Painting Music: Using artificial intelligence to create music from live painted drawings | Intellect Skip to content
1981
Volume 5, Issue 2
  • ISSN: 2057-0384
  • E-ISSN: 2057-0392

Abstract

This article describes the development of artificial intelligence (AI) techniques that monitor a painting or drawing evolving in real time and produce musical notes that relate to the individual elements of art as the artwork develops on the canvas. The article describes the practical approach required to capture the artwork unfolding in real time and then describes the framework used to develop the correlations between visual art and music. The AI technique exploits these areas of similarity within the two distinct artforms in order to respond to the live-painted elements and produce musical notes that reflect the development of the evolving artwork. A prototype of this system was implemented in a live stage performance at Aberdeen May Festival 2019 whose narrative centred on the question Other outputs of this project are a 20-minute film and a body of (tangible) visual artwork for digital platforms and gallery environments informed and inspired by AI. The integration of these disciplines through AI transforms a static artform into one that is dynamic, interactive, transformational, transient and temporal.

Loading

Article metrics loading...

/content/journals/10.1386/drtp_00033_1
2020-12-01
2024-04-27
Loading full text...

Full text loading...

References

  1. Fast, E., and Horvitz, E.. ( 2017;), ‘ Long-term trends in the public perception of artificial intelligence. ’, AAAI;, pp. 96369.
    [Google Scholar]
  2. Frigo, M., and Johnson, S. G.. ( 1998;), ‘ FFTW: An adaptive software architecture for the FFT. ’, in Proceedings of the International Conference on Acoustics, Speech, and Signal Processing, 15 May, Piscataway, NJ:: IEEE Signal Processing Society;, pp. 138184.
    [Google Scholar]
  3. Gagniuc, P. A.. ( 2017), Markov Chains: From Theory to Implementation and Experimentation, Hoboken, NJ:: John Wiley & Sons;.
    [Google Scholar]
  4. Graves, A.,, Liwicki, M.,, Fernandez, S.,, Bertolami, R.,, Bunke, H., and Schmidhuber, J.. ( 2009;), ‘ A novel connectionist system for improved unconstrained handwriting recognition. ’, IEEE Transactions on Pattern Analysis and Machine Intelligence, 31:5, pp. 85568.
    [Google Scholar]
  5. Graves, A., and Schmidhuber, J.. ( 2009;), ‘ Offline handwriting recognition with multidimensional recurrent neural networks. ’, in Y. Bengio,, D. Schuurmans,, J, Lafferty,, C. K. I. Williams, and A, Culotta. (eds), Advances in Neural Information Processing Systems 22, Vancouver, BC:: Neural Information Processing Systems Foundation;, pp. 54552.
    [Google Scholar]
  6. Kandinsky, W.. ( 1910;), Concerning the Spiritual in Art. , New York:: Dover Fine Art;.
    [Google Scholar]
  7. Kelley, P. G.,, Yang, Y.,, Heldreth, C.,, Moessner, C.,, Sedley, A.,, Kramm, A.,, Newman, D., and Woodruff, A.. ( 2020;), ‘ “Happy and assured that life will be easy 10 years from now”: Perceptions of artificial intelligence in 8 countries. ’, arXiv:2001.00081 , https://arxiv.org/pdf/2001.00081.pdf. Accessed 16 October 2020.
  8. Kohonen, T.,, Nieminen, I. T., and Honkela, T.. ( 2009;), ‘ On the quantization error in SOM vs. VQ: A critical and systematic study. ’, in Proceedings of the 7th International Workshop on Advances in Self-Organizing Maps, 8–10 June, Heidelberg:: Springer-Verlag;, pp. 13344.
    [Google Scholar]
  9. Kohonen, T.,, Schroeder, M. R., and Huang, T. S.. ( 2001), Self-Organizing Maps, , 3rd ed.., Heidelberg:: Springer-Verlag;.
    [Google Scholar]
  10. Linskens, E.. ( 2014), Music Improvisation Using Markov Chains, Maastricht:: Maastricht University;.
    [Google Scholar]
  11. Matlab ( 2019), Matlab Release 2019, Natick, MA:: The MathWorks, Inc;.
    [Google Scholar]
  12. Vi, C. T.,, Ablart, D.,, Gatti, E.,, Velasco, C., and Obrist, M.. ( 2017;), ‘ Not just seeing, but also feeling art: Mid-air haptic experiences integrated in a multisensory art exhibition. ’, International Journal of Human-Computer Studies, 108, December, pp. 114.
    [Google Scholar]
  13. Starkey, Andrew,, Steenhauer, Kate, and Caven, Jack. ( 2020;) ‘ Painting Music: Using artificial intelligence to create music from live painted drawings. ’, Drawing: Research, Theory, Practice, 5:2, pp. 209224, doi: https://doi.org/10.1386/drtp_00033_1
    [Google Scholar]
http://instance.metastore.ingenta.com/content/journals/10.1386/drtp_00033_1
Loading
/content/journals/10.1386/drtp_00033_1
Loading

Data & Media loading...

This is a required field
Please enter a valid email address
Approval was a success
Invalid data
An error occurred
Approval was partially successful, following selected items could not be processed due to error