Interactive Audio-Visual Systems in Music: Multi-Tool Approach by Simon Waite

Multi-tool suite of pieces Three compositions created with a multi-tool approach Simon Waite Research Programme: Digital Creativity

Contents Supporting Statement Research Output Project Details Page 2 Page 3 Page 4 Contextual Evidence Dissemination Page 11 Page 10 1

Project Details The output consists of three pieces: - Willow, - Unquiet - Brains Need Bodies Recorded live at the Staffordshire University Music & Sound studios, June 2018 2

300-Word Supporting Statement The three pieces created with the multi-tool approach investigate how interactive audio-visual systems might be used with traditional singer-songwriter material to create performances that exhibit a high degree of liveness (Sanden, 2013). The use of signal analysis and score-following techniques allow the human performer to play guitar and sing rather than operate controllers. The systems response components include multiple Max for Live devices creating melodic, harmonic and/or rhythmic accompaniment. Though unpredictable, the output of the devices within these systems were constrained in order to be compatible with the composed features of the song. These constraints were applied through the use of representational data from the instrumental/vocal audio signal, score-following techniques and sequenced material. These pieces have been published online, Brains Need Bodies Too has been performed at the Irish Sound Science and Technology Association (ISSTA) conference in 2019 whilst research findings have been shared at Loop 2017 (Berlin) and Innovations in Music 2017 (University of Westminster). 3

Research Output Willow: https://www.youtube.com/watch?v=Axzg8EchXcU Following the more experimental approaches of Rows, Columns, Collisions and I Begin Where You End, Willow demonstrates the use of indeterminate in a more traditional singer-songwriter context. The piece involves live recording of the guitar signal, which is used for loopers playing indeterminate melodies and granular synthesis to provide a background layer. When the guitar level exceeds a threshold, a spectral processing effect generates additional layers. The system is controlled by a score-follower that detects each section of the piece through analysis of guitar pitches (MIDI notes converted from the audio signal using Jam Origin s MIDI Guitar) and volume levels. The lyrics were written in a similar way to I Begin Where You End and are based on the story of the Willow Pattern (a popular ceramics pattern originated by Thomas Minton in Stoke-on-Trent). See https://youtu.be/42- BBrBItbo for an early version of the piece. 4

Research Output Score for Willow

Research Output Unquiet: https://www.youtube.com/watch?v=g_Xxo8w7G-g Unquiet was a piece that had existed as a demo version for a few years consisting of a guitar part, a drum loop and a vocal idea. A bassline was added, but something else was needed. Around the same time, I had begun work on an interactive-generative arpeggiator (Gen_Arp) using Max. This created unpredictable note patterns based on intervals between notes arriving at the input. Gen_Arp was converted to a Max for Live and used to generate a synth pattern, controlled by MIDI notes (using Jam Origin s MIDI Guitar) representing the live guitar signal. A backing vocal looping device that records and plays back the vocals from the live input and applies a variety of envelope settings was also added to the system. As with Willow, a score-follower detected the song s positions which changes the behaviour of the devices and controlled the playback of pre-recorded material from the demo. A demo of GenArp can be found at https://vimeo.com/70554334 6

Research Output Score for Unquiet

Research Output Brains Need Bodies Too: https://www.youtube.com/watch?v=tMOfAUw9tr0 / https://vimeo.com/user5921885 Brains Need Bodies Too uses a similar system as Willow and Unquiet. The guitar signal is converted to MIDI by Jam Origin s MIDI Guitar. Trigger melodies in the guitar part can then be used to detect the onset of the next song section (verse, chorus etc) using a score-following device. This device controls the behaviour of the various system devices. The piece explores the conflict and tension between humans and technology. This is reflected in the heads up display (HUD) approach of the visuals and the final section of the piece, which uses artificial intelligence to learn the melodies in the guitar part. Inspired by a Syd Barrett anecdote, the human performer keeps changing what they are playing while trying to follow the system s melodies, setting up an impossible cycle of imitation. 8

Research Output Score for Brains Need Bodies Too

Dissemination All 3 pieces recorded live at the Staffordshire University Music & Sound studios, June 2018 and made available on YouTube: https://www.youtube.com/watch?v=Axzg8EchXcU&list=PLEf_T--_c4UoAigrctftr7bVlqrq8XUKn Brains Need Bodies Too has been performed at the Irish Sound Science and Technology Association (ISSTA) conference in 2019 Research findings have been shared at Loop 2017 (Berlin) and Innovations in Music 2017 (University of Westminster). Further videos and software links available on: www.nationaltrevor.com 10

Contextual Evidence 11

Contextual Evidence 12