Things AIn't What They Used To Be

Proceedings of the 6th Conference on AI Music Creativity (AIMC 2025), Brussels, Belgium, September 10th-12th Conference on AI Music Creativity (AIMC) 2025 Things AIn’t What They Used To Be (composition, ~15 minutes) Mark Whitlam Department of Music University of Bristol [email protected] Abstract This performance work—titled in reference to Duke Ellington’s big band jazz classic, released over sixty years ago—offers a gentle provocation, contrasting traditional approaches to jazz improvisation with emerging paradigms in human–AI interaction. Combining real-time machine learning and deep learning tools, the piece stages a live collaboration between improvising human musicians and generative AI agents. Central to the work is a subversion of the established technique of the contrafact, whereby new melodies are composed over pre-existing chord progressions. Here, the process is inverted: AI agents are tasked with reharmonising composed melodic lines, thereby disrupting the expected harmonic framework. This indeterminacy both encourages and challenges the performers to find new musical responses. Leveraging technologies such as Somax2, RAVE, Mosaïque, and Google MediaPipe within MaxMSP, the system enables algorithmic agents to act as both collaborative and disruptive partners in the performance loop. These agents generate unexpected musical gestures and offer novel, interactive modalities that stimulate and provoke the performers. The result is an evolving musical language that emerges from the entangled dynamics of this extended network of human and machine improvisers. 1. The Actor-network of improvising musicians and AI agents Building on the widely used approach of corpus-based concatenative synthesis in electroacoustic composition (Hackbarth et al., 2013; Sturm, 2006), this composition uses a variety of AI agents, listening both to their human co-performers and to one another, creating a complex network of actants (Latour, 1996). This practice-based research looks through the lens of Actor-Network Theory and 4E Cognition to generate meaningful insights from the lived-experiences of improvising musicians to inform future compositional approaches with generative AI. It explores the complex dynamics of a co-improvising extended human-computer network and how interacting with AI agents may stimulate new musical language from human performers and how this informs novel compositional paradigms. This 15-minute performance features two sections of a longer piece originally written for tenor saxophone with live electronics and drum kit/percussion with live electronics, and various AI agents listening to both players and spatialised in Dolby Atmos 9.1.4. For this performance at AIMC, the saxophone is substituted for trumpet and the audio processing is rendered in stereo. 2 2. AI agents and their role in the human-machine network Drummer and composer Mark Whitlam is joined by trumpeter Celeste Cantor-Stephens for this performance, both of whom have been internationally performing musicians over the past two decades. They draw on this rich lived experience to navigate the often unpredictable interjections from their machine co-performers, but are also challenged to find new musical responses to the AI and one another. The performance uses a MaxMSP to combine multiple instances of IRCAM’s Somax2 providing musical gestures in response to both trumpet and percussion inputs. Somax2 draws upon various corpora of the composer’s own language and that of Bill Evans’ jazz standards Very Early and Nardis to provide melodic and harmonic responses to the human performers. Multiple instances of the AI agent allows for self-referential interaction with the various audio and MIDI corpora, adding greater emphasis on the agency of the artificial intelligence system in the live ensemble. Each time the piece is performed, new corpora of the previous performers’ playing are incorporated into the system’s ‘memory’, resulting in an ever-evolving, hauntological musical palette from which the AI draws and responds. 3. Embodied and enacted interactions Nabi et al. (2024) explored embodied interaction with the timbral transfer variational autoencoder RAVE (Caillon & Esling, 2022) through interdisciplinary research involving dancers equipped with wireless gyroscopic motion sensors. In contrast, the composer in this project uses MediaPipe as an interlocutor, with hand and body landmark recognition models transcoding performers’ movements into control data for two distinct AI agents. The first is RAVE, where dynamic navigation of the latent space generates novel timbres that stimulate further improvisatory responses from the performers. The second agent, Mosaïque (Thibault, 2024), employs corpusbased concatenative synthesis to arrange sonic fragments across a three-dimensional visual ‘map’. A projected image of this provides the audience with visual cues about the real-time interactions within the human-computer network, illustrating how the musicians’ gestures navigate and shape the sonic collage. Artist Biographies: Mark Whitlam is a composer, drummer/percussionist, and educator. He has recorded with and performed internationally with luminaries of the UK jazz scene, including ECM recording artists Andy Sheppard, Iain Ballamy, and Jason Rebello (Sting), as well as cross-genre projects with Portishead guitarist Adrian Utley and Goldfrapp’s Will Gregory. Mark’s compositions and performances have received wide airplay on BBC 2, 3, 6, and Jazz FM, and he has received commissions for an HBO television miniseries. He is currently undertaking a PhD in composition at the University of Bristol, exploring post-spectral language and the use of AI-based interactive agents within human, live, improvisation-led musical creation. Celeste Cantor-Stephens is a highly accomplished improviser, composer, writer, and teacher, and is active on the music scenes both in the UK and New York. Her versatile trumpet embraces the creative and exploratory, spanning a range of practices and traditions: from free improvisation to klezmer, via jazz, dub-reggae, classical, and more. She has collaborated with a diverse range of artists, from percussionist Billy Martin and Soundpainting pioneer Walter Thompson to the late dub maestro Lee ‘Scratch’ Perry. 3 Both Mark and Celeste teach undergraduate and postgraduate music students in the UK at Bath Spa University and BIMM University, Bristol. The composer would like to thank: The University of Bristol for funding towards this conference performance. Paul Hamesse at HD Studio, Brussels for drumkit hire. Musicians who have graciously contributed improvised performances for the RAVE and Mosaïque corpora: Lucy Anne-Allen (violin), Henrik Jensen and Ashley John Long (double bass), Jake McMurchie (saxophone), Joseph Vissaux (percussion), Nicholas Malcolm (trumpet). References Caillon, A., & Esling, P. (2022). RAVE: A variational autoencoder for fast and high-quality neural audio synthesis. In Proceedings of the 25th International Conference on Digital Audio Effects (DAFx22), Vienna, Austria, September 6–10, 2022. https://dafx2020.mdw.ac.at/proceedings/papers/DAFx20in22_paper_49.pdf Hackbarth, B., Schnell, N., Esling, P., & Schwarz, D. (2013). Composing morphology: Concatenative synthesis as an intuitive medium for prescribing sound in time. Contemporary Music Review, 32(1), 49–59. https://doi.org/10.1080/07494467.2013.774513 Latour, B. (1996). On actor-network theory: A few clarifications. Soziale Welt, 47(4), 369–381. http://www.jstor.org/stable/40878163 Nabi, S., Esling, P., Peeters, G., & Bevilacqua, F. (2024). Embodied exploration of deep latent spaces in interactive dance-music performance. In Proceedings of the 9th International Conference on Movement and Computing (MOCO ’24), Utrecht, Netherlands, May 2024. https://doi.org/10.1145/3658852.3659072 Sturm, B. L. (2006). Adaptive concatenative sound synthesis and its application to micromontage composition. Computer Music Journal, 30(4), 46–66. https://www.jstor.org/stable/4617983 Thibault, D. (2024, August 24). Mosaïque – Concatenative synthesis instrument for the practicing musicians [Workshop presentation]. AIMC 2024. https://aimc2024.pubpub.org/pub/buh7kcah/release/1