Music for Multimodal Agents – Experiment in Improvisation and Feedback Loop

Proceedings of the 6th Conference on AI Music Creativity (AIMC 2025), Brussels, Belgium, September 10th-12th Music for Multimodal Agents – Experiment in Improvisation and Feedback Loop Veera Jussila Artist, Machine Learning Engineer [email protected] Derek Yau Multi-instrumentalist [email protected] Abstract Music for AI Audience is a live, collaborative piece between custom deep learning software and a musician. In the 10-15 minute piece, multimodal LLM agents observe an improvised sound performance and let their minds wander through digital archives. This machine listening and machine seeing results in outputs that, on their part, act as prompts for the musician in the loop. The piece not only explores the creative possibilities of improvising in front of a non-human audience, but the potential and limits of machine sentience in experiencing art. 1 Introduction Music for AI Audience is a live performance based on a feedback loop between an improvising musician and a deep learning software. It investigates a form of performance which is primarily aimed at a non-human audience. Instead of controlling an AI system, the artist lets themselves be inspired by the reactions of the LLM-driven audience. The piece builds on Jussila’s research in subverting AI systems to introduce new forms of creative communication, and extends Yau’s practice in improvising. 2 Behind the scenes of AI audience 2.1 Agents as probe for investigation The software is a Python application programmed by Jussila. The starting point was to explore the sentient potential of LLM agents within a live performance. In her AI piece The Seeker, Thompson (2019) used machine vision to create a non-human entity that described the world through security cameras. Music for AI Audience investigates the more immediate, intimate potential of improvising for agents. Agents are AI systems that use text prompts and LLMs to understand and implement tasks. Using Päpper’s (2023) repository as an early reference, the current setup uses two agents to form the audience. 2.2 Audience with different sensing capabilities We can refer to the agents as Associator and Crawler. HuggingFace’s (2025) smolagents library is used for orchestration, the engine LLM set to Qwen’s (2024) Qwen2.5-Coder-32B-Instruct. Associator uses OpenCV video feed as its input. The tool of this agent is image search. Using Unum Cloud’s (2023) lightweight uform-gen model, it captions the webcam frames, describing the musician, instrument and the ambience. This caption is embedded using Sentence Transformers (Reimers and Gurewych, 2019) implementation of OpenAI’s (2021) CLIP model, and the embedding is used to return the best matching image from the database. The agent’s text generations are visible on the terminal, and snapshots of the musician and the resulting image associations are popping on screen, too. Crawler utilises microphone input. The tool of this agent is web search. The agent records 10 second snapshots of the music and captions these using a transformer-based model by Doh et al. (2023), describing the instrument and ambience of the performance. Using this caption, the agent searches 2 for related content online. The agent’s text generations are visible on the terminal, and the browsing sessions are popping on screen (Figure 1). Figure 1: Screenshot of AI audience outputs. The current setup forms a minimal MVP. The makeup of the AI audience is very modular, as its sensing capabilities and actions can be changed easily. 3 Going live The 10-15 minute performance is a collaboration between Jussila and Yau. The current demo consists of 1) Jussila navigating a laptop with a USB webcam and external microphone 2) Yau playing a cello 3) a laptop projection on the wall. The musician observes the projection to see the agents’ outputs and uses them as prompts. The projection shows agent-generated reasoning and captions, alongside the image associations and web searches run by the agents, plus occasional visual snapshots of the performance itself. This feedback loop unlocks a new dimension in live creation: instead of reading the audience’s body language or taking cues from fellow performers, the musician becomes part of the chain where their own output and body language powers the visual prompts (Figure 2). The musician is encouraged to use methods which aren’t strictly traditional, but instead involve colouring the sound via extended classical techniques, effects and layering to evoke interest and variety in agent output, such that the AI audience becomes part of the performance itself. 3 Figure 2: Music for AI Audience, live performance. Picture from video by Thomas Rosser. There is a meta element to this work, as Music for AI Audience is being watched by a human audience. Sometimes Jussila directs the webcam to the human audience, leading the visual agent to analyse the observers. The piece had its premiere in London in September 2024. 4 Discussion 4.1 Musician as part of prompting loop Interesting results can be observed when the musician allows themselves to be prompted like agents, by agents running on prompts. For example, the AI audience won’t tolerate much repetition, otherwise the outputs start to repeat themselves and become less valuable as prompts. Thus, the feedback loop pushes the musician to explore sound space that intrigues the machine listener. Additionally, moments of silence are analysed as input, sometimes resulting in thrilling interpretations - the sound of the hard-working GPU detected as a Tibetan singing bowl. Secondly, Music for AI Audience investigates machine sentience within live art experience, a realm that is thought to be dominantly human. In industry, part of AI’s attraction is in the (seeming) lack of ambiguity (McQuillan, 2019, p. 164). Creative computing, on the contrary, thrives in this grey area. In Museum of Borderlands, Jussila (2020) demonstrated how the deep learning classifier struggled to separate images of flowers from viruses, inviting a conversation about AI classifying ambiguous things as threats. Small datasets (Jussila, 2022) are another way to explore the expressive capabilities of AI. Broad (2019) has explored generative space by training models without data. Brain (2025) talks about eccentric engineering, where experimental software can be used to probe systems with more focus on non-human actors and environment. Music for AI Audience plays with the tension of agentic autonomy and the (human-induced) limitations of machines’ dreaming capabilities. 4.2 Future work Getting the agents to react to each other will open a whole new path of research. Jussila has already experimented with some pre-concert conversation by agents. It is also possible to build more developed audience members with personal history and music knowledge. Next tasks include adding more advanced listening and seeing models to add depth to the agents’ experience, and creating more activities the audience engages in (image generation with diffusion, 4 generating fiction, clicking deeper into web results). The agents could also observe a full improv orchestra. One option is to update the audience to react with sound, changing the dynamic to jamming. Already in its current form, Music for AI Audience integrates acoustic, minimal improvisation into the emerging field of creative computing. References Brain, T. (2025) Eccentric Engineering. Available at: https://tegabrain.com/Eccentric-Engineering (Accessed: 29 March 2025). Broad, T. (2019) un(stable) equilibrium. Available at: https://terencebroad.com/works/unstable-equilibrium (Accessed: 29 March 2025). Doh, S., Choi, K., Lee, J., and Nam, J. (2023) LP-MusicCaps: LLM-Based Pseudo Music Captioning. Available at: https://arxiv.org/pdf/2307.16372 (Accessed: 29 March 2025). HuggingFace (2025) smolagents [Software]. Available at: https://github.com/huggingface/smolagents (Accessed: 5 April 2025). Jussila, V. (2020) Museum of Borderlands. Available at: https://www.veerajussila.com/code-and-art/ museumof-borderlands (Accessed: 29 March 2025). Jussila, V. (2022) Small datasets. Available at: https://www.veerajussila.com/code-and-art/talk-small-datasets (Accessed: 29 March 2025). McQuillan, D. (2019) ‘The Political Affinities of AI’, in A. Sudmann (ed.) The Democratization of Artificial Intelligence: Net Politics in the Era of Learning Algorithms. Bielefeld: transcript Verlag, pp. 163–173. Available at: https://library.oapen.org/handle/20.500.12657/43874 (Accessed: 29 March 2025). OpenAI (2021) CLIP [Software]. Available at: https://github.com/openai/CLIP (Accessed: 29 March 2025). Päpper, M. (2023) LLM Agents [Software]. Available at: https://github.com/mpaepper/llm_agents (Accessed: 29 March 2025). Qwen (2024) Qwen2.5-Coder-32B-Instruct [Software]. Available at: https://huggingface.co/Qwen/Qwen2.5Coder-32B-Instruct (Accessed: 5 April 2025). Reimers, N. and Gurewych, I. (2019) Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. Available at: https://arxiv.org/abs/1908.10084 (Accessed: 7 April 2025). Thompson, N. (2019) The Seeker. Available at: https://nyethompson.net/works/the-seeker.html (Accessed: 29 March 2025). Unum Cloud (2023). UForm [Software]. Available at: https://github.com/unum-cloud/uform (Accessed: 29 March 2025).