Revival: Artistic Collaboration and Improvisation between Humans and AI in Music and Visual

Revival: Artistic Collaboration and Improvisation between Humans and AI in Music and Visual Keon Ju Maverick Lee School of Interactive Arts & Technology Simon Fraser University Vancouver, BC, Canada [email protected] Philippe Pasquier School of Interactive Arts & Technology Simon Fraser University Vancouver, BC, Canada [email protected] Jun Yuri Independent Artist Vancouver, BC, Canada [email protected] Abstract Revival is an original live audiovisual performance and improvisation by the artist collective K-Phi-A, merging human artistry with AI musicianship to craft electronic music accompanied by responsive visuals. The piece features real-time co-creative improvisation between a percussionist, an electronic music performer, and AI musical agents. These agents, trained on works by late composers as well as the collective’s own repertoire, actively respond to human input and emulate sophisticated musical styles. Complementing the sound, an AI-powered visual synthesizer—guided by a live VJ—generates evolving imagery in sync with the performance. Revival exemplifies the creative synergy between human performers and artificial intelligence in improvisational art. 1 Artwork concept Revival is an original live audiovisual artwork and performance by the artist collective K-PhiA, exploring the creative potential of real-time human-AI collaboration in musical improvisation. Blending electronic music, percussion, and generative visuals, the performance features a dynamic interplay between two human performers—a percussionist and a live electronic musician—and AI musical agents, including MACAT and MACataRT (Lee and Pasquier, 2024). Trained on a curated corpus consisting of the collective’s own compositions and works by deceased composers, these agents respond expressively to live input, emulating intricate musical structures and styles in real time. Accompanying the music is Autolume, an AI-powered visual synthesizer trained on public domain imagery (Kraasch and Pasquier, 2022). Operated by a live VJ, Autolume generates visuals that evolve in tandem with the musical texture, contributing to an immersive audiovisual experience. Revival foregrounds the affordances of AI as a co-creative partner, prioritizing improvisation over pre-composed interaction and highlighting collaborative agency between human performers and machine intelligence. At the core of our human-machine interaction framework lies a machine listening module embedded within the AI musical agent systems (Tatar and Pasquier, 2019). This module operates in conjunction with a conductor environment built using Chataigne, Ableton Live, and the Autolume visual engine, enabling real-time coordination across sonic and visual modalities. The performance is situated within a co-creative framework (Thelle and Wærstad, 2023) and developed through a research-creation methodology (Stévance and Lacasse, 2017), enabling an iterative, practice-led exploration of music and reactive audiovisuals as a unified artistic medium. Proceedings of the 6th Conference on AI Music Creativity (AIMC 2025), Brussels, Belgium, September 10th-12th Our approach emphasizes a small data mindset (Vigliensoni et al., 2022), advocating for ethical and transparent data use in creative AI. By focusing on curated, personalized datasets, we reduce reliance on large-scale scraping and mitigate risks related to intellectual property, stylistic appropriation, and environmental impact. This approach not only enhances artist agency and accountability but also aligns with sustainable and responsible AI development practices. 2 Real-time co-creation: human-AI music improvisation and visual systems 2.1 Musical agents for real-time co-creation and music improvisation Musical agents (Tatar and Pasquier, 2019) are autonomous systems capable of executing creative musical tasks by applying methods from Artificial Intelligence (Russell and Norvig, 2010) and Multi-Agent Systems (Van der Hoek and Wooldridge, 2008). As a central focus within the field of Musical Metacreation (Pasquier et al., 2017), these agents are designed to support real-time interaction, adaptability, and autonomous behavior. They operate across various creative roles, including generative composition, interactive improvisation, and live performance accompaniment. Drawing from both scientific and artistic perspectives, musical agents employ a variety of system architectures—ranging from cognitive to reactive and hybrid models—that enable dynamic, cocreative behavior in live musical contexts. In the Revival audiovisual performance, the artist collective K-Phi-A incorporates MACAT and MACataRT (Lee and Pasquier, 2024)—advanced musical agent systems developed at the Metacreation Lab for Creative AI 1 —as core components of a real-time human-AI improvisational framework. These agents are designed to accompany a human percussionist and a live electronic music performer by analyzing incoming audio using machine listening and generating context-sensitive musical responses. MACAT extends the MASOM architecture (Tatar and Pasquier, 2017) by integrating self-organizing maps (SOM) (Kohonen, 1990), affective computing, and the Factor Oracle (Assayag and Dubnov, 2004) to enable expressive concatenative sound synthesis (Schwarz, 2006) and realtime generative sequencing (Lee and Pasquier, 2024). MACataRT builds upon IRCAM’s CataRT system (Schwarz et al., 2006), augmenting it with temporal modeling and supporting both reactive and proactive improvisation modes using hybrid techniques based on audio mosaicing (Lazier and Cook, 2003) and learned sequence generation. These agents are trained on small, curated audio corpora tailored to the performers, following a “small data” methodology that emphasizes ethical transparency, artistic authorship, and environmental sustainability (Vigliensoni et al., 2022; Vigliensoni and Fiebrink, 2025). In Revival, MACAT supports the expressive potential of Keon Ju Maverick Lee’s electronic percussion setup by generating timbrally rich segments in response to real-time rhythmic and affective inputs, including MFCCs, chroma, valence, and arousal. Simultaneously, MACataRT engages with both Lee’s and Philippe Pasquier’s performances, generating and recombining audio segments in reaction to the evolving musical context. The machine listening modules in both systems analyze incoming signals and map them to a trained feature space, facilitating stylistically coherent and emotionally nuanced responses. Through these co-creative agents, Revival exemplifies a new paradigm in AI-driven music performance, where human musicians and intelligent systems engage in a continuous, improvisational dialogue (Thelle and Wærstad, 2023). 2.2 Audiovisual synthesizer and interactive visual music In the interactive component of our audiovisual system, spectral and Bark coefficients are extracted from the audio outputs of Keon, Philippe, and the musical agents. These coefficients are transmitted via Open Sound Control (OSC) messages to Autolume and VJ Amagi, enabling the generation of reactive visuals in real-time, which are shaped by VJ Amagi’s artistic techniques for visual music. Furthermore, the extracted audio features are conveyed to the Digital Multiplex (DMX) interface to control reactive lighting, facilitating visual representation for each performer. The reactive lighting system enhances the visual interpretation of each performer’s actions by colour-mapping specific audio features to the lighting system, thus enriching the audience’s performance experience. 1https://www.metacreation.net/projects/macat-macatart-systems 2 2.3 Coordinating sound and visual layers through conductor software We utilize Chataigne 2 software to configure our conductor system, which orchestrates communication, sound/visual parameter control, and timelines for our structured live music improvisation and audiovisual performance. The performance comprises multiple musical and visual themes, each employing distinct sound parameters for musical agents, percussion samples, audio samples, and visual elements. The conductor coordinates these elements in real-time through OSC messages to interact with the musical agents, visual synthesizer, and reactive lighting systems using DMX. Acknowledgments and Disclosure of Funding The artists gratefully acknowledge the generous support that made our artwork and music project possible. This project was funded by the Canada Council for the Arts (CCA), the Social Sciences and Humanities Research Council of Canada (SSHRC), and the Natural Sciences and Engineering Research Council of Canada (NSERC). Additional support was provided by the School of Interactive Arts and Technology (SIAT) at Simon Fraser University (SFU), as well as the Metacreation Lab for Creative AI, SIAT, SFU. Their contributions have been instrumental in fostering the development of this interdisciplinary, co-creative audiovisual work. References Assayag, G. and Dubnov, S. (2004). Using factor oracles for machine improvisation. Soft Computing, 8(9):604–610. Kohonen, T. (1990). The self-organizing map. Proceedings of the IEEE, 78(9):1464–1480. Kraasch, J. and Pasquier, P. (2022). Autolume-Live: Turning GANs into a Live VJing tool. In Proceedings of the Tenth Conference on Computation, Communication, Aesthetics & X (xCoAx), pages 168–185. Lazier, A. and Cook, P. (2003). MOSIEVIUS: Feature driven interactive audio mosaicing. In Digital audio effects (dafx). Citeseer. Lee, K. J. M. and Pasquier, P. (2024). Musical Agent Systems: MACAT and MACataRT. In In Proceedings of the Creativity and Generative AI NIPS (Neural Information Processing Systems) Workshop 2024. Pasquier, P., Eigenfeldt, A., Bown, O., and Dubnov, S. (2017). An introduction to musical metacreation. Computers in Entertainment (CIE), 14(2):1–14. Russell, S. J. and Norvig, P. (2010). Artificial intelligence a modern approach. London. Schwarz, D. (2006). Concatenative sound synthesis: The early years. Journal of New Music Research, 35(1):3–22. Schwarz, D., Beller, G., Verbrugghe, B., and Britton, S. (2006). Real-time corpus-based concatenative synthesis with catart. In 9th International Conference on Digital Audio Effects (DAFx), pages 279–282. Stévance, S. and Lacasse, S. (2017). creation in Music and the Arts: Towards a Collaborative Interdiscipline. Routledge. Tatar, K. and Pasquier, P. (2017). MASOM: A musical agent architecture based on self organizing maps, affective computing, and variable markov models. In Proceedings of the 5th International Workshop on Musical Metacreation (MUME 2017). Atlanta, Georgia, USA. Tatar, K. and Pasquier, P. (2019). Musical agents: A typology and state of the art towards musical metacreation. Journal of New Music Research, 48(1):56–105. 2https://benjamin.kuperberg.fr/chataigne/en 3 Thelle, N. J. W. and Wærstad, B. I. (2023). Co-creatives spaces: The machine as a collaborator. In Proceedings of the International Conference on New Interfaces for Musical Expression, pages 244–250. Zenodo. Van der Hoek, W. and Wooldridge, M. (2008). Multi-agent systems. Foundations of Artificial Intelligence, 3:887–928. Vigliensoni, G. and Fiebrink, R. (2025). Dataand interaction-driven approaches for sustained musical practices with machine learning. Journal of New Music Research, 0(0):1–14. Vigliensoni, G., Perry, P., and Fiebrink, R. (2022). A small-data mindset for generative ai creative work. In proceedings of the Generative AI and HCI - CHI 2022 Workshop, New Orleans, USA, Online. 4