Re i al: A is ic Collabo a ion and Imp o isa ion
be ween Humans and AI in Music and Visual
Keon Ju Ma e ick Lee
School o In e ac i e A s & Technology
Simon F ase Uni e si y
Vancou e , BC, Canada
[email p o ec ed]
Philippe Pasquie
School o In e ac i e A s & Technology
Simon F ase Uni e si y
Vancou e , BC, Canada
[email p o ec ed]
Jun Yu i
Independen A is
Vancou e , BC, Canada
[email p o ec ed]
Abs ac
Re i al is an o iginal li e audio isual pe o mance and imp o isa ion by he
a is collec i e K-Phi-A, me ging human a is y wi h AI musicianship o c a
elec onic music accompanied by esponsi e isuals. The piece ea u es eal- ime
co-c ea i e imp o isa ion be ween a pe cussionis , an elec onic music pe o me ,
and AI musical agen s. These agen s, ained on wo ks by la e compose s as well
as he collec i e’s own epe oi e, ac i ely espond o human inpu and emula e
sophis ica ed musical s yles. Complemen ing he sound, an AI-powe ed isual
syn hesize —guided by a li e VJ—gene a es e ol ing image y in sync wi h he
pe o mance. Re i al exempli ies he c ea i e syne gy be ween human pe o me s
and a i icial in elligence in imp o isa ional a .
1 A wo k concep
Re i al is an o iginal li e audio isual a wo k and pe o mance by he a is collec i e K-Phi-
A, explo ing he c ea i e po en ial o eal- ime human-AI collabo a ion in musical imp o isa ion.
Blending elec onic music, pe cussion, and gene a i e isuals, he pe o mance ea u es a dynamic
in e play be ween wo human pe o me s—a pe cussionis and a li e elec onic musician—and AI
musical agen s, including MACAT and MACa aRT (Lee and Pasquie , 2024). T ained on a cu a ed
co pus consis ing o he collec i e’s own composi ions and wo ks by deceased compose s, hese
agen s espond exp essi ely o li e inpu , emula ing in ica e musical s uc u es and s yles in eal
ime.
Accompanying he music is Au olume, an AI-powe ed isual syn hesize ained on public domain
image y (K aasch and Pasquie , 2022). Ope a ed by a li e VJ, Au olume gene a es isuals ha
e ol e in andem wi h he musical ex u e, con ibu ing o an imme si e audio isual expe ience.
Re i al o eg ounds he a o dances o AI as a co-c ea i e pa ne , p io i izing imp o isa ion o e
p e-composed in e ac ion and highligh ing collabo a i e agency be ween human pe o me s and
machine in elligence.
A he co e o ou human-machine in e ac ion amewo k lies a machine lis ening module embedded
wi hin he AI musical agen sys ems (Ta a and Pasquie , 2019). This module ope a es in conjunc ion
wi h a conduc o en i onmen buil using Cha aigne, Able on Li e, and he Au olume isual engine,
enabling eal- ime coo dina ion ac oss sonic and isual modali ies. The pe o mance is si ua ed wi hin
a co-c ea i e amewo k (Thelle and Wæ s ad, 2023) and de eloped h ough a esea ch-c ea ion
me hodology (S é ance and Lacasse, 2017), enabling an i e a i e, p ac ice-led explo a ion o music
and eac i e audio isuals as a uni ied a is ic medium.
P oceedings o he 6 h Con e ence on AI Music C ea i i y (AIMC 2025),
B ussels, Belgium, Sep embe 10 h-12 h
Ou app oach emphasizes a small da a mindse (Vigliensoni e al., 2022), ad oca ing o e hical and
anspa en da a use in c ea i e AI. By ocusing on cu a ed, pe sonalized da ase s, we educe eliance
on la ge-scale sc aping and mi iga e isks ela ed o in ellec ual p ope y, s ylis ic app op ia ion, and
en i onmen al impac . This app oach no only enhances a is agency and accoun abili y bu also
aligns wi h sus ainable and esponsible AI de elopmen p ac ices.
2 Real- ime co-c ea ion: human-AI music imp o isa ion and isual sys ems
2.1 Musical agen s o eal- ime co-c ea ion and music imp o isa ion
Musical agen s (Ta a and Pasquie , 2019) a e au onomous sys ems capable o execu ing c ea i e
musical asks by applying me hods om A i icial In elligence (Russell and No ig, 2010) and
Mul i-Agen Sys ems (Van de Hoek and Woold idge, 2008). As a cen al ocus wi hin he ield
o Musical Me ac ea ion (Pasquie e al., 2017), hese agen s a e designed o suppo eal- ime
in e ac ion, adap abili y, and au onomous beha io . They ope a e ac oss a ious c ea i e oles,
including gene a i e composi ion, in e ac i e imp o isa ion, and li e pe o mance accompanimen .
D awing om bo h scien i ic and a is ic pe spec i es, musical agen s employ a a ie y o sys em
a chi ec u es— anging om cogni i e o eac i e and hyb id models— ha enable dynamic, co-
c ea i e beha io in li e musical con ex s.
In he Re i al audio isual pe o mance, he a is collec i e K-Phi-A inco po a es MACAT and
MACa aRT (Lee and Pasquie , 2024)—ad anced musical agen sys ems de eloped a he Me ac ea ion
Lab o C ea i e AI
1
—as co e componen s o a eal- ime human-AI imp o isa ional amewo k.
These agen s a e designed o accompany a human pe cussionis and a li e elec onic music pe o me
by analyzing incoming audio using machine lis ening and gene a ing con ex -sensi i e musical
esponses. MACAT ex ends he MASOM a chi ec u e (Ta a and Pasquie , 2017) by in eg a ing
sel -o ganizing maps (SOM) (Kohonen, 1990), a ec i e compu ing, and he Fac o O acle (Assayag
and Dubno , 2004) o enable exp essi e conca ena i e sound syn hesis (Schwa z, 2006) and eal-
ime gene a i e sequencing (Lee and Pasquie , 2024). MACa aRT builds upon IRCAM’s Ca aRT
sys em (Schwa z e al., 2006), augmen ing i wi h empo al modeling and suppo ing bo h eac i e
and p oac i e imp o isa ion modes using hyb id echniques based on audio mosaicing (Lazie and
Cook, 2003) and lea ned sequence gene a ion.
These agen s a e ained on small, cu a ed audio co po a ailo ed o he pe o me s, ollowing a
“small da a” me hodology ha emphasizes e hical anspa ency, a is ic au ho ship, and en i onmen al
sus ainabili y (Vigliensoni e al., 2022; Vigliensoni and Fieb ink, 2025). In Re i al, MACAT suppo s
he exp essi e po en ial o Keon Ju Ma e ick Lee’s elec onic pe cussion se up by gene a ing imb ally
ich segmen s in esponse o eal- ime hy hmic and a ec i e inpu s, including MFCCs, ch oma,
alence, and a ousal. Simul aneously, MACa aRT engages wi h bo h Lee’s and Philippe Pasquie ’s
pe o mances, gene a ing and ecombining audio segmen s in eac ion o he e ol ing musical con ex .
The machine lis ening modules in bo h sys ems analyze incoming signals and map hem o a ained
ea u e space, acili a ing s ylis ically cohe en and emo ionally nuanced esponses. Th ough hese
co-c ea i e agen s, Re i al exempli ies a new pa adigm in AI-d i en music pe o mance, whe e
human musicians and in elligen sys ems engage in a con inuous, imp o isa ional dialogue (Thelle
and Wæ s ad, 2023).
2.2 Audio isual syn hesize and in e ac i e isual music
In he in e ac i e componen o ou audio isual sys em, spec al and Ba k coe icien s a e ex ac ed
om he audio ou pu s o Keon, Philippe, and he musical agen s. These coe icien s a e ansmi ed
ia Open Sound Con ol (OSC) messages o Au olume and VJ Amagi, enabling he gene a ion o
eac i e isuals in eal- ime, which a e shaped by VJ Amagi’s a is ic echniques o isual music.
Fu he mo e, he ex ac ed audio ea u es a e con eyed o he Digi al Mul iplex (DMX) in e ace o
con ol eac i e ligh ing, acili a ing isual ep esen a ion o each pe o me . The eac i e ligh ing
sys em enhances he isual in e p e a ion o each pe o me ’s ac ions by colou -mapping speci ic
audio ea u es o he ligh ing sys em, hus en iching he audience’s pe o mance expe ience.
1h ps://www.me ac ea ion.ne /p ojec s/maca -maca a -sys ems
2
2.3 Coo dina ing sound and isual laye s h ough conduc o so wa e
We u ilize Cha aigne
2
so wa e o con igu e ou conduc o sys em, which o ches a es communi-
ca ion, sound/ isual pa ame e con ol, and imelines o ou s uc u ed li e music imp o isa ion
and audio isual pe o mance. The pe o mance comp ises mul iple musical and isual hemes, each
employing dis inc sound pa ame e s o musical agen s, pe cussion samples, audio samples, and
isual elemen s. The conduc o coo dina es hese elemen s in eal- ime h ough OSC messages o
in e ac wi h he musical agen s, isual syn hesize , and eac i e ligh ing sys ems using DMX.
Acknowledgmen s and Disclosu e o Funding
The a is s g a e ully acknowledge he gene ous suppo ha made ou a wo k and music p ojec
possible. This p ojec was unded by he Canada Council o he A s (CCA), he Social Sciences
and Humani ies Resea ch Council o Canada (SSHRC), and he Na u al Sciences and Enginee ing
Resea ch Council o Canada (NSERC). Addi ional suppo was p o ided by he School o In e ac i e
A s and Technology (SIAT) a Simon F ase Uni e si y (SFU), as well as he Me ac ea ion Lab o
C ea i e AI, SIAT, SFU. Thei con ibu ions ha e been ins umen al in os e ing he de elopmen o
his in e disciplina y, co-c ea i e audio isual wo k.
Re e ences
Assayag, G. and Dubno , S. (2004). Using ac o o acles o machine imp o isa ion. So Compu ing,
8(9):604–610.
Kohonen, T. (1990). The sel -o ganizing map. P oceedings o he IEEE, 78(9):1464–1480.
K aasch, J. and Pasquie , P. (2022). Au olume-Li e: Tu ning GANs in o a Li e VJing ool. In
P oceedings o he Ten h Con e ence on Compu a ion, Communica ion, Aes he ics & X (xCoAx),
pages 168–185.
Lazie , A. and Cook, P. (2003). MOSIEVIUS: Fea u e d i en in e ac i e audio mosaicing. In Digi al
audio e ec s (da x). Ci esee .
Lee, K. J. M. and Pasquie , P. (2024). Musical Agen Sys ems: MACAT and MACa aRT. In In
P oceedings o he C ea i i y and Gene a i e AI NIPS (Neu al In o ma ion P ocessing Sys ems)
Wo kshop 2024.
Pasquie , P., Eigen eld , A., Bown, O., and Dubno , S. (2017). An in oduc ion o musical me ac e-
a ion. Compu e s in En e ainmen (CIE), 14(2):1–14.
Russell, S. J. and No ig, P. (2010). A i icial in elligence a mode n app oach. London.
Schwa z, D. (2006). Conca ena i e sound syn hesis: The ea ly yea s. Jou nal o New Music Resea ch,
35(1):3–22.
Schwa z, D., Belle , G., Ve b ugghe, B., and B i on, S. (2006). Real- ime co pus-based conca ena i e
syn hesis wi h ca a . In 9 h In e na ional Con e ence on Digi al Audio E ec s (DAFx), pages
279–282.
S é ance, S. and Lacasse, S. (2017). c ea ion in Music and he A s: Towa ds a Collabo a i e
In e discipline. Rou ledge.
Ta a , K. and Pasquie , P. (2017). MASOM: A musical agen a chi ec u e based on sel o ganizing
maps, a ec i e compu ing, and a iable ma ko models. In P oceedings o he 5 h In e na ional
Wo kshop on Musical Me ac ea ion (MUME 2017). A lan a, Geo gia, USA.
Ta a , K. and Pasquie , P. (2019). Musical agen s: A ypology and s a e o he a owa ds musical
me ac ea ion. Jou nal o New Music Resea ch, 48(1):56–105.
2h ps://benjamin.kupe be g. /cha aigne/en
3
Thelle, N. J. W. and Wæ s ad, B. I. (2023). Co-c ea i es spaces: The machine as a collabo a o . In
P oceedings o he In e na ional Con e ence on New In e aces o Musical Exp ession, pages
244–250. Zenodo.
Van de Hoek, W. and Woold idge, M. (2008). Mul i-agen sys ems. Founda ions o A i icial
In elligence, 3:887–928.
Vigliensoni, G. and Fieb ink, R. (2025). Da a- and in e ac ion-d i en app oaches o sus ained
musical p ac ices wi h machine lea ning. Jou nal o New Music Resea ch, 0(0):1–14.
Vigliensoni, G., Pe y, P., and Fieb ink, R. (2022). A small-da a mindse o gene a i e ai c ea i e
wo k. In p oceedings o he Gene a i e AI and HCI - CHI 2022 Wo kshop, New O leans, USA,
Online.
4
