Cybe nesis: An Imp o isa ional Pe o mance Sys em
Explo ing Ges u e Con ol o Ha dwa e Syn hesize s
Da id Piazza
CIRMMT
Uni e si é de Mon éal
200, a . Vincen -d’Indy, Mon éal
[email p o ec ed]
Abs ac
Cybe nesis is a pe o mance piece ha explo es he in e ac ion be ween human
ges u e, machine lea ning, and eal- ime sound syn hesis. Using a Leap Mo ion
con olle , he pe o me ’s hand mo emen s a e cap u ed and analyzed by cus om
so wa e. This ges u al da a ains a mul ilaye pe cep on (MLP), a o m o neu al
ne wo k, which in u n p edic s and in luences he in e nal s a e o a complex
ha dwa e sound syn hesis sys em. The co e o he pe o mance lies in he eal- ime
explo a ion o nonlinea mapping unc ions h ough linea eg ession, na iga ed
spa ially h ough he pe o me ’s lis ening and in ui i e hand mo emen s. This
c ea es a dynamic eedback loop whe e he pe o me and he model co-c ea e
he sonic ou pu , posi ioning he lea ning algo i hm no me ely as a ool, bu as
an ac i e pa icipan in he imp o isa ional p ocess media ed by he pe o me ’s
embodied in e ac ion.
1
In oduc ion: Machine Lea ning, Imp o isa ion, and Embodied In e ac ion
The in e sec ion o machine lea ning (ML) and musical c ea i i y con inues o yield no el o ms o
exp ession and in e ac ion. Cybe nesis is p esen ed as a con ibu ion o his ield, speci ically ocusing
on li e, imp o isa ional pe o mance using ML as a media ing laye be ween pe o me ges u e
and complex sound gene a ion. The wo k le e ages ML echniques o c ea e in ica e, e ol ing
ela ionships be ween embodied human inpu and he pa ame e s o a ha dwa e-based sound syn hesis
sys em, Des iny+’s Model Q2 phase modula ion syn hesize and En anglemen Space e ec s uni .
Cen al o Cybe nesis is he explo a ion o nonlinea mapping s a egies, mo ing beyond di ec
one- o-one co espondences be ween ges u e and sound (Hun e al., 2002). Ins ead, a mul ilaye
pe cep on (MLP) is ained based on he pe o me ’s hand mo emen s de ec ed by a Leapmo ion
in a ed came a, using he Fluid Co pus Manipula ion oolki (T emblay e al., 2021). This MLP
lea ns associa ions be ween hand posi ions and se s o a ying in e nal s a es wi hin he sound
syn hesis ha dwa e. The pe o me na iga es his complex con ol space h ough lis ening and sub le
hand mo emen s, engaging in an imp o isa ional dialogue wi h he sys em.
Du ing he piece, he pe o me is ac i ely shaping he beha iou o he p edic ion mechanism and
esponding o he sonic esul s in a igh eedback cycle. The model ac s as an in elligen in e media y,
ansla ing high-dimensional ges u al inpu ec o s in o con ol signals o a high-dimensional
syn hesis engine, bu i s beha iou is cons an ly in luenced and s ee ed by he pe o me ’s ac ions
and audi o y eedback (Ca amiaux e al., 2014; Van No e al., 2013). This c ea es a scena io whe e
agency is sha ed and nego ia ed be ween he human a is and he machine demons a ing in elligen
beha iou (Gio i e al., 2022), os e ing eme gen musical momen s cha ac e is ic o imp o isa ional
p ac ice (Bo go, 2006).
P oceedings o he 6 h Con e ence on AI Music C ea i i y (AIMC 2025),
B ussels, Belgium, Sep embe 10 h-12 h
2 Me hods: Sys em Design and In e ac ion
2.1 Ges u al Inpu : Leapmo ion Hand T acking
Hand ges u es p o ide he p ima y inpu o he sys em. A Leap Mo ion con olle cap u es spa ial
da a abou he pe o me ’s hands, including posi ion, o ien a ion, inge angles, and eloci y. This
high-dimensional da a s eam o e s signi ican exp essi e po en ial compa ed o adi ional con olle s
bu also p esen s challenges o di ec mapping o syn hesis pa ame e s (Leeuw, 2021). A Max/MSP
pa ch p ocesses his aw da a, ex ac ing ele an ea u es sui able o inpu in o he machine lea ning
model.
The co e o he in e ac ion logic esides in an MLP implemen ed in he pa ch. Unlike p ede ined
mapping unc ions, he MLP lea ns ela ionships be ween he p ocessed ges u al ea u es and NRPN
numbe alue a ays ep esen ing he in e nal s a e o he syn hesize .
Du ing pe o mance, he pe o me ’s ges u es p o ide inpu ec o s o he ained MLP. The MLP’s
ou pu laye hen gene a es p edic ions o he a ge syn hesize pa ame e s. While his p edic ion
p ocess is de e minis ic, he MLP in oduces a laye o abs ac ion and nonlinea i y, ans o ming
ges u al inpu s in o ha dwa e con ol signals based on a lea ned in e nal ep esen a ion (Fieb ink
and Ca amiaux, 2016; Esling e al., 2019). The pe o me in luences he aining da a o he MLP
h ough hei mo emen s, e ec i ely eaching he sys em p e e ed ges u e-s a e associa ions o
explo ing he eme gen possibili ies o he lea ned mapping.
2.2 Sound Syn hesis Ha dwa e
The a ge o he MLP’s p edic ions is a ha dwa e sound syn hesis pla o m de eloped by Des-
iny+. The key aspec is ha he ha dwa e possesses a complex in e nal s a e space, in ol ing
nume ous pa ame e s ha in e ac in non- i ial ways, making di ec manual con ol challenging
du ing imp o isa ion. Mo eo e , he physical componen s hemsel es only allow o modula e ou
o he pa ame e s a once. The MLP se es o na iga e his complexi y, ansla ing changes in he
pe o me ’s hand posi ions in o meaning ul changes wi hin he syn hesis engine, d awing pa allels
wi h con ol s a egies o modula sys ems (Whi e, 2022; Ca ey, 2023).
2.3 Imp o isa ional In e ac ion: Lis ening and Spa ial Explo a ion
The pe o mance i sel is an imp o isa ion based on na iga ing he possibili ies a o ded by he
sys em. The pe o me does no in e ac wi h he MLP o he ha dwa e h ough explici p og amming
o pa ame e weaking du ing he pe o mance. Ins ead, in e ac ion occu s h ough con inuous ges u e
and, c ucially, deep lis ening (Ca amiaux e al., 2014; Van No e al., 2013).
By mo ing hei hands in he Leapmo ion’s sensing space, he pe o me explo es he lea ned mapping
unc ion. The sonic ou pu om he ha dwa e syn hesize p o ides immedia e audi o y eedback.
This eedback in o ms he pe o me ’s subsequen ges u es, c ea ing a closed loop. The explo a ion is
spa ial: di e en egions o ypes o mo emen co espond o di e en sonic beha iou s as in e p e ed
by he MLP and ealised by he syn hesize . The pe o me lea ns o associa e ges u al mo i s wi h
sonic ou comes, guiding he imp o isa ion h ough his embodied, lis ening-based explo a ion o he
nonlinea , AI-media ed con ol space.
3 Rela ion o P ac ice Field
The use o machine lea ning as a c ea i e ool aligns wi h g owing in e es in AI applica ions o
music gene a ion, con ol, and in e ac ion (Fieb ink and Ca amiaux, 2016; S u m e al., 2019; Knees
e al., 2019). Speci ically, employing an MLP o eal- ime ges u e- o-syn hesis mapping con ibu es
o esea ch on in elligen pe o mance sys ems and hype ins umen s, whe e echnology ex ends he
pe o me ’s exp essi e capabili ies (Palacio-Quin in, 2017; Leeuw, 2021). The ocus on lea ning
mappings om in e ac ion a he han p e-de ining hem esona es wi h app oaches ha emphasize
da a-d i en and adap i e in e aces (Roma e al., 2019; Fasciani and Wyse, 2012).
The imp o isa ional na u e o he piece connec s o adi ions o ee imp o isa ion and li e elec onic
music pe o mance (Bo go, 2006; Collins e al., 2003). The sys em design, emphasizing a eedback
2
loop be ween pe o me ac ion, sys em esponse, and pe o me pe cep ion, e lec s concep s o
embodied cogni ion and enac ion in musical in e ac ion (Ca amiaux e al., 2014). The explo a ion
o complex mappings and po en ially unp edic able sonic beha iou s gene a ed by he in e ac ion
be ween ges u e, AI, and ha dwa e syn hesis can be ela ed o p ac ices in ol ing modula syn hesize s
and complex sys ems, whe e eme gen beha iou is o en emb aced (Whi e, 2022; Sla e , 1998; Ca ey,
2024).
Fu he mo e, he wo k implici ly p obes ques ions o agency in human-compu e in e ac ion (Gio i
e al., 2022; Landg a , 2018). By embedding an adap i e ML componen wi hin he con ol loop,
he sys em mo es beyond a simple ool pa adigm owa ds a mo e collabo a i e model, whe e he
"ins umen " ac i ely pa icipa es in shaping he musical ou come based on he pe o me ’s inpu .
4 Conclusion
Cybe nesis p esen s a pe o mance sys em whe e human ges u e and ML algo i hms collabo a e in
he eal- ime con ol o pa chable sound syn hesis ha dwa e. By u ilizing a Leap Mo ion con olle
and a mul ilaye pe cep on, he wo k acili a es an imp o isa ional explo a ion o complex mappings,
na iga ed h ough embodied in e ac ion and a en i e lis ening. This app oach highligh s he po en ial
o ML no jus as a ool o au oma ion and gene a ion, bu as a dynamic pa ne in c ea i e exp ession,
os e ing eme gen musical s uc u es h ough a igh eedback loop be ween he a is and he
compu e . The piece aims o con ibu e o he ongoing discou se on ML in music c ea i i y, in e ac i e
sys em design, and he e ol ing na u e o imp o isa ion in he age o in elligen echnologies. Fu u e
wo k could in ol e explo ing di e en ML a chi ec u es, inco po a ing mo e sophis ica ed eedback
mechanisms wi hin model, and in es iga ing he long- e m co-adap a ion be ween pe o me and
sys em.
Re e ences
Da id Bo go. Sync o Swa m: Musical Imp o isa ion and he Complex Dynamics o G oup
C ea i i y. In Lec u e No es in Compu e Science, Janua y 2006. doi: 10.1007/11780274_
1. URL
h ps://www. esea chga e.ne /publica ion/221350614_Sync_o _Swa m_
Musical_Imp o isa ion_and_ he_Complex_Dynamics_o _G oup_C ea i i y.
Bap is e Ca amiaux, Jules F ançoise, No be Schnell, and F édé ic Be ilacqua. Mapping
Th ough Lis ening. Compu e Music Jou nal, 38(3):34–48, Sep embe 2014. ISSN 0148-9267.
doi: 10.1162/COMJ_a_00255. URL
h ps://ieeexplo e.ieee.o g/abs ac /documen /
6899813. Con e ence Name: Compu e Music Jou nal.
Benjamin Ca ey. E godynamics o he Open Machine: Ma e ial Engagemen and Modula Syn-
hesis Pe o mance P ac ice. Con empo a y Music Re iew, 42(3):376–390, 2023. doi: 10.1080/
07494467.2023.2277563. URL
h ps://doi.o g/10.1080/07494467.2023.2277563
. Pub-
lishe : Rou ledge _ep in : h ps://doi.o g/10.1080/07494467.2023.2277563.
Benjamin Ca ey. Me as able In en ions: Simondonian conc e isa ion and echnical in en ion in modu-
la syn hesis p ac ice. O ganised Sound, 29(3):315–326, 2024. doi: 10.1017/S1355771824000128.
Nick Collins, Alex McLEAN, Julian Roh hube , and Ad ian Wa d. Li e coding in lap op
pe o mance. O ganised Sound, 8(3):321–330, Decembe 2003. ISSN 1469-8153, 1355-
7718. doi: 10.1017/S135577180300030X. URL
h ps://www.camb idge.o g/co e/
jou nals/o ganised-sound/a icle/abs/li e-coding-in-lap op-pe o mance/
08F42B84BBCA427C345030481A3DDA0D#access-block.
Philippe Esling, Nao ake Masuda, Ad ien Ba de , Romeo Desp es, and Axel Chemla-Romeu-San os.
Uni e sal audio syn hesize con ol wi h no malizing lows, July 2019. URL
h p://a xi .
o g/abs/1907.00971. a Xi :1907.00971 [cs].
S e ano Fasciani and Lonce L. Wyse. Adap ing gene al pu pose in e aces o syn hesis engines
using unsupe ised dimensionali y educ ion echniques and in e se mapping om ea u es o
pa ame e s. In In e na ional Con e ence on Ma hema ics and Compu ing, 2012. URL
h ps:
//api.seman icschola .o g/Co pusID:35264419.
3
Rebecca Fieb ink and Bap is e Ca amiaux. The Machine Lea ning Algo i hm as C ea i e Musical
Tool, No embe 2016. URL h p://a xi .o g/abs/1611.00379. a Xi :1611.00379.
A emi-Ma ia Gio i, Aa on Einbond, and Geo gina Bo n. Composing he Assemblage: P obing
Aes he ic and Technical Dimensions o A is ic C ea ion wi h Machine Lea ning. Compu e Music
Jou nal, 46(4):62–80, Decembe 2022. ISSN 0148-9267. doi: 10.1162/comj_a_00658. URL
h ps://doi.o g/10.1162/comj_a_00658.
Andy Hun , Yo Dd, Ma celo Wande ley, and Ma hew Pa adis. The Impo ance o Pa ame e
Mapping in Elec onic Ins umen Design. Jou nal o New Music Resea ch, 32, May 2002. doi:
10.1076/jnm .32.4.429.18853.
Pe e Knees, Ma kus Schedl, and Rebecca Fieb ink. In elligen music in e aces o lis ening and
c ea ion. In Companion P oceedings o he 24 h In e na ional Con e ence on In elligen Use
In e aces, IUI ’19 Companion, pages 135–136, New Yo k, NY, USA, Ma ch 2019. Associa ion
o Compu ing Machine y. ISBN 978-1-4503-6673-1. doi: 10.1145/3308557.3313110. URL
h ps://dl.acm.o g/doi/10.1145/3308557.3313110.
Edga Landg a . Imp o isa ion, pos humanism, and agency in a (Ge ha d Rich e pain ing).
Liminali ies, 14(1):207–222, 2018. Publishe : Liminali ies.
Hans Leeuw. Vi uoso mapping o he Elec umpe , a hype ins umen s a egy. In In e na ional
Con e ence on New In e aces o Musical Exp ession, June 2021. doi: 10.21428/92 beb44.
a8e0cceb. URL h ps://nime.pubpub.o g/pub/ xe52ym6/ elease/1.
Cléo Palacio-Quin in. 2008: Eigh Yea s o P ac ice on he Hype -Flu e: Technological and Mu-
sical Pe spec i es. In Alexande Re sum Jensenius and Michael J. Lyons, edi o s, A NIME
Reade , olume 3, pages 335–351. Sp inge In e na ional Publishing, Cham, 2017. ISBN
978-3-319-47213-3 978-3-319-47214-0. doi: 10.1007/978-3-319-47214-0_22. URL
h p:
//link.sp inge .com/10.1007/978-3-319-47214-0_22
. Se ies Ti le: Cu en Resea ch in
Sys ema ic Musicology.
Ge a d Roma, Owen G een, and Pie e Alexand e T emblay. Adap i e mapping o sound collec ions
o da a-d i en musical in e aces. In Ma celo Quei oz and Anna Xambó Sedó, edi o s, P oceedings
o he In e na ional Con e ence on New In e aces o Musical Exp ession, pages 313–318, Po o
Aleg e, B azil, June 2019. UFRGS. doi: 10.5281/zenodo.3672976. URL
h p://www.nime.
o g/p oceedings/2019/nime2019_pape 060.pd .
Dan Sla e . Chao ic Sound Syn hesis. Compu e Music Jou nal, 22(2):12–19, 1998. ISSN 0148-9267.
doi: 10.2307/3680960. URL
h ps://www.js o .o g/s able/3680960
. Publishe : The MIT
P ess.
Bob L S u m, Oded Ben-Tal, Úna Monaghan, Nick Collins, Do ien He emans, Elaine Chew, Gaë an
Hadje es, Emmanuel De u y, and F ançois Pache . Machine Lea ning Resea ch ha Ma e s
o Music C ea ion: A Case S udy. Jou nal o New Music Resea ch, 48(1):36–55, 2019. doi:
10.1080/09298215.2018.1515233. URL
h ps://hal.science/hal-03278041
. Publishe :
Taylo & F ancis (Rou ledge).
Pie e Alexand e T emblay, Ge a d Roma, and Owen G een. Enabling P og amma ic
Da a Mining as Musicking: The Fluid Co pus Manipula ion Toolki . Compu e Mu-
sic Jou nal, 45(2):9–23, June 2021. ISSN 0148-9267, 1531-5169. doi: 10.
1162/comj_a_00600. URL
h ps://di ec .mi .edu/comj/a icle/45/2/9/111383/
Enabling-P og amma ic-Da a-Mining-as-Musicking-The.
Doug Van No , Pauline Oli e os, and Jonas B aasch. Elec o/Acous ic Imp o isa ion and Deeply
Lis ening Machines. Jou nal o New Music Resea ch, 42(4):303–324, Decembe 2013. ISSN 0929-
8215. doi: 10.1080/09298215.2013.860465. URL
h ps://doi.o g/10.1080/09298215.
2013.860465. Publishe : Rou ledge.
Alex Whi e. Uns able S uc u e: The imp o ising modula syn hesise . O ganised Sound, 27(2):182–
192, Augus 2022. ISSN 1355-7718, 1469-8153. doi: 10.1017/S1355771821000595.
URL
h ps://www.camb idge.o g/co e/jou nals/o ganised-sound/
a icle/uns able-s uc u e- he-imp o ising-modula -syn hesise /
CD092D079B056EAEF1FF9CB0D083EEFE#.
4
