inySounds:
o oice and musebo ensemble
A ne Eigen eld
School o he Con empo a y A s
Simon F ase Uni e si y, Canada
[email p o ec ed]
Abs ac
An i onic wo k in which iny sounds – quie noises made by he human oice
ha a e ba ely audible – se e as an inpu o a noisy and exube an musebo
ensemble ha au onomously esponds, accompanies, and a gues wi h he li e
inpu . Musebo s a e in elligen musical agen s ha decide how o espond o
hei en i onmen – and each o he – on hei own, based upon hei in e nal
belie s, desi es, and in en ions.
1 De ailed P og am No es
Machine lea ning algo i hms a e wonde ul o si ing h ough da a and disco e ing ela ionships;
mo e challenging is how hese algo i hms can be used o gene a ion. I isn’ ha di icul , o
example, o ain a sys em o p o ide simila sounds o a da abase, gi en a li e sound. Bu wha ’s
he a is ic in e es in ha ? Simila ly, i isn’ ha di icul o ex ac li e pe o mance in o ma ion
om an imp o ising musician – ac i i y le el, gene al equency ange, imb e – so ha he
sys em esponds likewise. Bu , again, eac i e sys ems lose in e es ai ly quickly.
I ind i much mo e in e es ing when my musebo s go o on hei own, explo ing hei own ideas
h ough belie s hey may ha e o med inco ec ly and unin en ionally. Fo ha eason, I usually
build a lo o ambigui y in o my analysis, o p o ide con lic ing in o ma ion. Wha happens when
one musebo is su e o some hing, while ano he is absolu ely su e o some hing else? And wha i
a hi d musebo jus doesn’ ca e?
In inySounds, musebo s a e ained using a neu al ne on a co pus ha has been hand- agged o
alence and a ousal measu es, as well as p e-analysed o spec al in o ma ion. Howe e , he
co ela ion be ween audio ea u es (wha he musebo s a e lis ening o ) and a ec ( alence and
a ousal) isn’ di ec ; in assigning he la e , I may decide ha a sound om he co pus is complex
and ac i e, bu my easons o doing so may no use he same in o ma ion as he musebo s a e
p o ided wi h. Thus, a musebo may decide ha , based upon wha i has lea ned, a li e sound is
high alence / high a ousal, bu he lis ene may pe cei e i o he wise. This isn’ a law in he
sys em; i ’s a ea u e!
Las ly, my ole as o e see in he musebo ensemble allows me o u he dis up how he
musebo s apply hei knowledge. The co pus is o ganised seman ically (i.e. oice sounds, ki chen
sounds, anspo a ion sound, e c.); once a musebo is using a ce ain subdi ec o y, i can’ easily
swi ch o ano he . As a esul , i s choice o ela ed sound, whe he a ec i e o imb al, is limi ed o
wha is immedia ely a ailable o i . I he musebo s a e us a ed, hey ha en’ men ioned i o me
(ye ).
2 Musebo s
Musebo s a e pieces o so wa e ha au onomously c ea e music collabo a i ely wi h o he
musebo s (Bown e . al, 2015). They decide how o espond o hei en i onmen – and each o he –
on hei own, based upon hei in e nal belie s, desi es, and in en ions.
Since 2015, I ha e used musebo s in a a ie y o a wo ks (Eigen eld 2016, Eigen eld 2017,
Eigen eld 2018, Eigen eld and Ricke s 2019), and became hei main e angelis . I had been
wo king wi h agen s o almos a decade, and he musebo p o ocol allowed o a consis ency ha
P oceedings o he 6 h Con e ence on AI Music C ea i i y (AIMC 2025),
B ussels, Belgium, Sep embe 10 h-12 h
kindled oppo uni ies o adap and euse agen s ou side o hei o iginal c ea i e wo k.
Musebo s lack di ec in e ac i i y, as hey a e au onomous. As a esul , my ole du ing hei
pe o ma i e ac ions is less as a pe o me o conduc o , hen as a c i ical lis ene . This is
pa icula ly he case du ing he long pe iods o ine uning e e y musebo ensemble, which en ails
a g ea deal o lis ening and no e- aking in o de o disce n c ea i e au onomy om buggy code.
Musebo s a e no s aigh o wa d eac i e p ocesses; ins ead, hey ha e hei own belie s (in his
case, he incoming analysis da a), desi es, and in en ions. They will happily play on hei own, o
hey may eac e y closely o he li e pe o mance; mo e o en hen no , hey will o e hei own
“ ein e p e a ion” o he li e pe o mance, wi h indi idual eac ions o he analysis da a.
2 Con ex
The musebo ensemble in inySounds is a edeploymen o an ea lie me ac ea i e sys em, The
Indi e ence Engine, which is pa ially desc ibed elsewhe e (Eigen eld , 2014). Li e audio is
analysed o ea u es: spec al cen oid; spec al lux; loudness; ac i i y le el (onse de ec ion);
and Ba k band spec um. This in o ma ion is messaged o he audio musebo s and an e ec sBo .
This la e musebo adds e ec s – delay, pi ch shi , ime s e ch, ing modula ion, and dis o ion –
au onomously, based upon i s in e p e a ion o he analysis messages. Fo example, i will swi ch
e ec s when ac i i y is low, and add mo e p ocessing when lux is high.
The audio musebo s – in his case, ou ins ances o inySoundBo – ha e access o a la ge co pus
o p e-analysed sound iles; gi en a Ba k band spec al analysis ia he Conduc o , he audioBo s
will a emp o ind he closes ma ching eco dings om hei a ailable da abase. The audioBo s
au onomously begin and end playing based upon incoming messages, including ac i i y and lux,
as well as eac ing o whe he o he audioBo s a e ac i e o no .
Audio is gene a ed using a modi ied e sion o Ca aRT (Schwa z, 2007).
Re e ences
Bown, O., Ca ey, B., & Eigen eld , A. (2015) Mani es o o a Muse-bo Ensemble: A pla o m o
li e in e ac i e pe o mance be ween mul iple au onomous musical agen s. In P oceedings o he
In e na ional Symposium on Elec onic A , Vancou e .
Eigen eld , A. (2014) Gene a ing S uc u e – Towa ds La ge-scale Fo mal Gene a ion. In
P oceedings o he Ten h A i icial In elligence and In e ac i e Digi al En e ainmen Con e ence,
Raleigh.
Eigen eld , A ne. "Musebo s a One Yea : A Re iew." P oceedings o he Musical Me ac ea ion
Wo kshop. 2016.
Eigen eld , A ne. "Designing Music wi h Musebo s." P oceedings o he Fi h Con e ence on
Compu a ion, Communica ion, Aes he ics and X. 2017.
Eigen eld , A ne. Collabo a i e Composi ion wi h C ea i e Sys ems. In e na ional Symposium on
Elec onic A , Du ban, 2018.
Eigen eld , A ne, and Ka h yn Ricke s. “Unau ho ized: Collabo a ing wi h a Pe o me
Collabo a ing wi h C ea i e Sys ems.” Gene a i e A Con e ence, Rome, 2019.
Schwa z, D. (2007) Co pus-based Conca ena i e Syn hesis. In IEEE Signal P ocessing Magazine,
24(2).
2
