Syn he ic Cogni i e Feedback: Knowledge E osion by
Recu si e T aining o AI Gene a i e Models
E ic Guizzo∗
Independen Resea che
[email p o ec ed]
Abs ac
Syn he ic Cogni i e eedback is a ecu si e p ocess in which a gene a i e a i icial
in elligence (AI) model is ained on da a i has p oduced i sel . This loop can
ampli y in e nal biases, deg ade ou pu quali y, and de ach models om eal-wo ld
da a. As human-gene a ed da a becomes sca ce , such sys ems could inc easingly
ely on syn he ic in o ma ion, leading o possible scena ios whe e models a e
ained solely on ou pu s o o he models. To e lec on his phenomenon, we
p esen a musical piece in which, while a human pe o me plays, an AI is ained
in eal ime on he pe o me ’s pas ac ions and ecu si ely e ained on i s own
ou pu s. As he composi ion un olds, he model g adually o e ides human con ol
and e en ually akes ull command o he execu ion. The wo k highligh s he isks
o o e - elying on AI while neglec ing he de elopmen o human knowledge, and
encou ages e lec ion on he shi ing balance be ween au ho ship, o iginali y, and
machine-d i en c ea ion.
1 In oduc ion
Gene a i e machine lea ning models a e becoming inc easingly powe ul, g adually eplacing human
labo ac oss di e se ields and mu a ing he way knowledge and a a e gene a ed and consumed.
This g owing in luence ex ends beyond au oma ion, eshaping he e y logic o cul u al p oduc ion
and c ea i e p ac ice. As such ools g ow in scale and capabili ies, so does hei demand o la ge
olumes o aining da a. In esponse, esea che s s a ed using AI-gene a ed con en as inpu o
aining models (Schuhmann e al., 2022). Howe e , his p ac ice ini ia es a degene a i e p ocess:
when a model’s ou pu becomes he inpu o i s own e olu ion, a o m o syn he ic cogni i e
eedback loop a ises. This sel - e e ence ein o ces and ampli ies models’ own a i ac s, esul ing in
escala ing dis o ions and a g adual loss o ideli y, causing hem o lose ouch wi h eal-wo d da a
2
(Alemohammad e al., 2023; Shumailo e al., 2024; Gibney, 2024).
The use o eedback loops o e eal he inne p ope ies o sys ems has deep oo s in expe imen al
music (Mo is, 2007; Hawo h, 2021). Among o he s, Al in Lucie ’s I Am Si ing in a Room (Lucie ,
1969) e ealed he acous ic inge p in o a space by ecu si ely e- eco ding a spoken ex . In
no-inpu music he noises p oduced by a sel -pa ched mixing boa d a e explo ed as an unp edic able
sound pale e (Valle, 2012). William Basinski’s Disin eg a ion Loops cap u ed he physical decay o
magne ic ape as a powe ul e lec ion on memo y and loss (Basinski, 2002).
In his wo k, we ex end he eedback loop concep by placing a machine lea ning model wi hin i ,
using i no only as a gene a i e mechanism bu as a means o in e oga e i s s uc u al limi a ions.
This app oach highligh s a b oade issue: he consequences o elying exclusi ely on AI o ask
execu ion become inc easingly e iden as eal-wo ld da a g adually becomes obsole e. Fu u e models
may be ei he ained on ou da ed in o ma ion o on syn he ic da a p oduced by o he models, c ea ing
a sel - e e en ial cycle in which de ec s accumula e and magni y. This pe spec i e p omp s e lec ion
∗Fo me ly a ilia ed o Ci y S . Geo ge, Uni e si y o London.
2Also e e ed as model collapse.
P oceedings o he 6 h Con e ence on AI Music C ea i i y (AIMC 2025),
B ussels, Belgium, Sep embe 10 h-12 h
Figu e 1: Pe o mance s uc u e block diag am. Tempo al p og ession om le o igh .
on he po en ial implica ions o neglec ing he augmen a ion o ou di ec human knowledge in a o
o an exclusi e dependence on AI.
2 The Pe o mance
We p esen a musical piece ha in eg a es a human pe o me wi h an AI model ained li e and
subjec ed o syn he ic cogni i e eedback. Figu e 1 shows he pe o mance’s s uc u e.
The pe o me plays a cus om elec onic ins umen buil in Max Msp, con olled ia a ac ile in e ace
(as de ailed in Sec ion 2.1). Meanwhile, he AI model (desc ibed in Sec ion 2.2) is simul aneously
ained in eal ime on a da ase o pe o mance sco es p e iously eco ded by he same musician.
The model con inuously ecei es i s own ou pu s as new aining da a, c ea ing a cogni i e loop
e ec ha g adually deg ades he gene a ed sco es. Each ime a new sco e is p oduced, one o mo e
pe o me ’s con ols a e locked and aken o e by he AI, which begins manipula ing i by eading
he gene a ed in o ma ion. As he piece p og esses, he AI gains con ol o e mo e pa ame e s,
slowly o e aking he ins umen ’s in e ace. In he beginning, he AI ex ends he pe o me ’s
capabili ies by con olling pa ame e s in a way ha complemen s he o iginal ma e ial. Howe e , as
eedback a i ac s accumula e, he model g adually di e ges om he pe o me ’s in en . E en ually,
i akes ull con ol o he pe o mance, p oducing a sco e en i ely disconnec ed om he human
pe o me ’s exp essi e logic. The piece ends au oma ically once he gene a ed sco es no longe show
any meaning ul ela ion o he pe o me ’s o iginal wo k, comple ing he ansi ion om human
exp ession o machine ecu sion. By s uc u ing he piece so ha each ac ion co esponds o a
dis inc sonic ou pu , he audience can immedia ely disce n e e y mo emen o he musician and
hus unde s and he machine’s subsequen esponses. A isual ende ing o he ac ile in e ace is
p ojec ed du ing he pe o mance, o e ing a clea iew o wha he human and he AI a e doing.
2.1 Pe o me ’s In e ace
Figu e 2: P e- eco ded pe o mance sco es. All pa ame e s a e sampled a ixed in e als and he
esul ing ma e ial is olded in 3-dimensional ma ices.
The ins umen a he hea o he pe o me ’s se up can be played using inge -d umming echniques,
igge ing bo h pe cussi e and pi ched sound sou ces and ou ing hem h ough chains o spec al
2
p ocesso s, pi ch-shi ed delays, sa u a o s, and e e bs o u he shaping. The sys em o e s up o
32 con ollable pa ame e s, enabling managemen o he sou ce ma e ials, he e ec s’ beha iou and
hei ou ing. Since some pa ame e s can ha e con as ing e ec s, he pe o me e ains he abili y o
coun e ac o balance he in luence o a pa ame e de e mined by he AI
3
. To ain he model, he
pe o me eco ded a da ase con aining 20 8-minu es sessions. The eco ded sco es can be isualized
as RGB images (see Figu e 2).
Du ing he pe o mance, he ins umen con inuously communica es wi h he aining sys em. Each
ime a deg aded sco e is gene a ed, one o mo e andom con ols a e locked and d i en by he new
ma ix ins ead o he pe o me .
2.2 Sel - eeding Model
(a)
(b)
Figu e 3: Visual e ec o syn he ic cogni i e eedback aining on (a) images and (b) pe o mance
sco es. F om le o igh , he cumula i e a i ac s become inc easingly p onounced.
While p e ious wo ks (Alemohammad e al., 2023; Shumailo e al., 2024; Gibney, 2024) demon-
s a ed he e ec s o aining gene a i e models on hei own ou pu s om a scien i ic pe spec i e, he
ocus o his p oposal is o explo e his phenomenon om an a is ic poin o iew. To his end, we
de eloped a delibe a ely scaled-down se up designed o make eedback a i ac s clea ly pe cep ible,
3Fo example, low-pass il e ing can empe he ha shness in oduced by high sa u a ion.
3
while also allowing us o ain models in eal ime. The sco es a e gene a ed by a Va ia ional Au oen-
code (VAE) (Kingma e al., 2013)
4
, ained on he a o emen ioned pe o mance ma ices da ase .
S uc u ally, he ask esembles image gene a ion: he model comp esses and econs uc s inpu s
ia an encode –decode a chi ec u e wi h ou con olu ional and ansposed con olu ional blocks,
and a ully-connec ed bo leneck. T aining uses a bina y c oss-en opy loss and a Kullback–Leible
(KL) di e gence e m, ollowing s anda d VAE o mula ion. The a i ac s gene a ed by his ne wo k
h ough syn he ic cogni i e eedback a e isualized in Figu e 3a using sample images
5 6
,and in
Figu e 3b on an ac ual pe o mance sco e. The deg ada ion ype and amoun hea ily depend on he
ne wo k a chi ec u e and aining hype pa ame e s (lea ning a e, epochs, KL weigh e c.). These a e
uned o ensu e ha a single aining is comple ed wi hin a ew seconds
7
, while deg ada ion un olds
o e se e al minu es. The piece’s du a ion is calcula ed o ma ch a a ge un ime, wi h he numbe o
deg ada ion s eps kep close o he numbe o pe o me pa ame e s, ensu ing i ends sho ly a e he
AI akes ull con ol.
3 Final Rema ks
This wo k p esen s a musical expe imen ha eimagines he eedback loop as a c i ical lens on
AI. By aining a li e gene a i e model and subjec ing i o syn he ic cogni i e eedback, he piece
in es iga es he aes he ic and concep ual implica ions o ecu si e lea ning on syn he ic da a. As AI
akes on an inc easingly in luen ial ole, i highligh s he ension be ween indi idual c ea i i y and
knowledge shaped by collec i e con ibu ions, as encoded in AI sys ems. Beyond i s a is ic aims,
he p ojec p omp s deepe e lec ion on he dange s o sel - ein o cing algo i hms, unde sco ing he
need o ancho echnological ad ancemen in human insigh and c i ical awa eness.
Re e ences
Alemohammad, S., Casco-Rod iguez, J., Luzi, L., Humayun, A. I., Babaei, H., LeJeune, D.,
Siahkoohi, A., and Ba aniuk, R. G. (2023). Sel -consuming gene a i e models go mad. a Xi
p ep in a Xi :2307.01850, 4:14.
Basinski, W. (2002). The disin eg a ion loops.
h ps:// empo a y esidence.com/p oduc s/
166. Audio eco ding, o iginally eleased by 2062 Reco ds.
Gibney, E. (2024). Ai models ed ai-gene a ed da a quickly spew nonsense. Na u e, 632(8023):18–19.
Hawo h, C. (2021). Music and cybe ne ics in his o ical pe spec i e: In oduc ion o he special issue
edi ed by ch is ophe hawo h and e ic d o .
Kingma, D. P., Welling, M., e al. (2013). Au o-encoding a ia ional bayes.
Lucie , A. (1969). I am si ing in a oom.
h ps://www.lo ely.com/ i les/cd1013.h ml
.
Audio eco ding, eissued by Lo ely Music, L d., 1981/2001.
Mo is, J. M. (2007). Feedback ins umen s: Gene a ing musical sounds, ges u es, and ex u es in
eal ime wi h complex eedback sys ems. In ICMC.
Schuhmann, C., Beaumon , R., Vencu, R., Go don, C., Wigh man, R., Che i, M., Coombes, T.,
Ka a, A., Mullis, C., Wo sman, M., e al. (2022). Laion-5b: An open la ge-scale da ase o
aining nex gene a ion image- ex models. Ad ances in neu al in o ma ion p ocessing sys ems,
35:25278–25294.
Shumailo , I., Shumaylo , Z., Zhao, Y., Pape no , N., Ande son, R., and Gal, Y. (2024). Ai models
collapse when ained on ecu si ely gene a ed da a. Na u e, 631(8022):755–759.
Valle, A. (2012). Towa ds a Typology o eedback Sys ems. Ann A bo , MI: Michigan Publishing,
Uni e si y o Michigan Lib a y.
4Any o he gene a i e a chi ec u e can be used o he same pu pose.
5O iginal inpu images c ea ed by Alice Lo enzon: h ps://www.alicelo enzon.com
6
Anima ed audio isual e sions o hese p ocesses a e a ailable a :
h ps://d i e.google.com/d i e/
u/0/ olde s/1 mEzc6usBdIMOmdQBLaX6 xQwlAVboiX
. He e he deg aded audio is gene a ed using he
same syn he ic cogni i e eedback echnique applied o spec og ams.
7Wi h GPU accele a ion enabled.
4
