Gene ación de componen es pa a la
c eación y e aluación de poemas
musicales: Módulo de componen es pa a
la gene ación musical
T abajo de Fin de G ado
Cu so 2019–2020
Au o
Guille mo Ville a To es
Di ec o
Gonzalo Méndez Pozo
Pablo Ge ás
G ado en Ingenie ía In o má ica
Facul ad de In o má ica
Uni e sidad Complu ense de Mad id
Gene ación de componen es pa a la
c eación y e aluación de poemas
musicales: Módulo de componen es
pa a la gene ación musical
T abajo de Fin de G ado en Ingenie ía In o má ica
Depa amen o de Ingenie ía del So wa e e In eligencia
A i icial
Au o
Guille mo Ville a To es
Di ec o
Gonzalo Méndez Pozo
Pablo Ge ás
Con oca o ia: Feb e o2020
Cali icación: No a
G ado en Ingenie ía In o má ica
Facul ad de In o má ica
Uni e sidad Complu ense de Mad id
23 de ene o de 2020
Au o ización de di usión
El abajo i man e, ma iculado en el G ado en Ingenie ía en In o má ica
de la Facul ad de In o má ica, au o iza a la Uni e sidad Complu ense de
Mad id (UCM) a di undi y u iliza con ines académicos, no come ciales y
mencionando exp esamen e a su au o el p esen e T abajo Fin de G ado:
“Gene ación de componen es pa a la c eación y e aluación de poemas musi-
cales: Módulo de componen es pa a la gene ación musical”, ealizado du an e
el cu so académico 2019 - 2020 bajo la di ección de Gonzalo Mendez Pozo y
Pablo Ge ás en el Depa amen o de Depa amen o de Ingenie ía del So -
wa e e In eligencia A i icial, y a la Biblio eca de la UCM a deposi a lo en el
A chi o Ins i ucional E-P in s Complu ense con el obje o de inc emen a la
di usión, uso e impac o del abajo en In e ne y ga an iza su p ese ación
y acceso a la go plazo.
Guille mo Ville a To es
23 de ene o de 2020

Dedica o ia
A Mo , po es a en las buenas y en las malas.
En Mad id, en Ba celona o en el in del Mundo.
ii
Ag adecimien os
G acias a mi amilia, po el apoyo y la comp ensión ecibidos du an e los
úl imos años du an e mi iempo en la Uni e sidad.
G acias a los di ec o es de es e p oyec o, Gonzalo y Pablo, po su apoyo
incondicional y la ayuda p es ada, sus consejos y sus guías du an e odo es e
iempo.
G acias a Ma isa y Ca los, compañe os de a en u a desde el p ime has a
el úl imo día, sin oso os no hubie a sido lo mismo. G acias po hace me
dis u a del iaje y pode compa i lo con oso os.
Hace casi cinco años en é en la Facul ad de In o má ica sin sabe que
en ella descubi ía mi pasión, g acias a odas aquellas pe sonas que se han
c uzado en mi camino du an e odo el iempo que he enido la sue e de
es a en la Facul ad de In o má ica.
Don c y because i is o e , smile because i happened
D Seus
ix
3.1.6. MIDO........................... 25
3.1.7. LilyPond ......................... 25
3.1.8. Musesco e......................... 25
3.2. Hie a chical Da a Fo ma : HDF5 . . . . . . . . . . . . . . . . 25
4. De elopmen app oach owa ds music gene a ion 27
4.1. Music Theo y Concep s . . . . . . . . . . . . . . . . . . . . . 28
4.2. Da ase .............................. 29
4.2.1. Da ase ype: he ins umen . . . . . . . . . . . . . . 29
4.2.2. Da ase ype: no es, cho ds and songs . . . . . . . . . 29
4.2.3. Da ase ype: Gen e . . . . . . . . . . . . . . . . . . . 30
4.2.4. Da ase size........................ 30
4.3. Implemen ed models . . . . . . . . . . . . . . . . . . . . . . . 31
4.3.1. Ma ko Models . . . . . . . . . . . . . . . . . . . . . . 31
4.3.2. Neu al Ne wo ks . . . . . . . . . . . . . . . . . . . . . 40
4.3.3. A chi ec u e o he solu ion . . . . . . . . . . . . . . . 41
4.3.4. Neu al Ne wo ks . . . . . . . . . . . . . . . . . . . . . 42
4.3.5. A heo e ical app oach o Sen imen Analysis . . . . . 48
4.3.6. Lea ned lessons and esul compa ison . . . . . . . . . 50
4.4. Resul s and conclusions o he de eloping phase . . . . . . . . 51
5. Conclusiones y T abajo Fu u o 55
5.1. Composición pa a imp o isación ea al . . . . . . . . . . . . 55
5.2. Te minación de ob as clásicas . . . . . . . . . . . . . . . . . . 56
5.3. Composición pa a películas y ideos . . . . . . . . . . . . . . 57
5.4. In eg ación con asis en es de oz . . . . . . . . . . . . . . . . 57
5.5. Aplicaciones en el ámbi o educa i o . . . . . . . . . . . . . . . 58
5. Conclusions and Fu u e Wo k 59
5.1. Composi ion o hea e imp o isa ion . . . . . . . . . . . . . 60
5.2. Classical pieces comple ion . . . . . . . . . . . . . . . . . . . . 60
5.3. Composición pa a películas y ideos . . . . . . . . . . . . . . 61
5.4. Voice assis an in eg a ion . . . . . . . . . . . . . . . . . . . . 61
5.5. Applica ions in he educa ional en i onmen . . . . . . . . . . 61
5.6. Annex A: Ou pu Sample . . . . . . . . . . . . . . . . . . . . 63

Bibliog aphy 67
Lis o igu es
2.1. Neu al Newo k ypes depending on he s uc u e . . . . . . . 14
2.2. Ac i a ion unc ion ypes . . . . . . . . . . . . . . . . . . . . 16
2.3. Typical s uc u e o a Feed Fo wa d Neu al Ne wo k . . . . . 17
2.4. Typical s uc u e o a Mul ilaye Pe cep on . . . . . . . . . . 18
2.5. LSTM ypical s uc u e . . . . . . . . . . . . . . . . . . . . . 18
4.1. Co ela ion be ween he ch oma ic scale and he piano keys . 30
4.2. Ma ko Ma ix Diag am . . . . . . . . . . . . . . . . . . . . . 32
4.3. Ma ko Ma ix Ou pu . . . . . . . . . . . . . . . . . . . . . . 34
4.4. No es s a no a ion o midi ecuency co ela ion . . . . . . . 35
4.5. Ma ko Ma ix me ging . . . . . . . . . . . . . . . . . . . . . 35
4.6. Full Diag am Flow . . . . . . . . . . . . . . . . . . . . . . . . 36
4.7. Execu ion o he piece wi h Chopin Ope a 18 . . . . . . . . . 38
4.8. Execu ion o me ged pieces wi h Chopin Ope a 18 and Ope a 7 38
4.9. Diag am o an un olled neu al ne wo k . . . . . . . . . . . . . 40
4.10. D opou laye s and o e i ing . . . . . . . . . . . . . . . . . . 45
4.11. Hype bolic angen g aph . . . . . . . . . . . . . . . . . . . . 45
xix
Lis o ables
3.1. Midi co ela ion . . . . . . . . . . . . . . . . . . . . . . . . . . 24
xxi

Chap e 1
In oduc ion
“Music begins whe e he possibili ies o language end”
— Sibelius
1.1. Backg ound
This TFG is embedded on a la ge p ojec designed o gene a e music
o accompany, also gene a ed, poems. A s udy o hei ma ching is also
pe o med. The TFG ha co e he modules ela ed o Ly ics gene a ion and
music-ly ics ma ching we e p esen ed las Sep embe by Ma ia Luisa Qui oga
and Ca los Ma ín Tes illano in a TFG i led «Componen s gene a ion o
he c ea ion and e alua ion o musical poems»
Tha TFG had as aim o compose and e alua e he ma ching capabili ies
o bo h music and ly ics. Tha TFG ook, as pa o he inpu o compose
hose models, he ou pu o he p esen TFG, ha is he composed music.
1.2. Objec i e
My aim in his p ojec is o s udy he capaci y o compu e s o compose
music using di e en echniques belonging o he AI ield. These echniques
we e chosen due o hei e sa ili y, he duc ili y o he models and he
ools. Mo eo e , I conside AI o be an in e es ing esea ch ield due o he
exponen ial g ow h and he ending eme ging echnologies.
The scope o he p ojec , he e o e, is o design and implemen di e -
en models, using di e en echniques, able o gene a e music and he la e
1
2Chap e 1. In oduc ion
e alua ion o he esul o each one o he models.
Thus, he p ojec will ocus on he gene a ion and e alua ion o music
and he compa a i e s udy o he possibili ies o e ed by he ield o music
gene a ion h ough A i icial In elligence me hods.
In he same way, he de elopmen o his p ojec will allow me o deepen
he knowledge o he s uc u es and models used in AI and a he same ime,
o combine i wi h music.
Fu he mo e, i is in ended ha his p ojec can be ex ended and eused
by possible nex TFGs o se e as an inspi a ion o con inue inding u ili y
o he gene a i e models beyond he me e compu a ional po en ial o hese
models.
1.2.1. Mo i a ion
Music is one o he unique ways we humans communica e and ansmi
ideas and eelings. On he o he hand, he compu a ional capaci y o oday’s
compu e s and A i icial In elligence models allow us o c ea e sys ems capa-
ble o imi a ing human skills, bo h in e ms o he esul s ob ained and he
lea ning p ocess. A he same ime, s udying he in e nal p ocesses o hese
sys ems and he a ian s ha lead hese sys ems o gene a e be e esul s.
In o de o conside wha is good o bad music, he e a e a ied and
di e se c i e ia wi hin he music heo y esea che s hemsel es. Tha is why
du ing he de elopmen o his p ojec , when e alua ing he pieces esul ing
om he composi ion, we will no base ou sel es on objec i e, pa ame e ized
s udies and cha ac e is ics ha y o app oach such compa a i e s udies.
Addi ionally, and h ough he empi ical me hod o gene a ing music wi h
di e en echniques, we will y o pa ame e ize ce ain aspec s in ol ed
in he gene a ion o music in o de o compose be e pieces as well as o
imp o e he lea ning capaci y o he models used.
Since music se es, among o he hings, o ansmi eelings and each
piece o music ca ies wi h i a sen imen al deg ee, a s udy will be ca ied
ou which aims o analyse he associa ion and ela ionship be ween eelings
and he pieces o music. Fo his s udy, he esou ces used by he musical
compose s o ansmi eelings ( ugues, mishaps, dialogues,...) will be aken
in o accoun and he ac o s in e ening in his aspec will be de eloped.
In o de o ec ea e hese composi ional echniques, he di e en ial ac o s
1.2. Objec i e 3
o each o he echniques and when hey a e p esen in he musical pieces
will be s udied. The aim o his s udy is o see in which si ua ions hese
echniques a e use ul when exp essing eelings.
Th ough his s udy, he aim is o ca y ou he pa ame e iza ion o he
echniques in o de o ec ea e hem la e . To do so, he a iables ha make
possible he musical composi ion associa ed wi h a ce ain eeling will be
pa ame e ized, making possible he en ichmen o he musical pieces gene -
a ed.
The inal objec i e is o y o ge he A i icial In elligence sys ems used
o compose music, no o app oach human composi ions, bu o be able o
p o ide he composed pieces wi h eeling and meaning beyond he co ec
succession o no es and silences. This ac o , oge he wi h he possibili ies
o e ed by he models used, will be he di e en ial ac o in making his
p ojec an app oach o music gene a ion h ough he uni ica ion and use o
di e en composi ional echniques belonging o he ield o music heo y and
A i icial In elligence.
10 Chap e 2. S a e o he A
he 7094 o sing we e John Kelly and Ca ol Lock, bu he song was al eady
composed, he compu e only was able o play i (Ellis, 2006). The e o e was
no a p ope composi ion, e en hough esul s we e imp essing and plan ed a
seed among compu e esea che s. The ask was epea ed la e , when com-
pu e HAL 9000 who also sang his same song. This e en was ec ea ed
la e in 2001 in he ilm «A Space Odyssey»(Boylan, 1985).
I we dig in o he his o y o p ope compu e -based music composi ion,
we ha e o go back o 1951 when he compu e scien is Alan Tu ing, known
o dec yp ing he Nazi code du ing WWII, c ea ed a compu e capable o
composing li le musical pieces. Un o una ely, he compu e was des oyed
bu Jack Copeland and Jason Long (Copeland and Long, 2017) , wo e-
sea che s om he Uni e si y o Can e bu y in New Zealand back in 2016
we e capable o ep oducing he compu e and some o he pieces Alan Tu -
ing composed wi h ha compu e .
Compu e -based music composi ion is a wide and s udied ield, and we
can dis inguish among he esea che s and cu en s acco ding o he me hod
used and he app oach aken in o de o c ea e a compu e -gene a ed musi-
cal piece. In he o hcoming sec ions I will expose he main cu en s and
echniques used by esea che s who dig in o his ield.
2.2. Composi ion om p e ious lea ning
Mos esea che s s a by s udying he way humans compose music and
hey y o igu e ou wha can we lea n om hose echniques and p ocesses
and how o ansmi ha knowledge o a compu e , as well as ac o s ha
can be pu in o conside a ion when ying o make a compu e come up wi h
musical pieces.
Composing music is no hing else han o a ange a numbe o limi ed
no es ( ones, semi ones and silences) one a e ano he . A no e is o med by
a pi ch ( une) and a du a ion. Bu he e a e se e al ac o s ha a ise when
deciding wha no e o pu nex .
Humans compose music, in luenced by hei p e iously lea ned o hea d
music. The e o e, he i s hough would be o eed he compu e wi h music
pieces and ind a way o analyse hem in o de o ind common pa e ns o
s uc u e in o de o lea n om hose ac o s, and hen, y o add hem
in o he compu e composed pieces. This app oach is aken by Ca and

2.3. Mos used echniques and me hods 11
Zukowski (2018), in whose esea ch pape s come up wi h he explana ion
o he bi s and pieces ha a e aken in o conside a ion in o de o build a
model ha wo ks based on his me hodology.
Ano he impo an ques ion ha can be ound in many esea che s is he
one ela ed o aking in o conside a ion he con ol o e he p ocess when
gene a ing music. Con olling he ou pu in ol es ha he use is able o
change ce ain aspec s o he con igu a ion o he model i sel , o ce ain
aspec s di ec ly ela ed o he gene a ion p ocess. I he ou pu can be con-
olled o d i en while music is composed (gene a ion ime), he p ocess is
called «condi ional» (Ca and Zukowski, 2018), o he wise is called «uncon-
di ional». The e o e he i s ype o composi ion will allow he use o decide
o econ igu e ce ain aspec s so ha he ou pu adap s o hose changes in
o de o he i o be mo e pleasan o i he es ablished equi emen s.
2.3. Mos used echniques and me hods
Simpli ying music composi ion, we can de ine i as he p ocess h ough
which no es ( o med by pi ch and du a ion) a e pu one a e he o he o
o m a ull musical composi ion. O cou se, he e a e di e en a ia ions
o he same g oup o no es, bu he way music no es a e a anged has a
p obabilis ic a iable a ached. O cou se, he e a e se e al echniques being
used and poin s o iew om hey add ess his challenge, bu hey all sha e
his impo an ac we now wan o ecall.
A p ojec ha shows his ac e y well is he one c ea ed in PyPose
(William (2018)), which is a p ojec whe e music is composed me ely based
on p e iously se p obabili ies. Al hough Oli ei a’s wo k was mainly o-
cussed on Poe y Gene a ion wi h PoeT yMe, o T a-la-ly ics, bo h ela ed
o compu e gene a ion o ly ics, i se es o show he e a e many possibili ies
and applica ion o his echnique.
The Poe y Gene a ion me hods used by Oli ei a can illus a e e y well
he challenges aced by Hugo and he kind o p oblems I could be acing
du ing he de elopmen o my p ojec as well as he possible ways o ace
and esol e hose p oblems and challenges.
Due o he ac ha he ield ha co e s music gene a ion is widely open
and di e se we will s uc u e he ele an e e ences o wo k de eloped by
o he in es iga o s in a way ha se ed as inspi a ion o ou wo k.
12 Chap e 2. S a e o he A
Music Composi ion se s a e y big challenge o in es iga o s who wan s
o de elop hei abili ies as compu e and A i icial In elligence esea che s
as in o de o gene a e music i is necessa y o ha e a p e ious knowledge o
he ield.
When digging in o o he in es iga o s wo k, I selec ed hose who we e
mainly ocused on music gene a ion such as F açois Pache (Aucou u ie
and Pache , 2003). His wo k on music gene a ion and esea ch ela ed o
music composi ion and gen e classi ica ion o music d o e us o conclude
ha one possibili y was o use A i icial In elligence o de elop ou wo k.
We s a ed s udding se e al echniques applied o music gene a ion such
as Ma ko Chains and Long-Sho Te m Memo y Ne wo k ( om now on
LSTM), ha we la e complemen ed wi h sen imen analysis esea ch. Pa-
che is has de eloped se e al componen s o classi ica ion and composi ion
o music in di e en gen es and musical cu en s. On his wo k called "Taxon-
omy o musical Gen es" he in es iga es abou he di e en ac s ha makes
a piece o music belong o a ce ain Music Gen e.
To classi y music pieces and i hem in o a de e mined gen e he akes
in o accoun some ac s as Geog aphical inclusion, Ag ega ion, Repe i ion,
his o ical pe iod, e c. We ocused on ex ac ing om his wo k he musical
aspec on music gene a ion ha made him gene a e music om an speci ic
gen e o classi y music in o an speci ic gen e using he TFG de eloped by
Capa ini López, An onio and Pe ez Molina, Lau a ha in es iga ed "Gen e
classi ica ion o elec onic music"(An onio Capa ini López, 2017).
The wo k de eloped measu es he accu acy o music gen e classi ica ion
echniques. This wo k se ed us o see he impo an ac s conce ning gen e
classi ica ion and he e o e hose ones o ake in o accoun in o de o classi y
music pieces and ollow a gi en pa e n. This wo k se ed o connec wi h
he sen imen analysis a ea o ou wo k.
No necessa ily ocussed on music composi ion bu in c ea ion o RNN
models, i mus be ecalled he wo k o Andej Ka pa hy Ka pa hy (2015),
whose explana ion o he in e nal unc ioning o Neu al Ne wo ks was ound
o be e y use ul o unde s and in dep h he s uc u e o a Neu al Ne .
Du ing all he esea ch phase some impo an aspec s o ake in o accoun
on music gene a ion could be b ough up:
2.3. Mos used echniques and me hods 13
2.3.1. F equen ly Used Techniques
He ea e we will expose he main echniques used o c ea e compu e
models capable o gene a ing music composi ions. I is wo h ecalling ha
mos o hese models we e bo n in he ma hema ical ield, and hey u ned
ou o be use ul in o de o c ea e compu e -based gene a ed music.
2.3.1.1. Ma ko Models
This echnique is used as he s a ing poin by mos esea che s ha ake
in o accoun he s a is ical ac o music composi ion.
Ma ko Models a e models i s in oduced by And ey And eye ich Ma ko
in 1906 (Basha in e al., 2004). And eye ich was a Russian ma hema ician
ha concei ed o he i s ime his kind o models ha , la e would gi e
bi h o he nowadays known as Ma ko Chains.
A Ma ko model is a an au oma e sys em a anged in o m o s a e
chains. The change among s a es is only de e mined by he cu en s a e,
no aking in o accoun p e ious s a es (Kouemou and Dyma ski, 2011).
This s a es a e au onomous om each o he and he ela ions among
hem is s ochas ic, so he cu en s a e is he only ac o ha is aken in o
accoun o ind ou wha he nex ac o would be.
Pu ing i in o music e ms, he p obabili y model s udies he chance
o a no e o be ollowed by any o he gi en no e. This way wee c ea e a
b anch map whe e we se co ela ions among he di e en no es concei ed
in a composi ion.
This way we can s a c ea ing a b and new composi ion s a ing wi h
he p obabili ies o a no e o be ollowed by any o he gi en no e.
The mos common way o pu his ela ion down o pape is by c ea ing a
ela ion able whe e p obabili ies among he no es a e pic u ed. The model,
s a ing om he knowledge adqui ed om he aining song, will c ea e a
b and new song aking in o accoun he p obabili y able c ea ed by he
song.
The main ad an age o his model is ha , musicali y ac o s a e nea ly
always ensu ed, as long as he piece used o aining he model has in i
hese pa icula cha ac e is ics.
One example o a musician ha used Ma ko Models is Ianis Xenakis, a
musician ha in 1958 used a Ma ko Model o c ea e a piece o music called
14 Chap e 2. S a e o he A
Analogue (Xenakis, 1958).
2.3.1.2. Neu al Ne wo ks
A Neu al Ne wo k is a s uc u e o med by neu ons. A neu on is a
independen s uc u e ha pe o ms an ope a ion on a gi en inpu o deli e
an ou pu . The Neu al Ne wo k is o med by one o mo e neu ons a anged
in laye s. As wi h Neu ons i sel , Neu al Ne wo ks can ha e one o mo e
laye s.
Depending on he way laye s and Neu ons a e a anged, he e a e se e al
ypes o neu al ne wo ks. The Ne wo k can ecei e an inpu o med by one
o many a ays as an inpu and can deli e one o many a ays as esul .
As shown in he nex igu e, ex ac ed om (Ka pa hy, 2015), depending on
he ela ion among he numbe o ou pu s and inpu s, we can dis inguish
among one- o-one, one- o-many, many- o-one o many o many whe e he
i s a gumen indica es he numbe o inpu s and he second a gumen , he
numbe o ou pu s.
Figu e 2.1: Neu al Newo k ypes depending on he s uc u e
Among he di e en laye s, he ou pu is o wa ded, he e o e he ou pu
o he p e ious laye se es as inpu o he coming laye . A he same ime,
he ou pu o he la e laye s can also become pa o he inpu o he p e ious
laye s. This p ope y is called «backp opaga ion». Which u ns he Neu al
Ne wo k in o a Recu en Neu al Ne wo k.
As well as he ou pu o he la e laye s, he inal ou pu o he ne wo k
can also se e as he inpu o some o he p e ious laye s o i can be in-
co po a ed in o he da ase , u ning i in o he inpu o he main Neu al
Ne wo k.
This backp opaga ion capabili y o he Neu al Ne wo k, makes a huge
impac on he ou pu o he Neu al Ne wo k, and i is he main di e ence
2.3. Mos used echniques and me hods 15
among Ne wo k and he p e iously discussed models. This backp opaga ion
p ope y, makes i possible o he Neu al Ne wo k o lea n om p e ious
compossed music and i en iches he ou pu making i mo e human alike.
This capaci y o emembe p e ious ac o s, can allow he Ne wo k o com-
pose music wi h a gi en s uc u e (which o he models did no allow) and i
allows o he esul ing composi ion o be mo e human-alike.
One o he main ad an ages o his model, as well as he con inuous
lea ning ac o , i he ac ha i also akes in o accoun he e olu ion o
he pieces, he e o e, i well con igu ed and p ope ly ed, he Neu al Ne wo k
will allow us o come up wi h a composi ion ha p ese es he s uc u e o
a usual song (in o, e se, cho us, e se, cho us, and so on).
The main ad an age his kind o models c ea e is he powe ul po en ial
o his kind o models when i comes o analyzing music composi ions and
o he kind o s uc u es.
Neu al Ne wo k models p esen a big ad an age when gene a ing songs
a e he s a ing model o music composi ion, come o emb ace and esol e
a main issue ha appea s when using o he kind o echniques: he memo y
ac o .
To unde s and he memo y ac o we can i s s a by ecalling he
s uc u e and cohe ence along he play o any song. This makes he song
o be i hm consis en , bu eeling a iable. To make hings simple, we can
unde s and his as ollows: " he songs sounds good, bu i sound he same
all along".
The way Neu al ne wo ks ies o esol e his p oblem is by aking in o
accoun , no only he ela ion among he di e en no es, o he p obabili y
o a no e o be ollowed by any o he , bu also he main s uc u e o he
song and he di e en p obabili ies depending on he pa o he song we a e
c ea ing and lea ning om.
The esul o his echnique, and wha we ind o be he bigges ad an-
age, is o c ea e iches musical pieces and mo e human-alike composi ions.
Ka pa hy w o e a pape back in 2015 i led "The Un easonable e ec
o Recu en Neu al Ne wo ks" (Ka pa hy, 2015) on which he ela es he
unc ional model o a Neu al Ne wo k and hei possibili ies in compu a ional
c ea i i y and da a handling and gene a ion. He gene a es Shakespea e ex
and monologues om aining a Recu en Neu al Ne wo k.

16 Chap e 2. S a e o he A
The way a Recu en Neu al Ne wo k wo ks is simila o he Neu al
Ne wo ks ound on human b ains: he ne wo k mus ecei e a se o inpu
da a ha allows he model o «lea n» om i . I e a ing o e he da ase (a
numbe o speci ied epochs o i e a ions), he ne wo k is able o de e mine
he weigh s. The weigh is ela ed o he memo y o he neu al ne wo k, as
i de e mines he quan i y and/o he quali y o he connec ions among he
cells.
The P opaga ion unc ion is he esul ing po en ial go en om he in-
e ac ion o he cu en neu on wi h hei neighbou s.
And inally, he ac i a ion unc ions a e he ones ha de e mine he s a e
o li eliness o he neu on. The li eliness o he neu on is he c ucial ac o
ha de e mines i i akes pa on he unc ioning on he ne wo k.
The ac i a ion unc ion is based on he p opaga ion unc ion and he
p e ious o he in e ac ion.
In he ollowing igu e2we can see he mos common ypes o ac i a ion
unc ions:
Figu e 2.2: Ac i a ion unc ion ypes
Finally, when all p e ious calcula ions ha e been pe o med, he exi
unc ion deli e s he esul o he neighbou ing neu ons. I he neu on is
inac i e, which means ha he ac i a ion unc ion has no been igge ed,
he exi unc ion will no deli e any esul s, u ning he neu on in o an
isola ed cell.
2h ps:// owa dsda ascience.com/comple e-guide-o -ac i a ion- unc ions-
34076e95d044
2.3. Mos used echniques and me hods 17
2.3.1.3. Main Neu al Ne wo k Types
As said be o e in he heo e ical pa , a Neu al Ne wo k is o med by
neu ons a anged in laye s. The numbe o neu ons and hei a ibu es can
a y depending on he laye and hei ole in he model. Ke as allows us o
c ea e a model, s a ing as pa ame e o he laye s hei a ibu es and he
ype o NN. The a ibu es we can speci y can be: d opou laye s, numbe
o neu ons, ype o NN, ac i a ion unc ion,...
Se e al ypes o neu al ne wo ks can be de ined in o de o c ea e ou
model. I is wo h ecalling ha he e a e nea ly in ini e ypes o Neu al
Ne wo ks. This is due o he di e si y and a ie y o possibili ies on he
design o he ne wo k and hei beha iou .
He ea e we speci y some o he models ha we ook ook in o consid-
e a ion o he c ea ion o ou ne wo k and he easons why we decided o
use o disca d hem in each case:
Feed o wa d
This is he simples model o he neu al ne wo k possibili ies. Da a
a elling h ough he neu al ne wo k, only goes in one di ec ion onwa ds o
he ou pu nodes om he inpu nodes. I s simplici y elies on he posi ion
o he neu ons and design o he ne wo k. I is a i s app oach o NN
The ollowing igu e ex ac ed om (O’Shea and Nash, 2015) shows he
ypical s uc u e o a Feed Fo wa d Neu al Ne
Figu e 2.3: Typical s uc u e o a Feed Fo wa d Neu al Ne wo k
Mul ilaye Pe cep on
18 Chap e 2. S a e o he A
The ollowing igu e3shows he s uc u e o a Mul ilaye Pe cep on Neu-
al Ne wo k:
Figu e 2.4: Typical s uc u e o a Mul ilaye Pe cep on
The Mul ilaye Pe cep on (Rosenbla , 1961) is a neu al ne wo k ha ,
gi en a numbe o inpu s, hey deli e a se ies o disc e e ou pu s. This
ne wo k is no mally used, and gi es a good esul , when he aim is o classi y
a se ies o inpu s.
The main cha ac e is ic o his ne wo k is ha he numbe o ou pu s,
as he classes o classi y, can no be he same as he numbe o inpu s.
Recu en Neu al Ne wo ks - Long-Sho Te m Memo y
Ne wo ks
The main cha ac e is ic o his ne wo k is ha i exploi s he ac o
memo y-las ing aining ha he neu al ne wo ks a e expec ed o deli e .
The ollowing igu e4shows he usual s uc u e o a LSTM Ne wo k:
Figu e 2.5: LSTM ypical s uc u e
This is done by egis e ing, and e- eeding he inpu o he ne wo k and
i s neu ons wi h he esul s o p e ious ainings. This model also o e s
3h p://www.jpa hin o ma ics.o g/
4h ps://colah.gi hub.io/pos s/2015-08-Unde s anding-LSTMs/
2.3. Mos used echniques and me hods 19
he posibili y o back-p opaga ion. Backp opaga ion is a cha ac e is ic ha
elies on he ac ha he ou pu o some o he laye s o he ne wo k, can be
u ned in o he inpu o some o he o he laye s. This is a common ea u e
in Neu an ne wo k heo y.
This echnique gi es as a esul a be e unde s anding o he s uc u e
o he song used o aining he ne wo k. The esul is simila o he one we
would ge i we chop he song in o i s pa s, and use hose se e al pa s as he
inpu o he ne wo k. The esul o doing his en iches he music wi h aking
in o conside a ion he pa s a music composi ion is di ided on, and espec s
he changes in he eeling and sensa ions ansmi ed by music when i is
played. A ull heo y why his ypes o ne wo ks a e a good op ions when i
comes o compose music can be ound in he a icle by F ançois Pache i led
"Sampling a ia ion o sequences o s uc u ed music gene a ion" (Pache e
al., 2017). A mo e gene al app oach o his ac can be ound in (Ka pa hy,
2015)
This ne wo k, due o he capabili ies ha i o e s, and he na u e o i s
s uc u e, is conside ed he bes possible solu ion o his p ojec ’s aim and
will become one o he main axes and chosen echniques o he de elopmen
o he music gene a ion module.
2.3.1.4. Gene ic Algo i hms
A gene ic algo i hm, is a compu e algo i hm ha deploys he ac o o
DNA mix u e among wo pa en s. On he DNA s uc u e o a child, we can
ind common s uc u es ound on he DNAs coming om he pa en s ha
gene a ed he child.
B ough o music composi ion, his echniques mixes ac o s s udied in
wo musical pieces and pe o ms changes in o hose ac o s and he combine
hem o c ea e a b and new musical piece (Ho ne and Goldbe g, 1991).
The mos common a ia ions o his ac o s could be:
Pe mu a ion
Swapping posi ion and mixing alues among he ac o s o c ea e new
ela ions
C osso e s

Chap e 4
De elopmen app oach owa ds
music gene a ion
The ollowing chap e will pu down in pape he de elopmen o he c e-
a ed models o music gene a ion. The e o e, i con ains he echniques used,
he inal model c ea ed, and he es s pe o med wi h he ou pu gene a ed
will be explained he ea e .
In o de o cla i y in o ma ion o he de elopmen phase, I will spli his
chap e in o h ee main pa s: gene a ion, sen imen analysis and e alua ion
o he esul ing pieces acco ding o di e en c i e ia.
Each pa will con ain he key poin s o i s phase and hey will be p e-
sen ed as he na u al s eps o ollow o compose music using he de eloped
echniques.
Al hough i is spli in h ee appa en ly independen pa s, he de el-
opmen phase was an e olu iona y and con inuous imp o emen pa h and
du ing which i was necessa y o go back o p e ious s eps, phases and ech-
niques in o de o e-design o e-e alua e p e ious decisions. Al hough i
was hough o be an au onomous sys em and concie ed as a whole, he
pa s ha o med de de elopmen will be aken as independen pa s, and
ea ed as independen componen s acco ding o he echniques used in each
one o hem.
27
28 Chap e 4. De elopmen app oach owa ds music gene a ion
4.1. Music Theo y Concep s
In o de o s a de eloping he model, i is wo h s a ing by poin ing
ou some basic music concep s:
Music No a ion: he music no a ion is he sys em used o isualize music
sounds. Music no a ion is o med by s a , no es, silences and a ia ions.
S a : he s a is he se o ho izon al lines ha he no es will be placed
on.
Music no e: a music no e is he symbol deno ing a musical sound.
Pi ch: is he music equency o he no e.
Du a ion: de e mines he leng h o ime o which a no e is played.
Music cho d: a music cho d is a se o no es played in he same ime bea .
Bea : is he basic uni o ime used in music.
Tempo: is wha se s he speed o he bea in he composi ion.
Scale: is a g oup o no es a anged in ascending o descending o de . The
mos common scales a e he majo and mino scales. This scales will be used
o de elop he heo e ical app oach o he sen imen analysis sec ion.
Ch oma ic scale: a ch oma ic scale is he ange o no es by which he
music piece will be o med.
Ascending/descending o de : he o de speci ies he succession o no es
acco ding o he ela ion o hei pi ches. I he pi ches go om low o high,
he scale is said o be ascendan , o he wise, i is descendan .
4.2. Da ase 29
4.2. Da ase
In o de o ain ou model i was necessa y o de ine a da ase . A da ase
is a se o inpu s chosen o ain ou model.
Du ing he de eloping o he p ojec , se e al da ase s ha e been used.
The i s app oach was aken using he Magen a MAESTRO1da ase , a
da ase composed by piano i uoso composi ion.
Mo eo e , o he es ing o he Ma ko Models, he da ase was en iched
wi h ce ain classical composi ions ob ained om Classical A chi es2.
Addi ionally, due o he musical aining al eady acqui ed be o e en-
olling his p ojec , I could eco d mysel using an elec onic keyboa d and
h oughou he use o Audaci y3, a simple music manipula ion and eco ding
so wa e. Using his simple so wa e, I was be able o eco d mysel playing
ce ain pieces on he piano and aining he model using hose songs.
The main eason why I chose o eco d mysel playing hose pieces, and
he compa ison be ween he expec ed and ob ained esul using hem will
be explained in he ollowing sec ions.
4.2.1. Da ase ype: he ins umen
Du ing he de elopmen o he p ojec we will ocus on gene a ing music
composed and played o piano.
Piano is a widely known ins umen , wi h a ch oma ic scale big enough
o emb ace music pieces om di e en gen es, and scales, and he e o e we
will no limi he esul ing pieces o a ixed ch oma ic scale.
The ange o no es s ablished in he piano can be seen in he ollowing
igu e4:
4.2.2. Da ase ype: no es, cho ds and songs
As men ioned abo e, he da ase can be o med by he h ee main com-
ponen s ha music is o med wi h: no es, cho ds and songs.
Due o he ac ha he de eloped Ma ko Models, as i will be explained
1h ps://magen a. enso low.o g/da ase s/maes o
2h ps://www.classicala chi es.com/midi.h ml
3h ps://www.audaci y eam.o g/
4h p://bigi .ka ika u ize.com/piano-music-no es-cha /piano-key-cha -beginne s-
he e-a e-a- ew-easy-exe cises- o.h ml
30 Chap e 4. De elopmen app oach owa ds music gene a ion
Figu e 4.1: Co ela ion be ween he ch oma ic scale and he piano keys
la e , is ocused on he p obabilis ic app oach o p eceding no es, i is nec-
essa y o ha e a ull ange o no es p eceding one-ano he in o de o s udy
he p obabili y o a no e o be he succeeding one o a gi en no e.
Mo eo e , i we look a i om he esea ch app oach, as said p e iously,
he aim o his p ojec is no o ocus only in music gene a ion, we would also
like o gene a e music based on he lea ning o al eady-exis ing composi ions.
The e o e, i is necessa y o eed he models wi h he pieces i us supposed
o lea n om.
4.2.3. Da ase ype: Gen e
Du ing he de elopmen o he wo k classical music was used o ain
he models. The eason o i is he a ie y o no es ound in classical music
compa ed o o he gen es, and, he e o e he ampli ude on he lea ning p o-
cess i impulses. The ch oma ic a ie y o he classical gen e allows a deepe
s udy o he inpu ed songs.
4.2.4. Da ase size
The size o he chosen da ase is de e mined by he numbe o pieces ha
o ms i , hose pieces will se e as he inpu o he sys em. I is necessa y
o ake in o accoun he size o he da ase . As, i de e mines in a big way
he beha iou o he model. A da ase oo big will make he model o come
up wi h pieces incohe en and o med wi h a huge amoun o in o ma ion.
4.3. Implemen ed models 31
The eason why a big da ase , in ou case, deploys an un a ou able ou pu
is because he model ies o inco po a e in o he esul ing pieces, all he
lea ning acqui ed du ing he aining phase.
On he o he hand, a small da ase , as i was expe imen ed, makes he
model come up wi h a esul oo simila o he inpu pieces o no e.
These aspec s will be explained in he ollowing sec ions.
4.3. Implemen ed models
He ea e , he models c ea ed o gene a e he music pieces, hei s uc u e
and a chi ec u e will be explained. In his module wo main echniques will
be used: Ma ko Chains and Neu al Ne wo ks.
4.3.1. Ma ko Models
The Ma ko Model c ea ed o his p ojec is no hing else han a ma ix
whe e each componen ( o med by «x», and «y» a iables) is composed by
he p obabili y ha s a es he chance o a no e o p ecede ano he no e.
I we imagine an usual ma ix, he ows and columns ha will con o m
he ma ix will be he no es hemsel es ( o med by «pi ch» and «du a ion»),
and he e o e will be he axis o he esul ing ma ix. The body, on he o he
hand, will be he p obabili y o a no e o be ollowed by any o he no e. In
his way, i we ead he ma ix shown in igu e (X), we can say ha he no e
«XX» is ollowed by he no e «YY» wi h a p obabili y o «ZZ», on he o he
hand, no e «XX» is ollowed by no e «YY» wi h a highe p obabili y.
The ma ix will be cons uc ed om he da ase gi en o he model (in
his case, a song), s udying he numbe o imes a gi en no e o he piece is
p eceded by ano he speci ic no e, and he e o e, i s p obabili y.
Once he ma ix is cons uc ed by di e en p obabili ies o a no e o be
ollowed by ano he no e, we will s a by andomly picking an elemen o he
ma ix. Once an elemen has been chosen, we will maximize he p obabili ies
c ea ing a ee o one b anch, on which, we posi ion one elemen a e he
o he . The esul will be a lis o elemen s by hei coo dena es (x,y) whe e
«x» is he p eceeding no e and «y» he successo . The lis he e o e, i we
main ain only he «y» o all elemen s, and he «x» alue o he i s elemen ,
will be a lis o no es ha will be he sequence o no es ha will o m he

32 Chap e 4. De elopmen app oach owa ds music gene a ion
Figu e 4.2: Ma ko Ma ix Diag am
inal music piece.
In igu e 4.2, a high le el ep esen a ion o his ma ix has been designed.
The use o Ma ko Chains is all abou p edic ing he nex da a on a
sequence gi en some p obabili ies and maximizing hem.
The use o Ma ko Models o music composi ion, is based on he pi-
ano oll mechanism o implemen ing music, as i condenses he no es o he
musical composi ion ch oma ic scale in o a ma ix aking in o accoun he
appea ance, and «ac i a ing» he cell o he no es he song con ains.
4.3.1.1. A chi ec u e o he solu ion
In o de o de elop all he abo e men ioned model, he ask will s a by
di iding he implemen a ion in h ee submodules:
Gene a o Module
4.3. Implemen ed models 33
Con e e Module
Ma ko Module
This s uc u e o he solu ion allows o see he di e en s eps aken o
gene a e a ull musical composi ion using he Ma ko Models app oach. The
in e nal unc ioning o each componen is explained he ea e :
4.3.1.2. Gene a o Module
This class will be used as he main d i e o he compose me hod and
will be he main con olle o he model.
Fo he implemen a ion o his module, he Mido lib a y has been used
o he handling and manipula ion o he midi e en s.
On his class we will c ea e he Ma ko Chain om he speci ied song o
melody using he Ma ko Chain Module, and once c ea ed, we will gene a e
he new melody ha will o m he ou pu o he module.
The Ma ko Ma ix will be c ea ed using he Ma ko Chain class ha
will be explained la e . And i will be sa ed in a s uc u e o a dic iona y
ha will s o e he di e en p obabili ies o he ma ices.
Once he Ma ix is c ea ed, hen i is necessa y o c ea e a lis o no es,
ex ac ed om he esul ing ma ix, ha will be con e ed in o midi e en s
so hey can o m a b and new song. Fo ha pu pose, we will use he ge
nex unc ion ha will allow us o ge , acco ding o he cu en no e, he one
ha mus be he successo o he ac ual no e acco ding o he p obabili y
es ablished by he ma ix.
The du a ion o he pieces is de e mined by he se ing o a a iable ha
enuncia es he numbe o no e he esul ing pieces will ha e.
4.3.1.3. Con e e Module
This module will allow us o con e he midi ile in o a Ma ko Chain
by ex ac ing he da a ha he midi con ains and c ea es he lis o e en s
ha will be ea ed as he Ma ko Chain module o c ea e he Ma ix.
In o de o c ea e de nodes, i goes h ough he midi ile and ex ac s he
du a ion, pi ch and eloci y o he no es. F om ha in o ma ion, he class
will c ea e he sequence o he no es ha will o m he song in midi o ma .
34 Chap e 4. De elopmen app oach owa ds music gene a ion
The eloci y a gumen o he no e is how i known he s eng h wi h
which he no e is played on he composi ion. This a gumen will be a pa
o he sen imen analysis sec ion in he u he esea ch sec ion.
4.3.1.4. Ma ko Module
This las module is he one in cha ge o c ea ing he Ma ko Chain om
he in o ma ion pa sed by he Con e e module and manage his in o ma-
ion co ec ly o c ea e he new gene a ed melody.
This class will c ea e he ma ix o no es, o med by du a ion and pi ch
o each no e ha appea s on he piece.
The o m ha his ma ix will ake i is shown in he ollowing igu e.
Figu e 4.3: Ma ko Ma ix Ou pu
As we can see in he igu e, he columns and ows o he ma ix a e
o med by wo a gumen s: he i s a gumen e e s o he midi equency
co ela ed o he no e and i s du a ion. In he ollowing igu e we can see a
ep esen a ion:
Addi ionally o he speci ied unc ion o he module s a ed abo e, some
u ili ies unc ions ha e been used:
P in Ma ix:
4.3. Implemen ed models 35
Figu e 4.4: No es s a no a ion o midi ecuency co ela ion
This unc ion allows he use o see he ma ix buil om he inpu
ha has been gene a ed. I p in s he ma ix in he o ma speci ied
abo e.I akes as a sel -a gumen a chain o no es ex ac ed om he
midi ile.
Due o he ac ha a big amoun o no es may appea in a song, when
p in ing he ma ix only a small po ion o hose no es will be p in ed,
showing he use he mos signi ican pa o he ma ix, ha is, he
no es wi h he highes p obabili y o appea ance.
Combine wo ma ices: me ge
This u ili y allows joining wo ma ices o wo gi en music pieces.
This unc ion allows he model o ake wo a gumen s ins ead o one.
The unc ioning o his e sion o he model allows o ex ac he ma-
ices ela ed o each o he a gumen s (midi iles) and combine hem
o c ea e a new ma ix om which o gene a e he ou pu o he model,
as shown in he ollowing igu e.
Figu e 4.5: Ma ko Ma ix me ging
The combina ion o bo h ma ices is done by summing he alues o
he same componen s o he ma ix, ha is, in ou model, i wo no es
42 Chap e 4. De elopmen app oach owa ds music gene a ion
and he ou pu s, a MIDI ile and a music shee , will be passed o he e alua-
o , who, a e pe o ming he co esponding calcula ions and me hods, will
gi e he componen s eedback in he ollowing o m:
Changes on a ce ain no e o g oup o no es: he e alua o can iden-
i y which no es do no e ma ch wi h he gi en music and he Ly ics
gene a ed and ells he compose which no es o be changed.
The compose has a module on modi ica ions wi h u ili ies unc ions
in o de o pe o m a ailable modi ica ions.
Make music sadde o happie acco ding o he sen imen analysis pe -
o med.
In his case, he music compose can make use o he Ma ko Chain
modules o gene a e he esul ing Ma ko Ma ix o he composed
piece.
Then, i will calcula e de Ma ko Ma ix o a sadde o happie song,
depending on he equi emen s and will me ge hose wo ma ices. The
happie o sadde song will be picked om he co esponding da ase .
A e he new ma ix has been gene a ed, a new song will be c ea ed
and he esul , as he "pa e s" a e sad, i will be sad. Du ing he
de elopmen phase will see i , his p ocess has o be pe o med se e al
imes in o de o app ecia e he change in he music composi ion. In
case se e al uns a e needed, we will s aigh away apply as many as
needed o i o ake e ec .
4.3.4. Neu al Ne wo ks
To keep explo ing he possibili ies o Compu e Gene a ed Music he
nex app oach aken was o c ea e a simple model based on gene al Neu al
Ne wo ks using Ke as. The model will allow, o es he di e en Neu al
Ne woks models (such as LSTM), as well as se ing he pa ame e s o he
ne wo ks (ac i a ion unc ions) and de ining he c ea ed model.
The use o RNN will allow wo main hings:
C ea ion o music by he i e a ions o e a Neu al Ne wo k:

4.3. Implemen ed models 43
As we alk abou a RNN, he ne wo k is ed wi h ecu en da a. Tha
means ha he da a ha en e s in o he ne , is combined wi h he
da a gene a ed by p e ious s eps in he aining phase.
As we c ea e he model using LSTM, he esul ing ou pu will ake
in o accoun all inpu ega dless o how back in ime hey a e.
Ge id o he anished g adien p oblem and he e o e c ea e
iche composi ions and be e aining.
The Vanished G adien P oblem s a es ha , when aining a Neu al
Ne wo k, due o he na u e o he aining, he ne wo k will y o co e
all possible ou pu s, and ake in o conside a ion all a ian s along he
song and could cause he anishing o pa o he in o ma ion along
he way.
Tha ac d i es us o conclude ha he esul ing es s pe o med wi h
RNN no LSTM, did no p o ide he pieces wi h a de ined s uc u e,
a ian s o modula ion, and as we will see, c ea e pieces illed wi h
eigh h no es.
Due o he na u e o he LSTM ne wo k, he esul ing song should
ha e a de ined s uc u e and no be mono onic o seem oo andom.
4.3.4.1. Ne wo k ype
The app oach aken owa ds Music Gene a ion using Neu al Ne ow ks,
ocused in he de eloping o a Recu en Neu al Ne wo , and he use o a
Long-Sho Te m Memo y Ne wo k (LSTM).
The model was c ea ed using using Py hon Anaconda Spide as he main
ool o es and implemen he model.
4.3.4.2. Music pa sing
We s a ed by c ea ing he o ma ing inpu o he LSTM ne wo k as a
sequence o e en s using Music21. Gi en ha MIDI iles is coded his way,
he esul ing pa se is a simple implemen a ion ha sa es he midi e en s
in o an a ay.
Due o he ac ha RNN beha e be e wi h nume ical alues, a he
han music no a ion alues such as ABC no a ion, i was necessa y o c ea e
44 Chap e 4. De elopmen app oach owa ds music gene a ion
a pa sing unc ion o c ea e anspose he di e en inpu s in o a nume ic
alue.
The e o e, he esul ing inpu o he ne wo k will no be a midi ile, bu
a se o nume ical consecu i e alues.
4.3.4.3. Inpu imming
In o de o decide he amoun o in o ma ion gi en o he ne wo k as he
inpu pa ame e , as well as he in o ma ion ob ained om i , i is impo an
o « im» he da a in o smalle pieces.
4.3.4.4. Da a no maliza ion
I is no s ic ly necessa y o no malize he da a, bu i has been shown
ha he AI algo i hms pe o m be e wi h no malize da a when dealing
wi h big da ase s and eases he back p opaga ion aining p ocess o he
ne wo ks (Sola and Se illa, 1997).
Fo ha pu pose, we s a ed by pa sing he midi ile om he inpu and
u ning i in o a se ies o sequences o e en o ma ed in o de o become he
inpu o he Neu al Ne wo k. The o ma chosen is o de ine a co ela ion
lis o lis s o no es an in ege alues.
4.3.4.5. Model C ea ion
The nex s ep is o c ea e he model i sel . The model will be o med by
a Neu al Ne wo k o med by se e al aining laye s.
We will also include some egula iza ion laye s wi h D opou S i as a a
(2014). D opou is a echnique o egula iza ion o neu al ne wo ks ha
allows he d opou o andom neu ons along he ne wo k o a oid o e i ing.
See igu e 4.11.
In he model de ini ion, we will also include he ac i a ion unc ion cho-
sen. Du ing he de elopmen o he p ojec , se e al ac i a ion unc ions ha e
been es ed. The esul s ob ained o m he es s will be explained la e on
in he chap e de o ed o analyse he ob ained esul s.
The ac i a ion unc ions (see igu e 2.2) es ed a e:
Sigmoid
4.3. Implemen ed models 45
Figu e 4.10: D opou laye s and o e i ing
The sigmoid ac i a ion unc ion is a unc ion whose alues ange be-
ween 0 o 1 ha has he shape o an S
Hype bolic Tangen
I is de ined be ween 1 and -1
Figu e 4.11: Hype bolic angen g aph
Rec i ied Linea Uni (RELU)
This ac i a ion unc ion comes o esol e a p oblem a ached o he use
o he Sigmoid and Tangen unc ions called he G adien P oblem.
46 Chap e 4. De elopmen app oach owa ds music gene a ion
Apa om ixing he g adien p oblem, i also p o ides mo e sensi-
i i y o he ou pu o he nodes and a oids sa u a ion o he nodes.
The use o his unc ion also p o ides he possibili y o dec ease he
numbe o d opou laye s in he neu al ne wo k.
The unc ion wo ks in a simila way o he max unc ion in common
ma hema ics. I e u n he maximum o wo gi en alues, bu i has
he cha ac e is ic ha hal o he inpu o he RELU unc ion will
deli e a lowe bounda y alue p e iously se (no mally ze o), and a
some poin in he de elopmen o he unc ion, will s a pe o ming
like a linea unc ion.
This ac , makes he ac i a ion unc ion o il e he ou pu and disca d
o dele e he union o wo gi en nodes. We can imagine, he ac i a ion
unc ion wo ks like a swi ch, deac i a ing o ac i a ing he connec ions
o he nodes depending on he inpu ed alue.
He ea e , we will show some ad an ages we could see using his unc-
ion as he ac i a ion unc ion, some o hem ecalled in (Glo o e al.,
2011):
Compu a ional Simplici y
As i can be coded as a max unc ion.
Rep esen a ional Spa si y
This unc ion is capable o e u ning a ze o alue, unlike he o he s
ha will always e u n an app oxima e o ze o alue
Linea beha iou
As i beha es e y simila ly o he linea unc ion, bu he ac o
se ing he lowe bounda y, allows he unc ion o be mo e sensi i e
and e u ning a mo e il e ed ou pu .
Used in Deep Neu al Ne wo k
Due o he Rec i ied Linea ac i a ion, and he back p opaga ion o
he ecu en ne wo ks, i allows o he ne wo ks whe e i is used o be
ained as e and wi h be e esul s in compu a ional e ms, and as
i was shown du ing he es ing phase, be e musical esul s.
Mo eo e , Ke as allowes us o change some cha ac e is ics o he model
implemen ing di e en op imize s as well as di e en ac i a ion unc ions.
4.3. Implemen ed models 47
The op imize is a pa ame e o he Ke as Neu al Ne wo k model c ea ed
ha has he aim o op imizing he pe o mance o he Ne wo k and checking
he e ec s o ac i a ion unc ion modi ica ion in he esul ing music pieces.
The e a e se e al Op imize we ha e es ed:
Adel a (Zeile , 2012)
Adam (Kingma and Ba, 2014)
Adamx (Kingma and Ba, 2014)
Nadam (Doza , 2016)
Al hough hey all ha e been es ed, he one ha deli e ed he bes esul
was he RMSp op, as said in (Bengio, 2015).
The abo e men ioned pa ame e s we e se and adjus ed du ing he es ing
and de eloping phase and he in o ma ion abou hem has been collec ed
om he Ke as Documen a ion5and om he ci ed sou ces.
4.3.4.6. Ne wo k T aining
In o de o ain he model, he p e iously p epa ed da a will se e as
inpu .
To ain he model, i is necessa y o speci y he numbe o imes he
model will go h ough he aining da a se , ha is, he epochs o i e a ions,
as well as he numbe o i ems ha will o m he aining da ase .
A e he aining o he ne wo k has comple ed, he esul ing weigh s
a e sa ed in o a «.hd 5» ile.
4.3.4.7. Music Gene a ion
Finally, in o de o gene a e he esul ing piece, i is necessa y o combine
he in o ma ion collec ed h oughou he aining, ha is, he weigh s, and
inpu da a.
I is wo h ecalling ha he inpu da a, as well as he aining da a, ha e
o go h oughou he same pa sing. This implies ha we ha e o con e
he inpu da a in o he sequences o nume ical alues, as he aining da a
in o de o be inpu o he model as i has been p e iously explained.
5h ps://ke as.io/

48 Chap e 4. De elopmen app oach owa ds music gene a ion
To gene a e he a ay o no es, i is necessa y o speci y he numbe o
i ems, ha is, no es, and his allow us o de ine he leng h o he esul ing
piece.
The ou pu o he model, he e o e, will be an a ay o no es.
4.3.4.8. Ou pu handling: e e se pa sing
In o de o con e he a ay o no es and cho ds in o a music piece, i
is necessa y o decode i , ha is, pe o m he opposi e p ocess pe o med in
he Da a P epa a ion Sec ion.
The lis o no es, hen, will be con e ed in o a Music21 Objec and,
wi h ha i em, we can sa e he co esponding Midi ile, inally ob aining he
esul o he model.
The esul ing midi ile will can be subjec ed o he same p ocess as he
ou pu om he Ma ko model in o de o con e i in o a Music Shee
using Lilypond.
4.3.5. A heo e ical app oach o Sen imen Analysis
This pa o he p ojec s a es a heo e ical app oach o Sen imen Anal-
ysis, p o en and es ed wi h he models speci ied abo e.
In o de o c ea e music, we mus no o ge abou he sen imen al pa .
The Sen imen Analysis is he science o discipline ha s udies he ac s and
ing edien s ha a e necessa y o c ea e mo ing music pieces o composi ions.
D i en by he possibili ies o musical c ea ion ha he chosen me hods
p o ided us, i was conside ed he bes op ion o c ea e a pe sonal and own
s a egy o music composi ion o c ea e sen imen analysed music.
Gi en he ac ha his p ojec is hough o be included in he p ojec
p esen ed by Ma isa y Ca los, he s a egy mus be as simple as possible in
o de o he e alua o o gi e eedback abou he "sco e" o music ha will
be analysed.
Gi en he ac ha he ac o s o a iables ha he e alua o will ake
in o accoun a e di ec ly ela ed o s yle and music heo e ical app oach
inhe i ed om pieces ha we e he o igina o s pieces, ha is, he ou pu
will inhe i some ac o s om hei "pa en s" ha he e alua o will ake
in o accoun .
Bea ing ha in mind, we will ha e o ake hem in o accoun in o de o
4.3. Implemen ed models 49
egene a e music, se e al ac o s such as hose ones a e me onome ( empo),
key, mode, e c.
The e o e he main idea is o c ea e a pe sonalized da ase ha con ains
classi ied pieces o music and acco ding o he eedback gi en by he e alu-
a o , he e-gene a ion o modi ica ion o music, will ake in o accoun wo
main ac s: he o iginal da ase om which he song has been gene a ed and
he c ea ion o a new da ase o egene a e he whole song o pa o i o
comply wi h he gi en eedback.
The esul ing piece, he e o e, could be gene a ed by a Wagne Da ase ,
bu i he E alua o module c ea ed by Ca los, ells us o so en a he musical
composi ion, we may in oduce in o he da ase some Chopin sona as as hey
a e cha ac e ized o being in a slow empo and lowe pi ch o he ch oma ic
scale, compa ed o he Wagne composi ion.
4.3.5.1. Tempo & Scale
To igu e ou how his da ase s can be c ea ed we had o s ablish some
ini ial cons ain s o assump ions and i he eedback o he song spins a ound
he sadness o happiness o i . We ake in o accoun his wo ac o s: empo
and scale.
Tempo o he song:
We will assume, as s a ed in (Wasse mann e al., 2003), ha mos o
he slow music composi ions c ea ed in classical music, end o be quie
and calm o sad. The e o e, he songs c ea ed o composed om pieces
wi h his ing edien , a e mo e likely o be sad o calm.
Scale o he song
I he scale o he song is composed by majo cho ds o no es ha a e
no mally high, we assume he song is a happy one and he e o e he
esul ing pieces gene a ed om his piece a e mo e likely o be happie .
4.3.5.2. Veloci y
As old be o e, eloci y is an a gumen in a midi e en ha is used o
de e mine he s eng h wi h which he no e is played by he ins umen .
As we can all ecognize, agg essi e and happy songs, a e ela ed mo e o
no es being played wi h a high in ensi y, which in music would co espond
50 Chap e 4. De elopmen app oach owa ds music gene a ion
o o e o mezzo o e modula ions. The main aim o his ac is ha he
pieces y o ansmi a sense o joy and pleasu e o he lis ene .
On he o he hand, sad o melancholic songs a e no mally ela ed o a
so e play, a mo e delica e and p obably quie sound.
Bea ing his in mind, he e o e, by he analysis o he music pieces, we
can ecall ha he quie songs will be mo e likely ela ed o sad songs while
"ha de " songs will, somehow be ela ed o happy songs.
The sen imen analysis module will ake all his in o accoun and will
d i e he ins uc ions om which he music will be changed and modi ied
acco ding o he eedback gi en by he e alua o .
E en hough he sen imen analysis module is based on a simple p inciple,
as shown in he es ing phase, e u ns good esul s and he ou pu s can be
conside ed sen imen sensible. O cou se, i has some limi a ions, due o he
size o he da a se and he na u e o he me hodology, bu as he app oach
e u ns good esul s, i can be conside ed a success.
4.3.6. Lea ned lessons and esul compa ison
He ea e I will poin ou he mos signi ican keypoin s conce ning ou -
pu s and models c ea ion:
Impo ance o Pa ame e s
In bo h models he alues se o he pa ame e s ha con ol de model,
i u ned ou o be highly impo an , and condi ions he esul s ob-
ained h oughou he use o he model.
MIDI manipula ion
Rega ding MIDI and audio manipula ion has been su p ising he amoun
o possibili ies o handle and manipula e hose iles
To handle music objec s, se e al lib a ies and sou ces we e ound and
many o hem, es ed. We ocused on wo main modules, mido and
music21 as MIDI handle s, bu o he , such as p e y-midi we e es ed.
App oach o Piano Rolls
Du ing he i s ew s eps on he c ea ion o he Neu al Ne wo k model,
he possibili y o using piano olls as he inpu was concei ed. A piano
Roll was a s uc u e used a he begining o music his o y as a oll
4.4. Resul s and conclusions o he de eloping phase 51
pe o a ed pape which con ols he mo emen o he keys in a pianola
o simila ins umen .
This app oach was a ac i e, because, as a compu e s uc u e, he
piano oll is no hing mo e han a ma ix o a dic iona y. As he ap-
p oach seemed a ac i e we decided o s a explo ing his pa h and
we s a ing c ea ing he sys em o handle piano olls and con e hem
in o handleable music objec s.
No a ion
On he no a ion ield, we also ied di e en app oaches, such as ABC
no a ion. The use o ABC no a ion o ill he models we e disca ded
due o he p e iously explained ac conce ning he Ne wo k Capabil-
i ies and i s dec ease when using no nume ical alues.
ABC no a ion was a ac i e a he beginning due o he simplici y
and he amilia i y o he no a ion. The no a ion allowed simple CSV
modi ica ions o be pe o med as well as many o he ope a ions.
As can be seen, many pa hs leaded o emp y oads du ing he de eloping
o he Music Compose s, bu om all pa hs we lea ned some hing new, and
we do no disca d o use hem la e on in he u u e wo k.
All app oaches aken we e alid, bu we only kep he ones we hough ,
had possibili ies o se e ou aim.
4.4. Resul s and conclusions o he de eloping phase
Du ing he es ing phase o he de eloped models, some p oblems we e
aced and some conclusions we e deduc ed om he esul s.
We can ecall he nex conclusions depending on he model being es ed:
Neu al Ne wo ks
Composi ion Time:
Gi en he complexi y o he p ocess in compu a ional e ms, he Neu al
Ne wo k e u ned good esul s when ained o a long ime.
E en hough he aining ime highly imp o ed he esul , no much
di e ence was achie ed when he ime o e passed six o se en hou s.
58 Chap e 5. Conclusiones y T abajo Fu u o
5.5. Aplicaciones en el ámbi o educa i o
Tan o es e p oyec o, como las écnicas u ilizadas y los esul ados ob enidos
esul an ele an es pa a los en o nos educa i os de ca a a enseña música a
es udian es de di e en es edades.
Dado que la gene ación de música median e los mé odos u ilizados ex-
plo an los ac o es c ea i os inculados a la música, al mismo iempo que
incen i an la c ea i idad y la cu iosidad po los en o nos de desa ollo de
so wa e y ecnologías eme gen es.
Como se puede obse a , median e las líneas de inidas como abajo u-
u o, el espec o de aplicación de es e sis ema composi i o, así como las
écnicas u ilizadas, dejan abie o un amplio abanico de posibilidades pa a
la c eación de nue os sis emas y aplicaciones en di e en es ámbi os, con-
ibuyendo así a la di ulgación de las écnicas gene ado as y eo ía y gen-
e ación musical.

Chap e 5
Conclusions and Fu u e Wo k
The wo k de eloped du ing he mon hs o in es iga ion and implemen-
a ion o he models allowed me o e idence he g ea possibili ies ha he
music gene a ion ield has o o e , as well as digging in o he de eloped
AI- ela ed echniques.
The used echniques and he di e en implemen a ions de eloped and
es ed, which, as al eady said, ha e a ached hei own al eady-speci ied ad-
an ages and disad an ages, allowed me o e idence he possible applica ions
ha hose echniques o e , bea ing in mind hei na u e.
The esea ch phase o he p ojec , al hough some imes di icul due o
he ea ly s ages on which he echniques a e, and he amoun o in o ma ion
ela ed, allowed me o see he g owing p ocess and con inuous imp o emen s
hese echniques a e subjec ed o. These ac s, as well as he con inuous
lea ning and imp o emen p ocess ha e been g ea ly sa is ying.
The music composi ion gene a ed h oughou he execu ion o he mod-
els, e en hough can no be conside ed o be pe ec in musical e ms, can
be conside ed as a angible p oo o he possibili ies ha he Ma ko Models
and Neu al Ne wo ks ha e o o e .
In his inal chap e , I will expose he main ideas and uses hese ech-
niques can be subjec ed o, beyond he me e music gene a ion p ocess.
59
60 Chap e 5. Conclusions and Fu u e Wo k
5.1. Composi ion o hea e imp o isa ion
One o he use and imp o emen s ha I would like o see in he ollowing
yea s o come, is he use o hese echniques when c ea ing he sound ack
o a con e sa ion had by wo people. In he las ew yea s, he e has been an
inc easing endency o he p oduc ion o hea ical p oduc ions based on he
imp o isa ion o wo o mo e people. In hose p oduc ion, he use o music
is subjec ed o he low o he con e sa ion, and by na u e unp edic able and
ee.
The model mus be able o "lis en" he show and imp o ise, along he
ac o s, possible music accompanimen s ha i s he subjec b ough o ac-
ion. This composi ion mus be able o change along he way h oughou
he de eloping o he play, and adjus music pa ame e s such as dynamics,
o i he play.
5.2. Classical pieces comple ion
Ano he possibili y ha a ises om he use o hese models, is he com-
ple ion o classical music pieces ha , due o di e en ac o s, ne e go o
be inished. This implies ha , he model, s a ing om he s udying o clas-
sical composi ions, mus be able o, gi en a pa o an uncomple ed song,
comple e he gi en piece, ollowing he endencies and cha ac e is ics o he
o iginal compose .
This echnique would be aplicable on pieces such as he Moza Requiem,
in he A ia o choi «Lac ymosa», which, he was no be able o inish. Ad-
di ionally, his echnique could also be applicable o he Schube ’s Eigh h
Symphony, popula ly known as he «Un inished» o he B uckne ’s «Nine
Symphony», which, as «Lac ymosa» we e no inished by he o iginal com-
pose .
Big ech and echnological companies ha e be ocused on he de eloping
and a ising endency o hese echniques. This is he case o Huawei (Huawei,
2019), which ied o inish up he «Un inished Schube ’s Symphony». E en
hough he esul s ha e no been he bes ones expec ed (Man illa, 2019). ,
he ac ha companies such as Huawei in e es hemsel es on he de eloping
o hese echniques, ce ainly shows ha hey a e acing he igh way and
i is wo hy o keep esea ching.
5.3. Composición pa a películas y ideos 61
5.3. Composición pa a películas y ideos
As i has been explained in he sec ion de o ed o he «S a e o he A »,
he ge man compose Hans Zimme , decided o in es and ge on boa d on
a p ojec ocused on music gene a ion om compu e gene a ed echniques
h oughou he VJAM pla o m.
This echniques, as Hans Zimme , could be ocused on he c ea ion o
music o ilms, ideos and ca oons.
In his same esea ch line, I would like o ecall he s udied b ough up
in Muphic (Johan Be and, 2012), a composi ion sys em based on s a ic
images om which a music is composed ha ma ches he images.
5.4. Voice assis an in eg a ion
Du ing he de elopmen o he p ojec he e has been an unsuccess ul
app oach o he iabili y o include he models in o a skill c ea ed o oice
assis an s such as Google Home and Alexa. This skill would be able o
compose music using hese assis an s.
The inc ease in he use o hese assis an s ha e g own exponen ially in
he las ew yea s, and hey u ned ou o be an in e es ing ield o he de-
elopmen o skills o en ich hose sys ems. These skill would be capable, o
ins ance, o dynamically composing music pieces such as lullabies o ambien
music o encou age concen a ion. This c ea ions, would, a he same ime,
exploi he "Moza E ec "(Jenkins, 2001), which s a es ha he con inu-
ous hea ing o musical pieces composed by Moza , gi en hei na u e and
dynamic composi ion, causes imp o emen s in he long e ms o people’s
de elopmen capaci ies.
F om he ei e a ed s udied o Moza pieces, he skill will be able o
gene a e music based on endings used by Moza in his composi ions and,
he e o e, impulse he, so called, "Moza E ec " in he lis ene s.
5.5. Applica ions in he educa ional en i onmen
This p ojec , as well as he used echniques o de elop i , and he ob-
ained esul s, u ned ou o be ele an o educa ional en i onmen s in
o de o each music o s uden s o all ages.
62 Chap e 5. Conclusions and Fu u e Wo k
Gi en he ac ha music gene a ion h oughou used me hods exploi
and y o imi a e he c ea i i y ac o inhe en in human beings, a he same
ime, hey expec o impulse he use o eme ging echnologies and so wa e
de elopmen ac i i ies.
As i has been p o en, h oughou he di e en ac i i y lines de ined o
u u e wo d, he spec um concei ed o he applica ion o music gene a ion
sys ems, as well as he echniques used o de elop hem, lea e open a huge
ange o possibili ies o he c ea ion o o new sys ems and applica ions
in di e en ields, including popula iza ion o gene a ion and composi ion
musical heo y and echniques.
Annex A
5.6. Annex A: Ou pu Sample
The ollowing music shee is a sample o he ou pu s gene a ed by he
sys em
63













 
 

2







 

3





 


  
 
4
 




 

6





7




 
 

8







 

9









10 






 
 

11 


 


 

12 


 

 
13  
64 Chap e 5. Conclusions and Fu u e Wo k
2









15

Music eng a ing by LilyPond 2.18.2—www.lilypond.o g
5.6. Annex A: Ou pu Sample 65
Bibliog aphy
I I ha e seen u he han o he s, i is
by s anding upon he shoulde s o gian s.
Isaac New on
An onio Capa ini López, L. P. M. Gen e classi ica ion o elec onic
music. 2017.
Aucou u ie , J.-J. and Pache , F. Rep esen ing musical gen e: A s a e
o he a . Jou nal o New Music Resea ch, Vol. 32(1), 83–93, 2003.
Basha in, G. P.,Lang ille, A. N. and Naumo , V. A. The li e and
wo k o aa ma ko . Linea algeb a and i s applica ions, Vol. 386, 3–26,
2004.
Bengio, Y. Rmsp op and equilib a ed adap i e lea ning a es o noncon ex
op imiza ion. co abs/1502.04390, 2015.
Boylan, J. H. Hal in" 2001: A space odyssey": The lo e sings his song.
Jou nal o Popula Cul u e, Vol. 18(4), 53, 1985.
B aga, M. The e basize was da id bowie’s 1995 ly ic-w i ing mac app.
2016.
B yne , B. The piano oll: a aluable eco ding medium o he wen ie h
cen u y. PhD hesis, Depa men o Music, Uni e si y o U ah, 2002.
Ca , C. and Zukowski, Z. Gene a ing albums wi h sample nn o imi a e
me al, ock, and punk bands. In P oceedings o he 6 h In e na ional
Wo kshop on Musical Me ac ea ion (MUME 2018). 2018.
67