i
GAN Hype pa ame e s sea ch h ough
Gene ic Algo i hm
P ojec Wo k p esen ed as pa ial equi emen o ob aining he
Mas e ’s deg ee in Ad anced Analy ics
Umbe o Tamma o
ii
NOVA In o ma ion Managemen School
Ins i u o Supe io de Es a ís ica e Ges ão de In o mação
Uni e sidade No a de Lisboa
GAN hype pa ame e s sea ch
h ough gene ic algo i hm
Au ho :
Umbe o Tamma o (M20190806)
No a In o ma ion Managemen School
Mas e ’s in ad anced Analy ics
Supe iso :
P o . Mau o Cas elli
No a In o ma ion Managemen School
No embe 2021
iii
DEDICATION
Dedica o a Mamma, Papà e Fede ica.
Nonos an e la dis anza, il os o suppo o e a e o si a sen i e o e come semp e.
G azie di u o.
i
INDEX
1. INTRODUCTION ................................................................................................................................ 2
2. GENERATIVE ADVERSARIAL NETWORK ............................................................................................ 4
2.1 S uc u e o GAN ............................................................................................................................ 5
2.2 T aining GANs ................................................................................................................................. 6
2.3. Common P oblems o GANs ......................................................................................................... 7
2.3.1 Mode Collapse ........................................................................................................................ 7
2.3.2 Failu e o Con e ge ................................................................................................................ 8
2.3.3 Vanishing G adien s ............................................................................................................... 8
2.4 Wasse s ein GAN ........................................................................................................................... 9
2.5 Wasse s ein GAN wi h G adien Penal y ....................................................................................... 9
3. E olu iona y Compu ing ................................................................................................................ 10
3.1 Gene ic Algo i hm ........................................................................................................................ 11
3.2 Gene ic ope a o s ........................................................................................................................ 12
3.2.1 C osso e ope a o ............................................................................................................... 12
3.2.2 Mu a ion ope a o ............................................................................................................... 13
3.2.3 Selec ion ope a o ................................................................................................................ 13
3.3 Main Ad an ages ......................................................................................................................... 14
4. Rela ed Wo k ................................................................................................................................. 15
4.1 App oaches esea ched ............................................................................................................... 15
4.1.1 E ol ing connec ion weigh s ................................................................................................ 15
4.1.2 Ne wo k A chi ec u es ......................................................................................................... 16
4.1.3 Hype pa ame e s E olu ion ................................................................................................. 17
4.1.4 GAN-Speci ic App oaches ..................................................................................................... 18
5 Me hodology ................................................................................................................................... 20
5.1 P oposed App oach ...................................................................................................................... 20
5.1.1 Hype -Pa ame e s Encoding ................................................................................................ 20
5.1.2 Gene ic Ope a o s ................................................................................................................ 21
5.1.2 Fi ness Func ion.................................................................................................................... 22
5.1.3 Algo i hm s eps .................................................................................................................... 23
5.2 Da a .............................................................................................................................................. 24
5.3 Expe imen al Se up ...................................................................................................................... 26
5.3.1 Equipmen ............................................................................................................................ 26
5.3.2 Neu al Ne wo ks A chi ec u es ............................................................................................ 26
5.3.3 Pa ame e s o Gene ic Algo i hm ........................................................................................ 29
6. Resul s ............................................................................................................................................ 29
6.1 Me ics used ................................................................................................................................. 29
6.1.1 Basic S a is ics ...................................................................................................................... 30
6.1.2 Dis ibu ions ......................................................................................................................... 31
6.1.3 Co ela ion ............................................................................................................................ 31
6.2 Final esul .................................................................................................................................... 32
7. Conclusions and Fu u e Wo k ........................................................................................................ 33

i
LIST OF FIGURES
2.1. Human aces gene a ed by GAN ........................................................................................................ 4
2.2. Schema ic ep esen a ion o a GAN .................................................................................................... 5
2.3. P ocess o lea ning o a GAN ............................................................................................................... 6
2.4. P ocess o lea ning o a GAN ............................................................................................................... 7
3.1. Pheno ype-Geno ype o gene ic algo i hm ...................................................................................... 10
3.2. Flow cha o gene ic algo i hm ........................................................................................................ 11
3.3. One poin c osso e .......................................................................................................................... 12
3.4. Bi lip mu a ion................................................................................................................................. 13
3.5. Roule e wheel selec ion ................................................................................................................... 13
4.1. Ma ix encoding o a solu ion ........................................................................................................... 15
4.2. EA pa ame e s ................................................................................................................................... 17
4.3. O iginal GAN compa ed wi h E-GAN ................................................................................................. 18
5.1. Example o indi idual ........................................................................................................................ 20
5.2. Chosen gene ic ope a o s ................................................................................................................. 21
5.3. Values used o mu a ion .................................................................................................................. 21
5.4. Scheme o he algo i hm s eps ......................................................................................................... 23
5.5. Basic s a is ics able .......................................................................................................................... 24
5.6. Dis ibu ion o a iables .................................................................................................................... 25
5.7. Gene a o .......................................................................................................................................... 27
5.8. Disc imina o ..................................................................................................................................... 28
6.1. Bes pa ame e s ound ...................................................................................................................... 32
6.2. Resul s o expe imen ....................................................................................................................... 32
1
ABSTRACT
Recen de elopmen s in Deep Lea ning a e ema kable when i comes o gene a i e models. The main
eason o such p og ess is because o Gene a i e Ad e sa ial Ne wo ks (GANs) [1]. In oduced in a
pape by Ian Good ellow in 2014 GANs a e machine lea ning models ha a e made o wo neu al
ne wo ks: a Gene a o and a Disc imina o . These wo compe e amongs each o he o gene a e new,
syn he ic ins ances o da a ha esemble he eal one. Despi e hei g ea po en ial, he e a e p esen
challenges in hei aining, which include aining ins abili y, mode collapse, and anishing g adien . A
lo o esea ch has been done on how o o e come hese challenges, howe e , he e was no signi ican
p oo ound on whe he mode n echniques consis en ly ou pe o m anilla GAN. The pe o mances o
GANs a e also highly dependen on he da ase hey a e ained on. One o he main challenges is ela ed
o he sea ch o hype pa ame e s. In his hesis, we y o o e come his challenge by applying an
e olu iona y algo i hm o sea ch o he bes hype pa ame e s o a WGAN. We use Kullback-Leible
di e gence o calcula e he i ness o he indi iduals, and in he end, we selec he bes se o pa ame e s
gene a ed by he e olu iona y algo i hm. The pa ame e s o he bes -selec ed indi iduals a e
main ained h oughou he gene a ions. We compa e ou app oach wi h he s anda d hype pa ame e s
gi en by he s a e-o -a .
2
1. INTRODUCTION
A i icial neu al ne wo ks ha e success ully been used in a la ge numbe o applica ions, in ecen yea s
hei g ow h has led o ex ao dina y indings. Among hem, he Gene ic Ad e sa ial Ne wo ks a e hose
ha mos cap u ed he a en ion o he public. GANs ha e been used o a a ie y o gene a i e asks,
a ying om images o ime-se ies da a leading o imp essi e esul s. All hese algo i hms,
un o una ely, c ea e se e al p oblems since i is necessa y o es ablish se e al pa ame e s o hei
ne wo k design. Fo ins ance, in he case o mul ilaye pe cep ons, i is necessa y o es ablish lea ning
a e, ini ial weigh s, numbe o hidden laye s, and so on. The sea ch o he igh pa ame e s has become
one o he main challenges o ge ing hese ools o wo k. These p oblems usually a e app oached using
me hods as andom sea ch and g id sea ch. Al hough hey usually wo k ine, hey a e ime-consuming
and o en do no lead o he bes esul s. Among he possible al e na i es o ackle hese p oblems,
he e is one ha is leading o g ea esul s in li le ime: he e olu iona y algo i hms (EA). Inspi ed by
biological e olu ion, EAs imi a e e olu ion ia ep oduc ion, mu a ion na u al selec ion, and su i al o
he i es . They sea ch mo e o less andomly in he space o solu ions, paying special a en ion o hose
zones whe e he e alua ion unc ion is a maximum leading o as e and be e esul s compa ed o
adi ional me hods. In his hesis we p esen a possible way o apply EAs o iden i y he bes se o
hype pa ame e s o a WGAN-GP, ying o ou pe o m he s anda d se .
Ou wo k has been inspi ed a lo by he esea ch done by academics in ecen yea s. A lo o echniques
ha e been s udied on how o apply a gene ic algo i hm o he pa ame e s esea ch o neu al ne wo ks,
some ocused mo e on he s uc u e o he a chi ec u e and o he s on he uning o he hype -
pa ame e s. We ha e decided o ocus mainly on he sea ch o hype -pa ame e s. Using a p ede ined
ne wo k, we wan ed o see how much he esul s would change jus wi h gene ic esea ch o he bes
se o alues. Fundamen al o ou algo i hm, has been he esea ch published on [17]. Al hough ou
app oach has been simila o his one, we ha e ocused on a di e en se o pa ame e s o une. Also,
mos o he esea ch is ocusing on using and imp o ing GAN using con olu ional laye s. These
a chi ec u es happen o be he mos used because he images a e he mos common da a o wo k wi h
when dealing wi h GANs. In ou case, we wan ed o check i he esea ch would apply also o abula
da a, ha al hough can seem easie o wo k wi h, o en shows way mo e complica ed dis ibu ions.
The hesis is o ganized in he ollowing way:
3
In sec ion (2) we discuss he basic concep s o he GANs, gi ing he backg ound s udy conduc ed o he
hesis.
In sec ion (3) we explain he idea behind E olu iona y Algo i hms and how hey a e cons uc ed,
p esen ing he basic ope a o s wi h which hey a e buil .
In sec ion (4) we p esen a e iew o he a ious app oaches ound in he bibliog aphy and he di e en
aspec s ha a e possible o e ol e in neu al ne wo ks.
In sec ion (5) we p esen he wo k p ojec ha has been done, in oducing ou p oposed app oach. We
u he desc ibe he expe imen al se ups and he de ailed esul s o he p oposed me hods.
10
would lead o di icul ies in he in e ac ions be ween cos unc ion and weigh cons ain . The solu ion
o hese p oblems has been in oduced wi h an al e na i e o weigh clipping: g adien penal y.
A di e en iable unc ion, o be conside ed 1-Lipschi z mus ha e i s g adien s wi h he no m a mos 1
e e ywhe e. To achie e i hey cons ained di ec ly he g adien no m o he c i ic’s ou pu , so he new
objec i e unc ion p oposed is
The o mula is composed o wo pa s, he i s one being he o iginal c i ic loss and he second one
being he g adien penal y. This me hod has been p o ed o imp o e he aining speed and sample
quali y compa ed o weigh clipping.
3. EVOLUTIONARY COMPUTING
Since he in en ion o he compu e , he idea o e olu ion as a me apho o p oblem-sol ing has been
in oduced. In he 1970s and 1980s, his idea has been de eloped in o di e en algo i hmic
implemen a ions [8]. E olu iona y Compu ing has been used in he scope o op imiza ion p oblems.
These s ochas ic op imiza ion echniques a e based on na u al e olu ion and he i es s a egy ound
in biological o ganisms. They ha e been success ully applied o sol e complex op imiza ion p oblems
ac oss di e en a eas. Thei abili y o be sui able o such a a ie y o di e en p oblems is by cause o
he ac ha hey can be hough o as wo king on wo le els. On a highe le el, we ha e he pheno ype
ha ep esen s a collec ion o aspec s (like eye colo ) and ha is wha we change in his algo i hm. On
a lowe le el, in o de o wo k wi h his kind o abs ac concep , we ha e he geno ype, which is he
ep esen a ion o such abs ac aspec s in a o m ha can be manipula ed.
Figu e 3.1. Pheno ype-Geno ype mapping

11
This wo-le el ep esen a ion makes i possible o adap a small g oup o possible geno ypes o many
kinds o pheno ypes. F om his ield, a lo o algo i hms ha e been a ising, such as gene ic algo i hms,
e olu iona y s a egy, gene ic p og amming, and so on.
3.1 GENETIC ALGORITHM
Gene ic algo i hm (GA) has been i s ly p esen ed by J. Holland in 1975 [9]. GA is an algo i hm o mimic
he na u al e olu ion o species, i is widely applied o op imiza ion p oblems. The basic algo i hm is
o en called a simple gene ic algo i hm (SGA) and om his base, a lo o imp o ed algo i hms ha e been
in oduced.
Figu e 3.2. Flow cha o a Gene ic Algo i hm
In igu e 3.2 is possible o see he classical p ocess o he gene ic algo i hm. I s a s wi h a popula ion
o indi iduals (ch omosomes) andomly ini ialized. Then by applying di e en gene ic ope a o s i ies
12
o e ol e hese indi iduals o i e ones, in o de o make hem be e han he p e ious ones in e ms
o a ce ain measu e called i ness unc ion.
3.2 GENETIC OPERATORS
The gene ic ope a o s ha a e mean o imp o e he indi iduals a e h ee: selec ion, c osso e , and
mu a ion.
3.2.1 C osso e ope a o
The c osso e ope a o is he main p ocess o gene a e new indi iduals, i consis s o exchanging
in o ma ion be ween wo di e en indi iduals. The c osso e is inspi ed by ep oduc ion and he
biological c osso e .
Figu e 3.3. One-poin C osso e
The easies one is he one-poin c osso e , which consis s o selec ing a andom poin in he
ch omosomes and hen swapping he ails o he wo pa en s o p oduce o sp ing. The e a e mul iple
a ia ions o his ope a o , such as mul i-poin c osso e whe e you selec mo e han one poin o
uni o m c osso e whe e each gene is andomly selec ed o be swapped. In o de o decide which one
is he mos i ed o he p oblem we need o look i he e a e any cons ain s and how he me hod
selec ed would a ec he indi iduals. The c osso e is used o he explo a ion o he sea ch space (space
o all possible solu ions) because i makes mo e signi ican changes o he ch omosomes.
13
3.2.2 Mu a ion ope a o
Inspi ed by biology, he mu a ion is an al e a ion o he genes in a ch omosome. The main pu pose o
he mu a ion is o in oduce di e si y in he popula ion, in o de o a oid he con e gence o all simila
indi iduals. A classic example o mu a ion is he bi - lip mu a ion. In a bina y encoded ch omosome, one
o he andom genes is selec ed and hey a e lipped.
Figu e 3.4. Bi Flip Mu a ion
As o c osso e , no all he mu a ions a e well- i ed o e e y p oblem. The mu a ion is conside ed
essen ial o he con e gence o he gene ic algo i hm o i s abili y o a oid local minima by p e en ing
simila ch omosomes. Con a y o c osso e , he mu a ion is used o he exploi a ion o he
ch omosomes because i emains “close ” o he o iginal pa en .
3.2.3 Selec ion ope a o
The selec ion is he s age o he GA whe e he ch omosomes o be passed on o he nex gene a ion a e
chosen. This is one c ucial s ep o d i e he gene a ions in o be e solu ions. A good balance be ween
popula ion di e si y and good ch omosomes mus be ound in o de o a oid p ema u e con e gence.
In case o p ema u e con e gence, he popula ion, con e ging oo ea ly, would lead he algo i hm o
ge s uck in o local op ima esul ing in he ine iciency o he sea ch o he bes indi idual. One o he
mos popula ways o selec ing is basing he choice on he i ness o he indi iduals. The i ness o an
indi idual is de e mined by a unc ion ha akes as inpu he candida e solu ion and p oduces as ou pu
14
he sco e o de e mine how good o i is he solu ion. Usually, he i ness unc ion ma ches he objec i e
unc ion o he p oblem. One common i ness p opo iona e selec ion is he Roule e Wheel.
Figu e 3.5. Roule e Wheel Selec ion
The main idea behind he oule e wheel is ha be e indi iduals ge a highe chance o being chosen,
and hese chances a e p opo ional o i ness. Once de e mined he pe cen ages o he indi iduals
(po ions o he wheel) we spin he wheel n imes o selec n indi iduals. The selec ed indi iduals will be
pa o he nex gene a ion o indi iduals. As pe he o he ope a ions ha a e di e en ways o
implemen he selec ion o he indi iduals.
3.3 MAIN ADVANTAGES
The majo ad an ages o he E olu iona y Algo i hms compa ed wi h adi ional op imiza ion
echniques a e [10]:
• EAs end o escape mo e easily om he local op imum because o he andomness in oduced
by he ope a o s.
• EAs do no equi e p io knowledge on he speci ic domain e en hough, i a ailable, i could be
exploi ed.
• EAs a e concep ually simple and ela i ely easy o implemen .
• EAs do no equi e objec i e unc ion o be con inuous and can be used in algeb aic o m.
The e a e also some p oblems ega ding EA, such as poo pe o mance in handling cons ain s, long
compu a ion ime, and high compu a ional complexi y when he solu ion space is ha d o explo e.
15
4. RELATED WORK
In he la es yea s, gi en he di icul y o inding he bes pa ame e s o neu al ne wo ks, mul iple
expe imen s using gene ic algo i hms ha e been conduc ed. The app oaches o he usage o gene ic
algo i hms ha e been a ious. Th ee main app oaches can be ound: e olu ion o connec ion weigh s
ha is a global app oach o he ini ial weigh s and biases o ob ain a as con e gence; ne wo k
a chi ec u e e olu ion ha implies an adap a ion o he ne wo k opology o es ablish i s lea ning and
gene aliza ion abili y; lea ning pa ame e s and ule e olu ion ha is he adap a ion o he
hype pa ame e s and lea ning ules [11].
4.1 APPROACHES RESEARCHED
4.1.1 E ol ing connec ion weigh s
The ini ial weigh s o a neu al ne wo k can highly in luence he speed o con e gence in
backp opaga ion, since, depending on he poin o he sea ch space om which i has s a ed, be e o
wo se solu ions can be ob ained. Weigh e olu ion in ol es he se ing ini ial alues and he way he EA
changes hem. The simples me hod o all is based on making a andom ini ializa ion, o he me hods
equi e s a is ical analysis o he aining da a, which makes he p ocess mo e complica ed bu a he
same ime mo e powe ul. Chang e al. [12] esea ched he op imiza ion o he ini ial weigh s applied o
hip bone ac u e p edic ion. Thei s a egy was o de ine o each a i icial ne wo k he aining da a
and le he GA ind he op imum ini ial weigh s. The encoding o he weigh and biases consis ed o a
ep esen a ion o one digi . Gi en he ac ha he ini ial popula ion’s ange would a ec he sea ch
e iciency, hey p oposed o na ow he ange be ween -0.5 and 0.5. Fo he e alua ion o he
ch omosomes, he mean squa e e o has been used, ep esen ing how he solu ion was i o he
p oblem. In o de o a oid choosing bad solu ions hey pu a h eshold on he alida ion se ’s e o ; in
case he e o was highe han wha was expec ed he ch omosome would no be chosen.

16
The s opping c i e ia consis ed in s opping he algo i hms a e 100 epochs. The s udy ound ou ha ,
despi e all he di icul ies and limi a ions ound in he design o he algo i hm, using a simple GA has
been p o ed o be e ec i e o imp o ing he accu acy o neu al ne wo ks.
Figu e 4.1. Ma ix encoding o a solu ion.
The same indings we e ob ained by O i e e al. [13]. Thei s udy was also conduc ed on he ini ializa ion
o he weigh s. Di e en om he la e was he codi ica ion o a solu ion, ma ices we e used ins ead
o simple digi s. (Figu e 4.1). In he pape di e en ypes o c osso e ha e been used and has been
p o en ha he e a e signi ican di e ences be ween he esul s using di e en c osso e ope a o s.
O he han hese li le di e ences he esul s we e he same: uning he ini ial weigh s ia gene ic
algo i hm has been s a is ically demons a ed ha led o signi ican ly be e pe o mances.
4.1.2 Ne wo k A chi ec u es
Un il now, he a chi ec u es ha e been designed manually by expe s wi h enough expe ience. The
ne wo k s uc u e is e y impo an , a poo -designed a chi ec u e could lead o o e i o lead o
di icul ies in lea ning. The design o a op imal ne wo k a chi ec u e can be o mula ed as a sea ch
p oblem in he a chi ec u e space, whe e each poin ep esen s a possible ne wo k a chi ec u e [11].
17
One o he mos impo an algo i hms o he e olu ion o he opog aphy o he ne wo ks has been he
NEAT. In oduced by S anley & Mikkulainen, [14] i showed he ad an ages o e ol ing he a chi ec u e
simul aneously wi h he weigh s. The NEAT algo i hm uses a di ec encoding: e e y node and e e y
connec ion is s o ed in he DNA. F om his base, a chi ec u e esea che s ha e buil me hods o de elop
e olu iona y algo i hms o a chi ec u e design.
Es eban e al. [15] p oposed an e olu iona y p ocess o disco e he a chi ec u es au oma ically. In hei
me hod, each indi idual consis s in a ained a chi ec u e and du ing each e olu iona y s ep, wo
indi iduals a andom a e chosen and compa ed by hei i nesses. Gi en he high compu a ional
expense, hey de eloped a kind o g id compu ing solu ion, making he compe i ions in pa allel, he
compu e s in he a chi ec u e a e called wo ke s. This kind o scheme uses epea ed pai wise
compe i ions, which makes i an example o a ou namen selec ion ope a o , also i p e en s wo ke s
om idling when hey inish ea ly. The encoding used is sligh ly di e en han o he s: each neu al
ne wo k a chi ec u e is encoded as a g aph. The e ex in he g aph ep esen s a ank-3 enso (because
o images, hey always ha e h ee dimensions, he i s wo o he spa ial coo dina es and he hi d one
o he numbe o channels) o ac i a ions. The mu a ions hey chose we e due o he simila i y o he
ac ions ha a human designe may ake when imp o ing an a chi ec u e. Examples o mu a ions:
• Al e Lea ning Ra e
• Iden i y (Keep aining)
• Inse Con olu ional Laye (Inse a andom loca ion)
• Al e s ide (al e he s ide o he con olu ional laye s)
• Fil e size (ho izon al o e ical a andom)
To ge a sense o a iabili y hey epea ed he expe imen 5 imes and hey concluded ha hei neu o-
e olu ional app oach is capable o cons uc ing la ge and accu a e ne wo ks.
In hese ypes o app oaches, he popula ion size is a c ucial decision, gi en he ac ha he sea ch
space is huge, bigge popula ions le he algo i hm explo e he space mo e ho oughly.
4.1.3 Hype pa ame e s E olu ion
Hype pa ame e s a e essen ial o he lea ning success o he ne wo ks. Al hough he esea ch has
p oposed some s anda d alues, hey a e o en depending on he ype o da a and a chi ec u e you a e
wo king wi h. Fo example, some s udies ha e shown ha a se o hype -pa ame e s wo ked well in
18
simple ne wo ks bu did no ha e he same e ec in mo e complex a chi ec u es [16]. E olu iona y
Algo i hms ha e been p o ed o be e ec i e o he sea ch o hese pa ame e s.
Publica ion [17] has been he mos inspi ing o his hesis. In hei wo k, hey p esen a amewo k o
op imizing he hype -pa ame e s o a deep con olu ional neu al ne wo k. In he e olu iona y algo i hm,
each hype -pa ame e is encoded in a single gene o each indi idual. To a oid ob aining undesi ed
alues o he genes hey applied a ange and a esolu ion. The pa ame e s used in he expe imen we e
he ollowing:
Figu e 4.2. EA pa ame e s o he expe imen .
Whe e 𝑃𝑐 and 𝑃𝑚 a e espec i ely he p obabili y o c osso e and p obabili y o mu a ion. The hype -
pa ame e s hey sea ched o , consis ed o he ke nel sizes o he con olu ional laye s and he numbe
o il e s. The i ness unc ion chosen is simply he e o on he es se o he da ase used compu ed
a e 4000 i e a ions. Al hough he amewo k is no oo complex i has been p o ed o s ill be a mo e
powe ul ool han andom sea ch.
4.1.4 GAN-Speci ic App oaches
Mos o he ci ed app oaches could ha e been applied o di e en ypes o neu al ne wo ks, in mo e
ecen s udies some pape s ocused especially on GANs. In Ma ch 2018, Wang e al. p oposed he i s
e olu iona y gene a i e ad e sa ial ne wo k app oach (E-GAN). The main pu pose o hei wo k was o
s abilize GAN aining and imp o e gene a i e pe o mance. In he o iginal GAN, he Jansen-Shannon
di e gence is used as he me ic, bu mo e me ics ha e been in oduced. They de ised an e olu iona y
algo i hm ha e ol es a popula ion o gene a o s {𝐺}. To ake ad an age o di e en me ics E-GAN
uses p e-de ined objec i e unc ions ha al e na i ely ain he gene a o ’s weigh s. The e olu iona y
19
s ep consis s o 3 s ages: a ia ion, e alua ion, and selec ion. One o he main con ibu ions o E-GAN is
he a ia ion s ep whe e asexual ep oduc ion along wi h mu a ions is used o he nex gene a ion’s
indi iduals.
Figu e 4.3. O iginal GAN compa ed wi h E-GAN app oach.
In E-GAN he gene a o quali y was compa ed wi h con en ional GANs using FID sco e and
demons a ed ha E-GAN achie es con incing gene a i e pe o mance and minimizes aining p oblems
in con en ional GANs.
The ea e , la e in 2018, Abdullah e al. p oposed a spa ial coe olu iona y app oach called Towa ds
Dis ibu ed Coe olu iona y GANs (Lipizzane ) [19]. In which, he esea che s in es iga e he usage o
coe olu iona y algo i hms wi h con en ional GAN aining. They aimed o b idge he gap be ween wo ks
o deep lea ning and e olu iona y compu ing communi ies owa ds a be e unde s anding o g adien -
based and g adien - ee GAN dynamics. In he spa ial coe olu ion, GAN indi iduals a e dis ibu ed on a
g id whe e he local in e ac ion o indi iduals go e ns he i ness e alua ion, selec ion, and mu a ion.
Ano he GAN-speci ic in e es ing app oach was s udied by Rozie e e al. [20]. They ocused on he noise
used o he gene a o . Ins ead o andomly gene a ing a la en ec o z, hey used an e olu iona y
algo i hm o selec he bes one wi hou changing any hing in he aining o he ne wo k. They showed
ha he da a p oduced by he la en noise gene a ed h ough hei algo i hm, is o be e quali y while
p ese ing he di e si y o he o iginal GAN.
26
The di e ences in he dis ibu ions can be obse ed in igu e 5.6. O he han he dis ibu ions ano he
aspec o he da a was highligh ed by he g aphs. Is possible o obse e ha some o he columns ha e
ew disc e e alues (column_9 and column_11). As he WGAN canno di ec ly wo k wi h disc e e da a
we need o ackle his p oblem c ea ing some cons ain s. The WGAN in i s anilla o m can only
gene a e da a ha ha e decimals, so o hese a iables o be disc e e, we will ound up o he whole
numbe . This kind o pos -p ocessing would ha e no been needed i some o he GAN s uc u es would
ha e been used. Mo e ad anced a chi ec u es a e implemen ed wi h he possibili y o choosing, be o e
he aining, cons ain s o he gene a ion o he da a. Also, some o he a chi ec u es we e buil exac ly
hinking o he ca ego ical a iables like he CTGAN [23].
5.3 EXPERIMENTAL SETUP
5.3.1 Equipmen
The compu e used o his expe imen had an N idia GeFo ce RTX 2070i and he un o he algo i hm
ook abou 12 hou s. We ha e also ied some cloud al e na i es like Google Colab, bu un o una ely,
due o he long p ocess, i was no possible o ake ad an age o his ool.
5.3.2 Neu al Ne wo ks A chi ec u es
Gi en he ac ha we ha e used a simple WGAN, he s uc u e was composed o only wo Neu al
Ne wo ks. (In mo e complex s uc u es i is possible ha he numbe o neu al ne wo ks inc eases. The e
a e o en neu al ne wo ks o he encoding and decoding o he da a gi ing be e pe o mances in e ms
o aining.) In bo h gene a o and c i ic, gi en he da a was abula we decided on using dense laye s.
This a chi ec u e was chosen gi en ha he da a aken in o conside a ion was abula . We chose no o
use oo many laye s o keep he s uc u e simple and o a oid o e i ing. In case we had mo e complex
da a o deal wi h, he decision would ha e been di e en . The s uc u es a e no composed o many
laye s o he same easons. He eunde is possible o see he wo ne wo ks u ilized.

27
Figu e 5.7. Gene a o used
As we can see om igu e 5.7 he Gene a o is composed o 4 dense laye s wi h an inpu o a ec o o
32 (noise) and an ou pu o a ec o o size 11 ha co esponds o he numbe o columns o he da ase .
As ac i a ion unc ion, all o hem a e using a ‘ elu’. The sizes o he a ious laye s can be changed and
adap ed o he da a you a e wo king wi h.
28
.
Figu e 5.8. Disc imina o used
Al hough i could seem deepe , he disc imina o has he same numbe o laye s.
The only di e ence wi h he gene a o is ha we ha e used D opou s o slow down he lea ning p ocess
o he disc imina o bea ing in mind ha i is usually ained mo e imes han he gene a o .
29
5.3.3 Pa ame e s o Gene ic Algo i hm
Mul iple pa ame e s ha e been ied ou , bu he inal choice o he bes ade-o be ween ime and
good quali y o he expe imen we e hese ones:
• Popula ion size: 250
• P obabili y o c osso e : 0.60
• P obabili y o mu a ion: 0.10
• Numbe o Gene a ions: 20
Also, he decision has been in luenced by he academy ha o en ad ises using a li le pe cen age o
mu a ion and a bigge one o he c osso e . In he beginning, we ha e ied di e en p obabili ies o
he ope a o s. In he case o la ge alues, he indi iduals esul in ha ing a la ge di e si y, al hough i
is bene icial o explo e he ull sea ch space, you o en end up ge ing u he away om he local
op ima. One way o implemen he bene i o he di e si y gi en by he high p obabili ies o he
ope a o s would possibly be o implemen a decay a e. In case a decay a e is se , he pe cen ages
would ge lowe and lowe he mo e he gene a ions pass by. This would lead he algo i hm o explo e
be e he sea ch space a he beginning and once i eaches di e en local op ima i would i e a e close
o hem. Ano he cons ain ha we ha e was he popula ion size and he numbe o gene a ions, as i
ge s compu a ionally mo e expensi e, we could ha e no inc emen ed hese alues ha migh ha e had
helped he sea ch wi h high ope a o s’ p obabili ies.
6. RESULTS
In his chap e , we a e going o illus a e he esul s ob ained wi h he bes pa ame e s ound om he
gene ic algo i hm. A i s , we a e going o illus a e how we decided o measu e he quali y o he
gene a ed da a and he easons why we ha e chosen hose me ics. We will hen p oceed on showing
he pe o mance ob ained by compa ing he syn he ic da a wi h he o iginal one. The pe o mances o
he bes -pa ame e s da ase a e compa ed wi h he s a e-o -a pa ame e s da ase .
6.1 METRICS USED
Unde s anding how good he quali y o he gene a ed da a is no i ial and un il now, al hough a lo o
me ics ha e been de eloped, he e a e no g ounded ways o assess he da a quan i a i ely and
30
quali a i ely [22]. Un o una ely, as o he i ness unc ion, he e is no a one- i -all solu ion. This means
ha is impossible ( o da e) o measu e he quali y o a da ase wi h jus one me ic. So, o de e mine
he quali y o ou da a we decided o use 3 me ics ha o e all co e he di e en aspec s o he da a.
O cou se, mo e me ics could ha e used, bu ha depends a lo on wha i is he end goal o he
syn hesisa ion. In ou case, we a e jus looking o wa d o compa ing how he di e en gene a ed
da ase s a e, in e ms o simila i y o he o iginal da a.
6.1.1 Basic S a is ics
The i s hing we hough o was ha he gene a ed da a should ha e had simila beha iou o he eal
da a, so we expec ed all he basic s a is ics o be close o each o he :
• Mean
• S anda d De ia ion
• Pe cen iles
• Maximum
• Minimum
The main eason why you wan o see how close he basic s a is ics a e, is o see i he gene a ed alues
a e kep in he expec ed anges. Some imes, du ing he gene a ion o syn he ic da a, you could end up
wi h alues ha do no e lec eali y. Fo example, a lo o ime you da a canno ha e alues unde
ze o o alues ha exceed a ce ain h eshold. F om looking a he maximum and he minimum you can
al eady ha e a g asp o he ange o you gene a ed da a. To a oid his, i is o en implemen ed he use
o cons ain s on he ange o he possible alues ha you a iable can assume (could be implemen ed
ei he on he pos -p ocessing as we did o on he algo i hms hemsel es). On he o he hand, he mean,
s anda d de ia ion, and pe cen iles gi e you a u he unde s anding o how he alues a e dis ibu ed
along wi h he da ase , hus i you end up wi h some kind o mode collapse. To calcula e how simila
hese basic s a is ics we e o he o iginal da a we decided on compa ing hem calcula ing he pai wise
Euclidean dis ance be ween he s a is ics. In o de o do so, wo da a ames we e c ea ed: one
con aining all he a o emen ioned s a is ics o he eal da a, and one o he gene a ed da a. These
da ase s we e apidly c ea ed u ilizing he unc ion ‘desc ibe’ o pandas. One d awback o his me ic is
ha he esul ing alue is no scaled. This implies ha he e is no a uni e sal way o in e p e ing he
esul s as i could be o some me ic in a speci ied ange (i.e., 0 o 1). In ou case, as we a e using i jus
31
o compa e di e en da ase s gi es us jus he gene al idea o how good one sample is in espec o
ano he . As i is a dis ance, he smalle he alue, he close he gene a ed da a o he o iginal one.
6.1.2 Dis ibu ions
Ano he aspec ha was wo h calcula ing was he dis ibu ion o each a iable. Al hough looking a he
s a is ics gi es us a gene al idea o how he da a is beha ing, some es s gi e us a deepe unde s anding
o wha conce ns he dis ibu ions. To see he simila i y be ween he dis ibu ions, we decided o use
he ollowing wo s a is ical es s: Chi-Squa ed and Kolgomo o -Smi no .
The Chi-Squa ed es is used o compa e dis ibu ions o disc e e o ca ego ical a iables. The es
e u ns he esul ing p- alue so ha a small alue indica es ha we can ejec he null hypo hesis and
sugges s ha he dis ibu ions a e di e en .
The Kolmogo o –Smi no es is used o compa e he dis ibu ions o con inuous a iables using he
empi ical Cumula i e Dis ibu ion Func ion (CDF). I e u ns 1 minus he KS Tes D s a is ic, which
indica es he maximum dis ance be ween he expec ed CDF and he obse ed CDF alues.
Bo h he es s ou pu a alue be ween 0 and 1: i he dis ibu ions a e iden ical, you ge he maximum
alue (1), in case hey a e comple ely di e en you would ge he minimum alue (0). As bo h o hem
ha e he same ange, we decided o combine hem by calcula ing hei mean. As opposed o he la e
me ic, his would be use ul o gi e you in o ma ion on he da ase wi hou ha ing o compa e i o
o he s.
6.1.3 Co ela ion o Co a iances
Looking a how he da a is dis ibu ed and i i has he igh alues does no mean ha he da a
gene a ed is o high quali y. As you a e o en gene a ing mo e han one column you expec you
gene a ed da a o keep he same ela ions be ween he a iables o he o iginal da a. This is needed
because he gene a ed da a is o en used o sol e some machine lea ning asks, and in o de o do so
you need o keep all he possible in o ma ion. To unde s and he ela ionship ha exis s be ween he
a iables we ha e decided o look a he co a iance ma ix. As he co a iance is he join a iabili y o
wo andom a iables, he ma ix would gi e us all he in o we need. To calcula e he simila i y be ween
hese ma ixes, once again, he pai wise Euclidean dis ance has been used.

32
6.2 FINAL RESULT
The bes pa ame e s ound om ou algo i hm o he inal indi idual we e he ollowing:
Figu e 6.1. Bes Pa ame e s Found
This indi idual eached a i ness unc ion o 0.7854 meaning ha he dis ibu ions o he eal and
syn he ic samples we e e y close o each o he . F om he pa ame e s ound we can see ha he ones
ha a e he a hes om he s a e-o -a ones a e lea ning a e and be a 2. This means ha a e possibly
he ones ha in luence mo e he esul o he WGANs. The o he pa ame e s, al hough no equal, a e
close o he s anda d ones. One o he easons why hey a e so close could be he ac ha we pu some
cons ain s on he possible alues hey could ha e aken, bu a he same ime, I highly doub ha going
oo a om he s anda d ones would ha e led o op imal esul s.
F om he inal compa ison o he eal da a wi h he syn he ic one gene a ed by he bes indi idual, we
go hese esul s.
Figu e 6.2. Resul s o he expe imen
As we can see om he able, he pa ame e s ound by he gene ic algo i hm esul ed in ha ing be e
da a. The dis ibu ions a e mo e simila and he dis ances be ween s a is ics and co a iances a e
smalle . The esul s a e no ou s anding in e ms o di e ence, o example, we can see ha he
dis ances o he co ela ion a e close o each o he . As we al eady discussed, i is no s ange o ha e
alues ha a e no so dis an , as he pa ame e s a e no oo di e en . Ano he eason why i migh ha e
no been an excellen esul is ha Neu al ne wo k models a e s ochas ic. This means ha , gi en he
same model con igu a ion and he same aining da ase , a di e en in e nal se o weigh s will esul in
each ime he model is ained ha will in u n ha e a di e en pe o mance. The da a has been
gene a ed using he bes pa ame e s bu no he exac model used du ing he esea ch o hose.
33
7. CONCLUSIONS AND FUTURE WORK
In his hesis, we ha e shown ha al hough he quali y o he bes pa ame e ’s da a did no su pass by
a he quali y o s anda d pa ame e s, he p oposed gene ic app oach is be e han he no mal andom
sea ch. We ha e o ake in o accoun ha he e has no been done oo much wo k on he p e-p ocessing
o he da a, maybe app oxima ing he da a o a gaussian-like dis ibu ion would s abilize he aining o
he ne wo k and le ocus he ne wo ks on he hype -pa ame e s hemsel es. Fu he mo e, we ha e
p oposed a compa ison me hod ha p o ides an o e all iew o he quali y o he gene a ed samples.
Fu u e imp o emen s can be achie ed in a couple o di e en a eas o he p ojec . Fi s , we did no ha e
enough compu a ional powe o un big popula ions, and as we know ha he sea ch space is immense,
ha ing mo e indi iduals in he popula ion would al eady b ing be e esul s. The gene ic ope a o s ha
ha e been used we e basic, so as we saw om he li e a u e e iew, mo e complex ope a o s would
b ing be e indi iduals. Ano he imp o emen ha can be done is on he i ness unc ion; as he e is
no a uni e sal me ic o he quali y o he p oduced da a, using di e en i ness unc ions could b ing
be e esul s, as i has been done on [18]. O he han wo king on he gene ic algo i hm i sel some mo e
wo k can also be done on he GAN a chi ec u e ha ing some mo e s able ne wo ks could b ing bene i s
o he calcula ion o he i ness unc ion.
34
REFERENCES
[1] Ian J. Good ellow, J. P.-A. F. (2014), Gene a i e Ad e sa ial Ne wo ks,
h ps://a xi .o g/abs/1406.2661
[2] A. Agga wala, M. Mi al, G. Ba ineni, (2021), ‘Gene a i e ad e sa ial ne wo k: An
o e iew o heo y and applica ions’
[3] L. Huang, A. D. Joseph, B. Nelson, B. I. Rubins ein, and J. Tyga , ‘Ad e sa ial machine lea ning’, ACM
Wo kshop on Secu i y and A i icial In elligence, pp. 43–58, 2011.
[4] J. Gui, Z. Sun, Y. Wen, D. Tao, J. Ye, (2020), ‘A e iew on Gene a i e Ad e sa ial Ne wo ks: Algo i hms,
Theo y, and Applica ions’
[5] M. A jo sky, S. Chin ala, L. Bo ou, (2017), ‘Wasse s ein GAN’
[6] I. Gul ajani, F. Ahmed, M. A jo sky, V Dumoulin, A. Cou ille, (2017), ‘Imp o ed T aining o
Wasse s ein GANs’
[7] Wikipedia, ‘Lipschi z con inui y’, en.wikipedia.o g/wiki/Lipschi z_con inui y
[8] A. E. Eiben, J. Smi h, (2015), ‘F om e olu iona y compu a ion o he e olu ion o hings’, Na u e
[9] J. H. Holland. (1975), ‘Adap a ion in Na u al and A i icial Sys ems’, Uni e si y o Michigan
[10] K. Tang, X. Yuan, P. Liu, J. Yang, (2011), ‘Linea E olu iona y Algo i hm’, In echOpen
[11] P.A. Cas illo, M.G. A enas, J.J. Cas illo-Valdi ieso, J.J. Me elo, A. P ie o, G. Rome o, (2003),
‘A i icial Neu al Ne wo ks Design using E olu iona y Algo i hms’
[12] Y. Chang, J. Lin, J. Shieh, M. F. Abbod, (2012), ‘Op imiza ion he Ini ial Weigh s o A i icial
Neu al Ne wo ks ia Gene ic Algo i hm Applied o Hip Bone F ac u e P edic ion’, Hindawi Publishing
Co po a ion
[13] D. O i e, G. So osal, C. E. Bo ges, C. Ma in, A. Alonso-Vica io, (2014), ‘E olu iona y algo i hms
o hype pa ame e uning on neu al ne wo ks models’
[14] K. O. S anley, R. Miikkulainen, (2002), ‘E ol ing Neu al Ne wo ks h ough Augmen ing
Topologies’, Massachuse s Ins i u e o Technology
[15] E. Real, S. Moo e, A. Selle, S.Saxena, Y. L. Suema su, Jie Tan, Quoc V. Le, A. Ku akin, (2017),
‘La ge-Scale E olu ion o Image Classi ie s’, In e na ional Con e ence on Machine Lea ning, Sydney
35
[16] T.M. B euel, (2015), ‘The e ec s o hype pa ame e s on SGD aining o neu al ne wo ks’
[17] S. R. Young, D. C. Rose, T. P. Ka nowski, S.H. Lim, R. M. Pa on, (2015), ‘Op imizing Deep
Lea ning Hype -Pa ame e s Th ough an E olu iona y Algo i hm’
[18] C.Wangy, C. Xuz, X. Yao, D. Taoz, (2018), ‘E olu iona y Gene a i e Ad e sa ial Ne wo ks’
[19] T. Schmiedlechne , I. Ng Zhi Yong, A. Al-Dujaili, E. Hembe g, U. O’Reilly, (2018), ‘Lipizzane : A
Sys em Tha Scales Robus Gene a i e Ad e sa ial Ne wo k T aining’
[20] B. Rozie e, F. Tey aud, V. Hosu, H. Lin, J. Rapin, M. Zameshina, and O. Tey aud, (2020), ‘E olGAN:
E olu iona y Gene a i e Ad e sa ial Ne wo ks’, a Xi :2009.13311 1
[21] h ps://www.kaggle.com/c/Gi eMeSomeC edi /
[22] A. Bo ji, (2018), ‘P os and Cons o GAN E alua ion Measu e’
[23] L. Xu, M. Skola idou, A. Cues a-In an e, K. Vee amachaneni, (2019) ‘Modeling Tabula Da a using
Condi ional GAN’, a xi :1907.00503