GAN Hyperparameters search through Genetic Algorithm

i GAN Hype pa ame e s sea ch h ough Gene ic Algo i hm P ojec Wo k p esen ed as pa ial equi emen o ob aining he Mas e ’s deg ee in Ad anced Analy ics Umbe o Tamma o ii NOVA In o ma ion Managemen School Ins i u o Supe io de Es a ís ica e Ges ão de In o mação Uni e sidade No a de Lisboa GAN hype pa ame e s sea ch h ough gene ic algo i hm Au ho : Umbe o Tamma o (M20190806) No a In o ma ion Managemen School Mas e ’s in ad anced Analy ics Supe iso : P o . Mau o Cas elli No a In o ma ion Managemen School No embe 2021 iii DEDICATION Dedica o a Mamma, Papà e Fede ica. Nonos an e la dis anza, il os o suppo o e a e o si a sen i e o e come semp e. G azie di u o. i INDEX 1. INTRODUCTION ................................................................................................................................ 2 2. GENERATIVE ADVERSARIAL NETWORK ............................................................................................ 4 2.1 S uc u e o GAN ............................................................................................................................ 5 2.2 T aining GANs ................................................................................................................................. 6 2.3. Common P oblems o GANs ......................................................................................................... 7 2.3.1 Mode Collapse ........................................................................................................................ 7 2.3.2 Failu e o Con e ge ................................................................................................................ 8 2.3.3 Vanishing G adien s ............................................................................................................... 8 2.4 Wasse s ein GAN ........................................................................................................................... 9 2.5 Wasse s ein GAN wi h G adien Penal y ....................................................................................... 9 3. E olu iona y Compu ing ................................................................................................................ 10 3.1 Gene ic Algo i hm ........................................................................................................................ 11 3.2 Gene ic ope a o s ........................................................................................................................ 12 3.2.1 C osso e ope a o ............................................................................................................... 12 3.2.2 Mu a ion ope a o ............................................................................................................... 13 3.2.3 Selec ion ope a o ................................................................................................................ 13 3.3 Main Ad an ages ......................................................................................................................... 14 4. Rela ed Wo k ................................................................................................................................. 15 4.1 App oaches esea ched ............................................................................................................... 15 4.1.1 E ol ing connec ion weigh s ................................................................................................ 15 4.1.2 Ne wo k A chi ec u es ......................................................................................................... 16 4.1.3 Hype pa ame e s E olu ion ................................................................................................. 17 4.1.4 GAN-Speci ic App oaches ..................................................................................................... 18 5 Me hodology ................................................................................................................................... 20 5.1 P oposed App oach ...................................................................................................................... 20 5.1.1 Hype -Pa ame e s Encoding ................................................................................................ 20 5.1.2 Gene ic Ope a o s ................................................................................................................ 21 5.1.2 Fi ness Func ion.................................................................................................................... 22 5.1.3 Algo i hm s eps .................................................................................................................... 23 5.2 Da a .............................................................................................................................................. 24 5.3 Expe imen al Se up ...................................................................................................................... 26 5.3.1 Equipmen ............................................................................................................................ 26 5.3.2 Neu al Ne wo ks A chi ec u es ............................................................................................ 26 5.3.3 Pa ame e s o Gene ic Algo i hm ........................................................................................ 29 6. Resul s ............................................................................................................................................ 29 6.1 Me ics used ................................................................................................................................. 29 6.1.1 Basic S a is ics ...................................................................................................................... 30 6.1.2 Dis ibu ions ......................................................................................................................... 31 6.1.3 Co ela ion ............................................................................................................................ 31 6.2 Final esul .................................................................................................................................... 32 7. Conclusions and Fu u e Wo k ........................................................................................................ 33 i LIST OF FIGURES 2.1. Human aces gene a ed by GAN ........................................................................................................ 4 2.2. Schema ic ep esen a ion o a GAN .................................................................................................... 5 2.3. P ocess o lea ning o a GAN ............................................................................................................... 6 2.4. P ocess o lea ning o a GAN ............................................................................................................... 7 3.1. Pheno ype-Geno ype o gene ic algo i hm ...................................................................................... 10 3.2. Flow cha o gene ic algo i hm ........................................................................................................ 11 3.3. One poin c osso e .......................................................................................................................... 12 3.4. Bi lip mu a ion................................................................................................................................. 13 3.5. Roule e wheel selec ion ................................................................................................................... 13 4.1. Ma ix encoding o a solu ion ........................................................................................................... 15 4.2. EA pa ame e s ................................................................................................................................... 17 4.3. O iginal GAN compa ed wi h E-GAN ................................................................................................. 18 5.1. Example o indi idual ........................................................................................................................ 20 5.2. Chosen gene ic ope a o s ................................................................................................................. 21 5.3. Values used o mu a ion .................................................................................................................. 21 5.4. Scheme o he algo i hm s eps ......................................................................................................... 23 5.5. Basic s a is ics able .......................................................................................................................... 24 5.6. Dis ibu ion o a iables .................................................................................................................... 25 5.7. Gene a o .......................................................................................................................................... 27 5.8. Disc imina o ..................................................................................................................................... 28 6.1. Bes pa ame e s ound ...................................................................................................................... 32 6.2. Resul s o expe imen ....................................................................................................................... 32 1 ABSTRACT Recen de elopmen s in Deep Lea ning a e ema kable when i comes o gene a i e models. The main eason o such p og ess is because o Gene a i e Ad e sa ial Ne wo ks (GANs) [1]. In oduced in a pape by Ian Good ellow in 2014 GANs a e machine lea ning models ha a e made o wo neu al ne wo ks: a Gene a o and a Disc imina o . These wo compe e amongs each o he o gene a e new, syn he ic ins ances o da a ha esemble he eal one. Despi e hei g ea po en ial, he e a e p esen challenges in hei aining, which include aining ins abili y, mode collapse, and anishing g adien . A lo o esea ch has been done on how o o e come hese challenges, howe e , he e was no signi ican p oo ound on whe he mode n echniques consis en ly ou pe o m anilla GAN. The pe o mances o GANs a e also highly dependen on he da ase hey a e ained on. One o he main challenges is ela ed o he sea ch o hype pa ame e s. In his hesis, we y o o e come his challenge by applying an e olu iona y algo i hm o sea ch o he bes hype pa ame e s o a WGAN. We use Kullback-Leible di e gence o calcula e he i ness o he indi iduals, and in he end, we selec he bes se o pa ame e s gene a ed by he e olu iona y algo i hm. The pa ame e s o he bes -selec ed indi iduals a e main ained h oughou he gene a ions. We compa e ou app oach wi h he s anda d hype pa ame e s gi en by he s a e-o -a . 2 1. INTRODUCTION A i icial neu al ne wo ks ha e success ully been used in a la ge numbe o applica ions, in ecen yea s hei g ow h has led o ex ao dina y indings. Among hem, he Gene ic Ad e sa ial Ne wo ks a e hose ha mos cap u ed he a en ion o he public. GANs ha e been used o a a ie y o gene a i e asks, a ying om images o ime-se ies da a leading o imp essi e esul s. All hese algo i hms, un o una ely, c ea e se e al p oblems since i is necessa y o es ablish se e al pa ame e s o hei ne wo k design. Fo ins ance, in he case o mul ilaye pe cep ons, i is necessa y o es ablish lea ning a e, ini ial weigh s, numbe o hidden laye s, and so on. The sea ch o he igh pa ame e s has become one o he main challenges o ge ing hese ools o wo k. These p oblems usually a e app oached using me hods as andom sea ch and g id sea ch. Al hough hey usually wo k ine, hey a e ime-consuming and o en do no lead o he bes esul s. Among he possible al e na i es o ackle hese p oblems, he e is one ha is leading o g ea esul s in li le ime: he e olu iona y algo i hms (EA). Inspi ed by biological e olu ion, EAs imi a e e olu ion ia ep oduc ion, mu a ion na u al selec ion, and su i al o he i es . They sea ch mo e o less andomly in he space o solu ions, paying special a en ion o hose zones whe e he e alua ion unc ion is a maximum leading o as e and be e esul s compa ed o adi ional me hods. In his hesis we p esen a possible way o apply EAs o iden i y he bes se o hype pa ame e s o a WGAN-GP, ying o ou pe o m he s anda d se . Ou wo k has been inspi ed a lo by he esea ch done by academics in ecen yea s. A lo o echniques ha e been s udied on how o apply a gene ic algo i hm o he pa ame e s esea ch o neu al ne wo ks, some ocused mo e on he s uc u e o he a chi ec u e and o he s on he uning o he hype - pa ame e s. We ha e decided o ocus mainly on he sea ch o hype -pa ame e s. Using a p ede ined ne wo k, we wan ed o see how much he esul s would change jus wi h gene ic esea ch o he bes se o alues. Fundamen al o ou algo i hm, has been he esea ch published on [17]. Al hough ou app oach has been simila o his one, we ha e ocused on a di e en se o pa ame e s o une. Also, mos o he esea ch is ocusing on using and imp o ing GAN using con olu ional laye s. These a chi ec u es happen o be he mos used because he images a e he mos common da a o wo k wi h when dealing wi h GANs. In ou case, we wan ed o check i he esea ch would apply also o abula da a, ha al hough can seem easie o wo k wi h, o en shows way mo e complica ed dis ibu ions. The hesis is o ganized in he ollowing way: 3 In sec ion (2) we discuss he basic concep s o he GANs, gi ing he backg ound s udy conduc ed o he hesis. In sec ion (3) we explain he idea behind E olu iona y Algo i hms and how hey a e cons uc ed, p esen ing he basic ope a o s wi h which hey a e buil . In sec ion (4) we p esen a e iew o he a ious app oaches ound in he bibliog aphy and he di e en aspec s ha a e possible o e ol e in neu al ne wo ks. In sec ion (5) we p esen he wo k p ojec ha has been done, in oducing ou p oposed app oach. We u he desc ibe he expe imen al se ups and he de ailed esul s o he p oposed me hods. 10 would lead o di icul ies in he in e ac ions be ween cos unc ion and weigh cons ain . The solu ion o hese p oblems has been in oduced wi h an al e na i e o weigh clipping: g adien penal y. A di e en iable unc ion, o be conside ed 1-Lipschi z mus ha e i s g adien s wi h he no m a mos 1 e e ywhe e. To achie e i hey cons ained di ec ly he g adien no m o he c i ic’s ou pu , so he new objec i e unc ion p oposed is The o mula is composed o wo pa s, he i s one being he o iginal c i ic loss and he second one being he g adien penal y. This me hod has been p o ed o imp o e he aining speed and sample quali y compa ed o weigh clipping. 3. EVOLUTIONARY COMPUTING Since he in en ion o he compu e , he idea o e olu ion as a me apho o p oblem-sol ing has been in oduced. In he 1970s and 1980s, his idea has been de eloped in o di e en algo i hmic implemen a ions [8]. E olu iona y Compu ing has been used in he scope o op imiza ion p oblems. These s ochas ic op imiza ion echniques a e based on na u al e olu ion and he i es s a egy ound in biological o ganisms. They ha e been success ully applied o sol e complex op imiza ion p oblems ac oss di e en a eas. Thei abili y o be sui able o such a a ie y o di e en p oblems is by cause o he ac ha hey can be hough o as wo king on wo le els. On a highe le el, we ha e he pheno ype ha ep esen s a collec ion o aspec s (like eye colo ) and ha is wha we change in his algo i hm. On a lowe le el, in o de o wo k wi h his kind o abs ac concep , we ha e he geno ype, which is he ep esen a ion o such abs ac aspec s in a o m ha can be manipula ed. Figu e 3.1. Pheno ype-Geno ype mapping 11 This wo-le el ep esen a ion makes i possible o adap a small g oup o possible geno ypes o many kinds o pheno ypes. F om his ield, a lo o algo i hms ha e been a ising, such as gene ic algo i hms, e olu iona y s a egy, gene ic p og amming, and so on. 3.1 GENETIC ALGORITHM Gene ic algo i hm (GA) has been i s ly p esen ed by J. Holland in 1975 [9]. GA is an algo i hm o mimic he na u al e olu ion o species, i is widely applied o op imiza ion p oblems. The basic algo i hm is o en called a simple gene ic algo i hm (SGA) and om his base, a lo o imp o ed algo i hms ha e been in oduced. Figu e 3.2. Flow cha o a Gene ic Algo i hm In igu e 3.2 is possible o see he classical p ocess o he gene ic algo i hm. I s a s wi h a popula ion o indi iduals (ch omosomes) andomly ini ialized. Then by applying di e en gene ic ope a o s i ies 12 o e ol e hese indi iduals o i e ones, in o de o make hem be e han he p e ious ones in e ms o a ce ain measu e called i ness unc ion. 3.2 GENETIC OPERATORS The gene ic ope a o s ha a e mean o imp o e he indi iduals a e h ee: selec ion, c osso e , and mu a ion. 3.2.1 C osso e ope a o The c osso e ope a o is he main p ocess o gene a e new indi iduals, i consis s o exchanging in o ma ion be ween wo di e en indi iduals. The c osso e is inspi ed by ep oduc ion and he biological c osso e . Figu e 3.3. One-poin C osso e The easies one is he one-poin c osso e , which consis s o selec ing a andom poin in he ch omosomes and hen swapping he ails o he wo pa en s o p oduce o sp ing. The e a e mul iple a ia ions o his ope a o , such as mul i-poin c osso e whe e you selec mo e han one poin o uni o m c osso e whe e each gene is andomly selec ed o be swapped. In o de o decide which one is he mos i ed o he p oblem we need o look i he e a e any cons ain s and how he me hod selec ed would a ec he indi iduals. The c osso e is used o he explo a ion o he sea ch space (space o all possible solu ions) because i makes mo e signi ican changes o he ch omosomes. 13 3.2.2 Mu a ion ope a o Inspi ed by biology, he mu a ion is an al e a ion o he genes in a ch omosome. The main pu pose o he mu a ion is o in oduce di e si y in he popula ion, in o de o a oid he con e gence o all simila indi iduals. A classic example o mu a ion is he bi - lip mu a ion. In a bina y encoded ch omosome, one o he andom genes is selec ed and hey a e lipped. Figu e 3.4. Bi Flip Mu a ion As o c osso e , no all he mu a ions a e well- i ed o e e y p oblem. The mu a ion is conside ed essen ial o he con e gence o he gene ic algo i hm o i s abili y o a oid local minima by p e en ing simila ch omosomes. Con a y o c osso e , he mu a ion is used o he exploi a ion o he ch omosomes because i emains “close ” o he o iginal pa en . 3.2.3 Selec ion ope a o The selec ion is he s age o he GA whe e he ch omosomes o be passed on o he nex gene a ion a e chosen. This is one c ucial s ep o d i e he gene a ions in o be e solu ions. A good balance be ween popula ion di e si y and good ch omosomes mus be ound in o de o a oid p ema u e con e gence. In case o p ema u e con e gence, he popula ion, con e ging oo ea ly, would lead he algo i hm o ge s uck in o local op ima esul ing in he ine iciency o he sea ch o he bes indi idual. One o he mos popula ways o selec ing is basing he choice on he i ness o he indi iduals. The i ness o an indi idual is de e mined by a unc ion ha akes as inpu he candida e solu ion and p oduces as ou pu 14 he sco e o de e mine how good o i is he solu ion. Usually, he i ness unc ion ma ches he objec i e unc ion o he p oblem. One common i ness p opo iona e selec ion is he Roule e Wheel. Figu e 3.5. Roule e Wheel Selec ion The main idea behind he oule e wheel is ha be e indi iduals ge a highe chance o being chosen, and hese chances a e p opo ional o i ness. Once de e mined he pe cen ages o he indi iduals (po ions o he wheel) we spin he wheel n imes o selec n indi iduals. The selec ed indi iduals will be pa o he nex gene a ion o indi iduals. As pe he o he ope a ions ha a e di e en ways o implemen he selec ion o he indi iduals. 3.3 MAIN ADVANTAGES The majo ad an ages o he E olu iona y Algo i hms compa ed wi h adi ional op imiza ion echniques a e [10]: • EAs end o escape mo e easily om he local op imum because o he andomness in oduced by he ope a o s. • EAs do no equi e p io knowledge on he speci ic domain e en hough, i a ailable, i could be exploi ed. • EAs a e concep ually simple and ela i ely easy o implemen . • EAs do no equi e objec i e unc ion o be con inuous and can be used in algeb aic o m. The e a e also some p oblems ega ding EA, such as poo pe o mance in handling cons ain s, long compu a ion ime, and high compu a ional complexi y when he solu ion space is ha d o explo e. 15 4. RELATED WORK In he la es yea s, gi en he di icul y o inding he bes pa ame e s o neu al ne wo ks, mul iple expe imen s using gene ic algo i hms ha e been conduc ed. The app oaches o he usage o gene ic algo i hms ha e been a ious. Th ee main app oaches can be ound: e olu ion o connec ion weigh s ha is a global app oach o he ini ial weigh s and biases o ob ain a as con e gence; ne wo k a chi ec u e e olu ion ha implies an adap a ion o he ne wo k opology o es ablish i s lea ning and gene aliza ion abili y; lea ning pa ame e s and ule e olu ion ha is he adap a ion o he hype pa ame e s and lea ning ules [11]. 4.1 APPROACHES RESEARCHED 4.1.1 E ol ing connec ion weigh s The ini ial weigh s o a neu al ne wo k can highly in luence he speed o con e gence in backp opaga ion, since, depending on he poin o he sea ch space om which i has s a ed, be e o wo se solu ions can be ob ained. Weigh e olu ion in ol es he se ing ini ial alues and he way he EA changes hem. The simples me hod o all is based on making a andom ini ializa ion, o he me hods equi e s a is ical analysis o he aining da a, which makes he p ocess mo e complica ed bu a he same ime mo e powe ul. Chang e al. [12] esea ched he op imiza ion o he ini ial weigh s applied o hip bone ac u e p edic ion. Thei s a egy was o de ine o each a i icial ne wo k he aining da a and le he GA ind he op imum ini ial weigh s. The encoding o he weigh and biases consis ed o a ep esen a ion o one digi . Gi en he ac ha he ini ial popula ion’s ange would a ec he sea ch e iciency, hey p oposed o na ow he ange be ween -0.5 and 0.5. Fo he e alua ion o he ch omosomes, he mean squa e e o has been used, ep esen ing how he solu ion was i o he p oblem. In o de o a oid choosing bad solu ions hey pu a h eshold on he alida ion se ’s e o ; in case he e o was highe han wha was expec ed he ch omosome would no be chosen. 16 The s opping c i e ia consis ed in s opping he algo i hms a e 100 epochs. The s udy ound ou ha , despi e all he di icul ies and limi a ions ound in he design o he algo i hm, using a simple GA has been p o ed o be e ec i e o imp o ing he accu acy o neu al ne wo ks. Figu e 4.1. Ma ix encoding o a solu ion. The same indings we e ob ained by O i e e al. [13]. Thei s udy was also conduc ed on he ini ializa ion o he weigh s. Di e en om he la e was he codi ica ion o a solu ion, ma ices we e used ins ead o simple digi s. (Figu e 4.1). In he pape di e en ypes o c osso e ha e been used and has been p o en ha he e a e signi ican di e ences be ween he esul s using di e en c osso e ope a o s. O he han hese li le di e ences he esul s we e he same: uning he ini ial weigh s ia gene ic algo i hm has been s a is ically demons a ed ha led o signi ican ly be e pe o mances. 4.1.2 Ne wo k A chi ec u es Un il now, he a chi ec u es ha e been designed manually by expe s wi h enough expe ience. The ne wo k s uc u e is e y impo an , a poo -designed a chi ec u e could lead o o e i o lead o di icul ies in lea ning. The design o a op imal ne wo k a chi ec u e can be o mula ed as a sea ch p oblem in he a chi ec u e space, whe e each poin ep esen s a possible ne wo k a chi ec u e [11]. 17 One o he mos impo an algo i hms o he e olu ion o he opog aphy o he ne wo ks has been he NEAT. In oduced by S anley & Mikkulainen, [14] i showed he ad an ages o e ol ing he a chi ec u e simul aneously wi h he weigh s. The NEAT algo i hm uses a di ec encoding: e e y node and e e y connec ion is s o ed in he DNA. F om his base, a chi ec u e esea che s ha e buil me hods o de elop e olu iona y algo i hms o a chi ec u e design. Es eban e al. [15] p oposed an e olu iona y p ocess o disco e he a chi ec u es au oma ically. In hei me hod, each indi idual consis s in a ained a chi ec u e and du ing each e olu iona y s ep, wo indi iduals a andom a e chosen and compa ed by hei i nesses. Gi en he high compu a ional expense, hey de eloped a kind o g id compu ing solu ion, making he compe i ions in pa allel, he compu e s in he a chi ec u e a e called wo ke s. This kind o scheme uses epea ed pai wise compe i ions, which makes i an example o a ou namen selec ion ope a o , also i p e en s wo ke s om idling when hey inish ea ly. The encoding used is sligh ly di e en han o he s: each neu al ne wo k a chi ec u e is encoded as a g aph. The e ex in he g aph ep esen s a ank-3 enso (because o images, hey always ha e h ee dimensions, he i s wo o he spa ial coo dina es and he hi d one o he numbe o channels) o ac i a ions. The mu a ions hey chose we e due o he simila i y o he ac ions ha a human designe may ake when imp o ing an a chi ec u e. Examples o mu a ions: • Al e Lea ning Ra e • Iden i y (Keep aining) • Inse Con olu ional Laye (Inse a andom loca ion) • Al e s ide (al e he s ide o he con olu ional laye s) • Fil e size (ho izon al o e ical a andom) To ge a sense o a iabili y hey epea ed he expe imen 5 imes and hey concluded ha hei neu o- e olu ional app oach is capable o cons uc ing la ge and accu a e ne wo ks. In hese ypes o app oaches, he popula ion size is a c ucial decision, gi en he ac ha he sea ch space is huge, bigge popula ions le he algo i hm explo e he space mo e ho oughly. 4.1.3 Hype pa ame e s E olu ion Hype pa ame e s a e essen ial o he lea ning success o he ne wo ks. Al hough he esea ch has p oposed some s anda d alues, hey a e o en depending on he ype o da a and a chi ec u e you a e wo king wi h. Fo example, some s udies ha e shown ha a se o hype -pa ame e s wo ked well in 18 simple ne wo ks bu did no ha e he same e ec in mo e complex a chi ec u es [16]. E olu iona y Algo i hms ha e been p o ed o be e ec i e o he sea ch o hese pa ame e s. Publica ion [17] has been he mos inspi ing o his hesis. In hei wo k, hey p esen a amewo k o op imizing he hype -pa ame e s o a deep con olu ional neu al ne wo k. In he e olu iona y algo i hm, each hype -pa ame e is encoded in a single gene o each indi idual. To a oid ob aining undesi ed alues o he genes hey applied a ange and a esolu ion. The pa ame e s used in he expe imen we e he ollowing: Figu e 4.2. EA pa ame e s o he expe imen . Whe e 𝑃𝑐 and 𝑃𝑚 a e espec i ely he p obabili y o c osso e and p obabili y o mu a ion. The hype - pa ame e s hey sea ched o , consis ed o he ke nel sizes o he con olu ional laye s and he numbe o il e s. The i ness unc ion chosen is simply he e o on he es se o he da ase used compu ed a e 4000 i e a ions. Al hough he amewo k is no oo complex i has been p o ed o s ill be a mo e powe ul ool han andom sea ch. 4.1.4 GAN-Speci ic App oaches Mos o he ci ed app oaches could ha e been applied o di e en ypes o neu al ne wo ks, in mo e ecen s udies some pape s ocused especially on GANs. In Ma ch 2018, Wang e al. p oposed he i s e olu iona y gene a i e ad e sa ial ne wo k app oach (E-GAN). The main pu pose o hei wo k was o s abilize GAN aining and imp o e gene a i e pe o mance. In he o iginal GAN, he Jansen-Shannon di e gence is used as he me ic, bu mo e me ics ha e been in oduced. They de ised an e olu iona y algo i hm ha e ol es a popula ion o gene a o s {𝐺}. To ake ad an age o di e en me ics E-GAN uses p e-de ined objec i e unc ions ha al e na i ely ain he gene a o ’s weigh s. The e olu iona y 19 s ep consis s o 3 s ages: a ia ion, e alua ion, and selec ion. One o he main con ibu ions o E-GAN is he a ia ion s ep whe e asexual ep oduc ion along wi h mu a ions is used o he nex gene a ion’s indi iduals. Figu e 4.3. O iginal GAN compa ed wi h E-GAN app oach. In E-GAN he gene a o quali y was compa ed wi h con en ional GANs using FID sco e and demons a ed ha E-GAN achie es con incing gene a i e pe o mance and minimizes aining p oblems in con en ional GANs. The ea e , la e in 2018, Abdullah e al. p oposed a spa ial coe olu iona y app oach called Towa ds Dis ibu ed Coe olu iona y GANs (Lipizzane ) [19]. In which, he esea che s in es iga e he usage o coe olu iona y algo i hms wi h con en ional GAN aining. They aimed o b idge he gap be ween wo ks o deep lea ning and e olu iona y compu ing communi ies owa ds a be e unde s anding o g adien - based and g adien - ee GAN dynamics. In he spa ial coe olu ion, GAN indi iduals a e dis ibu ed on a g id whe e he local in e ac ion o indi iduals go e ns he i ness e alua ion, selec ion, and mu a ion. Ano he GAN-speci ic in e es ing app oach was s udied by Rozie e e al. [20]. They ocused on he noise used o he gene a o . Ins ead o andomly gene a ing a la en ec o z, hey used an e olu iona y algo i hm o selec he bes one wi hou changing any hing in he aining o he ne wo k. They showed ha he da a p oduced by he la en noise gene a ed h ough hei algo i hm, is o be e quali y while p ese ing he di e si y o he o iginal GAN. 26 The di e ences in he dis ibu ions can be obse ed in igu e 5.6. O he han he dis ibu ions ano he aspec o he da a was highligh ed by he g aphs. Is possible o obse e ha some o he columns ha e ew disc e e alues (column_9 and column_11). As he WGAN canno di ec ly wo k wi h disc e e da a we need o ackle his p oblem c ea ing some cons ain s. The WGAN in i s anilla o m can only gene a e da a ha ha e decimals, so o hese a iables o be disc e e, we will ound up o he whole numbe . This kind o pos -p ocessing would ha e no been needed i some o he GAN s uc u es would ha e been used. Mo e ad anced a chi ec u es a e implemen ed wi h he possibili y o choosing, be o e he aining, cons ain s o he gene a ion o he da a. Also, some o he a chi ec u es we e buil exac ly hinking o he ca ego ical a iables like he CTGAN [23]. 5.3 EXPERIMENTAL SETUP 5.3.1 Equipmen The compu e used o his expe imen had an N idia GeFo ce RTX 2070i and he un o he algo i hm ook abou 12 hou s. We ha e also ied some cloud al e na i es like Google Colab, bu un o una ely, due o he long p ocess, i was no possible o ake ad an age o his ool. 5.3.2 Neu al Ne wo ks A chi ec u es Gi en he ac ha we ha e used a simple WGAN, he s uc u e was composed o only wo Neu al Ne wo ks. (In mo e complex s uc u es i is possible ha he numbe o neu al ne wo ks inc eases. The e a e o en neu al ne wo ks o he encoding and decoding o he da a gi ing be e pe o mances in e ms o aining.) In bo h gene a o and c i ic, gi en he da a was abula we decided on using dense laye s. This a chi ec u e was chosen gi en ha he da a aken in o conside a ion was abula . We chose no o use oo many laye s o keep he s uc u e simple and o a oid o e i ing. In case we had mo e complex da a o deal wi h, he decision would ha e been di e en . The s uc u es a e no composed o many laye s o he same easons. He eunde is possible o see he wo ne wo ks u ilized. 27 Figu e 5.7. Gene a o used As we can see om igu e 5.7 he Gene a o is composed o 4 dense laye s wi h an inpu o a ec o o 32 (noise) and an ou pu o a ec o o size 11 ha co esponds o he numbe o columns o he da ase . As ac i a ion unc ion, all o hem a e using a ‘ elu’. The sizes o he a ious laye s can be changed and adap ed o he da a you a e wo king wi h. 28 . Figu e 5.8. Disc imina o used Al hough i could seem deepe , he disc imina o has he same numbe o laye s. The only di e ence wi h he gene a o is ha we ha e used D opou s o slow down he lea ning p ocess o he disc imina o bea ing in mind ha i is usually ained mo e imes han he gene a o . 29 5.3.3 Pa ame e s o Gene ic Algo i hm Mul iple pa ame e s ha e been ied ou , bu he inal choice o he bes ade-o be ween ime and good quali y o he expe imen we e hese ones: • Popula ion size: 250 • P obabili y o c osso e : 0.60 • P obabili y o mu a ion: 0.10 • Numbe o Gene a ions: 20 Also, he decision has been in luenced by he academy ha o en ad ises using a li le pe cen age o mu a ion and a bigge one o he c osso e . In he beginning, we ha e ied di e en p obabili ies o he ope a o s. In he case o la ge alues, he indi iduals esul in ha ing a la ge di e si y, al hough i is bene icial o explo e he ull sea ch space, you o en end up ge ing u he away om he local op ima. One way o implemen he bene i o he di e si y gi en by he high p obabili ies o he ope a o s would possibly be o implemen a decay a e. In case a decay a e is se , he pe cen ages would ge lowe and lowe he mo e he gene a ions pass by. This would lead he algo i hm o explo e be e he sea ch space a he beginning and once i eaches di e en local op ima i would i e a e close o hem. Ano he cons ain ha we ha e was he popula ion size and he numbe o gene a ions, as i ge s compu a ionally mo e expensi e, we could ha e no inc emen ed hese alues ha migh ha e had helped he sea ch wi h high ope a o s’ p obabili ies. 6. RESULTS In his chap e , we a e going o illus a e he esul s ob ained wi h he bes pa ame e s ound om he gene ic algo i hm. A i s , we a e going o illus a e how we decided o measu e he quali y o he gene a ed da a and he easons why we ha e chosen hose me ics. We will hen p oceed on showing he pe o mance ob ained by compa ing he syn he ic da a wi h he o iginal one. The pe o mances o he bes -pa ame e s da ase a e compa ed wi h he s a e-o -a pa ame e s da ase . 6.1 METRICS USED Unde s anding how good he quali y o he gene a ed da a is no i ial and un il now, al hough a lo o me ics ha e been de eloped, he e a e no g ounded ways o assess he da a quan i a i ely and 30 quali a i ely [22]. Un o una ely, as o he i ness unc ion, he e is no a one- i -all solu ion. This means ha is impossible ( o da e) o measu e he quali y o a da ase wi h jus one me ic. So, o de e mine he quali y o ou da a we decided o use 3 me ics ha o e all co e he di e en aspec s o he da a. O cou se, mo e me ics could ha e used, bu ha depends a lo on wha i is he end goal o he syn hesisa ion. In ou case, we a e jus looking o wa d o compa ing how he di e en gene a ed da ase s a e, in e ms o simila i y o he o iginal da a. 6.1.1 Basic S a is ics The i s hing we hough o was ha he gene a ed da a should ha e had simila beha iou o he eal da a, so we expec ed all he basic s a is ics o be close o each o he : • Mean • S anda d De ia ion • Pe cen iles • Maximum • Minimum The main eason why you wan o see how close he basic s a is ics a e, is o see i he gene a ed alues a e kep in he expec ed anges. Some imes, du ing he gene a ion o syn he ic da a, you could end up wi h alues ha do no e lec eali y. Fo example, a lo o ime you da a canno ha e alues unde ze o o alues ha exceed a ce ain h eshold. F om looking a he maximum and he minimum you can al eady ha e a g asp o he ange o you gene a ed da a. To a oid his, i is o en implemen ed he use o cons ain s on he ange o he possible alues ha you a iable can assume (could be implemen ed ei he on he pos -p ocessing as we did o on he algo i hms hemsel es). On he o he hand, he mean, s anda d de ia ion, and pe cen iles gi e you a u he unde s anding o how he alues a e dis ibu ed along wi h he da ase , hus i you end up wi h some kind o mode collapse. To calcula e how simila hese basic s a is ics we e o he o iginal da a we decided on compa ing hem calcula ing he pai wise Euclidean dis ance be ween he s a is ics. In o de o do so, wo da a ames we e c ea ed: one con aining all he a o emen ioned s a is ics o he eal da a, and one o he gene a ed da a. These da ase s we e apidly c ea ed u ilizing he unc ion ‘desc ibe’ o pandas. One d awback o his me ic is ha he esul ing alue is no scaled. This implies ha he e is no a uni e sal way o in e p e ing he esul s as i could be o some me ic in a speci ied ange (i.e., 0 o 1). In ou case, as we a e using i jus 31 o compa e di e en da ase s gi es us jus he gene al idea o how good one sample is in espec o ano he . As i is a dis ance, he smalle he alue, he close he gene a ed da a o he o iginal one. 6.1.2 Dis ibu ions Ano he aspec ha was wo h calcula ing was he dis ibu ion o each a iable. Al hough looking a he s a is ics gi es us a gene al idea o how he da a is beha ing, some es s gi e us a deepe unde s anding o wha conce ns he dis ibu ions. To see he simila i y be ween he dis ibu ions, we decided o use he ollowing wo s a is ical es s: Chi-Squa ed and Kolgomo o -Smi no . The Chi-Squa ed es is used o compa e dis ibu ions o disc e e o ca ego ical a iables. The es e u ns he esul ing p- alue so ha a small alue indica es ha we can ejec he null hypo hesis and sugges s ha he dis ibu ions a e di e en . The Kolmogo o –Smi no es is used o compa e he dis ibu ions o con inuous a iables using he empi ical Cumula i e Dis ibu ion Func ion (CDF). I e u ns 1 minus he KS Tes D s a is ic, which indica es he maximum dis ance be ween he expec ed CDF and he obse ed CDF alues. Bo h he es s ou pu a alue be ween 0 and 1: i he dis ibu ions a e iden ical, you ge he maximum alue (1), in case hey a e comple ely di e en you would ge he minimum alue (0). As bo h o hem ha e he same ange, we decided o combine hem by calcula ing hei mean. As opposed o he la e me ic, his would be use ul o gi e you in o ma ion on he da ase wi hou ha ing o compa e i o o he s. 6.1.3 Co ela ion o Co a iances Looking a how he da a is dis ibu ed and i i has he igh alues does no mean ha he da a gene a ed is o high quali y. As you a e o en gene a ing mo e han one column you expec you gene a ed da a o keep he same ela ions be ween he a iables o he o iginal da a. This is needed because he gene a ed da a is o en used o sol e some machine lea ning asks, and in o de o do so you need o keep all he possible in o ma ion. To unde s and he ela ionship ha exis s be ween he a iables we ha e decided o look a he co a iance ma ix. As he co a iance is he join a iabili y o wo andom a iables, he ma ix would gi e us all he in o we need. To calcula e he simila i y be ween hese ma ixes, once again, he pai wise Euclidean dis ance has been used. 32 6.2 FINAL RESULT The bes pa ame e s ound om ou algo i hm o he inal indi idual we e he ollowing: Figu e 6.1. Bes Pa ame e s Found This indi idual eached a i ness unc ion o 0.7854 meaning ha he dis ibu ions o he eal and syn he ic samples we e e y close o each o he . F om he pa ame e s ound we can see ha he ones ha a e he a hes om he s a e-o -a ones a e lea ning a e and be a 2. This means ha a e possibly he ones ha in luence mo e he esul o he WGANs. The o he pa ame e s, al hough no equal, a e close o he s anda d ones. One o he easons why hey a e so close could be he ac ha we pu some cons ain s on he possible alues hey could ha e aken, bu a he same ime, I highly doub ha going oo a om he s anda d ones would ha e led o op imal esul s. F om he inal compa ison o he eal da a wi h he syn he ic one gene a ed by he bes indi idual, we go hese esul s. Figu e 6.2. Resul s o he expe imen As we can see om he able, he pa ame e s ound by he gene ic algo i hm esul ed in ha ing be e da a. The dis ibu ions a e mo e simila and he dis ances be ween s a is ics and co a iances a e smalle . The esul s a e no ou s anding in e ms o di e ence, o example, we can see ha he dis ances o he co ela ion a e close o each o he . As we al eady discussed, i is no s ange o ha e alues ha a e no so dis an , as he pa ame e s a e no oo di e en . Ano he eason why i migh ha e no been an excellen esul is ha Neu al ne wo k models a e s ochas ic. This means ha , gi en he same model con igu a ion and he same aining da ase , a di e en in e nal se o weigh s will esul in each ime he model is ained ha will in u n ha e a di e en pe o mance. The da a has been gene a ed using he bes pa ame e s bu no he exac model used du ing he esea ch o hose. 33 7. CONCLUSIONS AND FUTURE WORK In his hesis, we ha e shown ha al hough he quali y o he bes pa ame e ’s da a did no su pass by a he quali y o s anda d pa ame e s, he p oposed gene ic app oach is be e han he no mal andom sea ch. We ha e o ake in o accoun ha he e has no been done oo much wo k on he p e-p ocessing o he da a, maybe app oxima ing he da a o a gaussian-like dis ibu ion would s abilize he aining o he ne wo k and le ocus he ne wo ks on he hype -pa ame e s hemsel es. Fu he mo e, we ha e p oposed a compa ison me hod ha p o ides an o e all iew o he quali y o he gene a ed samples. Fu u e imp o emen s can be achie ed in a couple o di e en a eas o he p ojec . Fi s , we did no ha e enough compu a ional powe o un big popula ions, and as we know ha he sea ch space is immense, ha ing mo e indi iduals in he popula ion would al eady b ing be e esul s. The gene ic ope a o s ha ha e been used we e basic, so as we saw om he li e a u e e iew, mo e complex ope a o s would b ing be e indi iduals. Ano he imp o emen ha can be done is on he i ness unc ion; as he e is no a uni e sal me ic o he quali y o he p oduced da a, using di e en i ness unc ions could b ing be e esul s, as i has been done on [18]. O he han wo king on he gene ic algo i hm i sel some mo e wo k can also be done on he GAN a chi ec u e ha ing some mo e s able ne wo ks could b ing bene i s o he calcula ion o he i ness unc ion. 34 REFERENCES [1] Ian J. Good ellow, J. P.-A. F. (2014), Gene a i e Ad e sa ial Ne wo ks, h ps://a xi .o g/abs/1406.2661 [2] A. Agga wala, M. Mi al, G. Ba ineni, (2021), ‘Gene a i e ad e sa ial ne wo k: An o e iew o heo y and applica ions’ [3] L. Huang, A. D. Joseph, B. Nelson, B. I. Rubins ein, and J. Tyga , ‘Ad e sa ial machine lea ning’, ACM Wo kshop on Secu i y and A i icial In elligence, pp. 43–58, 2011. [4] J. Gui, Z. Sun, Y. Wen, D. Tao, J. Ye, (2020), ‘A e iew on Gene a i e Ad e sa ial Ne wo ks: Algo i hms, Theo y, and Applica ions’ [5] M. A jo sky, S. Chin ala, L. Bo ou, (2017), ‘Wasse s ein GAN’ [6] I. Gul ajani, F. Ahmed, M. A jo sky, V Dumoulin, A. Cou ille, (2017), ‘Imp o ed T aining o Wasse s ein GANs’ [7] Wikipedia, ‘Lipschi z con inui y’, en.wikipedia.o g/wiki/Lipschi z_con inui y [8] A. E. Eiben, J. Smi h, (2015), ‘F om e olu iona y compu a ion o he e olu ion o hings’, Na u e [9] J. H. Holland. (1975), ‘Adap a ion in Na u al and A i icial Sys ems’, Uni e si y o Michigan [10] K. Tang, X. Yuan, P. Liu, J. Yang, (2011), ‘Linea E olu iona y Algo i hm’, In echOpen [11] P.A. Cas illo, M.G. A enas, J.J. Cas illo-Valdi ieso, J.J. Me elo, A. P ie o, G. Rome o, (2003), ‘A i icial Neu al Ne wo ks Design using E olu iona y Algo i hms’ [12] Y. Chang, J. Lin, J. Shieh, M. F. Abbod, (2012), ‘Op imiza ion he Ini ial Weigh s o A i icial Neu al Ne wo ks ia Gene ic Algo i hm Applied o Hip Bone F ac u e P edic ion’, Hindawi Publishing Co po a ion [13] D. O i e, G. So osal, C. E. Bo ges, C. Ma in, A. Alonso-Vica io, (2014), ‘E olu iona y algo i hms o hype pa ame e uning on neu al ne wo ks models’ [14] K. O. S anley, R. Miikkulainen, (2002), ‘E ol ing Neu al Ne wo ks h ough Augmen ing Topologies’, Massachuse s Ins i u e o Technology [15] E. Real, S. Moo e, A. Selle, S.Saxena, Y. L. Suema su, Jie Tan, Quoc V. Le, A. Ku akin, (2017), ‘La ge-Scale E olu ion o Image Classi ie s’, In e na ional Con e ence on Machine Lea ning, Sydney 35 [16] T.M. B euel, (2015), ‘The e ec s o hype pa ame e s on SGD aining o neu al ne wo ks’ [17] S. R. Young, D. C. Rose, T. P. Ka nowski, S.H. Lim, R. M. Pa on, (2015), ‘Op imizing Deep Lea ning Hype -Pa ame e s Th ough an E olu iona y Algo i hm’ [18] C.Wangy, C. Xuz, X. Yao, D. Taoz, (2018), ‘E olu iona y Gene a i e Ad e sa ial Ne wo ks’ [19] T. Schmiedlechne , I. Ng Zhi Yong, A. Al-Dujaili, E. Hembe g, U. O’Reilly, (2018), ‘Lipizzane : A Sys em Tha Scales Robus Gene a i e Ad e sa ial Ne wo k T aining’ [20] B. Rozie e, F. Tey aud, V. Hosu, H. Lin, J. Rapin, M. Zameshina, and O. Tey aud, (2020), ‘E olGAN: E olu iona y Gene a i e Ad e sa ial Ne wo ks’, a Xi :2009.13311 1 [21] h ps://www.kaggle.com/c/Gi eMeSomeC edi / [22] A. Bo ji, (2018), ‘P os and Cons o GAN E alua ion Measu e’ [23] L. Xu, M. Skola idou, A. Cues a-In an e, K. Vee amachaneni, (2019) ‘Modeling Tabula Da a using Condi ional GAN’, a xi :1907.00503