Online Learning Techniques for Neural Translation Systems

Escola Tècnica Supe io d’Enginye ia In o mà ica Uni e si a Poli ècnica de València Online Lea ning Techniques o Neu al T ansla ion Sys ems DEGREE FINAL WORK Deg ee in Compu e Enginee ing Au ho : Ke in Mon al á Mingue Tu o : F ancisco Casacube a Nolla Ál a o Pe is Ab il Cou se 2015-2016 Resum Els sis emes de aducció au omà ica s’han e se i desde la se a concepció pe a- duc o s p o essionals pe a accele a i acili a la se a asca. Aques os sis emes eben a- duccions edi ades p o essionalmen a a és del seu ús, lo que po pon ecialmen millo a el seu endimen . En les úl imes décades, Xa xes Neu onals A i icials s’han u ili za pe a desen olupa sis emes de aducció au omà ica comple s. En aques eball s’han u ili za dos algo i mes d’ap enen a ge online pe a millo a aduc o s neu onals, que ja ha ien acaba l’e apa d’en enamen . Una amplia gamma d’expe imen s s’han du a e me pe oba els hipe pa àme es óp ims pe els algo i - mes en cada asca de aducció, i desp és el endimen d’aques os sis emes, adap a s amb cadascun dels algo i mes ap enen a ge (con igu a s amb aques os hipe pa àme es óp ims), s’ha mesu a abans i desp és en qua e asques de aducció, i s’han ex e con- clusions sob e com han millo a o empi jo a . S’ha modi ica el codi del aduc o neu onal, s’ha implemen a un dels algo i mes p esen an s i s’ha desen olupa un nou codi pe o e i una in e ície pe a e expe imen s de o ma au oma i zada. Una al a amilia d’algo i mes ha sigu implemen ada i p obada amb els models i asques disponibles, sense esul a s posi ius. Finalmen , línies u u es d’in es igació en adap ació de aduc o s neu onals han si- gu conside ades i discu ides al inal d’aques eball, a més de la se a si uació en l’es a ac ual de la aducció au omà ica. Pa aules clau: ap enen a ge au omà ic, xa xes neu onals, xa xes neu onals ecu en s, aducció au omà ica, aducció neu onal Resumen Los sis emas de aducción au omá ica han sido u ilizados desde su concepción po aduc o es p o esionales pa a acele a y acili a su a ea. Es os sis emas eciben aduc- ciones edi adas p o esionalmen e a a és de su uso, que pueden po encialmen e mejo a su endimien o. En las úl imas décadas, se han u ilizado edes neu onales a i iciales pa a desa olla sis emas de aducción au omá ica comple os con éxi o. En es e abajo se han usado dos algo i mos de ap endizaje online pa a mejo a edes neu onales ya en enadas. Un amplio abanico de expe imen os se ha lle ado a cabo pa a encon a el conjun o óp imo de hipe pá ame os pa a los algo i mos en cada a ea, y se ha calculado el endimien o de es os sis emas, adap ados con cada algo i mo con sus hipe pa áme os óp imos encon ados de o ma empí ica. Se han ex aído conclusiones ace ca de la mejo a o empeo amien o de los aduc o es. Se ha lle ado a cabo una modi icación de la base de código del aduc o neu onal, jun o con la implemen ación de uno de los algo i mos y el desa ollo de una nue a he- amien a pa a p opo ciona una in e az pa a ealize expe imen os de o ma au oma- izada. O a amilia de algo i mos ha sido implemen ada y p obada con los modelos y a eas disponibles, con esul ados insa is ac o ios. Finalmen e, líneas po enciales de in es igación en adap ación de aduc o es neu o- nales han sido conside adas y discu idas al inal de es e abajo, jun o con su si uación en el es ado ac ual de la aducción au omá ica. Palab as cla e: ap endizaje au omá ico, edes neu onales, edes neu onales ecu en es, aducción au omá ica, aducción neu onal iii i Abs ac Machine T ansla ion sys ems ha e been used since hei incep ion by p o essional ansla o s o speed up and ease hei wo k. Those sys ems ecei e p o essionally edi ed ansla ions h ough hei use, which could po en ially imp o e hei pe o mance. In he las decades, A i icial Neu al Ne wo ks ha e been used o de elop comple e Machine T ansla ion sys ems o g ea success. In his wo k wo online lea ning algo i hms o A i icial Neu al Ne wo ks ha e been used o enhance al eady ained neu al ansla o s. A wide a ay o expe imen s ha e been ca ied ou o ind he op imal hype pa ame e s o he algo i hms in each ask, and hen he pe o mance o hose sys ems, adap ed wi h each algo i hm wi h hei em- pi ically ound op imal se o hype pa ame e s, has been measu ed be o e and a e in ou ansla ion asks, and conclusions ha e been ex ac ed on how hey imp o ed o wo sened. A modi ica ion o a neu al ansla o codebase has been ca ied ou , along wi h he implemen a ion o one o he algo i hms and he de elopmen o new codebase o p o- ide an in e ace o pe o m expe imen s in an au oma ed way. One addi ional amily o algo i hms has been implemen ed and es ed wi h he a ailable model and asks o no a ail. Finally, possible u u e lines o esea ch on adap a ion o neu al ansla o s ha e been conside ed and discussed a he end o his wo k, along wi h hei si ua ion in he cu en s a e o Machine T ansla ion. Key wo ds: machine lea ning, neu al ne wo ks, ecu en neu al ne wo ks, machine ansla ion, neu al ansla ion Con en s Con en s Lis o Figu es ii Lis o Tables ii Lis o Algo i hms iii 1 In oduc ion 1 1.1 Backg ound .................................... 1 1.2 Mo i a ion..................................... 1 1.3 Goals ........................................ 2 1.4 S uc u e...................................... 3 2 S a e o he a 5 2.1 S a is ical Machine T ansla ion . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.2 Neu al Machine T ansla ion . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.2.1 Recu en Neu al Ne wo ks . . . . . . . . . . . . . . . . . . . . . . . 7 2.2.2 Bidi ec ional Recu en Neu al Ne wo k . . . . . . . . . . . . . . . 10 2.2.3 Encode -Decode model . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.2.4 T aining .................................. 12 3 Online lea ning 13 3.1 T aining Neu al Ne wo ks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.2 Online lea ning algo i hms . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.2.1 S ochas ic G adien Descen . . . . . . . . . . . . . . . . . . . . . . . 14 3.2.2 AdaG ad.................................. 14 3.2.3 Passi e-Agg essi e............................ 14 4 Expe imen s 19 4.1 So wa e ...................................... 19 4.1.1 Theano................................... 19 4.1.2 G oundHog................................ 19 4.2 Expe imen a ion amewo k . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 4.3 Hype pa ame e sea ch ............................. 23 4.3.1 Xe ox ask................................. 23 4.3.2 EU ask................................... 25 4.4 Tes se expe imen s................................ 27 5 Conclusions and u u e wo k 29 Appendices Ac onyms 31 A. Ac onyms ..................................... 31 Bibliog aphy 33 Lis o Figu es 2.1 Example o Recu en Neu al Ne wo k. . . . . . . . . . . . . . . . . . . . . 7 2.2 Visualiza ion o a RNN un olded in ime. . . . . . . . . . . . . . . . . . . . 8 2.3 Illus a iono heGRU............................... 9 2.4 S uc u eo aBRNN................................ 10 4.1 Di e ences in BLEU in es se . . . . . . . . . . . . . . . . . . . . . . . . . . 27 4.2 Execu ion ime o expe imen s in he es se s. . . . . . . . . . . . . . . . . . 28 Lis o Tables 4.1 Co po a cha ac e is ics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 4.2 Ini ialBLEU..................................... 23 4.3 Hype pa ame e sea ch o he Xe ox ask, om English o Spanish, wi h SGDalgo i hm. .................................. 24 4.4 Hype pa ame e sea ch o he Xe ox ask, om Spanish o English, wi h SGDalgo i hm. .................................. 24 4.5 Hype pa ame e sea ch o he Xe ox ask, om English o Spanish, wi h AdaG adalgo i hm. ............................... 24 4.6 Hype pa ame e sea ch o he Xe ox ask, om Spanish o English, wi h AdaG adalgo i hm. ............................... 25 4.7 Hype pa ame e sea ch o he EU ask, om English o Spanish, wi h SGD algo i hm. ..................................... 25 4.8 Hype pa ame e sea ch o he EU ask, om Spanish o English, wi h SGD algo i hm. ..................................... 26 4.9 Hype pa ame e sea ch o he EU ask, om English o Spanish, wi h AdaG adalgo i hm. ............................... 26 4.10 Hype pa ame e sea ch o he EU ask, om Spanish o English, wi h AdaG adalgo i hm. ............................... 26 4.11 BLEU sco e ob ained by e aining on he es se ................ 27 ii Lis o Algo i hms 3.1 Passi e-Agg essi e app oxima ion pseudocode. . . . . . . . . . . . . . . . 17 4.1 Passi e-Agg essi e app oxima ion execu ion low. . . . . . . . . . . . . . . 21 iii CHAPTER 1 In oduc ion 1.1 Backg ound In 1949 Wa en Wea e laid he ounda ions o Machine T ansla ion (MT) by p oposing he use o compu e s o ackle he challenge o ansla ion. Consequen ly, in he 50s and 60s, se e al a emp s we e made o c ea e p ac ical ansla o s, and by he mid six ies he newly c ea ed ALPAC (Au oma ic Language P ocessing Ad iso y Commi ee)[15] published a epo in which hey pain ed he u u e o MT as bleak. Abou 10 yea s la e none heless, a e he concep ion o some new app oaches and he inc ease in p ocessing powe o compu e s, he e was a enewal o he expec ancies o MT, and esea ch and in e es ose back up. So a , MT is a challenge ha has been engaged wi h mul iple app oaches and me hod- ologies, bu mos o hem can, nowadays, be classi ied in o h ee ca ego ies[10]: Rule- Based app oaches, Co pus-Based ones and a hyb id app oach in be ween. Rule-Based Sys ems (RBS) equi e he use o human knowledge on languages, and he use o g am- ma s and ocabula ies. Co pus-Based Sys ems (CBSs) use big pa allel co po a o ex o he sys em o lea n hem and ep oduce hei ea u es. A big ad an age o CBSs o e Rule-Based Sys ems (RBSs) is ha language da a is eadily a ailable o almos any lan- guage in he wo ld, a big amoun o in o ma ion in se e al languages. F om books o news, mo ie sc ip s, ial ansc ip s and laws, he e is a la ge numbe o sou ces om which da a can be ex ac ed ha Co pus-Based ansla o s can use o imp o e he quali y o hei p oduc . Since CBSs a e ained on da a, hey also ha e (usually) he possibili y o scaling up and ou wi h ha dwa e, while RBSs a e usually limi ed by he quali y o hei ules. 1.2 Mo i a ion P o essional ansla o s use MT sys ems egula ly o speed up hei ask. The ansla ions ob ained h ough hose sys ems hough a e o i egula quali y, and many i no mos o hem mus be amended by human ansla o s o ob ain a sa is ac o y esul . Those co ec ions can be used o imp o e he MT sys em being used o, in he long e m, educe he numbe o co ec ions ha he p o essional needs o apply o u u e ansla ions. The limi being o cou se “pe ec ansla ions”, whe e no co ec ions a e needed. The co ec ion o machine-gene a ed ansla ions human ansla o s is known as pos - edi ion. In a pos -edi ion con ex , he human ansla o inpu s a sou ce sen ence, he so wa e gene a es a hypo hesis in he a ge language and he human co ec s he ans- la ion. In a simple sys em his co ec ion is only use ul o he use , since he sys em will 1 8S a e o he a x1 h1 y1 x2 h2 y2 x h −1 y ... h Inpu laye Hidden laye Ou pu laye Figu e 2.2: Visualiza ion o a RNN un olded in ime. x s ands o he - h inpu p ocessed by he ne wo k (al e na i ely, a he - h i e a ion), h is he hidden s a e o he ne wo k a i e a ion and y is he ou pu o he ne wo k a i e a ion . h = h(x ,h −1)(2.3) y = o(h )(2.4) hand odepend on he a chi ec u e o he ne wo k and he choice o ac i a ion unc- ions. RNNs, howe e , ha e one well-known sho coming when i comes o sequence p o- cessing: he anishing g adien p oblem[5]. G adien -based aining algo i hms upda e he ne wo k weigh s in p opo ion o he g adien o he e o unc ion wi h espec o each o hem. Since laye s “ u he away” om he ou pu laye (in e ms o connec- ions) ha e smalle con ibu ions o ou pu alues, his a o emen ioned g adien is in u n smalle . In he case o sequence-p ocessing RNNs, his means ha in o ma ion om elemen s a “in he pas ” is almos los . To sol e his p oblem, many RNN-based sys ems ha e success ully used he RNN a chi ec u es ha a e abou o be discussed. In he la e 90s, a amily o RNNs appea ed, called Long Sho Te m Memo y (LSTM) ne wo ks [14], which ha e shown e y good esul s in many sequence lea ning asks. They make use o a se o ga es o keep some kind o memo y. LSTM uni s a e ained o e ain di e en deg ees o memo y, which allows RNNs o success ully emembe long sequences. [25] used LSTMs o build a ela i ely simple neu al ansla o ha achie ed esul s compa able o hose o s a e-o - he-a SMT sys ems. Ano he ype o ga ed uni s a e GRUs, de eloped in [7] o a ph ase sco ing sys em based on Neu al Ne wo ks, as a pa o a Ph ase-Based ansla o . They a e simple han egula LSTM uni s, hey ha e ewe ga ing uni s, hus educing hei aining ime. They, howe e , ha e been shown o be as powe ul as LSTM. 2.2 Neu al Machine T ansla ion 9 RESET GATE 1− x h −1 h −1 z anh ⊙ h −1 x ~ h h −1 h UPDATE GATE x ⊙ ⊙ + Figu e 2.3: Illus a ion o he Ga ed Recu en Uni (GRU). The ese ga e adjus s how much o he p e iously s o ed in o ma ion is e ained and he upda e ga e adjus s how much o he newly acqui ed (cu en i e a ion) in o ma ion is kep . x and h ollow he same no a ion as in Figu e 2.2, while ˜ h ep esen s he upda ed s a e a i e a ion , is he ou pu o he ese ga e a i e a ion and z is he ou pu o he upda e ga e a i e a ion . A a gi en ime , a GRU cell holds a hidden s a e h . Gi en i s hidden s a e in he las i e a ion (a −1), i s cu en upda ed s a e ˜ h and he ou pu o he upda e ga e z , he cu en hidden s a e will be compu ed as ollows: h = (1−z )h −1+z ˜ h (2.5) whe e s ands o he elemen -wise mul iplica ion. The upda ed s a e ˜ h can be compu ed om he cu en inpu x , he p e ious hidden s a e h −1and he ou pu o he ese ga e : ˜ h = anh(Wx +U[ h −1]) (2.6) whe e Wand Ua e weigh ma ices, pa ame e s o he model. Bias e ms ha e been le ou o eadabili y. Finally, he ou pu o he ese and upda e ga es a e compu ed ollowing hese o mu- lae: =σ(W x +U h −1)(2.7) z =σ(Wzx +Uzh −1)(2.8) whe e W a e U ese ga e weigh ma ices, Wzand Uza e upda e ga e weigh ma ices and σis he elemen -wise logis ic unc ion. 10 S a e o he a x1 h 1 y1 x h −1 y ... h Inpu laye Fo wa d laye Ou pu laye xJ h J−1 yJ ... ... ... hb 2 hb hb +1 hb J Backwa d laye hb J h J hb h hb 1 h 1 Figu e 2.4: S uc u e o a BRNN. 2.2.2. Bidi ec ional Recu en Neu al Ne wo k In o de o imp o e he quali y o he ansla ions, when i comes o p ocessing a gi en wo d in he middle o a sen ence, we can choose o examine no only he wo ds ha p ecede bu also he ones ha ollow i . In o de o do ha , we need o p ocess he sen ence bo h ways: o wa d and backwa ds. The cu en a chi ec u e o RNN ha has been in oduced only accoun s o p e ious wo ds, bu we can include an addi ional hidden laye , independen o he p e ious one, ha will p ocess wo ds om las o i s . This a chi ec u e, in oduced in [24] as Bidi ec ional Recu en Neu al Ne wo k (BRNN), will allow he ne wo k o ake decisions based on he whole sen ence con ex . In his a chi ec u e, he e a e wo hidden s a es: o wa d (h ) and backwa d (hb), ha a e compu ed as ollows: h = h(x ,h −1)(2.9) hb = h(x ,h +1)(2.10) y = o(h ,hb )(2.11) 2.2.3. Encode -Decode model In he model p oposed by Cho e al. (2014) and Su ske e e al. (2014), we ha e a sys em composed o wo RNNs: an Encode and a Decode . The model used in his wo k is he one p oposed by Bahdanau e al. (2014), which is an ex ension o he a o emen ioned wo. 2.2 Neu al Machine T ansla ion 11 The Encode , which in ou case is a BRNN, encloses he inpu sen ence in a con ex ec o . By means o eading each elemen o he inpu sen ence, he hidden s a e o he ne wo k changes, and a e comple ing he eading i is a compendium o he sen ence. In ou case, he con ex ec o is ob ained om he conca ena ion o he hidden s a es o he o wa d ecu en laye and he backwa d ecu en laye . The Decode in u n has he ask o gene a e he ou pu sen ence om he a o emen- ioned con ex ec o . I e a i ely, he Decode will gene a e a wo d gi en he con ex ec o and he p e iously gene a ed wo ds. Once he Decode gene a es a special “end o line” wo d, he comple e sen ence is he sys em’s hypo hesis o he sou ce sen ence. Fo each wo d xj(1 ≤j≤J) in he sou ce sen ence, belonging o he sou ce ocab- ula y Vs, we p oduce a ec o xj∈ {0, 1}Vs, whe e e e y en y is se o ze o excep he one co esponding o xj, which is se o one. This is called one-ho codi ica ion. Then, he wo ds a e p ojec ed o a ixed-size con inuous ec o in he ollowing way: xj=Esxj(2.12) whe e xjis he embedding o wo d xjand Esis he sou ce language p ojec ion ma ix. The sequence o wo d embeddings, ep esen ed as x=x1, ...,xJ, is he inpu o he Encode . A e p ocessing each wo d xj, he hidden s a e o he Encode hjis eco ded. Since we ha e op ed o using a BRNN as he Encode , ou hidden s a e is ac ually hj= [h | j;hb| j]|(2.13) Once he sen ence has been ully p ocessed, an i e a i e p ocess begins. A each s ep, a non-linea unc ion qis applied o he sequence o hidden s a es and o he hidden s a e o he Decode ne wo k a he p e ious s ep, in o de o ob ain a con ex ec o c. In his wo k an a en ion mechanism has been used, he e o e he unc ion qis a weigh ed sum o he hidden s a es. This wo ks like an alignmen model, implemen ed by a Mul ilaye Pe cep on (MLP), be ween he sou ce sen ence and he a ge sen ence, and hus, we ha e a di e en con ex ec o ci o each s ep i: ci=q({h1,...,hJ},gi−1)(2.14) Thus, ciis a dynamic ep esen a ion o he inpu sen ence, based in he s a e o he Decode . This con ex ec o is hen ed o he Decode . The Decode p ocesses he con ex ec- o , and ou pu s Vs eal numbe s be ween ze o and one, whe e he i- h ou pu ep esen s he p obabili y o he i- h wo d in he a ge language o be nex in he ansla ion o he sou ce sen ence. This ou pu depends on ci,giand he wo d embedding ep esen a ion o he las emi ed wo d. This is implemen ed h ough a so max laye , which ensu es ha all he p obabili ies add up o one: y0 k=yk ∑Vs l=1yl (2.15) whe e yi ep esen s he i- h inpu o he so max laye , and y0 i ep esen s i s i- h ou pu , always be ween ze o and one. The e o e, he p obabili y o a wo d a ime-s ep iwould be: p(yi|y1, ..., yi−1,x;θ) = y| iϕ(Vη(yi−1,gi,ci)) (2.16) 12 S a e o he a whe e ϕ(·)is a so max unc ion, yiis he one-ho ec o ep esen a ion o wo d yi,Vis he weigh ma ix and ηis he ou pu o a RNN wi h GRU uni s and a maxou ou pu laye [12]. 2.2.4. T aining Following Equa ion (2.1), ou ne wo k aims o app oxima e P(y|x)in he ollowing way: P(y|x) = I ∏ i=1 P(yi|y1, ..., yi−1,x)(2.17) In o de o maximize P(y|x) o ou aining se , consis ing o a bilingual co pus o S sen ence pai s, and acco ding o Equa ion (2.17), we need o ind a se o pa ame e s o ou model ˆ θsuch as: ˆ θ=a gmax θ S ∏ s=1 I ∏ i=1 p(y(s) i|y(s) 1, ..., y(s) i−1,x(s);θ) =a gmax θ S ∑ s=1 I ∑ i=1 log(p(y(s) i|y(s) 1, ..., y(s) i−1,x(s);θ)) (2.18) whe e x(s)and y(s) ep esen he s- h sen ence o he aining se in he sou ce and a ge languages espec i ely, and Iis he leng h o he s- h a ge sen ence. Since each wo d o he sys em’s hypo hesis depends on p e iously gene a ed wo ds and he con ex ec o , and he con ex ec o depends only on he sou ce sen ence, bo h Encode and Decode can be ained as a whole o maximize he condi ional p obabili y o he a ge sen ences gi en he sou ce sen ences. So a , we ha e in oduced he knowledge ield o Machine T ansla ion. We ha e ou - lined he app oaches ha ha e been adop ed in he las decades o ad ance he quali y o ansla o s, and we ha e desc ibed s a e-o - he-a echniques ha powe he op ans- la ion so wa e in he ield. Finally, we ha e e iewed in dep h he Encode -Decode app oach and he Neu al Ne wo ks employed in i . Wha ollows is a desc ip ion o he wo k ha was ca ied ou in o de o pe o m he expe imen s. CHAPTER 3 Online lea ning 3.1 T aining Neu al Ne wo ks Ou objec i e in aining is o maximize he log-likelihood o he da a we use o aining, in an a emp o p oduce a sys em ha can gene alize ha se o ansla ions in o he o e all ansla ion ask. By ollowing Equa ion (2.18), we can une he sys em pa ame e s θ o maximize his sum o log-p obabili ies. The ainable pa ame e s o RNNs a e he weigh ma ices. Mos Neu al Ne wo k aining algo i hms a e i e a i e, hey upda e he ne wo k weigh s acco ding o a ule s ep by s ep, un il a gi en condi ion is accomplished. Upda e ules, s ep size and s opping condi ion de ine he di e en lea ning echniques. The ollowing is a classi ica ion by s ep size[19]: •Ba ch lea ning echniques a e hose ha upda e he weigh s o he ne wo k a e he whole aining se has been p ocessed. Once he sys em has e alua ed e e y sample, i s weigh s a e upda ed o i hose, and a new aining i e a ion begins. I he e mina ion condi ion is eached, he aining s ops. •Mini-ba ch lea ning echniques indica e ha he upda es mus be applied a e an a bi a y numbe o samples ha e been p ocessed. •Online lea ning echniques, inally, a e hose ha upda e he ne wo k weigh s a e e e y single sample. They a e a pa icula in e es o us, on accoun o hem being a pe ec i o he si ua ion desc ibed a he beginning o his wo k: imp o ing a sys em a e a new sample is ob ained. Wi h he goal o enhancing a neu al ansla o wi h i s use, a se up like he ollow- ing can be adop ed: he sys em gene a es ansla ions o a human ansla o , who inpu s co ec ed e sions o hose ansla ions, which he sys em uses one by one o upda e he weigh s o i s in e nal Neu al Ne wo k. 3.2 Online lea ning algo i hms In he nex sec ions we desc ibe he algo i hms we ha e chosen o compa e. 13 14 Online lea ning 3.2.1. S ochas ic G adien Descen SGD is a lea ning algo i hm[22] ha app oxima es G adien Descen by upda ing he weigh s o he sys em using he ollowing ule: θs=θs−1−η∇`s(θs−1)(3.1) This upda e is pe o med wi h he g adien o each sample o he aining se , hus app oxima ing he g adien o he whole se . In (3.1) ηis he lea ning a e, which se s he pace o he upda es o he model. This is he only pa ame e we can une in his algo i hm. I is usually se o alues lowe han one, in o de o modi y he model in small s eps owa ds minima in he e o unc ion. E e y ime he weigh s o he ne wo k a e upda ed, he sys em ies o imp o e i s pe - o mance owa ds he new sample, and in he p ocess, i s pe o mance wi h p e iously seen samples may ge wo se. In o de o y o achie e a good pe o mance in he a ge da a o he sys em as a whole, he lea ning a e is used. I he lea ning a e we e oo high, he sys em would agg essi ely y o i new samples a he expense o pas da a, esul ing likely in an o e all bad pe o mance. I i we e oo small, he ne wo k would conse a i ely lea n he da a, equi ing a e y high numbe o i e a ions, and hus a e y long ime, o be ained. 3.2.2. AdaG ad AdaG ad is a amily o adap i e, subg adien , online lea ning algo i hms de eloped in [11], based on SGD, ha is expec ed o ou pe o m i o high-dimensional, spa se ea- u es. The implemen a ion used in his wo k is an app oxima ion ob ained om [13]. I s upda e ule is as ollows: s= s−1−ηG−1/2 s∇ `(3.2) Gs=Gs−1+ (∇ `)2(3.3) whe e is any gi en weigh o θ,∇ ` ep esen s he g adien o he loss unc ion wi h he p e ious weigh se wi h espec o weigh be o e p ocessing sample sand Gs ep- esen s he sum o squa ed g adien s be o e p ocessing sample s. A any gi en ime, Gs=∑s i=1∇ `2 i. In his case, we also ha e a lea ning a e pa ame e ha we can une in o de o seek he op imal pe o mance o he algo i hm. 3.2.3. Passi e-Agg essi e Passi e-Agg essi e a e a amily o ma gin-based online lea ning algo i hms, p oposed in [8]. The goal o hose algo i hms is o ind, a each s ep, he model which, being as close as possible o he cu en one, achie es some gi en ma gin on he cu en sample. This is a cons ain op imiza ion p oblem ha is sol ed by he Lag ange mul iplie s echnique o ind an upda e ule ha mee s he condi ions. Since he ma gin equi emen migh be a ha d one, he PA-II and PA-III algo i hms include an “agg essi eness” hype pa ame e ha allows o a ade-o be ween he desi ed ma gin and he p oximi y o he cu en model. 3.2 Online lea ning algo i hms 15 These algo i hms ha e he ollowing upda e ule: θs+1=θs+sign(ys−ˆys)τsxs(3.4) whe e τsdepends on he pa icula algo i hm: τs=       `s ||xs||2, PA min(C,`s ||xs||2), PA-I `s ||xs||2+1 2C , PA-II (3.5) whe e Cis a pa ame e called agg essi eness in [8] and `sis he alue o he loss unc ion a ime . As we can see in Equa ion (3.4) and Equa ion (3.5), PA-I has no hype pa ame e s and PA-I and PA-II ha e one: C. In o de o sol e ˆ θ=a gminθ1 2||θ−θs||2s. . `(θ,xs,ys,hs)≤0 we use he Lag ange mul iplie s echnique: `(θ,xs,ys,hs) = log pˆ θ(hs|xs)−log pˆ θ(ys|xs)(3.6) We s a by ob aining Lag ange unc ion: L(θ,λ) = 1 2||θ−θs||2+λ`(θ,xs,ys,hs)(3.7) whe e λis a Lag ange mul iplie . Nex we ob ain he g adien , which would be ze o a he minimum: ∇θL(θ,λ) = θ−θs+λ∇θ`(θ,xs,ys,hs) = 0 (3.8) θ=θs−λ∇θ`(θ,xs,ys,hs)(3.9) A e wa ds, we ge he pseudo-dual unc ion: LD(θ,λ) = 1 2λ2||∇θ`(θ,xs,ys,hs)||2+λ`(θ,xs,ys,hs)(3.10) As we did be o e, we look o he minimum: ∂LD(θ,λ) ∂λ =λ||∇θ`(θ,xs,ys,hs)||2+`(θ,xs,ys,hs) = 0 (3.11) ˆ λ=−`(θ,xs,ys,hs) ||∇θ`(θ,xs,ys,hs)||2(3.12) Which is he op imal solu ion o he Lag ange mul iplie λ. Along wi h Equa ion (3.9), we can ob ain he pseudo op imal solu ion: θ=θs+`(θ,xs,ys,hs)∇θ`(θ,xs,ys,hs) ||∇θ`(θ,xs,ys,hs)||2(3.13) PA-I equi es sol ing ˆ θ=a gminθ1 2||θ−θs||2+Cξs. . `(θ,xs,ys,hs)≤ξ, which is done simila ly o equa ions 3.7 o 3.13: 16 Online lea ning L(θ,λ1,λ2) = 1 2||θ−θs||2+Cξ+λ1(`(θ,xs,ys,hs)−ξ)−λ2ξ ∇θL(θ,λ1,λ2) = θ−θs+λ1∇θ`(θ,xs,ys,hs) = 0 θ=θs−λ1∇θ`(θ,xs,ys,hs) ∂L(θ,λ1,λ2) ∂ξ =C−λ1−λ2=0→C=λ1+λ2 LD(θ,λ1,λ2) = 1 2λ2 1||∇θ`(θ,xs,ys,hs)||2+Cξ +λ1`(θ,xs,ys,hs)−(λ1+λ2)ξ ∂LD(θ,λ1,λ2) ∂λ1=λ1||∇θ`(θ,xs,ys,hs)||2+`(θ,xs,ys,hs) = 0 ˆ λ1=min(C,−`(θ,xs,ys,hs) ||∇θ`(θ,xs,ys,hs)||2) θ=θs−min(C,−`(θ,xs,ys,hs) ||∇θ`(θ,xs,ys,hs)||2)∇θ`(θ,xs,ys,hs)(3.14) whe eas PA-II equi es sol ing ˆ θ=a gmin θ 1 2||θ−θs||2+Cξ2s. . `(θ,xs,ys,hs)≤ξ L(θ,λ) = 1 2||θ−θs||2+Cξ2+λ(`(θ,xs,ys,hs)−ξ) ∇θL(θ,λ) = θ−θs+λ∇θ`(θ,xs,ys,hs) = 0 θ=θs−λ∇θ`(θ,xs,ys,hs) ∂L(θ,λ) ∂ξ =2Cξ−λ=0→ξ=λ 2C LD(θ,λ) = 1 2λ2||∇θ`(θ,xs,ys,hs)||2+Cλ 2C2 +λ`(θ,xs,ys,hs)−λλ 2C ∂LD(θ,λ) ∂λ =λ||∇θ`(θ,xs,ys,hs)||2+λ 2C+`(θ,xs,ys,hs)−λ C=0 ˆ λ1=−`(θ,xs,ys,hs) ||∇θ`(θ,xs,ys,hs)||2−1 2C θ=θs+`(θ,xs,ys,hs)∇θ`(θ,xs,ys,hs) ||∇θ`(θ,xs,ys,hs)||2−1 2C (3.15) As can be seen in Equa ion (3.13), θis ound in bo h sides o he equa ion, hus he need o he app oxima ion using ixed-poin i e a o s, which can be seen in Al- go i hm 3.1 o he PA algo i hm, while PA-I and PA-II a e iden ical, equi ing only a change in he upda e line o he co esponding o mula, o be like Equa ion (3.14) and Equa ion (3.15). Al hough he implemen a ion is u he explained in Sec ion 4.1.2, i is wo h no ing ha in he p elimina y expe imen s no p omising esul s we e achie ed wi h ei he o he h ee e sions o he algo i hm, and he e o e i was d opped om he expe imen a ion plan owa ds he end o he p ojec . 3.2 Online lea ning algo i hms 17 Inpu : Pa ame e s a he beginning o he i e a ion θs Sou ce sen ence xs Ta ge sen ence ys Hypo hesis hs Ou pu : θnew Ini ializa ion: θnew =θs epea 1. θold =θnew 2. θnew =θs+`(θold,xs,ys,hs)∇θ=θold `(θ,xs,ys,hs) ||∇θ=θold `(θ,xs,ys,hs)||2 un il θold == θnew Algo i hm 3.1: Passi e-Agg essi e app oxima ion pseudocode. 24 Expe imen s I e a ions Lea ning a e 0.05 0.1 0.2 0.4 0.8 1 66.2 66.0 65.9 64.2 61.0 3 66.0 66.2 - - - 5 66.4 66.2 65.3 60.9 - 10 66.5 66.4 66.1 60.8 - 20 66.6 66.8 - - - Table 4.3: Hype pa ame e sea ch o he Xe ox ask, om English o Spanish, wi h SGD algo- i hm. I e a ions Lea ning a e 0.05 0.1 0.2 0.4 0.8 1 69.9 69.9 69.8 68.7 63.7 370.1 69.8 70.6 68.4 61.3 5 70.5 70.0 69.5 69.3 60.0 Table 4.4: Hype pa ame e sea ch o he Xe ox ask, om Spanish o English, wi h SGD algo- i hm. SGD Table 4.3 shows gene al bu small imp o emen using a lea ning a e lowe han 0.2. I is possible ha sligh ly be e esul s could be achie ed wi h e en lowe lea ning a es, bu hose expe imen s ell ou side he scope o his wo k. The se o hype pa ame e s (0.05,5)was selec ed as a ade-o be ween pe o mance and quali y, since he bes esul (66.8) was achie ed pe o ming 20 i e a ions pe sample, se up which would be liable o slowing down oo much ansla ion so wa e. Table 4.4 none heless showed much be e esul s han Table 4.3, a guably because o di e ences in he models (di e en numbe o hidden nodes in hei ne wo ks, di e en aining ime...), o maybe because his ask was easie han he o me (is ansla ing om English o Spanish ha de han om Spanish o English?). The bes esul in his ba ch o expe imen s achie ed an imp o emen o 5.3 poin s in he BLEU sco e. AdaG ad Acco ding o Table 4.5, he model in he English-Spanish ask imp o ed he mos by pe o ming 20 aining i e a ions pe sample wi h a lea ning a e o 0.0001, and jus as well by pe o ming 3 i e a ions pe sample wi h a lea ning a e o 0.0005. Since speed I e a ions Lea ning a e 5e-5 1e-4 5e-4 1e-3 1 67.5 67.7 69.0 67.7 367.7 68.2 69.4 68.0 5 67.8 68.8 69.3 68.3 10 68.3 69.1 68.6 67.6 20 68.9 69.4 68.9 66.6 Table 4.5: Hype pa ame e sea ch o he Xe ox ask, om English o Spanish, wi h AdaG ad algo i hm. 4.3 Hype pa ame e sea ch 25 I e a ions Lea ning a e 5e-5 1e-4 5e-4 1e-3 5e-3 1 69.9 69.8 71.0 70.8 59.8 3 69.8 70.2 71.1 - - 569.8 70.1 71.7 70.5 57.6 Table 4.6: Hype pa ame e sea ch o he Xe ox ask, om Spanish o English, wi h AdaG ad algo i hm. I e a ions Lea ning a e 0.05 0.1 0.2 0.4 0.8 1 35.5 35.3 35.4 34.8 33.3 335.3 35.5 36.1 35.6 35.4 5 35.4 35.6 35.8 36.1 - Table 4.7: Hype pa ame e sea ch o he EU ask, om English o Spanish, wi h SGD algo i hm. is impo an in he con ex o his wo k, he la e has been chosen as he op imal se o hype pa ame e s o AdaG ad in his ask. I is wo h no ing ha none o he esul s o his able esul in a decline in he pe o - mance. Mo eo e , he a e age imp o emen was o 2.3 pe cen age poin s, which shows ha AdaG ad achie ed a much be e o e all imp o emen in his se han SGD. Ne - e heless, his ask is being used o hype pa ame e sea ch, so conclusions mus no be d awn om his compa ison, gi en ha we pu posely selec he bes esul o each o hem. In Table 4.6 we see imp o emen s o he same o de o Table 4.4, which leads o hink ha his ask is qui e adep a being e ained. 4.3.2. EU ask The EU co pus[16] was ob ained om he Bulle in o he Eu opean Union, which is pub- licly a ailable in all he o icial languages o he Eu opean Union. As in he Xe ox asks, only Spanish o English and English o Spanish we e used. The co pus was okenized as well, bu no ans o med o lowe case as we did in Sec ion 4.3.1. SGD Table 4.7 shows a case no encoun e ed so a : none o he esul s show an imp o emen in he quali y o ansla ions. The e a e se e al ac o s ha can con ibu e o his phe- nomenon. Fi s o all, his se is much smalle han ha o he p e iously shown asks: 400 sen ence pai s e sus 1012 in Xe ox. Since he BLEU sco e is a measu e ha a emp s o ma ch human judgemen when a e aged o e a co pus[21], i s alue when he se is small can be expec ed o be less eliable han he case whe e i is applied o a big co pus. Fu he mo e, he domain o he EU co pus is in all likelihood mo e complex han he domain o Xe ox. Xe ox co pus is ull o sho sen ences, wi h e y epe i i e wo ds, like p in e ea u es and op ions. Numbe s in Xe ox a e mos o he ime model iden i ie s, which do no change. EU bulle in is e y di e se, has long sen ences, a high quan i y o numbe s o eco ds, da es, pe cen ages, ile sizes, and so on. Also, he ini ial BLEU in bo h could no be any mo e di e en , and he ocabula y size in bo h languages in Xe ox is less han hal han in he EU models. 26 Expe imen s I e a ions Lea ning a e 0.05 0.1 0.2 0.4 0.8 1 35.1 35.1 35.1 35.3 34.4 335.0 34.8 35.2 36.0 35.5 5 35.1 35.7 35.8 35.8 34.8 Table 4.8: Hype pa ame e sea ch o he EU ask, om Spanish o English, wi h SGD algo i hm. I e a ions Lea ning a e 5e-5 1e-4 5e-4 1e-3 1 35.9 35.4 35.7 35.0 3 36.1 35.7 36.1 35.5 535.8 36.2 35.7 33.7 Table 4.9: Hype pa ame e sea ch o he EU ask, om English o Spanish, wi h AdaG ad algo- i hm. We can say wi hou a shade o doub ha he Xe ox ansla ion models in his wo k a e mo e e ec i e han hei EU coun e pa s. I emains a ques ion whe he his makes he ask o e aining easie o ha de , o whe he i is inma e ial o i . Fu he esea ch would be necessa y o answe i . Finally, gi en he assump ions o co ela ion be ween model and algo i hm pe o - mances in di e en asks ha we e issued in he in oduc ion o his wo k, we mus as- sume oo ha he expe imen wi h he es se ha in ol es his pa icula con igu a ion will de e io a e he model pe o mance, bu no as much as wi h he o he pa ame e s ha we e ied. A small imp o emen can be obse ed in some o he en ies o Table 4.8. E en hose se ups ha esul ed in a decline in quali y o ansla ions did so only by a e y small amoun . This could be a ibu ed o he small size o he de elopmen se , in compa ison o he o he co pus. AdaG ad The esul s in Table 4.9 appea o be e y simila o hose in Table 4.7: no imp o emen in any case. This can likely be a ibu ed o he same hypo he ical easons ha we e gi en o he esul s in Table 4.7. The bes esul o he able was chosen o u h e expe imen a ion. Table 4.10 shows sligh ly mo e p omising esul s han Table 4.8, e en hough sligh ly ewe cases we e a emp ed. I e a ions Lea ning a e 5e-5 1e-4 5e-4 1e-3 1 35.3 35.7 35.8 35.0 3 35.5 36.1 35.4 35.5 535.7 35.9 36.3 35.8 Table 4.10: Hype pa ame e sea ch o he EU ask, om Spanish o English, wi h AdaG ad algo- i hm. 4.4 Tes se expe imen s 27 Task BLEU Ini ial SGD AdaG ad Xe ox En-Es 55.2 55.9 57.1 Xe ox Es-En 46.1 51.7 50.4 EU En-Es 36.8 36.4 36.4 EU Es-En 35.7 34.8 36.0 Table 4.11: BLEU sco e ob ained in he expe imen s on he es se o each ask, bo h wi hou e- aining and by e aining wi h each algo i hm using he hype pa ame e s selec ed in Sec ion 4.3. Xe ox En-Es Xe ox Es-En EU En-Es EU Es-En −1 0 1 2 3 4 5 6 0.7 5.6 −0.4 −0.9 1.9 4.3 −0.4 0.3 ∆BLEU SGD AdaG ad Figu e 4.1: Di e ences in BLEU ob ained in he es se o each ask by e aining wi h each al- go i hm, using he hype pa ame e s selec ed in Sec ion 4.3. The alues used can be ound in Table 4.11. 4.4 Tes se expe imen s F om he da a in Table 4.11 we see ha AdaG ad ou pe o ms SGD in wo o ou ou cases, and ma ches i in ano he one. The da a, howe e , is insu icien o d aw con- clusions abou AdaG ad being in gene al be e o his scena io. SGD ou pe o med AdaG ad in Xe ox Spanish-English by a ela i ely la ge ma gin, in compa ison wi h he o he esul s. The imp o emen achie ed by each algo i hm on each model is plo ed in Figu e 4.1. Bo h algo i hms showed conside ably be e esul s in he Xe ox asks han in he EU ones, which could be a ibu ed o, among o he hings, he ask complexi y and/o he al eady-exis ing model pe o mance. The models ha ob ained a high BLEU ini ially im- p o ed hei esul s, hey p obably ook p o i o he da a and enhanced hei pa ame e s as o be e i he ansla ion ask. The models ha ob ained a ela i ely low BLEU in he ini ial measu emen may ha e no been able o assimila e he new da a and may ha e 28 Expe imen s Xe ox En-Es Xe ox Es-En EU En-Es EU Es-En 0 10 20 30 40 50 60 Execu ion ime / sen ence pai s (s) SGD AdaG ad Figu e 4.2: Execu ion ime o expe imens in he es se s. been nega i ely a ec ed by he pa ame e upda es (as hey may ha e in he expe imen s wi h he de elopmen se ), bu e en hen, he quali y o he hypo heses was no much wo se o , and AdaG ad algo i hm could achie e an imp o emen in he EU Es-En ask The decline in pe o mance obse ed by e aining in bo h EU asks in p e ious ex- pe imen s is consis en wi h hese esul s. The Xe ox English-Spanish ansla ion ask has p oduced much di e en esul s han p io expe imen s, ob aining much lowe imp o e- men s, while Xe ox Spanish-English has main ained a high deg ee o imp o emen wi h bo h algo i hms. In Figu e 4.2 we can obse e he di e en execu ion imes o each algo i hm, a e aged o e he numbe o sen ence pai s o he ele an se . We can obse e ha he execu ion ime o EU asks was much longe han o Xe ox asks, likely due o hei longe sen- ences and highe ocabula y size (which makes hei models e en la ge ). CHAPTER 5 Conclusions and u u e wo k We ha e de eloped a ull- ledged expe imen a ion en i onmen o online adap a ion o neu al ansla o s, able o pe o m au oma ed ba ches o expe imen s wi h p ope han- dling and s o ing o he esul s. We ha e selec ed and p ocessed da a o expe imen s, and pe o med se e al o hem wi h di e en se s o hype pa ame e s, and ha e ound he ones among hem ha wo k bes o each combina ion o ask and algo i hm. Wi h a di e en se bu o he same ask, we ha e pe o med expe imen s wi h bo h algo i hms con igu ed o hei empi ically- ound bes capaci ies and ha e ob ained a compa ison o how he algo i hms can imp o e o wo sen he quali y o neu al ansla o s. We ha e ob ained an i e a i e app oxima ion o he Passi e-Agg essi e amily o on- line lea ning algo i hms, p oduced an implemen a ion o ou expe imen a ion ame- wo k and obse ed he lack o p omising esul s. Due o his, no u he expe imen a ion was ca ied ou wi h hose algo i hms. In h ee ou o ou ansla ion asks, we obse ed a sligh co ela ion be ween he esul s in he de elopmen se and he esul s in he es se . AdaG ad algo i hm esul ed mo e p omising han SGD, bu he la e ou pe o med he o me in one ask. All hings conside ed, we canno s a e ha AdaG ad would be he bes choice o e e y ask, bu a p io i i is a be e candida e. Se e al ques ions emain unanswe ed, such as o why is he e such dispa i y in he esul s be ween de elopmen se and es se in he Xe ox English-Spanish ask, o why he EU asks pu up such a challenge agains bo h algo i hms. Fu he esea ch is equi ed o answe hose ques ions. Mo e expe imen s, in ol ing mo e da ase s and mo e language pai s a e equi ed, since neu al ansla o s a e usually ained and used o single language pai s, and he esul s obse ed using one o hem may no co espond a all wi h esul s ob ained om a di e en one. In his wo k we ha e used wo co po a and one language-pai (in bo h di- ec ions), and e en hen we ha e ound signi ican di e ences be ween asks (especially be ween bo h di ec ions o he Xe ox English o/ om Spanish asks). The size o he EU de elopmen se , especially when compa ed o ha o he Xe ox co pus, aises he ques ion o whe he he hype pa ame e sea ch o he EU asks may ha e been comp omised, o a he leas whe he he size o his se has handicapped he hype pa ame e sea ch o hose models. Mo e simila pai s o se s could be used o help dispel his doub . Finally, he app oxima ion o Passi e-Agg essi e algo i hms mus be e ised in sea ch o al e na i e me hods o ixed poin i e a o s, o e en ually es hem agains he chal- lenges p esen ed in his wo k, and compa e hem o he o he wo algo i hms we ha e used. 29 Ac onyms A. Ac onyms BLEU . . . . . . . . . . . BiLingual E alua ion Unde s udy BRNN . . . . . . . . . . . Bidi ec ional Recu en Neu al Ne wo k CBS . . . . . . . . . . . . Co pus-Based Sys em GRU . . . . . . . . . . . . Ga ed Recu en Uni LSTM . . . . . . . . . . . Long Sho Te m Memo y MLP . . . . . . . . . . . . Mul ilaye Pe cep on MT . . . . . . . . . . . . Machine T ansla ion NLP . . . . . . . . . . . . Na u al Language P ocessing NMT . . . . . . . . . . . Neu al Machine T ansla ion RBS . . . . . . . . . . . . Rule-Based Sys em RNN . . . . . . . . . . . Recu en Neu al Ne wo k SGD . . . . . . . . . . . . S ochas ic G adien Descen SMT . . . . . . . . . . . . S a is ical Machine T ansla ion 31 Bibliog aphy [1] Dzmi y Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neu al machine ansla- ion by join ly lea ning o align and ansla e. a Xi p ep in a Xi :1409.0473, 2014. [2] F édé ic Bas ien, Pascal Lamblin, Raz an Pascanu, James Be gs a, Ian Good el- low, A naud Be ge on, Nicolas Boucha d, Da id Wa de-Fa ley, and Yoshua Bengio. Theano: new ea u es and speed imp o emen s. a Xi p ep in a Xi :1211.5590, 2012. [3] Je ome R Bellega da. A la en seman ic analysis amewo k o la ge-span language modeling. In EUROSPEECH, 1997. [4] Yoshua Bengio, Réjean Ducha me, Pascal Vincen , and Ch is ian Jan in. A neu al p obabilis ic language model. J. Mach. Lea n. Res., 3:1137–1155, Ma ch 2003. [5] Yoshua Bengio, Pa ice Sima d, and Paolo F asconi. Lea ning long- e m dependen- cies wi h g adien descen is di icul . Neu al Ne wo ks, IEEE T ansac ions on, 5(2):157– 166, 1994. [6] M Asunción Cas ano, F ancisco Casacube a, and En ique Vidal. Machine ansla- ion using neu al ne wo ks and ini e-s a e models. Theo e ical and Me hodological Issues in Machine T ansla ion (TMI), pages 160–167, 1997. [7] Kyunghyun Cho, Ba Van Me iënboe , Cagla Gulceh e, Dzmi y Bahdanau, Fe hi Bouga es, Holge Schwenk, and Yoshua Bengio. Lea ning ph ase ep esen a- ions using nn encode -decode o s a is ical machine ansla ion. a Xi p ep in a Xi :1406.1078, 2014. [8] Koby C amme , O e Dekel, Joseph Keshe , Shai Shale -Shwa z, and Yo am Singe . Online passi e-agg essi e algo i hms. J. Mach. Lea n. Res., 7:551–585, Decembe 2006. [9] Jacob De lin, Rabih Zbib, Zhongqiang Huang, Thomas Lama , Richa d M Schwa z, and John Makhoul. Fas and obus neu al ne wo k join models o s a is ical ma- chine ansla ion. In ACL (1), pages 1370–1380. Ci esee , 2014. [10] Bonnie J Do , Pamela W Jo dan, and John W Benoi . A su ey o cu en pa adigms in machine ansla ion. Ad ances in compu e s, 49:1–68, 1999. [11] John Duchi, Elad Hazan, and Yo am Singe . Adap i e subg adien me hods o online lea ning and s ochas ic op imiza ion. The Jou nal o Machine Lea ning Resea ch, 12:2121–2159, 2011. [12] Ian J Good ellow, Da id Wa de-Fa ley, Mehdi Mi za, Aa on Cou ille, and Yoshua Bengio. Maxou ne wo ks. a Xi p ep in a Xi :1302.4389, 2013. 33