Learning how to divert attention in multilingual machine translation to mitigate gender bias

Bachelo ’s hesis - Enginee ing Physics ETSETB - Uni e si a Poli ècnica de Ca alunya Lea ning how o Di e A en ion in Mul ilingual Machine T ansla ion o Mi iga e Gende Bias Au ho : Pablo C uce a Ba e o P ojec supe iso s: Ma a Ruiz Cos a-jussà and Ca los Escolano Peinado June o 2021 Abs ac Deep lea ning has a oused o e he pas yea s as a p omising echnique in AI. Implemen ing algo i hms ha allow sys ems o lea n wi h applica ions in se e al ields has made a di e ence wi h he con en ional ule coding ha had mul iple limi a ions. Since success ul expe imen s wi hin his scope ha e aken place, he limi s o he de elopmen o in elligen machines seem o ha e aded ou . Howe e , his is s ill no a lawless p ocedu e: he e is a g ea numbe o imp o emen s ahead o be pe o med in o de o ob ain be e esul s. One o hese issues is he p esence o s e eo ypes in la ge da a se s, a gene alized p oblem in many applica ions o Deep Lea ning. In he case o Neu al Machine T ansla ion, his phenomenon leads o gende -biased inaccu a e ansla ions. In his hesis, ou commi men is o wo k on de eloping a modi ica ion o he a en ion mechanism (wi hin he a chi ec u e o NMT) ha can mi iga e such bias. A en ion con ols he amoun o con ex ha is used in Neu al Machine T ansla ion. The mo i a ion o his wo k is ha he a en ion mechanism o di e en ypes o wo ds, e.g. wo ds wi h di e en amoun s o dependencies, should be di e en . The concen a ion o di e sion o con ex in o ma ion bo h a encoding and decoding le els may di e depending on he s udied biases and language. We s udy he e ec o con olling his amoun o a en ion bo h by modi ying he a en ion mechanism a he encoding and decoding le el, o sel and in e -a en ion. We e alua e his o he case o mi iga ing biases in a mul ilingual se ing on a ecen NMT a chi ec u e: he T ans o me . The esul s show a simila quali y o ansla ion as he baseline model in e ms o accu acy, bu ou pe o ms he e e ence model in nea ly all gende -bias indica o s, yielding esul s ha a e encou aging o con inuing explo ing his line o esea ch. 2 Con en s Abs ac 2 Con en s 3 Acknowledgemen s 5 1.In oduc ion 6 1.1.Con ibu ions 6 1.2.Thesis o ganiza ion 7 2.Theo e ical amewo k 8 2.1.Fundamen als o Deep Lea ning 8 2.2.Neu al machine ansla ion (NMT) 9 2.3.Re iew o he s a e-o - he-a 10 2.3.1.CNN-based a chi ec u es 10 2.3.1.1.By eNe 10 2.3.1.2.Con olu ional Seq2Seq 10 2.3.2.An app oach o a en ion 11 2.3.3.The T ans o me 12 2.3.3.1.O e all model a chi ec u e 12 2.3.3.2.Mul i-head a en ion 14 2.3.3.3.Scaled Do -P oduc A en ion 14 3.Me hodology 19 3.1.How o modi y a en ion? An adap i e empe a u e pa ame e 20 3.2.Expe imen al implemen a ion 23 4.Expe imen s 25 4.1.Da ase s and P ep ocessing 25 4.1.1.BLEU sco e 25 4.1.2.WinoMT 25 4.2.Baseline model 26 4.2.1.Model se ings 26 4.2.2.Resul s 26 3 4.3.Fi s expe imen 29 4.3.1.App oach 29 4.3.2.Resul s and conclusions 29 4.4.Second expe imen 31 4.4.1.App oach 31 4.4.2.Resul s and conclusions 31 4.4.3.T ansla ion examples 34 5.Conclusions, u he esea ch and pe sonal e alua ion 36 Bibliog aphy/Re e ences 37 4 Acknowledgemen s Fi s o all, I would like o hank my p ojec supe iso s: Ma a Ruiz Cos a-jussà o b inging me he oppo uni y o being pa o he esea ch g oup and ge ing in o a wo ld ha I was looking o wa d o explo ing (Deep Lea ning) despi e knowing abou my sca ce backg ound in he ield, as well as en iching my p ojec wi h he as expe ience doing esea ch; and Ca los Escolano Peinado, o o e ing me a helping hand when I needed i he mos and being always he e o b inging me assessmen on p og amming (which equi es no only knowledge in compu e science bu also ons o pa ience), apa om p o iding me wi h nume ous ideas when some expe imen s ailed. I also would like o hank Ch is ine Bas a, a membe o he esea ch g oup, o helping me as well in con using momen s ega dless o no being one o he p ojec supe iso s. My colleagues (and iends) G aciela Ojeda and Júlia Sánchez ha e also made con ibu ions in his hesis, in he o m o mo al suppo and sha ing common ideas. Finally, I would like o hank my pa en s’ ask: al hough hey had no idea o he heo y behind his p ojec and hey s uggle wi h English, hey ha e been willing o ead his epo and ha e always eminded me ha I can go as a as I wan . 5 1.In oduc ion We a e li ing swee imes o he de elopmen o AI. Wi hin he pas decade, he e ha e aken place se e al success ul expe imen s wi hin he scope o AI and, specially, o Deep Lea ning. The g ow h ha his ield is expe iencing and he ac ha he e a e being implemen ed endless applica ions o he echniques used in i makes us hink ha A i icial In elligence has come o s ay. Howe e , he e is a lo ye o be explo ed. One he one hand, he e a e applica ions ahead o be disco e ed in which AI could play an in e es ing ole. On he o he hand, despi e he imp o emen s in he s a e-o - he-a echniques, he e is s ill an in e es ing wo k o be done o boos ing up some impe ec p ocedu es ha conce n b anches o AI such as Machine Lea ning (and Deep Lea ning). Among he lis o non- lawless applica ions o AI he e is Machine T ansla ion (MT), a sub ield o Na u al Language P ocessing (NLP), which is specially being de eloped hanks o he b eak h ough o Neu al Machine T ansla ion (NMT). The e is a cons an appea ance o new NMT a chi ec u es ying o de h one p e ious models in e ms o ansla ion accu acy, compu a ional cos o he ansla ion o model aining ime, which is a p oo o he exis ence o nume ous possible imp o emen s ha could ake place wi hin his scope. 1.1.Con ibu ions One o he impe ec ions ha we ha e ound is he p esence o biases in NMT models, and pa icula ly, gende bias. Mos models a e ained wi h la ge da ase s composed o millions o sen ences in which he male gende is some imes employed as he neu al gende (de aul ) and which mos ly p ese e gende s e eo ypes. This phenomenon causes ansla ions no o be neu al when i comes o gende . In his wo k, we will y o ackle his p oblem by implemen ing modi ica ions in a NMT a chi ec u e called The T ans o me [1], which elies en i ely on he concep o a en ion. In pa icula , we a e due o modi y he a en ion mechanism (bo h sel -a en ion and c oss-a en ion) ha his model makes use o a he encoding and decoding le els in a mul ilingual se ing. We belie e ha his will no only help o mi iga e biases bu also will imp o e he quali y o he ansla ion. 6 1.2.Thesis o ganiza ion This wo k is o ganized acco ding o he o de o appea ance o he necessa y concep s o unde s and he ollowing sec ions. Fi s o all, we will e iew he basic concep s behind he ield o ou app oach, as well as we will e iew he s a e-o - he-a models and echniques ha we will employ in he expe imen s. A e ha , we will p esen he p oposed modi ica ions in he a en ion mechanism, how hey wo k and how we will implemen hem. Finally we will co e he baseline model and he esul s ob ained bo h in his e e ence model and ou expe imen al implemen a ions, bo h in e ms o ansla ion accu acy and gende bias esul s. To ge a be e unde s anding o wha is o be implemen ed, we mus i s in oduce a se o heo y undamen als ha a e desc ibed in he nex sec ion: 7 2.Theo e ical amewo k 2.1.Fundamen als o Deep Lea ning Machine lea ning is a sub- ield o AI which has as a goal o p o ide compu ing sys ems wi h he abili y o pe o m ce ain asks (classi ica ion, eg ession, e c.) h ough lea ning. In his p ocess, he machine is ed wi h examples ha will help i o imp o e in i s ask. Wi hin his ield, we can ind Deep lea ning, a se o algo i hms which aims o co ec ly model asks ha equi e a high deg ee o abs ac ion. The basic uni o wha his algo i hm uses is he pe cep on. The pe cep on is a ma hema ical modeling o a biological neu on ha maps a se o inpu s (x1, x2,…, xN) o an ou pu yin he ollowing way: (1) Whe e w1,w2,…,wNa e a se o weigh s, w0is wha we e e o as bias and is ou ac i a ion unc ion (up o ou choice, bu some ac i a ion unc ions like anh on ReLU a e commonly used). Deep lea ning uses cascading laye s o pe cep ons, building a complex s uc u e ha is called Neu al Ne wo k. In be ween each laye , he e is wha we call a hidden s a e. Figu e 2.1: an example o a simple neu al ne wo k wi h h ee laye s, ou di e en inpu s, one ou pu and a hidden s a e consis ing o i e elemen s. Ci cum e ences ep esen pe cep ons, while he a ows ep esen he connec i i y be ween hem. 8 Neu al ne wo ks p oduce ou pu s ha will i ou ask be e o no depending on he alues o he weigh s ha a e assigned. In o de o op imize he alues o he weigh s, we mus de ine a loss unc ion ha will be he ma hema ical c i e ion o he machine o op imize he weigh s. The algo i hm used o modi y he weigh s while ensu ing he dec ease in he loss unc ion is ypically g adien descen . Since he e a e in ini e con igu a ions o neu al ne wo k s uc u es, se e al ypes o neu al ne wo ks ha e been desc ibed. In he ‘Re iew o he s a e-o - he-a ’ sec ion, we will men ion some o hem ela ed o a concep ha is o ou in e es : Neu al Machine T ansla ion. 2.2.Neu al machine ansla ion (NMT) The ield ha we a e wo king on is Na u al Language P ocessing (NLP). NLP e e s o he b anch o compu e science, and mo e speci ically, o he b anch o a i icial in elligence (AI), which deals wi h gi ing compu e s he abili y o unde s and spoken ex and wo ds in he same way as human beings. NLP combines compu a ional linguis ics ( ule-based modeling o human language) wi h s a is ical, machine lea ning, and deep lea ning models.[2] Wi hin his ield we can ind Neu al Machine T ansla ion (NMT), which is he machine ansla ion ask (gene a ing au oma ic ansla ions gi en a sou ce ex in a ce ain language o ano he language) ha makes use o a i icial neu al ne wo ks o ying o p edic wha he bes possible ou pu is. Se e al NMT a chi ec u es ha e been desc ibed, mos o hem in he e y ew pas yea s. We a e due o explain some o he cu ing-edge a chi ec u es ha ou pe o med hei p edecesso s. 9 So max A e ha , a so max unc ion is applied. So max is a unc ion commonly employed in neu al ne wo ks, specially in he las s eps, ha is used o mapping a se o inpu alues o a no malized se o ou pu s o he same size. This can be seen as he c ea ion o a p obabili y dis ibu ion om an o iginal se o compa ibili y sco es whose o de o magni ude is a bi a y1. The o mula o his unc ion is de ined as ollows: (3) The nume a o is s ic ly posi i e and inc easing wi h xigi en he na u e o he exponen ial unc ion and he denomina o is a cons an alue g ea e o equal han he nume a o . This means ha ega dless o he sign o he inpu , each o he componen s o he ou pu will be wi hin a bounded domain be ween 0 and 1, and i is i ial o see ha he sum o all componen s o he ou pu yields 1. In addi ion, he bigge a componen is (compa ed o he o he s), he mo e likely i becomes. In he p e ious s ep o his p ocess we commi ed o explain he impo ance o he scaling ac o . The main eason why his di ision is pe o med elies on he exponen ial beha io o he so max unc ion. The au ho s o his a en ion unc ion a gue ha he inpu sequences (que ies and keys) can be modelled as andom signals o mean 0 and a iance 1, and he e o e, a scala p oduc be ween wo dk-sized ec o s would yield a andom ou pu o mean 0 and a iance dk.[1] Fo la ge alues o dk(i.e. long sen ences), he a iance g ow h would cause some inpu componen s o he so max o end o ge signi ican ly la ge han o he s. This phenomenon would be mo e p onounced a e he applica ion o he so max ega ding he exponen ial na u e o his unc ion. The e ec o a so max on a andom inpu signal as he one men ioned be o e wi h and wi hou he scaling ac o can be obse ed in Figu e 2.4: 1In ou case, he o de o magni ude will no be a bi a y bu will be acco ding o a p obabili y dis ibu ion wi h pa ame e s ha will be desc ibed la e on 16 Figu e 2.4: simula ion o he ou pu o a so max unc ion applied o e a se o andom compa ibili y sco es o a wo d and a sequence o leng h N=20, wi h and wi hou he e ec o he scaling ac o In Figu e 2.4 we simula e he e ec o he so max upon a se o compa ibili y o sco es o a wo d wi h he wo ds in i s con ex , which co espond o he i s subplo . The ou pu p obabili ies o he so max unc ion wi hou including he scaling ac o dk(classical do -p oduc a en ion) a e depic ed in he subplo in he middle, while he las subplo shows how his ac o a ec s he ou pu p obabili ies o he so max. I can be obse ed ha a e applying he scaling he a en ion acqui es a less s eep p o ile, which means ha a en ion has been di e ed. We place so much emphasis on his because ou me hodology will ake ad an age o his phenomenon, as i will be explained in u he de ail in 3.1. In he case o he masked mul i-head a en ion, which is only used in he decode , masking is pe o med be o e he so max unc ion. Remembe ha he eason o doing his is o p e en om a ending o subsequen posi ions o he ou pu (wo ds ha ha e no been de e mined ye ). This is done by illing he compa ibili y sco es o elemen s o he ou pu co esponding o u u e ime s eps wi h sen inel alues ending o -∞, so he a en ion sco e becomes 0 a e he so max. 17 Ma ix mul iplica ion Finally, he ou pu o he so max unc ion is mul iplied by he ma ix Vusing ano he simple ma ix mul iplica ion. The e o e, he Scaled Do -P oduc A en ion module in ac p oduces a weigh ed sum o alue p ojec ions o wo d embeddings, in which weigh s a e he p obabili ies o hese alues o be he nex elemen o he ou pu sequence. 18 3.Me hodology Rega ding he main pu pose o his hesis, which is educing gende bias o machine ansla ion, we would like o ob ain he desi ed accu a e ansla ions o sen ences in which he e could be s e eo ypes. The p esence o such bias is mos ly gi en o ce ain wo ds in he ansla ion om languages ha do no speci y he gende o he pe son/animal in he wo d i sel bu in he con ex o languages ha use a di e en wo d o each gende . Fo ins ance, he English wo d nu se could be ansla ed in o Spanish as en e me o (masculine wo d o nu se) o en e me a ( eminine noun). Howe e , his ansla ion, i gene a ed au oma ically, will end o p oduce a emenine noun since his p o ession is adi ionally associa ed wi h he eminine gende , some imes igno ing he hin s in he con ex sugges ing ha his pe son migh be a boy. Rega ding ha a en ion e e s o he amoun o con ex used o gene a ing his ansla ion, we belie e ha using he app op ia e a en ion, he model would be able o de ec he gende o he subjec by ocusing on key wo ds. Le us see ano he example o wha a di e ed o ocused a en ion is, his ime ela ing a en ion wi h gende . Fo example, he sen ence “The doc o asked he nu se i he could help he .” con ains wo p onouns ha p o e ha he doc o is a woman and he nu se is a man. Howe e , he adi ional gende oles o hese p o essions a e he o he way a ound. A ocused a en ion would p oduce an ou pu ha ends o be mo e simila o a wo d-by-wo d ansla ion, and o he wo ds ‘doc o ’ and ‘nu se’ in pa icula , he model would end o ansla e hem as he mos likely ou pu o hose single wo ds, which co espond o he a ge s ha ha e been epea ed he mos o he model in he aining p ocess. Since we a e wo king wi h eno mous da ase s which p ese e gende s e eo ypes, a ocused a en ion mechanism will p oduce, o ins ance, he ollowing sen ence o English-Spanish ansla ion: “El doc o pidió a la en e me a si él podía ayuda la.” In his example, he model has no ela ed he nouns o he p onouns in he con ex , yielding a misin e p e a ion o he gende s o bo h p o essionals. Howe e , a mo e di e ed a en ion would ake in o accoun hese p onouns o he ansla ion o he nouns, which would help us ob ain a mo e accu a e ansla ion like: “La doc o a pidió al en e me o si podía ayuda la.” Ne e heless, an excess o di e sion is no some hing posi i e ei he : o e - ela ing wo ds ha ha e no hing o do wi h each o he would p oduce w ong ou pu s wi h comple ely senseless meanings. The e o e, we a e due o modi y he a en ion mechanism in an in elligen way in o de o p oduce be e esul s. 19 3.1.How o modi y a en ion? An adap i e empe a u e pa ame e In he p e ious sec ion, we ma hema ically desc ibed a en ion as a p obabili y sco e ha is compu ed o e e y wo d acco ding o how much i does ela e o he o he s. F om now on, we will call he ou pu o he las s ep o he compu a ion o he unc ion (so max) a en ion weigh s. Ou main idea s ands ha including a empe a u e pa ame e o he a en ion weigh s could egula e he amoun o con ex ha should be used o p oducing he ou pu gi en a ce ain inpu . Recalling he o mula o Scaled Do -P oduc A en ion, he e ec o his pa ame e will be implemen ed as ollows: (4) Whe e 𝜏, he empe a u e pa ame e , is calcula ed as: (5) Being some ac i a ion unc ion and wij an elemen o a se o lea ned weigh s. A di e en empe a u e ac o is compu ed o each sen ence in he inpu sequence. The jus i ica ion o his elies on he idea ha such a pa ame e could help he model edis ibu e he a en ion in a simila way ha he scaling ac o applied be o e he so max does. As shown in Figu e 3.1, a highe empe a u e will p oduce smalle g adien s han hose scaled by , leading o smoo he a en ion dis ibu ions. Simila ly, a low empe a u e will 1 𝑑𝑘 gene a e s eep peaks. 20 Figu e 3.1: simula ion o he e ec o including a empe a u e pa ame e on he a en ion weigh s o a andom se o wo ds and di e en empe a u es a ound 1 ( aw a en ion weigh s). Wo ds a e ep esen ed along he x-axis, empe a u e changes along he y-axis and he a en ion weigh s a e shown in he e ical di ec ion. Figu e 3.1 shows he a en ion weigh s (z-axis) ela ing a ce ain wo d wi h he N=20 wo ds in i s con ex ( ep esen ed along he x-axis) o di e en empe a u es, which a y along he y-axis. Wha is o pa icula in e es om his g aph is o compa e how a en ion changes compa ing i o he special case o 𝜏=1 ( he empe a u e e ec does no modi y he a en ion weigh s). As i can be obse ed, he wo d(s) ha we e mos ele an o he ansla ion o he wo d ha we a e e e ing o become e en mo e impo an when empe a u e dec eases (a en ion becomes mo e ocused). Howe e , as he empe a u e inc eases, he a en ion becomes mo e plana , which means ha i is di e ed. Despi e imp o ing he esul s o long sequences, we belie e ha his single alue migh 1 𝑑𝑘 no be he op imal scaling ac o a all imes: ou hypo hesis s a es ha o some cases i will be be e o ei he concen a e o di e a en ion, and ou con ol mechanism o do i will be 𝜏. Since we a e s ill unable o p edic wha he op imal alue o 𝜏is, we will y o compu e i using a i icial in elligence: aking as an inpu , a laye o a neu al ne wo k will p oduce 21 a calcula ion o a empe a u e pa ame e ha will be op imized using a ypical Deep Lea ning algo i hm, g adien descen . 22 3.2.Expe imen al implemen a ion E e y inpu sen ence is ep esen ed by a ma ix: e e y ow would ep esen he embedding o a speci ic wo d. In p ac ice, he inpu ex o he model is a enso o o de h ee, o which a new dimension is added so i con ains he in o ma ion o a di e en sen ence in each o he componen s[1]. ●Each componen o he i s dimension ep esen s a di e en sen ence. The e o e, i s size is called Ba ch size. ●The second dimension is ela ed o he numbe o wo ds pe sen ence. Since i is mo e p ac ical o use cubic da a s uc u es, all sen ences mus ha e he same leng h. To coun e ac his obs acle, a padding e ec is used in o de o lea e blank spaces in ph ases, le ing us deal wi h sen ences o di e en sizes. ●Finally, he hi d dimension is he wo d embedding, he size o which is h·dk=h·d =dmodel . Recall ha we a e wo king wi h eigh di e en heads, each o which uses a d -sized ec o o ep esen a wo d. These heads a e packed oge he in he same ma ix conca ena ed in a single dimension. Because we wan a empe a u e pa ame e o each o he sen ences, he ou pu ha we wan o p oduce be o e di iding by ou ac o is in ac a ec o o leng h Ba ch size. Then, each ma ix ep esen ing a sen ence (each elemen in he i s dimension is a ma ix ha s o es he embedding o a sen ence o i s p ojec ion) mus p oduce a scala pa ame e . How can his be implemen ed? By now, ou assump ion is ha his empe a u e ac o can be calcula ed as a weigh ed linea sum o each elemen in he ma ix, and a e ha , se o an ac i a ion unc ion. These weigh s would be lea ned o op imal esul s, so in ac , we could see hese ope a ions as he e ec o a single laye o a neu al ne wo k applied o a ec o ha esul s om he conca ena ion o ows o laye s. One o he d awbacks in his p ocedu e is he ac ha he size o he laye is compu ed as he p oduc be ween he size o he second dimension (leng h o sen ences) and he size o he hi d one (leng h o he wo d embedding o each head), bu i mus be kep cons an as well. Since hese wo dimensions depend on non- ixed pa ame e s, we should ind a way o u n a ma ix o a iable dimensions o a ixed-size inpu ma ix. A way o do his is using an Adap i e Pooling il e in 2D, which is an e ec i e me hod ha uses disc e iza ion by selec ing alues among a se o componen s. The alues ha a e p oduced could be selec ed by ei he selec ing max alues o pe o ming an a e age calcula ion on hem. The disad an age o doing his is ha , a e he pooling il e , some in o ma ion will be copied in case o sho e inpu s han he ou pu ma ix o missing in case o longe sequences. The o e all p ocedu e ha we will implemen is depic ed in Figu e 3.2: 23 Figu e 3.2: schema ic ske ch showing he main s eps o ou implemen a ion In Figu e 3.2 we in end o ep esen he s eps desc ibed abo e. Each colo ed g id o he inpu ep esen s he a en ion weigh s a a ce ain ime s ep o a sen ence. F om hese ma ices we ob ain squa e ma ices wi h ixed size ( ep esen ed by he 5x5 g id s ack) by pooling hem. Then, a linea laye compu es a weigh ed sum o each o he elemen s, which is hen se o an ac i a ion unc ion. The ou pu o his unc ion (deno ed by ) will be ou empe a u e pa ame e . 24 4.Expe imen s 4.1.Da ase s and P ep ocessing Fo ou expe imen s we will use he Eu opa l 7 co pus[11] as he aining se o sen ences, a da ase con aining millions o p ep ocessed examples o sen ence ansla ion be ween he main Eu opean languages. Fo he alida ion and es ing we will use new es 2012 and new es 20132da ase s. All da a will be p ep ocessed using s anda d Moses[12] sc ip s. Mo e conc e ely, he languages o ou choice will be English, F ench, Ge man and Spanish. We will ou line he esul s o he ask o all language pai s (bo h a sou ce and a ge languages) using he BLEU sco e as ou c i e ia o he quali y o he ansla ion. In addi ion, we will employ he WinoMT[13] algo i hm o de ec ing biases, using WinoBias[14] and Winogende [15] as he da ase s o hese c i e ia. 4.1.1.BLEU sco e Bilingual e alua ion unde s udy, commonly known as BLEU, is a me hod employed o measu ing he quali y o a ansla ion om a ce ain sou ce language o ano he .[16] This algo i hm compu es he numbe o co ec ly p edic ed ou pu s gi en a a ge sequence and a candida e sequence, he accu acy o which we wan o measu e. In ou case, we will employ BLEU-4, which akes in o accoun he p ecision o he ansla ion o each single wo d, wo d pai s and 3-g ams and 4-g ams o wo ds, as well as in oduces a penal y ac o o sho sen ences. 4.1.2.WinoMT WinoMT[13] is an algo i hm de eloped in o de o measu e gende biases aking as an inpu he ansla ion o h ee di e en ex da ase s w i en in English: a se o 3888 sen ences wi h equal p esence o masculine and emenine gende s and some neu al nouns ( om now on, neu al), a se o p o-s e eo ypically biased sen ences ( om now on, p o) and he an i-s e eo ypically e sion o he same se ( om now on, an i). I consis s o he e alua ion o h ee pa ame e s: ●Accu acy: measu es he pe cen age o gende s p edic ed co ec ly om he neu al da ase in he a ge language. ●ΔG: compu es he di e ence be ween he F1sco es ( ha is, a balanced pa ame e ha akes in o accoun bo h p ecision and ecall) o male and emenine nouns. ●ΔS: compu es he di e ence be ween he accu acies o p o and an i. 2A ailable a h p://www.s a m .o g/ 25 Howe e , hings change when he empe a u e is only applied in he encode . In his case, he bes esul s ha e been ob ained o he con igu a ion which combined he a e aging pooling il e wi h he bigge laye (sized 529). A e 14 epochs, he alid loss becomes 59.6279, which is a alue ha ep esen s a sligh inc ease o 0.51% wi h espec o he bes loss in he baseline model. We belie e ha his alue would ha e ou pe o med he baseline loss i i had been ained o as many epochs.3 The BLEU4 sco es ob ained o all language pai s a e p esen ed in Table 4.3. Ta ge language EN ES FR DE A e age EN - 29.48 29.31 21.72 26.84 ES 27.39 - 29.89 19.99 25.76 FR 26.14 29.06 - 19.26 24.82 DE 23.95 24.86 25.23 - 24.68 A e age 25.83 27.80 28.14 20.32 25.52 Table 4.3: BLEU4 sco e ob ained in ou second expe imen , using 𝜆=4, an a e aging pooling il e and a laye size o 529 A g aphical compa ison be ween he baseline and his model esul s is shown in Figu e 4.2. 3The aining p ocess did no each as many i e a ions as in 4.2.2 due o echnical issues o he machine. 32 Figu e 4.2: compa ison be ween he BLEU4 sco es ob ained bo h in he baseline and his expe imen As one can obse e om Table 4.3 and Figu e 4.2, ou model shows a sligh ly lowe pe o mance han he baseline model, yielding an a e age BLEU4 0.35 below han ou e e ence alue. Howe e , his di e ence is no much signi ican ega ding he ac ha he wo models ha e been ained o di e en imes.4 Le us now see he esul s o he WinoMT sco es o his model in Table 4.4. Accu acy ΔG ΔS ES 53.8% 15.7 6.2 FR 43.8% 17.5 16.4 DE 57.4% 1.1 0.5 A e age 51.7% 11.4 7.7 Table 4.4: WinoMT sco e ob ained in ou baseline model. Accu acy deno es he pe cen age o co ec ly p edic ed gende s in he ansla ion, ΔG deno es he di e ence be ween masuline and emenine F1sco es and ΔS deno es he di e ence be ween he p o-s e eo ypical and an i-s e eo ypical gende assignmen s. The unde lined esul s a e hose whose alues ou pe o m he baseline model. 4In addi ion, he BLEU4 sco e is subjec o he a ge sen ence used as a e e ence in he algo i hm and p o ides unde es ima ed esul s o candida e sen ences which con ain he same meaning as he sou ce bu di e in o m om he a ge . We will see la e ha his phenomenon has occu ed when es ing his model. 33 The compa ison be ween hese alues and hose ob ained in 4.2.2 a e shown in Table 4.5: Baseline Expe imen al model Di e ence (%) Accu acy (A g.) 52.6% 51.7% -1.7% ( ela i e o 52.6) ΔG (A g.) 15.4 11.4 -26.0% ΔS (A g.) 9.8 7.7 -21.4% Table 4.5: compa ison be ween he a e age WinoMT sco es ob ained in ou baseline model and he expe imen al model. We obse e ha ou model does mos ly ou pe o m he baseline in e ms o gende . The alues o ΔG and ΔS, which a e ou gende bias indica o s, ha e dec eased in a e age wi h espec o he baseline o all languages (excep o ΔG o F ench). Howe e , gende p edic ion accu acy is on a e age sligh ly lowe han he baseline model. The conclusion ha we d aw om hese ac s is ha ou model has no been able o ou pe o m he baseline in he o e all quali y o ansla ion, bu he modi ica ions ha ha e been implemen ed in he a en ion mechanism ha e somehow con ibu ed o he debiasing o he o iginal model. Ou p o o ype does no pay as much a en ion o he s e eo ypes in he aining da ase as he e e ence model, which is an indica o ha we ha e ul illed ou ini ial commi men o his wo k. As an addi ional no e, i would ha e been pleasing o ha e had he model es ed a e being ained o as long as he baseline. 4.4.3.T ansla ion examples In his sec ion we a ach signi ican en-es and es-en ansla ions ha would help us p o e ha he model wo ks co ec ly despi e he BLEU4 sco e, which is no a pe ec algo i hm since using synonyms is penalized ega dless o he ac ha i migh be a success ul ansla ion. 34 Sou ce sen ence Ta ge sen ence Hypo hesis desde es e úl imo pun o de is a , odas las ecnologías ac uales no son lo su icien emen e buenas pa a noso os ; es o da más abajo a los diseñado es de algo i mos . om he la e poin o iew , all cu en echnologies a e no good enough o us - his adds wo k o he algo i hm designe s . om his las poin o iew , all he cu en echnologies a e no good enough o us ; his gi es mo e wo k o algo i hms designe s . No way has handled i s oil weal h e y ca e ully - all bu a small pe cen age o money om he indus y is in es ed in a special und o he bene i o u u e gene a ions . No uega ha ges ionado su iqueza pe olí e a con mucha cau ela ; odo , excep o un pequeño po cen aje del dine o p oceden e de es a indus ia , se in ie e en un ondo especial en bene icio de las u u as gene aciones . No uega ha ges ionado muy cuidadosamen e su iqueza pe ole a : odo menos un pequeño po cen aje de dine o del sec o se in ie e en un ondo especial en bene icio de las gene aciones u u as . he summi also concluded wi h he join commi men o Chile and Pe u o accep a uling by he Hague Cou o adjudica e a bo de dispu e be ween he wo coun ies la cumb e concluyó ambién con el comp omiso conjun o de Chile y Pe ú de acep a un allo de la Co e de La Haya que di ima un di e endo on e izo en e ambos países . la Cumb e concluyó ambién con el comp omiso común de Chile y Pe ú de acep a una sen encia del T ibunal de La Haya pa a zanja una dispu a on e iza en e los dos países . las eo ías libe ales y los medios de comunicación a i man has a la saciedad que el es ado no debe ía apo a capi al a la economía , que di igi la economía conduce a su acaso . Libe al heo y and he Media incessan ly claim ha he S a e may no pa icipa e wi h capi al in i s own economy , and ha a con olled economy leads o economic uin libe al heo ies and he media claim ad nauseam ha he s a e should no b ing capi al o he economy , ha leading he economy leads o i s ailu e . Table 4.6: ansla ion examples be ween English and Spanish 35 5.Conclusions, u he esea ch and pe sonal e alua ion The i s hing ha we ha e been able o app ecia e no in he expe imen s bu in he e iew o he s a e o he a is he ac ha a en ion is a powe ul mechanism. In ac , i is so solid ha some cu ing-edge NMT models ely en i ely on i . Howe e , imp ecisions also a ise in he ask o ansla ion using hese a chi ec u es, being gende bias one o hese impe ec ions. Th oughou his wo k, we ha e p esen ed possible modi ica ions in he a en ion mechanism used in The T ans o me ha could help he model neu alize his inaccu acy using Deep Lea ning echniques in a mul ilingual se ing. The mos signi ican esul s a e ob ained when modi ying he a en ion in he decode , o he case o employing a e aging pooling il e s and big laye sizes. In ou bes se up, despi e a sligh d op in he BLEU sco e wi h espec o he o iginal model, a dec ease in he gende bias indica o s akes place, which is a symp om ha he app oach ha we ha e p esen ed is in he igh di ec ion. Mo e conc e ely, we ha e managed o dec ease he a e age ΔG sco e by 26% and he a e age ΔS sco e by 21.4% o Ge man, F ench and Spanish. I is impo an o poin ou ha he aining p ocess o his la e model did no go as a as he baseline which we ha e compa ed ou esul s wi h. As a u u e esea ch, i would be in e es ing o see how his model pe o ms a e ca ying ou he whole aining. O he in iguing possible modi ica ions could be ying o he alues o 𝜆, which has emained cons an in all execu ions o ou second expe imen , o inding ou wha he main eason o he quali y and loss d ops in he implemen a ions a he decode is. This has been a ewa ding p ojec ega dless o he esul s, which I conside o be good despi e he di icul ies. I had ne e en e ed he wo ld o Machine Lea ning (and e en less o Deep Lea ning) be o e, no in he ield o NLP, and he e o e he wo ld o NMT has been o ally new o me. Also, I had ne e wo ked wi h Py hon un il I s a ed his p ojec . The lack o knowledge o all o hese hings ha e made me spend conside able ime ca ching up wi h he necessa y backg ound o he p ojec . The ac ha he emo e machine in which we we e doing he expe imen s collapsed se e al imes du ing he aining p ocess (which akes days o comple e) did no help ei he . Tha is why I conside ha i has no been an easy job o me and ha he lea ning p ocess ha I ha e expe ienced has been in ense. Bu , as I ha e said, I ha e lea n so many hings ha we e new o me ha ha e made om his expe ience a wo hwhile one. I look o wa d o con inuing lea ning mo e abou AI and, in pa icula , he applica ions o Deep Lea ning so I acqui e su icien knowledge o ca ying ou ambi ious esea ch like his p ojec on my own. 36 Bibliog aphy/Re e ences [1] A. Vaswani, N. Shazee , N. Pa ma , J. Uszko ei , L. Jones, A. N. Gomez, L. Kaise and I. Polosukhin, A en ion is all you need. a Xi :1706.03762, 2017. [2] IBM, Na u al Language P ocessing. [Consul a ion da e: 02-06-2021, a ailable a h ps://www.ibm.com/cloud/lea n/na u al-language-p ocessing] [3] W. Zhang, K. I oh, J. Tanida and Y. Ichioka, Pa allel dis ibu ed p ocessing model wi h local space-in a ian in e connec ions and i s op ical a chi ec u e. Appl. Op . 29, 4790-4797, 1990. [4] N. Kalchb enne , L. Espehol , K. Simonyan, A. an den Oo d, A. G a es and K. Ka ukcuoglu, Neu al Machine T ansla ion in Linea Time. a Xi :1610.10099, 2016. [5] J. Geh ing, M. Auli, D. G angie , D. Ya a s, Y. N. Dauphin, Con olu ional Sequence o Sequence Lea ning. a Xi :1705.03122, 2017. [6] H. Lamba, In ui i e Unde s anding o A en ion Mechanism in Deep Lea ning. [Consul a ion da e: 17-04-2021, a ailable a h ps:// owa dsda ascience.com/in ui i e-unde s anding-o -a en ion-mechanism-in-deep-lea ning-6c9482aec 4 ]. [7] D. Bahdanau, K. Cho and Y. Bengio, Neu al Machine T ansla ion by Join ly Lea ning o Align and T ansla e. a Xi :1409.0473, 2014. [8] T. Luong, H. Pham and C. D. Manning, E ec i e App oaches o A en ion-based Neu al Machine T ansla ion. EMNLP 2015. [9] L. Weng, A en ion? A en ion! [Consul a ion da e: 19-04-2021, a ailable a h ps://lilianweng.gi hub.io/lil-log/2018/06/24/a en ion-a en ion.h ml]. [10] J. Alama , The Illus a ed T ans o me . [Consul a ion da e: 22-04-2021, a ailable a h ps://jalamma .gi hub.io/illus a ed- ans o me ]. [11] P. Koehn, Eu opa l: A pa allel co pus o s a is ical machine ansla ion. In MT summi , olume 5, pages 79–86, 2005. [12] P. Koehn, H. Hoang, A. Bi ch, C. Callison-Bu ch, M. Fede ico, N. Be oldi, B. Cowan, W. Shen, C. Mo an, R. Zens e al. Moses: Open sou ce oolki o s a is ical machine ansla ion. In P oceedings o he ACL: Demo Pape s, pages 177–180, 2017. [13] G. S ano sky, N. A. Smi h and L. Ze lemoye , E alua ing Gende Bias in Machine T ansla ion. a Xi :1906.00591, 2019. [14] R. Rudinge , J. Na adowsky, B. Leona d and B. Van Du me, Gende Bias in Co e e ence Resolu ion. In P oceedings o he ACL: Sho Pape s, pages 8-14, 2018. 37 [15] J. Zhao, T. Wang, M. Ya ska , V. O doñez, K. W. Chang, Gende Bias in Co e e ence Resolu ion: E alua ion and Debiasing Me hods. In P oceedings o he ACL: Sho Pape s, pages 15-20, 2018. [16] K. Papineni, S. Roukos, T. Wa d and W. J. Zhu, BLEU: a Me hod o Au oma ic E alua ion o Machine T ansla ion. In P oceedings o he 40 h Annual Mee ing o he ACL, pages 311-318, 2002. [17] C. Escolano, M. R. Cos a-jussà, J. A. R. Fonollosa, M. A e xe, Mul ilingual Machine T ansla ion: Closing he Gap be ween Sha ed and Language-speci ic Encode -Decode s. a Xi :2004.06575, 2020. [18] M. O , S. Eduno , A. Bae ski, A. Fan, S. G oss, N. Ng, D. G angie and M. Auli, ai seq: A Fas , Ex ensible Toolki o Sequence Modeling. In P oceedings o NAACL-HLT 2019: Demons a ions, 2019. [19] R. Mülle , S. Ko nbli h and G. E. Hin on, When does label smoo hing help? Pa o Ad ances in Neu al In o ma ion P ocessing Sys ems 32 (Neu IPS 2019), 2019. [20] D. P. Kingma and J. Ba. 2014. Adam: A me hod o s ochas ic op imiza ion. a Xi :1412.6980, 2014. 38