Depa amen o de Sis emas In o má icos y Compu ación
Uni e si a Poli ècnica de València
Online Lea ning in Neu al Machine
T ansla ion
MASTER THESIS
Más e en In eligencia A i icial, Reconocimien o de Fo mas e
Imagen Digi al
Au ho : Luis Ceb ián Chuliá
Tu o : F ancisco Casacube a Nolla
Ál a o Pe is Ab il
Cou se 2016-2017
Resum
La aducció de g an quali a es oba mol demanada en l’ac uali a . To i
que la aducció au omà ica o e ix unes p es acions accep ables, en alguns casos
no és su icien i és necessà ia la supe isió humana. Pe a acili a la asca de
aducció de l’humà, els sis emes de aducció au omà ica p enen pa en aques
p océs. Quan una no a o ació en el llengua ge o igen necessi a se aduïda,
es a s’in oduïx en el sis ema, el qual ob é com a eixida una hipò esi de aducció.
Lla o s, l’humà co egix aques a hipò esi ( ambé conegu com a pos -edi a ) pe a
ob ind e una aducció de majo quali a . Se capaços de ans e i el coneixemen
que l’ humà exhibix quan eali za la asca de pos -edició al sis ema de aducció
au omà ica és una ca ac e ís ica desi jable ja que s’ha demos a que un sis ema
de aducció mes p ecís ajuda a augmen a l‘e iciència del p océs de pos -edició.
Pel e que el p océs de pos -edició eque ix un sis ema ja en ena , les ècni-
ques d’ap enen a ge en línia són les adequades pe aques a asca. En es e eball,
es p oposen es algo i mes d’ap enen a ge en línia aplica s a un aduc o au-
omà ic neu onal en un escena i de pos -edició. Es os algo i mes es basen en
l’ap oximació en línia Passi e-Agg essi e en la qual el model s’ac uali za desp és
de cada mos a amb l’objec iu de compli un c i e i de co ecció al ma eix emps
que man é in o mació p è ia ap esa. L’objec iu és adap a i e ina un sis ema ja
en ena amb no es mos es al ol men e el p océs de pos -edició es du a e me
(pe an , el emps d’ac uali zació ha de man eni -se con ola ).
A més, es os algo i mes es compa en amb al es ben conegudes a ian s en
línia de l’algo i me de descens pe g adien es ocàs ic. Els esul a s mos en una
millo a en la quali a de les aduccions desp és d’aplica es os algo i mes, e-
duin així l’es o ç humà en el p océs de pos -edició.
Pa aules clau: Ap enen a ge en línia, T aducció au omà ica neu onal, Passi e-
Agg esi e
Resumen
La aducción de g an calidad es á muy demandada en la ac ualidad. A pesa
de que la aducción au omá ica o ece unas p es aciones acep ables, en algunos
casos no es su icien e y es necesa ia la supe isión humana. Pa a acili a la a ea
de aducción del humano, los sis emas de aducción au omá ica oman pa e en
es e p oceso. Cuando una nue a o ación en el idioma o igen necesi a se adu-
cida, es a se in oduce en el sis ema, el cual ob iene como salida una hipó esis de
aducción. El humano en onces, co ige es a hipó esis ( ambién conocido como
pos -edi a ) pa a ob ene una aducción de mayo calidad. Se capaz de ans e-
i el conocimien o que el humano exhibe cuando ealiza la a ea de pos -edición
al sis ema de aducción au omá ica es una ca ac e ís ica deseable pues o que se
ha demos ado que un sis ema de aducción mas p eciso ayuda a aumen a la
e iciencia del p oceso de pos -edición.
Debido a que el p oceso de pos -edición equie e un sis ema ya en enado, las
écnicas de ap endizaje en línea son las adecuadas pa a es a a ea. En es e aba-
jo, se p oponen es algo i mos de ap endizaje en línea aplicados a un aduc o
au omá ico neu onal en un escena io de pos -edición. Es os algo i mos se basan
iii
i
en la ap oximación en línea Passi e-Agg essi e en la cual el modelo se ac ualiza
después de cada mues a con el obje i o de cumpli un c i e io de co ección a
la ez que man eniendo in o mación p e ia ap endida. El obje i o es adap a y
e ina un sis ema ya en enado con nue as mues as al uelo mien as el p o-
ceso de pos -edición se lle a a cabo (po an o, el iempo de ac ualización debe
man ene se bajo con ol).
Además, es os algo i mos se compa an con o as bien conocidas a ian es en
línea del algo i mo de descenso po g adien e es ocás ico. Los esul ados mues-
an una mejo a en la calidad de las aducciones después de aplica es os algo-
i mos, educiendo así el es ue zo humano en el p oceso de pos -edición.
Palab as cla e: Ap endizaje en línea, T aducción au omá ica neu onal, Passi e-
Agg esi e
Abs ac
High quali y ansla ions a e in high demand hese days. Al hough machine
ansla ion o e s accep able pe o mance, i is no su icien in some cases and
human supe ision is equi ed. In o de o ease he ansla ion ask o he human,
machine ansla ion sys ems ake pa in his p ocess. When a sen ence in he
sou ce language needs o be ansla ed, i is ed o he sys em which ou pu s a
hypo hesis ansla ion. The human hen, co ec s his hypo hesis (also known as
pos -edi ing) in o de o ob ain a high quali y ansla ion. Being able o ans e
he knowledge ha a human ansla o exhibi when pos -edi ing a ansla ion o
he machine ansla ion sys em is a desi able ea u e, as i has been p o en ha a
mo e accu a e machine ansla ion sys em helps o inc ease he e iciency o he
pos -edi ing p ocess.
Because he pos -edi ing scena io equi es an al eady ained sys em, online
lea ning echniques a e sui ed o his ask. In his wo k, h ee online lea ning
algo i hms ha e been p oposed and applied o a neu al machine ansla ion sys-
em in a pos -edi ing scena io. They ely on he Passi e-Agg essi e online lea n-
ing app oach in which he model is upda ed a e e e y sample in o de o ul il
a co ec ness c i e ion while emembe ing p e iously lea ned in o ma ion. The
goal is o adap and e ine an al eady ained sys em wi h new samples on- he-
ly as he pos -edi ing p ocess akes place (hence, he upda e ime mus be kep
unde con ol).
Mo eo e , hese new algo i hms a e compa ed wi h well-s ablished online
lea ning a ian s o he s ochas ic g adien descen algo i hm. Resul s show im-
p o emen s on he ansla ion quali y o he sys em a e applying hese algo-
i hms, educing human e o in he pos -edi ing p ocess.
Key wo ds: Online Lea ning, Neu al Machine T ansla ion, Passi e-Agg esi e
Con en s
Con en s
Lis o Figu es ii
Lis o Tables ii
1 Mo i a ion 1
2 In oduc ion o MT 3
2.1 S a is ical machine ansla ion ...................... 3
2.1.1 Language model ......................... 4
2.1.2 T ansla ion model ......................... 5
2.1.3 Log-linea model .......................... 6
2.2 Assessmen ................................. 6
2.2.1 BLEU ................................ 7
2.2.2 TER ................................. 8
2.2.3 METEOR .............................. 8
3 Neu al machine ansla ion 11
3.1 Modelling language wi h neu al ne wo ks ............... 12
3.1.1 Con inuous wo d ep esen a ion: wo d embedding ..... 12
3.1.2 Dealing wi h sequences: ecu en neu al ne wo ks ..... 13
3.1.3 Dealing wi h con ex : bidi ec ional ecu en neu al ne wo ks 15
3.2 End- o-end ansla ion: encode -decode ................ 15
3.2.1 A en ion model .......................... 17
3.2.2 Decoding ansla ions: beam sea ch .............. 17
4 T aining neu al ne wo ks 19
4.1 The backp opaga ion algo i hm ..................... 19
4.1.1 Backp opaga ion h ough ime ................. 20
4.2 S ochas ic g adien descen ........................ 21
4.2.1 SGD wi h momen um ...................... 22
4.2.2 Adag ad .............................. 22
4.2.3 Adadel a .............................. 23
4.2.4 Adam ................................ 23
5 Online lea ning 25
5.1 Online lea ning amewo k ....................... 25
5.2 Passi e-Agg essi e online lea ning ................... 26
5.3 PA online lea ning applied o NMT ................... 27
5.3.1 Passi e-Agg essi e ia subg adien echniques ........ 28
5.3.2 Passi e-Agg essi e ia p ojec ed subg adien echniques . . 29
5.3.3 Passi e-Agg essi e ia SGD wi h egula iza ion ....... 29
6 Expe imen s and esul s 31
6.1 Expe imen al amewo k ......................... 31

i
CONTENTS
6.1.1 Task desc ip ion .......................... 31
6.1.2 So wa e .............................. 32
6.1.3 Co po a .............................. 32
6.1.4 NMT sys em ............................ 33
6.2 Resul s ................................... 34
6.2.1 Hype pa ame e con igu a ion ................. 34
6.2.2 Compa ison be ween OL algo i hms .............. 35
7 Fu u e wo k and conclusions 39
7.1 Fu u e wo k ................................ 39
7.1.1 Using o he loss unc ions .................... 39
7.1.2 Inclusion o OL s a egies in IMT ................ 40
7.2 Conclusions ................................ 40
Bibliog aphy 41
Lis o Figu es
2.1 Wo d aligmen ............................... 5
3.1 LSTM .................................... 14
3.2 Bidi ec ional ecu en neu al ne wo k ................. 15
3.3 Encode -decode ............................. 16
4.1 Backp opaga ion example ........................ 21
6.1 In luence o hype pa ame e s in PA algo i hms ............ 35
6.2 E ec o uning NMT sys ems in e ms o BLEU ............ 36
6.3 E ec o uning NMT sys ems in e ms o TER ............. 37
6.4 E olu ion o NMT sys ems ........................ 38
Lis o Tables
6.1 S a is ics o he Xe ox co pus ....................... 32
6.2 S a is ics o he Emea co pus ....................... 33
6.3 S a is ics o he TED co pus ....................... 33
6.4 G id sea ch alues ............................. 34
6.5 Bes con igu a ion PA ........................... 34
6.6 Op imiza ion hype pa ame e s ..................... 36
6.7 Resul s all asks .............................. 38
ii
CHAPTER 1
Mo i a ion
We ha e seen in mode n his o y how he imp o emen o anspo a ion and
elecommunica ions in as uc u e has had a big impac in he mo emen o peo-
ple. Now we a e mo e exposed o o he cul u es, p oduc s and wo ld iews han
e e be o e. As echnology imp o es, he access o any kind o in o ma ion has
become some hing we can’ li e wi hou . Howe e , di e en cul u es o en mean
di e en languages which implies he appea ance o obs acles when he need o
communica e is manda o y. Thus, he need o a ool ha helps us o o e come
his obs acle a ises. Machine T ansla ion (MT) in es iga es he use o so wa e o
ansla e ex o speech om one language in o ano he . I aims a p o iding he
bes possible ansla ion wi hou human assis ance.
This esea ch ield was bo n in 1947 when Wa en Wea e published a memo-
andum s a ing his belie s abou compu e ’s capabili y o ansla e one language
in o ano he using c ip og aphy, logic and linguis ic pa e ns. In he nex yea s
a lo o p ojec s eme ged in he Uni ed S a es, he So ie Union and Wes e n Eu-
ope bu he p ac ical esul s we e disappoin ing (Madsen,2009;Hu chins,2005).
This inally led o he publica ion o a epo om a special commi ee o med
by he Uni ed S a es named ALPAC (Au oma ic Language P ocessing Ad iso y
Commi ee) whe e hey s a ed ha he e was no u u e o good-quali y/cos -
e ec i e MT.
Du ing he 70’s he e was a new app oach o he MT ield ha elied on he
p emise ha a language is based on a se o g amma ical and syn ac ic ules. This
ule-based sys em needed a obus and ca e ully designed bilingual dic iona y o
linguis ic in o ma ion c ea ed by human expe s. The big e o ha his app oach
equi ed was a big issue and i was hen eplaced by he co pus-based sys ems.
A he same ime, scien is s ocused on de eloping ools ha would acili a e he
ansla ion p ocess a he han eplacing human ansla o s, leading o he de-
elopmen o ansla ion memo y (TM) and o he compu e assis ed ansla ion
(CAT) (Samson,2005) ools.
Co pus-based sys ems ely on a pa allel co pus o sen ences in a sou ce lan-
guage and i s ansla ion in a a ge language. Wi h enough numbe o samples,
hese sys ems can be ained o in e he ansla ions o new sen ences. Fo his
pu pose, s a is ical models a e applied o he ansla ion p ocess as hese a e good
a ex ac ing ele an in o ma ion om a se o ( ansla ion) examples.
1
8
In oduc ion o MT
Acco ding o he expe imen a ion made in (Papineni e al.,2002), N is se o 4,
and wn=1
N. The BLEU sco e anges om 0 o 1 being 1 he closes a candida e
ansla ion can be o he e e ence (iden ical in his case). This is he main measu e
we will use in he expe imen a ion sec ion.
TER
The ansla ion e o a e (TER) (Sno e e al.,2006) is ano he measu e used o
assess ansla ions. I measu es he minimum numbe o edi s needed o change
a hypo hesis so ha i exac ly ma ches he e e ence, no malized by he a e age
leng h o he e e ence (i we ha e mo e han one). I we ha e se e al e e ences
he TER sco e will co espond o he minimum numbe o edi s needed o ma ch
he closes e e ence.
The possible edi s include inse ion, dele ion and subs i u ion o single wo ds
as well as shi s o wo d sequences. A shi is jus a mo emen o con iguous
wo ds wi hin he hypo hesis. An impo an hing o no e is ha all edi s, includ-
ing shi s o any numbe o wo ds and dis ance ha e equal cos . In a pos -edi ing
scena io his measu e app oxima es he human e o equi ed o co ec a ans-
la ion p oduced by a MT sys em. Because we wan o educe he human e o
equi ed, we wan o achie e a low TER sco e.
METEOR
METEOR (Bane jee and La ie,2005) was designed o add ess some o he is-
sues ha BLEU in oduced, such as: he lack o ecall, he use o highe o de n-
g ams o model wo d o de o he lack o explici wo d-ma ching be ween ans-
la ion and e e ence. I is based on he unig am ma ching be ween he machine-
p oduced ansla ion and he human-p oduced e e ence ansla ion.
Gi en a pai o sen ences, METEOR gene a es an alignmen be ween he wo
s ings. In his con ex , an alignmen is a ma ching be ween unig ams, such ha
e e y unig am in each s ing maps o ze o o one unig am in he o he s ing.
In o de o gene a e ha alignmen he p ocess is di ided in wo phases, each o
which has di e en s ages.
In he i s phase, based on di e en c i e ia, di e en modules p oduce uni-
g am mappings be ween he wo s ings. The "exac " module p oduces ma ches
be ween wo unig ams i hey a e exac ly he same. The "po e s em" module
p oduces unig am ma ches i hey a e he same a e being s emmed. The "Wo d-
Ne synonymy" maps wo s ings i hey a e synonyms. This modules a e o de ed
by p io i y and a gi en ma ch will be chosen i s i i is possible o o m an align-
men and i i also has highe p io i y han o he ma ches p oduced by lowe
p io i y modules. Fo example, an exac ma ch will be always p e e ed.
In he second phase, he la ges subse o unig am mappings is selec ed such
ha he esul ing se cons i u es an alignmen as de ined abo e. I mo e han one
subse cons i u es an alignmen , his me ic selec s he one wi h ewe mapping
c osses. I bo h sen ences a e w i en one below he o he and a line is d awn

2.2 Assessmen
9
be ween he ma ching unig ams, a c oss is p oduced when hese lines in e sec
wi h ano he mapping.
Las ly, in o de o ob ain he METEOR sco e, an ha monic mean o p ecision
and ecall is compu ed o e he unig ams and penalized i hese unig ams a e in
di e en o de compa ed o he e e ence. A high sco e will mean high quali y
ansla ions.
CHAPTER 3
Neu al machine ansla ion
S a e-o - he-a MT sys ems ha e elied on he ph ase-based app oach o a long
ime. Howe e , a new neu al app oach has eme ged, being he i s echnology
ha has been able o challenge o me sys ems (Luong and Manning,2015;Jean
e al.,2015). Wi h he use o G aphic P ocesso Uni s (GPU), neu al machine
ansla ion (NMT) has been able o cope wi h he high compu a ional cos i e-
qui es o compe e wi h s a e-o - he-a ph ase-based sys ems (Ben i ogli e al.,
2016).
This ac eamed wi h he imp o emen s made o he encode -decode a chi-
ec u e (explained la e in his chap e ) such as he a en ion model, o he use o
ga ed ecu en uni s o cope wi h con ex in long sen ences, made NMT echnol-
ogy o ad ance by leaps and bounds. Apa om ha , NMT demons a ed i s
powe a he IWSLT12015 e alua ion campaign, whe e one o hese sys ems ou -
pe o med he up- o- hen s a e-o - he-a ph ase-based sys ems on he English-
Ge man ask, a pai o languages di icul due o he dispa i y in mo phology
and syn ax.
The nex yea , Google announced ha Google T ansla e made he leap o NMT
because neu al ne wo ks inc ease bo h luency and accu acy o i s ansla ions2.
Mic oso did he same wi h Mic oso T ansla o s a ing ha neu al ne wo ks
be e cap u e he con ex o ull sen ences be o e ansla ing hem, p o iding
much highe quali y ansla ions3. Howe e , despi e he easons hese big com-
panies ha e o shi o NMT we s ill lack a solid o mal backg ound ha suppo s
his echnology. Things like he ac i a ion unc ion choice, he numbe o uni s
pe laye o how many laye s o use, a e hings ha need a ho ough s udy o
unde s and hem. Fo now, we can only s udy NMT sys ems, by looking a he
ansla ions and see wha di e ences hem om he ones p oduced by o me
app oaches (Ben i ogli e al.,2016).
In his chap e we will gi e an o e iew o he NMT echnology and how i
can be used o pe o m he ask a hand. Fi s we will see how we can model he
language wi h neu al ne wo ks. Then we will p esen he cu en neu al a chi-
1In e na ional Wo kshop on Spoken Language T ansla ion.
2Found in ansla ion: Mo e accu a e, luen sen ences in Google T ansla e.
3Mic oso T ansla o launching Neu al Ne wo k based ansla ions o all i s speech lan-
guages.
11
12
Neu al machine ansla ion
ec u e used o ansla e. Las ly, we will see how we pe o m he sea ch ask o
ind he bes ansla ion.
Modelling language wi h neu al ne wo ks
Being able o ep esen language wi h neu al ne wo ks is no an easy ask. Fi s ,
we need o sol e he issue ha comes om he wo ds being ep esen ed as sym-
bols a he han numbe s. Fo his ask, we need a con inuous ep esen a ion
(a dense, eal- alued ec o ) ha bo h encapsula es ele an in o ma ion o such
symbols and educes he impac o he cu se o dimensionali y. This la e e m
e e s o he need o huge numbe o examples when lea ning complex unc ions
(such as language). This ac ge s wo se as he numbe o a iables inc eases (i.e.
he size o he ocabula y) since he model needs o disc imina e be ween a huge
numbe o combina ions o such alues.
Once we ha e sol ed he p e ious issue we ace ano he : we need o manage
he ac ha sen ences a e no o he same leng h. No only ha , he inpu and
ou pu sen ence can be o di e en leng h, so we need a way o p ocess in o ma-
ion o a iable leng h. Las ly, we need a way o cap u e con ex ual in o ma ion.
All hese h ee p oblems will be add essed in nex subsec ions.
Con inuous wo d ep esen a ion: wo d embedding
A con inuous ep esen a ion o a wo d is a ec o o ea u es which cha ac e -
ize he meaning o ha wo d. To unde s and his, i a human would ha e o
ex ac hese ea u es, he would choose g amma ical ea u es such as gende o
plu ali y. Wi h neu al ne wo ks we le he lea ning algo i hm disco e his eal-
alued ea u es ins ead. The idea is o map e e y wo d in he ocabula y o a
low-dimensional con inuous- alued ec o wi h he hope ha simila wo ds ge
simila ep esen a ions in he ea u e space. The main goal behind i is o allow
he model o gene alize be e o sequences ha a e no seen du ing aining bu
whose ea u es a e simila o hose who ha e been seen.
Al hough dis ibu ed wo d ep esen a ions we e i s p oposed in he ea ly
1980’s (Hin on,1986) and 1990’s (Cas año and Casacube a,1997), i was no un-
il he 2000’s ha i s app oaches using neu al ne wo ks appea ed o model lan-
guage. These models we e s udied i s in e ms o eed- o wa d ne wo ks (Ben-
gio e al.,2003) and la e in e ms o ecu en neu al ne wo ks (RNN) (Mikolo
e al.,2010,2011) and yielded good wo d ep esen a ion ha condensed linguis ic
egula i ies in he ec o ep esen a ions (Mikolo e al.,2013b).
One o he i s app oaches ha success ully used neu al ne wo ks o model
language was he one p oposed in Bengio e al. (2003). This wo k p ojec ed he
wo ds in o a eal- alued ec o . Then, i used a eed- o wa d ne wo k ocusing
on lea ning bo h he wo d ea u e ec o and he p obabili y dis ibu ion unc-
ion o wo d sequences in e ms o hese ea u e ec o s by maximizing he log-
likelihood o e a aining da a. The esul s ob ained wi h his app oach demon-
s a ed how dis ibu ed ep esen a ion o wo ds could be eamed up wi h neu al
ne wo ks o ou pe o m egula n-g am models.
3.1 Modelling language wi h neu al ne wo ks
13
All his wo k demons a ed ou s anding pe o mance in wo d-p edic ion bu
also he need o a mo e compu a ionally e icien model (specially he ones in-
ol ing RNN). Fo his pu pose, Mikolo e al. (2013a) p oposed a model ha
could lea n wo d ep esen a ion much as e wi h highe quali y han p e ious
app oaches. I showed he lea ned ela ionship be ween wo ds wi h simple al-
geb a by simply adding and sub ac ing ec o s o ob aining o he wo ds. Fo
example, he wo d Rome could be ob ained wi h: Pa is - F ance + I aly.
Dealing wi h sequences: ecu en neu al ne wo ks
When humans ead a sen ence, hey unde s and each wo d based on hei unde -
s anding o p e ious wo ds. As you ead, you b ain e ains in o ma ion ha is
used o unde s and wha is coming. Recu en neu al ne wo ks model his si -
ua ion na u ally. This is a majo ad an age since a a ie y o p oblems depend
on an a bi a y sequence o ime-dependen e en s, such as speech ecogni ion,
ansla ion o language modelling.
They ake in o accoun pas in o ma ion by means o a cycle be ween i s uni s.
This allows hem o ha e an in e nal s a e ha holds p esen and pas in o ma-
ion. In addi ion, depending on how hese cycles a e ou ed we ind di e en
a chi ec u es in he li e a u e. An example o his is he Elman (Elman,1990) and
Jo dan (Jo dan,1986) ne wo ks, also known as "simple ecu en ne wo ks" .
The Elman a chi ec u e wo ks as ollows: gi en a sequence o ec o s x=
x1...xT he ne wo k will p oduce a sequence o ou pu s y=y1...yT. A each ime
s ep (Eq. 3.1) he hidden s a e a ime ,s ge s calcula ed based on bo h, he inpu
a ime ,x and he hidden s a e o he p e ious ime s ep, s −1. Then, he ou pu
(Eq. 3.2 ) ge s calcula ed based on s .
s =σs(Wx +Us −1)(3.1)
y =σy(Vs )(3.2)
In hese equa ions, W,Uand Va e he weigh ma ices o he inpu , ecu en
and ou pu connec ions espec i ely, σsis an ac i a ion unc ion such as sigmoid
and σyis he ou pu ac i a ion unc ion, being usually, he so max unc ion (Eq.
3.18). Al e na i ely, he Jo dan a chi ec u e is simila bu he ecu en connec ion
is achie ed by aking in o accoun he p e ious ou pu a he han he p e ious
hidden s a e (Eq. 3.3).
s =σs(Wx +Uy −1)(3.3)
Howe e , one o he majo laws hese ne wo ks ha e is hei dependency on
he comple e pas in o ma ion. This p e en s hem om ocusing only on ecen
in o ma ion o on long- e m dependencies, as depic ed in (Bengio e al.,1994).
When aining hem, hey also ea u e he so-called anishing g adien p oblem
(explained in sec ion 4.1) which p e en s hem om p ope ly lea ning.
To add ess his, se e al new ecu en a chi ec u es ha e been p oposed, such
as he long sho e m memo y ne wo ks (Hoch ei e and Schmidhube ,1997)

14
Neu al machine ansla ion
known as LSTM (Fig. 3.1), o he ga ed ecu en uni s (GRU) (Cho e al.,2014)
which ha e he special ai o mi iga ing he anishing g adien p oblem. Bo h
o hese a chi ec u es ea u e pa ame ized ga es ha modula e how much inpu
in o ma ion is allowed o a ec he hidden s a e, how much pas in o ma ion is
le h ough o he nex ime s ep and how much in o ma ion is o go en. These
a e ainable pa ame e s ha allow he ne wo k o do his con enien ly.
In LSTM uni s, he memo y cell m (Eq. 3.4) depends on he p e ious memo y
cell m −1and he new in o ma ion ˜
m (Eq. 3.5) coming om he inpu o he
uni . The amoun o in o ma ion used o upda e m om bo h m −1and ˜
m is
modula ed by g (Eq. 3.6) and i (Eq. 3.7) which a e he ec o ou pu s o he
o ge ga e and inpu ga e whose alues ange om 0 o 1. By doing an elemen -
wise mul iplica ion  hey modula e he amoun o in o ma ion ha will be used
in he upda e, being 1 comple ely e ain and 0 comple ely o ge . The ou pu o
he uni s is modula ed by he ou pu ga e (Eq. 3.8) ge ing Eq. 3.9.
m =g m −1+i ˜
m (3.4)
˜
m = anh(WM
Ys −1+WM
Xx )(3.5)
g =σ(WF
Ys −1+WF
Xx )(3.6)
i =σ(WI
Ys −1+WI
Xx )(3.7)
o =σ(WO
Ys −1+WO
Xx )(3.8)
s =o  anh(m )(3.9)
In hese equa ions, WO
Y,WI
Y,WF
Yand WM
Ya e he ou pu ga e, inpu ga e, o -
ge ga e, and memo y cell ecu en weigh ma ices and WO
X,WI
X,WF
X,WM
Xa e
he ou pu ga e, inpu ga e, o ge ga e, and memo y cell inpu weigh ma ices.
σ(·)and anh(·)a e he sigma and hype bolic angen ac i a ion unc ions.
Inpu
Inpu
Fo ge
Ou pu
ga e
ga e
ga e
x
s −1
s −1s −1
s −1
x x
x
−1
s
Cell
˜
s
i
g
m −1
m
o
Figu e 3.1: LSTM uni
3.2 End- o-end ansla ion: encode -decode
15
Dealing wi h con ex : bidi ec ional ecu en neu al ne wo ks
When p ocessing sequences, he e a e imes when u u e e en s a e use ul and
p o ide mo e in o ma ion o he ne wo k in o de o be e pe o m he ask
a hand. Regula RNN ha e he limi a ion ha u u e in o ma ion can no be
eached om he cu en s a e.
Bidi ec ional ecu en neu al ne wo ks (BRNN) (Schus e and Paliwal,1997)
add ess his issue allowing u u e in o ma ion o be eachable om he cu en
s a e. They do his by connec ing wo hidden laye s o opposi e di ec ion o he
same ou pu laye , ha ing hen, in o ma ion o pas and u u e e en s.
Wi h his a chi ec u e we call o wa d laye he one ha p ocesses he sequence
in he posi i e ime di ec ion (le o igh ) and backwa d laye he one ha p o-
cesses he sequence in he nega i e ime di ec ion ( igh o le ). A each ime
s ep, he ou pu o bo h laye s a e combined (e.g. adding hem up) be o e apply-
ing he ou pu ac i a ion unc ion.
Fo wa d
s a es
Backwa d
s a es
-1 +1
Ou pu neu on
g oup
Hidden (s a e)
neu on g oup
Inpu s
G oup o
weigh s wi h
in o ma ion
low
Figu e 3.2: S uc u e o a bidi ec ional ecu en neu al ne wo k shown un olded in ime
o h ee ime s eps. Example bo owed om Schus e and Paliwal (1997).
.
End- o-end ansla ion: encode -decode
We ha e seen ha when a RNN p ocesses a sequence o leng h Ti p oduces an
ou pu sequence o he same leng h. In ansla ion howe e , sou ce and a ge
sen ence can (and usually a e) o di e en leng hs. To o e come his, wo simila
models we e p oposed by Su ske e e al. (2014) and Cho e al. (2014) which elied
on he encode -decode app oach. This wo k cons i u es a common amewo k
on which la e wo k aims o imp o e i in di e en ways.
The encode -decode app oach consis in a wo s ep p ocess ha i s , maps
he sou ce sen ence in o a ixed leng h ec o , and second, his ec o is decoded
o p oduce he a ge sen ence (possibly o di e en size). I aims o di ec ly
model he condi ional ansla ion p obabili y (Eq. 2.1).
The inpu o he sys em is a sequence o wo ds x=x1...xJp esen in he
ocabula y o he sou ce language Vxin a one-ho ep esen a ion. This means ha
we ha e o each wo d xja ocabula y-sized ec o ¯xj∈N|Vx|wi h all elemen s
16
Neu al machine ansla ion
se o ze o excep o he one loca ed a he posi ion o xjin Vxwhich is se o one.
Each wo d is p ojec ed o a ixed-leng h eal- alued ec o :
xj=Es¯
xj(3.10)
whe e xj∈Rdis he embedding o wo d xj,Es∈Rd×|Vx|is he sou ce languaje
p ojec ion ma ix and d he embedding size.
c
NULL
x1x2xT
y1y2yT0
y1yT0−1
Encode
Decode
Figu e 3.3: Ilus a ion o he encode -decode app oach.
The encode is an RNN ha eads each elemen o he sequence o wo d em-
beddings. A e eading each elemen in he posi i e di ec ion, he hidden s a e
h
jo he RNN is upda ed (Eq. 3.11). Once he end-o -sen ence symbol is eached,
he hidden s a e o he RNN c(Eq. 3.12), can be seen as a ixed-leng h summa y
o he whole inpu sequence.
h
j= (xj,h
j−1)(3.11)
c=h
J(3.12)
whe e is a non-linea unc ion.
The decode is ano he RNN which is ained o gene a e he ou pu sequence
by p edic ing he nex wo d yigi en he hidden s a e si, he p e ious gene a ed
wo d yi−1and c. The hidden s a e o he decode a ime iis compu ed as:
si= (si−1,yi−1,c)(3.13)
whe e is a non-linea ac i a ion unc ion (LSTM o GRU) ha p oduce he hid-
den s a e ec o si. The condi ional p obabili y o he nex wo d yiis app oxi-
ma ed (Eq. 3.14) assuming ha i depends on he p e ious wo d (and all p e ious
wo ds ha a e in si o some ex en ). In his case, g(·)∈R|Vy|is he so max unc-
ion (Eq. 3.18) ha p oduces a ec o o p obabili ies o size |Vy|, which is he size
o he ocabula y o he a ge language. ¯yi∈N|Vy|is he one-ho ep esen a ion
o he wo d yi.V∈R|Vy|×Lis he weigh ma ix and ϕ(·)is he L-sized ou pu
laye o a RNN.
P (yi|y1...yi−1,c)≈¯y0
ig(Vϕ(si,yi−1,c)) (3.14)
I is impo an o no e ha , du ing aining, we wan he decode o lea n o
ou pu he e e ence sen ence. In o de o help he model, we use he eache o c-
ing (Pascanu e al.,2013) echnique. This echnique consis o eeding he model
3.2 End- o-end ansla ion: encode -decode
17
wi h he co esponding wo d in he e e ence sen ence a ime s ep i−1 ins ead
o he p e iously gene a ed wo d.
A en ion model
One o he laws ha encode -decode models exhibi is ha hey ha e o con-
dense an a bi a y leng h sen ence in o a ixed-leng h ec o c. This is a bo leneck
when dealing wi h long sen ences, as no iced by Cho e al. (2014), and causes he
sys em o pe o m poo ly in hese si ua ions.
In o de o sol e his, Bahdanau e al. (2014) p oposed he so-called a en ion
model. The basic idea is o use a di e en con ex ec o depending on he cu en
decoding s age. Wi h his idea, Equa ion 3.14 is ew i en as:
P (yi|y1...yi−1,c)≈¯y0
ig(Vϕ(si,yi−1,ci)) (3.15)
whe e siis he RNN hidden s a e a ime i:
si= (si−1,yi−1,ci)(3.16)
We can see ha a each ime s ep, we use a di e en con ex ec o ci. This
ec o is calcula ed based on a sequence o anno a ions (h1,..., hJ)ex ac ed om
he sou ce sen ence. These anno a ions ha e in o ma ion ha s ongly desc ibe
he i- h wo d and su oundings. They a e calcula ed by conca ena ing he o -
wa d and backwa d hidden s a es o a BRNN. The anno a ion hj= [h
j;hb
j]∈R2S
whe e h
jand hb
ja e he hidden s a es o he o wa d and backwa d laye s o he
BRNN and Sis he size o he hidden s a e (we assume ha hey a e o he same
size). The e o e, ciis calcula ed as he weigh ed sum o he anno a ions:
ci=
J
∑
j=1
αijhj(3.17)
whe e αij is compu ed by he so max unc ion:
αij =exp(eij)
∑J
k=1exp(eik)(3.18)
being eij =a(si−1,hj)an alignmen model which sco es how ela ed he inpu s a
posi ion jand he ou pu a posi ion ia e. The alignmen model is pa ame ized
wi h a eed- o wa d neu al ne wo k which is ained join ly wi h he whole sys-
em.
Decoding ansla ions: beam sea ch
When decoding he ansla ion, he neu al sys em p o ides, a each ime s ep, a
se o p obabili ies (wi h he so max unc ion) o each wo d in he ocabula y.
Because we a e maximizing equa ion 2.1 and he ou pu p obabili ies o a gi en
ime s ep depend o e he p e ious chosen wo d, we canno jus choose he one
24
T aining neu al ne wo ks
decaying a e age o pas squa ed g adien s and an exponen ially decaying
a e age o pas g adien s m :
m =β1·m −1+ (1−β1)·∇`(Θ )(4.11)
=β2· −1+ (1−β2)·∇`(Θ )2(4.12)
The au ho s no iced ha a ea ly s ages, he algo i hm is biased owa ds ze o.
To coun e ac his hey co ec he abo e exp essions as ollows:
ˆm =m
1−β
1
(4.13)
ˆ =
1−β
2
(4.14)
lea ing he upda e ule as:
Θ +1=Θ −ρ
√ˆ +eˆm (4.15)
The de aul alues he au ho s p opose a e 0.9 o β1and 0.999 o β2.

CHAPTER 5
Online lea ning
Al hough compu a ional esou ces ha e e ol ed o he poin whe e i is ela i ely
cheap o ain a Pa e n Recogni ion (PR) sys em, i is s ill a ime-consuming p o-
cess ha akes days o e en weeks o comple e. As mo e and mo e da a is a ail-
able his p oblem is agg a a ed, specially on sys ems ha in e ac wi h changing
en i onmen s o ha equi e a quick esponse a he same ime hey lea n. In
hese cases, being able o adap ou sys em wi hou aining i om sc a ch be-
comes manda o y.
In his chap e we will p esen he online lea ning amewo k which will se e
o in oduce ou wo k, emphasizing i s applica ion o a pos -edi ing scena io in
MT. Nex we will explain he Passi e-Agg essi e (PA) online lea ning algo i hm.
I has he main goal o a oiding he ca as ophic in e e ence (McCloskey and
Cohen,1989) p oblem ha h ea ens neu al ne wo k sys ems in which hey end
o o ge p e iously lea ned in o ma ion upon lea ning new in o ma ion. Finally,
we will p opose h ee new PA-based algo i hms o he ask o NMT.
Online lea ning amewo k
PR sys ems ha e achie ed an accep able pe o mance in e y complex asks ha
in ol es s uc u ed ou pu and ambigui y such as machine ansla ion o image
desc ip ion. Al hough hese sys ems can each a high pe o mance on da a simi-
la o he one used o ain hem, hei pe o mance apidly d ops when he ask is
sligh ly di e en . In addi ion, ge ing enough manually anno a ed da a o a spe-
ci ic domain in o de o ain a whole sys em migh no be possible. The e o e,
i is use ul o ain wi h gene ic ou -o -domain da a and hen une he sys em
wi h domain-speci ic da a. Because a comple e e aining o he sys em migh be
in easible, online lea ning echniques (Blum,1998) a e chosen o his ask.
These echniques a e pa icula ly use ul in he compu e assis ed ansla ion
(CAT) and in e ac i e machine ansla ion (IMT) pa adigms, whe e human ans-
la o s wo k join ly wi h machine ansla o s in o de o e icien ly ob ain high
quali y ansla ions. In a pos -edi ing scena io, he MT sys em ansla es a sen-
ence and p opose his as a hypo hesis o a human ansla o which hen co ec s
he sen ence (i needed). Once a sen ence is co ec ed wha we ha e is a new
pai o sen ences ha can be used o ain he sys em. By using hese new ain-
25
26
Online lea ning
ing samples o ain ou model, we p og essi ely make he pos -edi ing p ocess
mo e e icien . Fi s , by making he sys em lea n om i s own e o s and second,
by easing he wo k o he human ansla o as i will ha e o co ec less e o s.
P io wo k in his a ea has also shown ha human ansla o s become mo e p o-
duc i e as he MT quali y imp o es (Ta sumi,2009).
The applica ion o such echniques has been s udied ho oughly (Ma ínez-
Gómez e al.,2012;La ie,2014;Ma ínez-Gómez e al.,2011) in classical ph ase-
based SMT sys ems. Resul s show ha signi ican imp o emen s can be achie ed
by adap ing he sys em by means o online lea ning in e ms o he e o he
human would need o co ec he hypo hesis p o ided. Howe e , he e has been
no published wo k ( o he bes o ou knowledge) ela ed o he applica ion o
hese echniques in NMT sys ems. This wo ks aims a p o iding new algo i hms
o apply online lea ning o hese sys ems.
Passi e-Agg essi e online lea ning
A a ie y o online lea ning me hods ha e been p oposed in li e a u e (Lu e al.,
2016). A classical OL me hod is he Pe cep on algo i hm (Rosenbla ,1958) which
upda es he model by adding a misclassi ied example (pa ame ized wi h a ac-
o ) o he cu en pa ame e s. F om his wo k, a lo o new online lea ning algo-
i hms ha e been de eloped based on he maximum ma gin c i e ion (C amme
and Singe ,2003;Gen ile,2001;Ki inen e al.,2004;Li and Long,2000) which ies
o sepa a e he classes as much as possible. One impo an echnique ha alls in
his ca ego y is he Passi e-Agg essi e (PA) online lea ning me hod (C amme
e al.,2006). This has been p o ed as a e y success ul and popula online lea n-
ing echnique o sol ing many eal-wo ld applica ions.
The idea behind he PA algo i hm is e y simple. When a new sample a i es
i is classi ied and a loss is ob ained, measu ing he deg ee o which he p edic ion
is w ong. The idea now is o ind a new se o pa ame e s (weigh s) ha make
he classi ie o co ec ly classi y he sample (agg essi eness) bu by emaining
ela i ely close o he p e ious se o pa ame e s (passi eness). We will ou line
he PA app oach in a bina y classi ica ion p oblem as explained in C amme e al.
(2006).
In a bina y classi ica ion p oblem, each sample x ∈Rdhas a unique label
y ∈ {+1, −1}associa ed. We assume ha he classi ica ion unc ion is based on
a ec o o weigh s Θ∈Rdwhich ake he o m o sign(Θ·x). The magni ude
|Θ·x|is in e p e ed as he deg ee o con idence in he p edic ion. We e e o
he e m y (Θ ·x )as he ma gin calcula ed a ime . Whene e he ma gin is
posi i e, he sample has been classi ied co ec ly. Howe e , we wan he classi ie
o achie e a co ec classi ica ion wi h some ma gin (in C amme e al. (2006) his
ma gin is se o 1). Then we de ine he loss ` ha su e s he classi ie whene e
he samples is classi ied inco ec ly:
`(Θ;(x,y)) = (0 i y(Θ·x)≤1
1−y(Θ·x)o he wise (5.1)
5.3 PA online lea ning applied o NMT
27
In o de o p og essi ely lea n he weigh ec o Θ, he algo i hm mus up-
da e i a e e e y sample. A ime , he new weigh ec o Θ +1is calcula ed by
sol ing his cons ained op imiza ion:
Θ +1=a gmin
Θ∈Rd
1
2kΘ−Θ k2s. . `(Θ;(x ,y )) = 0 (5.2)
In his algo i hm, i he loss is 0, he op imal solu ion is Θ +1=Θ . On he
o he hand, when he loss is g ea e han 0, he algo i hm o ces Θ +1 o sa is y
he cons ain `(Θ +1;(x ,y )) = 0 while being as close as possible o he p e ious
weigh ec o o p ese e knowledge o p e ious samples. I we de i e Eq. 5.2,
he upda e ule o he weigh ec o is:
Θ +1=Θ +τ y x whe e τ =`(Θ ;(x ,y ))
kx k2(5.3)
Howe e , his upda e ule is oo agg essi e and i migh esul in upda ing he
model in such a way ha in o de o mee he cons ain , i p oduces a lo o e o s
in la e samples because he model was changed d as ically. The e o e, a gen le
upda e s a egy is equi ed. To do his, he au ho s in oduce a non-nega i e slack
ξ a iable in o he op imiza ion p oblem de ined in Eq. 5.2:
Θ +1=a gmin
Θ∈Rd
1
2kΘ−Θ k2+Cξs. . `(Θ;x ,y )) ≤ξand ξ≥0 (5.4)
The pa ame e Ccon ols he in luence o he slack a iable ξ. The au ho s
coin he C a iable as he agg essi eness pa ame e o he algo i hm. The upda e
ule in his case is he same Θ +1=Θ +τ y x bu τ changes o:
τ =min{C,`(Θ ;(x ,y ))
kx k2}(5.5)
PA online lea ning applied o NMT
In his sec ion we p opose a new e sion o PA-based SGD. The de elopmen
o his wo ks is inspi ed by he applica ion o he MIRA (a PA-based) algo i hm
(C amme and Singe ,2003) used in adi ional SMT o une he weigh s o he
log-linea model. Also, he wo k de elop in Ma ínez-Gómez e al. (2012) o he
ask o online adap a ion gi es us a eel o wha can be achie ed when adap ing
a MT sys em wi h OL. As any o he PA echnique, ou algo i hm aims o pe o m
he minimum modi ica ion o he model pa ame e s while ul illing a co ec ness
c i e ion. In ou case, we use he loss unc ion o he neu al model o e iciency
easons (bu we could ha e used ano he loss unc ion such as BLEU).
28
Online lea ning
Le h be he hypo hesis gene a ed by he NMT sys em (using he pa ame e s
Θ a ime ) o he sou ce sen ence x . We conside ha Θ is inco ec i he
model assigns a lowe p obabili y o he a ge e e ence sen ence y han o h :
pΘ (y |x )<pΘ (h |x )(5.6)
When ha happens, we wan o sea ch o a b
Θsuch ha b
Θis close o Θ and
pΘ (y |x )>pΘ (h |x ). This is exp essed wi h he loss unc ion `:
`(b
Θ,x ,y ,h ) = log pb
Θ(h |x )−log pb
Θ(h |x )(5.7)
Wi h his loss unc ion we a e measu ing how la ge is he gap be ween he
p obabili y gi en o he hypo hesis and he p obabili y gi en o he e e ence. By
op imizing his unc ion we educe his gap wi h he idea ha p og essi ely, he
hypo hesis will be close (i.e. simila ) o he e e ence sen ence. Wi h he goal o
op imizing his unc ion, h ee a ia ions o PA-based algo i hms a e de eloped.
Passi e-Agg essi e ia subg adien echniques
In o de o ind b
Θ, he p oblem can be o mula ed as:
b
Θ=a gmin
Θ
1
2kΘ−Θ k2+Cξs. . `(b
Θ,x ,y ,h )≤ξand ξ≥0 (5.8)
being C he pa ame e ha con ols he agg essi eness o he algo i hm and ξ
a slack a iable (as discussed in Sec ion 5.2). F om his equa ion we ha e ξ≥
max(0, `(b
Θ,x ,y ,h )) and hen, we de ine F as he unc ion o op imize:
F (Θ,x ,y ,h ) = 1
2kΘ−Θ k2+Cmax(0, `(b
Θ,x ,y ,h )) (5.9)
aiming o ind he se o pa ame e s ha minimize his unc ion, i.e.:
b
Θ=a gmin
Θ
F (Θ,x ,y ,h )(5.10)
Because we a e op imizing a unc ion (F ) wi h discon inui y poin s (because
he max(·) unc ion has no de i a i e a 0) we ha e o use a subg adien me hod
(Sho e al.,2003). This esul s in he de i a i e ∂ΘF (Θ,x ,y ,h )being:
∂ΘF (Θ,x ,y ,h ) = 




Θ−Θ −C∇`(Θ,x ,y ,h )`(Θ,x ,y ,h )<0
Θ−Θ `(Θ,x ,y ,h )>0
[Θ−Θ ,Θ−Θ −C∇`(Θ,x ,y ,h )] `(Θ,x ,y ,h ) = 0
(5.11)
5.3 PA online lea ning applied o NMT
29
whe e, on he discon inui y poin (`(Θ,x ,y ,h ) = 0), we choose a alue in he in-
e al [Θ−Θ ,Θ−Θ −C∇`(Θ,x ,y ,h )]. We assume Θ−Θ when `(Θ,x ,y ,h ) =
0. Finally, we pe o m he upda e as:
Θk+1=Θk−ρ·∂ΘF (Θ,x ,y ,h )|Θk(5.12)
whe e ρis he lea ning a e, ∂ΘF is he subg adien o F wi h espec o Θ. We
ini ialize Θk=0=Θ and he upda e is pe o med k imes pe sample. We deno e
his PA ia subg adien echnique upda e ule as PAS.
Because we may wan o achie e a highe p obabili y o he e e ence sen ence
bu wi h some ma gin m:pΘ(y |x ) + m>pΘ(h |x )we can ew i e he subg a-
dien ∂ΘF wi h he discon inui y poin being m a he han ze o (as de ined in
Eq. 5.11).
Passi e-Agg essi e ia p ojec ed subg adien echniques
An ex ension o he PAS me hod is he p ojec ed PA subg adien me hod (PPAS)
in which he op imiza ion p oblem is e o mula ed (Boyd e al.,2003). We de ine
G (Θ,x ,y ,h )as max(0, `(b
Θ,x ,y ,h )). Then, Eq. 5.8 can be ew i en as:
b
Θ=a gmin
Θ
G (Θ,x ,y ,h )s. . kΘ−Θ k2≤C(5.13)
In his case, ∂ΘG (Θ,x ,y ,h )is de ined as:
∂ΘG (Θ,x ,y ,h ) = 




∇`(Θ,x ,y ,h )`(Θ,x ,y ,h )<0
0`(Θ,x ,y ,h )>0
[0, ∇`(Θ,x ,y ,h )] `(Θ,x ,y ,h ) = 0
(5.14)
and he upda e ule is pe o med in a wo s ep p ocess. Fi s , we calcula e he
in e media e weigh upda e ¯
Θk+1as:
¯
Θk+1=Θk−ρ∂ΘG (Θ,x ,y ,h )|Θk(5.15)
and second, we apply he p ojec ion ope a o , being Θk+1:
Θk+1=¯
Θk+1−Θ
k¯
Θk+1−Θ kC+Θ (5.16)
As in he p e ious case, we ini ialize Θk=0=Θ .
Passi e-Agg essi e ia SGD wi h egula iza ion
We can also op imize `(Θ,x ,y ,h )by means o SGD bu wi h a egula iza ion
e m, inspi ed by he PA p inciple o upda ing he model wi h new pa ame e s

30
Online lea ning
ha a e close o he cu en ones. We will e e o his algo i hm as PAR. Wi h his
in mind, we can de ine he minimiza ion p oblem as:
b
Θ=a gmin
Θ
(`(Θ,x ,y ,h ) + C
2kΘ−Θ k2)(5.17)
Because he minimiza ion has no discon inui y poin s we don’ need o apply
a subg adien me hod. The upda e ule hen, esul s in:
Θk+1=Θk−ρ·(∇`(Θ,x ,y ,h )|Θk+C(Θk−Θ )) (5.18)
As usual, Θk=0=Θ . We obse e ha when k=1 he upda e ule is plain
SGD wi h no egula iza ion e m:
Θk+1=Θk−ρ·(∇`(Θ,x ,y ,h )|Θk+
: 0
C(Θk−Θ ) ) (5.19)
in ha case, we in oduce a small Gaussian noise ν o he model pa ame e s in
o de o a oid eaching local minima:
Θk+1=Θk−ρ·(∇`(Θ,x ,y ,h )|Θk+Cν)(5.20)
CHAPTER 6
Expe imen s and esul s
A e he p oposal o h ee PA-based algo i hms in Sec ion 5.3, we will p oceed
o i s es ing and compa ison wi h o he well-known algo i hms ( hose desc ibed
in Sec ion 4.2). In his sec ion we will desc ibe he expe imen al se up used o
conduc he expe imen a ion. Fi s , we will desc ibe he ask in which we wan
o e alua e ou algo i hms. Then we will desc ibe he NMT sys em used and he
ools needed o wo k wi h i . A e desc ibing he co po a used, we will compa e
and discuss he esul s ob ained.
Expe imen al amewo k
Be o e p esen ing he esul s, we will p esen he ask a hand and desc ibe he
ools used o de elop he men ioned algo i hms and he co po a used o assess
hem.
Task desc ip ion
In o de o es hese algo i hms we will simula e (o he wise i would be oo
cos ly) a pos -edi ing scena io. We will simula e his scena io in h ee di e en
co pus o asks (de ailed in nex subsec ions). Fi s , we will ain h ee neu al
models using o each one he aining pa i ion o i s co esponding da ase .
Once ained, hese models will cons i u e he baseline models o each ask.
Then, we will e ine hese models wi h hei espec i e es o de elopmen
se o he co pus in which hey we e ained. Fo a gi en sou ce sen ence, he sys-
em p oduces i s ansla ion ( he sys em’s hypo hesis). This ansla ion is s o ed
o la e e alua ion. Then we ake he pos -edi ed sen ence (in his case, he e e -
ence sen ence) and upda e he sys em wi h i . This is epea ed un il he ull se o
sen ences a e ansla ed. We e alua e he s o ed hypo heses and compa e hem
wi h he ones p oduced by he baseline sys em. Ou hope is o see an imp o e-
men in he quali y o hese ansla ion. Wi h his p ocedu e we wan o see up o
wha ex en he OL algo i hm can e ine he baseline sys em.
In o de o measu e he impac o each algo i hm in he pos -edi ing e o
educ ion we use he TER me ic. In o de o assess he quali y o ansla ions we
will use he BLEU and METEOR me ics.
31
32
Expe imen s and esul s
So wa e
We ha e de eloped ou wo k using he NMT-Ke as oolki 1(Pe is,2017). This
(s ill in p og ess) so wa e p o ides ools o he NMT ask, such as: beam sea ch
decoding, suppo o GRU and LSTM ne wo ks, unknown wo d eplacemen ,
use o p e ained wo d embedding ec o s an mo e.
This so wa e is based on a Py hon lib a y called Ke as2(Cholle e al.,2015).
Ke as o e s an abs ac ion laye on op o Theano3(Theano De elopmen Team,
2016) (a lib a y o de ining and e alua ing ma hema ical exp essions in ol ing
mul i-dimensional a ays) and eases he ask o building and aining di e en
neu al ne wo k a chi ec u es. I has implemen ed u ili ies o sa e and load neu al
ne wo k models, moni o he aining p ocess and use p ede ined NN a chi ec-
u es.
All he algo i hms lis ed in Sec ion 5.3 ha e been implemen ed in Ke as and
in eg a ed in he NMT-Ke as oolki .
Co po a
In his wo k, we es all he algo i hm in h ee di e en co pus: Xe ox, Emea and
TED. We use he s anda d pa i ions and he English-F ench language pai o all
expe imen s. The ex is kep ue case and i is okenized using he sc ip om
Moses 4. In he nex subsec ions we will desc ibe hese co pus.
Xe ox
The Xe ox co pus (Ba achina e al.,2009) consis s in ansla ions o use manuals
o Xe ox p in e s. Table 6.1 shows all in o ma ion ega ding he pa i ions o
he English-F ench language pai .
T aining De elopmen Tes
Sen ences 52k 984 994
Vocabula y size English 14k 1.8k 1.7k
F ench 15.5k 1.9k 1.8k
Wo ds English 615k 10.9k 11.1k
F ench 676k 11.7k 11.8k
Table 6.1: S a is ics o he Xe ox co pus. In he able a e collec ed he numbe o sen ences
and ocabula y size o each pa i ion and language. k s ands o housands
1h ps://gi hub.com/l apeab/nm -ke as/
2h ps://ke as.io/
3h p://deeplea ning.ne /so wa e/ heano/
4h p://www.s a m .o g/moses/
6.1 Expe imen al amewo k
33
Emea
The Emea pa allel co pus (Tiedemann,2009) is made ou o documen s om he
Eu opean Medicine Agency. Table 6.2 shows all in o ma ion ega ding he pa i-
ions o he English-F ench language pai .
T aining De elopmen Tes
Sen ences 319k 500 1k
Vocabula y size English 52k 2.8k 4.5k
F ench 59.3k 2.9k 4.5k
Wo ds English 4.2M 10.3k 21.4k
F ench 4.8M 12.3k 25.7k
Table 6.2: S a is ics o he Emea co pus. In he able a e collec ed he numbe o sen ences
and ocabula y size o each pa i ion and language. k s ands o housands and M o
millions
TED
The TED co pus (Fede ico e al.,2011) ga he s ansc ibed TED alks. Table 6.3
shows all in o ma ion ega ding he pa i ions o he English-F ench language
pai .
T aining De elopmen Tes
Sen ences 159k 887 1.7k
Vocabula y size English 46.7k 3.4k 4.9k
F ench 58.2k 3.9k 4.9k
Wo ds English 2.17M 21.3k 33.5k
F ench 2.3M 21.8k 35.7k
Table 6.3: S a is ics o he TED co pus. In he able a e collec ed he numbe o sen ences
and ocabula y size o each pa i ion and language. k s ands o housands
NMT sys em
Fo each co pus, we ain a neu al model o e he aining pa i ion. Such model
(simila o he one depic ed in Bahdanau e al. (2014)) consis s in an encode -
decode LSTM ne wo k equipped wi h he a en ion mechanism. We made use
o single-laye ed LSTM due o ime limi a ions. The size o he hidden s a e o
each LSTM, he wo d embedding size and he a en ion mechanism laye is 512
(ou choice is based on he expe imen a ions ca ied ou in B i z e al. (2017)).
The baseline sys ems ha e been ained wi h he Adadel a algo i hm wi h he
de aul pa ame e s. Du ing aining, Gaussian noise (G a es,2011) and laye
no maliza ion (Ba e al.,2016) a e applied as egula iza ion me hods. We also
ea ly s opped he aining i he BLEU on he de elopmen se did no imp o e
in 100,000 upda es. The size o he beam is 6 in all expe imen s.
40
Fu u e wo k and conclusions
Inclusion o OL s a egies in IMT
The IMT scena io is igh ly ela ed wi h he pos -edi ing scena io. IMT ies o
collabo a e wi h he human on he ansla ion ask wi h he goal o minimizing
human e o . O en, he use only needs o poin ou he posi ion in he sen ence
in which i is w ongly ansla ed and he sys em will ou pu an al e na i e su -
ix s a ing om ha poin . IMT is an ac i e ield and ecen wo k has s udied
i s applica ion on NMT sys ems (Knowles and Koehn,2016;Pe is e al.,2017b).
As in he pos -edi ing scena io, being able o apply OL echniques in o an IMT
amewo k migh be bene icial in o de o de elop mo e adap i e and p oduc i e
MT sys ems.
Conclusions
In his wo k, h ee OL algo i hms ha e been p oposed and applied o an NMT
sys em. They ely on he Passi e-Agg essi e OL app oach in which he MT
model is upda ed a e e e y sample in o de o ul il a co ec ness c i e ion while
emembe ing p e iously lea ned in o ma ion. Because hese algo i hms ha e a
high numbe o hype pa ame e s o une, an ex ensi e g id sea ch has been ca -
ied ou in o de o ind he bes combina ion. In addi ion, a s udy has been made
o see how he di e en pa ame e s a ec each algo i hm’s pe o mance. Resul s
show ha , al hough hey depend on se e al hype pa ame e s, hose c i ical o
he well beha iou o he algo i hms a e, in e e y case, he agg essi eness hy-
pe pa ame e and he lea ning a e. O he pa ame e s such as he ma gin o he
numbe o i e a ions pe sample o e mino imp o emen s in he o e all pe o -
mance o he algo i hm.
These algo i hms ha e been es ed in a pos -edi ing scena io whe e an al eady
ained NMT sys em has o be e ined as soon as new samples a e a ailable. Each
new sample ha a i es o he sys em is ansla ed and co ec ed (we use he
e e ence sen ence ins ead). A loss unc ion is compu ed in o de o measu e
he dis ance be ween bo h sen ences. Once we ha e his, we use i o adap he
sys em, aiming o a oid he commi ed mis akes. In o de o e alua e how well
hese new algo i hms beha e, a compa ison be ween hese algo i hms and well-
known OL g adien descen a ian s has been ca ied ou . Resul s show ha OL
echniques help o e ine and adap he sys em on- he- ly, inc easing he quali y
o ansla ions and educing he human e o in a pos -edi ing scena io. We ha e
ound ha he p oposed PA-based algo i hms o e a compe i i e pe o mance:
hey imp o e he quali y o ansla ions in all asks. Ne e heless, adap i e SGD-
based algo i hms pe o med gene ally be e .
Finally, pa o he wo k de eloped in his hesis has been submi ed o he
2017 Con e ence on Empi ical Me hods in Na u al Language P ocessing (Pe is
e al.,2017a). Mo eo e , we plan o submi an a icle o a jou nal (ye o be de e -
mined).

Bibliog aphy
Ba, J. L., Ki os, J. R., and Hin on, G. E. (2016). Laye no maliza ion. a Xi p ep in
a Xi :1607.06450.
Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neu al machine ansla ion by
join ly lea ning o align and ansla e. a Xi p ep in a Xi :1409.0473.
Bane jee, S. and La ie, A. (2005). Me eo : An au oma ic me ic o m e alua-
ion wi h imp o ed co ela ion wi h human judgmen s. In P oceedings o he
acl wo kshop on in insic and ex insic e alua ion measu es o machine ansla ion
and/o summa iza ion, olume 29, pages 65–72.
Ba achina, S., Bende , O., Casacube a, F., Ci e a, J., Cubel, E., Khadi i, S., La-
ga da, A., Ney, H., Tomás, J., Vidal, E., e al. (2009). S a is ical app oaches o
compu e -assis ed ansla ion. Compu a ional Linguis ics, 35:3–28.
Bengio, Y., Ducha me, R., Vincen , P., and Jau in, C. (2003). A neu al p obabilis ic
language model. Jou nal o machine lea ning esea ch, 3:1137–1155.
Bengio, Y., Sima d, P., and F asconi, P. (1994). Lea ning long- e m dependencies
wi h g adien descen is di icul . IEEE ansac ions on neu al ne wo ks, 5(2):157–
166.
Ben i ogli, L., Bisazza, A., Ce olo, M., and Fede ico, M. (2016). Neu al e -
sus ph ase-based machine ansla ion quali y: a case s udy. a Xi p ep in
a Xi :1608.04631.
Bishop, C. M. (2006). Pa e n ecogni ion and machine lea ning. sp inge .
Blum, A. (1998). On-line algo i hms in machine lea ning. In Online algo i hms,
pages 306–325. Sp inge .
Bo ou, L. (1991). S ochas ic g adien lea ning in neu al ne wo ks. P oceedings o
Neu o-Nımes, 91.
Boyd, S., Xiao, L., and Mu apcic, A. (2003). Subg adien me hods. lec u e no es o
EE392o, S an o d Uni e si y, Au umn Qua e , 2004.
B i z, D., Goldie, A., Luong, T., and Le, Q. (2017). Massi e explo a ion o neu al
machine ansla ion a chi ec u es. a Xi p ep in a Xi :1703.03906.
B own, P. F., Cocke, J., Pie a, S. A. D., Pie a, V. J. D., Jelinek, F., La e y, J. D.,
Me ce , R. L., and Roossin, P. S. (1990). A s a is ical app oach o machine ans-
la ion. Compu a ional linguis ics, 16(2):79–85.
41
42
BIBLIOGRAPHY
B own, P. F., Pie a, V. J. D., Pie a, S. A. D., and Me ce , R. L. (1993). The ma h-
ema ics o s a is ical machine ansla ion: Pa ame e es ima ion. Compu a ional
linguis ics, 19(2):263–311.
Cas año, A. and Casacube a, F. (1997). A connec ionis app oach o machine
ansla ion. In Fi h Eu opean Con e ence on Speech Communica ion and Technology.
Chen, S. F. and Joshua, G. (1998). An empi ical s udy o smoo hing echniques
o language modeling. Technical epo , Ha a d Compu e Science G oup.
Chiang, D. (2012). Hope and ea o disc imina i e aining o s a is ical ansla-
ion models. Jou nal o Machine Lea ning Resea ch, 13:1159–1187.
Cho, K., Van Me iënboe , B., Gulceh e, C., Bahdanau, D., Bouga es, F., Schwenk,
H., and Bengio, Y. (2014). Lea ning ph ase ep esen a ions using nn encode -
decode o s a is ical machine ansla ion. a Xi p ep in a Xi :1406.1078.
Cholle , F. e al. (2015). Ke as.
h ps://gi hub.com/ cholle /ke as
.
Chow, Y.-L. and Schwa z, R. (1989). The n-bes algo i hm: An e icien p ocedu e
o inding op n sen ence hypo heses. In P oceedings o he wo kshop on Speech
and Na u al Language, pages 199–202. Associa ion o Compu a ional Linguis-
ics.
Chung, J., Gulceh e, C., Cho, K., and Bengio, Y. (2014). Empi ical e alua ion
o ga ed ecu en neu al ne wo ks on sequence modeling. a Xi p ep in
a Xi :1412.3555.
C amme , K., Dekel, O., Keshe , J., Shale -Shwa z, S., and Singe , Y. (2006). On-
line passi e-agg essi e algo i hms. Jou nal o Machine Lea ning Resea ch, 7:551–
585.
C amme , K. and Singe , Y. (2003). Ul aconse a i e online algo i hms o mul-
iclass p oblems. Jou nal o Machine Lea ning Resea ch, 3:951–991.
Duchi, J., Hazan, E., and Singe , Y. (2011). Adap i e subg adien me hods o on-
line lea ning and s ochas ic op imiza ion. Jou nal o Machine Lea ning Resea ch,
12:2121–2159.
Elman, J. L. (1990). Finding s uc u e in ime. Cogni i e science, 14(2):179–211.
Fede ico, M., Ben i ogli, L., Paul, M., and S üke , S. (2011). O e iew o he
IWSLT e alua ion campaign. In P oceedings o he In e na ional Wo kshop on Spo-
ken Language T ansla ion, pages 11–27.
Gen ile, C. (2001). A new app oxima e maximal ma gin classi ica ion algo i hm.
Jou nal o Machine Lea ning Resea ch, 2:213–242.
Golik, P., Doe sch, P., and Ney, H. (2013). C oss-en opy s. squa ed e o ain-
ing: a heo e ical and expe imen al compa ison. In P oceedings o he In e speech,
pages 1756–1760.
G a es, A. (2011). P ac ical a ia ional in e ence o neu al ne wo ks. In Ad ances
in Neu al In o ma ion P ocessing Sys ems, pages 2348–2356.
BIBLIOGRAPHY
43
Guo, J. (2013). Backp opaga ion h ough ime. Unpubl. ms., Ha bin Ins i u e o
Technology.
Hin on, G. E. (1986). Lea ning dis ibu ed ep esen a ions o concep s. In P o-
ceedings o he eigh h annual con e ence o he cogni i e science socie y, olume 1,
page 12. Amhe s , MA.
Hoch ei e , S., Bengio, Y., F asconi, P., and Schmidhube , J. (2001). G adien low
in ecu en ne s: he di icul y o lea ning long- e m dependencies.
Hoch ei e , S. and Schmidhube , J. (1997). Long sho - e m memo y. Neu al com-
pu a ion, 9(8):1735–1780.
Hu chins, J. (2005). The his o y o machine ansla ion in a nu shell. Re ie ed
Decembe , 20:2009.
Jean, S., Fi a , O., Cho, K., Memise ic, R., and Bengio, Y. (2015). Mon eal neu al
machine ansla ion sys ems o wm ’15. In P oceedings o Empi ical Me hods in
Na u al Language P ocessing, pages 134–140.
Jo dan, M. I. (1986). A ac o dynamics and pa allellism in a connec ionis se-
quen ial machine.
Kingma, D. and Ba, J. (2014). Adam: A me hod o s ochas ic op imiza ion. a Xi
p ep in a Xi :1412.6980.
Ki inen, J., Smola, A. J., and Williamson, R. C. (2004). Online lea ning wi h ke -
nels. IEEE ansac ions on signal p ocessing, 52:2165–2176.
Knese , R. and Ney, H. (1995). Imp o ed backing-o o m-g am language mod-
eling. In Acous ics, Speech, and Signal P ocessing, olume 1, pages 181–184. IEEE.
Knowles, R. and Koehn, P. (2016). Neu al in e ac i e ansla ion p edic ion. In
P oceedings o he Con e ence o he Associa ion o Machine T ansla ion in he Ame -
icas (AMTA), olume 1, pages 107–120.
Koehn, P. (2004). S a is ical signi icance es s o machine ansla ion e alua ion.
In P oceedings o Empi ical Me hods in Na u al Language P ocessing, pages 388–
395.
Koehn, P. (2010). S a is ical machine ansla ion.
La ie, M. D. C. D. A. (2014). Lea ning om pos -edi ing: Online model adap a-
ion o s a is ical machine ansla ion. P oceedings o he 14 h Con e ence o he
Eu opean Chap e o he Associa ion o Compu a ional Linguis ics (EACL 14), page
395.
LeCun, Y., Bengio, Y., and Hin on, G. (2015). Deep lea ning. Na u e, 521:436–444.
Li, Y. and Long, P. M. (2000). The elaxed online maximum ma gin algo i hm. In
Ad ances in neu al in o ma ion p ocessing sys ems, pages 498–504.
Lu, J., Zhao, P., and Hoi, S. C. (2016). Online passi e-agg essi e ac i e lea ning.
Machine Lea ning, 103:141–183.
44
BIBLIOGRAPHY
Luong, M.-T. and Manning, C. D. (2015). S an o d neu al machine ansla ion sys-
ems o spoken language domains. In P oceedings o he In e na ional Wo kshop
on Spoken Language T ansla ion.
Madsen, M. W. (2009). The limi s o machine ansla ion ( hesis). Cen e o Lan-
guage Technology, Uni . o Copenhagen, Copenhagen.
Ma ínez-Gómez, P., Sanchis-T illes, G., and Casacube a, F. (2011). Online lea n-
ing ia dynamic e anking o compu e assis ed ansla ion. Compu a ional
Linguis ics and In elligen Tex P ocessing, pages 93–105.
Ma ínez-Gómez, P., Sanchis-T illes, G., and Casacube a, F. (2012). Online adap-
a ion s a egies o s a is ical machine ansla ion in pos -edi ing scena ios.
Pa e n Recogni ion, 45(9):3193–3203.
McCloskey, M. and Cohen, N. J. (1989). Ca as ophic in e e ence in connec ionis
ne wo ks: The sequen ial lea ning p oblem. Psychology o lea ning and mo i a-
ion, 24:109–165.
Mikolo , T., Chen, K., Co ado, G., and Dean, J. (2013a). E icien es ima ion o
wo d ep esen a ions in ec o space. a Xi p ep in a Xi :1301.3781.
Mikolo , T., Ka a iá , M., Bu ge , L., Ce nock`
y, J., and Khudanpu , S. (2010). Re-
cu en neu al ne wo k based language model. In P oceedings o he In e speech,
olume 2, page 3.
Mikolo , T., Komb ink, S., Bu ge , L., ˇ
Ce nock`
y, J., and Khudanpu , S. (2011).
Ex ensions o ecu en neu al ne wo k language model. In Acous ics, Speech
and Signal P ocessing, pages 5528–5531. IEEE.
Mikolo , T., Yih, W.- ., and Zweig, G. (2013b). Linguis ic egula i ies in con in-
uous space wo d ep esen a ions. In P oceedings o he 2013 Con e ence o he
No h Ame ican Chap e o he Associa ion o Compu a ional Linguis ics: Human
Language Technologies (NAACL-HLT-2013), olume 13, pages 746–751.
Och, F. J. (2003). Minimum e o a e aining in s a is ical machine ansla ion. In
P oceedings o he Annual Mee ing o he Associa ion o Compu a ional Linguis ics,
pages 160–167.
Papineni, K., Roukos, S., Wa d, T., and Zhu, W.-J. (2002). Bleu: a me hod o
au oma ic e alua ion o machine ansla ion. In P oceedings o he 40 h annual
mee ing on associa ion o compu a ional linguis ics, pages 311–318.
Pascanu, R., Mikolo , T., and Bengio, Y. (2013). On he di icul y o aining ecu -
en neu al ne wo ks. In e na ional Con e ence on Machine Lea ning (3), 28:1310–
1318.
Penning on, J., Soche , R., and Manning, C. D. (2014). Glo e: Global ec o s o
wo d ep esen a ion. In P oceedings o Empi ical Me hods in Na u al Language
P ocessing, olume 14, pages 1532–1543.
Pe is, Á. (2017). NMT-Ke as.
h ps://gi hub.com/l apeab/nm -ke as
. Gi Hub
eposi o y.
BIBLIOGRAPHY
45
Pe is, Á., Ceb ián, L., and Casacube a, F. (2017a). Online lea ning o neu al
machine ansla ion pos -edi ing. a Xi p ep in a Xi :1706.03196.
Pe is, Á., Domingo, M., and Casacube a, F. (2017b). In e ac i e neu al machine
ansla ion. Compu e Speech & Language, 45:201–220.
Qian, N. (1999). On he momen um e m in g adien descen lea ning algo i hms.
Neu al ne wo ks, 12(1):145–151.
Rosenbla , F. (1958). The pe cep on: A p obabilis ic model o in o ma ion s o -
age and o ganiza ion in he b ain. Psychological e iew, 65:386.
Rumelha , D. E., Hin on, G. E., and Williams, R. J. (1988). Lea ning ep esen a-
ions by back-p opaga ing e o s. Cogni i e modeling, 5(3):1.
Samson, R. (2005). Compu e -assis ed ansla ion. T aining o he New Millennium:
Pedagogies o ansla ion and in e p e ing, 60:101.
Schus e , M. and Paliwal, K. K. (1997). Bidi ec ional ecu en neu al ne wo ks.
IEEE T ansac ions on Signal P ocessing, 45(11):2673–2681.
Shen, S., Cheng, Y., He, Z., He, W., Wu, H., Sun, M., and Liu, Y. (2016). Minimum
isk aining o neu al machine ansla ion. In P oceedings o he 54 h Annual
Mee ing o he Associa ion o Compu a ional Linguis ics, pages 1683–1692.
Sho , N., Zhu benko, N., Likho id, A., and S e syuk, P. (2003). Algo i hms o
nondi e en iable op imiza ion: De elopmen and applica ion. Cybe ne ics and
Sys ems Analysis, 39:537–548.
Sno e , M., Do , B., Schwa z, R., Micciulla, L., and Makhoul, J. (2006). A s udy
o ansla ion edi a e wi h a ge ed human anno a ion. In P oceedings o asso-
cia ion o machine ansla ion in he Ame icas, olume 200, pages 223–231.
Su ske e , I., Vinyals, O., and Le, Q. V. (2014). Sequence o sequence lea ning
wi h neu al ne wo ks. In Ad ances in neu al in o ma ion p ocessing sys ems, pages
3104–3112.
Ta sumi, M. (2009). Co ela ion be ween au oma ic e alua ion me ic sco es, pos -
edi ing speed, and some o he ac o s. P oceedings o he Twel h Machine T ans-
la ion Summi (MT-Summi XII), pages 332–339.
Theano De elopmen Team (2016). Theano: A Py hon amewo k o as com-
pu a ion o ma hema ical exp essions. a Xi e-p in s, abs/1605.02688.
Tiedemann, J. (2009). News om opus-a collec ion o mul ilingual pa allel co -
po a wi h ools and in e aces. In Recen ad ances in na u al language p ocessing,
olume 5, pages 237–248.
Zeile , M. D. (2012). Adadel a: an adap i e lea ning a e me hod. a Xi p ep in
a Xi :1212.5701.
Zens, R., Och, F. J., and Ney, H. (2002). Ph ase-based s a is ical machine ansla-
ion. In Annual Con e ence on A i icial In elligence, pages 18–32. Sp inge .