Escola Tècnica Supe io d’Enginye ia In o mà ica
Uni e si a Poli ècnica de València
Online Lea ning Techniques o Neu al T ansla ion
Sys ems
DEGREE FINAL WORK
Deg ee in Compu e Enginee ing
Au ho : Ke in Mon al á Mingue
Tu o : F ancisco Casacube a Nolla
Ál a o Pe is Ab il
Cou se 2015-2016
Resum
Els sis emes de aducció au omà ica s’han e se i desde la se a concepció pe a-
duc o s p o essionals pe a accele a i acili a la se a asca. Aques os sis emes eben a-
duccions edi ades p o essionalmen a a és del seu ús, lo que po pon ecialmen millo a
el seu endimen . En les úl imes décades, Xa xes Neu onals A i icials s’han u ili za pe
a desen olupa sis emes de aducció au omà ica comple s.
En aques eball s’han u ili za dos algo i mes d’ap enen a ge online pe a millo a
aduc o s neu onals, que ja ha ien acaba l’e apa d’en enamen . Una amplia gamma
d’expe imen s s’han du a e me pe oba els hipe pa àme es óp ims pe els algo i -
mes en cada asca de aducció, i desp és el endimen d’aques os sis emes, adap a s
amb cadascun dels algo i mes ap enen a ge (con igu a s amb aques os hipe pa àme es
óp ims), s’ha mesu a abans i desp és en qua e asques de aducció, i s’han ex e con-
clusions sob e com han millo a o empi jo a .
S’ha modi ica el codi del aduc o neu onal, s’ha implemen a un dels algo i mes
p esen an s i s’ha desen olupa un nou codi pe o e i una in e ície pe a e expe imen s
de o ma au oma i zada. Una al a amilia d’algo i mes ha sigu implemen ada i p obada
amb els models i asques disponibles, sense esul a s posi ius.
Finalmen , línies u u es d’in es igació en adap ació de aduc o s neu onals han si-
gu conside ades i discu ides al inal d’aques eball, a més de la se a si uació en l’es a
ac ual de la aducció au omà ica.
Pa aules clau: ap enen a ge au omà ic, xa xes neu onals, xa xes neu onals ecu en s,
aducció au omà ica, aducció neu onal
Resumen
Los sis emas de aducción au omá ica han sido u ilizados desde su concepción po
aduc o es p o esionales pa a acele a y acili a su a ea. Es os sis emas eciben aduc-
ciones edi adas p o esionalmen e a a és de su uso, que pueden po encialmen e mejo a
su endimien o. En las úl imas décadas, se han u ilizado edes neu onales a i iciales pa a
desa olla sis emas de aducción au omá ica comple os con éxi o.
En es e abajo se han usado dos algo i mos de ap endizaje online pa a mejo a edes
neu onales ya en enadas. Un amplio abanico de expe imen os se ha lle ado a cabo pa a
encon a el conjun o óp imo de hipe pá ame os pa a los algo i mos en cada a ea, y
se ha calculado el endimien o de es os sis emas, adap ados con cada algo i mo con sus
hipe pa áme os óp imos encon ados de o ma empí ica. Se han ex aído conclusiones
ace ca de la mejo a o empeo amien o de los aduc o es.
Se ha lle ado a cabo una modi icación de la base de código del aduc o neu onal,
jun o con la implemen ación de uno de los algo i mos y el desa ollo de una nue a he-
amien a pa a p opo ciona una in e az pa a ealize expe imen os de o ma au oma-
izada. O a amilia de algo i mos ha sido implemen ada y p obada con los modelos y
a eas disponibles, con esul ados insa is ac o ios.
Finalmen e, líneas po enciales de in es igación en adap ación de aduc o es neu o-
nales han sido conside adas y discu idas al inal de es e abajo, jun o con su si uación en
el es ado ac ual de la aducción au omá ica.
Palab as cla e: ap endizaje au omá ico, edes neu onales, edes neu onales ecu en es,
aducción au omá ica, aducción neu onal
iii
i
Abs ac
Machine T ansla ion sys ems ha e been used since hei incep ion by p o essional
ansla o s o speed up and ease hei wo k. Those sys ems ecei e p o essionally edi ed
ansla ions h ough hei use, which could po en ially imp o e hei pe o mance. In he
las decades, A i icial Neu al Ne wo ks ha e been used o de elop comple e Machine
T ansla ion sys ems o g ea success.
In his wo k wo online lea ning algo i hms o A i icial Neu al Ne wo ks ha e been
used o enhance al eady ained neu al ansla o s. A wide a ay o expe imen s ha e
been ca ied ou o ind he op imal hype pa ame e s o he algo i hms in each ask,
and hen he pe o mance o hose sys ems, adap ed wi h each algo i hm wi h hei em-
pi ically ound op imal se o hype pa ame e s, has been measu ed be o e and a e in
ou ansla ion asks, and conclusions ha e been ex ac ed on how hey imp o ed o
wo sened.
A modi ica ion o a neu al ansla o codebase has been ca ied ou , along wi h he
implemen a ion o one o he algo i hms and he de elopmen o new codebase o p o-
ide an in e ace o pe o m expe imen s in an au oma ed way. One addi ional amily o
algo i hms has been implemen ed and es ed wi h he a ailable model and asks o no
a ail.
Finally, possible u u e lines o esea ch on adap a ion o neu al ansla o s ha e been
conside ed and discussed a he end o his wo k, along wi h hei si ua ion in he cu en
s a e o Machine T ansla ion.
Key wo ds: machine lea ning, neu al ne wo ks, ecu en neu al ne wo ks, machine
ansla ion, neu al ansla ion
Con en s
Con en s
Lis o Figu es ii
Lis o Tables ii
Lis o Algo i hms iii
1 In oduc ion 1
1.1 Backg ound .................................... 1
1.2 Mo i a ion..................................... 1
1.3 Goals ........................................ 2
1.4 S uc u e...................................... 3
2 S a e o he a 5
2.1 S a is ical Machine T ansla ion . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2 Neu al Machine T ansla ion . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2.1 Recu en Neu al Ne wo ks . . . . . . . . . . . . . . . . . . . . . . . 7
2.2.2 Bidi ec ional Recu en Neu al Ne wo k . . . . . . . . . . . . . . . 10
2.2.3 Encode -Decode model . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.2.4 T aining .................................. 12
3 Online lea ning 13
3.1 T aining Neu al Ne wo ks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.2 Online lea ning algo i hms . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.2.1 S ochas ic G adien Descen . . . . . . . . . . . . . . . . . . . . . . . 14
3.2.2 AdaG ad.................................. 14
3.2.3 Passi e-Agg essi e............................ 14
4 Expe imen s 19
4.1 So wa e ...................................... 19
4.1.1 Theano................................... 19
4.1.2 G oundHog................................ 19
4.2 Expe imen a ion amewo k . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4.3 Hype pa ame e sea ch ............................. 23
4.3.1 Xe ox ask................................. 23
4.3.2 EU ask................................... 25
4.4 Tes se expe imen s................................ 27
5 Conclusions and u u e wo k 29
Appendices
Ac onyms 31
A. Ac onyms ..................................... 31
Bibliog aphy 33

Lis o Figu es
2.1 Example o Recu en Neu al Ne wo k. . . . . . . . . . . . . . . . . . . . . 7
2.2 Visualiza ion o a RNN un olded in ime. . . . . . . . . . . . . . . . . . . . 8
2.3 Illus a iono heGRU............................... 9
2.4 S uc u eo aBRNN................................ 10
4.1 Di e ences in BLEU in es se . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.2 Execu ion ime o expe imen s in he es se s. . . . . . . . . . . . . . . . . . 28
Lis o Tables
4.1 Co po a cha ac e is ics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.2 Ini ialBLEU..................................... 23
4.3 Hype pa ame e sea ch o he Xe ox ask, om English o Spanish, wi h
SGDalgo i hm. .................................. 24
4.4 Hype pa ame e sea ch o he Xe ox ask, om Spanish o English, wi h
SGDalgo i hm. .................................. 24
4.5 Hype pa ame e sea ch o he Xe ox ask, om English o Spanish, wi h
AdaG adalgo i hm. ............................... 24
4.6 Hype pa ame e sea ch o he Xe ox ask, om Spanish o English, wi h
AdaG adalgo i hm. ............................... 25
4.7 Hype pa ame e sea ch o he EU ask, om English o Spanish, wi h SGD
algo i hm. ..................................... 25
4.8 Hype pa ame e sea ch o he EU ask, om Spanish o English, wi h SGD
algo i hm. ..................................... 26
4.9 Hype pa ame e sea ch o he EU ask, om English o Spanish, wi h
AdaG adalgo i hm. ............................... 26
4.10 Hype pa ame e sea ch o he EU ask, om Spanish o English, wi h
AdaG adalgo i hm. ............................... 26
4.11 BLEU sco e ob ained by e aining on he es se ................ 27
ii
Lis o Algo i hms
3.1 Passi e-Agg essi e app oxima ion pseudocode. . . . . . . . . . . . . . . . 17
4.1 Passi e-Agg essi e app oxima ion execu ion low. . . . . . . . . . . . . . . 21
iii
CHAPTER 1
In oduc ion
1.1 Backg ound
In 1949 Wa en Wea e laid he ounda ions o Machine T ansla ion (MT) by p oposing
he use o compu e s o ackle he challenge o ansla ion. Consequen ly, in he 50s and
60s, se e al a emp s we e made o c ea e p ac ical ansla o s, and by he mid six ies
he newly c ea ed ALPAC (Au oma ic Language P ocessing Ad iso y Commi ee)[15]
published a epo in which hey pain ed he u u e o MT as bleak.
Abou 10 yea s la e none heless, a e he concep ion o some new app oaches and
he inc ease in p ocessing powe o compu e s, he e was a enewal o he expec ancies
o MT, and esea ch and in e es ose back up.
So a , MT is a challenge ha has been engaged wi h mul iple app oaches and me hod-
ologies, bu mos o hem can, nowadays, be classi ied in o h ee ca ego ies[10]: Rule-
Based app oaches, Co pus-Based ones and a hyb id app oach in be ween. Rule-Based
Sys ems (RBS) equi e he use o human knowledge on languages, and he use o g am-
ma s and ocabula ies. Co pus-Based Sys ems (CBSs) use big pa allel co po a o ex o
he sys em o lea n hem and ep oduce hei ea u es. A big ad an age o CBSs o e
Rule-Based Sys ems (RBSs) is ha language da a is eadily a ailable o almos any lan-
guage in he wo ld, a big amoun o in o ma ion in se e al languages. F om books o
news, mo ie sc ip s, ial ansc ip s and laws, he e is a la ge numbe o sou ces om
which da a can be ex ac ed ha Co pus-Based ansla o s can use o imp o e he quali y
o hei p oduc . Since CBSs a e ained on da a, hey also ha e (usually) he possibili y o
scaling up and ou wi h ha dwa e, while RBSs a e usually limi ed by he quali y o hei
ules.
1.2 Mo i a ion
P o essional ansla o s use MT sys ems egula ly o speed up hei ask. The ansla ions
ob ained h ough hose sys ems hough a e o i egula quali y, and many i no mos
o hem mus be amended by human ansla o s o ob ain a sa is ac o y esul . Those
co ec ions can be used o imp o e he MT sys em being used o, in he long e m, educe
he numbe o co ec ions ha he p o essional needs o apply o u u e ansla ions. The
limi being o cou se “pe ec ansla ions”, whe e no co ec ions a e needed.
The co ec ion o machine-gene a ed ansla ions human ansla o s is known as pos -
edi ion. In a pos -edi ion con ex , he human ansla o inpu s a sou ce sen ence, he
so wa e gene a es a hypo hesis in he a ge language and he human co ec s he ans-
la ion. In a simple sys em his co ec ion is only use ul o he use , since he sys em will
1
8S a e o he a
x1
h1
y1
x2
h2
y2
x
h −1
y
...
h
Inpu laye Hidden laye Ou pu laye
Figu e 2.2: Visualiza ion o a RNN un olded in ime. x s ands o he - h inpu p ocessed by he
ne wo k (al e na i ely, a he - h i e a ion), h is he hidden s a e o he ne wo k a i e a ion
and y is he ou pu o he ne wo k a i e a ion .
h = h(x ,h −1)(2.3)
y = o(h )(2.4)
hand odepend on he a chi ec u e o he ne wo k and he choice o ac i a ion unc-
ions.
RNNs, howe e , ha e one well-known sho coming when i comes o sequence p o-
cessing: he anishing g adien p oblem[5]. G adien -based aining algo i hms upda e
he ne wo k weigh s in p opo ion o he g adien o he e o unc ion wi h espec o
each o hem. Since laye s “ u he away” om he ou pu laye (in e ms o connec-
ions) ha e smalle con ibu ions o ou pu alues, his a o emen ioned g adien is in
u n smalle . In he case o sequence-p ocessing RNNs, his means ha in o ma ion om
elemen s a “in he pas ” is almos los . To sol e his p oblem, many RNN-based sys ems
ha e success ully used he RNN a chi ec u es ha a e abou o be discussed.
In he la e 90s, a amily o RNNs appea ed, called Long Sho Te m Memo y (LSTM)
ne wo ks [14], which ha e shown e y good esul s in many sequence lea ning asks.
They make use o a se o ga es o keep some kind o memo y. LSTM uni s a e ained o
e ain di e en deg ees o memo y, which allows RNNs o success ully emembe long
sequences. [25] used LSTMs o build a ela i ely simple neu al ansla o ha achie ed
esul s compa able o hose o s a e-o - he-a SMT sys ems.
Ano he ype o ga ed uni s a e GRUs, de eloped in [7] o a ph ase sco ing sys em
based on Neu al Ne wo ks, as a pa o a Ph ase-Based ansla o . They a e simple han
egula LSTM uni s, hey ha e ewe ga ing uni s, hus educing hei aining ime. They,
howe e , ha e been shown o be as powe ul as LSTM.

2.2 Neu al Machine T ansla ion 9
RESET
GATE
1−
x h −1
h −1
z
anh
⊙
h −1
x
~
h
h −1
h
UPDATE
GATE
x
⊙
⊙
+
Figu e 2.3: Illus a ion o he Ga ed Recu en Uni (GRU). The ese ga e adjus s how much o
he p e iously s o ed in o ma ion is e ained and he upda e ga e adjus s how much o he
newly acqui ed (cu en i e a ion) in o ma ion is kep . x and h ollow he same no a ion as in
Figu e 2.2, while ˜
h ep esen s he upda ed s a e a i e a ion , is he ou pu o he ese ga e a
i e a ion and z is he ou pu o he upda e ga e a i e a ion .
A a gi en ime , a GRU cell holds a hidden s a e h . Gi en i s hidden s a e in he las
i e a ion (a −1), i s cu en upda ed s a e ˜
h and he ou pu o he upda e ga e z , he
cu en hidden s a e will be compu ed as ollows:
h = (1−z )h −1+z ˜
h (2.5)
whe e s ands o he elemen -wise mul iplica ion.
The upda ed s a e ˜
h can be compu ed om he cu en inpu x , he p e ious hidden
s a e h −1and he ou pu o he ese ga e :
˜
h = anh(Wx +U[ h −1]) (2.6)
whe e Wand Ua e weigh ma ices, pa ame e s o he model. Bias e ms ha e been le
ou o eadabili y.
Finally, he ou pu o he ese and upda e ga es a e compu ed ollowing hese o mu-
lae:
=σ(W x +U h −1)(2.7)
z =σ(Wzx +Uzh −1)(2.8)
whe e W a e U ese ga e weigh ma ices, Wzand Uza e upda e ga e weigh ma ices
and σis he elemen -wise logis ic unc ion.
10 S a e o he a
x1
h
1
y1
x
h
−1
y
...
h
Inpu laye
Fo wa d laye Ou pu laye
xJ
h
J−1
yJ
...
...
...
hb
2
hb
hb
+1
hb
J
Backwa d laye
hb
J
h
J
hb
h
hb
1
h
1
Figu e 2.4: S uc u e o a BRNN.
2.2.2. Bidi ec ional Recu en Neu al Ne wo k
In o de o imp o e he quali y o he ansla ions, when i comes o p ocessing a gi en
wo d in he middle o a sen ence, we can choose o examine no only he wo ds ha
p ecede bu also he ones ha ollow i . In o de o do ha , we need o p ocess he
sen ence bo h ways: o wa d and backwa ds. The cu en a chi ec u e o RNN ha has
been in oduced only accoun s o p e ious wo ds, bu we can include an addi ional
hidden laye , independen o he p e ious one, ha will p ocess wo ds om las o i s .
This a chi ec u e, in oduced in [24] as Bidi ec ional Recu en Neu al Ne wo k (BRNN),
will allow he ne wo k o ake decisions based on he whole sen ence con ex .
In his a chi ec u e, he e a e wo hidden s a es: o wa d (h ) and backwa d (hb), ha
a e compu ed as ollows:
h
= h(x ,h −1)(2.9)
hb
= h(x ,h +1)(2.10)
y = o(h
,hb
)(2.11)
2.2.3. Encode -Decode model
In he model p oposed by Cho e al. (2014) and Su ske e e al. (2014), we ha e a sys em
composed o wo RNNs: an Encode and a Decode . The model used in his wo k is he
one p oposed by Bahdanau e al. (2014), which is an ex ension o he a o emen ioned
wo.
2.2 Neu al Machine T ansla ion 11
The Encode , which in ou case is a BRNN, encloses he inpu sen ence in a con ex
ec o . By means o eading each elemen o he inpu sen ence, he hidden s a e o he
ne wo k changes, and a e comple ing he eading i is a compendium o he sen ence.
In ou case, he con ex ec o is ob ained om he conca ena ion o he hidden s a es o
he o wa d ecu en laye and he backwa d ecu en laye .
The Decode in u n has he ask o gene a e he ou pu sen ence om he a o emen-
ioned con ex ec o . I e a i ely, he Decode will gene a e a wo d gi en he con ex
ec o and he p e iously gene a ed wo ds. Once he Decode gene a es a special “end
o line” wo d, he comple e sen ence is he sys em’s hypo hesis o he sou ce sen ence.
Fo each wo d xj(1 ≤j≤J) in he sou ce sen ence, belonging o he sou ce ocab-
ula y Vs, we p oduce a ec o xj∈ {0, 1}Vs, whe e e e y en y is se o ze o excep he
one co esponding o xj, which is se o one. This is called one-ho codi ica ion. Then, he
wo ds a e p ojec ed o a ixed-size con inuous ec o in he ollowing way:
xj=Esxj(2.12)
whe e xjis he embedding o wo d xjand Esis he sou ce language p ojec ion ma ix.
The sequence o wo d embeddings, ep esen ed as x=x1, ...,xJ, is he inpu o he
Encode . A e p ocessing each wo d xj, he hidden s a e o he Encode hjis eco ded.
Since we ha e op ed o using a BRNN as he Encode , ou hidden s a e is ac ually
hj= [h |
j;hb|
j]|(2.13)
Once he sen ence has been ully p ocessed, an i e a i e p ocess begins. A each s ep,
a non-linea unc ion qis applied o he sequence o hidden s a es and o he hidden s a e
o he Decode ne wo k a he p e ious s ep, in o de o ob ain a con ex ec o c. In his
wo k an a en ion mechanism has been used, he e o e he unc ion qis a weigh ed sum
o he hidden s a es. This wo ks like an alignmen model, implemen ed by a Mul ilaye
Pe cep on (MLP), be ween he sou ce sen ence and he a ge sen ence, and hus, we
ha e a di e en con ex ec o ci o each s ep i:
ci=q({h1,...,hJ},gi−1)(2.14)
Thus, ciis a dynamic ep esen a ion o he inpu sen ence, based in he s a e o he
Decode .
This con ex ec o is hen ed o he Decode . The Decode p ocesses he con ex ec-
o , and ou pu s Vs eal numbe s be ween ze o and one, whe e he i- h ou pu ep esen s
he p obabili y o he i- h wo d in he a ge language o be nex in he ansla ion o he
sou ce sen ence. This ou pu depends on ci,giand he wo d embedding ep esen a ion
o he las emi ed wo d. This is implemen ed h ough a so max laye , which ensu es
ha all he p obabili ies add up o one:
y0
k=yk
∑Vs
l=1yl
(2.15)
whe e yi ep esen s he i- h inpu o he so max laye , and y0
i ep esen s i s i- h ou pu ,
always be ween ze o and one.
The e o e, he p obabili y o a wo d a ime-s ep iwould be:
p(yi|y1, ..., yi−1,x;θ) = y|
iϕ(Vη(yi−1,gi,ci)) (2.16)
12 S a e o he a
whe e ϕ(·)is a so max unc ion, yiis he one-ho ec o ep esen a ion o wo d yi,Vis
he weigh ma ix and ηis he ou pu o a RNN wi h GRU uni s and a maxou ou pu
laye [12].
2.2.4. T aining
Following Equa ion (2.1), ou ne wo k aims o app oxima e P(y|x)in he ollowing way:
P(y|x) =
I
∏
i=1
P(yi|y1, ..., yi−1,x)(2.17)
In o de o maximize P(y|x) o ou aining se , consis ing o a bilingual co pus o S
sen ence pai s, and acco ding o Equa ion (2.17), we need o ind a se o pa ame e s o
ou model ˆ
θsuch as:
ˆ
θ=a gmax
θ
S
∏
s=1
I
∏
i=1
p(y(s)
i|y(s)
1, ..., y(s)
i−1,x(s);θ)
=a gmax
θ
S
∑
s=1
I
∑
i=1
log(p(y(s)
i|y(s)
1, ..., y(s)
i−1,x(s);θ)) (2.18)
whe e x(s)and y(s) ep esen he s- h sen ence o he aining se in he sou ce and a ge
languages espec i ely, and Iis he leng h o he s- h a ge sen ence.
Since each wo d o he sys em’s hypo hesis depends on p e iously gene a ed wo ds
and he con ex ec o , and he con ex ec o depends only on he sou ce sen ence, bo h
Encode and Decode can be ained as a whole o maximize he condi ional p obabili y
o he a ge sen ences gi en he sou ce sen ences.
So a , we ha e in oduced he knowledge ield o Machine T ansla ion. We ha e ou -
lined he app oaches ha ha e been adop ed in he las decades o ad ance he quali y o
ansla o s, and we ha e desc ibed s a e-o - he-a echniques ha powe he op ans-
la ion so wa e in he ield. Finally, we ha e e iewed in dep h he Encode -Decode
app oach and he Neu al Ne wo ks employed in i . Wha ollows is a desc ip ion o he
wo k ha was ca ied ou in o de o pe o m he expe imen s.
CHAPTER 3
Online lea ning
3.1 T aining Neu al Ne wo ks
Ou objec i e in aining is o maximize he log-likelihood o he da a we use o aining,
in an a emp o p oduce a sys em ha can gene alize ha se o ansla ions in o he
o e all ansla ion ask. By ollowing Equa ion (2.18), we can une he sys em pa ame e s
θ o maximize his sum o log-p obabili ies. The ainable pa ame e s o RNNs a e he
weigh ma ices. Mos Neu al Ne wo k aining algo i hms a e i e a i e, hey upda e he
ne wo k weigh s acco ding o a ule s ep by s ep, un il a gi en condi ion is accomplished.
Upda e ules, s ep size and s opping condi ion de ine he di e en lea ning echniques.
The ollowing is a classi ica ion by s ep size[19]:
•Ba ch lea ning echniques a e hose ha upda e he weigh s o he ne wo k a e
he whole aining se has been p ocessed. Once he sys em has e alua ed e e y
sample, i s weigh s a e upda ed o i hose, and a new aining i e a ion begins. I
he e mina ion condi ion is eached, he aining s ops.
•Mini-ba ch lea ning echniques indica e ha he upda es mus be applied a e an
a bi a y numbe o samples ha e been p ocessed.
•Online lea ning echniques, inally, a e hose ha upda e he ne wo k weigh s a e
e e y single sample. They a e a pa icula in e es o us, on accoun o hem being
a pe ec i o he si ua ion desc ibed a he beginning o his wo k: imp o ing a
sys em a e a new sample is ob ained.
Wi h he goal o enhancing a neu al ansla o wi h i s use, a se up like he ollow-
ing can be adop ed: he sys em gene a es ansla ions o a human ansla o , who
inpu s co ec ed e sions o hose ansla ions, which he sys em uses one by one
o upda e he weigh s o i s in e nal Neu al Ne wo k.
3.2 Online lea ning algo i hms
In he nex sec ions we desc ibe he algo i hms we ha e chosen o compa e.
13

14 Online lea ning
3.2.1. S ochas ic G adien Descen
SGD is a lea ning algo i hm[22] ha app oxima es G adien Descen by upda ing he
weigh s o he sys em using he ollowing ule:
θs=θs−1−η∇`s(θs−1)(3.1)
This upda e is pe o med wi h he g adien o each sample o he aining se , hus
app oxima ing he g adien o he whole se .
In (3.1) ηis he lea ning a e, which se s he pace o he upda es o he model. This is
he only pa ame e we can une in his algo i hm. I is usually se o alues lowe han
one, in o de o modi y he model in small s eps owa ds minima in he e o unc ion.
E e y ime he weigh s o he ne wo k a e upda ed, he sys em ies o imp o e i s pe -
o mance owa ds he new sample, and in he p ocess, i s pe o mance wi h p e iously
seen samples may ge wo se. In o de o y o achie e a good pe o mance in he a ge
da a o he sys em as a whole, he lea ning a e is used. I he lea ning a e we e oo
high, he sys em would agg essi ely y o i new samples a he expense o pas da a,
esul ing likely in an o e all bad pe o mance. I i we e oo small, he ne wo k would
conse a i ely lea n he da a, equi ing a e y high numbe o i e a ions, and hus a e y
long ime, o be ained.
3.2.2. AdaG ad
AdaG ad is a amily o adap i e, subg adien , online lea ning algo i hms de eloped in
[11], based on SGD, ha is expec ed o ou pe o m i o high-dimensional, spa se ea-
u es. The implemen a ion used in his wo k is an app oxima ion ob ained om [13]. I s
upda e ule is as ollows:
s= s−1−ηG−1/2
s∇ `(3.2)
Gs=Gs−1+ (∇ `)2(3.3)
whe e is any gi en weigh o θ,∇ ` ep esen s he g adien o he loss unc ion wi h
he p e ious weigh se wi h espec o weigh be o e p ocessing sample sand Gs ep-
esen s he sum o squa ed g adien s be o e p ocessing sample s. A any gi en ime,
Gs=∑s
i=1∇ `2
i.
In his case, we also ha e a lea ning a e pa ame e ha we can une in o de o seek
he op imal pe o mance o he algo i hm.
3.2.3. Passi e-Agg essi e
Passi e-Agg essi e a e a amily o ma gin-based online lea ning algo i hms, p oposed in
[8]. The goal o hose algo i hms is o ind, a each s ep, he model which, being as close
as possible o he cu en one, achie es some gi en ma gin on he cu en sample. This is
a cons ain op imiza ion p oblem ha is sol ed by he Lag ange mul iplie s echnique
o ind an upda e ule ha mee s he condi ions. Since he ma gin equi emen migh be
a ha d one, he PA-II and PA-III algo i hms include an “agg essi eness” hype pa ame e
ha allows o a ade-o be ween he desi ed ma gin and he p oximi y o he cu en
model.
3.2 Online lea ning algo i hms 15
These algo i hms ha e he ollowing upda e ule:
θs+1=θs+sign(ys−ˆys)τsxs(3.4)
whe e τsdepends on he pa icula algo i hm:
τs=






`s
||xs||2, PA
min(C,`s
||xs||2), PA-I
`s
||xs||2+1
2C
, PA-II
(3.5)
whe e Cis a pa ame e called agg essi eness in [8] and `sis he alue o he loss unc ion
a ime .
As we can see in Equa ion (3.4) and Equa ion (3.5), PA-I has no hype pa ame e s and
PA-I and PA-II ha e one: C.
In o de o sol e ˆ
θ=a gminθ1
2||θ−θs||2s. . `(θ,xs,ys,hs)≤0 we use he Lag ange
mul iplie s echnique:
`(θ,xs,ys,hs) = log pˆ
θ(hs|xs)−log pˆ
θ(ys|xs)(3.6)
We s a by ob aining Lag ange unc ion:
L(θ,λ) = 1
2||θ−θs||2+λ`(θ,xs,ys,hs)(3.7)
whe e λis a Lag ange mul iplie .
Nex we ob ain he g adien , which would be ze o a he minimum:
∇θL(θ,λ) = θ−θs+λ∇θ`(θ,xs,ys,hs) = 0 (3.8)
θ=θs−λ∇θ`(θ,xs,ys,hs)(3.9)
A e wa ds, we ge he pseudo-dual unc ion:
LD(θ,λ) = 1
2λ2||∇θ`(θ,xs,ys,hs)||2+λ`(θ,xs,ys,hs)(3.10)
As we did be o e, we look o he minimum:
∂LD(θ,λ)
∂λ =λ||∇θ`(θ,xs,ys,hs)||2+`(θ,xs,ys,hs) = 0 (3.11)
ˆ
λ=−`(θ,xs,ys,hs)
||∇θ`(θ,xs,ys,hs)||2(3.12)
Which is he op imal solu ion o he Lag ange mul iplie λ. Along wi h Equa ion (3.9),
we can ob ain he pseudo op imal solu ion:
θ=θs+`(θ,xs,ys,hs)∇θ`(θ,xs,ys,hs)
||∇θ`(θ,xs,ys,hs)||2(3.13)
PA-I equi es sol ing ˆ
θ=a gminθ1
2||θ−θs||2+Cξs. . `(θ,xs,ys,hs)≤ξ, which is
done simila ly o equa ions 3.7 o 3.13:
16 Online lea ning
L(θ,λ1,λ2) = 1
2||θ−θs||2+Cξ+λ1(`(θ,xs,ys,hs)−ξ)−λ2ξ
∇θL(θ,λ1,λ2) = θ−θs+λ1∇θ`(θ,xs,ys,hs) = 0
θ=θs−λ1∇θ`(θ,xs,ys,hs)
∂L(θ,λ1,λ2)
∂ξ =C−λ1−λ2=0→C=λ1+λ2
LD(θ,λ1,λ2) = 1
2λ2
1||∇θ`(θ,xs,ys,hs)||2+Cξ
+λ1`(θ,xs,ys,hs)−(λ1+λ2)ξ
∂LD(θ,λ1,λ2)
∂λ1=λ1||∇θ`(θ,xs,ys,hs)||2+`(θ,xs,ys,hs) = 0
ˆ
λ1=min(C,−`(θ,xs,ys,hs)
||∇θ`(θ,xs,ys,hs)||2)
θ=θs−min(C,−`(θ,xs,ys,hs)
||∇θ`(θ,xs,ys,hs)||2)∇θ`(θ,xs,ys,hs)(3.14)
whe eas PA-II equi es sol ing
ˆ
θ=a gmin
θ
1
2||θ−θs||2+Cξ2s. . `(θ,xs,ys,hs)≤ξ
L(θ,λ) = 1
2||θ−θs||2+Cξ2+λ(`(θ,xs,ys,hs)−ξ)
∇θL(θ,λ) = θ−θs+λ∇θ`(θ,xs,ys,hs) = 0
θ=θs−λ∇θ`(θ,xs,ys,hs)
∂L(θ,λ)
∂ξ =2Cξ−λ=0→ξ=λ
2C
LD(θ,λ) = 1
2λ2||∇θ`(θ,xs,ys,hs)||2+Cλ
2C2
+λ`(θ,xs,ys,hs)−λλ
2C
∂LD(θ,λ)
∂λ =λ||∇θ`(θ,xs,ys,hs)||2+λ
2C+`(θ,xs,ys,hs)−λ
C=0
ˆ
λ1=−`(θ,xs,ys,hs)
||∇θ`(θ,xs,ys,hs)||2−1
2C
θ=θs+`(θ,xs,ys,hs)∇θ`(θ,xs,ys,hs)
||∇θ`(θ,xs,ys,hs)||2−1
2C
(3.15)
As can be seen in Equa ion (3.13), θis ound in bo h sides o he equa ion, hus
he need o he app oxima ion using ixed-poin i e a o s, which can be seen in Al-
go i hm 3.1 o he PA algo i hm, while PA-I and PA-II a e iden ical, equi ing only a
change in he upda e line o he co esponding o mula, o be like Equa ion (3.14) and
Equa ion (3.15).
Al hough he implemen a ion is u he explained in Sec ion 4.1.2, i is wo h no ing
ha in he p elimina y expe imen s no p omising esul s we e achie ed wi h ei he o he
h ee e sions o he algo i hm, and he e o e i was d opped om he expe imen a ion
plan owa ds he end o he p ojec .
3.2 Online lea ning algo i hms 17
Inpu :
Pa ame e s a he beginning o he i e a ion θs
Sou ce sen ence xs
Ta ge sen ence ys
Hypo hesis hs
Ou pu : θnew
Ini ializa ion: θnew =θs
epea
1. θold =θnew
2. θnew =θs+`(θold,xs,ys,hs)∇θ=θold `(θ,xs,ys,hs)
||∇θ=θold `(θ,xs,ys,hs)||2
un il θold == θnew
Algo i hm 3.1: Passi e-Agg essi e app oxima ion pseudocode.
24 Expe imen s
I e a ions Lea ning a e
0.05 0.1 0.2 0.4 0.8
1 66.2 66.0 65.9 64.2 61.0
3 66.0 66.2 - - -
5 66.4 66.2 65.3 60.9 -
10 66.5 66.4 66.1 60.8 -
20 66.6 66.8 - - -
Table 4.3: Hype pa ame e sea ch o he Xe ox ask, om English o Spanish, wi h SGD algo-
i hm.
I e a ions Lea ning a e
0.05 0.1 0.2 0.4 0.8
1 69.9 69.9 69.8 68.7 63.7
370.1 69.8 70.6 68.4 61.3
5 70.5 70.0 69.5 69.3 60.0
Table 4.4: Hype pa ame e sea ch o he Xe ox ask, om Spanish o English, wi h SGD algo-
i hm.
SGD
Table 4.3 shows gene al bu small imp o emen using a lea ning a e lowe han 0.2. I
is possible ha sligh ly be e esul s could be achie ed wi h e en lowe lea ning a es,
bu hose expe imen s ell ou side he scope o his wo k. The se o hype pa ame e s
(0.05,5)was selec ed as a ade-o be ween pe o mance and quali y, since he bes esul
(66.8) was achie ed pe o ming 20 i e a ions pe sample, se up which would be liable o
slowing down oo much ansla ion so wa e.
Table 4.4 none heless showed much be e esul s han Table 4.3, a guably because o
di e ences in he models (di e en numbe o hidden nodes in hei ne wo ks, di e en
aining ime...), o maybe because his ask was easie han he o me (is ansla ing
om English o Spanish ha de han om Spanish o English?). The bes esul in his
ba ch o expe imen s achie ed an imp o emen o 5.3 poin s in he BLEU sco e.
AdaG ad
Acco ding o Table 4.5, he model in he English-Spanish ask imp o ed he mos by
pe o ming 20 aining i e a ions pe sample wi h a lea ning a e o 0.0001, and jus as
well by pe o ming 3 i e a ions pe sample wi h a lea ning a e o 0.0005. Since speed
I e a ions Lea ning a e
5e-5 1e-4 5e-4 1e-3
1 67.5 67.7 69.0 67.7
367.7 68.2 69.4 68.0
5 67.8 68.8 69.3 68.3
10 68.3 69.1 68.6 67.6
20 68.9 69.4 68.9 66.6
Table 4.5: Hype pa ame e sea ch o he Xe ox ask, om English o Spanish, wi h AdaG ad
algo i hm.

4.3 Hype pa ame e sea ch 25
I e a ions Lea ning a e
5e-5 1e-4 5e-4 1e-3 5e-3
1 69.9 69.8 71.0 70.8 59.8
3 69.8 70.2 71.1 - -
569.8 70.1 71.7 70.5 57.6
Table 4.6: Hype pa ame e sea ch o he Xe ox ask, om Spanish o English, wi h AdaG ad
algo i hm.
I e a ions Lea ning a e
0.05 0.1 0.2 0.4 0.8
1 35.5 35.3 35.4 34.8 33.3
335.3 35.5 36.1 35.6 35.4
5 35.4 35.6 35.8 36.1 -
Table 4.7: Hype pa ame e sea ch o he EU ask, om English o Spanish, wi h SGD algo i hm.
is impo an in he con ex o his wo k, he la e has been chosen as he op imal se o
hype pa ame e s o AdaG ad in his ask.
I is wo h no ing ha none o he esul s o his able esul in a decline in he pe o -
mance. Mo eo e , he a e age imp o emen was o 2.3 pe cen age poin s, which shows
ha AdaG ad achie ed a much be e o e all imp o emen in his se han SGD. Ne -
e heless, his ask is being used o hype pa ame e sea ch, so conclusions mus no be
d awn om his compa ison, gi en ha we pu posely selec he bes esul o each o
hem.
In Table 4.6 we see imp o emen s o he same o de o Table 4.4, which leads o hink
ha his ask is qui e adep a being e ained.
4.3.2. EU ask
The EU co pus[16] was ob ained om he Bulle in o he Eu opean Union, which is pub-
licly a ailable in all he o icial languages o he Eu opean Union. As in he Xe ox asks,
only Spanish o English and English o Spanish we e used. The co pus was okenized as
well, bu no ans o med o lowe case as we did in Sec ion 4.3.1.
SGD
Table 4.7 shows a case no encoun e ed so a : none o he esul s show an imp o emen
in he quali y o ansla ions. The e a e se e al ac o s ha can con ibu e o his phe-
nomenon. Fi s o all, his se is much smalle han ha o he p e iously shown asks:
400 sen ence pai s e sus 1012 in Xe ox. Since he BLEU sco e is a measu e ha a emp s
o ma ch human judgemen when a e aged o e a co pus[21], i s alue when he se is
small can be expec ed o be less eliable han he case whe e i is applied o a big co pus.
Fu he mo e, he domain o he EU co pus is in all likelihood mo e complex han he
domain o Xe ox. Xe ox co pus is ull o sho sen ences, wi h e y epe i i e wo ds, like
p in e ea u es and op ions. Numbe s in Xe ox a e mos o he ime model iden i ie s,
which do no change. EU bulle in is e y di e se, has long sen ences, a high quan i y
o numbe s o eco ds, da es, pe cen ages, ile sizes, and so on. Also, he ini ial BLEU in
bo h could no be any mo e di e en , and he ocabula y size in bo h languages in Xe ox
is less han hal han in he EU models.
26 Expe imen s
I e a ions Lea ning a e
0.05 0.1 0.2 0.4 0.8
1 35.1 35.1 35.1 35.3 34.4
335.0 34.8 35.2 36.0 35.5
5 35.1 35.7 35.8 35.8 34.8
Table 4.8: Hype pa ame e sea ch o he EU ask, om Spanish o English, wi h SGD algo i hm.
I e a ions Lea ning a e
5e-5 1e-4 5e-4 1e-3
1 35.9 35.4 35.7 35.0
3 36.1 35.7 36.1 35.5
535.8 36.2 35.7 33.7
Table 4.9: Hype pa ame e sea ch o he EU ask, om English o Spanish, wi h AdaG ad algo-
i hm.
We can say wi hou a shade o doub ha he Xe ox ansla ion models in his wo k
a e mo e e ec i e han hei EU coun e pa s. I emains a ques ion whe he his makes
he ask o e aining easie o ha de , o whe he i is inma e ial o i . Fu he esea ch
would be necessa y o answe i .
Finally, gi en he assump ions o co ela ion be ween model and algo i hm pe o -
mances in di e en asks ha we e issued in he in oduc ion o his wo k, we mus as-
sume oo ha he expe imen wi h he es se ha in ol es his pa icula con igu a ion
will de e io a e he model pe o mance, bu no as much as wi h he o he pa ame e s
ha we e ied.
A small imp o emen can be obse ed in some o he en ies o Table 4.8. E en hose
se ups ha esul ed in a decline in quali y o ansla ions did so only by a e y small
amoun . This could be a ibu ed o he small size o he de elopmen se , in compa ison
o he o he co pus.
AdaG ad
The esul s in Table 4.9 appea o be e y simila o hose in Table 4.7: no imp o emen
in any case. This can likely be a ibu ed o he same hypo he ical easons ha we e
gi en o he esul s in Table 4.7. The bes esul o he able was chosen o u h e
expe imen a ion.
Table 4.10 shows sligh ly mo e p omising esul s han Table 4.8, e en hough sligh ly
ewe cases we e a emp ed.
I e a ions Lea ning a e
5e-5 1e-4 5e-4 1e-3
1 35.3 35.7 35.8 35.0
3 35.5 36.1 35.4 35.5
535.7 35.9 36.3 35.8
Table 4.10: Hype pa ame e sea ch o he EU ask, om Spanish o English, wi h AdaG ad algo-
i hm.
4.4 Tes se expe imen s 27
Task BLEU
Ini ial SGD AdaG ad
Xe ox En-Es 55.2 55.9 57.1
Xe ox Es-En 46.1 51.7 50.4
EU En-Es 36.8 36.4 36.4
EU Es-En 35.7 34.8 36.0
Table 4.11: BLEU sco e ob ained in he expe imen s on he es se o each ask, bo h wi hou e-
aining and by e aining wi h each algo i hm using he hype pa ame e s selec ed in Sec ion 4.3.
Xe ox En-Es Xe ox Es-En EU En-Es EU Es-En
−1
0
1
2
3
4
5
6
0.7
5.6
−0.4
−0.9
1.9
4.3
−0.4
0.3
∆BLEU
SGD AdaG ad
Figu e 4.1: Di e ences in BLEU ob ained in he es se o each ask by e aining wi h each al-
go i hm, using he hype pa ame e s selec ed in Sec ion 4.3. The alues used can be ound in
Table 4.11.
4.4 Tes se expe imen s
F om he da a in Table 4.11 we see ha AdaG ad ou pe o ms SGD in wo o ou ou
cases, and ma ches i in ano he one. The da a, howe e , is insu icien o d aw con-
clusions abou AdaG ad being in gene al be e o his scena io. SGD ou pe o med
AdaG ad in Xe ox Spanish-English by a ela i ely la ge ma gin, in compa ison wi h he
o he esul s. The imp o emen achie ed by each algo i hm on each model is plo ed in
Figu e 4.1.
Bo h algo i hms showed conside ably be e esul s in he Xe ox asks han in he EU
ones, which could be a ibu ed o, among o he hings, he ask complexi y and/o he
al eady-exis ing model pe o mance. The models ha ob ained a high BLEU ini ially im-
p o ed hei esul s, hey p obably ook p o i o he da a and enhanced hei pa ame e s
as o be e i he ansla ion ask. The models ha ob ained a ela i ely low BLEU in he
ini ial measu emen may ha e no been able o assimila e he new da a and may ha e
28 Expe imen s
Xe ox En-Es Xe ox Es-En EU En-Es EU Es-En
0
10
20
30
40
50
60
Execu ion ime / sen ence pai s (s)
SGD AdaG ad
Figu e 4.2: Execu ion ime o expe imens in he es se s.
been nega i ely a ec ed by he pa ame e upda es (as hey may ha e in he expe imen s
wi h he de elopmen se ), bu e en hen, he quali y o he hypo heses was no much
wo se o , and AdaG ad algo i hm could achie e an imp o emen in he EU Es-En ask
The decline in pe o mance obse ed by e aining in bo h EU asks in p e ious ex-
pe imen s is consis en wi h hese esul s. The Xe ox English-Spanish ansla ion ask has
p oduced much di e en esul s han p io expe imen s, ob aining much lowe imp o e-
men s, while Xe ox Spanish-English has main ained a high deg ee o imp o emen wi h
bo h algo i hms.
In Figu e 4.2 we can obse e he di e en execu ion imes o each algo i hm, a e aged
o e he numbe o sen ence pai s o he ele an se . We can obse e ha he execu ion
ime o EU asks was much longe han o Xe ox asks, likely due o hei longe sen-
ences and highe ocabula y size (which makes hei models e en la ge ).
CHAPTER 5
Conclusions and u u e wo k
We ha e de eloped a ull- ledged expe imen a ion en i onmen o online adap a ion o
neu al ansla o s, able o pe o m au oma ed ba ches o expe imen s wi h p ope han-
dling and s o ing o he esul s. We ha e selec ed and p ocessed da a o expe imen s,
and pe o med se e al o hem wi h di e en se s o hype pa ame e s, and ha e ound
he ones among hem ha wo k bes o each combina ion o ask and algo i hm. Wi h a
di e en se bu o he same ask, we ha e pe o med expe imen s wi h bo h algo i hms
con igu ed o hei empi ically- ound bes capaci ies and ha e ob ained a compa ison o
how he algo i hms can imp o e o wo sen he quali y o neu al ansla o s.
We ha e ob ained an i e a i e app oxima ion o he Passi e-Agg essi e amily o on-
line lea ning algo i hms, p oduced an implemen a ion o ou expe imen a ion ame-
wo k and obse ed he lack o p omising esul s. Due o his, no u he expe imen a ion
was ca ied ou wi h hose algo i hms.
In h ee ou o ou ansla ion asks, we obse ed a sligh co ela ion be ween he
esul s in he de elopmen se and he esul s in he es se . AdaG ad algo i hm esul ed
mo e p omising han SGD, bu he la e ou pe o med he o me in one ask. All hings
conside ed, we canno s a e ha AdaG ad would be he bes choice o e e y ask, bu a
p io i i is a be e candida e.
Se e al ques ions emain unanswe ed, such as o why is he e such dispa i y in he
esul s be ween de elopmen se and es se in he Xe ox English-Spanish ask, o why he
EU asks pu up such a challenge agains bo h algo i hms. Fu he esea ch is equi ed o
answe hose ques ions.
Mo e expe imen s, in ol ing mo e da ase s and mo e language pai s a e equi ed,
since neu al ansla o s a e usually ained and used o single language pai s, and he
esul s obse ed using one o hem may no co espond a all wi h esul s ob ained om
a di e en one. In his wo k we ha e used wo co po a and one language-pai (in bo h di-
ec ions), and e en hen we ha e ound signi ican di e ences be ween asks (especially
be ween bo h di ec ions o he Xe ox English o/ om Spanish asks).
The size o he EU de elopmen se , especially when compa ed o ha o he Xe ox
co pus, aises he ques ion o whe he he hype pa ame e sea ch o he EU asks may
ha e been comp omised, o a he leas whe he he size o his se has handicapped he
hype pa ame e sea ch o hose models. Mo e simila pai s o se s could be used o help
dispel his doub .
Finally, he app oxima ion o Passi e-Agg essi e algo i hms mus be e ised in sea ch
o al e na i e me hods o ixed poin i e a o s, o e en ually es hem agains he chal-
lenges p esen ed in his wo k, and compa e hem o he o he wo algo i hms we ha e
used.
29

Ac onyms
A. Ac onyms
BLEU . . . . . . . . . . . BiLingual E alua ion Unde s udy
BRNN . . . . . . . . . . . Bidi ec ional Recu en Neu al Ne wo k
CBS . . . . . . . . . . . . Co pus-Based Sys em
GRU . . . . . . . . . . . . Ga ed Recu en Uni
LSTM . . . . . . . . . . . Long Sho Te m Memo y
MLP . . . . . . . . . . . . Mul ilaye Pe cep on
MT . . . . . . . . . . . . Machine T ansla ion
NLP . . . . . . . . . . . . Na u al Language P ocessing
NMT . . . . . . . . . . . Neu al Machine T ansla ion
RBS . . . . . . . . . . . . Rule-Based Sys em
RNN . . . . . . . . . . . Recu en Neu al Ne wo k
SGD . . . . . . . . . . . . S ochas ic G adien Descen
SMT . . . . . . . . . . . . S a is ical Machine T ansla ion
31
Bibliog aphy
[1] Dzmi y Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neu al machine ansla-
ion by join ly lea ning o align and ansla e. a Xi p ep in a Xi :1409.0473, 2014.
[2] F édé ic Bas ien, Pascal Lamblin, Raz an Pascanu, James Be gs a, Ian Good el-
low, A naud Be ge on, Nicolas Boucha d, Da id Wa de-Fa ley, and Yoshua Bengio.
Theano: new ea u es and speed imp o emen s. a Xi p ep in a Xi :1211.5590, 2012.
[3] Je ome R Bellega da. A la en seman ic analysis amewo k o la ge-span language
modeling. In EUROSPEECH, 1997.
[4] Yoshua Bengio, Réjean Ducha me, Pascal Vincen , and Ch is ian Jan in. A neu al
p obabilis ic language model. J. Mach. Lea n. Res., 3:1137–1155, Ma ch 2003.
[5] Yoshua Bengio, Pa ice Sima d, and Paolo F asconi. Lea ning long- e m dependen-
cies wi h g adien descen is di icul . Neu al Ne wo ks, IEEE T ansac ions on, 5(2):157–
166, 1994.
[6] M Asunción Cas ano, F ancisco Casacube a, and En ique Vidal. Machine ansla-
ion using neu al ne wo ks and ini e-s a e models. Theo e ical and Me hodological
Issues in Machine T ansla ion (TMI), pages 160–167, 1997.
[7] Kyunghyun Cho, Ba Van Me iënboe , Cagla Gulceh e, Dzmi y Bahdanau, Fe hi
Bouga es, Holge Schwenk, and Yoshua Bengio. Lea ning ph ase ep esen a-
ions using nn encode -decode o s a is ical machine ansla ion. a Xi p ep in
a Xi :1406.1078, 2014.
[8] Koby C amme , O e Dekel, Joseph Keshe , Shai Shale -Shwa z, and Yo am Singe .
Online passi e-agg essi e algo i hms. J. Mach. Lea n. Res., 7:551–585, Decembe
2006.
[9] Jacob De lin, Rabih Zbib, Zhongqiang Huang, Thomas Lama , Richa d M Schwa z,
and John Makhoul. Fas and obus neu al ne wo k join models o s a is ical ma-
chine ansla ion. In ACL (1), pages 1370–1380. Ci esee , 2014.
[10] Bonnie J Do , Pamela W Jo dan, and John W Benoi . A su ey o cu en pa adigms
in machine ansla ion. Ad ances in compu e s, 49:1–68, 1999.
[11] John Duchi, Elad Hazan, and Yo am Singe . Adap i e subg adien me hods o
online lea ning and s ochas ic op imiza ion. The Jou nal o Machine Lea ning Resea ch,
12:2121–2159, 2011.
[12] Ian J Good ellow, Da id Wa de-Fa ley, Mehdi Mi za, Aa on Cou ille, and Yoshua
Bengio. Maxou ne wo ks. a Xi p ep in a Xi :1302.4389, 2013.
33