P oceedings o he 2020 Con e ence on Empi ical Me hods in Na u al Language P ocessing, pages 7674–7684,
No embe 16–20, 2020. c
2020 Associa ion o Compu a ional Linguis ics
7674
T ansla ion A i ac s in C oss-lingual T ans e Lea ning
Mikel A e xe, Go ka Labaka, Eneko Agi e
HiTZ Cen e
Uni e si y o he Basque Coun y (UPV/EHU)
{mikel.a e xe,go ka.labaka,e.agi e}@ehu.eus
Abs ac
Bo h human and machine ansla ion play a
cen al ole in c oss-lingual ans e lea ning:
many mul ilingual da ase s ha e been c ea ed
h ough p o essional ansla ion se ices, and
using machine ansla ion o ansla e ei he
he es se o he aining se is a widely used
ans e echnique. In his pape , we show ha
such ansla ion p ocess can in oduce sub le
a i ac s ha ha e a no able impac in exis ing
c oss-lingual models. Fo ins ance, in na u-
al language in e ence, ansla ing he p emise
and he hypo hesis independen ly can educe
he lexical o e lap be ween hem, which cu -
en models a e highly sensi i e o. We show
ha some p e ious indings in c oss-lingual
ans e lea ning need o be econside ed in he
ligh o his phenomenon. Based on he gained
insigh s, we also imp o e he s a e-o - he-a
in XNLI o he ansla e- es and ze o-sho ap-
p oaches by 4.3 and 2.8 poin s, espec i ely.
1 In oduc ion
While mos NLP esou ces a e English-speci ic,
he e ha e been se e al ecen e o s o build
mul ilingual benchma ks
. One possibili y is o
collec and anno a e da a in mul iple languages
sepa a ely (Cla k e al.,2020), bu mos exis -
ing da ase s ha e been c ea ed h ough ansla ion
(Conneau e al.,2018;A e xe e al.,2020). This ap-
p oach has wo desi able p ope ies: i elies on ex-
is ing p o essional ansla ion se ices a he han
equi ing expe ise in mul iple languages, and i
esul s in pa allel e alua ion se s ha o e a mean-
ing ul measu e o he c oss-lingual ans e gap
o di e en models. The esul ing mul ilingual
da ase s a e gene ally used o e alua ion only, e-
lying on exis ing English da ase s o aining.
Closely ela ed o ha ,
c oss-lingual ans e
lea ning
aims o le e age la ge da ase s a ail-
able in one language— ypically English— o build
mul ilingual models ha can gene alize o o he
languages. P e ious wo k has explo ed 3 main
app oaches o ha end: machine ansla ing he
es se in o English and using a monolingual En-
glish model (TRANSLATE-TEST), machine ansla -
ing he aining se in o each a ge language and
aining he models on hei espec i e languages
(TRANSLATE-TRAIN), o using English da a o ine-
une a mul ilingual model ha is hen ans e ed
o he es o languages (ZERO-SHOT).
The da ase c ea ion and ans e p ocedu es de-
sc ibed abo e esul in a
mix u e o o iginal,1
human ansla ed and machine ansla ed da a
when dealing wi h c oss-lingual models. In ac ,
he ype o ex a sys em is ained on does no
ypically ma ch he ype o ex i is exposed o a
es ime: TRANSLATE-TEST sys ems a e ained on
o iginal da a and e alua ed on machine ansla ed
es se s, ZERO-SHOT sys ems a e ained on o ig-
inal da a and e alua ed on human ansla ed es
se s, and TRANSLATE-TRAIN sys ems a e ained on
machine ansla ed da a and e alua ed on human
ansla ed es se s.
Despi e o e looked o da e, we show ha
such
misma ch has a no able impac
in he pe o -
mance o exis ing c oss-lingual models. By using
back- ansla ion (Senn ich e al.,2016) o pa a-
ph ase each aining ins ance, we ob ain ano he
English e sion o he aining se ha be e e-
sembles he es se , ob aining subs an ial imp o e-
men s o he TRANSLATE-TEST and ZERO-SHOT ap-
p oaches in c oss-lingual Na u al Language In e -
ence (NLI). While imp o emen s b ough by ma-
chine ansla ion ha e p e iously been a ibu ed
o da a augmen a ion (Singh e al.,2019), we e-
jec his hypo hesis and show ha he phenomenon
is only p esen in ansla ed es se s, bu no in
o iginal ones. Ins ead, ou analysis e eals ha
1We use he e m o iginal o e e o non- ansla ed ex .
7675
his beha io is caused by sub le
a i ac s a ising
om he ansla ion
p ocess i sel . In pa icula ,
we show ha ansla ing di e en pa s o each
ins ance sepa a ely (e.g., he p emise and he hy-
po hesis in NLI) can al e supe icial pa e ns in he
da a (e.g., he deg ee o lexical o e lap be ween
hem), which se e ely a ec s he gene aliza ion
abili y o cu en models. Based on he gained in-
sigh s, we imp o e he s a e-o - he-a in XNLI,
and show ha some p e ious indings need o be
econside ed in he ligh o his phenomenon.
2 Rela ed wo k
C oss-lingual ans e lea ning.
Cu en c oss-
lingual models wo k by p e- aining mul ilingual
ep esen a ions using some o m o language mod-
eling, which a e hen ine- uned on he ele an
ask and ans e ed o di e en languages. Some
au ho s le e age pa allel da a o ha end (Conneau
and Lample,2019;Huang e al.,2019), bu aining
a model akin o BERT (De lin e al.,2019) on he
combina ion o monolingual co po a in mul iple
languages is also e ec i e (Conneau e al.,2020).
Closely ela ed o ou wo k, Singh e al. (2019)
showed ha eplacing segmen s o he aining da a
wi h hei ansla ion du ing ine- uning is help-
ul. Howe e , hey a ibu e his beha io o a da a
augmen a ion e ec , which we belie e should be
econside ed gi en he new e idence we p o ide.
Mul ilingual benchma ks.
Mos benchma ks
co e ing a wide se o languages ha e been c e-
a ed h ough ansla ion, as i is he case o XNLI
(Conneau e al.,2018) o NLI, PAWS-X (Yang
e al.,2019) o ad e sa ial pa aph ase iden i ica-
ion, and XQuAD (A e xe e al.,2020) and MLQA
(Lewis e al.,2020) o Ques ion Answe ing (QA).
A no able excep ion is TyDi QA (Cla k e al.,2020),
a con empo aneous QA da ase ha was sepa a ely
anno a ed in 11 languages. O he c oss-lingual
da ase s le e age exis ing mul ilingual esou ces,
as i is he case o MLDoc (Schwenk and Li,2018)
o documen classi ica ion and Wikiann (Pan e al.,
2017) o named en i y ecogni ion. Concu en o
ou wo k, Hu e al. (2020) combine some o hese
da ase s in o a single mul ilingual benchma k, and
e alua e some well-known me hods on i .
Anno a ion a i ac s.
Se e al s udies ha e
shown ha NLI da ase s like SNLI (Bowman e al.,
2015) and Mul iNLI (Williams e al.,2018) con ain
spu ious pa e ns ha can be exploi ed o ob ain
s ong esul s wi hou making eal in e en ial deci-
sions. Fo ins ance, Gu u angan e al. (2018) and
Poliak e al. (2018) showed ha a hypo hesis-only
baseline pe o ms be e han chance due o cues on
hei lexical choice and sen ence leng h. Simila ly,
McCoy e al. (2019) showed ha NLI models end
o p edic en ailmen o sen ence pai s wi h a high
lexical o e lap. Se e al au ho s ha e wo ked on
ad e sa ial da ase s o diagnose hese issues and
p o ide a mo e challenging benchma k (Naik e al.,
2018;Glockne e al.,2018;Nie e al.,2020). Be-
sides NLI, o he asks like QA ha e also been ound
o be suscep ible o anno a ion a i ac s (Jia and
Liang,2017;Kaushik and Lip on,2018). While
p e ious wo k has ocused on he monolingual sce-
na io, we show ha ansla ion can in e e e wi h
hese a i ac s in mul ilingual se ings.
T ansla ionese.
T ansla ed ex s a e known o
ha e unique ea u es like simpli ica ion, explici a-
ion, no maliza ion and in e e ence, which a e e-
e o as ansla ionese (Volansky e al.,2013). This
phenomenon has been epo ed o ha e a no able
impac in machine ansla ion e alua ion (Zhang
and To al,2019;G aham e al.,2019). Fo ins ance,
back- ansla ion b ings la ge BLEU gains o e-
e sed es se s (i.e., when ansla ionese is on he
sou ce side and o iginal ex is used as e e ence),
bu i s e ec diminishes in he na u al di ec ion
(Eduno e al.,2020). While connec ed, he phe-
nomenon we analyze is di e en in ha i a ises
om ansla ion inconsis encies due o he lack o
con ex , and a ec s c oss-lingual ans e lea ning
a he han machine ansla ion.
3 Expe imen al design
Ou goal is o analyze he e ec o bo h human
and machine ansla ion in c oss-lingual models.
Fo ha pu pose, he co e idea o ou wo k is o (i)
use machine ansla ion o ei he ansla e he ain-
ing se in o o he languages, o gene a e English
pa aph ases o i h ough back- ansla ion, and (ii)
e alua e he esul ing sys ems on o iginal, human
ansla ed and machine ansla ed es se s in com-
pa ison wi h sys ems ained on o iginal da a. We
nex desc ibe he models used in ou expe imen s
(
§
3.1), he speci ic aining a ian s explo ed (
§
3.2),
and he e alua ion p ocedu e ollowed (§3.3).
3.1 Models and ans e me hods
We expe imen wi h wo models ha a e ep esen-
a i e o he s a e-o - he-a in monolingual and
7676
c oss-lingual p e- aining: (i) ROBERTA (Liu e al.,
2019), which is an imp o ed e sion o BERT ha
uses masked language modeling o p e- ain an En-
glish T ans o me model, and (ii) XLM-R (Conneau
e al.,2020), which is a mul ilingual ex ension o
he o me p e- ained on 100 languages. In bo h
cases, we use he la ge models eleased by he au-
ho s unde he ai seq eposi o y.
2
As discussed
nex , we explo e di e en a ian s o he aining
se o ine- une each model on di e en asks. A
es ime, we y bo h machine ansla ing he es
se in o English (TRANSLATE-TEST) and, in he case
o XLM-R, using he ac ual es se in he a ge
language (ZERO-SHOT).
3.2 T aining a ian s
We y 3 a ian s o each aining se o ine- une
ou models: (i) he o iginal one in English (ORIG),
(ii) an English pa aph ase o i gene a ed h ough
back- ansla ion using Spanish o Finnish as pi o
(BT-ES and BT-FI), and (iii) a machine ansla ed
e sion in Spanish o Finnish (MT-ES and MT-FI).
Fo sen ences occu ing mul iple imes in he ain-
ing se (e.g., p emises epea ed o mul iple hy-
po heses), we use he exac same ansla ion o
all occu ences, as ou goal is o unde s and he in-
he en e ec o ansla ion a he han i s po en ial
applica ion as a da a augmen a ion me hod.
In o de o ain he machine ansla ion sys ems
o MT-XX and BT-XX, we use he big T ans o me
model (Vaswani e al.,2017) wi h he same se ings
as O e al. (2018) and Sen encePiece okeniza-
ion (Kudo and Richa dson,2018) wi h a join o-
cabula y o 32k subwo ds. Fo English-Spanish,
we ain o 10 epochs on all pa allel da a om
WMT 2013 (Boja e al.,2013) and Pa aC awl
5.0 (Espl
`
a e al.,2019). Fo English-Finnish, we
ain o 40 epochs on Eu opa l and Wiki Ti les
om WMT 2019 (Ba aul e al.,2019), Pa aC awl
5.0, and DGT, EUbookshop and TildeMODEL
om OPUS (Tiedemann,2012). In bo h cases,
we emo e sen ences longe han 250 okens, wi h
a sou ce/ a ge a io exceeding 1.5, o o which
langid.py
(Lui and Baldwin,2012) p edic s a
di e en language, esul ing in a inal co pus size
o 48M and 7M sen ence pai s, espec i ely. We
use sampling decoding wi h a empe a u e o 0.5
o in e ence, which p oduces mo e di e se ansla-
ions han beam sea ch (Eduno e al.,2018) and
pe o med be e in ou p elimina y expe imen s.
2h ps://gi hub.com/py o ch/ ai seq
3.3 Tasks and e alua ion p ocedu e
We use he ollowing asks o ou expe imen s:
Na u al Language In e ence (NLI).
Gi en a
p emise and a hypo hesis, he ask is o de e mine
whe he he e is an en ailmen ,neu al o con a-
dic ion ela ion be ween hem. We ine- une ou
models on Mul iNLI (Williams e al.,2018) o 10
epochs using he same se ings as Liu e al. (2019).
In mos o ou expe imen s, we e alua e on XNLI
(Conneau e al.,2018), which comp ises 2490 de-
elopmen and 5010 es ins ances in 15 languages.
These we e o iginally anno a ed in English, and he
esul ing p emises and hypo heses we e indepen-
den ly ansla ed in o he es o he languages by
p o essional ansla o s. Fo he TRANSLATE-TEST
app oach, we use he machine ansla ed e sions
om he au ho s. Following Conneau e al. (2020),
we selec he bes epoch checkpoin acco ding o
he a e age accu acy in he de elopmen se .
Ques ion Answe ing (QA).
Gi en a con ex
pa ag aph and a ques ion, he ask is o iden i y
he span answe ing he ques ion in he con ex .
We ine- une ou models on SQuAD 1.1 (Ra-
jpu ka e al.,2016) o 2 epochs using he same
se ings as Liu e al. (2019), and epo es esul s
o he las epoch. We use wo da ase s o e al-
ua ion: XQuAD (A e xe e al.,2020), a subse
o he SQuAD de elopmen se ansla ed in o 10
o he languages, and MLQA (Lewis e al.,2020)
a da ase consis ing o pa allel con ex pa ag aphs
plus he co esponding ques ions anno a ed in En-
glish and ansla ed in o 6 o he languages. In bo h
cases, he ansla ion was done by p o essional
ansla o s a he documen le el (i.e., when ans-
la ing a ques ion, he ex answe ing i was also
shown). Fo ou BT-XX and MT-XX a ian s, we
ansla e he con ex pa ag aph and he ques ions
independen ly, and map he answe spans using he
same p ocedu e as Ca ino e al. (2020).
3
Fo he
TRANSLATE-TEST app oach, we use he o icial ma-
chine ansla ed e sions o MLQA, un in e ence
o e hem, and map he p edic ed answe spans
back o he a ge language.4
3
We use Fas Align (Dye e al.,2013) o wo d alignmen ,
and disca d he ew ques ions o which he mapping me hod
ails (when none o he okens in he answe span a e aligned).
4
We use he same p ocedu e as o he aining se excep
ha (i) gi en he small size o he es se , we combine i wi h
WikiMa ix (Schwenk e al.,2019) o aid wo d alignmen , (ii)
we use Jieba o Chinese segmen a ion ins ead o he Moses
okenize , and (iii) o he ew unaligned spans, we e u n he
English answe .
7677
Model T ain en es de el bg u a i h zh hi sw u a g
Tes se machine ansla ed in o English (TRANSLATE-TEST)
ROBERTA
ORIG 91.2 82.2 84.6 82.4 82.1 82.1 79.2 76.5 77.4 73.8 73.4 76.7 70.5 67.2 66.8 77.7 ±0.6
BT-ES 91.6 85.7 87.4 85.4 85.1 85.1 83.6 81.3 81.5 78.7 78.2 81.1 76.3 72.7 71.5 81.7 ±0.2
BT-FI 91.4 86.0 87.4 85.7 85.7 85.4 84.4 82.3 82.1 79.0 79.3 81.8 77.6 73.5 73.6 82.3 ±0.2
XLM-R
ORIG 90.3 82.2 84.2 82.6 81.9 82.0 79.3 76.7 77.5 75.0 73.7 77.5 70.9 67.8 67.2 77.9 ±0.3
BT-ES 90.2 84.1 86.3 84.5 84.5 84.1 82.2 79.6 80.7 78.5 77.3 80.8 75.2 72.5 71.2 80.8 ±0.3
BT-FI 89.5 84.9 85.5 84.5 84.5 84.6 82.9 80.6 81.4 78.9 78.1 81.5 76.3 73.3 72.5 81.3 ±0.2
MT-ES 89.8 83.2 85.6 84.2 84.0 83.6 81.6 78.4 79.3 77.6 76.7 80.0 74.3 71.3 70.1 80.0 ±0.6
MT-FI 89.8 84.4 85.3 84.7 84.1 84.0 82.0 79.8 80.3 77.4 77.7 80.6 74.7 71.8 71.3 80.5 ±0.3
Tes se in a ge language (ZERO-SHOT)
XLM-R
ORIG 90.4 84.4 85.5 84.3 81.9 83.6 80.1 80.1 79.8 81.8 78.3 80.3 77.7 72.8 74.5 81.0 ±0.2
BT-ES 90.2 86.0 86.9 86.5 84.0 85.3 83.2 82.5 82.7 83.7 80.7 83.0 79.7 75.6 77.1 83.1 ±0.2
BT-FI 89.5 86.0 86.2 86.2 83.9 85.1 83.4 82.2 83.0 83.9 81.2 83.9 80.1 75.2 78.1 83.2 ±0.1
MT-ES 89.9 85.7 87.3 85.6 83.9 85.4 82.9 82.0 82.3 83.6 80.0 82.6 79.9 75.5 76.8 82.9 ±0.4
MT-FI 90.2 85.9 86.9 86.5 84.4 85.5 83.4 83.0 82.4 83.6 80.5 83.6 80.4 76.5 77.9 83.4 ±0.2
Table 1: XNLI de esul s (acc). BT-XX and MT-XX consis en ly ou pe o m ORIG in all cases.
Bo h o NLI and QA, we un each sys em 5
imes wi h di e en andom seeds and epo he
a e age esul s. Space pe mi ing, we also epo
he s anda d de ia ion ac oss he 5 uns. In ou e-
sul ables, we use an unde line o highligh he bes
esul wi hin each block, and bold ace o highligh
he bes o e all esul .
4 NLI expe imen s
We nex discuss ou main esul s in he XNLI de el-
opmen se (
§
4.1,
§
4.2), un addi ional expe imen s
o be e unde s and he beha io o ou di e en
a ian s (
§
4.3,
§
4.4,
§
4.5), and compa e ou esul s
o p e ious wo k in he XNLI es se (§4.6).
4.1 TRANSLATE-TEST esul s
We s a by analyzing XNLI de elopmen esul s
o TRANSLATE-TEST. Recall ha , in his app oach,
he es se is machine ansla ed in o English, bu
aining is ypically done on o iginal English da a.
Ou BT-ES and BT-FI a ian s close his gap by
aining on a machine ansla ed English e sion o
he aining se gene a ed h ough back- ansla ion.
As shown in Table 1, his b ings subs an ial gains
o bo h ROBERTA and XLM-R, wi h an a e age im-
p o emen o 4.6 poin s in he bes case. Qui e e-
ma kably, MT-ES and MT-FI also ou pe o m ORIG
by a subs an ial ma gin, and a e only 0.8 poin s be-
low hei BT-ES and BT-FI coun e pa s. Recall ha ,
o hese wo sys ems, aining is done in machine
ansla ed Spanish o Finnish, while in e ence is
done in machine ansla ed English. This shows
ha he loss o pe o mance when gene alizing
om o iginal da a o machine ansla ed da a is
subs an ially la ge han he loss o pe o mance
when gene alizing om one language o ano he .
4.2 ZERO-SHOT esul s
We nex analyze he esul s o he ZERO-SHOT ap-
p oach. In his case, in e ence is done in he es se
in each a ge language which, in he case o XNLI,
was human ansla ed om English. As such, di -
e en om he TRANSLATE-TEST app oach, nei he
aining on o iginal da a (ORIG) no aining on ma-
chine ansla ed da a (BT-XX and MT-XX) makes
use o he exac same ype o ex ha he sys em
is exposed o a es ime. Howe e , as shown in
Table 1, bo h BT-XX and MT-XX ou pe o m ORIG
by app oxima ely 2 poin s, which sugges s ha ou
(back-) ansla ed e sions o he aining se a e
mo e simila o he human ansla ed es se s han
he o iginal one. This also p o ides a new pe -
spec i e on he TRANSLATE-TRAIN app oach, which
was epo ed o ou pe o m ORIG in p e ious wo k
(Conneau and Lample,2019): while he o iginal
mo i a ion was o ain he model on he same lan-
guage ha i is es ed on, ou esul s show ha
machine ansla ing he aining se is bene icial
e en when he a ge language is di e en .
4.3 O iginal s. ansla ed es se s
So as o unde s and whe he he imp o emen s ob-
se ed so a a e limi ed o ansla ed es se s o
apply mo e gene ally, we conduc addi ional ex-
pe imen s compa ing ansla ed es se s o o iginal
ones. Howe e , o he bes o ou knowledge, all
7678
XNLI de Ou da ase
OR HT OR HT MT
Model T ain (en) (es) (es) (en) (en)
ROBERTA
ORIG 92.1 - - 78.7 79.0
BT-ES 91.9 - - 80.3 80.5
BT-FI 91.4 - - 80.5 80.5
XLM-R
ORIG 90.5 85.5 81.0 77.5 78.5
BT-ES 90.3 87.1 81.4 78.6 79.4
BT-FI 89.7 86.5 80.8 78.8 79.2
MT-ES 90.2 87.5 81.3 78.4 78.9
MT-FI 90.4 87.1 81.1 78.3 78.9
Table 2: NLI esul s on o iginal (OR), human ans-
la ed (HT) and machine ansla ed (MT) se s (acc).
BT-XX
and
MT-XX
ou pe o m
ORIG
in ansla ed se s,
bu do no ge any clea imp o emen in o iginal ones.
exis ing non-English NLI benchma ks we e c e-
a ed h ough ansla ion. Fo ha eason, we build
a new es se ha mimics XNLI, bu is anno a ed
in Spanish a he han English. We i s collec he
p emises om a il e ed e sion o CommonC awl
(Buck e al.,2014), aking a subse o 5 websi es
ha ep esen a di e se se o gen es: a newspa-
pe , an economy o um, a celeb i y magazine, a
li e a u e blog, and a consume magazine. We hen
ask na i e Spanish anno a o s o gene a e an en ail-
men , a neu al and a con adic ion hypo hesis o
each p emise.
5
We collec a o al o 2490 exam-
ples using his p ocedu e, which is he same size
as he XNLI de elopmen se . Finally, we c ea e a
human ansla ed and a machine ansla ed English
e sion o he da ase using p o essional ansla o s
om Gengo and ou machine ansla ion sys em
desc ibed in
§
3.2,
6
espec i ely. We epo esul s
o he bes epoch checkpoin on each se .
As shown in Table 2, bo h BT-XX and MT-XX
clea ly ou pe o m ORIG in all es se s c ea ed
h ough ansla ion, which is consis en wi h ou
p e ious esul s. In con as , he bes esul s on
he o iginal English se a e ob ained by ORIG, and
nei he BT-XX no MT-XX ob ain any clea imp o e-
men on he one in Spanish ei he .
7
This con i ms
ha he unde lying phenomenon is limi ed o ans-
la ed es se s. In addi ion, i is wo h men ioning
ha he esul s o he machine ansla ed es se in
English a e sligh ly be e han hose o he human
5
Unlike XNLI, we do no collec 4 addi ional labels o
each example. No e, howe e , ha XNLI kep he o iginal
label as he gold s anda d, so he addi ional labels a e i ele an
o he ac ual e alua ion. This is no en i ely clea in Conneau
e al. (2018), bu can be e i ied by inspec ing he da ase .
6We use beam sea ch ins ead o sampling decoding.
7No e ha he s anda d de ia ions a e a ound 0.3.
Compe ence Dis ac ion Noise
Model T ain AT NR WO NG LN SE
ROBERTA ORIG 72.9 65.7 64.9 59.1 88.4 86.5
BT-FI 56.6 57.2 80.6 67.8 87.7 86.6
XLM-R
ORIG 78.4 56.8 67.3 61.2 86.8 85.3
BT-FI 60.6 51.7 76.7 64.6 86.2 85.4
MT-FI 64.3 50.3 77.8 68.5 86.4 85.3
Table 3: NLI S ess Tes esul s (combined ma ched
& misma ched acc). AT = an onymy, NR = nume ical
easoning, WO = wo d o e lap, NG = nega ion, LN =
leng h misma ch, SE = spelling e o .
BT-FI
and
MT-FI
a e conside ably weake han
ORIG
in he compe ence
es , bu subs an ially s onge in he dis ac ion es .
ansla ed one, which sugges s ha he di icul y
o he ask does no only depend on he ansla ion
quali y. Finally, i is also in e es ing ha MT-ES is
only ma ginally be e han MT-FI in bo h Spanish
es se s, e en i i co esponds o he TRANSLATE-
TRAIN app oach, whe eas MT-FI needs o ZERO-SHOT
ans e om Finnish in o Spanish. This ein o ces
he idea ha i is aining on ansla ed da a a he
han aining on he a ge language ha is key in
TRANSLATE-TRAIN.
4.4 S ess es s
In o de o be e unde s and how sys ems ained
on o iginal and ansla ed da a di e , we un addi-
ional expe imen s on he NLI S ess Tes s (Naik
e al.,2018), which we e designed o es he o-
bus ness o NLI models o speci ic linguis ic phe-
nomena in English. The benchma k consis s o a
compe ence es , which e alua es he abili y o un-
de s and an onymy ela ion and pe o m nume ical
easoning, a dis ac ion es , which e alua es he
obus ness o shallow pa e ns like lexical o e lap
and he p esence o nega ion wo ds, and a noise
es , which e alua es obus ness o spelling e o s.
Jus as wi h p e ious expe imen s, we epo esul s
o he bes epoch checkpoin in each es se .
As shown in Table 3,ORIG ou pe o ms BT-FI
and MT-FI on he compe ence es by a la ge ma -
gin, bu he opposi e is ue on he dis ac ion es .
8
In pa icula , ou esul s show ha BT-FI and MT-FI
a e less elian on lexical o e lap and he p esence
o nega i e wo ds. This eels in ui i e, as ansla -
ing he p emise and hypo hesis independen ly—as
BT-FI and MT-FI do—is likely o educe he lexical
o e lap be ween hem. Mo e gene ally, he ans-
8
We obse e simila ends o BT-ES and MT-ES, bu
omi hese esul s o conciseness.
7679
la ion p ocess can al e simila supe icial pa e ns
in he da a, which NLI models a e sensi i e o (
§
2).
This would explain why he esul ing models ha e
a di e en beha io on di e en s ess es s.
4.5 Ou pu class dis ibu ion
Wi h he aim o unde s and he e ec o he p e i-
ous phenomenon in c oss-lingual se ings, we look
a he ou pu class dis ibu ion o ou di e en mod-
els in he XNLI de elopmen se . As shown in Ta-
ble 4, he p edic ions o all sys ems a e close o he
ue class dis ibu ion in he case o English. Ne -
e heless, ORIG is s ongly biased o he es o lan-
guages, and ends o unde p edic en ailmen and
o e p edic neu al. This can again be a ibu ed o
he ac ha he English es se is o iginal, whe eas
he es a e human ansla ed. In pa icula , i is
well-known ha NLI models end o p edic en ail-
men when he e is a high lexical o e lap be ween
he p emise and he hypo hesis (
§
2). Howe e , he
deg ee o o e lap will be smalle in he human
ansla ed es se s gi en ha he p emise and he
hypo hesis we e ansla ed independen ly, which
explains why en ailmen is unde p edic ed. In con-
as , BT-FI and MT-FI a e exposed o he exac same
phenomenon du ing aining, which explains why
hey a e no ha hea ily a ec ed.
So as o measu e he impac o his phenomenon,
we explo e a simple app oach o co ec his bias:
ha ing ine- uned each model, we adjus he bias
e m added o he logi o each class so he model
p edic ions ma ch he ue class dis ibu ion o
each language.
9
As shown in Table 5, his b ings
la ge imp o emen s o ORIG, bu is less e ec i e
o BT-FI and MT-FI.
10
This shows ha he pe o -
mance o ORIG was conside ably hinde ed by his
bias, which BT-FI and MT-FI e ec i ely mi iga e.
4.6 Compa ison wi h he s a e-o - he-a
So as o pu ou esul s in o pe spec i e, we com-
pa e ou bes a ian o p e ious wo k on he XNLI
es se . As shown in Table 6, ou me hod imp o es
he s a e-o - he-a o bo h he TRANSLATE-TEST and
he ZERO-SHOT app oaches by 4.3 and 2.8 poin s,
9
We achie e his using an i e a i e p ocedu e whe e, a
each s ep, we selec one class and se i s bias e m so he class
is selec ed o he igh pe cen age o examples.
10
No e ha we a e adjus ing he bias e m in he e alua ion
se i sel , which equi es knowing i s class dis ibu ion and
is hus a o m o chea ing. While use ul o analysis, a ai
compa ison would equi e adjus ing he bias e m in a sepa a e
alida ion se . This is wha we do o ou inal esul s in
§
4.6,
whe e we adjus he bias e m in he XNLI de elopmen se
and epo esul s on he XNLI es se .
EN EN →XX (a g)
Model T ain en neu con en neu con
ROBERTA
( ansla e- es )
ORIG 33.4 32.8 33.8 23.2 40.7 36.1
BT-FI 34.5 31.9 33.6 30.2 35.7 34.1
XLM-R
(ze o-sho )
ORIG 32.4 33.2 34.4 27.0 37.8 35.2
BT-FI 34.3 31.6 34.1 33.1 32.9 34.0
MT-FI 33.6 32.6 33.9 30.8 35.3 33.9
Gold S anda d 33.3 33.3 33.3 33.3 33.3 33.3
Table 4: Ou pu class dis ibu ion on XNLI de . All
sys ems a e close o he ue dis ibu ion in English, bu
ORIG
is biased owa d neu and con in he ans e lan-
guages. BT-FI and MT-FI alle ia e his issue.
Model T ain Base Unbias +∆
ROBERTA
( ansla e- es )
ORIG 77.7 ±0.6 80.6 ±0.2 2.9 ±0.5
BT-FI 82.3 ±0.2 82.8 ±0.1 0.4 ±0.2
XLM-R
(ze o-sho )
ORIG 81.0 ±0.2 82.4 ±0.2 1.4 ±0.3
BT-FI 83.2 ±0.1 83.3 ±0.1 0.1 ±0.1
MT-FI 83.4 ±0.2 83.8 ±0.1 0.4 ±0.2
Table 5: XNLI de esul s wi h class dis ibu ion un-
biasing (a e age acc ac oss all languages). Adjus ing
he bias e m o he classi ie o ma ch he ue class
dis ibu ion b ings la ge imp o emen s o
ORIG
, bu is
less e ec i e o BT-FI and MT-FI.
espec i ely. I also ob ains he bes o e all esul s
published o da e, wi h he addi ional ad an age
ha he p e ious s a e-o - he-a equi ed a ma-
chine ansla ion sys em be ween English and each
o he 14 a ge languages, whe eas ou me hod
uses a single machine ansla ion sys em be ween
English and Finnish (which is no one o he a ge
languages). While he main goal o ou wo k is no
o design be e c oss-lingual models, bu o ana-
lyze hei beha io in connec ion o ansla ion, his
shows ha he phenomenon unde s udy is highly
ele an , o he ex en ha i can be exploi ed o
imp o e he s a e-o - he-a .
5 QA expe imen s
So as o unde s and whe he ou p e ious indings
apply o o he asks besides NLI, we un addi-
ional expe imen s on QA. As shown in Table 7,
BT-FI and BT-ES do indeed ou pe o m ORIG o he
TRANSLATE-TEST app oach on MLQA. The imp o e-
men is modes , bu e y consis en ac oss di e en
languages, models and uns. The esul s o MT-ES
and MT-FI a e less conclusi e, p esumably because
mapping he answe spans ac oss languages migh
in oduce some noise. In con as , we do no ob-
7680
Model en es de el bg u a i h zh hi sw u a g
Fine- une an English model and machine ansla e he es se in o English (TRANSLATE-TEST)
BERT (De lin e al.,2019) 88.8 81.4 82.3 80.1 80.3 80.9 76.2 76.0 75.4 72.0 71.9 75.6 70.0 65.8 65.8 76.2
Robe a (Liu e al.,2019)91.3 82.9 84.3 81.2 81.7 83.1 78.3 76.8 76.6 74.2 74.1 77.5 70.9 66.7 66.8 77.8
P oposed (ROBERTA –BT-FI) 90.6 85.4 86.3 84.3 85.2 85.7 82.3 80.6 81.5 77.8 78.6 81.2 77.1 73.5 72.3 81.5
+ Unbiasing ( uned in de ) 90.5 85.8 86.6 84.6 85.5 85.8 82.9 81.2 82.3 78.7 79.7 82.3 77.6 74.4 72.9 82.1
Fine- une a mul ilingual model on all machine ansla ed aining se s (TRANSLATE-TRAIN-ALL)
Unicode (Huang e al.,2019) 85.6 81.1 82.3 80.9 79.5 81.4 79.7 76.8 78.2 77.9 77.1 80.5 73.4 73.8 69.6 78.5
XLM-R (Conneau e al.,2020) 88.7 85.2 85.6 84.6 83.6 85.5 82.4 81.6 80.9 83.4 80.9 83.3 79.8 75.9 74.3 82.4
Fine- une a mul ilingual model on he English aining se (ZERO-SHOT)
mBERT (De lin e al.,2019) 82.1 73.8 74.3 71.1 66.4 68.9 69.0 61.6 64.9 69.5 55.8 69.3 60.0 50.4 58.0 66.3
XLM (Conneau and Lample,2019) 85.0 78.7 78.9 77.8 76.6 77.4 75.3 72.5 73.1 76.1 73.2 76.5 69.6 68.4 67.3 75.1
Unicode (Huang e al.,2019) 85.1 79.0 79.4 77.8 77.2 77.2 76.3 72.8 73.5 76.4 73.6 76.2 69.4 69.7 66.7 75.4
XLM-R (Conneau e al.,2020) 88.8 83.6 84.2 82.7 82.3 83.1 80.1 79.0 78.8 79.7 78.6 80.2 75.8 72.0 71.7 80.1
P oposed (XLM-R –MT-FI) 88.8 84.8 85.7 84.6 84.2 85.7 82.9 81.8 82.0 82.1 79.9 81.8 79.8 75.9 76.7 82.4
+ Unbiasing ( uned in de ) 88.7 85.0 86.1 84.8 84.8 86.1 83.5 82.2 82.4 83.0 80.8 82.6 80.3 76.0 77.3 82.9
Table 6: XNLI es esul s (acc). Resul s o o he me hods a e aken om hei espec i e pape s o , i no
p o ided, om Conneau e al. (2020). Fo hose wi h mul iple a ian s, we selec he one wi h he bes esul s.
se e any clea imp o emen o he ZERO-SHOT
app oach on his da ase . Ou XQuAD esul s in
Table 8a e mo e posi i e, bu s ill inconclusi e.
These esul s can pa ly be explained by he
ansla ion p ocedu e used o c ea e he di e en
benchma ks: he p emises and hypo heses o XNLI
we e ansla ed independen ly, whe eas he ques-
ions and con ex pa ag aphs o XQuAD we e ans-
la ed oge he . Simila ly, MLQA made use o pa -
allel con ex s, and ansla o s we e shown he sen-
ence con aining each answe when ansla ing he
co esponding ques ion. As a esul , one can ex-
pec bo h QA benchma ks o ha e mo e consis en
ansla ions han XNLI, which would in u n di-
minish his phenomenon. In con as , he ques ions
and con ex pa ag aphs a e independen ly ans-
la ed when using machine ansla ion, which ex-
plains why BT-ES and BT-FI ou pe o m ORIG o
he TRANSLATE-TEST app oach. We conclude ha
he ansla ion a i ac s e ealed by ou analysis a e
no exclusi e o NLI, as hey also show up on QA
o he TRANSLATE-TEST app oach, bu hei ac ual
impac can be highly dependen on he ansla ion
p ocedu e used and he na u e o he ask.
6 Discussion
Ou analysis p omp s o econside p e ious ind-
ings in c oss-lingual ans e lea ning as ollows:
The c oss-lingual ans e gap on XNLI was
o e es ima ed.
Gi en he pa allel na u e o
XNLI, accu acy di e ences ac oss languages a e
commonly in e p e ed as he loss o pe o mance
when gene alizing om English o he es o lan-
guages. Howe e , ou wo k shows ha he e is
ano he ac o ha can ha e a much la ge impac :
he loss o pe o mance when gene alizing om
o iginal o ansla ed da a. Ou esul s sugges ha
he eal c oss-lingual gene aliza ion abili y o XLM-
Ris conside ably be e han wha he accu acy
numbe s in XNLI e lec .
O e coming he c oss-lingual gap is no wha
makes TRANSLATE-TRAIN wo k.
The o iginal
mo i a ion o TRANSLATE-TRAIN was o ain he
model on he same language i is es ed on. How-
e e , we show ha i is aining on ansla ed da a,
a he han aining on he a ge language, ha is
key o his app oach o ou pe o m ZERO-SHOT as
epo ed by p e ious au ho s.
Imp o emen s p e iously a ibu ed o da a
augmen a ion should be econside ed.
The
me hod by Singh e al. (2019) combines machine
ansla ed p emises and hypo heses in di e en
languages (
§
2), esul ing in an e ec simila o
BT-XX and MT-XX. As such, we belie e ha his
me hod should be analyzed om he poin o iew
o da ase a i ac s a he han da a augmen a ion,
as he au ho s do.
11
F om his pe spec i e, ha ing
he p emise and he hypo heses in di e en lan-
guages can educe he supe icial pa e ns be ween
hem, which would explain why his app oach is
be e han using examples in a single language.
11
Recall ha ou expe imen al design p e en s a da a aug-
men a ion e ec , in ha he numbe o unique sen ences and
examples used o aining is always he same (§3.2).
7681
Model T ain en es de a i zh hi a g
Tes se machine ansla ed in o English (TRANSLATE-TEST)
ROBERTA
ORIG 84.7 /71.4 70.1 / 49.7 60.5 / 41.2 55.7 / 32.5 65.6 / 40.8 53.5 / 26.0 42.7 / 20.7 61.8 ±0.1 / 40.3 ±0.2
BT-ES 84.4 / 71.2 70.9 / 50.7 61.0 / 41.6 56.5 / 33.3 66.7 / 41.8 54.4 / 27.1 43.0 / 21.1 62.4 ±0.1 / 41.0 ±0.2
BT-FI 83.8 / 70.4 70.3 / 50.1 61.1 / 41.9 56.5 / 33.4 66.8 / 42.1 54.9 / 27.5 42.8 / 21.3 62.3 ±0.1 / 40.9 ±0.2
XLM-R
ORIG 84.1 / 71.0 69.9 / 49.2 60.8 / 42.5 55.2 / 31.8 65.4 / 40.6 54.3 / 27.8 43.6 / 21.3 61.9 ±0.1 / 40.6 ±0.1
BT-ES 83.8 / 70.8 70.5 / 50.0 61.4 / 43.5 56.1 / 33.1 66.5 / 41.6 55.4 / 29.0 44.0 / 22.2 62.5 ±0.2 / 41.5 ±0.2
BT-FI 82.7 / 69.6 70.0 / 49.7 61.1 / 43.3 56.0 / 33.1 66.2 / 41.5 55.6 / 29.2 43.7 / 22.0 62.2 ±0.1 / 41.2 ±0.2
MT-ES 83.4 / 69.7 70.0 / 49.1 61.0 / 42.7 55.6 / 32.2 65.9 / 40.9 54.9 / 28.1 43.9 / 21.6 62.1 ±0.3 / 40.6 ±0.2
MT-FI 82.6 / 69.0 69.7 / 48.6 61.0 / 42.8 55.7 / 32.3 65.8 / 40.9 54.8 / 27.9 43.9 / 21.6 61.9 ±0.3 / 40.4 ±0.2
Tes se in a ge language (ZERO-SHOT)
XLM-R
ORIG 84.1 / 71.0 74.5 / 56.3 70.3 / 55.1 66.5 / 45.9 74.3 / 53.1 67.8 / 43.4 71.6 / 53.4 72.7 ±0.1 / 54.0 ±0.1
BT-ES 83.8 / 70.8 74.7 / 56.8 70.3 / 55.2 66.9 / 46.5 74.3 / 53.0 68.2 /43.8 71.4 / 53.6 72.8 ±0.2 /54.3 ±0.2
BT-FI 82.7 / 69.6 74.1 / 56.3 69.8 / 54.5 66.6 / 46.0 73.3 / 52.3 67.9 / 43.4 71.0 / 53.2 72.2 ±0.2 / 53.6 ±0.2
MT-ES 83.4 / 69.7 75.2 /57.3 70.5 / 55.1 67.5 /46.5 74.5 /53.2 67.5 / 42.5 71.7 / 52.7 72.9 ±0.3 / 53.9 ±0.4
MT-FI 82.6 / 69.0 74.1 / 56.0 70.2 / 54.6 66.9 / 46.0 73.7 / 52.6 67.2 / 41.5 71.9 / 53.4 72.4 ±0.2 / 53.3 ±0.4
Table 7: MLQA es esul s (F1 / exac ma ch).
Model T ain en es de el u a i h zh hi a g
XLM-R
(ze o-sho )
ORIG 88.2 82.7 80.8 80.9 80.1 76.1 76.0 80.1 75.4 71.9 76.4 79.0 ±0.2
BT-ES 87.9 83.5 80.5 81.2 80.7 76.8 77.4 80.2 76.4 73.0 76.9 79.5 ±0.3
BT-FI 87.1 82.5 80.2 80.7 79.8 75.7 76.6 79.4 75.7 71.5 76.8 78.7 ±0.3
MT-ES 87.1 84.1 80.3 81.2 80.1 76.0 77.4 80.9 76.7 72.7 77.1 79.4 ±0.3
MT-FI 86.3 81.4 80.2 80.5 80.2 76.6 77.0 80.3 77.6 74.5 77.8 79.3 ±0.2
Table 8: XQuAD esul s (F1). Resul s o he exac ma ch me ic a e simila .
The po en ial o TRANSLATE-TEST was unde es i-
ma ed.
The p e ious bes esul s o TRANSLATE-
TEST on XNLI lagged behind he s a e-o - he-a
by 4.6 poin s. Ou wo k educes his gap o only
0.8 poin s by add essing he unde lying ansla-
ion a i ac s. The eason why TRANSLATE-TEST
is mo e se e ely a ec ed by his phenomenon is
wo old: (i) he e ec is doubled by i s using
human ansla ion o c ea e he es se and hen ma-
chine ansla ion o ansla e i back o English, and
(ii) TRANSLATE-TRAIN was inad e en ly mi iga ing
his issue (see abo e), bu equi alen echniques
we e ne e applied o TRANSLATE-TEST.
Fu u e e alua ion should be e accoun o
ansla ion a i ac s.
The e alua ion issues
aised by ou analysis do no ha e a simple so-
lu ion. In ac , while we use he e m ansla ion
a i ac s o highligh ha hey a e an unin ended
e ec o ansla ion ha impac s inal e alua ion,
one could also a gue ha i is he o iginal da ase s
ha con ain he a i ac s, which ansla ion simply
al e s o e en mi iga es.
12
In any case, his is a
mo e gene al issue ha alls beyond he scope o
12
Fo ins ance, he high lexical o e lap obse ed o he
en ailmen class is usually ega ded a spu ious pa e n, so
educing i could be conside ed a posi i e e ec o ansla ion.
c oss-lingual ans e lea ning, so we a gue ha
i should be ca e ully con olled when e alua ing
c oss-lingual models. In he absence o mo e obus
da ase s, we ecommend ha u u e mul ilingual
benchma ks should a leas p o ide consis en es
se s o English and he es o languages. This
can be achie ed by (i) using o iginal anno a ions
in all languages, (ii) using o iginal anno a ions in
a non-English language and ansla ing hem in o
English and o he languages, o (iii) i ansla ing
om English, doing so a he documen le el o
minimize ansla ion inconsis encies.
7 Conclusions
In his pape , we ha e shown ha bo h human and
machine ansla ion can al e supe icial pa e ns in
da a, which equi es econside ing p e ious ind-
ings in c oss-lingual ans e lea ning. Based on he
gained insigh s, we ha e imp o ed he s a e-o - he-
a in XNLI o he TRANSLATE-TEST and ZERO-SHOT
app oaches by a subs an ial ma gin. Finally, we
ha e shown ha he phenomenon is no speci ic
o NLI bu also a ec s QA, al hough i is less p o-
nounced he e hanks o he ansla ion p ocedu e
used in he co esponding benchma ks. So as o
acili a e simila s udies in he u u e, we elease
7682
ou NLI da ase ,
13
which, unlike p e ious bench-
ma ks, was anno a ed in a non-English language
and human ansla ed in o English.
Acknowledgmen s
We hank No a A anbe i and Uxoa I
˜
nu ie a o
help ul discussion du ing he de elopmen o his
wo k, as well as he es o ou colleagues om he
IXA g oup ha wo ked as anno a o s o ou NLI
da ase .
This esea ch was pa ially unded by a Face-
book Fellowship, he Basque Go e nmen ex-
cellence esea ch g oup (IT1343-19), he Span-
ish MINECO (UnsupMT TIN2017-91692-EXP
MCIU/AEI/FEDER, UE), P ojec BigKnowledge
(Ayudas Fundaci
´
on BBVA a equipos de in es i-
gaci
´
on cien
´
ı ica 2018), and he NVIDIA GPU
g an p og am.
This esea ch is suppo ed ia he BETTER P o-
g am con ac #2019-19051600006 (ODNI, IARPA
ac i i y). The iews and conclusions con ained
he ein a e hose o he au ho s and should no be in-
e p e ed as necessa ily ep esen ing he o icial
policies, ei he exp essed o implied, o ODNI,
IARPA, o he U.S. Go e nmen . The U.S. Go -
e nmen is au ho ized o ep oduce and dis ibu e
ep in s o go e nmen al pu poses no wi hs and-
ing any copy igh anno a ion he ein.
Re e ences
Mikel A e xe, Sebas ian Rude , and Dani Yoga ama.
2020. On he c oss-lingual ans e abili y o mono-
lingual ep esen a ions. In P oceedings o he 58 h
Annual Mee ing o he Associa ion o Compu a-
ional Linguis ics, pages 4623–4637. Associa ion
o Compu a ional Linguis ics.
Lo¨
ıc Ba aul , Ondˇ
ej Boja , Ma a R. Cos a-juss`
a,
Ch is ian Fede mann, Ma k Fishel, Y e e G a-
ham, Ba y Haddow, Ma hias Huck, Philipp Koehn,
She in Malmasi, Ch is o Monz, Ma hias M¨
ulle ,
San anu Pal, Ma Pos , and Ma cos Zampie i. 2019.
Findings o he 2019 Con e ence on Machine T ans-
la ion (WMT19). In P oceedings o he Fou h Con-
e ence on Machine T ansla ion (Volume 2: Sha ed
Task Pape s, Day 1), pages 1–61, Flo ence, I aly. As-
socia ion o Compu a ional Linguis ics.
Ondˇ
ej Boja , Ch is ian Buck, Ch is Callison-Bu ch,
Ch is ian Fede mann, Ba y Haddow, Philipp
Koehn, Ch is o Monz, Ma Pos , Radu So icu , and
Lucia Specia. 2013. Findings o he 2013 Wo k-
shop on S a is ical Machine T ansla ion. In P oceed-
ings o he Eigh h Wo kshop on S a is ical Machine
13h ps://gi hub.com/a e xem/esxnli
T ansla ion, pages 1–44, So ia, Bulga ia. Associa-
ion o Compu a ional Linguis ics.
Samuel R. Bowman, Gabo Angeli, Ch is ophe Po s,
and Ch is ophe D. Manning. 2015. A la ge anno-
a ed co pus o lea ning na u al language in e ence.
In P oceedings o he 2015 Con e ence on Empi i-
cal Me hods in Na u al Language P ocessing, pages
632–642, Lisbon, Po ugal. Associa ion o Compu-
a ional Linguis ics.
Ch is ian Buck, Kenne h Hea ield, and Bas an Ooyen.
2014. N-g am coun s and language models om
he Common C awl. In P oceedings o he Nin h In-
e na ional Con e ence on Language Resou ces and
E alua ion (LREC’14), pages 3579–3584, Reyk-
ja ik, Iceland. Eu opean Language Resou ces Asso-
cia ion (ELRA).
Casimi o Pio Ca ino, Ma a R. Cos a-juss`
a, and Jos´
e
A. R. Fonollosa. 2020. Au oma ic Spanish ansla-
ion o he SQuAD da ase o mul i-lingual ques-
ion answe ing. In P oceedings o he 12 h Lan-
guage Resou ces and E alua ion Con e ence, pages
5515–5523, Ma seille, F ance. Eu opean Language
Resou ces Associa ion.
Jona han H. Cla k, Eunsol Choi, Michael Collins, Dan
Ga e e, Tom Kwia kowski, Vi aly Nikolae , and
Jennima ia Palomaki. 2020. TyDi QA: A bench-
ma k o in o ma ion-seeking ques ion answe ing in
ypologically di e se languages.T ansac ions o he
Associa ion o Compu a ional Linguis ics, 8:454–
470.
Alexis Conneau, Ka ikay Khandelwal, Naman Goyal,
Vish a Chaudha y, Guillaume Wenzek, F ancisco
Guzm´
an, Edoua d G a e, Myle O , Luke Ze le-
moye , and Veselin S oyano . 2020. Unsupe ised
c oss-lingual ep esen a ion lea ning a scale. In
P oceedings o he 58 h Annual Mee ing o he Asso-
cia ion o Compu a ional Linguis ics, pages 8440–
8451. Associa ion o Compu a ional Linguis ics.
Alexis Conneau and Guillaume Lample. 2019. C oss-
lingual language model p e aining. In Ad ances
in Neu al In o ma ion P ocessing Sys ems 32, pages
7059–7069.
Alexis Conneau, Ru y Rino , Guillaume Lample, Ad-
ina Williams, Samuel Bowman, Holge Schwenk,
and Veselin S oyano . 2018. XNLI: E alua ing
c oss-lingual sen ence ep esen a ions. In P oceed-
ings o he 2018 Con e ence on Empi ical Me hods
in Na u al Language P ocessing, pages 2475–2485,
B ussels, Belgium. Associa ion o Compu a ional
Linguis ics.
Jacob De lin, Ming-Wei Chang, Ken on Lee, and
K is ina Tou ano a. 2019. BERT: P e- aining o
deep bidi ec ional ans o me s o language unde -
s anding. In P oceedings o he 2019 Con e ence
o he No h Ame ican Chap e o he Associa ion
o Compu a ional Linguis ics: Human Language
Technologies, Volume 1 (Long and Sho Pape s),
