Deep e alua ion o hyb id a chi ec u es: simple me ics co ela ed wi h
human judgmen s
Go ka Labaka, A an za D´
ıaz de Ila aza,
Kepa Sa asola
Uni e si y o he Basque Coun y
[email p o ec ed],
[email p o ec ed],[email p o ec ed]
C is ina Espa˜
na-Bone ,
Llu´
ıs M`
a quez
Uni e si a Poli `
ecnica de Ca alunya
[email p o ec ed],
[email p o ec ed]
Abs ac
The p ocess o de eloping hyb id MT sys ems
is guided by he e alua ion me hod used o
compa e di e en combina ions o basic sub-
sys ems. This wo k p esen s a deep e alua-
ion expe imen o a hyb id a chi ec u e ha
ies o ge he bes o bo h wo lds, ule-based
and s a is ical. In a i s e alua ion human
assessmen s we e used o compa e jus he
single s a is ical sys em and he hyb id one,
he ule-based sys em was no compa ed by
hand because he esul s o au oma ic e alu-
a ion showed a clea disad an age. Bu a sec-
ond and wide e alua ion expe imen su p is-
ingly showed ha acco ding o human e al-
ua ion he bes sys em was he ule-based,
he one ha achie ed he wo s esul s us-
ing au oma ic e alua ion. An examina ion o
sen ences wi h con o e sial esul s sugges ed
ha linguis ic well- o medness in he ou pu
should be conside ed in e alua ion. A e ex-
pe imen ing wi h 6 possible me ics we con-
clude ha a simple a i hme ic mean o BLEU
and BLEU calcula ed on pa s o speech o
wo ds is clea ly a mo e human con o man
me ic han lexical me ics alone.
1 In oduc ion
The p ocess o de eloping hyb id MT sys ems is
guided by he e alua ion me hod used o compa e
di e en combina ions o basic subsys ems. Di ec
human e alua ion is mo e accu a e bu un o una ely
i is ex emely expensi e, so au oma ic me ics ha e
o be used in p o o ype de eloping. Howe e he
me hod should e alua e di e en sys ems wi h he
same c i e ia, and hese c i e ia should be as close as
possible o human judgmen .
I is well known ha ule-based and ph ase-
based s a is ical machine ansla ion pa adigms
(RBMT and SMT, espec i ely) ha e complemen-
a y s eng hs and weaknesses. Fi s , RBMT sys-
ems end o p oduce syn ac ically be e ansla ions
and deal wi h long dis ance dependencies, ag ee-
men and cons i uen eo de ing in a be e way,
since hey pe o m he analysis, ans e and gene a-
ion s eps based on syn ac ic p inciples. On he bad
side, hey usually ha e p oblems wi h lexical selec-
ion due o a poo handling o wo d ambigui y. Also,
in cases in which he inpu sen ence has an unex-
pec ed syn ac ic s uc u e, he pa se may ail and
he quali y o he ansla ion dec ease d ama ically.
On he o he side, ph ase-based SMT models usu-
ally do a be e job wi h lexical selec ion and gene al
luency, since hey model lexical choice wi h dis i-
bu ional c i e ia and explici p obabilis ic language
models. Howe e , ph ase-based SMT sys ems usu-
ally gene a e s uc u ally wo se ansla ions, since
hey model ansla ion mo e locally and ha e p ob-
lems wi h long dis ance eo de ing. They also end
o p oduce e y ob ious e o s, which a e annoying
o egula use s, e.g., lack o gende and numbe
ag eemen , bad punc ua ion, e c. Mo eo e , SMT
sys ems can expe ience a se e e deg ada ion o pe -
o mance when applied o co po a di e en om
hose used o aining (ou -o -domain e alua ion).
I is also well known ha he BLEU me ic (Pap-
ineni e al., 2002) is ac ually he mos used me ic in
s a is ical MT. Bu se e al doub s ha e a isen a ound
BLEU (Melamed e al., 2003; Callison-Bu ch e al.,
2006; Koehn and Monz, 2006). In addi ion o he
ac ha i is ex emely di icul o in e p e wha is
being exp essed in BLEU (Melamed e al., 2003),
imp o ing i s alue nei he gua an ees an imp o e-
men in he ansla ion quali y (Callison-Bu ch e al.,
2006) no o e s as much co ela ion wi h human
judgmen as was belie ed (Koehn and Monz, 2006).
Those p oblems ha e also been de ec ed when ans-
la ing o Basque (Mayo , 2007; Labaka, 2010).
In he las ew yea s, se e al new e alua ion me -
ics ha e been sugges ed o conside a highe le el
o linguis ic in o ma ion (Liu and Gildea, 2005;
Popo i´
c and Ney, 2007; Chan and Ng, 2008), and
di e en me hods o me ic combina ion ha e been
es ed. Due o i s simplici y, we decided o use he
idea p esen ed by Gim´
enez and M`
a quez (2008),
whe e he di e en simple me ics a e combined by
means o he a i hme ic mean.
In his wo k we p esen some su p ising esul s
we ha e achie ed in a deep e alua ion o a hyb id
a chi ec u e. In a i s s ep we used human e al-
ua ion o compa e jus he single s a is ical sys em
and he hyb id one, we did no compa e he ule-
based sys em by hand because he esul s o au o-
ma ic e alua ion showed a clea disad an age. Bu
a second and wide e alua ion expe imen su p is-
ingly showed ha acco ding o human e alua ion he
bes sys em was he ule-based, he one ha achie ed
he wo s esul s using au oma ic e alua ion. We
ied o make a diagnosis o his phenomenon, and
hen based on his we inally ound a simple bu
mo e human con o man me ic ha we plan o use
in aining new e sions o ou hyb id sys em.
In he nex sec ion o his pape we desc ibe he
hyb id sys em. Sec ion 3 p esen s he e alua ion ex-
pe imen s: he co po a used in hem, he i s expe i-
men compa ing jus he single s a is ical sys em and
he hyb id one, and he second and wide e alua ion
expe imen which compa es he all h ee sys ems.
Then Sec ion 4 desc ibes he p ocess o sea ching
o o he au oma ic me ics being mo e human con-
o man . And inally, he las sec ion is de o ed o
conclusions and u u e wo k.
2 The hyb id sys em, SMa xinT
S a is ical Ma xin T ansla o , SMa xinT in sho , is
a hyb id sys em con olled by he RBMT ansla o
and en iched wi h a wide a ie y o SMT ansla ion
op ions (Espa˜
na-Bone e al., 2011).
The wo indi idual sys ems a e a ule-based
Spanish-Basque sys em called Ma xin (Aleg ia e
al., 2007) and a s anda d ph ase-based s a is ical MT
sys em based on Moses which wo ks a he mo -
pheme le el allowing o deal wi h he ich mo phol-
ogy o Basque (Labaka, 2010).
The ini ial analysis o he sou ce sen ence is done
by Ma xin. I p oduces a dependency pa se ee,
whe e he bounda ies o each ph ase a e ma ked.
In o de o add hyb id unc ionali y wo new mod-
ules a e in oduced o he RBMT a chi ec u e (Fig-
u e 1): he ee en ichmen module, which inco -
po a es SMT addi ional ansla ions o each ph ase
o he syn ac ic ee; and a mono onous decoding
module, which is esponsible o gene a ing he i-
nal ansla ion by selec ing among RBMT and SMT
pa ial ansla ion candida es om he en iched ee.
The ee en ichmen module in oduces wo ypes
o ansla ions o he syn ac ic cons i uen s gi en by
Ma xin: 1) he SMT ansla ion(s) o e e y ph ase,
and 2) he SMT ansla ion(s) o he en i e sub ee
con aining ha ph ase. Fo example, he analysis o
he es agmen “a i m´
o el conseje o de in e io ”
(said he Sec e a y o in e io ) gi es wo ph ases:
he head “a i m´
o” (said) and i s child en “el conse-
je o de in e io ” ( he Sec e a y o in e io ). The ull
ule-based ansla ion is “Ba ne Sailbu ua baiez a u
zuen” and he ull SMT ansla ion is “esan zuen
he izaingo sailbu uak”. SMa xinT conside s hese
wo ph ases o he ansla ion o he ull sen ence,
bu also he SMT ansla ions o hei cons i uen s
(“esan zuen” and “he izaingo sailbu uak”). How-
e e , sho ph ases may ha e a w ong SMT ans-
la ion because o a lack o con ex . To o e come
his p oblem SMa xinT also uses he ansla ion o
a ph ase ex ac ed om a longe SMT ansla ion
(“he izaingo sailbu uak” in he p e ious example).
So, in o de o ansla e “a i m´
o el conseje o de in-
e io ” he sys em has p oduced 5 dis inc ph ases,
a numbe ha can be inc eased by conside ing a n-
bes lis o SMT ou pu s.
A e ee en ichmen , he ans e and gene a-
ion s eps o he RBMT sys em a e ca ied ou in a
usual way, and a inal mono onous decode chooses
among he op ions. A key aspec o he pe o -
mance o he sys em is he elec ion o he ea u es
Figu e 1: Gene al a chi ec u e o SMa xinT. The RBMT modules which guide he MT p ocess a e he g ey boxes.
o his decoding. The esul s we p esen he e a e
ob ained wi h a se o ele en ea u es. Th ee o hem
a e he usual SMT ea u es (language model, wo d
penal y and ph ase penal y). We also include ou
ea u es o show he o igin o he ph ase and he
consensus among sys ems (a coun e indica ing how
many di e en sys ems gene a ed he ph ase, wo bi-
na y ea u es indica ing whe he he ph ase comes
om he SMT/RBMT sys em o no , and he num-
be o sou ce wo ds co e ed by he ph ase gene a ed
by bo h indi idual sys ems simul aneously). Finally,
we use he lexical p obabili ies in bo h di ec ions
in wo o ms: a simila app oach o IBM-1 p ob-
abili ies modi ied o ake unknown alignmen s in o
accoun and a lexical p obabili y in e ed om he
RBMT dic iona y. We e e he eade o Espa˜
na-
Bone e al. (2011) o u he de ails.
3 Expe imen s
In ou expe imen s we e alua e bo h indi idual
sys ems and he inal hyb id: SMT, Ma xin and
SMa xinT. The language pai o applica ion is dic-
a ed by he ule-based sys em and, in his case,
Ma xin wo ks wi h he Spanish- o-Basque ansla-
ion. Basque and Spanish a e wo languages wi h
e y di e en mo phologies and syn axes.
3.1 Bilingual and monolingual co po a
The co pus buil o ain he SMT sys em consis s o
ou subse s: (1) six e e ence books ansla ed man-
ually by he ansla ion se ice o he Uni e si y o
he Basque Coun y (EHUBooks); (2) a collec ion
sen ences okens
EHUBooks Spanish 39,583 1,036,605
Basque 794,284
Consume Spanish 61,104 1,347,831
Basque 1,060,695
Elhuya TM Spanish 186,003 3,160,494
Basque 2,291,388
Euskal elTB Spanish 222,070 3,078,079
Basque 2,405,287
To al Spanish 491,853 7,966,419
Basque 6,062,911
Table 1: S a is ics on he bilingual collec ion o pa allel
co po a.
o 1,036 a icles published in Spanish and Basque
by he Consume E oski magazine1(Consume );
(3) ansla ion memo ies mos ly using adminis a-
i e language de eloped by Elhuya 2(Elhuya TM);
and (4) a ansla ion memo y including sho de-
sc ip ions o TV p og ammes (Euskal elTB). Table
1 shows some s a is ics on he co po a, gi ing some
igu es abou he numbe o sen ences and okens.
The aining co pus is hen basically made up o
adminis a i e documen s and desc ip ions o TV
p og ams. Fo de elopmen and es ing we ex ac ed
some adminis a i e da a o he in-domain e alua-
ion and selec ed one collec ion o news o he ou -
o -domain s udy, o aling h ee se s:
Elhuya de el and Elhuya es : 1,500 segmen s each,
ex ac ed om he adminis a i e documen s.
1h p:// e is a.consume .es
2h p://www.elhuya .o g/
NEWS es : 1,000 sen ences collec ed om Spanish
newspape s wi h wo e e ences.
Addi ionally, we collec ed a 21 million wo d
monolingual co pus, which oge he wi h he Basque
side o he pa allel bilingual co po a, builds up a
28 million wo d co pus. This monolingual co pus
is also he e ogeneous, and includes ex om wo
sou ces o news: he Basque co pus o Science and
Technology (ZT co pus) and a icles published by
Be ia newspape (Be ia co pus).
3.2 Fi s expe imen o e alua ion
Acco ding o he au oma ic e alua ion, ca ied ou
in he p e ious a icle and ex ended in Table 4, he
ule-based Ma xin sys em is clea ly he wo s sys em
ob aining he wo s sco es o bo h me ics (BLEU
and TER) in bo h es co po a. On he o he hand,
he e alua ion o he hyb id sys em a ies depending
on he es se . On he in-domain co po a (Elhuya
es se ), he BLEU sco e achie ed by SMa xinT is
sligh ly wo se han he sco es ob ained by he single
SMT sys em, bu be e acco ding o TER (Sno e
e al., 2006) e alua ion. The dis inc beha io be-
ween me ics and he small di e ences do no al-
low us o de ine a clea p e e ence be ween s a is i-
cal and hyb id sys ems. On he con a y, on he ou -
domain co po a (NEWS es se ), SMa xinT consis-
en ly a chi es be e sco es han any o he sys em.
Based on hese esul s, we s a ed ha he low in-
domain pe o mance o he Ma xin penalizes he hy-
b id sys em, p e en ing i o o e come he single
SMT sys em. Bu , in he ou -domain es se , whe e
he sco es o Ma xin we e no so a om he es
o he sys ems, ou hyb idiza ion echnique was able
o combine he bes o bo h sys ems ob aining he
bes ansla ion. In o de o e i y his asse ion, we
ca ied ou an human e alua ion, whe e we asked
ou e alua o s o de e mine he p e e ence be ween
he hyb id and he SMT ansla ions o 100 sen-
ences andomly chosen om he NEWS es se .
The igu es ob ained co obo a ed ha he hyb id
sys em ou pe o ms he single SMT sys em in he
ou -domain co po a.
3.3 Deepe e alua ion: Human e alua ion o
compa e he h ee sys ems
In o de o ge a mo e de ailed insigh o he pe -
o mance o ou sys ems, we ecen ly ex ended his
manual e alua ion o he es o he sys ems and es
co po a. Tha way, we selec ed ano he 100 sen-
ences om he Elhuya es se and asked he same
ou e alua o s o assess he p e e ence be ween he
h ee sys em pai s (SMT-Ma xin, SMT-SMa xinT,
Ma xin-SMa xinT).
Su p isingly, acco ding o his manual e alua ion
he bes sys em is he ule-based Ma xin sys em, he
wo s anked one using au oma ic e alua ion. E en
o in-domain e alua ion i is clea ly be e han he
s a is ical sys em and o simila quali y as he hyb id
one, ha is sligh ly supe io o he s a is ical sys em.
Fo ou -domain e alua ion he di e ences a e e y
clea : he ule-based Ma xin sys em clea ly ou pe -
o ms he hyb id sys em and his one ou pe o ms
he s a is ical sys em.
This can be seen in Table 2. The able shows he
numbe o imes ha a sys em is be e han he o he
o hose sen ences whe e he e was ull ag eemen
among e alua o s (Ag eemen ) and o he ull sub-
se (All). Resul s a e gi en o he h ee sys em pai s
on he wo es se s, he in-domain and he ou -o -
domain ones.
We con i med hese su p ising esul s o man-
ual e alua ion by examining some examples whe e
BLEU sco es did no e lec he di e ence o quali y
be ween ansla ion ou pu s. Le us analyze he ex-
ample shown in Table 3, ha is, he ansla ion o he
sou ce sen ence “Legasa cuen a ya con un con enio
sob e la ecupe aci´
on de bienes comunales.”. The
able shows he sou ce sen ence wi h i s meaning in
English oge he wi h wo ansla ion e e ences and
he ou pu gi en by he wo indi idual sys ems.
In his example, he ou pu o he ule-based sys-
em is adequa e, bu BLEU is unable o ecognize
some linguis ic equi alences: jadanik and jada a e
synonymous, as well as be esku a zea en ingu uan
and be esku a ze gainean. Simila ly he i onda-
sunak and he i-ondasunen a e almos he same be-
cause he “-” is op ional, and using Legasa ins ead o
Legasak is a common e o easy o unde s and. The
ollowing segmen s a e quasi equi alen s: hi za -
mena du and kon a zen du hi za men ba ekin. All
hese co espondences a e i ial o humans bu in-
isible o he BLEU me ic.
On he o he hand, he ou pu o he s a is ical sys-
em is ha de o unde s and. By using Legasako in-
s ead o Legasak, he sen ence becomes di icul o
Sys em 1 Tied Sys em 2
Elhuya (in-domain)
SMT s. SMa xinT Ag eemen 5 (9.4%) 31 (58.5%) 17 (32.1%)
All 25 (12.5%) 109 (54.5%) 66 (33.0%)
SMT s. Ma xin Ag eemen 14 (23.7%) 19 (32.2%) 26 (44.1%)
All 41 (20.5%) 79 (39.5%) 80 (40.0%)
SMa xinT s. Ma xin Ag eemen 19 (28.8%) 24 (36.4%) 23 (34.8%)
All 59 (29.5%) 82 (41.0%) 59 (29.5%)
NEWS (ou -domain)
SMT s. SMa xinT Ag eemen 15 (21.4%) 22 (31.4%) 33 (47.2%)
All 40 (20.0%) 74 (37.0%) 86 (43.0%)
SMT s. Ma xin Ag eemen 11 (17.7%) 13 (21.0%) 38 (61.3%)
All 32 (16.0%) 64 (32.0%) 104 (52.0%)
SMa xinT s. Ma xin Ag eemen 19 (26.4%) 13 (18.1%) 40 (55.5%)
All 49 (24.5%) 54 (27.0%) 97 (48.5%)
Table 2: Manual e alua ion o andom subse o 100 sen ences o each es co pus.
Sou ce Legasa cuen a ya con un con enio sob e
la ecupe aci´
on de bienes comunales.
(English) Legasa al eady has a con en ion on he e-
co e y o communi y p ope y.
Re . 1 Legasak hi za mena du jada he i onda-
sunak be esku a zea en ingu uan.
Re . 2 Legasak badauka ondasun komunalak
be esku a zea i bu uzko hi za mena.
Ma xin Legasa jadanik kon a zen du hi za men
ba ekin he i-ondasunen be esku a ze
gainean.
SMT legasako hi za mena du dagoeneko
be esku a zea i bu uzko ondasunak
komunalak.
SMa xinT dagoeneko legasako hi za mena
be esku a zea i bu uzko ondasun
komunalak
Table 3: Example whe e an unde s andable ansla ion
ob ained by Ma xin is penalized by BLEU, bu he con-
using SMT ansla ion ge s a good BLEU sco e.
unde s and, and he same happens wi h he s ange
end o he sen ence. Howe e , his ansla ion ob-
ains a good e alua ion sco e because e e y wo d bu
one is in he e e ences.
4 Sea ching o human con o man
au oma ic me ics
In iew o he la ge di e ence be ween he esul s
ob ained by s anda d au oma ic me ics and he man-
ual e alua ion, and conside ing ha he human e al-
ua o s alue syn ac ical co ec ness mo e han he
common lexical me ics (such as BLEU and TER)
do, we conside ed he possibili y o using me ics
ha use a highe le el o linguis ic in o ma ion (Liu
and Gildea, 2005; Popo i´
c and Ney, 2007; Gim´
enez
and M`
a quez, 2007; Chan and Ng, 2008). Thus,
in addi ion o he s anda d BLEU and TER, we ap-
plied hese same me ics o e he sequences o syn-
ac ic ca ego ies, pa s o speech (PoS), esul ing
BLEU PoS and TER PoS. Table 4 shows how he
me ics ha use linguis ic in o ma ion ob ain mo e
simila esul s o hose achie ed by he manual e al-
ua ion. Thus, in ou ou -domain e alua ion he me -
ics ha use PoS in o ma ion show he same p e e -
ence be ween sys ems han he human assessmen .
Tha is, Ma xin ge s he bes esul s, ollowed by
SMa xinT and SMT. Simila ly, in he in-domain es
se , he human p e e ence o SMa xinT o e he s a-
is ical sys em is clea e wi h his ype o me ics.
Despi e his, PoS based me ics can no ully com-
pensa e he high penal y ha Ma xin ecei es and
his sys em emains he lowes anked in he Elhuya
es se (in-domain), al hough he dis ance is sho e .
Howe e , hose esul s a e p o ably biased by he
ac ha bo h SMT and SMa xinT sys ems a e op-
imized o ise hei BLEU sco e. Thus, hey ge
a high lexical ma ching o he e e ence, a he ex-
pense o he syn ac ical co ec ness. Simila ly, he
use o me ics ha only ake in o accoun a e en
mo e speci ic aspec o ansla ion, such as he co-
incidence o PoS, a e no sui able o be used as he
unique me ic o he whole de eloping cycle. Using
such me ics on SMT pa ame e op imiza ion, o
example, could lead o ge ansla ions whose lex-
ical co ec ion is ully igno ed. So his kind o me -

BLEU TER BLEU PoS TER PoS comb BLEU comb all
Elhuya (in-domain)
Ma xin 5.25 84.51 25.63 52.82 15.44 7.88
SMT 14.53 71.60 30.78 48.82 22.65 11.53
SMa xinT 14.48 70.50 31.96 47.07 23.22 11.82
Elhuya (in-domain)
hand e alua ed sen ences
Ma xin 5.85 84.95 26.68 52.19 16.27 8.29
SMT 12.75 75.58 30.15 49.37 21.45 11.38
SMa xinT 13.37 75.09 31.39 48.63 22.38 11.38
NEWS (ou -domain)
Ma xin 11.65 72.39 39.19 42.40 25.42 12.93
SMT 14.45 70.18 31.09 48.65 22.77 11.59
SMa xinT 15.08 67.72 34.55 45.56 24.82 12.62
NEWS (ou -domain)
hand e alua ed sen ences
Ma xin 11.01 73.55 38.74 43.07 24.88 12.65
SMT 11.32 73.08 29.56 50.49 20.44 10.41
SMa xinT 13.64 70.42 35.34 46.82 24.49 12.45
Table 4: Au oma ic sco es o all indi idual and hyb id sys ems.
ics should be combined wi h me ics ha also ake
in o accoun o he aspec s o he ansla ion, as lex-
ical ma ching. In he li e a u e di e en me hods o
me ic combina ion ha e been es ed. Among o he
me hods, one can ind hose based on linea com-
bina ions (Pad´
o e al., 2009; Liu and Gildea, 2007;
Gim´
enez and M`
a quez, 2008), eg ession based al-
go i hms (Paul e al., 2007; Alb ech and Hwa,
2008) o a a ie y o supe ised machine lea ning
algo i hms (Qui k e al., 2005; Amig´
o e al., 2005).
Due o i s simplici y and he esul s achie ed,
we decided o use he idea p esen ed by Gim´
enez
and M`
a quez (2008), whe e he di e en me ics
a e combined jus by means o he a i hme ic mean.
This me hod o combina ion, despi e i s simplici y,
ob ained compe i i e esul s on he Me icsMATR
sha ed ask (Callison-Bu ch e al., 2010). Thus we
ha e de ined wo me ics ha combine lexical in-
o ma ion wi h PoS in o ma ion: (1) one ha com-
bines he ou me ics (BLEU, TER, BLEU PoS and
TER PoS) we es ed and (2) ano he one ha com-
bines only BLEU wi h BLEU PoS.
BLEU and BLEU PoS a e quali y measu es
(highe sco e means highe quali y) while TER and
TER PoS a e e o measu e (lowe sco e means
highe quali y). Due o he di e en na u e o he
me ics and o be able o combine all o hese ou
me ics by means o he a i hme ic mean, we had o
modi y he alues o TER o become quali y mea-
su es. Thus, he new me ics a e calcula ed using
he ollowing o mulas:
Comb BLEU = (BLEU +BLEU P oS)/2
Comb All = (BLEU +BLEU PoS + (100 −TER)
+ (100 −TER P oS)) /4
The wo me ics ha combine lexical me ics
wi h PoS in o ma ion ob ained esul s simila o
hose based only on PoS, in e ms o p e e ence be-
ween sys ems. In he same way, BLEU PoS and
TER PoS, Comb BLEU and Comb All es ablished
he same p e e ence o de as he manual e alua ion,
excep in he case o Ma xin in he in-domain es
se . Bu , unlike hose me ics based only on PoS in-
o ma ion, he combined me ics a e mo e sui able
as hey allow a be e syn ac ic adequacy while hey
main ain co ec lexical ma chings.
In addi ion o his co ela ion a he documen
le el, we also wan ed o check he co ela ion o
each me ic a sen ence le el whe e manual assess-
men s we e se . Fo each sen ence in which bo h hu-
man assessmen s ag ee, we ha e compa ed he e-
sul wi h he p e e ence o each me ic. To de ine
which is he p e e ence o each me ic, we consid-
e ed ha he au oma ic me ic p e e s a ansla ion
i one o he ansla ions ge s a sco e 10% highe
han he o he . In cases whe e he ela i e di e ence
is no highe han 10%, we conside ha he au o-
ma ic me ic is no able o disc imina e be ween he
wo ansla ions. Table 5 shows he pe cen age o
sen ences whe e each au oma ic me ic’s p e e ence
coincides wi h he one se by bo h human e alua o s
(we disca d he cases in which human e alua ions
ha e no ag eed).
BLEU TER BLEU PoS TER PoS comb BLEU comb all
Elhuya (in-domain)
SMT s. SMa xinT 34 (64%) 33 (62%) 33 (62%) 30 (56%) 35 (66%) 31 (58%)
SMT s. Ma xin 23 (39%) 23 (39%) 25 (42%) 22 (37%) 24 (41%) 26 (44%)
SMa xinT s. Ma xin 25 (38%) 29 (44%) 29 (44%) 27 (41%) 25 (38%) 28 (42%)
NEWS (ou -domain)
SMT s. SMa xinT 35 (50%) 31 (44%) 36 (51%) 38 (54%) 38 (54%) 34 (49%)
SMT s. Ma xin 31 (50%) 29 (47%) 42 (68%) 38 (61%) 39 (63%) 42 (68%)
SMa xinT s. Ma xin 38 (53%) 38 (53%) 39 (54%) 36 (50%) 46 (64%) 36 (50%)
Table 5: Sen ence by sen ence co ela ion be ween human e alua ion and au oma ic me ics.
These igu es show ha he me ics based on lin-
guis ic in o ma ion (bo h, hose ha only uses PoS
in o ma ion and hose ha combine i wi h lexical
in o ma ion) ge mo e coincidences han hose ha
only use lexical in o ma ion (BLEU o TER).
5 Conclusions
In his wo k we p esen an in-dep h e alua ion o
SMa xinT, a hyb id sys em ha is con olled by
he RBMT ansla o and en iched wi h a wide a-
ie y o SMT ansla ion op ions. The esul s o
he human e alua ion, whe e he ansla ion o he
wo indi idual sys ems and SMa xinT we e com-
pa ed in pai s, es ablished ha Ma xin, he RBMT
sys em, achie ed he bes pe o mance ollowed by
SMa xinT, while he SMT sys em gene a ed he
wo s ansla ions.
Those esul s, e y a om wha he au oma ic
me ics (BLEU and TER) show, co obo a e he al-
eady known inadequacy o he me ics ha measu e
only he lexical ma ching o compa ing sys ems
ha use so di e en ansla ion pa adigms. This kind
o me ics a e biased in a o o he SMT, as i hap-
pens in ou e alua ion, whe e he s a is ical sys em
achie es he bes esul s in he in-domain e alua ion,
e en when i gene a es he wo s ansla ions acco d-
ing o he manual assessmen .
To add ess hese limi a ions o he me ics ha a e
only based on lexical ma ching, we de ined a cou-
ple o me ics ha seek o ensu e he syn ac ic co -
ec ness, calcula ing he same exp essions bu a he
PoS le el. These me ics, which a e able o assess
he syn ac ic co ec ness, ha e shown a highe le el
o ag eemen wi h human assessmen s bo h a docu-
men and sen ence le el.
Ne e heless, he me ics ha assess speci ic as-
pec s o he ansla ion (such as PoS ma ching) do
no ensu e he absolu e quali y o he ansla ion,
and should be combined wi h egula lexical ma ch-
ing me ics. A he ime o combining hese me ics,
we op ed o simplici y and we used he a i hme ic
mean. This me hod, despi e i s simplici y, has al-
eady shown i s sui abili y be o e.
Ou combined me ics a e simple and able o
main ain a highe co ela ion wi h manual e alua-
ion han he usual lexical me ics, while ensu e he
lexical ma ching.
We a e planning o use his simple combina ion
o me ics in de eloping new e sions o ou hyb id
sys em. Simul aneously we a e adap ing linguis ic
ools o he Asiya Open Toolki 3 o es o he new
e alua ion me ics ha conside a highe le el o lin-
guis ic in o ma ion.
Acknowledgmen s
This wo k has been pa ially unded by he
Eu opean Communi y’s Se en h F amewo k P o-
g amme (FP7/2007-2013) unde g an ag eemen
numbe 247914 (MOLTO p ojec , FP7-ICT-2009-4-
247914), he Spanish Minis y o Science and In-
no a ion (OpenMT-2 p ojec , TIN2009-14675-C03-
01) and he Local Go e nmen o he Basque Coun-
y (Be ba ek p ojec , IE09-262).
Re e ences
Joshua Alb ech and Rebecca Hwa. 2008. Reg ession o
machine ansla ion e alua ion a he sen ence le el.
Machine T ansla ion, pages 1–27.
I˜
naki Aleg ia, A an za D´
ıaz de Ila aza, Go ka Labaka,
Mikel Le sundi, Ainge i Mayo , and Kepa Sa asola.
2007. T ans e -based MT om spanish in o basque:
Reusabili y, s anda diza ion and open sou ce. Lec u e
No es in Compu e Science, 4394:374–384.
En ique Amig´
o, Julio Gonzalo, Anselmo Pe˜
nas, and Fe-
lisa Ve dejo. 2005. Qa la: a amewo k o he e alua-
3h p://nlp.lsi.upc.edu/asiya
ion o ex summa iza ion sys ems. In P oceedings o
he 43 d Annual Mee ing on Associa ion o Compu a-
ional Linguis ics, ACL ’05, pages 280–289, S ouds-
bu g, PA, USA. Associa ion o Compu a ional Lin-
guis ics.
Ch is Callison-Bu ch, Miles Osbo ne, and Philipp
Koehn. 2006. Re-e alua ing he Role o BLEU in
Machine T ansla ion Resea ch. In P oceedings o he
In e na ional Con e ence o Eu opean Chap e o he
Associa ion o Compu a ional Linguis ics (EACL),
pages 249–256.
Ch is Callison-Bu ch, Philipp Koehn, Ch is o Monz,
Kay Pe e son, Ma k P zybocki, and Oma F. Zaidan.
2010. Findings o he 2010 join wo kshop on s a-
is ical machine ansla ion and me ics o machine
ansla ion. In P oceedings o he Join Fi h Wo kshop
on S a is ical Machine T ansla ion and Me icsMATR,
WMT ’10, pages 17–53, S oudsbu g, PA, USA. As-
socia ion o Compu a ional Linguis ics.
Yee Seng Chan and Hwee Tou Ng. 2008. MAXSIM:
A maximum simila i y me ic o machine ansla ion
e alua ion. In P oceedings o ACL-08: HLT, pages
55–62, Columbus, Ohio, June. Associa ion o Com-
pu a ional Linguis ics.
C is ina Espa˜
na-Bone , Go ka Labaka, A an za D´
ıaz de
Ila aza, Lluis M`
a quez, and Kepa Sa asola. 2011.
Hyb id machine ansla ion guided by a ulebased sys-
em. In P oceedings MT Summi XIII, Xiamen, China,
Sep empe .
Jes´
us Gim´
enez and Llu´
ıs M`
a quez. 2007. Linguis ic ea-
u es o au oma ic e alua ion o he e ogenous m sys-
ems. In P oceedings o he Second Wo kshop on S a-
is ical Machine T ansla ion, S a MT ’07, pages 256–
264, S oudsbu g, PA, USA. Associa ion o Compu-
a ional Linguis ics.
Jes´
us Gim´
enez and Llu´
ıs M`
a quez. 2008. A smo gas-
bo d o ea u es o au oma ic MT e alua ion. In P o-
ceedings o he Thi d Wo kshop on S a is ical Machine
T ansla ion, pages 195–198, Columbus, Ohio, June.
Associa ion o Compu a ional Linguis ics.
Philipp Koehn and Ch is o Monz. 2006. Manual and
Au oma ic E alua ion o Machine T ansla ion be ween
Eu opean Languages. In In P oceedings o NAACL
2006 Wo kshop on S a is ical Machine T ansla ion,
pages 102–121.
Go ka Labaka. 2010. EUSMT: Inco po a ing Linguis-
ic In o ma ion in o SMT o a Mo phologically Rich
Language. I s use in SMT-RBMT-EBMT hyb ida ion.
Ph.D. hesis, Uni e si y o he Basque Coun y.
Ding Liu and Daniel Gildea. 2005. Syn ac ic ea u es o
e alua ion o machine ansla ion. In P oceedings o
ACL Wo kshop on In insic and Ex insic E alua ion
Measu es o MT and/o Summa iza ion, pages 25–32.
Ding Liu and Daniel Gildea. 2007. Sou ce-language ea-
u es and maximum co ela ion aining o machine
ansla ion e alua ion. In HLT-NAACL’07, pages 41–
48.
Ainge u Mayo . 2007. Ma xin: e egele an oina i u-
ako i zulpen au oma ikoko sis ema. Ph.D. hesis, Eu-
skal He iko Unibe si a ea.
I. Dan Melamed, Ryan G een, and Joseph P. Tu ian.
2003. P ecision and Recall o Machine T ansla ion.
In NAACL ’03: P oceedings o he 2003 Con e ence
o he No h Ame ican Chap e o he Associa ion o
Compu a ional Linguis ics on Human Language Tech-
nology, pages 61–63, Mo is own, NJ, USA. Associa-
ion o Compu a ional Linguis ics.
Sebas ian Pad´
o, Michel Galley, Dan Ju a sky, and Ch is
Manning. 2009. Robus machine ansla ion e alua-
ion wi h en ailmen ea u es. In P oceedings o he
Join Con e ence o he 47 h Annual Mee ing o he
ACL and he 4 h In e na ional Join Con e ence on
Na u al Language P ocessing o he AFNLP: Volume
1 - Volume 1, ACL ’09, pages 297–305, S oudsbu g,
PA, USA. Associa ion o Compu a ional Linguis ics.
Kisho e Papineni, Salim Roukos, Todd Wa d, and Wei-
Jing Zhu. 2002. Bleu: A me hod o au oma ic e al-
ua ion o machine ansla ion. In P oceedings o he
40 h Annual Mee ing o he Associa ion o Compu a-
ional Linguis ics, pages 311–318.
Michael Paul, And ew Finch, and Eiichi o Sumi a. 2007.
Reducing Human Assessmen s o Machine T ansla-
ion Quali y o Bina y Classi ie s. In P oceedings o
he 11 h Con e ence on Theo e ical and Me hodologi-
cal Issues in Machine T ansla ion (TMI).
Maja Popo i´
c and He mann Ney. 2007. Wo d e o a es:
decomposi ion o e pos classes and applica ions o e -
o analysis. In P oceedings o he Second Wo kshop
on S a is ical Machine T ansla ion, S a MT ’07, pages
48–55, S oudsbu g, PA, USA. Associa ion o Com-
pu a ional Linguis ics.
Ch is Qui k, A ul Menezes, and Colin Che y. 2005.
Dependency eele ansla ion: syn ac ically in o med
ph asal sm . In P oceedings o he 43 d Annual
Mee ing on Associa ion o Compu a ional Linguis-
ics, ACL ’05, pages 271–279, S oudsbu g, PA, USA.
Associa ion o Compu a ional Linguis ics.
M. Sno e , B. Do , R. Schwa z, L. Micciulla, and
J. Makhoul. 2006. A S udy o T ansla ion Edi Ra e
wi h Ta ge ed Human Anno a ion. In P oceedings o
AMTA, pages 223–231.