Deep evaluation of hybrid architectures: simple metrics correlated with human judgments

Deep e alua ion o hyb id a chi ec u es: simple me ics co ela ed wi h human judgmen s Go ka Labaka, A an za D´ ıaz de Ila aza, Kepa Sa asola Uni e si y o he Basque Coun y [email p o ec ed], [email p o ec ed],[email p o ec ed] C is ina Espa˜ na-Bone , Llu´ ıs M` a quez Uni e si a Poli ` ecnica de Ca alunya [email p o ec ed], [email p o ec ed] Abs ac The p ocess o de eloping hyb id MT sys ems is guided by he e alua ion me hod used o compa e di e en combina ions o basic sub- sys ems. This wo k p esen s a deep e alua- ion expe imen o a hyb id a chi ec u e ha ies o ge he bes o bo h wo lds, ule-based and s a is ical. In a i s e alua ion human assessmen s we e used o compa e jus he single s a is ical sys em and he hyb id one, he ule-based sys em was no compa ed by hand because he esul s o au oma ic e alu- a ion showed a clea disad an age. Bu a sec- ond and wide e alua ion expe imen su p is- ingly showed ha acco ding o human e al- ua ion he bes sys em was he ule-based, he one ha achie ed he wo s esul s us- ing au oma ic e alua ion. An examina ion o sen ences wi h con o e sial esul s sugges ed ha linguis ic well- o medness in he ou pu should be conside ed in e alua ion. A e ex- pe imen ing wi h 6 possible me ics we con- clude ha a simple a i hme ic mean o BLEU and BLEU calcula ed on pa s o speech o wo ds is clea ly a mo e human con o man me ic han lexical me ics alone. 1 In oduc ion The p ocess o de eloping hyb id MT sys ems is guided by he e alua ion me hod used o compa e di e en combina ions o basic subsys ems. Di ec human e alua ion is mo e accu a e bu un o una ely i is ex emely expensi e, so au oma ic me ics ha e o be used in p o o ype de eloping. Howe e he me hod should e alua e di e en sys ems wi h he same c i e ia, and hese c i e ia should be as close as possible o human judgmen . I is well known ha ule-based and ph ase- based s a is ical machine ansla ion pa adigms (RBMT and SMT, espec i ely) ha e complemen- a y s eng hs and weaknesses. Fi s , RBMT sys- ems end o p oduce syn ac ically be e ansla ions and deal wi h long dis ance dependencies, ag ee- men and cons i uen eo de ing in a be e way, since hey pe o m he analysis, ans e and gene a- ion s eps based on syn ac ic p inciples. On he bad side, hey usually ha e p oblems wi h lexical selec- ion due o a poo handling o wo d ambigui y. Also, in cases in which he inpu sen ence has an unex- pec ed syn ac ic s uc u e, he pa se may ail and he quali y o he ansla ion dec ease d ama ically. On he o he side, ph ase-based SMT models usu- ally do a be e job wi h lexical selec ion and gene al luency, since hey model lexical choice wi h dis i- bu ional c i e ia and explici p obabilis ic language models. Howe e , ph ase-based SMT sys ems usu- ally gene a e s uc u ally wo se ansla ions, since hey model ansla ion mo e locally and ha e p ob- lems wi h long dis ance eo de ing. They also end o p oduce e y ob ious e o s, which a e annoying o egula use s, e.g., lack o gende and numbe ag eemen , bad punc ua ion, e c. Mo eo e , SMT sys ems can expe ience a se e e deg ada ion o pe - o mance when applied o co po a di e en om hose used o aining (ou -o -domain e alua ion). I is also well known ha he BLEU me ic (Pap- ineni e al., 2002) is ac ually he mos used me ic in s a is ical MT. Bu se e al doub s ha e a isen a ound BLEU (Melamed e al., 2003; Callison-Bu ch e al., 2006; Koehn and Monz, 2006). In addi ion o he ac ha i is ex emely di icul o in e p e wha is being exp essed in BLEU (Melamed e al., 2003), imp o ing i s alue nei he gua an ees an imp o e- men in he ansla ion quali y (Callison-Bu ch e al., 2006) no o e s as much co ela ion wi h human judgmen as was belie ed (Koehn and Monz, 2006). Those p oblems ha e also been de ec ed when ans- la ing o Basque (Mayo , 2007; Labaka, 2010). In he las ew yea s, se e al new e alua ion me - ics ha e been sugges ed o conside a highe le el o linguis ic in o ma ion (Liu and Gildea, 2005; Popo i´ c and Ney, 2007; Chan and Ng, 2008), and di e en me hods o me ic combina ion ha e been es ed. Due o i s simplici y, we decided o use he idea p esen ed by Gim´ enez and M` a quez (2008), whe e he di e en simple me ics a e combined by means o he a i hme ic mean. In his wo k we p esen some su p ising esul s we ha e achie ed in a deep e alua ion o a hyb id a chi ec u e. In a i s s ep we used human e al- ua ion o compa e jus he single s a is ical sys em and he hyb id one, we did no compa e he ule- based sys em by hand because he esul s o au o- ma ic e alua ion showed a clea disad an age. Bu a second and wide e alua ion expe imen su p is- ingly showed ha acco ding o human e alua ion he bes sys em was he ule-based, he one ha achie ed he wo s esul s using au oma ic e alua ion. We ied o make a diagnosis o his phenomenon, and hen based on his we inally ound a simple bu mo e human con o man me ic ha we plan o use in aining new e sions o ou hyb id sys em. In he nex sec ion o his pape we desc ibe he hyb id sys em. Sec ion 3 p esen s he e alua ion ex- pe imen s: he co po a used in hem, he i s expe i- men compa ing jus he single s a is ical sys em and he hyb id one, and he second and wide e alua ion expe imen which compa es he all h ee sys ems. Then Sec ion 4 desc ibes he p ocess o sea ching o o he au oma ic me ics being mo e human con- o man . And inally, he las sec ion is de o ed o conclusions and u u e wo k. 2 The hyb id sys em, SMa xinT S a is ical Ma xin T ansla o , SMa xinT in sho , is a hyb id sys em con olled by he RBMT ansla o and en iched wi h a wide a ie y o SMT ansla ion op ions (Espa˜ na-Bone e al., 2011). The wo indi idual sys ems a e a ule-based Spanish-Basque sys em called Ma xin (Aleg ia e al., 2007) and a s anda d ph ase-based s a is ical MT sys em based on Moses which wo ks a he mo - pheme le el allowing o deal wi h he ich mo phol- ogy o Basque (Labaka, 2010). The ini ial analysis o he sou ce sen ence is done by Ma xin. I p oduces a dependency pa se ee, whe e he bounda ies o each ph ase a e ma ked. In o de o add hyb id unc ionali y wo new mod- ules a e in oduced o he RBMT a chi ec u e (Fig- u e 1): he ee en ichmen module, which inco - po a es SMT addi ional ansla ions o each ph ase o he syn ac ic ee; and a mono onous decoding module, which is esponsible o gene a ing he i- nal ansla ion by selec ing among RBMT and SMT pa ial ansla ion candida es om he en iched ee. The ee en ichmen module in oduces wo ypes o ansla ions o he syn ac ic cons i uen s gi en by Ma xin: 1) he SMT ansla ion(s) o e e y ph ase, and 2) he SMT ansla ion(s) o he en i e sub ee con aining ha ph ase. Fo example, he analysis o he es agmen “a i m´ o el conseje o de in e io ” (said he Sec e a y o in e io ) gi es wo ph ases: he head “a i m´ o” (said) and i s child en “el conse- je o de in e io ” ( he Sec e a y o in e io ). The ull ule-based ansla ion is “Ba ne Sailbu ua baiez a u zuen” and he ull SMT ansla ion is “esan zuen he izaingo sailbu uak”. SMa xinT conside s hese wo ph ases o he ansla ion o he ull sen ence, bu also he SMT ansla ions o hei cons i uen s (“esan zuen” and “he izaingo sailbu uak”). How- e e , sho ph ases may ha e a w ong SMT ans- la ion because o a lack o con ex . To o e come his p oblem SMa xinT also uses he ansla ion o a ph ase ex ac ed om a longe SMT ansla ion (“he izaingo sailbu uak” in he p e ious example). So, in o de o ansla e “a i m´ o el conseje o de in- e io ” he sys em has p oduced 5 dis inc ph ases, a numbe ha can be inc eased by conside ing a n- bes lis o SMT ou pu s. A e ee en ichmen , he ans e and gene a- ion s eps o he RBMT sys em a e ca ied ou in a usual way, and a inal mono onous decode chooses among he op ions. A key aspec o he pe o - mance o he sys em is he elec ion o he ea u es Figu e 1: Gene al a chi ec u e o SMa xinT. The RBMT modules which guide he MT p ocess a e he g ey boxes. o his decoding. The esul s we p esen he e a e ob ained wi h a se o ele en ea u es. Th ee o hem a e he usual SMT ea u es (language model, wo d penal y and ph ase penal y). We also include ou ea u es o show he o igin o he ph ase and he consensus among sys ems (a coun e indica ing how many di e en sys ems gene a ed he ph ase, wo bi- na y ea u es indica ing whe he he ph ase comes om he SMT/RBMT sys em o no , and he num- be o sou ce wo ds co e ed by he ph ase gene a ed by bo h indi idual sys ems simul aneously). Finally, we use he lexical p obabili ies in bo h di ec ions in wo o ms: a simila app oach o IBM-1 p ob- abili ies modi ied o ake unknown alignmen s in o accoun and a lexical p obabili y in e ed om he RBMT dic iona y. We e e he eade o Espa˜ na- Bone e al. (2011) o u he de ails. 3 Expe imen s In ou expe imen s we e alua e bo h indi idual sys ems and he inal hyb id: SMT, Ma xin and SMa xinT. The language pai o applica ion is dic- a ed by he ule-based sys em and, in his case, Ma xin wo ks wi h he Spanish- o-Basque ansla- ion. Basque and Spanish a e wo languages wi h e y di e en mo phologies and syn axes. 3.1 Bilingual and monolingual co po a The co pus buil o ain he SMT sys em consis s o ou subse s: (1) six e e ence books ansla ed man- ually by he ansla ion se ice o he Uni e si y o he Basque Coun y (EHUBooks); (2) a collec ion sen ences okens EHUBooks Spanish 39,583 1,036,605 Basque 794,284 Consume Spanish 61,104 1,347,831 Basque 1,060,695 Elhuya TM Spanish 186,003 3,160,494 Basque 2,291,388 Euskal elTB Spanish 222,070 3,078,079 Basque 2,405,287 To al Spanish 491,853 7,966,419 Basque 6,062,911 Table 1: S a is ics on he bilingual collec ion o pa allel co po a. o 1,036 a icles published in Spanish and Basque by he Consume E oski magazine1(Consume ); (3) ansla ion memo ies mos ly using adminis a- i e language de eloped by Elhuya 2(Elhuya TM); and (4) a ansla ion memo y including sho de- sc ip ions o TV p og ammes (Euskal elTB). Table 1 shows some s a is ics on he co po a, gi ing some igu es abou he numbe o sen ences and okens. The aining co pus is hen basically made up o adminis a i e documen s and desc ip ions o TV p og ams. Fo de elopmen and es ing we ex ac ed some adminis a i e da a o he in-domain e alua- ion and selec ed one collec ion o news o he ou - o -domain s udy, o aling h ee se s: Elhuya de el and Elhuya es : 1,500 segmen s each, ex ac ed om he adminis a i e documen s. 1h p:// e is a.consume .es 2h p://www.elhuya .o g/ NEWS es : 1,000 sen ences collec ed om Spanish newspape s wi h wo e e ences. Addi ionally, we collec ed a 21 million wo d monolingual co pus, which oge he wi h he Basque side o he pa allel bilingual co po a, builds up a 28 million wo d co pus. This monolingual co pus is also he e ogeneous, and includes ex om wo sou ces o news: he Basque co pus o Science and Technology (ZT co pus) and a icles published by Be ia newspape (Be ia co pus). 3.2 Fi s expe imen o e alua ion Acco ding o he au oma ic e alua ion, ca ied ou in he p e ious a icle and ex ended in Table 4, he ule-based Ma xin sys em is clea ly he wo s sys em ob aining he wo s sco es o bo h me ics (BLEU and TER) in bo h es co po a. On he o he hand, he e alua ion o he hyb id sys em a ies depending on he es se . On he in-domain co po a (Elhuya es se ), he BLEU sco e achie ed by SMa xinT is sligh ly wo se han he sco es ob ained by he single SMT sys em, bu be e acco ding o TER (Sno e e al., 2006) e alua ion. The dis inc beha io be- ween me ics and he small di e ences do no al- low us o de ine a clea p e e ence be ween s a is i- cal and hyb id sys ems. On he con a y, on he ou - domain co po a (NEWS es se ), SMa xinT consis- en ly a chi es be e sco es han any o he sys em. Based on hese esul s, we s a ed ha he low in- domain pe o mance o he Ma xin penalizes he hy- b id sys em, p e en ing i o o e come he single SMT sys em. Bu , in he ou -domain es se , whe e he sco es o Ma xin we e no so a om he es o he sys ems, ou hyb idiza ion echnique was able o combine he bes o bo h sys ems ob aining he bes ansla ion. In o de o e i y his asse ion, we ca ied ou an human e alua ion, whe e we asked ou e alua o s o de e mine he p e e ence be ween he hyb id and he SMT ansla ions o 100 sen- ences andomly chosen om he NEWS es se . The igu es ob ained co obo a ed ha he hyb id sys em ou pe o ms he single SMT sys em in he ou -domain co po a. 3.3 Deepe e alua ion: Human e alua ion o compa e he h ee sys ems In o de o ge a mo e de ailed insigh o he pe - o mance o ou sys ems, we ecen ly ex ended his manual e alua ion o he es o he sys ems and es co po a. Tha way, we selec ed ano he 100 sen- ences om he Elhuya es se and asked he same ou e alua o s o assess he p e e ence be ween he h ee sys em pai s (SMT-Ma xin, SMT-SMa xinT, Ma xin-SMa xinT). Su p isingly, acco ding o his manual e alua ion he bes sys em is he ule-based Ma xin sys em, he wo s anked one using au oma ic e alua ion. E en o in-domain e alua ion i is clea ly be e han he s a is ical sys em and o simila quali y as he hyb id one, ha is sligh ly supe io o he s a is ical sys em. Fo ou -domain e alua ion he di e ences a e e y clea : he ule-based Ma xin sys em clea ly ou pe - o ms he hyb id sys em and his one ou pe o ms he s a is ical sys em. This can be seen in Table 2. The able shows he numbe o imes ha a sys em is be e han he o he o hose sen ences whe e he e was ull ag eemen among e alua o s (Ag eemen ) and o he ull sub- se (All). Resul s a e gi en o he h ee sys em pai s on he wo es se s, he in-domain and he ou -o - domain ones. We con i med hese su p ising esul s o man- ual e alua ion by examining some examples whe e BLEU sco es did no e lec he di e ence o quali y be ween ansla ion ou pu s. Le us analyze he ex- ample shown in Table 3, ha is, he ansla ion o he sou ce sen ence “Legasa cuen a ya con un con enio sob e la ecupe aci´ on de bienes comunales.”. The able shows he sou ce sen ence wi h i s meaning in English oge he wi h wo ansla ion e e ences and he ou pu gi en by he wo indi idual sys ems. In his example, he ou pu o he ule-based sys- em is adequa e, bu BLEU is unable o ecognize some linguis ic equi alences: jadanik and jada a e synonymous, as well as be esku a zea en ingu uan and be esku a ze gainean. Simila ly he i onda- sunak and he i-ondasunen a e almos he same be- cause he “-” is op ional, and using Legasa ins ead o Legasak is a common e o easy o unde s and. The ollowing segmen s a e quasi equi alen s: hi za - mena du and kon a zen du hi za men ba ekin. All hese co espondences a e i ial o humans bu in- isible o he BLEU me ic. On he o he hand, he ou pu o he s a is ical sys- em is ha de o unde s and. By using Legasako in- s ead o Legasak, he sen ence becomes di icul o Sys em 1 Tied Sys em 2 Elhuya (in-domain) SMT s. SMa xinT Ag eemen 5 (9.4%) 31 (58.5%) 17 (32.1%) All 25 (12.5%) 109 (54.5%) 66 (33.0%) SMT s. Ma xin Ag eemen 14 (23.7%) 19 (32.2%) 26 (44.1%) All 41 (20.5%) 79 (39.5%) 80 (40.0%) SMa xinT s. Ma xin Ag eemen 19 (28.8%) 24 (36.4%) 23 (34.8%) All 59 (29.5%) 82 (41.0%) 59 (29.5%) NEWS (ou -domain) SMT s. SMa xinT Ag eemen 15 (21.4%) 22 (31.4%) 33 (47.2%) All 40 (20.0%) 74 (37.0%) 86 (43.0%) SMT s. Ma xin Ag eemen 11 (17.7%) 13 (21.0%) 38 (61.3%) All 32 (16.0%) 64 (32.0%) 104 (52.0%) SMa xinT s. Ma xin Ag eemen 19 (26.4%) 13 (18.1%) 40 (55.5%) All 49 (24.5%) 54 (27.0%) 97 (48.5%) Table 2: Manual e alua ion o andom subse o 100 sen ences o each es co pus. Sou ce Legasa cuen a ya con un con enio sob e la ecupe aci´ on de bienes comunales. (English) Legasa al eady has a con en ion on he e- co e y o communi y p ope y. Re . 1 Legasak hi za mena du jada he i onda- sunak be esku a zea en ingu uan. Re . 2 Legasak badauka ondasun komunalak be esku a zea i bu uzko hi za mena. Ma xin Legasa jadanik kon a zen du hi za men ba ekin he i-ondasunen be esku a ze gainean. SMT legasako hi za mena du dagoeneko be esku a zea i bu uzko ondasunak komunalak. SMa xinT dagoeneko legasako hi za mena be esku a zea i bu uzko ondasun komunalak Table 3: Example whe e an unde s andable ansla ion ob ained by Ma xin is penalized by BLEU, bu he con- using SMT ansla ion ge s a good BLEU sco e. unde s and, and he same happens wi h he s ange end o he sen ence. Howe e , his ansla ion ob- ains a good e alua ion sco e because e e y wo d bu one is in he e e ences. 4 Sea ching o human con o man au oma ic me ics In iew o he la ge di e ence be ween he esul s ob ained by s anda d au oma ic me ics and he man- ual e alua ion, and conside ing ha he human e al- ua o s alue syn ac ical co ec ness mo e han he common lexical me ics (such as BLEU and TER) do, we conside ed he possibili y o using me ics ha use a highe le el o linguis ic in o ma ion (Liu and Gildea, 2005; Popo i´ c and Ney, 2007; Gim´ enez and M` a quez, 2007; Chan and Ng, 2008). Thus, in addi ion o he s anda d BLEU and TER, we ap- plied hese same me ics o e he sequences o syn- ac ic ca ego ies, pa s o speech (PoS), esul ing BLEU PoS and TER PoS. Table 4 shows how he me ics ha use linguis ic in o ma ion ob ain mo e simila esul s o hose achie ed by he manual e al- ua ion. Thus, in ou ou -domain e alua ion he me - ics ha use PoS in o ma ion show he same p e e - ence be ween sys ems han he human assessmen . Tha is, Ma xin ge s he bes esul s, ollowed by SMa xinT and SMT. Simila ly, in he in-domain es se , he human p e e ence o SMa xinT o e he s a- is ical sys em is clea e wi h his ype o me ics. Despi e his, PoS based me ics can no ully com- pensa e he high penal y ha Ma xin ecei es and his sys em emains he lowes anked in he Elhuya es se (in-domain), al hough he dis ance is sho e . Howe e , hose esul s a e p o ably biased by he ac ha bo h SMT and SMa xinT sys ems a e op- imized o ise hei BLEU sco e. Thus, hey ge a high lexical ma ching o he e e ence, a he ex- pense o he syn ac ical co ec ness. Simila ly, he use o me ics ha only ake in o accoun a e en mo e speci ic aspec o ansla ion, such as he co- incidence o PoS, a e no sui able o be used as he unique me ic o he whole de eloping cycle. Using such me ics on SMT pa ame e op imiza ion, o example, could lead o ge ansla ions whose lex- ical co ec ion is ully igno ed. So his kind o me - BLEU TER BLEU PoS TER PoS comb BLEU comb all Elhuya (in-domain) Ma xin 5.25 84.51 25.63 52.82 15.44 7.88 SMT 14.53 71.60 30.78 48.82 22.65 11.53 SMa xinT 14.48 70.50 31.96 47.07 23.22 11.82 Elhuya (in-domain) hand e alua ed sen ences Ma xin 5.85 84.95 26.68 52.19 16.27 8.29 SMT 12.75 75.58 30.15 49.37 21.45 11.38 SMa xinT 13.37 75.09 31.39 48.63 22.38 11.38 NEWS (ou -domain) Ma xin 11.65 72.39 39.19 42.40 25.42 12.93 SMT 14.45 70.18 31.09 48.65 22.77 11.59 SMa xinT 15.08 67.72 34.55 45.56 24.82 12.62 NEWS (ou -domain) hand e alua ed sen ences Ma xin 11.01 73.55 38.74 43.07 24.88 12.65 SMT 11.32 73.08 29.56 50.49 20.44 10.41 SMa xinT 13.64 70.42 35.34 46.82 24.49 12.45 Table 4: Au oma ic sco es o all indi idual and hyb id sys ems. ics should be combined wi h me ics ha also ake in o accoun o he aspec s o he ansla ion, as lex- ical ma ching. In he li e a u e di e en me hods o me ic combina ion ha e been es ed. Among o he me hods, one can ind hose based on linea com- bina ions (Pad´ o e al., 2009; Liu and Gildea, 2007; Gim´ enez and M` a quez, 2008), eg ession based al- go i hms (Paul e al., 2007; Alb ech and Hwa, 2008) o a a ie y o supe ised machine lea ning algo i hms (Qui k e al., 2005; Amig´ o e al., 2005). Due o i s simplici y and he esul s achie ed, we decided o use he idea p esen ed by Gim´ enez and M` a quez (2008), whe e he di e en me ics a e combined jus by means o he a i hme ic mean. This me hod o combina ion, despi e i s simplici y, ob ained compe i i e esul s on he Me icsMATR sha ed ask (Callison-Bu ch e al., 2010). Thus we ha e de ined wo me ics ha combine lexical in- o ma ion wi h PoS in o ma ion: (1) one ha com- bines he ou me ics (BLEU, TER, BLEU PoS and TER PoS) we es ed and (2) ano he one ha com- bines only BLEU wi h BLEU PoS. BLEU and BLEU PoS a e quali y measu es (highe sco e means highe quali y) while TER and TER PoS a e e o measu e (lowe sco e means highe quali y). Due o he di e en na u e o he me ics and o be able o combine all o hese ou me ics by means o he a i hme ic mean, we had o modi y he alues o TER o become quali y mea- su es. Thus, he new me ics a e calcula ed using he ollowing o mulas: Comb BLEU = (BLEU +BLEU P oS)/2 Comb All = (BLEU +BLEU PoS + (100 −TER) + (100 −TER P oS)) /4 The wo me ics ha combine lexical me ics wi h PoS in o ma ion ob ained esul s simila o hose based only on PoS, in e ms o p e e ence be- ween sys ems. In he same way, BLEU PoS and TER PoS, Comb BLEU and Comb All es ablished he same p e e ence o de as he manual e alua ion, excep in he case o Ma xin in he in-domain es se . Bu , unlike hose me ics based only on PoS in- o ma ion, he combined me ics a e mo e sui able as hey allow a be e syn ac ic adequacy while hey main ain co ec lexical ma chings. In addi ion o his co ela ion a he documen le el, we also wan ed o check he co ela ion o each me ic a sen ence le el whe e manual assess- men s we e se . Fo each sen ence in which bo h hu- man assessmen s ag ee, we ha e compa ed he e- sul wi h he p e e ence o each me ic. To de ine which is he p e e ence o each me ic, we consid- e ed ha he au oma ic me ic p e e s a ansla ion i one o he ansla ions ge s a sco e 10% highe han he o he . In cases whe e he ela i e di e ence is no highe han 10%, we conside ha he au o- ma ic me ic is no able o disc imina e be ween he wo ansla ions. Table 5 shows he pe cen age o sen ences whe e each au oma ic me ic’s p e e ence coincides wi h he one se by bo h human e alua o s (we disca d he cases in which human e alua ions ha e no ag eed). BLEU TER BLEU PoS TER PoS comb BLEU comb all Elhuya (in-domain) SMT s. SMa xinT 34 (64%) 33 (62%) 33 (62%) 30 (56%) 35 (66%) 31 (58%) SMT s. Ma xin 23 (39%) 23 (39%) 25 (42%) 22 (37%) 24 (41%) 26 (44%) SMa xinT s. Ma xin 25 (38%) 29 (44%) 29 (44%) 27 (41%) 25 (38%) 28 (42%) NEWS (ou -domain) SMT s. SMa xinT 35 (50%) 31 (44%) 36 (51%) 38 (54%) 38 (54%) 34 (49%) SMT s. Ma xin 31 (50%) 29 (47%) 42 (68%) 38 (61%) 39 (63%) 42 (68%) SMa xinT s. Ma xin 38 (53%) 38 (53%) 39 (54%) 36 (50%) 46 (64%) 36 (50%) Table 5: Sen ence by sen ence co ela ion be ween human e alua ion and au oma ic me ics. These igu es show ha he me ics based on lin- guis ic in o ma ion (bo h, hose ha only uses PoS in o ma ion and hose ha combine i wi h lexical in o ma ion) ge mo e coincidences han hose ha only use lexical in o ma ion (BLEU o TER). 5 Conclusions In his wo k we p esen an in-dep h e alua ion o SMa xinT, a hyb id sys em ha is con olled by he RBMT ansla o and en iched wi h a wide a- ie y o SMT ansla ion op ions. The esul s o he human e alua ion, whe e he ansla ion o he wo indi idual sys ems and SMa xinT we e com- pa ed in pai s, es ablished ha Ma xin, he RBMT sys em, achie ed he bes pe o mance ollowed by SMa xinT, while he SMT sys em gene a ed he wo s ansla ions. Those esul s, e y a om wha he au oma ic me ics (BLEU and TER) show, co obo a e he al- eady known inadequacy o he me ics ha measu e only he lexical ma ching o compa ing sys ems ha use so di e en ansla ion pa adigms. This kind o me ics a e biased in a o o he SMT, as i hap- pens in ou e alua ion, whe e he s a is ical sys em achie es he bes esul s in he in-domain e alua ion, e en when i gene a es he wo s ansla ions acco d- ing o he manual assessmen . To add ess hese limi a ions o he me ics ha a e only based on lexical ma ching, we de ined a cou- ple o me ics ha seek o ensu e he syn ac ic co - ec ness, calcula ing he same exp essions bu a he PoS le el. These me ics, which a e able o assess he syn ac ic co ec ness, ha e shown a highe le el o ag eemen wi h human assessmen s bo h a docu- men and sen ence le el. Ne e heless, he me ics ha assess speci ic as- pec s o he ansla ion (such as PoS ma ching) do no ensu e he absolu e quali y o he ansla ion, and should be combined wi h egula lexical ma ch- ing me ics. A he ime o combining hese me ics, we op ed o simplici y and we used he a i hme ic mean. This me hod, despi e i s simplici y, has al- eady shown i s sui abili y be o e. Ou combined me ics a e simple and able o main ain a highe co ela ion wi h manual e alua- ion han he usual lexical me ics, while ensu e he lexical ma ching. We a e planning o use his simple combina ion o me ics in de eloping new e sions o ou hyb id sys em. Simul aneously we a e adap ing linguis ic ools o he Asiya Open Toolki 3 o es o he new e alua ion me ics ha conside a highe le el o lin- guis ic in o ma ion. Acknowledgmen s This wo k has been pa ially unded by he Eu opean Communi y’s Se en h F amewo k P o- g amme (FP7/2007-2013) unde g an ag eemen numbe 247914 (MOLTO p ojec , FP7-ICT-2009-4- 247914), he Spanish Minis y o Science and In- no a ion (OpenMT-2 p ojec , TIN2009-14675-C03- 01) and he Local Go e nmen o he Basque Coun- y (Be ba ek p ojec , IE09-262). Re e ences Joshua Alb ech and Rebecca Hwa. 2008. Reg ession o machine ansla ion e alua ion a he sen ence le el. Machine T ansla ion, pages 1–27. I˜ naki Aleg ia, A an za D´ ıaz de Ila aza, Go ka Labaka, Mikel Le sundi, Ainge i Mayo , and Kepa Sa asola. 2007. T ans e -based MT om spanish in o basque: Reusabili y, s anda diza ion and open sou ce. Lec u e No es in Compu e Science, 4394:374–384. En ique Amig´ o, Julio Gonzalo, Anselmo Pe˜ nas, and Fe- lisa Ve dejo. 2005. Qa la: a amewo k o he e alua- 3h p://nlp.lsi.upc.edu/asiya ion o ex summa iza ion sys ems. In P oceedings o he 43 d Annual Mee ing on Associa ion o Compu a- ional Linguis ics, ACL ’05, pages 280–289, S ouds- bu g, PA, USA. Associa ion o Compu a ional Lin- guis ics. Ch is Callison-Bu ch, Miles Osbo ne, and Philipp Koehn. 2006. Re-e alua ing he Role o BLEU in Machine T ansla ion Resea ch. In P oceedings o he In e na ional Con e ence o Eu opean Chap e o he Associa ion o Compu a ional Linguis ics (EACL), pages 249–256. Ch is Callison-Bu ch, Philipp Koehn, Ch is o Monz, Kay Pe e son, Ma k P zybocki, and Oma F. Zaidan. 2010. Findings o he 2010 join wo kshop on s a- is ical machine ansla ion and me ics o machine ansla ion. In P oceedings o he Join Fi h Wo kshop on S a is ical Machine T ansla ion and Me icsMATR, WMT ’10, pages 17–53, S oudsbu g, PA, USA. As- socia ion o Compu a ional Linguis ics. Yee Seng Chan and Hwee Tou Ng. 2008. MAXSIM: A maximum simila i y me ic o machine ansla ion e alua ion. In P oceedings o ACL-08: HLT, pages 55–62, Columbus, Ohio, June. Associa ion o Com- pu a ional Linguis ics. C is ina Espa˜ na-Bone , Go ka Labaka, A an za D´ ıaz de Ila aza, Lluis M` a quez, and Kepa Sa asola. 2011. Hyb id machine ansla ion guided by a ulebased sys- em. In P oceedings MT Summi XIII, Xiamen, China, Sep empe . Jes´ us Gim´ enez and Llu´ ıs M` a quez. 2007. Linguis ic ea- u es o au oma ic e alua ion o he e ogenous m sys- ems. In P oceedings o he Second Wo kshop on S a- is ical Machine T ansla ion, S a MT ’07, pages 256– 264, S oudsbu g, PA, USA. Associa ion o Compu- a ional Linguis ics. Jes´ us Gim´ enez and Llu´ ıs M` a quez. 2008. A smo gas- bo d o ea u es o au oma ic MT e alua ion. In P o- ceedings o he Thi d Wo kshop on S a is ical Machine T ansla ion, pages 195–198, Columbus, Ohio, June. Associa ion o Compu a ional Linguis ics. Philipp Koehn and Ch is o Monz. 2006. Manual and Au oma ic E alua ion o Machine T ansla ion be ween Eu opean Languages. In In P oceedings o NAACL 2006 Wo kshop on S a is ical Machine T ansla ion, pages 102–121. Go ka Labaka. 2010. EUSMT: Inco po a ing Linguis- ic In o ma ion in o SMT o a Mo phologically Rich Language. I s use in SMT-RBMT-EBMT hyb ida ion. Ph.D. hesis, Uni e si y o he Basque Coun y. Ding Liu and Daniel Gildea. 2005. Syn ac ic ea u es o e alua ion o machine ansla ion. In P oceedings o ACL Wo kshop on In insic and Ex insic E alua ion Measu es o MT and/o Summa iza ion, pages 25–32. Ding Liu and Daniel Gildea. 2007. Sou ce-language ea- u es and maximum co ela ion aining o machine ansla ion e alua ion. In HLT-NAACL’07, pages 41– 48. Ainge u Mayo . 2007. Ma xin: e egele an oina i u- ako i zulpen au oma ikoko sis ema. Ph.D. hesis, Eu- skal He iko Unibe si a ea. I. Dan Melamed, Ryan G een, and Joseph P. Tu ian. 2003. P ecision and Recall o Machine T ansla ion. In NAACL ’03: P oceedings o he 2003 Con e ence o he No h Ame ican Chap e o he Associa ion o Compu a ional Linguis ics on Human Language Tech- nology, pages 61–63, Mo is own, NJ, USA. Associa- ion o Compu a ional Linguis ics. Sebas ian Pad´ o, Michel Galley, Dan Ju a sky, and Ch is Manning. 2009. Robus machine ansla ion e alua- ion wi h en ailmen ea u es. In P oceedings o he Join Con e ence o he 47 h Annual Mee ing o he ACL and he 4 h In e na ional Join Con e ence on Na u al Language P ocessing o he AFNLP: Volume 1 - Volume 1, ACL ’09, pages 297–305, S oudsbu g, PA, USA. Associa ion o Compu a ional Linguis ics. Kisho e Papineni, Salim Roukos, Todd Wa d, and Wei- Jing Zhu. 2002. Bleu: A me hod o au oma ic e al- ua ion o machine ansla ion. In P oceedings o he 40 h Annual Mee ing o he Associa ion o Compu a- ional Linguis ics, pages 311–318. Michael Paul, And ew Finch, and Eiichi o Sumi a. 2007. Reducing Human Assessmen s o Machine T ansla- ion Quali y o Bina y Classi ie s. In P oceedings o he 11 h Con e ence on Theo e ical and Me hodologi- cal Issues in Machine T ansla ion (TMI). Maja Popo i´ c and He mann Ney. 2007. Wo d e o a es: decomposi ion o e pos classes and applica ions o e - o analysis. In P oceedings o he Second Wo kshop on S a is ical Machine T ansla ion, S a MT ’07, pages 48–55, S oudsbu g, PA, USA. Associa ion o Com- pu a ional Linguis ics. Ch is Qui k, A ul Menezes, and Colin Che y. 2005. Dependency eele ansla ion: syn ac ically in o med ph asal sm . In P oceedings o he 43 d Annual Mee ing on Associa ion o Compu a ional Linguis- ics, ACL ’05, pages 271–279, S oudsbu g, PA, USA. Associa ion o Compu a ional Linguis ics. M. Sno e , B. Do , R. Schwa z, L. Micciulla, and J. Makhoul. 2006. A S udy o T ansla ion Edi Ra e wi h Ta ge ed Human Anno a ion. In P oceedings o AMTA, pages 223–231.