FEMsum: A Flexible Eclec ic Mul i ask
Summa ize A chi ec u e e alua ed in
Mul idocumen Tasks ?
.
Ma ia Fuen es, Ho acio Rod ´ıguez, Jo di Tu mo
TALP Resea ch Cen e , Uni e si a Poli `ecnica de Ca alunya Ba celona, Spain
Abs ac
This a icle desc ibes wo ypes o summa iza ion app oaches in eg a ed in a lexible
a chi ec u e o mul i ask summa iza ion. The i s ype is based on he use o lexical
ea u es, while he second one is g ounded on syn ac ic and seman ic in o ma ion.
All he app oaches ha e been e alua ed in expe imen s whe e, gi en a se o docu-
men s, hey a e expec ed o p oduce summa ies answe ing a use need (exp essed
by a que y) in a educed se o ele an ex ual agmen s. Thei pe o mance is
analyzed in wo di e en asks: w i en news and scien i ic o al p esen a ions.
Keywo ds: Tex Summa iza ion, Spon aneous Speech Summa iza ion.
1 In oduc ion
La ge amoun s o digi al in o ma ion a e p oduced on a daily basis in he con-
ex o human in e ac ion e en s, such as news epo s, scien i ic p esen a ions,
mee ings, e c. Documen s gene a ed om hese e en s can be o di e en na-
u e in e ms o hei media (e.g., w i en, audio, ideo), hei domain (e.g.,
jou nalism, poli ics, esea ch, business), o he scena io hey o igina e om
(e.g., newspape , elephone con e sa ion, poli ical speech, business mee ing,
?This esea ch has been pa ially unded by he Eu opean Commission (CHIL
p ojec , IST-2004506969) and he Spanish go e nmen (TIN-2004-0171-E). We
would like o hank all CHIL e alua o s. We a e especially g a e ul o Daniel Fe ´es,
En ique Al onseca and Rose Sau ´ı.
Email add esses: [email p o ec ed] (Ma ia Fuen es),
[email p o ec ed] (Ho acio Rod ´ıguez), [email p o ec ed] (Jo di Tu mo).
P ep in submi ed o Else ie 19 Decembe 2006
o scien i ic con e ence). In his con ex , au oma ic summa iza ion can de i-
ni ely help deal wi h he inc easing amoun o a ailable in o ma ion.
Au oma ic documen summa iza ion s ongly depends no only on he o iginal
documen ea u es, bu also on he use needs (e.g., size o he summa y,
ou pu media, con en ela ed o a que y). Cu en s a e-o - he-a wo k in
he ield e lec s such a iabili y. Mos o he esea ch is based on English
newspape and newswi e da a, like he wo k de eloped in he con ex o he
Documen Unde s anding Con e ence 1(DUC), in bo h o i s wo e sions:
single documen (SDS) and mul idocumen summa iza ion (MDS). Less e o
has been de o ed o summa izing spon aneous speech, mos o he esea ch
ocusing on b oadcas news, ypically ead aloud om a w i en ex . Cu en
wo k on o al p esen a ions ends o be based on one single documen , he
speech ansc ip , [7], [5], [2], al hough some o he wo k ocus on di ec ly
summa izing he speech signal o lec u es [8].
S udies such as Sh ibe g [12] show ha o al communica ion is ha de o p o-
cess han w i en ex . Fo ha eason, we p opose using a mul idocumen
summa ize capable o handling documen s om di e en media ypes. Com-
bining documen s om di e en media can help coun e ac no only he di -
icul ies in he p ocessing o o al communica ion, bu also hose e o s in o-
duced by au oma ic speech ecognize s (ASRs). The cu en wo d e o a e
(WER) in speake -independen ASRs is a ound 25% on close- alking mic o-
phone, and 50% on a -dis ance. In his a icle we p esen FEMsum, a lexible
eclec ic mul i ask summa ize which will be e alua ed in wo di e en sce-
na ios: w i en news and scien i ic o al p esen a ions. FEMsum is a lexible
sys em capable o dealing wi h di e en summa iza ion asks by combining
in o ma ion om documen s o di e en so , i a ailable, and ha akes in o
accoun he use needs as well as he pa icula ea u es o he documen s o
be summa ized.
Nex sec ion gi es an o e iew o FEMsum’s gene al a chi ec u e. Sec ion 3
desc ibes he componen s o ou ool. We will see ha combining hem in
di e en ways leads o dis inc summa ize app oaches, ocusing he e o e on
MDS asks. In pa icula , wo di e en app oaches ha e been explo ed he e. A
i s one, based on lexical in o ma ion (LEX), and a second one, which uses a
iche seman ic ep esen a ion in o de o a oid edundancy and imp o e he
cohesion o he esul ing summa y (SEM). Sec ion 4 discusses he expe imen al
esul s ob ained om e alua ing bo h app oaches in di e en use -o ien ed,
que y- ocused mul idocumen summa iza ion asks. Finally, we conclude in
Sec ion 5 wi h he conclusions om he esea ch.
1h p:/www-nlpi .nis .go /p ojec s/duc/
2
2 Func ional O e iew o FEMsum’s A chi ec u e
The au oma ic summa iza ion sys em p esen ed he e is a highly modula
and pa ame e izable sys em able o deal wi h di e en in o ma ion needs. An
o e iew o he sys em ea u ing i s basic unc ionali ies and global a chi ec-
u e is depic ed in Figu e 1. The unc ional equi emen s a e se by means o a
pa ame e se spli ed in o inpu and ou pu se ings. Inpu se ings conce n
he cha ac e is ics o he documen s o be summa ized, while ou pu se ings
apply o he con en and p esen a ion o he summa y.
Inpu se ings include he ollowing pa ame e s:
•Domain. The summa ize can be domain independen o domain es ic ed.
In he second case, addi ional knowledge sou ces can be included, such as:
lis s o equen colloca ions om scien i ic pape s [5]; o a se o gaze ee s
o help inc ease he accu acy and ob ain ine classes in he named en i ies
classi ica ion ask es ic ed o he geog aphical domain.
•Documen s uc u e. The sys em can ake in o accoun in o ma ion de i ed
om he documen s uc u e (posi ion, i le, sec ions, o a ailable ags).
•Language. Cu en ly English and Spanish a e suppo ed. The linguis ic p o-
cessing pe o mance depends hea ily on his pa ame e .
•Media. Di e en media a e conside ed depending on he scena io o be deal
wi h ( ideo, audio, well w i en ex o any kind o ex ual documen ).
•Uni . Bo h single documen s and collec ions o ela ed documen s a e used
o SDS and MDS espec i ely.
•Gen e. We conside bo h gen e independen and dependen : jou nalis ic o
scien i ic (pape s, spon aneous speech, au ho no es and slides) op ions.
Ou pu se ings include:
•Con en . In o de o ex ac he ele an agmen s om he inpu doc-
umen s, he sys em can ake in o accoun he wo ds om he associa ed
que y (in case he e is a na u al language ques ion o lis o keywo ds), o
all he wo ds in he collec ion (i he e is no such que y).
•Size. Numbe o wo ds o he summa y.
•Documen es ic ion. Used o il e ing ou some ypes o documen s, such
as hose coming om a speci ic media o gen e.
•Ou pu o ma . The summa y can be p esen ed o he use as ex , syn he-
sized oice om ex , o as an audio/ ideo eco ded segmen .
3
3 FEMsum componen s
In o de o achie e he unc ionali ies p esen ed abo e, he sys em is o ga-
nized in h ee main componen s (see Figu e 1): Rele an In o ma ion De ec o
(RID), Con en Ex ac o (CE), and Summa y Compose (SC). In addi ion,
he e is a Que y P ocessing componen (QP). No all he componen s a e
needed o all he app oaches. In ac , in he expe imen s epo ed he e only
RID and SC a e always used.
3.1 Rele an In o ma ion De ec o
The RID module p o ides a anked se o ele an Tex Uni s (TU). The
de ini ion o TU depends basically on he inpu media. Fo ins ance, in he
case o well w i en ex , he TU is he sen ence. Two di e en s a egies a e
used, depending on he exis ence o a use que y (a NL ques ion o a lis o
keywo ds). I such a que y exis s, o each documen se he p onoun e e ence
is sol ed, he ex is lemma ized and indexed, and a Passage Re ie al (PR)
so wa e (JIRS [6] in he epo ed expe imen s) is used o ob ain he mos
ele an TUs. The sys em e ie es he passages wi h he highes simila i y
be ween he la ges n-g am o he que y and he one in he passage. RID
e u ns NTUs om passages ela ed o he que y. The de aul alue o Nis
no ixed, bu i is he numbe o TUs om passages selec ed in some o he
execu ions based on pa icula use need. I he summa y is no que y-d i en,
he sys em can use all TUs in he documen se , o pe o m a selec ion and
anking using a *id me ic o e he whole se .
3.2 Con en Ex ac o
As can be seen in Figu e 2, he CE, consis s o h ee componen s: a Linguis ic
P ocesso (LP), a Candida es Simila i y Ma ix Gene a o (CSMG), and a
Candida es Selec o (CS). Inpu o CE is he se o NTUs p o ided by RID.
All hese TUs a e p ocessed by LP. Then, CSMG compu es he simila i ies
among hem, and he mos app opia e ones a e p oposed by CE o be pa o
he summa y.
3.2.1 Linguis ic P ocesso
The LP is illus a ed in Figu e 3. I consis s o a pipeline o gene al pu pose NL
p ocesso s pe o ming: okeniza ion, POS agging, lemma iza ion, ine g ained
4
named en i ies ecogni ion and classi ica ion (NERC), syn ac ic pa sing, se-
man ic labeling (wi h Wo dNe synse s, Magnini’s domain ma ke s, and Eu-
oWo dNe Top Concep On ology labels), discou se ma ke anno a ion, and
seman ic analysis. Some o hese ools a e language dependen (English and
Spanish), while o he s a e gene al ools uned o a speci ic language. The same
ools a e used o he linguis ic p ocessing o he RID esul and o he que y
(QP) –when needed. The speci ic ools o be used in each case depend on he
ype o TU in ol ed and he inpu language (see [4] o mo e de ails).
Tools used o Spanish include:
•F eeLing, which pe o ms okeniza ion, mo phological analysis, POS ag-
ging, lemma iza ion, and pa ial pa sing.
•ABIONET, a NERC on basic MUC ca ego ies (pe son, loca ion, o ganiza-
ion, o he s).
•Eu oWo dNe , used o ob ain he lis o synse s (wi hou a emp ing Wo d
Sense Disambigua ion), a lis o hype nyms o each synse up o he op o
he axonomy, and he Top Concep On ology class.
Tools used o English include:
•TnT, a s a is ical POS agge .
•Wo dNe lemma ize 2.0.
•ABIONET.
•Wo dNe
•A modi ied e sion o he Collins’ pa se which pe o ms ull pa sing and
obus de ec ion o e bal p edica e a gumen s.
•Alembic, a NERC wi h MUC classes, used in o de o boos ABIONET
pe o mance.
As a esul , sen ences a e en iched wi h lexical (sen ) and syn ac ic (sin ) lan-
guage dependen ep esen a ions. Fo each sen ence, i s syn ac ic cons i uen
s uc u e (including head speci ica ion) and he syn ac ic ela ions be ween
i s cons i uen s (subjec , di ec and indi ec objec , modi ie s) a e ob ained.
F om sen and sin , a seman ic ep esen a ion o he sen ence is p oduced,
he en i onmen (en ). The in o ma ion in each o hese componen s is he
ollowing:
Sen p o ides lexical in o ma ion o each wo d: o m, lemma, POS ag, se-
man ic class o NE, lis o WN o EWN synse s and, whene e possible,
de i a ional in o ma ion.
Sin con ains wo lis s: one eco ding he syn ac ic cons i uen s uc u e (ba-
sically nominal, p eposi ional, and e bal ph ases), and he o he ep esen -
ing he dependencies be ween hese cons i uen s.
En is a seman ic-ne wo k-like ep esen a ion compu ed using a p ocess ha
ex ac s he seman ic uni s (nodes) and he seman ic ela ions (edges) hold-
5

ing be ween he di e en okens in sen . Uni and ela ion ypes belong o an
on ology o abou 100 seman ic classes (as pe son, ci y, ac ion, magni ude,
e c.), and 25 ela ions be ween hem (mos ly bina y, as ime o e en , ac-
o o ac ion, loca ion o e en , e c.). Bo h classes and ela ions a e ela ed
by axonomic links (see [4] o de ails) allowing o inhe i ance. Table 1 p o-
ides an example o a sen ence en i onmen (en ), and Figu e 4 gi es he
comple e ep esen a ion o ano he sen ence.
3.2.2 Candida es Simila i y Ma ix Gene a o
CSMG is in cha ge o compu ing he simila i y ma ix among candida es. Fo
ha pu pose, i uses he en i onmen o each candida e TU (sen ence in he
epo ed expe imen s). En i onmen s a e ans o med in o labeled di ec ed
g aph ep esen a ion, whe e nodes a e assigned o posi ions in he sen ence
and labeled wi h he co esponding oken, and edges a e assigned o p edica es
(a dummy node, 0, is used o ep esen ing una y p edica es). Only una y
and bina y p edica es a e used. Figu e 5 is he g aph ep esen a ion o he
en i onmen in Table 1.
On op o his ep esen a ion, a ich panoply o lexico-seman ic p oximi y mea-
su es be ween sen ences ha e been buil . Each measu e combines wo compo-
nen s:
•A lexical componen which includes he se o common okens, i.e. hose
occu ing in bo h sen ences. The size o his se and he s eng h o he
compa ibili y links be ween i s membe s a e used o de ining he measu e.
A lexible way o measu ing oken-le el compa ibili y has been empi ically
se , anging om wo d- o m iden i y, lemma iden i y, o e lapping o Wo d-
Ne synse s, app oxima e s ing ma ching be ween Named En i ies e c. Fo
ins ance, ”Romano P odi” is lexically compa ible wi h ”R. P odi” wi h a
sco e o 0.5 and wi h ”P odi” wi h a sco e o 0.41. ”I aly” and ”I alian” a e
also compa ible wi h sco e 0.7.
•A seman ic componen , compu ed o e he subg aphs co esponding o he
se o lexically compa ible nodes. Fou di e en measu es ha e been de ined:
·S ic o e lapping o una y p edica es.
·S ic o e lapping o bina y p edica es.
·Loose o e lapping o una y p edica es.
·Loose o e lapping o bina y p edica es.
The loose e sions allow a elaxed ma ching o p edica es by climbing up in
he on ology o p edica es, e.g. p o ided ha A and B a e lexically compa -
ible, i en ci y(A) can ma ch i en p ope place(B),loca ion(B) o en i y(B).
Ob iously, loose o e lapping implies a penal y on he sco e.
Se e al ways o combining he simple sco es ha e been conside ed and es ed.
6
Once an app op ia e measu e has been selec ed, we can compu e he simila i y
be ween e e y sen ence pai .
3.2.3 Candida es Selec o
In o de o selec he candida es, h ee c i e ia ha e been aken in o accoun :
•Rele ance (wi h espec o he que y o any o he elemen )
•Densi y and cohesion
•An i- edundancy
CS p oceeds in he ollowing s eps:
Le Sim be he simila i y ma ix, Candida es a lis o candida e TUs, and
Summa y an o de ed lis o TUs o be included in he summa y.
(1) Se Candida es o he lis p o ided by RID componen .
(2) Se Summa y o he emp y lis .
(3) Se Sim o he ma ix con aining he simila i y alues be ween membe s
om Candida es.
(4) Fo each candida e in Candida es, compu e a sco e ha akes in o ac-
coun he ini ial ele ance sco e and he alues in Sim. The sco e used
is based on PageRank, as used by Mihalcea and Ta au [10], bu wi hou
making he dis inc ion be ween inpu and ou pu links.
(5) So Candida es by his sco e.
(6) Append he mos sco ed candida e ( he head o he lis ) o he Summa y
and emo e i om Candida es.
(7) In o de o p e en o e lapping, he S% TUs mos simila (using Sim) o
he one selec ed in he p e ious s ep a e emo ed as well om Candida es.
The R% leas sco ed TUs a e also emo ed om Candida es.
(8) I Candida es is no emp y go o 4.
3.3 Summa y Compose
Fo he summa y composi ion, wo di e en app oaches ha e been explo ed.
The i s one is based on lexical in o ma ion (LEX). The second one uses a
iche seman ic ep esen a ion in o de o a oid edundancy and o imp o e
he cohesion o he esul ing summa y (SEM).
The inpu se o candida e TUs in he LEX app oach consis s o hose TUs
p e iously de ec ed by he RID componen as ele an acco ding o he opic.
In con as , in he SEM app oach, he inpu se consis s o hose TUs ex-
ac ed by he CE componen . Summa y TUs a e selec ed by ele ance un il
7
he desi ed summa y size is achie ed. Fo each selec ed TU, i is checked
whe he he p e ious sen ence in he o iginal documen is also a candida e.
I posi i e, bo h a e added o he Summa y in he o de hey appea in he
o iginal documen .
4 Que y o ien ed MDS e alua ion
In his sec ion, we desc ibe he expe imen s we ca ied ou in o de o e alua e
ou sys em’s pe o mance when dealing wi h di e en que y-o ien ed MDS
asks. The e alua ion is pe o med on wo asks. On one hand, Sec ion 4.1
epo s he expe imen s and esul s o FEMsum in he in e na ional DUC
2006 e alua ion o w i en news scena io. On he o he hand, Sec ion 4.2
analyzes he esul s ob ained in a simila ask wi hin he amewo k o he
CHIL 2p ojec o scien i ic o al p esen a ion documen s.
The asks in he DUC 2006 and CHIL amewo ks di e in he ollowing
aspec s:
•que y (complex s. lis o keywo ds),
•summa y leng h (250 s. 100),
•numbe o inpu documen (25 s. abou 4),
•e alua ion es (50 opics s 20, 10 opics x 2 que ies).
•numbe o manual summa y models (4 abs ac based s. 3 ex ac based).
•domain (jou nalis ic s scien i ic),
•inpu media (well w i en ex s. aw ex comming om: spon aneous
speech, and documen s ela ed o an o al p esen a ion),
•gen e (w i en jou nalism s scien i ic spon aneous speech, esea ch pape s,
and au ho no es o slides om a p esen a ion).
4.1 Expe imen s in w i en news scena io
This sec ion desc ibes he DUC 2006 e alua ion amewo k, ou app oach
se ings o each kind o summa y p oduced in ou DUC 2006 pa icipa ion,
and he esul s ob ained.
4.1.1 E alua ion F amewo k
Fo he DUC 2006 e alua ion, we we e p o ided wi h 50 opics which had been
selec ed o be used as es da a. Each opic had assigned a clus e o 25 ela ed
2h p://chil.se e .de/se le /is/101/
8
ex ual news documen s, as well as a s a emen desc ibing he in o ma ion ha
could be answe ed using his documen clus e . The opic s a emen could be
in he o m o a ques ion o se o ela ed ques ions and can include backg ound
in o ma ion ha he assesso has conside ed would cla i y his/he in o ma ion
need. Fo each opic 4 manual summa ies we e p oduced a NIST.
The DUC baseline was a simply sys em ha e u ned all he leading sen-
ences (up o 250 wo ds) o he mos ecen documen . All 34 DUC 2006
pa icipa ing sys ems and he baseline we e e alua ed a wo le els: manu-
ally (Linguis ic quali y and Responsi eness) and au oma ically (me ics om
he package ROUGE, he Recall-O ien ed Unde s udy o Gis ing E alua ion
[9]). Manual e alua ion sco ed each aspec o a gi en summa y as 1: e y poo ,
2:poo , 3:accep able, 4:good, o 5: e y good. In addi ion, ou un is one o
he 21 pa icipan sys ems ha we e also manually e alua ed by means o he
py amid me hod [11].
4.1.2 FEMsum se ings in DUC 2006
Ou goal in pa icipa ing a DUC was o e alua e a numbe o aspec s o
ou sys em. We he e o e submi ed h ee di e en kinds o au oma ic sum-
ma ies in a single un: one lexically based (LEX), and wo seman ically based
(SEM150, SEM250). Ou o he 50 summa ies we we e expec ed o submi , 7
we e p oduced using he LEX app oach, 13 by means o he SEM150 s a egy,
and 30 by using he SEM250 one. Ou sys em was assigned he iden i ica ion
numbe 19.
Gi en he que y, a common c ucial s ep in all he app oaches is o de ec
he mos ele an TUs (sen ences in his expe imen s). We decided o ix a
maximum numbe o sen ences de ec ed as ele an by RID. Fo ha eason
we use he co pus o sen ences de ec ed as pa o a manual summa y in DUC
2005 p oposed by Copeck and Spakowicz [3]. Analyzing P ecision and Recall
Nwas empi ically ixed in a maximum o 250.
In he LEX app oach, ele an sen ences a e de ec ed by RID and hen SC is
applied o ob ain he summa ies. On he o he hand, in bo h SEM app oaches
he ini ial c i e ia o sen ence ele ance is ha sen ences om a same documen
a e conside ed o ha e a simila ele ance, independen ly o he RID sco e. In
he SEM250 s a egy, all he sen ences om he RID ou pu a e aken as CE
inpu , whe eas in SEM150 he inpu o CE is he clus e o 150 sen ences
om he i s documen s in he se . SEM150 ends o educe he numbe o
documen s whose con en is candida e o appea in he summa y.
9
Table 1. Sample o en i onmen buil om a sen ence.
Table 2. DUC FEMsum manually e alua ion o linguis ic quali y sco es by
app oach.
Table 3. DUC FEMsum manual con en esponsi eness sco e.
Table 4. DUC Con en esponsi eness sco es by app oach.
Table 5. DUC Con en esponsi eness sco es dis ibu ion by app oach
Table 6. DUC ROUGE measu es when conside ing 4 manual summa ies as
e e ences.
Table 7. CHIL ROUGE measu es when conside ed 3 manual summa ies as
e e ences.
Table 8. CHIL esponsi eness conside ing 3 human models when e alua ing
au oma ic summa ies and 2 when e alua ing human summa ies.
Table 9. CHIL esponsi eness sco es dis ibu ion by au oma ic sys em.
Figu e 1. FEMsum Global A chi ec u e
Figu e 2. Con en Ex ac o modul
Figu e 3. Linguis ic P ocesso cons i uen s
Figu e 4. Sample o a sen ence analysis and he co esponden en i onmen
ep eseen a ion
Figu e 5. Sample o he g aph ep esen a ion o an En i onmen
16

Table 1
Sample o en i onmen buil om a sen ence
”Romano P odi 1is 2 he 3p ime 4minis e 5o 6I aly 7”
i en p ope pe son(1), en i y has quali y(2), en i y(5), i en coun y(7),
quali y(4), which en i y(2,1), which quali y(2,5), mod(5,7), mod(5,4)
Table 2
DUC FEMsum manually e alua ion o linguis ic quali y sco es by app oach.
G amma icali y Non- edundancy Re e en ial cla i y Focus
FEMsum mean FEMsum mean FEMsum mean FEMsum mean
LEX 3,14 3,45 2,43 4,02 2,43 2,83 3,29 3,73
S150 3,00 3,60 4,15 4,19 3,08 3,09 3,77 3,84
S250 3,33 3,59 4,23 4,27 2,77 3,12 3,20 3,42
Table 3
DUC FEMsum manual con en esponsi eness sco e.
Sys em(ID) Sco e Mean Dis ance
Human(A-J) 4,75 2,19
Bes (27) 3,08 1,83
FEMsum(19) 2,60 0,04
+www(30) 2,58 0,02
Baseline(1) 2,04 -0,52
Mean(2-35) 2,56 S de 0,28
Table 4
DUC Con en esponsi eness sco es by app oach.
Mean(1-35) FEMsum Mean Dis ance
LEX 2,36 2,29 -0,07
SEM150 2,55 2,92 0,37
SEM250 2,58 2,53 -0,05
Table 5
DUC Con en esponsi eness sco es dis ibu ion by app oach
1: Ve y Poo 2: Poo 3: Accep able 4: Good 5: Ve y Good
LEX 14% 43% 43% 0% 0%
SEM150 7,5% 31% 31% 23% 7,5%
SEM250 6,7% 50% 26,7% 16,7% 0%
17
Table 6
DUC ROUGE measu es when conside ing 4 manual summa ies as e e ences.
Bes (24) FEMsum(19) +www(30) Baseline(1)
R-2 0,095 0,076 0,067 0,050
R-SU4 0,155 0,131 0,122 0,098
Table 7
CHIL ROUGE measu es when conside ed 3 manual summa ies as e e ences.
SDS LEX LEXnoT +www SEM
ROUGE-1 0,293 0,309 0,312 0,333 0,323
ROUGE-2 0,060 0,092 0,102 0,089 0,073
ROUGE-3 0,029 0,056 0,064 0,052 0,032
ROUGE-4 0,019 0,043 0,050 0,043 0,021
ROUGE-L 0,256 0,272 0,279 0,289 0,280
ROUGE-W1.2 0,089 0,098 0,100 0,104 0,098
ROUGE-S1 0,057 0,088 0,097 0,087 0,067
ROUGE-S4 0,064 0,089 0,095 0,094 0,073
ROUGE-S9 0,069 0,095 0,102 0,103 0,083
ROUGE-SU1 0,136 0,162 0,169 0,168 0,152
ROUGE-SU4 0,102 0,126 0,132 0,134 0,115
ROUGE-SU9 0,090 0,116 0,122 0,124 0,105
Table 8
CHIL esponsi eness conside ing 3 human models when e alua ing au oma ic sum-
ma ies and 2 when e alua ing human summa ies.
M1 M2 M3 SDS LEX LEXnoT +www SEM
3,625 3,400 3,375 1,250 1,775 2,025 1,800 1,800
Table 9
CHIL esponsi eness sco es dis ibu ion by au oma ic sys em.
M1 M2 M3 SDS LEX LEXnoT +www SEM
1: Ve y Poo 0% 0% 0% 70% 40% 15% 30% 35%
2: Poo 10% 5% 10% 25% 25% 50% 50% 40%
3: Accep able 20% 35% 30% 5% 35% 30% 15% 20%
4: Good 40% 40% 45% 0% 0% 5% 5% 5%
5: Ve y Good 30% 20% 15% 0% 0% 0% 0% 0%
18
pape s
slides
no es
aw
ex
Que y
Con en
Ex ac o
Summa y
Summa y
Compose
ASR
ansc ip
Use Need
OUTPUT Se ings
CONTENT
DOC_RESTRICT
SIZE
FORMAT
No ansc ip s
Tex
Num. wo ds
Keywo ds
Gene ic
Speech/ ideo
In o ma ion
Rele an
De ec o
NLs a emen
All
Sin hesizedTex
MDS
Independen
P esen a ion
Tex
English
No used
Used
SDS
Scien i ic
DOC_STRUCTURE
LANGUAGE
MEDIA
UNIT
Spanish
GENRE
Jou nalis ic
INPUT Se ings
DOMAIN
Independen
Scien i ic
News
Fig. 1. FEMsum Global A chi ec u e
Linguis ic
P ocesso
sen
en
sin
Tex Uni s Selec o
Candida es Rele an
Tex Uni s
sim
CONTENT EXTRACTOR
Candida es
Simila i y Ma ix
Gene a o
Fig. 2. Con en Ex ac o modul
Tokenize POS
Tagge Lemma ize NERC Syn ac ic
Chunke
Seman ic
Tagge Anno a o
DM Seman ic
Analize
Discou se
Ma ke s
Wo dNe
Use
Need
Tex
Uni s
sen
sin
en
LINGUISTIC PROCESSOR
Fig. 3. Linguis ic P ocesso cons i uen s
19
Fig. 4. Sample o a sen ence analysis and he co esponden en i onmen ep eseen-
a ion
Romano P odi
1
is
2
p ime
4
minis e
57
I aly
0
which_en i y which_quali y
mod mod
en i y_has_quali y
i_en_p ope _pe son
quali y
en i y
i_en_coun y
Fig. 5. Sample o he g aph ep esen a ion o an En i onmen
20