The UPC RDF- o-Tex Sys em a WebNLG Challenge 2020
Da id Be g´
es, Rose Can enys, Roge C eus, O iol Domingo, Jos´
e A. R. Fonollosa
Uni e si a Poli `
ecnica de Ca alunya, Ba celona
TALP Resea ch Cen e
{da id.be ges, ose .can enys
oge .c eus, o iol.domingo. oig}@es udian a .upc.edu
{jose. onollosa}@upc.edu
Abs ac
This wo k desc ibes he end- o-end sys em a -
chi ec u e p esen ed a WebNLG Challenge
2020. The sys em ollows he adi ional Ma-
chine T ansla ion (MT) pipeline, based on he
T ans o me model, applied in mos ex - o-
ex p oblems. Ou solu ion is en iched by
means o a Back T ansla ion s ep o e he o ig-
inal co pus. Thus, he sys em di ec ly elies on
lexicalise o ma since he syn he ic da a limi s
he use o delexicalisa ion.
1 In oduc ion
Na u al Language Gene a ion (NLG) can be di-
ided in o: ex - o- ex gene a ion o da a- o- ex
gene a ion, acco ding o Ga and K ahme (2017).
The WebNLG Challenge 2020 consis s in mapping
da a- o- ex . Mo e speci ically, he da a is a se
o Resou ce Desc ip ion F amewo k (RDF) iples
ex ac ed om DBpedia and he co esponding ex
is a e balisa ion o hese iples 1.
The in o ma ion s uc u e is based on RDF,
which consis o h ee elemen s:
h
subjec , p ed-
ica e, objec i. Thus, i es ablishes ela ions (p edi-
ca e) be ween en i ies (subjec , objec ). This can be
app ecia ed in Figu e 1. Howe e , his in o ma ion
s uc u e is no easy eadable nei he unde s and-
able, hence, i is ha d o people o comp ehend he
meaning o such da a.
The e has been a lo o p e ious wo k in he
NLG domain (Rei e and Dale,2000) in he pas
wo decades. Bon che a e al. (2004) wo k in
he medical domain, whe e hey use a adi ional
NLG app oach o gene a e sen ences om RDF
da a il e ing epe i i e RDF, and hen g oup cohe -
en iples agg ega ing he gene a ed sen ences in
o de o p oduce he inal ones. (Cimiano e al.,
2013) gene a e cooking ecipes om seman ic web
1h ps://webnlg-challenge.lo ia. /
challenge_2020/
da a, using a la ge co pus o ex ac lexicon in he
cooking domain, which is hen used in conjunc ion
wi h a adi ional NLG app oach o gene a e cook-
ing eceip s. (Duma and Klein,2013) use a me hod
which wo ks well on RDF iples in a seen domain
bu ails wi h unseen ones. Thei aim is o lea n
sen ence empla es om pa allel RDF da a and
ex co po a by means o aligning en i ies in RDF
iples wi h en i ies men ioned in sen ences, and
hen ex ac ing hese empla es om he aligned
sen ences by eplacing he en i y men- ion wi h a
unique oken.
We decided o only pa icipa e in he English
e sion o he RDF- o-Tex challenge (Cas o Fe -
ei a e al.,2020). We used a model based
on he T ans o me encoded-decode a chi ec u e
(Vaswani e al.,2017). Mo eo e , inspi ed by p e i-
ous wo k in he MT ield, we enla ged he o iginal
co pus by means o Back T ansla ion (BT) (Sen-
n ich e al.,2016).
The es o he documen is o ganised as ollows.
Fi s , in Sec ion 2we ake a deepe di e in o he
ask o mula ion. Nex , in Sec ion 3 he p ep o-
cessing plan is explained. Then, in Sec ion 4we
depic he T ans o me model a chi ec u e adap ed
o ou p oblem. The ea e , we b ie ly desc ibe
pos p ocessing in Sec ion 5. Finally, in Sec ion
6we summa ize he implemen a ion o BT o e
he o iginal challenge ollowed by b ie esul s and
conclusions in Sec ions 7and 8 espec i ely.
2 Task Fo mula ion
The goal o he RDF- o-Tex ask is o gene a e ex
om a se o iples, which a e wo ds es ablishing
ela ions be ween hem.
The inpu o ou sys em is da a in he o m o
iples ha can be deno ed as a se o RDF, i.e.
K:= { 1, ..., n}
. Each RDF
i
can be de ined as
hsi, pi, oii
, hese elemen s s and o subjec , p ed-
3 d In e na ional Wo kshop on Na u al Language Gene a ion om he Seman ic Web (WebNLG+),
Dublin, I eland (Vi ual), 18 Decembe 2020, pages 167–170, c
2020 Associa ion o Compu a ional Linguis ics
A ibu ion 4.0 In e na ional.
Figu e 1: Example o a knowledge g aph (a) wi h i s co esponding RDF iples (b) and i s na u al language
desc ip ion (c).
ica e and objec , espec i ely. No ice ha each
elemen can con ain mo e han one wo d, he e
is no p io es ic ion in ha sense. Fo ins ance,
he subjec ’Ba ack Obama’ would be encoded as
si= [Ba ack, Obama] = sij = [si1, si2]
, so
i
indica es he RDF in
K
and index
j
deno es he
wo d posi ion in each subjec , p edica e o objec
elemen .
Finally, we aim o gene a e a discou se
S
, which
consis s o a sequence o wo ds
[w1, ..., wm]
. The
esul ing discou se in
S
should be g amma ically
co ec and should also con ain all he in o ma ion
p esen in he iples.
3 P ep ocessing
In his sec ion we desc ibe he i s s eps pe o med
on da a. The e is a common s ep, delexicalisa ion,
ha has been pe o med since he e y beginning
o his challenge, back o 2017
2
. We decided o
a oid his s ep due o he implemen a ion o ou BT
me hod ha does no con ain he equi ed mapping:
om indi idual en i ies o gene ic wo ds.
The e y i s da a p ocessing guide is de ined
as ollows, and i is exempli ied in Table 1. Fi s o
all, we linea ise he RDF inpu and spli he camel-
Case no a ion. Then, Moses Tokenize (Koehn
e al.,2007) is applied o sepa a e punc ua ion om
wo ds, p ese ing special okens such as da es, and
no malize cha ac e s. Finally, By e Pai Encoding
(BPE) (Senn ich e al.,2015) is applied o enable
he model o be mo e obus o unseen da a. This is
a adi ional echnique ha inc eases he ansla ion
quali y o models. BPE is lea ned wi h he ain-
ing plus alida ion p ocedu e and is used o he
sou ce and a ge ocabula y. This way, he model
is ained o bo h ecei ing and p edic ing BPE
encoded ocabula y, also o he es se . Finally,
2h ps://webnlg-challenge.lo ia. /
challenge_2017/
he sys em implemen a ion allows o lea n embed-
dings om sc a ch om he ocabula y ha has
been al eady encoded wi h BPE.
4 The T ans o me Model
The T ans o me (Vaswani e al.,2017) is consid-
e ed a s a e-o - he-a encode -decode a chi ec u e,
wi h g ea success in a as ield o applica ions
such as MT. Following his success and aking ad-
an age o i s simple a chi ec u e, we p oposed a
simple ans o me app oach ained in an end- o-
end ashion. One o he main ai s ha enables
hese models o a ain such su p ising esul s, is
he a en ion mechanism, ha allows o model de-
pendencies ega dless hei dis ance in he inpu o
ou pu sequences. This capabili y is a undamen al
ea u e o RDF- o-Tex , as au oma ed gene a ion
o ex akes in o accoun he ela ionship be ween
wo ds ha may no appea consecu i ely.
4.1 Model Pa ame e s and Op imiza ion
Fo he model’s a chi ec u e, we used a o al o 3
laye s wi h 1,024-dimensional Feed Fo wa d Ne -
wo ks (FFN) and 8 a en ion heads, pe o ming
c oss + sel a en ion a each laye . We used 256-
dimensional embeddings wi h ixed sinusoidal posi-
ional encodings, sha ed ac oss he en i e ne wo k.
We used he Adam op imize wi h
b1=0.9
,
b2=0.98
and
= 10 −9
. We inc eased he
lea ning a e linea ly o a o al o 4,000 wa ming
s eps o 1e-03, and dec eased i ollowing an in-
e se squa e oo o mula om he eon. Addi ion-
ally, we applied se e al d opou echniques such as
d opou , g adien clipping and label smoo hing o
ou loss o mula.
Wi h his model con igu a ion, he pe o med
expe imen s concluded ha he bes choice was o
use 7,000 subwo ds o he BPE encoding.
168
RDF Inpu hBaku Tu kish Ma y s ’ Memo ial, na i eName, ” T¨
u k S¸ehi le i Anı ı ” i
hBaku Tu kish Ma y s ’ Memo ial, loca ion, Aze baijan i
Linea ise Baku Tu kish Ma y s ’ Memo ial na i eName ” T¨
u k S¸ehi le i Anı ı ” Baku Tu kish Ma y s ’ Memo ial loca ion Aze baijan
camelCase Remo al Baku Tu kish Ma y s ’ Memo ial na i e Name ” T¨
u k S¸ehi le i Anı ı ” Baku Tu kish Ma y s ’ Memo ial loca ion Aze baijan
BPE &Tokeniza ion Baku Tu kish Ma @@ y s ’ Memo ial na i e Name ” T@@ ¨
u k S¸@@ eh@@ i @@ l@@ e @@ i An@@ ı@@ @@ ı ” Baku
Tu kish Ma @@ y s ’ Memo ial loca ion Aze baijan
T ans o me The Baku Tu kish Ma @@ y s ’ Memo ial is loca ed in Aze baijan . The na i e name o he Baku Tu kish Ma @@ y s ’ ” T@@
¨
u k S¸@@ eh@@ i @@ l@@ e @@ i An@@ ı@@ @@ ı .
Sys em Ou pu The Baku Tu kish Ma y s ’ Memo ial is loca ed in Aze baijan . The na i e name o he Baku Tu kish Ma y s ’ Memo ial is Tu k
Sehi le i Anı ı .
Table 1: Exempli ica ion o each s ep in he sys em a chi ec u e using a es ins ance.
5 Pos p ocessing
The T ans o me model ou pu s a sequence o p e-
dic ed wo ds, hen, he sys em emo es he ok-
eniza ion as well as BPE. One example o he ou -
pu o he sys em is shown in Table 1.
6 Back T ansla ion
BT uns in a semi-supe ised en i onmen whe e
bo h pa allel co po a and monolingual da a in he
a ge language a e a ailable (Senn ich e al.,2015).
Fi s , BT ains an in e media e sys em on he pa -
allel da a which is used o ansla e he a ge mono-
lingual da a in o he sou ce language, i.e. ex - o-
RDF. The la e , esul s in a pa allel co pus whe e
he sou ce is syn he ic MT ou pu while he a ge
is genuine ex w i en by humans. A e wa ds, he
gene a ed syn he ic pa allel co pus is added o he
eal bi ex in o de o ain a inal model ha will
ansla e om he sou ce o he a ge language,
equi alen ly RDF- o- ex .
The pa allel da ase was al eady p o ided by he
challenge and con ains he ansla ion om RDF-
o- ex and ice- e sa. Hence, we jus needed an
ex e nal monolingual co pus o he a ge language
o pe o m augmen a ion o he sou ce da a. In
o de o do so, we implemen ed a dis ance-based
app oach o he aining da a since en i ies appea -
ing in he co pus we e anno a ed. Taking his in o
accoun , no only was he eam capable o sc ap-
ping Wikipedia pages o mos simila en i ies o he
ones in he o iginal co pus, bu we we e also able o
limi sc apping o he i s h ee pa ag aph in each
page wi hou loss o quali y. In o de o de e mine
he mos simila en i ies, an embedding dis ance
was compu ed ega ding Wikipedia2Vec (Yamada
e al.,2020) ha allows o que y o en i ies a he
han wo ds.
The cu en app oach o sol e he Back-
T ansla ion, ex - o-RDF, implies using pa sing
ees ha gua an ee ha elemen s in he RDF ap-
pea in he ex . Consequen ly, his implemen a ion
gene a ed cohe en da a wi h espec o ex . The
inal da ase , which in eg a ed he eal co pus and
he syn he ic one, has a ound 340,000 ins ances.
A de ailed desc ip ion o his expe imen can be
ound in (Domingo e al.,2020).
7 Resul s
In Table 2, we show he esul s ob ained in he es
se o he compe i ion. One ema kable aspec is
ha he e is no a signi ican di e ence be ween he
pe o mance in he seen and unseen domain ega d-
ing he METEOR, TER and ch F++ me ic. On he
o he hand, he e exis s a pe o mance d op based
on BLEU sco e in he unseen da a wi h espec o
he seen one.
Da a BLEU (↑)METEOR (↑)ch F++ (↑)TER (↓)
All 39.12 0.33 0.57 0.56
Seen Ca ego ies 51.85 0.37 0.64 0.47
Unseen Ca ego ies 29.46 0.31 0.52 0.60
Unseen En i ies 35.34 0.32 0.56 0.57
Table 2: Pe o mance o he sys em ega ding di e en
da a pa i ions in he es se .
8 Open discussion
An in e es ing capabili y ha was no implemen ed
would ha e been o ob ain delexicalised syn he ic
da a, so he model lea ned mo e gene ic ep esen-
a ions. Fu he mo e, i would be in e es ing o
enla ge he syn he ic co pus and use his syn he ic
co pus o ain mo e ele an models in he RDF-
o-Tex domain.
Acknowledgmen
We wan o hank anonymous e iewe s o hei
commen s on he pape . This wo k was suppo ed
by he p ojec ADAVOICE, PID2019-107579RB-
I00 / AEI / 10.13039/501100011033.
169
Re e ences
Kalina Bon che a and Yo ick Wilks. 2004. Au oma ic
epo gene a ion om on ologies: The miak ap-
p oach. In Na u al Language P ocessing and In o -
ma ion Sys ems, pages 324–335, Be lin, Heidelbe g.
Sp inge Be lin Heidelbe g.
Thiago Cas o Fe ei a, Clai e Ga den , Ch is an de
Lee, Nikolai Ilinykh, Simon Mille, Diego Mous-
salem, and Anas asia Shimo ina. 2020. The
2020 bilingual, bi-di ec ional webnlg+ sha ed ask
o e iew and e alua ion esul s (webnlg+ 2020). In
P oceedings o he 3 d WebNLG Wo kshop on Na -
u al Language Gene a ion om he Seman ic Web
(WebNLG+ 2020), Dublin, I eland (Vi ual). Associ-
a ion o Compu a ional Linguis ics.
Philipp Cimiano, Janna L¨
uke , Da id Nagel, and
Ch is ina Unge . 2013. Exploi ing on ology lex-
ica o gene a ing na u al language ex s om RDF
da a. In P oceedings o he 14 h Eu opean Wo k-
shop on Na u al Language Gene a ion, pages 10–
19, So ia, Bulga ia. Associa ion o Compu a ional
Linguis ics.
O iol Domingo, Da id Be g´
es, Rose Can enys, Roge
C eus, and Jos´
e A.R. Fonollosa. 2020. Enhancing
sequence- o-sequence modelling o RDF iples o
na u al ex . In P oceedings o he 3 d WebNLG
Wo kshop on Na u al Language Gene a ion om he
Seman ic Web (WebNLG+ 2020), Dublin, I eland
(Vi ual). ”Associa ion o Compu a ional Linguis-
ics”.
Daniel Duma and Ewan Klein. 2013. Gene a ing na -
u al language om linked da a: Unsupe ised em-
pla e ex ac ion. In P oceedings o he 10 h In e -
na ional Con e ence on Compu a ional Seman ics
(IWCS 2013) – Long Pape s, pages 83–94, Po sdam,
Ge many. Associa ion o Compu a ional Linguis-
ics.
Albe Ga and Emiel K ahme . 2017. Su ey o
he s a e o he a in na u al language gene a ion:
Co e asks, applica ions and e alua ion.CoRR,
abs/1703.09902.
Philipp Koehn, Hieu Hoang, Alexand a Bi ch, Ch is
Callison-Bu ch, Ma cello Fede ico, Nicola Be oldi,
B ooke Cowan, Wade Shen, Ch is ine Mo an,
Richa d Zens, Ch is Dye , Ondˇ
ej Boja , Alexand a
Cons an in, and E an He bs . 2007. Moses: Open
sou ce oolki o s a is ical machine ansla ion. In
P oceedings o he 45 h Annual Mee ing o he As-
socia ion o Compu a ional Linguis ics Companion
Volume P oceedings o he Demo and Pos e Ses-
sions, pages 177–180, P ague, Czech Republic. As-
socia ion o Compu a ional Linguis ics.
Ehud Rei e and Robe Dale. 2000. Building Na u al
Language Gene a ion Sys ems. S udies in Na u al
Language P ocessing. Camb idge Uni e si y P ess.
Rico Senn ich, Ba y Haddow, and Alexand a Bi ch.
2015. Neu al machine ansla ion o a e wo ds wi h
subwo d uni s.CoRR, abs/1508.07909.
Rico Senn ich, Ba y Haddow, and Alexand a Bi ch.
2016. Imp o ing neu al machine ansla ion mod-
els wi h monolingual da a. In P oceedings o he
54 h Annual Mee ing o he Associa ion o Compu-
a ional Linguis ics (Volume 1: Long Pape s), pages
86–96, Be lin, Ge many. Associa ion o Compu a-
ional Linguis ics.
Ashish Vaswani, Noam Shazee , Niki Pa ma , Jakob
Uszko ei , Llion Jones, Aidan N. Gomez, Lukasz
Kaise , and Illia Polosukhin. 2017. A en ion is all
you need.CoRR, abs/1706.03762.
Ikuya Yamada, Aka i Asai, Jin Sakuma, Hi oyuki
Shindo, Hideaki Takeda, Yoshiyasu Take uji, and
Yuji Ma sumo o. 2020. Wikipedia2 ec: An e icien
oolki o lea ning and isualizing he embeddings
o wo ds and en i ies om wikipedia. a Xi p ep in
1812.06280 3.
170