A deep lea ning app oach o
au oma ically gene a ing desc ip ions o
images con aining people
T abajo de Fin de G ado
Cu so 2017–2018
Au o
Ma a A acil Mu˜noz
Di ec o es
Gonzalo M´endez Pozo
Raquel He ´as Balles e os
G ado en Ingenie ´ıa In o m´a ica
Facul ad de In o m´a ica
Uni e sidad Complu ense de Mad id
A deep lea ning app oach o
au oma ically gene a ing desc ip ions o
images con aining people
T abajo de Fin de G ado en Ingenie ´ıa In o m´a ica
Depa amen o de Ingenie ´ıa del So wa e e In eligencia A i icial
Au o
Ma a A acil Mu˜noz
Di ec o es
Gonzalo M´endez Pozo
Raquel He ´as Balles e os
Con oca o ia: Sep iemb e 2018
G ado en Ingenie ´ıa In o m´a ica
Facul ad de In o m´a ica
Uni e sidad Complu ense de Mad id
Abs ac
Gene a ing image desc ip ions is a challenging A i icial In elligence p oblem wi h many
in e es ing applica ions such as obo s’ communica ion o helping isually impai ed people.
Howe e , i is a complex ask o compu e s: i equi es Compu e Vision algo i hms, o
unde s and wha he image depic s, and Na u al Language P ocessing algo i hms, o
gene a e a well- o med sen ence. Nowadays, deep neu al ne wo ks a e he s a e-o - he-a
in hese wo A i icial In elligence ields.
Fu he mo e, we belie e ha images ha con ain people a e desc ibed in a sligh ly
di e en manne and ha es ic ing an image desc ip ion gene a o model o hese images
may p oduce be e desc ip ions. The e o e, he main objec i e o his p ojec is o de elop
a Deep Lea ning model ha au oma ically p oduces desc ip ions o images con aining
people and o conclude i i is a good p ac ice he es ic ion o his kind o images. Fo
his pu pose, we ha e e iewed and s udied he li e a u e in he ield and we ha e buil ,
ained and compa ed ou di e en models using Deep Lea ning echniques and a GPU
o speed-up he compu a ion, as well as a big and comple e da ase .
Keywo ds
Deep Lea ning, Compu e Vision, Na u al Language P ocessing, image desc ip ion gene -
a ion, Ke as, GPU, da ase .

Resumen
Gene a desc ipciones de im´agenes es un p oblema de In eligencia A i icial con muchas
aplicaciones in e esan es como la comunicaci´on de obo s o ayuda a pe sonas con dis-
capacidad isual. Sin emba go, es una a ea compleja pa a un o denado : equie e algo-
i mos de isi´on po compu ado pa a en ende lo que la imagen ep esen a y algo i mos
de p ocesamien o de lenguaje na u al pa a gene a una ase bien o mada. Hoy en d´ıa,
las edes neu onales p o undas son el es ado del a e en es os dos campos de la In eligencia
A i icial.
Po o a pa e, c eemos que las im´agenes que con ienen pe sonas se desc iben de man-
e a lige amen e di e en e y que es ingi un modelo de gene aci´on de desc ipciones de
im´agenes a im´agenes de es e ipo puede p oduci mejo es desc ipciones. Po lo an o, el
p incipal obje i o de es e p oyec o es desa olla un modelo de ap endizaje p o undo que
p oduce au om´a icamen e desc ipciones de im´agenes que con ienen pe sonas y conclui si
es una buena p ´ac ica la es icci´on a es a clase de im´agenes. Pa a ello, hemos e isado y
es udiado la li e a u a y hemos cons uido, en enado y compa ado cua o modelos di e -
en es usando ´ecnicas de ap endizaje p o undo y una GPU pa a acele a los c´alculos, as´ı
como un da ase g ande y comple o.
Palab as cla e
Ap endizaje p o undo, isi´on po compu ado , p ocesamien o de lenguaje na u al, gen-
e aci´on de desc ipciones de im´agenes, Ke as, GPU, da ase .
ii
Con en s
1 In oduc ion and objec i es 1
1.1 Objec i es..................................... 2
1.2 Documen s uc u e ............................... 3
2 In oducci´on y obje i os 5
2.1 Obje i os ..................................... 6
2.2 Es uc u a del documen o . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3 S a e o he a 9
3.1 T adi ional algo i hms o image analysis . . . . . . . . . . . . . . . . . . . 10
3.2 T adi ional app oach o NLG . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.3 MachineLea ning................................. 11
3.4 DeepLea ning .................................. 13
3.4.1 Feed o wa d Neu al Ne wo ks . . . . . . . . . . . . . . . . . . . . . . 14
3.4.2 Con olu ional Neu al Ne wo ks . . . . . . . . . . . . . . . . . . . . . 15
3.4.3 Recu en Neu al Ne wo ks . . . . . . . . . . . . . . . . . . . . . . . 17
3.5 Image desc ip ion gene a o s . . . . . . . . . . . . . . . . . . . . . . . . . . 18
4 F amewo k 21
4.1 GPU........................................ 21
4.2 Deep Lea ning amewo k . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.3 Visualiza ion: Tenso Boa d . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.4 Imageda ase s .................................. 25
4.4.1 Da ase elec ion ............................. 28
ix
4CHAPTER 1. INTRODUCTION AND OBJECTIVES

Chap e 2
In oducci´on y obje i os
En nues a condici´on de humanos, cons an e e inconscien emen e hacemos desc ipciones
del mundo que nos odea pa a comunica nos con la gen e. Desc ibimos lo que emos
en muchas si uaciones di e en es, como po ejemplo cuando damos indicaciones a alguien
sob e c´omo i a alg´un si io (sigue ec o has a que eas el edi icio ojo y despu´es gi a a
la de echa a la calle que iene los ´a boles g andes y una uen e), cuando le con amos una
his o ia a un amigo (es aba de comp as cuando me encon ´e a Sa a de la mano de un se˜no
mayo al o) o cuando subimos una o o a In e ne (¡Pas´andomelo genial en la playa con
mi es ido azul nue o!). Mien as que es as desc ipciones las hacemos de mane a sencilla
y na u al, sin ene que pensa lo demasiado, gene a au om´a icamen e desc ipciones de lo
que hay en una imagen es una a ea muy compleja y exigen e pa a un o denado .
Gene a una buena desc ipci´on de una imagen equie e p ime o analiza y en ende
lo que apa ece en ella y despu´es gene a una ase en lenguaje na u al que explique esos
elemen os, es ´e bien o mada y sea concisa. Po lo an o, gene a desc ipciones de im´agenes
es m´as di ´ıcil que las ´ıpicas a eas del campo de isi´on po compu ado , como son la
clasi icaci´on de im´agenes o la de ecci´on de obje os, ya que conlle a no solo ´ecnicas de
Visi´on po Compu ado (CV), sino ambi´en P ocesamien o del Lenguaje Na u al (NLP);
dos campos de la In eligencia A i icial que po lo gene al han lle ado caminos sepa ados
y no se suelen es udia ni aplica en conjun o.
La gene aci´on de desc ipciones de im´agenes se puede conside a como un caso pa icu-
la de la aducci´on au om´a ica donde, en luga de aduci una ase dada a o o idioma,
hay que aduci una imagen de en ada a su desc ipci´on. La imagen de en ada debe se
codi icada a un ec o de ep esen aci´on y, despu´es, es a ep esen aci´on debe se decodi i-
cada a una ase en lenguaje na u al. Pa a log a es o, podemos combina esul ados del
es ado del a e de ambos campos: los esul ados de Visi´on po Compu ado pa a ob ene
el ec o de ep esen aci´on y los de NLP (en pa icula , el es ado del a e de la Gene aci´on
de Lenguaje Na u al (NLG)) pa a cons ui la desc ipci´on.
Es e p oblema an in e esan e de In eligencia A i icial ha despe ado el in e ´es de
g andes emp esas como Facebook o Google. El p oblema de desc ibi im´agenes au-
5
6CHAPTER 2. INTRODUCCI ´
ON Y OBJETIVOS
om´a icamen e no solo es a ac i o po su in e ´es acad´emico y cien ´ı ico, sino ambi´en
po sus muchas aplicaciones. La p incipal y m´as impo an e aplicaci´on es gene a des-
c ipciones pa a las pe sonas con alguna discapacidad isual, bien sea pa a ayuda les en
su d´ıa a d´ıa (po ejemplo, pa a mo e se po la ciudad o comp a en un supe me cado),
pa a ace ca les a la cul u a gene ando desc ipciones de los cuad os de un museo o, como
i imos en un mundo cla amen e dominado po o os, pa a hace m´as accesibles las edes
sociales, y las p´aginas de In e ne en gene al. Pe o ambi´en iene o as aplicaciones como
po ejemplo en ob´o ica pa a la comunicaci´on de los obo s ( ans o mando en ases lo
que en e iden i icando con qui´en es ´an hablando) o en medicina pa a desc ibi im´agenes
m´edicas.
Po o o lado, en los ´ul imos a˜nos hemos podido p esencia un inc e´ıble aumen o del
in e ´es en ob ene in o maci´on de los da os y es amos ac ualmen e en lo que llaman la
e a da a-d i en (di igida po los da os). Cada d´ıa, se c ean millones de nue os da os,
muchas emp esas ienen ecogida una g an can idad de da os y quie en saca les p o echo,
descub iendo pa ones ocul os y pe spec i as pa a ob ene buenas decisiones de nego-
cio. Debido a es a g an can idad de da os disponibles y al c ecimien o de la capacidad
compu acional pa a ealiza c´alculos complejos, las ´ecnicas de ap endizaje au om´a ico y
ap endizaje p o undo se han podido desa olla y es udia con i i´endose en el es ado del
a e en muchos campos, lo que ha hecho que es ´en en boca de odos.
Po lo an o, es e abajo busca auna es os dos concep os an popula es y a ac i os,
desa ollando un gene ado de desc ipciones de im´agenes u ilizando ´ecnicas de ap endizaje
p o undo.
2.1 Obje i os
Respec o a las desc ipciones de im´agenes, Be na di e al. [Be na di e al., 2016] dis inguen
dos en oques p incipales pa a su gene aci´on: modelos que gene an desc ipciones nue as
pa a una imagen de en ada y modelos que cons uyen la desc ipci´on ellenando plan illas o
usando desc ipciones asociadas a im´agenes simila es; noso os cen a emos nues o abajo
en los p ime os.
Vamos a limi a el alcance de es e abajo a gene a desc ipciones de im´agenes de
pe sonas, ya que conside amos que es un en oque in e esan e y pod emos cen a odos
los es ue zos en ello. El abajo elacionado en el campo de la gene aci´on au om´a ica
de desc ipciones de im´agenes no se suele cen a en un ema en conc e o, a ando de
aba ca odas las posibles desc ipciones e im´agenes y no le han dado demasiada a enci´on
a las im´agenes de pe sonas. Conside amos que no desc ibimos de la misma mane a a una
pe sona que a o os elemen os de una imagen, y po an o, al es ingi nos a im´agenes de
pe sonas podemos ob ene mejo es desc ipciones y esul ados.
Con odo es o, los obje i os p incipales de es e abajo son:
•Re isa y analiza la bibliog a ´ıa y el es ado del a e de las ´ecnicas de gene aci´on
2.2. ESTRUCTURA DEL DOCUMENTO 7
de desc ipciones de im´agenes.
•Es udia y en ende los modelos de ap endizaje p o undo y sus aplicaciones.
•Es udia los di e en es en o nos de abajo y he amien as en ocadas al ap endizaje
p o undo.
•Desa olla di e en es modelos que gene en au om´a icamen e desc ipciones de im´a-
genes de pe sonas y a a de ep oduci los esul ados del es ado del a e.
•E alua y compa a los dis in os modelos c eados y de e mina cu´al es el mejo y si
es sa is ac o io cen a los es ue zos y limi a nues os modelos a ca ac e ´ıs icas de
pe sonas.
2.2 Es uc u a del documen o
La es uc u a de es e documen o se ´a la siguien e: el Cap´ı ulo 3 da una explicaci´on muy
de allada del es ado del a e an o en el campo de la Gene aci´on de Lenguaje Na u al como
en el de Visi´on po Compu ado , en es e cap´ı ulo ambi´en se de allan algunos gene ado es
de desc ipciones de im´agenes. En el Cap´ı ulo 4 in oducimos el en o no de abajo en el
que desa olla emos es e abajo; en pa icula , explicamos las GPUs, el en o no cloud
compu ing, qu´e lib e ´ıas pa a ap endizaje p o undo y isualizaci´on usa emos y la elecci´on
del da ase .
El Cap´ı ulo 5 es el p incipal de es e abajo; a paso a paso po el p oceso de de inici´on
y en enamien o de los modelos. Explicamos en es e cap´ı ulo odas las ca ac e ´ıs icas y
componen es de los modelos y odas las decisiones omadas al de ini los y en ena los. El
Cap´ı ulo 6 es ´a dedicado a mos a los esul ados ob enidos, el p oceso de es ing y las
m´e icas m´as conocidas. Po ´ul imo, en el Cap´ı ulo 8 p esen amos las conclusiones del
abajo y damos posibles l´ıneas de abajo u u o.
8CHAPTER 2. INTRODUCCI ´
ON Y OBJETIVOS
Chap e 3
S a e o he a
Image analysis has been a challenging p oblem in esea che s’ minds o a long ime now.
A i s , his p oblem was ackled by a adi ional compu e ision app oach, wi h explici
p ocessing algo i hms. Then, Machine Lea ning models appea ed and hey we e he s a e-
o - he-a in his ield un il 2010, when aining complex deep neu al ne wo ks s a ed o
become a possibili y hanks o he use o GPUs compu a ional powe and he a ailabili y
o mo e da a. As o oday, hese neu al ne wo ks, in pa icula Con olu ional Neu al
Ne wo ks, a e he s a e-o - he-a in all image analysis challenges.
Simila ly, he ask o Na u al Language Gene a ion (NLG), has been aced by classical
s ep-by-s ep algo i hms, as well as Machine Lea ning and Deep Lea ning models. In
pa icula , Recu en Neu al Ne wo ks a e, as o oday, he s a e-o - he-a in gene a ing
na u al language sen ences and ex s.
Image desc ip ion combines bo h compu e ision and na u al language p ocessing
ields, and, nowadays, he s a e-o - he-a esul s a e a combina ion o Con olu ional Neu-
al Ne wo ks, ha ocus on ex ac ing image ea u es, and Recu en Neu al Ne wo ks, o
gene a e he sen ence.
In his chap e we will p esen p e ious wo k in he ield o image analysis and Na u al
Language Gene a ion, and how hese ields ha e been add essed ac oss ime, om he poin
o iew o adi ional and explici algo i hms o he one o Deep Lea ning models. We
s a wi h a b ie in oduc ion o adi ional algo i hms o compu e ision and language
gene a ion, we hen explain he main cha ac e is ics o Machine Lea ning and we con inue
wi h Deep Lea ning. In he las sec ion o his chap e , we discuss some well-known image
cap ion gene a o s.
9

10 CHAPTER 3. STATE OF THE ART
3.1 T adi ional algo i hms o image analysis
Image analysis consis s o he ex ac ion o impo an cha ac e is ics and ea u es o images
by digi al image p ocessing echniques, and i has many di e en ields o applica ion, such
as obo ics, secu i y, medicine o biology.
T adi ional compu e ision algo i hms, in gene al, wo k by ex ac ing ea u e ec o s
om images and using hem o classi y images. The e a e algo i hms esponsible o a
pa icula ask (noise educ ion, image segmen a ion, co ne de ec ion, edge de ec ion...),
ha hen wo k oge he wi h o he speci ic algo i hms o ca y ou he whole image
p ocessing ask. The use o hese echniques o ex ac use ul s uc u al in o ma ion
om images, such as edges, co ne s o colo s, educes signi ican ly he amoun o da a o
be p ocessed, as i il e s ou non- ele an da a o ocus only in he use ul in o ma ion
ex ac ed.
The main ea u e de ec ion algo i hms in compu e ision a e:
•Canny edge de ec o [Canny, 1986]. An edge is a sudden change in image b igh ness,
ha is, a poin whe e he image b igh ness has discon inui ies. These discon inui ies
usually co espond o changes in su ace o ien a ion, dep h, ma e ial p ope ies o
scene illumina ion. The e o e, an edge de ec o may help o iden i y he bounda ies
o objec s and su ace ma kings, as well as a ia ions in su ace o ien a ion. The
Canny edge de ec o is one o he bes edge de ec o algo i hms, and can de ec a
wide a ie y o edges in an image. I i s smoo hs he image by applying a Gaussian
il e , hen inds in ensi y g adien s and selec s he po en ial edges.
•Ha is co ne de ec o [Ha is and S ephens, 1988]. A co ne is in e p e ed as he
in e sec ion o wo edges. I is a e y impo an image ea u e as i is a poin in a ian
o ansla ion, o a ion and illumina ion. The Ha is co ne de ec o algo i hm
ocuses on he de ec ion o co ne s in an image, educing he dimensionali y o da a
o be p ocessed and i is used in many compu e ision applica ions such as mo ion
acking o s e eo ision (ex ac ing 3D cha ac e is ics om images). I has been
p o ed o be one o he mos accu a e algo i hms in dis inguishing be ween edges
and co ne s.
•SIFT (Scale-In a ian Fea u e T ans o m) [Lowe, 1999]. I is a ea u e desc ip ion
algo i hm used o de ec and desc ibe ea u es in images. SIFT ea u es a e in a ian
o image loca ion, scale and o a ion. A desc ip o ec o is compu ed o each one
o he poin s o in e es in he image, so ha hey a e also obus o changes in noise
and illumina ion. This algo i hm is use ul o many compu e ision applica ions,
such as objec ecogni ion, mo ion acking, na iga ion o 3D modeling.
•SURF (Speeded-Up Robus Fea u es) [Bay e al., 2006]. I is also a ea u e ex ac o
and desc ip o . This algo i hm is a speeded-up e sion o SIFT.
These objec de ec ion algo i hms usually come a e a p ocess o image segmen a ion,
3.2. TRADITIONAL APPROACH TO NLG 11
ha is, di iding he image in o i s cons i uen s pa s, and a e hen combined wi h adi-
ional Machine Lea ning algo i hms (SVM o K-Nea es neighbo ) o image classi ica ion.
3.2 T adi ional app oach o NLG
Na u al Language Gene a ion (NLG), in ol es p oducing unde s andable na u al language
ex s om some inpu da a. I has many di e en in e es ing applica ions, besides gen-
e a ing image desc ip ions, such as machine ansla ion, obo ics o ex ual summa ies o
ad anced da abases (e.g., inancial o wea he o ecas s da ase s) [Goldbe g e al., 1994,
Io danskaja e al., 1992, Wu e al., 2016].
T adi ional NLG algo i hms a e based on he ield o o mal language heo y and logic
ules on which we can build he language. As Rei e and Dale [Rei e and Dale, 1997]
s a e, adi ional algo i hms in NLG usually ollow hese s eps:
1. Con en de e mina ion: deciding which in o ma ion should be included in he ou -
pu . This p ocess c ea es a se o messages om he inpu da a, and i is ypically
done by il e ing and summa izing he da a and de ining he messages in some o mal
language, usually applica ion-dependen .
2. Documen s uc u ing: o ganizing and s uc u ing he se o messages so ha he
inal ex makes sense. The ou pu o his s ep is usually ep esen ed as a ee
s uc u e.
3. Agg ega ion: me ging ela ed messages in o sen ences o compac he in o ma ion.
This s ep is no manda o y bu enables luency and eadabili y.
4. Re e ing Exp essing Gene a ion: p oducing exp essions iden i ying objec s ha he
ex e e s o, as well as deciding abou p onouns. I is ela ed o lexicaliza ion.
5. Lexicaliza ion: selec ing he speci ic wo ds and ph ases o be used o ep esen
concep s and ela ions ha appea in he messages.
6. Realiza ion: applying ules o syn ax, mo phology and spelling o p oduce he inal
ou pu , such as adding p eposi ions, making plu als o adding punc ua ion ma ks.
3.3 Machine Lea ning
Machine Lea ning is he sub- ield o A i icial In elligence ha s udies compu e algo i hms
ha imp o e au oma ically h ough expe ience. In con as o adi ional algo i hms, ha
ha e speci ic ules o doing he asks hey a e equi ed o , Machine Lea ning algo i hms
lea n concep s wi hou being explici ly p og ammed o doing so, iden i ying pa e ns om
gi en examples o pe o m accu a ely on new, unseen da a by in e ing hese unco e ed
12 CHAPTER 3. STATE OF THE ART
pa e ns. These algo i hms can lea n om and make p edic ions on la ge olumes o da a,
no exclusi ely images.
The ield s a ed o be s udied in he mid 1980s and ea ly 1990s, and became mo e
popula a ound 2010 as mo e and mo e da a we e a ailable o lea ning and aining he
Machine Lea ning models.
This discipline has many applica ions in a wide a ie y o ields whe e designing and
p og amming explici algo i hms wi h good pe o mance is di icul o in easible. These
applica ions include oice ecogni ion, na u al language p ocessing, ansla ion, sea ch
engines, compu e ision, obo ics, medical diagnosis, inancial ma ke analysis, ad e is-
ing o ecommende sys ems [Pang e al., 2002, We nick e al., 2010, B idge e al., 2014,
Sa ikaya e al., 2014, Baik and Bala, 2004].
We can classi y Machine Lea ning echniques in o wo ca ego ies: supe ised lea ning
and unsupe ised lea ning. In supe ised lea ning, he Machine Lea ning algo i hm is
gi en a se o sample inpu s oge he wi h hei desi ed ou pu s, so ha i can ain
wi h hose examples o in e a gene alized unc ion ha maps inpu s in o ou pu s and
p edic s well when gi en a new example. The e o e, al eady labeled-da a a e equi ed in
his app oach.
The e a e also some special cases o supe ised lea ning such as semi-supe ised lea n-
ing, ha consis o a se o examples whe e no all o hem go wi h hei co esponding
desi ed ou pu , ha is, he e a e labeled and unlabeled da a on he se ; ac i e lea ning,
whe e he algo i hm has only a small ini ial se o labeled da a and i is able o in e ac-
i ely make que ies o he use o ind he desi ed ou pu s o some unlabeled da a; and
ein o cemen lea ning, whe e he e is some so o eedback ob ained om he ou side in
a dynamic en i onmen as an answe o i s ac ions.
Some o he main supe ised lea ning algo i hms and app oaches a e he ollowing:
•Decision ees and andom o es s: hey a e used as a p edic i e model. In a decision
ee, he b anches ep esen he obse a ions abou an i em, i s ea u es and he
lea es ep esen class labels. Random o es s [Ho, 1995] cons uc many decision
ees o make mo e accu a e p edic ions by ou pu ing a combina ion o he ou pu s
o all hose decision ees, ypically he mode o mean. Random o es s a e e y
popula as i is easy and clea o see and unde s and he decisions aken by hem,
while o he Machine Lea ning algo i hms a e mo e obscu e.
•Logis ic eg ession: i is a s a is ical me hod o model a bina y dependen a iable
(classes 0 and 1) in e ms o one o mo e explana o y a iables, using he logis ic
unc ion σ(z) = 1
1+e−z. This unc ion e u ns he p obabili y o belonging o class
1. The algo i hm’s ou pu is hen selec ed by compa ing he p obabili y o a gi en
h eshold (usually 0.5).
•Suppo Vec o Machines (SVM) [Co es and Vapnik, 1995]: i is a bina y classi i-
ca ion algo i hm ha gi en a se o poin s o wo di e en classes in a n-dimensional
3.4. DEEP LEARNING 13
space, sepa a es hem wi h he hype plane ha is si ua ed he u hes om all he
poin s. This algo i hm is only use ul i he poin s a e linea ly sepa able, i no , he
algo i hm was imp o ed by including ke nels, ha p ojec he se o poin s in a
bigge dimension space, o ob ain a hype su ace ha sepa a es he poin s in he
ini ial space.
•Neu al ne wo ks: i is a lea ning algo i hm whe e compu a ions a e s uc u ed in
connec ed neu ons ha ansmi signals om one o ano he . The e o e, each neu-
on’s inpu is he ou pu o ano he neu on (o he inpu o he algo i hm o he i s
laye o neu ons), and each neu on’s ou pu is compu ed as a non-linea unc ion o
he inpu .
•Nea es neighbo algo i hm [Al man, 1992]: i is used bo h o classi ica ion ( he e
a e a disc e e numbe o classes) and eg ession ( he a ge a iable is con inuous),
and i is based on he in o ma ion o he k closes poin s. In classi ica ion, he ou pu
is he mos common class among i s k neighbo s, and in eg ession he ou pu is he
mean o he alues o hese neighbo s.
Unsupe ised lea ning algo i hms lea n om a se o unlabeled da a, inpu examples
wi hou hei desi ed ou pu , in e ing a unc ion ha can desc ibe he s uc u e in he
da a. The e o e, he sys em mus be capable o inding pa e ns in he da a in o de o
be able o label he new inpu s; bu , as he da a used o ain he algo i hm is unlabeled,
he e is no way o e alua ing he accu acy o he pa e ns ound in i .
The ollowing a e he mos common unsupe ised lea ning algo i hms:
•Clus e ing: hese me hods di ide he da a se in o di e en subse s, o clus e s, so
ha he poin s in he same clus e a e mo e simila be ween hem han be ween
hose o o he g oups. The goal is o maximize he simila i y be ween poin s in he
same subse and o maximize he di e ence be ween poin s o di e en subse s.
•P incipal Componen Analysis (PCA) [Ho elling, 1933]: his algo i hm is used o
educe he dimensionali y o he da a se , wi hou losing any impo an in o ma ion
and il e ing edundancy, by making linea combina ions o he o iginal da a a iables
and selec ing he ones ha ha e he la ges possible a iance and a e unco ela ed.
3.4 Deep Lea ning
Deep lea ning is a Machine Lea ning echnique, based on neu al ne wo ks wi h many laye s
be ween he inpu and he ou pu ones [LeCun e al., 2015].
Al hough he heo y behind Deep Lea ning has been de eloped o many yea s now,
deep neu al ne wo ks need much compu e capaci y and labeled da a o pe o m apidly
and co ec ly he complex asks i is use ul o . The e o e, Deep Lea ning has only ecen ly
20 CHAPTER 3. STATE OF THE ART
Figu e 3.5: Cap ionbo
This de ice is e en able o lea n new aces and objec s i he use shows hem o Ho us in
di e en angles and speaking ou loud he name o he objec o he pe son.
As we ha e seen, mo e and mo e sophis ica ed image cap ion gene a o s, o di e en
pu poses and con ex s, a e being de eloped, showing he in e es and impac o his b and-
new ield and esul ing in a ich li e a u e.

Chap e 4
F amewo k
T aining and de eloping Deep Lea ning models equi es g ea compu a ional capaci y and
a good elec ion o he da ase . In his chap e , we explain wha a e GPUs and why a e
hey use ul o de eloping Deep Lea ning p ojec s, which Deep Lea ning amewo k and
isualiza ion ools we ha e selec ed and which da ase we use. We de eloped he p ojec
using Py hon as p og amming language.
4.1 GPU
GPUs (G aphics P ocessing Uni ) we e adi ionally c ea ed o compu e g aphics and
image p ocessing o accele a e he complex calcula ions o eal- ime 2D and 3D g aphics
ha esul ed in long p ocessing ime in CPUs.
GPUs a e build o op imized asks based on he SIMD pa allel concep (Single In-
s uc ion Mul iple Da a), ha is, he same ins uc ion applied epea edly o e a bunch o
da a. Those epea ed applica ions o he same ins uc ion a e all independen and can,
he e o e, be un simul aneously. Linea algeb a ope a ions a e inhe en ly pa allel and a e
he basis o polygon ans o ma ions o scene ende ing in compu e g aphics.
The e a e o he powe and ime consuming applica ions o pa allel na u e ha could
be op imized by using a GPU, bu hey in ol e e o mula ing he p oblem in o g aphics
p imi i es o be able o use hem, which is a e y complex ask. Fo ha eason, NVIDIA
de eloped CUDA (Compu e Uni ied De ice A chi ec u e), a pa allel compu ing oolki
simila o C p og amming language, o exploi GPUs capabili ies o gene al pu poses,
which is commonly called GPGPU (Gene al Pu pose G aphics P ocessing Uni ).
Linea algeb a is he basis o Deep Lea ning algo i hms, and also many Machine Lea n-
ing algo i hms. Neu ons wi hin a laye apply all he same unc ion o di e en inpu da a,
so hey could be also pa allelized and hence GPUs a e always used o aining hese
models.
21
22 CHAPTER 4. FRAMEWORK
Figu e 4.1: Deep Lea ning amewo ks popula i y
Acqui ing a GPU is no cheap, he e exis s cloud compu ing pla o ms, such as Amazon
Web Se ices1, Google Cloud Pla o m2o Mic oso Azu e3 ha con ain di e en se e s
wi h se e al GPUs o allow use s o un hei models mo e economically. GAIA esea ch
g oup has buil a se e wi h an NVIDIA Ti an X GPU in o de o main ain Deep Lea ning
p ojec s o s uden s and p o esso s and ga e us access o i , so we will use i o aining
his p ojec .
4.2 Deep Lea ning amewo k
CUDA has a e y low le el API and makes implemen ing Deep Lea ning compu a ional
g aphs conside ably edious. Fo his eason, many highe le el amewo ks we e c ea ed
o easily desc ibing and de eloping he complex asks ha Deep Lea ning in ol es.
The e is a wide numbe o amewo ks a ailable o Deep Lea ning aining, he ma-
jo i y o hem open-sou ce and de eloped by big companies o uni e si y eams such as
Mic oso , Google, In el o Be keley Uni e si y. The mos well-known and equen ly
used amewo ks a e To ch [Collobe e al., 2011], Tenso low [Abadi e al., 2016], Ca e
[Jia e al., 2014] and Theano [Theano De elopmen Team, 2016].
Among all hese amewo ks, Tenso low is p obably he mos popula one (as Fig-
u e 4.1, ex ac ed om Google images, shows), used in dis inguished companies such as
Ai bnb, Ube , D opbox o Twi e . I is an open-sou ce amewo k o ad anced nume i-
cal compu a ion, de eloped by he Google B ain eam in 2015, eplacing hei p opie a y
1h ps://aws.amazon.com
2h ps://cloud.google.com/
3h ps://azu e.mic oso .com
4.2. DEEP LEARNING FRAMEWORK 23
om ke as.models impo Sequen ial
om ke as.laye s impo Dense
# Building he model and lea ning con igu a ion
model = Sequen ial()
model.add(Dense(uni s=64, ac i a ion=’ elu’, inpu _dim=100))
model.add(Dense(uni s=10, ac i a ion=’so max’))
model.compile(loss=’ca ego ical_c ossen opy’, op imize =’sgd’,
me ics=[’accu acy’])
# T aining, e alua ion and p edic ion
# x_ ain, y_ ain, x_ es , y_ es and new_da a a e numpy a ays
model. i (x_ ain, y_ ain, epochs=5, ba ch_size=32)
loss_and_me ics = model.e alua e(x_ es , y_ es , ba ch_size=128)
p edic ion = model.p edic (new_da a, ba ch_size=128)
Figu e 4.2: A simple Ke as example code
Machine Lea ning sys em Dis Belie [Dean e al., 2012], c ea ed in 2011. Tenso low is
widely used o Deep Lea ning and Machine Lea ning and i is op imize o un models
bo h on CPUs o GPUs. I p o ides an API o Py hon as well as o o he p og amming
languages (C++, Haskell, Ja a...), bu he mos de eloped one is he Py hon lib a y.
Tenso low wo ks well o modeling Con olu ional Neu al Ne wo ks and Recu en
Neu al Ne wo ks. The basic objec in Tenso low lib a y is he enso , an abs ac ion o
mul idimensional a ays, being a 1-dimensional enso a ec o and a 2-dimensional enso
a ma ix. Deep Lea ning compu a ional g aphs a e build by desc ibing enso s’ ope a ions
and g adien s a e calcula ed au oma ically using a speci ic unc ion.
Howe e , p og amming in Tenso low can be di icul and he e exis s a highe le el
API ha allows lexible Tenso low implemen a ions in a mo e use - iendly manne . This
API is Ke as4, a high-le el open-sou ce Py hon lib a y o de eloping Deep Lea ning
p ojec s mo e easily and eadable, c ea ed in 2015 by a Google enginee and eleased
unde he MIT license. Ke as wo ks wi h Tenso low, as well as Theano o CNTK
[Seide and Aga wal, 2016], as backend. In 2017 Tenso low included i as pa o i s lib a y
and selec ed i as he p e e ed high-le el API.
The basic s uc u e in Ke as is he model, a se o connec ed laye s. Compu a ional
g aphs a e buil by s acking laye s wi h he add() unc ion, whe e he use only needs o
speci y he laye s and he inpu s’ dimensions. Ke as lib a y p o ides he well-known laye s
bu allows use s o also c ea e new ones. Con igu ing he aining p ocess is as simple as
applying o he model he compile() unc ion wi h he speci ic condi ions (loss unc ion,
op imize , me ics...). T aining, e alua ing he model and making p edic ions on unseen
examples is done wi h jus o he high le el unc ions: i (),e alua e() and p edic ()
4h ps://ke as.io/
24 CHAPTER 4. FRAMEWORK
espec i ely. Figu e 4.2 shows a simple example, ex ac ed om Ke as documen a ion,
o how o use hese unc ions o build a neu al ne wo k wi h wo ully-connec ed laye s.
The e o e, as ou needs a e co e ed in Ke as’ lib a y, i is widely ecommended o
Deep Lea ning s a e s and i is mo e use - iendly, we decided o de elop ou model in
Ke as on op o Tenso low. Ke as and Tenso low ins alla ion is done easily in Linux ia
he pip ins all command, c ea ing p e iously a Py hon i ual en i onmen .
4.3 Visualiza ion: Tenso Boa d
Deep Lea ning models a e composed o millions o complex calcula ions ha o en com-
plica e, o e en make i impossible, o debug and unde s and he pa ame e s and me ics,
wha happens du ing aining and wha is he neu al ne wo k lea ning. Fo his pu pose,
Tenso low de eloped Tenso boa d5, a sui e o isualiza ion ools inside Tenso low lib a y
ha allows use s o isualize compu a ion g aphs, me ics esul s and o he use ul cha s.
Tenso Boa d wo ks by sa ing log iles, con aining execu ion in o ma ion abou he
aining p ocess and he compu a ion g aph, in o a speci ic log di ec o y, and opening
Tenso Boa d web se e o access he g aphic in e ace.
The desi ed da a is ob ained wi h summa y ope a ions, enso ’s ope a ions ha p o-
duce se ialized in o ma ion abou a model ha needs o be ead wi h Tenso Boa d.
These summa y ope a ions ecei e he enso we wan o isualize and a meaning ul
name o dis inguish all o hem. Then, all he summa ies c ea ed a e combined using
.summa y.me ge all in o a single ope a ion ha gene a es all o hem. Finally, he se i-
alized summa y is w i en o he speci ied di ec o y in disk wi h .summa y.FileW i e .
To access he g aphic in e ace o Tenso Boa d o isualize he summa ies, we jus need
o go o h p://localhos :6006 a e yping he ollowing command in he command
line:
$ enso boa d --logdi pa h/ o/log-di ec o y
I is also possible o compa e isualiza ions o di e en execu ions by speci ying he
pa hs o he log di ec o ies in he p e ious command. The appea ance o Tenso Boa d
g aphic in e ace is shown in Figu e 4.3
Tenso Boa d con ains a wide a ie y o help ul isualiza ions: .summa y.scala ,
.summa y.his og am, .summa y.audio o .summa y.image, among o he s. The
i s one is he mos used, i displays he a ia ion o a scala me ic o e ime, usually he
loss o he lea ning a e. All hese isualiza ions admi some dynamic in e ac ions wi hin
he Tenso Boa d in e ace.
Using Tenso Boa d wi hin Ke as high-le el API i is e en easie . I is jus necessa y o
include he callback enso boa d unc ion while aining, speci ying he log di ec o y.
5h ps://gi hub.com/ enso low/ enso boa d
4.4. IMAGE DATASETS 25
Figu e 4.3: Tenso Boa d web se e
This unc ion will eco d da a o loss and accu acy me ics, as well as o he me ics
speci ied in he compile unc ion.
4.4 Image da ase s
Fo co ec ly aining and accu a ely e alua ing a good Machine Lea ning model, a sui able
selec ion o elabo a ion o he da ase is c ucial. Once we ha e he da ase , i is di ided
in o h ee disjoin subse s: aining, es and alida ion se s. T aining se is used o
aining he model, i ing he model’s pa ame e s; es se is used once he model is
ained, o p o ide an unbiased e alua ion o his inal model; alida ion se , also called
de elopmen se o de se , is used while aining he model, o une he hype pa ame e s
and o p e en o e i ing.
As Deep Lea ning esea che s ha e a s ong in e es in he image analysis ield, du -
ing hese yea s a lo o la ge and medium scale da ase s ha e been c ea ed o aining
hese models and we can ind many o hem a ailable online. The mos popula ones a e
explained below.

26 CHAPTER 4. FRAMEWORK
ImageNe
ImageNe 6is p obably he mos well-known image da ase in he Deep Lea ning and
machine lea ning indus y. This image da ase is used mainly o classi ica ion pu poses
and he classes a e o ganized based on he nouns in he Wo dNe [Mille , 1995] hie a chy.
This image da abase was c ea ed in 2009 [Deng e al., 2009] by que ying image sea ch
engines wi h Wo dNe ’s synse s. They ex ac ed a g ea numbe o candida e images
o each synse and hen il e ed hem by c owd-sou cing in Amazon Mechanical Tu k
(AMT)7. Since 2010, ImageNe also uns he ILSVRC (ImageNe La ge Scale Visual
Recogni ion Challenge) annual isual challenge [Russako sky e al., 2015]. Nowadays, his
challenge is hos ed in Kaggle8and i has h ee ca ego ies: image classi ica ion o 1000
classes, objec de ec ion (de ec he bounding box whe e he objec lays) o 200 classes
and objec de ec ion om ideo o 30 classes.
As o oday, ImageNe is one o he la ges image da abase a ailable. I has a o al o
14,197,122 anno a ed images o a wide a ie y o ca ego ies, wi h an a e age o o e i e
hund ed images pe synse . Fo he pe son’s high-le el ca ego y ( he one we ha e in e es
in), i holds a ound 952,000 images and i is di ided in 2,035 synse s wi h an a e age o
468 images pe synse . Images a e anno a ed wi h hei classi ica ion and 1,034,908 o
hem also wi h hei bounding boxes.
MS COCO
MS COCO9(Mic oSo Common Objec s in COn ex ) da ase [Lin e al., 2014] is one o
he mos widely used la ge-scale da ase s o image desc ip ion gene a ion and also o
objec de ec ion, objec segmen a ion and pe son keypoin s de ec ion.
Cu en ly, he MS COCO da ase consis s o o e 328,000 images o 91 basic objec
ypes in na u ally occu ing con ex s. Each image is labeled wi h a leas i e cap ions
desc ibing i , which leads o a o al o 2.5 million cap ions, and bounding boxes o each
objec ca ego y ha appea s in he pic u e. They also con ain 250,000 people anno a ed
wi h hei keypoin s.
This da ase has also gi en ise o image challenges o objec de ec ion, keypoin s de-
ec ion and image cap ioning. The la e is al eady closed, bu he e is an open e alua ion
se e and API [Chen e al., 2015] o compa e o s a e-o - he-a me hods using se e al
pe o mance me ics such as BLEU, ROUGE, METEOR and CIDE .
6h p://www.image-ne .o g/
7A well-known c owd-sou cing online pla o m. h ps://www.m u k.com/
8h ps://www.kaggle.com/
9h p://cocoda ase .o g/#home
4.4. IMAGE DATASETS 27
PASCAL
PASCAL10 (Pa e n Analysis S a is ical Modeling and Compu a ional Lea ning) Visual
Objec Classes p ojec comp ised challenges [E e ingham e al., 2010] o image classi ica-
ion, de ec ion and segmen a ion o eigh consecu i e yea s (2005-2012). Nowadays, he
challenges a e closed bu i p o ides an image da abase and ools o ob aining hese im-
ages and hei anno a ions, as well as an e alua ion se e . The la es da ase (PASCAL
2012 challenge) con ained 20 dis inc classes and mo e han 11,000 anno a ed images.
These images we e collec ed om Flick .
These da ase s ha e anno a ions o objec classi ica ion, de ec ion and segmen a ion,
bu a subse o he Pascal 2008 challenge da ase was used o c ea e an image desc ip ion
da abase, called Pascal1K. I con ains 1,000 images wi h objec s o di e en classes and
is anno a ed wi h i e desc ip ions, gene a ed by humans on Amazon Mechanical Tu k
c owd-sou cing pla o m. As i is a medium-scale da abase, i is gene ally used as a
benchma k o e alua ing image desc ip ion models.
Flick 8K and Flick 30K
Flick 8K11 [Hodosh e al., 2013] da ase and i s ex ension Flick 30K12 [Young e al., 2014],
con ain images om Flick , each one anno a ed wi h i e desc ip ions collec ed om he
c owd-sou ce pla o m Amazon Mechanical Tu k (AMT). Flick 8K consis s o app oxi-
ma ely 8,000 images and Flick 30K o a ound 30,000 images. These pic u es we e collec ed
by he Uni e si y o Illinois by que ying Flick o speci ic objec s and ac ions.
SBU
SBU Cap ioned Pho o Da ase was speci ically c ea ed o he Im2Tex image cap ion
gene a o [O donez e al., 2011]. I con ains 1 million Flick images labeled wi h hei
o iginal cap ions gene a ed by hei use s, collec ed by que ying Flick o speci ic objec s
and ac ions. I is a la ge-scale da ase bu no e y popula o aining image desc ip ion
models as he labeled cap ions may con ain subjec i e in o ma ion o in o ma ion no
con ained in he pic u e because hey we e gene a ed by he Flick use .
CIFAR
CIFAR-10 and CIFAR-100 (Canadian Ins i u e o Ad anced Resea ch) [K izhe sky, 2012]
a e image classi ica ion da ase s ha di e only in he numbe o classes hey ha e (CIFAR-
10 has en classes and CIFAR-100 one hund ed classes, g ouped in o wen y supe classes).
10Download page: h p://hos . obo s.ox.ac.uk/pascal/VOC/
11Fo m o downloading he Filck 8K da ase :h ps:// o ms.illinois.edu/sec/1713398
12Fo m o downloading he Filck 30K da ase : h ps:// o ms.illinois.edu/sec/229675
28 CHAPTER 4. FRAMEWORK
Figu e 4.4: COCO cap ion da ase
Bo h da ase s consis o 60,000 32x32 RGB images, CIFAR-10 holds 6,000 in each class
and CIFAR-100 holds 600. Ke as p o ides buil -in unc ions o load hese da ase s.
LabelMe
LabelMe13 is an image da abase and an online anno a ion ool [Russell e al., 2008] c ea ed
in he MIT Compu e Science and A i icial In elligence Labo a o y (CSAIL).
The da ase is dynamic, ee o use and open o public con ibu ion, and i consis s
o almos 200,000 images bu no all o hem a e ully anno a ed. I has mo e han
62,000 anno a ed images and mo e han 658,000 labeled objec s. The online anno a ion
ool allows he use o d aw polygons, que y images, b owse he da abase o download a
subse .
4.4.1 Da ase elec ion
Ha ing s udied he cha ac e is ics o all hese da ase s, we decided o use he MS COCO
da ase o aining and es ing ou model, as i con ains a wide numbe o images, i is one
o he mos commonly used o image desc ip ion and i p o ides ools o downloading
he da ase . They al eady p o ide a ain/ al/ es spli o he da ase , howe e , as his
da ase was eleased o hei annual challenges, he es se does no come labeled and
we will no be able o use i o es ing ou model, so we will c ea e ou own spli based
on he aining and alida ion se s.
13Download page: h p://labelme.csail.mi .edu
4.4. IMAGE DATASETS 29
Figu e 4.4 shows some examples o wha can be ound in COCO cap ion da ase .
In pa icula , we use he Py hon COCO API o download he MS COCO da ase and
o selec a subse o i con aining only images o people, o es ic o ou objec i es
he images we p o ide o he model. In addi ion, we also use he people’s subse (baby,
boy,gi l,man and woman ca ego ies) o he CIFAR-100 da ase o ine- une he image
ea u e-ex ac ion model we use, as explained in he nex chap e .
36 CHAPTER 5. DEEP LEARNING APPROACH
"""-----De ini ion o VGG16 ine- uned model-----"""
de model_de ini ion():
gg16_model = VGG16(weigh s="imagene ",
include_ op=False,
inpu _shape=(224,224,3))
# Replace op laye
gg16_ou pu = gg16_model.ou pu
x1 = Fla en()( gg16_ou pu )
x2 = Dense(4096, ac i a ion=’ elu’, name=" c1")(x1)
x3 = Dense(4096, ac i a ion=’ elu’, name=" c2")(x2)
p edic ion = Dense(100, ac i a ion=’so max’, name="p edic ions")(x3)
# C ea e model
model = Model(inpu = gg16_model.inpu , ou pu =p edic ion)
# Se i s laye s non- ainable
o laye in model.laye s[:12]:
laye . ainable = False
model.compile(op imize =SGD(l =1e-3, decay=1e-6, momen um=0.9),
loss=’ca ego ical_c ossen opy’,
me ics=[’accu acy’])
e u n model
Figu e 5.4: Py hon code o VGG16 ine- uned model de ini ion.
unc ion wi h a ba ch size o 32 images and 30 epochs3, and sa e i in a hd 54 ile in o
memo y using model.sa e().
5.3 Da a p epa a ion
Once we ha e ou ine- uned VGG16 model, we can s a de ining he models o gene a ing
image desc ip ions. As we explained in he i s sec ion o his chap e , we will use COCO
da ase , pa ly o en i ely depending on he model, o aining and es ing ou models.
Fo una ely, COCO p o ides a Py hon API o loading in o memo y and pa sing he
da ase in an easie way. To use his API, he da ase iles mus be downloaded om hei
webpage in an speci ic di ec o y s uc u e. We ha e downloaded he 2014 da ase because
i is he one ha con ains image desc ip ions as labels. A e cloning he gi hub p ojec
and unning he Make ile, he API can be used by impo ing he pycoco ools.coco
3An epoch is an en i e i e a ion o e he whole aining da ase . The ba ch size is he numbe o aining
samples ained be ween pa ame e s upda es.
4File o ma o s o e la ge amoun s o da a.

5.3. DATA PREPARATION 37
module. Mo e in o ma ion on how o use his API can be ound on hei webpage5and
hei gi hub p ojec 6.
All COCO images and desc ip ions mus be p ep ocessed in o de o co ec ly use
hem as he inpu s o ou models. E en hough wo o ou models wo k only wi h a
subse o COCO, he o he wo wo k wi h he whole da ase , so bo h image p ep ocessing
and desc ip ions p ep ocessing ha e been done in he whole da ase , lea ing he subse
ex ac ion o a la e s ep whe e we also pe o m he ain/ al/ es spli o he da ase as
we explained in Sec ion 4.4. In he ollowing subsec ions we explain how we ha e ca ied
ou hese wo asks.
5.3.1 Image p ep ocessing
COCO images do no need oo much p ep ocessing. Howe e , he Con olu ional Neu al
Ne wo k o ou models, ei he he VGG16 ne wo k o he ine- uned model, has al eady
p ecompu ed i s weigh s and hese weigh s will emain ozen when aining ou image
desc ip ions gene a ion models. The e o e, images ea u es could be p ecompu ed be o e
aining he models and sa ed o be used la e as an inpu ins ead o he images hemsel es.
This way we only calcula e hese ea u es once ins ead o e e y ime he aining p ocess
passes h ough one o hese images, sa ing compu a ional ime and memo y space.
This image p ep ocessing mus be epea ed o he VGG16 model and o ou ine-
uned model, o ex ac he ea u es ob ained by bo h models and use one o ano he
depending on which o he ou models we a e aining. Figu e 5.5 shows he ea u es
ex ac ion unc ion we ha e de ined, whe e model e e s o ei he VGG16() o o ou ine-
uned model, ea u es is he dic iona y ha will s o e he ex ac ed image ea u es and
images ids a e all COCO aining image ids ob ained wi h COCO API as ollows:
da aDi = ".."
da aType = " ain2014"
annFile = ’{}/anno a ions/ins ances_{}.json’. o ma (da aDi ,da aType)
coco = pycoco ools.coco.COCO(annFile)
imgs_ids = coco.ge ImgIds()
This p ocess is also done wi h he alida ion COCO da ase , changing da aType o
" al2014". To ex ac image ea u es, las laye o he model mus be emo ed using
laye s.pop() because his laye is he one ha p edic s which class he image belongs o
and we only ca e abou image ea u es. We hen load e e y image in he COCO da ase
using COCO API and Ke as load img() unc ion. Image a ge size is 224x224 as his
is he inpu size o VGG16 (and he e o e also o ou ine- uned model). Once we ha e
loaded he image in o memo y, i is con e ed o a numpy a ay and eshaped, using Ke as
unc ions om he ke as.p ep ocessing.image module. Finally, ea u es a e ex ac ed
5h p://cocoda ase .o g/download
6h ps://gi hub.com/cocoda ase /cocoapi
38 CHAPTER 5. DEEP LEARNING APPROACH
de ea u es_ex ac ion(model, ea u es, di ec o y, da aType, images_ids):
# emo e p edic ion laye om model
model.laye s.pop()
model = Model(inpu s=model.inpu s, ou pu s=model.laye s[-1].ou pu )
o id in images_ids:
ile = "0:0>12". o ma (id)
pa h = "/images//COCO__.jpg". o ma (di ec o y, da aType, da aType, ile)
img = load_img(pa h, a ge _size=(224, 224))
# img o 3D numpy a ay (heigh , weigh , colo channel) and eshape i
img = img_ o_a ay(img)
img = img. eshape((1, img.shape[0], img.shape[1], img.shape[2]))
# ge ea u es and s o e hem in dic
img = p ep ocess_inpu (img)
ea u e = model.p edic (img, e bose=1)
ea u es[id] = ea u e
e u n ea u es
Figu e 5.5: Image p ep ocessing: ea u es ex ac ion o COCO images.
using he p edic () unc ion and hen s o ed in o he dic iona y. A e ha ing ex ac ed
all aining and alida ion images ea u es, we sa e he dic iona y con aining hem in a
ile, using Py hon pickle lib a y7, o la e use.
5.3.2 Tex p ep ocessing
The image desc ip ions p ep ocessing ask ha we ha e implemen ed pe o ms a ious
ope a ions on he wo ds o clean hose desc ip ions and ob ain a sui able ocabula y
o lea ning, ha should be ep esen a i e and exp essi e enough. These ope a ions a e:
changing uppe case le e s o lowe case, emo ing punc ua ion ma ks, emo ing numbe s,
emo ing wo ds o leng h one (as hey a e no exp essi e), emo ing wo ds used less
han i e imes in he whole da ase and ans o ming English con ac ions such as don’
o do no . We also add ini ial and inal okens (START and END) o delimi ing he
desc ip ions.
Figu e 5.6 shows he p ep ocessing unc ion we ha e de ined o cleaning COCO
desc ip ions, whe e: wo d eq is a Py hon Coun e 8wi h he wo ds’ equencies and
desc ip ions is a lis con aining all he desc ip ions o COCO aining da ase and i is
ob ained using COCO API as ollows:
7h ps://docs.py hon.o g/3/lib a y/pickle.h ml
8h ps://docs.py hon.o g/3/lib a y/collec ions.h ml#collec ions.Coun e
5.4. DATASET SPLIT 39
de p ep ocess_desc(desc ip ions, wo d_ eq):
# ansla ion able o emo ing punc ua ion
able = s .make ans(’’, ’’, s ing.punc ua ion)
o desc_in o in desc ip ions:
desc = desc_in o[’cap ion’]
wo ds = desc.spli ()
# con e o lowe case
wo ds = [w.lowe () o w in wo ds]
# emo e numbe s
wo ds = [w o w in wo ds i no w.isnume ic()]
# ans o m con ac ions
wo ds = [con ac ions(w) o w in wo ds]
# emo e punc ua ion om each wo d
wo ds = [w. ansla e( able) o w in wo ds]
# emo e wo ds leng h < 2
wo ds = [w o w in wo ds i len(w)>1]
# emo e wo ds used < 5 imes
wo ds = [w o w in wo d_ eq.keys() i wo d_ eq[w]>=5]
# s o e in dic iona y wi h START and END okens
desc_in o[’cap ion’] = ’START ’ + ’ ’.join(wo ds) + ’ END’
Figu e 5.6: Tex p ep ocessing: cleaning COCO desc ip ions.
da aDi = ".."
da aType = " ain2014"
annFile = ’/anno a ions/cap ions_.json’. o ma (da aDi ,da aType)
coco = COCO(annFile)
anns_ids = coco.ge AnnIds()
desc ip ions = coco.loadAnns(anns_ids)
This p ocess is also epea ed wi h he alida ion desc ip ions, as we ha e done wi h im-
age p ep ocessing, changing da aType o " al2014". Each elemen o he desc ip ions
lis is a dic iona y con aining he image id he desc ip ion belongs o, an id o he de-
sc ip ion and he desc ip ion i sel . Fo ans o ming con ac ions, we ha e de ined a
dic iona y con aining he ans o ma ions and con ac ions() e u ns his ans o ma-
ion i he wo d is in ha dic iona y. Once we ha e p ep ocessed all aining and alida ion
desc ip ions, we me ge bo h lis s in o a single one and sa e i in a ile using pickle lib a y.
5.4 Da ase spli
A e ha ing p ep ocessed he en i e COCO da ase , we now spli i in o ain, alida ion
and es se s (wi h a 70%-20%-10% di ision) and di ide consequen ly he ea u es dic io-
na ies (VGG16 ea u es and ine- uned model ea u es) ob ained while p ep ocessing he
40 CHAPTER 5. DEEP LEARNING APPROACH
# ge ids o 35000 aining imgs gene al and wi h people
da aDi = ".."
da aType = " ain2014"
annFile = ’/anno a ions/ins ances_.json’. o ma (da aDi ,da aType)
coco = COCO(annFile)
ca s_ids = coco.ge Ca Ids(ca Nms=[’pe son’])
ain_people_ids = coco.ge ImgIds(ca Ids=ca s_ids)
ain_people_ids = andom.sample( ain_people_ids, 35000)
ain_ids = coco.ge ImgIds()
ain_imgs_ids = andom.sample( ain_ids, 35000)
# ge ids o 35000 alida ion imgs gene al and wi h people
da aDi = ".."
da aType = " al2014"
annFile = ’/anno a ions/ins ances_.json’. o ma (da aDi ,da aType)
coco = COCO(annFile)
ca s_ids = coco.ge Ca Ids(ca Nms=[’pe son’])
al_people_ids = coco.ge ImgIds(ca Ids=ca s_ids)
al_people_ids = andom.sample( al_people_ids, 15000)
al_ids = coco.ge ImgIds()
al_imgs_ids = andom.sample( al_ids, 10000)
# spli al people in al and es
al_people_ids, es _ids = ain_ es _spli ( al_people_ids, es _size=5000)
Figu e 5.7: Tes spli o COCO da ase .
images and he desc ip ions lis ob ained while p ep ocessing he desc ip ions.
As ou goal is o e alua e how well ou models desc ibe images con aining people, he
es se mus only include his kind o images. On he o he hand, he ain and alida ion
se s, may o may no include o he kind o images, depending on he model ha we a e
aining. The e o e, we will ha e wo di e en spli s o hese wo se s, one o hem o he
subse o images con aining people (excluding he ones in he es se ) and he o he one
o he whole da ase wi hou he es se .
T aining ou models wi h he whole COCO da ase (mo e han 300,000 images) was
impossible due o he cha ac e is ics o he se e we ha e used and o ou ime limi a ion.
In o de o educe ime and memo y consump ion, we decided o es ic ou aining
p ocess o only a subse o he COCO da ase : 35,000 images o he aining se , 10,000
images o alida ion and 5,000 images o he es se (70%-20%-10%).
To ob ain he aining se we ha e andomly selec ed, using andom.sample() Py hon
unc ion, 35,000 images o he whole COCO aining se and 35,000 images o all he COCO
aining images ha con ain people. Valida ion and es se s ha e been ex ac ed bo h
5.5. TRAINING PROCESS 41
om COCO alida ion se , andomly selec ing 15,000 COCO alida ion images con aining
people (5,000 o he es se and he emaining 10,000 o he alida ion se ) and 10,000
COCO alida ion images o any kind. This p ocess is shown in Figu e 5.7. We ha e
also ensu e ha he gene al aining and alida ion spli s a e ep esen a i e enough by
e i ying ha he andom subse s ha e enough images ha con ain people (30%-60%).
Once we ha e pe o med he da ase spli s, we di ide he ea u es dic iona ies and he
desc ip ions lis acco dingly o he di e en spli s and we sa e hese di isions in memo y
using pickle lib a y.
5.5 T aining p ocess
The i s s ep in he aining p ocess o he ou models is o load he iles ob ained wi h
he image and ex p ep ocessing and di ided when making he da ase spli . Following
he enume a ion we ha e de ined in Sec ion 5.1, Model 1 and Model 3 load image ea u es
ob ained wi h VGG16 and Model 2 and Model 4 load he ones ob ained wi h he ine-
uned VGG16 model. Subsequen ly, Model 1 and Model 2 load he ain and alida ion
spli o he whole da ase and Model 3 and Model 4 he one o he subse o images ha
con ain people.
The emaining s eps in he aining p ocess a e he same o he ou models, wi h
he only di e ences being he inpu s. A e loading he images ea u es and desc ip ions,
we c ea e and i a Tokenize o e all aining and alida ion desc ip ions, using Ke as
Tokenize () class. A Tokenize lea ns om a lis o ex s how o map he bag o wo ds
con ained in ha ex s o in ege alues and allowing o see each desc ip ion as an in ege
sequence.
The nex s ep is o de ine he model ha we a e going o ain. The model de ini ion
in Py hon using Ke as ollows he a chi ec u e ha we ha e de ined in Figu e 5.2 and i
is shown in Figu e 5.8. The hype pa ame e s a e ex ac ed om Tan i e al. obse a ions
[Tan i e al., 2017b]. ocab size,max leng h and emb ma ix a gumen s a e he size o
he whole bag o wo ds con ained in he desc ip ions, he leng h o he la ges desc ip ion
and he GloVe wo d embedding weigh ma ix espec i ely.
The unc ion s a s wi h he de ini ion o he Ke as inpu enso o he image ea u es
using Inpu () and he inpu shape is 4096 as i is he shape o he ea u es ec o ex ac ed
o each image. This inpu is passed o a Dense laye wi h a ReLu ac i a ion unc ion
and L2 egula iza ion o ob ain a 128 elemen ep esen a ion. Regula iza ion echniques
a e in oduced in Deep Lea ning aining p ocesses o p e en o educe o e i ing (good
lea ning o he aning se bu oo speci ic, leading o no gene aliza ion powe ). In pa ic-
ula , L2 egula iza ion adds an ex a e m o he cos unc ion p opo ional o he sum o
squa es o all he weigh s. We hen de ine he language model wi h a Ke as Inpu () en-
so o leng h max leng h and his inpu is passed o a wo d embedding laye wi h GloVe
p e- ained weigh s o map he spa se wo d ec o s o dense ones, as we explained in Sec-
ion 5.1.2. A e he Embedding laye we pe o m a 0.5 d opou [S i as a a e al., 2014],

42 CHAPTER 5. DEEP LEARNING APPROACH
de model_de ini ion( ocab_size, max_leng h, emb_ma ix):
# ea u e ex ac o model
img_inpu s = Inpu (shape=(4096,))
e1 = Dense(128, ke nel_ egula ize =l2(1e-8), ac i a ion=’ elu’,
name=" ea u es")(img_inpu s)
# desc ip ion model
desc_inpu s = Inpu (shape=(max_leng h,))
de1 = Embedding( ocab_size, 100, weigh s=[emb_ma ix],
ainable=False)(desc_inpu s)
de2 = D opou (0.5, name=" ex _d opou ")(de1)
de3 = LSTM(128, name="LSTM_laye ")(de2)
# me ge model
me ge_inpu s = [ e1,de3]
me1 = add(me ge_inpu s)
me2 = Dense(128, ac i a ion=’ elu’, name="decode ")(me1)
ou pu s = Dense( ocab_size, ke nel_ egula ize =l2(1e-8),
ac i a ion=’so max’, name="p edic ions")(me2)
model = Model(inpu s=[img_inpu s, desc_inpu s], ou pu s=ou pu s)
model.compile(loss=’ca ego ical_c ossen opy’, op imize =’adam’,
me ics=[’accu acy’])
model.summa y()
e u n model
Figu e 5.8: Image desc ip ion gene a ion model de ini ion wi h Ke as.
a egula iza ion echnique used o p e en o e i ing ha consis s in igno ing while ain-
ing some neu ons selec ed andomly. This is ollowed by a Ke as LSTM laye wi h 128
memo y uni s. Nex we me ge bo h he ou pu o he LSTM laye and he ou pu o he
Dense laye and apply a Dense laye wi h ReLU ac i a ion unc ion and a so max Dense
laye , wi h L2 egula iza ion, o e he ocabula y size o make he p edic ion o he nex
wo d. This model is ained wi h a ca ego ical c oss en opy loss (as he ou pu o he
ne wo k a e he p obabili ies o he wo ds in he ocabula y o be he nex wo d in he
desc ip ion) and Adam op imize (see Sec ion 5.5.1) wi h he de aul alues, as Tan i e
al. sugges [Tan i e al., 2017b].
Finally, he las s ep is o i he model wi h he aining and alida ion da a ha we
ha e loaded. As COCO is a e y big da ase and o ease memo y usage, we use a gene -
a o o p oduce aining and alida ion da a and i he model wi h he i gene a o
unc ion. To con ol he aining p ocess, we use he i gene a o unc ion wi h a
ModelCheckpoin callback, o sa e he model a e each epoch moni o ing he loss on he
alida ion se , and a Tenso Boa d callback, o sa e enso boa d logs o isualiza ion (as
we explained in Sec ion 4.3).
5.5. TRAINING PROCESS 43
Figu e 5.9: Op imiza ion algo i hms s eps eaching local op imum.
We ha e ained he ou models 35 epochs in he se e wi h he GPU. The whole
p ocess (VGG16 ine- une, da a p epa a ion and aining o he ou models) las ed a ound
eigh days.
5.5.1 Op imize s
As we explained in Sec ion 3.4, Deep Lea ning models lea n by ying o ind he op i-
mal pa ame e s ha minimize he cos unc ion and he g adien descen me hod is an
op imiza ion algo i hm o app oxima e a local op imum by compu ing he g adien o he
unc ion. Howe e , he cos unc ion depends on all aining samples, and he e o e so
does he g adien , which makes i oo expensi e o compu e o Deep Lea ning models
ha a e ypically ained on a big da ase . Fo his eason, as he cos unc ion is usually
a sum o e he aining se , an inc emen al app oach o calcula ing he g adien called
S ochas ic G adien Descen (SGD) was de eloped. I calcula es he g adien based only
on a small subse o he aining examples o speed up his calcula ion. This usually
in ol es aking mo e i e a ions o app oxima e he minimum, bu as e .
In SGD, weigh s a e upda ed using he same lea ning a e o all weigh s and all
i e a ions, making i di icul o slow o con e ge o he minimum i he lea ning a e is
oo big o o small. Many a ian s ha e been p oposed and de eloped in o de o upda e
his SGD algo i hm o y o each as e he local minimum.
Some o his a ian s a e AdaG ad [Duchi e al., 2011], Adam [Kingma and Ba, 2014]
and RMSP op. AdaG ad main ains a pe -pa ame e lea ning a e ins ead o ha ing he
44 CHAPTER 5. DEEP LEARNING APPROACH
same lea ning a e o all he pa ame e s, upda ing i based on he sum o p e ious g adien
alues. So does RMSP op, ha also has a lea ning a e o each pa ame e bu hese
lea ning a es a e upda ed based on he a e age o ecen g adien calcula ions, ha is,
based on how quickly he g adien is changing. Adam is an imp o emen o RMSP op
ha upda es he lea ning a es based on he a e age bu also on he g adien s a iance,
his is he one we use o aining ou model. Figu e 5.9, ob ained om Google Images,
shows how quickly hese algo i hms educe he cos unc ion.
As men ioned be o e, ou models ha e been ained using he Adam op imize .
Chap e 6
E alua ion and esul s
Once we ha e ou ou models de ined and ained, we can e alua e hem on he es se
and we can use hem o gene a e desc ip ions o new unseen images. In his chap e ,
we explain how we ha e es ed he models, we de ine he me ics ha we ha e used and
we show he esul s ob ained, including some examples o images and hei gene a ed
desc ip ions.
In Sec ion 5.4 we explained ha , as ou objec i e is o de e mine how well ou models
pe o m on desc ibing images ha con ain people, he di ision o he da ase was made
so ha he es se has only his kind o images. The e o e, he esul s o he e alua ion
p ocess and he examples ha we show in his chap e e e all o images con aining
people.
6.1 Tenso Boa d isualiza ion
As desc ibed in Sec ion 5.1, he ou models ha we ha e de eloped ollow he same
a chi ec u e bu di e on he echniques and da ase s used. The cha ac e is ics o hese
ou models a e he ollowing:
•Model 1: he mos gene al one. Uses a p e- ained VGG-16 CNN and i is ained
on MS COCO da ase .
•Model 2: uses a ine- uned e sion o VGG-16 and i is ained on MS COCO da ase .
•Model 3: uses a p e- ained VGG-16 CNN and i is ained on a subse o MS COCO
da ase o images con aining people.
•Model 4: uses a ine- uned e sion o VGG-16 and i is ained on a subse o MS
COCO da ase o images con aining people.
45
52 CHAPTER 6. EVALUATION AND RESULTS
Model 1: pe son in he snow holding snowboa d down snow
Model 3: man on skis in he snow
Figu e 6.8: Good desc ip ions.
Model 1: woman on beach wi h su boa d
Model 3: wo woman iding su boa d on he beach
Figu e 6.9: Good desc ip ions.

6.3. QUALITATIVE ANALYSIS 53
Model 1: man in baseball uni o m is holding ba
Model 3: baseball playe is swinging ba a baseball game
Figu e 6.10: Good desc ip ions.
Model 1: bi d lying on beach
Model 3: man is iding su boa d on he beach
Figu e 6.11: Di e en pe spec i es.
54 CHAPTER 6. EVALUATION AND RESULTS
Model 1: pizza wi h cheese and cheese on i
Model 3: pe son is si ing a able wi h pizza
Figu e 6.12: Di e en pe spec i es.
Model 1: pla e o ood wi h mea and ege ables on i
Model 3: pe son is si ing on able wi h some ood
Figu e 6.13: Di e en pe spec i es.
6.3. QUALITATIVE ANALYSIS 55
Model 1: pe son in he snow on skis on he snow
Model 3: man iding snowboa d down snow co e ed slope
Figu e 6.14: Bad desc ip ions.
Model 1: man holding ho dog in on o pizza
Model 3: man is si ing a able wi h pizza
Figu e 6.15: Bad desc ip ions.
56 CHAPTER 6. EVALUATION AND RESULTS
Chap e 7
Conclusions and u u e wo k
7.1 Conclusions
We conclude his p ojec ha ing accomplished he main objec i es es ablished a he
beginning. We ha e s udied and lea ned di e en Deep Lea ning echniques and ools,
ha a e nowadays e y impo an in he A i icial In elligence ield. We ha e also e iewed
all he bibliog aphy ega ding he image desc ip ion gene a ion p oblem.
We ha e ca ied ou his p ojec in a se e wi h a GPU p o ided by GAIA esea ch
g oup, as aining Deep Lea ning models needs a lo o compu ing capaci y. We ha e
used Ke as on op o Tenso low as he Deep Lea ning amewo k and Tenso Boa d o
gene a ing plo s o he aining p ocess. We ha e also analyzed he di e en a ailable
and well-known image da ase s in o de o selec he ones ha we e mo e sui able o ou
p ojec .
Finally, we ha e achie ed o de elop ou di e en Deep Lea ning models o au oma -
ically gene a e desc ip ions o images con aining people. Ou ou models we e all buil
using s a e-o - he-a Deep Lea ning echniques o Compu e Vision and o Na u al Lan-
guage Gene a ion: Con olu ional Neu al Ne wo ks o ex ac ing he image ea u es and
Recu en Neu al Ne wo ks o gene a ing he sen ence, as well as Feed o wa d Neu al
Ne wo ks o building he models’ ou pu s. These models we e success ully ained and
es ed on a big and popula image da ase .
To e alua e ou models we ha e analyzed he Tenso Boa d plo s gene a ed om he
aining da a, we ha e pe o med a quan i a i e analysis using he mos popula me ics
and a quali a i e analysis gene a ing desc ip ions o unseen images in o de o isualize
and con i m he esul s ob ained om he quan i a i e analysis.
These e alua ion analyses lead all o he same conclusions: we ob ained desi able
esul s om he e alua ion p ocess, simila o s a e-o - he-a esul s, o wo o he ou
models de eloped (Model 1 and Model 3). Howe e , ine- uning was no a good p ac ice,
as Model 2 and Model 4 ( he ones ha ine- uned he CNN) ha e poo e alua ion esul s,
57

58 CHAPTER 7. CONCLUSIONS AND FUTURE WORK
maybe because o he di e ences be ween he da ase s used o ine- uning and aining.
Ne e heless, Model 3 ha e sligh ly be e esul s han Model 1 so we can con i m ha
ou hough s we e in he igh pa h a he beginning o he p ojec : images con aining
people a e desc ibed di e en ly and so i is a good p ac ice o es ic he aining p ocess
o his kind o images.
7.2 Fu u e wo k
Conside ing concluded he main objec i es o his p ojec , he e a e many di ec ions in
which u u e wo k could go in o de o b oaden i . Fi s o all, da a quan i y and quali y
a e essen ial o a good aining, and hence he model could be mo e accu a e and ha e
be e e alua ion esul s wi h mo e, and mo e ep esen a i e, examples in he da ase and
labeled mo e p ecisely.
Ano he op ion is o imp o e ou ha dwa e capaci y by using mo e GPUs, a be e
one o by ano he way o ob aining mo e compu a ional powe in o de o being able o
ain he models mo e apidly.
In addi ion o all o hese changes o accomplish be e esul s wi h ou models, ano he
di ec ion we could ake in he u u e is o b oaden he scope o he model’s objec i e. We
could include mo e comple e image desc ip ions depic ing mo e complex cha ac e is ics o
he pe son in he image, such as he pe son’s eelings (e.g. i i is smiling o sad), o o he
cha ac e is ics ha a e no explici ly shown in he pic u e bu ha can be de i ed om i ,
no limi ing ou sel es o desc ibing wha i is s ic ly in he pic u e. Mo eo e , he model
could be ex ended o he desc ip ion o , no only people, bu o he objec s appea ing in
he pic u e, as well as he spa ial ela ionship be ween hose objec s and he people (e.g.
he man behind he window, he woman nex o he blue chai ).
The e a e also some eme ging asks ha combine bo h compu e ision and na u al
language ields, ying o go beyond wha i is s ic ly depic ed in he image, ha could be
in e es ing o s udy in a u u e wo k o his p ojec ; o example, he ask o VQA (Visual
Ques ion Answe ing) [Wu e al., 2017a], ha ies o ind an answe o a gi en ques ion
abou a gi en image, o isual s o y elling [Huang e al., 2016, Mos a azadeh e al., 2017],
ha gene a es a na a i e desc ip ion om he inpu image by making subjec i e assump-
ions o wha i is happening.
Ano he in e es ing possibili y is o implemen a machine lea ning model, using s a e-
o - he-a machine lea ning algo i hms such as xgboos , in o de o analyse he di e ences
be ween bo h deep lea ning and machine lea ning models (compu ing ime, da ase needs,
accu acy, e c).
Finally, al hough Con olu ional Neu al Ne wo ks a e he s a e-o - he-a , hey ha e
some limi a ions ha dec ease he le el o accu acy o he model. In he las mon hs,
a new kind o neu al ne wo ks ying o sol e hese limi a ions, called Capsule Neu al
Ne wo ks, ha e a isen. I would be a good u u e di ec ion o his p ojec o implemen
7.2. FUTURE WORK 59
he solu ion using his kind o ne wo k and o compa e i o he con olu ional one o see
i i yields be e esul s.
7.2.1 CNN limi a ions
Con olu ional Neu al Ne wo ks a e un il now he s a e-o - he-a app oach in he image
analysis ield, bu hey ha e some limi a ions when i comes o spa ial ela ionships.
Con olu ional Neu al Ne wo ks became a majo ad an age compa ed o adi ional
eed o wa d ne wo ks in he image ield, as hey allowed a 2-D ma ix as he inpu o
he model in con as wi h he need o la ening he pixel ma ix in o a ec o , which,
along wi h he con olu ion and pooling ope a ions, pe mi ed he model o be in a iance
o ansla ions.
As we ha e al eady discussed du ing his p ojec , he job o he con olu ional laye s
in a CNN is o de ec impo an ea u es in he image, wi h laye s close o he inpu
de ec ing simple ea u es and deepe laye s close o he ou pu combining hem o de ec
mo e complex ea u es. On he o he hand, pooling laye s, being max pooling he mos
widely used, help educing he image dimensionali y, and he e o e compu a ional ime,
and summa izing he impo an in o ma ion p esen in he image, as well as c ea ing
spa ial in a iance.
Howe e , his in a iance c ea ed by he max pooling laye s loses he exis ing spa ial
ela ionships be ween all hese ea u es. I does no ake in o accoun how he di e en
ea u es a e ela ed o each o he , since max pooling loses hei p ecise loca ions. This
makes he model ou pu a alse posi i e when he image has he componen s o an objec
bu no in he co ec o de . Fo example, conside ing i is a ace when i has wo eyes, a
nose and a mou h, bu he mou h is whe e an eye should be and ice e sa.
Fu he mo e, CNNs do no ake in o accoun he spa ial cha ac e is ics o each ea u e.
Tha is, hey do no ecognise an objec hey ha e al eady seen, shown now in a di e en
o ien a ion. Con inuing wi h he example abo e, an image wi h a ace u ned upside down
is no de ec ed as a ace by he CNN. To comba his, con olu ional models a e ained
by gi ing images o di e en possible angles explici ly o he ne wo k.
Ne e heless, he solu ion is no o emo e he max pooling laye s om he model,
as we eally need o in oduce some kind o in a iance; o he wise, he model will only
ecognise images e y simila o he ones in he aining se . Wha a be e model needs is
o adjus his in a iance wi h equi a iance, ha is, unde s anding o a ion and p opo ion
changes and adap ing acco dingly.
7.2.2 Capsule neu al ne wo ks
Geo ey E. Hin on, known as he a he o Deep Lea ning, has been discussing abou
con olu ional ne wo ks’ limi a ions and he need o sugges new models o co e hem, o
60 CHAPTER 7. CONCLUSIONS AND FUTURE WORK
Figu e 7.1: Capsule neu al ne wo k s uc u e (CapsNe ).
a long ime now. In 2011 he published a pape [Hin on e al., 2011] ega ding a p oposal o
including mo e complex spa ial cha ac e is ics o neu al ne wo ks, bu i has no been un il
ecen ly ha he has inally published wo pape s [Hin on e al., 2018, Sabou e al., 2017]
explaining his new sophis ica ed neu al ne wo k model, Capsule Ne wo ks.
In he pape hey ha e published [Sabou e al., 2017], hey p opose CapsNe , a capsule
neu al ne wo k model o he MNIST1da ase , epo ing be e esul s han CNNs and
wi hou he need o da a augmen a ion. This model is ep esen ed in Figu e 7.1.
Hin on belie es ha i is necessa y o p ese e pose’s ( ansla ion and o a ion) hie a -
chy be ween ea u es. This way i is easie o a model o ecognise ano he pe spec i e o
some hing i has al eady seen. These neu al ne wo ks y o o e come hose Con olu ional
Ne wo ks’ limi a ions by explici ly aking in o accoun hose spa ial ela ionships be ween
ea u es, whe eas CNNs do no ha e his 3-D space unde s anding.
This is achie ed by eplacing he scala ou pu s o adi ional neu ons wi h ec o
ou pu s encoding he ea u es’ loca ions and changing max-pooling wi h a dynamic ou ing
algo i hm. Thus, an objec p esence is de i ed om no only he p esence o i s cons i uen
pa s, bu also om hem being a he igh loca ions.
Acco ding o Hin on, A capsule is a g oup o neu ons whose ou pu s ep esen di e en
p ope ies o he same en i y. They a e nes ed laye s wi hin a laye , each one ocusing
on de ec ing a pa icula ea u e in he image and ou pu ing a ec o ep esen ing he
ea u e’s exis ence and i s pose p ope ies.
The leng h o he ec o ep esen s he p obabili y o he exis ence o he ea u e and
he spa ial p ope ies a e encoded in he ec o ’s di ec ion. This way, when he de ec ed
ea u e mo es o changes i s spa ial s a e, he leng h o he ec o does no change ( he
p obabili y emains he same) bu i changes i s o ien a ion.
In hese neu al ne wo ks, capsules in a lowe laye decide dynamically how o send
i s ou pu ec o o he nex laye ’s capsules. Each capsule in he lowe laye compu es,
o each possible pa en , a p edic ion ec o o he pose o a highe -le el capsule ea u e
1Da ase o handw i en digi s.
7.2. FUTURE WORK 61
(wha he highe capsule would see), by mul iplying i s ou pu by a weigh ma ix. When
se e al capsules in one laye ag ee on wha hey may ha e de ec ed, hey ac i a e he
co esponding capsule a he nex laye .
Hin on calls his me hod he ou ing-by-ag eemen algo i hm, and i subs i u es he
max-pooling algo i hm.
68 CHAPTER 8. CONCLUSIONES Y TRABAJO A FUTURO

Bibliog aphy
[DBL, 2015] (2015). IEEE Con e ence on Compu e Vision and Pa e n Recogni ion,
CVPR 2015, Bos on, MA, USA, June 7-12, 2015. IEEE Compu e Socie y.
[Abadi e al., 2016] Abadi, M., Ba ham, P., Chen, J., Chen, Z., Da is, A., Dean, J., De in,
M., Ghemawa , S., I ing, G., Isa d, M., Kudlu , M., Le enbe g, J., Monga, R., Moo e,
S., Mu ay, D. G., S eine , B., Tucke , P. A., Vasude an, V., Wa den, P., Wicke, M.,
Yu, Y., and Zhang, X. (2016). Tenso low: A sys em o la ge-scale machine lea ning.
CoRR, abs/1605.08695.
[Al man, 1992] Al man, N. S. (1992). An in oduc ion o ke nel and nea es -neighbo
nonpa ame ic eg ession. The Ame ican S a is ician, 46(3):175–185.
[Baik and Bala, 2004] Baik, S. and Bala, J. W. (2004). A decision ee algo i hm o
dis ibu ed da a mining: Towa ds ne wo k in usion de ec ion. In Lagan`a, A., Ga ilo a,
M. L., Kuma , V., Mun, Y., Tan, C. J. K., and Ge asi, O., edi o s, Compu a ional
Science and I s Applica ions - ICCSA 2004, In e na ional Con e ence, Assisi, I aly,
May 14-17, 2004, P oceedings, Pa IV, olume 3046 o Lec u e No es in Compu e
Science, pages 206–212. Sp inge .
[Bay e al., 2006] Bay, H., Tuy elaa s, T., and Gool, L. J. V. (2006). SURF: speeded up
obus ea u es. In Leona dis, A., Bischo , H., and Pinz, A., edi o s, Compu e Vision -
ECCV 2006, 9 h Eu opean Con e ence on Compu e Vision, G az, Aus ia, May 7-13,
2006, P oceedings, Pa I, olume 3951 o Lec u e No es in Compu e Science, pages
404–417. Sp inge .
[Bengio e al., 1994] Bengio, Y., Sima d, P. Y., and F asconi, P. (1994). Lea ning long-
e m dependencies wi h g adien descen is di icul . IEEE T ans. Neu al Ne wo ks,
5(2):157–166.
[Be na di e al., 2016] Be na di, R., C¸akici, R., Ellio , D., E dem, A., E dem, E., Ikizle -
Cinbis, N., Kelle , F., Musca , A., and Plank, B. (2016). Au oma ic desc ip ion gen-
e a ion om images: A su ey o models, da ase s, and e alua ion measu es. J. A i .
In ell. Res., 55:409–442.
69
70 BIBLIOGRAPHY
[B idge e al., 2014] B idge, J. P., Holden, S. B., and Paulson, L. C. (2014). Machine
lea ning o i s -o de heo em p o ing - lea ning o selec a good heu is ic. J. Au om.
Reasoning, 53(2):141–172.
[Canny, 1986] Canny, J. F. (1986). A compu a ional app oach o edge de ec ion. IEEE
T ans. Pa e n Anal. Mach. In ell., 8(6):679–698.
[Chen e al., 2015] Chen, X., Fang, H., Lin, T., Vedan am, R., Gup a, S., Doll´a , P., and
Zi nick, C. L. (2015). Mic oso COCO cap ions: Da a collec ion and e alua ion se e .
CoRR, abs/1504.00325.
[Chen and Zi nick, 2015] Chen, X. and Zi nick, C. L. (2015). Mind’s eye: A ecu en
isual ep esen a ion o image cap ion gene a ion. In IEEE Con e ence on Compu e
Vision and Pa e n Recogni ion, CVPR 2015, Bos on, MA, USA, June 7-12, 2015,
pages 2422–2431.
[Cho e al., 2014] Cho, K., an Me ienboe , B., G¨ul¸ceh e, C¸ ., Bahdanau, D., Bouga es,
F., Schwenk, H., and Bengio, Y. (2014). Lea ning ph ase ep esen a ions using RNN
encode -decode o s a is ical machine ansla ion. In [Moschi i e al., 2014], pages
1724–1734.
[Chung e al., 2014] Chung, J., G¨ul¸ceh e, C¸ ., Cho, K., and Bengio, Y. (2014). Em-
pi ical e alua ion o ga ed ecu en neu al ne wo ks on sequence modeling. CoRR,
abs/1412.3555.
[Collobe e al., 2011] Collobe , R., Ka ukcuoglu, K., and Fa abe , C. (2011). To ch7:
A ma lab-like en i onmen o machine lea ning.
[Co es and Vapnik, 1995] Co es, C. and Vapnik, V. (1995). Suppo - ec o ne wo ks.
Machine Lea ning, 20(3):273–297.
[Dean e al., 2012] Dean, J., Co ado, G. S., Monga, R., Chen, K., De in, M., Le, Q. V.,
Mao, M. Z., Ranza o, M., Senio , A., Tucke , P., Yang, K., and Ng, A. Y. (2012). La ge
scale dis ibu ed deep ne wo ks. In NIPS.
[Deng e al., 2009] Deng, J., Dong, W., Soche , R., Li, L., Li, K., and Li, F. (2009).
Imagene : A la ge-scale hie a chical image da abase. In 2009 IEEE Compu e Socie y
Con e ence on Compu e Vision and Pa e n Recogni ion (CVPR 2009), 20-25 June
2009, Miami, Flo ida, USA, pages 248–255. IEEE Compu e Socie y.
[Denkowski and La ie, 2014] Denkowski, M. J. and La ie, A. (2014). Me eo uni e sal:
Language speci ic ansla ion e alua ion o any a ge language. In P oceedings o
he Nin h Wo kshop on S a is ical Machine T ansla ion, WMT@ACL 2014, June 26-
27, 2014, Bal imo e, Ma yland, USA, pages 376–380. The Associa ion o Compu e
Linguis ics.
[Donahue e al., 2015] Donahue, J., Hend icks, L. A., Guada ama, S., Roh bach, M.,
Venugopalan, S., Da ell, T., and Saenko, K. (2015). Long- e m ecu en con olu ional
ne wo ks o isual ecogni ion and desc ip ion. In [DBL, 2015], pages 2625–2634.
BIBLIOGRAPHY 71
[Duchi e al., 2011] Duchi, J. C., Hazan, E., and Singe , Y. (2011). Adap i e subg adien
me hods o online lea ning and s ochas ic op imiza ion. Jou nal o Machine Lea ning
Resea ch, 12:2121–2159.
[E e ingham e al., 2010] E e ingham, M., Van Gool, L., Williams, C. K. I., Winn, J., and
Zisse man, A. (2010). The pascal isual objec classes ( oc) challenge. In e na ional
Jou nal o Compu e Vision, 88(2):303–338.
[Goldbe g e al., 1994] Goldbe g, E., D iedge , N., and Ki edge, R. I. (1994). Using
na u al-language p ocessing o p oduce wea he o ecas s. IEEE Expe , 9(2):45–53.
[G a es e al., 2013] G a es, A., Mohamed, A., and Hin on, G. E. (2013). Speech ecogni-
ion wi h deep ecu en neu al ne wo ks. In IEEE In e na ional Con e ence on Acous-
ics, Speech and Signal P ocessing, ICASSP 2013, Vancou e , BC, Canada, May 26-31,
2013, pages 6645–6649. IEEE.
[Ha is and S ephens, 1988] Ha is, C. G. and S ephens, M. (1988). A combined co ne
and edge de ec o . In Taylo , C. J., edi o , P oceedings o he Al ey Vision Con e ence,
AVC 1988, Manches e , UK, Sep embe , 1988, pages 1–6. Al ey Vision Club.
[Hend icks e al., 2016] Hend icks, L. A., Venugopalan, S., Roh bach, M., Mooney, R. J.,
Saenko, K., and Da ell, T. (2016). Deep composi ional cap ioning: Desc ibing no el
objec ca ego ies wi hou pai ed aining da a. In 2016 IEEE Con e ence on Compu e
Vision and Pa e n Recogni ion, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016,
pages 1–10.
[Hin on e al., 2011] Hin on, G. E., K izhe sky, A., and Wang, S. D. (2011). T ans o m-
ing au o-encode s. In Honkela, T., Duch, W., Gi olami, M. A., and Kaski, S., edi o s,
A i icial Neu al Ne wo ks and Machine Lea ning - ICANN 2011 - 21s In e na ional
Con e ence on A i icial Neu al Ne wo ks, Espoo, Finland, June 14-17, 2011, P oceed-
ings, Pa I, olume 6791 o Lec u e No es in Compu e Science, pages 44–51. Sp inge .
[Hin on e al., 2018] Hin on, G. E., Sabou , S., and F oss , N. (2018). Ma ix capsules
wi h EM ou ing. In In e na ional Con e ence on Lea ning Rep esen a ions.
[Ho, 1995] Ho, T. K. (1995). Random decision o es s. In Thi d In e na ional Con e ence
on Documen Analysis and Recogni ion, ICDAR 1995, Augus 14 - 15, 1995, Mon eal,
Canada. Volume I, pages 278–282. IEEE Compu e Socie y.
[Hoch ei e and Schmidhube , 1997] Hoch ei e , S. and Schmidhube , J. (1997). Long
sho - e m memo y. Neu al Compu a ion, 9(8):1735–1780.
[Hodosh e al., 2013] Hodosh, M., Young, P., and Hockenmaie , J. (2013). F aming image
desc ip ion as a anking ask: Da a, models and e alua ion me ics. J. A i . In ell.
Res., 47:853–899.
[Ho elling, 1933] Ho elling, H. (1933). Analysis o a complex o s a is ical a iables wi h
p incipal componen s. Jou nal o Educa ional Psychology, 24:417–441.
72 BIBLIOGRAPHY
[Huang e al., 2016] Huang, T. K., Fe a o, F., Mos a azadeh, N., Mis a, I., Ag awal, A.,
De lin, J., Gi shick, R. B., He, X., Kohli, P., Ba a, D., Zi nick, C. L., Pa ikh, D.,
Vande wende, L., Galley, M., and Mi chell, M. (2016). Visual s o y elling. In Knigh ,
K., Nenko a, A., and Rambow, O., edi o s, NAACL HLT 2016, The 2016 Con e ence o
he No h Ame ican Chap e o he Associa ion o Compu a ional Linguis ics: Human
Language Technologies, San Diego Cali o nia, USA, June 12-17, 2016, pages 1233–1239.
The Associa ion o Compu a ional Linguis ics.
[Io danskaja e al., 1992] Io danskaja, L., Kim, M., Ki edge, R. I., La oie, B., and
Polgu`e e, A. (1992). Gene a ion o ex ended bilingual s a is ical epo s. In 14 h In-
e na ional Con e ence on Compu a ional Linguis ics, COLING 1992, Nan es, F ance,
Augus 23-28, 1992, pages 1019–1023.
[Jia e al., 2014] Jia, Y., Shelhame , E., Donahue, J., Ka aye , S., Long, J., Gi shick,
R., Guada ama, S., and Da ell, T. (2014). Ca e: Con olu ional a chi ec u e o as
ea u e embedding. a Xi p ep in a Xi :1408.5093.
[Ka pa hy and Li, 2015] Ka pa hy, A. and Li, F. (2015). Deep isual-seman ic alignmen s
o gene a ing image desc ip ions. In [DBL, 2015], pages 3128–3137.
[Kingma and Ba, 2014] Kingma, D. P. and Ba, J. (2014). Adam: A me hod o s ochas ic
op imiza ion. CoRR, abs/1412.6980.
[Ki os e al., 2014a] Ki os, R., Salakhu dino , R., and Zemel, R. S. (2014a). Mul imodal
neu al language models. In P oceedings o he 31 h In e na ional Con e ence on Machine
Lea ning, ICML 2014, Beijing, China, 21-26 June 2014, pages 595–603.
[Ki os e al., 2014b] Ki os, R., Salakhu dino , R., and Zemel, R. S. (2014b). Uni y-
ing isual-seman ic embeddings wi h mul imodal neu al language models. CoRR,
abs/1411.2539.
[K izhe sky, 2012] K izhe sky, A. (2012). Lea ning mul iple laye s o ea u es om iny
images.
[K izhe sky e al., 2012] K izhe sky, A., Su ske e , I., and Hin on, G. E. (2012). Ima-
gene classi ica ion wi h deep con olu ional neu al ne wo ks. In Ba le , P. L., Pe ei a,
F. C. N., Bu ges, C. J. C., Bo ou, L., and Weinbe ge , K. Q., edi o s, Ad ances in
Neu al In o ma ion P ocessing Sys ems 25: 26 h Annual Con e ence on Neu al In o -
ma ion P ocessing Sys ems 2012. P oceedings o a mee ing held Decembe 3-6, 2012,
Lake Tahoe, Ne ada, Uni ed S a es., pages 1106–1114.
[Kulka ni e al., 2013] Kulka ni, G., P em aj, V., O donez, V., Dha , S., Li, S., Choi, Y.,
Be g, A. C., and Be g, T. L. (2013). Baby alk: Unde s anding and gene a ing simple
image desc ip ions. IEEE T ans. Pa e n Anal. Mach. In ell., 35(12):2891–2903.
[Kuzne so a e al., 2012] Kuzne so a, P., O donez, V., Be g, A. C., Be g, T. L., and
Choi, Y. (2012). Collec i e gene a ion o na u al image desc ip ions. In The 50 h
BIBLIOGRAPHY 73
Annual Mee ing o he Associa ion o Compu a ional Linguis ics, P oceedings o he
Con e ence, July 8-14, 2012, Jeju Island, Ko ea - Volume 1: Long Pape s, pages 359–
368. The Associa ion o Compu e Linguis ics.
[LeCun e al., 2015] LeCun, Y., Bengio, Y., and Hin on, G. E. (2015). Deep lea ning.
Na u e, 521(7553):436–444.
[Li e al., 2011] Li, S., Kulka ni, G., Be g, T. L., Be g, A. C., and Choi, Y. (2011). Com-
posing simple image desc ip ions using web-scale n-g ams. In Goldwa e , S. and Man-
ning, C. D., edi o s, P oceedings o he Fi een h Con e ence on Compu a ional Na u al
Language Lea ning, CoNLL 2011, Po land, O egon, USA, June 23-24, 2011, pages
220–228. ACL.
[Lin, 2004] Lin, C.-Y. (2004). Rouge: a package o au oma ic e alua ion o summa ies.
[Lin e al., 2014] Lin, T., Mai e, M., Belongie, S. J., Bou de , L. D., Gi shick, R. B.,
Hays, J., Pe ona, P., Ramanan, D., Doll´a , P., and Zi nick, C. L. (2014). Mic oso
COCO: common objec s in con ex . CoRR, abs/1405.0312.
[Lowe, 1999] Lowe, D. G. (1999). Objec ecogni ion om local scale-in a ian ea u es.
In ICCV, pages 1150–1157.
[Mao e al., 2014] Mao, J., Xu, W., Yang, Y., Wang, J., and Yuille, A. L. (2014). Deep
cap ioning wi h mul imodal ecu en neu al ne wo ks (m- nn). CoRR, abs/1412.6632.
[Mason and Cha niak, 2014] Mason, R. and Cha niak, E. (2014). Nonpa ame ic me hod
o da a-d i en image cap ioning. In P oceedings o he 52nd Annual Mee ing o he
Associa ion o Compu a ional Linguis ics, ACL 2014, June 22-27, 2014, Bal imo e,
MD, USA, Volume 2: Sho Pape s, pages 592–598. The Associa ion o Compu e
Linguis ics.
[Mille , 1995] Mille , G. A. (1995). Wo dne : A lexical da abase o english. Commun.
ACM, 38(11):39–41.
[Mi chell e al., 2012] Mi chell, M., Dodge, J., Goyal, A., Yamaguchi, K., S a os, K.,
Han, X., Mensch, A., Be g, A. C., Be g, T. L., and III, H. D. (2012). Midge: Gene a ing
image desc ip ions om compu e ision de ec ions. In Daelemans, W., Lapa a, M.,
and M`a quez, L., edi o s, EACL 2012, 13 h Con e ence o he Eu opean Chap e o he
Associa ion o Compu a ional Linguis ics, A ignon, F ance, Ap il 23-27, 2012, pages
747–756. The Associa ion o Compu e Linguis ics.
[Moschi i e al., 2014] Moschi i, A., Pang, B., and Daelemans, W., edi o s (2014). P o-
ceedings o he 2014 Con e ence on Empi ical Me hods in Na u al Language P ocessing,
EMNLP 2014, Oc obe 25-29, 2014, Doha, Qa a , A mee ing o SIGDAT, a Special
In e es G oup o he ACL. ACL.
[Mos a azadeh e al., 2017] Mos a azadeh, N., B ocke , C., Dolan, B., Galley, M., Gao,
J., Spi hou akis, G. P., and Vande wende, L. (2017). Image-g ounded con e sa ions:

74 BIBLIOGRAPHY
Mul imodal con ex o na u al ques ion and esponse gene a ion. In Kond ak, G. and
Wa anabe, T., edi o s, P oceedings o he Eigh h In e na ional Join Con e ence on
Na u al Language P ocessing, IJCNLP 2017, Taipei, Taiwan, No embe 27 - Decembe
1, 2017 - Volume 1: Long Pape s, pages 462–472. Asian Fede a ion o Na u al Language
P ocessing.
[Moze , 1989] Moze , M. C. (1989). A ocused backp opaga ion algo i hm o empo al
pa e n ecogni ion. Complex Sys ems, 3(4).
[O donez e al., 2011] O donez, V., Kulka ni, G., and Be g, T. L. (2011). Im2 ex : De-
sc ibing images using 1 million cap ioned pho og aphs. In Shawe-Taylo , J., Zemel,
R. S., Ba le , P. L., Pe ei a, F. C. N., and Weinbe ge , K. Q., edi o s, Ad ances in
Neu al In o ma ion P ocessing Sys ems 24: 25 h Annual Con e ence on Neu al In o -
ma ion P ocessing Sys ems 2011. P oceedings o a mee ing held 12-14 Decembe 2011,
G anada, Spain., pages 1143–1151.
[Pang e al., 2002] Pang, B., Lee, L., and Vai hyana han, S. (2002). Thumbs up?: Sen-
imen classi ica ion using machine lea ning echniques. In P oceedings o he ACL-02
Con e ence on Empi ical Me hods in Na u al Language P ocessing - Volume 10, EMNLP
’02, pages 79–86, S oudsbu g, PA, USA. Associa ion o Compu a ional Linguis ics.
[Papineni e al., 2002] Papineni, K., Roukos, S., Wa d, T., and Zhu, W. (2002). Bleu: a
me hod o au oma ic e alua ion o machine ansla ion. In P oceedings o he 40 h
Annual Mee ing o he Associa ion o Compu a ional Linguis ics, July 6-12, 2002,
Philadelphia, PA, USA., pages 311–318. ACL.
[Penning on e al., 2014] Penning on, J., Soche , R., and Manning, C. D. (2014). Glo e:
Global ec o s o wo d ep esen a ion. In [Moschi i e al., 2014], pages 1532–1543.
[Rei e and Dale, 1997] Rei e , E. and Dale, R. (1997). Building applied na u al language
gene a ion sys ems. Na u al Language Enginee ing, 3(1):57–87.
[Russako sky e al., 2015] Russako sky, O., Deng, J., Su, H., K ause, J., Sa heesh, S.,
Ma, S., Huang, Z., Ka pa hy, A., Khosla, A., Be ns ein, M. S., Be g, A. C., and Li,
F. (2015). Imagene la ge scale isual ecogni ion challenge. In e na ional Jou nal o
Compu e Vision, 115(3):211–252.
[Russell e al., 2008] Russell, B. C., To alba, A., Mu phy, K. P., and F eeman, W. T.
(2008). Labelme: A da abase and web-based ool o image anno a ion. In e na ional
Jou nal o Compu e Vision, 77(1-3):157–173.
[Sabou e al., 2017] Sabou , S., F oss , N., and Hin on, G. E. (2017). Dynamic ou ing
be ween capsules. In Guyon, I., on Luxbu g, U., Bengio, S., Wallach, H. M., Fe gus,
R., Vishwana han, S. V. N., and Ga ne , R., edi o s, Ad ances in Neu al In o ma ion
P ocessing Sys ems 30: Annual Con e ence on Neu al In o ma ion P ocessing Sys ems
2017, 4-9 Decembe 2017, Long Beach, CA, USA, pages 3859–3869.
BIBLIOGRAPHY 75
[Sa ikaya e al., 2014] Sa ikaya, R., Hin on, G. E., and Deo as, A. (2014). Applica ion o
deep belie ne wo ks o na u al language unde s anding. IEEE/ACM T ans. Audio,
Speech & Language P ocessing, 22(4):778–784.
[Seide and Aga wal, 2016] Seide, F. and Aga wal, A. (2016). Cn k: Mic oso ’s open-
sou ce deep-lea ning oolki . In P oceedings o he 22Nd ACM SIGKDD In e na ional
Con e ence on Knowledge Disco e y and Da a Mining, KDD ’16, pages 2135–2135, New
Yo k, NY, USA. ACM.
[S i as a a e al., 2014] S i as a a, N., Hin on, G. E., K izhe sky, A., Su ske e , I., and
Salakhu dino , R. (2014). D opou : a simple way o p e en neu al ne wo ks om
o e i ing. Jou nal o Machine Lea ning Resea ch, 15(1):1929–1958.
[Tan i e al., 2017a] Tan i, M., Ga , A., and Camille i, K. P. (2017a). Wha is he ole o
ecu en neu al ne wo ks ( nns) in an image cap ion gene a o ? In P oceedings o he
10 h In e na ional Con e ence on Na u al Language Gene a ion, INLG 2017, San iago
de Compos ela, Spain, Sep embe 4-7, 2017, pages 51–60.
[Tan i e al., 2017b] Tan i, M., Ga , A., and Camille i, K. P. (2017b). Whe e o pu he
image in an image cap ion gene a o . CoRR, abs/1703.09137.
[Theano De elopmen Team, 2016] Theano De elopmen Team (2016). Theano: A
Py hon amewo k o as compu a ion o ma hema ical exp essions. a Xi e-p in s,
abs/1605.02688.
[T an e al., 2016] T an, K., He, X., Zhang, L., and Sun, J. (2016). Rich image cap ioning
in he wild. In 2016 IEEE Con e ence on Compu e Vision and Pa e n Recogni ion
Wo kshops, CVPR Wo kshops 2016, Las Vegas, NV, USA, June 26 - July 1, 2016,
pages 434–441. IEEE Compu e Socie y.
[Vedan am e al., 2014] Vedan am, R., Zi nick, C. L., and Pa ikh, D. (2014). Cide :
Consensus-based image desc ip ion e alua ion. CoRR, abs/1411.5726.
[Vinyals e al., 2015] Vinyals, O., Toshe , A., Bengio, S., and E han, D. (2015). Show
and ell: A neu al image cap ion gene a o . In [DBL, 2015], pages 3156–3164.
[We nick e al., 2010] We nick, M. N., Yang, Y., B anko , J. G., You gano , G., and
S o he , S. C. (2010). Machine lea ning in medical imaging. IEEE Signal P ocessing
Magazine, 27(4):25–38.
[Wu e al., 2017a] Wu, Q., Teney, D., Wang, P., Shen, C., Dick, A. R., and an den
Hengel, A. (2017a). Visual ques ion answe ing: A su ey o me hods and da ase s.
Compu e Vision and Image Unde s anding, 163:21–40.
[Wu e al., 2017b] Wu, S., Wieland, J., Fa i a , O., and Schille , J. (2017b). Au oma ic
al - ex : Compu e -gene a ed image desc ip ions o blind use s on a social ne wo k
se ice. In P oceedings o he 2017 ACM Con e ence on Compu e Suppo ed Coope a i e
Wo k and Social Compu ing, CSCW ’17, pages 1180–1192, New Yo k, NY, USA. ACM.
76 BIBLIOGRAPHY
[Wu e al., 2016] Wu, Y., Schus e , M., Chen, Z., Le, Q. V., No ouzi, M., Mache ey, W.,
K ikun, M., Cao, Y., Gao, Q., Mache ey, K., Klingne , J., Shah, A., Johnson, M., Liu,
X., Kaise , L., Gouws, S., Ka o, Y., Kudo, T., Kazawa, H., S e ens, K., Ku ian, G.,
Pa il, N., Wang, W., Young, C., Smi h, J., Riesa, J., Rudnick, A., Vinyals, O., Co ado,
G., Hughes, M., and Dean, J. (2016). Google’s neu al machine ansla ion sys em:
B idging he gap be ween human and machine ansla ion. CoRR, abs/1609.08144.
[Yang e al., 2011] Yang, Y., Teo, C. L., III, H. D., and Aloimonos, Y. (2011). Co pus-
guided sen ence gene a ion o na u al images. In P oceedings o he 2011 Con e ence
on Empi ical Me hods in Na u al Language P ocessing, EMNLP 2011, 27-31 July 2011,
John McIn y e Con e ence Cen e, Edinbu gh, UK, A mee ing o SIGDAT, a Special
In e es G oup o he ACL, pages 444–454. ACL.
[Young e al., 2014] Young, P., Lai, A., Hodosh, M., and Hockenmaie , J. (2014). F om
image desc ip ions o isual deno a ions: New simila i y me ics o seman ic in e ence
o e e en desc ip ions. TACL, 2:67–78.