A deep learning approach for automatically generating descriptions of images containing people

A deep lea ning app oach o au oma ically gene a ing desc ip ions o images con aining people T abajo de Fin de G ado Cu so 2017–2018 Au o Ma a A acil Mu˜noz Di ec o es Gonzalo M´endez Pozo Raquel He ´as Balles e os G ado en Ingenie ´ıa In o m´a ica Facul ad de In o m´a ica Uni e sidad Complu ense de Mad id A deep lea ning app oach o au oma ically gene a ing desc ip ions o images con aining people T abajo de Fin de G ado en Ingenie ´ıa In o m´a ica Depa amen o de Ingenie ´ıa del So wa e e In eligencia A i icial Au o Ma a A acil Mu˜noz Di ec o es Gonzalo M´endez Pozo Raquel He ´as Balles e os Con oca o ia: Sep iemb e 2018 G ado en Ingenie ´ıa In o m´a ica Facul ad de In o m´a ica Uni e sidad Complu ense de Mad id Abs ac Gene a ing image desc ip ions is a challenging A i icial In elligence p oblem wi h many in e es ing applica ions such as obo s’ communica ion o helping isually impai ed people. Howe e , i is a complex ask o compu e s: i equi es Compu e Vision algo i hms, o unde s and wha he image depic s, and Na u al Language P ocessing algo i hms, o gene a e a well- o med sen ence. Nowadays, deep neu al ne wo ks a e he s a e-o - he-a in hese wo A i icial In elligence ields. Fu he mo e, we belie e ha images ha con ain people a e desc ibed in a sligh ly di e en manne and ha es ic ing an image desc ip ion gene a o model o hese images may p oduce be e desc ip ions. The e o e, he main objec i e o his p ojec is o de elop a Deep Lea ning model ha au oma ically p oduces desc ip ions o images con aining people and o conclude i i is a good p ac ice he es ic ion o his kind o images. Fo his pu pose, we ha e e iewed and s udied he li e a u e in he ield and we ha e buil , ained and compa ed ou di e en models using Deep Lea ning echniques and a GPU o speed-up he compu a ion, as well as a big and comple e da ase . Keywo ds Deep Lea ning, Compu e Vision, Na u al Language P ocessing, image desc ip ion gene - a ion, Ke as, GPU, da ase . Resumen Gene a desc ipciones de im´agenes es un p oblema de In eligencia A i icial con muchas aplicaciones in e esan es como la comunicaci´on de obo s o ayuda a pe sonas con dis- capacidad isual. Sin emba go, es una a ea compleja pa a un o denado : equie e algo- i mos de isi´on po compu ado pa a en ende lo que la imagen ep esen a y algo i mos de p ocesamien o de lenguaje na u al pa a gene a una ase bien o mada. Hoy en d´ıa, las edes neu onales p o undas son el es ado del a e en es os dos campos de la In eligencia A i icial. Po o a pa e, c eemos que las im´agenes que con ienen pe sonas se desc iben de man- e a lige amen e di e en e y que es ingi un modelo de gene aci´on de desc ipciones de im´agenes a im´agenes de es e ipo puede p oduci mejo es desc ipciones. Po lo an o, el p incipal obje i o de es e p oyec o es desa olla un modelo de ap endizaje p o undo que p oduce au om´a icamen e desc ipciones de im´agenes que con ienen pe sonas y conclui si es una buena p ´ac ica la es icci´on a es a clase de im´agenes. Pa a ello, hemos e isado y es udiado la li e a u a y hemos cons uido, en enado y compa ado cua o modelos di e - en es usando ´ecnicas de ap endizaje p o undo y una GPU pa a acele a los c´alculos, as´ı como un da ase g ande y comple o. Palab as cla e Ap endizaje p o undo, isi´on po compu ado , p ocesamien o de lenguaje na u al, gen- e aci´on de desc ipciones de im´agenes, Ke as, GPU, da ase . ii Con en s 1 In oduc ion and objec i es 1 1.1 Objec i es..................................... 2 1.2 Documen s uc u e ............................... 3 2 In oducci´on y obje i os 5 2.1 Obje i os ..................................... 6 2.2 Es uc u a del documen o . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 3 S a e o he a 9 3.1 T adi ional algo i hms o image analysis . . . . . . . . . . . . . . . . . . . 10 3.2 T adi ional app oach o NLG . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.3 MachineLea ning................................. 11 3.4 DeepLea ning .................................. 13 3.4.1 Feed o wa d Neu al Ne wo ks . . . . . . . . . . . . . . . . . . . . . . 14 3.4.2 Con olu ional Neu al Ne wo ks . . . . . . . . . . . . . . . . . . . . . 15 3.4.3 Recu en Neu al Ne wo ks . . . . . . . . . . . . . . . . . . . . . . . 17 3.5 Image desc ip ion gene a o s . . . . . . . . . . . . . . . . . . . . . . . . . . 18 4 F amewo k 21 4.1 GPU........................................ 21 4.2 Deep Lea ning amewo k . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 4.3 Visualiza ion: Tenso Boa d . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 4.4 Imageda ase s .................................. 25 4.4.1 Da ase elec ion ............................. 28 ix 4CHAPTER 1. INTRODUCTION AND OBJECTIVES Chap e 2 In oducci´on y obje i os En nues a condici´on de humanos, cons an e e inconscien emen e hacemos desc ipciones del mundo que nos odea pa a comunica nos con la gen e. Desc ibimos lo que emos en muchas si uaciones di e en es, como po ejemplo cuando damos indicaciones a alguien sob e c´omo i a alg´un si io (sigue ec o has a que eas el edi icio ojo y despu´es gi a a la de echa a la calle que iene los ´a boles g andes y una uen e), cuando le con amos una his o ia a un amigo (es aba de comp as cuando me encon ´e a Sa a de la mano de un se˜no mayo al o) o cuando subimos una o o a In e ne (¡Pas´andomelo genial en la playa con mi es ido azul nue o!). Mien as que es as desc ipciones las hacemos de mane a sencilla y na u al, sin ene que pensa lo demasiado, gene a au om´a icamen e desc ipciones de lo que hay en una imagen es una a ea muy compleja y exigen e pa a un o denado . Gene a una buena desc ipci´on de una imagen equie e p ime o analiza y en ende lo que apa ece en ella y despu´es gene a una ase en lenguaje na u al que explique esos elemen os, es ´e bien o mada y sea concisa. Po lo an o, gene a desc ipciones de im´agenes es m´as di ´ıcil que las ´ıpicas a eas del campo de isi´on po compu ado , como son la clasi icaci´on de im´agenes o la de ecci´on de obje os, ya que conlle a no solo ´ecnicas de Visi´on po Compu ado (CV), sino ambi´en P ocesamien o del Lenguaje Na u al (NLP); dos campos de la In eligencia A i icial que po lo gene al han lle ado caminos sepa ados y no se suelen es udia ni aplica en conjun o. La gene aci´on de desc ipciones de im´agenes se puede conside a como un caso pa icu- la de la aducci´on au om´a ica donde, en luga de aduci una ase dada a o o idioma, hay que aduci una imagen de en ada a su desc ipci´on. La imagen de en ada debe se codi icada a un ec o de ep esen aci´on y, despu´es, es a ep esen aci´on debe se decodi i- cada a una ase en lenguaje na u al. Pa a log a es o, podemos combina esul ados del es ado del a e de ambos campos: los esul ados de Visi´on po Compu ado pa a ob ene el ec o de ep esen aci´on y los de NLP (en pa icula , el es ado del a e de la Gene aci´on de Lenguaje Na u al (NLG)) pa a cons ui la desc ipci´on. Es e p oblema an in e esan e de In eligencia A i icial ha despe ado el in e ´es de g andes emp esas como Facebook o Google. El p oblema de desc ibi im´agenes au- 5 6CHAPTER 2. INTRODUCCI ´ ON Y OBJETIVOS om´a icamen e no solo es a ac i o po su in e ´es acad´emico y cien ´ı ico, sino ambi´en po sus muchas aplicaciones. La p incipal y m´as impo an e aplicaci´on es gene a des- c ipciones pa a las pe sonas con alguna discapacidad isual, bien sea pa a ayuda les en su d´ıa a d´ıa (po ejemplo, pa a mo e se po la ciudad o comp a en un supe me cado), pa a ace ca les a la cul u a gene ando desc ipciones de los cuad os de un museo o, como i imos en un mundo cla amen e dominado po o os, pa a hace m´as accesibles las edes sociales, y las p´aginas de In e ne en gene al. Pe o ambi´en iene o as aplicaciones como po ejemplo en ob´o ica pa a la comunicaci´on de los obo s ( ans o mando en ases lo que en e iden i icando con qui´en es ´an hablando) o en medicina pa a desc ibi im´agenes m´edicas. Po o o lado, en los ´ul imos a˜nos hemos podido p esencia un inc e´ıble aumen o del in e ´es en ob ene in o maci´on de los da os y es amos ac ualmen e en lo que llaman la e a da a-d i en (di igida po los da os). Cada d´ıa, se c ean millones de nue os da os, muchas emp esas ienen ecogida una g an can idad de da os y quie en saca les p o echo, descub iendo pa ones ocul os y pe spec i as pa a ob ene buenas decisiones de nego- cio. Debido a es a g an can idad de da os disponibles y al c ecimien o de la capacidad compu acional pa a ealiza c´alculos complejos, las ´ecnicas de ap endizaje au om´a ico y ap endizaje p o undo se han podido desa olla y es udia con i i´endose en el es ado del a e en muchos campos, lo que ha hecho que es ´en en boca de odos. Po lo an o, es e abajo busca auna es os dos concep os an popula es y a ac i os, desa ollando un gene ado de desc ipciones de im´agenes u ilizando ´ecnicas de ap endizaje p o undo. 2.1 Obje i os Respec o a las desc ipciones de im´agenes, Be na di e al. [Be na di e al., 2016] dis inguen dos en oques p incipales pa a su gene aci´on: modelos que gene an desc ipciones nue as pa a una imagen de en ada y modelos que cons uyen la desc ipci´on ellenando plan illas o usando desc ipciones asociadas a im´agenes simila es; noso os cen a emos nues o abajo en los p ime os. Vamos a limi a el alcance de es e abajo a gene a desc ipciones de im´agenes de pe sonas, ya que conside amos que es un en oque in e esan e y pod emos cen a odos los es ue zos en ello. El abajo elacionado en el campo de la gene aci´on au om´a ica de desc ipciones de im´agenes no se suele cen a en un ema en conc e o, a ando de aba ca odas las posibles desc ipciones e im´agenes y no le han dado demasiada a enci´on a las im´agenes de pe sonas. Conside amos que no desc ibimos de la misma mane a a una pe sona que a o os elemen os de una imagen, y po an o, al es ingi nos a im´agenes de pe sonas podemos ob ene mejo es desc ipciones y esul ados. Con odo es o, los obje i os p incipales de es e abajo son: •Re isa y analiza la bibliog a ´ıa y el es ado del a e de las ´ecnicas de gene aci´on 2.2. ESTRUCTURA DEL DOCUMENTO 7 de desc ipciones de im´agenes. •Es udia y en ende los modelos de ap endizaje p o undo y sus aplicaciones. •Es udia los di e en es en o nos de abajo y he amien as en ocadas al ap endizaje p o undo. •Desa olla di e en es modelos que gene en au om´a icamen e desc ipciones de im´a- genes de pe sonas y a a de ep oduci los esul ados del es ado del a e. •E alua y compa a los dis in os modelos c eados y de e mina cu´al es el mejo y si es sa is ac o io cen a los es ue zos y limi a nues os modelos a ca ac e ´ıs icas de pe sonas. 2.2 Es uc u a del documen o La es uc u a de es e documen o se ´a la siguien e: el Cap´ı ulo 3 da una explicaci´on muy de allada del es ado del a e an o en el campo de la Gene aci´on de Lenguaje Na u al como en el de Visi´on po Compu ado , en es e cap´ı ulo ambi´en se de allan algunos gene ado es de desc ipciones de im´agenes. En el Cap´ı ulo 4 in oducimos el en o no de abajo en el que desa olla emos es e abajo; en pa icula , explicamos las GPUs, el en o no cloud compu ing, qu´e lib e ´ıas pa a ap endizaje p o undo y isualizaci´on usa emos y la elecci´on del da ase . El Cap´ı ulo 5 es el p incipal de es e abajo; a paso a paso po el p oceso de de inici´on y en enamien o de los modelos. Explicamos en es e cap´ı ulo odas las ca ac e ´ıs icas y componen es de los modelos y odas las decisiones omadas al de ini los y en ena los. El Cap´ı ulo 6 es ´a dedicado a mos a los esul ados ob enidos, el p oceso de es ing y las m´e icas m´as conocidas. Po ´ul imo, en el Cap´ı ulo 8 p esen amos las conclusiones del abajo y damos posibles l´ıneas de abajo u u o. 8CHAPTER 2. INTRODUCCI ´ ON Y OBJETIVOS Chap e 3 S a e o he a Image analysis has been a challenging p oblem in esea che s’ minds o a long ime now. A i s , his p oblem was ackled by a adi ional compu e ision app oach, wi h explici p ocessing algo i hms. Then, Machine Lea ning models appea ed and hey we e he s a e- o - he-a in his ield un il 2010, when aining complex deep neu al ne wo ks s a ed o become a possibili y hanks o he use o GPUs compu a ional powe and he a ailabili y o mo e da a. As o oday, hese neu al ne wo ks, in pa icula Con olu ional Neu al Ne wo ks, a e he s a e-o - he-a in all image analysis challenges. Simila ly, he ask o Na u al Language Gene a ion (NLG), has been aced by classical s ep-by-s ep algo i hms, as well as Machine Lea ning and Deep Lea ning models. In pa icula , Recu en Neu al Ne wo ks a e, as o oday, he s a e-o - he-a in gene a ing na u al language sen ences and ex s. Image desc ip ion combines bo h compu e ision and na u al language p ocessing ields, and, nowadays, he s a e-o - he-a esul s a e a combina ion o Con olu ional Neu- al Ne wo ks, ha ocus on ex ac ing image ea u es, and Recu en Neu al Ne wo ks, o gene a e he sen ence. In his chap e we will p esen p e ious wo k in he ield o image analysis and Na u al Language Gene a ion, and how hese ields ha e been add essed ac oss ime, om he poin o iew o adi ional and explici algo i hms o he one o Deep Lea ning models. We s a wi h a b ie in oduc ion o adi ional algo i hms o compu e ision and language gene a ion, we hen explain he main cha ac e is ics o Machine Lea ning and we con inue wi h Deep Lea ning. In he las sec ion o his chap e , we discuss some well-known image cap ion gene a o s. 9 10 CHAPTER 3. STATE OF THE ART 3.1 T adi ional algo i hms o image analysis Image analysis consis s o he ex ac ion o impo an cha ac e is ics and ea u es o images by digi al image p ocessing echniques, and i has many di e en ields o applica ion, such as obo ics, secu i y, medicine o biology. T adi ional compu e ision algo i hms, in gene al, wo k by ex ac ing ea u e ec o s om images and using hem o classi y images. The e a e algo i hms esponsible o a pa icula ask (noise educ ion, image segmen a ion, co ne de ec ion, edge de ec ion...), ha hen wo k oge he wi h o he speci ic algo i hms o ca y ou he whole image p ocessing ask. The use o hese echniques o ex ac use ul s uc u al in o ma ion om images, such as edges, co ne s o colo s, educes signi ican ly he amoun o da a o be p ocessed, as i il e s ou non- ele an da a o ocus only in he use ul in o ma ion ex ac ed. The main ea u e de ec ion algo i hms in compu e ision a e: •Canny edge de ec o [Canny, 1986]. An edge is a sudden change in image b igh ness, ha is, a poin whe e he image b igh ness has discon inui ies. These discon inui ies usually co espond o changes in su ace o ien a ion, dep h, ma e ial p ope ies o scene illumina ion. The e o e, an edge de ec o may help o iden i y he bounda ies o objec s and su ace ma kings, as well as a ia ions in su ace o ien a ion. The Canny edge de ec o is one o he bes edge de ec o algo i hms, and can de ec a wide a ie y o edges in an image. I i s smoo hs he image by applying a Gaussian il e , hen inds in ensi y g adien s and selec s he po en ial edges. •Ha is co ne de ec o [Ha is and S ephens, 1988]. A co ne is in e p e ed as he in e sec ion o wo edges. I is a e y impo an image ea u e as i is a poin in a ian o ansla ion, o a ion and illumina ion. The Ha is co ne de ec o algo i hm ocuses on he de ec ion o co ne s in an image, educing he dimensionali y o da a o be p ocessed and i is used in many compu e ision applica ions such as mo ion acking o s e eo ision (ex ac ing 3D cha ac e is ics om images). I has been p o ed o be one o he mos accu a e algo i hms in dis inguishing be ween edges and co ne s. •SIFT (Scale-In a ian Fea u e T ans o m) [Lowe, 1999]. I is a ea u e desc ip ion algo i hm used o de ec and desc ibe ea u es in images. SIFT ea u es a e in a ian o image loca ion, scale and o a ion. A desc ip o ec o is compu ed o each one o he poin s o in e es in he image, so ha hey a e also obus o changes in noise and illumina ion. This algo i hm is use ul o many compu e ision applica ions, such as objec ecogni ion, mo ion acking, na iga ion o 3D modeling. •SURF (Speeded-Up Robus Fea u es) [Bay e al., 2006]. I is also a ea u e ex ac o and desc ip o . This algo i hm is a speeded-up e sion o SIFT. These objec de ec ion algo i hms usually come a e a p ocess o image segmen a ion, 3.2. TRADITIONAL APPROACH TO NLG 11 ha is, di iding he image in o i s cons i uen s pa s, and a e hen combined wi h adi- ional Machine Lea ning algo i hms (SVM o K-Nea es neighbo ) o image classi ica ion. 3.2 T adi ional app oach o NLG Na u al Language Gene a ion (NLG), in ol es p oducing unde s andable na u al language ex s om some inpu da a. I has many di e en in e es ing applica ions, besides gen- e a ing image desc ip ions, such as machine ansla ion, obo ics o ex ual summa ies o ad anced da abases (e.g., inancial o wea he o ecas s da ase s) [Goldbe g e al., 1994, Io danskaja e al., 1992, Wu e al., 2016]. T adi ional NLG algo i hms a e based on he ield o o mal language heo y and logic ules on which we can build he language. As Rei e and Dale [Rei e and Dale, 1997] s a e, adi ional algo i hms in NLG usually ollow hese s eps: 1. Con en de e mina ion: deciding which in o ma ion should be included in he ou - pu . This p ocess c ea es a se o messages om he inpu da a, and i is ypically done by il e ing and summa izing he da a and de ining he messages in some o mal language, usually applica ion-dependen . 2. Documen s uc u ing: o ganizing and s uc u ing he se o messages so ha he inal ex makes sense. The ou pu o his s ep is usually ep esen ed as a ee s uc u e. 3. Agg ega ion: me ging ela ed messages in o sen ences o compac he in o ma ion. This s ep is no manda o y bu enables luency and eadabili y. 4. Re e ing Exp essing Gene a ion: p oducing exp essions iden i ying objec s ha he ex e e s o, as well as deciding abou p onouns. I is ela ed o lexicaliza ion. 5. Lexicaliza ion: selec ing he speci ic wo ds and ph ases o be used o ep esen concep s and ela ions ha appea in he messages. 6. Realiza ion: applying ules o syn ax, mo phology and spelling o p oduce he inal ou pu , such as adding p eposi ions, making plu als o adding punc ua ion ma ks. 3.3 Machine Lea ning Machine Lea ning is he sub- ield o A i icial In elligence ha s udies compu e algo i hms ha imp o e au oma ically h ough expe ience. In con as o adi ional algo i hms, ha ha e speci ic ules o doing he asks hey a e equi ed o , Machine Lea ning algo i hms lea n concep s wi hou being explici ly p og ammed o doing so, iden i ying pa e ns om gi en examples o pe o m accu a ely on new, unseen da a by in e ing hese unco e ed 12 CHAPTER 3. STATE OF THE ART pa e ns. These algo i hms can lea n om and make p edic ions on la ge olumes o da a, no exclusi ely images. The ield s a ed o be s udied in he mid 1980s and ea ly 1990s, and became mo e popula a ound 2010 as mo e and mo e da a we e a ailable o lea ning and aining he Machine Lea ning models. This discipline has many applica ions in a wide a ie y o ields whe e designing and p og amming explici algo i hms wi h good pe o mance is di icul o in easible. These applica ions include oice ecogni ion, na u al language p ocessing, ansla ion, sea ch engines, compu e ision, obo ics, medical diagnosis, inancial ma ke analysis, ad e is- ing o ecommende sys ems [Pang e al., 2002, We nick e al., 2010, B idge e al., 2014, Sa ikaya e al., 2014, Baik and Bala, 2004]. We can classi y Machine Lea ning echniques in o wo ca ego ies: supe ised lea ning and unsupe ised lea ning. In supe ised lea ning, he Machine Lea ning algo i hm is gi en a se o sample inpu s oge he wi h hei desi ed ou pu s, so ha i can ain wi h hose examples o in e a gene alized unc ion ha maps inpu s in o ou pu s and p edic s well when gi en a new example. The e o e, al eady labeled-da a a e equi ed in his app oach. The e a e also some special cases o supe ised lea ning such as semi-supe ised lea n- ing, ha consis o a se o examples whe e no all o hem go wi h hei co esponding desi ed ou pu , ha is, he e a e labeled and unlabeled da a on he se ; ac i e lea ning, whe e he algo i hm has only a small ini ial se o labeled da a and i is able o in e ac- i ely make que ies o he use o ind he desi ed ou pu s o some unlabeled da a; and ein o cemen lea ning, whe e he e is some so o eedback ob ained om he ou side in a dynamic en i onmen as an answe o i s ac ions. Some o he main supe ised lea ning algo i hms and app oaches a e he ollowing: •Decision ees and andom o es s: hey a e used as a p edic i e model. In a decision ee, he b anches ep esen he obse a ions abou an i em, i s ea u es and he lea es ep esen class labels. Random o es s [Ho, 1995] cons uc many decision ees o make mo e accu a e p edic ions by ou pu ing a combina ion o he ou pu s o all hose decision ees, ypically he mode o mean. Random o es s a e e y popula as i is easy and clea o see and unde s and he decisions aken by hem, while o he Machine Lea ning algo i hms a e mo e obscu e. •Logis ic eg ession: i is a s a is ical me hod o model a bina y dependen a iable (classes 0 and 1) in e ms o one o mo e explana o y a iables, using he logis ic unc ion σ(z) = 1 1+e−z. This unc ion e u ns he p obabili y o belonging o class 1. The algo i hm’s ou pu is hen selec ed by compa ing he p obabili y o a gi en h eshold (usually 0.5). •Suppo Vec o Machines (SVM) [Co es and Vapnik, 1995]: i is a bina y classi i- ca ion algo i hm ha gi en a se o poin s o wo di e en classes in a n-dimensional 3.4. DEEP LEARNING 13 space, sepa a es hem wi h he hype plane ha is si ua ed he u hes om all he poin s. This algo i hm is only use ul i he poin s a e linea ly sepa able, i no , he algo i hm was imp o ed by including ke nels, ha p ojec he se o poin s in a bigge dimension space, o ob ain a hype su ace ha sepa a es he poin s in he ini ial space. •Neu al ne wo ks: i is a lea ning algo i hm whe e compu a ions a e s uc u ed in connec ed neu ons ha ansmi signals om one o ano he . The e o e, each neu- on’s inpu is he ou pu o ano he neu on (o he inpu o he algo i hm o he i s laye o neu ons), and each neu on’s ou pu is compu ed as a non-linea unc ion o he inpu . •Nea es neighbo algo i hm [Al man, 1992]: i is used bo h o classi ica ion ( he e a e a disc e e numbe o classes) and eg ession ( he a ge a iable is con inuous), and i is based on he in o ma ion o he k closes poin s. In classi ica ion, he ou pu is he mos common class among i s k neighbo s, and in eg ession he ou pu is he mean o he alues o hese neighbo s. Unsupe ised lea ning algo i hms lea n om a se o unlabeled da a, inpu examples wi hou hei desi ed ou pu , in e ing a unc ion ha can desc ibe he s uc u e in he da a. The e o e, he sys em mus be capable o inding pa e ns in he da a in o de o be able o label he new inpu s; bu , as he da a used o ain he algo i hm is unlabeled, he e is no way o e alua ing he accu acy o he pa e ns ound in i . The ollowing a e he mos common unsupe ised lea ning algo i hms: •Clus e ing: hese me hods di ide he da a se in o di e en subse s, o clus e s, so ha he poin s in he same clus e a e mo e simila be ween hem han be ween hose o o he g oups. The goal is o maximize he simila i y be ween poin s in he same subse and o maximize he di e ence be ween poin s o di e en subse s. •P incipal Componen Analysis (PCA) [Ho elling, 1933]: his algo i hm is used o educe he dimensionali y o he da a se , wi hou losing any impo an in o ma ion and il e ing edundancy, by making linea combina ions o he o iginal da a a iables and selec ing he ones ha ha e he la ges possible a iance and a e unco ela ed. 3.4 Deep Lea ning Deep lea ning is a Machine Lea ning echnique, based on neu al ne wo ks wi h many laye s be ween he inpu and he ou pu ones [LeCun e al., 2015]. Al hough he heo y behind Deep Lea ning has been de eloped o many yea s now, deep neu al ne wo ks need much compu e capaci y and labeled da a o pe o m apidly and co ec ly he complex asks i is use ul o . The e o e, Deep Lea ning has only ecen ly 20 CHAPTER 3. STATE OF THE ART Figu e 3.5: Cap ionbo This de ice is e en able o lea n new aces and objec s i he use shows hem o Ho us in di e en angles and speaking ou loud he name o he objec o he pe son. As we ha e seen, mo e and mo e sophis ica ed image cap ion gene a o s, o di e en pu poses and con ex s, a e being de eloped, showing he in e es and impac o his b and- new ield and esul ing in a ich li e a u e. Chap e 4 F amewo k T aining and de eloping Deep Lea ning models equi es g ea compu a ional capaci y and a good elec ion o he da ase . In his chap e , we explain wha a e GPUs and why a e hey use ul o de eloping Deep Lea ning p ojec s, which Deep Lea ning amewo k and isualiza ion ools we ha e selec ed and which da ase we use. We de eloped he p ojec using Py hon as p og amming language. 4.1 GPU GPUs (G aphics P ocessing Uni ) we e adi ionally c ea ed o compu e g aphics and image p ocessing o accele a e he complex calcula ions o eal- ime 2D and 3D g aphics ha esul ed in long p ocessing ime in CPUs. GPUs a e build o op imized asks based on he SIMD pa allel concep (Single In- s uc ion Mul iple Da a), ha is, he same ins uc ion applied epea edly o e a bunch o da a. Those epea ed applica ions o he same ins uc ion a e all independen and can, he e o e, be un simul aneously. Linea algeb a ope a ions a e inhe en ly pa allel and a e he basis o polygon ans o ma ions o scene ende ing in compu e g aphics. The e a e o he powe and ime consuming applica ions o pa allel na u e ha could be op imized by using a GPU, bu hey in ol e e o mula ing he p oblem in o g aphics p imi i es o be able o use hem, which is a e y complex ask. Fo ha eason, NVIDIA de eloped CUDA (Compu e Uni ied De ice A chi ec u e), a pa allel compu ing oolki simila o C p og amming language, o exploi GPUs capabili ies o gene al pu poses, which is commonly called GPGPU (Gene al Pu pose G aphics P ocessing Uni ). Linea algeb a is he basis o Deep Lea ning algo i hms, and also many Machine Lea n- ing algo i hms. Neu ons wi hin a laye apply all he same unc ion o di e en inpu da a, so hey could be also pa allelized and hence GPUs a e always used o aining hese models. 21 22 CHAPTER 4. FRAMEWORK Figu e 4.1: Deep Lea ning amewo ks popula i y Acqui ing a GPU is no cheap, he e exis s cloud compu ing pla o ms, such as Amazon Web Se ices1, Google Cloud Pla o m2o Mic oso Azu e3 ha con ain di e en se e s wi h se e al GPUs o allow use s o un hei models mo e economically. GAIA esea ch g oup has buil a se e wi h an NVIDIA Ti an X GPU in o de o main ain Deep Lea ning p ojec s o s uden s and p o esso s and ga e us access o i , so we will use i o aining his p ojec . 4.2 Deep Lea ning amewo k CUDA has a e y low le el API and makes implemen ing Deep Lea ning compu a ional g aphs conside ably edious. Fo his eason, many highe le el amewo ks we e c ea ed o easily desc ibing and de eloping he complex asks ha Deep Lea ning in ol es. The e is a wide numbe o amewo ks a ailable o Deep Lea ning aining, he ma- jo i y o hem open-sou ce and de eloped by big companies o uni e si y eams such as Mic oso , Google, In el o Be keley Uni e si y. The mos well-known and equen ly used amewo ks a e To ch [Collobe e al., 2011], Tenso low [Abadi e al., 2016], Ca e [Jia e al., 2014] and Theano [Theano De elopmen Team, 2016]. Among all hese amewo ks, Tenso low is p obably he mos popula one (as Fig- u e 4.1, ex ac ed om Google images, shows), used in dis inguished companies such as Ai bnb, Ube , D opbox o Twi e . I is an open-sou ce amewo k o ad anced nume i- cal compu a ion, de eloped by he Google B ain eam in 2015, eplacing hei p opie a y 1h ps://aws.amazon.com 2h ps://cloud.google.com/ 3h ps://azu e.mic oso .com 4.2. DEEP LEARNING FRAMEWORK 23 om ke as.models impo Sequen ial om ke as.laye s impo Dense # Building he model and lea ning con igu a ion model = Sequen ial() model.add(Dense(uni s=64, ac i a ion=’ elu’, inpu _dim=100)) model.add(Dense(uni s=10, ac i a ion=’so max’)) model.compile(loss=’ca ego ical_c ossen opy’, op imize =’sgd’, me ics=[’accu acy’]) # T aining, e alua ion and p edic ion # x_ ain, y_ ain, x_ es , y_ es and new_da a a e numpy a ays model. i (x_ ain, y_ ain, epochs=5, ba ch_size=32) loss_and_me ics = model.e alua e(x_ es , y_ es , ba ch_size=128) p edic ion = model.p edic (new_da a, ba ch_size=128) Figu e 4.2: A simple Ke as example code Machine Lea ning sys em Dis Belie [Dean e al., 2012], c ea ed in 2011. Tenso low is widely used o Deep Lea ning and Machine Lea ning and i is op imize o un models bo h on CPUs o GPUs. I p o ides an API o Py hon as well as o o he p og amming languages (C++, Haskell, Ja a...), bu he mos de eloped one is he Py hon lib a y. Tenso low wo ks well o modeling Con olu ional Neu al Ne wo ks and Recu en Neu al Ne wo ks. The basic objec in Tenso low lib a y is he enso , an abs ac ion o mul idimensional a ays, being a 1-dimensional enso a ec o and a 2-dimensional enso a ma ix. Deep Lea ning compu a ional g aphs a e build by desc ibing enso s’ ope a ions and g adien s a e calcula ed au oma ically using a speci ic unc ion. Howe e , p og amming in Tenso low can be di icul and he e exis s a highe le el API ha allows lexible Tenso low implemen a ions in a mo e use - iendly manne . This API is Ke as4, a high-le el open-sou ce Py hon lib a y o de eloping Deep Lea ning p ojec s mo e easily and eadable, c ea ed in 2015 by a Google enginee and eleased unde he MIT license. Ke as wo ks wi h Tenso low, as well as Theano o CNTK [Seide and Aga wal, 2016], as backend. In 2017 Tenso low included i as pa o i s lib a y and selec ed i as he p e e ed high-le el API. The basic s uc u e in Ke as is he model, a se o connec ed laye s. Compu a ional g aphs a e buil by s acking laye s wi h he add() unc ion, whe e he use only needs o speci y he laye s and he inpu s’ dimensions. Ke as lib a y p o ides he well-known laye s bu allows use s o also c ea e new ones. Con igu ing he aining p ocess is as simple as applying o he model he compile() unc ion wi h he speci ic condi ions (loss unc ion, op imize , me ics...). T aining, e alua ing he model and making p edic ions on unseen examples is done wi h jus o he high le el unc ions: i (),e alua e() and p edic () 4h ps://ke as.io/ 24 CHAPTER 4. FRAMEWORK espec i ely. Figu e 4.2 shows a simple example, ex ac ed om Ke as documen a ion, o how o use hese unc ions o build a neu al ne wo k wi h wo ully-connec ed laye s. The e o e, as ou needs a e co e ed in Ke as’ lib a y, i is widely ecommended o Deep Lea ning s a e s and i is mo e use - iendly, we decided o de elop ou model in Ke as on op o Tenso low. Ke as and Tenso low ins alla ion is done easily in Linux ia he pip ins all command, c ea ing p e iously a Py hon i ual en i onmen . 4.3 Visualiza ion: Tenso Boa d Deep Lea ning models a e composed o millions o complex calcula ions ha o en com- plica e, o e en make i impossible, o debug and unde s and he pa ame e s and me ics, wha happens du ing aining and wha is he neu al ne wo k lea ning. Fo his pu pose, Tenso low de eloped Tenso boa d5, a sui e o isualiza ion ools inside Tenso low lib a y ha allows use s o isualize compu a ion g aphs, me ics esul s and o he use ul cha s. Tenso Boa d wo ks by sa ing log iles, con aining execu ion in o ma ion abou he aining p ocess and he compu a ion g aph, in o a speci ic log di ec o y, and opening Tenso Boa d web se e o access he g aphic in e ace. The desi ed da a is ob ained wi h summa y ope a ions, enso ’s ope a ions ha p o- duce se ialized in o ma ion abou a model ha needs o be ead wi h Tenso Boa d. These summa y ope a ions ecei e he enso we wan o isualize and a meaning ul name o dis inguish all o hem. Then, all he summa ies c ea ed a e combined using .summa y.me ge all in o a single ope a ion ha gene a es all o hem. Finally, he se i- alized summa y is w i en o he speci ied di ec o y in disk wi h .summa y.FileW i e . To access he g aphic in e ace o Tenso Boa d o isualize he summa ies, we jus need o go o h p://localhos :6006 a e yping he ollowing command in he command line: $ enso boa d --logdi pa h/ o/log-di ec o y I is also possible o compa e isualiza ions o di e en execu ions by speci ying he pa hs o he log di ec o ies in he p e ious command. The appea ance o Tenso Boa d g aphic in e ace is shown in Figu e 4.3 Tenso Boa d con ains a wide a ie y o help ul isualiza ions: .summa y.scala , .summa y.his og am, .summa y.audio o .summa y.image, among o he s. The i s one is he mos used, i displays he a ia ion o a scala me ic o e ime, usually he loss o he lea ning a e. All hese isualiza ions admi some dynamic in e ac ions wi hin he Tenso Boa d in e ace. Using Tenso Boa d wi hin Ke as high-le el API i is e en easie . I is jus necessa y o include he callback enso boa d unc ion while aining, speci ying he log di ec o y. 5h ps://gi hub.com/ enso low/ enso boa d 4.4. IMAGE DATASETS 25 Figu e 4.3: Tenso Boa d web se e This unc ion will eco d da a o loss and accu acy me ics, as well as o he me ics speci ied in he compile unc ion. 4.4 Image da ase s Fo co ec ly aining and accu a ely e alua ing a good Machine Lea ning model, a sui able selec ion o elabo a ion o he da ase is c ucial. Once we ha e he da ase , i is di ided in o h ee disjoin subse s: aining, es and alida ion se s. T aining se is used o aining he model, i ing he model’s pa ame e s; es se is used once he model is ained, o p o ide an unbiased e alua ion o his inal model; alida ion se , also called de elopmen se o de se , is used while aining he model, o une he hype pa ame e s and o p e en o e i ing. As Deep Lea ning esea che s ha e a s ong in e es in he image analysis ield, du - ing hese yea s a lo o la ge and medium scale da ase s ha e been c ea ed o aining hese models and we can ind many o hem a ailable online. The mos popula ones a e explained below. 26 CHAPTER 4. FRAMEWORK ImageNe ImageNe 6is p obably he mos well-known image da ase in he Deep Lea ning and machine lea ning indus y. This image da ase is used mainly o classi ica ion pu poses and he classes a e o ganized based on he nouns in he Wo dNe [Mille , 1995] hie a chy. This image da abase was c ea ed in 2009 [Deng e al., 2009] by que ying image sea ch engines wi h Wo dNe ’s synse s. They ex ac ed a g ea numbe o candida e images o each synse and hen il e ed hem by c owd-sou cing in Amazon Mechanical Tu k (AMT)7. Since 2010, ImageNe also uns he ILSVRC (ImageNe La ge Scale Visual Recogni ion Challenge) annual isual challenge [Russako sky e al., 2015]. Nowadays, his challenge is hos ed in Kaggle8and i has h ee ca ego ies: image classi ica ion o 1000 classes, objec de ec ion (de ec he bounding box whe e he objec lays) o 200 classes and objec de ec ion om ideo o 30 classes. As o oday, ImageNe is one o he la ges image da abase a ailable. I has a o al o 14,197,122 anno a ed images o a wide a ie y o ca ego ies, wi h an a e age o o e i e hund ed images pe synse . Fo he pe son’s high-le el ca ego y ( he one we ha e in e es in), i holds a ound 952,000 images and i is di ided in 2,035 synse s wi h an a e age o 468 images pe synse . Images a e anno a ed wi h hei classi ica ion and 1,034,908 o hem also wi h hei bounding boxes. MS COCO MS COCO9(Mic oSo Common Objec s in COn ex ) da ase [Lin e al., 2014] is one o he mos widely used la ge-scale da ase s o image desc ip ion gene a ion and also o objec de ec ion, objec segmen a ion and pe son keypoin s de ec ion. Cu en ly, he MS COCO da ase consis s o o e 328,000 images o 91 basic objec ypes in na u ally occu ing con ex s. Each image is labeled wi h a leas i e cap ions desc ibing i , which leads o a o al o 2.5 million cap ions, and bounding boxes o each objec ca ego y ha appea s in he pic u e. They also con ain 250,000 people anno a ed wi h hei keypoin s. This da ase has also gi en ise o image challenges o objec de ec ion, keypoin s de- ec ion and image cap ioning. The la e is al eady closed, bu he e is an open e alua ion se e and API [Chen e al., 2015] o compa e o s a e-o - he-a me hods using se e al pe o mance me ics such as BLEU, ROUGE, METEOR and CIDE . 6h p://www.image-ne .o g/ 7A well-known c owd-sou cing online pla o m. h ps://www.m u k.com/ 8h ps://www.kaggle.com/ 9h p://cocoda ase .o g/#home 4.4. IMAGE DATASETS 27 PASCAL PASCAL10 (Pa e n Analysis S a is ical Modeling and Compu a ional Lea ning) Visual Objec Classes p ojec comp ised challenges [E e ingham e al., 2010] o image classi ica- ion, de ec ion and segmen a ion o eigh consecu i e yea s (2005-2012). Nowadays, he challenges a e closed bu i p o ides an image da abase and ools o ob aining hese im- ages and hei anno a ions, as well as an e alua ion se e . The la es da ase (PASCAL 2012 challenge) con ained 20 dis inc classes and mo e han 11,000 anno a ed images. These images we e collec ed om Flick . These da ase s ha e anno a ions o objec classi ica ion, de ec ion and segmen a ion, bu a subse o he Pascal 2008 challenge da ase was used o c ea e an image desc ip ion da abase, called Pascal1K. I con ains 1,000 images wi h objec s o di e en classes and is anno a ed wi h i e desc ip ions, gene a ed by humans on Amazon Mechanical Tu k c owd-sou cing pla o m. As i is a medium-scale da abase, i is gene ally used as a benchma k o e alua ing image desc ip ion models. Flick 8K and Flick 30K Flick 8K11 [Hodosh e al., 2013] da ase and i s ex ension Flick 30K12 [Young e al., 2014], con ain images om Flick , each one anno a ed wi h i e desc ip ions collec ed om he c owd-sou ce pla o m Amazon Mechanical Tu k (AMT). Flick 8K consis s o app oxi- ma ely 8,000 images and Flick 30K o a ound 30,000 images. These pic u es we e collec ed by he Uni e si y o Illinois by que ying Flick o speci ic objec s and ac ions. SBU SBU Cap ioned Pho o Da ase was speci ically c ea ed o he Im2Tex image cap ion gene a o [O donez e al., 2011]. I con ains 1 million Flick images labeled wi h hei o iginal cap ions gene a ed by hei use s, collec ed by que ying Flick o speci ic objec s and ac ions. I is a la ge-scale da ase bu no e y popula o aining image desc ip ion models as he labeled cap ions may con ain subjec i e in o ma ion o in o ma ion no con ained in he pic u e because hey we e gene a ed by he Flick use . CIFAR CIFAR-10 and CIFAR-100 (Canadian Ins i u e o Ad anced Resea ch) [K izhe sky, 2012] a e image classi ica ion da ase s ha di e only in he numbe o classes hey ha e (CIFAR- 10 has en classes and CIFAR-100 one hund ed classes, g ouped in o wen y supe classes). 10Download page: h p://hos . obo s.ox.ac.uk/pascal/VOC/ 11Fo m o downloading he Filck 8K da ase :h ps:// o ms.illinois.edu/sec/1713398 12Fo m o downloading he Filck 30K da ase : h ps:// o ms.illinois.edu/sec/229675 28 CHAPTER 4. FRAMEWORK Figu e 4.4: COCO cap ion da ase Bo h da ase s consis o 60,000 32x32 RGB images, CIFAR-10 holds 6,000 in each class and CIFAR-100 holds 600. Ke as p o ides buil -in unc ions o load hese da ase s. LabelMe LabelMe13 is an image da abase and an online anno a ion ool [Russell e al., 2008] c ea ed in he MIT Compu e Science and A i icial In elligence Labo a o y (CSAIL). The da ase is dynamic, ee o use and open o public con ibu ion, and i consis s o almos 200,000 images bu no all o hem a e ully anno a ed. I has mo e han 62,000 anno a ed images and mo e han 658,000 labeled objec s. The online anno a ion ool allows he use o d aw polygons, que y images, b owse he da abase o download a subse . 4.4.1 Da ase elec ion Ha ing s udied he cha ac e is ics o all hese da ase s, we decided o use he MS COCO da ase o aining and es ing ou model, as i con ains a wide numbe o images, i is one o he mos commonly used o image desc ip ion and i p o ides ools o downloading he da ase . They al eady p o ide a ain/ al/ es spli o he da ase , howe e , as his da ase was eleased o hei annual challenges, he es se does no come labeled and we will no be able o use i o es ing ou model, so we will c ea e ou own spli based on he aining and alida ion se s. 13Download page: h p://labelme.csail.mi .edu 4.4. IMAGE DATASETS 29 Figu e 4.4 shows some examples o wha can be ound in COCO cap ion da ase . In pa icula , we use he Py hon COCO API o download he MS COCO da ase and o selec a subse o i con aining only images o people, o es ic o ou objec i es he images we p o ide o he model. In addi ion, we also use he people’s subse (baby, boy,gi l,man and woman ca ego ies) o he CIFAR-100 da ase o ine- une he image ea u e-ex ac ion model we use, as explained in he nex chap e . 36 CHAPTER 5. DEEP LEARNING APPROACH """-----De ini ion o VGG16 ine- uned model-----""" de model_de ini ion(): gg16_model = VGG16(weigh s="imagene ", include_ op=False, inpu _shape=(224,224,3)) # Replace op laye gg16_ou pu = gg16_model.ou pu x1 = Fla en()( gg16_ou pu ) x2 = Dense(4096, ac i a ion=’ elu’, name=" c1")(x1) x3 = Dense(4096, ac i a ion=’ elu’, name=" c2")(x2) p edic ion = Dense(100, ac i a ion=’so max’, name="p edic ions")(x3) # C ea e model model = Model(inpu = gg16_model.inpu , ou pu =p edic ion) # Se i s laye s non- ainable o laye in model.laye s[:12]: laye . ainable = False model.compile(op imize =SGD(l =1e-3, decay=1e-6, momen um=0.9), loss=’ca ego ical_c ossen opy’, me ics=[’accu acy’]) e u n model Figu e 5.4: Py hon code o VGG16 ine- uned model de ini ion. unc ion wi h a ba ch size o 32 images and 30 epochs3, and sa e i in a hd 54 ile in o memo y using model.sa e(). 5.3 Da a p epa a ion Once we ha e ou ine- uned VGG16 model, we can s a de ining he models o gene a ing image desc ip ions. As we explained in he i s sec ion o his chap e , we will use COCO da ase , pa ly o en i ely depending on he model, o aining and es ing ou models. Fo una ely, COCO p o ides a Py hon API o loading in o memo y and pa sing he da ase in an easie way. To use his API, he da ase iles mus be downloaded om hei webpage in an speci ic di ec o y s uc u e. We ha e downloaded he 2014 da ase because i is he one ha con ains image desc ip ions as labels. A e cloning he gi hub p ojec and unning he Make ile, he API can be used by impo ing he pycoco ools.coco 3An epoch is an en i e i e a ion o e he whole aining da ase . The ba ch size is he numbe o aining samples ained be ween pa ame e s upda es. 4File o ma o s o e la ge amoun s o da a. 5.3. DATA PREPARATION 37 module. Mo e in o ma ion on how o use his API can be ound on hei webpage5and hei gi hub p ojec 6. All COCO images and desc ip ions mus be p ep ocessed in o de o co ec ly use hem as he inpu s o ou models. E en hough wo o ou models wo k only wi h a subse o COCO, he o he wo wo k wi h he whole da ase , so bo h image p ep ocessing and desc ip ions p ep ocessing ha e been done in he whole da ase , lea ing he subse ex ac ion o a la e s ep whe e we also pe o m he ain/ al/ es spli o he da ase as we explained in Sec ion 4.4. In he ollowing subsec ions we explain how we ha e ca ied ou hese wo asks. 5.3.1 Image p ep ocessing COCO images do no need oo much p ep ocessing. Howe e , he Con olu ional Neu al Ne wo k o ou models, ei he he VGG16 ne wo k o he ine- uned model, has al eady p ecompu ed i s weigh s and hese weigh s will emain ozen when aining ou image desc ip ions gene a ion models. The e o e, images ea u es could be p ecompu ed be o e aining he models and sa ed o be used la e as an inpu ins ead o he images hemsel es. This way we only calcula e hese ea u es once ins ead o e e y ime he aining p ocess passes h ough one o hese images, sa ing compu a ional ime and memo y space. This image p ep ocessing mus be epea ed o he VGG16 model and o ou ine- uned model, o ex ac he ea u es ob ained by bo h models and use one o ano he depending on which o he ou models we a e aining. Figu e 5.5 shows he ea u es ex ac ion unc ion we ha e de ined, whe e model e e s o ei he VGG16() o o ou ine- uned model, ea u es is he dic iona y ha will s o e he ex ac ed image ea u es and images ids a e all COCO aining image ids ob ained wi h COCO API as ollows: da aDi = ".." da aType = " ain2014" annFile = ’{}/anno a ions/ins ances_{}.json’. o ma (da aDi ,da aType) coco = pycoco ools.coco.COCO(annFile) imgs_ids = coco.ge ImgIds() This p ocess is also done wi h he alida ion COCO da ase , changing da aType o " al2014". To ex ac image ea u es, las laye o he model mus be emo ed using laye s.pop() because his laye is he one ha p edic s which class he image belongs o and we only ca e abou image ea u es. We hen load e e y image in he COCO da ase using COCO API and Ke as load img() unc ion. Image a ge size is 224x224 as his is he inpu size o VGG16 (and he e o e also o ou ine- uned model). Once we ha e loaded he image in o memo y, i is con e ed o a numpy a ay and eshaped, using Ke as unc ions om he ke as.p ep ocessing.image module. Finally, ea u es a e ex ac ed 5h p://cocoda ase .o g/download 6h ps://gi hub.com/cocoda ase /cocoapi 38 CHAPTER 5. DEEP LEARNING APPROACH de ea u es_ex ac ion(model, ea u es, di ec o y, da aType, images_ids): # emo e p edic ion laye om model model.laye s.pop() model = Model(inpu s=model.inpu s, ou pu s=model.laye s[-1].ou pu ) o id in images_ids: ile = "0:0>12". o ma (id) pa h = "/images//COCO__.jpg". o ma (di ec o y, da aType, da aType, ile) img = load_img(pa h, a ge _size=(224, 224)) # img o 3D numpy a ay (heigh , weigh , colo channel) and eshape i img = img_ o_a ay(img) img = img. eshape((1, img.shape[0], img.shape[1], img.shape[2])) # ge ea u es and s o e hem in dic img = p ep ocess_inpu (img) ea u e = model.p edic (img, e bose=1) ea u es[id] = ea u e e u n ea u es Figu e 5.5: Image p ep ocessing: ea u es ex ac ion o COCO images. using he p edic () unc ion and hen s o ed in o he dic iona y. A e ha ing ex ac ed all aining and alida ion images ea u es, we sa e he dic iona y con aining hem in a ile, using Py hon pickle lib a y7, o la e use. 5.3.2 Tex p ep ocessing The image desc ip ions p ep ocessing ask ha we ha e implemen ed pe o ms a ious ope a ions on he wo ds o clean hose desc ip ions and ob ain a sui able ocabula y o lea ning, ha should be ep esen a i e and exp essi e enough. These ope a ions a e: changing uppe case le e s o lowe case, emo ing punc ua ion ma ks, emo ing numbe s, emo ing wo ds o leng h one (as hey a e no exp essi e), emo ing wo ds used less han i e imes in he whole da ase and ans o ming English con ac ions such as don’ o do no . We also add ini ial and inal okens (START and END) o delimi ing he desc ip ions. Figu e 5.6 shows he p ep ocessing unc ion we ha e de ined o cleaning COCO desc ip ions, whe e: wo d eq is a Py hon Coun e 8wi h he wo ds’ equencies and desc ip ions is a lis con aining all he desc ip ions o COCO aining da ase and i is ob ained using COCO API as ollows: 7h ps://docs.py hon.o g/3/lib a y/pickle.h ml 8h ps://docs.py hon.o g/3/lib a y/collec ions.h ml#collec ions.Coun e 5.4. DATASET SPLIT 39 de p ep ocess_desc(desc ip ions, wo d_ eq): # ansla ion able o emo ing punc ua ion able = s .make ans(’’, ’’, s ing.punc ua ion) o desc_in o in desc ip ions: desc = desc_in o[’cap ion’] wo ds = desc.spli () # con e o lowe case wo ds = [w.lowe () o w in wo ds] # emo e numbe s wo ds = [w o w in wo ds i no w.isnume ic()] # ans o m con ac ions wo ds = [con ac ions(w) o w in wo ds] # emo e punc ua ion om each wo d wo ds = [w. ansla e( able) o w in wo ds] # emo e wo ds leng h < 2 wo ds = [w o w in wo ds i len(w)>1] # emo e wo ds used < 5 imes wo ds = [w o w in wo d_ eq.keys() i wo d_ eq[w]>=5] # s o e in dic iona y wi h START and END okens desc_in o[’cap ion’] = ’START ’ + ’ ’.join(wo ds) + ’ END’ Figu e 5.6: Tex p ep ocessing: cleaning COCO desc ip ions. da aDi = ".." da aType = " ain2014" annFile = ’/anno a ions/cap ions_.json’. o ma (da aDi ,da aType) coco = COCO(annFile) anns_ids = coco.ge AnnIds() desc ip ions = coco.loadAnns(anns_ids) This p ocess is also epea ed wi h he alida ion desc ip ions, as we ha e done wi h im- age p ep ocessing, changing da aType o " al2014". Each elemen o he desc ip ions lis is a dic iona y con aining he image id he desc ip ion belongs o, an id o he de- sc ip ion and he desc ip ion i sel . Fo ans o ming con ac ions, we ha e de ined a dic iona y con aining he ans o ma ions and con ac ions() e u ns his ans o ma- ion i he wo d is in ha dic iona y. Once we ha e p ep ocessed all aining and alida ion desc ip ions, we me ge bo h lis s in o a single one and sa e i in a ile using pickle lib a y. 5.4 Da ase spli A e ha ing p ep ocessed he en i e COCO da ase , we now spli i in o ain, alida ion and es se s (wi h a 70%-20%-10% di ision) and di ide consequen ly he ea u es dic io- na ies (VGG16 ea u es and ine- uned model ea u es) ob ained while p ep ocessing he 40 CHAPTER 5. DEEP LEARNING APPROACH # ge ids o 35000 aining imgs gene al and wi h people da aDi = ".." da aType = " ain2014" annFile = ’/anno a ions/ins ances_.json’. o ma (da aDi ,da aType) coco = COCO(annFile) ca s_ids = coco.ge Ca Ids(ca Nms=[’pe son’]) ain_people_ids = coco.ge ImgIds(ca Ids=ca s_ids) ain_people_ids = andom.sample( ain_people_ids, 35000) ain_ids = coco.ge ImgIds() ain_imgs_ids = andom.sample( ain_ids, 35000) # ge ids o 35000 alida ion imgs gene al and wi h people da aDi = ".." da aType = " al2014" annFile = ’/anno a ions/ins ances_.json’. o ma (da aDi ,da aType) coco = COCO(annFile) ca s_ids = coco.ge Ca Ids(ca Nms=[’pe son’]) al_people_ids = coco.ge ImgIds(ca Ids=ca s_ids) al_people_ids = andom.sample( al_people_ids, 15000) al_ids = coco.ge ImgIds() al_imgs_ids = andom.sample( al_ids, 10000) # spli al people in al and es al_people_ids, es _ids = ain_ es _spli ( al_people_ids, es _size=5000) Figu e 5.7: Tes spli o COCO da ase . images and he desc ip ions lis ob ained while p ep ocessing he desc ip ions. As ou goal is o e alua e how well ou models desc ibe images con aining people, he es se mus only include his kind o images. On he o he hand, he ain and alida ion se s, may o may no include o he kind o images, depending on he model ha we a e aining. The e o e, we will ha e wo di e en spli s o hese wo se s, one o hem o he subse o images con aining people (excluding he ones in he es se ) and he o he one o he whole da ase wi hou he es se . T aining ou models wi h he whole COCO da ase (mo e han 300,000 images) was impossible due o he cha ac e is ics o he se e we ha e used and o ou ime limi a ion. In o de o educe ime and memo y consump ion, we decided o es ic ou aining p ocess o only a subse o he COCO da ase : 35,000 images o he aining se , 10,000 images o alida ion and 5,000 images o he es se (70%-20%-10%). To ob ain he aining se we ha e andomly selec ed, using andom.sample() Py hon unc ion, 35,000 images o he whole COCO aining se and 35,000 images o all he COCO aining images ha con ain people. Valida ion and es se s ha e been ex ac ed bo h 5.5. TRAINING PROCESS 41 om COCO alida ion se , andomly selec ing 15,000 COCO alida ion images con aining people (5,000 o he es se and he emaining 10,000 o he alida ion se ) and 10,000 COCO alida ion images o any kind. This p ocess is shown in Figu e 5.7. We ha e also ensu e ha he gene al aining and alida ion spli s a e ep esen a i e enough by e i ying ha he andom subse s ha e enough images ha con ain people (30%-60%). Once we ha e pe o med he da ase spli s, we di ide he ea u es dic iona ies and he desc ip ions lis acco dingly o he di e en spli s and we sa e hese di isions in memo y using pickle lib a y. 5.5 T aining p ocess The i s s ep in he aining p ocess o he ou models is o load he iles ob ained wi h he image and ex p ep ocessing and di ided when making he da ase spli . Following he enume a ion we ha e de ined in Sec ion 5.1, Model 1 and Model 3 load image ea u es ob ained wi h VGG16 and Model 2 and Model 4 load he ones ob ained wi h he ine- uned VGG16 model. Subsequen ly, Model 1 and Model 2 load he ain and alida ion spli o he whole da ase and Model 3 and Model 4 he one o he subse o images ha con ain people. The emaining s eps in he aining p ocess a e he same o he ou models, wi h he only di e ences being he inpu s. A e loading he images ea u es and desc ip ions, we c ea e and i a Tokenize o e all aining and alida ion desc ip ions, using Ke as Tokenize () class. A Tokenize lea ns om a lis o ex s how o map he bag o wo ds con ained in ha ex s o in ege alues and allowing o see each desc ip ion as an in ege sequence. The nex s ep is o de ine he model ha we a e going o ain. The model de ini ion in Py hon using Ke as ollows he a chi ec u e ha we ha e de ined in Figu e 5.2 and i is shown in Figu e 5.8. The hype pa ame e s a e ex ac ed om Tan i e al. obse a ions [Tan i e al., 2017b]. ocab size,max leng h and emb ma ix a gumen s a e he size o he whole bag o wo ds con ained in he desc ip ions, he leng h o he la ges desc ip ion and he GloVe wo d embedding weigh ma ix espec i ely. The unc ion s a s wi h he de ini ion o he Ke as inpu enso o he image ea u es using Inpu () and he inpu shape is 4096 as i is he shape o he ea u es ec o ex ac ed o each image. This inpu is passed o a Dense laye wi h a ReLu ac i a ion unc ion and L2 egula iza ion o ob ain a 128 elemen ep esen a ion. Regula iza ion echniques a e in oduced in Deep Lea ning aining p ocesses o p e en o educe o e i ing (good lea ning o he aning se bu oo speci ic, leading o no gene aliza ion powe ). In pa ic- ula , L2 egula iza ion adds an ex a e m o he cos unc ion p opo ional o he sum o squa es o all he weigh s. We hen de ine he language model wi h a Ke as Inpu () en- so o leng h max leng h and his inpu is passed o a wo d embedding laye wi h GloVe p e- ained weigh s o map he spa se wo d ec o s o dense ones, as we explained in Sec- ion 5.1.2. A e he Embedding laye we pe o m a 0.5 d opou [S i as a a e al., 2014], 42 CHAPTER 5. DEEP LEARNING APPROACH de model_de ini ion( ocab_size, max_leng h, emb_ma ix): # ea u e ex ac o model img_inpu s = Inpu (shape=(4096,)) e1 = Dense(128, ke nel_ egula ize =l2(1e-8), ac i a ion=’ elu’, name=" ea u es")(img_inpu s) # desc ip ion model desc_inpu s = Inpu (shape=(max_leng h,)) de1 = Embedding( ocab_size, 100, weigh s=[emb_ma ix], ainable=False)(desc_inpu s) de2 = D opou (0.5, name=" ex _d opou ")(de1) de3 = LSTM(128, name="LSTM_laye ")(de2) # me ge model me ge_inpu s = [ e1,de3] me1 = add(me ge_inpu s) me2 = Dense(128, ac i a ion=’ elu’, name="decode ")(me1) ou pu s = Dense( ocab_size, ke nel_ egula ize =l2(1e-8), ac i a ion=’so max’, name="p edic ions")(me2) model = Model(inpu s=[img_inpu s, desc_inpu s], ou pu s=ou pu s) model.compile(loss=’ca ego ical_c ossen opy’, op imize =’adam’, me ics=[’accu acy’]) model.summa y() e u n model Figu e 5.8: Image desc ip ion gene a ion model de ini ion wi h Ke as. a egula iza ion echnique used o p e en o e i ing ha consis s in igno ing while ain- ing some neu ons selec ed andomly. This is ollowed by a Ke as LSTM laye wi h 128 memo y uni s. Nex we me ge bo h he ou pu o he LSTM laye and he ou pu o he Dense laye and apply a Dense laye wi h ReLU ac i a ion unc ion and a so max Dense laye , wi h L2 egula iza ion, o e he ocabula y size o make he p edic ion o he nex wo d. This model is ained wi h a ca ego ical c oss en opy loss (as he ou pu o he ne wo k a e he p obabili ies o he wo ds in he ocabula y o be he nex wo d in he desc ip ion) and Adam op imize (see Sec ion 5.5.1) wi h he de aul alues, as Tan i e al. sugges [Tan i e al., 2017b]. Finally, he las s ep is o i he model wi h he aining and alida ion da a ha we ha e loaded. As COCO is a e y big da ase and o ease memo y usage, we use a gene - a o o p oduce aining and alida ion da a and i he model wi h he i gene a o unc ion. To con ol he aining p ocess, we use he i gene a o unc ion wi h a ModelCheckpoin callback, o sa e he model a e each epoch moni o ing he loss on he alida ion se , and a Tenso Boa d callback, o sa e enso boa d logs o isualiza ion (as we explained in Sec ion 4.3). 5.5. TRAINING PROCESS 43 Figu e 5.9: Op imiza ion algo i hms s eps eaching local op imum. We ha e ained he ou models 35 epochs in he se e wi h he GPU. The whole p ocess (VGG16 ine- une, da a p epa a ion and aining o he ou models) las ed a ound eigh days. 5.5.1 Op imize s As we explained in Sec ion 3.4, Deep Lea ning models lea n by ying o ind he op i- mal pa ame e s ha minimize he cos unc ion and he g adien descen me hod is an op imiza ion algo i hm o app oxima e a local op imum by compu ing he g adien o he unc ion. Howe e , he cos unc ion depends on all aining samples, and he e o e so does he g adien , which makes i oo expensi e o compu e o Deep Lea ning models ha a e ypically ained on a big da ase . Fo his eason, as he cos unc ion is usually a sum o e he aining se , an inc emen al app oach o calcula ing he g adien called S ochas ic G adien Descen (SGD) was de eloped. I calcula es he g adien based only on a small subse o he aining examples o speed up his calcula ion. This usually in ol es aking mo e i e a ions o app oxima e he minimum, bu as e . In SGD, weigh s a e upda ed using he same lea ning a e o all weigh s and all i e a ions, making i di icul o slow o con e ge o he minimum i he lea ning a e is oo big o o small. Many a ian s ha e been p oposed and de eloped in o de o upda e his SGD algo i hm o y o each as e he local minimum. Some o his a ian s a e AdaG ad [Duchi e al., 2011], Adam [Kingma and Ba, 2014] and RMSP op. AdaG ad main ains a pe -pa ame e lea ning a e ins ead o ha ing he 44 CHAPTER 5. DEEP LEARNING APPROACH same lea ning a e o all he pa ame e s, upda ing i based on he sum o p e ious g adien alues. So does RMSP op, ha also has a lea ning a e o each pa ame e bu hese lea ning a es a e upda ed based on he a e age o ecen g adien calcula ions, ha is, based on how quickly he g adien is changing. Adam is an imp o emen o RMSP op ha upda es he lea ning a es based on he a e age bu also on he g adien s a iance, his is he one we use o aining ou model. Figu e 5.9, ob ained om Google Images, shows how quickly hese algo i hms educe he cos unc ion. As men ioned be o e, ou models ha e been ained using he Adam op imize . Chap e 6 E alua ion and esul s Once we ha e ou ou models de ined and ained, we can e alua e hem on he es se and we can use hem o gene a e desc ip ions o new unseen images. In his chap e , we explain how we ha e es ed he models, we de ine he me ics ha we ha e used and we show he esul s ob ained, including some examples o images and hei gene a ed desc ip ions. In Sec ion 5.4 we explained ha , as ou objec i e is o de e mine how well ou models pe o m on desc ibing images ha con ain people, he di ision o he da ase was made so ha he es se has only his kind o images. The e o e, he esul s o he e alua ion p ocess and he examples ha we show in his chap e e e all o images con aining people. 6.1 Tenso Boa d isualiza ion As desc ibed in Sec ion 5.1, he ou models ha we ha e de eloped ollow he same a chi ec u e bu di e on he echniques and da ase s used. The cha ac e is ics o hese ou models a e he ollowing: •Model 1: he mos gene al one. Uses a p e- ained VGG-16 CNN and i is ained on MS COCO da ase . •Model 2: uses a ine- uned e sion o VGG-16 and i is ained on MS COCO da ase . •Model 3: uses a p e- ained VGG-16 CNN and i is ained on a subse o MS COCO da ase o images con aining people. •Model 4: uses a ine- uned e sion o VGG-16 and i is ained on a subse o MS COCO da ase o images con aining people. 45 52 CHAPTER 6. EVALUATION AND RESULTS Model 1: pe son in he snow holding snowboa d down snow Model 3: man on skis in he snow Figu e 6.8: Good desc ip ions. Model 1: woman on beach wi h su boa d Model 3: wo woman iding su boa d on he beach Figu e 6.9: Good desc ip ions. 6.3. QUALITATIVE ANALYSIS 53 Model 1: man in baseball uni o m is holding ba Model 3: baseball playe is swinging ba a baseball game Figu e 6.10: Good desc ip ions. Model 1: bi d lying on beach Model 3: man is iding su boa d on he beach Figu e 6.11: Di e en pe spec i es. 54 CHAPTER 6. EVALUATION AND RESULTS Model 1: pizza wi h cheese and cheese on i Model 3: pe son is si ing a able wi h pizza Figu e 6.12: Di e en pe spec i es. Model 1: pla e o ood wi h mea and ege ables on i Model 3: pe son is si ing on able wi h some ood Figu e 6.13: Di e en pe spec i es. 6.3. QUALITATIVE ANALYSIS 55 Model 1: pe son in he snow on skis on he snow Model 3: man iding snowboa d down snow co e ed slope Figu e 6.14: Bad desc ip ions. Model 1: man holding ho dog in on o pizza Model 3: man is si ing a able wi h pizza Figu e 6.15: Bad desc ip ions. 56 CHAPTER 6. EVALUATION AND RESULTS Chap e 7 Conclusions and u u e wo k 7.1 Conclusions We conclude his p ojec ha ing accomplished he main objec i es es ablished a he beginning. We ha e s udied and lea ned di e en Deep Lea ning echniques and ools, ha a e nowadays e y impo an in he A i icial In elligence ield. We ha e also e iewed all he bibliog aphy ega ding he image desc ip ion gene a ion p oblem. We ha e ca ied ou his p ojec in a se e wi h a GPU p o ided by GAIA esea ch g oup, as aining Deep Lea ning models needs a lo o compu ing capaci y. We ha e used Ke as on op o Tenso low as he Deep Lea ning amewo k and Tenso Boa d o gene a ing plo s o he aining p ocess. We ha e also analyzed he di e en a ailable and well-known image da ase s in o de o selec he ones ha we e mo e sui able o ou p ojec . Finally, we ha e achie ed o de elop ou di e en Deep Lea ning models o au oma - ically gene a e desc ip ions o images con aining people. Ou ou models we e all buil using s a e-o - he-a Deep Lea ning echniques o Compu e Vision and o Na u al Lan- guage Gene a ion: Con olu ional Neu al Ne wo ks o ex ac ing he image ea u es and Recu en Neu al Ne wo ks o gene a ing he sen ence, as well as Feed o wa d Neu al Ne wo ks o building he models’ ou pu s. These models we e success ully ained and es ed on a big and popula image da ase . To e alua e ou models we ha e analyzed he Tenso Boa d plo s gene a ed om he aining da a, we ha e pe o med a quan i a i e analysis using he mos popula me ics and a quali a i e analysis gene a ing desc ip ions o unseen images in o de o isualize and con i m he esul s ob ained om he quan i a i e analysis. These e alua ion analyses lead all o he same conclusions: we ob ained desi able esul s om he e alua ion p ocess, simila o s a e-o - he-a esul s, o wo o he ou models de eloped (Model 1 and Model 3). Howe e , ine- uning was no a good p ac ice, as Model 2 and Model 4 ( he ones ha ine- uned he CNN) ha e poo e alua ion esul s, 57 58 CHAPTER 7. CONCLUSIONS AND FUTURE WORK maybe because o he di e ences be ween he da ase s used o ine- uning and aining. Ne e heless, Model 3 ha e sligh ly be e esul s han Model 1 so we can con i m ha ou hough s we e in he igh pa h a he beginning o he p ojec : images con aining people a e desc ibed di e en ly and so i is a good p ac ice o es ic he aining p ocess o his kind o images. 7.2 Fu u e wo k Conside ing concluded he main objec i es o his p ojec , he e a e many di ec ions in which u u e wo k could go in o de o b oaden i . Fi s o all, da a quan i y and quali y a e essen ial o a good aining, and hence he model could be mo e accu a e and ha e be e e alua ion esul s wi h mo e, and mo e ep esen a i e, examples in he da ase and labeled mo e p ecisely. Ano he op ion is o imp o e ou ha dwa e capaci y by using mo e GPUs, a be e one o by ano he way o ob aining mo e compu a ional powe in o de o being able o ain he models mo e apidly. In addi ion o all o hese changes o accomplish be e esul s wi h ou models, ano he di ec ion we could ake in he u u e is o b oaden he scope o he model’s objec i e. We could include mo e comple e image desc ip ions depic ing mo e complex cha ac e is ics o he pe son in he image, such as he pe son’s eelings (e.g. i i is smiling o sad), o o he cha ac e is ics ha a e no explici ly shown in he pic u e bu ha can be de i ed om i , no limi ing ou sel es o desc ibing wha i is s ic ly in he pic u e. Mo eo e , he model could be ex ended o he desc ip ion o , no only people, bu o he objec s appea ing in he pic u e, as well as he spa ial ela ionship be ween hose objec s and he people (e.g. he man behind he window, he woman nex o he blue chai ). The e a e also some eme ging asks ha combine bo h compu e ision and na u al language ields, ying o go beyond wha i is s ic ly depic ed in he image, ha could be in e es ing o s udy in a u u e wo k o his p ojec ; o example, he ask o VQA (Visual Ques ion Answe ing) [Wu e al., 2017a], ha ies o ind an answe o a gi en ques ion abou a gi en image, o isual s o y elling [Huang e al., 2016, Mos a azadeh e al., 2017], ha gene a es a na a i e desc ip ion om he inpu image by making subjec i e assump- ions o wha i is happening. Ano he in e es ing possibili y is o implemen a machine lea ning model, using s a e- o - he-a machine lea ning algo i hms such as xgboos , in o de o analyse he di e ences be ween bo h deep lea ning and machine lea ning models (compu ing ime, da ase needs, accu acy, e c). Finally, al hough Con olu ional Neu al Ne wo ks a e he s a e-o - he-a , hey ha e some limi a ions ha dec ease he le el o accu acy o he model. In he las mon hs, a new kind o neu al ne wo ks ying o sol e hese limi a ions, called Capsule Neu al Ne wo ks, ha e a isen. I would be a good u u e di ec ion o his p ojec o implemen 7.2. FUTURE WORK 59 he solu ion using his kind o ne wo k and o compa e i o he con olu ional one o see i i yields be e esul s. 7.2.1 CNN limi a ions Con olu ional Neu al Ne wo ks a e un il now he s a e-o - he-a app oach in he image analysis ield, bu hey ha e some limi a ions when i comes o spa ial ela ionships. Con olu ional Neu al Ne wo ks became a majo ad an age compa ed o adi ional eed o wa d ne wo ks in he image ield, as hey allowed a 2-D ma ix as he inpu o he model in con as wi h he need o la ening he pixel ma ix in o a ec o , which, along wi h he con olu ion and pooling ope a ions, pe mi ed he model o be in a iance o ansla ions. As we ha e al eady discussed du ing his p ojec , he job o he con olu ional laye s in a CNN is o de ec impo an ea u es in he image, wi h laye s close o he inpu de ec ing simple ea u es and deepe laye s close o he ou pu combining hem o de ec mo e complex ea u es. On he o he hand, pooling laye s, being max pooling he mos widely used, help educing he image dimensionali y, and he e o e compu a ional ime, and summa izing he impo an in o ma ion p esen in he image, as well as c ea ing spa ial in a iance. Howe e , his in a iance c ea ed by he max pooling laye s loses he exis ing spa ial ela ionships be ween all hese ea u es. I does no ake in o accoun how he di e en ea u es a e ela ed o each o he , since max pooling loses hei p ecise loca ions. This makes he model ou pu a alse posi i e when he image has he componen s o an objec bu no in he co ec o de . Fo example, conside ing i is a ace when i has wo eyes, a nose and a mou h, bu he mou h is whe e an eye should be and ice e sa. Fu he mo e, CNNs do no ake in o accoun he spa ial cha ac e is ics o each ea u e. Tha is, hey do no ecognise an objec hey ha e al eady seen, shown now in a di e en o ien a ion. Con inuing wi h he example abo e, an image wi h a ace u ned upside down is no de ec ed as a ace by he CNN. To comba his, con olu ional models a e ained by gi ing images o di e en possible angles explici ly o he ne wo k. Ne e heless, he solu ion is no o emo e he max pooling laye s om he model, as we eally need o in oduce some kind o in a iance; o he wise, he model will only ecognise images e y simila o he ones in he aining se . Wha a be e model needs is o adjus his in a iance wi h equi a iance, ha is, unde s anding o a ion and p opo ion changes and adap ing acco dingly. 7.2.2 Capsule neu al ne wo ks Geo ey E. Hin on, known as he a he o Deep Lea ning, has been discussing abou con olu ional ne wo ks’ limi a ions and he need o sugges new models o co e hem, o 60 CHAPTER 7. CONCLUSIONS AND FUTURE WORK Figu e 7.1: Capsule neu al ne wo k s uc u e (CapsNe ). a long ime now. In 2011 he published a pape [Hin on e al., 2011] ega ding a p oposal o including mo e complex spa ial cha ac e is ics o neu al ne wo ks, bu i has no been un il ecen ly ha he has inally published wo pape s [Hin on e al., 2018, Sabou e al., 2017] explaining his new sophis ica ed neu al ne wo k model, Capsule Ne wo ks. In he pape hey ha e published [Sabou e al., 2017], hey p opose CapsNe , a capsule neu al ne wo k model o he MNIST1da ase , epo ing be e esul s han CNNs and wi hou he need o da a augmen a ion. This model is ep esen ed in Figu e 7.1. Hin on belie es ha i is necessa y o p ese e pose’s ( ansla ion and o a ion) hie a - chy be ween ea u es. This way i is easie o a model o ecognise ano he pe spec i e o some hing i has al eady seen. These neu al ne wo ks y o o e come hose Con olu ional Ne wo ks’ limi a ions by explici ly aking in o accoun hose spa ial ela ionships be ween ea u es, whe eas CNNs do no ha e his 3-D space unde s anding. This is achie ed by eplacing he scala ou pu s o adi ional neu ons wi h ec o ou pu s encoding he ea u es’ loca ions and changing max-pooling wi h a dynamic ou ing algo i hm. Thus, an objec p esence is de i ed om no only he p esence o i s cons i uen pa s, bu also om hem being a he igh loca ions. Acco ding o Hin on, A capsule is a g oup o neu ons whose ou pu s ep esen di e en p ope ies o he same en i y. They a e nes ed laye s wi hin a laye , each one ocusing on de ec ing a pa icula ea u e in he image and ou pu ing a ec o ep esen ing he ea u e’s exis ence and i s pose p ope ies. The leng h o he ec o ep esen s he p obabili y o he exis ence o he ea u e and he spa ial p ope ies a e encoded in he ec o ’s di ec ion. This way, when he de ec ed ea u e mo es o changes i s spa ial s a e, he leng h o he ec o does no change ( he p obabili y emains he same) bu i changes i s o ien a ion. In hese neu al ne wo ks, capsules in a lowe laye decide dynamically how o send i s ou pu ec o o he nex laye ’s capsules. Each capsule in he lowe laye compu es, o each possible pa en , a p edic ion ec o o he pose o a highe -le el capsule ea u e 1Da ase o handw i en digi s. 7.2. FUTURE WORK 61 (wha he highe capsule would see), by mul iplying i s ou pu by a weigh ma ix. When se e al capsules in one laye ag ee on wha hey may ha e de ec ed, hey ac i a e he co esponding capsule a he nex laye . Hin on calls his me hod he ou ing-by-ag eemen algo i hm, and i subs i u es he max-pooling algo i hm. 68 CHAPTER 8. CONCLUSIONES Y TRABAJO A FUTURO Bibliog aphy [DBL, 2015] (2015). IEEE Con e ence on Compu e Vision and Pa e n Recogni ion, CVPR 2015, Bos on, MA, USA, June 7-12, 2015. IEEE Compu e Socie y. [Abadi e al., 2016] Abadi, M., Ba ham, P., Chen, J., Chen, Z., Da is, A., Dean, J., De in, M., Ghemawa , S., I ing, G., Isa d, M., Kudlu , M., Le enbe g, J., Monga, R., Moo e, S., Mu ay, D. G., S eine , B., Tucke , P. A., Vasude an, V., Wa den, P., Wicke, M., Yu, Y., and Zhang, X. (2016). Tenso low: A sys em o la ge-scale machine lea ning. CoRR, abs/1605.08695. [Al man, 1992] Al man, N. S. (1992). An in oduc ion o ke nel and nea es -neighbo nonpa ame ic eg ession. The Ame ican S a is ician, 46(3):175–185. [Baik and Bala, 2004] Baik, S. and Bala, J. W. (2004). A decision ee algo i hm o dis ibu ed da a mining: Towa ds ne wo k in usion de ec ion. In Lagan`a, A., Ga ilo a, M. L., Kuma , V., Mun, Y., Tan, C. J. K., and Ge asi, O., edi o s, Compu a ional Science and I s Applica ions - ICCSA 2004, In e na ional Con e ence, Assisi, I aly, May 14-17, 2004, P oceedings, Pa IV, olume 3046 o Lec u e No es in Compu e Science, pages 206–212. Sp inge . [Bay e al., 2006] Bay, H., Tuy elaa s, T., and Gool, L. J. V. (2006). SURF: speeded up obus ea u es. In Leona dis, A., Bischo , H., and Pinz, A., edi o s, Compu e Vision - ECCV 2006, 9 h Eu opean Con e ence on Compu e Vision, G az, Aus ia, May 7-13, 2006, P oceedings, Pa I, olume 3951 o Lec u e No es in Compu e Science, pages 404–417. Sp inge . [Bengio e al., 1994] Bengio, Y., Sima d, P. Y., and F asconi, P. (1994). Lea ning long- e m dependencies wi h g adien descen is di icul . IEEE T ans. Neu al Ne wo ks, 5(2):157–166. [Be na di e al., 2016] Be na di, R., C¸akici, R., Ellio , D., E dem, A., E dem, E., Ikizle - Cinbis, N., Kelle , F., Musca , A., and Plank, B. (2016). Au oma ic desc ip ion gen- e a ion om images: A su ey o models, da ase s, and e alua ion measu es. J. A i . In ell. Res., 55:409–442. 69 70 BIBLIOGRAPHY [B idge e al., 2014] B idge, J. P., Holden, S. B., and Paulson, L. C. (2014). Machine lea ning o i s -o de heo em p o ing - lea ning o selec a good heu is ic. J. Au om. Reasoning, 53(2):141–172. [Canny, 1986] Canny, J. F. (1986). A compu a ional app oach o edge de ec ion. IEEE T ans. Pa e n Anal. Mach. In ell., 8(6):679–698. [Chen e al., 2015] Chen, X., Fang, H., Lin, T., Vedan am, R., Gup a, S., Doll´a , P., and Zi nick, C. L. (2015). Mic oso COCO cap ions: Da a collec ion and e alua ion se e . CoRR, abs/1504.00325. [Chen and Zi nick, 2015] Chen, X. and Zi nick, C. L. (2015). Mind’s eye: A ecu en isual ep esen a ion o image cap ion gene a ion. In IEEE Con e ence on Compu e Vision and Pa e n Recogni ion, CVPR 2015, Bos on, MA, USA, June 7-12, 2015, pages 2422–2431. [Cho e al., 2014] Cho, K., an Me ienboe , B., G¨ul¸ceh e, C¸ ., Bahdanau, D., Bouga es, F., Schwenk, H., and Bengio, Y. (2014). Lea ning ph ase ep esen a ions using RNN encode -decode o s a is ical machine ansla ion. In [Moschi i e al., 2014], pages 1724–1734. [Chung e al., 2014] Chung, J., G¨ul¸ceh e, C¸ ., Cho, K., and Bengio, Y. (2014). Em- pi ical e alua ion o ga ed ecu en neu al ne wo ks on sequence modeling. CoRR, abs/1412.3555. [Collobe e al., 2011] Collobe , R., Ka ukcuoglu, K., and Fa abe , C. (2011). To ch7: A ma lab-like en i onmen o machine lea ning. [Co es and Vapnik, 1995] Co es, C. and Vapnik, V. (1995). Suppo - ec o ne wo ks. Machine Lea ning, 20(3):273–297. [Dean e al., 2012] Dean, J., Co ado, G. S., Monga, R., Chen, K., De in, M., Le, Q. V., Mao, M. Z., Ranza o, M., Senio , A., Tucke , P., Yang, K., and Ng, A. Y. (2012). La ge scale dis ibu ed deep ne wo ks. In NIPS. [Deng e al., 2009] Deng, J., Dong, W., Soche , R., Li, L., Li, K., and Li, F. (2009). Imagene : A la ge-scale hie a chical image da abase. In 2009 IEEE Compu e Socie y Con e ence on Compu e Vision and Pa e n Recogni ion (CVPR 2009), 20-25 June 2009, Miami, Flo ida, USA, pages 248–255. IEEE Compu e Socie y. [Denkowski and La ie, 2014] Denkowski, M. J. and La ie, A. (2014). Me eo uni e sal: Language speci ic ansla ion e alua ion o any a ge language. In P oceedings o he Nin h Wo kshop on S a is ical Machine T ansla ion, WMT@ACL 2014, June 26- 27, 2014, Bal imo e, Ma yland, USA, pages 376–380. The Associa ion o Compu e Linguis ics. [Donahue e al., 2015] Donahue, J., Hend icks, L. A., Guada ama, S., Roh bach, M., Venugopalan, S., Da ell, T., and Saenko, K. (2015). Long- e m ecu en con olu ional ne wo ks o isual ecogni ion and desc ip ion. In [DBL, 2015], pages 2625–2634. BIBLIOGRAPHY 71 [Duchi e al., 2011] Duchi, J. C., Hazan, E., and Singe , Y. (2011). Adap i e subg adien me hods o online lea ning and s ochas ic op imiza ion. Jou nal o Machine Lea ning Resea ch, 12:2121–2159. [E e ingham e al., 2010] E e ingham, M., Van Gool, L., Williams, C. K. I., Winn, J., and Zisse man, A. (2010). The pascal isual objec classes ( oc) challenge. In e na ional Jou nal o Compu e Vision, 88(2):303–338. [Goldbe g e al., 1994] Goldbe g, E., D iedge , N., and Ki edge, R. I. (1994). Using na u al-language p ocessing o p oduce wea he o ecas s. IEEE Expe , 9(2):45–53. [G a es e al., 2013] G a es, A., Mohamed, A., and Hin on, G. E. (2013). Speech ecogni- ion wi h deep ecu en neu al ne wo ks. In IEEE In e na ional Con e ence on Acous- ics, Speech and Signal P ocessing, ICASSP 2013, Vancou e , BC, Canada, May 26-31, 2013, pages 6645–6649. IEEE. [Ha is and S ephens, 1988] Ha is, C. G. and S ephens, M. (1988). A combined co ne and edge de ec o . In Taylo , C. J., edi o , P oceedings o he Al ey Vision Con e ence, AVC 1988, Manches e , UK, Sep embe , 1988, pages 1–6. Al ey Vision Club. [Hend icks e al., 2016] Hend icks, L. A., Venugopalan, S., Roh bach, M., Mooney, R. J., Saenko, K., and Da ell, T. (2016). Deep composi ional cap ioning: Desc ibing no el objec ca ego ies wi hou pai ed aining da a. In 2016 IEEE Con e ence on Compu e Vision and Pa e n Recogni ion, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 1–10. [Hin on e al., 2011] Hin on, G. E., K izhe sky, A., and Wang, S. D. (2011). T ans o m- ing au o-encode s. In Honkela, T., Duch, W., Gi olami, M. A., and Kaski, S., edi o s, A i icial Neu al Ne wo ks and Machine Lea ning - ICANN 2011 - 21s In e na ional Con e ence on A i icial Neu al Ne wo ks, Espoo, Finland, June 14-17, 2011, P oceed- ings, Pa I, olume 6791 o Lec u e No es in Compu e Science, pages 44–51. Sp inge . [Hin on e al., 2018] Hin on, G. E., Sabou , S., and F oss , N. (2018). Ma ix capsules wi h EM ou ing. In In e na ional Con e ence on Lea ning Rep esen a ions. [Ho, 1995] Ho, T. K. (1995). Random decision o es s. In Thi d In e na ional Con e ence on Documen Analysis and Recogni ion, ICDAR 1995, Augus 14 - 15, 1995, Mon eal, Canada. Volume I, pages 278–282. IEEE Compu e Socie y. [Hoch ei e and Schmidhube , 1997] Hoch ei e , S. and Schmidhube , J. (1997). Long sho - e m memo y. Neu al Compu a ion, 9(8):1735–1780. [Hodosh e al., 2013] Hodosh, M., Young, P., and Hockenmaie , J. (2013). F aming image desc ip ion as a anking ask: Da a, models and e alua ion me ics. J. A i . In ell. Res., 47:853–899. [Ho elling, 1933] Ho elling, H. (1933). Analysis o a complex o s a is ical a iables wi h p incipal componen s. Jou nal o Educa ional Psychology, 24:417–441. 72 BIBLIOGRAPHY [Huang e al., 2016] Huang, T. K., Fe a o, F., Mos a azadeh, N., Mis a, I., Ag awal, A., De lin, J., Gi shick, R. B., He, X., Kohli, P., Ba a, D., Zi nick, C. L., Pa ikh, D., Vande wende, L., Galley, M., and Mi chell, M. (2016). Visual s o y elling. In Knigh , K., Nenko a, A., and Rambow, O., edi o s, NAACL HLT 2016, The 2016 Con e ence o he No h Ame ican Chap e o he Associa ion o Compu a ional Linguis ics: Human Language Technologies, San Diego Cali o nia, USA, June 12-17, 2016, pages 1233–1239. The Associa ion o Compu a ional Linguis ics. [Io danskaja e al., 1992] Io danskaja, L., Kim, M., Ki edge, R. I., La oie, B., and Polgu`e e, A. (1992). Gene a ion o ex ended bilingual s a is ical epo s. In 14 h In- e na ional Con e ence on Compu a ional Linguis ics, COLING 1992, Nan es, F ance, Augus 23-28, 1992, pages 1019–1023. [Jia e al., 2014] Jia, Y., Shelhame , E., Donahue, J., Ka aye , S., Long, J., Gi shick, R., Guada ama, S., and Da ell, T. (2014). Ca e: Con olu ional a chi ec u e o as ea u e embedding. a Xi p ep in a Xi :1408.5093. [Ka pa hy and Li, 2015] Ka pa hy, A. and Li, F. (2015). Deep isual-seman ic alignmen s o gene a ing image desc ip ions. In [DBL, 2015], pages 3128–3137. [Kingma and Ba, 2014] Kingma, D. P. and Ba, J. (2014). Adam: A me hod o s ochas ic op imiza ion. CoRR, abs/1412.6980. [Ki os e al., 2014a] Ki os, R., Salakhu dino , R., and Zemel, R. S. (2014a). Mul imodal neu al language models. In P oceedings o he 31 h In e na ional Con e ence on Machine Lea ning, ICML 2014, Beijing, China, 21-26 June 2014, pages 595–603. [Ki os e al., 2014b] Ki os, R., Salakhu dino , R., and Zemel, R. S. (2014b). Uni y- ing isual-seman ic embeddings wi h mul imodal neu al language models. CoRR, abs/1411.2539. [K izhe sky, 2012] K izhe sky, A. (2012). Lea ning mul iple laye s o ea u es om iny images. [K izhe sky e al., 2012] K izhe sky, A., Su ske e , I., and Hin on, G. E. (2012). Ima- gene classi ica ion wi h deep con olu ional neu al ne wo ks. In Ba le , P. L., Pe ei a, F. C. N., Bu ges, C. J. C., Bo ou, L., and Weinbe ge , K. Q., edi o s, Ad ances in Neu al In o ma ion P ocessing Sys ems 25: 26 h Annual Con e ence on Neu al In o - ma ion P ocessing Sys ems 2012. P oceedings o a mee ing held Decembe 3-6, 2012, Lake Tahoe, Ne ada, Uni ed S a es., pages 1106–1114. [Kulka ni e al., 2013] Kulka ni, G., P em aj, V., O donez, V., Dha , S., Li, S., Choi, Y., Be g, A. C., and Be g, T. L. (2013). Baby alk: Unde s anding and gene a ing simple image desc ip ions. IEEE T ans. Pa e n Anal. Mach. In ell., 35(12):2891–2903. [Kuzne so a e al., 2012] Kuzne so a, P., O donez, V., Be g, A. C., Be g, T. L., and Choi, Y. (2012). Collec i e gene a ion o na u al image desc ip ions. In The 50 h BIBLIOGRAPHY 73 Annual Mee ing o he Associa ion o Compu a ional Linguis ics, P oceedings o he Con e ence, July 8-14, 2012, Jeju Island, Ko ea - Volume 1: Long Pape s, pages 359– 368. The Associa ion o Compu e Linguis ics. [LeCun e al., 2015] LeCun, Y., Bengio, Y., and Hin on, G. E. (2015). Deep lea ning. Na u e, 521(7553):436–444. [Li e al., 2011] Li, S., Kulka ni, G., Be g, T. L., Be g, A. C., and Choi, Y. (2011). Com- posing simple image desc ip ions using web-scale n-g ams. In Goldwa e , S. and Man- ning, C. D., edi o s, P oceedings o he Fi een h Con e ence on Compu a ional Na u al Language Lea ning, CoNLL 2011, Po land, O egon, USA, June 23-24, 2011, pages 220–228. ACL. [Lin, 2004] Lin, C.-Y. (2004). Rouge: a package o au oma ic e alua ion o summa ies. [Lin e al., 2014] Lin, T., Mai e, M., Belongie, S. J., Bou de , L. D., Gi shick, R. B., Hays, J., Pe ona, P., Ramanan, D., Doll´a , P., and Zi nick, C. L. (2014). Mic oso COCO: common objec s in con ex . CoRR, abs/1405.0312. [Lowe, 1999] Lowe, D. G. (1999). Objec ecogni ion om local scale-in a ian ea u es. In ICCV, pages 1150–1157. [Mao e al., 2014] Mao, J., Xu, W., Yang, Y., Wang, J., and Yuille, A. L. (2014). Deep cap ioning wi h mul imodal ecu en neu al ne wo ks (m- nn). CoRR, abs/1412.6632. [Mason and Cha niak, 2014] Mason, R. and Cha niak, E. (2014). Nonpa ame ic me hod o da a-d i en image cap ioning. In P oceedings o he 52nd Annual Mee ing o he Associa ion o Compu a ional Linguis ics, ACL 2014, June 22-27, 2014, Bal imo e, MD, USA, Volume 2: Sho Pape s, pages 592–598. The Associa ion o Compu e Linguis ics. [Mille , 1995] Mille , G. A. (1995). Wo dne : A lexical da abase o english. Commun. ACM, 38(11):39–41. [Mi chell e al., 2012] Mi chell, M., Dodge, J., Goyal, A., Yamaguchi, K., S a os, K., Han, X., Mensch, A., Be g, A. C., Be g, T. L., and III, H. D. (2012). Midge: Gene a ing image desc ip ions om compu e ision de ec ions. In Daelemans, W., Lapa a, M., and M`a quez, L., edi o s, EACL 2012, 13 h Con e ence o he Eu opean Chap e o he Associa ion o Compu a ional Linguis ics, A ignon, F ance, Ap il 23-27, 2012, pages 747–756. The Associa ion o Compu e Linguis ics. [Moschi i e al., 2014] Moschi i, A., Pang, B., and Daelemans, W., edi o s (2014). P o- ceedings o he 2014 Con e ence on Empi ical Me hods in Na u al Language P ocessing, EMNLP 2014, Oc obe 25-29, 2014, Doha, Qa a , A mee ing o SIGDAT, a Special In e es G oup o he ACL. ACL. [Mos a azadeh e al., 2017] Mos a azadeh, N., B ocke , C., Dolan, B., Galley, M., Gao, J., Spi hou akis, G. P., and Vande wende, L. (2017). Image-g ounded con e sa ions: 74 BIBLIOGRAPHY Mul imodal con ex o na u al ques ion and esponse gene a ion. In Kond ak, G. and Wa anabe, T., edi o s, P oceedings o he Eigh h In e na ional Join Con e ence on Na u al Language P ocessing, IJCNLP 2017, Taipei, Taiwan, No embe 27 - Decembe 1, 2017 - Volume 1: Long Pape s, pages 462–472. Asian Fede a ion o Na u al Language P ocessing. [Moze , 1989] Moze , M. C. (1989). A ocused backp opaga ion algo i hm o empo al pa e n ecogni ion. Complex Sys ems, 3(4). [O donez e al., 2011] O donez, V., Kulka ni, G., and Be g, T. L. (2011). Im2 ex : De- sc ibing images using 1 million cap ioned pho og aphs. In Shawe-Taylo , J., Zemel, R. S., Ba le , P. L., Pe ei a, F. C. N., and Weinbe ge , K. Q., edi o s, Ad ances in Neu al In o ma ion P ocessing Sys ems 24: 25 h Annual Con e ence on Neu al In o - ma ion P ocessing Sys ems 2011. P oceedings o a mee ing held 12-14 Decembe 2011, G anada, Spain., pages 1143–1151. [Pang e al., 2002] Pang, B., Lee, L., and Vai hyana han, S. (2002). Thumbs up?: Sen- imen classi ica ion using machine lea ning echniques. In P oceedings o he ACL-02 Con e ence on Empi ical Me hods in Na u al Language P ocessing - Volume 10, EMNLP ’02, pages 79–86, S oudsbu g, PA, USA. Associa ion o Compu a ional Linguis ics. [Papineni e al., 2002] Papineni, K., Roukos, S., Wa d, T., and Zhu, W. (2002). Bleu: a me hod o au oma ic e alua ion o machine ansla ion. In P oceedings o he 40 h Annual Mee ing o he Associa ion o Compu a ional Linguis ics, July 6-12, 2002, Philadelphia, PA, USA., pages 311–318. ACL. [Penning on e al., 2014] Penning on, J., Soche , R., and Manning, C. D. (2014). Glo e: Global ec o s o wo d ep esen a ion. In [Moschi i e al., 2014], pages 1532–1543. [Rei e and Dale, 1997] Rei e , E. and Dale, R. (1997). Building applied na u al language gene a ion sys ems. Na u al Language Enginee ing, 3(1):57–87. [Russako sky e al., 2015] Russako sky, O., Deng, J., Su, H., K ause, J., Sa heesh, S., Ma, S., Huang, Z., Ka pa hy, A., Khosla, A., Be ns ein, M. S., Be g, A. C., and Li, F. (2015). Imagene la ge scale isual ecogni ion challenge. In e na ional Jou nal o Compu e Vision, 115(3):211–252. [Russell e al., 2008] Russell, B. C., To alba, A., Mu phy, K. P., and F eeman, W. T. (2008). Labelme: A da abase and web-based ool o image anno a ion. In e na ional Jou nal o Compu e Vision, 77(1-3):157–173. [Sabou e al., 2017] Sabou , S., F oss , N., and Hin on, G. E. (2017). Dynamic ou ing be ween capsules. In Guyon, I., on Luxbu g, U., Bengio, S., Wallach, H. M., Fe gus, R., Vishwana han, S. V. N., and Ga ne , R., edi o s, Ad ances in Neu al In o ma ion P ocessing Sys ems 30: Annual Con e ence on Neu al In o ma ion P ocessing Sys ems 2017, 4-9 Decembe 2017, Long Beach, CA, USA, pages 3859–3869. BIBLIOGRAPHY 75 [Sa ikaya e al., 2014] Sa ikaya, R., Hin on, G. E., and Deo as, A. (2014). Applica ion o deep belie ne wo ks o na u al language unde s anding. IEEE/ACM T ans. Audio, Speech & Language P ocessing, 22(4):778–784. [Seide and Aga wal, 2016] Seide, F. and Aga wal, A. (2016). Cn k: Mic oso ’s open- sou ce deep-lea ning oolki . In P oceedings o he 22Nd ACM SIGKDD In e na ional Con e ence on Knowledge Disco e y and Da a Mining, KDD ’16, pages 2135–2135, New Yo k, NY, USA. ACM. [S i as a a e al., 2014] S i as a a, N., Hin on, G. E., K izhe sky, A., Su ske e , I., and Salakhu dino , R. (2014). D opou : a simple way o p e en neu al ne wo ks om o e i ing. Jou nal o Machine Lea ning Resea ch, 15(1):1929–1958. [Tan i e al., 2017a] Tan i, M., Ga , A., and Camille i, K. P. (2017a). Wha is he ole o ecu en neu al ne wo ks ( nns) in an image cap ion gene a o ? In P oceedings o he 10 h In e na ional Con e ence on Na u al Language Gene a ion, INLG 2017, San iago de Compos ela, Spain, Sep embe 4-7, 2017, pages 51–60. [Tan i e al., 2017b] Tan i, M., Ga , A., and Camille i, K. P. (2017b). Whe e o pu he image in an image cap ion gene a o . CoRR, abs/1703.09137. [Theano De elopmen Team, 2016] Theano De elopmen Team (2016). Theano: A Py hon amewo k o as compu a ion o ma hema ical exp essions. a Xi e-p in s, abs/1605.02688. [T an e al., 2016] T an, K., He, X., Zhang, L., and Sun, J. (2016). Rich image cap ioning in he wild. In 2016 IEEE Con e ence on Compu e Vision and Pa e n Recogni ion Wo kshops, CVPR Wo kshops 2016, Las Vegas, NV, USA, June 26 - July 1, 2016, pages 434–441. IEEE Compu e Socie y. [Vedan am e al., 2014] Vedan am, R., Zi nick, C. L., and Pa ikh, D. (2014). Cide : Consensus-based image desc ip ion e alua ion. CoRR, abs/1411.5726. [Vinyals e al., 2015] Vinyals, O., Toshe , A., Bengio, S., and E han, D. (2015). Show and ell: A neu al image cap ion gene a o . In [DBL, 2015], pages 3156–3164. [We nick e al., 2010] We nick, M. N., Yang, Y., B anko , J. G., You gano , G., and S o he , S. C. (2010). Machine lea ning in medical imaging. IEEE Signal P ocessing Magazine, 27(4):25–38. [Wu e al., 2017a] Wu, Q., Teney, D., Wang, P., Shen, C., Dick, A. R., and an den Hengel, A. (2017a). Visual ques ion answe ing: A su ey o me hods and da ase s. Compu e Vision and Image Unde s anding, 163:21–40. [Wu e al., 2017b] Wu, S., Wieland, J., Fa i a , O., and Schille , J. (2017b). Au oma ic al - ex : Compu e -gene a ed image desc ip ions o blind use s on a social ne wo k se ice. In P oceedings o he 2017 ACM Con e ence on Compu e Suppo ed Coope a i e Wo k and Social Compu ing, CSCW ’17, pages 1180–1192, New Yo k, NY, USA. ACM. 76 BIBLIOGRAPHY [Wu e al., 2016] Wu, Y., Schus e , M., Chen, Z., Le, Q. V., No ouzi, M., Mache ey, W., K ikun, M., Cao, Y., Gao, Q., Mache ey, K., Klingne , J., Shah, A., Johnson, M., Liu, X., Kaise , L., Gouws, S., Ka o, Y., Kudo, T., Kazawa, H., S e ens, K., Ku ian, G., Pa il, N., Wang, W., Young, C., Smi h, J., Riesa, J., Rudnick, A., Vinyals, O., Co ado, G., Hughes, M., and Dean, J. (2016). Google’s neu al machine ansla ion sys em: B idging he gap be ween human and machine ansla ion. CoRR, abs/1609.08144. [Yang e al., 2011] Yang, Y., Teo, C. L., III, H. D., and Aloimonos, Y. (2011). Co pus- guided sen ence gene a ion o na u al images. In P oceedings o he 2011 Con e ence on Empi ical Me hods in Na u al Language P ocessing, EMNLP 2011, 27-31 July 2011, John McIn y e Con e ence Cen e, Edinbu gh, UK, A mee ing o SIGDAT, a Special In e es G oup o he ACL, pages 444–454. ACL. [Young e al., 2014] Young, P., Lai, A., Hodosh, M., and Hockenmaie , J. (2014). F om image desc ip ions o isual deno a ions: New simila i y me ics o seman ic in e ence o e e en desc ip ions. TACL, 2:67–78.