Repositorio Institucional de Documentos

A machine lea ning app oach o ansien imaging econs uc ion Miguel Ángel Cosculluela G acia T abajo de in de Más e en Modelización e In es igación Ma emá ica, Es adís ica y Compu ación Uni e sidad de Za agoza Di ec o es del abajo: D . Julio Ma co Mu ia y P o . D . Diego Gu ié ez Pé ez 9 de eb e o de 2021 Abs ac The ecen ad ances in non-line-o -sigh imaging ha e made i possible o econs uc scenes hidden a ound a co ne , wi h po en ial applica ions in e.g. au onomous d i ing o medical imaging. By ope a ing a ame a es compa able o he speed o ligh , ecen i ual-wa e p opaga ion me hods le e age he empo al oo p in o indi ec ligh anspo a a isible auxilia y su ace o ake i ual pho os o objec s hidden om he obse e . Despi e hese ad ances, hese me hods ha e a c i ical compu a ional bo leneck: The econs uc ion qual- i y and he compu a ional pe o mance a e highly dependen on he esolu ion o he cap u e g id, which is ypically disc e ized in space and ime, leading o high p ocessing and memo y cons ain s. Inspi ed by ecen machine lea ning echniques, in his wo k we p opose a new compu- a ional imaging me hod o add ess hese limi a ions. Fo his pu pose we p opose o lea n implici ep esen a ions o he cap u ed da a using neu al ne wo ks, allowing us o con e he disc e e space o he cap u ed da a in o a con inuous one. Howe e , wo king di ec ly wi h he cap u ed da a is a complex ask due o i s huge size and i s high dynamic ange alues. In o de o a oid hese p oblems, we le e age ecen wa e-based phaso - ield imaging me hods o ans o m he ime- esol ed cap u ed da a in o se s o 2D complex- alued ields (i.e. phaso ields) a di e en equencies, which p o ides a mo e a o able ep esen a ion o machine lea ning me hods. Unde ou implici ep esen a ion o mula ion, we analyze he pe o mance o di e en neu- al ne wo k models o ep esen he complex s uc u e o phaso ields, s a ing om simple ep esen a ions, and i e a i ely p o iding mo e powe ul models o add suppo o he com- plexi y o he da a. We demons a e how ecen machine lea ning echniques based on mul i- laye pe cep ons wi h sine ac i a ion unc ions a e capable o ep esen ing phaso ields ana- ly ically in bo h spa ial and empo al equency domains, and in eg a e hem in o he phaso - ield amewo k o econs uc hidden geome y. We inally es his neu al model in di e en scenes, and measu e i s pe o mance a highe esolu ions no seen by he cap u ed da a. We show how he model is able o analy ically upsample all dimensions, and demons a e how ou implici ep esen a ion addi ionally wo ks as a denoise o he sou ce disc e ized phaso ield. iii i Abs ac Resumen Los ecien es a ances en imagen non-line-o -sigh han hecho posible la econs ucción de es- cenas ocul as a a és de una esquina, con la po encial aplicación en conducción au ónoma o imagen medica. Al ope a con o og amas po segundo ce canos a la elocidad de la luz, e- cien es mé odos de p opagación i ual de ondas ap o echan la huella empo al del anspo e de luz en una supe icie auxilia pa a oma o ones i uales de obje os ocul os al obse ado . A pesa de es os a ances, es os mé odos ienen un cuello de bo ella c i ico: La calidad econ- s ucción y el cos e de compu o son al amen e dependien es de la esolución de la malla de cap u a, la cual suele es a disc e izada en espacio y iempo, lo que conlle a g andes limi a- ciones de p ocesamien o y memo ia. Inspi ados en las ecien es écnicas de ap endizaje au omá ico, en es e abajo p oponemos un nue o mé odo de imagen compu acional pa a hace en e a es as limi aciones. Pa a es e p opósi o p oponemos ap ende ep esen aciones implíci as de los da os cap u ados usando edes neu onales, pe mi iéndonos con e i el espacio disc e o de los da os cap u ados en un espacio con inuo. Sin emba go, abaja di ec amen e con los da os cap u ados es una a ea compleja debido a su g an amaño y al al o ango dinámico de sus alo es. Pa a e i a es os p oblemas ap o echamos el ecien e mé odo de imagen de los campos de aso es basados en ondas pa a ans o ma los da os cap u ados esuel os en iempo en un conjun o de campos 2D de alo es complejos (como son los campos de aso es) a di e en es ecuencias, lo cual p o ee una ep esen ación más a o able pa a los mé odos de ap endizaje au omá ico. Siguiendo nues a o mulación de ep esen ación implíci a, hemos analizado el endimien o de di e en es modelos de edes neu onales pa a ep esen a la compleja es uc u a de los cam- pos de aso es, empezando po ep esen aciones simples, y p opo cionando de o ma i e a i a modelos más po en es pa a añadi sopo e pa a la complejidad de los da os. Demos amos como las écnicas ecien es de ap endizaje au omá ico basadas en p ecep ones mul icapa con unciones de ac i ación sinusoidales son capaces de ep esen a un campo de aso es analí ica- men e en los dominios espaciales y empo ales, e in eg a las den o del ma co de los campos de aso es pa a econs ui geome ía ocul a. Finalmen e p obamos es e modelo de ed neu onal con di e en es escenas y medimos su desempeño con mayo es esoluciones que no han sido usadas en el en enamien o. Mos amos como el modelo es capaz de gene a más mues as en odas las dimensiones y demos amos como nues a ep esen ación implíci a además unciona como un mé odo pa a elimina uido del campo de aso es disc e izado. i Resumen To my amily and iends in he labo a o y ha helped me du ing he ealiza ion o his mas e ’s hesis. I would like o hank my di ec o s, who o ien a ed me, and my pa ne who suppo ed me du ing his ime and who has gi en me he s eng h o keep on. ii Con en s Abs ac iii Resumen 1 In oduc ion 1 1.1 ThesisBackg ound ................................ 3 2 Rela ed Wo k 5 2.1 Non-line-o -sigh (NLOS) ansien imaging . . . . . . . . . . . . . . . . . . . 5 2.2 T ansien ende ing................................ 6 2.3 Upsamplingme hods ............................... 6 3 Backg ound 9 3.1 Neu al ne wo k op imiza ion . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 3.1.1 Mul ilaye pe cep on . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3.1.2 Ac i a ion unc ions . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.1.3 G adien descen op imiza ion . . . . . . . . . . . . . . . . . . . . . . 11 3.1.4 S ochas ic g adien descen . . . . . . . . . . . . . . . . . . . . . . . . 12 3.2 Non-line-o -sigh (NLOS) ansien imaging . . . . . . . . . . . . . . . . . . . 12 3.2.1 Phaso ields ............................... 13 4 Implici ep esen a ion o phaso ields 15 4.1 Neu al implici ep esen a ion o phaso ields . . . . . . . . . . . . . . . . . . 17 4.2 Da a ans o ma ion................................ 21 5 Resul s 25 6 Conclusions and u u e wo k 31 ix 6Chap e 2. Rela ed Wo k 2.2 T ansien ende ing In NLOS one o he mos aluable ools ha help o es and de elop new me hods is he sim- ula ion o he cap u e sys em and NLOS scenes and, in ou case, he possibili y o gene a e da a o machine lea ning algo i hms. The simula ion is done wi h a ende ing p ocess. Con- c e ely, he ende ing p ocess simula es he ligh and i s in e ac ions wi h he ma e in a i ual scene and cap u e he ligh wi h i ual senso s. The s anda d ende ing p ocess used in mos o he applica ions such as cinema o ideo games gene a es images in 2D wi h he ligh in he scene in eg a ed. This ype o ende is known as s eady-s a e ende ing. O he ype is he ansien ende ing me hods which allow simula ing how he ligh mo es h ough a scene, ende ing ideos a ame a es compa able o he speed o ligh . Howe e , adding he empo al dimension in ansien ende ing is no always possible. This is because some applica ions and me hods de eloped o s eady-s a e ende ing canno be applied easily o , in some cases, a all. To implemen a ansien ende e some wo ks p esen ed new ime- esol ed ligh anspo equa ions [24, 25]. O he wo ks gene alize Mon e-Ca lo me hods o s eady-s a e ende ing as pa h acing o a ime- esol ed e sion [26, 13]. O he algo i hms as pho on mapping we e also gene alized o ime- esol ed e sions [27]. The use and esea ch in ansien ende ing ha e g own wi h he de elopmen o NLOS an- sien imaging ield. I s use is ex emely use ul in he NLOS ield due o i helps o de elop and es NLOS me hods. This is due o he possibili y o gene a ing ansien p o ile in con olled condi ions wi hou he es ic ions o eal cap u e ha dwa e and use hem as e e ence da a. Following his las idea i is possible o gene a e da ase s and g ound u h o deep lea ning me hods. Fo example he wo k by Ma co e al. [28] uses a ansien ende e o gene a e a ime-o - ligh (TOF) da ase and ain a deep lea ning model o educe he e o in TOF cam- e as. O he in e es ing wo k by Liang e al. [29] p esen ed a comp ession me hod o ansien p o iles wi h a deep lea ning model ained wi h syn he ic ansien da a. Also, he wo k by Galindo e al. [30] p esen ed a public da ase wi h NLOS scenes. In con as , some NLOS me hods do no use he comple e ligh pa hs bu only he i s bounce [31, 32, 33]. This ype o ende ing is as e and he wo k by Chopi e e al. [34] uses his ype o ende ing o gene a e a bigge da ase and ain a deep lea ning model o econs uc NLOS scenes om hei ansien p o iles. 2.3 Upsampling me hods A cap u e p ocess by de ini ion canno ob ain he eal shape o he space bu only ge a ce ain numbe o samples (which is known as he esolu ion). Ha ing a highe esolu ion gi es mo e accu acy o he measu ed space. Howe e , he esolu ion is ypically limi ed by he cap u e ha dwa e o physical es ic ions in he cap u e p ocess. When he esolu ion is la ge enough, he o iginal shape can be ob ained wi h a simple linea in e pola ion because he changes be- ween samples p ac ically ollow a s aigh line. Bu in mos cases he esolu ion is oo low and ob aining he o iginal shape is a ha d p oblem. Wi h he e olu ion o neu al ne wo ks se - e al wo ks ha e demons a ed hei capabili ies o unde s anding he s uc u e o he da a and being able o upsample hem. Fo his p opose, he e a e wo main app oxima ions. The i s app oach consis s in upsampling o a highe esolu ion o a ixed size o scale a e (e.g. dupli- ca ing esolu ion). The second one uses he neu al ne wo k o, o a conc e e inpu like spa ial coo dina es, ime o equency, ge he alue o he o iginal da a o , i he inpu is p e iously unknown, hallucina e new samples. A machine lea ning app oach o ansien imaging econs uc ion 7 The i s app oach is commonly used in image supe - esolu ion. Wi h he idea o upsampling o ixed sizes, se e al neu al ne wo ks models ha e been de eloped om models which only need one low esolu ion image as an inpu [8] o models which le e age he in o ma ion om mul iple images such as ideo ames [35]. I he upsampling is done o e he empo al domain ins ead o he spa ial one he e ec is supe -slow mo ion ideos [36]. The second app oach is mo e lexible han he i s one because i is possible o sample in conc e e zones o a any desi ed esolu ion. Howe e , hey ha e he cons ha hei models a e no gene al and, o each scene, a new aining is equi ed. T aining and sampling a model ha e wo main ways o be implemen ed. In he i s way he model has as an inpu he scene o da a (o a ans o ma ion o hem) and he coo dina es o be sampled. This case ha e been es ed o comp ession models o ex u es images, ob aining a la en space o hem. Then he au ho s can decomp ess he la en space by sampling wi h neu al ne wo ks [37, 9] o coo di- na es in he o iginal ex u es o o new ones. The o he way i s he scene di ec ly in he neu al ne wo k. This is known as implici ep esen a ion and has he ad an age ha , once he neu al ne wo k is ained, he aining da a is no mo e used. This ype o ne wo ks has as an inpu he di e en ypes o coo dina es (spa ial, empo al, angula , e c.) and, as he ou pu , he co - esponding alues o ha coo dina es. Like he p e ious me hods, his ype o models can be used o gene a e samples in p e ious unknown posi ions. A use ul case is lea ning 3D scenes [38] which, in ende applica ions, a oids he equi emen o ende ing new iews. Close o ou app oach he e a e ecen simila wo ks which simpli y he aining p ocess and imp o e he esul s [10, 11] p esen ing a new me hod o ans o m he coo dina es and imp o ing sig- ni ican ly he esul s. Fu he mo e, he wo k o Si zmann e al. [12] shows he possibili ies o using pe iodical unc ions ins ead o he classical ac i a ion one o lea ning high equency de ails in se e al implici ep esen a ion p oblems such as image o ideo ep esen a ion. 8Chap e 2. Rela ed Wo k Chap e 3 Backg ound This chap e desc ibes he main ma hema ical, physical, and compu a ional aspec s ha his wo k builds upon. Fi s ly we in oduce a basic knowledge on neu al ne wo k explaining hem and he model used in his wo k wi h he mos common op imiza ion me hod used wi h neu al ne wo ks. Secondly we in oduce he non-line-o -sigh p oblem and explain he heo y o phaso ields, a s a e-o - he-a me hod o non-line-o -sigh econs uc ion. 3.1 Neu al ne wo k op imiza ion Neu al ne wo ks a e lea ning sys ems inspi ed in human neu ons and hei in e connec ions. They a e able o lea n a ans o ma ion and unde s and he ela ionship be ween he inpu and ou pu da a used in he aining p ocess. The capaci y o lea ning he ans o ma ion lies on he non-linea i ies (small changes in he inpu lead o big changes in he ou pu ) ha ne wo ks a e able o ind and lea n om a da ase . One o he main capabili ies o he neu al ne wo ks is he gene aliza ion, since once hey a e ained hey can be used wi h new unknown da a wi h a eally low compu a ional cos . An example o his a e he classi ica ion ne wo ks ha can be used e en in low powe ul sma phones wi h hei came a. Howe e , his powe ul me hod has a s ong equi emen o he majo i y o applica ions as neu al ne wo ks equi e da ase s wi h hund eds o housands o examples. A neu al ne wo k is o med by a se o unc ions called neu ons which a e in e connec ed wi h each o he . Each neu on has mul iple inpu s wi h which ope a es, p opaga ing he esul o he nex neu ons. This p ocess is epea ed om he i s neu ons ha compu e he inpu o he las ones which gene a e he ou pu . This sequen ial execu ion allows o de ing he neu ons in laye s. The e a e basically h ee ypes o laye s: he inpu laye , he ou pu laye and he hidden ones which a e all he laye s in be ween he inpu and he ou pu . The numbe o hese hidden laye s is a iable and can be ine uned depending on he p oblem which he neu al ne wo k is ying o sol e. The same s a egy can be ollowed ega ding he numbe o neu ons. Bo h a e hype -pa ame e s ha need o be ob ained expe imen ally. In he las yea s, he de elopmen o neu al ne wo ks has de i ed in a specializa ion o a - chi ec u es and di e en ypes o ne wo ks ha e shown be e achie emen s o ce ain ypes o da a o p oblems. Fo example, some a chi ec u es ha e shown high capaci y gene a ing a comp essed e sion o he inpu and also decomp essing i o he o iginal size [29]. Some wo ks ha e es ed his capabili y, adding he possibili y o ob ain alues o he o iginal da a 9 10 Chap e 3. Backg ound o ce ain coo dina es [9]. O he a chi ec u es ha e shown hei capabili y lea ning implici ep esen a ions o a desi ed scene o geome y and being able, once ained, o eco e he ull o iginal da a by e alua ing he neu al ne wo k in he poin s o he o iginal da a [10]. Mo eo e , in hese examples, he neu al ne wo ks ha e shown he abili y o hallucina e new unknown poin s, eco e ing mo e esolu ion han he o iginal. In his wo k bo h examples can be un- de s ood as a solu ion o ou p oblem bu we will ocus on he second app oach and lea n an implici ep esen a ion o he da a using a mul ilaye pe cep on a chi ec u e. 3.1.1 Mul ilaye pe cep on A pe cep on [39] is a unique neu on de ined as: y= n ∑ i=1 xiwi+b!,(3.1) whe e is a non-linea ac i a ion unc ion, like sigmoid o a hype bolic angen unc ion, wa e he weigh s o he pe cep on, xa e he inpu s and bis he bias. This equa ion can be exp essed in a ma icial o m: y= xwT+b.(3.2) Neu ons can be o ganized o ming laye s. The numbe o neu ons in each laye is decided depending on he numbe o he ou pu s o he laye since each neu on has only one ou pu alue. I mul iple laye s a e connec ed like in he igu e 3.1, he a chi ec u e is called mul i- laye pe cep on. The numbe o laye s and he numbe o neu ons in each one a e ob ained expe imen ally as hype -pa ame e s excep o he las laye , whose numbe o neu ons is equal o he numbe o ou pu s alues. Figu e 3.1: Mul ilaye pe cep on scheme. Image om [40]. A machine lea ning app oach o ansien imaging econs uc ion 11 3.1.2 Ac i a ion unc ions The capaci y o he neu al ne wo ks o sol e di icul p oblems lies on he non-linea i es ha hey a e able o lea n. They can lea n hem due o he non-linea i es o i s ac i a ion unc ions. The ac i a ion unc ion basically ans o m he ou pu o he neu on depending on ce ain con- di ions, which can change he whole dis ibu ion o he in e media e da a. The e a e mul iple ac i a ion unc ions such as hype bolic angen (equa ion 3.3), ec i ied linea uni (ReLU) [41] (equa ion 3.4), sigmoid (equa ion 3.5). anh(x) = ex−e−x ex+e−x(3.3) ReLU(x) = 0 i x≤0 xi x>0(3.4) sigmoid(x) = 1 1+e−x(3.5) The ype o unc ion o use depends on he p oblem and he cha ac e is ics o he p oblem o sol e. The selec ion o he ac i a ion unc ion could be unde s ood as an hype -pa ame e iza ion.. Despi e he necessi y o es ing di e en ac i a ion unc ions, he numbe o candida es can be educed based on he ype o unc ion ha has been used o simila p oblems in p e ious wo ks. Fo example, one o he mos used ac i a ion unc ions o wo k wi h images is he ReLU unc ion since i only p opaga es posi i es alues. This is specially in e es ing when he ou pu is an image because he s anda d ange o wo k wi h hem is in [0,1]. The cha ac e - is ics o he ac i a ion unc ions help he ne wo ks o ocus on he esolu ion o he p oblem. In con as , selec ing a bad ac i a ion unc ion can diminish and hinde he con e gence o he aining p ocess. 3.1.3 G adien descen op imiza ion The op imiza ion p ocess is done h ough he minimiza ion o a unc ion. In deep lea ning he unc ion o minimize is called he loss unc ion and measu es he pe o mance o he ne wo k o e a ask. The pe ec solu ion would be o ind he global minimum o he loss. Howe e , ha could be impossible due o he complexi y o he pa ame e space. On he o he hand, inding a local minimum is an easie ask ha can be done wi h g adien descen op imiza ion me hods. As a simpli ica ion, we can deno e he loss unc ion as y= (x)which de i a i e is 0(x) = dy/dx. I is known ha he de i a i e gi es he slope o a he poin x. Wi h he di ec ion o he slope we know in which di ec ion he unc ion will be minimized and i he de i a i e is equal o 0 hen he unc ion is in a minimum. can be de ined wi h se e al pa ame e s in he o m o x={x1,x2,...,xn}. The de i a i e o mus be done wi h pa ial de i a i es ∂ (x)/∂xi. The g adien o deno ed as ∇ (x)will be he ec o wi h all he pa ial de i a i es. Like he g adien gi es he slope o e e y pa ame e we can upda e hem in he nega i e di ec ion o he slope and descending i e a i ely in he g adien . The upda ing unc ion can be deno ed as: x +1=x −ε∇ (x ),(3.6) whe e x is x alues in he ac ual i e a ion and εis a small alue called lea ning a e ha con ols he speed o he descen . The lea ning a e can no be oo high because when he unc ion is nea o he minimum a huge lea n a e could p e en he unc ion o ind he local 12 Chap e 3. Backg ound minimum. On he o he hand i he lea ning a e is oo small he needed s eps o ge ing he minimum poin will be so high ha he lea ning p ocess would ake oo much ime. 3.1.4 S ochas ic g adien descen The p e ious de ini ion hold o a loss wi h a single aining da a. Howe e , in machine lea ning is ypical o ha e a hund ed o housands o aining da a. In ha case, he loss unc ion can be unde s ood as a sum o e he loss o all aining da a. Conside ing he pa ame e s o he model as θand any loss unc ions o a single da a as Lwe can deno e he loss L o all he example as L(θ) = E[L(x,y,θ)] = ∑n i=1Lx(i),y(i),θ n.(3.7) Then, o apply he g adien descen me hod is needed o calcula e he g adien wi h espec o he pa ame e s o he model θas ∇θL(θ) = ∑n i=1∇θLx(i),y(i),θ n.(3.8) Compu ing he g adien descen wi h his calcula ion is one o he mos success ul op ions. Howe e , as we commen ed abo e, he size o he da ase is huge and makes di icul aining wi h he whole da a due o memo y space es ic ions. The solu ion o ha is aining using a subse o da a which is called ba ch size, and can be de ined wi h any desi ed size. Fo he cases o eally small ba ches hey a e called miniba ches and o some ields hey shown be e esul s ha bigge ba ches. Deno ing he loss o hese ba ches as L0and he numbe o elemen s in he ba ch as n0 he g adien could be de ined as be ∇θL0(θ) = ∑n0 i=1∇θLx(i),y(i),θ n0.(3.9) Finally, he g adien descen o he pa ame e s using ba ches is θ =θ −1−ε∇θL0(θ −1).(3.10) This a ia ion o he g adien descen is called s ochas ic g adien descen and is one o he mos used me hods in he aining o neu al ne wo ks. As he o iginal me hod, his one can no con e ge o he global minimum bu , i will end in he nea es local minimum. 3.2 Non-line-o -sigh (NLOS) ansien imaging T ansien imaging me hods le e age he in o ma ion encoded in he empo al domain o a ime- esol ed ligh cap u e. One o he a eas ha can be add essed in his ield is he non-line-o - sigh (NLOS) econs uc ion. The imaging me hods de eloped in his a ea aim o econs uc scena ios ha a e hidden a ound a co ne , by analyzing hei indi ec illumina ion on a su ace isible o he came a. A ypical NLOS scene can be seen in igu e 3.2 whe e an ul a-sho lase pulse is emi ed o a wall (also called elay wall), he ligh p opaga es om he hidden scene, being e lec ed by he objec s, and pa o i goes o he elay wall and is eco ded by an ul a-high speed came a called single-pho on a alanche diode (SPAD) gene a ing a ansien p o ile Hcalled impulse esponse unc ion. These p o iles ha e he in o ma ion o he hidden A machine lea ning app oach o ansien imaging econs uc ion 13 scene encoded inside hem. Thus, His calcula ed o di e en poin s o he elay wall o he lase and he SPAD pa ame e s a e H(xl,xs, )whe e xlis he lase posi ion, xsis he senso posi ion and a speci ic empo al ins an . NLOS SPAD-lase cap u e se up SPAD lase Figu e 3.2: Sample se up o cap u ing a NLOS scene. An ul asho ligh pulse is emi ed o a elay wall and he senso cap u es he ligh esol ed in ime e lec ed by he scene o he elay wall. Image adap ed om Liu e al.[2]. The impulse esponse unc ion His used in se e al me hods o econs uc ing he hidden scene as we commen ed in sec ion 2.1. In his wo k we ocus in he phaso ield amewo k [2] which is one o he bes pe o ming me hods in NLOS imaging. This wo k gi es us he possibili y o ans o m he H unc ion in o a phaso ield which ans o ms he empo al eso- lu ion in o a complex magni ude a he elay wall wi h ampli ude and phase dimensions o a conc e e equency. 3.2.1 Phaso ields The wo k o Liu e al. [2] p esen s a new me hod o ansien imaging which ans o ms he NLOS p oblem in o a i ual line-o -sigh (LOS) p oblem. This ans o ma ion allows he au ho s o use classical op ic me hods in he NLOS domain. Vi ual illumina ion Vi ual ape u e Vi ual senso Vi ual lens (a) (b) (c) Figu e 3.3: Phaso ield s eps. (a) Vi ual illumina ion is p opaga ed h ough he scene. (b) A i ual ape u e cap u e he esponse o he scene o he i ual ligh . (c) A i ual lens ocus he cap u ed illumina ion o he i ual ape u e and imaging. Image adap ed om [2]. 14 Chap e 3. Backg ound The linea i y o H(xl,xs, )is used by phaso ields o compu e he esponse o he hidden scene (Fig. 3.3b) o any i ual complex- alued emission p o ile P(xl, )(Fig. 3.3a) a poin s xlin a i ual senso as P(xs, ) = ZL [P(xl, )∗H(xl,xs, )]dxl.(3.11) Th ough he p opaga ion o he ield P(xs, )wi h an imaging ope a o I(·) o any poin x in he hidden scene as I(x ) = Φ(P(xs, )).(3.12) This ope a o models a i ual lens and senso sys em (Fig. 3.3c) and i can be o mula ed in e ms o a Rayleigh-Somme eld di ac ion p opaga o [2]. In he case o p opaga ing monoch oma ic signals o a single equency ω his image o ma ion ope a o Φ(Pω(xs, )) is deno ed as Φ(Pω(xs, )) = ZS Pω(xs, )Lω(xs,x ) |x −xs|dxs 2 ,(3.13) whe e Lis a complex ope a o ha changes he phase o Pa equency ω. Fo he Φope a o we implemen i as a hin lens model ha ocuses in o a i ual image plane a a hidden loca ion x , ha ing Lω(xs,x ) = e−ik|x −xs|,(3.14) whe e k=ω/cis he wa enumbe , wi h c he speed o ligh . A posi ion x can be imaging by combining he ocused emission p o ile (Eq. 3.11) and he imaging ope a o (Eq. 3.13) as I(x ) = ZSZL [Pω(xl, )∗H(xl,xs, )] Lω(xs,x ) |x −xs|dxldxs 2 .(3.15) This me hod is ully compu a ional since he unique ex e nal da a ha i need is he H unc ion. Fu he mo e, he image o ma ion model is i ual and can be o mula ed in o he o ms. The same idea holds o he emission p o ile and o his wo k we will de ine i as a cons an -emission ligh sou ce as Pω(xl, ) = eiω .(3.16) Chap e 4 Implici ep esen a ion o phaso ields In he las decade, he ield o non-line-o -sigh (NLOS) has e ol ed imp o ing he quali y o he econs uc ions o hidden scenes. Howe e , he cap u e me hod has ba ely changed. NLOS me hods use a cap u ed impulse esponse unc ion o a hidden scene. To ob ain his unc ion a ligh is emi ed, by an ul a-sho lase , o a di use wall ( elay wall). Then i a els ac oss he hidden scene and goes back o he elay wall whe e i is inally cap u ed by a senso . The cap u e p ocess needs o be done sequen ially o each poin in he elay wall, esul ing in a slow p ocess wi h a bo leneck ha o ces o ha e impulses esponse unc ions wi h low spa ial esolu ion. Despi e he p oblem wi h he spa ial esolu ion, wo ks as Liu e al. [2] success ully econ- s uc high complexi y hidden scenes. Howe e , o ob ain mo e de ailed econs uc ions, i is manda o y o gene a e mo e spa ially dense impulse esponse unc ions. To ob ain hem he e a e wo op ions: c ea ing new ha dwa e o inc ease he spa ial esolu ion o using compu a- ional me hods o gene a e new poin s om he cap u ed ones. On he one hand c ea ing ha dwa e o inc ease he spa ial esolu ion o he senso is no a i ial wo k. The ac ual senso s a e single-pho on a alanche diode (SPAD). They ha e a esolu ion o 1x1 pixel and he e a e some e sions wi h a line o SPADs ob aining a ow o cap u es. Despi e hese ad ances, he nex gene a ions o 2D SPADs will ha e a low esolu ion nea 16x16 pixels which is insu icien o sol e he p oblem. On he o he hand, using ully compu a ional me hods can wo k wi h he ac ual cap u e sys ems. We can es and implemen mul iple algo i hms o e sions o hem in a ela i ely sho space o ime. In spi e o he ad an ages ha compu a ional me hods can gi e us, he main limi a ion o us o design and c ea e new cap u e ha dwa e is he necessi y o es ing i wi h eal cap u e sys ems which only a ew labo a o ies in he wo ld ha e. In con as o es compu a ional me hods we can use bo h eal and simula ed da a, and he e o e, alida e he me hod by using hem. Fo ha eason we ha e decided o implemen a compu a ional me hod and gene a e, h ough simula ion, ou cap u e da a. The p oblem ha we wan o alle ia e is he low densi y o alues in he spa ial domain o he impulse esponse unc ion o imp o e he quali y o he econs uc ions. We can say ha his p oblem can be unde s ood as an upsampling o supe esolu ion p oblem whe e he numbe o samples o a unc ion is inc eased by an algo i hm. This p oblem is ypical in images whe e he low esolu ion image is ans o med in o a high esolu ion one. The ype o me hods ha gi es be e esul s in his domain a e he machine lea ning based ones. Howe e , o wo k wi h 15 22 Chap e 4. Implici ep esen a ion o phaso ields Figu e 4.5: Example o H unc ion o a conc e e pai lase -senso . The op image is he ansien p o ile in i s o iginal scale. The bo om image is he ansien p o ile in loga i hmic scale. To sol e he p oblem wi h he high dynamic ange o he H unc ion, we p opose o use he phaso ields amewo k (sec ion 3.2.1) o NLOS econs uc ion. In he i s s ep o his ame- wo k he H unc ion is con ol ed (eq. 3.11) wi h a i ual illumina ion phaso o a conc e e equency ω(eq. 3.16). This phaso P(xs, )is a 3D complex ma ix, wo spa ial dimensions and one empo al, bu as he i ual ligh used in he con olu ion is cons an along he ime and he came a model is a con en ional came a, P(xs, )can be e alua ed in P(xs, =0)ob ain- ing a 2D complex ma ix and sol ing he p oblem wi h he high dynamic ange. Howe e , he equency ha modules he i ual illumina ion a ec s o he econs uc ion quali y as can be seen in igu e 4.6. Fu he mo e, he co ec equency o each H unc ion is no he same and needs o be ob ained expe imen ally. F equency Hidden scene +_ Figu e 4.6: Example o di e en econs uc ions wi h di e en equencies. Da a no maliza ion. In his wo k we ha e wo ypes o da a: he inpu pa ame e s o he model (coo dina es and in some models he equency) and he alues o he phaso which he model will lea n. Fo bo h op ions we es ed he [0,1]and [−1,1]no maliza ion bu he models A machine lea ning app oach o ansien imaging econs uc ion 23 did no i co ec ly in he bo de s o he ma ix. Since he p oblem was in he bounda ies we conside ha he p oblem was wi h he no maliza ion alues in he limi s o he ange and wi h a no maliza ion in (−1,1) his p oblem was pallia ed. Da a gene a ion. The cap u e o he H unc ions can be done om wo di e en sou ces, eal cap u e sys ems o h ough simula ion. The i s one can no be done in he labo a o y a his uni e si y due o we do no ha e he ha dwa e. Howe e , using eal cap u ed da a ha e some disad an ages. The cap u es ha e noise and hei condi ions can no be con olled comple ely. On he o he hand, simula ing a oids his p oblems due o he en i onmen can be con olled (e.g., a oid ex e nal ligh con amina ion) and can be gene a ed wi h less noise. Also, simula ing allow o c ea e idyllic condi ions e.g., pe ec lambe ian su aces o he elay wall. Fo ha eason in his wo k we simula e all he cap u es using he public ansien ende by Ja abo e al. [13]. Gene ic model s speci ic models. Neu al ne wo ks a e usually applied o gene ic appli- ca ions ha can be used once is ained wi h di e en da a, o example in supe esolu ion models, classi ica ion o some comp ession wo ks. Howe e , his models need o be ained wi h eno mous da ase s, in some cases millions o samples. In he con ex o his wo k his is no an op ion due o he amoun o ime ha is needed o gene a e a da ase o he size ha gene al models need. Fo ha eason in his wo k we choose o use one model pe unc ion (o scene) ha we wan o i . Since we a e i ing a single unc ion in he neu al model, and we wan o in e new alues, we need o a oid o e i ing in he model. In o he cases, i we wan ed o comple ely comp ess a unc ion and only e alua e a he o iginal poin s, o e i ing would be jus i ied. O he aspec ha is in ol ed in he o e i ing and in he capaci y o he neu al models o be able o i he disc e e unc ion is he numbe o model pa ame e s. The co ec numbe o hem needs o be ob ained expe imen ally. Mo eo e , i can depend on he ype o unc ion ha he model is i ing, complex unc ions can equi e mo e pa ame e s han he simples ones and i he numbe o pa ame e s is oo big, he model will be lazy and will o e i easily. In he con ex o his p oblem, we can conside an uppe bound o he numbe o pa ame e s, he o al amoun o alues in he disc e e unc ion because i is he numbe o alues needed o ep esen he same in o ma ion. In ou expe imen s we es ed wi h di e en numbe o pa ame e s and igu ed ou ha wi h mo e han hal he o al numbe o pa ame e s o he o iginal unc ion, he models end o o e i and can no gene a e well new poin s. On he o he hand, wi h less han he hal he models lea n he disc e e unc ion and a e able o gene a e new poin s smoo hly. Howe e , i he pa ame e s a e oo low, he models can no con e ge a all. 24 Chap e 4. Implici ep esen a ion o phaso ields Chap e 5 Resul s In he sec ion 4.1 we ha e es ed di e en neu al ne wo ks models o lea n an implici ep- esen a ion o a phaso ield. Speci ically, he neu al ne wo k model ob ained can lea n an implici ep esen a ion o a mul i- equency phaso ield. One o he ad an ages o lea ning a mul i- equency phaso ield is he elimina ion o he necessi y o calcula ing sepa a e implici ep esen a ions o each equency. This means ha i gi es he abili y o compu e any desi ed equency ha i is no in he sampled space. This is especially in e es ing due o he equency is di ec ly dependen o he quali y o he econs uc ions. Compu ing no el phaso ields wi h highe equencies and highe g id densi ies can p o ide sha pe esul s o he econs uc ed geome y. As such, he abili y o ou implici ep esen a ion o compu e no el empo al e- quencies and spa ial loca ions in he elay wall ha we e no p o ided by he sampled cap u ed da a is use ul o p o ide be e econs uc ions. Ano he applica ion o his mul i- equency model is he possibili y o use i wi h mo e complex ligh s and came as in he phaso ield amewo k. To use hem i will be equi ed ( heo e ically) he use o in ini e equencies. Wi h an implici ep esen a ion, his p oblem can be mo e ea able because he ep esen a ion a oids he necessi y o do any con olu ion o each equency. Du ing he analysis o he di e en neu al models (Sec ion 4) we ha e es ed hem wi h only one scene o simplici y. A e demons a ing ha he MLP wi h sine ac i a ion unc ions model p o ides he bes beha io in ep esen ing mul i- equency phaso ields, he e we p o- ide a deepe analysis o i s pe o mance in scenes o di e en complexi y. Howe e , be o e showing he di e en esul s, is impo an o unde s and he limi a ions o he i ual came a and he i ual ligh used in he phaso ield amewo k. The i ual came a has an ape u e size ha co esponds o he size o he sampled g id on he elay wall. A e using his ape u e o simula e a i ual came a ocused a ce ain dep h in he hidden scene, all geome y behind o ahead o he ocused plane is ou o ocus, in oducing se e al a i ac s in he image. Addi ion- ally, due o cap u e limi a ions, he i ual illumina ion is a enua ed adially om he cen e o he elay wall, which esul s in a enua ed econs uc ions o he geome y u he om he cen e o he esul ing images. This e ec is clea ly isible in all he p e ious esul s in he sec- ion 4.1. To analyze how he implici ep esen a ions can app oxima e he disc e e phaso ield we will compa e he in ensi y, he phase and he econs uc ion o bo h phaso ields, disc e e (g ound u h) and he one p edic ed by he implici ep esen a ion. We ha e selec ed sim- ple scenes o ha e low ou -o - ocus geome y o a oid ou -o - ocus a i ac s. Recons uc ing he in- ocus geome y in mo e complex scena ios would equi e es ima ing a la ge amoun o di e en phaso ields o each oxel o he econs uc ed scene, while simple scenes p o ide good esul s wi h a single mul i- equency phaso ield. We ega d as u u e wo k o p o ide 25 26 Chap e 5. Resul s mo e complex implici ep esen a ions o phaso ields in clu e ed scenes. As in he las model o he sec ion 4.1, we will es he capaci y o he model o gene a e new samples in he spa ial and equency domain. To ain and es he model we ha e decided o emo e 75% o he samples om a phaso ield wi h a spa ial esolu ion o 64x64, he e o e ge ing only 32x32 sampled poin s in he elay wall. Fo he equency, we ha e compu ed 151 di e en equencies and ained wi h he 90% o hem, ese ing he las 10% o es ing. Once he model is ained wi h he low esolu ion phaso ield, we eco e he o iginal esolu ion a speci ic equencies and spa ial loca ions in he elay wall ha he model has no been ained wi h. Fo es ing he model, a mul ilaye pe cep on wi h sine ac i a ion unc ions, we ha e used ou di e en scenes ( igu es 5.1-5.4). We s a by a plana bu de ailed scene ( igu e 5.1). This scene allows us o analyze he pe o mance unde di e en geome ic esolu ions allowing see- ing he le el o de ails ha he me hod is able o eco e . Conc e ely, he esul s ob ained o his scene show he capabili y o he model o lea n mo e complex phaso ields and upsample hem co ec ly.The gene al s uc u e is main ained and an impo an denoised e ec is app e- ciable. The econs uc ions a e quali a i ely simila . They lose a bi o de ail bu , bea ing in mind ha he model is ained wi h he 25% o he o iginal poin s, i is a good econs uc ion. As compa ison, in he igu e 5.2 can be seen how is a econs uc ion o a phaso ield wi h a esolu ion o 32x32 and a econs uc ion wi h a esolu ion o 64x64. In he e o map i can be seen wo hings: he e o de i ed om he noise no lea ned by he model and how he ne wo k ails mo e in he low alues. Figu e 5.1: Resul s o eco e ing a scene wi h mul iple planes wi h di e en sizes. The esul s show ha he implici ep esen a ion lea n co ec ly he shape o he phaso ields and emo e he noise o hem. The econs uc ions a e quali a i ely simila . Le phaso ield has a e- quency o 11.422 and he ig h one has a equency o 15,238. The second scene is o med by a plane wi h se e al conca i ies and i egula i ies ( igu e 5.3). No e how he esul ing ampli ude and phase o he phaso ield change due o he s uc- u al di e ences o he hidden geome y. Ne e heless, ou implici ep esen a ion is able o es ima e he phaso ield s uc u e a highe esolu ions, e y close o he g ound u h while also emo ing noise, and p o ides a simila econs uc ion esul . Ou implici ep esen a ion A machine lea ning app oach o ansien imaging econs uc ion 27 Figu e 5.2: Compa ison o he econs uc ion o he same scene wi h di e en esolu ion, 32x32 and 64x64 o he same equency (15.238). is capable o lea ning he gene al s uc u e, while simul aneously emo ing he noise in he am- pli ude and phase images. The e o maps show, as in he p e ious example, ha he model has mo e e o in he low alue poin s and also he gene al noise e o . The econs uc ions a e e y simila . Despi e ha he RMSE in he i s econs uc ion is highe , he esul is quali a i ely mo e simila han in he second econs uc ion. Figu e 5.3: Resul s o a scene wi h mul iple conca i ies and pikes. The econs uc ions do no show he shape co ec ly due o he ou -o - ocus geome y. The neu al model lea ns and eco e he shape o he phaso ield emo ing he noise om i . Finally, we ha e wo simila scenes ( igu es 5.4 and 5.5). Bo h a e o med by wo ec angula planes a wo di e en dep hs bu in each scene he dis ance be ween hem is di e en . The goal o hese wo scenes is o es he pe o mance o ou algo i hm unde he p esence o objec s ha may occlude hemsel es om he pe spec i e o he i ual came a. Fo hese wo cases ou implici neu al model lea ns he s uc u e o he phaso ields as well. Commen s and discussions. Ou neu al ep esen a ion o phaso ields has shown a high capaci y o lea n he gene al s uc u e o he phaso ield, being able o gene a e highe eso- lu ions ( eco e ing a 75% o he poin s in hese es s) and new equencies wi h simila esul s o he g ound u h. The RSME is simila in all o hem bu is bigge in he phaso ield wi h highe equency. This could be caused by he inc ease o de ails ha appea when he em- po al modula ion equency is inc eased. This inc ease in he empo al equency esul s in 28 Chap e 5. Resul s Figu e 5.4: Resul s o a scene wi h wo planes a di e en dep h. This scene allows us o es i he phaso ield wi h mul iple objec s and dep hs can be lea ned by he neu al model. The model lea ns co ec ly despi e he changes in he shape. Figu e 5.5: Resul s o a scene wi h wo planes a di e en dep h. In his case he plane in he back is semioccluded by he plane in he on . The neu al model lea ns co ec ly he shape and e en he s ong change o alues. an inc emen o he spa ial equency in he phaso ield, as he esul s show. Focusing on he econs uc ions, he p edic ed and he g ound u h a e also e y simila and can eco e e en he a i ac s p oduced by di ac ion e ec s. No e ha his is a p oblem inhe en in he phaso ields me hod [2], and no a esul o ou neu al model. The bigges quali a i e di e - ences be ween p edic ed and g ound u h appea in he econs uc ions done wi h he phaso ields wi h highe equencies, losing mo e de ails han wi h he low equency phaso ields. No e ha ou quan i a i e compa isons a e done wi h espec o he g ound u h solu ion wi h equi alen sampling densi y and empo al equency. While ou model yields a la ge RMSE when inc easing modula ion equency, inc easing sampling densi y and modula ion equency in phaso ields p o ides be e econs uc ions o he hidden scene, as ou quali a i e esul s show. The di e ences in RMSE o ou model wi h espec o lowe equencies a ise om he abili y o ou model o analy ically ep esen highe equencies, and no om he abili y o he da a o p ope ly econs uc he scene. A machine lea ning app oach o ansien imaging econs uc ion 29 Hidden scene Loss (L2) T aining ime (s) Figu e 5.1 0.009883426 677.92 Figu e 5.3 0.011194097 670.95 Figu e 5.5 0.0029691448 673.33 Figu e 5.4 0.0032978356 673.98 Table 5.1: Losses and aining imes o each scene. The aining p ocess ha e been done wi h a lea ning a e o 0.001 and a o al o 100000 epochs. The model in all ou es s has shown a denoising abili y. This e ec is p oduced by he capabili y o neu al ne wo ks o ocus on p edic able aspec s o he da a. Conc e ely, as he noise is some hing mo e andom han he s uc u e o he da a, he neu al ne wo ks ob ain be - e esul s lea ning he s uc u e and cha ac e is ics o he da a. Howe e , his e ec is only obse ed when he pa ame e s o he ne wo ks a e no oo high because, in ha case, he neu al ne wo k o e i s he da a and canno upsample. Rega ding he possible e ec o he noise in he e o maps, we hough ha i is possible ha he noise could mask he co ec le el o achie emen o his me hod. Howe e , o es i , i would be necessa y o use o he me ics ha ake mo e in o accoun he s uc u e o he nea by pixels as he s uc u al simila i y index mea- su e (SSIM). We will lea e he es o his hypo hesis o a u u e wo k. In he econs uc ions, he e ec o he exis ence o noise in he phaso ield does no appea o be de e minan o a leas he e ec is oo low o be seen. In hei e o maps, he i s di e ence ega ding he e o maps o he phaso ield is he lack o noise. The e o maps o he econs uc ions a e mo e smoo hly han he e o maps o he phaso ields. We hough ha he econs uc ion p ocess is esis an o his le el o noise. Wi h his idea, a u u e wo k ha could be done is an analysis o he ole ance o he econs uc ion me hod a di e en le els o noise and how his me hod can be used o denoising i . This is an in e es ing aspec because he eal cap u es ha e noise ha usually needs o be added o he syn he ic da a o compa e bo h. Fu he mo e, knowing ha , i could help o know he le el o noise ha his me hod can handle and use i o gene a e syn he ic da a as e . The aining p ocess o hese scenes is as (see able 5.1 o exac aining imes). In a ound ele en minu es each scene can be lea ned. The e alua ion p ocess o each equency is p ac i- cally ins an , less han one second, gi ing a huge po en ial o use i o mo e applica ions such as using mo e complex came as ha could equi e compu ing housands o di e en equen- cies. Mo eo e , he model can be e alua ed wi h mul iple equencies a he same ime wi hou a pe cep ible inc ease o ime. 30 Chap e 5. Resul s Chap e 6 Conclusions and u u e wo k In his wo k we ha e in oduced a new me hod o ep esen phaso ields in an implici o m by using neu al models o inc emen he e iciency and he quali y o he esul s o econs uc- ions o hidden scenes. Phaso ields a e a ecen wo k ha allows o ake i ual pho og aphs o a hidden scene as i was seen om a elay wall. Howe e , his me hod is limi ed by in i s esolu ion caused by he cap u e p ocess. Since each spa ial poin has o be sequen ially cap- u ed, i becomes unp ac ical and complex o cap u e high esolu ion in o ma ion. Al hough his p oblem could be alle ia ed wi h he de elopmen o new ha dwa e, his app oach would equi e he use o expensi e ha dwa e con igu a ions. Inspi ed by o he wo ks on implici ep esen a ions ha p o e hei abili y o c ea e a con in- uous space om disc e e numbe o samples. We ha e o mula ed a phaso ield using implici ep esen a ion which allows us o lea n a disc e e phaso ield and sample i in a con inuous space. This ans o ma ion emo es any esolu ion limi a ion. In o he wo ds, we could ob ain in ini e samples om i . Following ecen s udies and wo ks, we ha e es ed di e en models o di e en asks. S a ing by he mos simple (lea ning an implici ep esen a ion o a single- equency phaso ield wi hou changing he esolu ion), up o he mos complex ( ha lea ns an implici ep esen a ion o a mul i- equency phaso ield). This las model is a mul ilaye pe - cep on wi h sine ac i a ion unc ions ha can upsample he da a in he spa ial and equency domains. To e i y i he inal model can gene alize o di e en scenes we ha e es ed i wi h ou scenes. The model co ec ly upsample he da a o all he scenes gi ing a simila e o in all o hem. O he aspec obse ed is he denoising e ec , since he model eco e s smoo h phaso ields wi h he co ec s uc u e. This can be quali a i ely demons a ed, as he econs uc ed scenes a e almos iden ical o he g ound- u h ones. This wo k le some in e es ing esea ch as u u e a enues. Ou cu en me hod could be ex ended o suppo mo e clu e ed scenes wi h mo e complex imaging unc ions, whe e each loca ion o he scene would equi e a di e en illumina ion and lens unc ion, leading o he es ima ion o highe numbe o phaso ields pe scene. An aspec ha can be es ed is whe he he noise in he cap u es is ela ed wi h he di e ences in he econs uc ions. Rela ed o he noise, i could be in e es ing o analyze how he noise a ec s o he econs uc ions and i his model o o he ypes o ne wo k can denoise he phaso ield and wi h which le el o noise a e hey e ec i e. Mo eo e , his wo k could be expanded o he s udy o he whole ou dimensions in ol ed in phaso ields, in con as o he wo explo ed in his wo k. Ano he 31