A machine lea ning app oach o
ansien imaging econs uc ion
Miguel Ángel Cosculluela G acia
T abajo de in de Más e en Modelización e
In es igación Ma emá ica, Es adís ica y
Compu ación
Uni e sidad de Za agoza
Di ec o es del abajo: D . Julio Ma co
Mu ia y P o . D . Diego Gu ié ez Pé ez
9 de eb e o de 2021
Abs ac
The ecen ad ances in non-line-o -sigh imaging ha e made i possible o econs uc scenes
hidden a ound a co ne , wi h po en ial applica ions in e.g. au onomous d i ing o medical
imaging. By ope a ing a ame a es compa able o he speed o ligh , ecen i ual-wa e
p opaga ion me hods le e age he empo al oo p in o indi ec ligh anspo a a isible
auxilia y su ace o ake i ual pho os o objec s hidden om he obse e . Despi e hese
ad ances, hese me hods ha e a c i ical compu a ional bo leneck: The econs uc ion qual-
i y and he compu a ional pe o mance a e highly dependen on he esolu ion o he cap u e
g id, which is ypically disc e ized in space and ime, leading o high p ocessing and memo y
cons ain s.
Inspi ed by ecen machine lea ning echniques, in his wo k we p opose a new compu-
a ional imaging me hod o add ess hese limi a ions. Fo his pu pose we p opose o lea n
implici ep esen a ions o he cap u ed da a using neu al ne wo ks, allowing us o con e he
disc e e space o he cap u ed da a in o a con inuous one. Howe e , wo king di ec ly wi h he
cap u ed da a is a complex ask due o i s huge size and i s high dynamic ange alues. In
o de o a oid hese p oblems, we le e age ecen wa e-based phaso - ield imaging me hods
o ans o m he ime- esol ed cap u ed da a in o se s o 2D complex- alued ields (i.e. phaso
ields) a di e en equencies, which p o ides a mo e a o able ep esen a ion o machine
lea ning me hods.
Unde ou implici ep esen a ion o mula ion, we analyze he pe o mance o di e en neu-
al ne wo k models o ep esen he complex s uc u e o phaso ields, s a ing om simple
ep esen a ions, and i e a i ely p o iding mo e powe ul models o add suppo o he com-
plexi y o he da a. We demons a e how ecen machine lea ning echniques based on mul i-
laye pe cep ons wi h sine ac i a ion unc ions a e capable o ep esen ing phaso ields ana-
ly ically in bo h spa ial and empo al equency domains, and in eg a e hem in o he phaso -
ield amewo k o econs uc hidden geome y. We inally es his neu al model in di e en
scenes, and measu e i s pe o mance a highe esolu ions no seen by he cap u ed da a. We
show how he model is able o analy ically upsample all dimensions, and demons a e how ou
implici ep esen a ion addi ionally wo ks as a denoise o he sou ce disc e ized phaso ield.
iii
i Abs ac
Resumen
Los ecien es a ances en imagen non-line-o -sigh han hecho posible la econs ucción de es-
cenas ocul as a a és de una esquina, con la po encial aplicación en conducción au ónoma o
imagen medica. Al ope a con o og amas po segundo ce canos a la elocidad de la luz, e-
cien es mé odos de p opagación i ual de ondas ap o echan la huella empo al del anspo e
de luz en una supe icie auxilia pa a oma o ones i uales de obje os ocul os al obse ado .
A pesa de es os a ances, es os mé odos ienen un cuello de bo ella c i ico: La calidad econ-
s ucción y el cos e de compu o son al amen e dependien es de la esolución de la malla de
cap u a, la cual suele es a disc e izada en espacio y iempo, lo que conlle a g andes limi a-
ciones de p ocesamien o y memo ia.
Inspi ados en las ecien es écnicas de ap endizaje au omá ico, en es e abajo p oponemos
un nue o mé odo de imagen compu acional pa a hace en e a es as limi aciones. Pa a es e
p opósi o p oponemos ap ende ep esen aciones implíci as de los da os cap u ados usando
edes neu onales, pe mi iéndonos con e i el espacio disc e o de los da os cap u ados en un
espacio con inuo. Sin emba go, abaja di ec amen e con los da os cap u ados es una a ea
compleja debido a su g an amaño y al al o ango dinámico de sus alo es. Pa a e i a es os
p oblemas ap o echamos el ecien e mé odo de imagen de los campos de aso es basados en
ondas pa a ans o ma los da os cap u ados esuel os en iempo en un conjun o de campos 2D
de alo es complejos (como son los campos de aso es) a di e en es ecuencias, lo cual p o ee
una ep esen ación más a o able pa a los mé odos de ap endizaje au omá ico.
Siguiendo nues a o mulación de ep esen ación implíci a, hemos analizado el endimien o
de di e en es modelos de edes neu onales pa a ep esen a la compleja es uc u a de los cam-
pos de aso es, empezando po ep esen aciones simples, y p opo cionando de o ma i e a i a
modelos más po en es pa a añadi sopo e pa a la complejidad de los da os. Demos amos
como las écnicas ecien es de ap endizaje au omá ico basadas en p ecep ones mul icapa con
unciones de ac i ación sinusoidales son capaces de ep esen a un campo de aso es analí ica-
men e en los dominios espaciales y empo ales, e in eg a las den o del ma co de los campos de
aso es pa a econs ui geome ía ocul a. Finalmen e p obamos es e modelo de ed neu onal
con di e en es escenas y medimos su desempeño con mayo es esoluciones que no han sido
usadas en el en enamien o. Mos amos como el modelo es capaz de gene a más mues as en
odas las dimensiones y demos amos como nues a ep esen ación implíci a además unciona
como un mé odo pa a elimina uido del campo de aso es disc e izado.

i Resumen
To my amily and iends in he labo a o y ha helped me du ing he ealiza ion o his mas e ’s
hesis. I would like o hank my di ec o s, who o ien a ed me, and my pa ne who suppo ed
me du ing his ime and who has gi en me he s eng h o keep on.
ii
Con en s
Abs ac iii
Resumen
1 In oduc ion 1
1.1 ThesisBackg ound ................................ 3
2 Rela ed Wo k 5
2.1 Non-line-o -sigh (NLOS) ansien imaging . . . . . . . . . . . . . . . . . . . 5
2.2 T ansien ende ing................................ 6
2.3 Upsamplingme hods ............................... 6
3 Backg ound 9
3.1 Neu al ne wo k op imiza ion . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.1.1 Mul ilaye pe cep on . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.1.2 Ac i a ion unc ions . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.1.3 G adien descen op imiza ion . . . . . . . . . . . . . . . . . . . . . . 11
3.1.4 S ochas ic g adien descen . . . . . . . . . . . . . . . . . . . . . . . . 12
3.2 Non-line-o -sigh (NLOS) ansien imaging . . . . . . . . . . . . . . . . . . . 12
3.2.1 Phaso ields ............................... 13
4 Implici ep esen a ion o phaso ields 15
4.1 Neu al implici ep esen a ion o phaso ields . . . . . . . . . . . . . . . . . . 17
4.2 Da a ans o ma ion................................ 21
5 Resul s 25
6 Conclusions and u u e wo k 31
ix
6Chap e 2. Rela ed Wo k
2.2 T ansien ende ing
In NLOS one o he mos aluable ools ha help o es and de elop new me hods is he sim-
ula ion o he cap u e sys em and NLOS scenes and, in ou case, he possibili y o gene a e
da a o machine lea ning algo i hms. The simula ion is done wi h a ende ing p ocess. Con-
c e ely, he ende ing p ocess simula es he ligh and i s in e ac ions wi h he ma e in a i ual
scene and cap u e he ligh wi h i ual senso s. The s anda d ende ing p ocess used in mos
o he applica ions such as cinema o ideo games gene a es images in 2D wi h he ligh in
he scene in eg a ed. This ype o ende is known as s eady-s a e ende ing. O he ype is
he ansien ende ing me hods which allow simula ing how he ligh mo es h ough a scene,
ende ing ideos a ame a es compa able o he speed o ligh . Howe e , adding he empo al
dimension in ansien ende ing is no always possible. This is because some applica ions and
me hods de eloped o s eady-s a e ende ing canno be applied easily o , in some cases, a
all. To implemen a ansien ende e some wo ks p esen ed new ime- esol ed ligh anspo
equa ions [24, 25]. O he wo ks gene alize Mon e-Ca lo me hods o s eady-s a e ende ing
as pa h acing o a ime- esol ed e sion [26, 13]. O he algo i hms as pho on mapping we e
also gene alized o ime- esol ed e sions [27].
The use and esea ch in ansien ende ing ha e g own wi h he de elopmen o NLOS an-
sien imaging ield. I s use is ex emely use ul in he NLOS ield due o i helps o de elop and
es NLOS me hods. This is due o he possibili y o gene a ing ansien p o ile in con olled
condi ions wi hou he es ic ions o eal cap u e ha dwa e and use hem as e e ence da a.
Following his las idea i is possible o gene a e da ase s and g ound u h o deep lea ning
me hods. Fo example he wo k by Ma co e al. [28] uses a ansien ende e o gene a e a
ime-o - ligh (TOF) da ase and ain a deep lea ning model o educe he e o in TOF cam-
e as. O he in e es ing wo k by Liang e al. [29] p esen ed a comp ession me hod o ansien
p o iles wi h a deep lea ning model ained wi h syn he ic ansien da a. Also, he wo k by
Galindo e al. [30] p esen ed a public da ase wi h NLOS scenes. In con as , some NLOS
me hods do no use he comple e ligh pa hs bu only he i s bounce [31, 32, 33]. This ype o
ende ing is as e and he wo k by Chopi e e al. [34] uses his ype o ende ing o gene a e a
bigge da ase and ain a deep lea ning model o econs uc NLOS scenes om hei ansien
p o iles.
2.3 Upsampling me hods
A cap u e p ocess by de ini ion canno ob ain he eal shape o he space bu only ge a ce ain
numbe o samples (which is known as he esolu ion). Ha ing a highe esolu ion gi es mo e
accu acy o he measu ed space. Howe e , he esolu ion is ypically limi ed by he cap u e
ha dwa e o physical es ic ions in he cap u e p ocess. When he esolu ion is la ge enough,
he o iginal shape can be ob ained wi h a simple linea in e pola ion because he changes be-
ween samples p ac ically ollow a s aigh line. Bu in mos cases he esolu ion is oo low
and ob aining he o iginal shape is a ha d p oblem. Wi h he e olu ion o neu al ne wo ks se -
e al wo ks ha e demons a ed hei capabili ies o unde s anding he s uc u e o he da a and
being able o upsample hem. Fo his p opose, he e a e wo main app oxima ions. The i s
app oach consis s in upsampling o a highe esolu ion o a ixed size o scale a e (e.g. dupli-
ca ing esolu ion). The second one uses he neu al ne wo k o, o a conc e e inpu like spa ial
coo dina es, ime o equency, ge he alue o he o iginal da a o , i he inpu is p e iously
unknown, hallucina e new samples.

A machine lea ning app oach o ansien imaging econs uc ion 7
The i s app oach is commonly used in image supe - esolu ion. Wi h he idea o upsampling
o ixed sizes, se e al neu al ne wo ks models ha e been de eloped om models which only
need one low esolu ion image as an inpu [8] o models which le e age he in o ma ion om
mul iple images such as ideo ames [35]. I he upsampling is done o e he empo al domain
ins ead o he spa ial one he e ec is supe -slow mo ion ideos [36].
The second app oach is mo e lexible han he i s one because i is possible o sample in
conc e e zones o a any desi ed esolu ion. Howe e , hey ha e he cons ha hei models a e
no gene al and, o each scene, a new aining is equi ed. T aining and sampling a model
ha e wo main ways o be implemen ed. In he i s way he model has as an inpu he scene
o da a (o a ans o ma ion o hem) and he coo dina es o be sampled. This case ha e been
es ed o comp ession models o ex u es images, ob aining a la en space o hem. Then he
au ho s can decomp ess he la en space by sampling wi h neu al ne wo ks [37, 9] o coo di-
na es in he o iginal ex u es o o new ones. The o he way i s he scene di ec ly in he neu al
ne wo k. This is known as implici ep esen a ion and has he ad an age ha , once he neu al
ne wo k is ained, he aining da a is no mo e used. This ype o ne wo ks has as an inpu
he di e en ypes o coo dina es (spa ial, empo al, angula , e c.) and, as he ou pu , he co -
esponding alues o ha coo dina es. Like he p e ious me hods, his ype o models can be
used o gene a e samples in p e ious unknown posi ions. A use ul case is lea ning 3D scenes
[38] which, in ende applica ions, a oids he equi emen o ende ing new iews. Close o
ou app oach he e a e ecen simila wo ks which simpli y he aining p ocess and imp o e
he esul s [10, 11] p esen ing a new me hod o ans o m he coo dina es and imp o ing sig-
ni ican ly he esul s. Fu he mo e, he wo k o Si zmann e al. [12] shows he possibili ies o
using pe iodical unc ions ins ead o he classical ac i a ion one o lea ning high equency
de ails in se e al implici ep esen a ion p oblems such as image o ideo ep esen a ion.
8Chap e 2. Rela ed Wo k
Chap e 3
Backg ound
This chap e desc ibes he main ma hema ical, physical, and compu a ional aspec s ha his
wo k builds upon. Fi s ly we in oduce a basic knowledge on neu al ne wo k explaining hem
and he model used in his wo k wi h he mos common op imiza ion me hod used wi h neu al
ne wo ks. Secondly we in oduce he non-line-o -sigh p oblem and explain he heo y o
phaso ields, a s a e-o - he-a me hod o non-line-o -sigh econs uc ion.
3.1 Neu al ne wo k op imiza ion
Neu al ne wo ks a e lea ning sys ems inspi ed in human neu ons and hei in e connec ions.
They a e able o lea n a ans o ma ion and unde s and he ela ionship be ween he inpu and
ou pu da a used in he aining p ocess. The capaci y o lea ning he ans o ma ion lies on
he non-linea i ies (small changes in he inpu lead o big changes in he ou pu ) ha ne wo ks
a e able o ind and lea n om a da ase . One o he main capabili ies o he neu al ne wo ks is
he gene aliza ion, since once hey a e ained hey can be used wi h new unknown da a wi h a
eally low compu a ional cos . An example o his a e he classi ica ion ne wo ks ha can be
used e en in low powe ul sma phones wi h hei came a. Howe e , his powe ul me hod has
a s ong equi emen o he majo i y o applica ions as neu al ne wo ks equi e da ase s wi h
hund eds o housands o examples.
A neu al ne wo k is o med by a se o unc ions called neu ons which a e in e connec ed
wi h each o he . Each neu on has mul iple inpu s wi h which ope a es, p opaga ing he esul o
he nex neu ons. This p ocess is epea ed om he i s neu ons ha compu e he inpu o he
las ones which gene a e he ou pu . This sequen ial execu ion allows o de ing he neu ons in
laye s. The e a e basically h ee ypes o laye s: he inpu laye , he ou pu laye and he hidden
ones which a e all he laye s in be ween he inpu and he ou pu . The numbe o hese hidden
laye s is a iable and can be ine uned depending on he p oblem which he neu al ne wo k is
ying o sol e. The same s a egy can be ollowed ega ding he numbe o neu ons. Bo h a e
hype -pa ame e s ha need o be ob ained expe imen ally.
In he las yea s, he de elopmen o neu al ne wo ks has de i ed in a specializa ion o a -
chi ec u es and di e en ypes o ne wo ks ha e shown be e achie emen s o ce ain ypes
o da a o p oblems. Fo example, some a chi ec u es ha e shown high capaci y gene a ing
a comp essed e sion o he inpu and also decomp essing i o he o iginal size [29]. Some
wo ks ha e es ed his capabili y, adding he possibili y o ob ain alues o he o iginal da a
9
10 Chap e 3. Backg ound
o ce ain coo dina es [9]. O he a chi ec u es ha e shown hei capabili y lea ning implici
ep esen a ions o a desi ed scene o geome y and being able, once ained, o eco e he ull
o iginal da a by e alua ing he neu al ne wo k in he poin s o he o iginal da a [10]. Mo eo e ,
in hese examples, he neu al ne wo ks ha e shown he abili y o hallucina e new unknown
poin s, eco e ing mo e esolu ion han he o iginal. In his wo k bo h examples can be un-
de s ood as a solu ion o ou p oblem bu we will ocus on he second app oach and lea n an
implici ep esen a ion o he da a using a mul ilaye pe cep on a chi ec u e.
3.1.1 Mul ilaye pe cep on
A pe cep on [39] is a unique neu on de ined as:
y= n
∑
i=1
xiwi+b!,(3.1)
whe e is a non-linea ac i a ion unc ion, like sigmoid o a hype bolic angen unc ion,
wa e he weigh s o he pe cep on, xa e he inpu s and bis he bias. This equa ion can be
exp essed in a ma icial o m:
y= xwT+b.(3.2)
Neu ons can be o ganized o ming laye s. The numbe o neu ons in each laye is decided
depending on he numbe o he ou pu s o he laye since each neu on has only one ou pu
alue. I mul iple laye s a e connec ed like in he igu e 3.1, he a chi ec u e is called mul i-
laye pe cep on. The numbe o laye s and he numbe o neu ons in each one a e ob ained
expe imen ally as hype -pa ame e s excep o he las laye , whose numbe o neu ons is equal
o he numbe o ou pu s alues.
Figu e 3.1: Mul ilaye pe cep on scheme. Image om [40].
A machine lea ning app oach o ansien imaging econs uc ion 11
3.1.2 Ac i a ion unc ions
The capaci y o he neu al ne wo ks o sol e di icul p oblems lies on he non-linea i es ha
hey a e able o lea n. They can lea n hem due o he non-linea i es o i s ac i a ion unc ions.
The ac i a ion unc ion basically ans o m he ou pu o he neu on depending on ce ain con-
di ions, which can change he whole dis ibu ion o he in e media e da a. The e a e mul iple
ac i a ion unc ions such as hype bolic angen (equa ion 3.3), ec i ied linea uni (ReLU) [41]
(equa ion 3.4), sigmoid (equa ion 3.5).
anh(x) = ex−e−x
ex+e−x(3.3)
ReLU(x) = 0 i x≤0
xi x>0(3.4)
sigmoid(x) = 1
1+e−x(3.5)
The ype o unc ion o use depends on he p oblem and he cha ac e is ics o he p oblem o
sol e. The selec ion o he ac i a ion unc ion could be unde s ood as an hype -pa ame e iza ion..
Despi e he necessi y o es ing di e en ac i a ion unc ions, he numbe o candida es can be
educed based on he ype o unc ion ha has been used o simila p oblems in p e ious
wo ks. Fo example, one o he mos used ac i a ion unc ions o wo k wi h images is he
ReLU unc ion since i only p opaga es posi i es alues. This is specially in e es ing when
he ou pu is an image because he s anda d ange o wo k wi h hem is in [0,1]. The cha ac e -
is ics o he ac i a ion unc ions help he ne wo ks o ocus on he esolu ion o he p oblem.
In con as , selec ing a bad ac i a ion unc ion can diminish and hinde he con e gence o he
aining p ocess.
3.1.3 G adien descen op imiza ion
The op imiza ion p ocess is done h ough he minimiza ion o a unc ion. In deep lea ning he
unc ion o minimize is called he loss unc ion and measu es he pe o mance o he ne wo k
o e a ask. The pe ec solu ion would be o ind he global minimum o he loss. Howe e ,
ha could be impossible due o he complexi y o he pa ame e space. On he o he hand,
inding a local minimum is an easie ask ha can be done wi h g adien descen op imiza ion
me hods.
As a simpli ica ion, we can deno e he loss unc ion as y= (x)which de i a i e is 0(x) =
dy/dx. I is known ha he de i a i e gi es he slope o a he poin x. Wi h he di ec ion o
he slope we know in which di ec ion he unc ion will be minimized and i he de i a i e is
equal o 0 hen he unc ion is in a minimum. can be de ined wi h se e al pa ame e s in he
o m o x={x1,x2,...,xn}. The de i a i e o mus be done wi h pa ial de i a i es ∂ (x)/∂xi.
The g adien o deno ed as ∇ (x)will be he ec o wi h all he pa ial de i a i es. Like he
g adien gi es he slope o e e y pa ame e we can upda e hem in he nega i e di ec ion o he
slope and descending i e a i ely in he g adien . The upda ing unc ion can be deno ed as:
x +1=x −ε∇ (x ),(3.6)
whe e x is x alues in he ac ual i e a ion and εis a small alue called lea ning a e ha
con ols he speed o he descen . The lea ning a e can no be oo high because when he
unc ion is nea o he minimum a huge lea n a e could p e en he unc ion o ind he local

12 Chap e 3. Backg ound
minimum. On he o he hand i he lea ning a e is oo small he needed s eps o ge ing he
minimum poin will be so high ha he lea ning p ocess would ake oo much ime.
3.1.4 S ochas ic g adien descen
The p e ious de ini ion hold o a loss wi h a single aining da a. Howe e , in machine lea ning
is ypical o ha e a hund ed o housands o aining da a. In ha case, he loss unc ion can be
unde s ood as a sum o e he loss o all aining da a. Conside ing he pa ame e s o he model
as θand any loss unc ions o a single da a as Lwe can deno e he loss L o all he example
as
L(θ) = E[L(x,y,θ)] =
∑n
i=1Lx(i),y(i),θ
n.(3.7)
Then, o apply he g adien descen me hod is needed o calcula e he g adien wi h espec
o he pa ame e s o he model θas
∇θL(θ) =
∑n
i=1∇θLx(i),y(i),θ
n.(3.8)
Compu ing he g adien descen wi h his calcula ion is one o he mos success ul op ions.
Howe e , as we commen ed abo e, he size o he da ase is huge and makes di icul aining
wi h he whole da a due o memo y space es ic ions. The solu ion o ha is aining using a
subse o da a which is called ba ch size, and can be de ined wi h any desi ed size. Fo he cases
o eally small ba ches hey a e called miniba ches and o some ields hey shown be e esul s
ha bigge ba ches. Deno ing he loss o hese ba ches as L0and he numbe o elemen s in
he ba ch as n0 he g adien could be de ined as be
∇θL0(θ) =
∑n0
i=1∇θLx(i),y(i),θ
n0.(3.9)
Finally, he g adien descen o he pa ame e s using ba ches is
θ =θ −1−ε∇θL0(θ −1).(3.10)
This a ia ion o he g adien descen is called s ochas ic g adien descen and is one o he
mos used me hods in he aining o neu al ne wo ks. As he o iginal me hod, his one can no
con e ge o he global minimum bu , i will end in he nea es local minimum.
3.2 Non-line-o -sigh (NLOS) ansien imaging
T ansien imaging me hods le e age he in o ma ion encoded in he empo al domain o a ime-
esol ed ligh cap u e. One o he a eas ha can be add essed in his ield is he non-line-o -
sigh (NLOS) econs uc ion. The imaging me hods de eloped in his a ea aim o econs uc
scena ios ha a e hidden a ound a co ne , by analyzing hei indi ec illumina ion on a su ace
isible o he came a. A ypical NLOS scene can be seen in igu e 3.2 whe e an ul a-sho
lase pulse is emi ed o a wall (also called elay wall), he ligh p opaga es om he hidden
scene, being e lec ed by he objec s, and pa o i goes o he elay wall and is eco ded by an
ul a-high speed came a called single-pho on a alanche diode (SPAD) gene a ing a ansien
p o ile Hcalled impulse esponse unc ion. These p o iles ha e he in o ma ion o he hidden
A machine lea ning app oach o ansien imaging econs uc ion 13
scene encoded inside hem. Thus, His calcula ed o di e en poin s o he elay wall o he
lase and he SPAD pa ame e s a e H(xl,xs, )whe e xlis he lase posi ion, xsis he senso
posi ion and a speci ic empo al ins an .
NLOS SPAD-lase cap u e se up
SPAD
lase
Figu e 3.2: Sample se up o cap u ing a NLOS scene. An ul asho ligh pulse is emi ed o a
elay wall and he senso cap u es he ligh esol ed in ime e lec ed by he scene o he elay
wall. Image adap ed om Liu e al.[2].
The impulse esponse unc ion His used in se e al me hods o econs uc ing he hidden
scene as we commen ed in sec ion 2.1. In his wo k we ocus in he phaso ield amewo k
[2] which is one o he bes pe o ming me hods in NLOS imaging. This wo k gi es us he
possibili y o ans o m he H unc ion in o a phaso ield which ans o ms he empo al eso-
lu ion in o a complex magni ude a he elay wall wi h ampli ude and phase dimensions o a
conc e e equency.
3.2.1 Phaso ields
The wo k o Liu e al. [2] p esen s a new me hod o ansien imaging which ans o ms
he NLOS p oblem in o a i ual line-o -sigh (LOS) p oblem. This ans o ma ion allows he
au ho s o use classical op ic me hods in he NLOS domain.
Vi ual illumina ion Vi ual ape u e Vi ual senso
Vi ual lens
(a) (b) (c)
Figu e 3.3: Phaso ield s eps. (a) Vi ual illumina ion is p opaga ed h ough he scene. (b) A
i ual ape u e cap u e he esponse o he scene o he i ual ligh . (c) A i ual lens ocus
he cap u ed illumina ion o he i ual ape u e and imaging. Image adap ed om [2].
14 Chap e 3. Backg ound
The linea i y o H(xl,xs, )is used by phaso ields o compu e he esponse o he hidden
scene (Fig. 3.3b) o any i ual complex- alued emission p o ile P(xl, )(Fig. 3.3a) a poin s
xlin a i ual senso as
P(xs, ) = ZL
[P(xl, )∗H(xl,xs, )]dxl.(3.11)
Th ough he p opaga ion o he ield P(xs, )wi h an imaging ope a o I(·) o any poin
x in he hidden scene as
I(x ) = Φ(P(xs, )).(3.12)
This ope a o models a i ual lens and senso sys em (Fig. 3.3c) and i can be o mula ed
in e ms o a Rayleigh-Somme eld di ac ion p opaga o [2]. In he case o p opaga ing
monoch oma ic signals o a single equency ω his image o ma ion ope a o Φ(Pω(xs, )) is
deno ed as
Φ(Pω(xs, )) = ZS
Pω(xs, )Lω(xs,x )
|x −xs|dxs
2
,(3.13)
whe e Lis a complex ope a o ha changes he phase o Pa equency ω. Fo he Φope a o
we implemen i as a hin lens model ha ocuses in o a i ual image plane a a hidden loca ion
x , ha ing
Lω(xs,x ) = e−ik|x −xs|,(3.14)
whe e k=ω/cis he wa enumbe , wi h c he speed o ligh .
A posi ion x can be imaging by combining he ocused emission p o ile (Eq. 3.11) and he
imaging ope a o (Eq. 3.13) as
I(x ) = ZSZL
[Pω(xl, )∗H(xl,xs, )] Lω(xs,x )
|x −xs|dxldxs
2
.(3.15)
This me hod is ully compu a ional since he unique ex e nal da a ha i need is he H
unc ion. Fu he mo e, he image o ma ion model is i ual and can be o mula ed in o he
o ms. The same idea holds o he emission p o ile and o his wo k we will de ine i as a
cons an -emission ligh sou ce as
Pω(xl, ) = eiω .(3.16)
Chap e 4
Implici ep esen a ion o phaso ields
In he las decade, he ield o non-line-o -sigh (NLOS) has e ol ed imp o ing he quali y o
he econs uc ions o hidden scenes. Howe e , he cap u e me hod has ba ely changed. NLOS
me hods use a cap u ed impulse esponse unc ion o a hidden scene. To ob ain his unc ion
a ligh is emi ed, by an ul a-sho lase , o a di use wall ( elay wall). Then i a els ac oss
he hidden scene and goes back o he elay wall whe e i is inally cap u ed by a senso . The
cap u e p ocess needs o be done sequen ially o each poin in he elay wall, esul ing in a
slow p ocess wi h a bo leneck ha o ces o ha e impulses esponse unc ions wi h low spa ial
esolu ion.
Despi e he p oblem wi h he spa ial esolu ion, wo ks as Liu e al. [2] success ully econ-
s uc high complexi y hidden scenes. Howe e , o ob ain mo e de ailed econs uc ions, i is
manda o y o gene a e mo e spa ially dense impulse esponse unc ions. To ob ain hem he e
a e wo op ions: c ea ing new ha dwa e o inc ease he spa ial esolu ion o using compu a-
ional me hods o gene a e new poin s om he cap u ed ones.
On he one hand c ea ing ha dwa e o inc ease he spa ial esolu ion o he senso is no
a i ial wo k. The ac ual senso s a e single-pho on a alanche diode (SPAD). They ha e a
esolu ion o 1x1 pixel and he e a e some e sions wi h a line o SPADs ob aining a ow o
cap u es. Despi e hese ad ances, he nex gene a ions o 2D SPADs will ha e a low esolu ion
nea 16x16 pixels which is insu icien o sol e he p oblem. On he o he hand, using ully
compu a ional me hods can wo k wi h he ac ual cap u e sys ems. We can es and implemen
mul iple algo i hms o e sions o hem in a ela i ely sho space o ime. In spi e o he
ad an ages ha compu a ional me hods can gi e us, he main limi a ion o us o design and
c ea e new cap u e ha dwa e is he necessi y o es ing i wi h eal cap u e sys ems which only
a ew labo a o ies in he wo ld ha e. In con as o es compu a ional me hods we can use
bo h eal and simula ed da a, and he e o e, alida e he me hod by using hem. Fo ha eason
we ha e decided o implemen a compu a ional me hod and gene a e, h ough simula ion, ou
cap u e da a.
The p oblem ha we wan o alle ia e is he low densi y o alues in he spa ial domain o he
impulse esponse unc ion o imp o e he quali y o he econs uc ions. We can say ha his
p oblem can be unde s ood as an upsampling o supe esolu ion p oblem whe e he numbe o
samples o a unc ion is inc eased by an algo i hm. This p oblem is ypical in images whe e
he low esolu ion image is ans o med in o a high esolu ion one. The ype o me hods ha
gi es be e esul s in his domain a e he machine lea ning based ones. Howe e , o wo k wi h
15
22 Chap e 4. Implici ep esen a ion o phaso ields
Figu e 4.5: Example o H unc ion o a conc e e pai lase -senso . The op image is he
ansien p o ile in i s o iginal scale. The bo om image is he ansien p o ile in loga i hmic
scale.
To sol e he p oblem wi h he high dynamic ange o he H unc ion, we p opose o use he
phaso ields amewo k (sec ion 3.2.1) o NLOS econs uc ion. In he i s s ep o his ame-
wo k he H unc ion is con ol ed (eq. 3.11) wi h a i ual illumina ion phaso o a conc e e
equency ω(eq. 3.16). This phaso P(xs, )is a 3D complex ma ix, wo spa ial dimensions
and one empo al, bu as he i ual ligh used in he con olu ion is cons an along he ime and
he came a model is a con en ional came a, P(xs, )can be e alua ed in P(xs, =0)ob ain-
ing a 2D complex ma ix and sol ing he p oblem wi h he high dynamic ange. Howe e , he
equency ha modules he i ual illumina ion a ec s o he econs uc ion quali y as can be
seen in igu e 4.6. Fu he mo e, he co ec equency o each H unc ion is no he same and
needs o be ob ained expe imen ally.
F equency
Hidden scene +_
Figu e 4.6: Example o di e en econs uc ions wi h di e en equencies.
Da a no maliza ion. In his wo k we ha e wo ypes o da a: he inpu pa ame e s o he
model (coo dina es and in some models he equency) and he alues o he phaso which he
model will lea n. Fo bo h op ions we es ed he [0,1]and [−1,1]no maliza ion bu he models

A machine lea ning app oach o ansien imaging econs uc ion 23
did no i co ec ly in he bo de s o he ma ix. Since he p oblem was in he bounda ies we
conside ha he p oblem was wi h he no maliza ion alues in he limi s o he ange and wi h
a no maliza ion in (−1,1) his p oblem was pallia ed.
Da a gene a ion. The cap u e o he H unc ions can be done om wo di e en sou ces,
eal cap u e sys ems o h ough simula ion. The i s one can no be done in he labo a o y
a his uni e si y due o we do no ha e he ha dwa e. Howe e , using eal cap u ed da a
ha e some disad an ages. The cap u es ha e noise and hei condi ions can no be con olled
comple ely. On he o he hand, simula ing a oids his p oblems due o he en i onmen can
be con olled (e.g., a oid ex e nal ligh con amina ion) and can be gene a ed wi h less noise.
Also, simula ing allow o c ea e idyllic condi ions e.g., pe ec lambe ian su aces o he elay
wall. Fo ha eason in his wo k we simula e all he cap u es using he public ansien ende
by Ja abo e al. [13].
Gene ic model s speci ic models. Neu al ne wo ks a e usually applied o gene ic appli-
ca ions ha can be used once is ained wi h di e en da a, o example in supe esolu ion
models, classi ica ion o some comp ession wo ks. Howe e , his models need o be ained
wi h eno mous da ase s, in some cases millions o samples. In he con ex o his wo k his
is no an op ion due o he amoun o ime ha is needed o gene a e a da ase o he size ha
gene al models need. Fo ha eason in his wo k we choose o use one model pe unc ion (o
scene) ha we wan o i .
Since we a e i ing a single unc ion in he neu al model, and we wan o in e new alues,
we need o a oid o e i ing in he model. In o he cases, i we wan ed o comple ely comp ess
a unc ion and only e alua e a he o iginal poin s, o e i ing would be jus i ied. O he aspec
ha is in ol ed in he o e i ing and in he capaci y o he neu al models o be able o i he
disc e e unc ion is he numbe o model pa ame e s. The co ec numbe o hem needs o
be ob ained expe imen ally. Mo eo e , i can depend on he ype o unc ion ha he model is
i ing, complex unc ions can equi e mo e pa ame e s han he simples ones and i he numbe
o pa ame e s is oo big, he model will be lazy and will o e i easily. In he con ex o his
p oblem, we can conside an uppe bound o he numbe o pa ame e s, he o al amoun o
alues in he disc e e unc ion because i is he numbe o alues needed o ep esen he same
in o ma ion. In ou expe imen s we es ed wi h di e en numbe o pa ame e s and igu ed ou
ha wi h mo e han hal he o al numbe o pa ame e s o he o iginal unc ion, he models
end o o e i and can no gene a e well new poin s. On he o he hand, wi h less han he hal
he models lea n he disc e e unc ion and a e able o gene a e new poin s smoo hly. Howe e ,
i he pa ame e s a e oo low, he models can no con e ge a all.
24 Chap e 4. Implici ep esen a ion o phaso ields
Chap e 5
Resul s
In he sec ion 4.1 we ha e es ed di e en neu al ne wo ks models o lea n an implici ep-
esen a ion o a phaso ield. Speci ically, he neu al ne wo k model ob ained can lea n an
implici ep esen a ion o a mul i- equency phaso ield. One o he ad an ages o lea ning a
mul i- equency phaso ield is he elimina ion o he necessi y o calcula ing sepa a e implici
ep esen a ions o each equency. This means ha i gi es he abili y o compu e any desi ed
equency ha i is no in he sampled space. This is especially in e es ing due o he equency
is di ec ly dependen o he quali y o he econs uc ions. Compu ing no el phaso ields wi h
highe equencies and highe g id densi ies can p o ide sha pe esul s o he econs uc ed
geome y. As such, he abili y o ou implici ep esen a ion o compu e no el empo al e-
quencies and spa ial loca ions in he elay wall ha we e no p o ided by he sampled cap u ed
da a is use ul o p o ide be e econs uc ions. Ano he applica ion o his mul i- equency
model is he possibili y o use i wi h mo e complex ligh s and came as in he phaso ield
amewo k. To use hem i will be equi ed ( heo e ically) he use o in ini e equencies. Wi h
an implici ep esen a ion, his p oblem can be mo e ea able because he ep esen a ion a oids
he necessi y o do any con olu ion o each equency.
Du ing he analysis o he di e en neu al models (Sec ion 4) we ha e es ed hem wi h
only one scene o simplici y. A e demons a ing ha he MLP wi h sine ac i a ion unc ions
model p o ides he bes beha io in ep esen ing mul i- equency phaso ields, he e we p o-
ide a deepe analysis o i s pe o mance in scenes o di e en complexi y. Howe e , be o e
showing he di e en esul s, is impo an o unde s and he limi a ions o he i ual came a
and he i ual ligh used in he phaso ield amewo k. The i ual came a has an ape u e size
ha co esponds o he size o he sampled g id on he elay wall. A e using his ape u e o
simula e a i ual came a ocused a ce ain dep h in he hidden scene, all geome y behind o
ahead o he ocused plane is ou o ocus, in oducing se e al a i ac s in he image. Addi ion-
ally, due o cap u e limi a ions, he i ual illumina ion is a enua ed adially om he cen e
o he elay wall, which esul s in a enua ed econs uc ions o he geome y u he om he
cen e o he esul ing images. This e ec is clea ly isible in all he p e ious esul s in he sec-
ion 4.1. To analyze how he implici ep esen a ions can app oxima e he disc e e phaso ield
we will compa e he in ensi y, he phase and he econs uc ion o bo h phaso ields, disc e e
(g ound u h) and he one p edic ed by he implici ep esen a ion. We ha e selec ed sim-
ple scenes o ha e low ou -o - ocus geome y o a oid ou -o - ocus a i ac s. Recons uc ing
he in- ocus geome y in mo e complex scena ios would equi e es ima ing a la ge amoun o
di e en phaso ields o each oxel o he econs uc ed scene, while simple scenes p o ide
good esul s wi h a single mul i- equency phaso ield. We ega d as u u e wo k o p o ide
25
26 Chap e 5. Resul s
mo e complex implici ep esen a ions o phaso ields in clu e ed scenes.
As in he las model o he sec ion 4.1, we will es he capaci y o he model o gene a e
new samples in he spa ial and equency domain. To ain and es he model we ha e decided
o emo e 75% o he samples om a phaso ield wi h a spa ial esolu ion o 64x64, he e o e
ge ing only 32x32 sampled poin s in he elay wall. Fo he equency, we ha e compu ed 151
di e en equencies and ained wi h he 90% o hem, ese ing he las 10% o es ing. Once
he model is ained wi h he low esolu ion phaso ield, we eco e he o iginal esolu ion a
speci ic equencies and spa ial loca ions in he elay wall ha he model has no been ained
wi h.
Fo es ing he model, a mul ilaye pe cep on wi h sine ac i a ion unc ions, we ha e used
ou di e en scenes ( igu es 5.1-5.4). We s a by a plana bu de ailed scene ( igu e 5.1). This
scene allows us o analyze he pe o mance unde di e en geome ic esolu ions allowing see-
ing he le el o de ails ha he me hod is able o eco e . Conc e ely, he esul s ob ained o
his scene show he capabili y o he model o lea n mo e complex phaso ields and upsample
hem co ec ly.The gene al s uc u e is main ained and an impo an denoised e ec is app e-
ciable. The econs uc ions a e quali a i ely simila . They lose a bi o de ail bu , bea ing in
mind ha he model is ained wi h he 25% o he o iginal poin s, i is a good econs uc ion.
As compa ison, in he igu e 5.2 can be seen how is a econs uc ion o a phaso ield wi h a
esolu ion o 32x32 and a econs uc ion wi h a esolu ion o 64x64. In he e o map i can
be seen wo hings: he e o de i ed om he noise no lea ned by he model and how he
ne wo k ails mo e in he low alues.
Figu e 5.1: Resul s o eco e ing a scene wi h mul iple planes wi h di e en sizes. The esul s
show ha he implici ep esen a ion lea n co ec ly he shape o he phaso ields and emo e
he noise o hem. The econs uc ions a e quali a i ely simila . Le phaso ield has a e-
quency o 11.422 and he ig h one has a equency o 15,238.
The second scene is o med by a plane wi h se e al conca i ies and i egula i ies ( igu e
5.3). No e how he esul ing ampli ude and phase o he phaso ield change due o he s uc-
u al di e ences o he hidden geome y. Ne e heless, ou implici ep esen a ion is able o
es ima e he phaso ield s uc u e a highe esolu ions, e y close o he g ound u h while
also emo ing noise, and p o ides a simila econs uc ion esul . Ou implici ep esen a ion
A machine lea ning app oach o ansien imaging econs uc ion 27
Figu e 5.2: Compa ison o he econs uc ion o he same scene wi h di e en esolu ion,
32x32 and 64x64 o he same equency (15.238).
is capable o lea ning he gene al s uc u e, while simul aneously emo ing he noise in he am-
pli ude and phase images. The e o maps show, as in he p e ious example, ha he model has
mo e e o in he low alue poin s and also he gene al noise e o . The econs uc ions a e e y
simila . Despi e ha he RMSE in he i s econs uc ion is highe , he esul is quali a i ely
mo e simila han in he second econs uc ion.
Figu e 5.3: Resul s o a scene wi h mul iple conca i ies and pikes. The econs uc ions do
no show he shape co ec ly due o he ou -o - ocus geome y. The neu al model lea ns and
eco e he shape o he phaso ield emo ing he noise om i .
Finally, we ha e wo simila scenes ( igu es 5.4 and 5.5). Bo h a e o med by wo ec angula
planes a wo di e en dep hs bu in each scene he dis ance be ween hem is di e en . The
goal o hese wo scenes is o es he pe o mance o ou algo i hm unde he p esence o
objec s ha may occlude hemsel es om he pe spec i e o he i ual came a. Fo hese wo
cases ou implici neu al model lea ns he s uc u e o he phaso ields as well.
Commen s and discussions. Ou neu al ep esen a ion o phaso ields has shown a high
capaci y o lea n he gene al s uc u e o he phaso ield, being able o gene a e highe eso-
lu ions ( eco e ing a 75% o he poin s in hese es s) and new equencies wi h simila esul s
o he g ound u h. The RSME is simila in all o hem bu is bigge in he phaso ield wi h
highe equency. This could be caused by he inc ease o de ails ha appea when he em-
po al modula ion equency is inc eased. This inc ease in he empo al equency esul s in

28 Chap e 5. Resul s
Figu e 5.4: Resul s o a scene wi h wo planes a di e en dep h. This scene allows us o es
i he phaso ield wi h mul iple objec s and dep hs can be lea ned by he neu al model. The
model lea ns co ec ly despi e he changes in he shape.
Figu e 5.5: Resul s o a scene wi h wo planes a di e en dep h. In his case he plane in he
back is semioccluded by he plane in he on . The neu al model lea ns co ec ly he shape
and e en he s ong change o alues.
an inc emen o he spa ial equency in he phaso ield, as he esul s show. Focusing on
he econs uc ions, he p edic ed and he g ound u h a e also e y simila and can eco e
e en he a i ac s p oduced by di ac ion e ec s. No e ha his is a p oblem inhe en in he
phaso ields me hod [2], and no a esul o ou neu al model. The bigges quali a i e di e -
ences be ween p edic ed and g ound u h appea in he econs uc ions done wi h he phaso
ields wi h highe equencies, losing mo e de ails han wi h he low equency phaso ields.
No e ha ou quan i a i e compa isons a e done wi h espec o he g ound u h solu ion wi h
equi alen sampling densi y and empo al equency. While ou model yields a la ge RMSE
when inc easing modula ion equency, inc easing sampling densi y and modula ion equency
in phaso ields p o ides be e econs uc ions o he hidden scene, as ou quali a i e esul s
show. The di e ences in RMSE o ou model wi h espec o lowe equencies a ise om he
abili y o ou model o analy ically ep esen highe equencies, and no om he abili y o he
da a o p ope ly econs uc he scene.
A machine lea ning app oach o ansien imaging econs uc ion 29
Hidden scene Loss (L2) T aining ime (s)
Figu e 5.1 0.009883426 677.92
Figu e 5.3 0.011194097 670.95
Figu e 5.5 0.0029691448 673.33
Figu e 5.4 0.0032978356 673.98
Table 5.1: Losses and aining imes o each scene. The aining p ocess ha e been done wi h
a lea ning a e o 0.001 and a o al o 100000 epochs.
The model in all ou es s has shown a denoising abili y. This e ec is p oduced by he
capabili y o neu al ne wo ks o ocus on p edic able aspec s o he da a. Conc e ely, as he
noise is some hing mo e andom han he s uc u e o he da a, he neu al ne wo ks ob ain be -
e esul s lea ning he s uc u e and cha ac e is ics o he da a. Howe e , his e ec is only
obse ed when he pa ame e s o he ne wo ks a e no oo high because, in ha case, he neu al
ne wo k o e i s he da a and canno upsample. Rega ding he possible e ec o he noise in
he e o maps, we hough ha i is possible ha he noise could mask he co ec le el o
achie emen o his me hod. Howe e , o es i , i would be necessa y o use o he me ics ha
ake mo e in o accoun he s uc u e o he nea by pixels as he s uc u al simila i y index mea-
su e (SSIM). We will lea e he es o his hypo hesis o a u u e wo k. In he econs uc ions,
he e ec o he exis ence o noise in he phaso ield does no appea o be de e minan o a
leas he e ec is oo low o be seen. In hei e o maps, he i s di e ence ega ding he e o
maps o he phaso ield is he lack o noise. The e o maps o he econs uc ions a e mo e
smoo hly han he e o maps o he phaso ields. We hough ha he econs uc ion p ocess
is esis an o his le el o noise. Wi h his idea, a u u e wo k ha could be done is an analysis
o he ole ance o he econs uc ion me hod a di e en le els o noise and how his me hod
can be used o denoising i . This is an in e es ing aspec because he eal cap u es ha e noise
ha usually needs o be added o he syn he ic da a o compa e bo h. Fu he mo e, knowing
ha , i could help o know he le el o noise ha his me hod can handle and use i o gene a e
syn he ic da a as e .
The aining p ocess o hese scenes is as (see able 5.1 o exac aining imes). In a ound
ele en minu es each scene can be lea ned. The e alua ion p ocess o each equency is p ac i-
cally ins an , less han one second, gi ing a huge po en ial o use i o mo e applica ions such
as using mo e complex came as ha could equi e compu ing housands o di e en equen-
cies. Mo eo e , he model can be e alua ed wi h mul iple equencies a he same ime wi hou
a pe cep ible inc ease o ime.
30 Chap e 5. Resul s
Chap e 6
Conclusions and u u e wo k
In his wo k we ha e in oduced a new me hod o ep esen phaso ields in an implici o m
by using neu al models o inc emen he e iciency and he quali y o he esul s o econs uc-
ions o hidden scenes. Phaso ields a e a ecen wo k ha allows o ake i ual pho og aphs
o a hidden scene as i was seen om a elay wall. Howe e , his me hod is limi ed by in i s
esolu ion caused by he cap u e p ocess. Since each spa ial poin has o be sequen ially cap-
u ed, i becomes unp ac ical and complex o cap u e high esolu ion in o ma ion. Al hough
his p oblem could be alle ia ed wi h he de elopmen o new ha dwa e, his app oach would
equi e he use o expensi e ha dwa e con igu a ions.
Inspi ed by o he wo ks on implici ep esen a ions ha p o e hei abili y o c ea e a con in-
uous space om disc e e numbe o samples. We ha e o mula ed a phaso ield using implici
ep esen a ion which allows us o lea n a disc e e phaso ield and sample i in a con inuous
space. This ans o ma ion emo es any esolu ion limi a ion. In o he wo ds, we could ob ain
in ini e samples om i . Following ecen s udies and wo ks, we ha e es ed di e en models
o di e en asks. S a ing by he mos simple (lea ning an implici ep esen a ion o a single-
equency phaso ield wi hou changing he esolu ion), up o he mos complex ( ha lea ns an
implici ep esen a ion o a mul i- equency phaso ield). This las model is a mul ilaye pe -
cep on wi h sine ac i a ion unc ions ha can upsample he da a in he spa ial and equency
domains.
To e i y i he inal model can gene alize o di e en scenes we ha e es ed i wi h ou
scenes. The model co ec ly upsample he da a o all he scenes gi ing a simila e o in all o
hem. O he aspec obse ed is he denoising e ec , since he model eco e s smoo h phaso
ields wi h he co ec s uc u e. This can be quali a i ely demons a ed, as he econs uc ed
scenes a e almos iden ical o he g ound- u h ones.
This wo k le some in e es ing esea ch as u u e a enues. Ou cu en me hod could be
ex ended o suppo mo e clu e ed scenes wi h mo e complex imaging unc ions, whe e each
loca ion o he scene would equi e a di e en illumina ion and lens unc ion, leading o he
es ima ion o highe numbe o phaso ields pe scene. An aspec ha can be es ed is whe he
he noise in he cap u es is ela ed wi h he di e ences in he econs uc ions. Rela ed o he
noise, i could be in e es ing o analyze how he noise a ec s o he econs uc ions and i his
model o o he ypes o ne wo k can denoise he phaso ield and wi h which le el o noise
a e hey e ec i e. Mo eo e , his wo k could be expanded o he s udy o he whole ou
dimensions in ol ed in phaso ields, in con as o he wo explo ed in his wo k. Ano he
31