Ti le: Ad anced Me hods o Op ical
Cons ella ion Analysis and Comp ession
Au ho : Raúl López Ma ínez
Ad iso : Luis Velasco and Ma c Ruiz
Depa men : Compu e A chi ec u e
Uni e si y: UPC
Academic yea : 2021-2022
In e uni e si y Mas e
in S a is ics and
Ope a ions Resea ch
UPC-UB
Ad anced Me hods o Op ical Cons ella ion Analysis and Comp ession
palab a
2
Uni e si a Poli `ecnica de Ca alunya
Facul a de Ma em`a iques i Es ad´ıs ica
Mas e ’s Deg ee in S a is ics and Ope a ions Resea ch
Mas e ’s Deg ee Thesis
Ad anced Me hods o Op ical
Cons ella ion Analysis and Comp ession
Ra´ul L´opez Ma ´ınez
Supe ised by Luis Velasco and Ma c Ruiz
Sep embe , 2022
Thanks o all he eache s and s uden s o he acul y, hey we e one o he keys easons o me o lo e
hese s udies.
Thanks o my p ojec ad iso s o he in e es hey pu on his p ojec .
Thanks o Base is o encou aging me o keep s udying and ex ending my knowledge.
Thanks o my amily o always suppo ing my decisions du ing hese las 2 yea s and o educa ing
me in he alues ha made me he pe son who I am oday.
Thanks o all my iends and belo ed ones, i is always a pleasu e be su ounded by such nice people.
Special hanks o Ai o Muna and Ti`a Roig o being inc edible human beings. Thanks o all he momen s
we sha ed and e e y hing I lea n om hem. We miss you. Res in peace.

Abs ac
Op ical cons ella ions o e a highly dimensional ep esen a ion o he signals in op ical anspo ne wo k
echnologies and hey can be analyzed o se e al use cases such as op ical ne wo k heal h analysis and secu e
op ical ne wo ks. I is c ucial o ope a o s o implemen e icien moni o ing a chi ec u es ha mi iga e
po en ial d awbacks (such as high capaci y exhaus ion) while ensu ing he alidi y and eliabili y o widely
collec ed moni o ing da a. In his p ojec we aim o p o ide obus echniques o op ical cons ella ion
analysis and comp ession achie ing la ge comp ession a es wi h negligible in o ma ion loss. In pa icula ,
op ical cons ella ions will be cha ac e ised h ough pa ame ic p obabili y dis ibu ions and comp essed
h ough au oencode a chi ec u es.
Keywo ds
op ical ne wo ks, op ical cons ella ion, machine lea ning, deep lea ning, neu al ne wo ks, dimensionali y
educ ion, au oencode s
1
palab a
2
Con en s
1 In oduc ion 6
1.1 Mo i a ion........................................... 6
1.2 P e iouswo k ......................................... 6
1.3 Objec i es ........................................... 6
1.4 Documen o ganisa ion .................................... 7
2 Backg ound 8
2.1 Op icalCons ella ionda a................................... 8
2.2 Mix u edis ibu ions ..................................... 11
2.3 MachineLea ning ....................................... 12
2.4 Dimensionali yReduc ion................................... 14
2.5 Summa y............................................ 16
3 P ep ocess 17
3.1 The need o p ep ocess he da a . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.2 Spa ialdis ibu ions...................................... 17
3.3 Resul s............................................. 20
3.4 Summa y............................................ 22
4 Dimensionali y educ ion: da a econs uc ion 24
4.1 GaussianMix u es....................................... 24
4.2 Imageda a........................................... 25
4.3 Resul s............................................. 26
4.4 Summa y............................................ 31
5 Use case: Da a augmen a ion 32
5.1 Mo i a ion........................................... 32
5.2 Objec i es ........................................... 32
5.3 App oach............................................ 33
5.4 Resul s............................................. 33
5.5 Summa y............................................ 36
6 Concluding ema ks 37
6.1 Conclusions .......................................... 37
6.2 Pe sonal hough s....................................... 37
6.3 Fu u ewo k .......................................... 38
3
Ad anced Me hods o Op ical Cons ella ion Analysis and Comp ession
Figu e 2: Sequence o spans in a single link scena io (sou ce: [7]).
The da ase dis inguishes be ween h ee con igu a ions o he simula o : op imal, sub-op imal and deg a-
da ion. The di e en con igu a ions in oduce powe a ia ions ha esul in small changes in he op ical
cons ella ions wi hou wo sening ligh pa hs’ QoT.
The da ase consis s o a se o op ical cons ella ion ins ances, each o hem con o med by a se o 2048
poin s in he complex plane. The e a e 1250 ins ances om op imal and 500 om each o sub-op imal and
deg ada ion scena ios.
Figu e 3: One o he cons ella ion diag ams in OCATA 16QAM-singleLink-s1 da ase (sou ce: [7]).
Fo he execu ion o his p ojec , he da a has been spli ed (in a s a i ied manne ) so ha 20% is kep
o es ing he pe o mance o he de eloped models.
10

2.2 Mix u e dis ibu ions
In p obabili y heo y, a mix u e dis ibu ion[8] is he p obabili y dis ibu ion o a andom a iable which
de ini ion depends on a collec ion o o he andom a iables in he ollowing way:
•Fi s , a andom a iable om he collec ion is selec ed acco ding o gi en p obabili ies o selec ion.
•Finally, he alue o he andomly selec ed andom a iable is ealized.
The unde lying andom a iables can be uni a ia e o mul i a ia e ( andom ec o s), ha ing all o hem
he same dimension. In he second case, he mix u e dis ibu ion would be a mul i a ia e dis ibu ion.
Gi en a ini e o coun able se I, p obabili y densi y unc ions { i(
x)}i∈Iand weigh s {πi}i∈Isuch ha
πi≥0 and Pi∈Iπi= 1, he mix u e dis ibu ion is de ined by i s p obabili y densi y unc ion
(
x) = X
i∈I
πi· i(
x)
Usually, he mix u e componen s a e no a bi a y p obabili y dis ibu ions bu membe s o a pa ame ic
amily ( o example gaussians). In his case he densi y unc ion can be w i en in a sum o m as:
(
x;
θ) = X
i∈I
πi· i(
x;
θi)
whe e 
θiis he ec o o pa ame e s ha de ines he dis ibu ion o he componen io he mix u e and 
θ
is he conca ena ion o all he pa ame e s {
θi}i∈I.
Al hough we ha e only co e ed he case in which he collec ion o componen dis ibu ions is coun able,
he e’s an analogous kind o dis ibu ions o he case in which i is uncoun able: compound p obabili y
dis ibu ion. Howe e , in his p ojec we will be wo king wi h he pa icula case o ini e mix u es whe e
componen s a e om he same pa ame ic amily.
A his poin , one can al eady no ice ha mix u es o 16 2D-gaussian dis ibu ions can be a good i
o cha ac e ize he samples o 16QAM op ical cons ella ion diag ams (like he ones in he OCATA 6QAM-
singleLink-s1 da ase ) since hei symbols should be loca ed in clus e s a ound he heo ical posi ion o he
16 cons ella ion poin s.
Figu e 4: Densi y o a 2D Gaussian Mix u e wi h h ee componen s.
11
Ad anced Me hods o Op ical Cons ella ion Analysis and Comp ession
2.3 Machine Lea ning
Machine Lea ning (ML) is a ield ha alls in he in e sec ion be ween p obabili y heo y, s a is ics, op i-
miza ion and compu e science. I is de o ed o unde s anding and building me hods ha lea n om da a,
i.e, me hods ha use da a o imp o e he pe o mance o a ma hema ical model on pa icula asks [9].
Depending on he na u e o he asks, we can dis inguish be ween di e en ypes o machine lea ning
algo i hms. The wo main g oups a e supe ised and unsupe ised algo i hms.
•Supe ised Lea ning: We say ha a machine lea ning algo i hm is supe ised i i is ained h ough
inpu -ou pu example pai s.
•Unsupe ised Lea ning: Unsupe ised lea ning algo i hms a e he ones ha ha lea n pa e ns
om un agged da a. The machine builds la en ep esen a ions o da a h ough mimic y and hen
i is able o gene a e imagina i e con en om i .
The main machine lea ning algo i hms ha will be used in his p ojec all in o he ield o un-
supe ised lea ning.
•O he : O he supe ision le els in he spec um a e semi-supe ised lea ning and ein o cemen
lea ning.
Semi-supe ised lea ning comes om he impossibili y o ha e all you da a agged while s ill wan -
ing o ex ac he alue om he un agged da a. I combines unsupe ised and supe ised algo i hms.
Rein o cemen Lea ning (RL) is conce ned wi h how in elligen agen s migh ac in an en i on-
men in o de o maximize a no ion o cumula i e ewa d. The cumula i e ewa d is no hing else
han a pe o mance sco e ha he agen ecei es om he en i onmen as eedback depending on
he ou comes o he aken ac ions.
Gene alized linea models
In s a is ics, a gene alized linea model (GLM) is a gene aliza ion o o dina y linea eg ession. The GLM
gene alizes linea eg ession by making he linea model o be ela ed o he esponse a iable h ough a
link unc ion and by allowing he magni ude o he a iance o he measu emen s o be a unc ion o he
p edic ed alues. The pa icula GLM case o o dina y linea eg ession akes place when he link unc ion
is he iden i y. GLMs main ad an ages a e ha hey a e easy o implemen , as o i and highly explainable.
In a GLM, we assume ha he ou come Yo he dependen a iables Xis gene a ed om a pa icula
dis ibu ion in an exponen ial amily. The mean o he esponse dis ibu ion depends on he independen
a iables h ough he equa ion:
E(Y|X) = µ=g−1(X·β)
whe e E(Y|X) is he expec ed alue o Ycondi ional on X,X·βis he linea p edic o and gis he
link unc ion. GMLs equi e ha he link unc ion is mono onic and di e en iable o e he ange o possi-
ble alues o µ. The unknown pa ame e s o GLM, i.e he ec o β, a e mean o be es ima ed h ough da a.
12
In his p ojec , linea models will be used o ge baseline pe o mance me ics ha will be aimed o
be bea en by mo e complex models explained nex .
Deep Lea ning
The e m Deep Lea ning e e s o a amily o machine lea ning me hods based on A i icial Neu al Ne wo ks
wi h ep esen a ion lea ning. Lea ning can be supe ised, semi-supe ised o unsupe ised. A i icial Neu al
Ne wo ks a e composed by mul iple laye s, which a e ini e collec ions o GLMs also known as neu ons, o
p og essi ely ex ac highe -le el ea u es om he aw inpu .
The Deep Lea ning app oach is cu en ly s a e-o - he-a in p e y much e e y o he machine lea ning
and AI p oblems ha people do esea ch in. Ne e heless, i usually needs a as amoun o da a and
compu a ional esou ces o achie e such pe o mance.
Figu e 5: G aphical schema o an A i icial Neu al Ne wo k
This is an example o a pa icula ype o A i icial Neu al Ne wo ks whe e he e a e no cycles in he
edges ha join neu ons, wha is called a eed- o wa d ne wo k. The o he main ype o ANN, ecu en
neu al ne wo ks, ha e ecu en connec ions be ween neu ons and hey a e used in p oblems whe e ime
dependencies o he inpu ha e o be aken in o accoun by he ne wo k.
To easily unde s and how Neu al Ne wo ks wo k, i is be e o ocus on wha goes on in a neu on o
a hidden o ou pu laye .
Figu e 6: The compu a ions ha ake place in a neu on
13
Ad anced Me hods o Op ical Cons ella ion Analysis and Comp ession
This could be an a i icial neu al ne wo k ha is inside he p e ious one. The ou pu o a neu on is a GLM
o i s inpu alues, wi h i s weigh s and link unc ion. In deep lea ning he e m link unc ion is no used,
ins ead, hey call ac i a ion unc ion (sigma in he igu e abo e) o i s in e se. A e y common (bu no
he only) ac i a ion unc ion is he sigmoid, which leads o a neu on: a=σ(Pωixi) whe e σ(z) = 1
1+e−x.
The weigh s is wha an algo i hm aims o lea n. The weigh s o he whole ne wo ks a e ini ialised ( andomly
o no ) and hey a e ine- uned o op imise he pe o mance o he ne wo k in a speci ic ask based on
some aining da a.
In his p ojec , deep lea ning will be he go o app oach o y o bea he achie emen s o (gene al-
ized) linea model baselines.
2.4 Dimensionali y Reduc ion
The e m Dimensionali y Reduc ion e e s o ma hema ical ans o ma ions o da a ha ans o m i om
a high-dimensional space in o a lowe -dimensional one, also known as la en space, so ha he la en
ep esen a ion e ains he impo an p ope ies o he o iginal da a. Ideally, he dimension o he la en
space should be close o he in insic dimension o he da a.
Wo king in high-dimensional spaces can be undesi able o many easons:
•As a consequence o he cu se o dimensionali y, aw da a is o en spa se.
•Analyzing such da a can e en be compu a ionally in ac able.
Dimensionali y educ ion applica ion shines in ields ha deal wi h la ge numbe s o a iables and/o
obse a ions. Some o hese ields a e signal p ocessing, speech ecogni ion, neu oin o ma ics, and bioin-
o ma ics.
In his p ojec , dimensionali y educ ion echniques will be applied o op ical cons ella ions da a so he
p ojec alls in o he ield o signal p ocessing.
Among all he possibili ies ha Machine Lea ning o e in e ms o educ ion o dimension models, we
will be using P incipal Componen Analysis (PCA) as a baseline linea model and Deep Au oencode s will
be used o y o bea PCAs pe o mance.
P incipal Componen s Analysis
The P incipal Componen s o a ini e se o poin s ha li e in a eal coo dina e space a e a sequence o p
uni ec o s. They a e de ined so ha he i- h ec o is he di ec ion o he line ha bes i s he da a while
being o hogonal o he i s i−1 ec o s. The bes - i ing lines a e ob ained by leas squa es eg ession,
he e o e hey a e he ones ha minimize he a e age squa ed euclidean dis ance om he poin s o he
line. The di ec ions h ough his i e a i e me hod cons i u e an o hono mal basis o he p-dimensional
space whe e he aw da a li es in.
P incipal componen analysis (PCA) s ands o he p ocess o compu ing he p incipal componen s o
some da a and using hem o pe o m a change o basis on he da a.
14
PCA is commonly used o explo a o y da a analysis pu poses and o dimensionali y educ ion by p o-
jec ing he da a poin s on o only he i s ˆp<pp incipal componen s. This way, lowe -dimensional da a
is ob ained while p ese ing as much o he da a’s a iance as possible.
This me hod has he ad an age ha as o ain, easy o implemen and esul s a e in e p e able. How-
e e , i s main d awback is ha i assumes ha he componen s a e o hogonal linea combina ions o he
o iginal ea u es and his assump ion is no held in some kinds o da a.
Figu e 7: Illus a ion o he ob ained P incipal Componen s on 2-dimensional da a.
Deep Au oencode s
An encode is any unc ion ha maps some da a in o a lowe -dimensional space. A decode is a unc ion
ha does he same bu om he lowe -dimensional space o he o iginal inpu space. Any conca ena ion
o an encode and a decode is known as an au oencode a chi ec u e. When he encode and decode a e
neu al ne wo ks, we say ha he au oencode is a deep au oencode .
A Deep Au oencode is a ype o deep lea ning model ha is used o lea n e icien encodings o un-
labeled da a (unsupe ised lea ning). The encoding is e ined du ing he ain phase by a emp ing o
egene a e he o iginal inpu om he gene a ed encodings (o codes) h ough he decode . This way, he
ne wo k lea ns lowe -dimensional ep esen a ions o he da a by igno ing he insigni ican noise o he da a
ins ances.
Al hough in his p ojec we will be using au oencode s mainly o dimensionali y educ ion pu poses, hey
a e also applied o many o he p oblems, including acial ecogni ion, ea u e de ec ion, anomaly de ec ion
and syn he ic da a gene a ion.
15

Ad anced Me hods o Op ical Cons ella ion Analysis and Comp ession
Figu e 8: Schema o an au oencode ne wo k.
2.5 Summa y
Op ical cons ella ions is a way in which signals can be ep esen ed. In his p ojec , we will be wo king wi h
a da ase o op ical cons ella ions: he OCATA 16QAM-singleLink-s1 da ase . This da ase is public and
con ains samples o simula ed op ical cons ella ions unde di e en simula o con igu a ions.
When one wan s o cha ac e ise op ical cons ella ion samples by he spa ial p obabili y dis ibu ion o
i s symbols mix u e dis ibu ions can be p o ed o be a good i . The p oblem ha we will a emp o
sol e on his da a h ough machine lea ning is dimensionali y educ ion, which alls in o he ield o unsu-
pe ised lea ning. To do so, he di e en machine lea ning models ha will be ained a e PCA and Deep
Au oencode s.
16
3. P ep ocess
In his chap e , i s , i is going o be discussed why i is necessa y o p ep ocess he op ical cons ella ions
be o e applying mos o dimensionali y educ ion models. Then, one p ep ocess app oach will be sugges ed
o be used in he incoming chap e s. Finally, om he sugges ed app oach, di e en amilies o models will
be compa ed acco ding o some easonable me ic.
3.1 The need o p ep ocess he da a
The samples o he da ase come in abula o ma wi h 2048 columns ha con ain he alues o he sym-
bols. The column in which a symbol is loca ed is andom since he signals a e demodula ed also in andom
o de . The e o e, he columns o he da ase lack o meaning and he samples a e jus an uno de ed se o
demodula ed symbols.
Machine lea ning models can only lea n om meaning ul ea u es so, i i is desi ed o ain machine
lea ning models wi h i , i is needed o do some so o da a enginee ing in o de o cha ac e ize he sam-
ples h ough ano he se o ea u es. This ea u e enginee ing s ep will be e e enced o as p ep ocess in
his documen .
3.2 Spa ial dis ibu ions
One can hink o a sample as a sample o size 2048 o a andom a iable ha ollows a pa ame ic dis i-
bu ion wi h alues in R2. The pa ame e s o he dis ibu ions can be es ima ed h ough he symbols o
he sample and hey can be used as a se o ea u es.
Two amilies o spa ial dis ibu ions will be p oposed, bo h o hem being mix u e dis ibu ions:
•Gaussian Mix u es
As we ha e seen in he backg ound chap e , he 2D poin s o he samples a e supposed o o
con o m 16 clus e s, each o hem cen e ed a one o he 16 cons ella ion poin s o he 16QAM op i-
cal cons ella ion. Tha is why i makes sense o i 2D Gaussian Mix u es wi h 16 componen s. I is
done h ough he Expec a ion Maximiza ion algo i hm and he means o he gaussians a e ini ialised
a hei heo ical loca ion: he loca ion o he cons ella ion poin s o he 16QAM op ical cons ella ion.
This way, we ob ain a mix u e
(x0,x1) =
15
X
k=0
πk Nk(x0,x1)
whe e Nk(
x)=2πDe (Σk)−1
2exp(−1
2(
x−µk)TΣ−1
k(
x−µk)) is he densi y unc ion o he compo-
nen ko he Gaussian Mix u e.
17
Ad anced Me hods o Op ical Cons ella ion Analysis and Comp ession
Figu e 9: Op ical cons ella ion diag am (le ) and a isualiza ion o i s co esponding Gaussian Mix u e
ea u es ( igh ).
•Mix u es o uni o m dis ibu ions
The mos na u al way o isualize he samples as humans is in a 2D-sca e plo . By doing ha ,
we ge id o he lack o o de in he ea u es o he da ase . This way o ge id o he lack o o de
o isualizing he da a can be used also o p ep ocessing he da a. Besides, his is a mo e gene al
app oach han Gaussian Mix u es since he e is no assump ion ha he demodula ion noise ha
makes a symbol no o be loca ed in he heo ical cons ella ion poin posi ion is gaussian. Taking
in o accoun ha he e has been a lo o esea ch in compu e ision models and ha a sample can
well ep esen ed by an image i makes sense o p ep ocess he samples in o images.
The way ha we will do so is by compu ing he 2D his og am o he samples wi h a de e mina e
numbe o bins ( he numbe o bins de e mines he esolu ion o he image). Hence, a p ep ocessed
sample would be a n binsx×n binsyimage whe e he alue o each pixel is he numbe o poin s o
he sample ha all in o ha poin o he disc e iza ion o he space. We di ide all he alues by 2048
( he o al numbe o poin s) so ha all he alues sum up o one and hey de ine a disc e e p obabili y
dis ibu ion whe e each alue is he p obabili y ha a sampled poin alls in he co esponding 2D
bin o he his og am. We ob ain a disc e e p obabili y dis ibu ion
p(x∈bink) = πk
whe e he MLE o {πk}kis hei empi ical alue
ˆπk=nk
2048
he p opo ion o he sampled poin s ha all in o he co esponding bin.
18
Figu e 10: Op ical cons ella ion diag am (le ) and i s co esponding 2D His og am ( igh ).
A his poin , when we al eady ha e a disc e e dis ibu ion, we need o u n i in o con inuous
since ha he na u e o he heo ical andom a iable. We can do so by assuming an uni o m
dis ibu ion o e he space ha encloses a bin o he 2D his og am. Tha leads o he mix u e
π(x0,x1) = X
k∈{bins}
πk
1(x0,x1)∈k
Ak
=πk∗
Ak∗
whe e Akis he a ea o he bin kand k∗is he bin in which (x0,x1) alls in o. We will be using
squa ed bins o he same size so ha hey ha e all he same a ea. The e o e
π(x0,x1) = 1
AX
k∈{bins}
πk1(x0,x1)∈k=πk∗
A
whe e Ais he a ea ha e e y bin has. Since he only pa ame e s a e {πk}k, i can be easily de i ed
ha MLE in his case is also
ˆπk=nk
2048
No e ha he e we a e p oposing one possible disc e iza ion o he 2D space o be applied equally
o all he da a samples. Ins ead, one could de ine an adap i e way o disc e ize he da a ha akes
in o accoun pa icula i ies o an inpu o decide in which way i is going o be disc e ized. Anyways,
he selec ed app oach can be as p ecise as one wan s i he disc e iza ion is e ined enough. Tha is
p o ed by he ollowing limi compu a ion:
lim
A→0 π(x0,x1) = lim
A→0
πk∗
A= +∞
since πk∗is bounded be ween 0 and 1 (and alue 0 can be excluded). The e o e, he limi o he
likelihood o a model gi en a sample is also in ini e:
lim
A→0L(π|
X) = lim
A→0Y
(x0,x1)∈
X
π(x0,x1) = Y
(x0,x1)∈
X
lim
A→0 π(x0,x1)=+∞
19
Ad anced Me hods o Op ical Cons ella ion Analysis and Comp ession
whe e a ge k e e s he alue o he o iginal mix u e o uni o ms in bin k(analogous o p edk)
and Ais he (cons an ) a ea o a bin. No e ha , o e e y bin, he i s e m o he p oduc is
2048 · a ge k=nk he numbe o poin s in bin kin he o iginal sample. The second e m is he
log densi y o he econs uc ed dis ibu ion in bin kand i is di ided by 2048 o apply he same
e-scaling as we did in he p ep ocess chap e . The 2048 ac o s cancel one wi h each o he and he
las o mula is ob ained.
One can expec his au oencode model o pe o m be e han he ones p e iously in oduced o
wo main easons: is ained in he same me ic in which i is going o be e alua ed and he ou pu
does ne e need any pos -p ocess o be ans o med in a alid dis ibu ion.
4.3 Resul s
All he p e iously men ioned dimensionali y educ ion models ha e been ained se ing he numbe o
dimensions o he la en space (bo leneck size) o di e en alues. Fo mo e de ails on he hype pa ame e s
o he models check he model hype pa ame e s sec ion o he only appendix in his documen . The
pe o mance o he comp esso s will be compa ed in e ms o hei goodness-o - i , aking he o iginal
Gaussian Mix u es as he baseline models.
E alua ion
The main goodness-o - i me ic will be he log likelihood o he econs uc ed spa ial dis ibu ions. The
size o he la en space is also going o be some hing o ake in o accoun by looking a he how much
we educe he dimension o he ea u e space wi h espec o he Gaussian Mix u e baseline. This will be
measu ed h ough he a io o he numbe o pa ame e s o Gaussian Mix u es di ided by he bo leneck
size. This a io will be e e ed o as comp essing ac o .
Resul s
Le ’s i s analyse he pe o mance o he PCA au oencode on he Gaussian Mix u e ea u es.
Figu e 13: Likelihood o he PCA econs uc ion o Gaussian Mix u es.
Figu e 13 displays he pe o mance o he PCA au oencode on he Gaussian Mix u e ea u es ( econsX
s ands o he econs uc ed dis ibu ions wi h bo leneck size X). No e ha he econs uc ion o he
26

small dis ance samples is ha de o he model in his case. I has o do wi h he low a iance gaussians
being close o be degene a e dis ibu ions [10]. No e also how he PCA is able o ge good econs uc ions
when inc easing he la en space numbe o dimensions. PCA on gaussian ea u es doesn’ seem o be
he go o app oach since one can expec o ge be e comp ession a es on small dis ance connec ion
cons ella ions.
Nex , le ’s discuss he esul s o he PCA au oencode s on he image da a.
Figu e 14: Likelihood o PCA econs uc ion o image da a.
In his case, as seen in Figu e 14, PCA econs uc ions o image da a a e gene ally less p ecise han he
o iginal Gaussian Mix u es. I is also ema kable ha inc easing he numbe o dimensions in he la en
space leads o wo se econs uc ions. I is no con adic o y since, as i has been commen ed p e iously,
PCA op imizes he MSE o he pa ame e econs uc ions ins ead o he log likelihood o he econs uc ed
dis ibu ions. This app oach gi es econs uc ed dis ibu ions ha a e clea ly wo se han he baseline Gaus-
sian Mix u es so i won’ be conside ed u he in he p ojec .
Mo ing in o he analysis o he pe o mance o Deep Au oencode s on 64x64 images, as seen in igu e
15 and compa ing he shown esul s wi h he p e iously discussed ones, he econs uc ed dis ibu ions
a e mo e obus in he sense ha he expec ed decay o he log likelihood wi h he inc ease o he o al
dis ance in he samples is obse ed on bo h ain and es se . Besides, he econs uc ions a e as good
as he o iginal Gaussian Mix u es o be e han hem (in e ms o log likelihood) o o al dis ance alues
highe han 700 also on bo h se s. This beha iou is obse ed o all he ied alues o he numbe o
dimensions o he la en space.
27
Ad anced Me hods o Op ical Cons ella ion Analysis and Comp ession
Figu e 15: Likelihood o Deep Au oencode s’ econs uc ions o 64x64 images.
Figu e 16: Likelihood o Deep Au oencode s’ econs uc ions o 128x128 images.
Nex , igu e 16 illus a es he ob ained esul s o Deep Au oencode s ained on 128 ×128 images whe e
a simila beha iou is obse ed o all he ied bo leneck sizes.
The deep au oencode app oach is also a obus dimensionali y educ ion me hod o 128 ×128 images
acco ding o he log likelihood o he econs uc ed dis ibu ions. Fu he mo e, he econs uc ions a e
be e han he o iginal Gaussian Mix u es o dis ances la ge han app oxima ely 500. Mo eo e , on he
samples wi h o al dis ance smalle han 500, he pe o mance is p e y simila o he one o he o iginal
Gaussian Mix u es. We see how he log likelihood gap ha we had on he p e ious igu e o hose samples
has been closed h ough inc easing he esolu ion o he images.
Figu e 17 shows he comp essing ac o o he e olu ion o he mean log likelihood ained models compu ed
on he es se as we inc ease he comp essing ac o . This igu e clea ly shows ha he Deep Lea ning
app oach (deno ed by he e m DL in he igu e legend) is he one ha gi es bes esul s, leading o e-
cons uc ions ha a e on a e age be e han he o iginal gaussian mix u es wi h high comp essing ac o .
A his poin o he p ojec , we can jus keep he deep au oencode s wi h bo leneck size 3 o 64x64 and
128x128 images which lead o a comp essing ac o o 31.67 wi h espec o he o iginal Gaussian Mix u es
and a ac o 1365.33 wi h espec o he aw op ical cons ella ion.
The e olu ion o he log likelihood wi h espec o he connec ion o al dis ance a e displayed in able 2
and igu e 18.
28
Figu e 17: Log likelihood o he selec ed Deep Au oencode s.
Dis ance in e al Gaussian Mix u e 64x64 Images 128x128 Images 64x64 econs 128x128 econs
0-500 -1.192 -1.253 -0.915 -1.385 -1.271
501-1000 -2.348 -2.149 -1.651 -2.362 -2.312
1001-1500 -2.855 -2.520 -1.871 -2.824 -2.802
1501-2000 -3.212 -2.758 -1.988 -3.149 -3.117
Table 2: Mean log likelihood o he econs uc ed dis ibu ions h ough he selec ed au oencode s.
Figu e 18: Log likelihood o he selec ed Deep Au oencode s.
29
Ad anced Me hods o Op ical Cons ella ion Analysis and Comp ession
The p oposed comp ession models achie e pe o mance me ics ha a e compa able o he ones o he
baseline Gaussian Mix u es, being hem be e o la ge dis ance samples and sligh ly wo se o small dis-
ance ones. As he image esolu ion is inc eased om 64 ×64 o 128 ×128, he log likelihood o he
econs uc ed small dis ance images ge s eally close o he Gaussian Mix u es’ ones.
Figu e 19: Th ee examples o o iginal images and hei au oencoded images.
I is also in e es ing o ha e a look on he isualiza ion o he econs uc ed dis ibu ions. Figu e 19 shows
h ee examples o o iginal images and econs uc ed dis ibu ions o bo h o he used image esolu ions.
The e a e a ious ema kable aspec s in his igu e:
•Au oencode s a e ained o de ec pa e ns in he op ical cons ella ions and emo e wha could be
andom noise in he econs uc ions. No e how he econs uc ed dis ibu ions concen a e mos
o he densi y in he 16 cen e s o he heo ical cons ella ion poin s. This also happens when one
(heu is ically) assumes a Gaussian Mix u e model.
•Fo la ge dis ance connec ions, we a e mo e likely o ob ain econs uc ed dis ibu ions ha di e
mo e om a Gaussian Mix u e pa e n, which migh no always be a alid assump ion. Th ough
image da a, we a e using a less- es ic i e model in he sense ha he dispe sion a ound he heo ical
cons ella ion poin s may be no gaussian-like. This is one o eason why his kind o modelling leads
o be e esul s o la ge dis ance connec ions.
•Small dis ance samples econs uc ions look p e y simila o bo h o he image esolu ions. Ne -
e heless, we know ha he 128 ×128 a e gene ally be e . Tha comes om he ac ha he
30
disc e iza ion done in 128 ×128 images allows o mo e p ecision and he ob ained densi ies h ough
econs uc ion can i be e he ac ual cons ella ions.
Conclusions
To sum hings up, he conclusions ha one can ake om he ob ained esul s a e he ollowing:
•The app oach ha seems o p o ide a mo e obus solu ion o he da a econs uc ion p oblem is he
use o Deep Au oencode s on image da a wi h a cus om loss unc ion ha makes he au oencode
o ain di ec ly o ain on he log likelihood o he econs uc ions.
•The Deep Au oencode s’ pe o mance on 128x128 image da a is simila o he o iginal Gaussian
Mix u es’. I is ema kable ha such pe o mance is ob ained wi h a comp essing ac o o 31.67 on
he numbe o pa ame e s wi h espec o he Gaussian Mix u es’ ones.
•On 64x64 images, Deep Au oencode s gi e simila bu sligh ly wo se esul s, being his di e ence
la ge he smalle he o al dis ance o he connec ion.
4.4 Summa y
When aiming o use echniques o educe he dimension da a one has o choose how o p ep ocess he
da a ha will eed he machine lea ning models.
Gaussian mix u es’ pa ame e s ha e cons ain s ha suppose a p oblem when one wan s o au oencode
hem since a gene ic decode won’ always gene a e se s o pa ame e s whe e he cons ain s a e ul illed.
Ne e heless, he cons ain s on he pa ame e s o he mix u e o uni o ms a e easie o deal wi h.
P incipal Componen Analysis (wi h a pos -p ocessing o he ou pu in o de o ul ill cons ain s) has
been e alua ed on da a econs uc ion ia log likelihood o he econs uc ed dis ibu ions and i doesn’
seem o pe o m p e y well nei he in gaussian no in uni o m mix u es. One has o ake in o accoun
ha PCA op imizes MSE o he econs uc ed pa ame e s ins ead o log likelihood o he econs uc ed
dis ibu ion.
On he o he hand, he deep lea ning app oach gi es mo e obus econs uc ions o he o iginal dis-
ibu ions. The ac ha deep au oencode s a e di ec ly ained o maximize he log likelihood o he
econs uc ed dis ibu ions ia a cus om loss unc ion is key o ob aining such good pe o mance. Mo e-
o e , he applica ion o non-linea models (unlike PCA) also helps o boos he pe o mance. This app oach
le us ge econs uc ed dis ibu ions ha a e compa able (o sligh ly be e ) o he baseline Gaussian while
using mo e han 31 imes less ea u es.
31

Ad anced Me hods o Op ical Cons ella ion Analysis and Comp ession
5. Use case: Da a augmen a ion
This chap e aims o p esen , in a p oo o concep manne , an use case o au oencode s on op ical con-
s ella ion da a. In pa icula , au oencode s will be used o da a augmen a ion [11].
Fi s , he mo i a ion and objec i es o his use case will be in oduced. Nex , he echnical app oach
will be p esen ed. Finally, he pe o mance o he models will be e alua ed and some conclusions will be
d awn om he esul s.
5.1 Mo i a ion
The compu a ion behind he gene a ion o he op ical cons ella ion simula ed da a equi es a la ge amoun
o compu a ional esou ces, which scales wi h he sampling a e o he signal p ocessing. In such scena io,
i is easonable o aim o ex ac all he po en ial alue om he da a.
One hing one can y o do is o use he simula ed da a o gene a e syn he ic da a. The p ac ise o
ex ending a da ase wi h syn he ic da a is also known as Da a Augmen a ion and ha can lead o boos s
in he pe o mance o o he ML-based applica ions.
Da a augmen a ion is a echnique o a i icially c ea e new aining da a om exis ing aining da a.
This is done by applying domain-speci ic echniques o examples om he aining da a ha c ea e new
and di e en aining examples.
In machine lea ning, gene ally, he highe he dimension o he ea u e space, he mo e aining exam-
ples a e needed. In cases o high dimension ea u e space, which is he case o ou image da a, da a
augmen a ion is usually applied in o de o ha e mo e compac aining se s in bo h he la en and he
o iginal inpu space.
5.2 Objec i es
Ideally one would like o one would like o be able o econs uc he wha a sample wi h 2048 symbols
would look like om ano he wi h, le ’s say, only 1024 symbols. Second ones a e simula ed way mo e
e icien ly.
Ins ead, i has been decided o sol e a di e en p oblem: econs uc ing he spa ial dis ibu ions (p e-
p ocessed samples) using samples ha only ha e hal o he symbols symbols (1024). This would allow o
ex end he o iginal da ase wi h wo kinds o syn he ic da a:
•Encoded andom subse s o 1024 symbols o he o iginal samples.
•Encoded simula ed samples ha only con ain 1024 symbols.
The objec i e is o de elop au oencode s ha a e able o gene a e syn he ic spa ial dis ibu ions in o de
o enla ge he size o he o iginal ain se .
32
5.3 App oach
Taking in o accoun ha he Deep Au oencode s ha whe e de eloped in he p e ious chap e showed
good pe o mance i makes sense o use he same a chi ec u e. Tha is going o be he app oach in his
chap e .
The ac ha we will be dealing wi h a neu al ne wo k a chi ec u e (which allows on-line aining [12])
gi es us he ollowing h ee di e en aining scena ios:
•Ze o-sho p e ained models: use he p e ained models om he p e ious chap e wi hou any
addi ional ine- uning o his pa icula ask.
•Fine- uned p e ained models: load he p e ained models om he p e ious chap e and ine- une
hem o his pa icula ask.
•T ained om ze o models: ini ialise he weigh s o he models andomly and ain hem only on his
pa icula ask
All hese aining scena ios will be e alua ed on he same es se .
5.4 Resul s
All he p e iously men ioned app oaches ha e been ained o 64 ×64 and 128 ×128 images. The la en
space numbe o dimensions will be se o 8 o 64 ×64 images and 13 o 128 ×128 images since he las
chap e ’s esul s sugges i is a good op ion.
Thei pe o mance is going o be e alua ed acco ding o di e en c i e ia.
E alua ion
The es se o his ask consis s o , o each op ical cons ella ion o he o iginal es se , he sample plus
a se o 50 op ical cons ella ions wi h only 1024 symbols ha a e andom subse s o he symbols in he
o iginal sample.
The pe o mance o he Da a Augmen a ion au oencode s will be e alua ed on he log likelihood o he
econs uc ed spa ial dis ibu ions om he samples wi h only 1024 symbols gi en he o iginal es op ical
cons ella ion. In his case, ha ing likely enough econs uc ions would mean ha we gene ally only need
hal o he symbols o in e he spa ial dis ibu ion o he comple e op ical cons ella ion.
Resul s
Figu e 20 compa es he h ee aining scena ios ha we e in oduced p e iously o he i ed au oencode s.
No ice how he ze o-sho lea ning au oencode pe o ms p e y well bu he o he ones ha e sligh ly highe
log likelihood among he es se examples. I is decided o jus keep he p e ained and ine- uned au-
oencode s.
33
Ad anced Me hods o Op ical Cons ella ion Analysis and Comp ession
Figu e 20: Log likelihood o au oencode s o da a augmen a ion on di e en aining scena ios.
Figu e 21 displays he log likelihood o he econs uc ed masked images wi h espec o he o iginal dis-
ibu ions. The main conclusion one can ake om hese esul s is ha bo h image esolu ions lead o
econs uc ed dis ibu ion o he masked samples ha a e app oxima ely as likely as he o iginal Gaussian
Mix u e models ha we e i ed o he comple e samples. I is also ema kable ha , when wo king wi h
128x128 images, he esul s a e sligh ly be e o samples ha come om low-dis ance connec ions.
Figu e 21: Log likelihood o he da a augmen a ion au oencode s compa ed o he o iginal spa ial dis i-
bu ions’.
The espec i e nume ic esul s a e shown in able 3, whe e one can also no ice ha he ob ained log like-
lihood alues a e also simila o he ones o he econs uc ed dis ibu ions om he non-masked images.
Indeed, he econs uc ed dis ibu ions om he masked images can e en be equal o he econs uc ed
dis ibu ions om he o iginal images o a egula human eye and igu e 22 illus a es so.
34
Dis ance Gaussian Mix . 64x64 econs 128x128 econs masked 64x64 econs masked 128x128 econs
0-500 -1.192 -1.385 -1.271 -1.383 -1.281
501-1000 -2.348 -2.362 -2.312 -2.355 -2.330
1001-1500 -2.855 -2.824 -2.802 -2.821 -2.813
1501-2000 -3.212 -3.149 -3.117 -3.143 -3.144
Table 3: Log likelihood o he econs uc ed masked images h ough he selec ed au oencode s, compa ed
wi h he non-masked econs uc ions.
Figu e 22: Th ee example images and econs uc ions wi h hei espec i e masked images and econs uc-
ions.
Conclusions
A e ha ing a look a all he ob ained esul s, hese a e he main conclusions ha can be d awn om
hem:
•I is no necessa y o simula e 2048 symbols samples in o de o ob ain likely spa ial dis ibu ions o
he op ical cons ella ions. The p oposed au oencode -based app oach gene a es spa ial dis ibu ions
h ough samples ha ha e only hal o he symbols and he ob ained esul s on he es se show
ha hey a e compa able o he o iginal Gaussian Mix u es in e ms o likelihood o he p obabilis ic
models.
•Once again, now being in he masked op ical cons ella ions scena io, highe esolu ion images
35