1
1
Maximilian Oli e Rubin
Design explo a ions o in e ac i e i ual
en i onmen s using olume ic cap u e
anD isualisa ion echniques
A 250 yeA imes Amp o he bo AnicAl gA den o
he uni e si y o coimb A
Thesis in he con ex o he Mas e ’s in Design and Mul imedia, ad ised
by P o esso s Jo ge Ca los dos San os Ca doso and Ped o Filipe Ma ins
Ca alho and p esen ed o he Depa men o In o ma ics Enginee ing o he
Facul y o Sciences and Technology o he
Uni e si y o Coimb a.
Sep embe 2022
Kai os
A 250 yea imes amp o he Bo anical Ga den o he Uni e si y o Coimb a
Maximilian Oli e Rubin
Ad ised by P o esso s Jo ge Ca los dos San os Ca doso
and Ped o Filipe Ma ins Ca alho

Abs ac
Cap u ing en i onmen s and momen s in ime has always
been some hing humans ha e been ascina ed by – om
pain ings o analogue pho og aphy, o digi al came as. Wi h
he ecen de elopmen s o AR and VR echnologies, di ec
olume ic cap u e o eali y has been gaining ac ion, wi h
he po en ial o c ea ing imme si e, s a e-o - he-a isuals,
blu ing he lines be ween he eal and i ual wo lds.
Cap u ing and ansmi ing space i sel is one o he leading
challenges in he ield o imme si e media. Cap u ing a eas
such as si es o cul u al he i age is pa o his equa ion and
has gene ally been achie ed ei he by 360-deg ee ideo o
pho og aphy. Howe e , hese me hods limi he use ’s ull
sense o dep h o he image.
A solu ion o o e come his limi a ion consis s o cap u ing
eal-wo ld da a h ough eali y cap u e me hods, esul ing
in a se o da a poin s in space called a poin cloud. This
poin cloud hen passes h ough a econs uc ion p ocess,
ans o ming i in o a 3D polygonal mesh. Howe e ,
his p ocess limi s i s abili y o p o ide a deepe sense o
imme sion and can in oduce undesi able in e pola ions o
eali y. None heless, by using poin clouds as he basis o an
imme si e en i onmen , we can explo e new isual languages
and al e na e modes o pe cep ion.
The objec i e o his hesis is he c ea ion o a poin cloud-
based digi al imes amp o he Bo anical Ga den o he
Uni e si y o Coimb a, by cap u ing eali y and hus a
momen in ime ha can be un and isualised in 3D
space ( h ough Vi ual Reali y), in eal- ime. This digi al
imes amp consis s o h ee dis inc loca ions wi hin he
Ga den, con aining cen u y-old ee specimens. These a eas
we e digi ised h ough pho og amme y o c ea e poin
clouds, which we e hen used as he basis o an imme si e
audio- isual expe ience whose main goal was o con ey he
ambiance o he Ga den, along wi h showing people i s na u al
p ocesses usually hidden o he naked eye wi h he aim o
ekindle ou ela ionship wi h na u e.
This hesis esul ed in mul imedia con en consis ing o PC
and VR based expe iences, and 360 deg ee ideos. Bo h PC
and VR a ian s we e e alua ed by 22 olun ee s, esul ing in
a posi i e esponse dis ibu ion o he use ’s expe ience in
ela ion o hei ela ionship wi h he na u al wo ld.
Keywo ds: Vi ual Reali y, Da a Cap u e, Poin Clouds,
Pho og amme y, Imme si e Media.
Resumo
Cap u a ambien es e momen os oi algo que semp e ascinou
o homem – desde pin u as, a o og a ia analógica, a câma as
digi ais. Com os ecen es a anços em AR e VR, a cap u a
olumé ica da ealidade em ganho acção, endo o po encial
pa a c ia ambien es isuais ime si os, de ú ima ge ação,
esba endo as linhas en e o mundo eal e o i ual.
Cap u a e ansmi i o p óp io espaço é um dos p incipais
desa ios na á ea dos meios ime si os. Nes e sen ido, a cap u a
de á eas do pa imónio cul u al é um obje i o eco en e e
em sido ge almen e ealizada que po ídeo de 360 g aus,
que po o og a ia pano âmica. No en an o, es es mé odos
na u almen e limi am a sensação o al de p o undidade po
pa e do u ilizado .
Uma solução pa a ul apassa es a limi ação consis e em
cap u a dados do mundo eal a a és de mé odos de cap u a
olumé ica, esul ando num conjun o de pon os de dados
no espaço ambém conhecido po “nu em de pon os”. Es a
nu em de pon os passa po um p ocesso de econs ução,
ans o mando-a numa malha poligonal 3D. Con udo, es e
p ocesso limi a a sua capacidade de p opo ciona um sen ido
mais p o undo de ime são. No en an o, ao u iliza nu ens
de pon os como base de um ambien e ime si o, podemos
explo a no as linguagens isuais e modos al e na i os de
pe cepção.
O objec i o des a ese é a c iação de um imes amp digi al
do Ja dim Bo ânico da Uni e sidade de Coimb a, a a és
da cap ação da ealidade e, po an o, de um momen o no
empo que pode se execu ado e isualizado em espaço
3D (a a és da Realidade Vi ual), em empo eal. Es e
imes amp digi al consis e em ês locais dis in os do Ja dim
con endo espécimes de á o es cen ená ias. Es as á eas o am
digi alizadas a a és de écnicas de cap u a olumé ica
pa a c ia nu ens de pon os, que o am u ilizadas como base
de uma expe iência audio- isual ime si a cujo p incipal
objec i o é cap a o ambien e do Ja dim jun amen e com
mos a às pessoas os seus p ocessos na u ais no malmen e
escondidos a olho nu com a inalidade de ea i a a nossa
elação com a na u eza. Es a ese esul ou em con eúdo
mul imédia cons i uído po uma expe iência baseada
em PC e VR, e ídeos de 360 g aus. As a ian es PC e VR
o am a aliadas po 22 olun á os, esul ando numa
dis ibuição de espos a posi i a pa a as expe iências do
u ilizado em elação à sua elação com o mundo na u al.
Pala as-cha e: Realidade Vi ual, Cap u a de Dados,
Nu ens de Pon os, Fo og ame ia, Media ime si a.

1. In oduc ion
2
The essence o obse ing, desc ibing and emembe ing
in o ma ion and pe sonal expe iences has been wi h us o
as long as we can emembe , and has been he e y basis o
ou su i al as a species. A p ima y ins ance o his can be
seen in he p ehis o ic pain ings and eng a ings on walls o
ca es and ocks, which po ay he impo ance o ou human
need o communica e [1]. This o m o communica ion
le pe manen ma kings o he u u e, lea ing impo an
messages o he gene a ions o come.
To his day, he basis o e ec i e communica ion in he o m
o isual p omp s hasn’ changed. We ha e only go be e
a i and a e con inuously e ining and op imising he ways
we sha e in o ma ion. Now when we hink abou how a
we ha e come in ou abili y o exp ess ou hough s and
ideas wi h he help o VR (Vi ual Reali y) echnology, we
ha e almos eached he ipping poin o limi less pe sonal
exp ession and o making he imagina ion explici .
The desi e o show o ano he human being o o an audience
he eal s a e o one’s imagina ion, pe spec i e and ideas
is qui e an allu ing concep . Tu ning ideas, w i en and
spoken wo ds in o some hing we can see, can be conside ed
he pinnacle o pe sonal exp ession and ansmission o
in o ma ion. The same can be said when someone ells us
ha we “speak clea ly”, i ’s a isual me apho : hey “see”
wha you mean. In o he wo ds, i would be like a new kind o
communica ion, a o m o sha ing in o ma ion ha is simila
o elepa hy whe e hough s and ideas can be sha ed isually.
Technologies and p ojec s based on VR a e heading in his
di ec ion and he ecen hype su ounding he Me a e se
[2] is jus one o he many examples o p ojec s which will
adically change he way we communica e and pe cei e
eali y as we know i . Because o his, he e has been an in lux
o new me hods and imp o ed accessibili y when i comes o
cap u ing eali y o acili a e he eme gence o his upcoming
digi al ealm [3][4][5].
Cheape cap u ing de ices, imp o ed da a p ocessing
me hods and ad ancemen s in ha dwa e pe o mance
(especially when i comes o GPUs (G aphics P ocessing
Uni s)) a e allowing his ield o h i e, making i a ac i e
o digi al a is s and con en c ea o s. Poin clouds a e he
ounda ion o he p ocess o cap u ing eali y. A hei co e,
poin clouds a e essen ially iles ha con ain a mul i ude o
da a poin s in 3D space ha a e c ea ed by pe o ming a scan
o an objec o a loca ion.
3
The mos popula o ms o a aining hese scans is by using
ei he a LiDAR (Ligh De ec ion and Ranging) de ice, o
pho og amme y ( he p ocess o ex ac ing 3D da a om
digi al pho og aphs). Howe e , challenges s ill emain when
i comes o ans o ming poin cloud da a, whe he i be
om a lase scanne o om pho og amme ic cap u e, in o
ealis ic digi al models. The con e sion o he measu ed
da a in o a high quali y 3D polygonal su ace model is
s ill a esou ce-hea y ask. To make ma e s wo se, poin
clouds o en consis o incomple e, noisy and/o spa se da a,
especially when cap u ing la ge ou doo scenes, which in
u n u he nega i ely in luence he quali y o he inal
model [6].
A small numbe o a is s in he isual e ec s and i ual
imme si e ields ha e ecen ly app oached his challenge
in a di e en way. They s a ed explo ing poin clouds in
hei c ude o ma , aking ad an age o hei noisy na u e
o disco e new aes he ics and new isual languages. Wi h
his app oach, hey a e able o ake us on emo i e jou neys
h ough digi al agmen s o eali y. Mos o hese p ojec s
consis o p e- ende ed ideos while o he s consis o eal-
ime imme si e VR expe iences. Howe e , he la e a e
qui e demanding, ha dwa e-wise, usually ake place in sho
li ed exhibi ions and ypically ollow a p ede e mined ou e
wi hin hei i ual wo lds, hus limi ing he use s’ abili y o
ull con ol wi hin he i ual se ing.
This p ojec ’s main goal was he c ea ion o an imme si e
isual ep esen a ion o a memo y o he Bo anical Ga den
o he Uni e si y o Coimb a [7], by cap u ing eali y and
hus a momen in ime ha can be un and isualised in
3D space ( h ough VR), in eal- ime. In addi ion o se ing
as a digi al backup o he Ga den and commemo a ing i s
250 h anni e sa y, his p ojec allows anyone o access and
expe ience his loca ion emo ely, e en in ace o es ic ions
such as ones imposed by pandemics o physical limi a ions,
e ec i ely allowing unlimi ed isi s o his si e o cul u al
he i age [8].
This p ojec ocused on h ee dis inc a eas wi hin he
Bo anical Ga den, each o which highligh s memo able
cen ena y-old ee specimens: he Tilia x eu opaea [9], he
E y h ina C is a-Galli [10], and he Ficus Mac ophylla [11].
He e we also simula ed some o he a ea’s na u al p ocesses
usually hidden o he naked eye, wi h he aim o e oking a
sense o inquisi i eness wi hin ou use s.
4
These na u al p ocesses we e simula ed o e ou poin clouds
and consis o he a eas lo a and auna, including he ee’s
ascula sys ems and unde g ound oo ne wo ks, ollowing
he p emise ha i we can obse e he p ocesses o na u e
in VR, we may become mo e conscious o i s inne wo kings
in eal li e, and hus gain new le els o app ecia ion o he
na u al wo ld. These h ee a eas a e i ually in e connec ed,
simula ing snippe s o a memo y s oll h ough he Ga den,
allowing use s o eely a el om one memo y o he nex .
1.1 Why poin clouds?
One o he main conce ns behind VR is he challenge o
de elop in e ac i e and imme si e wo lds ha eel as close
o eali y as possible. Poin clouds a e ypically con e ed
in o 3D polygonal meshes in an e o o c ea e a close
eplica o wha was cap u ed, bu his some imes esul s
in a “so edged” appea ance which does no appeal o he
eye, especially a close p oximi y [6]. The ac o he ma e
is, poin clouds a e he basis o a cap u ed su ace and
hus can be conside ed he ab ic o ( i ual) eali y i sel ,
me apho ically speaking.
By isualising poin clouds in hei c ude and di used
s a e, ou ocus ge s aken away om he isual elemen ’s
mass, whils main aining a su icien le el o su ace de ail
o wha was cap u ed o ecogni ion, allowing ou mind
o ocus mo e on he gene al a mosphe e and one o he
locale. This opens new po en ials o explo e al e na e
modes o pe cep ion and new isual languages. So a he
han using compu e algo i hms o ill in missing su ace
in o ma ion wi h polygons, which a e me ely an in e pola ed
in e p e a ion o eali y, we can ake ad an age o ou b ain’s
inna e abili y o ac i ely in e p e ou pe cei ed eali y by
“ illing in he gaps”, ab ica ing he illusion o eali y, and
hus allowing o a deepe , mo e au hen ic connec ion
be ween he use and he i ual wo ld [12][13]. In o he
wo ds, e en i some poin cloud da a is missing, i s di used
and agmen ed cha ac e s ill has he capaci y o p o ide
a g ea e sense o p esence and au hen ici y compa ed o
3D polygon meshes. Mo eo e , his o m o isualising
da a opens new ga es o explo e he way ou memo ies and
d eams wo k, emula ing he way ou subconscious mind
s o es and eplays da a. Simila o poin clouds, d eams end
o be qui e incomple e, uzzy and some imes unclea , ye
we o en seem o emembe he gene al eelings hey lea e.
Tha ’s he eal goal he e, o be able o ec ea e he ambiance o
a momen los in ime.
5
1.2 Mo i a ion
Se e al ac o s in luenced he de elopmen o his p ojec .
Fi s , in addi ion o se ing as a digi al backup o he
Ga den and commemo a ing i s 250 h anni e sa y, his
p ojec allows anyone o access and expe ience his loca ion
emo ely, allowing unlimi ed isi s o his si e o cul u al
he i age.
Second, as a echnology-d i en species, we ha e been slowly
losing ouch wi h na u e and ha e become mo e dis an o
i s li e-gi ing bene i s [14]. This p ojec aims o igge ou
inna e “biophilia” [15] (a e m used o desc ibe ou a ilia ion
wi h mo he na u e, i s posi i e impac on ou men al heal h
and ou o e all well-being), by ekindling ou connec ion
o he na u al wo ld, no as a ool o subs i u e na u e and
eali y i sel , bu a he as a ool o enhance i .
Thi d, by u ilising and explo ing he po en ial o he la es
eali y cap u e and poin cloud isualisa ion echniques,
his p ojec seeks o e o lessly ansmi he eelings o a
momen cap u ed in ime h ough con empo a y echnology.
Simila o a pho og aph, bu in his case a ype o pho og aph
we can “jump in o”, so o speak. Allowing us o explo e he
in icacies o an al eady geome ically complex scene y.
Las ly, he conse a ion o he Bo anical Ga den is some hing
ha he Uni e si y o Coimb a is wo king on and his
has become a con inuous endea ou . Belonging o one o
UNESCO’s Wo ld He i age Si es [16], i is pa amoun ha he
Bo anical Ga den p ese es i s in eg i y and essence. Na u al
disas e s such as s o ms and he ini e li e-span o plan
species a e ine i able and ha e un o una ely aken ou some
his o ical ees and caused s uc u al damage o e he yea s.
Fo ins ance, he iconic Washing onia obus a palm ee, which
a he ime was he alles plan in he Ga den, go obli e a ed
by a ligh ning s ike in May o 1986 [17], no o men ion he
ecen hunde s o m “Leslie” in 2018, which caused a lo o
s uc u al damage o he Ga den [18] ( igu es 1, 3 and 4).
Fu he mo e, he e a e also signs o delibe a e ac s o
degene acy by he public, such as ca ings in he neoclassical
Cen al Squa e ( igu e 2). Al hough hese examples don’ all
belong o he h ee chosen a eas o cap u e, hese ins ances
a e jus o demons a e how exposed he Ga den is o
ex e nal o ces. Tha being he case, his p ojec will also
unc ion as a digi al imes amp, a so o backup, by digi ally
p ese ing he a eas wi hin he Ga den, wi h emphasis on
one o i s oldes ees: he E y h ina C is a-Galli, which is
cu en ly in a s uc u ally ulne able s a e ( igu e 5).

6
Figu e 1.
Figu e 3.
Figu e 2.
Figu e 4.
Figu e 5.
Figu e 1: A missing Tilia ee in Alameda das Tílias, 05/11/21; Figu e 2: Ca ings
in he Cen al Squa e ma ble, 05/11/21; Figu e 3: Polypo e mush ooms g owing
on he base o a che y ee in Cen al Squa e, indica ing unk o , 05/11/21;
Figu e 4: Damage o he ence’s neoclassical o namen ; Figu e 5: Cu en
s a e o he E y h ina C is a-Galli, 05/06/22.
7
1.3 Goals and Resul s
This p ojec ’s main goal consis ed o c ea ing a digi al
ep esen a ion o he Bo anical Ga den o he Uni e si y o
Coimb a using poin cloud-based cap u e echniques, by
cap u ing eali y and hus a momen in ime ha could hen
be un and isualised in 3D space, in eal- ime. Fu he mo e,
by wo king in a digi al en i onmen , all cons ain s p esen
in he eal wo ld we e unleashed, meaning ha he sky was
he limi when i came o how we could manipula e and
p esen hese da a. This lead o ano he goal which consis ed
o enhancing he sensa ion o being p esen in he Ga den,
along wi h e oking eelings o childlike wonde by senso y
s imula ion ia he isual and audi o y sys ems. Mo eo e ,
scien i ic in o ma ion on he cha ac e is ics o each a ea
we e ga he ed o he end goal o simula ing some o hei
na u al p ocesses. To u he inc ease i s imme si e ac o , a
VR compa ible e sion o his p ojec was also p oduced. Las
bu no leas , his wo k esul ed in a publica ion accep ed by
he In e na ional Con e ence on En e ainmen Compu ing,
in he ca ego y o Wo ks-in-P og ess pape s, ha will be
p esen ed in No embe 2022 in B emen, Ge many. This
submission will also be conside ed o he con e ence’s
S uden Compe i ion.
This imme si e expe ience o he Bo anical Ga den was
he end p oduc o his hesis, which was made publicly
a ailable o PC, a VR headse , along wi h 360-deg ee ideos
o be iewed wi hin a VR headse o h ough online ideo
s eaming pla o ms. Albei no ha ing he same imme si e
po en ial as he VR headse a ian , he 360-deg ee ideos
we e c ea ed o imp o e audience each. Rega ding he VR
a ian , a a he op imis ic ye easible goal was explo ed,
which consis ed o unning he imme si e expe ience
wi hin he Bo anical Ga den, ha is, he c ea ion o an in-
si u ins alla ion, whe e ou audience could u he indulge
hemsel es in he Ga den.
Fi s and o emos , o be able o achie e hese goals, a ew
s eps we e aken in o de o b ing he mos ou o his
expe ience and o enhance he sensa ion o being p esen
in he Ga den. These s eps consis ed o cap u ing as much
da a as possible in o de o s imula e as many senses as
possible wi hin he digi al en i onmen . In his case, hese
encompassed wo main a eas: he isual and audi o y. To
be able o acqui e hese da a, pho og amme ic cap u e was
used o eco d he isuals whils a se o mic ophones we e
used o audio cap u e o he goal o c ea ing a 3D ambien
sound e ec .
8
These acqui ed da a we e hen cleaned, p ocessed and
e ined o he inal digi al en i onmen . These da a we e
hen composed wi hin Uni y’s eal- ime game engine, whe e
u he isual explo a ion was done o see how hese da a
could be p esen ed and enhanced o c ea e a cap i a ing and
imme si e d eam-like expe ience.
Nex , in o de o gua an ee a quali y expe ience, pe o mance
and s abili y es s we e un o ensu e ha no issues occu ed
such as low ame a es o isual anomalies, as his would
ha e nega i ely impac ed ou desi ed ou come. As a inal
poin , a se o use e alua ion es s we e un o de e mine i
ou imme si e expe ience ul illed ou goals.
1.4 Scope
Na u ally, some bounda ies we e se in o de o achie e he
bes possible ou come wi h he limi ed esou ces a ailable.
This sec ion p esen s some conside a ions on he p ojec ’s
scope.
A pa o his p ojec was dedica ed o emula e he way we
expe ience d eam-like s a es and memo ies h ough he
exp essi e isual po en ial o poin clouds. Howe e , we only
sc a ched he su ace o his ma e and did no del e in o he
ine de ails o Onei ology ( he scien i ic s udy o d eams),
Neu oscience and/o Cogni i e Psychology.
Al hough ou ini ial plan was o cap u e mul iple loca ions
wi hin he Bo anical Ga den, he shee size o he Ga den
had o be aken in o accoun due o ime cons ain s and
echnical limi a ions. F om i s hi een hec a es o g ound,
h ee dis inc a eas o in e es we e selec ed con aining he
ollowing cen u y-old ees: he Tilia x eu opaea, he E y h ina
C is a-Galli, and he Ficus Mac ophylla.
Las ly, due o he wide- anging ield o da a cap u e and
sensing echnologies, and o he sake o his hesis, we
mos ly discussed pho og amme y, as i is he basis o he
p ac ical componen o his hesis, and did no add ess he
deepe scien i ic and ma hema ical componen s o emo e
sensing.
9
1.5 Documen S uc u e
This documen is s uc u ed as ollows: In oduc ion,
In oduc ion o Cap u ing Reali y, Rela ed Wo k, Me hodology
and Wo k Plan, P ojec De elopmen , E alua ion and las ly,
Conclusion.
In he i s chap e , an in oduc ion o he ield o s udy and
mo i a ions behind he hesis a e p esen ed, as well as i s
goals, objec i es and scope.
The second chap e , In oduc ion o Cap u ing Reali y, se es
o p o ide a deepe unde s anding o he ield o cap u ing
eal-wo ld da a by laying he ounda ions o his ma e .
This includes i s his o ical backg ound, i s me hodologies,
cu en challenges and limi a ions, ollowed by he la es
con empo a y me hods and de ices used in his ield.
The hi d chap e p esen s ela ed wo k and p ojec s o
p o ide a be e unde s anding o wha has al eady been
achie ed in he ield o using poin cloud da a in he con ex
o imme si e expe iences. In his chap e , each case s udy
is analysed o i s wo k low, mo i a ions and inspi a ion,
highligh ing pe inen in o ma ion ha aided he
de elopmen o he p ac ical wo k ea u ed in his hesis.
The ou h chap e con ains he Me hodology and Wo k
Plan, which de ined he s uc u e o he de elopmen
o he p ac ical componen o his hesis, ollowed by a
p ojec ime-line. This chap e se ed o gua an ee a s able
sys ema ic wo k low h oughou he de elopmen o his
hesis, including he me hods and p ocedu es ha we e used
o bes achie e ou desi ed goals and objec i es.
The i h, six h and se en h chap e s encompass all de ails
ega ding he de elopmen o he p ac ical componen s o
his hesis. This includes a small in oduc ion o he si e o
cap u e: he Bo anical Ga den o he Uni e si y o Coimb a,
ollowed by a sec ion o p elimina y expe imen s which
we e conduc ed o ge accus omed o he pho og amme ic
p ocessing pipeline. Nex , we ho oughly desc ibe he
de elopmen o Kai os, including all o i s a ian s, oge he
wi h all he s eps ha we e aken o achie e he inal
imme si e expe ience.
Chap e eigh , E alua ion, pe ains o he es ing phase o ou
p ojec whe e we ga he ed and analysed use eedback o ou
imme si e expe ience.
16
g ea conce n. Apple ecen ly in oduced LiDAR scanning
capabili ies in i s se ies o mobile phones and some able s,
hough no as accu a e as p o essional LiDAR de ices, hese
s ill open up many in e es ing applica ions o hand-held
scanning and a e g ea ly imp o ing audience each and
ecogni ion [31].
Figu e 10: Top Le : An illus a ion o a e es ial lase scanning sys em (TLS) in he con ex
o a o es . A ailable a : «h ps://a s els-cdn com/con en /image/1-s2 0-S0924271616000204-
g 1_l g jpg». Top Righ : Pho og aph o a well known e es ial lase scanne – he Leica P20
(manu ac u ed in 2014). A ailable a : «h ps://www. esea chga e.ne /p o ile/Jim-Chandle -2/
publica ion/337849886/ igu e/ ig1/AS:836700154654724@1576496270365/Leica-P20- e es ial-
lase -scanne -wi h-1-HDS- a ge -used- o - egis a ion-and png». Bo om: An example o a poin
cloud esul ing om a TLS scan o a public pa k. A ailable a : «h ps://i pinimg com/o iginals/
de/18/b /de18b 1c9b78e4b52981165b47796 3c jpg»

17
2.3 Pho og amme y
Pho og amme y (Pho os = ligh ; G amma = w i ing;
Me on = measu e) is de ined by he Ame ican Socie y o
Pho og amme y and Remo e Sensing as, “The a , science
and echnology o ob aining eliable in o ma ion abou
physical objec s and he en i onmen h ough p ocesses o
eco ding, measu ing and in e p e ing pho og aphic images
and pa e ns o elec omagne ic adian ene gy and o he
phenomena” [32].
To ecapi ula e and o ully quan i y he p og ession o
pho og amme y h oughou his o y, his ield o s udy can
be sepa a ed in o ou gene al s ages based on imp o emen s
in echnology.
The i s gene a ion o pho og amme y s a ed wi h he
in en ion o pho og aphy, which ini ia ed he in es iga ion
and expe imen a ion phase o emo e sensing.
The second gene a ion, conside ed he e a o analogue
pho og amme y, b ough ae ial su eying and s e eo
plo ing echniques ha we e es ablished and used du ing
he wo Wo ld Wa s, u ning pho og amme y in o a
epu able and popula su eying and mapping me hod. The
ma hema ical heo y was well known a his s age, bu he
lack o compu a ional powe hinde ed nume ical solu ions.
The hi d gene a ion, also known as he analy ical
pho og amme y gene a ion, began in he 1950’s du ing
he up ising o he compu e , which allowed o signi ican
ad ances o pho og amme ic de elopmen . Hellmu
H. Schmid was one o he i s pho og amme is s who
ook ad an age o he compu e , de eloping he basis o
analy ical pho og amme y using ma ix algeb a. Block
adjus men p og ams (a se o ools designed o imp o e
he geo e e encing accu acy) and ae ial iangula ion ( he
p ocess o de e mining he co ec posi ion and o ien a ion
o each image in a se ies o ae ial images) we e subsequen ly
in oduced in he la e six ies ha g ea ly imp o ed ae ial
iangula ion accu acy.
We a e now in he ou h gene a ion o pho og amme y.
This began wi h he ansi ion om analogue o digi al,
whe e digi al images a e used ins ead o ae ial pho og aphs.
Pho og amme ic p ocesses a e now managed by compu e
so wa e, p o iding huge cos and ime sa ings in p oducing
ypical pho og amme ic ou pu s [33][34].
18
2.3.1 The Pho og amme ic P ocessing Pipeline
The wide a ailabili y o digi al came as and he ecen
p og ess in compu e ision has led o he ise o accessible
3D modelling solu ions. Pho og amme ic cap u e
na u ally s a s wi h he cap u e o eal wo ld da a h ough
pho og aphs o ideo (as an image sequence). This cap u ed
da a hen ge s ed in o he pho og amme ic p ocessing
pipeline ( igu e 11), consis ing o a ew s ages ha need o be
ollowed in o de o p oduce a success ul digi al asse . This
pipeline gene ally consis s o he ollowing s ages: came a
alignmen , gene a ing dense cloud, build mesh, and build
ex u e.
Du ing he ini ial s age, came a alignmen , he
pho og amme y so wa e akes as inpu a se o images
o ideo ames and hei associa ed came a calib a ion
in o ma ion (i a ailable). I hen scans h ough e e y ame
and sea ches o ea u es. Fea u e de ec ion de e mines
salien poin s ( ea u es) in he images o ideo ames. These
ypically consis o dis inc i e a ibu es such as pa e ns,
s ong lines and edge poin s. These in u n help es ima e
he ollowing posi ioning o each came a ( ame) wi hin a
common coo dina e sys em which is ela ed o he wo ld
coo dina e sys em. I is usually desi able o eliably de ec
hese poin s a di e en iewpoin s and adial dis ances o
ensu e ha he same salien ea u es a e de ec ed in di e en
ames. A e he ea u e poin s ha e been ecognized in
se e al images, hey a e in e nally alida ed and sc eened o
alse co espondences in a s ep called geome ic alida ion.
The alida ed ea u es a e hen co ela ed wi hin e e y ame
o ge pu a i e ma ches which a e hen used o es ima e
he posi ion o e e y came a ame wi hin he scene [21]. As
a esul , a spa se poin cloud and a se o came a posi ions
a e o med [35]. No e ha his spa se poin cloud is me ely
a e e ence o he esul ing came a alignmen s and only a
ac ion o he o al da a. In o he wo ds, a “ alida ion s ep”
be o e s a ing any hea y du y compu a ional wo k.
Du ing he second phase, gene a ing dense cloud, 3D
coo dina es o he su ace poin s a e es ima ed based on he
ou pu o he p e ious phase. The line o sigh om e e y
came a o he objec is econs uc ed in o wha is called a ay
cloud. The in e sec ion o many ays de e mines he inal 3D
coo dina es o he objec [36]. Once his has been comple ed,
a dense poin cloud is gene a ed whe e each ma chable pixel
ge s i s own x, y, z coo dina es in 3D space [37] ( igu e 12).
The hi d phase, build mesh, consis s o gene a ing he su ace
o he scene o objec based on he p e iously p oduced dense
19
poin cloud. This s ep connec s each se o h ee adjacen
poin s in o a iangula ace, esul ing in a con inuous mesh
o e he su ace o he model.
In he inal s ep, build ex u e, o iginal images a e combined
in o a ex u e map and p ojec ed on o he mesh, esul ing
in a pho o- ealis ic digi al ep esen a ion o he o iginally
cap u ed objec .
Figu e 11: Le : The basis
o he pho og amme ic
pipeline. Righ : The
p incipal p ocedu es o
pho og amme y. The do ed
line indica es he connec ion
be ween image ea u es Pn,m,
ex e nal image o ien a ions
Rn, n, and he ie poin P1.
A ailable a : «h ps://www
esea chga e.ne / igu e/
The-p incipal-p ocedu es-o -
close- ange-pho og amme y-
The-do ed-line-indica es-
he_ ig2_347828301»
Figu e 12: Pho og amme ic cap u e o he Th ee G aces by An onio
Cano a, illus a ing he esul ing dense poin cloud and came a alignmen s.
A ailable a : «h ps://www ac um-a e com/ esou ces/images/ a/ echnology/
pho g amme y/cap u e3 jpg»
20
2.4 Challenges and Cu en Limi a ions
Pho og amme y
While many imp o emen s ha e been made in he ield
o pho og amme y o e he yea s, some hu dles s ill
pe sis and i is impo an o be awa e o i s challenges
and limi a ions o be able o assess each and e e y
si ua ion acco dingly. Many o he algo i hms used o
pho og amme ic p ocessing all sho in he econs uc ion
phase (build mesh), esul ing in a ailed o p oblema ic 3D
mesh. Mos pho og amme ic p ocessing so wa e p o ide
ho ough documen a ion o help ci cum en ailed o
p oblema ic meshes, howe e , hese ollowing limi a ions
should be conside ed in all con ex s.
2.4.1 Occlusions and Numbe o Pho og aphs
Necessa y
While no so common in indoo cap u e s udios, occlusions
a e a common issue when cap u ing ou doo objec s o
scenes. An occlusion is when an unwan ed objec ge s in
be ween he came a and he a ge objec , esul ing in
missing da a. Howe e , his can be easily o e come by aking
mo e images om di e en angles, p o iding su icien da a
o he p ocessing algo i hm o compensa e o he occluded
ame. Fo hea y occlusions, cap u ing images e e y 10º is
ecommended. Fo objec s wi h minimal occlusion, images
aken e e y 20º is su icien [20]. As a ule o humb, when in
doub , ake mo e pho og aphs.
2.4.2 Fea u e De ec ion
Du ing he came a alignmen p ocess, he algo i hm sea ches
o ea u e poin s wi hin mul iple images. Fo example,
aking images o blank walls o ha ing un ocused sho s will
mos likely cause he algo i hm o ail. The mo e acking
ea u es he scene o objec has, such as pa e ns and/o
s ong dis inc i e a ibu es, he be e he esul s. In some
cases acking ma ke s can be added o he objec o yield
be e esul s.
2.4.3 Re lec i e, T anspa en o Glossy Su aces
Re lec i e, anspa en and glossy su aces a e
pho og amme y’s bigges conce n. Objec s o his kind ha e
21
li le o no su ace de ail o he so wa e o pick up on.
The e is cu en ly no way a ound his o he han o coa
hese e lec i e su aces wi h some hing ha will scan,
his howe e is no a g ea solu ion and qui e imp ac ical,
especially when i comes o la ge scenes con aining buildings
wi h a lo o glass, o example.
2.4.4 En i onmen al Condi ions
Pho og amme ic cap u e is hea ily dependen on wea he
condi ions. Because o his and in o de o imp o e he
success ul a e o a cap u e, we should be ully awa e o he
ollowing en i onmen al ac o s: su ace mo emen due o
wind o o he o ces, as his may dec ease he abili y o he
p ocessing algo i hm o ma ch scene poin s, esul ing in a
e y noisy o incomple e poin cloud. Excessi e a ia ion
in ambien ligh ing du ing image collec ion, caused by
changing cloud pa e ns, o example, is also some hing we
should look ou o [38]. Though no as se e e as cons an
su ace mo emen , changing in ligh ing condi ions will
impac he ecogni ion o ea u es.
LiDAR
2.4.5 Re lec i i y Limi a ions
A no able d awback o LiDAR is ha i does no wo k well in
scenes wi h a lo o ligh e lec ions. LiDAR is also un eliable
when used o cap u e non e lec i e su aces due o he ac
ha his echnology elies on he p inciple o ligh e lec ion
[39].
2.4.6 En i onmen al Condi ions
Since LiDAR uses isible lase s o measu e dis ance, his
echnology is o limi ed use in ha sh en i onmen s, especially
in ce ain wea he condi ions such as og, ain, snow and/o
hail, making i essen ially “blind” in hese condi ions.
2.4.7 Da a P ocessing Requi emen
Since LiDAR can co e la ge a eas a a as a e, i ends o
gene a e a lo o da a. Meaning adequa e equipmen is c ucial
o be able o p ocess and in e p e he la ge amoun s o da a
i is able o cap u e [39].

22
2.5 Con empo a y Da a Cap u e
Today, da a cap u e has become one o he main ields
o in e es su ounding he c ea ion o ealis ic digi al
ep esen a ions o he eal wo ld, ega dless o he
echnology chosen o do so. When we ask ou sel es wha
conce ns we may ha e when i comes o cap u ing and
ep esen ing eali y, some ques ions may a ise such as: “how
should I cap u e his scene o objec ?” O , “wha me hods
should I use o achie e he bes esul s possible?” Fi s and
o emos , we ha e o be clea on wha we wan o cap u e, in
o de o de e mine how we should p oceed o cap u e hese
da a as an asse o as a scene o be used in VR.
Cap u ing single objec s has gene ally been accomplished
wi h ease wi hin s udios using mul i-came a sys ems.
Mul iple came as a e placed a ound he objec , ully
encompassing i . Gene ally, he highe he came a coun ,
he be e , as mo e da a will be a ailable o eed in o
pho og amme ic algo i hms ha in u n will p oduce i s
digi al ep esen a ion. Fu he mo e, being in a con olled
en i onmen (e.g. s udio), gene ally allows o a highe
quali y cap u e, such as ha ing ull con ol o e global
ligh ing – a key ac o o a success ul cap u e.
The mos p ominen s a e-o - he-a example o
pho og amme ic cap u e can be seen in Wol Digi al Wo ld’s
sphe ical-shaped chambe ( igu e 13). Wi h a diame e and
heigh o 6 and 7 me es, espec i ely, and con aining 200
Sony A7RIVA came as, i can cu en ly be conside ed he
wo ld’s mos ad anced indoo eali y scanne .
On he o he hand, Leica Geosys ems ha e ecen ly
announced hei s a e-o - he-a BLK se ies au onomous
LiDAR scanne s: he BLK2FLY, BLK ARC and he BLK2GO
( igu e 14), p o iding easy da a cap u e solu ions o a wide
ange o se ings.
The Leica BLK2FLY is a d one-s yle lase scanne wi h
ad anced obs acle a oidance. I c ea es poin clouds du ing
ligh and was designed o cap u e building ex e io s,
s uc u es and he en i onmen . This de ice can be se o
au onomously cap u e di icul - o-access building loca ions,
such as oo ops and acades.
The Leica BLK ARC was designed o in eg a e wi h obo ic
ca ie s o p o ide au onomous mobile lase scanning wi h
minimal o no human in e en ion. Use s a e able o plan
a scanning pa h and hen launch he BLK ARC o scan he
desi ed a ea, au onomously.
23
Las bu no leas , he BLK2GO is a po able hand-held
imaging lase scanne ha ec ea es 3D spaces on he go.
I is able o cap u e images and accu a e poin clouds in
eal ime and uses SLAM (simul aneous localisa ion and
mapping) echnology o eco d i s pa h h ough space. The
BLK2GO au oma ically cons uc s poin clouds om he
momen you s a he scanning session o when you u n
o he de ice [40]. All images and 3D da a ge me ged and
aligned au oma ically, making his a big imp o emen o
he no o ious pos manual poin cloud egis a ion and
alignmen p ocess – a common s ep which was necessa y in
p e ious gene a ion lase scanne s.
I is clea ha eali y cap u e o e s unique oppo uni ies
ac oss a ange o indus ies and sec o s, and echnology
is ad ancing apidly. The con inued implemen a ion
o au oma ed p ocedu es o analysing and ex ac ing
in o ma ion om 3D poin clouds will con inue o p o ide
e sa ili y, ease o use and lowe cos . This will make
LiDAR and pho og amme y a ailable o a wide ange o
use cases, including non- adi ional use s. Ad ances in
compu e ision, AI and machine lea ning a e al eady adding
new dimensions o cap u ing eali y. Despi e he ecen
imp o emen s in hese ields, we a e s ill in he ea ly s ages
o compu a ional and so wa e-based me hods ha will o m
he basis o u u e eali y cap u e.
Figu e 13: Wol Digi al
Wo ld’s sphe ical-
shaped pho og amme ic
cap u e chambe .
A ailable a : «h ps://
me ahe o io/»
24
Figu e 14: Leicas BLK ARC, BLK2GO, BLK2FLY, om le o igh , espec i ely.
A ailable a : «h ps://blk2021.com/wp-con en /uploads/2021/09/ARC_2GO_2FLY.jpg»
3. Rela ed Wo k: Imme si e Expe iences using
Poin Cloud Da a
32
Figu e 18: S ill om In he Eyes o he Animal, specula ing he way mosqui os see he en i onmen .
A ailable in «h ps://www. lick .com/pho os/ma shmallowlase eas /»
Figu e 19: Pho og aph o he in-si u ins alla ion o In he Eyes o he Animal showing he cus om
VR helme s along wi h he ac ile subpack elemen s. A ailable in «h ps://www. lick .com/pho os/
ma shmallowlase eas /»

33
3.4 P omenade
C ea ed by I alian isual a is Da ide Quayola, P omenade
[46] akes a mo e syn he ic app oach o cap u ing na u e.
This p ojec eme ged om a commission by he Swiss
manu ac u e o luxu y mechanical wa ches Audema s
Pigue . Mo eo e , he ela ionship be ween Quayola’s wo k
and Audema s Pigue is ela ed o hei in e play be ween
adi ion and expe imen a ion. In o he wo ds, i is a balance
be ween he mas e e isi ing he adi ion while a he same
ime e isi ing i and ein e p e ing i in a new way.
P omenade is a sho ilm ha p esen s a high quali y bi d’s-
eye iew o he o es s o he Vallée de Joux in Swi ze land,
nea he b and’s o iginal wa chmaking acili y. Quayola used
high p ecision LiDAR scanne s o collec hese da a, hen
ansla ed he o es in o a wa e o in e connec ed oo age,
ans o ming he documen ed landscape in o digi al a .
A lo o Quayola’s wo k is cen ed a ound explo ing di e en
ways o seeing and disco e ing new isual languages wi h
he help o echnology, equi alen o s udying hings wi h
“ he eyes o machines”. Quayola himsel s a ed ha he is no
pa icula ly in e es ed in he objec he is ying o cap u e, he
is mo e in e es ed in he ension be ween he old language
behind i and he new language he helps disco e .
Al hough Quayola uses high-p ecision LiDAR scanne s,
he s ill enjoys he small impe ec ions i b ings du ing he
ansla ion p ocess ( he ans o ma ion o he o es in o
he digi al) ( igu e 20). “They (LiDAR scanne s) canno see
pe ec ly. Tha ascina es me. I you imagine an imp essionis
pain ing, you a e educing he esolu ion. You a e no aiming
o pe ec ion. Limi a ion b ings ou he exp ession.” –
Quayola [47].
Fu he mo e, P omenade plays wi h he me ging o he
eal wo ld wi h echnology. Du ing he cou se o he ideo,
di e en isual e ec s al e na e be ween he eal and he
digi al, gi ing us a peculia hyb id machine-like iew o
he o es . To u he enhance hese epheme al g aphics, a
ca e ully c a ed sound ack was added and synch onised
wi h hese isual e ec s.
Quayola’s bold explo a ion o hese new isual languages
push bounda ies in he way we can ex ac and isualise
eal wo ld da a in he con ex o na u e, by p ocessing isual
in o ma ion wi h a di e en logic. E en hough P omenade
has s onge “ echy” isuals and a “ ossil s yle” colou pale e
34
( igu e 21), compa ed o he mo e na u al o ganic na u e o
he desi ed ou come o he p ac ical componen o his hesis,
i ’s accu a ely designed sound ack demons a es jus how
impo an audio is o ge a mo e angible expe ience om
wi hin he digi al en i onmen .
Figu e 20: Rende om P omenade, no e how well LiDAR was able o cap u e he in icacies o he
o es . A ailable in «h ps://whi ewall a /li es yle/quayola-cap u es- allee-de-joux-audema s-pigue »
Figu e 21: S ill om P omenade, illus a ing i s “ echy” isual language.
A ailable in «h ps://www audema spigue com/com/zh-han /news/a /quayola-p omenade h ml»
35
3.5 Clams Casinos’ Moon T ip Radio
The se ies o ideos used o accompany Moon T ip Radio’s
album [48] is ano he g ea example o poin cloud da a used
in a c ea i e way. De eloped by P ague based compu a ional
a is Jakub Val a , his cus om made isualise akes a mo e
psychedelic app oach o olume ic da a ep esen a ion.
Based on d one-cap u ed oo age and pho og amme y, he
o he wo ldly isual language chosen o his p e- ende ed
ideos accompany Casinos’ sound acks and a e eminiscen
o he bioluminescen co als seen in he opics. This is
e iden in he i s song o he album: Clams Casino – Rune
( igu e 22), which akes you on a jou ney h ough wha seems
like a moun ainous e ain composed o high con as ing
luo escen poin clouds.
As opposed o he p e ious ela ed p ojec s, his one
p oceeded o ollow-up and conclude he econs uc ion
phase o he pho og amme ic p ocessing pipeline, ha
is, econs uc ing he aces (mesh) o he cap u ed scene.
Though no so ob ious a i s glance, once we ake a close
look we can see ha he luo escen poin clouds a e in ac
es ing on a da k mesh. This mus ha e been decided due o
he chosen high con as ing isual language, whe e a loa ing
poin cloud would pe haps ende in a e y noisy and unclea
manne due o poin clouds inhe en “see h ough” cha ac e .
Likewise o Quayola’s accu a ely designed sound ack,
he syne gy o Jakub’s ideos wi h Casinos’ music ascend
ou senso y expe ience o ano he le el. Fo example, i we
play Casinos’ in ica e sonics wi hou any isual s imuli, o
play Jakub’s ideos wi hou any music, we lose hal o hei
exp essi e po en ial in bo h cases. Jakub’s ideos, P omenade
and especially In he Eyes o he Animal each us ha in o de
o ake ull ad an age o any gi en imme si e expe ience, we
should place a s ong ocus on s imula ing as many o ou
senses as easible, because highe imme si e quali y le els
elici highe le els o p esence, which in u n inc eases he
e ec i eness o he o e all expe ience.
36
Figu e 22: S ill om Clams Casino – Rune, showing he da k moun ainous e ain composed o
luo escen poin clouds. A ailable in «h ps://www.you ube.com/wa ch? =sWeG68lNcVA»
37
3.6 Andy Shau – ‘Clo e Ciga e e’
The pic u esque ack ‘Clo e Ciga e e’ [49], by Andy Shau ,
is a song w i en abou hal - o go en memo ies ha ook a
new app oach o s o y elling and cinema og aphy. Following
ha heme, he music ideo ou s h ough Shau ’s mind as
he e isi s To on o’s skyline es au an , his backya d, and la e
summe nigh s.
Di ec ed by Colin Medley and Ja ed Raab – a p oduc ion
eam based in To on o, he ideo clip was p oduced using a
a ie y o echnologies including LiDAR, pho og amme y
and a enowned ideo game engine – Un eal Engine. Sau ’s
wo ld was cap u ed hen digi ally econs uc ed in o s ill,
eli scenes which we e hen ende ed ou as one con inuous
ideo. Because he song was abou hal - o go en momen s,
Ja ed and Raab wan ed o c ea e some hing d eam-like and
impe ec . A e looking o solu ions, hey s umbled upon
Ruben F o’s wo ks, ha is when hey knew hey had ound
he igh oolse and aes he ic o a icula e he eeling o he
song. Howe e , ins ead o unning e e y hing wi hin Uni y,
like Ruben, hey decided o y ou Un eal Engine, which was
some hing hey we e al eady amilia wi h.
The eam ini ially wan ed o c ea e poin cloud based
anima ions, bu ha ing un a ew es s o jus a s ill scene
wi h he addi ion o a mosphe ic ligh ing and smoo h
came a passages, hey igu ed ou ha was all hey needed.
By ollowing his app oach, hey we e able o c ea e a d eam-
like jou ney h ough Andy’s wo ld ha sui ed his wa m, hazy
summe ibes. By wo king in a ully digi al en i onmen , hey
had a lo o e sa ili y when i came o pe ec ing he came a
acks, adding pos -e ec s, changing he ligh ing and e en
b inging o eign elemen s in o he scene ( igu es 23 and 24).
This p ojec mo i a ed ou incen i e o es Un eal Engine as
an op ion o ende ou poin clouds. This led o ou disco e y
o i s supe io olume ic ligh ing capabili ies and easie
poin cloud impo ing me hods, compa ed o i s i al – Uni y.

38
Figu e 23: S ill om Andy Shau – ‘Clo e Ciga e e’, showing i s d eam-like and impe ec na u e,
he sub le dep h o ield came a and ligh ing e ec s along wi h he bu ning ciga e e pos e ec .
A ailable in «h ps://www.you ube.com/wa ch? =A8QPF S2_ E»
Figu e 24: S ill om Andy Shau – ‘Clo e Ciga e e’, showing i s pos e ec s (bloom/ og), ligh ing
(neon ligh s abo e es au an en ance and axi headligh s) and o eign elemen s ( axi and blue
bicycle). A ailable in «h ps://www.you ube.com/wa ch? =A8QPF S2_ E»
39
3.7 Reali ies – Cologne Ca hed al
Pa o he p ac ical componen o his hesis add esses he
p ese a ion o he Bo anical Ga den as a digi al backup.
This is some hing ha is common in he Cul u al He i age
and A chaeological Documen a ion ields, which ocus
p ima ily on c ea ing he ines digi al eplicas o eali y
possible. Reali ies.IO is a company ha specialises in eali y
cap u e, pa icula ly in he ield o Cul u al He i age, whose
main goal is o allow you o isi si es o in e es all o e he
wo ld. Reali ies.IO so wa e is a ailable o ee on S eam and
is cu en ly compa ible wi h he HTC Vi e, Oculus Ri , and
Windows Mixed Reali y.
Reali ies – Cologne Ca hed al [50] allows you o ou h ough
a digi al eplica o he Go hic a chi ec u e o he Cologne
Ca hed al – ano he one o UNESCO Wo ld He i age Si es.
This VR ou also includes a po ion o he Ca hed al whe e
isi o s a e no no mally allowed, adding an ex a sense o
exci emen and wonde o he VR expe ience when you a e
able o en e i s sec e ooms and enclosed a eas.
Simila o Clams Casino – Rune music ideo, Reali ies.IO used
pho og amme y and p oceeded o econs uc he esul ing
poin clouds in o ully ex u ed 3D polygonal mesh ( igu e
25). This was e iden ly done o achie e he highes le el o
ealism, howe e , i s “so edges” and “low-poly” appea ance
a e s ill qui e no iceable a close p oximi y ( igu e 26). In his
con ex i can be seen as a ai decision, none heless, i s ill
demons a es he impe ec na u e o pho og amme y. Fo
example, by compa ing his p ojec o he agmen ed o m
o La Ca héd ale ( igu e 27), a p ojec by Benjamin Ba dou,
we can clea ly see how i di e s om Reali ies.IO’s isual
language. Al hough La Ca héd ale is no as isually e ined,
i s poin cloud ep esen a ion allows us o dig a bi deepe
in o i s space and ge a be e eeling o i s a mosphe e.
Reali ies – Cologne Ca hed al is a clea example o ou decision
o use poin clouds as a isual ep esen a ion medium o his
hesis a he han a econs uc ed mesh, o a oid he “so
edge” appea ance esul ing om he pho og amme ic mesh
ace econs uc ion p ocess.
40
Figu e 25: S ill om Reali ies – Cologne Ca hed al, illus a ing i s econs uc ed mesh amewo k.
A ailable in «h ps://s o e.s eampowe ed.com/app/708620/Reali ies__Cologne_Ca hed al/»
41
Figu e 26: S ill om Reali ies – Cologne Ca hed al, showing an example o he “so edge”
appea ance. A ailable in «h ps://s o e.s eampowe ed.com/app/708620/Reali ies__Cologne_
Ca hed al/»
Figu e 27: S ill om La Ca héd ale, by Benjamin Ba dou, demons a ing an al e na i e poin
cloud based isual language used in he con ex o a ca hed al. A ailable in «h ps://you u be/
N 9W1TQ TGA»
48
Sep JanOc FebNo Ma MayDec Ap Jun Jul
1.Da a Acquisi ion
2.Da a P ocessing
3.P ojec De elopmen
4.E alua ion and Op imisa ion
1.a
2.a
2.b
2.c
3.a
3.b
2.d
1.b
3.c
3.d
3.e
3.
3.g
4.a
4.b
1.c
Figu e 30: Gan cha o ou ini ial wo k plan, demons a ing how hese ou phases would
be ca ied ou du ing he execu ion ime o his hesis.

49
Changes o he Wo k Plan
Due o se e al unexpec ed echnical d awbacks desc ibed
h oughou chap e s 5, 6 and 7, along wi h he de elopmen
o addi ional showcase ma e ial, oge he wi h he
composi ion o he wo k-in-p og ess pape o ICEC 2022,
which was ini ia ed du ing mid-p ojec de elopmen ,
he ou phases o ou ini ial wo k-plan ook a no iceable
empo al shi . Figu e 31 shows an upda ed Gan cha
o how he de elopmen o ou p ojec eally wen , along
wi h an upda ed wo k plan. He e he main di e ences a e:
he sepa a ion o he p ocessing o ou pho og aphic and
audio da a due o hei complexi y, ollowed by he addi ion
o ac i i y #5: De eloping showcase ma e ial, whe e we
de eloped a websi e and 360-deg ee ideos.
1. Da a Acquisi ion
a. Explo e and de elop a wo k low o cap u ing da a;
b. Collec pho og amme ic da a;
c. Reco d audio.
2. Da a P ocessing
a. De elop a wo k low o da a e inemen ;
b. P epa e cap u ed da a o poin cloud p ocessing;
c. C ea e poin clouds;
d. Re ine poin clouds;
e. Re ine Audio.
3. P ojec De elopmen
a. Explo e he exp essi e po en ial o poin clouds;
b. De elop he s uc u e o he i ual uni e se;
c. Explo e ways o enhancing ou expe ience wi hin his digi al uni e se by adding
in e ac i e elemen s;
d. Compose he imme si e expe ience (PC);
e. Run p elimina y e alua ion es s;
. De elop he imme si e expe ience compa ible wi h a VR headse .
4. E alua ion and Op imisa ion
a. Fine- une he imme si e expe ience o achie e he bes quali y- o-pe o mance a io o
bo h PC and VR a ian s;
b. Run a inal se o use e alua ion es s.
5. De eloping showcase ma e ial
a. Websi e de elopmen ;
b. Audio- isual ma e ial de elopmen .
50
Sep JanOc FebNo Ma MayDec Ap Jun Jul Aug
1.Da a Acquisi ion
2.Da a P ocessing
3.P ojec De elopmen
1.a
2.a
2.b
3.a
2.d
1.b
3.b
3.c
3.d
3.e
3.
4.a
1.c
Sep
2.e
4.b
4.E alua ion and Op imisa ion
Figu e 31: Gan cha o how ou wo k plan eally wen du ing he ime-span o his hesis.
2.c
5.b
5. De eloping Showcase Ma e ial
5.b
5. P ojec De elopmen
52
Be o e s a ing p ojec de elopmen , a b ie in oduc ion
and his o ical backg ound o he Bo anical Ga den o
Coimb a will be made in o de o help acclima ise ou
no ion o his loca ion o cul u al he i age. A e his, we
p esen a se ies o ou p elimina y expe imen s, ocused on
explo ing he bes solu ions o poin cloud da a acquisi ion
h ough pho og amme y. Finally, wo ini ial p o o ypes a e
desc ibed.
5.1 Objec o S udy – The Bo anical
Ga den
The Uni e si y o Coimb a’s s a u es o 1772 s a ed ha : “in a
place in he icini y o he Uni e si y ha is o be conside ed
he mos app op ia e and compe en , a said Ga den will soon
be es ablished; so ha e e y kind o plan may be cul i a ed
he e; and pa icula ly hose o which can ui use ulness in
he ield o medicine, a e o be expec ed” [55].
Loca ed in he hea o he ci y, he Bo anical Ga den is an
iconic space in Coimb a and a p es igious loca ion in Po ugal
o i s scien i ic con ibu ions o bo any, consis ing o a
ple ho a o anscon inen al lo a ha ake you on a jou ney
o egions h oughou he wo ld.
Belonging o UNESCO’s lis o p ope ies egis e ed as Wo ld
He i age since 2013, his Ga den has been expanding i s
isibili y and consequen ly he p esence o isi o s, who, in
addi ion o being able o pa icipa e in he a ea o scien i ic
cul u e, can equally enjoy his public space o leisu e, whe e
hey can mee o casual encoun e s and explo e na u e by
s olling by a e and exo ic plan s om all pa s o he wo ld
wi h o he na u al landscape en husias s. Fu he mo e,
his Ga den has also expanded i s ole in en i onmen al
educa ion, as a ec ea ional space o a ious ac i i ies
su ounding men al heal h and well-being, ou s, conce s
and exhibi ions.
Looking back in ime, he 18 h cen u y b ough abou a
e olu ion o minds and signi ican scien i ic p og ess.
The Bo anical Ga dens, he e o e, we e c ea ed in o de o
complemen and encou age he ields o Na u al His o y
and Medicinal S udies o he Uni e si y o Coimb a [56].
This con ex led o he c ea ion o he Bo anical Ga den in
1772, by he adminis a ion o he Ma quis o Pombal, on
land dona ed by Benedic ine ia s. Ini ially named Ho o
Bo ânico, he Ga den ook up only he a ea ha is s ill known
as he Quad ado Cen al (Figu e 32: Cen al Squa e, poin 5).
53
The s a u es o he Uni e si y o Coimb a de e mined he
cul i a ion o all so s o medicinal lo a, including plan s
o o e seas o igin. The cul i a ion o plan s began in he
Cen al Squa e in 1774 and was only comple ed in 1790, wi h
he addi ion o a cen al oun ain – an ins alla ion which
s ill emains o p esen day. Du ing his ime, he Facul y
o Medicine s udied he healing p ope ies o plan s which
led o an upg ade consis ing o a se ies o ec angula beds
o u he cul i a ion o medicinal lo a. Today, he Ga den
s ill keeps o his scien i ic componen h ough a seedbank
p og am. Published o he i s ime in 1868, his seedbank
includes many Po uguese and exo ic species, including
se e al endange ed a ie ies, which ha e played a pionee ing
ole in he conse a ion o na u e.
A he p esen ime, he Bo anical Ga den co e s a ound
hi een hec a es o land and can be di ided in o di e en
le els, s ai s and a enues, each con aining hei own unique
space, lo a and a mosphe e. Below is an illus a i e map
o he Ga den wi h enume a ed a eas depic ing i s mos
p ominen loca ions ( igu e 32), ollowed by a close look and
desc ip ion o he lis ed loca ions ( igu es 33, 34, 35, 36, 37
and 38).
One o he main objec i es o he Bo anical Ga den is o
es ablish a close link be ween he Ga den and he public, by
awakening hei in e es in science, na u e and bo any. This
is one o he main easons his loca ion was chosen as he
basis o his hesis, by me ging simila ideas and objec i es
hus emphasising he Ga den’s essence h ough he help o
con empo a y echnology.
Main Ga e
Baixo Rele o de Luís Ca isso
Alameda das Tílias
T opical G eenhouse
Cen al Squa e
Cold G eenhouse
Figu e 32: Bi d’s-eye iew illus a ion o he Bo anical Ga den depic ing i s
mos p ominen loca ions. Adap ed om «h ps://www acebook com/media/
se /?se =a 424667424214088& ype=3»
1
1
2
2
3
3
4
4
6
6
5
5

54
Figu e 33. Figu e 34.
Figu e 36.
Figu e 38.
Figu e 33: Main Ga e en ance, con aining he ma ble s a ue o bo anis Félix de A ela B o e o.
Da es om 1887 [58] – pho og aph aken in he Bo anical Ga dens o Coimb a, 05/11/21.
Figu e 34: Pho og aph om he Cen al Squa e, looking up a he Main En ance, con aining he
s a ue o he di ec o o he Ga dens in 1918 – P o esso Luís Ca isso [57]. A ailable a «h ps://
www. acebook.com/Ja dimBo anicoUC/pho os/? e =page_in e nal». Figu e 35: Alameda das Tílias,
one o he mos emblema ic places in he Ga dens, which eminisces he old 19 h cen u y public
oo pa hs o Eu opean ci ies [57]. Pho og aph aken in he Bo anical Ga dens o Coimb a, 05/11/21.
Figu e 36: T opical G eenhouse – one o he oldes i on a chi ec u e buildings in Po ugal.
I s pe ec combina ion o i on and glass gi e his space i s su p ising beau y. Reopened in
2018 a e a ew yea s o es o a ion, i is home o mainly opical and sub opical plan s [57].
A ailable a «h ps://www. acebook.com/Ja dimBo anicoUC/pho os/? e =page_in e nal». Figu e 37:
Neoclassical oun ain in he middle o he Cen al Squa e, su ounded by geome ically designed
walls, hedges and aised beds con aining a wide a ie y o lowe ing auna, ansmi ing he
en i e a mosphe e o Roman icism o his a ea [57]. A ailable a «h ps://www acebook com/
Ja dimBo anicoUC/pho os/? e =page_in e nal». Figu e 38: Cold G eenhouse, buil in he 1950’s,
unde he di ec ion o Bo anis Abílio Fe nandes. This g eenhouse con ains lo a o humid and
shaded en i onmen s, su ounded by a cascading wa e all and a small s eam ha uns h ough
i s en i e y. A emale nude s a ue by sculp o Ma ins Co eia also inhabi s his a ea, named
“Bo ânica’’, who symbolises he Science o Plan s [57]. A ailable a «h ps://www acebook com/
Ja dimBo anicoUC/pho os/? e =page_in e nal».
4
6
12
Figu e 35.
3
Figu e 37.
5
55
5.2 P elimina y Expe imen s
Be o e s a ing any de ini i e de elopmen , ini ial es s we e
un in an e o o ge accus omed o he pho og amme ic
p ocessing pipeline. Mo eo e , hese es s we e ca ied ou o
de e mine an e icien wo k low o bes sa is y he p ac ical
goals and objec i es o his hesis.
5.2.1 Expe imen nº1 – 360º Came a
In an a emp o cap u e as much eal-wo ld da a in he
sho es amoun o ime, we es ed a 360º 4k ideo eco ding
de ice, mo e speci ically, he Samsung Gea 360. This idea
s emmed om he u ban-based p ojec s by Ruben F osali
(3.1 Memo ies o Tsukiji, page 26) and Benjamin Ba dou (3.2
Shinjuku, page 29), as hese we e also eco ded using 360º
came as.
In ha espec , we eco ded a sho 25 second ideo, whe e
we ollowed a p ede e mined pa h ha e ol ed a ound a
well known Tilia ee o he Bo anical Ga den ( igu e 39),
hus cap u ing a comple e 360º iew o he ee. In o de o
be able o ans o m his ideo clip in o a poin cloud, we
i s had o ende he 360º oo age in o a compa ible o ma
equi ed by he pho og amme ic p ocessing so wa e Agiso
Me ashape. This was accomplished wi h Samsung’s Gea 360
Ac ionDi ec o , a media p ocessing so wa e used o s i ch
he 360º ideo in o a s anda d ideo ame ( igu e 40). Basic
ideo colou co ec ion was also done wi hin Ac ionDi ec o ,
o b ing ou he shadows and o co ec any o e blown
highligh s in an e o o aid Me aShape’s ea u e de ec ion
p ocess.
This oo age was hen impo ed in o Me aShape’s
pho og amme ic p ocessing pipeline using i s de aul
p ocessing pa ame e s. This esul ed in a poin cloud o 27
million poin s ( igu e 41). A ew no es can be aken om his
ini ial es , namely he la ge amoun o noise in he esul ing
poin cloud wi h s ange loa ing sky and cloud pa icles, as
well as some e iden alignmen issues esul ing in duplica e
objec s ( igu e 42). None heless, his poin cloud was hen
impo ed in o Un eal Engine, using i s buil -in poin cloud
impo e , o be u he analysed in he con ex o a eal- ime
i ual en i onmen .
A i s , Un eal Engine was ha ing some di icul ies ende ing
he poin cloud ( igu e 43), i s s uc u e was incomple e
and pa s o he poin cloud would appea and disappea
in e mi en ly du ing iewpo mo emen . Howe e , his
issue was bypassed by inc easing he poin cloud plugin’s
56
de aul 1 million poin budge o a alue g ea e han 4
million ( igu e 44). I is o no e ha he la ge he poin
budge , he mo e de ailed he image becomes, howe e , his
comes wi h he bu den o exponen ially highe p ocessing
equi emen s, hus lowe ame- a es. In his expe imen and
wi h he ha dwa e used, we we e able o ende 20 million
poin s on-sc een, in eal- ime, a a s able ame- a e o 60
ames pe second.
Figu e 39: S ill om he aw 360º oo age o he Tilia ee.
Figu e 40: S ill om he now s i ched 360º oo age wi h imp o ed shadows and educed
highligh s.
57
Figu e 41: Sc een cap u e o he esul ing poin cloud wi hin CloudCompa e, no e i s noisy na u e,
s ange misaligned ee b anches and blue sky pa icles.
Figu e 42: Sc een cap u e showing e iden alignmen issues esul ing in duplica e objec s.
64
Figu e 53: Sc een cap u es illus a ing CloudCompa e’s colou il e ing addon ollowing he
emo al o he blue sky poin s (Le : O iginal, Righ : Cleaned).
5.2.5 Fi s P o o ype
Ha ing al eady explo ed di e en isual ep esen a ions
o ou poin clouds, we decided o op o a middle g ound
be ween accu acy and noise/ agmen a ion. Displaying
enough da a o allow us o unde s and ou su oundings, bu
no ealis ic enough so as o no emo e ou sel es om he
d eam-like s a e o eli ing a memo y.
I was unde s ood ha ha ing an ini ial high quali y da ase
o wo k wi h was he mos sensible cou se o ac ion. This
is because we can always educe he inal esolu ion o
ou poin cloud, bu can ha dly inc ease i . Fo his eason,
cap u ing eali y by aking high esolu ion pho og aphs
in RAW o ma was he chosen and inal me hod o da a
cap u e.
Ou inal p elimina y expe imen began wi h he cap u e
o he en i e Main En ance o he Bo anical Ga den, whe e
450 pho og aphs we e aken. To no miss any da a, se e al
ounds o pho og aphs we e sho , ollowing a me hod simila
o emula ing a one-eyed cybo g ying o make sense o his
space, by e ol ing a ound each and e e y objec o he scene
mul iple imes a di e en angles ( igu e 54).

65
Simila o he p e ious expe imen , hese pho og aphs
we e hen colou co ec ed and e- ouched wi hin Adobe
Ligh oom be o e ge ing impo ed in o Me ashape. This
ime, in an a emp o u he enhance accu acy and ea u e
de ec ion, all pho og aphs we e sligh ly sha pened and
de-noised. Fu he mo e, we decided o inc ease some o
Me ashape’s pa ame e s, namely he numbe o desi ed
ea u e poin s o he algo i hm o look o , as his was
hough o inc ease he numbe o poin s in he ou come
o he ollowing build dense-cloud p ocess. This pa ame e
alone inc eased he numbe o poin s o ou poin cloud o
174 million, esul ing in a a he la ge and hea y ile ( igu e
55).
Na u ally, a lo o loa ing poin s and alse co espondences
we e included in his sum and needed o be assessed wi hin
CloudCompa e ( igu e 56). Floa ing blue sky poin s we e
emo ed wi h he colou il e ing addon, hen he gene al
noise was educed wi h a noise il e (by emo ing any
o phaned loa e s g ea e han a ce ain adius o each and
e e y poin ), ollowed by a subsample o lowe he global
poin coun o 50 million poin s ( igu e 57). Al hough no ye
de ined, his subsample alue was pu posely used o imp o e
s o age capabili ies along wi h imp o ing esponsi eness and
pe o mance wi hin Un eal Engine.
This poin cloud was hen impo ed in o Un eal Engine,
whe e he ine de ails o ou selec ed isual language we e
explo ed o ge be e accus omed o he capabili ies o
Un eal Engine. Wi hin his i ual se up, se e al isual pos -
p ocessing ea u es we e explo ed o enhance he poin cloud
en i onmen . Adhe ing o a mo e ealis ic app oach, he e we
explo ed dynamic ligh ing, olume ic og and shadows, o
emula e he ligh ays we had obse ed adia ing h ough
he ee’s lea es and b anches ( igu es 58, 59 and 60). This
ollowed by he explo a ion o poin shape and size, in an
e o o ind a balance be ween ealism and he a mosphe e
o a d eam-like s a e ( igu e 61).
66
Figu e 54: Sc een cap u e o he esul ing spa se cloud wi hin CloudCompa e, showing i s a ious
angles o cap u e.
Figu e 55: Sc een cap u e o he esul ing poin cloud wi hin CloudCompa e, esul ing in a poin
cloud o 174 million poin s.
67
Figu e 56: Sc een cap u e wi hin CloudCompa e, illus a ing i s loa ing poin s and alse
co espondences.
Figu e 57: Sc een cap u e wi hin CloudCompa e, showing he educ ion o loa ing poin s and
alse co espondences a e an impe ec clean-up p ocess.
68
Figu e 59. Sc een cap u e wi hin Un eal Engine, illus a ing he olume ic og and dynamic
ligh ing e ec s implemen ed o added ealism.
Figu e 58. Sc een cap u e wi hin Un eal Engine, illus a ing he olume ic og and ligh ay
pos -p ocessing e ec s adia ing h ough he ee’s lea es and b anches.
69
Figu e 60. Sc een cap u e wi hin Un eal Engine, showing he explo a ion o poin size and shape.
Figu e 61: Sc een cap u e wi hin Un eal Engine, illus a ing he esul ing balance be ween ealism
and he a mosphe e o a d eam-like s a e.

70
5.4 Second P o o ype: High ideli y Tilia x
eu opaea
A e concluding he p e ious expe imen s and ha ing
es ablished mos o he me hodology, a mo e comple e, high
ideli y p o o ype was de eloped o include he majo i y o
he ea u es expec ed in he inal p ojec .
Du ing he de elopmen o ou ini ial p o o ype, we no iced
ha Un eal Engine’s buil -in poin cloud impo e was
showing some issues whils ende ing he poin s on sc een.
Fo easons s ill o be de e mined, some poin clus e s would
s a o licke depending on he playe ’s angle o iew wi hin
he i ual en i onmen , his issue became qui e no iceable
and nega i ely a ec ed ou use expe ience. Coinciden ally,
his issue is also appa en in Andy Shau ’s – “Clo e Ciga e e”
music ideo (men ioned in sec ion 3.6: page 37), which was
also de eloped using Un eal Engine, whe e agmen s o he
scene would disappea and eappea in an ab up ashion.
A e inding no solu ion o his p oblem, i was decided
ha we shi o e o Uni y’s game engine, as his was also
so wa e we we e mo e amilia wi h.
This high ideli y p o o ype consis s o a single a ea o he
Bo anical Ga den ha was cap u ed du ing No embe ,
2021 con aining he Tilia x eu opaea [59]. The de elopmen
o his p o o ype was di ided in o six s ages: 1. Poin Cloud
Rep esen a ion, 2. Rep esen a ion o Na u al P ocesses, 3.
In e ac ions, 4. Sound, 5. Locomo ion and A ea T a e sal, and
las ly 6. P elimina y E alua ion.
He e u he isual explo a ion was pe o med on ou poin
cloud, oge he wi h di e en app oaches o ep esen and
highligh some o he scenes na u al p ocesses associa ed
wi h he ee’s biology. Mo eo e , sound and use in e ac ion
which allow use s o di ec ly in luence he beha iou o he
expe ience was also explo ed, including locomo ion and
a ea a e sal. Las ly, a se o p elimina y use es s we e
pe o med o unde s and how people would eac o he
imme si e en i onmen .
Made up o 275 pho og aphs, his cap u e esul ed in a dense
cloud o 100 million poin s ( igu es 62 and 63). Howe e ,
a e unning in o some ini ial pe o mance issues (low
ame a e), i was disco e ed ha we could only ende
app oxima ely 20 million poin s a any gi en ime, meaning
ha ou ini ial idea o me ging mul iple a eas o he
Bo anical Ga den in a sandbox s yle open-wo ld was ou o
he ques ion. Du ing he de elopmen o his p o o ype we
71
Figu e 63: A close iew o he ee whe e we can see how he cap u e p ocess was done.
Figu e 62. Sc een cap u e wi hin Me ashape showing he alignmen o he 275 pho og aphs,
esul ing in a poin cloud o app oxima ely 100 million poin s.
u he disco e ed ha he g aphics API se wi hin Uni y’s
p e e ences hea ily in luenced he pe o mance o ou scene,
whe e Vulkan API would ou pe o m Di ec X11 by a ound
40 ames pe second, bo h using he exac same scene
pa ame e se ings ( igu es 64 and 65).
72
Figu e 64. Sc een cap u e o he scene being ende ed using he Vulkan API.
Figu e 65: Sc een cap u e o he scene being ende ed using Di ec X11, no e he change in ame
a e compa ed o he p e ious igu e (see op igh s a . ba s).
73
5.4.1 Poin Cloud Rep esen a ion
Th ee poin cloud ende ing me hods we e explo ed o
ind a balance be ween isual ideli y and compu a ional
pe o mance. Ou poin cloud was ini ially ende ed as
a s a ic objec wi h a ixed poin size, howe e , i s poin s
would become la ge a close p oximi y and hus diminish
he isual d eam-like aes he ic we we e looking o ( igu e
66). As an al e na i e, a “pixel absolu e” ende ing me hod
was explo ed so ha each poin would occupy he size o a
pixel a all imes ( igu e 67). None heless, wi h his me hod,
he poin s a hes away in he scene would ine i ably
scale much la ge han he ones close o he came a
(use ), esul ing in an unappealing con as o poin sizes.
Mo eo e , du ing hese es s we no iced a signi ican d op in
pe o mance depending on he se size o ou poin s, whe e
a smalle global poin size would ende much as e han
a la ge poin size. As a hi d a emp , Uni y’s VFX ( isual
e ec s g aph) was used o ende and anima e ou poin
cloud ( ia ex u e iles con aining i s posi ion and colou
da a), he e i s con inuous poin bu s ea u e wi h he
addi ion o o ce ields was explo ed. This me hod esul ed in
in e es ing isuals domina ed by oiling ex u es ( igu e 68),
howe e , i quickly became appa en how compu a ionally
hea ie his me hod was compa ed o he i s wo.
F om hese esul s i was decided ha he ini ial ixed poin
size ende ing me hod be used, his ime wi h a smalle
global poin size, as he inal ende ing me hod ( igu e
69). This me hod no only c ea ed a mo e uni o m look
h oughou he scene, bu also p oduced a mo e pho o
ealis ic and consis en appea ance a di e en esolu ions
(1080p, 1440p, and 4k).
Figu es 66 and 67: Compa ison be ween “ ixed poin size” (Le ) and “pixel
absolu e” (Righ ) poin ende ing me hods.
80
Figu e 79: A ea a e sal zones a ei he ex emi y o ou a ea.
~3.64m
~4.82m
~5.01m
Figu e 80: Rep esen a ion o he p oximi y sys em ha con ols he ade-ou
e ec . The ed poin s ( ) ep esen ing he p oximi y senso s, along wi h hei
espec i e dis ances o he use (came a).
Zone 1
Zone 2
Use (came a)
P oximi y Senso s
T igge Box

81
5.4.5 P elimina y E alua ion
This p o o ype was only a ac ion o ou desi ed goal a
he ime, ne e heless, we wan ed o pe o m p elimina y
es s wi h pa icipan s ex e nal o ou eam o unde s and
how people would eac o he imme si e en i onmen ,
and o also ga he ex a da a o submi ou WiP (Wo k-in-
P og ess) submission o he In e na ional Con e ence on
En e ainmen Compu ing [62].
Du ing a high-school isi o ou Uni e si y on he 27 h o
Ap il 2022, we an in-pe son play es s on six een s uden s
( igu e 81). Each pa icipan explo ed bo h de eloped
a ian s o ou i ual en i onmen : one a ian used
a keyboa d and mouse (wi h a ixed al i ude) o mo e
and explo e he en i onmen ; he second a ian , used a
gamepad con olle (wi h he abili y o ly). Du ing hese
es s we eco ded use analy ics, including play ime, and
use posi ion and o ien a ion acking wi hin he a ea.
We subsequen ly asked ou pa icipan s o answe a sho
eedback ques ionnai e whe e hey we e asked o desc ibe
hei expe ience in i e wo ds, wha a ian hey p e e ed
(keyboa d o gamepad con olle ), and las ly, a line o
commen s and/o sugges ions.
The a e age play ime o he keyboa d and mouse was 53
seconds whils he gamepad con olle was 62 seconds. The
mos popula wo ds we e “Realis ic” (11.8%), “Fun” (11.8%),
“In e es ing” (9.4%) and “Inc edible” (7.1%). Two hi ds o
he pa icipan s p e e ed he gamepad con olle o e he
keyboa d and mouse, howe e , his ac o was suspec ed o
be linked o he young demog aphic so u he es s we e
planned o be ca ied ou in he u u e, on a wide audience.
In e ms o use posi ion acking and despi e he a ea ha ing
a cen al ocal poin (being he Tilia ee), we no iced ha
mos use s p oceeded o explo e he en i e space, as can
be seen in he use posi ion acke s ( igu es 82 and 83).
Fu he mo e, we also no iced ha mos o ou pa icipan s
had he u ge o a el o he nex a ea, wan ing o explo e
mo e a eas.
Las bu no leas , some use s managed o bypass he
se bounda ies (wall collide s) o ou a ea, an issue ha
e en ually go esol ed by inc easing he hickness o he
wall collide s. In gene al, hese p elimina y esul s showed a
posi i e e alua ion and p o ided enough eedback o guide
us o con inue o expand and imp o e ou i ual wo ld.
Figu e 81: “Gi ls in ICT day” a he Depa men o
In o ma ics Enginee ing o he Uni e si y o Coimb a,
whe e we an in-pe son play es s on sixe een high-
school s uden s.
82
Figu e 83: Use posi ion
acking o he gamepad
con olle a ian wi h he
abili y o ly.
Figu e 82: Use posi ion
acking o he keyboa d and
mouse a ian wi h a ixed
al i ude. Each colou ep esen s
a pa icipan and he a ows
show in wha di ec ion he
playe was looking.
6. De elopmen o Kai os
84
The de elopmen o he main p ac ical componen o his
hesis was s a ed by es ablishing he numbe o a eas we
we e going o cap u e, wha na u al p ocesses we we e going
o include wi hin each a ea, how we would in e connec
hese loca ions along wi h how he use would in e ac
wi h said a eas. Ha ing al eady isi ed he Bo anical Ga den
mul iple imes, we op ed o h ee loca ions highligh ing
some o he oldes iconic ee specimens o he Ga den: he
Tilia x eu opaea [9], si ua ed nea he Alameda das Tílias,
he E y h ina C is a-Galli [10], loca ed wi hin he Cen al
Squa e, and he Ficus Mac ophylla [11], loca ed beside he cold
g eenhouse ( igu e 84).
A ea 1
Tilia x eu opaea
A ea 3
Ficus Mac ophylla
A ea 2
E y h ina C is a-Galli
Figu e 84: Map o he Bo anical
Ga den depic ing he h ee chosen
loca ions o cap u e.
85
A s a egy and cou se o ac ion was de ised on how we would
cap u e hese h ee loca ions. Ini ially, he cap u e o hese
a eas was planned wi hin he closes possible ime ame,
o concei e he idea o eplaying a memo y o a walk in he
Ga den, bu because each loca ion equi es a la ge se o
pho og aphs, aking a ound 30 o 50 minu es pe a ea, we
decided o spli he cap u e session in o h ee sepa a e pa s:
mo ning ( o he Tilia), noon ( o he C is a-Galli) and mid-
a e noon ( o he Ficus), simula ing a day’s passage.
Time was spen wi hin each a ea be o e commencing ou
cap u es, by ga he ing and analysing in o ma ion on he
isible na u al p ocesses occu ing in each a ea. Scien i ic
in o ma ion was hen collec ed o he ees oo sys em
s uc u es and hei symbio ic ela ionships wi h hei
su ounding en i onmen [63][64] ( igu e 85). This was
inspi ed by he hidden unde g ound wo ld o mic obes, wi h
emphasis on myco hizal ne wo ks [65], ha is, he symbio ic
ela ionships ha o m be ween ungi and plan s ( igu e 86).
6.1 Da a Cap u e and P ocessing
Da a cap u e commenced in a ch onological o de , s a ing
wi h he cap u e o he Tilia x eu opaea, ollowed by he
E y h ina C is a-Galli and ending wi h he Ficus Mac ophylla.
A lo mo e ca e was aken du ing he cap u e p ocess, as hese
da a would be used o ou inal imme si e en i onmen .
Because o his, a highe numbe o pho og aphs we e aken
o imp o e he di e si y o ou da ase , and hus, he quali y
o ou poin clouds.
Ou ini ial plan o cap u e all loca ions on he same day ailed
due o he cons an change o wea he condi ions, along
wi h exceeding ou came a’s limi ed ba e y and s o age
capaci ies. Consequen ly, each a ea go cap u ed on di e en
da es. Mo eo e , he i s wo a eas became a p io i y due o
hei ee’s limi ed lowe ing pe iods, gi ing a span o abou
wo weeks o cap u e and p ocess hese wo. Fo una ely
he Ficus Mac ophylla keeps i s lea es and ui ( igs) all yea
ound, making i less o a p io i y a he ime.
The success ul cap u es o ou a eas ook place on he 8 h o
June om 15:56pm o 16:30pm (Ficus Mac ophylla), on he 9 h
o June om 11:52am o 12:38pm (E y h ina C is a-Galli) and
on he 12 h o June om 9:20am o 9:44am (Tilia x eu opaea)
Ou pho og aphic da ase was impo ed and analysed wi hin
Adobe Ligh oom. He e he p ocess o colou co ec ion
ensued in an a emp o b ing ou as much de ail om he
Figu e 85: “Wo ld Wide Web” – Illus a ion depic ing
ees symbio ic ela ionship wi h hei su ounding
en i onmen . A ailable in: h ps://www a biope u o g/
en/blog-pos s/mo he - ees/
Figu e 86: A small pine ee g own in a glass
ank shows he amoun o whi e, inely b anched
symbio ic mycho hizal h eads ha help eed
he ee. A ailable in h ps://www csuchico edu/
egene a i eag icul u e/blog/how- ungus-migh -sa e-
us sh ml

86
Figu e 87: Colou co ec ion, wi hin Adobe Ligh oom, o
he a ea con aining he E y h ina C is a-Galli, in an e o
o b ing ou as much de ail om he scene as possible.
scenes as possible ( igu e 87). Ligh oom’s “Ma ch To al
Exposu e” ea u e was also used o au oma ically equalise
he exposu e alue o all pho og aphs in hopes o be e
Me ashape’s ea u e acking p ocess. Al hough no
ecommended by Agiso Me ashape’s use manual, we
had no choice bu o comp ess ou pho og aphic da ase
o he lossy comp ession o ma JPG, o a oid unning ou
o memo y (RAM) du ing Me ashape’s came a alignmen
p ocess.
Albei ha ing al eady gained expe ience om ou p elimina y
es s, new issues a ose conce ning ailed came a alignmen ,
unning ou o sys em memo y, and ailed objec alignmen
wi hin he scenes, consis ing o mul iple misaligned ee
unks and b anches ( igu es 88 and 89).
Pa adoxically, hese issues we e only sol ed by educing he
size o ou pho og aphic da ase h ough manual selec ion,
whe e pho og aphs deemed edundan and p oblema ic go
emo ed, such as hose wi h a lo o occlusions and ones
wi h simila angles o cap u e. F om an ini ial pho og aphic
da ase anging om 700 o 1100 pho og aphs pe a ea,
ou inal da ase consis ed o a ange o only 400 o 600
pho og aphs. Du ing he emo al p ocess, we ei e a ed he
pho og aph alignmen and “build dense-cloud” p ocesses
se e al imes un il we ended up wi h a success ul poin cloud.
87
The a ea con aining he E y h ina C is a-Galli was he mos
p oblema ic due o i s high concen a ion o ege a ion, low
p ominen ea u e- acking poin s and he cons an change
in wea he and ligh ing condi ions du ing ou cap u e
sessions, which nega i ely in luenced he accu acy and
ou come o ou poin cloud. We ended up e-cap u ing he
Tilia x eu opaea and E y h ina C is a-Galli se e al imes due o
he simila issues men ioned abo e.
Figu e 89: Rep esen a ion o ailed objec alignmen wi hin he scene con aining he C is a-Galli,
no e he duplica e b anches and wooden ee suppo poles.
Figu e 88: Rep esen a ion o a ailed came a alignmen wi hin he Tilia scene, no e he s ange
came a posi ioning a he bo om igh co ne .
88
When i came o he Ficus Mac ophylla, bo h cap u e and
p ocessing o his scene we e concluded in a single un wi h
ease, pe haps his was due o he highe p ominen ea u e
coun o he scene, oge he wi h mo e s able wea he
condi ions du ing he cap u e session ( igu e 90).
Figu e 90: Rep esen a ion o a success ul came a alignmen wi hin he scene con aining
he Ficus (spa se cloud), whe e he blue ec angles ep esen e e y pho og aph aken.
Figu e 92: Localised me ging, scaling and alignmen o he C is a-Galli’s hollow unk, using da a
om wo sepa a e poin clouds (Le ); Be o e (Cen e); A e (Righ ).
Figu e 91: Alignmen o wo sepa a e poin clouds
o he same a ea (highligh ed in ed and yellow), in
an e o o en ich ou inal poin cloud.
Despi e ha ing al eady ca ied ou mul iple cap u e sessions
on he E y h ina C is a-Galli, each session esul ed in an
insu icien poin cloud da ase . Fo ins ance, he i s cap u e
ailed o ep oduce he ee’s lea es, lowe s and b anches,
bu success ully managed o cap u e he ee’s hollow unk
wi h g ea de ail. The second and hi d cap u es esul ed in
poin clouds wi h a lo mo e lea , b anch and lowe de ail bu
ailed a ep oducing he ee’s unk. This issue go esol ed
by me ging, scaling and aligning sec ions o poin cloud da a
om all h ee cap u es in o one ( igu es 91 and 92).
89
6.2 Da a Re inemen
P oceeding wi h he e inemen o ou poin clouds, an
ini ial manual emo al o unnecessa y and ou -o -bounds
poin s wi hin Me ashape was execu ed using i s ee- o m
selec ion ool ( igu e 93). Subsequen ly, all undesi able
poin cloud da a such as blue sky and whi e cloud poin s
go emo ed h ough Me ashape’s colou selec ion ool,
oge he wi h CloudCompa e’s RGB il e ool, wi h a
cus om de ined ole ance pe cen age o each and e e y
colou emo al i e a ion ( igu e 94). Following he majo
clean-up wo k, we we e s ill le wi h a la ge olume o
poin s ha needed subsampling (i.e. dec ease he numbe o
poin s) o p ese e pe o mance o Uni y. This p ocess go
ini ia ed using CloudCompa e’s “Remo e duplica e poin s”
ea u e, whe e poin s wi h a adius o less han 0.0099 uni s
(CloudCompa e’s uni s o dis ance) om one ano he go
emo ed. This e icacious ool alone was able o emo e
mo e han hal he o al poin coun , eeing up space while
main aining isual cohe ence. These esul s conclude ha
ou pho og amme y de i ed poin clouds con ained a la ge
numbe o nea ly o e lapping poin s.
Figu e 93: Raw dense cloud o he a ea con aining
he Ficus ee, consis ing o 344 million poin s (Top).
Dense cloud a e manual clean-up, now wi h 261
million poin s (Bo om).
96
Figu e 106: The small Lasioglossum Dialic us (Le )
and he la ge Apis melli e a (Righ ), seen on he
Tilia a he Bo anical Ga den.
Se e al p e iously obse ed na u al p ocesses ( ha we e seen
du ing ou isi s o he Bo anical Ga den) we e implemen ed.
Ou o hese, wo dis inguishable bee species we e obse ed
and egis e ed wi hin he a ea con aining he Tilia x eu opaea,
which we e a he ime ga he ing nec a om he Tilia’s
ag an , pale yellow lowe s: he small Lasioglossum Dialic us
and he la ge Apis melli e a (also known as he Eu opean
honey bee), ( igu e 106). These we e bo h simula ed wi hin
VFX g aphs and also include use collision de ec ion.
Mo eo e , he Tilia’s lea posi ion da a was used as inpu o
he spawn posi ions o he smalle bees (encompassing he
en i e ee, as was obse ed in eal li e). Two swa ms o he
la ge honey bees we e manually placed wi hin he a ea,
posi ioned in wo sepa a e loca ions. He e p oximi y senso s
we e implemen ed ha igge when he use ge s oo close,
his inci es he bees which hen s a e ol ing a ound he
use in a mo e agi a ed ashion ( igu e 107).
Figu e 107: Rep esen a ion o he Apis melli e a honey bees wi hin Uni y.

97
Se e al Pie is apae bu e lies we e obse ed lu e ing
wi hin he a ea con aining he E y h ina C is a-Galli, in he
ose pa ches adjacen o he ee ( igu e 108). These go
simula ed by anima ing wo 2D ex u es (emula ing wings)
by scaling hem ollowing a sign wa e (see igu e 109).
Likewise o he bees, hese bu e lies also eac o he use ’s
p esence and dispe se when he use ge s oo close ( igu e
110).
Figu e 108: Pie is apae bu e lies. A ailable a : h ps://species
wikimedia o g/wiki/Pie is_ apae#/media/File:Pie is_ apae_
MHNT jpg
Figu e 110: Rep esen a ion o he Pie is apae bu e lies wi hin Uni y.
Figu e 109: Le : sc een cap u e o he bu e lies VFX g aph, showing he
p ocedu e o anima ing hei wings by scaling wo 2D ex u es ollowing a
sine wa e. Righ : Illus a i e ep esen a ion o he wing anima ion.
98
To accen ua e he C is a-Galli’s ed ag an lowe s, i was
decided o simula e he smell o nec a and pollen being
eleased om hese ( igu e 111). Pa icle sys ems whe e
posi ioned in he cen e o each lowe clus e , which
elease small pa icles ( ep esen ing he nec a and pollen).
Fu he mo e, each clus e con ains i s own p oximi y senso
ha bu s s pa icles i he use decides o collide o come in o
con ac wi h he lowe s. Ha ing an ini ial bu s quan i y
o 600 pa icles, o e e y consecu i e collision, i s bu s
quan i y ge s di ided by h ee ( igu e 112).
Figu e 111: Flowe clus e o he C is a-Galli, ich in nec a .
Figu e 112: Sc een cap u e o he C is a-Galli’s lowe pa icle sys ems – specula ing he elease o
pollen and nec a .
99
Du ing he cap u e session o he Ficus Mac ophylla, i was
no iced ha a lo o i s igs had allen o he g ound ( igu e
113), his is also e iden in he inal audio eco ding o he
a ea, whe e we can hea igs d opping. Hence, i was decided
o simula e hese igs using wo VFX g aphs. The i s
andomly spawns igs on he g ound on scene load ( igu e
115), he second emi s igs om he ee (using he ee’s
lea -posi ion da a as inpu ), whe e a andom ig ge s d opped
(emi ed) wi hin a andom ime-space be ween ou and
eigh seconds. A colou sample ange was hen added om
an ac ual ig o di e si y he colou s o ou pa icle sys em
( igu e 114). Addi ionally, a sub le o ce is applied o hei
nega i e y-axes, simula ing g a i y. Fo added ealism, once
hese all o he ee, hey ligh ly bounce o he g ound ( ia a
loo collide ) be o e coming o a hal .
Small clus e s o lies we e also spo ed wi hin he a ea
( igu e 116) and implemen ed wi hin he scene, ollowing
simila pa ame e s used o he bees su ounding he Tilia,
wi h he addi ion o ails ( igu e 117).
As a las no e, se e al bi ds we e obse ed ea ing igs o he
g ound, howe e , hese did no end up ge ing simula ed due
o ime cons ain s.
Figu e 113: Pho og aph o allen igs unde he Ficus.
Figu e 114: Pho og aph o one o he Ficus’s igs, oge he
wi h i s colou g adien sample.
Figu e 115: Rep esen a ion o allen igs wi hin Uni y, highligh ed in o ange.
100
Figu e 116: Small clus e s o lies spo ed nea he Ficus.
Figu e 117: Small clus e s o lies simula ion wi hin Uni y.
101
6.4 Locomo ion and A ea T a e sal
Taking he eedback da a om he p elimina y e alua ion
es s in o conside a ion, i was decided o keep bo h modes
o use in e ace: he keyboa d and mouse a ian , and he
game-pad con olle a ian , bo h con igu ed using he “ ly
came a” mode (g an ing he abili y o ly). Subsequen ly
a sc ip go implemen ed ha con inuously de ec s i a
con olle is connec ed o no and swi ches he inpu de ice
acco dingly.
Below is he splash sc een c ea ed o Kai os, con aining a
diag am o he mouse and keyboa d con ols as well as he
gamepad con olle .

102
Ou inal a ea a e sal sys em is based on he
implemen a ion om he high ideli y p o o ype. Figu e 118
shows a op iew o he Bo anical Ga den con aining all h ee
cap u ed a eas, including an illus a ion o he a ea a e sal
sys em whe e exis ing pa hways wi hin each a ea we e used
as exi and en y poin s.
Ha ing no iced ha he a ea con aining he E y h ina C is a-
Galli included ou pa hways, i became appa en o ake
ad an age o his by ins alling wo exi and wo en y poin s
o and om he o he a eas – a middle g ound be ween he
h ee ( igu e 119).
Adap ed om he a ea a e sal sc ip used o ou high
ideli y p o o ype, his sys em con ols he en y and exi
poin s o ou h ee a eas, allowing he use o a el om
one a ea o ano he . To achie e his, igge boxes we e
implemen ed and con olled by a sc ip ha cons an ly
calcula es he use ’s dis ance om he exi s – allowing o
a g adual scene ade-ou . These senso s de ec he use ’s
p oximi y and s a o ade he scene once he use ge s close
han 10 uni s o dis ance om ei he one o he h ee senso s.
Subsequen ly, he use ge s anspo ed o he nex a ea he
momen hey ouch he igge box.
Figu e 119: Top iew o A ea #2 E y h ina C is a-Galli, depic ing i s en y and exi poin s.
Bo om Righ : Snip o ou a ea a e sal sc ip , o a single a e sal ec angle, showing i s h ee
senso s (Objec s 1, 2 and 3), hei espec i e dis ances o he Use (came a), and he Alpha alue o
ou ade ansi ion.
A ea #2 – E y h ina C is a-Galli A ea #2 – E y h ina C is a-Galli
A ea #1 – Tilia x eu opaeaA ea #1 – Tilia x eu opaea
A ea #3 – Ficus Mac ophylla A ea #3 – Ficus Mac ophylla
Figu e 118: Top iew o he Bo anical Ga den illus a ing
all h ee a eas, oge he wi h hei en y and exi poin s.
Adap ed om Google Maps.
103
Figu e 120: Fou o he i e sma phones used o
ou i s eco ding a emp .
6.5 Sound Design
An expe imen al app oach was aken o achie e ou goal o
c ea ing a 3D ambien sound ep oduc ion o ou imme si e
expe ience, consis ing o ins alling mul iple mic ophones
wi hin each o he cap u ed a eas.
Be o e commencing any audio eco dings a he Bo anical
Ga den, a p elimina y audio es was pe o med in an e o
o de e mine i ou audio cap u e plan was in ac easible.
Two sma phones we e employed o his es : he Sony
XZ1 and XZ P emium, which we e placed on a lawn (in a
ga den), sepa a ed by oughly i e me e s. Each phone was
se o eco d a i s highes a ailable quali y (WAV o ma
wi h a 48kHz sample a e). A e s a ing bo h eco dings, a
loud clap signi ied he s a , o help synch onise and me ge
hese in o a single audio ile. The esul ing audio quali y was
decen ( ela i ely speaking), bu mos impo an ly we had
success ully c ea ed a binau al audio eco ding, by simula ing
sound as i i is being hea d li e (download link a ailable:
“Lawn 3D”, page 179).
Ha ing concluded his p elimina y es , we e u ned o
he Bo anical Ga den whe e we ins alled i e sma phones
( igu e 120) which go e enly dis ibu ed wi hin each
scene. Un o una ely, his equipmen go s olen jus a e
concluding he las se o eco dings, lea ing us wi hou ou
audio cap u e de ices and audio iles.
Adhe ing o ou ini ial plan, a second ound o audio cap u e
was done wi h a se o new mic ophones, including ones
o a highe calib e: he H1 Zoom, he H2n Zoom, ou Sony
a6300 came a and las ly, a Sony XZ1 sma phone ( igu e 121).
These de ices we e e enly dis ibu ed a ound each scene in a
ashion esembling a squa e ( igu e 122).
Figu e 121: The H2n Zoom, Sony a6300 came a,
Sony XZ1 sma phone and H1 Zoom – used o ou
inal audio eco dings.
Figu e 122: The posi ioning o ou equipmen wi hin he
scene con aining he Tilia, esembling a squa e.
104
Due o unp edic able wea he condi ions, windsc eens we e
made o hese mic ophones in a hope o minimise unwan ed
wind noise. Each a ea was eco ded (using a WAV o ma
wi h a 48kHz sample a e) o a pe iod o a ound 10 o 15
minu es. Howe e , be o e ini ialising ou audio eco dings,
mic ophone gain was ine uned o he H1 Zoom, H2n Zoom
and Sony a6300 o de e mine hei op imal signal- o-noise
a io ( igu e 123).
Ha ing success ully eco ded o all h ee o ou a eas,
all audio iles go impo ed in o P emie e P o, whe e he
synch onisa ion, clean-up and enhancemen p ocesses
we e ini ia ed. The synch onisa ion and clean-up o ou
clips was done by picking ou he peaks in he audio
acks ( ep esen ing he claps), ollowed by cu ing ou all
unwan ed sounds comp ised o human cha e , wind noise
and a ic sounds. The emo al o hese sounds was decided
due o hei o e ly dis ac ing na u e, whe e hese we e
p esumed oo in e e ing o he delica e cha ac e o he
a eas, especially in ega d o he a eas con aining he Tilia
x eu opaea and E y h ina C is a-Galli A c oss- ade audio
e ec was hen used as a bu e , o smoo h ou he audio cu s
be ween he sepa a ed (dele ed) clips. Nex , noise educ ion,
eble, bass and ampli ica ion adjus men s we e applied in
o de o uni y ou audio as a whole ( igu e 124).
Ul ima ely, all acks go impo ed in o Uni y. He e speake s
we e placed in he exac same posi ions as he mic ophones
had been placed in he eal wo ld ( igu es 125 and 126). Each
speake was hen ans o med in o a 3D sound emi e whe e
i s “spa ial blend” pa ame e along wi h i s min. and max.
emission dis ances we e adjus ed o c ea e ou desi ed 3D
audio e ec .
Figu e 123: Mic ophone gain adjus men s o
he H1 Zoom and H2n Zoom, oge he wi h hei
espec i e windsc eens.
Figu e 124: Audio edi ing wi hin P emie e P o,
showing he audio adjus men pa ame e s used in
an a emp o uni y all audio acks as a whole.
105
Figu e 126: Sc een cap u e depic ing he placemen o he 3D
sound emi ing speake s wi hin he i ual scene, in he exac same
posi ions as he mic ophones had been placed in he eal wo ld.
Figu e 125: Mic ophone placemen wi hin he a ea
su ounding he C is a-Galli – aking ad an age o
he ga den’s sh ubs as na u al wind-shields.
112
7.1 Vi ual Reali y Se up and De elopmen
Ou ques o implemen a Vi ual Reali y a ian ini ia ed
a he same ime as ou high ideli y p o o ype was being
de eloped, whe e he possibili y o unning ou imme si e
expe ience wi hin a VR headse was explo ed.
This began wi h he HTC Vi e (2016) ia S eam’s VR pla o m
(S eamVR). Ne e heless, a e impo ing ou poin cloud in o
Uni y ia Keiji o’s poin cloud impo e , ou p ojec ailed
o un – h owing an unexpec ed e o . This go esol ed
by changing Uni y’s g aphics API om Vulkan o Di ec X
11, howe e , his change also mean ha we we e ge ing a
conside able pe o mance educ ion o abou 20 ames pe
second. Besides his, s u e ing issues became appa en due
o high la ency as a consequence o Vi e’s cons an minimum
e esh a e o 90Hz (90 FPS) wi h a minimum la ency o
11.1 ms a a esolu ion o 3024 x 1680, o 1512 × 1680 pe eye.
A e del ing in o S eam VR’s se ings we we e able o sligh ly
imp o e un ime pe o mance by lowe ing he alue o he
“Th o ling and P edic ion” se ing, by o cing he ha dwa e
o un a a ixed slowe a e in o de o p o ide an o e all
smoo he expe ience.
In spi e o ou e o s, he high la ency issue pe sis ed,
esul ing in an o e all poo VR expe ience. As a las eso ,
ou ende me hod was swi ched o e o Uni y’s VFX g aph,
whe e ou ou pu poin s we e chosen o be ende ed as
p imi i es (quads wi h a ixed size). Ne e heless, only when
ou poin coun was educed o 2 million (o iginally 20
million) we e we able o achie e an accep able la ency a e,
his, howe e , d as ically educed he isual ideli y o ou
scene ( igu e 132). Fo his eason we decided o pu he HTC
Vi e aside o look o ano he solu ion.
Figu e 132: Sc een cap u e showing poo isual
ideli y, esul ing om he educ ion o he poin coun
o 2 million.

113
As a second a emp we es ed he Oculus Ques 2 (2020).
This equipmen seemed p omising due o i s highe
sc een esolu ion and wi eless acking capabili ies. To
ake ad an age o his headse ’s in e nal, And oid-based
ope a ing sys em, we ini ially ied building ou p ojec
as an app o And oid, as his would pe mi he launch o
ou imme si e expe ience wi elessly (as a s andalone).
Howe e , wi h his me hod we we e elying on he Ques ’s
Qualcomm Snapd agon XR2 chip o ende all o ou poin s.
Un o una ely he XR2 was only able o ende a meag e
200 housand poin s be o e unning in o pe o mance
issues. None heless, by connec ing he Ques 2 di ec ly o a
PCVR eady compu e , ia a Link cable, we e we able o ake
ad an age o ou highe pe o mance PC ha dwa e o ende
g aphics, ins ead o he s andalone mobile ha dwa e in he
headse . Mo eo e , his VR headse did no ha e any issues
unning o he p e iously p oblema ic Vulkan g aphics API,
meaning we had an addi ional pe o mance boos .
Se e al benchma ks we e pe o med using he Link cable
o he pu pose o de e mining he maximum quan i y o
poin s we could ende a a sus ainable ame a e. Al hough
no pe cep ible a i s glance (wi hin he headse ), we la e
disco e ed ha we we e ge ing low and a iable ame a es,
likely due o he ac ha we we e a emp ing o ende ou
scene a a esolu ion o 3616 x 1840 (o 1808 x 920 pe eye), a
a cons an e esh a e o 72 FPS ( he Ques ’s de aul e esh
a e). Fo his eason ou scene was changed o he E y h ina
C is a-Galli – a compu a ionally ligh e scene due o i s lowe
global poin coun .
An ini ial pe o mance op imisa ion was made by manually
lowe ing he poin coun wi hin he VFX g aph as a means
o eaching a s able ame a e o 72 FPS. Subsequen ly,
all anima ions go impo ed in o he scene ( oo sys ems,
bu e lies, unk, lowe and lea anima ions). This, howe e ,
nega i ely impac ed pe o mance once again, his ime
making ou scene luc ua e om 72 FPS o 36 FPS.
A mo e s uc u ed app oach was aken o y and s abilise
he ame a e, as lowe ing he poin coun u he would
only lead o a g ea e loss in isual ideli y. This app oach
consis ed o spli ing he poin cloud in o se e al sec ions
and subsec ions (g ound, scene ex emi ies, ee, ee’s
lea es, lowe s and scene bushes), as was done du ing
he de elopmen o he PC a ian o Kai os, allowing he
educ ion o ou scene’s poin coun o be made in a mo e
con olled ashion, by main aining mo e de ail (mo e poin s)
a he ocal poin o he scene and educing he poin coun a
he ex emi ies o he scene ( igu e 133).
Figu e 133: P ocess o sepa a ing he scene in o sec ions
and subsec ions in an e o o imp o e pe o mance
whils main aining as much de ail as possible.
114
Two op ions a ose when i came o use in e ac ion: o wo k
wi h he Ques ’s con olle s o expe imen wi h i s ecen ly
eleased hand acking capabili y, he la e being able o
de ec ou hands and inge s in eal- ime, h ough ou IR
(in a ed) came as loca ed on he Ques ’s headse , hen
displaying a model o hem wi hin he scene. This hand
acking sys em was explo ed i s , as i s new hand and inge
de ec ion sys em was eleased sho ly a e s a ing ou VR
de elopmen ( eleased on Ap il 19, 2022).
In o de o allow his sys em o wo k wi hin Uni y, he
download and impo o Uni y’s OVRPlaye Con olle was
necessa y, along wi h manually a aching i s hand p e abs o
bo h hands.
Ou goal consis ed o allowing he use o oam a ound
and in e ac wi h he scene solely using his hand acking
Figu e 134: Sc een cap u e o ou inal VR scene con aining 2.8 million poin s.
This, howe e , was no su icien o s abilise he ame a e
and a u he educ ion in he global poin coun had o be
done. F om he scenes o iginal poin coun o 20 million
poin s, ou inal VR a ea consis ed o only 2.8 million poin s.
As a las eso , he size o ou poin pa icles was inc eased
o add mo e olume o he scene in an a emp o coun e ac
i s low poin coun ( igu e 134).
115
ea u e, h ough ges u es and by a aching sphe e collide s
and a ac o s o ei he hand. We howe e we e only able o
success ully implemen he la e and had o ind ano he
solu ion o use locomo ion.
Ha ing ailed o mo e a ound he scene solely wi h
hand acking enabled, i was decided o ac i a e he
Ques ’s con olle s, along wi h modi ying he o icial
OVRPlaye Con olle sc ip o allow he use o “ ly” wi hin
he scene by poin ing in he desi ed di ec ion o ligh ia he
headse ’s o a ion/ il , and accele a ing ia he Le igge
bu on.
Ou nex ask consis ed o me ging he con olle ’s
unc ionali y wi h he hand acking sys em, un o una ely
he Ques 2 does no suppo he use o bo h inpu me hods
simul aneously. Fo his eason i was decided o keep
bo h a ian s, as sepa a e modes, whe e one can ly a ound
he scene using he con olle s, hen, by placing down he
con olle s, Uni y au oma ically swi ches o hand acking
whe e he use can hen in e ac wi h he scene.
Below is an illus a i e diag am o he con ols and inpu
me hods implemen ed o ou Ques 2 a ian .
116
The Righ con olle and Righ hand con ain a sphe e
collide and a ac o o in e ac wi h he scene ( igu es
135, 136, 137, 138 and 139), howe e , an issue a ose when
ying o ep esen bo h con olle meshes and hand meshes
du ing hese inpu ansi ions, whe e bo h meshes would
o e lap. Al hough impe ec , i was decided o p io i ise he
hand acking sys em o e he con olle s, by hiding hem.
Mo eo e , he eason behind adding only one sphe e collide
o he Righ con olle and hand was decided due o Uni y’s
incapabili y o pe o ming mul iple collide and a ac o
o ces ( o bo h Righ and Le ou pu s) on a single VFX
pa icle sys em. The only way a ound his issue consis ed
o spli ing each o ou VFX-based poin cloud pa icle
sys ems in hal (50% o he le hand and 50% o he igh
hand), his, howe e , esul ed in an une en a ac ion e ec
because only hal he pa icles would be a ac ed o ei he
hand, lea ing some pa icles una ec ed by hei collide and
a ac o o ces.
Figu e 135: Rep esen a ion o he Ques ’s hand
acking, oge he wi h he implemen ed sphe e
collide – highligh ed in ed.
Figu e 136: Demons a ion o he hand collision wi h he ee’s unk.
Figu e 137: Demons a ion o a bu e ly dispe sing as he use ies o ca ch i .
117
Figu e 138: Demons a ion o he lowe ’s “bu s ” e ec
upon hand collision.
Figu e 139: Demons a ion o he a ac ing o ce be ween he use and he ee’s ene gy.
In compa ison o he in e ac i e elemen s o he Desk op
(PC) a ian , an addi ion o ege a ion in e ac ion was
implemen ed in he VR a ian , allowing he use o
in e ac wi h he scene’s sh ubs, along wi h he lea es o
neighbou ing ees.
As a inal poin , all h ee loca ions we e o iginally planned
on being implemen ed o ou VR a ian , howe e , due o
pe o mance limi a ions, i was decided o solely include he
scene con aining he E y h ina C is a-Galli.

118
7.2 Pass h ough Explo a ion
In an e o o p oduce mo e con en o he Ques 2, i
was decided o explo e i s expe imen al pass h ough API.
Pass h ough is a mixed eali y- ocused ea u e ha allows
he use o see a eal- ime 3D isualisa ion o he physical
wo ld in he Oculus headse ia i s IR senso s. This ea u e
b ough up he idea o c ea ing an in-si u a ian consis ing o
a mixed eali y expe ience whe e he use could heo e ically
isualise he hidden p ocesses o he E y h ina C is a-Galli by
o e lapping ou poin clouds wi h he physical wo ld.
A p elimina y poin cloud cap u e es o ou esidency
was made be o e unning any in-si u es s. This consis ed
o ende ing ou poin cloud, ia Uni y’s VFX g aph, o e
he Ques ’s pass h ough s eam. A e co ec ing he scale,
o a ion and posi ion o ou poin cloud ( ela i e o he
physical wo ld), we we e able o c ea e an in e es ing
pa allelism be ween he eal and i ual wo lds, allowing
us o expe ience wo uni e ses simul aneously, so o speak
( igu e 140).
Figu e 140: POV o he Ques ’s pass h ough es ,
showing an o e lapped ep esen a ion o wo eali ies.
119
Figu e 141: In-si u equipmen se up.
An a emp o explo e he possibili ies o his idea was
conduc ed in-si u ( igu e 141), howe e , due o echnical
issues ela ed wi h he low po abili y and powe
equi emen s o ou se up, we we e no able o ca y ou
ou plan o ac ion wi hin he ime-span o his hesis.
Ne e heless, igu e 142 shows a simula ion o wha ou in-
si u a ian could ha e looked like, which is some hing ha
may be o in e es o u u e explo a ion.
120
Figu e 142: A simula ion o he in-si u a ian using he Ques 2 (POV), whe e one could
iew he scene’s hidden p ocesses, along wi h in e ac ing wi h said p ocesses h ough
physical con ac .
8. Public Feedback E alua ion
128
No a all
Pa icipan s
Figu e 154: How well Kai os
imp o ed ou olun ee s
app ecia ion o he na u al
wo ld.
G ea ly
These we e ollowed by a se o h ee mul i-choice linea
scale ques ions conce ning Kai os’s imp ession on ou
olun ee s. He e we asked how much he expe ience had
imp o ed hei app ecia ion o he na u al wo ld, i i made
hem eel any close o na u e, and i he expe ience had
inc eased hei desi e o explo e na u e in he eal wo ld.
On a scale o 1 ou o 10, he majo i y o ou olun ee s a ed
Kai os as ha ing imp o ed hei app ecia ion o he na u al
wo ld (a e age o 7.83), made hem eel close o na u e
(a e age o 8.55), and an inc ease in desi e o explo e na u e
in he eal wo ld (a e age o 8.27) ( igu es 154, 155 and 156).
No a all
Pa icipan s
Figu e 155: How well Kai os
made ou olun ee s eel close
o na u e.
Much close

129
Las bu no leas , a inal se o open ques ions we e gi en,
asking i any o he issues we e expe ienced du ing un- ime,
wha hey would add o he expe ience, ollowed by a sec ion
o commen s and/o sugges ions.
Some olun ee s exp essed hei desi e o mo e in e ac i e
elemen s wi hin he scenes, along wi h a guide (o a mini-
map) o show he use whe e o go, as some would become
los wi hin he a eas. Mo eo e , smalle sugges ions we e
also le , such as he addi ion o a blue sky and s a s.
No a all
Pa icipan s
Figu e 156: I Kai os inc eased
ou olun ee s desi e o explo e
na u e in eal li e.
Ve y much so
9. Making Kai os Publicly A ailable
132
Showcase ma e ial was p epa ed wi h he in en ion o
cap i a ing he public’s in e es and o be sha ed wi h he
Bo anical Ga den’s social media pla o ms and web pages.
This ask go ini ia ed h ough he de elopmen o a websi e
(h ps://kai os dei uc p /), whe e we explain he ideas and
mo i a ion behind ou p ojec as a b ie in oduc ion, igu es
depic ing how i go de eloped, wo download links o
bo h Desk op and VR a ian s ( igu e 157), along wi h ou
pape ha go accep ed a he In e na ional Con e ence on
En e ainmen Compu ing.
Figu e 157: Sc een cap u es o Kai os websi e. Top: on page. Bo om:
Download sec ion con aining links o bo h PC and VR a ian s.
A ailable a : h ps://kai os dei uc p /
133
In addi ion o his websi e, and due o he ha dwa e
equi emen s necessa y o un Kai os, we elec ed o c ea e
ideos consis ing o s e eoscopic (5K esolu ion) and
monoscopic (8K esolu ion) 360 deg ee ideos o ou
cap u ed a eas. The s e eoscopic a ian being aimed o he
Oculus Ques 2 and YouTube VR, and he monoscopic a ian
o be downloaded and iewed locally on a PC (a ailable a
h ps://kai os dei uc p /).
These ideos consis o wo minu e long passages h ough
ou cap u ed a eas, which we e c ea ed by se ing he scene
came as o ollow a p e-de e mined bézie cu e wi hin each
a ea ( igu e 158). To maximise he ou pu quali y o ou
ideos, i was decided o ende ou scenes as PNG image
sequences. This encompassed he use o Uni y’s in-buil
Reco de , whe e ou came a was se o ende a 360 deg ee
iew o i s su oundings, wi h an ou pu esolu ion o 8192
x 8192 pixels (s e eoscopic o e /unde o ma ), wi h a Cube
Map Size o 4096 pixels and a ame a e o 60 FPS ( igu e
159).
Figu e 158: Sc een cap u e wi hin Uni y, showing he came a’s p e-de e mined
pa h o 360 deg ee cap u e.

134
Figu e 159: The esul ing s e eoscopic (o e /
unde ) 360-deg ee image ende o he Tilia, wi h
a esolu ion o 8192 x 8192 pixels and a s e eo
sepa a ion o 0.065.
These image sequences we e hen impo ed in o P emie e
P o and ans o med in o ideos. In an a emp o gua an ee
smoo h ideo playback o he s e eoscopic a ian , o he
pla o ms men ioned p e iously, he H.265 ideo codec was
used a a bi a e o 120 Mbps. Highe bi a es we e es ed
in he e o o imp o e he isual quali y o ou ideos,
howe e , hese would cause ideo playback issues wi hin he
Ques 2 and YouTube.
Monoscopic 360 deg ee ideos we e subsequen ly ende ed,
using he same image sequences, bu his ime a a highe bi
a e o 400 Mbps. This was done o maximise ideo quali y,
howe e , hese a e aimed solely o local (Desk op) ideo
playback.
135
On a ela ed no e, a coope a ion was de eloped be ween
Kai os and he Depa men o In o ma ics Enginee ing’s
“PlugNPlay” display sys em. This ecen ly implemen ed
sys em consis s o an a ay o nine e ically s acked
and in e connec ed moni o s, p o iding he capaci y o
displaying con en a a maximum esolu ion o 9720 x 1920
pixels ( igu e 160).
The endea ou o execu ing Kai os on his sys em s emmed
om a mu ual cu iosi y o de e mine i Kai os could
success ully be un on his high esolu ion display a a s able
ame a e o 60 FPS ( igu e 161). Su p isingly his was he
case, which lead o a new ask: o c ea e a Kai os a ian as a
demo. o he inaugu a ion o PlugNPlay.
Figu e 160: PlugNPlay’s powe house, used o eed i s nine
displays, wi h he help o i s RTX 3080 Ti g aphics ca d.
Figu e 161: Ini ial es ing o Kai os on he PlugNPlay sys em.
A ew adjus men s needed o be made in o de o ully
ha ness he capabili ies o PlugNPlay, his included adjus ing
he scene’s came as FOV (Field O View), o encompass
he display’s wide aspec a io (app oxima ely 5:1) ( igu e
162), along wi h posi ioning mul iple s a iona y came as, a
di e en angles, wi hin each scene ( o alling wo pe scene).
A sc ip was hen added o au oma ically (o manually)
ansi ion be ween hese came as and scenes, simila o li e
wallpape backg ound ( igu e 163).
136
Las ly, a ease aile was c ea ed, consis ing o an in e laced
in e play be ween ideo oo age om he Bo anical Ga den
wi h i s digi al poin cloud ep esen a ion (a ailable a :
h ps://kai os dei uc p / ). A cus om audio ack, c ea ed by
Coimb a-based music a is Samuel Cadima, was also added,
which couples wi h a oice-o e , as a p elude o he na u al
p ocesses occu ing a he Bo anical Ga den o he Uni e si y
o Coimb a.
Figu e 162: P e- es ing Kai os PlugNPlay a ian wi hin Uni y, a a esolu ion o 9720 x 1920.
Figu e 163: Kai os unning on he PlugNPlay sys em.
10. Conclusion