Design explorations of interactive virtual environments using volumetric capture and visualisation techniques

1 1 Maximilian Oli e Rubin Design explo a ions o in e ac i e i ual en i onmen s using olume ic cap u e anD isualisa ion echniques A 250 yeA imes Amp o he bo AnicAl gA den o he uni e si y o coimb A Thesis in he con ex o he Mas e ’s in Design and Mul imedia, ad ised by P o esso s Jo ge Ca los dos San os Ca doso and Ped o Filipe Ma ins Ca alho and p esen ed o he Depa men o In o ma ics Enginee ing o he Facul y o Sciences and Technology o he Uni e si y o Coimb a. Sep embe 2022 Kai os A 250 yea imes amp o he Bo anical Ga den o he Uni e si y o Coimb a Maximilian Oli e Rubin Ad ised by P o esso s Jo ge Ca los dos San os Ca doso and Ped o Filipe Ma ins Ca alho Abs ac Cap u ing en i onmen s and momen s in ime has always been some hing humans ha e been ascina ed by – om pain ings o analogue pho og aphy, o digi al came as. Wi h he ecen de elopmen s o AR and VR echnologies, di ec olume ic cap u e o eali y has been gaining ac ion, wi h he po en ial o c ea ing imme si e, s a e-o - he-a isuals, blu ing he lines be ween he eal and i ual wo lds. Cap u ing and ansmi ing space i sel is one o he leading challenges in he ield o imme si e media. Cap u ing a eas such as si es o cul u al he i age is pa o his equa ion and has gene ally been achie ed ei he by 360-deg ee ideo o pho og aphy. Howe e , hese me hods limi he use ’s ull sense o dep h o he image. A solu ion o o e come his limi a ion consis s o cap u ing eal-wo ld da a h ough eali y cap u e me hods, esul ing in a se o da a poin s in space called a poin cloud. This poin cloud hen passes h ough a econs uc ion p ocess, ans o ming i in o a 3D polygonal mesh. Howe e , his p ocess limi s i s abili y o p o ide a deepe sense o imme sion and can in oduce undesi able in e pola ions o eali y. None heless, by using poin clouds as he basis o an imme si e en i onmen , we can explo e new isual languages and al e na e modes o pe cep ion. The objec i e o his hesis is he c ea ion o a poin cloud- based digi al imes amp o he Bo anical Ga den o he Uni e si y o Coimb a, by cap u ing eali y and hus a momen in ime ha can be un and isualised in 3D space ( h ough Vi ual Reali y), in eal- ime. This digi al imes amp consis s o h ee dis inc loca ions wi hin he Ga den, con aining cen u y-old ee specimens. These a eas we e digi ised h ough pho og amme y o c ea e poin clouds, which we e hen used as he basis o an imme si e audio- isual expe ience whose main goal was o con ey he ambiance o he Ga den, along wi h showing people i s na u al p ocesses usually hidden o he naked eye wi h he aim o ekindle ou ela ionship wi h na u e. This hesis esul ed in mul imedia con en consis ing o PC and VR based expe iences, and 360 deg ee ideos. Bo h PC and VR a ian s we e e alua ed by 22 olun ee s, esul ing in a posi i e esponse dis ibu ion o he use ’s expe ience in ela ion o hei ela ionship wi h he na u al wo ld. Keywo ds: Vi ual Reali y, Da a Cap u e, Poin Clouds, Pho og amme y, Imme si e Media. Resumo Cap u a ambien es e momen os oi algo que semp e ascinou o homem – desde pin u as, a o og a ia analógica, a câma as digi ais. Com os ecen es a anços em AR e VR, a cap u a olumé ica da ealidade em ganho acção, endo o po encial pa a c ia ambien es isuais ime si os, de ú ima ge ação, esba endo as linhas en e o mundo eal e o i ual. Cap u a e ansmi i o p óp io espaço é um dos p incipais desa ios na á ea dos meios ime si os. Nes e sen ido, a cap u a de á eas do pa imónio cul u al é um obje i o eco en e e em sido ge almen e ealizada que po ídeo de 360 g aus, que po o og a ia pano âmica. No en an o, es es mé odos na u almen e limi am a sensação o al de p o undidade po pa e do u ilizado . Uma solução pa a ul apassa es a limi ação consis e em cap u a dados do mundo eal a a és de mé odos de cap u a olumé ica, esul ando num conjun o de pon os de dados no espaço ambém conhecido po “nu em de pon os”. Es a nu em de pon os passa po um p ocesso de econs ução, ans o mando-a numa malha poligonal 3D. Con udo, es e p ocesso limi a a sua capacidade de p opo ciona um sen ido mais p o undo de ime são. No en an o, ao u iliza nu ens de pon os como base de um ambien e ime si o, podemos explo a no as linguagens isuais e modos al e na i os de pe cepção. O objec i o des a ese é a c iação de um imes amp digi al do Ja dim Bo ânico da Uni e sidade de Coimb a, a a és da cap ação da ealidade e, po an o, de um momen o no empo que pode se execu ado e isualizado em espaço 3D (a a és da Realidade Vi ual), em empo eal. Es e imes amp digi al consis e em ês locais dis in os do Ja dim con endo espécimes de á o es cen ená ias. Es as á eas o am digi alizadas a a és de écnicas de cap u a olumé ica pa a c ia nu ens de pon os, que o am u ilizadas como base de uma expe iência audio- isual ime si a cujo p incipal objec i o é cap a o ambien e do Ja dim jun amen e com mos a às pessoas os seus p ocessos na u ais no malmen e escondidos a olho nu com a inalidade de ea i a a nossa elação com a na u eza. Es a ese esul ou em con eúdo mul imédia cons i uído po uma expe iência baseada em PC e VR, e ídeos de 360 g aus. As a ian es PC e VR o am a aliadas po 22 olun á os, esul ando numa dis ibuição de espos a posi i a pa a as expe iências do u ilizado em elação à sua elação com o mundo na u al. Pala as-cha e: Realidade Vi ual, Cap u a de Dados, Nu ens de Pon os, Fo og ame ia, Media ime si a. 1. In oduc ion 2 The essence o obse ing, desc ibing and emembe ing in o ma ion and pe sonal expe iences has been wi h us o as long as we can emembe , and has been he e y basis o ou su i al as a species. A p ima y ins ance o his can be seen in he p ehis o ic pain ings and eng a ings on walls o ca es and ocks, which po ay he impo ance o ou human need o communica e [1]. This o m o communica ion le pe manen ma kings o he u u e, lea ing impo an messages o he gene a ions o come. To his day, he basis o e ec i e communica ion in he o m o isual p omp s hasn’ changed. We ha e only go be e a i and a e con inuously e ining and op imising he ways we sha e in o ma ion. Now when we hink abou how a we ha e come in ou abili y o exp ess ou hough s and ideas wi h he help o VR (Vi ual Reali y) echnology, we ha e almos eached he ipping poin o limi less pe sonal exp ession and o making he imagina ion explici . The desi e o show o ano he human being o o an audience he eal s a e o one’s imagina ion, pe spec i e and ideas is qui e an allu ing concep . Tu ning ideas, w i en and spoken wo ds in o some hing we can see, can be conside ed he pinnacle o pe sonal exp ession and ansmission o in o ma ion. The same can be said when someone ells us ha we “speak clea ly”, i ’s a isual me apho : hey “see” wha you mean. In o he wo ds, i would be like a new kind o communica ion, a o m o sha ing in o ma ion ha is simila o elepa hy whe e hough s and ideas can be sha ed isually. Technologies and p ojec s based on VR a e heading in his di ec ion and he ecen hype su ounding he Me a e se [2] is jus one o he many examples o p ojec s which will adically change he way we communica e and pe cei e eali y as we know i . Because o his, he e has been an in lux o new me hods and imp o ed accessibili y when i comes o cap u ing eali y o acili a e he eme gence o his upcoming digi al ealm [3][4][5]. Cheape cap u ing de ices, imp o ed da a p ocessing me hods and ad ancemen s in ha dwa e pe o mance (especially when i comes o GPUs (G aphics P ocessing Uni s)) a e allowing his ield o h i e, making i a ac i e o digi al a is s and con en c ea o s. Poin clouds a e he ounda ion o he p ocess o cap u ing eali y. A hei co e, poin clouds a e essen ially iles ha con ain a mul i ude o da a poin s in 3D space ha a e c ea ed by pe o ming a scan o an objec o a loca ion. 3 The mos popula o ms o a aining hese scans is by using ei he a LiDAR (Ligh De ec ion and Ranging) de ice, o pho og amme y ( he p ocess o ex ac ing 3D da a om digi al pho og aphs). Howe e , challenges s ill emain when i comes o ans o ming poin cloud da a, whe he i be om a lase scanne o om pho og amme ic cap u e, in o ealis ic digi al models. The con e sion o he measu ed da a in o a high quali y 3D polygonal su ace model is s ill a esou ce-hea y ask. To make ma e s wo se, poin clouds o en consis o incomple e, noisy and/o spa se da a, especially when cap u ing la ge ou doo scenes, which in u n u he nega i ely in luence he quali y o he inal model [6]. A small numbe o a is s in he isual e ec s and i ual imme si e ields ha e ecen ly app oached his challenge in a di e en way. They s a ed explo ing poin clouds in hei c ude o ma , aking ad an age o hei noisy na u e o disco e new aes he ics and new isual languages. Wi h his app oach, hey a e able o ake us on emo i e jou neys h ough digi al agmen s o eali y. Mos o hese p ojec s consis o p e- ende ed ideos while o he s consis o eal- ime imme si e VR expe iences. Howe e , he la e a e qui e demanding, ha dwa e-wise, usually ake place in sho li ed exhibi ions and ypically ollow a p ede e mined ou e wi hin hei i ual wo lds, hus limi ing he use s’ abili y o ull con ol wi hin he i ual se ing. This p ojec ’s main goal was he c ea ion o an imme si e isual ep esen a ion o a memo y o he Bo anical Ga den o he Uni e si y o Coimb a [7], by cap u ing eali y and hus a momen in ime ha can be un and isualised in 3D space ( h ough VR), in eal- ime. In addi ion o se ing as a digi al backup o he Ga den and commemo a ing i s 250 h anni e sa y, his p ojec allows anyone o access and expe ience his loca ion emo ely, e en in ace o es ic ions such as ones imposed by pandemics o physical limi a ions, e ec i ely allowing unlimi ed isi s o his si e o cul u al he i age [8]. This p ojec ocused on h ee dis inc a eas wi hin he Bo anical Ga den, each o which highligh s memo able cen ena y-old ee specimens: he Tilia x eu opaea [9], he E y h ina C is a-Galli [10], and he Ficus Mac ophylla [11]. He e we also simula ed some o he a ea’s na u al p ocesses usually hidden o he naked eye, wi h he aim o e oking a sense o inquisi i eness wi hin ou use s. 4 These na u al p ocesses we e simula ed o e ou poin clouds and consis o he a eas lo a and auna, including he ee’s ascula sys ems and unde g ound oo ne wo ks, ollowing he p emise ha i we can obse e he p ocesses o na u e in VR, we may become mo e conscious o i s inne wo kings in eal li e, and hus gain new le els o app ecia ion o he na u al wo ld. These h ee a eas a e i ually in e connec ed, simula ing snippe s o a memo y s oll h ough he Ga den, allowing use s o eely a el om one memo y o he nex . 1.1 Why poin clouds? One o he main conce ns behind VR is he challenge o de elop in e ac i e and imme si e wo lds ha eel as close o eali y as possible. Poin clouds a e ypically con e ed in o 3D polygonal meshes in an e o o c ea e a close eplica o wha was cap u ed, bu his some imes esul s in a “so edged” appea ance which does no appeal o he eye, especially a close p oximi y [6]. The ac o he ma e is, poin clouds a e he basis o a cap u ed su ace and hus can be conside ed he ab ic o ( i ual) eali y i sel , me apho ically speaking. By isualising poin clouds in hei c ude and di used s a e, ou ocus ge s aken away om he isual elemen ’s mass, whils main aining a su icien le el o su ace de ail o wha was cap u ed o ecogni ion, allowing ou mind o ocus mo e on he gene al a mosphe e and one o he locale. This opens new po en ials o explo e al e na e modes o pe cep ion and new isual languages. So a he han using compu e algo i hms o ill in missing su ace in o ma ion wi h polygons, which a e me ely an in e pola ed in e p e a ion o eali y, we can ake ad an age o ou b ain’s inna e abili y o ac i ely in e p e ou pe cei ed eali y by “ illing in he gaps”, ab ica ing he illusion o eali y, and hus allowing o a deepe , mo e au hen ic connec ion be ween he use and he i ual wo ld [12][13]. In o he wo ds, e en i some poin cloud da a is missing, i s di used and agmen ed cha ac e s ill has he capaci y o p o ide a g ea e sense o p esence and au hen ici y compa ed o 3D polygon meshes. Mo eo e , his o m o isualising da a opens new ga es o explo e he way ou memo ies and d eams wo k, emula ing he way ou subconscious mind s o es and eplays da a. Simila o poin clouds, d eams end o be qui e incomple e, uzzy and some imes unclea , ye we o en seem o emembe he gene al eelings hey lea e. Tha ’s he eal goal he e, o be able o ec ea e he ambiance o a momen los in ime. 5 1.2 Mo i a ion Se e al ac o s in luenced he de elopmen o his p ojec . Fi s , in addi ion o se ing as a digi al backup o he Ga den and commemo a ing i s 250 h anni e sa y, his p ojec allows anyone o access and expe ience his loca ion emo ely, allowing unlimi ed isi s o his si e o cul u al he i age. Second, as a echnology-d i en species, we ha e been slowly losing ouch wi h na u e and ha e become mo e dis an o i s li e-gi ing bene i s [14]. This p ojec aims o igge ou inna e “biophilia” [15] (a e m used o desc ibe ou a ilia ion wi h mo he na u e, i s posi i e impac on ou men al heal h and ou o e all well-being), by ekindling ou connec ion o he na u al wo ld, no as a ool o subs i u e na u e and eali y i sel , bu a he as a ool o enhance i . Thi d, by u ilising and explo ing he po en ial o he la es eali y cap u e and poin cloud isualisa ion echniques, his p ojec seeks o e o lessly ansmi he eelings o a momen cap u ed in ime h ough con empo a y echnology. Simila o a pho og aph, bu in his case a ype o pho og aph we can “jump in o”, so o speak. Allowing us o explo e he in icacies o an al eady geome ically complex scene y. Las ly, he conse a ion o he Bo anical Ga den is some hing ha he Uni e si y o Coimb a is wo king on and his has become a con inuous endea ou . Belonging o one o UNESCO’s Wo ld He i age Si es [16], i is pa amoun ha he Bo anical Ga den p ese es i s in eg i y and essence. Na u al disas e s such as s o ms and he ini e li e-span o plan species a e ine i able and ha e un o una ely aken ou some his o ical ees and caused s uc u al damage o e he yea s. Fo ins ance, he iconic Washing onia obus a palm ee, which a he ime was he alles plan in he Ga den, go obli e a ed by a ligh ning s ike in May o 1986 [17], no o men ion he ecen hunde s o m “Leslie” in 2018, which caused a lo o s uc u al damage o he Ga den [18] ( igu es 1, 3 and 4). Fu he mo e, he e a e also signs o delibe a e ac s o degene acy by he public, such as ca ings in he neoclassical Cen al Squa e ( igu e 2). Al hough hese examples don’ all belong o he h ee chosen a eas o cap u e, hese ins ances a e jus o demons a e how exposed he Ga den is o ex e nal o ces. Tha being he case, his p ojec will also unc ion as a digi al imes amp, a so o backup, by digi ally p ese ing he a eas wi hin he Ga den, wi h emphasis on one o i s oldes ees: he E y h ina C is a-Galli, which is cu en ly in a s uc u ally ulne able s a e ( igu e 5). 6 Figu e 1. Figu e 3. Figu e 2. Figu e 4. Figu e 5. Figu e 1: A missing Tilia ee in Alameda das Tílias, 05/11/21; Figu e 2: Ca ings in he Cen al Squa e ma ble, 05/11/21; Figu e 3: Polypo e mush ooms g owing on he base o a che y ee in Cen al Squa e, indica ing unk o , 05/11/21; Figu e 4: Damage o he ence’s neoclassical o namen ; Figu e 5: Cu en s a e o he E y h ina C is a-Galli, 05/06/22. 7 1.3 Goals and Resul s This p ojec ’s main goal consis ed o c ea ing a digi al ep esen a ion o he Bo anical Ga den o he Uni e si y o Coimb a using poin cloud-based cap u e echniques, by cap u ing eali y and hus a momen in ime ha could hen be un and isualised in 3D space, in eal- ime. Fu he mo e, by wo king in a digi al en i onmen , all cons ain s p esen in he eal wo ld we e unleashed, meaning ha he sky was he limi when i came o how we could manipula e and p esen hese da a. This lead o ano he goal which consis ed o enhancing he sensa ion o being p esen in he Ga den, along wi h e oking eelings o childlike wonde by senso y s imula ion ia he isual and audi o y sys ems. Mo eo e , scien i ic in o ma ion on he cha ac e is ics o each a ea we e ga he ed o he end goal o simula ing some o hei na u al p ocesses. To u he inc ease i s imme si e ac o , a VR compa ible e sion o his p ojec was also p oduced. Las bu no leas , his wo k esul ed in a publica ion accep ed by he In e na ional Con e ence on En e ainmen Compu ing, in he ca ego y o Wo ks-in-P og ess pape s, ha will be p esen ed in No embe 2022 in B emen, Ge many. This submission will also be conside ed o he con e ence’s S uden Compe i ion. This imme si e expe ience o he Bo anical Ga den was he end p oduc o his hesis, which was made publicly a ailable o PC, a VR headse , along wi h 360-deg ee ideos o be iewed wi hin a VR headse o h ough online ideo s eaming pla o ms. Albei no ha ing he same imme si e po en ial as he VR headse a ian , he 360-deg ee ideos we e c ea ed o imp o e audience each. Rega ding he VR a ian , a a he op imis ic ye easible goal was explo ed, which consis ed o unning he imme si e expe ience wi hin he Bo anical Ga den, ha is, he c ea ion o an in- si u ins alla ion, whe e ou audience could u he indulge hemsel es in he Ga den. Fi s and o emos , o be able o achie e hese goals, a ew s eps we e aken in o de o b ing he mos ou o his expe ience and o enhance he sensa ion o being p esen in he Ga den. These s eps consis ed o cap u ing as much da a as possible in o de o s imula e as many senses as possible wi hin he digi al en i onmen . In his case, hese encompassed wo main a eas: he isual and audi o y. To be able o acqui e hese da a, pho og amme ic cap u e was used o eco d he isuals whils a se o mic ophones we e used o audio cap u e o he goal o c ea ing a 3D ambien sound e ec . 8 These acqui ed da a we e hen cleaned, p ocessed and e ined o he inal digi al en i onmen . These da a we e hen composed wi hin Uni y’s eal- ime game engine, whe e u he isual explo a ion was done o see how hese da a could be p esen ed and enhanced o c ea e a cap i a ing and imme si e d eam-like expe ience. Nex , in o de o gua an ee a quali y expe ience, pe o mance and s abili y es s we e un o ensu e ha no issues occu ed such as low ame a es o isual anomalies, as his would ha e nega i ely impac ed ou desi ed ou come. As a inal poin , a se o use e alua ion es s we e un o de e mine i ou imme si e expe ience ul illed ou goals. 1.4 Scope Na u ally, some bounda ies we e se in o de o achie e he bes possible ou come wi h he limi ed esou ces a ailable. This sec ion p esen s some conside a ions on he p ojec ’s scope. A pa o his p ojec was dedica ed o emula e he way we expe ience d eam-like s a es and memo ies h ough he exp essi e isual po en ial o poin clouds. Howe e , we only sc a ched he su ace o his ma e and did no del e in o he ine de ails o Onei ology ( he scien i ic s udy o d eams), Neu oscience and/o Cogni i e Psychology. Al hough ou ini ial plan was o cap u e mul iple loca ions wi hin he Bo anical Ga den, he shee size o he Ga den had o be aken in o accoun due o ime cons ain s and echnical limi a ions. F om i s hi een hec a es o g ound, h ee dis inc a eas o in e es we e selec ed con aining he ollowing cen u y-old ees: he Tilia x eu opaea, he E y h ina C is a-Galli, and he Ficus Mac ophylla. Las ly, due o he wide- anging ield o da a cap u e and sensing echnologies, and o he sake o his hesis, we mos ly discussed pho og amme y, as i is he basis o he p ac ical componen o his hesis, and did no add ess he deepe scien i ic and ma hema ical componen s o emo e sensing. 9 1.5 Documen S uc u e This documen is s uc u ed as ollows: In oduc ion, In oduc ion o Cap u ing Reali y, Rela ed Wo k, Me hodology and Wo k Plan, P ojec De elopmen , E alua ion and las ly, Conclusion. In he i s chap e , an in oduc ion o he ield o s udy and mo i a ions behind he hesis a e p esen ed, as well as i s goals, objec i es and scope. The second chap e , In oduc ion o Cap u ing Reali y, se es o p o ide a deepe unde s anding o he ield o cap u ing eal-wo ld da a by laying he ounda ions o his ma e . This includes i s his o ical backg ound, i s me hodologies, cu en challenges and limi a ions, ollowed by he la es con empo a y me hods and de ices used in his ield. The hi d chap e p esen s ela ed wo k and p ojec s o p o ide a be e unde s anding o wha has al eady been achie ed in he ield o using poin cloud da a in he con ex o imme si e expe iences. In his chap e , each case s udy is analysed o i s wo k low, mo i a ions and inspi a ion, highligh ing pe inen in o ma ion ha aided he de elopmen o he p ac ical wo k ea u ed in his hesis. The ou h chap e con ains he Me hodology and Wo k Plan, which de ined he s uc u e o he de elopmen o he p ac ical componen o his hesis, ollowed by a p ojec ime-line. This chap e se ed o gua an ee a s able sys ema ic wo k low h oughou he de elopmen o his hesis, including he me hods and p ocedu es ha we e used o bes achie e ou desi ed goals and objec i es. The i h, six h and se en h chap e s encompass all de ails ega ding he de elopmen o he p ac ical componen s o his hesis. This includes a small in oduc ion o he si e o cap u e: he Bo anical Ga den o he Uni e si y o Coimb a, ollowed by a sec ion o p elimina y expe imen s which we e conduc ed o ge accus omed o he pho og amme ic p ocessing pipeline. Nex , we ho oughly desc ibe he de elopmen o Kai os, including all o i s a ian s, oge he wi h all he s eps ha we e aken o achie e he inal imme si e expe ience. Chap e eigh , E alua ion, pe ains o he es ing phase o ou p ojec whe e we ga he ed and analysed use eedback o ou imme si e expe ience. 16 g ea conce n. Apple ecen ly in oduced LiDAR scanning capabili ies in i s se ies o mobile phones and some able s, hough no as accu a e as p o essional LiDAR de ices, hese s ill open up many in e es ing applica ions o hand-held scanning and a e g ea ly imp o ing audience each and ecogni ion [31]. Figu e 10: Top Le : An illus a ion o a e es ial lase scanning sys em (TLS) in he con ex o a o es . A ailable a : «h ps://a s els-cdn com/con en /image/1-s2 0-S0924271616000204- g 1_l g jpg». Top Righ : Pho og aph o a well known e es ial lase scanne – he Leica P20 (manu ac u ed in 2014). A ailable a : «h ps://www. esea chga e.ne /p o ile/Jim-Chandle -2/ publica ion/337849886/ igu e/ ig1/AS:836700154654724@1576496270365/Leica-P20- e es ial- lase -scanne -wi h-1-HDS- a ge -used- o - egis a ion-and png». Bo om: An example o a poin cloud esul ing om a TLS scan o a public pa k. A ailable a : «h ps://i pinimg com/o iginals/ de/18/b /de18b 1c9b78e4b52981165b47796 3c jpg» 17 2.3 Pho og amme y Pho og amme y (Pho os = ligh ; G amma = w i ing; Me on = measu e) is de ined by he Ame ican Socie y o Pho og amme y and Remo e Sensing as, “The a , science and echnology o ob aining eliable in o ma ion abou physical objec s and he en i onmen h ough p ocesses o eco ding, measu ing and in e p e ing pho og aphic images and pa e ns o elec omagne ic adian ene gy and o he phenomena” [32]. To ecapi ula e and o ully quan i y he p og ession o pho og amme y h oughou his o y, his ield o s udy can be sepa a ed in o ou gene al s ages based on imp o emen s in echnology. The i s gene a ion o pho og amme y s a ed wi h he in en ion o pho og aphy, which ini ia ed he in es iga ion and expe imen a ion phase o emo e sensing. The second gene a ion, conside ed he e a o analogue pho og amme y, b ough ae ial su eying and s e eo plo ing echniques ha we e es ablished and used du ing he wo Wo ld Wa s, u ning pho og amme y in o a epu able and popula su eying and mapping me hod. The ma hema ical heo y was well known a his s age, bu he lack o compu a ional powe hinde ed nume ical solu ions. The hi d gene a ion, also known as he analy ical pho og amme y gene a ion, began in he 1950’s du ing he up ising o he compu e , which allowed o signi ican ad ances o pho og amme ic de elopmen . Hellmu H. Schmid was one o he i s pho og amme is s who ook ad an age o he compu e , de eloping he basis o analy ical pho og amme y using ma ix algeb a. Block adjus men p og ams (a se o ools designed o imp o e he geo e e encing accu acy) and ae ial iangula ion ( he p ocess o de e mining he co ec posi ion and o ien a ion o each image in a se ies o ae ial images) we e subsequen ly in oduced in he la e six ies ha g ea ly imp o ed ae ial iangula ion accu acy. We a e now in he ou h gene a ion o pho og amme y. This began wi h he ansi ion om analogue o digi al, whe e digi al images a e used ins ead o ae ial pho og aphs. Pho og amme ic p ocesses a e now managed by compu e so wa e, p o iding huge cos and ime sa ings in p oducing ypical pho og amme ic ou pu s [33][34]. 18 2.3.1 The Pho og amme ic P ocessing Pipeline The wide a ailabili y o digi al came as and he ecen p og ess in compu e ision has led o he ise o accessible 3D modelling solu ions. Pho og amme ic cap u e na u ally s a s wi h he cap u e o eal wo ld da a h ough pho og aphs o ideo (as an image sequence). This cap u ed da a hen ge s ed in o he pho og amme ic p ocessing pipeline ( igu e 11), consis ing o a ew s ages ha need o be ollowed in o de o p oduce a success ul digi al asse . This pipeline gene ally consis s o he ollowing s ages: came a alignmen , gene a ing dense cloud, build mesh, and build ex u e. Du ing he ini ial s age, came a alignmen , he pho og amme y so wa e akes as inpu a se o images o ideo ames and hei associa ed came a calib a ion in o ma ion (i a ailable). I hen scans h ough e e y ame and sea ches o ea u es. Fea u e de ec ion de e mines salien poin s ( ea u es) in he images o ideo ames. These ypically consis o dis inc i e a ibu es such as pa e ns, s ong lines and edge poin s. These in u n help es ima e he ollowing posi ioning o each came a ( ame) wi hin a common coo dina e sys em which is ela ed o he wo ld coo dina e sys em. I is usually desi able o eliably de ec hese poin s a di e en iewpoin s and adial dis ances o ensu e ha he same salien ea u es a e de ec ed in di e en ames. A e he ea u e poin s ha e been ecognized in se e al images, hey a e in e nally alida ed and sc eened o alse co espondences in a s ep called geome ic alida ion. The alida ed ea u es a e hen co ela ed wi hin e e y ame o ge pu a i e ma ches which a e hen used o es ima e he posi ion o e e y came a ame wi hin he scene [21]. As a esul , a spa se poin cloud and a se o came a posi ions a e o med [35]. No e ha his spa se poin cloud is me ely a e e ence o he esul ing came a alignmen s and only a ac ion o he o al da a. In o he wo ds, a “ alida ion s ep” be o e s a ing any hea y du y compu a ional wo k. Du ing he second phase, gene a ing dense cloud, 3D coo dina es o he su ace poin s a e es ima ed based on he ou pu o he p e ious phase. The line o sigh om e e y came a o he objec is econs uc ed in o wha is called a ay cloud. The in e sec ion o many ays de e mines he inal 3D coo dina es o he objec [36]. Once his has been comple ed, a dense poin cloud is gene a ed whe e each ma chable pixel ge s i s own x, y, z coo dina es in 3D space [37] ( igu e 12). The hi d phase, build mesh, consis s o gene a ing he su ace o he scene o objec based on he p e iously p oduced dense 19 poin cloud. This s ep connec s each se o h ee adjacen poin s in o a iangula ace, esul ing in a con inuous mesh o e he su ace o he model. In he inal s ep, build ex u e, o iginal images a e combined in o a ex u e map and p ojec ed on o he mesh, esul ing in a pho o- ealis ic digi al ep esen a ion o he o iginally cap u ed objec . Figu e 11: Le : The basis o he pho og amme ic pipeline. Righ : The p incipal p ocedu es o pho og amme y. The do ed line indica es he connec ion be ween image ea u es Pn,m, ex e nal image o ien a ions Rn, n, and he ie poin P1. A ailable a : «h ps://www esea chga e.ne / igu e/ The-p incipal-p ocedu es-o - close- ange-pho og amme y- The-do ed-line-indica es- he_ ig2_347828301» Figu e 12: Pho og amme ic cap u e o he Th ee G aces by An onio Cano a, illus a ing he esul ing dense poin cloud and came a alignmen s. A ailable a : «h ps://www ac um-a e com/ esou ces/images/ a/ echnology/ pho g amme y/cap u e3 jpg» 20 2.4 Challenges and Cu en Limi a ions Pho og amme y While many imp o emen s ha e been made in he ield o pho og amme y o e he yea s, some hu dles s ill pe sis and i is impo an o be awa e o i s challenges and limi a ions o be able o assess each and e e y si ua ion acco dingly. Many o he algo i hms used o pho og amme ic p ocessing all sho in he econs uc ion phase (build mesh), esul ing in a ailed o p oblema ic 3D mesh. Mos pho og amme ic p ocessing so wa e p o ide ho ough documen a ion o help ci cum en ailed o p oblema ic meshes, howe e , hese ollowing limi a ions should be conside ed in all con ex s. 2.4.1 Occlusions and Numbe o Pho og aphs Necessa y While no so common in indoo cap u e s udios, occlusions a e a common issue when cap u ing ou doo objec s o scenes. An occlusion is when an unwan ed objec ge s in be ween he came a and he a ge objec , esul ing in missing da a. Howe e , his can be easily o e come by aking mo e images om di e en angles, p o iding su icien da a o he p ocessing algo i hm o compensa e o he occluded ame. Fo hea y occlusions, cap u ing images e e y 10º is ecommended. Fo objec s wi h minimal occlusion, images aken e e y 20º is su icien [20]. As a ule o humb, when in doub , ake mo e pho og aphs. 2.4.2 Fea u e De ec ion Du ing he came a alignmen p ocess, he algo i hm sea ches o ea u e poin s wi hin mul iple images. Fo example, aking images o blank walls o ha ing un ocused sho s will mos likely cause he algo i hm o ail. The mo e acking ea u es he scene o objec has, such as pa e ns and/o s ong dis inc i e a ibu es, he be e he esul s. In some cases acking ma ke s can be added o he objec o yield be e esul s. 2.4.3 Re lec i e, T anspa en o Glossy Su aces Re lec i e, anspa en and glossy su aces a e pho og amme y’s bigges conce n. Objec s o his kind ha e 21 li le o no su ace de ail o he so wa e o pick up on. The e is cu en ly no way a ound his o he han o coa hese e lec i e su aces wi h some hing ha will scan, his howe e is no a g ea solu ion and qui e imp ac ical, especially when i comes o la ge scenes con aining buildings wi h a lo o glass, o example. 2.4.4 En i onmen al Condi ions Pho og amme ic cap u e is hea ily dependen on wea he condi ions. Because o his and in o de o imp o e he success ul a e o a cap u e, we should be ully awa e o he ollowing en i onmen al ac o s: su ace mo emen due o wind o o he o ces, as his may dec ease he abili y o he p ocessing algo i hm o ma ch scene poin s, esul ing in a e y noisy o incomple e poin cloud. Excessi e a ia ion in ambien ligh ing du ing image collec ion, caused by changing cloud pa e ns, o example, is also some hing we should look ou o [38]. Though no as se e e as cons an su ace mo emen , changing in ligh ing condi ions will impac he ecogni ion o ea u es. LiDAR 2.4.5 Re lec i i y Limi a ions A no able d awback o LiDAR is ha i does no wo k well in scenes wi h a lo o ligh e lec ions. LiDAR is also un eliable when used o cap u e non e lec i e su aces due o he ac ha his echnology elies on he p inciple o ligh e lec ion [39]. 2.4.6 En i onmen al Condi ions Since LiDAR uses isible lase s o measu e dis ance, his echnology is o limi ed use in ha sh en i onmen s, especially in ce ain wea he condi ions such as og, ain, snow and/o hail, making i essen ially “blind” in hese condi ions. 2.4.7 Da a P ocessing Requi emen Since LiDAR can co e la ge a eas a a as a e, i ends o gene a e a lo o da a. Meaning adequa e equipmen is c ucial o be able o p ocess and in e p e he la ge amoun s o da a i is able o cap u e [39]. 22 2.5 Con empo a y Da a Cap u e Today, da a cap u e has become one o he main ields o in e es su ounding he c ea ion o ealis ic digi al ep esen a ions o he eal wo ld, ega dless o he echnology chosen o do so. When we ask ou sel es wha conce ns we may ha e when i comes o cap u ing and ep esen ing eali y, some ques ions may a ise such as: “how should I cap u e his scene o objec ?” O , “wha me hods should I use o achie e he bes esul s possible?” Fi s and o emos , we ha e o be clea on wha we wan o cap u e, in o de o de e mine how we should p oceed o cap u e hese da a as an asse o as a scene o be used in VR. Cap u ing single objec s has gene ally been accomplished wi h ease wi hin s udios using mul i-came a sys ems. Mul iple came as a e placed a ound he objec , ully encompassing i . Gene ally, he highe he came a coun , he be e , as mo e da a will be a ailable o eed in o pho og amme ic algo i hms ha in u n will p oduce i s digi al ep esen a ion. Fu he mo e, being in a con olled en i onmen (e.g. s udio), gene ally allows o a highe quali y cap u e, such as ha ing ull con ol o e global ligh ing – a key ac o o a success ul cap u e. The mos p ominen s a e-o - he-a example o pho og amme ic cap u e can be seen in Wol Digi al Wo ld’s sphe ical-shaped chambe ( igu e 13). Wi h a diame e and heigh o 6 and 7 me es, espec i ely, and con aining 200 Sony A7RIVA came as, i can cu en ly be conside ed he wo ld’s mos ad anced indoo eali y scanne . On he o he hand, Leica Geosys ems ha e ecen ly announced hei s a e-o - he-a BLK se ies au onomous LiDAR scanne s: he BLK2FLY, BLK ARC and he BLK2GO ( igu e 14), p o iding easy da a cap u e solu ions o a wide ange o se ings. The Leica BLK2FLY is a d one-s yle lase scanne wi h ad anced obs acle a oidance. I c ea es poin clouds du ing ligh and was designed o cap u e building ex e io s, s uc u es and he en i onmen . This de ice can be se o au onomously cap u e di icul - o-access building loca ions, such as oo ops and acades. The Leica BLK ARC was designed o in eg a e wi h obo ic ca ie s o p o ide au onomous mobile lase scanning wi h minimal o no human in e en ion. Use s a e able o plan a scanning pa h and hen launch he BLK ARC o scan he desi ed a ea, au onomously. 23 Las bu no leas , he BLK2GO is a po able hand-held imaging lase scanne ha ec ea es 3D spaces on he go. I is able o cap u e images and accu a e poin clouds in eal ime and uses SLAM (simul aneous localisa ion and mapping) echnology o eco d i s pa h h ough space. The BLK2GO au oma ically cons uc s poin clouds om he momen you s a he scanning session o when you u n o he de ice [40]. All images and 3D da a ge me ged and aligned au oma ically, making his a big imp o emen o he no o ious pos manual poin cloud egis a ion and alignmen p ocess – a common s ep which was necessa y in p e ious gene a ion lase scanne s. I is clea ha eali y cap u e o e s unique oppo uni ies ac oss a ange o indus ies and sec o s, and echnology is ad ancing apidly. The con inued implemen a ion o au oma ed p ocedu es o analysing and ex ac ing in o ma ion om 3D poin clouds will con inue o p o ide e sa ili y, ease o use and lowe cos . This will make LiDAR and pho og amme y a ailable o a wide ange o use cases, including non- adi ional use s. Ad ances in compu e ision, AI and machine lea ning a e al eady adding new dimensions o cap u ing eali y. Despi e he ecen imp o emen s in hese ields, we a e s ill in he ea ly s ages o compu a ional and so wa e-based me hods ha will o m he basis o u u e eali y cap u e. Figu e 13: Wol Digi al Wo ld’s sphe ical- shaped pho og amme ic cap u e chambe . A ailable a : «h ps:// me ahe o io/» 24 Figu e 14: Leicas BLK ARC, BLK2GO, BLK2FLY, om le o igh , espec i ely. A ailable a : «h ps://blk2021.com/wp-con en /uploads/2021/09/ARC_2GO_2FLY.jpg» 3. Rela ed Wo k: Imme si e Expe iences using Poin Cloud Da a 32 Figu e 18: S ill om In he Eyes o he Animal, specula ing he way mosqui os see he en i onmen . A ailable in «h ps://www. lick .com/pho os/ma shmallowlase eas /» Figu e 19: Pho og aph o he in-si u ins alla ion o In he Eyes o he Animal showing he cus om VR helme s along wi h he ac ile subpack elemen s. A ailable in «h ps://www. lick .com/pho os/ ma shmallowlase eas /» 33 3.4 P omenade C ea ed by I alian isual a is Da ide Quayola, P omenade [46] akes a mo e syn he ic app oach o cap u ing na u e. This p ojec eme ged om a commission by he Swiss manu ac u e o luxu y mechanical wa ches Audema s Pigue . Mo eo e , he ela ionship be ween Quayola’s wo k and Audema s Pigue is ela ed o hei in e play be ween adi ion and expe imen a ion. In o he wo ds, i is a balance be ween he mas e e isi ing he adi ion while a he same ime e isi ing i and ein e p e ing i in a new way. P omenade is a sho ilm ha p esen s a high quali y bi d’s- eye iew o he o es s o he Vallée de Joux in Swi ze land, nea he b and’s o iginal wa chmaking acili y. Quayola used high p ecision LiDAR scanne s o collec hese da a, hen ansla ed he o es in o a wa e o in e connec ed oo age, ans o ming he documen ed landscape in o digi al a . A lo o Quayola’s wo k is cen ed a ound explo ing di e en ways o seeing and disco e ing new isual languages wi h he help o echnology, equi alen o s udying hings wi h “ he eyes o machines”. Quayola himsel s a ed ha he is no pa icula ly in e es ed in he objec he is ying o cap u e, he is mo e in e es ed in he ension be ween he old language behind i and he new language he helps disco e . Al hough Quayola uses high-p ecision LiDAR scanne s, he s ill enjoys he small impe ec ions i b ings du ing he ansla ion p ocess ( he ans o ma ion o he o es in o he digi al) ( igu e 20). “They (LiDAR scanne s) canno see pe ec ly. Tha ascina es me. I you imagine an imp essionis pain ing, you a e educing he esolu ion. You a e no aiming o pe ec ion. Limi a ion b ings ou he exp ession.” – Quayola [47]. Fu he mo e, P omenade plays wi h he me ging o he eal wo ld wi h echnology. Du ing he cou se o he ideo, di e en isual e ec s al e na e be ween he eal and he digi al, gi ing us a peculia hyb id machine-like iew o he o es . To u he enhance hese epheme al g aphics, a ca e ully c a ed sound ack was added and synch onised wi h hese isual e ec s. Quayola’s bold explo a ion o hese new isual languages push bounda ies in he way we can ex ac and isualise eal wo ld da a in he con ex o na u e, by p ocessing isual in o ma ion wi h a di e en logic. E en hough P omenade has s onge “ echy” isuals and a “ ossil s yle” colou pale e 34 ( igu e 21), compa ed o he mo e na u al o ganic na u e o he desi ed ou come o he p ac ical componen o his hesis, i ’s accu a ely designed sound ack demons a es jus how impo an audio is o ge a mo e angible expe ience om wi hin he digi al en i onmen . Figu e 20: Rende om P omenade, no e how well LiDAR was able o cap u e he in icacies o he o es . A ailable in «h ps://whi ewall a /li es yle/quayola-cap u es- allee-de-joux-audema s-pigue » Figu e 21: S ill om P omenade, illus a ing i s “ echy” isual language. A ailable in «h ps://www audema spigue com/com/zh-han /news/a /quayola-p omenade h ml» 35 3.5 Clams Casinos’ Moon T ip Radio The se ies o ideos used o accompany Moon T ip Radio’s album [48] is ano he g ea example o poin cloud da a used in a c ea i e way. De eloped by P ague based compu a ional a is Jakub Val a , his cus om made isualise akes a mo e psychedelic app oach o olume ic da a ep esen a ion. Based on d one-cap u ed oo age and pho og amme y, he o he wo ldly isual language chosen o his p e- ende ed ideos accompany Casinos’ sound acks and a e eminiscen o he bioluminescen co als seen in he opics. This is e iden in he i s song o he album: Clams Casino – Rune ( igu e 22), which akes you on a jou ney h ough wha seems like a moun ainous e ain composed o high con as ing luo escen poin clouds. As opposed o he p e ious ela ed p ojec s, his one p oceeded o ollow-up and conclude he econs uc ion phase o he pho og amme ic p ocessing pipeline, ha is, econs uc ing he aces (mesh) o he cap u ed scene. Though no so ob ious a i s glance, once we ake a close look we can see ha he luo escen poin clouds a e in ac es ing on a da k mesh. This mus ha e been decided due o he chosen high con as ing isual language, whe e a loa ing poin cloud would pe haps ende in a e y noisy and unclea manne due o poin clouds inhe en “see h ough” cha ac e . Likewise o Quayola’s accu a ely designed sound ack, he syne gy o Jakub’s ideos wi h Casinos’ music ascend ou senso y expe ience o ano he le el. Fo example, i we play Casinos’ in ica e sonics wi hou any isual s imuli, o play Jakub’s ideos wi hou any music, we lose hal o hei exp essi e po en ial in bo h cases. Jakub’s ideos, P omenade and especially In he Eyes o he Animal each us ha in o de o ake ull ad an age o any gi en imme si e expe ience, we should place a s ong ocus on s imula ing as many o ou senses as easible, because highe imme si e quali y le els elici highe le els o p esence, which in u n inc eases he e ec i eness o he o e all expe ience. 36 Figu e 22: S ill om Clams Casino – Rune, showing he da k moun ainous e ain composed o luo escen poin clouds. A ailable in «h ps://www.you ube.com/wa ch? =sWeG68lNcVA» 37 3.6 Andy Shau – ‘Clo e Ciga e e’ The pic u esque ack ‘Clo e Ciga e e’ [49], by Andy Shau , is a song w i en abou hal - o go en memo ies ha ook a new app oach o s o y elling and cinema og aphy. Following ha heme, he music ideo ou s h ough Shau ’s mind as he e isi s To on o’s skyline es au an , his backya d, and la e summe nigh s. Di ec ed by Colin Medley and Ja ed Raab – a p oduc ion eam based in To on o, he ideo clip was p oduced using a a ie y o echnologies including LiDAR, pho og amme y and a enowned ideo game engine – Un eal Engine. Sau ’s wo ld was cap u ed hen digi ally econs uc ed in o s ill, eli scenes which we e hen ende ed ou as one con inuous ideo. Because he song was abou hal - o go en momen s, Ja ed and Raab wan ed o c ea e some hing d eam-like and impe ec . A e looking o solu ions, hey s umbled upon Ruben F o’s wo ks, ha is when hey knew hey had ound he igh oolse and aes he ic o a icula e he eeling o he song. Howe e , ins ead o unning e e y hing wi hin Uni y, like Ruben, hey decided o y ou Un eal Engine, which was some hing hey we e al eady amilia wi h. The eam ini ially wan ed o c ea e poin cloud based anima ions, bu ha ing un a ew es s o jus a s ill scene wi h he addi ion o a mosphe ic ligh ing and smoo h came a passages, hey igu ed ou ha was all hey needed. By ollowing his app oach, hey we e able o c ea e a d eam- like jou ney h ough Andy’s wo ld ha sui ed his wa m, hazy summe ibes. By wo king in a ully digi al en i onmen , hey had a lo o e sa ili y when i came o pe ec ing he came a acks, adding pos -e ec s, changing he ligh ing and e en b inging o eign elemen s in o he scene ( igu es 23 and 24). This p ojec mo i a ed ou incen i e o es Un eal Engine as an op ion o ende ou poin clouds. This led o ou disco e y o i s supe io olume ic ligh ing capabili ies and easie poin cloud impo ing me hods, compa ed o i s i al – Uni y. 38 Figu e 23: S ill om Andy Shau – ‘Clo e Ciga e e’, showing i s d eam-like and impe ec na u e, he sub le dep h o ield came a and ligh ing e ec s along wi h he bu ning ciga e e pos e ec . A ailable in «h ps://www.you ube.com/wa ch? =A8QPF S2_ E» Figu e 24: S ill om Andy Shau – ‘Clo e Ciga e e’, showing i s pos e ec s (bloom/ og), ligh ing (neon ligh s abo e es au an en ance and axi headligh s) and o eign elemen s ( axi and blue bicycle). A ailable in «h ps://www.you ube.com/wa ch? =A8QPF S2_ E» 39 3.7 Reali ies – Cologne Ca hed al Pa o he p ac ical componen o his hesis add esses he p ese a ion o he Bo anical Ga den as a digi al backup. This is some hing ha is common in he Cul u al He i age and A chaeological Documen a ion ields, which ocus p ima ily on c ea ing he ines digi al eplicas o eali y possible. Reali ies.IO is a company ha specialises in eali y cap u e, pa icula ly in he ield o Cul u al He i age, whose main goal is o allow you o isi si es o in e es all o e he wo ld. Reali ies.IO so wa e is a ailable o ee on S eam and is cu en ly compa ible wi h he HTC Vi e, Oculus Ri , and Windows Mixed Reali y. Reali ies – Cologne Ca hed al [50] allows you o ou h ough a digi al eplica o he Go hic a chi ec u e o he Cologne Ca hed al – ano he one o UNESCO Wo ld He i age Si es. This VR ou also includes a po ion o he Ca hed al whe e isi o s a e no no mally allowed, adding an ex a sense o exci emen and wonde o he VR expe ience when you a e able o en e i s sec e ooms and enclosed a eas. Simila o Clams Casino – Rune music ideo, Reali ies.IO used pho og amme y and p oceeded o econs uc he esul ing poin clouds in o ully ex u ed 3D polygonal mesh ( igu e 25). This was e iden ly done o achie e he highes le el o ealism, howe e , i s “so edges” and “low-poly” appea ance a e s ill qui e no iceable a close p oximi y ( igu e 26). In his con ex i can be seen as a ai decision, none heless, i s ill demons a es he impe ec na u e o pho og amme y. Fo example, by compa ing his p ojec o he agmen ed o m o La Ca héd ale ( igu e 27), a p ojec by Benjamin Ba dou, we can clea ly see how i di e s om Reali ies.IO’s isual language. Al hough La Ca héd ale is no as isually e ined, i s poin cloud ep esen a ion allows us o dig a bi deepe in o i s space and ge a be e eeling o i s a mosphe e. Reali ies – Cologne Ca hed al is a clea example o ou decision o use poin clouds as a isual ep esen a ion medium o his hesis a he han a econs uc ed mesh, o a oid he “so edge” appea ance esul ing om he pho og amme ic mesh ace econs uc ion p ocess. 40 Figu e 25: S ill om Reali ies – Cologne Ca hed al, illus a ing i s econs uc ed mesh amewo k. A ailable in «h ps://s o e.s eampowe ed.com/app/708620/Reali ies__Cologne_Ca hed al/» 41 Figu e 26: S ill om Reali ies – Cologne Ca hed al, showing an example o he “so edge” appea ance. A ailable in «h ps://s o e.s eampowe ed.com/app/708620/Reali ies__Cologne_ Ca hed al/» Figu e 27: S ill om La Ca héd ale, by Benjamin Ba dou, demons a ing an al e na i e poin cloud based isual language used in he con ex o a ca hed al. A ailable in «h ps://you u be/ N 9W1TQ TGA» 48 Sep JanOc FebNo Ma MayDec Ap Jun Jul 1.Da a Acquisi ion 2.Da a P ocessing 3.P ojec De elopmen 4.E alua ion and Op imisa ion 1.a 2.a 2.b 2.c 3.a 3.b 2.d 1.b 3.c 3.d 3.e 3. 3.g 4.a 4.b 1.c Figu e 30: Gan cha o ou ini ial wo k plan, demons a ing how hese ou phases would be ca ied ou du ing he execu ion ime o his hesis. 49 Changes o he Wo k Plan Due o se e al unexpec ed echnical d awbacks desc ibed h oughou chap e s 5, 6 and 7, along wi h he de elopmen o addi ional showcase ma e ial, oge he wi h he composi ion o he wo k-in-p og ess pape o ICEC 2022, which was ini ia ed du ing mid-p ojec de elopmen , he ou phases o ou ini ial wo k-plan ook a no iceable empo al shi . Figu e 31 shows an upda ed Gan cha o how he de elopmen o ou p ojec eally wen , along wi h an upda ed wo k plan. He e he main di e ences a e: he sepa a ion o he p ocessing o ou pho og aphic and audio da a due o hei complexi y, ollowed by he addi ion o ac i i y #5: De eloping showcase ma e ial, whe e we de eloped a websi e and 360-deg ee ideos. 1. Da a Acquisi ion a. Explo e and de elop a wo k low o cap u ing da a; b. Collec pho og amme ic da a; c. Reco d audio. 2. Da a P ocessing a. De elop a wo k low o da a e inemen ; b. P epa e cap u ed da a o poin cloud p ocessing; c. C ea e poin clouds; d. Re ine poin clouds; e. Re ine Audio. 3. P ojec De elopmen a. Explo e he exp essi e po en ial o poin clouds; b. De elop he s uc u e o he i ual uni e se; c. Explo e ways o enhancing ou expe ience wi hin his digi al uni e se by adding in e ac i e elemen s; d. Compose he imme si e expe ience (PC); e. Run p elimina y e alua ion es s; . De elop he imme si e expe ience compa ible wi h a VR headse . 4. E alua ion and Op imisa ion a. Fine- une he imme si e expe ience o achie e he bes quali y- o-pe o mance a io o bo h PC and VR a ian s; b. Run a inal se o use e alua ion es s. 5. De eloping showcase ma e ial a. Websi e de elopmen ; b. Audio- isual ma e ial de elopmen . 50 Sep JanOc FebNo Ma MayDec Ap Jun Jul Aug 1.Da a Acquisi ion 2.Da a P ocessing 3.P ojec De elopmen 1.a 2.a 2.b 3.a 2.d 1.b 3.b 3.c 3.d 3.e 3. 4.a 1.c Sep 2.e 4.b 4.E alua ion and Op imisa ion Figu e 31: Gan cha o how ou wo k plan eally wen du ing he ime-span o his hesis. 2.c 5.b 5. De eloping Showcase Ma e ial 5.b 5. P ojec De elopmen 52 Be o e s a ing p ojec de elopmen , a b ie in oduc ion and his o ical backg ound o he Bo anical Ga den o Coimb a will be made in o de o help acclima ise ou no ion o his loca ion o cul u al he i age. A e his, we p esen a se ies o ou p elimina y expe imen s, ocused on explo ing he bes solu ions o poin cloud da a acquisi ion h ough pho og amme y. Finally, wo ini ial p o o ypes a e desc ibed. 5.1 Objec o S udy – The Bo anical Ga den The Uni e si y o Coimb a’s s a u es o 1772 s a ed ha : “in a place in he icini y o he Uni e si y ha is o be conside ed he mos app op ia e and compe en , a said Ga den will soon be es ablished; so ha e e y kind o plan may be cul i a ed he e; and pa icula ly hose o which can ui use ulness in he ield o medicine, a e o be expec ed” [55]. Loca ed in he hea o he ci y, he Bo anical Ga den is an iconic space in Coimb a and a p es igious loca ion in Po ugal o i s scien i ic con ibu ions o bo any, consis ing o a ple ho a o anscon inen al lo a ha ake you on a jou ney o egions h oughou he wo ld. Belonging o UNESCO’s lis o p ope ies egis e ed as Wo ld He i age since 2013, his Ga den has been expanding i s isibili y and consequen ly he p esence o isi o s, who, in addi ion o being able o pa icipa e in he a ea o scien i ic cul u e, can equally enjoy his public space o leisu e, whe e hey can mee o casual encoun e s and explo e na u e by s olling by a e and exo ic plan s om all pa s o he wo ld wi h o he na u al landscape en husias s. Fu he mo e, his Ga den has also expanded i s ole in en i onmen al educa ion, as a ec ea ional space o a ious ac i i ies su ounding men al heal h and well-being, ou s, conce s and exhibi ions. Looking back in ime, he 18 h cen u y b ough abou a e olu ion o minds and signi ican scien i ic p og ess. The Bo anical Ga dens, he e o e, we e c ea ed in o de o complemen and encou age he ields o Na u al His o y and Medicinal S udies o he Uni e si y o Coimb a [56]. This con ex led o he c ea ion o he Bo anical Ga den in 1772, by he adminis a ion o he Ma quis o Pombal, on land dona ed by Benedic ine ia s. Ini ially named Ho o Bo ânico, he Ga den ook up only he a ea ha is s ill known as he Quad ado Cen al (Figu e 32: Cen al Squa e, poin 5). 53 The s a u es o he Uni e si y o Coimb a de e mined he cul i a ion o all so s o medicinal lo a, including plan s o o e seas o igin. The cul i a ion o plan s began in he Cen al Squa e in 1774 and was only comple ed in 1790, wi h he addi ion o a cen al oun ain – an ins alla ion which s ill emains o p esen day. Du ing his ime, he Facul y o Medicine s udied he healing p ope ies o plan s which led o an upg ade consis ing o a se ies o ec angula beds o u he cul i a ion o medicinal lo a. Today, he Ga den s ill keeps o his scien i ic componen h ough a seedbank p og am. Published o he i s ime in 1868, his seedbank includes many Po uguese and exo ic species, including se e al endange ed a ie ies, which ha e played a pionee ing ole in he conse a ion o na u e. A he p esen ime, he Bo anical Ga den co e s a ound hi een hec a es o land and can be di ided in o di e en le els, s ai s and a enues, each con aining hei own unique space, lo a and a mosphe e. Below is an illus a i e map o he Ga den wi h enume a ed a eas depic ing i s mos p ominen loca ions ( igu e 32), ollowed by a close look and desc ip ion o he lis ed loca ions ( igu es 33, 34, 35, 36, 37 and 38). One o he main objec i es o he Bo anical Ga den is o es ablish a close link be ween he Ga den and he public, by awakening hei in e es in science, na u e and bo any. This is one o he main easons his loca ion was chosen as he basis o his hesis, by me ging simila ideas and objec i es hus emphasising he Ga den’s essence h ough he help o con empo a y echnology. Main Ga e Baixo Rele o de Luís Ca isso Alameda das Tílias T opical G eenhouse Cen al Squa e Cold G eenhouse Figu e 32: Bi d’s-eye iew illus a ion o he Bo anical Ga den depic ing i s mos p ominen loca ions. Adap ed om «h ps://www acebook com/media/ se /?se =a 424667424214088& ype=3» 1 1 2 2 3 3 4 4 6 6 5 5 54 Figu e 33. Figu e 34. Figu e 36. Figu e 38. Figu e 33: Main Ga e en ance, con aining he ma ble s a ue o bo anis Félix de A ela B o e o. Da es om 1887 [58] – pho og aph aken in he Bo anical Ga dens o Coimb a, 05/11/21. Figu e 34: Pho og aph om he Cen al Squa e, looking up a he Main En ance, con aining he s a ue o he di ec o o he Ga dens in 1918 – P o esso Luís Ca isso [57]. A ailable a «h ps:// www. acebook.com/Ja dimBo anicoUC/pho os/? e =page_in e nal». Figu e 35: Alameda das Tílias, one o he mos emblema ic places in he Ga dens, which eminisces he old 19 h cen u y public oo pa hs o Eu opean ci ies [57]. Pho og aph aken in he Bo anical Ga dens o Coimb a, 05/11/21. Figu e 36: T opical G eenhouse – one o he oldes i on a chi ec u e buildings in Po ugal. I s pe ec combina ion o i on and glass gi e his space i s su p ising beau y. Reopened in 2018 a e a ew yea s o es o a ion, i is home o mainly opical and sub opical plan s [57]. A ailable a «h ps://www. acebook.com/Ja dimBo anicoUC/pho os/? e =page_in e nal». Figu e 37: Neoclassical oun ain in he middle o he Cen al Squa e, su ounded by geome ically designed walls, hedges and aised beds con aining a wide a ie y o lowe ing auna, ansmi ing he en i e a mosphe e o Roman icism o his a ea [57]. A ailable a «h ps://www acebook com/ Ja dimBo anicoUC/pho os/? e =page_in e nal». Figu e 38: Cold G eenhouse, buil in he 1950’s, unde he di ec ion o Bo anis Abílio Fe nandes. This g eenhouse con ains lo a o humid and shaded en i onmen s, su ounded by a cascading wa e all and a small s eam ha uns h ough i s en i e y. A emale nude s a ue by sculp o Ma ins Co eia also inhabi s his a ea, named “Bo ânica’’, who symbolises he Science o Plan s [57]. A ailable a «h ps://www acebook com/ Ja dimBo anicoUC/pho os/? e =page_in e nal». 4 6 12 Figu e 35. 3 Figu e 37. 5 55 5.2 P elimina y Expe imen s Be o e s a ing any de ini i e de elopmen , ini ial es s we e un in an e o o ge accus omed o he pho og amme ic p ocessing pipeline. Mo eo e , hese es s we e ca ied ou o de e mine an e icien wo k low o bes sa is y he p ac ical goals and objec i es o his hesis. 5.2.1 Expe imen nº1 – 360º Came a In an a emp o cap u e as much eal-wo ld da a in he sho es amoun o ime, we es ed a 360º 4k ideo eco ding de ice, mo e speci ically, he Samsung Gea 360. This idea s emmed om he u ban-based p ojec s by Ruben F osali (3.1 Memo ies o Tsukiji, page 26) and Benjamin Ba dou (3.2 Shinjuku, page 29), as hese we e also eco ded using 360º came as. In ha espec , we eco ded a sho 25 second ideo, whe e we ollowed a p ede e mined pa h ha e ol ed a ound a well known Tilia ee o he Bo anical Ga den ( igu e 39), hus cap u ing a comple e 360º iew o he ee. In o de o be able o ans o m his ideo clip in o a poin cloud, we i s had o ende he 360º oo age in o a compa ible o ma equi ed by he pho og amme ic p ocessing so wa e Agiso Me ashape. This was accomplished wi h Samsung’s Gea 360 Ac ionDi ec o , a media p ocessing so wa e used o s i ch he 360º ideo in o a s anda d ideo ame ( igu e 40). Basic ideo colou co ec ion was also done wi hin Ac ionDi ec o , o b ing ou he shadows and o co ec any o e blown highligh s in an e o o aid Me aShape’s ea u e de ec ion p ocess. This oo age was hen impo ed in o Me aShape’s pho og amme ic p ocessing pipeline using i s de aul p ocessing pa ame e s. This esul ed in a poin cloud o 27 million poin s ( igu e 41). A ew no es can be aken om his ini ial es , namely he la ge amoun o noise in he esul ing poin cloud wi h s ange loa ing sky and cloud pa icles, as well as some e iden alignmen issues esul ing in duplica e objec s ( igu e 42). None heless, his poin cloud was hen impo ed in o Un eal Engine, using i s buil -in poin cloud impo e , o be u he analysed in he con ex o a eal- ime i ual en i onmen . A i s , Un eal Engine was ha ing some di icul ies ende ing he poin cloud ( igu e 43), i s s uc u e was incomple e and pa s o he poin cloud would appea and disappea in e mi en ly du ing iewpo mo emen . Howe e , his issue was bypassed by inc easing he poin cloud plugin’s 56 de aul 1 million poin budge o a alue g ea e han 4 million ( igu e 44). I is o no e ha he la ge he poin budge , he mo e de ailed he image becomes, howe e , his comes wi h he bu den o exponen ially highe p ocessing equi emen s, hus lowe ame- a es. In his expe imen and wi h he ha dwa e used, we we e able o ende 20 million poin s on-sc een, in eal- ime, a a s able ame- a e o 60 ames pe second. Figu e 39: S ill om he aw 360º oo age o he Tilia ee. Figu e 40: S ill om he now s i ched 360º oo age wi h imp o ed shadows and educed highligh s. 57 Figu e 41: Sc een cap u e o he esul ing poin cloud wi hin CloudCompa e, no e i s noisy na u e, s ange misaligned ee b anches and blue sky pa icles. Figu e 42: Sc een cap u e showing e iden alignmen issues esul ing in duplica e objec s. 64 Figu e 53: Sc een cap u es illus a ing CloudCompa e’s colou il e ing addon ollowing he emo al o he blue sky poin s (Le : O iginal, Righ : Cleaned). 5.2.5 Fi s P o o ype Ha ing al eady explo ed di e en isual ep esen a ions o ou poin clouds, we decided o op o a middle g ound be ween accu acy and noise/ agmen a ion. Displaying enough da a o allow us o unde s and ou su oundings, bu no ealis ic enough so as o no emo e ou sel es om he d eam-like s a e o eli ing a memo y. I was unde s ood ha ha ing an ini ial high quali y da ase o wo k wi h was he mos sensible cou se o ac ion. This is because we can always educe he inal esolu ion o ou poin cloud, bu can ha dly inc ease i . Fo his eason, cap u ing eali y by aking high esolu ion pho og aphs in RAW o ma was he chosen and inal me hod o da a cap u e. Ou inal p elimina y expe imen began wi h he cap u e o he en i e Main En ance o he Bo anical Ga den, whe e 450 pho og aphs we e aken. To no miss any da a, se e al ounds o pho og aphs we e sho , ollowing a me hod simila o emula ing a one-eyed cybo g ying o make sense o his space, by e ol ing a ound each and e e y objec o he scene mul iple imes a di e en angles ( igu e 54). 65 Simila o he p e ious expe imen , hese pho og aphs we e hen colou co ec ed and e- ouched wi hin Adobe Ligh oom be o e ge ing impo ed in o Me ashape. This ime, in an a emp o u he enhance accu acy and ea u e de ec ion, all pho og aphs we e sligh ly sha pened and de-noised. Fu he mo e, we decided o inc ease some o Me ashape’s pa ame e s, namely he numbe o desi ed ea u e poin s o he algo i hm o look o , as his was hough o inc ease he numbe o poin s in he ou come o he ollowing build dense-cloud p ocess. This pa ame e alone inc eased he numbe o poin s o ou poin cloud o 174 million, esul ing in a a he la ge and hea y ile ( igu e 55). Na u ally, a lo o loa ing poin s and alse co espondences we e included in his sum and needed o be assessed wi hin CloudCompa e ( igu e 56). Floa ing blue sky poin s we e emo ed wi h he colou il e ing addon, hen he gene al noise was educed wi h a noise il e (by emo ing any o phaned loa e s g ea e han a ce ain adius o each and e e y poin ), ollowed by a subsample o lowe he global poin coun o 50 million poin s ( igu e 57). Al hough no ye de ined, his subsample alue was pu posely used o imp o e s o age capabili ies along wi h imp o ing esponsi eness and pe o mance wi hin Un eal Engine. This poin cloud was hen impo ed in o Un eal Engine, whe e he ine de ails o ou selec ed isual language we e explo ed o ge be e accus omed o he capabili ies o Un eal Engine. Wi hin his i ual se up, se e al isual pos - p ocessing ea u es we e explo ed o enhance he poin cloud en i onmen . Adhe ing o a mo e ealis ic app oach, he e we explo ed dynamic ligh ing, olume ic og and shadows, o emula e he ligh ays we had obse ed adia ing h ough he ee’s lea es and b anches ( igu es 58, 59 and 60). This ollowed by he explo a ion o poin shape and size, in an e o o ind a balance be ween ealism and he a mosphe e o a d eam-like s a e ( igu e 61). 66 Figu e 54: Sc een cap u e o he esul ing spa se cloud wi hin CloudCompa e, showing i s a ious angles o cap u e. Figu e 55: Sc een cap u e o he esul ing poin cloud wi hin CloudCompa e, esul ing in a poin cloud o 174 million poin s. 67 Figu e 56: Sc een cap u e wi hin CloudCompa e, illus a ing i s loa ing poin s and alse co espondences. Figu e 57: Sc een cap u e wi hin CloudCompa e, showing he educ ion o loa ing poin s and alse co espondences a e an impe ec clean-up p ocess. 68 Figu e 59. Sc een cap u e wi hin Un eal Engine, illus a ing he olume ic og and dynamic ligh ing e ec s implemen ed o added ealism. Figu e 58. Sc een cap u e wi hin Un eal Engine, illus a ing he olume ic og and ligh ay pos -p ocessing e ec s adia ing h ough he ee’s lea es and b anches. 69 Figu e 60. Sc een cap u e wi hin Un eal Engine, showing he explo a ion o poin size and shape. Figu e 61: Sc een cap u e wi hin Un eal Engine, illus a ing he esul ing balance be ween ealism and he a mosphe e o a d eam-like s a e. 70 5.4 Second P o o ype: High ideli y Tilia x eu opaea A e concluding he p e ious expe imen s and ha ing es ablished mos o he me hodology, a mo e comple e, high ideli y p o o ype was de eloped o include he majo i y o he ea u es expec ed in he inal p ojec . Du ing he de elopmen o ou ini ial p o o ype, we no iced ha Un eal Engine’s buil -in poin cloud impo e was showing some issues whils ende ing he poin s on sc een. Fo easons s ill o be de e mined, some poin clus e s would s a o licke depending on he playe ’s angle o iew wi hin he i ual en i onmen , his issue became qui e no iceable and nega i ely a ec ed ou use expe ience. Coinciden ally, his issue is also appa en in Andy Shau ’s – “Clo e Ciga e e” music ideo (men ioned in sec ion 3.6: page 37), which was also de eloped using Un eal Engine, whe e agmen s o he scene would disappea and eappea in an ab up ashion. A e inding no solu ion o his p oblem, i was decided ha we shi o e o Uni y’s game engine, as his was also so wa e we we e mo e amilia wi h. This high ideli y p o o ype consis s o a single a ea o he Bo anical Ga den ha was cap u ed du ing No embe , 2021 con aining he Tilia x eu opaea [59]. The de elopmen o his p o o ype was di ided in o six s ages: 1. Poin Cloud Rep esen a ion, 2. Rep esen a ion o Na u al P ocesses, 3. In e ac ions, 4. Sound, 5. Locomo ion and A ea T a e sal, and las ly 6. P elimina y E alua ion. He e u he isual explo a ion was pe o med on ou poin cloud, oge he wi h di e en app oaches o ep esen and highligh some o he scenes na u al p ocesses associa ed wi h he ee’s biology. Mo eo e , sound and use in e ac ion which allow use s o di ec ly in luence he beha iou o he expe ience was also explo ed, including locomo ion and a ea a e sal. Las ly, a se o p elimina y use es s we e pe o med o unde s and how people would eac o he imme si e en i onmen . Made up o 275 pho og aphs, his cap u e esul ed in a dense cloud o 100 million poin s ( igu es 62 and 63). Howe e , a e unning in o some ini ial pe o mance issues (low ame a e), i was disco e ed ha we could only ende app oxima ely 20 million poin s a any gi en ime, meaning ha ou ini ial idea o me ging mul iple a eas o he Bo anical Ga den in a sandbox s yle open-wo ld was ou o he ques ion. Du ing he de elopmen o his p o o ype we 71 Figu e 63: A close iew o he ee whe e we can see how he cap u e p ocess was done. Figu e 62. Sc een cap u e wi hin Me ashape showing he alignmen o he 275 pho og aphs, esul ing in a poin cloud o app oxima ely 100 million poin s. u he disco e ed ha he g aphics API se wi hin Uni y’s p e e ences hea ily in luenced he pe o mance o ou scene, whe e Vulkan API would ou pe o m Di ec X11 by a ound 40 ames pe second, bo h using he exac same scene pa ame e se ings ( igu es 64 and 65). 72 Figu e 64. Sc een cap u e o he scene being ende ed using he Vulkan API. Figu e 65: Sc een cap u e o he scene being ende ed using Di ec X11, no e he change in ame a e compa ed o he p e ious igu e (see op igh s a . ba s). 73 5.4.1 Poin Cloud Rep esen a ion Th ee poin cloud ende ing me hods we e explo ed o ind a balance be ween isual ideli y and compu a ional pe o mance. Ou poin cloud was ini ially ende ed as a s a ic objec wi h a ixed poin size, howe e , i s poin s would become la ge a close p oximi y and hus diminish he isual d eam-like aes he ic we we e looking o ( igu e 66). As an al e na i e, a “pixel absolu e” ende ing me hod was explo ed so ha each poin would occupy he size o a pixel a all imes ( igu e 67). None heless, wi h his me hod, he poin s a hes away in he scene would ine i ably scale much la ge han he ones close o he came a (use ), esul ing in an unappealing con as o poin sizes. Mo eo e , du ing hese es s we no iced a signi ican d op in pe o mance depending on he se size o ou poin s, whe e a smalle global poin size would ende much as e han a la ge poin size. As a hi d a emp , Uni y’s VFX ( isual e ec s g aph) was used o ende and anima e ou poin cloud ( ia ex u e iles con aining i s posi ion and colou da a), he e i s con inuous poin bu s ea u e wi h he addi ion o o ce ields was explo ed. This me hod esul ed in in e es ing isuals domina ed by oiling ex u es ( igu e 68), howe e , i quickly became appa en how compu a ionally hea ie his me hod was compa ed o he i s wo. F om hese esul s i was decided ha he ini ial ixed poin size ende ing me hod be used, his ime wi h a smalle global poin size, as he inal ende ing me hod ( igu e 69). This me hod no only c ea ed a mo e uni o m look h oughou he scene, bu also p oduced a mo e pho o ealis ic and consis en appea ance a di e en esolu ions (1080p, 1440p, and 4k). Figu es 66 and 67: Compa ison be ween “ ixed poin size” (Le ) and “pixel absolu e” (Righ ) poin ende ing me hods. 80 Figu e 79: A ea a e sal zones a ei he ex emi y o ou a ea. ~3.64m ~4.82m ~5.01m Figu e 80: Rep esen a ion o he p oximi y sys em ha con ols he ade-ou e ec . The ed poin s ( ) ep esen ing he p oximi y senso s, along wi h hei espec i e dis ances o he use (came a). Zone 1 Zone 2 Use (came a) P oximi y Senso s T igge Box 81 5.4.5 P elimina y E alua ion This p o o ype was only a ac ion o ou desi ed goal a he ime, ne e heless, we wan ed o pe o m p elimina y es s wi h pa icipan s ex e nal o ou eam o unde s and how people would eac o he imme si e en i onmen , and o also ga he ex a da a o submi ou WiP (Wo k-in- P og ess) submission o he In e na ional Con e ence on En e ainmen Compu ing [62]. Du ing a high-school isi o ou Uni e si y on he 27 h o Ap il 2022, we an in-pe son play es s on six een s uden s ( igu e 81). Each pa icipan explo ed bo h de eloped a ian s o ou i ual en i onmen : one a ian used a keyboa d and mouse (wi h a ixed al i ude) o mo e and explo e he en i onmen ; he second a ian , used a gamepad con olle (wi h he abili y o ly). Du ing hese es s we eco ded use analy ics, including play ime, and use posi ion and o ien a ion acking wi hin he a ea. We subsequen ly asked ou pa icipan s o answe a sho eedback ques ionnai e whe e hey we e asked o desc ibe hei expe ience in i e wo ds, wha a ian hey p e e ed (keyboa d o gamepad con olle ), and las ly, a line o commen s and/o sugges ions. The a e age play ime o he keyboa d and mouse was 53 seconds whils he gamepad con olle was 62 seconds. The mos popula wo ds we e “Realis ic” (11.8%), “Fun” (11.8%), “In e es ing” (9.4%) and “Inc edible” (7.1%). Two hi ds o he pa icipan s p e e ed he gamepad con olle o e he keyboa d and mouse, howe e , his ac o was suspec ed o be linked o he young demog aphic so u he es s we e planned o be ca ied ou in he u u e, on a wide audience. In e ms o use posi ion acking and despi e he a ea ha ing a cen al ocal poin (being he Tilia ee), we no iced ha mos use s p oceeded o explo e he en i e space, as can be seen in he use posi ion acke s ( igu es 82 and 83). Fu he mo e, we also no iced ha mos o ou pa icipan s had he u ge o a el o he nex a ea, wan ing o explo e mo e a eas. Las bu no leas , some use s managed o bypass he se bounda ies (wall collide s) o ou a ea, an issue ha e en ually go esol ed by inc easing he hickness o he wall collide s. In gene al, hese p elimina y esul s showed a posi i e e alua ion and p o ided enough eedback o guide us o con inue o expand and imp o e ou i ual wo ld. Figu e 81: “Gi ls in ICT day” a he Depa men o In o ma ics Enginee ing o he Uni e si y o Coimb a, whe e we an in-pe son play es s on sixe een high- school s uden s. 82 Figu e 83: Use posi ion acking o he gamepad con olle a ian wi h he abili y o ly. Figu e 82: Use posi ion acking o he keyboa d and mouse a ian wi h a ixed al i ude. Each colou ep esen s a pa icipan and he a ows show in wha di ec ion he playe was looking. 6. De elopmen o Kai os 84 The de elopmen o he main p ac ical componen o his hesis was s a ed by es ablishing he numbe o a eas we we e going o cap u e, wha na u al p ocesses we we e going o include wi hin each a ea, how we would in e connec hese loca ions along wi h how he use would in e ac wi h said a eas. Ha ing al eady isi ed he Bo anical Ga den mul iple imes, we op ed o h ee loca ions highligh ing some o he oldes iconic ee specimens o he Ga den: he Tilia x eu opaea [9], si ua ed nea he Alameda das Tílias, he E y h ina C is a-Galli [10], loca ed wi hin he Cen al Squa e, and he Ficus Mac ophylla [11], loca ed beside he cold g eenhouse ( igu e 84). A ea 1 Tilia x eu opaea A ea 3 Ficus Mac ophylla A ea 2 E y h ina C is a-Galli Figu e 84: Map o he Bo anical Ga den depic ing he h ee chosen loca ions o cap u e. 85 A s a egy and cou se o ac ion was de ised on how we would cap u e hese h ee loca ions. Ini ially, he cap u e o hese a eas was planned wi hin he closes possible ime ame, o concei e he idea o eplaying a memo y o a walk in he Ga den, bu because each loca ion equi es a la ge se o pho og aphs, aking a ound 30 o 50 minu es pe a ea, we decided o spli he cap u e session in o h ee sepa a e pa s: mo ning ( o he Tilia), noon ( o he C is a-Galli) and mid- a e noon ( o he Ficus), simula ing a day’s passage. Time was spen wi hin each a ea be o e commencing ou cap u es, by ga he ing and analysing in o ma ion on he isible na u al p ocesses occu ing in each a ea. Scien i ic in o ma ion was hen collec ed o he ees oo sys em s uc u es and hei symbio ic ela ionships wi h hei su ounding en i onmen [63][64] ( igu e 85). This was inspi ed by he hidden unde g ound wo ld o mic obes, wi h emphasis on myco hizal ne wo ks [65], ha is, he symbio ic ela ionships ha o m be ween ungi and plan s ( igu e 86). 6.1 Da a Cap u e and P ocessing Da a cap u e commenced in a ch onological o de , s a ing wi h he cap u e o he Tilia x eu opaea, ollowed by he E y h ina C is a-Galli and ending wi h he Ficus Mac ophylla. A lo mo e ca e was aken du ing he cap u e p ocess, as hese da a would be used o ou inal imme si e en i onmen . Because o his, a highe numbe o pho og aphs we e aken o imp o e he di e si y o ou da ase , and hus, he quali y o ou poin clouds. Ou ini ial plan o cap u e all loca ions on he same day ailed due o he cons an change o wea he condi ions, along wi h exceeding ou came a’s limi ed ba e y and s o age capaci ies. Consequen ly, each a ea go cap u ed on di e en da es. Mo eo e , he i s wo a eas became a p io i y due o hei ee’s limi ed lowe ing pe iods, gi ing a span o abou wo weeks o cap u e and p ocess hese wo. Fo una ely he Ficus Mac ophylla keeps i s lea es and ui ( igs) all yea ound, making i less o a p io i y a he ime. The success ul cap u es o ou a eas ook place on he 8 h o June om 15:56pm o 16:30pm (Ficus Mac ophylla), on he 9 h o June om 11:52am o 12:38pm (E y h ina C is a-Galli) and on he 12 h o June om 9:20am o 9:44am (Tilia x eu opaea) Ou pho og aphic da ase was impo ed and analysed wi hin Adobe Ligh oom. He e he p ocess o colou co ec ion ensued in an a emp o b ing ou as much de ail om he Figu e 85: “Wo ld Wide Web” – Illus a ion depic ing ees symbio ic ela ionship wi h hei su ounding en i onmen . A ailable in: h ps://www a biope u o g/ en/blog-pos s/mo he - ees/ Figu e 86: A small pine ee g own in a glass ank shows he amoun o whi e, inely b anched symbio ic mycho hizal h eads ha help eed he ee. A ailable in h ps://www csuchico edu/ egene a i eag icul u e/blog/how- ungus-migh -sa e- us sh ml 86 Figu e 87: Colou co ec ion, wi hin Adobe Ligh oom, o he a ea con aining he E y h ina C is a-Galli, in an e o o b ing ou as much de ail om he scene as possible. scenes as possible ( igu e 87). Ligh oom’s “Ma ch To al Exposu e” ea u e was also used o au oma ically equalise he exposu e alue o all pho og aphs in hopes o be e Me ashape’s ea u e acking p ocess. Al hough no ecommended by Agiso Me ashape’s use manual, we had no choice bu o comp ess ou pho og aphic da ase o he lossy comp ession o ma JPG, o a oid unning ou o memo y (RAM) du ing Me ashape’s came a alignmen p ocess. Albei ha ing al eady gained expe ience om ou p elimina y es s, new issues a ose conce ning ailed came a alignmen , unning ou o sys em memo y, and ailed objec alignmen wi hin he scenes, consis ing o mul iple misaligned ee unks and b anches ( igu es 88 and 89). Pa adoxically, hese issues we e only sol ed by educing he size o ou pho og aphic da ase h ough manual selec ion, whe e pho og aphs deemed edundan and p oblema ic go emo ed, such as hose wi h a lo o occlusions and ones wi h simila angles o cap u e. F om an ini ial pho og aphic da ase anging om 700 o 1100 pho og aphs pe a ea, ou inal da ase consis ed o a ange o only 400 o 600 pho og aphs. Du ing he emo al p ocess, we ei e a ed he pho og aph alignmen and “build dense-cloud” p ocesses se e al imes un il we ended up wi h a success ul poin cloud. 87 The a ea con aining he E y h ina C is a-Galli was he mos p oblema ic due o i s high concen a ion o ege a ion, low p ominen ea u e- acking poin s and he cons an change in wea he and ligh ing condi ions du ing ou cap u e sessions, which nega i ely in luenced he accu acy and ou come o ou poin cloud. We ended up e-cap u ing he Tilia x eu opaea and E y h ina C is a-Galli se e al imes due o he simila issues men ioned abo e. Figu e 89: Rep esen a ion o ailed objec alignmen wi hin he scene con aining he C is a-Galli, no e he duplica e b anches and wooden ee suppo poles. Figu e 88: Rep esen a ion o a ailed came a alignmen wi hin he Tilia scene, no e he s ange came a posi ioning a he bo om igh co ne . 88 When i came o he Ficus Mac ophylla, bo h cap u e and p ocessing o his scene we e concluded in a single un wi h ease, pe haps his was due o he highe p ominen ea u e coun o he scene, oge he wi h mo e s able wea he condi ions du ing he cap u e session ( igu e 90). Figu e 90: Rep esen a ion o a success ul came a alignmen wi hin he scene con aining he Ficus (spa se cloud), whe e he blue ec angles ep esen e e y pho og aph aken. Figu e 92: Localised me ging, scaling and alignmen o he C is a-Galli’s hollow unk, using da a om wo sepa a e poin clouds (Le ); Be o e (Cen e); A e (Righ ). Figu e 91: Alignmen o wo sepa a e poin clouds o he same a ea (highligh ed in ed and yellow), in an e o o en ich ou inal poin cloud. Despi e ha ing al eady ca ied ou mul iple cap u e sessions on he E y h ina C is a-Galli, each session esul ed in an insu icien poin cloud da ase . Fo ins ance, he i s cap u e ailed o ep oduce he ee’s lea es, lowe s and b anches, bu success ully managed o cap u e he ee’s hollow unk wi h g ea de ail. The second and hi d cap u es esul ed in poin clouds wi h a lo mo e lea , b anch and lowe de ail bu ailed a ep oducing he ee’s unk. This issue go esol ed by me ging, scaling and aligning sec ions o poin cloud da a om all h ee cap u es in o one ( igu es 91 and 92). 89 6.2 Da a Re inemen P oceeding wi h he e inemen o ou poin clouds, an ini ial manual emo al o unnecessa y and ou -o -bounds poin s wi hin Me ashape was execu ed using i s ee- o m selec ion ool ( igu e 93). Subsequen ly, all undesi able poin cloud da a such as blue sky and whi e cloud poin s go emo ed h ough Me ashape’s colou selec ion ool, oge he wi h CloudCompa e’s RGB il e ool, wi h a cus om de ined ole ance pe cen age o each and e e y colou emo al i e a ion ( igu e 94). Following he majo clean-up wo k, we we e s ill le wi h a la ge olume o poin s ha needed subsampling (i.e. dec ease he numbe o poin s) o p ese e pe o mance o Uni y. This p ocess go ini ia ed using CloudCompa e’s “Remo e duplica e poin s” ea u e, whe e poin s wi h a adius o less han 0.0099 uni s (CloudCompa e’s uni s o dis ance) om one ano he go emo ed. This e icacious ool alone was able o emo e mo e han hal he o al poin coun , eeing up space while main aining isual cohe ence. These esul s conclude ha ou pho og amme y de i ed poin clouds con ained a la ge numbe o nea ly o e lapping poin s. Figu e 93: Raw dense cloud o he a ea con aining he Ficus ee, consis ing o 344 million poin s (Top). Dense cloud a e manual clean-up, now wi h 261 million poin s (Bo om). 96 Figu e 106: The small Lasioglossum Dialic us (Le ) and he la ge Apis melli e a (Righ ), seen on he Tilia a he Bo anical Ga den. Se e al p e iously obse ed na u al p ocesses ( ha we e seen du ing ou isi s o he Bo anical Ga den) we e implemen ed. Ou o hese, wo dis inguishable bee species we e obse ed and egis e ed wi hin he a ea con aining he Tilia x eu opaea, which we e a he ime ga he ing nec a om he Tilia’s ag an , pale yellow lowe s: he small Lasioglossum Dialic us and he la ge Apis melli e a (also known as he Eu opean honey bee), ( igu e 106). These we e bo h simula ed wi hin VFX g aphs and also include use collision de ec ion. Mo eo e , he Tilia’s lea posi ion da a was used as inpu o he spawn posi ions o he smalle bees (encompassing he en i e ee, as was obse ed in eal li e). Two swa ms o he la ge honey bees we e manually placed wi hin he a ea, posi ioned in wo sepa a e loca ions. He e p oximi y senso s we e implemen ed ha igge when he use ge s oo close, his inci es he bees which hen s a e ol ing a ound he use in a mo e agi a ed ashion ( igu e 107). Figu e 107: Rep esen a ion o he Apis melli e a honey bees wi hin Uni y. 97 Se e al Pie is apae bu e lies we e obse ed lu e ing wi hin he a ea con aining he E y h ina C is a-Galli, in he ose pa ches adjacen o he ee ( igu e 108). These go simula ed by anima ing wo 2D ex u es (emula ing wings) by scaling hem ollowing a sign wa e (see igu e 109). Likewise o he bees, hese bu e lies also eac o he use ’s p esence and dispe se when he use ge s oo close ( igu e 110). Figu e 108: Pie is apae bu e lies. A ailable a : h ps://species wikimedia o g/wiki/Pie is_ apae#/media/File:Pie is_ apae_ MHNT jpg Figu e 110: Rep esen a ion o he Pie is apae bu e lies wi hin Uni y. Figu e 109: Le : sc een cap u e o he bu e lies VFX g aph, showing he p ocedu e o anima ing hei wings by scaling wo 2D ex u es ollowing a sine wa e. Righ : Illus a i e ep esen a ion o he wing anima ion. 98 To accen ua e he C is a-Galli’s ed ag an lowe s, i was decided o simula e he smell o nec a and pollen being eleased om hese ( igu e 111). Pa icle sys ems whe e posi ioned in he cen e o each lowe clus e , which elease small pa icles ( ep esen ing he nec a and pollen). Fu he mo e, each clus e con ains i s own p oximi y senso ha bu s s pa icles i he use decides o collide o come in o con ac wi h he lowe s. Ha ing an ini ial bu s quan i y o 600 pa icles, o e e y consecu i e collision, i s bu s quan i y ge s di ided by h ee ( igu e 112). Figu e 111: Flowe clus e o he C is a-Galli, ich in nec a . Figu e 112: Sc een cap u e o he C is a-Galli’s lowe pa icle sys ems – specula ing he elease o pollen and nec a . 99 Du ing he cap u e session o he Ficus Mac ophylla, i was no iced ha a lo o i s igs had allen o he g ound ( igu e 113), his is also e iden in he inal audio eco ding o he a ea, whe e we can hea igs d opping. Hence, i was decided o simula e hese igs using wo VFX g aphs. The i s andomly spawns igs on he g ound on scene load ( igu e 115), he second emi s igs om he ee (using he ee’s lea -posi ion da a as inpu ), whe e a andom ig ge s d opped (emi ed) wi hin a andom ime-space be ween ou and eigh seconds. A colou sample ange was hen added om an ac ual ig o di e si y he colou s o ou pa icle sys em ( igu e 114). Addi ionally, a sub le o ce is applied o hei nega i e y-axes, simula ing g a i y. Fo added ealism, once hese all o he ee, hey ligh ly bounce o he g ound ( ia a loo collide ) be o e coming o a hal . Small clus e s o lies we e also spo ed wi hin he a ea ( igu e 116) and implemen ed wi hin he scene, ollowing simila pa ame e s used o he bees su ounding he Tilia, wi h he addi ion o ails ( igu e 117). As a las no e, se e al bi ds we e obse ed ea ing igs o he g ound, howe e , hese did no end up ge ing simula ed due o ime cons ain s. Figu e 113: Pho og aph o allen igs unde he Ficus. Figu e 114: Pho og aph o one o he Ficus’s igs, oge he wi h i s colou g adien sample. Figu e 115: Rep esen a ion o allen igs wi hin Uni y, highligh ed in o ange. 100 Figu e 116: Small clus e s o lies spo ed nea he Ficus. Figu e 117: Small clus e s o lies simula ion wi hin Uni y. 101 6.4 Locomo ion and A ea T a e sal Taking he eedback da a om he p elimina y e alua ion es s in o conside a ion, i was decided o keep bo h modes o use in e ace: he keyboa d and mouse a ian , and he game-pad con olle a ian , bo h con igu ed using he “ ly came a” mode (g an ing he abili y o ly). Subsequen ly a sc ip go implemen ed ha con inuously de ec s i a con olle is connec ed o no and swi ches he inpu de ice acco dingly. Below is he splash sc een c ea ed o Kai os, con aining a diag am o he mouse and keyboa d con ols as well as he gamepad con olle . 102 Ou inal a ea a e sal sys em is based on he implemen a ion om he high ideli y p o o ype. Figu e 118 shows a op iew o he Bo anical Ga den con aining all h ee cap u ed a eas, including an illus a ion o he a ea a e sal sys em whe e exis ing pa hways wi hin each a ea we e used as exi and en y poin s. Ha ing no iced ha he a ea con aining he E y h ina C is a- Galli included ou pa hways, i became appa en o ake ad an age o his by ins alling wo exi and wo en y poin s o and om he o he a eas – a middle g ound be ween he h ee ( igu e 119). Adap ed om he a ea a e sal sc ip used o ou high ideli y p o o ype, his sys em con ols he en y and exi poin s o ou h ee a eas, allowing he use o a el om one a ea o ano he . To achie e his, igge boxes we e implemen ed and con olled by a sc ip ha cons an ly calcula es he use ’s dis ance om he exi s – allowing o a g adual scene ade-ou . These senso s de ec he use ’s p oximi y and s a o ade he scene once he use ge s close han 10 uni s o dis ance om ei he one o he h ee senso s. Subsequen ly, he use ge s anspo ed o he nex a ea he momen hey ouch he igge box. Figu e 119: Top iew o A ea #2 E y h ina C is a-Galli, depic ing i s en y and exi poin s. Bo om Righ : Snip o ou a ea a e sal sc ip , o a single a e sal ec angle, showing i s h ee senso s (Objec s 1, 2 and 3), hei espec i e dis ances o he Use (came a), and he Alpha alue o ou ade ansi ion. A ea #2 – E y h ina C is a-Galli A ea #2 – E y h ina C is a-Galli A ea #1 – Tilia x eu opaeaA ea #1 – Tilia x eu opaea A ea #3 – Ficus Mac ophylla A ea #3 – Ficus Mac ophylla Figu e 118: Top iew o he Bo anical Ga den illus a ing all h ee a eas, oge he wi h hei en y and exi poin s. Adap ed om Google Maps. 103 Figu e 120: Fou o he i e sma phones used o ou i s eco ding a emp . 6.5 Sound Design An expe imen al app oach was aken o achie e ou goal o c ea ing a 3D ambien sound ep oduc ion o ou imme si e expe ience, consis ing o ins alling mul iple mic ophones wi hin each o he cap u ed a eas. Be o e commencing any audio eco dings a he Bo anical Ga den, a p elimina y audio es was pe o med in an e o o de e mine i ou audio cap u e plan was in ac easible. Two sma phones we e employed o his es : he Sony XZ1 and XZ P emium, which we e placed on a lawn (in a ga den), sepa a ed by oughly i e me e s. Each phone was se o eco d a i s highes a ailable quali y (WAV o ma wi h a 48kHz sample a e). A e s a ing bo h eco dings, a loud clap signi ied he s a , o help synch onise and me ge hese in o a single audio ile. The esul ing audio quali y was decen ( ela i ely speaking), bu mos impo an ly we had success ully c ea ed a binau al audio eco ding, by simula ing sound as i i is being hea d li e (download link a ailable: “Lawn 3D”, page 179). Ha ing concluded his p elimina y es , we e u ned o he Bo anical Ga den whe e we ins alled i e sma phones ( igu e 120) which go e enly dis ibu ed wi hin each scene. Un o una ely, his equipmen go s olen jus a e concluding he las se o eco dings, lea ing us wi hou ou audio cap u e de ices and audio iles. Adhe ing o ou ini ial plan, a second ound o audio cap u e was done wi h a se o new mic ophones, including ones o a highe calib e: he H1 Zoom, he H2n Zoom, ou Sony a6300 came a and las ly, a Sony XZ1 sma phone ( igu e 121). These de ices we e e enly dis ibu ed a ound each scene in a ashion esembling a squa e ( igu e 122). Figu e 121: The H2n Zoom, Sony a6300 came a, Sony XZ1 sma phone and H1 Zoom – used o ou inal audio eco dings. Figu e 122: The posi ioning o ou equipmen wi hin he scene con aining he Tilia, esembling a squa e. 104 Due o unp edic able wea he condi ions, windsc eens we e made o hese mic ophones in a hope o minimise unwan ed wind noise. Each a ea was eco ded (using a WAV o ma wi h a 48kHz sample a e) o a pe iod o a ound 10 o 15 minu es. Howe e , be o e ini ialising ou audio eco dings, mic ophone gain was ine uned o he H1 Zoom, H2n Zoom and Sony a6300 o de e mine hei op imal signal- o-noise a io ( igu e 123). Ha ing success ully eco ded o all h ee o ou a eas, all audio iles go impo ed in o P emie e P o, whe e he synch onisa ion, clean-up and enhancemen p ocesses we e ini ia ed. The synch onisa ion and clean-up o ou clips was done by picking ou he peaks in he audio acks ( ep esen ing he claps), ollowed by cu ing ou all unwan ed sounds comp ised o human cha e , wind noise and a ic sounds. The emo al o hese sounds was decided due o hei o e ly dis ac ing na u e, whe e hese we e p esumed oo in e e ing o he delica e cha ac e o he a eas, especially in ega d o he a eas con aining he Tilia x eu opaea and E y h ina C is a-Galli A c oss- ade audio e ec was hen used as a bu e , o smoo h ou he audio cu s be ween he sepa a ed (dele ed) clips. Nex , noise educ ion, eble, bass and ampli ica ion adjus men s we e applied in o de o uni y ou audio as a whole ( igu e 124). Ul ima ely, all acks go impo ed in o Uni y. He e speake s we e placed in he exac same posi ions as he mic ophones had been placed in he eal wo ld ( igu es 125 and 126). Each speake was hen ans o med in o a 3D sound emi e whe e i s “spa ial blend” pa ame e along wi h i s min. and max. emission dis ances we e adjus ed o c ea e ou desi ed 3D audio e ec . Figu e 123: Mic ophone gain adjus men s o he H1 Zoom and H2n Zoom, oge he wi h hei espec i e windsc eens. Figu e 124: Audio edi ing wi hin P emie e P o, showing he audio adjus men pa ame e s used in an a emp o uni y all audio acks as a whole. 105 Figu e 126: Sc een cap u e depic ing he placemen o he 3D sound emi ing speake s wi hin he i ual scene, in he exac same posi ions as he mic ophones had been placed in he eal wo ld. Figu e 125: Mic ophone placemen wi hin he a ea su ounding he C is a-Galli – aking ad an age o he ga den’s sh ubs as na u al wind-shields. 112 7.1 Vi ual Reali y Se up and De elopmen Ou ques o implemen a Vi ual Reali y a ian ini ia ed a he same ime as ou high ideli y p o o ype was being de eloped, whe e he possibili y o unning ou imme si e expe ience wi hin a VR headse was explo ed. This began wi h he HTC Vi e (2016) ia S eam’s VR pla o m (S eamVR). Ne e heless, a e impo ing ou poin cloud in o Uni y ia Keiji o’s poin cloud impo e , ou p ojec ailed o un – h owing an unexpec ed e o . This go esol ed by changing Uni y’s g aphics API om Vulkan o Di ec X 11, howe e , his change also mean ha we we e ge ing a conside able pe o mance educ ion o abou 20 ames pe second. Besides his, s u e ing issues became appa en due o high la ency as a consequence o Vi e’s cons an minimum e esh a e o 90Hz (90 FPS) wi h a minimum la ency o 11.1 ms a a esolu ion o 3024 x 1680, o 1512 × 1680 pe eye. A e del ing in o S eam VR’s se ings we we e able o sligh ly imp o e un ime pe o mance by lowe ing he alue o he “Th o ling and P edic ion” se ing, by o cing he ha dwa e o un a a ixed slowe a e in o de o p o ide an o e all smoo he expe ience. In spi e o ou e o s, he high la ency issue pe sis ed, esul ing in an o e all poo VR expe ience. As a las eso , ou ende me hod was swi ched o e o Uni y’s VFX g aph, whe e ou ou pu poin s we e chosen o be ende ed as p imi i es (quads wi h a ixed size). Ne e heless, only when ou poin coun was educed o 2 million (o iginally 20 million) we e we able o achie e an accep able la ency a e, his, howe e , d as ically educed he isual ideli y o ou scene ( igu e 132). Fo his eason we decided o pu he HTC Vi e aside o look o ano he solu ion. Figu e 132: Sc een cap u e showing poo isual ideli y, esul ing om he educ ion o he poin coun o 2 million. 113 As a second a emp we es ed he Oculus Ques 2 (2020). This equipmen seemed p omising due o i s highe sc een esolu ion and wi eless acking capabili ies. To ake ad an age o his headse ’s in e nal, And oid-based ope a ing sys em, we ini ially ied building ou p ojec as an app o And oid, as his would pe mi he launch o ou imme si e expe ience wi elessly (as a s andalone). Howe e , wi h his me hod we we e elying on he Ques ’s Qualcomm Snapd agon XR2 chip o ende all o ou poin s. Un o una ely he XR2 was only able o ende a meag e 200 housand poin s be o e unning in o pe o mance issues. None heless, by connec ing he Ques 2 di ec ly o a PCVR eady compu e , ia a Link cable, we e we able o ake ad an age o ou highe pe o mance PC ha dwa e o ende g aphics, ins ead o he s andalone mobile ha dwa e in he headse . Mo eo e , his VR headse did no ha e any issues unning o he p e iously p oblema ic Vulkan g aphics API, meaning we had an addi ional pe o mance boos . Se e al benchma ks we e pe o med using he Link cable o he pu pose o de e mining he maximum quan i y o poin s we could ende a a sus ainable ame a e. Al hough no pe cep ible a i s glance (wi hin he headse ), we la e disco e ed ha we we e ge ing low and a iable ame a es, likely due o he ac ha we we e a emp ing o ende ou scene a a esolu ion o 3616 x 1840 (o 1808 x 920 pe eye), a a cons an e esh a e o 72 FPS ( he Ques ’s de aul e esh a e). Fo his eason ou scene was changed o he E y h ina C is a-Galli – a compu a ionally ligh e scene due o i s lowe global poin coun . An ini ial pe o mance op imisa ion was made by manually lowe ing he poin coun wi hin he VFX g aph as a means o eaching a s able ame a e o 72 FPS. Subsequen ly, all anima ions go impo ed in o he scene ( oo sys ems, bu e lies, unk, lowe and lea anima ions). This, howe e , nega i ely impac ed pe o mance once again, his ime making ou scene luc ua e om 72 FPS o 36 FPS. A mo e s uc u ed app oach was aken o y and s abilise he ame a e, as lowe ing he poin coun u he would only lead o a g ea e loss in isual ideli y. This app oach consis ed o spli ing he poin cloud in o se e al sec ions and subsec ions (g ound, scene ex emi ies, ee, ee’s lea es, lowe s and scene bushes), as was done du ing he de elopmen o he PC a ian o Kai os, allowing he educ ion o ou scene’s poin coun o be made in a mo e con olled ashion, by main aining mo e de ail (mo e poin s) a he ocal poin o he scene and educing he poin coun a he ex emi ies o he scene ( igu e 133). Figu e 133: P ocess o sepa a ing he scene in o sec ions and subsec ions in an e o o imp o e pe o mance whils main aining as much de ail as possible. 114 Two op ions a ose when i came o use in e ac ion: o wo k wi h he Ques ’s con olle s o expe imen wi h i s ecen ly eleased hand acking capabili y, he la e being able o de ec ou hands and inge s in eal- ime, h ough ou IR (in a ed) came as loca ed on he Ques ’s headse , hen displaying a model o hem wi hin he scene. This hand acking sys em was explo ed i s , as i s new hand and inge de ec ion sys em was eleased sho ly a e s a ing ou VR de elopmen ( eleased on Ap il 19, 2022). In o de o allow his sys em o wo k wi hin Uni y, he download and impo o Uni y’s OVRPlaye Con olle was necessa y, along wi h manually a aching i s hand p e abs o bo h hands. Ou goal consis ed o allowing he use o oam a ound and in e ac wi h he scene solely using his hand acking Figu e 134: Sc een cap u e o ou inal VR scene con aining 2.8 million poin s. This, howe e , was no su icien o s abilise he ame a e and a u he educ ion in he global poin coun had o be done. F om he scenes o iginal poin coun o 20 million poin s, ou inal VR a ea consis ed o only 2.8 million poin s. As a las eso , he size o ou poin pa icles was inc eased o add mo e olume o he scene in an a emp o coun e ac i s low poin coun ( igu e 134). 115 ea u e, h ough ges u es and by a aching sphe e collide s and a ac o s o ei he hand. We howe e we e only able o success ully implemen he la e and had o ind ano he solu ion o use locomo ion. Ha ing ailed o mo e a ound he scene solely wi h hand acking enabled, i was decided o ac i a e he Ques ’s con olle s, along wi h modi ying he o icial OVRPlaye Con olle sc ip o allow he use o “ ly” wi hin he scene by poin ing in he desi ed di ec ion o ligh ia he headse ’s o a ion/ il , and accele a ing ia he Le igge bu on. Ou nex ask consis ed o me ging he con olle ’s unc ionali y wi h he hand acking sys em, un o una ely he Ques 2 does no suppo he use o bo h inpu me hods simul aneously. Fo his eason i was decided o keep bo h a ian s, as sepa a e modes, whe e one can ly a ound he scene using he con olle s, hen, by placing down he con olle s, Uni y au oma ically swi ches o hand acking whe e he use can hen in e ac wi h he scene. Below is an illus a i e diag am o he con ols and inpu me hods implemen ed o ou Ques 2 a ian . 116 The Righ con olle and Righ hand con ain a sphe e collide and a ac o o in e ac wi h he scene ( igu es 135, 136, 137, 138 and 139), howe e , an issue a ose when ying o ep esen bo h con olle meshes and hand meshes du ing hese inpu ansi ions, whe e bo h meshes would o e lap. Al hough impe ec , i was decided o p io i ise he hand acking sys em o e he con olle s, by hiding hem. Mo eo e , he eason behind adding only one sphe e collide o he Righ con olle and hand was decided due o Uni y’s incapabili y o pe o ming mul iple collide and a ac o o ces ( o bo h Righ and Le ou pu s) on a single VFX pa icle sys em. The only way a ound his issue consis ed o spli ing each o ou VFX-based poin cloud pa icle sys ems in hal (50% o he le hand and 50% o he igh hand), his, howe e , esul ed in an une en a ac ion e ec because only hal he pa icles would be a ac ed o ei he hand, lea ing some pa icles una ec ed by hei collide and a ac o o ces. Figu e 135: Rep esen a ion o he Ques ’s hand acking, oge he wi h he implemen ed sphe e collide – highligh ed in ed. Figu e 136: Demons a ion o he hand collision wi h he ee’s unk. Figu e 137: Demons a ion o a bu e ly dispe sing as he use ies o ca ch i . 117 Figu e 138: Demons a ion o he lowe ’s “bu s ” e ec upon hand collision. Figu e 139: Demons a ion o he a ac ing o ce be ween he use and he ee’s ene gy. In compa ison o he in e ac i e elemen s o he Desk op (PC) a ian , an addi ion o ege a ion in e ac ion was implemen ed in he VR a ian , allowing he use o in e ac wi h he scene’s sh ubs, along wi h he lea es o neighbou ing ees. As a inal poin , all h ee loca ions we e o iginally planned on being implemen ed o ou VR a ian , howe e , due o pe o mance limi a ions, i was decided o solely include he scene con aining he E y h ina C is a-Galli. 118 7.2 Pass h ough Explo a ion In an e o o p oduce mo e con en o he Ques 2, i was decided o explo e i s expe imen al pass h ough API. Pass h ough is a mixed eali y- ocused ea u e ha allows he use o see a eal- ime 3D isualisa ion o he physical wo ld in he Oculus headse ia i s IR senso s. This ea u e b ough up he idea o c ea ing an in-si u a ian consis ing o a mixed eali y expe ience whe e he use could heo e ically isualise he hidden p ocesses o he E y h ina C is a-Galli by o e lapping ou poin clouds wi h he physical wo ld. A p elimina y poin cloud cap u e es o ou esidency was made be o e unning any in-si u es s. This consis ed o ende ing ou poin cloud, ia Uni y’s VFX g aph, o e he Ques ’s pass h ough s eam. A e co ec ing he scale, o a ion and posi ion o ou poin cloud ( ela i e o he physical wo ld), we we e able o c ea e an in e es ing pa allelism be ween he eal and i ual wo lds, allowing us o expe ience wo uni e ses simul aneously, so o speak ( igu e 140). Figu e 140: POV o he Ques ’s pass h ough es , showing an o e lapped ep esen a ion o wo eali ies. 119 Figu e 141: In-si u equipmen se up. An a emp o explo e he possibili ies o his idea was conduc ed in-si u ( igu e 141), howe e , due o echnical issues ela ed wi h he low po abili y and powe equi emen s o ou se up, we we e no able o ca y ou ou plan o ac ion wi hin he ime-span o his hesis. Ne e heless, igu e 142 shows a simula ion o wha ou in- si u a ian could ha e looked like, which is some hing ha may be o in e es o u u e explo a ion. 120 Figu e 142: A simula ion o he in-si u a ian using he Ques 2 (POV), whe e one could iew he scene’s hidden p ocesses, along wi h in e ac ing wi h said p ocesses h ough physical con ac . 8. Public Feedback E alua ion 128 No a all Pa icipan s Figu e 154: How well Kai os imp o ed ou olun ee s app ecia ion o he na u al wo ld. G ea ly These we e ollowed by a se o h ee mul i-choice linea scale ques ions conce ning Kai os’s imp ession on ou olun ee s. He e we asked how much he expe ience had imp o ed hei app ecia ion o he na u al wo ld, i i made hem eel any close o na u e, and i he expe ience had inc eased hei desi e o explo e na u e in he eal wo ld. On a scale o 1 ou o 10, he majo i y o ou olun ee s a ed Kai os as ha ing imp o ed hei app ecia ion o he na u al wo ld (a e age o 7.83), made hem eel close o na u e (a e age o 8.55), and an inc ease in desi e o explo e na u e in he eal wo ld (a e age o 8.27) ( igu es 154, 155 and 156). No a all Pa icipan s Figu e 155: How well Kai os made ou olun ee s eel close o na u e. Much close 129 Las bu no leas , a inal se o open ques ions we e gi en, asking i any o he issues we e expe ienced du ing un- ime, wha hey would add o he expe ience, ollowed by a sec ion o commen s and/o sugges ions. Some olun ee s exp essed hei desi e o mo e in e ac i e elemen s wi hin he scenes, along wi h a guide (o a mini- map) o show he use whe e o go, as some would become los wi hin he a eas. Mo eo e , smalle sugges ions we e also le , such as he addi ion o a blue sky and s a s. No a all Pa icipan s Figu e 156: I Kai os inc eased ou olun ee s desi e o explo e na u e in eal li e. Ve y much so 9. Making Kai os Publicly A ailable 132 Showcase ma e ial was p epa ed wi h he in en ion o cap i a ing he public’s in e es and o be sha ed wi h he Bo anical Ga den’s social media pla o ms and web pages. This ask go ini ia ed h ough he de elopmen o a websi e (h ps://kai os dei uc p /), whe e we explain he ideas and mo i a ion behind ou p ojec as a b ie in oduc ion, igu es depic ing how i go de eloped, wo download links o bo h Desk op and VR a ian s ( igu e 157), along wi h ou pape ha go accep ed a he In e na ional Con e ence on En e ainmen Compu ing. Figu e 157: Sc een cap u es o Kai os websi e. Top: on page. Bo om: Download sec ion con aining links o bo h PC and VR a ian s. A ailable a : h ps://kai os dei uc p / 133 In addi ion o his websi e, and due o he ha dwa e equi emen s necessa y o un Kai os, we elec ed o c ea e ideos consis ing o s e eoscopic (5K esolu ion) and monoscopic (8K esolu ion) 360 deg ee ideos o ou cap u ed a eas. The s e eoscopic a ian being aimed o he Oculus Ques 2 and YouTube VR, and he monoscopic a ian o be downloaded and iewed locally on a PC (a ailable a h ps://kai os dei uc p /). These ideos consis o wo minu e long passages h ough ou cap u ed a eas, which we e c ea ed by se ing he scene came as o ollow a p e-de e mined bézie cu e wi hin each a ea ( igu e 158). To maximise he ou pu quali y o ou ideos, i was decided o ende ou scenes as PNG image sequences. This encompassed he use o Uni y’s in-buil Reco de , whe e ou came a was se o ende a 360 deg ee iew o i s su oundings, wi h an ou pu esolu ion o 8192 x 8192 pixels (s e eoscopic o e /unde o ma ), wi h a Cube Map Size o 4096 pixels and a ame a e o 60 FPS ( igu e 159). Figu e 158: Sc een cap u e wi hin Uni y, showing he came a’s p e-de e mined pa h o 360 deg ee cap u e. 134 Figu e 159: The esul ing s e eoscopic (o e / unde ) 360-deg ee image ende o he Tilia, wi h a esolu ion o 8192 x 8192 pixels and a s e eo sepa a ion o 0.065. These image sequences we e hen impo ed in o P emie e P o and ans o med in o ideos. In an a emp o gua an ee smoo h ideo playback o he s e eoscopic a ian , o he pla o ms men ioned p e iously, he H.265 ideo codec was used a a bi a e o 120 Mbps. Highe bi a es we e es ed in he e o o imp o e he isual quali y o ou ideos, howe e , hese would cause ideo playback issues wi hin he Ques 2 and YouTube. Monoscopic 360 deg ee ideos we e subsequen ly ende ed, using he same image sequences, bu his ime a a highe bi a e o 400 Mbps. This was done o maximise ideo quali y, howe e , hese a e aimed solely o local (Desk op) ideo playback. 135 On a ela ed no e, a coope a ion was de eloped be ween Kai os and he Depa men o In o ma ics Enginee ing’s “PlugNPlay” display sys em. This ecen ly implemen ed sys em consis s o an a ay o nine e ically s acked and in e connec ed moni o s, p o iding he capaci y o displaying con en a a maximum esolu ion o 9720 x 1920 pixels ( igu e 160). The endea ou o execu ing Kai os on his sys em s emmed om a mu ual cu iosi y o de e mine i Kai os could success ully be un on his high esolu ion display a a s able ame a e o 60 FPS ( igu e 161). Su p isingly his was he case, which lead o a new ask: o c ea e a Kai os a ian as a demo. o he inaugu a ion o PlugNPlay. Figu e 160: PlugNPlay’s powe house, used o eed i s nine displays, wi h he help o i s RTX 3080 Ti g aphics ca d. Figu e 161: Ini ial es ing o Kai os on he PlugNPlay sys em. A ew adjus men s needed o be made in o de o ully ha ness he capabili ies o PlugNPlay, his included adjus ing he scene’s came as FOV (Field O View), o encompass he display’s wide aspec a io (app oxima ely 5:1) ( igu e 162), along wi h posi ioning mul iple s a iona y came as, a di e en angles, wi hin each scene ( o alling wo pe scene). A sc ip was hen added o au oma ically (o manually) ansi ion be ween hese came as and scenes, simila o li e wallpape backg ound ( igu e 163). 136 Las ly, a ease aile was c ea ed, consis ing o an in e laced in e play be ween ideo oo age om he Bo anical Ga den wi h i s digi al poin cloud ep esen a ion (a ailable a : h ps://kai os dei uc p / ). A cus om audio ack, c ea ed by Coimb a-based music a is Samuel Cadima, was also added, which couples wi h a oice-o e , as a p elude o he na u al p ocesses occu ing a he Bo anical Ga den o he Uni e si y o Coimb a. Figu e 162: P e- es ing Kai os PlugNPlay a ian wi hin Uni y, a a esolu ion o 9720 x 1920. Figu e 163: Kai os unning on he PlugNPlay sys em. 10. Conclusion