The Imperial Cathedral in Königslutter (Germany) as an Immersive Experience in Virtual Reality with Integrated 360° Panoramic Photography

applied sciences A icle The Impe ial Ca hed al in Königslu e (Ge many) as an Imme si e Expe ience in Vi ual Reali y wi h In eg a ed 360◦Pano amic Pho og aphy Alexande P. Walmsley and Thomas P. Ke s en * Ha enCi y Uni e si y Hambu g, Pho og amme y & Lase Scanning lab, Übe seeallee 16, 20457 Hambu g, Ge many; Alexande .Walmsley@hcu-hambu g.de *Co espondence: Thomas.Ke s en@hcu-hambu g.de Recei ed: 6 Janua y 2020; Accep ed: 7 Feb ua y 2020; Published: 23 Feb ua y 2020   Abs ac : As i ual eali y (VR) and he co esponding 3D documen a ion and modelling echnologies e ol e in o inc easingly powe ul and es ablished ools o nume ous applica ions in a chi ec u e, monumen p ese a ion, conse a ion/ es o a ion and he p esen a ion o cul u al he i age, new me hods o c ea ing in o ma ion- ich in e ac i e 3D en i onmen s a e inc easingly in demand. In his a icle, we desc ibe he de elopmen o an imme si e i ual eali y applica ion o he Impe ial Ca hed al in Königslu e , in which 360 ◦ pano amic pho og aphs we e in eg a ed wi hin he i ual en i onmen as a no el and complemen a y o m o isualiza ion. The Impe ial Ca hed al (Kaise dom) o Königslu e is one o he mos impo an examples o Romanesque a chi ec u e no h o he Alps. The Ca hed al had p e iously been subjec ed o lase -scanning and eco ding wi h 360 ◦ pano amic pho og aphy by he Pho og amme y & Lase Scanning lab o Ha enCi y Uni e si y Hambu g in 2010. Wi h he ecen apid de elopmen o consume VR echnology, i was subsequen ly decided o in es iga e how hese wo da a sou ces could be combined wi hin an imme si e VR applica ion o ou ism and o a chi ec u al he i age p ese a ion. A specialised echnical wo k low was de eloped o build he i ual en i onmen in Un eal Engine 4 (UE4) and in eg a e he pano ama pho og aphs so as o ensu e he seamless in eg a ion o hese wo da ase s. A simple mechanic was de eloped using he na i e UE4 node-based p og amming language o swi ch be ween hese wo modes o isualisa ion. Keywo ds: 3D modelling; 3D ep esen a ion; game engine; lase scanning; pano amic pho og aphy; i ual eali y 1. In oduc ion Vi ual eali y has ecen ly become a much b oade ield, inding applica ions in medicine, a chi ec u e, mili a y aining, and cul u al he i age, among o he ields. Wi h his g ow h has come some disc epancy in he de ini ion o he medium: while in some ields VR is used o e e o 360 ◦ imme si e pano amas and ideos, in o he ields i e e s o ully- ealised in e ac i e CGI en i onmen s. These wo “kinds” o VR ha e adi ionally been app oached e y di e en ly, owing o highly di e ging wo k lows and he di e en da a sou ces equi ed. Howe e , he e a e cu en ly no e ec i e ways o b inging oge he hese wo kinds o da a (each o which ha e hei own complemen a y ad an ages in isualisa ion) in o a single VR applica ion. This is pa icula ly impo an o applica ions in cul u al he i age, whe e documen a ion o en akes he o m o mul iple di e en kinds o complemen a y da a (e.g., w i en, pho og aphic, 3D, ideo and ield eco dings, among o he o ms). Vi ual eali y is de ined by Me iam Webs e as “an a i icial en i onmen which is expe ienced h ough senso y s imuli (such as sigh s and sounds) p o ided by a compu e and in which one’s Appl. Sci. 2020,10, 1517; doi:10.3390/app10041517 www.mdpi.com/jou nal/applsci Appl. Sci. 2020,10, 1517 2 o 11 ac ions pa ially de e mine wha happens in he en i onmen ” [ 1 ]. This e y b oad de ini ion allows o mos mode n applica ions o VR o be aken in o accoun . Addi ional de ini ions may be ound in li e a u e by Dö ne e al. [2], F eina and O [3], and Po man e al. [4]. In he ollowing we p esen he de elopmen wo k low o a oom-scale i ual eali y expe ience o a cul u al he i age monumen which in eg a es a high- esolu ion CGI en i onmen wi h 360 ◦ pano amic pho og aphy, allowing he use o “ oggle” be ween he i ual and he eal en i onmen s om wi hin he VR headse . This implemen a ion has he ad an age o exploi ing he po en ial o he in e ac i i y o a eal- ime game engine en i onmen wi h he high- ideli y o high dynamic ange image (HDRI) pano amic pho og aphy. 2. Rela ed Wo k While much c edi o he gene aliza ion o VR echnology and i s inc easing accessibili y is due o he ideo game indus y, which has in es ed hea ily in pushing he indus y o wa d [ 5 ], VR is now being employed in a wide ange o disciplines. To da e, VR has been success ully used o , among o he applica ions, i ual su ge y, i ual he apy, and ligh and ehicle simula ions. In he ield o cul u al he i age, VR has been ins umen al in he de elopmen o he ield o i ual he i age [ 6 – 8 ]. A he Ha enCi y Uni e si y Hambu g, se e al VR p ojec s conce ning his subjec ha e al eady been ealized. The own museum in Bad Segebe g, housed in a 17 h-cen u y ownhouse, was digi ally cons uc ed o a VR expe ience using he HTC Vi e P o [ 9 ]. Th ee his o ical ci ies (as well as hei su ounding en i onmen s) ha e been de eloped as VR expe iences: Duisbu g in 1566 [ 10 ], Segebe g in 1600 [ 11 ], and S ade in 1620 [ 12 ]. In addi ion, wo eligious and cul u al monumen s a e also a ailable as VR expe iences: he Selimiye Mosque in Edi ne, Tu key [ 13 ], and a wooden model o Solomon’s Temple [14]. The amoun o wo k speci ically ega ding he eal- ime VR isualiza ion o cul u al he i age monumen s is cu en ly limi ed bu g owing. Recen museum exhibi s using eal- ime VR o isualize cul u al he i age include Ba a ia 1627 a he Wes ies Museum in Hoo n, Ne he lands [ 15 ], and Viking VR, de eloped o accompany an exhibi a he B i ish Museum [ 16 ]. A numbe o ecen esea ch p ojec s also ocus on he use o VR o cul u al he i age isualiza ion [ 17 – 20 ], as well as on aspec s beyond isualisa ion, including ec ea ing he physical en i onmen al s imuli [ 21 ]. The cu en pape con ibu es o his g owing discussion by seeking o in eg a e 360 ◦ pano ama pho og aphs wi hin an imme si e eal- ime isualiza ion o a cul u al he i age monumen . A his s age, only e y limi ed wo k ega ding pano amic pho og aphy in eg a ion in eal- ime VR is known o he au ho s [22]. 3. The Impe ial Ca hed al (Kaise dom) in Königslu e , Ge many The own o Königslu e , some 20 km eas o B aunschweig (Lowe Saxony, Ge many), is domina ed by he Impe ial Ca hed al, known in Ge man as he Kaise dom (Figu e 1). One o he mos imp essi e examples o Romanesque a chi ec u e no h o he alps, he ca hed al’s cons uc ion was begun unde he di ec ion o Kaise Lo ha III, Ge man empe o om 1133 onwa ds [ 23 , 24 ]. The chu ch was buil in he o m o a h ee-aisled c oss-shaped column basilica. The ca hed al is no able pa icula ly o i s epea ed a chi ec u al e e ences o no he n I alian a chi ec u al s yles o he ime, indica ing ha i migh be he wo k o an I alian a chi ec o indeed someone who was well- a elled in hose egions. Among he mos impo an ea u es o he ca hed al is an o namen al hun ing ieze, which hugs he ex e nal wall o he main aisle (see Figu e 1cen e). Be ween 2002 and 2006, es o a ion was ca ied ou on he ex e io o he ca hed al, ollowed by he in e io be ween 2006 and 2010. The ca hed al measu es 75 m in leng h, 42 m in wid h, and 56 m in heigh . Appl. Sci. 2020,10, 1517 3 o 11 Appl. Sci. 2020, 10, x FOR PEER REVIEW 3 o 11 Figu e 1. Pano amic iew o he Impe ial Ca hed al in Königslu e , Ge many ( op), he hun ing ieze on he ex e nal wall o he main apsis o he ca hed al wi h s o y- elling igu es (cen e), and a pano amic iew o he in e io o he ca hed al (bo om). 4. Me hodology 4.1. P ojec Wo k low The o e all wo k low o he p oduc ion o he VR expe ience o he Kaise dom is schema ically ep esen ed in Figu e 2. Special ocus was gi en o achie ing a ealis ic 1:1 ep esen a ion o he ca hed al, including he in eg a ion o pano amic pho os in he VR expe ience (see Sec ion 4.6). The p ojec was di ided in o i e majo phases o de elopmen (Figu e 2): (1a) da a acquisi ion by e es ial lase scanning wi h one Riegl VZ-400 scanne (ou side) and wo Zolle + F öhlich IMAGER 5006 scanne s (inside), (1b) egis a ion and geo- e e encing o scans using RiScan P o and Lase Con ol, (1c) segmen a ion o poin clouds in o objec iles, (2a) 3D solid modelling wi h Au oCAD using segmen ed poin clouds, (2b) gene a ion o pano amic images using PTGui, (3) ex u e mapping o polygon models using Au odesk Maya and Subs ance Pain e , (4a) placemen o meshes and building he scene wi hin he UE4 game engine, (4b) in eg a ion o mo ion con ol and in e ac ions in UE4, (4c) in eg a ion o 360° pano amic image y, and (5) imme si e and in e ac i e isualisa ion o he ca hed al in he VR sys em HTC Vi e P o using S eam VR 2.0 as an in e ace be ween he game engine and he Head Moun ed Display (HMD). Figu e 1. Pano amic iew o he Impe ial Ca hed al in Königslu e , Ge many ( op ), he hun ing ieze on he ex e nal wall o he main apsis o he ca hed al wi h s o y- elling igu es ( cen e ), and a pano amic iew o he in e io o he ca hed al (bo om). 4. Me hodology 4.1. P ojec Wo k low The o e all wo k low o he p oduc ion o he VR expe ience o he Kaise dom is schema ically ep esen ed in Figu e 2. Special ocus was gi en o achie ing a ealis ic 1:1 ep esen a ion o he ca hed al, including he in eg a ion o pano amic pho os in he VR expe ience (see Sec ion 4.6). The p ojec was di ided in o i e majo phases o de elopmen (Figu e 2): (1a) da a acquisi ion by e es ial lase scanning wi h one Riegl VZ-400 scanne (ou side) and wo Zolle +F öhlich IMAGER 5006 scanne s (inside), (1b) egis a ion and geo- e e encing o scans using RiScan P o and Lase Con ol, (1c) segmen a ion o poin clouds in o objec iles, (2a) 3D solid modelling wi h Au oCAD using segmen ed poin clouds, (2b) gene a ion o pano amic images using PTGui, (3) ex u e mapping o polygon models using Au odesk Maya and Subs ance Pain e , (4a) placemen o meshes and building he scene wi hin he UE4 game engine, (4b) in eg a ion o mo ion con ol and in e ac ions in UE4, (4c) in eg a ion o 360 ◦ pano amic image y, and (5) imme si e and in e ac i e isualisa ion o he ca hed al in he VR sys em HTC Vi e P o using S eam VR 2.0 as an in e ace be ween he game engine and he Head Moun ed Display (HMD). Appl. Sci. 2020,10, 1517 4 o 11 Appl. Sci. 2020, 10, x FOR PEER REVIEW 4 o 11 Figu e 2. Wo k low o he de elopmen o he i ual eali y (VR) expe ience. 4.2. Da a Acquisi ion The da a acquisi ion was al eady desc ibed in 2012 in Ke s en and Linds aed [25] and is summa ised in he ollowing. The lase scan da a o he Kaise dom was acqui ed a 55 s a ions inside he ca hed al by wo Zolle + F oehlich IMAGER 5006 (www.z -lase .com) e es ial lase scanne s, and a 8 s a ions ou side he ca hed al by one Riegl VZ-400 (www. iegl.com) on 5 Janua y and 23 June 2010 (Figu e 3). In o al, he scanning ook 15 h. The scanning esolu ion was se o high (6 mm @ 10 m) o he IMAGER 5006 and o 5 mm a objec space o he Riegl VZ-400. The p ecision o he geode ic ne wo k s a ions was 2.5 mm, while he p ecision o he con ol poin s o lase scanning was 5 mm. In o de o la e colou ise he poin cloud, as well as o he building o he i ual ou , 360° pano amic pho os we e aken a each IMAGER 5006 scan s a ion and a a ew supplemen a y s a ions using a Nikon DSLR came a (see Sec ion 3.4). Figu e 3. Geode ic 3D ne wo k (blue and g een lines) and posi ion o he scan s a ions (IMAGER 5006 = yellow iangles, Riegl VZ-400 = ed do s) a he ca hed al (le ), Riegl VZ-400 poin cloud o ( op igh ) and 2D p esen a ion o an IMAGER 5006 scan (bo om igh ). 4.3. 3D Modelling The 3D modelling was also desc ibed in 2012 in Ke s en and Linds aed [25] and is b ie ly summa ised in he ollowing. The gene a ed poin cloud, being oo la ge o impo di ec ly in o a CAD p og am, was i s segmen ed and hen ans e ed o Au oCAD using he plugin Poin Cloud. Once impo ed, he ca hed al was blocked ou manually wi h a 3D mesh by ex uding polylines along he su aces and edges o he poin cloud s uc u e. This me hod has he ad an age o no gene a ing Figu e 2. Wo k low o he de elopmen o he i ual eali y (VR) expe ience. 4.2. Da a Acquisi ion The da a acquisi ion was al eady desc ibed in 2012 in Ke s en and Linds aed [ 25 ] and is summa ised in he ollowing. The lase scan da a o he Kaise dom was acqui ed a 55 s a ions inside he ca hed al by wo Zolle +F oehlich IMAGER 5006 (www.z -lase .com) e es ial lase scanne s, and a 8 s a ions ou side he ca hed al by one Riegl VZ-400 (www. iegl.com) on 5 Janua y and 23 June 2010 (Figu e 3). In o al, he scanning ook 15 h. The scanning esolu ion was se o high (6 mm @ 10 m) o he IMAGER 5006 and o 5 mm a objec space o he Riegl VZ-400. The p ecision o he geode ic ne wo k s a ions was 2.5 mm, while he p ecision o he con ol poin s o lase scanning was 5 mm. In o de o la e colou ise he poin cloud, as well as o he building o he i ual ou , 360 ◦ pano amic pho os we e aken a each IMAGER 5006 scan s a ion and a a ew supplemen a y s a ions using a Nikon DSLR came a (see Sec ion 4.4). Appl. Sci. 2020, 10, x FOR PEER REVIEW 4 o 11 Figu e 2. Wo k low o he de elopmen o he i ual eali y (VR) expe ience. 4.2. Da a Acquisi ion The da a acquisi ion was al eady desc ibed in 2012 in Ke s en and Linds aed [25] and is summa ised in he ollowing. The lase scan da a o he Kaise dom was acqui ed a 55 s a ions inside he ca hed al by wo Zolle + F oehlich IMAGER 5006 (www.z -lase .com) e es ial lase scanne s, and a 8 s a ions ou side he ca hed al by one Riegl VZ-400 (www. iegl.com) on 5 Janua y and 23 June 2010 (Figu e 3). In o al, he scanning ook 15 h. The scanning esolu ion was se o high (6 mm @ 10 m) o he IMAGER 5006 and o 5 mm a objec space o he Riegl VZ-400. The p ecision o he geode ic ne wo k s a ions was 2.5 mm, while he p ecision o he con ol poin s o lase scanning was 5 mm. In o de o la e colou ise he poin cloud, as well as o he building o he i ual ou , 360° pano amic pho os we e aken a each IMAGER 5006 scan s a ion and a a ew supplemen a y s a ions using a Nikon DSLR came a (see Sec ion 3.4). Figu e 3. Geode ic 3D ne wo k (blue and g een lines) and posi ion o he scan s a ions (IMAGER 5006 = yellow iangles, Riegl VZ-400 = ed do s) a he ca hed al (le ), Riegl VZ-400 poin cloud o ( op igh ) and 2D p esen a ion o an IMAGER 5006 scan (bo om igh ). 4.3. 3D Modelling The 3D modelling was also desc ibed in 2012 in Ke s en and Linds aed [25] and is b ie ly summa ised in he ollowing. The gene a ed poin cloud, being oo la ge o impo di ec ly in o a CAD p og am, was i s segmen ed and hen ans e ed o Au oCAD using he plugin Poin Cloud. Once impo ed, he ca hed al was blocked ou manually wi h a 3D mesh by ex uding polylines along he su aces and edges o he poin cloud s uc u e. This me hod has he ad an age o no gene a ing Figu e 3. Geode ic 3D ne wo k (blue and g een lines) and posi ion o he scan s a ions (IMAGER 5006 = yellow iangles, Riegl VZ-400 = ed do s) a he ca hed al ( le ), Riegl VZ-400 poin cloud o ( op igh ) and 2D p esen a ion o an IMAGER 5006 scan (bo om igh ). 4.3. 3D Modelling The 3D modelling was also desc ibed in 2012 in Ke s en and Linds aed [ 25 ] and is b ie ly summa ised in he ollowing. The gene a ed poin cloud, being oo la ge o impo di ec ly in o a CAD Appl. Sci. 2020,10, 1517 5 o 11 p og am, was i s segmen ed and hen ans e ed o Au oCAD using he plugin Poin Cloud. Once impo ed, he ca hed al was blocked ou manually wi h a 3D mesh by ex uding polylines along he su aces and edges o he poin cloud s uc u e. This me hod has he ad an age o no gene a ing oo la ge a ile, while e aining isual con ol o he buil model using a supe imposed poin cloud. Figu e 4 shows he inal cons uc ed 3D CAD model o he en i e ca hed al in ou di e en pe spec i e iews. Appl. Sci. 2020, 10, x FOR PEER REVIEW 5 o 11 oo la ge a ile, while e aining isual con ol o he buil model using a supe imposed poin cloud. Figu e 4 shows he inal cons uc ed 3D CAD model o he en i e ca hed al in ou di e en pe spec i e iews. Fo some smalle de ails on he ca hed al, he au oma ed meshing unc ions in Geomagic we e used o quickly gene a e a mesh di ec ly om he poin cloud (Figu e 5). This wo ks by means o a simple iangula ion algo i hm, which wo ks be e o mo e complex and i egula shapes and su aces. Figu e 4. Cons uc ed 3D model o he impe ial ca hed al in Königslu e , Ge many—View o he ou on s in Au oCAD. Figu e 5. Gene a ion o small complex objec s o he ca hed al using he meshing unc ion in Geomagic o he segmen ed poin clouds. 4.4. Pano amic Pho og aphy In o de o subsequen ly colou ise he poin cloud, as well as o gene a e a i ual online ou o he ca hed al, a se ies o 360° pano amic pho os we e aken a each IMAGER 5006 scan s a ion using a Nikon DSLR came a wi h a nodal poin adap e . Supplemen a y pano amic pho os we e also aken a 10 addi ional loca ions ou side he ca hed al, as well as 19 u he poin s wi hin he ca hed al. These we e aken wi hou any lase -scanning a ge s o ex aneous objec s p esen in he sho . The acquisi ion and p ocessing o he pano amic pho og aphy was also desc ibed in 2012 in Ke s en and Linds aed [25]. Fo be e unde s anding o he whole wo k low, he p ocessing o he pano amic pho og aphy is b ie ly summa ised in he ollowing. Each se o pho og aphs consis s o 16 images— Figu e 4. Cons uc ed 3D model o he impe ial ca hed al in Königslu e , Ge many—View o he ou on s in Au oCAD. Fo some smalle de ails on he ca hed al, he au oma ed meshing unc ions in Geomagic we e used o quickly gene a e a mesh di ec ly om he poin cloud (Figu e 5). This wo ks by means o a simple iangula ion algo i hm, which wo ks be e o mo e complex and i egula shapes and su aces. Appl. Sci. 2020, 10, x FOR PEER REVIEW 5 o 11 oo la ge a ile, while e aining isual con ol o he buil model using a supe imposed poin cloud. Figu e 4 shows he inal cons uc ed 3D CAD model o he en i e ca hed al in ou di e en pe spec i e iews. Fo some smalle de ails on he ca hed al, he au oma ed meshing unc ions in Geomagic we e used o quickly gene a e a mesh di ec ly om he poin cloud (Figu e 5). This wo ks by means o a simple iangula ion algo i hm, which wo ks be e o mo e complex and i egula shapes and su aces. Figu e 4. Cons uc ed 3D model o he impe ial ca hed al in Königslu e , Ge many—View o he ou on s in Au oCAD. Figu e 5. Gene a ion o small complex objec s o he ca hed al using he meshing unc ion in Geomagic o he segmen ed poin clouds. 4.4. Pano amic Pho og aphy In o de o subsequen ly colou ise he poin cloud, as well as o gene a e a i ual online ou o he ca hed al, a se ies o 360° pano amic pho os we e aken a each IMAGER 5006 scan s a ion using a Nikon DSLR came a wi h a nodal poin adap e . Supplemen a y pano amic pho os we e also aken a 10 addi ional loca ions ou side he ca hed al, as well as 19 u he poin s wi hin he ca hed al. These we e aken wi hou any lase -scanning a ge s o ex aneous objec s p esen in he sho . The acquisi ion and p ocessing o he pano amic pho og aphy was also desc ibed in 2012 in Ke s en and Linds aed [25]. Fo be e unde s anding o he whole wo k low, he p ocessing o he pano amic pho og aphy is b ie ly summa ised in he ollowing. Each se o pho og aphs consis s o 16 images— Figu e 5. Gene a ion o small complex objec s o he ca hed al using he meshing unc ion in Geomagic o he segmen ed poin clouds. 4.4. Pano amic Pho og aphy In o de o subsequen ly colou ise he poin cloud, as well as o gene a e a i ual online ou o he ca hed al, a se ies o 360 ◦ pano amic pho os we e aken a each IMAGER 5006 scan s a ion Appl. Sci. 2020,10, 1517 6 o 11 using a Nikon DSLR came a wi h a nodal poin adap e . Supplemen a y pano amic pho os we e also aken a 10 addi ional loca ions ou side he ca hed al, as well as 19 u he poin s wi hin he ca hed al. These we e aken wi hou any lase -scanning a ge s o ex aneous objec s p esen in he sho . The acquisi ion and p ocessing o he pano amic pho og aphy was also desc ibed in 2012 in Ke s en and Linds aed [ 25 ]. Fo be e unde s anding o he whole wo k low, he p ocessing o he pano amic pho og aphy is b ie ly summa ised in he ollowing. Each se o pho og aphs consis s o 16 images—one poin ing owa ds he sky, h ee owa ds he g ound and 12 pho os o he 360 ◦ ci cle in he ho izon al posi ion. The so wa e PTGui au oma ically gene a ed a sphe ical pano ama wi h 11,700 ×5850 pixels (ca. 43 MB) o each came a s a ion. These pano ama images we e con e ed in o a se o six cube images (in o al ca. 5 MB). The pano ama iewing so wa e KRpano (h ps://k pano.com) was ini ially used o gene a e an in e ac i e i ual ou o web b owse s (Figu e 6). The ou can be iewed a h ps://www.koenigslu e -kaise dom.de/ i uelleTou / ou .h ml (mus ha e Adobe Flash 9/10 enabled). In his b owse -based ou , all sphe ical pano amas a e linked o each o he ia ho spo s o ia he o e iew map (bo om- igh co ne ). This p o ides a quick and con enien way o na iga ing h ough he pano amas, simply by clicking on he ele an map icon. Appl. Sci. 2020, 10, x FOR PEER REVIEW 6 o 11 one poin ing owa ds he sky, h ee owa ds he g ound and 12 pho os o he 360° ci cle in he ho izon al posi ion. The so wa e PTGui au oma ically gene a ed a sphe ical pano ama wi h 11,700 × 5850 pixels (ca. 43 MB) o each came a s a ion. These pano ama images we e con e ed in o a se o six cube images (in o al ca. 5 MB). The pano ama iewing so wa e KRpano (h ps://k pano.com) was ini ially used o gene a e an in e ac i e i ual ou o web b owse s (Figu e 6). The ou can be iewed a h ps://www.koenigslu e -kaise dom.de/ i uelleTou / ou .h ml (mus ha e Adobe Flash 9/10 enabled). In his b owse -based ou , all sphe ical pano amas a e linked o each o he ia ho spo s o ia he o e iew map (bo om- igh co ne ). This p o ides a quick and con enien way o na iga ing h ough he pano amas, simply by clicking on he ele an map icon. Figu e 6. In e ac i e i ual ou h ough he impe ial ca hed al using ull sphe ical pano ama pho og aphy on se e al s a ions inside and ou side o he building, including an o e iew map o s a ions (bo om- igh co ne ). 4.5. Game Engine Un eal and VR Sys em HTC Vi e A game engine is a simula ion en i onmen whe e 2D o 3D g aphics can be manipula ed h ough code. De eloped p ima ily by he ideo games indus y, hey p o ide ideal pla o ms o he c ea ion o VR expe iences o o he pu poses (e.g., cul u al he i age), as many o he necessa y unc ionali ies a e al eady buil in, elimina ing he need o enginee hese ea u es independen ly. While he e a e dozens o app op ia e game engines ha could be used, he mos popula o small s udios and p oduc ion eams end o be he Uni y engine (Uni y Technologies, San F ancisco, Cali o nia, USA), C yEngine (C y ek, F ank u am Main, Ge many) and Un eal Engine (Epic Games, Ca y, No h Ca olina, USA). Fo his p ojec , he Un eal Engine was chosen o i s ad an age in he buil -in bluep in s isual coding sys em, which allows use s o build in simple in e ac ions and anima ions wi hou any p io knowledge o C++, he p og amming language on which he engine is buil [26]. The speci ic ha dwa e equi ed o un VR is a VR headse , wo “ligh house” base s a ions, wo con olle s, and a VR- eady PC. Fo his p ojec , he HTC Vi e P o was chosen as a headse . The ligh houses a e needed o ack he use ’s mo emen in 3D space (Figu e 7), while he con olle s a e used o mapping in e ac ions in he i ual wo ld. T acking is achie ed wi h a gy oscope, accele ome e , and lase posi ion senso wi hin he VR headse i sel , and can de ec mo emen s wi h an accu acy o 0.1° [27]. Figu e 7 shows he se up o he VR sys em HTC Vi e P o, including he in e ac ion a ea (blue) o he use . Figu e 6. In e ac i e i ual ou h ough he impe ial ca hed al using ull sphe ical pano ama pho og aphy on se e al s a ions inside and ou side o he building, including an o e iew map o s a ions (bo om- igh co ne ). 4.5. Game Engine Un eal and VR Sys em HTC Vi e A game engine is a simula ion en i onmen whe e 2D o 3D g aphics can be manipula ed h ough code. De eloped p ima ily by he ideo games indus y, hey p o ide ideal pla o ms o he c ea ion o VR expe iences o o he pu poses (e.g., cul u al he i age), as many o he necessa y unc ionali ies a e al eady buil in, elimina ing he need o enginee hese ea u es independen ly. While he e a e dozens o app op ia e game engines ha could be used, he mos popula o small s udios and p oduc ion eams end o be he Uni y engine (Uni y Technologies, San F ancisco, Cali o nia, USA), C yEngine (C y ek, F ank u am Main, Ge many) and Un eal Engine (Epic Games, Ca y, No h Ca olina, USA). Fo his p ojec , he Un eal Engine was chosen o i s ad an age in he buil -in bluep in s isual coding sys em, which allows use s o build in simple in e ac ions and anima ions wi hou any p io knowledge o C++, he p og amming language on which he engine is buil [26]. Appl. Sci. 2020,10, 1517 7 o 11 The speci ic ha dwa e equi ed o un VR is a VR headse , wo “ligh house” base s a ions, wo con olle s, and a VR- eady PC. Fo his p ojec , he HTC Vi e P o was chosen as a headse . The ligh houses a e needed o ack he use ’s mo emen in 3D space (Figu e 7), while he con olle s a e used o mapping in e ac ions in he i ual wo ld. T acking is achie ed wi h a gy oscope, accele ome e , and lase posi ion senso wi hin he VR headse i sel , and can de ec mo emen s wi h an accu acy o 0.1 ◦ [ 27 ]. Figu e 7shows he se up o he VR sys em HTC Vi e P o, including he in e ac ion a ea (blue) o he use . Appl. Sci. 2020, 10, x FOR PEER REVIEW 7 o 11 Figu e 7. Componen s and schema ic se up o he VR sys em HTC Vi e P o wi h in e ac ion a ea (blue). 4.6. Implemen a ion in Vi ual Reali y In o de o b ing he model in o i ual eali y, some changes had o be made o he mesh and ex u es in o de o make hem un mo e e icien ly wi hin he game engine. The s ic pe o mance c i e ia o VR mean ha e e y e o needs o be made o op imize he models and ensu e ha a su icien ly high ame a e (ideally 90 ames pe second, hough o many applica ions abo e 40 is su icien ) can be achie ed. Much o his pa o he wo k low was done manually. Fi s , he mesh was spli in o di e en pa s in o de o make he da a olume o he iles smalle and he e o e speed up he ime aken o each i e a ion o he ex u ing p ocess. Because UE4’s buil - in ende engine ende s only hose meshes and ex u es ha a e wi hin he sc een-space o he iewe a any one ime, a logical app oach is o sepa a e he in e io om he ex e io meshes, so as o unload he ex e io da a when he use is inside he ca hed al and ice e sa when hey a e ou side. The wo p incipal pa s o he Kaise dom— he cen al na e and he clois e s—we e also p ocessed sepa a ely o he same eason. In a ew a eas o he model, such as he sou he n side o he clois e , addi ional modelling was done in o de o supplemen he scan da a. A low-poly ci y model p o ided by he company CPA So wa e GmbH (Siegbu g, Ge many) was used as a basis o model low-poly buildings in he a ea a ound he Ca hed al. As hese buildings we e no cen al o he expe ience, hey we e modelled only in low de ail so as no o ake up oo much ende ing space on he GPU. Buildings u he away om he Ca hed al, which we e only isible on he pe iphe y o he i ual en i onmen , we e le in hei aw o m (simple g ey ec angula meshes) o a oid any ex aneous modelling wo k. Much o he wo k in he VR op imiza ion p ocess was dedica ed o he p oduc ion o high- quali y ex u es sui able o eal- ime VR. The e is a undamen al ade-o he e be ween he quali y o he ex u es needed o appea pho o ealis ic a close ange and he da a s eaming limi o he engine (which a ies due o ha dwa e and so wa e speci ica ions). As a ule, c ea ing a pho o ealis ic en i onmen o VR equi es high-quali y ex u es in o de o boos he expe ience o imme sion. While he Un eal Engine au oma ically implemen s le el-o -de ail algo i hms o educe he load on he ende engine, a ce ain amoun o manual op imiza ion mus be done in addi ion o achie e pe o mance goals. As such, ex u e esolu ion was a ied depending on how a he ex u e would Figu e 7. Componen s and schema ic se up o he VR sys em HTC Vi e P o wi h in e ac ion a ea (blue). 4.6. Implemen a ion In Vi ual Reali y In o de o b ing he model in o i ual eali y, some changes had o be made o he mesh and ex u es in o de o make hem un mo e e icien ly wi hin he game engine. The s ic pe o mance c i e ia o VR mean ha e e y e o needs o be made o op imize he models and ensu e ha a su icien ly high ame a e (ideally 90 ames pe second, hough o many applica ions abo e 40 is su icien ) can be achie ed. Much o his pa o he wo k low was done manually. Fi s , he mesh was spli in o di e en pa s in o de o make he da a olume o he iles smalle and he e o e speed up he ime aken o each i e a ion o he ex u ing p ocess. Because UE4’s buil -in ende engine ende s only hose meshes and ex u es ha a e wi hin he sc een-space o he iewe a any one ime, a logical app oach is o sepa a e he in e io om he ex e io meshes, so as o unload he ex e io da a when he use is inside he ca hed al and ice e sa when hey a e ou side. The wo p incipal pa s o he Kaise dom— he cen al na e and he clois e s—we e also p ocessed sepa a ely o he same eason. In a ew a eas o he model, such as he sou he n side o he clois e , addi ional modelling was done in o de o supplemen he scan da a. A low-poly ci y model p o ided by he company CPA So wa e GmbH (Siegbu g, Ge many) was used as a basis o model low-poly buildings in he a ea a ound he Ca hed al. As hese buildings we e no cen al o he expe ience, hey we e modelled only in low de ail so as no o ake up oo much ende ing space on he GPU. Buildings u he away om he Ca hed al, which we e only isible on he pe iphe y o he i ual en i onmen , we e le in hei aw o m (simple g ey ec angula meshes) o a oid any ex aneous modelling wo k. Appl. Sci. 2020,10, 1517 8 o 11 Much o he wo k in he VR op imiza ion p ocess was dedica ed o he p oduc ion o high-quali y ex u es sui able o eal- ime VR. The e is a undamen al ade-o he e be ween he quali y o he ex u es needed o appea pho o ealis ic a close ange and he da a s eaming limi o he engine (which a ies due o ha dwa e and so wa e speci ica ions). As a ule, c ea ing a pho o ealis ic en i onmen o VR equi es high-quali y ex u es in o de o boos he expe ience o imme sion. While he Un eal Engine au oma ically implemen s le el-o -de ail algo i hms o educe he load on he ende engine, a ce ain amoun o manual op imiza ion mus be done in addi ion o achie e pe o mance goals. As such, ex u e esolu ion was a ied depending on how a he ex u e would be om eye-le el in he i ual en i onmen . 4K ex u es (4096 × 4096 px) we e used o high-de ail ex u es ha would appea a eye le el, while 2K ex u es (2048 × 2048 px) we e used o ex u es ha appea well abo e eye le el ( o example, he ceiling and oo ex u es). While many o he ex u es o his p ocess we e adap ed om pho os aken a he Kaise dom, supplemen a y pho o- ex u es we e sou ced om a c ea i e commons licensed CGI ex u e da abase (h ps:// ex u eha en.com/). Fo hose ma e ials wi h mo e exagge a ed elie , such as he building s one and oo iles, no mal maps we e also added and accen ua ed wi h a pa allax displacemen e ec buil wi h he na i e UE4 ma e ial c ea ion ools. The 3D models wi h hei co esponding ex u es we e expo ed in o UE4 o placemen and eal- ime isualiza ion (Figu e 8A,B). The e sion o UE4 used in his case was 4.22. Addi ional elemen s such as plan meshes, clouds, og, en i onmen al ligh ing, and audio we e added o heigh en he sense o pho o ealism and imme sion. In addi ion, simple in e ac ions we e in eg a ed in o de o help he use na iga e a ound he en i onmen . Fi s ly, a elepo a ion mechanic was implemen ed, allowing he use o jump om loca ion o loca ion. This mechanic makes use o a simple ay- ace , p e-buil in o UE4, ha allows he use o poin o any loca ion in he i ual wo ld and check ha he loca ion is a alid elepo a ion poin acco ding o a ce ain se o c i e ia ( hese c i e ia, including he space a ailable and he slope angle a he loca ion, a e calcula ed by UE4 wi h i s “Na iga ion Mesh” ea u e). I he loca ion is alid as a elepo a ion poin , he use can elepo he e wi h he click o he igge bu on on he con olle (Figu e 8D). In addi ion, au oma ic doo -opening anima ions we e added o se e al doo s in he ca hed al, allowing use s o mo e be ween di e en pa s o he building as in he eal wo ld. A sho aile o he i ual en i onmen can be iewed online (h ps://www.you ube.com/wa ch? =hmO0JOdlLgw). Once he eal- ime en i onmen was buil and VR in e ac ions se up, he 360 ◦ pano amas could be in eg a ed. A simple mechanism was implemen ed in he UE4 engine o make each pano ama iewable. This mechanism was made up o : (1) a isual clue in he i ual wo ld ha indica ed whe e he pano ama was loca ed. As an example we used a glowing ing, which s ands ou well om he es o he en i onmen (Figu e 8C)—a wide a ie y o o he isual clues may be app op ia e; (2) A igge box o e lapping wi h he ing, coupled wi h a unc ion ha i es when a ce ain bu on is p essed on he HTC Vi e mo ion con olle ; (3) A sepa a e, emp y map o le el in he UE4 edi o ; and (4) A skybox in he emp y le el on o which o p ojec he cube-map pano ama. Using his mechanism, he playe can app oach a glowing ing, ep esen ing a pano ama aken on ha spo , p ess a bu on on he mo ion con olle , and be anspo ed in o he 360 ◦ pano ama. By oggling he bu on p ess on he mo ion con olle , he playe can come ou o he pano ama and be placed back in he i ual wo ld (Figu e 9). Ce ain a ia ions in his mechanic we e es ed (e.g., p ojec ing he pano amic pho o on he inside o a sphe e in he i ual wo ld, hen using a bu on on he mo ion con olle o al e na ely showing and hiding his sphe e when he playe was in he igh a ea), bu he me hod desc ibed abo e was ound o p o ide he simples and mos obus way o oggling be ween he pano amic pho os in he scene while e aining he o iginal pe spec i e o he pho og aphs. The inished e sion o he VR expe ience was es ed wi h he HTC Vi e P o headse unning on a PC wi h an 8-Co e In el Xeon CPU (3.10 GHz), an NVIDIA GTX 1070i GPU, and 32.0 GB RAM. Wi h his se up, he expe ience achie ed an a e age ame a e o 40–50 ames pe second. Appl. Sci. 2020,10, 1517 9 o 11 Appl. Sci. 2020, 10, x FOR PEER REVIEW 8 o 11 be om eye-le el in he i ual en i onmen . 4K ex u es (4096 × 4096 px) we e used o high-de ail ex u es ha would appea a eye le el, while 2K ex u es (2048 × 2048 px) we e used o ex u es ha appea well abo e eye le el ( o example, he ceiling and oo ex u es). While many o he ex u es o his p ocess we e adap ed om pho os aken a he Kaise dom, supplemen a y pho o- ex u es we e sou ced om a c ea i e commons licensed CGI ex u e da abase (h ps:// ex u eha en.com/). Fo hose ma e ials wi h mo e exagge a ed elie , such as he building s one and oo iles, no mal maps we e also added and accen ua ed wi h a pa allax displacemen e ec buil wi h he na i e UE4 ma e ial c ea ion ools. Figu e 8. Two iews o he Kaise dom, inside (A) and ou side (B). A hi d image (C) shows an example o he elepo a ion mechanic in ac ion. Image (D) shows an example o a isual clue placed in he i ual wo ld, whe e a pano amic pho o can be iewed. The 3D models wi h hei co esponding ex u es we e expo ed in o UE4 o placemen and eal- ime isualiza ion (Figu e 8A,B). The e sion o UE4 used in his case was 4.22. Addi ional elemen s such as plan meshes, clouds, og, en i onmen al ligh ing, and audio we e added o heigh en he sense o pho o ealism and imme sion. In addi ion, simple in e ac ions we e in eg a ed in o de o help he use na iga e a ound he en i onmen . Fi s ly, a elepo a ion mechanic was implemen ed, allowing he use o jump om loca ion o loca ion. This mechanic makes use o a simple ay- ace , p e-buil in o UE4, ha allows he use o poin o any loca ion in he i ual wo ld and check ha he loca ion is a alid elepo a ion poin acco ding o a ce ain se o c i e ia ( hese c i e ia, including he space a ailable and he slope angle a he loca ion, a e calcula ed by UE4 wi h i s “Na iga ion Mesh” ea u e). I he loca ion is alid as a elepo a ion poin , he use can elepo he e wi h he click o he igge bu on on he con olle (Figu e 8D). In addi ion, au oma ic doo - opening anima ions we e added o se e al doo s in he ca hed al, allowing use s o mo e be ween di e en pa s o he building as in he eal wo ld. A sho aile o he i ual en i onmen can be iewed online (h ps://www.you ube.com/wa ch? =hmO0JOdlLgw). Once he eal- ime en i onmen was buil and VR in e ac ions se up, he 360° pano amas could be in eg a ed. A simple mechanism was implemen ed in he UE4 engine o make each pano ama iewable. This mechanism was made up o : (1) a isual clue in he i ual wo ld ha indica ed whe e he pano ama was loca ed. As an example we used a glowing ing, which s ands ou well om he es o he en i onmen (Figu e 8C)—a wide a ie y o o he isual clues may be app op ia e; (2) A igge box o e lapping wi h he ing, coupled wi h a unc ion ha i es when a ce ain bu on is p essed on he HTC Vi e mo ion con olle ; (3) A sepa a e, emp y map o le el in he UE4 edi o ; and AB C D Figu e 8. Two iews o he Kaise dom, inside ( A ) and ou side ( B ). A hi d image ( C ) shows an example o he elepo a ion mechanic in ac ion. Image ( D ) shows an example o a isual clue placed in he i ual wo ld, whe e a pano amic pho o can be iewed. Appl. Sci. 2020, 10, x FOR PEER REVIEW 9 o 11 (4) A skybox in he emp y le el on o which o p ojec he cube-map pano ama. Using his mechanism, he playe can app oach a glowing ing, ep esen ing a pano ama aken on ha spo , p ess a bu on on he mo ion con olle , and be anspo ed in o he 360° pano ama. By oggling he bu on p ess on he mo ion con olle , he playe can come ou o he pano ama and be placed back in he i ual wo ld (Figu e 9). Ce ain a ia ions in his mechanic we e es ed (e.g., p ojec ing he pano amic pho o on he inside o a sphe e in he i ual wo ld, hen using a bu on on he mo ion con olle o al e na ely showing and hiding his sphe e when he playe was in he igh a ea), bu he me hod desc ibed abo e was ound o p o ide he simples and mos obus way o oggling be ween he pano amic pho os in he scene while e aining he o iginal pe spec i e o he pho og aphs. The inished e sion o he VR expe ience was es ed wi h he HTC Vi e P o headse unning on a PC wi h an 8-Co e In el Xeon CPU (3.10 GHz), an NVIDIA GTX 1070i GPU, and 32.0 GB RAM. Wi h his se up, he expe ience achie ed an a e age ame a e o 40–50 ames pe second. Figu e 9. View om he same posi ion in he i ual wo ld, wi h he pano ama swi ched o (A) and on (B). 5. Conclusions and Ou look This pape p esen ed he in e es and wo k low in c ea ing a VR isualiza ion wi h in eg a ed 360° pano amic pho og aphy o he Kaise dom in Königslu e . The combina ion o hese wo kinds o media— eal- ime 3D isualiza ion and HDRI pano amic pho og aphy—allows he in e ac i e and imme si e po en ial o he o me o complemen he high- ideli y and pho o ealism o he la e . While adi ionally hese wo “kinds” o VR ha e emained sepa a e, i is impo an o in es iga e ways o in eg a ing hem in o de o build expe iences ha a e able o in eg a e di e en kinds o da a. This is pa icula ly impo an o hose ields, such as he i age, whe e documen a ion can ake mul iple o ms, such as pho og aphs, objec s, 3D da a, o w i en documen s. The u u e de elopmen o he i ual museum, o example, depends on being able o in eg a e di e en kinds o da a in o a i ual space ha can be na iga ed in ui i ely in i ual eali y [28]. Fu he applica ions o he wo k low desc ibed abo e can also be en isioned. In ano he ecen p ojec , a ec ea ion o he own o S ade (Lowe Saxony) in he yea 1620 [12], pano amic pho og aphy is implemen ed so ha use s can jump be ween he eal- ime isualiza ion o he own in 1620 and 360° pho os om he mode n day. This implemen a ion allows use s o di ec ly jux apose he his o ic and con empo a y ci y, as an en y poin o compa ing he his o ical condi ions o he wo pe iods. In pa icula , his ea u e could ha e ex a meaning o use s who a e al eady amilia wi h he own, by e ealing he pe haps unknown his o y o ce ain well-known loca ions. While eal- ime 3D isualiza ions on hei own may p o ide a ce ain deg ee o imme sion, he in eg a ion o di e en kinds o da a in hese i ual wo lds, such as pano amic pho og aphy, can g ea ly en ich he expe ience by in i ing he use o compa e di e en kinds o isualiza ions. In addi ion, by aking eal- ime isualisa ions beyond being simply s a ic i ual wo lds h ough he in eg a ion o di e en kinds o in o ma ion, VR becomes much mo e powe ul as a ool o educa ion in museums. Cul u al he i age monumen s such as he Kaise dom o Königslu e a e pa icula ly sui ed o VR exhibi ion due a subs an ial exis ing audience ha may be looking o new ways o ex end hei isi o expe ience. By ex ending eal- ime isualisa ions h ough pano amic A B Figu e 9. View om he same posi ion in he i ual wo ld, wi h he pano ama swi ched o ( A ) and on (B). 5. Conclusions and Ou look This pape p esen ed he in e es and wo k low in c ea ing a VR isualiza ion wi h in eg a ed 360 ◦ pano amic pho og aphy o he Kaise dom in Königslu e . The combina ion o hese wo kinds o media— eal- ime 3D isualiza ion and HDRI pano amic pho og aphy—allows he in e ac i e and imme si e po en ial o he o me o complemen he high- ideli y and pho o ealism o he la e . While adi ionally hese wo “kinds” o VR ha e emained sepa a e, i is impo an o in es iga e ways o in eg a ing hem in o de o build expe iences ha a e able o in eg a e di e en kinds o da a. This is pa icula ly impo an o hose ields, such as he i age, whe e documen a ion can ake mul iple o ms, such as pho og aphs, objec s, 3D da a, o w i en documen s. The u u e de elopmen o he i ual museum, o example, depends on being able o in eg a e di e en kinds o da a in o a i ual space ha can be na iga ed in ui i ely in i ual eali y [28]. Fu he applica ions o he wo k low desc ibed abo e can also be en isioned. In ano he ecen p ojec , a ec ea ion o he own o S ade (Lowe Saxony) in he yea 1620 [ 12 ], pano amic pho og aphy