A rapid prototyping tool to produce 360° video-based immersive experiences enhanced with virtual/multimedia elements

ScienceDi ec A ailable online a www.sciencedi ec .com P ocedia Compu e Science 138 (2018) 441–453 1877-0509 © 2018 The Au ho s. Published by Else ie L d. This is an open access a icle unde he CC BY-NC-ND license (h ps://c ea i ecommons.o g/licenses/by-nc-nd/4.0/) Selec ion and pee - e iew unde esponsibili y o he scien i ic commi ee o he CENTERIS - In e na ional Con e ence on ENTERp ise In o ma ion Sys ems / P ojMAN - In e na ional Con e ence on P ojec MANagemen / HCis - In e na ional Con e ence on Heal h and Social Ca e In o ma ion Sys ems and Technologies. 10.1016/j.p ocs.2018.10.062 10.1016/j.p ocs.2018.10.062 1877-0509 © 2018 The Au ho s. Published by Else ie L d. This is an open access a icle unde he CC BY-NC-ND license (h ps://c ea i ecommons.o g/licenses/by-nc-nd/4.0/) Selec ion and pee - e iew unde esponsibili y o he scien i ic commi ee o he CENTERIS - In e na ional Con e ence on ENTERp ise In o ma ion Sys ems / P ojMAN - In e na ional Con e ence on P ojec MANagemen / HCis - In e na ional Con e ence on Heal h and Social Ca e In o ma ion Sys ems and Technologies. A ailable online a www.sciencedi ec .com ScienceDi ec P ocedia Compu e Science 138 (2018) 441–453 www.else ie .com/loca e/p ocedia 1877-0509 © 2018 The Au ho s. Published by Else ie L d. This is an open access a icle unde he CC BY-NC-ND license (h ps://c ea i ecommons.o g/licenses/by-nc-nd/4.0/) Selec ion and pee - e iew unde esponsibili y o he scien i ic commi ee o he CENTERIS - In e na ional Con e ence on ENTERp ise In o ma ion Sys ems / P ojMAN - In e na ional Con e ence on P ojec MANagemen / HCis - In e na ional Con e ence on Heal h and Social Ca e In o ma ion Sys ems and Technologies. 10.1016/j.p ocs.2018.10.062 CENTERIS - In e na ional Con e ence on ENTERp ise In o ma ion Sys ems / P ojMAN - In e na ional Con e ence on P ojec MANagemen / HCis - In e na ional Con e ence on Heal h and Social Ca e In o ma ion Sys ems and Technologies, CENTERIS/P ojMAN/HCis 2018 A apid p o o yping ool o p oduce 360º ideo-based imme si e expe iences enhanced wi h i ual/mul imedia elemen s Telmo Adão*,a, Luís Páduab, Miguel Fonsecab, Luís Ag ellosc, Joaquim J. Sousaa, Luís Magalhãesd, Emanuel Pe esa a INESC TEC and Uni e si y o T ás-os-Mon es e Al o Dou o, 5000-801 Vila Real, Po ugal b Uni e si y o T ás-os-Mon es e Al o Dou o, School o Sciences and Technology, 5000-801 Vila Real, Po ugal c GEMA - Digi al and Technology Agency, Cais das Ped as N.º 8, 4050-465 Po o, Po ugal d ALGORITMI Cen e , Uni e si y o Minho, 4800-058 Guima ães, Po ugal Abs ac While he popula i y o i ual eali y (VR) g ows in a wide ange o applica ion con ex s – e.g. en e ainmen , aining, cul u al he i age and medicine –, i s economic impac is expec ed o each a ound 15bn USD, by he yea o 2020. Wi hin VR ield, 360º ideo has been spa king he in e es o de elopmen and esea ch communi ies. Howe e , edi ing ools suppo ing 360º pano amas a e usually expensi e and/o demand p og amming skills and/o ad anced use knowledge. Besides, applica ion app oaches o quickly and in ui i ely se up such 360º ideo-based VR en i onmen s complemen ed wi h di e se ypes o pa ame e izable i ual asse s and mul imedia elemen s a e s ill ha d o ind. The eby, his pape aims o p opose a sys em speci ica ion o simply and apidly con igu e imme si e VR en i onmen s composed o su ounding 360º ideo sphe es ha can be complemen ed wi h pa ame e izable mul imedia con en s – namely 3D models, ex and spa ial sound –, whose beha io can be ei he ime- ange o use -in e ac ion dependen . Mo eo e , a p elimina y p o o ype ha ollows a subs an ial pa o he p e iously men ioned speci ica ion and implemen s he enhancemen o 360º ideos wi h ime- ange dependen i ual asse s is p esen ed. P elimina y es s e alua ing usabili y and use sa is ac ion we e also ca ied ou wi h 30 pa icipan s, om which encou aging esul s we e achie ed. © 2018 The Au ho s. Published by Else ie L d. This is an open access a icle unde he CC BY-NC-ND license (h ps://c ea i ecommons.o g/licenses/by-nc-nd/4.0/) Selec ion and pee - e iew unde esponsibili y o he scien i ic commi ee o he CENTERIS - In e na ional Con e ence on ENTERp ise In o ma ion Sys ems / P ojMAN - In e na ional Con e ence on P ojec MANagemen / HCis - In e na ional Con e ence on Heal h and Social Ca e In o ma ion Sys ems and Technologies. Keywo ds: Vi ual Reali y; 360º Videos Edi ing; Imme si e Digi al Pano amas; Mul imedia; 3D model; 3D ex ; Spa ialized Sound; P o o yping. * Co esponding au ho . Tel.: +351 259 350 357. E-mail add ess: elmoadao@u ad.p A ailable online a www.sciencedi ec .com ScienceDi ec P ocedia Compu e Science 138 (2018) 441–453 www.else ie .com/loca e/p ocedia 1877-0509 © 2018 The Au ho s. Published by Else ie L d. This is an open access a icle unde he CC BY-NC-ND license (h ps://c ea i ecommons.o g/licenses/by-nc-nd/4.0/) Selec ion and pee - e iew unde esponsibili y o he scien i ic commi ee o he CENTERIS - In e na ional Con e ence on ENTERp ise In o ma ion Sys ems / P ojMAN - In e na ional Con e ence on P ojec MANagemen / HCis - In e na ional Con e ence on Heal h and Social Ca e In o ma ion Sys ems and Technologies. 10.1016/j.p ocs.2018.10.062 CENTERIS - In e na ional Con e ence on ENTERp ise In o ma ion Sys ems / P ojMAN - In e na ional Con e ence on P ojec MANagemen / HCis - In e na ional Con e ence on Heal h and Social Ca e In o ma ion Sys ems and Technologies, CENTERIS/P ojMAN/HCis 2018 A apid p o o yping ool o p oduce 360º ideo-based imme si e expe iences enhanced wi h i ual/mul imedia elemen s Telmo Adão*,a, Luís Páduab, Miguel Fonsecab, Luís Ag ellosc, Joaquim J. Sousaa, Luís Magalhãesd, Emanuel Pe esa a INESC TEC and Uni e si y o T ás-os-Mon es e Al o Dou o, 5000-801 Vila Real, Po ugal b Uni e si y o T ás-os-Mon es e Al o Dou o, School o Sciences and Technology, 5000-801 Vila Real, Po ugal c GEMA - Digi al and Technology Agency, Cais das Ped as N.º 8, 4050-465 Po o, Po ugal d ALGORITMI Cen e , Uni e si y o Minho, 4800-058 Guima ães, Po ugal Abs ac While he popula i y o i ual eali y (VR) g ows in a wide ange o applica ion con ex s – e.g. en e ainmen , aining, cul u al he i age and medicine –, i s economic impac is expec ed o each a ound 15bn USD, by he yea o 2020. Wi hin VR ield, 360º ideo has been spa king he in e es o de elopmen and esea ch communi ies. Howe e , edi ing ools suppo ing 360º pano amas a e usually expensi e and/o demand p og amming skills and/o ad anced use knowledge. Besides, applica ion app oaches o quickly and in ui i ely se up such 360º ideo-based VR en i onmen s complemen ed wi h di e se ypes o pa ame e izable i ual asse s and mul imedia elemen s a e s ill ha d o ind. The eby, his pape aims o p opose a sys em speci ica ion o simply and apidly con igu e imme si e VR en i onmen s composed o su ounding 360º ideo sphe es ha can be complemen ed wi h pa ame e izable mul imedia con en s – namely 3D models, ex and spa ial sound –, whose beha io can be ei he ime- ange o use -in e ac ion dependen . Mo eo e , a p elimina y p o o ype ha ollows a subs an ial pa o he p e iously men ioned speci ica ion and implemen s he enhancemen o 360º ideos wi h ime- ange dependen i ual asse s is p esen ed. P elimina y es s e alua ing usabili y and use sa is ac ion we e also ca ied ou wi h 30 pa icipan s, om which encou aging esul s we e achie ed. © 2018 The Au ho s. Published by Else ie L d. This is an open access a icle unde he CC BY-NC-ND license (h ps://c ea i ecommons.o g/licenses/by-nc-nd/4.0/) Selec ion and pee - e iew unde esponsibili y o he scien i ic commi ee o he CENTERIS - In e na ional Con e ence on ENTERp ise In o ma ion Sys ems / P ojMAN - In e na ional Con e ence on P ojec MANagemen / HCis - In e na ional Con e ence on Heal h and Social Ca e In o ma ion Sys ems and Technologies. Keywo ds: Vi ual Reali y; 360º Videos Edi ing; Imme si e Digi al Pano amas; Mul imedia; 3D model; 3D ex ; Spa ialized Sound; P o o yping. * Co esponding au ho . Tel.: +351 259 350 357. E-mail add ess: elmoadao@u ad.p 442 Telmo Adão e al. / P ocedia Compu e Science 138 (2018) 441–453 2 Telmo Adão e al. / P ocedia Compu e Science 138 (2018) 441–453 1. In oduc ion Expe ience simula ion using compu e -based syn he ic en i onmen s is he co e o i ual eali y (VR) esea ch and de elopmen ield [1], which has been success ully applied o se e al o he a eas (such as medicine [2], p oduc design and manu ac u ing [3], educa ion [4], ou ism [5], cul u al he i age [6] and human-compu e in e ac ion [7]), in a wide ange o con ex s (e.g. en e ainmen , aining and ehabili a ion) and goals (e.g. lea ning imp o emen , knowledge dissemina ion, public pa icipa ion p omo ion and medical pa ien eco e y suppo ). Cu en ly, VR is an a ea o g ea ele ance, whose economic impac is expec ed o be a ound 15bn USD, by he yea o 2020 [8]. Among he VR en i onmen classes, he e is one ha has been igge ing buzz nea by de elopmen and esea ch communi ies: 360º ideos ha allow a sphe ical iew o s eamed image y da a. These kind o ideos has been used o documen a ies, ou ism, houses and in e io design, hobby p og ams, deba e p og ams, making o , s and-up comedy, among o he s [9] and se e al a e he modali ies o p esen such con en as, o example, compu e au oma ic i ual en i onmen s (CAVE), con en ional sc eening de ices and head-moun ed displays (HMD) [9], [10]. O cou se, modali y selec ion should ollow he in ended equi emen s o imme si eness and p esence, since his aspec ends up ha ing impac in use expe ience ( o example, HMDs ha e highe capabili ies o imme si eness when compa ed wi h con en ional displays). Mo e speci ically, 360º ideo allows o se up imme si e en i onmen s, by p o iding a comple e ci cula (p e iously eco ded o li e) ideo s eaming a ound a ixed cen al acquisi ion sys em, hus o e coming educed co e ing angles ha is a limi a ion o adi ional ideog aphy [11][12]. Rega ding acquisi ion sys em, wo ypes o equipmen can be applied: a single came a wi h cu ed lenses o a ull ield o iew (FoV) co e ing o a mul i-came a se up, whe e each came a shoo s pa o he space o be la e s i ched in a single g oup [13]. The esul ing p oduc is a closed loop ideo ha allows an use o isualize i in po ions o as a whole – depending on a ailable FoV p o ided by iewe applica ion –, by simply and eely change iewpoin o ien a ion [13], [14], wi h po en ial bene i s o sense o p esence [12] (a opic add essed by o he s udies [15]–[17]). E o s o p o ide 360º ideo wi h mo e in e ac i i y and usage simplici y ha e been p esen ed ([18], [19], [11], [12], [20], [21], [22]), while o he s [23] ocused a en ions on quali y o expe ience (QoE), poin ing ou he s ong impac o as came a mo ion ideos in use s’ cybe sickness. Cu en ly, a wide a ie y o consume -g ade came as o eco d ideog aphic pano amas is a ailable, which has le e aging new ma ke s ha ely hei business models in his ype o mul imedia con en s. Social media ne wo ks and ideo pla o ms a e also ollowing his end, by p o iding suppo o 360º ideo s eaming [21]. Al hough, in wha ega ds o ideo edi ing, besides mos o he exis ing ools a e cos ly and demand p o essional le el knowledge o ope a e wi h, no all o hem suppo 360º pano amas and e en ewe p o ide eedom o in ui i ely se up VR imme si e en i onmen s complemen ed wi h pa ame e izable i ual elemen s, mo e speci ically 3D models, ex and spa ial sound, a leas , no wi hou p og amming and/o ad anced use skills. Sca ci y o such au ho ing ools has been iden i ied a ew yea s ago [24]. In [25]–[27], ex and pho os we e posi ioned in ce ain a eas o in e es o complemen he na a o ’s oice o o enhance he spa ial and con ex ual pe cep ion. Be ning e al. [24] used pano amic 360º ideos as a base o as augmen ed eali y (AR) scena ios p o o yping. The de eloped ool enables o inse 2D elemen s in o he ideo eed, as well as o ans o m hei posi ions, sizes and opaci ies. Ex ensible ma kup language (XML) iles a e used o sa e scena ios and o la e play hem in a iewe applica ion. Au ho s also epo ed ha hei ool could be easily handled – e en by no ice use s – and he c ea ion o bo h simple and ad anced scena ios could be ca ied in less han one hou . F ei as e al. [28] p oposed an in o ma ion sys em o he managemen and isualiza ion o mul isenso y con en s. Following he same line, Coelho e al. [29] p oposed a ool ha enables o edi and isualize 360º ideos enhanced wi h audio, hap ic and ol ac o y s imuli. The de eloped ool enabled o selec each s imulus’ ime ange and in ensi y (e.g. quan i y o smell sp ay o be eleased), o p e iew and o ine une he expe ience. A pilo e alua ion ega ding sys em’s usabili y was ca ied ou wi h 10 pa icipan s, being all o hem able o success ully pe o m he eques ed asks. Only one o hem asked o help. This pape aims o p opose a gene al sys em speci ica ion o easily and apidly con igu e imme si e VR en i onmen s cons i u ing isual expe iences composed o a single su ounding 360º ideo sphe e ha can be complemen ed wi h pa ame e izable mul imedia con en s namely 3D models, ex and spa ial sound. Mo eo e , hese con en s a e planned o be ime ange dependen o esponsi e o in e ac ion igge s (e.g. came a eye gaze, use ’s aycas in e sec ion, Telmo Adão e al. / P ocedia Compu e Science 138 (2018) 441–453 443 Telmo Adão e al. / P ocedia Compu e Science 138 (2018) 441–453 3 e c.). A p elimina y p o o ype ollowing mos o he p oposed speci ica ion – mo e speci ically, ime-based asse embedding – has been implemen ed in Uni y 3D (San F ancisco, Cali o nia, Uni ed S a es o Ame ica - USA) and is unde analysis as well. Mo eo e , esul s o es s made wi h 30 pa icipan s ha olun ee ed o expe imen and e alua e usabili y and o e all sa is ac ion a e discussed. The emaining o his pape is o ganized as ollows: Sec ion 2 desc ibes he sys em’s speci ica ion; in Sec ion 3, he i s p o o ype e sion is p esen ed; p o o ype’s e alua ion esul s a e p esen ed and discussed in Sec ion 4. Finally, Sec ion 5 ends his wo k by p esen ing some conclusions and ema ks owa ds u u e de elopmen s. 2. Gene al sys em speci ica ion o edi and enhance 360º ideo sphe es wi h i ual/mul imedia con en s The p oposed sys em o edi and enhance 360º ideo-based en i onmen s wi h i ual/mul imedia con en s is composed o wo undamen al unc ionali y g oups: one o se ing up expe iences (manage -side) and o he o isualize he compiled he esul (spec a o -side). Rega ding he o me , i includes unc ionali ies o se up an imme si e s age wi h he espec i e 360º ideo, o impo pa ame e izable asse s, o speci y e en s o each impo ed asse – ime dependen and/o based in in e ac ion – and, inally, o p e iew he expe ience be o e deli e ing i o he spec a o , allowing imely adjus men s. Asse s and e en s ollow a con igu a ion sequence. Fi s ly, 3D models, sounds o ex ual desc ip ions mus be impo ed and hen e en s can be associa ed o each one. In wha ega ds e en s, wo ypes a e planned o be in eg a ed: a ime-based e en speci ies he objec s’ ime- o-li e while in e ac ion-based ones de ine a esponse o an in e ac ion (e.g. eye-gaze o con ac ). The la e g oup in e aces wi h he spec a o by p o iding a 360º expe ience ha , besides isualiza ion, also suppo s in e ac ion. The eby, du ing expe iences, a spec a o may: (1) see isual/audible objec s being imely ac i a ed/deac i a ed wi h he ideo’s cou se; and (2) be p o ided wi h in e ac ion-based objec s’ eedback. Fig. 1 p esen s he g oups o unc ionali ies and espec i e use ype assignmen . Fig. 1. G oups o gene al unc ionali ies included in he p oposed 360º ideo edi ing sys em. While he manage is esponsible by se ing up expe iences ha can include ime- o in e ac ion-based asse s (objec s), spec a o is he sys em’s ac o o whom expe iences a e deli e ed. Rega ding expe ience manage asks, he e y i s s ep o con igu ing a 360º ideo-based expe ience is o c ea e a new p ojec o load an exis ing one. Then, ideo 360º needs o be selec ed by use o loaded acco dingly wi h he loaded p ojec se ings. As long as he manage emains in he applica ion (un il qui ing), asse s can be loaded in o he p ojec and, hen, a pa ame e iza ion s ep may ake place, which includes con ex -based se ing de ini ion and ime-/in e ac ion-based e en a achmen . Fig. 2 p esen s he simple o m o a wo k low depic ing he a o emen ioned unc ionali ies. 444 Telmo Adão e al. / P ocedia Compu e Science 138 (2018) 441–453 4 Telmo Adão e al. / P ocedia Compu e Science 138 (2018) 441–453 Fig. 2. Expe ience se up om he manage pe spec i e. P ojec s can be ini ialized o loaded om his o y. To se up an imme si e en i onmen , ideo 360º can be selec ed ( o new p ojec s) o loaded ( om exis ing p ojec se ings) and asse s can be added, edi ed o dele ed. Mo eo e , he possibili y o adjus ing each asse se ings is a ailable, which includes di ec ly ela ed pa ame e s o associa ed igge s ( ime-/in e ac ion-based). 2.1. Asse s and pa ame e s Each asse ype has i s own g oup o di ec ly ela ed pa ame e s. The eby se ings a e done in a con ex -awa e basis. Suppo ed ypes a e he ollowing:  3D model: co esponds o a mesh g oup ha can be impo ed om ile (e.g. obj, bx). Besides posi ion inside he en i onmen , manage can also de ine objec ’s o a ion and scale;  Tex : is a mesh suppo ing cha ac e chains. As 3D models, i can be placed inside he en i onmen , acco ding o manage ’s will. Also, ex ual desc ip ion inside he mesh can be edi ed, as well as i s scale;  Sound: ep esen s an audi o y asse . I can be assigned o any posi ion inside he en i onmen and i s sound olume can be pa ame e ized. A no ewo hy ea u e o his asse ype is he spa iali y as i p o ides o he use an in ui i e insigh abou he sound playing sou ce posi ion, du ing expe iences. 2.2. Asse s’ con olling igge s Along wi h asse s pa ame e iza ion, a manage can a ach igge s. As i was p e iously men ioned, wo ypes can be speci ied: ime- and (in e ac ion) e en -based. Rega ding he o me , i consis s in de ining a ime- o-li e o a gi en asse , i.e., a ime ange o show up he objec in he 360º en i onmen . Two alues wi hin ideo ime bounda ies mus be p o ided, co esponding o a s a and o an end ime. The la e ype ega ds o in e ac ion e en s, in which an in e ac ion lis ene and a pos -in e ac ion igge e en – o simply pos - igge e en – mus be speci ied. Essen ially, in e ac ion lis ene es ablishes which use ac ion ac i a es pos - igge e en . In spi e o s ill being opened o new hypo hesis, he ollowing lis p o ides he planned in e ac ion lis ene s o be conside ed o in eg a ion in he p oposed sys em:  Eye-gaze: de ec ed when he use s a es o a gi en objec o a gi en pe iod o ime ( ypically, a ew seconds);  Con ac : occu s i some in e ac ion me apho is applied o ac ually each he i ual objec o in e es . In a 3D en i onmen , such me apho can be implemen ed by, o example, emo e aycas – i.e., i ual adjus able ay capable o eaching i ual asse s – o p oximi y picking con olle – i.e., physical de ice wi h posi ion acking capabili ies, such as HTC Vi e (HTC co po a ion, Xindian, New Taipei Ci y, Taiwan), o con e use ’s eal loca ion in o a i ual en i onmen posi ion. Telmo Adão e al. / P ocedia Compu e Science 138 (2018) 441–453 445 Telmo Adão e al. / P ocedia Compu e Science 138 (2018) 441–453 5 Pos - igge e en s a e elemen s ha de ine which ac ions will be pe o med upon a gi en objec , a e an in e ac ion e en de ec ion. The ollowing p o isional lis p esen s he planned pos - igge e en s o be in eg a ed in he p oposed sys em:  Cus omized beha io engagemen : displaying o p e-con igu ed isual/audi o y s imuli ( o example, ansla ions, pa hing de ini ions, o a ions, scaling ope a ions and/o ma e ial/colo changings a ec ing 3D models/ ex meshes and, o audio elemen s, mixable sound e ec s such as acous ic ans o ma ions and/o playing speed al e a ions);  Sel -exclusion: in p ac ice, his co esponds o an objec ’s sel -deac i a ion o des uc ion;  New asse spawning: asse s may be con igu ed o be spawned a e in e ac ion wi h a gi en objec o in e es , ei he o be added o he i ual en i onmen o o eplace he e e ed objec . A simpli ied lowcha depic ing a possible wo k low o se ing up he a o emen ioned ime/in e ac ion igge s is p esen ed in Fig. 3. Fig. 3. Asse s’ igge con igu a ion wo k low. In case o ime-based igge , a s a and an end ime mus be p o ided (asse ’s ime- o-li e). O he wise, an in e ac ion e en lis ene is equi ed o be de ined (eye-gaze o con ac ), along wi h a pos -in e ac ion igge (cus omized beha iou engagemen , sel -exclusion, new asse spawning). The ollowing sec ion p esen s a p o o ype applica ion implemen ing he add essed sys em speci ica ion, bu only conside ing ime-based igge s. I aims o show p oposed sys em’s cu en de elopmen s a e. 3. P o o ype implemen a ion add essing ime-based asse ’s in eg a ion in 360º ideo-based VR en i onmen s To ake ad an age o a powe ul mains eam game engine o mul i-pla o m de elopmen , which p o ides lexible suppo o easily ope a e in 3D en i onmen s, p oposed sys em’s p o o ype was implemen ed in Uni y 3D ( e sion 2018.1). Along wi h a powe ul g aphical in e ace enabling o di ec ly in e ac wi h a 3D s age o which componen s can be added by simple d ag-and-d op mouse ope a ion, Uni y 3D also suppo s collision mapping, ex u e manipula ion, sc een space ambien occlusion and beha io -o ien ed p og amming, whose code sc ip s migh be di ec ly a ached o game objec s placed in he 3D s age. A cu en s age, his p o o ype is implemen ing manage ’s side au ho ing ool o se up 360º ideo-based expe iences ha can be p e- isualized. I consis s in a 3D ool ha isola es he manage inside o a sphe e and allows him o selec he ideo and o eely place objec s and pa ame e ize hem acco ding o hei ype. Mo eo e , each objec can be se wi h ime s o show up and ade ou om he ideo. P ojec s can be sa ed in o Ja aSc ip Objec No a ion (JSON) o ma iles, o be la e accessed and edi ed o , simply, o be used o display imme si e expe iences o spec a o s. 3.1. Imme si e 360º ideos P o o ype implemen s a sphe e o 25 Uni y uni s (equi alen o 25m, by de aul ) ha cons i u es he cocoon o 360º ideo display. To be able o ep oduce ideos o i s cen al poin di ec ion – i.e., o ha e in e io p ojec ion – sphe e no mals we e in e ed. A se ies o 10 expe imen s we e ca ied ou o isually check he adap abili y o he sphe e in displaying di e en MP4 ideos, in which a p ope ende ing and a consis en ex u e applica ion beha io 446 Telmo Adão e al. / P ocedia Compu e Science 138 (2018) 441–453 6 Telmo Adão e al. / P ocedia Compu e Science 138 (2018) 441–453 was obse ed. Howe e , some sligh ly no iceable dis o ions we e iden i ied, which is a qui e common occu ence ela ed wi h sphe ical ideos s i ching. This issue was conside ed negligible, since i alls ou o his pape ’s scope. 3.2. Suppo ed Objec s Th ee ypes o i ual objec s a e suppo ed by he de eloped p o o ype: 3D models, spa ialized sound and ex meshes (Fig. 4). All o hem can be eely placed wi hin he sphe ical space. In case o i ual 3D models, manage can selec among cubic p imi i es o 3D polygonal meshes impo ed om ha d d i e. Fo he la e , a ee asse om Uni y Asse S o e [30] was in eg a ed in he p o o ype p ojec o suppo wa e on iles (.OBJ) impo a un ime. Rega ding ope a ions ha can be made o ans o m i ual 3D models, o a ions, scaling and anima ions a e allowed. While o a ions can occu a ound X, Y and Z axis, indi idually, scaling is applied o he 3-axis locked oge he . Rega ding anima ions, a he cu en de elopmen s age, only a non-s opping o a ion a ound Y axis is suppo ed. Spa ialized sound is always posi ioned o a ed owa ds cen al sphe e posi ion, o shoo he audible s imulus di ec ly o he use , du ing expe iences. This ype o asse equi es a wa e ile picked up om ha d d i e by he manage , which ep esen s he sound o play. I s g aphical ep esen a ion in edi ing mode is a 3D sound speake ha can be selec ed and eposi ioned. Pa ame e iza ion includes sound olume ( anging om 0 – so e – o 1 – loude ) and a boolean lag de ining i he asse should play in a loop o no . Sound spa ializa ion is managed by Uni y’s Audio componen , as long as spa ial blend p op ie y is se o 1 (which is no , by de aul ). Addi ionally, 3D sound se ings such as dopple le el and olume ollo can be modi ied. In he p oposed p o o ype, each sound asse spawned a un ime has i s spa ial blend p op ie y se o 1 and uses he de aul 3D audio se ings, since hey a e good enough o he cu en sound spa ializa ion needs. Fo in o ma i e pu poses, he p oposed p o o ype allows o in eg a e pa ame e izable i ual ex meshes wi hin he imme si e sphe e space. Jus as sound asse s, ex ual meshes keep hei o a ion owa ds sphe e cen e , ega dless posi ion modi ica ions. Locked XYZ scaling and ex edi ing cons i u e he se o pa ame e s ha can be changed by he manage o ans o m ex ual asse s. a) b) c) d) Fig. 4. Types o i ual asse s suppo ed o in eg a ion by he p oposed p o o ype o se ing up 360º ideo-based en i onmen s: a) 3D model p imi i e (only cubes a e suppo ed, cu en ly); b) 3D model impo ed om ha d d i e; c) Spa ial sound ep esen a ion ( isible in edi ing mode); and c) ex ual mesh. 3.3. In e ac ion P o o ype’s in e ac ion is implemen ed conside ing a minimalis se o inpu s, o he sake o simplici y: mouse d agging, le mouse bu on, keyboa d’s le Al and, also, keyboa d’s numbe s 1 and 2 om he ypew i e sec ion (TS). Table 1 disc imina es suppo ed p o o ype’s ac ions and co esponding con ols’ combina ions. Telmo Adão e al. / P ocedia Compu e Science 138 (2018) 441–453 447 Telmo Adão e al. / P ocedia Compu e Science 138 (2018) 441–453 7 Table 1. P o o ype’s a ailable ac ions and espec i e keyboa d/mouse con ols o pe o m hem. Ac ion Con ols’ combina ion To na iga e wi h he mouse in ee mode, enabling menus access (main menu, dialog con ols, e c.). Mouse cu so d agging. To isualize 360º ideo all a ound, by mo ing aycas . Le Al + mouse cu so d agging. To mo e away o o b ing close aycas posi ione . Le Al + TS 1 o le Al + TS 2, espec i ely. To place an asse in he en i onmen . Le mouse click in one o he i s ou main menu bu ons ( om le o igh ). To d ag elemen s in he en i onmen ( eposi ioning). (1) Le Al + le mouse click + mouse d agging. To mo e away o o b ing close asse s, using aycas . (1) Le mouse click + TS 1 o le mouse click + TS 2, espec i ely. To pa ame e ize asse s h ough con ex -based menu ( o mo e in o ma ion, check p e ious subsec ions). (1) Mouse d agging o na iga e o he igh sided con ex - based menu; mouse clicking o selec op ion. To change asse ’s ime- o-li e, h ough con ex -based dialog. (2) Mouse clicking/d agging in he minimum/maximum handle s belonging o he con ex -based dialog’s ime- line. To change asse ’s addi ional pa ame e s, h ough con ex - based dialog (2) Mouse clicking o na iga e check/uncheck: anima ion o 3D models and loop playing o sound asse s. (1) Requi es o ha e an asse selec ed, by o ien ing he aycas owa ds elemen o in e es . (2) Requi es (1) o be sa is ied, ollowing con ex -based dialog in oca ion h ough edi ing op ions selec ion in he con ex -based menu. A ea u e wo hy o highligh ega ding o a ions in e ac ion is he ci cula colo ed gizmo ha is shown o p o ide a isual in ui ion o he axis unde which a gi en asse will be o a ed. This isual aid is ac i a ed whene e he mouse cu so is o e a o a ion bu on (by e en igge ) and will assume he o ien a ion o he desi ed o a ion axis (X, Y o Z, depending on he bu on he mouse is o e laying). In wha conce ns o edi ing menus, hese a e (exclusi ely) p esen ed conside ing he con ex o he ask ha is being pe o med a a gi en momen . When he manage is eely na iga ing, besides a aycas o posi ioning/selec ing i ual asse s in he 360º ideo-based en i onmen , main menu is p esen ed in he sc een bo om o allow i ual asse s’ a achmen ( ha assume he aycas poin e posi ion), sa e, p e iew and qui he expe ience p ojec . I asse a achmen is eques ed, a dialog shows up wi h a pa ame e izable empo al line allowing o de ine s a and ending ime o espec i e ende ing ( ull ideo ime ange is assumed by de aul ). Ano he op ion depending on he asse ype ha is being added is shown along wi h ha ime-line: o i ual 3D models, a checkbox o selec /unselec anima ion ( o a ion a ound Y axis); o spa ial sound, he e is also a checkbox o oggle loop playing; ex meshes do no ha e his second pa ame e a ailable. The same ype o dialogs is shown i he manage is edi ing a p e iously a ached asse , bu wi h a dele e bu on a ailable o wipe ou he elemen ha is being handled (in selec ion), in case o need. When he manage is selec ing an asse lying in he scene, an asse ype-based menu is p ope ly p esen ed a he igh ma gin o he sc een. Edi op ion is ans e sal o all ypes. Fo unanima ed i ual 3D models, scaling, o a ion and eo ien a ion ( o u n he asse ’s ace owa ds he sphe e cen e ) op ions a e made a ailable. O he wise (i he 3D model is se o be anima ed), o a ion op ions a e hidden. Rega ding audio asse s, a olume slide anging be ween 0 (so e ) and 1 (loude ) is shown in he selec ion-ac i e menu. Fo ex meshes, such menu p esen s – besides an edi ing bu on – scaling op ions o make he desc ip ions bigge o smalle . Finally, when he op ion o sa e he expe ience p ojec is eques ed, a window pops up p omp ing a s o ing olde and a ile name. A sc eensho o he wo king p o o ype is p esen ed in Fig. 5, in which se e al asse s can be seen, namely a selec ed cube (wi h he aycas poin ing o i ), a i ual model o a chai , a spa ial audio sou ce and se e al ex meshes agging each one o hem. 448 Telmo Adão e al. / P ocedia Compu e Science 138 (2018) 441–453 8 Telmo Adão e al. / P ocedia Compu e Science 138 (2018) 441–453 Fig. 5. P o o ype’s sc eensho , depic ing se e al ypes o i ual asse s embedded in a 360º ideo-based en i onmen . The ollowing elemen s can be seen: a i ual chai (nex o eal chai s), a spa ial audio sou ce, a i ual cube ha is selec ed by aycas , a con ex -based menu ( igh pa o he sc een) ega ding cube selec ion and an a a a pa ially occluded by ha menu. 3.4. JSON o ma o sa ing/loading/p e iewing expe ience p ojec s To suppo VR expe ience p ojec s’ sa ing/loading/p e iewing, JSON iles a e used o s o e he s a e o he asse s lying in he en i onmen – and espec i e p op ie ies – a a gi en momen . The main di e ence be ween a JSON ile o sa e and a JSON ile o p e iew is ha he o me s o es da a de ini i ely – o u u e access (e.g. expe ience loading) –, while he la e is empo a ily gene a ed only o suppo he p e iew mode and is dele ed when use qui s p e iewing. Rega dless, bo h iles keep he e y same s uc u e: a pa h o he 360º ideo o play akes he op o each ile and he emaining o i is ese ed o a ays, g ouping he i ual en i onmen 's asse s pe ype, i.e., a ays o p imi i es, 3D models, sounds and ex meshes. Nex sec ion ega ds o p o o ype e alua ion, whe ein es s wi h pa icipan s – o e alua e usabili y and use sa is ac ion – and espec i e esul s a e analyzed and discussed. 4. P o o ype e alua ion Expe imen al design used o e alua e he p oposed p o o ype and espec i e esul s p ocessed o e da a ga he ed om ques ionnai es a e bo h p esen ed and discussed in his sec ion. To assess he p oposed sys em’s p o o ype wi h espec o bo h usabili y and sa is ac ion, es s wi h olun ee s we e ca ied ou , in an expe ience assembling con ex . They we e p o ided wi h a p o ocol speci ying a se o asks o accomplish. In he end, hey we e in i ed o answe o a couple o ques ionnai es. 4.1. Ma e ial and me hods An academic disse a ion de ense simula ion was he expe imen selec ed o e alua e he p oposed p o o ype. The eby, he public academic e en s oom o he School o Sciences and Technology (Escola de Ciências e Tecnologia – ECT, Pole I), in eg a ed in he Uni e si y o T ás-os-Mon es e Al o Dou o (UTAD) campus, was eco ded wi h a 360º came a o be used as expe imen 's ideo-sphe e backg ound. The expe imen consis ed in asking o 30 pa icipan s o p ope ly popula e he ideo-sphe e wi h i ual asse s – mo e speci ically, 3D models, sound and ex – con igu ed wi h a ime ange, i.e., ime- o-li e in he scene. An adap a ion s age was ca ied ou o le he pa icipan s ge amilia ized wi h he p o o ype’s in e ac ion. Fi s ly, hey we e ins uc ed how o use he so wa e and hen, hey we e asked o eely explo e i . A e wa ds, hey we e Telmo Adão e al. / P ocedia Compu e Science 138 (2018) 441–453 449 Telmo Adão e al. / P ocedia Compu e Science 138 (2018) 441–453 9 challenged o accomplish a p o ocol wi h 6 conc e e asks. In he end, eedback ega ding unc ionali y-cen e ed usabili y and o e all sa is ac ion was ga he ed h ough a couple o ques ionnai es, which will be analyzed in a u he subsec ion. 4.1.1. Expe imen al 360º ideo-based en i onmen The 360º ideo con en used o his expe ience was eco ded using a Ricoh The a V (Ricoh Company L d., Tokyo, Japan) cap u ing de ice ha is cons i u ed by a couple o 180º FOV came as and suppo s 4K esolu ion a 30 ames pe second ( ps). As p e iously men ioned, ideo eco ding ook place in he academic e en s oom o ECT, UTAD, which can be di ided in h ee main sec ions ha mus be explained o pa icipan s be o e hey pe o m he expe imen al ial: audience, ju y’s able (bo h placed in opposi e oom’s sides) and candida e’s spo (placed in he oom’s cen e ). Mos o he i ual models o enhance he 360º ideo-based en i onmen we e downloaded om public access eposi o ies1, wi h ee usage licensing: a bo le o wa e , a lap op, an a a a and an audio speake ep esen ing sound asse we e he models used om hose eposi o ies. Rega ding he cube p imi i e ha is made a ailable o in eg a ion in his de elopmen s age, i is na i ely p o ided by Uni y amewo k. These elemen s we e placed in a speci ic olde o be easily accessed by pa icipan s du ing expe imen al es s. 4.1.2. Sample cha ac e iza ion Mos o he pa icipan s we e s uden s o Communica ion and Mul imedia (CM) cou se (≈66,67%) while he emaining sample was composed o s uden s o In o ma ics and o he simila cou ses ( om now on e e ed as IOSC). Elemen s in bo h g oups we e mos ly good o ad anced compu e use s and had con ac ed wi h VR en i onmen s and 360º ideos in p e ious occasions. A balanced ega ding ideo pos -/p oduc ion skills was e i ied: app oxima ely hal o he pa icipan s poin ed ou o be inexpe ienced o o ha e a basic/a e age knowledge in such opic, while he emaining sample conside ed o ha e good/ad anced capabili ies. O he obse a ions include academic quali ica ion and gende : among he pa icipan s, 15 we e bachelo s uden s, 12 we e a ending o mas e ’s deg ee, he emaining 3 we e in PhD p og ams; 4 indi iduals we e emale. 4.1.3. P oceedings and ask p o ocol Volun ee s p o ided indi idual con ibu ion o p o o ype’s es ing. They s a ed o being in o med abou his es s’ con ex , as well as p o o ype’s goal. Then, hey we e p esen ed wi h a consen decla a ion o sign, hus ag eeing o be olun a ily in eg a ed in he es ing sample and, also, o p o ide hei anonymized da a o scien i ic p ocessing pu poses. P o o ype and main unc ionali ies we e shown and explained, as well as he o ganiza ion o he public academic e en s oom eco ded o he 360º ideo ha composes he imme si e en i onmen . A e wa ds, pa icipan s we e in i ed o eely explo e he p oposed so wa e wi h he goal o le ing hem ge amilia wi h i (adap a ion s age). When hey el com o able o s a he o icial es , a p o ocol o p oduce a 360º ideo-based imme si e en i onmen simula ing a public disse a ion de ense was p o ided o hem, in which he ollowing 6 conc e e asks’ se s we e eques ed:  To place a p imi i e (cube) nea by audience’s a ea, along wi h a ex ual desc ip ion “Audience”; equi ed ime ange o bo h elemen s is 2-10 seconds;  To place wo bo les o wa e upon ju ies’ able and make hem look like hey a e seamlessly suppo ed; and, also, a ex ual desc ip ion “a gue ” a ound he same able; hese elemen s a e shown be ween 20 h and 30 h second;  To edi he las ex ual elemen and o change i s con en o “Ju y”, as well as o adjus i s showing in e al o 20- 35 seconds; o dele e one o he bo les p e iously impo ed;  To place a lap op upon he candida e’s able and change i s o a ion, scale and posi ion a will; his asse is shown he en i e ideo; 1 Tu boSquid (h ps://www. u bosquid.com/) and CGT ade (h ps://www.cg ade .com/).