TangibleMRCreate: Intuitive Authoring of Mixed Reality Content

International Conference on Artificial Reality and Telexistence Eurographics Symposium on Virtual Environments (2023) A. Campbell, C. Krogmeier, and G. Young (Editors) Poster TangibleMRCreate: Intuitive Authoring of Mixed Reality Content X. Li1and J. Dingliana1 1School of Computer Science and Statistics, Trinity College Dublin, Ireland Abstract In this poster, we present work-in-progress exploring the intuitive authoring of Mixed Reality (MR) content using physical object manipulation, tangible 3D user interfaces, and the creation of MR within MR itself. To assess this approach, we implemented an MR-based system enabling novice users to tangibly create MR scenes within MR. The system, prototyped on the Microsoft HoloLens 2, allows users to create digital scenes by manipulating a physical box and a tangible 3D user interface. A pilot study was conducted to provide a preliminary evaluation and to inform future system development and deeper study design. CCS Concepts •Human-centered computing →Mixed / augmented reality; 1. Introduction Despite the burgeoning popularity of Mixed Reality (MR), the authoring of MR applications is still limited to a relatively small community of creators. One reason for this is that MR authoring typically involves the use of desktop development tools (e.g., Unity3D) which not only separates the creation process from the actual physical environment but also requires technical skills, often including competence at 3D transformations, which are known to be difficult for novice users to master [SWLR21]. To address this, we investigate the intuitive authoring of MR content by leveraging the paradigms of tangible 3D user interfaces, manipulation and interaction of physical objects, and the creation of MR within MR itself. Tangible 3D user interfaces provide tactile feedback from physical objects [Ish08], which is particularly beneficial for novice users who require more precise interactions. The manipulation and interaction with physical objects replicate a physical experience. Users can easily and intuitively move and rotate these objects to place digital models [BIAI17]. This method simplifies object alignment and naturally accounts for collisions with the real-environment. For example, objects ’snap’ to physical surfaces such as a table without the need for a physics engine, collision detection, nor environment sensing. The creation of MR within MR is important for scenarios where consideration of the real-environment is necessary, for instance, when planning illumination, it would be useful to directly account for light sources in the real environment. While modern development tools can, in theory, capture such properties from the real world and transfer them to virtual environments, this is an expensive process to achieve robustly in real-time. In similarly motivated previous work, Lee et al. [LSR11] proposed tangible user interfaces for authoring AR content intuitively. Their system requires specific hardware RFID Tags, which may not be available or affordable for all users and requires careful environment setup in advance to support spatial positioning. Muender et al. [MRD→19] compared the effects of tangibles with different production processes (uniform-shaped objects, Lego-built objects, and 3D-printed tangibles) for 3D scene creation in virtual reality (VR), while Matviienko et al. [MLM→21] used multiple tangible physical objects and a tablet to support touch input for assisting children to create VR content without programming knowledge. In contrast to these, our work aims to be simpler and more accessible, allowing users to place models using a single physical object, a simple box which is re-attachable to different digital models, without the need for additional devices, props, nor advanced setup. Figure 1: Components of our system: (a) Physical object manipulation. (b) Tangible 3D user interface. (c) Creation of MR within MR itself. To explore the efficacy of our proposed approach, we designed a system, TangibleMRCreate, which enables users to tangibly create MR scenes within MR by manipulating a physical box and interacting with a tangible 3D user interface, as shown in Fig. 1.We implemented the system on the Microsoft Hololens 2, but future deployment on Video See-through displays, such as the Apple Vision Pro, may provide better seamless rendering quality. We then use this in a pilot study to determine potential advantages over traditional desktop development tools and pure gestural interaction. © 2023 The Authors. Proceedings published by Eurographics - The European Association for Computer Graphics. This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited. The following is an author-preprint of the abstract published by Eurographics on the EG Digital Library The final published version is available at: https://doi.org/10.2312/egve.20231339 X. Li & J. Dingliana / TangibleMRCreate: Intuitive Authoring of Mixed Reality Content 2. Design and Implementation The design of our system can be divided into three parts: 1. Start and Model Assets Import, 2. Tangible Creation, and 3. Save and Upload, as shown in Fig. 2. The main component of the system, Tangible Creation, was developed using Unity3D 2021.3.4f1, with Microsoft HoloLens 2 as the MR headset. Additionally, the Mixed Reality Toolkit (MRTK) 2.8.3 was used for hand tracking and system configuration, and the Vuforia Engine 10.16.3 was employed for target tracking. A physical box (8 x 6 x 3 cm, Vuforia-provided texture) is used as a tracking target manipulated by the user to place digital 3D models in the scene, replicating a physical hands-on creative experience, while allowing users to easily and intuitively handle relatively complex tasks such as interactive object rotations in 3D. The physical box also provides natural collisions with the realenvironment, providing additional feedback and enabling users to easily ’snap’ digital models to surfaces in the real scene. Open TangibleMRCreate on Hololens 2 Select the attached model Browse the tangible menu Open scene Live collaboration Transform the physical box Confirm placement Upload scene Search 3D models Join Invite Import Target tracking 1. Start & Model Assets Import 2. Tangible Creation 3. Save & Upload Save scene Figure 2: Overview of TangibleMRCreate system. We also employ tangible interaction in a menu for the selection of 3D model assets. MR apps typically employ virtual menu windows floating in thin air, which, due to their ethereal nature, can be unintuitive for novice users, who sometimes struggle to robustly complete simple tasks such as clicking buttons or pressing keys. To address this, we project the menu interface onto physical elements in the environment, for instance, a wall, effectively providing tactile feedback and stability for more precise and robust interactions. 3. Results and Conclusions We conducted a preliminary pilot study involving six participants (25 to 35 years). Three were experienced users of Unity and HoloLens (P1-P3), two were novices with HoloLens but familiar with Unity (P4-P5), and one was a novice in both HoloLens and Unity (P6). We encouraged them to compare our method with pure gestural and traditional desktop paradigms. The participants were tasked with replicating, as closely as possible, the 3D layout of three preset scenes of varying difficulty, in two ways (using our system for tangible creation, and pure gestural interaction via pinchand-drag gestures implemented in MRTK). During the task, participants were encouraged to speak out loud and share their thoughts on the interactions. All participants responded warmly to the interactive manipulation of the physical object, the tangible 3D user interface, and scene creation within MR. Compared to desktop tools, participants found the tangible creation more intuitive but with some limitations. P4: "I love this; I can touch it, move it, and position it as I want, that is enough for me and it’s faster than using Unity. In Unity, you have to work with 3D transformations." P2 noted the advantage of being able to view the real environment during the creation process, eliminating the need to spend time scanning and building in Unity. However, P2 also mentioned, "It is easy to use and feels very intuitive, but I prefer using Unity where I can precisely position objects through code and the 3D transformation interface." Compared to pure gestural interaction, participants found tangibility more comfortable and intuitive. P6 commented, "The physical box is easy to use for a first-timer. Gestural interaction is a little confusing." P1 noted that "Gestural interaction feels like it has too much freedom, especially the rotation; it’s hard to control. If it’s just for placing objects like in this test, the physical box would be a better choice." P1 and P3 suggested that in other situations, like sculpting, gestures might be more suitable. P5 noted that "Even though gestural interaction is a bit difficult to control, the box loses tracking if my fingers block its surface." Feedback from the pilot study provided overall encouraging results and valuable guidance for further development of a more comprehensive system to support the efficient creation of MR content. Potential additions include: (a) improving the size, weight and quality of the physical box for a better hand-held experience; (b) supporting intuitive scaling transforms; (c) natural interactions with other virtual objects, such as allowing them to snap together; (d) enabling shared MR experiences, where users can collaboratively interact and co-author digital media assets for MR; and (e) improving tracking robustness in scenarios where fingers occlude markers. This work was conducted with the financial support of a Trinity College Dublin Doctoral Research Award, and the Horizon Europe Framework Program under grant agreement 101070109. References [BIAI17] BESANÇON L., ISSARTEL P., AMMI M., ISENBERG T.: Mouse, tactile, and tangible input for 3D manipulation. In 2017 CHI Conf. on Hum. Factors in Comp. Sys. (2017), ACM, p. 4727–4740. 1 [Ish08] ISHII H.: Tangible bits: Beyond pixels. In Proceedings of the 2nd International Conference on Tangible and Embedded Interaction (2008), TEI ’08, Association for Computing Machinery, p. xv–xxv. 1 [LSR11] LEE J. Y., SEO D. W., RHEE G. W.: Tangible authoring of 3D virtual scenes in dynamic augmented reality environment. Computers in Industry 62, 1 (2011), 107–119. 1 [MLM→21] MATVIIENKO A., LANGER M., MÜLLER F., SCHMITZ M., MÜHLHÄUSER M.: Vrtangibles: Assisting children in creating virtual scenes using tangible objects and touch input. In Extended Abstracts of 2021 CHI Conf. on Hum. Factors in Comp. Sys. (2021), ACM. 1 [MRD→19] MUENDER T., REINSCHLUESSEL A. V., DREWES S., WENIG D., DÖRING T., MALAKA R.: Does it feel real? using tangibles with different fidelities to build and explore scenes in virtual reality. In 2019 CHI Conf. on Hum. Factors in Comp. Sys. (2019), ACM. 1 [SWLR21] SUSELO T., WÜNSCHE B. C., LUXTON-REILLY A.: Using Mobile Augmented Reality for Teaching 3D Transformations. In Proc. ACM Technical Symposium on Computer Science Education (Mar. 2021), ACM, pp. 872–878. 1 © 2023 The Authors. Proceedings published by Eurographics - The European Association for Computer Graphics.