Music(re)ality: A Collaborative Improvisation between Virtual and Real World

Music(re)ality: A Collaborative Improvisation between Virtual and Real World YICHEN WANG, The Australian National University, Australia CHARLES MARTIN, The Australian National University, Australia Additional Key Words and Phrases: mixed reality musical interface and performance, head-mounted augmented reality ACM Reference Format: 1 PROGRAM NOTES Music(re)ality is a collaborative musical performance between the virtual and real world. Two musicians will present a musical improvisation, with one performing with an iPad instrument and the others using a freehand augmented reality musical instrument. While musicians are physically located in the space, the music jamming will happen across a virtual and real environment. How will the collaboration happen and what is a mixed reality musical performance? Is it through sonic-only feedback or performers’ musical gestures? 2 PROJECT DESCRIPTION Music(re)ality is a collaborative musical performance between the virtual and real-world space. New types of mixed reality musical instruments have been shown unique qualities of 3D musical interaction [ 1 ], it is still yet to see how these instruments are used with other musical instruments in a collaborative context. Collaborative music-making is well-studied [ 3 ] and often requests some levels of communicative cues or information sharing [ 2 ], particularly in the mixed reality context where the musician’s view is naturally blocked by hardware equipment [ 4 ]. With limited information flows between musicians and others when performing with the mixed reality musical instrument, what is it like and how can they reach a level of “musical agreement”? In this performance, two musicians will present a musical improvisation, one using an iPad instrument and the other using a freehand augmented reality musical instrument. While both musicians exist in reality and being able to see each other for collaboration, this AR instrument only exists in the “augmented” world. The AR instrument was designed with freehand interaction which the musician is able to perform flexible musical gestures for different sonic possibilities. The performer’s in-app view is shared on the stage screen for their collaborator and audiences. This musical performance is not only the author’s musical practice of their own instrument, but also attempts to understand opportunities and challenges of mixed reality musical instruments in music ensemble. Licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). Copyright remains with the author(s). Yichen Wang and Charles Martin. 2023. Music(re)ality: A Collaborative Improvisation between Virtual and Real World. 1, 1 (May 2023), 2 pages. Music Proceedings of the International Conference on New Interfaces for Musical Expression NIME’23,31 May–2 June, 2023, Mexico City, Mexico. 2 Wang and Martin 3 PERFORMANCE NOTES This a pre-recorded performance. One author will join the “open-jack” for live demonstration. The AR instrument itself is open-sourced on GitHub and can be deployed to most VR and AR devices (E.g., Oculus Pro, Quest; Microsoft HoloLens Gen 2). 4 MEDIA LINKS •Video: https://youtu.be/j-f-6y0zYuk REFERENCES [1] Florent Berthaut. 2020. 3D interaction techniques for musical expression. Journal of New Music Research 49, 1 (2020), 60–72. [2] Florent Berthaut and Luke Dahl. 2020. Adapting & Openness: Dynamics of Collaboration Interfaces for Heterogeneous Digital Orchestras. In New Interfaces for Musical Expression. [3] Tina Blaine and Sidney Fels. 2003. Contexts of collaborative musical experiences. In Proceedings of the 2003 conference on New interfaces for musical expression. 129–134. [4] Michael F Schober. 2006. Virtual environments for creative work in collaborative music-making. Virtual Reality 10, 2 (2006), 85–94.