Unspoken

Unspoken YICHEN WANG∗and SANDY MA∗,The Australian National University, Australia Additional Key Words and Phrases: mixed reality musical interface and performance, ensemble performance, digital materiality, interactive musical system ACM Reference Format: 1 PROGRAM NOTES Unspoken is a mixed reality duet performance that explores the aesthetic possibilities of augmented reality technology as a medium for collaborative musical expression. Two performers with their tangible musical systems are communicated of each other’s gestural and spatial musical intention through a collaborative augmented reality interface. 2 PROJECT DESCRIPTION In the NIME community, the tactility of musical performance unfolds through the complex relationships between the musicians and their instruments. With sensors, motion capture and tracking technologies, the transformation of tactile sensations into sonic textures magnifies the musician’s innate interpretations of their physical senses and facilitates their unspoken intentions in sonic practices [ 1 , 2 ]. This performance advances the tactility in a mixed reality context, melding the tangible aspects of sound production with the symbolic significance and the dynamic interplay of hand gestures and movements in a collective augmented space. Two performers will engage in a 10-minute improvisational dialogue equipped with augmented reality (AR) headsets within this hybrid environment, whilst sitting away from each other to restrict visual communication. Mediated through a collaborative AR interface, each performer will be shown the other’s hand movements to help facilitate the expression of their intentions through the unspoken in gestural and spatial actions. One performer manipulates a physical keyboard synthesiser, while the other uses physical drawing materials to control a digital synthesizer, finding a shared space for different practices to reach autonomous commonality. To juxtapose different forms of tactile soundmaking with a recognisable keyboard, this performance introduces a digital synthesiser system in Pure Data [ 3 ] that applies the ML5 image classifier [ 4 ] implemented in a web application to capture the performer drawing and manipulate the sound based on the classified object. This work is built upon the author’s ongoing academic research in AR systems that facilitate collaborative performance by transmitting information about a musician’s bodily movements and instrument in supporting their awareness. Here, we seek out the aesthetic qualities of the AR technology in the form of collaborative musical expression. Attached to this submission is a demonstration video of the proposed performance linked in the section below. ∗Both authors contributed equally to this research. Licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). Copyright remains with the author(s). Yichen Wang and Sandy Ma. 2024. Unspoken. 1, 1 (September 2024), 2 pages. Music Proceedings of the International Conference on New Interfaces for Musical Expression NIME’24, 4–6 September, 2024, Utrecht, The Netherlands 2 Yichen Wang and Sandy Ma 3 MEDIA LINKS •Video Demo: https://www.youtube.com/watch?v=H5AdOE8QnKk 4 PERFORMANCE NOTES Practical Requirements •Two tables (one for each performer) •Venue should ideally be indoors without bright lights to ensure AR Headset functionality Technical Requirements • Projector with video mixer (preferably) for both performers’ video inputs. Performance requires at least two HDMI input. •2 * dual 1/4 inch audio channels •Internet connection Performers to bring •Router •Synthesiser systems (including drawing materials) •AR Headsets ETHICAL STANDARDS To our knowledge, there are no ethical conflicts involved in the making and presentation of this work. REFERENCES [1] Victor Evaristo Gonzalez Sanchez, Charles Patrick Martin, Agata Zelechowska, Kari Anne Vadstensvik Bjerkestrand, Victoria Johnson, and Alexander Refsum Jensenius. 2018. Bela-Based Augmented Acoustic Guitars for Sonic Microinteraction. In Proceedings of the International Conference on New Interfaces for Musical Expression, Thomas Martin Luke Dahl, Douglas Bowman (Ed.). Virginia Tech, Blacksburg, Virginia, USA, 324–327. https://doi.org/10.5281/zenodo.1302599 [2] Sasha Leitman. 2020. Sound Based Sensors for NIMEs. In Proceedings of the International Conference on New Interfaces for Musical Expression, Romain Michon and Franziska Schroeder (Eds.). Birmingham City University, Birmingham, UK, 182–187. https://doi.org/10.5281/zenodo.4813309 [3] Miller Puckette. 2023. Pure Data. http://msp.ucsd.edu/software.html. Accessed: 31 Jan 2024. [4] The ml5.js Team. 2023. ml5.js: Friendly Machine Learning for the Web. https://ml5js.org/. Accessed: 27 Dec 2023.