Improvising within "GravField": A Participatory Live-coding Performance Exploring How Digital Objects Mediate Intercorporeal Movements in Collocated Mixed Reality

Improvising within "GravField": A Participatory Live-coding Performance Exploring How Digital Objects Mediate Intercorporeal Movements in Collocated Mixed Reality BOTAO AMBER HU∗,Reality Design Lab, USA YUEMIN HUANG, East China Normal University, China MINGZE CHAI, Independent, China XIAOBO AARON HU, Independent, China YILAN ELAN TAO, Reality Design Lab, USA REM RUNGU LIN, The Hong Kong University of Science and Technology (Guangzhou), China Fig. 1. GravField is an experimental intercorporeal performance system designed for collocated mixed reality environments. In this system, mixed reality headset wearers collaboratively improvise using their body movements to alter the auditory and augmented visual feedback, which is dynamically orchestrating by a live musician. This figure presents the augmented reality screenshot of an actual GravField performance from a third-person spectator perspective. As mixed reality technologies evolve, they blur the boundaries between digital and physical worlds, prompting us to reevaluate how we engage with digital objects, our bodies, and our consensus reality. Inspired by contact improvisation, we introduce GravField, a live performance mixed reality system that uses metaphorical audiovisual mediators to guide interdependent behaviors among participants. These mediators—incorporating springs, ropes, and magnetic fields—shape bodily movement and social dynamics through dynamic audio feedback, assigned by a live-musician/live-coder. We invite participants to join performances to explore how digital objects mediate intercorporeal interaction and entangled embodied behavior with technology. This work investigates the concept of "digital physics" in mixed reality, where digital objects shape somatic experiences to examine the relationships between embodiment, interpretation, alterity, and background, drawing from post-phenomenology. ∗Corresponding author Authors’ Contact Information: Botao Amber Hu, Reality Design Lab, New York City, USA, [email protected]; Yuemin Huang, East China Normal University, Shanghai, China, [email protected]; Mingze Chai, Independent, Shanghai, China, [email protected]; Xiaobo Aaron Hu, Independent, Shanghai, China, [email protected]; Yilan Elan Tao, Reality Design Lab, New York City, USA, [email protected]; Rem RunGu Lin, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China, [email protected]gz.edu.cn. This work is licensed under a Creative Commons Attribution 4.0 International License. NIME ’25, June 24–27, 2025, Canberra, Australia ©2025 Copyright held by the owner/author(s). NIME ’25, June 24–27, 2025, Canberra, Australia Botao Amber Hu et al. Additional Key Words and Phrases: Augmented Reality, Multiplayer, Bodily Play, Collocated Mixed Reality, Live Orchestrator, Collaborative Movement, Improvisation, Embodied Interaction, Intercorporeality, Entangled Assemblages 1 Program Notes How do digital objects mediate our intercorporeal movements in the future of mixed reality? We invite you and a friend to explore and interact with digital objects together in GravField. GravField is an experimental, participatory live performance set in a collocated mixed reality (MR) environment, where participants wearing MR headsets collectively create real-time music through their body movements, guided by digital objects (metaphorical audiovisual mediators) controlled by a live-musician/live-coder. This setup tracks intercorporeal signals—such as distance, area formation, relative height differences, and head-motion synchronization—and maps them into metaphorical audiovisual mediators, offering cues about "digital physics." As live-musicians adjust these mappings, the constantly evolving patterns shape and influence participants’ improvisation of their own bodily movements. In turn, these evolving movements continually alter how the sound is generated, creating an immersive, collaborative feedback loop that reimagines how we move, connect, and create in shared augmented spaces. 2 Project Description GravField (short for "Gravitational Field") is an experimental and participatory performance system in a collocated mixed reality environment. Participants wearing mixed reality headsets are invited to join and collectively create real-time music through their collaborative body movements, guided by digital objects (metaphorical audiovisual mediators) dynamically assigned by live-musicians/live-coders. This experiment harnesses intercorporeal signals derived from participants’ body movements, including factors like distance, area formation, relative height differences, and head motion synchronization. Live-coders dynamically map these signals into metaphorical audiovisual patterns in augmented reality, providing participants with cues about the relationships between players. The live-coded patterns shape and influence participants’ improvisation of body movements, while adjustments in live coding dynamically alter how these movements generate sound, influencing the overall improvisational experience. In addition, spectators outside the live performance area can enjoy the mixed reality performance through projection screens and TVs, enhancing audience engagement. GravField aims to explore the vast potential of intercorporeal signals, creating a communicative, playful, and co-creative space where players’ interconnected bodies become the instruments of expression. 2.1 Motivation and related works In recent years, the Human-Computer Interaction (HCI) community has increasingly focused on the experiential body, developing numerous methodologies and tools for movement-based interactions. This shift emphasizes somatic and artistic relationships in interactive system design [ 4 , 5 , 7 – 9 ]. Researchers use wearable and mobile technologies to record biodata and transform it into somadata, deepening our understanding of body expression [ 1 ]. As the CHI community explores new mechanisms of lived body interaction, Merleau-Ponty’s phenomenological concept of intercorporeality [ 2 , 10 ] has become particularly relevant. This concept highlights how intercorporeal interactions facilitate sense-making practices in cognitive processes [ 11 ]. While some projects, like the contact improvisation mixed reality art Cell Space [ 6 ], have begun exploring intercorporeality through Mixed Reality technology, there remains a significant gap in research on collocated MR experiences focusing on intercorporeal interactivity. Though embodied interaction is fundamental to MR head-mounted displays, few studies have examined how multiple users’ physical movements can interact with virtual mediators as augmented overlays to create engaging collaborative behaviors in shared MR spaces. Unlike Francoise et al.’s study on player body movement and sound interaction in live-coding systems [ 3 ], our research examines intercorporeality between multiple participants in a performance context. We create a continuous feedback loop where live coders dynamically adjust the digital physics of objects, fostering an ongoing dialogue between participants, digital physics, and live coders. 2.2 Roles In this performance, there are four types primary participants: Immersants, Live-Coders, Spectators, as shown in Figure 2. • Immersants: As primary participants, immersants are dancers equipped with headworn optical see-through Augmented Reality headset, ensuring their connection to the real world and other dancers without isolation or latency. They also see augmented links between dancers, enhancing their awareness of the intercorporeal "field" collaboratively shaped by live-coders and dancers. • Live-coders: As performers who engage in the real-time creation and manipulation of augmented affordance to generate augmented reality visual and audio pattern. They use programming languages and digital audio workstation to execute codes and assign audio effects on the fly. Improvising within "GravField" NIME ’25, June 24–27, 2025, Canberra, Australia • Spectators: Equipped with handheld AR devices, Spectators view the performance in real-time, observing the augmented interconnections from a first-person perspective. They can enter the performance field, transforming from passive observers to active participants, blurring the boundaries between themselves, the dancers, and other spectators. • Bystander: Those without devices can still engage with "GravField" by watching through a monitor screen, offering a third-person view of the immersive experience. This allows a broader audience to appreciate the complex interplay of connections and movements within the performance. Fig. 2. The system setting of GravField Fig. 3. Live-coding affordance pattern: The Rope 2.3 Immersant’s Experience When the performance begins, three spectators are invited to wear the mixed reality head-mounted device HoloKit, transforming into ’immersants’ in the process. Through the stereoscopic AR display, all immersants perceive themselves NIME ’25, June 24–27, 2025, Canberra, Australia Botao Amber Hu et al. Fig. 4. Live-coding affordance pattern: The Spring Fig. 5. Live-coding affordance pattern: The Magnetic Field as connected by a virtual line. This line assumes a crucial role in our audio-visual system, providing diverse possibilities through its various forms. As the immersants move, the live-coder dynamically generates the line’s shape and corresponding sound effects in response to their movements. These audio-visual feedbacks spark the immersants’ imagination, fostering whole-body engagement and intercorporeal collaboration with fellow immersants. 2.4 Live-coding Metaphorical Audiovisual Mediators Live-coders dynamically create metaphorical audiovisual mediators (or simply "affordance patterns") based on intercorporeal signals (Fig. 3). Affordances are patterns or structures within the GravField system that respond to specific intercorporeal signals derived from participants’ movements. These signals include factors such as distance, area formation, relative height differences, and head motion synchronization. Participants will be introduced to the affordance patterns developed thus far, such as the rope, the spring, and magnetic field patterns. • The rope affordance pattern (see Fig.3 encourages participants to swing collaboratively to gain maximum feedback. By synchronizing their swinging motions, participants create a continuous and rhythmic audiovisual experience. The chain pattern emphasizes the importance of coordination and shared movement, allowing participants to explore the concepts of rhythm, synchronization, and collective expression. Participants will engage in swinging movements together, experiencing the immersive and captivating nature of the chain pattern in generating collaborative music. • The spring affordance pattern (see Fig.4) connects bodies with strong auditory feedback, encouraging participants to play with the distance between them. As participants move closer together, the auditory feedback strengthens, reinforcing their proximity and fostering a sense of unity. Conversely, when participants move apart, the auditory feedback diminishes, prompting them to explore the musical possibilities of distance. Participants will experiment with the spring pattern, exploring the relationship between distance and sound, and discovering how it influences their collaborative music creation. Improvising within "GravField" NIME ’25, June 24–27, 2025, Canberra, Australia • The magnetic field affordance pattern (see Fig.5) is designed to attract neighboring body relationships. By creating a spatial magnetic vector field between participants, it encourages individual exploration and expression while maintaining a sense of proximity and collaboration. Participants will learn how to utilize the magnetic field pattern to shape the dynamics and interactions between their bodies, creating distinct musical outcomes. By engaging with these affordance patterns, immersants explore the dynamics and interactions between their bodies, ultimately contributing to the collaborative music creation. 2.5 Audience Engagement Spectators outside the performance area can observe the mixed reality performance through projection screens and TVs. This allows a broader audience to appreciate the intricate interplay of connections and movements within the performance. By providing spectators with a glimpse into the immersive experience, GravField enhances audience engagement and expands the reach of the performance. 2.6 Conclusion GravField presents an innovative and participatory live performance that explores the potential of intercorporeal signals in a collocated mixed reality environment. Through collaborative body movements, immersants generate real-time music guided by live-coders’ programmed settings. The interplay between intercorporeal signals and live coding shapes the audio-visual experience, fostering a communicative, playful, and co-creative space where interconnected bodies become instruments of expression. By extending the immersive experience to spectators through projection screens and TVs, GravField broadens audience engagement and showcases the complex dynamics of the performance. 3 Technical Notes Our proposal for the NIME 2025 conference is a 1-hour exhibition installation for improvised performance featuring "GravField". We require a bright performance space, a projector or TV to display GravField in real time, stereo audio output, a Mac running the GravField executable program (built with Unity). We will provide HoloKits, and audience can use either their own iPhones or those provided by us to run the GravField app, which can be downloaded from the Apple App Store. For a clearer understanding of the setup, please refer to the accompanying performance video. 4 Media Links •Video: https://vimeo.com/955522087 •Project URL: https://reality.design/project/gravfield 5 Ethical Statements We adhere to NIME’s ethical guidelines by ensuring that our creative and technological practices respect both human and non-human agents. References [1] Miquel Alfaras, Vasiliki Tsaknaki, Pedro Sanches, Charles Windlin, Muhammad Umair, Corina Sas, and Kristina Höök. 2020. From Biodata to Somadata. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376684 [2] Sara Eriksson, Åsa Unander-Scharin, Vincent Trichon, Carl Unander-Scharin, Hedvig Kjellström, and Kristina Höök. 2019. Dancing With Drones: Crafting Novel Artistic Expressions Through Intercorporeality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300847 [3] Jules Françoise, Sarah Fdili Alaoui, and Yves Candau. 2022. CO/DA: Live-Coding Movement-Sound Interactions for Dance Improvisation. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI ’22). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3491102.3501916 [4] Kristina Höök, Caroline Hummels, Katherine Isbister, Patrizia Marti, Elena Márquez Segura, Martin Jonsson, Florian ’Floyd’ Mueller, Pedro A.N. Sanches, Thecla Schiphorst, Anna Ståhl, Dag Svanaes, Ambra Trotto, Marianne Graves Petersen, and Youn-kyung Lim. 2017. Soma-Based Design Theory. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA ’17). Association for Computing Machinery, New York, NY, USA, 550–557. https://doi.org/10.1145/3027063.3027082 [5] Caroline Hummels, Kees C. J. Overbeeke, and Sietske Klooster. 2007. Move to Get Moved: A Search for Methods, Tools and Knowledge to Design for Expressive and Rich Movement-Based Interaction. Personal and Ubiquitous Computing 11, 8 (Dec. 2007), 677–690. https://doi.org/10.1007/s00779006-0135-y [6] Rem RunGu Lin, Botao Amber Hu, Koo Yongen Ke, Wei Wu, and Kang Zhang. 2024. Cell Space: Augmented Awareness of Intercorporeality. Proc. ACM Comput. Graph. Interact. Tech. 7, 4 (July 2024), 53:1–53:10. https://doi.org/10.1145/3664213 [7] Lian Loke, Astrid T. Larssen, Toni Robertson, and Jenny Edwards. 2007. Understanding Movement for Interaction Design: Frameworks and Approaches. Personal and Ubiquitous Computing 11, 8 (Oct. 2007), 691–701. https://doi.org/10.1007/s00779-006-0132-1 NIME ’25, June 24–27, 2025, Canberra, Australia Botao Amber Hu et al. [8] Lian Loke and Toni Robertson. 2013. Moving and Making Strange: An Embodied Approach to Movement-Based Interaction Design. ACM Transactions on Computer-Human Interaction 20, 1 (April 2013), 7:1–7:25. https://doi.org/10.1145/2442106.2442113 [9] Elena Márquez Segura, Laia Turmo Vidal, Asreen Rostami, and Annika Waern. 2016. Embodied Sketching. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16). Association for Computing Machinery, New York, NY, USA, 6014–6027. https: //doi.org/10.1145/2858036.2858486 [10] Maurice Merleau-Ponty, Donald A. Landes, Taylor Carman, Claude Lefort, and Maurice Merleau-Ponty. 2014. Phenomenology of Perception (this ed. 1. publ. in paperback ed.). Routledge, London New York. [11] Christian Meyer, Jürgen Streeck, and J. Scott Jordan (Eds.). 2017. Intercorporeality: Emerging Socialities in Interaction. Oxford University Press, New York, NY.