State Change: Merging Adaptive Music Technology with Disability Aesthetics

This work is licensed under a Creative Commons Attribution 4.0 International License. NIME ’25, June 24–27, 2025, Canberra, Australia © 2025 Copyright held by the owner/author(s). State Change: Merging Adaptive Music Technology with Disability Aesthetics MOLLY JOYCE, University of Virginia 1 Abstract State Change is a forthcoming album that transforms surgical records into musical lyrics and uses aural material generated by accelerometers and motion capture systems. By intertwining the medical and the aural, the album offers a sonic exploration of acquired disability, transforming personal and embodied experiences into a musical language. My interest in disability is rooted in my own lived experience. At the age of seven, I was involved in a car accident that nearly resulted in the amputation of my left hand. This life-altering event set me on a journey of adapting my relationship with music, leading me to experiment with different instruments and, more recently, to explore music technologies that align with my physical capabilities. These include accelerometers and motion capture devices, which have become central to my creative process. The featured track from the album, August 6, 1999, utilizes the MUGIC (Music/User Gesture Interface Control) device, an innovative motion sensor developed by violinist and composer Mari Kimura. The MUGIC translates hand movements into software parameters: “rotation” (pronation and supination) triggers high-frequency sine tones, while “yaw” (vertical axis rotation) triggers low-frequency sine tones. This track integrates lyrical text drawn from the surgical records of my first post-accident operation, alongside layered vocal elements that convey the emotional weight of that experience. Together, these elements create a deeply personal and evocative sonic representation of my journey with disability. 2 Program Notes August 6, 1999 is the opening track from State Change, a forthcoming album that utilizes surgical records as musical lyrics, combined with aural material produced from various accelerometer and motion capture systems. The album underscores the aural with the medical, ultimately seeking a musical manifestation of acquired disability. State Change showcases my experimentation with adaptive music technologies, such as the MUGIC (Music/User Gesture Interface Control) device. Developed by violinist and composer Mari Kimura, the MUGIC is a small, versatile, and affordable prototype motion sensor that uses movements to control software parameters. Fig. 1. Screenshot from video for “August 6, 1999.” NIME ’25, June 24–27, 2025, Canberra, Australia Molly, Joyce 3 Project Description State Change is an album that utilizes surgical records as musical lyrics, combined with aural material produced with various accelerometer and motion capture systems. The album seeks to underscore the aural with the medical, ultimately seeking a musical manifestation of acquired disability. My interest in disability stems from my personal experience acquiring a disability. At the age of seven, I was involved in a car accident that nearly amputated my left hand. Following the accident, I transitioned from various musical instruments to find one I could adapt to. More recently, this has led to engaging with various music technologies that fit my body, such as accelerometers and motion capture technology. State Change showcases my experimentation with these technologies, with the surgical records as lyrical material and production and engineering support from Grammy-award-winning producer William Brittelle and engineer Michael Hammond at Figure 8 Studios in Brooklyn, NY. Guest appearances include by Fire-Toolz (Angel Marcloid). 4 Performance Notes This will be a video playback/remote performance. August 6, 1999 utilizes the MUGIC (Music/User Gesture Interface Control) device, a small, versatile, and affordable prototype motion sensor that uses movements to control software parameters. Developed by violinist and composer Mari Kimura, it can connect via USB connection or Wi-Fi to a computer and translates accelerometer/movement data to the computer via software such as Ableton and Max. For this song, I applied the MUGIC parameters of “rotation” (based on pronation and supination of the hand) to trigger high-frequency sine tones layered onto each other and “yaw” (rotation on a vertical axis) to trigger low-frequency sine tones. While this approach, explicitly having precise pitches triggered by the positional orientation of the MUGIC (almost like a keyboard), was challenging in that it is difficult to have the device in a consistent aerial position, it also freed me to do something I could never do on traditional keyboard instruments. If I wanted to have a cluster of nearby pitches played together, I would have to lay most of my forearm down on the keyboard to get that effect, and inadvertently, I usually get unwanted pitches in that cluster. While I have grown to welcome the “mistakes” in this, especially with disability artistry and aesthetics embracing imperfections, such as Tobin Siebers’ concept of “disability aesthetics” and prizing physical and mental difference as a significant value in itself (2010), it also resulted from my attempt to conform to instruments I would never conform to. 5 Media Link(s) Video: https://youtu.be/a2_Xx39ytQw 6 Ethical Standards This work makes use of medical records provided by the author/artist, with full informed consent for their use in this project. The author/artist affirms that they have the right to share and use these records for creative and academic purposes. State Change NIME ’25, June 24–27, 2025, Canberra, Australia Funding for this project has been provided by the University of Virginia and the Virginia Commission for the Arts. Their support has enabled the development and dissemination of this work, though the views expressed herein are those of the author and do not necessarily reflect those of the funding institutions. References [1] Mugic Motion. (n.d.). MUGIC® | Revolutionize your creativity with advanced motion sensor technology . Retrieved January 30, 2025, from https://www.mugicmotion.com/ [2] Larsen, J.V., Overholt, D., & Moeslund, T.B. (2016). The Prospects of Musical Instruments For People with Physical Disabilities. New Interfaces for Musical Expression . [3] Murray-Browne, T., Mainstone, D., Bryan-Kinns, N., & Plumbley, M. (2011). The medium is the message: Composing instruments and performing mappings. New Interfaces for Musical Expression . [4] Siebers, T. (2010). Disability Aesthetics . University of Michigan Press. [5] Skuse, A. H. C., & Knotts, S. (2020). Creating an Online Ensemble for Home Based Disabled Musicians: Disabled Access and Universal Design - why disabled people must be at the heart of developing technology. Proceedings of the International Conference on New Interfaces for Musical Expression, 115--120. https://doi.org/10.5281/zenodo.4813266 [6] Tanaka, A. (2010). Mapping Out Instruments, Affordances, and Mobiles. New Interfaces for Musical Expression . [7] Wanderley, M. (2001). Gestural Control of Music. International Workshop -Human Supervision and Control in Engineering and Music. http://recherche.ircam.fr/equipes/analysesynthese/wanderle/pub/kassel