Unboxing: Public-Space Performance With Wearable-Sensors And SuperCollider

Unboxing: Public-Space Performance With Wearable-Sensors And SuperCollider TAKUMI IKEDA, Composer, Japan HANAKO ATAKE, Dancer, Japan IANNIS ZANNOS, Ionian University, Greece Additional Key Words and Phrases: Mobile Performance System, Wireless IMU, Outdoor Performance, Electronic Sound Performance ACM Reference Format: 1 PROGRAM NOTES This piece is a mobile outdoor performance where a dancer wearing mobile wireless IMU sensors controls sound generated by a laptop through their movements. The performance is mobile, and can take place in any available public space. The dancer’s movements acquired by the sensors drive sound generation algorithms running on SuperCollider and output from a mobile speaker. Through this work, we are showing that a performance that is not bound by location is possible through a relatively inexpensive and easy-to-construct performance system. The title “Unboxing” refers to escaping from the economic, social, political, and artistic constraints of conventional performances. It also alludes to “unboxing” as an internet meme in online videos where one does not know what is contained in the box before it is opened - as the performance data and the resulting sound structures cannot be evaluated beforehand. While use of motion sensors in music technology is gradually becoming widespread, most of the basic questions regarding its aesthetic function and nature as performance form remain unanswered. Why employ motion sensors instead of other controllers? What is the intention of including movement in music in the first place? etc. Dancers are arguably more aware than musicians about how to express themselves by moving their bodies. However, they are less accustomed than musicians to having their movements directly converted into sound. Even if a musician or programmer, for musical reasons, assigns different functions to each of the three motion axes of a sensor, it is very unlikely that the dancer will move as the musician or programmer intended. Therefore, it is necessary for the musician to take into account the way dancer’s movements when programming the sensors. In our work, the dancer discovers how to perform with the sensors while also considering their choreography. Then the musician adjusts the program according to the dancer’s choreography. The performance is gradually brushed up over iteration of these processes. In light of the above, describing this piece simply as music or dance is inappropriate. The present performance thus ventures into the field of uncategorisable performances, which represent at the same time a threat to some systems and a clue to escape the system, as necessary condition for defining new performance conditions that lead to paradigms better Licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). Copyright remains with the author(s). Takumi Ikeda, Hanako Atake, and Iannis Zannos. 2023. Unboxing: Public-Space Performance With Wearable-Sensors And SuperCollider. 1, 1 (May 2023), 4 pages. Music Proceedings of the International Conference on New Interfaces for Musical Expression NIME’23, May 31–June 2, 2023, Mexico City, Mexico 2 Ikeda et al. suited for performance with motion sensors. Unboxing therefore refers here also to venturing beyond the traditional context of computer-aided performances. This project aims to open up computer music creativity to a wider audience through frugal technology and escape Western-centric concepts of music and dance. Performances with sound and electronics as their main components which also take into account the full circumstances of the creation context, rather than just the sound, require a better descriptive name. We propose the term “electronic sound performance” as alternative. In general, computer-aided performances can easily be black-boxed. In other words, the system’s internal structure is hidden, and the relationship between inputs and outputs can frequently be difficult to understand objectively. However, understanding causality between actions and sounds is an integral part of enjoyment and appreciation of in musical performances. That is why it is essential to make the black box as transparent as possible, i.e., to make the relationship between inputs and outputs easy to understand. The same can be said for non-electronic music. “Unboxing” in music can be defined as the presentation of sound as the sounds themselves, rather than using sound to express something other than sound, or in musical composition, counter-measures such as simplifying or pre-disclosing the methods and processes of forming music. Our performance investigates this principle as an alternative way for building performances with sensors in conjunction to novel performance settings, of which outdoors street performance is a prime example. In some cultures, the existence of plazas as public spaces is not axiomatic. In areas where the culture of the plaza does not exist, sometimes a road serves as a plaza. However, such a location serves only as a temporary performance space, and any attending audience groups will eventually disband. Street performances are extremely rare in such contexts. In societies that do not support plaza culture style, arts are usually allowed to exist only in closed spaces such as theaters, concert halls, and galleries whose function is predefined. However, it can be argued that a major factor inhibiting street performances are prevailing cultural stereotypes, rather than the institutions themselves. Aural and visual stimuli as well as activities from passersby in an open space environment are perceived as interference or “noise” that distracts from the performance itself. In our approach, we take a different stance: The performance aims to creatively embrace the different conditions of the open space environment in which it is embedded. The performer is obliged to function within these constraints, which do not exist in predestinated closed spaces, and must moreover decide how to either respond to the situation or do nothing. The present performance exposes both the dancer and the live coder jointly to these conditions and sets them the task to face together on the spot the challenge of creating new performance constraints outside the box of predefined computer-music performance settings (see Figure 1). 2 PROJECT DESCRIPTION Our mobile performance system in this piece consists of the following elements. All hardware devices are commercially available. •2 x wireless IMU sensors : M5StickC Plus (M5Stack) [1] •USB-powered Wi-Fi router : TL-WR802N (TP-LINK) [2] •Optional battery-powered speaker •Windows PC •SuperCollider [3] M5StickC Plus is a thumb-sized microcomputer with a lithium polymer battery that is rechargeable via a USB cable (see Figure 2). It can run continuously for about one hour in this system by minimizing the duration of the LCD screen Unboxing: Public-Space Performance With Wearable-Sensors And SuperCollider 3 Fig. 1. Outdoor performance. display and other programming tricks, and communicates wirelessly over Wi-Fi. A windows PC powers the Wi-Fi router over USB. Additionally, the use of a battery-powered speaker allows for outdoor performance. Fig. 2. Sensors and Wifi-Router. The ESP32-based M5StickC Plus supports Wi-Fi and Bluetooth. The data from the IMU sensor inside M5StickC Plus is transmitted over the network in Open Sound Control (OSC) format by the program. ArduinoOSC [ 4 ] is used to send OSC messages. When SuperCollider receives IMU data via OSC messages, it writes the values to the control busses via appropriate numeric mapping to make them available to the sound synthesis programs. The performance combines synthesis programs are controlled by the dancer’s movements with autonomously operating programs. Programs controlled by the dancer’s movements are programmed to account for the musical effects and the dancer’s movements. If we compare it to an orchestral piece in concerto form, the dancer’s program corresponds to the soloist and the autonomous program to the orchestra. Rather than leaving all controls to the soloist, giving multiple perspectives to the controls allows for more dynamic and varied expression. 4 Ikeda et al. 3 PERFORMANCE NOTES We are aiming for a prerecorded video presentation as it is difficult for us to travel to the conference location. The video at the URL noted in this paper is a test version. If this project is accepted, we will again record and submit a completed version of the video. The performer of this piece is a dancer wearing wireless IMU sensors. The dancer performs in various locations throughout the town. The performance locations include points usually deemed inappropriate as performance contexts, such as shopping streets, near the railroad, and a major road, to explore various possibilities for outdoor performance. 4 MEDIA LINKS •Video: https://www.youtube.com/watch?v=2XG5qEXe3Es ACKNOWLEDGMENTS The authors would like to thank Professor Satoru Takaku for his support. REFERENCES [1] https://docs.m5stack.com/en/core/m5stickc_plus [2] https://www.tp-link.com/us/home-networking/wifi-router/tl-wr802n/ [3] https://supercollider.github.io/ [4] https://github.com/hideakitai/ArduinoOSC