transcriptions

This work is licensed under a Creative Commons Attribution 4.0 International License. NIME ’25, June 24–27, 2025, Canberra, Australia © 2025 Copyright held by the owner/author(s). transcriptions TAKUMA KIKUCHI, Keio university, Faculty of Environment, and Information Studies, Japan RIKI SAITO, Keio university, Faculty of Environment, and Information Studies, Japan RISAKO SHIBATA, Researcher, Keio Research Institute at SFC, Japan ATSUYA TSUCHIDA, Keio university, Faculty of Policy Management, Japan KENSHIRO TAIRA, Keio university, Faculty of Environment, and Information Studies, Japan NIMISHA ANAND, Keio university, Faculty of Policy Management, Japan RYOHO KOBAYASHI, College of Performing and Visual Arts, J. F. Oberlin University, Japan YUTA UOZUMI, Keio university graduate school, Graduate School of Media and Governance, Japan SHINYA FUJII, Keio university, Faculty of Environment, and Information Studies, Japan Program Notes “transcriptions” is an improvisational musical performance piece in which two performers recursively mimic each other's movements, utilizing postural sensing and tactile feedback. The misalignments and errors that occur in the process of imitation and the dynamic changes in the relationship between the performers caused by the interveners (the System Jockey and the Intervener) are intertwined to produce unpredictable movements and sounds. Performers wear special suits equipped with gyro-sensors and exciters. The gyrosensor converts the postural movement data into vibrations, which are transmitted to the exciter of the other performer. The performers respond to these vibrations and move, mimicking each other's postures. As this chain of imitation is repeated, errors and misalignments due to tactile perception, physical ability, initial position, differences in posture, spatial constraints, and other factors accumulate, and new movements emerge. In this work, all sounds are generated from the performers' movements. Contact microphones are attached to the performers' suits to capture the sound generated by their movements. The sounds are processed and output, and electronic sounds are generated from its volume information, so that the relationships between the performers and the changes in their movements can be expressed sonically. The relationships among the performers, and the interaction between the system and the performance environment generate the performers' movements and sounds, and the system jockey and the intervener intervene in these interactions. The result is an improvisational performance in which nonlinear changes in movement and sound are intertwined with intentional control. “transcriptions” is a work that presents a new form of improvisational expression by actively utilizing creative emergence through chains of imitation. NIME '25 June 24-27, 2025, Canberra, Australia Kikuchi et al. Fig. 1. transcriptions Project Description This work is an improvisational musical performance using a suit equipped with a gyro-sensor and exciter, and employs mutual imitation between two performers through tactile feedback and motion sensing. In the process of imitation, various factors such as differences in physical characteristics, interpretation of tactile feedback, and spatial constraints interact to cause errors and deviations in movement. The accumulation and circulation of these errors and misalignments results in the creation of new movements that the performers themselves do not anticipate. Contact microphones attached to the suit capture the sound emitted from the performer's movements, after which it is processed and output. This allows the dynamics of the movements and the relationship between the two performers, which is unique to this system, to be directly reflected in the sound. This performance consists of two performers, a System Jockey, an Intervener, a movement system, and a sound system. 1.1 Movement System This system uses a suit as an interface to sense the performer's movements and provide tactile feedback. The suit contains a gyro sensor, exciter, and LEDs, each of which plays the following roles: 1.1.1 Gyro sensors It is worn on the upper back of the suit and detects thoracic spine tilt transitions. This extracts abstracted postural shifts of the performer. The acquired data is wirelessly transmitted to the laptop and used to generate haptic feedback. 1.1.2 Exciters An exciter is a type of speaker. They do not have cones but receive audio signals and make resonant sounds by causing objects they come in contact with to vibrate. The suit has a total of three exciters attached to both the shoulders and back. These vibrate to provide movement guidance to the performer. transcriptions NIME '25 June 24-27, 2025, Canberra, Australia 1.1.3 LED Installed in the same position as the exciter, the LED visually represents the vibration feedback received by the performer. The light intensity changes in response to the vibration of the exciter, emitting light at a maximum of 31,000 mcd. Fig.2: Schematic Diagram of the Performer Suit Through this suit, the performer's movement data is acquired in real time and tactile feedback is generated. The data acquired by the gyroscopic sensor is analyzed by Max for Live to generate the corresponding vibration patterns. The generated vibration is output as an audio signal and transmitted as tactile feedback to the exciter worn by the other performer. The performer receives the exciter's vibration as a “push” and moves his/her body in that direction (See Fig. 3). This movement is captured again by the gyro sensor and sent to Max for Live, which generates a new vibration. By repeating this feedback loop, the transmission of vibrations according to the mapping induces the performers to move, resulting in the performers alternately mimicking each other's postural transitions. LEDs also emit light in response to the exciter's vibration, visually communicating the vibration feedback to the audience. Fig.3: Diagram illustrating the rule of movement. NIME '25 June 24-27, 2025, Canberra, Australia Kikuchi et al. Fig.4: Movement System Workflow Fig.5: Movement System Diagram 1.2 Sound System In this system, sound is processed in the following ways: 1.2.1 Input from contact microphone The sound of accessories and clothing rubbing against each other is acquired from multiple contact microphones attached to the suits worn by the performers. 1.2.2 Sound processing and generation The laptop processes the microphone sounds in real time and processes the raw sound. It also analyzes the input volume information from the contact microphones and generates electronic sounds based on this information. The sound processing/generation is performed as follows: transcriptions NIME '25 June 24-27, 2025, Canberra, Australia • Processing of Raw sound: The sound acquired from each performer is output from separate speakers. The input sound of each performer is processed separately so that the frequency range is different for each performer's sound. This creates a clear dialogue, like a turn-taking process, when the Performers move alternatingly. On the other hand, when both performers influence each other at the same time, the high and low tones blend together and are perceived as a single integrated sound. • Generation of Electronic Sound: The higher the volume of the contact microphones, the higher the grain density of a metallic pulsing sound. This sound increases in tempo as the performer moves more violently, reinforcing the dynamics of the movement with sound. The average of the input volume of all microphones over a period of time is calculated, and the louder this average is, the louder the generated drone sound is. This acoustic processing makes the vicissitudes of the performance development audible, allowing the viewer to perceive the dynamism of the performance sonically, as well as visually. In addition, the time that the input volume of the contact microphones attached to each performer is below a certain value is measured, and a low rumbling sound crescendos according to the performer's stagnancy. This makes audible the equilibrium of forces produced by performers pulling each other's limbs or pushing each other's bodies. This creates a sense of tension, giving the audience an intuitive sense of the struggle for power between the performers. 1.2.3 Sound Output The system jockey decides which sounds are output according to the performance situation. Sounds are mixed in real-time accordingly, and output from the speakers. Through this process, the dynamics and relationships of the performer's movements are reflected in the sound, creating a viewing experience in which the performer's movements and the sound seem to be one. Through this process, the dynamics and relationships of the performer's movements are reflected in the sound, creating an experience in which the performer's movements and the sound seem to be unified. Fig.6: Sound System Diagram 1.3 Performer Two performers move alternately in a chess turn-like format. They attempt to predict each other's movement from the tactile feedback generated by the movement system and mimic each other's NIME '25 June 24-27, 2025, Canberra, Australia Kikuchi et al. movements. Performers do not move voluntarily, as they require tactile feedback and external factors (such as being pushed) in order to move. 1.4 System Jockey The System jockey manipulates the movement system and sound system. Specifically, they perform the following actions: 1. Setting the amount of time a performer can move in one turn and affects the speed of the Performer's movement. 2. Adjusting the pause time between turns. This controls the rhythm of the movement and determines when the performer begins the next movement. 3. Stopping the movement system, forcing the performer to stop moving. 4. Selecting the sounds to be output Through these manipulations, the interpretation of the vibrational feedback and the intervention in the body's state of being intentionally generates “displacement” to foster the development of the performance. 1.5 Intervener The Intervener physically interferes with the performer and is responsible for initiating and transforming the imitation loop. Specifically, the intervener performs the following actions: 1. Alter the performer's posture at the start of the imitation, thereby manipulating the conditions of imitation and influencing the process of imitation. 2. Intervening by pushing the Performer to generate the initial action and initiate the chain of imitation. 3. Add physical constraints to the performers by tying their bodies to each other with tape, chains, or other props. Physical constraints create movement displacement and alter the mimetic process. Through these actions, the Intervener creates intentional “deviations” within the mimetic loop, encouraging diversity of movement and changes in sonic expression, thereby creating the development of the performance. Table 1. Hardware Configuration Hardware Quantity Note/Purpose Laptop 2 1 for Sound System, 1 for Movement System Audio Interface 2 1 for Sound System, 1 for Movement System(exciter) Performer Suit 2 Includes gyroscope sensors, exciters, and LEDs PA Speaker 2 Output Sound which generates Sound System Exciter Power Amplifier 1 Powers the exciters in the Performer Suit transcriptions NIME '25 June 24-27, 2025, Canberra, Australia Contact Microphone 8 Captures sound from performer’s movement Mat 1 For the safety of the performers Wi-fi Router 1 Connects gyro sensor to PC Stage Light 1 Provides stage illumination Extension Cable 2 Provides power connectivity Arduino Uno 2 Controls LEDs TRS Cable 2 Connects Sound System components (Audio Interface to Speaker) Note: All equipment is assumed to be self-supplied. 2 Performance notes 2.1 Spatial Requirement The most suitable spatial requirement for this work is an indoor or outdoor stage that can provide a rectangular space with a length of 5m and a width of 7m. A simple stage will be set up with a 2 cmthick mat on the ground for the performance (Fig. 7). A Processing Unit (contained in a cabinet) was set up behind the stage, to control the movement system and sound system using a laptop, and a system jockey was stationed there during the performance. The two performers move around the stage while performing, and the Intervener enters the stage only when interfering with the performers and waits at the side of the stage during the rest of the time. In addition, one speaker will be placed at the ends of the stage on each side to output sound. A PAR light is placed on the ground in the center in front of the stage and is turned on only during the performance to illuminate the stage. It is preferred that the audience watches the performance from the front of the stage, but this can be changed according to space limitations and the layout of the venue. In consideration of actor safety, from the viewpoint of securing the movement routes, cable lengths and wiring will be safely managed, and the length between speakers and lights will be long enough to allow actors to pass through. Of course, the size of the space can be adjusted according to the exhibition venue. The performance time can also be adjusted within a given time, such as short (15 minutes) or long (30 minutes) versions, as there is an aspect of improvisation. With regard to these aspects, the experience of the exhibition debut at the ZOUNO-HANA FUTURESCAPE PROJECT 2024 in December 2024 demonstrates the feasibility of this work as a work that can be realized with minimum preparation time, and flexibility. NIME '25 June 24-27, 2025, Canberra, Australia Kikuchi et al. Fig.7 Layout Diagram Fig.8 Processing Unit Diagram 2.2 FLOOR PLAN Based on the debut of the ZOU-NO-HANA FUTURESCAPE PROJECT 2024 held in December 2024, this work will have the system set up in advance, and will be brought in and placed on the stage just prior to the performance. This allows for flexibility in the event of limited turnaround time with other performances. The following is a description of the setup plan and equipment configuration for the performance: 1. Pre-setup in the backyard, etc.: Laptop, audio interface, vibration control equipment, etc. are all stored together in a machine box, and power, wiring, and operation checks are completed offstage. Exciters, contact microphones, and other devices worn by the performers are also attached to their suits in advance. transcriptions NIME '25 June 24-27, 2025, Canberra, Australia 2. Stage implementation: Just before the performance, the machine box and speakers are moved into place, power is secured, and a final check is made. Since the connections have already been made, the system can be up and running quickly. A total of about 1400W of power is required to control exciter vibration and sound output. Once the performance is over, the equipment, including the machine box, can be dismantled easily so as not to interfere with subsequent performances. In past exhibitions, we have been able to operate effectively with a plan in which the equipment is prepared in advance in the backyard, etc., so that loading, performance, and unloading can be done in the minimum amount of time required. Media Link(s) Video: https://youtu.be/F0hwyvr1jEM 3 Acknowledgments This work was supported by JST Moon Shot Type R&D Project, JPMJMS2215. Projects Supported by the Ishii-Ishibashi Fund (The Keio University Grant for Early Career Researchers) We would also like to thank all those involved in the ZOU-NO-HANA FUTURESCAPE PROJECT 2024 and the SFC ORF(Open Research Forum) 2024 for their cooperation in the exhibition. 4 Ethical Standards This work was conducted in accordance with ethical standards and relevant guidelines. This work was funded by JST Moonshot Type R&D Project, JPMJMS2215. There are no conflicts of interest, financial or non-financial. The performers, System Jockey and Intervener understood the purpose and methods of the study and the risks and benefits to be gained, obtained informed consent, and physical and psychological well-being was considered to be paramount. In particular, safety aspects were carefully considered to ensure that physical restraints and vibratory stimulation were not excessive, and a system was in place to allow performers, System Jockey, and Intervener to withdraw from participation at any time. In addition, sensor data and video/audio materials obtained in the course of the study were used for research purposes only, and personally identifiable information was anonymized and stored in a secure manner. No harm was done to Performer, System Jockey, or Intervener at any stage.