Playing the sound image – individual immersive sound performance by a performer wearing parametric speakers

This work is licensed under a Creative Commons Attribution 4.0 International License. NIME ’25, June 24–27, 2025, Canberra, Australia © 2025 Copyright held by the owner/author(s). Playing the sound image – individual immersive sound performance by a performer wearing parametric speakers RISAKO SHIBATA, Keio University, Faculty of Environment and Information Studies, Japan MIKI KANDA, Keio University, Faculty of Policy Management, Japan KENTA TANAKA, Researcher, Keio Research Institute at SFC, Japan RYOHO KOBAYASHI, College of Performing and Visual Arts, J. F. Oberlin University, Japan YUTA UOZUMI, Keio University Graduate School, Graduate School of Media and Governance, Japan SHINYA FUJII, Keio University, Faculty of Environment, and Information Studies, Japan Program Notes This is an improvisational performance in which dynamic spatial movement of sound is created in real time by a performer wearing parametric speakers. Parametric speakers differ greatly from ordinary speakers in that they have a super narrow directivity. Therefore, only the audience in a straight line can hear the sound. Listening to the direct sound brings the sensation that the sound field has suddenly appeared near the ears, as well as tactile auditory stimulation. In addition, the sound is easily reflected by walls and objects, creating a virtual sound source on the spot. In this work, parametric speakers are attached to both palms and shins of the performer and made wireless. The composer also remotely selects and adjusts the sound parameters of the sound source emitted from the speakers. The performer crosses the acoustically transforming space and enters between the audience, changing the distance and direction between the walls and the audience, creating a shift in the sound image. The sound image changes abruptly due to the switch between direct and reflected sound, and pitch changes due to the rapid movement of the speakers. The performer's movement changes the hearing, response, and relationship between the performer, the composer, and the audience, bringing to light more complex sound source movement and improvisational acoustic experiences. Fig. 1. Performance at SFC, Keio Univ. NIME '25 June 24-27, 2025, Canberra, Australia Shibata et al. Fig.2. Performance for archive at SFC, Keio Univ. Project Description This work aims to extend the immersive and audiovisual experience through the spatial movement of sound, by exploring ways of creating spatial acoustics and acoustic/improvisational body expression performance. The actual physical movement of the sound output device adds further physical and spatial information to the sound. We believe that this sonic precision is effective in enhancing the immersive experience of spatial acoustics. In this performance, the use of a super directional sound source will maximize the immersive experience of the work through the spatial movement and reflection of sound. In addition, in the order of performance production, the first step is to design the movement patterns of the sound source, then to consider the physical movements to realize these patterns, and finally to create the sounds that match these patterns. Rules for interaction with the audience through sound will also be devised. The performer becomes his or her own personal composer for each audience member, shaping the sound image and manipulating the space through physical movement. We will propose a new way of creating richer and more stimulating three-dimensional sound. Parametric speaker Parametric speaker is an acoustic device that uses ultrasonic waves as a carrier wave and modulates its amplitude or frequency to give sound waves a sharp directivity. Compared to audible sound, ultrasonic waves have a sharper directivity and exhibit behavior similar to that of light, such as traveling in a straight line and being reflected. By using these waves as a carrier wave, sound from a parametric speaker hardly diffuses. In contrast to audible sound radiated from ordinary speakers, which diffuse and propagate in a semicircle around the speaker due to the wave diffraction effect and other factors, the sound from parametric speakers has a linear audible area (fig. 3). In this performance, a parametric speaker was produced with the “Parametric Speaker Experiment Kit (fig. 4)” manufactured by Tri-State Ltd. Table I. Parametric Speaker Characteristics Modulation method FM（frequency modulation） Carrier frequency 40.3kHz±0.1kHz Directivity Approx.20° Power source 12VDC Size Base part 98mm×65mm ultrasonic oscillator 98mm×55mm Playing the sound image. NIME '25 June 24-27, 2025, Canberra, Australia From reports and user feedback, the performance is good in the 1kHz to 4kHz sound range, but low and higher frequency components are attenuated and behave like a bandpass filter. To take advantage of this performance and ensure sound volume, sounds below about 20 Hz and above about 10000 Hz were significantly cut during sound creation. In addition, because of sound leakage, sounds can be perceived slightly outside of the assumed linear sound field. Direct sound includes a tactile element that pierces the ear. It also has the effect of making the sound seem closer, as if the sound field were at one's ear. When reflected on a wall or other surface, reflected sound is generated with the same directivity as direct sound, with the sound image at the reflection position. Fig.3. audible image of the speaker Fig.4. Parametric speaker Performer suit In this performance, the performer wears a total of four parametric speakers on both palms and both shins(fig.5). The super-directional speakers are protected by the legs of electronic components using acrylic plates made with a laser cutter and spacers for the base(fig.4). They are attached to both palms and both shins by wrapping them with rubber straps.This prevents the attachment from weakening when the speaker moves, and also holds the speaker in place on the curved surfaces of the shins. The performer suit is processed based on tights and a long-sleeved top with an easily recognizable silhouette. The suit has a pocket in the back where the control boards for the four parametric speakers, the Bluetooth receiver, the audio splitter, and the dry cell batteries for the power supply are secured so that they do not fall out. In addition, the cables from the speakers are attached to the arms, legs, and back of the clothing with velcro over the cables to keep them in place and in line with the body (fig.5,6). NIME '25 June 24-27, 2025, Canberra, Australia Shibata et al. Fig.5. Performer suit diagram Fig.6. Performer suit Performance System The performance basically proceeds with pre-made choreography and track music(fig.2,7). The performer dances and gradually incorporates improvisational movements in response to the audience's reactions. The sound source is played by the composer on a Laptop and transmitted to the Performer's parametric speakers. The composer plays multiple tracks, manipulating the parameters of the effects, switching the volume of the tracks, and playing fine pulsing sampled sounds as the performer's movements progress and improvise. The tracks are prepared with different patterns of sounds that best fit the performer's movements and the resultant spatial movement of the sounds. These patterns were created during the production phase through repeated prototyping of movements and sounds. Playing the sound image. NIME '25 June 24-27, 2025, Canberra, Australia Fig.7. System diagram Sequence Performance is divided into four major phases. 2.4.1 Phase1 We show that the sound source movement is realized by body movements, and at the same time the body movements are aurified as sound image movement. Simple patterns of sound source movement are defined and sequenced, and then the body movements that realize these patterns are performed. Creation of sound movement. This phase was assembled with geometric sound source movement: straight lines, arcs/circles, free curves, spirals, waves, regular hexagons/polygons, and so on. Fig.8. Sound movement image Sound movement into Choreography. We identified and patterned the various body movements that can realize each sound source movement. In particular, linear sound movement is possible by moving the body by walking or by simultaneously bending and stretching multiple joints aligned in a straight line(fig.9). Movements on a circle are possible by bending and stretching or rotating a single joint. (fig.10). In addition, the simultaneous parallelism of these movements can produce even more complex and geometric sound source movements. For example, walking in a circle on the floor while drawing a circle with the palm of the hand can generate epicycloidal sound source movement. Fig.9. Choreography that produces a linear movement of sound NIME '25 June 24-27, 2025, Canberra, Australia Shibata et al. Fig.10. Choreography that produces a circular movement of sound Sound Design: When linear sound movement takes place, white noise was used because it is considered an ethereal sound that does not produce a change in hearing for those who are not on that straight line. A circle is a figure that represents the circulation and repetition of time and form. Spatial acoustics caused by slow circular motion brings a sense of harmony and equilibrium with space. Therefore, we created an ambient-like sound source by playing a sustained sound of 1000 Hz or higher. For the arc-like sound source movement created by the shin speakers during walking, we used a relatively easy-to-hear sustained sound of around 3000 Hz to emphasize the dynamism of the feet. For complex movements that combine straight lines and circles, an LFO was used with a volume and pitch that had a sense of folds. The composer changed the volume of the track to match each movement. 2.4.2 Phase2 This phase incorporates the interaction between the Performer and the audience. 1. The Performer faces each audience member, arms outstretched in front of him or her, and waves his or her hands in front of him or her. The volume is kept as low as possible to prevent sound leakage to other audience members. By waving their hands quickly toward their ears, the performers create a tactile auditory experience, as if wind were blowing against their ears. Use of white noise without pitch, but the experience is similar to the Doppler effect, in which a sense of pitch is perceived by moving at high speeds. 2. From the center of the space, the sound approaches in a straight line toward one audience member. The composer creates a sense of tension as the performer approaches by playing a series of pulse sounds in response. It brings about an illusionary sensation that the sound image localization does not change even though the position of the parametric speakers is Playing the sound image. NIME '25 June 24-27, 2025, Canberra, Australia getting closer. For the pulse sound, a metal collision sound was used, which is easy to hear the sound image clearly and creates a sense of urgency. 2.4.3 Phase3 Performer incorporates interaction aimed at multiple audiences at the same time. The performer holds the hyper-directional speakers in both palms and shins still, pointing them at the ears of the four separate audiences surrounding him. The performer then changes aim to the other audiences and continues to pose for them in turn. As s/he changes poses, an unintended spatial shift of the sound source occurs, creating a complex curve in space. The composer plays various sampled sounds during the pauses and stillness. 2.4.4 Phase4 The performer combines previous movements and moves freely through the audience. The composer selects a track while watching the Performer, using the pattern from Phase 1. Gradually, the movement returns to circular arc and straight line movements, and the performance ends with a standstill. Performance notes Technical equipment Provided by artist Laptop ×1 MIDI Controller ×1 Bluetooth Receiver ×1~ Parametric speaker ×4~ Equipment requirements Stage lights （speaker ×2~4) table ×1（for composer） Approx. 120W Performance time Approx.12min Space requirements ～200 ㎡（It is preferable to be close to the audience or a stage where the performers can move among the audience.） Set-up time 2h Technical equipment Provided by artist Laptop ×1 MIDI Controller ×1 Bluetooth Receiver ×1~ Parametric speaker ×4~ Space Requirement The experience of three performance staging at Open Research Forum (ORF) held at Keio University's SFC in October 2023 demonstrates the feasibility of this project. The performances were staged in a semi-circular domed space with a diameter of 8.2m (fig.11). For this work, we prefer a narrow space and a space that is horizontal and close to the viewer to allow for interaction, but this is not an absolute requirement. The music, choreography and performance development can be adjusted in advance to suit the venue. NIME '25 June 24-27, 2025, Canberra, Australia Shibata et al. Fig.11. space image Set-up Setup will take at least 2 hours, but can proceed backstage. The Performer will be wearing a suit and hyper-directional speakers. In order to design a more immersive sound depending on the size and shape of the venue, we prefer to rehearse or take a preliminary look in the same space as the performance prior to the day before the show. Technical Requirements Wi-Fi is essential, so we request either wireless or wired internet access. In terms of power, 120W is required, and a single outlet plug is fine. Depending on the actual size and shape of the stage, we will consider adding speakers. Power is expected to vary approximately 150W accordingly. It is preferable to have power for battery charging for parametric speakers in the setup space as well. Every effort will be made to achieve performance. It would be ideal to be able to freely change the stage lighting during the performance. We would like to darken the entire venue so that the performers cannot be seen, allowing the audience to immerse themselves in the sound. We can provide personnel to operate the lighting. Media Link(s) • Video: https://youtu.be/fPONmF3s8nM • Sound: https://soundcloud.com/lisco-325716502/playing-the-sound-image-231126_mix_volupfor-nime2025 Ethical Standards All sounds used in this work were recorded and created by me. The sources of electronic components used in the work are clearly indicated. I respect and strictly abide by copyright rules in all creative and scholarly activities. This work does not involve unethical research on human or animal subjects. References [1] 秋月電子通商,”パラメトリック・スピーカー増設キット:キット一般 秋月電子通商-電子部品・ネット 通販”. Accessed Feb.01,2024.[Online.] Available: https://akizukidenshi.com/catalog/g/g102815/ [2] Curtis Roads, The Computer Music Tutorial, MIT Press, 1996.