Cross-modal Analysis of Spatial-Temporal Auditory Stimuli and Human Micromotion when Standing Still in Indoor Environments

Cross-Modal Analysis of Spatial-Temporal Auditory Stimuli and Human Micromotion when Standing Still in Indoor Environments Jinyue Guo1[0009000374381483],JimTorresen 2,3[0000000305560288],and Alexander Refsum Jensenius3[0000000161718743] 1RITMO Centre for Interdisciplinary Studies in Rhythm, Time, and Motion 2Department of Musicology, University of Oslo, Norway 3Department of Informatics, University of Oslo, Norway {jinyueg,jimtoer,alexanje}@uio.no Abstract. This paper examines how a soundscape influences human stillness. We are particularly interested in how spatial and temporal features of a soundscape influence human micromotion and swaying patterns. The analysis is based on 345 Ambisonics audio recordings of different indoor environments and corresponding accelerometer data captured at the chest of a person standing still for ten minutes. We calculated the temporal and spatial correlation between the person’s quantity of motion and the sound energy of the Ambisonic recordings. While no clear temporal correlations were found, we discovered a correlation between the spatial directionality of the micromotion and the sound direction of arrival. The results suggest a potential entrainment between the directionality of environmental sounds and human swaying patterns, which have not been thoroughly studied previously compared to the temporal or spectral features of indoor soundscapes. Keywords: Environmental rhythms ·Human micromotion ·Spatial audio ·Cross-modal analysis. 1Introduction Even when humans try to stand still, we always move slightly [7]. This is based on postural sway driven by continuous muscular activity in the legs combined with respiratory phases and involuntary postural adjustments [3]. We use the term micromotion to describe the smallest motion that humans can produce and experience, typically at a millimeter scale [6]. Together with meso-level (centimeter-scale) and macro-level (meter-scale) motion, micromotion has been studied as a phenomenon of embodied music cognition, playing an essential role in the studies of how humans perceive and interact with musical rhythms [11,2]. All rights remain with the authors under the Creative Commons Attribution 4.0 International License (CC BY 4.0). Proc. of the 17th Int. Symposium on Computer Music Multidisciplinary Research, London, United Kingdom, 2025 Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 871 J. Guo et al. Fig. 1. Screenshots from the “fisheye” view from some of the 360-degree videos in the StillStanding dataset that is used for the analysis of relationships between soundscapes and human stillness. The act of standing still reveals a rich tapestry of involuntary micromotion, which can be measured and analyzed to uncover patterns of embodied perception and environmental responsiveness, and that can be explained as part of a continuous action–perception loop between a human and the environment [4]. Past research has focused on how musical sounds affect humans when they are standing still. However, there are few studies on the effects of non-musical sounds, including everyday sounds in real-world environments. This study examines the impact of soundscapes on body motion during intentional stillness in indoor environments, each with distinct sonic characteristics. While previous studies have mainly focused on the temporal and spectral features of indoor soundscapes, we have analysed the spatial (directional) information of soundscapes based on Ambisonics recordings. This is correlated with accelerometer data from a person standing still in each room. The goal is to gain adeeperunderstandingofaparticulartypeofsound–motion relationship and how sound contributes to our sense of presence, balance, and spatial awareness. Section 2 introduces some relevant concepts before describing the collection of our StillStanding dataset in Section 3. Then, we explore the dataset in Section 4 using cross-modal correlation analysis between the accelerometer data and the Ambisonics audio recordings. To avoid confusion in terminologies, we use the terms “sound”, and “motion” when describing the physical stimuli,“auditory” and “visual” to explain perception of such stimuli, and “audio”, “video”, and “accelerometer data” when describing recorded data. 2Background 2.1 Human Micromotion Previous studies have shown that micromotion patterns, such as respiration, pulse, and postural adjustments, can be affected by music of different genres, Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 872 Spatial–Temporal Sound–Motion Analysis of Still-Standing tempi, rhythmic patterns, and spectral characteristics [3]. Furthermore, lab experiments have revealed that differences in music-related micromotion behavior are related to age, exercise level, social interactions [8], and level of empathy [21]. There have also been explorations into using micromotion mechanisms for artistic purposes [7,5,12]. 2.2 Spatial-Temporal Stimuli Architectural elements in indoor environments, such as light, color, natural views, sound, smell, and specific geometries, have been shown to have psychological and physiological effects [17]. Blinking indicator lights, ventilation hums, ticking clocks, and intermittent appliance sounds are examples of spatial-temporal stimuli. These stimuli are inherently dynamic, varying across both spatial locations and temporal patterns. Such stimuli are often processed subconsciously, yet they contribute significantly to spatial orientation and temporal awareness, particularly in complex environments [18]. Recent research has explored how such human-observable signals can be systematically captured and modeled using sensor networks and machine learning techniques, enabling applications in ambient intelligence [20] and integrating physical and cognitive ergonomics [13]. 2.3 Environmental Rhythms Our interest in spatial-temporal stimuli is from the observation that they may exhibit rhythmic structures, which we refer to as environmental rhythms. This is inspired by Lefebvre’s rhythmanalysis,positingthatrhythms—definedasrepetitive patterns in time and space—shape human experience [10]. Lefebvre proposes that everyday life is structured by natural (e.g., bodily cycles, day and night), social (e.g., work schedules, traffic), and technical (e.g., media, machines) rhythms. People attend to both cyclical rhythms (repeating, like seasons) and linear rhythms (progressive, like a workday). Lefebvre emphasizes that space is not static but produced and experienced through these temporal patterns. 2.4 Embodied Music Cognition Lefebvre’s reflections on how time, space, and the body interact in everyday life resonate well with recent music cognition theories, which emphasize that music (and its rhythms) is inherently multimodal [11]. Rhythmic structures can arise from the integration of auditory and visual stimuli, such as synchronized sonic pulses and light flashes. More intricate rhythmic experiences emerge when auditory features—such as loudness, pitch, and timbre—correlate with visual attributes like color, brightness, and spatial positioning [19]. This expanded notion of rhythm underscores the intricate interplay between environmental stimuli and perceptual processes, suggesting that rhythm is not merely a musical construct but a broader phenomenon embedded in everyday sensory experiences. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 873 J. Guo et al. Fig. 2. The recording setup used for data collection. The GoPro Max and the Zoom H3-VR were attached with a fixed relative direction on a tripod raised to chest level. 2.5 Spatial Audio and Ambisonics Spatial audio refers to a set of audio technologies that enable immersive sonic experiences, closely mimicking how we perceive sounds in the real world [16]. Traditional speaker setups focus on either single-source-based listening (mono) or various types of “stereo” setups (often with two speakers) that create a “twodimensional” sound field. Spatial audio, on the other hand, allows listeners to perceive three-dimensional depth and height, as well as the position and motion of sound sources. This enhances immersion in applications such as film, virtual reality (VR), augmented reality (AR), and immersive media. Ambisonics is a full-sphere surround sound format that captures and renders sound in all directions around a listener [23]. While it is arguably still a niche protocol, commercial Ambisonics recorders now make it convenient to perform field recordings. Unlike channel-based systems (e.g., 5.1 or 7.1) that are designed for fixed speaker setups, Ambisonics is scene-based, allowing it to be decoded into arbitrary numbers and setups of speakers. This versatility makes it useful not only for audio playback but also for analytical purposes. 3Thesound–motionSubsetofStillStandingDataset To investigate the impact of soundscapes on human micromotion, we utilized asubsetoftheStillStanding dataset, comprising 365 recordings of 10-minute standstill sessions, collected by the last author. The dataset comprises immersive audio and video recordings (Figure 1), sensor data from a smartphone, physiological data from a sports watch, and corresponding text descriptions. 3.1 Data Collection Each 10-minute recording was taken during a daily standstill session, performed around noon, throughout 2023, in a different room each time. Recordings began Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 874 Spatial–Temporal Sound–Motion Analysis of Still-Standing Fig. 3. A frame from the ‘.lrv’ video showing the “fisheye view” from the two lenses in the 360-degree camera. The mobile phone is hung around the person’s neck. and ended with a clap for synchronization and to measure the room impulse response. After each session, the participant described the room and documented their subjective experience. Recordings of the environment were made with a 360-degree video camera (GoPro Max) and an Ambisonics audio recorder (Zoom H3-VR) mounted in a fixed setup on a tripod (Figure 2). The standalone audio recorder was added to ensure high-quality audio recordings in a standard format. While GoPro claims that the Max camera can record “spatial audio,” its spatial audio format is nonstandard, rendering it unusable for analytical purposes [15]. Environmental, motion, and physiological data were captured using an Android smartphone. Attached to a string around the neck, this phone captured swaying and micromotion from the upper body (Figure 3). The microphone and light sensors on the phone can also be used for environmental analysis. Sensor data from the phone was captured using the Physics Toolbox Sensor Suite 1 (ver. 2023.01.21) by Vieyra Software. The data collection focused on indoor spaces, while some recordings were made in semi-indoor environments, such as a bus shelter with only two walls and a ruin with walls but no roof. To avoid recording other people, most sessions were made in private rooms (e.g. cabins, bedrooms, or storage rooms) or public rooms without other people present (e.g. conference rooms, offices, or libraries). 3.2 Data Processing Each recording session lasted for approximately 10 minutes and was trimmed to exactly 8 minutes to remove synchronization claps at the beginning and end, as well as postural adjustments that were irrelevant to the analysis. For the current Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 875 J. Guo et al. study, we rely on the first-order Ambisonics (FOA) audio recordings from the Zoom H3VR and accelerometer data from the mobile phone. The Ambisonics audio files were synchronized with the other file types by manually identifying the timestamp of the beginning clap in each audio recording. This could then be aligned with the peak in the loudness measurement from the .CSV files recorded on the smartphone. Due to some erroneous recordings, we used a subset of 345 out of the 365 recordings in the analysis. 4SoundandMotionAnalyses 4.1 Temporal Sound–Motion Correlation Previous experiments suggest a strong temporal correlation between human micromotion and rhythmic and musical sound stimuli. We were therefore curious to investigate whether a correlation exists in irregular and non-musical sound stimuli of indoor environments. Fig. 4. Temporal correlation analysis of all samples. There is no clear linear correlation between the temporal changes of QoM and audio loudness. We calculated the quantity of motion (QoM) as the root mean square (RMS) of the three accelerometer axes captured by the mobile phone sensors. The sound intensity level was recorded with the Physics Toolbox Sensor Suite on the same mobile phone. Then, the temporal sound–motion correlation was obtained by calculating the Pearson Correlation Coefficient (PCC) between the QoM sequence and the loudness sequence. Figure 4 shows the temporal sound–motion correlation of all StillStanding samples. Most recordings are very close to the r-value of zero, and only 36 out of 345 recordings have p<0.05,regardlessof Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 876 Spatial–Temporal Sound–Motion Analysis of Still-Standing the average loudness (x-axis) or mean QoM of the session (color). The results suggest that there is no temporal correlation between the audio loudness and the QoM in the StillStanding sessions. The change in correlation suggests that the prevalent and structural music stimuli, i.e. rhythmic components, are necessary for the entrainment between auditory stimuli and micromotion. 4.2 Directional Audio Energy The Ambisonics recordings allow us to investigate spatial correlations between body motion and environmental sounds. To that end, we first need to calculate the directional audio energy Eamb[]. We do that in the following steps: 1. Setup a horizontal virtual speaker array =[180,175,··· ,0,··· ,175] with elevation of 0. 2. Decode the Ambisonics signal to the speaker array with a mode-matching decoder: x[, n]=D[n], where x[, n]are decoded virtual speaker signals, [n]is the B-format, first-order Ambisonics signal, and Dis the decoder matrix where each loudspeaker is represented by the spherical harmonics spectrum of a Dirac delta, expressed as Legendre Polynomials [22,14]. 3. The decoded virtual speaker signals are then windowed into a small frame, and the RMS energy of each is calculated: E[, m]=qPN n=0 x[, m +n]2, where E[, m]is the RMS energy at angle and time frame m,andNis the window size. 4. Finally, we calculate the overall audio energy Eamb[]by summing the maximum directional energy of each time frame: Eamb[]=Pm2ME[|= argmax(E[, m]),m]. Figure 5 shows examples of the calculated directional audio energy maps of the StillStanding dataset. The energy difference between the FOA recordings in different directions is low, since most recordings in the StillStanding dataset primarily contain quiet background sounds between 40 and 70 dB SPL. Therefore, the sum of energy in each direction across all frames does not vary significantly. However, if we only add the energy of the maximum direction at each frame, as in step 4, the resulting array clearly shows the differences. The proposed method can effectively extract the overall characteristics of different indoor soundscapes, revealing distinct directionality patterns. The scale of energy also varies significantly among the samples, as SS#301 has the highest energy at around 0.7, while SS#276 reaches a value over 8. 4.3 Directional Acceleration Energy We calculate the directional acceleration energy Eacc[]from the accelerometer data captured by the smartphone in the following steps: 1. Remove the average value of each acceleration axis to clear the offset. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 877 J. Guo et al. Fig. 5. A collage of directional audio energy of 15 Ambisonics recordings from the StillStanding dataset. The examples are picked at an equal interval of days. 2. Convert the transversal acceleration points from cartesian coordinates to polar coordinates: ✓[n]=arctan(az[n],a y[n]),r =paz[n]2+ay[n]2, where ✓is the angular coordinate, rthe radial coordinate, nthe discrete timestamp, az,a ythe transversal plane accelerations (since the phone was hung horizontally). 3. For each angular bin of , sum the energy of all points within the range of 5: Eacc()=P✓[n]2[2.5,+2.5)r[n],2[180,175,··· ,0,··· ,175]. Figure 6 shows examples of accelerometer data maps (green dots) and the calculated directional acceleration energy (blue bars). Usually, people tend to sway back and forth (anterior–posterior) due to the stabilization effects of the feet [9]. Thus, deviations from such an anterior–posterior swaying pattern may be due to environmental factors. As expected, Figure 6 shows that the scale of acceleration energy remains the same in most recordings, while a few, such as SS#031, have more motion. 4.4 Spatial Sound–Motion Correlation After obtaining Eacc[]and Eamb[]for the 345 recordings included in the analysis, the Pearson correlation between the two arrays is calculated. Figure 7 shows aplotofthecalculateddirectionalaccelerationenergy(left)anddirectionalaudio energy (right) of one plot (SS#046). The spatial correlation between motion directionality and audio directionality is considerably higher than the temporal Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 878 Spatial–Temporal Sound–Motion Analysis of Still-Standing Fig. 6. Acollageofdirectionalaccelerationenergyof15sessionsofaccelerometerdata from the StillStanding dataset. The examples are picked at an equal interval of days. Fig. 7. Spatial correlation analysis of SS#046, a session in a concert hall. Left:Plot of acceleration (green) and summed directional energy (blue bins). Right:Plotof directional audio energy calculated from the Ambisonics audio recordings. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 879