Cross-Modal Coupling Between Pitch and the Perception of Space, and its Modulation by Contextual Priors

Cross-modal coupling between pitch and the perception of space, and its modulation by contextual priors. Zofia Anna Hołubowska1,2[0009-0002-1113-4509], Marlen Herkenberg1 and Marc Schönwiesner1,3[0000-0002-2023-1207] 1 Institute for Biology, Faculty of Life Sciences, University of Leipzig, Talstraße 33, 04103 Leipzig, Germany 2 Max-Planck-Institute for Human and Cognitive Brain Sciences, Stephanstraße 1a, 04103 Leipzig, Germany 3 Department of Psychology, University of Montreal, Pavillon Marie-Victorin, 90 avenue Vincent d'Indy, Montréal, Quebec, Canada [email protected] Abstract. Frequency-elevation mapping (FEM) describes how listeners systematically mislocalize sounds in vertical space: high-pitched sounds appear elevated while low-pitched sounds seem lower than their actual positions. This phenomenon shapes how we experience sound in space, yet its underlying mechanisms remain debated, with some suggesting the main influence of statistical learning based on sounds naturally occurring in the environment, whereas others supporting the main linguistic or cultural contribution. We investigated FEM using violin, flute, and spectrally-matched artificial sounds across nine frequencies (200–1600 Hz) and five elevation angles. Twenty-nine participants localized these sounds while we manipulated both spectral centroid and musical context to disentangle local and global context contributions to the effect. Our results show that FEM is primarily driven by fundamental frequency rather than spectral content. While spectral centroid slightly influenced elevation perception at low frequencies, this effect vanished at higher frequencies. Crucially, musical instruments showed significantly weaker FEM effects compared to artificial sounds with identical spectral properties. Local context also mattered: the interval between successive notes influenced spatial perception. These findings show that musical knowledge can outweigh fundamental psychoacoustic expectations. Recognition of a musical instrument as a coherent sound source capable of producing multiple pitches reduces FEM effect. This demonstrates how musical and contextual understanding actively shape our perception of auditory space, transforming abstract frequency relationships into a meaningful spatial experience within musical environment. Keywords: spatial perception, cross-modal perception, frequency-elevation mapping. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 105 2 Hołubowska, Herkenberg and Schönwiesner 1 Introduction When we talk about sounds, cross-modal connections are omnipresent, as we borrow vocabulary from other sensory modalities to describe our experience [1]. For instance, when we say, that one sound was “higher” than another, we typically refer to an increase in fundamental frequency, not to a change in spatial location. Crucially, this mapping is not just a linguistic convention – it reflects a perceptual phenomenon. The frequencyelevation mapping (FEM), also known as the Pratt effect, was first described in 1930 [2], when participants hearing pure tones ranging from 256 to 4096 Hz showed linear relation between the pitch of presented sound and its perceived spatial location, namely sound of higher frequency were perceived higher in space. Roffler and Butler [3] later confirmed Pratt’s effect using complex tones of nine fundamental frequencies (250– 7200 Hz). They found that high-frequency tones were perceived several degrees higher than low-frequency ones, and this bias persisted despite changes in spatial position of participants and the presence or absence of visual cues. Notably, it has been found that children, as young as 4-month-old infants showed the same pitch–height bias [4], arguing against a purely linguistic or learned origin (however see [5] for different conclusions). Recent work suggests that FEM arises from natural scene statistics, rather than linguistical associations. Parise, Knorre, and Ernst [6] analysed real-world sounds and found a robust statistical correlation between pitch and source elevation: high-frequency sounds in nature are more likely to come from above than low-frequency sounds. Moreover, they show that the anatomy of the ear reinforces the mapping, as the filtering properties of the pinnae reflect the higher frequency-higher location mapping. Thus, the explanation of the FEM cannot be derived solely from linguistic or cultural associations through music notation, for instance, but is also reflected in statistics of the natural environment and the anatomy of the auditory system. In our daily experience, we are surrounded by statistical learning on many levels. Although we might be exposed to the general natural scene statistics of higher sounds coming from elevated locations, we also learn, that single sound sources have stable spatial locations. For example, a single violin produces sounds ranging from 200 to 2600 Hz [7]. Thus, we have an additional global prior of a stable sound source of a musical instrument, with changing sound frequency. On a more local level, pitch perception in musical sequences involves not only absolute frequencies but also relative relationships between successive tones. The same frequency could be perceived differently depending on the preceding musical context. This contextual dependency might influence frequency-elevation mapping by these relative pitch relationships rather than operating solely on absolute frequencies. Musical intervals – the pitch distance between consecutive notes – could therefore modulate spatial perception by establishing a directional context that guides elevation judgments. Evidence from audio-visual studies supports this, demonstrating that sequential pitch changes can influence the perceived motion of visual objects [8], suggesting that perception of direction in pitch and space is dynamically linked. While FEM is a well-established effect, its sensitivity to contextual knowledge remains unclear. We hypothesized that both global priors (e.g., knowledge of sound Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 106 Cross-modal coupling between pitch and the perception of space 3 source stability) and local priors (e.g., pitch relations between tones) modulate FEM strength. To test this, we compared spatial localization of real musical instruments – recognized as coherent, pitch-varying sources – with spectrally-matched artificial sounds. This design disentangles the effects of acoustics, sound source identity, and context on frequency-space mapping in music. 2 Methods and Materials 2.1 Participants Twenty-nine healthy participants without self-reported hearing deficits took part in the study (19 females, 9 males, 1 diverse; Mage = 24.55±6.04). They received a compensation for participation in the study (10€ per hour). Participants gave a written consent before the study. Musicality of participants was assessed using the Goldsmith Music Sophistication Index (Gold-MSI; [9]). This study was carried out with the approval of the local ethics committee. 2.2 Stimuli Participants were presented with tones of twelve different fundamental frequencies, each lasting one second. These frequencies were organized into four bins, each centered around one of the following values: 208 Hz, 554 Hz, 1480 Hz, and 3951 Hz. For each central frequency, two additional tones were included – one a semitone below and one a semitone above – resulting in three frequencies per bin. The frequency bins were later used for data aggregation and for testing the main effects of pitch on sound localization performance. Experimental stimuli were created in four conditions: violin, flute, complex tone: harmonic complex tones with fixed harmonic structure (7 harmonics of amplitude -10, -20, -30, -40, -50, -60, and -70 dB), and synthetic violin: generated as harmonic complexes whose component frequencies and relative amplitudes were derived from spectral analyses of recorded violin tones, thereby replicating the instrument’s harmonic profile. The selection of the stimuli allowed us to group stimuli based on two criteria. They can be grouped based on the matched spectral centroid, where violin and synthetic violin have similar spectral centroid, and systematically higher than flute and complex tone (Fig.1A and B). Alternatively, the sounds can be grouped based on musical vs. artificial sound contrast (violin-flute and complex tone-synthetic violin). The spectral centroid for each tone, a weighted average frequency of a spectrum, was computed using slab package in python [10]. Artificial sounds (complex tone and synthetic violin) were created using slab package [10]. Musical sounds were generated in MuseScore (MuseScore Studio 4.4.2) as MIDI files and recorded with the instruments’ library of GarageBand (version 10.4.12). Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 107 4 Hołubowska, Herkenberg and Schönwiesner 2.3 Procedure The experiment was conducted in an anechoic chamber to ensure free-field acoustic conditions. Participants were seated facing a speaker array mounted on a vertical arc with five loudspeakers positioned at azimuth 0° and elevations of +25°, +12.5°, 0°, – 12.5°, and –25°, all placed at a distance of 150 cm from the participant’s head (Fig. 1C). The 0° loudspeaker was aligned with the participant’s ear. The loudspeakers were hidden behind an acoustically transparent curtain to avoid visual cue of the sound location. Stimuli were generated, loaded, and presented through a digital signal processor (RX8.1, Tucker-Davis Technologies). Participants were presented with three condition blocks (violin, flute, and complex tone or synthetic violin) in the randomized order. Each block consisted of the presentation of 180 stimuli (each of the 12 frequencies was presented 3 times from each of 5 possible locations), with a break after 90 stimuli. After each sound, participants were asked to point their head towards the perceived direction of the sound. Head movements were recorded using a sensor-based head-tracking system mounted on top of participant’s head, with a laser pointer attached above the right ear allowing participants to point at perceived source locations. Button press was used to confirm localization responses. Fig. 1. (A) Spectral centroid as a function of the fundamental frequency for the four experimental conditions. (B) Results of a one-way ANOVA confirmed a significant effect of condition on spectral centroid values (F(3, 113) = 8.22, p < .001). Post-hoc pairwise comparisons (Tukey HSD, α = .05) revealed significantly higher centroids in both the violin and synthetic violin conditions compared to flute and complex tone. No significant difference was observed between violin and synthetic violin (p = .709), nor between flute and complex tone (p = .996), showing effective matching of centroid in these condition pairs. (C) A schematic of experimental set-up. Participants were seated in front of a vertical array of five loudspeakers, hidden behind the acoustically transparent curtain. A B C + 25o + 12.5o 0o -25o -12.5o 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 Frequency (kHz) 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 Spectral Centroid (kHz) flute complex tone violin synthetic violin Condition 0 1.0 2.0 3.0 4.0 Spectral Centroid (kHz) ns ns Condition flute complex tone synthetic violin violin Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 108 Cross-modal coupling between pitch and the perception of space 5 2.4 Analysis We analysed localization responses from 29 participants across four conditions (violin, flute, complex tone and synthetic violin) using linear mixed-effects models (statsmodels package in python [11]). The dependent variable in all models was the elevation difference, defined as the angular difference (in degrees) between the presented and perceived sound source location. Data from the highest frequency bin (>3500 Hz) were excluded from the analysis. Because these frequencies exceed the typical range of flute and violin, our synthesized tones in this bin lacked harmonic content and resembled near-pure tones. Such spectrally poor stimuli are challenging to localize in elevation due to the absence of the spectral cues necessary for vertical spatial mapping [3, 12], particularly when compared with spectrally rich tones from lower frequency bins, and from other conditions (e.g., complex tone). To replicate the frequency-elevation mapping effect, we fitted a linear mixedeffects model to model the elevation difference as a function of frequency, with the random effect of participant. We added condition and the interaction between frequency and condition, to test whether the strength or direction of the frequency-elevation mapping varies across conditions. To evaluate whether music-based or centroid-based grouping better explains listeners’ responses, we compared two mixed-effects models using AIC and BIC. Finally, in order to assess the role of musical experience, we compared the slopes of frequency effect on the elevation difference with the results of the Gold-MSI questionnaire, specifically the General Index and Musical Training subscales. 3 Results 3.1 The strength of frequency-elevation mapping differs between conditions To determine whether the mapping between frequency and perceived elevation differed across stimulus conditions, we fitted a linear mixed-effects model with fixed effects of frequency, condition, and their interaction, and with a random effect of participant. Consistent with the previous results, the model revealed a significant main effect of frequency, indicating that higher-pitched sounds were generally perceived as originating from higher elevations (β = 0.008, z = 12.02, p < .001). This effect was further modulated by condition, as demonstrated by significant frequency × condition interactions. The frequency-elevation mapping was stronger in artificially created tones and attenuated in natural musical timbres (flute and violin) (Fig. 2A). Specifically, compared to the complex tone condition (baseline), the effect was significantly reduced in both the flute condition (β = - 0.004, z = -4.42, p < .001) and the violin condition (β = -0.004, z = -4.32, p < .001). This indicates that although an upward bias with increasing frequency was still present in these conditions, it was consistently weaker than in the complex tone condition. Synthetic violin condition did not differ significantly from complex tone (β = -0.000, p = .718). Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 109 6 Hołubowska, Herkenberg and Schönwiesner 3.2 Spectral centroid does not predict perceived elevation To test whether the spectral centroid of a stimulus predicts perceived elevation independently of its fundamental frequency, we extended the model by adding orthogonalized spectral centroid as a fixed effect. This ensured that centroid effects were tested independently of highly correlated fundamental frequency. The results showed that frequency was a strong predictor of perceived elevation (β = 0.010, z = 10.88, p < .001), consistent with prior models. While conditiondependent frequency interactions remained significant, the inclusion of spectral centroid added only a marginal improvement. Specifically, the main effect of spectral centroid was statistically significant but small (β = 0.006, p = .027), suggesting a minimal influence on elevation perception. To further disentangle these contributions, we computed Pearson correlations between spectral centroid and elevation difference separately for each fundamental frequency (Fig. 2B). Only the three lowest frequencies – 196 Hz (r = 0.082, p = .003), 208 Hz (r = 0.115, p < .001), and 220 Hz (r = 0.101, p < .001) – showed weak, but statistically significant positive correlations between spectral centroid and elevation difference. At higher frequencies, the correlation coefficients were near zero and not significant (p > .25). Fig. 2. (A) Differences between perceived and presented sound elevation as a function of fundamental frequency, plotted across conditions. (top) Raw data aggregated across all participants in the sound localization task. (bottom) Fitted linear mixed-effects model showing frequencyelevation trends, with slopes presented per condition. (B) Elevation differences across three frequency bins, showing correlations with spectral centroid. A weak positive relationship is observed only in the lowest frequency bin; no systematic relationship is present at higher frequencies. (C) Elevation difference plotted against the interval between the current and previous tone. Larger upward pitch intervals are associated with higher perceived elevation, suggesting a contribution of local pitch context to the differences between perceived and presented sound location. 200 300 400 500 −40 0 40 Elevation Difference f₀= 208 Hz r= 0.10, p < .000 600 800 1000 1200 Spectral Centroid (Hz) −40 0 40 f₀= 554 Hz r = 0.02, p= .28 1500 2000 2500 −40 0 40 f₀= 1480 Hz r= 0.02, p= .16 −20 −15 −10 −5 0 5 Interval (in semitones) −60 −40 −20 0 20 40 60 Elevation difference A B C 196 208 220 523 554 587 1397 1480 1568 −10.0 −7.5 −5.0 −2.5 0.0 2.5 5.0 7.5 Elevation Difference Condition violin flute complex tone synthetic violin 200 400 600 800 1000 1200 1400 1600 Frequency (Hz) −6 −4 −2 0 2 4 6 8 Elevation Difference Condition violin flute complex tone synthetic violin Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 110 Cross-modal coupling between pitch and the perception of space 7 Finally, to evaluate whether grouping conditions based on musical versus artificial timbres or spectral centroid better accounts for listeners’ elevation perception, we compared the corresponding mixed-effects models using the Akaike Information Criterion (AIC) and the Bayesian Information Criterion (BIC). The model incorporating music-based grouping showed a better overall fit (AIC = 101393; BIC = 101446) than the model based on centroid grouping (AIC = 101400; BIC = 101452). This suggests that perceptual categorization of stimuli as musical or non-musical captures variance in elevation perception more effectively than a purely acoustically defined centroid-based classification. 3.3 Local frequency change modulates elevation perception To investigate whether the interval between successive tones contributes to location perception, we included both absolute frequency and interval size (in semitones) as predictors in a mixed-effects model, along with their interaction with condition. This model tested whether local melodic context – the pitch difference between consecutive tones – influences the frequency–elevation mapping (Fig. 2C). As in previous models, frequency remained a strong predictor of elevation difference. Crucially, interval size also significantly predicted perceived elevation (β = 0.137, p = .014), indicating that tones that followed larger upward intervals were more likely to be localized higher in space, independent of their absolute frequency. This confirms that not only pitch height, but also pitch change, influences spatial perception. 3.4 Musical experience doesn’t drive the frequency-elevation mapping We analyzed the relationship between musicality scores and the frequency–elevation slope across participants to examine whether individual musical experience influences the strength of the frequency–elevation mapping. Both General Musicality (GM) and Musical Training (MT) scores, derived from the Gold-MSI questionnaire, were tested in separate linear mixed-effects models with participant as a random effect. Neither GM (β = –0.000, p = .819) nor MT (β = –0.000, p = .711) significantly predicted the slope of the frequency–elevation relationship, indicating that musical background of participants did not systematically modulate the effect. 4 Discussion In this study we investigated the frequency-elevation modulation, and how it can interact with global and local priors. We also explored whether the effect is driven by the spectral content of the sound, or by its fundamental frequency. Our results show that FEM is a robust phenomenon, occurring consistently across both artificial and musical sounds and among participants with varying levels of musical expertise. Crucially, we found that the perceived elevation of pitched sounds is primarily determined by their fundamental frequency and by higher-level features Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 111 8 Hołubowska, Herkenberg and Schönwiesner such as sound source identity (e.g., recognition as a musical instrument), rather than by spectral centroid alone. Specifically, variations in spectral centroid among sounds sharing the same fundamental frequency did not predict differences in perceived sound elevation. Previous studies have shown that manipulating the spectral content of noise bursts by enhancing or attenuating parts of the frequency spectrum can change how participants perceive sound elevation [13]. In our experiment, the spectral centroid only weakly predicted perceived elevation at the lowest frequencies. For higher frequencies, however, the difference in elevation perception is driven solely by the fundamental frequency. This difference might be due to the less reliable pitch estimation in lower frequencies [14]. Taken together, these findings indicate that, when pitch is clearly defined, differences in harmonic composition (i.e., spectral centroid) contribute little to the FEM, leaving the fundamental frequency as the dominant cue. Importantly, the strength of FEM was not uniform across conditions. It was significantly reduced in the musical context, violin and flute, compared to the acoustically matched artificial sounds, despite different spectral profiles of the two instruments. This shows that the modulation of FEM is not driven by spectral differences between instruments, but rather by listeners’ prior knowledge of a stable sound source. In other words, global priors about the sound source interfere with elevation mapping. A similar pattern emerged with local priors, when relative change between subsequent tones also influenced the perceived sound location. Such interpretation aligns with theories of statistically optimal cue integration, where prior expectations are integrated with current sensory input to increase perceptual precision [15, 16]. Frequency-elevation mapping is a phenomenon explained and shaped both by linguistic and cultural associations [17] as well as natural scene statistics and ear anatomy [6]. While these explanations show that the effect is strong and operates on multiple levels of perception, we argue that FEM can be still modulated by priors. Both longterm (global) and short-term (local) expectations modulate FEM, showing that a basic psychoacoustic judgments are dynamically shaped by context. Our results point to FEM as a higher-level perceptual bias – integrating source recognition, categorization, and learned environmental statistics with elevation cues – rather than a fixed, low-level property of hearing. Acknowledgments. Z.A.H. is supported by the Leipzig Excellence Fund for Young Researchers and the German Academic Scholarship Foundation. Disclosure of Interests. The authors have no competing interests to declare that are relevant to the content of this article. References 1. Hamilton-Fletcher G, Pisanski K, Reby D, et al (2018) The role of visual experience in the emergence of cross-modal correspondences. Cognition 175:114–121. https://doi.org/10.1016/j.cognition.2018.02.023 2. Pratt CC (1930) The spatial character of high and low tones. Journal of Experimental Psychology 13:278–285. https://doi.org/10.1037/h0072651 Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 112 Cross-modal coupling between pitch and the perception of space 9 3. Roffler SK, Butler RA (1968) Factors That Influence the Localization of Sound in the Vertical Plane. The Journal of the Acoustical Society of America 43:1255–1259. https://doi.org/10.1121/1.1910976 4. Dolscheid S, Hunnius S, Casasanto D, Majid A (2014) Prelinguistic Infants Are Sensitive to Space-Pitch Associations Found Across Cultures. Psychol Sci 25:1256–1261. https://doi.org/10.1177/0956797614528521 5. Dolscheid S, Çelik S, Erkan H, et al (2023) Children’s associations between space and pitch are differentially shaped by language. Developmental Science 26:e13341. https://doi.org/10.1111/desc.13341 6. Parise CV, Knorre K, Ernst MO (2014) Natural auditory scene statistics shapes human spatial hearing. Proc Natl Acad Sci USA 111:6104–6108. https://doi.org/10.1073/pnas.1322705111 7. Piston W (1955) Orchestration. W. W. Norton & Company 8. Teramoto W, Hidaka S, Sugita Y (2010) Sounds Move a Static Visual Object. PLoS ONE 5:e12255. https://doi.org/10.1371/journal.pone.0012255 9. Müllensiefen D, Gingras B, Musil J, Stewart L (2014) The Musicality of Non-Musicians: An Index for Assessing Musical Sophistication in the General Population. PLoS ONE 9:e89642. https://doi.org/10.1371/journal.pone.0089642 10. Schönwiesner M, Bialas O (2021) s(ound)lab: An easy to learn Python package for designing and running psychoacoustic experiments. JOSS 6:3284. https://doi.org/10.21105/joss.03284 11. Seabold S, Perktold J (2010) Statsmodels: Econometric and Statistical Modeling with Python. Austin, Texas, pp 92–96 12. Middlebrooks JC (1992) Narrow-band sound localization related to external ear acoustics. The Journal of the Acoustical Society of America 92:2607–2624. https://doi.org/10.1121/1.404400 13. Rajendran VG, Gamper H (2019) Spectral manipulation improves elevation perception with non-individualized head-related transfer functions. J Acoust Soc Am 145:EL222–EL228. https://doi.org/10.1121/1.5093641 14. Cedolin L, Delgutte B (2005) Pitch of Complex Tones: Rate-Place and Interspike Interval Representations in the Auditory Nerve. Journal of Neurophysiology 94:347–362. https://doi.org/10.1152/jn.01114.2004 15. Wolmetz M, Elhilali M (2016) Attentional and Contextual Priors in Sound Perception. PLOS ONE 11:e0149635. https://doi.org/10.1371/journal.pone.0149635 16. Hohwy J (2017) Priors in perception: Top-down modulation, Bayesian perceptual learning rate, and prediction error minimization. Consciousness and Cognition 47:75–85. https://doi.org/10.1016/j.concog.2016.09.004 17. Holler J, Drijvers L, Rafiee A, Majid A (2022) Embodied Space‐pitch Associations are Shaped by Language. Cognitive Science 46:e13083. https://doi.org/10.1111/cogs.13083 Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 113