Image Sonification as Unsupervised Domain Transfer

Image Sonification as Unsupervised Domain Transfer Bálint Laczkó1[0009→0001→8337→2509], Marie E. Rognes2,3[0000→0002→6872→3710], and Alexander Refsum Jensenius1[0000→0001→6171→8743] 1RITMO Centre for Interdisciplinary Studies in Rhythm, Time, and Motion, Department of Musicology, University of Oslo, Norway 2Dept. Numerical Analysis and Scientific Computing, Simula Research Laboratory, Norway; 3K. G. Jebsen Centre for Brain Fluid Research, Oslo, Norway Abstract. The process of image sonification maps visual features into perceived auditory features. Most established sonification methods rely on identifying salient visual features in the input data and then mapping their distribution to a proportional distribution of auditory features. However, this approach requires both domain expertise and manual feature engineering. Here, we propose a new method of image sonification, leveraging recent advances in representation learning and domain transfer. Our approach introduces a pair of variational auto-encoder models that learn disentangled latent representations of the images and sounds, respectively, and a separate network that maps between these representations. The resulting sonification system encodes images into the latent space and then decodes them as sounds. Both representations and their mapping are learned in an entirely unsupervised manner. When evaluating the system in an interactive real-time setting, we observed that the model successfully learned disentangled representations of image and sound factors in our synthetic datasets. Keywords: Image sonification · Domain transfer · Representation learning · Variational Auto-Encoder. 1 Introduction Sonification is the transformation of data relationships into perceived auditory relationships [15]. A key challenge in this process is aligning the auditory display with human auditory perception. Based on the perceptual distance between sonified data points, the listener should be able to judge the relative distances between data points in the feature space (e.g., “A is about twice as far from C as B is from C”). We refer to this as the requirement of proportionality, where distances in the data feature space (given a specific metric, e.g., Euclidean distance) relate monotonically to perceptual distances. Some existing definitions of sonification align well with our requirement of proportionality. Nattkemper et al. describe the requirements of identity, where identical data features must result in identical sonifications, and similarity, where sonifications of a set All rights remain with the authors under the Creative Commons Attribution 4.0 International License (CC BY 4.0). Proc. of the 17th Int. Symposium on Computer Music Multidisciplinary Research, London, United Kingdom, 2025 Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 596 B. Laczkó et al. of data points “must be perceived similar or different, corresponding to their distance in the parameter space” [23, p. 2]. Hermann’s criterion of a sonification reflecting “objective properties and relations” in the source data [14, p. 2] suggests the listener should be able to perceive those data relations based on listening, which requires preserving proportional relationships. Parameter Mapping Sonification (PMSon) is the most common sonification method [7]. This method involves selecting a feature associated with the dataset and mapping it to an input parameter of a synthesizer, which in turn generates the sound stream. A PMSon system can include one or several such mappings between data features and synthesizer parameters. However, more mappings make it harder to satisfy the requirement of proportionality. Successful examples of PMSons, such as metal detectors (mapping magnetic force to synthesizer frequency), or car parking assistance (mapping distance to pulse rate), typically involve only one mapping. One reason for this could be that monitoring multiple parameters that change simultaneously requires more attention and focus. Furthermore, some acoustic parameters are known to influence each other in perception (e.g., amplitude influences perceived pitch) [15]. Depending on the synthesizer architecture, changes to input parameters may not produce proportional changes in the perceptual distances of the synthesized sounds [21]. Image sonification is the transformation of data relationships in image media to perceived auditory relationships. The process highlights an additional challenge: choosing the data features for sonification that best represent the relationships between data points. This can be hard when working with image data where individual pixels lack descriptive power. For example, the approximate age of a person depicted in an image cannot be easily defined as a function of pixel values. Imaging-driven research fields, such as biology and medical diagnostics, therefore often rely on handcrafted features (acquired from software such as CellProfiler [3]). Then, there is no apparent connection between low-level image features (e.g., mean and variance of pixel intensities) and biological phenomena (e.g., the malignancy of a tissue sample). Representation learning, an area of Deep Learning (DL), examines how features can be learned through automatic optimization instead of manual selection, and it has demonstrated rapid advances in learning representations of images, sounds, and other data. In this paper, we propose a new method for image sonification, which builds upon recent advancements in representation learning and domain transfer. We introduce a pair of FactorVAE models [18] that learn disentangled latent representations of images and sounds, and a Mapper network that maps between these representations. This construction is designed to satisfy our requirement of proportionality. The representations and mappings in our system are learned entirely unsupervised. We have tested the system with synthetic image and sound datasets, where we controlled the underlying factors of variation and evaluated the learned mapping in an interactive real-time setting. This controlled prototype demonstrated the system’s viability and has helped establish a proof of concept for future applications. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 597 Image Sonification as Unsupervised Domain Transfer 2 Background 2.1 Image Sonification Image sonification has been applied most prominently in sensory substitution systems, notably as navigation guidance for the blind and visually impaired [5]. It has also been used to explore biomedical image datasets, either to aid medical diagnostics [11], understand cell behavior [24], or for artistic expression and science communication [13]. Besides these two major areas, image sonification has also been applied in education [26], for movement feedback [12], and for reconnaissance [1] and surveillance [17] systems. In these applications, image sonification typically frees up or substitutes for vision in other tasks, reducing visual overload in data display or overcoming visualization limitations. 2.2 Deep Learning in Image Sonification The use of DL in image sonification has mainly focused on image processing tasks, such as classification [30], object detection [25], segmentation [31], and generation [6], while applying established mapping and sound generation techniques (such as parameter mapping and additive synthesis, respectively). However, in recent years, studies have emerged using DL to form links between image and sound features. Milazzo et al. collected a dataset of candle flame images, where each captured image showed the flame excited by a sine wave emitted by a speaker directed at the candle [22]. An image classifier Convolutional Neural Networks (CNN) was then trained using the pitches of the excitation signals as labels, so that a video of candle flame could be sonified at inference time using the learned mapping. Fink et al. utilized a scene recognition network to retrieve relevant ambient sounds from a large sound dataset previously labeled with scenes derived from video frames in an unsupervised manner [9]. Gambardella et al. sonified video feeds using the learned latent representations of each video frame as spectra to be decoded to sounds via the inverse Fast Fourier Transform (FFT) [10]. The studies mentioned above demonstrate various methods for creating mappings between image and sound data using DL. However, they are all based on supervised mappings. This means there is a prior decision about “the correct” mapping, i.e., the particular images and sounds that should be paired. To our knowledge, our proposed image sonification system is novel in that it learns the mapping between images and sounds in an unsupervised manner, focusing solely on preserving the distribution of the image data in the domain of perceived sound. 2.3 Learning Disentangled Representations Representation learning is a topic within DL that focuses on discovering the best features that describe the distribution within a given dataset. Samples from the input dataset are mapped to learned latent vectors either according to supervised [30] or unsupervised labels. A high-quality representation is crucial for solving downstream tasks, such as classification or generation. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 598 B. Laczkó et al. An Auto-Encoder (AE) is a type of neural network based on the concept of pairing an encoder that performs dimensionality reduction (data compression) and a decoder that reconstructs (decompresses) the data from the encoded latent representation. However, since the encoder has no restriction on how it organizes its latent space (i.e., the distribution of projected data points), AEs often suffer from poorly organized latent spaces, where semantically similar inputs are not necessarily close together, rendering them ineffective for tasks like clustering and feature learning. In a Variational Auto-Encoder (VAE), each point in the latent representation (the state after encoding and before decoding) is modeling a variational distribution instead of a single deterministic point [19]. During optimization, the model regularizes its latent space organization to enable smooth transitions between similar data points. VAE-s trained with an optimal balance between the reconstruction and regularization objectives are better suited for learning global and local relationships within the data than regular AE-s. VAE-s can also learn “disentangled” representations, where each latent dimension captures one (and preferably only one) aspect in the data. A notable VAE variant that excels in disentangled representation learning is the betaVAE algorithm [16]. BetaVAE-s achieve disentangled representations by assigning a stronger weight (beta) to the regularization objective during training. While yielding a more disentangled representation, this strategy often comes at the expense of reconstruction quality. Another VAE design for disentangled representation learning is FactorVAE by Kim et al. [18], where an adversarial discriminator network forces the VAE’s encoder to produce uncorrelated latent dimensions. They demonstrated that FactorVAE-s can produce more disentangled latent representations and more accurate reconstructions than betaVAE-s. 2.4 Unsupervised Cross-Modal Domain Transfer Recently, there has been an increasing focus on so-called multi-modal representation learning and modality alignment within DL. The term “multi-modality” is challenging when working between technology (data types) and psychology (human perception) [4]. Most research on supervised multi-modal representations focuses on coupling data types and how they are observed in ground truth data. Theodoritis et al. implemented a system of supervised domain transfer, where, in one experiment, food images were mapped to the respective lists of their components. Another experiment focused on aligning hand-pose images with corresponding motion capture data [28]. In their approach, the desired mapping between data types is known in advance. Similar priors cannot always be present in sonification, where multiple mappings could exist that satisfy the requirement of proportionality, and any appropriate mapping must be found, typically through iterative design. Tian et al. demonstrate how a VAE can learn to coordinate visual and auditory representations in an unsupervised way, allowing domain transfer between different media [29]. Their design also enforces semantic alignment, ensuring that labels remain consistent (e.g., written digits of zeros will be matched with recordings of spoken zeros). The benefit of their proposed architecture is that it can build upon pre-trained latent variable models and their “bridging VAE” can transfer in both directions (from images to sounds and vice versa). Similarly, the AudioViewer model from Song et al. creates Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 599 Image Sonification as Unsupervised Domain Transfer an unsupervised VAE-based one-way mapping between audio and video, effectively visualizing the fluctuations in an audio stream as proportional fluctuations between image frames in a video [27]. Since their model only aims for a one-way domain transfer, the mapping network can be significantly less complex and completely label-free. 3 Methods: A Domain Transfer Approach Here, we address the question of how to integrate representation learning and unsupervised domain transfer into a novel image sonification paradigm. Our approach relies on three components: we use the FactorVAE architecture [18] to learn a pair of latent representations of an image and a sound dataset respectively and then use a Multi-Layer Perceptron (MLP) [20] to learn a mapping between the latent spaces of the image and sound VAE-s (Figure 1). In this section, we first introduce the main algorithm, then present a synthetic dataset for model testing, and finally, an evaluation of the approach. 3.1 Architectures and algorithm We base the FactorVAE models on CNN-s, where the latent space between the Encoder and the Decoder is chosen to have 2 dimensions. This is our way of informing the model about the number of factors of variation it should examine within a dataset. We train the models using the reconstruction loss (i.e., the error between the input data sample and its reconstructed counterpart), measured by Mean Absolute Error (MAE), and the regularization loss (i.e., how different the latent distribution is from an ideal Gaussian normal distribution), measured by Kullback-Leibler Divergence (KLD). To guide the model towards learning disentangled representations, we also include the FactorVAE’s additional loss term, which measures the correlation between latent dimensions. This is estimated by a separate discriminator network that is discarded after training. To obtain better-quality reconstructions from the Decoder, we employ warm-up annealing for the regularization factor, which is dynamically increased as long as the reconstruction loss remains under a certain threshold. This way, we can aim for a specific reconstruction “quality level” and automatically find the strongest possible regularization that the model can sustain with that target quality. After training the image FactorVAE and sound FactorVAE models, they are frozen (i.e., their weights are kept unchanged) for the rest of the optimization process, in which the MLP Mapper is trained using latent samples of images (encoded by the image FactorVAE) as input. The Mapper’s objective is twofold. First, it must project the latent representations of images into the latent space of the sound FactorVAE while preserving the intersample distance relationships of the samples. This proportionality must be enforced independently in each latent dimension, rather than combined. This is because if the FactorVAE-s successfully disentangled the underlying factors, each latent dimension in their representations will correspond to one (and only one) factor. Per-axis proportionality prevents the Mapper from mixing latent dimensions, preserving the factor-todimension correspondence that both FactorVAEs learned. The Mapper can “squeeze” or “stretch” its projections along a dimension to fit the target latent space. This proportionality loss term alone—measured via MAE—would not be enough since it would Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 600 B. Laczkó et al. Fig. 1: The proposed model architecture. The blue group denotes the image model, the green group the sound model, and the orange block represents the Mapper. Red arrows indicate loss criteria. The semi-transparent red path indicates how images are mapped to sound at test time. only facilitate the Mapper to reproduce its input. The second objective of the Mapper is to project samples within the bounds of the target latent space’s distribution. It is important that the resulting projections align with the distribution of encoded sound samples in the sound latent space, as samples from outside this distribution would yield spurious reconstructions. We use the idea of a cycle consistency loss from AudioViewer [27]. The cycle consistency loss measures how well two mappings—such as A-to-B and B-to-A—work as inverses of each other. Ideally, input A mapped to B and then back to A should be very similar to its original form. The cycle consistency loss measures the discrepancy between the round-trip result and the input. This loss ensures the Mapper stays within the learned data distribution of the target network, preventing unrealistic outputs. This loss term can conflict with the proportionality loss, and during optimization, a trade-off is established based on the relative weights of the two terms. After training the Mapper, the resulting system can be used to map a group of images to a group of sounds by using the Encoder of the image FactorVAE to produce latent representations. The Mapper projects these representations to the latent space of the sound FactorVAE, where its Decoder transforms the latent points to sounds. This pipeline is highlighted in Figure 1 as the semi-transparent red path. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 601 Image Sonification as Unsupervised Domain Transfer (a) Samples from the image dataset. (b) Samples from the sound dataset. Fig. 2: Samples of the image and sound datasets. (a) Four random samples from the image dataset (128 →128 px). The horizontal and vertical coordinates of the 2 →2 px white square are the only varying factors in the dataset. (b) 64 random samples from the sound dataset. Sine waves are represented as time-averaged mel-spectrograms of 64 bands, where each row corresponds to an individual band. Pitch (based on MIDI) and amplitude are the only variables that change in this dataset. 3.2 Synthetic image and sound datasets Since neural networks are black-box architectures, it can be difficult to evaluate whether their learned representations are meaningful if applied to a complex experimental dataset. Here, we instead first aim to determine whether our system can successfully identify known exact factors of variation by generating synthetic datasets of images and sounds. We generate image and sound datasets with 2 factors of variation each (Figure 2). In the image domain, we generate images of a white square over a black background, with the horizontal and vertical positions as the varying factors. Initial experimentation revealed difficulties in model learning due to the lack of signal in the images (apart from 4 active pixels). Therefore, we also included an exponential decay in intensity centered around the square’s position (Figure 2a). This diffusive effect did not introduce another factor of variation in the image, and significantly sped up model convergence. In the sound domain, we presented sine waves with varying pitch and amplitude to the sound FactorVAE model as time-averaged Mel-spectrograms (Figure 2b). The images were 128 →128 px and the size of the white square was kept at 2 →2 px. This combination of image and square sizes yielded a total of 15,876 possible XY coordinates for the square. For the sound dataset, combinations of MIDI pitch and amplitude values were sampled from a uniform random distribution at every step in the training loop. The MIDI pitch values were converted to frequencies, and sine waves were generated at a sample rate of 48,000 Hz, with a duration of 8,192 samples, or 170.67 ms. These were then converted to mel-spectrograms with 64 bands representing frequencies from 60 Hz to 1200 Hz (a range corresponding to our pre-defined MIDI pitch range). Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 602 B. Laczkó et al. The spectrograms were averaged over the time dimension, and the magnitudes of the bands were converted to dB values and then scaled into a 0–1 range. The sounds of sine waves were presented to the model as 1-dimensional, 64-number-long vectors. 3.3 Evaluation method The objective of our proposed system is to simulate a PMSon, where, in our proofof-concept dataset, either the horizontal position is mapped to pitch and the vertical position to loudness, or vice versa. We first evaluated whether each model successfully learned a disentangled representation of all factors of variation, based solely on the pixels or mel bands, respectively, and the prior knowledge that there were 2 factors of variation in each dataset. Disentanglement was measured qualitatively by plotting the latent distribution produced by the models and color coding the latent points using the parameters that generated the corresponding input sample. Additionally, we decoded linearly interpolated latent points, where one dimension was kept fixed and the other was varied, to determine whether these interpolations could generate images and spectrograms in which only one factor could be changed (e.g., the vertical position of the square in the images or the pitch of the sine wave in the spectrograms). The second evaluation stage focused on the mapping between the image and sound domains. To qualitatively test this, we set up an interactive app through Cycling ’74 Max, where the cursor position on the XY pad generated image inputs to the network. The predicted spectrogram was then matched to the closest from our sound dataset, whose pitch and loudness parameters were synthesized in real time. Using this interface, we could not only inspect the proportionality of the mapping but also its consistency over extended use. 4 Results Both FactorVAE models converged successfully. Disentanglement was tested by encoding both datasets with their respective models and plotting the latent spaces, where the parameters used to generate the input data (i.e., white square X and Y position, and sine wave MIDI pitch and amplitude) were used to color code the encoded latent points. Figure 3 shows the latent space learned by the image model, color coded by the white square X and Y position parameters. From only observing pixels the model clearly identified the two varying factors and associated each with a single latent dimension. A similarly successful disentanglement can be observed in Figure 4, where the two varying parameters of the sine waves (MIDI pitch and amplitude) are modeled on separate latent dimensions in the sound model’s latent space. Additionally, both latent spaces have a rectangular distribution that matches the underlying data structure despite the VAE’s Gaussian prior. The mapping was evaluated in the interactive setting described above by moving the XY pad cursor. 5 Discussion The mapping showed a clear overall tendency but was not deterministic, with slight variations in pitch and loudness when revisiting the same position, due to the variaProc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 603 Image Sonification as Unsupervised Domain Transfer (a) Image latent space colored by X. (b) Image latent space colored by Y. Fig. 3: The latent space of the encoded images colored by the white square’s X position (a) and Y position (b) parameters used for generating the input images. (a) The first latent dimension encodes Y position (increasing from right to left), and (b) the second latent dimension encodes X position (increasing from top to bottom). (a) Sound latent space colored by MIDI Pitch. (b) Sound latent space colored by Amplitude. Fig. 4: The latent space of the encoded sounds colored by the MIDI Pitch (a) and Amplitude (b) parameters used for generating the input sine waves. (a) The first latent dimension encodes Amplitude (increasing from left to right), and (b) the second latent dimension encodes MIDI Pitch (increasing from top to bottom). Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 604