Extracting Sonic Trajectories

Master in Sound and Music Computing Universitat Pompeu Fabra Extracting Sonic Trajectories Tito Scutari Supervisor: Sergi Jorda Co-Supervisor: Behzad Haki 2025 Master in Sound and Music Computing Universitat Pompeu Fabra Extracting Sonic Trajectories Tito Scutari Supervisor: Sergi Jorda Co-Supervisor: Behzad Haki 2025 Contents 1 Introduction 1 1.1 Overview .................................. 1 1.2 Motivation.................................. 2 1.2.1 Extractinggroove.............................. 2 1.2.2 Universal neural encoders are everywhere . . . . . . . . . . . . . . . . . 2 1.3 Objectives.................................. 2 1.3.1 Tracking sonic trajectories . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.3.2 Interpretability of representations . . . . . . . . . . . . . . . . . . . . . 3 1.3.3 Test bench for representations . . . . . . . . . . . . . . . . . . . . . . . 3 1.4 Structure of the Report . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.4.1 Stateoftheart ............................... 4 1.4.2 Exploratory experiments . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.4.3 FirstStudy ................................. 4 1.4.4 SecondStudy ................................ 5 1.4.5 Conclusion.................................. 5 2 State of the art 6 2.1 Groove.................................... 6 2.2 Onsetdetection............................... 7 2.2.1 Traditional approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.2.2 Deep Learning approaches . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.3 Trajectories ................................. 9 2.4 Representations............................... 9 2.4.1 Traditional representations . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.4.2 Latentspaces ................................ 10 3 Exploratory experiments 13 3.1 Introduction................................. 13 3.2 Methodology ................................ 13 3.2.1 Synth .................................... 14 3.2.2 Testset ................................... 14 3.2.3 Heatmaps .................................. 15 3.2.4 Magnitude.................................. 16 3.2.5 Distance................................... 16 3.2.6 Cosinesimilarity .............................. 16 3.3 Results.................................... 16 3.3.1 Amplitude.................................. 16 3.3.2 Pitch..................................... 16 3.3.3 Noise..................................... 17 3.3.4 Frequencycontent ............................. 18 3.3.5 Delayandreverb .............................. 20 4 Study 1: Single Modulations 22 4.1 Introduction................................. 22 4.2 Setup .................................... 22 4.2.1 Dataset ................................... 23 4.2.2 Representations............................... 25 4.2.3 Metrics and measurements . . . . . . . . . . . . . . . . . . . . . . . . . 25 4.2.4 Procedure.................................. 27 4.3 Results.................................... 28 4.3.1 Exampleplot ................................ 28 4.3.2 Summarytable ............................... 31 4.4 Discussion.................................. 31 5 Study 2: Double Modulations 33 5.1 Introduction................................. 33 5.2 Setup .................................... 33 5.2.1 Dataset ................................... 34 5.2.2 Representations............................... 36 5.2.3 Metrics and measurements . . . . . . . . . . . . . . . . . . . . . . . . . 36 5.2.4 Procedure.................................. 38 5.3 Results.................................... 38 5.3.1 Exampleboxplots.............................. 39 5.3.2 Tables .................................... 40 5.4 Discussion.................................. 42 6 Conclusions and discussion 45 6.1 Overview .................................. 45 6.2 Representations............................... 45 6.3 Metrics ................................... 46 6.4 First Study: Single Modulations . . . . . . . . . . . . . . . . . . . . . . 46 6.5 Second Study: Double Modulations . . . . . . . . . . . . . . . . . . . . 47 6.6 FutureDirections.............................. 47 List of Figures 48 List of Tables 50 Bibliography 51 A Linear Analysis of Modulation Representation 53 A.1 Setup .................................... 54 A.1.1 Dataset ................................... 54 A.1.2 Methodology ................................ 54 A.1.3 Stage 1: Per-Sample Model Fitting . . . . . . . . . . . . . . . . . . . . 55 A.1.4 Stage 2: Generalization with an Average Model . . . . . . . . . . . . . 55 A.2 Results.................................... 55 A.2.1 Per-Sample Model Performance . . . . . . . . . . . . . . . . . . . . . . 55 A.2.2 Generalization Model Performance . . . . . . . . . . . . . . . . . . . . 56 A.3 Discussion.................................. 57 Acknowledgement I would like to express my sincere gratitude to: Sheila, Navid and Justin for an unforgettable year Behzad, Błażej, Lonce and Martin and the UPF faculty for all the inspirations and insights Milo, Doma, Fernando, Satya, Vivek and all the classmates for the best nerdy vibes the memory of Giorgio, for all the support in the past 10 years the memory of MoRenato, for teaching me the love for music and all the friends and family back in Italy, for making me what I am 4Chapter 1. Introduction •First study: systematic evaluation of single modulations. •Second study: systematic evaluation of double modulations. •Conclusion: summary of findings, limitations, and possible future directions. 1.4.1 State of the art This chapter reviews the main concepts and previous work relevant to this project. It starts with the idea of groove in music, both from a theoretical and perceptual perspective. It then covers traditional and deep learning approaches to onset detection, discusses the notion of sonic trajectories for sound analysis, and surveys various audio representations, both classic signal-processing methods and modern neural latent spaces, that can be used for trajectory extraction. 1.4.2 Exploratory experiments This section describes initial experiments using synthesized audio signals with known modulations. The goal is to see if different representations can capture these modulations in a meaningful way. These experiments help motivate the choice of representations and metrics, and offer an early look at the strengths and weaknesses of each approach before developing the main algorithm. 1.4.3 First Study This chapter presents a systematic evaluation of single modulations. A dataset of synthetic sounds was generated with one parameter modulated at a time; amplitude, frequency, filter cutoff, or oscillator shape. For each case, trajectories were extracted and compared with ground-truth signals through correlation measures. The goal is to establish a baseline, showing the sensitivity and stability of different representations under simple and isolated conditions. 1.4. Structure of the Report 5 1.4.4 Second Study This chapter extends the evaluation to double modulations. Two parameters vary simultaneously, creating interactions between trajectories. The analysis focuses on how metrics respond when modulations overlap or interfere, and whether one can still be reliably tracked in the presence of another. The aim is to test robustness and generalization of representations under more complex and realistic scenarios. 1.4.5 Conclusion The final chapter summarizes the main findings of the project, discussing what worked, what didn’t, and why. It reflects on the usability of different representations, the usefulness of the developed metrics and procedures, and the overall potential of this approach for audio analysis and creative applications. It also suggests directions for further research, including improvements to the testing framework and application to real-world audio. Chapter 2 State of the art This chapter illustrates the current state of the art of the various topics pertinent to the work, it is divided into 2 groups of sections, the first group reviews the concept of groove from a more theoretical point of view, covering the psychoacoustical and cognitive aspects (Section 2.1) and proceeds to go over past and current technical approaches for onset detection (Section 2.2), which is the most relevant MIR topic related to groove. The second group is an overview of the idea of sonic trajectories (Section 2.3), which is one of the scopes used for music analysis, especially for the more experimental genres, where most of the structures are built on timbral aspects of sound and proceeds to present various more techical representations and algorithms (Section 2.4) used to analyze sound and music especially from a timbral perspective that can be useful for a trajectory-based interpretation of sound. 2.1 Groove In the musicology discourse, groove is an important concept, yet elusive and multifaceted, depending on genres, eras, and contexts. It is closely related to the idea of rhythm, musical gesture, dance, immersion, and is deeply correlated with motor areas of the brain, as explained by Etani [1]. Defining groove is not a simple task, several definitions have been proposed, an interesting general one that should be taken into account was proposed by Duman et al. in 2024 [2]: “Groove is a par6 2.2. Onset detection 7 ticipatory experience (related to immersion, movement, positive affect, and social connection) resulting from subtle interaction of specific music- (such as timeand pitch-related features), performance-, and/or individual-related factors.”. It is thus clear that groove is an incredibly wide term, for our purposes it had been narrowed down to computable rhythmic audio features, in the literature [3] mainly related to percussive elements and more generally events that provide a perceptual quantization of time. In the field of MIR those events are mainly treated with various onset detection techniques. 2.2 Onset detection Onset detection refers to the process of locating the beginning of a musical note or sound, often associated with the transient phase where the signal exhibits rapid changes. It is essential for applications such as automatic music transcription, beat tracking, and synchronization in music production. The task is particularly challenging in polyphonic music, where multiple instruments play simultaneously, leading to overlapping signals. 2.2.1 Traditional approaches Traditional digital signal processing methods for onset detection rely on analyzing various signal properties to detect abrupt changes indicative of onsets [4]. These can be categorized into several sub-approaches depending on the audio representation used: Time domain A common technique is energy-based detection, where the signal’s energy is calculated over short windows, and onsets are identified when energy exceeds a predefined threshold [4]. For example, detecting sudden increases in amplitude is straightforward but can lead to false positives in noisy or amplitude-modulated signals. 8Chapter 2. State of the art Frequency domain These involve the transformation of the signal into the frequency domain using techniques such as the fast Fourier transform (FFT). Metrics such as spectral flux [4], which measures the rate of change in spectral energy between consecutive frames, are widely used. Other metrics include spectral centroid, tracking the center of mass of the spectrum, which can highlight frequency shifts at onsets. These methods are more robust than time-domain approaches but require more computational resources due to FFT processing. Superflux is in this category of onset detection algorhithms and is the most widely used, as it is a standard go to for Librosa and Essentia, the most widespread MIR toolkits. Other advanced methods of this kind take into account also phase changes of the components, detecting onsets when simultaneous changes in phase occur. 2.2.2 Deep Learning approaches Deep learning methods, particularly Convolutional Neural Networks (CNNs), have gained prominence in onset detection by leveraging neural networks to learn patterns directly from data, often represented as spectrograms or other transforms (wavelet, CQT). They learn spatial patterns that correspond to onsets, such as sudden changes in frequency content. A notable early example is the work by Schlüter and Böck [5], which showed CNNs outperforming traditional methods on a dataset with 26,000 annotated onsets. RNNs, in particular LSTMs are also able to outperform traditional methods, as shown by Marchi et al. [6]. Nowadays most advanced models use a combination of the two, as shown by one of the most advanced models for polyphonic strings onset detection developed in 2023 [7]. It is important to note that although those models outperform traditional methods the costs to develop them are very high, both from a computational point of view and data sourcing wise. Running them is also expensive, GPU acceleration might be needed and not all of them can run in real time on a common laptop. In the vast panorama of onset detection models Dance Dance Convolution is the one used as of today by the system, it is based on a convolution architecture trained on Dance Dance Revolution 2.3. Trajectories 9 annotations and shows outstanding real-time performance for transient-like onset detection. This is ideal for detecting percussive sounds but falls short when onsets are softer or happen in the pitch or timbre domain. 2.3 Trajectories Sound gestures and trajectories in sound art and electroacoustic music refer to how sounds move and evolve, capturing both performer actions and spatial dynamics. Smalley’s spectromorphology [8], introduced in 1986, describes the temporal shaping of sound spectra, providing a tool to analyze these aspects. While smalley describes them from an analytical point of view, composers have been using similar concept’s representing them with graphic notation. This notation uses visual symbols to represent music, offering flexibility beyond traditional notation. Xenakis’s UPIC system translates drawings into sound, while Cardew’s "Treatise" (1967) leaves interpretation open, both influencing sound gestures (Xenakis UPIC, Cardew Treatise). MIR techniques, model and generate music by computational representations that collide with the ones used by musicologists and composers. Though direct research is sparse, the integration of those representations could lead to interesting results. 2.4 Representations Sound representations are methods used to transform raw audio signals into structured formats that emphasize specific characteristics, making it easier to analyze, process, or interpret the audio data. They are essential in fields like audio processing, speech recognition, music analysis, and machine learning for audio-related tasks. Each representation highlights different aspects of sound—such as amplitude, frequency, or abstract features—depending on the intended application. 2.4.1 Traditional representations Traditional representations focus on decomposing audio signals into interpretable components, often based on frequency or perceptual scales. These methods are 10 Chapter 2. State of the art well-established and widely used in signal processing. Fourier transform The Fourier Transform decomposes a signal into its frequency components, and for audio, the Short-Time Fourier Transform (STFT) is commonly used to capture how these frequencies evolve over time [4]. It divides the audio into short segments and computes the Fourier Transform for each, resulting in a time-frequency representation. Constant Q Transform The Constant Q Transform is a time-frequency representation where each frequency bin has a constant Q factor, meaning the bandwidth is proportional to the frequency. This results in a logarithmic frequency scale, aligning with human perception of pitch. Unlike STFT, which uses linear frequency spacing, CQT uses logarithmic spacing, with lower frequencies having narrower bandwidths and higher frequencies having wider bandwidths [9]. This is particularly effective for musical signals. Nonnegative Matrix Factorization NMF is a matrix decomposition technique that factors a non-negative matrix (e.g., a spectrogram) into two non-negative matrices: a dictionary matrix (basis spectra) and an activation matrix (how these bases are activated over time). This can be interpreted as representing the audio as a combination of basis spectra. This representation is widely used in the MIR field especially for source separation. 2.4.2 Latent spaces Latent space representations leverage machine learning, particularly neural networks, to create abstract, high-level representations of data [10]. These are often used for tasks requiring similarity measurements, semantic understanding, generation, or compression, and they represent a shift toward data-driven approaches. 2.4. Representations 11 CLAP and semantics Using contrastive learning, it is possible to learn mappings between audio clips and their textual descriptions into a shared embedding space where related concepts, like the sound of rain and the phrase "rain falling", are close together, and unrelated ones are far apart. This cross-modal approach bridges the gap between sound and meaning, enabling tasks like searching for audio with text or understanding audio content semantically, thus building a very semantically rich and meaningful latent space, the downside is that long samples are needed, and these embeddings cannot be calculated frame in real-time, content is therefore very meaningful and rich, but temporal resolution in today’s models is very poor. CLAP [11] is the most widespread model that has been trained in this way and its architecture enables cross-modal tasks like typing "dog barking" to find a sound clip or feeding an audio file to get a text description. Its contrastive learning approach also allows zero-shot classification: you can classify audio into categories never seen during training by using text labels (e.g., "happy music" vs. "sad music"). By focusing on high-level semantics, CLAP makes audio understandable in human terms, which is a crucial for applications like content retrieval or audio annotation. RAVE and live use Some models focus on real-time audio synthesis and manipulation via latent spaces. they usually leverage a variational autoencoder (VAE) type of neural network that learns a compressed, probabilistic representation of data to capture the essence of audio in a way that’s both efficient and flexible. The idea here is to encode audio into a compact "latent space" and then decode it back into sound, enabling not just reconstruction but also the creation of new audio. This makes them useful for generative applications, such as synthesizing new sounds or morphing existing ones, all while keeping latency low enough for live use. A well known example is RAVE [12] that uses an architecture that shines for real-time performance and is optimized to process audio fast enough for live applications like music production or interactive sound design. Its generative nature means that the latent space can 12 Chapter 2. State of the art be sampled to create entirely new sounds or tweak the latent variables to transfer styles (e.g., making a drum sound like a synth). SoundStream and compression Another use of latent spaces is for building neural audio codecs built for efficient compression and high-quality reconstruction. The goal is to shrink audio into a compact form for storage or streaming while keeping it sounding great when played back. The result is a very fast and complete latent space but it is optimized for compression, not for understandability and semantics. A successful example is SoundStream [13], which is trained end-to-end with neural networks, combining compression, quantization, and perceptual quality optimization into one seamless system. It’s designed to work in real time and adapt to different compression needs. SoundStream’s architecture supports variable bitrates and its real-time efficiency means it can encode and decode on the fly, perfect for streaming. Chapter 3 Exploratory experiments 3.1 Introduction The idea of this set of experiments is to examine different representations with synthesized signals in which some modulation occurs and visualize if there is some correlation between the representation’s vectors and the modulation. The representations considered are STFT spectrum, MFCCs, CQT for the traditional side, DAC and Music2Latent latent spaces for the machine learning side. Those representations have been chosen firstly based on widespread adoption, novelty and performance. In particular STFT is the most used transform for MIR related processing, MFCC is also widely used and the data compression it performs packs the information in very small vectors. CQT is chosen because of its psychoacoustical and musical relevance. The two neural encoders are very different DAC is made for data compression, works on raw audio with very high time resolution, music2latent is instead trained on complex spectrograms. Those two were chosen as examples of good performing, recent but also very different neural representations. 3.2 Methodology The methodology involves visually inspecting the representations to determine if there is an apparent correlation between the modulation signal and movements in 13 20 Chapter 3. Exploratory experiments sented with respect to distortion and filtering but since it is still a frequency-related phenomenon it seems quite trackable by frequency domain representations. Figure 8: Impact of fm amount on CQT magnitude Figure 9: Impact of fm amount on music2latent similarity Figure 10: cqt vector magnitudes 3.3.5 Delay and reverb Delay and reverb are maybe the least interesting aspect, since there are already many ways to remove reverb, also in real-time and the difference between the energy between reverb and direct signal clearly correlates with the amount of reverb. Nevertheless, it is interesting to see how those kinds of processing appears in the latent space and other representations. 3.3. Results 21 Figure 11: music2latent magnitudes Figure 12: DAC magnitudes Chapter 4 Study 1: Single Modulations 4.1 Introduction This first study focuses on very simple cases, in order to give an idea of what the various representations can achieve, if a metric scores low on those tests, it is unlikely that it will be useful on more complex tasks. The idea is straightforward: using basic synthesized sound in which one single parameter is modulated with a smooth curve, different representations(series of vectors), are then calculated and those metrics(time series) extracted and correlation between the metric and the modulation applied. If the correlation is high (close to 1.0the metric is able to track the modulation). Jupyter notebook available at: sonic-trajectories/blob/main/study-1.ipynb 4.2 Setup This section describes the setup of the experiment, focusing on the dataset of synthesized sounds and then listing representations and metrics used. All the processing applied is explained in detail in order to ensure replicability. 22 4.2. Setup 23 4.2.1 Dataset The dataset is fully synthesized with minisynth, a tiny library developed for the occasion. The library is available at: pypi.org/project/minisynth/ MiniSynthSubtractive The synthesizer used is a single oscillator subtractive (non-resonant lowpass 12db/oct) synthesizer with 4 parameters: •Base frequency (exponential mapping [50hz −1000hz]): this is the base frequency of the oscillator •Amplitude (exponential mapping [0.0−1.0]): the amplitude of the signal •Filter cutoff(exponential mapping [200hz −5000hz]): cutoff frequency of the lowpass filter •Wave mix (linear mapping [0.0−1.0]): the mix between square and sawtooth waves This architecture has been chosen because even in it’s very minimal design it is able to produce a good variety of sonic trajectories, directly associated with the parameters: •Absolute pitch: the base frequency knob can control the absolute vertical placement in the spectrum, in a perceptual relevant range and mapping. •Amplitude: this directly controls the amplitude of the singla in a perceptually relevant mapping. •Absolute spectrum shape: the filter cutoff controls the absolute shape of the spectrum, mainly in the part that contains harmonics. 24 Chapter 4. Study 1: Single Modulations •Relative spectrum shape: the wave mix is able to change the balance between odd and even harmonics changing the relative shape of the spectrum with respect to the fundamental. Additionally the synthesizer is implemented so that any parameter can be modulated at sample rate speed. The sounds The dataset consists of 1620 2slong sounds, a little less than 1hin total. For every sound one single parameter is positively modulated with a single cycle of a cosine wave at 5 different level of modulation amount: (0.1,0.2,0.3,0.4,0.5) of their respective ranges. For example the amplitude of a sound with an amplitude base level of 0.25 with a 0.2modulation amount will start at 0.45 decrese to 0.25 and rise back to 0.45 in parameter range terms, in actual amplitude those values would be exponentially mapped (0.25 = 0.1,0.45 = 0.23). The base values of the non-modulated parameters can be one of the three between 0.25,0.5,0.75 of the parameter range and the modulated parameter base can be one of the three between 0.0,0.25,0.50 to account for the additive modulation. The complete dataset consists of all the possible combinations of the aformentioned settings: 3 levels for each one of the four parameters, 4 possible modulation targets, 5 possible modulation amounts: 3×3×3×3×4×5 = 1620 In this way, even if just on a minimal set of points per parameter, it is easy to understand the interactions and the dependencies between various parameters. It is to be mentioned, though, that some representations (DAC in our case) work on raw audio and representations have a periodicity based on the ratio between the fundamental frequency in the sound and the window size and so they behave differently at different frequencies and the results of the experiment could vary due 4.2. Setup 25 to this phenomenon. Since those type of representations are quite widespread for compression tasks, further investigation on these issues could have interesting and potentially useful results. The direct consequence of the dataset that contains all the possible combinations of a small set of possibilities is that the dataset is actually not stored anywhere and can be generated directly, possibly with just an iterator and it is thus just a tiny algorithm with ranges and mappings. That said in the actual study intermediate steps have been saved for validation and debugging purposes. 4.2.2 Representations The representations used are: •FFT spectrogram: w= 4096, h = 512 hann window •MFCCs: 20 coefficients w= 2048, h = 512 hann window •CQT: in db h= 512 •DAC: provided compressed latents (72 dimensions) are used •music2latent: 4 encoders shifted by 1024 to achieve 4x oversampling 4.2.3 Metrics and measurements On every representation the following metrics are calculated: Magnitude Frobenius norm of vectors as a time series: Mt=∥xt∥2=v u u t n X i=1 (xt,i)2 26 Chapter 4. Study 1: Single Modulations Cosine similarity Cosine similarity between subsequent vectors: St=at·bt ∥at∥2∥bt∥2+ε=Pn i=1 at,i bt,i qPn i=1 a2 t,i qPn i=1 b2 t,i +ε Distances Distances between subsequent vectors: Dt=∥xt+1 −xt∥2=v u u t n X i=1 (xt+1,i −xt,i)2 Processing Every metric is smoothed with a flat window 0.25s, which ensures the permanence of the sub-audio rate changes and removes most of the noise, normalized in the range [0,1] and stretched to fit the modulator length (l= 1000) 1. Smoothing with a flat window of length w(corresponding to 0.25 s): ˜m[t] = 1 w (w−1)/2 X k=−(w−1)/2 m[t+k] 2. Normalization to the range [0,1]: ˆm[t] = ˜m[t]−min( ˜m) max( ˜m)−min( ˜m) 3. Stretching to fit modulator length l= 1000: mresampled[n] = ˆm"n l·N#, n = 0,1, . . . , l −1 4.2. Setup 27 Correlation Correlation, in the form of pearson correlation coefficient, is the best indicator of the accuracy of the extraction. Perfect correlation (ρ= 1) means that the output of the algorithm is linearly related to the modulator. It is defined as: ρx,y =cov(x, y) σxσy Polarity can be flipped, depending on the representation and on the specific feature, since we are interesting in matching mainly the movement it makes sense to consider: |corr(a(x), m)| . 4.2.4 Procedure The procedure for the experiment is the following: 1. Generate all the possible combinations of parameters (stored in a pandas dataframe). 2. Synthesize the sounds from the parameters and calculate each representation, for each representatio a h5 file is created and the indexed representations are stored. This is done because the encoding step, especially for neural representations, is more computationally intensive. 3. For each entry in each representation (5×1620) the three metrics are calculated, smoothed and correlation with the cosine is computed. The result of this process is a dataframe with 15 correlation columns, one for every metricrepresentation pair. 4. boxplots are generated, filtering the dataframe based on what parameter is modulated and grouped by the other variables. 28 Chapter 4. Study 1: Single Modulations 4.3 Results In order to understand the results several boxplots have been produced, the full set can be examined in the notebook(sonic-trajectories/blob/main/study-1.ipynb). In every boxplot only the data with a single representation, metric and modulation target is considered, for example a boxplot could countain the correlations of the magnitude of mfcc for amplitude modulation. In each boxplot correlation sits on the y axis, on the x axis the variable of which we are testing the influence is placed. In this way we can have a clear idea of how sensitive the representation is, for example we can understand how much modulation is needed to be sensed, and if the sensitivity changes across frequencies, amplitudes, etc. 4.3.1 Example plot The following are a set of plots that are what is used in this study to understand which metric is the best to track filter cutoff modulations and how other variables influence the tracking. Every plot is now described in detail to understand how to read those and the ones contained in the appendix. In alle the five figures correlation for filter cutoff is plotted on the y axis, this mean that we are considering only samples in which filter cutoff modulation is happening, no other modulation is applied on this subset of samples. Those samples, as those with other targets of modulations, represent a fourth of the 405, and all the combinations of variables as previously explained while having filter cutoff modulation. Figure 13: music2latent filter cutoff modulation against modulation amount 4.3. Results 29 The plot in figure 13 has the modulation amount on the x axis and we can see the impact on the tracking, it is clear that the amount of modulation influences the tracking done via magnitude, but the one done with cosine similarity is not really affected. Of all the metrics, and this will be confirmed by other plots, the distances seems the best to track filter cutoff modulation with music2latent representations. Figure 14: music2latent filter cutoff modulation against amplitude In the second plot(Figure 14) on the x axis the amplitude is plotted, the levels correspond to 0.25,0.50,0.75 of the range, exponentially mapped. Magnitude is highly sensitive to amplitude, the average correlation rises from 0.5to 0.8, the other metrics are not particularly influenced by the variation in amplitude, and are thus more stable. Similarly to the previous plot, distances is the best perfoming metric. Figure 15: music2latent filter cutoff modulation against frequency For the third plot(Figure 15) on the x axis the base frequency is plotted, the levels correspond to 0.25,0.50,0.75 of the range, exponentially mapped. This is the more chaotic plot of the five, for magnitude correlation is at its highest at low frequences, 36 Chapter 5. Study 2: Double Modulations 5.2.2 Representations The representations used are: •FFT spectrogram: w= 4096, h = 512 hann window •MFCCs: 20 coefficients w= 2048, h = 512 hann window •CQT: in db h= 512 •DAC: provided compressed latents (72 dimensions) are used •music2latent: 4 encoders shifted by 1024 to achieve 4x oversampling 5.2.3 Metrics and measurements On every representation the following metrics are calculated: Magnitude Frobenius norm of vectors as a time series: Mt=∥xt∥2=v u u t n X i=1 (xt,i)2 Cosine similarity Cosine similarity between subsequent vectors: St=at·bt ∥at∥2∥bt∥2+ε=Pn i=1 at,i bt,i qPn i=1 a2 t,i qPn i=1 b2 t,i +ε 5.2. Setup 37 Distances Distances between subsequent vectors: Dt=∥xt+1 −xt∥2=v u u t n X i=1 (xt+1,i −xt,i)2 Processing Every metric is smoothed with a flat 0.25swindow, this ensures that we’re keeping the sub-audio rate changes and remove most of the noise, normalized in the range [0,1] and stretched to fit the modulator length (l= 1000) 1. Smoothing with a flat window of length w(corresponding to 0.25 s): ˜m[t] = 1 w (w−1)/2 X k=−(w−1)/2 m[t+k] 2. Normalization to the range [0,1]: ˆm[t] = ˜m[t]−min( ˜m) max( ˜m)−min( ˜m) 3. Stretching to fit modulator length l= 1000: mresampled[n] = ˆm"n l·N#, n = 0,1, . . . , l −1 Correlation Correlation, in the form of a Pearson correlation coefficient, is the best indicator of the accuracy of the extraction. Perfect correlation (ρ= 1) means that the output of the algorithm is linearly related to the modulator. It is defined as: ρx,y =cov(x, y) σxσy 38 Chapter 5. Study 2: Double Modulations Polarity can be flipped, depending on the representation and on the specific feature, since we are interested in matching mainly the movement, it makes sense to consider: |corr(a(x), m)| . 5.2.4 Procedure The procedure for the experiment is the following: 1. Generate all the possible combinations of parameters (stored in a pandas dataframe) 2. Synthesize the sounds from the parameters and calculate each representation, for each representatio a h5 file is created and the indexed representations are stored. This is done because the encoding step, especially for neural representations, is more computationally intensive. 3. For each entry in each representation (5×4374) the three metrics are calculated, smoothed and correlation with the cosine is computed. The result of this process is a dataframe with 60 correlation columns, one for every parametermetric-representation group. 4. boxplots are generated, filtering the dataframe based on what parameter is modulated and grouped by modulation balance between first and second modulation and modulation offset, i.e., the phase of modulation 2. For this tables are also necessary to evaluate the influence of other modulations on the one that it is being tracked, for this tasks boxplots would be too many and difficult to navigate so only the average is used. 5.3 Results In order to understand the results boxplots have been produced, each boxplot is built with the data points of a single representation, a single metric and a single 5.3. Results 39 modulation target, it is important to remember that two parameters are being modulated in each sample, and the boxplots consider a data point valid if one of the two is the one selected. Thus the boxplots are not useful to understand how modulations interact based on the combiantion of parameters, for this purpose tables are presented in the next section. Boxplots are used to understand how the balance between the two modulations and the phase of the second modulation influence the tracking of the first. The boxplots have correlation on the y axis and either balance or offset on the x axis. All the plots are examinable in the notebook(sonictrajectories/blob/main/study-2.ipynb). 5.3.1 Example boxplots The following examples are two the boxplots of filter cutoff tracking in music2latent, the entries for used are the ones that contain filter cutoff modulation, which are half of the dataset (2187), which is the case for any specific parameter modulation. These plots don’t take into account what is the other modulated parameter, they are all aggregated. Each plot is explained in detail in the following paragraphs. Figure 20: music2latent filter cutoff modulation against second modulation offset In Figure 20 the x axis is the offset of the second modulation, that means that we always consider filter cutoff modulation having phase 0 and the modulation of the other parameter having phase 0,π 2, π. It can be seen that the correlation does’nt vary a lot between the different metrics, to chose the best one the results of study 1 should be considered. It is clear that when the second modulation is in phase or shifted by π(inverted polarity) the tracking has performs better, getting close to 40 Chapter 5. Study 2: Double Modulations the results of study 1. When second modulation is shifted by π 2the performance degrades drastically meaning that if the modulations are not synchronous the disturbance is very high and filter cutoff becomes very hard to track with those metric in music2latent. Figure 21: music2latent filter cutoff modulation against modulation amount In Figure 21 the x axis is the balance between the filter cutoff modulation and the modulation on the other parameter, aggregated. As for the first plot in order to choose the best metric the results of study 1 should be considered. In this case the plot shows that the difference at the various degrees of balance is not really relevant and the cutoff tracking is not really influenced by the belance between the modulation amounts. 5.3.2 Tables In this section, we present tables for each parameter and their combinations. These tables highlight the most important results of the experiment, as they show how different parameters interact and interfere with each other’s tracking. Each table includes only half of the samples, those containing one chosen parameter modulation. In the tables, each column shows the average correlation of the primary modulation when a secondary modulation (listed in that column) is also present. For the frequency modulation tracking results (Table 2), the table shows that CQT is the best performing representation, which is aligned with the results of study 1. The best metrics are cosine and distances, and the most disturbing modulation 5.3. Results 41 freq amp freq cutoff freq shape all freq spectrum magnitude 0.63 0.57 0.59 0.59 spectrum distances 0.64 0.60 0.61 0.61 spectrum similarity 0.63 0.63 0.55 0.60 dac magnitude 0.51 0.52 0.49 0.51 dac distances 0.51 0.45 0.42 0.46 dac similarity 0.54 0.49 0.43 0.49 music2latent magnitude 0.55 0.56 0.55 0.56 music2latent distances 0.63 0.57 0.57 0.59 music2latent similarity 0.61 0.58 0.59 0.59 cqt magnitude 0.64 0.57 0.53 0.58 cqt distances 0.69 0.65 0.63 0.65 cqt similarity 0.69 0.64 0.61 0.65 mfcc magnitude 0.63 0.61 0.52 0.59 mfcc distances 0.60 0.60 0.60 0.60 mfcc similarity 0.62 0.60 0.59 0.61 Table 2: Frequency averages is oscillator waveshape. Modulating the waveshape means change the harmonic content, thus the content in the higher dimensions of the representation varies a lot, and this explains the disturb. The amplitude modulation table (Table 3) shows that CQT magnitude is the best metric to track amplitude, this is coherent with study 1, the most influential secondary modulation is frequency, and this happens because of how the energy is distributed is spread across the bins and how the norm is calculated. Amplitude is overall the easiest modulation to track, and it is quite understandable that calculating total energy per frame, which is what the norm is doing works well for this no matter what the secondary modulation happens to be. As for study 1, MFCC magnitude is the best at tracking frequency cutoff, this is not surprising since MFCC describe the overall shape of the spectrum which is what we are changing when modulating the filter’s cutoff frequency. The most impactful secondary modulation is amplitude, which is not surprising since it greatly impacts the first coefficients, which are often the ones that contain the information realted to a soft filter like the one we are using. 42 Chapter 5. Study 2: Double Modulations freq amp amp cutoff amp shape all amp spectrum magnitude 0.76 0.67 0.72 0.72 spectrum distances 0.68 0.66 0.70 0.68 spectrum similarity 0.72 0.78 0.80 0.77 dac magnitude 0.49 0.46 0.50 0.48 dac distances 0.58 0.73 0.73 0.68 dac similarity 0.59 0.76 0.71 0.69 music2latent magnitude 0.56 0.62 0.66 0.61 music2latent distances 0.75 0.83 0.83 0.80 music2latent similarity 0.75 0.85 0.83 0.81 cqt magnitude 0.83 0.89 0.89 0.87 cqt distances 0.59 0.79 0.75 0.71 cqt similarity 0.65 0.76 0.74 0.71 mfcc magnitude 0.85 0.80 0.86 0.84 mfcc distances 0.45 0.54 0.53 0.51 mfcc similarity 0.45 0.62 0.59 0.55 Table 3: Amplitude averages Shape modulation (Table 5) is the one that shows the biggest difference with study one, CQT distances, which was the best metric in study one, performs significantly worse when other other modulations are in place and the best performing metric is now music2latent distances, significantly more resistant. The most disturbing modulation is frequency, which makes sense since what is to be tracked in music2latent is the relative structure of spectral peaks and frequency moves them around. Summary table Table 6 summarizes the results of study 2, the absolute best metric n average is MFCC magnitude but that’s mostly due to its performance at tracking filter cutoff. Better all rounders are CQT magnitude and music2latent distances. 5.4 Discussion This study shows a drastic decrease in performance at tracking specific modulations with those metrics compared to study one. The interaction between the various modulations interferes a lot with the tracking of most single parameters, with the main exceptions being amplitude modulation across most of representations and 5.4. Discussion 43 freq cutoff amp cutoff cutoff shape all cutoff spectrum magnitude 0.67 0.58 0.69 0.64 spectrum distances 0.61 0.54 0.67 0.61 spectrum similarity 0.71 0.65 0.78 0.71 dac magnitude 0.45 0.38 0.55 0.46 dac distances 0.47 0.58 0.63 0.56 dac similarity 0.49 0.65 0.59 0.57 music2latent magnitude 0.53 0.65 0.59 0.59 music2latent distances 0.56 0.73 0.72 0.67 music2latent similarity 0.59 0.71 0.69 0.66 cqt magnitude 0.64 0.68 0.81 0.71 cqt distances 0.51 0.58 0.69 0.59 cqt similarity 0.53 0.54 0.70 0.59 mfcc magnitude 0.82 0.74 0.85 0.80 mfcc distances 0.43 0.39 0.59 0.47 mfcc similarity 0.43 0.44 0.59 0.49 Table 4: Filter cutoff averages MFCCs for tracking filter cutoff. Traditional representations are still a very valid way to track modulations, especially CQT, but this study reveals that metrics are quite harder to pick in a more realistic scenario. Where distances were performing as a good all rounder now both magnitude and similarity are needed, which has the upside that different metrics correlate to different things but also that there’s no single metric that conveys the overall groove. Neural representations, music2latent in particular, are much more resistant and the performance is not impacted as much as for traditional representation. This could be a very interesting feature of those type of representations, but testing on a more varied dataset is needed to confirm its usefulness. 44 Chapter 5. Study 2: Double Modulations freq shape amp shape cutoff shape all shape spectrum magnitude 0.52 0.52 0.49 0.51 spectrum distances 0.49 0.53 0.50 0.51 spectrum similarity 0.52 0.62 0.61 0.59 dac magnitude 0.46 0.43 0.48 0.45 dac distances 0.37 0.57 0.55 0.50 dac similarity 0.39 0.60 0.57 0.52 music2latent magnitude 0.56 0.59 0.60 0.58 music2latent distances 0.59 0.69 0.68 0.66 music2latent similarity 0.54 0.68 0.64 0.62 cqt magnitude 0.56 0.65 0.63 0.61 cqt distances 0.51 0.59 0.56 0.55 cqt similarity 0.50 0.53 0.51 0.51 mfcc magnitude 0.57 0.65 0.64 0.62 mfcc distances 0.42 0.41 0.45 0.43 mfcc similarity 0.41 0.44 0.44 0.43 Table 5: Wave shape averages frequency amp cutoff shape overall cqt distances 0.65 0.71 0.59 0.55 0.63 cqt magnitude 0.58 0.87 0.71 0.61 0.69 cqt similarity 0.65 0.71 0.59 0.51 0.62 dac distances 0.46 0.68 0.56 0.50 0.55 dac magnitude 0.51 0.48 0.46 0.45 0.48 dac similarity 0.49 0.69 0.57 0.52 0.57 mfcc distances 0.60 0.51 0.47 0.43 0.50 mfcc magnitude 0.59 0.84 0.80 0.62 0.71 mfcc similarity 0.61 0.55 0.49 0.43 0.52 music2latent distances 0.59 0.80 0.67 0.66 0.68 music2latent magnitude 0.56 0.61 0.59 0.58 0.58 music2latent similarity 0.59 0.81 0.66 0.62 0.67 spectrum distances 0.61 0.68 0.61 0.51 0.60 spectrum magnitude 0.59 0.72 0.64 0.51 0.62 spectrum similarity 0.60 0.77 0.71 0.59 0.67 Table 6: Study 2 overall summary, best are in bold Chapter 6 Conclusions and discussion 6.1 Overview This work explored sonic trajectories as a way to represent continuous changes in sound, moving beyond onset-based analysis to capture variations in amplitude, pitch, timbre, and spectral shape. A trajectory extraction framework was proposed, combining traditional signal-processing representations and neural latent spaces with simple but robust metrics. Exploratory experiments established the feasibility of the approach, and two systematic studies evaluated the capacity of different representations and metrics to follow both isolated and interacting modulations. Taken together, the results highlight both the promise of trajectory-based methods for music information retrieval and sound analysis, and the challenges that remain when dealing with overlapping or multidimensional changes. 6.2 Representations The results across both studies show that representation choice has a decisive influence on trajectory extraction. Traditional representations remain highly competitive: Constant-Q Transform proved reliable in tracking both frequencyand amplitude-related changes, while MFCC magnitudes excelled at following filter cutoff modulations. These methods, despite their age, continue to provide robust base45 52 BIBLIOGRAPHY of the 18th International Audio Mostly Conference, 136–142 (ACM, Edinburgh United Kingdom, 2023). URL https://dl.acm.org/doi/10.1145/3616195. 3616206. [8] SMALLEY, D. Spectromorphology: explaining sound-shapes. Organised Sound 2, 107–126 (1997). [9] Schörkhuber, C. Constant-q transform toolbox for music processing (2010). URL https://api.semanticscholar.org/CorpusID:12358579. [10] Mikolov, T., Chen, K., Corrado, G. & Dean, J. Efficient estimation of word representations in vector space (2013). URL https://arxiv.org/abs/1301. 3781.1301.3781. [11] Elizalde, B., Deshmukh, S., Ismail, M. A. & Wang, H. Clap: Learning audio concepts from natural language supervision (2022). URL https://arxiv.org/ abs/2206.04769.2206.04769. [12] Caillon, A. & Esling, P. Rave: A variational autoencoder for fast and highquality neural audio synthesis (2021). URL https://arxiv.org/abs/2111. 05011.2111.05011. [13] Zeghidour, N., Luebs, A., Omran, A., Skoglund, J. & Tagliasacchi, M. Soundstream: An end-to-end neural audio codec. CoRR abs/2107.03312 (2021). URL https://arxiv.org/abs/2107.03312.2107.03312. Appendix A Linear Analysis of Modulation Representation Jupyter notebook available at: sonic-trajectories/blob/main/study-3.ipynb This appendix details a third study, an extension of the metric-based methods in Studies 1 and 2. The main goal is to find if control parameter modulations can be linearly reconstructed from the vectors of an audio representation. This analysis asks two questions: 1. For a single audio sample, can a linear combination of a representation’s dimensions reconstruct the ground-truth modulation signal? How good is this reconstruction? 2. If these linear combinations exist, do they work for other samples? Is a filter cutoff sweep, for example, always encoded in the same dimensions for different sounds? 53 54 Appendix A. Linear Analysis of Modulation Representation A.1 Setup A.1.1 Dataset The datset for this study is significantly different from the previous, it consists of 1000 10s (more than 3h) minisynth synthesized samples with random sub audio modulations. Samples have 1,2,3 or 4 concurrent modulations, all different with each other. This ensure the maximum variety possible for this minisynth class, which means that if the method works on those minisynth sounds it works on all of them. The dataset is available at huggingface.co/datasets/inspektral/minisynth1ksub-points-v1 and renderable via minisynth (see notebook). A.1.2 Methodology The method is a two-stage analysis using linear regression. The assumption is that the modulation signal M(a time series) is a linear combination of the dimensions of the audio representation R(a matrix of time frames x dimensions). The relationship is: M≈R·w Where: •Mis the ground-truth modulation vector (length T). •Ris the representation matrix (T×D). •wis a weight vector (length D). The optimal weight vector wis found using ordinary least squares (OLS), which minimizes the squared difference between the real modulation Mand the predicted one R·w. A.2. Results 55 A.1.3 Stage 1: Per-Sample Model Fitting In the first stage, the process is applied to each audio sample. For each sample i, with representation Riand modulation Mi, a weight vector wiis computed. The model performance is measured with the Pearson correlation between the groundtruth modulation Miand the reconstructed modulation Ri·wi. A high correlation (close to 1.0) means the modulation is linear for that sample’s representation. A.1.4 Stage 2: Generalization with an Average Model The second stage tests if the model generalizes. The weight vectors (wi) for all samples with the same modulated parameter (e.g., filter cutoff) are averaged to create a single vector, wavg: wavg =1 N N X i=1 wi This average vector is a general model for that parameter and representation. Its performance is tested on each sample j. Performance is the Pearson correlation between the ground-truth Mjand the reconstructed modulation Rj·wavg. Good performance implies the modulation is encoded in the same way for all samples. Poor performance suggests the encoding is context-dependent, using different dimensions or weights for each sample. A.2 Results The results from the two-stage analysis answer the initial questions. A.2.1 Per-Sample Model Performance The analysis shows that for any sample, a linear combination of dimensions can be found to reconstruct the modulation. As shown in Figure 22, the correlation scores 56 Appendix A. Linear Analysis of Modulation Representation for the per-sample models were close to 1.0for most parameters and representations. This shows that the modulation information is present and linear within the representation’s dimensions. Figure 22: Correlation histograms for per-sample model fitting. For all four modulation types in the ‘music2latent‘ representation, the correlations are heavily skewed towards 1.0, indicating near-perfect linear reconstruction. A.2.2 Generalization Model Performance The performance of the average model depended on the modulated parameter and the representation. •Good Generalization: Sometimes, the average model performed well, which means the encoding was consistent. For example, amplitude modulation worked well, as most representations capture energy. •Poor Generalization: In other cases, the model’s performance was much worse. The results varied, with poor results for many samples. For example, the ‘music2latent‘ model for filter cutoff gave varied results, while its model for oscillator shape performed poorly overall. A.3. Discussion 57 Figure 23: Correlation distribution for the generalized average model on ‘music2latent‘. The filter cutoff model (left) shows a wide, inconsistent distribution of correlations, while the oscillator shape model (right) performs poorly overall, with most correlations below 0.4. This shows that while the modulation is linear in each sample, the dimensions and weights change for each sample. A change in base frequency, for example, can make the representation use different dimensions to encode a filter sweep. A.3 Discussion These findings add to the results of Studies 1 and 2. The simple metrics of magnitude, distance, and similarity show general change, but this study shows the information is more structured. The near-perfect per-sample result is important. It suggests that even in complex latent spaces, the representation of features has structure; it is linear on a per-sample basis. The mixed results of the average model are also important. It answers questions about interpretability and consistency. To be interpretable, a feature like filter cutoff should be represented in the same way. The results show this is not always true; the encoding depends on context. This is a key problem in using audio representations: a feature can be represented differently for each sound.