Freesound Loop Generator

Master in Sound and Music Computing Universitat Pompeu Fabra Freesound Loop Generator Ada Salvador Avalos Supervisor: Lonce Wyse Co-Supervisor: Dmitry Bogdanov, Pablo Alonso August 2025 Contents 1 Introduction 1 1.1 Motivation.................................. 2 1.2 ResearchObjectives ............................ 3 1.3 StructureoftheThesis........................... 4 2 State of the Art 6 2.1 Generative Models for Latent Space Manipulation . . . . . . . . . . . . 7 2.1.1 Variational Autoencoders . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.1.2 TransformerModels ............................ 8 2.1.3 Alternatives Approaches: Generative Adversarial Networks (GANs), Diffusion and Hybrid Models . . . . . . . . . . . . . . . . . . . . . . . . 10 3 Methods 15 3.1 Dataset ................................... 15 3.1.1 DataPreprocessing............................. 16 3.2 Exploratory Dataset Analysis . . . . . . . . . . . . . . . . . . . . . . . 22 3.2.1 StyleEncoding ............................... 22 3.2.2 Style Activation Distribution Analysis . . . . . . . . . . . . . . . . . . 23 3.2.3 Style Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 3.3 Realtime Audio Variational AutoEncoder (RAVE) . . . . . . . . . . . . 28 3.3.1 Architecture and Training Strategy . . . . . . . . . . . . . . . . . . . . 28 3.3.2 Latent Space Compression Method . . . . . . . . . . . . . . . . . . . . 29 3.3.3 Reconstruction from Compressed Latents . . . . . . . . . . . . . . . . . 30 3.3.4 Implementation of the Training Procedure . . . . . . . . . . . . . . . . 31 3.4 Real-Time Neural Audio Synthesis and Morphing System . . . . . . . . 32 3.4.1 Playback Manipulation . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 3.4.2 Encoding .................................. 33 3.4.3 Latent Interpolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 3.4.4 Real-timeSynthesis............................. 34 3.4.5 Decoding .................................. 36 3.5 EvaluationMetrics ............................. 37 3.5.1 Subjective Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 3.6 Experiments................................. 38 3.6.1 Conditioned Morpher Transformer Model . . . . . . . . . . . . . . . . . 38 3.6.2 DataPreprocessing............................. 38 3.6.3 ModelArchitecture............................. 39 3.6.4 TrainingProcedure............................. 43 3.6.5 Audio Morphing Inference System . . . . . . . . . . . . . . . . . . . . . 47 3.6.6 EvaluationMetrics ............................. 50 3.6.7 Results.................................... 51 4 Results 53 4.1 5-pointLikertScales ............................ 53 4.1.1 Agreement Scale Responses (Q1–Q3) . . . . . . . . . . . . . . . . . . . 54 4.1.2 Intensity Scale Responses (Q4–Q5, Q8–Q9) . . . . . . . . . . . . . . . 55 4.2 Identification of Musical Aspects for each Dimension . . . . . . . . . . 56 4.3 Perceived Control: Uniform vs. Per-Dimension Blending . . . . . . . . 58 5 Discussion 59 5.1 Conclusion.................................. 60 5.2 FutureWork................................. 61 List of Figures 62 List of Tables 63 Bibliography 64 A Questionnaire for Subjective Evaluation 69 A.1 Linear Interpolation in Latent Space Using a Uniform Ratio Across All Dimensions ................................. 69 A.2 Per-Dimension Ratio Control in Latent Space . . . . . . . . . . . . . . 70 A.3 Exploring Dimensionality in Latent Space . . . . . . . . . . . . . . . . 71 B Model Architecture 72 C Materials for Reproducibility 74 Acknowledgement First, I would like to express my gratitude to my supervisor and co-supervisors for their guidance and advice throughout this project, for patiently answering my questions, and helping me overcome doubts that opened the way to new perspectives. I am truly grateful for they availability and willingness to support me. I would also like to thank my friends for their patience and understanding, for always listening, and for offering support even when I wasn’t around due to all the work involved. I am truly grateful for their encouragement and presence throughout this journey. Finally, I am especially grateful to my family for their endless support, care, and belief in me. Without them, I could never have made it this far. Abstract This project investigates the synthesis of new audio loops using neural networks, focusing on creative sound generation through latent space manipulation. Using the FreeSound Loop Dataset, audio samples were preprocessed through tempo normalization as well as beat and downbeat alignment to ensure rhythmic consistency and structural coherence, enabling musically relevant synthesis. The system is built around a neural autoencoder, specifically the RAVE model, which compresses audio loops into compact latent representations and reconstructs them with high-fidelity. New loops are generated by interpolating between two encoded examples, producing smooth transitions and hybrid sounds that blend characteristics from both sources. Genre classification guides latent space traversal, supporting stylistic selection and controlled blending of sounds. Furthermore, realtime manipulation of individual latent dimensions expands the system’s potential for interactive audio applications, such as live performances or dynamic sound design. Subjective evaluation demonstrated that interpolated loops are perceptually coherent and musically meaningful. Listeners reported that latent space trajectories provide expressive and controllable tools for creative composition, and, that varying the number of latent dimensions influences musical richness without compromising perceptual continuity. By combining deep generative models, musically informed preprocessing, and usercontrollable synthesis techniques, this work presents a flexible framework for creative loop-based audio generation, bridging high-quality synthesis with practical usability. Keywords: Audio Generation; Neural Audio Synthesis; Latent Space Interpolation; Interactive Sound Design; Loop-Based Music Chapter 2 State of the Art The field of deep learning has witnessed rapid advancements, particularly in generative models. Traditional methods, such as variational autoencoders (VAEs) [3], have demonstrated significant capabilities in modeling complex data distributions through the learning of latent spaces. More recently, transformer-based architectures [7] have redefined how latent spaces can be explored and manipulated, offering good scalability and performance across various domains. This chapter examines state-of-the-art methodologies relevant to latent space manipulation and interpolation, with a particular focus on morphing techniques across various generative models. Section 2.1 places special emphasis on variational autoencoders (VAEs) –particularly the RAVE model [12]– and transformer architectures, alongside generative adversarial networks (GANs) [4], diffusion models [6] [5], and hybrid approaches. The analysis of how these models synthesize audio provides insights into how architectural differences influence the quality and coherence of the generated output. 6 2.1. Generative Models for Latent Space Manipulation 7 2.1 Generative Models for Latent Space Manipulation Morphing techniques enable smooth transitions between different data points in the latent space, allowing for interpolation, attribute editing, and controlled transformations. Different classes of generative models offer unique approaches to morphing. 2.1.1 Variational Autoencoders Variational autoencoders (VAEs) [3] are generative models designed to learn a probabilistic mapping from observed data to a structured latent space. Unlike deterministic autoencoders, VAEs introduce stochasticity through a probabilistic encoderdecoder framework, where the encoder maps input data to a distribution over latent variables rather than a single point. This stochastic nature promotes robustness and smoothness in the learned latent space, enabling meaningful interpolations and traversals. The latent space of a VAE is typically modeled as a Gaussian distribution, where the encoder outputs parameters defining a mean and variance for each input. During training, latent variables are sampled from these distributions, and the decoder reconstructs the original input from the sampled points. The model is optimized by maximizing a variational lower bound, which includes both a reconstruction loss and a regularization term enforcing the latent space to approximate a predefined prior distribution, usually a standard Gaussian. Early research demonstrated that VAEs could morph between distinct data points by linearly interpolating between their corresponding latent representations. Due to the continuous and structured nature of the latent space, transitions between points produce coherent, gradual transformations. Additionally, the standard VAE framework often produces entangled latent spaces, where different generative factors are not well-separated. This lack of disentanglement makes morphing operations less interpretable and can cause undesired blending 8Chapter 2. State of the Art of unrelated features during interpolation. Several approaches have been proposed to address these limitations by improving the structure and interpretability of the latent space. For example, hierarchical VAEs [13], they introduce a multi-level latent space architecture, where higher-level latent variables capture global structure while lower-level variables capture finer details. This hierarchical organization allows smoother and more coherent interpolations by modeling complex data distributions more effectively. Furthermore, structured latent space models, such as Beta-VAEs [14], modify the training objective to encourage disentanglement [15]. By introducing a hyperparameter βto control the trade-off between reconstruction fidelity and latent regularization, Beta-VAEs promote the separation of independent generative factors. Such disentangled representations enable more interpretable and controlled morphing operations. While VAEs are well-suited for generating smooth transitions within a structured latent space, balancing reconstruction quality with meaningful latent representation remains a fundamental challenge. Enforcing smoothness through strong regularization can degrade reconstruction quality, leading to oversmoothed outputs that lack detail. Conversely, relaxing the regularization can enhance fidelity but at the expense of losing coherence in interpolations. The RAVE model [12] overcomes several traditional limitations of VAEs by combining adversarial regularization and multi-resolution spectral reconstruction loss. This enables high-fidelity and real-time audio generation, making RAVE particularly well-suited for interactive applications such as live audio loop creation and manipulation. 2.1.2 Transformer Models Transformer models [7] have revolutionized sequence modeling by employing selfattention mechanisms to process input data in a parallelized manner, bypassing the need for recurrent or convolutional structures. They learn contextual relationships over long distances, making them particularly effective for tasks that require preserving coherence across extended sequences. The self-attention mechanism com- 2.1. Generative Models for Latent Space Manipulation 9 putes dependencies between all elements of an input sequence simultaneously, allowing transformers to capture complex patterns and long-range interactions that traditional architectures struggle to model. This capacity for understanding global dependencies is especially valuable for morphing tasks, where smooth transitions between diverse inputs are essential. For example, LoopNet[16] employs attention mechanisms to capture temporal dependencies within musical loops. It learns to recognize patterns and structures over multiple time scales, facilitating the generation of coherent and continuous audio loops. Wave-U-Net [17], originally designed as a convolutional model for audio source separation, has been adapted with transformer architectures to enhance its capacity for modeling long-range dependencies. This integration provides improved control over generating structured audio outputs across extended sequences. Even MelNet [18], a model designed for generating high-fidelity audio spectrograms using hierarchical architectures, supports the generation of structured outputs that retain fine-grained details over extended durations. Most recently, Transformer-XL [19], a model that is a variant of the standard transformer, addresses the challenge of generating long sequences by incorporating recurrence mechanisms. In typical transformers, the model’s attention mechanism is limited to a fixed-length context window, which restricts the ability to capture longrange dependencies. Transformer-XL solves this limitation by introducing a memory mechanism that allows the model to retain and reuse hidden states from previous segments of input data across different training steps. This enables the model to handle much longer sequences efficiently, making it well-suited for tasks like language modeling, where maintaining long-term dependencies is crucial for coherent output. This ability to persist memory over long distances is particularly useful when generating continuous data, such as long audio sequences, where consistency and context preservation are essential. In addition, AudioLM [20] is a hierarchical model designed for high-quality audio generation. Unlike traditional models that directly generate audio waveforms or spectrograms, AudioLM leverages a two-level hierarchical approach to generate au- 10 Chapter 2. State of the Art dio. It learns to represent audio data at both low-level (fine-grained) and high-level (abstract). The low-level representations capture more granular features of audio, such as acoustic properties, while the high-level representations capture broader patterns like musical structures or speech context. The hierarchical nature of AudioLM enables it to model both local and global dependencies within the data. This makes it particularly powerful for tasks that require contextual accuracy, such as generating speech or music, where understanding both the micro-level (like phonemes or notes) and macro-level (like sentence or melody structure) is critical. By learning at multiple levels of abstraction, AudioLM can generate realistic audio that maintains coherence over long durations. Transformers offer a powerful framework for modeling complex sequential data, particularly due to their ability to capture long-range temporal dependencies through self-attention mechanisms. This makes them well-suited for tasks requiring structural coherence, such as audio loop synthesis and morphing. Although most transformerbased systems in the literature focus on generation or translation tasks, there is growing interest in exploring their potential for latent space traversal and style conditioning. However, established architectures like VAEs currently dominate this space in terms of coherence and fidelity. 2.1.3 Alternatives Approaches: Generative Adversarial Networks (GANs), Diffusion and Hybrid Models Generative Adversarial Networks Generative adversarial networks (GANs)[4], have become a foundational tool for generating realistic samples by training a generator against a discriminator in a competitive framework. The generator aims to produce data indistinguishable from the real dataset, while the discriminator attempts to discern between real and generated samples. The learned latent space in GANs has been extensively studied for its capability to support linear and nonlinear morphing techniques. Such techniques include style transfer, attribute-based editing, and smooth transitions between dif- 2.1. Generative Models for Latent Space Manipulation 11 ferent data samples. For example, conditional GANs (cGANs) [21] introduce additional conditioning variables to guide the latent space exploration, giving users more control over the generated outputs. These conditioning variables can include attributes such as tempo, key, or genre, which can be specified to influence the generated audio or visual content. Early GAN architectures often suffered from training instability and mode collapse. However, advancements such as StyleGAN [22] and StyleGAN2 [23] introduced significant improvements. These models proposed a style-based generator architecture where a mapping network transforms input vectors into intermediate latent codes. This approach enabled hierarchical disentanglement of features and fine-grained control over generated samples, enhancing the model’s ability to perform smooth and coherent morphing. Furthermore, StyleGAN3 [24] was introduced, improving upon its predecessors by addressing aliasing issues and ensuring better spatial consistency. This version introduced Fourier features and an improved generator architecture, allowing for smoother transitions and enhanced morphing quality by preserving geometric consistency across interpolations. Recent work by Hung et al.[25] explored loop morphing using StyleGAN, StyleGAN2, and UNAGAN[26], showing that smooth transitions can be achieved across different domains, including visual and auditory data. Their approach demonstrated the effectiveness of GAN-based models in learning coherent, seamless transformations between different latent representations. In particular, StyleGAN’s mapping network allows for non-linear control over attributes, which has been effectively exploited in various morphing tasks. The ability to traverse the latent space in controlled directions enables transitions between attributes, styles, or even domains. For audio-based generation and morphing, several GAN architectures have been employed successfully. UNAGAN demonstrated the feasibility of applying GANs to sequence-to-sequence generation, showcasing robust morphing capabilities in speech synthesis and music generation. Other notable models include Parallel WaveGAN[27] 12 Chapter 2. State of the Art and MelGAN [28], which have proven effective for high-quality audio waveform generation. GANsynth [29], in particular, introduced a GAN-based approach for audio synthesis that operates directly in the frequency domain, achieving high-quality, fast generation of musical audio. Even with recent advances such as pSp (pixel2style2pixel) [30] and e4e (Encoder for Editing) [31], which enhance latent interpretability and enable more meaningful manipulations in StyleGAN-generated images, GANs remain less flexible than VAEs for structured latent interpolations. Their application to audio is still largely unexplored, making them less suitable for real-time, controllable loop morphing. Diffusion models Denoising diffusion probabilistic models (DDPMs) have recently gained popularity for their high-quality generative capabilities, offering a different approach to generative modeling compared to variational autoencoders (VAEs) and generative adversarial networks (GANs). Unlike these models, diffusion models use an iterative process to gradually transform noise into meaningful samples through a sequence of denoising steps. The foundational work by Sohl-Dickstein et al. [6] introduced the idea of using a Markov chain to progressively add noise to data, effectively destroying its structure over several time steps. By learning to reverse this process, a model can generate realistic samples from pure noise. This formulation laid the groundwork for more advanced diffusion models by defining a structured generative process governed by probabilistic transitions. Further refinement came with the work of Ho et al. [5], which introduced a simpler and more efficient training objective known as the denoising score-matching loss. This approach improved training stability and generation quality by optimizing the model to predict the original data from a corrupted version at each time step, rather than directly modeling the entire reverse process. Recent diffusion-based models, such as Imagen [32], which employs a cascaded diffusion process conditioned on text descriptions, and Stable Diffusion [33], which introduces a latent diffusion approach wherein the generative process operates in 2.1. Generative Models for Latent Space Manipulation 13 a compressed latent space rather than directly in pixel space, have demonstrated highly efficient and scalable generation. These approaches not only achieve state-ofthe-art photorealistic image synthesis but also support controlled interpolation and morphing through latent conditioning, text prompts, or attribute guidance. The iterative nature of the denoising process allows for finer control over generated samples, making them particularly effective for morphing tasks. During interpolation, they can produce smooth and coherent transitions by operating at various noise levels. This approach prevents abrupt or unrealistic transformations, providing a structured mechanism for generating continuous morphs. Furthermore, the flexibility of diffusion models allows them to be guided by external inputs (e.g., text prompts or conditioning vectors), enabling controlled and interpretable interpolations. However, diffusion models are computationally demanding, often requiring hundreds of iterative steps to generate a single sample. Although latent diffusion variants reduce this burden to some extent, real-time or interactive applications –such as loop morphing in performance contexts– remain impractical due to latency and resource requirements. Hybrid Approaches Combining different generative model architectures leverages the strengths of each to create models that are more capable of handling complex tasks, especially when these tasks require both high-fidelity generation and coherent temporal structure. Hybrid approaches typically involve combining two or more model types—such as GANs, RNNs, transformers, and diffusion models—to improve overall performance. A hybrid approach is to combine GANs—such as StyleGAN or StyleGAN2—with RNNs or transformers to address different aspects of generative tasks. GANs are typically effective for modeling high-quality structures such as images or audio textures. RNNs and transformers, on the other hand, excel at capturing temporal dependencies and sequential structure, making them ideal for tasks like modeling musical rhythms, progressions, or coherence over time. For example, in Style-conditioned 14 Chapter 2. State of the Art Music Generation with Transformer-GANs by Wang et al. [34] the authors present a music generation algorithm that creates compositions from scratch based on specified target styles. It incorporates a style-conditioned linear transformer to model MIDI event sequences and a style-conditioned patch discriminator within a GAN framework to enhance the modeling of music sequences. Some models like SpecDiff-GAN [35], FastDiff 2 [36] suggest using GANs to generate an initial structure or coarse representation of data, followed by diffusion models to refine and add intricate details. This method leverages the GAN’s ability to capture the overall data distribution and the diffusion model’s strength in modeling complex, high-frequency components, resulting in high-fidelity outputs. These hybrid methods demonstrate that integrating complementary architectures can yield good results in generating complex audio loops, especially when aiming to balance stylistic diversity with coherent temporal evolution. However, models that combine elements of VAEs, GANs, and diffusion architectures remain complex to train, resource-intensive, and often harder to interpret. The added architectural overhead can also limit adaptability in interactive music-making scenarios, where efficiency and transparent control are crucial. Chapter 3 Methods This chapter outlines the methodology employed in this work. Section 3.1 begins with a discussion of the chosen dataset, followed by the preprocessing steps described in Subsection 3.1.1. Section 3.2 presents an exploratory data analysis, emphasizing the style distribution of the subset used to train the model. Subsequently, Section 3.3 describes the selected model for real-time audio synthesis and its training implementation, while Section 3.4 covers its inference process, associated applications, and modulation capabilities. Finally, Section 3.5 outlines the evaluation metrics employed, and Section 3.6 concludes with an experimental study of a custom-built transformer model, including its dedicated preprocessing pipeline, model architecture, training procedure, inference process, evaluation metrics, and comparative results against the core system model. 3.1 Dataset The FreeSound Loop Dataset [37] was selected for its rich combination of highquality audio loops and detailed metadata annotations, making it well-suited for music analysis and generation tasks. The dataset comprises 9,455 loops collected from the Freesound platform [38], each annotated with tempo (BPM), musical key, 15 22 Chapter 3. Methods 3.2 Exploratory Dataset Analysis The dataset used for training consisted of 5 hours of audio, corresponding to 1125 audio files drawn from the preprocessed corpus described in Subsection 3.1.1. To avoid ordering bias, the files were randomly shuffled prior to selection. This subset was chosen to provide a representative sample of the larger dataset while maintaining computational feasibility during model training. To better understand the selected subset, an analysis was conducted focusing on the distribution of music styles and the classification of audio files according to these styles. This analysis made it possible to assess how balanced the subset was across different sub-genres and to identify potential biases that could have influenced model performance. 3.2.1 Style Encoding The MAEST (Music Audio Efficient Spectrogram Transformer) model [40] was employed for automatic genre distribution and classification. It is a transformer-based model designed for music tagging. Unlike conventional CNN-based approaches, MAEST is optimized for short audio segments and supports multi-label classification over hundreds of genres. In the pipeline, the audio signal was first converted into a log-mel spectrogram, using an FFT size of 1024, a hop length of 320, and 80 mel frequency bins. The resulting spectrogram was then resized to match the input dimensions required by the MAEST architecture. The pre-trained checkpoint discogs-maest-5s-pw-129e was employed, which performs genre predictions on 5-second audio segments and outputs probabilities across 400 musical styles. To obtain meaningful activation probabilities, the auxiliary function predict_labels() was used; it applies a sigmoid activation and averages the predictions across time dimension. The full 400-dimensional activation vector was used as the style representation to 3.2. Exploratory Dataset Analysis 23 analyze the distribution and classification of genres within the subset. 3.2.2 Style Activation Distribution Analysis Median and IQR Analysis To see how musical styles are represented in the dataset, the activation probabilities of the 400 style categories were analyzed across all audio files. Each audio file was associated with a 400-dimensional style probability tensor, representing the likelihood of each style being present. The median activation probability and the interquartile range (IQR) were computed for each style across the dataset. Figure 1 displays the top 20 styles ranked by median activation, with error bars indicating the IQR. Figure 1: Median Activation of style probability vectors. Electronic subgenres dominate top ranks, with Electronic—Abstract,Electronic— Experimental and Electronic—Glitch showing the highest median activations. This suggests a strong presence of electronic textures and characteristics in the dataset. The IQR values further reveal the variability in style activation. Some styles, such as Electronic—Glitch show high median values but also large IQRs, indicating that they are often prominent but with substantial variation in strength across the dataset. In contrast, styles like Rock—Goregrind exhibit a low median combined a with wide IQRs, implying a more sporadic and inconsistent activation pattern. Meanwhile, 24 Chapter 3. Methods styles such as Electronic–Industrial present both low median and low IQR values, suggesting a small but consistent presence across the dataset. PCA Projection To further explore the structure of the style representations in the dataset, Principal Component Analysis (PCA) was applied to the 400-dimensional style activation vectors. Each style representation was projected into a 2D space defined by the first two principal components, which capture the most significant variance in the style probability distributions. Figure 2 visualizes this projection, with points color-coded by their predicted parent genre. Parent genres were determined by averaging activation scores within genrespecific subsets of the 400 style categories and assigning each file the genre with the highest mean activation. Figure 2: PCA projection of 400-dimensional style probabilities across audio files. The first principal components account for 36.14% and 19.76% of the total variance, respectively. Colors indicate predicted parent genre. The PCA projection reveals overlapping cluster but also some degree of genrespecific separation. Notably: 3.2. Exploratory Dataset Analysis 25 •Electronic and Non-Music files are more widely spread, forming diffuse clusters, likely due to the diversity of styles within these categories. •Stage & Screen and Reggae show tighter groupings, suggesting more consistent style activations across the files. •Rock and Hip Hop appear more dispersed, though they exhibit localized tendencies, potentially reflecting hybrid or overlapping style characteristics. This projection supports the idea that while many genres share stylistic similarities (as evidenced by cluster overlap), the model captures enough stylistic variation to organize files by genre in low-dimensional space. 3.2.3 Style Classification Each audio sample was represented with a style probability vector, with each element corresponding to a sub-genre. These sub-genres were then grouped into broader parent genres, as listed in Table 1. Classification was performed by calculating the mean probability across sub-genres within each parent genre, ensuring fairness across genres with differing number of styles. 26 Chapter 3. Methods Parent Genre Sub-Genre Electronic 106 Rock 91 Latin 35 Folk, World, & Country 27 Hip Hop 26 Jazz 25 Pop 16 Funk / Soul 15 Classical 13 Non-Music 13 Blues 12 Reggae 11 Stage & Screen 4 Brass & Military 3 Children’s Music 3 Table 1: Number of sub-genres (styles) associated with each parent genre. Based on the classification procedure described above, Table 2 shows the distribution of the 1,125 classified files across parent genres. Genres not listed had no assigned samples and were removed. 3.2. Exploratory Dataset Analysis 27 Parent Genre Number of files (%) Electronic 195 (17.3%) Rock 107 (9.5%) Hip Hop 127 (11.3%) Funk / Soul 2 (0.2%) Non-Music 590 (52.4%) Reggae 47(4.2%) Stage & Screen 29 (2.6%) Brass & Military 28 (2.5%) Table 2: Number of files classified into each parent genre. 28 Chapter 3. Methods 3.3 Realtime Audio Variational AutoEncoder (RAVE) For the purpose of real-time audio manipulation through latent space control, the Realtime Audio Variational AutoEncoder (RAVE) [12] was selected for its high synthesis quality and low latency. The subsections that follow provide a structured description of the RAVE model. First, the overall architecture and the two-stage training procedure are presented. Followed by a discussion of the latent space compression strategies for effective manipulation and the reconstruction approach from reduced latent representations. Finally, the specific training implementation used in this work is described. 3.3.1 Architecture and Training Strategy The Realtime Audio Variational AutoEncoder (RAVE) is a deep generative model that consists of an encoder that maps input audio spectrograms into a compact latent vector, capturing perceptually relevant features, and a decoder that reconstructs audio from this latent representation. To achieve efficient real-time performance, RAVE employs a multi-band synthesis strategy, where the waveform is decomposed into several sub-bands that are predicted in parallel and then recombined. This architecture not only reduces computational complexity but also enables highquality, low-latency audio generation. The latent space is designed to be structured and compressible, allowing for flexible audio manipulations such as interpolation between samples and real-time modulation of individual latent dimensions. RAVE is trained with a two-stage training procedure: •Stage 1: Representation Learning In this phase, the model is trained as a variational autoencoder (VAE). Through it differs from standard VAE implementations by using the multiscale spectral distance [41]. This spectral distance is crucial for audio applications as it avoids penalizing irrelevant phase variations that would occur with raw waveform L2 loss. The goal is to learn a latent representation that captures the 3.3. Realtime Audio Variational AutoEncoder (RAVE) 29 perceptually relevant features of audio while being robust to phase differences. •Stage 2: Adversarial Fine-Tuning Once the encoder has learned a meaningful representation, it is frozen to preserve the latent space structure. The decoder continues training with a combination of three loss components: – Adversial loss: To fool the discriminator and improve the synthesis realism. – Continued spectral distance loss: To maintain reconstruction fidelity. – Feature matching loss [28]: To match discriminator feature maps between real and generated audio. This multi-objetive approach ensures that synthesis quality improves without compromising the stability and meaningfulness of the latent representation. 3.3.2 Latent Space Compression Method RAVE addresses the challenge of identifying the most informative dimensions in the learned latent space to enable more effective manipulation and analysis. The goal is to find the minimal subset of dimensions in the latent vector zthat retains enough information for high-fidelity reconstruction. The method uses post-training Singular Value Decomposition (SVD) to distinguish between informative and uninformative latent dimensions. However, applying SVD directly to samples Z∈Rb×d, here bis the number of audio samples in the batch and dis the latent dimensionality, would be problematic due to high variance in collapsed dimensions that have converged to prior. To address this, a modified matrix Z′∈Rb×dis constructed where each row represents the mode (most likely value) of the posterior distribution for sample i: Z′ i= arg max zqϕ(z|x)(3.9) 30 Chapter 3. Methods Where, z∼qϕ(z|x)denotes a latent vector zsampled from the posterior distribution qϕ(z|x)of the encoder given an input audio x, parameterized by ϕ. For Gaussian posteriors, this corresponds to the mean distribution. The matrix Z′is centered by subtracting the mean across samples, ensuring that collapsed dimensions (which have constant values) become zero after centering. SVD is applied to the centered matrix Z′: Z′=UΣVT(3.10) Where U∈Rb×bcontains the left singular vectors representing directions in sample space, Σ∈Rb×dis a diagonal matrix of singular values, and V∈Rd×dcontains the right singular vectors, representing the directions in the latent space. A fidelity parameter f∈[0,1] determines the minimal number of dimensions rfto retain: Prf i=1 Σii Pd i=1 Σii ≥f(3.11) Where Σii refers to the i-th singular value in Σ, and indicates the contribution of each latent dimension to the reconstruction variance. This allows any latent vector zto be projected to a compact representation zf∈Rrf containing only the most informative dimensions. 3.3.3 Reconstruction from Compressed Latents For reconstruction, the compressed latent zfis concatenated with random noise ϵ∼ N(0, I)for the uninformative dimensions, forming a full latent vector ˜z= [zf;ϵ]VT, which is then passed through the decoder to generate audio. Here, ϵis a multivariate normal distribution with zero mean and identity covariance, and [zf;ϵ] indicates concatenation along the latent dimension. The paper demonstrates that with f= 0.99, the latent dimensionality can be reduced from 128 to 24 dimensions for string music and 16 for speech while maintaining reconstruction quality. 3.3. Realtime Audio Variational AutoEncoder (RAVE) 31 3.3.4 Implementation of the Training Procedure RAVE was trained for a total of 2 million steps following the two-stage procedure described in Subsection 3.3.1. The first 1 million steps corresponded to representation learning, while the second 1 million steps corresponded to adversarial fine-tuning. The v2 architecture was employed, which is an improved continuous model optimized for faster and higher quality generation. Several additional configurations were applied during training. The causal setting enforced the model to rely on exclusively on past waveform samples, which is essential in real-time scenarios as it reduces the perceived latency, although at the expense of reconstruction quality. The noise setting introduced a noise synthesizer in the decoder, which improves modeling of sounds containing significant noisy components. Single-channel audio input was also used to ensure consistent processing across the dataset. After training, the model was exported as a TorchScript file for deployment. The –streaming option was enabled, which activates cached convolutions and ensures compatibility with real-time audio processing. 38 Chapter 3. Methods 3.6 Experiments 3.6.1 Conditioned Morpher Transformer Model In addition to utilizing the RAVE model, a custom transformer model was developed to perform audio morphing while incorporating conditioning on style and BPM (beats per minute) as the quantitative representation of musical tempo. 3.6.2 Data Preprocessing The audio preprocessing pipeline consisted of four main stages: audio normalization, tempo adjustment, genre classification, and neural audio coding. Raw audio files from the FreeSound Loop Dataset were processed to create suitable standardized representations for the model. Audio Normalization and Standardization All input audio files were standardized to a consistent format with the following specifications: •Sample rate: fs= 44.1kHz •Duration: T= 5 seconds •Channels: Mono (stereo files were converted by averaging channels) •Target samples: N=T×fs= 220,500 samples Tempo Normalization To ensure rhythmic consistency across the dataset, an optional tempo normalization procedure targeting BPMtarget = 120 was implemented, as described in Subsection 3.1.1. However, this normalization was not applied during model training. 3.6. Experiments 39 Genre Classification and Style Encoding The style embedding was extracted using the MAEST model [40], as described in Subsection 3.2.1 for classification purposes. Specifically, the pre-trained checkpoint discogs-maest-5s-pw-129e was used to obtain the full 400-dimensional activation vectors for the whole dataset. Neural Audio Coding To compress and represent audio in a discrete latent space, the Discrete Audio Codec (DAC) [42] was employed –a state-of-the-art neural audio codec designed for highfidelity audio reconstruction. DAC encodes audio signals into compact sequences of discrete tokens through a fully convolutional encoder-decoder architecture, followed by Residual Vector Quantization for efficient discretization. The 44 kHz DAC model with GPU-based encoding was used to ensure fast and uniform token generation across the dataset. 3.6.3 Model Architecture The model employs a transformer-based encoder-decoder architecture with separate FiLM conditioning layers for style and BPM [43]. Within the encoder, audio tokens are processed through multiple codebook embeddings, with FiLM layers modulating features according to style and BPM. Representations from both source and target inputs are then linearly interpolated. Finally, the decoder generates output logits using cross-attention to the interpolated representations, incorporating the same dual conditioning mechanisms. 40 Chapter 3. Methods Figure 3: Overview of the model. Source(s) and target(t) inputs (x) are encoded (E) into latent space representations and decoded (D) into output (ˆy). Source (zs) and target (zt) latent representations are interpolated using α∈[0,1] to produce zmorph. Conditioning vectors include cs= [bs, σs],ct= [bt, σt], and cdecoder = [bdecoder, σdecoder], with cdecoder ∈ {ccustom, ct}. Where σand brepresent the style and BPM vectors, respectively. A detailed diagram is provided in Figure 8 in Appendix B. Input Stage The model takes two audio sequences for morphing, corresponding to the source and target domains. Each domain consists of audio codes x∈ZB×K×L +from K= 9 codebooks, a style vector σ∈RB×400 +, and a BPM scalar b∈RB×1 +, where Bis the batch size and Lis the maximum sequence length. Embedding Stage Audio tokens from each codebook are processed through separate embedding layers (mapping from V= 1024, the vocabulary size, to dmodel = 64 dimensions), then mean-pooled across codebooks and enhanced with positional encoding. The conditioning information (style and BPM) is processed through separate embedding networks: •Style conditioning: Linear projection from 400 to 64 dimensions 3.6. Experiments 41 •BPM conditioning: MLP with Linear →LayerNorm →ReLU →Linear → LayerNorm, which transforms the scalar BPM to a 64-dimensional embedding. Feature-wise Linear Modulation (FiLM) The model employs separate FiLM layers for style and BPM conditioning allowing independent modulation of features. Given input embedded features xemb ∈ RB×L×dmodel , style condition cσ∈RB×dmodel and BPM condition cb∈RB×dmodel , each FiLM layer computes: FiLM(xemb, c) = γ(c)⊙xemb +β(c)(3.14) Where γ(c)and β(c)are learned MLP projections (Linear →ReLu →Linear) of the conditioning vector, and ⊙denotes element-wise multiplication. The combined FiLM operation applies both style and BPM modulations additively: xout =xemb + (FiLMσ(xemb, cσ)−xemb)+(FiLMb(xemb, cb)−xemb)(3.15) Preserving both conditioning signals while maintaining the original feature structure. Encoder Stage Six FiLM-conditioned transformer blocks encode each sequence. Each block contains masked multi-head self-attention (8 heads, dk= 16; masked to ignore padding positions), followed by Add & Norm, Dual FiLM modulation (as described in Subsection 3.6.3), a feed-forward network (Linear →ReLU →Linear), another Add & Norm, and a second dual FiLM layer. The output then passes through a latent projector (Linear →LayerNorm →ReLU →Linear) producing latent representation of shape [B, L, dmodel]. 42 Chapter 3. Methods Latent Interpolation Source and target latent are interpolated in the latent space: zmorph = (1 −α)·zs+α·zt(3.16) Where α∈[0,1] controls the morphing ratio, and zsand ztdenote the source and target latents, respectively. Decoder Stage Six FiLM-conditioned transformer blocks decode the morphed latent using target conditioning. Each decoder block follows a structured pipeline: 1. Masked multi-head self-attention is applied first, followed by Add & Norm and the first dual FiLM conditioning layer, which modulates features using both style and BPM conditions. 2. The block performs multi-head cross-attention to the encoded memory, applies another Add & Norm operation, and introduces a second dual FiLM conditioning stage. 3. Finally, the features pass through a feed-forward network, apply a third Add & Norm step, and are processed by a third dual FiLM conditioning, ensuring comprehensive style and BPM modulation throughout the decoding. The decoder input is zero-initialized with a shape [B, Lt, dmodel], where Ltis the target sequence length, and enhanced with positional encoding before passing through the FiLM-conditioned layers, which attend to the morphed latent representation. Nine separate heads, one per codebook, compute logits independently, each of shape [B, Lt, V ]. 3.6. Experiments 43 3.6.4 Training Procedure Dataset Construction and Data Loading A paired dataset was constructed from DAC-encoded audio loops. Each training sample consisted of a source-target pair (xs, xt)where xsand xtwere different encoded audio loops, which enables the model to learn morphing transformations between different musical sequences. The dataset loader was implemented with robust handling of variable-length sequences through dynamic padding and sequence length tracking. For each batch, sequences were padded to the maximum length within the batch, and attention masks were computed to ensure proper handling of padded positions during training. Each sample contained: •Source and target DAC codes: xs,xt∈RK×L •Style probability vectors: σs, σt∈R400 •BPM values: bs, bt∈R •Actual sequence lengths: ls, lt∈N Curriculum Learning Strategy To archive more stability during training, a curriculum learning strategy was implemented to progressively increase the complexity of morphing ratios over time: αcurriculum(e) =              {0.0,1.0}if e E<0.3 {0.0,0.25,0.75,1.0}if 0.3≤e E<0.6 U(0,1) if e E≥0.6 (3.17) where edenotes the current epoch, Ethe total number of epochs, and U(0,1) is the uniform distribution over [0,1]. 44 Chapter 3. Methods This curriculum begins with extreme values of α(i.e., pure source or target reconstruction), then introduces intermediate ratios, and finally explores the full interpolation space. Loss Function Design Training was conducted using a simplified morphing loss function that directly supervises the model to reconstruct the target sequence, regardless of the morphing ratio: Lmorph =1 K K X k=1 LCE(ˆ yk,xt,k)(3.18) Where K= 9 denotes the number of codebooks, ˆ ykrepresents the predicted logits for the codebook k, and xt,k are the target sequence tokens for codebook k. The crossentropy loss LCE is computed with the masking to handle variable-length sequences: LCE(ˆ yk,xt,k) = −1 |M| X τ∈M log p(xτ t,k |ˆ yτ k)(3.19) where Mrepresents the set of valid (no-padded) time steps τbased on the target sequence length. The model learns to decode the interpolated representation, as described in Subsection 3.6.3, into coherent audio sequences through consistent supervision against the target sequence. The morphing behavior emerges implicitly from the latent space interpolation and the FiLM-based conditioning mechanism. Training Configuration The model was trained on 2010 audio files using the AdamW optimizer and the following hyperparameters: 3.6. Experiments 45 Parameter Value Learning rate 1×10−4 Weight decay 1×10−5 Batch size 4 Gradient accumulation steps 8 Effective batch size 32 Maximum epochs 300 Gradient clipping 0.5 Table 3: Training hyperparameters The OneCycleLR scheduler was employed with a peak learning rate reached at 10% of total training steps. Gradient accumulation was applied to simulate longer batch sizes while maintaining memory efficiency. Training Procedure The model was trained using mini-batches of paired source-target examples, following curriculum learning strategy for interpolation ratios (see Subsection 3.6.4). For each batch, source and target sequences were independently encoded, interpolated in the latent space according to the sampled ratio α, and decoded under target conditioning. A morphing loss Lmorph (see Subsection 3.6.4) was then computed against target tokens, averaged across codebooks with full masking variable-length handling. Optimization used gradient accumulation over Naccum steps and clipping for stability. The full training loop is summarized in Algorithm 1. 46 Chapter 3. Methods Algorithm 1 Training Loop 1: for each batch (xs, xt)in training set do 2: Sample interpolation factor α∼curriculum(e) 3: Encode source: zs←Encode(xs, σs, bs) 4: Encode target: zt←Encode(xt, σt, bt) 5: Interpolate representations: zmorph ←(1 −α)zs+αzt 6: Use target conditioning directly: cdecoder ←(σt, bt) 7: Decode output: ˆy ←Decode(zmorph,cdecoder) 8: Compute loss: L←Lmorph(ˆy, xt) 9: Normalize loss: L←L/Naccum and backpropagate 10: if Step % Naccum = 0 then 11: Clip gradient by norm (max=0.5), update parameters, reset gradients 12: end if 13: end for Validation and Early Stopping The model was evaluated across multiple fixed morphing ratios α∈ {0.0,0.25,0.5,0.75,1.0} to assess reconstruction quality at endpoints α= 0.0and α= 1.0, interpolation smoothness across intermediate values, and morphing effectiveness at the midpoint α= 0.5. Early stopping was triggered if validation loss did not improve for 15 consecutive epochs, preventing overfitting while ensuring convergence. Model Checkpointing During training, the best model based on validation loss was saved, along with periodic checkpoints every 10 epochs, a final model state at the end of training, and curriculum epoch metadata for resuming training. This strategy supported training resumption and helped model selection based on morphing performance across different interpolation ratios. 3.6. Experiments 47 3.6.5 Audio Morphing Inference System The inference system implements a neural audio morphing framework that operates on Discrete Audio Codec (DAC) representations. It enables controlled interpolation between the audio loops using the developed Conditioned Morpher Transformer model, allowing control over musical style, BPM, and structural characteristics. Input Processing Pipeline The inference system applies the same preprocessing pipeline used for training data preparation to ensure consistency between training and inference phases. Audio Preprocessing Given source and target audio files, the system applies preprocessing to ensure consistent format and temporal alignment, including resampling, truncating or padding. DAC Encoding The preprocessed audio is encoded using the Discrete Audio Codec to obtain quantized representations suitable for the model. Style and Tempo Feature Extraction The inference system applies the same style and tempo feature extraction methods described in Subsection 3.6.2. Musical style features are extracted using the MAEST model to generate 400-dimensional style activation tensors, while tempo information is obtained using Essentia’s RhythmExtractor2013 algorithm. These features are computed for both source and target audio files to provide the conditioning information required for the morphing process. Neural Morphing Architecture Conditioned Morpher Transformer model (CMT) At inference, the model takes source and target DAC codes, style and BPM tensors, and a morphing ratio. Optional custom style and BPM can also be provided, which are applied directly without interpolation. The CMT model generates logits for each codebook independently. 54 Chapter 4. Results 4.1.1 Agreement Scale Responses (Q1–Q3) Responses to the Agreement Scale, which assessed perceptual balance, coherence, and usability of the morphs, are presented in Table 5. Question Main response distribution Median Mode Q1 5.8% Neutral/Unsure, 59.3 % Agree, 34.9% Strongly Agree 4.0 4 Q2 7.0% Neutral/Unsure, 47.7% Agree, 45.3% Strongly Agree 4.0 4 Q3 62.8% Agree, 37.2% Strongly Agree 4.0 4 Table 5: Agreement Scale Responses (Q1–Q3). Participants expressed high levels of agreement across all three questions, The midpoint of the morph (50% blend) was generally perceived as a balanced combination of both original loops (Q1; 59.3% agreed, 34.9% strongly agreed, median = 4.0). Interpolated transitions were rated as musically coherent, with changes in rhythm, timbre, and structure perceived as making musical sense (Q2; 47.7% agreed, 45.3% strongly agreed, median = 4.0). Finally, the morphs were considered musically usable for applications such as seamless transitions, genre blending or timbre transformations (Q3; 62.8% agreed, 37.2 % strongly agreed, median = 4.0). The distribution of participant responses to the Agreement Scale questions is visualized in Figure 4. 4.1. 5-point Likert Scales 55 Figure 4: Percentage distributions of Likert-scale responses to Agreement Scale questions (Q1–Q3) 4.1.2 Intensity Scale Responses (Q4–Q5, Q8–Q9) The Intensity Scale, which measured how strongly participants perceived specific qualities or changes related to the morphing process, is presented in Table 6. Question Main response distribution Median Mode Q4 64.0% Strongly, 36.0% Very strongly 4.0 4 Q5 68.6% Strongly, 31.4% Very strongly 4.0 4 Q8 52.3% Not at all, 47.7% Slightly 1.0 1 Q9 58.1% Strongly, 41.9% Very strongly 4.0 4 Table 6: Intensity Scale responses (Q4–Q5, Q8–Q9). Participants consistently perceived the morphing as gradual (Q4; 64.0% strongly, 36.0% very strongly, median = 4.0). They also reported that the morph appeared to target specific musical features such as rhythm, timbre and texture (Q5; 68.6% 56 Chapter 4. Results strongly, 31.4% very strongly, median = 4.0). In contrast, changes in the number of latent dimensions were judged to have little impact on the smoothness or continuity of the morph (Q8; 52.3% not at all, 47.7% slightly, median = 1.0). However, variations in latent dimensions were perceived to strongly affect the expressiveness of the morph (Q9; 58.1% strongly, 41.9% very strongly, median = 4.0). The distribution of participant responses to the Intensity Scale questions is visualized in Figure 5. Figure 5: Percentage distributions of Likert-scale responses to Intensity Scale questions (Q4–Q5, Q8–Q9) 4.2 Identification of Musical Aspects for each Dimension Question 6 assessed which musical aspects –such as rhythm, timbre, and structure– participants perceived as most affected during the morphing transitions. The distribution of responses across the five dimensions is reported in Table 7. 4.2. Identification of Musical Aspects for each Dimension 57 Musical Feature Dim 1 Dim 2 Dim 3 Dim 4 Dim 5 Rhythm 20.5 % (84) 40.7 % (81) 48.8 % (78) 0.8% (1) – Timbre 20.8 % (85) 12.6 % (25) 50.6 % (81) 57.9 % (77) – Pitch/ Harmony 20.5 % (84) 0.5 % (1) – 39.8 % (53) – Structure/ Arrangement 19.6 % (80) 17.1 % (34) – – – Texture/ Layering 18.6 % (76) 29.1 % (58) – 1.5 % (2) – No noticeable change – – 0.6 % (1) – 100 % (85) Table 7: Distribution of participant responses across different dimensions and musical features, showing both percentages and counts. Overall, responses revealed distinct tendencies depending on the latent dimension. For Dimension 1, participants reported changes distributed fairly evenly across all features (≈20% each). Dimension 2 emphasized rhythm (40.7%) and texture/layering (29.1%). Dimension 3 was dominated by rhythm (48.8%) and timbre (50.6%). Dimension 4 showed strong emphasis on timbre (57.9%) and pitch/harmony (39.8%). Finally, dimension 5 was constantly perceived as having no noticeable change (100.0%). The distribution of responses per dimension is visualized in Figure 6. 58 Chapter 4. Results Figure 6: Participant-identified musical features most affected across dimensions (Q6). 4.3 Perceived Control: Uniform vs. Per-Dimension Blending Question 7 evaluated whether participants perceived per-dimension blending as providing greater precision and control compared to uniform blending. The distribution of responses is shown in Figure 7. Figure 7: Participant responses on perceived precision and control of per-dimension blending (Q7). All participants unanimously acknowledge that using a per-dimension ratio felt more precise and controlled than using a uniform ratio. Chapter 5 Discussion The findings from the subjective evaluation provide insight into the perceptual and musical validity of latent space interpolation for loop-based audio synthesis. With respect to perceptual blend and musical coherence RQ1, participants consistently rated the interpolated loop as perceptually meaningful and musically coherent. The midpoint morphs were perceived as balanced combinations of both source loops, and transitions were judged musically coherent, with changes in different musical aspects perceived as intentional and musically logical. Furthermore, analysis across latent dimensions revealed that specific dimensions corresponded to identifiable musical features, reinforcing the interpretability and musical validity of latent space morphing. In terms of usability and control RQ2, listeners reported that the interpolations were musically usable within compositional or performance contexts. The gradual transformations were reliable perceived and appeared to convey stylistic characteristics associated with different genres. Per-dimension blending was consistently judged as more precise and controllable than uniform blending. These observations suggest that latent space trajectories can function as expressive tools for style fusion and creative manipulation. Regarding latent space fidelity RQ3, varying the fidelity of the latent representa59 60 Chapter 5. Discussion tions was found to affect expressiveness more than perceptual continuity. Although transitions remained smooth even at lower fidelities, higher fidelity settings were consistently associated with greater richness and musical detail, highlighting a trade-off between compactness and perceptual richness in generative loop synthesis. Taken together, these results demonstrate that latent interpolation not only yields perceptually coherent blends but also affords practical usability in creative audio contexts. At the same time, they reveal that model fidelity can significantly shape the qualitative character of the outputs, being an important parameter for both design and evaluation of generative systems. 5.1 Conclusion This thesis developed a generative audio system for creative loop manipulation, demonstrating how latent space interpolation can be used to synthesize, transform, and blend audio loops. The system enables perceptual coherent and musically meaningful transitions, supporting applications such as mashups, stylistic blending and audio morphing. The evaluation confirmed that latent space interpolation offers perceptually coherent and musically meaningful transformations, showing its value as a tool for creative loop manipulation. Rather than focusing solely on synthesis quality, the system emphasizes controllability and expressiveness, aligning with the needs of musicians and producers in loop-based workflows. An additional exploratory experiment with a Transformer model conditioned on style and tempo (CMT) further highlighted the challenges of balancing controllability with synthesis quality. While style conditioning showed potential, tempo conditioning was less effective, and objective evaluation revealed that its acoustic, perceptual, and ability to represent musical structure did not match the RAVEbased system. Overall, this work confirms that neural generative models, when combined with latent space manipulation strategies, offer a practical and expressive approach for 5.2. Future Work 61 creative loop synthesis and transformation. 5.2 Future Work Future directions include the development of a plugin implementation system, which would make the tool accessible in common digital workstations (DAW) environments. This would allow musicians and producers to explore latent space interpolation directly within their workflows. In parallel, improvements to the Conditioned Morpher Transformer model will be pursued. In particular, refining the conditioning mechanisms for tempo and investigating alternative strategies to enhance both controllability and competitive performance. A more comprehensive evaluation, including subjective listening studies, will also be necessary to assess the perceptual impact of conditioning and to better understand its implications for creative applications. List of Figures 1 Median Activation of style probability vectors. . . . . . . . . . . . . . 23 2 PCA projection of 400-dimensional style probabilities across audio files. The first principal components account for 36.14% and 19.76% of the total variance, respectively. Colors indicate predicted parent genre. ................................... 24 3 Overview of the model. Source(s) and target(t) inputs (x) are encoded (E) into latent space representations and decoded (D) into output (ˆy). Source (zs) and target (zt) latent representations are interpolated using α∈[0,1] to produce zmorph. Conditioning vectors include cs= [bs, σs],ct= [bt, σt], and cdecoder = [bdecoder, σdecoder], with cdecoder ∈ {ccustom, ct}. Where σand brepresent the style and BPM vectors,respectively. ........................... 40 4 Percentage distributions of Likert-scale responses to Agreement Scale questions(Q1–Q3) ............................ 55 5 Percentage distributions of Likert-scale responses to Intensity Scale questions (Q4–Q5, Q8–Q9) . . . . . . . . . . . . . . . . . . . . . . . . 56 6 Participant-identified musical features most affected across dimensions(Q6).................................. 58 7 Participant responses on perceived precision and control of per-dimension blending(Q7)................................ 58 8 Modelarchitecture ............................ 73 62 List of Tables 1 Number of sub-genres (styles) associated with each parent genre. . . . 26 2 Number of files classified into each parent genre. . . . . . . . . . . . . 27 3 Training hyperparameters . . . . . . . . . . . . . . . . . . . . . . . . 45 4 FAD Evaluation Results: Comparison between RAVE model and the CMTmodel................................ 52 5 Agreement Scale Responses (Q1–Q3). . . . . . . . . . . . . . . . . . . 54 6 Intensity Scale responses (Q4–Q5, Q8–Q9). . . . . . . . . . . . . . . . 55 7 Distribution of participant responses across different dimensions and musical features, showing both percentages and counts. . . . . . . . . 57 63 70 Appendix A. Questionnaire for Subjective Evaluation 1. Does the midpoint of the morph (50% blend) sound like a perceptually balanced combination of both original loops? Scale: Strongly Disagree Disagree Neutral / Unsure Agree Strongly Agree 2. Do the interpolated transitions feel musically coherent — e.g., do changes in rhythm, timbre, and structure make musical sense and feel intentional? Scale: Strongly Disagree Disagree Neutral / Unsure Agree Strongly Agree 3. Does the interpolation between the two loops sound musically usable — for example, could it be used as a seamless transition, genre blend, or timbre transformation within a musical piece? Scale: Strongly Disagree Disagree Neutral / Unsure Agree Strongly Agree 4. Can you clearly perceive a gradual transformation in sound as the morph progresses from one loop (in a specific genre) to another? Scale: Not at all Slightly Moderately Strongly Very Strongly A.2 Per-Dimension Ratio Control in Latent Space In the second part of the demonstration video, titled "Per-Dimension Ratio Control in Latent Space", participants observed how each dimension was individually morphed within the Max/MSP environment. These dimensions were identifiable as the five separate wires coming out of the encode objects after linear interpolation, leading into the decoder. Participant were then asked to answer Questions 5-7 regarding the use of perdimension ratio control: 5. Do the changes you hear during the morph seem to target specific musical features (e.g., rhythm, timbre, texture)? Scale: Not at all Slightly Moderately Strongly Very Strongly A.3. Exploring Dimensionality in Latent Space 71 6. Can you identify which musical aspects (e.g., rhythm, timbre, structure) were most affected in the transition? Multiple choice per each dimension: •Rhythm — e.g., changes in beat, tempo, or groove •Timbre — e.g., the "color" or tone quality (bright, dark, buzzy, etc.) •Pitch / Harmony — e.g., melody shape, harmonic feel •Structure / Arrangement — e.g., buildup, breakdown, or change in loop form •Texture / Layering — e.g., thickness, number of layers or instruments •No noticeable change 7. Compared to uniform blending, did per-dimension blending feel more precise and controlled? Scale: Less controlled About the same More controlled A.3 Exploring Dimensionality in Latent Space In the final part of the demonstration video, titled "Exploring Dimensionality in Latent Space", participants observed how changing the number of dimensions affects the decoded audio. Participants were then asked to answer Questions 8 and 9 based on their observations: 8. When fewer or more latent dimensions are used (i.e., different fidelity settings), how strongly do you notice an effect on the smoothness or continuity of the audio morphing between sounds? Scale: Not at all Slightly Moderately Strongly Very Strongly 9. When fewer or more latent dimensions are used, how strongly do you notice an effect on the musical expressiveness or richness of the morphing? Scale: Not at all Slightly Moderately Strongly Very Strongly Appendix B Model Architecture This appendix presents a detailed diagram of the model architecture. The transformerbased encoder-decoder structure, the FiLM-based conditioning mechanism for style and BPM, and the feature interpolation process–including the layers in each block– are illustrated. Figure 8 provides a schematic representation of the architecture, showing the flow of data from input to output and the application of dual conditioning. 72 73 Figure 8: Model architecture Appendix C Materials for Reproducibility All code, demonstration videos, and materials necessary to reproduce the core RAVE-based morphing system are available in the project repository: https:// github.com/AdaSalvadorAvalos/freesound-loop-generator. This repository includes: •All scripts used for the ML pipeline •A pre-trained model checkpoint •The interactive Max/MSP patches •The demonstration video used in the subjective evaluation •A notebook for analyzing the evaluation results Additionally, a separate repository for the experimental approach is available at https://github.com/AdaSalvadorAvalos/conditioned-morpher-transformer-model, which contains: •All scripts used for the ML pipeline •A pre-trained model checkpoint 74 75 •Objective evaluation comparing the RAVE-based model with the developed model •The web interface