Improving the Semantic Structure of Neural Audio Codecs

Master thesis on Sound and Music Computing Universitat Pompeu Fabra Improving the Semantic Structure of Neural Audio Codecs Ángel Monsalve Fernández Supervisor: Dr. Lonce Wyse July 2025 Contents 1 Introduction 1 1.1 AudioRepresentations ........................... 1 1.1.1 Latent Spaces in Audio Representation Learning . . . . . . . . . . . . . 1 1.1.2 Semantic Structure in Audio Latent Spaces . . . . . . . . . . . . . . . . 3 1.2 NeuralAudioCodecs............................ 4 1.2.1 From Traditional to Neural Audio Codecs . . . . . . . . . . . . . . . . 4 1.2.2 Neural Audio Codecs as Audio Representation Learners . . . . . . . . . 5 1.2.3 TheSemanticGap ............................. 7 1.2.4 Emerging Semantic Codecs . . . . . . . . . . . . . . . . . . . . . . . . 8 1.2.5 ALMTokenizer: Enabling Query-Based Compression . . . . . . . . . . 9 1.3 Motivation.................................. 10 1.4 Objectives.................................. 11 2 Methods 12 2.1 Architecture................................. 12 2.1.1 Generator .................................. 12 2.1.2 Discriminator................................ 15 2.2 Losses .................................... 16 2.2.1 GeneratorLoss ............................... 16 2.2.2 DiscriminatorLoss ............................. 20 2.3 Training................................... 20 2.4 Datasets and Data Preprocessing . . . . . . . . . . . . . . . . . . . . . 21 2.5 Evaluation.................................. 21 2.5.1 Signal Reconstruction . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.5.2 SemanticStructure............................. 22 3 Results 26 3.1 Training................................... 26 3.2 Signal Reconstruction . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 3.3 SemanticStructure............................. 28 3.3.1 Projections ................................. 28 3.3.2 Clustering.................................. 30 3.3.3 Linear Separability Tests . . . . . . . . . . . . . . . . . . . . . . . . . . 31 3.3.4 Instrument Classification . . . . . . . . . . . . . . . . . . . . . . . . . . 32 3.3.5 Interpolation Smoothness Tests . . . . . . . . . . . . . . . . . . . . . . 32 3.3.6 Zero-shot Timbre Transfer . . . . . . . . . . . . . . . . . . . . . . . . . 33 4 Discussion 34 4.1 Discussion.................................. 34 4.2 FurtherWork................................ 35 4.3 Conclusions ................................. 37 List of Figures 39 List of Tables 40 Bibliography 41 Acknowledgement I would first like to express my deepest gratitude to my supervisor, Dr. Lonce Wyse, for his tireless guidance, patience, and support throughout this project. His dedication made it possible for me to bring this work to completion, and it has been a real pleasure to work under his supervision. I am also grateful to my former flatmates, Davide, Diego, and Mafe, for creating a space outside of academia where I could rest and recharge. Their friendship offered me the distance I needed to see the challenges of this project from a different perspective, and for that I owe them much. My heartfelt thanks go to Amalia, my partner and new flatmate, whose presence has given me direction and clarity. Her support has been a constant reminder of the path I want to follow in life. Finally, I would like to thank the entire Music Technology Group (MTG), and especially my classmates in the MSc in Sound and Music Computing, for filling this year with joy, discovery, knowledge, and culture. It has been truly comforting to share the journey with people who understand the struggle of listening to the same audio over and over again, waiting for something to change. Abstract Neural audio codecs have achieved remarkable compression efficiency by learning latent representations optimized for waveform fidelity. However, these codecs often lack explicit semantic structure, limiting their effectiveness for downstream tasks that require meaningful audio abstractions. Query-based compression, as introduced by ALMTokenizer, offers a path to infuse global context into discrete audio tokens by interleaving learnable [CLS] embeddings among frame-level features and leveraging Transformer attention to aggregate semantic information. This thesis implements a reproducible pipeline that adapts the ALMTokenizer paradigm using a frozen EnCodec front-end. By inserting one [CLS] query token every w frames, the model enables bitrate-on-demand through a tunable window length, while a Transformer encoder–decoder architecture captures long-range dependencies and reconstructs waveforms via a paired decoder. Quantization layers are omitted in this implementation to focus analysis on the raw contextual embeddings. To assess the semantic organization of the resulting latent space, we extract [CLS] embeddings from the Good-sounds dataset and perform an evaluation of the resulting latents. Our analyses show that although ALMTokenizer reconstructions lag behind EnCodec in perceptual quality, its embeddings exhibit stronger semantic organization. Clustering, projection, and classification experiments reveal clearer groupings by instrument, note, and octave, while interpolation suggests smoother latent transitions. This highlights a trade-off: EnCodec excels at fidelity, whereas ALMTokenizer provides embeddings better suited for semantic tasks. By releasing the implementation and methodology, this thesis offers a foundation for future research on semantically structured audio codecs. Keywords: neural audio codecs; semantic structure; query-based compression; audio representation learning; transformer Chapter 1 Introduction 1.1 Audio Representations 1.1.1 Latent Spaces in Audio Representation Learning When we hear a sound, our brain does not process the raw waveform sample by sample. Instead, it constructs an internal representation that captures the essential features of the auditory event and links them to our prior experiences and memories. This process, often referred to as auditory imagery or mental representation, allows us to recognize a voice in a crowd, identify a tune after a single note, or recall the emotion conveyed by a sound, all without attending to the physical waveform itself [1]. In machine learning, latent spaces aim to mirror this abstraction: they map highdimensional audio signals into a lower-dimensional embedding where semantically similar inputs lie close together. A latent space is the continuous, multidimensional feature space learned by the encoder of an autoencoder, variational autoencoder (VAE), or similar architecture. It compresses the data into a compact representation while preserving the information needed for reconstruction or downstream tasks [2]. In the context of audio, these embeddings may capture timbral qualities, phonetic content, or rhythmic patterns, depending on the training objective. 1 8Chapter 1. Introduction encode the linguistic content [28]. In essence, there is a growing realization that codec design matters for high-level audio tasks: the latent should ideally preserve semantic integrity, not just acoustic fidelity. 1.2.4 Emerging Semantic Codecs Several research efforts are now explicitly focusing on semantic audio codecs, aiming at altering the codec design or training to yield tokens that are both highly compressive and carry semantic meaning. A popular approach consists in using a two-stage encoding: first extract a highlevel representation and then an acoustic residual. For example, SemantiCodec uses a dual-encoder architecture: a semantic encoder (built on AudioMAE features) produces tokens capturing the semantic content, and an acoustic encoder captures the remaining low-level details [30]. The semantic encoder output is quantized using k-means codebooks derived from a large audio dataset, effectively clustering audio frames into discrete semantic units. Another example is MimiCodec, which targets ultra-low bitrate (≈1kbps) by distilling semantic knowledge from a pretrained model (WavLM) into the first quantization layer of a codec [22]. MimiCodec thus has a semantic codebook for the first layer (ensuring tokens correlate with speech content), followed by additional layers for refinement. However, MimiCodec was tailored for speech and required external supervision (knowledge distillation) to inject semantics. Another notable work, X-Codec, explicitly combines a HuBERT-based semantic encoder with an acoustic encoder, and introduces a semantic reconstruction loss in training [31]. By quantizing after merging those features, X-Codec produces tokens that significantly improve phonetic discriminability (measured via ABX tests) and downstream speech generation word error rates. In a very recent work, ALMTokenizer pushes this idea further to support both speech and general audio [23]. It introduces a Transformer-based encoder that compresses a sequence of audio frames into a smaller set of tokens by attending over 1.2. Neural Audio Codecs 9 context, rather than treating each frame independently. This query-based compression allows the model to pick up longer patterns and encode them with fewer tokens, thereby encoding more semantic context per token. In the following section, we will dive deep into this novel semantic codec architecture. 1.2.5 ALMTokenizer: Enabling Query-Based Compression ALMTokenizer is a neural audio codec tokenizer designed to produce discrete, semantically rich audio tokens at extremely low bitrates. Rather than quantizing every frame, it interleaves a small number of learnable [CLS] query embeddings into a Transformer-based encoder–decoder architecture. These query tokens attend over windows of raw audio frame embeddings, aggregating contextual information before being quantized via residual vector quantization (RVQ) and decoded back into waveform through a paired decoder Query-based compression, the core innovation of ALMTokenizer, replaces per-frame quantization with a mechanism in which only these sparse, context-aggregating queries are retained. By inserting one [CLS] token every wframes and discarding the intermediate frame slots after encoding, the model compresses a long sequence of frames into a much shorter sequence of queries. This approach harnesses the Transformer’s self-attention to capture long-range dependencies, enabling each token to summarize dozens of milliseconds of audio content in a single vector. This paradigm delivers several key advantages. First, it achieves bitrate ondemand by simply tuning the window length w: fewer queries mean fewer tokens per second and hence lower bitrate, while more queries recover finer temporal detail. Second, because each query token attends to its neighbors, the resulting tokens carry global semantic context that per-frame quantization often misses. Third, by compressing context into fewer tokens, ALMTokenizer produces much shorter sequences, alleviating the computational burden on downstream sequence models and improving long-term coherence for generative tasks. Empirical evaluations confirm that ALMTokenizer’s query-based compression de- 10 Chapter 1. Introduction livers perceptual quality comparable to much higher-bitrate neural codecs while operating at only a fraction of their token rates. Listeners consistently judge its reconstructions as nearly indistinguishable from richer references, and objective intelligibility and fidelity metrics place it at the forefront of ultra-low-bitrate models. Crucially, the semantic density of its [CLS] tokens translates into real gains on downstream tasks: speech recognition error rates drop, and emotion or speakeridentification systems perform more accurately when fed ALMTokenizer embeddings [23]. 1.3 Motivation The motivation for this work stems from both practical and theoretical gaps in current neural audio codec research. First, although the ALMTokenizer paper introduces a powerful query-based compression paradigm that promises bitrate ondemand and richer semantic content, the authors have not released their implementation alongside the preprint, limiting reproducibility and broader community adoption. Reproducing and extending such a model is therefore essential to validate its claims, explore its design choices, and integrate its innovations into downstream audio-language applications. Beyond codec design, there is a pressing need to reconcile compression efficiency with the sequence-modeling demands of audio-language Transformers. High token rates (e.g., 150 tokens/s in EnCodec) provide fine-grained acoustic detail but lead to prohibitively long sequences for autoregressive or self-attention models, restricting their ability to capture long-term structure. Conversely, extreme compression can degrade perceptual quality. A semantically structured latent space at moderate token rates could enable Transformers to model content over extended contexts more effectively, improving tasks like speech recognition, emotion classification, and audio generation. This work explores semantic compression by reproducing and simplifying the querybased framework of ALMTokenizer and evaluating its semantic properties. The goal 1.4. Objectives 11 is to contribute open, reproducible methods and insights that could support the development of future audio codecs with improved bitrate efficiency and semantic clarity. 1.4 Objectives The primary aim of this thesis is to investigate and enhance the semantic organization of neural audio codec latents, focusing on the novel query-based compression approach. To this end, we pursue the following two specific objectives: 1. Reproduce and adapt the ALMTokenizer paradigm: we will implement a query-based compression pipeline by interleaving learnable [CLS] query tokens into the latent stream of a frozen EnCodec front-end, enabling bitrate on-demand through a tunable window length. 2. Characterize the semantic structure of the resulting latent space: we will extract [CLS] embeddings from the Good-sounds dataset and compare them against baseline EnCodec latents via (a) projections, (b) unsupervised clustering and projections, (c) linear separability tests, (d) supervised instrument classification, (e) interpolation smoothness assessments and (f) timbre transfer capabilities. Chapter 2 Methods 2.1 Architecture This section details the dual-module architecture of our adversarial audio codec, which consists of a generator that compresses and reconstructs waveforms and an ensemble of discriminators that provide multiscale perceptual feedback. A schema of the whole architecture, including both the generator and the discriminator is presented in Figure 2. 2.1.1 Generator The generator part of our custom ALMTokenizer implementation is composed of the following modules: Patchify and Unpatchify (encoder/decoder frontends), [CLS] tokens interleaving and retrieval functions, Transformer modules and mask tokens interleaving and retrieval functions. Patchify and Unpatchify Modules The Patchify and Unpatchify modules serve as the encoder and decoder front-ends, effectively transforming raw audio signals to vector representations (z). Our model uses the pretrained 24 kHz EnCodec encoder as the Patchify module and its corresponding decoder as Unpatchify, both frozen during training to ensure high-fidelity 12 2.1. Architecture 13 Figure 2: Schematic of the ALMTokenizer generator (top) and discriminator (bottom). The discriminator schema is taken from the EnCodec paper [17]. waveform processing without additional front-end optimization. The encoder transforms the raw audio waveform into a sequence of frame embeddings via successive one-dimensional strided convolutions, producing a tensor of shape T×dwhere T is the number of frames and dthe embedding dimension (in our case, d= 128). Unpatchify mirrors this process with transposed convolutions that reconstruct the waveform from decoded embeddings, guaranteeing convertibility between time and latent domains. 14 Chapter 2. Methods [CLS] Token Interleaving and Retrieval A custom routine interleaves a learnable [CLS] token immediately after every w frames, where wis the chosen window size. By varying w, it is possible to control how many [CLS] tokens are inserted, and thus the effective bitrate, enabling true bitrate on-demand. Once these tokens pass through the transformer encoder (described below), each [CLS] will gather context from the surrounding frame embeddings through self-attention. After passing through the transformer encoder, the contextrich [CLS] tokens are retrieved from the sequence, to form the new semantic-rich latent frames. During training, wis randomly chosen for each batch, taking values in the range from 2 to 10. This ensures better generalization for different values of w. During evaluation, we use w= 3, so that after every three frame vectors produced by the Patchify module, one learnable [CLS] token is inserted. Transformer Modules Once the [CLS] tokens have been interleaved among the frame embeddings, the combined sequence is fed into our transformer encoder, which is responsible for infusing each [CLS] vector with contextual information from its neighbors. We employ rotary positional embeddings (RoPE) [32] to encode temporal order, and restrict the attention mechanism to a causal sliding window with a default size of 16 frames (approximately 213 ms, given 75 frames per second). This design enforces local temporal dependencies while keeping computation efficient. During decoding, the retrieved [CLS] queries are merged with learnable mask tokens (described below) and fed into a symmetric transformer decoder, which re-expands the sequence back to its original length before Unpatchify restores the waveform. Both the transformer encoder and decoder consist of 12 identical layers, each featuring 32 attention heads and a feed-forward network of 256 and 512 dimensions for the encoder and decoder respectively. 2.1. Architecture 15 Mask Tokens Interleaving and Retrieval During decoding, we reinstate the original token positions by inserting a single learnable mask embedding into each frame slot that was dropped during compression and placing the retrieved [CLS] tokens back in their query locations. This interleaved sequence is then fed into the transformer decoder, where the mask tokens signal gaps to be filled using the contextual information carried by the [CLS] vectors. After decoding, only the frame-position outputs proceed to Unpatchify, while the mask and [CLS] slots are discarded. Quantization The original ALMTokenizer implementation employs three layers of residual vector quantization (RVQ) to quantize each query embedding in sequence. Each layer has its own codebook, half of which is initialized using k-means centroids derived from wav2vec 2.0 features for speech and the other half from BEATs features for general sounds. In our implementation, we eliminate quantization entirely. Our goal is not to minimize the bitrate of the compressed signal but to explore the latent space generated by the model. Accordingly, we feed the raw [CLS] tokens directly, without any quantization step. This choice simplifies training by removing both the RVQ module and the autoregressive (AR) loss. 2.1.2 Discriminator Discrimination is performed using the same multi-scale STFT discriminator architecture introduced in EnCodec, which we adopt directly from their official implementation [17]. We use four discriminators, each operating on spectrograms at different resolutions. Specifically, the STFTs are computed with FFT sizes of 256, 512, 1024, and 2048, with matching window lengths and hop sizes of 64, 128, 256, and 512 samples, respectively. This multi-scale configuration allows the discriminators to capture both fine and coarse temporal structures in the audio. 16 Chapter 2. Methods Each discriminator follows the EnCodec design: the complex STFT (real and imaginary parts concatenated) is processed through a stack of 2D convolutional layers, starting with a 3x9 convolution with 32 filters, and continuing with progressively deeper convolutions that incorporate strided downsampling along the frequency axis and dilations of 1, 2, and 4 along the time axis. All layers use LeakyReLU activations and weight normalization, and the network concludes with a 3x3 convolution to produce the discriminator logits. 2.2 Losses In this subsection we introduce the loss terms that guide the training of the model. We define the time-domain reconstruction loss, the frequency-domain spectral loss, the adversarial loss and the feature-matching loss. Figure 3 illustrates the set of loss functions that is computed for the ALMTokenizer model. 2.2.1 Generator Loss The generator features a composite loss that combines four terms that guide the model toward perceptually realistic and semantically rich reconstructions. Reconstruction Loss Reconstruction error quantifies the discrepancy between the original and reconstructed audio by combining two complementary measures. First, we compute the time-domain loss as the pointwise L1distance between the original waveform (x) and its reconstruction (ˆx): Ltime = x−ˆx 1 Second, we form the frequency-domain loss using a multiscale mel-spectrogram criterion. Both the original signal and the reconstruction are transformed into melspectrograms at multiple resolutions. At every scale, we compute (1) the mean absolute error between the mel-spectrogram magnitudes, and (2) the RMSE error 2.2. Losses 17 Figure 3: Schematic of the losses computed during training. All the losses are shown within a red box. 24 Chapter 2. Methods We deliberately use a very high C parameter in the SVC (C= 107), ensuring that the classifier strongly penalizes misclassifications and fits the data as closely as possible, so that the reported accuracy reflects the intrinsic linear separability of the embeddings rather than the effect of strong regularization. Instrument Classification To further test the representational power of the latents, we train a small random forest classifier (RF) with 100 estimators on the embeddings. By comparing downstream performance when using our query-based [CLS] tokens versus raw EnCodec frame vectors (without additional processing), we can measure how semantic structuring in the latent space translates into concrete gains on a classification task. To assess such performance, we compute and compare the accuracy and precision of both instances of the classifier. Interpolation Smoothness Tests We investigate the interpolation behavior between latent codes from different instruments. Specifically, we linearly interpolate between the two [CLS] vectors representing the centroids of all samples belonging to a given instrument, note, or octave. Each intermediate point is then decoded back to audio, allowing us to listen to the resulting sequence. In a semantically structured latent space, these traversals should produce perceptually smooth transitions, with timbre and character morphing gradually. By contrast, an unstructured space would yield abrupt or incoherent changes. While this test is inherently subjective, we provide exemplary audio snippets in our project repository so that readers can evaluate the continuity and plausibility of these interpolations themselves. Although Good-Sounds contains carefully recorded samples, an instrument’s timbre can vary significantly over the course of a note. To reduce artifacts in the generated audio, we compute centroids only from the most stable portions of the signal. For sounds with available annotations, we used the segment spanning from the annotated attack or decay to the annotated release or offset. Because no annotations were 2.5. Evaluation 25 provided for the onset of the sustain phase, we relied on these available boundaries instead. Under these criteria, only the flute, clarinet, and trumpet recordings were suitable for centroid construction. Zero-shot Timbre Transfer Finally, we explore whether moving the latent representation of an audio towards that of a target instrument can effectively change its timbre while preserving pitch and rhythm. The procedure starts by encoding an input sound into latent frames, both with EnCodec and with our ALMTokenizer. From these frames, we compute the centroid of the input audio in latent space. We also compute the centroid of the target instrument by averaging all latent frames corresponding to the stable part of that instrument in the Good-Sounds dataset. The difference between these two centroids defines a timbre direction. Once we have this direction, we shift all latent frames of the input audio along it until their average aligns with the centroid of the target instrument. After the shift, the modified latents are decoded back into audio. We will evaluate the results by listening and assessing whether the transformed audio contains characteristic elements of the target instrument while still preserving the pitch and timing of the original audio. Chapter 3 Results The code used for implementing the architecture and characterizing its output has been made available at https://github.com/angelmf97/almtokenizer 3.1 Training In Figure 4, we plot the evolution of each loss term over the first 300 training steps. 0.150 0.155 0.160 0.165 Value L_time 0.32 0.33 0.34 0.35 0.36 0.37 0.38 Value L_freq 1.0 1.2 1.4 1.6 Value L_adv 0.175 0.200 0.225 0.250 0.275 0.300 Value L_feat 0 10 20 30 40 50 60 70 Epoch 0.25 0.50 0.75 1.00 1.25 1.50 1.75 Value L_mae 0 10 20 30 40 50 60 70 Epoch 2.50 2.75 3.00 3.25 3.50 3.75 Value L_total 0 10 20 30 40 50 60 70 Epoch 1.93 1.94 1.95 1.96 1.97 1.98 Value L_disc Train Test Training Metrics Figure 4: Loss curves for the train (blue) and test dataset (orange). Overall, the total loss of the generator (L_total steadily declines, demonstrating that the model is converging. Looking more closely at the reconstruction losses, we see a pronounced drop in the frequency-domain spectral loss (L_freq). This indicates that the spectrograms of the reconstructed audio are becoming ever more similar to those 26 3.2. Signal Reconstruction 27 of the original signal. By contrast, the time-domain L1 loss (L_time) increases during the first stages of training. This rise is not inherently problematic: time-domain error measures mathematical discrepancies between waveforms, and two signals can be perceptually identical despite substantial sample-by-sample differences, and the discriminator is agnostic to these differences, since it only evaluates spectrograms. The MAE loss decays smoothly and consistently, showing that the transformer encoder in the generator is effectively embedding contextual information into each frame. Even when portions of the input are masked, the model learns to reconstruct them accurately. During training, the adversarial loss, discriminator loss, and feature-matching loss reach a steady equilibrium. All three remain stable, oscillating only within a narrow range, which suggests that the generator and discriminator are evenly matched and that neither dominates the other. This implies that, even as the discriminators increase their strength, the generator learns to produce outputs that better align with their internal feature representations, pointing to an improvement in perceptual quality. 3.2 Signal Reconstruction We provide audio examples comparing reconstructed signals (after encoding and decoding) with their original counterparts at https://angelmf97.github.io/almtokenizer/ The quality of the reconstructions is far from optimal. The generated audios exhibit a metallic character and audible artifacts, which shows that the model is not yet capable of reproducing the full richness of the original signals. Compared to stateof-the-art codecs such as EnCodec, the gap in perceptual quality is clear. However, it is important to emphasize that the reconstructed sounds remain perfectly recognizable, and in the case of speech samples the content is intelligible. Achieving this level of reconstruction with a relatively small dataset and under limited computational resources represents a significant accomplishment and provides a solid basis for further improvement. 28 Chapter 3. Results 3.3 Semantic Structure 3.3.1 Projections Figure 5 presents the t-SNE projections of the embeddings produced by EnCodec and ALMTokenizer for the Good-Sounds dataset. We omit the legend, focusing instead on the overall structure and grouping of the embeddings rather than individual labels. Visually, the clusters formed by ALMTokenizer embeddings are noticeably tighter than those of EnCodec across all three semantic attributes examined (instrument, note, and octave). This tighter grouping suggests that the latent space learned by ALMTokenizer encodes semantic information more effectively than traditional neural codecs, which may lead to improved performance in downstream tasks that depend on semantic distinctions. In particular, the space shows a clearer organization by pitch, as the naturally 20 0 20 40 t-SNE 1 40 20 0 20 40 t-SNE 2 instrument 20 0 20 40 t-SNE 1 40 20 0 20 40 t-SNE 2 note 20 0 20 40 t-SNE 1 40 20 0 20 40 t-SNE 2 octave EnCodec 60 40 20 0 20 40 t-SNE 1 40 20 0 20 40 t-SNE 2 instrument 60 40 20 0 20 40 t-SNE 1 40 20 0 20 40 t-SNE 2 note 60 40 20 0 20 40 t-SNE 1 40 20 0 20 40 t-SNE 2 octave ALMTokenizer Figure 5: t-SNEs of the ground truth labels of the embeddings generated by EnCodec (top) and ALMTokenizer (bottom). 3.3. Semantic Structure 29 emerging clusters align better with note and octave labels, suggesting that pitch is a key factor structuring the latent space. While t-SNE can be a useful tool, it is also prone to producing misleading impressions of the data. To obtain a more reliable view of class separability, we instead employ Linear Discriminant Analysis (LDA), which projects the samples onto the directions that maximize variability between groups (i.e., those that best separate instruments, notes, or octaves). Figure 6 shows the LDA projections for these three label types, comparing ALMTokenizer with EnCodec. In ALMTokenizer, the groups appear slightly more distinctly separated. This suggests that the class boundaries in ALMTokenizer are more linear and better defined. 7.5 5.0 2.5 0.0 2.5 5.0 LDA 1 10.0 7.5 5.0 2.5 0.0 2.5 5.0 LDA 2 instrument 4202468 LDA 1 6 4 2 0 2 4 6 LDA 2 note 420246810 LDA 1 6 4 2 0 2 4 6 LDA 2 octave EnCodec 642024 LDA 1 6 4 2 0 2 4 6 LDA 2 instrument 420246 LDA 1 4 2 0 2 4 6 LDA 2 note 420246 LDA 1 4 2 0 2 4 6 LDA 2 octave ALMTokenizer Figure 6: LDAs of the ground truth labels of the embeddings generated by EnCodec (top) and ALMTokenizer (bottom). 30 Chapter 3. Results 3.3.2 Clustering We explore a range of values for kto identify the number of clusters that best aligns with the instrument labels. For each k, we compute the Akaike Information Criterion (AIC). Figure 7 reports the value of this metric across different choices of kfor both EnCodec and ALMTokenizer. Because AIC penalizes model complexity, the curves typically descend and then rise as kincreases, helping avoid overfitting. The optimal kis 34 and 44 for EnCodec and ALMTokenizer, respectively. Figure 7 also displays the same t-SNE projections as before, now colored by the cluster assignments obtained with the optimal number of clusters. The projection confirms better defined clusters in our model. Then, we compute the external validation metrics. The Adjusted Rand Index (ARI) measures how well the predicted clusters match the true labels, correcting for chance. The Normalized Mutual Information quantifies the amount of shared information between the clustering and the ground truth. Finally, the Homogeneity Score evaluates whether each cluster contains only members of a single class. Table 2 reports the values obtained for each of the three chosen metrics. ALMTokenizer consistently outperforms EnCodec across all of them. These results indicate that the latent space of ALMTokenizer is better organized with respect to the semantic structure of instruments, notes and octaves than that of EnCodec. Feature EnCodec ALMTokenizer Adjusted Rand Index (↑) 0.011 0.029 Normalized Mutual Information Score (↑) 0.268 0.411 Homogeneity Score (↑) 0.206 0.326 Table 2: External clustering evaluation metrics for EnCodec embeddings vs. ALMTokenizer embeddings. Bold for the best result in each feature. 3.3. Semantic Structure 31 EnCodec 20 25 30 35 40 45 50 55 60 Number of clusters 800000 750000 700000 650000 600000 550000 500000 AIC Model Selection: EnCodec ALMTokenizer 20 25 30 35 40 45 50 55 60 Number of clusters 950000 900000 850000 800000 750000 700000 650000 600000 550000 AIC Model Selection: ALMTokenizer 40 20 0 20 40 t-SNE 1 40 20 0 20 40 t-SNE 2 cluster 60 40 20 0 20 40 60 t-SNE 1 40 20 0 20 40 t-SNE 2 cluster Figure 7: Plot of the AICs computed at each value of k(top) and t-SNEs of the clustering of the embeddings generated by EnCodec and ALMTokenizer (bottom). 3.3.3 Linear Separability Tests Table 3 reports the raw accuracies achieved by an SVM classifier with a linear kernel on EnCodec and ALMTokenizer embeddings for three semantic attributes: instrument, note and octave. ALMTokenizer outperforms EnCodec in every case, reaching 0.32 versus 0.44 for instrument, 0.15 versus 0.30 for note and 0.30 versus 0.34 for octave. These results indicate that ALMTokenizer embeddings capture semantic distinctions more effectively than traditional neural codecs, yielding more discriminative representations for downstream classification tasks. 32 Chapter 3. Results Feature EnCodec ALMTokenizer Accuracy on Instrument (↑) 0.32 0.44 Accuracy on Note (↑) 0.15 0.30 Accuracy on Octave (↑) 0.30 0.34 Table 3: Raw accuracies obtained by the SVM classifier for EnCodec embeddings vs. ALMTokenizer embeddings. Bold for the best result in each feature. 3.3.4 Instrument Classification Table 4 summarizes the accuracy and precision obtained from training a random forest classifier to categorize samples by instrument, note, and octave. In all three cases, ALMTokenizer outperforms the baseline, achieving consistently higher values for both metrics. This suggests that the representations learned by ALMTokenizer capture musically relevant features and can be more useful for downstream MIR tasks such as instrument classification. Feature Model Accuracy (↑) Precision (↑) Instrument EnCodec 0.51 0.53 ALMTokenizer 0.67 0.70 Note EnCodec 0.46 0.47 ALMTokenizer 0.60 0.61 Octave EnCodec 0.64 0.68 ALMTokenizer 0.74 0.77 Table 4: Raw accuracies and precisions obtained by the Random Forest classifier for EnCodec embeddings vs. ALMTokenizer embeddings. Bold for the best result in each feature. 3.3.5 Interpolation Smoothness Tests We included a series of examples showcasing sound interpolation (or morphing) at https://angelmf97.github.io/almtokenizer/. The procedure consists of selecting two groups of sounds, computing their centroids, and then generating interpolated vectors between those centroids. These interpolated vectors are subsequently passed through EnCodec and ALMTokenizer for audio synthesis. 3.3. Semantic Structure 33 Upon listening, we observed that ALMTokenizer produces audio of lower perceptual quality compared to EnCodec. However, it appears to capture the semantic structure of the sounds more effectively. In particular, in interpolation cases where pitch is involved, the audio generated by ALMTokenizer clearly transitions through intermediate pitches along the path, while EnCodec does not exhibit this behavior. 3.3.6 Zero-shot Timbre Transfer The experiments on zero-shot timbre transfer can be found at the following page: https://angelmf97.github.io/almtokenizer/ In these tests, an input sound is shifted in latent space toward the centroid of a target instrument, and then decoded back to audio. The resulting examples make clear that, although the overall reconstruction quality is still limited and artifacts remain audible, the latent space learned by ALMTokenizer encodes semantic structure more effectively than EnCodec. This stronger organization allows the transferred sounds to convey a clearer sense of the intended target timbre, making the transformation feel more purposeful and consistent, even if it is still far from musically convincing. List of Tables 1 Comparison of frames per second (FPS), tokens per second (TPS), codebook size (CS) and bitrate (BR) across models. Adapted from [23] 5 2 External clustering evaluation metrics for EnCodec embeddings vs. ALMTokenizer embeddings. Bold for the best result in each feature. 30 3 Raw accuracies obtained by the SVM classifier for EnCodec embeddings vs. ALMTokenizer embeddings. Bold for the best result in eachfeature................................. 32 4 Raw accuracies and precisions obtained by the Random Forest classifier for EnCodec embeddings vs. ALMTokenizer embeddings. Bold for the best result in each feature. . . . . . . . . . . . . . . . . . . . . 32 40 Bibliography [1] Hubbard, T. L. Auditory imagery: Empirical findings. Psychological Bulletin 136 (2010). [2] Bergmann, D. What is latent space? (2025). URL https://www.ibm.com/ think/topics/latent-space. [3] Baevski, A., Zhou, H., Mohamed, A. & Auli, M. wav2vec 2.0: A framework for self-supervised learning of speech representations (2020). URL http://arxiv. org/abs/2006.11477. [4] Hsu, W.-N. et al. Hubert: Self-supervised speech representation learning by masked prediction of hidden units (2021). URL http://arxiv.org/abs/2106. 07447. [5] Huang, P.-Y. et al. Masked autoencoders that listen (2023). URL http:// arxiv.org/abs/2207.06405. [6] Liu, H. et al. Audioldm: Text-to-audio generation with latent diffusion models (2023). URL http://arxiv.org/abs/2301.12503. [7] Nakashima, R., Ozaki, R. & Taniguchi, T. Unsupervised phoneme and word discovery from multiple speakers using double articulation analyzer and neural network with parametric bias. Frontiers in Robotics and AI 6(2019). [8] Hawley, S. H. & Tackett, A. R. Operational latent spaces. Journal of Audio Engineering Society (2024). 41 42 BIBLIOGRAPHY [9] Lu, H. et al. Disentangled speech representation learning for one-shot crosslingual voice conversion using β-vae (2022). URL http://arxiv.org/abs/ 2210.13771. [10] Wyse, L., Kamath, P. & Gupta, C. Sound model factory: An integrated system architecture for generative audio modelling (2022). URL http://arxiv.org/ abs/2206.13085. [11] García, H. F., Nieto, O., Salamon, J., Pardo, B. & Seetharaman, P. Sketch2sound: Controllable audio generation via time-varying signals and sonic imitations (2025). URL http://arxiv.org/abs/2412.08550. [12] Strawn, J. & Pohlmann, K. C. Principles of digital audio. Computer Music Journal 10 (1986). [13] Smith, J. O. & Abel, J. S. Iso11172-3: Information technology - coding of moving pictures and associated audio for digital storage media at up to about 1.5 mbit/s - part 3: Audio. ISEJTC 129 WG 11 (1993). [14] Valin, J., Vos, K. & Terriberry, T. Definition of the opus audio codec. Internet Engineering Task Force (IETF) (2012). [15] Wu, H. et al. Towards audio language modeling – an overview (2024). URL http://arxiv.org/abs/2402.13236. [16] Zeghidour, N., Luebs, A., Omran, A., Skoglund, J. & Tagliasacchi, M. Soundstream: An end-to-end neural audio codec (2021). URL http://arxiv.org/ abs/2107.03312. [17] Défossez, A., Copet, J., Synnaeve, G. & Adi, Y. High fidelity neural audio compression (2022). URL http://arxiv.org/abs/2210.13438. [18] Kumar, R., Seetharaman, P., Luebs, A., Kumar, I. & Kumar, K. High-fidelity audio compression with improved rvqgan (2023). URL http://arxiv.org/ abs/2306.06546. BIBLIOGRAPHY 43 [19] Ji, S. et al. Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling (2025). URL http://arxiv.org/abs/2408.16532. [20] Parker, J. D. et al. Scaling transformers for low-bitrate high-quality speech coding (2024). URL http://arxiv.org/abs/2411.19842. [21] Zhang, X., Zhang, D., Li, S., Zhou, Y. & Qiu, X. Speechtokenizer: Unified speech tokenizer for speech large language models (2024). URL http://arxiv. org/abs/2308.16692. [22] Défossez, A. et al. Moshi: a speech-text foundation model for real-time dialogue (2024). URL http://arxiv.org/abs/2410.00037. [23] Yang, D. et al. Almtokenizer: A low-bitrate and semantic-rich audio codec tokenizer for audio language modeling (2025). URL http://arxiv.org/abs/ 2504.10344. [24] Juang, B. H. & Gray, A. H. Multiple stage vector quantization for speech coding. In ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, vol. 1982-May (1982). [25] Borsos, Z. et al. Audiolm: a language modeling approach to audio generation (2023). URL http://arxiv.org/abs/2209.03143. [26] Agostinelli, A. et al. Musiclm: Generating music from text (2023). URL http: //arxiv.org/abs/2301.11325. [27] Borsos, Z. et al. Soundstorm: Efficient parallel audio generation (2023). URL http://arxiv.org/abs/2305.09636. [28] Wang, C. et al. Neural codec language models are zero-shot text to speech synthesizers (2023). URL http://arxiv.org/abs/2301.02111. [29] Sun, S., Krishna, K., Mattarella-Micke, A. & Iyyer, M. Do long-range language models actually use long-range context? (2021). URL http://arxiv.org/ abs/2109.09115. 44 BIBLIOGRAPHY [30] Liu, H. et al. Semanticodec: An ultra low bitrate semantic audio codec for general sound (2024). URL http://arxiv.org/abs/2405.00233http://dx. doi.org/10.1109/JSTSP.2024.3506286. [31] Ye, Z. et al. Codec does matter: Exploring the semantic shortcoming of codec for audio language model (2024). URL http://arxiv.org/abs/2408.17175. [32] Su, J. et al. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing 568 (2024). [33] Fonseca, E., Favory, X., Pons, J., Font, F. & Serra, X. Fsd50k: An open dataset of human-labeled sound events. IEEE/ACM Transactions on Audio Speech and Language Processing 30 (2022). [34] Picas, O. R., Rodriguez, H. P., Dabiri, D. & Serra, X. Good-Sounds Dataset (2017). URL https://zenodo.org/record/820937. [35] Maaten, L. V. D. & Hinton, G. Visualizing data using t-sne. Tech. Rep. (2008). [36] FISHER, R. A. The use of multiple measurements in taxonomic problems. Annals of Eugenics 7(1936). [37] Rao, C. R. The utilization of multiple measurements in problems of biological classification. Journal of the Royal Statistical Society Series B: Statistical Methodology 10 (1948). [38] Elizalde, B., Deshmukh, S., Ismail, M. A. & Wang, H. Clap: Learning audio concepts from natural language supervision (2022). URL http://arxiv.org/ abs/2206.04769. [39] Alonso-Jiménez, P., Serra, X. & Bogdanov, D. Efficient supervised training of audio transformers for music representation learning (2023). URL https: //arxiv.org/abs/2309.16418.2309.16418.