Blaming the Algorithm: An Analysis of Interpolation Failures in MusicVAE's Latent Space

Blaming the Algorithm: An Analysis of Interpolation Failures in MusicVAE’s Latent Space⋆ Weixi Zhai1,* 1Quanzhou Normal University, 398 Donghai Street, Fengze District, Quanzhou, Fujian 362000, China Abstract Inspired by Der Greif magazine’s Issue 12: Blame the Algorithm, which critiques algorithmic censorship and highlights the limitations of algorithms in content moderation, this study examines the implicit aesthetic biases in AI music systems by analyzing MusicVAE’s treatment of experimental musical content. We extracted and paired melody tracks from the POP909 dataset: 909 conventional pop melodies and 909 systematically "glitched" counterparts featuring note deletions, extreme pitch manipulations, and temporal disruptions. Despite comprehensive full-parameter fine-tuning on a balanced dataset, the system consistently failed to process the experimental sequences, rejecting them at multiple architectural stages. Our analysis revealed four distinct failure modes: representational filtering during data conversion, architectural incompatibility in sequence processing, optimization penalties against irregular patterns, and generation collapse in interpolation tasks. These findings demonstrate that contemporary AI music systems operate within implicit aesthetic boundaries that categorically exclude certain forms of musical expression. This study contributes to human-centered Music Information Research (MIR) by exposing how algorithmic design choices can marginalize experimental practices and by questioning whose creativity gets computational support. We argue for more inclusive generative models that can embrace diverse musical expressions beyond conventional pop structures. Keywords AI creativity limitations, MusicVAE, Algorithmic bias, Experimental music generation, Creative boundaries 1. Introduction AI music generation has rapidly evolved in recent years, but remains largely dominated by prompt-driven and inspiration-based creativity. Most widely adopted systems rely on Transformers [ 1 , 2 , 3 ], diffusion models [ 4 , 5 , 6 , 7 ], and flow-based approaches [ 8 ], prioritizing fidelity, controllability, and multimodal conditioning. Applications such as MusicGen[ 9 ], YuE[ 10 ], ChatMusician[ 11 ], and Amuse[ 12 ] illustrate related trends, from text-to-music generation to cross-modal inspirations from text, images, and video. Together, these systems highlight how generative models are increasingly designed as partners in creative workflows through prompt-based interaction. Beyond prompt-based paradigms, interpolation in latent space offers an alternative mode of exploration, enabling musicians to traverse between melodies or styles and discover unexpected HCMIR25: 3rd Workshop on Human-Centric Music Information Research, September 20th, 2025, Daejeon, Korea *Corresponding author. $[email protected] (W. Zhai) https://wishzhai.github.io/ (W. Zhai) 0009-0006-2698-8549 (W. Zhai) ©2025 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). blends. It remains an important device for musical creativity. MusicVAE 1 explicitly frames interpolation as a “musical palette,” analogous to how painters blend colors. What counts as “good” or “bad” in art is always defined by aesthetic judgment, and therefore subjective. In this project, we ask a simple but generative question: what happens when the musical palette is tasked with blending between what the model encodes as “well-formed” music and material that deliberately resists those norms? Our inspiration stemmed from Der Greif 2 magazine’s Issue 12: Blame the Algorithm, a photography publication that explored algorithmic censorship through curatorial experiment. The editors issued a call for images likely to be rejected by platform systems—those considered too private, too violent, too political, or too subversive. Instead of making the selections themselves, they delegated the task to a former Facebook content moderator—a human agent once tasked with implementing algorithmic content policy. The result was a curated dataset of ninety photographs, manually categorized as either “good” or “bad.” This binary labeling exposed the blurry, subjective boundaries between acceptable and unacceptable imagery. By combining human judgment with algorithmic logic, the editors offered a critique of platform governance and the moral labor embedded in content moderation. They blamed not just the algorithm, but the system that teaches humans to judge like machines. Motivated by this conceptual framework, we designed a musical counterpart to test the structural boundaries of a generative model. We established a baseline dataset of well-formed melodies from the POP909 [ 13 ] corpus and a corresponding test dataset of stylistically perturbed melodies, created by systematically corrupting the originals to disrupt pitch, rhythm, and control signals. We then attempted to interpolate between the well-formed sequences and their perturbed counterparts using a fine-tuned MusicVAE checkpoint. The results were definitive: MusicVAE failed to generate meaningful interpolations with the perturbed data, rejecting sequences at the converter level as too chaotic for its quantization and embedding routines. This failure revealed algorithmic assumptions about acceptable musical content. Like Der Greif ’s visual experiment, our system could not process what lay outside its learned aesthetic bounds. This raises critical questions: what kinds of creativity do generative models exclude, and can such tools be reimagined to recognize artistic practices rooted in noise, disruption, or resistance? 2. Methodology According to the official MusicVAE documentation, the latent space is designed to satisfy three key properties: (1) Expression, enabling faithful encoding and reconstruction of real musical examples; (2) Realism, ensuring that any point in the space decodes to a musically plausible output; and (3) Smoothness, such that nearby points correspond to similar musical ideas. While smoothness emphasizes local coherence, this raises an important question: can interpolations across more distant regions of the latent space or fine-tuning with stylistically divergent data such as glitch-processed POP909 yield outputs that are not only plausible but 1https://magenta.tensorflow.org/music-vae 2https://dergreif.org/ also more creatively novel? To test whether this palette can be expanded to accommodate alternative aesthetics, we designed the following full pipeline3, illustrated in Figure 1. Figure 1: workflow from POP909 dataset through melody extraction, glitch transformation, dataset combination, fine-tuning, to bias diagnostic analysis. 2.1. Dataset and Glitch Transformation We used the POP909 dataset to create balanced training data: 909 clean melody segments and 909 glitch-transformed counterparts. Each 2-bar melody was processed through MusicVAE’s cat-mel_2bar_big configuration, converting MIDI to NoteSequence format that retains discrete note events (pitch, velocity, duration) while discarding controller information. Glitch transformations applied five systematic perturbations: •Note Deletion: 𝐷=|𝑀deleted|/|𝑀total|= 0.6 •Pitch Bends: 𝑃𝐵(𝑛𝑖) = {𝑝𝑗∼ 𝒰(−8192,+8191)}𝑘 𝑗=1, 𝑘 ∈[5,15] •Controller Chaos: 𝐶𝐶(𝑐, 𝑡)=𝑣𝑡∼ 𝒰(0,127),|𝒞| = 9 • Temporal Displacement: 𝑡′ 𝑖=𝑡𝑖+𝛿, 𝛿 ∼ 𝒰(−0.1,+0.1) ; 𝑑′ 𝑖=𝑑𝑖·𝛾, 𝛾 ∈ {0.25,0.5,1.5,2.0} • System Interruptions: Random insertion of “All Notes Off” and “Reset Controllers” messages 2.2. Full-Parameter Fine-Tuning Strategy To accommodate the radical structural divergence between glitch music and conventional pop melodies, we adopted a full-parameter fine-tuning strategy for MusicVAE. Unlike selective or transfer-layer tuning, this approach allows all learnable components of the model to adjust, thereby maximizing its capacity to internalize the irregularities and disruptions inherent in glitch aesthetics. The decision was motivated by three primary concerns: (1) the significant reduction in note density introduced by our glitch pipeline, (2) the breakdown of temporal regularity, and (3) the injection of unpredictable control signals—all of which deviate significantly from the clean, structured data on which MusicVAE was originally trained. While full-parameter fine-tuning 3Our implementation is available at https://github.com/wishzhai/PopGlitch-VAE can partly mitigate this mismatch, the pre-training bias toward coherence and regularity remains a crucial contextual factor in interpreting the model’s failures. The full scope of parameter updates included the bidirectional LSTM encoder (to relearn representations from sparse inputs), the latent space estimators (to adjust the statistical mapping between clean and glitch distributions), the stacked LSTM decoder (to accommodate unpredictable timing), and the output projection layers (to synthesize dissonant or fragmented pitch sequences). We trained the model for 10,000 steps using a batch size of 32 and a learning rate of 0.0001, optimized with Adam and a loss function combining reconstruction loss with KL divergence regularization. 3. Results We designed diagnostic tests to detect structural biases in the MusicVAE pipeline. The framework operates through controlled perturbations across four processing stages: data representation (MIDI-to-NoteSequence conversion), architecture (BiLSTM encoding), training objectives (latent space optimization), and output regression (LSTM decoding). 3.1. Systematic Bias Detection After 10,000 steps of full-parameter fine-tuning on a balanced glitch-transformed dataset (909 clean files, 909 glitch files) using MusicVAE’s cat-mel_2bar_big checkpoint, we analyzed bias manifestation across these four stages. Our findings indicate that even under aggressive parameter adaptation with balanced training data, MusicVAE systematically excludes experimental musical features. As illustrated in Figure 2, four critical failure points emerge, and this analysis of failure mechanisms can provide the community with insights to guide future improvements, ultimately enhancing the creative potential of music generation systems. Figure 2: Overview of bias detection points. The flowchart shows the standard data flow from MIDI input through NoteSequence conversion, BiLSTM encoder, latent space, LSTM decoder, to generated output. Four bias points are identified: (1) Data Representation bias during conversion, (2) Architecture bias in the encoder, (3) Training Objectives bias in latent space optimization, and (4) Output Regression bias in the decoder. Bias 1: Data Representation Detection Point: NoteSequence conversion stage Exclusion Mechanism: Systematic filtering of experimental features during MIDI-to-sequence parsing Evidence: Our diagnostic analysis confirms that pitch bend data (full MIDI range +/- 8192) and controller changes across nine types are eliminated during conversion. The system explicitly reports "MusicVAE only processes note sequences, not controllers" and "Most glitch features are lost at model input stage." Feature analysis rates Controller Chaos as "Impossible" and Pitch Bend as "Extremely Difficult" to learn, with the reason "Model architecture does not support CC data." This preprocessing stage structurally excludes the majority of glitch-specific features before any trainable parameters are engaged. To make failure, evaluation, and rejection explicit at this stage, we define and apply the following: failure is quantified by a controller/pitch-bend retention rate 𝑟ctrl = (𝑁′ bend +𝑁′ cc)/(𝑁bend +𝑁cc) measured per controller type; in this run, 𝑟ctrl = 0 by construction since all controller and bend events are discarded. Evaluation further includes reporting the preservation rate of onset/duration perturbations after quantization; rejection is recorded whenever 𝑟ctrl = 0 or when quantization snaps at least 90% of onset perturbations below one grid step, indicating that temporal irregularities are effectively erased at input. In this experiment, rejection is already met by 𝑟ctrl = 0. Bias 2: Architecture Detection Point: BiLSTM encoder processing Exclusion Mechanism: Inability to encode structurally irregular sequence patterns Evidence: The bidirectional LSTM encoder consistently failed to process glitch-derived sequences in this run, aborting the encode/decode path with the explicit error "Can only extract subsequences from unquantized NoteSequence." We operationalize extreme temporal displacement as onset deviations that substantially exceed the quantization grid (e.g., sustained off-grid timing or highly variable inter-onset intervals) and would, if encodable, be assessed by latent separation between a normal 8-note probe and a chaotic 7-note probe, as well as by interpolation behavior. In the diagnostic framework, rejection at the architectural stage is declared when the pipeline fails prior to encoding with the above explicit error (pre-encoding rejection). Because the encode/decode path aborted in this run, latent-space distance and interpolation-based metrics are not computed and are marked as not applicable for this experiment; nevertheless, the pre-encoding rejection itself constitutes the architectural failure signal. Even after full-parameter fine-tuning, this outcome indicates fundamental incompatibility with non-sequential or fragmented musical structures. Bias 3: Training Objectives Detection Point: Latent space optimization and loss computation Exclusion Mechanism: Systematic penalization of experimental patterns through objective design Evidence: Loss function analysis indicates consistent bias against irregular structures: the reconstruction objective tends to generate normal sequences and penalizes sparse or irregular patterns (e.g., heavy note deletion), while the KL term suppresses extreme latent representations, pushing toward the prior and filtering radical departures from the training distribution. We make rejection explicit by comparing per-sequence loss decomposition for matched clean vs. glitch probes and declaring rejection when the glitch probe exhibits a persistent and positive loss gap across steps (with both reconstruction and KL components reported). In this particular run, because the pipeline rejected glitch sequences before encoding, these differential loss measurements could not be computed; the bias statement here therefore characterizes the objective-level mechanism rather than a numerical gap from this run. Bias 4: Output Regression Detection Point: LSTM decoder and output generation Exclusion Mechanism: Inability to reconstruct or interpolate structurally unconventional sequences Evidence: The decoder in practice tends to regress to conventional musical patterns when upstream representations fail to carry glitch-specific structure. We quantify generation-stage failure, when reachable, by two measures: the on-grid rate (fraction of decoded onsets aligned to the quantization grid) and the deletion-retention rate (fraction of intended deletions preserved after decoding); rejection is declared when interpolations or reconstructions yield on-grid rates >= 90% together with deletion-retention <= 20% across seeds and interpolation steps, indicating collapse back to standard structures. In this run, decoding and interpolation could not be executed due to pre-encoding rejection at the architectural stage; thus these generation-stage metrics are not computed and are marked as not applicable for this experiment. 3.2. Interpolation Failure Analysis Our clean-to-glitch interpolation experiments demonstrated failure at the fundamental encoding stage. With the available dataset in this study (909 clean files, 909 glitch files) and successful model loading, the system repeatedly aborted when processing glitch-derived sequences with the explicit error "Can only extract subsequences from unquantized NoteSequence." This constitutes pre-encoding rejection in our diagnostic and prevents latent interpolation from being attempted. Root Cause Analysis This failure occurs before interpolation can even begin. The diagnostic confirms that the model loads successfully and can process the normal test probe, but the encode/decode test on glitch-derived sequences fails with the same explicit error regardless of training duration or parameter optimization strategy. The model lacks the architectural affordances required to interpret structurally unconventional musical sequences. Our comprehensive diagnosis identifies the core problems as follows: architecture limitation (MusicVAE does not process MIDI controllers), feature loss (most glitch features disappear at input stage), and loss bias (objectives penalize irregular features). When attempting to process glitch sequences, the system encounters categorical representational exclusion rather than gradual quality degradation, thereby delineating boundaries of musical legitimacy that exclude glitch aesthetics. 4. Discussion Our findings expose the implicit aesthetic biases in AI music systems. MusicVAE’s rejection of glitch-modified sequences reveals a design philosophy that favors "normal" music, raising critical questions about inclusivity in computational creativity tools. This multi-level exclusion, from data representation to optimization, suggests that addressing these limitations requires fundamental architectural changes. Like Der Greif ’s visual experiments, our study reveals encoded musical values within generative models. Future research should expand beyond MusicVAE, exploring diverse architectures and developing systems that respect both technological capabilities and artistic freedom in human-AI collaboration. References [1] S. Li, Y. Sung, Melodydiffusion: Chord-conditioned melody generation using a transformerbased diffusion model, Mathematics 11 (2023) 1915. URL: https://www.mdpi.com/ 2227-7390/11/8/1915/pdf. doi:10.3390/math11081915. [2] Z. Fei, M. Fan, C. Yu, J. Huang, Flux that plays music, In IEEE Conference on Artificial Intelligence (CAI) (2024). URL: https://arxiv.org/pdf/2409.00587. [3] S. Kim, K. Takeda, T. Toda, Sequence-to-sequence network training methods for automatic guitar transcription with tokenized outputs, in: Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), 2023. URL: https://archives.ismir.net/ ismir2023/paper/000062.pdf. [4] G. Mittal, J. Engel, C. Hawthorne, I. Simon, Symbolic music generation with diffusion models, in: Proceedings of the 22nd International Society for Music Information Retrieval Conference, 2021. URL: https://archives.ismir.net/ismir2021/paper/000058.pdf. [5] Y. Ma, A. Øland, A. Ragni, B. M. S. Del Sette, C. Saitis, Foundation models for music: A survey, arXiv preprint arXiv:2408.14340 (2024). URL: https://arxiv.org/pdf/2408.14340. [6] Z. Tian, Y. Jin, Z. Liu, R. Yuan, X. Tan, Q. Chen, Audiox: Diffusion transformer for anythingto-audio generation, arXiv preprint arXiv:2503.10522 (2025). URL: https://arxiv.org/pdf/ 2503.10522. [7] J. Zhu, K. Sakurai, R. Togo, T. Ogawa, Structured polyphonic music generation with diffusion transformer, IEEE Access (2024). URL: https://ieeexplore.ieee.org/abstract/document/ 10760546/. [8] Y. Zhang, Improving controllability and editability for pretrained text-to-music generation models, arXiv preprint arXiv:2411.12641 (2024). URL: https://arxiv.org/pdf/2411.12641. [9] J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Synnaeve, Y. Adi, A. Défossez, Simple and controllable music generation, 2024. URL: https://arxiv.org/abs/2306.05284. arXiv:2306.05284. [10] R. Yuan, H. Lin, S. Guo, G. Zhang, J. Pan, Y. Zang, H. Liu, Y. Liang, W. Ma, X. Du, X. Du, Z. Ye, T. Zheng, Y. Ma, M. Liu, Z. Tian, Z. Zhou, L. Xue, X. Qu, Y. Li, S. Wu, T. Shen, Z. Ma, J. Zhan, C. Wang, Y. Wang, X. Chi, X. Zhang, Z. Yang, X. Wang, S. Liu, L. Mei, P. Li, J. Wang, J. Yu, G. Pang, X. Li, Z. Wang, X. Zhou, L. Yu, E. Benetos, Y. Chen, C. Lin, X. Chen, G. Xia, Z. Zhang, C. Zhang, W. Chen, X. Zhou, X. Qiu, R. Dannenberg, J. Liu, J. Yang, W. Huang, W. Xue, X. Tan, Y. Guo, Yue: Scaling open foundation models for long-form music generation, 2025. URL: https://arxiv.org/abs/2503.08638.arXiv:2503.08638. [11] R. Yuan, H. Lin, Y. Wang, Z. Tian, S. Wu, T. Shen, G. Zhang, Y. Wu, C. Liu, Z. Zhou, L. Xue, Z. Ma, Q. Liu, T. Zheng, Y. Li, Y. Ma, Y. Liang, X. Chi, R. Liu, Z. Wang, C. Lin, Q. Liu, T. Jiang, W. Huang, W. Chen, J. Fu, E. Benetos, G. Xia, R. Dannenberg, W. Xue, S. Kang, Y. Guo, ChatMusician: Understanding and generating music intrinsically with LLM, in: L.-W. Ku, A. Martins, V. Srikumar (Eds.), Findings of the Association for Computational Linguistics: ACL 2024, Association for Computational Linguistics, Bangkok, Thailand, 2024, pp. 6252–6271. URL: https://aclanthology.org/2024.findings-acl.373/. doi: 10.18653/ v1/2024.findings-acl.373. [12] Y. Kim, S.-J. Lee, C. Donahue, Amuse: Human-ai collaborative songwriting with multimodal inspirations, arXiv preprint arXiv:2412.18940 (2024). [13] Z. Wang*, K. Chen*, J. Jiang, Y. Zhang, M. Xu, S. Dai, G. Bin, G. Xia, Pop909: A popsong dataset for music arrangement generation, in: Proceedings of 21st International Conference on Music Information Retrieval, ISMIR, 2020.