Are you really listening? Boosting Perceptual Awareness in Music-QA Benchmarks

ARE YOU REALLY LISTENING? BOOSTING PERCEPTUAL AWARENESS IN MUSIC-QA BENCHMARKS Yongyi Zang1Sean O’Brien2Taylor Berg-Kirkpatrick2 Julian McAuley2Zachary Novack2 1Independent Researcher 2University of California, San Diego [email protected], {seobrien,tberg,jmcauley,znovack}@ucsd.edu ABSTRACT Large Audio Language Models (LALMs), where pretrained text LLMs are finetuned with audio input, have made remarkable progress in music understanding. However, current evaluation methodologies exhibit critical limitations: on the leading Music Question Answering benchmark, MuChoMusic, text-only LLMs without audio perception capabilities achieve surprisingly high accuracy of up to 56.4%, much higher than chance. Furthermore, when presented with random Gaussian noise instead of actual audio, LALMs still perform significantly above chance. These findings suggest existing benchmarks predominantly assess reasoning abilities rather than audio perception. To overcome this challenge, we present RUListening, a framework that enhances perceptual evaluation in Music-QA benchmarks. We introduce the Perceptual Index (PI), a quantitative metric that measures a question’s reliance on audio perception by analyzing log probability distributions from text-only language models. Using this metric, we generate synthetic, challenging distractors to create QA pairs that necessitate genuine audio perception. When applied to MuChoMusic, our filtered dataset successfully forces models to rely on perceptual information—text-only LLMs perform at chance levels, while LALMs similarly deteriorate when audio inputs are replaced with noise. These results validate our framework’s effectiveness in creating benchmarks that more accurately evaluate audio perception capabilities. 1. INTRODUCTION Large language models (LLMs) have achieved impressive reasoning capabilities [1] and strong zeroand few-shot performance across NLP tasks [2], but are limited to only processing textual information. This constraint has driven the development of Multimodal LLMs (MLLMs), which extend LLMs to process, reason over, and generate multimodal content like images or videos [3]. Large Audio Lan- © Y. Zang, S. O’Brien, T. Berg-Kirkpatrick, J. McAuley and Z. Novack. Licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). Attribution: Y. Zang, S. O’Brien, T. Berg-Kirkpatrick, J. McAuley and Z. Novack, “Are you really listening? Boosting Perceptual Awareness in Music-QA Benchmarks”, in Proc. of the 26th Int. Society for Music Information Retrieval Conf., Daejeon, South Korea, 2025. Figure 1. Text-only LMs and LALMs’ performance on the Music QA benchmark MuChoMusic [5]. OpenMU is finetuned on Llama 3 8B, yet performs worse than it. guage Models (LALMs) specifically add audio perception and reasoning capabilities to LLMs. Evaluating LALMs is challenging, as conventional metrics like BLEU [4] struggle with diverse outputs. QA frameworks like MuChoMusic [5] address this by transforming evaluation into classification tasks with predefined choices, making them wellsuited for assessing music capabilities in LALMs. However, we discover a concerning issue: text-only models often select correct answers even without multimodal input, nearly matching the performance of multimodal models. We evaluated 11 text-only LLMs against state-of-the-art LALMs on the premier Music QA benchmark MuChoMusic [5] (see Figure 1). Surprisingly, we found that text-only models can perform well even without audio perception ability, with eight models reaching accuracy over 50%, two of which are even of similar parameter size as LALMs. Even more telling, OpenMU [6]—a LALM finetuned from Llama 3 8B— performs worse on this benchmark than its text-only Llama 3 8B foundation, despite having access to the audio. As mentioned in the MuChoMusic paper and per our re-evaluation (See Fig. 2), when presented with gaussian noise as input, the LALMs only show very limited performance decline no where near 247 Figure 2. LALM performance with original input vs. gaussian noise input on MuChoMusic [5]. to chance level. We present a hypothesis for this phenomenon: the strong initialization of text-only reasoning capabilities allows LLMs to solve QA benchmarks without true audio perception, creating an illusion of understanding. To address this challenge, we introduce RUListening, a framework to boost existing QA benchmarking datasets, where we generate distractors that require active perception to be distinguished from correct answers. Starting with audio descriptions, questions, and correct answers, we prompt a text-only model to generate plausible yet incorrect candidates. We define "perceptual index" (PI) as the need for perceptual information, calculated from logprobabilities of distractors being selected by a text-only model. We optimize based on this metric to select four distractors per question/answer pair. We additionally employ a leave-one-out strategy for 4-fold cross-validation, ensuring robust assessment of models’ perceptual capabilities. Empirically, filtering MuChoMusic through RUListening reduces text-only models to near-chance performance, confirming reasoning alone cannot solve these questions. When audio inputs for LALMs are replaced with gaussian noise, their performance also plummets to near-or-belowchance levels, confirming sensitivity to perceptual abilities. Additionally, we find the PI metric (derived from a single text-only LM) strongly correlates with performance across all text-only LMs, validating our methodology’s generalizability and effectiveness at boosting genuine audio perception capabilities. To the best of our knowledge, this represents the first research to evaluate text-only LMs on Music QA benchmarks, exploring the reasoning and perception ability separately for LALMs, and the first to propose such a methodology for boosting QA benchmarks to specifically emphasize perceptual capabilities. We believe our work advances the community’s approach to benchmarking LALMs. We open-source all code and evaluation scripts at https://github. com/yongyizang/AreYouReallyListening and RUL-MuChoMusic at https://huggingface.co/ datasets/yongyizang/RUListening under MIT License to facilitate further research. 2. RELATED WORK 2.1 LALMs Large Audio Language Models (LALMs) combine audio encoders with fine-tuned LLMs to process audio alongside text tokens. Pengi [7] pioneered this architecture, achieving state-of-the-art results on audio classification tasks. This breakthrough inspired numerous open-source models including LTU [8], LTU-AS [9], SALMONN [10], FUTGA [11], AudioGPT [12], GAMA [13], JMLA [14], and Audio Flamingo [15], plus open-access alternatives like Qwen-Audio [16] and Qwen2-Audio [17]. Research has prioritized scaling parameters and datasets over improving data quality or audio representations [18]. While these models show enhanced performance on basic tasks, they still face limitations in real-world applications [19]. 2.2 Music QA Benchmarks LALM benchmarks evaluate either specific musical attributes (tonality, genre, instrument identification) or overall music understanding through audio description and musical inquiry tasks [20–23]. For question-answer pairs, many works [5, 20, 21, 23, 24] use the MusicCaps collection [25], while others [21, 22] create new datasets by using LLMs to convert existing annotations from MusicCaps or MagnaTagaTune [26] into structured QA formats, producing datasets like MusicQA and MusicInstruct. MMAU [27] represents a recent advancement that balances information extraction (perception) and reasoning questions. Some research focuses on evaluating models trained on symbolic music representations [5, 28–30], with MuChin [28] using non-multiple-choice Chinese text and both MusicTheoryBench and ZIQI-Eval targeting textoriented LLMs through symbolic notation rather than audio. Meanwhile, multimodal capability evaluation appears in works like AIR-Bench [31], which includes musicrelated assessments within broader audio comprehension, and MuChoMusic [5], which employs LLMs with human verification to generate question-answer pairs from audio descriptions, creating more robust benchmarks for comprehensive music understanding evaluation. 2.3 Multimodal Perception Benchmarks Various benchmarks assess multimodal reasoning abilities. Beyond those discussed above, MMMU [32] provides a multi-discipline dataset for evaluating vision models’ multimodal reasoning, while mementos [33] tests reasoning over long image sequences. However, perception assessment remains relatively underexplored compared to reasoning evaluation. Chen et al. [34] found that many vision language model benchmark questions can be answered without visual input or rely on textual components from training data. They developed a filtering methodology using text-only language models to answer questions, and using their accuracy to determine the degree of reliance for a question on visual modality. To our knowledge, no similar work exists for audio or music language models. Proceedings of the 26th ISMIR Conference, Daejeon, Korea, September 21-25, 2025 248 3. REASONING IS ENOUGH TO SOLVE CURRENT MUSIC QA BENCHMARK We begin by assessing the extent to which current Music QA benchmark requires perception. To do so, we evaluate text-only LMs, which have no perception but strong reasoning capabilities, on the MuChoMusic benchmark, comparing them against LALMs, which have both perception and reasoning capabilities. This comparison allows us to quantify the importance of perceptual abilities in successfully addressing music-related questions. For text-only LMs, we evaluate 11 SOTA models across <3B, <8B, <32B, <72B and >72B parameter ranges: Gemma 2B and Llama 3.2 3B; Llama 3 8B [35] and Qwen 2.5 7B [36]; Mixtral 8x7B [37] and Gemma 27B [38]; Mixtral 8x22B [39], Qwen 2.5 72B, and Llama 3.1 70B; and Llama 3.1 405B and DeepSeek V3 671B [40] for larger models. For LALMs, we evaluated top MuChoMusic benchmark performers including Audio Flamingo 2 [18], OpenMU [6], Qwen Audio [16] and Qwen2-Audio [16], reporting results from original model papers or the MuChoMusic paper when available. We observed that the standard evaluation prompt provided by MuChoMusic often resulted in the models declining to answer, with responses indicating they could not perceive the audio. To complete the evaluation, we modified the prompt to: “Provide your best guess of this question. You must guess one, even if you did not hear the audio. Think step by step.” This change prompted text-only LMs to generate answers. For LALMs, we couldn’t reproduce the reported results (actual performance was lower) due to fluctuations from text decoding hyperparameters. For fair comparison, we cite the numbers from their original papers. Figure 1 presents the evaluation results. Notably, eight text-only LMs are able to achieve an overall accuracy of more than 50%, with two of them (Llama 3 8B and Qwen 2.5 7B) of similar size as LALMs. Even more surprisingly, although OpenMU was fine-tuned from Llama 3 8B, its performance falls short of the base Llama 3 8B model. As mentioned in the MuChoMusic paper ( [5], see Fig. 6b) and per our re-evaluation 1(see Fig. 2), when presented with gaussian noise as input, the LALMs only show very limited performance decline nowhere near chance level; one LALM even showed performance improvement. These findings suggest that reasoning capabilities alone, even without perception, may be sufficient to solve the MuChoMusic benchmark. But how is this possible? How can text-only models answer music QAs without hearing the music? Through manual inspection, we identified two key reasons: First, MuChoMusic’s design includes a "correct but irrelevant" distractor providing contextual information. While this 1OpenMU [6] failed to be evaluated under noise input for MuChoMusic since its language capabilities become too severely degraded and rejects to answer the question, despite our best effort by changing hyperparameters and prompts. distractor is helpful in analyzing error modes, text-only LMs can exploit it to make educated guesses. Second, we hypothesize that text-only LMs are using their inherent "world prior" knowledge from their training. They may understand that 80s pop songs likely feature synthesizers rather than orchestras, and folk songs typically contain acoustic rather than distorted electric guitars. This broad world knowledge could have made much of the multimodal input redundant for answering questions, reducing the benchmark’s effectiveness in evaluating LALMs’ true perceptual capabilities. To provide a concrete understanding of this phenomenon, we display a prototypical example through Llama 3.1 70B’s response to the following question 2: "Which instrument initiates the piece? Options: (A) Synthesizer (B) Rueful tune (C) Acoustic guitar (D) Vocals" Llama 3.1 70B was able to reason through this question without audio input yet arriving at the correct answer (C). It starts by eliminating impossible answers: To make an educated guess, I’ll consider common musical patterns and the options provided. Option (B) “Rueful tune” doesn’t seem to be a specific instrument, so I’ll exclude that choice. Then it attempts to leverage its prior knowledge about distribution of music: Between the remaining options, I’ll consider which instrument is most likely to initiate a piece. Synthesizers (A) can be used to create a wide range of sounds, but they’re not always the first choice to start a song. Vocals (D) can initiate a piece, but it’s not uncommon for an instrument to set the tone before the vocals begin. Acoustic guitar (C) is a popular choice to start a song, especially in genres like folk, rock, and pop. It’s often used to create a simple, yet effective melody that sets the tone for the rest of the piece. Based on this reasoning, my guess is: C. 4. CONSTRUCTING RULISTENING 4.1 Methodology We define a Music-QA benchmark as a set of audioquestion-answers triplets (a, q, Y )for audio clip a, question q, and set of answers Y, and can further decompose Y={c∪D}where cis the correct answer and Dis the set of incorrect distractors. Under this definition, an effective benchmark for audio perception should present questions that are challenging without audio but solvable with audio access. Formally, let ptext(Y|q)represent the total probability over all given answers for a text-only LM (i.e. ptext(Y|q) = Pyi∈Yptext(yi|q)), and pLALM(Y|q, a) represent the corresponding probability for a LALM. Ideally, if one wants to measure the multimodal perception abilities of LALMs, a Music-QA question should illicit a noticeable information gain when conditioning on 2We provide more examples of this in Appendix A. Proceedings of the 26th ISMIR Conference, Daejeon, Korea, September 21-25, 2025 249 the audio, i.e., p(c|q, a)≫p(c|q). Using this principle for benchmark design gives us two options for increasing the information gain: create (a, q, Y )triplets that are unimodally difficult (i.e. reduce p(c|q)), or design questions and correct answers highly perceptually aligned with audio (i.e. increase p(c|q, a)). We prioritize the former as the latter is problematic: constructing new QA-pairs is unscalable with current systems, and using LALMs to automate this would contaminate the benchmark’s evaluative purpose and rely too much on questionable LALM capabilities. We therefore focus on creating benchmark items where questions challenge text-only LMs while maintaining the expert-verified relationship between (a, q, c). We formalize this as finding optimal distractor sets D∗that maximize the probability of text-only models selecting incorrect answers. We define the need for perceptual information as "perceptual index," or PI: PI(q, Y, D) = ptext(D|q) ptext(Y|q)(1) which is equivalent to the QA-normalized error probability. This metric ranges from 0 to 1, with values closer to 1 indicating questions where a text-only model is more likely to select incorrect answers (i.e., ptext(D|q)≫ ptext(c|q)). Since we cannot modify the audio, question, or correct answer without compromising the integrity of the expert-verified content, we restrict our optimization to finding distractor sets that maximize this perceptual index metric. Importantly, PI does not perfectly correlate with the entropy of the answer space; a high PI may reflect a model that is confidently incorrect (selecting a wrong answer with high probability), thus exhibiting low entropy. We prefer PI over entropy as our optimization target precisely because PI captures the maximum possible performance gap between modalities—the distance between being confidently wrong (high PI) and correct is necessarily larger than between being uncertain (high entropy) and correct, thereby providing a stronger signal for identifying questions that genuinely require perceptual information. 4.2 Generating Distractors Set To arrive at a set of distractors D∗that maximizes PI, we begin with generating a large pool of possible distractors D, then filter through them to arrive at the highest PI set of distractors. We leverage the DeepSeek-V3 model to do this. We use a prompt template including question text, audio description, and correct answer, and prompt the LLM to generate multiple candidates. This process happens for multiple times, allowing us to sample multiple batches for diversity. Finally, we apply cleaning and deduplication processes. We explicitly prompt the model to maintain stylistic consistency across answers, and only generate answers that are 1) plausible and 2) distinctly different from the correct answer. We use in-context learning examples to enforce structured output using XML tags, then extract possible distractors using regular expressions and apply text normalization. Figure 3. Semantic distribution of distractors. To analyze the distribution of generated distractors, we employed two distinct models: T5 [41], a text-only transformer encoder, and the text branch of CLAP [42], a joint audio-text representation model. For each distractor and correct answer pair, we calculated the cosine similarity between their respective embeddings. The T5 similarity distribution captures the natural language semantic relationships, while CLAP’s text encoder reveals the music domain-specific relationships. Figure 3 presents histograms of these semantic similarity distributions. We observe that distractors cluster tightly in the text semantic space yet spread more widely in the music semantic space, indicating items that are textually similar (e.g., “Acoustic guitar” and “Electric guitar”) but musically distinct. This confirms our distractors maintain musical variety while minimizing textual differences (thus preventing leakage). 4.3 Filtering Based on Perceptual Index After obtaining the distractor set D, we begin filtering for D∗. To calculate the probability for each answer to be selected ptext(y|q), we use the log probability of a lightweight text-only LLM Qwen 2.5 7B. Specifically, we prompt the model with: “Provide your best guess of this question. The question is: {question} The answer candidates are: (A) ... (B) ... (C) ... (D) ... Answer without the parenthesis. The most likely answer is” Then take the log probability for the immediate next token to be A, B, C or D to represent the probability for each corresponding answer to be selected following the methodology of [43]. Empirically, we found this method worked well when evaluating <4distractors, likely due to how real-world multiple choice questions are often with four choices. As such, we begin by randomly selecting sets of three distractors, then evaluate them with the correct answer in random order. For each set of four choices, we take the distractor with highest probability; we recursively do this, until we are left with four distractors. These four distractors have highest ptext(D|q), and thus reasonably approximates the set D∗that yields the largest PI(q, Y, D). During evaluation, we implement a leave-one-out strategy: within the four distractors, we remove one at each iteration. This approach provides four distinct answer passes for each QA pair. Our design serves two purposes: (1) having 4 answers aligns with the real-world distribution of multiple-choice questions; and (2) it enhances our robustness against variations in distractors. Proceedings of the 26th ISMIR Conference, Daejeon, Korea, September 21-25, 2025 250 Figure 4. Correlation between Perceptual Index (PI) and question accuracy on RUL-MuChoMusic. Text-only LMs show stronger negative correlation, indicating greater influence from lack of perception. 5. RESULTS We evaluate the aforementioned 11 text-only LMs and select the top-performing 4 LALMs on MuChoMusic, and our proposed modifed version RUL-MuChoMusic. For all models evaluated, we report both the mean performance and 95% confidence intervals. 5.1 Validity of Perceptual Index To validate the Perceptual Index (PI) as an effective surrogate for overall LLM performance, we analyzed questionlevel accuracy across all 11 LLMs (44 response passes). For each question, we calculated the correlation between accuracy across all attempts and the PI. We observe a strong negative Pearson correlation of -0.738 as shown in Figure 4(a), indicating a highly significant relationship where high PI corresponds to low question accuracy. These results confirm PI effectively predicts text-only LMs’ ability to answer questions using solely textual information. Similarly, calculating the correlation between PI and question-level accuracy across all 4 LALMs (16 passes) reveals a weaker negative Pearson correlation of -0.331, as shown in Fig. 4(b), suggesting the need for perception is significantly higher. This validates PI as an effective metric for optimizing distractor sets to maximize the performance gap between text-only LMs and LALMs. Additionally, we plot the perceptual index distribution across all questions for both MuChoMusic and RULMuChoMusic. The only difference between these benchmarks is the distractor set. As shown in Figure 5, MuChoMusic PI values follow an approximately Gaussian distribution with mean 0.427 and larger variance, indicatFigure 5. Distribution of PI on MuChoMusic and RULMuChoMusic. MuChoMusic exhibits overall less reliance on perceptual modality compared to RUL-MuChoMusic. ing many questions can be answered substantially through text modality alone without requiring music information. This aligns with our observation that text-only language models score highly on MuChoMusic. In contrast, RULMuChoMusic achieves a significantly higher PI distribution with mean 0.861 and lower variance, demonstrating greater dependence on music modality for correct answers. For identical questions, our generated and filtered distractors consistently increase PI compared to the original benchmark (mean increase of 0.338), with some questions showing increases exceeding 0.9. 3These results confirm our generation and filtering pipeline effectively reduces text-only answering capability, creating a more robust multimodal evaluation benchmark. 5.2 Benchmark Results for Text-only LMs and LALMs We present comprehensive results for text-only LLMs and LALMs in Figure 6. Several key patterns emerge from our analysis. Across all models, we observe a consistent decrease in accuracy scores, indicating that RUL-MuChoMusic presents a greater challenge than MuChoMusic; text-only LMs perform at near-chance levels, validating our approach. Importantly, OpenMU (4thplace) outperforms its text-only subcomponent (Llama 3 8B, 12th-place), suggesting enhanced music perception capabilities. The text-only LMs that managed to place in the top-10 possess much larger parameter counts (405B, 72B, 27B, 671B, 70B, and 56B) compared to the sub-7B audio models. Though RUListening effectively increases unimodal difficulty (see Sec. 5.1), most LALMs besides Qwen2Audio demonstrate relatively poor performance, as multimodal difficulty was not used in construction. Due to Qwen2-Audio’s broad use across various tasks [44], its strong performance is expected. To quantitatively assess whether poor results stem from inherent model limitations or benchmark design flaws, we evaluated all LALMs using 10-second samples of random Gaussian noise to probe their sensitivity to audio input. Results appear in Figure 7. While all models previously performed above chance, noise inputs drove performance to near or below chance levels. Qwen2-Audio showed the most dramatic performance degradation, while Audio Flamingo 2 demon3We present examples of highest and lowest distractor PI change in Appendix C. Proceedings of the 26th ISMIR Conference, Daejeon, Korea, September 21-25, 2025 251 Figure 6. Benchmarking results on RUL-MuChoMusic. Error bar displays 95% confidence interval. Figure 7. LALM performance with original input vs. gaussian noise input on RUL-MuChoMusic. strated the least sensitivity to noise, possibly related to its weaker reasoning abilities. When comparing to MuChoMusic [5], only 2 LALMs show significant degradation with noise input, yet nowhere near chance-level performance, suggesting RUListening provides stronger evaluation of audio perception. Examining LALM response patterns reveals additional insights. 4Audio Flamingo 2 [18] exhibits limited reasoning ability, often generating direct answers. In contrast, Qwen2-Audio frequently produces extended reasoning chains. This suggests reasoning capability may be crucial for success on Music QA benchmarks, as also demonstrated by recent research exploring LALM multimodal fine-tuning techniques for reasoning models. 6. DISCUSSION While establishing RUL-MuChoMusic as a more effective perception-testing benchmark compared to MuChoMusic, we acknowledge a fundamental limitation in our work: the quality of our benchmark is inherently constrained by the quality of the provided question-answer pairs. Our manual inspection revealed several issues with the original dataset. Some problems stem from the LLMassisted methodology used to create question-answer pairs from human captions. Questions with IDs 448 and 665 have "Not specified in the description" as correct answers, while eight others contain phrases like "based on the description" despite no description being provided during benchmarking. Human captions sometimes include uninferable metadata—we found 17 questions/answers with 4We include response examples in Appendix B. "recorded in" phrases, though recording location cannot be determined solely from audio. Some questions are challenging even for human experts, such as identifying specific banjo types (question ID 730). Some correct answers inadequately address their questions—for instance, question ID 832 asks "Who is the primary vocalist in the song?" with "Male" as the correct answer, and question ID 7 asks "What is used at the very beginning to make the track sound vintage?" with the overly simplistic answer "Effect." To quantify this issue, we employed Claude 3.7 Sonnet to evaluate whether questions and answers made sense based on the audio text captions. The model identified 201 out of 1187 pairs (16.9%) as problematic. However, we observed that Claude itself made errors in this evaluation process. For example, it incorrectly flagged question ID 1125, claiming that the correct answer "Digital bass sound" was inconsistent with the audio description mentioning a "synthesizer bassline that is repeating." These issues highlight a broader challenge in Music QA benchmark construction: human-written benchmarks are time-consuming to develop, yet LLMs are error-prone when used as discriminators or assistants. The question of how to effectively balance these approaches remains an important area for future research. We hope our work serves as a starting point in encouraging researchers to critically examine the effectiveness of Music QA benchmarks. 7. CONCLUSION We introduce RUListening, a methodology for improving the perceptual relevance of LALM QA benchmarks. By demonstrating that text-only LMs outperform LALMs on existing benchmarks, we revealed that current music QA benchmarks test reasoning rather than perception. We generate distractors that maximize perceptual necessity through our Perceptual Index metric, creating a benchmark where text-only models perform at chance levels, and LALMs fall to chance level when presented with gaussian noise input. Though QA benchmarks remain constrained by their underlying question-answer pairs, RUListening offers a practical path toward developing multimodal benchmarks that genuinely require engagement with non-textual data—an approach potentially valuable for other multimodal domains beyond music. Proceedings of the 26th ISMIR Conference, Daejeon, Korea, September 21-25, 2025 252 8. ACKNOWLEDGMENTS We sincerely thank the co-first authors of MuchoMusic [5], Benno Weck and Ilaria Manco, for proposing a methodology for effective music QA benchmark creation and for their helpful early discussions that helped shape this paper. 9. REFERENCES [1] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou et al., “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems, vol. 35, pp. 24 824–24 837, 2022. [2] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” arXiv preprint arXiv:2009.03300, 2020. [3] J. Wu, W. Gan, Z. Chen, S. Wan, and P. S. Yu, “Multimodal large language models: A survey,” in 2023 IEEE International Conference on Big Data (BigData). IEEE, 2023, pp. 2247–2256. [4] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002, pp. 311–318. [5] B. Weck, I. Manco, E. Benetos, E. Quinton, G. Fazekas, and D. Bogdanov, “Muchomusic: Evaluating music understanding in multimodal audiolanguage models,” arXiv preprint arXiv:2408.01337, 2024. [6] M. Zhao, Z. Zhong, Z. Mao, S. Yang, W.-H. Liao, S. Takahashi, H. Wakaki, and Y. Mitsufuji, “Openmu: Your swiss army knife for music understanding,” arXiv preprint arXiv:2410.15573, 2024. [7] S. Deshmukh, B. Elizalde, R. Singh, and H. Wang, “Pengi: An audio language model for audio tasks,” 2023. [8] Y. Gong, H. Luo, A. H. Liu, L. Karlinsky, and J. R. Glass, “Listen, think, and understand,” in The Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https: //openreview.net/forum?id=nBZBPXdJlC [9] Y. Gong, A. H. Liu, H. Luo, L. Karlinsky, and J. Glass, “Joint audio and speech understanding,” in 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2023, pp. 1–8. [10] C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. MA, and C. Zhang, “SALMONN: Towards generic hearing abilities for large language models,” in The Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/forum?id=14rn7HpKVk [11] J. Wu, Z. Novack, A. Namburi, J. Dai, H.-W. Dong, Z. Xie, C. Chen, and J. McAuley, “Futga-mir: Enhancing fine-grained and temporally-aware music understanding with music information retrieval,” 2025. [12] R. Huang, M. Li, D. Yang, J. Shi, X. Chang, Z. Ye, Y. Wu, Z. Hong, J. Huang, J. Liu et al., “Audiogpt: Understanding and generating speech, music, sound, and talking head,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, 2024, pp. 23 802– 23 804. [13] S. Ghosh, S. Kumar, A. Seth, C. K. R. Evuru, U. Tyagi, S. Sakshi, O. Nieto, R. Duraiswami, and D. Manocha, “GAMA: A large audio-language model with advanced audio understanding and complex reasoning abilities,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. AlOnaizan, M. Bansal, and Y.-N. Chen, Eds. Miami, Florida, USA: Association for Computational Linguistics, Nov. 2024, pp. 6288–6313. [Online]. Available: https://aclanthology.org/2024.emnlp-main.361/ [14] X. Du, Z. Yu, J. Lin, B. Zhu, and Q. Kong, “Joint music and language attention models for zero-shot music tagging,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 1126–1130. [15] Z. Kong, A. Goel, R. Badlani, W. Ping, R. Valle, and B. Catanzaro, “Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities,” in Forty-first International Conference on Machine Learning, 2024. [Online]. Available: https://openreview.net/forum?id=WYi3WKZjYe [16] Y. Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou, “Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models,” arXiv preprint arXiv:2311.07919, 2023. [17] Y. Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y. Leng, Y. Lv, J. He, J. Lin et al., “Qwen2-audio technical report,” arXiv preprint arXiv:2407.10759, 2024. [18] S. Ghosh, Z. Kong, S. Kumar, S. Sakshi, J. Kim, W. Ping, R. Valle, D. Manocha, and B. Catanzaro, “Audio flamingo 2: An audio-language model with longaudio understanding and expert reasoning abilities,” arXiv preprint arXiv:2503.03983, 2025. [19] M. R. Morris, J. Sohl-Dickstein, N. Fiedel, T. Warkentin, A. Dafoe, A. Faust, C. Farabet, and S. Legg, “Position: Levels of AGI for operationalizing progress on the path to AGI,” in Proceedings of the 41st International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, Eds., vol. 235. PMLR, 21–27 Proceedings of the 26th ISMIR Conference, Daejeon, Korea, September 21-25, 2025 253 Jul 2024, pp. 36 308–36 321. [Online]. Available: https://proceedings.mlr.press/v235/morris24b.html [20] C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. MA, and C. Zhang, “SALMONN: Towards Generic Hearing Abilities for Large Language Models,” in The Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/forum?id=14rn7HpKVk [21] Z. Deng, Y. Ma, Y. Liu, R. Guo, G. Zhang, W. Chen, W. Huang, and E. Benetos, “MusiLingo: Bridging music and text with pre-trained language models for music captioning and query response,” in Findings of the Association for Computational Linguistics: NAACL 2024, K. Duh, H. Gomez, and S. Bethard, Eds. Mexico City, Mexico: Association for Computational Linguistics, Jun. 2024, pp. 3643– 3655. [Online]. Available: https://aclanthology.org/ 2024.findings-naacl.231 [22] S. Liu, A. S. Hussain, C. Sun, and Y. Shan, “Music understanding llama: Advancing text-to-music generation with question answering and captioning,” in ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024, pp. 286–290. [23] J. Gardner, S. Durand, D. Stoller, and R. Bittner, “Llark: A multimodal instruction-following language model for music,” in Proceedings of the 41st International Conference on Machine Learning (ICML), 2024. [24] Y. Vasilakis, R. Bittner, and J. Pauwels, “I can listen but cannot read: An evaluation of two-tower multimodal systems for instrument recognition,” arXiv preprint arXiv:2407.18058, 2024. [25] A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchi, M. Sharifi, N. Zeghidour, and C. Frank, “Musiclm: Generating music from text,” arXiv preprint arXiv:2301.11325, 2023. [26] E. Law, K. West, M. Mandel, M. Bay, and J. Stephen Downie, “Evaluation of algorithms using games: The case of music tagging,” in Proceedings of the 10th ISMIR Conference, 2009. [27] S. Sakshi, U. Tyagi, S. Kumar, A. Seth, R. Selvakumar, O. Nieto, R. Duraiswami, S. Ghosh, and D. Manocha, “Mmau: A massive multi-task audio understanding and reasoning benchmark,” arXiv preprint arXiv:2410.19168, 2024. [28] Z. Wang, S. Li, T. Zhang, Q. Wang, P. Yu, J. Luo, Y. Liu, M. Xi, and K. Zhang, “Muchin: A chinese colloquial description benchmark for evaluating language models in the field of music,” arXiv preprint arXiv:2402.09871, 2024. [29] R. Yuan, H. Lin, Y. Wang, Z. Tian, S. Wu, T. Shen, G. Zhang, Y. Wu, C. Liu, Z. Zhou et al., “Chatmusician: Understanding and generating music intrinsically with llm,” arXiv preprint arXiv:2402.16153, 2024. [30] J. Li, L. Yang, M. Tang, C. Chen, Z. Li, P. Wang, and H. Zhao, “The music maestro or the musically challenged, a massive music evaluation benchmark for large language models,” arXiv preprint arXiv:2406.15885, 2024. [31] Q. Yang, J. Xu, W. Liu, Y. Chu, Z. Jiang, X. Zhou, Y. Leng, Y. Lv, Z. Zhao, C. Zhou, and J. Zhou, “Air-bench: Benchmarking large audiolanguage models via generative comprehension,” CoRR, vol. abs/2402.07729, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2402.07729 [32] X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun et al., “Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 9556–9567. [33] X. Wang, Y. Zhou, X. Liu, H. Lu, Y. Xu, F. He, J. Yoon, T. Lu, G. Bertasius, M. Bansal et al., “Mementos: A comprehensive benchmark for multimodal large language model reasoning over image sequences,” arXiv preprint arXiv:2401.10529, 2024. [34] L. Chen, J. Li, X. Dong, P. Zhang, Y. Zang, Z. Chen, H. Duan, J. Wang, Y. Qiao, D. Lin et al., “Are we on the right way for evaluating large vision-language models?” arXiv preprint arXiv:2403.20330, 2024. [35] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan et al., “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024. [36] A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei et al., “Qwen2.5 technical report,” arXiv preprint arXiv:2412.15115, 2024. [37] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand et al., “Mixtral of experts,” arXiv preprint arXiv:2401.04088, 2024. [38] G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ramé et al., “Gemma 2: Improving open language models at a practical size,” arXiv preprint arXiv:2408.00118, 2024. [39] “Cheaper, Better, Faster, Stronger | Mistral AI — mistral.ai,” https://mistral.ai/news/mixtral-8x22b, [Accessed 28-03-2025]. [40] A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan et al., “Deepseek-v3 technical report,” arXiv preprint arXiv:2412.19437, 2024. Proceedings of the 26th ISMIR Conference, Daejeon, Korea, September 21-25, 2025 254 [41] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified textto-text transformer,” Journal of machine learning research, vol. 21, no. 140, pp. 1–67, 2020. [42] Y. Wu*, K. Chen*, T. Zhang*, Y. Hui*, T. BergKirkpatrick, and S. Dubnov, “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” in IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP, 2023. [43] N. Sachdeva, B. Coleman, W.-C. Kang, J. Ni, L. Hong, E. H. Chi, J. Caverlee, J. McAuley, and D. Z. Cheng, “How to train data-efficient llms,” arXiv preprint arXiv:2402.09668, 2024. [44] R. Yuan, H. Lin, S. Guo, G. Zhang, J. Pan, Y. Zang, H. Liu, Y. Liang, W. Ma, X. Du et al., “Yue: Scaling open foundation models for long-form music generation,” arXiv preprint arXiv:2503.08638, 2025. Proceedings of the 26th ISMIR Conference, Daejeon, Korea, September 21-25, 2025 255