Harmonising Music Information Retrieval with Semantics: from Data Integration to Multimodality

i i “output” — 2025/10/28 — 13:54 — page 1 — #1 i i i i i i Dottorato di ricerca in Computer Science and Engineering Ciclo 37 Settore Concorsuale 01/B - INFORMATICA Settore Scientifico Disciplinare INF/01 - INFORMATICA Harmonising Music Information Retrieval with Semantics: from Data Integration to Multimodality Coordinatore Dottorato Prof. Ilaria Bartolini Supervisore: Prof. Valentina Presutti Esame finale anno 2025 i i “output” — 2025/10/28 — 13:54 — page 2 — #2 i i i i i i i i “output” — 2025/10/28 — 13:54 — page I — #3 i i i i i i i i “output” — 2025/10/28 — 13:54 — page II — #4 i i i i i i “Science cannot tell us a word about why music delights us, of why and how an old song can move us to tears. Science can, in principle, describe in full detail all that happens in the latter case in our sensorium and ‘motorium’ from the moment the waves of compression and dilation reach our ear to the moment when certain glands secrete a salty fluid that emerges from our eyes.” Erwin Schrödinger i i “output” — 2025/10/28 — 13:54 — page III — #5 i i i i i i i i “output” — 2025/10/28 — 13:54 — page I — #6 i i i i i i Abstract Inthe era of big data and machine learning, the fragmentation of musical datasets and the lack of standardised representations continue to hinder advancements in Music Information Retrieval (MIR). The multifaceted nature of music complicates both the representation of content – with small, task-specific datasets scattered across various formats, and context (metadata), where there is a lack of a consistent, structured terminology. These challenges increase the effort required for data collection and pre-processing, reduce reproducibility, and limit the scalability of MIR models. To address these issues, this thesis proposes a unified semantic model to foster interoperability and advance MIR tasks. A specific instance of this fragmentation can be found in harmonic annotations, where harmony – an essential musical dimension – is inconsistently represented across datasets, formats, and notational systems. Taking harmony as a use case and leveraging the proposed semantic model, this thesis develops a standardised workflow to harmonise previously disconnected datasets, enabling the creation of new, large-scale unified corpora. Building on these harmonised datasets, a key contribution of the thesis is the exploration of harmonic similarity, which is used to organise, explore, and reveal connections across diverse tracks, historical periods, and genres. To this end, we implement novel state-of-the-art harmonic similarity functions, advancing current research in MIR. Moreover, by utilising the created large, varied data collections, we uncover deeper relationships within music and explore how diverse stylistic excerpts can be used for generative tasks. While integrating symbolic data offers significant advantages, certain limitaI i i “output” — 2025/10/28 — 13:54 — page II — #7 i i i i i i tions persist. Primary challenges include the limited diversity of annotated data, often biased toward a narrow range of musical genres, and the inherent ambiguity and subjectivity in harmonic annotations. Such challenges have led MIR tasks like Audio Chord Estimation (ACE) to hit a “glass ceiling,” where neither increasing computational power nor the volume of data has led to improved results. To address these issues, this thesis explores a multimodal approach aimed at enhancing the analysis and understanding of harmonic data by jointly leveraging audio and chord annotations. We first propose a novel method for enriching the dataset with audio annotations aligned to the existing symbolic data. Building on this foundation, we introduce a new model for ACE that embeds formalised music theory concepts such as consonance and dissonance, addressing both training and evaluation challenges in the field. This model aims to mitigate the limitations of chord vocabulary imbalance and annotation subjectivity, ultimately improving the state-of-the-art in audio-based harmonic analysis. i i “output” — 2025/10/28 — 13:54 — page III — #8 i i i i i i i i “output” — 2025/10/28 — 13:54 — page IV — #9 i i i i i i Contents Abstract II 1 Introduction 1 1.1 Problem Definition and Research Questions . . . . . . . . . . . . . 2 1.1.1 Challenges in Representing Music: The Need for Unified SemanticModels ......................... 2 1.1.2 Harmonising Symbolic Data: Addressing Fragmentation in HarmonicDatasets....................... 6 1.1.3 Exploring Harmonic Similarity: Leveraging Large-Scale Corpora for Deeper Musical Understanding . . . . . . . . . . . 7 1.1.4 Limitations of Symbolic Data Integration: A Multimodal Approach............................ 9 1.1.5 Limitations in Audio Chord Estimation: Chord Imbalance andSubjectivity ........................ 11 1.2 ThesisContribution.......................... 13 1.3 ThesisStructure............................ 17 2 Background 21 2.1 Music Theory and Structure . . . . . . . . . . . . . . . . . . . . . 22 2.1.1 Fundamentals of Music Theory . . . . . . . . . . . . . . . . 23 2.1.2 FacetsofMusic......................... 30 2.1.3 Focus on Harmony . . . . . . . . . . . . . . . . . . . . . . . 34 2.2 MusicTechnology ........................... 39 2.2.1 Historical Background . . . . . . . . . . . . . . . . . . . . . 39 2.2.2 Music Information Retrieval Tasks . . . . . . . . . . . . . . 40 2.2.3 HarmonyinMIR........................ 42 2.3 Music Representation . . . . . . . . . . . . . . . . . . . . . . . . . 43 2.3.1 Representing Musical Metadata . . . . . . . . . . . . . . . . 44 2.3.2 Representing Musical Content . . . . . . . . . . . . . . . . 46 IV i i “output” — 2025/10/28 — 13:54 — page 2 — #16 i i i i i i Chapter 1. Introduction models for handling and analysing vast digital music collections. As music archives expanded in size and complexity, new challenges emerged around how to represent and access different aspects of music. MIR, which established itself as a formal research field in the early 2000s [120], focuses on addressing these challenges by developing techniques for extracting meaningful information from diverse music data representations, such as audio recordings, symbolic data, or music metadata. Central tasks such as retrieval,recommendation, and browsing have become core paradigms of the field, enhancing user interactions and facilitating seamless access to music collections [221]. Despite these advancements, a consistent and meaningful representation of both musical content and metadata remains an unresolved problem due to the complex nature of music itself. Music consists of a variety of structural elements, such as melody, harmony, and rhythm, that can be represented differently in terms of notation and conceptual model, depending on the task to be addressed or the available dataset(s). In addition, music is deeply embedded in diverse cultural and historical contexts, adding further complexity to the metadata that any retrieval or analysis system should account for [41]. This complexity has led to the fragmentation of music data into isolated, task-specific formats, notations, and conceptual models, making it increasingly difficult to develop unified frameworks that scale across diverse datasets, styles and genres. We argue that addressing this fragmentation by establishing standardised approaches to music representation is essential for advancing MIR models and improving the performances on a broad range of MIR tasks. 1.1 Problem Definition and Research Questions 1.1.1 Challenges in Representing Music: The Need for Unified Semantic Models The challenge of understanding and processing the multiple dimensions of music within MIR is often referred to as the “multifaceted challenge” and these facets are categorised by Downie [119] into seven distinct categories: pitch,temporal, harmonic,timbral,editorial,textual, and bibliographic. While the first six facets primarily address music content, encapsulating the diverse elements that make up a musical piece, the bibliographic facet extends beyond the musical content itself, capturing broader contextual information, such 2 i i “output” — 2025/10/28 — 13:54 — page 3 — #17 i i i i i i 1.1. Problem Definition and Research Questions as the historical, cultural, and publication data surrounding a musical work. This facet is also known as knowledge about music [296], or more commonly, music metadata. The distinction between content and context forms the basis of two major paradigms within MIR: Content-based MIR, which deals with the intrinsic properties of music, usually further categorised into signal and symbolic representations, and Context-based MIR, which focuses on external, contextual information. Both paradigms are widely used for tasks such as music similarity, recommendation, and browsing, either in isolation or in combination [221]. Effectively representing both the content and context of music presents one of the central challenges within MIR. Capturing the context, such as cultural or historical information, is particularly complex. A significant part of the difficulty stems from the lack of a consistent and universally accepted terminology for describing music metadata. For instance, in conventional databases, the “artist” is typically the central entity, contrasting with the emphasis on the “composer” in Western classical music and with the concept of “performer” in Western popular music. Similarly, the concept of a “composition” in the classical tradition diverges significantly from terms like “track” or “song,” which dominate in most contemporary systems. These problems have repercussions on many MIR tasks, both when searching for data in a database and when integrating data from different datasets. Furthermore, musical heritage encompasses a diversity of human expressions and experiences across different cultural and historical contexts. This diversity is reflected in the metadata, where multiple sources (e.g., theatres, conservatoires, instruments) must be connected to their broader social and artistic contexts (e.g., scholars, musicians, intellectuals) across different languages and time periods [41]. These issues are exacerbated when the context includes conflicting or incomplete information, such as varying names for the same artist across genres or historical periods. For instance, music metadata often lacks a unified structure to account for oral traditions (common in folk music), where works evolve through verbal transmission rather than being formally composed and notated. Semantic models have been proposed to standardise the description of music metadata (e.g. [324, 64]), however, these models are mostly limited in scope. They focus on specific historical periods or genres and are tailored to narrow requirements, limiting their applicability to other styles or broader contexts. As a result, they are insufficient for generalising across diverse musical collections, 3 i i “output” — 2025/10/28 — 13:54 — page 4 — #18 i i i i i i Chapter 1. Introduction making it difficult to create a unified approach for organising and accessing music across varying genres and traditions. The problem is further compounded when addressing music content. In fact, unlike other forms of knowledge, music lacks a consensual, shared meaning [296]. Whereas language or images usually point to specific objects or concepts, music consists of abstract elements—such as notes, chords, and sounds—that exist within their own domain, detached from external reference points. Historically, music notation has been the primary method for representing these abstract elements in the context of Western music. Originally developed to preserve the work over time and to allow performers to recreate the composer’s original intent [25], music notation has evolved over a thousand years [24], reflecting the growing complexity of Western music and its associated performance practices. Computationally, music is typically represented in one of two ways: signal representations and symbolic representations [403]. Signal representations consist of recordings from sound sources. These representations are content-unaware and unstructured, making the extraction of information a complex task. In contrast, symbolic representations denote discrete musical events and provide a structured format that is context-aware, facilitating easier data extraction and analysis [395]. Depending on the task, different derivations of both audio representations (such as MFCCs or Chroma Features) and symbolic representations (such as MIDI or MusicXML) have been proposed [282]. This multiplicity of possible representations, known as the Multirepresentational challenge [119], has become increasingly significant. Each representation has unique implications for computational analysis: for instance, MIDI [106] representations capture performance details effectively but lack the expressive nuances of score-based formats, like MEI [174]. Selecting appropriate representations ensures that the data is both relevant and informative, which in turn enhances accuracy and reduces computational load [252]. The challenges of representing music content have led to a fragmentation in the field, where different datasets are represented using diverse symbolic representations, each tailored to serve specific tasks and applications. This fragmentation poses two major disadvantages: first, it requires significant resources for collecting, harmonising, and pre-processing disconnected and inconsistently represented data; second, it hampers the comparison and reproducibility of research results, as results derived from different datasets are not easily aligned [302]. As a consequence, the need for data interoperability has become increasingly critical. Data 4 i i “output” — 2025/10/28 — 13:54 — page 5 — #19 i i i i i i 1.1. Problem Definition and Research Questions integration must address syntactic alignment (i.e., consistent data formats and structures) as well as semantic consistency, ensuring that the meaning and the relationships between musical elements are preserved across different representations. Achieving this requires data conversion as well as the formal modelling of musical information, such as by means of ontologies. Several ontologies have been developed to address various aspects of music, but they remain limited in scope. Most focus on modelling elements of music notation [332, 214, 315, 314], and some target specific formats, such as MIDI [272], or address a narrow set of audio signal features [7]. We argue that addressing the fragmentation of music data and the limitations of existing ontologies is essential for advancing MIR. This thesis proposes the development of a unified semantic model that integrates both music content and context, supporting interoperability and enhancing the reproducibility of MIR tasks. By streamlining data collection and pre-processing, such a model would improve the effectiveness of various MIR processes and provide a scalable, adaptable framework for handling diverse musical styles, genres, and formats. Furthermore, semantic models offer the advantage of continuous integration, as they can be extended and adapted to accommodate new, unforeseen use cases while maintaining interoperability. This adaptability ensures that the model remains relevant and scalable as the field evolves, supporting a wide range of tasks and musical data formats. This raises the first research question: RQ1 Is it possible to design an ontology representing both contextand content-based music information, by extending and generalizing over existing models? What features shall this ontology include in order to account the diversity of musical styles, genres, and formats? In addressing this fragmentation, we hypothesise that a unified semantic model can overcome limitations of existing ontologies by extending and generalising them across diverse musical domains. This model should aim to harmonise disparate data sources and provide a scalable, adaptable framework for supporting varied MIR tasks – thus setting the foundation for examining additional aspects of contentand context-based data integration. 5 i i “output” — 2025/10/28 — 13:54 — page 6 — #20 i i i i i i Chapter 1. Introduction 1.1.2 Harmonising Symbolic Data: Addressing Fragmentation in Harmonic Datasets A significant case where this challenge shows clearly is the case of harmonic annotations, where inconsistency across datasets, formats, and notations epitomises the broader issues within MIR. Harmony is a prominent dimension of music, also known as its “vertical dimension”. It is informally defined as “combining notes in music to produce a pleasing effect greater than the sum of its parts” [74]. Chords are the basic constituents of harmony, and sequences of them define the harmonic structure of a piece. A chord is defined as a simultaneous occurrence of several music sounds, producing harmony [165]. Depending on the notational system and the annotation conventions, a chord can be associated, for example, with a name or label. Chords form the basis of harmonic progressions that underpin much of Western music, defining its tonal structure and flow. Chord sequences and harmonic progressions provide a framework for understanding a piece’s key, tension, and resolution, making harmony an essential aspect for music analysis. Computationally, the automatic analysis of chord progressions has supported several tasks in information retrieval – from the detection of cadences [215], structures in music [60], to the introduction of harmonic similarity measures for cover song detection [227, 104], classification [305], and generation [67]. Available datasets containing chord annotations vary significantly across several dimensions. They differ in terms of data formats, with annotations stored in formats such as LAB,CSV,TXT, and MXL. The notational systems used for representing chords also vary widely, including Harte,Leadsheet,Roman numerals, and ABC notation. Finally, the datasets differ in the type of music representation they annotate, with some focusing on symbolic representation, where time is expressed in beats and measures, and others on audio data, where time is expressed in seconds. These variations underscore the need for harmonised data representations in MIR, highlighting the prominence and challenge of developing systems that are both syntactically and semantically interoperable in the context of harmony. Taking harmony as a use case, this thesis investigates a workflow to harmonise symbolic data annotated in different formats and syntaxes, addressing both contextand content-based information about music. The goal is to explore whether it is possible to create a standardised workflow to unify existing datasets and to facilitate the creation of new corpora of harmonised data. 6 i i “output” — 2025/10/28 — 13:54 — page 7 — #21 i i i i i i 1.1. Problem Definition and Research Questions This leads to a key research question: RQ2 What strategies can be developed to create a large-scale, unified symbolic music dataset that standardises diverse digital formats and annotation practices, enhancing consistency and accessibility for music analysis? This pursuit of a harmonised symbolic dataset not only seeks to mitigate the fragmentation inherent in current music datasets but also aims to improve the accessibility and utility of harmonic data in music computing. The expected contributions of this work include the development of methodologies that can effectively unify diverse datasets and the demonstration of how such unified data can enable more sophisticated analyses and applications in a wide range of MIR tasks. 1.1.3 Exploring Harmonic Similarity: Leveraging Large-Scale Corpora for Deeper Musical Understanding The creation of large, harmonised corpora of symbolic harmonic annotations opens new avenues for more comprehensive and scalable exploration of harmonic similarity at a scale that was previously unattainable. By unifying and standardising diverse datasets, we can now study harmonic similarity across a much broader range of music, encompassing various genres, historical periods, and styles. This unprecedented scale allows for more meaningful insights and deeper analysis of harmonic content, which were difficult to achieve with smaller, fragmented datasets. Since the advent of MIR, similarity has served as a fundamental paradigm for organising musical datasets. Large music collections require effective organisation to support meaningful exploration, enabling users to uncover new connections and insights. As Pampalk notes, “the value of a large music collection is limited by how efficiently a user can explore it” [297]. While contextual metadata is crucial for structuring music collections, content-based similarity provides an objective framework for comparing pieces independently of metadata, which is often inconsistent or incomplete in music datasets. Moreover, by comparing the harmonic structures of many pieces, it becomes possible to abstract rules and processes that define a particular musical style or genre. The analysis of these large-scale datasets can also improve our understanding of the generative processes behind music, revealing the evolutionary paths that 7 i i “output” — 2025/10/28 — 13:54 — page 8 — #22 i i i i i i Chapter 1. Introduction have shaped its development over time [393]. This deeper understanding not only contributes to the study of musicology but also has practical applications, such as generating new music based on learned patterns or refining algorithms for music classification and recommendation systems. Most content-based music similarity research has focused on audio data [361, 124]. However, a significant drawback of these approaches lies in their reliance on end-to-end algorithms, which often fail to provide interpretable explanations for why certain tracks are considered similar. This lack of transparency can result in biased similarity measures and obscure the commonalities between distinct tracks, leading to challenges in understanding the reasoning behind the outcomes [231]. An alternative to these audio-based methods is symbolic similarity, which offers a more explainable and interpretable approach. Over the past decade, symbolic music similarity has been applied to a variety of tasks, including cover song detection [101], genre classification [12], variation recognition [157], music search [84], and plagiarism detection [408]. While melodic similarity has received substantial attention in this domain, the study of harmonic similarity has not garnered as much focus in recent years. To the best of our knowledge, the state-of-the-art methods in this area include the Tonal Pitch Step Distance (TPSD) [104] and the Chord Sequence Alignment System (CSAS) [175]. Moreover, available approaches to harmonic similarity tend to consider tracks as similar only when their global harmonic profiles align, offering limited insights into local harmonic similarities and hindering the possibility of exploring shared patterns among songs. A further objective of this thesis is to develop a more nuanced and scalable approach to harmonic similarity. By leveraging symbolic harmonic annotations, the aim is to explore methods that account for both global and local harmonic structures. A key goal is to demonstrate that large corpora can support this type of research by enabling comprehensive, scalable studies of harmonic similarity that were previously not possible with fragmented data. In this perspective, similarity measures, together with access to large corpora of harmonic data, open up new possibilities for creating exploratory tools that support musicological research and creative applications. Such tools enable researchers to navigate and analyse extensive harmonic datasets, revealing hidden patterns, connections, and trends across diverse musical works, genres, and historical periods. Additionally, these similarity measures may offer practical applications for composers, inspiring them with harmonic ideas drawn from a broad 8 i i “output” — 2025/10/28 — 13:54 — page 9 — #23 i i i i i i 1.1. Problem Definition and Research Questions range of genres and styles and sparking creativity through harmonic suggestions that may not be immediately apparent. This leads to the third research question: RQ3 How can novel harmonic similarity measures be developed to capture both global and local harmonic structures? How can these similarity measures be applied to large corpora to support scalable musicological research and assist creative applications? This research aims not only to establish advanced similarity measures that make full use of harmonised, large-scale corpora but also to demonstrate how these enriched datasets can drive innovation in both analytical and creative MIR applications. By exploring these avenues, we aim to validate the potential of integrated data as a foundational resource for improved musicological analysis and creative exploration. 1.1.4 Limitations of Symbolic Data Integration: A Multimodal Approach Despite significant progress in symbolic data integration, large harmonic corpora still reveal limitations that hinder the full potential of data-driven approaches. Two key challenges persist: (i) the scarce diversity and balancing of the available harmonic datasets and (ii) the inherent ambiguity and subjectivity of chord annotations [302]. The first challenge refers to the lack of diversity and balance in the available datasets. For instance, the ChoCo dataset, although being the largest corpus of chord annotations to date in terms of size and diversity, is heavily skewed towards mainstream Western genres, with nearly 80% of the data derived from pop and rock music. This bias towards a few dominant genres is critical because chord vocabulary and harmonic structure can vary widely across musical styles, resulting in long-tail distributions that are notoriously challenging to model computationally [302], particularly in the context of Deep Learning (DL) applications. This issue, often referred to as chord vocabulary imbalance, restricts the system’s ability to accurately handle less frequent chords, further contributing to biased outcomes. Another critical issue that cannot be addressed via data integration is the ambiguity and subjectivity inherent in chord annotations. Annotators may interpret harmonic structures differently, leading to significant variation in the labelling of chord sequences [227]. This problem arises because defining a chord in a musical 9 i i “output” — 2025/10/28 — 13:54 — page 10 — #24 i i i i i i Chapter 1. Introduction context can be highly subjective. For instance, distinguishing between a chord sequence and a melodic line is often open to interpretation. Additionally, annotators might vary in the level of detail they focus on, such as whether to include rapid approach chords or arpeggiated chords. They may also be influenced by the instrument playing the harmonic line, such as piano or guitar, rather than focusing on the broader harmonic structure of the piece. Furthermore, a specific set of notes can be labelled differently depending on the context and the harmonic function the annotator identifies within the piece. A particularly famous case illustrating the subjectivity of chord annotation is the opening chord of The Beatles’ A Hard Day’s Night, which has generated decades of debate among musicians and analysts. Various interpretations have been offered, ranging from George Harrison’s description of an F chord with a G on top to other music theorists proposing G7sus4,G11sus4, or even Dmin11. These challenges highlight the need for a more comprehensive approach to analysing harmonic data, that goes beyond the integration of symbolic annotations alone. To address these issues, this thesis advocates for a multimodal approach, i.e. combining symbolic annotations and audio signals to capture multiple dimensions of musical content [66]. Multimodal integration offers several key advantages: 1. It enables the exploration of tasks that inherently require audio, such as Audio Chord Estimation (ACE). 2. It mirrors the human approach to chord transcription by combining both audio and symbolic data, providing an effective means to address interannotator agreement. 3. Audio data provides additional information, such as timbre, dynamics, and articulation, that symbolic data alone cannot capture. 4. This approach aligns with the growing trend in the MIR community to integrate symbolic and audio data, offering a more comprehensive understanding of musical content [358, 66]. To create a multimodal dataset that integrates symbolic chord annotations with audio signals, the first step is to retrieve the corresponding audio and align it with the symbolic data. However, only about 12% of the 20,000 annotated tracks in the ChoCo dataset are aligned with audio, highlighting the urgent need for a more efficient method to align audio with chord annotations. 10 i i “output” — 2025/10/28 — 13:54 — page 11 — #25 i i i i i i 1.1. Problem Definition and Research Questions An effective audio-to-chord alignment method would not only fill this gap but also enable the creation of new multimodal datasets. A promising approach is to leverage crowd-sourced chord annotations from platforms such as Ultimate Guitar1, e-chords2, and Chordie3, which collectively offer millions of annotated songs, particularly from underrepresented genres such as electronic, metal, hip hop, reggae, and country. These platforms present a valuable opportunity to expand MIR datasets beyond the mainstream Western genres. However, these repositories typically lack timing and duration information, making them unsuitable for MIR tasks that rely on temporal alignment between audio and symbolic data. Although various approaches in the literature focus on aligning audio with symbolic data—primarily using Dynamic Time Warping (DTW) algorithms [288]—none are specifically designed for aligning audio with chord annotations. Building on the case of harmonic annotations discussed in RQ2, this thesis proposes the development of a method for aligning symbolic chord annotations with audio, referred to as audio-to-chord alignment, and leveraging this approach to generate new multimodal data from crowd-sourced datasets. We intend to address the following research question: RQ4 Is it possible to align chord annotations with audio without prior time information, thus enabling the creation of enriched, multimodal datasets? 1.1.5 Limitations in Audio Chord Estimation: Chord Imbalance and Subjectivity The proposed multimodal dataset, which integrates homogenised symbolic chord annotations from diverse sources and aligns them with audio signals, holds significant potential for advancing Audio Chord Estimation (ACE) – a critical task in MIR. ACE automates the transcription of chords directly from audio recordings, offering a scalable solution for music transcription and analysis. Its applications are far-reaching, impacting fields such as music analysis, musicology, contentbased retrieval, and music education. 1https://www.ultimate-guitar.com/ 2https://www.e-chords.com/ 3https://www.chordie.com/ 11 i i “output” — 2025/10/28 — 13:54 — page 18 — #32 i i i i i i Chapter 1. Introduction discusses usage scenarios and applications, while data availability and licensing are covered in Section 4.6. The chapter concludes with a summary in Section 4.7. Chapter 5, Similarity, explores harmonic similarity. Section 5.2 reviews the state of the art in symbolic harmonic similarity. Section 5.3 introduces LHARP, an algorithm for harmonic similarity exploration. Harmory is presented in Section 5.4, discussing its segmentation and similarity algorithms, Knowledge Graph construction, and implications for computational creativity. The chapter concludes with a summary in Section 5.5. Chapter 6 discusses integrating symbolic and audio data. Section 6.2 reviews audio-to-score alignment and ACE methods. Section 6.3 presents ChordSync, a novel alignment technique, followed by an analysis of inter-annotator agreement in Section 6.4.1. Section 6.4 discusses ACE improvements using consonance-based label smoothing and focal loss mechanisms. The chapter closes with a summary in Section 6.5. Finally, Chapter 7 summarises the contributions of this thesis by revisiting the research questions presented in Chapter 1, discussing the solutions developed for each question, and analysing perspectives for future work. 18 i i “output” — 2025/10/28 — 13:54 — page 19 — #33 i i i i i i i i “output” — 2025/10/28 — 13:54 — page 20 — #34 i i i i i i i i “output” — 2025/10/28 — 13:54 — page 21 — #35 i i i i i i CHAPTER 2 Background This chapter provides the theoretical and methodological foundation for understanding the core contributions of this thesis. By drawing on established concepts in music theory, MIR, and KR, it highlights the challenges of music representation and retrieval, particularly in the context of harmonic analysis and multimodal data integration. The chapter begins by exploring fundamental concepts of music theory in Section 2.1, focusing on key concepts and definitions of music and their relevance to computational music analysis. In Section 2.1.2, the discussion covers the main musical dimensions (or facets [119]) such as melody, rhythm, and timbre. Harmony, a central focus of this thesis, is further detailed in Section 2.1.3, where the chords and harmonic structures are explored in greater detail. Following this, the chapter turns to Music Technology in Section 2.2, where the field’s evolution and key tasks are discussed. This section connects theoretical musical knowledge with computational tasks, laying the groundwork for the subsequent analysis of data representation. In Section 2.3, the chapter delves into the core topic of music representa21 i i “output” — 2025/10/28 — 13:54 — page 22 — #36 i i i i i i Chapter 2. Background tion, focusing both on the representation of metadata (Section 2.3.1) and music content (Section 2.3.2). The section then addresses the signal representations of music in Section 2.3.3, highlighting its strengths and the challenges it presents. This is followed by a discussion on symbolic representation in Section 2.3.4, which explores how symbolic music data like MIDI and MusicXML are used in computational tasks, and the complications introduced by fragmented notational systems. The final subsection, Section 2.3.5, focuses on KR, introducing semantic models and ontologies and their potential to address the multirepresentational challenges identified earlier. Lastly, in Section 2.4, the chapter covers the emerging field of multimodal approaches in MIR, with Section 2.4.1 providing a definition of multimodality and challenges to integrating symbolic and audio data. Through this exploration, the chapter establishes the necessary background for the thesis’ contributions, especially in areas concerning music representation, harmonic analysis, and multimodal integration. 2.1 Music Theory and Structure The purpose of this section is to provide the foundational music theory concepts that will serve as a reference throughout the thesis. These concepts are essential for the analysis and discussion of musical structures, particularly in the context of MIR. Music transcends time and cultural boundaries, yet each historic epoch, culture, and subculture has created its own unique way of expressing itself musically. This wide variety of expression gives rise to what is referred to as the “Multicultural Challenge” [119]. Music theory endeavours to define and explain what music is by offering a generalized representation, yet the diversity of musical expressions necessitates a variety of theoretical frameworks. These theories differ based on the specific music they aim to describe, the cultural and geographical context of the theorists, and the purposes for which the theories are intended [177]. This chapter will delve into the music theories used to analyse the Western artmusic tradition. While these theories provide profound insights into this particular musical repertoire, they offer limited perspectives on the broader, universal nature of music. It is important to note that this section is not intended to be a comprehensive 22 i i “output” — 2025/10/28 — 13:54 — page 23 — #37 i i i i i i 2.1. Music Theory and Structure reference for music theory. For readers seeking a more detailed and comprehensive account of the field, we refer to [235, 69, 177, 26]. The main aim of this section is to provide context for the topics discussed in the thesis, with a focus on the most explored aspects of music theory, such as harmony, while intentionally overlooking other topics and concepts that fall outside the scope of this research. 2.1.1 Fundamentals of Music Theory In this chapter, we delve into theoretical concepts that are rooted in the commonpractice period of Western music, which spans from the Baroque to the Romantic periods (ca. 1650. 1900). During these three centuries, compositions were often structured around a gravitational centre, a fundamental phenomenon known as tonality. This centre, or tonic, is typically a single pitch labelled using letters from Athrough G, with possible modifiers such as “flat,” “sharp,” “major,” and “minor.” Such centres of gravity in music, providing a point around which all pitches orbit, have been a part of musical structures since antiquity and continue to resonate in contemporary music across various genres, including film scores, popular and commercial music, folk music, and jazz [235], albeit in evolving forms. While the strict adherence to tonal centres characterized much of earlier Western music, modern compositions often experiment with these foundations, incorporating atonal structures, polytonality, and microtonal music which challenge the traditional roles of the tonic [177]. Despite these innovations, the concept of a central pitch remains influential, adapted and reinterpreted in genres ranging from minimalism to progressive rock and modern jazz. Pitch and Pitch Classes The term pitch describes the attribute of a sound–such as an individual musical note–that determines its position on a scale. In Western culture, pitch is perceptually recognized as the attribute that allows sounds to be judged as “higher” and “lower,” in the sense traditionally associated with musical melodies [313]. Pitch is perceived based on what the ear interprets as the fundamental frequency of the sound, even in cases where this frequency is an auditory illusion, such as with difference tones, and not actually present in the sound wave [184]. The frequency of a sound is determined by the rate of vibrations produced by the sound source, such as the plucked string of a violin or the vibrating reed of 23 i i “output” — 2025/10/28 — 13:54 — page 24 — #38 i i i i i i Chapter 2. Background Figure 2.1: Music score illustrating various enharmonic intervals, highlighting equivalent pitch pairs. a clarinet in response to airflow. These regular vibrations occur at a speed–often measured in Hertz (Hz)–which directly influences pitch: higher frequencies yield higher pitches and lower frequencies result in lower pitches. For example, a sound at 880Hz is typically perceived as higher than one at 40 Hz. When two pitches are related by a frequency ratio of 2 : 1, they are separated by an octave. Despite the difference in frequency, pitches separated by an octave are perceptually similar, leading to the convention of labelling them with the same letter name (Athrough G), distinguished only by a number that defines the octave (e.g., A3,A4) [235]. This cyclical nature of pitch perception is foundational in Western music theory, where the octave serves as a reference point for organizing pitches. The concept of pitch class encompasses all pitches that share a specific relationship, such as being separated by one or more octaves. This relationship is termed an “equivalence” because, within a particular musical context, pitches within the same class are considered interchangeable or equivalent [339]. Pitches within the same pitch class bear the same letter name irrespective of their octave designation. For instance, the pitch class “C” includes all Cs across various octaves (C1,C2,C3, etc.). In Western tonal music, there are twelve distinct pitch classes within each octave, corresponding to the twelve semitones of the chromatic scale. The concept of pitch class simplifies the manipulation and analysis of musical materials, especially in the study of harmonic progressions. In this context, the crucial aspect to consider is the relationships between pitches rather than the specific octaves in which they are played. Pitches that sound the same but are named differently are termed enharmonically equivalent. For example, as depicted in Figure 2.1, C#and Db are enharmonically equivalent, as are Ab and G#. Although C#and D♭ sound identical on a piano, they serve different functions within their respective musical contexts [69]. 24 i i “output” — 2025/10/28 — 13:54 — page 25 — #39 i i i i i i 2.1. Music Theory and Structure Figure 2.2: Visualization of the C major scale, displayed on both a piano keyboard and a musical staff, illustrating the corresponding notes across both representations and scale degrees. Scales A scale is defined as a sequence of pitches arranged in ascending or descending order [122]. The diatonic scale, predominant in Western music, is a seven-note sequence where each pitch, denoted by letters Athrough G, appears once per octave. The term “diatonic,” meaning “through the tones,” describes the division of the octave into seven steps, with the cycle completing by repeating the initial pitch at the octave [235]. The major scale, a primary example of a diatonic scale, adheres to a fixed pattern of whole steps (tones) and half steps (semitones), specifically arranged as W−W−H−W−W−W−H(see Figure 2.2). This configuration imparts the major scale with its characteristic bright and uplifting sound. Conversely, minor scales, known for their more sombre tone, typically feature a lowered third scale degree, distinctly altering their emotional effect [49]. Each note in the seven-tone diatonic scale is assigned a functional name, reflecting its role within the scale. Figure 2.3 shows these names for each degree in both major and minor scales [26]. The pattern of intervals in the major scale can be transposed to any starting pitch, a process known as transposition. For instance, transposing the C major scale to start on G results in the G major scale. 25 i i “output” — 2025/10/28 — 13:54 — page 26 — #40 i i i i i i Chapter 2. Background Figure 2.3: Scale degrees and their corresponding names within both major (C major) and minor (C minor) scales. Major and minor scales dominate the landscape of Western classical tonal music. However, six other scales, known as modes or Church modes, were prevalent in Medieval and Renaissance music (pre-1600) and continue to appear in modern folk and popular music [368]. Additionally, several less common scales also exist in Western music. These include the pentatonic scale, and nondiatonic scales such as the chromatic scale and the octatonic scale [235]. Intervals A fundamental concept in music theory is the interval, which refers to the distance between two pitches [247]. Intervals are designated by the number of diatonic notes (notes with distinct letter names) they span. For example, the distance from C to Fis a fourth because it spans four letter names: C,D,E, and F. Intervals that encompass one octave or less are classified as simple, while those extending beyond an octave are termed compound. Intervals possess qualities that further define their musical characteristics [235]. Intervals that include the tonic (keynote) and span to the fourth and fifth scale degrees of a major scale are termed perfect, as are unison and the octave, typically denoted with the letter P(e.g., P4for a perfect fourth). Intervals extending from the tonic to the second, third, sixth, and seventh degrees are classified as major, marked by an uppercase M(e.g., M6for a major sixth). When a major interval is decreased by one half step, it becomes a minor interval, achievable by raising the lower note or lowering the upper note, and is 26 i i “output” — 2025/10/28 — 13:54 — page 27 — #41 i i i i i i 2.1. Music Theory and Structure Figure 2.4: Illustration of various intervals showing diminished, minor, major, augmented, and perfect qualities. denoted by a lowercase m. Conversely, augmenting a perfect or major interval by one half step transforms it into an augmented interval, marked by the letter A. Similarly, diminishing a perfect or minor interval by one half step changes it to adiminished interval, indicated by the letter d. Figure 2.4 displays examples of these interval types. The inversion of an interval involves switching the positions of the tones, with the lower tone becoming the higher one, or vice versa [247]. Intervals can manifest in two forms: if the tones occur in succession, the interval is considered melodic. If they sound simultaneously, the interval is harmonic. Harmonic intervals are further categorized as either consonant or dissonant. Consonant intervals, which sound harmonious and stable, are often utilized by composers at points of resolution. In contrast, dissonant intervals, characterized by tension and instability, are used to drive the music forward, demanding resolution and thus are unsuitable for concluding sections [368]. Tonality and Key The concept of tonality refers to the orientation of melodies and harmonies towards a referential (or tonic) pitch class. In the broadest possible sense, however, it refers to systematic arrangements of pitch phenomena and relations between them [205]. Each key is built upon a diatonic scale. In any given key, the tonic is the central pitch around which other pitches revolve, creating varying levels of tension and resolution [235]. The structure of the diatonic scale organizes these pitches in a 27 i i “output” — 2025/10/28 — 13:54 — page 34 — #48 i i i i i i Chapter 2. Background We further discuss the challenges that can be encountered when dealing with the bibliographic facet in Section 2.3. Harmonic Facet The harmonic facet of music, due to its significant relevance in the context of this thesis, warrants a dedicated section (c.f. Section 2.1.3). 2.1.3 Focus on Harmony Harmony is a prominent dimension of the Western tonal music, also known as the “vertical dimension”, which is concerned with “combining notes in music to produce a pleasing effect greater than the sum of its parts” [74]. Harmony is a widely studied component in music theory [312, 351], and music analysis[188]; where functional harmony provides a set of rules for moving to and from the tonic – the most stable note in a piece, allowing to relate chords to each other, and to the main harmony. Moreover, harmony, along with metrical structure is ubiquitous: roughly speaking, every piece, in fact every moment of every piece, has a metrical structure and a harmonic structure [379]. The foundations of modern harmony began to take shape during the thirteenth century with the development of organum, progressing through the medieval period into the Renaissance. During the Renaissance (1450–1600), the concept of harmony evolved through the study and practice of counterpoint, which involved the interweaving of independent melodic lines [344]. Counterpoint emphasized the consonant and dissonant relationships between these lines, laying the groundwork for the vertical (harmonic) perspective of combining notes. This exploration of consonance and dissonance within multiple melodic layers became a cornerstone of harmony, as theorists sought to formalize the principles governing these relationships [70]. The Baroque period (1600–1750) saw the development of accompanying a melody with chords through the figured bass system [406], where keyboard performers improvised an accompaniment from a given bass line marked with symbols indicating the chords to be played. This practice was predominant throughout the Baroque era, not only in keyboard accompaniments but also in solo songs and ensemble compositions. The seminal work in the theory of harmony was Jean-Philippe Rameau’s “Traité de l’harmonie”, which introduced and discussed the inversion of chords, a concept that has significantly influenced theoretical 34 i i “output” — 2025/10/28 — 13:54 — page 35 — #49 i i i i i i 2.1. Music Theory and Structure Figure 2.8: Examples of triads: augmented, major, minor, and diminished, illustrating the distinct interval structures that define each type. perspectives in music [26]. Chords Chords are the basic constituents of harmony, which jointly define the harmonic structure of a piece. Individually, a chord is defined as a “simultaneous occurrence of three or more music sounds, producing harmony” [70]. Depending on the notational system and the annotation conventions, a chord can be associated with a name, or label. Atriad, the simplest form of a chord, comprises any three-tone combination. It is identified by the root note upon which it is built—for example, a “C major triad” consists of the root C, along with a major third and a perfect fifth above it. Triads are classified into four types based on their quality: major,minor, diminished, and augmented. The arrangement of these intervals defines the overall sound and character of the triad [26]. Figure 2.8 shows examples of each of such triads. The position of a triad within a piece, whether it appears in root position or as an inversion, significantly influences its harmonic function. In root position, the root note is the lowest pitch, while in inversions, either the third or fifth takes the lowest position, changing the chord’s texture and perception. Further complexity is added by extending these triads to form larger chords such as seventh,ninth,eleventh, and thirteenth chords, each stacking intervals on top of the original triad [165], as shown in Figure 2.9. The seventh chord, for instance, adds a note a third above the triad’s fifth, 35 i i “output” — 2025/10/28 — 13:54 — page 36 — #50 i i i i i i Chapter 2. Background Figure 2.9: Examples of chord extensions: Illustration of a major triad and its progressive extensions to seventh, ninth, eleventh, and thirteenth chords, built on the same root note. creating a distinctive interval of a seventh from the root in its root position. The dominant seventh chord, built on the fifth scale degree of the major and minor scales, combines a major triad with a minor seventh, contributing to its pivotal role in tonal harmony. Each type of seventh chord possesses a unique sound, dictated by the combination of the triad type and the seventh interval [70]. In musical analysis, Roman numerals are employed to identify and differentiate triads and their inversions based on their scale degrees. These analytical symbols help in understanding the placement and function of chords within the overall key structure. Additionally, figured bass notation, a system from the Baroque period, uses figures placed under a bass line to specify the intended harmony, streamlining the notation process for continuo players and emphasizing key harmonic elements [312]. Harmony not only structures the musical piece but also deeply influences its emotional impact. Major triads are often associated with positive emotions, while minor triads convey negative feelings. Diminished triads suggest tension and suspense, and augmented triads evoke a sense of unease or mystery. These emotional associations highlight the powerful affective role of harmony in music, resonating with listeners from diverse cultural backgrounds [225, 74]. Harmonic Progression Harmonic progression refers to the sequence in which chords succeed each other within a musical composition. Historically, from the Baroque through the Classical and Romantic periods, composers have utilized harmonic progression as a primary organizational tool. The transition from one chord to another not only propels the music forward but also adds a dynamic element that is distinct from 36 i i “output” — 2025/10/28 — 13:54 — page 37 — #51 i i i i i i 2.1. Music Theory and Structure Figure 2.10: Illustration of a basic I−V−Iharmonic progression, showing the movement from tonic to dominant and back to tonic, creating a sense of resolution. the contributions of melody or rhythm alone [177]. In tonal harmony, which pervades various musical styles, the structure of a piece is largely shaped by its chord progressions. In tonal music, the tonic chord represents the pinnacle of stability. Chords that deviate from the tonic generally introduce tension, while those that return to the tonic tend to resolve this tension, providing a sense of fulfilment. Consequently, tonal compositions typically conclude with the tonic triad, and often commence with it as well. The relationships between successive chords are governed by two principal forces, both related to the roots of the chords: (i) the relationship of the chords to the prevailing tonality, and (ii) the intervals formed by the roots of adjacent chords. Triads built on each scale degree relate back to the tonic triad, which serves as both the point of rest and the ultimate goal of harmonic progression. To fully understand harmonic movement, it is insightful to analyse chord progressions based on the interval between the roots of consecutive chords [26]. By examining the intervallic relationships between the roots of successive chords, harmonic progressions can be grouped into meaningful categories. One of the most fundamental progressions in tonal harmony involves the dominant harmony leading back to the tonic, which establishes a sense of closure. The I−V−Iprogression, often written in this standard notation, begins on the tonic (providing a state of stability), moves to the dominant (which has a strong tendency to resolve back to the tonic), and finally returns to the tonic, offering a conclusive sense of arrival and completion [368]. Harmonic Cadences Aharmonic cadence serves as musical punctuation, providing closure to a phrase or section of music. Cadences can vary in strength, 37 i i “output” — 2025/10/28 — 13:54 — page 38 — #52 i i i i i i Chapter 2. Background with some signalling the end of a complete musical thought (comparable to a period) and others suggesting continuation, akin to a comma [368]. Most cadences involve either the Vor Ichord, with the dominant chord frequently appearing as a seventh chord (V7). Several types of cadences are commonly encountered in Western tonal music [26]: •Perfect Authentic Cadence (PAC): This is the strongest cadence, consisting of a V−Iprogression in major keys (or V−iin minor keys), with both chords in root position and the tonic note as the highest sounding pitch in the final chord. The PAC provides a powerful sense of finality. •Imperfect Authentic Cadence (IAC): This is a slightly weaker form of the PAC. It occurs when one or both chords are inverted, when the highest tone in the tonic chord is not the tonic itself, or when the diminished seventh chord (viio6) substitutes for the Vchord. •Half Cadence (HC): The half cadence ends on the dominant (V) chord, creating a sense of pause that anticipates continuation. Common forms include I−V,IV −V, and ii −V, with the Phrygian half cadence (iv6-V) as a notable variant in minor keys. •Plagal Cadence: The plagal cadence moves from IV to Iin major keys (or iv to iin minor keys). This cadence provides a softer sense of resolution than the authentic cadence. •Deceptive Cadence (DC): In this cadence, the progression begins with Vbut resolves to a chord other than I, typically the vi chord in major keys (V I in minor). This progression provides an unexpected twist, delaying the sense of finality. •Rhythmic Cadence: Phrase endings frequently feature characteristic rhythmic patterns that emphasize the cadence. These rhythmic cues add to the sense of closure or pause in the music. These cadences, especially the V-I relationship in the PAC, are fundamental to the experience of resolution in Western tonal music. Cadences help to shape musical phrases, each resembling a self-contained musical thought, with clear beginnings and endings. 38 i i “output” — 2025/10/28 — 13:54 — page 39 — #53 i i i i i i 2.2. Music Technology 2.2 Music Technology 2.2.1 Historical Background The convergence of music and computational sciences has significantly reshaped the ways we study, interpret, and engage with music. In recent decades, progress in computational methods has provided researchers with powerful tools to analyse extensive music collections and uncover structural aspects of music with unprecedented depth. These advancements have enabled the examination of vast quantities of musical data—surpassing traditional human limitations–and have transformed fields such as musicology, music analysis, and creative music applications. This transformation began in the mid-20th century when the first computational applications to music were conceptualised in the 1950s and gradually implemented during the 1960s [30]. At that time, computational models enabled researchers to address challenges in music analysis that traditional methods could not manage—such as identifying patterns across large datasets, conducting statistical analyses, and developing formalised representations of music. By the late 20th century, the digitisation of music and the rapid growth of digital archives accelerated the demand for intelligent systems capable of managing and retrieving musical information. This development laid the foundation for MIR, a field dedicated to creating models for handling and analysing vast digital music collections. MIR has developed as a distinct field of research from the 1990s onwards, culminating in the establishment of the International Society for Music Information Retrieval (ISMIR)1, a dedicated conference series at the turn of the millennium. The field addresses unique challenges in music retrieval, adopting since its foundation an interdisciplinary approach: MIR is a [...] interdisciplinary research area encompassing computer science and information retrieval, musicology and music theory, audio engineering and digital signal processing, cognitive science, library science, publishing, and law. Its agenda, roughly, is to develop ways of managing collections of musical material for preservation, access, research, and other uses. [119] Its goal is to manage and utilize musical materials for various purposes like preservation, access, and research. Content analysis, which involves the automatic 1https://ismir.net/ 39 i i “output” — 2025/10/28 — 13:54 — page 40 — #54 i i i i i i Chapter 2. Background extraction of music descriptors directly from audio, along with the development of innovative interfaces and infrastructure, are central to MIR’s objectives [221]. 2.2.2 Music Information Retrieval Tasks MIR integrates various paradigms for accessing music information, tailored to suit different user interactions and needs, as analysed by Knees & Schedl [221]. These paradigms include: •Retrieval: This paradigm involves users actively expressing a specific musicrelated need through a query, which can be in forms such as text, symbolic music representation, or audio. The system retrieves and potentially ranks results—audio clips, scores, or metadata—corresponding to the query. This method aligns with traditional information retrieval but is specialized to accommodate the distinctive characteristics of music data. •Browsing: Unlike retrieval, browsing allows users to explore a music collection without a predetermined goal. This paradigm supports an interactive and iterative process facilitated by intuitive user interfaces, enabling users to discover music items serendipitously. •Recommendation: In this paradigm, the system proactively filters and suggests music items based on the user’s past actions or stated preferences, which may be explicitly provided or inferred through behavioural patterns such as previous queries or playback history. Recommendations provide a personalized experience by predicting user preferences and offering music choices accordingly. These paradigms are applied in various contexts within the field of MIR, ranging from systems that allow queries by humming to sophisticated algorithms for theme detection using symbolic data representations like Musical Instrument Digital Interface (MIDI) [282]. Further, technologies such as music fingerprinting identify songs from brief audio clips, even in noisy environments, and cover song identification algorithms analyse components like melody and harmonic progressions to find different renditions of the same song [221]. While a single, unified taxonomy for MIR tasks is challenging and beyond the scope of this work, various resources provide substantial insights and methodologies for specific tasks. Textbooks such as [282, 241] and ISMIR conference 40 i i “output” — 2025/10/28 — 13:54 — page 41 — #55 i i i i i i 2.2. Music Technology proceedings are instrumental for gaining a deeper understanding of these tasks. Furthermore, [252] proposes a semi-comprehensive taxonomy of MIR tasks, categorized broadly as follows: •Tonality and Harmony: Mode, chord, and key detection. •Melody and Pitch: Melody estimation, pitch and multi-pitch detection, note tracking, automatic music transcription. •Rhythm: Onset detection, beat and downbeat tracking, metre estimation, tempo estimation. •Temporal Alignment: Score following, audio-to-score alignment, score alignment. •Source Separation: Musical instrument source separation, harmonic-percussive source separation. •Timbre-related Tasks: Musical instrument identification, playing technique detection. •Clip-level Classification: Music tagging, genre recognition, emotion/- mood recognition. •Content-based Audio Retrieval: Audio identification, audio matching, cover song detection. •Temporal Segmentation: Music detection, music structure segmentation, time boundary identification. •Visual Score Input: Optical music recognition and subtasks, including staff line identification and music symbol identification. •Performance-related Understanding: Technique identification, performer identification, performance assessment, difficulty estimation. While a comprehensive overview of all efforts and tasks within the MIR field is unfeasible, Ma et al. [252] highlight several methodological trends. They observe that the bulk of MIR research primarily utilizes audio as the input modality, with a smaller proportion relying on symbolic representations. The focus is predominantly on Western tonal music, relegating non-Western cultures and folk or traditional music to lesser prominence. A key differentiator among these tasks lies 41 i i “output” — 2025/10/28 — 13:54 — page 42 — #56 i i i i i i Chapter 2. Background in the required temporal granularity, ranging from clip-level classifications such as audio tagging to tasks demanding fine temporal resolution, such as pitch detection and onset detection. This variability in temporal demands, coupled with the global diversity of musical cultures, presents challenges in developing a universal music foundation model. Nevertheless, Ma et al. also note that the intrinsic connections among various tasks, such as the relationship between onset detection and beat tracking, could lead to the development of versatile music representations and MIR models capable of addressing multiple tasks, albeit potentially at the cost of reduced musical diversity. 2.2.3 Harmony in MIR Harmony plays a pivotal role in MIR as it forms the foundation of numerous analytical tasks that interpret the vertical dimension of music. Key harmonyrelated MIR tasks include: •Audio Chord Estimation (ACE): ACE involves the automatic detection of chords in audio recordings, returning a symbolic chord sequence that represents the harmonic structure. It is one of the fundamental tasks in MIR, with applications in music transcription, similarity, and analysis [259, 146, 410, 267, 411, 343]. •Key Detection: This task identifies the key of a musical piece, which provides the tonal context for interpreting harmonic progressions and melody [423, 59, 220, 133]. •Cadence Detection: Cadence detection identifies harmonic cadences, which mark the ends of musical phrases and are critical for understanding musical form and structure [33, 215]. •Harmonic Similarity: This task measures the similarity between harmonic progressions, facilitating tasks such as cover song identification and music recommendation [104, 3, 100, 175]. •Functional Harmonic Analysis: Functional harmonic analysis seeks to annotate chords with their functional roles (e.g., tonic, dominant), providing a deeper layer of harmonic meaning and enhancing the understanding of chord progressions [289, 61, 76, 331]. 42 i i “output” — 2025/10/28 — 13:54 — page 43 — #57 i i i i i i 2.3. Music Representation •Harmonic Change Detection: This task detects points in a piece where significant harmonic changes occur, which can reveal shifts in mood, tension, or form within the music [329, 108]. •Harmonic Predictive Modelling: In predictive modelling, systems are trained to anticipate upcoming harmonic structures, often used in generative applications or music prediction algorithms [82, 402]. These harmony-related MIR tasks are not solely ends in themselves; they also play a critical role in enhancing broader MIR paradigms, such as retrieval, recommendation, and browsing, by enabling more sophisticated and contextually rich interactions with music data. Moreover, harmony analysis contributes to improving other MIR tasks, including version identification [345], genre classification [201], and music structure segmentation [60], enriching the scope and depth of computational music analysis. 2.3 Music Representation A foundational aspect of MIR research is the representation and structured organisation of music, which serves as the basis for different methods of accessing and interacting with musical content. Historically, the paradigms in MIR have revolved around two primary forms of music information access: context-based MIR and content-based MIR [221]. •Content-Based MIR focuses on the direct analysis of the audio signal itself, covering elements that can be extracted such as rhythm, timbre, melody, harmony, and even the mood of the music piece. These aspects represent the core components of a music piece that contribute to its unique auditory signature and perceptual impact. •Context-Based MIR involves aspects of music that cannot be directly derived from the audio signal. This includes metadata such as reviews, liner notes, album artwork, country of origin, recording decade, and marketing strategies. These elements, often termed as cultural features, community metadata, or context-based features, provide a broader cultural and contextual perspective on the music, influencing how it is perceived, understood, and valued across different listener communities. 43 i i “output” — 2025/10/28 — 13:54 — page 50 — #64 i i i i i i Mel-Frequency Cepstral Coefficients (MFCCs) are a critical audio feature extraction technique that encapsulates the characteristics of human speech perception [65] (c.f. Figure 2.11b). Similarly to Log Mel Spectrograms, the process initiates with the conversion of audio signals to the frequency domain via FFT, followed by the application of STFT for analysing temporal changes. Next, Mel filters are employed to emulate the non-linearity of human ear. The computation of MFCCs involves taking the logarithm of the energies in each Mel filter, followed by a Discrete Cosine Transform (DCT). Owing to their effectiveness in capturing the configurations of the vocal tract, MFCCs are predominantly used in speech recognition and speaker identification applications [385, 253]. Constant-Q Transform (CQT) enhances audio processing in music analysis by providing a log-frequency spectrogram aligned with the musical scale [1] (c.f. Figure 2.11c). Unlike the linear Fourier Transform, the CQT operates on a logarithmic scale that reflects the exponential nature of musical pitch, facilitating the extraction of note frequencies. A distinctive feature of CQT is that the ratio of centre frequency to bandwidth remains constant (denoted by Q), which allows variable filter lengths that optimize performance. While not as popular as the log Mel spectrogram, the CQT is beneficial for tasks such as music Representation Learning (RL) [244], particularly when related to harmony [301, 211]. Furthermore, to better capture harmonic relationships in music audio signal, the Harmonic Constant-Q Transform (HCQT) was introduced. The HCQT is a three-dimensional array, indexed by harmonic, frequency, and time: H[h, t, f], where it measures the h-th harmonic of frequency fat time t. The index h= 1 represents the fundamental harmonic, and H[h]denotes the h-th harmonic of the base CQT H[1]. Chroma Features or chromagrams, capture the twelve pitch classes of Western music, distilling the harmonic essence of a composition regardless of timbre or instrumentation (see Figure 2.11d). As described in Section 2.1, human perception of pitch is periodic; pitches differing by an octave are perceived as similar in “color” and play a related harmonic role. This perceptual property allows each pitch to be divided into two components: tone height (the octave number) and chroma (the pitch spelling attribute within the set {C, C#, D, ..., B}). Enumerating chroma values, this set can be mapped to [0 : 11], where 0corresponds to chroma C,1to C#, and so forth. A pitch class, then, is the set of all pitches sharing the same chroma, such as {..., C0, C1, C2, C3, ...}, and for simplicity, the terms i i “output” — 2025/10/28 — 13:54 — page 51 — #65 i i i i i i 2.3. Music Representation (a) Mel Spectrogram (b) MFCCs (c) Constant-Q Transform (CQT) (d) Chroma Features Figure 2.11: Audio features extracted from the first 15 seconds of “Do I Wanna Know?” by Arctic Monkeys. (a) shows the Mel spectrogram; (b) depicts the MFCCs; (c) presents the Constant-Q Transform (CQT); and (d) displays the Chroma features. 51 i i “output” — 2025/10/28 — 13:54 — page 52 — #66 i i i i i i Chapter 2. Background chroma and pitch class are often used interchangeably. Chroma features aggregate spectral information associated with a pitch class into a single coefficient. Formally, given a pitch-based log-frequency spectrogram YLF :Z×[0 : 127] →R≥0as defined in equation (3.4), a chromagram Z×[0 : 11] →R≥0can be derived by summing all pitch coefficients that share the same chroma: C(n, c) := X {p∈[0:127]:pmod 12=c} YLF(n, p)(2.1) where c∈[0 : 11] represents each chroma class. Because of their ability to distil music into pitch class profiles, chroma features are effective in identifying key harmonic elements like chords, key signatures, and modulations. Their alignment with the equal-tempered scale of Western music makes them particularly valuable for applications in melodic transcription [22] and chord recognition [146]. However, chromagrams can exhibit noise in lowerfrequency regions, and harmonic overtones often spread energy across multiple chroma bands. For instance, when playing C3, the third harmonic resonates in the G4chroma band, and the fifth harmonic in E5[282]. 2.3.4 Symbolic Representation of Music Symbolic music representation encodes musical elements into structured, computerreadable formats, allowing detailed manipulation and analysis of musical data [255]. Unlike audio recordings, symbolic representations capture abstract musical aspects– such as pitch, rhythm, harmony, and structure–independently of any specific performance, making them versatile tools for analysis and retrieval. To be effective, Music Representation Systems (MRSs) must meet several essential requirements based on the nature of musical data. First, they need to be multi-dimensional, capturing both quantifiable elements like pitch and tempo, alongside qualitative elements such as performance instructions. This multidimensionality supports essential MIR tasks like interval detection and rhythmic grouping while also allowing for nuanced analysis and playback. However, symbolic systems must balance precision with flexibility, as traditional notation often only approximates the musical experience [74]. Data abstraction is critical in symbolic representation systems, enabling musical features to be encoded independently of specific units, such as Hertz for pitch 52 i i “output” — 2025/10/28 — 13:54 — page 53 — #67 i i i i i i 2.3. Music Representation Figure 2.12: Symbolic formats illustrating a B5 note in the key of Eb with a time signature of 3/4. Subplots include: (a) score representation, (b) LilyPond, (c) **kern, (d) ABC, (e) GUIDO, (f) MusicXML, and (g) MEI. or seconds for time [404]. This abstraction facilitates generalised musical operations foundational to MIR tasks and ensures that symbolic representations can be consistently interpreted across applications. Representing musical time and tempo in symbolic systems also poses challenges, as these elements need to balance the expressive temporal dynamics essential to music with the consistency required for reliable representation. Typical approaches use symbolic time abstractions that retain the key relationships among musical events, though synchronising real-time with metrical time remains an active research area [83]. For MIR and musicological applications, MRSs allow efficient sharing, retrieval, and collaborative analysis of music data. These systems range from general-purpose formats to specialised systems supporting tasks such as recording, analysis, or generative applications [403]. In this work, we examine MRSs that facilitate computational and analytical approaches, focusing on applications in MIR, musicology, and Deep Learning (DL) systems. Figure 2.12 presents the symbolic representations discussed in this work, showcasing diverse syntaxes used for encoding music data. Musical Instrument Digital Interface (MIDI) is a protocol developed to facilitate digital communication between musical instruments and computers [342]. 53 i i “output” — 2025/10/28 — 13:54 — page 54 — #68 i i i i i i Chapter 2. Background Initially introduced in the early 1980s, MIDI was conceived as a means to control electronic musical instruments remotely and in real-time, primarily by encoding performance information rather than actual sound [342, 13]. This standard is based on two basic messages: note-on and note-off, which encode with them details such as pitch, velocity, and timing, thereby functioning more as a digital score than an audio recording. MIDI files, formalized through the Standard MIDI File (SMF) format in 1988, allow for the exchange of MIDI data across different systems, preserving not only musical notes but also tempo and time signatures, which are vital for synchronization [106]. Despite its widespread use, MIDI faces limitations in its representation capabilities. For instance, MIDI encodes pitch based on a fixed, equal-tempered scale, and does not accommodate microtonal variations, which can be crucial in nonWestern musical contexts [252]. Moreover, the protocol’s focus on performance data–such as dynamics and articulations–makes it ideal for applications requiring control over musical expression but less so for those needing detailed notational information, like key signatures, chords, or structural details [22]. However, MIDI’s structure, which organizes data into a series of timed events rather than a continuous stream, allows for intricate manipulations of musical timing, albeit at the expense of more nuanced musical descriptions such as phrasing or the subtleties of harmonic relationships [404, 282, 241]. MusicXML stands as a pivotal format in the digital music domain [159], encapsulating the intricacies of musical notation within a universally transferable XML-based framework. Initially designed to serve as an online sheet music and music software exchange medium, MusicXML was aimed to do for music notation what MIDI did for electronic musical instruments. Unlike MIDI, which is primarily focused on performance data, MusicXML covers a broader spectrum, including both the visual representation of music scores and detailed encoding of musical elements like pitch, duration, and dynamics, thus ensuring both human and machine readability [282]. Rooted in the Extensible Markup Language (XML), MusicXML allows for a detailed representation of almost every musical element and offers flexibility in terms of data hierarchy, supporting both part-wise and time-wise score representations. This flexibility is enhanced by the use of XSLT3, which can alternate between hierarchical representations to simulate the structure of musical scores. 3https://www.w3.org/TR/xslt-30/ 54 i i “output” — 2025/10/28 — 13:54 — page 55 — #69 i i i i i i 2.3. Music Representation Such capabilities make MusicXML particularly suited for applications that span musical notation, performance, analysis, and retrieval. Despite its strengths, MusicXML is not without its limitations. The format is complex and verbose, which can introduce challenges in data processing and interpretation. While it excels in notational accuracy and interoperability between various music software, its application in training DL models for music is limited. Moreover, encoding or decoding MusicXML can be cumbersome for AI models that thrive on larger datasets and longer context lengths [252]. Music Encoding Initiative (MEI) is a community-driven, open-source endeavour to define a system for encoding musical documents into a machine-readable format [174]. Developed to preserve and share detailed information about musical notations, MEI utilizes an XML-based schema to encode both the intellectual and physical properties of music notation documents, thereby facilitating consistent search, retrieval, display, and exchange of musical data across various platforms [252]. The core of MEI is structured into 23 modules, each designed to encapsulate unique characteristics of musical source encoding, expressed in an XML metaschema language known as the “One Document Does-it-all” (ODD) format. This modular and extensible structure supports encoding a diverse range of music notation systems beyond just common Western notation, including mensural (Renaissance) and neume (Medieval) notations . In contrast to MusicXML, which primarily facilitates interchange between notation editors, one of MEI’s primary goals is to create a semantically rich model for music notation that goes beyond mere visual imitation to preserve the unique structure and semantics of each notation system. This enables the encoding of traditional facsimile, critical editions, and performance editions, promoting the development of extensive and international archives of notated music which serve as crucial resources for music editions, performances, analyses, and research [174]. However, the complexity, focus on detailed musical notation and emphasis on visual representation rather than audio features, the large data files, and limited integration with Machine Learning (ML) tools make MEI less suitable for DL applications that require more streamlined and uniform data formats [252]. LilyPond is a music notation software that emphasizes the creation of visually pleasing sheet music via a high-level description file [290]. It is similar to LaTeX in its approach, allowing users to input musical notation in plain text format 55 i i “output” — 2025/10/28 — 13:54 — page 56 — #70 i i i i i i Chapter 2. Background which is then compiled into professionally engraved scores. Developed with a comprehensive syntax that allows to cover a wide range of musical symbols and formatting options, LilyPond is capable of handling complex scores that require professional-quality output [252]. The software functions by reading and processing files that contain formal representations of the music, outputting to formats such as PostScript or PDF. The pitch in LilyPond is indicated using lowercase letters, and octave adjustments are made with apostrophes (’) or commas (,), with each symbol raising or lowering the pitch by one octave respectively. Alterations to pitch such as sharps and flats are added by appending ′is′for sharps and ′es′for flats to the note name. Durations are noted by their reciprocal values; for example, a quarter note is represented as ′4′and a half note as ′2′[290]. While it offers extensive control over musical notation, LilyPond’s complexity and focus on visual presentation can introduce challenges when being employed for MIR tasks, particularly in tasks like automatic transcription of music [252]. **kern is a symbolic music representation system developed as part of the Humdrum Toolkit, created by David Huron in the 1980s [204]. This system is designed to encode Western music notation efficiently, focusing on common practice music notation and is used for computational music analysis. The format represents pitch, rhythm, meter, and articulation in a clear and straightforward syntax that is both human-readable and easily processed by computers. The **kern format is notable for its extensibility and flexibility, enabling users to include detailed musical parameters and metadata, which is crucial for symbolic music analysis, pattern recognition, and music generation. Models using **kern can analyse musical structures, recognize stylistic features, and generate compositions that comply with specific musical conventions. In **kern, pitch information is represented using a combination of upper and lowercase letters that denote octaves: for instance, the lowercase ‘c’ represents Middle C (C4), while multiple ‘c’s (e.g., ‘ccc’) indicate higher octaves, and uppercase ‘C’ denotes lower octaves (e.g., ‘C’ for C3). Accidental symbols are also incorporated, with ‘#’ indicating a sharp, ‘-’ a flat, and ‘n’ a natural note. Durations are marked numerically, with ‘4’ for a quarter note and ‘8’ for an eighth, among others [204]. While **kern is highly effective for notational accuracy and computational analyses, its focus on the notational aspect may omit some expressive details 56 i i “output” — 2025/10/28 — 13:54 — page 57 — #71 i i i i i i 2.3. Music Representation found in performance data, such as dynamics and exact timing nuances [252]. ABC Notation was developed by Chris Walshaw in 1997 and is the de facto standard for folk and traditional music notation, offering a simple, text-based approach to represent music4. It is written using ASCII characters including letters, digits, and punctuation marks, which makes it highly accessible and widely adopted for sharing music online. ABC notation consists of two parts: a header that contains metadata such as the tune’s title, meter, default note length, key, and reference numbers, and a body that describes the actual musical content—notes, rests, bars, and other musical symbols. The notes in ABC are denoted using the English note names, with uppercase letters (A–G) representing the lower octave and lowercase letters (a–g) for the higher octave. Octave modifiers include the comma (,) to indicate lower octaves and the apostrophe (’) for higher octaves. For instance, “C,” represents a low C, while “c” indicates a high C. Rests are denoted by ‘z’ or ‘x’, and their duration can be modified similarly to notes. Musical nuances such as dynamics, articulations, and decorations are expressed using textual expressions enclosed in exclamation marks (e.g., !trill!) . ABC’s structure allows for straightforward conversion between ABC and other music notation formats, notably MIDI. This facilitates its use in a wide range of applications, from educational tools to software that performs complex tasks like automatic transcription and music generation. For example, the software package ABC Music includes tools like abc2midi, which converts ABC files to MIDI, supporting features such as multivoiced files, guitar chord expansion, voice transposition, and percussion accompaniment. Despite its simplicity, ABC notation’s ability to encode detailed musical information makes it suitable for computational musicology and the development of music Artificial Intelligence (AI) models. It’s particularly valued in projects that require converting textual descriptions into symbolic music notation, owing to its compatibility with natural language notations. This compatibility enhances the potential of ABC notation in training text-to-symbolic music models, such as those aimed at generating music from textual descriptions [252]. GUIDO Music Notation (GMN) is designed to be a robust, yet straightforward format for encoding musical scores. Developed with simplicity and computational efficiency in mind, GMN offers a flexible method for representing a wide 4https://abcnotation.com/ 57 i i “output” — 2025/10/28 — 13:54 — page 58 — #72 i i i i i i Chapter 2. Background range of musical styles and complexities [196, 195]. It uses a clear and concise syntax to denote pitches, durations, dynamics, articulations, and other musical elements. The hierarchical structure of GUIDO allows for well-organized musical information, making it accessible for various digital music applications. The main advantage of GMN is its balance between human readability and machine processability. This balance facilitates tasks such as symbolic music analysis, composition, and pattern recognition in ML models. These models can leverage GUIDO’s structured format to explore musical structures, generate new compositions, and identify stylistic patterns across genres. Moreover, the readability of GMN aids in debugging and data interpretation during model development and evaluation phases. However, GMN’s simplicity might come at the cost of omitting detailed expressive nuances, which are often necessary for complex performance analysis and intricate compositions, making it potentially less detailed than formats like MusicXML or MEI. 2.3.5 Knowledge Representation of Music Given the complex and multifaceted nature of music, knowledge representation offers a structured approach to capture and model its conceptual and relational elements. These approaches go beyond simple data encoding, which focuses on technically structuring music data into formats that are readable and analysable by computer systems. While data encoding aids retrieval and basic analysis, knowledge representation seeks to model deeper, conceptual aspects of music, such as genre, instrumentation, and notation, in a structured format [75]. This distinction underscores that data encoding primarily deals with technical readability, while knowledge representation addresses the complexity of musical interpretation, relational structures, and high-level descriptions that cannot be derived directly from raw audio or symbolic data alone. The field of knowledge representation has its origins in mid-20th-century logic and artificial intelligence, with early developments like semantic networks and symbolic logic [210]. Today, the Semantic Web represents a prominent tool for advancing these ideas, offering a framework for creating machine-interpretable data on a global scale. Semantic Web technologies, including ontologies and linked data, play a pivotal role in representing and linking music-related knowledge. Ontologies provide for58 i i “output” — 2025/10/28 — 13:54 — page 59 — #73 i i i i i i 2.3. Music Representation mal definitions of the concepts and relationships within a given domain, allowing data to be tied to its meaning in a standardized way [324]. This approach enables diverse data sources–ranging from manually annotated music scores and editorial data to social and automatically generated content–to be logically interconnected, moving beyond simple text-based searches to context-aware, semantically driven queries [135]. In the field of MIR, these representations are crucial. By enabling the integration of heterogeneous datasets and allowing for sophisticated, context-rich queries, Semantic Web technologies bring new depth to music research, offering a distributed knowledge environment that fosters interoperability and enriches musical analysis [52]. Fundamentals of Semantic Web Technologies The Semantic Web, pioneered by Tim Berners-Lee, aims to transform the Web from a collection of documents into an interconnected web of data, enabling machines to process and interpret information in ways that enhance interoperability, data integration, and automated reasoning [29]. This transformation relies on standardized frameworks for structuring, linking, and querying data, with Resource Description Framework (RDF), Web Ontology Language (OWL), and SPARQL forming the core technologies essential for creating and managing knowledge graphs and ontologies. Knowledge Graphs The term KG has a long history, with origins dating back to at least 1972 [350]. However, its modern usage gained traction following Google’s 2012 announcement of the Google Knowledge Graph [362], which was subsequently adopted by major tech companies, including Amazon [233], IBM [112], Uber [173], and others. At the core of KGs is the idea of using graph-based structures to represent and integrate diverse data sources, with the goal of accumulating and conveying knowledge about the world [293]. In KGs, nodes represent entities, while edges represent relationships between these entities, offering an intuitive and flexible abstraction for complex, interconnected data. KGs are particularly advantageous for tasks that require integrating, managing, and extracting value from diverse data sources, as they allow flexible schema design, support handling incomplete data, and are well-suited for domains with intricate relationships, such as social networks, biological data, bibliographic records, and transport systems [11]. 59 i i “output” — 2025/10/28 — 13:54 — page 66 — #80 i i i i i i i i “output” — 2025/10/28 — 13:54 — page 67 — #81 i i i i i i CHAPTER 3 Representing Musical Knowledge 3.1 Introduction Musical heritage encompasses a diversity of human expressions and experiences, leaving heterogeneous traces that are difficult to describe, connect, and preserve [178]. Western music cultural heritage developed through varied sources: musical contents and objects (such as tunes, scores, melodies, notations, recordings, etc.) linked to tangible objects (theatres, conservatoires, instruments, etc.) but also to their cultural and historical contexts, opinions and stories told by people with diverse social and artistic roles (scholars, writers, students, intellectuals, musicians, politicians, journalists, etc.), and facts expressed in different styles and perspectives (memoire, reportage, news, biographies, reviews) in different languages (English, Italian, French, Spanish, and German) and across centuries [41]. This diversity creates unique opportunities as well as challenges for researchers and practitioners attempting to study and preserve music heritage. However, the fragmented nature of the data limits our ability to fully understand the cultural significance and historical trajectory of musical works. As such, 67 i i “output” — 2025/10/28 — 13:54 — page 68 — #82 i i i i i i Chapter 3. Representing Musical Knowledge there is a pressing need for frameworks that can integrate both musical content and context, enabling better preservation, study, and dissemination of music in its full richness. 3.1.1 Challenges and Requirements for Interoperability Music data can describe two main musical dimensions: content, which encompasses intrinsic elements like pitch, harmony, and rhythm, and context, which includes broader information such as cultural, historical, and publication metadata. Music metadata – also known as contextual, bibliographic, or documentary data – plays a crucial role in identifying and describing musical works, their creators, recordings, and performances. For the music industry, metadata is vital for efficiently managing and distributing music, supporting tasks such as search, recommendation, and cataloguing [296]. Accurate metadata ensures that artists are properly credited and compensated [363], and for musical heritage, metadata facilitates the preservation and dissemination of works across different cultural and historical contexts [156]. In this regard, metadata also promotes diversity and inclusivity by highlighting lesser-known genres and artists, thereby fostering a more comprehensive understanding of global musical traditions [105]. However, challenges arise due to the inconsistency and fragmentation of metadata across various systems and musical traditions, each with unique conventions for describing elements like composers, performers, or works. For example, the term “composition” in classical music may diverge from the concept of “track” or “song” in popular music, leading to fragmentation that hampers interoperability between systems and datasets. Alongside metadata, musical content itself – whether in the form of symbolic data (such as music scores, or MIDI files) or audio recordings – poses its own challenges for computational representation (see Section 2.3). Audio data is unstructured and requires extensive feature extraction, while symbolic data is more structured but often not as expressive as audio signal [395]. Moreover, numerous audio and symbolic representations have been proposed, each tailored to specific tasks or genres. However, interoperability among these representations is often limited. This has led to significant data fragmentation issues in the MIR field, with small datasets structured in varied formats to suit specific tasks and applications. This fragmentation poses two main challenges: first, it demands substantial 68 i i “output” — 2025/10/28 — 13:54 — page 69 — #83 i i i i i i 3.1. Introduction resources for collecting, harmonising, and preprocessing disparate data; second, it hinders comparison and reproducibility, as findings from different datasets are often misaligned. As a result, achieving data interoperability is increasingly critical. Effective data integration must go beyond syntactic alignment (i.e., standardised formats and structures) to also ensure semantic consistency, preserving the meaning and relationships of musical elements across representations. Domain specificity hampers interoperability Existing ontologies for music data are typically tailored to specific use cases and requirements, limiting their general applicability. For metadata, Music Ontology (MO) [324] leans towards modelling discographic data with a focus on contemporary music, whereas DOREMUS [64] is inherently rooted in classical music. Nevertheless, when drifting from discographic data and classical music, or attempting to reuse both models, addressing e.g. cultural heritage requirements while fostering interoperability becomes difficult. Indeed, a model reflecting the view and the interpretations ascribable to a musical genre, stakeholder, or dataset type may be difficult to reuse and extend to other domains. For instance, a music artefact may originate from oral transmission or be the result of a creative process that does not necessarily entail a formal composition process. The latter is common in songwriting, but also in folk music whenever a set of tunes (collected from different manuscripts) allows for the identification of a tune family [391]. Similarly, when expressing relationships between musical artefacts (alias derivations), it is important not to impose any modelling bias that may constrain possible interpretations (e.g. an arrangement having proper musical identity vs simply providing a different instrumentation). This is commonly referred to as “dominance of concept” [64], whose definition should be left to users depending on their data and domain expertise. The same limitations apply to ontologies designed for music content data, which are often created to represent specific types of musical information. For example, some ontologies focus on the semantics of music notation [213], while others target data from particular representation systems, such as MIDI [272]. These focused approaches hinder interoperability between systems and formats, as they lack a flexible structure for integrating diverse types of musical content. Rather than attempting to achieve consensus on musical concepts and jargon, accounting for the interoperability calls for an abstraction layer for music data 69 i i “output” — 2025/10/28 — 13:54 — page 70 — #84 i i i i i i Chapter 3. Representing Musical Knowledge (“zoom-out”) that can then be specialised, extended, and adapted to address domain-specific requirements (“zoom-in”). Expressivity is needed at different levels Another requirement for interoperability and reuse across various data sources is providing expressivity at different degrees, i.e. the possibility to conveniently describe music data at the right level of detail. For example, one data source may have granular/detailed information that requires high semantic expressivity (a composition process spread over different time, places, and involving more artists); whereas others may have basic (only the name of an artist is known) or even incomplete and uncertain information (a composition tentatively attributed to an artist). For metadata, the WikiProject Music1has been successful in providing expressivity to represent music metadata from different sources. As an extreme case of ontological flexibility, the schema underlying Wikidata – an open-ended, multidomain KG built collaboratively like Wikipedia – is not specified in a previously agreed ontology, and the high expressivity overly adds complexity to the model. This is due to Wikidata’s scope being the most general. Provenance is fundamental for data integration Accounting for provenance is a central requirement for both cultural heritage and music industry. This becomes fundamental when integrating KGs from different datasets and stakeholders – as every single bit of data (each triple) should be attributable to a dataset/KGs. Furthermore, integrating provenance is also needed within the context of a single dataset, at least for claims and links. Claims-Interpretations Cultural heritage applications often require representing debatable statements or claims [86, 87]. These are usually the result of an interpretation process based on factual or documentary evidence (a dataset, a manuscript, etc.), and following a methodology and/or theory. Examples include personal information (e.g. the year/place of birth of a composer), and authorship claims (e.g. a composition being attributed to an artist). Links and identifiers These includes links to artists’ official websites, fan pages, discussion forums, music reviews, record shops; as well as identifiers from mu1https://en.wikipedia.org/wiki/Wikipedia:WikiProject_Music 70 i i “output” — 2025/10/28 — 13:54 — page 71 — #85 i i i i i i 3.1. Introduction sic databases (e.g. MusicBrainz, Discogs, AllMusic), streaming platforms (e.g. Deezer, Spotify), and authoritative sources (e.g. ISNI, ISWC, ISRC). As most links and identifiers are crowdsourced or automatically inferred by entity linking algorithms, modelling provenance here promotes traceability and accountability of data sources. 3.1.2 Towards a Unified Model for Music Representation The ontology engineering efforts described in this chapter have been conducted within the framework of the H2020 Polifonia project2. The Polifonia project brings together memory institutions, museums, music archives, scholars, commercial organisations, and citizens who ask complex questions (e.g. “Which tunes share melodic patterns and geographical origin?”; “How do libretto and music relate, e.g. in describing an emotion?”; “Can we trace the evolution of tonality and transition from modal to tonal?”) across these multi-perspective and multi-modal sources. This demands the integration of musicological (notes, chords, modes, theories), historical (events, persons, places, objects) and archival/preservation (metadata, descriptors) data and perspectives. The project comprises 4 cultural institutions (CNAM, NISV, MiC, KNAW) and 10 pilots with a large variety and number of requirements. Ontologies and KGs have the potential to overcome these challenges, and shed light on this wealth of resources by extracting, materialising and linking new music history knowledge that was previously overlooked and therefore missing [277, 43]. Although various ontologies have been proposed to model some aspects of musical heritage interest, they are individually insufficient to overcome the challenge of integrating the notation, metadata, and historical contexts needed for multiperspective cultural analyses; thus leaving questions about the relationship between musical theory (melodies, tonalities, chords) and culture (historical events, architecture, geography) unanswered. To date, no available ontological framework integrates music metadata, notation, annotation, source provenance, and cultural heritage object descriptions. To the best of our knowledge, no toolkits exist to support knowledge engineering tasks around the lifecycle of competency questions, which is a central project requirement given the large number of variety of stakeholders, pilots and questions. 2https://polifonia-project.eu/ 71 i i “output” — 2025/10/28 — 13:54 — page 72 — #86 i i i i i i Chapter 3. Representing Musical Knowledge 3.1.3 Our contribution In this chapter, we describe the Polifonia Ontology Network (PON), a set of new ontologies formalising the semantics of music representation, metadata, annotation, analysis, mediums of performance (instruments), and historical sources (provenance), enabling the creation of interoperable knowledge graphs from music datasets. These ontologies address RQ1 (c.f. Section 1.1.1) by establishing a unified framework for representing music-related data, enhancing interoperability and paving the way to the creation of harmonised KGs of musical data. The contributions detailed in this chapter have been published across several peer-reviewed works [94, 88, 52, 27]. To achieve this, we apply and extend XD [36], a well-known ontology design methodology where ontological requirements are gathered from a comprehensive inventory of CQs, and modularity is fostered through the reuse of ODPs [151, 192]. We also release the PolifoniaCQ dataset, a collection of 361 competency questions on musical heritage. Further, we validate PON and provide evidence of its current and planned (re)use by three different types of users: (i) the Polifonia pilots, using them to generate musical culture KGs; (ii) a number of industrial and institutional stakeholders and early adopters, planning to use PON to annotate their in-house datasets; and (iii) a survey run in the Semantic Web and Music Technology communities showing intentions of use. More specifically, the contributions of this chapter are as follows: •Extensions to XD centred around CQ extraction and enhancement, including both methodological (a CQ-elicitation framework to mirror use cases from domain experts) and technological (a toolkit for assisted design and iterative improvement of CQs through language models) aspects (Section 3.3). •PolifoniaCQ, a new dataset of competency questions driving the design and the evaluation of PON, with associated stories and personas (Section 3.3.1). •The Polifonia Ontology Network (PON v1.0) resources, available on GitHub3 and including 15 (CC-BY 4.0) ontology modules (Section 3.4). •Evidence of reuse and impact from music stakeholders, applications within Polifonia, and interest from various research communities (Section 3.6). 3https://github.com/polifonia-project/ontology-network 72 i i “output” — 2025/10/28 — 13:54 — page 73 — #87 i i i i i i 3.1. Introduction •Example-driven validation of the model, focused on the data elicited from four different stakeholders. •Code support to create Music Meta KGs without expert knowledge of the model, with automatic alignments to the MO, DOREMUS, and Wikidata. 3.1.4 Chapter Structure The chapter begins with a review of related works in the field of knowledge engineering for music, also including a discussion on methodologies and workflows for ontology engineering (Section 3.2). Section 3.3 outlines the methodology adopted for this work, which encompasses two key processes. First, requirement collection (Section 3.3.1) details the process of gathering the needs and expectations of various stakeholders involved in the Polifonia project. Second, Ontology Network Design and Development (Section 3.3.2) describes the design choices and development processes for creating a robust and scalable ontology network (PON) that integrates both music content and metadata. Section 3.4 provides an in-depth description of the Polifonia Ontology Network, focusing on its structure, the modules it incorporates, and how it supports the integration of diverse types of music data. A dedicated focus is given to the Music Meta Module (Section 3.5), which is specifically designed to handle the diverse and complex metadata requirements of musical heritage. Section 3.6 reviews the adoption and reuse of PON within and beyond the Polifonia project. Evidence of its use in Polifonia pilots is presented in Section 3.6.1, while Section 3.6.2 summarises interest in PON reuse based on a survey of the Semantic Web and Music Technology communities. Section 3.6.3 describes early PON adoption by Polifonia Stakeholder Network members and resulting synergies for validating and annotating cultural and industrial datasets. The chapter concludes with Section 3.7, which summarises ontology engineering achievements, discusses ongoing challenges, and outlines future directions for PON. 73 i i “output” — 2025/10/28 — 13:54 — page 74 — #88 i i i i i i Chapter 3. Representing Musical Knowledge 3.2 Related Work Ontologies play a fundamental role in the representation and management of knowledge, by providing common vocabularies to describe resources and queries. Several ontologies exist in the music domain for addressing diverse applications, dealing with both music content and metadata at different levels of specificity. MusoW [85] is a catalogue indexing online music resources, including ontologies and KGs. Here, we focus on music ontologies and categorise them according to their reference domain: (i) metadata; (ii) music theory; (iii) music notation; and (iv) audio features. In this section, we explore the key areas of ontology engineering in music, beginning with ontologies for describing both the context and content of music. Table 3.1 presents a taxonomy of music ontologies, categorised by their domain, scope, and the year of the latest release, providing an overview of the diverse landscape of existing models. Next, we examine the primary methodologies that have been developed for ontology engineering, focusing on their evolution from early frameworks to more collaborative and agile approaches. 3.2.1 Ontologies for Describing Music Context Ontologies such as the Music Ontology (MO) [324] and DOREMUS [249] play a foundational role in describing high-level metadata about musical works, composers, and performances. When looking at these ontologies, MO leans towards modelling discographic data with a focus on contemporary music, whereas DOREMUS is inherently rooted in classical music. These ontologies have been demonstrated to model metadata from MusicBrainz and BBC Music [326], and from classical music libraries and radio broadcasts for concerts programming [64], respectively. Their specificity makes them appealing when downstream applications show considerable overlap in terms of requirements and data. Examples include the reuse of MO in the WASABI project [45], to support the semantic annotation of audio music (emotions, lyrics, structures), but also for music recommendation [337] and listening [4]; and the adoption of DOREMUS by Philarmonie de Paris, Biblioteque National de France, and Radio France. Nevertheless, when drifting from discographic data and classical music, or attempting to reuse both models, addressing e.g. cultural heritage requirements while fostering interoperability becomes difficult. 74 i i “output” — 2025/10/28 — 13:54 — page 75 — #89 i i i i i i More specialised ontologies like the OMAC Ontology [347] provide an in-depth description of musical claims and interpretations, which are essential for musicological research. Other ontologies focus on modelling emotional responses to music. For instance, the COMUS Ontology [334] captures emotional states by integrating both contextual factors and user preferences, while UniEmotion [216] classifies tags into positive emotions, negative emotions, and factual descriptors, offering a structured approach to emotion-based music categorisation. Additionally, the Performed Music Ontology focuses on capturing detailed information about live performances, while the OnVIE Ontology [376] extends this to the mediums used in musical performances. The Musical Instrument Taxonomies [224] and the Smart Music Instrument Ontology [388] further contribute by providing conceptual models for the classification and description of instruments, especially in the context of the Internet of Musical Things [387]. Ontologies like the ArCo ontology [54] are also crucial for connecting music with its cultural and historical significance. By situating music within broader contexts of cultural heritage, such frameworks allow researchers to examine how different periods and styles influence musical creation and perception. Other related efforts include [43, 206], which aim to integrate music data into larger heritage databases. Despite their contributions, existing context-related ontologies face challenges in terms of scalability and interoperability. Many models, while rich in metadata, struggle to integrate with other ontologies due to a lack of standardised alignment practices. The Polifonia Ontology Network addresses these issues by aligning context models with existing web resources while ensuring that provenance information is consistently maintained across datasets [52]. 3.2.2 Ontologies for Describing Music Content In addition to context, other ontologies were modelled to capture intrinsic musical properties like music theory, audio signal features, and notation, serving as essential tools for computational analysis, musicology, and creative applications in MIR. Some ontologies describe different elements ascribable to music theory. The Music Theory Ontology (MTO) [332] provides a detailed framework for encoding theoretical music concepts, allowing researchers to model elements such as harmony, tonality, and rhythm. More specific ontologies, like the Functional Harmony Ontology [214], take this further by reasoning about harmonic sequences i i “output” — 2025/10/28 — 13:54 — page 82 — #96 i i i i i i Chapter 3. Representing Musical Knowledge modules shaping the architecture of the Polifonia Ontology Network. To streamline this process, we analysed the CQ embedding space generated and projected by IDEA. This is done by computing the sentence-level embeddings (a feature vector of fixed size) for each CQ in the PolifoniaCQ dataset. The latter can be considered as a point in a high dimensional space – providing a numerical summary of the question’s meaning [68]. Embeddings are computed via SentenceBERT [333] due to its state of the art performance on a number of question-related tasks, including multi-lingual search and paraphrase detection. An interactive visualisation of the PolifoniaCQ embeddings is available from a live Tensorboard Projector [365] which is set up and synchronised via IDEA7. The qualitative analysis of the embedding space, in addition to density-based clustering analysis under various parametrisations, have jointly facilitated the identification of common requirements (as nested clusters) and enabled the interactive exploration of the PolifoniaCQ dataset via similarity (c.f. Figure 3.2). Matching CQs to ODPs. For each module/ontology, an XD iteration starts from selecting a coherent set of CQs. To address those requirements, existing solutions (ODPs) from ontologies or online catalogues of patterns are considered for reuse, extension, and specialisation. For instance, a CQ such as “Where and when a situation took place?” can be matched to the TimeIndexedSituation8 ODP, which represents temporal situations. Here, IDEA supports the identification of “the CQ set” via the multi-lingual search feature. For example, an ontology designer looking for CQs related to places may express a search query as shown below in Listing 3.1. Listing 3.1: Search results for query “questions related to places” with similarity score. 10.377 Where were the places in which musicians played ? 20.368 Which are all organs near to geographic coordinates x, y? 30.341 What are geographically distinct features of organs from a region ? 40.287 Where is the church / bell tower ? 50.285 What is the provenance of the event attendees ? 60.275 Which tunes which share melodic patterns or geographical origin ? 70.265 What places did a musician visited in her career ? 80.263 Where is the Bell Tower ? 90.246 Where was a musical composition performed ? 10 0.238 In which buildings was a musical composition performed ? 7https://polifonia-project.github.io/idea/category/competency-questions 8http://ontologydesignpatterns.org/wiki/Submissions:TimeIndexedSituation 82 i i “output” — 2025/10/28 — 13:54 — page 83 — #97 i i i i i i 3.4. The Polifonia Ontology Network (PON) Direct/indirect ontology reuse. Depending on the project’s requirements, reuse of ontologies and ODPs is direct and/or indirect [53]. The former approach directly includes/imports ontologies or part of them (e.g. individual entities, relations) thus introducing a dependency to any possible changes and availability. In indirect reuse, relevant entities and patterns from other ontologies are used as templates (replicated and extended) while being aligned to ensure interoperability. In Polifonia, we follow a hybrid approach: ArCo ontology [54] is directly reused since its development and maintenance involves one of the project’s partners (MiC), while others (such as DOREMUS) are indirectly reused and aligned. Validation and testing. Ontology modules have been developed in close collaboration with domain experts and pilot leaders throughout the whole development cycle. This has allowed the ontology design team to leverage the domain expertise in Polifonia to technically validate our modules at different stages: from the collection and analysis of requirements, to iterations of ontology designs. Validation was facilitated by IDEA (at the CQ-level), and, at the modelling level, by the Graphical Framework For OWL Ontologies (Graffoo) notation [132] – providing a powerful visual language for coproduction activities. This has also been achieved through data snippets provided by the pilots, which have been modelled by our ontologies and triggered further iterations of improvements. Overall, the involvement of domain experts from different institutions and background (complementary views and notions), the 10 pilots in the Polifonia project (reasonable diversity of application domains), and the use of collaborative workflows have also contributed to mitigate bias in the development of PON. 3.4 The Polifonia Ontology Network (PON) The Polifonia Ontology Network (PON) provides a modular backbone of music ontologies to address both cultural heritage and more general queries in the music domain. As illustrated in Figure 3.3, PON v1.0 comprises 15 ontology modules that are organised thematically (colours, horizontal view) and hierarchically, to highlight their dependencies (vertical view). At the bottom of the architecture lies our Core module (providing general-purpose elements of design, ODPs, and alignments) and the reused ontologies. Four foundational models provide interoperability across PON through their abstract design: Source,Instrument,Music Meta, and Music Representation. These are specialised and extended in the 83 i i “output” — 2025/10/28 — 13:54 — page 84 — #98 i i i i i i Chapter 3. Representing Musical Knowledge Figure 3.3: Overview of the main modules in the Polifonia Ontology Network, with Polifonia’s pilots as early adopters (grey circles). Foundational models (Source, Instrument,Music Meta,Music Representation) provide the backbone of PON, built on top of the Core module while leveraging the main ontologies reused directly or indirectly. upper levels to add functionalities and contextualise specific domains. A summary of PON modules is given in Table 3.2, with links to the repositories storing the modules with documentation, diagrams, and examples. Through our foundational models, PON ontologies can be applied to a wide set of music projects, and the modular design simplifies extensibility and maintenance. To facilitate this process, further documentation and tutorials are also being made available at https://polifonia-project.github.io/ontology-network/. An example of use involving 5 PON modules (besides Core) is shown in Figure 3.4. 3.4.1 Foundational models and their extensions and specialisations The Music Meta module provides a rich and flexible ontology to describe music metadata related to artists, compositions, performances, recordings, broadcasts, and links. Music Meta focuses on provenance and interoperability – essential requirements for the integration of music datasets, which is currently hampered by the specificity of existent ontologies. The model is based on the InformationRealisation ODP [150], allowing to reduce the complexity of FRBR-based models, whose application in the music domain has raised concerns [335]. Given the relevance of this module, we will explore it in greater detail in Section 3.5. 84 i i “output” — 2025/10/28 — 13:54 — page 85 — #99 i i i i i i 3.4. The Polifonia Ontology Network (PON) Module Prefix Outline Repository Core core: Elements of general reuse and ontology design patterns /core-ontology Music Meta mm: Achieving interoperability of music metadata /music-meta-ontology Music Representation mr: Foundational model to describe arbitrary musical content /music-representation-ontology Music Instrument mop: Instruments and their evolution through time and space /music-instrument-ontology Source src: Musical sources and their context of production /source-ontology Tunes tunes: A specialisation of Music Meta for folk music /tunes-ontology CoMeta com: An extension of Music Meta to represent music corpora /cometa-ontology Music Projection mp: Achieving interoperability of music notation systems /music-projection-ontology Organs organ: A rich descriptive model of organs and building methods /organs-ontology Bells bell: Describing bells, bell towers and bell ringers /bell-ontology Music Algorithm mx: Computational methods for music and their parametrisation /music-algorithm-ontology Music Analysis ma: Music analysis through reasoning using modal-tonal theories /music-analysis-ontology Music Annotation ann: A wrapper of ontologies for music annotations (audio, symbolic) /music-annotation-ontology PON (full) pon: The whole Polifonia Ontology Network (imports all modules). /ontology-network Table 3.2: Overview of the modules in the Polifonia Ontology Network. All URIs are also accessible from https: // github. com/ polifonia-project/ ontology-network . The Tunes module extends and specialises Music Meta for folk music. The main novelty consists in grouping and describing tunes into “tune families” depending on their melodic similarity (an association requiring rich provenance description of the musicological analysis on the source); which also extends to lyrics families. CoMeta reuses and extends Music Meta to describe arbitrary music collections, corpora, and datasets. Here, metadata is described at the collection-level (data curator, annotations provided, availability of audio music, etc.), and at the contentlevel, (e.g., the title, artist, release of each piece in a dataset). The design of CoMeta is informed by a survey of Music Information Retrieval datasets [271]. The Music Representation module provides a comprehensive schema to describe the analysis of musical objects (a score, an audio track, etc.) interpreted according to a theory. Fragments of a musical object (elements of a music object whose temporal location is uniquely identifiable) are described by annotations provided by an agent (e.g. expert annotator, algorithm). An annotation is either the subjective result of an analysis (e.g. a chord played in a specific section) or objective in nature (e.g. a note in a digital score). Each annotation describes some music content (e.g. notes, chords, etc.), which we refer to as a musical projection [256]. Annotations are formalised via our Music Annotation Pattern [89]. whereas the definition of music projections is delegated to the Music Projection module. The generality of the module and its abstraction over the represented content enables the interoperability of different music annotation schemas. The module is aligned to MusicOWL [213], Music Notation Ontology [62], Music Note 85 i i “output” — 2025/10/28 — 13:54 — page 86 — #100 i i i i i i Chapter 3. Representing Musical Knowledge Ontology [315], and our JAMS ontology (c.f. Section3.4.2). The Music Projection module formalises musical entities that can be subject of an annotation. This ranges from traditional musical notation (e.g. note, chords) to informal annotations (e.g. mood, danceability). The module is aligned with MusicOWL, Music Notation Ontology, Music Note Ontology, Music Theory Ontology [332], Chord Ontology [135], and Roman Chord Ontology 9. This allows to integrate existing domain ontologies. Notably, we also harmonise different chord representations (Chord Ontology, the Roman Chord Ontology and the Tonality Ontology) based on the Unified Model of Chords in Western Harmony [187]. The Instrument Module describes musical instruments as mediums of performance and their technical properties. Given that numerous taxonomies of instruments into groups and families exist (e.g. Hornbostel-Sachs, MIMO, MusicBrainz) and finding common categorisations is an open problem [224], our module provides an abstraction capable to express arbitrary classifications. This is achieved by leveraging the Information-Realisation and the Collection ODPs. Overall, the module allows to: (i) refer to instruments as entities (an instrumentation of a piece for “piano” and “viola”) as well as conceptually (e.g. a viola has 4 strings); (ii) support the integration with different taxonomies and vocabularies, such as [248]; (iii) describe the evolution of instruments in time and space (e.g. a viola as a cultural heritage object being relocated). This provides a foundational level where contributors can “plug” their instrument-specific ontologies [421]. The Bells module extends Instrument to describe bells by means of measurable, intrinsic aspects such as weight, materials, conservation status. The main entities contextualising bells are: (i) the author(s), such as the foundry who built the bell; (ii) the agencies that played some role e.g. the agency that took care of cataloguing the bell; (iii) the place(s) where it has been located; (iv) the tower(s) where the bell has been included; (v) the tools that the set of bells is played with; (vi) documents related to the bells, e.g. bibliographies, protective measures. The Organs module extends Instrument to describe organs as (i) a musical instrument consisting of parts; and (ii) as a focal point of projects detailing its changes throughout time. To address the former, we used the Parthood pattern from the DOLCE ontology 10. The entities of the ODP, Whole and Part make 9https://github.com/polifonia-project/roman-chord-ontology 10http://www.ontologydesignpatterns.org/ont/dul/DUL.owl 86 i i “output” — 2025/10/28 — 13:54 — page 87 — #101 i i i i i i 3.4. The Polifonia Ontology Network (PON) possible the specification of the whole instrument and its parts. In the ontology, the Whole entity refers to the organ instrument, and the Part entity refers to the parts of the organ that are Console,WindSystem,Case,Division, and Action. The Source module represents various sources of music-related information. These include manuscripts, textbooks, articles, interviews, reviews, comments, memoirs, etc. of different scope and format (physical, digital). The module aims to provide general support to describe information related to the creator and type of the source, the time and place when/where it was created, the context of production and usage, and the subject and goals. Although this conceptualisation leans towards bibliographical sources, the module provides expressivity to indicate multimedia documents (e.g. images of scores, audio recording, video). For example, a video recording of a performance can be considered as a musical source – providing documentary evidence of a composition e.g. during an event. The Meetups module describes encounters between people in the musical world in Europe from c. 1800 to c. 1945. Historical meetups, which are the main subject of this module, are described by means of four main components: the people involved in the meetup, for instance, the person that is the subject of interest and the people interacting in the event, the place where the encounter took place (e.g., city, country, venue), the type of event, the reason (e.g., music making, personal life, business, among others) and the date when it took place. The MusicBO module is developed by following a KG-to-ontology process [269]. Ontological axioms, grouped into patterns, are empirically generated from the MusicBO knowledge graph – which is built from a textual corpus on music performances and encounters between music-related agents in Bologna since the 17th century. Such patterns include information about the probability of axioms to happen (as they are derived from the data). For instance, the probability of instances of the pattern compose situation (the process of creating art) to have NaturalPerson as range of the artist property, is higher than the probability of having an Organisation as a composer. In sum, the content of the ontology module is highly dependent on the KG, and the most populated and described entities are: persons, places, organisations, works of art, theatres, and books. 87 i i “output” — 2025/10/28 — 13:54 — page 88 — #102 i i i i i i Legend rdf:type Prefixes core: mm: mr: mp: organs: ann: https://w3id.org/polifonia/ontology/core/ https://w3id.org/polifonia/ontology/music-meta/ https://w3id.org/polifonia/ontology/music-representation/ https://w3id.org/polifonia/ontology/music-projection/ https://w3id.org/polifonia/ontology/organs/ https://w3id.org/polifonia/ontology/music-annotation/ Highway Star live version rdf:type First Chord Highway Star record version rdf:type Second Chord mr:Fragment rdf:type A_1 A_2 Analysis A mr:isAnalysedIn mr:hasAnnotation ann:JAMSAnnotation ann:includesObervation rdf:type F5 ann:hasChordSubject mp:Chord rdf:type G5 ann:hasChordSubject mp:Chord ann:includesObservation rdf:type A_2_F2 mr:describesFragment rdf:type A_2_F1 mr:describesFragment mr:Fragment First Chord skos:exactMatch Second Chord skos:exactMatch mr:involvesAnalyst rdf:type expert_1 rdf:type Annotator mr:involvesRole core:AgentRole mr:involvesAgent core:Agent rdf:type B_1 B_2 mr:Annotation mr:hasObservation rdf:type Fsus2 mr:hasSubject mp:Chord rdf:type G5 mr:hasSubject mp:Chord mr:hasObservation rdf:type B_2_F2 mr:describesFragment rdf:type B_2_F1 mr:describesFragment mr:Fragment First Chord skos:exactMatch Second Chord skos:exactMatch rdf:type Annotatorexpert_2 mr:involvesAgent core:AgentRole Analysis B mr:involvesAnalyst mr:isAnalysedIn mr:hasAnnotation rdf:type rdf:type "0"^^xsd:integer rdf:type ann:hasStart MusicTimeIndex ann:hasMusic TimeDuration ann:hasValue "37"^^xsd:integer ann:hasValue ann:MusicTime IndexComponent ann:MusicTime Duration ann:MusicTime Interval seconds ann:hasValueType ann:hasMusic TimeInterval rdf:type "3"^^xsd:integer rdf:type core:hasStartMusic TimeIndex core:hasMusic TimeDuration core:hasValue "23"^^xsd:integer core:hasValue core:MusicTimeIndex Component core:MusicTime Duration characters core:hasValueType core:hasMusicTimeInterval mm:MusicEnsemble rdf:type Deep Purple rdf:type Ian Gillan Jon Lord Ian Paice Ritchie Blackmore rdf:type IanPaiceMembershipA mm:MusicEnsemble Membership rdf:type HighwayStar MusicEntity Roger Glover mm:MusicEntity HighwayStarLivePerf rdf:type HighwayStarStudioPerf mm:MusicPerformance rdf:type HighwayStar StudioPerf Recording Proc HighwayStarLivePerf JonLord HighwayStar LivePerf Recording Proc mm:RecordingProcess mm:Recording mm:hasMembership mm:isCreatorOf mm:hasMember mm:isRecordedBy Osaka, Japan mm:hasPlace core:involvesAgent core:involves IanPaiceMembershipA TimeInterval "1968"^^xsd:int "1976"^^xsd:int core:hasEndTime core:hasStartTime core:hasTimeInterval mm:producesRecording core:realises core:realises mm:producesRecording rdf:type rdf:type "61"^^xsd:integer Hammond_C3 organs:Organ core:includesPart core:includesWhole organs:Console rdf:type organs:Manual Keyboard organs:hasRange rdf:type core:includesPart organs:Pedal Keyboard "25"^^xsd:integer organs:hasRange core:hasTimeInterval core:startTime "1935"^^xsd:integer rdf:type Harmonic_percussion organs:PlayingAids mm:Musician core:hasTimeInterval "16-08-1972"^^xsd:int core:hasStartTime core:hasEndTime Hammond_C3 core:involvesInstrument Class Individual object property datatype property rdf:type Figure 3.4: Graffoo [132] example of “Highway Star” by Deep Purple using 5 PON modules to describe: metadata information (Music Meta, bottom), instrument (Organs, bottom-right) and annotation of musical content on two audio recordings via the Music Representation, Projection, and Annotation modules, either related to a studio (top-right) or a live (top-left) performance of the same piece. We remark how the two musical annotations are made interoperable via PON despite their profound differences (JAMS [203] and text, respectively) as they refer to the same fragment. i i “output” — 2025/10/28 — 13:54 — page 89 — #103 i i i i i i 3.4. The Polifonia Ontology Network (PON) 3.4.2 Modules for analysis and annotation of music The Music Algorithm module formalises algorithms that can operate on music metadata (using the Meta module), and musical content (via the representation module). The module commitments are similar to those defined by Diamantini et al. in [117]. Indeed, an algorithm is characterised by three main components: a formalisation, which can be theoretical (e.g. pseudocode) or executable (e.g. using a programming language); a parametrisation (e.g. input data); and the kind of task it solves. The latter defines a set of entities that are processed alongside the input and output data requirements and the final goal achieved. The module allows theoretical and quantitative performances to be represented in the context of the algorithm’s parametrisation. Through an abstract and general definition, the formalisation in Music Algorithm can be seen as a general pattern, capable of representing any algorithm regardless of the domain of application. In the context of music, the output of the algorithm is considered an analysis, which is then represented via the Representation module. The Music Analysis module allows for the analysis of musical pieces using historical and present-day established musical theories: the modal and tonal theories. Through the use of this framework, different subjective analyses can be unified – overcoming the limitations imposed by a “global” theoretical perspective. Different theoretical viewpoints can be used for the interpretation of the same piece. Currently, two historical theories are implemented: Zarlino (1558) and Praetorius (1619) [168, 169]. Through the use of formal reasoning and a comprehensive axiomatisation, the ontology is able to automatically infer the theoretical interpretations of a musical piece and its evolution in time and space. The Music Annotation module provides different music annotation models to accommodate musicological and information retrieval use cases. The primary objective of this module is to enhance support for other descriptional systems, thus increasing interoperability and conversion possibilities from various music annotation formats. Indeed, all our models are logically interconnected through Music Representation. A fully fledged annotation model here is the JAMS Ontology [90] 11, which is detailed in Dataset 4 (c.f. Section 4.3.1). This ontology mimics the structure of a JAMS (JSON Annotated Music Specification for Reproducible MIR Research) document [203]. It semantically describes and connects 11https://github.com/polifonia-project/jams-ontology 89 i i “output” — 2025/10/28 — 13:54 — page 90 — #104 i i i i i i Chapter 3. Representing Musical Knowledge all the elements of the JAMS specification (Annotation,Observation, etc.), including the music metadata and the annotation contents using the Music Meta and Representation modules, respectively. 3.5 The Music Meta Ontology Music Meta is part of Polifonia Ontology Network (PON), from which it imports the CORE module (c.f. Section 3.4). The ontology (prefixed as mm) is available at the following URI: https://w3id.org/polifonia/ontology/music-meta/, and is released as open source project under the CC-BY 4.0 on GitHub12. From FRBR to Information Objects/Realisations At the core of Music Meta lies the use of the Information-Realisation (IR) ODP [150]. An information object is a non-physical social object carrying information that can have one or multiple materialisations (information realisations). Each realisation is a particular physical object, or event, realising the information object, or involving the latter as a participant. Both information object and realisation are intended as Information Entities (IEs), i.e. (social) objects created and/or used to communicate, reason, and specify new entities. This allows to distinguish between a piece of information (e.g. the content of a composition) from how it is materialised (e.g. as a performance). On the other hand, both the Music Ontology [324] and DOREMUS [64] are built on top of different flavours of FRBR13 (FRBRer and FRBRoo, respectively). FRBR is a conceptual model describing bibliographic resources at four levels: Work,Expression,Manifestation, and Item. In contrast, the two levels of the IR pattern map to Expression and Item, since Work and Manifestation are said to provide non-informative conceptualisations [150]. Moreover, [335] argues that FRBR’s Works – intended as “entities that pre-exist expressions”, cannot represent improvisations or traditional music, as they do not derive from a formal composition process leading to a realisation. FRBR’s Work is often ambiguously intended as an entity retrospectively created for grouping multiple expressions for cataloguing needs. As for the Manifestation level, while its representation is straightforward in the bibliographic domain (e.g. the printed version of a book), its correspondence in the music domain is not fully intuitive, as it may relate to 12https://github.com/polifonia-project/music-meta-ontology 13https://www.ifla.org 90 i i “output” — 2025/10/28 — 13:54 — page 91 — #105 i i i i i i 3.5. The Music Meta Ontology mm:MusicArtist core:AgentRole core:Role core:involvesAgent core:involvesRole rdfs:subClassOf mm:Musicianmm:MusicEnsemble rdfs:subClassOf mm:MusicAlgorithm core:isMemberOf core:hasMember mm:MusicGenre mm:hasGenre mm:Award mm:nominatedForAward mm:receivedAward core:Personcore:Place mm:wasFormedIn xsd:dateTime core:activityStartDate core:activityEndDate core:MusicDataset mm:isTrainedOn core:Alias core:hasAlias core:hasLanguage xsd:string core:name core:Language core:name mm:isInfluencedBy mm:hasCollaboratedWith mm:MusicEnsembleMembership mm:MusicEnsemble mm:involvesMusic Ensemble mm:MusicArtist mm:involvesMemberOf MusicEnsemble core:TimeInterval core:hasTimeInterval core:Role core:involvesRole * * core:isMemberOf core:hasMember * Prefixes rdf: rdfs: owl: core: mm: http://www.w3.org/1999/02/22-rdf-syntax-ns# http://www.w3.org/2000/01/rdf-schema# http://www.w3.org/2002/07/owl# https://w3id.org/polifonia/ontology/core/ https://w3id.org/polifonia/ontology/music-meta/ Figure 3.5: Describing music artists as musicians, music ensembles, and algorithms using the Graffoo notation (yellow boxes are classes, blue/green arrows are object/- datatype properties, purple circles are individuals, green polygons are datatypes). either a recording, a score, a compact disc, or all the above – thereby introducing complexity and ambiguity. Nevertheless, being aligned to two levels of FRBR, the IR ODP makes our model leaner and flexible, while still achieving interoperability with FRBR-based (music) ontologies. In fact, IE patterns are meant to boost the semantic integration of contents, tools, platforms, resources that are silo-ed or non-interoperable [150]. 3.5.1 Main elements of design From Polifonia’s CQs14, we identified those related to metadata, and aimed for a model capable of addressing the requirements in Section 3.1.1. Music Meta follows a hierarchical design (where each level extends the former to add expressiveness) and is complemented by data transformation rules to conveniently translate one level into another. To enable data integration from existing knowledge bases and datasets, we align Music Meta to other ontologies: the Music Ontology, DOREMUS, and Wikidata, after having identified common/similar classes and properties. Music artists To represent music creatives the class mm:MusicArtist generalises over musicians (mm:Musician), ensembles (mm:MusicEnsemble), and computational methods (mm:MusicAlgorithm), as illustrated in Figure 3.5. Musicians are seen as a 14https://github.com/polifonia-project/stories 91 i i “output” — 2025/10/28 — 13:54 — page 98 — #112 i i i i i i Chapter 3. Representing Musical Knowledge 12345 0 5 10 15 20 25 30 count 1.6% 6.6% 16.4% 27.9% 47.5% My research or project work related to music data comprises multiple stakeholders with varied requirements. 12345 0 5 10 15 20 25 count 1.6% 6.6% 42.6% 24.6% 24.6% I have found the reuse or extension of the existing ontologies to be a challenging task (choose 3 if not applicable). 12345 0 5 10 15 20 count 3.3% 3.3% 36.1% 29.5% 27.9% I have found the interoperability of two or more existing ontologies (e.g. combining them) to be a challenging task (choose 3 if not applicable). 12345 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0 count 3.3% 6.6% 32.8% 26.2% 31.1% I think that the existing ontologies are lacking in their coverage of concepts related to music history and musical cultural heritage. 12345 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0 count 1.7% 1.7% 33.9% 32.2% 30.5% I think that there is a lack of high-quality, varied, and large datasets containing competency questions about music history and cultural heritage. 12345 0 5 10 15 20 25 30 count 3.3% 0.0% 18.0% 24.6% 54.1% I am interested in using an ontology that describes the metadata related to artists, compositions, performances, recordings, and links pertaining to different musical genres, time periods and datasets. 12345 0 5 10 15 20 25 30 count 1.6% 1.6% 16.4% 29.5% 50.8% I am interested in using an ontology that represents the sources of music such as manuscripts, collections and related information such as creator, type and context of the source. 12345 0 5 10 15 20 25 count 6.6% 9.8% 13.1% 27.9% 42.6% I am interested in using an ontology dedicated to the description and categorization of musical instruments as well as their evolution over time and place. 12345 0 5 10 15 20 25 count 6.6% 14.8% 21.3% 16.4% 41.0% I am interested in using an ontology that enables interoperability between different music notations while reusing existing ontologies. 12345 0 5 10 15 20 25 count 5.0% 3.3% 26.7% 25.0% 40.0% I am interested in using a high-quality, varied, and large dataset of competency questions about music history and cultural heritage. Figure 3.10: Selection of questions 4, 6, 7-14 from the online survey, where responses are expressed on a Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). 98 i i “output” — 2025/10/28 — 13:54 — page 99 — #113 i i i i i i 3.7. Conclusions 3.6.3 Adoption by Polifonia Stakeholders In addition to internal and potential adopters, industrial and institutional stakeholders in the Polifonia Stakeholder Network have also expressed interest to use PON resources. These include the Digital Music Observatory, concerning the use of Music Meta and Source to annotate the numerous music resources of the consortium; and the Université Catholique de Louvain where AnneEmmanuelle Ceulemans uses the Music Anlysis module for studying the annotation of cadences in Josquin des Prez (composer of High Renaissance music). We have also planned work with Deezer,Songfacts, and MusicID for the evaluation, extension, and reuse of Music Meta driven by their resources; and collaborations with the EU H2020 MuseIT21 project to extend the ChoCo KG. 3.6.4 Availability, sustainability, and FAIRness PON namespaces are introduced in Section 3.4, and permanent Uniform Resource Identifiers (URIs) were created with the W3C Permanent Identifier Community Group. PON is under version control on public GitHub repositories (c.f. Table 3.2), and all repositories are also published on Zenodo (with associated DOIs) under the CC-BY 4.0 licence. The storage of all resources on GitHub guarantees their persistence beyond the project, with the University of Bologna and the Italian Ministry of Culture (MiC) committed to host and maintain PON on the long term. We also remark that PON is reused as a sibling ontology project of ArCo by MiC [54]. 3.7 Conclusions This chapter presented the creation and development of the Polifonia Ontology Network (PON), a collection of expressive ontologies aimed at addressing the challenges of interoperability in musical cultural heritage. Through the Polifonia project, we applied and extended the XD methodology for ontology engineering, incorporating multidisciplinary and domain-specific requirements. Our methodological innovations, such as the IDEA framework for NLP-assisted ontology codesign, helped ensure that the design of PON was grounded in real-world needs and technological advancements. 21https://www.muse-it.eu/ 99 i i “output” — 2025/10/28 — 13:54 — page 100 — #114 i i i i i i Chapter 3. Representing Musical Knowledge PON (v1.0) comprises 15 new ontologies, along with the release of the PolifoniaCQ dataset containing 361 competency questions, all made available under an open license (CC-BY 4.0). Furthermore, we provide evidence of current and potential reuse by institutional and industrial stakeholders, demonstrating the practical relevance and impact of PON. As a next step, we plan to perform an extensive competency question-driven evaluation of PON’s modules to further refine the ontologies. Additionally, we will continue to support stakeholders and early adopters in reusing, extending, and maintaining the ontologies and knowledge graphs over the long term. This includes ongoing work to specialise the Music Meta model for the integration and release of new music knowledge graphs, both in the cultural heritage and music industry domains. Furthermore, we plan to extend PON modules by incorporating novel music theories, for instance through the Music Analysis Module, to broaden the scope and applicability of the ontology. 100 i i “output” — 2025/10/28 — 13:54 — page 101 — #115 i i i i i i i i “output” — 2025/10/28 — 13:54 — page 102 — #116 i i i i i i i i “output” — 2025/10/28 — 13:54 — page 103 — #117 i i i i i i CHAPTER 4 Harmonising Fragmented Data: A Comprehensive Workflow for Symbolic Data Integration 4.1 Introduction As discussed in Chapter 3, music data is fragmented across numerous datasets, each using its own conventions, tailored to address specific tasks or applications. This fragmentation arises from representational issues in both metadata and music content, addressed in RQ1 (Section 1.1.1). While the metadata representation often lacks standardization and interoperability, making it difficult to integrate different datasets, the content representation is scattered across a multitude of formats and notational systems (c.f. Section 2.3). This fragmentation is well exemplified by datasets containing harmonic data. As discussed in Chapter 2.1.3, harmony is a widely studied dimension in music theory [312, 351] and music analysis [188]; where functional harmony provides a set of rules for moving to and from the tonic – the most stable note in a piece, allowing to relate chords to each other, and to the main harmony. Chords are the basic constituents of harmony, which jointly define the harmonic 103 i i “output” — 2025/10/28 — 13:54 — page 104 — #118 i i i i i i Chapter 4. Harmonising Fragmented Data: A Comprehensive Workflow for Symbolic Data Integration structure of a piece. Individually, a chord is defined as a simultaneous occurrence of several music sounds, producing harmony [165]. Depending on the notational system and the annotation conventions, a chord can be associated with a name, or label. For example, the chord G7 (typically read as “G dominant seventh”) in the key of C major, contains the notes G−B−D−Fand may create tension partly due to the tritone relation between B(leading tone) and F(the seventh of the chord). These intervals to the root characterise the intrinsic harmonic properties of chords, as well as the relationships with other chords in the same harmonic progression [32]. Perceptually, some chords sound more stable, final and resolved, while others sound unstable and tense – a phenomenon that is salient both to young children and to adults, even from diverse cultures. However, the definition of harmony differs vastly across time, genre, and individuals [225], reflecting a great heterogeneity in terms of harmony perception [189, 265]; and in this work, we focus on Western tonal music tradition. In this regard, harmony exerts an affective role: major harmonies tend to represent positive emotions (happiness, joy, triumph, etc.); minor triads express “negative” emotions (sadness, anger, etc.); diminished triads (chains of minor thirds) indicate suspense and other disorienting sentiments, while augmented triads (all major third intervals) tend to create senses of spookiness, extreme dark emotions, and mystery [74]. Computationally, the automatic analysis of chord progressions has addressed several tasks in information retrieval – from the detection of cadences, patterns, structures in music, to the introduction of harmonic similarity measures for cover song detection, symbolic search, and content-based music linking. Progress in machine learning research has also sparked interest in computational creativity applications, such as arrangement generation, continuation, infilling, and automatic music composition with harmonic conditioning [128] (e.g. generating melodies from a given harmonic template) to name a few. To account for the evolution of harmony and explain its subjective and genrespecific differences, while enabling the aforementioned applications, the availability of large, diverse, and reliable chord data is fundamental. However, several different chord notations exist (Harte, Roman, ABC, Leadsheet, etc.), each with different levels of expressiveness, in a large number of disconnected chord datasets that are hard to combine [52]. This poses a challenge for combining existing chord datasets into larger ones. Existing approaches address this issue by focusing on 104 i i “output” — 2025/10/28 — 13:54 — page 105 — #119 i i i i i i 4.1. Introduction scale, and publishing large numbers of chord annotations. For example, UltimateGuitar1offers a collection of 1.1M+ songs annotated by a community of 12M+ musicians. Chordify2addresses the challenge of scalable chord annotation by applying methods for automated chord estimation. However, none of these approaches solves the problem of integrating chord datasets complying with the following desiderata: (a) high quality of the data; (b) precise timing information; (c) release through open licences; (d) use of different chord notations; (e) diversity of music genres; and (f) large scale. The problem is exacerbated by the little reuse of standard formats for music annotation. In the context of this thesis, music annotation is defined, in a broad sense, as the outcome of a music analysis carried out by a domain expert on the musical surface (a score, a recording) to identify and locate elements of interest (e.g. chords, segments, patterns, etc.), following an established methodology. For example, if the goal of a harmonic analysis is to identify chords from a composition, a music annotation may correspond to a list of chords together with a reference to their onset and offset (i.e. when they occur in the piece). 4.1.1 Our contribution In this chapter, we introduce the Chord Corpus (ChoCo), a comprehensive KG of harmonic annotations and a workflow designed to facilitate the development of musical harmony Knowledge Graphs leveraging PON (c.f. Section 3). These contributions, published in [90], directly address RQ2 (Section 1.1.2) by focusing on strategies for unifying symbolic music datasets to standardize diverse digital formats and annotation practices, thereby tackling the fragmentation challenges within existing chord datasets. The workflow we present encompasses the curation, transformation, and integration of over 20,000 human-made, high-quality harmonic annotations from 18 highly heterogeneous chord datasets (desiderata a,b,f), following the JAMS data structure as annotation model. The resulting annotations are rich in provenance data (e.g. metadata of the annotated work, authors of annotations, identifiers, etc.) and refer to both symbolic music notation and audio recordings, while encompassing different notation systems (desideratum d). After semantically enriching, extending, and standardising these annotations under the JAMS definition, 1https://www.ultimate-guitar.com/ 2https://chordify.net 105 i i “output” — 2025/10/28 — 13:54 — page 106 — #120 i i i i i i Chapter 4. Harmonising Fragmented Data: A Comprehensive Workflow for Symbolic Data Integration we use the PON ontologies described in Chapter 3 to release the ChoCo Knowledge Graph – providing fine-grained semantic descriptions of chords, opportunities for chord interoperability, and 4K+ links to external datasets. All data and code are released using open data licences (desideratum c). We also show evidence of interest and use of ChoCo, and postulate its value for the SW and MIR communities at enabling the study of harmony through large scale data. Specifically, the main contributions are summarised as follows: •A generalised data curation framework to semantically integrate MIR harmonic datasets and represent chords from a large variety of formats (JSON, CSV, LAB, TXT, SQL, MusicXML, iReal, mgu, sku, ABC, etc.) as JAMS annotations. •A large dataset and KG standardising, enriching, and integrating 18 existing chord collections in the literature. ChoCo is released both as a JAMS dataset and an RDF Knowledge Graph, to accommodate the requirements and needs of different communities (MIR, Musicology, SW, etc.). •Demonstrations of the utility of both the workflow and the resulting chord corpus. These examples illustrate its relevance to both MIR and SW fields, including the use of the workflow to describe other musical elements, such as melodic patterns [353]. •Evidence of potential adoption and community interest, by conducting a survey targeting potential users, asking ten questions related to the relevance of chord data in their work, and their interest in adopting the dataset and workflow. ChoCo achieves interoperability of harmonic datasets at three levels: metadata, annotation format, and chord notation. The interoperability at metadata and annotation format levels is implemented by integrating metadata from different sources, at the parsing level, and by leveraging the JAMS annotation standard to store harmonic annotations, consistently. Chord notation interoperability is achieved by converting chords to three reference notational systems (desideratum d) – bridging them via the Harte notation [181]. The outcome of this approach enables the use of these integrated collections as if they belonged to the same dataset and underpins the automatic generation of Music Knowledge Graphs. In addition to the conversions, ChoCo provides the original annotations in each JAMS file, along with rich provenance descriptions that keep track of the original sources. 106 i i “output” — 2025/10/28 — 13:54 — page 107 — #121 i i i i i i 4.2. Related Work 4.1.2 Chapter Structure This chapter is organized as follows: we begin with a review of the related work in Section 4.2, providing context for the challenges and existing approaches to harmonizing and integrating chord datasets in MIR. The development of ChoCo is detailed in Section 4.3, where we present the methodology used for creating the dataset. This includes an overview of the data incorporated into ChoCo, the process of converting the data into a unified format, the conversion of chord annotations, and the creation of the resulting KG. We then provide descriptive statistics in Section 4.3.2, offering an overview of ChoCo data at two different levels: the metadata associated with the music tracks, including their identifiers and links, and the content of the music annotations. In Section 4.4, we present the technical validation of the dataset, demonstrating the reliability and completeness of ChoCo. The chapter continues with usage notes in Section 4.5, where we discuss how both the dataset and the workflow have been employed, along with potential future applications. Data availability and licensing information are outlined in Section 4.6, providing details on how ChoCo can be accessed and used under open licenses. Finally, we present the conclusions in Section 4.7, summarizing the main contributions and future directions for the workflow and its applications. 4.2 Related Work In the last decade, numerous systems and formats have been proposed for representing and storing musical annotations [187]. Some have been more successful than others, but no system has prevailed as a reference standard. Some systems are focused on symbolic music and are domain-specific (e.g. DCMLab, RomanText for harmonic analyses), embed annotations in the score (MusicXML, ABC, etc.), or propose variations of tabular formats to account for audio and symbolic music (LAB and xLAB). In the audio domain, JAMS (JSON Annotated Music Specification) [203] has emerged as a system to uniformly represent music annotations of different types and granularity, that is efficiently built on top of the JSON serialisation standard. JAMS is also supported by software libraries for dataset manipulation [34] and for the evaluation of MIR methods [323]. However, combined efforts of MIR and SW researchers to address (chord) anno107 i i “output” — 2025/10/28 — 13:54 — page 114 — #128 i i i i i i Chapter 4. Harmonising Fragmented Data: A Comprehensive Workflow for Symbolic Data Integration namespace (the term namespace in JAMS has a different sense than a Web namespace) that identifies the type of the annotation’s subject (e.g., chords, structural segments, emotions, patterns, keys, etc.). These annotations also include metadata to document the annotation process (e.g. whether the annotation is manually produced or inferred by an algorithmic method, the name of the annotator or software, information about the annotation tools, rules and validation). •sandbox, described as an unrestricted place to store any additional data. Listings 4.1 and 4.2 show excerpts of an example JAMS file from the Isophonics collection [258] annotating chords for Queen’s Bohemian Rhapsody, taken from the Isophonics collection. Although JAMS has an implicit focus for audio-based annotations, its definition and structure are flexible enough to be easily extendable to the symbolic domain. This is also confirmed by the modular design of the codebase, where additional namespaces can be registered by a user, by simply providing regular expressions to validate the annotation content (e.g. a new chord notation). In other words, any arbitrary music annotation can be described within JAMS as long as the atomic observations (e.g. the individual occurrences of chords making up the progression) are described in terms of: time, a temporal anchor specifying the onset of the observation; duration,value (e.g. Bb:maj7), and confidence, a scalar in [0,1] expressing a level of certainty by the annotator (or algorithm). Therefore, the only elements distinguishing audio from symbolic annotations, are the temporal specifications (time and duration), which are described in absolute (seconds) or metrical (measure and beat/offset) terms, respectively. For symbolic annotations, we number measures and beats from 1 for convenience, without attempting to emulate exact musical (editorial) practice for cases like anacrustic openings. JAMification of datasets Considering the diversity of annotation formats and conventions for data organisation (the way content is scattered across folders, files, database tables, etc.), each chord dataset in ChoCo (c.f. Table 4.1) undergoes a standardisation process lending to the creation of a JAMS dataset. This is needed to aggregate all relevant annotations of a piece (chord, key, etc.) in a single JAMS file, and to extract 114 i i “output” — 2025/10/28 — 13:54 — page 115 — #129 i i i i i i 4.3. ChoCo: a Chord Corpus and a Data Transformation Workflow for Musical Harmony Knowledge Graphs content metadata from the relevant sources. The content metadata of a (music) dataset is indeed crucial to identify, describe and retrieve the actual musical content being annotated. This typically includes the title of each piece, artists (composers and/or performers), and cataloguing information (album/release or collected work), ideally with the provision of identifiers (e.g. MusicBrainz IDs). Nevertheless, only the Mozart Piano Sonata collection [188] provides complete content metadata in a csv file, as usually expected from a music dataset. When content metadata is missing, this may be found online (HTML pages, supplementary material), from articles/reports documenting the collection, by resolving any cross-reference among files and dataset-specific identifiers, extracted from the actual score (or better, the dataset-specific representation of the score). Alternatively metadata can be derived from the organisation of files in folders. For example, Michael Jackson/Essential Michael Jackson [Disc 01]/1-16_Beat_it.lab indicates author, album, disc, track number and title, respectively. This organisation varies as the datasets vary – a consequence of the lack of a standard “datasheet for datasets” in the music domain [152]. 1{ 2" sandbox ": {} , 3" annotations ": [ 4{ 5" data ": [ 6{ 7" duration ": 0.459 , 8" confidence ": 1.0 , 9" value ": "N", 10 " time ": 0.0 11 }, 12 { 13 " duration ": 3.663 , 14 " confidence ": 1.0 , 15 " value ": " Bb :maj6 ", 16 " time ": 0.459 17 }, 18 { 19 " duration ": 0.789 , 20 " confidence ": 1.0 , 21 " value ": "C :7" , 22 " time ": 4.122 23 }, 24 ... 25 ], Listing 4.1: Excerpt of the three first chords in a JAMS file annotating Queen’s Bohemian Rhapsody. 1"annotation_metadata": { 2"annotation_tools": "", 3"curator": { 4" name ": " Matthias Mauch ", 5" email ": "m. mauch@qmul . uk " 6}, 7" annotator ": {} , 8" version ": 1.0 , 9" corpus ": " Isophonics ", 10 "annotation_rules": "", 11 " validation ": "" , 12 " data_source ": "" 13 }, 14 " namespace ": " chord " , 15 " sandbox ": {} 16 }, ... ], 17 " file_metadata ": { 18 " jams_version ": "0.2.0" , 19 " title ": " Bohemian Rhapsody ", 20 " identifiers ": {} , 21 " release ": "" , 22 " duration ": 358.293 , 23 " artist ": " Queen " 24 } 25 } Listing 4.2: Annotation and file metadata in a JAMS file annotating Queen’s Bohemian Rhapsody. 115 i i “output” — 2025/10/28 — 13:54 — page 116 — #130 i i i i i i Chapter 4. Harmonising Fragmented Data: A Comprehensive Workflow for Symbolic Data Integration The same issue applies to the extraction, pre-processing, and standardisation of harmonic annotations from these collections, some of which were never released as chord datasets (Weimar Jazz Database,Wikifonia,iReal Pro,Nottingham). Harmonic annotations can be encoded in different formats (LAB, XLAB, RomanText, CSV, DCMLab, JSON, SQL, TXT), or extracted from symbolic music (MusicXML, ABC) and backing tracks in proprietary encodings (iReal, MGU). As each collection shows a specific combination of the mentioned issues (different organisation of content and metadata, different annotation formats and conventions), this step required considerable effort. The result of this standardisation process may improve the usability of these resources for music researchers, and simplifies the KG construction process. In addition, for the symbolic subsets, we also include time signatures (initial time signature and subsequent metrical changes) as annotations in each JAMS file (using a dedicated timesig namespace); which makes it easier to interpret the temporality of each chord (onset and duration) at hand. Following the standardisation process, each of these 18 JAMS datasets represents a novel contribution per se, due to the heterogeneity of annotation formats and practices, and the limited availability of content metadata in their original version. This also includes CASD, a collection that provides chords in JAMS format, but lacks local key annotations, which were retrieved from Billboard (we remind that CASD is already a subset of Billboard). Conversion of chord notations As shown in Table 4.1, the third element of divergence besides annotation formats and provision of content metadata, is the notation system used to represent chords. To address this issue we perform the following actions: (i) decomposition of domain-specific notations to chord constituting elements; (ii) conversion of the decomposed chord to the Harte framework; (iii) conversion of chord progressions by iteratively applying steps (i) and (ii) to all the chords in a sequence/progression. This yields a new JAMS file with the converted chord annotations. For all the above steps, specific software was developed for processing the different annotation types contained in the original datasets. There are three main types of chords that are processed: Roman Numerals chords (e.g. C min:viio7/V), Polychords (e.g. E4,G#4,B4), Leadsheet chords (e.g. Gm7/F). With Leadsheet chords we refer to a broader category, although each dataset using this format 116 i i “output” — 2025/10/28 — 13:54 — page 117 — #131 i i i i i i 4.3. ChoCo: a Chord Corpus and a Data Transformation Workflow for Musical Harmony Knowledge Graphs encoding output ABC Notation Nottingham decomposition Leadsheet Notation Wikifonia Jazz-corpus Ireal-Pro Weimar Polychords Wikifonia validation decomposition Roman Numerals Notation Mozart-pianosonatas When-inRome Rock-corpus Lark Grammar Parsing music21 Parsing Harte Encoder INPUT: JAMS File OUTPUT: JAMS FileChonverter Harte Notation Figure 4.3: Overview of the Chonverter workflow, describing how different chord notations are converted to the Harte notation. proposes a different flavour of this notation. For example, a G minor chord in Wikifonia is annotated as G min, whereas the same chord is annotated as Gin the Jazz-corpus. As outlined in Table 4.1, each dataset uses a flavour of the same notation to represent chords, with the exception of Wikifonia, where some annotations use both Leadsheet and Polychords even for the same progression; and the Jazz Corpus, providing chords encoded in both Roman Numerals and Leadsheet. Figure 4.3 provides a taxonomy of the different notational flavours, together with a schematic overview of the conversion workflow. In step (i), a chord is first decomposed into its components (e.g. C major → C, E, G). For this purpose, the Chonverter uses a family of tools depending on the source notation. Roman numerals are decomposed using the roman module of music21 [79], a Python library for computational musicology. As Polychords already provide note constituents by definition, this step is limited to preprocessing the symbols associated to the different pitches in a chord. Polychords are usually mixed with chords annotated in other notations (e.g. Leadsheet), so it is necessary to differentiate the type of chords when parsing. Finally, for each Leadsheet flavour, a context free grammar was created to parse the original annotation of the chord. A different grammar was created for each dataset containing annota117 i i “output” — 2025/10/28 — 13:54 — page 118 — #132 i i i i i i Chapter 4. Harmonising Fragmented Data: A Comprehensive Workflow for Symbolic Data Integration tions in leadsheet format, namely Weimar Jazz Database,Wikifonia, and iReal Pro, using the Lark library9. Notably, the ABC notation used in Nottingham is similar to the Leadsheet notation and was therefore processed in the same way. This process is more intuitive for all collections natively using the Harte notation, as the latter already accounts for the description of chord pitches [181]. After all chords are decomposed as lists of pitches, it is then possible to associate a shorthand (a string) to each list according to the Harte notation (Step (ii)). The Chonverter achieves this via music21 and defines rules for composing Harte chords. New JAMS files are produced after the last step, each providing a new annotation (with chord_harte as namespace). Whenever an original annotation uses Leadsheet or Polychord notations, the new annotation replaces the original, since the conversion provides a generalisation of the different flavours via a syntactic transformation. Instead, if the original annotation contains Roman Numerals chords, the new (converted) annotation is added to the existing one, since the Roman Numerals contain information that would otherwise be lost, i.e. the harmonic functions that the chords hold within the piece. The Chonverter module performs a syntactic conversion of chord labels. However, converting Roman Numeral also requires taking into account the key of the song. Moreover, a distinction has to be made between key-relative and absolute chords. Some music is always played in the same key, while other pieces are frequently transposed. For example, symphonies are often performed in a fixed key, while lieder are typically performed in multiple keys depending on the singer’s vocal range. Datasets like When in Rome contain transcriptions of these key-flexible works. Even in these cases, chords in ChoCo are always converted by taking into account the tonality provided by the original dataset for that piece. However, whenever this happens, the generated conversion, although correct, may only be one of several possible conversions. The JAMS Ontology and the ChoCo Knowledge Graph To represent JAMS annotations as Linked Data (LD) we designed an ontology that formally represents the JAMS data model. The JAMS Ontology is part of the Polifonia Ontology Network (c.f. Chapter 3), from which we reused 4ontology modules (Core,Music Meta,Music Representation and Music Projection). More 9https://github.com/lark-parser/lark 118 i i “output” — 2025/10/28 — 13:54 — page 119 — #133 i i i i i i 4.3. ChoCo: a Chord Corpus and a Data Transformation Workflow for Musical Harmony Knowledge Graphs specifically, the JAMS Ontology is part of the Music Annotation module, which directly imports it. Table 4.2 provides links to ChoCo’s resources, including the JAMS Ontology and KG. The JAMS Ontology formally defines the semantics of music annotations that are encoded using JAMS. To improve compliance with the ontology and facilitate the generation of LD, we have established conventions for including relevant information in the creation phase of the JAMS files. In essence, the JAMS Ontology tackles the limitations of the current JAMS model, mainly on two fronts: (i) at the level of metadata, enabling the alignment and linking of tracks belonging to different datasets, and also, with external resources available on the Web; (ii) at the annotation level, allowing to describe data (e.g. a chord) by semantically annotating its components (e.g. root, quality, inversions, etc.) rather than using a label. Concerning the first level, the JAMS Ontology inherits all the benefit of the proposed PON and Music Meta module, as docuented in Section 3.5. Moreover, the proposed model also allows to correctly interpret the content of the annotation with great level of detail, for example, modelling temporal information both in real time (seconds) and in beats, depending on the type of annotation at hand. To achieve this, additional data is dumped by the JAMifier in the Sandbox of each JAMS file, and new annotation types were created by contributing new namespaces. The JAMS Ontology provides a common conceptual, formal model to interpret JAMS annotations and is available online at the following URI: https://w3id.org/polifonia/ontology/jams/ Our ontological requirements can be summarised as follows: •the resulting KG must represent JAMS files and JAMS annotations as such, including their provenance and process-related information: e.g. source dataset, annotator, confidence of each observation, etc; •temporal information must be expressed according to the type of the annotation’s subject, i.e. audio or score; •chords must be represented according to the data model of these notation families: Harte and Roman Numerals. To model this ontology, we reused the Music Annotation Pattern [94], an ODP [147] for modelling different types of music annotations and their related time 119 i i “output” — 2025/10/28 — 13:54 — page 120 — #134 i i i i i i Chapter 4. Harmonising Fragmented Data: A Comprehensive Workflow for Symbolic Data Integration ID Competency question CQ1 What is the content of the observations contained in a JAMS Annotation? CQ2 Who is the composer of a musical object? CQ3 Who is the performer of a musical object? CQ4 Who/what is the annotator of an annotation/observation, and what is its type? CQ5 What is the time frame addressed by an annotation, within a musical object? CQ6 What is its start time (i.e. the starting time of the time frame)? CQ7 Which are the observations included in an annotation? CQ8 Given an observation, what is the starting point of the time frame it addresses, within its target musical object? CQ9 Given an observation, what is its addressed time frame, within its target musical object? CQ10 What is the key of a composition/performance? CQ11 What is the value of an observation? CQ12 What is the confidence of an observation? CQ13 What are the chords of a composition/performance? Table 4.3: Competency questions (CQs) addressed by the JAMS Ontology. references. We remark that the terminology used in the JAMS documentation10 is adopted to define the JAMS Ontology vocabulary. In particular, the following terms are (re-)used: •Annotation: an annotation is defined as a group of Observations (see below) that share certain elements, such as the method used and the type of annotation’s subject (e.g. chords, notes, patterns); •Observation: an observation is defined as the content of an annotation, and includes all the elements that characterise the observation. For example, in the case of an annotation containing chords, each observation corresponds to a chord, and specifies, in addition to the chord value, the temporal information and its confidence. We also apply the same methodology described in Section 3.3: the CQs defined for the ontology are listed in Table 4.3, while the corresponding SPARQL queries are available in the JAMS Ontology repository11. Figure 4.4 shows a fragment of the JAMS Ontology modelling a JAMS Annotation. On the left (box A), we define the classes and properties for representing the song’s metadata, by reusing the Music Meta module from PON. Main classes and properties of this ontology are detailed in Section 3.5. The connection between the Music Meta ontology and the JAMS Ontology happens at the level of mm:Recording,mm:Score, and mm:AbstractScore, where the union of the three form a mr:MusicContent, which can be annotated by a jams:JAMSAnnotation. A core class of the JAMS Ontology is jams:JAMSAnnotation. It captures the 10https://jams.readthedocs.io/en/stable/ 11https://github.com/polifonia-project/jams-ontology 120 i i “output” — 2025/10/28 — 13:54 — page 121 — #135 i i i i i i 4.3. ChoCo: a Chord Corpus and a Data Transformation Workflow for Musical Harmony Knowledge Graphs Prefixes core: jams: mm: mr: prov: https://w3id.org/polifonia/ontology/core/ https://w3id.org/polifonia/ontology/jams/ https://w3id.org/polifonia/ontology/music-meta/ https://w3id.org/polifonia/ontology/music-representation/ http://www.w3.org/ns/prov# jams:JAMSAnnotation jams:JAMSFile prov:wasDerivedFrom jams:Annotator jams:AnnotatorType jams-res:program jams:hasAnnotator jams:hasAnnotatorType jams-res:expert_human jams-res:crowd prov:wasDerivedFrom prov:wasMemberOf jams:Dataset rdf:type jams:MusicTimeInterval jams:hasMusic TimeInterval B jams:JAMSObservation jams:includes Observation jams:AnnotationType jams:hasAnnotation Type mm:CreativeProcess mm:MusicEntity mm:creates core:AgentRole core:has AgentRole mm:MusicArtist core:involvesAgent mr:MusicContent mm:AbstractScore mm:hasMusic EntityPart mm:Score mm:isRealisedIn mm:Recording Process mm:Recording mm:produces Recording mm:isRecordedBy core:hasTime Interval core:TimeInterval owl:unionOf jams:hasJAMS Annotation A core:involvesAgent Figure 4.4: Fragment of the JAMS ontology describing JAMS files and their provenance, musical objects and JAMS annotations. annotation, from a file encoded with the JAMS format, on a musical object (its target): either a recording or a score. A JAMS annotation entity and its musical object are put in relation by means of the property jams:hasJAMSAnnotation. An annotation is performed by an annotator jams:Annotator, has a time validity jams:hasMusicTimeInterval, and contains information of a certain type jams:AnnotationType (e.g. chords, keys, etc.). The validity indicates to which time frame, within a musical object, the annotation refers. For example, if an annotation reports the observation of a certain key, that key refers to a segment of the target musical object. Annotators may be of different types (e.g. expert annotator, software program), and are defined by the class jams:AnnotatorType. Finally, a jams:JAMSAnnotation is composed of a set of jams:JAMSObservation. Figure 4.5 depicts the JAMS Ontology fragment that models JAMS observations. The Provenance Ontology [238] is reused to model the provenance of JAMS annotations (Figure 4.4, box B). Each JAMS annotation derives from a JAMS file (jams:JAMSFile) which is either taken or derived (for example, translated from a file in a different format to the JAMS format) from a dataset jams:Dataset. A key aspect of observations and annotations is the identification of the musical object fragment they refer to. We model musical object fragments as musical time intervals core:MusicTimeInterval. Musical time intervals can be expressed in different ways, depending on the type of musical object. For example, if the subject of an observation (and in turn of an annotation) is a recording, then we 121 i i “output” — 2025/10/28 — 13:54 — page 122 — #136 i i i i i i Chapter 4. Harmonising Fragmented Data: A Comprehensive Workflow for Symbolic Data Integration mp:MusicProjection rdfs:subClassOf jams:JAMSObservationjams:JAMSAnnotation mp:Chord core:MusicTimeInterval core:MusicTimeDurationcore:MusicTimeIndex core:hasEndMusic TimeIndex core:hasMusicTimeDuration core:MusicTime IndexComponent core:hasMusicTime IndexComponent rdfs:Literal core:hasValue core:MusicTime ValueType core:hasValueType core:hasValueType core:hasValue jams:hasMusicTimeInterval jams:hasMusicTimeInterval C Prefixes rdfs: jams: mr: mp: http://www.w3.org/2000/01/rdf-schema# https://w3id.org/polifonia/ontology/jams/ https://w3id.org/polifonia/ontology/music-representation/ https://w3id.org/polifonia/ontology/music-projection/ jams:includesObservation jams:hasSubject jams:hasChordSubject core:hasStartMusic TimeIndex core:hasMusic TimeIndex xsd:int jams:hasConfidence Figure 4.5: Fragment of the JAMS Ontology describing JAMS annotations and JAMS observations. The red block C highlights how the time information has been modelled for handling different types of formats and standards. most probably identify its fragments in terms of seconds. If we deal with scores, we may want to use a combination of measures and beats. To make the ontology as flexible as possible for expressing musical time intervals, we model them as being defined by musical time indexes (core:MusicTimeIndex). Each musical time interval has a start time index and an end time index (plus potentially infinite internal time indexes). A musical time index is defined by one or more components (core:MusicTimeIndexComponent), each substantiated by a value (core:hasValue) and a value type (core:MusicTimeValueType). A musical time interval also has a duration (core:MusicTimeDuration) which is expressed by means of a value and a value type (usually seconds for recordings and beats for scores). Figure 4.6 shows an example of data from the Wikifonia subset (wikifonia_39) annotated using the JAMS Ontology. Starting from the individual highlighted by the red box (pon-res:AutumnInRomeComposition) we can trace information related to the piece entitled "Autumn in Rome". The file includes two annotations (Wikifonia39KeyAnnotation and Wikifonia39ChordAnnotation), derived from a score, hence their temporal information is expressed as a combination of beats and measures. The chord annotation (pon-res:Wikifonia39ChordAnnotation) contains two observations, the first starting at the beginning of the first measure 122 i i “output” — 2025/10/28 — 13:54 — page 123 — #137 i i i i i i 4.3. ChoCo: a Chord Corpus and a Data Transformation Workflow for Musical Harmony Knowledge Graphs Figure 4.6: Example of data modelled using JAMS Ontology, extracted from a track from the Wikifonia dataset. The track is annotated from a score, therefore annotations and observations contain time references expressed in beat and measure. 123 i i “output” — 2025/10/28 — 13:54 — page 130 — #144 i i i i i i Chapter 4. Harmonising Fragmented Data: A Comprehensive Workflow for Symbolic Data Integration 100101102103 no. of chords (log-x scale) 0.0 0.5 1.0 1.5 Density Number of chord observations in annotations 0 20 40 60 80 100 no. of chords 0.00 0.01 0.02 0.03 0.04 0.05 0.06 Number of unique chord classes in annotations Figure 4.11: Distribution of the number of chord observations per annotation (left, linear scale) and their distinct chord classes (right, log-x scale). The latter can also be considered as the cardinality of the chord set used by each annotation. 100101102 seconds (log-x scale) 0.0 0.5 1.0 1.5 2.0 2.5 Density Average duration of chord observations (audio) 100101102 beats (log-x scale) 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Average duration of chord observations (score) Figure 4.12: Distribution of chord durations for audio (left, seconds) and symbolic (right, beats) annotations on log-x scale. of chord observations in the progression; then averaging the resulting chord frequencies across all annotations. Analogously, Figure 4.13 (right) reports the same statistics after removal of consecutively repeated chords. This pre-processing step aims to mitigate consecutive repetitions (which may arise due to the different temporal granularity of chord observations, or possess a harmonic function) from inflating the chord count. Regardless of the counting method, the three most common chords in the dataset are: C:maj,G:maj, and F:maj. A similar analysis is also reported for chord n-grams, which are typically used 0 10000 20000 30000 40000 50000 60000 70000 80000 count C:maj G:maj F:maj D:maj A:maj Bb:maj G:7 C:7 D:7 Eb:maj A:min E:maj A:7 E:min D:min chords classes Absolute occurrences of chord classes 012345 frequency 1e 6 C:maj G:maj F:maj D:maj A:maj Bb:maj E:maj G:7 N A:min D:7 Eb:maj C:7 E:min D:min Relative chord occurrences per annotation 0 10000 20000 30000 40000 50000 60000 70000 count C:maj G:maj F:maj D:maj A:maj G:7 Bb:maj C:7 D:7 Eb:maj A:min A:7 E:maj E:min F:7 Absolute occurrences of chord classes (no rep) 012345 frequency 1e 6 C:maj G:maj F:maj D:maj A:maj Bb:maj G:7 E:maj D:7 A:min C:7 Eb:maj N E:min F:7 Relative chord occurrences per annotation (no rep) Figure 4.13: Absolute and relative occurrences of chord classes in the original annotations (left, centre-left), and after removal of consecutively repeated chords (right, centre-right). Absolute occurrences are counted and accumulated throughout the corpus, whereas relative occurrences are first aggregated per annotation, as frequencies, then averaged across the whole dataset. Note that the “N” chord class denotes the “silent chord” as per the Harte notation (obtained for all subsets). 130 i i “output” — 2025/10/28 — 13:54 — page 131 — #145 i i i i i i 4.3. ChoCo: a Chord Corpus and a Data Transformation Workflow for Musical Harmony Knowledge Graphs to find harmonic patterns in songs. To avoid trivial n-grams, these are computed after removal of consecutive repetitions (e.g. G:7, G:7, C:maj becoming G:7, C:maj). Table 4.4 ranks the first 10 n-grams based on their relative count (frequency). To conclude, the number of chord annotations for which the identity of the annotators is known is 796 (3.9% of the dataset). Order Rank Chord 1 Chord 2 Chord 3 Chord 4 Frequency Occurrences 2 1 G:maj C:maj - - 9.894371e-07 11560 2 C:maj G:maj - - 9.314316e-07 9968 3 C:maj F:maj - - 8.578674e-07 9837 4 D:maj G:maj - - 8.447899e-07 11229 5 G:7 C:maj - - 8.270923e-07 12590 6 G:maj D:maj - - 8.236944e-07 9591 7 F:maj C:maj - - 7.588854e-07 8547 8 D:7 G:maj - - 7.092709e-07 10673 9 A:maj D:maj - - 6.319998e-07 6925 10 C:7 F:maj - - 6.247398e-07 10362 3 1 G:maj C:maj G:maj - 4.156081e-07 4487 2 C:maj F:maj C:maj - 4.022300e-07 4167 3 D:maj G:maj D:maj - 3.518498e-07 4473 4 C:maj G:maj C:maj - 3.210295e-07 3209 5 G:maj D:7 G:maj - 2.757892e-07 3411 6 G:maj D:maj G:maj - 2.755515e-07 3483 7 C:maj G:7 C:maj - 2.685492e-07 3371 8 F:maj C:maj F:maj - 2.601499e-07 2660 9 A:maj E:maj A:maj - 2.201239e-07 1767 10 A:maj D:maj A:maj - 2.151695e-07 2450 4 1 G:maj C:maj G:maj C:maj 1.984606e-07 1933 2 C:maj G:maj C:maj G:maj 1.897574e-07 1746 3 C:maj F:maj C:maj F:maj 1.840459e-07 1693 4 F:maj C:maj F:maj C:maj 1.759950e-07 1509 5 D:maj G:maj D:maj G:maj 1.647309e-07 2256 6 G:maj D:maj G:maj D:maj 1.609514e-07 2105 7 D:7 G:maj D:7 G:maj 1.587393e-07 1873 8 A:maj E:maj A:maj E:maj 1.497483e-07 998 9 E:maj A:maj E:maj A:maj 1.453102e-07 1067 10 G:7 C:maj G:7 C:maj 1.338413e-07 1593 Table 4.4: Summary of the most common chord n-grams (n= 2,3,4), ranked by their relative occurrence (frequency) per chord annotation. The last column reports the corresponding total number of n-gram occurrences in the dataset (no aggregation). 131 i i “output” — 2025/10/28 — 13:54 — page 132 — #146 i i i i i i Chapter 4. Harmonising Fragmented Data: A Comprehensive Workflow for Symbolic Data Integration 4.4 Technical Validation To validate the data transformation workflow presented in Section 4.3.1 (Figure 4.1), focusing on the output of the JAMifier (generation of JAMS files from arbitrary chord collections) and the Chonverter (chord alignment and conversion) modules, we conducted two separate analyses: a groundtruth evaluation of JAMS files, and an expert validation of chord conversions. 4.4.1 Validation of the JAMifier As the goal of the JAMifier is to automatically generate a JAMS dataset given a music collection providing chord annotations and metadata in different formats, notations, and conventions, this first evaluation addresses the following question. How complete and accurate are ChoCo’s JAMS files – for metadata and harmonic annotations, after the JAMification? To answer this question, we carried out a series of tests to compare a sample of generated JAMS files with those that are expected from this process. This required the creation of a groundtruth dataset of JAMS files that were manually produced by two human annotators from a given template (the backbone of a JAMS file), and through manual inspection of the original collections. For example, given a sample of the Wikifonia subset, the validator was expected to fill the JAMS template by: opening the MusicXML file of each assigned piece; inserting the relevant metadata (title, composer, duration, etc.) into the appropriate fields; and finding the (Leadsheet) chord labels annotated on the score – to create a JAMS Observation out of each of them. Annotators were first instructed on the task, and a preliminary annotation trial was performed to assess their reliability. After the trial, annotators received 4 templates for each subset and produced 72 gold JAMS files in total. The corresponding JAMification output is then compared to the groundtruth to measure: (i) the coverage and the accuracy of the metadata; and (ii) the coverage and error of chord and key annotations. For the metadata, coverage is computed as the proportion of metadata fields in the gold JAMS that can also be found in the generated JAMS, regardless of their values. For example, if title,composers,genre, and duration are the expected metadata fields for a given JAMS file, and the generated counterpart only provides records for title and duration, coverage would account for 0.5(even if both title and duration are incorrect). To provide a complementary view, metadata accuracy 132 i i “output” — 2025/10/28 — 13:54 — page 133 — #147 i i i i i i 4.4. Technical Validation of common fields is computed as the normalised Levenshtein similarity among the generated and expected values for strings; or as the relative variance from the expected value for numerical fields (e.g. duration). The accuracies are then averaged for each JAMS file. The results of this evaluation are reported in Table 4.5, aggregated for each subset and separated from the identifiers that were extracted from the JAMification (e.g. MusicBrainz, Wikidata – c.f. Section 4.3.2). Overall, maximum accuracy and coverage are attained for most collections, and all the possible identifiers are always extracted with no errors. metadatametadata identifiersidentifiers coverage ↑accuracy ↑coverage ↑accuracy ↑ subset biab-internet-corpus 0.95 ±0.1 0.9243 ±0.0835 - - billboard 1.0 ±0.0 1.0 ±0.0 - - chordify 1.0 ±0.0 1.0 ±0.0 1.0 ±0.0 1.0 ±0.0 ireal-pro 1.0 ±0.0 1.0 ±0.0 - - isophonics 1.0 ±0.0 1.0 ±0.0 - - jaah 0.8036 ±0.0595 1.0 ±0.0 1.0 ±0.0 1.0 ±0.0 jazz-corpus 1.0 ±0.0 1.0 ±0.0 - - mozart-piano-sonatas 0.875 ±0.0 1.0 ±0.0 1.0 ±0.0 1.0 ±0.0 nottingham 1.0 ±0.0 1.0 ±0.0 - - real-book 1.0 ±0.0 1.0 ±0.0 - - robbie-williams 1.0 ±0.0 1.0 ±0.0 - - rock-corpus 1.0 ±0.0 1.0 ±0.0 - - rwc-pop 1.0 ±0.0 0.9999 ±0.0001 - - schubert-winterreise 1.0 ±0.0 1.0 ±0.0 1.0 ±0.0 1.0 ±0.0 uspop2002 1.0 ±0.0 0.9661 ±0.062 - - weimar 1.0 ±0.0 0.9878 ±0.0243 1.0 ±0.0 1.0 ±0.0 when-in-rome 0.7976 ±0.0558 0.9608 ±0.0694 - - wikifonia 0.95 ±0.1 0.95 ±0.1 - - Table 4.5: Average coverage and accuracy of metadata and identifiers in the generated JAMS files, per ChoCo subset. The dash symbol denotes a subset that does not provide any identifiers. For the harmonic annotations in the JAMS files, comparison with the gold counterparts is focused on coverage and error – reported independently for times (e.g. the onset of a chord occurrence), durations (e.g. how long a chord occurrence spans), and labels (e.g. a C:maj chord) of the observations in each annotation. The evaluation is thus in line with the structure of an observation in JAMS’ annotations (see Section 4.3.1 and Listings 4.1, 4.2). In this case, coverage measures the amount of the overlap between the generated and the expected observation 133 i i “output” — 2025/10/28 — 13:54 — page 134 — #148 i i i i i i Chapter 4. Harmonising Fragmented Data: A Comprehensive Workflow for Symbolic Data Integration fields, without taking order into account (this is because an extra observation may have been inserted by the annotator, thus breaking the desired alignment for comparison). For example, if (C:maj, G:maj, D:7, F:maj) and (N, C:maj, G:maj, D:7) are the labels of a generated chord annotation and the corresponding gold, respectively, the silent chord “N” breaks the alignment of those sequences. In this case, coverage would still be 3/4, as all the other chord labels are included in generated annotation. Instead, errors are computed from a 1-to-1 comparison of fields – which are assumed to be aligned. The latter can be reported according to the unit of measure of each field: seconds and beats for time and duration, and normalised Levenshtein distance for labels (string values). Key coverages ↑Key coverages ↑Key errors ↓Key errors ↓Chord coverages ↑Chord coverages ↑Chord errors ↓Chord errors ↓ subset type time duration label time duration label time duration label time duration label billboard audio 1.0 ±0.0 1.0 ±0.0 1.0 ±0.0 0.0 ±0.0 0.0 ±0.0 0.0 ±0.0 1.0 ±0.0 1.0 ±0.0 1.0 ±0.0 0.0 ±0.0 0.0 ±0.0 0.0 ±0.0 chordify audio 1.0 ±0.0 1.0 ±0.0 1.0 ±0.0 0.0 ±0.0 0.0 ±0.0 0.0 ±0.0 1.0 ±0.0 1.0 ±0.0 1.0 ±0.0 0.0 ±0.0 0.0 ±0.0 0.0 ±0.0 isophonics audio 1.0 ±0.0 1.0 ±0.0 1.0 ±0.0 0.0 ±0.0 0.0 ±0.0 0.0 ±0.0 1.0 ±0.0 1.0 ±0.0 1.0 ±0.0 0.0 ±0.0 0.0 ±0.0 0.0 ±0.0 jaah audio 1.0 ±0.0 1.0 ±0.0 1.0 ±0.0 0.0 ±0.0 0.0 ±0.0 0.0 ±0.0 0.95 ±0.1 0.95 ±0.1 1.0 ±0.0 0.06 ±0.13 0.06 ±0.13 0.0 ±0.0 robbie-williams audio 1.0 ±0.0 1.0 ±0.0 1.0 ±0.0 0.0 ±0.0 0.0 ±0.0 0.0 ±0.0 1.0 ±0.0 1.0 ±0.0 1.0 ±0.0 0.0 ±0.0 0.0 ±0.0 0.0 ±0.0 rwc-pop audio - - - - - - 1.0 ±0.0 0.53 ±0.45 1.0 ±0.0 0.0 ±0.0 0.0 ±0.0 0.0 ±0.0 schubert-winterreise audio 1.0 ±0.0 1.0 ±0.0 1.0 ±0.0 0.0 ±0.0 0.0 ±0.0 0.0 ±0.0 1.0 ±0.0 1.0 ±0.0 1.0 ±0.0 0.0 ±0.0 0.0 ±0.0 0.0 ±0.0 uspop2002 audio - - - - - - 1.0 ±0.0 0.3 ±0.26 1.0 ±0.0 0.0 ±0.0 0.0 ±0.0 0.0 ±0.0 weimar audio 1.0 ±0.0 1.0 ±0.0 1.0 ±0.0 0.0 ±0.0 0.0 ±0.0 0.0 ±0.0 1.0 ±0.0 1.0 ±0.0 1.0 ±0.0 0.0 ±0.0 0.0 ±0.0 0.0 ±0.0 biab-internet-corpus score 1.0 ±0.0 1.0 ±0.0 1.0 ±0.0 0.0 ±0.0 0.0 ±0.0 0.0 ±0.0 0.95 ±0.1 1.0 ±0.0 1.0 ±0.0 0.05 ±0.1 0.0 ±0.0 0.0 ±0.0 ireal-pro score 1.0 ±0.0 1.0 ±0.0 1.0 ±0.0 0.0 ±0.0 0.0 ±0.0 0.0 ±0.0 1.0 ±0.0 1.0 ±0.0 1.0 ±0.0 0.0 ±0.00 0.0 ±0.0 0.0 ±0.0 jazz-corpus score 1.0 ±0.0 1.0 ±0.0 1.0 ±0.0 0.0 ±0.0 0.0 ±0.0 0.0 ±0.0 1.0 ±0.0 1.0 ±0.0 1.0 ±0.0 0.0 ±0.0 0.0 ±0.0 0.0 ±0.0 mozart-piano-sonatas score 0.5 ±0.58 0.0 ±0.0 0.5 ±0.58 62.55 ±125.03 139.75 ±83.75 0.25 ±0.29 0.85 ±0.3 0.88 ±0.25 0.75 ±0.5 0.25 ±0.5 0.15 ±0.19 0.15 ±0.3 nottingham score 1.0 ±0.0 1.0 ±0.0 1.0 ±0.0 0.0 ±0.0 0.0 ±0.0 0.0 ±0.0 0.75 ±0.25 1.0 ±0.0 1.0 ±0.0 0.85 ±0.6 0.0 ±0.0 0.0 ±0.0 real-book score 1.0 ±0.0 1.0 ±0.0 1.0 ±0.0 0.0 ±0.0 0.0 ±0.0 0.0 ±0.0 1.0 ±0.0 1.0 ±0.0 1.0 ±0.0 0.0 ±0.0 0.0 ±0.0 0.0 ±0.0 rock-corpus score 1.0 ±0.0 1.0 ±0.0 1.0 ±0.0 0.0 ±0.0 0.0 ±0.0 0.0 ±0.0 1.0 ±0.0 1.0 ±0.0 1.0 ±0.0 0.0 ±0.0 0.0 ±0.0 0.0 ±0.0 schubert-winterreise score 1.0 ±0.0 1.0 ±0.0 1.0 ±0.0 0.0 ±0.0 0.0 ±0.0 0.0 ±0.0 1.0 ±0.0 1.0 ±0.0 1.0 ±0.0 0.0 ±0.0 0.0 ±0.0 0.0 ±0.0 when-in-rome score 1.0 ±0.0 1.0 ±0.0 1.0 ±0.0 0.0 ±0.0 0.0 ±0.0 0.0 ±0.0 1.0 ±0.0 1.0 ±0.0 1.0 ±0.0 0.0 ±0.0 0.0 ±0.0 0.0 ±0.0 wikifonia score 1.0 ±0.0 1.0 ±0.0 1.0 ±0.0 0.0 ±0.0 0.0 ±0.0 0.0 ±0.0 1.0 ±0.0 0.92 ±0.17 0.75 ±0.35 0.0 ±0.0 0.1 ±0.2 0.11 ±0.18 Table 4.6: Evaluation of chord and key annotations in the generated JAMS files on the test samples, reported for times, durations, and labels of their observations, and averaged for each subset. Coverage of observation values ranges from 0 (all the expected values are not found in the generated annotation) to 1 (all the expected values are included). Errors are given as seconds (audio) or beats (symbolic) for times and durations, respectively; and as normalised text similarities for labels. Table 4.6 reports the results of this last evaluation for both key and chord annotations, where each metric is averaged by subset (mean and standard deviation). Results show good coverage and minimum error for most subsets, thus confirming the quality of the JAMification output. An exception is the Mozart Piano Sonata collection, for which low coverage and high errors are reported for key annotations. After having manually compared the JAMS sample for this subset, we found that the observations annotated by our validators in the gold set used a different temporal granularity (e.g. merging two consecutively repeated observations and aggregating their time and duration), compared to the JAMification output. Although this affected the evaluation results, both these annotations can be deemed equivalent. 134 i i “output” — 2025/10/28 — 13:54 — page 135 — #149 i i i i i i 4.4. Technical Validation 4.4.2 Validation of the Chonverter Following the data transformation workflow illustrated in Figure 4.1, we recall that the output of the JAMification step that does not natively provide Harte chord labels undergoes an alignment/conversion process through the Chonverter. First, the Chonverter aligns chord labels to one of the three chord families introduced in Section 4.3.1, namely: Leadsheet (Harte), Roman, and Polychord. Then, a syntactic conversion is performed on each chord class, independently, to infer the corresponding Harte label. Evaluating the output of the Chonverter can thus be formulated as follows. How accurate and musically plausible are the chord alignment and chord conversion steps? Conversely to the previous evaluation, addressing this question requires musical expertise and familiarity with different chord notations. Therefore, we performed a 2-step evaluation with music experts to validate the alignment and the conversion rules. Four participants with at least 5 years of musical training were recruited for this experiment. Participants were first introduced to the task, and asked to express their level of familiarity with the different chord notations, and the validation methodology. Given the nature of the validation, no personal record was recorded from participants and minimal risk clearance was granted from the Research Ethics Office of King’s College London (registration number: MRSP21/22-32842). Step 1 The first step focused on validating the context-free grammars used to parse chords in the original formats and aligning them to the corresponding chord families. Participants were presented with 3 different grammars, including 250 mapping rules to validate. Whenever a rule was deemed incorrect, participants were asked to provide the expected mapping. Step 2 Once chords were converted, the final result of the conversion was validated. This step also allowed for the validation of other conversion types that were not validated in Step 1, such as Roman numerals and Polychords. In addition, even for annotations originally provided in Leadsheet, this step allows for the validation of added/removed notes and inversions. The first step allowed to validate all the grammar rules used for decomposing leadsheet chords into their constituting degrees. Each grammar consists of a set 135 i i “output” — 2025/10/28 — 13:54 — page 136 — #150 i i i i i i Chapter 4. Harmonising Fragmented Data: A Comprehensive Workflow for Symbolic Data Integration Subset Validated chords Chord type Correct conversions Incorrect conversions Accuracy ↑ ireal-pro 39 leadsheet 37 2 0.949 rock-corpus 40 roman 40 0 1.000 weimar 37 leadsheet 37 0 1.000 when-in-rome 40 roman 40 0 1.000 wikifonia 40 leadsheet 39 1 0.975 average 196 all 193 3 0.985 Table 4.7: Evaluation of chord conversions performed by music experts on a selection of ChoCo subsets. of shorthands grouped into classes. For example, the class referring to minor chords is composed of the shorthands "m" and "min". Each class is then mapped to the degrees that compose that type of chord: for minor chords, the degrees associated with that class are 1, b3,5. This type of validation was required due to the limited musical background of the dataset’s curators. All grammar rules reported incorrect by the experts were corrected and revised. A total of 27 rules within the validated grammars were updated. The corrections were of two main types: i) correct shorthands but incorrect degrees: the group of shorthands assigned to degrees was correct, but the degrees into which the chord was decomposed had one or more errors; ii) inconsistent group of shorthands: the grouping of shorthands in classes was incorrect. In this case, the shorthand(s) not belonging to the class was moved to the correct class if it existed, otherwise a new class was created. This implies that the preliminary chord alignment of the Chonverter is potentially error free. The second validation step consisted in distributing spreadsheets in which the original chords were shown in the first column whereas the second column showed the chords converted by the Chonverter module. Before starting this validation phase, all participants were provided with a thorough documentation of all types of annotation used, including Harte. Furthermore, chords annotated in the Roman Numeral format, which had not been validated in the previous step, were tested for the first time. Experts were asked to mark whether the conversion to the Harte format was correct or not. The evaluation results are as the percentage of corrected chords out of the total (Table 4.7). 4.5 Usage Notes The availability of a large chord dataset, providing high-quality harmonic annotations with temporal information, content metadata, and links to external 136 i i “output” — 2025/10/28 — 13:54 — page 137 — #151 i i i i i i 4.5. Usage Notes resources, is of considerable interest to several research communities. In the field of MIR, chord datasets are a fundamental prerequisite for training and evaluating content-based music algorithms that can accommodate a variety of tasks – from chord recognition and cover song detection, to automatic composition systems. For musicology and computational music analysis, the scale and diversity of ChoCo [90] would enable large scale cross-corpus studies across different musical periods, genres, and artists (e.g. uncovering potential influences), and the KG can also be leveraged to run complex queries entailing certain musicological properties of chords, rather than relying exclusively on their notation-specific label. Also the SW community would benefit from the introduction of high quality chord data that can be linked to existing Web resources. In turn, this opens up new scenarios and research opportunities for the aforementioned communities. 4.5.1 Applications and tasks Given the diversity, size, and quality of the corpus, we expect ChoCo to enable novel applications in Music Technology, other than supporting the design and the evaluation of methods addressing specific tasks in both MIR and computational music analysis. Besides the aforementioned applications in music listening and recommendation, another case study involves the advancement of systems for machine creativity. In the context of our work, these include automatic (or semi-automatic) composition, with particular focus on arrangement generation [357] (generating a chord progression, possibly given a melody to accompany); and melody generation through harmonic conditioning [128] (generating a melody to play along with a chord progression that is provided as a harmonic template). In ChoCo v1.0, this is enabled by the integrated Wikifonia and Nottingham collections; and in future versions, with melodic data from Rock Corpus,Weimar, and the Band-in-a-Box collections. Not only does ChoCo support the creative capabilities of such systems – by providing a considerable amount of quality training data, but it also contributes to their automatic evaluation. In fact, the evaluation of music generation systems has recently attracted a growing interest in the field, due to the concerning ethical implications these tools are raising [369]. On one hand, this involves the extraction of statistical features quantifying the degree of alignment between a generated repertoire and the training material, with respect to certain musical properties [414]; on the other hand, it concerns the detection of potential sources 137 i i “output” — 2025/10/28 — 13:54 — page 138 — #152 i i i i i i Chapter 4. Harmonising Fragmented Data: A Comprehensive Workflow for Symbolic Data Integration of plagiarism in generated music within and beyond the training set [418]. Another application domain that can benefit from the Chord Corpus is that of music pedagogy. For example, TheoryTab15 allows users to choose from a repertoire of popular songs and visualise their harmonic/melodic structure during playback – with chords encoded in both Leadsheet and Roman notations, and projected in such a way as to facilitate the theoretical understanding of the song. Chordify uses chord recognition systems to infer and align chord progressions from audio recordings, and provides support for practising them with guitar, piano, and ukulele. Despite their value, both the technology and the data powering these commercial tools are not openly available, thereby decreasing their overall wider use. In contrast, ChoCo provides an attractive open and linked solution, with its modular architecture enabling the semantic description of chords according to the desired level of complexity and granularity (e.g. an educational ontology for chords might provide a simpler vocabulary). This makes it more suitable for educational purposes. In the context of MIR, the use of ChoCo can support a multitude of tasks. The nature of its contribution is twofold: (i) it provides an unprecedented amount of training data, which is often essential for the effectiveness of supervised methods; (ii) it contributes to the development of graph-based methodologies for music analysis that can leverage the semantic representation of chord progressions. For instance, a central research area in MIR is music similarity, which in turn encompasses a number of interrelated tasks, including cover song detection – useful for music cataloguing and to support court decisions in music plagiarism [281]; and content-based music retrieval, aiming to search scores or performances from musical repositories using either symbolic queries, singing (alias query-by-humming), or by playing a smart instrument [389]. Another example of a MIR task that would benefit from ChoCo is music structure analysis [96], which is concerned with the detection and labelling of structural segments related to musical form – a task that strongly relies on the use of harmonic/melodic features [17]. Other tasks of interest include music tagging [31], such as music genre/style classification and composer/artist identification. Finally, examples of tasks of musicological interest that would benefit from ChoCo include pattern mining,cadence detection, and local key identification. 15https://www.hooktheory.com/theorytab 138 i i “output” — 2025/10/28 — 13:54 — page 139 — #153 i i i i i i 4.5. Usage Notes 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 count Audio music Symbolic music (e.g. MusicXML) Both, jointly (e.g. multi-modal analyses) Other I do research or industrial practice in: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 count RDF data Structured data (e.g CSV, JSON, etc.) Unstructured data from the Web (e.g. text, web pages, etc.) Databases (e.g. MusicBrainz, Genius, etc.) Other My work on content-based music data focuses, or will focus, on: Figure 4.14: Overview of responses to Questions 2 (music domains, left), and 3 (data types, right) in the survey. 4.5.2 Online survey Since ChoCo is a new resource for the SW, MIR and Musicology communities, we discuss here evidence for potential adoption. To gather such evidence, we performed an online survey in which we directly ask potential adopters 10 questions regarding their background, relevance, and interest in working with chord data. The online survey was distributed in the SW, International Society for MIR (ISMIR), and Digital Musicology mailing lists, gathering a total of N= 53 responses. The survey was conducted via Google Forms – without recording any personal data from participants or any contact information. Results are illustrated in Figures 4.14 and 4.15. Except for questions 1-3 and 12 (multiple choices), all questions ask participants to quantify the agreement with the statement made from 1 (absolutely disagree) to 5 (absolutely agree), 3 being a neutral response (neither agree nor disagree). In the first three questions we assess the background of the respondents, finding that 38 work in MIR, 27 in Musicology, 13 in SW, and 5 are also involved in other fields (AI, Music Theory, Music Interaction). Most respondents do research or industrial practice using audio (29) or symbolic music (33), or both (18), focusing primarily on structured data when conducting content-based music studies (Figure 4.14). Nevertheless, music researchers also make extensive use of unstructured data and music databases, and 13 of them (24% of respondents), utilise RDF data. From questions 4-11 we found that: 64% of respondents have encountered the need for chord datasets providing high-quality timed annotations of harmonic progressions, covering one or more genres/styles; 47% believe that currently existing chord datasets are not of sufficient size for their practise (whereas 41.5% have a neutral position); about 60.3% argue that such datasets do not provide content metadata sufficiently rich and informative to their needs (with another 35.8% being neutral); and 51% believe that links to external resources (e.g. MusicBrainz, 139 i i “output” — 2025/10/28 — 13:54 — page 242 — #256 i i i i i i Bibliography [1] Jakob Abeßer, Stefan Balke, Klaus Frieler, Martin Pfleiderer, and Meinard Müller. Deep learning for jazz walking bass transcription. In Audio Engineering Society Conference: 2017 AES International Conference on Semantic Audio. Audio Engineering Society, 2017. [2] Manel Achichi, Pasquale Lisena, Konstantin Todorov, Raphaël Troncy, and Jean Delahousse. DOREMUS: A graph of linked musical works. In International Semantic Web Conference, pages 3–19. Springer, 2018. [3] Elie Adam, E Noune, and Yasmina Yared. A system for music similarity search based on harmonic content. Beirut, Lebanon, 2010. [4] Alessandro Adamou, Simon Brown, Helen Barlow, Carlo Allocca, and Mathieu d’Aquin. Crowdsourcing Linked Data on listening experiences through reuse and enhancement of library data. International Journal on Digital Libraries, 20(1):61–79, 2019. [5] Eytan Agmon. Functional Harmony Revisited: A Prototype-Theoretic Approach. Music Theory Spectrum, 17(2):196–214, 10 1995. [6] Andrea Agostinelli, Timo I. Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, Matt Sharifi, Neil Zeghidour, and Christian Frank. MusicLM: Generating Music From Text, 2023. 242 i i “output” — 2025/10/28 — 13:54 — page 243 — #257 i i i i i i Bibliography [7] Alo Allik, György Fazekas, and Mark B. Sandler. An Ontology for Audio Features. In Mandel et al. [254], pages 73–79. [8] Greg Aloupis, Thomas Fevens, Stefan Langerman, Tomomi Matsui, Antonio Mesa, Yurai Núñez Rodríguez, David Rappaport, and Godfried T. Toussaint. Algorithms for Computing Geometric Measures of Melodic Similarity. Computer Music Journal, 30(3):67–76, 2006. [9] William G Andrews and Molly Sclater. Materials of Western Music. Alfred Music Publishing, 1997. [10] Renzo Angles, Marcelo Arenas, Pablo Barceló, Aidan Hogan, Juan L. Reutter, and Domagoj Vrgoc. Foundations of Modern Query Languages for Graph Databases. ACM Computing Surveys, 50(5):68:1–68:40, 2017. [11] Renzo Angles and Claudio Gutiérrez. Survey of graph database models. ACM Computing Surveys, 40(1):1:1–1:39, 2008. [12] Marcelo Gabriel Armentano, Walter A. De Noni, and Hernán F. Cardoso. Genre classification of symbolic pieces of music. J. Intell. Inf. Syst., 48(3):579–599, 2017. [13] MIDI Manufacturers Association. The Complete MIDI 1.0 Detailed Specification: Incorporating All Recommended Practices. MIDI Manufacturers Association, 1996. [14] Agnes Axelsson and Gabriel Skantze. Multimodal User Feedback During Adaptive Robot-Human Presentations. Frontiers Comput. Sci., 3:741148, 2021. [15] Tadas Baltrusaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal Machine Learning: A Survey and Taxonomy. IEEE Trans. Pattern Anal. Mach. Intell., 41(2):423–443, 2019. [16] Ana M Barbancho, Isabel Barbancho, Lorenzo J Tardón, and Emilio Molina. Database of Piano Chords: An Engineering View of Harmony. Springer, 2013. [17] Mark A Bartsch and Gregory H Wakefield. Audio thumbnailing of popular music using chroma-based representations. IEEE/ Transactions on Multimedia, 7(1):96–104, 2005. 243 i i “output” — 2025/10/28 — 13:54 — page 244 — #258 i i i i i i Bibliography [18] Sean Bechhofer, Simon Dixon, George Fazekas, Thomas Wilmering, and Kevin Page. Computational analysis of the live music archive. In Cumming et al. [77]. [19] Sean Bechhofer, Kevin Page, and David De Roure. Hello Cleveland! Linked Data publication of live music archives. In 2013 14th International Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS), pages 1– 4. IEEE, 2013. [20] Juan Pablo Bello, Elaine Chew, and Douglas Turnbull, editors. ISMIR 2008, 9th International Conference on Music Information Retrieval, ISMIR 2008, September 2008. [21] Juan Pablo Bello and Jeremy Pickens. A Robust Mid-Level Representation for Harmonic Content in Music Signals. In ismir2005 [207], pages 304–311. [22] Emmanouil Benetos, Simon Dixon, Zhiyao Duan, and Sebastian Ewert. Automatic music transcription: An overview. IEEE Signal Processing Magazine, 36(1):20–30, 2019. [23] Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives, 2013. [24] Ian D. Bent. musical notation, July 2024. [25] Ian D. Bent, David W. Hughes, Robert C. Provine, Richard Rastall, Anne Kilmer, David Hiley, Janka Szendrei, Thomas B. Payne, Margaret Bent, and Geoffrey Chew. Notation, 2001. [26] B. Benward and M.N. Saker. Music in Theory and Practice. Number v. 1 in Music in Theory and Practice. McGraw-Hill, 2003. [27] Jacopo Berardinis, Valentina Carriero, Albert Meroño-Peñuela, Andrea Poltronieri, and Valentina Presutti. The Music Meta Ontology: A Flexible Semantic Model for the Interoperability of Music Metadata. In Sarti et al. [349]. [28] Adam Berenzweig, Beth Logan, Daniel P. W. Ellis, and Brian P. W. Whitman. A large-scale evaluation of acoustic and subjective music-similarity measures. Comput. Music J., 28(2):63–76, June 2004. 244 i i “output” — 2025/10/28 — 13:54 — page 245 — #259 i i i i i i Bibliography [29] Tim Berners-Lee, James Hendler, and Ora Lassila. The semantic web. Scientific american, 284(5):34–43, may 2001. [30] Lawrence F. Bernstein and Joseph P. Olive. Computers and the 16th-century Chanson a pilot project at the University of Chicago. Computers and the Humanities, 3(3):153–160, Jan 1969. [31] Thierry Bertin-Mahieux, Douglas Eck, and Michael Mandel. Automatic tagging of audio: The state-of-the-art. In Machine audition: Principles, Algorithms and Systems, pages 334–352. IGI Global, 2011. [32] Emmanuel Bigand, Richard Parncutt, and Fred Lerdahl. Perception of musical tension in short chord sequences: The influence of harmonic function, sensory dissonance, horizontal motion, and musical training. Perception & Psychophysics, 58(1):125–141, Jan 1996. [33] Louis Bigo, Laurent Feisthauer, Mathieu Giraud, and Florence Levé. Relevance of musical features for cadence detection. In Gómez et al. [158], pages 355–361. [34] Rachel Bittner, Magdalena Fuentes, David Rubinstein, Andreas Jansson, Keunwoo Choi, and Thor Kell. mirdata: Software for Reproducible Usage of Datasets. In Flexer et al. [139], pages 99–106. [35] Eva Blomqvist, Karl Hammar, and Valentina Presutti. Engineering Ontologies with Patterns - The eXtreme Design Methodology. In Ontology Engineering with Ontology Design Patterns - Foundations and Applications, volume 25 of Studies on the Semantic Web. IOS Press, Amsterdam, 2016. [36] Eva Blomqvist, Valentina Presutti, Enrico Daga, and Aldo Gangemi. Experimenting with eXtreme Design. In Knowledge Engineering and Management by the Masses. EKAW 2010, volume 6317, pages 120–134. Springer, Berlin, Heidelberg, 2010. [37] Margaret A. Boden. Understanding creativity. The Journal of Creative Behavior, 1992. [38] Margaret A. Boden. The creative mind: Myths and mechanisms. Routledge, 2004. 245 i i “output” — 2025/10/28 — 13:54 — page 246 — #260 i i i i i i Bibliography [39] Paul M. Bodily and Dan Ventura. Explainability: An Aesthetic for Aesthetics in Computational Creative Systems. In François Pachet, Anna Jordanous, and Carlos León, editors, Proceedings of the Ninth International Conference on Computational Creativity, ICCC 2018, Salamanca, Spain, June 25-29, 2018, pages 153–160. Association for Computational Creativity (ACC), 2018. [40] Carl Boettiger. rdflib: A high level wrapper around the redland package for common rdf applications, 2018. [41] Thomas Bottini, Valentina Anita Carriero, Jason Carvalho, Philippe Cathé, Fiorela Ciroku, Enrico Daga, Marilena Daquino, Achille Davy-Rigaux, Marco Guillotel-Nothmann, Gurrieri, Philo van Kemenade, Eleonora Marzi, Albert Meroño Peñuelala, Paul Mulholland, Elena Musumeci, Valentina Presutti, and Andrea Scharnhorst. D1.1 Roadmap and pilot requirements 1st version. Technical report, EU Commission, The Polifonia consortium, 2021. [42] Jean-Pierre Briot, Gaëtan Hadjeres, and François-David Pachet. Deep learning techniques for music generation, volume 1. Springer, 2020. [43] George Bruseker, Nicola Carboni, and Anaïs Guillem. Cultural heritage data management: the role of formal ontology and CIDOC CRM. Heritage and archaeology in the digital age: acquisition, curation, and dissemination of spatial cultural heritage data, pages 93–131, 2017. [44] Nick Bryan-Kinns, Berker Banar, Corey Ford, Courtney N. Reed, Yixiao Zhang, Simon Colton, and Jack Armitage. Exploring xai for the arts: Explaining latent space in generative music, 2023. [45] Michel Buffa, Elena Cabrio, Michael Fell, Fabien Gandon, Alain Giboin, Romain Hennequin, Franck Michel, Johan Pauwels, Guillaume Pellerin, Maroua Tikat, et al. The WASABI dataset: cultural, lyrics and audio analysis metadata about 2 million popular commercially released songs. In The Semantic Web: 18th International Conference, ESWC 2021, Virtual Event, June 6–10, 2021, Proceedings 18, pages 515–531. Springer, 2021. [46] Morgan Buisson, Pablo Alonso-Jiménez, and Dmitry Bogdanov. Ambiguity Modelling with Label Distribution Learning for Music Classification. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 611–615, 2022. 246 i i “output” — 2025/10/28 — 13:54 — page 247 — #261 i i i i i i Bibliography [47] Benjamin Burger, Phillip M Maffettone, Vladimir V Gusev, Catherine M Aitchison, Yang Bai, Xiaoyan Wang, Xiaobo Li, Ben M Alston, Buyi Li, Rob Clowes, et al. A mobile robotic chemist. Nature, 583(7815):237–241, 2020. [48] John Ashley Burgoyne, Jonathan Wild, and Ichiro Fujinaga. An Expert Ground Truth Set for Audio Chord Recognition and Music Analysis. In Klapuri and Leider [219], pages 633–638. [49] Bryan R. Burnham, Emma Long, and Jake Zeide. Pitch direction on the perception of major and minor modes. Attention, Perception, & Psychophysics, 83(1):399–414, Jan 2021. [50] Alison Callahan, Jose Cruz-Toledo, Peter Ansell, and Michel Dumontier. Bio2RDF Release 2: Improved Coverage, Interoperability and Provenance of Life Science Linked Data. In Philipp Cimiano, Óscar Corcho, Valentina Presutti, Laura Hollink, and Sebastian Rudolph, editors, The Semantic Web: Semantics and Big Data, 10th International Conference, ESWC 2013, Montpellier, France, May 26-30, 2013. Proceedings, volume 7882 of Lecture Notes in Computer Science, pages 200–212. Springer, may 2013. [51] Julio José Carabias-Orti, Francisco J. Rodríguez-Serrano, Pedro VeraCandeas, Nicolás Ruiz-Reyes, and Francisco J. Cañadas-Quesada. An audio to score alignment framework using spectral factorization and dynamic time warping. In Müller and Wiering [286], pages 742–748. [52] Valentina Anita Carriero, Fiorela Ciroku, Jacopo de Berardinis, Delfina Sol Martinez Pandiani, Albert Meroño-Peñuela, Andrea Poltronieri, and Valentina Presutti. Semantic integration of mir datasets with the polifonia ontology network. In Lee et al. [239]. [53] Valentina Anita Carriero et al. The landscape of ontology reuse approaches. Applications and Practices in Ontology Design, Extraction, and Reasoning, 49:21, 2020. [54] Valentina Anita Carriero, Aldo Gangemi, Maria Letizia Mancinelli, Ludovica Marinucci, Andrea Giovanni Nuzzolese, Valentina Presutti, and Chiara Veninata. ArCo: The Italian cultural heritage knowledge graph. 247 i i “output” — 2025/10/28 — 13:54 — page 248 — #262 i i i i i i Bibliography In The Semantic Web–ISWC 2019: 18th International Semantic Web Conference, Auckland, New Zealand, October 26–30, 2019, Proceedings, Part II 18, pages 36–52. Springer, 2019. [55] Valentina Anita Carriero, Aldo Gangemi, Maria Letizia Mancinelli, Andrea Giovanni Nuzzolese, Valentina Presutti, and Chiara Veninata. Patternbased design applied to cultural heritage knowledge graphs. Semantic Web, 12(2):313–357, 2021. [56] Tristan Carsault, Jérôme Nika, and Philippe Esling. Using Musical Relationships Between Chord Labels in Automatic Chord Extraction Tasks. In Gómez et al. [158], pages 18–25. [57] Shan Carter and Michael Nielsen. Using artificial intelligence to augment human intelligence. Distill, 2(12):e9, 2017. [58] Yunkee Chae, Junghyun Koo, Sungho Lee, and Kyogu Lee. Exploiting time-frequency conformers for music audio enhancement. In Proceedings of the 31st ACM International Conference on Multimedia, MM ’23, page 2362–2370, New York, NY, USA, 2023. Association for Computing Machinery. [59] Wei Chai and Barry Vercoe. Detection of key change in classical piano music. In ismir2005 [207], pages 468–473. [60] Ruofeng Chen and Ming Li. Music Structural Segmentation by Combining Harmonic and Timbral Information. In Klapuri and Leider [219], pages 477–482. [61] Tsung-Ping Chen, Li Su, et al. Functional Harmony Recognition of Symbolic Music Data with Multi-task Recurrent Neural Networks. In Gómez et al. [158], pages 90–97. [62] Samira Si-said Cherfi, Christophe Guillotel, Fayçal Hamdi, Philippe Rigaux, and Nicolas Travers. Ontology-Based Annotation of Music Scores. In Óscar Corcho, Krzysztof Janowicz, Giuseppe Rizzo, Ilaria Tiddi, and Daniel Garijo, editors, Proceedings of the Knowledge Capture Conference, K-CAP 2017, pages 10:1–10:4, New York, NY, USA, 2017. Association for Computing Machinery. 248 i i “output” — 2025/10/28 — 13:54 — page 249 — #263 i i i i i i Bibliography [63] Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu. Selfsupervised learning with random-projection quantizer for speech recognition, 2022. [64] Pierre Choffé and Françoise Leresche. DOREMUS: connecting sources, enriching catalogues and user experience. In 24th IFLA World Library and Information Congress, pages 1–20, 2016. [65] Keunwoo Choi, György Fazekas, Kyunghyun Cho, and Mark Sandler. A tutorial on deep learning for music information retrieval, 2018. [66] Anna-Maria Christodoulou, Olivier Lartillot, and Alexander Refsum Jensenius. Multimodal music datasets? Challenges and future goals in music processing. International Journal of Multimedia Information Retrieval, 13(3):37, Aug 2024. [67] Ching-Hua Chuan, Elaine Chew, et al. A hybrid system for automatic generation of style-specific accompaniment. In Proceedings of the 4th international joint workshop on computational creativity, pages 57–64. Goldsmiths, University of London London, 2007. [68] Kenneth Ward Church. Word2Vec. Natural Language Engineering, 23(1):155–162, 2017. [69] J.P. Clendinning and E.W. Marvin. The Musician’s Guide to Theory and Analysis: Third Edition. W.W. Norton, 2016. [70] Richard Cohn, Brian Hyer, Carl Dahlhaus, Julian Anderson, and Charles Wilson. Harmony, 2001. [71] Nick Collins, V Ruzicka, and Mick Grierson. Remixing AIs: mind swaps, hybrainity, and splicing musical models. In Proc. The Joint Conference on AI Music Creativity, 2020. [72] Simon Colton, John William Charnley, and Alison Pease. Computational Creativity Theory: The FACE and IDEA Descriptive Models. In ICCC, pages 90–95. Mexico City, 2011. [73] Simon Colton and Dan Ventura. You Can’t Know my Mind: A Festival of Computational Creativity. In Simon Colton, Dan Ventura, Nada Lavrac, and Michael Cook, editors, Proceedings of the Fifth International Conference 249 i i “output” — 2025/10/28 — 13:54 — page 250 — #264 i i i i i i Bibliography on Computational Creativity, ICCC 2014, Ljubljana, Slovenia, June 10-13, 2014, pages 351–354. computationalcreativity.net, 2014. [74] Norman D Cook and Takashi X Fujisawa. The psychophysics of harmony perception: Harmony is a three-tone phenomenon. Empirical Musicology Review, 2006. [75] T. Crawford and L. Gibson. Modern Methods for Musicology: Prospects, Proposals, and Realities. Digital Research in the Arts and Humanities. Taylor & Francis, 2016. [76] Markus Cremer. A system for harmonic analysis of polyphonic music. In Audio Engineering Society Conference: 25th International Conference: Metadata for Audio. Audio Engineering Society, 2004. [77] Julie Cumming, Jin Ha Lee, Brian McFee, Markus Schedl, Johanna Devaney, Cory McKay, Eva Zangerle, and Timothy de Reuse, editors. Proceedings of the 21th International Society for Music Information Retrieval Conference, ISMIR 2020, October 2020. [78] Sally Jo Cunningham, Zhiyao Duan, Xiao Hu, and Douglas Turnbull, editors. Proceedings of the 18th International Society for Music Information Retrieval Conference, ISMIR 2017, October 2017. [79] Michael Scott Cuthbert and Christopher Ariza. music21 : A Toolkit for Computer-Aided Musicology and Symbolic Music Data. In Downie and Veltkamp [121], pages 637–642. [80] Marco Cuturi and Mathieu Blondel. Soft-dtw: a differentiable loss function for time-series. In Doina Precup and Yee Whye Teh, editors, International conference on machine learning, volume 70 of Proceedings of Machine Learning Research, pages 894–903. PMLR, PMLR, 08 2017. [81] Enrico Daga, Luigi Asprino, Paul Mulholland, and Aldo Gangemi. FacadeX: An Opinionated Approach to SPARQL Anything. In Mehwish Alam, Paul Groth, Victor de Boer, Tassilo Pellegrini, and Harshvardhan J. Pandit, editors, Volume 53: Further with Knowledge Graphs, volume 53, pages 58– 73. IOS Press, August 2021. 250 i i “output” — 2025/10/28 — 13:54 — page 251 — #265 i i i i i i Bibliography [82] David Dalmazzo, Ken Déguernel, and Bob L. T. Sturm. The chordinator: Modeling music harmony by implementing transformer networks and token strategies. In Artificial Intelligence in Music, Sound, Art and Design: 13th International Conference, EvoMUSART 2024, Held as Part of EvoStar 2024, Aberystwyth, UK, April 3–5, 2024, Proceedings, page 52–66, Berlin, Heidelberg, 2024. Springer-Verlag. [83] Roger B. Dannenberg. A Brief Survey of Music Representation Issues, Techniques, and Systems. Computer Music Journal, 17, 11 1993. [84] Roger B. Dannenberg, William P. Birmingham, Bryan Pardo, Ning Hu, Colin Meek, and George Tzanetakis. A comparative evaluation of search techniques for query-by-humming using the MUSART testbed. Journal of the American Society for Information Science and Technology, 58(5):687– 701, 2007. [85] Marilena Daquino and Enrico Daga. MusoW: Music data on the Web. [86] Marilena Daquino, Valentina Pasqual, and Francesca Tomasi. Knowledge Representation of digital Hermeneutics of archival and literary Sources. Knowledge Representation of digital Hermeneutics of archival and literary Sources, pages 59–76, 2020. [87] Marilena Daquino, Valentina Pasqual, Francesca Tomasi, and Fabio Vitali. Expressing Without Asserting in the Arts. In CEUR WORKSHOP PROCEEDINGS, volume 3160, 2022. [88] Jacopo de Berardinis, Valentina Anita Carriero, Nitisha Jain, Nicolas Lazzari, Albert Meroño-Peñuela, Andrea Poltronieri, and Valentina Presutti. The polifonia ontology network: Building a semantic backbone for musical heritage. In Terry R. Payne, Valentina Presutti, Guilin Qi, María Poveda-Villalón, Giorgos Stoilos, Laura Hollink, Zoi Kaoudi, Gong Cheng, and Juanzi Li, editors, The Semantic Web – ISWC 2023, pages 302–322, Cham, 2023. Springer Nature Switzerland. [89] Jacopo de Berardinis, Albert Meroño-Peñuela, Andrea Poltronieri, and Valentina Presutti. The Music Annotation Pattern. In Vojtech Svátek, Valentina Anita Carriero, María Poveda-Villalón, Christian Kindermann, and Lu Zhou, editors, Proceedings of the 13th Workshop on Ontology Design 251 i i “output” — 2025/10/28 — 13:54 — page 258 — #272 i i i i i i Bibliography [146] Takuya Fujishima. Realtime Chord Recognition of Musical Sound: a System Using Common Lisp Music. In Proceedings of the 1999 International Computer Music Conference, ICMC 1999, Beijing, China, October 22-27, 1999. Michigan Publishing, 1999. [147] Aldo Gangemi. Ontology Design Patterns for Semantic Web Content. In Yolanda Gil, Enrico Motta, V. Richard Benjamins, and Mark A. Musen, editors, The Semantic Web - ISWC 2005, 4th International Semantic Web Conference, ISWC 2005, Galway, Ireland, November 6-10, 2005, Proceedings, volume 3729 of Lecture Notes in Computer Science, pages 262–276. Springer, 2005. [148] Aldo Gangemi, Nicola Guarino, Claudio Masolo, Alessandro Oltramari, and Luc Schneider. Sweetening Ontologies with DOLCE. In Asunción Gómez-Pérez and V. Richard Benjamins, editors, Knowledge Engineering and Knowledge Management. Ontologies and the Semantic Web, 13th International Conference (EKAW 2002), pages 166–181, Berlin, Heidelberg, 2002. Springer, Springer Berlin Heidelberg. [149] Aldo Gangemi and Peter Mika. Understanding the Semantic Web through Descriptions and Situations. In Robert Meersman, Zahir Tari, and Douglas C. Schmidt, editors, On The Move to Meaningful Internet Systems 2003: CoopIS, DOA, and ODBASE, pages 689–706, Berlin, Heidelberg, 2003. Springer Berlin Heidelberg. [150] Aldo Gangemi and Silvio Peroni. The Information Realization Pattern. In Pascal Hitzler, Aldo Gangemi, Krzysztof Janowicz, Adila Krisnadhi, and Valentina Presutti, editors, Ontology Engineering with Ontology Design Patterns - Foundations and Applications, volume 25 of Studies on the Semantic Web, pages 299–312. IOS Press, 2016. [151] Aldo Gangemi and Valentina Presutti. Ontology Design Patterns. In Steffen Staab and Rudi Studer, editors, Handbook on Ontologies, International Handbooks on Information Systems, pages 221–243. Springer, Berlin, Heidelberg, 2009. [152] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. Datasheets for datasets. Communications of the ACM, 64(12):86–92, 2021. 258 i i “output” — 2025/10/28 — 13:54 — page 259 — #273 i i i i i i Bibliography [153] Shaghayegh Gharghabi, Yifei Ding, Chin-Chia Michael Yeh, Kaveh Kamgar, Liudmila Ulanova, and Eamonn Keogh. Matrix profile VIII: domain agnostic online semantic segmentation at superhuman performance levels. In 2017 IEEE international conference on data mining (ICDM), pages 117– 126. IEEE, 2017. [154] Deepanway Ghosal and Maheshkumar H Kolekar. Music genre recognition using deep neural networks and transfer learning. In Interspeech, pages 2087–2091, 2018. [155] Konstantinos Giannos and Emilios Cambouropoulos. Symbolic Encoding of Simultaneities: Re-Designing the General Chord Type Representation. In 8th International Conference on Digital Libraries for Musicology, DLfM ’21, page 67–74, New York, NY, USA, 2021. Association for Computing Machinery. [156] Stamatios Giannoulakis, Nicolas Tsapatsoulis, and Nikos Grammalidis. Metadata for intangible cultural heritage. In Proceedings of the 13th international joint conference on computer vision, imaging and computer graphics theory and applications (VISAPP 2018), pages 634–645, 2018. [157] Mathieu Giraud, Ken Déguernel, and Emilios Cambouropoulos. Fragmentations with Pitch, Rhythm and Parallelism Constraints for Variation Matching. In Mitsuko Aramaki, Olivier Derrien, Richard Kronland-Martinet, and Sølvi Ystad, editors, Sound, Music, and Motion, pages 298–312, Cham, 2014. Springer International Publishing. [158] Emilia Gómez, Xiao Hu, Eric Humphrey, and Emmanouil Benetos, editors. Proceedings of the 19th International Society for Music Information Retrieval Conference, ISMIR 2018, September 2018. [159] Michael Good. MusicXML for Notation and Analysis. In The Virtual Score, Volume 12: Representation, Retrieval, Restoration. The MIT Press, 05 2001. [160] Mark RH Gotham. Connecting the Dots: Engaging Wider Forms of Openness for the Mutual Benefit of Musicians and Musicologists. Empirical Musicology Review, 16(1):34–46, 2021. [161] Masataka Goto, Hiroki Hashiguchi, Takuichi Nishimura, and Ryuichi Oka. RWC Music Database: Popular, Classical and Jazz Music Databases. In 259 i i “output” — 2025/10/28 — 13:54 — page 260 — #274 i i i i i i Bibliography ISMIR 2002, 3rd International Symposium on Music Information Retrieval, pages 287–288, October 2002. [162] Maarten Grachten, Josep Lluís Arcos, and Ramón López de Mántaras. Melodic similarity: Looking for a good abstraction level. In ISMIR 2004, 5th International Conference on Music Information Retrieval, October 2004. [163] Mark Granroth-Wilding and Mark Steedman. A robust parser-interpreter for jazz chord sequences. Journal of New Music Research, 43(4):355–374, 2014. [164] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd International Conference on Machine Learning, ICML ’06, page 369–376, New York, NY, USA, 2006. Association for Computing Machinery. [165] George Grove, Stanley Sadie, and K. Marie Stolba. The New Grove Dictionary of Music and Musicians. Macmillan Publishers, London ; Washington, D.C., 1980. [166] Dan Gruen, Thyra Rauch, Sarah Redpath, and Stefan Ruettinger. The use of stories in user experience design. International Journal of HumanComputer Interaction, 14(3-4):503–534, 2002. [167] Michael Gruninger and Maria S. Fox. The role of competency questions in enterprise engineering. In Benchmarking — Theory and Practice. IFIP Advances in Information and Communication Technology, pages 83–95. Springer, Boston, MA, 1994. [168] Christophe Guillotel-Nothmann. Knowledge extraction and modelling in the project Thesaurus Musicarum Germanicarum. In Prague DH Workshops Session III: Editions, 2020. [169] Christophe Guillotel-Nothmann and Anne-Emmanuelle Ceulemans. Das diatonisch-chromatische System zur Zeit des Michael Praetorius. Eine digitale Neuerschließung des Syntagma Musicum (1619) in Verbindung mit dem Tanzzyklus Terpsichore (1612). In Musik im Umbruch. Michael Praetorius zum 400. Todestag, Harrassowitz, 2021. 260 i i “output” — 2025/10/28 — 13:54 — page 261 — #275 i i i i i i Bibliography [170] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. Conformer: Convolution-augmented transformer for speech recognition. In Helen Meng, Bo Xu, and Thomas Fang Zheng, editors, Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020, pages 5036–5040. ISCA, 2020. [171] Zixun Guo, Jaeyong Kang, and Dorien Herremans. A domain-knowledgeinspired music embedding space and a novel attention mechanism for symbolic music modeling. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence, AAAI’23/IAAI’23/EAAI’23. AAAI Press, 2023. [172] Chitralekha Gupta, Rong Tong, Haizhou Li, and Ye Wang. Semi-supervised lyrics and solo-singing alignment. In Gómez et al. [158]. [173] Ferras Hamad, Isaac Liu, and Xian Xing Zhang. Food Discovery with Uber Eats: Building a Query Understanding Engine. Uber Engineering Blog, June 2018. https://eng.uber.com/uber-eats-query-understanding/. [174] Andrew Hankinson, Perry Roland, and Ichiro Fujinaga. The music encoding initiative as a document-encoding framework. In Klapuri and Leider [219]. [175] Pierre Hanna, Matthias Robine, and Thomas Rocher. An alignment based system for chord sequence retrieval. In Proceedings of the 9th ACM/IEEECS joint conference on Digital libraries, pages 101–104, 2009. [176] Nicholas Harley and Geraint Wiggins. An ontology for abstract, hierarchical music representation. In Müller and Wiering [286]. [177] J.P.E. Harper-Scott and J. Samson. An Introduction to Music Studies. Cambridge University Press, 2009. [178] Mickey Hart. Preserving Our Musical Heritage: A Musician’s Outreach to Audio Engineers. Journal of the Audio Engineering Society, 49(7/8):667– 670, 2001. 261 i i “output” — 2025/10/28 — 13:54 — page 262 — #276 i i i i i i Bibliography [179] Christopher Harte. Towards automatic extraction of harmony information from music signals, 2010. [180] Christopher Harte, Mark Sandler, and Martin Gasser. Detecting harmonic change in musical audio. In Proceedings of the 1st ACM Workshop on Audio and Music Computing Multimedia, AMCMM ’06, page 21–26, New York, NY, USA, 2006. Association for Computing Machinery. [181] Christopher Harte, Mark B. Sandler, Samer A. Abdallah, and Emilia Gómez. Symbolic Representation of Musical Chords: A Proposed Syntax for Text Annotations. In ismir2005 [207], pages 66–71. [182] Olaf Hartig. Foundations of RDF* and SPARQL* (An Alternative Approach to Statement-Level Metadata in RDF). In Alberto Mendelzon Workshop on Foundations of Data Management, 2017. [183] Curtis Hawthorne, Ian Simon, Adam Roberts, Neil Zeghidour, Josh Gardner, Ethan Manilow, and Jesse H. Engel. Multi-instrument Music Synthesis with Spectrogram Diffusion. In Rao et al. [330], pages 598–607. [184] Bruce Haynes and Peter Cooke. Pitch, 2001. [185] Qi He, Bee-Chung Chen, and Deepak Agarwal. Building The LinkedIn Knowledge Graph. LinkedIn Blog, October 2016. https://engineering.linkedin.com/blog/2016/10/ building-the-linkedin-knowledge-graph. [186] James A. Hendler, Jeanne Holm, Chris Musialek, and George Thomas. US Government Linked Open Data: Semantic.data.gov. IEEE Intelligent Systems, 27(3):25–31, 2012. [187] Johannes Hentschel, Fabian C. Moss, Andrew McLeod, Markus Neuwirth, and Martin Rohrmeier. Towards a Unified Model of Chords in Western Harmony. In Stefan Münnich and David Rizo, editors, Music Encoding Conference Proceedings 2021, pages 143–149. Humanities Commons, 2022. [188] Johannes Hentschel, Markus Neuwirth, and Martin Rohrmeier. The annotated Mozart Sonatas: Score, Harmony, and Cadence. Transactions of the International Society for Music Information Retrieval, 4(1), 2021. 262 i i “output” — 2025/10/28 — 13:54 — page 263 — #277 i i i i i i Bibliography [189] Paul Hindemith and Arthur Mendel. The Craft of Musical Composition. 1. Theoretical Part. Schott, 1970. [190] Keiji Hirata, George Tzanetakis, and Kazuyoshi Yoshii, editors. Proceedings of the 10th International Society for Music Information Retrieval Conference, ISMIR 2009. International Society for Music Information Retrieval, October 2009. [191] Thomas S. Hischak and Dai Griffiths. Lyrics, 2001. [192] Pascal Hitzler, Aldo Gangemi, Krzysztof Janowicz, Adila Krisnadhi, and Valentina Presutti, editors. Ontology Engineering with Ontology Design Patterns - Foundations and Applications, volume 25 of Studies on the Semantic Web. IOS Press, Amsterdam, 2016. [193] Pascal Hitzler, Markus Krötzsch, Bijan Parsia, Peter F. Patel-Schneider, and Sebastian Rudolph. OWL 2 Web Ontology Language Primer (Second Edition), W3C Recommendation 11 December 2012. W3c recommendation, World Wide Web Consortium, dec 2012. [194] Aidan Hogan, Eva Blomqvist, Michael Cochez, Claudia d’Amato, Gerard de Melo, Claudio Gutierrez, Sabrina Kirrane, José Emilio Labra Gayo, Roberto Navigli, Sebastian Neumaier, et al. Knowledge graphs. Synthesis Lectures on Data, Semantics, and Knowledge, 12(2):1–257, 2021. [195] Holger Hoos, Keith A. Hamel, Kai Renz, and Jürgen Kilian. Representing score-level music using the GUIDO music-notation format. Computing in Musicology, 12, 2001. [196] Holger H Hoos, Keith Hamel, Kai Renz, and Jürgen Kilian. The guido notation format: A novel approach for adequately representing score-level music. In ICMC, volume 98, pages 451–454, 1998. [197] Cheng-Zhi Anna Huang, Tim Cooijmans, Adam Roberts, Aaron C. Courville, and Douglas Eck. Counterpoint by Convolution. In Flexer et al. [139], pages 211–218. [198] Cheng-Zhi Anna Huang, Hendrik Vincent Koops, Ed Newton-Rex, Monica Dinculescu, and Carrie J Cai. AI song contest: Human-AI co-creation in songwriting. In Cumming et al. [77]. 263 i i “output” — 2025/10/28 — 13:54 — page 264 — #278 i i i i i i Bibliography [199] Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, and Douglas Eck. Music Transformer: Generating Music with Long-Term Structure. In International Conference on Learning Representations, 2019. [200] Jiawen Huang, Emmanouil Benetos, and Sebastian Ewert. Improving lyrics alignment through joint pitch detection, 2022. [201] Yin-Fu Huang, Sheng-Min Lin, Huan-Yu Wu, and Yu-Siou Li. Music genre classification based on local feature selection using a self-adaptive harmony search algorithm. Data Knowl. Eng., 92:60–76, 2014. [202] Eric J. Humphrey, Taemin Cho, and Juan Pablo Bello. Learning a robust Tonnetz-space transform for automatic chord recognition. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2012, Kyoto, Japan, March 25-30, 2012, pages 453–456. IEEE, 2012. [203] Eric J. Humphrey, Justin Salamon, Oriol Nieto, Jon Forsyth, Rachel M. Bittner, and Juan P. Bello. JAMS: A JSON annotated music specification for reproducible MIR research. In Wang et al. [398], pages 591–596. [204] David Huron. Music information processing using the humdrum toolkit: Concepts, examples, and lessons. Computer Music Journal, 26(2):11–26, 2002. [205] Brian Hyer. Tonality, 2001. [206] Antoine Isaac and Bernhard Haslhofer. Europeana linked open data–data. europeana. eu. Semantic Web, 4(3):291–297, 2013. [207] ISMIR 2005, 6th International Conference on Music Information Retrieval, September 2005. [208] Kurt Jacobson, Yves Raimond, and Mark B. Sandler. An Ecosystem for Transparent Music Similarity in an Open World. In Hirata et al. [190], pages 33–38. [209] Alexander Refsum Jensenius. Best versus good enough practices for open music research. Empirical Musicology Review, 16(1):5–15, 2021. 264 i i “output” — 2025/10/28 — 13:54 — page 265 — #279 i i i i i i Bibliography [210] Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and Philip S. Yu. A survey on knowledge graphs: Representation, acquisition, and applications. IEEE Transactions on Neural Networks and Learning Systems, 33(2):494–514, 2022. [211] Junyan Jiang, Ke Chen, Wei Li, and Gus Xia. Large-vocabulary Chord Transcription Via Chord Structure Decomposition. In Flexer et al. [139], pages 644–651. [212] Cyril Joder, Slim Essid, and Gaël Richard. Learning optimal features for polyphonic audio-to-score alignment. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 21(10):2118–2128, 2013. [213] Jim Jones, Diego de Siqueira Braga, Kleber Tertuliano, and Tomi Kauppinen. MusicOWL: The Music Score Ontology. In Proceedings of the International Conference on Web Intelligence, WI ’17, pages 1222–1229, New York, NY, USA, 2017. Association for Computing Machinery. [214] Spyridon Kantarelis, Edmund Dervakos, Natalia Kotsani, and Giorgos Stamou. Functional Harmony Ontology: Musical Harmony Analysis with Description Logics. Web Semant., 75(C), January 2023. [215] Emmanouil Karystinaios and Gerhard Widmer. Cadence detection in symbolic classical music using graph neural networks. In Rao et al. [330]. [216] Hyon Hee Kim. A Semantically Enhanced Tag-Based Music Recommendation Using Emotion Ontology. In Intelligent Information and Database Systems, Berlin, Heidelberg, 2013. Springer Berlin Heidelberg. [217] Jaehun Kim, Julián Urbano, Cynthia Liem, and Alan Hanjalic. One Deep Music Representation to Rule Them All?: A Comparative Analysis of Different Representation Learning Strategies. Neural Computing and Applications, 32(4):1067–1093, 2020. [218] Arto Klami, Theo Damoulas, Ola Engkvist, Patrick Rinke, and Samuel Kaski. Virtual laboratories: transforming research with ai. Data-Centric Engineering, 5:e19, 2024. [219] Anssi Klapuri and Colby Leider, editors. Proceedings of the 12th International Society for Music Information Retrieval Conference, ISMIR 2011. University of Miami, October 2011. 265 i i “output” — 2025/10/28 — 13:54 — page 266 — #280 i i i i i i Bibliography [220] Peter Knees, Ángel Faraldo Pérez, Herrera Boyer, Richard Vogl, Sebastian Böck, Florian Hörschläger, Mickael Le Goff, et al. Two data sets for tempo estimation and key detection in electronic dance music annotated from user corrections. In Müller and Wiering [286], pages 364–370. [221] Peter Knees and Markus Schedl. Music similarity and retrieval. In Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’13, page 1125, New York, NY, USA, 2013. Association for Computing Machinery. [222] Peter Knees and Markus Schedl. A survey of music similarity and recommendation from music context data. ACM Trans. Multimedia Comput. Commun. Appl., 10(1), dec 2013. [223] Stefan Koelsch. Toward a neural basis of music perception–a review and updated model. Frontier in Psychology, 2:110, 2011. [224] Sefki Kolozali, Mathieu Barthet, György Fazekas, and Mark B Sandler. Knowledge Representation Issues in Musical Instrument Ontology Design. In Klapuri and Leider [219], pages 465–470. [225] Hendrik Vincent Koops, Bas de Haas, John Ashley Burgoyne, Jeroen Bransen, Anna Kent-Muller, and Anja Volk. Annotator subjectivity in harmony annotations of popular music. Journal of New Music Research, 48(3):232–252, 2019. [226] Hendrik Vincent Koops, W. Bas de Haas, Jeroen Bransen, and Anja Volk. Automatic chord label personalization through deep learning of shared harmonic interval profiles. Neural Comput. Appl., 32(4):929–939, February 2020. [227] Hendrik Vincent Koops, W. Bas de Haas, John Ashley Burgoyne, Jeroen Bransen, Anna Kent-Muller, and Anja Volk. Annotator subjectivity in harmony annotations of popular music. Journal of New Music Research, 48(3):232–252, may 2019. [228] Filip Korzeniowski and Gerhard Widmer. A fully convolutional deep auditory model for musical chord recognition. In Francesco A. N. Palmieri, Aurelio Uncini, Kostas I. Diamantaras, and Jan Larsen, editors, 26th IEEE International Workshop on Machine Learning for Signal Processing, MLSP 266 i i “output” — 2025/10/28 — 13:54 — page 267 — #281 i i i i i i Bibliography 2016, Vietri sul Mare, Salerno, Italy, September 13-16, 2016, pages 1–6. IEEE, 2016. [229] Harald Kosch. MPEG-7 and multimedia database systems. SIGMOD Rec., 31(2):34–39, June 2002. [230] S.M. Kostka and D. Payne. Tonal Harmony, with an Introduction to Twentieth-century Music. McGraw-Hill, 2004. [231] Dominik Kowald, Markus Schedl, and Elisabeth Lex. The Unfairness of Popularity Bias in Music Recommendation: A Reproducibility Study. In Joemon M. Jose, Emine Yilmaz, João Magalhães, Pablo Castells, Nicola Ferro, Mário J. Silva, and Flávio Martins, editors, Advances in Information Retrieval, pages 35–42, Cham, 2020. Springer International Publishing. [232] Michael Krause, Christof Weiß, and Meinard Müller. Soft dynamic time warping for multi-pitch estimation and beyond. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023. [233] Arun Krishnan. Making search easier: How Amazon’s Product Graph is helping customers find products more easily. Amazon Blog, August 2018. https://blog.aboutamazon.com/innovation/making-search-easier. [234] Allison Lahnala, Gauri Kambhatla, Jiajun Peng, Matthew Whitehead, Gillian Minnehan, Eric Guldan, Jonathan K. Kummerfeld, Anıl Çamcı, and Rada Mihalcea. Chord Embeddings: Analyzing What They Capture and Their Role for Next Chord Prediction and Artist Attribute Prediction. In Juan Romero, Tiago Martins, and Nereida Rodríguez-Fernández, editors, Artificial Intelligence in Music, Sound, Art and Design, volume 12693 of Lecture Notes in Computer Science, pages 171–186, Cham, 2021. Springer International Publishing. [235] S.G. Laitz. The Complete Musician: An Integrated Approach to Tonal Theory, Analysis, and Listening. Number v. 1 in The Complete Musician: An Integrated Approach to Tonal Theory, Analysis, and Listening. Oxford University Press, 2008. [236] Agnieszka Lawrynowicz. Creative AI: A new avenue for the Semantic Web? Semantic Web, 11:69–78, 2020. 267 i i “output” — 2025/10/28 — 13:54 — page 274 — #288 i i i i i i Bibliography [291] Oriol Nieto and Juan Pablo Bello. Systematic Exploration of Computational Music Structure Research. In Mandel et al. [254], pages 547–553. [292] Nottingham Database. https://ifdo.ca/~seymour/nottingham/ nottingham.html. [293] Natasha F. Noy, Yuqing Gao, Anshu Jain, Anant Narayanan, Alan Patterson, and Jamie Taylor. Industry-scale Knowledge Graphs: Lessons and Challenges. ACM Queue, 17(2):20, 2019. [294] Nicola Orio and Antonio Rodà. A Measure of Melodic Similarity based on a Graph Representation of the Music Structure. In Hirata et al. [190], pages 543–548. [295] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA, 2024. Curran Associates Inc. [296] Francois Pachet. Knowledge management and musical metadata. Idea Group, 12, 2005. [297] Elias Pampalk. Computational models of music similarity and their application in music information retrieval. PhD thesis, Technische Universität Wien, 2006. [298] Letitia Parcalabescu, Nils Trost, and Anette Frank. What is Multimodality? In Lucia Donatelli, Nikhil Krishnaswamy, Kenneth Lai, and James Pustejovsky, editors, Proceedings of the 1st Workshop on Multimodal Semantic Representations (MMSR), pages 1–10, Groningen, Netherlands (Online), June 2021. Association for Computational Linguistics. [299] Ferran Parés, Dario Garcia Gasulla, Armand Vilalta, Jonatan Moreno, Eduard Ayguadé, Jesús Labarta, Ulises Cortés, and Toyotaro Suzumura. Fluid communities: A competitive, scalable and diverse community detection al274 i i “output” — 2025/10/28 — 13:54 — page 275 — #289 i i i i i i Bibliography gorithm. In International Conference on Complex Networks and their Applications, pages 229–240. Springer, 2017. [300] Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le. Specaugment: A simple data augmentation method for automatic speech recognition. In Interspeech 2019. ISCA, September 2019. [301] Jonggwon Park, Kyoyun Choi, Sungwook Jeon, Dokyun Kim, and Jonghun Park. A Bi-Directional Transformer for Musical Chord Recognition. In Flexer et al. [139], pages 620–627. [302] Johan Pauwels, Ken O’Hanlon, Emilia Gómez, and Mark B. Sandler. 20 years of automatic chord recognition from audio. In Flexer et al. [139], pages 54–63. [303] D. Pedler. The Songwriting Secrets Of The Beatles. Music Sales, 2010. [304] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global Vectors for Word Representation. In Alessandro Moschitti, Bo Pang, and Walter Daelemans, editors, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 2529, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1532–1543. ACL, 2014. [305] Carlos Pérez-Sancho, David Rizo, José M Iñesta, Pedro J Ponce De León, Stefan Kersten, and Rafael Ramirez. Genre classification of music by tonal harmony. Intelligent Data Analysis, 14(5):533–545, 2010. [306] Silvio Peroni. A simplified agile methodology for ontology development. In OWL: Experiences and Directions–Reasoner Evaluation, pages 55–69. Springer, 2016. [307] Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep Contextualized Word Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, volume 1, pages 2227–2237, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. 275 i i “output” — 2025/10/28 — 13:54 — page 276 — #290 i i i i i i Bibliography [308] Bastian Pfleging, Stefan Schneegaß, and Albrecht Schmidt. Multimodal interaction in the car: combining speech and gestures on the steering wheel. In AutomotiveUI, pages 155–162. ACM, 2012. [309] Martin Pfleiderer, Klaus Frieler, Jakob Abeßer, Wolf-Georg Zaddach, and Benjamin Burkhart, editors. Inside the Jazzomat - New Perspectives for Jazz Research. Schott Campus, 2017. [310] Azzurra Pini, Jer Hayes, Connor Upton, and Medb Corcoran. AI Inspired Recipes: Designing Computationally Creative Food Combos. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, CHI EA ’19, page 1–6, New York, NY, USA, 2019. Association for Computing Machinery. [311] Helena Sofia Pinto, Steffen Staab, and Christoph Tempich. DILIGENT: Towards a fine-grained methodology for DIstributed, Loosely-controlled and evolvInG Engineering of oNTologies. In ECAI, volume 16, page 393. Citeseer, 2004. [312] W. Piston. Harmony. W. W. Norton, 1962. [313] C.J. Plack, A.J. Oxenham, and R.R. Fay. Pitch: Neural Coding and Perception. Online access: LexisNexis Nexis Advance UK. Springer, 2005. [314] Andrea Poltronieri and Aldo Gangemi. The HaMSE ontology: Using semantic technologies to support music representation interoperability and musicological analysis. In Proceedings of the 1st workshop on multisensory data and knowledge (MDK), 2021. [315] Andrea Poltronieri and Aldo Gangemi. The Music Note Ontology. In Karl Hammar, Cogan Shimizu, Hande Küçük McGinty, Luigi Asprino, and Valentina Anita Carriero, editors, Proceedings of the 12th Workshop on Ontology Design and Patterns (WOP 2021), Online, October 24, 2021., November 2021. [316] Andrea Poltronieri, Valentina Presutti, and Martín Rocamora. ChordSync: A Conformer-based Audio-to-Chord Synchroniser. In Proceedings of the 2024 Sound and Music Computing Conference. Sound and Music Computing Network, July 2024. 276 i i “output” — 2025/10/28 — 13:54 — page 277 — #291 i i i i i i Bibliography [317] Andrea Poltronieri, Xavier Serra, and Martín Rocamora. From discord to harmony: Decomposed consonance-based training for improved audio chord estimation. In Proceedings of the 26th International Society for Music Information Retrieval Conference (ISMIR). International Society for Music Information Retrieval, 2025. [318] Valentina Presutti, Enrico Daga, Aldo Gangemi, and Eva Blomqvist. eXtreme Design with Content Ontology Design Patterns. In Eva Blomqvist, Kurt Sandkuhl, François Scharffe, and Vojtech Svátek, editors, Proceedings of the Workshop on Ontology Patterns (WOP 2009), volume 516 of CEUR Workshop Proceedings. CEUR-WS.org, 2009. [319] Thomas Prätzlich, Jonathan Driedger, and Meinard Müller. Memoryrestricted multiscale dynamic time warping. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 569–573, 2016. [320] Michael Pulis and Josef Bajada. Siamese neural networks for content-based cold-start music recommendation. In Proceedings of the 15th ACM Conference on Recommender Systems, RecSys ’21, page 719–723, New York, NY, USA, 2021. Association for Computing Machinery. [321] Alexander M Putman and Robert M Keller. A transformational grammar framework for improvisation. In First International Conference on New Music Concepts, 2015. [322] Colin Raffel and Daniel P. W. Ellis. Optimizing dtw-based audio-to-midi alignment and matching. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 81–85, 2016. [323] Colin Raffel, Brian McFee, Eric J Humphrey, Justin Salamon, Oriol Nieto, Dawen Liang, Daniel PW Ellis, and C Colin Raffel. MIR-EVAL: A Transparent Implementation of Common MIR Metrics. In Wang et al. [398], pages 367–372. [324] Yves Raimond, Samer Abdallah, Mark Sandler, and Frederick Giasson. The Music Ontology. In Dixon et al. [118], pages 417–422. [325] Yves Raimond, Tristan Ferne, Michael Smethurst, and Gareth Adams. The 277 i i “output” — 2025/10/28 — 13:54 — page 278 — #292 i i i i i i Bibliography BBC World Service Archive prototype. Journal of Web Semantics, 27–28:2– 9, 2014. [326] Yves Raimond and Mark Sandler. Evaluation of the music ontology framework. In The Semantic Web: Research and Applications: 9th Extended Semantic Web Conference, ESWC 2012, Heraklion, Crete, Greece, May 27-31, 2012. Proceedings 9. Springer, 2012. [327] Dhanesh Ramachandram and Graham W. Taylor. Deep Multimodal Learning: A Survey on Recent Advances and Trends. IEEE Signal Processing Magazine, 34(6):96–108, 2017. [328] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821–8831. PMLR, 2021. [329] Pedro Ramoneda Franco and Gilberto Bernardes de Almeida. Revisiting harmonic change detection. In Audio Engineering Society Convention, volume 149, oct 2020. [330] Preeti Rao, Hema A. Murthy, Ajay Srinivasamurthy, Rachel M. Bittner, Rafael Caro Repetto, Masataka Goto, Xavier Serra, and Marius Miron, editors. Proceedings of the 23rd International Society for Music Information Retrieval Conference, ISMIR 2022, December 2022. [331] Christopher Raphael and Joshua Stoddard. Functional harmonic analysis using probabilistic models. Computer Music Journal, 28(3):45–52, 2004. [332] Sabbir M. Rashid, David De Roure, and Deborah L. McGuinness. A Music Theory Ontology. In Proceedings of the 1st International Workshop on Semantic Applications for Audio and Music, SAAM ’18, page 6–14, New York, NY, USA, 2018. Association for Computing Machinery. [333] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, EMNLP/IJCNLP, pages 3980–3990. Association for Computational Linguistics, 2019. [334] Seungmin Rho, Seheon Song, Eenjun Hwang, and Minkoo Kim. COMUS: Ontological and Rule-Based Reasoning for Music Recommendation System. 278 i i “output” — 2025/10/28 — 13:54 — page 279 — #293 i i i i i i Bibliography In Advances in Knowledge Discovery and Data Mining, Berlin, Heidelberg, 2009. Springer Berlin Heidelberg. [335] Jenn Riley. Application of the Functional Requirements for Bibliographic Records (FRBR) to Music. In Bello et al. [20], pages 439–444. [336] Adam Roberts, Jesse Engel, Colin Raffel, Curtis Hawthorne, and Douglas Eck. A hierarchical latent vector model for learning long-term structure in music. In International conference on machine learning, pages 4364–4373. PMLR, 2018. [337] Miguel Ángel Rodríguez-García, Luis Omar Colombo-Mendoza, Rafael Valencia-García, Antonio A Lopez-Lorca, and Ghassan Beydoun. Ontologybased music recommender system. In Distributed Computing and Artificial Intelligence, 12th International Conference, pages 39–46. Springer, 2015. [338] Francisco Jose Rodriguez-Serrano, Julio Jose Carabias-Orti, Pedro VeraCandeas, and Damian Martinez-Munoz. Tempo driven audio-to-score alignment using spectral decomposition and online dynamic time warping. ACM Trans. Intell. Syst. Technol., 8(2), oct 2016. [339] John Roeder. Pitch class, 2001. [340] Carles Roig, Lorenzo J Tardón, Ana M Barbancho, and Isabel Barbancho. Submission to MIREX: symbolic melodic similarity task, November 2013. [341] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022. [342] J. Rothstein. MIDI: A Comprehensive Introduction. Computer music and digital audio series. A-R Editions, 1992. [343] Luke O. Rowe and George Tzanetakis. Curriculum learning for imbalanced classification in large vocabulary automatic chord recognition. In Lee et al. [239], pages 586–593. [344] Klaus-Jürgen Sachs and Carl Dahlhaus. Counterpoint, 06 2023. 279 i i “output” — 2025/10/28 — 13:54 — page 280 — #294 i i i i i i Bibliography [345] Justin Salamon, Joan Serrà, and Emilia Gómez. Melody, bass line, and harmony representations for music version identification. In Proceedings of the 21st International Conference on World Wide Web, pages 887–894, 2012. [346] Amaia Salvador, Michal Drozdzal, Xavier Giró-i Nieto, and Adriana Romero. Inverse cooking: Recipe generation from food images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10453–10462, 2019. [347] Emilio M. Sanfilippo and Richard Freedman. Ontology for Analytic Claims in Music. In Silvia Chiusano, Tania Cerquitelli, Robert Wrembel, Kjetil Nørvåg, Barbara Catania, Genoveva Vargas-Solar, and Ester Zumpano, editors, New Trends in Database and Information Systems, pages 559–571, Cham, 2022. Springer International Publishing. [348] KC Santosh, Bart Lamiroy, and Laurent Wendling. DTW–Radon-based shape descriptor for pattern recognition. International Journal of Pattern Recognition and Artificial Intelligence, 27(03):1350008, 2013. [349] Augusto Sarti, Fabio Antonacci, Mark Sandler, Paolo Bestagini, Simon Dixon, Beici Liang, Gaël Richard, and Johan Pauwels, editors. Proceedings of the 24th International Society for Music Information Retrieval Conference, ISMIR 2023, November 2023. [350] Edward W. Schneider. Course Modularization Applied: The Interface System and Its Implications For Sequence Control and Data Analysis. In Association for the Development of Instructional Systems (ADIS), Chicago, Illinois, April 1972, 1973. [351] A. Schoenberg, R.E. Carter, and W. Frisch. Theory of Harmony. University of California, 2010. [352] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Katrin Erk and Noah A. Smith, editors, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, volume 1, pages 1715–1725, Berlin, Germany, August 2016. Association for Computational Linguistics. [353] Abdul Shahid, Danny Diamond, and James McDermott. Patterns2KG: JAMS Pipeline for Modeling Music Patterns. In Antonis Bikakis, Roberta 280 i i “output” — 2025/10/28 — 13:54 — page 281 — #295 i i i i i i Bibliography Ferrario, Stéphane Jean, Béatrice Markhoff, Alessandro Mosca, and Marianna Nicolosi Asmundo, editors, Proceedings of the International Workshop on Semantic Web and Ontology Design for Cultural Heritage co-located with the International Semantic Web Conference 2023 (ISWC 2023), Athens, Greece, November 7, 2023, volume 3540 of CEUR Workshop Proceedings. CEUR-WS.org, 2023. [354] Bidisha Sharma, Chitralekha Gupta, Haizhou Li, and Ye Wang. Automatic lyrics-to-audio alignment on polyphonic music using singing-adapted acoustic models. In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 396–400, 2019. [355] Mohamadreza Sheikh Fathollahi and Farbod Razzazi. Music similarity measurement and recommendation system using convolutional neural networks. International Journal of Multimedia Information Retrieval, 10:43–53, 2021. [356] Siddharth Sigtia, Nicolas Boulanger-Lewandowski, and Simon Dixon. Audio Chord Recognition with a Hybrid Recurrent Neural Network. In Müller and Wiering [286], pages 127–133. [357] Ian Simon, Dan Morris, and Sumit Basu. MySong: automatic accompaniment generation for vocal melodies. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 725–734, 2008. [358] Federico Simonetta, Stavros Ntalampiras, and Federico Avanzini. Multimodal Music Information Processing and Retrieval: Survey and Future Challenges. In 2019 International Workshop on Multilayer Music Representation and Processing (MMRP), pages 10–18, 2019. [359] Federico Simonetta, Stavros Ntalampiras, and Federico Avanzini. Audio-toscore alignment using deep automatic music transcription. In 2021 IEEE 23rd International Workshop on Multimedia Signal Processing (MMSP), pages 1–6, 2021. [360] Elena Simperl and Markus Luczak-Rösch. Collaborative ontology engineering: a survey. The Knowledge Engineering Review, 29(1):101–131, 2014. [361] Jagendra Singh, Mohammad Sajid, Chandra Shekhar Yadav, Shashank Sheshar Singh, and Manthan Saini. A Novel Deep Neural-based Music Recommendation Method considering User and Song Data. In 2022 6th Interna281 i i “output” — 2025/10/28 — 13:54 — page 282 — #296 i i i i i i Bibliography tional Conference on Trends in Electronics and Informatics (ICOEI), pages 1–7, 2022. [362] Amit Singhal. Introducing the Knowledge Graph: things, not strings. Google Blog, May 2012. https://www.blog.google/products/search/ introducing-knowledge-graph-things-not/. [363] Camila Sitonio and Alberto Nucciarelli. The impact of blockchain on the music industry. Calgary: International Telecommunications Society (ITS), 2018. [364] William C. Sleeman, Rishabh Kapoor, and Preetam Ghosh. Multimodal Classification: Current Landscape, Taxonomy and Future Directions. ACM Comput. Surv., 55(7), December 2022. [365] Daniel Smilkov, Nikhil Thorat, Charles Nicholson, Emily Reif, Fernanda B Viégas, and Martin Wattenberg. Embedding projector: Interactive visualization and interpretation of embeddings. arXiv preprint arXiv:1611.05469, 2016. [366] Temple F Smith, Michael S Waterman, et al. Identification of common molecular subsequences. Journal of molecular biology, 147(1):195–197, 1981. [367] Daniel Stoller, Simon Durand, and Sebastian Ewert. End-to-end lyrics alignment for polyphonic music using an audio-to-character recognition model, 2019. [368] J.N. Straus. Elements of Music. Oxford University Press, 2021. [369] Bob Sturm, Maria Iglesias, Oded Ben-Tal, Marius Miron, and Emilia Gómez. Artificial intelligence and music: open questions of copyright law and engineering praxis. In Arts, volume 8, page 115. MDPI, 2019. [370] Bob Sturm, Joao Felipe Santos, and Iryna Korshunova. Folk music style modelling by recurrent neural networks with long short term memory units. In Müller and Wiering [286]. [371] Mari Carmen Suárez-Figueroa, Asunción Gómez-Pérez, and Mariano Fernández-López. The NeOn methodology for ontology engineering. In Ontology engineering in a networked world, pages 9–34. Springer, 2012. 282 i i “output” — 2025/10/28 — 13:54 — page 283 — #297 i i i i i i Bibliography [372] Iiris Sundin, Alexey Voronov, Haoping Xiao, Kostas Papadopoulos, Esben Jannik Bjerrum, Markus Heinonen, Atanas Patronov, Samuel Kaski, and Ola Engkvist. Human-in-the-loop assisted de novo molecular design. Journal of Cheminformatics, 14(1):1–16, 2022. [373] Christopher Sutton, Yves Raimond, and Matthias Mauch. The OMRAS2 Chord Ontology. http://purl.org/ontology/chord/, 2007. [374] Iman SH Suyoto and Alexandra L Uitdenbogerd. Simple orthogonal pitch with ioi symbolic music matching. In Downie and Veltkamp [121]. [375] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the Inception Architecture for Computer Vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 2818–2826. IEEE Computer Society, 2016. [376] Kimmy Szeto. Ontology for Voice, Instruments, and Ensembles (OnVIE): Revisiting the Medium of Performance Concept for Enhanced Discoverability. Code4Lib Journal, 54, 2022. [377] Nazif Can Tamer, Yigitcan Özer, Meinard Müller, and Xavier Serra. Highresolution violin transcription using weak labels. In Sarti et al. [349]. [378] David Temperley. Kostka-Payne corpus. http://davidtemperley.com/ kp-stats/. [379] David Temperley. The Cognition of Basic Musical Structures. The MIT Press Series. MIT Press, 2001. [380] Nicholas Temperley. The Problem of Definitive Identification in the Indexing of Hymn Tunes. Music Reference Services Quarterly, 2(3-4):227–239, 1993. [381] Florian Thalmann, Alfonso Perez Carrillo, György Fazekas, Geraint A Wiggins, and Mark Sandler. The mobile audio ontology: Experiencing dynamic music objects on mobile devices. In 2016 IEEE Tenth International Conference on Semantic Computing (ICSC), pages 47–54. IEEE, 2016. [382] The Real Book, 2004. [383] Matthias Thiemel. Dynamics, 2001. 283