LyricSIM: A novel dataset and benchmark for similarity detection in Spanish song lyrics

LyricSIM: A novel dataset and benchmark for similarity detection in Spanish song lyrics LyricSIM: Un nuevo dataset y benchmark para la detecci´on de similitud en letras de canciones en espa˜nol Alejandro Benito-Santos,1Adri´an Ghajari,2Pedro Hern´andez,3V´ıctor Fresno,2 Salvador Ros,2Elena Gonz´alez-Blanco3 1Universidad de Salamanca 2Universidad Nacional de Educacion a Distancia 3IE University ab[email protected] Abstract: In this paper, we present a new dataset and benchmark tailored to the task of semantic similarity in song lyrics. Our dataset, originally consisting of 2775 pairs of Spanish songs, was annotated in a collective annotation experiment by 63 native annotators. After collecting and refining the data to ensure a high degree of consensus and data integrity, we obtained 676 high-quality annotated pairs that were used to evaluate the performance of various state-of-the-art monolingual and multilingual language models. Consequently, we established baseline results that we hope will be useful to the community in all future academic and industrial applications conducted in this context. Keywords: semantic textual similarity, annotation task, dataset, benchmark, cultural heritage, song lyrics. Resumen: En este trabajo presentamos un nuevo conjunto de datos y benchmark orientados a la tarea de similitud sem´antica en letras de canciones. Nuestro conjunto de datos, originalmente formado por 2775 pares de canciones en Espa˜nol, fue anotado en un experimento de anotaci´on colectivo por 63 anotadores nativos. Despu´es de recoger y refinar los datos para asegurar un alto grado de consenso e integridad en los datos, obtuvimos 676 pares anotados de alta calidad que fueron empleados para evaluar el rendimiento de diferentes modelos del lenguaje monoling¨ues y multiling¨ues pertenecientes al estado del arte. En consecuencia, obtuvimos unos resultados base que esperamos sean de utilidad a la comunidad en todas aquellas aplicaciones acad´emicas e industriales futuras que se realicen en este contexto. Palabras clave: similitud sem´antica en textos, tarea de anotaci´on, conjunto de datos, letras de canciones. 1 Introduction The success of music streaming services is mainly based on the tailor-made playlist they offer to users based on their listening habits and 54% of consumers say playlists are replacing albums in their listening habits1. To build those lists and fit them to the user’s preferences, music streaming providers have developed recommender systems, which provide personalized suggestions based on the user´s behavior and some specific parameters related to the musicality of a song (Fell, 2020), oftentimes overlooking the song lyrics. A song is comprised of two distinct com1https://midiaresearch.com/blog/musicsubscriber-market-shares-2022 ponents: the music and the lyrics. The lyrics consist of text, typically presented in poetic form, and describe the artist‘s message and embody a unique combination of linguistic, artistic, and cultural elements, making them a special expression of writing that contains different features that could help us to improve song recommender systems. The lyrics thus provide a rich source of unstructured data and qualitative information that is not captured by metadata traditionally used in song recommendation such as beat, tempo, pitch, instrument, or mood (acoustic). Due to their unique writing style, the song lyrics present distinct challenges for semantic similarity modeling, a task that has traditionally been evaluated in the SemEVal tasks Procesamiento del Lenguaje Natural, Revista nº 71, septiembre de 2023, pp. 149-163 recibido 31-03-2023 revisado 08-05-2023 aceptado 12-05-2023 ISSN 1135-5948 DOI 10.26342/2023-71-12 ©2023 Sociedad Española para el Procesamiento del Lenguaje Natural that were organized between 2012 and 2017 (Agirre et al., 2012; Agirre et al., 2013; Agirre et al., 2014; Agirre et al., 2015). This evaluation framework operates under the assumption that a model that performs well for the general STS task is also likely to perform well for tasks that are tailored towards specific applications. However, there has been a significant discrepancy between the performance of models in STS and their performance in specific STS-based tasks such as MT Metrics (MTM) or Passage Retrieval (PR) (Abe et al., 2022). We argue existing STS-based evaluations2may not effectively capture their nuances and specificities, leading to suboptimal performance in lyric-related tasks such as recommendation, search, and cultural analysis, which hinders research and development efforts in this area. In addition, there is currently a gap in the availability of benchmark datasets that are specific to the domain of song lyrics (Chandrasekaran and Mago, 2022). Therefore, a dataset of song lyrics annotated for similarity would not only facilitate the development of more accurate and applicable semantic similarity models but also enable a deeper understanding of the relationships between lyrics and music in this rich and diverse linguistic context. Since Spanish as a language contains a rich cultural and linguistic diversity and the Latin music industry represents a significant portion of the global music market, with a vast array of genres and styles, ranging from traditional folk music to contemporary pop, rock, or urban styles, the lyrics songs in Spanish offer unique challenges and opportunities for the exploration of semantic similarity in the domain of music. In addition, the development of a dataset comprising Spanish song lyrics annotated for similarity tailored to the Spanish language and culture is crucial to advance the understanding and modeling of semantic similarity in the context of music and lyrics within the Spanish-speaking world. In this paper, we introduce LyricSIM, a novel dataset for Spanish song lyric similarity designed to address the domain-specific characteristics of lyrics and facilitate the development of more accurate and applicable semantic similarity models for this domain. Our dataset comprises a diverse collection 2http://nlpprogress.com/english/semantic_ textual_similarity.html of paired song lyrics in Spanish, annotated with similarity scores based on various aspects such as theme, message, emotions, literal meaning, and cultural context. On the other hand, we also have assessed the performance of various state-of-the-art models on our dataset for the semantic similarity task taking into account the unique features of song lyrics. The obtained results make up a new benchmark for the semantic similarity tasks based on the lyrics of songs. By developing a dataset specifically tailored to song lyric similarity and assessing the performance of SOTA models over this dataset, this paper aims to bridge the gap between general-purpose semantic similarity tasks and domain-specific applications, ultimately contributing to the advancement of NLP research in the context of music and lyrics research and analysis. 2 Related Work To the best of our knowledge, no benchmark datasets currently exist for the study of semantic similarity between song lyrics. Although datasets containing song lyrics are available in different languages, they do not include semantic similarity annotations between song pairs, only collecting the lyrics. This information makes our dataset a very good fit to design models aimed at identifying plagiarism or detecting unlicensed versions of copyrighted songs, among others. The benchmark dataset we can consider most closely related to ours is the 4MuLA dataset introduced in (da Silva, Silva, and Marcacini, 2020), which contains structured information that can be applied to several tasks. This dataset is obtained from a lyrics-focused platform and includes additional user-provided annotations. It includes Latin music genres that are underrepresented in other benchmark datasets. The dataset provides acoustic features, extracted tags, and lyrics in English, Portuguese, or Spanish, making it suitable for lyrics-, audio- , or multimodal-based genre classification, music and artist similarity, and popularity regression. Furthermore, the lyrics in the dataset can be used for crossor multilingual text analysis, such as discourse analysis or measuring the differences between emotion transmitted by audio and lyrics. However, it lacks annotations about the semantic similarity between the song lyrics, which is what the Alejandro Benito-Santos, Adrián Ghajari, Pedro Hernández, Víctor Fresno, Salvador Ros, Elena González-Blanco 150 benchmark dataset we propose provides. Given that song lyrics often share similarities with poetry more than other literary genres, it stands to reason that poetry evaluation collections could be leveraged in the study of similarity between song lyrics, even when accounting for the differences between songs and poems. In (Li et al., 2021), a dataset was proposed in order to evaluate the semantic understanding of poetry models through poem matching. The objective was to advance research efforts focused on integrating deep semantics into the generation and comprehension system of Chinese classical poetry. In (Haider et al., 2020) the annotation with experts leaded to an agreement of kappa = .70, resulting in a dataset for large scale analysis. The authors conducted first emotion classification experiments based on BERT, showing that identifying aesthetic emotions was challenging. 3 Annotation Task Given that this was our first attempt at characterizing similarity in song lyrics, and that we wanted to obtain a broad vision of the problem, we decided to limit the number of annotators to three per pair and to favor diversity in the song lyrics. Through a crowdsourcing platform, participants were chosen from a pool of a total of 63 annotators who took part in the study. They were asked to rate the similarity between pairs of lyrics of Spanish songs using a six-point semantic differential scale ranging from 0, for completely different items, to 5 for outstandingly similar items (see details in Section 3.2). They were instructed to evaluate the similarity of pairs of song lyrics based on various criteria, such as the primary theme or context of the lyrics, the message conveyed, the emotions or feelings expressed, the literal meaning, the vocabulary employed, the relationship between the sender and receiver, the language style, and the sociocultural context of the song (see Annex 1 and 2 for a complete task description in Spanish and a translation into English, respectively). Participants were advised to follow their intuition if they had doubts or if the instructions were insufficient to provide a response. The aim of this task was to obtain a comprehensive dataset of similarity annotations for Spanish song lyrics, taking into account multiple dimensions of lyrical content. 3.1 Dataset Description The dataset we prepared for the annotation task contains 75 song lyrics in Spanish that were selected for their diversity and popularity and for representing a wide range of music genres and themes. Also, we included song lyrics of varying lengths (M= 77.61, SD = 34.88, see histogram in Figure 1), to check whether this variable had any influence in the participants’ perception of similarity. 0 20 40 60 80 100 120 140 160 length (in words, binned) 0 1 2 3 4 5 6 7 8 9 10 Number of song lyrics Figure 1: Histogram of song lyrics length in the dataset used to conduct the annotation task. The lengths were approximately normally distributed with a mean length of 77.61 and a standard deviation of 34.88. Although pairs were randomly assigned from the pool of 75 songs, we ensured that enough pair combinations with disparate lengths were paired together. A representation of these pairs can be seen in Figure 2. 3.2 Semantic Differential Scale We employed a 6-point numeric scale in our annotation experiment, similar to the one that was conceived for the SemEval tasks (Agirre et al., 2012). It is worth noting that, as in the original SemEval tasks, our scale devotes one level to total dissimilarity (level 0), and five other different grades to capture a subtler range of semantic similarity (levels 1-5) in an increasing order of intensity. However, we changed the wording of the categories to fit the broader context of similarity between song lyrics. A description of the full scale is provided below: •Completely different (0): the lyrics are entirely dissimilar. LyricSIM: A novel dataset and benchmark for similarity detection in Spanish song lyricsLyricSIM: A novel dataset and benchmark for similarity detection in Spanish song lyrics 151 0 20 40 60 80 100 120 140 160 Length of song 1 (in words) 0 20 40 60 80 100 120 140 160 Length of song 2 (in words) 4 13 22 14 14 10 7 9 8 6 4 6 5 9 2 31 36 17 19 32 31 30 25 17 17 17 10 22 9 8 1 7 3 8 2 17 8 4 1 3 19 11 25 20 6 7 11 12 2 4 10 14 2 9 3 1 3 3 3 2 1 0.0 0.5 1.0 1.5 2.0 mean avg. rating 10 20 30 Count of Records Figure 2: Scatterplot showing composition of pairs in the filtered dataset. The circles represent the size of combination cases (axes X and Y). The size of the circle depicts the number (in white) of particular cases matching lyrics of length Xto lyrics of length Y. The color scale represents the average of mean scores on each combination of lengths. •Barely any similarity (1): the lyrics share minor aspects without semantic importance, such as language style or sociocultural context. •Little similarity (2): there is no semantic similarity (lyrical situation, message, feelings), but the lyrics can be considered thematically (literal meaning) related. •Basic similarity (3): the lyrics resemble each other in message, feelings of the protagonist/singer, lyrical situation, or literal meaning. •Notable similarity / missing details (4): the lyrics share the same message and feelings but differ in lyrical situations and/or literal meaning. •Outstanding similarity (5): the lyrics share the same message, emotions, intentions, and lyrical situation, differing only in lexicon and genre. Upon collection of the results, we obtained more 8,325 pair-wise similarity values corresponding to the comparison of 2775 pairs by three different participants each. An overview of the collected annotations can be seen to the left of Figure 3. 0 500 1,000 1,500 2,000 2,500 3,000 3,500 Number of annotations Original 0 1 2 3 4 5 Rating 0 500 1,000 1,500 2,000 2,500 3,000 3,500 Filtered 0 1 2 3 4 5 Rating Figure 3: Comparison of the number of labels before (left, in blue) and after (right, in red) applying the filtering criteria. Although the difference in size between the two datasets is notable (approximately 75% of the original annotations were discarded in the process), the filtered dataset contains only high-quality annotations. 4 Data Refinement To maximize cost-effectiveness, we did not set any restrictions on the number of pairs that could be annotated by each participant, which resulted in an unbalanced distribution of annotation authoring. To mitigate this potential bias, we took the necessary steps to ensure that the resulting dataset included annotations that could be used as ground truth in future studies. This meant that we kept only those pairs in which a high degree of consensus (see 4.2) between the three annotators could be established. 4.1 Refinement In order to obtain high-quality annotations, we filtered the collected data using different criteria aimed at reducing rating variability. To model and filter the annotation data, as in the STS Core tasks, we departed from the assumption that annotations of similarity (scores from 1 to 5) are fundamentally different from those of dissimilarity (0). This effect can be seen in Figure 4, which shows the kernel density estimations (KDEs) of rating distributions in the STS and LyricSIM datasets. The chart reveals that in all cases, ratings are biased towards the inferior ratings of the Alejandro Benito-Santos, Adrián Ghajari, Pedro Hernández, Víctor Fresno, Salvador Ros, Elena González-Blanco 152 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 ratings 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 0.55 density lyricsim sts-belief sts-forums sts-headlines sts-images sts-students source Figure 4: Comparison of rating distributions between the STS datasets and LyricSIM via kernel density estimation (KDE). In both cases, despite differences in the scale interpretation, the distributions are positively skewed, with STS datasets showing a higher bias towards the first point of the scale. scale, specially in the STS data (around 50% probability of seeing a rating of 0). Thus, we relied on the observation that the probability of seeing a dissimilar pair was hypothetically much higher, and consequently we made a clear distinction between dissimilar and similar items in an attempt to further characterize the tail of the distribution. For dissimilarities, we take only those pairs in which all three annotators agreed that the pair was dissimilar (i.e., all annotators assigned a score of 0 to the pair), giving out a total of 837 pair-wise dissimilarities. In case of similarities, we aimed to capture pairs in which 2 out of 3 annotators agreed exactly on the same score. To avoid including doubtful cases, we chose to exclude pairs in which the third annotator assigned a very different score (i.e., 2 or more points apart from the mode), resulting in the selection of 676 highquality similarity pairs or 24.36% of the original dataset. In total, the refined dataset contains 2,028 high-quality annotations of pairwise similarity and dissimilarity judgments for a total of 75 distinct song lyrics. The 676 high-quality pairs are compared side by side to the unfiltered data in 3, providing a visual estimation of the reduction in size per rating label in the refined dataset. This same data can be found in table format in Table 1. 4.2 Reliability To further assess the reliability of our dataset, we calculated inter-annotator agreement using Krippendorff’s reliability alpha (αk), which gave a value of 0.90 (in the dataset rating count percent Original 0 3058 36.73% 1 3014 36.20% 2 1058 12.71% 3 746 8.96% 4 347 4.17% 5 102 1.23% Filtered 0 837 41.27% 1 705 34.76% 2 360 17.75% 3 88 4.34% 4 34 1.68% 5 4 0.20% Table 1: Number of ratings in the original and filtered datasets. unrefined data, it was 0.27). Krippendorff’s reliability alpha is a metric that generalizes other metrics that are responsible for quantifying the reliability between annotators (inter-rater reliability). It can be used for both ordinal and nominal annotations, as well as with any number of annotators. K-alpha yields a value between 0 and 1, where 1 represents full agreement. However, there are different criteria regarding when to consider that there is enough agreement between annotators. According to the general consensus, a common threshold is to consider that there is enough agreement when K-alpha is greater than 0.8. Using this metric, the resulting inter-annotator agreement of the high-quality annotations dataset was found to be substantial, with a coefficient alpha of 0.90, indicating that the dataset is reliable and thus can be used for future research in the field. Table 2 shows a comparison between STS gold standard datasets used in the SemEval 2014 STS Core task (Agirre et al., 2014) and ours, including dataset size (in sentence pairs), number of annotators per pair and computed Krippendorff’s alpha-reliability scores (Krippendorff, 2004), when possible. Unfortunately, we could not find the results of the STS Spanish tasks disaggregated by annotator. Thus, we derived our results from the English tasks data (available at http://ixa2.si.ehu.es/stswiki/images/ 2/21/STS2015-en-rawdata-scripts.zip). LyricSIM: A novel dataset and benchmark for similarity detection in Spanish song lyricsLyricSIM: A novel dataset and benchmark for similarity detection in Spanish song lyrics 153 Dataset Size Avg.Length K-α STS-images 1500 9.6±3.04 0.82 STS-students 1500 10.44 ±3.34 0.72 STS-headlines 2000 7.5±2.24 0.79 STS-belief 2000 13.01 ±6.83 0.63 STS-forums 1500 15.04 ±3.28 0.66 Avg. STS 0.72 LyricSIM 676 77.61 ±34.88 0.90 Table 2: Comparison between STS datasets and LyricSIM displaying dataset size (in pairs), average sentence/utterance length and calculated Krippendorff alpha (K-alpha). Text sizes in LyricSIM are considerably larger than their STS counterparts and Kalpha varies greatly in the STS datasets, ranging from 0.63 (min) to 0.82 (max). 5 Evaluation In this section, we use the refined dataset introduced in the previous section to analyze how SOTA language models perform in the similarity detection task. We have followed a similar 85-5-10 split as the one used in other studies (Guti´errez-Fandi˜no et al., 2022; Agerri and Agirre, 2023), resulting in 638 song pairs for the train set, 38 song pairs for the development set and 68 song pairs for the test set. To ensure balanced representation of each class, we used stratified sampling during the splitting process. The metrics used for model assessment were Spearman’s Rank Correlation and Pearson’s Correlation (Cer et al., 2017; Devlin et al., 2019). For the purpose of creating a reference point for our dataset’s behavior when subjected to traditional, less complex models, a Support Vector Machine (SVM) model was also trained. Further, to obtain a more holistic measure of the models’ performance, a combined score was also calculated, computed by taking the arithmetic mean of both metrics (Guti´errez-Fandi˜no et al., 2022) (Agerri and Agirre, 2023). 5.1 Language Models We present a brief overview of the models that were selected for evaluation and discuss their respective training and architectural characteristics. By exploring these models in greater depth, we can better understand the nuances of their performance on the refined dataset, ultimately allowing us to make informed decisions regarding their suitability for similarity detection tasks in the Spanish language. The language model architectures we have selected are BERTIN, RoBERTa-base-bne (MarIA base), RoBERTa-large-bne (MarIA large), Sentence Transformer, ALBERTI, DeBERTa, XML-RoBERTa base and XMLRoBERTa large. There are several reasons that led us to choose these five models for our study. First, we aimed to include a significant representation of both monolingual Spanish models, as our dataset comprises Spanish song lyrics, and multilingual models, which have demonstrated superior performance over monolingual models in Spanish tasks. Additionally, we selected both the base and large versions of these models to analyze the resulting metrics after training. Our selection was influenced by an article that evaluated Spanish Language Models(Agerri and Agirre, 2023). The models we chose exhibited high performance in terms of their STS official combined scores, as shown in Table 2 of the referenced article. BERTIN and MarIA have become prominent monolingual models for the Spanish language, as they have been trained on extensive Spanish datasets. BERTIN was trained on the Spanish portion of mC4, which contains approximately 416 million documents and 235 billion words in 1TB of uncompressed data, along with other datasets such as Wikipedia, OpenSubtitles, and Europarl. MarIA was trained on a 570GB corpus of clean and deduplicated texts extracted from the Spanish Web Archive, built by the National Library of Spain between 2009 and 2019. Sentence Transformer was chosen due to its core focus on STS tasks during training, which is highly relevant to our study. ALBERTI was selected because it is domainadapted for poetry, a semantic context closely related to song lyrics. The reasons for choosing DeBERTa, XML-RoBERTa base, and XML-RoBERTa large models are manifold. DeBERTa is selected due to its superior training scores compared to RoBERTa and BERT, incorporating cutting-edge techniques such as Disentangled Attention and Enhanced Mask Decoder, which enhance its performance. XML-RoBERTa base and XML-RoBERTa large models are chosen for their extensive parameter counts, with the base version comprising 270M parameters and the large version containing 550M paAlejandro Benito-Santos, Adrián Ghajari, Pedro Hernández, Víctor Fresno, Salvador Ros, Elena González-Blanco 154 rameters. Additionally, these models are multilingual, trained on 100 languages, making them highly versatile and capable of handling diverse language tasks. In the following, we provide an overview of these models representing both monolingual and multilingual approaches. All these models were trained on various corpora with distinct architectures and training parameters which we discuss in the following section. •BERTIN (Rosa et al., 2022) employs a novel technique called ”perplexity sampling” for pre-training Spanish language models. This method reduces the amount of data and training steps needed compared to traditional approaches, while still achieving competitive results. BERTIN utilizes a two-step training process with different sequence lengths and batch sizes. •The MarIA (Guti´errez-Fandi˜no et al., 2022), RoBERTa-base-bne and RoBERTa-large-bne models (Liu et al., 2019), are state-of-the-art NLP models for the Spanish language. They have been trained on a massive corpus of Spanish text data derived from the National Library of Spain’s selective crawls. The MarIA models employ a single-epoch training approach with no dropout, focusing on tasks such as sentiment analysis, part-of-speech tagging, and named entity recognition. •Sentence-Transformer (Reimers and Gurevych, 2019) is a modification of pre-trained BERT and RoBERTa networks that produce semantically meaningful sentence embeddings. It uses a siamese and triplet network structure, reducing computational overhead while maintaining accuracy. •ALBERTI3is a BERT-based multilingual model trained on poetry datasets, including Spanish resources. Although no publication detailing the corpus and training methodology exists, the model leverages domain adaptation to capture patterns and features specific to poetry. •DeBERTa (He et al., 2021), Decodingenhanced BERT with Disentangled Attention, refines BERT and RoBERTa 3https://huggingface.co/flax-community/ alberti-bert-base-multilingual-cased through disentangled attention and superior masked decoding. The model’s virtual adversarial training enhances generalization, improving efficiency in pre-training and performance in NLU and NLG tasks. An expansive DeBERTa version surpasses human performance on the SuperGLUE benchmark, a notable achievement in macro-average score. •XML-RoBERTa (Conneau et al., 2020) enhances cross-lingual understanding (XLU) through extensive study of unsupervised cross-lingual representations. Introducing XLM-R, a transformerbased multilingual model pre-trained on text in 100 languages, achieving stateof-the-art results in cross-lingual classification, sequence labeling, and question answering. The authors delve into key factors and trade-offs between positive transfer, capacity dilution, and performance across languages. They demonstrate that a single large model can effectively encompass all languages without sacrificing per-language performance. 5.2 Training Parameters Details In this section, we delve deeper into the training parameters employed by the monolingual and multilingual models, highlighting the similarities and differences that may contribute to their respective performance in the similarity detection task. We want to compile the training parameters in a list format, which we have obtained after reading the papers of each of these models. These parameters serve as a starting point for replicating and investigating the results obtained by the research and development teams who worked on each of these models. The training parameters for the selected models reveal substantial differences in the pre-training procedures employed by the monolingual and multilingual models. For example, BERTIN follows a twostep training process with varying sequence lengths and batch sizes, while the MarIA models employ a single-epoch training approach with no dropout. These variations in training procedures and corpora used for pre-training contribute to the distinct performance characteristics of each model. As researchers continue to develop and evaluate Spanish language models, it is crucial to assess how these differences in training parameLyricSIM: A novel dataset and benchmark for similarity detection in Spanish song lyricsLyricSIM: A novel dataset and benchmark for similarity detection in Spanish song lyrics 155 ters impact the models’ effectiveness in downstream tasks. The selected Spanish language models offer valuable insights into the development and application of transformer-based language models for the Spanish language. 5.3 Fine-tuning The fine-tuning process was carried out following the same practices found in the aforementioned studies. We used the same scripts as the MarIA team4(Guti´errez-Fandi˜no et al., 2022), which are based on HuggingFace Transformers library (Wolf et al., 2020), with minor modifications to adapt them to our dataset structure. To maintain consistency across the models, we initialized each one of them with a random head and employed a fixed seed for reproducibility. We conducted a grid search over the following search space: •Weight decay: 0.1, 0.01 •Learning rate: 1e-5, 2e-5, 3e-5, 5e-5 •Batch size: 8, 16, 32 Due to memory constraints, especially for larger models, in cases where the batch size exceeded capacity, gradient accumulation was used achieving the same effective batch size; the rest of the hyperparameters remain the same as the HuggingFace defaults. The maximum sequence length was 512 tokens for all models, chosen to accommodate all sentence pairs in the dataset. To prevent overfitting, we trained each model for a maximum of 5 epochs with Adam optimizer and a linear decaying learning rate, selecting the checkpoint with the highest score according to development set. Finally, we perform evaluation on the test set for each model, with the best checkpoint from the previous step. The optimal configuration of hyperparameters per model is shown in Table A2. 5.4 Results The results are shown in Table 3 for each model, fine-tuned as described. Among the eight models investigated, MarIA large demonstrated the highest performance in terms of the combined score metric, while MarIA’s base counterpart ranked fourth; mDeBERTa model closely followed MarIA large, exhibiting competitive results. The XLM-RoBERTa large and base models are 4https://github.com/PlanTL-GOB-ES/lmspanish fifth and sixth respectively, showing again a larger model performing better than its base variant, while Sentence Transformer based on the same architecture performed better than both, achieving third place. ALBERTI, the only BERT based model of this study, was only slightly behind some of the other models, placing seventh. BERTIN, although not dramatically behind the SVM, placed last. These findings suggest that the MarIA large model is the most effective in capturing semantic textual similarity on songs, with mDeBERTa as a strong contender. The results for the evaluation on the development set, used as criteria for selecting the best checkpoint from the fine-tuning process and hyperparameter selection, can be found in Table A1 of the appendix. A comparative analysis of the development and test set results may provide further insights into the generalization capabilities of the models and the effectiveness of the fine-tuning process. 6 Discussion In this section, we will delve into the results obtained from the data refining process and the fine-tuning of the baseline models, highlighting the insights and implications of these outcomes for our understanding of semantic similarity in Spanish song lyrics. 6.1 Evaluation Results It is noteworthy that the highest performing model, MarIA large, is a Spanish-specific model. This observation suggests that models trained on a particular language may have an advantage in capturing the subtleties of that language for this particular task, although multilingual models can still achieve competitive performance. Further studies are recommended to provide a more comprehensive understanding. Furthermore, our results validate that larger models (e.g. Maria large) lead to improved performance compared to their base counterparts. This result emphasizes the need to factor in model size and computational demands when choosing a model for practical applications. Another interesting observation is the difference in performance between the BERTbased model, ALBERTI, and the other RoBERTa and XLM-RoBERTa-based models. Surprisingly, even though this model was trained on poems, which exhibit a structure more closely related to songs than the data Alejandro Benito-Santos, Adrián Ghajari, Pedro Hernández, Víctor Fresno, Salvador Ros, Elena González-Blanco 156 model name STS combined combined spearmanr pearson BERTIN 79.45 85.72 86.32 85.12 MarIA large 84.11 90.02 89.94 90.11 MarIA base 85.33 86.75 89.02 84.5 XLM-RoBERTa base 83.47 86.45 88.19 84.72 XLM-RoBERTa large 84.04 86.74 88.15 85.33 Sentence Transformer XLM-R - 88.91 89.55 88.28 mDeBERTa3 83.61 89.15 89.07 89.23 ALBERTI - 86.35 88.29 84.42 SVM (RBF) - 86.18 86.57 85.78 Table 3: Test combined scores for all the models considered (best in bold). STS dataset results from other studies (Agerri and Agirre, 2023) have been added for illustration purposes. used to train the other models, performs relatively worse. A possible avenue for exploration would be the tokenization process used by this model, as it does not take into account line-break characters (\n) commonly used to delimit verses and stanzas, which in turn could impact the model’s ability to learn the structure of a song. This also accentuates the importance of considering the structure and formatting of the input data when selecting a model for a specific task. Similarly, the advantage in performance of Sentence Transformer, which is based on the XLM-RoBERTa architecture, in comparison to the other XLM-RoBERTa models may be partially attributed to the supplementary fine-tuning process applied to sentence pairs for semantic textual similarity. We argue this fine-tuning might have the potential to refine the model’s ability to capture semantic relationships more effectively, which would explain this observation in the data. The results also revealed that the performance of the SVM model, a comparatively simpler machine learning architecture, was not significantly different from some of the more complex transformer-based models (e.g. BERTIN). This could potentially indicate that the dataset size might not be large enough or sufficiently complex to highlight the strengths of transformer models. These findings have practical implications for the development of real-world applications related to the task at hand: the performance of the MarIA large and mDeBERTa models suggests that they may be well-suited for tasks such as song recommendation, lyric analysis, or music information retrieval. 6.2 Similarity vs Length In addition to the findings discussed earlier, an interesting observation that we could derive from the annotation data was that similarity scores increased with the total length of the lyrics in a pair (see Figure 5), suggesting that longer lyrics may contain more opportunities for shared vocabulary and thematic elements to contribute to overall semantic similarity. Although more research is needed in this area, current evidence points to the potential utility of considering vocabulary overlap as an important factor in determining semantic similarity between pieces of text (Abe et al., 2022). As we discuss in the next section, there is a currently a lack of examples of annotated pairs exhibiting varying degrees of similarity (low, medium or high), which we aim to resolve in future studies. 7 Conclusion and Future Work In this study, we have laid the groundwork for assessing semantic similarity in the context of Spanish song lyrics by presenting a dataset and baseline results from pre-trained SOTA models. Our results provide insights into the performance of these models, revealing potential strengths and weaknesses as well as opportunities for future research. Beyond that, we have provided a reflection on the data collection process typical of the similarity annotation task, and a detailed characterization of the filtering process that we followed to increase the quality of the annotation data. Finally, we provide all the code and data necessary to reproduce our research at the repository located at https: //github.com/linhd-postdata/lyricsim. To further advance this research area, during the course of our research we identified LyricSIM: A novel dataset and benchmark for similarity detection in Spanish song lyricsLyricSIM: A novel dataset and benchmark for similarity detection in Spanish song lyrics 157