Using Autotagging for Classification of Vocals in Music Signals

FACULDADE DE ENGENHARIA DA UNIVERSIDADE DO PORTO Using Autotagging for Classification of Vocals in Music Signals Nuno Hespanhol DISSERTATION Mestrado Integrado em Engenharia Informática e Computação Supervisor: Fabien Gouyon June 2013 c Nuno Hespanhol, 2013 Using Autotagging for Classification of Vocals in Music Signals Nuno Hespanhol Mestrado Integrado em Engenharia Informática e Computação Approved in oral examination by the committee: Chair: Gabriel de Sousa Torcato David External Examiner: Paulo Jorge Freitas Oliveira Novais Supervisor: Fabien Gouyon June 2013 Abstract Modern society has drastically changed the way it consumes music. During these last recent years, listeners have become more demanding in how many songs they want to have accessible and require to access them faster than ever before. The modern listener got used to features like automatic music recommendation and searching for songs that, for example, have “female vocals” and “ambient” characteristics. This has only been possible due to sophisticated autotagging algorithms. However, there has been an increasing belief in the research community that these algorithms often report over optimistic results. This work approaches this issue, in the context of automatic vocal detection, using evaluation methods that are rarely seen in literature. Three methods are conducted for the evaluation of the classification model developed: same dataset validation, cross dataset validation and filtering. The cross dataset experiment shows that the concept of vocals is generally specific per dataset rather than universal as expected. The filtering experiment, which consists of iteratively applying a random filterbank, shows drastic performance drops, in some cases, from a global f-score of 0.72 to 0.27. However, these filters have been showed not to affect the human ear’s ability to distinguish vocals, by conducting a listening experiment with over 150 candidates. Additionally, a comparison between two binarization algorithms - maximum and dynamic threshold - is performed and shows no significance difference. The results are reported on three datasets that have been widely used within the research community, on which a mapping from its original tags to the vocals domain was performed and which is made available to other researchers. i ii Resumo A sociedade moderna mudou drasticamente a maneira como consome música. Durante estes últimos anos, os ouvintes tornaram-se mais exigentes em relação ao número de músicas que querem ter acessíveis e querem acedê-las mais rapidamente do que anteriormente. O ouvinte moderno habituou-se a funcionalidades como recomendação automática de música e à possibilidade de pesquisar músicas com características, como por exemplo, “female vocals” e “ambient”. Este tipo de funcionalidades só foram tornadas realidade devido a sofisticados algoritmos de autotagging. Contudo, existe uma crença pela comunidade de investigação que estes algoritmos reportam muitas vezes resultados demasiadamente otimistas. Este trabalho aborda este problema, no contexto de deteção automática de voz, usando métodos de avaliação raramente vistos na literatura. Três métodos são realizados para a avaliação do modelo de classificação desenvolvido: validação entre o mesmo dataset, validação entre datasets e filtragem iterativa aleatória. A avaliação entre datasets mostra que o conceito de vocais é de uma maneira geral específico por dataset em vez de universal, como seria de esperar. A experiência dos filtros, que consiste em iterativamente aplicar um filterbank aleatório, mostra drásticas baixas na performance do sistema, em alguns casos, de um f-score global de 0.72 para 0.27. Contudo, através da realização de uma experiência perceptiva com mais de 150 candidatos, mostra-se que estes filtros não afectam a capacidade do ouvido humano de distinguir vocais. Adicionalmente, é realizada também uma comparação entre dois métodos de binarização - máximo e limite dinâmico - que não mostra uma diferença significativa entre eles. Os resultados são reportados em três datasets que foram largamente utilizados pela comunidade de investigação, sobre os quais é realizado um mapeamento das suas tags originais para o domínio vocal e disponibilizado para que outros investigadores possam usá-los. iii iv Acknowledgments I would like to thank several people that directly or indirectly made this thesis possible. First and foremost, I must thank Prof. Fabien Gouyon for his guidance and support throughout this thesis. A special thanks to João Lobato Oliveira, that even though wasn’t a formal co-supervisor to this work in many ways behaved as such providing invaluable insight and advice. I have also been assisted by members from the SMC group at INESC which I would like to thank. Last but not least, I would also like to thank my family and closest friends for their comprehension, patience and support during the course of this work. v LIST OF FIGURES xii List of Tables 2.1 Sampling rates, its maximum available frequencies and corresponding general uses. 7 2.2 Common bit depths and where they are normally used . . . . . . . . . . . . . . . 8 2.3 Truth Table highlighting Type I and Type II errors in red. . . . . . . . . . . . . . 19 2.4 Example of a confusion matrix. . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.5 MIREX 2012 results for Audio Tag Classification - Major Miner task. . . . . . . . 21 2.6 MIREX 2012 results for Audio Tag Classification - Mood task............ 21 3.1 Statistics for the datasets used in the experiments. . . . . . . . . . . . . . . . . . 24 3.2 Listening evolution in CAL500. . . . . . . . . . . . . . . . . . . . . . . . . . . 26 3.3 Listening evolution in Magtag5k. . . . . . . . . . . . . . . . . . . . . . . . . . . 27 3.4 Analysis via number of tag occurrences for each tag in the MSD24k dataset. . . . 29 3.5 MSD24k selected tags to include in Nonvocals class and in which percentage. . . 29 3.6 Listening evolution in MSD24k. . . . . . . . . . . . . . . . . . . . . . . . . . . 30 3.7 Distribution of Vocals and Nonvocals per dataset and globally. . . . . . . . . . . 30 5.1 Comparison of dynamic threshold and maximum binarization algorithms. . . . . 38 5.2 Same Dataset experiment results on CAL500. . . . . . . . . . . . . . . . . . . . 39 5.3 Same Dataset experiment results on Magtag5k. . . . . . . . . . . . . . . . . . . 39 5.4 Same Dataset experiment results on MSD24k. . . . . . . . . . . . . . . . . . . . 40 5.5 Cross Dataset Experiment training with CAL500. . . . . . . . . . . . . . . . . . . 41 5.6 Cross Dataset Epxeriment training with Magtag5k. . . . . . . . . . . . . . . . . 42 5.7 Cross Dataset Experiment training with MSD24k. . . . . . . . . . . . . . . . . . 42 5.8 Summary of Filters experiment results for first and last iteration. . . . . . . . . . 44 6.1 Results of a paired t-test onquestion1. ...................... 53 B.1 ListeningExperimentData............................... 61 xiii LIST OF TABLES xiv Abbreviations FEUP Faculty of Engineering of the University of Porto GMM Gaussian Mixture Models HMM Hidden Markov Models INESC Institute for Systems and Computer Engineering of Porto ISMIR International Society for Music Information Retrieval MFCC Mel Frequency Cepstral Coefficients MIDI Musical Instrument Digital Interface MIR Music Information Retrieval MIREX Music Information Retrieval Evaluation eXchange MSD Million Song Dataset SMC Sound and Music Computing Group SVM Support Vector Machines xv Chapter 1 Introduction 1.1 Motivation Music is one of the most ancient art forms. It dates back to the prehistoric ages and has forever played a very important role in society. It is an an essential part of way of life across countries, religions and cultures. Indeed, there has been and always will be a need for society in general to be able to access and consume music. In these last few years however, music consumption has changed drastically. There have appeared several commercial services such as iTunes, Amazon Music, Spotify and Last.fm that offer music collections in the order of the millions. In addition, these collections are constantly growing, therefore making the process of accessing and consuming them more difficult than ever before. A way to make it easier for the listeners is to make use of metadata, that is, data about the data itself, normally called music tags. Music tags are simply keywords that are used to describe music content. Autotagging can then be defined as the process of automatically associating these kinds of tags with music content with the ultimate goal of helping future listeners to more easily find what they are looking for. A fundamental question about this process of annotating music with tags is the implications of it being manual or automatic. Naturally, both have its advantages and disadvantages. It is clear that manual annotation will require more time, but will probably be more accurate than automatic processes. A perfect example of the manual annotation process would be the Pandora 1 Internet Radio. It is a commercial service that offers its listeners the ability to create automatic intelligent playlists based on their musical preferences. In order for this to be possible, more than 40 musicologists have been paid to annotate musical files with over 400 attributes since 2000. Considering the ever growing number of music that are edited every day, it is easy to see how this is a big weakness of this kind of process. However, one may argue that the quality of the musical tags is superior to the ones obtained from automatic algorithms of tagging. Still, it is clear that it is an obvious limitation 1www.pandora.com 1 Introduction to annotate big collections of music such as iTunes and Amazon only manually, therefore, making automatic tagging a necessity. 1.2 Context 1.2.1 MIR Research The work developed during the course of this dissertation is inserted in the field of Music Information Retrieval (MIR). MIR is the interdisciplinary science of retrieving information from music. The first research works on automatic genre classification date back to 2001 [ TC02 ]. As for music autotagging, the first research findings date back to late 2005 [ ME05 , TBL06 ]. Therefore, music autotagging is still considered a very recent research problem which is part of the research field of Music Information Retrieval (MIR). This research field most important conference is the International Society for Music Information Retrieval Conference (ISMIR) which first happened in 2000. MIR has reached a certain level of maturity and it’s now entering what this society founders call it its “teen” years [ DBC09 ], since its still in its early stages when compared to other research areas such as signal processing and speech processing [Sor12]. We would like computers to help us discover, manage, and describe the many new songs that become available every day. The goal of autotagging is not to replace humans: the best description of an album is still (and will most likely continue to be) the one of a music expert. It is impossible though for any group of experts to listen to every music piece on the Internet and summarize it in order for others to discover it. That is where autotagging comes in. For instance, services like Pandora or Last.fm 2 both provide the feature of automatic recommendations based on listeners’ musical taste. In order to achieve this, they assume that if a listener liked songs Aand Band you liked A, you might like Bas well. These kinds of algorithm have proven to be extremely efficient. However, it leaves two major problems: • cold start problem: new songs on the music databases tend to not be very popular since they are still unknown and because of that they tend to not be tagged and ultimately never popular. • extreme popularity problem: songs that eventually get popular will only tend to get more popular over time, since they are the ones being recommended all the time. It is in these two particular cases that music autotagging algorithms can improve such services. Ultimately, we can say that the goal of music autotagging is to do for music what Google and Yahoo! did for web documents and change the way we handle music [BMEM10]. There have been attempts to unite the research community to join efforts in order to tackle this ultimate goal. That is how MIREX, M usic I nformation R etrieval E valuation e X change, was born in 2005. It consists of an annual evaluation campaign for MIR algorithms, including autotagging ones, which tries to rank these algorithms and to promote cooperation between researchers. 2www.last.fm 2 Introduction 1.2.2 Research at INESC Porto This dissertation was developed in partnership with the Sound and Music Computing Group 3 (SMC) of the Institute of Engineering and Computer Systems of Porto 4 (INESC). This research group combines basic research in signal processing, pattern recognition, music and human-computer interaction, and aims at contributing to make computers better understand, model and generate sounds and music. 1.2.3 Personal trajectory The work presented in this dissertation spans over five months and it gave me the chance to apply some knowledge I have obtained during my private, professional and student life. The inclination for music started rather early having studied in music at a conservatory, where, apart from learning the guitar and piano, I also learned about composition and acoustics. Despite this I never considered working or studying in the music research area until the later years in my degree where I choose as an elective subject Automatic Music Generation. A great challenge developing this work had to deal with learning about Digital Signal Processing, in which I had no background whatsoever before starting this work in February, coming from a more Software Engineering background rather than an Electric Engineering one. 1.3 Objectives The objectives for this dissertation were: •to develop and thoroughly evaluate a system capable of identifying vocals in music signals; • to propose new methods of evaluation that complement those commonly seen in literature such as cross dataset testing and iterative random filterbank; •to do a critical study on current state-of-the-art vocals segmentation algorithms; • to conduct a listening experiment with human candidates to validate the system and methods proposed. 1.4 Relevant Contributions The most important contribution of this work is the systematic evaluation of an autotagging system in the context of the detection of the presence of human voice in music audio signals. Three different evaluations were applied to the system developed, two of which (cross dataset and iterative random filterbank) are rarely seen in literature and which present a new perspective on evaluation of autotagging systems. A cross dataset experiment showed that high performance in a 3http://smc.inescporto.pt 4http://inescporto.pt 3 Introduction typical cross-validation within the same dataset does not guarantee a generalization of the model to other dataset, while iterative random filterbank showed that such filtering has a drastic impact on a system’s performance having been observed drops in global f-score from 0.72 to 0.27. Additionally, a mapping into the vocals domain of three datasets widely used within the research community was made available 5 . Lastly, some guidelines for conducting listening experiments with human candidates are described that could be of use to others researchers looking to conduct a listening experiment. 1.5 Structure of the Dissertation This dissertation is structured in 7 chapters that somewhat resemble the chronologically way in which the present work took place. The current chapter, Chapter 1, contextualizes the research conducted, its motivations, aims and main contributions to the field. Chapter 2 provides the theoretical background and explanations to many of the concepts that were of fundamental importance to the work developed, ending with a comparison of current state-of-the-art systems by other researchers. Chapter 3 describes the process of creating the three datasets in which all the work was based. Chapter 4 presents the framework that was used for the experiments conducted. Chapter 5 details the experiments conducted and its results. Chapter 6 describes the design and results of a listening experiment done with over 150 human candidates to further validate findings from the experiments reported in the previous chapter. Finally, Chapter 7 ends with the discussion of the results, conclusions and future work. 5http://paginas.fe.up.pt/ ei08067/dokuwiki/doku.php. 4 Chapter 2 The Autotagging Ecosystem This chapter presents the general framework of an autotagging system as well as the theoretical background of all the concepts it involves. It ends with a summary of the performance of some systems at the MIREX competition and a comparison of current state-of-the-art system for the task of voice segmentation. 2.1 Overview A general overview of an autotagging system is given in Figure 2.1. Basically, the goal is to train a machine learning algorithm using a subset of a database of songs whose tags are already known. For this training to happen, the first step is to extract the low-level features of the sound signal. Then, the dimension reduction takes place, which tries to remove redundancy and simplify the dimensionality of the problem. That is when the machine learning algorithm is ready to learn the data. Many different algorithms can be used, which will be discussed in Section 2.5. This training will create a model that can then be used to automatically assign tags to the rest of the songs in the database. The final step is the evaluation by analyzing the performance of the system. This is possible because the data classified by the system is part of a dataset that is known beforehand. Metrics such as percentages of success and F-measures are used to access a system performance, as discussed in Section 2.6. 2.2 Feature Extraction It would be very useful for any MIR task that music files were available under a structured and easy-to-parse format such as MIDI 1 or MusicXML 2 . However, that is not generally the case and music content is most of the times only available in a way that matters to the listeners, which is of a waveform. Therefore, the first step in virtually any MIR algorithm is to extract information that characterizes audio signals. This step is called Feature Extraction. 1 Stands for Musical Instrument Digital Interface, a communication protocol introduced in 1983 that for the first time proposed a standard way to structure music data and communications between systems and applications. 2A standard open format to represent Western musical notation. 5 The Autotagging Ecosystem As a quick summary of these spectral features, Figure 2.12 shows how there is a clear different behavior for each of them over music,speech and noise. Figure 2.12: Spectral Features change in Music, Speech and Noise. Adapted from [Gia08]. 2.2.6 MFCCs The first step towards understanding the MFCCs is understanding the Mel Scale. The Mel scale was created by [ SVN37 ] and is widely used in tasks involving pitch perception as it is the case with many MIR applications. It can be defined as a perceptual scale of pitches judged by listeners to be equal in distance from one another. This scale relates perceived frequency, or pitch, of a pure tone to its actual measured frequency. Humans are much better at discerning small changes in pitch at low frequencies than they are at high frequencies. Incorporating this scale makes our features match more closely what humans hear. Figure 2.13: The Mel Scale. It was built by playing experimental subjects a reference tone and asking them to adjust a second tone until it sounds twice as high or half as high in pitch. By varying the reference tone 12 The Autotagging Ecosystem and by testing a large number of subjects it was possible to build up a scale which relates pitch (in Hertz) to a subjective perceptual scale of pitch. 1000 Hz was arbitrarily selected and its pitch in mels was chosen to be 1000 mels. 500 mels is perceived as half the pitch of 1000 mels, whilst 2000 mels is perceived as being twice the pitch of 1000 mels. The MFCCs are the coefficients of the Mel Frequency Cepstrum (MFC). Usually, MIR applications only make use of the first 13 coefficients. They were introduced by [ DM80 ] in the 1980’s, and have been state-of-the-art ever since [ mfc ]. It is also common to use the first and second order derivatives of the MFCC, which are called ∆ MFCCs (delta) and ∆∆ MFCCs (double delta), respectively. The process for obtaining these coefficients can be described as follows and as Figure 2.14 schematically shows: 1. Calculate the FFT of the signal, in windows. 2. Map the spectrum obtained onto the mel scale, using the Mel Filterbank (Figure 2.15). 3. Compute the logarithm of the powers at each of the mel frequencies (since the human ear perceives loudness logarithmically rather than linearly4). 4. Apply the Discrete Cosine Transform5(DCT) of the list of mel log powers. 5. The MFCCs are the amplitudes of the resulting spectrum, which is called the Mel Cepstrum. Figure 2.14: MFCC calculation process. Figure 2.15: The Mel Filterbank. Adapted from [mfc]. 4 As an example, for a human ear to perceive a sound two times more loud, it is necessary to apply a eight times more energy to it 5 It is very similarly to the DFT, but, instead of using harmonically-related complex exponential functions, it uses real-valued cosine functions. 13 The Autotagging Ecosystem 2.3 Dimension Reduction During the Feature Extraction process a lot of data can arise, many of which (i) might not even be relevant to the classifier or (ii) might be too much for it to be able to compute in reasonable time. Therefore, methods have been proposed to reduce the dimension of this problem, many times without loosing any information at all. Some of them are: •Principal Component Analysis (PCA) •Linear Discriminant Analysis (LDA) •Independent Component Analysis (ICA) •Non-negative Matrix Factorization (NMF) •Relevant Component Analysis (RCA) In the framework used and described in Chapter 4, considering the relatively low number of features to be used no Dimension Reduction algorithm will be used. Therefore, the description of these algorithms is out of the scope of the present dissertation. 2.4 Data As mentioned in Section 2.4, there are many ways to obtain tags, namely through surveys, social tags, games and/or web documents. The datasets used for training the classifier are of extreme importance in the training process. It is obvious that if the dataset is poorly notated, the classifier will be badly trained and consequently it will classify many inputs wrongly. That is why there are many datasets shared among the research community such as the ones presented in Chapter 3that took careful consideration to be built. Music tags are simply keywords that are used to describe music content. For example, if we consider the song Smoke on the Water by Deep Purple, its music tags directly extracted from the Last.fm site, are shown below. They are represented under the format of a tag cloud, in which each tag is bigger according to its frequency of tagging. Figure 2.16: Tag cloud for the song “Smoke on the Water” by “Deep Purple” on Last.fm. Music tags can be related to various facets of the music: •Emotion (e.g. epic) 14 The Autotagging Ecosystem •Musical instruments (e.g. guitar solo, keyboards) •Genre (e.g. rock, pop, reggae) •Date (e.g. 70s, 80s, 90s) •Usage (e.g. seen live, riot, war) There are also several ways to use metadata to describe the music, but they fall into three basic categories: editorial metadata, cultural metadata, and acoustic metadata [Pac11]. •Editorial : data obtained directly from the editor, such as song titles, album name, date and place of recordings, composers, performers, etc. •Cultural : information that is produced by the environment or culture, which typically results from the analysis of emerging patterns, categories or associations of sources like Google searches, Wikipedia articles or music databases (i.e. Allmusic6) •Acoustic : it is the entirely objective kind of metadata regarding the music content. Typical examples are beats per second, metric measure, low-level descriptors (Mel Frequency Cepstral Coefficients, MFCCs). They can be obtained through five different methods [TBL08]: •Surveys : paying people (musicologists or not) to assign tags to music files. This is how the dataset CAL5007was built through the payment to undergraduate students to do that job. •Social tags : the most paradigmatic example of this kind of tagging is the popular site Last.fm where users voluntarily tag songs. Recently, there has been a keen interest on literature on how to better make use of this this kind of tagging. There are naturally a few obvious downsides to it, as for example, the quality of the tags being low, but, it has been show to improve a lot of autotagging algorithms. •Games : there have been authors exploring the gamification of music tagging with great results, such as ListenGame [ TLB07 ], Tag-a-Tune [ LvAD07 ], and MajorMiner [ ME07 ]. For instance, the Tag-a-Tune game is a two-player game where the players listen to a song and are asked to enter “free text” tags until they both enter the same tag. •Web documents : this method for obtaining tags is done through web mining of music websites, artist biographies, album reviews, songs reviews, Wikipedia articles. •Autotags : all previous methods require the tagging process to be done by humans. This method relies on automatic processes by computers to do so, and it consists of the system this dissertation is going to explore. 6www.allmusic.com 7http://cosmal.ucsd.edu/cal/projects/AnnRet/ 15 The Autotagging Ecosystem 2.5 Classifier Machine Learning is the branch of Artificial Intelligence that studies the construction of systems that can learn from large amounts of data. Therefore, it uses generalization to solve many of its problems. In the particular case of music autotagging, the basic idea is to have a relatively large database of songs (see Chapter 3) from which the machine learning algorithms can construct a model that can then be applied to new songs the model has never seen before. Machine Learning algorithms are divided into three main categories: • Supervised learning: when there is a what is a called a ground truth set of elements to train the model, before actually running it on new data. Ground truth means facts that are assumed as a fundamental truth. This kind of approach to training machine learning systems is the most popular in music autotagging algorithms and is the one to have shown better results. Hence, so much effort have been put into creating ground truth databases of songs (see Chapter 3). • Unsupervised learning: when there is no prior knowledge about the elements to be learned. This kind of learning is not very common in music autotagging, although it has been used in music genre classification, which is a particular case of the music autotagging problem. • Semi-supervised learning: this approach consists of a mixture of the previous two - it combines the use of both labeled and unlabeled data to create a classifier. Again, in the case of music autotagging, this would mean to have both songs in the training database with music tags and others with no music tags. This module of the autotagging system is the one responsible for automatically linking tags to audio features. There are many algorithms: •Support vector machines (SVM) •Gaussian Mixture Models (GMM) •K-Nearest Neighbors (KNN) •Artificial Neural Networks (ANN) SVMs are one of the most widely used machine learning algorithms. Their performance as a classifier has been demonstrated and they can also handle regression, but they have the big disadvantage with its training speed, which is in the order of N2 , N being the number of audio files in the database. Since this will be the classifier to be used in framework (Chapter 4) a more detailed explanation is given below (Section 2.5.1). Gaussian mixture is a learning algorithm that models a distribution using gaussians. Gaussian mixtures are more powerful (in terms of representation capacity) than an algorithm that only classifies because it estimates the likelihood of a data point. K-Nearest Neighbors (KNN) is one of the most simple, and surprisingly effective, machine learning techniques [SLC07]. 16 The Autotagging Ecosystem Neural networks have not been been used in recent papers on automatic tagging of audio, but there is no reason why it would not work efficiently [BMEM10]. 2.5.1 Support Vector Machines SVMs are a type of supervised learning approach pattern recognition algorithms. They basically binary classify an input instance as one out of two classes, having previously been trained with some labeled data. Its process can be described as follows. 1. Inputs are formulated as feature vectors (in our case using the features of the signal). 2. Mapping these vectors into a feature space by the use of a kernel function. 3. Optimally separate the classes of training vectors to build a model of classification. There are three main types of SVM classifiers: the hard-margin, soft-margin and non-linear. 2.5.1.1 Hard Margin Linear SVM There are lots of possible solutions for choosing the plane (or hyperplane if in a higher dimensional space) to separate two classes of data in the feature space. As can be seen in Figure 2.17 both the black and green planes separate the pink and blue classes, and there is an infinite more number of planes that do so. SVM, however, guarantees that it will find the optimum solution to maximize its margin to the feature vectors (the instances of the data that are in the dotted-lines and that directly affect the selection of the optimum solution). In Figure 2.17 it’s clear how both black and green planes are a solution, but with the green providing a bigger margin and therefore yielding better classification results. Figure 2.17: A hard-margin linear SVM example. 2.5.1.2 Soft Margin Linear SVM There will be times however, where the data won’t allow for linear separability of its classes (even in higher dimensions). A soft-margin SVM solves this problem by allowing mislabeled examples. It then tries to find the plane (or hyperplane) that splits the examples as cleanly as possible, while still maximizing the distance to the nearest clean example. The degree of allowed mislabeled 17 The Autotagging Ecosystem examples is controlled through the C constant called Slack variable. Figure 2.18 clearly illustrates how this trade-off of allowing for a few mislabeled examples (circled in dark green) for the cost of linearly separating the two classes. Figure 2.18: A soft-margin linear SVM example. This kind of SVM gives a greater error on the training dataset comparing to a hard-margin SVM, but improves generalization to the test dataset, not to mention making it possible to linearly separate data which wouldn’t be possible otherwise. 2.5.1.3 Nonlinear SVM Figure 2.19: Kernel function transformation to higher dimensional space example.8 The idea of a nonlinear SVM is to gain linearly separation by mapping the data to a higher dimensional space. The following set on the right of Figure 2.19 can not be separated by a linear function, but can be separated by a quadratic one. A φ transformation is then applied to the input vectors to map them to a higher dimension space, through the means of a kernel function. The most popular choice is the radial basis function kernel, also known as RBF and is expressed by the following equation, being γthe only variable parameter: K(x,x0) = e−γkx−x0k2for γ>0 8http://www.imtech.res.in/raghava/rbpred/algorithm.html 18 The Autotagging Ecosystem 2.6 Evaluation This part of the system consists of evaluating whether the automatic tagging was done correctly or not. Most commonly, autotagging system publications present their evaluation statistics considering two scenarios: globally (considering all tags of a system) and per tag (considering the average of the performance per tag). In classification algorithms, as it is the case with any autotagging algorithm, it is important to bear in mind the so called Type I and Type II errors. Type I errors account for the False Positives (FP), that is the instances that are considered of a class but aren’t, whereas Type II errors account for the False Negatives (FN), that is the instances that aren’t considered of a class but should. A more visual explanation of this can be seen in Table 2.3. Predicted Class True False Actual Class True True Positive (TP) Type I Error False Type II Error True Negative (TN) Table 2.3: Truth Table highlighting Type I and Type II errors in red. In the same line of reasoning, a typical way of presenting results of machine learning classification methods, it’s through a confusion matrix, as shown below in Table 2.4 for the context of our work. Predicted Class Vocals Nonvocals Actual Class Vocals 220 2 Nonvocals 22 7 Table 2.4: Example of a confusion matrix. This basically means that: •220 excerpts were correctly identified as having Vocals. •2 excerpts had someone singing, but the system assumed they didn’t (Type I error). •22 excerpts didn’t have anyone singing, but the system assumed they did (Type II error). •7 excerpts were correctly predicted as being Nonvocals. While the confusion matrix might be a good way to visually present machine learning classification results, for actually benchmarking a system it is common to summarize all of these four variables (TP, TN, FP, FN) using the following evaluation metrics, each of them with a particular meaning on a system’s performance: 19 The Autotagging Ecosystem Precision is the ratio of predicted classes that are relevant. Precision =TP T P +FP Recall is the ratio of relevant classes that were predicted. Recall =TP T P +FN F-score is a weighted harmonic mean average measure of both precision and recall. Fscore =2 1 p+1 R =2PR P+R Accuracy is the ratio of correctly predicted classes over all possible classes. Accuracy =T P +TN T P +FP +TN +FN 2.7 MIREX The Music Information Retrieval Evaluation eXchange (MIREX) is an annual evaluation campaign for Music Information Retrieval (MIR) algorithms, coupled to the International Society (and Conference) for Music Information Retrieval (ISMIR) [ 15 ]. MIREX project tries to rank MIR algorithms in each of its categories and above all to promote cooperation between researchers. There are many categories, for the music autotagging task the ones relevant are Audio Genre Classification,Audio Music Mood Classification and in particular Audio Tag Classification. The MIREX defines the Audio Tag Classification task as “This task will compare various algorithms’ abilities to associate descriptive tags with 10-second audio clips of songs”. The first edition of this task was in 2008 and was held ever since to 2012, the last happening of ISMIR, conference in which MIREX results were disclosed and presented below in Table 2.5 and Table 2.6. As mentioned in Section 2.6 the main metric used for ranking the tagging systems is the F-measure. MIREX divides the Audio Tag Classification task into two, considering they are tested against two different databases of songs, namely MajorMiner9and Mood10 datasets. The algorithm that performs the best in the Major Miner dataset is the [Ham11], which uses a Principal Mel-Spectrum Components and a combination of temporal pooling functions. 9http://majorminer.org/info/intro 10The Mood tag dataset is derived from mood related tags on last.fm 20 The Autotagging Ecosystem ID Participants F-Measure AUC-ROC PH2 Philippe Hamel 0.50 0.88 SSKSS1 K. Seyerlehner, M. Schedl, P. Knees, R. Sonnleitner, J. Schluter 0.49 0.89 BA2 Simon Bourguigne, Pablo Daniel Aguero 0.49 0.77 BA1 Simon Bourguigne, Pablo Daniel Aguero 0.49 0.78 GT2 George Tzanetakis 0.47 0.89 RWJ1 Jia-Min Ren, Ming-Ju Wu, Jyh-Shing Roger Jang 0.46 0.85 RWJ2 Jia-Min Ren, Ming-Ju Wu, Jyh-Shing Roger Jang 0.46 0.84 RJ1 Jia-Min Ren, Jyh-Shing Roger Jang 0.44 0.83 HN1 Jorge Herrera, Juhan Nam 0.34 0.70 Table 2.5: MIREX 2012 results for Audio Tag Classification - Major Miner task. ID Participants F-Measure AUC-ROC SSKSS1 K. Seyerlehner, M. Schedl, P. Knees, R. Sonnleitner, J. Schluter 0.49 0.87 PH2 Philippe Hamel 0.46 0.84 RWJ1 Jia-Min Ren, Ming-Ju Wu, Jyh-Shing Roger Jang 0.45 0.76 RJ1 Jia-Min Ren, Jyh-Shing Roger Jang 0.43 0.74 RWJ2 Jia-Min Ren, Ming-Ju Wu, Jyh-Shing Roger Jang 0.43 0.77 BA1 Simon Bourguigne, Pablo Daniel Aguero 0.42 0.78 BA2 Simon Bourguigne, Pablo Daniel Aguero 0.41 0.77 GT2 George Tzanetakis 0.37 0.86 HN1 Jorge Herrera, Juhan Nam 0.37 0.69 Table 2.6: MIREX 2012 results for Audio Tag Classification - Mood task. 21 Dataset Construction 4a_subtle_use_of_vocal_counterpoint 5a_subtle_use_of_vocal_harmony 6abstract_lyrics 7ambiguous_lyrics 8an_unintelligible_vocal_delivery 9angry_lyrics 10 clear_pronunciation 11 consistent_rhyme_patterns 12 explicit_lyrics 13 french_lyrics 14 funny_lyrics 15 great_lyrics 16 heartbreaking_lyrics 17 heavy_use_of_vocal_harmonies 18 humorous_lyrics 19 interweaving_vocal_harmony 20 narrative_lyrics 21 offensive_lyrics 22 paired_vocal_harmony 23 political_lyrics 24 romantic_lyrics 25 sad_lyrics 26 southern_rap 27 spoken_word 28 upbeat_lyrics 29 use_of_call-and-response_vocals 30 vocal_duets 31 vocal_harmonies 32 vocal_samples Listing 3.5: MSD24k tags related to vocals, but, for the reasons mentioned above, weren’t considered for the class Vocals. As already mentioned, this dataset is quite large compared to both the CAL500 and Magtag5k. In fact, after this initial separation into Vocals and Nonvocals, the Nonvocals class was with 20519 songs against 1178 in the Vocals. There were two reasons for reducing the Nonvocals class. Firstly, it was impossible to manually check all of the 20519 Nonvocals songs without distributing the work to some kind of community and that was not an option. Secondly, having so many Nonvocals songs wouldn’t be a balanced dataset, that, as mentioned in Chapter 3is of the utmost importance when defining a ground truth dataset. In order to reduce the number of Nonvocals songs, it was done an analysis of the occurrences of each tag in the first separation of the Vocals and Nonvocals classes, as presented in the table below. The ratio of the occurrence of each tag in the Nonvocals on this initial separation is also presented. Ratio(tagx) = number of occurrences of tagxin Nonvocals number of occurrences of tagxin Vocals +number of occurrences of tagxin Nonvocals That basically means that if the ratio was high, most of that tags occurrences was in the Nonvocals dataset, therefore, there was more confidence that particular tag would be a good choice 28 Dataset Construction for the Nonvocals class. Again, the whole dataset was simply too big for manual verification, so, just the tags that looked promising were picked, both taking into account the ratios and general musical knowledge. Tag N#Nonvocals #Vocals Ratio a_breathy_male_lead_vocalist 0 106 0.0 a_busy_bass_line 4 2 0.67 a_busy_horn_section 11 134 0.08 a_clear_focus_on_recording_studio_production 12 284 0.0 . . .. . .. . .. . . vocal_harmonies 0 240 0.0 vocal_samples 25 0 1.0 west_coast_rap_roots 1 0 1.0 western_swing 16 2 0.89 Table 3.4: Analysis via number of tag occurrences for each tag in the MSD24k dataset. For all the tags, the ones presented in Table 3.4 were picked. The selection column indicates the percentage of songs from that tag to be selected from the dataset. This was done empirically, but with the goal of trying to obtain a relatively balanced dataset. For example, since there was considerable more songs with tag electro (2862 songs) than with underground_hip_hop (69 songs), a lower percentage was used to avoid the issue of overfitting, namely 5% instead of 100%. Tag #Nonvocals #Vocals RatioHSelection underground_hip_hop 69 0 1.0 100% turntablism 242 0 1.0 100% new_age_instrumental 18 0 1.0 100% instrumental_hip_hop 90 0 1.0 100% trance 1338 26 0.98 5% techno 1918 38 0.98 5% jazz_fusion 504 18 0.97 5% drumnbass 54 2 0.96 100% electro 2862 130 0.96 5% industrial 969 46 0.95 5% classical 317 20 0.94 100% acoustic_guitar 351 24 0.94 100% Table 3.5: MSD24k selected tags to include in Nonvocals class and in which percentage. With the Nonvocals class now considerably reduced from 20519 songs to 1364, the manual verification process could take place. As expected, for the Nonvocals class, it was found a high number of False cases, since most of the songs selected for this class were a result of an educated 29 Dataset Construction guess. However, it is safe to say that the False cases would have been much higher if a random selection had been made instead. Vocals Nonvocals Totals Before Listening 1178 1372 2550 True 1152 506 1658 False 26 0 26 Dubious 5 0 5 After Listening 1147 532 1679 Table 3.6: Listening evolution in MSD24k. 3.4 Overview To conclude this chapter, the distribution of the Vocals and Nonvocals class per dataset in from which all experiment from Chapter 5were run are presented in Table 3.7 and Figure 3.1. The datasets are also made available at http://paginas.fe.up.pt/~ei08067/dokuwiki/ doku.php. Vocals Nonvocals Total Ratio 4 CAL500 444 58 502 7.66 Magtag5k 1627 724 2351 2.25 MSD24k 1147 532 1679 2.16 Total 3218 1314 4532 Table 3.7: Distribution of Vocals and Nonvocals per dataset and globally. Additionally, for a better visualization of the distribution of the original tags considering the three datasets for the Vocals and Nonvocals classes the following two tag clouds are presented Figure 3.2 and 3.3). Even empirically, it is clear how there are some tags that have a direct relationship with singing content i.e. classical and strings. 4Number of Vocals over number of Nonvocals. 30 Dataset Construction Figure 3.1: Distribution of Vocals and Nonvocals per dataset. Figure 3.2: Tag cloud for Vocals for all 3 datasets. Figure 3.3: Tag cloud for Nonvocals for all 3 datasets. 31 Dataset Construction 32 Chapter 4 Framework In this chapter the overall framework used for the experiments mentioned in Chapter 5is described. A schematic visualization of all the parts involved can be seen in Figure 4.1. Figure 4.1: Overview of the framework. 33 Framework 4.1 Feature Extraction For extracting the low-level features of the musical excerpts the Marsyas 1 ( M usic A nalysis, R etrieval and Sy nthesis for A udio S ignals) software package was used. It is an open source package that provide MIR researchers and enthusiasts with a general, extensible and flexible architecture that allows for easy experimentation and fast performance. In particular, the command line application provided with the Marysas framework bextract was used. The basic command was run for extracting the features: bextract -mfcc -zcrs -ctd -rlf -flx -ws 1024 -as 400 -sv -fe <dataset>.mf -w <dataset>.arff The output of this command is an ARFF 2 file, of which there is an example in Appendix D. A brief explanation of each of its flags is given. Features -mfcc for extracting MFCCs 0-12 (Section 2.2.6). -zcrs for extracting the Zero Crossing Rate (Section 2.2.2). -ctd for extracting the Centroid Power (Section 2.2.3). -rlf for extracting the Roll-off Power (Section 2.2.4). -flx for extracting the Flux Power (Section 2.2.5) Other -ws setting the analysis window to 1024 samples (Section 2.2.1.6). -as setting the accumulator size to 400 analysis windows (Section 2.2.1.6). -fe stands for feature extraction, meaning to only extract the features and not to train the classifier, which is done later in kea, another Marsyas application. -sv stands for single vector, meaning to turn on single vector feature extraction where one feature vector is extracted per file. Default -hs stands for hop analysis that is set by default to 512 samples (Section 2.2.1.6). -m stands for memory and sets the size of how many analysis windows make up a texture window (Section 2.2.1.6). Default value is 40. Marsyas extract all these features by applying a Hamming window (described in Section 2.2.1.4). Also, for each feature, there are four output values: the average of the averages of the texture windows, the average of the standard deviations of the texture windows, the standard deviation of the averages of the texture windows and the standard deviation of the standard deviations of the texture windows. 1http://marsyas.info/ 2 stands for Attribute-Relation File Format and was developed as part of the Machine Learning Project at the Department of Computer Science of The University of Waikato for use with the Weka - a machine learning software. 34 Framework 4.2 Learning Algorithm For the classification part of the framework, the Marsyas’s command line application kea was used. The following command was used for all the experiments, with variables train , test and affinities naturally varying according to experiment. kea -m tags -w <train>.arff -tw <test>.arff -pr <affinities> By default kea uses a soft margin non linear SVM with RBF kernel with parameters C=1.0 and γ=4 . Even though the application kea of the Marsyas framework is being used, the actual software package that runs the SVM routines is libsvm [ CL11 ], a popular SVM implementation package that serves as basis for many machine learning applications. 4.3 Evaluation For the evaluation of the proposed framework the three metrics most commonly seen in literature are used: Precision,Recall and F-Score. A cross validation within the same dataset (Same Dataset Experiment) and between datasets (Cross Dataset Experiment) is done. Finally, a filtering experiment is also conducted. 4.4 Data As [ MDLG11 ] clearly shows not using artist filtering preprocessing can result in over optimistic results that not clearly reflect the performance of a system. Therefore, artist filtering was used when considering the separation of the datasets into folds. Considering the size of the datasets and the number of tags per dataset (which was only two, Vocals and Nonvocals), it didn’t make sense to separate the datasets into many folds, for instance 10, a common value seen in autotagging algorithms with 10+ classes. Therefore, CAL500 being a dataset with only 58 Vocals songs was only divided into two folds. Magtag5k and MSD24k being larger in size and having a better distribution of tags were both divided into three. 35 Framework 36 Chapter 5 Experiments In this chapter, it will be described the experiments conducted to evaluate the system developed. 5.1 Binarization Methods The task of classifying a song as Vocals or Nonvocals is more formally called as a Binary Classification Problem. This means that every instance of the data (in this case, song excerpts) has to belong to one and only one class out of the two possible. In other words, every song in the datasets will be classified as either Vocals or Nonvocals, not both of them, not none of them. The predictions of a machine learning algorithm before being normalized (more commonly called affinities) are continuous values. Therefore, these values should be binarized so that a classification can in fact occur. For binarizing these continuous values, there are a few possibilities, such as using: • a threshold value: a predefined value below which all values to binarize (affinities) are considered to be part of one class whereas, all other ones above it, are considered to be part of the other class. For example, all affinities above 0.5 will be considered as 0 (not representative of a class) and all equal and above will be considered 1 (representative of that class). This method is also referred to as cutoff value. • a dynamic threshold value: very similar method to the previous one, with the particular difference thast the threshold value is set dynamically according to the distribution of the classes in the training set. That is, if the distribution in the training dataset is of, let’s say, 70% Vocals and 30% Nonvocals, then the threshold value will be set to guarantee that same distribution in the test set. • the maximum value: simply choosing the maximum affinity out of the possible classes as the predicted one. For example, if a song excerpt has an affinity of 0.3 for the class Vocals and 0.8 for the class Nonvocals, the class with the maximum affinity (in this case, the Nonvocals) will be the chosen one. 37 Experiments 1. Generate a random filterbank (considering the aforementioned constraints). 2. Apply the filterbank to the data. 3. Use the transformed signals in a way similar to the Same Dataset Experiment. 4. Repeat from 1. with the instances of the data that didn’t change classification when compared to its untransformed version. In other words, this experiment can be described as N Same Dataset experiments in which Nis the number of filters generated. Figure 5.2 shows the results of this experiment. Note that this experiment was only run on the subset of the data that was correctly predicted in the Same Dataset experiment, so, in practice, not the whole datasets is being tested. A summary of the values for the first and last iteration of each dataset is presented in Table 5.8. Figure 5.2: Evolution of global and per tag f-score (top) as well as number of songs that flip classification (bottom) by iteration number of random filter generated. CAL500 Magtag5k MSD24k Iteration 1 17 1 36 1 75 Excerpts not flipped 449 371 1857 540 1406 282 F-measure per tag 0.53 0.45 0.58 0.25 0.66 0.18 F-measure global 0.93 0.84 0.76 0.37 0.72 0.27 Table 5.8: Summary of Filters experiment results for first and last iteration. As a quick analysis of the graphs show, in all three datasets it is clear how there is a great number of songs that are flipping its original correct prediction, due to the fact of the filterbank being 44 Experiments applied. It is also clear how both the global and per tag f-score in all three datasets behave similarly, as expected, decreasing according to the increase in the amount of excerpts being misclassified. What this experiment clearly shows, is that even though apparently the audio file is not being changed too much, the system looses all its capability of effectively detecting the presence of vocals. 45 Experiments 46 Chapter 6 Listening Experiment This chapter describes both the design and findings of the experiment conducted with human candidates, in order to further explore the findings of the Filters Experiment described in Section 5.4. An extensive use of the words original and filtered is made in this particular chapter, and so a clear definition is necessary. •original excerpt: a music file directly taken from a publicly available dataset (for more details on the datasets used, see Chapter 3). For example, it can be assumed that this is the version one would get from ripping directly from the original CD recording. •filtered excerpt: an original excerpt to which a randomly generated filterbank has been applied. Please note that for each original excerpt there are several corresponding filtered versions. The script and general interface for the experiment can be found in the Annex. 6.1 Goals In Section 5.4, it was shown how iteratively applying a filterbank to excerpts in the datasets would change its classification, that is, excerpts that were classified as Vocals in the ground truth were changing to Nonvocals and vice-versa. Furthermore, the filters being applied weren’t significantly changing the sounds properties of the excerpts. This was clear evidence that supported the hypothesis of current state-of-the-art features used in Music Genre Classification not working as well as they were expected to within the domain of vocals identification. However, to fully validate this evidence, it was necessary to conduct an experiment with human subjects to evaluate if an human would exhibit the same behavior, or in other words, if an human would still be able to accurately identify the presence of vocals in the filtered excerpts. In addition, there was also interest in evaluating how different an human considered an original and its corresponding filtered version to be. The goals of the experiment can be summarized as such: 1. evaluate if an human can still detect the presence (or not) of vocals in the filtered excerpts; 47 Listening Experiment 2. evaluate to which extent an human ear can guess if an excerpt is original or filtered. 6.2 Data Selection When doing experiments with computers there is no problem using large amounts of data. However, the same isn’t true for experiments that involve human participation, since time available for participation is more of a constraint. For this reason, it was necessary to carefully select some excerpts that correctly represented the whole data, or in other words, to have a good sampling of the datasets considering the experiment to conduct. That said, the following guidelines were taking into account for the selection of the excerpts to include in the experiment: •have the same number of Vocals and Nonvocals excerpts; •have the same number of originals and filtered excerpts; • select the nmost “aggressive” filters for the filtered excerpts, with “aggressive” meaning the iteration in which most songs flip its classification; •have the same number of excerpts from Magtag5k and MSD24k.1. These guidelines resulted in the selection of 6 excerpts per class (Vocals and Nonvocals) per dataset (Magtag5k and MSD24k) per version (original and filtered), totaling 48 excerpts of 30 seconds each. The reason for the number 6 lies in the fact of designing a relatively short experiment, more precisely, around 10/15 minutes, since it was desirable that the candidates maintained their full attention throughout the whole experiment. As it will be clear in the next paragraphs, these 48 excerpts (of 30 seconds each) were divided into two different sets of 24, therefore making it a 12 minutes listening experiment for each of the candidates. The selected excerpts were picked randomly given the above mentioned constraints. However, they were all listened one-by-one prior to the realization of the experiment to make sure they were in good audio quality conditions. A complete list can be found in Appendix B. Please note that the Iteration column is associated with the random generated band-pass filter and is presented to show how the filtered excerpts are obtained from applying different filters (see Section 5.4 for more details). 6.3 Design Evaluating the first goal - evaluate if an human can still detect the presence (or not) of vocals in the filtered excerpts - was a relatively simple task. In fact, it was just a matter of mixing original with filtered excerpts and ask the candidate if he/she could detect the presence of vocals. On the other 1 The CAL500 dataset was not considered here, since its song clips are not of 30 seconds duration, but rather full songs. To keep the experiment as short as possible, this dataset wasn’t considered. This wasn’t seen as problem though, since there was no evidence that each dataset had its own intrinsic sound properties. The other two, Magtag5k and MSD24k, were however considered, but just one would most probably have yielded similar results. 48 Listening Experiment hand, for the second one - evaluate to which extent an human ear can guess if an excerpt is original or filtereda more careful analysis had to be done. For that, the following options were considered: 1. To show the same candidate the two versions of an excerpt, the original and the corresponding filtered one without him/her knowing which one is which and ask him to guess. Even though this might seem like a possible solution, it wasn’t. For instance, if the candidate was presented with a filtered excerpt first (even though he/she doesn’t know it) and after with the corresponding original one, he would most likely take the first excerpt as reference to answer, therefore not providing a totally uninformed answer, which would go against the purpose of the experiment. 2. Don’t show the same candidate the two versions of an excerpt to avoid the issue the previous option raises. Since the aim was to compare results between original and corresponding filtered excerpts without showing both versions to the same candidate to avoid informed answers, it was necessary to ask them to different candidates. A point can be made that this approach might not be correct, since every listener can have its own perception of the song. However, this was not posed as a problem, since identifying vocals is a relatively straightforward task and therefore such problem is not expected to happen. For the reasons mentioned above, the second option was chosen. This implied that the data collected for the experiment had to be divided in two different disjoint sets, so that no candidate was presented with the original and filtered excerpt of the same song, in order to avoid direct comparison. Also, to avoid any pattern that showing the same excerpts in a particular order could imply, a random ordering within the experiment set was made for each candidate. 6.4 Population Additionally, a few considerations normally done in listening experiments were also taking into account, such as: •only allowing the submission of an answer after the candidate has heard the full excerpt; •only possible to hear each excerpt once; • ask the candidate to setup their system sound level at the beginning of the experiment and leave it unaltered until its end, since altering it might have influence in timbre perception. (see Figure C.3). Taking into account the specificity of the experiment, which dealt a lot with music listening skills, it was necessary to ask some screening questions related to music knowledge, such as: 1. If the candidate had any prior formal training in music; 49 Listening Experiment 2. If the candidate was an avid music listener; 3. How the candidate would be listening to the experiment. Regarding the population of this experiment, a few numbers follows. The total number of candidates was of 154, with an age average of 24.8,91 being males and 62 females. 75 of the candidates say they had any formal training in music while 79 say they don’t. About being an avid listener,72 candidates say they are, 55 say they don’t and 27 position themselves in between. Finally, 101 of the candidates say they have no experience with sound engineering techniques, while 53 say they do. For a complete reference of the questions in the questionnaire, please refer to Figure C.2 as Annex. 6.5 Software For conducting this experiment, there was two options: (i) do it in-lab, that is bringing people to a controlled environment and conduct the experiment and (ii) distribute it over the Internet. Considering the implications an in-lab experiment would entail, such as getting the necessary number of candidates within the time frame expected, early-on it was decided for the distributed version. This implied designing a web page that would serve the purpose of the experiment. Considering the very specificity of the experiment itself a web page from scratch was developed. It uses the HTML5 audio capabilities to play audio in the browser. Figure 6.1 shows an overall view on how the web page works, with the yellow markers highlighting the process of choosing the candidate group in the experiment, according to how many completed versions there are already - naturally, the group with less completed answers at the moment, is the one chose for the current candidate. The web page is still accessible in http://paginas.fe.up.pt/~ei08067/exp/ . An example answer file is also provided in Appendix E. For a more visual representation of the layout of the pages, please refer to Appendix C. As can be seen in Figure 6.1, the web page communicates with a very simple REST API (written in PHP) that has only two possible function calls: NEW when: a new candidate hits the Next button after filling its personal details. sends: the personal details of the candidate. returns: the candidate number and group. does: creates an answer file in the server with the details of the candidate (personal, browser, date, group). 50 Listening Experiment ANSWER when: a candidate hits the Next button after hearing an excerpt. sends: candidate number and the answers to the current excerpt. returns: ok if successful. does: adds a line in the candidate’s answer file with current excerpt .wav filename and answers. The main reason for developing this kind of architecture, rather than an even simpler one, was due to the fact that it had to be possible for more than one candidate to be doing the experiment at the same time, hence the use of a candidate number id and the consequent API. Figure 6.1: Overview of Listening Experiment Page Architecture. 6.6 Results As mentioned in Section 6.1, the purpose of the experiment had two different distinct goals. For the first one, a more formal hypothesis testing was conducted, since it was of utmost importance to validate the results of the Filters Experiment described in Section 5.4. For the second goal, the results are present in a more intuitive way, also considering the more subjective nature of the question itself. For the experiment there was a 251 total number of candidates that started the experiment: 251, while 154 (77 for Experiment A and 77 for Experiment B) actually finishing the experiment. Overall, there was dropout rate of 38.6%, which considered the time it took to fully complete the experiment was understandable. 6.6.1 First Question - Detecting Vocals The first question of the listening experiment was the following: 51 Listening Experiment Can you hear a human voice in this excerpt? (consider singing, speaking, shouting, whistling, etc.) The type of experiment conducted for this particular question can be described as One-Factor Two-Levels Within-Subject design [Lud07]. Any experiment implies the description of its independent and dependent variables. In this one, the independent variable is filtering the excerpts, while the dependent variable is the distribution of the answers on the filtered excerpts. The factor of this experiment is applying a filter on an excerpt. And for that same factor, there are two possible levels: the original and the filtered. It’s called within-subjects (sometimes also referred to as repeated measures), since the same subjects are used in each level (original and filtered) of a given factor (the excerpts). The Control Group can be described as the distribution of the answers from original excerpts, while the Experiment Group is the distribution of the answers from the filtered excerpts. The claim, can then be described as the human ear perception of presence of vocals is not affected by most “aggressive” filters applied on the Filters Experiment (Section 5.4). As remainder, the filters used on Filter Experiment were a 96 channel equally spaced filterbank constrained with an abruption set to 0.9. When conducting hypothesis testing, both the null hypothesis and alternative hypothesis should be defined, in this case as follows: H0:Q1(o) = Q1(f) H1:Q1(o)6=Q1(f) However, there is a significant difference between this experiment formulation and a typical hypothesis testing. Normally, the interest is in proving that the alternative hypothesis should be true, considering there is significant evidence not to consider the null one. In this case, however, what is intended is to do the opposite, to show that spite of the large sampling data used, there is no evidence that shows the null hypothesis can be disregarded. The author is aware that is not viable to prove the null hypothesis, as [ Arb13 ] clearly explains. However, given the problem at hand no other formulation was possible. What will be done throughout this section is to present how in spite of the large sample data used, considering the problem, there no evidence to discard the null hypothesis. Although, this isn’t proving it it shows some clear evidence that for it to be false is very unlikely. Figure 6.2 shows the percentage of correct answers for question 1. According to the claim that the perception of vocals isn’t affected by the filters used, it was expected to see most of the bars with similar heights, which can be said to be the case. Figure 6.3 shows the same data but in difference of correct answers per excerpt. It is clear how most excerpts have similar responses, since most bars are close to the y axis. However, some excerpts originate not so consensus answers, such as excerpt number 9,11,14,21 and 24. A careful listen to this excerpt shows that there is some reason for this to happen. With no exception, all of this particular excerpts are somewhat of 52 Listening Experiment Figure 6.2: Percentage of correct answers per excerpt for question 1. more dubious nature, having only a very short vocal presence (less than one second, for example) or have a vocal presence just at the very end of the excerpt (last second). Still, a a paired t-test is performed (with a tail number of 2), which yields the following results: Degrees of freedom 23 T-value 0.5 P-value 0.6 Table 6.1: Results of a paired t-test on question 1. Results from Table 6.1, show that, at a significance level of α=0.05 , it is completely unfeasible to reject the null hypothesis, since pα . In other words, most hypothesis test try to reject the null hypothesis, what happens when p<α , which in this case it not true at all. Again, by not rejecting the null hypothesis, it is not being proved its truthfulness, but, considering the relatively large sampled data, it’s safe to say there is several evidence to consider it not false, at the very minimum. 53 Dataset Construction Log File f\jacob_heringman_and_catherine_king-alonso_mudarra_songs_and_solos-09-o_gelosia_de_amanti-0-29.mp3 f\satori-healing_sounds_of_tibet-01-moon_night-233-262.mp3 [SPEECH_AT_END] 1\jacob_heringman-holburns_passion-27-a_toy_lute-30-59.mp3 3\jacob_heringman-siena_lute_book-09-ricercata_mlb11_da_milano-88-117.mp3 6\doc_rossi-demarzi6_sonatas_for_cetra_o_kitara-11-sonata_iii_largo-146-175.mp3 8\justin_bianco-phoenix-03-unseen_facts-88-117.mp3 9\american_baroque-dances_and_suites_of_rameau_and_couperin-13-minuets_1_2_suite_from_les_fetes_dhebe_rameau-117-146.mp3 9\janine_johnson-german_keyboard_masters-05-auf_das_heilige_pfingstfest_pachelbel-88-117.mp3 b\hanneke_van_proosdij-harpsichord_suites_of_chambonnieres-18-suite_in_c_major__courante_iris-88-117.mp3 b\jacob_heringman-jane_pickeringes_lute_book-02-a_toye-0-29.mp3 c\o_fickle_fortune-a_celebration_of_robert_burns-19-set_of_jigs-204-233.mp3 d\daniel_ben_pienaar-book_2_cd1_welltempered_clavier-21-prelude_and_fugue_no__11_in_f_major_bwv_880_praeludium-146-175.mp3 f\heavy_mellow-acoustic_abstracts-05-midnight_chimes-146-175.mp3 f\jacob_heringman_and_catherine_king-alonso_mudarra_songs_and_solos-22-romanesca_o_guardame_las_vacas_4_course_guitar-59-88.mp3 f\magnaloops-electronica_loops_1-43-osxivilion554-0-29.mp3 [VOCALS_NOTVOCALS] 5\burnshee_thornside-rock_this_moon-08-miss_your_love_forever_featuring_lilling-0-29.mp3 [DUBIOUS] 2\magnatune_compilation-electronica-06-indidginus_dusty_lands-233-262.mp3 5\domased-selection-07-wild_ride-30-59.mp3 7\rocket_city_riot-pop_killer-03-feel_alive-175-204.mp3 a\jade_leary-the_lost_art_of_human_kindness-12-earth_beyond_a_finite_thought-494-523.mp3 b\magnatune_compilation-rock-11-cargo_cult_alchemy-146-175.mp3 7\wicked_boy-the_treatment-05-strange_days-88-117.mp3 9\artemis-orbits-06-subterranean_hidden_kisses_mix__hands_upon_black_earth-262-291.mp3 e\burning_babylon-stereo_mash_up-01-7_nine_skank-88-117.mp3 Listing A.1: Magtag5k log file 60 Appendix B Listening Experiment Data ID Dataset Class Iteration Artist Title 1 MSD24k Vocals 12 Hole Heaven Tonight 2 MSD24k Vocals 2 David Cassidy & The Partridge Family I Think I Love You 3 MSD24k Vocals 57 Rage Against The Machine Mic Check 4 MSD24k Vocals 6 JayMay Gray Or Blue 5 MSD24k Vocals 4 Albert Hammond Jr GfC 6 MSD24k Vocals 5 Black Kids Hurricane Jane 7 MSD24k Nonvocals 14 Rodrigo y Gabriela Hanuman 8 MSD24k Nonvocals 8 Secret Garden Song From A Secret Garden 9 MSD24k Nonvocals 22 Nicolay Fantastic 10 MSD24k Nonvocals 3 Deep Dish Deep Dish 11 MSD24k Nonvocals 16 Infected Mushroom Bombat 12 MSD24k Nonvocals 13 Supervielle Forma 13 Magtag5k Vocals 9 Briddes Roune Lutel Wot Hit Any Mon 14 Magtag5k Vocals 7 Indidginus Spiritual Spearmints 15 Magtag5k Vocals 15 Jacob Heringman And Catherine King Villancico Agora Viniesse Un Viento 16 Magtag5k Vocals 10 Mercy Machine Stark Love 17 Magtag5k Vocals 2 Jami Sieber In The Silence 18 Magtag5k Vocals 4 The Kokoon Face 19 Magtag5k Nonvocals 3 Apa Ya Apa Ya Pradha 20 Magtag5k Nonvocals 14 Jacob Heringman Waissel Polish Dance 21 Magtag5k Nonvocals 6 Justin Bianco Siren 22 Magtag5k Nonvocals 5 Jacob Heringman Newsidler Adieu Mes Amours 23 Magtag5k Nonvocals 11 Seth Carlin Sonata in Bb Kv 333 Allegretto Grazioso (Mozart) 24 Magtag5k Nonvocals 8 Ehren Starks Lines Build Walls Table B.1: Listening Experiment Data. 61 Listening Experiment Data 62 Appendix C Listening Experiment Interface Figure C.1: First page of the questionnaire. Description of the experiment. 63 Listening Experiment Interface Figure C.2: Second page of the questionnaire. Screening questions. 64 Listening Experiment Interface Figure C.3: Third page of the questionnaire. Sound Setup. 65 Listening Experiment Interface Figure C.4: The questions page of the questionnaire. 66 Appendix D Arff File Example % Created by Marsyas @relation cal12_mag123_test.arff @attribute Mean_Acc400_Mean_Mem40_ZeroCrossings_HopSize512_WinSize1024_AudioCh0 real @attribute Mean_Acc400_Mean_Mem40_Centroid_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Mean_Acc400_Mean_Mem40_Rolloff_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Mean_Acc400_Mean_Mem40_Flux_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Mean_Acc400_Mean_Mem40_MFCC0_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Mean_Acc400_Mean_Mem40_MFCC1_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Mean_Acc400_Mean_Mem40_MFCC2_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Mean_Acc400_Mean_Mem40_MFCC3_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Mean_Acc400_Mean_Mem40_MFCC4_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Mean_Acc400_Mean_Mem40_MFCC5_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Mean_Acc400_Mean_Mem40_MFCC6_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Mean_Acc400_Mean_Mem40_MFCC7_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Mean_Acc400_Mean_Mem40_MFCC8_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Mean_Acc400_Mean_Mem40_MFCC9_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Mean_Acc400_Mean_Mem40_MFCC10_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Mean_Acc400_Mean_Mem40_MFCC11_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Mean_Acc400_Mean_Mem40_MFCC12_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Mean_Acc400_Std_Mem40_ZeroCrossings_HopSize512_WinSize1024_AudioCh0 real @attribute Mean_Acc400_Std_Mem40_Centroid_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Mean_Acc400_Std_Mem40_Rolloff_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Mean_Acc400_Std_Mem40_Flux_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Mean_Acc400_Std_Mem40_MFCC0_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Mean_Acc400_Std_Mem40_MFCC1_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Mean_Acc400_Std_Mem40_MFCC2_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Mean_Acc400_Std_Mem40_MFCC3_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Mean_Acc400_Std_Mem40_MFCC4_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Mean_Acc400_Std_Mem40_MFCC5_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Mean_Acc400_Std_Mem40_MFCC6_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Mean_Acc400_Std_Mem40_MFCC7_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Mean_Acc400_Std_Mem40_MFCC8_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Mean_Acc400_Std_Mem40_MFCC9_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Mean_Acc400_Std_Mem40_MFCC10_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Mean_Acc400_Std_Mem40_MFCC11_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Mean_Acc400_Std_Mem40_MFCC12_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Std_Acc400_Mean_Mem40_ZeroCrossings_HopSize512_WinSize1024_AudioCh0 real @attribute Std_Acc400_Mean_Mem40_Centroid_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Std_Acc400_Mean_Mem40_Rolloff_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Std_Acc400_Mean_Mem40_Flux_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Std_Acc400_Mean_Mem40_MFCC0_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Std_Acc400_Mean_Mem40_MFCC1_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Std_Acc400_Mean_Mem40_MFCC2_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Std_Acc400_Mean_Mem40_MFCC3_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Std_Acc400_Mean_Mem40_MFCC4_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Std_Acc400_Mean_Mem40_MFCC5_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Std_Acc400_Mean_Mem40_MFCC6_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Std_Acc400_Mean_Mem40_MFCC7_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Std_Acc400_Mean_Mem40_MFCC8_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Std_Acc400_Mean_Mem40_MFCC9_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Std_Acc400_Mean_Mem40_MFCC10_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Std_Acc400_Mean_Mem40_MFCC11_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Std_Acc400_Mean_Mem40_MFCC12_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Std_Acc400_Std_Mem40_ZeroCrossings_HopSize512_WinSize1024_AudioCh0 real 67 Arff File Example @attribute Std_Acc400_Std_Mem40_Centroid_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Std_Acc400_Std_Mem40_Rolloff_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Std_Acc400_Std_Mem40_Flux_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Std_Acc400_Std_Mem40_MFCC0_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Std_Acc400_Std_Mem40_MFCC1_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Std_Acc400_Std_Mem40_MFCC2_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Std_Acc400_Std_Mem40_MFCC3_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Std_Acc400_Std_Mem40_MFCC4_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Std_Acc400_Std_Mem40_MFCC5_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Std_Acc400_Std_Mem40_MFCC6_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Std_Acc400_Std_Mem40_MFCC7_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Std_Acc400_Std_Mem40_MFCC8_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Std_Acc400_Std_Mem40_MFCC9_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Std_Acc400_Std_Mem40_MFCC10_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Std_Acc400_Std_Mem40_MFCC11_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute Std_Acc400_Std_Mem40_MFCC12_Power_powerFFT_WinHamming_HopSize512_WinSize1024_AudioCh0 real @attribute output {cal12_mag123_test} @data % filename ..\..\mag\wav\0\william_brooks-bitter_circus-01-the_gift-88-117.wav 0.074980,0.054144,0.125415,0.101078,-40.193601,2.705833,-0.543432,0.873664,-0. 135541,0.611070,0.261652,0.242124,0.340740,0.158596,0.092557,0.153308,-0.00456 2,0.018943,0.017169,0.040647,0.090665,3.545981,0.834907,0.625059,0.568093,0.45 7108,0.484083,0.509933,0.461722,0.426579,0.423766,0.465065,0.428401,0.388648,0 .019437,0.014858,0.034728,0.023312,6.402241,0.690896,0.510121,0.429690,0.21408 5,0.288941,0.376290,0.256485,0.247653,0.339819,0.148527,0.208551,0.188240,0.00 8423,0.007615,0.018011,0.018501,6.606448,0.466660,0.160579,0.210611,0.090238,0 .154525,0.165228,0.099030,0.100694,0.080730,0.124886,0.105281,0.074527,cal12_m ag123_test % filename ..\..\mag\wav\0\william_brooks-bitter_circus-02-try_it_like_this-88-117.wav 0.047659,0.036763,0.067530,0.128668,-40.828464,5.165160,-0.977301,0.722968,-0. 608061,0.712947,-0.248283,0.007119,-0.070354,-0.115457,-0.007304,-0.119530,0.2 38864,0.012734,0.010967,0.024505,0.106926,3.262384,0.937428,0.681022,0.489417, 0.495998,0.480388,0.500754,0.484698,0.449125,0.412358,0.441460,0.408414,0.3762 95,0.008472,0.006486,0.014740,0.031299,6.658873,1.116054,0.480703,0.276408,0.3 33904,0.380896,0.295072,0.362178,0.298321,0.297704,0.176098,0.221404,0.264504, 0.007017,0.006618,0.013151,0.019757,6.380625,0.619832,0.185431,0.107789,0.1481 80,0.114934,0.120978,0.095206,0.088286,0.109016,0.118492,0.083367,0.079322,cal 12_mag123_test % filename ..\..\mag\wav\0\william_brooks-bitter_circus-03-seven_promises-117-146.wav 0.076671,0.052370,0.118735,0.109734,-40.126877,2.623657,0.024332,0.728177,-0.3 35325,0.247121,0.260596,0.252637,0.264246,-0.323057,0.057755,0.390968,0.234352 ,0.019912,0.019929,0.038499,0.102009,3.330445,0.859786,0.647371,0.584332,0.496 104,0.473556,0.466057,0.432250,0.443897,0.436244,0.356939,0.481489,0.357760,0. 014252,0.010012,0.022072,0.024087,6.587749,0.604038,0.279371,0.231552,0.273156 ,0.121427,0.316747,0.158059,0.269846,0.246839,0.250262,0.259504,0.207816,0.008 504,0.006635,0.013040,0.015428,6.108352,0.323304,0.210572,0.135321,0.098274,0. 109326,0.100174,0.080558,0.124466,0.139885,0.067613,0.125634,0.067169,cal12_ma g123_test Listing D.1: Example .arff file 68 Appendix E Answer File Example 2013-06-04 21:17:27 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.4 (KHTML, like Gecko) Chrome /22.0.1229.79 Safari/537.4 A Rui Ribeiro,<email>,20,m,yes,yes,2,no 1,17.wav,yes,orig 2,23.wav,no,orig 3,10f.wav,no,orig 4,18f.wav,yes,orig 5,9.wav,no,orig 6,7.wav,yes,orig 7,15.wav,yes,orig 8,21.wav,yes,orig 9,14f.wav,no,orig 10,4f.wav,yes,orig 11,11.wav,cant,orig 12,22f.wav,yes,orig 13,24f.wav,no,orig 14,8f.wav,no,orig 15,12f.wav,no,filt 16,3.wav,yes,orig 17,1.wav,yes,orig 18,2f.wav,yes,orig 19,20f.wav,yes,orig 20,13.wav,no,orig 21,19.wav,yes,orig 22,6f.wav,yes,filt 23,16f.wav,yes,orig 24,5.wav,yes,orig Listing E.1: Example answer file 69