Towards Emotion Recognition: A Persistent Entropy Application

TOWARDS EMOTION RECOGNITION: A PERSISTENT ENTROPY APPLICATION A PREPRINT Rocio Gonzalez-Diaz Dept. of Applied Mathematics I University of Seville [email protected] Eduardo Paluzo-Hidalgo Dept. of Applied Mathematics I University of Seville [email protected] José F. Quesada Dept. of Computer Science and Artificial Intelligence University of Seville [email protected] November 27, 2018 ABSTRACT Emotion recognition and classification is a very active area of research. In this paper, we present a first approach to emotion classification using persistent entropy and support vector machines. A topology-based model is applied to obtain a single real number from each raw signal. These data are used as input of a support vector machine to classify signals into 8different emotions (calm, happy, sad, angry, fearful, disgust and surprised). Keywords Persistent Homology ·Persistent Entropy ·Emotion Recognition ·Support Vector Machine. 1 Introduction Emotion recognition is not a trivial task and different approaches have been explored so far (see for example [1]). Additionally, its applications are really important, such as gathering and processing satisfaction feedback in customers’ services, generating statistical studies over a population, using emotion recognition to improve spoken language understanding during a conversation. Furthermore, it can help in human interaction as in KRISTINA project1, where emotion recognition is applied in order to help the interaction between health professionals and migrated patients. Among the different theories about emotions proposed in the specialized literature, we follow the model described in [2] and [3], where a discrete theory of emotions is given, differentiating several basic groups of emotions (neutral, happy, sad and surprised) and organizing them in a spatial model. In [4] a review of different emotional speech recognition techniques can be consulted. Topological data analysis is a well substantiated field useful to extract information from data (see [5]). Concretely, a recent tool in this area called persistent entropy has been successfully applied to distinguish discrete piecewise-linear functions (see [6]). In this paper, persistent entropy is used to model arousal (i.e., emotional state) and emotion recognition as follows. First, speech signals are considered as piecewise linear functions. Second, persistent entropy is computed from the lower-star filtration obtained from these functions. This persistent entropy embedding can be considered as a summary of the features that appear in raw signals, as intensity and intonation. The stability theorem for persistent entropy computed from lower-star filtrations [6] guarantees right comparison between signals and robustness against noise. Finally, a support vector machine is used to classify emotions via persistent entropy values. As far as our knowledge, no topology approaches have been previously applied to emotion recognition. 1http://kristina-project.eu/en/ APREPRINT - NOVEMBER 27, 2018 This paper is organized as follows: Basic emotion theory, the notions of persistent homology and persistent entropy, and machine learning knowledge required for the model are introduced in Section 2. In Section 3, the methodology followed in the experiments is explained. Results obtained from different training approaches are shown in Section 4. Finally, Section 5 provides conclusions and future work ideas. 2 Background In this paper, different tools are mixed up in order to propose a unified and coherent framework for emotion classification. In this section, the basic concepts about acoustics, topology, machine learning and statistics are introduced. Acoustic and Psychoacoustic Features. Emotions constitute the main field largely studied by psychologists. Following [3], we consider that emotions can be modeled spatially in a circle, being arousal and valence their main characteristic features. Accordingly, prosodic attributes of speech [7] are strongly related with emotion recognition. This research area takes into account several features of speech, in conjunction with gesticulation of the speaker. Some of those features are: pitch signal, number of harmonics, vocal tract, and speech energy. Along this paper, just the physical features of the acoustic signal along with the processing results available from this signals (such as the contour of speech signal which is a feature affected by the arousal of the speaker), will be taken into account. The inclusion of visual features will be proposed in Section 5 as a natural continuation of this research. Sentences will be processed, assuming that certain attributes, like the fundamental frequency, intensity and duration, of a sound are meaningful for emotion production and recognition. These attributes are encapsulated under the notion of prosody. Depending on the prosodic pattern, a sentence can have very different emotional features. For example, happiness is linked usually with large fundamental frequency and, loudness, in contrast with sadness, normally related to the opposite. For further explanations about psychoacoustics, [8] can be consulted. In the literature, some emotion classification techniques have been proposed (see [9]). Some of them employ prosody contours information of speech in order to recognize emotions, as, for example: artificial neural networks, the multichannel hidden Markov model, and the mixture of hidden Markov models. For a further approximation to paralinguistic theory see [10]. Topology background. Topological data analysis (TDA) studies the shape of data. In our case, we apply topological data analysis tools to distinguish between piecewise linear function shapes. For an introduction to topological data analysis, [11] can be consulted. Persistent entropy is the main tool from TDA that will be used in this paper. It sums up persistent homology information which “measures" homological features of shapes and of functions. Informally, homology provides the number of n-dimensional holes, called the n-th Betti numbers and denoted by βn. Intuitively, β0is the number of connected components, β1the number of tunnels and β2the number of cavities. However, for dimensions higher than 2, we lose the intuition about what a hole is. Definition 1 (Betti number, informal, [12]) If Xis a topological space, then Hn(X)≃Zβnis called the n-th homology group of Xif the power βnis the number of independent n-dimensional ’holes’ in X. We call βnthe n-th Betti number of X. Finally, the homology of Xis defined as H(X) = {Hn(X)}∞ n=0. Observe that the concept of homology is not useful in practice. For example, suppose a dataset Vof 10 points sampling a circumference. We expect that H0(V)≃Zsince a circumference has one connected component. However, the exact 0-th homology of Vis Z10. Therefore, we need a tool to compute the homology of the underlying space sampled by a dataset. Following this idea, Edelsbrunner et al. [11] introduced the concept of persistent homology together with an efficient algorithm and its visualization as a persistence diagram. Carlsson et al. [13] reformulated and extended the initial definition and gave an equivalent visualization method called persistence barcodes. Given a dataset Vand a simplicial complex Kconstructed from it, persistent homology measures homology by a filtration during time, obtaining births and deaths of each homology class (’hole’). Consequently, those classes that persist are better candidates to be representatives of the homology of the underlying space. Definition 2 (Abstract simplicial complex) Let Vbe a finite set. A family Kof subsets of Vis an abstract simplicial complex if for every subsets σ∈Kand µ⊂V, we have that µ⊂σimplies µ∈K. A subset in Kof m+ 1 elements of Vis called a m-simplex and Vis called the set of vertices of K. 2 APREPRINT - NOVEMBER 27, 2018 Definition 3 (Filtration) Given a set Vand a simplicial complex Kconstructed from it, a filtration is a finite increasing sequence of simplicial complexes: ∅=K0⊂K1⊂K2⊂. . . ⊂Kn=K A particular filtration that will be used in this paper is the lower-star filtration. Definition 4 (Lower-star filtration [11]) Let Kbe a simplicial complex with real (distinct) values specified on the set Vof all the vertices in K. Since vertices have distinct function values, then they can be ordered incrementally: f(u1)< f(u2)< . . . < f(un). The lower star of uiis the subset of simplices of Kfor which uiis the vertex with maximum function value, Ki={σ∈K:for all vertex vof σ⇒f(v)≤f(ui)}. Once the lower-star filtration is obtained, persistent homology can be computed as follows. The inclusion Ki⊂Kj induces a homomorphism fi,j :H(Ki)→H(Kj)on homology. Its image is the persistent homology, letting βij pbe the number of n-dimensional ‘holes’ that are born at Kiand die entering Kj. During the computation of persistent homology along the filtration, an elder rule is applied. For example, when there are two connected components that get joined at some Kj, the older one (the one that was born earlier) remains, and the younger one dies. A persistence barcode is a representation of births and deaths of homology classes along time using bars. An example is shown in Fig. 1. Finally, once persistence barcodes are obtained, persistent entropy can be computed. Definition 5 (Persistent entropy [6]) Given a filtered simplicial complex {K(t) : t∈F}, and the corresponding persistence barcode B={ai= [xi, yi) : i∈I}, the persistent entropy Eof the filtered simplicial complex is calculated as follows: E=−X i∈I pilog(pi) where pi=li L, li=yi−xi, and L=Pi∈Ili. In the case of an interval with no death time, [xi,∞), the corresponding barcode [xi, m)will be considered, where m= max {F}+ 1. The robustness of persistent homology to noise is guaranteed thanks to the following result, letting a stable comparison between signals. Theorem 1 ([6]) Given two functions, f:V→Rand g:V→R, defined on a set of vertices Vof Rn, then for every ε > 0, there exists δ > 0such that ||f−g||∞≤δ⇒ |E(f)−E(g)| ≤ ε. Machine Learning Background. Machine learning techniques are nowadays widely applied to solve classification problems. A classification technique will use a ‘training’ dataset D={(~vi, ci)|~vi∈Rn, ci∈ {0, . . . , k}, i ∈ {1, . . . , m} } where {0, . . . , k}are the different possible classes. From this dataset, the classification algorithm will produce a classification model. This model can lately be applied to new inputs in order to predict the corresponding classes. There exist several classification techniques in machine learning. In our case, we focus our attention on support vector machine (see [14], [15], [16] and [17, Chapter 5]. A support vector machine is a supervised learning technique that construct a hyperplane, driven by a linear function b+Pm i=1 αi~vT i~vi, or a set of them that can be used to classify data. When this data is not linearly separable, a kernel trick is applied: the space is mapped to higher dimensions using a kernel function, k(~v,~v0) = φ(~v)T·φ(~v0). Therefore, a support vector machine just creates hyperplanes that work as decision boundaries for classification after applying a deformation of the dataset in order to get a linearly separable representation. Then, formally, a support vector machine within a kernel makes predictions using the following function: f(~v) = b+ m X i=1 αik(~v,~vi) 3 APREPRINT - NOVEMBER 27, 2018 A B C D E F A C E A B C D E F A C E A B C D E F A C E Table 1: Left: Lower-star filtration. Right: The associated persistence barcode of the corresponding lower-star filtration pictured on the left. Kernels Linear k(~v,~v0) = ~vT·~v0 Polynomial of degree d k(~v, ~v0) = (~vT·~v0+c)d Gaussian k(~u,~v) = N(~u −~v; 0, σ2~ I) 4 APREPRINT - NOVEMBER 27, 2018 where αis a vector of coefficients, kthe kernel and bis a bias term. Finally, the coefficients are chosen as a result of an optimization problem of the separation margin between classes. Different kernel-based functions can be used, for example: where N(~v;~µ, Σ) is the standard normal density. Performance Metrics. Basically, we are dealing with a classification problem. Therefore, our main metric will be the accuracy, considered as the percentage of well classified data in a dataset: Accuracy =m n where mis the number of well-classified data and nis the size of the full dataset used in the test. Statistical Tool. The correlation coefficient of two random variables is a measure of their linear dependence. One correlation coefficient largely known and applied is the Pearson’s correlation coefficient [18]: Pearson’s correlation coefficient ρ(A, B) = cov(A, B) σAσB where cov(A, B)is the covariance and σthe standard deviation. 3 Methodology As was previously anticipated, the shape of the wave of a speech signal can be meaningful to emotional speech recognition. Roughly speaking, we will compute persistent entropy to the lower-star filtration of the raw signal and then, we classify the signals by comparing these numbers using a support vector machine. Let us now explain in details the methodology applied in this paper: Step 1. Subsampling of the signal. The size of each signal is reduced in order to face the complexity of the persistent homology algorithm. Besides, every signal of the dataset needs to be subsampled into the same size in order to fulfill the assumptions of Theorem 1. For example, we subsampled the signal pictured in Fig. 1 from 196997 points to 10000. The subsampling process was done uniformly on the signal, maintaining its shape and main distribution of the spikes. Furthermore, the experiments of Section 4 were also done using the dataset without subsampling reaching similar results. Then, we could assert that this type of subsampling does not loose relevant information for this approach. Step 2. Introduction of imperceptible noise. Signals are slightly perturbed to fulfill the requirement of lower-start filtrations (see Definition 2): two points in the signal can not have the same height. Step 3. Persistence barcode computation. The lower-star filtration technique is applied to the signals generated in Step 2, obtaining the associated persistence barcode. For example, the barcode associated to the signal of Fig. 1 can be seen in Fig. 2. Step 4. Persistent entropy computation. Persistent entropy is computed applying the formula given in Definition 5 to the persistence barcodes obtained in Step 3. Step 5. Support vector machine classification. This step consists of the application of several support vector machines with different kernels in order to infer results and develop a classification predictor to emotions. The different possible kernels, previously introduced in the paper, are tested and the one with better accuracy is chosen. 4 Experiments The work-flow presented in the previous section was applied to the RAVDESS dataset [19]. This dataset is composed by 24 actors interpreting 60 audios each on different emotions and different intensity. Concretely, there are 4audios for the neutral emotion and 8audios for each of the seven remaining emotions. Consequently, there are 1440 different audios. In Fig. 3, a box-plot of the persistent entropy of the 1440 audios grouped by the different emotions can be seen. We can infer that persistent entropy values vary depending on both the emotion and the person. It seems that there 5 APREPRINT - NOVEMBER 27, 2018 Figure 1: Raw signal intensity graph of an angry emotion interpreted by the actor number 1 of the RAVDESS dataset. Figure 2: Barcode of the signal shown in Fig. 1. The horizontal axis represents time. Every horizontal (blue) line represents the life of a 0-dimensional homology class. 6 APREPRINT - NOVEMBER 27, 2018 Male actor Female actor Male actor 0.43 0.23 Female actor 0.23 0.49 Table 2: Mean values for the correlation coefficients of the entropy values grouped by sexes. exists characteristic personal values and the range of every emotion can be really wide. For example, the persistent entropy values of the audio number 20 in Fig. 3, that is an example of happiness, varies from 5.1713 to 0.6923 depending on the person. Besides, the existing overlapping between the boxes tells us that emotions can not be distinguished from the rest by just the persistent entropy values of every script as a feature. This failure approximation is illustrated and explained in Experiment 1. However, some emotions can be differentiated by pairs even with this ‘naive’ approximation. One thing that appealed our attention is the visual correlation that persistent entropy values tend to have per sexes as shown in Fig. 4 and Fig. 5. Even if the range is lower or higher depending on the person, in general, the peaks appear on the same places. To illustrate it, let us consider the correlation matrix between persistent homology values of the 60 audios grouped in the ones belonging to females and the ones belonging to males. We obtain that persistent entropy values are moderately correlated between same sex audios and badly correlated between different sexes (see Table 2). We think that it could be interesting the use of more sophisticated measures of similarity apart from correlation. Furthermore, correlation results give us clues to the need of developing emotion classification within the dataset separated by sexes to reach better classification accuracy. Besides, we consider that persistent entropy values could even be a nice approach to people identification and not just to emotion recognition. However, this approach is far from the scope of this paper and its preliminary nature. In all the following experiments we use as the classification technique a support vector machine with fold cross validation and the kernel that provides the better accuracy from the ones explained previously. The training dataset will be the 1440 persistent entropy values grouped by different ways trying to get the features needed to reach our goal. In the first experiment we try the brute force approach using every script as a point of the training dataset. Then, in the second experiment, every point correspond to an emotion within its 24 persistent entropy values by the 24 different actors. Finally, in the last experiment, the dataset is grouped by actors and emotions. Experiment 1: Each persistent entropy value will be a point of the training dataset. In this case, 20.3% of accuracy is reached within a linear kernel. Some conclusions can be pointed out from this failed approach: The emotion recognition problem is a multidimensional one, in the sense that a 1-dimensional embedding is not enough to an acceptable classification result. Furthermore, this was anticipated by the overlapping of the different boxes at the box-plot of persistent entropy values showed in Fig. 3. Besides, the non correlation between persistent entropy values per sexes is a matter not taken into account in this experiment. Experiment 2: Each point of the dataset is a vector of 24 features which correspond to the persistent entropy value of the same emotion interpreted by the 24 different actors. The dataset was separated in 40 points for training dataset and 20 points for test dataset and a gaussian kernel was used. Then, 92.5% of accuracy was reached on the training dataset and 90% on the test dataset. Furthermore, 96.66% accuracy was obtained on the full dataset. In our opinion, this experiment presents two main drawbacks. The first one is its difficult applicability as it needs 24 features of every emotion. However, withing long audio recordings, it could be cut into pieces and obtain enough features to classify. The other drawback is the small dataset we have for this experiment because of the way it has been grouped. Experiment 3: In this experiment, each point of the dataset consists of a vector of 8features, corresponding each feature to the persistent entropy value of the same emotion interpreted by the same actor. By this, the following accuracy Table 3 for classification by pair of emotions was obtained using a second degree polynomial kernel. Considering other results in the literature like [1] where 71% of accuracy was reached using Artificial Neural Networks, our results are really promising. However, we are still far from the 83% of accuracy reached in [20] using a multi-task hierarchical model. But we can say that, with just a first approximation, we could reach similar accuracy than those that already exist in the literature. Furthermore, as we are considering here just intensity and one type of filtration, only some features that characterize emotions are taken into account. Then, it gives us a nice starting point in order to improve the model by using different features of the signal and different filtrations. 5 Conclusions and future work A persistent entropy application has been developed in order to extract information from raw audio signals and solve a classification problem using support vector machine. Furthermore, a descriptive analysis of the computed persistent 7 APREPRINT - NOVEMBER 27, 2018 Figure 3: (90◦rotated figure) Horizontal axis represents the different 60 audios. Vertical axis represents persistent entropy. The big (red) rectangle clusters encloses persistent entropies of the audios per emotion (the respective emotion is indicated in the horizontal axis. The small (blue) rectangles are quartiles for the persistent entropy values. The vertical (blue) dashed lines mean the range of values of persistent entropy values. The (red) points are outliers. The horizontal (red) small lines are the mean persistent entropy value for the corresponding audio. 8 APREPRINT - NOVEMBER 27, 2018 Figure 4: Horizontal axis represents the different audios of actresses. Vertical axis represents persistent entropy value. The different persistent entropy values for the 60 audios of the same actress are connected by an straight line. We can see that shapes are correlated (see Table 2), showing that they tend to have the same peaks and downs. Figure 5: Horizontal axis represents the different audios of male actors. Vertical axis represents persistent entropy value. The different persistent entropy values for the 60 audios of the same actor are connected by an straight line. We can see that shapes are correlated (see Table 2), showing that they tend to have the same peaks and downs. Feelings Calm Happy Sad Angry Fearful Disgust Surprised Calm 77.1% 68.8% 81.2% 79.2% 72.9% 60.4% Happy 62.5% 64.6% 60.4% 58.3% 64.6% Sad 75% 62.5% 70.8% 60.4% Angry 68.8% 77.1% 70.8% Fearful 72.9% 72.9% Disgust 75% Surprised Table 3: Prediction accuracy from pair of emotions using different support vector machine within different kernels. 9