Análisis topológico de datos: aplicación al reconocimiento de emociones

Análisis topológico de datos: aplicación al reconocimiento de emociones Guillermo Aguirre Carrazana Análisis topológico de datos: aplicación al reconocimiento de emociones Guillermo Aguirre Carrazana Memoria presentada como parte de los requisitos para la obtención del título de Máster Universitario en Matemática Avanzada por la Universidad de Sevilla. Tutorizada por Prof. Rocío González Díaz Prof. Eduardo Paluzo Hidalgo Contents Abstract 1 Resumen 3 1 Introduction 5 2 Biometrics for emotion recognition 9 2.1 Emotions.................................. 10 2.1.1 Emotion modeling . . . . . . . . . . . . . . . . . . . . . . . . 11 2.2 Topology to model emotions . . . . . . . . . . . . . . . . . . . . . . . 12 2.3 State-of-the-art approaches for emotion recognition . . . . . . . . . . 13 3 An overview of tools from TDA 19 3.1 Simplicialcomplexes ........................... 19 3.2 From point cloud to simplicial complexes . . . . . . . . . . . . . . . . 21 3.2.1 Algebraic methods to compute filtrations . . . . . . . . . . . 23 3.2.2 Geometric methods . . . . . . . . . . . . . . . . . . . . . . . . 25 3.3 Homology ................................. 26 3.4 Persistenthomology............................ 27 ii análisis topológico de datos: aplicación al reconocimiento de emociones 3.4.1 Persistence diagrams . . . . . . . . . . . . . . . . . . . . . . . 30 3.4.2 Distances between persistence diagrams . . . . . . . . . . . . 31 3.4.3 Persistent entropy . . . . . . . . . . . . . . . . . . . . . . . . 33 3.4.4 Time-varying systems . . . . . . . . . . . . . . . . . . . . . . 35 3.5 Persistence time series . . . . . . . . . . . . . . . . . . . . . . . . . . 37 3.5.1 Dynamic time warping . . . . . . . . . . . . . . . . . . . . . . 37 3.5.2 Embedding delay parameter selection . . . . . . . . . . . . . 40 3.6 Machine learning background . . . . . . . . . . . . . . . . . . . . . . 40 3.6.1 Machine learning with KNN Algorithms . . . . . . . . . . . . 42 3.6.2 Performance metrics . . . . . . . . . . . . . . . . . . . . . . . 43 3.6.3 Statisticaltools .......................... 45 4 Topological audio-visual emotion recognition 47 4.1 A topological model for video signals . . . . . . . . . . . . . . . . . . 47 4.1.1 Filtration of the cell complex . . . . . . . . . . . . . . . . . . 49 4.1.2 Persistent homology and topological signature . . . . . . . . 49 4.2 A topological model for audio signals . . . . . . . . . . . . . . . . . . 50 4.2.1 Time series as point clouds . . . . . . . . . . . . . . . . . . . 54 4.2.2 From point clouds to persistence diagrams . . . . . . . . . . . 54 4.3 An audio-visual combination topological model . . . . . . . . . . . . 55 5 Experimentation 57 5.1 Datasetsdescription............................ 57 5.1.1 Video-only dataset . . . . . . . . . . . . . . . . . . . . . . . . 58 5.2 Experimentation on video-only dataset . . . . . . . . . . . . . . . . . 59 CONTENTS iii 5.2.1 Classifying emotions using the facial landmarks . . . . . . . . 60 5.3 Experimentation on audio dataset . . . . . . . . . . . . . . . . . . . . 63 5.3.1 Experiment 1: audio dataset . . . . . . . . . . . . . . . . . . . 64 5.3.2 Experiment 2: audio dataset . . . . . . . . . . . . . . . . . . . 69 5.3.3 Experiment 3: audio dataset . . . . . . . . . . . . . . . . . . . 70 5.4 Combination of datasets . . . . . . . . . . . . . . . . . . . . . . . . . 71 5.5 Comparison with current papers . . . . . . . . . . . . . . . . . . . . . 73 6 Design and Implementation 75 6.1 Analysis of the implementation of the algoritm for the video-only dataset ................................... 75 6.2 Analysis of the implementation of the algoritm for the audio dataset . 81 7 Conclusions and future works 87 Abstract Emotion recognition consists of a series of processes to detect human emotions from facial human expressions. Humans interact with each other primarily through speech, but also through body gestures to emphasize a certain part of the conversation and exhibit emotions. The automatic recognition of a person’s emotional state has become a very active research field that involves scientists specialized in different areas such as artificial intelligence, computer vision or psychology, among others. Our main objective in this work is to develop a novel approach, based on topological data analysis, for the recognition and classification of emotions that combines features extracted from videos and audios, in their respective spaces of representation, of people expressing emotions. 2 Biometrics for emotion recognition Biometrics are biological measurements or physical characteristics used to generate mathematical models of physical (e.g. hand geometry, iris, fingerprints, gait, and so on) and behavioral (e.g. signal, patterns) features to recognize patterns for identification. The biometric details that are different for each person are used as distinct biometric data. A person’s emotional state can influence concentration, task solving, and decisionmaking skills. Emotional communication occurs through non-verbal channels, like body movements, the tone and pitch of the voice, gestures displayed through body language, and physical distance between the peoples speaking. Expressing and recognizing emotions help us understand the intentions of others, build better relationships, avoid or resolve conflicts better, and move past difficult mental and physical states more easily. Facial expressions, eye contact, hand gestures, and other cues can transmit more information than just words or language. It can emphasize a message, distract and redirect your attention to highlight details, and influence the course of your actions. Among the non-verbal components that have emotional meaning, facial expressions are one of the main information channels in interpersonal communication. Then, it is natural that research devoted to facial emotion has been increasing attention over the last years with applications not only in cognitive sciences but also in artificial intelligence. The recognition of people and the identification of their corresponding emotions allow better interaction with computers, systems, and environments. Emotion recognition plays an important role in the prediction of social behavior [77], in emotional intelligence as an ability [78], and the experience of empathy [38]. 10 análisis topológico de datos: aplicación al reconocimiento de emociones Biometric patterns are categorized in physiological characteristics and behavioral characteristics. Whereas a biometric system based on physiological characteristics is more reliable, behavioral characteristics may be easier to integrate within certain specific applications. Examples of physiological characteristics include fingerprint, face recognition, palmprint, hand geometry, iris recognition, and retina. Behavioral characteristics are related to the pattern of behavior of a person including gait and voice. This chapter addresses aspects of human emotion from a biometric point of view for the classification of emotion tasks. Specifically, emotions will be seen as a hidden biometric related to physiological and behavioral characteristics. Section 2.1 focuses on explaining the biometric foundations of emotional state and the motivation of its use against other techniques. The three principal approaches for modeling emotions we use will be explained. The idea of why topology is a useful tool to model emotion will be explained in Section 2.2. Finally, the state-of-the-art of the main approaches for emotion recognition existing in the literature is detailed in Section 2.3. 2.1 Emotions Emotions are a vital part of humans, playing a valuable role in how a person perceives and understands the environment. Emotions can be seen as a kind of information to guide us to interact with the world. They can restrict or expand our behavior according to the situation. As emotions are so important, physiologists have formed several theories about how the emotions are generated and how important is the information they contain. Then, researchers have studied emotions and their theoretical development. The ever-growing theory raises several important questions that researchers must address as they bring emotion to their field [47]. For example: 1. Is it the physiological or the cognitive aspect of an emotional experience that primarily determine which emotion is being experienced? 2. Are emotions culturally specific or shared across cultures? 3. Are either emotions themselves or the causes that elicit them? To answer these questions, we can say that our emotional states are combinations of physiological arousal, psychological appraisal, and subjective experiences. We defined them as the components of one emotion. Two people who faced the same 2. biometrics for emotion recognition 11 conjuncture could have a different emotion. Over time, different theories of emotion have been developed to explain how the various components of emotion interact with each other. In the last decade, methods and algorithms have increased to facilitate emotion analysis; starting with manual methods using questionnaires made mostly by psychologists, continuing with complex methods involving computational algorithms. At present, emotion recognition through computers has a lot of application and it has been constantly developed by researchers. 2.1.1 Emotion modeling Summing up, researchers have distinguished three principal approaches for modeling emotions [46]: •Categorical approach: Based on the idea that only a small number of emotions exist, divided into six basic emotions: happiness, sadness, anger, fear, surprise, and disgust [48]. •Dimensional approach: Based on the idea that emotional states are not independent. On the contrary, they are related in a systematic way [48], covering the variability in three dimensions: 1. Valence: how positive or negative emotion is. 2. Arousal: how excited or apathetic an emotion is. 3. Dominance:the degree of power. •Appraisal approach: Based on the appraisal theory [25], it emphasizes the distinct components of emotions, and is often denominated componential view. This theory assumes that one emotion contains different emotional components including the subjective experience of the emotion itself, and it is given by a situational context. The occurrence of such emotional components can vary across different situations. For example, emotion based on a person’s experience, opportunities for action goals, motivation, feelings, and reactions. Emotions can be represented with a limited number of independent affective dimensions, modeled spatially in a circle, being arousal and valence their main characteristic features. See Figure 2.1 where the eight emotions considered in this thesis are placed regarding their arousal and valence. As we can see in Figure 2.1, the arousal 12 análisis topológico de datos: aplicación al reconocimiento de emociones dimension is related to activity in both our mind and body, indicating the intensity of feeling along a single dimension ranging from sleep to frantic excitement. It links to attributes such as stimulated-relaxed, excited-calm, and wide awake-sleepy to define arousal [6]. The organs are stimulated with perception and the influence of the stimulation is related with arousal dimension. For example, when people are stimulated, they become aroused. Thus, it indicates how calming, soothing or exciting this stimulus is. The valence is usually represented in the horizontal axis and indicates the degree of pleasantness. It also indicates the emotional value that is associated with a certain stimulus and it is used to categorize emotions. It refers to the positive and negative character of emotions or some of its aspects. Figure 2.1: The eight emotions considered in this thesis placed spatially depending on their arousal and valence. 2.2 Topology to model emotions Data analysis technologies including machine learning are based on statistical analysis that, until nowadays, has usually been the main technique of any research. Statistical analysis makes assumptions such as the data follows a normal distribution, so it is known that expected performance cannot be obtained if the data does not follow well-known distributions. 2. biometrics for emotion recognition 13 Topological Data Analysis (TDA) has emerged as an important approach for characterizing the behavior of datasets using techniques from topology. Tools from TDA, specifically persistent homology, allow assigning shape descriptors to large and noisy data across a range of spatial scales. Unlike deep learning, this technique does not rely on any label data training. TDA focuses on measuring topological summaries (e.g., connected components, loops, and voids) of the resulting patterns obtained from the data analyzed. Learning high-level features from emotional utterances and creating a hierarchical representation of the signal is our main purpose. To address the problem, we will use TDA as a data analysis method that can capture in detail features and information by focusing on the shape of the data without using any statistical technique. 2.3 State-of-the-art approaches for emotion recognition Emotion recognition has been an important topic for researchers where several machine learning techniques have been used. The three main theories that can be distinguished are discrete theory for categorical models, dimensional theory for dimensional models, and appraisal theory. Discrete models describe an emotional state as discrete labels using basic emotions such as: sad and happy. In this way, complex affective emotions cannot be expressed. Dimensional models consider an emotional state as a point in a continuous space modeling more complicated emotions. Typically, an emotional state is covered by three dimensions: arousal (level of affective activation), valence (pleasure), and dominance (a measure of power or control), these dimensions describe many emotional states. Appraisal theory [101] focuses on detailing the mental processes underlying the incitement of emotions. An emotional state is interpreted as a control set of stimulus evaluations. While emotions can be expressed in different ways, automatic recognition has mainly focused on facial expressions and speech [102]. There are a few articles about body gestures and posture on emotional recognition summarized in [91]. Nevertheless, the most intuitive model is to detect facial expressions and analyze the emotion later. This is very useful in the case of a short distance from the person. 14 análisis topológico de datos: aplicación al reconocimiento de emociones Recognizing emotion from facial expressions has several advantages (+) and disadvantages (-) such as: + There are many datasets available for facial expressions. + The tools created for other purposes on facial recognition can be used. + It is considered the natural way to identify the emotional states. - It is determinant to have a high quality of the video and a good segmentation. - Actors can fake the movements involved in facial emotions. - The information of the context is not provided, thus sometimes the outputs are misleading. Data is sequences of images together with audio signals that can represent the temporal characteristics of emotion. Usually, in the process, it is necessary to enhance the quality of the image, for example, by changing the contrast and brightness or removing the noise. In the literature, there are several algorithms to solve illumination problems. Besides, it is required to extract the face from the video to apply later a detection face algorithm, where the most famous is the Viola-Jones algorithm. One relevant analysis of this approach is found in [130]. Another step is to extract features from the video: geometric features, appearance features or a mix of geometric and appearance features are the principal ones. Geometric features focus on the relationship between facial components. Features based on the position and angle of landmark facial points [39] will be used in this work. The appearance features are extracted from the global face region or from the analysis of regions containing different types of information [118]. Principal component analysis, Gavor-wavelets features, and local binary pattern histogram are the principal methodologies implemented in time real for this approach. The hybrid features bring better results in certain cases; the strength of both approaches generates an approach with fewer weaknesses [56]. Another approach is to recognize emotions using text whose solution is useful in different fields as data mining, information filtering systems, human-computer interaction, and psychology. As we see in [2], the solution to this problem is based on identifying keywords and assigning emotion to a text selected from a set of predefined emotion labels. Emotion recognition in the text is one of the difficult tasks in natural language processing and is very important as the principal medium of humancomputer interactions through emails, chats, messages, forums, blogs, and other social 2. biometrics for emotion recognition 15 platforms. There are different levels of complexity. For example, one emotion can be expressed through the meaning of words and their relations or could be expressed by metaphors, irony, or sarcasm. Another important branch focuses on the study of some parameters from the audio signal, such as a voice tone or prosody, highlighting the verbal communication. The features extracted from audio signals are used as input to emotion classification algorithms. Recall that a speech signal is the principal way of communication between humans. There exist efficient methods based on speech for machines to recognize human voices. However, we are still far from having a natural interaction between humans and machines because understanding the emotional state of the speaker is still an ongoing goal. This fact has motivated a recent research field namely speech emotion recognition. It is very useful for application webs and computer tutorials where the response of these systems to the user depends on the detected emotion [109], such as a diagnostic tool for the therapist [34] used in call center applications and mobile communication [75]. Speech emotion recognition task is very challenging because it is not clear which features are more distinguishable between emotions. There also exist other problems like variability, introduced by the existence of different sentences, speakers, speaking styles, and rates. Another issue is how a certain emotion is expressed. Generally, it depends on the speaker, his culture, and the environment or if he/she is speaking in his/her mother tongue. Although there are few works about multi-lingual classification [53], most work has focused on mono-lingual classification assuming there is no cultural difference among speakers. Emotion does not have a common theoretical definition, people know the emotion when they feel them. Nevertheless, researchers were able to study and define different aspects of emotion. This approach has a variety of applications, and it can be redirected to the voice recognition field, call center, or customer services. The approach to design a system for emotion recognition based on speech primarily includes two phases known as the feature extraction phase and features classification phase. The election of features for speech representation is needed to design an appropriate classification scheme and to choice, a useful speech database for evaluating the algorithm developed. There are many reviews on speech emotion recognition such as [125, 32] surveying the speech features and the classification techniques used in this task. In the works of this area, it is common to divide the speech signal into frames, within each frame 16 análisis topológico de datos: aplicación al reconocimiento de emociones the signal is considered stationary. Later, local or global features are extracted from each frame. Local features, as pitch and energy, are extracted from each frame. Global features are computed as statistics from all speech features obtained. There are a lot of different opinions in the literature about which local or global features are more suitable for emotion recognition. Top papers addressing global features [114, 54] agreed that they are superior to local features in terms of classification accuracy and performance. However, researchers have asserted that global features are efficient only to distinguish between the emotion of high arousal versus lower arousal, failing on the classification of emotions with similar arousal such as, for example, anger versus joy. Another disadvantage is given by the loss of temporal information on the signal. Ayadi et. al. [32] exposed the idea that it is unreliable to use complex classifiers such as the hidden Markov model and the support vector machine with global features since the number of training vectors may not be sufficient for reliably estimating model parameters. A third approach for feature extraction is based on segmentation speech signals and later computation of a feature vector for each phoneme segmented, observing the variation in the spectral shapes of the same phoneme under different emotions [71]. The poor performance of phoneme segmentation algorithms is a problem in this approach. An alternative method is to analyze the voiced segment (caused by vibrations of the vocal cord and are oscillatory) rather than each phoneme since it is easier to implement and feasible. In the literature, many speech features have been applied. They can be grouped into four categories: 1. Continuous features: pitch, energy, formants. 2. Qualitative features: voice quality, harsh, tense, breathy. 3. Spectral features: ordinary linear predictor coefficients, short-time coherence method, least-squares modified Yule-Walker equations and others. 4. Features based on the Teager energy operator. On the categorical approach, the states are limited to a fixed number and could be hard to focus on a complex emotional state. Nevertheless, these types of emotions are included in the dimensional approach. Some researchers address the pros and cons of each feature but, until now, no one can identify which category is the best one. In the last years, speech signal processing has been engaged by deep learning to a significant degree achieving competitive results in several applications. For example, 2. biometrics for emotion recognition 17 Zhao in [134] constructed two convolutional neural networks and long short-term memory (CNN LSTM) networks: one 1D CNN LSTM network and one 2D CNN LSTM network, to learn local and global emotion-related features from speech and "logmel" spectrogram, respectively. Most of the existing approaches are designed for specific databases, that make the full solution to this problem still tough. While the system is trained on a particular database, it continues facing different issues as ethnicity, appearance, culture, sex, age contextual meaning of sentences, and noise of background in the signal. According to this, the algorithm implemented in [72] does not work well when dealing with the natural environment. To recognize emotional states in a natural environment is challenging due sometimes we have to manage large volumes of unsegmented, nonprototypical, and non-preselected data. A model that allows us to work with different kinds of information into it is essential. Then, to increase the performance of emotion recognition, there are lines of research mixing multimodal approaches in that audio feature with facial features and body gestures that overcome the performance of the mono-modal approaches. This problem has been approached, analyzing audio features such as spectral or voice quality and video features including local phase quantization. This problem has been approached, analyzing audio features such as spectral or voice quality and video features including local phase quantization. For example, Nicole et al. [90] developed a system that performs emotion recognition based on multi-modal features as facial appearance, head movements of speakers, and spectral features from audio combined via a regression classifier. Deep learning has been used in audio-visual emotion recognition to improve the linear relationship between the features by capturing complex non-linear feature interactions in multimodal data [64]. According to what is discussed in this chapter, Topology Data Analysis opens its door to set the problem of classifying emotional state. It will be necessary to find a topological signature for emotions. 24 análisis topológico de datos: aplicación al reconocimiento de emociones A paradigm on TDA is to create an estimation  𝑈𝜖(𝕏)of any underlying submanifold, ⊂ℝ𝑑, from which 𝕏is the sample and then consider its homology, through what is known as persistence homology or through Betti numbers as it is addressed (see [95]). Readers interested in homology theory may refer to [52, 50]. An important result in this area is the Nerve Lemma [9], which establishes that the Czech complex and the neighborhood set  𝑈𝜖(𝕏)are homotopically equivalent and, in particular, they have the same homology groups. However, since the definition of the Czech Complex is essentially combinatorial, the Czech complex is computationally more accessible and therefore more widely used in applications than the neighborhood set. The interest in working with the Czech complexes is since it is primarily a high-dimensional complex analogous to a geometric graph. For an extensive study on geometric random graphs, we can cite [98]. The Czech topology is closely related to the topology of the alpha complex. However, when the dimension is greater than three its calculation becomes impractical. Figure 3.5: The Czech complex  𝐶𝜖(𝕏)for 𝕏= {𝑥1, 𝑥2,⋯, 𝑥6}and 𝜖 > 0. The complex contains six vertices, two edges and one triangle. Many times, there is not feasible to compute the Czech complex and the alpha complex can be applied only on dimension three or less. Another method use to compute filtrations, closely related to the previous one and easier to compute, is explained below. Vietoris-Rips complexes The Vietoris-Rips complex was introduced by Vietoris in [126] to extend the simplicial homology to a theory of homology of spaces in more general metrics. Although they are generally not as fast as alpha complexes at low dimensions, its calculation can be efficient in high dimensions. 3. an overview of tools from tda 25 Definition 3.6 (Vietoris-Rips complex). Let 𝕏= {𝑥1, 𝑥2,⋯, 𝑥𝑛}be a collection of points in ℝ𝑑and let 𝜖 > 0. The complex  𝑅𝜖(𝕏)is defined as: 𝜎= {𝑥1, 𝑥2,⋯, 𝑥𝑘} ∈  𝑅𝜖(𝕏)←→ ||𝑥𝑖−𝑥𝑗||≤𝜖∀𝑖, 𝑗 ∈ {1,⋯, 𝑘}. The definitions of the Czech and Vietoris-Rips complexes are not limited only to the case of the Euclidean space, they can be defined for a set of points in any metric space [17]. They satisfy the following property that plays an important role in TDA. Lemma 3.1 ([26]).Let 𝕏be a finite set of points in ℝ𝑑and let 𝜖≥0. Then there is a chain of inclusion maps:  𝑅𝜖(𝕏)→ 𝐶√2𝜖(𝕏)→ 𝑅√2𝜖(𝕏). This means that a topological property that persists under the inclusion  𝑅𝜖(𝕏)→  𝑅𝜖′(𝕏)with 𝜖′≥√2𝜖is a topological feature of  𝐶𝜖′(𝕏). The main idea is that the information about the topological features which persist under the previous inclusion reveal more information than if the same information is considered separately. In [40], from a computational point of view, it is analyzed that the Vietoris-Rips complex is computationally less expensive than the corresponding Czech complex, despite having more simplices. An unsatisfactory aspect of the previous results is the dependence on a priory knowledge of 𝜖. One way to make a good choice of 𝜖is to consider homological invariant multi-scale that encode changes in the shape of homology when 𝜖changes. In [93], it is presented how to select the parameter 𝜖; for enough small 𝜖, the complex is a discrete set and for higher 𝜖, the complex has a high dimension. 3.2.2 Geometric methods A persistent problem in the methods presented above is that the simplices can have very high dimensions. Then, we address below some complexes exposed in [10, 70] that arise from computational geometry techniques to approach the problem. 26 análisis topológico de datos: aplicación al reconocimiento de emociones Delaunay complex To avoid the computational problems of Czech complexes and Vietoris-Rips complexes, it becomes necessary to limit the number of simplices in high dimensions. Delaunay complexes yield geometric tools to achieve this task and, nowadays, most of the simplicial complexes used are based on variations of Delaunay complexes. Definition 3.7 (Voronoi diagram and Delaunay complex). Let 𝑆be a set of points in ℝ𝑑. Define 𝑉𝑠as the set of points of ℝ𝑑that is closest to 𝑠∈𝑆than to any of the points of 𝑆. That is, for 𝑠∈𝑆, let us define: 𝑉𝑠= {𝑥∈ℝ𝑑|𝑑(𝑥, 𝑠)≤𝑑(𝑥, 𝑠′) ∀𝑠′∈𝑆}. The collection of the sets 𝑉𝑠is a cover for ℝ𝑑and it is called the Voronoi decomposition of ℝ𝑑concerning 𝑆. The nerve of this cover is called the Delaunay complex of 𝑆, denoted by 𝐷𝑒𝑙(𝑆;ℝ𝑑). The construction of this complex is costly in high dimensions, although efficient algorithms for the computation of Delaunay complexes for 𝑑= 2 and 𝑑= 3 have been developed. See [120] for more details on Voronoi diagrams and Delaunay complexes. 3.3 Homology The 𝑛-dimensional homology group of a topological space represents the 𝑛-dimen- sional holes of the space. Intuitively, a 0-dimensional hole is a connected component, a 1-dimensional hole is a tunnel and a 2-dimensional hole is a cavity. Higherdimensional homology groups do not have so clear intuition on the space but classical results ensure that 𝑛-dimensional homology groups are topological invariant, that is, they are invariant under homeomorphisms1. For each integer 𝑛 > 0, the 𝑛-dimensional homology group of a topological space can be computed as follows. Suppose we have a topological space structured as a cell complex, being an 𝑛-dimensional cell a topological space homeomorphic to an 𝑛-dimensional ball. A 0-dimensional ball is a point(vertex), a 1-dimensional curve (edge). A 2-dimensional ball is homeomorphic to a disk and so on. A cell complex 𝐾is then a collection of cells constructed inductively: 1A homeomorphism is a bicontinuous and bijective function between two topological spaces. 3. an overview of tools from tda 27 (1) The 0-skeleton 𝐾(0) of 𝐾(i.e., the set of 0-cells) are a set of points in an ambient 𝑑-dimensional space ℝ𝑑. (2) Inductively, form the 𝑛-skeleton 𝐾(𝑛)of 𝐾from the (𝑛− 1)-skeleton 𝐾(𝑛−1) by attaching 𝑛-cells via homeomorphisms. The boundary of an 𝑛-cell 𝜎can be informally defined as the (𝑛− 1)-cells in the (𝑛−1)-skeleton 𝐾(𝑛−1) of 𝐾used to attach the 𝑛-cell 𝜎. This way, if two cells intersect, they intersect at their boundaries. If a cell is in 𝐾, then the cells in its boundary are in 𝐾, where, for example, the boundary of an edge is its two endpoints (vertices). By abuse of language, we will say that a vertex is in the boundary of an 𝑛-cell if the vertex was one of the cells in 𝐾(𝑛−1) needed to attach the 𝑛-cell 𝜎. A maximal cell of 𝐾is a cell that is not in the boundary of any other cell of 𝐾. The dimension of 𝐾is the dimension of the cell of a higher dimension in 𝐾. From now on, we will assume that 𝐾has a finite number of cells. We can formally sum cells of same dimension 𝑛to obtain the 𝑛-dimensional chain group 𝐶𝑛(𝐾), for each 𝑛. Moreover, the boundary operator is extended to a linear map 𝜕𝑛from 𝐶𝑛(𝐾)to 𝐶𝑛−1(𝐾)in the obvious way: the boundary of an edge is the alternative sum of its endpoints, and so on. Since the boundary of the boundary of a cell is always zero, then the image 𝐵𝑛(𝐾)of 𝜕𝑛+1 is a subgroup of the kernel 𝑍𝑛(𝐾)of 𝜕𝑛and then, the 𝑛-dimensional homology group of 𝐾is the quotient group 𝐻𝑛(𝐾) = 𝐵𝑛(𝐾)∕𝑍𝑛(𝐾). An element 𝛼of 𝐻𝑛(𝐾)is called an 𝑛-dimensional homology class of 𝐾. The rank 𝛽𝑝of 𝐻𝑝(𝕏)is called the 𝑝-th Betti number where, in the case of the three first dimensions, an intuitive meaning exists since 𝛽0is the number of connected components, 𝛽1the number of holes and 𝛽2the number of voids [33]. Simplicial complexes can be seen as particular cases of cell complexes. Simplicial homology consists of the homology groups of simplicial complexes and it is used to summarise the global connectivity of a topological space 𝕏decomposed as a simplicial complex, associating it with a sequence of abelian groups. 3.4 Persistent homology The concept of homology is not useful in practice due to its lack of discrimination. This is why Edelsbrunner et al., in [30], introduced a more discriminating tool based 28 análisis topológico de datos: aplicación al reconocimiento de emociones Figure 3.6: The circumference has one connected component and one hole (𝛽0= 1, 𝛽1= 1). The sphere has one connected component and one void (𝛽0= 1, 𝛽1= 0, 𝛽2= 1). The torus has one connected component, two holes and one void (𝛽0= 1, 𝛽1= 2, 𝛽2= 1). on homology, called persistent homology, together with an efficient algorithm for computing it. Later, Carlsson et al., in [135], reformulated and extended the initial definition. Persistent homology applied to a filtration of a simplicial complex is in charge of keeping track of the moment 𝑖where a homology class is born and the moment 𝑗where the same class is died leading to topological descriptors as persistence diagrams or barcodes. Specifically, given a real-valued function 𝑓∶𝕏→ℝdefined for a triangulable subspace of ℝ𝐷, persistence homology describes the variation in the topology of the lower level sets 𝑓−1(−∞, 𝑡]when 𝑡increases from −∞ to +∞. For example considering the lower level sets 𝐿𝑡= {𝑥∈𝕏∶𝑓(𝑥)≤𝑡}, the index 𝑡means a scale parameter that leads the subspace filtration {𝐿𝑡}such that 𝐿𝑡⊆ 𝐿𝑠,∀𝑡≤𝑠. The filtration leads a family {𝐻(𝐿𝑡) ∶ 𝑡∈ℝ}of homology groups and the inclusion 𝐿𝑡→𝐿𝑠leads a family of homomorphisms 𝐻(𝐿𝑡)→𝐻(𝐿𝑠). Filtering functions Given a cell complex 𝐾, persistent homology measures homology by filtration during a time, obtaining births and deaths of each homology class. Concretely, recall that a filtration is a finite increasing sequence of simplicial complexes: ∅ = 𝐾0⊂ 𝐾1⊂ 𝐾2⊂⋯⊂ 𝐾𝑛=𝐾 Assuming that the vertices of a simplicial complex are points in the 𝑑-dimensional space ℝ𝑑, then a filtration can be derived from a real-valued map ℎon the set 𝑉of 3. an overview of tools from tda 29 Figure 3.7: Top: Example of a filtration obtained using the height function on its vertices. Bottom: Associated 1-dimensional persistent homology. vertices of 𝐾. In fact, there are several ways to obtain a filtration from ℎ. For example, the one used in [44] for developing an emotion recognition method was the lowerstar filtration. The one used in the first part of this work is different and defined as follows: For each maximal simplex 𝜎in 𝐾the ℎ-value of 𝜎is defined as: ℎ(𝜎) = 𝑚 ∑ 𝑖=1 ℎ(𝑣𝑖) assuming that {𝑣1,…, 𝑣𝑚}are the vertices in the boundary of 𝜎. Then, the filtration is: ∅ = 𝐾ℎ[0] ⊂ 𝐾ℎ[𝓁1]⊂ 𝐾ℎ[𝓁2]⊂⋯⊂ 𝐾ℎ[𝓁𝑛] = 𝐾. where 𝐾ℎ[𝓁𝑖]is composed by the maximal simplices of 𝐾with ℎ-value less or equal to 𝓁𝑖and all the simplices in their boundary, 0<𝓁1<𝓁2<⋯<𝓁𝑛and 𝓁𝑛being large enough such that ℎ−1(−∞,𝓁𝑛] = 𝐾. See Figure 3.7. Working over a field as the ground ring, persistent homology captures homology variations throughout a filtration. Assuming we have a filtration of the form: ∅ = 𝐾0⊂ 𝐾1⊂ 𝐾2⊂⋯⊂ 𝐾𝑛=𝐾, for each 𝑡and 𝑛in ℤ, the 𝑛-dimensional homology group 𝐻𝑛(𝐾𝑡)is a vector space. For every 𝑎≤𝑏and 𝑛, consider the linear maps 𝑣𝑎,𝑏 𝑛∶𝐻𝑛(𝐾𝑎)→𝐻𝑛(𝐾𝑏)induced by the inclusion 𝐾𝑎→ 𝐾𝑏. The family of vector spaces (𝐻𝑛(𝐾𝑡))𝑡∈ℤtogether with the linear maps (𝑣𝑎,𝑏 𝑛)𝑎≤𝑏is called the 𝑛-dimensional persistent homology of the given filtration. 30 análisis topológico de datos: aplicación al reconocimiento de emociones To compute the persistent homology of a cell complex, the notion of AT-model [45] is used in this work. We assume that the ground ring is ℤ2. See Algorithm 1 in page 50. 3.4.1 Persistence diagrams Based on what was stated in the previous section, considering that the homology class 𝛼was born in 𝐻𝑛(𝐾𝑡)and died in 𝐻𝑛(𝐾𝑠), set that 𝑏(𝛼) = 𝑡and 𝑑(𝛼) = 𝑠. Representing every class 𝛼by a point (𝑏(𝛼), 𝑑(𝛼)) results a multi-set of points in ℝ2 with the corresponding horizontal axis at the birth of the class and the vertical axis at death. The persistence of 𝛼is the difference 𝑝𝑒𝑟𝑠(𝛼) = 𝑑(𝛼)−𝑏(𝛼), where in a general context, the set of points with infinity persistence corresponds to points of the form (𝑡, +∞). Definition 3.8 (persistence diagram). A persistence diagram is a multiset of points in ℝ2with the diagonal Δ = {(𝑡, 𝑠) ∈ ℝ2|𝑡=𝑠}, where every point on the diagonal has infinite multiplicity. Each point in the persistence diagram corresponds to a pair of birth and death time of a homology class. That is, if a point (𝑡, 𝑠)belongs to a persistence diagram representing the persistent homology associated to a filtration, then there is a homology class 𝛼that was born in 𝐻𝑛(𝐾𝑡)and dies entering 𝐻𝑛(𝐾𝑠). When the pairs [𝑡, 𝑠]represent intervals in the real plane then the set of such intervals is called persistence barcode. Definition 3.9 (total persistence degree). The total persistence of degree 𝑝of a persistence diagram 𝑑is defined as: 𝑃 𝑒𝑟𝑠𝑝(𝑑) = ∑ 𝑥∈𝑑 (𝑝𝑒𝑟𝑠(𝑥))𝑝. Note that the persistence diagram is included in the half-plane above the diagonal Δsince deaths always occur after births. In [16], it is shown how these diagrams are well defined for any metric space and in particular for any compact metric space. The features of higher persistence are represented by the points furthest from the diagonal while nearby points to the diagonal may be interpreted as topological noise. Persistence diagrams are stable since a small change in the input function produces a small change in the diagram. There are different choices of metrics in the 3. an overview of tools from tda 31 space of persistence diagrams, analogous to the variety of metrics in the space of functions. In a general way, a metric distance is interpreted as a distance 𝐿𝑝in the function space of a discrete space. A natural family of metrics is discussed in [121]. Approaches to study the persistence diagrams from a probabilistic and statistical point of view are presented in [122, 103, 89, 18]. To use the persistence diagrams as a true statistic tool, natural questions about these summaries arise: •Is it possible to define probability measures on these summaries? •Can we establish relationships between the sampling distribution of the data and the distribution in the topological summary? •Can we compute the mean and variance? There is a variety of reasons for characterizing the statistical properties of diagrams. For example, given a point cloud 𝑆, it is advantageous to work with subsampling of the data which produce point clouds smaller and later to calculate the mean and variance of the persistence diagram set. In statistic terms, it consists of calculating a bootstrap estimation of a persistence diagram from data [18]. But these procedures require a good definition of the mean and variance. In the literature, some progress has been developed. In [84], Mileyko proved that the space of diagrams (𝐷𝑔𝑚𝑝, 𝑊𝑝) is a Polish space (with complete and separable metric), and therefore it is possible to define the Frechet mean. In particular, it has been proven that the Frechet measure of a finite set of persistence diagrams always exists but is not necessarily unique. 3.4.2 Distances between persistence diagrams Let us define several distances between persistence diagrams for pairwise comparison between them. Definition 3.10 (Wasserstein distance). The Wasserstein distance between two persistence diagrams, 𝑑1and 𝑑2is defined as: 𝑊𝑝(𝑑1, 𝑑2)=(𝑖𝑛𝑓𝛾∑ 𝑥∈𝑑1||𝑥−𝛾(𝑥)||𝑝 ∞) 1 𝑝 where 𝛾is a set of all bijections between 𝑑1and 𝑑2, it is not-empty due the diagonal. Particular case: For 𝑝= ∞, the bottleneck distance is defined as follows: 32 análisis topológico de datos: aplicación al reconocimiento de emociones 𝐵∞(𝑑1, 𝑑2) = inf 𝛾sup 𝑡∈𝑑1||𝑡−𝛾(𝑡)||∞. The empty persistence diagram is denoted by 𝑑∅, and is the diagram which includes only the diagonal; note that 𝑃 𝑒𝑟𝑠𝑝(𝑑)=2𝑝(𝑊𝑝(𝑑, 𝑑∅))𝑝for 𝑡∈𝑑. The metric 𝑊𝑝is used to define the following persistence diagram space [84]: 𝐷𝑝= {𝑑|𝑊𝑝(𝑑, 𝑑∅)<∞} = {𝑑|𝑃 𝑒𝑟𝑠𝑝(𝑑)<∞}; 𝑝≥1. The bottleneck distance is the most used metric to compare persistence diagrams due to its optimal properties and stability [29]. An intuitive idea of this distance is shown in Figure 3.8. Figure 3.8: Part a) shows two persistence diagrams, part b) shows the match given by choosing the bijection which minimises the sum of the distances. Note that one corner-point represented as triangle is matched with its projection on the diagonal. Now, let us describe several stability results on persistent homology, supporting the idea that an algorithm designed using persistent homology tools will produce “similar” outputs for “similar” inputs. Theorem 3.1 ([22]). Let 𝑓, 𝑔 ∶𝑋→ℝbe two tame 2Lipschitz function on a metric space 𝑋whose triangulation grows polynomially with constant exponent 𝑗≥1. Then, there are constants 𝑐 > 1and 𝑘≥𝑗such that the 𝑝-th Wasserstein distance between their corresponding persistence diagrams, denoted by 𝑑𝑓and 𝑑𝑔, satisfies: 𝑊𝑝(𝑑𝑓, 𝑑𝑔)≤𝑐||𝑓−𝑔||1−𝑘∕𝑝 ∞,∀𝑝≥𝑘. 2A function is tame if there is a finite number of elements in the set {𝐻𝑚(𝑓−1(−∞, 𝑎]} and such set consists of homology groups whose ranks are finite, for 𝑎∈ℝand 𝑚≥0being integer. 3. an overview of tools from tda 33 When 𝑝= ∞, the constant 𝑐is not longer necessary, obtaining the following most commonly used simplified version. Corollary 3.1 ([30] p. 183).Let 𝐾be a simplicial complex, and let 𝑓, 𝑔 ∶𝐾→ℝbe two monotonic functions If 𝑑𝑓and 𝑑𝑔denote the corresponding persistence diagrams obtained from 𝑓and 𝑔, then: 𝐵∞(𝑑𝑓, 𝑑𝑔)≤||𝑓−𝑔||∞. As a consequences of the previous theorem, the following theorem is guaranteed. Theorem 3.2 ([15]). Consider two finite metric spaces 𝑋and 𝑌. Let 𝑑𝑓, 𝑑𝑔be the two persistence diagrams obtained, respectively, from Vietoris-Rips filtration. Then, 𝐵∞(𝑑𝑓, 𝑑𝑔)≤𝐺𝐻(𝑋, 𝑌 ) where 𝐺𝐻 denotes the Gromov-Hausdorff distance. Due to these results, we can assert that stability results are simpler when using the bottleneck distance than when using the Wasserstein distance. Given the above intuitive idea and the notation, the stability of the persistence diagram is defined as: Theorem 3.3 (Stability Theorem [30]). For a triangulable space 𝑋, two continuous and tame functions 𝑓, 𝑔 ∶𝑋→ℝ, and any dimension 𝑝≥0, the bottleneck distance of the two 𝑝-dimensional persistence diagrams 𝑑𝑓and 𝑑𝑔is bounded by the distance between the functions: 𝐵∞(𝑑𝑓, 𝑑𝑔)≤||𝑓−𝑔||∞ The proof of this result is very technical and uses commutative diagrams of vector spaces of homology classes. It can be consulted in [21]. 3.4.3 Persistent entropy Due to the number of applications based on TDA, there are numerous available software packages to calculate and represent the persistent homology in almost all coding languages. A nice study of the performance of software packages is available in [93]. Persistence diagrams, persistence barcodes, and, more recently, persistence landscapes constitute a compact way to represent the information obtained from persistent homology. Although these topological summaries are metric spaces used to 40 análisis topológico de datos: aplicación al reconocimiento de emociones In order to analyze the evolution of a system in time as a collection of observations, windowing is defined as a uniform partition of the composition according to its subdivision in bars. 3.5.2 Embedding delay parameter selection Time-delay embedding is a uniform subsampling of the original time series data according to an embedding parameter 𝜏. It has been considered primarily in the context of analyzing dynamical system [119], where a time-delay embedding of time series data was used to recover the underlying dynamics of a system. In order to perform the phase space reconstruction, it is very important to select an appropriate pair of embedding dimension 𝑚and 𝜏. The precision of these parameters is directly related to the characteristics of the attractors in phase space reconstruction. Aiming to obtain these values, there are two different approaches to the literature. The first one is that 𝜏 and 𝑚are not correlated with each other; that is, both can be selected independently. Then, different criteria can be used for both parameters. For time delay 𝜏there are three different approaches: 1. Multiple auto-correlation and non-bias multiple auto-correlation [58]. 2. Series auto-correlation such as auto-correlation, mutual information [36], highorder correlations [1]. 3. Phase space extension, e.g., fill factor [12], wavering product [11], average displacement [104] and others. Another approach more practical consists of using that 𝑚and 𝜏are closely related since the time series data is not infinitely long and it is difficult to avoid noise. A lot of experiments done by researchers indicate that 𝑚and 𝜏are related with time window 𝑡𝑤= (𝑚− 1)𝜏for the reconstruction of the phase space. From a practical point of view, this is more reasonable and the combination algorithm for dimensional embedding and delay time have become an important line of analysis in the category of the chaotic time series analysis. 3.6 Machine learning background Machine learning usually refers to changes in systems that perform tasks associated with artificial intelligence. Such changes might be either improvement to already per- 3. an overview of tools from tda 41 forming systems or initial synthesis of new systems. Such tasks involve recognition, classification, planning, prediction, and others. Learning is a spectrum very wide. The field of machine learning has been divided into several subfields focusing on different types of learning tasks. In order to provide an idea of this wide field, we summarize the four parameters existing along which learning paradigms can be classified [113]. 1. Supervised versus unsupervised: Learning involves an interaction between learner and the environment, then we can divide it according to this kind of interaction. The first one is the difference between supervised and unsupervised learning. On supervised learning, we know the values of the function 𝑓for the 𝑛samples in the training set. Then, to assume that there is a hypothesis that matches the function for the elements of the training set involves that the hypothesis will be a good guess for 𝑓. In the unsupervised case, a training set of vectors without function values is provided. Then, the problem here is to partition the training set into subsets in some appropriate way. This setting is very useful in taxonomic problems in which it is desired to research ways to classify data into meaningful categories. 2. Active versus passive: An active learner interacts with the environment at training time for example performing experiments. On another side, the passive learner only observes the information provides by the environment. 3. Online versus batch learning protocol: The distinction between environments in which a learner has to respond online, along with her learning process and the environment in which the learner applies experience acquired after had the opportunity to process a large amount of data. 4. "Helpfulness of the teacher": When a scientist learns about the nature of the data, the environment plays the role of the teacher. To model such learning scenarios, we assert that the training data (or learning experience) is generated by some random process. Then, learning happens when the learner’s input is given by an adversarial called the teacher. Nowadays, machine learning techniques are widely applied to solve classification problems. A classification technique will use a training dataset: 𝐷={(𝑣𝑖, 𝑐𝑖)|𝑣𝑖∈ℝ𝑛, 𝑐𝑖∈ {0,…, 𝑘}, 𝑖 ∈ {1,…, 𝑚}} where the values 𝑐𝑖are the different 𝑘possibles classes that can exists, 𝑣𝑖vectors of dimension 𝑛that we are going to use in the classification. Through this dataset, the classification algorithm will produce a classification model. 42 análisis topológico de datos: aplicación al reconocimiento de emociones The different existing settings in machine learning offers very useful tools; one of these is the Support Vector Machine (SVM) paradigm for learning linear predictors in high dimensional feature spaces. The high dimensionality of the feature space establishes a challenge given by sample complexity and computational complexity. A SVM is a supervised learning technique that construct a hyperplane, driven by a linear function 𝑏+ 𝑚 ∑ 𝑖=1 𝛼𝑖𝑣𝑖 𝑇𝑣𝑖, or a set of them that can be used to classify data. When this data is not linearly separable, a kernel trick is applied. The space is mapped to higher dimensions using a kernel function, 𝑘(𝑣,  𝑣′) = 𝜙(𝑣)⋅𝜙(𝑣′). Therefore, a support vector machine just creates hyperplanes that work as decision boundaries for classification after applying a deformation of the dataset to get a linearly separable representation. Then, formally, a SVM within a kernel makes predictions using the following function: 𝑓(𝑣) = 𝑏+ 𝑚 ∑ 𝑖=1 𝛼𝑖𝑘(𝑣, 𝑣𝑖). SVM techniques were originally conceived as efficient methods for pattern recognition and classification. Researchers have used these settings in several linear and nonlinear applications such as speech recognition, image processing, array processing, communication systems. discriminant analysis, clustering, and many other applications. 3.6.1 Machine learning with KNN Algorithms What are KNN algorithms? K-Nearest-Neighbor (KNN) algorithms are supervised learning machine methods used in classification and regression task. It assumes that similar features exist in close proximity. In other words, similar things are near to each other. Its main goal is captures the idea of similarity, that can be defined as distance, proximity, or closeness. The idea is easy to use, and does not make assumptions about the data. Its accuracy 3. an overview of tools from tda 43 depends on the quality of the data, and in the algorithm should find the optimal number of nearest neighbors (K-value) [82]. How do KNN algorithms work? •INPUT: (𝑥1, 𝑦1),⋯,(𝑥𝑛, 𝑦𝑛) ∈ 𝑋x𝑌(having 𝑋as the input space and 𝑌their class labels) •Load the dataset. •Select the optimal value K of the neighbors. •Calculate 𝑑𝑖=𝑑(𝑥, 𝑥𝑖); 𝑖= 1,2,⋯, 𝑛, where 𝑑denotes the distance defined. •Sort in ascending order the collection of distances and indices. •Take the first 𝐾distances from this sorted list. •Get the labels of the selected 𝐾entries. •In case of regression, return the mean of the 𝐾labels, and in classification return the mode. •OUTPUT: 𝐷𝐾 𝑥assigning 𝑥to its most frequent class Choosing the right value for 𝐾 To select the right 𝐾for the specific dataset in analysis, the KNN algorithm runs several times with different values of 𝐾, and selects the 𝐾that reduces the number of errors found, while keeping the ability of the algorithm to accurately make predictions with data than it has not seen before. 3.6.2 Performance metrics After knowing how to implement a machine learning model and get outputs in form of a probability or a class, the first model is rarely the best one. Then, it is needed to evaluate the quality of our machine learning model in order to improve the model until it performs as best as it can. In the literature, different performance metrics have been used to evaluate these algorithms. Basically, in this work, we will focus on the classification problem. Thus, we will use classification performance metrics such as accuracy, the area under the curve, precision, etc. When conducting an experiment, the metric is chosen to evaluate the model is very important. Choosing the metric involves how the performance of the algorithm 44 análisis topológico de datos: aplicación al reconocimiento de emociones is measured and compared. Next, let us see the most used metrics and the metric chosen in this work. 1. Confusion matrix: It is a very intuitive metric and used to find the correctness and accuracy of the model. It is useful if we need to classify the data into two or more types of classes. This technique itself is not a performance measure, but almost all metrics are based on it. In this case, we will define confusion matrix as a table with two dimensions (“Actual” and “Predicted”), and sets of “classes” in both dimensions. Usually, each column of the matrix represents the instances of the predicted class by the model applied, while each row the instance of the actual class in other words the class observed. Then, diagonal elements indicate correct predictions, while the off-diagonals represent incorrect predictions. Actual Predicted Positive (1) Negative (0) Positive (1) TP FP Negative (0) FN TN The terms associated with the confusion matrix are: •True Positive (TP): the actual and predicted classes of the data point are true. •True Negative (TN): both classes are negative. •False Positive (TP): the model predicts incorrectly, the actual class is false and the prediction class is true. •False Negative (FN): the actual class of the data point is true, and the model predicts it is false. It depends on the problem and the context if we need to minimize the false negatives or positives. 2. Accuracy in the classification problem: it is considered as the percentage of well-classified data in a dataset: Accuracy = 𝑚 𝑛 where 𝑚is the number of well-classified data and 𝑛is the size of the full dataset used in the test. Accuracy is a good measure when the target variable classes in the data are nearly balanced. The “target variable” is the variable whose values are to be modeled and predicted by other variables (analogous to the dependent variable in linear regression). It should never be used if the target variable in 3. an overview of tools from tda 45 the data belong to one class. Let see one example, to understand better these terms. In our happy emotion detection task with 200 videos of people expressing emotions, but only 10 people express happiness. Let’s say our model is very bad and predicts every video of emotions as no expressing happiness. In doing so, it has classified those 190 non-happy emotions videos correctly and 10 happiness video as Non-happy. Then even though the model is terrible at predicting happy emotion, the accuracy of such a bad model is also 95%. 3. Precision: It is a measure that evaluates how close a measurement comes to another measurement. 4. Sensitivity: It measures how often the test captures the cases correctly. For example, positive results in the test for a person who has the disease. 5. Specificity: It measures the ability to correctly generate a negative result. For example, if a person does not have the condition that is being tested for it. 3.6.3 Statistical tools The correlation coefficient of two random variables is a measure of their linear dependence. One correlation coefficient largely known and applied is the Pearson correlation coefficient. 𝜌(𝐴, 𝐵) = cov(𝐴, 𝐵) 𝜎𝐴𝜎𝐵 where cov(𝐴, 𝐵)is the covariance and 𝜎the standard deviation. This chapter demonstrates, the all theory that proceeds of Topology Data Analysis is useful for developing algorithms based on it and get promising results in the task of emotion classification. We started defining simplicial complexes as a set of simplices that satisfies certain conditions. This concept defines an approximation to the underlying space 𝕏(point cloud). To complete this step, we studied the most common algebraic and geometric methods existing. The 𝑛-dimensional homology group of a topological space and the persistent homology to work with the lack of discrimination of the homology theory were introduced. Given a complex 𝐾the persistent homology measures homology by filtration during a time, so we explained some filtration techniques and the algorithm applied in this thesis to calculate it. To summarise the information, we faced the 46 análisis topológico de datos: aplicación al reconocimiento de emociones persistence diagrams techniques and their approaches to study from a probabilistic and statistical point. To get a description of the persistent homology as a number, the entropy persistent technique is tackled in this chapter. A time-varying system was defined and how to combine the techniques from TDA as persistent homology and time series. To calculate the distance between time-series under certain restrictions, we introduced the Dynamic time warping technique. For extracting topological information from time-series, we studied Takens’ embedding theorem and its conditions. By last, this chapter developed a background in Machine learning techniques and the different techniques that later we will apply in the experimentation chapter. Once the background needed to understand the rest of the thesis is given, let us explain, in the next chapter, the flow work composed by different algorithms that this thesis follows. 4 Topological audio-visual emotion recognition In this chapter, we develop an efficient method for emotion recognition, combining facial landmark points from video signal and voice from audio signal, using persistent homology tools and machine learning techniques. In order to get an efficient combination, we set a topological model to get competitive accuracy in classification task using video from emotions. Later, another topological model is built to face the classification task in audio signals of emotions. Finally, we combine both audio and video signals and propose a method that follows the strategies One-vs-Rest and One-vs-One for multi-class classification. 4.1 A topological model for video signals In this section, we introduce the construction of the cell complex 𝐾which represents the input video (i.e., the image sequence) with a person expressing emotions. For that, we will concatenate the topological information computed from the video signals mentioned in the section above. We start the procedure by extracting the landmark points on the face in each frame of the input video sequence as it is shown in Figure 4.1. With the intention of a fair comparison with other state-of-the-art methods, we will use the videos provided in the RAVDESS dataset [74]. Secondly, using the spatial positions of the landmark points in one frame, a cell complex, with vertices being the landmark points, is computed. In this thesis, this cell complex is computed from a Delaunay triangulation with vertices being the landmark points. 48 análisis topológico de datos: aplicación al reconocimiento de emociones Figure 4.1: Example of computation of the landmark points of a face in one frame of an image sequence obtained from a video of the RAVDESS dataset. Thirdly, landmark points with the same label in consecutive frames are joined by an edge. A 2-dimensional cell is obtained when the two endpoints of an edge are joined to the two endpoints of the corresponding edge in the neighbor frame. A 3- dimensional cell is obtained when the vertices of a triangle are joined with the vertices of the corresponding triangle in the neighbor frame. See Figure 4.2, where the cells of the 3-dimensional cell complex 𝐾are pictured all in the same frame. Figure 4.2: Cells of the 3-dimensional cell complex 𝐾obtained from an input video with set of vertices drawn in the same frame. 4. topological audio-visual emotion recognition 49 4.1.1 Filtration of the cell complex The next step in this process is to sort the cells of 𝐾in order to obtain a filtration. The partial ordering of these cells is dictated by a function 𝑓∶𝐾→ℝ, satisfying that if a cell 𝜎is a face of another cell 𝜎′in 𝐾then 𝑓(𝜎)≤𝑓(𝜎′)(i.e., 𝜎appears before or at the same time that 𝜎′) in the ordering. In this work, eight different filtrations (two horizontal, two vertical, and four obliques) are used to obtain eight different persistence barcodes similar to the method explained in [68, 69]. This way, all the small movements of the landmark points through the video sequence will be captured in the persistence barcodes. For each plane 𝜋, it defines the filter function 𝑓𝜋∶𝐾→ℝwhich assigns to each vertex of 𝐾its distance to the plane 𝜋, and to any other cell of 𝐾, the biggest distance of its vertices to 𝜋. Ordering its cells according to the values of the function, the filtration 𝐾𝜋for 𝐾associated with the plane 𝜋is computed. Lamar et. al. [68, 69] presented a version of the same procedure for gait recognition where a bunch of simplices is added to the filtration, all the simplices of 𝜕𝐾(𝐼)with equal distance to the reference plane. The method is robust to variation in the number of simplices, thanks to this way in which the different times define sets of simplices with different cardinalities, and it is robust to noise. 4.1.2 Persistent homology and topological signature The topological signature of the landmarkpoint face videos is obtained by computing the persistent entropy of each filtration. Let us notice that if 𝑝𝑗≤1, then 𝑙𝑜𝑔(𝑝𝑗)≤0 and the entropy of a persistence barcode is always positive. Intuitively, the entropy measures how different intervals of the barcodes are in length. The algorithm to compute the persistent homology is described in Algorithm 1. Intuitive idea of Algorithm 1: The output of Algorithm 1 is the persistence diagram and, for that, the filtration of the complex is represented in the following way: A vertex, where two or more line segments meet is defined by a numerical value. The edges and triangles are defined by the index position of the face which forms it in the filtration used. Let us describe an example of the complex formed by a triangle 1,2,3denoted by 123 and let us consider the filtration defined as: {1,2,3,12,13,23,123}. The index 3 56 análisis topológico de datos: aplicación al reconocimiento de emociones from the topological signature of the input video which consists of a vector in the 9-dimensional space ℝ9(see Figure 4.6). In the literature, there are different approaches to tackle the multi-classification problems. To split the multi-class classification dataset into multiple binary classification datasets and set a binary classification model on each we will follow the next two strategies: "One-vs-Rest" and "One-vs-One". Consult the next chapter to deepen in knowledge about these methods. In this chapter, we have explained the algorithms developed in this thesis with a theoretical focus detailing some parameters that in the implementation are necessary to research. The next chapter focuses on an aspect more practical of our workflow, with the results of the experimentation work. 5 Experimentation In this chapter, the experiments that support the effectiveness of the use of topological signatures on video and audio signals for the emotion recognition task are exposed. Hereafter, Section 5.1 addresses the database used in the experiments. The protocol for the experiments on the video-only dataset is explained in Section 5.2, and Section 5.3 focuses on the protocol for the experiments on the audio dataset. Finally, we explain the configuration followed for combining both datasets in Section 5.4. 5.1 Datasets description For experimentation, the Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS) is used [74]. This database contains the vocalization of two statements in a neutral North American accent by 24 professional actors (12 female, 12 male). Each expression is produced at two levels of emotional intensity (normal, strong), with an additional neutral expression. The intensity fulfils an important role in emotional theory [28, 107]. The strong intensity is useful when we are looking for clear emotional examples. However, the normal intensity is used if we are interested in providing classification for daily life [59]. The RAVDESS database [74] is based on multi-modals inputs: physiological sensors, video, depth sensors, and standard input devices, providing more information to the recognition process. Because of the information it provides ,it is widely used in emotion recognition. The RAVDESS database is composed of 7356 files. All actors produced 104 different vocalizations, consisting of 60 spoken expressions and 44 sung expressions. These vocalizations are available in three formats: audio-video, video-only, and audio-only. This produced 312 files per actor forming 7356 files with a total of 4320 speech record- 58 análisis topológico de datos: aplicación al reconocimiento de emociones ings and 3036 song recordings. For this thesis, we are going to work with the videoonly dataset and the audio-only dataset. Some examples of the RAVDESS database can be seen in Figure 5.1. 5.1.1 Video-only dataset The RAVDESS video dataset contains tracked facial landmark movements for all 2452 trials. The information offered includes facial landmark detection, head pose estimation, facial action unit recognition, and eye-gaze estimation. All the information is collected on 2452 CSV files. Each actor has 104 tracked trials (60 speech, 44 song). File naming convention Each of the RAVDESS files has a unique filename describing different features. The filename consists of a 7-part numerical identifier. In the next experiments, we are going to focus on the modality "video-only" represented by 02 in the first digits of the identifier. Then, in order, the identifiers are the following: •Modality (02 = video only) •Vocal channel (01 = speech) •Emotion (01 = neutral, 02 = calm, 03 = happy, 04 = sad, 05 = angry, 06 = fearful, 07 = disgust, 08 = surprised.) •Statement (01 = "Kids are talking by the door", 02 = "Dogs are sitting by the door"). •Repetition (01 = 1st repetition, 02 = 2nd repetition). •Actor (01 to 24) Odd numbered actors are male, even numbered actors are female. Filename example: 02-01-04-01-02-13.CSV Meaning the following: 02 Video-only 01 Speech 04 Sad 01 Statement "Kids are talking by the door." 02 Second Repetition 02 14th Actor Male, as the ID number is odd 5. experimentation 59 Figure 5.1: Example of the eight RAVDESS emotions of audio dataset (speech channel). 5.2 Experimentation on video-only dataset The first experimentation consists of the computation of a simplicial complex from the facial landmarks of nine frames of each video of the RAVDESS dataset. Here, we hypothesize that summarising an emotion video by a set of a few key-frames in terms of the variability of facial expressions will be enough to describe and efficiently learn the contained emotion. Then, we proceed with the application of Algorithm 1 in page 50 using eight different filter functions from which we can compute their persistent entropy values. More details on the implementation can be consulted in Section 6. 60 análisis topológico de datos: aplicación al reconocimiento de emociones We focused on the 60 speech videos collected in the RAVDESS dataset. The total tracked files used is 24 actors × 60 speech trials, i.e., a total of 1440 files. In the RAVDESS dataset, the tracking results are provided as individual CSV files with values separated by comma. The resolution of all input videos is 1280×720, the output units are in pixels and their range of values goes from (0,0) (top left corner) to (1280,720) (bottom right corner). Figure 5.2 shows the landmark points on a face. Figure 5.2: Location of 2D landmark points. 5.2.1 Classifying emotions using the facial landmarks This subsection focuses on the behavior of the 62 landmarks points of the face around the video from one emotion. The first task in the experimentation topic is which classifier to select. In the literature, several pattern classifiers are explored for developing speech systems like speech recognition, speaker recognition, emotion classification, speaker verification, and so on [27, 112]. Many researchers explored several classification methods, such as neural networks [133], gaussian mixture model [13], kernel regression [88], k-nearest neighbors [24] and support vector machines [73]. Each classifier has advantages and limitations over the others. In this thesis, after making some tests with different classifiers and according to the characteristic of our database, we decided to apply the k-nearest neighbor family of classifiers which typically have good predictive accuracy in low dimensions. Such classification uses feature similarity to predict the feature of new data points. 5. experimentation 61 A feature will be assigned to a new data point based on how closely it is from the points in the training dataset. It is possible to use different metrics to determine that distance. Given a set of points 𝑋∈ℝ𝑛and a distance function, the k-nearest neighbor (kNN) method finds the 𝑘closest points in our dataset to a query point or set of points. kNN-based algorithms are widely used as benchmark machine learning rules 3.6.1. The classifiers with better output for our data were Fine KNN, which distinguishes finely in detail between classes. The number of neighbors was set to 𝑘= 1, which means that the feature is simply assigned to the class on the single nearest neighbor. Evaluation Procedure 1. We need two kinds of attributes in the data: the columns associated with the features and one column with the label. 2. To understand the model performance, we divide the dataset into a training set and test set. 3. Then, we build a KNN classifier model for 𝐾= 1 using the library and functions explained on 6. 4. We estimate how accurately the classifier can predict the type of emotion. 5. Accuracy is computed by comparing actual test set values and predicted values. The dataset was randomly split in 20% of the observations for the test set, and 80% for the training set. In this experiment, each point of the dataset consists of a vector of eight features obtained from the eight filtrations applied to each video. These features are the persistent entropy values previously calculated in the algorithm. Table 5.1: Accuracy rates reached using the entropy values associated with one filtration Acurracy Filt1 Filt2 Filt3 Filt4 Filt5 Filt6 Filt7 Filt8 Training_Data 88,7% 81,3% 84,93% 89,7% 85,6% 87,3% 82,7% 85,4% Test_Data 86,2% 79,0% 83,5% 88,3% 84% 86,2% 81,1% 83,4% Total_Data 89,6% 80,7% 87,0% 89,9% 88,1% 86,1% 84,4% 87,0% The first experiment focuses on classify the data using only the persistent entropy values associated with one filtration. Since we defined eight different filtrations, we will have eight classifications associated with them. Table 5.1 offers the accuracy 62 análisis topológico de datos: aplicación al reconocimiento de emociones rates reached in both parts of the dataset, and the last row shows the accuracy of the trained model on the 100% of the dataset. The first direction Filt1, horizontal (axis 𝑋) going from bottom to top, and the fourth one Filt4, going orthogonal to the third direction of view (45 degrees with 𝑋 and 𝑌) reach better accuracy than others. However, the classification ranges in each filtration are quite similar with a mean of 86,6% in the analysis of the full dataset. Once this experiment is done, it allows to confirm that every filtration offers relevant information for the emotion classification. Then, a vector of eight dimension is built that groups each persistent entropy value associated to each filtration defined. Figure 5.3: Confusion Matrix. The second experiment on this dataset focuses on mixing the information of the eight filtrations. Then, the dataset comprises vectors of eight features with the same 5. experimentation 63 target (the type of emotion) as the first experiment. Figure 5.3 shows the confusion matrix plot associated to the training set. This matrix allows us to understand how the kNN Fine classifier performed in each class, identifying the areas where the classifier has performed poorly. The numbers mean emotions: 2 = calm, 3 = happy, 4 = sad, 5 = angry, 6 = fearful, 7 = disgust, 8 = surprised. As we can see, the happy emotion has a lower rate than the others. Then, movements and gestures that people make when expressing happiness in the video-dataset analyzed are less significant to characterize such emotion. Table 5.2: Accuracy rates reached on video signals. Emotions Training Data Test Data Full Data calm 98% 94% 99% happy 92% 90% 95% sad 98% 95% 99% angry 96% 93% 98% fearful 98% 96% 99% disgust 96% 94% 97% surprised 98% 97% 98% Table 5.3: General Accuracy rates reached on video signals. Total Training Dataset Test Dataset Full Dataset General Accuracy 96,57% 94,14% 97,85% On the training dataset, the accuracy was reached 96,4%, reflected on the matrix. On test dataset, 94,1% was reached. And, on the full dataset, 97,8% was achieved. These rates of accuracy allow us to claim that the topological information extracted from the action in the video signals is relevant and decisive to classify emotions. Table 5.2 and Table 5.3 shows a summary of this experiment with the accuracy achieved in the video-only dataset. 5.3 Experimentation on audio dataset The second experimentation is focused on analyzing raw signals of the audio dataset from RAVDESS. In this case of audio signals, the dataset is composed of 24 actors 64 análisis topológico de datos: aplicación al reconocimiento de emociones interpreting 60 audios of different emotions. Each person contains four audios representing neutral emotions and eight audios for each of the seven remaining emotions. 5.3.1 Experiment 1: audio dataset Figure 5.4 shows examples of the raw signal from each emotion. We can see the differences in intensity, amplitude, and information which contain each audio signal. For the first experiment, we follow the next idea: For each audio, we partition the raw signal into a series of segments and compute automatically the optimal delay coordinate embedding. Each stage is totally configurable, where the segment size, window size, delay parameters should be controlled. The choice of the embedding dimension 𝑚and time delay 𝜏determine the number of points in the point cloud and it is an important step in the algorithm (see subsection 5.3.1). For more information, Section 6 includes the explanation of the codes for the computation of these parameters. How to choose the delay and the embedding dimension optimal for this dataset? We do not know these parameters a priory. Consider two or three embedding plots should be the more realistic case, but we need to formalize this. Ín the literature, we found two measures that give an idea about which delay and dimension to choose. Next, we will see the explanation and reference of these measures. Choice of 𝜏and 𝑚 Mutual Information [127] is a measure of similarity between two labels of the same data, where we need to calculate the minimum 𝑥𝑚𝑖𝑛 and maximum 𝑥𝑚𝑎𝑥 of the timeseries analyzed. Later, the interval [𝑥𝑚𝑖𝑛, 𝑥𝑚𝑎𝑥]is divided into a greater number of pieces. 𝑈𝑘denotes the probability of one element of the time series is in the 𝑘-th piece and 𝑈ℎ,𝑘(𝜏)the probability that 𝑥𝑖is in the 𝑘-th piece when 𝑥𝑖+𝜏is in the 𝑘-th piece. Then the mutual information is formulated by: 𝐼(𝜏)=− 𝑁 ∑ ℎ=1 𝑁 ∑ 𝑘=1 𝑈ℎ,𝑘(𝜏)𝑙𝑜𝑔 𝑈ℎ,𝑘(𝜏) 𝑈ℎ𝑈𝑘 . The first minimum of 𝐼(𝜏)as a function of 𝜏gives the optimal delay, since there we obtain the largest information by adding 𝑥𝑖+𝜏. 5. experimentation 65 Figure 5.4: Examples of the raw signal from each emotion. 72 análisis topológico de datos: aplicación al reconocimiento de emociones on each [37]. There are two famous strategies for this approach: the One-vs-Rest and One-vs-One. One-vs-Rest: A heuristic method that involves dividing the multi-class dataset into multiple binary classification problems. Then, each binary classifier will be trained on each binary classification problem and finally, the predictions are made according to the most confident model [85]. One-vs-One: A heuristic method that involves dividing the datasets into one dataset for each class versus every other class. Then, this implies an increase in the number of datasets due we will have one for each pair of classes. If the number of classes is high, it is not recommendable to use this method in code efficiency [85]. The library Skelarn has implemented the One-vs-Rest and One-vs-One setups to make the multi-class classification. Both setups were tested for the combination model task and One-vs-Rest reached better results with respect to the One-vs-One approach, one example of its application will be explained in 6. Table 5.6: Accuracy rates reached by combining audio and video signals. Emotions Training Data Test Data Full Data calm 90,2% 86,5% 92% happy 89,4% 87,6% 90,8% sad 95% 93% 96% angry 90% 88% 92,4% fearful 93,3% 90,2% 95,3% disgust 89% 87,5% 92,6% surprised 88,5% 85,5% 90% Table 5.6 displays the accuracy for the combination using One-vs-Rest setting. The approach One-vs-Rest implemented for the mixed model of information makes the predictions according to the most confident model. We can see that the previous accuracy obtained in the video-only dataset is affected by the introduction of audio information, but this decrease in values of classification is not so higher, and we reached results promising and competitive in the area. In the training dataset, the accuracy was reached 90,7%; on test dataset, 88,3% was reached. And, on the full dataset, 92,4% was achieved. 5. experimentation 73 5.5 Comparison with current papers Lately, hybrid neural networks combining Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) have become the state-of-art for emotion recognition. As we argued in this thesis, audio-video emotion recognition is a challenging task. Deep learning has played an important role in most of the challenges that exist in the biometrics recognition task. Guo et al. [49] proposed an audiovisual-based hybrid network that combines the predictions of five models for emotion recognition in the wild. The overall accuracy of the proposed method achieves 55,61 and 51,15 classification accuracy on the audio and video-only dataset, respectively, while the challenge baseline is of accuracy 38,81. This paper uses a different database (AFEW database [66]) but with the same amount of emotions to analyze and with similar conditions. Moskovin et al. [87] developed a method based on convolutional and recurrent neural networks for real-time human emotion recognition by audio-visual data. Based on a series of experiments, the optimal hyper-parameters of the neural network, as well as parameters of data processing, were chosen. The results achieved for the emotion classification in working with the only audio channel is of 61%, with video channel 69%. To fuse the results of two neural networks is 73%, they showed the information only in the full dataset. They use the Acted Facial Expressions from the Wild Database [79]. Issa et al. [55] face only the audio emotion recognition task using a convolutional neural network. Their baseline model includes one-dimensional convolutional layers combined with dropout, batch normalization, and activation layer. Table 5.7 shows how our method behaves with respects to principal papers from state-of-art. The values showed are achieved on the training dataset to present the results as the papers compared. Table 5.7: Comparison of our method to start-of-the-art methods Paper Audio dataset Video-only dataset Audio-video dataset Guo et al. \[49] 55,61% 51,15% 38,81% Moskovin et al. \[87] 61% 69% 73% Issa et al. (Model E) [55] 86,1% Our method 80,48% 96,6% 90,7% 74 análisis topológico de datos: aplicación al reconocimiento de emociones We can conclude that our method increases the previous better accuracy reached in this area, except in the case of the audio signal, that we need to work on to increase our results. More information about the functions, library, software, and algorithms used in this thesis is explained in the next section. 6 Design and Implementation This chapter will explain the use of different programming languages and certain libraries applied in this thesis, as well as discuss the implementation of the complete system. The code can be consulted in https://github.com/Cimagroup/AudioVisual- EmotionRecognitionUsingTDA. 6.1 Analysis of the implementation of the algoritm for the video-only dataset According to the first issue of the thesis, the useful functions existing, and the previous works focused on this issue, we have chosen the language Matlab to code the first part. The speech videos collected in the RAVDESS dataset are provided as individual CSV files with coordinates of the facial landmark points separated by a comma. The first issue of the algorithm was to define a point cloud {(𝑥𝑖, 𝑦𝑖)} and apply the 2D Delaunay Triangulation, a function offered by Matlab. Each element in DT.ConnectivityList is a vertex ID and each rows represents a triangle in the triangulation. Then, we applied the function triangulation to create the 2D triangulation data in matrix format. 1000 DT = delaunayTriangulation(x,y) %Conne c tivity l i s t 1002 l i s t = DT . C on n e c t i v i t y L i s t # c r e a t e s a 2D or 3D t r i a n g u l a t i o n r e p r e s e n t a t i o n using the t r i a n g u l a t i o n c o n n e c t i v i t y l i s t DT and the p oi n ts in matrix P . 1004 Delaunay_base = t r i a n g u l a t i o n ( l i s t , DT . po in ts ) Listing 6.1: Delaunay Triangulation. 76 análisis topológico de datos: aplicación al reconocimiento de emociones Now that our data is represented using triangulation, we can perform topological and geometric queries. The algorithm uses this representation to built the topological complex. It takes the landmark points with the same label in consecutive frames and then joined by an edge (worked with 9frames). The next function defined in the thesis calculates the eight filtration of the cell complexes obtained from the four view directions (2by each direction.) 1000 function f i l t e r s = f i l t r a t i o n s ( complex ) 1002 pts_central =[]; n= s i z e ( complex . pts , 1 ) ; 1004 %n= max ( complex . cuad ( : , 4 ) ) fo r i =1: len ( complex ) 1006 pts_central(i ,:) = sum ( complex . pts ( complex . cuad ( i , : ) , : ) ) / 4 ; end 1008 maxP = max (max (abs( complex . pts ) ) ) + 10 0 00; 1010 %sort triangles 1012 [X , indX ] = sortrows ( p t s _ c e n t r a l ) ; %ordering the p oin ts by the elements of the f i r s t column 1014 l i s t P t s Y ( : , 1 ) = p t s _ c e n t r a l ( : , 2 ) ; %r e v e rs e the order of x and y 1016 l i s t P t s Y ( : , 2 ) = p t s _ c e n t r a l ( : , 1 ) ; 1018 [X , indX ] = sortrows ( p t s _ c e n t r a l ) ; 1020 [Y , indY ] = sortrows ( l i s t P t s Y ) ; 1022 PtsDistXY = arrayfun (@( x , y ) ( abs ( x+y ) ) , p t s _ c e n t r a l ( : , 1 ) , p t s _ c e n t r a l ( : , 2 ) ) ; 1024 tem = PtsDistXY ; tem ( : , 2 : 3 ) = p t s _ c e n t r a l ; 1026 [XY , indXY ] = sortrows ( tem ) ; 1028 PtsDistYX = arrayfun (@( x , y ) ( abs( x−y + maxP ) / sqrt ( 2 ) ) , pts_central (: ,1) ,pts_central (: ,2) ); 1030 tem = PtsDistYX ; tem ( : , 2 : 3 ) = p t s _ c e n t r a l ; 1032 [YX , indYX ] = sortrows ( tem ) ; 1034 6. design and implementation 77 complexX = complex . cuad ( indX , : ) ; 1036 subcomplex = complexX ; %to i n v e r t order ind = [ s i z e ( subcomplex , 1 ) : −1:1]; 1038 subcomplex = subcomplex(ind , : ) ; complexX_inv = subcomplex ; 1040 complexY = complex . cuad ( indY , : ) ; 1042 subcomplex = complexY ; ind = [ s i z e ( subcomplex , 1 ) : −1:1]; 1044 subcomplex = subcomplex(ind , : ) ; complexY_inv = subcomplex ; 1046 complexXY = complex . cuad ( indXY , : ) ; 1048 subcomplex = complexXY ; ind = [ s i z e ( subcomplex , 1 ) : −1:1]; 1050 subcomplex = subcomplex(ind , : ) ; complexXY_inv = subcomplex ; 1052 complexYX = complex . cuad ( indYX , : ) ; 1054 subcomplex = complexYX ; %r ev er s e order ind = [ s i z e ( subcomplex , 1 ) : −1:1]; 1056 subcomplex = subcomplex(ind , : ) ; complexYX_inv = subcomplex ; 1058 f i l t e r s . complexX = unique ( complexX , ’ rows ’ ,’stable ’) ; 1060 f i l t e r s . complexY = unique ( complexY , ’ rows ’ ,’stable ’) ; f i l t e r s . complexXY = unique ( complexXY , ’ rows ’ ,’stable ’) ; 1062 f i l t e r s . complexYX = unique ( complexYX , ’ rows ’ ,’stable ’) ; 1064 f i l t e r s . complexX_inv = unique ( complexX_inv , ’ rows ’ ,’ s t a b l e ’ ) ; f i l t e r s . complexY_inv = unique ( complexY_inv , ’ rows ’ ,’ s t a b l e ’ ) ; 1066 f i l t e r s . complexXY_inv = unique ( complexXY_inv , ’ rows ’ ,’ s t a b l e ’ ) ; f i l t e r s . complexYX_inv = unique ( complexYX_inv , ’ rows ’ ,’ s t a b l e ’ ) ; 1068 end The next function is defined with the aim to split the cell complex obtained to see the features which born and died 2-cell on the complex. Later, the components which survived get the index where the square is. 1000 function[ index_cuad , complejo ] = complex_wtsquare ( complejo ) 1002 index_cuad = [ ] ; 1004 fo r i =1: length( complejo) 78 análisis topológico de datos: aplicación al reconocimiento de emociones i f numel ( complejo { i } ) ==4 1006 index_cuad = [ index_cuad i ] ; end 1008 end complejo ( index_cuad ) = [ ] ; After we get the eight filtrations in each direction defined, we applied the next function to convert the complex in a matrix way. Once we had the matrix, we apply the Incremental Algorithm described in the previous section (see Algorithm 1 in page 50) to get the persistence intervals. The interval who has death equal to zero means that never died. 1000 function matrix = complex2matrix ( complex ) 1002 s = c e l l f u n (@( x ) s i z e ( x , 2 ) , complex ) ; 1004 dim = unique ( s ) ; matrix = ones ( s i z e ( s , 2 ) , 3 ) ∗−1; 1006 fo r i =1: s i z e ( complex , 2 ) 1008 i f ( s ( i ) ==1) matrix ( i , 1 ) = complex { i } ; 1010 end v = complex { i } ; 1012 f =0; i f (s i z e ( v , 2 ) ==2) 1014 f = (~ isempty (fi nd ( v ==1) ) && ~ isempty (f i nd ( v ==37) ) ) ; end i f ( s ( i ) ==2) 1016 v = complex { i } ; in1 = 0 ; inn2 =0; 1018 con = 0 ; fo r j =i −1: −1:1 1020 i f (s i z e ( complex { j } , 2 ) ==1 && v ( 1 , 1 ) ==complex { j } ) matrix ( i , 1 ) = j −1; 1022 con = con + 1 ; end 1024 i f (s i z e ( complex { j } , 2 ) ==1 && v ( 1 , 2 ) ==complex { j } ) matrix ( i , 2 ) = j −1; 1026 con = con + 1 ; end 1028 i f ( con ==2) con = 0 ; 1030 break ; 6. design and implementation 79 end 1032 end 1034 end 1036 i f ( s ( i ) ==3) v = complex { i } ; 1038 in1 = 0 ; inn2 =0; con = 0 ; 1040 fo r j =i −1: −1:1 i f (s i z e ( complex { j } , 2 ) ==2 && ( ~ isempty (fi nd ( v ( 1 , 1 ) ==complex { j } ) ) && ~ isempty (fi nd ( v ( 1 , 2 ) ==complex { j } ) ) ) ) 1042 matrix ( i , 1 ) = j −1; con = con + 1 ; 1044 end i f (s i z e ( complex { j } , 2 ) ==2 && ( ~ isempty (fi nd ( v ( 1 , 1 ) ==complex { j } ) ) && ~ isempty (fi nd ( v ( 1 , 3 ) ==complex { j } ) ) ) ) 1046 matrix ( i , 2 ) = j −1; con = con + 1 ; 1048 end i f (s i z e ( complex { j } , 2 ) ==2 && ( ~ isempty (fi nd ( v ( 1 , 2 ) ==complex { j } ) ) && ~ isempty (fi nd ( v ( 1 , 3 ) ==complex { j } ) ) ) ) 1050 matrix ( i , 3 ) = j −1; con = con + 1 ; 1052 end i f ( con ==3) 1054 con = 0 ; break ; 1056 end 1058 end end 1060 end 1062 end The next code for our algorithm gets the persistence intervals, and later calculates the persistent entropy based on these intervals. 1000 matrix = complex2matrix ( Complejo { k } ) ; 1002 [ cc ]= Persistence_new ( matrix ) ; cc= sortrows ( cc , 1 ) ; 1004 a l l =1:length( matrix ) ; 80 análisis topológico de datos: aplicación al reconocimiento de emociones missing = s e t d i f f ( all , cc ( : , 1 ) ) ’ ; 1006 missing ( : , 2 ) =0; dd= sortrows ( [ cc ; missing ] , 1 ) ; 1008 i i = Index { k } ; i i =[0 , i i ] ’ ; 1010 fo r h =1: length( dd ) dd ( h , 1 ) =max(cumsum ( ( i i −dd ( h , 1 ) <0) ) ) ; 1012 dd ( h , 2 ) =max(cumsum ( ( i i −dd ( h , 2 ) <0) ) ) ; end 1014 dd ( dd ( : , 2) == 0 , : ) = [ ] ; dd ( dd ( : , 1) == dd ( : , 2 ) , : ) = [ ] ; 1016 [ entropy ] = per_entropy ( dd ) ; diagram { k }= dd ; 1018 ent { k }= entropy ; To classify, we used the APP Classification Learner which offers Matlab. Thanks to that, it is possible to explore supervised machine learning using various classifiers. The automated training allows us to find the best classification model, includes decision trees, discriminant analysis, support vector machines, logistic regression, nearest neighbors, naive Bayes, and ensemble classification. The model uses a training dataset to train it and, later, it uses a validation dataset to test it and get the accuracy of the model. After we export the model to the workspace as trained_model (variable), we can predict the model with any sample. 1000 model= trainedModel . p re di ct F cn ( Sample ) ; 1002 %dataset 1004 T = c v p a r t i t i o n ( Entropy \ _ p e r s i s t e n c e . l ab e l , ’HoldOut ’ , 0 . 2 ) ; %20% use fo r t e s t i n g data t r i i d x = t r a i n i n g (T ) ; 1006 t e s t i n g = d a t a s e t (~ t r i i d x , : ) ; t r a i n i n g = Entropy \ _ p e r s i s t e n c e ( t r i i d x , : ) ; 1008 %Use c l a s s i f i c a t i o n l e a rn i n g apps % S e l e c t t r a i n i n g to t r a i n and v a l i d a t e 1010 %Export Model %C l a s s i f y the t e s t i n g d a ta s et using trainedModel 1012 model = trainedModel . pr ed ict Fc n ( t e s t i n g ) ; % Perform accuracy using f o r example , loss , accuracy , confusion . . 6. design and implementation 81 6.2 Analysis of the implementation of the algoritm for the audio dataset Considering the context of the project and the existing software environment, the language Python was chosen as the main programming language to implement the second part. This is a trade-off about to find a balance between the ease of implementation due to the language’s machine learning libraries, and the ease of use for future developers. Regarding the libraries, there are some used to work with audio signals. The first one is the Librosa library [80], whose overall goal is to provide a set of functions necessary to create audio information retrieval systems. Preprocessing of the audio signal was done using Librosa. Another useful library applied is pandas [81] that provides ways to store and organize data-sets efficiently. Based on the paradigm of scikit-learn library, the giotto-tda library arises to simplify the principal items for topological machine learning. The library currently supports the application of persistent homology to graph and time-series data. To calculate persistent homology, it is necessary to instantiate a Vietoris-Rips Persistence transformer and calculate the persistence diagram for the collection of point clouds. The topological features can be a connected component, 1D hole/loop, 2D cavity, or more generally 𝑛-dimensional void that exists in the data at scales between its birth and death values. The homology dimension of the feature is stored as the third input in that triplet. 1000 from gtda . homology import VietorisRipsPersistence 1002 VR = V i e t o r i s R i p s P e r s i s t e n c e ( homology_dimensions =[0 , 1 , 2 ] ) # Parameter expl ai ned in the t e x t diagrams = VR . f i t _ t r a n s f o r m ( poi nt _c lou ds ) 1004 diagrams . shape Listing 6.2: VietorisRipsPersistence As we explained in the thesis, the Embedding Theorem is practical when we have a time-series, which is expected to be guided by a non-linear system. The embedding dimension 𝑑and time delay 𝜏can be choose manually. However, there are two techniques that can be applied to get these parameter automatically. Mutual Information to calculate 𝜏and False nearest neighbours to get 𝑑. The next two function show how 88 análisis topológico de datos: aplicación al reconocimiento de emociones it. The idea was similar to the previous work of our group but, in this case, we use a matrix distance based on the bottleneck distance of the persistence diagram as the input of the classification algorithm. This way, the accuracy increases. Nevertheless, we think that more research on audio emotional recognition task can be done. The inclusion of other types of information that the audio offers, mix them, or the use of an auxiliary neural network to achieve high-level features should improve the accuracy of our model. Furthermore, to get better results in the classification of emotion task, we have combined the topological information obtained from the video dataset with the topological information obtained from the audio dataset. The setting One-vs-Rest allowed to achieve competitive results and be in concordance with one of the most relevant conclusions of the KRISTINA project: "The combination of visual and audio features can develop better predictions than using them separately". The following future works are plan to be explored: •To Expand to a 𝑛-dimensional feature vector of raw audio signals by extracting other types of information. •To divide the landmark points into different subsets to determine regions or pairs of regions that contain discriminative landmark points for each facial expression. •To reduce dimension could be an important task in the future since introducing new subsets of landmark points, the dimension of the feature vector could increase. •To analyze 3D data of the video sequence. The RAVDESS dataset provides landmarks points in the 3-dimensional space ℝ3. Taking advantage of the depth information of the landmark points could be a challenging problem for the future. •Instead of only focus on the face, another interesting point is to consider the silhouettes of the people that appear in the videos which include the face and the shoulder movements and that could have relevant information for the emotion recognition purpose. •The temporal relation between audio and visual features is not well-explored in the literature. In a future study, we will try to integrate this relation into our proposed system. Bibliography [1] Albano, A., Passamante, A., and Farrell, M. E. Using higher-order correlations to define an embedding window. Physica D: Nonlinear Phenomena 54, 1-2 (1991), 85–97. [2] Alswaidan, N., and Menai, M. E. B. A survey of state-of-the-art approaches for emotion recognition in text. Knowledge and Information Systems (2020), 1–51. [3] Atienza, N., Escudero, L. M., Jimenez, M. J., and Soriano-Trigueros, M. Persistent entropy: a scale-invariant topological statistic for analyzing cell arrangements. arXiv preprint arXiv:1902.06467 (2019). [4] Atienza, N., González-Díaz, R., and Soriano-Trigueros, M. On the stability of persistent entropy and new summary functions for tda. arXiv preprint arXiv:1803.08304 (2018). [5] Avots, E., Sapiński, T., Bachmann, M., and Kamińska, D. Audiovisual emotion recognition in wild. Machine Vision and Applications 30, 5 (2019), 975–985. [6] Bakker, I., Van der Voordt, T., Vink, P., and de Boon, J. Pleasure, arousal, dominance: Mehrabian and russell revisited. Current Psychology 33, 3 (2014), 405–421. [7] Bergomi, M. G., and Baratè, A. Homological persistence in time series: an application to music classification. Journal of Mathematics and Music (2020), 1–18. [8] Binchi, J., Merelli, E., Rucco, M., Petri, G., and Vaccarino, F. jholes: A tool for understanding biological complex networks via clique weight rank persistent homology. Electron. Notes Theor. Comput. Sci. 306 (2014), 5–18. 90 análisis topológico de datos: aplicación al reconocimiento de emociones [9] Borsuk, K. On the imbedding of systems of compacta in simplicial complexes. Fundamenta Mathematicae 35, 1 (1948), 217–234. [10] Botnan, M. B. Three approaches in computational geometry and topology: Persistent homology, discrete differential geometry and discrete morse theory. Master’s thesis, Institutt for matematiske fag, 2011. [11] Buzug, T., and Pfister, G. Comparison of algorithms calculating optimal embedding parameters for delay time coordinates. Physica D: Nonlinear Phenomena 58, 1-4 (1992), 127–137. [12] Buzug, T., and Pfister, G. Optimal delay time and embedding dimension for delay-time coordinates by analysis of the global static and local dynamical behavior of strange attractors. Physical review A 45, 10 (1992), 7073. [13] Calò, D. G. Gaussian mixture model classification: A projection pursuit approach. Computational statistics & data analysis 52, 1 (2007), 471–482. [14] Camplani, M., and Cannas, B. The role of the embedding dimension and time delay in time series forecasting. IFAC Proceedings Volumes 42, 7 (2009), 316–320. [15] Chazal, F., Cohen-Steiner, D., Guibas, L. J., Mémoli, F., and Oudot, S. Y. Gromov-hausdorff stable signatures for shapes using persistence. In Computer Graphics Forum (2009), vol. 28, Wiley Online Library, pp. 1393–1403. [16] Chazal, F., De Silva, V., Glisse, M., and Oudot, S. The structure and stability of persistence modules. arXiv preprint arXiv:1207.3674 21 (2012). [17] Chazal, F., De Silva, V., and Oudot, S. Persistence stability for geometric complexes. Geometriae Dedicata 173, 1 (2014), 193–214. [18] Chazal, F., Fasy, B. T., Lecci, F., Rinaldo, A., Singh, A., and Wasserman, L. On the bootstrap for persistence diagrams and landscapes. arXiv preprint arXiv:1311.0376 (2013). [19] Cherniakov, M. An introduction to parametric digital filters and oscillators. Wiley Online Library, 2003. [20] Chintakunta, H., Gentimis, T., Gonzalez-Diaz, R., Jimenez, M.-J., and Krim, H. An entropy-based persistence barcode. Pattern Recognition 48, 2 (2015), 391 – 401. BIBLIOGRAPHY 91 [21] Cohen-Steiner, D., Edelsbrunner, H., and Harer, J. Stability of persistence diagrams. Discrete & computational geometry 37, 1 (2007), 103–120. [22] Cohen-Steiner, D., Edelsbrunner, H., Harer, J., and Mileyko, Y. Lipschitz functions have l p-stable persistence. Foundations of computational mathematics 10, 2 (2010), 127–139. [23] Cohen-Steiner, D., Edelsbrunner, H., and Morozov, D. Vines and vineyards by updating persistence in linear time. In Proceedings of the twenty-second annual symposium on Computational geometry (2006), pp. 119–126. [24] Cunningham, P., and Delany, S. J. k-nearest neighbour classifiers–. arXiv preprint arXiv:2004.04523 (2020). [25] Dalgleish, T., and Power, M. Handbook of cognition and emotion. John Wiley & Sons, 2000. [26] De Silva, V., and Ghrist, R. Coverage in sensor networks via persistent homology. Algebraic & Geometric Topology 7, 1 (2007), 339–358. [27] Devi, J. S., Yarramalle, S., and Nandyala, S. P. Speaker emotion recognition based on speech features and classification techniques. International Journal of Image, Graphics and Signal Processing 6, 7 (2014), 61. [28] Diener, E., Larsen, R. J., Levine, S., and Emmons, R. A. Intensity and frequency: dimensions underlying positive and negative affect. Journal of personality and social psychology 48, 5 (1985), 1253. [29] d’Amico, M., Frosini, P., and Landi, C. Natural pseudo-distance and optimal matching between reduced size functions. Acta applicandae mathematicae 109, 2 (2010), 527–554. [30] Edelsbrunner, H., and Harer, J. Computational topology: an introduction. American Mathematical Soc., 2010. [31] Edelsbrunner, H., and Mücke, E. P. Three-dimensional alpha shapes. ACM Transactions on Graphics (TOG) 13, 1 (1994), 43–72. [32] El Ayadi, M., Kamel, M. S., and Karray, F. Survey on speech emotion recognition: Features, classification schemes, and databases. Pattern Recognition 44, 3 (2011), 572–587. 92 análisis topológico de datos: aplicación al reconocimiento de emociones [33] Fasy, B. T., Lecci, F., Rinaldo, A., Wasserman, L., Balakrishnan, S., Singh, A., et al. Confidence sets for persistence diagrams. The Annals of Statistics 42, 6 (2014), 2301–2339. [34] France, D. J., Shiavi, R. G., Silverman, S., Silverman, M., and Wilkes, M. Acoustical properties of speech as indicators of depression and suicidal risk. IEEE transactions on Biomedical Engineering 47, 7 (2000), 829–837. [35] Frank, J., Mannor, S., and Precup, D. Activity and gait recognition with timedelay embeddings. In AAAI (2010), vol. 2, Citeseer, p. 102. [36] Fraser, A. M. Information and entropy in strange attractors. IEEE transactions on Information Theory 35, 2 (1989), 245–262. [37] Garcia Cifuentes, C. Multi-class Classification with Machine Learning and Fusion. Universitat Politècnica de Catalunya, 2009. [38] Gery, I., Miljkovitch, R., Berthoz, S., and Soussignan, R. Empathy and recognition of facial expressions of emotion in sex offenders, non-sex offenders and normal controls. Psychiatry research 165, 3 (2009), 252–262. [39] Ghimire, D., and Lee, J. Geometric feature-based facial expression recognition in image sequences using multi-class adaboost and support vector machines. Sensors 13, 6 (2013), 7714–7734. [40] Ghrist, R. Barcodes: the persistent topology of data. Bulletin of the American Mathematical Society 45, 1 (2008), 61–75. [41] Gidea, M., and Katz, Y. Topological data analysis of financial time series: Landscapes of crashes. Physica A: Statistical Mechanics and its Applications 491 (2018), 820–834. [42] Giesen, J., and John, M. The flow complex: a data structure for geometric modeling. Computational Geometry 39, 3 (2008), 178–190. [43] Gonzalez-Diaz, R., Jimenez, M.-J., and Krim, H. Towards minimal barcodes. In International Workshop on Graph-Based Representations in Pattern Recognition (2013), Springer, pp. 184–193. [44] Gonzalez-Diaz, R., Paluzo-Hidalgo, E., and Quesada, J. F. Towards emotion recognition: A persistent entropy application. In Computational Topology in Image Context (2019), R. Marfil, M. Calderón, F. Díaz del Río, P. Real, and A. Bandera, Eds., Springer International Publishing, pp. 96–109. BIBLIOGRAPHY 93 [45] Gonzalez-Diaz, R., and Real, P. On the cohomology of 3d digital images. Discrete Applied Mathematics 147, 2-3 (2005), 245–263. [46] Grandjean, D., Sander, D., and Scherer, K. R. Conscious emotional experience emerges as a function of multilevel, appraisal-driven response synchronization. Consciousness and cognition 17, 2 (2008), 484–495. [47] Griffiths, P. E. Emotions. A Companion to cognitive science (2017), 197–203. [48] Gunes, H., and Pantic, M. Automatic, dimensional and continuous emotion recognition. International Journal of Synthetic Emotions (IJSE) 1, 1 (2010), 68–99. [49] Guo, X., Polanía, L. F., and Barner, K. E. Audio-video emotion recognition in the wild using deep hybrid networks. arXiv preprint arXiv:2002.09023 (2020). [50] Hatcher, A. Algebraic topology. cambridge university press, 2002. [51] Hegger, R., Kantz, H., and Schreiber, T. Practical implementation of nonlinear time series methods: The tisean package. Chaos: An Interdisciplinary Journal of Nonlinear Science 9, 2 (1999), 413–435. [52] Hilton, P. J., and Wylie, S. Homology theory: An introduction to algebraic topology. CUP Archive, 1967. [53] Hozjan, V., and Kačič, Z. Context-independent multilingual emotion recognition from speech signals. International journal of speech technology 6, 3 (2003), 311–320. [54] Hu, H., Xu, M.-X., and Wu, W. Fusion of global statistical and segmental spectral features for speech emotion recognition. In Eighth Annual Conference of the International Speech Communication Association (2007). [55] Issa, D., Demirci, M. F., and Yazici, A. Speech emotion recognition with deep convolutional neural networks. Biomedical Signal Processing and Control 59 (2020), 101894. [56] Jain, V., Lamba, P. S., Singh, B., Namboothiri, N., and Dhall, S. Facial expression recognition using feature level fusion. Journal of Discrete Mathematical Sciences and Cryptography 22, 2 (2019), 337–350. [57] Jenke, R., Peer, A., and Buss, M. Feature extraction and selection for emotion recognition from eeg. IEEE Transactions on Affective computing 5, 3 (2014), 327– 339. 94 análisis topológico de datos: aplicación al reconocimiento de emociones [58] Jiayu, L., Yueke, W., Zhiping, H., and Zhenkang, S. Selection of proper timedelay in phase space reconstruction of speech signals [j]. Signal Processing 3 (1999). [59] Kamińska, D., Sapiński, T., and Pelikant, A. Recognition of emotion intensity basing on neutral speech model. In Man-Machine Interactions 3. Springer, 2014, pp. 451–458. [60] Kantz, H., and Schreiber, T. Nonlinear time series analysis, vol. 7. Cambridge university press, 2004. [61] Kennel, M. B., Brown, R., and Abarbanel, H. D. Determining embedding dimension for phase-space reconstruction using a geometrical construction. Physical review A 45, 6 (1992), 3403. [62] Keogh, E., and Kasetty, S. On the need for time series data mining benchmarks: a survey and empirical demonstration. Data Mining and knowledge discovery 7, 4 (2003), 349–371. [63] Khasawneh, F. A., and Munch, E. Chatter detection in turning using persistent homology. Mechanical Systems and Signal Processing 70 (2016), 527–541. [64] Kim, Y., Lee, H., and Provost, E. M. Deep learning for robust feature generation in audiovisual emotion recognition. In 2013 IEEE international conference on acoustics, speech and signal processing (2013), IEEE, pp. 3687–3691. [65] Kleinsmith, A., and Bianchi-Berthouze, N. Affective body expression perception and recognition: A survey. IEEE Transactions on Affective Computing 4, 1 (2012), 15–33. [66] Kossaifi, J., Tzimiropoulos, G., Todorovic, S., and Pantic, M. Afew-va database for valence and arousal estimation in-the-wild. Image and Vision Computing 65 (2017), 23–36. [67] Kryzhanovsky, B., Dunin-Barkowski, W., and Redko, V. Advances in neural computation, machine learning, and cognitive research. Neuroinformatics (2017). [68] Lamar-Leon, J., Alonso-Baryolo, R., Garcia-Reyes, E., and Gonzalez-Diaz, R. Persistent homology-based gait recognition robust to upper body variations. In 2016 23rd International Conference on Pattern Recognition (ICPR) (Dec 2016), pp. 1083–1088. BIBLIOGRAPHY 95 [69] Lamar-León, J., Baryolo, R. A., Reyes, E. B. G., and González-Díaz, R. Persistent-homology-based gait recognition. CoRR abs/1707.06982 (2017). [70] Lazar, E. A., Mason, J. K., MacPherson, R. D., and Srolovitz, D. J. Statistical topology of three-dimensional poisson-voronoi cells and cell boundary networks. Physical Review E 88, 6 (2013), 063309. [71] Leinonen, L., Hiltunen, T., Linnankoski, I., and Laakso, M.-L. Expression of emotional–motivational connotations with a one-word utterance. The Journal of the Acoustical society of America 102, 3 (1997), 1853–1863. [72] Li, W., Tsangouri, C., Abtahi, F., and Zhu, Z. A recursive framework for expression recognition: from web images to deep models to game dataset. Machine Vision and Applications 29, 3 (2018), 489–502. [73] Liu, T.-Y., Yang, Y., Wan, H., Zeng, H.-J., Chen, Z., and Ma, W.-Y. Support vector machines classification with a very large-scale taxonomy. Acm Sigkdd Explorations Newsletter 7, 1 (2005), 36–43. [74] Livingstone, S. R., and Russo, F. A. The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english. PloS one 13, 5 (2018), e0196391. [75] Ma, J., Jin, H., Yang, L. T., and Tsai, J. J.-P. Ubiquitous Intelligence and Computing: Third International Conference, UIC 2006, Wuhan, China, September 3-6, 2006, Proceedings (Lecture Notes in Computer Science). Springer-Verlag, 2006. [76] Maria, C., Boissonnat, J.-D., Glisse, M., and Yvinec, M. The gudhi library: Simplicial complexes and persistent homology. In International Congress on Mathematical Software (2014), Springer, pp. 167–174. [77] Marsh, A. A., Kozak, M. N., and Ambady, N. Accurate identification of fear facial expressions predicts prosocial behavior. Emotion 7, 2 (2007), 239. [78] Mayer, J. D., Caruso, D. R., and Salovey, P. The ability model of emotional intelligence: Principles and updates. Emotion review 8, 4 (2016), 290–300. [79] McDuff, D., Kaliouby, R., Senechal, T., Amr, M., Cohn, J., and Picard, R. Affectiva-mit facial expression dataset (am-fed): Naturalistic and spontaneous facial expressions collected. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops (2013), pp. 881–888. 96 análisis topológico de datos: aplicación al reconocimiento de emociones [80] McFee, B., Raffel, C., Liang, D., Ellis, D. P., McVicar, M., Battenberg, E., and Nieto, O. librosa: Audio and music signal analysis in python. In Proceedings of the 14th python in science conference (2015), vol. 8, pp. 18–25. [81] McKinney, W., et al. pandas: a foundational python library for data analysis and statistics. Python for High Performance and Scientific Computing 14, 9 (2011). [82] MELLO, R. F., and Ponti, M. A. Machine Learning: A Practical Approach on the Statistical Learning Theory. Springer, 2018. [83] Merelli, E., Piangerelli, M., Rucco, M., and Toller, D. A topological approach for multivariate time series characterization: the epileptic brain. In Proceedings of the 9th EAI International Conference on Bio-inspired Information and Communications Technologies (formerly BIONETICS) (2016), pp. 201–204. [84] Mileyko, Y., Mukherjee, S., and Harer, J. Probability measures on the space of persistence diagrams. Inverse Problems 27, 12 (2011), 124007. [85] Mohri, M., Rostamizadeh, A., and Talwalkar, A. Foundations of machine learning. MIT press, 2018. [86] Morozov, D. Homological illusions of persistence and stability. Duke University, 2008. [87] Moskvin, A., and Shishkin, A. Deep learning based human emotional state recognition in a video. Journal of Modeling and Optimization 12, 1 (2020), 51–59. [88] Muller, K.-R., Mika, S., Ratsch, G., Tsuda, K., and Scholkopf, B. An introduction to kernel-based learning algorithms. IEEE transactions on neural networks 12, 2 (2001), 181–201. [89] Munch, E., Turner, K., Bendich, P., Mukherjee, S., Mattingly, J., Harer, J., et al. Probabilistic fréchet means for time varying persistence diagrams. Electronic Journal of Statistics 9, 1 (2015), 1173–1204. [90] Nicole, J., Rapp, V., Bailly, K., Prevost, L., and Chetouani, M. Audio-visual emotion recognition: A dynamic, multimodal approach. Preprint-HAL archives (2014). [91] Noroozi, F., Kaminska, D., Corneanu, C., Sapinski, T., Escalera, S., and Anbarjafari, G. Survey on emotional body gesture recognition. IEEE transactions on affective computing (2018). BIBLIOGRAPHY 97 [92] Ofodile, I., Kulkarni, K., Corneanu, C. A., Escalera, S., Baro, X., Hyniewska, S., Allik, J., and Anbarjafari, G. Automatic recognition of deceptive facial expressions of emotion. arXiv preprint arXiv:1707.04061 2 (2017). [93] Otter, N., Porter, M. A., Tillmann, U., Grindrod, P., and Harrington, H. A. A roadmap for the computation of persistent homology. EPJ Data Science 6, 1 (2017), 17. [94] Oudre, L., Jakubowicz, J., Bianchi, P., and Simon, C. Classification of periodic activities using the wasserstein distance. IEEE transactions on biomedical engineering 59, 6 (2012), 1610–1619. [95] Owada, T., Adler, R. J., et al. Limit theorems for point processes under geometric constraints (and topological crackle). The Annals of Probability 45, 3 (2017), 2004–2055. [96] Pan, S., and Duraisamy, K. Data-driven discovery of closure models. SIAM Journal on Applied Dynamical Systems 17, 4 (2018), 2381–2413. [97] Pedregosa, F., Varoqaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12 (2011), 2825–2830. [98] Penrose, M., et al. Random geometric graphs, vol. 5. Oxford university press, 2003. [99] Pereira, C. M., and de Mello, R. F. Persistent homology for time series and spatial data clustering. Expert Systems with Applications 42, 15-16 (2015), 6026– 6038. [100] Pławiak, P., Sośnicki, T., Niedźwiecki, M., Tabor, Z., and Rzecki, K. Hand body language gesture recognition based on signals from specialized glove and machine learning algorithms. IEEE Transactions on Industrial Informatics 12, 3 (2016), 1104–1113. [101] Plutchik, R. The nature of emotions: Human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice. American scientist 89, 4 (2001), 344–350.