Looking for Emotions on a Single EEG Signal

Looking for Emotions on a Single EEG Signal Royl´an Quesada-Tabares , Alberto J. Molina-Cantero(B), Jos´e I. Escudero-Fombuena , Manuel Merino-Monge , Isabel M. G´omez-Gonz´alez , Clara Lebrato-V´azquez , and Juan A. Castro-Garc´ıa Departamento de Tecnolog´ıa Electr´onica, Universidad de Sevilla, Seville, Spain [email protected], [email protected] Abstract. This work aims at demonstrating that it is possible to detect emotions using a single EEG channel with an accuracy that is comparable to that obtained in studies carried out with devices that have a high number of channels. In this article the Neurosky Maindwave device, which only a single electrode at the FP1 position, the MatLab and the IBM SPSS Modeler were used to acquire, process and classify the signals respectively. It is remarkable the accuracy achieved in relation to the inexpensive hardware employed for the acquisition of the EEG signal. The result of this study allows us to determine when the brain response is more intense after undergoing the subject, in the experimentation, to the stimuli that generate those emotions. This let us decide which brain power bands are most significants and which moments are the most appropriate to carry out this detection of emotions. Keywords: Emotions ·Signal processing ·Single EEG channel · Classification analysis ·Dynamic properties 1 Introduction Several theories have tried to explain the origin and nature of emotions. One of them posits the existence of a relative low number of basic emotions (families of emotions) which are universal for all human beings and independent of any cultural environment [8,12]. At least six emotional families have been proposed. They are: happiness, sadness, disgust, anger, fear and surprise. This theory includes, in turn, a locationist model that assumes that each emotional category starts from a specific place of the brain and body. Specifically, fear is located in the amygdala; disgust, in the insula; anger in the orbito-frontal cortex (OFC) and sadness in the anterior cortex of the cingulate (ACC) [28]. Another theory is based on the so-called constructionist model, in which it is asserted that emotions are psychological events that emerge from basic physiological operations and in which a large number of neural networks, spread in the brain, work together to obtain a meaningful experience. In [13] the authors propose a model with four components: core affect, some body sensory input that is experienced as pleasant/unpleasant with some degree of excitation; conceptualization, which links the body sensations with previous experiences to endow them with meaning; emotional words, used as support of emotional categories that are not clearly differentiable from the sensitive point of view; and executive attention, which focuses on some of the incoming stimuli. Some neuroimaging results have corroborated that, unlike that the locationist model predicts, a region activated during a basic emotion, was also for at least one other emotion [13]. This suggests the existence of neural networks that interact with each other to generate the emotions, instead of precise places (locationist model). For example the amygdala is recruited for both fear and disgust, so it takes different functionalities depending on the neural network that uses it. There is no a dominant theory about another, taking into account the results of some neuroimaging studies, which can be interpreted differently according to the procedure used for the treatment of the data. [11] summarizes the existing controversy pointing out that the future should include analysis with animal models and studies of patients with brain injuries, which have been reported a tendency towards a locationist theory. The dimensional theory considers that emotions can be represented in an N- dimensional space, where two of the coordinate axis would explain most of the emotional variations. These axis are called Valence and Arousal [19]. Valence is related to pleasure and varies from low values (very unpleasant) to high values (very pleasant). Arousal is related to the intensity of emotion, ranging from very low to very high. The dimensional model is also called the circumplex model, which is a dimensional representation of the core affect in the constructionist model. Although it would seem more appropriate to use this representation for the constructionist model, in fact, the six basic emotions can also be characterized according to their valence and arousal. Emotions can be detected in several ways: through the analysis of gestures (facial), speech [16] or the activity of various physiological signals, such as the electroencephalogram (EEG). In this work we analyze the EEG to obtain the activity of the brain during experiment in which specific emotions are elicited. There are a large number of commercial devices for measuring and recording the EEG activity. They vary in price and the way in transmitting the information (wired or wireless). We opted for the use of a wireless device, for its advantages in mobility. Several devices with these characteristics can be found in the market: Emotiv [10], Neurosky Mindwave [10] and Enobio [9]. Emotiv offers better results than the Neurosky Mindwave when it is used for the evaluation of cognitive processes [7]. However, the Neurosky Mindwave usability and prices make it very competitive. This device has been widely used by the scientific community for the development of various applications such as the detection of sleepiness [27], level of attention [14], stress [6,17], and so on. It is a device that offers developers and researchers the possibility to make a treatment of the measured signal, but also comes integrated with a system that processes and delivers to the user characteristics of the post-processing, which will not be used in our case. For the study and comprehension of the EEG signal, the analysis of the bands is widely used [20]. They are: δ(which ranges from 0 up to 4 Hz), θ(4–8 Hz), α (8–12 Hz), β(13–30 Hz) and γ(>30 Hz). Another feature we have used to study the EEG signal is the fractal dimension [22,29]. The fractal dimension shows the complexity of the signal. There are several ways to calculate it, in this study we used the Higuchi’s algorithm [26]. A fractal dimension close to 2 indicates that the signal is very complex, however a value close to 1 means that the signal is close to a line. This study aims to keep on analyzing the features extracted from a single EEG signal, which started in our previous work [18], but adding the statistical significance when different temporal windows of data are used to make the classification of the emotional states elicited by sets of images. 2 Methodology 2.1 Materials The EEG signal was captured using the Neurosky Mindwave, which has a fixed electrode located at Fp1, uses a sampling frequency at 512 Hz, a 12 bits analogdigital converter and a bluetooth interface to transmit the raw signal. To read, save, show the pictures and process data we used Matlab 8.4.0.150421 (R2014b). Finally, the IBM SPSS Modeler, a software with data mining tools, was employed to analyzed and classify the features extracted from raw EEG signals. IBM SPSS Modeler offers a wide variety of modeling methods from automated learning, artificial intelligence and statistics [5]. 2.2 Experimentation The experimentation consisted in showing 60 images (Table 1) selected from the IAPS (International Affective Picture System), grouped in three different sets of valence and arousal pairs (Fig. 1). An overall of three different arousal and valence values were chosen: 2.306 ±0.43, 5.063 ±0.24 and 6,921 ±0.032 for valence and sets 1, 2 and 3 respectively, whereas for arousal, 6.1890 ±0.04, 3,020 ±0.02 and 4,551 ±0.02 for respective sets. Seven people took part in the experiment with a mean age of 29.85 and a standard deviation of 8.97. Before starting the experiment, the SAM test (Self-assessment manikin) was applied to each person to evaluate her/his initial valence and arousal at that moment. Each picture was displayed on a computer screen for six seconds (Fig. 2) following the same experiment schedule performed in [1] with a resting period of time among images of 4 s, in which a cross was shown on the screen. A Matlab software application was responsible in displaying the images, registering the temporary marks to identify the time when the pictures appeared and were removed from the screen, and in receiving the raw EEG data sent by the Neurosky. All the information was stored for further processing. Table 1. Selected pictures obtained from the IAPS database. The arousal and valance for each picture and its averages are also shown. Set 1 Set 2 Set 3 File Val e nce Arousal File Va l enc e Arousal File Va len c e Arousal 2352.2 2.090 6.250 2038 5.090 2.940 1540 7.150 4.540 2683 2.620 6.210 2102 5.160 3.030 1731 7.070 4.560 3103 2.070 6.060 2381 5.250 3.040 2030 6.710 4.540 3150 2.260 6.550 2411 5.070 2.860 2152 6.930 4.500 3195 2.060 6.360 2850 5.220 32153 6.980 4.400 3550.1 2.350 6.290 2880 5.180 2.960 2306 7.080 4.460 6021 2.210 6.060 2890 4.950 2.950 2344 6.720 4.710 6022 2.140 6.090 6150 5.080 3.220 2362 6.740 4.600 6200 2.710 6.210 7003 4.970 3.160 2373 6.970 4.500 6213 2.190 6.010 7004 53.070 2391 7.110 4.630 6244 2.330 5.990 7010 4.930 3.010 2655 6.880 4.570 6350 2.310 6.380 7013 4.980 34616 6.860 4.430 6370 2.230 6.330 7018 5.180 3.120 5199 6.930 4.700 6570.1 2.190 6.240 7035 4.950 3.060 5300 6.910 4.360 6825 2.380 6.290 7054 5.220 2.950 5890 6.670 4.600 9301 2.260 67056 4.900 3.020 5994 6.800 4.610 9330 2.210 5.890 7057 5.070 3.070 7400 6.840 4.560 9610 2.480 6.460 7161 5.020 3.070 7430 6.910 4.550 9621 2.700 6.110 7165 4.980 2.980 7472 7.080 4.640 9903 2.330 67180 5.060 2.880 7481 7.080 4.550 mean: 2.306 ±0.43 6.189 ±0.040 mean: 5.063 ±0.024 3.020 ±0.020 mean: 6.921 ±0.032 4.551 ±0.020 A webcam recorded the experiment in order to contrast any possible anomaly in the signal. Finally, people were asked to fill in the SAM test for the 60 images. To do this, they were shown the same pictures again, in the same order. Fig. 1. Valence and arousal values of the selected pictures. Source [18]. 2.3 Signal Processing The EEG signal was split in epochs of 512 samples with an overlapping of 87.5% (or hop size of 64 samples). Then, each epoch was automatically analyzed to know if it was contaminated with artifacts: blinks, eye movements, muscle activity (EMG) or motion artifacts. In this specific case, the main source of artifacts are blinks and ocular movements, due to the electrode position, very close to the eye, although the electrical activity of the frontal an temporal muscles and motion artifacts are also important. The automatic identification of contaminated epochs is based on obtaining two features: the difference between the maximum and minimum value (Min- Max) in the epoch and the total energy of the signal (ESF) after applying a Savitzky-Golay lowpass filter (order 2 and length 35) [21]. Figure 3shows the feature space. Epochs contaminated by muscular activity have values of the Min- Max feature similar or a bit higher to those of the epochs with only EEG, but with more energy from the filtered signal (ESF). Blinking or EEG-only epochs have similar values in the ESF feature but differ from MinMax. Epochs with motion artifact contain values of these features that surround those obtained by other types of artifacts. An artificial neural network (ANN) was trained to detect epochs containing valid EEG data or blinks, but the results were similar to those obtained by setting thresholds in each dimension of the feature space (as shown in Fig. 3). Namely, the accuracy in detecting epochs contaminated with blinks was 98% whereas the accuracy in detecting valid EEG was of 96%. Setting thresholds in the feature space makes the identification method of valid epochs to be conservative, reducing the number of false positives at the cost of increasing the number of false negatives. This means that is better to reject some valid epochs than to accept contaminated epochs. While motion and EMG artifact are infrequent, the ocular ones do not. It is known that ocular artifacts affect, mainly, the lowest energy bands (δ,θand part of α), so not to remove them from the epoch (or not to discard the epoch) could distort the frequency analysis. There are techniques for removing ocular artifacts from the EEG signal. One of the most well-known technique is based on the analysis of independent components (ICA) [24], but, for its application, at least two EEG channels are needed, which do not exist in our case. [25] shows a technique to eliminate such artifacts in a single channel. It is based on applying a Savitzky-Golay low pass filter with the same characteristics as the one used for the calculation of the ESF feature. Therefore, if an epoch is identified as a blinking container, such a filter is applied before proceeding to the frequency analysis. Epochs containing valid EEG data or blinks, which were previously removed, were then windowed with a Hamming function to reduce spectral leakage. Then the squared fast Fourier transform (FFT) was applied to each segment to obtained the typical energy bands: δ,θ,α,βand γ. The fractal dimension was also calculated based on Higuchi’s algorithm [15], so a total of 6 values per epoch were obtained. Fig. 2. Experimental sequence. Source [18]. Fig. 3. Feature spaces values for motion artifacts (pink), muscular activity (blue), blinks (green) and EEG (black). The selected areas to identify EEG and EEG+blinks epochsarealsoshown.Source[18] (Color figure online) 2.4 Analysis For each subject, the average of five typical power bands, the fractal dimension and some ratios such as θ/βand θ/αfor all valid epochs were obtained. This yields an overall of 60 ×8 numerical values for each subject, grouped into three sets, according to the valence/aerosal values shown in Table 1, or, in other words, (3 ×20 ×8) features. Next, we applied three different types of analysis: statistical, classification and temporal. The first seeks to find significant differences among features and sets. The second analysis seeks to obtain the accuracy in detecting one out of the three emotional sets through a classifier based on decision trees. The last one studies the dynamic in inducing an emotion. Study 1. Here, we looked for the statistical significance of some features among sets. For doing so, we employed the Kruskal-Wallis test, which is a nonparametric alternative to the analysis of variance (ANOVA). It serves to contrast the hypothesis that k quantitative samples haven been obtained from the same population. It requires samples (or features from pictures) to be obtained randomly, which is completely fulfilled by the experiment design. The statistical analysis is individually applied to each participant and feature, so 8 ×7(features ×subjects) analysis were carried out. Remind that for each picture there are 8 features estimated using a six-second segment of data. Study 2. The IBM SPSS Modeler software was used in this analysis to generate a decision tree with the C5.0 classification algorithm. The option to partition the data, to ensure that the information used to test the model is not the same as the one employed to generate it, was activated. The input parameters to the classifier are the 8 x 20 x 3 (features x pictures x sets). The outputs are the predicted sets. The classifier was trained for each person, so 7 training and testing processes were accomplished. Study 3. Features in study 1 were obtained by using the whole 6s segment of data associated to the visualization of each picture. Our hypothesis is that features are not steady during the whole segment, but they show a temporal evolution. This is based on the idea that any subject takes a while to respond to the emotion elicited by the picture. As can be seen in Fig. 4, initially it must be a delay time (D), after the onset of the picture, in which the subject’s responsiveness is low and start growing up. During the interval ΔT , the feature reaches its maximum (minimum) value and probably began declining. It might be better to use only the interval ΔT instead of the whole segment for testing the statistical significance. 3 Results 3.1 Study 1 Table 2shows the absolute values of the features obtained for Study 1, grouped according to the set of images and subject. Statistically significant values, with ap<0.05 have been also highlighted. Subjects 3, 6 and 7 obtained values significant for the fractal dimension, subjects 5 and 7 in the αband and subject 5intheγpower band too. There isn’t a clear relationship between features’ behavior and changes on valence and arousal. For example, for subject 3, the fractal dimension has a concave shape when moving from set 1 to set 3 whereas for the subject 6, the same feature, has a growing monotone behavior. Table 2. Averaged values of each feature, subject and set of images. Values for power bands are scaled by 107. In bold the features that obtained significant results according to the Kruskal-Wallis test. Most of data contained in the table has been obtained from [18]. Set1 Set2 Set3 Subject δ θ α β γ θ β θ αhfd δ θ α β γ θ β θ αhfd δ θ α β γ θ β θ αhfd 16,04 5,12 2,3 3,21 2.85 1,59 2,22 1,58 3,36 3,65 1,95 3,08 2.85 1,18 1,86 1,59 4,02 3,98 1,96 2,89 2.47 1,37 2,02 1,58 22,06 2,49 2,63 3,9 5.01 0,63 0,94 1,66 2,1 2,59 2,95 4,08 4.32 0,63 0,87 1,65 2,19 3,26 2,85 3,77 4.13 0,86 1,14 1,65 34,32 5,48 4,37 8,64 3.73* 0,63 1,25 1,57* 4,58 5,83 4,98 8,61 3.34* 0,67 1,17 1,55* 4,06 5,63 4,67 8,3 3.81 0,67 1,2 1,58* 42,96 3,29 1,77 5,34 6.06 0,61 1,85 1,64 2,15 1,91 1,1 5,11 5.77 0,37 1,73 1,64 2,45 2,2 1,29 5,14 5.82 0,42 1,69 1,65 52,18 2,64 1,56* 3,81 4.38 0,69 1,69 1,68 1,61 2,09 1,1* 3,51 4.78 0,59 1,88 1,68 2,29 2,88 1,47* 3,6 4.05 0,8 1,95 1,67 64,68 4,94 3,09 4,5 3.32 1,09 1,6 1,58* 4,54 4,64 3,09 4,73 4.09 0,98 1,5 1,6* 3,56 4,4 3,05 4,76 4.51 0,92 1,44 1,63* 71,11 1,24 0,69* 4,51 1.29 0,27 1,78 1,71* 2,62 1,9 1,32* 4,53 9.52 0,41 1,43 1,68* 2,13 2,53 1,64* 5,07 1.03 0,49 1,53 1,69* Subjects 5 and 7 show significant changes in the αband. However, like with the fractal dimension, there isn’t a regular pattern in this feature. For example, while subject 7 shows an increase in α, subject 5, has a concave behavior. Subject 3 also obtained significant values in γband with a concave behavior among sets. Fig. 4. Temporal evolution of a feature after the onset of the stimulus (t=0). There is an initial delay, D, followed by the interval of interest, ΔT , in which the feature reaches its maximum or minimum value. 3.2 Study 2 Table 3shows the accuracy in classifying the 3 sets. For them all, the accuracy is over 70%, reaching up to near 92% for subject 5. In average, the accuracy was around 81%. 3.3 Study 3 We hypothesized that there must be a temporal evolution in any feature after the onset of the stimulus. To illustrate this Fig. 5shows the average temporal evolution of αand γpower bands for subject 4 and set of pictures. It can be seen that power bands temporally changes as time goes by. For example, in set 3, it would be better to obtain this features in the middle of the interval for the αband.