Digitizing arquetypal human expereience through physiological signals

Digitizing arquetypal human expereience through physiological signals Leonid I. Ivonin ADVERTIMENT La consulta d’aquesta tesi queda condicionada a l’acceptació de les següents condicions d'ús: La difusió d’aquesta tesi per mitjà del r e p o s i t o r i i n s t i t u c i o n a l UPCommons (http://upcommons.upc.edu/tesis) i el repositori cooperatiu TDX ( h t t p : / / w w w . t d x . c a t / ) ha estat autoritzada pels titulars dels drets de propietat intel·lectual únicament per a usos privats emmarcats en activitats d’investigació i docència. No s’autoritza la seva reproducció amb finalitats de lucre ni la seva difusió i posada a disposició des d’un lloc aliè al servei UPCommons o TDX. No s’autoritza la presentació del seu contingut en una finestra o marc aliè a UPCommons (framing). Aquesta reserva de drets afecta tant al resum de presentació de la tesi com als seus continguts. En la utilització o cita de parts de la tesi és obligat indicar el nom de la persona autora. ADVERTENCIA La consulta de esta tesis queda condicionada a la aceptación de las siguientes condiciones de uso: La difusión de esta tesis por medio del repositorio institucional UPCommons (http://upcommons.upc.edu/tesis) y el repositorio cooperativo TDR (http://www.tdx.cat/?localeattribute=es) ha sido autorizada por los titulares de los derechos de propiedad intelectual únicamente para usos privados enmarcados en actividades de investigación y docencia. No se autoriza su reproducción con finalidades de lucro ni su difusión y puesta a disposición desde un sitio ajeno al servicio UPCommons No se autoriza la presentación de su contenido en una ventana o marco ajeno a UPCommons (framing). Esta reserva de derechos afecta tanto al resumen de presentación de la tesis como a sus contenidos. En la utilización o cita de partes de la tesis es obligado indicar el nombre de la persona autora. WARNING On having consulted this thesis you’re accepting the following use conditions: Spreading this thesis by the i n s t i t u t i o n a l r e p o s i t o r y UPCommons (http://upcommons.upc.edu/tesis) and the cooperative repository TDX (http://www.tdx.cat/?localeattribute=en) has been authorized by the titular of the intellectual property rights only for private uses placed in investigation and teaching activities. Reproduction with lucrative aims is not authorized neither its spreading nor availability from a site foreign to the UPCommons service. Introducing its content in a window or frame foreign to the UPCommons service is not authorized (framing). These rights affect to the presentation summary of the thesis as well as to its contents. In the using or citation of parts of the thesis it’s obliged to indicate the name of the author. DIGITIZING ARCHETYPAL HUMAN EXPERIENCE THROUGH PHYSIOLOGICAL SIGNALS leonid i.ivonin Ivonin, Leonid I. Digitizing archetypal human experience through physiological signals / Eindhoven: Eindhoven University of Technology, 2014 A catalogue record is available from the Eindhoven University of Technology Library ISBN 978-90-386-3600-9 NUR 984 Keywords: Affective computing / Psychology / Archetype / Modeling / Psychophysiology / Unconscious / Emotion / Machine learning Latex typesetting and cover design: Leonid I. Ivonin Reproduction: GVO Drukkers & Vormgevers B.V. Copyright © 2014 by Leonid I. Ivonin DIGITIZING ARCHETYPAL HUMAN EXPERIENCE THROUGH PHYSIOLOGICAL SIGNALS proefschrift ter verkrijging van de graad van doctor aan de Technische Universiteit Eindhoven, op gezag van de rector magnificus prof.dr.ir. C.J. van Duijn, voor een commissie aangewezen door het College voor Promoties, in het openbaar te verdedigen op woensdag 14 mei 2014 om 16:00 uur door Leonid Igorevich Ivonin geboren te Lyubertsy, Rusland Dit proefschrift is goedgekeurd door de promotoren en de samenstelling van de promotiecommissie is als volgt: voorzitter: prof.dr.ir. A.C. Brombacher 1epromotor: prof.dr. M. Rauterberg 2epromotor: prof.dr. A. Català (Universitat Politècnica de Catalunya) co-promotor: dr. W. Chen leden: prof.dr. M. Díaz (Universitat Politècnica de Catalunya) prof.dr.ir. L.M.G. Feijs prof.dr.-ing. J. Ziegler (Universität Duisburg-Essen) This dissertation was produced under Erasmus Mundus Joint Doctorate Program in Interactive and Cognitive Environments. The research was conducted towards a joint double PhD degree between the following partner universities: TECHNISCHE UNIVERSITEIT EINDHOVEN UNIVERSITAT POLITÈCNICA DE CATALUNYA ACKNOWLEDGMENTS This PhD Thesis has been developed in the framework and according to the rules of the Erasmus Mundus Joint Doctorate on Interactive and Cognitive Environments EMJD ICE [FPA n° 2010-0012] with the cooperation of the following Universities: Alpen-Adria-Universität Klagenfurt – AAU Queen Mary, University of London – QML Technische Universiteit Eindhoven – TU/e Università degli Studi di Genova – UNIGE Universitat Politècnica Catalunya – UPC According to ICE regulations, this PhD Thesis has been awarded also of Italian PhD title by the Università degli Studi di Genova. SUMMARY digitizing archetypal human experience through physiological signals The problem of capturing human experience is relevant in many application domains. In fact, the process of describing and sharing individual experience lies at the heart of human culture. Throughout the courses of our lives we learn a great deal of information about the world from other people’s experience. Besides the ability to share utilitarian experience such as whether a particular plant is poisonous, humans have developed a sophisticated competency of social signaling that enables us to express and decode emotional experience. The natural way of sharing emotional experiences requires those who share to be co-present during this event. However, people have overcome the limitation of physical presence by creating a symbolic system of representations. This advancement came at a price of losing some of the multidimensional aspects of primary, bodily experience during its projection into the symbolic form. Recent research in the field of affective computing has addressed the question of digitization and transmission of emotional experience through monitoring and interpretation of physiological signals. Although the outcomes of this research represent a great step forward in developing a technology that supports sharing of emotional experiences, they do not seem to help in preserving the original phenomenological experience during the aforementioned projection. This circumstance is explained by the fact that in affective computing the focus of investigation has been aimed at emotional experiences which can be consciously evaluated and described by individuals themselves. Therefore, generally speaking, applying an affective computing technique for capturing emotions of an individual is not a deeper or more precise way to project her experience into the symbolic form than asking this person to write down a description of her emotions on a piece of paper. One can say that so far the research in affective computing has aimed at delivering technology that could automate the projection but it has not considered the problem of improving the projection in order to preserve more of the multidimensional aspects of human experience. This dissertation examines whether human experience, which individuals are not able to consciously transpose into the symbolic representation, can still be captured using the techniques of affective computing. First, a theoretical framework for description of human experience which is not accessible for conscious awareness was formulated. This framework was based on the work of Carl Jung who introduced a vii model of a psyche that includes three levels: consciousness, the personal unconscious and the collective unconscious. Consciousness is the external layer of the psyche that consists of those thoughts and emotions which are available for one’s conscious recollection. The personal unconscious represents a repository for all of an individual’s feelings, memories, knowledge and thoughts that are not conscious at a given moment of time. The collective unconscious is a repository of universal modes and behaviors that are similar in all individuals. According to Jung, the collective unconscious is populated with archetypes. Archetypes are prototypical categories of objects, people and situations that existed across evolutionary time and in different cultures. When an archetype becomes activated and is experienced by a person, it will result in a complex in the personal unconscious. A complex in the personal unconscious is a conglomeration of emotions and ideas that are specific to the person and are product of the archetype. In this thesis, the unconscious experience that is related to archetypes was defined as the archetypal experience. It seemed reasonable to begin our inquiry into the digitization of the unconscious human experience with considering the problem of recognizing the archetypal experience because archetypes are supposed to be common among individuals. Moreover, they provide a convenient way to conceptualize the unconscious experience. Having defined our theoretical framework we conducted an experiment where visual and auditory stimuli from standardized databases for elicitation of conscious emotions were demonstrated to subjects. Apart from the stimuli for conscious emotions, the subjects were exposed to stimuli that represented the archetype of the self. During presentation of the stimuli cardiovascular signals of the subjects were recorded. The experimental results indicated that heart rate responses of the participants were unique for every category of the stimuli including the archetypal one. These findings gave an impulse to perform another study where a broader spectrum of archetypal experiences was examined. In our second study, we made a switch from visual and auditory stimuli to audiovisual stimuli because it was expected that videos would be more efficient in elicitation of conscious emotions and archetypal experiences than still pictures or sounds. The number of archetypes was increased, and overall, subjects were stimulated to feel eight various types of the archetypal experience. We also prepared stimuli for conscious emotions. In this experiment, physiological signals included cardiovascular, electrodermal, respiratory activities and skin temperature. The statistical analysis suggested that the archetypal experiences could be differentiated based on the physiological activations. Moreover, several prediction models were constructed based on the collected physiological data. These models demonstrated an ability to viii contents xv 6.4Discussion 119 6.4.1Self-Reports Data 119 6.4.2Physiological Data 120 6.4.3ArcheSense 122 6.4.4Limitations 124 6.5Conclusion 125 7 general discussion 127 7.1Implicit Mental Experience and Archetypes 127 7.2Physiological Correlates of Affective Pictures and Sounds 128 7.3Physiological Correlates of Archetypal Film Clips 130 7.3.1Study 1131 7.3.2Study 2133 7.4Selection and Validation of Archetypal Stimuli 135 7.5Research Questions 136 7.6Future Work 137 7.7Applications 138 7.8Conclusion 139 8 appendix 141 8.1Entertainment System for The Unconscious 141 8.2Alternative Representation of Emotional States 143 8.2.1Introduction and Approach 143 8.2.2Results 145 8.2.3Discussion and Conclusion 146 8.3Source Code of ArcheSense 149 bibliography 151 LIST OF FIGURES Figure 1The two-dimensional affective space. 17 Figure 2Three modes of representing changes in the components of emotion. 19 Figure 3Three cognitive systems: perception, intuition, and reasoning. 20 Figure 4Examples of mandala symbols. 23 Figure 5Depiction of relations between elements in psychological and physiological domains. 27 Figure 6Different types of mandala symbols. 34 Figure 7The timeline of a stimulus presentation. 38 Figure 8Five categories of stimuli are plotted in the affective space. 40 Figure 9Changes in heart rate from the baseline for different types of media. 44 Figure 10 Changes in heart rate from the baseline for different categories of stimuli. 44 Figure 11 The deceleration of heart rate for different types of media and categories of the stimuli. 46 Figure 12 A subject is seated in the laboratory. 56 Figure 13 The procedure of the experiment is illustrated with a flowchart. 57 Figure 14 Heart rate responses of the subjects to the film clips. 65 Figure 15 The two modes of operation supported by the tool. 79 Figure 16 The architecture of the proposed tool. 83 Figure 17 The data model of the proposed tool. 85 Figure 18 A general overview of the proposed tool for evaluation of affective experience. 87 Figure 19 The navigation of Graphical User Interface (GUI) screens. 88 Figure 20 Screenshots of the GUI.90 Figure 21 The pattern of the subjects’ heart rate signals corresponding to the presentation of neutral film clips. 108 Figure 22 Classification framework for entertainment applications. 141 Figure 23 The conceptual design of the entertainment system based on Kansei Mediation interaction model. 142 Figure 24 Representations of the affective space. 144 xvi Figure 25 Four categories of the stimuli are plotted in the affective space with Cartesian and the modified polar coordinate systems. 146 LIST OF TABLES Table 1Sets of basic emotions. 15 Table 2An overview of the stimuli used in the experiment. 36 Table 3Mean values of the heart rate in beats per minute for different media and categories. 41 Table 4Changes in heart rate in beats per minute for different media and categories. 43 Table 5Sources of the film clips for elicitation of the archetypal experience. 53 Table 6Sources of the film clips for elicitation of explicit emotions. 54 Table 7Classification performance corresponding to the datasets of physiological features. 66 Table 8Comparison of the classification performance for the physiological data and the self-reports. 67 Table 9Sources of the film clips for the archetypal experience (part 1). 97 Table 10 Sources of the film clips for the archetypal experience (part 2). 98 Table 11 Sources of the film clips for the explicit emotions. 100 Table 12 Mean values and standard errors of the SelfAssessment Manikin (SAM) ratings. 110 Table 13 Classification results obtained for recognition of the archetypes and the explicit emotions from the self-reports. 111 Table 14 Classification results obtained for recognition of the archetypes and the explicit emotions from cardiovascular responses of the participants. 113 Table 15 Classification results obtained for recognition of the archetypes and the explicit emotions from the skin conductance of the participants. 114 Table 16 Classification results obtained for recognition of the archetypes and the explicit emotions from the complete dataset of the physiological signals. 115 xvii Table 17 Comparison of the classification accuracy achieved using the self-report questionnaires and the physiological data. 116 Table 18 The means and the standard errors of the recognition accuracies calculated based on the withinsubject classification. 117 Table 19 Confusion matrix of the model obtained with Principal Component Analysis (PCA) and k-Nearest Neighbor (kNN) techniques. Each column represents the instances in a predicted class. Each row represents the instances in the actual class. 118 Table 20 Inferential statistics of the self-assesment ratings in the affective space using Cartesian and the modified polar coordinate systems. 147 ACRONYMS AdaBoost Adaptive Boosting ANOVA Analysis of Variance ANS Autonomic Nervous System ARAS Archive for Research in Archetypal Symbolism CNS Central Nervous System DFT Discrete Fourier Transform DTW Dynamic Time Warping DWT Discrete Wavelet Transform ECG Electrocardiogram EEG Electroencephalography fMRI functional Magnetic Resonance Imaging GSR Galvanic Skin Response GUI Graphical User Interface HCI Human Computer Interaction HMM Hidden Markov Model HR Heart Rate xviii acronyms xix HRV Heart Rate Variability IADS International Affective Digitized Sounds IAPS International Affective Picture System kNN k-Nearest Neighbor LDA Linear Discriminant Analysis LMM Linear Mixed Model MANOVA Multivariate Analysis of Variance MEG Magnetoencephalography MRI Magnetic Resonance Imaging PCA Principal Component Analysis RR Respiration Rate SAM Self-Assessment Manikin SCL Skin Conductance Level SCR Skin Conductance Response ST Skin Temperature SVM Support Vector Machine TMS Transcranial Magnetic Stimulation 1 INTRODUCTION "Until you make the unconscious conscious, it will direct your life and you will call it fate." C.G. Jung 1.1 background For a long time people assumed that the mind is able to observe its own inner processes and, in accordance with this point of view, scientific psychology started a scholarly investigation of the consciousness. Consciousness is at once one of the most intriguing research topics in science and something very familiar to everybody. There is no single definition of consciousness. Some authors refer to it as “experience of experiencing, the knowledge of knowing, the sense of sensing”, while others define it as “an alert cognitive state in which you are aware of yourself and your situation” (Blackmore,2005). Wundt, Titchener and other pioneers of experimental psychology began their studies of the mind with introspection methods that involved analysis of thoughts, feelings and memories self-reported by trained subjects. This approach presumed that the subjects are able to consciously perceive all the aspects of their mental lives, explain and reduce them to elementary emotions and thoughts. However, later experimental observations suggested that the consciousness is restrained from the access to many mental structures and processes. Work of Helmholtz, Freud and Janet exposed the role of the mind’s unconscious processes which operate outside of phenomenal awareness and yet have an impact on conscious experiences of people. The unconscious processing enables people to adjust to the world, make decisions and set goals while their consciousness is busy with other tasks. As Wilson and Bar-Anan (2008) pointed out, people frequently know very little about their own minds because much of the mental processes, both at lowand high-levels, are automatic and hidden from the self-awareness. Without fast low-level unconscious decisions we would be pondering over simple matters, such as whether to use left or right hand to grab a cup of coffee. On the other hand, intuition, an ability to unconsciously analyze deep behavioral patterns and gain knowledge without conscious inference, is a good illustration of high-level processing in the unconscious mind. 1 2 introduction One of the most influential theories of the unconscious was proposed by Jung (1981) who developed it based on the observations of his patients and through analyzing folklore from various cultures and epochs. According to Jung, the mind is comprised of three parts: the ego, which is identified with the consciousness; the personal unconscious, which contains unconscious processes specific to a particular individual; and the collective unconscious, which is a reservoir of the experience shared by all human beings. The collective unconscious does not develop individually but is inherited and substitutes a common psychic substance of a universal nature. The content of the collective unconscious is described with the concept of archetypes - universal forms of experiences and feelings that lead to typical and recognizable patterns of behavior. Archetypes are very close analogies to instincts because the latter are impersonal, inherited traits that present and motivate human behavior long before any consciousness develops. However, these two phenomena are not interchangeable because instincts are seen as impulses to actions without conscious motivation while archetypes are the pre-existent forms of apperception that determine human experience. Although the conscious mind cannot observe the unconscious, it is sometimes confronted with manifestations of archetypes that may emerge in, for instance, dreams. As it is known from the field of psychophysiology, affective and cognitive processes of people can be indirectly assessed through measurement of their physiological parameters such as heart rate or brain activity. This fact enabled researchers to approach the investigation of the unconscious mind from physiological perspective (see, for example, (Bornemann et al.,2012) or (Kimura et al.,2004)). Interestingly enough, there are different points of view on the role of the unconscious in mental life. The research of Miller (1992) on the usage of psychophysiological recordings for therapy to improve consciousness illustrates the attitude, according to which, the unconscious is “dumb” and only capable to assist in highly routinized activities. Another perspective on the unconscious is found in the work of Bargh and Morsella (2008) who argued that the higher mental processes including judgment and social behavior could function in the absence of conscious control, and thus, provide evidence of the intelligent and adaptive nature of the unconscious mind. This point of view was also supported by recent research in neuroscience (van Gaal and Lamme,2012). The history of the psychological thought on structure and operation of the mind evolved from the assumption that our mental lives are completely conscious to recognition and investigation of the unconscious processes. The history often repeats itself and so it did when about a decade ago Picard (2000) started a new research direction in Human Computer Interaction (HCI) that was aimed at study and development of computer systems with a capability to recognize human affect. This research track named affective computing focuses on 1.1 background 3 sensing, interpretation and simulation of conscious emotions. Later, Fairclough (2009) proposed to widen the scope of affective computing and, besides emotions, include other psychological states in the scope of investigation. He coined the term physiological computing for description of this research area. The advantages of psychophysiological user interfaces such as increased adaptive capability, effortless and extended communication bandwidth have attracted the attention of HCI researchers (Hudlicka,2003;Pantic and Rothkrantz,2003) and have stimulated investigations associated with computer systems that can recognize and mimic human cognitive and affective states (McDuff et al.,2012;Scheirer et al.,2002). However, while studying recognition of conscious affective and cognitive states of users, HCI researchers largely ignored the unconscious minds of people. Sensing and interpretation of user experience beyond the levels of cognition and affect, in the domain of ancestral instincts and inborn behaviors, is not well understood and remains a largely unexplored area of HCI. One of the possible reasons that explains the negligence of the unconscious in the HCI domain is the lack of noticeable practical applications that would motivate research in this direction. Indeed, thinking about hypothetical human-computer interaction scenarios it is difficult to imagine a situation when the unconscious aspect of users’ mental lives has to be considered. The only exception could be interaction with entertainment systems, such as intelligent movie recommenders or biofeedback controlled games. Yet, if we look outside of the HCI discipline, there are several areas that would greatly benefit from the ability to recognize and interpret unconscious experiences of people. First of all, let us consider a general problem of design and presentation of products. At the early stages of product development designers need an input about customers’ preferences and experiences with initial prototypes. This information helps the designers to create better products that will be in demand on the marketplace. It turns out that collection of the insights about customers’ values and experiences with products requires a particular attention to their unconscious minds because, as Zaltman (2003) pointed out, “ninety-five percent of thinking takes place in our unconscious minds – that wonderful, if messy, stew of memories, emotions, thoughts, and other cognitive processes we are not aware of or that we cannot articulate”. Although, this thoughtprovoking claim may seem hard to believe in, it has been supported by the experimental evidence (Bargh and Chartrand,1999;Wilson,2004; Wilson and Bar-Anan,2008;Wegner,2003). Current trends in marketing research – a discipline that steers the design of products and communicates their value to potential customers – confirm the importance of the unconscious mind. An emerging field of marketing science called neuromarketing applies neuroimaging methods to product marketing in order to uncover the hidden preferences of customers that cannot be identified with traditional tools such as surveys, focus 4 introduction groups or market tests (Ariely and Berns,2010). The main promise of neuromarketing is that through measurement of brain activities it will deliver accurate information about the unconscious experience of customers. Another problem that bears attention is the evaluation of performance of media. In the same way as product designers need information about the hidden preferences of their future customers, media creators require a measure of viewers’ unconscious feelings in order to produce powerful and impressive media content. The established methods for the evaluation of a media including various rating schemes and questionnaires can only collect the feedback that the viewers decide or able to report. For this reason, everything that is inaccessible for their conscious minds also remains concealed from the questionnaires. However, as the research in psychology demonstrated, some of the most important decisions may be made unconsciously, and in this case, the viewers will lack awareness of how they felt while watching media. Similarly to the claim with regard to evaluation of the customer experience, neuromarketing professionals assert that the viewers’ unconscious responses to presentation of a media can be decoded from brain imaging data. While it might be still too early to conclude whether neuromarketing is hope or hype (Ariely and Berns, 2010), the necessity for a measure of the viewers’ unconscious experiences is evident. Moreover, the ability to recognize and interpret the unconscious experiences is likely to be highly appreciated by the community called Quantified Self (Rivera-Pelayo et al.,2012) that becomes increasingly widespread. Members of this community promote new kinds of lifelogging techniques because of their interest in self-knowledge and selfimprovement through continuous self-tracking with wearable computers. The knowledge about the processes in one’s unconscious mind would bring the self-discovery to the ultimate level that could not be attained otherwise. It may also help to improve an individual’s wellbeing by providing insights about the hidden reasons of depression or anxiety. Although we can continue the list of the problems requiring a measure of the unconscious experiences, the aforementioned examples have already clearly illustrated the importance of it outside of the HCI domain. Furthermore, research in neuromarketing has already been targeting some of these problems. However, the approach commonly utilized in neuromarketing studies has several disadvantages that may be critical in certain circumstances. First, the brain imaging technologies such as functional Magnetic Resonance Imaging (fMRI) or Magnetoencephalography (MEG) that dominate in neuromarketing have high operating costs. Therefore, only hospitals or large companies can afford the price of brain scans that among the cost of equipment includes personnel and overhead expenses. A less expensive alternative 1.3 novelty,main objectives,and outline of the thesis 11 important one is the limited generalizability of the results. We did our utmost to ensure that the obtained models provide valid predictions on unknown data sample by applying an appropriate statistical techniques. However, the reported results should be repeated in several other studies for the ultimate confirmation of their generalizability. In order to streamline preparation and execution of future experiments on recognition of the archetypal experience, we made a decision to develop a tool that would facilitate administration of studies, collection of physiological data and quantitative analysis of the obtained recordings. This tool would enable us conduct studies faster and with less effort. Moreover, as we plan to freely distribute our tool online, other researchers working in related areas may benefit from it and, perhaps, investigate the phenomenon of the collective unconscious with new populations of subjects originating from various regions of the world. The findings from a variety of studies, which are performed by different researchers in a standardized manner thanks to the use of the tool, would help us to arrive the ultimate conclusion about the relationship between human experience in the domain of the collective unconscious and physiological signals of the ANS. Besides, such a tool could be useful in evaluation of customer experience with products or media. As we pointed our in the previous section, this tool will be superior to the state of the art approaches in three aspects: recognition of the archetypal experience, cost effectiveness and flexibility. In Chapter 5we describe the process of design and implementation of this tool that went through several iterations. We also explain the methodology of practical use of our tool. Chapter 6provides description of our third study that was aimed at recognition of the archetypal experiences by means of the tool we developed. Although the general design of the experiment was similar to the previous study, there were differences that have to be highlighted. The most significant modification was made with regard to the duration of video clips for induction of archetypal experiences. It was decreased from five minutes to one minute due to practical considerations. Additionally, more video clips for every archetype were prepared and demonstrated to participants. Also, in this study we focused on measurement of only two physiological signals – cardiovascular and electrodermal responses – because, according to the data from the previous experiment, they contributed to classification performance the most. The results of the third study generally confirmed our previous observations. Next, we present a general discussion of our research findings and their relation to the two questions we formulated above. This discussion can be found in Chapter 7and also includes our point of view on the future work in evaluation of archetypal human experience. Finally, the dissertation ends with a conclusion and complementary information provided in the appendix. 2 STATE OF THE ART 2.1 introduction People always demonstrated extensive interest in the exploration of mental experience. Although this subject has received substantial attention in science, it is still associated with more questions than answers. This circumstance can be explained by the fact that mental experience is a complex phenomenon involving thoughts, emotion, perception, memory, and imagination. In this thesis, the focus is on the experience related to emotion and corresponding conscious and unconscious mental processes. An overview of the current state of affairs with regard to emotion, archetypes, and the collective unconscious is provided in the present chapter. Moreover, we review the available knowledge about the relation between psychological states of people and patterns of activations in their physiological signals, such as heart rate or skin conductivity. 2.2 emotion Emotion is obviously important in human existence. At once it is one of the most intriguing topics in the modern science and something familiar to everybody. Questions about emotion are fundamental in psychology and play an important role in understanding of mind and behavior (Barrett,2006). Despite of this fact, researchers dealing with emotion have not been able to achieve a clear consensus on many aspects related to emotion (LeDoux,1995). There is a disagreement about how to define emotion, which methodology is better for structuring and classifying various emotional states, whether different emotion have recognizable physiological signatures, the role of conscious and unconscious processes in emotion, and so on. The concept of emotion is familiar to everyone. However, replacing the intuitive understanding of emotion with operational definitions and transferring it into the domain of scientific study is not a trivial task (Bradley,2000). As a matter of fact, Kleinginna and Kleinginna (1981) had already counted more than one hundred proposals for scientific definitions of emotion in 1981. One of the first theoretical definitions for emotion was introduced by James (1884) who discussed the relationship between the environment, physiological activations, and emotional experiences. James proposed the theory which maintains “... that the bodily changes follow directly the perception of the exciting fact, and that our feeling of the same changes as they occur is the emo13 14 state of the art tion.” A similar idea was independently proposed by Lange a year later. Eventually, this theory became known as the James-Lange theory. The main feature of the James-Lange theory is that it makes an accent on the bottom-up processes involved in the origination of emotion. A number of later theories were inspired by the ideas of James and Lange. With this regard, it is necessary to mention the behavior theory developed by Ryle (1949), the account of Zajonc (1980) who pointed out that affect is a post-cognitive phenomenon, the perceptual theory of emotion proposed by Prinz (2004), the somatic marker hypothesis formulated by Damasio (2005), and the notion of Izard (2009) who sees emotion feeling as a phase of neurobiological activity. Although each of these theories is different from other ones in some aspects, they all to certain extent share a common idea that emotion originate in a bottom-up fashion. In other words, according to these accounts, bodily changes initiate generation of emotional states. There is also a number of theories that offer an alternative point of view on the origination of emotion. These theories were inspired by the CannonBard hypothesis that argued for the top-down generation of emotion (Cannon,1927). More specifically, Cannon suggested the idea that emotional experiences originate from cognitive processes in the mind. Similar approach was adapted by the theorists who developed appraisal theory of emotion. For instance, Arnold (1960) and Lazarus (1991) proposed ideas that all emotions are preceded by appraisal judgments. According to this account, changes in bodily states are the outcome of experiencing emotional states caused by appraisal judgments. Besides theories adopting bottom-up and top-down concepts of emotion generation, there are frameworks that integrate these two approaches. Scherer (2005b) defined emotion as “an episode of interrelated, synchronized changes in the states of all or most of the five organismic subsystems in response to the evaluation of an external or internal stimulus event as relevant to major concerns of the organism.” According to this definition, both bodily and cognitive components take part in the origination of emotion. Overall, one can conclude that the problem of defining emotion is not easy. It is a complex phenomenon that has an interconnected relationship with cognitive processes and bodily activations. Since the last decade, the role of emotion in human-computer interaction (HCI) has been becoming more and more important (Fairclough, 2009;Gunes and Pantic,2010). Unlike the conventional paradigms of HCI, the affective interaction takes into account emotional states of users, and therefore, brings a new modality to the communication channel. As emotional aspects are highly important in the interaction of people, Picard (2000) proposed that computers could also benefit from the capability to sense an affective state of a user, adjust its operation to the sensed state, and provide emotionally rich feedback. One of the difficulties on the way towards affective HCI is that there is no 2.2 emotion 15 straightforward way to structure and categorize emotions. Similarly to the situation with the definition of emotion, psychology has a number of theories that aim to explain the mechanism of affective processes in humans. Here we provide a brief review of three theories of emotion. They focus on structuring various emotional states and are frequently used in HCI applications that consider affective states of users. 2.2.1The Theory of Basic Emotions The theory of basic emotions is probably the earliest attempt to define and classify emotions. It states that there is a set of basic emotions that can be combined and used to describe any arbitrary emotion. The pool of basic emotions varies from one theorist to another. For instance, Ekman et al. (1982) proposed a set that consists of anger, disgust, fear, joy, sadness, and surprise, while the selection of James (1884) included only fear, grief, love, and rage. Several sets of basic emotions developed by different theorists is provided in Table 1to give a general idea about possible combinations. theorists basic emotions Damasio (2000)Happiness, sadness, fear, anger, surprise, disgust Ekman et al. (1982) Anger, disgust, fear, joy, sadness, surprise Frijda (1987)Desire, joy, pride, surprise, distress, anger, aversion, contempt, fear, shame Gray and McNaughton (2000)Rage/terror, anxiety, joy Izard (1993)Anger, contempt, disgust, distress, fear, guilt, interest, joy, shame, surprise James (1884) Fear, grief, love, rage Mowrer (1960) Pain, pleasure Oatley and Johnson-laird (1987)Anger, disgust, fear, happiness, sadness Plutchik (1980)Acceptance, anger, anticipation, disgust, joy, fear, sadness, surprise Weiner and Graham (1988)Happiness, sadness Table 1: A selection of sets of basic emotions. 16 state of the art Although the theory of basic emotions does not seem implausible from the folk theory point of view, it received a considerable amount of critique. The obvious weakness of this theory is related to the existence of several different sets of basic emotions. Indeed, if there are certain basic or fundamental emotions, then why scientists cannot agree on them? Moreover, the notion of a basic emotional state is questionable by itself because there is no exact definition for it. For a more detailed discussion of this issue please refer to (Ortony and Turner, 1990) and (Barrett,2006). Despite of the critique, the theory of basic emotions was embraced by many psychologists, albeit they sometimes eschewed the term ’basic emotions’ and preferred to say that some of emotions (e.g., being pleased, approving or liking) are easier to elicit than more complex emotions, such as reproach or pity (Ortony et al., 1990). The outcomes of this theory have been frequently used in the design of computer systems for recognition of emotions. The framework of basic emotions was particularly suitable for the cases when researches aimed at creation of the systems that were capable of identifying emotional states of users from a specific set (Kapoor et al.,2007; Resnicow et al.,2004;Lisetti and Nasoz,2004). 2.2.2The Dimensional Theory of Emotion Another widespread framework for representation of emotional states comes from the dimensional theory of emotion. The idea of reduction of a complex multidimensional phenomenon to a more simple representation that involves a low number of meaningful dimensions can be found in many fields of science. For instance, the position of an object in the 3D space may be described in various ways but the description can always be simplified to just three variables. Since the last century, psychologists have tried to develop a low-dimensional representation of human emotion, so that the representation is reasonably simple to work with, and at the same time, robust enough to cover a broad range of emotional states. Most commonly, dimensional theories of emotion propose to use an affective space with three dimensions (e.g., Osgood et al. (1975)) that has axes aligned with activation-arousal, evaluationpleasantness, and potency-control. However, many researchers of emotion preferred to utilize a two-dimensional space for description of the affect and emotion (Lang,1984;Russell,1979;Tellegen et al.,1999;Russell,2003). As can be seen at Figure 1, they used dimensions of arousal (relaxed vs. aroused) and valence (pleasant vs. unpleasant) to define an emotional state. Some theorists came up with a combination of the basic and dimensional theories of emotion. For instance, Plutchik (1980) considered that emotion is multidimensional and any emotional state can vary in intensity, similarity, and polarity but at the same time he also talked about eight basic emotions. There is a number of drawbacks in dimen- 2.2 emotion 17 Figure 1: The two-dimensional affective space where the horizontal axis depicts pleasure-displeasure and the vertical axis represents the level of arousal (taken from (Russell,1980, p. 1173)). sional theories as it was pointed out by Grandjean et al. (2008) and Fontaine et al. (2007). First, emotion researchers could not agree on the number and nature of the dimensions that provide an optimal framework for description of emotion. The next problem is that the reduction of complex emotional states to a low-dimensional representation leads to the failure in differentiating between certain types of emotions (e.g., anger being very close to fear). Furthermore, dimensional theories lack an explanatory mechanism that allows to predict patterns of emotional response. Overall, the dimensional theory of emotion seems to provide the framework that so far has found the widest application in the affective computing area (Gunes and Schuller,2013). It likely gained the popularity due to the simplicity in implementation, robustness, and the ease of integration with the approaches for recognition of emotion from physiological data. For examples of applications that rely on this framework please refer to (Cowie et al.,2000;Mandryk and Atkins, 2007). 18 state of the art 2.2.3The Appraisal Theory of Emotion Appraisal theories of emotion are younger than the basic and dimensional ones and are based on the pioneering work presented in (Arnold, 1960;Lazarus,1966,1991). It should be noted that, similar to other theories of emotion, the appraisal theories have a number of variations (e.g., structural and process appraisal models), but according to the goals of this thesis, only the most influential theories will be discussed. The basic assumption of the appraisal theories is that stimuli do not have an intrinsic value, rather the meaning of a stimulus, determined by a particular human in a particular context and at a particular moment of time, leads to elicitation and differentiation of emotions (Barrett, 2006). Appraisal theorists see the evaluation and the resulting emotion as a continuous and changing in time process, where the variability is caused by changes in environment and reappraisals of the situation (Scherer,2005a). The major structural components of the appraisal theories are four evaluation checks (Grandjean et al.,2008): 1. How relevant is this event for me? Does it directly affect me or my social reference group? (relevance) 2. What are the implications or consequences of this event and how do they affect my well-being and my immediate or long-term goals? (implications) 3. How well can I cope with or adjust to these consequences? (coping potential) 4. What is the significance of this even for my self-concept and to social norms and values? (normative significance) Another assumption made by some appraisal theorists is that the majority of emotional processes are unconscious and only some of them emerge into the consciousness for certain periods of time (Scherer, 2005a). It was suggested that processing of an emotional episode involves several synchronized processes and each of them is largely automated. However, if a regulation of these processes is required at the high level of cognition, the information about the emotional event has to, at least partly, emerge into the consciousness. This idea can be illustrated with Figure 2. Circle A represents a reflection of changes in the monitoring structures of a human nervous system. Circle B, only partially overlapping with circle A, expresses the part of emotion processing that enters conscious awareness of people. The last circle C depicts the ability of a person to verbally report the subjective experience during an emotional episode, and therefore, share it with other people including researchers who observe this individual. Although appraisal theories are still far in their development from the state where they can provide a comprehensive model of emotion, 2.3 archetypes and the collective unconscious 19 Figure 2: Three modes of representing the changes in the components of emotion: unconscious, conscious, and verbal (taken from (Scherer,2005a, p. 332)). they avoid some of the drawbacks associated with the two theories of emotion discussed above. Appraisal theories enable scientists to make highly specific predictions about the determinants that elicit and distinguish emotions. Moreover, a concrete procedure underlying emotional response pattering is suggested, and richness of emotion is addressed, allowing researchers to differentiate between emotional differences of individuals (Grandjean et al.,2008). To the best of our knowledge, the translation of the appraisal theory into an engineering framework remains a challenging endeavor (Sander et al.,2005), and therefore, it does not seem possible to provide examples of implementations that utilize the appraisal emotion model in affective computing applications. 2.3 archetypes and the collective unconscious Similarly to the situation with emotion, several definitions of the consciousness have been proposed but this concept still remains vague. Sometimes it is equated with attention, sometimes with the ability of verbal report, and sometimes operationalized in terms of the behavioral dissociation between different performance measures (Norman, 2010). Although the ultimate definition of the consciousness still requires further development, functions of the consciousness have been largely clarified and include reasoning, solving problems, learning languages, and so on. For the most part of human history, only the concepts of conscious thought and intentional behavior were consid- 20 state of the art ered in a scientific investigation. Contemporary psychological science remains dedicated to the conscious-centric model of the higher mental processes (Bargh and Morsella,2008). However, recent years have seen an increased attention being given to the unconscious aspects of the mind (Bargh and Morsella,2008;Dijksterhuis and Nordgren,2006; Gigerenzer,2008;Norman,2010;Rauterberg,2010). 2.3.1The Unconscious It has been demonstrated that there are higher order cognitive and affective processes to which individuals may have little or no direct introspective access (Wilson and Bar-Anan,2008). This fact seems surprising and controversial, but the experimental findings suggest that people are not very well aware of and not able to report on their cognitive processes (Nisbett and Wilson,1977). Thus, a considerable part of human experience is tied to a deeper level of psyche, which due to unavailability for conscious awareness is conceptualized as the unconscious (Wilson and Bar-Anan,2008). Since the phenomenon of the unconscious is still to be fully understood by the scientific community, there has not yet been an established definition developed. In order to avoid ambiguity and confusion, the unconscious mental processes have been operationally defined by Bargh and Morsella (2008) “in terms of a lack of awareness of the influences or effects of a triggering stimulus and not of the triggering stimulus itself”. This definition emphasizes the important distinction between unconscious and subliminal by resolving the common confusion about these two phenomena. People outside of psychological science often equate the unconscious with processing of stimuli, which are too weak or short to enter the conscious awareness, and therefore, are referred to as subliminal. In fact, the unconscious information processing is not necessarily associated with presentations of subliminal stimuli and runs continuously as a parallel background process in the human mind (Rauterberg,2010). Figure 3: Three cognitive systems: perception, intuition, and reasoning (adapted from (Kahneman,2003)). 2.4 physiological signals 27 dents Zhou et al. (2011); Wu et al. (2010). On the other hand, affective physiological computing is aimed at increase of pleasure in interaction with computer systems and is well suited for domains such as entertainment or computer-based learning (Stickel et al.,2009). Naturally, there is an overlap between these two branches of physiological computing (Novak et al.,2012) due to the fact that cognition and affect are interrelated in the human psyche. Psychophysiology attracted researchers from technical fields by providing a possibility to determine psychological parameters of users from the evaluation of tangible physiological data. It is worth to emphasize that most of the engineering systems utilizing emotion recognition based on physiological data referred to emotion models (sometimes implicitly) from the theory of basic emotions and the dimensional theory of emotion (Duric et al.,2002;Lisetti and Nasoz,2002; Kapoor et al.,2007;Maat and Pantic,2006). Nevertheless, physiological approach in emotion recognition has a number of obvious drawbacks, that make development of robust emotion recognition systems more challenging. The main weakness of this approach comes from the fact that physiological emotion recognition heavily depends on ANS. However this system is clearly not created exclusively for emotion processing, but is also in charge of keeping a human body alive. Therefore, sometimes it is difficult to say if a specific change in physiological parameters accounts for an emotional event or it is just a usual adjustment of the human body to, for instance, temperature in a room. Figure 5: Depiction of logical relations between elements in psychological (Ψ) and physiological (Φ) domains (Cacioppo and Tassinary,1990, p. 20). Panel A: Links between the psychological elements and individual physiological responses. Panel B: Links between the psychological elements and the physiological response pattern. Panel C: Links between the psychological elements and the profile of physiological responses across time. 28 state of the art A good discussion and analysis of the difficulties related to the inferring psychological meaning from physiological data was provided in (Cacioppo and Tassinary,1990). Here, the gist of their argument will be briefly outlined. First, let us introduce two sets, one is for psychological events (Set Ψ) and another one is for physiological events (Set Φ). Possible relations between events in psychological and physiological domains are presented at Figure 5. They are one-to-one,one-to-many, many-to-many, and null relation. In null relation the element in the psychological domain is not related to the element in the physiological domain. Naturally, only a one-to-one relation allows to infer psychological significance from physiological signals. Therefore, many-to-one and many-to-many relations should be simplified by redefining what constitutes an element in the psychological or physiological domain. For instance, any subset of elements in Set Φassociated with one or more psychological elements can be represented in Set Φ0by a single element representing a profile of physiological responses. However, search of one-to-one relations proved to be a difficult task (Strongman, 1996). While research in physiological computing has made a considerable progress in recognition of cognitive and affective states of the users, the investigation has been primary focused on conscious psychological states (Ivonin et al.,2013b). Thus, sensing a deeper level of human experiences defined by the unconscious processes remains a largely unexplored area. As it was stated above, there is an initial evidence from psychophysiology (Miller,1992) that the unconscious experiences of people can be indirectly assessed with their physiological signals. This fact implies that although the unconscious processes are hidden from the conscious mind, traces of the unconscious can be observed from bodily activations. There is a number of physiological metrics that hold promise in application for obtaining information regarding unconscious experiences of a person including galvanic skin resistance, respiration, blood pressure, temperature, heart rate, and electroencephalogram (Kreibig,2010). However, a further investigation is required in order to evaluate the feasibility of sensing the users’ unconscious mental processes in HCI scenarios by means of physiological computing. To the best of our knowledge, the state of the art in physiological computing does not provide a solution for digitizing archetypal human experience. 2.5 outlook As it was explained above, it may be possible to observe the archetypal experiences of individuals through measures of their physiological signals. This opportunity seems interesting and attractive because it would enable us to complement the capability of sensing human experience in the domain of conscious affective and cognitive states with 2.5 outlook 29 an additional facility for interpreting the archetypal experiences. Overall, it would mean that psychophysiological interfaces and applications could benefit from more detailed information about human experience. As the problem of digitization and transmission of human experience is essential in many application areas (see (Lahlou,2010) for an elaborate discussion), it seems that a technique for more detailed and deeper capture of human experience will not be excessive. 3 PSYCHOPHYSIOLOGICAL EFFECTS OF PERCEIVING AFFECTIVE PICTURES AND SOUNDS 3.1 introduction We began our investigation into understanding and capturing unconscious human experience with a study where a relationship between mental experience of people and their physiological parameters could be examined in a controlled environment. Since mental experiences of people are too broad and complex to start with, it was reasonable to focus on the individual components of one’s internal world. We proposed to first consider emotions because they are undoubtedly one of the most important components of one’s mental life. Moreover, it was demonstrated that emotions can be recognized from physiological signals of a human body (Villon and Lisetti,2006;Healey,2000;Cacioppo and Tassinary,1990;Fairclough,2009). This latter fact is important in the context of developing automatic tools for digitization of human experience. As it was illustrated in Chapter 2, questions about emotion are fundamental in psychology and play an important role in understanding mind and behavior (Barrett,2006). Recently, an interesting idea about this role was presented. According to this idea, emotion is a media of communication between the unconscious and the conscious in the human mind (Rauterberg,2010). Emotion is seen as the conscious perception of the complex mapping processes from the unconscious space into the low-dimensional space of the conscious. Although recent studies have proved that it is possible to recognize emotion based on physiological signals and this research direction looks rather promising (Picard,2010), the primary focus has been on conscious emotion, which people are aware of and can report. However, Berridge and Winkielman 2003 argued that emotion can be unconscious as well. According to Kihlstrom (1999, p. 432), explicit emotion refers to the person’s conscious awareness of an emotion, feeling, or mood state; implicit emotion, by contrast, refers to changes in experience, thought or action that are attributable to one’s emotional state, independent of his or her conscious awareness of that state”. This chapter is (partly) based on: Ivonin, L., Chang, H.-M., Chen, W., Rauterberg, M.: A new representation of emotion in affective computing. Proceeding of International Conference on Affective Computing and Intelligent Interaction 2012. pp. 337-343. Lecture Notes in Information Technology, Taipei (2012). Ivonin, L., Chang, H.-M., Chen, W., Rauterberg, M.: Unconscious emotions: quantifying and logging something we are not aware of. Personal and Ubiquitous Computing. 17, 663-673 (2013). 31 32 effects of perceiving affective pictures and sounds We believe that tools for capturing human experience should take into account both conscious and unconscious emotions. Unconscious emotions might even be of higher interest and importance than conscious emotions for some users of such tools because they are hidden from their conscious awareness. To the best of our knowledge, recognition of unconscious emotions has not been studied yet, and thus, it is necessary to investigate whether physiological signals can be used for this purpose. In our study, we focused on the signal of heart rate, which has been proved to be one of the physiological signals that are related to emotional states Palomba et al. (1997); Villon and Lisetti (2007); Gunes and Pantic (2010); Mandryk and Atkins (2007). Importantly for development of tools for evaluation of human experience in realistic scenarios, heart rate can be unobtrusively measured with wearable sensors. In emotion recognition studies, various sets of stimuli are utilized to elicit emotional states in participants. For this purpose, specialized databases of stimuli have been developed and validated (Lang et al.,2008;Bradley and Lang, 1999). However, in case of unconscious emotions such sets of stimuli have not been clearly identified yet. Therefore, in our study, we had to introduce archetypal stimuli (Gronning et al.,2007) as a new kind of stimuli that could be applied to evoke unconscious emotions. Another difference of our study from the large part of the previous work in this direction is that we targeted five different emotional states, while other studies tended to focus on a fewer number of emotions. For instance, according to van den Broek et al. (2009), most of the studies included three to four emotional states. Based on the aforesaid, an experiment was set up in a laboratory setting for elicitation of emotions (both conscious and unconscious) with visual and auditory stimuli and for measurement of any changes in heart rate of participants in response to presentation of the stimuli. Conscious emotions were included in the experiment for control purposes. This experiment should (1) clarify if different types of emotional stimuli evoke diverse heart rate responses and (2) if unconscious emotions can be recognized from heart rate. 3.2 visual and auditory affective stimuli In order to elicit emotional feelings under laboratory conditions, it is common to use visual and auditory stimuli. Based on the previous work in this field, there are publicly available databases with affective pictures and sounds that cover the most common emotions and have been successfully tested (Lang et al.,2008;Bradley and Lang,1999; Dan-Glauser and Scherer,2011). Unfortunately, there are no databases that contain stimuli that are capable of eliciting unconscious emotional experiences. In view of the aforementioned, for the experiment it was necessary to pick appropriate content from one of currenly available 3.2 visual and auditory affective stimuli 33 databases for elicitation of conscious emotions and to choose stimuli that are capable of evoking unconscious emotion. 3.2.1Affective Pictures and Sounds In the field of emotion research, International Affective Picture System (IAPS) (Lang et al.,2008) and International Affective Digitized Sounds (IADS) (Bradley and Lang,1999) are widely used to investigate the correlation between self-reported feelings of subjects and the stimuli that are demonstrated to them. In our study, IAPS and IADS were selected as the sources of experimental material due to the facts that these databases consistently cover the emotional affective space, have relatively complete content including pictures and sound clips, and provide detailed instructions about usage of the databases. Next, we had to identify the remaining stimuli for unconscious emotion. 3.2.2Archetypal Pictures and Sounds Jung (1981) postulated the concept of collective unconsciousness, arguing that in contrast to the personal psyche, the unconsciousness has some contents and modes of behavior that are identical in all individuals. This means that the collective unconsciousness is identical in all human beings and, thus, constitutes a common psychic substrate of a universal nature which is present in every human being. Jung further posited that the collective unconsciousness contains archetypes: ancient motifs and predispositions to patterns of behavior that manifest symbolically as archetypal images in dreams, art or other cultural forms (Jung,1964). According to Jung’s personal confrontation with the unconsciousness, he tried to translate the emotions into images, or rather to find the images that were concealed in the emotions (Jung, 1989). According to the record of Jung’s patients, archetypal symbols are essential for representation of one’s emotions at the unconscious level. Jung further argued that mandala (see Figure 6, sub-figures b and c), a circular art form, is an archetypal symbol representing the self and wholeness (Jung,1981,1989). The fundamental and more generic form of mandala consists of a circle with a dot in its center (Figure 6, sub-figure a). This pattern can also be found in different cultural symbols, such as the Celtic cross, the aureole, and rose windows. Since Jung’s argument, mandala drawings have been applied for practical use in the art and psychotherapeutic fields as basic tools for self-awareness, self-expression, conflict resolution, and healing (Bush, 1988;Curry and Kasser,2005;Kim et al.,2009a;Schrade et al.,2011; Slegelis,1987). Recent studies have discovered that mandala could be a promising tool for non-verbal emotional communication (Schrade et al.,2011;Elkis-Abuhoff et al.,2009;DeLue,1999;Cox and Cohen, 2000;Henderson et al.,2007). For patients with post-traumatic stress 34 effects of perceiving affective pictures and sounds (a) (b) (c) Figure 6: Different types of mandala: (a) basic form of mandala (Kazmierczak, 1990); (b) Buddha mandala (Wikipedia,2013d); (c) Mandala of the six Chakravartins (Wikipedia,2013c). disorder (PTSD), therapists can diagnose patients’ emotional statuses through the mandalas drawn by them while these patients are not willing or not able to discuss sensitive information regarding childhood abuse (Cox and Cohen,2000). Furthermore, in another case concerning breast cancer patients, mandala drawings, as a non-invasive assessment tool, allowed the physician to extract valuable information that may have been otherwise blocked by conscious processes (ElkisAbuhoff et al.,2009). The above studies have shown the potential of mandala to be a promising tool to convey unconscious emotions. Based on the work of Jung, the Archive for Research in Archetypal Symbolism (ARAS) was established (Gronning et al.,2007). ARAS is a pictorial and written archive of mythological, ritualistic, and symbolic pictures from all over the world and from all epochs of human history. Therefore, we assumed that the archetypal content of ARAS might enable us to elicit unconscious emotion and included the archetypal symbols in our experiment. Very little information is available about archetypal sounds. We ascertained that ‘Om’ and Solfeggio Frequencies are considered to be archetypal sounds (Wikipedia,2013f,a). ’Om’ or ’Aum’ represents a sacred syllable in Indian religions (Wikipedia,2013f). ’Om’ is the reflection of the absolute reality without beginning or end and embracing all that exists (Maheshwarananda,2004). Next, Solfeggio frequencies are a set of six tones that were used long ago in Gregorian chants and Indian Sanskrit chants. These chants had special tones that were believed to impart spiritual blessings during religious ceremonies (Wikipedia, 2013a). Solfeggio frequencies represent the fundamental sound that is both used in Western Christianity and Eastern Indian religions; therefore, we considered them as archetypal sounds. In addition to the archetypal symbols, we also included the archetypal sounds in our experiment to take into account the effect of auditory stimuli. Our expectations are that unlike the content of IAPS 3.3 materials and methods 35 and IADS the visual and auditory archetypal stimuli may evoke unconscious emotions. 3.3 materials and methods 3.3.1Participants Thirty-four healthy subjects, including 15 males and 19 females, participated in our experiment. Most of the participants were students and researchers associated with Eindhoven University of Technology in the Netherlands. The participants had diverse nationalities: 15 from Asia (China, India, Indonesia, and Taiwan), eight from Europe (Belgium, the Netherlands, Russia, Spain, and Ukraine), eight from the Middle East (Turkey and United Arab Emirates), and three from South America (Colombia and Mexico). The subjects had a mean age of 26 years and 9months, ranging from 18 to 50 years (one under 20 years old, two above 40 years old). The participants provided informed consent prior to the start of the experiment and were financially compensated for their time. 3.3.2Stimuli IAPS and IADS contain huge amounts of visual and audio stimuli, including 1194 pictures and 167 sound clips. Due to the limit of time and resources, we had to reduce the amount of materials for the experiment. To keep the validity of these two databases with the shrunk size, three selective principles were applied. First, four featured categories in the affective space of IAPS and IADS had to remain, they were Positive and Arousal (PA), Positive and Relax (PR), Neutral (NT), and Negative (NG). Second, the selected stimuli of each category had to reflect the original dataset. For example, the PA picture category mainly consisted of Erotic Couple, Adventure, Sports, and Food. Thus, the selected PA picture category should contain these clusters as well. Last, stimuli that can best represent the category should be selected first. For example, for PA category, the most positive and arousing content should be included first. The same criteria were used to select the materials for the fifth category Archetypal Content (AR), with the only difference that the distribution of Archetypal Content in the affective space is not yet defined. To sum up, there were two kinds of media, which were pictures and sound clips; each media contained five categories, which were mentioned above as PA, PR, NT, NG, and AR; each category comprised of 6stimuli (see Table 2). In total, the materials for the experiment included 30 pictures and 30 sound clips. The study followed the method used in IAPS and IADS (Lang et al.,2008;Bradley and Lang,1999). 36 effects of perceiving affective pictures and sounds Media Category Content Stimuli (Description and Code Number) Resource Picture Positive and Arousing Erotic couples, Adventure, Sports, Food EroticCouple (4652), EroticCouple (4668), Cupcakes (7405), Sailing (8080), Bungee (8179), RollerCoaster (8490)IAPS Positive and Relaxing Babies, Nature Butterfly (1605), Rabit (1610), Baby (2060), NeutBaby (2260), Nature (5760), Clouds (5891)IAPS Neutral Neutral objects, Mushrooms Mushroom (5530), RollinPin (7000), HairDrier (7050), Book (7090), Lamp (7175), Cabinet (7705)IAPS Negative Human threats, Animal threats, Accident, Disgust, Illness, Grief BurnVictim (3053), BabyTumor (3170), AimedGun (6230), Attack (6350), Vomit (9321), DeadMan (9412)IAPS Archetypal Archetypal images Mandala(3Hc.041), Mandala(3Pa.208), Mandala(5Ef.007), Mandala(7Ao.014), Mandala001 (Jung,1964), the Wheel of Life (Wikipedia,2013e) ARAS Sound Positive and Arousing Erotic, Gamble, Adventure, Cheering Crowds, Baby laughing Baby (110),EroticFem1(201), EroticCouple2(215), SportsCrowd (352), RollerCoaster (360), Casino2(367)IADS Positive and Relaxing Nature, Rain, Classical Music Seagull (150), Robin (151), Brook (172), Giggling (230), CorkPour (726), Beethoven (810)IADS Neutral Neutral objects or behaviors CountryNight (171), Yawn (262), Lawnmower (376), Rain1 (377), Clock (708), BrushTeeth (720)IADS Negative Screams, Crying, Accidents, Disgust, Animal Threats, Human Threats Bees (115),Vomit (255), BabiesCry (260), Femscream3(277), Victim (286), CarWreck (424)IADS Archetypal Archetypal sounds SF396Hz (Welch,2013), SF417Hz (Welch,2013), SF528Hz (Welch,2013), SF639 Hz (Welch,2013), SF741 Hz (Welch, 2013), Om_Meditation (YouTube,2013) Other Table 2: An overview of the stimuli used in the experiment. 3.4 results 43 Media Category Interval-2Interval-3Interval-4 Mean Std. Error Mean Std. Error Mean Std. Error Picture Archetypal -1.243 0.312 -1.468 0.293 -1.585 0.380 Positive and Relaxing -0.379 0.479 -0.601 0.501 -0.687 0.624 Positive and Arousing -1.732 0.393 -1.922 0.311 -2.132 0.331 Neutral -1.903 0.338 -1.938 0.307 -1.997 0.339 Negative -2.204 0.330 -1.830 0.313 -1.397 0.428 Sound Archetypal -2.336 0.362 -2.541 0.339 -2.682 0.398 Positive and Relaxing -1.367 0.383 -1.602 0.372 -1.806 0.468 Positive and Arousing -2.155 0.411 -2.221 0.395 -2.297 0.445 Neutral -2.169 0.305 -1.935 0.273 -1.734 0.361 Negative -2.323 0.335 -2.153 0.333 -1.842 0.385 Table 4: Changes in heart rate in beats per minute for different media and categories (negative values mean deceleration of heart rate) (N=34). 44 effects of perceiving affective pictures and sounds Figure 9: Changes in heart rate (HR) from the baseline, which is defined by the interval-1, for different types of media during the interval-3. Figure 10: Changes in heart rate (HR) from the baseline, which is defined by the interval-1, for different categories of stimuli during the interval3. ever, taking into account the large sample size (N = 2040), the results of Box’s M test can be neglected (Burns and Burns,2008). A combination of the discriminant functions revealed a significant relation (p = 0.019) between the predictor variables and the categories of emotional stimuli, and the classification results showed that 25.0percent of the original cases and 23.3percent of cross-validated grouped cases were correctly classified. Although these classification rates are not very high, they are still above the chance level, which, in the case of five categories of stimuli, equals to 20.0percent. Therefore, the achieved classification rates represent an improvement of 25.0percent for the original cases and 16.5percent for the cross-validated grouped cases in comparison to the chance level. 3.5 discussion 45 3.5 discussion The content of IAPS and IADS databases was successfully used to induce a range of emotions in a way that is similar to our experiment (Palomba et al.,1997;Winton et al.,1984). Therefore, we considered that every category of stimuli that comes from the databases would result in a unique combination of emotional feelings of the participants. Based on the literature review, we also expected that each of the categories of stimuli would result in a distinct pattern of physiological responses. In addition to the content of IAPS and IADS, the archetypal stimuli were included in the experiment. To the best of our knowledge no one has used the archetypal stimuli in the emotion research so far; hence, we did not exactly know what kind of physiological changes they might induce. However, we assumed that the physiological reaction should be different from the response to the IAPS and IADS stimuli. Based on the analysis of the data obtained with the SAM instrument, the differences between the five categories of stimuli were significant and their positions in the affective space were consistent with the previous research (Bradley and Lang,1999;Lang et al.,2008). Nevertheless, the distributions of points corresponding to Archetypal and Neutral categories were located close to each other in the affective space. This finding could be explained with the fact that participants were not able to consciously interpret emotional feelings associated with Archetypal category. In this case, they seemed to rate their emotional experience as neutral. The experimental results related to the physiological data enabled us to draw several other conclusions. The first and the most important conclusion was that the results confirmed our hypotheses regarding the unique pattern of physiological response for every (including archetypal) category of stimuli. Indeed, for both types of analysis that were performed, namely, for the analysis of the average heart rate during the demonstration of a stimulus and for the analysis of the change in average heart rate during the demonstration of a stimulus, the statistical tests showed a significant main effect of the category of the stimuli on heart rate. However, it is necessary to note that for the change of the average heart rate, the test was significant only for the interval-2, while for the average heart rate the test was significant for the interval2, the interval-3, and the interval-4. According to the previous research about the response of heart rate to emotional stimuli (Palomba et al., 1997), the absence of the significant main effect of category during the interval-3and the interval-4can be explained by the fact that the largest change in heart rate happens during the first two seconds of a stimulus presentation. Figure 11 illustrates the average heart rate changes measured on the interval-2with a reference to the interval-1. It can be observed that, 46 effects of perceiving affective pictures and sounds independent of the media type and the category of stimuli, heart rate exhibits a general decelerating response. This finding agrees with the previously reported results (Palomba et al.,1997;Winton et al.,1984) and is explained by the Laceys’ model (Lacey and Lacey,1970), which describes the effect of attention on heart rate. According to the Laceys’ theory of intake and rejection, the deceleration of heart rate occurs due to the diversion of attention to an external task, for instance, perception of a visual or auditory stimulus. On the other hand, when the attention has to be focused on an internal task and the environment has to be rejected, heart rate tends to accelerate. Figure 11: The deceleration of heart rate for different types of media and categories of the stimuli during the interval-2. The next conclusion is that negative stimuli evoked larger deceleration of heart rate in comparison to positive and neutral stimuli. This pattern is also in agreement with the previous studies (Winton et al., 1984) and is usually explained by the Laceys’ theory. The fact that our experimental results are highly consistent with the literature confirms again the validity of our study. We also analyzed changes in heart rate for different types of media and found that auditory stimuli lead to a higher speed of the heart rate deceleration in comparison to visual stimuli. From our point of view, the participants might consider sounds to be more significant and unexpected events because they could not influence the perception of an 3.5 discussion 47 auditory stimulus. For example, if a subject did not like a negative visual stimulus, she always could close her eyes and avoid the stimulus. However, she could not avoid an auditory stimulus in the same manner. Therefore, an intake of an auditory stimulus affected heart rate stronger than a visual stimulus. It is particularly interesting to look at the effect of the archetypal stimuli on heart rates of participants. According to the statistical analysis, there is a significant difference between the influence of visual and auditory archetypal stimuli on the heart rate of the participants during the interval-3(F(11,24) = 2875, p = 0.015 (Wilks’ Lambda)). Surprisingly, the archetypal sounds evoked even stronger heart rate deceleration than the negative sounds. This phenomenon is hard to explain; however, we have an idea that is based on one of the original purposes of the archetypal sounds, which is to support people in meditation practice. Indeed, to achieve a proper mental state during the meditation, people have to move away from their conscious experiences and free their mind from thoughts (Jain et al.,2007). This exercise is difficult because the conscious mind hardly can be idle. Therefore, people use the archetypal sounds (for instance, the famous ’Om’ sound) to keep the conscious concentrated on these sounds while they meditate. Then, one can infer that the archetypal sounds efficiently capture an attention of individuals. This, in turn, allows us to explain the strong heart rate deceleration with the Laceys’ theory. The pattern of heart rate change is influenced by the archetypal pictures to a lesser extent than by the archetypal sounds. This might be that, because for the revelation of archetypal features of the pictures, participants have to be deeply engaged in the contemplation of archetypal pictures. As mandalas and meditative sounds are religious symbols and, therefore, can possibly elicit conscious emotions in some people, one might question if the psychological responses of the participants to the archetypal stimuli was indeed unconscious. In order to investigate this question, it is reasonable to assume that people from Asia are more familiar with mandala than people from other regions of the world because mandala is a religious symbol in Hinduism and Buddhism. Therefore, a presentation of mandala to Asian people might elicit conscious emotional response. However, mandala does not appear in, for example, European religions, and, for this reason, it is unlikely that European people consciously know this symbol. As in our study we had participants from various geographical locations (15 from Asia, 8from Europe, 8from the Middle East, and 3from South America), it was possible to compare emotional responses to mandala between the participants who come from Asia and the participants who come from other regions of the world. Analysis showed that for the archetypal stimuli there is no statistically significant main effect of the geographical region (Asia or non-Asia) on changes in heart rate of the partici- 48 effects of perceiving affective pictures and sounds pants (F(11,23) = 1.724,p=0.13 (Wilks’ Lambda)). Therefore, we can conclude that emotional responses to the archetypal stimuli do not depend on the familiarity of the participants with these symbols. 3.6 conclusion and future directions As was pointed out earlier, we see emotion as an important component of one’s mental life. Recent research in the area of affective computing has demonstrated that emotional states of people can be recognized from their physiological signals (van den Broek et al.,2009). However, based on this research, it is not clear if unconscious emotions have corresponding patterns of physiological signals. Our study provided evidence, which implies that unconscious emotions also affect the heart rate. Moreover, deceleration of heart rate, which followed presentation of the archetypal stimuli to the participants, was different from deceleration of heart rate after demonstration of other stimuli. It is obvious that even theoretically heart rate alone will not allow precise classification of emotions because emotion is a multidimensional phenomenon, and heart rate provides just one dimension. Our experimental results support this point of view with the classification rate of 16.5percent above the chance level, which is lower than in some studies that focus on a fewer number of emotions and utilize various physiological signals in addition to heart rate (e.g., (Kreibig et al.,2007)). This result is probably best explained by the fact that we targeted many broad emotional categories, which considerably complicates recognition of emotion. Nevertheless, this study provided an important foundation for development of tools for evaluation of emotional experiences of people related to archetypes. Therefore, it is necessary to continue the work in this direction and further investigate the patterns of physiological responses to the archetypal stimuli. From our point of view, other features of the ECG signal could also be useful for measurement of emotions. Along with heart activities, other physiological signals, such as galvanic skin response and respiration rate, should be taken into account. Additional physiological signals are important because eventually they might allow establishing one-to-one links between the profile of physiological response patterns and emotional experiences across time (Cacioppo and Tassinary,1990). Another approach to improvement of the classification performance could be increasing the effectiveness of eliciting affective experiences. As Healey (2011) demonstrated, more precise affective labels lead to more differentiated physiological features. Therefore, improvements in the procedure of eliciting affective states are also likely to have a positive effect on the classification accuracy. 4 RECOGNITION OF THE ARCHETYPAL EXPERIENCE FROM PHYSIOLOGICAL SIGNALS 4.1 introduction During our first study reported in Chapter 3, the initial evidence was obtained that unconscious emotional experience of people is manifested in the patterns of cardiovascular activations. The statistical tests indicated a significant relationship between the categories of emotional stimuli and the features extracted from the ECG recordings of the subjects. Among the categories of the stimuli there were ones representing explicit emotions and one with stimuli for elicitation of unconscious emotional states. Nevertheless, that study was limited in several important aspects. The most significant limitation was discovered in the classification analysis. It turned out that the best classification accuracy achieved with five categories of stimuli was only around 23 percent. Although this result is better than the chance level, obviously, it is not sufficient for most of the practical applications. Next limitation was related to the variety of unconscious emotional experiences induced in the participants. There was only one category of stimuli that contained pictures and sounds selected for elicitation of an implicit emotional state. This category corresponded to the archetype of the self and represented only a very narrow band in the spectrum of possible archetypal experiences. For this reason, in our future research it was necessary to extend the number of archetypes included in the consideration. Based on the analysis of the first limitation outlined above, we concluded that the low classification accuracy may be explained with the fact that the visual and auditory stimuli were too weak or too brief in order to elicit a strong emotional response in the participants. Therefore, we proposed to use video clips for the elicitation of affective states in our next experiments. Our hypothesis that video clips would be more powerful in elicitation of emotional and archetypal experiences was supported by the literature in this field (Rottenberg et al.,2007). The limitation related to the variety of archetypal experiences also had to be addressed in the future studies. A straightforward approach for tackling this limitation was to develop a pool of stimuli that covers a broader range of archetypal experience. Having clarified the ways to This chapter is (partly) based on: Ivonin, L., Chang, H.-M., Chen, W., Rauterberg, M.: Automatic recognition of the unconscious reactions from physiological signals. In: Holzinger, A. et al. (ed.) SouthCHI 2013, LNCS 7946. pp. 16-35. Springer-Verlag, Berlin Heidelberg (2013). 49 50 recognition of the archetypal experience address the problems identified in the previous experiment we were set to conduct a new study. In the new study our primary goal was the evaluation of the possibility to sense the unconscious experiences of the users in an automatic and unobtrusive manner. However, as the unconscious is a complex phenomenon, the scope of our study was limited to the collective unconscious. Unlike the personal unconscious that is highly diverse and individual, the collective unconscious consists of the universal archetypes. For this reason, it is better suited for computing applications where a range of common archetypal experiences could be employed for system adaptation to psychological states of the users. More specifically, this study was aimed at investigating the feasibility of sensing and distinguishing various archetypal experiences of the users based on the analysis of physiological signals such as heart rate and skin conductance. Besides the patterns of physiological activations, we took into account introspective reports provided by the participants after confronting with stimuli. Our hypothesis for this experiment was that physiological data may provide a better reflection of archetypal experiences because, according to Jung (1981), archetypes are implicit and not readily available for conscious recollection. The archetypal experiences were elicited with film clips that were developed in collaboration with the ARAS (Gronning et al.,2007), which is an organization that since the early 1930s has been collecting and annotating mythological, ritualistic, and symbolic images from all over the world and possesses a profound expertise in archetypes and their representations. Apart from the film clips for elicitation of archetypal experiences, we also introduced clips to induce several explicit emotions. This way, we could later compare our findings with the state of the art on the explicit emotion recognition. The film clips were organized in categories in such a way that every category corresponded to one of the archetypes or explicit emotions. During presentation of the film clips, physiological signals modulated by the ANS of the subjects were monitored. We preferred to focus on the signals related to the ANS and avoid measurement of activations of the Central Nervous System (CNS) due to practical considerations. The CNS measures often impose considerable limitations on the design of studies. For instance, fMRI requires participants to be placed in a scanner. Another common CNS measure is EEG. While this is a more flexible approach than the usage of an fMRI scanner, the necessity of wearing obtrusive equipment on the scalp does not help subjects to feel natural and relaxed during interaction with products or media. In our study we monitored the following ANS signals: ECG, skin conductance, respiration, and skin temperature. After every film clip, the subjects were required to provide introspective report about their feelings using the SAM ratings. Upon completion of the study we used the collected data and information about categories of the film clips for training of several classifica- 4.2 materials and methods 51 tion models. Then, prediction performance of the models built using the introspective reports was compared with the models based on the physiological data. Furthermore, in the evaluation of performance we distinguished between models specific to the archetypal experiences and the explicit emotions. 4.2 materials and methods 4.2.1Experimental Design 4.2.1.1Stimuli An appropriate set of stimuli was required for the elicitation of the archetypal experiences in the experiment. According to Jung, symbolic representations of archetypes have been present across cultures for thousands of years. They were commonly used in artwork, myths, storytelling, and continue to be employed in modern mass media (Faber and Mayer,2009). Therefore, the set of stimuli can be constructed by extracting powerful archetypal appearances from a rich variety of media sources. However, a decision has to be made not just about which archetypes should be selected but also regarding the type of media to use. Past research in affect elicitation have applied different media types for emotion induction in laboratory conditions, including images and sounds (Bradley and Lang,1999;Lang et al.,2008), music (Eich et al.,2007), and films (Gross and Levenson,1995). These media types differ from one another in many aspects. For instance, still images and sounds are commonly presented to subjects for very short periods of time and have a high temporal resolution. On the other hand, music and film clips accommodate a lower degree of temporal resolution lasting for several minutes and deliver heterogeneous cognitive and affective activations. In comparison with the other types of media, film clips are powerful in capture of attention because of their dynamic display that includes both visual and auditory modalities (Gross and Levenson,1995). They also have a relatively high degree of ecological validity, meaning that their dynamic display resembles real life scenarios. Another characteristic of film clips is the ability to elicit intensive emotional responses that lead to activations in cognitive, experiential, central physiological, peripheral physiological and behavioral systems (Rottenberg et al.,2007). Taking into account the pros and cons of each media type film clips were chosen for this study because they effectively elicit emotions and last for several minutes. The latter fact was important for calculation of heart rate variability parameters that require at least 5minutes of data (Camm et al.,1996). With regard to the archetypal stimuli we assumed that the media with a high affective impact would also have a large influence on the collective unconscious. 52 recognition of the archetypal experience For this reason, film clips were utilized for the induction of both the explicit emotions and the archetypal experiences. A total of eight archetypes (anima, animus, hero-departure, heroinitiation, hero-return, mentor, mother, and shadow) were selected for this study. The number of chosen archetypes was a compromise: there were more interesting archetypes to study, but an increase in subjects’ psychological states would make the classification more challenging. For this reason, only films depicting the most common archetypes (Jung,1981) formed our pool of stimuli. The archetypes of anima, animus and shadow were chosen based on the work of Jung (1964). Three archetypes of a hero represent important stages in the hero’s journey described by Campbell (2008), who studied stories about heroes in myths, literature, and religion across cultures, places, and time. From his findings, he identified that a prototypical journey, which a hero undertakes in a narrative, includes stages of departure, initiation, and return. The archetype of mentor also comes from the research of Campbell and represents a character that helps the hero to acquire knowledge and power. Mother is another major archetype (Maloney,1999) that was picked for this experiment. Film clips that embody these eight archetypes needed to be selected. Similar to the previous studies that employed films (Rottenberg et al., 2007) we obtained our clips by extracting fragments from full-length commercial movies. However, our choices had to be evaluated and, if necessary, corrected by external experts in the area of archetypal research. Therefore, as it was mentioned earlier, we sought collaboration with the ARAS (Gronning et al.,2007). Based on the fruitful cooperation with ARAS and their feedback, our set of archetypal stimuli was constructed from the clips, which were obtained from the movies specified in Table 5. Copies of the film clips cannot be shared due to the fact that they were extracted from commercial movies. However, all of the movies are freely available on the market and the film clips can be easily re-created using the timings provided in Table 5. Apart from the archetypal stimuli, we also included in the study five film clips for elicitation of explicit emotions. The purpose of the stimuli for explicit emotions was to facilitate the comparison of psychophysiological responses to them and the archetypal films. Emotions or feelings are commonly represented in affective computing with the dimensional model (Russell,1980). This model projects emotions in the affective space with two or three dimensions. In case of two dimensions, an emotional state in the affective space is characterized by values of arousal and valence. The dimension of arousal ranges from calm to aroused states, while the dimension of valence ranges from negative to positive states (Ivonin et al.,2012). For this study five explicit emotions, amusement, fear, joy, sadness and neutral state, were selected. They uniformly cover the two-dimensional affective space. According to the previous work in this field (Lang et al.,1993), the 4.2 materials and methods 59 0.1Hz respectively in order to remove unnecessary noise. The Respiration Rate (RR) was calculated according to the recommendations provided by the manufacturer of respiration measurement hardware (TMSI BV). RR was then averaged over the duration of a film clip with a non-overlapping moving window of 10 seconds. Skin Temperature (ST) varies due to localized changes in the blood flow defined by vascular resistance or arterial blood pressure, which are consequently modulated by the sympathetic nervous system (Kim et al.,2004). The variations in ST caused by confrontation of individuals with affective stimuli have been previously reported in literature (Ekman et al.,1983) and justify our decision to include ST into this study. ST is a slow changing signal, thus, the raw data was harmlessly resampled to 64 Hz in order to speed up the calculation. A low pass filter of 10 Hz was applied to the resampled signal for removal of high frequency noise. Finally, ST was smoothed with a non-overlapping moving window of 10 seconds. 4.2.2Statistical Analysis As stated in the introduction, one of the motivations for this study was the question whether the patterns of physiological responses to various archetypal experiences are different and, furthermore, if the difference is statistically significant. We were also interested how physiological activations modulated by the explicit emotions of the participants are different comparing to their responses elicited by the archetypal stimuli. A number of statistical tests had to be conducted in order to answer these questions. Each subject watched all the film clips that formed our sets of stimuli for the explicit emotions and the archetypal experiences. Thus, the study had repeated-measures design where physiological measurements were made on the same individual under changing experimental conditions. An appropriate statistical test for this type of design would be MANOVA for repeated measures (O’Brien and Kaiser,1985). However, certain assumptions of this test were violated for some of the physiological signals’ features in our study. Namely, MANOVA does not allow inclusion of time-varying covariates in the model and an unequal number of repeated observations per an experimental condition. The former requirement could not be fulfilled because the physiological baselines that were introduced to the statistical model as covariates consisted of multiple data points. Although this assumption could easily be satisfied by transforming a number of data points into a single feature, we preferred to preserve the richness of our dataset and refrained from, for instance, averaging the baseline record. The latter prerequisite of MANOVA demands an equal number of repeated measurements per experimental condition. It could not be met due to the fact that the film clips presented during the experiment had slightly 60 recognition of the archetypal experience different length and, consequently, the size of vectors with physiological data varied. While all the clips lasted for approximately five minutes, there was a considerable difference between some of the stimuli. The shortest film clip had duration of four minutes and 46 seconds whereas the longest one was six minutes and 35 seconds. The limitations of MANOVA can be overcome if the statistical analysis is performed with Linear Mixed Model (LMM). LMMs are parametrical statistical models for clustered, longitudinal or repeated-measures data that characterize the relationships between continues dependent variables and predictor factors (West et al.,2006). LMMs have another advantage over MANOVA – they allow participants with missing data points to be included in the analysis. In contrast, MANOVA drops the entire dataset of a subject even if just one data point is absent. The general specification of an LMM for a given participant ican be defined as follows: Yi=Xiβ+Ziui+εi In this equation Yiis a vector of continues responses for the i-th subject and Xiis a design matrix that contains values of the covariates associated with the vector of fixed-effect parameters β. The Zimatrix is comprised of covariates that are associated with random effects for the i-th subject. The vector or random effects is assumed to follow a multivariate normal distribution and is denoted with ui. Finally, the εi vector represents residuals. A more elaborate introduction into LMMs can be found in, for instance, (West et al.,2006) or (Cnaan et al.,1997). A software implementation of statistical procedures included in SPSS Version 19 (SPSS, Inc.) was utilized to answer the research questions pointed out earlier. Physiological responses of the subjects were treated as dependent variables (continuous responses), the film clips represented fixed variables and the physiological baselines measured during the presentation of the video with a breathing pattern before each stimulus were used as covariates. The LMMs main effect tests whether the patterns of the participants’ physiological responses are different between various stimuli. The HRV features were analyzed with MANOVA as they met the requirements of this method. All statistical tests used a0.05 significance level. 4.2.3Data Mining Techniques The statistical analysis can enable us to determine whether or not it is possible to distinguish the archetypal experiences of people based on the patterns of their physiological activations corresponding to each of the archetypes. However, a statistically significant difference between the physiological responses associated with various archetypes does not allow evaluation of the practical feasibility to accurately predict psychological states of the participants. On the other hand, a prediction 4.2 materials and methods 61 model that maps physiological signals and the unconscious states of the users is what will enable development of systems for digitization of human experience. Thus, besides the statistical analysis, data mining techniques were applied to the dataset in order to obtain a predictive model that would facilitate evaluation of the classification accuracy among the archetypal experiences. 4.2.3.1Normalization Physiological signals vary highly between subjects, so the primary goal of normalization is to reduce the variance and make physiological data from different individuals comparable. As this study involved more than one participant, the data had to be normalized. There are different approaches to the normalization of physiological data (Novak et al., 2012). We chose to utilize the approach that involves subtraction of baseline values from the data corresponding to stimuli presentations. The result of the subtraction was then normalized to a range from 0 to 1for each subject separately. This normalization method was well suited for the design of our experiment where a baseline condition was recorded before each film clip. 4.2.3.2Selection of Classification Method According to the goals of this experiment, a classification method that takes a vector of physiological features and assigns a psychological label to it was required. Psychophysiological studies have utilized a number of different supervised classification methods and the selection of a particular method is generally defined by the type of stimuli, the design of the experiment, and the research objective. In this study, film clips with duration of approximately five minutes were shown to subjects and, thus, the physiological data that was recorded for each stimulus formed temporal sequences. The problem of time sequence classification is different from the recognition of static information due to the fact that a classifier has to learn the dynamic patterns of the data instead of its static attributes. There are three main types of sequence classification methods (Xing et al.,2010): • Feature based classification, which involves the calculation of features that describe time series and then the application of them in conventional classification methods for static data. • Sequence distance based classification. It relies on a distance measure that characterizes the similarity of time sequences. Frequently used distance functions include Euclidian distance and Dynamic Time Warping (DTW) distance. Once a distance measure is defined, a conventional classification method can be used. • Model based classification. This approach requires a statistical model, which is capable of learning data sequences, and it is 62 recognition of the archetypal experience the only method that does not reduce the time series classification problem to a representation amenable to classifiers for static data. The Hidden Markov Model (HMM) is the most popular instrument for temporal classification and widely applied in the speech recognition domain. We found the feature based method to be more suitable for a number of reasons. First, it provides a convenient way to include non-temporal attributes, such as some HRV features or the gender of subjects, into the analysis, which DTW and HMM do not (Kadous and Sammut,2005). Second, contrary to HMM, this method does not require a large amount of training data (Kadous and Sammut,2005). Third, the creation of template streams in the sequence distance method, which would represent a typical time series corresponding to psychological states, is not trivial. Once time sequences of physiological data were transformed into feature vectors, three types of classifiers were used in order to obtain a predictive model. These classifiers were chosen based on the history of their previous applications in recognition of emotions (Novak et al., 2012). kNN is a simple classification algorithm that is still reasonably accurate and can be useful as a baseline measure for the judgment of performance achieved with other methods. The second classifier was Support Vector Machine (SVM) that constructs a set of hyperplanes for classification purposes. The third selected classifier was naïve Bayes, which builds a probabilistic model based on training data and then assigns each feature vector to a particular class. The fourth classification method wasLinear Discriminant Analysis (LDA). This algorithm is transparent, meaning its results can easily be understood and interpreted by humans; it works well with small samples of data; and it is easy to implement (Novak et al.,2012). Finally, the fifth classification method was the C4.5algorithm for generation of decision trees. The decision trees were used in conjunction with Adaptive Boosting (AdaBoost) (Freund and Schapire,1997) in order to achieve higher accuracy. In order to ensure that a classification algorithm is not trained and tested on the same dataset, a cross-validation technique was employed. We chose leave-one-out cross-validation because it provides an accurate assessment of the classification rate. 4.2.3.3Extraction of Features An essential prerequisite of the classification is the extraction of feature vectors from data sequences. The main goal pursued by the extraction of features is a compression of data sequences to smaller sets of static features. The sliding window, the Discrete Wavelet Transform (DWT) and the Discrete Fourier Transform (DFT) (Agrawal et al.,1993;Chan, 2003;Geurts,2001) are three methods for conversion of time series to static data. The sliding window method performs best with low fre- 4.2 materials and methods 63 quency and relatively short time sequences because an increase of the signal’s frequency and length leads to generation of high dimensional feature vector. For long and high frequency data series the DWT and DFT approaches have been introduced. The idea behind these methods is the transformation of a sequence from the time domain to the timefrequency plane (DWT) or to the frequency domain respectively (DFT). Taking into consideration the aspects of our setup, the sliding window method for extraction of feature vectors was an appropriate way to prepare the dataset for the classification. Another name of this approach is segmentation since it first involves partition of a time axis into multiple segments with equal length and then averaging of temporal data along the segments (Geurts,2001). Having chosen the feature extraction method, we proceeded with the application of this method to physiological signals. A total of 38 features were obtained from ECG data: 30 features were extracted from HR temporal data based on the average value of HR calculated along non-overlapping segments of 10 seconds; eight conventional HRV features described earlier were taken without any modification (SDNN, RMSSD, SDSD, total power, VLF, LF, HF, and LF/HF). The skin conductance signal was compressed to a vector consisting of 60 features. First, 30 features were obtained from SCL, averaged over 30 segments of 10 seconds each. The remaining 30 features were extracted by averaging the absolute amplitude of SCR along the same 30 segments. Respiration data was reduced to 30 features that represented the average RR calculated for each of 30 segments. Finally, 30 features were obtained from the skin temperature signal, averaged in a similar manner over 30 segments. Therefore, in total 158 features were prepared for the classification. 4.2.3.4Dimension Reduction In data mining, it is important to identify an optimal number of features that are relevant for building an accurate prediction model. If the number of features is very large, the feature space volume also becomes vast, thus making the classification difficult. Moreover, if the training sample is not significantly large, overfitting may occur and lead to poor predictive performance. Therefore, it is beneficial to reduce the number of features as much as possible in order to build a computationally efficient and robust model for classification (Guyon and Elisseeff,2003). Various techniques have been proposed for feature reduction, including PCA. This method has been applied in psychophysiological studies and enable the transformation of a large number of features into a smaller set of uncorrelated components Novak et al. 2012. Dimension reduction with PCA transformed 158 features into 27 components. Overall, we had 25 samples per each class. There were eight classes that corresponded to the archetypes and five classes 64 recognition of the archetypal experience corresponding to the explicit emotions. The data of different subjects was combined in a single dataset. 4.3 results 4.3.1Statistical Analysis The initial motivation of this study was to explore the relationships between the archetypal experiences and their physiological correlations. The statistical analysis was to answer the question whether the archetypal experiences of the participants elicited with the film clips have a significant effect on their physiological signals. The features extracted from ECG, skin conductance,respiration and skin temperature recordings were arranged to form three types of datasets: one with the data for the explicit emotions, another with the data for the archetypal experiences and the unified dataset. LMMs were fit to each of the datasets with the HR features. The analysis, which the HR entered as a dependent variable, demonstrated a significant interaction effect between the film clips and the HR baselines for all the datasets: the explicit emotions dataset, [F(4,541.443) = 2.513, p = 0.041], the archetypal experiences dataset [F(7,1028.618) = 3.503,p=0.001] and the unified dataset, [F(12,1521.573) = 3.929, p <= 0.001]. As the HRV features were calculated over the whole duration of every stimulus and were represented with a single data point, they could be easily analyzed with MANOVA for repeated measures. This test showed a significant main effect of the film clips on the HRV parameters of the participants’ physiological responses for two of the datasets: the explicit emotions dataset, [F(32,329.811) = 2548, p <= 0.001 (Wilks’ lambda)] and the unified dataset, [F(96,1903.193) = 1987, p <= 0.001 (Wilks’ lambda)]. However, the same test for the archetypal experiences dataset was not significant, [F(56,872.323) = 1281,p= 0.085 (Wilks’ lambda)]. The relationship between the SCL features and the presentations of the stimuli was investigated with LMMs. The statistical tests indicated a significant interaction effect between the film clips and the SCL baselines for the explicit emotions dataset [F(4,2884,487) = 42.130, p <= 0.001], the archetypal experiences dataset [F(7,5880.869) = 38.795, p <= 0.001] and the unified dataset [F(12,9868.854) = 27.615, p <= 0.001]. Next, we ran analysis for the SCR features in a similar manner. A significant interaction effect between the film clips and the baseline covariates was discovered for the explicit emotions dataset, [F(4,707.582) = 13.473, p <= 0.001], the archetypal experiences dataset, [F(7,1391.923) =11.401, p <= 0.001] and the unified dataset, [F(12,2109.957) = 10.667, p <= 0.001]. 4.3 results 65 Then, we looked at the respiration data and performed tests with LMMs that were fit to the RR measurements. The interaction between the film clips and the baseline RR did not demonstrated significance for the archetypal experiences dataset, [F(7,1071.446) = 1.070, p = 0.380] and the unified dataset [F(12,1686.540) = 1.667, p = 0.068]. Nevertheless, the same test was significant for the explicit emotions dataset, [F(4, 611.304) = 2.931,p=0.020]. Finally, the features of the skin temperature recordings were analyzed. Again, LMMs built on the ST data were used for the statistical testing. However, we could not complete the analysis because the statistical software did not achieve a convergence within 100 of iterations. For illustrative purposes, the data of the HR signal that contributed to the discriminant functions the most is presented on Figure 14. The mean values and 95% confidence intervals of the HR are indicated for several of the archetypal stimuli. Figure 14: Heart rate responses of the subjects to the film clips. The mean values and 95% confidence intervals of the HR are represented with the bold lines and the vertical bars. 4.3.2Classification After the statistical analysis an evaluation of several predictive models was conducted. This evaluation was aimed at answering the question of how accurate the archetypal experiences can be predicted and classified by computational intelligence algorithms from the physiological data and introspective self-reports. Similar to the statistical analysis the classification was performed on three collections of data records: the explicit emotions dataset, the archetypal dataset and the unified dataset that integrated all the available data. Every selected classification method (kNN,SVM, naïve Bayes, LDA and AdaBoost with decision trees) was applied to each of the three datasets. The analysis on each of the datasets was performed using the extracted features as attributes. The archetypes or the explicit emotions presented in the clips served as class labels. The model constructed with the kNN method was able to correctly classify 26.5percent of the instances belonging to the archetypal dataset. 66 recognition of the archetypal experience However, the same classification approach resulted in the recognition rate of 30.4percent for the explicit emotions dataset and of 13.8percent for the unified dataset. The number of nearest neighbors equal to 20 (k= 20) lead to the optimal performance in all the cases. The SVM algorithm provided worse classification accuracy than kNN for the archetypal dataset (23.0percent). On the other hand it demonstrated better recognition rate on the explicit emotions dataset (36.8percent) and the unified dataset (17.2percent). The naïve Bayes approaches enabled us to achieve comparable performance on the archetypal dataset (26.0percent) and the explicit emotions dataset (27.2percent). The LDA classifier demonstrated one of the best results on the archetypal (29.5percent) and the explicit (36.8percent) datasets.The recognition rate on the unified dataset was higher with the LDA method (19.4percent) comparing to the naïve Bayes classifier (13.8percent). Finally, decision trees in conjunction with AdaBoost led to the poorest classification results: 19.0 percent for the archetypal dataset, 24.8percent for the explicit emotions dataset and 8.9percent for the unified dataset. A summary of the classification results is provided in Table 7. Dataset kNN SVM Naïve Bayes LDA AdaBoost Archetypal 26.5 23.0 26.0 29.5 19.0 Explicit 30.4 36.8 27.2 36.8 24.8 Unified 13.8 17.2 13.8 19.4 8.9 Table 7: Classification performance achieved with different methods for the archetypal, the explicit and the unified datasets of physiological features. The values are specified in percent. Following the analysis of physiological data, we looked at the SAM ratings provided by the subjects after viewing the film clips. Their reports included information about three factors that are often used in emotion modeling: valence, arousal, and dominance. According to the dimensional theory of affect, a combination of these factors is fairly sufficient for description of an emotional state of a person. Similarly to analysis of the physiological data, we had to train several prediction models to answer the question of how accurately the archetypal experiences and the explicit emotions could be predicted based on the participants’ reports. The LDA method was used for this purpose. Analysis of the data corresponding to the archetypal clips indicated that a prediction model could achieve a classification accuracy of 28.5percent. It was considerable above the chance level but could not outperform the results obtained using the physiological data. The same analysis was repeated for the explicit emotions. With the LDA classification method we could predict the explicit emotions based on the reports of the subjects with an accuracy of 52.0percent. A comparison of these findings 4.4 discussion 67 and the classification results obtained using the same method from the physiological data is presented in Table 8. Dataset Physiological Data Introspective Reports Archetypal 29.5 28.5 Explicit 36.8 52 Table 8: Comparison of the classification performance for the physiological data and the self-reports. The values are specified in percent. 4.4 discussion According to Jung, people share certain impersonal traits, which do not develop individually but are inherited and universal. He introduced the concept of archetypes in order to describe the contents of the unconscious psyche. While it is not clear whether Jungian model is valid, the notion of archetypes found applications in various areas of psychological science. In this study, the feasibility of capturing archetypal human experience through physiological data was evaluated. Moreover, a comparison of two approaches for identification of archetypes in human experience was carried out: introspective reports and physiological measures. 4.4.1Statistical Analysis A number of statistical tests were run on the collected data. Their outcomes gave evidence of a significant relationship between some of the physiological signals and the psychological conditions of the subjects. Whereas the patterns in three out of the four measurements reflected the induced explicit emotions, no association could be inferred from the skin temperature signal. These findings were anticipated and go along with the state of the art of physiological computing. Unfortunately, the skin temperature signal did not justify our expectations and, from our point of view, its variations are too slow to successfully contribute to the differentiation of emotions. The archetypal states of the participants demonstrated statistically significant relationship with the HR,SCL and SCR features extracted from the ECG and skin conductance signals. In comparison with the explicit emotions, the archetypal experiences lead to observable activations in a smaller number of the physiological features. Nevertheless, our results show that the patterns of physiological responses to various archetypal experiences are different and the difference is statistically significant. Furthermore, the analysis performed with the unified dataset, which integrated the explicit 68 recognition of the archetypal experience emotions and the archetypal experiences, supported the hypothesis of possibility to distinguish between these two types of stimuli. 4.4.2Classification Accuracy 4.4.2.1Physiological Measures The results of our experiment demonstrate that prediction models constructed with established data mining techniques and trained on the physiological data of the subjects achieved an accuracy that is considerably higher than the chance level. The models were obtained with five different classification methods (kNN, Naïve Bayes, AdaBoost,SVM, and LDA) featuring classification rates from 19.0percent to 29.5percent for the archetypal experiences. Explicit emotions could be recognized with an accuracy ranging from 24.8percent to 36.8percent. It is difficult to compare the results related to the archetypal experiences with the state of the art because there is not many studies that examined them from the psychophysiological perspective. However, we can directly set these findings against the observations obtained in the experiment that was described in Chapter 3. The comparison suggests that the modifications made in our approach to analysis of physiological signals and design of the experiment lead to considerable improvements in the classification performance. Prior to conducting this experiment, we hypothesized that two aspects may enable us to mitigate the limitations from the previous study. The first one was related to a more rigorous analysis of physiological data that takes into account a wider array of physiological signals and extracts additional features from these signals. The second aspect, in which this study differed from the previous one, was associated with the type of media utilized for elicitation of explicit emotions and archetypal experiences. We made a switch from using still pictures and sounds to presenting video clips for elicitation of affective states in the participants. Both of these modifications proved to be useful and facilitated the improvement in recognition performance from 23.3to 29.5percent. This comparison uses the data obtained from the unified dataset because, similar to the setting of the previous experiment, it includes both archetypes and explicit emotions. In order to have an additional benchmark, the results obtained for the explicit emotions can be set against previous studies that dealt with recognition of affect. Based on the review provided in (Novak et al.,2012), the predictive power of our models is on par with affect recognition studies in terms of classification accuracy. There are studies where higher accuracies have been reported, for instance in (Healey and Picard,2005;Picard et al.,2001;Sakr et al.,2010;Pantic et al.,2011) researchers were able to achieve classification precision of up to 97.4percent. While we acknowledge their accomplishments, it is 5.1 introduction 75 Although psychophysiological measurements seem to offer several advantages over self-reports, their adoption in practical scenarios is held back by two significant limitations. First, application and interpretation of physiological measurements is not trivial and requires researches to have background knowledge in the processing of physiological signals and classification algorithms. Second, previous studies demonstrated recognition accuracy for various sets of emotions of around 60-80 percent (Novak et al.,2012). Clearly, the recognition rate has to be improved in order to allow a wider application of psychophysiological techniques in practice. We aimed at addressing the first limitation in more detail and leaving the second limitation for future research. The task of identifying patterns in physiological activations of the ANS requires a significant investment of a researcher’s time. Generally, this process includes the following steps that have to be performed (Novak et al.,2012): recording of physiological signals, extraction of features, reduction of dimensionality, and classification or estimation. Each of these steps demands theoretical competencies and practical skills that are specific to the domains of psychophysiology (Cacioppo,2000) and affective computing (Picard,2000). Therefore, an application of psychophysiological measurements in evaluation of human experience requires allocation of significant research efforts on technical aspects that are not directly related to the primary research question. Our goal was to develop and validate a tool that would simplify and streamline the evaluation of affective experience based on psychophysiological observations. The steps of the measurement procedure should be integrated, and if possible, automated into a straightforward and easy-to-use instrument. Moreover, the proposed tool should address another limitation of psychophysiological measurements that has not been mentioned yet. This limitation is related to the fact that, traditionally, physiological measurements require a laboratory environment. While there were notable exceptions (e.g. (Healey and Picard,2005; Plarre et al.,2011)), most of the studies were performed in laboratory settings that allowed monitoring of physiological signals in appropriate detail using stationary equipment. On the other hand, researchers may want to investigate emotional experiences of people in ecologically valid, realistic environments (Maly et al.,2011). Therefore, the proposed tool should be suitable not just for stationary settings but also for naturalistic and mobile scenarios. The development process of the proposed instrument for evaluation of human experience in realistic contexts is reported in this chapter. The initial evaluation of the instrument will be presented in the next chapter. 76 archesense:a tool for evaluation of affective experience 5.2 related work As it was pointed out above, there seem to be a necessity in an instrument that would remove some of the limitations that are associated with evaluation of human experience using psychophysiological measurements. The first step in development of such a tool was to make an overview of existing tools that may to certain extent address the problems that we identified. A quick glance at the literature in the fields of human-computer interaction and consumer psychology reveals that there are many tools (Hole and Williams,2007;Desmet,2005;Huisman et al.,2013;Isbister et al.,2006;McDonagh et al.,2002;Bradley and Lang,1994) that are aimed at supporting researchers in evaluation of human experience. For instance, the SAM (Bradley and Lang, 1994) that was mentioned earlier is a common instrument for measuring emotional arousal, valence, and dominance. Another example is the Product Emotion Measurement Instrument (Desmet,2005) that enables measurement of how individuals emotionally respond to products. Most of these tools, however, are based on verbal or non-verbal self-reports. Therefore, they are not directly related to psychophysiological observations and cannot help one to carry out an evaluation of emotional states using physiological signals. If we look specifically for the instruments that facilitate evaluation of affective experiences based on psychophysiology, the choice is scarce. This circumstance is likely explained by the fact that in comparison with introspective reports physiological observations is still a considerably novel approach for evaluation of human experience. It offers interesting advantages over introspective reports but has not reached a wide adoption yet. Next, we will provide a review of the existing tools for evaluation of human experience from physiological data. The Affective Evaluation Studio (Perakakis and Potamianos,2013) is tool for evaluation of user experience with visual interfaces using EEG signal. Other physiological signals are currently not supported. It facilitates usability research by providing detailed information about affective experiences of users. The tool has limited functionality and is in the early stage of development. A personal computer is required to collect and process data, and therefore, this application is not particularly suitable for mobile environments. Data mining and classification algorithms are not included in the Affective Evaluation Studio. There is no information available about the accuracy of this tool. Finally, it is not publicly available. The IVE (Maly et al.,2011) is a tool for evaluation of user behavior in realistic environments. It also enables researchers to analyze physiological signals, such as GSR and HRV. The collected physiological data can be used to track the level of stress. This tool is particularly useful for a post-hoc analysis that involves data sources of different nature (for instance, physiological data and video recordings). On the other 5.2 related work 77 hand, the IVE tool does not provide any support for collection, processing and classification of physiological data. In fact, the developers of this tool recommended a third-party application for extraction of HRV features. While this tool has a dedicated webpage with description and screenshots, the source code and executable files seem to be unavailable for public access. PsychLog (Gaggioli et al.,2011) is a mobile phone platform built for collection of information about psychological and physiological states of users. The primary application of this platform is research in mental health. Using PsychLog researchers can administer self-report questionnaires, record cardiovascular activities of users, and accelerometer data. The data collection function has an excellent implementation in PsychLog. This tool establishes connection with an ECG sensor and receives physiological data wirelessly by means of Bluetooth transfer. Such an approach is particularly well suited for studies that are to be performed in mobile environments. PsychLog also has a sensing module that processes ECG in order to locate positions of heart beats. Unfortunately, the next steps of analyzing ECG data cannot be performed on a mobile phone. The data needs to be exported to a workstation where further processing and classification algorithms should be applied using Matlab (MathWorks, Inc.) computing environment. Therefore, this tool does not facilitate the full sequence of analysis steps and is mainly focused on collecting of ECG data. PsychLog is an open source application that is freely available for the general public. The Smart Sensor Integration (SSI) framework (Wagner et al.,2009) was developed to support online and offline emotion recognition by offering dedicated tools for data segmentation, extraction of features, and classification. The SSI is not restricted to work with a particular type of input signal, and therefore, it does not offer capabilities for recording physiological data. The data has to be collected by a researcher and imported into the framework. Therefore, the SSI covers the complete pipeline of identifying patterns in physiological activations except recording of physiological signals. Although the SSI framework was designed to support any kind of physiological signals, we only were able to locate a report describing implementation of this framework for recognition of emotional speech. This framework was designed for workstations and cannot be run on mobile devices. The SSI framework is publicly available including the source code. Based on the overview of the existing tools for evaluation of human experience, we concluded that none of these instruments completely satisfied the requirements formulated in the Introduction section. Only two of the tools are publicly available: PsychLog and the SSI. None of them allows truly mobile scenarios because at some stage physiological data has to be transferred to a workstation and processed by researchers or engineers with third-party tools, such as Matlab (MathWorks, Inc.). In fact, a combination of PsychLog and the SSI framework 78 archesense:a tool for evaluation of affective experience could be very close to the formulated requirements if PsychLog supported recording of more than one physiological signal and the SSI framework was designed for mobile platforms. Overall, it is clear that existing instruments cannot efficiently address the limitations that are associated with evaluation of affective experience using psychophysiological measurements. Moreover, some of the tools we mentioned were not built with this particular goal (e.g., PsychLog was made for monitoring of mental health). For this reason, we made a decision to proceed with development of a new tool that should simplify and streamline evaluation of affective experience with integrated features of mobility, portability, recording and processing of physiological signals. 5.3 design considerations Our goal was to develop an instrument that provided a solid technical framework for performing recognition and analysis of physiological signals to evaluate affective experience. The tool should follow the principle “it just works” in order to be accessible for a broad community of social scientists and other people who do not have a particularly strong technical background in signal processing. Moreover, it should provide a user interface suitable for mobile environments. Before implementation of the tool, it was important to take into account several design considerations. They would provide a foundation for further implementation of this instrument. 5.3.1Representation of Emotional States One of the first questions that have to be answered in the design of a tool for evaluation of affective experience is the representation of emotion. As we explained in Chapter 2, one of the earliest theory to define and classify emotions suggests that there is a set of basic emotions which can be used to describe any arbitrary emotion. The set of basic emotions varies from one theorist to another (Ekman et al.,1982; Frijda,1987;Izard,1993;James,1884;Oatley and Johnson-laird,1987; Plutchik,1980). Nevertheless, some of them share popular emotions like fear or anger. The basic emotions theory has been frequently used in the design of computer systems for emotion recognition (Lisetti and Nasoz,2002;Kapoor et al.,2007;Resnicow et al.,2004). An advantage of representing affective states using a set of labels or categories is that this way of representation can be easily extended to more general psychological states of people (e.g., related to cognition or motivation). Therefore, the proposed tool with such a representation of emotional states could be used in a larger number of scenarios. Finally, such a representation would suit the design of our next experiment. 5.3 design considerations 79 Figure 15: The two modes of operation supported by the tool. 5.3.2Training and Evaluation Modes The tool should support at least two modes of operation: training and evaluation. In the training mode, researchers would be able to train a prediction model that could be later used for recognition of psychological states. In this mode, the tool should guide researchers through the training process starting from defining parameters of the model, such as expected psychological states, durations of stimuli for elicitation of these states, number of stimuli per each state, and so on. When the basic parameters of the model are set, the tool should facilitate the collection of physiological data from a number of individuals while they are exposed to the stimuli defined in the model. As soon as the physiological data is collected from a sufficient number of subjects, the tool should offer researchers a straightforward way to build the prediction model based on this data. In the evaluation mode, researchers would be able to perform evaluation of affective experience using one of the prediction models trained beforehand. The tool would perform online analysis and classification of physiological data streamed from the sensors. It would also visualize the outcome of recognition in real time. The evaluation and training modes have to be separated because in a general case researchers would not have a prediction model suitable for their scenario. Therefore, such a model has to be first built in the training mode and later applied in evaluation of human experience. In order to explain the modes of operation supported by the tool two flowcharts were prepared. At Figure 15 these flowcharts illustrate how the tool should operate when it is in the training mode and in the evaluation mode. As it was explained above, the training mode serves the purpose of obtaining a prediction model suitable for recognition of a particular pool of psychological states. Researchers start with defining parameters of the model, and then, conduct individual sessions with subjects for collection of physiological data corresponding to the elicited psychological states. When a sufficient amount of data is obtained the 80 archesense:a tool for evaluation of affective experience researchers will train the prediction model using one of the available algorithms. The evaluation mode of the tool should enable researcher to get an estimation of the individuals’ psychological states in a real-time manner. In this mode, researchers would expose the subjects to a product or media, to which their affective responses have to be evaluated. To certain extent this mode of operation is similar to the training mode. In both cases, the tool would record the physiological data from the sensors attached to the participants. The online filtering and processing of physiological signals would also take place in each of these scenarios. The difference is that in the evaluation mode the collected data would be fed to the previously trained model in order to obtain a prediction regarding the current psychological state of the individual. The outcome of the predication should be visualized in the user interface of the tool and stored on the disk for future evaluations of this experimental session. The tool may support different types of visualization. Along with conventional bar chats it could employ pictorial resources representing the psychological states of the participants. 5.3.3Sensor Integration A seamless integration with the sensors for monitoring of physiological signals is essential for the proposed tool. The tool would need to have an interface for acquiring data from one of the commercial sensor platforms publicly available on the market. Moreover, the tool should implement a number of commands for configuring the sensors. For instance, it may be necessary to deactivate certain sensors or set a different sampling rate. It would be beneficial to ensure that the proposed tool supports wireless exchange of information with sensors for physiological signals because it would increase portability of the tool and allow for more versatility in experimental designs. 5.3.4Signal Recording and Features Extraction The physiological data obtained from the sensors has to go through initial processing procedures and stored on the disk for later retrieval. Depending on the type of a physiological signal, it may be necessary to apply several filters in order to eliminate noise. An ECG signal, for instance, commonly requires cleaning with low-pass, high-pass, and notch filters (Camm et al.,1996). If the processing power of the tool is sufficient to carry our filtering in real time, it would be advantageous to perform this task immediately after reading the input from the sensors. In this way, the physiological data saved on the disk will be clean from noise and artifacts. Besides filtering, certain physiological signals require an extraction of features. Again, we use the ECG signal as an example. It is a raw physiological signal from which a number 5.3 design considerations 81 of features, such as mean heart rate for a given time interval or various measures of heart rate variability, can be extracted (Novak et al., 2012). Similarly to filtering, it could make sense to perform extraction of certain features in a real-time manner because researchers may want to see an online visualization of these features (e.g., momentary heart rate). Moreover, the raw physiological data will not be directly used for training of prediction models. For this reason, the disk space can be saved by storing only the series of extracted features. Obviously, an option for saving a backup copy of the raw physiological data is also required. Finally, it is necessary to visualize in the user interface the input received from the sensors and, in certain cases, extracted features in order to enable a quick discovery of artifacts and malfunction of the sensors. 5.3.5Model Training As soon as researchers collected a sufficient amount of physiological data they can proceed with training a prediction model that would map physiological signals and psychological states of the individuals. The tool should have a dedicated functionality for an overview of the collected data. In the overview area, researchers should be able to conveniently inspect the data, manipulate the dataset with various data mining techniques, and apply one of the available classification algorithms in order to obtain the prediction model. There should be a possibility to evaluate performance of the model either using crossvalidation or holdout validation. When researchers are satisfied with the prediction model, the tool should offer them a capability to save it on the disk for later use. 5.3.6Storage Throughout the possible usage scenarios the tool has to provide functionality for saving the data permanently on the disk. Most importantly, recordings of physiological signals after filtering and initial processing have to be stored for future analysis. Although this analysis will normally be facilitated by the tool itself, researchers may also prefer to use other external applications. For this reason, the storage format must be open and transparent. In comparison with desktop computers portable devices have limited disk space. Due to this circumstance, the tool has to ensure that the data files are lightweight and do not embed redundant or unnecessary information. 82 archesense:a tool for evaluation of affective experience 5.3.7Export of Data Researchers should have a possibility for exporting the data from the tool. The exported data could, for instance, be used in third party instruments or shared with peer scientists. Moreover, a website could be set up where prediction models trained by different researchers could be uploaded and shared. Each model would be accompanied with the meta-data including information about the psychological states this model includes and the demographics of the individuals whose physiological data was used for its training. Obviously, the uploaded information has to be free from any personal data of the subjects. There are two occasions in the workflow of the tool when the export function is particularly relevant. First, after completion of the training procedure it is appropriate to offer researches to export the recorded physiological signals and the prediction model. Next, after completion of an evaluation session is another occasion when researchers are likely need to extract the data from the tool. The proposed tool could offer several options for the export, such as manipulation of the file system of the device, e-mail, or cloud storage. 5.4 system architecture Based on the design considerations formulated above we proceeded with definition of a high-level architecture for the proposed tool. It is important to start the development process with outlining a macroscopic architecture because early design decisions that will considerably impact the tool’s implementation depend on a comprehensible abstraction of the system. Our main design requirement was that the tool should be usable while demanding no or very little technical knowledge of physiological signal processing. In other words, the proposed tool should “just work” and save researchers from the necessity to invest much time into setting up of a technical infrastructure for measurement and analysis of physiological data. Therefore, the architecture of the tool should include components that cover all the activities in the pipeline of physiological data processing. Figure 16 shows the architecture that was developed. Similarly to the approach taken in design of context-aware applications (Chen and Kotz,2000), in the proposed architecture, we decoupled the part directly related to sensing from other components of the system. This decision was motivated by the goal to maintain flexibility by providing support for various types of physiological sensors. One or more sensors will interact with the tool through the driver that would provide control over the hardware through software calls. When the tool invokes a function in the driver, it will issue a sequence of commands to the connected sensors. For instance, there could be a function to start streaming physiological data or change the sampling frequency. 5.4 system architecture 83 Figure 16: The architecture of the proposed tool. Once the driver receives data from the sensors, it may send a message to the tool with enclosed data packets. In order to ensure high speed of data exchange the driver will likely have to be hardware-dependent and specific to the operating system of the computer hosting the tool. The physiological data from each of the sensors will be aggregated in the Data Acquisition component of the tool. This module will, if necessary, convert the raw data coming from the driver into an appropriate format and make adjustments in the case when the sensors are streaming at different frequencies. Optionally, users of the tool may choose to enable the backup procedure that will save the raw physiological data to the local data storage. From the Data Acquisition module the data will be passed to the Signal Processing module. In the Signal Processing component of the tool, the physiological data will be treated with various techniques for filtering and removal of artifacts. An appropriate set of techniques will be chosen based on the nature of a physiological signal. Next, algorithms for features extraction may be applied to the data. For instance, if one of the sensors connected to the proposed tool provides measurement of respiration activities via an inductive belt, it will be necessary to obtain respiration rate from the raw signal. When processing of the signals is complete and the tool operates in the training mode, the signals will be saved to the data storage and used again later for training of a prediction model. If the tool is in the evaluation mode, an estimation of the current psychological state will be performed based on the physiological data and chosen prediction model. The Analysis module of the tool contains a number of data mining and classification methods. The data mining methods could be used 84 archesense:a tool for evaluation of affective experience in order to prepare the collected physiological data for training of a prediction model. In most of the cases, it could be beneficial to reduce dimensionality of the obtained vectors of physiological features. When training of a new model is completed this module may offer the users some statistics about the training and performance of the model in the evaluation. Once researchers are satisfied with the prediction model, it could be saved on the disk and will become available for usage in the online evaluation mode. The proposed tool should not be a closed system. While it offers broad capabilities for recognition of psychological states from physiological data, there may be situations where the collected data needs to be analyzed externally or shared with other researchers. Therefore, the Export module was included in the architecture of the system. This module will contain several routines that enable exchange of data with external environments. Various parameters that define the exact behavior of the tool in different scenarios will be maintained in the Configuration module. The Session Management module will be offering guidance to researchers when they need to collect data for training of a model or perform an evaluation. For instance, it could control timings of the stimuli presentation and the sequence in which they have to be demonstrated. 5.5 data model As the proposed tool has to frequently store large amounts of data on the disk, a data model will play an important role in the overall performance of the tool. During the development of a concept for the data model we faced a dilemma. From one point of view, it is necessary to ensure that the data model allows for extremely fast writing on the disk because the sensors stream physiological signals in real time and their sampling frequencies may be high. From another point of view, the data model should present the collected data in a structured and transparent way because it is necessary for an easy interpretation of the data outside of the tool. Therefore, it may be beneficial to have several types of data containers for different purposes. Some of the data containers may be temporary. For instance, the stream of physiological data from the sensors could be temporarily saved into a data container with a simple structure. Depending on the hardware capabilities of the device hosting the tool, this data container could be encapsulated in a text or binary file. If possible, the preference should be given to text files because they are easier to understand and users can make more sense of them. When recording of physiological data into a temporary data container is complete, the data could be transferred to a permanent data container with a well-defined structure. The temporary container should be dismissed after this point. 5.6 implementation 91 the sensors sends messages with new chunks of data to this thread’s message queue. The messages are processed as soon as the message queue is ready to do so. ECG signal is treated by the application with low-pass, high-pass, and notch filters. Filtering was necessary to remove high frequency noise (above 100 Hz), low frequency components, such as respiration (below 0.5Hz), and mains hum (50 Hz). ECG is a rich signal, and in the psychophysiological domain it is commonly used for the derivation of the HR and HRV. The heart rate is simply a measure of the number of heart beats per minute (Neuman,2010). It is extracted from the ECG signal by detecting beats with an algorithm described in (Afonso et al.,1999). The tool also calculates the average heart rate over a nonoverlapping moving window which duration is set as one of the parameters of a model. As we explained in the previous chapters, skin conductance characterizes the electrodermal activity of skin and is related to changes in eccrine sweating, which are regulated by the sympathetic branch of the autonomic nervous system. Skin conductance consists of tonic and phasic components. The tonic component corresponds to relatively slow changes in skin conductance over longer time intervals, which can last from tens of seconds to tens of minutes. It is indicative of a general level of arousal, and thus, is called a SCL. On the other hand, the phasic component or SCR reflects high frequency variations of the conductivity and is directly related to observable stimuli (Figner and Murphy,2011). In order to obtain SCL from the raw skin conductance signal received from the sensors, the tool set a low pass filter to 1Hz. For SCR, a high pass filter was additionally applied at 0.5Hz. 5.6.3Training of Prediction Models Training of prediction models requires implementation of data mining techniques. Previous research in affective computing and psychophysiology demonstrated that a variety of methods could be used for training of models. It is not easy to say which methods are better because the answer to this question critically depends on the properties of the data, the goal of the study, and the desires of a researcher (Novak et al.,2012). Techniques giving a good result in one scenario may not be the optimal choice in another situation. For this reason, it will be beneficial if the proposed tool offers capabilities for an easy integration of widespread data mining methods according to the requirements of researchers. Such functionality will require the tool to have implementations of these methods. Although we could invest our time into implementation of major algorithms for reduction of dimensionality and classification, it does not seem like a good approach. First, it would require a big commitment in terms of time and development resources. Second, when development is complete, the algorithms would have to 92 archesense:a tool for evaluation of affective experience be thoroughly tested. This would demand equal or even large amount of time as development. Therefore, we sought an opportunity to re-use implementations of the algorithms from popular open-source software suits. Our particular interest was in machine learning software written in Java programming language because ArcheSense was also built using this language, and therefore, integration would be simpler. One of the most widespread software workbenches offering researchers easy and free access to state-of-the-art techniques in machine learning is WEKA (Hall et al.,2009). WEKA seemed to be an excellent choice for our purposes because it is written in Java and has achieved recognition in the data mining community for its collection of algorithms. Successful attempts of porting WEKA to Android have already been reported (Liu et al.,2012). One of the implementations of WEKA for Android was integrated in ArcheSense. This implementation contains most of the major data mining algorithms that could be necessary in the context of psychophysiological studies. Before an algorithm can be used, it has to be added to the GUI of the Analysis screen. As of writing, the GUI of ArcheSense provides access to implementations of two data mining algorithms: PCA (Martinez and Kak,2001) and kNN.PCA can be effectively used for reduction of dimensionality. kNN is one of the simplest classification algorithms that can be used in a prediction model. Nevertheless, it has gained considerable popularity in psychophysiology (Novak et al.,2012). We released the first version of ArcheSense with only few data mining techniques available. Although more available algorithms are likely to be appreciated by users, our main research objective was to develop and validate a tool for evaluation of affective experience. Therefore, we needed a minimum viable prototype of such an application. The prototype does not necessarily have to provide support for dozens of data mining techniques in order to test the proposed framework. In the future, we will be gradually expanding the pool of available algorithms. If ArcheSense finds traction among researchers, they may contribute their own techniques as well. 5.6.4Data Storage ArcheSense implements the data model that was described above. It uses JSON for storing information about models and associated physiological data. Each model is saved in a separate file. Also, there are temporary files that are used exclusively during the online recording of physiological signals. The temporary files have a simple structure where the data received from different sensors is saved in a plain-text form. Prediction models trained using routines exported from WEKA are stored in separate files generated with serialization functions. The data files are organized in folders located at the shared external storage of a tablet computer running Android. The data collected with 5.7 evaluation 93 ArcheSense can be easily exported from the tablet computer either by e-mail or using a USB cable connected to a workstation. ArcheSense provides export of data in JSON and ARFF (Hall et al.,2009) formats. 5.7 evaluation An evaluation of the proposed tool was conducted in a study where psychophysiological responses of people to various video clips were investigated. This kind of studies is typical for the field of affective computing. The design of the proposed experiment was similar to the one described in Chapter 4with several exceptions. As we explained in the Introduction section of this chapter, the most important difference of the new study was overcoming the limitation related to the number of film clips per each category of the archetypal experience. The main objective of this experiment was to explore the relationship between archetypal experiences elicited with video clips and corresponding activations in physiological signals of the subjects. Moreover, we wanted to obtain a computational model describing this relationship. Instead of using separate instruments for recording of physiological signals, extraction of features, reduction of dimensionality, and classification we attempted to perform all these steps of physiological signals processing with ArcheSense. The findings obtained from the new study are reported in Chapter 6. They include both the results that are relevant to the main research questions of this thesis and the outcomes of evaluating the performance of ArcheSense. 6 IMPROVEMENTS IN RECOGNITION OF THE ARCHETYPAL EXPERIENCE FROM PHYSIOLOGICAL SIGNALS 6.1 introduction Having completed the development of a tool for evaluation of human experience (see Chapter 5), we proceeded with preparation of a study that was supposed to address the limitations identified during our first attempt to recognize the archetypal experience induced with video clips from physiological signals reported in Chapter 4. The new tool should have enabled us to streamline the collection, processing, and analysis of physiological data. As we explained earlier, the generalization of our previous findings was limited by the fact that each type of the archetypal experience was represented with a single film clip. In order to overcome this limitation, the new experiment should had an improved design that allocated several video clips per each category of the archetypal experience. Taking into account several factors, such as the overall duration of the study and the length of individual stimuli, we decreased the number of archetypes from eight to seven, made the film clips shorter (one minute instead of five minutes) and increased the number of film clips per an archetype from one to three. Additionally, the new study gave us an opportunity for performing the first evaluation of ArcheSense and answering the question whether a portable tool that we developed could successfully replace traditional instruments for signal processing, data mining, and training of prediction models. We formally defined the major goal of this study as to evaluate the feasibility of sensing and distinguishing various archetypal experiences of individuals based on the analysis of physiological signals, such as heart rate and skin conductance. For the achievement of our research goal, an experiment was designed where a range of archetypal experiences and explicit (conscious) emotions were elicited in individuals, and their physiological data was measured in real-time with wearable sensors. The physiological data obtained from the wearable sensors was processed by ArcheSense. We also asked the subjects to provide conscious self-reports about their feelings. The archetypal experiences and the explicit emotions were induced by means of film clips. This chapter is (partly) based on: Ivonin, L., Chang, H.-M., Díaz, M., Català, A., Chen, W., Rauterberg, M.: Beyond Cognition and Affect: Sensing the Unconscious. Behaviour & Information Technology. In press (2014). 95 96 improvements in recognition of the archetypal experience After the experimental session, a set of relevant features was extracted from the physiological signals and statistical analysis of both physiological and self-reports’ data was conducted. Then, several standard data mining techniques were applied to the features in order to obtain prediction models that allow for a meaningful classification of subjects’ psychological states in accordance with each of the archetypes and the explicit emotions. Finally, we analyzed the obtained findings, proposed an approach to improve classification accuracy, and performed its initial verification. 6.2 materials and methods 6.2.1Stimuli As our goal in this study was to explore the relationship between archetypal experiences of people and activations of the ANS, it was necessary to develop a method for elicitation of these pre-existent forms of apperception in our subjects. While the state of the art still lacks an established method for the elicitation of archetypal experiences, it seems reasonable to follow the approach taken in our previous study. The essence of this approach is to expose participants of a study to manually selected audiovisual material. It is, therefore, a passive method of elicitation. Unlike the approaches involving confederate interaction procedures, this method may not provide psychological responses of high intensity, but it ensures high degree of standardization (Rottenberg et al.,2007). We again chose film clips as a method of eliciting archetypal experiences and explicit emotions in our participants because we did not experience any problems with them in the previous study. Seven common archetypes were included in our study. They were selected based on their appearance in the work of Jung (1981) and others (Walle,1986;Campbell,2008;Faber and Mayer,2009;Munteanu et al.,2010). These archetypes were: anima, hero initiation, hero departure, hero rebirth, hero return, mentor, and shadow. Four out of the seven archetypes chosen were closely related. They represented important stages of the hero’s journey, which was described by Campbell (2008). Campbell identified a prototypical journey that a hero undertakes in a general narrative and divided it into several stages. The archetype of mentor is found in the research of Campbell as well and signifies a character that supports the hero in acquiring knowledge and power. The archetype of anima represents the female aspect of the male psyche, and the archetype of shadow constitutes qualities of the personality that the conscious ego tends to reject. Next, we needed film clips embodying these archetypes that would be demonstrated to the participants of the experiment. It was decided to have three clips taken from different sources to present each of the 6.2 materials and methods 97 archetypes because, in this case, we ensured that computational intelligence algorithms would perform recognition of the archetypes and not the film clips. Similar to the studies that employed films (Gross and Levenson,1995) and our own previous study, we obtained the clips by editing fragments of full-length commercial movies. Film clip Movie Start End Anima (1)American Beauty (Mendes,1999)0:16:15 0:17:17 Anima (2) Malena (Tornatore,2000)0:19:18 0:20:20 Anima (3)Perfume: The Story of a Murderer (Tykwer,2006) 0:18:03 0:18:18 0:21:20 0:22:15 Hero Departure (1) V for Vendetta (McTeigue,2005)0:41:55 0:43:03 Hero Departure (2)Braveheart (Gibson,1995)0:10:10 0:10:46 0:14:13 0:14:43 Hero Departure (3) The Lord of the Rings: The Fellowship of the Ring (Jackson,2001) 2:21:12 2:21:47 2:22:37 2:23:06 2:23:10 2:23:16 Hero Initiation (1) V for Vendetta (McTeigue,2005)1:23:29 1:24:34 Hero Initiation (2)Braveheart (Gibson,1995)2:07:39 2:08:37 2:08:47 2:08:58 Hero Initiation (3) The Matrix (Wachowski and Wachowski,1999)2:02:25 2:03:25 Hero Rebirth (1) V for Vendetta (McTeigue,2005)1:24:59 1:26:00 Hero Rebirth (2)Braveheart (Gibson,1995)2:15:39 2:16:15 2:17:35 2:18:01 Hero Rebirth (3) The Matrix (Wachowski and Wachowski,1999)2:04:35 2:05:45 Table 9: Sources of the film clips for the archetypal experience (part 1). The selection of the fragments was guided by our experience gained from the collaboration with the ARAS that took place during the study reported in Chapter 4.ARAS is an organization that, since the early 1930s, has been collecting and annotating mythological, ritualistic, and symbolic images from all over the world (Gronning et al.,2007). 98 improvements in recognition of the archetypal experience Film clip Movie Start End Hero Return (1)V for Vendetta (McTeigue,2005) 2:02:40 2:03:04 2:03:22 2:04:06 Hero Return (2) Braveheart (Gibson,1995) 2:48:56 2:49:08 2:49:11 2:49:53 2:49:54 2:50:09 Hero Return (3) The Matrix Revolutions (Wachowski and Wachowski,2003) 1:53:40 1:53:47 1:54:02 1:54:05 1:54:33 1:54:50 1:55:24 1:55:39 1:56:02 1:56:29 Mentor (1) The Lord of the Rings: The Fellowship of the Ring (Jackson,2001) 2:03:05 2:04:10 Mentor (2)The King’s Speech (Hooper,2010) 1:42:13 1:42:44 1:42:58 1:43:18 1:45:33 1:45:52 Mentor (3)The Lion King (Allers and Minkoff,1994) 0:24:38 0:25:05 0:25:29 0:26:06 Shadow (1) The Lord of the Rings: The Two Towers (Jackson, 2002) 1:35:19 1:36:20 Shadow (2)Fight Club (Fincher, 1999)1:48:24 1:49:32 Shadow (3)The Dark Knight (Nolan, 2008)1:24:22 1:25:30 Table 10: Sources of the film clips for the archetypal experience (part 2). It did not seem appropriate to select stimuli for induction of the archetypal experiences based on their visual or audio properties. Instead, the film clips were edited based on their symbolic qualities. Symbols play an important role in connecting the internal psychological phenomena and the external physical world (Varela et al.,1992). The symbolic approach is also well-aligned with the work of Jung who identified similar symbolic representations of archetypes across cultures and epochs of human history. The outcome of the selection can be found in Table 9and Table 10. In these tables, we provided information that is necessary to obtain exactly the same film clips as 6.2 materials and methods 99 were used in our study. Unfortunately, we cannot share the film clips themselves because they were extracted from the commercial movies protected by copyright. Besides archetypal experiences we wanted our subjects to feel explicit emotions as well. There were three main reasons for this. First, we needed a possibility to benchmark our findings against the state of the art in affective computing. Second, it was interesting to compare the results of recognition for archetypal experiences and explicit emotions. Third, it was necessary to analyze the differences in self-reports of the participants in order to confirm they were not consciously aware of the archetypes. Similarly to archetypal experiences, explicit emotions were elicited with film clips. As it was demonstrated in Chapter 2, emotions or feelings are commonly represented in affective computing with the dimensional model. This model projects emotions in the affective space with two or three dimensions. In the case of two dimensions, an emotional state in the affective space is characterized by values of arousal and valence (Russell,1980). The dimension of arousal ranges from calm to aroused states, while the dimension of valence ranges from negative to positive states. Similarly to the previous study, we did not want to choose too many explicit emotions, but at the same time, the number of emotions should be sufficient to uniformly cover the two-dimensional affective space. Therefore, five emotional states were chosen. Four of them were located in each of quadrants of the affective space, and the fifth was situated close to the origin. In accordance with their location, we tagged these emotional states as active-pleasant, active-unpleasant, passive-pleasant, passiveunpleasant, and neutral. The film clips for elicitation of these explicit emotional states were identified based on the literature in elicitation of affect and our experience with the previous experiment. The work of Gross and Levenson (1995) and Soleymani et al. (2011) provides guidance with regard to application of video in emotion research and even proposes sets of film clips that can be readily used as emotional stimuli. Unfortunately, we could not easily reuse film clips from the previous study because the length of the videos had to be considerably adjusted. In the same manner as for the archetypes, three film clips were selected for every explicit emotion. Therefore, in total, we had 15 affective film clips that are listed in Table 11. Again, the clips could not be shared due to the copyright restrictions, but the data in Table 11 should be sufficient to create videos identical to the ones used in our experiment. 6.2.2Experimental Design Our experiment was conducted at the usability laboratory of the Polytechnic University of Catalonia. The laboratory was divided into two rooms. The inner room was set up for presentation of video clips on 100 improvements in recognition of the archetypal experience Film clip Movie Start End AP (1)Mr. Bean (Atkinson and Curtis, 1990)0:06:10 0:07:13 AP (2) Funny cats (YouTube,2008)0:00:00 0:01:01 AP (3)Funny clip with mice and a dog (MrBallonRond,2012) 0:04:11 0:04:44 0:10:08 0:10:36 AU (1) Hannibal (Scott,2001)1:44:50 1:45:50 AU (2)American History X (Kaye, 1998)1:52:07 1:53:10 AU (3)The Silence of the Lambs (Demme,1991)1:39:38 1:40:40 NE (1)Coral Sea Dreaming: Awaken (Hannan,2010)0:08:01 0:09:01 NE (2)Coral Sea Dreaming: Awaken (Hannan,2010)0:04:31 0:05:31 NE (3)Coral Sea Dreaming: Awaken (Hannan,2010)0:38:48 0:39:48 PP (1)The Lion King (Allers and Minkoff,1994)0:47:51 0:48:52 PP (2)Mr. Bean’s Holiday (Bendelack, 2007)1:17:19 1:18:19 PP (3) Love Actually (Curtis,2003)0:10:17 0:11:21 PU (1)The Thin Red Line (Malick, 1998)1:07:08 1:08:09 PU (2) Forrest Gump (Zemeckis,1994)2:05:55 2:07:04 PU (3) Up (Docter,2009)0:10:22 0:11:26 Table 11: Sources of the film clips for the explicit emotions. AP: Activepleasant; AU: Active-unpleasant; NE: Neutral; PP: Passive-pleasant; PU: Passive-unpleasant. a large screen and accommodated subjects during the study. It had a comfortable couch located at the distance of three meters in front of the screen and was equipped with surveillance cameras for observation of the participants. The video clips were projected on the screen with a beamer. Sound was delivered via wireless headphones. The wall between the inner and the outer room had windows made of tinted glass, and therefore, the subjects located in the inner room could not see researchers who administered a session of the experiment. The outer room had a computer that was used for managing playback of 6.3 results 107 trees were used in combination with Adaptive Boosting (Freund and Schapire,1997) in order to achieve higher accuracy. It was important to guarantee that the classification algorithms are not trained and tested on the same dataset because we wanted to obtain subject-independent results. Therefore, a leave-one-out cross-validation technique was employed for assessments of the classification performance. Prior to performing classification, it was beneficial to reduce the dimensionality of the dataset with physiological data. Generally, reduction of the dimensionality is a recommended step in data mining procedures. There are various techniques for the reduction of the features space including PCA and LDA. These two approaches are particularly common for the reduction of dimensionality. For this study, we tried both of these approaches. As of this writing, ArcheSense only supported PCA, and therefore, it was the only method that could be applied automatically. Additionally, we manually employed LDA method because PCA does not capitalize on between-class information, while LDA uses both withinand between-class information for better performance (Martinez and Kak,2001). Two aspects of LDA should be mentioned here. First, strictly speaking, LDA is not a feature selection but a feature extraction method that obtains the new attributes by a linear combination of the original dimensions. The reduction of dimensionality is achieved by keeping the components with highest variance. Second, LDA can be used for both the identification of important features and classification (Novak et al.,2012). While the aforementioned methodology was utilized for betweensubject classification, later we also had to perform within-subject classification, and due to the small number of data samples, it required a special approach for the reduction of dimensionality. We borrowed this approach from the domain of image recognition where the highdimensional datasets with small sample size are common. In these circumstances, the traditional LDA algorithm cannot be used because its within-class scatter matrix is always singular (Yang and Yang,2003). A popular technique to address this difficulty is called PCA plus LDA (Belhumeur et al.,1997;Yu and Yang,2001). In this approach, PCA is applied to reduce the dimensionality before using LDA.PCA plus LDA approach was verified both by experience and theoretically (Yang and Yang,2003). In PCA reduction, we kept the minimum number of variables that were required to explain at least 90 percent of the variance in a dataset. 6.3 results Upon completion of the study, ArcheSense enabled us to perform quick exploratory data analysis. The tool averaged and plotted physiological data of the subjects on several line charts that are well-suited for the display of a sequence of variables in time. We could switch 108 improvements in recognition of the archetypal experience the display between the charts corresponding to different types of the archetypal experience, emotional states, and physiological variables. Data visualization is important because it allows researchers to quickly examine large amounts of data and efficiently expose trends and issues. The exploratory analysis of the HR signal revealed that presentation of each video clip generally led to a decelerating response in HR. This pattern of response had place even with neutral stimuli and is illustrated on Figure 21. The deceleration of heart rate due to diversion of attention to an external task, such as perception of an audiovisual stimulus, is a known effect that is explained by Laceys’ theory of intake and rejection (Lacey and Lacey,1970). Based on our previous study and other literature in the related fields (Winton et al.,1984;Palomba et al.,1997) we anticipated this effect and made adjustments in the data analysis procedure in order to account for the decelerating response present across all categories of the stimuli. This adjustment enabled us to highlight the differences in responses to various stimuli and improve the classification accuracy. Figure 21: The pattern of the subjects’ heart rate signals corresponding to the presentation of neutral film clips. The mean values and 95% confidence intervals of the HR are represented with the bold lines and the vertical bars. Following the exploratory analysis, we exported the obtained dataset from ArcheSense. Three goals were pursued by this export. First, several statistical procedures could be performed on the dataset using third-party software packages for statistical analysis. ArcheSense was developed with the idea to simplify the collection of physiological data and training of machine learning algorithms but it is reasonable to assume that some researchers would also like to perform statistical 6.3 results 109 analysis of the obtained data. Second, it was necessary to compare the performance of ArcheSense in training of prediction models with other data mining tools. Third, as of this writing, ArcheSense only supported a limited number of data mining techniques and we wanted to analyze the obtained dataset with more advanced methods. 6.3.1Statistical Analysis Next, several statistical tests were conducted. We started with analysis of the self-report evaluations provided by the subjects after watching the film clips. MANOVA for repeated measurements was performed for the SAM ratings of valence, arousal and dominance. It demonstrated a significant main effect of the archetypes presented in the film clips on the SAM ratings [F(18,351.210) = 10.060, p < 0.001 (Wilks’ lambda)]. Similarly, the explicit emotions exhibited in the film clips had a significant main effect on the SAM ratings provided by the participants [F(12,227.826) = 25.301, p < 0.001 (Wilks’ lambda)]. Estimated marginal means of the SAM ratings can be found in Table 12. Then, we added gender of the participants as a between-subject factor to the MANOVA tests in order to see if women and men rated their psychological experiences in a different manner. The results of the tests indicated that the interaction effect between the subjects’ gender and the archetypes was not significant [F(18,334.240) = 1.166, p = 0.288 (Wilks’ lambda)]. Neither was significant the interaction effect between the gender of the participants and the explicit emotions [F(12,217.243) = 1.476,p=0.135 (Wilks’ lambda)]. When the statistical analysis of the SAM ratings was complete, we looked into the physiological data of the subjects. Multivariate analysis of variance conducted for the features extracted from the physiological signals indicated there is a significant main effect of the archetypes pictured in the film clips on physiological responses of the subjects [F(216,583.757) = 1.396,p=0.001 (Wilks’ lambda)]. Another MANOVA test was performed in order to see the relationship between the physiological data and the explicit emotions presented in the film clips. The outcome of this test was significant as well [F(144,213.766) = 1.985, p <0.001 (Wilks’ lambda)]. Next, we examined if there was a connection between gender of the participants and their physiological responses to the film clips. The gender was added into the analysis as a between-subject variable. The results of the MANOVA tests demonstrated there were no significant interaction effects neither between the archetypes and the gender [F(216, 548.182) = 1.034,p=0.379 (Wilks’ lambda)] nor between the explicit emotions and the gender [F(144,197.835) = 0.872, p = 0.808 (Wilks’ lambda)]. Our statistical analysis uncovered several interesting findings. There were significant relationships between the archetypes and the SAM rat- 110 improvements in recognition of the archetypal experience Class Valence Arousal Dominance Mean SE Mean SE Mean SE Anima 5.879 0.298 4.515 0.418 5.803 0.360 Hero Departure 4.015 0.233 4.485 0.372 4.515 0.400 Hero Initiation 3.864 0.271 4.909 0.392 4.439 0.353 Hero Rebirth 5.924 0.282 4.455 0.361 6.197 0.299 Hero Return 6.318 0.298 4.818 0.381 6.742 0.330 Mentor 6.273 0.198 3.455 0.279 6.348 0.300 Shadow 4.591 0.229 4.576 0.426 5.212 0.330 Active-pleasant 7.232 0.360 3.522 0.333 6.768 0.282 Active-unpleasant 2.623 0.237 5.986 0.386 3.522 0.371 Neutral 7.406 0.293 1.580 0.130 6.812 0.366 Passive-pleasant 8.043 0.176 3.087 0.343 7.391 0.246 Passive-unpleasant 3.478 0.293 4.014 0.313 4.130 0.389 Table 12: Mean values and standard errors (SE) of the SAM ratings. The ratings were provided by the participants after viewing the film clips. The left column indicates which archetype or explicit emotion was presented in the film clips. ings, between the explicit emotions and the SAM ratings, between the archetypes and the physiological responses, and between the explicit emotions and the physiological responses. In order to further explore these findings, we needed to build and evaluate predictive models that would quantify these relationships. The evaluation was performed through comparison of classification accuracies achieved by the predictive models obtained with five different methods (kNN,SVM, naïve Bayes, LDA and AdaBoost with decision trees). 6.3.2Classification We started with models for prediction of the archetypal experiences based on the SAM ratings. Due to the fact that in our study there were video clips for elicitation of seven different archetypal experiences, the classification task was considerably difficult. Moreover, four out of the seven archetypes were related to a hero. This circumstance added even more confusion into the subjects’ self-reports. For this reason, we divided the set of films picturing the archetypes into four subsets. Every subset included the archetypes of anima, mentor, shadow, and one of the hero archetypes. The best classification accuracy (28%) for the complete set of archetypes was achieved with the kNN classifier. Similarly, the kNN method demonstrated the most accurate result (42%) for 6.3 results 111 the subset that included the archetype of hero departure. For the subset with the archetype of hero initiation, the precision of classification was between 40.9percent (with SVM classifier) and 43.1percent (with AdaBoost classifier). It was also the most accurately predicted subset out of the four. The subset with the archetype of hero rebirth featured the lowest classification rate (38.4%) among all of the subsets. Finally, for the subset that included the archetype of hero return, the classification methods enabled us to achieve the accuracy of 40.6percent. A more detailed overview of the classification results can be found in Table 13. Categories of the film clips NkNN SVM Naïve Bayes LDA AdaBoost AN, HD, HI, HRB, HRT, ME, SH 728.024.224.624.7 25.5 AN, HD, ME, SH 442.040.238.040.9 37 AN, HI, ME, SH 4 42.4 40.942.042.043.1 AN, HRB, ME, SH 438.437.736.238.0 34.4 AN, HRT, ME, SH 440.639.939.539.5 39.9 AP, AU, NE, PP, PU 550.449.049.048.4 47.3 AU, NE, PP, PU 4 63.4 63.863.064.963.0 Table 13: Classification results obtained for recognition of the archetypes and the explicit emotions from the self-reports. The first column reports the categories of the film clips that were included into the classification. The number of the categories is specified in the second column. Other columns state classification accuracy (in percent) achieved with various classification methods. AN: Anima; HD: Hero Departure; HI: Hero Initiation; HRB: Hero Rebirth; HRT: Hero Return; ME: Mentor; SH: Shadow; AP: Active-pleasant; AU: Active-unpleasant; NE: Neutral; PP: Passive-pleasant; PU: Passive-unpleasant. Our next step was to see how accurately the explicit emotions presented in the film clips could be differentiated based on the SAM ratings given by the subjects. For this purpose, we performed classification with the same classification algorithms as were used for the film clips with the archetypes. The analysis was conducted with two datasets: the complete dataset that included the self-reported data for all the film clips featuring the explicit emotions and the dataset that was obtained from the complete dataset by removing the data related to the active-pleasant emotional state. The motivation for introduction of the second dataset was justified by the fact that one of the film clips for active-pleasant emotion turned out to be controversial. Our observations of the participants’ facial expressions during the study indicated 112 improvements in recognition of the archetypal experience that some of them found this clip to be unpleasant or confusing, while other subjects perceived it as funny. Therefore, we expected considerable variations in the subjects’ self-reports. Moreover, classification results of the second dataset for the explicit emotions and any of the four reduced datasets for the archetypes could be easily compared because they had the same number of classes. The best classification accuracy (50.4%) for the complete dataset of the explicit emotions was achieved with the kNN method. According to our expectations, the classification of the subset that did not include the data corresponding to the active-pleasant emotion was noticeably more precise (64.9%). Table 13 provides more details on the classification results for the explicit emotions. Having conducted the analysis of the SAM rating, our next goal was to evaluate the feasibility of recognizing the archetypes and the explicit emotions from the physiological data of the participants. This evaluation was performed in three steps. On the first step, recognition of the archetypes and the explicit emotions was carried out using the feature variables extracted from cardiovascular responses of the subjects. For the second step, only the features of the skin conductance signal were used. Finally, on the third step, we utilized the complete set of the features extracted from the ECG and skin conductance signals. The breakdown of the analysis process into three steps enabled us to discover and compare the importance of different physiological signals with respect to the classification. Similarly to the analysis of the SAM ratings, we used five classification methods for every step. The cardiovascular data of the subjects enabled us to classify the film clips corresponding to the seven archetypes with the accuracy of up to 29.3percent. This recognition rate was achieved with the SVM algorithm. The recognition performance reached 44.6percent (LDA) if the number of archetypes in the classification was decreased by isolating four smaller subsets following the approach taken during the analysis of the self-reports. The accuracy of classification for each of the subsets was slightly above 40 percent, ranging from 41.1percent (for the subset with the archetype of hero initiation) to 44.6percent (for the subset with the archetype of hero rebirth). For three out of the four subsets, the best results were achieved with the LDA algorithm. The classification of the explicit emotions based on the cardiovascular data was possible with the accuracy of up to 35.9percent (LDA) in case of five classes of the emotions and up to 43.1percent (Naïve Bayes) if the data corresponding to the active-pleasant emotional state was excluded. A detailed overview of the recognition results for the ECG data is presented in Table 14. Next, we performed analysis based on the skin conductance data of the participants. The analysis followed the same procedure as in the case of the cardiovascular data. The prediction model trained based on the responses in skin conductivity of the subjects to presentation of 6.3 results 113 Categories of the film clips NkNN SVM Naïve Bayes LDA AdaBoost AN, HD, HI, HRB, HRT, ME, SH 7 25.529.327.229.0 18.5 AN, HD, ME, SH 4 38.4 41.342.842.4 31.9 AN, HI, ME, SH 4 39.3 40.037.841.135.6 AN, HRB, ME, SH 4 38.4 40.240.644.642.4 AN, HRT, ME, SH 4 42.8 43.840.644.235.5 AP, AU, NE, PP, PU 5 30.7 35.132.835.926.4 AU, NE, PP, PU 4 38.4 39.943.141.7 34.4 Table 14: Classification results obtained for recognition of the archetypes and the explicit emotions from the cardiovascular responses of the participants. The first column reports the categories of the film clips that were included into the classification. The number of the categories is specified in the second column. Other columns state classification accuracy (in percent) achieved with various classification methods. AN: Anima; HD: Hero Departure; HI: Hero Initiation; HRB: Hero Rebirth; HRT: Hero Return; ME: Mentor; SH: Shadow; AP: Activepleasant; AU: Active-unpleasant; NE: Neutral; PP: Passive-pleasant; PU: Passive-unpleasant. the archetypal film clips enabled us to classify seven archetypes with the accuracy of 28 percent (LDA). Then, the original dataset was split into four subsets in such a way that every subset included only one of the hero archetypes. The classification performance varied from 39.5 percent to 45.5percent among the subsets. Similarly to the ECG signal, the most accurate results were obtained with the LDA method. The explicit emotions were predicted with the precision of 40.6percent (LDA) in case of five classes and 46.0percent (SVM) in case of four classes. Additional information about the classification is provided in Table 15. Finally, we integrated the features extracted from the ECG and the skin conductance signals into a unified dataset and built several prediction models in order to evaluate the feasibility of recognizing the archetypes and the explicit emotions from the physiological data. In the case of classifying seven archetypes, the accuracy was in the range between 28.4percent (AdaBoost) and 36.7percent (LDA). When the data was rearranged into several subsets, in such a manner that each of them corresponded to only four archetypes, the classification performance achieved 57.1percent (LDA). This result was accomplished on the subset with the archetype of hero initiation. The recognition of the 114 improvements in recognition of the archetypal experience Categories of the film clips NkNN SVM Naïve Bayes LDA AdaBoost AN, HD, HI, HRB, HRT, ME, SH 7 21.0 23.023.728.018.0 AN, HD, ME, SH 4 36.6 39.935.540.937.0 AN, HI, ME, SH 4 37.5 44 44.745.522.5 AN, HRB, ME, SH 4 29.739.537.339.522.8 AN, HRT, ME, SH 4 37.0 39.135.939.527.2 AP, AU, NE, PP, PU 5 38.240.640.040.630.4 AU, NE, PP, PU 4 40.946.041.744.6 24.3 Table 15: Classification results obtained for recognition of the archetypes and the explicit emotions from the skin conductance of the participants. The first column reports the categories of the film clips that were included into the classification. The number of the categories is specified in the second column. Other columns state classification accuracy (in percent) achieved with various classification methods. AN: Anima; HD: Hero Departure; HI: Hero Initiation; HRB: Hero Rebirth; HRT: Hero Return; ME: Mentor; SH: Shadow; AP: Activepleasant; AU: Active-unpleasant; NE: Neutral; PP: Passive-pleasant; PU: Passive-unpleasant. explicit emotions demonstrated similar outcomes. With four classes of the emotions, 57.2percent of the cases were accurately classified using the LDA method. When five explicit emotions were included into the analysis, the recognition rate decreased to 50.7percent (LDA). In Table 16, we provided further details about the analysis of the complete dataset of the physiological signals. As we completed the analysis of the participants’ self-reports and their physiological responses to the film clips it was necessary to put the results next to each other to facilitate a comparison and further discussion. This was done in Table 17, which reports the best classification accuracies achieved on the datasets that were built based on the selfreports and the physiological data. Additionally, this table illustrates the relationship between the recognition performance and the number of the film clips’ categories in the datasets. Although up to this point of the analysis we focused on conducting between-subject classification, there is an opinion that due to physiological differences between individuals the algorithms for recognition of affective states work better if they were personalized. For instance, judging from the review provided in (Novak et al.,2012), the studies that performed within-subjects recognition of psychological states 6.3 results 115 Categories of the film clips NkNN SVM Naïve Bayes LDA AdaBoost AN, HD, HI, HRB, HRT, ME, SH 7 33.4 34.633.436.728.4 AN, HD, ME, SH 4 52.9 50.753.351.4 50.7 AN, HI, ME, SH 4 54.1 56.055.657.145.8 AN, HRB, ME, SH 4 49.2 51.450.052.938.0 AN, HRT, ME, SH 456.152.253.652.9 49.3 AP, AU, NE, PP, PU 5 47.5 49.050.150.744.1 AU, NE, PP, PU 4 54.7 55.157.256.2 41.6 Table 16: Classification results obtained for recognition of the archetypes and the explicit emotions from the complete dataset of the physiological signals. The first column reports the categories of the film clips that were included into the classification. The number of the categories is specified in the second column. Other columns state classification accuracy (in percent) achieved with various classification methods. AN: Anima; HD: Hero Departure; HI: Hero Initiation; HRB: Hero Rebirth; HRT: Hero Return; ME: Mentor; SH: Shadow; AP: Activepleasant; AU: Active-unpleasant; NE: Neutral; PP: Passive-pleasant; PU: Passive-unpleasant. achieved better results than the experiments where between-subjects approach were utilized. Although the main focus of our study was on investigating the feasibility of developing algorithms for betweensubjects recognition of the archetypal experiences, we could also analyze the data from the participants individually. After we split the original dataset into 23 subsets in such a way that every dataset contained physiological data for one individual, two types of analysis were conducted. First, we trained prediction models using one of the classification algorithms introduced above to recognize the archetypal experiences of the subjects. Before the training took place, we had to considerably reduce the number of features in order to avoid the unbalanced classification problem. The reduction was performed using the technique described in the Methods section. It should be noted that, although the actions for prevention of the unbalanced design were implemented, our results for the within-subject classification are still likely to be overoptimistic and should be interpreted as preliminary. A dedicated study with an emphasis on the within-subject design is required for more reliable evaluations. Having completed the classification for each of the participants, we calculated separately the means and standard er- 116 improvements in recognition of the archetypal experience Categories of the film clips N Selfreports Physiological data AN, HD, HI, HRB, HRT, ME, SH 728.0 36.7 AN, HD, ME, SH 442.0 53.3 AN, HI, ME, SH 443.1 57.1 AN, HRB, ME, SH 438.4 52.9 AN, HRT, ME, SH 440.6 56.1 AP, AU, NE, PP, PU 550.4 50.7 AU, NE, PP, PU 464.9 57.2 Table 17: Comparison of the classification accuracy achieved using the selfreport questionnaires and the physiological data. AN: Anima; HD: Hero Departure; HI: Hero Initiation; HRB: Hero Rebirth; HRT: Hero Return; ME: Mentor; SH: Shadow; AP: Active-pleasant; AU: Active-unpleasant; NE: Neutral; PP: Passive-pleasant; PU: Passiveunpleasant. rors of the recognition accuracies across the whole population and for two gender groups. The best classification rate (70.3percent) between seven archetypes was achieved with the SVM technique for the entire population of the subjects. If the participants were divided into gender groups, the recognition accuracy for the men was 72.8percent (SVM), while for the women only 67.0percent (SVM). Second, the steps taken for the classification of the archetypal experiences were repeated in order to obtain prediction models for the explicit emotions. In this analysis, the LDA and the SVM techniques demonstrated almost identical performance. The five explicit emotions were classified with an average accuracy of 86.6percent (LDA) on the dataset that consisted of the physiological data from the participants of both genders. Similarly to the recognition of the archetypes, we could more reliably predict the explicit emotions for men (88.2%) rather than for women (84.7%). A detailed overview of the classification results is presented in Table 18. 6.3.3Performance of ArcheSense In order to evaluate the performance of ArcheSense, we performed training of a prediction model using ArcheSense and a desktop version of WEKA software suite. It was expected that if the same dimensionality reduction technique and the classification algorithm were used in WEKA and ArcheSense, the accuracy of obtained models would be equal. This observation would enable us to validate the performance of ArcheSense in the area of data mining. As at the time of this ex- 6.4 discussion 123 usefulness and helped us to save considerable amount of time on the first step of data collection. Second, ArcheSense running on a tablet computer gave us the flexibility that is not available in conventional approaches to collection of physiological data. Even though the reported study was conducted in the laboratory settings, the portability of ArcheSense enabled researchers to freely move around the subjects and between two rooms holding the tablet with a real-time stream of physiological data visualized by this tool. It was particularly helpful in situations when the quality of physiological recordings was bad due to the poor electrical contact between skin of a subject and an electrode. The immediate visual feedback provided by ArcheSense enables us to quickly identify such problems and efficiently respond to them. Without doubt, the flexibility of ArcheSense as a data collection instrument is likely to be even more significant advantage in realistic and mobile scenarios. Besides providing assistance in the data collection, ArcheSense also simplified and automated the extraction of featured from the physiological recordings. In our experiment, two physiological signals were captured: ECG and skin conductivity. Several features for the future classification had to be extracted from the raw data corresponding to both of these signals. ECG data needed to be processed in order to locate heart beats and obtain beat-to-beat intervals because they would serve as an input for calculation of heart rate and HRV features. Similarly, the raw skin conductivity data had to be analyzed in order to obtain such features as skin conductance level and skin conductance response. Traditionally, researchers have to either adjust third-party signal processing tools or develop their own software routines for extraction of physiological features. None of these options seems convenient or leads to the efficient utilization of researchers’ time. For this reason, the capability of ArcheSense to automatically extract features like heart rate from the physiological signals was particularly helpful for us. According to the settings related to the recognition window, ArcheSense divided physiological data corresponding to each of the film clips into 12 non-overlapping segments. A segment, therefore, lasted for five seconds, and the temporal data was averaged over its duration. This procedure was performed for HR,SCL, and SCR signals. For the SCR, the tool additionally calculated absolute values of the signal (Figner and Murphy,2011). Then, ArcheSense performed fusion of physiological data coming from different signals through concatenation. As an outcome of the transformation we had an integrated dataset consisting of 36 features that could be used for statistical analysis and classification. As it was explained in the Methods section, we further extended this dataset by including features that at the time of the experiment could not be obtained with ArcheSense. Prior to performing training of a prediction model, we used ArcheSense to reduce the dimensionality of the obtained dataset with PCA algorithm. 124 improvements in recognition of the archetypal experience Overall, based on our observations during the experiment, it seems that ArcheSense is a viable tool for observation of human experience through the physiological measurements. This instrument can potentially replace separate instruments for recording of physiological signals, extraction of features, reduction of dimensionality, and classification. Moreover, the portability aspect of this tool enables researchers to investigate emotional experiences of people in ecologically valid, realistic environments. ArcheSense significantly simplified and streamlined our study for recognition of emotional responses to the film clips. Nevertheless, we identified two important limitations of the current version of ArcheSense. The first limitation is related to the small number of available data mining methods. The second limitation is associated with the narrow capabilities for calculation of various physiological features. The origin of both limitations is directly linked to the fact that, as of writing, ArcheSense was at the early stage of development, and therefore, implemented only a bare minimum of features in order to be functional. Despite of the limited functionality, the implementation of ArcheSense was sufficient to validate the proposed framework for an instrument that can facilitate evaluation of affective experiences based on physiological data. As our results indicate that ArcheSense has a potential to become a fully featured all-in-one instrument for evaluation of affective experience, it is necessary to outline several future directions. First of all, we plan to address the limitations that became apparent in the aforementioned study. Next, it is necessary to assess performance of the tool in the evaluation mode. A separate study has to be conducted for this purpose. Another study could be performed in order to assess the usability aspects of the developed tool. Finally, ArcheSense should be complemented with additional capabilities, such as convenient sharing of collected physiological data and prediction models. The sharing could take place at a portal connected to the Internet. Researchers would be able to collaborate with each other by sharing datasets in a uniform representation. 6.4.4Limitations The present study has some limitations. One limitation is the relatively small number of participants. If we talk about the within-subject classification then it is necessary to mention that more data samples per subject would be beneficial. Another limitation is that the participants did not move much during the presentation of the stimuli and, thus, movement artifacts in the physiological measurements were minimal. In many practical scenarios it is reasonable to expect movements of users. For this reason, additional filters for the elimination of the noise generated by movements have to be introduced. Finally, additional 6.5 conclusion 125 studies are still necessary for the the final confirmation of the generalizability of our findings. 6.5 conclusion Besides explicit emotional feelings that are easily available for conscious recollection, people have unconscious experiences that nevertheless drive their decisions, motivations, and behaviors. Unlike the explicit emotions, the unconscious or implicit experiences have received little attention in the HCI field. In this study, we investigated whether the archetypal experiences of users, which in part constitute the unconscious, produce distinct patterns of physiological responses and estimated the feasibility of the automated recognition of such experiences with wearable sensors for measurement of cardiovascular and electrodermal activities. Seven archetypes and five explicit emotions were included in the study and presented to subjects by means of the film clips. Following the presentation of every video the participants were asked to provide a conscious report about their feelings. The statistical analysis demonstrated that the film clips with both the archetypes and the explicit emotions led to significantly different psychophysiological responses. Then, the data mining techniques were applied to the subjects’ self-reports and their physiological data in order to construct several prediction models. In case of the archetypal film clips, the models trained on the physiological data demonstrated better performance (57.1%) than the models built based on the self-reports (43.1%). We encountered an opposite finding during the evaluation of the models for the explicit emotions. The models that were built based on the SAM reports featured higher classification accuracy (64.9%) than the models trained on the physiological data (57.2%). Thus, it seems that by using the physiological signals the archetypes could be distinguished as accurately as the explicit emotions. Moreover, our research findings suggest the subjects had more conscious awareness about the explicit emotions rather than the archetypal experiences. Although the classification performance for the archetypes was considerably higher than the chance level, it may not be robust enough for practical applications. Therefore, we carried out a preliminary evaluation to see whether a switch from the between-subject to within-subject modeling could benefit the recognition accuracy. Our analysis was performed on small data samples and may be overoptimistic, but it indicated that the classification rate for seven classes of the archetypes could be improved up to 70.3percent by using within-subject models. Overall, our findings suggest the archetypes could be identified in human experience through physiological measurements even though they may not always be consciously recognized by the individuals. Additional goal of this study was to evaluate the tool that could facilitate evaluation of affective experiences through the measurement 126 improvements in recognition of the archetypal experience of physiological activations. Although the area of affective computing and other fields requiring the means for objective evaluation of human experience are growing, researchers still miss an instrument that could automate and streamline psychophysiological measurements. Therefore, in Chapter 5, we introduced a framework and provided an initial implementation of such an instrument. Our implementation is called ArcheSense and is publicly available at Google Play. Based on the initial findings obtained during this study, we could conclude that ArcheSense considerably simplified and automated the technical aspects of conducting a psychophysiological experiment. More specifically, recording of physiological signals, extraction of features, reduction of dimensionality, and classification or estimation were handled by ArcheSense. Finally, we hope to continue development of ArcheSense as a general research tool for measuring affective responses of people through their physiological activations. 7 GENERAL DISCUSSION 7.1 implicit mental experience and archetypes We started this dissertation with considering the problem of capturing and digitizing mental experience of people. As it was explained in Chapter 1, sharing of experience is crucial in many aspects of human life. For instance, constitution of groups requires individuals to share experience in short and long periods of time. Lahlou (2010) argued that sharing experience in the short term strongly constitutes participation. When sharing takes place in the long term, it supports the creation of a common mythical past and belief in a shared destiny of a project. Emotional component of experience plays a particularly important role in the establishment of groups. It is also crucial in the empathic competences of humans that enable them, to certain extent, make sense of the overt behavior of other people using analogies from their own experience (Hatfield et al.,2009). Therefore, transmission of human experience goes beyond the mere exchange of information and involves a social dimension comprising of the multidimensional facets of direct, primary, and bodily experience (Lahlou,2010). A variety of methods for capturing and transferring the aspects of human experience related to the objective knowledge have been developed. On the other hand, measuring the emotional dimension of human experience represents a more challenging task and was first considered in the 1990s when the discipline of affective computing was introduced as “computing that relates to, arises from, and deliberately influences emotion” (Picard,2010). Researchers working in the field of affective computing have invented a number of techniques that make possible capturing and digitization of users’ emotional states. The focus of their research, however, always stayed on the explicit emotional states. That is, emotional states that can be consciously interpreted and reported by people using, for instance, a questionnaire. While this approach seems plausible, based on the research findings from the domain of psychology, it becomes clear that the implicit emotional experience of people also demands attention from the scientific community. In this dissertation, we attempted to evaluate the feasibility of developing methods that enable recognition and digitization of the implicit mental experience. The major difficulty associated with capturing the implicit experience is directly linked to the fact that humans find it very difficult or impossible to describe this kind of mental experience. For this reason, very few knowledge about evaluating or quantifying the implicit 127 128 general discussion (or unconscious) experience exists. One of the frameworks for description of the unconscious experience of people that found a wide application in personality psychology and marketing was developed by Jung (1981). Using the concept of archetypes, he provided a conceptual structure for understanding how the unconscious mental experience is organized. Archetypes are prototypical categories of situations, objects, and people that are common for the most of the cultures and have existed across evolutionary time. When an archetype is experienced, it produces a distinguished array of implicit emotional feelings. Taking into account the prominence of Jung’s framework, we used it in order to more clearly formulate the goals of this research. Utilizing on the notion of archetypes, we could narrow down the general problem of recognition and digitization of the implicit mental experience to a more specific task of capturing the unconscious experience related to various archetypes. Research in psychophysiology and affective computing has demonstrated that psychological states of an individual can be successfully linked with physiological activations in a human body. For instance, an exposure to a stressful situation may produce a change in heart rate. The relationship between psychological states and physiological signals is particularly valuable because it provides a capability for an objective evaluation of the individuals’ mental states. This evaluation bypasses the conscious thinking of the individuals and cannot be influenced by their subjective reports. Thanks to this property of physiological data, it is sometimes referred to as ’honest signals’(Pentland and Pentland,2008). Our main hypothesis in this dissertation was that physiological data may help us to capture not only the conscious emotional experience but also the archetypal experience. Moreover, we wanted to see whether different types of the archetypal experience could be reliably recognized by a computer system in an automated fashion. This research challenge was approached with a series of three empirical studies that enabled us to shed some light on the problem of digitizing the archetypal experience of people. In this chapter, we will discuss our main findings, outline future research directions, review the practical applications, and draw the final conclusion. 7.2 physiological correlates of perceiving affective pictures and sounds Our first study relied on a pool of pictures and sounds selected beforehand for elicitation of the archetypal experience in the participants. This method of elicitation was chosen based on the state of the art in affective computing. As explained in Chapter 3, our review indicated that two databases consisting of pictures and sounds (Lang et al.,2008; Bradley and Lang,1999) are ones of the most common instruments for the elicitation of the emotional experience. Although one may point 7.2 physiological correlates of affective pictures and sounds 129 out several disadvantages of these databases, such as very brief exposure to the stimuli (approximately 6seconds) or the content that is starting to look old-fashioned nowadays, they are still one of the standard approaches in the research on emotion. For this reason, it was logical to use images and sounds for elicitation of the archetypal experience as well. As our first study was relatively simple and included only one type of the archetypal experience, it was relatively easy to find the required set of stimuli. The pictures and sounds were selected based on their symbolic meaning that corresponded to the archetype of the self. Additionally to the stimuli for the archetypal experience, standard stimuli (from IAPS and IADS) for the explicit emotions were included in the study. The main question that we expected to answer after the first study was whether a particular kind of the archetypal experience would result in a homogenous pattern of activations in the ANS of the participants. Depending on the answer to this question it would have been necessary to either continue the inquiry into capturing of the archetypal experience or change the topic of the investigation. Taking into consideration the previous work on correlation of ECG features to emotions (Fairclough and Venables,2006) and the simplicity of the experiment, only this physiological signal was monitored. This circumstance enabled us to speed up the processing of the data and reduce the complexity of the statistical analysis. Besides recording the physiological data of the subjects we also collected their self-reports using the SAM instrument. The analysis of the collected data led to several important observations. First of all, the collected SAM ratings confirmed that the distribution of the stimuli for explicit emotions in the affective space was consistent with the previous research (Lang et al.,2008;Bradley and Lang,1999). The location of the archetypal category of stimuli in the affective space was very close to the neutral category. Therefore, it seems that the participants consciously described their feelings towards the archetypal content as neutral. Next, several statistical tests that were performed for different time intervals of the physiological data indicated a significant main effect of the category of the stimuli on the average heart rate of the subjects. As our dataset included the physiological recordings corresponding to both the archetypal experience and the explicit emotions, the outcomes of the statistical analysis implied that individual categories of the stimuli were characterized by considerably different patterns of the cardiovascular responses. Moreover, results of the classification that treated the physiological data as a predictor of the category of the stimuli indicated that a particular emotional or archetypal state could be predicted with an accuracy above the chance level. There were also two interesting observations that did not have a direct relation to the main research question. The first one is that sounds had a stronger effect on the participants as evidenced by large decelerations of their HR. The second observation was that 130 general discussion the cultural background of the subjects did not significantly influence their responses to the archetypal content. Although the statistical analysis and the classification provided an initial evidence that the experience related to the archetype of the self could lead to a recognizable pattern in a change of the participants’ heart rate, the classification accuracy was considerably lower than other studies in affective computing reported. One of the most likely explanations of the poor classification performance was related to the intentional simplicity of this experiment. The study was designed in a way that enabled us to quickly confirm or discard the main hypothesis and, based on the outcome, either proceed with more sophisticated experiments or adjust the research vector. Since the collected data suggested a positive answer to the principal research question and provided a substantial foundation for the future research, we began planning our next experiment by identifying the weaknesses that needed to be addressed foremost. Overall, it seemed that three limitations had to be overcome in order to achieve more complete and reliable classification results: (1) the number of archetypes in consideration, (2) the assortment of physiological measurements, and (3) the intensity of the stimuli. The number of archetypes in consideration had to be increased because the technique for digitization of the archetypal experience should cover at least several most common archetypes. A study of a larger pool of archetypes would also enable us to discover the differences between responses to individual archetypes. The assortment of physiological measurements had to be expanded in order to develop more accurate prediction models that could benefit from a rich source of physiological data. Finally, the stimuli for elicitation of the archetypal experience had to be modified because during the experiment we observed that demonstration of pictures and sounds for brief moments of time could not strongly affect the emotional states of the participants. Moreover, certain mental experiences may require larger temporal ranges to develop. 7.3 physiological correlates of watching film clips with archetypal appearances Having obtained the initial evidence that exposure of people to the stimuli with archetypal content may result in a noticeable change of their heart rate, we proceeded with a more detailed investigation of the methods for capturing the archetypal experience using physiological signals. This investigation required us to conduct two empirical studies, which outcomes are discussed in the next two subsections. 7.3 physiological correlates of archetypal film clips 131 7.3.1Study 1 This study was a considerable improvement in terms of the methodological procedure and the physiological measurements in comparison with the previous experiment. First of all, we replaced pictures and sounds with film clips that proved to be more powerful in capturing the attention of the participants thanks to their dynamic display including visual and audio modalities. We expected that application of the film clips in this study would enable us to elicit more intensive psychophysiological reactions in the subjects. Another modification was related to the diversity of the archetypal experience being investigated. We performed a survey of the literature about archetypal symbolism and identified eight of the most important and frequently found archetypes. Also, the array of physiological measures was considerably expanded. In this study, four physiological signals including cardiovascular, electrodermal, respiratory activities, and skin temperature were monitored. Similarly to the experiment with pictures and sounds the participants were required to provide retrospective evaluations of their feelings using the SAM instrument. Moreover, there was a number of less significant modifications that were thoroughly described in Chapter 4. The amount of the experimental data collected in this experiment was considerably larger than in the previous study. The analysis of the collected data was started with running several statistical tests that suggested a significant relationship between most of the physiological signals monitored during the experiment and the categories of the archetypal film clips demonstrated to the participants. A notable exception was the signal of skin temperature that apparently had a very low time resolution. Probably, the variations in the skin temperature were too slow to reflect the changes in psychological conditions of the subjects. Overall, the results of the statistical analysis were in line with the findings obtained in our previous study. Next, we proceeded with training prediction models and evaluating their classification performance. In this area of analysis, a considerable improvement of the classification accuracy comparing to the previous experiment was expected because the prediction models could be trained on a larger amount of the physiological recordings. The training of the prediction models was performed with five well-established data mining techniques. Using the obtained models we could distinguish between five classes of the explicit emotion with an accuracy of up to 36.8percent. This classification performance was close to the best results reported by other researchers in affective computing. Based on the fact that we could reproduce their results with regard to the explicit emotions, one could conclude that our methodology was correct. The results related to the evaluation of the prediction models trained on the physiological data corresponding to the archetypal experience 132 general discussion were close to the observations for the explicit emotions. In fact, taking into account a large number of classes, the classification accuracy for the archetypal experience was even slightly better than for the explicit emotions and amounted up to 29.5percent. Besides comparing the results with other studies in recognition of affect, we could also set them against the observations from the previous experiment. In comparison with the previous study, this prediction model performed considerably better. As a next step of analysis, we looked at the data collected with the SAM instrument. The analysis showed that the data from the introspective reports of the participants could predict the category of the film clips worse than the physiological recordings. Further examination revealed that the prediction models trained on the SAM data could provide more reliable classification for the explicit emotions than for the archetypal experience. Based on these observations, one could speculate that since the participants were not consciously aware of the archetypal properties of the film clips, they were not able to provide reliable self-reports. On the other hand, the accuracy of the models obtained from the SAM data was higher for the films eliciting the explicit emotions because the subjects could easily recognize and describe their psychological state using the rating scales. If the results acquired using the SAM ratings and the physiological data were compared with each other, it was clear that, in case of the archetypal film clips, the classification techniques based on the physiological signals outperformed the models built using the data from the self-reports. This observation suggested that even though accurate self-reports could not be provided by the participants, their physiological signals still responded to the presented stimuli. In general, this experiment enabled us to collect a significantly large amount of physiological data that could be used to build robust prediction models. Moreover, the scope of the previous study was considerably broadened by introducing eight types of the archetypal experience corresponding to the common archetypes. Unfortunately, we could not avoid several limitations in the design of the experiment. The most critical limitation was related to the number of the film clips utilized in the study. As each category of the explicit emotions or the archetypal experience was represented with only one video clip, there was a potential threat that the data mining algorithms could produce prediction models specific to a particular clip rather than to a whole category. This fact significantly limited the generalizability of our findings to other experimental settings. Furthermore, it was difficult to make any firm conclusions based solely on this experiment. For this reason, it was decided to design and carry out another study that would have similar experimental settings but at the same time could address the limitations we identified. A discussion of the findings obtained in that study is presented in the next subsection.