Optimizing the digital customer journey—Improving user experience by exploiting emotions, personas and situations for individualized user interface adaptations

Märtin, Christian; Bissinger, Bärbel Christine; Asta, Pietro Article — Published Version Optimizing the digital customer journey—Improving user experience by exploiting emotions, personas and situations for individualized user interface adaptations Journal of Consumer Behaviour Provided in Cooperation with: John Wiley & Sons Suggested Citation: Märtin, Christian; Bissinger, Bärbel Christine; Asta, Pietro (2021) : Optimizing the digital customer journey—Improving user experience by exploiting emotions, personas and situations for individualized user interface adaptations, Journal of Consumer Behaviour, ISSN 1479-1838, Wiley, Hoboken, NJ, Vol. 22, Iss. 5, pp. 1050-1061, https://doi.org/10.1002/cb.1964 This Version is available at: https://hdl.handle.net/10419/288144 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. http://creativecommons.org/licenses/by-nc-nd/4.0/ SPECIAL ISSUE ARTICLE Optimizing the digital customer journey—Improving user experience by exploiting emotions, personas and situations for individualized user interface adaptations Christian Märtin | Bärbel Christine Bissinger | Pietro Asta Faculty of Computer Science, Augsburg University of Applied Sciences, Augsburg, Germany Correspondence Christian Märtin, Faculty of Computer Science, Augsburg University of Applied Sciences, Augsburg, Germany. Email: [email protected] Abstract This paper discusses a novel approach for exploiting emotions and situation-aware software adaptation methods for individualizing some of the touch points of the digital customer journey and thereby optimizing customer experience and effectiveness of e-commerce applications. Our approach uses emotion recognition, eye-tracking, and other individual tracking methods as well as customer personas for adapting interactive web applications by accessing a flexible adaptation framework at runtime. The framework allows for individualization at runtime by applying situation-aware adaptations. Two experimental customer studies were carried out in the e-commerce domain in order to provide a basis for exploitable emotion- and persona-related situational changes. The results of the studies were used to demonstrate the potential of our situation analytic adaptation approach with examples from a commercial beauty-products e-business portal. 1|INTRODUCTION One aspect of digitalization in marketing is the design of IT-based solutions for the steps or cycles of the customer journey (Følstad & Kvale, 2018). The customer journey is the customer's interaction at several touch points with a service or several services of one or more service providers in order to achieve a specific goal (Halvorsrud et al., 2016). A generic customer journey is divided into the following five phases: awareness, where the customer is made aware of the product or service, favorability, where the interest of the customer is increased, so that the customer begins to take a closer look at the product and to inform herself about it, consideration, which increasingly triggers the customer's desire to own the product, intent to purchase, where the customer's intention to buy the product is being initiated, and, finally conversion, where the product will ultimately be bought by the customer. It also makes sense to add a post purchase phase to the customer journey. A distinction is made between direct and indirect touch points. The website, advertising spots and advertisement in general are understood as direct touch points. Indirect touch points for example include rating portals, user forums and blogs and can only be influenced to a limited extent by the provider. Optimizing the customer journey is very important and necessary for the success of e-commerce providers. By using suitable tracking technologies, the behavior of consumers can be analyzed in real-time. Analysis can also reveal all contact points created through advertising. Thanks to this knowledge, it is possible to identify optimization potential. It must be the provider's goal at every touch point, to create a situation that leads to optimum user experience (UX) for the potential customer (Stein & Ramaseshan, 2016). UX in the customer journey is often described as customer experience (CX). CX aims at “a customer's cognitive, emotional, behavioral, sensorial, and social”reactions to the offerings of a provider or a business “during the customer's entire purchase journey”(Lemon & Verhoef, 2016). This paper is a revised and extended version of (Märtin et al., 2020). In our previous research we have designed and engineered the SitAdapt system that is able to observe activities and recognize emotions of users Received: 1 November 2020 Revised: 10 March 2021 Accepted: 18 May 2021 DOI: 10.1002/cb.1964 This is an open access article under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License, which permits use and distribution in any medium, provided the original work is properly cited, the use is non-commercial and no modifications or adaptations are made. © 2021 The Authors. Journal of Consumer Behaviour published by John Wiley & Sons Ltd. 1050 J Consumer Behav. 2023;22:1050–1061. wileyonlinelibrary.com/journal/cb during their work in any interactive application environment. By applying situation rules the system exploits the gathered data in order to individually adapt the user interface and the behavior of the observed interactive application (Märtin et al., 2019). In this paper we lay the foundation for using SitAdapt for accompanying individual users during the different phases of the customer journey in the e-business domain. We create a suitable experimental setting for analyzing the following research questions: •Q1: When are emotions triggered in e-business environments and how can they be exploited for optimizing the digital customer journey? •Q2: Can predefined personas lead to better customer experience in e-business environments? •Q3: Can direct observation of users lead to more individualized adaptations of the interactive e-business applications? •Q4: Can situation-aware adaptations help to achieve better task accomplishment of the users? •Q5: Can situation-aware adaptations help to achieve the providers' business goals more effectively? The data we collected during our experimental studies give preliminary answers to these research questions and provide valuable input for triggering and targeting larger and more specific subsequent studies. The remainder of this paper is organized as follows: •Section 2gives an overview of related and previous work relevant to the scope of this paper. •Section 3presents two experimental studies that evaluate the role of emotions and customer personas in order to raise customer experience for the b2c part of an existing e-commerce portal for beauty products. •Section 4discusses, how SitAdapt can exploit the findings and results of the studies by specifying and triggering appropriate situation rules. •Section 5concludes the paper and gives an outlook on our planned future work. 2|MATERIALS AND METHODS This section discusses some of the methods from the disciplines that have influenced and inspired our work. We also discuss the functionality of the SitAdapt system. 2.1 |Emotions, decision-making and user experience 2.1.1 | Emotions and decision-making In our society, emotions and emotional reactions are often perceived with a negative connotation even though emotions play an important role in all human actions. Emotions directly influence interaction between humans as well as human–computer-interaction (HCI). This means that emotions are not only linked to social environments or to the entertainment industry, but that they are also essential for human communication, human perception and decision-making (Picard, 2000). In the past, scientists were assuming that human beings primarily behave in a rational way. Today's expertise and research prove the opposite. It is now known that emotions trigger actions and are involved in human processes like •Evaluation of situations •Motivation •Preparation •Actions •Regulations •Social Tasks (Broschart & Monschein, 2017) These processes cover a broad field of behavior and activities. It can be seen that the power of emotions influences the decision making processes in many situations and areas of life. In 1994, a study by the neuroscientist Ant onio Dam asio showed, how emotions affect decision making. Using a strategic card game, Dam asio could prove that emotions are essential for long-term planning. Some of the participants of the study had a disorder of the cerebral cortex and were not able to evaluate actions emotionally. These players were at a disadvantage compared to the other participants, because the average persons could use their emotions to help them make long-term decisions which was necessary in this strategic card game (Dam asio, 2006; Müsseler & Rieger, 2017). We human beings are often not aware of how much influence emotions have on our choices. Even if we assume that we make a decision based on rational reasons, this is usually not the case - especially for long-term considerations. In some situations, it is also possible that emotions are the only reason for our judgment. In these cases, no logical element is necessary to pass a resolution. However, since we often consider a logical component to be important in our judgment, the logical element is added afterwards in order to reinforce and rationalize the emotionally made decision (Broschart & Monschein, 2017). This is how people avoid negative emotions, which would arise when opinions, attitudes, perceptions or thoughts do not match. This phenomenon is called cognitive dissonance, which is perceived as an uncomfortable tension for people and which needs to be reduced in order to build a coherent overall picture and a harmonious balance (Moser, 2015). The opinion that decisions need to be made rational leads to the phenomenon that humans subconsciously add a logical component after the decision has already been made. This shows that supposedly rational decisions can often be traced back to emotions without people being aware of it. Even though emotions have such a high significance, there is no explicit definition for it and many different statements can be found. Kleinginna and Kleinginna have collected the similarities of around 100 assertions and definitions and have created a working definition that combines cognitive, mental, physiological and physical aspects (Kleinginna Jr. & Kleinginna, 1981). One possibility for categorization MÄRTIN ET AL.1051 are fundamental emotion models which are based on Darwin's findings of basic emotions. A famous example are the six basic emotions discussed by Ekman, which refer to facial expressions and should be valid regardless of a person's language and culture (Ekman, 1992a; Ekman, 1992b; Stürmer & Schmidt, 2014). Another option to categorize emotions are dimensional models. Two prominent dimensions are valence and arousal which are for example used in the model of Russel. The dimension of valence indicates the extent to which the emotion is felt positively or negatively. The arousal dimension shows how strong the emotion is and how relevant the environment is (Valenza et al., 2014). 2.1.2 | Emotions and user experience Since emotions play a central role in human behavior and decisionmaking, they are meaningful for marketing and advertising measures, in classical and digital marketing. The choices of the customers determine whether a product or a company will be accepted or declined in the market. Consumer excitement, great anticipation and loyalty to a brand or products are achievements which can only be reached by creating and maintaining emotional value. This is not only valid for an individual product, but for the whole experience users gather with the products, brands or companies. While usability describes the extent to which an interactive system is effective, efficient and satisfying to use in a specified context of use, user experience (UX) considers satisfaction before, during and after use (UXQB, 2020). For online presence or digital touch points a smooth interaction due to compliance with usability rules, that is, an expected and faultless operation, is nowadays a prerequisite. A friendly user interface still leads to a competitive advantage, but does not mean that users are addressed emotionally or that they have a great experience by using the interface. This is only achieved when users feel excitement and a joy of use (Dorau, 2011). When emotions are taken into account while creating and adapting user interfaces, more intelligent and human like interfaces can be designed. The attention of the user can thus be controlled more easily, which results in a more natural perception for the user. Computer systems and user interfaces can be adapted to people and not the other way around (Picard, 2000). 2.2 |Business intelligence in e-commerce Whenever consumers enter the individual phases of the customer journey and get into contact with a provider through direct and indirect touch points, huge amounts of data are produced and transmitted to the respective provider. In order to generate valuable knowledge from such data, business intelligence (BI) can be applied. BI combines the use of various methods and technologies that serve the collection, administration, evaluation and presentation of mainly external but also internal data in digital form (Gluchwoski & Chamoni, 2016). By using adequate tools data patterns, cross-connections and correlations can be discovered and trends can be predicted. In the focus of the applied analytical methods are data mining that uses statistical algorithms as well as the hypothesis-based online analytics processing (OLAP), which allows a multi-dimensional analysis of data. In (Chen et al., 2012) the authors predicted for the now current generation of BI and analysis tools to focus on mobile and sensor-based content, location-awareness, person-centered and context-relevant analysis as well as mobile visualization techniques and HCI aspects. Results from BI analysis can have an impact on the design of ecommerce sites and provide insights on how to organize content parts, advertising, navigation and presentational aspects. In our approach the large quantities of user and situational data recorded by the SitAdapt 2.1 recording component (see Section 2.4) can be accessed by BI tools. Analysis results can therefore evolutionarily influence customer experience in all phases and for all direct touch points in future sessions. 2.3 |Persona-based design Another technique with positive impact on the customer journey and e-commerce in general can be the use of personas. In the context of this paper a “persona is an archetype of a class of users synthesizing goals and behavior patterns as well as skills, attitudes and environment. The user's characteristics […] must be effective for the design problem at hand.”(De Marsico & Levialdi, 2004). For each persona, an avatar is defined, that is equipped with authentic features such as name, photo, curriculum vitae, age, income, marital status, hobbies, education, etc. (Klünen, 2019). In our approach to optimizing the customer journey, a persona represents a certain customer group of a provider. Most information required to design personas, as well as all knowledge about their preferences, results from expert analysis of existing customer data. The use of personas helps us to create a customized approach for specific customer groups, that is, to work with specific visual, verbal and audio attributes. Also, persona-specific product preferences are taken into account. This allows the personaspecific tailoring of the marketing measures, in particular the choice of advertising material to be used, as well as of the web design of the ecommerce site. Personas are also often used in HCI and software engineering (Jahavery et al., 2009). In (McGinn & Kotamraju, 2008) an alternative approach to finding personas is discussed: In order to reach specific HCI design goals, a survey was conducted with members of the different target groups, resulting in useful personas with attributes specific to tasks of each user group, but without organizational overhead. 2.4 |Context- and situation-awareness The concept of context-aware computing was first proposed for distributed mobile computing by (Schilit & Theimer, 1994). In addition to technical aspects the definition of context also included 1052 MÄRTIN ET AL. environmental and social attributes. Later the term situationawareness appeared in psychology and the cognitive sciences with the aim to support correct task accomplishment of human operators in complex situations by defining situation-dependent guidance (Flach et al., 2004). In recent years interactive software has made huge steps towards understanding of and reacting to varying situations. To capture the individual requirements of a situation, (Chang, 2016) proposes that a situation consists of an environmental context Ethat covers the user's operational environment, a behavioral context Bthat covers the user's social behavior by interpreting his or her actions, and a hidden context Mthat includes the users' mental states and emotions. 2.4.1 | SitAdapt Our own work, the SitAdapt system (Märtin et al., 2019), (Herdin & Märtin, 2020) was inspired by Chang. The system (Figure 1)usesa broad set of visual and other observation and monitoring tools and has been tested and evaluated for a number of applications from different e-business domains. All data recorded in a user session are stored in very fine-grained situation profiles (minimum time resolution: 1/60 s). Possible adaptations of the target web application are planned and premodeled at development time. At runtime they are triggered by situation rules and generated by activating and exploiting domain-dependent and independent actions and/or HCI- patterns. For applying SitAdapt inthee-commercedomainwehave exploited the results of two experimental user studies in order to optimize our approach (see Section 3). The current software version, the SitAdapt 2.1 system, is now able to cover broad parts of the customer journey. Section 4demonstrates the capabilities of the SitAdapt system with application examples from e-commerce. 2.4.2 | SitAdapt architecture SitAdapt 2.1 consists of the following parts: •The data interfaces use the different APIs of the devices (eyetracker, wristband, facial expression recognition software interface, metadata from the application) to collect data about the user. SitAdapt 2.1 uses two different data types for generation and adaptation of the user interface received from the different input devices. FIGURE 1 Structure and components of the SitAdapt system (Märtin et al., 2019) TABLE 1 Data input from the eye tracking system (Herdin & Märtin, 2020) Attribute Possible values Description LeftPupilDiameter Between circa 2.0 and 8.0 Describes the dilation of the subject's left pupil in mm RightPupilDiameter Between circa 2.0 and 8.0 Describes the dilation of the subject's right pupil in mm LeftEyeX Between 0.0 and 1.0 Indicates the normalized xcoordinate of the subject's left eye LeftEyeY Between 0.0 and 1.0 Indicates the normalized ycoordinate of the subject's left eye RightEyeX Between 0.0 and 1.0 Indicates the normalized xcoordinate of the subject's right eye RightEyeY Between 0.0 and 1.0 Indicates the normalized ycoordinate of the subject's right eye MÄRTIN ET AL.1053 •The recording component synchronizes the different input records with a timestamp. In Table 1, for instance, the attribute value ranges are listed that can be received from the eye tracking system API. Table 2 shows the possible attributes and values from emotion tracking. •The database writer stores the data from the recording component and from the browser in the database, where the raw situations and situation profiles are managed. It also controls the communication with the rule editor. •The rule editor (Figure 2) allows the definition and modification of situation rules, for example, for specifying the different user states and the resulting actions. The rule editor can use all input data types and attribute values as well as their temporal changes for formulating rule conditions. At runtime rules are triggered by the situation analytics component for adapting the user interface, if the conditions of one or more rules apply. However, situation rules can also activate HCI- patterns in a pattern repository (Märtin et al., 2019). •The situation analytics component analyzes and assesses situations by exploiting the observed data. Situation rules are triggered when the rule conditions are satisfied. Situation rules interact with the situation profiles stored in the SitAdapt 2.1 database. •The evaluation/decision component uses the data that are provided by the situation analytics component to decide whether an adaptation of the user interface is currently meaningful and necessary. The component evaluates one or more applicable situation rules and has to solve possible conflicts between the rules. •The adaptation component generates the necessary modifications of the interactive target user application. 3|RESULTS 3.1 |Experimental user study 1: Can emotions be exploited for UX optimization in e-commerce? To discover what triggers emotions of users, and how this information can be used to optimize the user experience, we did a test in our lab with customers and potential customers of Dr. Grandel GmbH, a German manufacturer in the cosmetics sector. The experimental study was focused on emotional responses at digital touch points. In the study the real world b2c website of the company with an interactive tool, an online shop and an online magazine was used. Other elements were three different online advertising measures. We wanted to find out whether the usage of these different elements triggers emotional responses of the users. In case they do, we wanted to know which emotions were shown and what the concrete triggers for the measured responses were. 3.1.1 | Research subject The first part of the study was centered around an interactive tool on the website, which helps users to find the most suitable products for them. The users were asked to describe themselves with the support of predefined answers and pictures. This is a very individual procedure and lots of decisions need to be made that involve emotions. Another part of the research with the website was focused on the online shop and the process from product search to product selection. The last element of the test that was related to the website included an online-magazine with editorial articles for entertainment and news about trends and products in the cosmetics industry. Additionally, selected advertisements on external websites, advertising banners on the own online channels as well as newsletters and Instagram posts were analyzed. All these elements were tested in order to find out, whether they trigger emotions that can be measured. According to Jakob Nielsen, the best results for the usability evaluation of an interactive application will appear when only five users are testing the target object with as many small tests as possible (Williams, 2004). As we differentiated between two user groups, we used two groups, each of them comprising of four people, which is recommended when testing with different user groups (Tullis & Albert, 2013). One group consisted of female customers of the company who already knew the company and its products. The second group consisted of female users, who did not know the products yet, but who did fit into the target group of the company's products. 3.1.2 | Test setting and tools The experimental study included a pretest to gather information about the earlier experience of the participants and to assign the participants to the different groups, a lab test, which was the main part of the study, and a post-test to enquire the individual personal opinion of the participants. The lab test was divided into three test scenarios with eight tasks which were building on one another. During the test, the participant was seated in an ordinary office room in front of a personal computer. The researchers could observe the participants through a one-way mirrored window and with a webcam that recorded the sessions and logged interaction details. To measure emotional responses of the participants during different stimuli, Tobii Studio eye-tracking software, FaceReader 7 facial expression recognition software and an Empatica E4 wristband to measure the heart rate and skin conductance as indicators for some emotional states were used. 3.1.3 | Selected results of the study Interactive tool The interactive tool with the individual result could trigger emotions which could be measured. Figure 3shows a section of the results of the facial expression recognition software during the interaction of a participant with the tool. Since emotions do not always occur in a pure form, it is possible that the facial expression shows several 1054 MÄRTIN ET AL. emotional states. The FaceReader software therefore shows the analyzed expressions proportionally. For classifying basic emotions FaceReader uses a deep neural network trained with 10,000 annotated facial images for detecting the basic emotions in the algorithmically runtime-generated 3D model which shows the exact position of 500 relevant key points in the currently observed face. The TABLE 2 Input data from emotion tracking (Märtin et al., 2019) Attribute Possible values Description Angry between 0.0 and 1.0 Indicates to what degree the user appears angry Disgusted between 0.0 and 1.0 Indicates to what degree the user appears disgusted Happy between 0.0 and 1.0 Indicates to what degree the user appears happy Neutral between 0.0 and 1.0 Indicates to what degree the user appears neutral Sad between 0.0 and 1.0 Indicates to what degree the user appears sad Scared between 0.0 and 1.0 Indicates to what degree the user appears scared. Surprised between 0.0 and 1.0 Indicates to what degree the user appears surprised Contempt between 0.0 and 1.0 Indicates to what degree the user appears full of contempt Contempt is defined as the feeling that a person or a thing is beneath consideration, worthless, or deserving scorn Valence between 1.0 and 1.0 Describes how positive/negative an emotion is. 1 means very negative. Arousal between 0.0 and 1.0 Describes how strong the emotion is. A person that is yelling in anger, for example, has a higher arousal than a person who is only pulling their eyebrows together. Quality between 0.0 and 1.0 Roughly describes the quality of the observation Age Agerange (“from-to”) Describes a numeric interval for the estimated age of the user Beard ”None,”“Some,”“Full”Describes the extent of the subject's lower facial hair Moustache “None,”“Some,”“Full”Describes the extent of the subject's upper facial hair. Glasses ”Yes,”“No”Describes whether or not the user is wearing glasses Ethnicity “Caucasian,”“Eastern Asian,”“South Asian,”“African,”“Other” Describes the ethnicity of the subject based on visual appearance. Gaze Direction Identity “Left,”“Right,”“Forward”Describes in which direction the subject is looking. Identity “unknown person,”“no identification”Identifies the user based on the current Noldus Face-Reader session's profiles. Left Eye “Open,”“Closed”Describes the state of the user's left eye. Right Eye “Open,”“Closed”Describes the state of the user's right eye. Mouth “Open,”“Closed”Describes the state of the user's mouth. Left Eyebrow “Raised,”“Lowered,”“Neutral”Describes the state of the user's left eyebrow. Right Eyebrow “Raised,”“Lowered,”“Neutral”Describes the state of the user's right eyebrow. MÄRTIN ET AL.1055 FaceReader software measures the intensity of each basic emotion in a range between 0 and 1. Note, however, that the sum of the intensities of all currently involved emotions is not typically equal to 1, as the intensity of each emotion is handled separately (Loijens & Krips, 2019). In this example, the dominant expression of the participant during the usage of the tool is happy which proves a positive user experience during the interaction for this user. The results also show (see Figure 4a,b) that the arousal and the valence, which are indicators for emotional reactions and are also measured by FaceReader, change with the beginning of the interaction with the tool. The valence indicates, whether an emotional state is positive or negative. The arousal shows the activeness of an emotional state. This example illustrates the correlation of valence and arousal for physical reactions. In the second part of the graph, at 09:37:00, the interaction with the tool starts. For the observed user the graph proves that the activeness of emotional states is higher and more positive during an interactive touchpoint. Magazine and newsletter Figure 5shows the emotional reaction of one of the participants, when an unexpected visual appeared, after a button leading to some promotional action was pressed. The values of the basic emotions happy and surprised are reaching quite significant levels. In the next example (Figure 6) a participant is looking at a specific winter skin cream. Upon reading the detailed description of the product Winter Silk Cream, the user's emotional state significantly changes to happy. A situation rule could now exploit this knowledge to give additional information about other winter products. Advertisements In the next example (Figure 7a,b), the system has gathered a priori knowledge about the varying gaze behavior of participants, who are known customers of the business or participants who do not know the products by distinguishing between the lab-created heat maps. These heat maps show that customers focus on the images, while noncustomers concentrate on the textual description of the products. The gaze behavior with respect to this image can be used to categorize anonymous users. The customer experience during the FIGURE 2 SitAdapt rule editor (Märtin et al., 2019) FIGURE 3 Emotions measured by FaceReader 7 software 1056 MÄRTIN ET AL. prepurchase phase can be improved. When the system assumes a returning customer, the focus of her further customer journey will be put on showing esthetic images, while in the other case more descriptive information will be given during the rest of the customer journey. These examples of our experimental findings show that the potential for UX optimization in e-commerce by exploiting emotions is very high and promising. Thus research question Q1 can be answered positively. The findings provide valuable input for the analysis of research questions Q3,Q4,andQ5inSection4.InSection4we discuss, how the observation FIGURE 4 (a) Valence of measured emotions during the test for a specific participant; (b) Arousal of measured emotions during the test for a specific participant FIGURE 5 Visual for a special promotion and measured emotional values FIGURE 6 A significant value of the emotion “happy”is observed for a candidate, when she is finding the right product MÄRTIN ET AL.1057