UEQ KPI Result Report of the Evaluation of Amazon and Skype

UEQ KPI Result Report of the Evaluation of Amazon and Skype Andreas Hinderks University of Seville Martin Schrepp SAP SE Francisco José Domínguez Mayo University of Seville María José Escalona Cuaresma University of Seville Jörg Thomaschewski University of Applied Science Emden/Leer December 2017 Contents Contents 1 Introduction 2 2 The Study 3 2.1 Participants.......................................... 3 2.2 DataAnalysisinGeneral .................................. 5 2.3 UEQKPI........................................... 5 3 Amazon Results 7 3.1 UEQKPI........................................... 7 3.2 UEQOverview ........................................ 7 3.3 UEQ Scales - Mean, Standard Deviation, and Confidence . . . . . . . . . . . . . . . . 8 3.4 UEQ Scales - Cronbach’s Alpha Coefficient . . . . . . . . . . . . . . . . . . . . . . . . 8 3.5 UEQBenchmark....................................... 9 3.6 Perceived Importance of UEQ Scales - Overview . . . . . . . . . . . . . . . . . . . . . 9 3.7 Perceived Importance of UEQ Scales - Mean, Standard Deviation and Confidence . . . 10 3.8 Correlation between Scale Mean and Perceived Importance . . . . . . . . . . . . . . . 10 4 Amazon Compare Countries 11 4.1 UEQ Scales - Mean, Standard Deviation, and Confidence . . . . . . . . . . . . . . . . 11 4.2 UEQ Scales - Cronbach’s Alpha Coefficient . . . . . . . . . . . . . . . . . . . . . . . . 12 4.3 Perceived Importance of UEQ Scales - Mean, Standard Deviation and Confidence . . . 12 4.4 UEQScales-T-Test..................................... 13 5 Skype Results 14 5.1 UEQKPI........................................... 14 5.2 UEQ-Overview ....................................... 14 5.3 UEQ Scales - Mean, Standard Deviation, and Confidence . . . . . . . . . . . . . . . . 15 5.4 UEQ Scales - Cronbach’s Alpha Coefficient . . . . . . . . . . . . . . . . . . . . . . . . 15 5.5 UEQBenchmark....................................... 16 5.6 Perceived Importance of UEQ Scales Overview . . . . . . . . . . . . . . . . . . . . . . 16 5.7 Perceived Importance of UEQ Scales - Mean, Standard Deviation, and Confidence . . 17 5.8 Correlation between Scale Mean and perceived Importance . . . . . . . . . . . . . . . 17 6 Skype Compare Countries 18 6.1 UEQKPI........................................... 18 6.2 UEQ Scales - Mean, Standard Deviation and Confidence . . . . . . . . . . . . . . . . . 18 6.3 UEQ Scales - Cronbachs Alpha-Coefficient . . . . . . . . . . . . . . . . . . . . . . . . . 19 6.4 Perceived Importance of UEQ Scales - Mean, Standard Deviation and Confidence . . . 19 6.5 UEQScales-T-Test..................................... 20 II Contents List of Figures 21 List of Tables 22 Bibliography 24 1 Introduction 1 Introduction In many companies, decisions are mostly made based on the key figures, such as turnover, profit, employee satisfaction, or the Net Promoter Score (NPS). Managers are accustomed to information being summarized and available via key figures. This leads to the demand for the user experience to also be mapped in a single KPI (UX KPI). Our main goal is to show the calculation of a UX KPI in practice by conducting two studies evaluating Amazon and Skype. In addition, we want to determine whether there are difference between the values of the UEQ scales and the perceived importance per UEQ scale. For this purpose, we have added six additional questions to the User Experience Questionnaire (UEQ) to determine the perceived importance of each scale of the UEQ. In this paper, we describe the detailed results of our studies. 2 The Study 2 The Study The study has been conducted in England, Spain, and Germany through online and paper versions of the questionnaire. For the English and German dataset, we have choosen a social panel (Prolific Academic1) to collect the data. We have collected the Spanish dataset from the University of Seville. First of all, the participants were asked to choose between Amazon and Skype. In the next step, they had to complete the UEQ [Laugwitz et al. 2008]. The UEQ is a semantic differential of 26 items that measures user experience in terms of attractivity, perspicuity, efficiency, dependability, stimulation, and novelty [Laugwitz et al. 2008]. Based on the six UEQ scales, the participants were asked in the following step to assess how important the scale and/or product quality is to them for the chosen product. In Table 2.1 the detailed statements are listed. Finally, the participants had to state their age and gender. Each participant only evaluated one product. 2.1 Participants A total of 882 participants took part in the study. Not all participants answered all items seriously. To detect the somewhat random or non serious answers and clean up the datasets, we used a simple heuristic [Schrepp 2016]. In the end, we identified 87 answers as problematic and deleted them from the datasets. The remaining answers are categorized as follows: Amazon Skype England 238 (158 females, 96 males) 215 (172 females, 82 males) Spain 51 (no data available) 62 (no data available) Germany 144 (52 females, 102 males) 85 (40 females, 52 males) Total 433 362 Table 2.2: Number of Participants Choosing Amazon or Skype 1https://www.prolific.ac 3 The Study Scale Statements Attractiveness [ENG] Amazon should look attractive, enjoyable, friendly and pleasant. [SPA] El sistema tiene que ser atractivo, agradable y simpático. [GER] Das Produkt soll attraktiv, angenehm und sympathisch wirken. Efficiency [ENG] I should perform my tasks with Amazon fast, efficient and in a pragmatic way. [SPA] El sistema debe ayudar a que yo haga mis tareas de forma rápida, eficiente y pragmática. [GER] Das Produkt soll mir helfen, meine Aufgaben schnell, effizient und pragmatisch zu erledigen. Perspicuity [ENG] Amazon should be easy to understand, clear, simple, and easy to learn. [SPA] El sistema debe ser entendible, claro y fácil de aprender. [GER] Das Produkt soll übersichtlich, verständlich und leicht zu lernen sein. Dependability [ENG] The interaction with Amazon should be predictable, secure and meets my expectations. [SPA] El uso del sistema debe ser predecible de usar y fácil de controlar. [GER] Die Bedienung des Produkts soll in der Bedienung vorhersehbar und gut kontrollierbar sein. Stimulation [ENG] Using Amazon should be interesting, exiting and motivating. [SPA] Trabajar con el sistema debe ser interesante, emocionante y de activación. [GER] Das Arbeiten mit dem Produkt soll interessant, spannend und aktivierend sein. Novelty [ENG] Amazon should be innovative, inventive and creatively designed. [SPA] El sistema debe ser original, innovadora y creativa. [GER] Das Produkt soll originell, innovativ und kreativ gestaltet sein. Table 2.1: Additional questions to assess the importance of the parameters to the participant. 4 The Study The average age is 34 years (35 for females, 33 for males) for the English dataset and 29 years (31 for females, 28 for males) for the German dataset. We have no socio-demographic information available for the Spanish dataset. 2.2 Data Analysis in General We have analyzed the results of the studies on Amazon and Skype in terms of the following aspects: •UEQ KPI: See detailed information and development of the UEQ KPI in Chapter 2.3. •UEQ Overview: Values for each UEQ scale [Schrepp 2015]. •UEQ Scale Statistic: Mean, standard deviation, and confidence value for each UEQ scale [Schrepp 2015]. •UEQ Scales Consistency: Cronbach’s Alpha Coefficient for each UEQ scale [Cronbach 1951]. •UEQ Benchmark: Comparison of the results with the UEQ Benchmark [Schrepp 2017b]. •Perceived Importance: Values of the perceived importance of each UEQ scale. The results are calculated on the basis of the additional statemens (see Table 2.1). •Perceived Importance Statistic: Mean, standard deviation and confidence of the perceived importance of each UEQ scale. In addition we calculate the correlation between Scale Mean and Perceived Importance. In addition to the analysis for Amazon and Skype, we have compared the results from the three countries. 2.3 UEQ KPI The central idea of our approach is to combine for every participant the value and perceived importance of each UEQ scale [Schrepp 2017a]. Therefore, we calculated for each participant the relative importance of each scale and multiplied this with the value of the UEQ scale. Finally, we have generated a UX KPI based on the UEQ. We have named the resulting value UEQ KPI. The steps are described in detail below. 5 The Study •The result of the UEQ represents a value per participant Paiand for every scale: attractiveness Ai, perspicuity Pi, efficiency Ei, dependability Di, stimulation Si, and novelty Ni. •The result of the six additional questions is a value of the perceived importance of each scale: attractiveness ai, perspicuity pi, efficiency ei, dependability di, stimulation si, and novelty ni. •We have calculated the relative importance per participant using the given perceived importance of each scale. For example for attractiveness Awi=ai/(ai+pi+ei+di+si+ni). •The UEQ KPI per participant can be calculated by multiplying the mean value of the UEQ scale with its relative importance: UEQ KPIi=Awi∗Ai+Pwi∗Pi+Ewi∗Ei+Dwi∗Di+ Swi∗Si+Nwi∗Ni. The UEQ KPI has been calculated using the following formula: UEQ KP I =1 n n X i=1 (Awi∗Ai+Pwi∗Pi+Ewi∗Ei+Dwi∗Di+Swi∗Si+Nwi∗Ni)(2.1) We have used this formula to calculate the UEQ KPI for the two studies conducted for Amazon and Skype. In the following chapters, we present the results of the studies in detail. 6 Amazon Results 3 Amazon Results 3.1 UEQ KPI UEQ KPIAmazon Overall = 1.336 NAmazon Overall = 433 3.2 UEQ Overview Figure 3.1: Amazon Overall - UEQ Overview NAmazon Overall = 433 Confidence as error bar. 7 Skype Results 5 Skype Results 5.1 UEQ KPI UEQ KPISkype Overall = 0.896 NAmazon Overall = 362 5.2 UEQ - Overview Figure 5.1: Skype Overall - UEQ Overview NSkype Overall = 362 Confidence as error bar. 14 Skype Results 5.3 UEQ Scales - Mean, Standard Deviation, and Confidence Scale Mean Std. Dev. N Confidence Confidence interval Attractiveness 1.086 1.123 362 0.116 0.971 1.202 Perspicuity 1.255 1.145 362 0.118 1.137 1.373 Efficiency 0.924 1.106 362 0.114 0.810 1.038 Dependability 0.860 0.998 362 0.103 0.758 0.963 Stimulation 0.727 1.052 362 0.108 0.619 0.836 Novelty 0.444 1.015 362 0.105 0.340 0.549 Table 5.1: Skype Overall - UEQ Scales - Mean, Standard Deviation and Confidence Std. Dev. ≡Standard Deviation 5.4 UEQ Scales - Cronbach’s Alpha Coefficient Scale Alpha Attractiveness 0.920 Perspicuity 0.832 Efficiency 0.798 Dependability 0.710 Stimulation 0.790 Novelty 0.739 Table 5.2: Skype Overall - UEQ Scales - Cronbachs Alpha-Coefficient NSkype Overall = 362 15 Skype Results 5.5 UEQ Benchmark Figure 5.2: Skype Overall - UEQ Benchmark NSkype Overall = 362 5.6 Perceived Importance of UEQ Scales Overview Figure 5.3: Skype Overall - Perceived Importance of UEQ Scales - Overview NSkype Overall = 362 Confidence as error bar. 16 Skype Results 5.7 Perceived Importance of UEQ Scales - Mean, Standard Deviation, and Confidence Scale Importance -3/3 Std. Dev. N Confidence Confidence interval Attractiveness 5.474 1.474 1.3080 346 0.138 1.336 1.612 Perspicuity 6.218 2.218 0.9871 362 0.102 2.117 2.320 Efficiency 6.366 2.366 0.8228 361 0.085 2.281 2.451 Dependability 6.199 2.199 0.8643 362 0.089 2.110 2.288 Stimulation 4.925 0.925 1.4462 362 0.149 0.776 1.074 Novelty 5.014 1.014 1.3894 362 0.143 0.871 1.157 Table 5.3: Skype Overall - Perceived Importance of UEQ Scales - Mean, Standard Deviation and Confidence NSkype Overall = 362 -3/3 ≡Importance value from 1 to 7 adjusted to -3 to +3 Std. Dev. ≡Standard Deviation 5.8 Correlation between Scale Mean and perceived Importance Scale Correlation Attractiveness 0,149 Perspicuity 0,172 Efficiency 0,139 Dependability 0,155 Stimulation 0,237 Novelty 0,159 Table 5.4: Skype - Correlation between Scale Means and Perceived Importance 17 Skype Compare Countries 6 Skype Compare Countries 6.1 UEQ KPI Figure 6.1: Skype - Compare Countries - UEQ KPI NSkype ENG = 215; NSkype SPA = 62; NSkype GER = 85 6.2 UEQ Scales - Mean, Standard Deviation and Confidence Scale England Spain Germany Mean Std. Dev. Conf. Mean Std. Dev. Conf. Mean Std. Dev. Conf. Attractiveness 1.119 1.134 0.152 1.182 0.984 0.245 0.933 1.184 0.252 Perspicuity 1.331 1.190 0.159 1.165 0.865 0.215 1.126 1.203 0.256 Efficiency 1.045 1.121 0.150 0.683 1.012 0.252 0.794 1.099 0.234 Dependability 0.914 1.023 0.137 0.891 0.939 0.234 0.703 0.970 0.206 Stimulation 0.691 1.072 0.143 0.956 0.952 0.237 0.653 1.058 0.225 Novelty 0.407 1.004 0.134 0.560 1.029 0.256 0.453 1.037 0.220 Table 6.1: Skype - Compare Countries - UEQ Scales - Mean, Standard Deviation and Confidence 18 Skype Compare Countries NSkype ENG = 215; NSkype SPA = 62; NSkype GER = 85 6.3 UEQ Scales - Cronbachs Alpha-Coefficient Scale ENG Alpha SPA Alpha GER Alpha Attractiveness 0.924 0.896 0.928 Perspicuity 0.862 0.655 0.824 Efficiency 0.840 0.727 0.764 Dependability 0.715 0.712 0.679 Stimulation 0.806 0.771 0.766 Novelty 0.720 0.782 0.779 Table 6.2: Amazon - Compare Countries - UEQ Scales Cronbachs Alpha-Coefficient NSkype ENG = 215; NSkype SPA = 62; NSkype GER = 85 6.4 Perceived Importance of UEQ Scales - Mean, Standard Deviation and Confidence Scale England Spain Germany Mean Std. Dev. Conf. Mean Std. Dev. Conf. Mean Std. Dev. Conf. Attractiveness 1.414 1.434 0.192 1.696 1.072 0.310 1.506 1.065 0.226 Perspicuity 2.251 0.877 0.117 2.145 1.143 0.285 2.188 1.129 0.240 Efficiency 2.526 0.682 0.091 2.246 0.789 0.198 2.047 1.045 0.222 Dependability 2.316 0.804 0.108 2.097 0.882 0.219 1.976 0.951 0.202 Stimulation 0.823 1.540 0.206 1.242 1.237 0.308 0.953 1.318 0.280 Novelty 1.047 1.400 0.187 1.290 1.348 0.335 0.729 1.357 0.289 Table 6.3: Amazon - Compare Countries - Perceived Importance of UEQ Scales - Mean, Standard Deviation and Confidence NSkype ENG = 215; NSkype SPA = 62; NSkype GER = 85 19 Skype Compare Countries 6.5 UEQ Scales - T-Test ENG-SPA ENG-GER SPA-GER Attractiveness 0.672 No Sig. Diff. 0.217 No Sig. Diff. 0.168 No Sig. Diff. Perspicuity 0.226 No Sig. Diff. 0.184 No Sig. Diff. 0.820 No Sig. Diff. Efficiency 0.017 Sig. Diff. 0.078 No Sig. Diff. 0.527 No Sig. Diff. Dependability 0.869 No Sig. Diff. 0.097 No Sig. Diff. 0.239 No Sig. Diff. Stimulation 0.063 No Sig. Diff. 0.782 No Sig. Diff. 0.072 No Sig. Diff. Novelty 0.301 No Sig. Diff. 0.728 No Sig. Diff. 0.534 No Sig. Diff. Table 6.4: Skype - Compare Countries - UEQ Scales - T-Test Alpha level = 0.05 NSkype ENG = 215; NSkype SPA = 62; NSkype GER = 85 Sig. Diff. ≡Significant Difference 20 LIST OF FIGURES List of Figures 3.1 Amazon Overall - UEQ Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 3.2 Amazon Overall - UEQ Benchmark . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 3.3 Amazon Overall - Perceived Importance of UEQ Scales - Overview . . . . . . . . . . . 9 4.1 Amazon - Compare Countries - UEQ KPI . . . . . . . . . . . . . . . . . . . . . . . . . 11 5.1 Skype Overall - UEQ Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 5.2 Skype Overall - UEQ Benchmark . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 5.3 Skype Overall - Perceived Importance of UEQ Scales - Overview . . . . . . . . . . . . 16 6.1 Skype - Compare Countries - UEQ KPI . . . . . . . . . . . . . . . . . . . . . . . . . . 18 21 LIST OF TABLES List of Tables 2.2 Number of Participants Choosing Amazon or Skype . . . . . . . . . . . . . . . . . . . 3 2.1 Additional questions to assess the importance of the parameters to the participant. . . 4 3.1 Amazon Overall - UEQ Scales - Mean, Standard Deviation and Confidence . . . . . . 8 3.2 Amazon Overall - UEQ Scales - Cronbachs Alpha-Coefficient . . . . . . . . . . . . . . 8 3.3 Amazon Overall - Perceived Importance of UEQ Scales - Mean, Standard Deviation andConfidence........................................ 10 3.4 Amazon - Correlation between Scale Means and Perceived Importance . . . . . . . . . 10 4.1 Amazon - Compare Countries - UEQ Scales - Mean, Standard Deviation and Confidence 11 4.2 Amazon - Compare Countries - UEQ Scales Cronbachs Alpha-Coefficient . . . . . . . 12 4.3 Amazon - Compare Countries - Perceived Importance of UEQ Scales - Mean, Standard DeviationandConfidence .................................. 12 4.4 Amazon - Compare Countries - UEQ Scales - T-Test . . . . . . . . . . . . . . . . . . . 13 5.1 Skype Overall - UEQ Scales - Mean, Standard Deviation and Confidence . . . . . . . . 15 5.2 Skype Overall - UEQ Scales - Cronbachs Alpha-Coefficient . . . . . . . . . . . . . . . 15 5.3 Skype Overall - Perceived Importance of UEQ Scales - Mean, Standard Deviation and Confidence .......................................... 17 5.4 Skype - Correlation between Scale Means and Perceived Importance . . . . . . . . . . 17 6.1 Skype - Compare Countries - UEQ Scales - Mean, Standard Deviation and Confidence 18 6.2 Amazon - Compare Countries - UEQ Scales Cronbachs Alpha-Coefficient . . . . . . . 19 22 LIST OF TABLES 6.3 Amazon - Compare Countries - Perceived Importance of UEQ Scales - Mean, Standard DeviationandConfidence .................................. 19 6.4 Skype - Compare Countries - UEQ Scales - T-Test . . . . . . . . . . . . . . . . . . . . 20 23