Approaches to manage the user experience process in Agile software development: A systematic literature review

Information and Software Technology 150 (2022) 106957 Available online 30 May 2022 0950-5849/© 2022 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/bync-nd/4.0/). Contents lists available at ScienceDirect Information and Software Technology journal homepage: www.elsevier.com/locate/infsof Approaches to manage the user experience process in Agile software development: A systematic literature review Andreas Hinderks a,∗, Francisco José Domínguez Mayoa, Jörg Thomaschewski b, María José Escalona a aUniversity of Seville, Spain bUniversity of Applied Science Emden/Leer, Germany ARTICLE INFO Keywords: User experience management UX process User experience UX Usability HCI Agile methods Agile Systematic literature review ABSTRACT Context: Software development companies use Agile methods to develop their products or services efficiently and in a goal-oriented way. But this alone is not enough to satisfy user demands today. It is much more important nowadays that a product or service should offer a great user experience — the user wants to have some positive user experience while interacting with the product or service. Objective: An essential requirement is the integration of user experience methods in Agile software development. Based on this, the development of positive user experience must be managed. We understand management in general as a combination of a goal, a strategy, and resources. When applied to UX, user experience management consists of a UX goal, a UX strategy, and UX resources. Method: We have conducted a systematic literature review (SLR) to analyse suitable approaches for managing user experience in the context of Agile software development. Results: We have identified 49 relevant studies in this regard. After analysing the studies in detail, we have identified different primary approaches that can be deemed suitable for UX management. Additionally, we have identified several UX methods that are used in combination with the primary approaches. Conclusions: However, we could not identify any approaches that directly address UX management. There is also no general definition or common understanding of UX management. To successfully implement UX management, it is important to know what UX management actually is and how to measure or determine successful UX management. 1. Introduction Today’s users expect a high level of satisfaction while interacting with a product. They expect to be able to use the product without any major effort to finish their tasks in a quick and efficient manner. Moreover, for a product to succeed, it is important to consider hedonic interaction qualities — i.e. those that are not directly targetoriented [1]. In summary, the user wants to have a positive user experience while interacting with any product or service. A well-known definition of user experience is given in ISO 9241- 210 [2]. Here user experience is defined as ‘a person’s perceptions and responses that result from the use or anticipated use of a product, system or service’. Therefore, user experience is viewed as a holistic concept that includes all types of emotional, cognitive, or physical reactions concerning the concrete or even only the assumed usage of a product formed before, during, and after use. ∗Corresponding author. E-mail addresses: [email protected] (A. Hinderks), [email protected] (F.J. Domínguez Mayo), [email protected] (J. Thomaschewski), [email protected] (M.J. Escalona). A different interpretation defines user experience as a set of distinct quality criteria [1] that includes the classical usability criteria or pragmatic qualities, such as efficiency, controllability, or learnability, and non-goal directed or hedonic quality criteria [3] like stimulation, novelty, or aesthetics [4]. This definition has the advantage that it splits the general notion of user experience into a number of quality criteria, thereby describing the distinct and relatively well-defined aspects of user experience. Software development companies use Agile methods to develop products or services more efficiently. Agile methods (e.g. Scrum [5], Kanban [6], or Extreme Programming (XP) [7]) reduce the time taken to develop a product available in the market [8]. The iterative approach to developing software minimizes the risk of developing software that is not in line with what is needed in the market [9]. By performing https://doi.org/10.1016/j.infsof.2022.106957 Received 20 October 2020; Received in revised form 6 September 2021; Accepted 19 May 2022 Information and Software Technology 150 (2022) 106957 2 A. Hinderks et al. Fig. 1. User Experience Management based on McKeown [12]. retrospectives [5] at the end of an iteration, both product quality and Agile process quality can be improved. To develop the best possible product with great user experience, it is essential to have the right management in place in terms of UX. To the best of our knowledge, there is no approved definition of UX management in the literature. There is also no common understanding of what UX management is or how to apply it (Section 2). We generally understand management based on the explanations of Drucker [10] and Stone [11] — it is a combination of a goal, a strategy, and resources. When applied to UX, user experience management consists of a UX goal, a UX strategy, and UX resources (Fig. 1) based on the work of McKeown [12]. For example, a UX goal can be fixed to improve UX for a particular factor or quality criteria of UX. For this purpose, a UX strategy can be developed from different UX methods. For instance, to reach the UX goal, you can conduct a survey with a UX questionnaire, such as the User Experience Questionnaire [13] or the SUPR-Q [14], before and after the development. The UX strategy is that the results from the questionnaires after the development should be better than what they were before the development. Both UX strategy and UX resources are necessary to achieve the UX goal. It should be known before the next development iteration, whose requirements positively supported the UX goal. In this way can the UX goal be achieved in a goal-oriented manner. This paper reports the findings of a systematic literature review (SLR) in the field of approaches to manage the user experience process by focusing on Agile software development. This SLR will be addressed by the following research questions: •RQ1: Which approaches are suitable for UX management in an agile context? •RQ2: What conclusions can be deducted from the studies found? •RQ3: How can user experience in Agile software development be planned and controlled for a product backlog item or a requirement before the development? •RQ4: What retrospective proposals exist to improve the efficiency and effectiveness of the user experience process in terms of Agile software development? This paper is structured as follows: Section 2briefly summarizes the related work and presents gap analysis. Section 3present the review method including research questions of this SLR, search strategy, selection process, quality assessment, and data extraction. Section 4outlines the results and key findings of our study as well as the answers to our research questions. Section 5discusses the meaning of the findings and the limitations of our study. The paper ends with Section 6, with conclusions and ideas for future work. 2. Background and related work As already mentioned in the introduction, we did not find a definition of UX management nor a common understanding of the term in the literature. We searched for ‘‘user experience management’’ and similar terms in IEEEXpore, Science Direct, Scopus, Springer Link and ACM. The full search string we used was: (‘‘user experience management’’) OR (‘‘manage user experience’’) OR (‘‘ux management’’) OR (‘‘manage ux’’). In the end, we found five relevant paper. In these paper, there are various approaches or descriptions of UX management. The term UX management is often used without any explanation. We present the five paper in the next two paragraphs. In the literature, the term UX management is used differently. Szóstek [15] used the term UX management in the context of team building and empowerment. This includes career planning and development, team management, and training of individual team members. Anderson et al. [16] used a similar approach — in addition to building a UX team, they proposed that a C-level executive focus on user experience is necessary for UX management to have any corporate influence at all. For the implementation of UX management, Anderson et al. [16] and Rosenberg [17], for example, offered various patterns that provide support at the levels of planning, decision, tactics, and conflict. Another approach is the use of a UX maturity model. The advantage of using such a model is that it determines the current maturity level of an organization. Thus, its weaknesses can be identified. But the decisive factor is which dimensions are mapped in the UX maturity model. For example, the Total User Experience Management (TUXM) [18] model contains elements such as UX objectives, integrated design system, strategic communication, continual improvement, fact-based decisionmaking, and a T-type design team. The Nielsen Corporate Usability Maturity Model [19], on the other hand, comprises dimensions such as the developers’ attitude towards usability, the management’s attitude towards usability, the usability practitioner’s role, usability methods and techniques, and strategic usability. At first glance, it is noticeable that the TUXM model contains the dimension called UX objectives which is not present in Nielsen’s model. Conversely, the Nielsen model is more focused on practical implementation. The testing of a suitable UX maturity model should be carried out before deployment and tailored to the needs of the organization [19]. To the best of our knowledge, we did not find any paper in the literature that considers both managing UX process and Agile software development. But we found papers that analyse the integration of UX or similar methods and Agile software development. Therefore, the next section summarizes an overview of SLRs regarding the integration of UX and Agile software development. 2.1. Summary of related literature reviews In the literature, there are many reviews that investigate the integration of HCI and Agile methods. The term ‘Agile methods’ is used in the same way by all SLRs. However, there are differences in the processes or methods from the HCI area being used or integrated. The range of methods includes classic usability engineering, user-centred design (UCD) or human-centred design (HCD) [2], and UX methods in general, as well as design thinking. The next paragraphs briefly summarize the SLRs found on the basis of our search results (Section 4.1). In Fig. 2, the SLRs are arranged in chronological order on a timeline. The 2010 SLR by Bruun [20] investigated whether developers are trained in usability engineering so that they can apply usability engineering methods themselves. One of the main results of the SLR is the following finding: usability engineering is mainly published with Information and Software Technology 150 (2022) 106957 3 A. Hinderks et al. Fig. 2. Systematic Reviews in a Chronological Order. a university or laboratory focus. Only a small part was dedicated to training the developers of usability methods. In the end, only one study could be identified that covered the essential aspects, namely userbased methods, training costs, focus on organizational contexts, and practitioners. In 2011, Silva et al. [21] conducted an SLR on the integration of Agile methods and user-centred design. The authors analysed how usability problems are handled in Agile projects. The authors identified the following key aspects that play an essential role in integration: little upfront design, prototyping, user stories, user testing, inspection evaluation, and one sprint ahead. In 2014, Salah et al. [22] analysed the current state of Agile and User-Centred Design Integration (AUCDI). Their analysis should identify the factors for the integration of Agile method and usercentred design. Besides, the authors examined the challenges and key aspects to ensure a successful integration. The identified key aspects are lack of allocated time for upfront activities, difficulty of modularization, optimizing the work between developers and UCD practitioners, performing usability testing, and lack of documentation. In 2014, the SLR of Jurca et al. [23] analysed the literature to derive findings and recommendations for the integration of Agile and UX (Agile-UX). One finding is that Agile-UX methods are not anchored in companies and therefore do not receive the necessary support. Besides, UX designers are reworked and not part of the development team, but they are responsible for several development teams. This has been shown to reduce the efficiency and effectiveness of UX designers. Salvador et al. [24] conducted in 2014 an SLR with the focus on which usability methods are used in Agile methods and when. The most commonly used usability methods include: fast prototyping, individual inquiry, formal tests, and heuristic evaluations. These methods are used about 50% during the implementation phase and 40% during the design phase. Only 10% of the usability methods used are implemented automatically. In 2015, Silva et al. [25] performed a systematic mapping and analysed publications from significant Agile and HCI conferences. The objective was to answer the research questions on how Agile UCD is understood and which techniques are used in Agile UCD. Agile UCD is generally understood in the same way. It applies equally to the phases research, design, prototype, and evaluate. The most common technique is the implementation of usability test on lightweight prototypes. In 2015, Brhel et al. [26] published an SLR by stating the principles of user-centred Agile software development (UCASD). The authors aimed to assess the current state of the art regarding the integration of Agile software development and user-centred design. Using a coding system, the authors extracted five derived principles: separate product discovery and product creation, iterative and incremental design and development, parallel interwoven creation tracks, continuous stakeholder involvement, and artefact-mediated communication. In 2016, Magües et al. [27] conducted a systematic mapping study (SMS) in order to determine the current status of the integration of usability techniques in Agile processes. To that end, 31 studies were analysed and the usability techniques used were assigned to the development phase (requirements engineering, design, and evaluation). The most frequently used usability techniques for requirements engineering are ‘personas’; for design, ‘low-fi prototyping’; and for evaluation, ‘usability expert evaluations’. Likewise in 2016, Caballero et al. [28] conducted a literature review to investigate the extent to which Agile teams integrated UCD methods in their Agile software development process. One result was that the most frequently used Agile methods are Scrum and XP. The three main UCD methods, which represent 70% of the methods used, are prototypes, user stories, and usability testing. Also in 2016, the results from the SLR by Magües et al. [29] were further analysed in a mapping study by Magües et al. [27]. According to Brhel et al. [26], the selected studies were classified into the categories ‘process integration’ (48%), ‘practice integration’ (19%), ‘team integration’ (17%), and ‘technology’ (4%). The remaining papers could not be directly allocated. In conclusion, the authors concluded that there are no formalized suggestions for integrating usability techniques in Agile software development. In 2017, the systematic literature review by Schön et al. [30] focused on approaches and methods for involving stakeholders in the process of Agile requirements engineering. A total of 27 papers were analysed. The most important result: there is no common understanding of the user perspective in Agile software development. However, four methods (Human-Centred Design, Design Thinking, Contextual Inquiry, and participatory design) were identified that integrate knowledge of user needs in Agile software development. Garcia et al. [31] conducted a systematic mapping study in 2017. The purpose was to investigate artefacts used in communication between Agile methods and User-Centred Design. A total of 20 artefacts were identified and examined, such as prototype, user story, scenario, sketch, persona, and card like the design card or the task-case card. During the development iteration, about 56% of the artefacts are used. The rest are used during the discovery or planning phase. In a meta-study in 2017, Hoda et al. [32] examined SLRs that treat Agile software development. A restriction to HCI was not made. The aim was to identify which developments in Agile software development can be recognized by the SLRs investigated. One finding is that the significant integration of established domains such as usability, CMMI, and global software engineering can be recognized. Usability is the second-most common (18%) integrated domain. To evaluate how the Design Thinking approach is used in conjunction with Agile software development methods, Pereira & Russo [33] in 2018 used a systematic literature review. In total, 29 articles were Information and Software Technology 150 (2022) 106957 4 A. Hinderks et al. collected, categorized, and reviewed. The results show that most integrated models are applied throughout the software lifecycle. In most cases, the design thinking approach of the International Organization for Standardization (ISO) was integrated in Scrum as an Agile method. In 2018, Silva et al. [34] analysed the results obtained by Brhel’s SLR [26] concerning the state-of-the-art integration of Agile methods and the user experience design. The outcome from the respective publications was divided into three dimensions: process and practice, people and social, and technology and artefacts. As a result, the individual outcomes were arranged on a timeline so that the chronological sequence of the publications was visualized. The authors stated in their analysis that solutions are already being offered for the dimensions process and practice and people and social concerning integration. Finally, the authors concluded that technology and artefacts are still missing to integrate Agile methods and user experience design with Agile UXD. In a meta-study conducted in 2019, Curcio et al. [35] examined SLRs concerning Agile methods and usability. The fundamental question concerned how usability methods could be integrated in Agile software development. It was found that there are different levels of integration — process, practices, team, and technology integration. The biggest challenges are issues related to tests, time, work balance, modularization, feedback, prioritization, and documentation. Another important finding is that the type of integration that has evolved from two independent teams (parallel track) to one team during the search period. In total, we presented 16 SLRs for the integration of UX in Agile software development. The number of SLRs indicates that the integration has met the scientific interest. Besides, the SLRs show that everyone has a different focus on integration. Finally, the SLRs presented here show positive progress in the integration of UX in Agile software development. 2.2. Gap analysis In a further step, we investigated the research questions of SLRs. We assigned each research question to the category UX strategy, UX resources, or UX goal depending on the objective of the research questions. The categorization was done based only on the purposes of the research questions. The results of the research questions were not further investigated. A total of 47 research questions from the 16 SLRs were examined. Twenty-nine research questions were assigned to the category UX strategy, 7 UX resources, and 0 UX goal. The remaining 11 research questions could not be assigned to any of the categories. The results of the research show a focus on UX strategy. This is remarkable in that a UX strategy should always start with a UX goal as a prerequisite or objective. Only if a UX goal has been defined, a corresponding UX strategy can be selected. Every UX strategy indeed leads to a UX goal, but the definition of this goal is undefined and therefore, not manageable. From our point of view, all three categories have to be covered if managing UX is needed. The studies from the related literature review deal in different levels with the integration of UX or HCI in Agile software development. Garcia et al. [31] and Bruun [20] for example provide approaches for measuring the success of the applied approaches in terms of improving UX. However, we found that the next step is to focus on the strategic pursuit of user experience improvement. It was assumed that the integration of UX methods in Agile software development improves the user experience of the product, but this cannot be measured. However, to determine whether a goal has been achieved, the previous, expected, and post-implementation state should be measured [12]. This is the only way to determine whether the UX strategy and UX resources used have achieved the UX goal (Fig. 3). In other words: Has desired UX been achieved? We did not find a systematic literature review which investigates user experience management by focusing on Agile software development directly. The research questions tend to focus on UX strategies and perhaps UX resources, but not on UX goals. For this purpose, we conducted this SLR. Fig. 3. Desired Outcome vs. Results based on the Strategic Planning Cycle [12]. Fig. 4. Phases of an SLR. 3. Research methodology Appropriate guidelines have been followed for conducting a systematic review, particularly the guidelines for SLR in software engineering by [36]. According to these guidelines, our SLR consists of three main phases. Fig. 4 shows the most important stages of each phase. Owing to the high number of retrieved studies, we used the SLR Tool [37] and the software Citavi in order to manage information obtained in an efficient manner. We used the SLR tool when performing the SLR (managing the paper, developing the review protocol, documenting the search, and conducting quality assessment). In our literature database managed with Citavi, we imported the result of the SLR to use the management and citation functions. 3.1. Objectives and research questions In the beginning, we did informal research on UX management or related terms. We conducted the informal research with Science Direct, Springer Link, IEEEXplore, Scopus, and ACM with the keyword ‘user experience management’ and variations of it. The result was presented at the beginning of Section 2. However, during the research, we also found that the term ‘UX management’ is neither sufficiently defined nor explained in the literature. Further, we found through GAP analysis (Section 2.2) that there was a research gap in the goal, strategy, and resources concerning UX management. Besides, our informal research revealed that the number of papers found was too small and their content too widely scattered. However, we found approaches that allow UX management, as described in Section 2.2. This is the basis for our research questions. RQ1: Which approaches are suitable for UX management in an agile context? This question aims to identify approaches that can potentially be used for UX management. We did not expect that the approaches that had been found could be used explicitly for UX management. Otherwise, we could have already identified approaches in the literature search (Section 2). Our analysis was intended to list approaches that were generally successful or had a high acceptance concerning UX methods in Agile software development. Information and Software Technology 150 (2022) 106957 5 A. Hinderks et al. Table 1 Keywords used for search. Category Keywords Agile Agile, Kanban, Scrum, Lean, Extreme Programming, Design Thinking User Experience User Experience, UX, Usability, HCD, HCI, HMI, UCD RQ2: What conclusions can be deduced from the studies found? Concerning these research questions, we wanted to find out whether, in addition to approaches, other findings on UX management could also be derived from the studies. Not every study contains an approach that can be used directly for UX management. We instead assumed that studies would be found which described the integration of user experience methods and Agile software development. These aspects also need to be considered and analysed. RQ3: How can user experience in Agile software development be planned and controlled for a product-backlog item or a requirement before development? Management also implies a goal — what is to be achieved so that the necessary strategy and resources can be selected? The third research question aims to identify approaches that can be used to estimate UX product-backlog items before development. The result of the estimation is to figure out the potential UX that can be reach if the product-backlog item will be developed. In this way, the estimation of UX can be used to determine where there is potential to achieve a potential UX goal. Further, the question remains as to what extent the estimated UX can be expressed in the form of product-backlog items or requirements. RQ4: What retrospective proposals exist to improve the efficiency and effectiveness of the user experience process in terms of Agile software development? In terms of these research questions, our goal was to identify proposals for improving the UX process. We had to consider the fact that Agile software development was usually iterative. This means that after each iteration, there is the possibility of improving the UX process. 3.2. Search strategy and data sources Based on the research questions and research objectives, we developed a search strategy. This strategy contains the search string, the search space, and the process to select the relevant papers. The first step is to create a set of keywords. Since UX management has not been sufficiently covered in the literature (Section 2.2), the set of keywords consists of Agile and user experience as far as related terms are included. In practice, it has been shown that Agile is often not directly addressed, but rather Kanban,Scrum etc. Agile frameworks, like Scrum, or agile methods, like Kanban, are often used as a keyword in combination with Agile Methods. For this reason, we have included agile frameworks and methods in the search string. We also included the term design thinking because our experience has shown that useful publications have also been found with this term. In the second step, we extended the keywords by alternative spellings and synonyms. These were extracted from the previously analysed literature (Section 2). Finally, we consolidated and optimized the list of keywords. The final list of keywords is shown in Table 1. The set of keywords was then transferred to a search string in the next step. This is as follows: (agile OR kanban OR scrum OR lean OR ‘‘extreme programming’’ OR ‘‘design thinking’’) AND (‘‘user experience’’ OR ux OR usability OR hcd OR hci OR hmi OR ucd) Table 2 Search space with specification of search strategy (TAK = Title, Abstract, and Keywords) and number of paper. Library Search strategy Number IEEEXplore Full Text 863 Science Direct TAK 61 SCOPUS TAK 1,308 SpringerLink Full Text 3,874 ACM TAK 26 This search string was adapted to the syntax of the respective search spaces as these had partially differed. The actual logic, however, had not been changed. The search space included digital libraries, journals, and conference proceedings. A complete list of the search space is shown in Table 2. The search was conducted at all search spaces in January 2020. Without any restriction – i.e. plain full-text search of the search engine – N𝑃0=44,637 (Fig. 5) papers were found. It should be noted that IEEEXplore and Springer Link had problems restricting the search to title, abstract, and keywords. Both did not offer the possibility to search for title, abstract, and keywords together. The conversation with the support of the respective providers has also led to no result. These problems were partially resolved by the owner of the search engines, but they led us to a slightly different strategy. Wherever possible, we downloaded the paper and put the abstract and keywords into plain text. From 4496 paper, we were able to extract the abstract and keywords from 2733 paper. Finally, we conducted an own search limited to title, abstract, and keywords using the SLR Tool [37] on all N𝑃1=4496 paper. The result was that we had to check N𝑃1=4496 (Fig. 5) paper initially. By conducting the internal search function of the SLR Tool [37], we could reduce the result by searching only on title, abstract, and keywords so that the amount of paper to be examined was N𝑃2=1253 (Fig. 5). This data set was further examined by us, as described in Section 3.3. 3.3. Study selection In the previous section, it was described how the number of papers was limited by the search criteria (Step N𝑃2). In a further step, we reduced the number of papers from N𝑃2=1253 to N𝑃3=196 by scanning the title. We only included those papers that were interesting and valuable for our SLR in terms of their title. The title should be recognized that the paper is mainly about ‘agile’ and ‘user experience’. If the title indicated that the paper only applied agile and UX methods, the paper was excluded. In the following step, we reduced the number of papers to N𝑃4=110 by reading the abstract. We applied the same criteria we used one step ago. All the decisions are traceably logged by the SLR tool. In each step of the reduction, a set of selection criteria were applied. These which are divided into inclusion and exclusion criteria. The inclusion criteria were: papers written in English; papers under peer-reviewed papers; and papers presenting approaches to integrate user experience methods (or similar) in Agile development processes. Exclusion criteria were: no full books; papers whose full text were not available; papers only presenting lessons learned, ideas, guidelines or recommendations; papers introducing a panel talk or a workshop at a conference; papers with results that had already been published; papers that were not focused on Agile development; papers introducing tools whose underlying methodology was not comprehensibly described (black box). After the study selection, we performed a snowballing process according to Wohlin [38]. We applied forward snowballing (search in papers that cite the paper) and backward snowballing (search in the reference list of the paper). Snowballing has the advantage that we Information and Software Technology 150 (2022) 106957 6 A. Hinderks et al. Fig. 5. Search Process comprising Phases and Inclusive Forward and Backward Snowballing. Table 3 Quality checklist for empirical studies. Item Assessment criteria Score Description QA1 Was more than one study conducted? −1 Only one study was conducted 0Two studies were conducted 1More than two studies were conducted QA2 Was the target group selected randomly? −1 No, randomized group of participants 1Yes, randomized group of participants QA3 Is the data analysis process appropriate? −1 No analysis has been taken 0One statistical analysis has been taken 1The dataset is well analysed QA4 Is the result of the statistical analysis appropriate? −1 No or poor results 0The results are okay 1The results are good enough can identify additional papers important to the SLR that were not identified via the SLR method itself in addition to the systematic search. The additionally found papers (N =29) were inserted in step N𝑃4 (Fig. 5). The numbers shown in Fig. 5 include the papers added by the snowballing. In total, seven additional papers for data extraction were added at the end. 3.4. Quality assessment The papers selected in the previous section (N𝑃4=110) were evaluated with a quality assessment. We developed a checklist (Table 3) based on the recommendations of Kitchenham and Charters [36] (Table 3) to evaluate case studies, literature research, interviews, and surveys. Methods and models were excluded because they are qualitative studies. To better classify the papers, we classified the papers according to case studies, literature research, interviews, surveys, methods and models. This classification is based on our own created system. We manually reviewed these methods and models by reading and evaluating the paper. The evaluation was based on the basic orientation of the study and whether it is suitable for our SLR. The overall aim was to identify studies of low quality and then exclude them from our study. In the end, every paper was rated with a sum of the individual result. We decided to include those articles with a score greater than or equal to 1. The SLRs determined were checked to see whether the SLR was carried out in a traceable manner. Also, we checked whether the SLR was performed according to a standard published in the literature. The SLRs were then reduced to N𝑃6.1=16. Models and methods were generally reviewed for evaluation or validation. The aim was to determine whether the method or model was generally successfully applied in a study. After validation, we reduced the methods to N𝑃6.2=6 and models to N𝑃6.3=10. After the user of the quality assessment, we reduced the case studies to N𝑃6.4=3, literature research N𝑃6.5=4, interviews N𝑃6.6=8, and surveys to N𝑃6.7=2. In the end, we included 49 papers out of 110 papers in our SLR study. 3.5. Data extraction and analysis According to Kitchenham’s and Charters’s guidelines [36], a form for data extraction was set up. We used the SLR Tool [37] in order to gather summaries, etc. The SLR Tool also supported the data extraction with regard to defined attributes from the protocol: •Basic information: title, authors, publication date, DOI and URL •Publication data: journal, conference, date (of conference), publisher, volume, issue, pages, keywords and abstract •List of included references (if available) Information and Software Technology 150 (2022) 106957 7 A. Hinderks et al. Table 4 Distribution according to research methods and year. Research method 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 Total SLR 1 1 32332116 32% Methods 1 22 1 6 12% Models 1 2 1 1 1 21 1 10 21% Case studies 1 1 1 3 6% Literature research 1 214 8% Interviews 1 1 12 1 1 1 8 17% Surveys 2 2 4% Total 2 3 0 3 2 4 1 10 5 8 6 2 3 49 100% In addition to the automatically extracted data, we have determined the following attributes manually: •Paper category: e.g. SLRs, models, methods, case studies, literature research, interviews, and surveys •Used UX methods: e.g. personas, prototypes, usability evaluations •Agile methodology: e.g. Scrum, XP, Agile in general •UX process integration: Parallel track or one track •Development phase of usage: before, during, or after •Short summary •Results and contributions •Personal assessment In some cases, it was not possible to fill every attribute. In this case, we filled the attribute with ‘not specified’. In the last step, we checked the content of all studies and assigned them to particular research questions. The aim was to have a list of studies per research question so that we can use to answer them. The research questions are answered in the next section. 4. Results In our work, we have selected 49 relevant studies. In this section, we will present the studies individually or in total, if necessary, to answer the research questions. The first part of this chapter gives an overview of the selected studies. In the second part, the individual research questions will be answered based on the studies. 4.1. Summary of the studies Our search was limited to ‘Agile’ and ‘UX’ (and similar terms). An explicit restriction to UX management or similar was not made. This is because already in our first literature review it was found that the search result would not be sufficient to answer the research questions (Section 2). For this reason, the studies included in our SLR cover a broad spectrum. Incidentally, all the studies have been examined for their applicability in the field of UX management. In journals, 15 (31%) of the included studies and 34 (69%) in conference proceedings were published. In Table 4, all 49 studies analysed in this SLR are grouped by research method and year. 15 (31%) of the studies were published between 2007 and 2013 and 34 (69%) between 2014 and 2019. It should also be noted that our research was conducted in January 2020, so there may still be late publications that are not included in this SLR. All the 49 included studies were assigned different underlying research methods (Table 4). Of the studies included, 16 (32%) are structured literature reviews (SLR). These SLRs have not been used to answer the research questions as they themselves answer research questions that differ from ours. However, they have been presented in section Related Work (Section 2.1) and form the basis of the gap analysis (Section 2.2). Further, six (12%) of the studies included in this study can be assigned to the category Methods [39–44]. Methods are to be understood rather generally in this category. Usually, a procedure is described, how a problem was solved. A total of eight different methods were presented in these studies. Of the eight methods, four known methods were newly combined (Tool for A/B test [44], Checklist for a possible maturity model [43], Personas [40], Nielsen’s heuristics [42]), three new methods were presented (Usability Goals Achievement Metric (UGAM) [39], Index of Integration (IoI) [39], Web business process refactoring (WBPR) [41]), and one method was supplemented (UserX Story [42]). The 10 (21%) Models [45–54] included can be divided into four frameworks, three conceptual models, two processes, and one lifecycle. For the sake of simplicity, they have been assigned to the research method models, since in some cases even the author of the study has not specified the research method. All three (6%) Case Studies [55–57], four (8%) Literature Research [19,58–60], eight (17%) Interviews [55,61–67], and two (4%) Surveys [68,69] have been conducted in economic enterprises in such a way that the results and conclusions from each study are very practical. The Case Studies can be summarized as ‘classic case studies’. As Literature Research, we have categorized studies that answer research questions based on the literature. In this case, the difference between an SLR and Literature Research depends on the used method to conduct study. The Interviews were partly conducted as semi-structured interviews. Finally, the included Surveys were conducted in the classical way with a self-developed questionnaire. Of the total of 49 studies, 16 SLRs are included, which we discussed in Section 2. Of the 33 remaining studies, 38 individual studies are included and analysed in this section. Two are Multi-Case Studies with four and two Case Studies, respectively, and two Methods are presented in one study. For this reason, 38 individual studies are selected and presented in our paper. The corresponding studies with several individual studies are numbered accordingly in round brackets. These are described in the next five sections. We further analyse the approaches in the 38 individual studies. We use the term approach as a generic term for a method, a model, a case study, literature research, an interview, or a survey. 4.1.1. User experience or usability The 38 individual studies were examined as to whether each of them used the concept of usability or user experience. In some cases, it was not possible to assign the concept, as both concepts were used. In this case, the concept that was the most present in the individual study had been selected. Of the 38 individual studies, 16 (42%) use the concept usability and 22 (58%) the concept of user experience. 4.1.2. Used Agile method We also examined the 38 individual studies according to the Agile method used. A total of 21 (55%) of the studies have not been assigned to a specific method. This is because the authors of the paper do not address explicit Agile methods. This means that it could be used for any Agile methods. The rest is divided between Scrum 11 (29%) and Extreme Programming 6 (16%). Information and Software Technology 150 (2022) 106957 8 A. Hinderks et al. 4.1.3. Type of integration of UXin Agile methods We also examined the 38 individual study to determine whether UX methods were used within the development team or outside as an additional parallel track/iteration. Of the individual studies examined, 11 (29%) integrated UX methods in the development team. The UX methods are usually used by UX professionals as well as developers. The eight (21%) individual studies, on the other hand, which use a UX team with its organization, in addition to the development team, are different. With an additional UX team, this team does not necessarily have to work exclusively with the development team, but can also work for several teams. The remaining 19 (50%) individual studies are not further specified for integration. One reason for this is that the individual study condenses results in such a way that the type of integration can no longer be deduced from them. Another reason may be that individual UX professionals only work with the development team and therefore do not constitute a team in their own right. 4.1.4. Proposed approaches In the next step, we extracted the proposed approaches from the 38 studies. In total, we were able to extract 18 unique primary (Table 5) approaches from 24 individual studies out of the 38 studies. We analysed these in a further step. The results are presented in Sections 4.1.5 and 4.2.1. 4.1.5. Time period of use In a further step, all 18 approaches were analysed with their temporal applicability in development. The aim was to examine each approach about the phase of development in which this approach is to be or was used. The breakdown was made according to before, during, or after development. If an approach can be used in several phases of development, it was also assigned to those phases. In total, 21 (88%) approaches can be used before development, while 15 (63%) approaches can be used during development and 13 (54%) after development. In the next paragraphs, we answer the individual research questions from Section 1using the 38 individual studies and the 18 primary approaches extracted from the 38 individual studies. All studies, except for the SLRs determined, serve as a basis. 4.2. (RQ1) which approaches are suitable for UX management in an agile context? The first research questions seek to answer which of the 18 primary approaches extracted from the studies are suitable for the management of UX. One criterion for determining when an approach is suitable is when the presented approach has been successfully applied within Agile software development. Thus, the approach presented in the study is suitable for UX management for the first time. The quality assessment (Section 3.4) already ensures that the approach has been sufficiently evaluated or validated. Table 5 lists all the primary approaches that were mainly presented or used in the individual studies. This means that in addition to the primary approaches, additional UX methods have been used. The main focus for answering RQ1 is on the primary approaches. Additionally used UX methods can support the primary approach, but they would not bring the desired success if used in isolation. The additional UX methods (Table 6) that were used in combination with the primary approaches are described in Section 4.2.2. 4.2.1. Primary approaches presented in studies In the following sections, we present the highlights found in the included studies. In Table 5, we list all 18 primary approaches presented in the studies (Section 4.1.4). These approaches are potentially suitable for managing UX processes because they integrate UX methods into agile software development. The most frequently identified approach is ‘Upfront UCD Design’. A second UCD team will be added to the actual development team. This team works in a parallel track (Section 4.1.3), always one iteration ahead of the development iteration. In this parallel track, prototypes [55,61,70] are usually created in various forms. These prototypes are then handed over to the development team and developed in the next iteration of iterations. Silva et al. chose a similar approach [50] — their framework defines the tasks of the individual teams within the parallel track concerning the user experience. The second-most frequently used method or approach is ‘Communication/Collaboration’ — a simple but successful approach. The approach is implemented in various ways. Ferreira et al. [55], for example, chose four different approaches in their study. The UI designers and developers worked together and constantly exchanged in this study. The UI designer usually developed a UI prototype based on a specification or a user story. This was then discussed together with the developers and then implemented by the development team. The result was that the developers had a much better understanding of the goal of the specification or user story. Ferreira et al. [70] chose a similar approach in another study. Øvad et al. [56] and Øvad and Larsen [71] followed a different approach. Both approaches aim to teach the developers selected usability and UX methods so that they can use them independently in their development work. The approaches include ‘A/B Testing’ [56,71] as well as ‘Focus Workshops’ [71] and ‘Contextual Interviews’ [71]. The ‘Cruiser Lifecycle’ developed by Memmel et al. [45] aimed at integrating the methods of human–computer interaction (HCI) in Agile software development. The lifecycle mainly consists of three phases. The result of the first phase (Initial Requirements Up-Front) is a collection of artefacts like use case diagrams, scenarios, and prototypes. These are then processed further in the second phase (Initial Conceptual Phase). In the second phase, the release plan, system metaphor, UI design, and UI specification are developed. In the third phase (Construction & Test Phase), the product is developed and then its usability is evaluated. If new requirements are to be implemented, work can begin again from Phase 1. Xiong and Wang [48] and Humayoun et al. developed similar phase models [49]. All of them are based on the fact that there is a certain number of phases in which certain HCI or UX tasks are to be completed. Singh [47] extended Scrum to ‘U-Scrum’ by adding another role called ‘Usability Product Owner’. This is to ensure that the usability represented by the newly created role is already considered when requirements are created. Both the Product Owner (PO) and the Usability Product Owner (UPO) work on the same level, but with different focuses. The PO corresponds to the defined role in the Scrum Method [5]. In contrast, the UPO has to defining a user experience vision. This should be considered in the requirements analysis. Finally, the UPO is responsible for the usability design and for creating the requirements together with the PO. Wolkerstorfer et al. [46] integrated UCD methods within Extreme Programming (XP) to an ‘Agile Usability Process’. Before the first iteration of development, conducting user studies, personas, and usability tests are necessary. During the development, usability expert evaluations of the product increment are performed simultaneously with the unit tests established in XP. These provide direct feedback to the development as well as work for the next iteration. Joshi at al. [39] created the ‘Usability Goals Achievement Metric (UGAM)’. We will present this approach in Section 4.4 in more detail. The second metric presented in this paper is ‘Index of Integration (IoI)’ [39]. This metric represents, on a scale from 0 to 100, the level of integration of HCI activities in software development activities. With both UGAM and IoI, the development team can know how good their developed product regarding UX happens to be. ‘Web business process refactoring’ (WBPR), developed by Distante et al. [41], is a framework to capture the usability improvements of business process web applications. A WBPR includes the following Information and Software Technology 150 (2022) 106957 9 A. Hinderks et al. Table 5 18 Primary approaches presented in studies. No Primary approach Studies Total 1 Upfront UCD design [55](3), [55](4), [61], [70](1) 4 2 Communication/Collaboration [55](1), [55](2), [70](2) 3 3 Teaching UX Methods [56], [71] 2 4 Cruiser Lifecycle [45] 1 5 U-Scrum [47] 1 6 Agile Usability Process [46] 1 7 Inter-Combined Model [48] 1 8 Usability Goals Achievement Metric (UGAM) [39] 1 9 Index of Integration (IoT) [39] 1 10 Three-Fold Integration Framework [49] 1 11 UXD & AD Framework [50] 1 12 Agile Usability Model [52] 1 13 Agile UX Model [51] 1 14 Web Business Process Refactoring (WBPR) [41] 1 15 UserX Story [42] 1 16 Checklist for User-Centeredness of Agile Processes [43] 1 17 CSWR Framework [44] 1 18 SIBAP [53] 1 Total 24 Table 6 Methods used in combination with primary approaches. No Method Studies Total 1 Prototyping (Low/High) [45], [61], [48], [63], [55](2), [55](3), [55](4) 7 2 Personas [61], [46], [48], [40], [42], [54] 6 3 Task/Usage Scenarios [45], [48], [54] 3 4 Acceptance Test [55](2), [55](3) 2 5 Focus Groups [45], [71] 2 6 Expert Reviews [45], [46] 2 7 UX Questionnaires [45], [48] 2 8 Contextual inquiry [61], [71] 2 9 Usability Testing [61], [63] 2 10 User Evaluation [46], [54] 2 11 Interviews [48], [54] 2 12 A/B Testing [56], [71] 2 13 Card Sorting [45] 1 14 Brainstorming [45] 1 15 Usability Inspection [45] 1 16 FlexREQ [52] 1 Total 38 criteria: intent (usability qualities such as effectiveness, efficiency, and satisfaction), bad smell (an indicator of the lack of usability), motivation (description of the problem), and examples (describes the application). For each of these criteria, the authors provide instructions for creating WBPRs within the framework. Navarro et al. [53] developed the ‘Script-Based Aspect-Oriented GUI Prototyping (SIBAP)’ framework. The objective of the development was to create prototypes using a scripting language that can be used by both designers and developers. The prototype is the common artefact of designers and developers. The designers use the framework to create a prototype, which is then implemented by the developers. Thus, there is no media break between the prototyping of the designers and the development. In summary, all the approaches presented in this section aim to promote a collaboration between UX professionals and developers. Some approaches directly promote communication, while others try to promote the exchange via a process or framework. 4.2.2. Additional UX methods included in primary approaches UX methods are used in various ways. Usually, UX methods are used in addition to a process or a framework. Table 6 lists all the UX methods that were additionally used to the 24 primary approaches in the studies. The two most frequently used methods are Prototyping and Personas. Prototyping and personas can be used as artefacts for the communication between UI designers and developers. The UI designers either develop a prototype together with the developers or work on it before the actual development. Personas, on the other hand, are usually used permanently. Various methods are used to determine the requirements — these are task/usage scenarios, focus groups, contextual inquiry, user evaluation, interviews, A/B testing, card sorting, brainstorming, and FlexREQ. The following methods are used to measure and evaluate the user experience: acceptance test, expert reviews, UX questionnaires, usability testing, and usability inspection. The UX methods listed in Table 6 were identified mainly by Jia et al. [68] as UX methods that serve in connection with the integration of UX in Agile software development. According to the list of Jia et al. [68] these are workshops, prototyping, interviews, scenarios, personas, field studies, usability goals, usability evaluation, questionnaires, and heuristic evaluation. What is interesting about the study, however, is the difference between the frequency of use and the evaluation of each individual method. The Personas method is rated quite positively, but 38% of the participants consulted it only once a year. In contrast, over 60% of the users use Prototypes at least two or three times a month. 4.3. (RQ2) what conclusions can be deducted from the studies found? This question aims to identify findings that are useful in the context of UX management. To answer this research question, we analysed all 38 studies. Both frameworks – user-centred design (UCD) and Agile methods – were developed independent of each other. Some studies have investigated to what extent the two frameworks can be integrated. The fundamental difference between the two frameworks is the focus. Agile methods focus on code production and the work of the developers,