Research Protocol

II Research Protocol For the paper entitled: “Between Policy and Practice: GenAI Adoption in Agile Software Development Teams” Submitted to the International Conference on Agile Software Development (XP) 2026 Authors: Michael Neumann, Lasse Bischof, Nic Elias Hinz, Abdullah Altun, Luca Stockmann, Dennis Schrader, Ana Carolina Ahaus, Erim Can Demirci, Benjamin Gabel, Maria Rauschenberger, Philipp Diebold, Henning Fritzemeier, Adam Przybylek III Table of Contents List of figures…………………………………………………………………………………….II List of tables…………………………………………………………………………………….III 1 Insight Inc. Case: GQM Visualization ................................................................................... 1 1.1 GQM Table – G1: Understanding the Framework Conditions for GenAI Usage ........................ 1 1.2 GQM Table – G2: Identifying Use Cases for GenAI .................................................................... 2 1.3 GQM Table – G3: Identifying Perceived Benefits and Barriers .................................................. 3 2 Insight Inc. Case: Interviews ................................................................................................ 4 2.1 Interview Guide .......................................................................................................................... 4 2.2 Execution of the interviews ....................................................................................................... 5 2.3 Overview of the interview participants ..................................................................................... 6 3 Insight Inc. Case: Clustering ................................................................................................. 8 3.1 Initial Clustering ......................................................................................................................... 8 3.2 Observations from the Clusters ............................................................................................... 13 4 Grey Matter Technologies ................................................................................................. 20 List of figures Figure 1: GQM-Approach ............................................................................................... 1 Figure 2: GQM-Approach: G1 ........................................................................................ 1 Figure 3: GQM-Approach: G2 ........................................................................................ 2 Figure 4: GQM-Approach: G3 ........................................................................................ 3 Figure 5: Organisation of the interviews ......................................................................... 5 Figure 6: Clustering: Usage & Framework conditions .................................................... 9 Figure 7: Clustering: Practical Applications and Effects ............................................... 10 Figure 8: Clustering: Perspectives and Future ............................................................. 11 Figure 9: Clustering: Additional Aspects ....................................................................... 12 Figure 10: Observations from the Clusters ................................................................... 13 IV Figure 11: Observations from the Clusters: Tool Landscape ....................................... 14 Figure 12: Observations from the Clusters: Governance ............................................. 15 Figure 13: Observations from the Clusters: Usage Patterns ........................................ 16 Figure 14: Observations from the Clusters: Impact Factors ......................................... 17 Figure 15: Observations from the Clusters: Future Use Cases .................................... 18 Figure 16: Observations from the Clusters: Support Requirements ............................. 19 Figure 17: Goal Question Metric for Interview guideline Grey Matter Technology ....... 20 Figure 18: Thematic Analysis RQ 1 Grey Matter Technology ...................................... 21 Figure 19: Thematic Analysis RQ 2 Grey Matter Technology ...................................... 22 Figure 20: Thematic Analysis RQ 3 Benefits Grey Matter Technology ......................... 23 Figure 21: Thematic Analysis RQ 3 Barriers Grey Matter Technology ......................... 24 Figure 22: Thematic Analysis Results Overview Grey Matter Technology ................... 25 Figure 23: Thematic Analysis Results RQ 1 Dinoco ..................................................... 26 Figure 24: Thematic Analysis Results RQ 2 Dinoco ..................................................... 27 Figure 25: Thematic Analysis Results RQ 3 Dinoco ..................................................... 28 List of tables Table 2: GQM-Approach: G1 ......................................................................................... 2 Table 3: GQM-Approach: G2 ......................................................................................... 2 Table 4: GQM-Approach: G3 ......................................................................................... 3 Table 5: Participant Profiles ........................................................................................... 6 Table 6: Cluster 1 – Usage patterns & Framework conditions ....................................... 9 Table 7: Cluster 2 – Practical application & effects ...................................................... 10 Table 8: Cluster 3 – Perspectives & Future .................................................................. 11 Page 1 1 Insight Inc. Case: GQM Visualization To support transparency and traceability in the research design, the full GQM structure was visualized in the form of a diagram (see figure 1). This diagram shows the logical flow from the overall research goals to the guiding questions and finally to the individual metrics. Each of the three main goals (G1–G3) is shown as a separate branch. These are further divided into the relevant sub-questions and associated metrics. The visualization was created to clearly illustrate how the interview topics were derived and how they relate back to the core research questions. The structure was designed using the online tool draw.io and was based on the initial research framework developed during the planning phase. It also served as a reference when preparing and structuring the interview guide. Figure 1: GQM-Approach 1.1 GQM Table – G1: Understanding the Framework Conditions for GenAI Usage Figure 2: GQM-Approach: G1 The following table on the next page shows how the first research goal (G1) was broken down into guiding questions and associated metrics. This structure served as a foundation for formulating the corresponding interview questions. It ensured that the interview responses could later be clearly assigned to the research objective and evaluated in a structured way. Page 2 Table 1: GQM-Approach: G1 Question (Q) Metric (M) Q1.1: Which GenAI tools do you use? M1.1.1: List of tools Q1.2: Which of them are licensed? M1.2.1: License status (free / paid) Q1.3: Do you use GenAI tools more for direct problem-solving or for idea support? M1.3.1: Share of users who primarily use GenAI for direct problem-solving vs. share who use it for idea support M1.3.2: Type of application (e.g., documentation, communication, facilitation) Q1.4: Are there any guidelines or rules for the use of GenAI tools? M1.4.1: Existence of guidelines (yes/no) M1.4.2: Level of awareness of the guidelines (e.g., % of coaches who are aware of them) 1.2 GQM Table – G2: Identifying Use Cases for GenAI Figure 3: GQM-Approach: G2 The following table outlines the second goal (G2) of the GQM model. It breaks down how the goal of identifying relevant and potential GenAI use cases for Agile Coaches was operationalized into questions and metrics. This structure helped guide the main part of the interviews and enabled a clear categorization during the analysis. Table 2: GQM-Approach: G2 Question (Q) Metric (M) Q2.1: What possible use cases do Agile Coaches see for GenAI in their work context in the future? M2.1.1: Number of use cases mentioned per person M2.1.2: Usefulness rating per use case (e.g., scale 1–5) M2.2.1: Mention of new task areas Page 3 Q2.2: Are there any new tasks or areas of responsibility emerging from the use of GenAI at Bagilstein? M2.2.2: Frequency of mentions Q2.3: Are there differences in GenAI usage depending on experience level or context? M2.3.1: Comparison of usage between junior and senior coaches M2.3.2: Differences based on team size/industry/project type 1.3 GQM Table – G3: Identifying Perceived Benefits and Barriers Figure 4: GQM-Approach: G3 The following table shows the breakdown of the third goal (G3) of the GQM model. This part focuses on understanding the advantages, challenges, support needs, and limits of using GenAI in agile roles. The questions and metrics were directly integrated into the interview guide to gather relevant and comparable insights. Table 3: GQM-Approach: G3 Question (Q) Metric (M) Q3.1: What are the perceived benefits of using GenAI? M3.1.1: Frequency of mentioned benefits (e.g., efficiency, creativity, time saving) M3.1.2: Perceived usefulness (scale) Q3.2: What barriers make the use of GenAI more difficult? M3.2.1: Frequency of mentioned barriers (e.g., data protection, distrust, lack of training) M3.2.2: Perceived risk (scale) Q3.3: What kind of support do Agile Coaches wish for in the future? M3.3.1: Need for training or further education (scale or free text) M3.3.2: Desire for clearer guidelines or frameworks (yes/no, free text) Q3.4: Are there tasks where GenAI cannot be used – and if so, why? M3.4.1: Mentioned tasks M3.4.2: Reasons for not using GenAI in these tasks Page 4 2 Insight Inc. Case: Interviews 2.1 Interview Guide The interview guide was developed based on the results of the GQM-approach. It was separated into three parts. In the first section, to collect general information about the background of the person interviewed, the questions were related to general information, the professional background and the current position at Bagilstein. These questions are not based on the GQM-Approach Introductory questions (part one): • Could you briefly introduce yourself? (e.g., name, age, academic background) • What is your position at Bagilstein? • How long have you been working in this role (e.g., as a Scrum Master)? • How long have you been with Bagilstein? • What previous positions or roles have you held? • What is your role in relation to clients? • Have you had any prior experience with GenAI tools? The second part, the main part, the questions, which were developed based on the GQMapproach, were asked. Main questions (part two): • Which GenAI tools do you currently use? • Which of these tools are licensed? • Do you use GenAI tools more as a direct solution (e.g., letting the tool complete a task) or more as a support to reflect on or improve your own ideas? • Are there any guidelines or policies in place regarding the use of GenAI tools? • What potential do you see for Agile Coaches to use GenAI in their work going forward? • Are new tasks or areas of responsibility emerging due to the use of GenAI at Bagilstein? • Do you observe any differences in how GenAI is used depending on a person’s level of experience or the specific context? • What are the main benefits of using GenAI tools? • What barriers or challenges make the use of GenAI more difficult? Page 5 Figure 5: Organisation of the interviews • What kind of support would you like to see in the future regarding GenAI use? • Are there any tasks for which GenAI cannot be used? If so, why? The third part contains typical questions to conclude an interview. These questions are as well as the introductory questions not based on the GQM-approach. Final questions (part three): • Is there any exchange among colleagues regarding GenAI? • Do you have any questions for us? • Were there any questions you expected us to ask but we didn’t? 2.2 Execution of the interviews The interviews were conducted between May 12 and May 19, 2025. The participants were invited via email. The initial contact with the participants was established by the CEO of Bagilstein. To organize the interview appointments, we used a shared Google Spreadsheet, which is shown in the next figure. We provided available time slots, and each interview was conducted by two of us together. For data protection reasons, the names of the interviewees were anonymized and replaced with codes ranging from P1 to P7. We used Microsoft Teams to record the interviews and those were automatically transcribed using Microsoft Teams’ built-in transcription tool. The first interview we conducted served as a pilot interview to test the interview guide. Since the conversation went well and the guide proved effective, we decided to include the results of this first interview in the overall analysis. Page 6 2.3 Overview of the interview participants The following table shows a brief overview of the interview participants' backgrounds. We gathered this information through the initial questions in our interviews to better understand the context behind their responses, such as their roles, experience levels, and areas of focus (see table 4). Table 4: Participant Profiles Interviewee Role at Bagilstein Professional Background Experience with GenAI Age/Education P1 Agile Coach and Scrum Master for 5 years Program manager at a bank (IAM, project lead, team lead, product owner) ChatGPT 57 years old P2 Scrum Master and Kanban expert for 6 years Long-time self-employed, leadership experience ChatGPT, Gemini, AI image editing tools 49 years old, chemistry degree + nano degree in digitization P3 Agile Coach and Scrum Master since early 2025 13 years of experience: developer, Agile Coach, Product Owner ChatGPT, technical understanding 32 years old, IT specialist + computer science studies P4 Longer-term collaboration, external roles as Agile Coach, Traditional project management, agile roles: PO, Scrum No GenAI use No info on age or degree Page 13 3.2 Observations from the Clusters Figure 10: Observations from the Clusters After all interviews had been transcribed, the responses were sorted into topic-based clusters using a digital whiteboard in Miro. The clustering was based on the structure of the interview guide and the underlying GQM model. The visual representation in Miro was created collaboratively and served as a working basis for the evaluation phase. Each cluster represents a thematic focus, such as Tool Landscape, Governance, Usage Patterns, Impact Factors, Future Use Cases, and Support Requirements. For each topic, digital notes were used to capture individual statements from the interviews. These notes were marked with participant codes (P1–P7) to ensure anonymization and traceability. The layout of the board follows a left-to-right logic: starting with framework-related topics and moving toward future-oriented perspectives. Within individual clusters, similar statements were grouped together. In some cases, such as the cluster Impact Factors, additional subdivisions were introduced to improve clarity (e.g., separating benefits, barriers, and experience/context). The purpose of the clustering was to prepare the material for further analysis – not to interpret it yet. The Miro board helped the team to structure large amounts of qualitative data, identify patterns, and keep the research process transparent and collaborative. The full clustering can be seen in figures 11-16. Page 14 Figure 11: Observations from the Clusters: Tool Landscape Page 15 Figure 12: Observations from the Clusters: Governance Page 16 Figure 13: Observations from the Clusters: Usage Patterns Page 17 Figure 14: Observations from the Clusters: Impact Factors Page 18 Figure 15: Observations from the Clusters: Future Use Cases Page 19 Figure 16: Observations from the Clusters: Support Requirements Page 20 4 Grey Matter Technologies Figure 17: Goal Question Metric for Interview guideline Grey Matter Technology Page 21 Figure 18: Thematic Analysis RQ 1 Grey Matter Technology Page 22 Figure 19: Thematic Analysis RQ 2 Grey Matter Technology