The Impact of LLM-Assistants on Software Developer Productivity: A Systematic Literature Review - Supplementary material.

Table 1: Overall goal of primary studies. Primary Study - Goal Strategy Procedure [PS20] - Evaluate Code Compose, an internal AIassisted coding system at Meta Field Study Survey (Open Ended feedback) [PS25] - Case study of developing a business app through SDLC phases using systematic prompting Experimental Simulation Case study, Concept Implementation (Proof of Concept) [PS27] - Evaluate an LLM-based tool for generating Ansible YAML code Field Study Survey [PS9] - Compare novice coding with and without Copilot in terms of performance and experience Laboratory Experiment Controlled experiment, Survey [PS4] - Compare AI-pair and non-AI-pair projects in a mobile game studio Field Study Case study, Survey, Interviews [PS6] - Investigates perceived productivity of GenAI integration in software development Field Study Case Study, Survey [PS21] - Evaluates an IDE-integrated, prompt-less conversational LLM Laboratory Experiment Controlled experiment: Within-subject, Survey [PS3] - Investigate the impact of GenAI on labor productivity across domains Sample Study Survey [PS2] - Modeling productivity and output quality of 70 large global companies using AI tools Sample Study Survey, Interview, Case Study [PS28] - Measuring GitHub Copilot’s impact on developer productivity for an automotive organization Field Study Case Study, Survey [PS10] - Evaluate the benefits of using IDE autocompletion Laboratory Experiment Controlled experiment: Between-subjects, Survey [PS35] - Evaluate developers’ performance with neural code translation Laboratory Experiment Case Study, Controlled experiment: Within-subjects, Survey [PS36] - Investigate if code generation in PyCharm improves productivity Laboratory Experiment Controlled experiment, Survey, Concept Implementation [PS33] - Whether working with ChatGPT was helpful in coding and development tasks Laboratory Experiment Controlled experiment: Between-groups, Survey [PS34] - Explore how LLM-assistants impact productivity using the SPACE framework Laboratory Experiment Controlled experiment: Within-subjects, Survey [PS23] - Understand how access to Google Bard affects productivity and trust in coding Laboratory Experiment Controlled experiment: Within-subjects, Survey [PS13] - Gain insights into programmers’ views on ChatGPT and other AI tools Experimental Simulation Survey [PS16] - Compare performance of Stack Overflow vs. ChatGPT usage Laboratory Experiment Controlled experiment: Between-groups, Survey [PS32] - Compare highlighting conditions for AI code completion Laboratory Experiment Controlled experiment: Within-subjects, Survey, Interview [PS7] - Explore how professional developers perceive AI-based tools in their work Field Study Case Studies, Surveys [PS14] - Comparing human-human vs. human-agent interaction using wizard-of-Oz Laboratory Experiment User Study, Interview, Survey [PS5] - Compare novices’ performance using ChatGPT vs. web search Laboratory Experiment Controlled experiment, Survey [PS18] - How LLMs affect students’ learning and practice of software testing Laboratory Experiment Controlled experiment: Between-subjects, Survey 1 Primary Study - Goal Strategy Procedure [PS17] - Understand developers’ daily and continuous experiences with AI code assistants Field Study Interview [PS15] - Understand developers’ practices with AI tools to identify its benefits and challenges Sample Study Survey [PS8] - Explore interaction patterns and challenges of using an AI tutor for students Field Experiment Concept Implementation, Surveys [PS11] - Practical usefulness of ChatGPT for professional software engineers Field Study Observational Study, Survey [PS22] - Initial experiences using a RAG-based coding LLM-assistant Laboratory Experiment User Study, Survey [PS12] - Expert opinions on future GenAI impact on labor market Judgment Study Delphi Study, Interviews, Surveys [PS26] - What influences engineers to adopt GenAI tools Sample Study Surveys [PS37] - GitHub Copilot usage and self-reported productivity Sample Study Case Study, Survey [PS19] - Understand how programmers interact with GitHub Copilot Experimental Simulation User Study, Survey [PS30] - Prevent unhelpful prompts that affect software productivity Laboratory Experiment Concept Implementation (Proof of Concept) [PS29] - Explore the role of bots on software development productivity Formal Theory - [PS31] - Future of AI agents and programmers Opinion Paper - [PS24] - Futuristic scenario on AI’s impact on developers (2024–2030) Opinion Paper - [PS1] - Roadmap of how AI reshapes software engineering Formal Theory - 2 Table 2: Study distribution across investigated SPACE dimension combinations. SPACE Dimensions Studies Satisfaction [PS7] Performance [PS3] Communication [PS1] Activity + Efficiency [PS10] Performance + Activity [PS30] Performance + Efficiency [PS2, PS12, PS25] Satisfaction + Communication [PS6, PS8, PS11] Satisfaction + Efficiency [PS15, PS17, PS22, PS26] Satisfaction + Performance [PS9, PS13, PS32, PS35] Performance + Communication + Efficiency [PS29] Satisfaction + Activity + Efficiency [PS27] Satisfaction + Performance + Communication [PS24] Satisfaction + Communication + Efficiency [PS21, PS31] Satisfaction + Performance + Activity [PS5, PS18, PS20] Satisfaction + Performance + Efficiency [PS4, PS16, PS23, PS33, PS36] Satisfaction + Performance + Communication + Efficiency [PS14] All dimensions [PS19, PS28, PS34, PS37] 3 References Primary Studies [PS1] Silvia Abrah˜ ao et al. “Software Engineering by and for Humans in an AI Era”. In: ACM Transactions on Software Engineering and Methodology (2025). [PS2] Jacques Bughin. “The role of firm AI capabilities in generative AI-pair coding”. In: Journal of Decision Systems (2024), pp. 1–22. [PS3] Jacques Bughin. “What drives the corporate payoffs of using generative artificial intelligence?” In: Structural Change and Economic Dynamics 71 (2024), pp. 658–668. [PS4] Tianyi Chen. “The Impact of AI-Pair Programmers on Code Quality and Developer Satisfaction: Evidence from TiMi studio”. In: Proceedings of the 2024 International Conference on Generative Artificial Intelligence and Information Security. 2024, pp. 201–205. [PS5] Rudrajit Choudhuri et al. “How far are we? the triumphs and trials of generative ai in learning software engineering”. In: Proceedings of the IEEE/ACM 46th international conference on software engineering. 2024, pp. 1–13. [PS6] Mariana Coutinho et al. “The role of generative ai in software development productivity: A pilot case study”. In: Proceedings of the 1st ACM International Conference on AI-Powered Software. 2024, pp. 131–138. [PS7] Nicole Davila et al. “An industry case study on adoption of ai-based programming assistants”. In: Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice. 2024, pp. 92– 102. [PS8] Eduard Frankford et al. “Ai-tutoring in software engineering education”. In: Proceedings of the 46th International Conference on Software Engineering: Software Engineering Education and Training. 2024, pp. 309– 319. [PS9] Nicholas Gardella, Raymond Pettit, and Sara L Riggs. “Performance, Workload, Emotion, and Self-Efficacy of Novice Programmers Using AI Code Generation”. In: Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1. 2024, pp. 290–296. [PS10] Shaokang Jiang and Michael Coblenz. “An Analysis of the Costs and Benefits of Autocomplete in IDEs”. In: Proceedings of the ACM on Software Engineering 1.FSE (2024), pp. 1284–1306. [PS11] Ranim Khojah et al. “Beyond code generation: An observational study of chatgpt usage in software engineering practice”. In: Proceedings of the ACM on Software Engineering 1.FSE (2024), pp. 1819–1840. [PS12] Kathrin Komp-Leukkunen. “How ChatGPT shapes the future labour market situation of software engineers: A Finnish Delphi study”. In: Futures 160 (2024), p. 103382. [PS13] Mohammad Amin Kuhail et al. ““Will I be replaced?” Assessing ChatGPT’s effect on software development and programmer perceptions of AI tools”. In: Science of Computer Programming 235 (2024), p. 103111. [PS14] Sandeep Kaur Kuttal et al. “Trade-offs for substituting a human with an agent in a pair programming context: the good, the bad, and the ugly”. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 2021, pp. 1–20. [PS15] Jenny T Liang, Chenyang Yang, and Brad A Myers. “A large-scale survey on the usability of ai programming assistants: Successes and challenges”. In: Proceedings of the 46th IEEE/ACM international conference on software engineering. 2024, pp. 1–13. [PS16] Jinrun Liu et al. “Chatgpt vs. stack overflow: An exploratory comparison of programming assistance tools”. In: 2023 IEEE 23rd International Conference on Software Quality, Reliability, and Security Companion (QRS-C). IEEE. 2023, pp. 364–373. [PS17] Wendy Mendes, Samara Souza, and Cleidson De Souza. “” You’re on a bicycle with a little motor”: Benefits and Challenges of Using AI Code Assistants”. In: Proceedings of the 2024 IEEE/ACM 17th International Conference on Cooperative and Human Aspects of Software Engineering. 2024, pp. 144–152. 4 [PS18] Simone Mezzaro, Alessio Gambi, and Gordon Fraser. “An empirical study on how large language models impact software testing learning”. In: Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering. 2024, pp. 555–564. [PS19] Hussein Mozannar et al. “Reading between the lines: Modeling user behavior and costs in AI-assisted programming”. In: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 2024, pp. 1– 16. [PS20] Vijayaraghavan Murali et al. “AI-assisted Code Authoring at Scale: Fine-tuning, deploying, and mixed methods evaluation”. In: Proceedings of the ACM on Software Engineering 1.FSE (2024), pp. 1066–1085. [PS21] Daye Nam et al. “Using an llm to help with code understanding”. In: Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. 2024, pp. 1–13. [PS22] Gustavo Pinto et al. “Developer Experiences with a Contextualized AI Coding Assistant: Usability, Expectations, and Outcomes”. In: Proceedings of the IEEE/ACM 3rd International Conference on AI EngineeringSoftware Engineering for AI. 2024, pp. 81–91. [PS23] Crystal Qian and James Wexler. “Take it, leave it, or fix it: measuring productivity and trust in humanAI collaboration”. In: Proceedings of the 29th International Conference on Intelligent User Interfaces. 2024, pp. 370–384. [PS24] Ketai Qiu et al. “From Today’s Code to Tomorrow’s Symphony: The AI Transformation of Developer’s Routine by 2030”. In: arXiv preprint arXiv:2405.12731 (2024). [PS25] Asha Rajbhoj et al. “Accelerating software development using generative ai: Chatgpt case study”. In: Proceedings of the 17th innovations in software engineering conference. 2024, pp. 1–11. [PS26] Daniel Russo. “Navigating the complexity of generative ai adoption in software engineering”. In: ACM Transactions on Software Engineering and Methodology 33.5 (2024), pp. 1–50. [PS27] Priyam Sahoo et al. “Ansible Lightspeed: A Code Generation Service for IT Automation”. In: Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 2024, pp. 2148–2158. [PS28] Danie Smit et al. “The impact of GitHub Copilot on developer productivity from a software engineering body of knowledge perspective”. In: (2024). [PS29] Margaret-Anne Storey and Alexey Zagalsky. “Disrupting developer productivity one bot at a time”. In: Proceedings of the 2016 24th ACM SIGSOFT international symposium on foundations of software engineering. 2016, pp. 928–931. [PS30] Zhensu Sun et al. “Don’t Complete It! Preventing Unhelpful Code Completion for Productive and Sustainable Neural Code Completion Systems”. In: 2023 IEEE/ACM 45th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). IEEE. 2023, pp. 324–325. [PS31] Steven L Tanimoto. “Five Futures with AI Coding Agents”. In: Companion Proceedings of the 7th International Conference on the Art, Science, and Engineering of Programming. 2023, pp. 32–38. [PS32] Helena Vasconcelos et al. “Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions”. In: ACM Transactions on Computer-Human Interaction (2024). [PS33] Wei Wang et al. “Rocks coding, not development: A human-centric, experimental evaluation of LLM-supported SE tasks”. In: Proceedings of the ACM on Software Engineering 1.FSE (2024), pp. 699–721. [PS34] Thomas Weber et al. “Significant productivity gains through programming with large language models”. In: Proceedings of the ACM on Human-Computer Interaction 8.EICS (2024), pp. 1–29. [PS35] Justin D Weisz et al. “Better together? an evaluation of ai-supported code translation”. In: Proceedings of the 27th International Conference on Intelligent User Interfaces. 2022, pp. 369–391. [PS36] Frank F Xu, Bogdan Vasilescu, and Graham Neubig. “In-ide code generation from natural language: Promise and challenges”. In: ACM Transactions on Software Engineering and Methodology (TOSEM) 31.2 (2022), pp. 1– 47. [PS37] Albert Ziegler et al. “Productivity assessment of neural code completion”. In: Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming. 2022, pp. 21–29. 5