Online Appendix for "The Invisible Hand of AI Libraries Shaping Open Source Projects and Communities"

The Invisible Hand of AI Libraries Shaping Open Source Projects and Communities Online Appendix Matteo Espositoa, Andrea Janesb, Valentina Lenarduzzia,c, Davide Taibia,c aUniversity of Oulu, Finland, bFree University of Bozen-Bolzano, Italy, cUniversity of Southern Denmark, Vejle, Denmark [email protected], [email protected], v[email protected], da[email protected] CONTENTS I Background & Related Work 1 I-A OSS in Empirical Software Engineering 1 I-B AI libraries in Software Engineering . . 1 I-C AI libraries in OSS . . . . . . . . . . . 2 II Empirical Study Design 2 III Controlling for Confounding Factors in RQ2 and RQ3 2 III-A Sampling-Level Controls . . . . . . . . 3 III-B Analysis-Time Controls . . . . . . . . . 3 III-C Domain-Related Considerations . . . . . 3 III-D Robustness and Interpretation . . . . . . 3 IV Software Metrics 3 V Repository Metrics 3 References 3 I. BACKGROUND & RELATED WORK One distinguishing feature of OSS is that its source code is openly accessible for anyone to use, modify, and distribute. In contrast to proprietary software developed and owned by individual companies, OSS fosters collaborative and communitydriven development Fortunato et al. [1]. These characteristics make OSS a suitable candidate for diverse research in software engineering Gyimothy et al. [2]. A. OSS in Empirical Software Engineering Mockus et al. [3] compare commercial and open-source methodologies using Mozilla and Apache as examples. They find that while commercial projects rely on explicit coordination, OSS projects like Apache use decentralized communication. Mozilla faces challenges due to module interdependencies, while Apache’s approach emphasizes modularity. Leveraging the open-source community for low-interdependence tasks can improve productivity. Similarly, Gyimothy et al. [2] conducted a study to assess the quality and reliability of OSS systems, mainly focusing on Mozilla, a widely used web and email suite. They calculated object-oriented metrics proposed by Chidamber et al. [4] to analyze fault-proneness in Mozilla’s source code. Moreover, Yu et al. [5] conducted a large-scale empirical study on Java annotations, analyzing 1,094 OSS projects on GitHub. They investigated annotation usage, evolution, and impact, revealing 10 novel findings. Similarly, we are also interested in the context of OSS projects. Nonetheless, the author focused on Java annotations, while our study focuses on AI libraries and technologies. Arberdour et al. [6] highlights the significance of the OSS community’s research on OSS quality, emphasizing the shift towards empirical data over subjective opinions. Moreover, they discuss the consensus in the literature on the key components of OSS quality. The authors review the state of the art, extracting lessons and exploring the divergences and potential inclusion of OSS quality approaches into closedsource software development methods. Esposito et al. [7] investigated the hidden bugs, i.e., buggy untouched methods, that remained dormant for various releases of OSS projects. Similarly, Esposito et al. [8] performed a large-scale analysis on OSS vulnerability severity according to SonarQube and the National Vulnerability (NVD) Datasets to assess whether the NVD vulnerability severity was correlated with SQ severity levels. Finally, Esposito et al. [9] performed a large-scale OSS-oriented test of static analysis security testing tools. Moreover, recent studies employed the ready availability of OSS development history to improve JIT defect prediction B. AI libraries in Software Engineering AI stimulated new research approaches to each software engineering task [10]. Wan et al. [11], investigated how incorporating machine learning into systems impacts software development practices. They conducted both qualitative interviews and a quantitative survey to identify differences between developing machine learning systems and traditional systems. The study revealed significant disparities in software engineering aspects such as requirements, design, testing, and process, as well as in work characteristics like skill variety, problem-solving, and task identity. Moran et al. [12] devised an automated approach called ReDraw to streamline the transformation of GUI mock-ups into code for user-facing software. ReDraw utilizes computer vision techniques and deep convolutional neural networks. A data-driven algorithm then generates a hierarchical GUI structure, enabling automatic assembly of prototype applications for the Android platform. Esposito et al. [13], compiled a systematic dataset review on the usage of AI in the field of vulnerability prediction. Furthermore, Liu et al. [14] conducted an empirical study on Stack Overflow to explore developers’ challenges in finding suitable ML/DL libraries for AI tasks. Based on their findings, they proposed MLTaskKG, a task-oriented ML/DL library recommendation approach. MLTaskKG utilizes a knowledge graph to capture AI tasks, ML/DL models, implementations, repositories, and their relationships. By extracting knowledge from various sources, including ML/DL resource websites, papers, and frameworks, MLTaskKG recommends libraries matching developers’ requirements. Xu et al. [15], address the challenge of turning conceptual ideas into code, especially when dealing with unfamiliar library APIs, by investigating the promise and challenges of using ML for code generation and retrieval within the PyCharm IDE. They developed a plugin for PyCharm that combines code generation and retrieval functionality and orchestrates virtual environments to collect user events. While qualitative surveys indicate positive developer experiences, quantitative results regarding increased productivity, code quality, and program correctness are inconclusive. The study identifies critical points for improving future machine learning-based code assistants and demonstrates developers’ preferences between code generation and retrieval. Finally, Wang et al. [16] conducted a 12-year Systematic Literature Review on 1,428 ML/DL-related SE papers published between 2009 and 2020. Their analysis highlighted the impacts, complexities, and challenges of applying ML/DL techniques to SE tasks. They categorized the rationales for ML/DL technique selection into five themes and examined issues related to reproducibility, replicability, and model choices in SE tasks. C. AI libraries in OSS Dilhara et al. [17] presented a large-scale empirical study to examine the challenges developers face utilizing ML libraries in Software-2.0 development. Key findings include a significant increase in the adoption of ML libraries, various usage patterns, and unique challenges such as the binary incompatibility of trained ML models. Our study focuses on the broader AI spectrum of libraries, evaluating the impact of adopting them. Tufano et al. [18] investigated how developers use Large Language Models, like OpenAI’s ChatGPT, in OSS projects. Out of the 1,501 instances of potential ChatGPT usage in open-source projects, the authors narrowed them down to 467 actual ChatGPT instances. [18] categorized these instances into 45 tasks developers automate using ChatGPT, hence helping developers understand how to use LLMs effectively and providing researchers with insight into tasks that can benefit and those that do not from automated solutions. Our study focuses on the usage of AI libraries, thus including LLMs, in OSS projects, intending to investigate the impact that the adoption of such libraries has on community engagement and the technical ecosystem. II. EMPIRICAL STUDY DESIGN RQ1(Adoption). How widespread is the adoption of AI-related libraries in open-source software projects? RQ1.1Does AI adoption differ between Java and Python projects? This RQ aims to measure the extent of AI library adoption in OSS projects. AI adoption captures whether and how much a project integrates AI features and capabilities through external components, i.e., libraries. According to [19]–[21], we can analyze evidence of a project dependency on three levels: •Dependency-level evidence: importing or referencing AI/ML libraries (e.g., TensorFlow, PyTorch, Scikit-learn, Hugging Face Transformers). •Code-level evidence: using classes, functions, or APIs belonging to these libraries. •Commit-level or release-level evidence: identifying when AI libraries were introduced and whether they continue to be maintained or expanded. In this first RQ, we aim to measure the dependency-level evidence, specifically whether the OSS project includes references to AI libraries in its list of dependencies. Furthermore, we are keen to focus on Java and Python because they represent two dominant yet contrasting ecosystems in software development [22]. Python is the de facto language for AI and machine learning, offering a rich ecosystem of libraries, including TensorFlow, PyTorch, and scikit-learn [23], [24]. In contrast, Java is widely adopted in enterprise and large-scale software systems, where AI integration is emerging [25]. Studying both allows us to capture differences between experimentation-oriented (Python) and productionoriented (Java) environments. Other languages were excluded as they either lack mature AI ecosystems (e.g, Go) [26], [27], or they are more challenging, e.g., steeper learning curve, memory management complexity, that may affect adoption compared to higher-level languages (e.g. C++) [28], or are less representative of large, diverse open-source communities suitable for comparative analysis [23]. III. CONTROLLING FOR CONFOUNDING FACTORS IN RQ2 AND RQ3 Observational studies on open-source software are inherently exposed to confounding factors, as project characteristics may simultaneously influence both technology adoption and the metrics of interest. On the one hand, this study aim to identify potential non-trivial confounding factors to use in a upcoming cohort study. On the other hand, in the TABLE I REPOSITORY INCLUSION CRITERIA Criterion Condition Archived or Fork Repository is neither archived nor a fork Last Activity The last activity on the repository is less than six months old Contributors Repository has at least three contributors Star Count Repository has 50 or more stars SBOM Availability The GitHub repository allows access to the SBOM context of RQ2 and RQ3, project size, popularity, activity, and application domain are particularly relevant, as they may correlate with both AI library adoption and repositoryor code-level metrics. To mitigate these risks, our study adopts a multi-level strategy that combines sampling controls,analysis-time adjustments, and robustness checks. A. Sampling-Level Controls At the data collection stage, we apply strict inclusion criteria to reduce variance stemming from unstable or atypical repositories. Specifically, we exclude forks and archived projects, require recent activity within six months prior to data collection, enforce a minimum of three contributors, and include only repositories with at least 50 GitHub stars. The star threshold is used as a proxy for project popularity, following prior empirical work (Table I). These criteria help reduce confounding effects related to project immaturity, inactivity, and extremely low visibility, which are known to affect both adoption behavior and software metrics. B. Analysis-Time Controls To further account for confounding in RQ2 and RQ3, we explicitly control for project characteristics during analysis. Rather than relying solely on aggregate comparisons between AI-adopting and non-adopting projects, we perform stratified analyses based on key confounding variables. In particular, projects are grouped into homogeneous strata according to size-related indicators (e.g., code volume) and popularity (e.g., star-count quantiles). Comparisons are then conducted within each stratum, reducing the likelihood that observed differences are driven by systematic differences in project scale or visibility. In addition, when feasible, we apply matching-based comparisons, pairing AI-adopting projects with non-adopting projects that exhibit similar values for size, popularity, and project age. C. Domain-Related Considerations Project domain represents a more challenging confounding factor to operationalize at scale. To address this, we leverage coarse-grained domain indicators derived from repository metadata, such as GitHub topics and tags, when available and READMEs. These indicators are used to support stratification or matching, enabling comparisons among projects operating in similar application contexts. Nonetheless, we acknowledge that domain effects cannot be entirely eliminated and explicitly treat residual domain-related confounding as a threat to internal validity. D. Robustness and Interpretation Finally, to ensure the stability of findings, we assess whether conclusions for RQ2 and RQ3 remain consistent across different strata and matched samples, and we report effect sizes alongside statistical significance. This approach supports a more cautious and transparent interpretation of results, emphasizing practical relevance rather than relying solely on pvalues. IV. SOFTWARE METRICS Software metrics provide quantitative insights into various aspects of software design, implementation, and maintainability [7]. They help developers and project managers assess code quality, identify potential issues, and improve software reliability and performance [7]. Regarding such metrics, Table II presents those that can be collected using the ”full metric” parameter in the SciTool’s Understand static analysis tool. These metrics, including cyclomatic complexity, line counts, and class coupling, measure different dimensions of code, such as its complexity, readability, cohesion, and interdependencies, offering a comprehensive view of software quality. V. REPOSITORY METRICS Repository metrics provide valuable insights into the activity, health, and community engagement of a software project hosted on platforms like GitHub. These metrics help developers, maintainers, and stakeholders understand the level of collaboration, issue resolution, and code evolution in the repository. The table below lists key repository metrics, their descriptions, and their classifications according to [29]–[32]. REFERENCES [1] L. Fortunato and M. Galassi, “The case for free and open source software in research and scholarship,” Philosophical Transactions of the Royal Society A, vol. 379, no. 2197, p. 20200079, 2021. [2] T. Gyimothy, R. Ferenc, and I. Siket, “Empirical validation of objectoriented metrics on open source software for fault prediction,” IEEE Transactions on Software Engineering, vol. 31, no. 10, pp. 897–910, 2005. [3] A. Mockus, R. T. Fielding, and J. D. Herbsleb, “Two case studies of open source software development: Apache and mozilla,” ACM Trans. Softw. Eng. Methodol., vol. 11, no. 3, p. 309–346, jul 2002. [Online]. Available: https://doi.org/10.1145/567793.567795 [4] S. R. Chidamber and C. F. Kemerer, “A metrics suite for object oriented design,” IEEE Transactions on software engineering, vol. 20, no. 6, pp. 476–493, 1994. [5] Z. Yu, C. Bai, L. Seinturier, and M. Monperrus, “Characterizing the usage, evolution and impact of java annotations in practice,” IEEE Trans. Software Eng., vol. 47, no. 5, pp. 969–986, 2021. [Online]. Available: https://doi.org/10.1109/TSE.2019.2910516 [6] M. Aberdour, “Achieving quality in open-source software,” IEEE Software, vol. 24, no. 1, pp. 58–64, 2007. [7] M. Esposito and D. Falessi, “Uncovering the hidden risks: The importance of predicting bugginess in untouched methods,” in 2023 IEEE 23rd International Working Conference on Source Code Analysis and Manipulation (SCAM). IEEE, 2023, pp. 277–282. TABLE II SOFTWARE METRICS FROM SCITOOLS’UNDERSTAND. ACRONYM:LLANGUAGE:JJAVA,PPYTHON,BBOTH;GGRANULARITY:CCLASS,FFUNCTION,MMETHOD,PRPROJECT Name Description L G AvgCountLineBlank Average number of blank lines per method or function. B M AvgCountLineCode Average number of lines of code per method or function. B M AvgCyclomatic Average cyclomatic complexity of methods/functions within a class or file. B C/F AvgCountLineComment Average number of comment lines per method or function. B M CountClassBase Number of immediate base classes a class inherits from. B C CountClassCoupled Number of unique classes that a class is directly coupled to. B C CountClassDerived Number of classes that directly inherit from a given class. B C CountDeclClass Number of class declarations within a file or namespace. B F CountDeclFunction Number of function declarations within a file or namespace. P F CountDeclInstanceMethod Number of instance method declarations within a class. B C CountDeclInstanceVariablePrivate Number of private instance variable declarations within a class. B C CountDeclMethod Number of method declarations within a class. B C CountLineBlank Number of blank lines in the code (readability metric). B F CountLineCode Number of lines containing executable code. B F CountLineComment Number of lines containing comments in the code. B F CountStmt Total number of statements in the code. B F Cyclomatic Cyclomatic complexity of a single method or function. B M MaxCyclomatic Maximum cyclomatic complexity among all methods/functions. B C/F MaxInheritanceTree Maximum depth of the inheritance tree for a class. B C MaxNesting Maximum nesting level of control structures. B M PercentLackOfCohesion Lack of cohesion in a class (higher = lower cohesion). B C SumCyclomatic Sum of cyclomatic complexities of all methods/functions. B C/F [8] M. Esposito, S. Moreschini, V. Lenarduzzi, D. H¨ astbacka, and D. Falessi, “Can we trust the default vulnerabilities severity?” in 2023 IEEE 23rd International Working Conference on Source Code Analysis and Manipulation (SCAM). IEEE, 2023, pp. 265–270. [9] M. Esposito, V. Falaschi, and D. Falessi, “An extensive comparison of static application security testing tools,” in Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering. ACM, 2024, pp. 69–78. [10] J. Mostow, “Foreword what is ai? and what does it have to do with software engineering?” IEEE Transactions on Software Engineering, vol. SE-11, no. 11, pp. 1253–1256, 1985. [11] Z. Wan, X. Xia, D. Lo, and G. C. Murphy, “How does machine learning change software development practices?” IEEE Transactions on Software Engineering, vol. 47, no. 9, pp. 1857–1871, 2021. [12] K. Moran, C. Bernal-C´ ardenas, M. Curcio, R. Bonett, and D. Poshyvanyk, “Machine learning-based prototyping of graphical user interfaces for mobile apps,” IEEE Transactions on Software Engineering, vol. 46, no. 2, pp. 196–221, 2020. [13] M. Esposito and D. Falessi, “Validate: A deep dive into vulnerability prediction datasets,” Information and Software Technology, p. 107448, 2024. [14] M. Liu, C. Zhao, X. Peng, S. Yu, H. Wang, and C. Sha, “Task-oriented ml/dl library recommendation based on a knowledge graph,” IEEE Transactions on Software Engineering, vol. 49, no. 8, pp. 4081–4096, 2023. [15] F. F. Xu, B. Vasilescu, and G. Neubig, “In-ide code generation from natural language: Promise and challenges,” ACM Trans. Softw. Eng. Methodol., vol. 31, no. 2, mar 2022. [Online]. Available: https://doi.org/10.1145/3487569 [16] S. Wang, L. Huang, A. Gao, J. Ge, T. Zhang, H. Feng, I. Satyarth, M. Li, H. Zhang, and V. Ng, “Machine/deep learning for software engineering: A systematic literature review,” IEEE Transactions on Software Engineering, vol. 49, no. 3, pp. 1188–1231, 2023. [17] M. Dilhara, A. Ketkar, and D. Dig, “Understanding software-2.0: A study of machine learning library usage and evolution,” ACM Trans. Softw. Eng. Methodol., vol. 30, no. 4, jul 2021. [Online]. Available: https://doi.org/10.1145/3453478 [18] R. Tufano, A. Mastropaolo, F. Pepe, O. Dabi´ c, M. D. Penta, and G. Bavota, “Unveiling chatgpt’s usage in open source projects: A mining-based study,” 2024. [19] N. B. Tarrega, M. Zivkovic, A. Oprescu, and S. PCS, “Measuring the impact of library dependency on maintenance.” in SATToSE, 2020. [20] R. G. Kula, D. M. German, A. Ouni, T. Ishio, and K. Inoue, “Do developers update their library dependencies? an empirical study on the impact of security advisories on library migration,” Empirical Software Engineering, vol. 23, no. 1, pp. 384–417, 2018. [21] I. Pashchenko, H. Plate, S. E. Ponta, A. Sabetta, and F. Massacci, “Vulnerable open source dependencies: Counting those that matter,” in Proceedings of the 12th ACM/IEEE international symposium on empirical software engineering and measurement, 2018, pp. 1–10. [22] S. Cass, “The top programming languages 2024: Typescript and rust are among the rising stars,” IEEE Spectrum, August 2024, accessed: 2024-12-08. [Online]. Available: https://spectrum.ieee.org/top-program ming-languages-2024 [23] H. Li and C.-P. Bezemer, “Bridging the language gap: an empirical study of bindings for open source machine learning libraries across software package ecosystems,” Empirical Software Engineering, vol. 30, no. 1, p. 6, 2025. [24] A. Khandare, N. Agarwal, A. Bodhankar, A. Kulkarni, and I. Mane, “Analysis of python libraries for artificial intelligence,” in Intelligent Computing and Networking, V. E. Balas, V. B. Semwal, and A. Khandare, Eds. Singapore: Springer Nature Singapore, 2023, pp. 157–177. TABLE III REPOSITORY METRICS CLASSIFICATION. ACRONYMS:AMACTIVITY;CEM - COMMUNITY ENGAGEMENT;ITM - ISSUE TRACKING;PRM - PULL REQUEST;RMM - RELEASE MANAGEMENT;WM - WORKFLOW;TM - TECHNOLOGY METRICS. Metric Name Description Group Branches Number of branches in the repository. AM Closed Issues Number of issues that have been resolved or closed. ITM Closed Pulls Number of pull requests that have been merged or closed. PRM Comments Total number of comments on issues and pull requests. ITM Commits Total number of commits made in the repository. AM Contributors Number of unique contributors who have contributed to the repository. CEM Dependencies Number of external dependencies listed in the repository. RMM Forks Number of times the repository has been forked. CEM Issues Total number of issues in the repository (open and closed). ITM Languages Programming languages used in the repository, measured by lines of code. TM Open Issues Number of currently open issues in the repository. ITM Open Pulls Number of currently open pull requests. PRM Pulls Total number of pull requests (open and closed). PRM Releases Total number of releases published in the repository. RMM Star Count Number of stars the repository has received. CEM Subscribers Number of people subscribed to repository notifications. CEM Tags Total number of tags created in the repository. AM Topics Topics or keywords associated with the repository. TM Watchers Number of users watching the repository. CEM Workflows Number of workflows configured for CI/CD in the repository. WM [25] Azul Systems, “2025 state of java survey & report – java’s continuing influence in the enterprise,” https://www.azul.com/blog/2025-state-of-j ava-survey-confirms-majority-of-enterprise-apps-are-built-on-java-2/, 2025, accessed: 2025-11. [26] JetBrains. (2023) Does machine learning in go have a future? Accessed: 2025-11-09. [Online]. Available: https://blog.jetbrains.com/go/2023/06/ 22/does-machine-learning-in-go-have-a-future/ [27] ——. (2024) Is golang still growing? go language popularity trends in 2024. Accessed: 2025-11-09. [Online]. Available: https: //blog.jetbrains.com/research/2025/04/is-golang-still-growing-go-langu age-popularity-trends-in-2024/ [28] GeeksforGeeks. (2024) C++ libraries for machine learning. Accessed: 2025-11-09. [Online]. Available: https://www.geeksforgeeks.org/mach ine-learning/c-libraries-for-machine-learning/ [29] M. D’Ambros, M. Lanza, and R. Robbes, “An extensive comparison of bug prediction approaches,” in 2010 7th IEEE working conference on mining software repositories (MSR 2010). IEEE, 2010, pp. 31–41. [30] G. Bavota and B. Russo, “Four eyes are better than two: On the impact of code reviews on software quality,” in 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2015, pp. 81–90. [31] Y. Bugayenko, “Cam: A collection of snapshots of github java repositories together with metrics,” arXiv preprint arXiv:2403.08488, 2024. [32] GitHub, Inc., GitHub Documentation, 2025. [Online]. Available: https://docs.github.com/en