Software Heritage 10 years of archiving, sharing, and studying source code at the ultimate scale

Software Heritage 10 years of archiving, sharing, and studying source code at the ultimate scale Stefano Zacchiroli Software Heritage Télécom Paris, Polytechnic Institute of Paris 23 Jan 2025 10 Ans du Séminaire Codes Sources, Sorbonne Université Paris, France THE GREAT LIBRARY OF SOURC E CODE Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Software Heritage 23 Jan 2025 1 / 20 Outline 1Introduction 2Software Heritage 3Public code evolution through the Software Heritage looking glass 4Selected highlight: Open Science 5Selected highlight: Source Code Compliance 6Selected highlight: AI and transparent LLMs 7Conclusion Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Software Heritage 23 Jan 2025 2 / 20 About the speaker Professor of Computer Science, Télécom Paris, Polytechnic Institute of Paris Free/Open Source Software activist (25+ years) Debian Developer & Former 3x Debian Project Leader Former Open Source Initiative (OSI) director Software Heritage co-founder & CSO Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Software Heritage 23 Jan 2025 2 / 20 Previously, on Séminaire Codes Sources Flashback to 16 Dec 2015, on the 1st year of this seminar, talk "Large-scale source code archival, publishing, and indexing with Debsources" by yours truly (slides): Debsources in a nutshell 1an infrastructure to publish Debian source code on the Web 2a notable instance indexing all Debian source code to date: http://sources.debian.net For developers: browse/search source code syntax highlighting pinpoint code lines, annotate For data miners: Debian evolution over time 20+ years of FOSS history live change monitoring Stefano Zacchiroli (UPD / IRILL) Debsources Séminaire Codes Sources 8 / 45 Highlight #2: programming languages top-5 most popular programming languages in Debian over time Recent trends (post-etch, 2007): Cstill leads, steady (absolute) growth C stops losing (relative) ground to C++ decrease of Perl/Shell popularity Python rises (more maintainable glue code?) Lisp halves its popularity Java no longer under-represented Stefano Zacchiroli (UPD / IRILL) Debsources Séminaire Codes Sources 28 / 45 Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Software Heritage 23 Jan 2025 3 / 20 Previously, on Séminaire Codes Sources (cont.) Flashback to 16 Dec 2015, on the 1st year of this seminar, talk "Large-scale source code archival, publishing, and indexing with Debsources" by yours truly (slides): What made Debsources possible? Source code: availability “The commons is the cultural and natural resources accessible to all members of a society [. . . ]” licensing terms “The software commons consists of all computer software which is available at little or no cost and which can be altered and reused with few restriction” organization ñpackage & version namespaces ñintrinsic identifiers (e.g., SHA256) What would it take to do the same for the entire software commons? And what can we do with it once we have it? Stefano Zacchiroli (UPD / IRILL) Debsources Séminaire Codes Sources 32 / 45 The Software Heritage Project Our mission Collect,organise,preserve and share all the software that lies at the heart of our culture and our society. Joint work with Roberto Di Cosmo Stefano Zacchiroli (UPD / IRILL) Debsources Séminaire Codes Sources 38 / 45 Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Software Heritage 23 Jan 2025 4 / 20 Outline 1Introduction 2Software Heritage 3Public code evolution through the Software Heritage looking glass 4Selected highlight: Open Science 5Selected highlight: Source Code Compliance 6Selected highlight: AI and transparent LLMs 7Conclusion Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Software Heritage 23 Jan 2025 5 / 20 Software Heritage in a nutshell www.softwareheritage.org THE GREAT L I BRA RY OF SO URCE CO D E Collect, preserve and share all software source code Preserve our heritage, enabling better software and better science for all Reference catalog find and reference all software source code Universal archive preserve and share all software source code Research infrastructure enable analysis of all software source code Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Software Heritage 23 Jan 2025 5 / 20 Auniversal software archive, as a shared infrastructure One infrastructure, open and shared The largest archive ever built Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Software Heritage 23 Jan 2025 6 / 20 The archive under the hood Git loader Mercurial loader Debian source package loader pypi source package loader . . . Software Heritage Archive Merkle DAG + blob storage Loading & deduplication dsc dsc hg hg hg git git git git svn svn svn tar zip software origins Package repos Forges GitHub lister GitLab lister Debian lister PyPi lister . . . Distros ... Scheduling Listing (full/incremental) tar origins snapshots releases revisionsrevisions directoriesdirectories contents Global development history permanently archived inauniform data model over 20 billion unique source files from over 300 million software projects ~2PB (compressed) blobs, ~50 B nodes, ~700 B edges Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Software Heritage 23 Jan 2025 7 / 20 Diversity, equity, and inclusion (cont.) Geographic gap — key findings Early decades of public code dominated by contributions from North America, followed by a period of alternating dominance between North America and Europe. Since then geographic diversity has increased constantly, with raising importance of contributions from Central and South America. Geo and Gender gap: the trend of increased female contributions is global, with the exception of some regions in Asia where it is either slower or flat. Rossi and Zacchiroli. Geographic diversity in public code contributions. MSR 2022 Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Software Heritage 23 Jan 2025 13 / 20 Outline 1Introduction 2Software Heritage 3Public code evolution through the Software Heritage looking glass 4Selected highlight: Open Science 5Selected highlight: Source Code Compliance 6Selected highlight: AI and transparent LLMs 7Conclusion Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Software Heritage 23 Jan 2025 14 / 20 Addressing key needs for open science (ARDC) Archive (20B+ files, 320M+ projects) Git loader Mercurial loader Debian source package loader pypi source package loader . . . Software Heritage Archive Merkle DAG + blob storage Loading & deduplication dsc dsc hg hg hg git git git git svn svn svn tar zip software origins Package repos Forges GitHub lister GitLab lister Debian lister PyPi lister . . . Distros ... Scheduling Listing (full/incremental) tar origins snapshots releases revisionsrevisions directoriesdirectories contents Reference (50 billion SWHIDs) Describe Intrinsic metadata from source code Contributed the Codemeta generator Cite/Credit Contributed biblatex-software style Software Citation from the archive! Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Software Heritage 23 Jan 2025 14 / 20 A few adoption indicators ●Recommendations in ANR 2023 guidelines (p. 17) ●HAL+SWH in the Open Science software booklet Policy Users and collaborations Graphics Replicability Stamp Initiative Projects Outline 1Introduction 2Software Heritage 3Public code evolution through the Software Heritage looking glass 4Selected highlight: Open Science 5Selected highlight: Source Code Compliance 6Selected highlight: AI and transparent LLMs 7Conclusion Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Software Heritage 23 Jan 2025 16 / 20 Tech preview: swh-scanner Problem Detect unknown files in an open source project Detect proprietary files leaked on a public forge Solution swh-scanner :open source and open data source code scanner for compliance workflows, backed by the largest public archive of public source code. Design Software Heritage archive as source of truth about public code Merkle DAG model and SWHIDs for maximum scanning efficiency Output: source tree partition into known/unknown + provenance information Package: pypi.org/project/swh.scanner (GPL 3+) Ref.: Serafini, Zacchiroli. Efficient Prior Publication Identification for Open Source Code. OpenSym 2022: 12:1-12:8 Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Software Heritage 23 Jan 2025 16 / 20 Outline 1Introduction 2Software Heritage 3Public code evolution through the Software Heritage looking glass 4Selected highlight: Open Science 5Selected highlight: Source Code Compliance 6Selected highlight: AI and transparent LLMs 7Conclusion Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Software Heritage 23 Jan 2025 17 / 20 Looking for founding principles at Software Heritage Question: are we asking too much? 15 Findings from BigCode: The Stack v2 and StarCoder2 Released February 28th 2024 Yes one can build the best open LLM for code available while fully adhering to the Software Heritage principles for responsible LLMs, … and even more: the full training pipeline is made public too! 16 Dataset Model Roadmap: CodeCommons project (BPI-funded, 2025-2027) Addressing the current limitations in LLMs for code Vision: create the world reference for LLMs for code Massive, transparent and up to date, with qualified information Traceability of the contents, code attribution Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Software Heritage 23 Jan 2025 19 / 20 License dataset Stefano Zacchiroli A Large-scale Dataset of (Open Source) License Text Variants MSR 2022 (best dataset paper) + Empir. Soft. Eng. 28(6): 147 (2023) preprint: https://arxiv.org/abs/2308.11258 Dataset 6.9 million unique full texts of FOSS license variants Detected using filename patterns across the entire SWH archive LICENSE,COPYRIGHT,NOTICE, etc. Metadata: file lengths measures, detected MIME type, detected SPDX license (via ScanCode), example origin repository, oldest public commit of origin, ground truth Use cases Empirical studies on FOSS licensing, including phylogenetics Training of automated license classifiers NLP analyses of legal texts Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Software Heritage 23 Jan 2025 4 / 13 The Software Heritage Filesystem (SwhFS) The Software Heritage Filesystem (SwhFS) is a user-space POSIX filesystem that enables browsing parts of the Software Heritage archive as if it were locally available. Code: forge.softwareheritage.org/source/swh-fuse Documentation: docs.softwareheritage.org/devel/swh-fuse Thibault Allançon, Antoine Pietri, Stefano Zacchiroli The Software Heritage Filesystem (SwhFS): Integrating Source Code Archival with Development ICSE 2021 (Tool track): The 43rd Intl. Conference on Software Engineering https://arxiv.org/abs/2102.06390 Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Software Heritage 23 Jan 2025 5 / 13 The Software Heritage Filesystem (SwhFS) — example $ mkdir swhfs $ swh fs mount swhfs/ # mount the archive $ cd swhfs/ $ cat archive/swh:1:cnt:c839dea9e8e6f0528b468214348fee8669b305b2 #include <stdio.h> int main(void) { printf("Hello, World!\n"); } $ cd archive/swh:1:dir:1fee702c7e6d14395bbf5ac3598e73bcbf97b030 $ ls | wc -l 127 $ grep -i antenna THE_LUNAR_LANDING.s | cut -f 5 # IS THE LR ANTENNA IN POSITION 1 YET # BRANCH IF ANTENNA ALREADY IN POSITION 1 Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Software Heritage 23 Jan 2025 6 / 13 The Software Heritage Filesystem (SwhFS) — example (cont.) $ cd archive/swh:1:rev:9d76c0b163675505d1a901e5fe5249a2c55609bc $ ls -F history/ meta.json@ parent@ parents/ root@ $ jq ’.author.name, .date, .message’ meta.json "Michal Golebiowski-Owczarek" "2020-03-02T23:02:42+01:00" "Data:Event:Manipulation: Prevent collisions with Object.prototype ..." $ find root/src/ -type f -name ’*.js’ | xargs cat | wc -l 10136 Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Software Heritage 23 Jan 2025 7 / 13 Graph compression Paolo Boldi, Antoine Pietri, Sebastiano Vigna, Stefano Zacchiroli Ultra-Large-Scale Repository Analysis via Graph Compression SANER 2020, 27th Intl. Conf. on Software Analysis, Evolution and Reengineering. IEEE Research question Is it possible to efficiently perform software development history analyses at the scale of Software Heritage archive on a single, relatively cheap machine? Idea Apply state-of-the-art graph compression techniques from the field of Web graph / social network analysis. Results The entire archive graph (35 B nodes, 500 B edges) can be loaded in 300 GiB and then traversed at the cost of tens of ns per edge (= a few hours for a full single-thread visit). Java and gRPC APIs available: docs.softwareheritage.org/devel/swh-graph/grpc-api.html Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Software Heritage 23 Jan 2025 8 / 13 Graph compression pipeline MPH BV compress BFS Permute Compressed graph (forward) Transpose Compressed graph (backward) Merkle DAG MPH: minimal perfect hash, mapping Merkle IDs to 0..N-1 integers BV compress: Boldi-Vigna compression (based on MPH order) BFS: breadth-first visit to renumber Permute: update BV compression according to BFS order (Re)establishing locality Key for good compression is a node ordering that ensures locality and similarity Which is very much not the case with Merkle IDs, ...but is the case again after BFS reordering Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Software Heritage 23 Jan 2025 9 / 13 Is my FOSS code (known to be) vulnerable? "The Common Vulnerabilities and Exposures (CVE) system provides a reference-method for publicly known information-security vulnerabilities and exposures." e.g., CVE-2014-0160 (AKA: Heartbleed), CVE-2021-44228 (AKA: Log4Shell) Tooling A number of state-of-the-art tools in FOSS compliance can: 1Scan your local code base, 2Compare it with a knowledge base that knows about CVEs, 3Emit warnings like: "you include/depend on code affected by the following CVEs: ...". CVE ↔code matching heuristics are based for the most part on package metadata (in particular: version numbers) Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Software Heritage 23 Jan 2025 10 / 13 Example — osv.dev OSV.dev:"A distributed vulnerability database for Open Source. An open, precise, and distributed approach to producing and consuming vulnerability information for open source." Service (operated by Google) and data format (standardized by OpenSSF) that: Crawls vulnerability information from many places (GitHub, distros, package manager repos, ...). Provides efficient APIs to query the information and integrate it into compliance toolchains. Is my code affected by a known CVE? By commit hash: $ curl -X POST -d \ ’{"commit": "6879efc2c1596d11a6a6ad296f80063b558d5e0f"}’ \ "https://api.osv.dev/v1/query" {"vulns":[{"id":"OSV-2020-484","summary":"Heap-buffer-overflow in AAT... Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Software Heritage 23 Jan 2025 11 / 13 Reasoning on the global commit graph State-of-the-art tech (e.g., OSV.dev by Google) to query whether a given commit is known to be vulnerable do not know about all code out there. They crawl project repos related to known vulnerabilities, but not their forks (across multiple forges!). Real-world example (one out of many we have identified): Linux kernel vulnerability: GSD-2022-1004193 Example of vulnerable commit: b13baccc3850ca8b8cccbf8ed9912dbaa0fdf7f3 Fix commit: a92d44b412e75dd66543843165e46637457f22cc False negative commit in fork (Raspberry PI Linux): c7c7a1a18af4c3bb7749d33e3df3acdf0a95bbb5 Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Software Heritage 23 Jan 2025 12 / 13 An universal knowledge base about public code vulnerabilities Vision Software Heritage is the perfect (and only) place where to build an universal knowledge base that maps known vulnerabilities to public code artifacts. SWH can provide an open data API mapping SWHIDs to CVEs, that knows about all public commits and can be leveraged to increase open source security. EU Cyber Resilience Act (CRA) Key helper to abide to CRA obligations, coming into effect ~Q3 2026. SWH funding member of the Open Regulatory Compliance Working Group. Roadmap Context: SWHSec project (PTCC-funded, 2023-2027). Current status: working prototype that processes OSV.dev data and use it to "color" the entire SWH commit graph (~5 billion commits) with vulnerability information. Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Software Heritage 23 Jan 2025 13 / 13