Software Heritage Large-Scale Research on Public Code Development

Software Heritage Large-Scale Research on Public Code Development Stefano Zacchiroli Télécom Paris, Institut Polytechnique de Paris [email protected] 23 Jan 2024 — LaBRI, Bordeaux THE GREAT LIBRARY OF SOURC E CODE Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 1 / 29 Outline 1Preface 2Software as knowledge 3Software Heritage 4Research highlights in empirical software engineering 5Conclusion Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 2 / 29 About the speaker Professor of Computer Science, Télécom Paris, Institut Polytechnique de Paris Free/Open Source Software activist (20+ years) Debian Developer & Former 3x Debian Project Leader Former Open Source Initiative (OSI) director Software Heritage co-founder & CTO Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 2 / 29 Outline 1Preface 2Software as knowledge 3Software Heritage 4Research highlights in empirical software engineering 5Conclusion Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 3 / 29 Software is dual-form knowledge “The source code for a work means the preferred form of the work for making modifications to it." GPL Licence Hello World Program (excerpt of binary) 4004e6: 55 4004e7: 48 89 e5 4004ea: bf 84 05 40 00 4004ef: b8 00 00 00 00 4004f4: e8 c7 fe ff ff 4004f9: 90 4004fa: 5d 4004fb: c3 Program (source code) /* Hello World program */ #include<stdio.h> void main() { printf("Hello World"); } Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 3 / 29 Software is dual-form knowledge “The source code for a work means the preferred form of the work for making modifications to it." GPL Licence Hello World Program (excerpt of binary) 4004e6: 55 4004e7: 48 89 e5 4004ea: bf 84 05 40 00 4004ef: b8 00 00 00 00 4004f4: e8 c7 fe ff ff 4004f9: 90 4004fa: 5d 4004fb: c3 Program (source code) /* Hello World program */ #include<stdio.h> void main() { printf("Hello World"); } Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 3 / 29 Software is dual-form knowledge “The source code for a work means the preferred form of the work for making modifications to it." GPL Licence Hello World Program (excerpt of binary) 4004e6: 55 4004e7: 48 89 e5 4004ea: bf 84 05 40 00 4004ef: b8 00 00 00 00 4004f4: e8 c7 fe ff ff 4004f9: 90 4004fa: 5d 4004fb: c3 Program (source code) /* Hello World program */ #include<stdio.h> void main() { printf("Hello World"); } Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 3 / 29 Software is dual-form knowledge “The source code for a work means the preferred form of the work for making modifications to it." GPL Licence Hello World Program (excerpt of binary) 4004e6: 55 4004e7: 48 89 e5 4004ea: bf 84 05 40 00 4004ef: b8 00 00 00 00 4004f4: e8 c7 fe ff ff 4004f9: 90 4004fa: 5d 4004fb: c3 Program (source code) /* Hello World program */ #include<stdio.h> void main() { printf("Hello World"); } Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 3 / 29 Software source code is precious human knowledge Harold Abelson, Structure and Interpretation of Computer Programs (1st ed.) 1985 “Programs must be written for people to read, and only incidentally for machines to execute.” Apollo 11 source code (excerpt)Quake III source code ( excerpt ) Len Shustek, Computer History Museum 2006 “Source code provides a view into the mind of the designer.” Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 4 / 29 Software Heritage in a nutshell www.softwareheritage.org THE GREAT LIBRA RY OF SO URCE CO D E Collect, preserve and share all software source code Preserving our heritage, enabling better software and better science for all Reference catalog find and reference all software source code Universal archive preserve and share all software source code Research infrastructure enable analysis of all software source code Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 6 / 29 Software Heritage in a nutshell www.softwareheritage.org THE GREAT LIBRA RY OF SO URCE CO D E Collect, preserve and share all software source code Preserving our heritage, enabling better software and better science for all Reference catalog find and reference all software source code Universal archive preserve and share all software source code Research infrastructure enable analysis of all software source code Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 6 / 29 Software Heritage in a nutshell www.softwareheritage.org THE GREAT LIBRA RY OF SO URCE CO D E Collect, preserve and share all software source code Preserving our heritage, enabling better software and better science for all Reference catalog find and reference all software source code Universal archive preserve and share all software source code Research infrastructure enable analysis of all software source code Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 6 / 29 Software Heritage in a nutshell www.softwareheritage.org THE GREAT LIBRA RY OF SO URCE CO D E Collect, preserve and share all software source code Preserving our heritage, enabling better software and better science for all Reference catalog find and reference all software source code Universal archive preserve and share all software source code Research infrastructure enable analysis of all software source code Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 6 / 29 Archiving goals Targets: VCS repositories & source code releases (e.g., tarballs, packages) We DO archive file content (= blobs) revisions (= commits), with full metadata releases (= tags), ditto where (origin) & when (visit) we found any of the above ... in a VCS-/archive-agnostic canonical data model We DON’T archive (yet) homepages, wikis BTS/issues/code reviews/etc. mailing lists Long term vision: play our part in a "semantic wikipedia of software" Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 7 / 29 Data flow deb deb hg hg hg git git git git svn svn svn pypi pypi software origins Package repos Software Heritage Archive Forges GitHub lister GitLab lister Debian lister Git loader Mercurial loader Debian source package loader PyPI lister PyPI loader Merkle DAG + blob storage . . . . . . Distros ... Scheduling Listing (full/incremental) Loading & deduplication Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 8 / 29 Merkle trees Merkle tree (R. C. Merkle, CRYPTO 1987) Combination of tree hash function Classical cryptographic construction fast, parallel signature of large data structures widely used (e.g., Git, blockchains, IPFS, ...) built-in deduplication Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 9 / 29 Merkle trees Merkle tree (R. C. Merkle, CRYPTO 1987) Combination of tree hash function Classical cryptographic construction fast, parallel signature of large data structures widely used (e.g., Git, blockchains, IPFS, ...) built-in deduplication Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 9 / 29 Data model Origin + url: str branches Snapshot * * Release directory Revision * ** * Merkle DAG target 1 entries parents snapshots + id: sha1 Directory entries + id: sha1 + id: sha1 + author: str + name: str + message: str + timestamp: datetime + id: sha1 + author: str + message: str + timestamp: datetime Content + id:sha1 Aglobal graph linking together fully deduplicated source code artifact (files, commits, directories, releases, etc.) to the places that distribute them (e.g., Git repositories), providing a unified view on the entire Software Commons. Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 10 / 29 The archive: a (giant) Merkle DAG origin visit https://forge.softwareheritage.org/source/helloworld.git snapshot 1 timestamp Fri Feb 9 12:38:45 2018 +0100 0861db5e… <<Revision>> a3ee21ad… +author = "Stefano Zacchiroli <zack@…>" +message = "add build toolchain …" +timestamp = Thu Feb 8 10:49:29 2018 +0100 +directory: Directory +parents: Revision list <<Directory>> b94a90cd… +entries ".gitignore" "Makefile" "hello.c" <<Content>> 225ae01b… +data = "all: hello\n\n…" <<Revision>> 43ef7dcd… +author = "Stefano Zacchiroli <zack@…>" +message = "add licensing information and README" +timestamp = Thu Feb 8 10:54:09 2018 +0100 +directory: Directory +parents: Revision list <<Directory>> fa8c0908… +entries ".gitignore" "COPYING" "Makefile" "README.md" "hello.c" <<Content>> a1afd006… +data = "…Yet another…" <<Release>> edf82f21… +author = "Stefano Zacchiroli <zack@…>" +name = "1.0" +message = "1.0 release" +timestamp = Thu Feb 8 15:51:00 2018 +0100 +target <<Snapshot>> 0861db5e… +branches HEAD refs/heads/master refs/tags/1.0 Archive content after visit 1 <<Content>> c839dea9… +data = "#include …" <<Directory>> 6ca2e444… +entries "hello.c" <<Revision>> 1886826f… +author = "Stefano Zacchiroli <zack@…>" +message = "implement a trivial …" +timestamp = Thu Feb 8 10:44:35 2018 +0100 +directory: Directory +parents: Revision list = None Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 11 / 29 Graph dataset Use case: large scale analyses of the most comprehensive corpus on the development history of free/open source software. Antoine Pietri, Diomidis Spinellis, Stefano Zacchiroli The Software Heritage Graph Dataset: Public software development under one roof MSR 2019: 16th Intl. Conf. on Mining Software Repositories. IEEE preprint: http://deb.li/swhmsr19 Dataset Relational representation of the full graph as a set of tables Available as open data: docs.softwareheritage.org/devel/swh-dataset/graph Chosen as subject for the MSR 2020 Mining Challenge Formats Local use: set of Apache ORC files (10+ TiB in total) Live usage: Amazon Athena (SQL-queriable), Azure Data Lake Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 14 / 29 Graph dataset — example SELECT COUNT(*) AS c, word FROM ( SELECT LOWER(REGEXP_EXTRACT(FROM_UTF8( message), ’^\w+’)) AS word FROM revision) WHERE word != ’’ GROUP BY word ORDER BY COUNT(*) DESC LIMIT 5; Count Word 71 338 310 update 64 980 346 merge 56 854 372 add 44 971 954 added 33 222 056 fix Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 15 / 29 Graph dataset — example SELECT COUNT(*) AS c, word FROM ( SELECT LOWER(REGEXP_EXTRACT(FROM_UTF8( message), ’^\w+’)) AS word FROM revision) WHERE word != ’’ GROUP BY word ORDER BY COUNT(*) DESC LIMIT 5; Count Word 71 338 310 update 64 980 346 merge 56 854 372 add 44 971 954 added 33 222 056 fix Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 15 / 29 Graph dataset — example Fork arity i.e., how often is a commit based upon? SELECT fork_deg, count(*) FROM ( SELECT id, count(*) AS fork_deg FROM revision_history GROUP BY id) t GROUP BY fork_deg ORDER BY fork_deg; 0.1 1 10 100 1000 10000 100000 1x106 1x107 1x108 1x109 1x1010 0.1 1 10 100 1000 10000 100000 1x106 1x107 Number of nodes Degree Merge arity i.e., how large are merges? SELECT merge_deg, COUNT(*) FROM ( SELECT parent_id, COUNT(*) AS merge_deg FROM revision_history GROUP BY parent_id) t GROUP BY merge_deg ORDER BY merge_deg; 0.1 1 10 100 1000 10000 100000 1x106 1x107 1x108 1x109 1x1010 0.1 1 10 100 1000 10000 100000 1x106 1x107 Number of nodes Degree Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 16 / 29 Graph dataset — example Fork arity i.e., how often is a commit based upon? SELECT fork_deg, count(*) FROM ( SELECT id, count(*) AS fork_deg FROM revision_history GROUP BY id) t GROUP BY fork_deg ORDER BY fork_deg; 0.1 1 10 100 1000 10000 100000 1x106 1x107 1x108 1x109 1x1010 0.1 1 10 100 1000 10000 100000 1x106 1x107 Number of nodes Degree Merge arity i.e., how large are merges? SELECT merge_deg, COUNT(*) FROM ( SELECT parent_id, COUNT(*) AS merge_deg FROM revision_history GROUP BY parent_id) t GROUP BY merge_deg ORDER BY merge_deg; 0.1 1 10 100 1000 10000 100000 1x106 1x107 1x108 1x109 1x1010 0.1 1 10 100 1000 10000 100000 1x106 1x107 Number of nodes Degree Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 16 / 29 License dataset Stefano Zacchiroli A Large-scale Dataset of (Open Source) License Text Variants MSR 2022 (best dataset paper) + Empir. Soft. Eng. 28(6): 147 (2023) preprint: https://arxiv.org/abs/2308.11258 Dataset 6.9 million unique full texts of FOSS license variants Detected using filename patterns across the entire SWH archive LICENSE,COPYRIGHT,NOTICE, etc. Metadata: file lengths measures, detected MIME type, detected SPDX license (via ScanCode), example origin repository, oldest public commit of origin, ground truth Use cases Empirical studies on FOSS licensing, including phylogenetics Training of automated license classifiers NLP analyses of legal texts Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 17 / 29 The Software Heritage Filesystem (SwhFS) The Software Heritage Filesystem (SwhFS) is a user-space POSIX filesystem that enables browsing parts of the Software Heritage archive as if it were locally available. Code: forge.softwareheritage.org/source/swh-fuse Documentation: docs.softwareheritage.org/devel/swh-fuse Thibault Allançon, Antoine Pietri, Stefano Zacchiroli The Software Heritage Filesystem (SwhFS): Integrating Source Code Archival with Development ICSE 2021 (Tool track): The 43rd Intl. Conference on Software Engineering https://arxiv.org/abs/2102.06390 Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 18 / 29 The Software Heritage Filesystem (SwhFS) — example $ mkdir swhfs $ swh fs mount swhfs/ # mount the archive $ cd swhfs/ $ cat archive/swh:1:cnt:c839dea9e8e6f0528b468214348fee8669b305b2 #include <stdio.h> int main(void) { printf("Hello, World!\n"); } $ cd archive/swh:1:dir:1fee702c7e6d14395bbf5ac3598e73bcbf97b030 $ ls | wc -l 127 $ grep -i antenna THE_LUNAR_LANDING.s | cut -f 5 # IS THE LR ANTENNA IN POSITION 1 YET # BRANCH IF ANTENNA ALREADY IN POSITION 1 Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 19 / 29 The Software Heritage Filesystem (SwhFS) — example (cont.) $ cd archive/swh:1:rev:9d76c0b163675505d1a901e5fe5249a2c55609bc $ ls -F history/ meta.json@ parent@ parents/ root@ $ jq ’.author.name, .date, .message’ meta.json "Michal Golebiowski-Owczarek" "2020-03-02T23:02:42+01:00" "Data:Event:Manipulation: Prevent collisions with Object.prototype ..." $ find root/src/ -type f -name ’*.js’ | xargs cat | wc -l 10136 Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 20 / 29 Graph compression Paolo Boldi, Antoine Pietri, Sebastiano Vigna, Stefano Zacchiroli Ultra-Large-Scale Repository Analysis via Graph Compression SANER 2020, 27th Intl. Conf. on Software Analysis, Evolution and Reengineering. IEEE Research question Is it possible to efficiently perform software development history analyses at the scale of Software Heritage archive on a single, relatively cheap machine? Idea Apply state-of-the-art graph compression techniques from the field of Web graph / social network analysis. Results The entire archive graph (35 B nodes, 500 B edges) can be loaded in 300 GiB and then traversed at the cost of tens of ns per edge (= a few hours for a full single-thread visit). Java and gRPC APIs available: docs.softwareheritage.org/devel/swh-graph/grpc-api.html Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 21 / 29 ... and more (open research leads) Cybersecurity SWHSec (CampusCyber project) — main question: how can we leverage Software Heritage as a knowledge base to increase the security of open source software? AI Project classification at the scale (size and heterogeneity) of Software Heritage (ongoing work with LabRI+UniBo) Code Commons: producing research datasets for ethical training of LLMs on Software Heritage code 1giving back to humanity 2precisely identify training inputs 3opt-out Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 28 / 29 ... and more (open research leads) Cybersecurity SWHSec (CampusCyber project) — main question: how can we leverage Software Heritage as a knowledge base to increase the security of open source software? AI Project classification at the scale (size and heterogeneity) of Software Heritage (ongoing work with LabRI+UniBo) Code Commons: producing research datasets for ethical training of LLMs on Software Heritage code 1giving back to humanity 2precisely identify training inputs 3opt-out Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 28 / 29 ... and more (open research leads) Cybersecurity SWHSec (CampusCyber project) — main question: how can we leverage Software Heritage as a knowledge base to increase the security of open source software? AI Project classification at the scale (size and heterogeneity) of Software Heritage (ongoing work with LabRI+UniBo) Code Commons: producing research datasets for ethical training of LLMs on Software Heritage code 1giving back to humanity 2precisely identify training inputs 3opt-out Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 28 / 29 Outline 1Preface 2Software as knowledge 3Software Heritage 4Research highlights in empirical software engineering 5Conclusion Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 29 / 29 Conclusion Software Heritage archives public code and its history as a huge Merkle DAG Analyzing it at scale (35/500 B nodes/edges) is a significant big data undertaking Gold mine of research leads in and around empirical software engineering Research on Software Heritage www.softwareheritage.org/publications Questions? Ask me! Contact Stefano Zacchiroli /stefano[email protected] /@[email protected] Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 29 / 29 Appendix Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 1 / 3 Software Heritage Identifiers (SWHIDs) (full spec) An emerging standard in Linux Foundation’s SPDX 2.2 IANA-registered "swh:" URI prefix WikiData property P6138 Examples Apollo 11 AGC excerpt Quake III rsqrt Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 2 / 3 Software Heritage Identifiers (SWHIDs) (full spec) An emerging standard in Linux Foundation’s SPDX 2.2 IANA-registered "swh:" URI prefix WikiData property P6138 Examples Apollo 11 AGC excerpt Quake III rsqrt Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 2 / 3 Software Heritage Identifiers (SWHIDs) (full spec) An emerging standard in Linux Foundation’s SPDX 2.2 IANA-registered "swh:" URI prefix WikiData property P6138 Examples Apollo 11 AGC excerpt Quake III rsqrt Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 2 / 3 Software Heritage Identifiers (SWHIDs) (full spec) An emerging standard in Linux Foundation’s SPDX 2.2 IANA-registered "swh:" URI prefix WikiData property P6138 Examples Apollo 11 AGC excerpt Quake III rsqrt Stefano Zacchiroli Software Heritage 2024-01-23, LaBRI 2 / 3