Software Heritage A research platform for large-scale source code archival

Software Heritage A research platform for large-scale source code archival Stefano Zacchiroli Télécom Paris — [email protected], @zacchiro 19 Oct 2021 IDIA Prototype/Software/Platform Day Télécom Paris THE GREAT LIBRARY OF SOURC E CODE Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 1 / 18 About me Professor, Télécom Paris, ACES team (newbie!) Free/Open Source Software activist (20+ years) Debian Developer & Former 3x Debian Project Leader Former Open Source Initiative (OSI) director Software Heritage co-founder & CTO Research interests Software engineering, Free/Open Source Software (FOSS) Digital commons Computer security Software supply chain Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 2 / 18 Outline 1Why we must preserve the history of software source code 2How we can preserve our software heritage 3Demo time! 4Scientific challenges 5Preserving our software commons: the present and the future Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 3 / 18 Software source code is precious knowledge Harold Abelson, Structure and Interpretation of Computer Programs (1st ed.) 1985 “Programs must be written for people to read, and only incidentally for machines to execute.” Apollo 11 source code (excerpt) Quake III source code ( excerpt ) Len Shustek, Computer History Museum 2006 “Source code provides a view into the mind of the designer.” Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 3 / 18 Software source code is precious knowledge Harold Abelson, Structure and Interpretation of Computer Programs (1st ed.) 1985 “Programs must be written for people to read, and only incidentally for machines to execute.” Apollo 11 source code (excerpt) Quake III source code ( excerpt ) Len Shustek, Computer History Museum 2006 “Source code provides a view into the mind of the designer.” Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 3 / 18 Software source code is precious knowledge Harold Abelson, Structure and Interpretation of Computer Programs (1st ed.) 1985 “Programs must be written for people to read, and only incidentally for machines to execute.” Apollo 11 source code (excerpt) Quake III source code ( excerpt ) Len Shustek, Computer History Museum 2006 “Source code provides a view into the mind of the designer.” Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 3 / 18 Software source code is precious knowledge Harold Abelson, Structure and Interpretation of Computer Programs (1st ed.) 1985 “Programs must be written for people to read, and only incidentally for machines to execute.” Apollo 11 source code (excerpt) Quake III source code ( excerpt ) Len Shustek, Computer History Museum 2006 “Source code provides a view into the mind of the designer.” Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 3 / 18 Calling for source code preservation: UNESCO UNESCO, Inria, Software Heritage invite 40 international experts meet in Paris ... The call is published on Feb 2019 “[We call to] support efforts to gather and preserve the artifacts and narratives of the history of computing, while the earlier creators are still alive” https://en.unesco.org/foss/paris-call-software-source-code Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 4 / 18 Calling for source code preservation: UNESCO UNESCO, Inria, Software Heritage invite 40 international experts meet in Paris ... The call is published on Feb 2019 “[We call to] support efforts to gather and preserve the artifacts and narratives of the history of computing, while the earlier creators are still alive” https://en.unesco.org/foss/paris-call-software-source-code Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 4 / 18 Source code history — for security and transparency Where does reused software come from? Do you know where it comes from? the software you ship the software you use the software you acquire the software that has that bug has that vulnerability KYSW: Know Your SoftWare Like KYC in banking, KYSW is now essential all over IT... Sec. 4. Enhancing Software Supply Chain Security ensuring and attesting, to the extent practicable, to the integrity and provenance of open source software May 2021 POTUS Executive Order Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 6 / 18 Source code history — for security and transparency Where does reused software come from? Do you know where it comes from? the software you ship the software you use the software you acquire the software that has that bug has that vulnerability KYSW: Know Your SoftWare Like KYC in banking, KYSW is now essential all over IT... Sec. 4. Enhancing Software Supply Chain Security ensuring and attesting, to the extent practicable, to the integrity and provenance of open source software May 2021 POTUS Executive Order Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 6 / 18 Source code history — for security and transparency Where does reused software come from? Do you know where it comes from? the software you ship the software you use the software you acquire the software that has that bug has that vulnerability KYSW: Know Your SoftWare Like KYC in banking, KYSW is now essential all over IT... Sec. 4. Enhancing Software Supply Chain Security ensuring and attesting, to the extent practicable, to the integrity and provenance of open source software May 2021 POTUS Executive Order Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 6 / 18 Fragile Like all digital information, FOSS is fragile link rot: projects are created, moved around, removed business-driven code loss (e.g., Gitorious, Google Code, Bitbucket) data rot: physical media with legacy software decay If a website disappears you go to the Internet Archive... where do you go if (a repository on) GitHub or GitLab goes away? Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 7 / 18 Fragile Like all digital information, FOSS is fragile link rot: projects are created, moved around, removed business-driven code loss (e.g., Gitorious, Google Code, Bitbucket) data rot: physical media with legacy software decay If a website disappears you go to the Internet Archive... where do you go if (a repository on) GitHub or GitLab goes away? Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 7 / 18 Outline 1Why we must preserve the history of software source code 2How we can preserve our software heritage 3Demo time! 4Scientific challenges 5Preserving our software commons: the present and the future Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 8 / 18 Software Heritage in a nutshell www.softwareheritage.org THE GREAT LIBRA RY OF SO URCE CO D E Collect, preserve and share all software source code Preserving our heritage, enabling better software and better science for all Reference catalog find and reference all software source code Universal archive preserve all software source code Research infrastructure enable analysis of all software source code Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 8 / 18 Software Heritage in a nutshell www.softwareheritage.org THE GREAT LIBRA RY OF SO URCE CO D E Collect, preserve and share all software source code Preserving our heritage, enabling better software and better science for all Reference catalog find and reference all software source code Universal archive preserve all software source code Research infrastructure enable analysis of all software source code Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 8 / 18 Software Heritage in a nutshell www.softwareheritage.org THE GREAT LIBRA RY OF SO URCE CO D E Collect, preserve and share all software source code Preserving our heritage, enabling better software and better science for all Reference catalog find and reference all software source code Universal archive preserve all software source code Research infrastructure enable analysis of all software source code Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 8 / 18 Software Heritage in a nutshell www.softwareheritage.org THE GREAT LIBRA RY OF SO URCE CO D E Collect, preserve and share all software source code Preserving our heritage, enabling better software and better science for all Reference catalog find and reference all software source code Universal archive preserve all software source code Research infrastructure enable analysis of all software source code Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 8 / 18 Software Heritage intrinsic Identifiers (SWHID) (full spec) An emerging standard in Linux Foundation’s SPDX 2.2 IANA registered, WikiData property P6138 Examples: Apollo 11 AGC excerpt Quake III rsqrt Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 11 / 18 Software Heritage intrinsic Identifiers (SWHID) (full spec) An emerging standard in Linux Foundation’s SPDX 2.2 IANA registered, WikiData property P6138 Examples: Apollo 11 AGC excerpt Quake III rsqrt Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 11 / 18 Software Heritage intrinsic Identifiers (SWHID) (full spec) An emerging standard in Linux Foundation’s SPDX 2.2 IANA registered, WikiData property P6138 Examples: Apollo 11 AGC excerpt Quake III rsqrt Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 11 / 18 Outline 1Why we must preserve the history of software source code 2How we can preserve our software heritage 3Demo time! 4Scientific challenges 5Preserving our software commons: the present and the future Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 12 / 18 A walkthrough Browse the archive Trigger archival of your preferred software in a breeze Get and use SWHIDs (full spec available online) The Apollo 11 AGC source code example Cite software with the biblatex-software style from CTAN Example use in a research article: compare Fig. 1 and conclusions in the 2012 version in the updated version using SWHIDs and Software Heritage Example in a journal: an article from IPOL Curated deposit in SWH via HAL, see for example: LinBox, SLALOM, Givaro, NS2DDV, SumGra, Coq proof, ... Rescue landmark legacy software, see the SWHAP process with UNESCO Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 12 / 18 Outline 1Why we must preserve the history of software source code 2How we can preserve our software heritage 3Demo time! 4Scientific challenges 5Preserving our software commons: the present and the future Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 13 / 18 A revolutionary research infrastructure designed for source code The graph of Software Development Snapshots Releases Revisions Directories Contents All software development in a single graph ... The blockchain of Software Development ... a single Merkle graph! Apillar of Open Science Reference archive of Research Software Reference platform for Big Code Asingle, uniform data structure Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 13 / 18 A revolutionary research infrastructure designed for source code The graph of Software Development Snapshots Releases Revisions Directories Contents All software development in a single graph ... The blockchain of Software Development ... a single Merkle graph! Apillar of Open Science Reference archive of Research Software Reference platform for Big Code Asingle, uniform data structure Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 13 / 18 A revolutionary research infrastructure designed for source code The graph of Software Development Snapshots Releases Revisions Directories Contents All software development in a single graph ... The blockchain of Software Development ... a single Merkle graph! Apillar of Open Science Reference archive of Research Software Reference platform for Big Code Asingle, uniform data structure Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 13 / 18 A revolutionary research infrastructure designed for source code The graph of Software Development Snapshots Releases Revisions Directories Contents All software development in a single graph ... The blockchain of Software Development ... a single Merkle graph! Apillar of Open Science Reference archive of Research Software Reference platform for Big Code Asingle, uniform data structure Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 13 / 18 Focus on Academia: growing adoption (selection) HAL software curated deposit workflow Curated Archiving of Research Software Artifacts International Journal of Digical Curation, 2020 Reference archive for swmath.org See code links, e.g. SemiPar package IPOL (image processing) archive (deposit) reference BibLaTeX eLife (life sciences) archive (save code now) reference JTCAM (mechanics) instructions for authors biblatex-software in journal L A T EX class Policy: France National Plan for Open Science Policy: Europe EOSC SIRS report SWHIDs archive Guidelines 1Prepare your public repository README, AUTHORS & LICENSE files 2Save your code http://save.softwareheritage.org/ 3Reference your work (full repository, specific version or code fragment) summary ICMS 2020 Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 15 / 18 An international, non profit initiative built for the long term Sharing the vision And many more ... www.softwareheritage.org/support/testimonials Donors, members, sponsors Platinum sponsors Silver sponsors Bronze sponsors Gold sponsors Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 16 / 18 An international, non profit initiative built for the long term Sharing the vision And many more ... www.softwareheritage.org/support/testimonials Donors, members, sponsors Platinum sponsors Silver sponsors Bronze sponsors Gold sponsors Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 16 / 18 You may help! Foster the adoption of research best practices archive and reference relevant source code (save code now, and deposit) use Software Heritage and biblatex-software in articles, journals, and books rescue and preserve landmark legacy source code with SWHAP Engage with Software Heritage as a researcher use the archive for your own software-related experiments work with us to tackle open technical and research problems Engage with Software Heritage as an organization become a member/sponsor build a Software Heritage mirror contribute to the preservation mission Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 17 / 18 You may help! Foster the adoption of research best practices archive and reference relevant source code (save code now, and deposit) use Software Heritage and biblatex-software in articles, journals, and books rescue and preserve landmark legacy source code with SWHAP Engage with Software Heritage as a researcher use the archive for your own software-related experiments work with us to tackle open technical and research problems Engage with Software Heritage as an organization become a member/sponsor build a Software Heritage mirror contribute to the preservation mission Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 17 / 18 You may help! Foster the adoption of research best practices archive and reference relevant source code (save code now, and deposit) use Software Heritage and biblatex-software in articles, journals, and books rescue and preserve landmark legacy source code with SWHAP Engage with Software Heritage as a researcher use the archive for your own software-related experiments work with us to tackle open technical and research problems Engage with Software Heritage as an organization become a member/sponsor build a Software Heritage mirror contribute to the preservation mission Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 17 / 18 Thank you! Resources archive archive.softwareheritage.org stay posted softwareheritage.org/newsletter blog softwareheritage.org/blog References (selected; full list at softwareheritage.org/publications) Jean-François Abramatic, Roberto Di Cosmo, Stefano Zacchiroli Building the Universal Archive of Source Code Communication of the ACM, October 2018 Antoine Pietri, Diomidis Spinellis, Stefano Zacchiroli The Software Heritage Graph Dataset: Large-scale Analysis of Public Software Development History MSR 2020: 17th Intl. Conf. on Mining Software Repositories. IEEE Roberto Di Cosmo Archiving and Referencing Source Code with Software Heritage International Congress on Mathematical Software (ICMS), 2020 MESRI Second French Plan for Open Science www.ouvrirlascience.fr/second-national-plan-for-open-science, 2001 Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 18 / 18 Archiving goals Targets: VCS repositories & source code releases (e.g., tarballs, packages) We DO archive file content (= blobs) revisions (= commits), with full metadata releases (= tags), ditto where (origin) & when (visit) we found any of the above ... in a VCS-/archive-agnostic canonical data model We DON’T archive (yet) homepages, wikis BTS/issues/code reviews/etc. mailing lists Long term vision: play our part in a "semantic wikipedia of software" Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 1 / 3 Data model Origin + url: str branches Snapshot * * Release directory Revision * ** * Merkle DAG target 1 entries parents snapshots + id: sha1 Directory entries + id: sha1 + id: sha1 + author: str + name: str + message: str + timestamp: datetime + id: sha1 + author: str + message: str + timestamp: datetime Content + id:sha1 Aglobal graph linking together fully deduplicated source code artifact (files, commits, directories, releases, etc.) to the places that distribute them (e.g., Git repositories), providing a unified view on the entire Software Commons. Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 2 / 3 The archive: a (giant) Merkle DAG origin visit https://forge.softwareheritage.org/source/helloworld.git snapshot 1 timestamp Fri Feb 9 12:38:45 2018 +0100 0861db5e… <<Revision>> a3ee21ad… +author = "Stefano Zacchiroli <zack@…>" +message = "add build toolchain …" +timestamp = Thu Feb 8 10:49:29 2018 +0100 +directory: Directory +parents: Revision list <<Directory>> b94a90cd… +entries ".gitignore" "Makefile" "hello.c" <<Content>> 225ae01b… +data = "all: hello\n\n…" <<Revision>> 43ef7dcd… +author = "Stefano Zacchiroli <zack@…>" +message = "add licensing information and README" +timestamp = Thu Feb 8 10:54:09 2018 +0100 +directory: Directory +parents: Revision list <<Directory>> fa8c0908… +entries ".gitignore" "COPYING" "Makefile" "README.md" "hello.c" <<Content>> a1afd006… +data = "…Yet another…" <<Release>> edf82f21… +author = "Stefano Zacchiroli <zack@…>" +name = "1.0" +message = "1.0 release" +timestamp = Thu Feb 8 15:51:00 2018 +0100 +target <<Snapshot>> 0861db5e… +branches HEAD refs/heads/master refs/tags/1.0 Archive content after visit 1 <<Content>> c839dea9… +data = "#include …" <<Directory>> 6ca2e444… +entries "hello.c" <<Revision>> 1886826f… +author = "Stefano Zacchiroli <zack@…>" +message = "implement a trivial …" +timestamp = Thu Feb 8 10:44:35 2018 +0100 +directory: Directory +parents: Revision list = None Stefano Zacchiroli Software Heritage 2021-10-19, IDIA 3 / 3