Software Heritage: Building the Software Pillar of Open Science

Software Heritage Building the Software Pillar of Open Science Stefano Zacchiroli 4 Nov 2025 Open Access Week, Institut Polytechnique de Paris Palaiseau, France THE GREAT LIBRARY OF SOURC E CODE Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 1 / 25 Outline 1Introduction 2(Open) Source for Open Science 3Software Heritage in the big picture 4Demo time 5Much more than an archive 6Conclusion Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 2 / 25 About the speaker Professor of Computer Science, Télécom Paris, Polytechnic Institute of Paris Free/Open Source Software activist (20+ years) Debian Developer & Former 3x Debian Project Leader Former Open Source Initiative (OSI) director Software Heritage co-founder & chief scientific officer Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 2 / 25 Software Source Code is Precious Knowledge Harold Abelson, Structure and Interpretation of Computer Programs (1st ed.) 1985 “Programs must be written for people to read, and only incidentally for machines to execute.” Apollo 11 source code (excerpt)Covid Sim ( excerpt ) Len Shustek, Computer History Museum 2006 “Source code provides a view into the mind of the designer.” Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 3 / 25 (Open) Source Code comes from all over the world Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 4 / 25 (Open) Source Code grows at an exponential rate 1990 1995 2000 2005 2010 2015 2020 Year 105 106 107 108 109 Number of Objects Evolution of Number of Objects per Year Revisions Contents Revisions Fitted Growth Annual Growth: 27.30% Doubling Time: 2.87 years Contents Fitted Growth Annual Growth: 40.25% Doubling Time: 2.05 years Rousseau, Di Cosmo, Zacchiroli: Software provenance tracking at the scale of public source code. Empir. Softw. Eng. 25(4): 2930-2959 (2020) Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 5 / 25 Outline 1Introduction 2(Open) Source for Open Science 3Software Heritage in the big picture 4Demo time 5Much more than an archive 6Conclusion Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 6 / 25 Software Source Code: pillar of Open Science Software powers modern research Over 20% of articles using software across all disciplines share it 2024 French Open Science Monitor Key pillar: software Links are important Nota Bene software may be a tool, a research outcome and a research object access to the source code is essential! Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 6 / 25 Source code is special (software is not data) Software evolves over time projects may last decades the development history is key to its understanding Complexity millions of lines of code large web of dependencies easy to break, difficult to maintain research software a thin top layer sophisticated developer communities python3-matplotlib python3-dateutil python3-six (>= 1.4) python3:any python-matplotlib-data (>= 3.0.2-2) python3-pyparsing (>= 1.5.6) libjs-jquery libjs-jquery-ui python3-numpy (>= 1:1.14.3) python3 (<< 3.8) (>= 3.7~) python3-numpy-abi9 python3-cycler (>= 0.10.0) python3-kiwisolver libfreetype6 (>= 2.2.1) libpng16-16 (>= 1.6.2-1) python3-pil python3-tk (>= 1.5) (>= 3.2~) tzdata [python3] [python3] {debconf} debconf-2.0 (>= 0.5) [debconf] {cdebconf} fonts-lyx ttf-bitstream-vera (>= 3.3.2-2~) jquery javascript-common (>= 1.7) (<< 3.8)(>= 3.7~)python3.7:any libblas3 libblas.so.3 liblapack3 liblapack.so.3python3-pkg-resources python3-minimal (= 3.7.3-1) python3.7 (>= 3.7.3-1~) libpython3-stdlib (= 3.7.3-1) python3.7-minimal (>= 3.7.3-1~) {dpkg} install-info (>= 1.13.20) libpython3.7-minimal (= 3.7.3-2) libexpat1 (>= 2.1~beta3) libssl1.1 (>= 1.1.1) libpython3.7-stdlib (>= 0.5) (= 3.7.3-2) mime-support libbz2-1.0 liblzma5 (>= 5.1.1alpha+20120614) libdb5.3 libffi6 (>= 3.0.4) libmpdec2 libncursesw6 (>= 6) libtinfo6 (>= 6) libreadline7 (>= 7.0~beta) libsqlite3-0 (>= 3.7.15) libuuid1 (>= 2.20.1) bzip2file xz-utils (= 1.0.6-9) libmagic1 (= 1:5.35-4) libmagic-mgc (= 1:5.35-4) (>= 5.2.2)xz-lzma (= 6.1+20181013-2) libgpm2 (>= 6) readline-common (>= 1.15.4) libreadline-common (>= 1.16.1) uuid-runtime (>= 2.25-5~) (>= 2.31.1) adduserlibsmartcols1 (>= 2.27~rc1) libsystemd0 (>= 0.5) passwd (>= 5.1.1alpha+20120614) libgcrypt20 (>= 1.8.0) liblz4-1 (>= 0.0~r122) libgpg-error0 (>= 1.25) libgpg-error-l10n (= 3.7.3-2) (= 3.7.3-2) (>= 3.7.3-1~) [python3.7] [python3.7] libgfortran5 (>= 8) libquadmath0 (>= 4.6) ... -6gcc-9-base (= 9-20190428-1) (>= 4.6) (= 9-20190428-1) (>= 8) (>= 4.6) ... -3- (>= 3.3.2-2~) (<< 3.8)(>= 3.6~) (>= 1.6.2-1) (<< 3.8) (>= 3.7~) (>= 2.2.1) [mime-support] python3-pil.imagetk libimagequant0 (>= 2.11.10) libjpeg62-turbo (>= 1.3.1) liblcms2-2 (>= 2.2+git20110628) libtiff5 (>= 4.0.3) libwebp6 (>= 0.5.1) libwebpdemux2 (>= 0.5.1) libwebpmux3 (>= 0.6.1-2) python3-olefile (<< 3.8)(>= 3.7~) (= 6.0.0-1) (>= 3.4.1-2) (>= 3.7.1-1~)(<< 3.9) blt (>= 2.4z-9) tk8.6-blt2.5 (>= 2.5.3) libtcl8.6 (>= 8.6.0) libtk8.6 (>= 8.6.0)(= 2.5.3+dfsg-5) (>= 8.6.0) (>= 8.6.0) blt4.2blt8.0 blt8.0-unoff (>= 2.2.1) (>= 8.6.0-2) libfontconfig1 (>= 2.12.6) libxext6 libxft2 (>> 2.1.1) libxss1 (>= 2.3.5)(>= 2.12.6) libxrender1 x11-common libjpeg62 (>= 5.1.1alpha+20120614) (>= 1.3.1) libjbig0 (>= 2.0) (>= 0.5.1) libzstd1 (>= 1.3.2) (>= 0.5.1)(>= 0.5.1) Matplotlib library Python dependencies Real dependencies Fake OS dependencies induced by package granularity Precious, endangered executable and human readable knowledge key people passing away, platforms (GoogleCode, Gitorious, etc.) closing down ... no organised effort to catalog and archive it Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 7 / 25 Software Heritage in a nutshell www.softwareheritage.org Unveiled in 2016 THE GREAT LIBRA RY OF SO URCE CO D E Collect, preserve and share all software source code Preserving our heritage, enabling better software and better science for all Reference catalog find and reference all software source code Universal archive preserve and share all software source code Research infrastructure enable analysis of all software source code Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 13 / 25 Auniversal software archive, as a shared infrastructure One infrastructure open and shared The largest archive ever built Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 14 / 25 An international, non profit initiative built for the long term Sharing the vision And many more ... www.softwareheritage.org/support/testimonials Donors, members, sponsors Platinum sponsors Gold sponsors Silver sponsors Bronze sponsors Diamond sponsors Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 15 / 25 The archive under the hood Git loader Mercurial loader Debian source package loader pypi source package loader . . . Software Heritage Archive Merkle DAG + blob storage Loading & deduplication dsc dsc hg hg hg git git git git svn svn svn tar zip software origins Package repos Forges GitHub lister GitLab lister Debian lister PyPi lister . . . Distros ... Scheduling Listing (full/incremental) tar origins snapshots releases revisionsrevisions directoriesdirectories contents Global development history permanently archived inauniform data model over 24 billion unique source files from over 375 million software projects ~2PB (compressed) blobs, ~50 B nodes, ~900 B edges Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 16 / 25 The Software Hash persistent identifier (SWHID) Software Hash Identifiers (SWHID) see swhid.org 50+B intrinsic, decentralised, cryptographically strong identifiers, SWHIDs In SPDX 2.2; IANA "swh:"; WikiData P6138; standard ISO 18670 Full fledged source code references for traceability, integrity and reproducibility Examples: Apollo 11 AGC,Quake III rsqrt; Guidelines available: HOWTO and ICMS 2020 Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 17 / 25 Outline 1Introduction 2(Open) Source for Open Science 3Software Heritage in the big picture 4Demo time 5Much more than an archive 6Conclusion Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 18 / 25 A walkthrough Browse (e.g. Apollo 11 [excerpt], your work may be already there !) Trigger archival, use the updateswh browser extension, configure the webhooks Get and use SWHIDs (full specification available online) Cite software with biblatex-software package from CTAN Overleaf ACMART template available Example in journals: article from IPOL Example with Parmap: devel on Github,archive in SWH,curated deposit in HAL Extracting all the software products for Inria,for CNRS,for CNES,for LIRMM or for Rémi Gribonval using HalTools Curated deposit in SWH via HAL, see for example: LinBox,SLALOM,Givaro, NS2DDV,SumGra,Coq proof, ... Example use in research articles: compare Fig. 1 and conclusions in the 2012 version and the updated version SWHID in a replication experiment Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 18 / 25 Addressing ARDC: an example is worth 1000 words Software metadata: codemeta.json example from Parmap, created with Codemeta generator software citation (see detailed HOWTO) and biblatex-software Integration with the HAL national french open access archive Curated deposit: metadata quality due to moderation examples: LinBox,SLALOM,Givaro,NS2DDV,SumGra,Coq proof, ... generation of reports, cv, web pages: for Inria,for CNRS,for CNES,for LIRMM or for Rémi Gribonval using HalTools Software Heritage + a curated metadata repository allows to address all needs ... researcher, engineer: archival, reference, credit, CV etc. with a little effort from them labs, organizations: track and report software production, curated catalog Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 19 / 25 Selected adoption indicators: users From Melissa Harrison’s OSEC 2022 talk Use on replicabilitystamp.org HAL+SWH in the Open Science software booklet Funding agencies recommendations ANR 2023 guidelines (p. 17) Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 20 / 25 Selected adoption indicators: infrastructures EOSC SIRS report: Software Source Code and Open Science, 2020 Connect scholarly ecosystem with the whole software ecosystem See e.g. the French public administration open source catalog Ongoing work: FAIRCORE4EOSC A full workpackage: connectors with InvenioRDM (Zenodo), episcience, Dagstuhl, swMath, etc. Software Heritage mirror for the European Open Science Cloud (EOSC) standardisation of CodeMeta and SWHID Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 21 / 25 Appendix Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 1 / 10 Outline 7Hands-on examples 8Background on key concepts on identifiers 9Extrinsic identifiers are fragile Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 1 / 10 Get Swhid for a code snippet in the archive video Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 1 / 10 Spice up publications with Swhid Parmap_code_swh-poster.png video Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 2 / 10 Get a citation for an archived code video Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 3 / 10 Use the updateswh browser extension video Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 4 / 10 Outline 7Hands-on examples 8Background on key concepts on identifiers 9Extrinsic identifiers are fragile Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 5 / 10 Identification vs Location an example is worth a thousand words Identification of a book Goal: identify a book one ISBN number per published book ISO 2108 standard specification Location of (a copy of) a book Goal: find (a copy of) a book many locations (locations can change!) many approaches for call numbers we are interested in identification, not in location Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 5 / 10 Extrinsic vs Intrinsic identifiers In a nutshell (for more info see this dedicated blog post) Main difference: how the relation between identifier and designated object is created and maintained. Persistence is a key desired property. Extrinsic Intrinsic relation register convention persistence external1internal pre-internet passport number, Music/Chemistry notations ISBN, SSN, etc. e.g. NaCl is table salt internet era DOI, Handle, Ark, etc. cryptographic hashes e.g.: git, bitcoin, SWHID Software development has massively adopted intrinsic identifiers since 2006 Git, GitHub, GitLab, pull requests, Guix, Nix, etc. all rely on intrinsic identifiers replicability/reproducibility need intrinsic identifiers SWHID is directly compatible with git! 1"persistence... is a function of administrative care" RFC 3650 (Handle System Overview, 2003) Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 6 / 10 Choosing identifiers: a matter of granularity 10.15497/RDA00053 Top concept layers vs. bottom artifact layers Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 7 / 10 SWHID computation: a worked example Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 8 / 10 SWHID computation: a worked example Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 8 / 10 SWHID computation: a worked example Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 8 / 10 SWHID computation: a worked example Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 8 / 10 SWHID computation: a worked example Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 8 / 10 SWHID computation: a worked example Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 8 / 10 SWHID computation: a worked example Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 8 / 10 Outline 7Hands-on examples 8Background on key concepts on identifiers 9Extrinsic identifiers are fragile Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 9 / 10 Disruption of the web of reference Web links are not permanent (even permalinks) there is no general guarantee that a URL... which at one time points to a given object continues to do so T. Berners-Lee et al. Uniform Resource Locators. RFC 1738. URLs used in articles decay! Analysis of IEEE Computer (Computer), and the Communications of the ACM (CACM): 1995-1999 the half-life of a referenced URL is approximately 4 years from its publication date. D. Spinellis. The Decay and Failures of URL References. Communications of the ACM, 46(1):71-77, January 2003. Similar findings in Lawrence, S. et al. Persistence of Web References in Scientific Research, IEEE Computer, 34(2), pp. 26–31, 2001. Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 9 / 10 The Digital Object Identifier (DOI) Example: https://doi.org/10.1109/MSR.2015.10 to find what 10.1109/MSR.2015.10 is, go to a resolver (e.g. doi.org) this returns http://ieeexplore.ieee.org/ document/7180064/ at this URL we find ... Architecture of the DOI infrastructure DOI resolution can change content at URL can change no intrinsic way of noticing persistence based on good will of multiple parties Stefano Zacchiroli Software Heritage Nov 2025, OA week IP Paris 10 / 10