Building Blocks for a Safer Open Source Supply Chain Reproducible Builds and Software Heritage

Building Blocks for a Safer Open Source Supply Chain Reproducible Builds and Software Heritage Stefano Zacchiroli Software Heritage Télécom Paris, Polytechnic Institute of Paris 29 May 2024 INFORTECH Day 2024, Université de Mons Mons, Belgium THE GREAT LIBRARY OF SOURCE CODE Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 1 / 30 Outline 1Introduction 2Open Source Software Supply Chain — Attacks 3Reproducible Builds 4Open Source Software Supply Chain — KYSW 5Software Heritage 6Conclusion Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 2 / 30 About the speaker Professor of Computer Science, Télécom Paris, Polytechnic Institute of Paris Free/Open Source Software activist (20+ years) Debian Developer & Former 3x Debian Project Leader Former Open Source Initiative (OSI) director Software Heritage co-founder & CTO Reproducible Builds board member Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 2 / 30 Outline 1Introduction 2Open Source Software Supply Chain — Attacks 3Reproducible Builds 4Open Source Software Supply Chain — KYSW 5Software Heritage 6Conclusion Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 3 / 30 The software supply chain Supply chain: the set of activities required by an organization to deliver goods or services to consumers. Software supply chain: the set of software components and software services required to deliver an IT product or service to users. libraries, runtimes, and other software component dependencies base system (operating system, package manager, compiler, ...) development tools and platform (e.g., IDEs, build system, GitHub/GitLab, CI/CD, ...) etc. Key artifact for audits: SBOM = Software Bill of Materials Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 3 / 30 (An) open source development workflow Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 4 / 30 Supply chain attacks Asoftware supply chain attack is a particular kind of cyber-attack that aims at injecting malicious code into an otherwise legitimate software product. Notable examples NotPetya (2017): ransomware concealed in an update of a popular accounting software, hitting Ukranian banks and major corps (B$) CCleaner (2017): malicious version of a popular MS Windows maintenance tool, distributed via the vendor website SolarWinds (2020): malicious update of the SolarWinds Orion monitoring software, shipping a delayed-activation trojan. Breached into several US Gov. branches as well as Microsoft XZ (2024): "Jia Tan" social engineers their way into becoming maintainer of XZ and plant a backdoor targeting SSH, allowing remote command execution Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 5 / 30 Open source supply chain attacks Is this specific to Free/Open Source Software (FOSS)? No. But modern FOSS package ecosystems are heavily intertwined. Examples: NPM (JavaScript), PyPI (Python), Crates (Rust), Gems (Ruby), etc. 100/10k/1M packages, depending on each other due to code reuse opportunities. Reverse transitive dependencies grow fast. A single package could be required by thousands of others. Example: removing left-pad, a 8-line(!) library to align strings, from NPM broke "many thousands of projects" in 2016, including high-profile ones from Big Tech. For an attacker, code injection into (transitively) popular leaf packages has a low opportunity cost. Also, entirely open FOSS package ecosystems (!= Linux distros) can be easy to infiltrate. Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 6 / 30 Attack tree — Injection (image from: Ohm et al. Backstabber’s Knife Collection: A Review of Open Source Software Supply Chain Attacks. DIMVA 2020) Attacker’s goal: package P containing malicious code is available from download from a distribution platform and P is a reverse transitive dependency of a legitimate package. Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 7 / 30 Build reproducibility in the small How hard could it be to ensure build reproducibility? After controlling for source code, build deps., and toolchain, two main classes of issues arise in practice: 1Uncontrolled build inputs: when toolchains allow the build process to be affected by the surrounding environment. Intuition: this is the build engineering equivalent of breaking encapsulation in programming 2Build non-determinism that gets encoded in final built artifacts. Let’s see some real-world examples... Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 12 / 30 Build reproducibility in the small How hard could it be to ensure build reproducibility? After controlling for source code, build deps., and toolchain, two main classes of issues arise in practice: 1Uncontrolled build inputs: when toolchains allow the build process to be affected by the surrounding environment. Intuition: this is the build engineering equivalent of breaking encapsulation in programming 2Build non-determinism that gets encoded in final built artifacts. Let’s see some real-world examples... Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 12 / 30 Build paths fprintf (stderr, "DEBUG: boop (%s:%s\n", __FILE__, __LINE__); The __FILE__ C preprocessor macro "expands to the name of the current input file". This results in non reproducibility when the program is built from different directories, e.g., /home/lamby/tmp vs. /home/zack/tmp. Fix: introducted gcc -ffile-prefix-map option (and related -fdebug-prefix-map) to support embedding relative (rather than absolute) paths Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 13 / 30 Filesystem ordering NAME readdir - read a directory SYNOPSIS #include <dirent.h> struct dirent *readdir(DIR *dirp); [...] The order in which filenames are read by successive calls to readdir() depends on the filesystem implementation; it is unlikely that the names will be sorted in any fashion. [...] Fix: impose a deterministic order in build systems/recipes, e.g., via an explicit sort() Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 14 / 30 Build reproducibility in the large Let’s now assume we have fixed all micro-issues that impede build reproducibility How do we go about making large FOSS software collections reproducible? Experiment: making all Debian packages build reproducibly from source Debian: one of the most popular GNU/Linux distro, esp. in the server market 30’000+ (source) packages, 1+B lines of code Goals: 1Empirical experiment to identify common causes of non-reproducibility 2Real impact (if successful) due to Debian popularity in the market Approach Establish a corresponding Quality Assurance process and soft-enforce it using Continuous Integration (CI). Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 15 / 30 Adversarial rebuilding How do you find build reproducibility issues, at scale? Mass-rebuild all packages... ...building each of them twice... ...in two build environments configured to differ as much as possible: Clock set 18 months in the future in 2nd build Changing: host name, locales, kernel Reverse filesystem ordering using disorderfs ... 30+ variations in total Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 16 / 30 Reproducible Debian — Evolution over time Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 17 / 30 The Reproducible Builds ecosystem https://reproducible-builds.org/ 2014: project started by Debian developers for Debian needs fun Joined since: Arch Linux, coreboot, F-Droid, Fedora, FreeBSD, Guix, NixOS, openSUSE, Qubes, Tails, ... 2017 milestone: Tails (live distro used by Snowden to exfiltrate NSA documents) publishes a fully reproducible ISO to improve end-user verifiability R-B is an independent project hosted by Software Freedom Conservancy and supported by 3rd-party sponsors (e.g., Google, The Linux Foundation, Ford Foundation, Siemens) Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 18 / 30 Outline 1Introduction 2Open Source Software Supply Chain — Attacks 3Reproducible Builds 4Open Source Software Supply Chain — KYSW 5Software Heritage 6Conclusion Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 19 / 30 KYSW (Know Your SoftWare) Like KYC in banking, KYSW is now essential all over IT... Vertical approach: secure your software Improve security of each component separately By law: e.g. EU Cyber Resilience Act By practice: e.g. https://best.openssf.org/ Horizontal approach: all the supply chain Sec. 4. Enhancing Software Supply Chain Security ensuring and attesting, to the extent practicable, to the integrity and provenance of open source software May 2021 POTUS Executive Order Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 19 / 30 Software Heritage, in a nutshell www.softwareheritage.org TH E GRE AT LIBRA RY OF SOU RCE CODE Collect, preserve and share all software source code Preserving our heritage, enabling better software and better science for all Reference catalog find and reference all software source code Universal archive preserve and share all software source code Research infrastructure enable analysis of all software source code Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 21 / 30 Software Heritage, in a nutshell www.softwareheritage.org TH E GRE AT LIBRA RY OF SOU RCE CODE Collect, preserve and share all software source code Preserving our heritage, enabling better software and better science for all Reference catalog find and reference all software source code Universal archive preserve and share all software source code Research infrastructure enable analysis of all software source code Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 21 / 30 Software Heritage, in a nutshell www.softwareheritage.org TH E GRE AT LIBRA RY OF SOU RCE CODE Collect, preserve and share all software source code Preserving our heritage, enabling better software and better science for all Reference catalog find and reference all software source code Universal archive preserve and share all software source code Research infrastructure enable analysis of all software source code Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 21 / 30 The largest software archive, a shared infrastructure One infrastructure open and shared The largest archive ever built Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 22 / 30 The largest software archive, a shared infrastructure One infrastructure open and shared The largest archive ever built Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 22 / 30 The largest software archive, a shared infrastructure One infrastructure open and shared The largest archive ever built Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 22 / 30 A peek under the hood: a universal archive dsc dsc hg hg hg git git git git svn svn svn tar zip software origins Package repos Forges GitHub lister GitLab lister Debian lister PyPi lister . . . Distros ... Listing (full/incremental) tar Global development history permanently archived inauniform data model over 18 billion unique source files from over 290 million software projects ~1.5PB (compressed) blobs, ~35 B nodes, ~500 B edges Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 23 / 30 A peek under the hood: a universal archive Git loader Mercurial loader Debian source package loader pypi source package loader . . . Software Heritage Archive Merkle DAG + blob storage Loading & deduplication dsc dsc hg hg hg git git git git svn svn svn tar zip software origins Package repos Forges GitHub lister GitLab lister Debian lister PyPi lister . . . Distros ... Scheduling Listing (full/incremental) tar origins snapshots releases revisionsrevisions directoriesdirectories contents Global development history permanently archived inauniform data model over 18 billion unique source files from over 290 million software projects ~1.5PB (compressed) blobs, ~35 B nodes, ~500 B edges Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 23 / 30 A peek under the hood: a universal archive Git loader Mercurial loader Debian source package loader pypi source package loader . . . Software Heritage Archive Merkle DAG + blob storage Loading & deduplication dsc dsc hg hg hg git git git git svn svn svn tar zip software origins Package repos Forges GitHub lister GitLab lister Debian lister PyPi lister . . . Distros ... Scheduling Listing (full/incremental) tar origins snapshots releases revisionsrevisions directoriesdirectories contents Global development history permanently archived inauniform data model over 18 billion unique source files from over 290 million software projects ~1.5PB (compressed) blobs, ~35 B nodes, ~500 B edges Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 23 / 30 Referencing all source code artifacts with SWHIDs Software Heritage Identifiers (SWHID) link to full docs 35+B intrinsic, decentralised, cryptographic Full fledged source code references for traceability, integrity and reproducibility Linux Foundation SPDX 2.2 IANA-registered "swh:" WikiData property P6138 Examples: Apollo 11 AGC excerpt,Quake III rsqrt Guidelines available, see the HOWTO ISO standardization underway, see swhid.org Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 24 / 30 Industry use cases (selection) Open Source complete and corresponding source code distribution (Intel) Software Heritage members can: archive source code in Software Heritage, distribute only the SWHID Traceability and integrity (OIN for the Linux System Definition) Software Heritage members can: archive source code in Software Heritage track it and verify its integrity using its SWHID And much more! cybersecurity: just launched SWHSec project swhsec.github.io AI: providing high-quality data for ethical code LLMs an open (source & data) source code scanner for open compliance Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 27 / 30 Industry use cases (selection) Open Source complete and corresponding source code distribution (Intel) Software Heritage members can: archive source code in Software Heritage, distribute only the SWHID Traceability and integrity (OIN for the Linux System Definition) Software Heritage members can: archive source code in Software Heritage track it and verify its integrity using its SWHID And much more! cybersecurity: just launched SWHSec project swhsec.github.io AI: providing high-quality data for ethical code LLMs an open (source & data) source code scanner for open compliance Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 27 / 30 swh-scanner Vision swh-scanner is an open source and open data source code scanner for open compliance workflows, backed by the largest public archive of FOSS source code. Design Query Software Heritage as source of truth about public code Leverages the Merkle DAG model and SWHIDs for maximum scanning efficiency E.g., no need to query the back-end for files contained in a known directory File-level granularity Output: source tree partition into known (= published before) v. unknown Source: gitlab.softwareheritage.org/swh/devel/swh-scanner License: GPL-3+ Package: pypi.org/project/swh.scanner Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 28 / 30 Outline 1Introduction 2Open Source Software Supply Chain — Attacks 3Reproducible Builds 4Open Source Software Supply Chain — KYSW 5Software Heritage 6Conclusion Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 29 / 30 Reproducible Builds ↔Software Heritage Software Heritage provides key ingredients for R-B pipelines: on-demand archival (e.g., of VCS commits referenced by build recipes) + long-term availability We have implemented this by integrating the GNU Guix package manager with Software Heritage Ludovic Courtès, Timothy Sample, Simon Tournier, Stefano Zacchiroli Source Code Archiving to the Rescue of Reproducible Deployment. ACM REP 2024 (to appear) Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 29 / 30 Learn more reproducible-builds.org softwareheritage.org Piergiorgio Ladisa, Henrik Plate, Matias Martinez, Olivier Barais SoK: Taxonomy of Attacks on Open-Source Software Supply Chains IEEE S&P 2023 Chris Lamb, Stefano Zacchiroli Reproducible Builds: Increasing the Integrity of Software Supply Chains IEEE Softw. 39(2): 62-70 (2022) Roberto Di Cosmo, Stefano Zacchiroli Software Heritage: Why and How to Preserve Software Source Code iPRES 2017: Intl. Conf. on Digital Preservation Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 30 / 30 Appendix Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 1 / 8 Outline 7Reproducible Builds 8Software Heritage 9Software Heritage — Datasets 10 Efficient traversal of the full graph Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 2 / 8 Challenges Debian reached 95% reproducible packages, can we go all the way? Yes, it’s just busy/constant maintenance work. Working with upstream and spreading r-b culture helps a lot. How to make signed buld artifacts reproducible (without distributing signing keys)? Detached signatures. (Painful for distribution channels.) How do end-user verify build artifacts before installation? Particularly challenging on locked-down mobile environments/stores. How little trusted code is acceptable? Bootstrappable Builds managed to bootstrap from a 6 KiB trusted ELF binary to GCC via TCC. Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 2 / 8 Outline 7Reproducible Builds 8Software Heritage 9Software Heritage — Datasets 10 Efficient traversal of the full graph Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 3 / 8 Example: most popular commit verbs (stemmed) Query using Amazon Athena SELECT COUNT(*) AS C, word FROM ( SELECT word_stem(lower(split_part( trim(from_utf8(message)),’ ’, 1))) AS word FROM revision WHERE length(message) < 1000000) WHERE word != ’’ GROUP BY word ORDER BY C DESC LIMIT 20; Total cost: approximately .5 euros Results Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 6 / 8 Example: most popular commit verbs (stemmed) Query using Amazon Athena SELECT COUNT(*) AS C, word FROM ( SELECT word_stem(lower(split_part( trim(from_utf8(message)),’ ’, 1))) AS word FROM revision WHERE length(message) < 1000000) WHERE word != ’’ GROUP BY word ORDER BY C DESC LIMIT 20; Total cost: approximately .5 euros Results Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 6 / 8 Outline 7Reproducible Builds 8Software Heritage 9Software Heritage — Datasets 10 Efficient traversal of the full graph Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 7 / 8 Going beyond SQL State-of-the-art graph compression from social networks Paolo Boldi, Antoine Pietri, Sebastiano Vigna, Stefano Zacchiroli Ultra-Large-Scale Repository Analysis via Graph Compression SANER 2020, 27th Intl. Conf. on Software Analysis, Evolution and Reengineering. IEEE Results Full graph structure (25 B nodes, 350 B edges) in 200 GiB RAM traversal time is tens of ns per edge bidirectional traversals implemented beware: metadata access is still off RAM Java and gRPC APIs available docs.softwareheritage.org/devel/swh-graph/grpc-api.html Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 7 / 8 Examples assume graph service on localhost:50091 Find all origins containing a given content grpc_cli call localhost:50091 swh.graph.TraversalService.Traverse "\ src: ’swh:1:cnt:8722d84d658e5e11519b807abb5c05bfbfc531f0’, direction: BACKWARD, \ mask: {paths: [’swhid’,’ori.url’]}, return_nodes: {types: ’ori’}" Gives a list of origins including "https://github.com/rdicosmo/parmap", encoded as "swh:1:ori:8903a90cff8f07159be7aed69f19d66d33db3f86" (beware: this is not a SWHID!) Shortest provenance path of a content in a given origin grpc_cli call localhost:50091 swh.graph.TraversalService.FindPathBetween "\ src: ’swh:1:ori:8903a90cff8f07159be7aed69f19d66d33db3f86’, \ dst: ’swh:1:cnt:8722d84d658e5e11519b807abb5c05bfbfc531f0’, \ mask: {paths: [’swhid’]}" | egrep ’swhid’ connecting to localhost:50091 swhid: "swh:1:ori:8903a90cff8f07159be7aed69f19d66d33db3f86" swhid: "swh:1:snp:1527a93b039d70f6a781b05d76b77c6209912887" swhid: "swh:1:rev:82df563aecf86b9164eee7d10d40f2d8cbd1c78d" swhid: "swh:1:dir:484db39bb2825886191837bb0960b7450f9099bb" swhid: "swh:1:dir:4d15e44b378fe39dd23817abee756cd47ad14575" swhid: "swh:1:cnt:8722d84d658e5e11519b807abb5c05bfbfc531f0" Rpc succeeded with OK status Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 8 / 8 Examples assume graph service on localhost:50091 Find all origins containing a given content grpc_cli call localhost:50091 swh.graph.TraversalService.Traverse "\ src: ’swh:1:cnt:8722d84d658e5e11519b807abb5c05bfbfc531f0’, direction: BACKWARD, \ mask: {paths: [’swhid’,’ori.url’]}, return_nodes: {types: ’ori’}" Gives a list of origins including "https://github.com/rdicosmo/parmap", encoded as "swh:1:ori:8903a90cff8f07159be7aed69f19d66d33db3f86" (beware: this is not a SWHID!) Shortest provenance path of a content in a given origin grpc_cli call localhost:50091 swh.graph.TraversalService.FindPathBetween "\ src: ’swh:1:ori:8903a90cff8f07159be7aed69f19d66d33db3f86’, \ dst: ’swh:1:cnt:8722d84d658e5e11519b807abb5c05bfbfc531f0’, \ mask: {paths: [’swhid’]}" | egrep ’swhid’ connecting to localhost:50091 swhid: "swh:1:ori:8903a90cff8f07159be7aed69f19d66d33db3f86" swhid: "swh:1:snp:1527a93b039d70f6a781b05d76b77c6209912887" swhid: "swh:1:rev:82df563aecf86b9164eee7d10d40f2d8cbd1c78d" swhid: "swh:1:dir:484db39bb2825886191837bb0960b7450f9099bb" swhid: "swh:1:dir:4d15e44b378fe39dd23817abee756cd47ad14575" swhid: "swh:1:cnt:8722d84d658e5e11519b807abb5c05bfbfc531f0" Rpc succeeded with OK status Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Building Blocks for a Safer Open Source Supply Chain 29 May 2024 8 / 8