Source Code Tracking at Software Heritage Scale for compliance, open science, and security

Source Code Tracking at Software Heritage Scale for compliance, open science, and security Stefano Zacchiroli Software Heritage — [email protected], @zacchiro 30 Nov 2021 #swh5years — Sponsors meeting UNESCO THE GREAT LIBRARY OF SOURC E CODE Stefano Zacchiroli Source Code Tracking at SWH Scale 30 Nov 2021, #swh5years 1 / 19 Outline 1FOSS Source Code Tracking... 2... for Open Compliance 3... for Open Science 4... for Security 5Wrapping up Stefano Zacchiroli Source Code Tracking at SWH Scale 30 Nov 2021, #swh5years 2 / 19 The largest free/open source software archive The largest public source code archive in the world (and growing!) Stefano Zacchiroli Source Code Tracking at SWH Scale 30 Nov 2021, #swh5years 2 / 19 Automation and storage Git loader Mercurial loader Debian source package loader pypi source package loader . . . Software Heritage Archive Merkle DAG + blob storage Loading & deduplication dsc dsc hg hg hg git git git git svn svn svn tar zip software origins Package repos Forges GitHub lister GitLab lister Debian lister PyPi lister . . . Distros ... Scheduling Listing (full/incremental) tar origins snapshots releases revisionsrevisions directoriesdirectories contents Full development history permanently archived inauniform data model. Stefano Zacchiroli Source Code Tracking at SWH Scale 30 Nov 2021, #swh5years 3 / 19 Meet the Software Heritage Identifiers (SWHIDs) (full spec) An emerging standard in Linux Foundation’s SPDX 2.2 IANA-registered "swh:" URI prefix WikiData property P6138 Examples Apollo 11 AGC excerpt Quake III rsqrt Stefano Zacchiroli Source Code Tracking at SWH Scale 30 Nov 2021, #swh5years 4 / 19 Meet the Software Heritage Identifiers (SWHIDs) (full spec) An emerging standard in Linux Foundation’s SPDX 2.2 IANA-registered "swh:" URI prefix WikiData property P6138 Examples Apollo 11 AGC excerpt Quake III rsqrt Stefano Zacchiroli Source Code Tracking at SWH Scale 30 Nov 2021, #swh5years 4 / 19 Meet the Software Heritage Identifiers (SWHIDs) (full spec) An emerging standard in Linux Foundation’s SPDX 2.2 IANA-registered "swh:" URI prefix WikiData property P6138 Examples Apollo 11 AGC excerpt Quake III rsqrt Stefano Zacchiroli Source Code Tracking at SWH Scale 30 Nov 2021, #swh5years 4 / 19 Meet the Software Heritage Identifiers (SWHIDs) (full spec) An emerging standard in Linux Foundation’s SPDX 2.2 IANA-registered "swh:" URI prefix WikiData property P6138 Examples Apollo 11 AGC excerpt Quake III rsqrt Stefano Zacchiroli Source Code Tracking at SWH Scale 30 Nov 2021, #swh5years 4 / 19 Meet the Software Heritage Identifiers (SWHIDs) (full spec) An emerging standard in Linux Foundation’s SPDX 2.2 IANA-registered "swh:" URI prefix WikiData property P6138 Examples Apollo 11 AGC excerpt Quake III rsqrt Stefano Zacchiroli Source Code Tracking at SWH Scale 30 Nov 2021, #swh5years 4 / 19 Tech preview: swh-scanner Vision swh-scanner is an open source and open data source code scanner for open compliance workflows, backed by the largest public archive of FOSS source code. Design (of the current prototype) query the Software Heritage archive as source of truth about public code leverages the Merkle DAG model and SWHIDs for maximum scanning efficiency e.g., no need to query the back-end for files contained in a known directory file-level granularity output: source tree partition into known (= published before) v. unknown Code: forge.softwareheritage.org/source/swh-scanner (GPL 3+) Package: pypi.org/project/swh.scanner Stefano Zacchiroli Source Code Tracking at SWH Scale 30 Nov 2021, #swh5years 7 / 19 swh-scanner — Example $ pip install swh.scanner $ swh scanner scan -f json /srv/src/linux/kernel { [...] "/srv/src/linux/kernel/auditsc.c": { "known": true, "swhid": "swh:1:cnt:814406a35db163080bbf937524d63690861ff750" }, "/srv/src/linux/kernel/backtracetest.c": { "known": true, "swhid": "swh:1:cnt:a2a97fa3071b1c7ee6595d61a172f7ccc73ea40b" }, "/srv/src/linux/kernel/bounds.c": { "known": true, "swhid": "swh:1:cnt:9795d75b09b2323306ad6a058a6350a87a251443" }, "/srv/src/linux/kernel/bpf": { "known": true, "swhid": "swh:1:dir:fcd9987804d26274fee1eb6711fac38036ccaee7" }, "/srv/src/linux/kernel/capability.c": { "known": true, "swhid": "swh:1:cnt:1444f3954d750ba685b9423e94522e0243175f90" }, [...] } 0,53s user 0,61s system 145% cpu 1,867 total Stefano Zacchiroli Source Code Tracking at SWH Scale 30 Nov 2021, #swh5years 8 / 19 swh-scanner — Example (cont.) $ du -sh --exclude=.git /srv/src/linux 1,1G /srv/src/linux $ time swh scanner scan -f json -x *.git /srv/src/linux { [...] "/srv/src/linux/arch": { "known": true, "swhid": "swh:1:dir:590c329d3548b7d552fc913a51965353f01c9e2f" }, [...] "/srv/src/linux/scripts/kallsyms.c": { "known": true, "swhid": "swh:1:cnt:0096cd9653327584fe62ce56ba158c68875c5067" }, "/srv/src/linux/scripts/kconfig": { "known": false, "swhid": "swh:1:dir:548afc93bd01d2fba0dfcc0fd8c69f4b082ab8c6" }, "/srv/src/linux/scripts/kconfig/.conf.o.cmd": { "known": false, "swhid": "swh:1:cnt:0d8be19e430c082ece6a3803923ad6ecb9e7d413" }, [...] } 20,84s user 1,52s system 103% cpu 21,540 total Stefano Zacchiroli Source Code Tracking at SWH Scale 30 Nov 2021, #swh5years 9 / 19 swh-scanner — Example (cont.) Interactive mode to drill-down and inspect unknown files: $ swh scanner scan -f sunburst -x *.git /srv/src/linux Stefano Zacchiroli Source Code Tracking at SWH Scale 30 Nov 2021, #swh5years 10 / 19 swh-scanner — Going further swh-scanner shows that it is possible to create a source code scanner that is both open source and backed by the most comprehensive open data FOSS archive. Roadmap swh-scanner is not a production-ready scanner. The following features are still missing: license information →in-house scanning + ClearlyDefined provenance information →Software Heritage crawling info increase granularity to snippet/SLOC Some of these are low-hanging fruits, some require substantial R&D investments. Feedback welcome feel free to play with swh-scanner, feedback is very welcome! caveat: intensive use will result in hitting the API rate-limit Stefano Zacchiroli Source Code Tracking at SWH Scale 30 Nov 2021, #swh5years 11 / 19 swh-scanner — Going further swh-scanner shows that it is possible to create a source code scanner that is both open source and backed by the most comprehensive open data FOSS archive. Roadmap swh-scanner is not a production-ready scanner. The following features are still missing: license information →in-house scanning + ClearlyDefined provenance information →Software Heritage crawling info increase granularity to snippet/SLOC Some of these are low-hanging fruits, some require substantial R&D investments. Feedback welcome feel free to play with swh-scanner, feedback is very welcome! caveat: intensive use will result in hitting the API rate-limit Stefano Zacchiroli Source Code Tracking at SWH Scale 30 Nov 2021, #swh5years 11 / 19 swh-scanner — Going further swh-scanner shows that it is possible to create a source code scanner that is both open source and backed by the most comprehensive open data FOSS archive. Roadmap swh-scanner is not a production-ready scanner. The following features are still missing: license information →in-house scanning + ClearlyDefined provenance information →Software Heritage crawling info increase granularity to snippet/SLOC Some of these are low-hanging fruits, some require substantial R&D investments. Feedback welcome feel free to play with swh-scanner, feedback is very welcome! caveat: intensive use will result in hitting the API rate-limit Stefano Zacchiroli Source Code Tracking at SWH Scale 30 Nov 2021, #swh5years 11 / 19 Outline 1FOSS Source Code Tracking... 2... for Open Compliance 3... for Open Science 4... for Security 5Wrapping up Stefano Zacchiroli Source Code Tracking at SWH Scale 30 Nov 2021, #swh5years 12 / 19 Prior Art Search & Plagiarism Detection Use case 1: Researcher (Prior Art Search) Verify that the novelty status of the replication package of a paper under submission matches expectations: Original code written for the experiment should be novel Reused 3rd-party FOSS components should not be novel (verifying this also helps with spotting undesirable local patches) Use case 2: Open Science Publisher (Plagiarism Detection) Verify that the source code part of papers, submitted as original work by the authors, are in fact original. This is already standard publisher procedure for the textual part of submitted papers, but it isn’t yet for software source code. Stefano Zacchiroli Source Code Tracking at SWH Scale 30 Nov 2021, #swh5years 12 / 19 Prior Art Search & Plagiarism Detection — Example Let’s verify that in the replication package of our ICSE 2021 paper about swh-fuse we have used a public, archived version of the package: $ swh scanner scan -f ndjson replication-package/swh-fuse/ {".": {"swhid": "swh:1:dir:3d4f903b[...]", "known": true}} ... Let’s check that the rest of the replication package is novel (at submission time, it will be archived in Software Heritage at camera-ready time): $ swh scanner scan -f ndjson replication-package {".": {"swhid": "swh:1:dir:14ecd6[...]", "known": false}} ... Researchers can integrate these checks in their pre-submission checklists Publishers can integrate these checks into existing plagiarism detection pipelines, making results available to scientific editors and reviewers Stefano Zacchiroli Source Code Tracking at SWH Scale 30 Nov 2021, #swh5years 13 / 19 How do we go from file/dir SWHIDs back to repositories? At the scale of Software Heritage, querying efficiently all the places where a given SWHID can be found in is a challenging R&D problem, that nobody has ever tackled in its full generality (see Rousseau et al., Empir. Softw. Eng. (2020) for the scientific details). We are working on a complete software provenance index for the archive Meanwhile we have building blocks in place that address specific parts of the problem, based on a compressed graph representation of the archive (see Boldi et al., SANER (2020) for sci. details). They come with an API you can play with. Code: forge.softwareheritage.org/source/swh-graph (GPL 3+) Doc: docs.softwareheritage.org/devel/swh-graph API: docs.softwareheritage.org/devel/swh-graph/api.html Stefano Zacchiroli Source Code Tracking at SWH Scale 30 Nov 2021, #swh5years 16 / 19 Graph API — Example (All) repositories containing vulnerable versions of d1_both.c: $ swh-whereis swh:1:cnt:0a84f957118afa9804451add380eca4719a9765e https://github.com/mathiassamuelson/openssl-dane-ms https://gitorious.org/baserock-morphs/openssl.git https://github.com/tack/openssl_tack https://gitorious.org/myopenssl/myopenssl.git https://bitbucket.org/xreach/android-external-openssl.git [...] Where swh-whereis is a trivial wrapper around the swh-graph API: curl --silent --fail --location \ "${API_URL}/graph/leaves/${SWHID}/?direction=backward&resolve_origins=true" Caveats no filtering/sorting on commit timestamps (yet) no filtering on path (e.g., OpenSSL should be included in a sub-dir, etc) Stefano Zacchiroli Source Code Tracking at SWH Scale 30 Nov 2021, #swh5years 17 / 19 Graph API — Example (cont.) Same API, compliance use case Find software that has been licensed under the original 1.0 version of the Affero GPL license (SWHID: swh:1:cnt:8f3209754390bbc58953d49701ed45c9d4a1a47f). $ curl https://archive.softwareheritage.org/api/1/graph/randomwalk/\ swh:1:cnt:8f3209754390bbc58953d49701ed45c9d4a1a47f/ori/\ ?direction=backward&limit=-1&resolve_origins=true https://github.com/uwsampa/grappa/ note the random walk, for spot checks, examples, etc. (Grappa has since been re-released under the BSD 3-Clause license) Stefano Zacchiroli Source Code Tracking at SWH Scale 30 Nov 2021, #swh5years 18 / 19 Outline 1FOSS Source Code Tracking... 2... for Open Compliance 3... for Open Science 4... for Security 5Wrapping up Stefano Zacchiroli Source Code Tracking at SWH Scale 30 Nov 2021, #swh5years 19 / 19 Wrapping up TH E G REAT LIBR ARY OF S OURCE CO D E www.softwareheritage.org @swheritage thanks to its data model, Software Heritage provides a global view on the largest public collection of software source code artifacts this global view enables source code tracking of public code (FOSS, and more) at an unprecedented scale global source code tracking is a key building block for addressing use cases in domains such as license compliance,open science, and security we are laying the foundations for addressing these use cases Contacts Stefano Zacchiroli /[email protected] /@zacchiro /@[email protected] Stefano Zacchiroli Source Code Tracking at SWH Scale 30 Nov 2021, #swh5years 19 / 19 Web API — Integrate your tools with the Software Heritage archive RESTful API to programmatically access the Software Heritage archive https://archive.softwareheritage.org/api/ Features pointwise browsing of the archive ... snapshots  revisions  directories  contents ... full access to the metadata of archived objects crawling information when have you last visited this Git repository I care about? where were its branches/tags pointing to at the time? Endpoint index https://archive.softwareheritage.org/api/1/ Stefano Zacchiroli Source Code Tracking at SWH Scale 30 Nov 2021, #swh5years 1 / 2 Anatomy of a KYSW toolchain source: A Community Take on the License Compliance Industry, Stefano Zacchiroli, FOSDEM 2016, Legal and Policy Issues devroom, https://upsilon.cc/~zack/talks/2016/2016-01-31-fosdem-compliance.pdf A source code scanner is the key ingredient of all KYSW toolchains: it scans a local source code base and compares it to a FOSS knowledge base, summarizing findings. Stefano Zacchiroli Source Code Tracking at SWH Scale 30 Nov 2021, #swh5years 2 / 2