Large-Scale RDM Without a Centralization Cost
Hanke, Michael; Riße, Matthias
- Publisher
- Zenodo
- Language
- en
Abstract
Software development and data curation or analysis share many of their issues: keeping track of the evolution of files – ideally with information on how, why, when, and by whom – , organizing collaboration with multiple people, keeping track of known issues and other TODOs, discussing changes, making versions available to others, automating tasks, and more. Often you will even have to write code as part of a data project, blurring the line between the two even more. In the free and open-source software development world these issues already have well established solutions: a version control system keeps track of your projects history and ongoing development, a forge can serve as a collaboration hub, CI/CD services provide flexible automation. So, why not apply them to our data management needs? Git, git-annex and DataLad together build the foundation of a decentralized version control system for large datasets, with DataLad being specifically tailored towards Research Data Management. Forgejo-aneksajo on the other hand extends Forgejo – a well-established free software git forge – with git-annex support, turning it into a versatile self-hostable (meta-)data collaboration hub for this ecosystem.
Full text
Scale-Free Collaboration Networks of Digitally Sovereign Actors Use Cases for Decentralized RDM Cross-Research Field Activity Michael Hanke, Matthias Riße Large-Scale RDM Without a Centralization Cost Funding Reproducible HPC Workflows With Actionable Provenance Inputs Pipeline Results sub-100123 Dedicated ephemeral workspace for each compute job Dataset created with unique identifier Link input data Link processing pipeline Develop workflow Workflow execution Result consolidation/ merge Dataset with file content present Dataset with file content absent Content identity and metadata availability File content Ephemeral workspace Free, Self-Hostable (Meta)Data Platform Built on Industry Standards Scalable Metadata System Indexing Existing Infrastructure SUBMIT QUERY KNOWLEDGE QUERY EDIT CURATION Standardized Interface to Data Identity and Availability Metadata Maintained Internal Processes and Infrastructures for Data Acquisition and Storage Neuroscience & Medicine ATRIS Advanced Simulation Climate & Energy Systems Information Earth & Environment Findability and access through homogeneous APIs 1 2 3 Portable Structure to Remote-Process "cannot-move" Data Productivity Suite for "in-flight" Datasets CI/CD Automation: Processing Quality Assurance / Control Distribution ... Actions Project Management: Issue tracker Change proposals Wiki Versioning Releases ... Enhancements: git-annex data storage UI integration aneksajo Ecosystem of Interoperable (Scientific) Cyberinfrastructure sciebo NIH data archive F o r g e j o - A n e k s a j o
