Publishing Large Datasets: Experiences with Data Scaling and Hybrid Model Publication

Kerzenmacher, Tobias; Barthlott, Sabine; Schneider, Matthias; Bach, Felix; Soltau, Kerstin; Hofmann, Stefan

doi:10.5281/zenodo.17815953

Publishing Large Datasets: Experiences with Data Scaling and Hybrid Model Publication

Kerzenmacher, Tobias; Barthlott, Sabine; Schneider, Matthias; Bach, Felix; Soltau, Kerstin; Hofmann, Stefan

Publisher: Zenodo
DOI: 10.5281/zenodo.17815953
Repository: zenodo.org
Language: en

Abstract

The publication of large datasets poses significant challenges for research data management, particularly regarding storage, access, citability, and adherence to FAIR principles. We present our experience publishing a 25 TB dataset from the Infrared Atmospheric Sounding Interferometer (IASI) covering October 2014 to June 2019 (MUSICA IASI v3.2.1) using a hybrid approach. A representative one-day subset was deposited in the institutional repository RADAR4KIT with a DOI, while the complete dataset is archived at the Large Scale Data Facility (LSDF) provided by the Scientific Computing Center (SCC) at KIT and accessed via a THREDDS Data Server (TDS) hosted by IMKASF at KIT. We describe the workflow, metadata harmonisation, persistent linking strategy, and lessons learned from this publication process.

Full text

KIT – The University in the Helmholtz Association The authors thank the data infrastructure staff at FIZ Karlsruhe, IMKASF, and KIT, as well as the supporting funding agencies. The Leibniz Science Campus Digital Transformation of Research (DiTraRe) is funded by the Leibniz Association (W74/2022). Example netCDF header 2. Hybrid Workflow Representative subset → RADAR4KIT (citable, DOI: 10.35097/408) Full dataset → LSDF (Large Scale Data Facility), data accessed via THREDDS data server Long-term archive → bwDataArchive 1. The Challenge Multi-terabyte satellite datasets (25 TB MUSICA IASI v3.2.1) exceed upload limits due to archival restrictions on bwArchive (~600 GB) Need FAIR, citable, and accessible solutions Ensure reproducibility, metadata compliance, and long-term preservation Institute of Meteorology and Climate Research Atmospheric Trace Gases and Remote Sensing Publishing Large Datasets: Experiences with Data Scaling and Hybrid Model Publication Big Data, Small Uploads: A Hybrid Workflow for Large Datasets Tobias Kerzenmacher, Sabine Barthlott, Matthias Schneider — KIT, Karlsruhe, Germany Felix Bach, Kerstin Soltau, and Stefan Hofmann — FIZ Karlsruhe, Eggenstein-Leopoldshafen, Germany Contact: T[email protected] 3. Representative Subset One day (17.5 GB) of the large satellite data (25 TB, 1734 days) selected. Captures retrieval configuration and structure. Enables testing, validation, and reproducibility. Includes MD5 checksums, ReadMe Includes stable links to the LSDF archive via the THREDDS catalogue 4. Lessons Learned Hybrid model balances persistence and scalable access Metadata harmonization and clear documentation are essential THREDDS server at IMKASF to the full data set on the LSDF. 5. Outlook Repository improvements: Terminology Service 4 NFDI (TS4NFDI), WebDAV upload Additional metadata fields, single-file downloads, Zenodo-style versioning Specialized repositories for multiterabyte datasets may emerge Landing page for the published data on RADAR4KIT. THREDDSRADAR4KIT © ESA © Simon Raffeiner/SCC/KIT