Integrating Diverse Research Data into one Repository: The BASS metadata.xlsx Crawler
Ribas-Ribas, Mariana; Bibi, Riaz; Wurl, Oliver; Badewien, Thomas H.; Feenders, Christoph
- Publisher
- Zenodo
- Language
- en
Abstract
Initiated in 2023, the DFG Research Unit Biogeochemical Processes and Air–SeaExchange in the Sea-Surface Microlayer (BASS) explores air–sea exchange processesthrough multidisciplinary field campaigns, mesocosm and laboratory experiments, andmodeling. During the expected eight-year project duration (two phases of four yearseach), BASS encompasses nine subprojects per phase, involving 25 principalinvestigators, 50 employees and students, and IT specialists, a total of approximately 75potential users. Handling and sharing diverse data from these sources require strong,compatible research data management (RDM) workflows that follow FAIR Findable,Accessible, Interoperable and Reusable (FAIR) principles. However, sketching a suitabledata model for the integration of such diverse and dynamic data in advance was out ofscope. Our case study shows an open-source-based RDM system designed to helpscientists without programming skills within BASS. The workflow focuses on theLinkAhead platform, improved by a custom automation tool called the metadata.xlsxcrawler: This simple tool allows data-providers to enrich their metadata dynamically, byentering it into an Excel template spreadsheet stored alongside the corresponding datafiles. The metadata.xlsx crawler maps the entries to the data model or extends itautomatically, if needed. Using a familiar, offline-capable Excel format, researchers record metadata during datacollection with minimal technical barriers. Extracted metadata adheres to PANGAEArepository standards, including controlled vocabularies and provenance information,making repository submissions easier despite the current lack of direct export featuresfrom LinkAhead. This method connects simple offline metadata capture with integratedRDM workflows, supporting consistent, FAIR-compliant data archiving across variousinstitutions and disciplines. As we prepare for BASS Phase 2, we plan to continue usingand improving this approach. Future updates include saving search queries to improvemetadata discoverability and enhancing metadata fields to boost findability andinteroperability.
Full text
Biogeochemical processes and Air–sea exchange in the Sea-Surface microlayer (BASS) PD Dr. Mariana Ribas Ribas FAIR in Action: Open RDM Integration 01.10.2025, Göttingen Multidisciplinary Research Network Driving BASS DFG Research Unit: BASS
2 Multidisciplinary observational, experimental and modelling data Heterogeneous Data Formats and Workflows
3 Sample Collection → Processing → Analysis → Storage → Sharing Mesocosm and Wind Wave Tunnel study Helgoland joint field campaign Baltic Sea campaign Research cruise Meteor expedition Discrete Observational Data Remote Sensing Data Satellite observations Research aircraft sensors High-Resolution Data In situ sensors continuous data Autonomous platforms Micro profiler Heterogeneous Sources of Data
4 Different subprojects use different sampling strategies, instruments, formats, and vocabularies Why Integration was hard? • Tabular data: XLSX, CSV, TXT • Documents: PDF • Images: JPG, PNG, TIFF • Videos: MP4 How can we bring all these into one database in a FAIR and integrated way?
3 / 9 LinkAhead oFully open-source software (AGPLv3) oLinked data files and metadata oEasy and powerful search → find data across subprojects instantly oSemantic core: Fast and easy tuning of the data model oLinkAhead can be easily extended or modified according to user requirements oIntegrates into existing workflows (This BASS phase could be integrated in next) oAPI available to link electronic lab notebook oAutomated export → FAIR-ready metadata for repositories (e.g., PANGAEA) The Bridge: LinkAhead + Metadata.xlsx Crawler 5
6 Integrating Diverse Research Data into One Repository: The BASS metadata.xlsx Crawler BASS Metadata.xlsx Concept • One central metadata spreadsheet template for all SPs •Excel-based template – no extra software needed •Familiar tool (excel) – lowering the barrier for adoption •Aligned with FAIR and repository (PANGAEA/LinkAhead). • Ensures standardized structure while staying flexible • Collects essential metadata: • Dataset title, description, keywords • Instrument/method • Spatial/temporal coverage • Contact/PI information
7 The Crawler •Crawler automatically reads submitted metadata.xlsx files • Transforms Excel rows into standardized metadata (JSON/XML) •Standard fields (parameter, unit, instrument, date, location) •Optional fields: Add extra properties (e.g., calibration ID, serial no.) → without changing the template • Prepares FAIR-ready metadata for LinkAhead / PANGAEA • Detects errors, missing values, and enforces controlled vocabularies (e.g., units, methods). • Benefits: • Reduces manual curation workload • Ensures consistency across datasets • Early QA/QC for metadata before submission Scientists fill Excel → Crawler Processes → LinkAhead → PANGAEA
“Final” Data Management System 8
Lessons Learned •FAIR in practice: Metadata harmonization makes data Findable and Interoperable •Training and guidance are essential – clear instructions needed for consistent metadata •Automation reduces errors – crawler catches missing values and enforces vocabularies •Challenges: • Convincing scientists to fill templates properly • Balance required → completeness vs. user-friendliness is a trade-off • Controlled vocabularies still evolving Integration achieved: 9 SPs feeding into one database This template + crawler workflow can scale to other consortia Reusable model: 9
