The Life of Research Data: How to Keep It Organized and Accessible

www.nfdi4biodiversity.org @NFDI4Biodiv #NFDI4Biodiv 2025-10-29 Ivaylo Kostadinov GFBio e.V. Slide deck by Daniel Tschink et. Al. The Life of Research Data: How to Keep It Organized and Accessible 2025-10-29 Dr. Ivaylo Kostadinov Technical Lead • RDM champion • Team Leader for Data and Software Solutions at the German Federation for Biological Data • Background in Bioinformatics & Marine Microbiology Lecturer Homepage https:// www.gfbio-ev.de https://www.nfdi4biodiversity.org/de YouTube https://www.youtube.com/@NFDI4Biodiv GFBio Service Help Desk [email protected] or [email protected] 2025-10-29 Source: IDC’s Digital Universe Study 2014 https://www.emc.com/leadership/digital-universe/2014iview/executive-summary.html ‘The rising tide of data – nearly as many digital bits as there are stars in the universe.’ The age of ‘big data’ 2025-10-29 −The age of ‘big data’: the digital universe grew by about a factor of 500, from 0.13 zettabytes to 64 zettabytes per year between 2005 and 2020 −The number of papers has surged exponentially >3 million papers per year −The number of active peer-reviewed English-language journals increased to >33,000 © Data Age 2025, International Data Cooperation, Redgate Blog Volume of Data created and replicated worldwide Statista Digital Economy Compass 2019 The age of ‘big data’ From Data to Wisdom 2025-10-29 … knowledge is applied to make informed decisions. Weekly measurements of ice sheet thickness. 30% reduction in Arctic sea ice over the past 100 years. … combine different data sets in a meaningful way to understand relationships, trends, and implications. https://commons.wikimedia.org/wiki/File:DIKW_Pyramid.svg CC BY-SA 4.0 Human-induced CO₂ emission lead to climate shifts. Policymakers implement carbon reduction strategies. … process, structure, give meaning to your data set. 2025-10-29 Availability of research data with time How long are Research Data usually available after publication? 0 0.25 0.5 0.75 1 0 5 10 15 20 25 Data extant (assuming author responded) Age of paper (years) 2025-10-29 •Data being lost are estimated to increase by 17% in every year after publication. •Find a working e-mail address for the first, last, or corresponding author fell by 7% per year. •"Overall, we only received 19.5% of the requested data sets, and only 11% for articles published before 2000.” Missing data As research articles age, the odds of their raw data being extant drop dramatically. Vines, Timothy H. et al. Current Biology, 2014, Volume 24, Issue 1, 94-97 Availability of research data with time Data availability ✓Published and openly accessible data drives exploration of biodiversity related questions! ✓Trend: Data Papers; e.g. Biodiversity Data Journal, Ecological Archives Reichman, O. J., Jones, M. B. Schildhauer, M. P. (2011) https://doi.org/10.1126/science.1197962 Chavan, V., Penev, L. (2011) https://doi.org/10.1186/1471-2105-12-S15-S2 2025-10-29 “…we estimate that less than 1% of the ecological data collected is accessible after publication of associated results. Rather than providing direct access to data, we share interpretations of distilled data through presentations and publications.” 2025-10-29 Research Funding in Germany ? Types of Research Data Type Definition Example Primary/Raw Data Primary data are the original data derived from your research Observational data, experimental data, i mages, … Secondary Data Secondary data are data derived from your primary data. Derived/compiled data (selection, correction, aggregation,…), d ocumentation ( methods, laboratory books, test protocols ), procedures (algorithms, software), … Metadata Data that provides information about other data Name author, company/model/camera, calibration/settings, … Research information Administrative information Employees, running times, financing of projects, funding applications, … 2025-10-29 Data Life Cycle •Conceptual tool which helps to understand the different steps from generating data to knowledge creation •Data sharing and reuse begins with good data practice •In reality, Data life cycle is implemented depending on the project and the data 2025-10-29 Cioffi, Matt, & Goldman, Julie. (2023). Harvard Biomedical Research Data Lifecycle (Version 5). Zenodo. https://doi.org/10.5281/zenodo.80761 68 Licensed under CC-BY-NC ELIXIR (2021) Research Data Management Kit. A deliverable from the EU-funded ELIXIR-CONVERGE project (grant agreement 871075). URL: https://rdmkit.elixir-europe.org Modified after https://www.gfbio.org/training/ material/data-life-cycle/ licensed under CC-BY-NC Data should be … 2025-10-29 Data should be as open as possible as closed as necessary RDM should be done according to these principles! Wilkinson, M., Dumontier, M., Aalbersberg, I. et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci Data 3, 160018 (2016). https://doi.org/10.1038/sdata.2016.18 FAIR Data Principles Wilkinson, et al., Scientific Data, 2016 http://doi.org/10.1038/sdata.2016.18 2025-10-29 What FAIR is NOT 2025-10-29 •A standard •Equal to open data •A quality, but a quantity •Only for humans or only for machines •Only for life sciences •Equal to RDF, Linked Data, or Semantic Web B. Mons et al., doi:10.3233/ISU-170824 FAIR Data Self Assessment Tools 2025-10-29 https://ardc.edu.au/resource/fair-data-selfassessment-tool/ https://www.f-uji.net/ Anusuriya Devaraju, & Robert Huber. (2020). F-UJI - An Automated FAIR Data Assessment Tool. Zenodo. https://doi.org/10.5281/zenodo.6361400 2025-10-29 ELIXIR (2021) Research Data Management Kit. A deliverable from the EU-funded ELIXIR-CONVERGE project (grant agreement 871075). URL: https://rdmkit.elixir-europe.org 2025-10-29 ELIXIR (2021) Research Data Management Kit. A deliverable from the EU-funded ELIXIR-CONVERGE project (grant agreement 871075). URL: https://rdmkit.elixir-europe.org •Start with a research question. •Think about data management, sharing and re-use as early as possible and create a data management plan. •Consider the data to collect, the methods, how to backup, preserve and share your data. TIP: Others authors’ data can foster new ideas and hypotheses. •Reinforce or validate your research by reusing data collected by others. How to do the planning? 2025-10-29 Check for existing data management recommendations or guidelines of your institution, department or funder. Funding Guidelines - DFG 2025-10-29 Guidelines on the Handling of Research Data in Biodiversity Research Guidelines on the Handling of Research Data Guidelines for Safeguarding Good Scientific Practice 2025-10-29 ELIXIR (2021) Research Data Management Kit. A deliverable from the EU-funded ELIXIR-CONVERGE project (grant agreement 871075). URL: https://rdmkit.elixir-europe.org Collection and Documentation •Systematic naming convention for trials, samples and variables •Standardized sampling protocols or software if available •Predefine names of output files from measuring devices •Instantly document relevant metadata information •Systematic and consistent folder structure 2025-10-29 File & Folder Naming 2025-10-29 •Keep file names short (max.32 characters) and descriptive •Use abbreviations and acronyms consistently •Use ASCII-characters •Avoid spaces, full stops and special characters such as: & ; * % $ £ ] { ! @ / •In case of sequential numbering, use leading zeros: e.g., 001, 002, 003 •In case you need to rename files, use renaming software •Windows: Ant Renamer, Rename-IT, Bulk Rename Utility •Mac: Renamer, Name Changer •Linux: GNOME Commander, GPRename •Unix: For Unix, the command “rename” can be helpful to find and rename files with regular expressions [project name]_[state]_[year]_[dataset]_[analysisID].ext Version Control •Keep the original version of the data file the same and save! •Establish a consistent convention and document it: •file history/version table •version control software, e.g. Git •Avoid imprecise labels, such as “final” •Ordinal numbers for major version changes, decimal for minor changes OR dates to distinguish between successive versions [project name]_[dataset]_[processID]_[v1.1].ext [project name]_[dataset]_[processID]_[YYYYMMDD].ext 2025-10-29 Collecting experimental data Summarize basic information about data in a structured way to make the data comprehensible and machine-readable. structured attribute-value pairs 2025-10-29 https://ikai.philfak.unikoeln.de/sites/ikai/_ processed_/c/0/c sm_Smartphone_Cy bertracker_201910 14_153619_672347b 468.jpg QField Smart audio data collector (Smatrix) https://docs.qfield.org/de/how-to/digitize/ Metadata - “Data about Data” ✓Can be understood and interpreted by people who were not involved in their collection ✓Ensure long-term usage ✓Can be machine readable 2025-10-29 42 Species abundance Helgoland 20th January 2023 Bottom Trawling Hydractinia echinata (Fleming, 1828) 54°11′00″N, 7°53′34″E 20 m below NHN Subtidal Who What When Where How Daniel Collecting data from online sources 2025-10-29 How do you choose the right database? Collecting data from online sources Repositories/databases should be trustworthy Check the websites •Information about provenance •Licensing information •Information about validation criteria •Choose repositories based on your data type https://www.re3data.org/, https://fairsharing.org/ •Example: https://www.ioer-monitor.de/methodik/#c245 2025-10-29 Storage vs. Preservation Storage = mid-term •During active project phase •Local computer or server •Institutional cloud storage Preservation = long-term •Aims at integrity of data •Long-term repositories with curation routines •Good scientific practice = 10 y Accessibility: Data can be retrieved, displayed and used. Authenticity: Data have not been manipulated or faked. Longevity: Data are reusable for long-term, independently of software and hardware decay. 3… 2… 1… backup! at least 3 copies of a file on at least 2 different media with at least 1 off site 2025-10-29 2025-10-29 ELIXIR (2021) Research Data Management Kit. A deliverable from the EU-funded ELIXIR-CONVERGE project (grant agreement 871075). URL: https://rdmkit.elixir-europe.org Top 10 data issues •Table descriptions, markings, plots, statistics, empty rows/columns, additional comments in data table •Wrong data types (e.g. float, int, object) •Wrong date/time format & UTC vs local time •Wrong format of latitude/longitude •Ambiguous NAs (e.g. nan, N/A, -999.99) •Abbreviations •Excessive decimal points ≠ sensor accuracy •Wrong decimal separator is ‘,’ instead of ‘.’ •Spelling, e.g. in species names •Leading/trailing/double white spaces 2025-10-29 NFDI4Biodiversity Tools – Data cleaning •Standardise and clean data (e.g. CSV, JSON, XML, ODS, XLS) •Get an overview of a data set •Resolve inconsistencies •Split data up into more granular parts •Match local data up to other data sets •Enhance a data set with data from other sources •Save a set of data cleaning steps to replay on multiple files •Free, open source desktop application that uses web browser as a graphical interface •No internet connection needed •Data or commands are not sent to a remote server •Does not modify your original dataset •All actions can be reversed, captured and shared 2025-10-29 https://openrefine.org NFDI4Biodiversity Tools – Data annotation •Add ontologies to Excel spreadsheets •Standardise sampling protocols •Enables users to import Excel spreadsheets, or generate new ones from scratch. •Ontologies can be imported from local file systems, the web, or from the BioPortal ontology repository •Individual cells, or whole columns or rows can be marked with the required ranges of ontology terms and an •Individual spreadsheet can be annotated with terms from multiple ontologies •Free, open source Java application interacting with Microsoft documents •Enables researcher to consistently annotate data without the need to explore and understand the numerous standards and ontologies available •Everything is embedded in the Excel spreadsheet 2025-10-29 https://rightfield.org.uk/about.html 2025-10-29 ELIXIR (2021) Research Data Management Kit. A deliverable from the EU-funded ELIXIR-CONVERGE project (grant agreement 871075). URL: https://rdmkit.elixir-europe.org FAIR Data analysis 2025-10-29 Provide your… •code; e.g. Git repositories (~64% not at all comfortable) •execution environment; e.g. Bioconda, binder project (~90% not at all comfortable) •data analysis workflows; e.g. Jupyter, R Markdown (~40% not at all comfortable) This is the part you know… …but how about this? Data analysis e.g. R, Python 82% Advanced 18% Basic Provide analysis workflows using Git 2025-10-29 Git is a distributed version control system (VCS) •Enables version tracking, branching, and merging •Facilitates collaboration •Cross-platform •Open source, free •Most common tool worldwide Provide analysis workflows using Git 2025-10-29 Code or data table Researchers Share Merge Local repository Remote repository 2025-10-29 ELIXIR (2021) Research Data Management Kit. A deliverable from the EU-funded ELIXIR-CONVERGE project (grant agreement 871075). URL: https://rdmkit.elixir-europe.org Storage vs. Preservation or: preservation is more than a backup Storage = mid-term •During active project phase •Local computer or server •Institutional cloud storage Preservation = long-term •Aims at integrity of data •Long-term repositories with curation routines •Good scientific practice = 10 y Accessibility: Data can be retrieved, displayed and used. Authenticity: Data have not been manipulated or faked. Longevity: Data are reusable for long-term, independently of software and hardware decay. 3… 2… 1… backup! at least 3 copies of a file on at least 2 different media with at least 1 off site 2025-10-29 Data preservation 2025-10-29 •Preserve data for at least 10 years after the end of the project as required by funder, publisher and institution policies •Preserve as many data as possible, especially unique data or that cannot be easily re-generated; e.g. raw data, analysis, workflows Services in NFDI4Biodiversity 2025-10-29 Data submission, versioning and publication Including long-term preservation (extended GFBio services) Helpdesk Individual support for researchers and data centers (extended GFBio service) Support with integration and harmonization of data (GFBio data centers) Provision of collaborative workspaces With support for scientific workflows and provenance management Education and Training Tailored events, tools and materials for teaching Basic tools for data managers Validation, transformation, automated quality checks various de.NBI tools Terminology service (extended GFBio service) Search portals and API for data and tools (extended GFBio services) Elastic compute service (Infrastructureas-a-service) In future: Research Data Commons Services 2025-10-29 GFBio Service: DMP Support 2025-10-29 1. Use the GFBio DMP tool https://dmp.gfbio.org 2. Send a support request We will 3. Review your input 4. Get in contact with you 5. Provide your personalized DMP GFBio Service: DMP Support 2025-10-29 4. Get in contact with you 5. Provide your personalized DMP GFBio Service: Collaborating Workbenches 2025-10-29 https://www.gfbio.org/tools GFBio Service: Data Submission and Brokerage 2025-10-29 https://submissions.gfbio.org GFBio Service: Data Submission 2025-10-29 •Single drop-off point for heterogeneous data •Individual support •Submission templates •Persistent Identifiers (e.g. DOI) •Individual open access agreements (embargo) https://submissions.gfbio.org Data types and repositories Typical types of data our partners can handle: ●Occurrence Data (observations of species) ●Environmental data (e.g. temperature, rainfall) ●Trait data (e.g. seed number and mass) ●Molecular data (e.g. sequences) ●Experimental and laboratory measurements ●Multimedia (photographs, audio, video), e.g. of observed specimens ●Orthophotos produced using a drone ●Digital surface models ●Model code ●Statistics Data Centers at Natural Science Collections Data Centers specialized on Plant, Nucleotide, and Environmental Data 2025-10-29 https://www.nfdi4biodiversity.org/datenzentren/ GFBio Service: Data Search 2025-10-29 map filters Semantic search (beta version) using the GFBio terminology service https://search.gfbio.org GFBio Service: Data Integration and Analysis 2025-10-29 https://vat.gfbio.org Visualization, Analysis & Transformation of spatio-temporal (biodiversity) data