Quality Time: Rethinking Data Trust in the Age of AI and (Cross-Domain) Collaboration

Workshop Documentation Quality Time: Rethinking Data Trust in the Age of AI and (Cross-Domain) Collaboration 22. September 2025, NFDI4Earth Plenum 2025 Hannes Thiemann Andrea Lammert Ivonne Anders Deutsches Klimarechenzentrum (DKRZ), Hamburg Date: 26.11.2025 Citation Thiemann, H., Lammert, A., & Anders, I. (2025). Quality Time: Rethinking Data Trust in the Age of AI and (Cross-Domain) Collaboration. Zenodo. https://doi.org/10.5281/zenodo.17858384 License This work is licensed under a Creative Commons “Attribution 4.0 International” license. Acknowledgement This work has been funded by the German Research Foundation (DFG) through the project NFDI4Earth (DFG project no. 460036893, https://www.nfdi4earth.de/) within the German National Research Data Infrastructure (NFDI, https://www.nfdi.de/). Funded by the German Research Foundation (DFG) through the project NFDI4Earth. Abstract This document summarizes the 90-minute workshop “Quality Time: Rethinking Data Trust in the Age of AI and Cross-Domain Collaboration”, held during the NFDI4Earth Plenum 2025. The workshop combined introductory keynote talks with an interactive World Café to explore interdisciplinary approaches to research data quality, the role of AI in quality assurance, transparent quality cultures, and mechanisms for translating quality concepts across domains. Key messages include the urgent need for scalable, AI-ready data quality procedures, shared principles for communicating uncertainty, and practical blueprints that help communities operationalize interoperable, trustworthy, and purpose-specific data quality across domains. Table of content 1. Background 3 2. Aim of the Workshop 3 3. Workshop format 3 4. Key Note speeches – Fresh Perspectives on Data Quality 4 5. World Café 5 6. Conclusion 6 Funded by the German Research Foundation (DFG) through the project NFDI4Earth. 1. Background The quality of research data is becoming increasingly important in the context of growing data volumes, increasing heterogeneity, and new digital technologies. Feedback on the NFDI4Earth interim report showed that a holistic and cross-consortium approach to data quality is urgently needed. In Earth system sciences, there are different quality cultures and established, often manual procedures that have reached their limits regarding scalability. There is an urgent need to transition from manual curation to AI-assisted workflows to handle the increasing data heterogeneity and volume transparently. Against this backdrop, the workshop was developed to bring together key issues of data quality, highlight existing practices from various disciplines, and discuss ways in which established principles can be meaningfully further developed with innovative technologies such as AI or knowledge graphs, together with partner consortia such as FAIRagro. 2. Aim of the Workshop The workshop aimed to bridge the gap between existing discipline-specific quality practices and the new cross-cutting challenges in dealing with research data. The goal was to critically examine established data quality assurance procedures and, at the same time, demonstrate how modern technologies—especially AI-based tools—can meaningfully expand and scale these practices. Another key aspect was the exchange between different consortia in order to identify common principles, translation mechanisms, and possible joint approaches that both ensure continuity and create space for innovation. 3. Workshop format The workshop was designed as a compact, 90-minute session that combined content input with interactive exchange. At the beginning, participants were given an introduction to the topic through three short keynote speeches, each presenting different perspectives on data quality and future quality assurance. These inputs laid the technical foundations and focused specifically on the topics of human-machine interaction, trustworthy data processes, and interdisciplinary quality cultures. This was followed by a World Café format, which gave participants the opportunity to discuss topics in small groups at three thematically oriented tables. Each round was moderated by the respective expert from the keynote section, allowing for in-depth discussion and practical reflection on the previously established topics. The rotation of the groups created a dynamic exchange in which different perspectives were brought together and key challenges and solutions were collected. The structured combination of technical input and collaborative discussion made it possible to highlight both concrete technical findings and cross-consortium perspectives. Funded by the German Research Foundation (DFG) through the project NFDI4Earth. 4. Key Note speeches – Fresh Perspectives on Data Quality Sebastian Mieruch (AWI) Human + Machine: The new Dream-Team to tackle data QC challenges The first keynote speech addressed the question of how AI can be used to effectively support established quality assurance processes in marine research. The starting point was classic, expert-driven quality management, which, despite its proven precision, is increasingly reaching its limits—especially in view of ever-growing amounts of data. The AI tool SalaciaML presented shows that new technologies cannot replace human expertise, but can scale it. The algorithm was trained on expert-curated data, can detect misclassifications, and serves as an adaptive assistant that makes quality control more efficient. At the same time, the final evaluation remains the task of human experts. The presentation demonstrated that while AI is essential for QC, it requires a robust "Human-in-the-loop" approach. AI agents should serve as adaptive assistants for pre-processing, leaving the final trust-building validation to human experts. Marthe Klöcking (Uni Münster) Turning Rubble into Gold: making data quality shine for Open Science The second presentation introduced the GeoROC geochemical data repository, which serves as an example of highly curated, trustworthy data collections. The presentation contrasted GeoROC's quality assurance with uncontrolled data collections, such as those from web scraping processes. Particular attention was paid to the challenges of heterogeneous long-tail data and the enormous manual effort involved in its curation. At the same time, it was emphasized that these curated collections are by no means outdated. The presentation showed that quality consists not only of data itself, but also of transparent processes and comprehensible criteria that must remain continuously visible and scalable, especially with regard to open science in practice. Markus Möller (JKI, FAIRagro) Bridges not Walls: How can we build interoperability between diverse quality cultures without forcing limiting standards? The third impulse showed how different quality cultures can be linked without restricting them through uniform standards. The Digital Soil Mapping Community clearly demonstrated that the central challenges lie primarily in communicating quality and uncertainties. Quality metrics are often only understandable within individual disciplines and are rarely communicated in a way that is appropriate to the scale. It was emphasized that data quality is always purpose-specific (“data fitness for purpose”) and must therefore be contextualized. He presented concepts such as the Scaling Ladder and the SICOM framework, which help to structure complex quality information and make it comparable across different levels. Interoperability is achieved not so much through enforced standardization, but through translation mechanisms that make different quality logics visible and connectable. Modern technologies such as LLMs have been identified as key factors in building these bridges between different quality cultures. Funded by the German Research Foundation (DFG) through the project NFDI4Earth. 5. World Café Three parallel moderated tables were set up in the workshop, which were supervised by the three keynote speakers. The participants rotated in groups from table to table and discussed different key topics. Table 1 – “Humans & Machines” (Andrea / Sebastian Mieruch) This table focused on the role of humans in interaction with AI tools for quality assurance. The discussion centered on the greatest opportunities and obstacles in the use of AI in everyday work, such as a lack of trust, unclear responsibilities, and insufficient transparency in algorithmic decisions. Another focus was on the question of which tasks should remain in human hands and which activities could be reliably performed by machines. Summary of the discussion Both, the opportunities and the risks were discussed in detail and both are considered to be high. The loss of human perspective and experience is offset by significant time savings and automation. These contradictions can only be resolved through transparency, excellent and repeatedly validated test data, and regular quality controls. Table 2 – “The Code of Trust” (Hannes / Marthe Klöcking) The second table focused on the topic of transparent and open quality culture. The discussion revolved around the question of how implicit expert knowledge can be translated into explicit, machine-readable, and FAIR-accessible quality information. The participants exchanged views on what information they typically lack when they want to use a data set from a different field and what elements a “QC certificate” would absolutely have to contain. In addition, cultural barriers that stand in the way of an open quality culture were discussed – for example, a lack of incentives or concerns about evaluation – as well as possible ways to overcome these hurdles. Summary of the discussion The discussion showed that key quality characteristics such as consistency, accuracy, completeness, and rich metadata are essential for trustworthy data. Participants emphasized that both raw data and processes must be documented transparently and that unified quality standards are still lacking within the community. Human expertise remains important in quality control despite increasing automation. Major challenges include handling very large datasets and the absence of incentives for quality work. At the same time, it was highlighted that high-quality data is crucial for AI, and that AI itself can help improve quality processes. Table 3 – “Bridges instead of walls” (Ivonne / Markus Möller) The third table discussed how interdisciplinary quality cultures can be linked. Instead of focusing on standardization, the focus was on translation mechanisms that make quality information from one discipline usable for others. Funded by the German Research Foundation (DFG) through the project NFDI4Earth. Practical examples were discussed in which participants had worked with data from other disciplines and wondered what rules were used to assess quality there. Overarching principles of good data quality – such as transparency, traceability, and contextualization – were identified. Summary of the discussion The discussion highlighted that clear quality categories and transparent uncertainty information are essential for reliable data use. However, missing or inconsistent standards make comparability and communication between data producers and users difficult. Uncertainties arise at many stages and are often not documented sufficiently. At the same time, there is little motivation to provide quality metrics because the effort is high and the benefits are not always visible. Good organizational structures and shared practices facilitate communication and support better quality processes. Overall, there is a clear need for understandable standards, common metrics, and practical guidance to make data quality more consistent and usable. 6. Conclusion The workshop impressively demonstrated the extent of shared interest in sustainable quality assurance across disciplines. Participants engaged enthusiastically in the discussions and contributed a wide range of experiences from their respective professional contexts. Across the groups, it became evident that AI-readiness - specifically the quality of training data - is a critical bottleneck. Although there is broad agreement, that high-quality data is essential, the community currently lacks unified criteria or semantic standards to define it, as requirements differ strongly between domains. Participants emphasized that developing such criteria is urgent, since lacking or inconsistent quality measures risk undermining the reliability and robustness of future AI-systems in Earth System Sciences. There was broad consensus that manual curation alone is no longer sufficient to handle the increasing data heterogeneity. Combining established quality practices with AI-assisted automation was identified as a key opportunity to scale quality assurance processes effectively. However, discussion also revealed a gap in practical implementation guidance. While the community is ready to adopt new technologies, participants expressed a strong demand for reusable “Blueprints” and validated workflows that operationalize data quality and semantic interoperability for diverse institutional contexts. Overall, the workshop highlighted that while different perspectives on transparency and standardization exist, these lead to valuable exchange rather that contradiction. The event confirmed that the community is eager to move from theoretical discussion to the practical application of interoperable, AI-supported solutions. Funded by the German Research Foundation (DFG) through the project NFDI4Earth. Annex World Café notes - Table 1 - “Humans & Machines” (Andrea Lammert/ Sebastian Mieruch) Funded by the German Research Foundation (DFG) through the project NFDI4Earth. World Café notes - Table 2 - “The Code of Trust” (Hannes Thiemann / Marthe Klöcking) Funded by the German Research Foundation (DFG) through the project NFDI4Earth. World Café notes - Table 3 - “Bridges instead of walls” (Ivonne Anders / Markus Möller) Funded by the German Research Foundation (DFG) through the project NFDI4Earth.