DataTools4Heart_Milestone MS11_Dedicated project space for DT4H available at the OpenEBench benchmarking platform

Page 1 of 6 DataTools4Heart A European Health Data Toolbox for Enhancing Cardiology Data Interoperability, Reusability and Privacy Milestone 11: Dedicated project space for DT4H available at the OpenEBench benchmarking platform Reference MS11_ DataTools4Heart_BSC_30092025 Lead Beneficiary BSC Author(s) Carles Hernandez-Ferrer, Laia Codó, Josep Lluís Gelpí Dissemination level Public Type - Official Delivery Date 30/09/2025 Date of validation of the WP leader 30/09/2025 Date of validation by the Project Coordinator 30/09/2025 Project Coordinator Signature DataTools4Heart is funded by the European Union’s Horizon Europe Framework Under Grant Agreement No. 101057849. Page 2 of 6 Version Log Issue Date Version Involved Comments 29/09/2025 0.1 Carles Hernandez-Ferrer Initial version 30/09/2025 0.2 Laia Codó, Josep Lluís Gelpí Second version 30/09/2025 Final Cristian Izquierdo, Xènia Puig, Karim Lekadir Revised and corrected final version Executive Summary This milestone (MS11) reports the availability of a dedicated project space for DataTools4Heart (DT4H) within the ELIXIR OpenEBench benchmarking platform. The DT4H benchmarking hub will systematically integrate benchmarking efforts undertaken across the project on tools, scientific outcomes, user experience, software quality, and overall performance. It provides a centralised and FAIR-compliant environment for transparent, reproducible, and community-driven benchmarking of DT4H toolbox components (e.g., NLP models, federated learning algorithms, synthetic datasets). Page 3 of 6 Table of Contents Version Log ............................................................................................................................................ 2 Executive Summary ............................................................................................................................... 2 Acronyms ............................................................................................................................................... 3 List of tables ........................................................................................................................................... 3 List of figures .......................................................................................................................................... 3 1 Introduction ......................................................................................................................................... 4 2 Benchmarking in DT4H ....................................................................................................................... 4 3 Infrastructure ....................................................................................................................................... 5 3.1 DT4H @ OpenEBench .................................................................................................... 5 4 Future Work ............................................................................................................................. 6 Acronyms OEB: OpenEBench FL: Federated Learning NLP: Natural Language Processing FAIR: Findable, Accessible, Interoperable, Reproducible List of tables Table 1: Overview of XXXX (example table caption). 4 List of figures Figure 1: Illustration of XXXX (example figure caption) 5 Page 4 of 6 1 Introduction This document outlines the technical capabilities and operational framework of the benchmarking platform designed as part of the DataTools4Heart (DT4H) European project. It serves as a centralised hub for recording a technical summary of project’s benchmarking decisions, goals, agreements, and capabilities. Crucially, will provide the structured space to: ● List the datasets used for benchmarking (including references to the DT4H's data catalogue). ● Detail the complete roster of benchmarked assets. ● Comprehensive description and results for all benchmarking events. Along the task, the benchmarking hub will be populated by a series of benchmarking efforts performed within the DT4H project frame. The overarching goal is to elucidate the right tool for the right scenario by enabling consistent, objective evaluation across different technical areas. This consistency is critical for advancing the state-of-the-art in heart disease data processing and analysis. 2 Benchmarking in DT4H Benchmarking in DT4H is transversal, covering scientific outcomes, user experience, software quality, and the overall performance of the platform. Assets to be evaluated include: - baseline NLP tools and transformer-based AI models, - federated learning models trained under novel schemas (e.g. centre dropout, unbiased aggregation methods) - synthetic cardiology dataset (CardioSyn) Such benchmarking efforts are carried out by the relevant WPs (see Table 1. Benchmarking-related tasks) and later integrated as structured evaluations into the DT4H OpenEBench portal, ensuring transparent comparison and long-term availability of results. Table 1. Benchmarking-related tasks T6.4 Development of an evaluation environment for DT4H toolbox T3.6 Integration of clinical NLP models into federated learning framework T4.2 Centre dropout for efficient federated learning T5.4 Technical Evaluation of Synthetic Data Sets for Cardiology T4.3 Unbiased aggregation in federated learning T7.4 Multi-centre evaluation and feedback loop Of these, T6.4 is responsible for linking DT4H with OpenEBench and adapting the platform to fit the benchmarking approach followed by each of the other tasks. 3 Infrastructure The main achievement introduced in this milestone is the DT4H benchmarking platform built upon the existing infrastructure of OpenEBench. Page 5 of 6 OpenEBench is the ELIXIR Benchmarking and Technical Monitoring platform. It is a publicly available, community-driven platform designed to centralise and standardise the results of technical assessments, validations, and benchmarks for bioinformatics tools and workflows. Its core capabilities include: ● Standardisation: Providing standardised metrics, methods, and reporting schemas for tool evaluation. ● Centralisation: Acting as a repository for benchmarking results across diverse domains. ● Reproducibility: Facilitating the creation of reproducible benchmarking events, or challenges. ● Community Engagement: Allowing different scientific teams to compare their tools and methods fairly in a transparent environment. OpenEBench provides the necessary framework and technical specifications to ensure DT4H's benchmarking efforts are transparent, consistent, and comparable with wider European life science initiatives. 3.1 DT4H @ OpenEBench DT4H has been integrated at OpenEBench as a new dedicated project space (see Figure 1. DataTools4Heart space at OpenEBench). Accessible at the OpenEBench portal (Home https://openebench.bsc.es/ → Projects → Datatools4Heart card), it will integrate a comprehensive view of DT4H assessment areas, produced metrics, computation pipelines and quality indicators. Participating WPs will contribute evaluation outputs to this space, ensuring harmonisation across the consortium. DataTools4Heart project space https://openebench.bsc.es/projects/OEBC014 Furthermore, this section of the platform acts as a centralised hub of links to external tools, data sources, and other platforms that have been used to benchmark and compare tools, ensuring users have a single point of reference for all related technical documentation. Page 6 of 6 Figure 1. DataTools4Heart space at OpenEBench 4 Future Work While the benchmarking platform is already operational, the current version contains only the skeleton of the future hub. Next steps include keep working in collaboration with the rest of the Benchmarking Working Group for completing the list of evaluable toolbox components (datasets, models, tools) and both: - Organising continuous evaluation and feedback loops (multi-centre, user-driven) - Preparing thematic benchmarking events (e.g., NLP model evaluation, federated learning robustness, synthetic data validation) The result of this effort would be a centralised benchmarking hub for DT4H tools, organised by thematic area (benchmarking event) and task (challenge). Every benchmarking-related task of the DT4H project is to collect and submit its outcome to populate the centralised DT4H space in OpenEBench. Currently, most of these tasks are under progress.