DataTools4Heart_Milestone MS10_3 FL innovations implemented, optimised and tested across the network, i.e. Centre Dropout, Unbiased Aggregation and Uncertainty-Awareness

Page 1 of 11 DataTools4Heart A European Health Data Toolbox for Enhancing Cardiology Data Interoperability, Reusability and Privacy Milestone MS10 3 FL innovations implemented, optimised and tested across the network, i.e. Centre Dropout, Unbiased Aggregation and Uncertainty-Awareness Reference MS10_ DataTools4Heart_UB_30092025 Lead Beneficiary University of Barcelona Author(s) Grzegorz Skorupko , Jorge Fabila, Cristian Izquierdo, Xenia Puig. Dissemination level Public Type Milestone Official Delivery Date 30/09/2025 Date of validation of the WP leader 15/09/2025 Date of validation by the Project Coordinator 26/09/2025 Project Coordinator Signature DataTools4Heart is funded by the European Union’s Horizon Europe Framework Under Grant Agreement No. 101057849. Page 2 of 11 Version Log Issue Date Version Involved Comments First draft V1.0 Jorge Fabila, Grzegorz Skorupko, Cristian Izquierdo 1st Draft 30/09/2025 V2.0 Cristian Izquierdo, Xenia Puig, Karim Lekadir Final Version Executive Summary This milestone serves as a proof that the three different FL implementations (Centre Dropout, Unbiased Aggregation and Uncertainty Awareness) have been designed and tested internally, and ready for deployment. Page 3 of 11 Table of Contents Version Log ............................................................................................................................................ 2 Executive Summary ............................................................................................................................... 2 Acronyms ............................................................................................................................................... 3 Centre Dropout: Experiments and Evaluation ....................................................................................... 4 Method Overview ............................................................................................................................... 4 Experimental Setup ............................................................................................................................ 4 Experimental Results ......................................................................................................................... 4 Training Time and Efficiency .............................................................................................................. 6 Experiments Summary ....................................................................................................................... 7 Weight Smoothing: Experiments and Evaluation .................................................................................. 7 Method Overview ............................................................................................................................... 7 Experimental Setup ............................................................................................................................ 7 Experimental Results ......................................................................................................................... 7 Averaged Performance Across Centres ............................................................................................ 8 Experiments Summary ....................................................................................................................... 8 Uncertainty aware Federated Learning ................................................................................................. 9 Conclusions .......................................................................................................................................... 11 Bibliography ......................................................................................................................................... 11 Acronyms FL: Federated Learning CVD: Cardiovascular Disease Page 4 of 11 Centre Dropout: Experiments and Evaluation Method Overview Centre Dropout is a methodological extension of federated learning aimed at improving both efficiency and fairness in collaborative model training, introduced by [1]. Instead of requiring all participating institutions to contribute updates in every round, Centre Dropout randomly or selectively excludes a fraction of centres while proportionally scaling the training load of the remaining ones. This reduces communication overhead, accelerates training by avoiding synchronization with the slowest sites, and ensures that smaller centres are not consistently overshadowed by large institutions in the aggregation process. Over multiple rounds, the approach balances institutional contributions while preserving overall training effort. This method is designed to provide a flexible middle ground between standard federated averaging and more sequential training schemes, making it highly suitable for heterogeneous medical datasets. Experimental Setup To assess the effectiveness of Centre Dropout in practice, we conducted experiments using the UK Biobank dataset. Due to delays in data access from partnering institutions, this large-scale publicly available dataset was selected to provide a representative benchmark for federated learning experiments. ● Dataset: 225,355 samples originating from 12 acquisition centres were considered in their original distribution. ● Prediction task: From this cohort, 3,350 patients with cardiovascular disease (CVD) as the main cause of death were selected for the classification task. ● Model: A Logistic Regression classifier trained on 12 clinical features was used as the baseline model. ● Evaluation metric: Balanced Accuracy was chosen to account for the significant label imbalance present in the training data. ● Validation strategy: Data from the Stoke centre was entirely held out for external validation. ● Reproducibility: Each run was repeated eight times with randomized splits to ensure robustness of results. Experimental Results Three sets of analyses were performed: 1. Sample distribution across centres Fig. 1 illustrates the distribution of available samples per acquisition centre. The dataset was highly imbalanced, with some centres contributing a disproportionately large number of cases (Newcastle) while others remained relatively underrepresented (Wrexham, Swansea). Page 5 of 11 Figure 1: Distribution of samples across 12 acquisition centres in the UKBiobank dataset. 2. Federated vs. local training performance Fig. 2 compares local models (trained independently per centre) with the federated model. Results demonstrate a consistent performance improvement of the federated approach, particularly for smaller centres that otherwise suffer from insufficient training data. This confirms the expected benefit of FL in enhancing generalization across heterogeneous clinical sites. Figure 2: Comparison of Balanced Accuracy between local and federated training across centres. A trend of improved performance for federated training, especially in smaller centres, is observed. Page 6 of 11 3. Benchmarking Centre Dropout methods Fig. 3 presents results from experiments with different Centre Dropout variants, including the “Less participants every odd round” strategy at dropout fractions of 0.3, 0.7, and 0.8. Overall, performance remained comparable to the baseline federated model (all centres participating in every round). Interestingly, the smallest centre (Wrexham) achieved a mild performance increase under the “Less participants every odd” configuration, as it contributed proportionally more often to the aggregation process. Figure 3: Benchmark of Centre Dropout methods (“Less participants every odd round”) with varying dropout fractions (0.3, 0.7, 0.8). Performance remains comparable to the baseline, with mild improvement for the smallest centre (Wrexham). Training Time and Efficiency To quantify the efficiency gains, we compared training times across Centre Dropout configurations. The results are summarized in Table 1. All dropout variants resulted in reduced training time compared to standard FL. The “Less participants every odd (0.7)” setup achieved the largest efficiency improvement, reducing training time by approximately 20% while incurring only a 0.6% loss in Balanced Accuracy. Table 1: Training time reduction and Balanced Accuracy performance under different Centre Dropout configurations. Method Balanced Accuracy Training time [s] Locally trained 0.675 ±0.018 - Federated (baseline) 0.723 ±0.013 72.103 ±1.121 Centre dropout: Random dropout (0.5) 0.723 ±0.012 68.979 ±2.332 Fast at odd rounds (0.3) 0.721 ±0.012 64.479 ±1.824 Less participants every odd (0.3) 0.722 ±0.012 64.601 ±0.951 Less participants every odd (0.7) 0.719 ±0.011 57.256 ±1.972 Less participants every odd (0.8) 0.711 ±0.014 59.676 ±1.905 Page 7 of 11 Experiments Summary These findings highlight the potential of Centre Dropout to accelerate federated training without compromising predictive performance. Configurations with higher dropout fractions (e.g., 0.7) deliver substantial efficiency gains while maintaining nearly identical Balanced Accuracy. This confirms the method’s value as a practical mechanism to scale federated learning across heterogeneous, multicentre healthcare datasets. As a next step, the consortium will extend these experiments to real-world partner datasets as soon as access is available, to validate Centre Dropout under the specific conditions of cross-institutional medical data integration Weight Smoothing: Experiments and Evaluation Method Overview Weight Smoothing is a federated learning strategy designed to reduce bias towards data-rich centres during the aggregation phase, proposed by [1]. In conventional federated averaging [2], client updates are weighted proportionally to dataset size, which can lead to overrepresentation of large centres and underrepresentation of smaller ones. Weight Smoothing mitigates this issue by adjusting the aggregation weights along a spectrum between equal centre contributions (balanced voting) and sample-size–based contributions. By controlling the balance between these two extremes, the method aims to promote fairness while preserving the statistical robustness of the global model. Experimental Setup To evaluate Weight Smoothing within the project, we conducted experiments using the UK Biobank dataset in the same setup as for Centre Dropout evaluation. Due to delays in data access from partnering institutions, this dataset provided a representative large-scale benchmark. ● Dataset: 225,355 samples from 12 acquisition centres in the original distribution. ● Prediction task: Selection of 3,350 patients with cardiovascular disease (CVD) as the main cause of death for binary classification. ● Model: Logistic Regression trained on 12 input features. ● Evaluation metric: Balanced Accuracy, reflecting the strong class imbalance in the dataset. ● Validation strategy: Data from the Stoke centre was held out entirely for external validation. ● Reproducibility: Each experiment was repeated eight times with randomized splits. Experimental Results Three Weight Smoothing strategies were compared: ● Equal sample weights (federated baseline): aggregation proportional to the number of samples per centre. ● Equal centre weights: all centres contribute equally, independent of dataset size. ● Lower quartile smoothing hybrid scheme weighting 0.25 centre-balanced and 0.75 samplesize–based contributions. Page 8 of 11 Fig. 4 presents Balanced Accuracy across all participating centres for the three methods. In this case, no substantial performance differences were observed between strategies. This outcome may be explained by the relatively homogeneous population distribution across UK Biobank centres, which limits cross-site variability. Additionally, since Logistic Regression involves far fewer trainable parameters than deep learning models, the potential benefit of weight smoothing in mitigating aggregation bias is likely reduced compared to prior studies [1]. Figure 4: Balanced Accuracy across centres for three Weight Smoothing strategies: Equal sample weights, Equal centre weights, and Lower quartile smoothing. Averaged Performance Across Centres To further assess the effects, results were averaged across all centres (Table 2). The Equal sample weights method (baseline) provided the best overall performance. Nevertheless, all Weight Smoothing configurations outperformed locally trained models, confirming the advantage of federated training even under simple linear modelling assumptions. Table 2. Averaged Balanced Accuracy across centres under different Weight Smoothing configurations. Method Balanced Accuracy Training time [s] Locally trained 0.718 ±0.012 - Weight Smoothing: Equal sample weights (baseline) 0.723 ±0.013 72.103 ±1.121 Equal centre weights 0.721 ±0.014 73.818 ±3.774 Lower Quartile 0.723 ±0.013 71.611 ±1.527 Experiments Summary The results suggest that Weight Smoothing did not substantially affect model performance in the current setup, likely due to the homogeneous sample distribution across UK Biobank centres and the simplicity of the Logistic Regression model. Nonetheless, the experiments confirm that federated approaches consistently outperform local training, and more pronounced effects of Weight Smoothing are expected in future experiments with more heterogeneous partner datasets from DT4H consortium. Page 9 of 11 Uncertainty aware Federated Learning In many machine learning applications, especially in AI for healthcare, it is important not only to predict labels accurately but also to estimate the uncertainty of those predictions. Traditional Bayesian Neural Networks (BNNs) explicitly model the posterior distribution over weights but are often computationally expensive and difficult to scale. Monte Carlo (MC) Dropout offers a practical approximation to Bayesian inference. It interprets dropout (originally designed as a regularization technique) to sample from an approximate posterior [3]. A model with high variance means it is not confident about its predictions. One way to include this information in the federated setting is to weight this measure during the aggregation process. Federated averaging usually weights model contributions based on sample size, to avoid overrepresenting centres with many samples and to balance the aggregation. In this case, we also include entropy, which is a measure of the models’ variance: models with higher variance (i.e., greater uncertainty) contribute less than those with lower variance. Entropy (H, as was called by Shannon) is defined as: Where each prediction p is the average of the T stochastic forward passes: And each forward pass is the output of a given NN: Starting from the aggregation based on number of samples we have: The weights are defined based on the number of samples and the entropy (H), ϵ is only to avoid dividing by 0: