The Hubverse: Streamlining Collaborative Infectious Disease Modeling for Public Health Impact
Krystalli, Anna; Consortium of Infectious Disease Modeling Hubs
- Publisher
- Zenodo
- Language
- en
Abstract
Predictive models have become an essential tool for public health decision-making during infectious disease outbreaks [1]. Yet the rapid proliferation of models, especially during the COVID-19 pandemic, has created a fragmented landscape marked by inconsistent metrics, overlapping or conflicting forecasts, and limited comparability. This has posed serious challenges for decision-makers trying to identify reliable, policy-relevant insights. Collaborative modeling hubs offer a promising solution by coordinating model submissions, promoting transparency, and facilitating ensemble modeling, where aggregated model outputs usually outperform individual ones. Hubs also improve communication between modeling teams and stakeholders by aligning outputs with public health priorities. The hubverse [2] is a modular, open-source software ecosystem designed to support the setup and operation of these hubs. It introduces a set of data standards for probabilistic model output data, as well as utilities for setting up and administering a hub, validation and ensembling tools, visualization templates, and mechanisms for model evaluation and public-facing communication. The hubverse defines five primary user roles: hub administrators, modelers, analysts, stakeholders, and developers, and supports each with tailored tools. Importantly, the hubverse is built primarily on open-source software (R, Python, JavaScript, Arrow) and freely available platforms like GitHub, making it accessible even to research groups with limited resources. While hub administration remains the most technical aspect, the ecosystem is designed to reduce barriers across all roles. Automation, templating, and interactive interfaces lower the technical burden and make engaging meaningfully with hub products easier for a wider range of users, including public health stakeholders. This talk introduces the hubverse through real-world examples, including its recent adoption by the CDC’s FluSight influenza forecasting hub. We will highlight how this infrastructure is helping standardize infectious disease modeling efforts and support evidence-based decision-making at all levels of public health response.
Full text
The Hubverse: Streamlining Collaborative Infectious Disease Modeling US-RSE Conference 2025 Anna Krystalli 7 October 2025 [email protected] R-RSE SMPC Consortium of Infectious Disease Modeling Hubs 1
Background 3
❌ The problem Infectious disease modeling has scaled rapidly… But the landscape is fragmented: Inconsistent formats Redundant or conflicting forecasts Lack of coordination between modelers and stakeholders “Comparing the accuracy of forecasting applications is difficult because forecasting methods, forecast outcomes, and reported validation metrics varied widely.” Chretien et al., PLOS ONE, 2014 4
✨ The promise of modeling hubs Modeling hubs coordinate collaborative forecasting: Provide centralised location for effort coordination Define data standards and modeling targets Improve transparency and comparability Aggregate forecasts enabling ensembles Facilitate timely public health decision-making , American Journal of Public Health Reich, et al.2022 “Collaborative Hubs: Making the Most of Predictive Epidemic Modeling” 5
🕰 Project origins Pre-COVID: Forecasting code base existed for CDC influenza hubs During COVID: That code was reused for new COVID-19 hubs + demand internationally (e.g.Europe) for similar setups ❗ Problem: Each hub required manual editing of source code ➡ Need for generalisation, modularity, and configurability Figure credits: Alex Vespignani and Nicole Samay 6
🌐 Enter the hubverse An open-source software ecosystem to power modeling hubs: GitHub repositories for centralising hub activity Data standards for infectious disease modeling data Schema-driven configuration for modeling tasks + hub setup Modular tools for validation, access, evaluation, ensembling, communication and hub administration Supports full lifecycle: from hub set up, data submission to decision-making 7
Hubverse overview 9
☑ Standardised Data Modeling hubs are built around a shared data standard: Modeling task definition: targets (response variables), standard predictors, output types (e.g. mean , quantiles ) Structured hub layout: consistent file system for organizing submissions Standard model output format: for file content and naming ✅ Enables comparability, validation, and streamlined data access 10
⚙ Config-driven hub setup Hub administrators configure hubs using structured JSON config files: admin.json : hub-level metadata. task.json : modeling task specification: Task IDs: Targets (response), horizons, locations (predictors) etc. Output types: accepted model outputs e.g. mean , median , quantiles , cdf , pmf , samples . Configs are validated against a shared JSON schema 11
Real-world example: Used by US CDC to monitor influenza severity Weekly forecasts from 40 teams across 70 different models. Hosted on GitHub + S3 cloud mirror. Managed using full hubverse stack since 2023/2024 season. CDC FluSight Hub https://github.com/cdcepi/FluSight-forecast-hub 19
📁 File structure: model output (CDC FluSight) Model outputs committed by teams to versioned directories > one directory per model > one file per modeling round. 20
✅ Model output validation with Model outputs submitted through PRs and validated through GitHub Actions hubValidations 21
📂 Accessing model output via Connect to Arrow dataset of forecast submissions Query and collect data See more in . Python analogue also available. hubData library(hubData)1 2 hub_path <- s3_bucket(3 "cdcepi-flusight-forecast-hub"4 )5 hub_con <- connect_hub(6 hub_path,7 skip_checks = TRUE8 )9 hub_con10 hub_connection 9 columns reference_date: date32[day] target: string horizon: int32 target_end_date: date32[day] location: string output_type: string output_type_id: string value: double model_id: string # Filter for one model and forecast date using dplyr1 library(dplyr)2 hub_con |>3 filter(4 model_id == "CADPH-FluCAT_Ensemble",5 target_end_date == "2023-10-28"6 ) |>7 collect_hub()8 # A tibble: 92 × 9 model_id reference_date target horizon target_end_date location output_type * <chr> <date> <chr> <int> <date> <chr> <chr> 1 CADPH-Flu… 2023-10-14 wk in… 2 2023-10-28 06 quantile 2 CADPH-Flu… 2023-10-14 wk in… 2 2023-10-28 06 quantile 3 CADPH-Flu… 2023-10-14 wk in… 2 2023-10-28 06 quantile 4 CADPH-Flu… 2023-10-14 wk in… 2 2023-10-28 06 quantile 5 CADPH-Flu… 2023-10-14 wk in… 2 2023-10-28 06 quantile 6 CADPH-Flu… 2023-10-14 wk in… 2 2023-10-28 06 quantile 7 CADPH-Flu… 2023-10-14 wk in… 2 2023-10-28 06 quantile 8 CADPH-Flu… 2023-10-14 wk in… 2 2023-10-28 06 quantile 9 CADPH-Flu… 2023-10-14 wk in… 2 2023-10-28 06 quantile 10 CADPH-Flu… 2023-10-14 wk in… 2 2023-10-28 06 quantile # ℹ 82 more rows # ℹ 2 more variables: output_type_id <chr>, value <dbl> Accessing data vignette hub-data 22
🌐 Ensembling with hubEnsembles Combine models using simple or weighted rules forecast_df <- hub_con |>1 filter(2 model_id %in%3 c(4 "CADPH-FluCAT_Ensemble",5 "CEPH-Rtrend_fluH",6 "CFA_Pyrenew-Pyrenew_HE_Flu"7 ),8 output_type == "quantile"9 ) |>10 collect_hub()11 12 13 hubEnsembles::simple_ensemble(14 forecast_df, 15 agg_fun = median,16 model_id = "simple-ensemble-median"17 )18 # A tibble: 282,716 × 9 model_id reference_date target horizon target_end_date location output_type * <chr> <date> <chr> <int> <date> <chr> <chr> 1 simple-en… 2023-10-14 wk in… -1 2023-10-07 01 quantile 2 simple-en… 2023-10-14 wk in… -1 2023-10-07 01 quantile 3 simple-en… 2023-10-14 wk in… -1 2023-10-07 01 quantile 4 simple-en… 2023-10-14 wk in… -1 2023-10-07 01 quantile 5 simple-en… 2023-10-14 wk in… -1 2023-10-07 01 quantile 6 simple-en… 2023-10-14 wk in… -1 2023-10-07 01 quantile 7 simple-en… 2023-10-14 wk in… -1 2023-10-07 01 quantile 8 simple-en… 2023-10-14 wk in… -1 2023-10-07 01 quantile 9 simple-en… 2023-10-14 wk in… -1 2023-10-07 01 quantile 10 simple-en… 2023-10-14 wk in… -1 2023-10-07 01 quantile # ℹ 282,706 more rows # ℹ 2 more variables: output_type_id <chr>, value <dbl> 23
📈 Dashboard - forecasts 24
🩺 Dashboard - Evaluates forecasts against target (observed) data. model evaluations 25
💡 Lessons & wider relevance ✅ Standards + automation reduce friction 🧰 Open source keeps it free & accessible 🏥 Collaborative infrastructure empowers public health 🌍 Standardised, open data fuels downstream use cases like training, education, and reproducible research Introduction So far in this course we have focused on building, visualising combining “toy” forecast models in somewhat synthetic sett you will work with real forecasts from an existing modeling h many of the challenges involved with real-time forecasting, a f di d d li ff Evaluating real-world outbreak fo NFIDD SI… Getting started Authors Sessions Reference https://nfidd.github.io/sismid/sessions/real-worldforecasts.html 26
