Cloud-based Workflows for Antarctic Atmospheric Rivers: Successes and Challenges

Cloud-based Workflows for Antarctic Atmospheric Rivers: Successes and Challenges James Butler1, Michelle Maclennan2, Fernando Pérez1, Jon McAuliffe1,3 1UC Berkeley, Department of Statistics, 2British Antarctic Survey, 3The Voleon Group December 16, 2025 AGU25 Session IN23A Open-Source Geospatial Workflows in the Cloud: Tools and Techniques for Data Access, Analysis, Visualization, Storytelling, and Sharing in the Python and Jupyter Ecosystem II Oral What are Antarctic Atmospheric Rivers (ARs)? Varied impacts on Antarctic Ice Sheet Credit: NASA Antarctica What are Antarctic Atmospheric Rivers (ARs)? Varied impacts on Antarctic Ice Sheet Credit: NASA Antarctica What are Antarctic Atmospheric Rivers (ARs)? Varied impacts on Antarctic Ice Sheet Credit: NASA Antarctica What are Antarctic Atmospheric Rivers (ARs)? Varied impacts on Antarctic Ice Sheet Credit: NASA Credit: NASA Larsen B collapse Antarctica What are Antarctic Atmospheric Rivers (ARs)? Varied impacts on Antarctic Ice Sheet Credit: NASA Blanchard-Wrigglesworth (2023) Credit: NASA Larsen B collapse 40°C T. Anomaly Antarctica A Question To Tackle On a multidecadal, Antarctic-wide, storm-by-storm basis, how are their landfalling impacts and characteristics associated? A Question To Tackle On a multidecadal, Antarctic-wide, storm-by-storm basis, how are their landfalling impacts and characteristics associated? … but there’s no single storm-by-storm catalog on climatological scales A Question To Tackle On a multidecadal, Antarctic-wide, storm-by-storm basis, how are their landfalling impacts and characteristics associated? … but there’s no single storm-by-storm catalog on climatological scales Preliminary work for our study 1. Catalog of AR storm events 2. Workflow for atmospheric impacts + characteristics … A Cloud-Based Workflow n=3101 Stream days from S3 via earthaccess … search_data() open() xr.open_mfdataset() List of DataGranules Pointers to nc4 in S3 Chunked Dataset A Cloud-Based Workflow n=3101 Stream days from S3 via earthaccess Subset + Mask Variable DataArray … search_data() open() xr.open_mfdataset() 2m temperature List of DataGranules Pointers to nc4 in S3 Chunked Dataset A Cloud-Based Workflow n=3101 ( ) .agg() .compute() Stream days from S3 via earthaccess Subset + Mask Variable DataArray … Compute and Store Quantity search_data() open() xr.open_mfdataset() 2m temperature List of DataGranules Pointers to nc4 in S3 Chunked Dataset A Cloud-Based Workflow n=3101 ( ) .agg() .compute() Stream days from S3 via earthaccess Subset + Mask Variable DataArray … Compute and Store Quantity search_data() open() xr.open_mfdataset() 2m temperature Rinse and repeat List of DataGranules Pointers to nc4 in S3 Chunked Dataset A Cloud-Based Workflow n=3101 Implemented on full catalog for variables of interest from one MERRA-2 dataset Successes! MERRA-2 inst1_2d_asm_Nx Max 2m Temperature Anomaly Max 2m Temperature Anomaly over AIS Avg. Integrated Water Vapor over AIS Max. Integrated Water Vapor over AIS Max. Integrated Water Vapor Anomaly over AIS Max. Sea Level Pressure Gradient ~55 mins. on 8 CPUs HUGE speedups in earthaccess version 0.15.1 vs. 0.10.0 (version I started using in 2024) Successes! Successes! Greatly improve reproducibility and organization of primary project! Before: a set of several folders on Perlmutter scratch… Successes! Greatly improve reproducibility and organization of primary project! Before: a set of several folders on Perlmutter scratch… And yes, it did get purged once! Challenges