Collection of requirements for shared data infrastructure and distributed processing

Collection of requirements for shared data infrastructure and distributed processing (architecture, services) Milestone 10 Authors: Pierre-Antoine Bretonnière 2 Views and opinions expressed are those of the author(s) only and do not necessarily reflect those of the European Union or the European Climate, Infrastructure and Environment Executive Agency (CINEA). Neither the European Union nor CINEA can be held responsible for them. The UK participation in the project is funded by UK Research and Innovation (UKRI) under the UK government’s Horizon Europe funding guarantee, and Canadian participation in the EXPECT project draws upon research supported by the Government of Canada’s New Frontiers in Research Fund (NFRF). Document information Grant agreement 101137656 Project title Towards an Integrated Capability to Explain and Predict Regional Climate Changes Project acronym EXPECT Project start date 1 April 2024 Related work package WP7 Related task(s) T7.1/T7.2/T7.3 Lead organisation BSC CNS Authors Pierre-Antoine Bretonnière Submission date March 2025 Dissemination level PU (Public) History Date Notes 28/03/2025 Final document submitted Disclaimer: Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Climate, Infrastructure and Environment Executive Agency (CINEA). Neither the European Union nor the granting authority can be held responsible for them. 3 Views and opinions expressed are those of the author(s) only and do not necessarily reflect those of the European Union or the European Climate, Infrastructure and Environment Executive Agency (CINEA). Neither the European Union nor CINEA can be held responsible for them. The UK participation in the project is funded by UK Research and Innovation (UKRI) under the UK government’s Horizon Europe funding guarantee, and Canadian participation in the EXPECT project draws upon research supported by the Government of Canada’s New Frontiers in Research Fund (NFRF). Table of content 1 About 4 2 Executive summary 4 3 Survey elaboration and content 5 4 Survey results 6 5 Annex 1 8 4 Views and opinions expressed are those of the author(s) only and do not necessarily reflect those of the European Union or the European Climate, Infrastructure and Environment Executive Agency (CINEA). Neither the European Union nor CINEA can be held responsible for them. The UK participation in the project is funded by UK Research and Innovation (UKRI) under the UK government’s Horizon Europe funding guarantee, and Canadian participation in the EXPECT project draws upon research supported by the Government of Canada’s New Frontiers in Research Fund (NFRF). 1 About The climate system is changing rapidly and some regions have seen increases in extremes beyond what is expected from climate model simulations. To support targeted climate adaptation strategies, EXPECT will enable trustworthy assessments and predictions of regional climate change including extremes by developing a prototype operational capability for integrated attribution and prediction of climate. This ambitious goal is closely aligned with the WCRP Lighthouse Activity on Explaining and Predicting Earth System Change. EXPECT will identify and quantify the mechanisms by which physical processes govern regional climatic changes, including extremes, on inter-annual to multi-decadal time scales. It will do so by exploiting newly available climate simulations and Earth Observations (EOs), and by combining machine learning (ML) with physical methods. The research will target fundamental knowledge gaps related to atmospheric circulation and land-atmosphere interactions, which represent major limitations in current climate predictions and projections, and in particular in understanding changes in European summer extremes. To underpin the research, and benefiting the wider research community, EXPECT will develop tools to efficiently analyse a variety of large data sets in combination that are hosted in different repositories across institutions. This will facilitate the exploitation of recent investments into high-resolution climate models and EO data. EXPECT will further build data science capacity for the scientifically robust, efficient and reproducible analysis of the massive data assets, including novel ML approaches, and provide training for the climate science community and the next generation of researchers in particular. EXPECT will thus deliver significant scientific and technological advances for society and the climate science community that will last well beyond the project, in support of WCRP’s strategic objectives. 2 Executive summary This milestone reports the elaboration process and the results of the survey sent to the whole consortium to get their requirements in terms of data needs, access, and computing. This survey aims at defining and scaling the work from Theme 4, especially D7.1 « Data Infrastructure Setup », D9.1 « Distributed dask computation Service », D14.2 and 15.2 « Update and final version of the DMP », M11 « Report on status of tool support for high-resolution data in CF context and FAIRness enabling components » and M12 «Report on distributed data analysis workflows including outreach material enabling future exploitation ». 5 Views and opinions expressed are those of the author(s) only and do not necessarily reflect those of the European Union or the European Climate, Infrastructure and Environment Executive Agency (CINEA). Neither the European Union nor CINEA can be held responsible for them. The UK participation in the project is funded by UK Research and Innovation (UKRI) under the UK government’s Horizon Europe funding guarantee, and Canadian participation in the EXPECT project draws upon research supported by the Government of Canada’s New Frontiers in Research Fund (NFRF). 3 Survey elaboration and content The survey was created at the beginning of the project (June 2024) to gather the requirements on data types, frequencies, where people access the data from, the formats they use, etc. Getting this information will allow the partners of Theme 4 (BSC, CINECA, DKRZ, UREAD) to design the access to the data for people who need the tools and data. The survey was designed in collaboration with the sister project AI4PEX, who made a similar survey, and both were homogenized to get a shared survey. It was sent in June 2024, with reminders in September 2024, and March 2025, which helped gather information at different stages of maturity of the work from the different themes. In total, 20 respondents answered. The questions are listed below: • What datasets are you planning to use in your work in EXPECT? • What data frequencies do your analysis need? • What variables are you planning to use? • Where do you usually get your data from? • Do your tools need a specific data format? • What tools do you usually use to load/process the data? • Do you have a rough idea about the total data volumes you might need for your analysis? • How much data (in GB) is usually loaded into memory during a typical runtime job? • Where do you plan to run the analysis? 6 Views and opinions expressed are those of the author(s) only and do not necessarily reflect those of the European Union or the European Climate, Infrastructure and Environment Executive Agency (CINEA). Neither the European Union nor CINEA can be held responsible for them. The UK participation in the project is funded by UK Research and Innovation (UKRI) under the UK government’s Horizon Europe funding guarantee, and Canadian participation in the EXPECT project draws upon research supported by the Government of Canada’s New Frontiers in Research Fund (NFRF). • Where do you usually run/plan to run your analysis? • What type of hardware does your analysis need? • Do you have specific memory or system requirements (number of cores, memory,...) ? • The preparation and optimization of the infrastructure is the responsibility of Theme 4 WGs. Still, do you have some people in your WP/institute ready to dedicate a few hours to discuss technical implementation with Theme 4? 4 Survey results The details of the answers to the survey, question by question, are shown in Annex. The main conclusions are summarized here. All the work packages have been represented in the answers, except for WP11 (Laying the foundation for communication and dissemination), which is expected as the WP doesn’t use directly data or the technical infrastructure. The rest of the WPs have inequal representation but with significant number of answers for each. The main datasets that people are planning to use come from ESGF (CMIP6, LSFMIP), the Climate Datastore or mars (ERA5 or mars seasonal forecasts) and some other external High Resolution Climate models data (Climate Digital Twin DestinE, EERIE, NextGEMS). CMIP6 and ERA5, which are available in Jasmin and DKRZ are the most requested datasets (>80% of users planning to use them). From these datasets, some high frequency data are expected (~80% for daily, and 30% hourly or as high as possible), which combined to the spatial resolution of the datasets means that some very high volumes of data are going to be used, especially taking into account that more than 75% of the people answering plan to use 3D data. Users seem to have 2 kinds of usage for the data: either downloading the data from the CDS and ESGF (~60%) and Mars (~25%) to their local machines, or taking advantage of the prefetched data from institutional repositories such as CEDA, DKRZ, BSC,… (>50%), but the data access seems to be the main driver to choose where to run the analysis (more than half of the people answered “Wherever I have the data and computing power” to the question about where they planned to run their analysis). 7 Views and opinions expressed are those of the author(s) only and do not necessarily reflect those of the European Union or the European Climate, Infrastructure and Environment Executive Agency (CINEA). Neither the European Union nor CINEA can be held responsible for them. The UK participation in the project is funded by UK Research and Innovation (UKRI) under the UK government’s Horizon Europe funding guarantee, and Canadian participation in the EXPECT project draws upon research supported by the Government of Canada’s New Frontiers in Research Fund (NFRF). Most of the users are expecting NetCDF data (pure Netcdf 61%, CMOR NetCDF 1 answer) but more surprisingly, people claim that their analysis tools are not tied to a specific format (27%), which gives some freedom to the developers of Theme 4 to develop or enhance new tools. The typical jobs are not defined clearly as people have given very wide ranges of memory usage and volume of data (“I don’t know” is the first answer to the memory and volume usage). Finally, as expected, most of the people (>75%) don’t plan to dedicate technical support to theme 4 and the developments will have to be delivered with only the technical validation. 8 Views and opinions expressed are those of the author(s) only and do not necessarily reflect those of the European Union or the European Climate, Infrastructure and Environment Executive Agency (CINEA). Neither the European Union nor CINEA can be held responsible for them. The UK participation in the project is funded by UK Research and Innovation (UKRI) under the UK government’s Horizon Europe funding guarantee, and Canadian participation in the EXPECT project draws upon research supported by the Government of Canada’s New Frontiers in Research Fund (NFRF). 5 Annex 1 9 Views and opinions expressed are those of the author(s) only and do not necessarily reflect those of the European Union or the European Climate, Infrastructure and Environment Executive Agency (CINEA). Neither the European Union nor CINEA can be held responsible for them. The UK participation in the project is funded by UK Research and Innovation (UKRI) under the UK government’s Horizon Europe funding guarantee, and Canadian participation in the EXPECT project draws upon research supported by the Government of Canada’s New Frontiers in Research Fund (NFRF). 16 Views and opinions expressed are those of the author(s) only and do not necessarily reflect those of the European Union or the European Climate, Infrastructure and Environment Executive Agency (CINEA). Neither the European Union nor CINEA can be held responsible for them. The UK participation in the project is funded by UK Research and Innovation (UKRI) under the UK government’s Horizon Europe funding guarantee, and Canadian participation in the EXPECT project draws upon research supported by the Government of Canada’s New Frontiers in Research Fund (NFRF).