NFDI4Earth IG High-Performance Computing in Earth System Sciences - Activity Report 2023-2025

IG High-Performance Computing in Earth System Sciences (HPC in ESS): Activity Report Q3/2023-Q3/2025 for the IG: Stephan Frickenhaus, Alfred Wegener Institute (AWI, Bremerhaven & U. Bremen) Stephan Hachinger, Leibniz Supercomputing Centre (LRZ, Garching b.M.) 1 2 IG HPC in ESS: Motivation Why FAIR is easier said than done in HPC Observation: Repositories are (almost) perfect ... but: ... most “average” repositories cannot hold huge datasets as produced in HPC (in particular in ESS). ... marrying RDM and FAIR principles with computational workflows is not easy. FAIR „vs.“ HPC in ESS: Lots of challenges! ▪Technical: Datasets produced too large for many repositories – satellite data & global/large-scale high-resolution datasets ▪Methodical: Reproducibility –simulation/analytics output depends on compiler switches, libraries, etc. “Wild” community software usage: netCDF, eccodes, … ▪Administrative: Computing needs grants and accounting, but researchers want to quickly work where their data are. ▪Practical: “Everyone talks while I have 1PB to handle!” Grouping challenges: Metadata & Findability –Access –Interoperability –Federation & Compute-to-Data 3 IG HPC in ESS: Motivation Activity report for the last years 4 25 meetings since last activity report (mid 2023) Online-meeting participants ~12 distinct persons ~7 participants every time on average Institute participation (number of people on mailing list): AWI (5), DKRZ (3), DLR (2), FZJ-JSC (1), GEOMAR (1), GWDG (1), HEREON (2), HLRS (2), LRZ (5), MPI-BGC (1), TU Berlin (1), TUDD (3), TUDD-ZIH (3), TUM (1), GFZ (1) ▪Output: ▪Update to our concept paper: FAIR@HPC - Improving HPC usage in ESS by FAIR data and compute services https://doi.org/10.5281/zenodo.14796888 ▪Pilot CAPICE ▪Networking: ▪NHR – e.g. NHR Conference, Darmstadt Sep. 2024 –Talk: Data Projects & CAPICE ▪GCS – e.g. talk on ESS data publication at JSC in IG, exchange on Data Projects and NFDI4Earth RDM needs with InHPC-DE project ▪Other: ▪Talk at NFDI4Earth booth – European Geosciences Union (EGU) General Assembly 2025 ▪Suggestions for embedding HPC-related FAIR aspects in the renewal proposal We updated our concept paper… 5 •Original Publicly available concept paper: https://doi.org/10.5281/zenodo.6565405 6 … evaluating our recommendations w/r/t fulfilment and validity: finish-up in a workshop on N4E Plenary ‘24 1. An infrastructure registry for NFDI4Earth to find matching computing resources 2. Approaches to access computing resources through Single Sign On 3. S eamless cross-centre and cross-system computing access for users including accounting Do we have something there? 4. I nteroperability & interconnection of computing systems 5. Robust computing environments for re-usability and reproducibility (e.g. containerisation) Containers = wishful thinking in HPC or not? Alternatives? ( Spack, Package MGMT, Conda, …) Discussion Discussion 6. Establishing (meta - )data formats in collaboration with standardisation bodies, RDA, etc. 6.a) Max. ~10 general storage formats HDF5, NetCDF, ZARR, GeoTIFF, OGC Endorsed Formats, what else? 6.b) Max. ~10 metadata formats DataCite , ISO 19115 what else? 6.c) NFDI4Earth recommends these formats Who recommends? Label/Interoperability building blocks? 6.d) Support for these formats at all computing centres in NFDI4Earth … to be checked 7. Standardised software environments at computing centres, documented within NFDI or NFDI4Earth 8. Recommendations on RSE and software packaging 9. Recommendations on quality control Does that make sense? 10. Political work towards viable solutions across state/national borders Discussion Discussion Discussion Discussion 11. Issue NFDI4Earth recommendations on efficient data access and transfer for Big Data 12. Removal of administrative barriers regarding federated data/compute access 13. Data-portal and storage solutions with spatiotemporal and layer-/field-based data selection & download 14. Establish standardised (meta-)data interface to instituteor subdiscipline-specific interfaces 14.a) Metadata interfaces, starting with OAI -PMH 14.b) Advance usage of standard data interfaces, e.g. S3 15. Clarification and NFDI4Earth index for LTA 16. Interactive HPC including checkpointing 17. Concepts for services reacting on data events (e.g. alerting) Discussion Discussion Discussion Are recommendations realistic? Does it make sense if we treat certain topics like quality control? blue: recommendations we could more easily assess in teleconferences yellow: recommendations we needed to discuss on-site in the workshop 7 Written assessment of recommendation status + new recommedations: our concept paper update „(…) HPC centres and systems enable large-scale generative or discriminative Artificial Intelligence (…) applications. As we lay out below, we see a particular need in this context to adapt our recommendation catalogue to cover (…) i) data lakes for data-intensive science, ii) indexing and data catalogues, iii) increasing usage of AI and ML methods in the ESS, and iv) the development of data-project concepts and storage grant schemes at supercomputing sites.“ Concept paper 2024/25 update on Zenodo – check it out! https://doi.org/10.5281/ zenodo.14796888 N4E pilot CAPICE – developed in the IG 8 CAPICE – Combined Analysis and Publication of ICE Sheet data ▪Ice sheet modeling + Earth observation data + Calving front prediction ▪federate access/jobs across DKRZ/LRZ ▪despite challenges, principles proven viable: https://doi.org/10.5281/zenodo.14410879 ▪model example made FAIR: https://doi.org/10.5281/zenodo.14538166 CAPICE data flow schemeCAPICE infrastructures involved Want to get active with us ? Send a mail to stephan.frickenhaus [@] awi.de and/or stephan.hachinger [@] lrz.de ! ▪https://nfdi4earth.de/2participate/get-involved-by-interest-groups/ig-high-performance-computing-in-earth-system-sciences ▪Every 2nd Monday in the month we meet at 9:00 am