D1.4 - Support for emerging system architectures

Report on support for emerging system architectures MultiXscale Deliverable 1.4 Deliverable Type: Report Delivered in June, 2025 MultiXscale EuroHPC Centre of Excellence for Multiscale Modelling Acknowledgement Funded by the European Union. This work has received funding from the European High Performance Computing Joint Undertaking (JU) under grant agreement No 101093169. Disclaimer Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European High Performance Computing Joint Undertaking (JU). Neither the European Union nor the granting authority can be held responsible for them. MultiXscale Deliverable 1.4 Page ii Project and Deliverable Information Project Title MultiXscale: EuroHPC Centre of Excellence for Multiscale Modelling Project Ref. Grant Agreement 101093169 Project Website https://www.multixscale.eu EuroHPC Project Officer Dr. Matteo Mascagni Deliverable ID D1.4 Deliverable Nature Report Dissemination Level Public Contractual Date of Delivery Project Month 30 (30th June, 2025) Actual Date of Delivery 27th June, 2025 Description of Deliverable Report on support in the shared software stack for emerging system architectures Document Control Information Document Title: Report on support for emerging system architectures ID: D1.4 Version: As of June, 2025 Status: Accepted by Steering Committee Available at: https://www.multixscale.eu/deliverables Document history: Internal Project Management Link Review Review Status: Reviewed Authorship Written by: Pedro Santos Neves (RIJKSUNIGRON) Contributors: Bob Dröge (RIJKSUNIGRON), Kenneth Hoste (UGent), Lara Peeters (UGent), Caspar van Leeuwen (SURF), Satish Kamath (SURF), Alan O’Cais (UB), Richard Topouchian (UiB), Thomas Röblitz (UiB), Julián Morillo (BSC) Reviewed by: Thomas Röblitz (UiB), Kenneth Hoste (UGent) Approved by: Alan Ó Cais (UB) Document Keywords Keywords: MultiXscale, HPC, software, applications , infrastructure 27th June, 2025 Disclaimer: This deliverable has been prepared by the responsible Work Package of the Project in accordance with the Consortium Agreement and the Grant Agreement. It solely reflects the opinion of the parties to such agreements on a collective basis in the context of the Project and to the extent foreseen in such agreements. Copyright notices: This deliverable was co-ordinated by Pedro Santos Neves1(RIJKSUNIGRON) on behalf of the MultiXscale consortium with contributions from Bob Dröge (RIJKSUNIGRON), Kenneth Hoste (UGent), Lara Peeters (UGent), Caspar van Leeuwen (SURF), Satish Kamath (SURF), Alan O’Cais (UB), Richard Topouchian (UiB), Thomas Röblitz (UiB), Julián Morillo (BSC) . This work is licensed under the Creative Commons Attribution 4.0 International License. To view a copy of this license, visit: http://creativecommons.org/licenses/by/4.0 cb [email protected] MultiXscale Deliverable 1.4 Page iii Contents Executive Summary 1 1 Introduction 2 1.1 Scope of the deliverable ................................................ 2 1.2 Target audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.3 Deliverable outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.4 Partner contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 2 CPU targets 3 2.1 Criteria for supporting new CPU targets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2.2 Emerging CPU targets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.3 Other relevant CPU targets .............................................. 4 3 Accelerator targets 5 3.1 NVIDIA GPUs ...................................................... 5 3.2 AMD ROCm support for AMD GPUs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 4 RISC-V 7 4.1 Current status ...................................................... 7 5 Additional improvements 8 5.1 Installations of additional CPU targets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 5.2 Development repository ................................................ 8 6 Conclusion and outlook 9 References 10 List of Tables 1 Supported CPU microarchitectures and Compute Unified Device Architecture (CUDA) Compute Capability (CC). An N denotes software builds are done natively on a given CPU-GPU pair (meaning GPUspecific tests are run). All CPU targets can utilize available GPU-enabled software installations. . . . . . 5 MultiXscale Deliverable 1.4 Page 1 Executive Summary In this report, we summarize the process for supporting new CPU microarchitectures and GPUs in the European Environment for Scientific Software Installations (EESSI) software stack within the MultiXscale Centre of Excellence. The main purpose of EESSI is to provide optimized software applications seamlessly to users while supporting the widest possible range of systems, from laptops to the largest exascale High Performance Computing (HPC) clusters such as the JUPITER system. Consequently, keeping up with new and emerging system architectures is critical to ensure new systems also have access to the optimized software stack. This work ensures that emerging system architectures are available through EESSI’s continuous deployment, and thus, that an increasing number of users can benefit from optimized software installations. We describe the progress regarding the support for accelerators, in particular NVIDIA and AMD GPUs. The support for GPUs was substantially improved, and the number of available build environments increased. The effort to expand the software stack with a large matrix of CPU architectures and GPU combinations is high, but its expansion is well underway. Thanks to the access of numerous European High Performance Computing Joint Undertaking (EuroHPC) sites, Tier-1 and Tier-2 sites across Europe, and the previously used Microsoft Azure and Amazon AWS cloud platforms, we are diversifying our build infrastructure, which gives us access to critical CPU and GPU combinations and allows us the spread the computational effort among different systems and providers. Besides the gradual increase of the total number of available software packages since the official release of the first EESSI version (2023.06 in late 2023), 7 new CPU microarchitectures are supported, raising the count of the initially supported architectures from 6 to 13. These include both x86_64 and aarch64 CPU families, such as AMD Zen 4 and NVIDIA Grace. The separate riscv.eessi.io software stack has also vastly grown, consisting now of a much larger number of RISCV compatible applications. The method to build it have also been brought closer to the general software building workflow for software.eessi.io, which is a major milestone towards integrating the "experimental" RISC-V software stack into the production version served via /cvmfs/software.eessi.io. We also detail the process of adding new CPU targets and accelerators to the existing software stack and how the decision of such additions is taken. By having clear processes and documentation EESSI can keep growing sustainably while meeting the already high expectations of the user community. Finally, the development repository dev.eessi.io is available to MultiXscale CoE researchers, allowing them to build, test, and deploy their software to EuroHPC sites and beyond in a quick and flexible way. We expect this to speed up development cycles, enable testing and Continuous Integration (CI) on actual hardware to increase software quality; all of which are desirable aspects of modern software development. MultiXscale Deliverable 1.4 Page 2 1 Introduction 1.1 Scope of the deliverable This deliverable concerns the results achieved in Task 1.2. We describe the progress, milestones, and ongoing work in supporting optimized software builds for emerging CPU architectures in EESSI. In addition, we discuss lessons learned while developing our procedure to add additional CPU targets to existing EESSI versions, which was done repeatedly for the first production version of EESSI – 2023.06. We detail the status and ongoing development of the ARM software stack which now includes the addition of NVIDIA Grace to the existing generic, Neoverse N1, Neoverse V1 targets and we provide an outline to enable support for Fujitsu A64FX. Regarding the x64_86 CPU microarchitectures, we describe the inclusion and parity of builds for AMD Genoa (zen4) in EESSI as well as Intel Sapphire Rapids. Similarly, we describe our work to support AMD accelerators and ROCm. Lastly, we describe the ongoing work developing the RISC-V software stack in the dedicated CernVM File System (CernVM-FS) repository riscv.eessi.io. 1.2 Target audience This deliverable is aimed at HPC system administrators, users and researchers interested in software builds optimized for specific CPU microarchitectures and accelerators (GPUs). While primarily aiming for using EESSI on HPC systems, researchers and users at large who are interested in maximizing the performance of their scientific software will also benefit from the advances described in this report. 1.3 Deliverable outline We cover the progress in supporting new CPU targets in Section 2and include the criteria used to decide on the addition of new CPU microarchitectures. In Section 3we outline the improvements to the existing support for NVIDIA GPUs, and include a description of how the support for NVIDIA accelerator generations is integrated. In the same section, we also outline the important steps taken and progress achieved in supporting AMD ROCm and AMD GPUs. We provide details on the current status of the RISC-V software stack in Section 4and give an overview of the steps taken towards integrating it with software.eessi.io. Lastly, in Section 5we describe improvements to the support of emerging architectures such as a newly created workflow for quickly and robustly adding new CPU targets to the production software repository and on supporting developers in the MultiXscale CoE. Lastly, we introduce the development repository dev.eessi.io and describe how it can help the workflow of MultiXscale CoE developers. 1.4 Partner contributions Partners in Task 1.2 (RIJKSUNI, BSC, UGent, SURF, UiB, UB) contributed as planned to the work described in this deliverable. MultiXscale Deliverable 1.4 Page 3 2 CPU targets In this section, we elaborate on new CPU targets added to the EESSI software stack, which constitutes an update to the work already reported in Deliverable D1.1 "Report on shared software stack prototype" [1]. At the time of writing, the EESSI shared software stack provides optimized software for the following aarch64 CPU microarchitectures: • Neoverse N1 • Neoverse V1 • NVIDIA Grace (Neoverse V2) • Fujitsu A64FX (work in progress) The supported x86_64 CPU microarchitectures: • AMD Zen 2 • AMD Zen 3 • AMD Zen 4 • Intel Haswell • Intel Skylake • Intel Cascade Lake • Intel Ice Lake End-users with access to systems where the EESSI software repository software.eessi.io is available need only load the EESSI module and and key environment variables, e.g., $MODULEPATH, will be configured such that the best fitting / optimized installations are accessible via Lmod commands. module use /cvmfs/software.eessi.io/init/modules module load EESSI/2023.06 Software installations are located in subdirectories with standardised names (reflecting the output of archspec and archdetect), for example, applications optimized for the ARM CPU microarchitecture Neoverse N1 are installed in the directory: /cvmfs/software.EESSI.io/software/versions/2023.06/software/linux/aarch64/neoverse_n1/ Consequently, when new CPU targets are added to the software stack, a new subdirectory is created under aarch64 or x86_64 where appropriate. We detail the procedure for adding new CPU targets in Section 5.1. For clients using a CPU that is not natively supported, the stack will fall back to one of the other targets that is fully compatible with the client’s CPU. Ultimately, there is a generic fallback without microarchitecture-specific optimizations, but that will only be used for processors that have fewer CPU features than the oldest relevant. 2.1 Criteria for supporting new CPU targets The current number of CPU microarchitectures used both in single user or HPC contexts is too large to cover comprehensively. More importantly, the differences in instruction sets between very similar CPU microarchitectures can be very small (for instance, Intel Haswell and Intel Broadwell are very similar). In such cases, the potential performance differences between both CPU microarchitectures are very small or even negligible. Furthermore, in order to natively compile optimized software, we need access to infrastructure providing the necessary hardware. Therefore, we focus only on building and delivering optimized software installations a subset of all existing CPU microarchitectures, while ensuring broad compatibility and optimized builds are available for the community. We decide on which the relevant system architectures to be included directly in EESSI are by following a number of criteria: • New system architectures that become available in EuroHPC systems, or that will be available in the near future; • The addition of instructions that bring significant performance advantages and that are not available already in the instruction sets of the currently supported CPU targets; • Ubiquity of given system architectures in national systems (Tier-1 and Tier-2 sites); MultiXscale Deliverable 1.4 Page 4 • Requests by the community for a particular system architecture for which there is a compelling case for inclusion. As an example, see the request2on the support portal for the inclusion of the Intel Sapphire Rapids software stack. • Availability and access to hardware resources on which to build the corresponding software stack. 2.2 Emerging CPU targets The NVIDIA Grace software stack has been added to software.eessi.io and it is now fully on par with other CPU targets, that is, it consists of over 1,000 software installations at the time of writing (end of June 2025), with more being added as part of the standard process of installing new software. NVIDIA Grace software was not built on the existing AWS and Azure cloud clusters. Software builds were possible due to the cooperation with the Jülich Supercomputing Centre (JSC) system administrators, who granted a few EESSI developers access to JURECA3, which is equipped with some NVIDIA Grace hardware4. The collaboration with JSC system administrators and user support managers has been beneficial for both parties - JSC and MultiXscale/EESSI - because users on JUPITER will get access to a broad range of software installations from early on, and it may be used on other systems equipped with this CPU microarchitecture. The AMD Zen 4 and Intel Sapphire Rapids software stacks are on par with the already present CPU targets. This was done via the standard procedure using instances of the build-test-deploy bot running on Magic Castle clusters on Microsoft Azure and AWS. As part of the build process, a CI pipeline was devised to compare the new with existing software stacks, in order to validate that all software has been built and is available. 2.3 Other relevant CPU targets While not an emerging architecture, as it was launched in 2019, building a Fujitsu A64FX software stack is of particular relevance for the EuroHPC system Deucalion5, since one of its main compute partitions is powered by this architecture. Software builds for A64FX were obtained through access to this system, and at the time of writing approximately 54% of the necessary software installations were ingested into the software.eessi.io CernVM-FS repository, amounting to a total of 525 installations at the time of writing. Due to the more limited availability of this architecture in cloud providers such as AWS and Microsoft Azure, the necessary instance of the build-test-deploy bot runs on the Deucalion system itself, for which Minho Advanced Computing Center (MACC) system administrators kindly provided us access via a service account. In all other aspects the build process is identical to that of building for other CPU targets. Support for Intel Cascade Lake and Ice Lake has been added recently and their software substacks are now fully on par with those of the remaining architectures. Intel Cascade Lake is particularly relevant because such platform is available in AWS coupled with NVIDIA Compute Capability (CC) 7.5 enabled accelerators. Hence, having the Intel Cascade Lake CPU software stack in place immediately enables us to build CC 7.5 compatible applications in AWS cloud instances, which, due to said availability, may be popular for the community at large. 2https://gitlab.com/eessi/support/-/issues/68 3https://www.fz-juelich.de/en/ias/jsc/systems/supercomputers/jureca 4https://apps.fz-juelich.de/jsc/hps/jureca/evaluation-platform-overview.html#grace-hopper-nodes 5https://www.macc.fccn.pt/resources MultiXscale Deliverable 1.4 Page 5 3 Accelerator targets In deliverables D1.1 "Report on shared software stack prototype" [1] and D1.3 "Report on stable, shared software stack" [2] we reported on the (initial) support for NVIDIA GPUs. In this section, we will elaborate on the progress and extended support that has been added, and we will outline the current work and plans for adding support for the AMD ROCm ecosystem to EESSI in order to support AMD GPUs as well. 3.1 NVIDIA GPUs Because software installations are optimized for each system architecture they are built on, a suitable combination of CPU and GPU architecture to build the software is needed. The features available to each NVIDIA GPU are defined as Compute Unified Device Architecture (CUDA) Compute Capability (CC) and depend on the GPU itself and the version of CUDA available on the system. For example, a device that supports CUDA CC 8.0 can use all features up to and including version 8.0. This means it is fully compatible with software built using features from version 7.0 or earlier. At the software level, not all applications can be optimised for more than one CUDA CC, including software that is integral to MultiXscale [2], which makes building the software stack with different combinations of CPU microarchitectures and GPUs all the more important. We followed a similar approach when choosing which CPU architectures and CUDA Compute Capability combinations to focus on. Our priority is to focus on combinations present in existing and upcoming EuroHPC systems, and covering a wide enough spectrum of CPU and GPU architectures so that most users can find either a matching combination, or a suitable fallback. At the moment, optimized software including key MultiXscale applications are available for combinations such as AMD Zen 2 and Zen 3 CPUs with CUDA CC 8.0, as well as for NVIDIA Grace CPUs with CUDA CC 9.0 (that is, compatible with Grace Hopper systems, which will power the EuroHPC HPC JUPITER). CUDA CC CPU microarchitecture 7.0 8.0 9.0 x86_64 generic intel haswell skylake cascadelake N icelake N sapphirerapids amd zen2 N zen3 N zen4 N aarch64 generic neoverse_n1 neoverse_v1 nvidia grace N Table 1: Supported CPU microarchitectures and CUDA CC. An N denotes software builds are done natively on a given CPU-GPU pair (meaning GPU-specific tests are run). All CPU targets can utilize available GPU-enabled software installations. In Table 1we show that we are building GPU software for all CPU families for all supported CUDA CCs. However, we only can test GPU installations on build nodes that provide a GPU (denoted by N in the Table 1). The subtleties of device code, ptx code and compatibility for NVIDIA GPUs are discussed in detail in CUDA CC support issue for EESSI, and are beyond the scope of the current discussion. 3.2 AMD ROCm support for AMD GPUs Although AMD datacenter GPUs intended for high-performance computing have been around for several years now, it can still be considered as a somewhat emerging platform in terms of adoption and software supporting this platform. In addition, the AMD ROCm software ecosystem itself is still evolving quite quickly. As an example, broader compatibility guarantees between ROCm software and kernel mode drivers was only introduced with ROCm version 6.4 (released April 11, 2025) [3]. Prior to this, user space software ecosystem updates required administrators to update the GPU drivers (and all software would need to be recompiled!). Nevertheless, it is a very interesting accelerator architecture to consider next to NVIDIA, also because the EuroHPC LUMI system provides a large number of AMD GPUs. For this reason, we have recently started working on adding support for the ROCm ecosystem to the EESSI software stack. MultiXscale Deliverable 1.4 Page 6 Most of the work so far has been put into adding support for a more recent ROCm version in EasyBuild. Currently, EasyBuild only supports ROCm version 4.5.0, which was released in 2021. Since then, the ROCm ecosystem and EasyBuild have evolved quite a bit. Thus, we could simply bump the versions of the existing build recipes. Instead, we opted for a new start on building support for recent versions of ROCm in EasyBuild. Additionally, the AMD ROCm software ecosystem is in a comparatively less mature state than NVIDIA’s ecosystem [4], which presents additional challenges. The ROCm compilers are based on an AMD fork of the LLVM compilers, and recently the EasyBuild installation procedures for LLVM compilers were completely overhauled 6. We have based our work on these new installation procedures for the ROCm LLVM compilers. The interest in having ROCm support built into EasyBuild and EESSI has also been picked up by AMD, and we are currently co-designing the necessary updates to the EasyBuild recipes with the AMD ROCm team 7. It is worth mentioning a recent external contribution to EESSI by Inuits in the form of a comprehensive documentation page titled Overview of ROCm Ecosystem (v6.4.1-20250526)8, which was accompanied by a blog post9to highlight the new documentation page. We hope that this additional documentation will help clarify the ROCm ecosystem and serve as a go-to reference for its overall structure and design. 6https://github.com/easybuilders/easybuild-easyblocks/pull/3373 7https://gitlab.com/eessi/support/-/issues/71 8https://www.eessi.io/docs/site_specific_config/rocm/ 9https://www.eessi.io/docs/blog/2025/05/26/rocm/