Plasma-PEPSC D1.12 Updated Assessment of Developments and Revision of Scientific Challenge, Enabling Next-Generation Plasma Accelerators for Real World Applications with PIConGPU

HORIZON JU Research and Innovation Actions HORIZON-EUROHPC-JU-2021-COE-01-01 European High-Performance Computing Joint Undertaking Plasma Exascale-Performance Simulations CoE 101093261 D1.12 Updated assessment of developments and revision of scientific challenge - Enabling Next-Generation Plasma Accelerators for Real World Applications with PIConGPU WP1: Plasma Simulations - Codes and Grand Challenges Date of preparation (latest version): 30/06/2025 Copyright©2023 – 2027 The Plasma-PEPSC Consortium D1.12: Enabling Next-Generation Plasma Accelerators with PIConGPU 2 DOCUMENT INFORMATION Deliverable Number D1.12 Deliverable Name Updated assessment of developments and revision of scientific challenge - Enabling Next-Generation Plasma Accelerators for Real World Applications with PIConGPU Due Date 30/06/2025 Deliverable lead HZDR Authors Michael Bussmann (HZDR), Klaus Steiniger (HZDR), Alexander Debus (HZDR), Julian Lenz (HZDR), Tapish Narwal (HZDR), Franz P¨oschel (HZDR), Richard Pausch (HZDR) and Jeffrey Kelling (HZDR) Responsible Author Michael Bussmann (HZDR) E-mail: [email protected] Keywords laser-plasma interaction, laser-plasma accelerators PIConGPU, alpaka, openPMD, TWEAC WP/Task WP1/Task D1.12 Nature R Dissemination Level PU Final Version Date 30/06/2025 Reviewed by Gabin Schieffer (KTH), Luca Pennati (KTH) Jonah Ekelund (KTH), Stefan Costea (UL) & Leon Kos (UL) D1.12: Enabling Next-Generation Plasma Accelerators with PIConGPU 3 DOCUMENT HISTORY Partner Date Comment Version HZDR 05/05/2025 Skeleton version 0.1 HZDR 05/06/2025 First draft updated with content 0.2 HZDR 06/06/2025 Final draft ready for internal review 0.3 KTH 06/06/2025 Final draft updated for internal review 0.4 HZDR 18/06/2025 Revised draft after internal review 0.4 KTH 23/06/2023 Final cleanup for submission 1.0 D1.12: Enabling Next-Generation Plasma Accelerators with PIConGPU 4 Executive Summary In this deliverable, we summarize our activities toward the scientific challenge of simulating compact laser-plasma accelerators in the Traveling-Wave Electron Acceleration (TWEAC) geometry for electron acceleration beyond 10 GeV energy. We realized in our earlier work that PIConGPU’s capabilities of modeling the complex laser fields in TWEAC geometries were insufficient to accurately simulate the target test case. Therefore, we developed and implemented a new Traveling-wave Thomsonscattering (TWTS) laser model TWTSTight which significantly improved modeling accuracy. It is well tested and ready to be used in production runs. Two further challenges were discovered when using the new laser model. First, the improved modeling comes at the cost of longer compute times. Second, we observed unforeseen numerical issues, specifically laser field reflections, were observed at the boundaries of the simulation box. The causes for these were identified and a mitigation strategy can be formulated, however, this is subject to experimentation and requires more time. Our earlier investigations also found that our performance portability library alpaka, on top of which PIConGPU is built, cannot be well analyzed by compilers of the new EPI compute architectures, resulting in poor auto-vectorization. After analyzing the problem, we started the development of alpaka3 now, implementing the idea of alpaka from scratch in a new development, in order to better express vector unit parallelism, aid autovectorization and increase usability. While first applications of our users were already ported to alpaka3, further development is necessary to port PIConGPU to the new library. A solution to the challenge of the I/O bottleneck for simulation data write-out and analysis has been developed and successfully implemented in production. The solution utilizes data streaming directly from the producer to the consumer, i. e. a PIConGPU simulation and a Machine Learning application, respectively, via openPMD with ADIOS2. This allows us to circumvent the lack of adequate disk capacity and bandwidth by enabling data to reside in-memory and be distributed via network between the two applications running on different nodes of the compute cluster at the same time. We have extended PIConGPU’s CI infrastructure to include runtime testing (both unit tests and integration tests). While we run unit tests automatically, integration tests, crucial for validating full physics workflows, are manually triggered, as they consume significant CI resources. Performing meaningful integration tests regularly, e. g. with every pull request or on a daily or weekly basis, is a very important but ultimately too big task for standard, shared CI resources. To address this, we request CASTIEL2’s support in accessing CI runners on EuroHPC systems, which would allow us to run large-scale integration tests automatically, ensuring readiness for deployment on EuroHPC systems and improving code reliability. In summary, we are progressing steadily along the work plan, having resolved key challenges, while adapting it to address new, unforeseen issues. D1.12: Enabling Next-Generation Plasma Accelerators with PIConGPU 5 Contents 1 Revision of the Scientific Challenge 6 1.1 Physicschallenges............................... 6 1.2 Computational resources and numerical challenges . . . . . . . . . . . . . 6 2 Summary of Developments 7 2.1 PIConGPU software release and continued development . . . . . . . . . . 7 2.2 Developingalpaka3.............................. 8 2.3 Utilizing the data streaming capability in PIConGPU via openPMD/ADIOS2inproduction............................. 8 2.4 Runtime tests in a Continuous Integration pipeline . . . . . . . . . . . . 8 2.5 Python interface for PIConGPU . . . . . . . . . . . . . . . . . . . . . . . 10 3 Revision of Gap Analysis 10 4 Update on Roadmap 10 5 Conclusion 11 D1.12: Enabling Next-Generation Plasma Accelerators with PIConGPU 6 1 Revision of the Scientific Challenge The initial report D1.3 defined the scientific challenges as follows. 1.1 Physics challenges •Identify the most suitable Traveling-Wave Electron Acceleration (TWEAC) regimes for stable acceleration. •Demonstrate acceleration of electrons in a single stage beyond 10 GeV with TWEAC. •Assess which injection techniques of standard Laser-Plasma Accelerators (LPAs) can be employed for TWEAC. •Evaluate possibilities for injection of high-brightness electrons. •Explore new capabilities of TWEAC with respect to electron injection techniques. As our scientific goal did not change, these challenges are up-to-date. The first revision D1.7 added •Identify TWEAC regimes where the laser pulse can be modeled with existing methods Since then, our physics-related work focused on improving the laser modeling capabilities of PIConGPU in order to accurately simulate small interaction angle regimes where we expect highest efficiency of laser to electron energy transfer. The newly implemented TWTSTight laser accurately models the vector fields of the laser pulse, taking weak laser components into account, see Pull Requests (PRs) #5231, #5273, #5274, #5305. Specifically small interaction angle regimes of TWEAC, where lasers are tightly focused and propagate over long distances, can be accurately simulated now. The item added in D1.7 is therefore achieved and cleared. 1.2 Computational resources and numerical challenges The initial report D1.3 defined the numerical challenges as follows. •Long-term stability: Simulations of TWEAC beyond 10 GeV will run on the order of 106timesteps. Stability of the solvers, in terms of artificial amplification of noise or small errors, needs to be studied. •Accuracy: If stability of the simulation setup is given, we want to ensure highfidelity simulation results, i. e. keep errors due to numerical inaccuracies low. We will use higher order schemes for the solvers, i. e. field solver, particle pusher, and particle shape. These schemes are already implemented. •Simulation volume size: Due to the line focus geometry, the laser pulses used in TWEAC have a large width in the plane transverse to the focal plane compared to the cylindrical symmetric laser pulses used in standard Laser-Wakefield Accelerators (LWFAs). In order to capture the plasma cavity formation correctly in the D1.12: Enabling Next-Generation Plasma Accelerators with PIConGPU 7 (a) (b) Figure 1: Current challenges in the simulation model of TWEAC: (a) Distorted field due to field artifacts when inserting TWEAC lasers from the side without a temporal window plus field reflections due to gracing incidence of the laser on the Perfectly Matching Layer boundaries in small interaction angle scenarios; (b) expected field distribution from a reference implementation. simulation, large plasma volumes have to be simulated. Together with the increased accuracy requirement, this increases the required computational resources by orders of magnitude compared to standard LWFA simulations. As before, these challenges still exist and are being actively worked on. The challenge ‘Laser model accuracy: implement and test a cylindrical quasi-Gaussian vector beam’, which was newly defined in D1.7 has been met with the new laser implementation. However, while testing the improved laser model, we observed further challenges in other modeling aspects of the simulation, see fig. 1. These are currently being worked on: •Directly inserting TWEAC lasers from the side without a temporal window generates initial field artifacts. A Blackman-Nuttal window (ref. [2]) will be implemented in order to enable a shallow field upramp. •Settings of the Perfectly Matching Layers boundaries have to be optimized since there are significant reflections visible in small interaction angle scenarios where the laser almost co-propagates to the transverse boundaries. 2 Summary of Developments 2.1 PIConGPU software release and continued development On 18th December 2024, a new release of PIConGPU was published on Github: https: //github.com/ComputationalRadiationPhysics/picongpu. Highlights of this PIConGPU release are a new atomic physics model FLYonPIC 2.0, the shadowgraphy plugin as advanced synthetic diagnostic of laser plasma interactions, a D1.12: Enabling Next-Generation Plasma Accelerators with PIConGPU 8 new laser profile FromOpenPMDPulse allowing to load electric fields in transverse space and time domain into the simulation via incidentField, enhanced OpenPMD functionality, and support for the RISC-V ecosystem. In addition, between 1st December 2024 and 1st June 2025 112 Pull Requests, by 16 contributors in total, were merged into mainline PIConGPU. These add functionality, ease usage, extend documentation, and fix bugs. 2.2 Developing alpaka3 A substantial amount of work is currently being invested into a redesign and new implementation of our performance-portability layer alpaka, the new library being alpaka3, see https://github.com/alpaka-group/alpaka3. During the project lifetime, we found within the consortium that alpaka cannot fully utilize vector units of RISC-V, the upcoming European exascale compute architecture, due to failing auto-vectorization by the compiler, see e. g. deliverables D1.7 and D2.5. The new development of alpaka3 started in November 2024 and includes – among other things – a hardware-agnostic kernel language that can accurately express the various levels of parallelism found in modern computing hardware. As a first step, this will aid auto-vectorization. This is already starting to receive confirmation within the consortium, see deliverable D2.5. But it will also allow the backend to implement explicit parallelization strategies, including vectorization, to further tune performance. Future benchmarking and collaboration within the consortium will show if this could be beneficial beyond the auto-vectorization capabilities of modern compilers. 2.3 Utilizing the data streaming capability in PIConGPU via openPMD/ADIOS2 in production The last roadmap, presented in deliverable 1.7, featured item ‘3.2 Extend and test streaming capability in PIConGPU using openPMD/ADIOS2 to be used in production’. This item represented the method to realize item ‘1.3. In-transit data analysis to tackle the data deluge’ of the same roadmap. Both can be regarded as completed as will be shown in the following. We have recently demonstrated a streaming workflow in which data from a running PIConGPU simulation is streamed directly to a machine-learning (ML) framework, circumventing the file system bottleneck, see fig. 2. That is, no simulation data is stored on hard drives of the file system. Data is transformed in transit, concurrently to the simulation and the training of the model. The results are published on arXiv, cf. ref. [1], and are accepted for publication in the proceedings of the 39th IEEE International Parallel & Distributed Processing Symposium. 2.4 Runtime tests in a Continuous Integration pipeline We have expanded our CI infrastructure to include runtime tests, which include both unit tests and integration tests (see PR #4723). These tests are designed to validate the correctness and performance of both the physics algorithms and the computer science D1.12: Enabling Next-Generation Plasma Accelerators with PIConGPU 9 (c) (d) Figure 2: (a-c) Three aspects to streaming between loosely coupled producer and consumer: (a) streaming without going through storage unlocks more bandwidth; (b) reducing simulation data close to the producer lowers bandwidth requirements; (c) to distributed producer and consumer, system topology presents communication paths with vastly different bandwidths which must be reconciled with the loosely-coupled application’s communication requirements. (d) High-level overview of the software stack for data exchange between PIConGPU and the ML framework. components of the PIConGPU codebase. Currently, unit tests are automatically executed as part of every CI run, ensuring rapid feedback on core functionalities. However, the integration tests, which are essential for validating full physics simulation workflows, require significantly more resources on a scale that no longer fit standard CI pipelines. We thus currently trigger them manually, even though we already added them to our CI workflows. This constraint limits the frequency and immediacy of integration testing. We noted that with our limited CI resources, running the full integration tests on every CI run was prohibitively expensive in terms of time and resources. It further had the negative impact of slowing down development, since it created a bottleneck in the development workflow by increasing the turnaround time for validating changes. To address this limitation, we would be happy to work with CASTIEL2 to obtain access to CI runners on EuroHPC resources, since running these tests on a cluster would be much more feasible. This would enable us to incorporate integration tests for largescale physics simulations directly into the CI pipeline, which would facilitate both the testing and deployment readiness on production EuroHPC systems and also testing the physics to increase code reliability. As a consequence, we regard the items ‘2. Test suite for continuous integration and code interface for automatization to secure fidelity of physics results in code development’ and ‘3.3. Implement execution of run time tests in continuous integration’ of the roadmap