CEEC Webinar: Exploring CI/CD Tools and Workflows for Research and HPC Software Projects

Exploring CI/CD Tools and Workflows for Research and HPC Software Projects 2 Profile •Michael Zikeli | [email protected] •PhD student in RSE •CI/CD Maintainer •Friedrich-Alexander-University ErlangenNuerenberg (FAU) •Chair for System Simulation (LSS) •Software: •waLBerla (Simulation Framework) •HyTeG (Simulation Framework) •ExaStencils (Code Generator) •pystencils and lbmpy (Code Generator) 26.11.2025 2 4 CX in HPC 11/26/2025 4 Coding Build Dependencies Functionality Performance Portability Reproducibility User Accessibility Monitoring Development Orchestration in RSE for HPC •Automatically build & test every change •Ensures correctness across compilers, architectures, and systems Continuous Integration (CI) •Automatically produce and ship reproducible builds and environments •Ensures software availability Continuous Delivery / Deployment (CD) •Automatically measure and track performance over time •Validates scaling behaviour Continuous Benchmarking (CB) 5 Showcase: •widely applicable Lattice Boltzmann from Erlangen (waLBerla) •walberla.net/ •A massively parallel open-source framework for multi physics applications. •C++ Codebase •Runs on state-of-the-art CPU and GPU supercomputes. (LUMI, SuperMUC, …) •Uses compiletime codegeneration … •as interface between models and code •to optimize compute kernels and communication patterns •to facilitate code portability (e.g. GPU) •Focuses on CFD Domain •Lattice Boltzmann Method (LBM) •Coupling to other methods: •Particle Dynamics (DEM) •Phase fields •Free-Surface Flows •Temperature 26.11.2025 6 Showcase: -CI •Codebase at self hosted GitLab •i10git.cs.fau.de/walberla/walberla •CI pipeline orchestrator is GitLab CI •YAML based Pipeline configurations •GitLab CI schedules CI-Jobs from pipeline •Each Job defines: •Its execution environment as Image •Hardwae requirements per Tags •Specific build flags •Artifacts to keep after execution •GitLab Runner executes queued CI-Jobs •Runners are taged with their specs •OS: Linux or MacOS •GPU: NVIDIA for Linux •CPU ISO: X86 Linux or ARM MacOS 26.11.2025 CI Job definition example: 7 waLBerla – CI Pipeline 26.11.2025 7 CI Jobs divided into 6 stages: CI pipeline is triggered by every code „push“. BUT some jobs only run on specific branches: GitLa b Pages Branch Commits to feature branches High execution frequency Small test space Saves resources MR Commits to branches with open merge requests Large test space Used as Acceptance Test Stable Commit to main branch Regression Test + Performance Regression Updates GitLab Pages Documentation Coverage report 8 CI-Stage –Testsuite •Frequent small test set to catch errors during coding •Executes on all branches •Framework is built with different •Compilers: gcc, clang, icx •Build Parameters: e.g. CUDA, MPI, OMP, AVX, … •All CI Tests are performed for build configuration •Unit, Integration, Regression, … •Direct Feedback from GitLab GUI + Mail Notification •More details stored in build artifacts. 26.11.2025 8 TestSuite-Job-Passed TestSuite-Job-Failed 11 CMake Presets & Fragments •waLBerla uses CMake Presets •to specify the build configuration •Alternative to •`ccmake` •passing `-W<cacheVariable>` •Clean build config definition •Allows generation of CMakePresets.json •Eases Reproducability •We define common configurations as “Fragments” •Reduces code duplications in presets •Fasilitates cleaner Presets •Fragment features include: •Enabling external libraries like OpenMesh, Metis, etc. •Setting the build type; e.g., Release, Debug •Choosing your build tool; e.g., Ninja, Unix Make •Enabling support for GPU, Python, MPI, and more 26.11.2025 11 12 CI-Stage –Code Quality •Statis Analysis Tests •Clang-tidy for C++ linting •Helped to modernize waLBerla to C++20 •We use own clang-tidy setup •Linting for python codes •Flake8 + MyPy for python linting •Executed with Cmake and Nox 26.11.2025 12 20 CI-Images –3 Steps •Creating waLBerla build environment in 3 CI-Stages: 1. Build BaseContainer •Start from GPU vendor image •Prepare Spack 2. Install Dependencies •Using spack binary cache 3. Create CI-Images •Generate spack environments •Create Image from spack environment 26.11.2025 20 21 CI-Image –Generation 26.11.2025 21 •Valid spack environment file •Mark required changes with macros •Python environment generation script •Define variations in environment spec •Use Jinja2 Templates to override macros •Create environment folder per image •Each folder contains generated environment file •CI loops over environment folders to create images 22 CI Images –Spack Buildcache •Accellerate Spack installations •Already build binaries can be stored in Build Cache •Lookup binaries in cache before each build •If available binary matches the spec: ➔Binary is pulled instead of build •GitLab Container Registry can host Build Caches •Same Registry for CI-Images and Build Caches •Note: Public build caches exist: •cache.spack.io 26.11.2025 22 23 Deployment / Delivery •Current Deployment = Code Release •Manuel Release via •GitLab Release for Versioning •Zenodo Publication for a DOI •Has hook for GitHub •Automatic Mirror to other Git Servers •For GitHub and EU Projects •Binary Deployments •Container often not appropriate for HPC •Deploying a binary artefact is not enough •HPC codes need target optimizations ➔Usually want to build directly on target system 26.11.2025 23 As hardware gets more powerful efficiency becomes even more important! 24 RDM –Bridge to CD •Research Data Management (RDM) Repository •Internal first attempt for benchmark CD •Focus on FAIR Principles •Content of RDM-Collection •Buildand Run Scripts •Performance Data •Team Guidbook for systems •Requires manuel execution on site •NO Continous Deployment. 26.11.2025 24 25 WaLBerla –EuroHPC Landscape 26.11.2025 25 26 Tools –Package Managers •System Admin oriented: E.g., RPM (Redhat) und Deb (Ubuntu, Debian) •User oriented: Mostly EasyBuild and Spack; Other exist, e.g. Nix or pip for python, … •Software deployment tools for complex research software (and HPC) •Build scientific libraries (MPI, HDF5, PETSc, etc.) •Manage dependencies, toolchains, updates, etc. •Enable reproducibility of software environments on HPC clusters •Provide environment modules for users •Facilitates concurrent version (multiple-version of same software on a system) Note: Spack used in waLBerla to manage build dependencies in CI •Not to deploy waLBerla itself (yet) •Future developments will likely include Spack and EasyBuild 26.11.2025 26 27 Compare –Package Managers 26.11.2025 27 Spack • By LLNL (US) | Internationally used • Flexible dependency organization •E.g. ~fortran+Ofast @gcc12 target=x86_64 •Convenient for users and researchers •Allows many options (Maybe too many?) • Define Package as Python classes •Build options →`variant` •Dependencies→`depents_on` •Calls a function for each parameter • Example PETSC •>800 lines •> 40 variants + dep versions →∞ Variations • Pre-build binary stack •Public Build Caches (e.g. E4S) • Main package repository •github.com/spack/spack-packages •Very Large Repository EasyBuild • European Solution, By Gent University (NL) • Hierarchical decency organization •One dependency chain per eb-config file •Convenient for maintainer and admins •Ready-to-use recipes; fewer decisions needed • Define Package as EasyBuild configs (DSL) •Build options →`toolchainopts` •Dependencies→`dependencies` •Defines a lists for each parameter • Example PETSC •46 lines •3 build options & 10 PETSc.eb files • Pre-build binary stack •EESSI (European) (based on CernVM-FS) • Main package repository •github.com/easybuilders/easybuildeasyconfigs 28 Tools –Benchmark Orchestration 26.11.2025 28 •Test / Benchmark orchestration tools for complex research software (and HPC) • Initially developed for Acceptance and Regression tests of HPC systems •Natively designed to work with workload managers like SLURM •Abstraction of test dependencies and execution •Require definition of at least: System, Build, Parameterset •Spans Testmatirx / -pool from set of parameters •Utilizes data staging; Own folders for execution and results per test •Eases result collection •Support reproducibility (workspace snapshotting, environment tracking) •One job per test; Risks sheduler problems; LovedLoathed by system admins •Common Tools ReFrame or JUBE •Both European solutions •Other exist, e.g. Pavilion2, Ramble, … Note: Different CoE requre the use of different Benchmark Orchestrators 29 Tool –ReFrame •By CSCS Zürich (CH) •Uses Python based DSL •Job divided in 5 stages; Most stages are required •Convenient features: •Natively provides result collection from logfiles by defining regexes •Handles complex execution graphs •Simple validation logic for quick feedback •Opinion: •Stronger and more flexible •You can do basically everything if you call your own functions correctly •Power to customize can become a pitfall •Convenient features quickly become annoying when modelling simple tasks •Initial barrier relatively high; Time-consuming first setup •Nice documentation 26.11.2025 29