D4.4 Containerized workflows

LUMI AI Factory Service Center Empowering Europe’s AI Ecosystem D4.4 Containerized workflows 2 D4.4 Containerized workflows D4.4 Containerized workflows 3 Project Title LUMI AI Factory Service Center Project Acronym LUMI-AIF Project Number 101234208 Type of Action HORIZON-JU-RIA Topic HORIZON-JU-EUROHPC-2025-AI-01-IBA-01 Starting Date of Project 01.03.2025 Ending Date of Project 29.02.2028 Duration of the Project 36 months Website lumi-ai-factory.eu Work Package WP4 Task T4.2. Provision of rich selection of ready-to-use AI tools Lead Authors Ville Ahlgren (CSC) Contributors Juha Hulkkonen (CSC) Tomáš Martinovic (IT4I) Pauliina Somekoski (CSC) Peer Reviewers Markus Koskela (CSC) Version 1.0. Due Date 29.8.2025 Submission Date 29.8.2025 Dissemination level x PU: Public SEN: Sensitive – limited under the conditions of the Grant Agreement EU-RES. Classified Information: RESTREINT UE (Commission Decision 2005/444/EC) EU-CON. Classified Information: CONFIDENTIEL UE (Commission Decision 2005/444/EC) EU-SEC. Classified Information: SECRET UE (Commission Decision 2005/444/EC) D4.4 Containerized workflows 4 Version History Revision Date Editors Comments 0.1 5.6.2025 Pauliina Somerkoski Ville Ahlgren Initial version 0.2 24.7.2025 Ville Ahlgren Added bulk of the document's content and structure. 0.3 25.7.2025 Juha Hulkkonen Added descriptions for terms. 0.4 4.8.2025 Ville Ahlgren Overall polish, finalize chapters Next Steps and Workflow Files, and add chapter AI Models, Software, and Use. 0.5 5.8.2025 Ville Ahlgren Minor grammatical fixes and restructuring the Workflow Catalog chapter. 0.6 12.8.2025 Ville Ahlgren Added references and improved several sections based on feedback. 0.7 13.8.2025 Tomáš Martinovic Added clarifying sentence to section Workflows, Containerized and a couple of examples to Vision-Language workflow description. 0.8 18.8.2025 Ville Ahlgren Minor updates to multiple chapters based on received feedback. 0.9 27.8.2025 Ville Ahlgren Changed dissemination level to public, final version ready for submission. 1.0 29.8.2025 Anna Luoma Final quality check performed by the PMO, sent to official review. D4.4 Containerized workflows 5 Glossary of Terms Item Description Aitta AI Inference service developed by CSC. Airflow Data-engineering oriented workflow engine. AWQ Activation Aware Quantization is an AI model compression technique, designed to enable efficient inference while maintaining accuracy. Docker Docker is a platform to package software environments into units called containers. CI/CD CI Continuous Integration and CD Continuous Delivery are practices in which code changes are automatically tested, integrated, and delivered. Flash Attention An efficient algorithm used in large language models to optimize memory access. GPQA Graduate-Level Google-Proof Questions and Answers is a benchmark designed to evaluate the reasoning and problem-solving abilities of large language models. HPC-API High-Performance Computing - Application Programming Interface. Llama A family of large language models developed by Meta AI. LLM Large Language Model. LUST LUMI User Support Team, responsible for the official LUMI software stack and user support. Lustre Lustre is a high-performance distributed file system, commonly used in largescale computing environments, such as HPC clusters. MLOps Machine Learning Operations is a set of practices that aims to streamline the development, deployment, and maintenance of machine learning models MI250x AMD GPU designed for high-performance computing and AI workloads, used in systems like LUMI. MMLU Massive Multitask Language Understanding is a benchmark for evaluating the multitask accuracy and general knowledge of large language models. Mistral Mistral is a family of large language models developed by the company Mistral AI. Kubeflow A popular workflow system for machine learning operations Podman An open-source container management tool alternative to Docker. PyTorch PyTorch is an open-source machine learning library developed by Meta AI, widely used for developing deep learning models. Qwen Qwen is a family of large language models developed by Alibaba Cloud. ROCm AMD's open source software stack designed for GPU-accelerated high performance computing (HPC) and machine learning workloads. Singularity A container platform targeted for high-performance computing (HPC) use case. STT Speech-to-Text (STT) refers to technologies or models that convert spoken language into written text, commonly used in voice recognition systems. TensorFlow TensorFlow is an open-source machine learning platform developed by Google, used for developing, training, and deploying machine learning models. Ubuntu A popular and generic Linux distribution widely used with machine learning and AI related applications. VLLM VLLM is an open-source Python library and inference engine designed for efficient and fast inference and serving of large language models. D4.4 Containerized workflows 6 YAML YAML is a human-readable data serialization format often used for configuration files and data exchange between languages with different data structures. YOLO You Only Look Once (YOLO) is a real-time object detection algorithm in computer vision that predicts bounding boxes and class probabilities for objects in images in a single evaluation. Executive Summary One important part of the LUMI AI Factory Service Center's mission is to contribute to empowering Europe's AI ecosystem by developing and making available ready-to-use AI software solutions. Central to this effort is the provision of Containerized Workflows, which simplify the deployment of AI capabilities across various applications. The Containerized Workflows consists of pre-built container images with tailored software, example data sets and pre-tested models to improve the accessibility of AI usage and development. The development of these workflows is linked with the broader LUMI AI Factory software ecosystem, which includes HighPerformance Computing API (HPC-API) and Machine Learning Operations (MLOps) tools. Together, these elements will form an integrated environment where the development of different components supports each other. For instance, the container base images for various parts of the ecosystem, and the Containerized Workflows will share the same foundations. The key objectives of developing the Containerized Workflows concept include delivering the first complete workflows early on. Additionally, new workflows will be continuously added throughout the lifetime of the LUMI AI Factory Service Center based on emerging opportunities and user needs. Initially, the focus is on developing workflows for Large Language Models (LLMs), including text processing workflows for tasks such as dataset curation, LLM fine-tuning, and LLM evaluation. An important part of this development effort involves establishing processes, best practices, and tooling for creating these workflows. The concept of Containerized Workflows aims to include everything necessary to help new users get started quickly. It also provides a proven starting point for expert users to build upon for more specialized tasks. To support this, documentation with examples, openly licensed models tested with the workflows, and example datasets will be available directly within the environment. This document provides an update on the status of the Containerized Workflows development and outlines the next steps in this ongoing effort. D4.4 Containerized workflows 7 Table of Contents 1. Introduction ............................................................................................................... 8 2. Background ................................................................................................................ 9 2.1 AI Models, Software, and Use Cases 9 2.2 Current Use of Containers 10 2.3 Workflows, Containerized 11 3. Design and Implementation ...................................................................................... 13 3.1 Container Images 13 3.2 Workflow Files 14 3.3 Additional Artifacts 14 3.4 Installation and Directory Structure 15 3.5 Current Status 15 4. Workflow Catalog ..................................................................................................... 16 4.1 LLM Text Processing 16 4.2 Vision-Language Batch Processing 17 4.3 YOLO Image Processing Workflow 18 4.4 Planned Future Workflows 19 5. Next Steps ................................................................................................................ 20 6. References ............................................................................................................... 21 D4.4 Containerized workflows 8 1. Introduction In recent years, the landscape of artificial intelligence has undergone significant advancements, with a growing number of AI tools and models becoming accessible and practical for a wide range of applications [1]. These developments have contributed to the democratization of AI, enabling researchers and practitioners across various domains to incorporate advanced technologies into their work [2]. As the use of AI continues to expand, the need for structured, scalable approaches to streamline deployment and integration has become increasingly evident. To address these emerging needs and opportunities in AI adoption, the LUMI AI Factory Service Center is developing AI software solutions, including the Containerized Workflows concept, designed to streamline the execution of AI-related tasks. With Containerized Workflows concept, this is achieved by offering pre-prepared software, container images, datasets, models, and documentation as packages, tested to work properly together. This effort is part of the larger LUMI AI Factory software ecosystem development, which includes developing tools and APIs such as the High-Performance Computing API (HPC-API), Machine Learning Operations (MLOps) tools, and AI model inference platforms (Aitta). These components collectively enhance the functionality and ease of use of AI software on LUMI and the future LUMI-AI supercomputers. The Containerized Workflows work package is set to deliver a selection of functional workflows early on, with continuous updates based on user needs and technological advancements throughout the project. The initial focus is on developing workflows for Large Language Models (LLMs), including text processing, fine-tuning, and model evaluation tasks as well as workflows of different image processing tasks. The development process for the Containerized Workflows involves establishing processes, best practices, and tools for creating and maintaining these workflows and the related artifacts. Key considerations include identifying use cases that address specific research needs, guiding the development of tailored workflows. The goal is to provide comprehensive workflow packages that enable both novice and expert users to quickly deploy and customize AI workflows, supported by detailed documentation, example datasets, and pre-tested models within the LUMI environment. D4.4 Containerized workflows 9 2. Background The following sections provide an overview of key aspects underlying the demand for the Containerized Workflows as an approach for delivering AI capabilities on high-performance computing environments. These include the growing availability of open AI models and software tools [1], the adoption of container technologies in high-performance computing environments [3], and the opportunity for structured, reproducible workflows to accelerate the adoption of different AI tools in various domains and practical applications. 2.1 AI Models, Software, and Use Cases In recent years, there has been a significant increase in the availability of open AI models and AI-related tools. Examples include general-purpose language models such as LLaMA [4], Mistral [5], and Poro [6], domain-specific models for areas like protein structure prediction (AlphaFold [7]), as well as widely used software frameworks like PyTorch [8], Hugging Face Transformers [9] and optimized inference engines such as vLLM [10]. A large selection of resources is now freely accessible, enabling researchers and developers to build on these advanced capabilities without relying on proprietary systems, or coming up with custom solutions from scratch, for example when creating automated document analysis pipelines, training climate models, or developing computer vision systems for scientific imaging. While the availability of open models and tools offers great opportunities, it also introduces significant complexity, especially for those new to the field. Many high-quality pre-trained models are openly available across domains such as language, vision, speech, and generative tasks. Examples include: • Natural Language Processing: Models used in tasks like classification, summarization, and translation, for example automatically categorizing thousands of incoming customer support emails by topic, summarizing lengthy research reports for quick review, or translating technical documentation to multiple languages for international teams. • Computer Vision: Models used in object detection, classification, and segmentation, for example identifying defects on a manufacturing line in real time, classifying satellite images to detect land use changes, or segmenting microscopic images for medical diagnosis. • Speech Recognition: Models used for transcription and other audio-based tasks, for example converting meeting recordings into searchable transcripts, enabling voice-based commands in industrial equipment, or generating captions for live broadcasts. • Generative Models: Models used for image generation from text prompts, audio synthesis, and video generation, for example creating photorealistic marketing visuals from a short description, synthesizing voiceovers in multiple languages for training videos, or producing short animations for educational content. These models are often released under licenses that permit reuse and adaptation, and are typically accompanied by example code and documentation [11]. One important distinction to make is between open-weight models and fully open models. Open-weight models provide trained parameters for use in inference or fine-tuning but may withhold training data, code, or full methodology. Fully open models release these alongside the weights, enabling better D4.4 Containerized workflows 16 4. Workflow Catalog This chapter lists the currently implemented workflows and introduces few new workflows planned to be implemented in future. 4.1 LLM Text Processing Status: Initial implementation has been completed, and the workflow is available for LUMI user. Description: The LLM Text Processing workflow is designed to facilitate efficient text processing using Large Language Models (LLMs). This workflow is implemented as a ready-to-go container image and recipe, specifically tailored for tasks such as dataset curation. It leverages the vLLM library to process input files based on user-provided instructions and generate corresponding output files. The vLLM engine implements optimized tensor-parallel batched inference for high performance, while the workflow as a whole provides a tested, ready-to-use setup that allows users to get started quickly without extensive configuration or troubleshooting. Key Features: • Input and Output Handling: The workflow processes files in a specified input directory and generates output files in a designated output directory. Each input file is processed according to the given instructions, with the output being either a single file or one output file per input file. • Task Configuration: The workflow allows for command-line or YAML configuration file inputs, providing flexibility in defining the processing tasks. Users can specify the model to be used, input and output file paths, and instructions for processing. • Custom Scripts and Models: Users can customize the workflow script and select from a set of ready-to-go models that have been tested to work efficiently within the workflow. • Scalability: The processing speed scales well with the number of input files. Optimal performance is achieved with a larger number of input files, leveraging the efficiency of batched processing used in vLLM. Planned Improvements: • Allow multi-step processing without reloading the language model. Usage overview: To utilize the LLM Text Processing workflow, users can begin by copying one of the provided example directories from the workflow's examples folder into their own working directory. The examples serve as templates and include all necessary files and configurations to run a text processing job. Once the example directory is copied, users can submit the job to the computing cluster using a batch script included in the directory. This script requires users to specify their project account to allocate the necessary resources for job execution. D4.4 Containerized workflows 17 During job execution, users can monitor the progress and status using standard Slurm commands. This allows for real-time tracking of the job's output and overall progress once it has started. Upon completion of the job, the workflow generates output files in the specified output directory. These files contain the processed results based on the input files and instructions provided. For detailed specific examples, users can refer to the example README files located within the workflow example directories. These files provide comprehensive instructions, sample task configurations in YAML format, and example instruction files that guide users through setting up and customizing their text processing tasks. 4.2 Vision-Language Batch Processing Status: Initial implementation has been completed, and the workflow is available for LUMI user. Description: The Vision-Language Batch Processing workflow is designed to facilitate efficient image processing, such as object detection, classification, segmentation, motion-tracking, using visionlanguage models. This workflow leverages container and with Transformers library to process input images based on user-provided instructions and generate corresponding output files with textual answers. Key Features: • Input and Output Handling: The workflow processes images in a specified input directory and generates output text files in a designated output directory. Each input image is processed according to the given instructions, with the output being one text file per input image. • Task Configuration: The workflow uses a YAML configuration file to define the processing tasks. Users can specify the model to be used, input and output file paths, and instructions for processing. • Custom Scripts and Models: Users can customize the workflow script and select from a set of ready-to-go models that have been tested to work properly within the workflow. • Scalability: The processing speed scales well with the number of input images, based the efficiency of batched processing provided in the Transformers library. Planned Improvements: • Enable using quantized models like AWQ to improve loading times and overall performance. • Conduct an additional code review to ensure the quality and maintainability of the workflow. Usage Overview: To utilize the Hugging Face Vision-Language Batch Processing workflow, users can begin by submitting the provided SLURM script to launch a demo job on LUMI. The script requests GPU resources and sets sensible defaults for interactive testing. Once the job is submitted, the container execution begins, and the vision-language pipeline is executed. The pipeline reads a YAML task specification, loads a Qwen2.5 vision-language model, loops over the input images, asks the prompt, and stores the model's answer in the corresponding output text files. D4.4 Containerized workflows 18 Users can create their own tasks by preparing input images, writing a prompt, creating a task.yaml file that points to the inputs, outputs, and instructions, and running the job by adapting the last line of the run_container.sh script to point to their YAML and re-submitting with sbatch. For detailed specific examples, users can refer to the example README files located within the workflow example directories. These files provide comprehensive instructions, sample task configurations in YAML format, and example instruction files that guide users through setting up and customizing their image processing tasks. 4.3 YOLO Image Processing Workflow Status: Initial implementation has been completed, and the workflow is available for LUMI user. Description: The YOLO Image Processing Workflow is designed to facilitate efficient image processing using YOLO (You Only Look Once) models. This workflow is implemented as a ready-to-go container image and a workflow recipe, suited for tasks such as object classification. It processes input image files and generates corresponding output files with class predictions in a human-readable format. Key Features: • Input and Output Handling: The workflow processes images in a specified input directory and generates output text files in a designated output directory with one text file per input image. • Task Configuration: The workflow uses a YAML configuration file to define the processing tasks. Users can specify the model directory and file, input and output file paths. • Scalability: The processing speed scales well with the number of input images, leveraging the efficiency of batched processing. Planned Improvements: • None at the moment Usage Overview: To utilize the YOLO Image Processing Workflow, users can begin by copying one of the provided example directories from the workflow's examples folder into their own working directory. The examples serve as templates and include all necessary files and configurations to run an image processing job. Once the example directory is copied, users can submit the job to the computing cluster using a batch script included in the directory. This script requires users to specify their project account to allocate the necessary resources for job execution. During job execution, users can monitor the progress and status using standard Slurm commands. This allows for real-time tracking of the job's output and overall progress once it has started. Upon completion of the job, the workflow generates output text files in the specified output directory. These files contain the class predictions based on the input images. D4.4 Containerized workflows 19 For detailed specific examples, users can refer to the example README files located within the workflow example directories. These files provide comprehensive instructions, sample task configurations in YAML format, and example instruction files that guide users through setting up and customizing their image processing tasks. 4.4 Planned Future Workflows Looking forward to expanding the selection of available workflows, we are considering introducing several new workflows. These upcoming workflows should be beneficial for addressing several opportunities in research and other use cases. Below is our current list of planned and ones in idea stage. LLM Fine-tuning Status: Planned to be implemented Description: This workflow will provide ready-to-go recipes and container images for fine-tuning pretrained and instruction-tuned foundation models using custom datasets. The workflow will include a set of tested models readily available on LUMI, container images, example datasets, and example job files for launching training jobs. This will enable users to adapt existing models to specific tasks or domains, improving their performance and relevance. The workflow will support a variety of pre-trained models and model architectures. LLM Evaluation Status: Planned to be implemented Description: This workflow will offer a containerized solution for running standard LLM evaluation benchmarks, such as GPQA (Graduate-Level Google-Proof Q&A) and MMLU (Massive Multitask Language Understanding). Standard benchmarks are central for assessing the performance and capabilities of language models, providing insights into model strengths and weaknesses. Speech to Text Status: Under consideration Description: This workflow will enable transcribing audio files to text using Speech-to-Text (STT) models. This capability can be useful multiple research and application areas. The workflow will include pre-trained models, container images, and example audio files to demonstrate the transcription process. Workflow Frameworks Status: Under consideration Description: We are evaluating the usefulness of providing workflows enabling the use of workflow engines such as Nextflow, OpenWDL, Snakemake and StreamFlow. D4.4 Containerized workflows 20 5. Next Steps The next phase of development for the Containerized Workflows includes increasing the coverage of the workflows for different tasks, enhancing the functionality, accessibility, and user experience of the existing workflows, and streamlining the Containerized Workflows development process as well as finalizing a process for user support requests related to the workflows. Building on the initial set of provided workflows of LLM Text Processing, Vision-Language Batch Processing, and YOLO Image Processing Workflows, we will proceed to expand the catalog of available workflows. The next phase is planned to include the development of workflows for LLM fine-tuning and LLM benchmark evaluation. Additionally, we will explore workflows for audio processing, such as Speech-to-Text. We will also continue tracking the software and AI model landscape to identify opportunities for integrating new technologies and tools to ensure that the Containerized Workflows concept remain up-to-date and effectively meet the needs of the user community. Maintaining the created workflows is crucial. This involves regularly updating the containers images to the latest software versions to improve performance, fix bugs, and maintain the security of the software components. Additionally, system upgrades on LUMI may break compatibility with the workflows, which may require additional work to ensure continuous operation and smooth user experience. We aim to provide a stable and efficient environment for users to continuously accomplish their AI-related tasks. Continuous improvement of the development processes is needed for achieving high standards for the Containerized Workflows concept. This involves adopting best practices in software development, containerization, and efficient collaboration between the contributing partner organizations. By refining the processes, we aim to ensure the workflows are robust, reliable, and beneficial to the users. Code reviews, automated testing, and CI/CD pipelines are planned to be implemented to streamline the development process. Additionally, we are looking to establish clear channels for user communication and feedback to enable us to gather insights and address any issues the users may face promptly. We are looking to provide regular updates to keep the users informed about improvements and upcoming workflows. In conclusion, the groundwork for the Containerized Workflows has been successfully laid with the initial set of workflows and supporting infrastructure now available on LUMI. This foundation enables us to move forward with expanding functionality, improving usability and improving the development process. Moving forward, our focus is on collaboration, maintainability, and scalability of the concept. Through ongoing technical refinement and user engagement, we strive to make sure that the Containerized Workflows concept becomes a sustainable and impactful toolset for AI development and research on LUMI, and at later stage, on LUMI-AI. D4.4 Containerized workflows 21 6. References [1] "Recent Trends in Artificial Intelligence Technology: A Scoping Review [arXiv:2305.04532v3]," 2023. [2] "Exploring the Latest Trends in AI Technologies: A Study on Current State, Application and Individual Impacts [10.4236/jcc.2024.128002]," 2024. [3] "Containerisation for High Performance Computing Systems: Survey and Prospects [arXiv:2212.08717]," 2022. [4] "Huggingface Llama-3.3-70B-Instruct Model Card," [Online]. Available: https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct. [5] "Huggingface Mistral-Small-3.2-24B-Instruct-2506 Model Card," [Online]. Available: https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506. [6] "Huggingface Poro-34B-chat Model Card," [Online]. Available: https://huggingface.co/LumiOpen/Poro-34B-chat. [7] "AlphaFold," [Online]. Available: https://deepmind.google/science/alphafold/. [8] "Pytorch," [Online]. Available: https://pytorch.org . [9] "Huggingface Transformers," [Online]. Available: https://huggingface.co/docs/transformers/en/index. [10] "vLLM," [Online]. Available: https://docs.vllm.ai/en/latest/. [11] "An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registry [arXiv:2303.02552]," 2023. [12] "Comprehensive Analysis of Transparency and Accessibility of ChatGPT, DeepSeek, and other SoTA Large Language Models [arXiv:2502.18505v1]," 2025. [13] "Tensorflow," [Online]. Available: https://www.tensorflow.org. [14] "Numpy," [Online]. Available: https://numpy.org/. [15] "Pandas," [Online]. Available: https://pandas.pydata.org. [16] "LUMI Documentation, Container jobs," [Online]. Available: https://docs.lumisupercomputer.eu/runjobs/scheduled-jobs/container-jobs/. D4.4 Containerized workflows 22 [17] "LUMI EasyBuild containers," [Online]. Available: https://github.com/Lumi-supercomputer/LUMIEasyBuild-containers. [18] "Enabling Message Passing Interface Containers on the LUMI Supercomputer [a rXiv:2407.19928]," 2024. [19] "LUMI Documentation - Singularity," [Online]. Available: https://docs.lumisupercomputer.eu/software/containers/singularity/. [20] "Singularity Documentation - Security in Singularity," [Online]. Available: https://docs.sylabs.io/guides/3.5/user-guide/security.html. [21] "LUMI Documentation, GPU nodes - LUMI-G," [Online]. Available: https://docs.lumisupercomputer.eu/hardware/lumig/. [22] "LUMI Documentation, Network and interconnect," [Online]. Available: https://docs.lumisupercomputer.eu/hardware/network/. [23] "Radeon Open Compute Documentation," [Online]. Available: https://rocm.docs.amd.com/en/latest/. [24] "NVidia Cuda," [Online]. Available: https://developer.nvidia.com/cuda-toolkit. [25] "A Systematic Literature Review on Graphics Processing Unit Accelerated Realm of HighPerformance Computing [10.47941/ijce.1813]," [Online]. [26] "LUMI-container Image Docker Recipes," [Online]. Available: https://github.com/sfantao/LUMIcontainers/tree/main/RecipesDocker. [27] "Podman," [Online]. Available: https://podman.io/. [28] "Docker documentation, Dockerfile," [Online]. Available: https://docs.docker.com/reference/dockerfile/. [29] "LUMI User Documentation/LUMI AI Factory," [Online]. Available: https://docs.lumisupercomputer.eu/software/local/lumi-aif. [30] "Is Open Source the Future of AI? A Data-Driven Approach. [arXiv:2501.16403v1]," 2025. [31] "How Far Behind Are Open Models?," [Online]. Available: https://epoch.ai/blog/open-modelsreport.