D5.1 Tooling for object storage

LUMI AI Factory Service Center Empowering Europe’s AI Ecosystem D5.1 Tooling for object storage 2 Deliverable 5.1 Tooling for object storage D5.1 Tooling for object storage 3 Project Title LUMI AI Factory Service Center Project Acronym LUMI-AIF Project Number 101234208 Type of Action HORIZON-JU-RIA Topic HORIZON-JU-EUROHPC-2025-AI-01-IBA-01 Starting Date of Project 01.03.2025 Ending Date of Project 29.02.2028 Duration of the Project 36 months Website lumi-ai-factory.eu Work Package WP5 Task T5.3, T5.1 Lead Authors Martin Golasowski (IT4I), Jan Martinovič (IT4I) Contributors Abdulrahman Azab (Sigma2), Jarno Laitinen (CSC), Lorand Janos Szentannai (Sigma2) Peer Reviewers Pauliina Somerkoski (CSC) Version 1.0 Due Date 31.8.2025 Submission Date 29.8.2025 Dissemination level X PU: Public SEN: Sensitive – limited under the conditions of the Grant Agreement EU-RES. Classified Information: RESTREINT UE (Commission Decision 2005/444/EC) EU-CON. Classified Information: CONFIDENTIEL UE (Commission Decision 2005/444/EC) EU-SEC. Classified Information: SECRET UE (Commission Decision 2005/444/EC) D5.1 Tooling for object storage 4 Version History Revision Date Editors Comments 0.1 9.7.2025 Martin Golasowski Initial structure of the document 0.2 7.8.2025 Martin Golasowski Introduction complete 0.3 22.8.2025 Abudlrahman Azab, Martin Golasowski, Jarno Laitinen Consolidated description of national resources, added section about data transfers 0.4 26.8.2025 Martin Golasowski, Lorand Janos Szentannai Finished data spaces and summary section, added Norwegian resources 0.5 29.8.2025 Martin Golasowski Finalization, LUMI AIF abbreviation fix, AI declaration added 1.0 29.8.2025 Anna Luoma Final quality checked performed by the PMO, sent to official review. Declaration on the Use of AI Assistance This report has been prepared with the support of ChatGPT and GPT-5-enabled Microsoft Copilot, which were used to assist in drafting text, checking language and content consistency, and gathering background information. All content has been thoroughly reviewed, fact-checked, and edited by the authors to ensure accuracy and alignment with the objectives of the report. Glossary of Terms Item Description ACL Access Control List API Application Programming Interface AWS Amazon Web Services CLI Command Line Interface – terminal client EOSC European Open Science Cloud FTP File Transfer Protocol GUI Graphical User Interface GPU Graphics Processing Unit HPC High Performance Computing HTTP HyperText Transfer Protocol IETF Internet Engineering Task Force D5.1 Tooling for object storage 5 iRODS Integrated Rule-Oriented Data System ISO International Organization for Standardization NFS Network File System POSIX Portable Operating System Interface SSH Secure Shell – protocol for remote command line interfaces and data transfer URL Uniform Resource Locator Executive Summary Description of object storage concepts, implementation and available tooling that should be provided along with support of the LUMI AIF users. Integration with computing resources focusing on efficient data staging and caching is discussed as well as connection to external data repositories and resources such as European Data Spaces. Available storage resources per each LUMI AIF partner country are summarized along with overview of common technologies available in European HPC centers. D5.1 Tooling for object storage 6 Table of Contents 1. Introduction ............................................................................................................... 7 State of the art in the object storages 7 LUMI object storage system 7 S3 implementations 7 2. Current tooling for object storage ............................................................................... 8 Command line tools 9 Software Development Kits (SDKs) 9 Graphical User Interfaces (GUIs) 9 AI oriented storage management solutions 10 Other object-oriented solutions 10 Connection to external data resources 11 3. Integration with computing resources....................................................................... 12 Staging on computing resources 13 3.1.1 SSH based access 13 3.1.2 In-cluster staging 13 3.2 Local data caching – ingestion from external resources 14 4. External data repositories and resources .................................................................. 15 National resources across the consortium partners 15 4.1.1 Czechia 15 4.1.2 Denmark 16 4.1.3 Estonia 16 4.1.4 Finland 17 4.1.5 Norway 17 4.1.6 Poland 18 Object storages across European HPC centers 18 5. Summary .................................................................................................................. 18 References ...................................................................................................................... 19 D5.1 Tooling for object storage 7 1. Introduction State of the art in the object storages The concept of object storage emerged in the early 2000’s as organizations faced increasing challenges storing and managing large volumes of unstructured data. Its development aimed to overcome the limitations of traditional storage systems in terms of scalability, metadata handling, and access methods. One of the first major milestones was the launch of Amazon S3 (Simple Storage Service) in 2006, which showcased how object storage could power large-scale cloud infrastructure. Since then, it has become a foundational technology in cloud computing, used widely by public cloud providers, enterprises, and software-defined storage solutions for its durability, cost-effectiveness, and ease of integration with APIs. However, despite widespread usage, there is no standard for the S3 protocol and there can be support issues between different clients and versions. There are different implementations of the protocol, which implement the S3 API. Contrary to the HTTP as an example, there are no IETF or ISO standards for S3 API. It is worth mentioning that object storage is primarily offered as a service with a well-defined API, which is supposed to be either by expert users or developers. There are many use-cases for object storages like backup or archive storage and collection of data from IoT devices. However, all these use cases often build their own interfaces and services on top of the object storage API. AI related use-cases are no different, appropriate tooling and services must be built on top of these interfaces making the technical complexity transparent to the user. LUMI object storage system LUMI-O is an object storage service accessible through a web interface and S3 API, and LUMI AIF continues to operate similar service. LUMI-O is based on completely different technology than the Lustre parallel filesystem (LUMI-P and LUMI-F). Also, unlike the Lustre parallel file system, LUMI-O is a separate service from LUMI compute partitions. We discuss this topic in more detail in the chapter "Integration with Computing Resources" below. LUMI-O offers a total of 30 PB storage space for storing, sharing, and staging of data. By default, 150 TB of storage space is allocated per LUMI project, but projects can request for more LUMI-O storage space by contacting the LUMI helpdesk. S3 implementations This section provides a list of common implementations of the S3 API, including the AWS itself. It is worth noting that many of the solutions offer good scalability within a single datacenter, and scaling across multiple locations requires building a complex network infrastructure, including load balancers and D5.1 Tooling for object storage 8 points-of-presence to hide the physical limits of a single datacenter to the users (i.e. hyperscaling). Given the distributed nature of compute and storage resources across Europe, good data transfer solution is needed to move data efficiently between locations, which is a feature that is usually not offered by the solutions listed below. Amazon S3 (Simple Storage Service) is a highly scalable, durable, and secure object storage service provided by Amazon Web Services (AWS). Launched in 2006, S3 was one of the first services offered by AWS and remains a cornerstone of its cloud infrastructure. S3 allows users to store and retrieve any amount of data from anywhere on the web using a simple web interface or RESTful API. It organizes data into buckets and assigns each object a unique key, enabling fine-grained access control, versioning, and metadata tagging. With built-in features like lifecycle policies, cross-region replication, and integration with AWS analytics and machine learning services, S3 is widely used for backup, disaster recovery, data lakes, content delivery, and application hosting. Several open-source and commercial implementations offer S3-compatible object storage for organizations seeking alternatives to AWS or for on-premises deployments. MinIO is a high-performance, lightweight, and Kubernetes-native object storage system designed for cloud-native applications. Fully compatible with the S3 API, MinIO supports advanced features like erasure coding, encryption, versioning, and multi-tenant isolation. Its simplicity, speed, and scalability make it a popular choice for private cloud, edge, and hybrid cloud environments. Ceph RGW (RADOS Gateway) is another prominent S3-compatible solution, part of the broader Ceph distributed storage system. Ceph RGW provides a gateway interface that allows users to access Ceph's object storage using S3 or Swift APIs. Ceph is known for its high availability, fault tolerance, and unified storage capabilities, offering block, object, and file storage in a single platform. While more complex to deploy than MinIO, Ceph RGW is favored in large-scale enterprise or research environments requiring deep customization and integration with other storage modalities. OpenStack Swift is an open-source, distributed object storage system designed to store and retrieve unstructured data at scale. It is a core component of the OpenStack cloud platform and provides a highly available, fault-tolerant storage backend minimizing single point of failure. Swift is ideal for storing large volumes of data such as backups, media files, and archives. It supports features like replication, eventual consistency, and metadata, and can be accessed via a RESTful API or compatible clients. Swift is commonly used in private clouds or service provider environments where open-source control and scalability are key requirements. OpenStack Swift offers S3 API support through middleware such as Swift3 or radosgw (when used with Ceph), enabling compatibility with S3-compatible tools and applications. This allows users to interact with Swift using familiar S3 clients while benefiting from Swift's scalable, open-source architecture. 2. Current tooling for object storage The object storage tooling consists mainly of Software Development Kits (SDKs) for various programming environments and applications providing CLIs and desktop GUIs. These tools interact with the S3 directly, therefore it is a user responsibility to set up the common required parameters such as URL of the endpoint, access and secret keys, zone and region name, etc. D5.1 Tooling for object storage 9 Command line tools s3cmd Website: https://s3tools.org/s3cmd A Python-based CLI tool for managing S3 buckets and files. It supports operations like uploading, downloading, syncing, and modifying metadata or Access Control Lists (ACLs) to enforce data access policies. It's well-suited for scripting and terminal-based workflows that don't require programming. s5cmd Website: https://github.com/peak/s5cmd A high-performance CLI tool written in Go, designed for fast, parallel S3 operations such as batch uploads, downloads, and sync. It is ideal for users handling large-scale or time-sensitive S3 workloads, offering significant speed advantages over traditional tools. It offers superior performance to the native s3cmd/boto based implementations. Software Development Kits (SDKs) Libraries with bindings to many programming languages exists, here we list the most popular ones including Boto3, which is part of the official AWS SDK. AWS SDK for Python (Boto3) Website: https://boto3.amazonaws.com/v1/documentation/api/latest/index.html The official Python SDK for AWS services, including full support for S3 operations like uploads, multipart transfers, and bucket management. AWS SDK for Go (v2) Website: https://aws.github.io/aws-sdk-go-v2/ Modern, modular SDK for Go that supports S3 operations and integrates easily with Go-based cloud services or automation tools. MinIO JavaScript Client SDK Website: https://min.io/docs/minio/linux/developers/javascript/minio-js.html Lightweight Node.js SDK for working with S3-compatible storage, ideal for use with MinIO but works with any S3 endpoint. Graphical User Interfaces (GUIs) GUIs are the most user-friendly way to access the object storages. There is a number of desktop applications allowing streamlined interaction with objects and buckets as well as web based interfaces, such as the native MinIO web application. Here are two examples of popular GUI application with S3 support: D5.1 Tooling for object storage 16 compliance with FAIR principles by offering access through MyAccessID and implementing the DataCite metadata model. On a national level, a coordinated national layer of research-data repositories is being developed within the European Open Science Cloud framework under EOSC CZ Initiative 10 . The core component is the National Repository Platform for Research Data 11 , which provides repository infrastructure, including a general-purpose national repository, while also enabling integration with institutional and domainspecific repositories. Metadata from these repositories are harvested into the National Metadata Directory 12 , which serves as the central discovery service. To ensure interoperability and alignment with EOSC, the metadata are exposed in standard formats such as DCAT-AP, making datasets findable both nationally and internationally. This platform will be connected to the LUMI AI Factory, thereby facilitating availability of large datasets in the AI Factory ecosystem. The access to large datasets will be aligned with the goals of the EOSC CZ Initiative, which actively implements the Open Science approach and FAIR principles in the research data management domain in the Czech Republic. The Czech national node of the European Open Science Cloud is currently being established, aiming of building a National Data Infrastructure as a common platform for sharing, managing and accessing data and computing resources for research purposes. EOSC CZ initiative also includes activities for support of the integration of a distributed large data transfer service into the Czech EOSC ecosystem, which will allow the staging of the data close to the computing resources (such as GPU enabled HPC clusters). 4.1.2 Denmark In DK there are several repositories with accessibility policies that are available to being used by the LUMI AI factory. The “Datavejviser” acts as the main entry point for publicly available datasets. The specifications are derived from the European Commission's specification DCAT-AP 2.0, which aims to standardize and establish coherence and interoperability between data portals in member countries. The European Commission's specification likewise builds upon the W3C specification DCAT (Data Catalog Vocabulary). These repositories are reachable via the CKAN-API (https://docs.ckan.org/en/2.9/api/) (https://github.com/ckan/ckanext-dcat#rdf-dcat-endpoints) The CKAN’s Action API is a, RPC-style API that exposes all of CKAN’s core features to API clients. All of a CKAN website’s core functionality can be used by external code. 4.1.3 Estonia UT HPC has been operating an object storage system https://s3.hpc.ut.ee since 2024. It is based on open source object storage MinIO. Object storage became relevant for UT HPC as users asked for a storage solution that is S3 compatible, from there MinIO was chosen. S3 is currently being used for HPC jobs, web apps and as a storage solution for a myriad of offered services. The future currently holds a need to 10 EOSC CZ Initiative: https://www.eosc.cz/en 11 Czech National Repository Platform for Research Data: https://data.narodni-repozitar.cz 12 Czech National Metadata Directory: https://nma.eosc.cz/ D5.1 Tooling for object storage 17 migrate over to a CEPH or SeaweedFS based solution, but as long as evaluations are under way nothing is set. 4.1.4 Finland CSC has been operating Allas object storage system (https://docs.csc.fi/data/Allas/ ) based on CEPH backend since 2019. Therefore CSC has experience on operating such system before it was taken into use also in LUMI. The main purpose for Allas is to store data for project lifetime, which does not need to be actively processed on the HPC systems. CSC has scripts, which will delete data from Lustre scratch space of the project based on the file timestamp and quota of the scratch. CSC has been implementing wrapper scripts called "a-commands" for the rclone with support on packaging and compressing the files. This is helpful in cases where there is a large amount of files. CSC has been supporting both Swift and S3 protocols, but is now migrating to S3, which is the only protocol LUMI is using. CSC has provided guidance how to use Allas from LUMI. A script called allas-conf support swift and S3 configuring the access. CSC advices also graphical user interfaces such as CyberDuck and has made a simple web UI for the Allas backend. Other alternative browser based access is the OpenStack Horizon (very limited) and CSC specific sensitive data transfer tool including encryption. For back-up there is a script for the Restic tool. 4.1.5 Norway Sigma2, Norway’s national research infrastructure provider, has been operating the National eInfrastructure for Research Data (NIRD) since 2017. NIRD, now in its second iteration and operational since the beginning of 2023, was designed with user-focus, scalability, and adaptability in mind. It serves as a generic storage infrastructure comprising two independent but interconnected resources: NIRD Data Peak and NIRD Data Lake. NIRD tenants and users have the option to apply for resources from either NIRD Data Peak or NIRD Data Lake, or both, depending on their specific functional requirements and usage patterns. These resources provide cost-efficient data and storage services to users. NIRD is built on IBM hardware and software, utilizing IBM Storage Scale (GPFS) as its core. It offers functionalities such as tenant separation, data lifecycle management, transparent tiering, automatic replication, snapshots, storage policies, encryption, metadata cataloging and more. NIRD Data Lake provides unified fileand object-storage access through redundant, high-availability protocol nodes, and the object storage endpoint is https://s3.nird.sigma2.no. The object storage service utilizes IBM’s High-Performance Object (HPO) storage with NooBaa as its core. It supports the S3 protocol and S3 APIs and is integral to Sigma2’s data-centric approach, where data is at the core of research and services - such as cloud services (NIRD Service Platform) - are built around it and tightly integrated to provide a seamless user experience. Currently, Sigma2 is working on further enhancing the workflow for Norwegian HPC users, and enabling data staging and caching between remote HPC systems and NIRD based on the S3 protocol. In addition to NIRD’s generic storage infrastructure, Sigma2 coordinates Norway’s national services for sensitive research data. These services provide secure, policy-compliant environments for storing, processing, and sharing data classified under GDPR and national health and privacy regulations. The solutions are designed to support projects across health, social sciences, and other domains where sensitive data handling is mandatory. While physically and logically separated from the generic NIRD D5.1 Tooling for object storage 18 services, the sensitive data platforms form part of Norway’s broader e-infrastructure ecosystem and are being aligned with European initiatives for Trusted Research Environments (TREs). 4.1.6 Poland AI storage is provided as S3 service via the PLGrid portal. Two zones of S3 Storage are available, geographically distributed to two datacenters, located 10 km from each other. Access to service is available by grants accepted via PLGrid portal. S3 is one of storage available for the users, S3 is mainly used for long term storage that is not needed for computational purposes in near future. There are modules available on supercomputers for using s3 on computing clusters (for example rclone or minio client). We have service documentation available for users (https://guide.s3p.cloud.cyfronet.pl/quickstart_en.html). Additionally, a WEB interface based on filestash is available for users to get access to this type of service. Users can check via a dedicated portal their status of grants, quotas and check number of objects or data amount. This object storage is planned to be integrated with LUMI/LUMI AI for the AI service provision. Object storages across European HPC centers The European HPC centers also operate object storages. Below we provide a non-exhaustive list of several HPC centers, illustrating that S3 access is becoming quite common, although its local implementation type varies a lot. • Jülich Supports S3 protocol. Access token to the MinIO cluster is acquired via web portal. MinIO client (mc), python boto3 and s3cmd tools are advised 13 • SURF object storage 14 is implemented with Ceph RADOS Gateway and the authentication is via OpenStack Keystone. Both S3 and Swift are supported. User guide lists s3cmd, awscli, S5cmd, rclone, python swift client, curl and Cyberduck (GUI).Also read-only mounting via S3QL, S3FS, rclone and Goofys is mentioned as back-up solution. • MeluXina uses DDN S3DS system 15 , which supports S3 protocol. Credential are received via helpdesk. Their user guide has a table comparing the features of s3cmd, awscli, CyberDuck and python Boto3. 5. Summary In this document we provide a brief description of object storage as a concept for storing data compared to traditional files and folders. We provide a comprehensive list of existing client applications and various usage models. Important topic of this document is user experience with object storage and efficient data movement close to the compute resources when executing complex AI workloads. We describe several concepts which will be leveraged to provide tooling for the LUMI AIF users, which are expected to use many very large datasets repeatedly, therefore appropriate data locality must be ensured through transparent caching a reasonable amount of abstraction. 13 Jülich object storage: https://apps.fz-juelich.de/jsc/hps/just/object-storage.html 14 SURF: https://servicedesk.surf.nl/wiki/spaces/WIKI/pages/29557216/Object+Store 15 MeluXina: https://docs.lxp.lu/web_services/s3/ D5.1 Tooling for object storage 19 Significant data movement is expected in LUMI AIF, between the AI factory itself, local storages as well as remote data resources and repositories. The document also contains an overview of national storage resources relevant for AI use-cases across the LUMI AIF consortium partners as well as storages operated by European HPC centers. We provide description of tooling suitable for common AI use-cases. As there are many technologies and protocols involved in data movement, the LUMI AIF should provide tailored support for its users by providing appropriate tooling and automation of the data transfers by working closely with the users and ensuring that optimal data access patterns are used. References [1] NAKAMURA, Ryo; KUGA, Yohei. Multi-threaded scp: Easy and Fast File Transfer over SSH. In: Practice and Experience in Advanced Research Computing 2023: Computing for the Common Good. 2023. p. 320-323.