D4.3 Inference Service documentation

LUMI AI Factory Service Center Empowering Europe’s AI Ecosystem D4.3 Inference Service documentation 2 D4.3 Inference service documentation D4.3 Inference Service documentation 3 Project Title LUMI AI Factory Service Center Project Acronym LUMI-AIF Project Number 101234208 Type of Action HORIZON-JU-RIA Topic HORIZON-JU-EUROHPC-2025-AI-01-IBA-01 Starting Date of Project 01.03.2025 Ending Date of Project 29.02.2028 Duration of the Project 36 months Website lumi-ai-factory.eu Work Package WP4 Task T4.3 Lead Authors Juha Hulkkonen, Tomáš Martinovič Contributors Pauliina Somerkoski, Emma Hintsala, Jan Martinovič, Juho Keränen Peer Reviewers Abdulrahman Azab, Heidi Laine, Katja Mankinen Version v1.0 Due Date 30.4.2025 Submission Date 30.4.2025 Dissemination level X PU: Public SEN: Sensitive – limited under the conditions of the Grant Agreement EU-RES. Classified Information: RESTREINT UE (Commission Decision 2005/444/EC) EU-CON. Classified Information: CONFIDENTIEL UE (Commission Decision 2005/444/EC) EU-SEC. Classified Information: SECRET UE (Commission Decision 2005/444/EC) D4.3 Inference Service documentation 4 Version History Revision Date Editors Comments 0.1. 28.3.2025 Juha Hulkkonen Initial draft 0.2. 17.4.2025 Juho Keränen, Jan Martinovič, Tomáš Martinovič, Pauliina Somerkoski Draft for review 0.3. 25.4.2025 Pauliina Somerkoski Edited based on review comments 1.0 28.4. Pauliina Somerkoski Final version. Glossary of Terms Item Description Aitta AI Inference service created in CSC LumiLingo Chatbot for Nordic languages created as demonstration application on top of Aitta LAIFS LUMI AI Factory Service Center LLM Large Language Model API Application Programming Interface REST API Representational State Transfer Application Programming Interface Web UI Web browser User Interface D4.3 Inference Service documentation 5 Executive Summary LUMI AI Factory's objectives are advancing the development of AI solutions, including offerings like inference service. We expect to see fast-paced growth in computational needs in this area. This document describes two inference platforms: Aitta and EXA4MIND. Aitta inference platform is developed by CSC, and EXA4MIND developed by the EXA4MIND project consortium (IT4I and METU). The inference services will be provided by LUMI AI Factory. Inference refers to the process of using an AI model to generate predictions, produce outputs or perform specific tasks based on new input data. Inference enables real-world applications of Large Language Models (LLMs) and other AI models, allowing users to interact with the model by providing prompts — text inputs that guide the model’s responses. Aitta inference platform is a service to host AI models on supercomputers. With Aitta, you can interact with LLMs efficiently using simple API (Application Programming Interface) calls or simple Web UI (Web browser User Interface). Users will be able to run their self-trained models, use curated portfolio of models. Easy integration to existing model repositories will be provided. EXA4MIND inference service is developed on efficient leveraging of HPC resources in a project initiative that is building a platform for extreme data (i.e. data that is at the upper or lower limits of expectations that should be accepted by the system), enabling advanced analytics on supercomputers. EXA4MIND inference service solution complements the Aitta inference service with the main difference in the task queue implementation. This opens the possibility for further Aitta extensions and optimization based on the incoming requirements. D4.3 Inference Service documentation 6 Table of Contents 1. Aitta in General ......................................................................................................... 7 1.1 Introduction to Aitta 7 1.2 Value 7 1.3 Future work 7 2. How to use Aitta ........................................................................................................ 9 2.1 Training Materials 9 3. Aitta Inference Service Technical Description ........................................................... 10 3.1 Aitta Inference Service Technical Description 10 3.1.1 Web user interface (FastAPI & Gradio) 11 3.1.2 Aitta Backend Service API 11 3.1.3 Task and Job Management (Celery & Redis) 12 3.1.4 Model Metadata (MongoB) 12 3.1.5 High-End Application Execution Middleware (HEAppE) 13 3.1.6 Authentication 13 4. EXA4MIND inference server ..................................................................................... 15 4.1 Introduction 15 4.2 Architecture 15 4.2.1 Web user interface (FastAPI) 15 4.2.2 High-End Application Execution Middleware (HEAppE) 15 4.2.3 Instances and deployed models DB 15 4.2.4 Task queue and scheduling 16 4.2.5 Logs 16 4.3 Future work 16 D4.3 Inference Service documentation 7 1. Aitta in General 1.1 Introduction to Aitta Aitta is a general purpose scalable service component to host AI models on a supercomputer. Aitta is offered for LUMI AI Factory Service Center customers to run their AI models in a reliable and powerful computing system safely in Finland, while harnessing powers of energy efficient supercomputers. Aitta offers a set of curated models via an easy-to-use modern web interface and API endpoints with its own Python library for programmatic usage. The fundamental development idea behind Aitta is that users can run arbitrary models with any type of data they need for their research and development purposes. Aitta saves users from having to set up the servers, or from buying cloud service capacity from third parties. Aitta offers a user-friendly experience, allowing users to focus on their primary tasks and objectives. AI inference platform Aitta is currently in development stage, originally created to complement CSC's growing AI service portfolio. With LUMI AI Factory the service is to be offered for a wider variety of international use cases and users. CSC offers a variety of computing services, including HPC capacity, scientific support, and training materials, for development of cutting edge AI models, including generative models. Aitta complements the services with providing suitable environments for running heavy AI models. At the moment Aitta is heavily focused on Large Language Models (LLMs) and our current set of available models includes Nordic language models Poro, Viking, GPT-SW3, FinGPT with modern chat interface. 1.2 Value Aitta can be seen as a part of commitment for efficient and sustainable use of resources by offering scalability to computing resources and making supercomputers more multi-purpose instruments. Computing resources used for training can be also used for inferencing the models. Running AI models may require substantial computational power, which Aitta uses efficiently by allocating resources on demand. Resources are allocated automatically and freed after a certain period of time. Aitta will offer various backends to ensure effective use of resources: cloud and container cloud for running small experimental models quickly, national supercomputers and LUMI and LUMI AI for largest and heaviest workloads. 1.3 Future work In the future, a feature to upload self-trained models will be developed. With help of other MLOps solutions planned to be developed in LUMI AI Factory, users can deploy their models into Aitta. Additionally, the curated portfolio of models, that now consists of LLM models, will be developed further. Access key management will be improved greatly to allow users to integrate Aitta into their D4.3 Inference Service documentation 8 own services. Currently Aitta development has been concentrating more on use via API, so Aitta has only a basic user interface, that will be developed further to be more user-friendly. D4.3 Inference Service documentation 9 2. How to use Aitta Aitta can be used via web user interface or REST API. REST API (Representational StateTransfer Application Programming Interface) is a common web service interface. Aitta has a simple web UI, see Aitta's webpage. The current Web UI is made mainly for testing and development purposes. The WebUI is currently also used to get the access key for programmatic use. Aitta makes models and inference tasks available via REST API. Aitta REST API is also OpenAI compatible. AITTA API reference documentation documents the API in detail describing endpoints, parameters, and responses. To be able to use this API, users need to log in to Aitta's webpage to create API keys for access. A Python client to be used with Aitta API is also developed. This client for programmatic use is published to PyPi repository With just a few lines of code, one can utilize this client to e.g. integrate and work with LLMs via the Aitta API. While its development is ongoing and uploading custom models is not yet available, Aitta already provides models for users to explore through the web UI and API endpoints using Python libraries aittaclient and openai. In the near future, it will also be possible to create embeddings (numerical representations of real-world objects) of the data using Aitta. 2.1 Training Materials Introduction to Aitta -training is online and available via the Noppe service. Noppe is CSC's service for learning and course purposes, offering web applications for working with data and programming. Noppe supports Jupyter and RStudio based applications. The training materials are currently located and will be maintained in a Github repository. In need of using instructions with private computer, the material sources can be downloaded from Github. D4.3 Inference Service documentation 16 Scaling is focused on the HPC job sides through adding jobs with HEAppE. For keeping track of the individual HPC jobs, there is a DB which stores two main things: the compute jobs created by HEAppE and related metadata, and the deployed models on these compute jobs. The second item is necessary so we know which models are already available, or whether the users would have to load a new model. 4.2.4 Task queue and scheduling Each compute job will have a process that acts as a task queue and load balancer using various strategies such as round-robin. There can be multiple AI models deployed in the worker processes (using either multiple single-GPU models or multi-GPU models deployed on one or more compute nodes), with communication between the workers and the main process handled by ZeroMQ. 4.2.5 Logs For each compute job there will be logs created containing information about user queries, responses, and AI model state. These are mainly used for developers to debug any errors in the server operation. 4.3 Future work There are multiple features that we would like to add in the future. Among these we are considering adding retrieval augmented generation (RAG) support, persistence of queues on the compute job to allow restarts in case of failures during the query execution, adding chat history to the user interface, and making the solution more general by allowing different models besides LLMs, such as image generation, audio generation, etc.