D5.3 Tools for integrating dynamic and real-time data streams

LUMI AI Factory Service Center Empowering Europe’s AI Ecosystem D5.3 Tools for integrating dynamic and real-time data streams 2 D5.3 Tools for integrating dynamic and real-time data streams D5.3 Tools for integrating dynamic and real-time data streams 3 Project Title LUMI AI Factory Service Center Project Acronym LUMI-AIF Project Number 101234208 Type of Action HORIZON-JU-RIA Topic HORIZON-JU-EUROHPC-2025-AI-01-IBA-01 Starting Date of Project 01.03.2025 Ending Date of Project 29.02.2028 Duration of the Project 36 months Website lumi-ai-factory.eu Work Package WP5 Data access and integration Task Task 5.3: Provision of tools for object storage Lead Authors Heidi Laine (CSC), Jarno Laitinen (CSC) Contributors Abdulrahman Azab (Sigma2) Peer Reviewers Pauliina Somerkoski (CSC), Dhanya Pushpadas (Sigma 2) Version 1.0 Due Date 31.08.2025 Submission Date 29.08.2025 Dissemination level X PU: Public SEN: Sensitive – limited under the conditions of the Grant Agreement EU-RES. Classified Information: RESTREINT UE (Commission Decision 2005/444/EC) EU-CON. Classified Information: CONFIDENTIEL UE (Commission Decision 2005/444/EC) EU-SEC. Classified Information: SECRET UE (Commission Decision 2005/444/EC) D5.3 Tools for integrating dynamic and real-time data streams 4 Version History Revision Date Editors Comments 0.1 11.08.2025 Heidi Laine First version for project internal peer review 0.5 25.08.2025 Heidi Laine Edits based on peer reviewer comments 0.9 29.08.2025 Heidi Laine Edits on case study descriptions, spellchecking, final formatting, addition of glossary 1.0 29.08.2025 Anna Luoma Final quality check performed by the PMO, sent to review. Declaration on the Use of AI Assistance This report has been prepared with the support of GPT-5-enabled Microsoft Copilot, which was used to assist in drafting text, checking language, and gathering background information. All content has been thoroughly reviewed, fact-checked, and edited by the authors to ensure accuracy and alignment with the objectives of the report. Glossary of Terms Item Description 5G Fifth-generation mobile network technology that provides low-latency, high-bandwidth wireless connectivity suitable for real-time IoT and edge scenarios. Active learning Machine-learning approach where a model requests labels for the most informative samples to improve quickly with limited annotation effort. AI/ML inference Running a trained model to generate predictions on new data, often with low-latency requirements for streaming use cases. Apache Airflow Workflow orchestration platform for scheduling and managing data pipelines (mainly batch). Apache Flink Distributed stream-processing engine for stateful computations over unbounded and bounded data streams with low latency. Apache Kafka Distributed event-streaming platform providing durable, scalable pub/sub messaging with topics and partitions. Apptainer (Singularity) Container runtime commonly used in HPC to package and run applications reproducibly without elevated privileges. D5.3 Tools for integrating dynamic and real-time data streams 5 Aspera / FASP High-speed file transfer protocol (UDP-based) optimized for long-distance, high-latency networks. Backpressure Mechanism in streaming systems to slow producers or intermediate stages when consumers cannot keep up, preventing data loss. Batch processing Processing data in discrete chunks at scheduled times, as opposed to continuous real-time processing. CAN bus Controller Area Network; vehicle bus standard used to enable communication between automotive components. cPouta / Rahti CSC cloud services: cPouta (IaaS) and Rahti (Kubernetes PaaS) used for containerized or VM-based workloads. CSC CSC – IT Center for Science (Finland). CSV Comma-Separated Values; simple text format for tabular data exchange. Data governance Policies and processes ensuring data quality, security, compliance, and lifecycle management. Data lake Centralized storage for raw, structured and unstructured data at any scale for analytics and ML. Data minimization GDPR principle: collect and process only the data necessary for a specific purpose. Data provenance Documented lineage of data, including origin, transformations, and ownership, supporting reproducibility and FAIR. Data warehouse Schema-oriented analytical storage optimized for structured, query-intensive workloads. DPIA Data Protection Impact Assessment; GDPR requirement to assess and mitigate privacy risks of a processing activity. Edge computing Processing performed close to data sources (e.g., devices, gateways) to reduce latency and bandwidth usage. Elastic scaling Ability of a system to scale resources up/down automatically based on load. Event-driven architecture Design where events trigger processing and downstream actions, common in streaming setups. EuroHPC European High-Performance Computing Joint Undertaking. FAIR Data principles: Findable, Accessible, Interoperable, Reusable. FASP See Aspera / FASP. FlexE Flexible Ethernet; technology to aggregate/slice Ethernet links for high-capacity, deterministic transport. gRPC High-performance, binary RPC framework using Protocol Buffers; supports streaming calls. Globus Research data transfer and sharing platform that provides secure, reliable, resumable high-throughput transfers. Grafana Visualization and dashboarding tool often paired with time-series databases and Prometheus metrics. D5.3 Tools for integrating dynamic and real-time data streams 6 Helm chart Kubernetes packaging format describing a set of resources for installing/configuring applications. HPC High-Performance Computing; large-scale compute systems optimized for parallel workloads. HTTP/HTTPS Web protocol (secure variant with TLS) used for APIs and data exchange. ICEYE Microsatellite (SAR) data provider; example in the document for near-real-time environmental monitoring. Idempotency Property where repeating an operation yields the same result; important for at-least-once processing. InfluxDB Time-series database optimized for high-ingest telemetry and metrics. IoT Internet of Things; network of connected devices and sensors producing telemetry data. JupyterHub Multi-user environment for running Jupyter notebooks on shared infrastructure. Kafka partition Subdivision of a Kafka topic enabling parallelism and scalability across brokers/consumers. Kubernetes Container orchestration platform for deploying, scaling, and managing containerized applications. LiDAR Light Detection and Ranging; sensor producing high-rate 3D point clouds, often streamed over UDP. LoRaWAN Low-power, long-range wireless protocol for IoT devices with modest data rates. Lustre High-performance parallel file system widely used in HPC environments. LUMI EuroHPC pre-exascale supercomputer hosted in Kajaani, Finland. LUMI AIF LUMI AI Factory; services enabling AI/HPC workflows, including data access and integration. LUMI-AI Forthcoming AI-optimized expansion to LUMI improving training and real-time processing capabilities. LUMI-G GPU partition of LUMI (AMD MI250X GPUs) for accelerated AI/ML and compute workloads. Metadata Descriptive information about data (schema, provenance, identifiers) enabling discovery and reuse. Micro-batching Processing incoming data in very small batches to balance latency and throughput. MinIO S3-compatible object storage software commonly used for on-premises data lakes. MQTT Lightweight pub/sub messaging protocol for constrained devices and unreliable networks. NB-IoT Narrowband IoT; cellular LPWAN technology for low-power, low-throughput devices. D5.3 Tools for integrating dynamic and real-time data streams 7 NIRD Norwegian research data infrastructure used in the SeaBee case for storage and processing. NIS2 EU directive establishing cybersecurity risk-management and reporting obligations for essential/important entities. OEE Overall Equipment Effectiveness; KPI combining availability, performance, and quality in manufacturing. ONNX Runtime Inference engine for models in ONNX format; enables portable, high-performance inference. ORC Optimized Row Columnar; columnar storage format for efficient analytics. Parquet Columnar storage format optimized for analytics and compression. PCAP Packet Capture; file format for recording network traffic (e.g., UDP LiDAR frames). Predictive maintenance Using analytics/ML on telemetry to predict component failure and schedule maintenance proactively. Prometheus Monitoring system and time-series database focused on metrics collection and alerting. Protocol translation Converting between protocols or payload formats (e.g., MQTT→Kafka, binary→JSON/Parquet). Pseudonymization Replacing identifiers with pseudonyms to reduce privacy risks while enabling analysis. RBAC Role-Based Access Control; authorization model assigning permissions to roles rather than individuals. Rclone Command-line tool for syncing and copying data across storage systems (e.g., S3, POSIX, cloud object stores). Real-time analytics Continuous analysis of streaming data with low latency to drive immediate decisions. ROCm AMD’s open compute platform for GPU acceleration (LUMI-G GPUs). RUL Remaining Useful Life; predicted time until a component fails or requires servicing. SeaBee Coastal monitoring initiative using drones; case study demonstrating hybrid streaming and HPC processing. Slurm HPC workload manager used on LUMI to schedule and run jobs. Spark Streaming Streaming library for Apache Spark enabling micro-batch and continuous processing. Stream processing Continuous computation over data streams to derive metrics, alerts, and features with minimal delay. Telegraf Agent for collecting, processing, and sending metrics/logs to databases such as InfluxDB. TensorRT NVIDIA inference optimizer/runtime (relevant for portability comparisons; not used on AMD GPUs in LUMI-G). D5.3 Tools for integrating dynamic and real-time data streams 8 Threshold-based alerting Generating alerts when metrics cross predefined values (e.g., temperature spikes). Time-series database Database optimized for time-indexed data (telemetry, metrics), e.g., InfluxDB, TimescaleDB, Prometheus. TLS/SSL Transport-layer encryption ensuring confidentiality and integrity for protocols like HTTPS, MQTT, Kafka. UDP User Datagram Protocol; low-latency, connectionless transport commonly used for high-rate sensor streams. WebSockets Full-duplex communication channel over a single TCP connection for bidirectional real-time messaging. Zigbee Low-power wireless mesh protocol for short-range IoT networks. D5.3 Tools for integrating dynamic and real-time data streams 9 Executive Summary This deliverable presents the initial results of Task 5.3, Provision of Interfaces for Dynamic and Real-Time Data, within the LUMI AI Factory Service Center. The report outlines the technical, architectural, and operational requirements for integrating dynamic and real-time data streams into the LUMI environment, and assesses the current capabilities of the infrastructure in relation to these needs. Dynamic data refers to continuously changing information generated by sensors, user interactions, or automated systems. Its integration into high-performance computing (HPC) environments is essential for enabling responsive, data-driven AI workflows. The report identifies key challenges such as lowlatency ingestion, protocol interoperability, edge computing, and secure data handling. The first version of the LUMI AI Factory tools includes components for high-speed data transfer, stream processing, edge integration, and observability. These tools are designed to support heterogeneous use cases, including environmental monitoring, predictive maintenance, and smart infrastructure. A comparative analysis highlights areas where LUMI’s current capabilities align with requirements and where further development is needed. Case studies, such as the SeaBee coastal monitoring pilot, demonstrate practical implementation scenarios and inform future development priorities. The report concludes with a roadmap for enhancing real-time data support, including ingestion middleware, protocol translation services, and AI/ML inference capabilities. It is likely that streaming data use cases are so heterogeneous that there cannot be one solution architecture. The end-users need to adapt the solution of their use cases. LUMI AI Factory can provide open source software component in containers and provide guidance for them. D5.3 Tools for integrating dynamic and real-time data streams 16 • Automated alerts are sent to maintenance teams when anomalies are detected. • Predictive insights from HPC simulations are integrated into the dashboard for proactive decision-making. Key Benefits Immediate visibility into production performance. • Reduced downtime through early anomaly detection. • Optimized resource allocation via predictive analytics. • Secure and compliant data handling across the pipeline. 2.3 Predictive maintenance Scenario: A large energy company operates wind farms across Europe. Each turbine is equipped with IoT sensors that monitor vibration, temperature, and power output. The goal is to predict component failures before they occur, reducing downtime and maintenance costs. Step 1: Continuous Data Collection Sensors on turbine components (gearboxes, bearings, blades) generate telemetry data every second. • Edge devices preprocess the data, compressing and normalizing it into JSON format. • Data is transmitted securely via MQTT to the ingestion layer. Step 2: Ingestion and Buffering An Apache Kafka cluster receives the data streams and partitions them by turbine ID. • Kafka ensures durability and provides a buffer to handle network fluctuations or HPC maintenance windows. Step 3: Real-Time Anomaly Detection • A Flink-based stream processing job consumes data from Kafka. • The job applies: • Threshold-based alerts for critical parameters (e.g., temperature spikes). • Lightweight ML models for early anomaly detection at the edge or ingestion layer. Step 4: HPC Integration for Predictive Modeling • Aggregated and enriched data is streamed to the LUMI HPC environment. • HPC nodes run deep learning models trained on historical failure data to predict Remaining Useful Life (RUL) of components. • Models are updated periodically using the latest streaming data for improved accuracy. Step 5: Visualization and • Predictions and alerts are displayed on a Grafana dashboard for operations teams. D5.3 Tools for integrating dynamic and real-time data streams 17 Maintenance Scheduling • The system automatically generates maintenance tickets for turbines predicted to fail within a defined time window. • Integration with ERP systems ensures spare parts and technicians are scheduled proactively. Key Benefits • Reduced unplanned downtime by predicting failures days or weeks in advance. • Optimized maintenance costs through condition-based servicing. • Improved safety by preventing catastrophic failures. • Scalable solution capable of handling thousands of turbines and millions of data points per day. 2.4 Smart infrastructure and environments Scenario: A metropolitan city deploys a network of IoT sensors and connected devices to monitor traffic flow, air quality, and energy consumption in real time. The objective is to optimize traffic management, reduce emissions, and improve urban living conditions through data-driven decisions. Step 1: Data Collection from Distributed Sources • IoT sensors embedded in traffic lights, roadways, and public transport vehicles collect data on: • Vehicle counts and speeds • Air quality metrics (CO₂, NO₂, PM2.5) • Energy usage in smart streetlights • Edge gateways aggregate and normalize the data into JSON format for transmission. Step 2: Ingestion and Routing • Data streams are transmitted via MQTT and HTTP/HTTPS to the city’s central ingestion layer. • An Apache Kafka cluster partitions the data by category (traffic, air quality, energy) for efficient processing. Step 3: Real-Time Analytics • A Flink-based stream processing pipeline analyzes: • Traffic congestion patterns in real time. • Air quality anomalies and pollution hotspots. • Energy consumption trends for adaptive lighting control. • Event-driven triggers adjust traffic signals dynamically and activate low-emission zones when pollution exceeds thresholds. Step 4: HPC Integration for Predictive Modeling • Aggregated data is streamed to the LUMI HPC environment for: • Traffic flow simulations to predict congestion under different scenarios. • Air quality forecasting using AI-driven models. • Energy optimization models for smart grid integration. D5.3 Tools for integrating dynamic and real-time data streams 18 Step 5: Visualization and Citizen Engagement • Real-time dashboards display traffic and air quality data for city operators. • Citizens access live updates via a mobile app, including alternative route suggestions and pollution alerts. • Predictive insights inform long-term urban planning decisions. Key Benefits • Reduced congestion through adaptive traffic control. • Improved air quality via proactive interventions. • Energy savings through smart lighting and grid optimization. • Enhanced citizen experience with real-time information and safer environments. 3. Requirements for dynamic and real time data streams Developing support for dynamic and real-time data streams in the LUMI AI Factory environment requires a careful analysis of multiple types of requirements. These include technical requirements, such as the ability to ingest and process high-velocity data streams with low latency, ensure data integrity, and scale to handle large volumes. Architectural requirements must also be considered, including event-driven processing, buffering mechanisms, and integration with HPC workflows. In addition, operational requirements, such as resilience during maintenance, interoperability with existing tools, and support for diverse protocols, are essential for reliability. Finally, governance and compliance requirements, including security, access control, and adherence to FAIR principles, must be addressed to ensure trust and sustainability. This chapter outlines these requirements and explains why they are critical for enabling real-time and dynamic data capabilities in research and innovation workflows. 3.1 Technical, architectural, and operational requirements Streaming data from IoT devices to a data analysis environment involves several key requirements across hardware, software, and network layers. Naturally the use case defines the needed functions. LUMI AIF environment streaming data capabilities will be continuously developed based on experiences from pilot case studies, first two of which are presented in chapter 7. Case Studies in this report. In this chapter we present on possible classification of the essential components. The suitability of the classification will be examined in the afore mentioned case studies. Device-Level Requirements • Sensors and Actuators: Devices must be equipped with sensors to collect data (e.g., temperature, motion, humidity). • Connectivity Modules: Support for communication protocols like Wi-Fi, Bluetooth, Zigbee, LoRaWAN, NB-IoT, or 5G. Communication and Networking D5.3 Tools for integrating dynamic and real-time data streams 19 • Low-Latency, High-Bandwidth Networks: Especially important for real-time applications (e.g., 5G or Ethernet). • Reliable Protocols: • MQTT: Lightweight, ideal for constrained devices. • CoAP: Optimized for low-power devices. • HTTP/HTTPS: Common for RESTful APIs. • WebSockets: For bi-directional, real-time communication. • Security: TLS/SSL encryption, VPNs, and secure authentication (e.g., OAuth2, token-based access). Data Ingestion and Streaming Platforms • Message Brokers: Tools like Apache Kafka, RabbitMQ, or MQTT brokers to buffer and route data. • Log collection and shipping tools (e.g., Logstash and , Filebeat) • Streaming Workflow Engines for computation and analytics (e.g. Apache Airflow, Temporal, Argo ) • Stream Processing Engines for orchestration. • Scalability: Ability to handle high-throughput and bursty data loads. Data Storage and Management • Time-Series Databases: InfluxDB, TimescaleDB, or Prometheus for storing sensor data. • Data Lakes/Warehouses: For long-term storage and batch analytics (e.g., AWS S3, Azure Data Lake, Google BigQuery). • Metadata Management: Tagging and indexing for efficient querying. Data Analysis Environment • Real-time stream processing engine tools like Apache Flink • Real-Time Dashboards: Tools like Grafana, Kibana, or Power BI for visualization. • Machine Learning Pipelines: Integration with platforms like TensorFlow, PyTorch, or cloud ML services. • Alerting and Automation: Triggering actions based on thresholds or anomalies. For this could be configured for example Apache Spark Streaming or and Apache Flink 1 . Some of the tools might have so called Helm Chart to launch and configure the setup in the user's name space. AI algorithms can be called also from tools like Flink, and Kafka (via API). Some software may be available in the cluster to be consumed by all users, which make it easier for the users. Also, some software may require special privileges to install those, which is not possible via Helm Chart based method. In the worst case, not all software can be installed on the Kubernetes. 1 https://doi.org/10.1109/ISDFS58141.2023.10131800 D5.3 Tools for integrating dynamic and real-time data streams 20 For log management there are specialized tools. One needs to be aware of the possible limitations of the platform such as the containers cannot use root user account. The software can be made readily available in form of containers and helm charts. 3.2 Integration capabilities Integrating dynamic and real-time data streams into a High-Performance Computing (HPC) environment such as LUMI requires specialized capabilities that bridge the gap between traditional batch-oriented HPC workflows and the continuous, event-driven nature of streaming data. These capabilities ensure that data flows seamlessly from distributed sources to HPC resources for processing, analytics, and AI workloads. 1. Multi-Protocol Data Ingestion • Requirement: Ability to ingest data from heterogeneous sources using protocols such as MQTT, Kafka, HTTP/HTTPS, and WebSockets. • Rationale: IoT devices, APIs, and external systems often use different communication standards. A flexible ingestion layer ensures interoperability and minimizes integration overhead. 2. Real-Time Buffering and Backpressure Management • Requirement: Middleware that can buffer incoming streams and manage backpressure to prevent data loss during HPC job scheduling or maintenance windows. • Rationale: HPC systems are optimized for batch jobs, which may not align with continuous data arrival. Buffering decouples producers from consumers, ensuring reliability. 3. Stream Processing Framework Integration • Requirement: Support for frameworks like Apache Flink or Spark Streaming, deployed in containerized environments or via orchestration platforms (e.g., Kubernetes). • Rationale: Enables real-time analytics, event-driven triggers, and pre-processing before data reaches HPC compute nodes. 4. Hybrid Edge-HPC Integration • Requirement: Edge agents capable of filtering, aggregating, and translating data before transmission to the HPC center. • Rationale: Reduces bandwidth usage, minimizes latency, and ensures only relevant data is sent to the HPC system. 5. Protocol and Format Translation Services • Requirement: Services to convert IoT-native protocols and formats (e.g., MQTT, CoAP, binary payloads) into HPC-compatible formats (e.g., JSON, Parquet). • Rationale: Ensures seamless integration with HPC storage and analytics pipelines. 6. Secure and Compliant Data Transfer D5.3 Tools for integrating dynamic and real-time data streams 21 • Requirement: End-to-end encryption (TLS/SSL), quantum-safe transport, and role-based access control. • Rationale: Protects sensitive data and ensures compliance with regulations such as GDPR. 7. Observability and Monitoring • Requirement: Real-time dashboards and alerting systems (e.g., Grafana, Prometheus) for monitoring data flows, latency, and system health. • Rationale: Provides visibility into streaming pipelines and enables proactive issue resolution. 8. Scalable Storage and Retrieval • Requirement: Integration with high-performance parallel file systems (e.g., Lustre) and timeseries databases for short-term analytics. • Rationale: Supports both real-time and historical analysis without compromising performance. 9. AI/ML Pipeline Integration • Requirement: Ability to feed streaming data into AI inference services or online learning pipelines. • Rationale: Enables adaptive AI models and real-time decision-making. 3.3. Security and compliance General technical security requirements • Secure Data Transmission • Use TLS/SSL encryption for all streaming protocols (MQTT, HTTP, Kafka). • Implement quantum-safe encryption for long-term confidentiality (as tested in LUMI’s optical link). • Authentication and Authorization • Enforce strong authentication (OAuth2, token-based access). • Apply role-based access control (RBAC) for HPC resources and data pipelines. • Network Segmentation and Firewalls • Isolate ingestion services from core HPC compute nodes. • Use firewalls and VPNs for external connections. • Data Governance and Compliance Layer • Real-time data validation and anonymization for sensitive information. • Implement policy-driven access control aligned with GDPR and other regulations. • Monitoring and Audit Logging • Continuous monitoring of data flows using tools like Prometheus and Grafana. • Maintain immutable audit logs for traceability and compliance reporting. • Resilience and Fault Tolerance • Buffering and backpressure management (e.g., Kafka) to prevent data loss during outages. D5.3 Tools for integrating dynamic and real-time data streams 22 • Redundant ingestion nodes for high availability. • Secure Container Orchestration • Deploy ingestion and processing services in Kubernetes namespaces with strict security policies. • Avoid root privileges in containers; enforce Pod Security Standards. Security and Compliance Measures in EU Context Measure Description Relevant EU Regulation TLS/SSL & Quantum-Safe Encryption Secure data transmission for streaming protocols GDPR Art. 32 (Security of Processing), NIS2 (Cybersecurity) Strong Authentication & MFA OAuth2, certificates, multifactor authentication for access control GDPR Art. 32, NIS2 Role-Based Access Control (RBAC) Restrict access to HPC resources and data pipelines based on roles GDPR Art. 25 (Data Protection by Design), NIS2 Data Minimization & Anonymization Remove or pseudonymize personal data before ingestion GDPR Art. 5 (Data Minimization), Art. 32 Network Segmentation & Firewalls Isolate ingestion services from HPC compute nodes NIS2 (Network and Information Security) Immutable Audit Logs Maintain logs for accountability and incident reporting GDPR Art. 30 (Records of Processing), NIS2 Incident Response Plan Procedures for breach notification within 24–72 hours GDPR Art. 33 (Breach Notification), NIS2 Data Protection Impact Assessment Assess risks for streaming projects GDPR Art. 35 (DPIA) Vendor Risk Management Security checks and contractual clauses for third-party data providers GDPR Art. 28 (Processor Obligations), NIS2 Training & Awareness GDPR and cybersecurity training for staff GDPR Art. 39 (DPO Tasks), NIS2 Compliance Audits & Pen Testing Regular audits and vulnerability assessments GDPR Art. 32, NIS2 Organizational Measures • Data Protection Policies • Define clear policies for data classification, retention, and deletion. • Ensure compliance with GDPR and sector-specific regulations. • Access Management Procedures • Regularly review and update user permissions. • Implement least privilege principle for all roles. • Incident Response Plan • Establish a documented process for security breaches or data leaks. D5.3 Tools for integrating dynamic and real-time data streams 23 • Conduct regular incident response drills. • Vendor and Third-Party Risk Management • Assess security posture of external data providers. • Include data protection clauses in contracts. • Training and Awareness • Provide security training for developers, data engineers, and HPC operators. • Promote awareness of phishing and social engineering risks. • Compliance Audits • Schedule regular internal and external audits. • Maintain documentation for regulatory inspections. 4. First version of LUMI AIF tools for dynamic and real time data streams As of mid-2025, the LUMI supercomputer is being prepared for advanced real-time and dynamic data streaming capabilities, particularly in support of AI and high-performance computing (HPC) workloads. Here are the key developments: • High-Speed Data Transfer Infrastructure: A major milestone was achieved in June 2025 when CSC, SURF, and Nokia successfully tested a 1.2 terabit per second (Tbit/s) quantum-safe fiberoptic connection between Amsterdam and Kajaani, Finland—where LUMI is hosted. This infrastructure is designed to support massive, continuous data flows ("elephant flows") and is crucial for real-time data streaming in HPC and AI applications • Flexible Ethernet and Optical Networking: The test utilized Flexible Ethernet (FlexE) and high-capacity optical transport technologies, enabling LUMI to handle large-scale, real-time data transfers across long distances. This is essential for applications like training large AI models or streaming sensor data from IoT devices • Real-Time Environmental Monitoring: LUMI is already being used for real-time analysis of radar data from ICEYE’s microsatellite system. This allows for near-instantaneous generation of terrain images to detect environmental events like floods or fires, regardless of weather conditions • Support for AI Factories and LUMI-AI: The infrastructure upgrades are also in preparation for LUMI-AI, a next-generation AI-optimized supercomputer. This system will further enhance realtime data processing capabilities, supporting AI factories and other data-intensive applications LUMI Capabilities vs. Real-Time & Dynamic Data Streaming Requirements Category General Requirements for Real-Time & Dynamic Data LUMI Capabilities (Based on Docs & Public Info) Data Ingestion Continuous, high-throughput ingestion from distributed sources (e.g., IoT, sensors, APIs) Supports high-speed data transfer via 1.2 Tbit/s optical link; integration with satellite data (e.g., ICEYE) D5.3 Tools for integrating dynamic and real-time data streams 24 Latency Low-latency processing and response times for real-time decision-making Designed for high-performance computing; latency depends on workload and architecture (batch-oriented by default) Streaming Protocols Support for MQTT, Kafka, WebSockets, HTTP/2, etc. Not natively documented; integration likely requires custom middleware or edge preprocessing Edge Integration Ability to process data at the edge before sending to central systems No native edge computing layer; relies on external systems to preprocess before ingestion Scalability Elastic scaling to handle bursty or growing data streams Extremely scalable compute and storage infrastructure (toptier HPC system) Real-Time Analytics Stream processing engines (e.g., Flink, Spark Streaming) for on-the-fly analytics Not explicitly documented; users can deploy custom analytics pipelines using available compute nodes Data Storage Time-series databases, data lakes, and fast-access storage for streaming data High-performance parallel file systems (e.g., Lustre); suitable for large-scale data storage and retrieval Security & Compliance End-to-end encryption, access control, GDPR/industry compliance Quantum-safe data transfer tested; secure infrastructure managed by CSC and EuroHPC AI/ML Integration Real-time model inference and online learning capabilities Supports large-scale AI training; LUMI-AI expansion will enhance real-time AI capabilities Monitoring & Observability Real-time dashboards, logs, and metrics for data flow visibility Monitoring tools available for HPC jobs; real-time observability for streaming not natively documented LUMI offers exceptional compute and data transfer capabilities, making it well-suited for high-volume, high-performance workloads. However, for real-time and dynamic data streaming, it typically requires custom integration layers or external tools to meet the responsiveness and protocol flexibility expected in IoT and streaming environments. 5. Architecture of the Tools Package 5.1 Core components and design principles The first version of the LUMI AI Factory (LUMI AIF) tools for dynamic and real-time data streams builds upon the high-performance capabilities of the LUMI supercomputer while addressing the unique D5.3 Tools for integrating dynamic and real-time data streams 25 requirements of streaming and IoT-driven workloads. The design emphasizes modularity, scalability, and interoperability to accommodate diverse use cases and evolving technologies. Core Components High-Speed Data Transfer Layer • Utilizes the recently tested 1.2 Tbit/s quantum-safe optical link between Kajaani and Amsterdam to support massive, continuous data flows. • Incorporates Flexible Ethernet (FlexE) and advanced optical networking for lowlatency, high-throughput connectivity. Data Ingestion and Buffering Services • Middleware layer for ingesting real-time streams from IoT devices, APIs, and external systems. • Supports integration with message brokers (e.g., Apache Kafka, MQTT) and provides buffering to decouple data producers from consumers. Stream Processing Framework • Containerized deployment of Apache Flink or Spark Streaming for real-time analytics. • Enables event-driven processing and integration with AI/ML pipelines. Edge Integration Layer • Lightweight agents for preprocessing and filtering data at the edge, reducing bandwidth and latency. • Supports protocol translation (e.g., MQTT → JSON/Parquet) before forwarding to LUMI. Storage and Persistence • High-performance parallel file systems (e.g., Lustre) for large-scale storage. • Optional integration with time-series databases for short-term retention and analytics. Monitoring and Observability • Dashboards and alerting systems (e.g., Grafana, Prometheus) for real-time visibility into data flows and system health. Security and Compliance Layer • End-to-end encryption, quantum-safe data transfer, and role-based access control. • Compliance with GDPR and other relevant regulations. Design Principles • Modularity: Each component is loosely coupled, enabling independent scaling and upgrades. D5.3 Tools for integrating dynamic and real-time data streams 32 Context University of Oulu ITEE Facility 3 has Toyota RAV4 vehicles for vehicular research. The vehicle is equipped with several sensors such as lidar, two thermal cameras and stereocamera. They produce data at a high rate, at least 10gb/min, which amounts to hundreds if not terabytes of data per driving session. The data will be sent to the cloud to the supercomputer, where it will be processed and machine learning algorithms will be run on it. In the future, the plan is to have the processed data and possibly commands passed back onto the vehicle to enable e.g. self-driving car functionalities. Ideal case would be a full digital twin of the car, which could be run automatically. Handling large amounts of data with minimal delay in real-time is crucial for this use case. Current Challenge Poor connection, high data rate and the need for minimizing delay are all current challenges. The current setup is based on a MQTT connection to Rahti, but MQTT connection does not suit well for sending heavy video material through it. Also, our future work includes setting up ML algorithms run on CSC services from the data gotten from the vehicle. Currenetly we have not tried to run any ML algorithms on e.g. Mahti from the data from the Rahti and vehicle in real time, as Mahti has the SLURM batch jobs and queuing. Data Management Requirements for Streaming • Real-time processing • Streaming large amount of data from different sensor types • Streaming also video à need for e.g. UDP connection (others possible as well, but this seems to be a suitable candidate for video streaming) • Relaying data back to the car • Minimal delay / near-real-time functioning of the whole setup Future Vision In the future ideally the car could relay all the sensor data it receives to the CSC cloud and supercomputers, the algorithms could be run on CSC services, and then they would be passed back onto the vehicles. Ideally this would enable us to have a real-time digital twin of the vehicle, that could be either automatically driven or driven remotely. Pilot Study Design: Smart car–LUMI AIF Integration Objectives • Replace Rahti/cPouta backend with LUMI for preprocessing and HPC workloads. • Add LiDAR (UDP→PCAP) alongside CAN and prove end-to-end ingest with micro-batching. Scope • Use existing Jetson CAN-bus integration and extend it with LiDAR UDP streaming. 3 https://www.oulu.fi/en/university/faculties-and-units/faculty-information-technology-and-electrical-engineering D5.3 Tools for integrating dynamic and real-time data streams 33 • Migrate the current MQTT → Telegraf → InfluxDB → Grafana pipeline to LUMI. • Integrate the graphical management UI into the Jetson, enabling local control over: • Enabled sensors. • Target InfluxDB buckets. • UDP/MQTT endpoint configuration. • Demonstrate end-to-end real-time multi-sensor data streaming with monitoring dashboards in Grafana. Evaluation metrics • Performance: Measured latency and throughput for CAN + LiDAR streams. • Scalability: How the system handles additional sensors or higher data rates. • Reliability: LonPacket loss, synchronization accuracy, and system uptime. Technical setup Component LUMI AIF Adaptation Compute LUMI VMs running Docker Compose for InfluxDB, Telegraf, Grafana, Mosquitto for example LiDAR UDP-based PCAP streaming at high throughput Jetson environment Python-based CAN + LiDAR collection, REST API backend + graphical UI Data transfer Globus for bulk (resumable, scheduled); Aspera/FASP (UDP) for accelerated near‑real‑time hops when WAN conditions require it Networking MQTT for CAN data; UDP for LiDAR Data workflows to test • Baseline CAN streaming: Verify existing MQTT → InfluxDB pipeline works after migration. • LiDAR UDP integration: Benchmark packet ingestion rates and packet loss tolerance. • Combined ingestion: Stress test simultaneous CAN + LiDAR data flow into InfluxDB. Data management requirements • Real-time ingestion: Handle simultaneous CAN, LiDAR, and optional video streams. • High-throughput UDP handling: Optimize network buffer sizes and Telegraf configuration. • Dynamic data routing: Enable the Jetson UI to reconfigure targets without backend redeployment. • Visualization and monitoring: Grafana dashboards must expose latency, throughput, and packet statistics. • Security: o Token-based access for InfluxDB and Grafana. o Security groups for UDP, MQTT, and HTTPS. D5.3 Tools for integrating dynamic and real-time data streams 34 o Optional VPN tunneling for sensor data. 7. Future developments This chapter outlines the key development opportunities identified during the current phase of work. It does not represent a fixed implementation plan but rather highlights areas where enhancements could significantly improve functionality and performance. A dynamic development roadmap, reflecting priorities and timelines, will be maintained and regularly updated on the LUMI AI Factory website (to be published during September 2025). This ensures transparency and provides stakeholders with an up-todate view of planned and ongoing improvements. To support real-time and dynamic data integration, organizations are adopting technologies such as: • Stream processing engines (e.g., Apache Kafka, Flink, Spark Streaming) • Event-driven architectures that trigger actions based on data changes • Edge computing to process data closer to its source • Cloud-native platforms that scale dynamically with data volume These technologies allow for seamless integration of data from IoT devices, mobile apps, enterprise systems, and external APIs into a cohesive analytical environment. Based on the comparison between LUMI’s current capabilities and the general requirements for dynamic and real-time data streaming, the following key service development needs emerge: 1. Real-Time Data Ingestion Layer • Need: A middleware or gateway service to ingest and buffer real-time data streams from IoT devices or external APIs. • Why: LUMI does not natively support streaming protocols like MQTT or Kafka. • Solution Direction: Develop or integrate a scalable ingestion layer that can preprocess and route data to LUMI’s compute nodes. 2. Stream Processing Framework Integration • Need: Support for real-time analytics engines such as Apache Flink, Spark Streaming, or cloudnative equivalents. • Why: LUMI is optimized for batch HPC workloads; real-time stream processing is not documented. • Solution Direction: Containerized or orchestrated deployment of stream processing frameworks on LUMI or in hybrid cloud-edge setups. 3. Edge Computing Enablement • Need: Preprocessing and filtering data at the edge before transmission to LUMI. • Why: Reduces latency, bandwidth usage, and offloads non-critical processing. • Solution Direction: Develop edge agents or microservices that handle local computation and stream only relevant data to LUMI. D5.3 Tools for integrating dynamic and real-time data streams 35 4. Protocol and Format Translation Services • Need: Services to translate IoT-native protocols (e.g., MQTT, CoAP) into formats LUMI can process (e.g., JSON, CSV, Parquet). • Why: LUMI lacks native support for lightweight IoT protocols. • Solution Direction: Build protocol adapters or use existing open-source bridges. 5. Real-Time Monitoring and Observability Tools • Need: Dashboards and alerting systems for monitoring data flows and system health in real time. • Why: HPC systems like LUMI are not typically equipped with real-time observability tools. • Solution Direction: Integrate tools like Grafana, Prometheus, or custom dashboards tailored to streaming workloads. 6. AI/ML Inference Services • Need: Real-time model inference capabilities for streaming data. • Why: LUMI is optimized for training large models, but real-time inference is not documented. • Solution Direction: Deploy lightweight inference services (e.g., ONNX Runtime, TensorRT) on edge or cloud nodes, with LUMI used for model training and updates. 7. Data Governance and Compliance Layer • Need: Real-time data validation, anonymization, and access control mechanisms. • Why: Streaming data often includes sensitive or regulated information. • Solution Direction: Implement policy-driven data governance services that operate inline with data streams. 8. Conclusion Real-time and dynamic data integration is a critical enabler for advanced AI and HPC applications. This report has identified the core requirements for supporting such capabilities within the LUMI AI Factory environment and presented the first version of tools designed to meet these needs. While LUMI offers exceptional compute and data transfer infrastructure, additional components are required to fully support streaming data workflows. These include ingestion layers, stream processing frameworks, edge computing agents, and observability tools. Security and compliance measures must also be integrated to ensure trustworthy and sustainable data handling. The case studies presented validate the relevance of these requirements and demonstrate the potential for scalable, responsive AI workflows. Future development will focus on modular, user-centric solutions that enable elastic scaling, protocol interoperability, and real-time decision support. By addressing these priorities, the LUMI AI Factory will strengthen its role as a European hub for AI innovation, supporting research and development across domains that rely on dynamic data—from environmental science to industrial automation.