ELASTIC - D2.1: Lightweight and Robust Orchestrating Mechanisms - Initial Version

Horizon Europe Framework Programme HORIZON JU Research and Innovation Action Reliable Services and Smart Security Efficient, portabLe And Secure orchesTration for reliable servICes D2.1: Lightweight and Robust Orchestrating Mechanisms - Initial Version Abstract: This deliverable describes the first version of the design of lightweight, efficient, and secure orchestration mechanisms that leverage Wasm and eBPF technologies. It explains the state of the art in WebAssembly Orchestration, including integration in Kubernetes and WebAssembly-native orchestration platforms such as wasmCloud. It details the improved Wasm orchestrator “Propeller”, providing an alternative for environments where Kubernetesbased solutions do not fit. For example, it uses the WebAssembly Micro runtime (WAMR) in order to support truly tiny devices with real-time operating systems such as Zephyr. It strikes a fine balance between compatibility with the existing cloud-native ecosystem and support for low-resource edge devices by reusing some key cloud-native components and replacing some components with those more suitable for IoT devices. For example, it supports regular OCI registries but swaps out HTTP for MQTT communication. It also discusses our innovations in AI/ML workload orchestration and our innovations for bringing WebAssembly to the cyberphysical edge by creating high-performance standardised interfaces to connect WebAssembly applications to external hardware using protocols such as I2C and USB. Finally, it brings these aspects together in a single proposed architecture and reflects on the current state and the next steps for this work. Contractual Date of Delivery 28/02/2025 Actual Date of Delivery 28/03/2025 Deliverable Security Class Public Editor Merlijn Sebrechts (IMEC) Contributors IMEC, AMA, TUC Internal Reviewers Davide Miola (POLITO) Vladimir Urošević (ZEN) ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 2 - February 28, 2025 The ELASTIC Consortium Part. No. Participant organisation name Participant Short Name Role Country 1 POLYTECHNEIO KRITIS TUC Coordinator EL 2 ERICSSON AB ERS Principal Contractor SE 3 OY L M ERICSSON AB ERF Principal Contractor FI 4 TELEFONICA INNOVACION DIGITAL SL TID Principal Contractor ES 5 THALES SIX GTS FRANCE SAS THS Principal Contractor FR 6 THALES DIS FRANCE SAS THD Principal Contractor FR 7 INTERUNIVERSITAIR MICROELECTRONICA CENTRUM IMEC Principal Contractor BE 8 ULTRAVIOLET CONSULT DOO UVC Principal Contractor RS 9 AALTO KORKEAKOULUSAATIO SR AAL Principal Contractor FI 10 LUNDS UNIVERSITET LUN Principal Contractor SE 11 ABSTRACT MACHINES SAS AMA Principal Contractor FR 12 PRIVREDNO DRUSTVO ZENTRIX LAB DRUSTVO SA OGRANICENOM ODGOVORNOSCU PANCEVO ZEN Principal Contractor RS 13 POLITECNICO DI TORINO POLITO Principal Contractor IT ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 3 - February 28, 2025 Document Revisions & Quality Assurance Internal Reviewers 1. Davide Miola, POLITO 2. Vladimir Urošević, ZEN Revisions Version Date By Overview 3.0 27/03/2025 TUC Final approval and submission 2.0 26/03/2025 Merlijn Sebrechts Final version 1.1 21/03/2025 Dhouha Ayed Comments from the STM 1.0 12/03/2025 POLITO, ZEN Approval from the IRs 0.9 06/03/2025 Merlijn Sebrechts 3rd draft 0.8 05/03/2025 TUC Comments from the PC, Quality check & fix formatting issues 0.7 03/03/2025 Merlijn Sebrechts, Drasko Draskovic 2nd draft 0.6 20/02/2025 POLITO, ZEN Comments on the 1st draft 0.5 12/02/2025 ALL 1st complete draft 0.4 25/11/2024 ALL Initial draft 0.3 04/11/2024 ALL Comments on the TOC 0.2 10/10/2024 Drasko Draskovic Create TOC 0.1 30/07/2024 Merlijn Sebrechts Create document and fill in template Disclaimer The work described in this document has been conducted within the ELASTIC project. This project has received funding from the Smart Networks and Services Joint Undertaking (SNS JU) under the European Union’s Horizon Europe research and innovation programme under Grant Agreement No 101139067. This document does not reflect the opinion of the European Union, and the European Union is not responsible for any use that might be made of the information contained therein. This document contains information that is proprietary to the ELASTIC Consortium partners. Neither this document nor the information contained herein shall be used, duplicated, or communicated by any means to any third party, in whole or in parts, except with prior written consent of the ELASTIC Consortium. ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 4 - February 28, 2025 Table of Contents List of Tables............................................................................................................................. 6 List of Figures ........................................................................................................................... 7 List of Abbreviations ................................................................................................................ 8 Executive Summary ............................................................................................................... 10 1 Introduction .................................................................................................................... 11 1.1 Purpose and Scope of the Document ................................................................................. 11 1.2 Relation to Work Packages, Deliverables and Activities ................................................ 11 1.3 Contribution to WP2 and Project Objectives .................................................................. 12 1.4 Structure of the Document ................................................................................................. 12 2 Objectives ........................................................................................................................ 13 3 Analysis of Existing Wasm Orchestrators ................................................................... 15 3.1 Orchestrating lightweight FaaS workloads ...................................................................... 15 3.1.1 Spinkube ......................................................................................................................................... 15 3.1.2 wasmCloud ..................................................................................................................................... 15 3.1.3 Erlang BEAM-like lightweight scheduling .................................................................................... 16 3.2 Lightweight and robust supervision with WebAssembly ............................................... 16 3.3 WebAssembly for cyber-physical devices ......................................................................... 17 3.3.1 WebAssembly Micro Runtime (WAMR) ...................................................................................... 17 3.3.2 Wasmtime ....................................................................................................................................... 18 3.3.3 Lack of WASI support for physical hardware interfaces ............................................................... 18 4 Architecture and Components ...................................................................................... 19 5 Architecture of Improved Wasm Orchestrator........................................................... 22 5.1 Features ............................................................................................................................... 22 5.2 How It Works ...................................................................................................................... 23 5.3 System Architecture ........................................................................................................... 23 5.4 Components ......................................................................................................................... 24 5.4.1 CLI .................................................................................................................................................. 24 5.4.2 Manager .......................................................................................................................................... 25 5.4.3 Proplet............................................................................................................................................. 25 5.4.4 Embedded Proplet .......................................................................................................................... 26 5.4.5 Proxy............................................................................................................................................... 36 5.4.6 SuperMQ ........................................................................................................................................ 36 5.4.7 Communication .............................................................................................................................. 37 5.5 OCI Registry for Wasm Workloads ................................................................................. 39 5.5.1 Wasm to OCI .................................................................................................................................. 40 5.5.2 Proxy Service.................................................................................................................................. 42 5.5.3 Proxy Service Architecture ............................................................................................................. 43 5.5.4 Streaming System ........................................................................................................................... 44 5.5.5 Running The Proxy......................................................................................................................... 44 5.5.6 Chunk Management ........................................................................................................................ 46 5.5.7 Performance Features ..................................................................................................................... 46 ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 5 - February 28, 2025 6 AI/ML Workload Orchestration................................................................................... 47 6.1 AI/ML Libraries and Wasm Runtime Support ............................................................... 47 6.2 Training and Inference of Wasm AI Workloads ............................................................. 47 6.3 WASI Support for AI Acceleration .................................................................................. 48 6.4 Execution of Wasm AI/ML Algorithms using Propeller ................................................ 49 7 Cyber-physical Wasm Workload Orchestration ......................................................... 51 7.1 Improving Wasm Runtime Support for Edge Computing ............................................. 51 7.1.1 Architectural overview ................................................................................................................... 51 7.1.2 System Interface Standards ............................................................................................................ 54 7.1.3 I2C Support in WASI ..................................................................................................................... 54 7.1.4 USB Support in WASI ................................................................................................................... 55 7.2 Performance of Wasm for cyber-physical systems .......................................................... 57 7.2.1 I2C performance in WASI .............................................................................................................. 57 7.2.2 I2C implementation in Wasmtime .................................................................................................. 57 7.2.3 I2C implementation in WAMR ...................................................................................................... 58 7.2.4 I2C cold start execution times ........................................................................................................ 58 7.2.5 USB performance in WASI ............................................................................................................ 59 8 Serverless Security Tools for ELASTIC Framework ................................................. 65 8.1 SW-based Mechanisms ....................................................................................................... 65 8.1.1 Lightweight Security orchestrator for Edge devices ...................................................................... 65 8.1.2 eBPF code security Analyser .......................................................................................................... 66 8.1.3 IoT Resource Allocation Model Against Mobility Attacks ........................................................... 66 8.1.4 Lightweight Attribute-based Access Control (ABAC) solution .................................................... 67 8.1.5 Data protection at-rest at the edge with TEE solution .................................................................... 67 8.1.6 WasmHAL-Trust: Automation tooling for confidential computing environments ........................ 67 8.1.7 WASI Security capabilities ............................................................................................................ 68 8.2 HW-based Mechanisms ...................................................................................................... 68 8.2.1 AI Intrusion Detection component ................................................................................................. 68 8.2.2 Hardware-based encryption accelerator ......................................................................................... 69 9 Evaluation metrics.......................................................................................................... 70 10 Conclusions and Next Steps ....................................................................................... 72 11 References ................................................................................................................... 73 ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 6 - February 28, 2025 List of Tables Table 1: Static Memory Allocation Breakdown of Compiled Firmware on ESP32-S3 ........................ 31 Table 2: MQTT Event Topics in Propeller ............................................................................................ 37 Table 3: Task Execution Messaging Topics .......................................................................................... 38 Table 4: Proplet Management Messaging Topics .................................................................................. 38 Table 5: File Transfer Messaging Topics............................................................................................... 38 Table 6: Proplet Management API Endpoints ....................................................................................... 39 Table 7: Task Management API Endpoints ........................................................................................... 39 Table 8: Health & Metrics API Endpoints ............................................................................................. 39 Table 9: WASI-I2C Proposal Portability Criteria .................................................................................. 55 Table 10: KPI Status Update for T2.1 Contributions Related to Objective 2 ........................................ 70 ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 7 - February 28, 2025 List of Figures Figure 1. A simplified overview of the ELASTIC architecture ............................................................. 19 Figure 2. Detailed overview of the ELASTIC architecture ................................................................... 20 Figure 3. Propeller Architecture ............................................................................................................. 24 Figure 4. Successful starting of a Wasm task ........................................................................................ 28 Figure 5. Duplicate task detected; previous instance stopped before restarting execution .................... 28 Figure 6. Successful termination of a Wasm task .................................................................................. 29 Figure 7. Failed stop attempt due to a missing task ............................................................................... 30 Figure 8. Logs confirming Zephyr build system initialisation and toolchain detection ........................ 31 Figure 9. Build logs confirming WAMR integration into the Zephyr build system, showing enabled runtime features, memory allocation, and execution configurations ..................................................... 32 Figure 10. The top-left shows the embedded proplet publishing execution results to the Manager via MQTT, the top-right captures the sent payload with input parameters, and the bottom displays a detailed MQTT transaction log of task execution, status updates, and result publication in the embedded proplet ................................................................................................................................................................ 33 Figure 11. Logs capturing errors for real-time debugging ..................................................................... 34 Figure 12. Real-time MQTT keep-alive message exchange between the broker and the client ............ 35 Figure 13. MQTT messages published successfully with QoS 1........................................................... 36 Figure 14. Proxy Service Architecture for MQTT-HTTP Communication and Wasm Distribution .... 42 Figure 15. Proxy Service Architecture ................................................................................................... 43 Figure 16. WASI Support for AI Acceleration: Overview of WASI-NN API for Running ML Inference in WebAssembly Runtimes .................................................................................................................... 49 Figure 17. Execution of Wasm-based AI/ML Algorithms in the Propeller Framework across Cloud, Edge, and IoT Environments.................................................................................................................. 50 Figure 18. Schematic representation of the proof-of-concept architecture that enables I2C and USB hardware interface support in Wasm applications. The application and device drivers run as Wasm guest components and connect to the host components via WASI. The host components provide ACL and capability-based security and use underlying OS-specific APIs to implement the interfaces ............... 52 Figure 19. WASI-USB Proposal and Prototype Implementations: Demonstrating the API with a PacMan Clone and USB Device Driver in WebAssembly .................................................................................. 56 Figure 20. Mean cold start execution times for writing a four-digit number to the display and reading the temperature from the HTS221 sensor, comparing native, Wasmtime, and WAMR implementations ................................................................................................................................................................ 59 Figure 21. Size of the runtime binary on different test platforms with and without the USB host component. The size increase of the runtime is the worst on Windows at 1.9% larger ........................ 60 Figure 22. Runtime memory usage on the different platforms with and without the USB host component. The memory usage overhead on Linux is at most 1.7%. Windows shows a larger overhead of 11% .. 60 Figure 23. USB flash drive sequential read speeds across various platforms for both native and Wasm guest applications show minimal overhead on high-powered machines, while lower-powered devices experience a larger overhead. The reported speeds represent the median throughput........................... 61 Figure 24. Round-trip time for a 32-byte USB bulk transfer across different platforms, comparing native and Wasm guest applications. x86 Linux shows minimal outliers and reasonable overhead, whereas AArch64 displays greater overhead. The reported times represent the median round-trip time. .......... 62 Figure 25. Round-trip time for a 32-byte USB interrupt transfer across different platforms, comparing native and Wasm guest applications. The latency is seemingly slightly lower for the WebAssembly API compared to the native API; however, this is an artefact of the scheduling of USB interrupt endpoints and how the benchmark application times the latency of the function calls. ......................................... 63 Figure 26. Round-trip time for a 32-byte USB isochronous transfer across different platforms, comparing native and Wasm guest applications. Similarly to the results from the interrupt endpoint benchmarks, the latency is slightly lower on WebAssembly compared to the native API. ................... 64 ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 8 - February 28, 2025 List of Abbreviations ABAC Attribute-Based Access Control ACL Access-control List AI Artificial Intelligence AOT Ahead Of Time API Application Programming Interface CLI Command Line Interface CNCF Cloud Native Compute Foundation CRD Custom Resource Definitions CVM Confidential Virtual Machine EC European Commission FaaS Function as a Service FDE Full Disk Encryption GPU Graphics Processing Unit HAL Hardware Abstraction Layer IDL Interface Description Language IDS Intrusion Detection System IoT Internet of Things IPC Inter Process Communication IR Intermediate Representation JWT Javascript Web Token JIT Just In Time K8s Kubernetes LWT Last Will and Testament MANO Management and Orchestration ML Machine Learning mTLS Mutual TLS OCI Open Container Initiative OS Operating System PEP Policy Enforcement Point PDP Policy Decision Point QoS Quality of Service RBAC Role-Based Access Control RTOS Real-Time Operating System ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 9 - February 28, 2025 RTT Round-Trip Time SIMD Single Instruction Multiple Data SDK Software Development Kit SLA Service-Level Agreement TEE Trusted Execution Environment TPU Tensor Processing Unit WAMR WebAssembly Micro Runtime WASI WebAssembly System Interface Wasm WebAssembly WIT WebAssembly Interface Type WP Work Package XDP Express Data Path ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 16 - February 28, 2025 • Developer API: Developers interact with wasmCloud by creating Kubernetes Custom Resource Definitions (CRDs). This method allows users to deploy and manage Wasm components using familiar Kubernetes tools and workflows, streamlining the development process and reducing the learning curve for those already acquainted with Kubernetes. In a sense, wasmCloud’s use of the Kubernetes API is a perfect example of how Kubernetes is gradually evolving into the Kubernetes-as-an-API pattern instead of Kubernetes-as-a-platform [7]. One key advantage of this approach compared to the one taken by SpinKube is that wasmCloud can take full advantage of the efficiencies, composability, and distributed potential of WebAssembly. For example, by letting go of the Kubernetes overlay network, it can support wRPC natively, bringing the WebAssembly Component Model and WASI preview 2 to serverless WebAssembly orchestration. Furthermore, it can take full advantage of the existing serverless WASI interfaces such as message-bus, even when these conflict with the Kubernetes assumption that all communication happens using TCP/IP. However, this approach also presents certain limitations. wasmCloud operates with a level of independence from Kubernetes' native features, meaning that functionalities such as overlay networks and persistent storage do not automatically integrate. Instead, wasmCloud utilises NATS as its overlay network, bypassing Kubernetes' default networking solutions. Additionally, wasmCloud has its own system for providing workloads access to resources, which may require additional configuration and management efforts. 3.1.3 Erlang BEAM-like lightweight scheduling Erlang’s BEAM-like lightweight scheduling was initially considered for orchestrating serverless AI workloads due to its efficient concurrency model and fault tolerance. However, we ultimately moved away from this approach for two key reasons. First, client nodes needed to run on resource-constrained microcontrollers, where a full BEAM runtime was not feasible due to its memory footprint and processing requirements. This limitation made it impractical for lightweight edge deployments. Second, we required lightweight and secure communication between client nodes, the Manager, and other nodes, relying on event-driven messaging protocols such as MQTT rather than Erlang-native IPC. BEAM’s built-in distributed communication mechanisms, while powerful, assume a homogeneous runtime environment and introduce additional overhead, making them less suitable for our decentralised and heterogeneous architecture. As such, it becomes clear the Erlang BEAM-like lightweight scheduling is not the correct path forward for robust Wasm orchestration in the edge. This is enforced by the decision of wasmCloud to move away from an Elixir/Erlang-based implementation in favour of an in-house Rust solution, recognising the need for a more efficient and adaptable orchestration model for serverless AI workloads at the edge. 3.2 Lightweight and robust supervision with WebAssembly The primary orchestration targets of Kubernetes are high-resource cloud clusters. Running Kubernetes on low-resource clusters suffers from relatively high control plane overhead costs, which hinders adoption in the edge market segment. In complex cloud-native application deployments, Operators are used to automate actions on the Kubernetes cluster state that would otherwise be performed by a human operator. These Operators are one of the main cost drivers of the Kubernetes control plane. To react to changes in the Kubernetes cluster state, the ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 17 - February 28, 2025 Operators must run as long-living processes. Even if the operator’s control loop is idle, the container and process still use cluster resources. For complex applications that use many Operators, these overhead costs quickly accumulate and account for a significant portion of the resource utilisation. This is especially problematic for low-resource deployments. Since global edge application configuration and deployment is a complex task, it is often abstracted by the service provider and included in a Function as a Service (FaaS) offering. FaaS applications are better suited for low-resource edge environments thanks to their fast ondemand scaling properties. Specifically, some edge FaaS platforms use Wasm, like Cloudflare Workers and Fastly Compute@Edge. WebAssembly is used to securely isolate workloads with reduced overhead and scale-to-zero capabilities. Prior work investigated the use of WebAssembly to turn the Kubernetes control plane into serverless functions, resulting in lightweight supervision processes [8]. The downside of this approach, however, is the additional latency of starting the control plane functions each time a change is detected, and the limited support for the existing ecosystem of Kubernetes Operators. To solve the first challenge, proactively loading control plane functions when the runtime expects a change to happen might prove useful. The second problem is twofold. Firstly, the framework only supports the rust-based kube-rs framework instead of the much more popular Golang Kubernetes SDK. Secondly, using it requires a patched version of kube-rs that might not support all existing kube-rs Operators. 3.3 WebAssembly for cyber-physical devices Ensuring drivers are up-to-date and secure, and applications can securely connect to external hardware is crucial for supply chain security, as they frequently have elevated system privileges or direct access to hardware. W3C Wasm and the WASI emerge as a promising solution to shrink the trusted computing base and further enhance embedded development and maintainability by providing a trusted runtime, modern development methodologies, toolchains and software development kits. Wasm was originally created to execute binary code within the browser to improve the performance of web applications and is now actively used outside the browser via WASI. WASI facilitates operating system communication within Wasm by providing a set of portable Application Programming Interfaces (APIs). There are two significant WebAssembly runtimes for Edge and IoT computing, both maintained by the Bytecode Alliance: WAMR and Wasmtime. 3.3.1 WebAssembly Micro Runtime (WAMR) The WebAssembly Micro Runtime (WAMR) is a lightweight standalone runtime with a small footprint, high performance and highly configurable features. Its use cases include embedded, IoT, edge Trusted Execution Environment (TEE), smart contract, and cloud native. Some of its features include a small runtime binary size of about 85kB for the interpreter and about 50kB for AOT and low memory usage. Both the AOT and JIT are LLVM based to achieve nearnative speed. It is also fully compliant to Wasm and implements WASI, as well as a subset for embedded environments. More recent Wasm/WASI proposals are also implemented, such as socket support, SIMD, reference types and threads. Benchmarks often show WAMR to have the best performance and the smallest size. Moreover, due to its support for many different runtime modes including AOT, JIT and interpreter, WAMR is a very flexible platform able to run on both very small and very large devices. ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 18 - February 28, 2025 3.3.2 Wasmtime Wasmtime is the reference WebAssembly runtime used to showcase the state of the art in WASI support. It is a relatively fast runtime because it is built on the Cranelift code generator to translate the target-independent IR quickly to high-quality machine code at runtime or aheadof-time. Cranelift is comparable to LLVM with its own IR (and thus can theoretically be used in non-Wasm use cases) and supports four back-ends (for different target architectures). The speed of code generated by Cranelift (used within Wasmtime) is 14% slower than LLVM (used within WAMR). The compilation speed is however faster than LLVM. Wasmtime has a strong focus on security by using Rust’s runtime safety guarantees, reviews, fuzzing, and a security policy. They also have a strong collaboration with academic researchers to verify critical parts of both Wasmtime and Cranelift. 3.3.3 Lack of WASI support for physical hardware interfaces One of the main challenges with using Wasm and WASI for applications on embedded and IoT devices, is that they require access to hardware interfaces to enable communication with the physical system in which they operate. Wasm does not currently meet the demands of cyberphysical systems, particularly in accessing connected hardware, and requires standardised WASI interfaces to be implemented in Wasm runtimes to maintain its portability and enable communication with hardware across diverse devices. As such, improving this situation is the initial focus of our research. ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 19 - February 28, 2025 4 Architecture and Components To solve the challenges with the state of the art in serverless orchestration, we are developing several components interlinking as part of the broader overarching ELASTIC architecture. Figure 1 shows a simplified overview of the ELASTIC architecture with six larger planes. Each plane links to the rest of the architecture using one or more overarching interfaces. Of note for this deliverable is the Orchestration plane, responsible for deploying and managing workloads, monitoring the workload lifecycle, and orchestrating control plane actions across the other planes. This plane has three main interfaces. • The “Attestation” interface informs the Trust & Access Control plane when a workload is ready to be attested and verified. • The “Standardised Workload Packages” interface is used to ensure hand-off of workloads to the Isolation plane, to execute the workloads in an isolated environment. This interface will build on existing standardised workload package formats such as OCI images and extend them to include additional metadata relevant to the ELASTIC technologies. • The “Observe, Check, Act” interface is responsible for monitoring the lifecycle state of running workloads and infrastructure so that any differences between the desired state of workloads and the actual state of workloads can be reconciled. Figure 1. A simplified overview of the ELASTIC architecture Diving deeper, each plane can be expanded into its own internal architecture containing multiple interconnected components, as shown in Figure 2. This deliverable specifically addresses the preliminary versions of the components highlighted in yellow. Figure 2. Detailed overview of the ELASTIC architecture ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 21 - February 28, 2025 The following architecture components are discussed in this deliverable: ● The Propeller Orchestrator is a novel serverless orchestrator solving the challenges of robustness and fault tolerance of WebAssembly serverless functions on low-resource devices down to microcontrollers with Real Time Operating Systems. Its architecture and implementation are discussed further in Section 5, together with the definition of the Workload packages interface. This component will be showcased as part of Demonstrator 1 (An IoT data fabric as a native 6G infrastructure capability) and Demonstrator 2 (IT/OT - Privacy-preserving confidential computing platform to migrate in-premise sensitive IT services to the cloud). ● The Confidential AI Orchestrator is a novel orchestrator for Confidential AI workloads. It solves the challenges of integrating serverless with AI at the edge. Its architecture and implementation are discussed further in Section 6. This component will be showcased as part of Demonstrator 1. ● The Wasm Operator component solved the challenges of running serverless orchestrators on devices with low resources. Its architecture and implementation will be discussed further in D2.4. This component will be showcased as part of Demonstrator 1. ● The WasmHAL-hardware component is a hardware abstraction layer that solves the challenges of giving serverless WebAssembly workloads access to external hardware on IoT and edge devices. Its architecture and implementation are discussed further in Section 7. This component will be showcased as part of Demonstrator 1. ● The WASI Security component solves the challenge of mediating access control for serverless functions in a security and performance context. The overall security approach of ELASTIC is discussed in Section 8. The full architecture and implementation of WASI Security will be discussed further in D2.4. This component will be showcased as part of Demonstrator 1. ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 22 - February 28, 2025 5 Architecture of Improved Wasm Orchestrator Propeller 3 is a distributed Wasm workload orchestrator designed to facilitate the management and deployment of Wasm applications across heterogeneous computational environments. It extends from high-performance cloud servers to resource-constrained edge devices, integrating flexibility, security, and performance. While Kubernetes has emerged as the de facto standard for container orchestration [9], its architecture is not inherently optimised for orchestrating Wasm workloads across diverse platforms due to differences in execution models, resource management, and runtime integration [10]. SpinKube represents a significant advancement in integrating Wasm into Kubernetes (K8s), refining previous approaches such as WOK and Krustlet [1]. It leverages the runwasi interface to enable Wasm execution within containerd. The above integration facilitates seamless deployment alongside conventional containers, yielding substantial performance enhancements. By maintaining full compatibility with existing Kubernetes tools and workflows, SpinKube provides a scalable, cost-effective, and developer-friendly solution for orchestrating Wasm workloads within Kubernetes environments. However, SpinKube does not fully address the need for a comprehensive and adaptable orchestration framework capable of managing Wasm workloads across a broader range of computational platforms. For example, SpinKube requires the relatively complex and resource-hungry Kubernetes control plane and does not work on non-Linux platforms such as the Zephyr real-time operating system. In contrast, Mechanoid, an open-source framework, is specifically designed for building and executing WebAssembly applications on embedded systems and IoT devices [11]. Its primary objective is to facilitate the development of secure and extensible applications while leveraging the latest advancements in both WebAssembly and embedded computing. However, Mechanoid lacks the capability to orchestrate Wasm workloads across heterogeneous platforms. Propeller addresses the above limitations by offering a unified and adaptable framework for managing and deploying Wasm workloads efficiently across heterogeneous platforms. It is designed to bridge the gaps by providing a unified, flexible, and scalable approach to Wasm workload orchestration. While it is not intended as a direct replacement for Kubernetes, it functions as a complementary solution targeting diverse computational infrastructure such as Realtime Operating Systems that cannot run Kubernetes. 5.1 Features Propeller enables seamless cloud-edge orchestration, allowing Wasm workloads to be deployed and managed across a wide range of computing environments. High-performance cloud infrastructures handle large-scale computations, while resource-constrained microcontrollers at the edge execute lightweight tasks efficiently. This distributed execution model ensures that workloads are processed in the most suitable environment based on their computational requirements. With the fast boot times of Wasm, workloads achieve near-instant execution, making Propeller particularly well-suited for latency-sensitive applications such as real-time analytics and interactive services. Additionally, Propeller supports FaaS deployments, enabling users to define event-driven functions that dynamically scale based on demand. This capability 3 https://github.com/absmach/propeller ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 23 - February 28, 2025 improves resource utilisation and enhances operational efficiency by ensuring that computational resources are allocated only when necessary. To facilitate seamless artefact management, Propeller integrates with OCI-compliant registries, allowing for efficient storage, retrieval, and distribution of Wasm workloads. This ensures interoperability with existing containerised ecosystems, streamlining deployment processes. Furthermore, the inclusion of the WebAssembly Micro Runtime (WAMR) [12] in the Zephyr RTOS extends execution capabilities to highly resource-constrained embedded devices, making Propeller an ideal solution for industrial automation, automotive applications, and smart sensors. For secure and efficient communication in distributed IoT networks, Propeller integrates with SuperMQ [13], a modern, scalable, and secure open-source IoT cloud platform written in Go. SuperMQ provides seamless protocol bridging, supporting HTTP, MQTT, WebSocket, and CoAP, enabling flexible communication between users and devices and acting as a middleware layer for IoT applications. Security remains a fundamental aspect of Propeller’s architecture. By enforcing strict workload isolation and end-to-end secure communication channels, Propeller mitigates risks associated with untrusted execution environments, ensuring confidentiality, integrity, and reliability in Wasm-based workloads. 5.2 How It Works Propeller is a flexible platform for managing and executing distributed tasks across cloud, edge, and IoT devices. It combines Wasm and SuperMQ for seamless workload orchestration. Users write lightweight Wasm workloads using existing toolchains, which are then pushed to an OCIcompliant registry for easy deployment. The platform allows users to orchestrate task deployment and manage proplets - orchestration agents deployed in the form of cluster (equivalent to Kubernetes kubelets). The setup begins by provisioning SuperMQ. Devices such as "Manager" and "Proplet" are created with secure communication channels. Tasks are created via the manager's RESTful API, where Wasm modules are uploaded or pulled from the registry and associated with tasks. Propeller manages task scheduling; delegates work and ensures communication between a proplet and the manager. 5.3 System Architecture Propeller manages and executes tasks across multiple nodes (proplets). It leverages MQTT for communication, a manager service for task orchestration, and a proxy service for container image distribution. The architecture of Propeller is illustrated in Figure 3. The system is composed of the following key components: • CLI: Command Line Interface for interacting with the Propeller system. • Manager: Central service responsible for task management and proplet coordination. • Proplet: Worker nodes that execute tasks. • Proxy: Service for fetching and distributing container images from a registry to proplets. • SuperMQ: Internal event-driven Infrastructure for communication between services. ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 24 - February 28, 2025 Figure 3. Propeller Architecture 5.4 Components 5.4.1 CLI The Command-Line Interface (CLI) provides a user-friendly mechanism for interacting with the Propeller system. Built with a modular architecture, it supports three core functionalities: task management, resource provisioning, and system configuration. ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 25 - February 28, 2025 Under task management, the CLI enables users to create, view, update, delete, start, and stop tasks. To create a task, users can execute the tasks create <name> command, specifying a descriptive name for the new task. Tasks are created with unique identifier <id> and timestamps, ensuring traceability and organisation. The tasks view <id> command retrieves and displays task details based on its unique identifier. Task updates can be made using the tasks update <id> command, while tasks can be removed from the system with the tasks delete <id> command. For task execution, the tasks start <id> and tasks stop <id> commands allow users to start and stop tasks as needed. All the above operations interact with the backend system via the Propeller Software Development Kit (SDK), ensuring seamless communication. The CLI also streamlines resource provisioning for Propeller components by automating the setup of domains, managers, proplets, and communication channels. During the above process, users are prompted to input their credentials, such as username and password, for authentication. Resource names can be either user-defined or automatically generated by the CLI. Once the configuration is complete, the process outputs a config.toml file containing all necessary details for subsequent operations. Additionally, the CLI establishes and validates connections between managers and proplets to ensure that the system is fully operational. The above-automated provisioning reduces complexity and enhances user productivity by standardising the process. 5.4.2 Manager The Manager serves as the central coordination unit within the Propeller system, responsible for orchestrating task execution, managing proplets, and facilitating communication between system components. It ensures the efficient allocation of resources, supervises task lifecycle transitions, and provides mechanisms for monitoring and system observability. The Manager's primary responsibility is the oversight of task lifecycle management. Unlike the CLI, which serves as a user-facing interface for task operations, the Manager operates as the backend controller that handles task scheduling, state transitions, and interactions with system components. It maintains a registry of proplets and uses a round-robin scheduling algorithm to distribute tasks among the proplets. The scheduling ensures a balanced workload across available proplets. To ensure reliability, the Manager monitors the liveness of proplets through MQTT subscriptions and updates the status of the proplets for maintaining system integrity and ensuring that tasks are assigned only to operational proplets. To support system monitoring and debugging, the Manager incorporates middleware for logging, tracing, and metrics collection. Logging captures detailed records of internal operations, while distributed tracing, implemented via OpenTelemetry, provides visibility into execution flows across the system. Metrics collected through Prometheus offer insights into performance and resource utilisation, enabling operators to monitor the health of the Manager and its interactions with other system components. Configuration of the Manager is handled through environment variables and the config.toml configuration file, allowing it to be customised for different deployment environments. The Manager stores tasks information, proplets information and task-to-proplet mappings. 5.4.3 Proplet The proplet in the Propeller system serves as the lightweight execution environment for running tasks. Designed to operate efficiently within distributed environments, the proplet is responsible for executing Wasm tasks, communicating with the Manager, and maintaining system-wide ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 32 - February 28, 2025 To establish a direct connection between the application and WAMR’s execution environment, the build system explicitly includes WAMR’s core runtime headers and source files. This guarantees that the WebAssembly engine is properly compiled and linked within the firmware, ensuring smooth execution of Wasm workloads. Finally, WAMR is embedded into the Zephyr application as a dedicated library using zephyr_library_named(wamr_lib). The application then links WAMR with the Zephyr build system through target_link_libraries(app PRIVATE wamr_lib), allowing the Wasm execution environment to be tightly integrated with the Zephyr firmware. The setup, as seen in Figure 8 and Figure 9 ensures that the embedded proplet maintains memory isolation and runtime efficiency while executing WebAssembly workloads, making it well-suited for embedded devices. Figure 9. Build logs confirming WAMR integration into the Zephyr build system, showing enabled runtime features, memory allocation, and execution configurations 5.4.4.4 Wasm Handler The Wasm handler, implemented in wasm_handler.c and wasm_handler.h, serves as the primary interface between the embedded proplet and the embedded device. It is responsible for reading binary WebAssembly modules, validating their integrity, and loading the Wasmmodules into WAMR’s runtime. The validation step ensures that corrupted or malformed modules do not compromise system stability. Once a Wasm module is loaded, the handler initialises the runtime environment to ensure proper execution. The initialisation includes allocating a dedicated stack (16 KB) and heap (16 KB) using wasm_runtime_instantiate(). To support concurrent execution of multiple Wasm workloads, the handler maintains an array where each running module is tracked with a unique task ID. The implementation of the embedded proplet also ensures that Wasm modules can be explicitly terminated, freeing memory and execution slots when no longer needed. For Wasm modules to interact with external hardware and networking components, the embedded proplet exposes controlled system interfacing methods. The methods provide secure access to essential embedded system capabilities, including: ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 33 - February 28, 2025 • Publishing results over MQTT: The handler enables the embedded proplet to send execution results to the Manager using publish_results(), facilitating seamless communication with the Manager as seen in Figure 10. • Interacting with external inputs: Embedded proplets receive dynamic inputs through a defined input structure (inputs[MAX_INPUTS]), allowing parameterised execution of WebAssembly code as seen in Figure 10. Figure 10. The top-left shows the embedded proplet publishing execution results to the Manager via MQTT, the top-right captures the sent payload with input parameters, and the bottom displays a detailed MQTT transaction log of task execution, status updates, and result publication in the embedded proplet • Performing logging and debugging: The embedded proplet integrates with Zephyr’s logging framework, ensuring that execution logs and error messages are captured for real-time monitoring and debugging as seen in Figure 11. ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 34 - February 28, 2025 Figure 11. Logs capturing errors for real-time debugging Since embedded devices have limited resources, the Wasm handler enforces strict memory isolation and execution constraints to maintain system reliability. The isolation is achieved through: • Creating execution environments: Each Wasm module operates within a sandboxed memory region, using wasm_runtime_create_exec_env(), preventing unintended access to system memory. • Limiting execution time and memory usage: The Wasm handler enforces a predefined memory allocation to prevent system-wide memory exhaustion. Execution timeouts can be defined. • Handling runtime exceptions: The system continuously monitors for execution errors using wasm_runtime_get_exception(). If an error occurs, it is logged, and the execution is halted to prevent cascading failures. Furthermore, the orchestrator assigns Wasm tasks to available embedded proplets based on resource availability, with the Wasm handler dynamically managing execution priorities. Each embedded proplet periodically reports its workload status to the orchestrator, enabling real-time load balancing and task reassignment if needed. 5.4.4.5 Networking and Connectivity The networking and connectivity components of the embedded proplet are built upon the networking stack of Zephyr, providing robust support for Wi-Fi and IP-based communication. The configuration file enables Wi-Fi networking and network management, allowing devices to establish and maintain wireless connections effectively. The system also supports general networking capabilities through the network management layer of Zephyr, which enables runtime control and configuration of networking interfaces. The system relies on DHCPv4 for dynamic IP address allocation, ensuring that each embedded proplet can automatically obtain an IP address when connecting to the network. The above ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 35 - February 28, 2025 eliminates the need for static IP configurations and allows seamless integration into existing network infrastructures. Additionally, the networking stack supports both TCP and UDP, ensuring compatibility with various communication protocols used in distributed systems. To manage multiple network interfaces, the configuration allows up to two IPv4 addresses, configured through CONFIG_NET_IF_MAX_IPV4_COUNT=2, per network interface. Though IPv6 support is available, it is disabled in the configuration file, as the embedded proplet currently prioritises IPv4 networking. At the link-layer level, the system enables Ethernet and Wi-Fi management, ensuring that edge devices connect using standard networking interfaces. The configuration file also specifies a maximum of two managed Wi-Fi interfaces, allowing the system to handle multiple Wi-Fi connections efficiently. Packet and buffer management is fine-tuned to optimise networking performance for embedded devices, where memory and processing power are constrained and can lead to dropped packets, increased retransmissions, and degraded communication efficiency. The configuration sets CONFIG_NET_BUF_RX_COUNT=64 and CONFIG_NET_BUF_TX_COUNT=64, ensuring that sufficient buffers are allocated for incoming and outgoing network packets to reduce packet loss and improve transmission reliability. Similarly, CONFIG_NET_PKT_RX_COUNT=32 and CONFIG_NET_PKT_TX_COUNT=32 define the number of packet descriptors available for processing network traffic, balancing memory usage and network performance. The dedicated memory allocations for networking tasks CONFIG_NET_RX_STACK_SIZE=2048 and CONFIG_NET_TX_STACK_SIZE=2048, further improve the responsiveness and stability of the system in handling concurrent network operations. Additionally, CONFIG_NET_MAX_CONTEXTS=10 ensures that multiple networking contexts can be managed concurrently, allowing seamless handling of multiple network sockets, connections, or protocols. A key aspect of connectivity in the embedded proplet is the integration of MQTT for messagebased communication. The configuration enables the MQTT library and socket support for seamless data exchange between embedded Propeller nodes and the orchestrator. The MQTT client is configured to maintain session state and use a keep-alive mechanism to ensure continuous connectivity as shown in Figure 12. Figure 12. Real-time MQTT keep-alive message exchange between the broker and the client ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 36 - February 28, 2025 To enhance reliability, the embedded proplet supports the Last Will and Testament (LWT) feature of MQTT. The feature ensures that in the event of an unexpected disconnection of the embedded proplet, a predefined message is sent to the broker, notifying the Manager of the disconnection event. Additionally, the embedded proplet leverages Quality of Service (QoS) levels, as shown in Figure 13, to provide varying degrees of message reliability, ensuring that critical messages are received without duplication or loss. The configuration also allows for adaptive reconnection strategies, ensuring that embedded proplets can re-establish connections in case of temporary network disruptions. Figure 13. MQTT messages published successfully with QoS 1 5.4.5 Proxy The Proxy serves as an intermediary between HTTP-based registries and the MQTT-based communication framework that is implemented in the distributed architecture of Propeller. Its primary role is to bridge HTTP - MQTT communication protocol by efficiently fetching Wasm files from OCI registries and streaming them to proplets using MQTT in manageable chunks. The proplet tracks the progress of chunk transmission to ensure the complete delivery of each container. Once the image chunks are delivered, the proplet reassembles the image locally for execution. Configuration of the Proxy is managed through environment variables or a config.toml file, enabling flexibility across deployment environments. MQTT settings include the broker URL, client credentials, and channel ID, while HTTP registry settings specify the registry URL, authentication method, and chunk size. The above configurability allows the Proxy to adapt to various operational requirements and ensures compatibility with different registry and communication setups. 5.4.6 SuperMQ SuperMQ - an open-source messaging system - is designed to be highly scalable and secure, making it an ideal solution for building modern, event-driven systems. Unlike traditional service meshes that often require heavy overhead and complex configurations [14], SuperMQ offers a simplified approach to service communication. With its multi-protocol connectivity, including support for HTTP, MQTT, WebSocket, and CoAP, SuperMQ ensures seamless ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 37 - February 28, 2025 messaging across diverse applications [15]. This flexibility makes it particularly useful for IoT, real-time data pipelines, and other event-driven architectures. It also emphasises security by supporting mutual TLS (mTLS), JWT, and fine-grained access control through ABAC and RBAC policies [16]. SuperMQ simplifies the process of locating and interacting with services. Traditional service meshes often require a complex infrastructure to manage service discovery and communication between services, which can involve managing numerous YAML files and container configurations. However, with SuperMQ, the need for knowing the precise location or IP address of services is eliminated. Instead, services simply need to know the message subjects (or topics) they need to interact with. This allows for a more flexible, decoupled system where services can communicate without being tightly bound to specific addresses. Whether services are running in Kubernetes, on bare-metal hardware, or in edge environments, SuperMQ enables efficient communication without the complexity of traditional service meshes. Moreover, SuperMQ's approach to communication resilience and routing control ensures that systems are robust and can scale easily. With built-in features such as retries, timeouts, circuitbreaking, and rate limiting, SuperMQ handles the intricacies of maintaining reliable communication between services. This can be especially beneficial when managing rolling updates or handling failures in distributed systems. By integrating these features with the power of a global SuperMQ cluster, Propeller is built to be resilient, scalable message infrastructure without the complexity of managing multiple service mesh configurations. 5.4.7 Communication Communication in the Propeller system is facilitated through two primary protocols: MQTT and HTTP. Each protocol plays a distinct role in enabling efficient and reliable interactions between the Manager, proplets, Proxy, and CLI. 5.4.7.1 MQTT MQTT serves as the primary messaging protocol in Propeller, facilitating event-driven, asynchronous communication between key system components: the Manager, proplets, and Proxy. MQTT messaging in Propeller is structured into four primary categories: 1. Connection and Authentication: When a client (Manager, proplet, Proxy, or CLI) connects to the MQTT broker, SuperMQ handles authentication and connection management. Upon successful connection, the system records an event to track client activity. Table 2 below lists MQTT events related to client connection and authentication, detailing when system components establish or disconnect from MQTT sessions. Table 2: MQTT Event Topics in Propeller Event Topic Description Client Connects supermq.mqtt/connect Published when a client establishes an MQTT session Client Disconnects supermq.mqtt/disconnect Published when a client disconnects 2. Task Execution Messaging: The Manager and proplets communicate via MQTT for task lifecycle management. The Manager publishes task instructions, while proplets subscribe to receive and execute the instructions. The proplet publishes task results back ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 38 - February 28, 2025 to the Manager. The bidirectional communication ensures synchronisation between the proplet and the Manager. Table 3 shows the MQTT topics used for task lifecycle management, facilitating communication between the Manager and proplets for task creation, execution, and status updates. Table 3: Task Execution Messaging Topics Event Topic Description Task Creation /control/manager/create The Manager creates a task Start Task /control/manager/start The Manager signals proplets to start a task Stop Task /control/manager/stop The Manager signals a proplet to stop a running task Task Status Update /control/manager/taskupdate proplets publish task status Task Completion /control/proplet/results proplets publish task results 3. Proplet Management Messaging: The proplet publishes periodic "alive" status updates to indicate its availability and responsiveness to the Manager. The updates allow the Manager to monitor the operational status of all proplets and ensure that they are responsive to incoming tasks. Table 4 details the MQTT topics used for proplet management, including registration and periodic liveness updates to the Manager. Table 4: Proplet Management Messaging Topics Event Topic Description Proplet Registration /control/proplet/create Proplets announce availability to the Manager Liveness Updates /control/proplet/alive Proplets send periodic heartbeats 4. File Upload and Chunking: To accommodate large Wasm binaries, MQTT enables chunked file transfers, preventing network congestion and optimising performance. Table 5 describes the MQTT topics used for handling large Wasm binary transfers via chunked messaging, ensuring efficient file distribution to proplets. Table 5: File Transfer Messaging Topics Event Topic Description Fetch request /registry/proplet Request file chunks from OCI registry File chunk transfer /registry/server Receive file chunks from registry To maintain robust communication, Quality of Service (QoS) levels, message retention, and session persistence are utilised. QoS levels ensure that critical messages, such as task commands or task results, are delivered reliably even in the event of network interruptions. Retained messages allow newly connected proplets to receive the most recent task or configuration updates, ensuring consistency in operations. Session persistence ensures that messages intended for temporarily disconnected components are delivered once they reconnect, minimising data loss and ensuring system integrity. ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 39 - February 28, 2025 5.4.7.2 HTTP HTTP is primarily used for interactions between the CLI and the Manager. The Manager exposes RESTful endpoints that enable the CLI to facilitate task and proplet management. The API is organised into three primary categories: 1. Proplet Management Endpoints: Enables querying and monitoring of proplet instances. Table 6 lists the available HTTP endpoints for proplet management. Table 6: Proplet Management API Endpoints Event Endpoint Description GET /proplets Lists all available proplets GET /proplets/{propletID} Fetches details of a specific proplet 2. Task Management Endpoints: Provides full lifecycle management for tasks, including execution control and Wasm binary uploads. Table 7 presents the HTTP endpoints used for creating, updating, and managing tasks. Table 7: Task Management API Endpoints Event Endpoint Description POST /tasks Creates a new task GET /tasks Retrieves a paginated list of tasks GET /tasks/{taskID} Fetches details of a specific task PUT /tasks/{taskID} Updates a task's metadata DELETE /tasks/{taskID} Deletes a task POST /tasks/{taskID}/start Starts the execution of a task POST /tasks/{taskID}/stop Stops a running task PUT /tasks/{taskID}/upload Uploads a Wasm binary for a task 3. Health & Metrics Endpoints: Facilitates system health monitoring and telemetry collection. Table 8 outlines the HTTP endpoints available for health checks and metric retrieval. Table 8: Health & Metrics API Endpoints Event Endpoint Description GET /health Checks the health status of the Manager GET /metrics Provides Prometheus-compatible metrics 5.5 OCI Registry for Wasm Workloads The Open Container Initiative (OCI) is a collaborative project that establishes open industry standards for container runtimes, image formats, and distribution to ensure interoperability across different container ecosystems. By defining a standardised container image format and ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 40 - February 28, 2025 runtime specifications, OCI enables portability, security, and consistency in containerised applications. The OCI Registry is a critical component of this ecosystem, providing a standardised way to store, distribute, and manage container images and artefacts. It ensures compliance with OCI specifications, allowing developers to push, pull, and sign images securely while supporting multi-cloud and hybrid-cloud deployments. OCI Registries enhance container workflows by offering versioning, access control, and integration with CI/CD pipelines, making them fundamental for modern software distribution. 5.5.1 Wasm to OCI The wasm-to-oci tool represents an innovative approach to Wasm module distribution by leveraging existing container registry infrastructure. This integration creates a seamless bridge between WebAssembly deployment and established container ecosystems. Container registries have evolved beyond traditional Docker images to become general-purpose artefact storage systems. 5.5.1.1 How it works This leverages the OCI Artifacts proposal, whose goal is to enable the distribution of more cloud-native artefacts using existing registry infrastructure and uses it to store WebAssembly modules as single-layer blobs in the registry. The primary function of wasm-to-oci lies in its ability to treat WebAssembly modules as container images. This approach maintains the lightweight and portable nature of WebAssembly while taking advantage of robust container registry infrastructure. 5.5.1.2 Push Operation The compilation of the WebAssembly module begins with the creation of an example programme - addition.wasm - by running the make addition command, which places the resulting binary in the build directory. The push action then makes it easier to convert locally built WebAssembly modules into container images that adhere to OCI specifications and are optimised for registry persistence. Standardised artefact storage and retrieval are made possible by this transformation process, which complies with the distribution specification of the Open Container Initiative. The conversion process includes encapsulating the WebAssembly binary in a layer-based structure together with the necessary manifest files and metadata that define OCI images. This procedure is illustrated in the following example: $ ls build add.wasm $ wasm-to-oci push build/add.wasm docker.io/<username>/addition:latest Pushed: docker.io/addition:latest Size: 124962 Digest: sha256:9c82cbe576ee947c00435ac8053a800a1969f4757ae4a81f870f714674afc91a During this operation, the tool: • Converts the Wasm module to an OCI image format • Uploads it to the specified registry • Generates a SHA256 digest for content verification • Returns size and digest information ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 41 - February 28, 2025 5.5.1.3 Pull Operation The pull operation retrieves Wasm modules from OCI registries and restores them to their original format: $ wasm-to-oci pull docker.io/<username>/addition:latest --out addition.wasm Pulled: docker.io/<username>/addition:latest Size: 124962 Digest: sha256:4c7915b4c1f9b0c13f962998e4199ceb00db39a4a7fa4554f40ae0bed83d9510 This Wasm file can be executed as follows: $ wasmtime --invoke add addition.wasm 1 2 warning: using `--invoke` with a function that takes arguments is experimental and may break in the future warning: using `--invoke` with a function that returns values is experimental and may break in the future 3 This process: • Downloads the module from the registry • Extracts the Wasm content • Enables immediate execution with Wasm runtimes 5.5.1.4 Technical Implementation The wasm-to-oci tool leverages the OCI Artifacts proposal to enable WebAssembly module distribution through existing registry infrastructure. At its core, the implementation stores WebAssembly modules as single-layer blobs in the registry, using a specialised manifest structure. The tool utilises a custom manifest format that adheres to OCI specifications. Here's an example of the manifest structure: { "schemaVersion": 2, "config": { "mediaType": "application/vnd.wasm.config.v1+json", "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", "size": 2 }, "layers": [ { "mediaType": "application/vnd.wasm.content.layer.v1+wasm", "digest": "sha256:4c7915b4c1f9b0c13f962998e4199ceb00db39a4a7fa4554f40ae0bed83d9510", "size": 1624962 } ] } ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 48 - February 28, 2025 Inference, on the other hand, is where Wasm excels as it is lightweight, highly portable, and well-suited for deployment across browsers, edge devices, and serverless environments. With its ability to run pre-trained models securely in a sandboxed environment, Wasm offers significant advantages for edge AI and real-time applications. Performance optimisations such as SIMD acceleration, model quantisation, and precompiled binaries can further enhance inference speed and efficiency. 6.3 WASI Support for AI Acceleration The WASI-NN proposal provides a high-level API for running machine learning (ML) inference within WebAssembly runtimes, specifically targeting non-browser environments like standalone Wasm engines. It enables Wasm programs to access host-provided ML functionalities, using frameworks like OpenVINO™ [19] for peak performance, especially on x86 architectures. The above stems from the limitations of WebAssembly's architecture, which cannot expose specialised instructions for ML acceleration or compile directly to auxiliary devices like GPUs or TPUs. WASI-NN aims to address the limitations by abstracting the hardware-specific details and offering a format-agnostic solution for loading and executing ML models. The specification of WASI-NN is designed as a "graph loader" API, differing from other APIs like WebNN that allow users to build ML computation graphs. WASI-NN supports the loading, binding, and evaluating of pre-trained models without requiring users to build the computational graph node-by-node. This simplifies the process for developers by relying on existing tools to handle model compilation for the appropriate devices. The process to use WASI-NN involves: 1. Loading a model and its weights as byte arrays. 2. Initialising the execution context and binding tensors. 3. Executing the inference. 4. Retrieving the output tensors. Figure 16 illustrates the WASI support for AI acceleration, showing the key steps in the WASINN process for running machine learning inference in WebAssembly runtimes. ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 49 - February 28, 2025 Figure 16. WASI Support for AI Acceleration: Overview of WASI-NN API for Running ML Inference in WebAssembly Runtimes 6.4 Execution of Wasm AI/ML Algorithms using Propeller Propeller provides a robust and flexible framework for orchestrating WebAssembly-based AI/ML workloads across cloud, edge, and IoT environments. By leveraging Propeller’s distributed architecture, AI models can be efficiently deployed, managed, and executed on heterogeneous computing platforms, ensuring optimal performance, security, and resource utilisation. The execution of AI/ML algorithms in Wasm runtimes through Propeller enables seamless migration of workloads between high-performance cloud servers and resourceconstrained edge devices, making it particularly well-suited for low-latency inference, federated learning, and secure AI execution in TEEs. In the Propeller ecosystem, AI/ML models are compiled into Wasm modules and stored in OCIcompliant registries for efficient distribution. The Propeller Manager is responsible for workload orchestration, scheduling execution based on available compute resources, security constraints, and real-time performance requirements. Using SuperMQ, a secure and scalable messaging layer, Propeller ensures low-latency communication between the orchestration layer and distributed proplets, lightweight execution nodes deployed across cloud, edge, and embedded environments. For inference tasks, Wasm-based AI models are dynamically dispatched to proplets running on edge devices, ensuring minimal latency and real-time processing. Propeller integrates with WebAssembly Micro Runtime (WAMR) and WASI-NN, enabling execution on low-power IoT devices, mobile platforms, and microcontrollers while maintaining compatibility with standard ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 50 - February 28, 2025 AI inference frameworks such as OpenVINO™ and TensorFlow Lite. In security-sensitive environments, AI workloads can be executed inside TEEs, where Wasm’s sandboxing capabilities further enhance isolation and protect AI models from unauthorised access. Remote attestation mechanisms ensure workload integrity, making Propeller an ideal solution for privacy-preserving AI applications in healthcare, finance, and industrial automation. Figure 17 illustrates the execution of Wasm-based AI/ML algorithms within the Propeller framework across diverse environments, highlighting key components such as workload orchestration, edge device inference, and secure execution in TEEs. Figure 17. Execution of Wasm-based AI/ML Algorithms in the Propeller Framework across Cloud, Edge, and IoT Environments ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 51 - February 28, 2025 7 Cyber-physical Wasm Workload Orchestration This chapter explains our initial improvements for WebAssembly support for cyber-physical devices. Our contribution includes two in-progress standards and three prototype implementations that were heavily benchmarked to ensure feasibility on low-resource devices. These contributions have been highlighted in one peer-reviewed paper and three master’s theses. • Accepted paper: “Cyber-physical WebAssembly: Secure Hardware Interfaces and Pluggable Drivers” [20]. • Master’s thesis: “Advancing the I2C proposal for WebAssembly System Interface” [21]. • Master’s thesis: “Design and Prototype of a WebAssembly-Native USB Interface” [22]. • Master’s thesis: “Standardising USB in the WebAssembly System Interface” [23]. Our two standard proposals are already upstream in the WebAssembly GitHub organisation, and our prototype implementations are provided on GitHub as open-source code. • wasi-i2c standard: ○ In-progress standard: https://github.com/WebAssembly/wasi-i2c ○ prototype: https://github.com/idlab-discover/i2c-wasm-components • wasi-usb standard: ○ In-progress standard: https://github.com/WebAssembly/wasi-usb ○ Prototype 1: https://github.com/idlab-discover/usb-wasm ○ Prototype 2: https://github.com/Wouter01/USB_WASI 7.1 Improving Wasm Runtime Support for Edge Computing 7.1.1 Architectural overview The goal of this work is to create standardised interfaces to connect WebAssembly applications to physical hardware. Our work initially focuses on connecting to hardware using the I2C and USB protocols. Figure 18 shows a diagram of the proof-of-concept architecture to enable I2C and USB hardware interface support in Wasm applications. Both the application and the peripheral device drivers run as WebAssembly components. These connect to the runtime via standardised generic I2C and USB WebAssembly System Interfaces. The runtime itself has several host components, one for each interface, that implement the interfaces and an provide access-control list (ACL) and capability-based security. The host component itself communicates with the host operating system (OS) and uses OS-specific APIs for hardware access. The remainder of this section examines important concepts in more detail. ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 52 - February 28, 2025 Figure 18. Schematic representation of the proof-of-concept architecture that enables I2C and USB hardware interface support in Wasm applications. The application and device drivers run as Wasm guest components and connect to the host components via WASI. The host components provide ACL and capability-based security and use underlying OS-specific APIs to implement the interfaces Concept 1: Interface Worlds An interface world describes the system calls that are supported by the runtime using the Component Model's Interface Description Language (IDL) called Wasm Interface Type (WIT). For example, the USB world contains a WIT description of all the system calls that Wasm components can use to communicate with USB devices. Concept 2: Host Components The host component is the host-side implementation of the system call API. It is embedded in the runtime and runs as native code directly on the underlying OS, as it requires access to native APIs of the host OS to communicate with hardware devices. As such, the host component is highly dependent on the specific host OS interfaces. Because it is embedded in the runtime, the host component also depends on the specific implementation details of the runtime. Therefore, the implementation of the host component is unique for every combination of runtime and underlying operating system. Any new system or runtime that wishes to support certain interface worlds will require a modified implementation of the host component. However, this burden can be lessened by using higher-level cross-platform libraries to implement the host component, rather than depending directly on the OS through system calls. This way, one implementation of the host component can be compiled for multiple operating systems. Unfortunately, using a higher-level library is not always feasible, especially on certain lowpowered embedded devices due to device-specific limitations, requiring a custom host component implementation for those devices. Concept 3: Guest Components Guest components contain the application logic and device drivers or a hardware abstraction layer (HAL). These can be made up of multiple separate components, or the application logic can embed the driver and HAL logic inside of it. A guest component specifies which interfaces ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 53 - February 28, 2025 it requires through imports and the interfaces it provides through exports using a WIT world description. This informs the runtime about which host component interfaces to connect to the guest component, allowing the guest component to interact with hardware devices through host component function calls. Guest components are contained within a sandbox by the runtime, ensuring they remain isolated and cannot access system resources without authorisation. Concept 4: Wasm Runtime The Wasm runtime resolves the imports and exports for all components and links them together. To successfully resolve imports and exports, the runtime must support the WIT world that a component implements. This can be achieved by a host component providing the necessary implementations for the guest component's imported functions. After linking, the guest component has access to the imported functions from the world it implements. The runtime, in turn, has access to the functions that the world exports. To begin execution, the runtime must call one of those exported functions. Since (our) interface worlds are not concerned with how a component starts execution, we delegate component start to the wasi-cli [24] world, which specifies the component’s entry point. Two popular Wasm runtimes are Wasmtime [25] and WebAssembly Micro Runtime (WAMR) [26], both maintained under the supervision of the Bytecode Alliance [27]. Concept 5: Capability-Based Security One of the strengths of WASI is its capability-based security mechanism. This mechanism implicitly allows granular control over what resources an application can access based on the API definition. If a resource is not provided, the module cannot interact with it, implicitly revoking the capability for that resource. For example, WASI file system APIs apply capabilitybased security so that applications can only access files and directories to which they have been granted access [28]. Otherwise, the resource does not exist. Although WASI enables the implementation of capability-based security for system interfaces, it does not stipulate the level of granularity at which these security controls should be constructed. The capability-based security model is inherently defined by the API, which also determines the granularity of the security measures. This implies that access control must define the resources to be protected, which is impractical for fine granularity and challenging when control layers are orthogonal or complementary. Consequently, we propose an extension of capability-based security with explicit access control lists (ACLs). Concept 6: Access Control List (ACL) Based Security Access control lists add a layer of granular and orthogonal access control, complementing capability-based access control. Unlike capability-based access control, which requires changes to the API, ACLs can be implemented without affecting the existing API. For example, an ACL rule might allow an application to read from a device but not write to it. Furthermore, ACLs offer greater flexibility and can be fine-tuned to the environment and runtime as needed. Based on these advantages, we propose that standardised interfaces use capability-based security for high-level concepts and use an additional ACL to further restrict permissions. Our implementation of ACL-based access control necessitates the incorporation of logic specific to the runtime, environment, and peripheral. Whenever the runtime creates a new module instance, an I/O context is created and linked to the corresponding execution environment. Whenever a device interaction is performed, the corresponding I/O context is retrieved from the execution environment, and depending on the type of device interaction and the I/O context's properties, the action is permitted or denied. ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 54 - February 28, 2025 The I/O context's properties may refer to permitted interaction types, such as read or write. Additionally, specific values may be blocked, which is necessary for device interaction with arbitrary data, but, for example, prevents changes in the configuration. Furthermore, the context may specify how connections should be established with end devices, preventing communication with other peripherals sharing the same bus. 7.1.2 System Interface Standards To ensure compatibility with future toolchains and platforms, it is important to standardise our cyber-physical WebAssembly interfaces. Standardisation will enable the integration of support for these interfaces into runtimes, compiler toolchains, HAL libraries, and drivers. To initiate and facilitate the standardisation process [29], IMEC joined the WebAssembly System Interface Subgroup of the W3C WebAssembly Community Group [30]. 7.1.3 I2C Support in WASI The WASI-I2C proposal is co-championed by representatives from both IMEC and Siemens and is currently in the second phase of the standardisation process, with ongoing efforts to fulfil the criteria to pass the vote to the third phase. You can find the latest version of the standard proposal in the upstream wasi-i2c repository [31] and two runtime implementations: one for WAMR and one for Wasmtime, in the i2c-wasm-components repository [32]. The primary goal is to provide an interface that WASI programs can use to read and write data over an I2C connection. Although I2C is in some respects not that different from SPI, the purpose of this proposal is to solely focus on I2C. The WASI-I2C proposal contains three WIT files adhering to the component model: delay, i2c, and world. The delay and i2c files are closely aligned with the interfaces from embedded-hal [33] v1.0, which served as inspiration, although the use of the embedded-hal Rust crate is not required to implement the proposal. The world file provides a way to import all the described interfaces of the proposal at once, thus currently importing all interfaces from the delay and i2c files. The proposal describes two handles that dynamically identify and provide access to resources, similar to a file descriptor. One handle is designated for I2C and the other for delays. These handles offer extensive access to the resource, meaning you currently cannot restrict the I2C connection to specific addresses. While a more fine-graded control model is feasible, consultations with the embedded special interest group (SIG-Embedded [34]) and industry partners have indicated that there is currently no demand for this. Because there exist various ways to obtain an I2C handle, the i2c interface lacks an explicit constructor for the i2c resource. Instead, worlds that utilise this interface should provide their own mechanism to obtain I2C handles, typically by passing handles into exported functions as parameters or by having imported functions return them. The i2c resource includes essential methods such as read and write to interact with the I2C device. Although delaying program execution is not inherently associated with I2C communication, certain guest implementations depend on it, which is why it is also included in the current proposal. In the future, delays could either be separated into their own proposal or merged with other embedded-related proposals to avoid the need for each delay-dependent proposal to establish its own standards. The main downside of merging proposals is that it could significantly increase the workload for the proposal champions, who would need to be ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 55 - February 28, 2025 interested and committed to maintaining these additional communication protocols. However, having centralised proposals could streamline the process for potential implementers. The WASI-I2C proposal is currently purely synchronous. An asynchronous API could be implemented today by creating a separate WIT API that depends on wasi-io [35] polling, but a better solution for asynchronous support is to wait for the component model to natively implement async. The reasoning behind this is that a single WIT description can describe both a synchronous and asynchronous API. Caller and callee can then each independently choose if they want to work synchronously or asynchronously. We identified that there is no immediate need for asynchronous support, therefore we will wait until native async support is in the component model. A key element of a proposal is its portability criteria. These criteria demonstrate that the champions' specification is not overly tailored to a specific platform. Typically, there need to be at least two entirely independent implementations in separate Wasm runtimes. However, given the diversity of platforms that can interface with I2C, this approach is not feasible for the WASI-I2C proposal. The champions, along with representatives from Siemens, Intel, Aptiv, Xiaomi, Amazon, Midokura, Sony, Atym, and Bosch, have collectively established a set of portability criteria for the proposal, as outlined in Table 9. To minimise the potential variety in memory size, flash size, and processor speed, reference hardware was provided for each criterion. Although the Intel x86 architecture is not explicitly mentioned, its support is implied since Wasmtime is being developed using this architecture. The NUCLEO-F412ZG was selected as it serves as Siemens' reference board. The remaining two targets were selected to demonstrate compatibility with both a real-time operating system (RTOS) and Linux, as well as ARM and RISC-V architectures. In addition to the target requirements, it is crucial that the interface is designed in a way that minimises its memory usage, ensuring that there is sufficient memory available for the application itself. Table 9: WASI-I2C Proposal Portability Criteria Platform Architecture Reference Hardware Linux ARM64 ESP32-C2 RTOS RISC-V ESP32-C2 RTOS ARM32 NUCLEO-F412ZG 7.1.4 USB Support in WASI The WASI-USB proposal is championed by IMEC and is currently in the first phase of the standardisation process. The current version of the standard is available in the wasi-usb GitHub repository [36]. We have two implementations showing this API in action. The first codebase USB_WASI tries to follow the standard proposal as close as possible and will be used going forward to continue the standard [37]. The second codebase contains a lot more features, including the demo shown below (Figure 19), where we run a rudimentary PacMan clone and a USB device driver in WebAssembly [38]. ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 56 - February 28, 2025 Figure 19. WASI-USB Proposal and Prototype Implementations: Demonstrating the API with a PacMan Clone and USB Device Driver in WebAssembly The WASI USB proposal adds an API for controlling USB devices. The API is meant to be used with any kind of USB device and is more low-level than, for example, accessing USB devices through the filesystem. The API design is based on the libusb [39] library, which is an often-used library for accessing USB devices in native programs. Just like libusb, this API aims to be a thin wrapper around the USB standard and Operating System APIs, with improved ergonomics where necessary. Access control can be applied to limit the devices a component can access. Support in web browsers is currently not considered. Research was done to evaluate the feasibility, but the conclusion was that while possible, the API would have too many limitations (based on the current support for USB devices in web browsers). The WASI-USB proposal contains a preliminary version of the USB API interface and is described using the WIT format. The key components of the proposal are outlined below. The usb interface contains a usb-device and a device-handle resource. The static enumerate function of the usb-device resource allows the caller to retrieve all available USB devices on the bus, returning them as a list of usb-device instances. Moreover, there are three other methods that can be used in any of these instances. The device-descriptor and configurations methods provide additional information about a device and are always available without requiring exclusive access to the device. Conversely, the open method is only available under specific conditions. This method establishes a context for device communication. It is important to note that opening a device can fail for various reasons. For instance, some operating systems may refuse to open a device if the executable lacks the necessary permissions. These permissions are managed at the Wasm runtime level, rather than being specific to individual Wasm modules. To develop a robust and user-friendly API, functions that require an ‘open’ device are scoped to the device-handle resource, which is only obtainable when the device is opened successfully. The device-handle resource encompasses all the methods necessary for device communication, with most methods closely related to similar libusb functions. The events interface allows a guest to poll for connection event updates. This allows the guest to detect “connect” and “disconnect” events and react accordingly. The events interface ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 57 - February 28, 2025 includes a single update function that returns a device-connection-event variant. The returned case of device-connection-event will vary based on the specific event that has occurred, for example pending when no new event update is available. With the anticipated async support in WASI Preview 3, the function can be adapted to receive event updates through an asynchronous stream rather than polling. The current WASI-USB proposal applies the capability-based security model exclusively at the device level. In this approach, only devices with specified vendor and product ID pairs are permitted. This is the most straightforward option and is similar to how WebUSB protects devices [40]. Community feedback will be essential to determine whether the capability-based security model should be extended to other levels, based on its perceived usefulness. Multiple components may attempt to access the same USB device simultaneously. However, this should be avoided, as each component typically expects exclusive access to a device. To address this, the runtime should monitor which components are using specific devices and prevent other components from accessing a device that is already in use. This can be achieved by returning an error when a component attempts to open a device that is already in use. This approach is consistent with most operating systems, where only one program can access a USB device at any given time. 7.2 Performance of Wasm for cyber-physical systems To analyse the overhead of cyber-physical applications in WebAssembly, we built proof of concepts that implement I2C and USB device drivers and multiple applications that interface with I2C and USB peripherals as WebAssembly components. 7.2.1 I2C performance in WASI We built two proof-of-concept implementations to evaluate I2C cyber-physical Wasm systems. The first one focuses on the Wasmtime runtime using the component model and the second one is implemented to work with the WAMR runtime. The reasoning behind these two separate proofs-of-concept is that discussions with industry and the community have shown that WAMR is the preferred choice for low-powered embedded use cases. We implement and evaluate two proof-of-concept applications for I2C: communicating with an I2C seven-segment display and reading temperature and humidity data from an HTS221 sensor. The proof-of-concept for the seven-segment display was tested on a Raspberry Pi 4B and a Raspberry Pi Pico microcontroller, both on Wasmtime and WAMR. Reading data from the HTS221 sensor was tested on a Raspberry Pi 3B on Wasmtime only. The proof-of-concept implementation includes a host implementation along with several guest implementations, i.e., communicating with the seven-segment display and reading data from the HTS221 sensor. 7.2.2 I2C implementation in Wasmtime The host implementation handles the I2C connection with the device, whereas the guests are responsible for device-specific logic, such as reading temperature or displaying time. The host and guest components interact through two WIT worlds, one for the display and the other for the HTS221 sensor. These WIT worlds import the necessary interfaces from the WASI-I2C proposal that are implemented by the host, allowing the guest to communicate with the I2C device. Additionally, the WIT worlds export device-specific functions, such as gettemperature, implemented by the guest. Although these worlds were written for our specific proofs-of-concept, they aim to be as generic as possible to allow future support for other ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 64 - February 28, 2025 Figure 26. Round-trip time for a 32-byte USB isochronous transfer across different platforms, comparing native and Wasm guest applications. Similarly to the results from the interrupt endpoint benchmarks, the latency is slightly lower on WebAssembly compared to the native API. ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 65 - February 28, 2025 8 Serverless Security Tools for ELASTIC Framework Serverless architecture frameworks provide substantial advantages across a wide range of applications and is supported by a diverse range of complex platforms and products. As a cloudnative execution model, serverless computing enables developers to design and deploy applications without managing the underlying infrastructure. However, despite its benefits in scalability and cost efficiency, serverless computing introduces distinct security challenges that need to be addressed through emerging technologies and research advancements. To address these challenges, the ELASTIC framework incorporates specialized hardwareand software-based security mechanisms with focus on enhancing data integrity, confidentiality, and workload isolation. Specifically, the proposed security tools strengthen the security of the ELASTIC orchestration and scheduling framework by implementing modern, robust access control and capability-based security mechanisms that mitigate potential attacks. All the described tools are integral components of ELASTIC and will be further detailed in the corresponding deliverables of WP1, WP2, WP3, and WP4. A concise overview of their characteristics and the security benefits they offer is provided in the following sections. 8.1 SW-based Mechanisms ELASTIC supports a wide range of software-based security solutions associated with data integrity, authentication, and intrusion detection. The mechanisms are integrated into different parts of the ELASTIC framework offering a high level of security for serverless computing environments. 8.1.1 Lightweight Security orchestrator for Edge devices The ELASTIC lightweight orchestration agent is designed to manage security functions in accordance with industry-specific security policies, ensuring the protection of services deployed across diverse execution environments, including cloud and edge computing infrastructures. The agent's inherent flexibility and adaptability facilitate the seamless integration of security policies, allowing for the dynamic definition and redefinition of security levels in response to evolving requirements. To address the constraints of resource-limited edge devices, the agent leverages Wasm runtimes and TEE technologies to enable the deployment of sophisticated security controls while minimising computational overhead. A key feature of this orchestration agent is its modular security function deployment, designed to accommodate the heterogeneity of edge devices, which vary in processing architecture, memory availability, and network connectivity. The integration of Wasm ensures platform-independent execution of security applications, thereby enhancing interoperability across diverse system environments. The agent also incorporates advanced scheduling algorithms that dynamically prioritise workloads based on security events and compliance with predefined security service level agreements (SLAs) tailored to specific industry verticals. To maintain execution integrity, it employs remote attestation mechanisms, ensuring that only authenticated Wasm workloads are executed within the secure enclave. Additionally, the agent collects real-time telemetry data, providing critical insights into system performance and security posture, which enables the continuous refinement of security policies. In scenarios where deviations from security policies are detected or potential threats emerge in monitoring logs, the orchestration agent autonomously triggers remediation processes, such as scaling security workloads, deploying additional security measures, or reconfiguring security ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 66 - February 28, 2025 controls. These capabilities collectively establish the agent as a self-adaptive, trust-based security framework capable of responding dynamically to evolving threats while maintaining operational efficiency in resource-constrained environments. This proactive and autonomous approach significantly enhances system resilience, ensuring robust and responsive security postures in dynamic operational landscapes. 8.1.2 eBPF code security Analyser The development of eBPF programs is often hindered by the complexity of diagnosing and resolving errors raised by the eBPF verifier. This challenge primarily comes from the lack of early detection tools, leading to the late identification of errors at load time rather than at compile time. Furthermore, the error messages generated by the verifier are difficult for programmers to interpret, as they pertain to eBPF bytecode rather than the original C source code. The eBPF code security analyser is a software tool designed to analyse eBPF C source code, identifying security vulnerabilities that may cause verification failures. By providing developers with early-stage, human-readable error messages, this tool significantly improves the debugging process. In addition, it proposes corrections directly related to the original C source code, thereby streamlining and accelerating code fixes. The tool will be integrated into the ELASTIC architecture as a development support utility, assisting programmers in writing secure and verifier-compliant eBPF code. In addition, the proposed tool will be particularly beneficial for various eBPF-based architecture components including eBPF distributed state synchronisation and accelerated microservices interaction. The key advantage of the static eBPF code security analyser within the ELASTIC ecosystem lies in its ability to enhance the eBPF development process by providing clear, early, and actionable feedback to developers. Unlike current eBPF development workflows, where error messages are often cryptic and delayed until execution, this tool ensures that programmers can efficiently diagnose and resolve issues at the source code level, improving both productivity and code security. 8.1.3 IoT Resource Allocation Model Against Mobility Attacks The emerging 6G mobile network offers novel services and capabilities, requiring robust mobility prediction for efficient resource allocation. However, adversarial mobility patterns can degrade prediction accuracy, leading to suboptimal network performance. To address this, the ELASTIC tool introduces a solution that publishes a mobility dataset containing legitimate and adversarial user equipment trajectories in 5G networks. Also, it provides an AutoML-based framework to identify optimal mobility prediction models while considering adversarial behaviours. The proposed solution automates data pre-processing, feature engineering, model selection, and hyperparameter tuning, ensuring robust predictions without significant performance loss. This component enhances core network security against mobility-based attacks while facilitating high-precision mobility prediction. Also, unlike the previous works, which focused on mobility prediction methods, the ELASTIC solution uniquely addresses adversarial resilience. By leveraging AutoML to retrain models against attack patterns, the network performance is safeguarded, offering telecommunication providers a novel, secure, and scalable mobility prediction framework. ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 67 - February 28, 2025 8.1.4 Lightweight Attribute-based Access Control (ABAC) solution This solution enforces secure attribute-based access control (ABAC) policies in Wasm orchestration, ensuring high security levels between Wasm containers and their environment. It introduces Policy Enforcement Points (PEPs) in the user plane, interacting with control plane components for authentication, authorisation, and auditing—aligning with NIST Zero Trust Architecture (NIST SP 1800-35) principles. Its main key features include the edge-optimised PEPs enforcing ABAC policies via a Policy Decision Point (PDP) in the control plane, the secure access control for Management and Orchestration (MANO) operations and Wasm containers' interactions with the Edge computing host and the integration with the ELASTIC workload identity framework, utilising attestationbased authentication before authorisation. The critical limitations that the proposed solution addresses are based on the fact that existing Wasm orchestration frameworks lack flexible ABAC policies or fail to support resourceconstrained Edge devices. The proposed tool ensures continued authorisation enforcement during DoS attacks, network disruptions, or infrastructure malfunction and it leverages ELASTIC Wasm workload identity and Wasm-specific attributes for fine-grained access control over container interactions, both locally and remotely. By integrating workload identity and attestation into Wasm-based Edge environments, this solution significantly enhances the security and resilience of the ELASTIC FaaS orchestration system. 8.1.5 Data protection at-rest at the edge with TEE solution The increasing deployment of Wasm workloads within TEEs necessitates robust mechanisms for securing data. ELASTIC will focus on developing an encryption component, ensuring the transparent interception and encryption of Wasm workload data within TEEs for secure communication. This tool is a novel encryption component designed to provide transparent data-at-rest protection for Wasm workloads operating in TEEs. The proposed solution ensures that encryption occurs seamlessly in the background, without requiring user intervention or application-level awareness. A critical security feature of this approach is the use of cryptographic keys stored within a dedicated hardware root of trust, thereby enhancing the reliability and robustness of the encryption process. Designed for edge environments, the component is optimised for low resource consumption and is implemented on the RISC-V architecture with Keystone as the preferred TEE. Addressing existing TEE limitations, the solution integrates tamper-resistant Secure Elements to mitigate side-channel attacks and support advanced encryption methods, such as Full Disk Encryption (FDE) and file-level encryption. Importantly, it extends Wasm’s security model without compromising its platform-agnostic nature. Seamlessly integrating with other ELASTIC project components, this contribution significantly enhances data security in edge environments, reinforcing cryptographic integrity and ensuring confidentiality in nextgeneration computing architectures. 8.1.6 WasmHAL-Trust: Automation tooling for confidential computing environments The ELASTIC Wasm HAL-Trust component is a key tool enabling secure and portable execution of Wasm workloads across different confidential container platforms. The tool facilitates architecture-agnostic and privacy-preserving execution of workloads in multi- ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 68 - February 28, 2025 stakeholder and multi-tenant environments. At its core, the Hardware Abstraction Layer (HAL) specification plays a crucial role in achieving these objectives by providing a unified and interoperable interface for accessing platform trust functions. The Wasm HAL-Trust component extends the WASI standard to support TEEs. This extended interface allows security-sensitive WebAssembly applications to interact with TEE functionalities in a standardised way. The component serves as a reference implementation of the Wasm HAL-Trust interface, providing a baseline for adapting and integrating TEE HALs across different ELASTIC-compatible TEE platforms. By enhancing WASI compatibility with TEE-specific security functions, i.e., secure migration, protected storage, and parameter sealing, the Wasm HAL-Trust component ensures interoperability across TEEs. As an essential part of the ELASTIC architecture, it supports trusted execution and contributes to the development, testing, and standardisation of secure WebAssembly computing environments. 8.1.7 WASI Security capabilities WebAssembly System Interface (WASI) is a standardised system interface designed to enable Wasm execution beyond the browser, ensuring portability, security, and consistency across server-side and edge computing applications. This work introduces a novel security adaptation framework for WebAssembly runtimes, facilitating standardised access control and policy mediation for Wasm components interacting with external resources. The proposed framework establishes a bridge between WebAssembly runtimes and higher-level security orchestrators, providing a consistent and enforceable mechanism for defining, interpreting, and enforcing component permissions. It introduces a policy assertion framework to specify access control rules governing Wasm component interactions, thereby enabling standardised security policy enforcement at runtime and improving interoperability across different Wasm environments. In contrast to existing Wasm runtimes such as Wasmtime and WAMR, which offer limited resource mediation and lack a unified security policy format, this work presents a runtimeagnostic plugin that standardises policy definition, interpretation, and enforcement. The framework comprises (1) a standardised format for policy assertion documents, (2) tools for creating and managing security policies, and (3) a plugin capable of integrating with multiple Wasm runtimes to dynamically enforce security policies. Last, this work introduces a universal security policy model, fostering cross-runtime compatibility, and enhancing security governance for WebAssembly workloads in heterogeneous execution environments. 8.2 HW-based Mechanisms ELASTIC incorporates two hardware security mechanisms: a hardware encryption tool and an artificial intelligence-based intrusion detection tool. These mechanisms are mapped on either reconfigurable platforms or GPU-based platforms, leveraging their high-performance capabilities and real-time processing efficiency. 8.2.1 AI Intrusion Detection component The ELASTIC AI intrusion detection tool leverages deep packet inspection with a focus on efficient network data pattern matching. It employs a hardware-based solution implemented on a reconfigurable hardware platform [47][48], enabling parallelised processing and high-speed interaction with external networks through an Ethernet interface. Performance has been further enhanced by deploying the tool on an advanced hardware platform, achieving greater parallelism and seamless integration within cluster-based environments. ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 69 - February 28, 2025 Moreover, the tool is integrated with additional technologies, such as eBPF, to support a broad spectrum of monitoring and performance optimisation tasks within the ELASTIC framework. AI frameworks are also incorporated into the hardware-based IDS, enabling advanced security through intelligent analysis and adaptive capabilities. The proposed tool provides significant advantages to the ELASTIC framework. Firstly, by utilising standalone hardware-based solutions, it effectively offloads the primary network infrastructure from the computationally intensive demands of security-related high-workload operations. Additionally, the hardware-based architecture enables real-time processing of realworld network workloads through a multi-pattern matching algorithm built on an advanced and innovative data structure. Lastly, the reconfigurable intrusion detection system (IDS) is designed to integrate seamlessly with cluster-distributed frameworks, enabling low-latency and real-time monitoring within the ELASTIC platform. 8.2.2 Hardware-based encryption accelerator The hardware-based cryptography module integrates security frameworks into the ELASTIC network, providing robust data protection through cryptographic and certification keys. The system is designed to process network workloads in real-time while offering adaptive security mechanisms that dynamically adjust cryptographic techniques to streaming data characteristics. The module features a security-enhanced core supporting advanced cryptographic operations, integrating multiple IP cores, like AES, for secure encryption, authentication, and data integrity. By utilising hardware acceleration and modern cryptographic primitives, the solution ensures high performance, scalability, and resilience, addressing ELASTIC's stringent security requirements. Developed for seamless integration into the ELASTIC framework, the module will use opensource platforms, such as OpenTitan, to enhance transparency and adaptability. Its deployment on FPGA-based systems ensures optimised performance for real-time operations, while integration with an AI-based Intrusion Detection System (IDS) enables a multi-layered security approach. The module is built for being tailored for 6G networks to meet ultra-low latency, high throughput, and advanced security demands. This approach enhances cryptographic robustness and aligns the ELASTIC framework with the requirements of next-generation communication systems. ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 70 - February 28, 2025 9 Evaluation metrics This section outlines the Key Performance Indicators (KPIs) related to Objective 2 of the ELASTIC project. These KPIs are critical for assessing the progress of the components covered by this deliverable. In Table 10, we provide the current status, the associated result, and a description for each KPI, explaining its purpose and outlining the approach we are taking to meet it. Work on all KPIs has already begun, and we have a well-defined roadmap in place for each one to ensure successful completion by the end of Task 2.1. Table 10: KPI Status Update for T2.1 Contributions Related to Objective 2 KPI 2.1 Reduce attack detection and analysis time by >=25% In progress R2.1 Low-latency, real-time monitoring of cluster-distributed eBPF/XDP and Wasm, with embedded low-power and hardware-based modules This KPI focuses on the performance of the cybersecurity algorithms within the ELASTIC framework by leveraging hardware-accelerated platforms. Security mechanisms need to be deployed on embedded low-power hardware solutions to enable real-time security assessment of the ELASTIC eBPF/XDP and Wasm frameworks. The integration of hardware-based security solutions offloads computationally intensive security tasks to dedicated hardware, ensuring real-time analysis and enabling a faster response to potential attacks while maintaining energy efficiency. Progress has already been made toward achieving this KPI. Specifically, a hardware-accelerated attack detection solution, implementing the Snort intrusion detection algorithm, has been successfully deployed on a hardware-based platform, namely the Xilinx FPGA Alveo using the AMD Alveo U50 Data Center Accelerator Card. This hardware solution leverages a hardware-based signature-matching algorithm to enhance the efficiency of intrusion detection processing. The next steps for fully covering the KPI will focus on analysing the performance of the hardwarebased solution and comparing it against the initial software-optimised implementation. In addition, the next step includes the developed hardware-based intrusion detection framework with the ELASTIC cluster-distributed frameworks, including eBPF and Wasm, to enhance system-wide security and performance. KPI 2.6 Orchestration agents use on average 50% less memory in a constantly-active scenario and 60% less memory in a wait-and-observe scenario, compared to standard non-serverless approaches. In progress R2.7 Development of an open-source framework to create memory-efficient serverless FaaS orchestration agent packages in architecture-agnostic WebAssembly modules. This KPI focuses on reducing the memory usage of the Kubernetes control plane, specifically to enable Kubernetes to be used for orchestration of serverless workloads on low-resource edge devices. This work is still ongoing and will be reported on in detail as part of D2.4. To meet this KPI, suspending control plane functions to disk is an important step. This functionality is already possible, but results in greatly increased control plane latency. Therefore, our current work is extending this platform to proactively predict when control plane functions will be required, to restart them right before they are needed. This reduces the latency overhead of suspending to disk. A second important area of work concerns support for realistic and existing Operators, to better understand the true memory savings in the field. To understand this better, we are improving the developer tooling around this framework and are increasing API support of our framework to make it easier to port and run existing Operators. Once these extensions are complete, we will perform a thorough performance evaluation to quantify and show the resulting performance improvements. ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 71 - February 28, 2025 KPI 2.7 Security evaluation of an application secured with the developed approach; publish at least one scientific conference or journal paper related to the solutions. In progress R2.8 Secure use of local data in distributed applications through the development of mechanisms that allow networked applications to securely process sensitive data while reducing or eliminating the risk of an attacker abusing the application to reveal data en masse from a participating endpoint This KPI focuses on evaluating the security benefits of our approach and publishing about the resulting security improvements. Although the bulk of our security work will be reported as part of D2.4, we have already shared preliminary results of improving the security and mediation of WASI APIs as part of Section 7. This has already resulted in an accepted conference publication to the IEEE NOMS 2025 conference [7.1]. Our ongoing work continues in this manner, by building a robust framework for both ACL and capability-based access control mechanisms in WebAssembly/WASI runtimes. This includes creating a metadata format to signal workload permissions from the orchestrator to the runtime, and multiple runtime plugins to enforce these permissions. To meet this KPI, IMEC will collaborate with AAL to provide a demo WebAssembly application and secure it with the approach. We are planning to publish a second paper, specifically a joint publication about this approach, the use case, and the security improvements. ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 72 - February 28, 2025 10 Conclusions and Next Steps In this first iteration of the lightweight and robust orchestrating mechanisms survey and development, we have focussed on the challenges of robustness and fault tolerance, integrating serverless with AI at the edge, connecting serverless workloads to external hardware, and proposing a number of security mechanisms for serverless workloads. Firstly, we built the Propeller orchestrator that allows robust deployment and management of WebAssembly workloads to the edge and IoT devices up to the microcontroller scale. We also built an initial version of the AI/ML workload orchestration engine focussed on confidentiality and edge support. Furthermore, we proposed two WebAssembly System Interface (WASI) standards, providing open-source implementations of these standards and a thorough benchmark evaluation that shows the minimal overhead of this approach. Finally, we investigated and proposed a number of security mechanisms to improve the security of serverless workloads across the cloud-edge continuum. Future work will continue on this approach by shifting the focus to the security of WASI interfaces, proposing both ACL-based and capability-based mediation of WASI APIs, and developing a standardised format to describe WASI security capabilities and policies per workload. Further security work will focus on analysing the performance of hardware-based low-latency monitoring of Wasm workloads and comparing it against software-optimised methods. Furthermore, we will thoroughly analyse the performance, robustness and user experience of the Propeller orchestrator, and update the platform to better meet the needs of holistic cloud-edge-IoT orchestration. All these innovations will be combined with the components from other WPs and be tested in the ELASTIC Demonstrator scenarios. ELASTIC D2.1 HORIZON-JU-SNS-2023/№ 101139067 ELASTIC - 73 - February 28, 2025 11 References [1]. M. Butcher, “The history and evolution of WebAssembly in Kubernetes.” Fermyon. [Online]. Available: https://www.fermyon.com/blog/history-and-evolution-ofwebassembly-in-kubernetes. [Accessed: Feb. 25, 2025]. [2]. T. Goethals, M. Sebrechts, M. Al-Naday, B. Volckaert, and F. De Turck, “A functional and performance benchmark of lightweight virtualization platforms for edge computing,” in Proc. IEEE Int. Conf. Edge Comput. Commun. (EDGE), Jul. 2022, pp. 60–68. doi: 10.1109/EDGE55608.2022.00020. [3]. T. Goethals, F. De Turck, and B. Volckaert, “FLEDGE: Kubernetes compatible container orchestration on low-resource edge devices,” in Internet of Vehicles. Technologies and Services Toward Smart Cities: 6th Int. Conf. IOV 2019, Kaohsiung, Taiwan, Nov. 18–21, 2019, Proc., Berlin, Germany: Springer-Verlag, 2019, pp. 174– 189. doi: 10.1007/978-3-030-38651-1_16. [4]. C. Carrión, “Kubernetes scheduling: Taxonomy, ongoing issues and challenges,” ACM Comput. Surv., vol. 55, no. 7, pp. 138:1–138:37, Dec. 2022, doi: 10.1145/3539606. [5]. “Platform overview | wasmCloud.” wasmCloud. [Online]. Available: https://wasmcloud.com/docs/concepts/. [Accessed: Feb. 25, 2025] [6]. “wasmCloud on Kubernetes | wasmCloud.” wasmCloud. [Online]. Available: https://wasmcloud.com/docs/kubernetes/. [Accessed: Feb. 25, 2025]. [7]. “Kubernetes as a platform vs. Kubernetes as an API | Containers.” Amazon Web Services (AWS). [Online]. Available: https://aws.amazon.com/blogs/containers/kubernetes-as-aplatform-vs-kubernetes-as-an-api-2/. [Accessed: Feb. 25, 2025] [8]. M. Sebrechts, T. Ramlot, S. Borny, T. Goethals, B. Volckaert, and F. De Turck, “Adapting Kubernetes controllers to the edge: On-demand control planes using Wasm and WASI,” in Proc. IEEE 11th Int. Conf. Cloud Netw. (CloudNet), Nov. 2022, pp. 195– 202. doi: 10.1109/CloudNet55617.2022.9978884. [9]. L. Larsson, W. Tärneberg, C. Klein, E. Elmroth, and M. Kihl, “Impact of etcd deployment on Kubernetes, Istio, and application performance,” Softw.: Pract. Exp., vol. 50, no. 10, pp. 1986–2007, 2020, doi: 10.1002/spe.2885. [10]. “WebAssembly vs. Kubernetes: Understand the relationship,” Search IT Operations, TechTarget. [Online]. Available: https://www.techtarget.com/searchitoperations/tip/WebAssembly-vs-KubernetesUnderstand-the-relationship. [Accessed: Feb. 25, 2025]. [11]. hybridgroup/mechanoid, “Mechanoid framework.” The Hybrid Group. [Online]. Available: https://github.com/hybridgroup/mechanoid. [Accessed: Feb. 15, 2025] [12]. bytecodealliance/wasm-micro-runtime, “Wasm micro-runtime.” Bytecode Alliance. [Online]. Available: https://github.com/bytecodealliance/wasm-micro-runtime. [Accessed: Feb. 25, 2025] [13]. “Overview | SuperMQ.” SuperMQ Documentation. [Online]. Available: https://docs.supermq.abstractmachines.fr/. [Accessed: Feb. 25, 2025] [14]. M. Ganguli, S. Ranganath, S. Ravisundar, A. Layek, D. Ilangovan, and E. Verplanke, “Challenges and opportunities in performance benchmarking of service mesh for the