WebAssembly Benefits and Application Scenarios at CERN

WebAssembly benefits and application scenarios at CERN AUGUST 2024 AUTHOR: Riccardo Carissimi IT-PW-PI SUPERVISORS: Diana Gaponcic Alberto Pimpo CERN openlab Report // 2024 PROJECT SPECIFICATION WebAssembly (often abbreviated to WASM) allows developers to write code in languages like C, C++, or Rust and compile it into a platform-independent binary format1. Support for WebAssembly in the cloud native ecosystem is a growing trend with multiple existing open source projects. WASM offers a powerful combination of cross-language compatibility, portability, and performance. This project will focus on evaluating the existing WASM ecosystem and projects and experimenting with a runtime for Kubernetes nodes at CERN, covering both self managed and OpenShift/PaaS deployments. 1https://webassembly.org/ WebAssembly benefits and application scenarios at CERN 2 CERN openlab Report // 2024 ABSTRACT This report explores the benefits and potential uses of WebAssembly (Wasm) at CERN. WebAssembly is a technology that allows code written in various programming languages to run efficiently and securely on different platforms, including web browsers, servers, and edge devices. Initially designed to improve web applications' performance, Wasm is now being considered for broader applications at CERN, such as real-time data processing on edge devices and GPU-accelerated tasks. The report discusses the evolution of WebAssembly, highlighting its advantages like fast execution, security, and cross-platform compatibility. It also compares different Wasm runtimes and their integration into containerized environments, which is relevant for deploying Wasm at scale. The report also covers practical aspects of using WebAssembly in a Kubernetes cluster. It provides a guide for setting up runwasi with containerd or configure crun properly to deploy WebAssembly applications. Additionally, the report compares WebAssembly runtimes like Wasmtime, Wasmedge, and Wasmer, and briefly discusses WasmCloud, a platform for distributed Wasm applications that can integrate with Kubernetes. Challenges noted include the current limitations of WebAssembly features and language support. Despite these, WebAssembly shows great promise for certain applications. It is not yet a complete replacement for traditional container runtimes in every situation, particularly for performance-critical tasks. However, its advantages in terms of portability, quick startup times, and security make it a valuable tool for specific use cases at CERN. WebAssembly benefits and application scenarios at CERN 3 CERN openlab Report // 2024 TABLE OF CONTENTS INTRODUCTION 6 NaCl 6 asm.js 6 WebAssembly 7 WASM Benefits 8 WebAssembly APIs 8 WebAssembly Use Cases 9 FaaS/Serverless Platforms 9 Embedded Sandboxing 9 High-Performance Web Applications 9 Machine Learning in the Browser 10 Porting Legacy Applications 11 Cross-Platform Development 11 Gaming 11 Potential use cases at CERN 11 Edge devices 11 GPU workloads 12 Container Runtimes Compared 12 Wasmtime 13 Wasmedge 13 Wasmer 13 Integrating WebAssembly 14 Using CRUN 14 Using runwasi 15 Runtime verification 16 Test deployment 16 CRUN 16 runwasi 17 Double-checking using strace 17 Node selection with RuntimeClass 18 WasmCloud 18 Setting it up 19 Pre-requisites 19 Installation 19 Deploying an application 19 WebAssembly benefits and application scenarios at CERN 4 CERN openlab Report // 2024 Rough edges we found along the way 20 Work in progress 20 Actual language compatibility 20 "Near" native performance 20 Performance Analysis 20 Execution modes 21 Standalone performance 21 Cold start evaluation 22 Final considerations 24 WebAssembly benefits and application scenarios at CERN 5 CERN openlab Report // 2024 INTRODUCTION In the paper that formally introduced the world to WebAssembly, the authors indicate that the motivation was about rising to meet the needs of modern, web-delivered software in ways that JavaScript alone could not2. The goal was to provide software that is safe, fast, portable and compact. These attributes are crucial for the performance and security demands of today's diverse and distributed computing environments. In order to provide software's portability, we need to consider two main issues: software is dependent on APIs (such as OpenGL, POSIX or Win32) that are not platform-agnostic. The second main issue is that software is going to run on different hosts, with diverse hardware and security capabilities. Until now, JavaScript has been the only way to solve these problems since it is ubiquitous, portable and provides some sense of security when run in a sandboxed environment. Unfortunately, there is a difference between JavaScript code running as a client compared to running it on a server. Its single-threaded design complicates long-running tasks or highly concurrent ones and due to its origins as a dynamic language, there are several classes of optimizations that are unavailable as options to even the fastest and most modern JavaScript runtimes. NaCl It was for these and other reasons that Google began to consider an alternative approach to safe, fast, and portable client-side web development. In 2011, Google released a new open source project called Native Client (NaCl). The idea was to provide near-native speed execution of code in the browser while running in a limited privilege sandbox for safety reasons3. The technology was successful in the sense that the performance demonstrated in the browser was only minimally off of native execution speeds. The problem was that the NaCl binaries would need to be generated and maintained for each target platform and would only run in Chrome. They also ran in an out-of-process space, so they could not directly interact with other Web APIs or JavaScript code. NaCl deserves a tremendous amount of credit for moving the industry in this direction, but it was ultimately too fiddly and too Chrome-specific to carry the open web forward. asm.js Mozilla's attempt focused on a subset of JavaScript that could be optimized ahead-of-time (AOT). Utilizing the LLVM-based clang frontend parser via the Emscripten toolchain, compiled C and C++ code could be efficiently converted into this JavaScript subset, making it portable and reasonably performant. However, the resulting file was significantly larger than its native counterpart due to the need to include standard library functions and browser-specific hooks, highlighting the difference between portable code and portable applications. Despite its limitations in performance compared to Google's NaCl/PNaCl, asm.js demonstrated the feasibility of deploying C and C++ applications in browsers with minimal code changes. It served as a 3Brian Sletten. WebAssembly: The Definitive Guide. O’Reilly Media, 2021. 2Andreas Haas, Andreas Rossberg, Derek L. Schuff, Ben L. Titzer, Michael Holman, Dan Gohman, Luke Wagner, Alon Zakai, and JF Bastien. Bringing the web up to speed with webassembly. SIGPLAN Not., 52(6):185–200, jun 2017 WebAssembly benefits and application scenarios at CERN 6 CERN openlab Report // 2024 significant stepping-stone, showing that optimized, sandboxed JavaScript code could be generated from various languages using LLVM-based toolchains. WebAssembly Developers across the web ecosystem recognized the need for a high-performance, sandboxed, and portable code solution that worked within existing web infrastructure. This culminated in the initiation of WebAssembly in 2015, led by Brendan Eich, the creator of JavaScript. WebAssembly was conceived as a "binary syntax for low-level safe code"4that could initially work with asm.js but eventually evolve to support multiple programming languages at a low level. WebAssembly (abbreviated Wasm) is a binary instruction format for a stack-based virtual machine. Wasm is designed as a portable compilation target for programming languages, enabling deployment on the web for client and server applications. You can take code written in over 30 different languages and compile it into a .wasm file, and then can execute that file in the browser, on a server, or even on a car. Despite a running joke in the WebAssembly community that WebAssembly is “neither web nor assembly,” the name conveniently suggests its functionality. WebAssembly serves as a target platform with a set of instructions that resemble assembly language. The fact that WebAssembly modules are frequently going to be delivered over the web as another type of URL-addressable resource justifies the inclusion of the word Web in the name. 4Brendan Eich. From asm.js to webassembly. https://brendaneich.com/2015/06/from-asm-js-to-webassembly/, 2015. Accessed: 2024-07-04. WebAssembly benefits and application scenarios at CERN 7 CERN openlab Report // 2024 WASM Benefits WebAssembly excels because of the following five characteristics5: ● Portable: The binary format for Wasm bytecode is standardized, meaning that any runtime capable of executing Wasm will be able to run any Wasm code. This is similar to Java’s promise of “write once, run anywhere”. ● Universal: Many languages can compile into Wasm. This support goes beyond systems languages like C, C++, and Rust to include garbage-collected, high-level languages like Go, Python, and Ruby. A full list of languages that compile to Wasm can be found here. ● “Near-Native Performance”: Wasm is often described as having “near-native performance”. What this actually means is that WebAssembly is almost always faster than JavaScript, especially for compute-intensive workloads, and averages between 1.45 and 1.55 times slower than native code6, but results do vary by runtime7. ● Fast Startup Time: The cold start time of Wasm is important enough that it warrants a category of its own. On the server, it can achieve 10-100x faster cold starts than Docker containers8because it does not need to create a new OS process for every container. In the browser, decoding Wasm and translating it to machine code is faster than parsing, interpreting, and optimizing JavaScript, and so Wasm code can begin executing at peak performance more quickly than JavaScript can. ● Secure: WebAssembly was designed with the web in mind and so security was a priority. Code running in a Wasm runtime is memory sandboxed and capability constrained, meaning that it is restricted to doing what it is explicitly allowed to do. While sandboxed, Wasm code can still be granted access to the underlying system, including system-level interfaces and hardware features. WebAssembly APIs The evolution of WebAssembly extended further with the introduction of the WebAssembly System Interface (WASI). WASI is a standard that provides WebAssembly with the ability to perform system-level operations, such as file and network access, in a secure and portable manner. By defining a set of system calls that are platform-independent, WASI enables WebAssembly applications to run outside the browser in various environments, including servers, IoT devices, and more. This extends the applicability of WebAssembly beyond the web, making it a versatile tool for developers across different domains9. This is the isolation mechanism used by the CRUN runtime when running WebAssembly applications on a Kubernetes cluster. Thanks to APIs like WebGPU and WebNN, WebAssembly can now also serve as a platform for GPU or NPU accelerated workloads, breaking its previous limitation to CPU workloads, such as those typically handled by the Java Virtual Machine (JVM). WebGPU and WebNN allow developers to write generic code capable of running on any hardware without depending on vendor-specific APIs like CUDA. This cross-hardware compatibility offers significant advantages, since it makes it possible to have the same container image running machine learning workloads being able to run on any kind of node, without being restricted to the GPU vendor or the CPU architecture. 9Wasi api docs. https://wasi.dev/. Accessed: 2024-07-08. 8Gackstatter, P. (2021). A WebAssembly container runtime for serverless edge computing [Diploma Thesis, Technische Universität Wien]. reposiTUm. https://doi.org/10.34726/hss.2021.85181. 7Frank Denis. Performance of webassembly runtimes in 2023. https://00f.net/2023/01/04/webassembly-benchmark-2023/. Accessed: 2024-07-10. 6Abhinav Jangda et al. Not So Fast: Analyzing the Performance of WebAssembly vs. Native Code. USENIX ATC 19, jul 2029. 5Harshal Sheth. Pay attention to WebAssembly. https://harshal.sheth.io/2022/01/31/webassembly.html. Accessed 2024-08-08. WebAssembly benefits and application scenarios at CERN 8 CERN openlab Report // 2024 WebAssembly Use Cases Originally designed to enable high-performance web applications, WebAssembly has evolved to become a versatile technology with a wide range of practical use cases10. In this section, we explore the practical applications of WebAssembly and how it is revolutionizing the way the web is built and experienced. FaaS/Serverless Platforms Function-as-a-service platforms need to execute user-provided code quickly and safely. Since serverless platforms are often used to run code for short durations, startup times are a particularly important metric. The ultra-fast cold starts and broad language support make Wasm a good choice for serverless workloads11. The CDN-edge computing platforms provided by Cloudflare Workers12 and Fastly Compute@Edge13 already provide the ability to run WebAssembly. Fastly claims 100x faster startup times than other offerings in the market, and attributes this speedup to their WebAssembly-based compiler and runtime. Embedded Sandboxing The idea of embedding WebAssembly within other applications is useful beyond plugin systems. In fact, it can be used to sandbox entire third-party libraries or to construct layers of security for first-party code. Firefox is leading the way14 in this area by protecting itself against bugs in third-party libraries, like the ones they use for spell checking or image decoding. In conjunction with a tool called RLBox, which provides a tainting layer, they can protect against vulnerabilities in those libraries without resorting to heavy-handed process isolation. High-Performance Web Applications WebAssembly allows developers to execute code at near-native speed in the browser, bridging the performance gap between web and desktop applications. By compiling high-level languages like C, C++, and Rust to Wasm, developers can achieve significant speed improvements for computationally intensive tasks, making complex applications run seamlessly in the browser. To give an idea, let's suppose we want to perform matrix multiplication efficiently on the browser15. We can do it by writing some simple code in C. 15 Trung Le. Unleashing the power of webassembly — practical use cases for a seamless web experience. https://link.medium.com/g7SKWfyOYKb, 2024. Accessed: 2024-07-05. 14 Bobby Holley. WebAssembly and Back Again: Fine-Grained Sandboxing in Firefox 95. https://hacks.mozilla.org/2021/12/webassembly-and-back-again-fine-grained-sandboxing-in-firefox-95/. Accessed 2024-08-08. 13 MJ Jones. How Compute is tackling the most frustrating aspects of serverless. https://www.fastly.com/blog/how-compute-edge-is-tackling-the-most-frustrating-aspects-of-serverless/. Accessed 2024-08-08. 12 Cloudflare docs - webassembly (wasm). https://developers.cloudflare.com/workers/runtime-apis/webassembly/. Accessed: 2024-07-05. 11 It’s worth noting that there are other alternatives. AWS Lambda, for example, uses the lightweight Firecracker MicroVM. In a head-to-head, WebAssembly does have some performance advantages. 10 Webassembly docs - use cases. https://webassembly.org/docs/use-cases/. Accessed: 2024-07-05. WebAssembly benefits and application scenarios at CERN 9 CERN openlab Report // 2024 Runtime verification Test deployment In order to verify the correct configuration of WebAssembly with CRUN, we deploy the following demo application. Save this file to deployment.yaml and run it in your cluster using kubectl apply -f deployment.yaml. Feel free to edit this deployment as desired. apiVersion: apps/v1 kind: Deployment metadata: name: wasm-deployment labels: app: wasm-test spec: replicas: 4 selector: matchLabels: app: wasm-test template: metadata: labels: app: wasm-test annotations: module.wasm.image/variant: compat-smart spec: containers: - name: wasm-test image: wasmedge/example-wasi-http ports: - containerPort: 1234 In order to run WebAssembly workloads on crun we need to add an annotation to Pods (module.wasm.image/variant: compat-smart) in order to instruct crun itself that containers in the Pod might need a WASM runtime to be executed. Alternatively, we can set a logical link called crun-wasm that points to crun as our runtime. This is equivalent to using the annotation module.wasm.image/variant: compat that executes all workloads as WASM ones by default. This will make newly instantiated linux containers not run on the cluster. We can verify the status of the running Pods with the appropriate command (kubectl get pods -o wide). We can also test the application by forwarding the appropriate port and fetching the demo webpage. kubectl port-forward deployment/wasm-deployment 8080:1234 curl localhost:8080 CRUN We want to verify that CRUN is actually running the containers in the nodes. In order to do it, we need to log in on a node in the cluster that is running a Pod related to the previously specified deployment. We can get the container ID of the running deployment by querying the running containers using sudo crictl ps -a. We can then look into the runtimeType field of that container using sudo crictl inspect CONTAINERID | grep "runtimeType". On our systems, we get that the runtimeType is io.containerd.runc.v2. This is a shim that calls CRUN in order to execute containers on the node. WebAssembly benefits and application scenarios at CERN 16 CERN openlab Report // 2024 runwasi Using runwasi we can get rid of the annotation, since we are not using CRUN. This is the deployment that we can apply to our cluster, once runwasi is installed. apiVersion: apps/v1 kind: Deployment metadata: name: wasm-deployment labels: app: wasm-test spec: replicas: 4 selector: matchLabels: app: wasm-test template: metadata: labels: app: wasm-test spec: containers: - name: wasm-test image: wasmedge/example-wasi-http ports: - containerPort: 1234 In order to verify that the correct runtime is being used, we can replicate the steps specified in the previous section. A low level WASM runtime should be set as the runtimeType, like io.containerd.wasmedge.v1. When running linux containers runwasi will default to youki, a low level runtime written in Rust26.This makes runwasi be able to execute both types of workloads, but unfortunately it cannot be changed with crun or runc. Also, since youki is embedded in the runwasi binary, it will not show up using strace. Double-checking using strace We can verify which runtimes are actually being called by looking at the system calls made by containerd. We can do it by performing sudo strace -f -p "pidof containerd" -o stracelog and looking at the execve in the strace file. For instance, we get the following when applying our deployment using the CRUN runtime. 8140 execve("/usr/bin/crun", ["crun", "--root", "/run/containerd/runc/k8s.io" , "--log", "/run/containerd/io.containerd.ru"..., "--log-format", "json", "create", "--bundle", "/run/containerd/io.containerd.ru"..., "--pid-file", "/run/containerd/io.containerd.ru"..., "6645bdee3357eecd5772b9d57d0b0cea"...], 0xc0001e8700 /* 13 vars */) = 0 On the other hand, this is what we get when using runwasi with wasmedge. 7131 execve("/usr/local/bin/containerd-shim-wasmedge-v1", ["/usr/local/bin/containerd-shim-w"..., "-namespace", "k8s.io", "-address", "/run/containerd/containerd.sock", "-publish-binary", "/var/usr/local/bin/containerd", "-id", "94f00e035351fc03dd358968ab2d911b"..., "start"], 0xc001986300 /* 14 vars */ <unfinished ...> 26 Unfortunately, this is not documented. You can find the relevant piece of code here: https://github.com/containerd/runwasi/blob/4e0e396bd117f5c0b90a90025abc9842beaa941e/crates/containerd-shim-w asm/src/sys/unix/container/executor.rs#L45-L68 WebAssembly benefits and application scenarios at CERN 17 CERN openlab Report // 2024 Node selection with RuntimeClass We can also enable WebAssembly or runwasi only on certain nodes in the cluster. We can label nodes accordingly using a predefined label. In the following example we use the label runtime=runwasi. kubectl label nodes NODE runtime=runwasi We can use the following RuntimeClass to enable the Pods to be scheduled only on those nodes. RuntimeClasses are a feature for selecting the container runtime configuration for running pods within the same cluster. By specifying a RuntimeClass in a pod's configuration, we can select which container runtime handler will be used to run that specific pod27. apiVersion: node.k8s.io/v1 kind: RuntimeClass metadata: name: wasmedge scheduling: nodeSelector: runtime: runwasi handler: wasmedge Another option is to not define a nodeSelector, thus enabling us to select the runtime that is going to run the workload without imposing node restrictions. apiVersion: node.k8s.io/v1 kind: RuntimeClass metadata: name: wasmedge handler: wasmedge WasmCloud WasmCloud is another approach to running WebAssembly workloads. Unlike other crun and runwasi, WasmCloud is a platform specifically designed for building distributed, actor-based applications. WasmCloud actors are the smallest deployable unit and they handle messages delivered to them by the host runtime via capability providers. Actors can also invoke exposed functions of capability providers which have been explicitly assigned to them. Capability providers on the other hand represent code which is not part of the core business logic, such as an HTTP server, or a message broker. While actors are compiled to WebAssembly, this is not required for capability providers28. WasmCloud includes lattice, a self-forming, self-healing mesh network using NATS that provides a unified, flattened topology. It integrates well with Kubernetes, but does not depend on it. It can be set up natively, but as of now the recommended way to run it is in a pod on a cluster. When considering whether to deploy WasmCloud, it is crucial to consider if the actual performance benefits are worth the switch to a new platform. It is true that it integrates well with Kubernetes, but 28 Kjorveziroski, Vojdan, Sonja Filiposka, and Anastas Mishev. "Evaluating webassembly for orchestrated deployment of serverless functions." 2022 30th Telecommunications Forum (TELFOR). IEEE, 2022. 27 Runtimeclass documentation. https://kubernetes.io/docs/concepts/containers/runtime-class/. Accessed: 2024-07-23. WebAssembly benefits and application scenarios at CERN 18 CERN openlab Report // 2024 transitioning or integrating it cannot be hassle free. As of the time of writing, WasmCloud does not support some of the tools that populate the Kubernetes ecosystem, like Istio29. Moreover, WasmCloud represents a different paradigm of deploying applications that may require training and time to adapt to. We argue that an accurate assessment of the actual benefits of moving to this new technology is needed, so that they can be counterweighted by the cost of transitioning itself. This complexity has the potential to be relevant for an organization like CERN. Setting it up Pre-requisites Before proceeding with the installation, we had to enable the required CSI drivers on our test cluster30. This might not be necessary on every cluster, based on the storage one might want to use. It is done by adding the following arguments when creating the cluster through openstack: --merge-labels --labels cinder_csi_enabled=True. Installation We followed the instructions provided by the WasmCloud team. You can find the version we used at the time of writing here. When installing NATS we had to set the StorageClass manually, since our cluster does not provide a default one. This can be easily done by overwriting the value when installing the NATS helm chart. To do that, just use the following command instead of the one provided in the instructions. helm install my-release-name path/to/chart \ --set config.jetstream.fileStore.pvc.storageClassName="STORAGE_CLASS" \ --set config.resolver.pvc.storageClassName="STORAGE_CLASS" Alternatively, we can also set the default StorageClass for the entire cluster, using the following command. kubectl annotate storageclass STORAGE_CLASS storageclass.kubernetes.io/is-default-class="true" Deploying an application We created a small demo project to test the cluster setup with WasmCloud. This demo serves as a simple example to ensure everything is working correctly and can act as a baseline for building more complex, production-ready applications in the future. By starting with this demo, developers can easily expand and adapt it to meet the needs of more sophisticated projects. Instructions can be found in the project's "Readme" file. 30 Kubernetes @ CERN Storage Docs. https://kubernetes.docs.cern.ch/docs/storage/. Accessed 2024-08-08. 29 Here you can find the list of integrations officially supported by WasmCloud. https://wasmcloud.com/docs/integrations/. WebAssembly benefits and application scenarios at CERN 19 CERN openlab Report // 2024 Rough edges we found along the way As we integrated WebAssembly into our workflows, we encountered a few challenges that highlighted some rough edges in the technology. Based on our expertise, these are not deal breakers and should be fixed in the years to follow. Work in progress WebAssembly is still a relatively young technology, and many of its features are under active development. This means that certain functionalities, like threading support or advanced debugging tools, are still maturing31. For instance, multi-threading in WebAssembly is possible, but it requires specific configurations and can be complex to implement correctly. These limitations can sometimes slow down development or necessitate workarounds that add complexity to projects. While the WebAssembly ecosystem is evolving rapidly, developers need to be aware that they might encounter unfinished or experimental features that could impact their workflows. Actual language compatibility WebAssembly supports a broad range of programming languages, but the level of maturity and stability varies significantly across them. While languages like C, C++, and Rust have robust support and produce highly optimized WASM code, others like Java, C#, and Python are still catching up32. The tooling for these languages might not yet provide the same level of performance, and certain language-specific features may not be fully supported. For instance, we found the support for Java to be very premature and not suited for a production ready project. This inconsistency can be a hurdle for teams looking to port existing applications or libraries to WebAssembly, as they might face limitations or need to rely on less mature tools, which can introduce bugs or inefficiencies. "Near" native performance WebAssembly is often praised for its "near-native" performance, but in practice, there are scenarios where this claim doesn’t hold up. While WebAssembly can achieve impressive speeds, it doesn’t always match the performance of native code. As we highlighted previously, it is almost always faster than JavaScript, especially for compute-intensive workloads, and averages between 1.45 and 1.55 times slower than native code33. Additionally, the lack of direct access to hardware features (like SIMD) in some environments can limit the potential performance gains. For tasks where performance is critical and the overhead introduced by WebAssembly becomes noticeable, traditional containerization technologies like Docker may still be preferred. Containers offer near-native performance and greater control over the execution environment, making them better suited for workloads that demand the highest levels of efficiency. Performance Analysis Several studies and benchmarks have been conducted to compare the performance of WASM runtimes, both in general contexts and within containerized environments like Kubernetes. In this 33 Abhinav Jangda et al. Not So Fast: Analyzing the Performance of WebAssembly vs. Native Code. USENIX ATC 19, jul 2029. 32 WASM programming languages support list. https://github.com/appcypher/awesome-wasm-langs. Accessed 2024-08-08. 31 WASM proposals list. https://github.com/WebAssembly/proposals. Accessed 2024-08-08. WebAssembly benefits and application scenarios at CERN 20 CERN openlab Report // 2024 section, we'll delve into the performance implications of running WebAssembly workloads on Kubernetes. Execution modes A Wasm module can typically be run in one of three compilation modes: Interpreter, Just-In-Time (JIT), or Ahead-of-Time (AOT)34. The decision depends on the user’s preference for resource usage, execution speed, etc. - AOT, which facilitates quick startup, exceptionally small footprint, and results in almost native speed. - Interpreter, which offers small memory usage, little footprint, and comparatively slow. - JIT, it allows to run WASM at almost native speed while maintaining a platform-neutral distribution. However, compilation might be expensive during runtime. Standalone performance In an earlier investigation, Jangda et al. attempt to understand the efficiency of Wasm binaries35. However, this study is focused solely on web browsers and does not provide sufficient insights into standalone runtimes. Another study by Wen and Weber introduces an operating system designed to run WASM applications in edge environments, comparing their results to native Linux execution36. They report improved execution speed over native code while retaining the security advantages of WASM. Building on this, Wang addresses the limitations of prior studies by offering a more comprehensive evaluation of standard WASM runtimes37. This research examines the five most widely-used independent WASM runtimes. The study also introduces a new benchmark suite, WABench, which includes tools from established benchmarks along with complete applications from various domains. The goal is to assess the performance efficiency of standalone Wasm runtimes. Wang explores the impact of JIT compilers, AOT compilation, and Wasm compiler optimizations on performance. The findings indicate that wasmtime, wavm, and wasmer, which utilize JIT compilation, outperform wasm3 and wamr, which rely on interpretation. Overall, compared to the native execution, the five standalone WebAssembly runtimes introduce an average performance slowdown of 1.67× (Wasmtime), 3.54× (WAVM), 1.59× (Wasmer), 6.99× (Wasm3), and 9.57× (WAMR), respectively. This work serves as a baseline for performance analysis due to its integration of a broad range of benchmarks, though it is limited to x86-based architectures and lacks multi-architecture support. In summary, the paper notes the following: the performance degradation of interpretation-based runtimes is consistent across both shortand long-running benchmarks. The Cranelift JIT demonstrates promising results in the JetStream, MiBench, and PolyBench benchmark suites. In all 37 W. Wang, “How far we’ve come–a characterization study of standalone webassembly runtimes,” in 2022 IEEE International Symposium on Workload Characterization (IISWC), pp. 228–241, IEEE, 2022. 36 E. Wen and G. Weber, “Wasmachine: Bring the edge up to speed with a webassembly os,” in 2020 IEEE 13th International Conference on Cloud Computing (CLOUD), pp. 353–360, IEEE, 2020. 35 Abhinav Jangda et al. Not So Fast: Analyzing the Performance of WebAssembly vs. Native Code. USENIX ATC 19, jul 2029. 34 Kakati and Brorsson, “WebAssembly Beyond the Web.” in 2023 3rd International Conference on Intelligent Technologies (CONIT), pp. 1-8, 2023. WebAssembly benefits and application scenarios at CERN 21 CERN openlab Report // 2024 benchmarks, WAVM exhibits higher memory usage, and in general, WASM runtimes consume more memory than native applications. Spies and Mock examine the performance and usability of WASM in three non-web environments: desktop, server, and IoT devices38. The authors conclude that Wasm has potential for use in non-web environments, especially in resource-constrained settings where it offers a lightweight alternative to native code. Cold start evaluation Comparative analysis of performance differences in terms of the initial start up delay between traditional Docker containers and WebAssembly modules has been done by Long et al.39, finding that WASM starts up 10 times faster than the same workload running in a containerized environment. Kjorveziroski et al.40 compares the cold start latency and execution speed of WebAssembly modules to that of containers in cases where the WebAssembly runtime is integrated with the interface of a container runtime. This is done by analyzing the cold start delays and total execution times of three WebAssembly runtimes: WasmEdge, Wasmer, and Wasmtime to the performance of the containerd container runtime, using distroless and distro-oriented container images. The research also introduces a new benchmarking suite41 comprised of 13 different sequential applications, representing both the functionality of common serverless functions as well as more theoretical microbenchmarks. Each benchmarking function has been evaluated with all runtimes using both JIT and AOT compilation. Results are compared to distroless and distro-oriented container images. The results indicate that WebAssembly runtimes generally exhibit superior execution times in most tests (10 out of 13), when comparing WASM AOT execution to the performance of distroless and distro-based container images. 41 WASM Serverless Benchmarks. https://github.com/korvoj/wasm-serverless-benchmarks 40 Kjorveziroski and Filiposka, “WebAssembly as an Enabler for Next Generation Serverless Computing.” in Journal of Grid Computing, pp. 34, 2023. 39 Long, J., Tai, H.-Y., Hsieh, S.-T., Yuan, M.J.: A lightweight design for serverless function-as-a-service. IEEE Software vol.38(1), pp. 75–80(2021). https://doi.org/10.1109/MS. 2020.3028991 arXiv:2010.07115 [cs] (2019) 38 B. Spies and M. Mock, “An evaluation of webassembly in non-web environments,” in 2021 XLVII Latin American Computing Conference (CLEI), pp. 1–10, IEEE, 2021. WebAssembly benefits and application scenarios at CERN 22 CERN openlab Report // 2024 However, for sustained compute performance in complex tasks like image processing or matrix multiplication, additional improvements are necessary for WebAssembly runtimes. In contrast, container runtimes currently deliver better performance due to their higher compute capabilities, though they suffer from cold start issues. The potential impact of multi-threading support on WebAssembly performance is still uncertain and will only be clarified once it becomes a stable feature. When analyzing JIT compilation, Wasmtime demonstrated the best performance in 8 out of the 13 cases. However, it is important to note that WasmEdge was not included in the JIT comparison due to its reliance on interpretation rather than true JIT compilation. Additionally, Wasmer was excluded from several tests that require I/O access to the underlying filesystem. Indeed, even though Wasmer does support WASI, it was not possible to execute these functions with crun’s WebAssembly integration, limiting its participation in the benchmarking. Distroless and distro-oriented container images, included in the comparison, do not use JIT compilation, thus serving as a static performance baseline for comparison. WebAssembly benefits and application scenarios at CERN 23 CERN openlab Report // 2024 Among the three WebAssembly runtimes analyzed, Wasmtime, when integrated with containerd, provided the best performance in 8 out of the 13 cases. WasmEdge followed, excelling in 4 cases, while Wasmer led in just one test. When considering JIT compilation, Wasmtime was able to be 36.4% faster than distro-oriented container images and introduce a small 26.9% overhead over distroless images, both compiled ahead-of-time. Final considerations WebAssembly stands out with its significantly faster cold start times compared to container runtimes and much smaller artifact sizes, which are at least an order of magnitude smaller than OCI container images. However, WebAssembly is not yet a complete substitute for container runtimes in serverless environments, as containers continue to offer superior sustained compute performance, essential for more resource-intensive tasks. By integrating WebAssembly runtimes with the APIs and CLIs of existing container runtimes it is possible to achieve the best of both worlds, while reusing the existing mature tooling and orchestration tools developed primarily for container based workloads. This approach addresses the orchestration challenge for server-side WebAssembly execution without needing entirely new solutions, instead relying on existing components. WebAssembly benefits and application scenarios at CERN 24