A Methodology for Reproducible and Portable Experiment Workflows

A Methodology for Reproducible and Portable Experiment Workflows Henning Stubbe1, Sebastian Gallenmüller1,∗ , Georg Carle1 TUM School of Computation, Information and Technology, Technical University of Munich, Boltzmannstr. 3, 85748, Garching near Munich, Germany Abstract Testbeds allow the creation of research prototypes to test new ideas through practical experiments. This central role in validating ideas makes them irreplaceable tools for data-driven research in computer science. Various testbeds were created to provide testbeds for the scientific community. To simplify testbed usage, frameworks help to authenticate users, allocate resources, and run experiments. Each testbed typically implements its own framework using a specific API to realize experiments. Such an experiment design impedes the portability of experiments between different testbeds. In this paper, we present a solution where we port the pos experiment controller to the Chameleon and CloudLab testbed. The well-structured pos experiment workflow allows the creation of inherently reproducible experiments. Previously, the experiments using the pos workflow were only possible in dedicated testbeds. By introducing the portability feature, these experiments can run on Chameleon and CloudLab. We demonstrate that experiments can be executed on any of the mentioned platforms without changing the experiment definition. Based on these results, we discuss how the portability feature will be used in the upcoming SLICES-RI testbeds to create reproducible and easily-shareable experiments. Keywords: Repeatability, Reproducibility, Network Experiment, Portability, plain orchestrating service (pos) This publication is an extension of the paper „The pos Experiment Controller: Reproducible & Portable Network Experiments“ originally published at the 19th Wireless On-demand Network systems and Services Conference (WONS’24) [1]. 1. Introduction Computer networks and connectivity to the Internet have found their way into our everyday lives, where they have become a commodity for many products and services. Industrial manufacturing, the health sector, and transport systems have developed into interconnected systems relying on the implicit availability of network connectivity. Such services expect real-time or near real-time communication for a smooth and reliable operation. At the same time, networked systems have reached a level of complexity that requires a thorough understanding to ensure that networks provide connectivity with the required quality. The complexity of network systems arises from a combination of multiple layers of (virtualized) software in end hosts and programmable network components such as switches or smart network interface cards (NICs) or complex connected hardware components. ∗Corresponding author Email addresses: [email protected] (Henning Stubbe), [email protected] (Sebastian Gallenmüller), [email protected] (Georg Carle) Virtualization, simulation, and emulation are tools to recreate real, complex systems as a target platform for research in a cost-efficient way. However, recreating such complex environments is a highly challenging task, especially the interplay of all the different hardware and software network components. Minor differences between the original and the recreated system may impact the measurement results, especially in the real-time domain where fractions of a second matter. Hardware testbeds help solve this problem, allowing the recreation of systems based on bare-metal hardware. Research based on such a testbed can use bare-metal hardware and the same software stack as the original systems, ensuring a realistic delay behavior. Emulated, simulated, or virtualized systems can behave differently compared to the original systems [2]. The reason for the different behavior are effects that may be hidden, intentionally or unintentionally, from the experimenters, such as the delay of network connections or undocumented features / bugs of hardware components. A hardware-based testbed will be subjected to the same effects as the original systems, increasing the confidence in the results gained. If effects are well understood, simulation and emulationbased research platforms can be powerful tools for creating experiments with a high number of participating entities cost-efficiently. To gain this understanding, multiple testbeds were created that offer bare-metal hardware access [3, 4, 5, 6]. Despite offering similar functionality, all testbeds typically require experimenters to use a Management Node Experiment Controller Testbed Orchestrator Experiment Node Experiment Node Experiment Node (a) Step 1: Testbed orchestrator preparing experiment nodes Management Node Experiment Controller Testbed Orchestrator Experiment Node Experiment Node Experiment Node (b) Step 2: Experiment controller performing experiment Figure 1: Two components of the management node testbed-specific API for their experiments. The different APIs limit the execution of experiments to a specific testbed, preventing simple portability of experiments between testbeds. Consequently, creating portable experiments requires additional effort from the experimenters to make their experiments compatible with the API of other testbeds. This lack of native portability severely limits the reproduction of experiments across testbeds. In this work, we want to demonstrate a novel approach to make experiments portable. Our approach uses pos [3], a framework with a focus on the creation of reproducible experiments. The pos framework implements the API that is used by experiments. To make the experiments portable, this API must be supported by other testbeds. We achieve this by porting the pos framework to other testbeds. To test the feasibility of our approach, we demonstrate that an unmodified experiment workflow can be executed on other testbeds such as CloudLab [4] and Chameleon [5]. The paper is structured as follows. In Section 2, different testbeds are presented with a focus on their design and usability for experimenters. Section 3 presents the high-level design of our approach. In Section 6, we discuss how the presented approach can be used in future testbeds. The artifacts published in addition to the paper are explained in Section 7. Section 8 concludes the paper. 2. Background and Related Work In this section, we investigate different aspects of current testbeds, focusing on portability and reproducibility. Testbed orchestration vs. experiment control. We identified two components necessary to successfully operate a testbed: The first component, the testbed orchestrator, handles low-level operations to manage or orchestrate the resources of a testbed. This testbed orchestrator prepares and configures the environment to run the experiments. The orchestrator typically offers an inventory of available resources that helps researchers identify relevant resources. A common feature in multi-user environments is the reservation of testbed resources, e.g., via a calendar. Based on the available resources and their reservation, the orchestrator initializes the specified resources and prepares them to become ready for experimentation. The second component, the experiment controller, is responsible for high-level tasks to control experiments. The main task of this component is the execution of the experiment workflow in a prepared environment. Typically, the experiment controller expects a particular structure, or even a domainspecific language (DSL), to define experiments. The experiment controller can utilize resources prepared by the previous orchestration steps to perform experiments. Based on the definition of the experiment workflow, experiments can involve the creation of measurement results, evaluations, or plots. The separation between testbed orchestrator and experiment controller is shown in Figure 1. To make experiments reproducible, the orchestrator needs to create the state of the experiment environment reproducibly. Portability is ensured if an orchestrator creates the environment for the experiment controller across different platforms and the experiment controller performs the experiment according to a defined workflow. The following overview uses the previously introduced concepts to categorize existing approaches for testbed orchestration and experiment control. Reproducibility. The Association for Computing Machinery (ACM) defines reproducibility as a three-stage process [7]: The first stage, repeatability, is reached if the same team obtains results on the same experimental setup; the second stage, reproducibility, is achieved if a different team recreates an experiment using the same experimental setup; in the third stage, replicability, a different team uses a different experimental setup to recreate an experiment. Testbeds mainly target the experimental setup. Therefore, stages one and two can be achieved through technical measures [3]. Experiments become repeatable for researchers if the testbed ensures that experiments are executed fully automated and all experiments start from a well-defined state. By sharing access to a common experiment setup, e.g., a testbed, other researchers can reproduce experiments. Technical measures cannot ensure the creation of an independent experiment setup through a different research group. Therefore, a testbed cannot ensure reaching third-stage replicability. However, by creating and publishing well-documented experimental artifacts, experimenters can support potential replications. 2.1. Testbed Overview Testbeds come in a wide variety, reflecting the different requirements of research domains. At the same time, testbed operators are conscious of the existence of other testbeds and, hence, strive to join forces for a common goal. These joint works resulted in successful cooperation, such as GENI [8], whose successor FABRIC [9] recently took over and has enabled access to experiment resources across a number of testbeds, including Chameleon and CloudLab. Similarly, on the European side, there is ongoing work towards providing not only individual testbeds but also opening access to different testbeds to researchers. One 2 manifestation of this work was Fed4FIRE [10]. A federation of multiple European testbeds. A more recent project that plans to establish a more tightly integrated European testbed is SLICES-RI [11]. Connecting testbeds. A common theme between CloudLab, Chameleon, and Fed4FIRE [10] testbeds is their spatial distribution. The CloudLab and Chameleon testbeds are distributed across different sites. Typically, organizations that contribute to the operation of a testbed operate their own on-site testbeds, leading to the distributed structure. From the experimenter’s point of view, all sites can be managed via the same API. The common API requires little to no work from the experimenter when moving the experiment between different sites. The Fed4FIRE testbeds follow a more open approach for collaboration between different testbeds. Fed4FIRE offered three levels: (1) association, (2) light federation, and (3) advanced federation. Associated testbeds are not required to provide any shared API or tooling. The two more advanced levels use a shared solution to authenticate users. The highest level of federation provides a common set of basic services for experimenters across testbeds. Depending on the association/federation level, experimenters have to use different APIs. This diversity of APIs increases the complexity of porting experiments between different testbeds. Existing approaches for testbed orchestration. The genilib [12] is a library developed by the GENI initiative that was created to provide an interface to orchestrate different testbeds, such as CloudLab [4]. Testbed users use the API of this library to orchestrate the testbed. The Chameleon testbed [5] uses OpenStack [13] to manage the testbed resources. Users can use the OpenStack APIs to orchestrate this testbed. pos [3] is a testbed framework designed to create reproducible experiments. Using live Linux images ensures that any machine state is erased on reboot. In addition, pos users need to fully automate the orchestration process for a reproducible setup. Orchestration is handled via a command line interface (CLI), i.e., any application that can utilize the CLI will be able to use the pos orchestrator. Existing approaches for experiment control. Along with the continuous need for experiments comes the community’s striving for tools to simplify their handling. One representative of this is the Common Workflow Language (CWL) [14]. This programming language provides scaleable tooling for automated execution of experiments, implementing two standards: 1. details on available tools with their inand outputs and 2. description of composition rules for known tools. CWL has several implementations using various platforms such as SSH-accessible systems, Kubernetes, AWS, or Azure to run experiments. It is a popular choice for processing data-driven experiments in life sciences [15]. CWL is well-suited to run purely software-based experiments that do not require specialized hardware or network topologies. While also focusing on the execution of experiments, the cOntrol and Management Framework (OMF) [16] approached this topic differently. Rakotoarivelo et al. developed a central controller accepting jobs from experiments to execute on the testbed’s hardware. Using a Ruby-based DSL to describe experiments, OMF produces experiment outputs available via SQL or HTTP interfaces. NEPI [17] is a platform that provides different backends to execute experiments: a physical testbed, a network emulator, and a network simulator. The NEPI framework provides an abstraction layer on top of its supported backends utilized by the experiments. Experiments use a Python API to control the experimental workflow. In recent years, Chameleon integrated services for experiment control [5]. The preferred interface for experiment specification are Jupyter notebooks and Chameleon’s Jupyter hub; to store these experiment artifacts, researchers are encouraged to use Trovi. Jupyter notebooks aggregate all steps of the experiment workflow. Researchers are free to structure the experiment workflow according to their personal preferences. The pos framework [3] also integrates a component for experiment control. It relies not on a specific API or library to define experiments. The user writes scripts executed on the allocated experiment nodes, and the scripts can be started via the pos CLI from the management node. Benefits of the pos approach. In contrast to the previously mentioned frameworks, pos does not require users to learn and apply APIs or DSLs to orchestrate testbeds or describe experiments. Users can express the steps for orchestration and experimental control using any programming language or software. Interaction with pos is kept minimal and handled via a CLI that can be utilized by experimenter-defined scripts. This ensures a high degree of freedom for its users while pos’ properties ensure a reproducible execution of experiments. 2.2. Investigated Testbeds For this paper, we want to focus on a specific type of testbed, targeting a similar domain and offering off-theshelf server hardware: Chameleon, CloudLab, and pos. All three offer low-level access to resources, therefore, a high degree of freedom for the experimenter. 2.2.1. Chameleon Operating since 2015, the Chameleon testbed [5] provides an experiment platform for researchers. Designed for a broad set of experiment domains, this testbed provides networked compute on a heterogeneous device set. The available hardware is listed on the testbed’s webpage. The desired resources, e.g., networks and machines, can be reserved for a given interval, assuming sufficient availability. Another limiting factor when reserving is that resource allocation requires a virtual currency; each project accepted on the testbed has a limited amount of said currency to spend. During the reserved interval, researchers can configure their resources, e.g., the disk image to be used or 3 how nodes are connected, and conduct their experiments. A recent addition to Chameleon is Trovi, a service hosting Jupyter-notebook-based experiments and artifacts. The motivation for Trovi and Jupyter notebooks is the attempt to provide a repository for collecting, publishing, and sharing different experiments. Other researchers can easily access Trovi to reproduce existing experiments or to derive their own experiments. 2.2.2. CloudLab CloudLab [4] is built on Emulab [18], a purpose-built testbed management software. Designed as a federated testbed, CloudLab spans across multiple sites in the USA. While sites vary in the kind of offered hardware, there is a tendency towards commercial off-the-shelf servers. Each type of hardware is generally available several times, thus enabling experiments among the same kind of hardware. Similar to Chameleon, connections between machines are implemented via testbed-configurable data center switches. As with servers, the kind of switch differs between sites. Experiment setups in CloudLab can be configured either via an XML description or via a Python script, which, based on provided libraries, allows the generation of this XML description. This setup description includes, e.g., the number of machines required, their type and disk image, or the intra-experiment links. Additionally, optional information, such as experiment documentation or commands to execute on boot, can be configured this way. The resulting experiment description can be shared with other researchers and simplifies recreating the same experiment setup as needed. To conduct an experiment, researchers can either search for a suitable time slot by entering their requirements or try to immediately begin experimenting. In the latter case, though, more evolved reservations are more likely to be infeasible, as many resource types are well-utilized. Part of the experiment configuration associated with a reservation is, e.g., the disk image to use on the reserved nodes. Once allocated for the researcher, machines will be configured as instructed; this configuration commonly includes tasks such as deploying SSH keys or mounting a persistent NFS share. Before the allocation terminates, the researcher may, provided sufficient availability, request an extension of their experiment. After the experiment, nodes are reimaged with a default configuration and made inaccessible. 2.2.3. pos: Dedicated Deployment The pos framework consists of two components: a testbed orchestrator and an experiment controller [3]. During the past years, pos-managed testbeds were used for research and teaching at the Technical University of Munich. Only a virtualized testbed has been publicly available, with a limited amount of resources. To organize access to testbed resources between multiple researchers, the pos framework supports a calendarbased resource reservation and allocation. Researchers can note their intention to use resources during any interval. In their reserved interval, experiments can be conducted by applying the pos methodology. That is, researchers may allocate their reserved nodes, configure arbitrary images to be live-booted, and make use of other features of the pos framework to implement their experiment. 2.2.4. pos: Virtualized Deployment The virtualized deployment of pos considered in this work is similar to the dedicated deployment (cf. Section 2.2.3). The only difference between virtualized and dedicated deployment is the type of the experiment nodes: the dedicated deployment uses bare-metal servers; in contrast, in the virtualized deployment, virtual machines (VMs) are used. Each physical machine present in the dedicated deployment is represented by a VM running; all VMs run on a single physical host. One benefit of this approach is the reduced number of physical machines required to host the testbed and perform experiments [3]. 3. Analysis & Design In this section, we want to briefly discuss the drivers that shaped today’s testbed landscape. Based on these drivers, we deduct the requirements for portable and reproducible experiments. Finally, we suggest a high-level design to meet our requirements. 3.1. Requirements Analysis Initially, we analyze the requirements for testbeds with a focus on the aspects of portability and reproducibility. Requirement (1): Low-effort reproducibility. To foster the way toward more reproducible research, the ACM introduced badges that can be awarded to papers for providing artifacts for their papers [19]. Artifact Evaluation (AE) Committees evaluate the artifacts and award different badges according to the quality of the artifacts and their documentation. A survey among AE participants, paper authors, and artifact evaluators found the evaluation process useful but time-consuming [19]. Testbeds, such as Chameleon and CloudLab, were proposed as resource providers for artifact evaluators [20, 21]. The creation of portable experiments can help lower the effort for the evaluation process. Authors and reviewers that evaluate portable experiments can use different testbeds to run experiments, and having a shared platform to run experiments simplifies debugging. Testbeds such as Chameleon and CloudLab have already shown their commitment to the long-term availability of the platforms. This ensures that artifacts can be reproduced for a longer term, whereas individual researchers or research groups may not be able to provide this long-term commitment. 4 Requirement (2): Low-effort portability. Porting experiments between testbeds is highly attractive. Testbeds offer different hardware platforms. By supporting a portable experiment format, researchers can profit from this hardware diversity by running their experiments on multiple platforms. The current testbed landscape also allows this. However, the effort is higher if the experiment has to be adapted to the different APIs of the different testbeds. Requirement (3): Portability for international collaboration. Sharing testbed resources internationally can be challenging if research is funded by different countries. USbased testbeds, funded by US taxpayers, naturally prefer US-based research. The same is true for European research. Though cross-country access to testbeds is possible, we used it ourselves for this paper, the amount of resources offered to international guests may be limited. Portable experiments can help solve this issue. Researchers can exchange their experiments and execute them on testbeds that fit their funding regimes. Requirement (4): Experiment API. Testbed users rely on the stability and availability of APIs for their experiments. Removing the traditional APIs will alienate long-term users familiar with the testbed-specific APIs. At the same time, removing traditional APIs will break existing experiments that can no longer be executed on these testbeds. Converting testbeds to a new API will lead to the same problems. Therefore, removing APIs or converting to a new API are unlikely to happen for established testbeds. The only solution that remains is the addition of new APIs while retaining backward compatibility for older experiments. However, adding new APIs typically requires the support of the testbed operator, who needs to implement and deploy software that offers the new APIs. This additional effort for the operator can be a reason why new APIs are rarely added. 3.2. Requirements vs. existing testbeds Designed with reproducibility in mind, pos already integrates features that meet Requirement (1). To ensure reproducibility, the pos experiment workflow includes all scripts necessary to run and document an experiment. The pos workflow requires the experimenter to integrate code that automates experiment configuration, execution, and evaluation. This ensures completeness of the experiment description, which is a necessary precondition to ensure portability, i.e., Requirement (2). Requirements (3) and (4) are not yet addressed by pos in its current form. In the following, we suggest and implement an approach that solves these issues. Therefore, we extend pos to deploy the pos API to other testbeds (cf. Requirement (4)) without creating the deployment overhead for testbed operators. As a result of this work, the pos API proposed by Gallenmüller et al. [3] becomes available outside of traditional pos-powered testbeds. In our approach, we avoid any overhead for testbed operators due to the execution of pos-based experiments. To address Requirement (3), international collaboration, we selected the US-based testbeds CloudLab and Chameleon as target platforms to run the portable pos experiments. CloudLab and Chameleon also provide the means to create reproducible experiments. However, the testbeds can be used to create reproducible experiments, but they do not ensure reproducibility [22]. In pos, resetting the systems before each experiment and the requirement to automate all steps in the experimental workflow enforce the creation of reproducible experiments—a property that we call reproducibility-by-design. Whereas CloudLab and Chameleon experiments can be created in a way that fulfills Requirement (1), pos inherently supports it, making pos an advantageous platform to implement the other requirements. 3.3. High-level Design The high-level architecture of pos follows a two-layer design, consisting of the testbed orchestrator as the lower layer and the experiment controller running on top (cf. Figure 1). The testbed orchestrator contains all testbedspecific functionality; the experiment controller only utilizes the interface provided by the testbed orchestrator. To achieve portability, only the testbed orchestrator needs to be modified when porting the pos framework to other testbeds; the experiment controller remains unchanged. With the experiment controller left untouched, experiments using the pos experiment controller do not need to be modified. Preserving the experiment scripts reduces the workload for experimenters, as they do not need to adapt their experiment code to new testbeds. At the same time, the reproducibility-by-design is conserved, as testbed orchestration and experiment controller ensure that all properties of the pos methodology are maintained. Also, apart from portability, another benefit of our approach comes to mind: experiment metadata. To automatically enrich experiment results with information about the experiment environment, e.g., network topology and hardware involved, pos captures and stores said information for each experiment conducted. While performed automatically for scenarios where pos acts as testbed orchestrator and experiment controller, this capability is extensible to other testbeds, providing the required information in a machine-readable fashion. Looking at commonalities between scientific testbeds, we find that such settings feature in particular: 1. a set of entities available for experiments, 2. a management facility to control entities available for experiments, and 3. optional additional services providing supplementing functionality to users, e.g., data storage for experiment results. Based on the premise of limited assumptions, only the former two can be considered when designing the pos experiment controller. However, the use of pos should not prevent the use of features present in the hosting testbed. A high-level overview of the architecture of the pos framework is depicted in Figure 2. 5 pos Management Node Experiment Node Experiment Node Experiment Node Hosted Testbed Experiment Experiment Management Node Additional Service Additional Service Additional Service Hosting Testbed Figure 2: Deployment of pos controller inside other testbeds There, a differentiation between the hosting and the hosted testbed is made. In this work, hosting testbed refers to the testbed that provides the physical infrastructure and means to orchestrate it. Moreover, the hosting testbed may, as mentioned, feature additional services surpassing the baseline requirements imposed by our approach. A typical example of a hosting testbed is a widely used testbed, such as Chameleon or CloudLab. Compared with the hosting testbed, the hosted testbed in Figure 2 refers to the testbed whose features are used to perform the actual experiment. For this task, we propose the use of our pos experiment controller. In this case, the hosted testbed is an instance of pos that exists inside another testbed, i.e., the hosting testbed. Subsequently, we will demonstrate that examples of hosting testbeds include Chameleon and CloudLab. Embedded in the hosting testbed, the pos framework provides a subset of the hosting testbed’s available functionality, ultimately providing an abstraction of it. Inside this hosted testbed’s abstraction, there is, again, a differentiation of resources. On the one hand, the pos experiment controller itself is running on a managed experiment node. On the other hand, one or multiple experiment nodes, provided by the hosting testbed, are managed through the pos experiment controller. Note that both the managing and the managed experiment nodes may appear indistinguishable in their type to the hosting testbed’s management node. Additionally, depending on the hosting testbed’s design, nodes used by the hosted testbed may stem from one or multiple experiments of the hosting testbed. For example, the pos experiment controller and experiment nodes managed by it may be part of different experiments on the hosting testbed, cf. Figure 2. Running the pos framework in a testbed relies on the availability of certain functionality to be exposed via an API, namely: 1. the ability to configure the power state of an experiment node, i.e., turning it off and on again, and 2. the means to control the boot process of experiment nodes, e.g., by allowing network boot or customizing the disk image to boot. With these requirements met, the pos framework can be ported to a testbed. Presuming a successful port, the pos framework can then expose its experiment API. This API provides experiments with abilities such as: 1. defining the state of their experiment nodes, e.g., power state or running operating system, 2. scheduling execution of experiment scripts on experiment nodes, 3. synchronizing between experiment nodes, and 4. exporting experiment artifacts to the pos management node. To summarize, to host the pos experiment controller inside a variety of testbeds, a minimal set of functions, akin to all observed testbeds and required to allow hosting our approach, was determined. Building on this, the pos framework was extended to enable interfacing with dedicated as well as representative existing testbeds. Details on the implementations will be provided subsequently. The architecture of the pos experiment controller is summarized in Figure 2 and prominently isolates the hosted from the hosting testbed. 4. Implementation As indicated previously, we implemented our approach for a selection of testbeds. This selection is based on the experiences reported by Nussbaum [22]; he surveyed available testbeds with a focus on Chameleon, CloudLab, and Grid’5000 [6]. A more recent survey by Gomez et al. [23] confirms this list of available mediumto large-scale cloudcomputing or general purpose testbeds. While the authors introduce further testbeds with matching parameters, e.g., P4Campus [24], these testbeds’ mission highlight them as designed for a specific goal. Thus, we did not consider them for the first set of supported testbeds. However, the generality of the pos methodology makes its application also of interest to other domains, e.g., IoT. Given that CloudLab and Grid’5000 support GENI [8], we base our implementation on this API, thus achieving compatibility with both testbeds. Support for Chameleon is achieved by adding support for OpenStack’s API to pos. Consequently, the presented approach is not limited to the chosen testbeds. While the pos methodology is impartial to the language and tools used to conduct experiments, researchers are not. Interactive and visual tools such as Jupyter [25] enjoy great popularity. In a previous work, Demchenko et al. [26] investigated reproducible research and tools suitable for this task. They show how the pos methodology and Jupyter notebooks can be combined to this effect. Building on this, this section shows how Jupyter notebooks can be integrated with the proposed pos experiment controller. This section continues with information about implementation considerations for the respective testbeds. As each implementation follows the overarching architecture, this description is focused on particularities related to the individual testbeds. An overview of the different implementation approaches is given in Figure 3. 4.1. Extending pos: Software Architecture Perspective Extending pos, i.e., pos’ testbed orchestrator, to support an additional hosting testbed corresponds to providing a translation between the action of the two parties. For 6 pos Management Node Experiment Node Experiment Node Experiment Node Experiment (a) Dedicated pos deployment pos Management Node Experiment Node (VM) Experiment Node (VM) Experiment Node (VM) Experiment (b) Virtualized pos deployment pos Management Node Experiment Node Experiment Node Experiment Node Hosted Testbed Experiment Testbed Site Trovi (c) Chameleon pos Management Node Experiment Node Experiment Node Experiment Node Hosted Testbed Experiment Experiment Clearing House Aggregate Manager Aggregate Manager Aggregate Manager (d) CloudLab Figure 3: Implementation details depend on the hosting testbed example, a researcher’s command to reboot an experiment node, issued via pos, must be translated by pos to interactions with the hosting testbed. The hosting testbed’s response to these interactions must be translated back and interpreted by pos, to provide the researcher with consistent behavior. To facilitate the ideally transparent translation between the hosting testbed and pos, pos defines an internal abstraction that must be implemented to enable the testbed orchestrator to interact with a hosting testbed. This internal abstraction enables pos and its testbed orchestrator to execute basic operations, such as starting or stopping an experiment node. Consequently, the internal abstract is the foundation of pos’ testbed orchestrator implementation. Next to managing the power state of an experiment node, ensuring unique access to an experiment node for the duration of an experiment defines the second pillar of pos’ internal abstraction. The selection of the proper translation layer for interactions between pos and its hosting testbed is part of pos’ configuration. There, each available experiment node is associated with properties, such as its hostname; these properties include a reference to the appropriate translation layer for the hosting testbed. 4.2. pos: Dedicated Deployment In the dedicated deployment, cf. Figure 3a, the hosting testbed is non-existent and the pos experiment controller is solely responsible for managing the infrastructure. This implies increased control of the experiment infrastructure. At the same time, it also imposes a considerable burden on the experimenter. As an example, the hosting testbeds may provide additional recovery features of integration of long-term data storage or user authentication. Such features are not provided by the pos experiment controller. However, as indicated, the benefit of accepting this burden is a tighter control of the infrastructure. For example, some hosting testbeds may only provide virtualized resources, thus limiting the experimenter’s control over resources and potentially subjecting their experiments to the actions of others. While a pos testbed typically relies on a CLI to run experiments, Jupyter notebook-based interaction is also feasible. To run a Jupyter notebook-based experiment on the dedicated pos testbed, the experimenter first needs to provision Jupyter in a Python virtual environment on the testbed’s management server. Once the Jupyter notebook is installed and running, the experimenter needs to make this instance accessible, e.g., by forwarding the respective port via SSH. After this setup phase, experimenters can interact with pos either via the CLI or an optional Python library. Based on established Python libraries [27], this customized library wraps pos’ internal HTTP API. For experimenters seeking to describe their experiments in a programming language, this library exposes both highand low-level functions to describe a pos experiment. CLI and Python library are functionally equivalent, i.e., experimenters can choose their perferred approach without sacrificing functionality. 4.3. CloudLab Neither CloudLab nor pos are designed with a specific experiment script format in mind. On the contrary, both testbed and methodology are deliberately open in terms of options available to their users. Thus, generally, any approach to deploy the pos experiment controller would be feasible. For consistent deployment across testbeds, we decided to implement a Jupyter notebook-based deployment. As highlighted in Section 2.2.2, to start an experiment on CloudLab for the deployment of the pos experiment controller, a description of the experiment is required. I.e., among others, the number of nodes and their type, their connections, and disk image. For the pos experiment controller, a dedicated experiment was created. Using a Debian bullseye cloud image [28], we use Debian’s cloudinit [29] capabilities to deploy Jupyter and setup scripts to install the pos experiment controller to the experiment node. Once CloudLab provisioned the experiment node and cloud-init’s configuration has concluded, a Jupyter instance is available. To set up the pos experiment controller, the experimenter executes cells of one of the provisioned setup scripts, a Jupyter notebook. Since the pos experiment controller needs to impersonate the experimenter when interacting with CloudLab, the experimenter will be asked to provide means to authenticate with CloudLab. Subsequent steps of this notebook will deploy pos using Ansible. Finally, the deployment instantiates a second CloudLab experiment. Unless configured otherwise, two nodes are instantiated and integrated into pos. With pos configured and the sample topology in place, the second deployed Jupyter notebook containing the sample experiment can be executed. Figure 3d summarizes this implementation approach. There, “Clearing House” refers to CloudLab’s central entity for resource management, while the “Aggregate Manager” provides an entity that provides an interface to available resources [8]. 7 4.4. pos: Virtualized Deployment The deployment of pos in the virtualized testbed differs from the dedicated deployment described in Section 4.2. For instance, access to the testbed is granted via a webshell. Despite that, the implementation of the virtualized and the dedicated deployment do not differ as the features of pos used are unchanged; this similarity is shown in Figure 3b. However, due to the change in access to the testbed, the workflow changes slightly: instead of running a Jupyter server on the management node, we suggest converting the involved notebooks to Python scripts and subsequently executing those. 4.5. Chameleon Jupyter notebooks [25] are at the center of Chameleon’s workflow. While the use of notebooks is not required, their use is encouraged through the availability of APIs, examples, and documentation on their integration. Given that a core concept of the pos experiment controller is its embed-ability into different testbed contexts, its implementation takes this into account. Specifically, the pos experiment controller provides a Python API to control experiments. As a result, the interaction with the controller is well-suited for Jupyter notebooks. The most prominent execution engine for Jupyter notebooks, also called kernel, uses Python as a programming language. To run an experiment on Chameleon, a Jupyter notebook should be provided on Chameleon’s experiment scriptsharing and archiving platform Trovi. Thus, our implementation, as shown in Figure 3c, follows this approach and consists of a Jupyter notebook to interact with Chameleon. The provided Jupyter notebook attempts to allocate all resources, required for later operation, in Chameleon. I.e., one node featuring the pos experiment controller, the pos management node, and other nodes used to conduct experiments with. E.g., for the experiment discussed later, apart from the pos management node, two experiment nodes are requested to function as device under test and load generator. After the required resources have been provided by Chameleon, the implementation is otherwise unable to proceed, management and experiment nodes are configured. In particular, the nodes’ disk images are set. While the management node is configured to run on Debian bullseye, the experiment nodes are configured to boot iPXE [30]. We selected the former due to its proven reputation as a stable and well-tested platform. Besides, the selection of a PXE booting image is needed to apply the pos methodology. As mentioned, this methodology includes live-booting the experiment node’s operating system. To this end, PXE is used to serve the desired operating system to the experiment nodes. Even though numerous devices support PXE booting, its use often requires BIOS reconfiguration. BIOS reconfiguration is not supported on all hosted testbeds we encountered and may suffer from partial or flaky PXE implementations. Therefore, experiment pos Management Node DuT LoadGen Evaluator Figure 4: Experiment setup nodes boot the iPXE PXE implementation and, thus, mitigate these issues. With the management node booted, the provided Jupyter notebook will continue to set up a pos experiment controller on the same. This setup step relies on Ansible [31] and, apart from installing the software itself, ensures preconditions for conducting experiments, such as the availability of bootable images, are met. Once the setup script concludes, a brief functionality test is performed to validate the success of the deployment. With this test succeeding, the desired experiment may be performed. The provided Jupyter notebook concludes with instructions to tear down the setup, once all work has been performed. We encourage releasing experiment resources instead of relying on automatic experiment termination. The specialization of the high-level architecture depicted in Figure 2 for the implementation in Chameleon is shown in Figure 3c. Of particular note is the interaction with Trovi as an additional service. Trovi hosts the Jupyter notebook describing the experiment. Another specialization is the way each testbed allocates resources and how to map the pos experiments onto the hosting testbed: In Chameleon, for integration of the pos experiment controller, both management and experiment nodes can be part of a single experiment. On CloudLab, we use two separate experiments for the management node and the experiment nodes. 5. Evaluation To show the usability of the developed pos experiment controller, we revisit a previously conducted pos experiment [3, 32]. I.e., we execute the previously described experiment, maintaining the previous configuration but expanding the considered testbeds. We revisit the experiment on the four supported testbeds: (1) our local bare-metal pos deployment, (2) a virtualized pos deployment, (3) the Chameleon testbed, and (4) the CloudLab testbed. The considered experiment investigates the achievable throughput of a device under test (DuT), when subjected to constant bit rate traffic by a load generator (LoadGen). Evaluation of the experiment observations is done after that by an evaluator script. Management of the experiment is done by the pos experiment controller. Both DuT and LoadGen run on a Debian buster live system. 8 Figure 4 depicts the overall experiment setup. There, the test traffic is generated by MoonGen [33] on the load generator. The nature of the traffic is varied between different experiment rounds depending on two parameters. The first parameter is the packet size, here, either 64 B or 1450 B. Ranging from 10 kpps to 3000 kpps, the second parameter is the requested packet rate. The packet size is chosen to reflect both the smallest and the largest possible packet size transmittable without fragmentation on any of the investigated testbeds. In particular, on Chameleon, the usual MTU of 1500 B is not available, possibly due to the use of VXLAN as a tunneling protocol between the running virtual machines. The presented investigations and previous studies [32] have shown a perfect linear scaling of the Linux router with the number of processed packets. Therefore, we investigated only the minimum and maximum packet size of the investigated platform as additional measurements with other packet sizes provide no additional insights. From the load generator, the generated traffic is sent to the DuT. The latter is configured to act as a Linux-based forwarder, i.e., traffic received on the ingress interface is emitted unchanged on the egress interface. Packets from the DuT’s egress then arrive again at the load generator. Experiments are described via and were conducted from Jupyter notebooks [25]. While the pos framework and, thus, the pos experiment controller is agnostic to the language used to contain experiment instructions, there is a noticeable preference for this format by, e.g., the Chameleon community. In our experiment Jupyter notebooks, we separated the experiment setup from its execution. As a result, the setup of the hosting testbeds, e.g., Chameleon or CloudLab, is independent of the execution of the pos experiment. Thus, the pos experiment Jupyter notebook remains the same. This aligns with the proposed design of having a hosted testbed providing an API independent of its hosting environment. Figures 5 and 6 summarize the measurement results obtained from executing the same experiment scripts on multiple testbeds. The individual results are discussed in more detail hereafter. 5.1. pos: Dedicated Deployment For the dedicated deployment, the experiment was conducted on two dedicated machines. A DuT featuring an Intel Xeon E5-2640 v2 running at 2.0 GHz with 32 GB of memory. As load generator, we used an Intel Xeon E52640 v2 with 16 GB RAM. Both DuT and LoadGen were equipped with a 10 Gbit/s Intel X540-AT2. Results in Figure 5a indicate a linear relationship between requested and received traffic for generated packet rates below 0.5 Mpps. There, for every investigated combination of the parameter, the DuT was able to forward the traffic such that the load generator received any packets sent. For higher packet rates, however, results are more diverse. Here, the 0 0.5 1 1.5 2 2.5 3 0 0.5 1 1.5 2 2.5 3 pkt rate [Mpps] Average Packet Rate [Mpps] 64 B TX 64 B RX 1450 B TX 1450 B RX (a) Dedicated pos deployment 0 0.5 1 1.5 2 2.5 3 0 0.5 1 1.5 2 2.5 3 pkt rate [Mpps] Average Packet Rate [Mpps] 64 B TX 64 B RX 1450 B TX 1450 B RX (b) CloudLab Figure 5: Measurement result comparison: bare-metal experiment results load generator’s transmitted rate continues to grow linearly until approx. 2 Mpps and 1 Mpps for packets of size 64 B and 1450 B, respectively. Afterward, despite requesting higher packet rates, the load generator is unable to comply. In contrast to that, the packet transmitted back from the DuT, independent of the packet size, never surpasses a rate of just below 0.5 Mpps. 5.2. CloudLab The CloudLab experiment used two machines of type c220g2 for the LoadGen and DuT. These nodes feature an Intel Xeon E5-2660 v3, running at 2.2 GHz, as well as 160 GB RAM each. Moreover, this node type has dualport Intel X520 NICs, which were used as experiment interfaces. Similar to the dedicated deployment, the results in Figure 5b show a linear relationship between requested and received traffic. I.e., with the given experiment, no deviation of the received traffic from the requested traffic is noticeable. 5.3. pos: Virtualized Deployment In the virtualized setting, the VMs are hosted on a system equipped with two Intel Xeon Silver 4214 12-core CPUs running at 2.2 GHz and with 384 GB RAM. Apart from the pos management node providing both orchestrator and controller, two experiment nodes, provided by virtualized machines, are provisioned on this host system. Each experiment node is assigned four cores and 7.4 GB of RAM. The virtualized machines are connected via two virtual links provided by the KVM-based hypervisor. For both investigated packet sizes, the experiment results, as depicted in Figure 6a, look alike. The load generator is, for all investigated packet rates, able to provide the 9