Towards Adaptive Resource Management and Control on Edge Platform for AI Applications

Towards Adaptive Resource Management and Control on Edge Platform for AI Applications Yongho Kim, Rajesh Sankaran, Charlie Catlett, Nicola Ferrier, and Pete Beckman Mathematics and Computer Science Argonne National Laboratory Lemont, USA [email protected], [email protected], [email protected], [email protected] Sean Shahkarami NAISE Northwestern University Evanston, USA [email protected] Abstract—Dynamic, fine-grain resource management is needed for AI and machine learning workloads in multi-tenant environments, as these now represent an important use case for edge computing. In this work, we present a programming model that addresses this need by exposing applications control knobs and metrics to the edge scheduler for resource management. Our initial demonstration of this programming model shows its potential for making adaptive scheduling decisions in edge computing platforms. Index Terms—edge computing, edge scheduler, resource management and control I. INTRODUCTION Waggle [1] is a cyberinfrastructure (CI) that offers an edge computing platform for edge Artificial Intelligence / Machine Learning (AI/ML) applications. Over the past decade through projects like the Array of Things [2] and Sage [3], we have deployed over 200 Waggle edge nodes (wirelessly connected hardware/software devices with embedded computing and sensing capabilities) in both urban and rural settings around the world. Our research team has been advancing state-ofart ML models and algorithms using Waggle [4], [5] and has published scientific AI@Edge applications in our open edge code repository (ECR)1. Through Waggle and the ECR we are creating a venue where scientists and engineers create AI applications and share them with the community, enabling adaptation and use for diverse research and lowering the barrier-to-entry for scientific adoption of AI-enabled edge applications. The ability to manage resources on edge platform while efficiently scheduling applications is the key to providing a CI that can be used in a multi-tenant environment. To this end, the Waggle edge scheduler should provide adequate resources to those edge applications for expected quality of service (QoS) and adapt itself to any resource constraints such as those caused by power/network shortage and resource conflicts from multi-tenant nature of the system. At the same time, Waggle edge applications are enabled and encouraged to expose control knobs to enable the scheduler to adjust their resource demands. In this paper, we propose a programming model that abstracts user application and exposes control knobs to the scheduler to change different resource profiles 1Sage Edge Code Repository: https://portal.sagecontinuum.org/apps/explore of the application at runtime. As a proof of concept, we have partially implemented the model and collected resource consumption of user applications registered in ECR and using the model. II. THE PROPOSED MODEL We will first describe the Waggle edge scheduler that is run on every Waggle node and makes autonomous local scheduling decisions. Then, we will detail the proposed programming model and its connection to the scheduler. A. Waggle Edge Scheduler The Waggle edge scheduler [6] manages AI/ML applications on a node. User jobs describe “science rules”, comprising IF-THEN conditions about their intended context, to let the scheduler know when they prefer the application to be executed. For instance, a flood measurement algorithm would optimally run during and after precipitation events, but not during droughts. The scheduler then makes scheduling decisions on allocating resource to the applications. Kubernetes is the runtime system used to launch applications in containers, connecting them to the Waggle node services. However, applications may not be scheduled even though their “science rule” is valid when the current resource is depleted, or, they may crash and fail if the application’s resource needs exceeds availability of a given resource. Knowing application resource profiles will allow the scheduler to allocate the approximate resource for the application and to free up resource by changing the application’s resource profile. B. Interfacing Framework The proposed programming model employs the sidecar pattern [7] – an add-on program assisting the main program/service – to interact with user applications. The model exposes the sidecar container to the scheduler or a code developer for monitoring application performance and for controlling the application by changing the application configuration. The main advantage of the model is that it minimizes code changes in the application while making the application dynamic for controls. When user applications are wrapped with PyWaggle – Waggle’s Python programming interface – the sidecar can control the flow of application at function level, which will produce different resource profiles. Waggle Edge Computing Node Waggle Edge Computing Node Waggle Edge Scheduler Waggle Cloud Scheduler Users Portal Kubernetes Backend Resource Monitoring Downloading Science Goals and AI@Edge Applications Science Rule Evaluation sensors AI@Edge Application AI@Edge Application AI code in functions flow control algorithms app configuration container cgroup control knobs Application Container output data Waggle Sidecar Container Resource monitoring meta data insertion & data control Resource management / flow monitoring and control Waggle Software Stack For sensor/data services Fig. 1: Overview of Waggle Edge scheduler and the proposed model design shown in the red rectangle. III. DEMONSTRATION AND EXPERIMENTATION To show applicability of the proposed model, we implemented the model in Waggle nodes and used it to collect resource metrics of existing scientific AI/ML edge applications. Figure 2 shows resource profiles of an edge application run on the embedded computing resource (NVIDIA Jetson NX) in a Waggle node. The sidecar container running alongside the application container reported the application container’s resource consumption to a local time-series database. When the given resource is limited (See Figure 2b) the application took longer to complete its task. When the scheduler further limited memory the application crashed with an out-ofmemory (OOM) error. As demonstrated, the proposed model can let the scheduler receive the current resource consumption and possibly adjust resource allocation to prevent the crash. It could also be possible to lower the resource use if the application can be configured to use a lighter ML model through the sidecar interface, preventing the OOM error. IV. CONCLUSION Edge computing will face increasing and interesting challenges in areas including resource management, system control, and optimization as AI applications continue to grow with larger ML models such as transformers and with complex workflows involving interactions between Cloud and Edge. The proposed model is a conceptual design of how to make AI/ML applications dynamic in their performance and quality of the result. We expect that this model will enable finegrained control of AI/ML applications at the edge and efficient resource management based on objectives of application workflows. REFERENCES [1] P. Beckman, R. Sankaran, C. Catlett, N. Ferrier, R. Jacob, and M. Papka, “Waggle: An open sensor platform for edge computing,” in 2016 IEEE SENSORS. IEEE, 2016, pp. 1–3. (a) no resource limit (b) limited to 1 logical CPU core and 1 GB of memory, no limit for GPU Fig. 2: Performance profiling of the traffic state estimation. The application records a 30 second video and uses Yolo V7 and DeepSort to analyze the video for vehicle detection and tracking. Note that the Jetson NX used the shared memory between CPU and GPU. [2] C. Catlett, P. Beckman, N. Ferrier, M. E. Papka, R. Sankaran, J. Solin, V. Taylor, D. Pancoast, and D. Reed, “Hands-on computer science: The Array of Things experimental urban instrument,” Computing in Science & Engineering, vol. 24, no. 01, pp. 57–63, Jan. 2022. [3] P. Beckman, C. Catlett, I. Altintas, E. Kelly, and S. Collis, “Mid-Scale RI-1: Sage: A Software-Defined Sensor Network (NSF OAC 1935984),” Oct. 2019. [4] R. C. Jackson, B. A. Raut, D. Dematties, S. M. Collis, N. Ferrier, P. Beckman, R. Sankaran, Y. Kim, S. Park, S. Shahkarami et al., “Arming the edge: Designing edge computing-capable machine learning algorithms to target arm doppler lidar processing,” Artificial Intelligence for the Earth Systems, pp. 1–38, 2023. [5] S. Park, Y. Kim, N. J. Ferrier, S. M. Collis, R. Sankaran, and P. H. Beckman, “Prediction of solar irradiance and photovoltaic solar energy product based on cloud coverage estimation using machine learning methods,” Atmosphere, vol. 12, no. 3, p. 395, 2021. [6] Y. Kim, S. Park, S. Shahkarami, R. Sankaran, N. Ferrier, and P. Beckman, “Goal-driven scheduling model in edge computing for smart city applications,” Journal of Parallel and Distributed Computing, vol. 167, pp. 97–108, 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0743731522001009 [7] B. Burns and D. Oppenheimer, “Design patterns for container-based distributed systems,” in 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 16), 2016. GOVERNMENT LICENSE The submitted manuscript has been created by UChicago Argonne, LLC, Operator of Argonne National Laboratory (“Argonne”). Argonne, a U.S. Department of Energy Office of Science laboratory, is operated under Contract No. DE-AC0206CH11357. The U.S. Government retains for itself, and others acting on its behalf, a paid-up nonexclusive, irrevocable worldwide license in said article to reproduce, prepare derivative works, distribute copies to the public, and perform publicly and display publicly, by or on behalf of the Government. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doepublic-accessplan).