Delay optimal schemes for Internet of Things applications in heterogeneous edge cloud computing networks

Citation: Lakhan, A.; Mohammed, M.A.; Abdulkareem, K.H.; Jaber, M.M.; Nedoma, J.; Martinek, R.; Zmij, P. Delay Optimal Schemes for Internet of Things Applications in Heterogeneous Edge Cloud Computing Networks. Sensors 2022, 22, 5937. https://doi.org/10.3390/ s22165937 Academic Editor: Antonio Guerrieri Received: 9 July 2022 Accepted: 5 August 2022 Published: 9 August 2022 Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. Copyright: © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/). sensors Article Delay Optimal Schemes for Internet of Things Applications in Heterogeneous Edge Cloud Computing Networks Abdullah Lakhan 1,2 , Mazin Abed Mohammed 3, Karrar Hameed Abdulkareem 4,5 , Mustafa Musa Jaber 6,7, Jan Nedoma 8,* , Radek Martinek 9and Petr Zmij 10 1Department of Cybersecurity and Computer Science, Dawood University of Engineering and Technology, Karachi City 74800, Sindh, Pakistan 2Institute of Artificial intelligence and Blockchain, Guangzhou University, Waihuan West Road, University Town, Guangzhou 510006, China 3College of Computer Science and Information Technology, University of Anbar, Anbar 31001, Iraq 4College of Agriculture, Al-Muthanna University, Samawah 66001, Iraq 5College of Engineering, University of Warith Al-Anbiyaa, Karbala 56001, Iraq 6Department of Computer Science, Dijlah University College, Baghdad 00964, Iraq 7Department of Medical Instruments Engineering Techniques, Al-Farahidi University, Baghdad 10021, Iraq 8Department of Telecommunications, VSB-Technical University of Ostrava, 708 00 Ostrava, Czech Republic 9Department of Cybernetics and Biomedical Engineering, VSB-Technical University of Ostrava, 708 00 Ostrava, Czech Republic 10 Industrial Engineering—Brose Group, Prumyslovy Park 302, 742 21 Koprivnice, Czech Republic *Correspondence: [email protected] Abstract: Over the last decade, the usage of Internet of Things (IoT) enabled applications, such as healthcare, intelligent vehicles, and smart homes, has increased progressively. These IoT applications generate delayedsensitive data and requires quick resources for execution. Recently, softwaredefined networks (SDN) offer an edge computing paradigm (e.g., fog computing) to run these applications with minimum end-to-end delays. Offloading and scheduling are promising schemes of edge computing to run delay-sensitive IoT applications while satisfying their requirements. However, in the dynamic environment, existing offloading and scheduling techniques are not ideal and decrease the performance of such applications. This article formulates joint and scheduling problems into combinatorial integer linear programming (CILP). We propose a joint task offloading and scheduling (JTOS) framework based on the problem. JTOS consists of task offloading, sequencing, scheduling, searching, and failure components. The study’s goal is to minimize the hybrid delay of all applications. The performance evaluation shows that JTOS outperforms all existing baseline methods in hybrid delay for all applications in the dynamic environment. The performance evaluation shows that JTOS reduces the processing delay by 39% and the communication delay by 35% for IoT applications compared to existing schemes. Keywords: JTOS; CLIP; SDN; task scheduling; framework; dynamic environment 1. Introduction These days, the usage of industrial automation applications in the Internet of Things (IoT) enabled paradigm has been growing progressively in practice. Industrial automation applications are smart homes, smart agriculture, smart healthcare, and smart transport with different data analytics sensors that offload their data to the cloud server for execution. Recently, Internet of Things (IoT) applications, such as healthcare, autonomous vehicles, and smart homes, are increasing progressively. IoT brings the efficient resource environment for the industrial automation applications with the collaboration of fog and cloud networks. With millions of sensors and intelligent devices, various applications can be developed that generate vast amounts of data and have stringent latency requirements. Sensors 2022,22, 5937. https://doi.org/10.3390/s22165937 https://www.mdpi.com/journal/sensors Sensors 2022,22, 5937 2 of 30 These applications include smart grids, innovative healthcare, intelligent vehicles, intelligent buildings, and many more. Typically, in cloud computing, data are sent to remote data centers for computing and storage. However, with the advent of IoT platforms, novel applications and use cases have emerged with bandwidth and latency requirements that cannot be met by traditional cloud computing. Hence, a new computing paradigm was created to cater to applications with low latency requirements. Fog computing extends cloud computing by bringing the storage and computing facility to the edge of the network, reducing bandwidth requirement and latency. Fog computing, also called edge computing, does not replace cloud computing. Fog nodes are used for short-term analytics with limited data. Resource-intensive computing and long-term analytics take place at cloud data centers. The Open Fog Consortium (OFC) is an open-source multi-vendors hybrid architecture that allows the IoT applications that leverage fog and cloud architectures mutually [1]. Core mobile cloud computing (MCC) offers unlimited shareable services to the users [ 2 ]. However, it is often incurred with long end-to-end latency due to the multiple hops away from users. Fog computing is a subset of cloud computing that enables cloud services at the edge of users’ networks with ultra-low latency [ 3 ]. Typically, IoT sensors and devices are resource constrained and generate huge amounts of data that are processed by applications [ 4 ]. IoT-based applications comprise various tasks that can be computed or data-intensive. Due to the high demand for computing and other resources, compute-intensive tasks are offloaded. Task offloading is a method which transfers all compute-intensive parts of an application either to the fog or public cloud for processing [ 5 ]. The major challenge in task offloading is to decide whether the task is to be sent to the fog or cloud for processing [ 6 ]. Though the decision is based on several factors, bandwidth, data, and latency remain the most critical. Further, the heterogeneous nature of IoT applications’ data, along with the high quality of service (QoS) requirements in terms of latency and efficiency, makes the decision of task offloading more complex [ 7 ]. In order to meet latency requirements, the tasks are offloaded jointly in heterogeneous computing nodes to solve scheduling. Task offloading problems for IoT applications have been widely addressed separately by many researchers. For instance, these studies [ 8 – 14 ] investigated the delay optimal offloading problem in the mobile edge cloud (MEC). The aim is to reduce device energy consumption and accelerate application performance on resource-constrained devices. Furthermore, the authors [ 15 – 18 ] addressed the issue of task scheduling in the MEC. The goal is to schedule edge cloud resources to improve application performance and decrease the energy consumption of cloud nodes. The joint optimization problem has been investigated by these studies [ 6 , 7 ] to address task offloading and resource allocation problems mutually for IoT applications. However, the work mentioned earlier did not focus on task offloading and task scheduling problems jointly to meet the requirements of latency-sensitive tasks in the fog cloud networks. Therefore, there is a need for a framework that can optimize IoT application performance by considering both task offloading and scheduling problems in fog cloud networks. This article formulates delay-efficient joint offloading and scheduling as combinatorial integer linear programming (CLIP) problems in heterogeneous fog cloud networks for industrial automation applications. All the problem constraints are linear integers and objective functions to be minimized for each task. The industrial automation applications based on IoT are smart homes, augmented reality (A.G), E-Business (E-Factories), and E-Healthcare; they offload their workloads to the fog cloud for the processing in the system. Furthermore, heterogeneous fog cloud networks are the combination of different capacity fog computing and core cloud computing. This research aims to minimize the total delays (i.e., network delay and cloud delay) of a task during offloading and scheduling. Each IoT application consists of independent tasks. Each task includes workload attributes and deadline constraints. The fog clouds are dispersed and connected with users via the base station (BS). All BSs are linked via switches and managed by a software-defined network (SDN) control [8]. Sensors 2022,22, 5937 3 of 30 The considerations in our article are quite different from the existing ones in several aspects: • Generally, former studies [ 7 – 14 ] made offloading decisions based on single criteria, such as either mobile battery power threshold or application total time limitation. Whereas, in a dynamic environment, a single threshold value-based offloading decision is not accurate; thus, we consider the multi-criteria task offloading-based decisions more accurate, and they incur lower overhead during the task offloading decision. • Existing task scheduling problems generally involve soft deadlines in the homogeneous edge only or core cloud-only resources [ 15 – 23 ]. Nonetheless, the frameworks proposed in those research works consider mutually heterogeneous fog cloud resources for IoT applications. Furthermore, each task has a hard deadline and must meet the stringent latency requirements during offloading and scheduling. • The high-level placement policy of the proposed fog-cloud architecture for real-time and delay-sensitive applications is different from earlier works’ architectures regarding offloading decisions and task prioritizing for scheduling. The state of the art formulated the network delay and computational delay for IoT tasks. The key objective is to minimize the delay of tasks in the system. There are many types of delays in the fog cloud network. For instance, network delay, communication delay, wait for delay, and processing delay. However, these studies only considered static offloading and static scheduling in their solutions without considering the mobility factors and dynamic aspects of functions in the system. Therefore, real-time, multi-parameter aware offloading and scheduling in dynamic environments are widely ignored in the state-of-the-art studies. This study focuses on two types for each task, such as network delay and computational delay. The network will optimize by the offloading technique, and computational delay will optimize by dynamic scheduling in the study. The Major contributions of this article are summarized below: 1 The problem considered the hybrid delay, a combination of network delay and computation delay under a dynamic environment where transient failure in resources always occurs. This study suggests an architecture that shows how to solve the joint offloading and scheduling problem in different steps. The study considered the following steps: offloading, sequencing, scheduling, and transient failure awareness for IoT applications. The architecture aims to construct an environment to facilitate applications to run with distributed resources in the network. The architecture components or steps are discussed in the proposed solution in detail. 2 The fuzzy multi-criteria task offloading method is proposed, which makes an optimal offloading decision that adopts changes during the offloading decision process, ensuring that the network context changes do not degrade the task offloading performance. 3 The work devises latency efficient task sequence in which their competent order arranges all tasks, thereby meeting all application requirements. 4 The task scheduling method with topological sorting, searching, and transient failure methods proposed by the study to deal with the robustness of the applications during scheduling in the network. This study formulated the CLIP problem for IoT, which is different from existing studies in the following way. This work formulated the joint optimization as the CLIP problem, which is a well-known NP-Hard problem. To solve the joint optimization problem, the JTOS framework is proposed. For offloading and scheduling, the prediction of delay and QoS of applications are satisfied during the process. We consider the round-trip delay between users and BS and BS to computing nodes in the network delay. Existing studies only considered either user to node delay or user to BS delay. This study introduced a novel transient failure method, which can handle any transient failure tasks during the process. The previous works focused on the failure of tasks based on checkpointing and or primary backup method. These methods cannot handle transient failure and consume much more resources of the nodes during the recovery of tasks from the failure state. Sensors 2022,22, 5937 4 of 30 The rest of the article is organized as follows. Section 2elaborates related work, followed by Section 3, which describes the proposed description and formalizes the problem under study. A heuristic is proposed for the considered problem in Section 4that describes the proposed algorithm: JTOS. Section 5discusses the experiments and results, whereas Section 6concludes the article. 2. Related Work Recently, the usage of IoT with different computing nodes (e.g., fog/edge computing) is increasing day by day [ 24 ]. Many partitioning, offloading, and scheduling frameworks, architectures, and methods have been suggested to improve the energy, cost, and delay of applications and solve the CLIP problem. In the literature, many efforts have been made to solve the different IoT application problems. We analyze the actions of existing studies in the table to solve the CLIP problem (Table 1). Table 1. Existing offloading methods. Research Parameters Decision Profiling Environment Problem Objective [1] Single Para. Static Network Fixed ILP Min.Energy [2] Single Para. Static Program Fixed LP Min.Energy [3] Single Para. Static REST API Fixed MLP Min Computation [4] Two Para. Dynamic RPC Adaptive Concave Max.Utilization [5] Multi-Para. Dynamic Monitoring Adaptive Quadratic Max. Throughput [6,7] Multi-Para. Dynamic Resource Adaptive Integer-Constraints Min.Delay [8–10] Multi-Para. Dynamic Monitoring Adaptive Quadratic Min.Energy [8,11] Multi-Para Hybrid Monitoring Mobility ILP Min Rent [12–14] Many-Para. Hybrid SDN-Controller Mobility ILP Min.Cost [15–17] Many-Para. Hybrid SDN-Controller Mobility LP Min.Budget [18,19] Many-Para. Hybrid OS Mobility ILP Min.renting cost [20,21] Many-Para. Hybrid OS Mobility ILP Min.Energy [1,3,5,8,11] GA Many Fog-Cloud VMs Olog(m×n)Offloading [2,4,6,10,12] PSO Many Edge/Cloud VMs Olog(m×n)2Joint-ORA [13–19] Heuristics Many Edge VMs M×N×NOffloading [20–23,25,26] DRL Many Edge-Cloud Containers M×M×NScheduling Debashis et al. [ 1 ] suggested a multi-leveling offloading method to solve the partitioning and offloading problem in distributed mobile cloud architecture. The goal was to minimize the power and latency of the application during the process in the architecture. The study considered the single parameter offloading (e.g., battery energy/delay), static offloading, network profiling technology, and fixed resources (e.g., mobile and cloud resources) and solved as integer linear programming. However, due to the long distance between mobile users and the cloud, the offloading faced end-to-end latency issues during the process. Shahryari et al. [ 2 ] investigated energy-delay-aware offloading for latencysensitive applications. The study formulated this problem as linear programming with non-linear constraints. The single parameter offloading (e.g., battery energy/delay), static offloading, program technology, and fixed resources (e.g., mobile and fog resources) have been considered. However, due to the high ratio of user requests, the resource-constraint fog node faced an overloading situation during the offloading process. Aruba et al. [ 3 ] formulated a scheduling convex optimization problem for the IoT in a fog cloud network to minimize the lateness of the applications. The study considered two types of workload, including latency-sensitive and delay-tolerant with multi-parameters (e.g., workload size and offloading time) in the static resource environment. The study presented the window algorithm for application profiling which may decide whether the workload of the Sensors 2022,22, 5937 5 of 30 application offloads or not in the distributed fog cloud network. Lin and Fan et al. [ 4 , 5 ] suggested a SDN-based fog cloud network for IoT applications. The offloading and resource allocation-aware schemes are widely suggested to minimize the end-to-end delay of applications. The combinatorial optimization (e.g., Concave) and linear programming (e.g., Quadratic) based objectives were optimized. The multi-parameters (e.g., local execution, network execution, and computing execution delay were taken into consideration) during offloading and resource allocation. The network changes are also considered an adaptive environment where offloading is performed based on new available values instead of old network contents. The authors in [ 6 – 10 ] investigated joint offloading and resource placement problems for IoT applications in distributed fog/cloudlet cloud networks. These studies considered the dynamic and adaptive environment where network contents and resource placement can change at runtime. The proposed scheduler and offloader engine adopt any runtime changes during the initial process. Based on the experience, a new offloading decision will be more optimal than the existing one. The offloading decision is dynamic concerning resource placement and offloading, where remote procedure call methods are formulated as the integer constraints and the quadratic problem. The objective function was a convex function, i.e., minimize delay, and all variables are convex linear integers in the considered problem. The authors in [ 8 , 11 – 14 ] investigated resource provisioning and cost-aware offloading and scheduling problems for IoT applications in the distributed mobile fog cloud network. The SDN controller was implemented to facilitate mobility features of applications during roaming among networks. The dynamic environment and hybrid static and dynamic offloading decisions were taken into consideration. The resource cost, budget, rent and application delay, and energy objectives were optimized as joint linear integer programming optimization problems. All coarse-grained workloads are scheduled under their maximum threshold level to avoid any violence of users. The authors in [ 15 – 19 ] suggested that frameworks and architectures solve deadline-aware offloading and scheduling problems for IoT applications. They formulated problems as integer linear programming where all objective functions and constraints are linear and integer variables. The budget, energy, renting cost, scheduling cost, and offloading delay objectives are optimized via different optimization methods. The orchestrator controller (e.g., SDN and system components) are implemented to offer mobility-aware services to the IoT vehicle applications during their roaming features. The dynamic offloading decision (e.g., static and dynamic) and dynamic workload assignment in fog cloud network were considered during the problem formulation. Table 1shows the efforts of existing studies in the area. Furthermore, the authors in [ 1 , 3 , 5 , 8 , 11 ] proposed optimization algorithms based on a genetic algorithm (GA) with many components (e.g., resource searching, resource allocation and offloading) for IoT applications. The suggested that mixture architectures are based on fog-cloud nodes with virtual machine implementation to serve users’ requests. The proposed methods contained average time complexity during the process of applications in the network. The linear and global searches are parts of GA during the execution of applications. The works [ 2 , 4 , 6 , 10 , 12 ] devised optimization techniques based on particle swarm optimization (PSO) meta-heuristics with many components (e.g., resource searching, composition, allocation, and migration). The fog-cloud was implemented with virtual machines to process the workload of applications with their constraints. To obtain the complexity of the algorithms, the lightweight and linear search-based iterative model was devised by studies. The studies [ 13 – 19 ] devised different optimization heuristics based on Bundle branch-bound, Subgradient linear search, Interior-Points, and Cutting-plane Hungarian with different types of searching methods. These methods search for the best resource among edge/fog cloud nodes before allocating tasks to them. The time complexity is lower because these studies implemented linear search, where all optimal solutions select randomly based on integer parameters. Sensors 2022,22, 5937 6 of 30 Recently, the complexities of IoT applications are increasing day by day. For instance, healthcare applications contain real-time tasks, which require continuous attention from network and computation nodes. Due to the uncertainty of the network due to mobility and traffic, the ratio of failure tasks, delay, and deadline can occur widely. The uncertainty and fluctuation in computing nodes can lead to the violence of the quality of service of applications in the system. Therefore, all existing conventional heuristics cannot adapt to any dynamic changes and do not support the complex requirements of applications. Recently, dynamic environment aware deep reinforcement learning (DRL) aware heuristics were proposed [ 20 – 23 , 25 – 27 ] to solve the complex IoT problems. The DRL approaches can work better in a dynamic environment via different states where all states are independent. These approaches are achieving long-term goals in terms of offloading and resource allocations. However, many issues remain in both conventional methods and machine learning approaches when solving the joint offloading and scheduling problem for IoT applications. 3. Proposed Solution This section aims to discuss the importance and all steps of the proposed architecture. As mentioned above, this study considers the joint offloading and scheduling optimization problem as a CLIP for IoT applications in the distributed network. The proposed architecture consists of three main layers: IoT application layer, agent layer, and resource layer, as shown in Figure 1. All layers are managed and controlled by the agent layer. The application layer consists of different applications, where each application is composed of various independent tasks. However, resource-constrained local devices (e.g., limited battery, computational capability, and storage) cannot run all tasks locally on devices. Therefore, a Fuzzy multi-criteria method (FMCM) is implemented at the application layer, which boosts the performance of all applications via an offloading process based on QoS requirements (e.g., deadline and hybrid delay). Initially, FMCM generates an offloading result based on QoS requirements and sends it to the agent layer. Then, the agent layer allows devices to offload their tasks to the system for efficient execution. The agent layer accepts offloaded tasks based on their offloading result for further execution. Initially, all tasks are sorted into topological order based on their deadlines and total delay. The main reason behind sorting is that all tasks have different sizes, deadlines, and resource requirements. Therefore, delay-sensitive tasks with the lowest deadline must have high priority compared to delay-tolerant tasks with long deadlines. Furthermore, based on sorting order, all tasks are scheduled onto different computing nodes based on their offloading results. In the dynamic environment, in different timezones, the ratio of user traffic is different; therefore, due to the ingests percentages of users in peak hours (8 am to 4 pm), the transient failure of resources often occurs in the system. The transient failure aware (e.g., detection, retry, and familiar failure schemes) handles all transient failure tasks and runs all applications robustly without losing their generosity. The orchestrator, multilayers SDN scheduler controller is the primary controller in the agent layer, responsible for managing load balancing and connectivity of all computing nodes and all layers, and it helps to monitor resource status in the network. The resource layer combination of fog nodes and cloud nodes is connected with the SDN controller and BSs. Here, the SDN controller allows devices to make the offloading request to the associated BSs and ensures that BSs are directly connected with computing nodes for further processing. For instance, BS 1 was directly connected with the fog node k1 to process task v1 and v6 . Whereas BS 2 connected with fog k2 and processed the tasks v3 and v2 . The delay-tolerant tasks are scheduled on fog nodes; however, all delay-tolerant tasks must be scheduled on cloud computing for efficient processing. For instance, v5 and v4 and the rest of the tasks are offloaded to the remote cloud by the SDN controller for execution via the Internet. Sensors 2022,22, 5937 7 of 30 TasksTopological Ordering V1V2V3V4 V1V3V2VN Offloading V5V6VN V6V5 V4 Smart Home Sensors A.G. Sensors E-Business Sensors E-Healthcare Sensors IoT Application Layer 1 Fog-Cloud Agent Layer 2 Fuzzy Multi-criteria Method Task Offloading Results Orchestrator Multi-Layers SDN Scheduler Controller V6 V1 V3 V2 V5 V4 Fog k1 Base-station 1 Base-station 2 Fog k2 Cloud k3 Internet VN Geographically Distributed Resource Layer 3 Figure 1. Joint Hybrid Delay Optimal Offloading and Task Scheduling for IoT Applications in Fog-Cloud Architecture. 3.1. System Model Scenario As we mentioned above, the study considered leaving dynamic requests of users in the system. Therefore, resources, network channel capacity, and QoS (quality of service) and quality of experience (QoE) requirements of applications are dynamic. The scenario of IoT applications in the fog cloud network is defined as follows. Initially, all IoT applications (e.g., mobile devices, sensors, cameras, vehicles, etc.) are connected to the BSs. Each application is only connected with one BS at a time. In comparison, many base stations are connected with fog nodes which are dynamically distributed and managed by the SDN control plane. All the BSs are connected with fog nodes via different fiber optics switches. The SDN control plane is also associated with the remote cloud via wire Internet to process the delay-tolerant tasks of applications during execution. All BSs and fog nodes are resource constraints; therefore, SDN control is also responsible for load balancing among BSs and computing nodes. 3.2. Problem Formulation The architecture leverages three different core technologies: IoT sensor-based technologies, wireless technologies (i.e., WiFi, Bluetooth, and cellular network), and computing frameworks (e.g., fog node and cloud node). The considered joint optimization task is offloading and the scheduling problem for IoT applications incurred two kinds of delays: network delay and computation delay. Thus, total delays for each task have network delay and computation delay during offloading and execution. The notation of the study is described in Table 2. Sensors 2022,22, 5937 8 of 30 Table 2. Mathematical notation. Notation Description ANumber of all IoT applications GThe Gth application of A NNumber of tasks of application G viith independent task vi↔bRound-trip delay between viand b BSet of base stations b yib The coverage base stations b↔kThe Round-trip delay between band k Tnet iTotal network delay for task vi MSet of computing nodes k kth computing nodes of M ekResource capacity of computing node kth kth ζjThe speed capability of jth of kth computing node WiThe requested workload of task vi W0 iThe result of task vi diThe deadline of task vi Te iThe execution time of task vi xij The assignment of task vion kjcloud FiThe finish time of task vi Td The lateness of the vi CThe channel capacity BwThe bandwidth of the channel in hertz SSignal power over the bandwidth NsInterference over the bandwidth S NsThe signal-to-noise ratio 3.3. Dynamic Environment The problem constraints, such as computing resources, network contexts, and task size, are highly dynamic in the study. This work considered the in and out user requests in the system as a dynamic environment without mobility features. Although the mobility feature is a part of the dynamic environment, this study did not consider the mobility of the application in the current version of the work. 3.4. Network Delay The network delay for IoT applications is comprised of round-trip delays, such as the delay between a task vi and associated base station b and k th computing node. The notation vi↔b shows round-trip delay between task vi and assigned BS b . Whereas the notation b↔k shows round-trip delay between the base station b and computing node k . Each application is composed of different types of tasks, such as video, audio, image, text, and so on. However, it is not trivial to know in advance what amount of task data will be carried via a channel per second or at what rate the channel transfers task data to the cloud for execution. There are many factors that can interrupt communication links during task offloading and downloading, such as noise, inference, and intermittency in the wireless network [ 28 ]. Thus, our task offloading decision method adopts dynamic changes of network contexts during offloading. The binary variable y={ 0, 1 } shows either the offloaded task i being implied in base station b in the coverage area yib = 1 or not yib = 0. The communication delay for a task during offloading and downloading between BS and computing node can be estimated in the following way: Tnet i=yib(vi↔b+b↔k)×C. (1) Sensors 2022,22, 5937 9 of 30 Equation (1) determines the maximum capacity C of the channel and measures the network delay of each task, which is the sum of delay between the user and BS and BS to any particular computing node during offloading and downloading. In simple terms, it is round-trip delay between a user to BS and BS to computing node. rc =Wi UB ·wc +W0 i DB ·wclog21+S N. (2) In Equation (2), the variable rc determines the usage ratio of channel resources. Where UB . wc is the uploading bandwidth (i.e., not fixed) of the wireless channel, wc determines the hertz (H), and UB . wc is the downloading bandwidth. The variable S is the signal power of the network in watts, and N is the present noise and inference in the wireless during task offloading to the fog cloud network. Whereas the variable log2(Wi UB·wc )+( W0 i UB·wc ) is the amount of data offloaded and downloaded to and from BS and computing. The network delay of all tasks between all BSs and computing nodes is determined in the following way. Tnet = A ∑ G=1 N ∑ i=1 yG ib(vG i↔b+b↔k), (3) b∈B,k∈M. Equation (3) calculates the network delay of all tasks during offloading and downloading . Computation Delay The aim of the study is to offload and schedule all independent tasks of different IoT applications to the heterogeneous computing nodes k∈M to reduce the maximum delay of all tasks. The tasks are represented by {v1 , v2 , v3 , . . . , vN} . Each task vi has workload Wi{vi= 1, . . . , N} and deadline di , which is defined by the user when tasks are offloaded to the proper computing node. All computing nodes are heterogeneous and represented by {k1 , k2 , . . . , kM} . Each computing node has a different computing speed which is depicted as ζk , where k= 1, 2, . . . M . We denote the computing resources of all computing nodes in this way, i.e., ek={k= 1, . . . , kM} . The notation ek denotes particular resources of the node k . To reduce the computation delay of all submitted tasks, we assign each vi to the ideal computing node to meet the deadline constraint of all tasks with minimum delay. Meanwhile, a task vi is executed by a single cloud k . The decision variable is employed xik and either task vi is assigned to k or not. The execution time of the task vi on k th computing node can be estimated as given in Equation (4). The computation delay on a particular node of a task iis determined in the following way: Te i=xij Wi ζk . (4) Equation (4) measures the execution delay of a task on the particular computing node. Similarly, the execution delay of all tasks on all computing nodes is determined in the following way. Te= A ∑ G=1 N ∑ i=1 xG ij WiG ζk where k ∈M(5) Equation (5) calculates the computation delay of all tasks on heterogeneous computing nodes. The considered problem is mathematically modeled as below: Ttotal =Tnet +Te. (6) Sensors 2022,22, 5937 16 of 30 Algorithm 2: Task Offloading Phase Input :∑A G=1,ω={f w1,f w2. . . , ω} Output:results[i,k] 1begin 2foreach (G=1 to A) do 3 Construct fuzzy weight tasks and resource matrix of each application based on Equation (16); 4The reliability index measured based on Equation (18); 5RI =λmax−n (n−1)RR =RI (n−1)×Random−Index ; 6Normalized the tasks index based on Equation (20); 7NM(xi×k)dc =M(xi×k)dc ∑4 n=1M2 i×k ; 8Construct the decision matrix based on Equation (21); 9Mw=ωv×N; 10 foreach (dc=1 to DC) do 11 Generate position solution and negative solution based on Equation (22); 12 i←k←J+; 13 i←k←J−; 14 determines the best choice and worst choice of each task to computing based on Equation (24); 15 i←k←D+ v; 16 i←k←D− v; 17 Construct the optimal and ideal rank of each task onto optimal computing based on Equation (26); 18 results[i,k]←Hv←Ttotal; 19 if (rc ≤C) then 20 Offload: results[i,k]; 21 results[i,k]for all applications; 22 End Table 5. Sequence-1: Sorting Tasks by Their Descending Order of Resource Ttotal. Fog Nodes Cloud Node Application k1k2k3k4 G1v2,v5v3v1,v6v4 G2v6v2,v1v5v3,v4 G3v6v3v2v4,v5 G4v1,v4v5v1v3,v6 The topological ordering of deadline sequence is shown in Table 6. Sensors 2022,22, 5937 17 of 30 Table 6. Sequence-2: Sorting Tasks by Their Descending Order of Resource di. Fog Nodes Cloud Node Application k1k2k3k4 G1v5,v2v3v6,v1v4 G2v6v1,v2v5v4,v3 G3v6v3v2v4,v5 G4v1,v4v5v1v6,v3 4.3. Dynamic Task Scheduling The study devises the scheduler method Algorithm 3for the workload sorting and execution in the system. The dynamic scheduling is the mapping process in this study which schedules all tasks onto different computing based on their given ranks or orders. The task scheduling scheme is an iterative model where the objective of each task will improve during the runtime of execution. The scheduler only handles computing node failure tasks instead of offloading failure. The study only considered the node failure tasks, which are transitory failed during their process. This study devises a task scheduling Algorithm 3 which takes a sorted list of tasks and resources as an input. We only determined the communication offloading and downloading time of tasks in the system. As we assumed, we have fixed bandwidth for the offloading and downloading task data between users and servers in the article. The study exploited the scheduler in which tasks are received in random format and stored in the queue before being scheduled in the system. All steps of Algorithm 3defined as below. • The workload of all tasks must be less than the capacity of the computing nodes before scheduling in the network as defined between 2 and 5 steps. • All tasks are sorted based on sequence-1 and sequence-2. • If the task deadline is satisfied at any node without wasting any resource, the objective of each task is calculated based on Equation (7). • There are three statuses of each task with different activities, such as start, i.e., status-1, and status = 2 shows progress, and status shows the finished process at a particular node as defined in 9 to 20 steps. • If any task is incurred with transient failure, it will be added to the failure list. • The failure list will recover tasks based on the failure aware method with their detection, retry, and failure aware strategies. • However, due to the dynamic environment, the initial solution is not optimal. Therefore, the searching method searches for the optimal solution based on the current values of the network. • The searching method will improve the initial solution of tasks from the neighborhood structure. • The neighborhood structure consists of different objective functions of each task in the dynamic environment, where the scheduler will choose the best one via the linear searching method. • The scheduler will control both the failure and dynamic environment for all tasks in the system. Sensors 2022,22, 5937 18 of 30 Algorithm 3: Task Scheduling Scheme Input:results[i∈N,k∈K]; ,ek 1begin 2foreach (vi=1 to N ∈A) do 3Initially all tasks are sorted according to Sequence-1; 4if (Wi←k≤ek)then 5Sort all tasks based on deadline order according to Sequence-2; 6if (i ←Fi≤di)then 7i←k←Ttotal; 8Calculate the initial scheduling based on equation (7); 9if (statusi=1)then 10 Return Status=Started; 11 else 12 Failure[i,k,in f o]←i←k; 13 Return Status=2; 14 foreach (i=1 to Failure[i,k,in f o])do 15 if (Failure[i,k,in f o]6=empty) then 16 Call Transient Failure Method; 17 Call Searching Method; 18 Optimize: Ttotal ←T∗ total; 19 Calculate the new scheduling based on in f o; 20 i←k←T∗ total; 21 Status=3; 22 End of Failure Tasks; 23 End of Inner Iteration; 24 End of Main Loop; 4.4. Solution Searching Method The study considered the dynamic environment, where users can enter and leave the network at runtime. At different timezones, the objective function of each task could be changed. In the night timezone, the user’s traffic is low, and all requests are entertained in a better way. However, during peak time, the traffic is high and the objective function will suffer from time to time. Due to this dynamic environment, the study constructs the different solutions for each task via neighborhood structure. The study proposes a novel searching engine, i.e., Algorithm 4, which determines the best solution for each task based on offloading results in the system. The study defines all steps of Algorithm 4as follows. • Initially, the study constructs the neighborhood structure of available solutions. • The initial objective of each task is Ttotal. • The initial objective of each task Ttotal will be compared with another solution when a new solution is better than the existing ones, and it will return a new optimal solution, i.e., T∗ total. • All the failure tasks or scheduled tasks always obtained an optimal solution, i.e., T∗ total in the dynamic environment. • The length of the search method is limited, and each solution is compared with a new solution in the non-linear way; when a new one is the better existing solution, the searching mechanism will stop searching. This way, the number of search steps will be reduced and remain optimal. Sensors 2022,22, 5937 19 of 30 Algorithm 4: Searching Optimal Solution Input:Neighborhood Structure: NS =G←results[i∈N,k∈K]∈A 1begin 2Ttotal ←=1 Initial Solution; 3while (Ttotal ≤NS) do 4foreach (G ←results[i∈N,k∈K]∈A) do 5Ttotal = (Ttotal,NS); 6T∗ total=Local Search (Ttotal); 7if (f(T∗ total)≤f(Ttotal)then 8Ttotal =1; 9else 10 Ttotal+ = 1; 11 Stop when criteria are met; 12 return T∗ total; 13 End 4.5. Transient Failure Aware Method To understand the failure-aware mechanism, the study discusses a case study of the real-world practice of IoT applications. There are two types of failure in distributed computing that are often considered: communication node failure and computing node failure. However, this study considers the transient failure of computing nodes in a dynamic environment. Figure 2illustrates the execution process of application G 1 with its tasks on different computing devices. The information is the history of a task from beginning to end. Each task has three statuses: s 1 shows that a task has started its execution on a particular node. s 2 illustrates that a task is still in the process of execution on any node, whereas s 3 ensures the execution of a task is finished successfully. The tasks v2 and v5 of application G 1 are scheduled on k1 . Let us assume that a task v5 failed at the computing node k1 . The detection strategy saves information on the failure of a task from the point of failure and sends it to the retry strategy. At the same time, the retry strategy tries to recover s 2 status with two possible iterations left, i.e., three before the deadline, as shown in Figure 2. The task v1 recovered with three retried operations on computing node k1 and final failure aware (FA) policy return success status to the system. In another case, a task v5 on computing node k1 failed, and the retry strategy tried possible iterations. However, a task failure exceeds its deadline limit, and then the FA will mark it as a failure. The v5 will reschedule from the scheduler from the start for execution. Algorithm 5handles the transient failure aware process of all applications robustly without violating their performances during execution. The transient failure steps of Algorithm 5are explained as below. • Initially, the failure list of all tasks saved those tasks which have failure status during scheduling. • The retry variable ret =0 and max-iteration (max-ite) has a limited three attempts to recover the transient failure aware process of tasks. • The detection will return the information of tasks when they are failed on different computing nodes. • The retry strategy will retry tasks from their point of failure with three iterations. The retry duration is only 30 s, and the gap between the first iteration and the second iteration is about 15 s. • In the end, if the tasks are retried under their deadlines, then FA returns finished status. Otherwise, it will inform the scheduler of the tasks to be scheduled again from scratch. Sensors 2022,22, 5937 20 of 30 v2 k1 info s1 s2 s3 G1 Start Progress Finished v5 k1 info s1 s2 s3 G1 di ret1 ret2 ret3 Retry FA v5 Detect v3 k2 info s1 s2 s3 G1 v1 k3 info s1 s2 s3 G1 di ret1 ret2 ret3 Retry FA v1 k1 Detect k3 Figure 2. Transient Computing Node Failure Aware. 4.6. Time Complexity of Jtos A study mentioned above shows that the JTOS framework consists of different components, such as offloading, sequencing, searching, and scheduling. Therefore, the time complexity of JTOS is determined by various elements. The time complexity of offloading is divided into three phases: parameters, normalization, and weighting, and it is equal to O(n×n) . In comparison, the time complexity of the positive ideal solution and negative ideal solution becomes n. The ranking of each task becomes n . Therefore, the total complexity of the offloading algorithm is equal to O(n×n) . The task sequence is divided into two ordered, such as Ttotal and deadline, and the time complexity becomes Olog(n×n) . The Ttotal ordering becomes n and it is the same for the deadline, which becomes n . Therefore, the total time complexity is equal to Olog(n×n) . The searching for an optimal solution for each task to the resource is equal to n×m . n is several tasks, and m matches each resource’s iteration during searching. Furthermore, all tasks are scheduled in n order to the optimal solution. The total time complexity of searching and scheduling becomes (n×m+n) . The final time complexity transient failure algorithm is divided into three parts: detection, retry, and failure aware event. The detection strategy finds the failure of tasks when they have failed status in n time, and the same for the retry method, i.e., n . The failure awareness is the decision scheme for all tasks, and then it is equal to n . Therefore, the total time complexity of transient failure is equal to n+n+n. Sensors 2022,22, 5937 21 of 30 Algorithm 5: Transient Failure Aware Schemes Input: Failure[i,k,s,in f o]∀A 1begin 2ret=0; 3Max −ite =3; 4while (ret ≤Max −ite) do 5Call Detection policy; Failure[i,k,s,in f o]∀A; 6if (si==1) then 7Apply Retry Policy; 8Failure[i,k,s,in f o]; 9if (si== 2)then 10 i←k←Failure[i,k,s,in f o]; 11 ret++; 12 if (si== 3& vi←Ttotal ≤di)then 13 Failure Aware Scheme Apply; 14 Return Status=success; 15 else 16 Return Status=Fail; 17 Three rules condition run until executed all tasks; 18 End of Main; 5. Performance Evaluation and Experimental Settings This section evaluates the performances of the proposed algorithms on the different workloads of IoT applications in a dynamic environment. The performance evaluation consists of many sub-parts, from the parameter setting to the result in the discussion. 5.1. Existing Simulation Tools Many existing simulation tools and their approaches suggested different frameworks to solve the CLIP problem in distributed computing, as shown in Table 7. These tools consist of the following parts: control plane network, framework, implementation, environment, and the problem type. The control plane is a centralized system, which handles the entire system within the system. For instance, the SDN control plane enables many BS and fog nodes and manages their management during the process. Due to offloading, these studies considered the BS, wireless access point, and Bluetooth as a network channel for offloading. The algorithm framework is most important here, and many works suggest their methods based on GA, iterative heuristic, PSO, and optimization techniques to solve the problem. The computing nodes, such as fog node and cloud, offer two kinds of servicing model based on container and virtual machines. However, recently, results are witnessed that container-type resources outperform virtual machines concerning the delay in the system. Table 7. Existing SDN Fog Cloud Simulation Tool. Tool Control Plane Network Framework Implementation Environment Problem Edge-Fog [30] SDN BSs GA Fog VMs Dynamic CILP Fog-Torch [31] SDN Wireless HEFT Fog VMs Dynamic CILP Fog-TorchII [32] Agent BSs SA Edge VMs Dynamic CILP iFogSim [33] Agent BSs Iterative Edge VMs Dynamic CILP FogDirSim [34] Controller BSs Heuristic Edge VMs Dynamic CILP FogNetSim++ [35] Controller BSs Meta-Heuristic Fog VMs Dynamic CILP FogWorkFlowSim [36] Master Node Wireless Searching Fog VMs Dynamic CILP YAFS [37] Hadoop Wireless Min-Max Fog Container Dynamic CILP FogDirMime [38] Handler Link PSO Fog Container Dynamic CILP fogbus [39] Monitor Nodes MET Fog VMs Dynamic CILP MobFogSim [39] Profiler Wireless MCT Fog Container Dynamic CILP Sensors 2022,22, 5937 22 of 30 5.2. Proposed Simulation Tool The evaluation part is crucial to evaluate the performances of applications based on proposed schemes. We exploited the multi-variance analysis of variance (ANOVA) method to find out the ideal parameters of the proposed algorithm. The simulation parameters are organized in Table 8. The workload analysis of mobile cloud applications with different task types (i.e., image, text, and video) is explained in Table 9. We use a fog cloud network based on android emulators, i.e., Amazon GenyMotion, running as virtual images on the AWS product, as a service (PaaS) on virtual machines and on the desktop machine. We also implemented a cloud-based android emulator running as a virtual image on a desktop machine as software (SaaS). We constructed a virtual cloud, i.e., edge cloud, that will be scaled up and down on-demand while emulator and configurations are performed in the cloud. We designed JTOS heuristic in the JAVA language with an advanced application programming interface (API) and tested it on Intel (R) Core (TM) i5-3475 CPU @ 3.30 GHz, 10 G Memory machine. We implemented the 64 bit X86 Amazon Machine Image (AMI) mobile cloud environment with Android 7.0 nougat for mobile cloud applications. We installed mobile applications APK’s (i.e., Android Packages) on 64 bit X86 AMI. The configuration of heterogeneous computing nodes resources defined in Table 10 with their characteristics and specifications. Table 8. Simulation Parameters. Simulation Parameters Values Windows OS Linux Amazon GenyMotion (virtual box) Centos 7 Runtime X86-64-bit AMI Languages JAVA, XML, Python Android Phone Google Nexus 4, 7, and S Experiment Repetition 160 times Simulation Duration 12 h Simulation Monitoring Every 1 h Evaluation Method ANOVA Single and Multi-Factor Amazon On Demand Service EC2 t3 Android Operating System GenyMotion Application interface Desktop, Cloud or Mobile Applications: Task Offloading Attributes Weights C5GB B0.130 S N0.062 RR 0.052 vi↔b0.1∼100 ms b↔k0.05∼50 ms ω{0.1, 0.4, 0.3, 0.5} TZ =1 12 am to 8 am TZ =2 8 am to 4 pm TZ =3 4 pm to 12 am Table 9. Workload of IoT Applications. Workload Wi(MB) W0 i(MB) Image Tasks Video Tasks Text Tasks N G1825 5.2 100 100 300 500 G2631 6.3 150 200 350 700 G3645 7.4 100 200 500 800 G4755 8.7 200 200 600 1000 Sensors 2022,22, 5937 23 of 30 Table 10 defines resource specification of computing nodes with their characteristics and features. Table 10. Heterogenous Fog Cloud Nodes Resource Specification. Resources Fog Clouds Public Cloud k1k2k3k4 ζkSmall 2 V CPU Medium 4 V CPU Large 8 V CPU Extra Large 24 V CPU CORE 1 1 2 4 ek500 GB 1000 GB 1500 GB 3000 GB Run −Time X86 X86 X86 X86 5.3. Data Performance Method We tested four benchmark mobile cloud applications; their specifications are exemplified in Table 9. We tested applications that generate data (i.e., configuration file obtained data via profiling technologies and task scheduling heuristics) of different applications via analysis of variance (ANOVA). At the same time, ANOVA is an efficient parametric technique accessible for examining algorithm-generated data of mobile cloud applications from experiments. We exploit t-tests and dependent and independent random variables in the one way ANOVA method to note the proposed method’s efficiency. To compute the recital of the JTOS, we exploit RPD (relative percentage deviation) statistical analysis. It evaluates the power consumption consumed by different parameters and frameworks, plus algorithm permutation throughout the parameter space of component calibration. The RPD estimation can be as demonstrated in the following Equation (27): RPD(%) = Ttotal −T∗ total T∗ total ×100%, (27) where Ttotal is the objective function of the article that executes the tasks via the proposed IoT tasks fog cloud architecture and JTOS algorithm. Furthermore, T∗ total is the delay optimal and scheduling in the proposed architecture in the distributed computing environment. 5.4. Baseline Approaches The study implemented the recently published article methods to compare the performance of the proposed JTOS framework based on different components. Recently, two computing models, virtual machines and container-based services, have been widely used to run various IoT applications. However, many additional features, such as service start time, pre-allocation, and post-allocation of resources, have utilization and delay effects on both applications and the system. The following research methods are assumed as baseline approaches in the experimental part. (1) Baseline 1: The experiment environment implemented existing method frameworks [ 1 – 10 ] that are considered dynamic situations during offloading and resource allocation. They suggested joint offloading and GA and PSO-aware resource allocation methods to CILP problems with the fixed environment (e.g., where users entered and left at different timezone). They considered a virtual machine-based fog cloud network and SDN controller with different base stations and wireless access points. (2) Baseline 2: The experiment environment implemented existing method frameworks [11,14,19,22,39] that are considered dynamic situation during offloading and resource allocation. They suggested joint offloading and iterative and search-aware greedy heuristics-based resource allocation methods to CILP problems with the fixed environment (e.g., where users entered and left at different timezone). They considered a docker container-based fog cloud network and SDN controller with different base stations and wireless access points. Sensors 2022,22, 5937 24 of 30 5.5. Sdn Fog Nodes Offloading Scenario Sdn Controller Configuration NetSim [ 40 ] is a widely used simulation modeling for the SDN control plane to handle distributed fog and cloud computing via different base stations. This study extended the SDN class control plane and base-station classes as an abstract class from the NetSim model for the experiment purpose in the proposed work. The SDN control plane monitors the channel capacity and resource of computing nodes. The SDN control plane is a management manager that helps offload the engine to reduce the total delay of applications. However, in different timezones, the uncertainty of applications could be added due to traffic and load on nodes. The study considered the following assumptions during offloading for all tasks. 1. Each task viis only finished and its execution or fail once it is assigned to any node. 2. Each task has a different total delay in a different timezone. 3. The study only considered the node failure and ignored the base-station failure. 4. The task migration and pre-emption are not allowed. 5. The dynamic environment only considered enter and leaving requests in the network. It is different from mobility, and mobility is not considered in this study. 6. The failure tasks reschedule from scratch if they completely failed on their computing nodes. 7. The roundtrip time between user and base station and base station to computing nodes is fixed in the same timezone. However, in a different timezone, there would be a different network roundtrip during offloading. The offloading of tasks in different timezones has an impact on the objective of each application. Whereas in a dynamic environment, users with different applications can enter and leave at any time. However, the traffic of users becomes dense from 8 am to 10 pm. Therefore, in the simulation, the timezone is divided into three zones, such as TZ = 1, i.e., 12 am to 8 am; this time, the traffic is more minor, and total delay (communication delay and computation delay) becomes less. Whereas TZ = 2, i.e., 8 am to 4 pm, the traffic becomes very high, applications’ objective function has a huge impact on their performances. In the final timezone TZ = 3, i.e., 4 pm to 12 am, the traffic becomes less than daytime peak time. In the joint offloading and allocation optimization problem, existing studies only consider the random and linear offloading without considering the timezone. Furthermore, they offload tasks based on task size and required CPU requirements. However, they did not consider the deadline and total delay of applications in their offloading scheme. Figure 3shows FMCM in JTOS gained lower delay of application with random tasks as compared to all baseline approaches. The study evaluated the offloading of all applications at different timezones with JTOS with a different number of arbitrary tasks. Hence it is proved that the multi-criteria offloading in a dynamic environment with different timezones outperforms all conventional offloading methods in terms of RPD%. In the following way, the proposed FMCM works better than the existing offloading method: (1) All existing offloading methods in baseline 1 and baseline 2 just focused on compute-intensive or data-intensive task offloading in the full offloading scheme. Whereas, in full offloading, all tasks were offloaded in the associated external server for execution. The task offloading scheme could not meet the QoS requirement of applications, such as deadline and objective function based on data size. (2) The study proposed a fuzzy multi-criteria method that offloads all tasks based on data size, execution time, total delay, and deadline of each application in a different timezone. Different timezones have different offloading results; FMCM in JTOS always adopts the environment and makes an offloading decision based on the current values in the system. Sensors 2022,22, 5937 25 of 30 200 400 TZ=1: Number of Random Tasks 0.0123 0.01235 0.0124 0.01245 0.0125 0.01255 0.0126 RPD% Baseline1 JTOS Baseline2 200 400 TZ=2 Number of RandomTasks 0.044 0.046 0.048 0.05 0.052 0.054 0.056 0.058 0.06 0.062 RPD% Baseline1 JTOS Baseline2 200 400 TZ=3 Number of Random Tasks 0.0123 0.01235 0.0124 0.01245 0.0125 0.01255 0.0126 RPD% Baseline1 JTOS Baseline2 200 400 600 800 All Applications Random Tasks 0.02 0.025 0.03 0.035 0.04 0.045 0.05 0.055 0.06 0.065 0.07 RPD% Baseline1 JTOS Baseline2 Figure 3. Offloading Performance In Different Timezones. 5.6. Proposed Task Scheduling Performance against State-of-the-Art Approaches Scheduling (e.g., resource allocation) is always a challenging problem in the dynamic environment. Whereas ordering of tasks, i.e., ranks, searching for particular resources, and transient failure often happen in the background. Due to different requirements of tasks in size, deadline, and additional computing speed of nodes, they are assigning tasks appropriately based on their ranks. For instance, v1 has a small deadline compared to v2 , which has both a large size and deadline. Therefore, in this case, the scheduler should not assign them to power machines for their execution. In this way, a lot of resources are wasted. We implemented the existing GA, PSO, and HEFT-based methods as existing studies used in their resource allocation model and evaluated all applications’ performance in terms of RPD% in the dynamic environment. Figure 4shows the performance of all random tasks of different applications at different timezones onto heterogeneous computing nodes. Initially, all algorithms rank to each task before scheduling in the network, then search the appropriate resource for the execution of every task. Different searches such as liner search informed search and bi-searched are implemented as baseline approaches in the simulation tool. Figure 4illustrates that JTOS outperforms all existing resource allocation policies in terms of RPD% of applications in different timezones with their characteristics. These are the reasons why JTOS outperforms all existing baseline approaches: (1) The first reason is that they exploited a simple sorting strategy, i.e., all tasks are ranked based on their size. However, this way, they will violence their deadline during scheduling. (2) The second reason, all tasks are sorted based on the available resource of machines. However, in this way, the slack time gap between tasks leads to lateness for each task. (3) All existing resource allocation schemes were used for exploration and exploitation searching during scheduling for all applications. However, due to the ample space of candidate searching, the scheduling will face a lot of overhead during the assignment of