AIOTI Strategic Reseach and Innovation Agenda - Advancing Next-Generation IoT and Edge Computing Research and Innovation
Abstract
The AIOTI SRIA introduces a mission-oriented research and innovation approach that answers societal and market needs, maintains, and extends industrial leadership, protects the environment, ensures security, privacy, safety, and energy efficiency solutions, while prioritising research and innovation capabilities and education. The AIOTI SRIA addresses research and innovation priorities for future IoT and edge computing technologies and applications that will drive changes across industrial sectors, the European economy, and society. The priorities include convergence with next-generation Tactile IoT, decentralised and distributed architectures, IoT knowledge-driven edge processing, artificial intelligence (AI) and trustworthiness.
Full text
© AIOTI. All rights reserved.
© AIOTI. All rights reserved. 2 Executive Summary The Alliance for IoT and Edge Computing Innovation (AIOTI) Strategic Research and Innovation Agenda (SRIA) aims both at identifying key Internet of Things (IoT) and edge computing technologies and applications research priorities and at providing a vision on how the future of the IoT domain will look like, up until the 2030 timeframe. The AIOTI SRIA is a roadmap (2023-2030) for future IoT and edge computing research and innovation actions in Europe, proposing specific themes, sub-themes, and priorities, which help identifying gaps and areas where research and innovation advancements are most needed. The identified research priorities form the reference for concrete actions to be implemented in different research programmes by various stakeholders, such as industry, researchers, small-andmedium enterprises, academia, entrepreneurs, the public sector, and the overall society. The SRIA delivers a direct contribution in aligning with the United Nations Sustainable Development Goals (SDG) needs and the European Green Deal objectives, while developing international cooperation to solve global challenges using IoT and cloud computing technologies. The research and innovation efforts in IoT and edge computing require to be based on a longterm programming approach that provides continuity across technology and applications efforts over several years. The AIOTI SRIA introduces a mission-oriented research and innovation approach that answers societal and market needs, maintains, and extends industrial leadership, protects the environment, ensures security, privacy, safety, and energy efficiency solutions, while prioritising research and innovation capabilities and education. The AIOTI SRIA addresses research and innovation priorities for future IoT and edge computing technologies and applications that will drive changes across industrial sectors, the European economy, and society. The priorities include convergence with next-generation Tactile IoT, decentralised and distributed architectures, IoT knowledge-driven edge processing, artificial intelligence (AI) and trustworthiness. The IoT and edge computing technologies have evolved rapidly in addressing the grand challenge of developing human centred IoT technologies and applications, which require increased communication and coordination amongst policy makers, end-users, and experts in the IoT and edge computing fields. The extensive use of IoT and edge computing in different industrial sectors and the move from cloud to edge processing must be accompanied by new distributed architectures and end-toend (E2E) IoT security. The convergence of connectivity, IoT, edge computing, AI, and Distributed Ledger Technologies (DLT) will be essential to next-generation Internet applications and advancements. The topics presented in the AIOTI SRIA are aligned with the issues addressed by other European partnerships to improve the European IoT ecosystem's sustainability and manage human wellbeing, particularly for developing safe, secure, and trustworthy IoT and edge computing technologies.
© AIOTI. All rights reserved. 3 Table of Content 3.1 Technological developments ....................................................................................................................... 16 3.2 Main Trends, Issues and Challenges............................................................................................................. 19 3.3 Research Priorities Timeline............................................................................................................................ 20 4.1 Technological developments ....................................................................................................................... 21 4.2 Main Trends, Issues and Challenges............................................................................................................. 22 4.3 Research Priorities Timeline............................................................................................................................ 23 5.1 Technological developments ....................................................................................................................... 24 5.2 Main Trends, Issues and Challenges............................................................................................................. 25 5.3 Research Priorities Timeline............................................................................................................................ 28 6.1 Technological developments ....................................................................................................................... 29 6.2 Main Trends, Issues and Challenges............................................................................................................. 30 6.3 Research Priorities Timeline............................................................................................................................ 31 7.1 Technological developments ....................................................................................................................... 32 7.2 Main Trends, Issues and Challenges............................................................................................................. 33 7.3 Research Priorities Timeline............................................................................................................................ 35 8.1 Technological developments ....................................................................................................................... 36 8.2 Main Trends, Issues and Challenges............................................................................................................. 38 8.3 Research Priorities Timeline............................................................................................................................ 38
© AIOTI. All rights reserved. 4 9.1 Technological developments ....................................................................................................................... 39 9.2 Main Trends, Issues and Challenges............................................................................................................. 40 9.3 Research Priorities Timeline............................................................................................................................ 42 10.1 Technological developments ....................................................................................................................... 43 10.2 Main Trends, Issues and Challenges............................................................................................................. 45 10.3 Research Priorities Timeline............................................................................................................................ 46 11.1 Technological developments ....................................................................................................................... 47 11.2 Main Trends, Issues and Challenges............................................................................................................. 48 11.3 Research Priorities Timeline............................................................................................................................ 49 12.1 Technological developments ....................................................................................................................... 50 12.2 Main Trends, Issues and Challenges............................................................................................................. 51 12.3 Research Priorities Timeline............................................................................................................................ 53 13.1 Technological developments ....................................................................................................................... 54 13.2 Main Trends, Issues and Challenges............................................................................................................. 55 13.3 Research Priorities Timeline............................................................................................................................ 56 14.1 Technological developments ....................................................................................................................... 57 14.2 Main Trends, Issues and Challenges............................................................................................................. 57 14.3 Research Priorities Timeline............................................................................................................................ 59 15.1 Technological developments ....................................................................................................................... 60 15.2 Main Trends, Issues and Challenges............................................................................................................. 61 15.3 Research Priorities Timeline............................................................................................................................ 62 16.1 Technological developments ....................................................................................................................... 63 16.2 Main Trends, Issues and Challenges............................................................................................................. 64 16.3 Research Priorities Timeline............................................................................................................................ 65
© AIOTI. All rights reserved. 5 17.1 Technological developments ....................................................................................................................... 66 17.2 Main Trends, Issues and Challenges............................................................................................................. 66 17.3 Research Priorities Timeline............................................................................................................................ 67 18.1 Technological developments ....................................................................................................................... 68 18.2 Main Trends, Issues and Challenges............................................................................................................. 69 18.3 Research Priorities Timeline............................................................................................................................ 71
© AIOTI. All rights reserved. 6 Table of Figures Figure 1 Edge IoT research and innovation topics ....................................................................................................... 12 Figure 2 Edge computing granularity across the computing continuum .................................................................. 17 Figure 3 Edge cloud continuum representation ........................................................................................................... 18 Figure 4 Xcontinuum paradigm ................................................................................................................................... 21 Figure 5 Matter and Thread edge IoT connectivity ...................................................................................................... 27 Figure 6 Edge IoT digital twin technology evolution ..................................................................................................... 37 Figure 7 Internet of things senses .................................................................................................................................... 44 Figure 8 Edge IoT system dependability characteristics.............................................................................................. 68
© AIOTI. All rights reserved. 7 List of Tables Table 1 IoT and edge computing research priorities ................................................................................................... 20 Table 2 IoT edge and X-Continuum research priorities ............................................................................................... 23 Table 3 Intelligent connectivity research priorities ....................................................................................................... 28 Table 4 Energy-efficient intelligent IoT and edge computing systems research priorities ...................................... 31 Table 5 Heterogeneous cognitive edge IoT mesh research priorities ........................................................................ 35 Table 6 IoT Digital Twins, Modelling and Simulation Environments research priorities ............................................. 38 Table 7 Swarm systems research priorities .................................................................................................................... 42 Table 8 Internet of things senses research priorities ..................................................................................................... 46 Table 9 Decentralised and distributed edge IoT systems research priorities ............................................................ 49 Table 10 Federated learning and AI for edge IoT systems research priorities .......................................................... 53 Table 11 Operating systems and orchestration concepts for edge IoT systems research priorities ....................... 56 Table 12 Dynamic programming tools and environments for edge IoT systems research priorities ...................... 59 Table 13 Heterogeneous edge IoT systems integration research priorities ............................................................... 62 Table 14 Edge IoT sectorial and Cross-Sectorial Open Platforms research priorities ............................................... 65 Table 15 IoT Verification, Validation and Testing Methods research priorities .......................................................... 67 Table 16 IoT Trustworthiness and Edge Computing Systems Dependability research priorities .............................. 71
© AIOTI. All rights reserved. 8 List of Acronyms AI Artificial Intelligence AIOp Artificial Intelligence Operations AIOTI Alliance for IoT and Edge Computing Innovation AIOTI SRIA AIOTI Strategic Research and Innovation Agenda ANDSF Access Network Discovery and Selection Function ANN Artificial Neural Network API Application Programming Interface AR Augmented Reality ASIC Application-Specific Integrated Circuit ASSP Application-Specific Standard Product AV Audio Video BS Base Station CBOR Concise Binary Object Representation CN Cognitive Network CPU Central Processing Unit D2D Device-Device DL Deep Learning DLT Distributed Ledger Technologies DMO Direct Mode Operation DSP Digital Signal Processing DT Digital Twin E2E End to End ENISA European Union Agency for Network and Information Security FL Function Level FPGA Field-Programmable Gate Array FW Firmware GPU Graphics Processing Unit H2H Host to Host, Human to Human HE Horizon Europe HMI Human-Machine Interface HW Hardware IaaS Infrastructure as a Service ID Identifiers Definition IETF OSCORE Internet Engineering Task Force Object Security for Constrained RESTful Environments IoT Internet of Things IoT DT IoT Digital Twin IoTS Internet of Things Senses IT/OT Information/Operation Technology JSON-LD JavaScript Object Notation for Linked Data
© AIOTI. All rights reserved. 9 KPI Key Performance Indicator () LDS Laser-Direct Structuring LoRa Long Range Radio LPWAN Low-Power Wide Area Network M2H Machine to Human M2M Machine to Machine MEC Multi-access Edge Computing ML Machine Learning MLOp Machine Learning Operations mMTC Machine Type Communication MR Machine Reasoning NPN Non-Public-Network NPU Network Processing Unit O-RAN Open Radio Access Network OEM Original Equipment Manufacturer OS Operating System OTA Over-the-Air PaaS Platform as a Service PCI Peripheral Component Interconnect PLC Programmable Logic Controller QoI Quality of Information QoS Quality of Service RAN Radio Access Network RAT Radio Access Technology RF Radio Frequency RISC-V reduced Instruction Set Computer principles 5th generation RU (Multi-)Resource Unit SCADA Supervisory Control and Data Acquisition SDG Sustainable Development Goals SDO Service Data Objects SLO Service Level Objective SoC System-on-a-Chip SRE Site Reliability Engineers SRIA Strategic Research and Innovation Agenda SW Software TPU Tensor Processing Unit TSN Time-Sensitive Networking VR Virtual Reality Wi-Fi Wireless Fidelity XAI Explainable AI XR eXtended Reality VV&T Verification, Validation and Testing
© AIOTI. All rights reserved. 16 3. IoT and Edge Computing Granularity The concept of data processing, analysis and storage using the centralised cloud computing paradigm, comprising of a set of technologies, infrastructure, services, and applications, is no longer aligned with the always increasing demand of cellular, e.g., massive Machine Type Communication (mMTC), services and wireless intelligent connectivity usage scenarios 9 . Such scenarios aim at ensuring that the always growing number of IoT devices connected to the network can operate according to their specification, do not operate under conditions limiting the capabilities, and fulfil the expected Quality of Service (QoS). A new approach is therefore looked for, so to avoid creating silos and issues regarding both connectivity and real-time data processing and storage. Edge computing provides the mechanisms for distributing data processing and redefines the IoT landscape by moving data processing and analytics at the edge by using AI/ML techniques and ensuring an advanced level of embedded security. Edge computing allow an effective deployment of real-time applications, considering that the processing is performed close to the data source. It also reduces the order of magnitude of transmitted data, by not transmitting the extensive amount of raw data created by IoT devices, rather just sending smaller amount of data, thanks to storage and local processing capabilities. Several benefits can be derived by this approach, e.g., a decrease in needed communication bandwidth and data storage requirements, as well as in the data attack surface, thus Improving security, privacy data protection, and finally also reducing the overall energy consumption. Edge computing solutions are implemented through different deployment forms such as cloudlet 10 , dew 11 , mobile edge 12 , fog computing 13 , etc., that have been developed over the last few years. 3.1 Technological developments Edge computing is defined as a paradigm that can be implemented using different architectures built to support an IoT distributed infrastructure of data processing (signals, image, voice, etc.) with edge IoT devices operating close to the points of collection (data sources) and utilisation. The edge computing distributed paradigm provides computing capabilities to the IoT nodes and devices of the edge of the network (or edge domain) to improve the performance (energy efficiency, latency, etc.), operating cost, security and reliability of applications and services. Edge computing performs data analysis by minimising the distance between IoT nodes and devices and reducing the dependence on centralised resources that serve them while minimising network hops. 9 B. Raaf et al., “Key technology advancements driving mobile communications from generation to generation”, in Intel Technology Journal 18 (1), 2014. 10 D. Bhatta and L. Mashayekhy, "Physics-Inspired Mobile Cloudlet Placement in Next-Generation Edge Networks," 2022 IEEE International Conference on Edge Computing and Communications (EDGE), Barcelona, Spain, 2022, pp. 159-168, doi: 10.1109/EDGE55608.2022.00031. 11 Z. S. Ageed, S. R. M. Zeebaree, M. A. M. Sadeeq, R. K. Ibrahim, H. M. Shukur and A. Alkhayyat, "Comprehensive Study of Moving from Grid and Cloud Computing Through Fog and Edge Computing towards Dew Computing," 2021 4th International Iraqi Conference on Engineering Technology and Their Applications (IICETA), Najaf, Iraq, 2021, pp. 68-74, doi: 10.1109/IICETA51758.2021.9717894. 12 J. Lee and W. Na, "A Survey on Mobile Edge Computing Architectures for Deep Learning Models," 2022 13th International Conference on Information and Communication Technology Convergence (ICTC), Jeju Island, Korea, Republic of, 2022, pp. 2346-2348, doi: 10.1109/ICTC55196.2022.9952954. 13 I. Martinez, A. S. Hafid and A. Jarray, "Design, Resource Management, and Evaluation of Fog Computing Systems: A Survey," in IEEE Internet of Things Journal, vol. 8, no. 4, pp. 2494-2516, 15 Feb.15, 2021, doi: 10.1109/JIOT.2020.3022699.
© AIOTI. All rights reserved. 17 IoT edge computing capabilities include a steady operating procedure across different platform infrastructures to deliver processing services to remote IoT devices, application integration, orchestration, and service delivery requirements. The edge computing technologies are meant to properly consider hardware (HW) limitations and cost constraints, to effectively handle limited or intermittent network connections, and to implement methods to satisfy the most diverse requirement sets, e.g., IoT applications requiring low latency or greatly differing in data rates. For intelligent IoT applications, the edge computing concept is mirrored in the development of different edge computing levels (micro, deep, meta), that incorporate the computing and intelligence continuum from the sensors/actuators, processing units, controllers, gateways, onpremises servers to the interface with multi-access, fog, and cloud computing. Edge IoT devices and their functions cover the edge computing, communication, and data analytics capabilities, so to make it possible to call such devices smart or intelligent. An edge IoT device is designed around the computing units (CPUs, GPUs/FPGAs, ASICs platforms, AI accelerators/processing), communication network, storage infrastructure and the applications (workloads) that run on it. The edge domain can scale from a few IoT devices to tens of thousands of IoT devices, distributed in different locations with a unique identity. The IoT devices in the edge computing environment are physically separated and connected using wireless or wired connections in different kinds of topologies, e.g., by using a mesh network 14 . The IoT edge devices can even operate semi-autonomously, especially in the case of sensors spread in remote locations, using remote management administration tools. The edge devices can be optimised based on different aspects, like processing, memory, energy, connectivity, size, cost. Their performance and capabilities are of course constrained by these parameters. A description of the micro-, deepand meta-edge concepts is provided in the following sections and aligned with other European partnerships 15 . The granularity of the edge computing environments is illustrated in Figure 2. Figure 2 Edge computing granularity across the computing continuum The micro-edge describes the intelligent sensors, machine vision, and IoT devices that generate insight data and are implemented using microcontrollers built around processors that can well 14 Karthika K.C, "Wireless mesh network: A survey," 2016 International Conference on Wireless Communications, Signal Processing and Networking (WiSPNET), Chennai, India, 2016, pp. 1966-1970, doi: 10.1109/WiSPNET.2016.7566486. 15 Vermesan, O., Pétrot, F., Coppola, M., Schneider, M. and Höß, A. (Authors). Industrial AI Technologies for Next-Generation Autonomous Operations with Sustainable Performance, in "Intelligent Edge-Embedded Technologies for Digitising Industry" (Chapter 1), River Publishers Series in Communications, June 2022. ISBN: 9788770226110, e-ISBN: 9788770226103. Online at: https://www.riverpublishers.com/pdf/ebook/chapter/RP_9788770226103C1.pdf
© AIOTI. All rights reserved. 18 cope with costs and power consumption constraints. The distance from the data source generated by the sensors is minimised. The compute resources process this raw data in line and produce insight data with minimal latency. The HW devices of the micro-edge physical sensors/actuators generate from raw data insight data and/or actuate based on physical objects by integrating AI-based elements into these devices and running AI-based techniques for inference and self-training. Intelligent micro-edge allows IoT real-time applications to become ubiquitous and merged into the environment where various IoT devices can sense their environments and react fast and intelligently with a very low power budget energy-efficient gain. AI capabilities integrated into IoT devices significantly enhance their capabilities (functionality, performances, low latency, low power consumption, high processing power) and usefulness, especially when the full power of these networked devices is harnessed – a trend called AI on edge 16 , 17 , 18 . The deep-edge comprises intelligent controllers like Programmable Logic Controllers (PLC), Supervisory control and data acquisition (SCADA) elements, machine vision connected embedded systems, networking equipment, gateways and computing units that aggregate data from the sensors/actuators of the IoT devices that generate data. Deep edge processing resources are implemented with performant processors and microcontrollers such as Intel iseries, Atom, ARM M7+, etc., including CPUs, GPUs, Tensor Processing Units (TPU), and ASICs. The system architecture, including the deep edge, depends on the envisioned functionality and deployment options considering that these devices' cores are controllers: PLCs, gateways with cognitive capabilities that can acquire, aggregate, understand, react to data, exchange, and distribute information 19 . The meta-edge integrates processing units, typically located on-premises, implemented with high-performance embedded computing units, edge machine vision systems, edge servers (e.g., high-performance CPUs, GPUs, FPGAs, etc.) that are designed to handle computeintensive tasks, such as processing, data analytics, AI-based functions, networking, and data storage 20 . This classification is closely related to the distance between the data source and processing, impacting overall latency. A high-level representation of the edge cloud continuum is represented in Figure 3 and a rough estimation of the communication latency and the distance from the data sources are presented below. Figure 3 Edge cloud continuum representation • Micro-edge latency below 1 ms, range from mm to 15 m. • Deep-edge latency below 2-5 ms, range up to 1 km. 16 Vermesan, O., Pétrot, F., Coppola, M., Schneider, M. and Höß, A. (Authors). Industrial AI Technologies for Next-Generation Autonomous Operations with Sustainable Performance, in "Intelligent Edge-Embedded Technologies for Digitising Industry" (Chapter 1), River Publishers Series in Communications, June 2022. ISBN: 9788770226110, e-ISBN: 9788770226103. Online at: https://www.riverpublishers.com/pdf/ebook/chapter/RP_9788770226103C1.pdf 17 Dhiraj Joshi; Nirmit Desai; Shyama Prosad Chowdhury; Wei‐Han Lee; Luis Bathen; Shiqiang Wang; Dinesh Verma, "AI at the Edge: Challenges, Applications, and Directions," in IoT for Defense and National Security, IEEE, 2023, pp.133-160, doi: 10.1002/9781119892199.ch9. 18 A. Munir, E. Blasch, J. Kwon, J. Kong and A. Aved, "Artificial Intelligence and Data Fusion at the Edge," in IEEE Aerospace and Electronic Systems Magazine, vol. 36, no. 7, pp. 62-78, 1 July 2021, doi: 10.1109/MAES.2020.3043072. 19 Vermesan, O., Pétrot, F., Coppola, M., Schneider, M. and Höß, A. (Authors). Industrial AI Technologies for Next-Generation Autonomous Operations with Sustainable Performance, in "Intelligent Edge-Embedded Technologies for Digitising Industry" (Chapter 1), River Publishers Series in Communications, June 2022. ISBN: 9788770226110, e-ISBN: 9788770226103. Online at: https://www.riverpublishers.com/pdf/ebook/chapter/RP_9788770226103C1.pdf 20 Vermesan, O., Pétrot, F., Coppola, M., Schneider, M. and Höß, A. (Authors). Industrial AI Technologies for Next-Generation Autonomous Operations with Sustainable Performance, in "Intelligent Edge-Embedded Technologies for Digitising Industry" (Chapter 1), River Publishers Series in Communications, June 2022. ISBN: 9788770226110, e-ISBN: 9788770226103. Online at: https://www.riverpublishers.com/pdf/ebook/chapter/RP_9788770226103C1.pdf
© AIOTI. All rights reserved. 19 • Meta-edge latency below 10 ms, range up to 50 km. • MEC latency 10-5 ms, range up to 75 km. • Fog latency 10-20 ms, range up to 100 km. • Far-edge latency 20-50 ms, range up to 200 km.Cloud and data centres latency 50-100 ms, range up to 1000 km. IoT application deployments at the edge can provide more energy-efficient processing solutions by integrating various computing architectures at the edge (e.g., neuromorphic chips, CPUs, GPUs, ASICs, FPGAs), reducing data traffic and data storage. Edge computing reduces the latency and bandwidth constraints of the communication network and Internet connectivity by processing locally and distributing computing resources, intelligence, and software stacks among the computing network IoT devices. 3.2 Main Trends, Issues and Challenges Edge computing moves service provisioning closer to producers and users of such services. It provides low-latency, mobility support, data analytics close to the data source, and reduced energy consumption. In IoT, there are many expectations for AI-based applications. AI processing happens mainly in the cloud. With the improvement of AI enabling technologies, AI processing is moving into IoT devices. Intelligent edge devices will integrate software to train the AI model for different applications and executable software that runs the AI algorithms on the IoT devices. Machine Learning (ML) and Deep Learning (DL) will have unique roles for intelligent IoT devices at the edge. As a subset of AI, ML enables machines to recognise patterns and make predictions by analysing data instead of using explicit programming. For a higher level of accuracy, ML can be upgraded to DL. DL is a class of ML algorithms that progressively uses a hierarchy of multiple layers to extract higher-level features from the raw input. With DL, a computer can train itself with an extensive data set collected for this purpose. The result is an Artificial Neural Network (ANN) that contains all the information to carry out the task. The ANN uses the knowledge acquired in training to infer data features from new incoming data.
© AIOTI. All rights reserved. 20 Achieving a middleware architecture that integrates all the levels of the edge granularity and manage to effectively handle heterogeneous IoT resources, including networking and computing, is a new challenge for IoT/edge computing, leading to a unified networking and computing architecture. Deployment of DL at the IoT edge has several challenges such as identification of leading performance indicators of edge DL algorithms, exploitation of trade-offs between the indicators to improve the efficiency of the algorithms, effective combination of the simplified DL models at the IoT edge with the DL model in the cloud, coordination between training and inference and energy-efficient deployment of DL at the IoT edge. 3.3 Research Priorities Timeline Table 1 IoT and edge computing research priorities Topic Short Term Medium Term Long term 2023-2024 2025-2027 2028-2030 AI Develop effective AI-based solutions that can exploit the benefit of DL and Federated Learning In different ways for the different needs of the Edge Granularity How to deliver an IoT/edgecloud continuum that takes care of all the specificities of all the Involved layers in a smooth way. Distinction of the several Edge Granularity totally transparent to both systems and humans (final users). Accelerators Adapt, by using dedicated accelerators like ASICs or reconfigurable ones like FPGAs (depending on the constraints and requirements in focus), the different needs of the different edge granularity. Exploit the smaller technology node so to embed in edge devices advanced accelerators functionalities. No distinction possible anymore between FPGA and dedicated HW. New HW architectures Promising new pre-commercial architectures (like neuromorphic computing ones) will be assessed against existing traditional architectures based on CPU/GPUs or vector processors. Edge devices containing novel HW architectures that can run AI algorithms so to achieve better results than the currently existing HW architecture based on CPU and GPU. Synergy between legacy and novel architectures and usage of one or the other according to the real-time needs of the use case in focus, taking into consideration also real-time selfadaptation to the different workloads. Security Design a secure-byconstruction distributed architecture that can impact all the different granularities of the Edge. Deploy such architecture across different verticals and industrial domains. Take on board quantum and post-quantum novel approach to ensure the highest possible security level at all layers and kind of devices in a transparent way for final users.
© AIOTI. All rights reserved. 21 4. IoT Edge and X-Continuum Paradigm Distributed IoT digital platforms and infrastructures for collecting, processing, computing, communicating, and running analytics are evolving towards an interconnected ecosystem allowing complex applications to be executed from IoT edge to high-performance computing capabilities. The IoT digital continuum includes computing resources placed at optimal processing points in the IoT system from the cloud data centre to edge IoT systems and endpoint devices that integrate E2E capabilities such as sensing, processing, connectivity, computing, storage, intelligence, security, safety, and privacy as illustrated in Figure 4. Figure 4 Xcontinuum paradigm Implementing IoT E2E capabilities in such a distributed continuum is challenging and requires reconciling different application requirements and constraints with IoT infrastructure design choices including the energy consumption continuum. One main challenge in this domain is how to accurately reproduce relevant behaviours of IoT applications workflow and representative settings of the IoT physical infrastructure, including AIbased learning/training and inferencing underlying the complex distributed continuum from micro-, deep-, meta-edge to cloud. 4.1 Technological developments IoT data processing, computing and communication no longer lean only on traditional approaches that send all data to centralised cloud facilities for processing, rather leverage on the distributed resources close to where the data is generated, so to extract insights in real-time while keeping efficient resource usage and preserving security and privacy constraints. The IoT digital continuum seamlessly combines resources, processing, and storage capabilities, and services at the centralised cloud, IoT edge, and in-transit, along the distributed data path.
© AIOTI. All rights reserved. 22 The development of different dedicated systems for data processing on each component of the continuum lacks a holistic approach and a trustworthy architecture for implementing future IoT ubiquitous computing systems. One of the main challenges in this context is related to the complexity of deploying large-scale, real-life IoT applications on such heterogeneous infrastructures, which breaks down to configuring many system-specific parameters and harmonising many requirements or constraints regarding processing, communication latency, storage, energy, network efficiency, interoperability, mobility, security, and data privacy. The IoT digital continuum requires an IoT digital infrastructure jointly used by complex application workflows, commonly combining real-time data generation, processing, computation, and analytics. The combined edge-cloud infrastructures deliver IoT applications' virtual distributed-centralised computing and storage resources to perform stream and batch analytics on comprehensive historical data, AI model training, and complex simulations. IoT edge infrastructures complement these virtual resources with distributed computing and storage capabilities close to IoT intelligent devices that sense the environment. The IoT elements include nodes, programable logic controllers, gateways, routers, micro-servers, and embedded high-performance units distributed across the edge continuum and used for final or in-transit processing on data aggregated from multiple IoT edge devices to reduce further data volumes that need to be transferred and processed in the clouds and data centres. Lightweight frameworks, based on message brokers using IoT lightweight protocols, enable distributed and hierarchical processing and intelligent aggregation, minimising latency, and bandwidth usage. The IoT digital continuum requires an efficient architecture that focuses on both horizontal and vertical resource distribution in the IoT edge-to-cloud continuum to secure seamless E2E services across the whole continuum. As a result, IoT system architectures and management mechanisms are needed to seamlessly encompass computing, storage, and networking. 4.2 Main Trends, Issues and Challenges The implementation of IoT continuum-X requires research that integrates ideally open-source intelligence tools, HW and SW platforms, and systems, thus addressing the non-functional aspects of IoT systems with multiple elements as part of the continuum-X. Another issue is how to guarantee embedded interoperability into continuum-X with transparent aggregation, distribution and logging of participants and system activities when using IoT distributed edge computing services. The research effort must address how to collect simultaneously a huge amount of data from a very disperse, heterogeneous and numerous numbers of sources when real-time decisions are to be taken. Future research activities include addressing the resource management for continuum-X capabilities by managing the resources on the processing points and identifying the optimal E2E capabilities and their relevance, depending on the IoT application context. Work is required on providing solutions for dynamic placement of edge computing platforms to minimise the network access delay depending on the context, IoT activity at the edge and IoT application scalability requirements.
© AIOTI. All rights reserved. 23 Further research is needed to provide collaborative and hierarchical computing to distribute the IoT devices' workload among various distributed processing/computing points and allocate optimal resources to each device based on the continuum-X capabilities and requirements. The research in the following years should address the infrastructure edge and the device edge together and provide solutions for optimising the integration of the two. Research in context-aware, scalable heterogeneous continuum-X optimisation for IoT applications is required due to the tremendous diversity of workloads and applications being run across a variety of different IoT edge devices. Security, safety, and monitoring of critical E2E capabilities of IoT systems are essential for many IoT applications. The integration of solutions for addressing these elements of the continuum-X holistically into IoT platforms should be prioritised. To provide continuum-X implementation flexibility on different IoT platforms, new research directions should address developing and deploying containers at the edge that address challenges related to connectivity, distribution, and synchronisation, and leverage the different IoT architecture to deploy new solutions and applications. Developing edge AI for IoT applications requires offloading AI models learning/training to IoT devices to retrain the AI models locally with near real-time data collected by the IoT devices for efficient data processing. Future research is required to identify how the computational tasks, AI models, inputs, parameters, and weights, are instantiated on IoT devices considering the specific requirements for the continuum-X elements and how their allocation changes during execution. Addressing IoT distributed AI and federated learning/training considering the continuum-X elements requires defining a new generation of tools and mechanisms that enable finegranularity computation, processing, communication, coordination, and mobility management across the edge. Finally, new research approaches are required to address the fragmentation in IoT to tackle the E2E challenges, by providing several interoperable, integrated HW, software, and services approaches to address the continuum-X. 4.3 Research Priorities Timeline Table 2 IoT edge and X-Continuum research priorities Topic Short Term Medium Term Long term 2023-2024 2025-2027 2028-2030 Continuum refinement High-level IoT-edge-Cloud continuum solutions for basic services, protocols, and resource allocation. More refined continuum support of functionalities and lower layers management. Fully supported and smooth management of all the edge granularities, types, services resources, and functions, in a fully heterogeneous ecosystem. Common shared data space Definition of domain-specific edge IoT applications aligned to common European data spaces. Definition and implementation of common European data spaces. Fully compatible interwork of data spaces with all geographical areas. Instantiation Piecewise and vertical-specific instantiation of all kinds of resources. Coherent and homogeneous instantiation of all kinds of resources. No distinction between the kinds of resources, all are treated as If they were homogeneous and available everywhere.
© AIOTI. All rights reserved. 24 5. Intelligent Connectivity Networks and connectivity are becoming more heterogeneous and IoT devices use more and more a variety of equipment with different wireless access technologies. With the constant increase of the numbers of users (new contracts) and especially of things (sensors, metering systems, mobile/static robots, vehicles, drones, etc.) joining each day the communication network, the need for a smarter and more effective way of handling the related growing data created by those devices is becoming more and more an issue that needs the attention of the IoT community. To handle such complexity and to manage in a smart way how the communication system can properly manage such an increasing amount of data (scalability problem), there’s the need to define and elaborate on the intelligent connectivity concept, to optimise the usage of the networks but also to decrease the overall energy consumption of the overall IoT system. Intelligent connectivity can be seen as the smooth synergy of different technologies and domains, so to offer final users of a communication system the best possible QoS, according to the given resource, available technology, and energy constraints. 5.1 Technological developments With the technology miniaturization that still proceeds at almost yearly cadence, and recent advancements in battery technology, and the more effective use of out-of-device processing capability services and applications, it is possible to increase the number of functions and the communication capabilities of terminals, so to properly handle most of the issues described above and make intelligent connectivity finally a reality. The next wave of IoT devices at the edge will therefore use a kind of intelligent connectivity that relies on the consolidation and synergy of different communication technologies. Whereas the Internet is today the largest worldwide communication network, such intelligent connectivity for IoT still requires further development to allow high demand in bandwidth and quality in signals and protocols, in support of more critical content and data exchange. The requirements of real-time response may be, amongst others, safety and mission-critical (like in telemedicine, for example) and, consequently, IoT communications solutions are numerous and diverse: • Artificial Intelligence, especially when declined in distributed AI at the edge of the network, but also as a set of methods to more intelligently manage the huge and diverse amount of data created and consumed by terminals. • Advanced communication protocols: to accommodate the different needs of the different verticals that make use of the communication network, dedicated and always improving communication protocols are to be defined and deployed. Standards bodies constantly update and add new features to each new generation of the network they define. • Edge Processing: the need for low latency of several applications and the complexity of the data to be managed request that always more data shall be handled and processed as close as possible to the source and destination of that data, i.e., at the edge. The MultiAccess Edge Computing (MEC) concept, in constant evolution, and its related features can be seen as a cornerstone in the quest for a more efficient data management at the Edge.
© AIOTI. All rights reserved. 25 • Energy Consumption: this is a system parameter that becomes more and more important at each new generation of the communication network. Reason for that is not only the need for CO2 reduction to tame the disruptive climate change consequences, but also sheer economic reasons, as increasing the number of servers and chips needed to handle the raising amount of data exchanged in the world, implies a constant and steep raise of the energy demand to accommodate the needed QoS and reliability of such communications. Therefore, it is key that in the forthcoming Intelligent Connectivity paradigm low energy consumption is taken as a key design constraint for all new chips, platforms, and SW suites. • Spectrum management: with the introduction of several access technologies (LoRa, Wi-Fi, cellular, etc.) and the commercial availability of new spectrum bands for new application (mmWave bands under and above 100 GHz in addition to the standards sub-6 GHz bands), there is the need to make effective use of all the different bands now made available. That effective use has the advantage not only to increase the coverage, the resiliency, and the performance, but also, to enable offering the lowest energy consuming access technology in a specific vertical use case, also reducing in a dynamic way the energy consumed to deliver a specific service. Dynamic usage of different kind of spectrum, of non-adjacent bands, and even mixing different bands coming from different access technologies, based on local availability, will change the way services can be delivered from the cost and from the quality points of view. Finally, it is worth mentioning that also drones and satellite accesses and related bands will soon be broadly available for commercial use, and the availability of coverage everywhere will change the way we think of connectivity and related services. • Multi-Access capable devices: to deliver higher QoS, to handle the communication bottlenecks (both in the frontand the back-haul) in highly densely populated areas, and to fulfil Key Performance Indicators (KPI) of the applications requesting a huge amount of bandwidth, e.g., for Augmented Reality / Virtual Reality (AR/VR) services, the need of a handset capable of making use of the most suitable available access technologies and bands is becoming more and more important. • Seamless connectivity broker: that is needed to optimise the connectivity to a specific network, according to several technical and business parameters. The seamless roaming should avoid that data get lost during the switching procedure; AI/ML technologies should also contribute to make the most relevant choice. An essential characteristic of the next generation IoT is the requirements range from high reliability and resilience in the communication network to ultra-low latency and increased capacity at the communication channel. IoT Intelligent Connectivity solutions are also very dependent on the context in which they are applied and whether it is necessary to respond to strict energy efficiency constraints or cover large outdoor areas, deep indoor environments or vehicles moving at high speeds. 5.2 Main Trends, Issues and Challenges The next-generation IoT must address the convergence between different cells and radiation and develop new management models to control roaming while exploiting the coexistence of many different cells and radio access technology (RAT). New management protocols to handle user assignments regarding cells and technology will have to be deployed in the mobile core network to access network resources more efficiently.
© AIOTI. All rights reserved. 32 7. Heterogeneous Cognitive Edge IoT Mesh IoT edge can be formed by a mesh network of intelligent IoT devices using edge IoT platforms, including AI model training and ML inference that process, analyse, store information locally close to the data sources, and communicate and exchange information with other edge devices, computing units and across the computing continuum, made of cloud platforms and data centres. Cognitive compute continuum applied in decentralised environments, including edge mesh infrastructures, can operate efficiently and reliably without a central entity for full knowledge and management about available resources and current workloads. Autonomous computing techniques and ML should be broadly applied, so to be able to adapt to unforeseen situations and anticipate future circumstances, such that the cognitive edge IoT devices become self-aware, self-managed, self-protected, self-healing, and self-optimising. 7.1 Technological developments IoT and edge computing systems evolve around the fundamental context management processes: acquisition, modelling, reasoning, and distribution, while distributing the processes based on the context information. Novel IoT and edge computing designs bring innovative adaptive and behavioural-awareness capabilities to the IoT, so to set the foundations for developing the next-generation cognitive and self-adaptive IoT systems. The huge number of heterogeneous edge IoT devices need to be integrated into mesh networks, in which the infrastructure nodes connect directly, dynamically, and nonhierarchically to other nodes and cooperate with one another to efficiently route data to and from edge IoT devices. The mesh network topologies allow multiple routes for exchanging information among connected nodes. The mesh approach increases the network's resilience in case of a node or connection failure. More extensive edge IoT mesh networks may include multiple routers, switches and other IoT devices operating as nodes. In heterogenous cognitive IoT networks, context-awareness is becoming critical for IoT systems as the processing is moving towards pervasive, swarm evolutionary computing. Resources and services are remotely accessed by edge IoT devices from anywhere in the network at any time. The environment in which data is exchanged and processed can change dynamically as different ad-hoc heterogenous networks are created. Creating self-configured mesh IoT networks, where things integrate self-perception and automated response, can support this. Embedding appropriate plug-and-play intelligent and autonomous edge IoT devices facilitate the transition into IoT networks with self-X capabilities (self-configuring, self-healing, selfoptimising, self-protecting, etc.). Such self-configured mesh IoT networks form a flexible architecture where things are active and locally integrate almost every functional and operational requirement.
© AIOTI. All rights reserved. 33 The IoT end-devices can manage critical aspects such as security, safety, and trustworthiness and embed federated, decentralised ML and decision-making mechanisms, enabling disruptive cross-domain applications with high complexity and scale. The security techniques used by edge IoT devices can be crucial for obtaining contextual information from other end devices using a lightweight and E2E approach based on current standards (e.g., IETF OSCORE 24 ). The process is linked to an initial bootstrapping by which a legitimate device can be successfully and securely deployed in the network. The modelling of context information should be based on current standards that guarantee the representation format's interoperability using flexible approaches (e.g., JSON-LD 25 , or additional representations based on CBOR 26 ). In this case, the primary purpose is to achieve a trade-off between lightness and expressiveness to represent the dependencies among devices' contextual information. 7.2 Main Trends, Issues and Challenges The evolution of IoT and edge computing system will be based on the integration of AI-based mechanisms. Moving part of the system intelligence down to the end devices brings a range of advantages and allows a widespread collective-intelligence web. The use of intelligence on the device (e.g., TinyML 27 and other similar embedded ML applications, algorithms, HW, and SW) boosts the evolution of end devices to intelligent IoT objects that can offer, among other advantages, a much more autonomous behaviour. Integrating AI within edge IoT devices is one step towards developing cognitive and evolutive IoT systems. The integration of cognitive self-evolution capabilities in the entire E2E network chain (edge, dew, fog, and cloud) goes beyond the current limits of AI by permitting end devices and, by extension, the whole system to grow and refine its intelligence. The development of evolutive artificial cognitive mechanisms and their integration with the selfx concepts pave the way for designing IoT devices and systems with increased intelligence that will permit them to meta-learn from new situations, scenarios, and environmental changes. The cooperation among individual meta-smart IoT objects can create a higher intelligence layer for tackling large-scale issues. The decision process is not isolated but starts from a complete view of the scenario and gathers collaborative efforts. The heterogeneous cognitive and mobile IoT edge mesh solutions utilise low-level IoT devices for the decision-making process, the gateway devices and the micro servers distributed across the edge continuum. The data collection, processing, and decision-making tasks are distributed among these edge devices within heterogeneous networks. The computation tasks and data are shared using a cognitive mesh network of edge IoT devices, gateways, and micro servers. Heterogeneous cognitive and mobile IoT edge mesh systems offer distributed processing, low latency, fault tolerance, better scalability, security, and privacy. These features are essential for critical applications that need higher reliability, real-time processing, mobility support, and context awareness. 24 https://datatracker.ietf.org/doc/rfc8613/ 25 https://json-ld.org/ 26 https://cbor.io/ 27 https://www.tinyml.org/
© AIOTI. All rights reserved. 34 Distributed computations on edge IoT systems imply addressing heterogeneous cognitive features for IoT applications and developing a fault-tolerant robust collective computing framework suited for multi-IoT device systems that include decentralised operations applied to dynamic environments. Mobile IoT edge mesh characteristics, such as scalability, heterogeneity, mobility, and ubiquitous networking, require a robust cooperative and cognitive computing framework that addresses the edge IoT systems characteristics, including new security solutions adequate to these new IoT-based systems. The research needs to provide IoT platform-agnostic solutions with scalable edge IoT devices in a self-healing and self-organising network. In a heterogeneous mesh network environment, communication and cognitive coexistence are critical to allow maximum robustness to RF disturbances and minimize negative effects due to the co-existence of various wireless and cellular networks. The cooperation between edge IoT devices and the dynamic communication infrastructures of the edge and swarm systems create the heterogeneous cognitive and distributed intelligence framework needed to optimally support edge IoT mesh applications. The heterogeneous cognitive and mobile IoT edge mesh research must address topics at the intersection between IoT, AI, connectivity, and edge computing. These topics relate to context awareness, autonomous control, ambient intelligence, semantic reasoning, and cognitive IoT to enable distributed intelligence in IoT. The edge IoT distributed intelligence must be addressed holistically by developing cognitive platforms integrating features such as distributed data collection, data analytics, networking, data management, edge IoT device management, resource management, service management, orchestration, and federated learning. Several research questions are related to defining the distributed mesh network and the cognitive computing model, the distribution of data processing, and the global optimisation (energy, processing, time, etc.) of communication and computation. New computation algorithms must be developed for distributed computing to be performed by the various heterogeneous, resource-constraint edge IoT devices operating with dynamic communication and intermittent connectivity.
© AIOTI. All rights reserved. 35 7.3 Research Priorities Timeline Table 5 Heterogeneous cognitive edge IoT mesh research priorities Topic Short Term Medium Term Long term 2023-2024 2025-2027 2028-2030 Architecture Architectural models and metaembedded operating systems with integrated stack for wireless mesh networking. AI-based cognitive mesh architectures that integrate components and modules addressing context awareness, autonomous control, ambient intelligence, semantic reasoning, and federated learning. Dynamic cognitive mesh architectures with AI and context-based configuration capabilities integrating features for swarm intelligence and distributed processing capabilities. Cognitive Capabilities Evolutive artificial cognitive mechanisms for edge IoT systems and the integration with mesh networks topologies. Development of new algorithms and SW/HW self-X capabilities. Integration of cognitive selfevolution capabilities in the entire mesh network and across the edge granularity including scalable AI capabilities across the continuum. Computing Models Computation algorithms for distributed computing applied to different heterogeneous, resource-constraint edge IoT devices across the edge continuum. Heterogeneous cognitive edge IoT mesh frameworks integrating features such as distributed data collection, data analytics, networking, data management, edge IoT device management, resource management, service management, orchestration, and federated learning. Meta-mesh computing models for global optimisation (energy, processing, time, etc.) of distributed systems.
© AIOTI. All rights reserved. 36 8. IoT Digital Twins, Modelling and Simulation Environments An IoT DT is a virtual representation of an IoT device that models the device's characteristics, properties, environmental conditions, behaviours, and functions over the operational lifetime, based on real-time data and information synchronised automatically and bi-directionally at a specified frequency and accuracy. An IoT DT uses simulation, ML, and reasoning to simulate various scenarios in different IoT applications and help optimise and improve the overall IoT system functionalities and services. The real-time feature represents a vital characteristic to define IoT DTs, considering that the realtime instances vary according to IoT applications. In many IoT applications, time values are not defined identically, and such issue should be carefully considered when designing IoT DT instances. The synchronisation between the physical IoT device and its virtual representation in the simulation environment and the synchronisation of the events and scenarios in the simulation platform is critical for the performance of the whole IoT system. 8.1 Technological developments IoT DTs support optimal decision-making and effective and efficient actions of the IoT devices in IoT applications, using real-time and historical data to represent the past and present and simulate predicted activities tailored to use cases based on domain knowledge, and implemented in Information Technology / Operation Technology (IT/OT) systems. The IoT DTs can expose a set of services to execute certain operations and produce data describing the physical activity of the IoT devices that they virtualise. How to choose and optimize characteristics, properties, environmental conditions, behaviours, and functions over the operational lifetime of an IoT device that are mapped to the IoT DT is a matter that needs further research; moreover, further analysis is needed also to properly develop IoT DT user interfaces that can connect the IoT DTs with other devices, with humans, and with other DTs coming from different domains (e.g., telco DTs or connectivity DTs). The lifecycle evolution of the IoT DTs must take into consideration the updatability and upgradability of the IoT devices, including new features addressing the dependability characteristics. Operational intelligence is used to build IoT DTs as it supports digitising the IoT and edge computing infrastructure, monitoring operations in real-time, predicting events, taking actions based on intelligence and engaging with different stakeholders. The virtual representation of an IoT DT reflects all the relevant dynamics, characteristics, critical components, and essential properties of the IoT physical device throughout its life cycle. The creation and update of IoT DTs rely on timely and reliable multi-sense wireless sensing, while the cyber-physical interaction relies on timely and dependable wireless control over many interaction points, where wireless interfaces of the IoT device are embedded. In future networks, IoT DTs will be a valuable tool to create novel and disruptive solutions, especially for vertical industries, that are enabled by a large scale of real-time, robust, and seamless interactions among, for example, machines, humans, and environments. IoT DTs need to be scaled up in IoT applications, thus enabling for the broadest possible population a sustainable living with systematic climate mitigation measures, improving society's resilience in crises by actively monitoring and simulating a huge number of future scenarios, and potentially helping transform the whole societal structure, so to make it more robust and capable of addressing the environmental (but not only) challenges of the future.
© AIOTI. All rights reserved. 37 IoT DTs must possess a minimum set of attributes to be integrated into IoT applications and platforms to optimise the functions and services of these applications. These IoT DT attributes are summarised below: • Abstractness – free from details which are specific to implementations. • Correctness – give a correct replication of the IoT ecosystem and its devices. • Completeness – updated vis a vis the functionality in the real-world system. • Expandability – adapt easily to emerging technologies and applications. • Parameterised – accessible for analysis, design, and implementation. • Reproducible – be able to replicate the same result for the same input as the real system. • Scalability – must be able to operate at any scale. • Soundness – exhibit only the functionality available in the real-world system. IoT DTs will continue to grow in industrial and production environments, leading to the new designing approach called massive twinning. It will enable to go beyond the current levels of agility of production, thus allowing more efficient interaction of production means to encompass a more significant extent of the respective processes, and achieve the transfer of massive volumes of data, as well as, often, reach unprecedented performance and reliability levels. The evolution of edge IoT digital twin technology is illustrated in Figure 6. Figure 6 Edge IoT digital twin technology evolution IoT DTs, as part of IoT technologies and applications, are being expanded to support more applications, use cases and vertical industries, as well as combined with more technologies, such as speech, augmented reality for an immersive experience and AI capabilities, enabling to look inside the IoT DT by eliminating the need to go and check the physical IoT device.
© AIOTI. All rights reserved. 38 8.2 Main Trends, Issues and Challenges The research today focuses on developing the virtual model representation of an IoT device, the evolving data sets relating to the IoT domain, the mechanisms to dynamically synchronise and adjust the virtual representation following the changes into the physical IoT device, and the simulation environment in which the IoT DTs will operate. The IoT DT mapping of the physical environments into the digital world is facilitated by IoT simulation platforms and SW leveraged to create a digital model and a virtual representation of the physical IoT device. IoT digital virtual representation can be used to manipulate and control the real-world IoT device through a teleoperation DT modelling solution. The research areas for IoT DTs must consider the scope and augmentations of the IoT DT and the operational environment combined with the functions needed to realise the IoT DTs' communication capabilities, and the update frequency required for providing the optimal precision of the IoT DT, based on data measured and acquired during the operation and use of physical IoT devices. Further research needs to investigate the different levels of IoT and edge computing intelligence through cognitive functions, implemented in the physical/digital and virtual devices. Understanding, defining, and designing the simulation capabilities of the future IoT platforms, which will be able to provide different fidelity levels of simulation tuned by input parameters, time dependency, behaviour, and prediction aspects, intelligence, and IoT device complexity, are very challenging tasks, which require further investigation. 8.3 Research Priorities Timeline Table 6 IoT Digital Twins, Modelling and Simulation Environments research priorities Topic Short Term Medium Term Long term 2023-2024 2025-2027 2028-2030 IoT DT Models Aggregation of heterogeneous IoT DT models. Energy-efficient models. E2E features and optimisation. Horizontal and vertical integration of IoT DT models. IoT DT that is capable of modelling and simulating the future state and behaviour of the IoT device. IoT DT Modelling and Simulation Platforms IoT DT platforms at the edge. Virtual sensing and actuation functions and simulations. Predictive modelling platforms. Modelling and simulation of energy efficiency. Integrated IoT platforms with virtual simulation environments including XR. IoT DT Security IoT DT security features integrated. IoT DTs counterfeiting identification and mitigation. Automatic recognition of fake DTs and their isolation or elimination. IoT DT Connectivity Simulation and modelling of the communication channels. Define influence of the environments on the communication parameters of the IoT DTs. Virtual platforms for the connectivity of IoT devices.
© AIOTI. All rights reserved. 39 9. IoT Swarm Systems Swarm intelligence for edge IoT systems deals with various IoT devices coordinated using selforganisation, decentralised and distributed control. It consists of simple autonomous edge IoT devices and agents that come as emergent collective intelligence. Swarm intelligence addresses the area of IoT system collective behaviour and is inspired by social swarms in nature such as bird flocks, ant colonies and honeybees. Swarm and edge computing combined with IoT are used to implement the collective behaviour of physical and virtual AI decentralised and distributed IoT systems. From that concept new terms such as "artificial intelligence of things", "internet or intelligent things" are derived and are emerging to address collective intelligence in IoT systems with no centralised control infrastructure. These new developments create new "Internet of Robotic Things", "Internet of Intelligent Mobile Things" and "Intelligent IoT Devices Colonies" applications that apply the principle of swarm and edge computing to fleets and groups of edge IoT devices, supporting an always increasing number of items as the technology improves. 9.1 Technological developments The future edge IoT swarm systems bring fundamental research challenges in cross-domain IoT resource orchestration. For instance, in designing new multi-intelligent-agent-based edge IoT frameworks. Each IoT device could be associated in the future with an agent or a DT at the edge. The agents could utilise swarm intelligence to jointly optimise the operations and information flow for their respective IoT devices. The move towards collective intelligence in edge IoT systems requires intelligent orchestrators for heterogeneous systems leading to swarm computing concepts. To achieve that, new intelligent programming orchestrations for distributed and decentralised open architectures and methods for updating/upgrading over-the-air (OTA) IoT smart devices are required. Collaborative functions ask for leveraging edge IoT swarm systems to improve network connectivity, enhancing information collection ability through each edge IoT device and intranetworking issues for a swarm of edge IoT devices. The energy spent on edge IoT devices during swarm task execution is a key aspect to be considered. Total energy consumption includes static energy consumption and dynamic energy consumption during different tasks and the connectivity to other IoT devices and edge processing. The development of collaborative edge IoT swarm systems raise the issue of vulnerability to attacks by hackers from other fleets or via the interface with the cloud. The inherent distributed architecture for edge IoT swarm systems makes them more robust and secure and implies that the information is shared, stored, exchanged, and analysed locally and inside the swarm fleet, making it harder for hackers to access sensible data.
© AIOTI. All rights reserved. 40 The real-time processing and response of edge IoT swarm systems make it hard for malicious attackers to detect devices' sensitive information. ' The edge IoT systems interfaces to other edge-cloud infrastructure derive several security problems that have to be considered (swarm identity (ID) leakage, forgery, tampering, spam, jamming, swarm IoT device impersonation). The distributed nature of edge IoT swarm systems require new efficient service discovery protocols to design such that IoT devices and users can identify and locate the relevant swarm services and providers to satisfy the application requirements. Real-time optimisation protocols for edge IoT swarm systems are required as future mobile IoT applications are dynamic and involve using online edge resource orchestration capabilities and provisioning to continuously handle dynamic edge IoT swarm devices workloads and tasks. The distributed edge computing architecture for edge IoT swarm systems shifts the research on how trust and security are addressed. The development of decentralised trust solutions with services provided by different secured and trustworthy edge IoT entities is part of the new E2E implementations. New efficient security mechanisms for edge IoT swarm systems are required to ensure IoT device authentication, data integrity, and mutual IoT swarm platform verification and validation. New secure routing schemes and trust network topologies are key for edge IoT swarm systems. The development edge IoT swarm systems must consider the balance between the HW and SW constraints for edge IoT swarm devices combined with the level of intelligence implemented in the edge devices and their connectivity capabilities to perform in a swarm fleet adequately. As a result, developing SW, HW, and AI-based algorithms for handling computation offloading from edge to the IoT swarm device is a critical issue to be addressed. The inherent distributed architecture of future edge IoT swarm systems consider interoperability, collaboration as part of the system, and the platforms operating these systems include efficient algorithms to facilitate collaboration. Edge IoT swarm systems may display a considerable level of heterogeneity in terms of IoT devices, level of intelligence, connectivity, and processing capabilities and, therefore, efficient IoT platforms that deal with this heterogeneity are highly desired. The collaborative intelligent interaction among the IoT swarms requires further research on federated learning to execute or train ML models in edge IoT devices as part of a swarm fleet. 9.2 Main Trends, Issues and Challenges The research in IoT edge swarm systems complements the traditional AI approach to cognition and combines individual reasoning, collaborative intelligence, and mobile and sensing intelligence. The IoT swarm computing must address the interaction with the environment in which the swarm IoT fleets operate, including research concerning federated learning processes and simple perceptions of forms, recognition of other IoT devices, and the swarm fleets' interaction with other fleets and humans.
© AIOTI. All rights reserved. 41 The evolution of mobile edge IoT fleets that mimic complex behaviours must consider the development of techniques to model and simulate the interaction of simple edge IoT devices, to explore psychological ideas and the nature of collaborative intelligence in mobile IoT fleets. Research on expanding the concept of intelligent agents receiving real-time data combined with mesh computing applied to IoT edge swarm systems can support the advancement in collaborative intelligence for edge IoT. Edge IoT swarm-based intelligence assumes that a group of IoT swarm devices or intelligent agents can perform tasks without explicit representations of the environment, and the IoT swarm devices or agents operate combining planning with reactivity. The self-organisation of flow patterns in edge IoT swarms research is a challenge, especially in providing solutions for intelligent and efficient behaviour of the whole IoT swarm fleet when we combine the limited intelligence of the individual IoT devices. Further research needs to study the essential elements of edge IoT swarm dynamics providing mechanisms for implementing such behaviours and HW, SW, algorithms for modelling, simulating, and integrating self-organisation agents into edge IoT devices. One of the major challenges of edge IoT swarm systems is to encode and deploy the collective behavioural characteristics of a fleet of edge IoT swarm-based fleets into the behaviour of the individual edge IoT devices. Research on complex adaptive behaviour is needed to provide HW, SW, and connectivity mechanisms to implement interactions between edge IoT devices as distinct from behaviour that directly results from the operations of individual edge IoT devices. The advances in edge IoT swarm systems are directly linked with the research in providing optimised mechanisms and techniques for providing the ability to mobile IoT swarm devices to separate, cohere, align, and avoid obstacles. Separation is the ability to operate with other IoT devices and keep a certain separation distance from different devices and avoid crowding too closely together in a fleet. The cohesion function allows the IoT devices to approach or form a group of edge IoT devices. The alignment feature supports the devices to identify the expected next movement and synchronise the movements of IoT devices relative to each other. Navigating and avoiding obstacles (e.g., perception and processing efficient collision detection algorithms) is another critical challenge to be addressed by research in IoT swarm-based fleets. The work in edge IoT swarm systems must be combined with the latest advances in neuromorphic computing with edge IoT devices using neuromorphic HW and collaborative neuromorphic operating systems (OS). The connectivity solutions in edge IoT swarm systems becomes critical in implementing collaborative intelligence. New ultra-low-power mesh communication protocols must be combined with neuromorphic architecture to provide self-X functions to edge IoT swarm systems.
© AIOTI. All rights reserved. 48 Here, micro-networks of potential different ownership and with a potentially external security management might share parts of the infrastructure with wide area networks, paving the way for a set-up where a private network with partly owned infrastructure and a private trust policy are integrated in a public network. 11.2 Main Trends, Issues and Challenges Decision making and optimisation: taking into consideration an E2E system made of three elements, the edge, the network, and the cloud (where information flows from the edge through the network to the cloud and vice versa), critical issues are what are these decisions, and equally important, where do those decisions take place, with which implications on the overall network. The main consideration is that any network is a complex entity, since it is distributed, heterogeneous in different aspects (data streams, computation, and storage capabilities, required QoS, etc.), time varying (in some but not all aspects), and resource limited. The application of the decision theory to the more and more important problem of distributed resource allocation is still a hot research question that needs to be addressed 36 . What is looked for is a framework or model that would allow to decide where in a network (edge, cloud, intermediate stages?), both processing and decisions about resource allocation are made, and that does that in a way that is responsive in near real-time. Research is needed in novel IoT distributed architectures to address the convergence of low latency, Tactile Internet, edge processing, AI and distributed security based on ledger or other technologies, and an effective deployment of multi-access edge computing (MEC). Developing specific architectural requirements for distributed intelligence and context awareness at the edge is a future research topic, especially when considering the integration with mesh network architectures, so to form knowledge-centric networks for IoT, capable of serving many different applications coming from numerous vertical sectors. Research on orchestration of IoT heterogeneous networks, adaptation of software defined radio and networking technologies for IoT, considering built-in E2E distributed security as well as hardware-based security solutions 37 , trustworthiness, and privacy issues in edge computing environment must be extended, addressing the federation and cross-platform Integration for edge IoT applications. 36 A. Mukherjee, P. Goswami, M. A. Khan, L. Manman, L. Yang and P. Pillai, "Energy-Efficient Resource Allocation Strategy in Massive IoT for Industrial 6G Applications," in IEEE Internet of Things Journal, vol. 8, no. 7, pp. 5194-5201, 1 April1, 2021, doi: 10.1109/JIOT.2020.3035608. 37 J. Gopika Rajan. and R. S. Ganesh, "Hardware Based Data Security Techniques in IOT: A Review," 2022 3rd International Conference on Smart Electronics and Communication (ICOSEC), Trichy, India, 2022, pp. 408-413, doi: 10.1109/ICOSEC54921.2022.9952021.
© AIOTI. All rights reserved. 49 11.3 Research Priorities Timeline Table 9 Decentralised and distributed edge IoT systems research priorities Topic Short Term Medium Term Long term 2023-2024 2025-2027 2028-2030 Decision making Partially distributed decisionmaking mechanisms and techniques. Federated sub-system sharing a common decision mechanism Fully distributed decisionmaking methods. Federated systems of systems sharing the same decision mechanisms Fully distributed and federated systems, using heterogeneous decision mechanisms targeted to specific QoS or vertical sectors. Security aspects Established Privacy preserving techniques. Block-chain based security methods applied to selected vertical sectors. Scalable block-chain based security mechanisms addressing some vertical sectors. Trustworthy and auto-adapting communication among several vertical sectors. Secure-by-design system of systems, endowed with swarm intelligence and hardwarebased security measures, fully Decentralised security procedures, self-adapting to real-time demands of heterogenous actors. Learning mechanisms Distributed learning. Federated learning. Distributed and federated learning. Continuous learning, selfadapting to the dynamic changing environment of a heterogeneous network supporting several vertical sectors.
© AIOTI. All rights reserved. 50 12. Federated Learning, Artificial Intelligence technologies and learning for edge IoT Systems To perform according to the devised expectations 38 , the new distributed IoT architectures for computing optimisation across the edge continuum need to improve responsiveness by reducing decision-making latency, to increase data security and privacy, to decrease power consumption, using less network bandwidth, thus maximizing efficiencies, reliability, and autonomy. The IoT edge contains computing capabilities scaled across the micro-, deepand meta-edge to process workloads, including the latest technology like AI model training and ML inference and signal processing, using signal conditioning 39 followed by neural networks 40 computing. The neural network computing and memory requirements are significantly reduced by using signal conditioning on the raw data. In this context, federated learning as a distributed ML technique, which creates a global model by learning from multiple decentralised edge clients, is a significant technological development that can be implemented in distributed IoT architectures across the edge continuum. Federated learning uses complex methods for handling distributed data training by enabling the cooperative training of common AI models, by combining and averaging locally calculated updates submitted by edge IoT devices. Federated learning permits training new models on multiple edge IoT devices simultaneously without the need to have data stored in a central cloud. 12.1 Technological developments Integrating AI-based techniques across the edge continuum requires a new layer of edge processing infrastructure and scalable, energy-efficient modules for AI-based processing 41 . Federated learning methods offer several advantages, including scalability and data privacy, with ML and DL algorithms that can be executed on edge IoT devices, delivering faster realtime insights for increased IoT application efficiency. Bringing AI to the edge increase the efficiency of processing the data locally and reduces latency and the cost of connectivity for many IoT applications. In the distributed data exchange environment, federated learning/training combined with the IoT heterogeneous compute based on various underlying processing architectures (CPUs, GPUs, NPUs, neuromorphic, etc.) can provide the solution for future IoT edge intelligent heterogeneous systems. 38 J. Lin, W. Yu, N. Zhang, X. Yang, H. Zhang and W. Zhao, "A Survey on Internet of Things: Architecture, Enabling Technologies, Security and Privacy, and Applications," in IEEE Internet of Things Journal, vol. 4, no. 5, pp. 1125-1142, Oct. 2017, doi: 10.1109/JIOT.2017.2683200. 39 R. Tirupathi and S. K. Kar, "Design and analysis of signal conditioning circuit for capacitive sensor interfacing," 2017 IEEE International Conference on Power, Control, Signals and Instrumentation Engineering (ICPCSI), Chennai, India, 2017, pp. 1717-1721, doi: 10.1109/ICPCSI.2017.8392007. 40 Z. Li, F. Liu, W. Yang, S. Peng and J. Zhou, "A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects," in IEEE Transactions on Neural Networks and Learning Systems, vol. 33, no. 12, pp. 6999-7019, Dec. 2022, doi: 10.1109/TNNLS.2021.3084827. 41 Vermesan, O. and Nava, M.D. (Eds.). Intelligent Edge-Embedded Technologies for Digitising Industry. River Publishers Series in Communications, June 2022. ISBN: 9788770226110, e-ISBN: 9788770226103. Online at: https://www.riverpublishers.com/pdf/ebook/RP_E9788770226103.pdf
© AIOTI. All rights reserved. 51 The federated learning/training must consider the underlying connectivity systems, including Bluetooth, Wi-Fi, LPWAN, mesh, 5G and beyond applied to the IoT application topology, range, and power requirements. The convergence of connectivity, AI/ML, and processing can ease the implementation of different IoT federated learning/training solutions for industrial and consumer applications. The concept of IoT federated learning/training is combined with the IoT edge mesh, which allocates the decision-making tasks among edge devices within the network. The computation and processing tasks and data are shared using an IoT networks of edge devices, routers, PLCs, and micro-servers. The IoT edge mesh combined with federated learning/training provide distributed processing, low latency, fault tolerance, scalability, security, and privacy, as required by IoT applications, which demand higher reliability, real-time processing, mobility support, and context awareness. Cooperative IoT computing based on federated learning/training provides better usage of resources, reduced latency, due to easy access to local resources, better services, as IoT devices can cooperate to get better information, reduced communication with centralised entities, and improved security and privacy as data remain most of the time within a local network. Enabling distributed intelligence in IoT/edge computing or swarm computing applications is difficult due to a set of known problems, e.g., synchronisation, consensus, cooperation etc. In addition, due to scalability and complexity issues of IoT systems, it is challenging to determine how to generate, coordinate and federate the intelligence, which edge IoT device provides intelligent functionality and how different edge IoT devices cooperate, transfer, and acquire intelligence. 12.2 Main Trends, Issues and Challenges The challenge for many IoT applications is that federated learning/training uses multiple entities collaborating to solve ML problems under the coordination of a central server. This approach for edge networks creates many issues concerning security and privacy and the data itself. Federated learning raises several risks and weaknesses in terms of computational complexity in the case of heterogeneous edge IoT devices that may have limited computing resources, inadequate wireless connectivity quality, or may use different OSs. Another edge IoT federated learning challenge relates to communication delays expressed as the latency between edge IoT devices and the ML system. Decreasing latency is critical for AI‐based edge IoT devices operating in real-time applications such as industrial equipment. Replacing the client-server process of the federated learning/training model with fully decentralised learning replaces communication with the server by peer-to-peer communication between individual clients. Optimisation algorithms are necessary to implement federated learning/training at the edge, considering edge IoT devices' constraints and resource limitations as part of the edge continuum. The IoT devices limited bandwidth can restrain scalability, but this is solved in IoT edge mesh architectures, as data is sent to multiple edge IoT devices that share data with other devices. The communication bottleneck issue is resolved due to the distributed nature of the system.
© AIOTI. All rights reserved. 52 At the same time, the computation tasks are offloaded to different edge IoT devices, operation which speeds up the processing time and increases the efficiency of federated learning/training, leading to better response time, reduced make span, and higher throughput. The distribution of loads drives the edge IoT systems to be more flexible and robust, as, in the case of a device failure, other devices can share the load of the failed IoT device. IoT systems are dynamic, as devices can be mobile, added, removed, or changed in configuration; all of the above require new context-aware solutions for distributed security and privacy algorithms. The heterogeneity of computing, communication and AI technologies requires AI-based algorithms that are portable across different IoT edge environments. Communication technology is intrinsically heterogeneous for what concerns data rate, transmission range, and bandwidth. The IoT SW solutions depend on the HW, and programming models are needed to execute workloads simultaneously at multiple HW levels. Research on standard protocols and interfaces should address the integration of AI-based algorithms and lightweight protocols for communicating with different devices in a heterogeneous environment. More research is needed to develop distributed learning/training algorithms and to maximise the average time between errors and optimise the availability by minimising the failure probability and average recovery time in the OTA learning/training process. The IoT federated learning/training must consider the hybrid computing method that combines HW/SW and AI techniques across the edge continuum. Hybrid computing implies the integration of specialised advanced AI processors at different computing levels for both high-level and lowlevel operations. Further research is needed to develop optimised algorithms for federated learning/training to complement and effectively leverage the computing capabilities with the AI-based processors and other types of processor architectures.
© AIOTI. All rights reserved. 53 12.3 Research Priorities Timeline Table 10 Federated learning and AI for edge IoT systems research priorities Topic Short Term Medium Term Long term 2023-2024 2025-2027 2028-2030 Federate learning approaches Techniques and methods to integrate federated learning into IoT/edge computing systems. Management of edge IoT systems, by addressing mesh network security and management by leveraging ML. Research on central training data sets and edge IoT local data sets. Edge IoT intelligence architectures and AI frameworks for federated learning. Development of tools and tool chains for dedicated edge IoT federated learning. Methods for providing reference training datasets for performing standard federated learning application tuning. Benchmarking techniques and methods for edge IoT federated learning. Scalability and portability of AIbased models for federated learning across the edge granularity. Federated learning architecture and frameworks Advanced architectural approaches for the federated learning server integrated into mesh networking environments. Extend the capabilities of opensource federated learning frameworks. Communication and computation efficiency of the federated learning architectures, synchronisation optimisation among edge IoT devices. Federated learning architectures addressing task scheduling, dynamic resource allocation to achieve lowlatency services. Hardware platforms for federated learning HW requirements for implementing federated learning in edge IoT computing environments. HW heterogenous solutions to minimise memory transfer, increase energy efficient and improve computational speed. Computation offloading and content caching using dynamic cache allocation techniques, context-aware offloading algorithms adapted to resource-constrained (e.g., limited storage and capacity) edge IoT devices. HW/SW/AI algorithms heterogeneity management. Understanding of the effect of system heterogeneity on the AI model aggregation efficiency, accuracy and the divergence or convergence of optimisation processes.
© AIOTI. All rights reserved. 54 13. Operating Systems and Orchestration Concepts for edge IoT Systems The extension of the IoT edge and edge-cloud federation is redefining the use of OSs and orchestration concepts across the IoT-edge-cloud continuum in real-time applications. The increased use of embedded systems combined with processing units at the edge requires new ways of virtualising the computing capabilities and the OSs across the diversity of the HW components at the IoT edge. This also requires more frequent updates and upgrades of FW, SW, algorithms, and security patches for edge IoT devices. Challenges arise with more functions and services implemented in FW/SW/algorithms and the increasing complexity of electronic component interoperability, including managing agile and vulnerable edge IoT devices. 13.1 Technological developments The development of ubiquitous meta-OSs 42 to provide an integration framework that implements processing and a better traceability, safety, and security across the IoT edge must consider several challenges. Ubiquitous meta-OSs for IoT edge applications must embed features and functions to allow the applications to be autonomous, cooperative, situational, evolvable, and trustworthy. As the heterogeneity of edge IoT increases, edge IoT devices need to act with a valid secure method, both using centralised and distributed intelligence. The developments of ubiquitous meta-OSs for IoT edge applications are not yet ready to ensure scalability and provide the functionalities and capabilities that would allow Billions of edge IoT devices to work independently of each other and cooperate in real-time. Among the biggest challenges for ubiquitous meta-OSs for IoT edge applications one can mention how to effectively provide real-time features to applications and OS capabilities to heterogeneous edge IoT devices, and how to interact with federated platforms and interface with cloud provider agnostic solutions. Building supporting platforms would have a substantial impact in providing solutions to those issues. The challenges for autonomous orchestration of distributed edge systems are related to maximising utilisation while assuring the Service Level Objective (SLO) of workloads running at resource constraint edges and dealing with inhomogeneous/heterogeneous platforms at the IoT edge. The above-mentioned challenges need to address critical features, like network slicing, that open for an entirely new set of use cases. For example, given a set of network slices, how do we assure that each one will hit Its performance targets while at the same time not over-provisioning resources at large. Overall, the orchestration needs to become more autonomous and lift the burden of Site Reliability Engineers (SRE) running the systems to date. Artificial Intelligence Operations (AIOps) and Machine Learning Operations (MLOps) techniques and methodologies could form the basis for genuinely autonomous systems at the edge. 42 P Trakadas, et Al., “A Reference Architecture for Cloud–Edge Meta-Operating Systems Enabling Cross-Domain, Data-Intensive, ML-Assisted Applications: Architectural Overview and Key Concepts”, in Sensors 2022, 22(22), 9003; https://doi.org/10.3390/s22229003.
© AIOTI. All rights reserved. 55 The heterogeneity of IoT edge brings a paradigm shift towards application-driven components and systems, interoperability, and open-source HW, SW solutions. There is a strong need for a data strategy and establishing trusted data exchange frameworks across OEMs, system integrators, and component vendors. Missionand safety-critical real-time edge IoT applications require guaranteed latency and bandwidth, the integrity of data, security, resilience, and controlled mesh networking. Ubiquitous meta-OSs for IoT edge applications can act as orchestrators that aggregate and compose services according to users' requirements and coordinate their execution in a coherent and smart way. Ubiquitous meta-OS for IoT edge and orchestrating methods are needed to simplify the development, orchestration, and security of distributed IoT edge architectures and solutions. The development of new solutions using at the edge containers, virtual machines, and unikernels 43 provides a flexible foundation for expanding distributed edge computing deployments with a choice of heterogeneous HW, applications, and federated edge-clouds infrastructures. This brings further challenges to ensure distributed firewall, open orchestration APIs, support for virtual machines, containers, and unikernels application deployment models. 13.2 Main Trends, Issues and Challenges New research and key innovations are needed to address orchestration in the future. For instance, novel interfaces between the legacy IoT architecture and the new one will be required. This will allow users to exploit innovations in the control planes (e.g., moving from centralised systems to more decentralised ones) and new HW features in heterogeneous (from the compute capability point of view) platforms, so to be able to place mission-critical workloads at the edge. OSs and orchestration concepts for decentralised and distributed edge IoT systems are needed to provide an integrated environment for the next-generation intelligent IoT applications. Such applications embed IoT in distributed computing systems operating in a continuum at the edge across microdeep-, meta-edge and are interfaced and federated with the cloud solutions. IoT/edge computing platforms and the new IoT applications can create new business models relying on meta-OS orchestration, distributed edge intelligence, storage, and resources heterogeneity. Ubiquitous meta-OSs for IoT edge and orchestrating mechanisms are required for heterogeneous systems with lightweight virtualisation, virtual machines, microservices and containers. Key issues to be addressed are data pre-processing at the edge, edge analytics, scalability, efficiency, dependability, trustworthiness, adaptability, and transparency. The issues are even more critical considering the evolution of IoT towards Tactile IoT and IoTS. Research activities must solve several critical technical challenges, including the availability of a distributed architecture of ubiquitous meta-OSs, scalability, performance and applicability issues, self-X features, and autonomous considerations. Ubiquitous meta-OSs developments for IoT edge must align with the emerging new computing paradigms such as IoT swarm, including intelligent robotics things, organic computing, broad usage of AI, and neuromorphic concepts. 43 http://unikernel.org/
© AIOTI. All rights reserved. 56 The development of new SW-defined abstractions and capabilities for edge IoT devices is needed to support the management, application development, and communications between devices across the edge and cloud continuum. Context-awareness in the edge IoT system is related to the context information, the real-time operation and the dynamicity of processing, scalability and managing the information with the support of the ubiquitous meta-OS. Research should address the issue of creating a flexible context modelling framework to provide means of presenting, maintaining, sharing, protecting, reasoning, and querying context information. In this context, the IoT edge meta-OS must include adaptability and self-X features to allow the IoT application to adapt its behaviour according to the context. The IoT system can then be reconfigured to adapt to these context changes. Self-X functions, including self-organising, self-healing, self-protecting, self-diagnosing, selfreconfiguring, require a form of self-awareness to understand the state of the edge IoT system and are to be embedded into the meta-OS, which must realise its state and configuration, and the state and the configuration of the resources it controls and manages. Research in developing models of edge IoT HW and ubiquitous meta-OS-level SW configuration is necessary to describe the topologies and the rationale for the designs of IoT devices and their integration into the distributed IoT edge applications architecture. The development of new software-defined abstractions and capabilities for edge IoT devices is needed to support the management, application development, and communications between devices across the edge and cloud continuum. The meta-OS features for IoT edge require adapting to the characteristics of these future applications that will be context-sensitive, adaptive, customised, and reconfigurable. The functional requirements behind the concepts of IoT systems reconfigurability, contextawareness, adaptability, and customisation mean that the meta-OS supports and allows that the IoT system’s HW and SW configurations can seamlessly change at runtime. 13.3 Research Priorities Timeline Table 11 Operating systems and orchestration concepts for edge IoT systems research priorities Topic Short Term Medium Term Long term 2023-2024 2025-2027 2028-2030 Decentralization techniques Mature federated learning techniques that provide clear advantages in selected verticals and use cases Advanced federated learning techniques suitable for all verticals New paradigm of distributed management Context awareness Understanding the key surrounding parameters and functionalities that can guarantee an advance context awareness Context awareness improved with semantic capabilities, i.e., abstracting from the sheer heterogeneous data and going up in the abstraction layer New context awareness paradigm Operating Systems New meta-OS that can proof advantages against existing traditional OSs Enhanced distributed meta-OSs that adapt resource computation and storage allocation to the run-time environment on a ms scale Fully autonomous and reconfigurable management of the system resources in the IoT-edge-cloud continuum with fully distributed decision capabilities
© AIOTI. All rights reserved. 57 14. Dynamic Programming Tools and Environments for Decentralised and Distributed IoT Systems Current programming environments and tools for IoT 44 provide a centralised approach (either on-premises or cloud-based) where the main component transforms and processes most of the computation on data supplied by the edge and far edge devices. This approach unfortunately implies several drawbacks, for instance unused edge computational power, a single point of failure, and violation of data boundaries (private, technological, etc.). In this context, new research is needed in leveraging the resources available in lower-tier devices to improve overall dependability, performance, scalability, observability, and reproducibility of IoT systems using new programming and tools for distributed IoT. 14.1 Technological developments Extensive edge IoT distributed systems are difficult to implement. Programming these systems is challenging due to the requirement to assert numerous control paths resulting from the innumerable interleaving of messages and failures, produces by a huge number of heterogeneous devices. The edge IoT distributed systems developments need to consider the integration of lightweight meta-OSs, distributed databases, middleware, mesh networking, and application architecture types. This means addressing the design and analysis of distributed algorithms, programming languages, compilers, SW tools and dynamic middleware environments. Managing edge IoT distributed systems infrastructure requires dedicated sets of programming tools. The dynamic programming tool for coding, modelling, simulating, and visualising the current operational state of all edge IoT devices, including verification and validation components and debugging and testing modules notifying when the failure occurs in the distributed IoT systems. The edge IoT distributed systems bring inherent challenges such as concurrency, partial failure, node dynamism, or asynchrony in their design and implementation. The increasing complexity of the concurrent activities and reactive behaviours in edge IoT distributed systems are unmanageable by the existing programming models, tools, and abstraction mechanisms. 14.2 Main Trends, Issues and Challenges The research should consider transforming and decomposing data flow and partitioning of data exchange between the distributed IoT nodes while detecting current conditions in deciding when to move computations along the edge continuum. 44 A recent survey of the most used tools can be found at: https://www.iotforall.com/top-iot-tools-and-platforms-for-iot-development-and-developers.
© AIOTI. All rights reserved. 64 The next-generation edge IoT platforms require a move from specific vertical industrial sectors to horizontal IoT platforms that implement distributed architectures based on edge computing capabilities operating across several industrial sectors. Such platforms need to integrate several connectivity and processing technologies, advanced analytics and data management capabilities that enable developers to easily design and deploy IoT solutions faster, accelerating time to insight for applications operating across several industrial sectors. 16.2 Main Trends, Issues and Challenges Specific research challenges for next-generation edge IoT platforms are to address the issues of heterogeneity, scalability and federated learning (FL) by implementing distributed architectures that guarantee the security and privacy of the data exchanged by an extensive number of intelligent edge IoT devices while subduing interoperability issues. The research advances in the area of cognitive cloud platforms require new solutions for the federation of edge IoT platforms and the orchestration of edge-cloud domains for optimising the use of the resources, improving service quality, reducing the energy, the inefficient flow of data, and the costs. Further research is needed to address the technological and semantic interoperability issues among heterogeneous IoT devices and platforms in the context of implementing distributed architectures and the integration of new technologies such as swarm computing. The research needs to focus on minimising the complexity of collecting and processing vast amounts of real-time data generated by intelligent IoT devices, address scalability and security issues. The increased focus on IoT solutions built using open-source SW and HW, which are based on open specifications allowing portability and reducing IoT applications development, require further research on open-source technologies. The research and innovation priorities must address the convergence of technologies for edge IoT platforms combining the technological development in a long list of areas, e.g., sensing, processing, communication, computing, AI, and storage technologies. The research priorities need to support the development of new open, edge IoT horizontal platforms combining the distributed functions for mobile edge computing and the interfaces to synergize and federate with other edge or cloud platforms. The advances in intelligent industrial IoT applications, Tactile IoT and autonomous/robotic systems solutions require real-time response and computing at the edge of the networks and FL across the edge. New research is required to provide AI algorithms to operate in a heterogeneous distributed IoT application context, including a federation of edge IoT and cloud platforms. AI and IoT FL advances require new development frameworks to enable the effective development of edge IoT platforms embedding AI-based components at different IoT architectural layers. Further development of edge IoT platforms needs to support the creation of intelligent, self-X functions (e.g., self-organising, self-healing, self-configuring, self-managing) integrated into the platforms to deal with the intelligent autonomous IoT and swarm systems.
© AIOTI. All rights reserved. 65 These functions include SW/HW and AI-based algorithms that enable automated tasks (e.g., to create, provision, configure, troubleshoot) that manage fleets of IoT devices and gateways securely, remotely, in volume or individually. Various AI techniques, rule engines, event stream processing, data visualisation and ML need to be developed to address the micro-, deep, and meta-edge and integrated into the IoT platforms. The IT and OT security measures and features must be considered and developed in the context of distributed architecture and across the edge-cloud continuum. The advances in IoT/edge computing need to address secure data storage, efficient data retrieval and dynamic data collection as part of a processing framework for IoT along the computing continuum considering the functions of data pre-processing, storage and analytics based on both edge and cloud computing infrastructures. 16.3 Research Priorities Timeline Table 14 Edge IoT sectorial and Cross-Sectorial Open Platforms research priorities Topic Short Term Medium Term Long term 2023-2024 2025-2027 2028-2030 Simulation capabilities Edge IoT platforms can simulate and run a IoT devices in a specific vertical domain Edge IoT platforms across multiple IoT verticals domains and supporting partial twinning. Edge AI platforms working and seamless simulating all industrial domains and make use of advanced twinning. Interoperability among platforms Each edge IoT platform has its own API to expose its services to the outside world Edge IoT platforms that can exchange data among themselves, making use of the recently defined data spaces at European level. Edge IoT distributed platforms with embedded AI capabilities applied to industrial sectors and use Research on advanced edge AI platforms to exchange abstract data sets and make use of all defined data spaces at the European level. Convergence of technologies Edge IoT platforms that combine distributed architectures converging mesh, DLT and AI technologies. Advanced edge IoT AI-based platforms with integrated cognitive functions for federated learning and other emerging distributed learning technologies. Advanced edge IoT platforms including digital twining technologies to address the metaverse continuum for novel immersive applications.
© AIOTI. All rights reserved. 66 17. IoT Verification, Validation and Testing (VV&T) Methods The broad range of existing security certification schemes for products, systems, domains, solutions, services, and organizations derives on a heterogeneous environment of solutions, making it difficult to understand what is needed to achieve a certain level of security in each context or technology. This heterogeneity also makes comparing certified devices more difficult, especially when different certification approaches, countries, and contexts are used. Currently, there is no unified solution that copes with these problems, facilitating the process of comparing and assessing the security levels. 17.1 Technological developments Due to the dynamism of security defined by the new vulnerabilities discovered, affecting the security or the change of the domain that have different security requirements (e.g., IoT devices that passes from a medical to a home domain), the certification approach must adapt to these changing conditions. An agile certification process is required to ensure such security level is up to date during the lifecycle of a device. In addition, the approach must cope with the business requirements and needs of the market. It means that security certification approaches should be efficient and cost-effective, so the market product launch is not delayed. 17.2 Main Trends, Issues and Challenges While pure functional verification of IoT systems still poses challenges, e.g., due to HW/SW integration issues, AI at the edge, and the distributed nature of IoT systems, verifying nonfunctional properties, such as cybersecurity or resource consumption, usually is at least as challenging. Also, integrating untrusted third party IoT devices in sensitive or safety-critical systems often comes with the challenge to prove the absence of hidden “unwanted” functionality (e.g., backdoors), which can go as deep as needing to employing techniques like binary code analysis. The edge IoT distributed, heterogeneous systems employ multi-languages for implementation or for services edge IoT devices provide. Distributed systems use different forms of concurrent programming that increase HW concurrency and sources of heterogeneity. Developing reliable, safe, and secure edge IoT distributed systems is a trade-off process that must be able to also scale among the different level of performance that are needed in the different configurations. The development requires novel models, logical notations, verification techniques, extensions, improvements, and combinations of existing ones to catch the behaviour of such systems and ensure that they meet various specific requirements. In this context, advanced formal methods applied to the verification and validation of edge IoT concurrent and distributed heterogeneous systems can provide new understandings of the efficiency of these methods, when compared with other approaches. Given the financial pressure on IoT device development, any verification activity must be cost effective and efficient, hence, the automation of verification tasks is an important trend. Cost and efficiency of verification also is heavily linked to the proper usage of design methodologies like “security-by-design”, as rigorous design will significantly cut down testing efforts. Verification of non-functional properties like cybersecurity lags behind functional verification, even if the former can have even bigger financial risk attached.
© AIOTI. All rights reserved. 67 Research shall focus on new and improved methods for the automated verification of nonfunctional HW/SW properties of IoT systems. In addition, the functional verification of edge IoT AI-based solutions, which is a largely open challenge, must be addressed. Verification research on edge IoT system should consider virtual environments (incl. digital twins), and design approaches promising greater efficiency at reduced error - and vulnerability rate into account. This may include formal development/rigorous design, low-code/model-based and other approaches. Finally, investigating automating and improving the scalability of verification approaches for third-party “black-box” systems, which look at the HW and binary SW level of IoT systems, is required. The heterogeneity of edge IoT distributed systems makes deploying and managing these systems complex. The distributed architecture of these systems adds to the complexity, as the IoT systems are deployed across networks and spatial boundaries. Distributed systems display a high level of concurrency, which is a challenge in estimating the consistency and correctness of these systems, mainly when their behaviours are not appropriately defined. In the edge distributed systems, IoT devices become reactive systems, constantly interacting with the environment, and adjusting their states. The distributed environment and the changing behaviour of edge IoT devices drive the IoT applications to be very dynamic and, therefore, exposed to unexpected behaviour. Identifying unexpected behaviour and assuring that the demanded behaviour is followed can create a challenge in dynamic systems. Intended behaviours in edge IoT distributed systems can be defined as properties. Model-checking techniques are used to see whether these properties hold onto the formal specification of the system. In this context, property specification, verification and virtual validation can help detect errors in edge IoT dynamic and distributed systems. Further research must consider upcoming technologies such as HW/SW neuromorphic and swarm computation. Of further interest is the automation and integration of the design/verification/debug cycle with operational data: automated repair, diagnosis, and updates for systems in the field based on observed failures. 17.3 Research Priorities Timeline Table 15 IoT Verification, Validation and Testing Methods research priorities Topic Short Term Medium Term Long term 2023-2024 2025-2027 2028-2030 Edge IoT AIbased systems VV&T Virtual validation of edge IoT based system including AIbased components. VV&T of self-X behaviours (e.g., selfconfiguration, self-healing, and self-adaption, etc.) of edge IoT systems. Development of VV&T methods and techniques for real time and time constrained AI-based edge IoT distributed systems. Distributed systems VV&T Validation methods for edge IoT distributed systems focusing on both proof-based verification and systematic testing. Mesh networking systems integrated as part of edge IoT infrastructure. Edge IoT distributed systems embedding digital twin solutions. Heterogenous systems VV&T HW/SW neuromorphic and swarm computation applied to edge IoT distributed systems. Heterogenous IoT distributed systems comprising of DLT platforms, edge AI frameworks and various ML implementations. Federation of multi-blockchainbased data processing, edge IoT computing and swarm intelligence. Heterogenous edge IoT interoperability testing.
© AIOTI. All rights reserved. 68 18. IoT Trustworthiness and Edge Computing Systems Dependability Trustworthiness of the edge IoT systems can be defined as the degree of confidence one has that the edge IoT system performs as expected. Designing IoT edge trustworthiness into edge IoT systems requires the identification of characteristics and mechanisms of trust that can be embedded into the edge IoT system. Trust in edge IoT systems is realised at the intersection of several dependability characteristics, as shown in Figure 8. As a result, trust in an edge IoT system is a concept with multiple dimensions, combining dependability characteristics with human and machine behaviour. In this context, there is a need for a greater understanding of how individuals interact with edge IoT devices and how edge devices interact with other devices/things concerning the extension of trust. 18.1 Technological developments The dependability 50 of a system reflects the user's degree of trust in that system. Dependability characteristics applied to edge IoT systems are safety, security, reliability, resilience, availability, connectability, and maintainability as illustrated in Figure 8. Figure 8 Edge IoT system dependability characteristics The security aspects for IoT are critical due to the amount, diversity, and potential impact of security threats on everyday critical infrastructures. Security aspects represent one of the most pressing barriers to the adoption of large-scale IoT deployments. 50 Laprie, JC. (1995). Dependability — Its Attributes, Impairments and Means. In: Randell, B., Laprie, JC., Kopetz, H., Littlewood, B. (eds) Predictably Dependable Computing Systems. ESPRIT Basic Research Series. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-79789-7_1.
© AIOTI. All rights reserved. 69 Recent cyberattacks, such as the Mirai 51 or BrickerBot 52 IoT Botnets 53 , highlight the need to design appropriate protection mechanisms for the next generation of IoT devices. To address cybersecurity concerns, one of the most ambitious initiatives in Europe is the definition of a cybersecurity certification framework. In this direction, the recent "Cybersecurity Act" 54 represents the European Commission effort to strengthen the European Union Agency for Network and Information Security (ENISA) 55 role for the definition of this framework by providing additional guidelines and challenges for its realization. The definition of a cybersecurity framework requires efforts from different knowledge domains to satisfy the requirements and needs of various stakeholders, such as manufacturers, institutions, legal firms, and consumers. On the one hand, the very broad range of existing security certification schemes for products, systems, domains, solutions, services, and organizations derives on a heterogeneous environment of solutions, making difficult understanding what is needed to achieve a certain level of security in each context or technology. This heterogeneity also makes comparing certified devices more difficult, especially when these devices are certified with different certification approaches, countries, and contexts. Currently, there is not a unified solution that copes with these problems, facilitating the process of comparing and assessing the security level. On the other hand, due to the dynamism of security (e.g., a new vulnerability discovered affecting the security or a change of domain such as a device that passes from a medical to a home domain, implying different security requirements), the certification approach must consider these fast-changing conditions, which could affect the device’s security level. IoT Trustworthiness requires integrating stakeholders of the critical sector (network operators, technology suppliers, cybersecurity solution providers, standard and certification bodies, etc.) for defining cybersecurity standards and test procedures. 18.2 Main Trends, Issues and Challenges With IoT devices taking over critical tasks in system control and/or health areas, dependability is one major trend: data provided must be reliable and trustworthy. Dependable edge IoT systems will incorporate devices and services of multiple different vendors which will only increase the challenge. For example, such systems need to cope with data privacy, cybersecurity, reliability, and safety at the same time. Without proper standards and planning, guaranteeing all these attributes over a set of diverse devices from different vendors (being deployed in different networks and data centres) is next to impossible. 51 G. Tatebatake and S. Yamaguchi, "Mathematical Modeling and Analysis of the Dictionary Attack Mechanism in IoT Malware Mirai," 2022 IEEE International Conference on Consumer Electronics-Asia (ICCE-Asia), Yeosu, Korea, Republic of, 2022, pp. 1-5, doi: 10.1109/ICCE-Asia57006.2022.9954838. 52 C. Kolias, G. Kambourakis, A. Stavrou and J. Voas, "DDoS in the IoT: Mirai and Other Botnets," in Computer, vol. 50, no. 7, pp. 80-84, 2017, doi: 10.1109/MC.2017.201. 53 R. Raman, "Detection of Malware Attacks in an IoT based Networks," 2022 Sixth International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC), Dharan, Nepal, 2022, pp. 430-433, doi: 10.1109/I-SMAC55078.2022.9987253. 54 https://digital-strategy.ec.europa.eu/en/policies/cybersecurity-act. 55 https://www.enisa.europa.eu/
© AIOTI. All rights reserved. 70 Dependability management over the complete product lifecycle will become key and include techniques such as AI-augmented DevOps, safety and security-by design, dependability architectures, low-code development, code-re-use, and proper security management over the whole product lifecycle. In a more technology-dependent world, cybersecurity concerns have attracted an increasing interest from companies, regulatory bodies, and end users. With the IoT, such aspects are exacerbated due to the amount, diversity, and potential impact of security threats on everyday critical infrastructures. Indeed, nowadays, security aspects represent one of the most important barriers for the adoption of large-scale IoT deployments. Therefore, an agile certification process is required to ensure such security level is up to date during the lifecycle of a device. In addition, the approach must cope with the business requirements and needs from the market. It means that security certification approaches should be efficient and cost effective, so the product launch in the market is not delayed. As a result of this process, a cybersecurity label is looked for, which contains the security level achieved by the device. It should provide a clear visibility of the level of security achieved but also give a non-ambiguous and complete representation of the results of the cybersecurity evaluation process. This is rather difficult to achieve, because in comparison to the energy label, which measure a physical quantity, the measurement of security is far more complex, involving several security properties or dimensions. In addition, the label should be able to show in real time the security offered by the device. The IoT and edge computing systems have challenges in addressing the automated cybersecurity evaluation towards a more agile and cost-effective certification, dealing with the IoT dynamism. Finally, there is also the need for objectives and evidence-based evaluation methodologies that would allow for a homogeneous evaluation, including comparability aspects. This limits the security management mechanisms addressing the whole lifecycle of the IoT device, dealing with the security changes that might invalidate the certificate.
© AIOTI. All rights reserved. 71 18.3 Research Priorities Timeline Table 16 IoT Trustworthiness and Edge Computing Systems Dependability research priorities Topic Short Term Medium Term Long term 2023-2024 2025-2027 2028-2030 Trustworthiness Edge IoT trustworthiness of ML models and explainability of AI (XAI) models used in Federated Learning. Trustworthiness models for edge IoT self-adapting systems based on digital twin and AI-based technologies. Frameworks providing guidelines, good practices and standards oriented to end-toend trust in edge IoT systems. Dependability Define methods and tools to support the dependability system properties definition, composition and validation and relate the properties to different standards addressing different technologies. Dependability properties at different layers of the edge IoT architecture by considering scalable concepts for HW, SW, connectivity, Al algorithms (inference, learning) and the design of flexible heterogeneous architectures that optimise the use of computing resources and the use of resource-constrained devices. Virtualisation and simulation tools for managing the evaluation of edge IoT system dependability. Benchmarking Define different benchmarking methods and techniques of trust for an edge IoT system and provide compliance to an agreed-upon standard via certification schemes. Benchmarking heterogenous edge IoT systems based on DLT, digital twins, mesh networking and AI technologies. Benchmarking reference data sets for training edge IoT systems for federated learning.
© AIOTI. All rights reserved. 72 Contributors Editors: Ovidiu Vermesan, SINTEF Valerio Frascolla, Intel Reviewer: Damir Filipovic, AIOTI Secretary General Contributors: Alessandro Liani, Video Systems, Italy André Bourdoux, IMEC, Belgium Antonio Skarmeta, Universidad de Murcia, Spain Antonio Soriano Asensi, Universitat de Valencia, Spain Asbjorn Hovsto, Hafenstrom, Norway Charles Sturman, Huawei, UK Cian O Murchu, Tyndall National Institute, Ireland Daniela Buleandra, SIMAVI, Romania François Fischer, FSCOM, France George Suciu, BEIA Consult International SRL, Romania Georgios Karagiannis, Huawei, Germany Ignacio Lacalle, Universitat Politecnica de Valencia, Spain Ilia Pietri, Intracom Telecom Solutions, Greece Jara Pascual, Collabwith Group, Netherlands Jesus Angel Garcia Sanchez, Indra Sistemas Joachim Hillebrand, Virtual Vehicle Research, Austria Juho Pirskanen, Wirepas, Finland Konstantinos Loupos, Inlecom, Greece Konstantinos Thivaios, NETCOMPANY INTRASOFT, Luxembourg
© AIOTI. All rights reserved. 73 Lazaros Karagiannidis, Institute of Communication and Computer Systems, Greece Luis Munoz, University of Cantabria, Spain Luis Perez-Freire, Fundacion Centro Tecnoloxico de Telecomunicacions de Galicia, Spain Marcin Plociennik, Poznan Supercomputing and Networking Center, Poland Marco Gonzalez Hierro, IKERLAN, Spain Mario Drobics, AIT Austrian Institute of Technology GmbH, Austria Martin Serrano, Insight SFI research Centre for Data Analytics, Ireland Maurizio Spirito, Fondazione LINKS, Italy Mikel Larranaga, Fundacion Tekniker, Spain Mirko Presser, Aarhus University, Denmark Monica Florea, SIMAVI, Romania Natalie Samovich, Enercoutim, Portugal Oltion Xhezo, Vodafone, UK Ovidiu Vermesan, SINTEF, Norway Pedro Maló, UNPARALLEL Innovation, Portugal Pedro Ruiz, Integrasys, Spain Philippe Sayegh, Verses Global, Netherlands Pierre Yves Danet, 48deg79min-Consulting, France Pieter Becue, IMEC, Belgium Pietro Dionisio, Medea, Italy Ranga Rao Venkatesha Prasad, Technical University Delf, Netherlands Ricardo Vitorino, Ubiwhere, Portugal Roumen Nikolov, Virtech, Bulgaria Roy Bahr, SINTEF, Norway Rute Sofia, fortiss, Germany Sean McGrath, University of Limerick, Ireland Sergio Gusmeroli, Politecnico di Milano, Italy Srdjan Krco, DunavNET doo, Serbia Valerio Frascolla, Intel, Germany Veronica Quintuna Rodriguez, Orange, France