Deliverable D3.3 Initial analysis of architectural enablers and framework

A holistic flagship towards the 6G network platform and system, to inspire digital transformation, for the world to act together in meeting needs in society and ecosystems with novel 6G services Deliverable D3.3 Initial analysis of architectural enablers and framework Hexa-X-II project has received funding from the Smart Networks and Services Joint Undertaking (SNS JU) under the European Union’s Horizon Europe research and innovation programme under Grant Agreement No 101095759. Date of delivery: 30/04/2024 Version: 1.1 Project reference: 101095759 Call: HORIZON-JU-SNS-2022 Start date of project: 01/01/2023 Duration: 30 months Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 2 / 195 Document properties: Document Number: D3.3 Document Title: Initial analysis of architectural enablers and framework Editor(s): Mårten Ericson (EAB), Panagiotis Botsinis (APP), Merve Saimler (EBY), Ozgur Akgul (NFI), Sokratis Barmpounakis (WIN), Milan Groshev (UC3) Authors: Ozgur Akgul (NFI), Bassem Arar (TUD), Michael De Angelis (NXW), Sokratis Barmpounakis (WIN), Jaap van de Beek (LTU), Giacomo Bernini (NXW), Sonai Biswas (TUD), , Panagiotis Botsinis (APP), Pere Garau Burguera (AAU), Panagiotis Charatsaris (ICC), Panagiotis Demestichas (WIN), Maria Diamanti (ICC), Toni Dimitrovski (TNO), Sameh Eldessoki (APP), Mårten Ericson (EAB), Mohammad Asif Habibi (RPTU Kaiserslautern), Apostolos Kousaridas (NGE), Milan Groshev (UC3), Alperen Gundogan (APP), Hasanin Harkous (NGE), Hamed Hellaoui (NFI), Selim Ickin (EAB), Paola Iovanna (EAB), Grigorios Kakkavas (ICC), Bahare M. Khorsandi (NGE), Slawomir Kukliński (OPL), Gerald Kunzmann (NGE), Hannes Larsson (EAB), Marvin Manalastas (NFI), Beatriz Mendes (UBW), Swaraj S. Nande (TUD), Antonio de la Oliva (UC3), Ece Ozturk (NGE), Torgny Palenius (SON), Symeon Papavasliou (ICC), Ignacio Labrador Pavón (ASA), Janusz Pieczerak (OPL), Bartosz Rak (OPL), Merve Saimler (EBY), Hans D. Schotten (RPTU Kaiserslautern), Erin Seder (NXW), Vivek Sharma (SON), Mohammad Soliman (NGE), Heiko Straulino (NGE), Halina Tarasiuk (OPL), Olav Tirkkonen (AAU), Nassima Toumi (TNO), Vasilis Tsekenis (WIN), Antonio Varvara (TIM), Stefan Wänstedt (EAB), Zi Ye (LTU), Milan Zivkovic (APP), Marcin Ziółkowski (OPL) Contractual Date of Delivery: 30/04/2024 Dissemination level: PU1 Status: Final Version: 1.1 File Name: Hexa-X-II_D3.3_v1.1 Revision History Revision Date Issued by Description 0.1 2023-09-25 Hexa-X-II WP3 Template for Deliverables/IRs 0.2 2023-12-15 Hexa-X-II WP3 Cleaned-up version 1 SEN = Sensitive, only members of the consortium (including the Commission Services). Limited under the conditions of the Grant Agreement PU = Public Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 3 / 195 0.3 2024-01-16 Hexa-X-II WP3 Reduced pages, added Appendix 0.4 2024-02-1 Hexa-X-II WP3 Internal full review 0.5 2024-02-15 Hexa-X-II WP3 Addressing review comments 0.6 2024-03-02 Hexa-X-II WP3 External review 0.7 2024-03-26 Hexa-X-II WP3 GA version 0.8 2024-04-15 Hexa-X-II WP3 GA version, minor updates 1.0 2024-04-28 Hexa-X-II WP3 Final version submitted to EC 1.1 2025-10-29 Hexa-X-II WP3 Clean-up of some track changes Abstract This is the second public deliverable from Hexa-X-II project work package 3 – “Initial analysis of architectural enablers and framework”. This deliverable develops and analyses innovative enablers required for a data driven architecture to power new services, both for communication and for beyond communications. In addition, the deliverable describes new means for a modular cloud-native network to improve flexibility and reduce signalling, as well as enablers for new access and flexible topologies for improved reliability. The architecture inherently uses AI, both for orchestration and as a service, and incorporates NTN for enable ubiquitous coverage. Keywords 6G architecture, data-driven architecture, AIaaS, modular networks, JCAS, flexible topologies, cloud transformation Disclaimer Funded by the European Union. The views and opinions expressed are however those of the author(s) only and do not necessarily reflect the views of Hexa-X-II Consortium nor those of the European Union or Horizon Europe SNS JU. Neither the European Union nor the granting authority can be held responsible for them. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 4 / 195 Executive Summary The Hexa-X-II project is a flagship initiative bringing together key stakeholders in Europe for 6G research, continuing the work of the Hexa-X project. Hexa-X-II includes the key industry players in telecom and major research institutes; a combination of innovative knowledge capable of introducing new value chains for future connectivity solutions. Furthermore, the Hexa-X-II project comprises several work packages that span over important parts of the 6G ecosystem. In this report, results from work in WP3, which deals with the 6G architecture design, are presented. The overarching objective of WP3 is to develop a 6G architecture framework with innovative enablers for beyond communication services. The architecture should be data driven to efficiently power new services, modular to support cloud-native networks and to improve signalling as well as allow new access and flexible topologies for improved reliability. This is the second public deliverable from WP3, called D3.3 “Initial analysis of architectural enablers and framework”. The main objective of this deliverable is to analyse the WP3 enablers introduced in previous deliverable and at the same time identify and define additional requirements that are important for the 6G architecture. The document comprises descriptions of the enablers and studies of the different solutions within the enabler. Each enabler is thereafter summarized, including the benefits and the implications if the enabler would be implemented in the 6G architecture. An important aspect of the architecture is how to support the migration between 5G and 6G. Therefore, the deliverable analyses the outcome of the 4G to 5G migration, and discusses the lessons learned and how to apply them to the 5G to 6G migration. One proposal is to use spectrum sharing for the 5G and 6G integration. The deliverable thereafter analyses the artificial intelligence (AI) enablers for the 6G data-driven architecture. The enablers comprise architectural means and protocols, machine learning (ML) Operations (MLOps), data operations (DataOps), AI as a service (AIaaS), and intent-based management. The AI enablers form a robust framework for seamlessly integrating AI into the compute continuum of 6G networks. To enable flexibility without increasing complexity, 6G needs an easily deployable architecture of modules (i.e., network functions) that can scale to current needs. The deliverable analyses the network modularisation enablers, which focus on different granularities of a module as well as on the evolution of the modules as a result of different deployments and use cases. The idea is to optimize the 6G network functions or modules according to certain key performance indicators such as latency (end to end), procedure completion time, and control plane signalling, while at the same time reduce complexity and improve sustainability. The deliverable thereafter analyses the enablers for new access and flexible topologies which consist of network of networks, multi-connectivity and end to end context awareness management. The network of networks enabler aims to develop so-called subnetworks and integrate these sub-networks into a seamless and ubiquitous communication system. One main component for this is to inherently support non-terrestrial networks (NTN) in 6G, with improved interoperability between satellites and integration with the terrestrial networks. Another type of subnetwork is a network of User Equipment (UE) based on mutual trust, where a management node, i.e., a more capable UE, controls other UEs and helps them perform certain procedures. The deliverable also investigates methods to improve the multi-connectivity as well as aggregating different radio technologies, such as 6G cellular and wireless local area network (WLAN). The context awareness enabler aims to allow network components to dynamically adapt to the context to ensure the expected end to end quality of a service (QoS) and use resources in a more efficient manner. One way to introduce context awareness to the transport network is that a resource orchestrator creates an abstracted view of transport resources and employs a software defined transport controller for resource management, ensuring that the QoS associated with a network slice is met. The beyond communication enabler deals with how to realize new 6G services such as sensing and compute offloading, and how to expose resulting data and relevant service capabilities in a secure, privacy-preserving and efficient manner. To reduce the overhead from data exposure (especially from sensing), the deliverable proposes new functionality that is needed to aggregate and fuse data as well as ensure data privacy and trust. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 5 / 195 One way to improve sensing quality is obviously to carry out more radio measurements prior to exposure of the measurement report to the requesting application, preferably measurements that are geographically distributed. Involving more network nodes (UEs or base stations) naturally leads to architectural challenges of centralized vs. distributed inference and processing of the measurements. The provision of sensing services by next generation communication systems necessitates the introduction of a management function of the sensing. The sensing management function controls the sensing process such as which nodes to involve, processing of sensing data and facilitating an efficient coordination of sensing procedures. As far as compute offloading is concerned, the scope includes investigating new procedures for offloading computations and for controlling which compute node will perform the computations. The off-the-shelf cloud is suitable for a big subset of multimedia human-scale applications, but it has its limitations when it comes to supporting the upcoming latency sensitive 6G use cases. The cloud transformation deals with how to develop the 6G cloud platform for the telecom system. The deliverable proposes an enabler for the integration and orchestration of the compute continuum, including extreme edge resources. The enabler analyses the needed architectural interfaces and components for seamless orchestration and management of the complete compute continuum composed of cloud, edge and extreme edge resources. The analysis results in a proposal of new architectural mechanisms for the Multi-access Edge Computing (MEC) framework to incorporate the extended compute continuum and account for the extreme edge devices. Multi-domain/Multicloud federation is the capability to aggregate cloud services provided by multiple domains and providers into a single, coherent cloud. To this end, the cloud continuum should provide intent-based interfaces for cloud services and the core network should provide intent-based interfaces for network services. The deliverable also includes a detailed description of the two proof-of-concepts (PoCs) belonging to WP3. The first PoCs is called “Distributed ML model training and inference” for a remote-controlled robot use case. With cross-network function training, it demonstrates collaborative training without moving and sharing data between the entities. The cross-network function training enables an ML model to train with richer input obtained from different layers in the network stack. This way, the efficacy of the ML model can be improved. The second PoC “Trustworthy flexible topologies in 6G, leveraging on beyond communication aspects” investigates means for a network that enables versatile, robust and dynamic applications not only intended towards public use but also for industry needs, demonstrating the adaptability and resilience of the 6G network in a warehouse scenario. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 6 / 195 Table of Contents List of Tables ................................................................................................................................ 9 Acronyms and abbreviations .................................................................................................... 13 1 Introduction ......................................................................................................................... 21 1.1 Objectives..................................................................................................................... 21 1.2 Enabler definition and Methodology ........................................................................... 21 1.3 Structure ....................................................................................................................... 22 2 6G Architecture overview .................................................................................................. 23 2.1 WP3 objectives mapping to the E2E 6G system blueprint .......................................... 23 2.2 5G to 6G Migration ...................................................................................................... 24 2.2.1 Introduction ............................................................................................................. 24 2.2.2 Evolved 5GC and 6GC ........................................................................................... 25 2.2.3 5G-6G Multi-Radio Spectrum Sharing ................................................................... 26 2.2.4 Lower layer split ..................................................................................................... 26 3 AI enablers for data-driven architecture .......................................................................... 28 3.1 Data-driven architectural means and protocols ............................................................ 29 3.1.1 Introduction ............................................................................................................. 29 3.1.2 Architectural Implications....................................................................................... 30 3.1.3 Evaluation ............................................................................................................... 32 3.1.4 Summary ................................................................................................................. 33 3.2 MLOps ......................................................................................................................... 34 3.2.1 Introduction ............................................................................................................. 34 3.2.2 Architectural Implications....................................................................................... 35 3.2.3 Evaluation ............................................................................................................... 40 3.2.4 Summary ................................................................................................................. 42 3.3 AIaaS ............................................................................................................................ 43 3.3.1 Introduction ............................................................................................................. 43 3.3.2 Architectural Implications....................................................................................... 44 3.3.3 Evaluation ............................................................................................................... 46 3.3.4 Summary ................................................................................................................. 48 3.4 DataOps ........................................................................................................................ 49 3.4.1 Introduction ............................................................................................................. 49 3.4.2 Architectural Implications....................................................................................... 49 3.4.3 Evaluation ............................................................................................................... 50 3.4.4 Summary ................................................................................................................. 51 4 Network modularisation ..................................................................................................... 53 4.1 6G Network modularisation ......................................................................................... 53 4.1.1 Introduction ............................................................................................................. 53 4.1.2 Module design and composition ............................................................................. 54 4.1.3 E2E module interfaces and interaction ................................................................... 60 4.1.4 Summary ................................................................................................................. 65 4.2 E2E service design in modular 6G ............................................................................... 66 4.2.1 Introduction ............................................................................................................. 66 4.2.2 Extended/E2E network modularity in UP and CP .................................................. 66 4.2.3 Network autonomy and adaptiveness via modularization ...................................... 72 4.2.4 Summary ................................................................................................................. 76 5 Architectural enablers for new access and flexible topologies ........................................ 78 5.1 Network of networks .................................................................................................... 78 5.1.1 Introduction ............................................................................................................. 78 5.1.2 Architectural implications ....................................................................................... 79 Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 7 / 195 5.1.3 Evaluations .............................................................................................................. 82 5.1.4 Summary ................................................................................................................. 85 5.2 Multi-connectivity ........................................................................................................ 86 5.2.1 Introduction ............................................................................................................. 86 5.2.2 Architectural implications ....................................................................................... 86 5.2.3 Evaluations .............................................................................................................. 90 5.2.4 Summary ................................................................................................................. 92 5.3 E2E context awareness management ........................................................................... 93 5.3.1 Introduction ............................................................................................................. 93 5.3.2 Architectural implications ....................................................................................... 94 5.3.3 Evaluations .............................................................................................................. 97 5.3.4 Summary ............................................................................................................... 100 6 Architectural enablers for network beyond communications ....................................... 102 6.1 Exposure and data management ................................................................................. 102 6.1.1 Introduction ........................................................................................................... 102 6.1.2 Architectural implications ..................................................................................... 103 6.1.3 Preliminary workflows and evaluation ................................................................. 104 6.1.4 Summary ............................................................................................................... 105 6.2 JCAS protocols, signalling and procedures................................................................ 106 6.2.1 Introduction ........................................................................................................... 106 6.2.2 Architectural implications ..................................................................................... 107 6.2.3 Preliminary workflows and evaluation ................................................................. 109 6.2.4 Summary ............................................................................................................... 110 6.3 Compute offloading protocols, signalling and procedures ......................................... 111 6.3.1 Introduction ........................................................................................................... 111 6.3.2 Architectural implications ..................................................................................... 111 6.3.3 Preliminary workflows and evaluation ................................................................. 113 6.3.4 Summary ............................................................................................................... 116 6.4 Application-/Device-specific BCS optimisation architectural enablers ..................... 117 6.4.1 Introduction ........................................................................................................... 117 6.4.2 Architectural implications ..................................................................................... 118 6.4.3 Preliminary workflows and evaluation ................................................................. 120 6.4.4 Summary ............................................................................................................... 124 7 Virtualisation and Cloud transformation ....................................................................... 126 7.1 Integration and orchestration of extreme edge resources in the compute continuum 126 7.1.1 Introduction ........................................................................................................... 126 7.1.2 Architectural modifications................................................................................... 127 7.1.3 Preliminary workflows and evaluation ................................................................. 132 7.1.4 Summary ............................................................................................................... 134 7.2 Multi-domain/multi-cloud federation ......................................................................... 135 7.2.1 Introduction ........................................................................................................... 135 7.2.2 Architectural modifications................................................................................... 136 7.2.3 Preliminary workflows and evaluation ................................................................. 140 7.2.4 Summary ............................................................................................................... 143 7.3 Cloud transformation with quantum technologies ..................................................... 144 7.3.1 Introduction ........................................................................................................... 144 7.3.2 Architectural modifications................................................................................... 145 7.3.3 Summary ............................................................................................................... 146 8 Proof of Concepts .............................................................................................................. 147 8.1 Component-PoC #B.2: Distributed Model Training and Inference ........................... 147 8.1.1 Remote controlled robot use case ......................................................................... 147 8.1.2 Distributed AI-Enabled Technology: Split Learning ............................................ 148 8.1.3 The input node ...................................................................................................... 151 Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 8 / 195 8.1.4 The generalization node ........................................................................................ 152 8.1.5 The output node .................................................................................................... 152 8.1.6 Kubernetes pods and settings ................................................................................ 153 8.1.7 Measurement Points and Evaluation Method ....................................................... 153 8.1.8 Results ................................................................................................................... 153 8.2 Component-PoC #B.3: Trustworthy flexible topologies in 6G, leveraging on “beyond communication” aspects ............................................................................................ 154 8.2.1 Inventory management use-case ........................................................................... 154 8.2.2 Flexible topology use-case .................................................................................... 155 9 Summary and Conclusions ............................................................................................... 158 10 References .......................................................................................................................... 160 11 Annex A: Further details of the studies .......................................................................... 167 11.1 Detailed Versions of Studies in MLOPs .................................................................... 167 11.1.1 Federated learning approach between different city verticals ............................... 167 11.1.2 Strategies and mechanisms for distributed AI and AIaaS functions management 171 11.1.3 Intent Based Management ..................................................................................... 172 11.2 Network modularisation ............................................................................................. 173 11.2.1 5G Service Based Architecture ............................................................................. 173 11.2.2 RAN modularity .................................................................................................... 175 11.3 Architectural enablers for new access and flexible topologies .................................. 175 11.3.1 Multi-connectivity ................................................................................................. 175 11.3.2 E2E context awareness management .................................................................... 181 11.4 Decentralised compute-continuum smart management ............................................. 185 Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 9 / 195 List of Tables Table 1-1 WP3 Objectives ............................................................................................................................... 21 Table 3-1: Data driven Architectural means and Protocols enabler ................................................................ 34 Table 3-2: MLOps Enabler .............................................................................................................................. 42 Table 3-3: AIaaS Enabler ................................................................................................................................ 48 Table 3-4: DataOps Enabler ............................................................................................................................ 51 Table 4-1 Advantages and drawbacks of fine-grained modularisation ........................................................... 56 Table 4-2 Sources of delay of inter-NF interactions in a virtualized environment ......................................... 59 Table 4-3 Network modularisation enabler summary ..................................................................................... 65 Table 4-4 Summary of NGAP functionality and corresponding SBA functionality ....................................... 70 Table 4-5 Benefits and implications of "E2E service design in modular 6G" enabler. ................................... 77 Table 5-1 Benefits and implications of "Network of networks" enabler ......................................................... 85 Table 5-2 MC simulation parameters for the scenario 1 and 2 ........................................................................ 90 Table 5-3 Benefits and implications of "Multi-connectivity" enabler. ............................................................ 93 Table 5-4 Benefits and implications of "E2E context awareness management" enabler .............................. 100 Table 6-1 Exposure and data management summary table ........................................................................... 105 Table 6-2 JCAS protocols, signaling, and procedures summary table. ......................................................... 110 Table 6-3 Compute offloading protocols, signalling, and procedures summary table. ................................. 116 Table 6-4 Application-/Device-specific BCS data consuming functions summary table ............................. 124 Table 7-1: Benefits and implications of "Integration and orchestration of extreme edge resources" enabler 134 Table 7-2 Cloud Continuum nodes placement with a geographic service diameter equal to 50 km ............. 141 Table 7-3 Cloud Continuum nodes placement with a geographic service diameter equal to 150 km ........... 142 Table 7-4 Benefits and implications of “Multi-cloud/multi-domain federation” enabler ............................. 144 Table 7-5:Benefits and implications of “Cloud transformation with quantum technologies” enabler .......... 146 Table 11-1 Intent based management enabler summary ............................................................................... 172 Table 11-2 Cell-free parameters .................................................................................................................... 175 Table 11-3 Computational complexity of different path computation algorithms ........................................ 184 Table 11-4 Object detection models details .................................................................................................. 185 Table 11-5 Hardware setup details ................................................................................................................ 185 List of Figures Figure 2-1 Illustrative mapping of WP3 objectives to the system blueprint ................................................... 23 Figure 2-2 General architecture options for 6G. .............................................................................................. 25 Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 16 / 195 FTN Flexible Topology Node GA Genetic Algorithm GCL Guarded Command Language gNB gNodeB GTD Global Topology Database GUAMI Globally Unique AMF ID (GUAMI). HFL Hierarchical Federated Learning HPC High-Performance Computing I-UPF Intermediate UPF IAB Integrated Access and Backhaul IDC Input Data Compression IFFT Inverse Fast Fourier Transform IID Independent Identically Distributed INC Integration of Network and Compute IPD Infrastructure Partitions Database IRN Infrastructure Registry Node IRS Infrastructure Registry Service ISL Inter-Satellite Link ISM Ingress Steering Module ISP Internet Service Providers ISPN Infrastructure Status Prediction Node ISPS Infrastructure Status Prediction Service JCAS Joint Communications and Sensing KPI Key Performance Indicator KVI Key Value Indicator LCM LifeCycle Management LEO Low Earth Orbit LIM Lawful Intercept Module LLS Lower Layer Split LMF Location Management Function LWA LTE-WLAN Aggregation LWIP LTE-WLAN Radio Level Integration using IPSec Tunnel M-NFVO Modified NFVO M-VIM Modified VIM M&O Management and Orchestration MAC Medium Access Control MBSM Multimedia Broadcast Services Module MC Multi-Connectivity Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 17 / 195 MDAS Management Data Analytics System ME Market Equilibrium MEC Multi-access Edge Computing MEO MEC Orchestrator MEP Multi-access Edge Platform MEPM MEC Platform Manager MgtN Management Node MLOps Machine Learning Operations MN Master Node MNO Mobile Network Operator MP Management Plane MRSS Multi-Radio Spectrum Sharing MST Minimum Spanning Tree N6TM N6 Tunnelling Module NARD NS Allocated Resources Database NF Network Function NFV Network Function Virtualization NGAP NG application protocol NN Neural Network NOMA Non-Orthogonal Multiple Access NR New Radio NRF NF Repository Function NS Network Service NSA Non-Standalone NSCR NS Charging Record NSRD NS Requirements Database NSTS Network Slice Topology Selection NTN Non-Terrestrial Network NW Network NWDAF Network Data Analytics Function O-RAN Open RAN ODM On-Demand Function ON Offloading Node OSR O-RAN Slice Request OSS Operation Support System OTA Over-The-Air PbNF Procedure-based NF PCF Policy Control Function Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 18 / 195 PCT Procedure Completion Time PHY Physical Layer PNF Physical Network Function PNI-NPN Public network integrated Non-Public Network architecture PoC Proof of Concept PoI Partition of Infrastructure PRTC Primary Reference Time Clock PSCell Primary Secondary Cell PTP Precision Time Protocol QC Quantum Computing Qgate Quantum Gate QKD Quantum Key Distribution QoCS Quality of Compute Service QoE Quality of Experience QoS Quality of Service RAN Radio Access Network RAT Radio Access Technology RB Radio Bearer RDAE Resource Layer Data Analytic Engine RedM Redundancy Module RegNF Registration NF RIC RAN Intelligent Controller RL Reinforcement Learning RLC Radio Link Control RNIS Radio Network Information Service RNTI Radio Network Temporary Identifier ROF Resource Layer Orchestrated Function ROP Resource Layer Operator Portal RP Resource Plane RRC Radio Resource Control RSE Resource layer Security Engine RSMA Rate-Splitting Multiple Access RU Radio Unit SBA Service Based Architecture SBI Service-Based Interface SCF Sensing Control Function SDLA Semantic Deep Learning Analyzer SDN Software Defined Network Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 19 / 195 SDNDC SDN Domain Controller SE Satisfaction Equilibrium SeMF Sensing Management Function SeRS Sensing Radio Signals SESM Semantic Edge Slicing Module SFC Service Function Chain SLA Service Level Agreement SMF Session Management Function SN Secondary Node snCP Subnetwork Control Plane SPF Sensing Processing Function SR Service Request SRN Service Registry Node SRS Services Registry Service SSB Synchronisation Signal Block SubNW Subnetwork T-GM Telecom Grandmaster TCO Total Cost of Ownership TDD Time Division Duplex tMEC traditional MEC TN Terrestrial Network TNL Transport Network Layer ToA Time of Arrival ToD Time of Departure ToF Time-of-Flight TS Traffic Source TSN Time-Sensitive Networking UALCMP User Application LCM Proxy UAV Unmanned Aerial Vehicle UDM Unified Data Management UDR User Data Repository UDSF Unstructured Data Storage Function UE User Equipment UL Uplink UL-CL Uplink Classifier ULM Uplink Module UP User Plane UPA User Plane Adapter Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 20 / 195 UPF User Plane Function URLLC Ultra-Reliable Low-Latency Communications VIM Virtual Infrastructure Manager VM Virtual Machine VME VNF Migration Engine VNO Virtual Network Operator WCA WLAN-Cellular Aggregation WFQ Weighted Fair Queuing WLAN Wireless Local Area Network, also referred to as WiFi in the report WN Worker Node WPO Work Package Objective WT WLAN Terminal XR Extended Reality Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 21 / 195 1 Introduction The Hexa-X-II project is a flagship initiative bringing together key stakeholders in Europe for 6G research, continuing the Hexa-X project work. Hexa-X-II includes the key industry players in telecom and major research institutes; a combination capable of introducing new value chains for future connectivity solutions. Furthermore, the Hexa-X-II project comprises several work packages that spans over important parts of the 6G ecosystem. In this report results from work in WP3 are presented, which deals with the 6G architecture design. The overarching objective of WP3 is to develop a 6G architecture framework and innovative enablers for a data driven architecture capable of powering new services, such as beyond communications services, a modular cloud-native network for improved signalling as well as new access and flexible topologies for improved reliability. This is the second public deliverable from WP3, called D3.3 “Initial analysis of architectural enablers and framework”. 1.1 Objectives The main objective of this deliverable is to analyse the WP3 enablers and at the same time define new requirements that are important for the 6G architecture. The long-term objectives of WP3 are presented in Table 1-1. The three Work Package Objectives (WPO) for WP3 are the 6G architecture for AI and beyond communications, how to combine the cloud technology for a modular, scalable and extendable architecture and an architecture for flexible topologies. Table 1-1 WP3 Objectives Objective Objective description Chapter WPO3.1: 6G architecture for AI and beyond communications Develop and analyse a 6G architecture framework and new innovative enablers for the beyond communications and data driven architecture, identify requirements a data-driven architecture will have on protocols, interfaces, data, and network nodes. Chapter 3 and 6 WPO3.2: Combine the cloud technology for a modular, scalable and extendable architecture Define and analyse solutions that combine cloud technology flexibility with distributed processing nodes into self-contained modules with minimum dependency that can be used to extend and scale the network deployments in stepwise manner Chapter 4 and 7 WPO3.3: Architecture for flexible topologies Develop and analyse new access for flexible topologies and local communications, including different types of multiconnectivity, node roles and node coordination, as well as design control and management solutions for programmable and context-aware transport Chapter 5 1.2 Enabler definition and Methodology In this deliverable, the term enabler is used extensively. In WP3, enabler is defined as a technical area with common objectives. The technical area (or enabler) may contain several different types of solutions (or components) aiming for the same objective. For example, the enabler Network of networks in Section 5.1, aims to develop a seamless and ubiquitous communication system. To solve this, several different solutions (or components) are necessary, for example the use of Non-Terrestrial Networks (NTN) and different types of (terrestrial) sub-networking solutions. These solutions applied together aim to increase the coverage and reliability of the networks. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 22 / 195 The methodology of this deliverable is to investigate and analyse the architectural implications for each enabler and the different components belonging to the enabler. The architectural implications describe how the architecture need to be modified in order to implement and introduce the enabler in the 6G architecture. For some of the enablers there is also a dedicated evaluation section with more focus on simulation results, etc. Next, the enablers are summarized based on their benefits and implications in a 6G system. As can be seen in Table 1-1, the WP3 objectives cover broad areas (technically broader areas than the enablers in many cases), and in the final deliverables, we will also aim to analyse how different enablers will work together in order to fulfil the WP3 objectives. 1.3 Structure This document is structured as follows: Chapter 2 gives a brief overview of the envisioned architecture and 5G to 6G migration. Chapter 3 describes the AI enablers for a data driven architecture. Chapter 4 describes the network modularisation, i.e., how to build an architecture of modules that can scale based on current needs. Chapter 5 describes new 6G radio access for flexible topologies and local communications. Chapter 6 describes the “beyond communications” services in 6G, such as sensing and computing. Chapter 7 describes the virtualization and cloud transformation. In Chapter 8, the status of the proof of concepts activities within WP3 are described. Chapter 9 summarizes and concludes the deliverable. Chapter10 contains the references and Chapter 11 provides an Annex with details from some of the enablers and studies. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 23 / 195 2 6G Architecture overview 2.1 WP3 objectives mapping to the E2E 6G system blueprint In [HEX223-D22], the Hexa-X-II view of the 6G E2E system blueprint was defined, see Figure 2-1. The blueprint consists of four layers: • Application, • Network-centric application, • Network functions, and • Infrastructure layers. In addition to this, is also shows the so called “Pervasive functionalities” which can reside in any of the four layers. Figure 2-1 will be used throughout this deliverable to indicate where different enablers are placed on the 6G End-to-End (E2E) system blueprint. In this section, we will make a similar assessment for the objectives listed in Table 1-1. The objective WPO3.1 “6G architecture for AI and beyond communications” includes data driven functionalities such as Machine Learning Operations (MLOps), Data Operations (DataOps) and exposure of AI services. It also contains the Beyond Communication Network (BCN) functions such as sensing and compute offloading and how to expose these services. As can be seen in Figure 2-1, this objective belongs to the Network function layer, and will have new functionality in the Radio Access Network (RAN) and in the Core Network (CN). The next objective WPO3.2 “Combine the cloud technology for a modular, scalable and extendable architecture” addresses how the RAN and CN NFs can communicate with each other in an efficient manner and the E2E placement of functions (including the devices in some cases). The objective should also address the cloud transformation and the so-called cloud continuum. As can be seen in Figure 2-1, this objective maps to both the infrastructure (in particular the cloud continuum) and the network function layers. The last objective is WPO3.3 “Architecture for flexible topologies”. This objective maps to the Network function layer (see Figure 2-1) and develops existing and new 6G subnetworks, including NTN, mesh networks and device networks. The objective also develops new multi-connectivity solutions. Figure 2-1 Illustrative mapping of WP3 objectives to the system blueprint Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 24 / 195 2.2 5G to 6G Migration 2.2.1 Introduction For the migration from 4G to 5G, the industry was concerned that the envisioned deployment of the 5G RAN could be delayed due to a delayed introduction of the 5GC. Consequently, 3GPP agreed to deliver an “early drop” of New Radio (NR, i.e., 5G RAN) in Rel-15 which supports only the so-called Non-Standalone (NSA). The NSA option meant that the UE could connect to a 5G RAN i.e., either connected to the 4G CN (i.e., Evolved Packet Core (EPC)) or to the 5GC. Further on, several different options of dual connectivity between 4G and 5G was also standardised [38.300], where 4G RAN and 5G RAN can be used in any combination connected to either EPC or 5GC. One of these options that also is implemented and deployed in the network is Evolved Universal Terrestrial Radio Access (E-UTRA) and New Radio (NR) Dual Connectivity (DC) (ENDC), where the 4G RAN is the master node and the 5G RAN the secondary node, and EPC is used as core network. During the remainder of release 15 (Rel-15), focus was shifted so that the complete Rel-15 specifications also specify SA. So far, several years after 5G’s launch, only a small fraction of networks has yet transitioned to the full SA 5G architecture, i.e., using both 5G RAN and 5GC.Therefore, to reduce the 5G-6G interworking and deployment complexity, the goal should be to have fewer architecture options specified for the deployment of 5GC as a 6G requirement. This would also avoid delays introducing key 6G services. As a result, the CN for 6G could be an evolution of the 5G CN so that the networks can gradually extend the support for new 6G services without the need to replace the CN. The 6G radio access should support a single-RAT architecture only, i.e., a 6G UE that connects via the 6G radio interface establishes a connection to the CN for 6G without any complex inter-RAT multi-connectivity. Interoperability with 5G and older standards could be managed via existing core network handover or reestablishment. The above considerations aid in enabling the 5G-6G migration with a focus on a selected set of key architecture options to be specified, that will later be developed and deployed. The 6G RAT will support operation on newly assigned 6G spectrum but also on all bands supported by 5G which then enables dynamic spectrum sharing with 5G. Thus, operators can use existing deployments both for 5G and 6G and dynamically share the spectrum resources between 5G and 6G as needed. The main architecture options evaluated for 6G migration (cf. Figure 2-2) are as follows. • Option 1: 6G RAN is connected to 5G RAN using DC. One of the objectives of 6G RAN is to enhance capacity, while in this option base capacity and coverage is provided by 5G. The core network is based on 5GC. Note that some updates to 5GC are still needed to support the introduction of 6G RAN as secondary RAT. However, option 1 is less likely to be selected for standardization since it requires 6G to support more than one RAT and will also slow-down the introduction of the new 6G services. • Option 2: 6G RAN is deployed standalone and connected to an Evolved 5GC (E-5GC) (see section 2.2.2). 6G intra-RAT multi-connectivity using an enhanced carrier aggregation (CA) and/or 6G-6G DC (see more in section 5.2) can be used to combine capacity and coverage bands that are dynamically shared with 5G via Multi-Radio Spectrum Sharing (MRSS) [HEX223-D43]. • Option 3: 6G RAN is deployed in stand-alone mode with a 6G Core (6GC) (cf. Section 2.2.2), which allows for non-backward compatible changes such as a completely new RAN-Core or NAS interface or significant refactoring between RAN and Core functions. Depending on the amount of refactoring, network functions may be alike (e.g., 5G and 6G AMF), while the 6GC would still have many commonalities with the 5GC such as the Service Based Architecture (SBA) framework or shared network functions like the Unified Data Management (UDM), PCF, Network Repository Function (NRF), etc., can exist. Mobility/inter-working between 5GS and 6G System (6GS) is core-based and realized via inter-RAT handover. Note that although the different general architecture options are presented here for the sake of completeness, a decreased set of options considering the 6G architecture E2E blueprint is presented in [HEX223-D22]. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 25 / 195 Figure 2-2 General architecture options for 6G. 2.2.2 Evolved 5GC and 6GC The E-5GC is built upon 5GC NFs which can be enhanced to handle 6G requirements. Moreover, new 6G NFs can be defined within E-5GC when justified. The exact list of the shared/dedicated functions is subject to further study, e.g., use of evolved or shared 5G Network Functions (NF) and definition of new 6G NF(s). The E-5GC serves both legacy 5G RAN and new 6G RAN while supporting the 5G and 6G features at the same time (with the latter requiring 6G UEs and 6G RAN to be used). In the design, 5G SBA framework is continued to be used in both the E-5GC and the 6GC. 5GC and 6GC will have shared network functions such as UDM, NRF and User Plane Function (UPF). Additionally, new network functions or new services for existing 5G NFs (i.e., evolved 5G NFs or 6G NFs) can be added to support new 6G use cases and applications. Figure 2-3 depicts how this extensibility can be utilized for an evolution of the 5GC to introduce 6G functionalities in a backward compatible manner. Figure 2-3 Evolved 5G core architecture (Option 2) New 6G functionality is introduced by adding new NFs (e.g., NFx in Figure 2-3) or by enhancing 5GC NFs with new functionality, i.e., “shared NFs” (illustrated as blue triangles). The new 6G NFs are accessed only by new 6G UEs, 6G RAN, or new 6G services, while new functions / services can access and reuse the services provided by 5G network functions (e.g., UDM, NRF) through the common SBA framework. However, larger refactoring of functional splits (e.g., changing RAN-Core split) between network functions would render E5GC infeasible and break backward compatibility. In this case, a clear separation between 5GC and 6GC should be preferred as depicted in Figure 2-4 (Option 3), where only few NFs would be shared. Regardless of the migration option, the granularity at which core components interact with each other will continue to stay at NF level definition. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 32 / 195 Figure 3-3 AI-Native architecture for efficient AI/ML model orchestration Figure 3-3 shows the AI-native architecture and components to support MLOps CLs for AI/ML model orchestration: - AI/ML model training: This function fetches data from the repository function and/or the data collection function to create the input dataset, then performs AI/ML model training and validation. Optionally, if the new model training relies on another model (e.g., in case of Transfer Learning or Continual Learning), the function retrieves the base model from the repository. This function may also use a sandbox environment including Digital Twins for safe AI/ML model training and validation. - AI/ML model inference agent: This function performs inference using the trained model retrieved from the model repository. The agent also provides model performance reports to the model monitoring function. - AI/ML model monitoring: This function is responsible for collecting and aggregating different performance reports from the AI/ML model inference agents and determining the model’s average accuracy across agents. The function sends alerts to the AI/ML orchestration function when model performance drops below a certain threshold. - Data collection function: This function performs data collection and pre-processing prior to storage. It interfaces with different data sources to collect data on demand, or periodically to maintain data accuracy and freshness (e.g., for keeping Digital Twins up to date). - Data and model repository: This repository serves to store and provide AI/ML models, Digital Twins, datasets and their related metadata, such as accuracy and dependency relationships. - AI/ML model orchestration: This function is the main decision component of the AI-Native framework. Similar to a service orchestrator, it manages the lifecycle of the model and its different versions by collecting monitoring data, making decisions and communicating with the other functions to trigger different operations with the aim of maintaining model performance in an automated manner. 3.1.3 Evaluation The set of principles outlined for the 6G system in [HEX223-D21] serves as a comprehensive framework that intricately intertwines with the architectural means and protocols. These principles, ranging from support for advanced digital services and full automation to flexibility in various network scenarios, scalability, resilience, and considerations for security, privacy, and sustainability, lay the foundation for an innovative and adaptive architecture. The relationship between these principles and architectural means and protocols becomes evident in the way they shape the fundamental design decisions. Each principle addresses specific aspects of network functionality, service management, security, and environmental impact, providing a holistic approach that influences the protocols, interfaces, and overall architectural structure. This synergy ensures that the resulting 6G architecture not only meets the requirements of advanced communication services but also aligns with principles of automation, flexibility, resilience, and sustainability. Cooperative learning, with its demand for the classification of data with varying privacy sensitivity levels, represents a paradigm shift in the landscape of wireless communication. Currently, cooperative learning operates primarily on the application level, lacking a tailored network architecture, protocols, and procedures Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 33 / 195 that cater specifically to stringent requirements of cooperative learning on privacy/security and data accuracy. Section 3.1.2.1 has the following impacts on the design principles: • Principle 1, Support and exposure of 6G services and capabilities: Section 3.1.2.1 supports efficient incorporation of 6G services, such as computational offloading, when network nodes with low capability, having to perform a certain task, need to offload data and/or computation to more capable collaborative node. • Principle 6, Persistent security, and privacy: The proposed collaborative learning model supports a persistent security and privacy principle. The proposed framework and workflows in Section 3.1.2.2 not only align with the principles but also play a crucial role in automating ML lifecycle management. Moreover, it brings forth efficiencies by allowing early prediction, detection, and prevention of ML model performance degradation, emphasizing the symbiotic relationship between the overarching principles and the intricate details of architectural means and protocols. The study has the following impacts to the principles: • Principle 1, Support and exposure for 6G services and capabilities: The solution developed within Section 3.1.2.2 will enhance support for future 6G services and capabilities by providing and maintaining AI/ML models that can be used by the 6G system (e.g., Zero-Touch network and service orchestration) and the services deployed on it. • Principle 2, Full automation and optimization: The output of Section 3.1.2.2 will support full automation and optimization by providing and maintaining optimized AI/ML models that support 6G system management automation. • Principle 4, Network scalability: The AI-native architecture will support network scalability by training, updating, or re-training AI/ML models and Digital Twins, and deploying new model instances as needed, in a dynamic and automated manner. • Principle 5, Resilience and availability: The trained AI/ML models and Digital Twins can also be used to predict or detect events that might affect the system’s availability or performance. The trained AI/ML models and Digital Twins also provide decisions for preventing or mitigating those events, and for the overall optimization of the system and its relevant KPIs. 3.1.4 Summary The shift to cooperative learning in 6G networks necessitates novel architectural frameworks to enable effective capability sharing among UEs and gNBs. This includes the development of control structures, discovery methods, and signalling mechanisms. Integral to this transition is the implementation of privacypreserving measures to address varying sensitivity levels. Proposed AI-native architectures integrate AIaaS functions, emphasizing protocols for data exchange and management. Key components such as programmable network monitoring, DataOps, MLOps, and Cooperative Learning governance are crucial for synchronization and performance optimization. Ensuring reliability within the cooperative learning environment demands regular data collection and updates. Data-driven architectural means and protocols are instrumental in integrating immersive reality technologies [HEX223-D12] seamlessly, optimizing network performance, prioritizing data traffic, and dynamically allocating resources. They ensure security, privacy, and enhance user experiences while enabling predictive maintenance. Additionally, they support autonomous robots [HEX223-D12] by facilitating local ad hoc connectivity, efficient resource allocation, and collaborative decision-making. These protocols also enable the implementation of digital twins [HEX223-D12] across industries, ensuring real-time data ingestion and executing specialized algorithms. In the context of 6G networks, they prioritize privacy protection, establish trusted environments, and enable precision healthcare, ensuring safety and reliability in critical scenarios. Moreover, the mapping of this enabler to the 6G system blueprint is illustrated in Figure 3-1. Table 3-1 summarizes the main benefits and implications of the data-driven architectural means and protocols enabler. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 34 / 195 Table 3-1: Data driven Architectural means and Protocols enabler Description The structure, design, and principles governing AI component interaction and data handling. Benefits KPI improvement Interoperability ensures seamless communication among architecture components, integrating diverse systems effectively for efficient data flow. Scalability accommodates data growth and processing demands, allowing expansion without performance loss. API Response Time directly affects user experience and process efficiency, ensuring quick data processing for timely decision-making. Design principles [HEX223-D21] Principles #1, #2, #4 and #5 Dependencies / Basis for another enabler MLOps, DataOps, and AIaaS depend on architectural means and protocols for seamless integration and efficient operation. Architectural means provide structure, while protocols establish communication standards, ensuring compatibility and scalability. Implications Requirements Requirements for a data-driven architecture within an architectural means and protocols perspective include standardized protocols for integration, scalability, robust security, metadata management, and interoperability. Additional requirements cover version control, monitoring, as well as protocols for cross-functional collaboration, API design, and adaptability to emerging technologies. Standard relations & regulations 3GPP TR 23.700-82 [23.700-82], TS 23.288 [23.288], TR 38.817-01 [38.817-01] and TR 38.817-01 [38.817-02] 3.2 MLOps 3.2.1 Introduction The arrival of 6G is aims to revolutionize connectivity and data processing. This next frontier in wireless communication holds the promise of unparalleled speed, ultra-low latency, and a diverse range of applications. At the core of this transformative shift lies the integration of MLOps within the 6G framework, a strategic union that reshapes how ML is developed, deployed, and managed. MLOps, a comprehensive set of tools, addresses the ML development lifecycle, including data preparation, model training, deployment, and monitoring. The MLOps in the 6G landscape becomes particularly crucial due to the inherent distribution of datasets and the need to train ML models where data is collected, to uphold privacy considerations. This distribution mandates the efficient management of distributed AI functions across the telecommunications system, a challenge that MLOps tackles by minimizing communication, computation, storage, and energy costs during data collection, training, and inference. The collaborative and decentralized nature of MLOps algorithms facilitates model training and inference across various network functions while preserving privacy. The proposed MLOps solution within the 6G framework brings forth several notable benefits. Firstly, it introduces Input Data Compression (IDC) through a split neural network, which efficiently compresses input datasets with multiple attributes into a reduced dimensionality, reducing communication overhead during the transmission of model outputs or activations. Additionally, the MLOps enables Cross-Network Function Distributed (CNFD) ML model training and inference, allowing the fusion of output from ML models trained on different network domains, such as RAN, core network, or applications, while safeguarding privacy. Furthermore, the architecture facilitates Distributed ML Model Generalization by deploying multiple output neural network models that simultaneously serve different tasks, thus reducing the number of models to maintain and train, along with associated memory and compute requirements. The solution also supports Domain Adaptation, making the generalization node agnostic to different data distributions, further contributing to the reduction of required ML models. Lastly, the integration enables Distributed ML Model Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 35 / 195 Layer Offloading, allowing for dynamic adjustments in the computation environment by seamlessly offloading certain neural network model layers between computation nodes during training or inference. These benefits collectively preserve user privacy through private Federated Learning (FL) and enable distributed data-driven network decisions by learning from UE, RAN, core network, and application data within the 6G ecosystem. 3.2.2 Architectural Implications The architecture experiences a paradigm shift as MLOps seamlessly integrates into the existing infrastructure, potentially requiring the introduction of novel network functions rather than modifications. The orchestration of intricate ML models, especially in a distributed setting, becomes a central challenge, demanding adaptability to dynamic changes in compute, memory, and energy capacities. Continuous monitoring and feedback loops are imperative to ensure the freshness of datasets and trained models across a network of distributed nodes. Furthermore, the integration of privacy-preserving architectural components becomes crucial, requiring additional considerations for the management of privacy-sensitive data exchange between nodes. MLOps not only enhances the robustness and efficiency of ML systems but also poses architectural challenges that mandate a holistic and adaptive approach to accommodate the evolving landscape of AI. In the context of distributed ML settings, establishing communication links between computation nodes is imperative, especially considering the iterative nature of information exchange during multiple rounds of model parameter exchange. This ongoing exchange of information is vital not only during training but also in the collaborative inference phase of split neural networks. Unlike the conventional centralized ML model training, where decentralized nodes acted as data sources, in distributed ML training and inference decentralized compute nodes take on the additional roles of ML model trainers and aggregators. This shift necessitates effective orchestration to dynamically adapt, place, and offload ML tasks based on the changing capacity of the execution environment, considering factors such as compute, memory, energy, and communication. The orchestration process must optimize various KPIs, including model efficacy, training time, and resource utilization. The integration of privacy-preserving architectural components within the network introduces modifications or novel network functions to facilitate privacy-preserving data collection and learning. This adds complexity to the network architecture, requiring thoughtful consideration of privacy implications and the incorporation of mechanisms to ensure secure and private handling of sensitive data. Additionally, the deployment of intricate ML models in a distributed manner poses challenges, necessitating continuous monitoring and feedback loops to ensure the freshness and reproducibility of diverse datasets. As the network scales with an increasing number of distributed nodes engaged in ML model training, robust infrastructure becomes essential. Integrating MLOps practices further complicates the scenario, demanding compatibility with existing systems and careful management to streamline the deployment and management of distributed ML models. Overall, the implications from the MLOps perspective underscore the need for sophisticated orchestration, privacy preservation measures, and robust infrastructure to navigate the complexities of distributed ML in the evolving network landscape. MLOps plays a pivotal role, acting as an enabler for both AIaaS and Intent-based Management. Within MLOps, tools and processes are designed and employed to manage the entire lifecycle of AI services, translating user intents into actionable models and services. This emphasizes the significance of MLOps in operationalizing and optimizing AI services throughout their lifecycle. Moreover, the synergy between MLOps and DataOps becomes apparent, forming a crucial dependency to efficiently manage ML models and the diverse datasets used for training. The integration of MLOps and DataOps practices ensures a harmonized approach to handling data and models, contributing to the seamless execution of privacy-aware and collaborative ML initiatives within the network architecture. KPIs and KVIs play a pivotal role in assessing the effectiveness and efficiency of MLOps within the context of distributed ML, particularly in the split neural network setting. These metrics provide a comprehensive view of the operational aspects and impact of MLOps tools and mechanisms. The identified KPIs and KVIs include robustness, flexibility, communication cost, computation and memory cost, fault tolerance, model security, ML model efficacy, improved privacy/security, data accuracy, power consumption and resource utilization. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 36 / 195 3.2.2.1 Distributed Model Training and Inference This section presents a distributed AI-enabled technology called split learning on different scenarios. The research questions addressed in this section are related to: • cross-network function ML: how to perform training and inference on a distributed ML model consisting of input data of different attributes from different network elements such as RAN, core network, or from different network functions, without moving original local datasets from where they are located? • model generalization with joint training: how to train minimal number of ML models with minimal computation and memory requirements that serve different use cases and tasks simultaneously? • model generalization with domain adaptation: how to train an ML model for multiple data domains, i.e., datasets with different distributions, simultaneously? (Note on terminology: the well-known domain adaptation in ML (e.g., data distribution) should not be confused with the same terminology used in communication networks (e.g., RAN, core network), and here we refer to ML terminology). • model layer offloading: how much reduction in CPU utilization and memory allocation be achieved by offloading model layers in between split learning compute nodes? The distributed ML in this section consists of three main types of nodes: input node, generalization node, and output node. The global Neural Network (NN) model is split into these node types, and every node type contains portion of the global NN model. A node type in this demo is defined as the group of at least one neural network model. The neural network models that are deployed at the input and output of the global neural network are called input and output nodes, respectively. The input node contains the input dataset, i.e., ML features; and the output node contains the target labels. The intermediate NN model is deployed in the generalization node. The nodes communicate in full synchronization. These nodes are illustrated in Figure 3-4. Figure 3-4 Illustration of a split learning setting. Input node consists of three entities that each contains input data and compute capability. The three input entities are Physical, MAC, and IP, and each entity has one local NN model that trains on the local input data. The generalization node does not have access to raw dataset; it is rather an intermediate entity with a NN model (serving as a compute element) with the goal of taking high compute tasks related to the training of a split learning model. It also serves with the goal of generalizing the intermediate learned representations from input nodes, so called encodings or activations, to multiple use cases deployed in the output node. The output node consists of two NN models serving for two use case tasks, i.e., video bitrate estimation and video delay estimation. The Physical and MAC entities can be co-located in gNB; the IP entity and the generalization node can be co-located in UPF; and the output consumer entities that contains video bitrate and video delay data can be co-located in output consumer such as external application or some other external entity. More detailed description on the proof of concept (PoC #B.2) and results are presented in Chapter 8.1. 3.2.2.2 Privacy-aware data collection and learning Private federated learning and private data aggregation enable privacy-preserving learning on user data. Privacy is preserved by combining the local noise addition with secure aggregation techniques in such a way that correct statistical information of data can be still learned without revealing personally identifiable information. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 37 / 195 Being put in the future cellular context, aggregated data originated from several UEs is collected from a defined sampling area and over a defined sampling time window. The data shared with the network does not capture individual private data, but it could consist of statistical (mean, variance, distribution, etc.) and/or analytical insights (embedding, graph, correlations, etc.) of the users in the network. Some examples of such data could be a capture of cellular metrics (e.g., signal strength), Quality of Experience (QoE) metrics, location, motion information, etc. The granularity of this sampling could be agreed between UE vendor and network vendor such that it respects the user privacy and serves the intended utility. If the network needs private individual data of a UE to infer the best config/parameters, e.g., location for handover, it can share the computation model (inference model) to the UE who can run the model to decide which configuration the NW should use. Such inference models can be trained by private federated learning that preserve user privacy. Privacy preserving architecture proposed in D3.2 introduced new architectural elements such as UE aggregation unit (performs privacy-preserving data aggregation by leveraging secure aggregation techniques and/or data anonymization to collect statistical information of UE data) and data-driven network control unit, (an entity at the network side that configures the base station by performing automation, optimization, and intelligence services, based on both network and UE data). This architecture is based on privacy-preserving cryptographic protocols like Prio [CB17], depicted in Figure 3-5, currently being standardized in IETF Privacy Preserving Measurement Workgroup [GPR+23]. It consists of a Collector, an entity that obtains the aggregated UE data, and Aggregators (Leader and Helper), which are responsible for UE data collection. This approach is already efficiently implemented in Apple’s ecosystem [TWM+23] [CCC+23]. Leader (main) is an aggregator responsible for coordinating the data aggregation. It receives the encrypted UE data and orchestrates the process of computing the aggregate result as requested by the Collector. Helper is an aggregator assisting the Leader with the computation. Privacy is preserved under the assumption that the Leader and the Helper do not collide, i.e., they do not contain the same data shares. The Leader and the Helper send aggregated shares to the Collector that computes the final aggregation result. Figure 3-5 Prio-based aggregation system architecture. The physical realization of the different Prio entities in the cellular network architecture can have different variants. The Leader and Helper can be realized as a Network Data Analytics Function (NWDAF) service or reside in Data Network (DN), while Collector can reside in DN to preserve UE ownership of the data. An example of potential privacy preserving architecture in 6G network is shown in Figure 3-6. The Leader is implemented in NWDAF, thus leveraging mostly CN resources for aggregating encrypted UE data shares, while Helper resides in DN, assisting the Leader. Application function can be in core network (AF1), e.g., for network optimization/OAM, or in data network (AF2), e.g., for analytics. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 38 / 195 Figure 3-6 An example of Privacy preserving architecture in 6G network. 3.2.2.3 Incentive mechanism design for wireless federated learning networks Consider a wireless Hierarchical Federated Learning (HFL) network where edge servers facilitate the aggregation and transmission of end user’' model parameters to a server in the cloud. It is evident that the performance of the HFL procedure mainly depends on the end users’ discretion to invest their resources (e.g., battery, communication, computing resources), and that is when the problem of providing sufficient incentives arises. Several incentive mechanisms have been proposed so far for the typical FL setting, based on game, auction, contract, and matching theories [TZL+22]. Depending on the objective pursued (e.g., latency requirement, model accuracy, transmission feasibility), different commodities are traded between the FL network parties, such as Central Processing Unit (CPU) resource, power resource, or size of end users’ dataset used. Especially, considering an HFL network, where multiple edge servers exist, then the potential that these servers belong to different providers makes the problem of incentive provisioning even more challenging. Apparently, inherent competition arises between them in how to attract end users that yield uniform data distributions under budget constraints, where the budget may regard resources or monetary rewards offered by the servers to the end users. The term uniform refers to Independent and Identically Distributed (IID) data distribution, which is crucial for maintaining statistical consistency across the edge servers that, in turn, minimizes bias and variance, thereby enhancing the convergence and generalization performance of the federated learning system. The allocation problem can be modelled as a Fisher market [BCD+14], where a set of buyers (i.e., edge servers) aims to purchase multiple goods in the sense of employing end users to maximize its utility under constrained budget availability. In a Fisher market, the prices and allocation of goods are derived to clear the market, meaning that the goods are optimally allocated to maximize the buyer’' utility and fairly priced such that the corresponding sellers would not deviate from the resulting allocation. The latter equilibrium point is referred to as Market Equilibrium (ME). Motivated by the notably limited literature on the joint client association and incentive provisioning problem in HFL networks, a price-driven client association based on the Fisher market modelling is proposed. The proposed framework manages to model and account for the competitive behaviour of both the edge servers and the end users when performing user-to-edge-server association and pricing, respectively. In this context, the edge servers play the role of the buyers that invest a budget in the form of monetary rewards to motivate end users to participate in the HFL process. The user-to-edge-server association and the pricing are determined based on the ME, at which point the edge servers attain maximum utility from associating with end users that yield balanced data distributions to address the challenging issue of non-IID data. The end users, in turn, are priced fairly so that no other association would be more beneficial for them. 3.2.2.4 Federated learning approach between different city verticals While FL addresses some issues regarding data privacy by removing the need to transfer raw data to the central server, it does not make it impervious to attacks on the privacy or security of data. Malicious devices on the edge network can not only intercept the model uploaded by the central server and use that information to infer raw data on other device’' uploaded models, but they can also upload data poisonous to the global model aggregation. Some solutions to this are vulnerability detection of edge devices (using Generative Adversarial Networks to protect model parameters, weight-based anomaly detection against poisoning attacks and privacypreserving data, data filtering in dealing with poisoning attacks), creating a trust system to label the edge Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 39 / 195 devices on their performance, protecting against backdoor attacks introduced by data points with unusual features, among others. Figure 3-7: Collaborative edge-computing model In vertical FL, which is our case, the data are complementary; parking and traffic data, for example, are combined to predict someone’s route preferences. Finally, in federated transfer learning, a pre-trained foundation model designed to perform one task, like detecting parking availability, is trained on another dataset to do something else, like identify routes with minor traffic. The Collaborative edge-computing model, see Figure 3-7, is using the edge nodes for local learning, will address the issues of data privacy and resilience. All data concerning city A will remain locally. What is updated on the central node are the algorithms used without all the private information, and therefore without the concerns of what information types and privacy, and without all the amount of data locally present on each edge. These will be managed by a centralised Machine Learning Catalogue, shared by all, and all these communications will be encrypted in transit. Regarding network requirements needed to ensure communications, Heterogeneous communication systems that can support various innovative wireless technologies, services and applications are required. To do so, the 6G Radio Access Network is deployed standalone. This solution will ensure shared communication between the nodes, enabling the exchange of information in a secure and reliable medium. All these services must be integrated and standardised to achieve a global service, with model aggregation, directly fed from all the nodes after all the local modelling and training. The solution will present AIaaS, due to this architecture that will provide an edge to the cloud continuum with high availability and resilience. FL and Edge Computing integration become critical in the context of 6G. The real-time communication needs of FL setups are well-suited to the ultra-low latency and high bandwidth capabilities of 6G networks. Nearer Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 40 / 195 to the end users, edge devices allow for collaborative model learning and improvement without sacrificing data privacy. 3.2.3 Evaluation By embracing the principles of full automation and optimization, flexibility to diverse scenarios, and persistent security and privacy, MLOps becomes a strategic enabler. It ensures that AI models seamlessly adapt to dynamic network conditions, address diverse use cases, and uphold stringent security and privacy standards. The relationship between these principles and MLOps reflects a collaborative effort to infuse intelligence and adaptability into the 6G system, harnessing the potential of ML within a framework defined by overarching principles for a robust and responsive wireless communication network. The distributed nature of ML models, deployable across nodes with computational capabilities, introduces considerations such as the potential need for additional secure hardware. Furthermore, ML model training may necessitate increased bandwidth to share parameters without degrading existing communication link quality. The integration of privacy-preserving architectural elements, driven by MLOps, mandates corresponding changes in architecture and novel signalling protocols to accommodate varied data privacy levels. On the positive side, MLOps significantly enhances the efficiency of the ML lifecycle, streamlining, and automating various processes. This not only accelerates model development, training, and deployment but also underscores the importance of version control and reproducibility, ensuring heightened reliability of ML models. The mention of a split 6G data architecture reflects adherence to Principle 3, which emphasizes the design's adaptability to various network scenarios, including subnetworks and non-public networks, ensuring optimal performance across diverse environments. The utilization of split neural networks for distributed ML resonates with Principle 6, which highlights the importance of persistent security and privacy. This is achieved by addressing challenges such as data protection regulations and bandwidth limitations, showcasing a commitment to safeguarding user data. The integration of privacy-aware data collection and a privacy-preserving architecture in support of security and privacy aligns with Principle 10, which underscores the aim of minimizing the environmental footprint and promoting sustainable networks. This principle emphasizes the need for justifying any increased environmental footprint with added value, cost efficiency, and societal benefits. The acknowledgment of distributed data-driven network decisions contributing to sustainable development aligns with the broader goal of environmental consciousness, reinforcing the importance of responsible and eco-friendly technological advancements. Lastly, the emphasis on collaborative and decentralized model training through MLOps corresponds to the overarching objective of optimizing and automating network operations, aligning with the principles of full automation and optimization (Principle 2). This principle underscores the importance of utilizing distributed AI/ML agents to manage and optimize the system without human interaction, promoting efficiency and autonomy in network and service management operations. 3.2.3.1 Wireless hierarchical federated learning: On the accuracy-energy trade-off HFL is an extension of FL that introduces a hierarchical structure to the model aggregation process. HFL suggests adding an extra layer of edge model aggregation where edge servers facilitate the aggregation and transmission of end user’' model parameters to a server in the cloud. In this way, better resource utilization across the network can be achieved while reducing backhaul network traffic and thus the required convergence time of the FL procedure and the consumed energy at the end-user devices. Nevertheless, an inconvenient assignment of users to the different edge servers can cause an unequal distribution of data among the edge servers and an imbalanced traffic load on the RAN, bringing exactly opposite effects in the HFL procedure. To reap the maximum benefits of such a hierarchical structure, several research works, e.g., [MAM+22], [LYC+22], [LCW+20], [ZLH23], are devoted to the appropriate user association to the available edge servers and the allocation of the radio resources. Consider a wireless HFL network as the one considered in Section 3.2.2.3. The users associated with the same edge server transmit their local models over the same time and frequency resources by multiplexing through the power-domain Non-Orthogonal Multiple Access (NOMA) technique. The joint problem of user association Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 41 / 195 and uplink transmission power allocation is formulated as a non-cooperative game in satisfaction form [PTL+10] to achieve a trade-off between training model accuracy and energy consumption for the users in a distributed manner. The solution concept pursued is the so-called Satisfaction Equilibrium (SE) point, where each end user’s minimum acceptable trade-off value is satisfied. The SE point is determined by a Reinforcement Learning (RL) algorithm. The users act as agents that explore their action space, i.e., possible associations and transmission power levels, by observing their achieved trade-off value provided as feedback from the cloud server. The details regarding the system model, problem formulation, algorithm design, and performance evaluation can be found in [CDP23]. The proposed SE solution concept is evaluated by comparison against (i) the "Random States", according to which each user randomly selects its associated edge server and transmission power from a feasible set of values, and (ii) the "Satisfaction Game - Energy" that refers to a satisfaction game where the users aim to solely reach a maximum energy consumption threshold without accounting for the achieved training model accuracy. Figure 3-8 Convergence behaviour and test accuracy of the global HFL model Figure 3-9 Total UE energy consumption and network’s entropy In Figure 3-8 the variation of the global model's accuracy achieved over the test set is presented as a function of the cloud iterations, i.e., epochs, under the proposed and the two comparative approaches. The proposed framework outperforms the two comparative approaches that do not explicitly consider the performance of the learning process striking to achieve a uniform distribution of the data to the different edge servers in the HFL network via proper association. Figure 3-9 depicts the total energy consumed by the users (left axis) along with the total information entropy achieved in the system (right axis) after the convergence of the proposed and the two comparative approaches. The information entropy expresses the users’ data distribution among the different edge servers. The highest the total entropy is, the closer the network is to the IID case that results in high training model accuracy. It is shown that the proposed framework provides low energy consumption for the users and the highest information entropy. Figure 3-10 illustrates the total energy consumed by the users for communication along with the total entropy achieved in the HFL network, under different minimum trade-off values. It is observed that as the pursued trade-off value increases and the user’' requirements get stricter, the total user’' energy decreases, whereas the network's entropy increases. The data are more efficiently distributed to the different edge servers while maintaining interference levels low. Figure 3-11 shows that the presence of more users results in higher energy consumption by each of them due to increased interference levels. On the contrary, the higher number of users provokes an increment in the network's entropy, since the additional users joining the HFL network bring a wider variety of samples, allowing for better distribution of the dataset’s classes among the edge servers. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 48 / 195 3.3.4 Summary AIaaS emerges as a crucial enabler for advanced capabilities, driven by user-friendly APIs and customizable models. The framework targets distributed ML techniques, managed through Runtime and Training Managers. Integration with DataOps and MLOps ensures efficient data management and lifecycle support. Evaluation metrics encompass various KPIs and KVIs, emphasizing performance, sustainability, and trustworthiness. Architectural implications prioritize flexibility and scalability, aiming for seamless integration and optimal performance in 6G networks. AIaaS enhances various technological applications. For autonomous robots, it provides capabilities like computer vision, task planning, natural language processing, anomaly detection, and adaptive learning, ensuring effective navigation and task performance in dynamic environments. In the realm of smart transportation, AIaaS improves vehicle capabilities for object detection, navigation assistance, and contextaware communication, enhancing safety and efficiency in urban areas. Additionally, it strengthens digital twin applications by offering data processing, monitoring, predictive maintenance, and simulation, improving decision-making and efficiency across industries. Furthermore, AIaaS supports human-centric 6G services by providing privacy protection, security enhancement, personalized healthcare, and safety monitoring, ensuring compliance with regulations and enhancing trust among users. Moreover, the mapping of this enabler to the 6G system blueprint is illustrated in Figure 3-1. Table 3-3 summarizes the main benefits and implications of the data-driven architectural means and protocols enabler. Table 3-3: AIaaS Enabler Description AIaaS represents a paradigm shift where the mobile network becomes a catalyst for innovative use cases by providing accessible AI capabilities through a variety of APIs, thereby eliminating the need for application developers to build and manage their AI infrastructure. Benefits KPI improvement From an AIaaS perspective, the most critical KPIs are model efficacy, scalability, and API availability/reliability. These metrics ensure accurate model performance, the ability to handle growing demands, and seamless access to AI services, respectively. Design principles [HEX223-D21] Principles #1, #2, #3, #4 and #8 Dependencies / Basis for another enabler DataOps, architectural means and protocols, and MLOps rely on AIaaS for seamless integration and operation within the architecture. AIaaS provides access to AI capabilities without the need for infrastructure management, enabling these components to leverage advanced AI functionalities. This integration allows DataOps to streamline data workflows by incorporating AI capabilities for enhanced data processing and analysis. Architectural means and protocols can be designed to seamlessly integrate AIaaS, aligning communication protocols and data exchange formats to accommodate AI-driven processes efficiently. Additionally, MLOps can utilize AIaaS for model deployment, monitoring, and optimization, ensuring the operational efficiency and effectiveness of ML models within the architecture. Implications Requirements Key requirements for a data-driven architecture within the MLOps perspective include ensuring high-quality data, implementing scalable storage, and processing, and facilitating data versioning. Automation through CI/CD pipelines and robust monitoring is crucial, along with a focus on model explainability and interpretability. Security measures and compliance with regulations, feedback loops for continuous improvement and resource optimization are essential. Standard relations & regulations Standards: ETSI GS ZSM 012 [ZSM012], 3GPP TR 23.700-82 [23.70082] Regulations: EU AI Act Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 49 / 195 3.4 DataOps 3.4.1 Introduction 6G and DataOps represent cutting-edge technologies poised to shape the future of communication and data management. Expected to be fully operational by the end of the decade, 6G could revolutionize industries supporting technologies like advanced AI applications. With potential speeds reaching terabits per second, 6G aims to create a digital landscape that enables immersive experiences and seamless connectivity. On the other hand, DataOps is a collaborative and agile approach to data management. It emphasizes the streamlined management of data throughout its lifecycle, from acquisition to analysis. By integrating DevOps practices into the data domain, DataOps fosters automation, continuous integration, and collaboration among data professionals. This methodology enhances the speed and accuracy of data analytics, reducing the time it takes to derive insights and ensuring a smooth flow of information across different departments. The convergence of 6G and DataOps holds promise, as the high-speed, low-latency capabilities of 6G can enhance real-time data exchange, offering transformative possibilities for innovation, efficiency, and data-driven decision-making. The synergy between DataOps and 6G brings along with several benefits. Efficient data utilization is facilitated allowing the aggregation of insights from distributed devices without the necessity of transferring the complete dataset to a central entity, which proves advantageous for 6G networks, designed to manage extensive data, as it significantly reduces network traffic and bandwidth usage. Furthermore, empowering edge devices and endusers to actively participate in the learning process fosters distributed intelligence. This not only promotes a collaborative and decentralized learning environment but also enables real-time decision-making at the edge of the network. Such capabilities align with the evolving needs of advanced technologies like 6G, where the ability to process and act upon data locally is increasingly crucial. The focus is set on understanding the distributed intelligent management solutions, a federated approach facilitates seamless collaboration between data-producing components and systems within a management framework. These components deliver either cleansed data or locally trained AI/ML models to a centralized management entity through standardized interfaces. This approach prioritizes efficient data sharing and model training while safeguarding sensitive information and enables secure data sharing only when necessary or at specified intervals, maintaining the integrity of sensitive information. 3.4.2 Architectural Implications The correct functioning of AI is related to data quality. Data shall be delivered, pre-processed, and stored where and when required. This imposes requirements on a flexible data ingestion architecture, which can refer to DataOps. DataOps serves as a pivotal force orchestrating efficiency and collaboration. Its central role involves attention to data quality assurance, employing processes to cleanse and validate data, ensuring the precision essential for effective AI model training. DataOps aligns the efforts fostering seamless communication and a shared vision within AI initiatives. Through adept orchestration of automation, continuous integration, and monitoring, DataOps accelerates the AI development lifecycle while incorporating scalability, flexibility, and robust security and compliance measures. From a DataOps perspective, in Section 3.4.2.1 critical KPIs and KVIs guide the assessment and optimization. Model Convergence measures the efficiency of the learning process, ensuring models reach stable and accurate states. Network Efficiency evaluates data transfer effectiveness, addressing latency, congestion, and bandwidth usage. Model Inference assesses real-time decision-making speed and accuracy. Data Ingestion metrics gauge the efficiency of incorporating diverse data sources. User Engagement and Satisfaction KPIs provide insights into end-user experiences. Model Reusability ensures adaptability across contexts, maximizing the utility of ML models. These metrics collectively drive continuous improvement within the DataOps framework, creating an efficient and responsive data ecosystem. Additionally, the capability to leverage diverse data sources is demonstrated in Section 3.4.2.1. This facilitates the development of personalized ML models tailored to individual preferences, contributing to enhanced user experiences, targeted services, and improved decision-making across diverse network domains. The scope of Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 50 / 195 ML applications is expanded, allowing its application to various industries without the necessity of sharing data with operators. This not only ensures data privacy but also unlocks the potential for new, data-driven insights and decision-making capabilities within specific verticals and among application owners. The principles of efficient and secure data management in DataOps environments are aligned with by these features mentioned in Section 3.4.2.1. 3.4.2.1 E2E 6G Network Slice Instance Employing Distrusted Intelligence Solutions In distributed intelligent management solutions, data-producing components and systems within a management framework seamlessly deliver either cleansed data or trained local AI/ML models to a centralized management entity via standardized interfaces. This federated approach ensures efficient data sharing and model training while protecting sensitive information. The centralized entity, as depicted in Figure 3-14, plays a critical role in aggregating individual local models or training cleansed data to construct a comprehensive global model for a specific optimization function. This global model leverages the collective learnings from local models and cleansed data, enhancing the overall performance of the management framework. Figure 3-14 Enabling Efficient Data Sharing and Model Training Across Management Systems Local models, also referred to as domain-level models in this article, are trained within specific management domains using the data or network data stored on those domains. This approach maintains data privacy and security by keeping confidential information within the respective management domains, allowing for secure data sharing only when necessary or at specified intervals. The global model, trained on cleansed data or trained local models stored in the central repository of the centralized management entity, encapsulates the collective knowledge from all management systems. It functions as a coordinator, periodically aggregating updates from local models and incorporating them into its own parameters, ensuring continuous improvement and optimization across the management framework. 3.4.3 Evaluation From a DataOps perspective, a comprehensive approach in Section 3.4.2.1 is highlighted, showcasing the leveraging of diverse data sources, the facilitation of personalized machine learning models, and the enabling of efficient data utilization. DataOps principles are adhered to, emphasizing the aggregation of insights from distributed devices to streamline data operations. The empowerment of edge devices and end-users in the Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 51 / 195 learning process resonates with the decentralized nature of DataOps, fostering distributed intelligence and facilitating real-time decision-making at the network’s edge. In the context of 6G, the approach in Section 3.4.2.1 stands out for its potential to handle vast amounts of data, resulting in the reduction of network traffic and bandwidth usage. The capability to apply the approach across a wide range of applications and industries without the need to share data with operators aligns with the diverse and dynamic requirements of 6G networks. The integration of standardized AI/ML approaches, such as the 3GP’'s Management Data Analytics System (MDAS), underscores compatibility with 6G technologies, offering a pathway for intelligence incorporation into various management functions across different domains. 3.4.4 Summary Effective AI relies on high-quality data, driving the need for a flexible data ingestion architecture like DataOps. DataOps ensures data quality through processes like cleansing and validation, crucial for AI model training. Automation and continuous monitoring accelerate the AI development lifecycle while emphasizing scalability, flexibility, and robust security. Key KPIs and KVIs within DataOps focus on model convergence, network efficiency, model inference, data ingestion, user engagement, satisfaction, and model reusability. Leveraging diverse data sources enables personalized ML models, enhancing user experiences and decision-making while ensuring data privacy. These principles align with efficient and secure data management in DataOps environments, fostering an efficient data ecosystem. DataOps enhances diverse technological applications. For Seamless Immersive Reality, it efficiently manages data integration, ensuring quality and deploying machine learning models for immersive experiences. In autonomous robots, DataOps integrates sensor data, processes it in real-time, and manages machine learning models, improving efficiency and safety. Similarly, in vehicle networks, DataOps optimizes performance, fosters communication, and ensures security, enabling smart transportation. Moreover, for Digital Twin use cases, DataOps integrates and analyses data for improved decision-making and optimization. Lastly, in humancentric 6G services, DataOps ensures efficient data management, real-time analytics for personalized healthcare, and privacy protection, enhancing reliability and scalability. Moreover, the mapping of this enabler to the 6G system blueprint is illustrated in Figure 3-1. Table 3-4 summarizes the main benefits and implications of the data-driven architectural means and protocols enabler. Table 3-4: DataOps Enabler Description DataOps ensures data quality by cleansing and validating data, crucial for precise AI model training. It fosters collaboration and efficiency accelerating development through automation, integration, and monitoring. Additionally, it ensures scalability, flexibility, and robust security measures. Benefits KPI improvement From a DataOps perspective, the most critical KPIs are accuracy rate, automation rate, and data drift detection rate. These metrics ensure the reliability of data processing, efficiency of operations, and proactive identification of changes in data patterns, respectively. Design principles [HEX223-D21] Principles #1, #2, #3, #4, #5, #6, #7 and #8 Dependencies / Basis for another enabler AIaaS, architectural means and protocols, and MLOps depend on DataOps for efficient data management and collaboration. Architectural means and protocols can integrate DataOps seamlessly, ensuring compatibility and interoperability. MLOps relies on DataOps for efficient data preparation, enabling smooth deployment and monitoring of ML models. AIaaS benefits from DataOps by leveraging its streamlined data workflows and efficient processes Implications Requirements Key requirements for a data-driven architecture within the DataOps perspective include implementing robust data quality management, establishing automated end-to-end data pipelines, and implementing version control for data artifacts. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 52 / 195 Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 53 / 195 4 Network modularisation A modular 6G design is introduced in [HEX223-D32] where the architectural aspects for 5G evolution as well as its implications have been discussed. In this deliverable, the four fundamental enabler blocks of the modular 6G design (i.e., presented in [HEX223-D32]) are aggregated into two main aspects of network modularization, namely 6G network modularization and E2E service design in modular 6G. “6G network modularization” details a methodological approach on module design, where the module can incorporate different 5G NFs or micro-services that are allocated within the same network container. In other words, the modularization study can be considered as the design and analysis of new 6G NFs. Whereas “E2E service design in modular 6G” focuses on how the modules with various degrees of granularity and composition are utilized for various deployment options and scenarios to achieve a high degree of operational flexibility and efficiency, e.g., improved latency, throughput, better sustainability via scalability etc. Figure 4-1 maps these two enablers to the 6G E2E system blueprint that has been proposed in [HEX223-D22]. Network modularisation enablers present fundamental changes to the way the 6G network functions and their interfaces are designed and the E2E interactions of these network functions, i.e., represented with the solid red line on Figure 4-1. Moreover, the envisioned 6G pervasive functionalities can impact as well as be impacted by the enablers of this section, i.e., represented with the dotted red line on Figure 4-1. In the following subsections, the details of architectural changes on the network functions and pervasive functionalities layers and their preliminary evaluations are presented. At the end of this section, the summary sections detail the benefits and the implications of the architectural changes. Figure 4-1 Mapping of the network modularisation enablers to the 6G E2E system blueprint of [HEX223-D22]. 4.1 6G Network modularisation 4.1.1 Introduction A multitude of services and use cases have been envisioned for 6G [HEX223-D12] that encompasses various requirements and deployment scenarios. However, to achieve the KPIs posed by these services, the network architecture (i.e., the NF design, the relevant interfaces and procedures) needs to be optimized per service needs, which requires a highly flexible and efficient network design. Flexibility and efficiency have also been the fundamental principles in the previous technologies. For example, with 4G networks the idea of separating the control signalling and the user data streams, i.e., Control and User Plane Separation (CUPS), was introduced. However, the low granularity of CP network functions (NFs) was a major limitation on the flexibility. To solve this limitation, the 5GC is built upon an SBA, where the CP NFs have a higher degree of Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 54 / 195 functional decomposition. Through the interactions of these NFs, it is possible to execute relatively complex 5GC procedures. This highly granular structure of the 5G core ensures an increased flexibility and scalability. However, 5G SBA also increased the inter-NF dependencies. For example, to perform a UE registration procedure more than six network function needs to interact with each other [23.502]. Consequently, the current design can lead to increased signalling cost due to the increased number of new modules that need to be interacted to execute a procedure which would also impact the latencies. The modularization as a candidate solution is revisiting the functional composition of the NFs and optimize them for specific deployment options or procedures. Although a module can be built using 5G NFs, it is also possible to decompose a 5G NF and create more than one module from this NF. As the research community’s focus shifts towards 6G, the modular network architecture and how to design such modular NFs become a major point. The design of a module (or modular NF) depends on not only the service based KPIs but also the network provider’s KVIs as well as the deployment options. To this end, a better balance between NF granularity and the number of required interactions between the system elements is required, so that NFs can be added, updated, and replaced in a flexible and modular manner. The idea is to minimize the number of NFs involved in the different procedures by aggregating common ones into new network modules. Thus, it will be possible to reduce today’s 5G functional dependencies that create long sequential procedures with several variants. In this chapter, the 6G network modularization enabler and its implications are described. First the impacts of different module granularities are investigated. As summarized above, a high granularity of the control plane would bring both advantages and disadvantages. To understand the potential of network modularization, a clear understanding of the different granularity options and their implications on the E2E performance is needed. A module can contain different network functions or functionalities (e.g., microservices), and there are different ways to determine how to create a network module (cf., Figure 4-2). In this enabler, different decomposition options and their performances are presented. Finally, the high degree of flexibility is achieved via the interactions between different NFs in 5GC. In 6G, taking advantage of the findings of 5G, the inter-NF or inter-module integration needs to be streamlined, i.e., by optimizing where possible, removing unnecessary interactions, and preserving the already optimum ones. In the last part of this enabler, these streamlined interactions are presented. 4.1.2 Module design and composition The current design of the 5GC enhances flexibility and agility in introducing new functions to the core network. In this architecture, individual NFs, such as the Access and Mobility Management Function (AMF) and the Session Management Function (SMF), are characterized by specific logic for execution, with each NF generating services i.e., consumable by others. The core network facilitates basic procedures like UE registration, UE deregistration, and PDU session establishment by defining interactions and information exchange among these NFs [23.502]. However, the collaborative execution of procedures by different NFs leads to an increased volume of signalling traffic as they exchange messages and information elements. Furthermore, the Procedure Completion Time (PCT), representing the time required for a procedure’s full execution, is prolonged due to inter-NF communication among the involved NFs [GSH+22]. Exploring alternative designs for 6G involves considering new control plane core network functions to minimize interNF signalling and reduce PCT in the system. The placement of 5GC NFs is typically static and depends on RAN network topology and mobile network connectivity to other networks. The core NFs placement optimization goal is to minimise network reaction time, the overall signalling traffic and network cost. Users’ locations and user generated traffic depend on time-of-day, e.g., sport event can cause a serious local traffic growth. To manage signalling congestion and to increase 5GC programmability, the CP NFs have relatively high degree of functional decomposition in 5G SBA. An important feature of 5GC is the ability to add new CP NFs that can interact with other NFs to create more advanced CP services. 5GC CP NFs can be implemented in the cloud which gives various benefits. Firstly, the scaling mechanism of the cloud can allocate resources to a virtualised NF if needed, for example to handle more requests by an NF. Secondly, virtualised NFs can be migrated or cloned during network runtime. However, the use of stateful NFs can make such processes complex. An important performance implication of SBA in 5GC can be the delay in transferring and processing CP messages. The transfer delay comprises a component related to the physical distance between NFs, a delay related to a message bus and a Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 55 / 195 delay related to virtualisation (typically not negligible), whereas the processing delay can be controlled by allocating resources to a NF using scaling. In particular, NF can be scaled in response to signalling traffic growth to avoid NF overload. Figure 4-2 Decomposition of 5G NFs with changing granularity In addition to the delay, there are multiple implications of the NF granularity and the module creation, i.e., outlined in Table 4-1. Therefore, to evaluate the different architecture options and procedures, a multitude of design criteria must be considered that affect the KPIs of the system [AKE+24], including but not limited to • the affinity constraints, e.g., the nodes are meant to be placed in different physical locations; • latency limitations, which concern how a procedure is constructed based on the current architecture; • reliability requirements, which manifest themselves in affinity and anti-affinity rules, persistent memory, etc.; • new technologies, e.g., cloud, virtualization, IP protocols, and quantum technologies; • unique requirements posed by the use cases such as Joint Communication and Sensing; • signalling traffic volume; • self-sufficiency, i.e., indicates how many times a certain function depends on another function to complete a task; • backward compatibility; • the number of failure points that indicates how many times a functional entity requires a re-start of a procedure resulting from a failure to send/receive a message; • service (including CP/Management Plane services) reaction time (delay) and other factors understood as service KPIs; • service-CP/CP/MP overhead (control/management traffic volume minimization); • data type and characteristics, i.e., dealt within procedure execution which would impact how many and what type of data processing mechanisms/libraries are needed to be deployed in a node; • easiness of functions migration and cloning without disturbing other services. Customised techniques can be used to optimise the virtualisation-related delay, e.g., by the selection of a suitable inter-NF message passing mechanism. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 56 / 195 Figure 4-3 Impact of modularization granularity on performance Generally, there is a trade-off between NF granularity and the overhead of inter-NF interactions, as illustrated in Figure 4-3. Low granularity modules reduce the number of inter-NF interactions but long paths to an NF increase response time, while highly granular NFs can be placed closer to the traffic source or sink to reduce response time, but the number of inter-NF interactions increases. When the approach defines more granular modules, the execution time of procedures will be shorter due to the reduction of messages round-trip time. High granularity of NFs should be combined with identification of services that a chain of such NFs can handle. Such a service can be linked with mobility management, security or a user. Table 4-1 Advantages and drawbacks of fine-grained modularisation Advantages Disadvantages Scalability management: efficient and targeted scale-up and scale-down of the resources assigned to the fine-grained modules. Performance: the latency increases when traversing more modules to complete the NF execution. Maintenance: easier to identify, isolate, and replace faulty fine-grained modules. Interfaces definition: interfaces between modules should be defined for cross-vendor deployments. Development: faster when developing different independent modules in parallel. Note that this may come at the cost of increased complexity in terms of integration and testing. Management: managing a bigger number of modules and their interactions increases the management overhead. Deployment: flexible in placing modules in distributed deployments. Signalling: more signalling and data exchange is needed between fine-grained modules. Reusability: efficient when reusing fine-grained modules in different implementations. Context data management: more memory transactions are needed by the modules to read and store stateful data. Different granularity level of a module raises different potentials and drawbacks that need to be considered based on the specific use cases as well as the overall KPIs. Following the granularity of a module, the second criteria is to choose how to create the module. In this deliverable, two major modularization methods are considered, namely Procedure based decomposition and Dynamic decomposition and placement. Procedure based decomposition: The primary objective of this design is to disrupt the strong interdependencies between NFs by introducing Procedure-based NFs (PbNF). Unlike the current 5G architecture where the logic needed to execute a full procedure such as UE registration or UE deregistration is distributed among different NFs, the procedure-based decomposition methodology consolidates the logic for executing a complete procedure into a single PbNF that has a higher modular granularity compared to the current 5G Core NFs. For example, one PbNF is the UE registration NF which is made up of the processing logic that is distributed in the following 5G NFs: AMF, Authentication Service Function (AUSF), PCF, NRF, UDM, and User Date Repository (UDR). The design and implementation of this methodology is described in the following and is based on [GSH+22]. The implementation of the PbNFs is done by reusing the source code of the stateless Free5GC v3.0.6 simulator [Free5GC]. For instance, the source code of the NF handling the Registration procedure (Registration NF (RegNF)) includes logic that, in an SBA system, is distributed among AMF, AUSF, PCF, and UDM. Consequently, the processing logic as well as the services involved in this procedure execution is consolidated Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 57 / 195 into one RegNF as shown in Figure 4-4. Since NRF, UDR, and Unstructured Data Storage Function (UDSF) serve as service-discovery mechanisms or database abstraction layers whose implementation depends on the operator’s backend database, the NFs are not integrated into the new PbNFs. Figure 4-4 Integrating processing blocks from different NFs into UE Registration Procedure-based NF During the implementation, the AMF is used as the basic NF whose source code is expanded by incorporating additional processing blocks code to realize the PbNFs. Additionally, a consolidated UE context is constructed to encompass all information that was previously distributed across various NFs. Special consideration is given to identifying any redundant data to prevent unnecessary duplication and overhead. Given the diverse procedures handled by different NFs in proposed system, it becomes imperative to implement at least procedural statelessness. Consequently, upon completion of a procedure, a PbNF serializes the UE context and state, transmitting it to the UDSF, which maintains the most up-to-date versions. The local context is then purged to avoid undue memory resource consumption. Subsequently, when a subsequent procedure is initiated, the responsible PbNF retrieves the information from UDSF before proceeding to fulfil the request. Utilizing the previously outlined methodology, UE RegNF, PDU Session Establishment NF, PDU Session Release NF, and UE Deregistration NF PbNFs are implemented to perform the tasks associated with UE Registration, PDU Session Establishment, PDU Session Release, and UE Deregistration procedures, respectively. Figure 4-5 shows a 5G or 6G system based on the proposed PbNF. The PbNFs could be deployed in edge locations to mimic the distributed deployments of AMF. It still interacts with the other unmodified 5G NFs such as UDR, NRF, UDSF, and UPF. The RAN&UE Emulator Node depicted in Figure 4-5 is implemented as a gNB and UE Emulator application. The purpose of developing this application is to be able to generate scalable control plane traffic and evaluate the performance of the updated 5G system that incorporates PbNFs. Furthermore, the Emulator Node includes tools to collect and report different performance metrics such as PCT. In future work, this design will be evaluated and compared against the current 5G system as a baseline while considering different quantitative and qualitative metrics. These include the PCT, the volume of signalling as well as the impact on deployment flexibility, duplicated services and interface terminations, and redundant implementation and testing. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 64 / 195 Zenoh and Zenoh-Flow were selected to implement the proposed approach of a data-centric SBA for edgenative 6G networks. Both present a unique set of capabilities that make them a perfect fit for the Data-centric Service Routing layer and Service Dataflow, respectively. Zenoh /zeno/ is a pub/sub/query protocol unifying data in motion, data at rest, and computations. Moreover, it is fully decentralized and provides enhanced capabilities, like the dynamic discovery of nodes, data priority, and low network overhead. Such capabilities have already proven its suitability as communication protocol between edge applications [Sab21]. Zenoh-Flow is a dataflow programming framework that leverages data centricity and location transparency, thus allowing for applications to span across the continuum. It provides a declarative dataflow graph definition, seamless operation across the continuum (including automatic deployment), fosters code reuse by the means of composite operators, and achieves high performance with low latency and higher throughput than state-of-theart solutions such as Istio, HTTP and gRPC. Zenoh-Flow functionalities are already being leveraged as a dataflow layer for network intelligence algorithms in the context of automated network management [GCF+22]. Open5GS an open-source implementation for 5G Core and EPC (Release 17 at the time of writing), was deployed, along with emulated gNBs and User Equipment (UE). This setup was used to obtain traces of the selected 5G workflows, which were later used to replay the behaviour of the NFs, their reference points, interactions, and exchanged data. Our experiments include not only the proposed solution, but also HTTP, gRPC, Istio, MQTT, and Kafka solutions due to their adoption in today’s 5G systems. The validation has been carried over our internal testbed composed of 3 machines (AMD Ryzen 7 5800X @ 4.0 GHz, 32GB of DDR4 3200MHz RAM, running Ubuntu 20.04 LTS), directly interconnected via 100GbE fiber-based NICs over a ring topology. The 5G workflow selected for evaluation purposes is the PDU Session Establishment [29.502]. This workflow follows a Request/Reply pattern. For this workflow, the proposed solution is compared against (i) HTTP, as the protocol defined in the 3GPP for the 5G SBA communication, (ii) gRPC, a widely used protocol for SBAbased interactions, and (iii) Istio, a widely used open-source service-mesh implementation. Results are shown in Figure 4-9 , depicting the total time required to complete the entire PDU session establishment exchange. Figure 4-9 Workflow completion time for different protocols. (lower is better). The proposed solution performs one or more orders of magnitude faster than HTTP, gRPC, and Istio, regardless of service discovery being performed. HTTP and gRPC show similar performance. The reasons lie on (i) a more efficient data exchange; (ii) faster serialization; and (iii) less wire overhead, thus less transmission time. On the other hand, for the service-mesh implementation, the reason lies in how the service mesh is implemented, consisting of a chain of proxies to provide location transparency, thus increasing overall latency. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 65 / 195 4.1.4 Summary The envisioned extreme KPIs targets as well as the efficiency and flexibility goals of the 6G system shift the research communities focus towards revisiting the 5G network functions and creating new modular structures, i.e., optimized towards particular aspects. This, however, raises the fundamental question of how the modularization should be performed, e.g., with which granularity or which methodology and how should, if any, the E2E interfaces and interactions be evolving. This enabler aims at answering these questions by stating the advantages and disadvantages of various options and presenting some preliminary results. Through modularization, the network function composition can be designed to meet specific requirements i.e., set by the service or the operator. Consequently, it is possible to revisit the design to meet extreme requirements e.g., less than 1ms latencies i.e., set by collaborative robots and digital twins. Moreover, in addition to the measurable goals such as QoS metrics, it can also optimize the energy consumption and complexity. For the use cases that requires a high degree of flexibility, e.g., immersive experience and fully connected world, the modules designed in a highly granular way. The work in this section explains the different strategies to perform this modularization, the relevant evolution of the E2E module interface and interaction design and the possible implications to the E2E performance. As a fundamental methodology, the network modularization can impact all the other enablers, i.e., including but not limited to Joint Communications and Sensing (JCAS) protocols and AIaaS. Consequently, it would have implications to all the 3GPP standards, including but not limited to [23.501], [23.502] and [38.413]. Table 4-3 Network modularisation enabler summary Description This enabler focuses on different granularities of a module, how to design a module and the evolution of the modules based on different deployment locations and use cases. This enabler also investigates the streamlining the interactions between different modules/domains Benefits KPI improvement Note that the major argument of this enabler is the capability to optimize the 6G NFs or modules according to certain KPIs. Therefore, it would decrease latency (E2E), procedure completion time, CP signalling, and it will reduce complexity. It will also increase efficiency via direct signalling and less interfaces while enhancing network scalability, flexibility (i.e., both deployment and execution), and reliability. It will provide uniform and reliable service for all users Design principles [HEX223-D21 As a fundamental enabler that has direct impact on the NF design and 6G architecture, it has impact on all the design principles provided by [HEX223-D21], with particular impact on Flexibility to different network scenarios (#3) by flexible composition of NFs/modules, Network scalability (#4), Resilience and availability(#5), Persistent security and privacy (#6), Internal interfaces are cloud optimized (#7), Separation of concerns of network functions (#8), Network simplification in comparison to previous generations (#9) through decreased number of interfaces and interaction to execute a procedure. Dependencies / Basis for another enabler This enabler is a fundamental enabler that does not have any dependency to any other enablers. However, as it analyses and proposes new NF compositions and architecture, it can serve as a basis for all other enablers, including but not limited to E2E service design in modular 6G, JCAS protocols, and AIaaS. Implications Requirements Firstly, this enabler requires the management functions between RAN and CN to be revisited and changing the core NF design and implementation. As the new NF compositions evolve, new interfaces might be needed for inter-node network-level coordination. Finally, redesign of the architecture considering new approaches is needed which might need a “clean slate” approach. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 66 / 195 Standard relations & regulations The network composition changes as well as the relevant interfaces and interactions would impact all the 3GPP standards including but not limited to [23.501], [23.502], [38.413]. Needed resources All the solution is mostly software based 4.2 E2E service design in modular 6G 4.2.1 Introduction Extending the service heterogeneity in 5G, 6G is envisioned to host a larger number of different services [HEX223-D12]. These services pose strict KPI and QoS requirements, which require the network resources to be optimized according to the needs of the hosted services. To optimize the shared network resources to meet the conflicting needs of these services, 5G utilized the network slicing concept, which is built upon the idea of creating virtual and dedicated resources (i.e., physical and computing) i.e., customized for specific use cases. However, the semi-static nature of the network slicing that is built upon strict SLAs (Service Level Agreements) limits the flexibility and places strains on the efficiency of the network. In order to provide a seamless and effective coexistence among different services, the 6G networks need to have a high level of autonomy and adaptiveness to the dynamic needs of the network. This, however, requires an E2E vision in the network modularization that would cover both UP and CP, namely the deployment location of the network modules as well as the specific use case(s) and their KPIs must be considered in the module design phase. Moreover, the network autonomy and adaptiveness need to be ensured in this modular 6G architecture. The modules should be designed considering the reusability and autonomy requirements in multi-cloud environments, which would also allow the network providers to scale the available modules on demand. The respective interfaces to support such dynamic control need to be provided in the cloud continuum. The capability to have this high degree of dynamism in control operations would also bring a more efficient multitenancy in 6G. In addition to the regulatory implications, this enhanced multi-tenancy requires a revisit of the modular structure to determine the optimal degree of control over the modules that should be provided to a tenant, to ensure scalability, reliability and efficiency in this shared network. Finally, the maturity of AI/ML technology raises the opportunity to integrate certain AI functionalities inside the module design and further enhance the efficiency by proactively customizing the modules to the transient network states. Indeed, this integration requires a clear understanding of the extent of AI functionalities to be placed in the modules and how they should be exposed to the other modules/domains. In this enabler, the analysis starts with the E2E implications of modularization and its architectural affects to both CP and UP. In particular the modularization in RAN, the RAN-CN interactions and modular UPF are discussed. The analysis is followed by a presentation of the network autonomy and adaptiveness via modularization where the potential of intent-based management/orchestration to manage the modular placement is discussed. Finally, quantum-based synchronisation to meet the extremely accurate inter-module synchronisation is detailed. 4.2.2 Extended/E2E network modularity in UP and CP 4.2.2.1 RAN modularity The ever-increasing complexity and heterogeneity of wireless networks requires solutions that make RAN open. The flexible deployments brought by disaggregation and virtualization of RAN components allow for increased reconfigurability, interoperability and resiliency of the networks [PBD+23]. The move from the monolithic systems of the past towards open and flexible RANs is likely to continue in 6G, improving upon the principles of openness of the current and past generations. Disaggregation of RAN components and its related functional splits defined by 3GPP for 5G NR [38.401] is contemplated as a possible architectural option for RAN in the shift towards 6G. The RAN functional splits proposed by 3GPP [38.801] divide the functionalities into CUs, DUs and RUs. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 67 / 195 Inter-cell interference is an intrinsic limiting factor of network-centric cellular networks [AZD+16]. Users located near the cell edges experience the worst network performance. By shifting the view from networkto user-centric networks, this hindering limitation can be mitigated, providing uniform and reliable performance for all users. Cell-free MIMO is a technology in which each user is served by a group of RUs located around that specific user, effectively eliminating cell boundaries by providing a cell-free experience to users. That is, from the users’ perspective, there are no cell boundaries during data transmission, and no perceived reconnection as users move [NAY+17]. RAN disaggregation is a crucial aspect for the practical implementation of cell-free networks, as there is a need for a particular level centralization between RUs, to enable joint processing. Certain functional split solutions allow for flexible and scalable cell-free solutions. Most of the focus is put in the functional split between the RU and the DU. There are some aspects to consider when selecting suitable splits, such as module complexity and fronthaul throughput and latency requirements. As more functions are placed in the RU, coordination between different RUs by the DU becomes more difficult or even impossible, as the channel estimates must be in the DU. The corresponding splits consider the separation of physical layer (PHY) functionalities into the RU and DU. The interface between the RU and the DU, the fronthaul, must support UP and CP traffic between these two nodes. There is a trade-off between the RU complexity and cost, and the amount of data that goes through the fronthaul, as well as the latency requirements. As more functions are placed at the RU, it becomes more complex and costly, whereas the total fronthaul traffic decreases. Additionally, the requirements on fronthaul latency, i.e., between the RU and the DU, become less stringent [38.801]. Figure 4-10 Logical RAN architecture enabling distributed cell-free operation. Users may be served by different overlapping clusters of RRHs. For this, RU resources can be divided to different DUs. A possible implementation of cell-free operation on disaggregated RANs is with the introduction of the concept of RU multi-clustering, as depicted in Figure 4-10. As shown, in this case the RU performs low PHY functionalities such as cyclic prefix insertion or removal and (Inverse) Fast Fourier Transform ((I)FFT). The DU performs the rest of the PHY functionalities as well as providing functionality associated with the MAC and Radio Link Control (RLC). Finally, the upper layers (Packet Data Convergence Protocol (PDCP) and Radio Resource Control (RRC)) are implemented at the CU. The red lines represent the additional interface connections that make possible this implementation, compared to a conventional cellular Distributed MIMO (D-MIMO) network. The RU-DU split corresponds to split 7-2, whereas the DU-CU split corresponds to option 2, as defined by 3GPP [38.801]. The RUs are initially clustered depending on their physical locations, with one DU managing each of the resulting RU clusters, forming the so-called basic clustering. By allowing RUs to be connected to multiple DUs, multi-clustering is formed by the means of shifting, in which the RUs at the borders of the basic clustering are also connected to the neighbouring DUs, giving to each DU a fraction of their radio resources. Each DU therefore manages multiple clustering of RUs, each of them (aside from the basic clustering itself) resulting from a shift of the basic clustering in a certain direction, where new RUs become part of that cluster, and some are removed. The total bandwidth is orthogonally divided in the same number of partitions as clustering’s there are in the network (the basic clustering and the new ones obtained Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 68 / 195 from the new RU-DU connections). Each partition is then associated to a clustering, effectively defining multiple orthogonal (in frequency) D-MIMO instances. By allowing negotiation between DUs on which resources are allocated to the RUs they share, it is possible to formulate a user scheduling and resource allocation problem that aims to improve the downlink spectral efficiency of users, especially of those located at the cluster edges. Ideally, users will be served by an RU cluster that places them at the centre, making the network more user-centric. Comparing the results of this approach to other benchmarks, such as single clustering D-MIMO and collocated Massive MIMO, it can be seen that it provides better service for users, especially for those who suffer from the cell-edge problem. This is presented in Figure 4-11, which shows the empirical Cumulative Distribution Function (CDF) of the average downlink spectral efficiency for the different approaches studied. The gain is higher in users which suffer from high co-channel interference in single clustering D-MIMO, as can be seen by comparing the spectral efficiency of the 5th percentile. Additionally, since the functional split defines the front-haul as carrying frequency domain samples, and the partition is also done in the frequency domain, the proposed solution does not increase the fronthaul traffic, compared to a single-clustering solution (D-MIMO). Figure 4-11 Empirical CDF of the user average spectral efficiency, comparing RRH multi-clustering with different benchmarks. As mentioned in section 2.2.4, the LLS may work well with Cell-free MIMO instead of the aforementioned higher layer split of CU and DU. The LLS is a separation of baseband (RAN stack) and RU by a fronthaul interface. This interface is typically inside the physical layer (PHY, L1). The use of LLS instead of HLS and a split between CU and DU may reduce the number of signalling messages required to acquire and utilize all relevant information for each UE. 4.2.2.2 RAN – CN interface in modular 6G So far, the discussion has focussed on the modularity and interfaces of the RAN architecture. This section provides a more detailed look at the interfaces between the RAN and the CN and how to improve the E2E modularity. The control plane signalling between RAN and CN communicates via the N2 interface and always communicates via the AMF (regardless of what function in CN being the “final destination”). One reason for using the AMF as proxy is security, including isolation between RAN and CN and a single NAS security termination point for the UE. Any CN function accessing RAN needs to go via the AMF. Procedures supported over N2 are for example PDU Session Management, UE Context Management, UE Mobility Management, Paging, Transport of NAS Messages and NG Interface Management. So, can the N2 interface be made more modular for 6G? One option is to reuse a Service-Based Interface (SBI) as in SBA. One advantage with a SBI interface could be to reuse the SBA framework (e.g., registration, discovery, authorization) also for RAN – CN signalling. However, a RAN-CN SBI has challenges that are described in the following. A possible consequence that would need to be addressed is handling a high number of processes for service registration and discovery from many RAN functions. Furthermore, SBA is considered to be more “chatty” than previous Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 69 / 195 approaches and this chattiness could impact RAN-CN signalling performance. Security can also be an issue and some alternative to the AMF proxy needs to be found. Alternative here means some functions that provides a similar level of security. The N2 interface between the current RAN and AMF supports the so-called NG application protocol (NGAP). The underlaying transport protocol for the NGAP is the SCTP. Currently, the NGAP relies on ASN.1 binary format with a specific order of the information (the so-called information elements, IEs) where each IE is associated with specific ID field allowing quicker parsing. NG Setup can be triggered after the Transport Network Layer (TNL) Association, handled by SCTP, between RAN-AMF has become active, see Figure 4-12. Figure 4-12 NGAP association and the AMF connection to the RAN The binding between an TNL Association and an NG-AP instance are handled as part of the so-called NG interface management procedures. The NG interface mandates the usage of SCTP as transport protocol and the application logic of the NG interface is defined leveraging on SCTP features such as semi-permanent connections and real-time redundancy information. The architectures of the RAN and CN CP are expected to undergo changes in 6G for the support of 6G radio features, new applications and services envision for 6G, energy efficient design, and more. Here the objective is to investigate how to design efficient network functions and how these functions interact in the architecture, in terms of combined KPIs such as latency, failure points, dependencies and number of messages. [HEX223D32] discusses KPIs to use for evaluating efficient signalling and a “KPI map”, the NF design, SBA and SBI on general terms. Here, we consider three evolution directions for 6G RAN-CN CP interfaces. The first is considered a stepwise evolution to 5G NGAP while the other two are more revolutionary. Figure 4-13 Considered RAN-CN CP options for 6G (Option A-C) In the first option (i.e., Option A), existing architecture and interfaces are used where 6G RAN communicates via NGAP with the AMF, or a 6G equivalent of it. The main features of this approach include RAN maintaining ASN.1 encoded interfaces, 6G AMF maintaining the gateway functionality between RAN and CN NFs, thereby being an evolution of 5G. The nature of 6G AMF and the required modifications to this NF are to be studied. It can either follow an enhancement to AMF functionalities or a completely new NF design. In option B, the RAN-CN interface is changed by introducing a Service-Based Interface (SBI) between 6G CN and RAN. In this case, this interface would enable the 6G RAN to directly expose and consume services towards and from the CN, respectively. Similarly, it would enable all CN NFs to directly expose and consume Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 70 / 195 services towards and from the 6G RAN. Communication between CN NF and RAN is performed over defined service APIs and operations. This approach supports the usage of cloud-friendly mechanisms and protocols and enables harmonization of services across the RAN and CN domains, such as security, exposure and discovery, etc. It is important to ensure that RAN and CN domains remain treated separately at standardization and implementation to maintain flexibility (such as RAN sharing), inter-vendor operability, etc. In any case, this option requires significant effort in standardization, implementation, optimization, and deployment. As a trade-off between options A and B, a hybrid approach is a possible evolution of the 6G RAN-CN CP interface where service-based interaction is introduced with minimal impact on the existing point-to-point interactions and without creating duplication among the two interfaces. This can allow cloud-friendliness, deployment flexibility, and room for performance optimization for certain functionalities and services while decreasing the standardization and implementation impact when compared to option B. Yet, the consequences of implementing multiple protocol stacks in 6G RAN have to be analysed carefully as there might be complications in management, interoperability, testing, etc. Table 4-4 summaries the NGAP procedures related to NG interface management and a possible equivalent SBA functionality, with focus on differences. The table includes the main management functions supported by NGAP, such as setup, error handling and overload functionalities. Table 4-4 Summary of NGAP functionality and corresponding SBA functionality NG management procedure: Possible equivalent SBA functionality: NG/N2 Setup Can be replaced by SBA Service discovery/registration. Note that in 5G it is the RAN that triggers NG Setup towards AMF. When using service registration/discovery, it has to be understood if RAN should be discovered or not (and eventually if RAN should register its services). NG Setup also has the functional role to make RAN operational. RAN/AMF configuration such as PLMN info, Slice support, Default Paging DRX, Globally Unique AMF ID (GUAMI). The configuration parameters are few and should be possible to get from e.g., the NRF via Service discovery/registration and status notification. AMF load management and AMF load balancing The AMF load management still needs to be part of the SBA solution, new or modified procedures need to be standardized and implemented. NOTE: to be understood whether indications about load management should be handled via 3GPP procedures or if it is possible to handle this in the cloud platform with limited or no implications to 3GPP procedures. Adding new AMF, replace AMF In NGAP, the new AMF sends out the GUAMI to the gNBs; can this functionality be moved to the NRF and discovery/registration of SBA. The UE context can still be stored in the UDSF if an AMF is replaced. Multiple TNL Association management TNL Association management is not needed in SBA; this is handled inherently by the SBA. NG Reset (e.g., to clean out context in RAN at failure) This can be handled through NRF which keeps track of function’s status (e.g., service/function de-registration when decommissioned). NOTE: to be understood how to handle the reset of signalling relevant to only one specific UE and not to the whole function. AMF status, Overload start/stop Possibly this can be handled by NRF changing some service information (or by direct signalling between RAN/AMF). NOTE: NG overload start has also functional value of e.g., informing RAN about which signalling should be prioritized (or rejected) in case of AMF load. To be understood how this could be realized. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 71 / 195 Error indication between RAN and AMF The AMF or the RAN can understand the cause of an error and take action based on this. This cannot be replaced with SBA framework functionality As a summary of Table 4-4, SBA can replace the NGAP functionality for most cases. However, we do not see any actual benefits by adopting SBA over NGAP. Moreover, the error handling and possibly the AMF load balancing that exist for the 5G NG management is not directly supported by SBA, so these functionalities probably need to be added on top of the normal SBA functionality. One principle should be that the signalling application is responsible for handling the features required by its logic (following an E2E principle), e.g., determine when a signalling transaction is completed. Transport layer re-transmissions of messages can still be used, but knowledge of application signalling losses should be at application level since this gives more freedom on how an underlying transport is deployed or used. For example, it will then be possible to use multiple transport associations without explicitly managing those associations on application layer or possible to deploy application and transport layers in different container without relying on the assumption that receptions of messages at transport layer implies also safe reception at application. The main benefit in moving the E2E responsibility to the application is that the coupling with the signalling transport goes away, e.g., removing the binding of application signalling to long lived SCTP associations, removing the reliance on SCTP association monitoring. However, this does not necessarily require the adoption of a SBI between RAN and CN, and it should be possible to include this e2e functionalities also to application logic of point-to-point interfaces between RAN and CN (e.g., relying on application layer acks transaction status handling as well as application-level connection monitoring if needed). 4.2.2.3 CN UPF modularity Finally, while the control plane of 5G Core networks is designed to follow the SBA, the user plane and in particular UPF is still dealt with as one big monolithic block with many functions and features to support (approximately 20 based on 3GPP TS 23.501 Release 18). Big monolithic design of functions hinders the ability to scale in/out resources assigned for subfunctions at a fine-granular level, slows the development cycle in terms of deployment and debugging, reduces the flexibility in using different technologies for developing different components of the design, etc. [AAE16] As a concrete example, the UPF may handle an asymmetric volume of uplink/downlink traffic to be processed at the Core Network. In this case, a modular design of UPF would enable scaling in/out the processing resources for each traffic direction independently based on the demand. Another example is when the UPF is expected to run some features, such as Lawful Intercept, over a specific period of time and over different locations. In this case, the module that implements the Lawful Intercept functionality could be provisioned with processing resources independent from UPF to cope with the volume of the demand, and thus the processing resources would be utilized more efficiently and sustainably. To this end, we propose a modular design for the UPF of the next generation Core Networks. The following entities are proposed [HAR22]: Ingress Steering Module (ISM): This module handles the incoming packets when they arrive at the UPF. First, it checks the validity of the received packets based on defined admission control rules to drop the invalid packets. Then, it steers the packets to uplink or downlink modules. This module can implement load balancer mechanisms to balance the processing workload when multiple replicas of the following modules (i.e., uplink function or downlink function) exist. Downlink Module (DLM): This module handles the packets coming from the DN to the gNB and later to the UE. It implements all the packet processing required to fulfil the basic tasks assigned to the UPF when handling downlink data traffic. Uplink Module (ULM): This module handles the packets coming from the UE through the gNB to the DN. It implements all the packet processing required to fulfil the basic tasks assigned to the UPF when handling uplink data traffic. On-Demand Modules (ODM)s: This list of modules includes all optional functionalities assigned to the UPF, which can be activated on-demand, and which are considered discrete compared to the basic packet processing. Example of On-Demand functions could be: Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 72 / 195 1. Lawful Intercept Module (LIM): This module handles the lawful interception of user data for some devices and over a certain period of time on a per-demand basis. 2. Enterprise specific Processing Module (EPM): This module handles Time Sensitive Networking/Communications Translator functionality, or similar functionality defined for supporting integration of 5GS to DetNet, or other similar future enterprise specific handling. 3. ATSSS processing Module (ATSSSM): This module supports ATSSS with all relevant steering functions like MP-TCP, potential future steering functions like QUIC, MP-QUIC, etc., and all existing and future steering modes. Flavors of this module might be steering function specific like, e.g., ATSSS microservice for MP-TCP, ATSSS microservice for MP-QUIC. 4. Redundancy Module (RedM): This module handles user plane traffic duplication and elimination for high reliability like, e.g., support of redundant N3/N9 tunnels. 5. Cellular IoT Module (CIoTM): This module handles IoT related user plane traffic supporting extended buffering, Reliable Data Service etc. 6. Multicast Broadcast Services Module (MBSM): This module supports MBS, MB-UPF functionalities. 7. N6 Tunnelling Module (N6TM): This module serves the various N6 tunnelling options like for example MPLS. 8. DPI Module (DPIM): A typical user plane module applicable in many cases. The optional UPF functionalities provided by the ODMs can be flexibly activated/deactivated on demand and can also be scaled in/out over time as needed. This modular design facilitates discrete specialization, and the best integrated result Figure 4-14 shows the modularized UPF in the Core Network. The packet processing in the user plane will be defined in the spirit of Service Function Chains (SFC), where these SFCs are created as chains made up of the modules defined above. Figure 4-14 Modular UPF design integrated in the E2E mobile network system 4.2.3 Network autonomy and adaptiveness via modularization The Cloud Continuum is a seamless integration of various types of clouds that extends from the centralized cloud to the on-premises equipment, passing through the far-edge and the near-edge. This extended cloud is hence distributed in nature and possibly constituted by several heterogeneous cloud technologies. Potentially 6G network functionalities should be deployed coherently on all the Cloud Continuum, while being effectively deployed only where the specific function, micro-function or module is needed and only for the time it is strictly needed. Also, different decomposition and interactions between network modules may be implemented in the 6G system differently from the current 5G SBA model. As a matter of fact, as depicted in the previous sections, 6G system may be further disaggregated to encompass submodules of network functions that are composed and reused by several functions to provide the resources needed for fulfilling a services request. Consequently, the modularization model can evolve during 5G and subsequently 6G and be highly dynamic. Therefore, an orchestration model, which does not bind to the specific architecture of current network modules, is needed; it must be natively extensible and manage orchestration in a generic way, moving away from static 5G workflow-based orchestration models. In this context, a, flexible orchestrator, should deliver carrier-grade, simple, open and cloud native intent-based automation. It must be natively expandable and not related to specific network architecture or specific network modules. It should be based on common automation Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 73 / 195 templates that materially simplify the deployment and management of multi-vendor cloud infrastructure and network functions across large scale edge deployments. It must enable faster onboarding of network functions to production including provisioning of underlying cloud infrastructure with a true cloud native approach and should reduce the costs of adoption of cloud and network infrastructure. Extension components must be a key feature supported by such an orchestrator to enable the use of custom resources allowing the management of the upcoming modules and their components. The orchestrator behaviour should be extensible without modifying the code by linking controllers to one or more custom resources. Consequently, the orchestrator must be able to manage a huge number of clusters of servers across the telco network, handling a variety of infrastructure technologies with a uniform and consistent user experience, automatically installing and configuring additional plugins, especially for enhancing networking capabilities and applying configuration customizations to tune performance parameters. The development of a network orchestrator hinges on its intent-driven nature, a crucial element for ensuring its effectiveness. While the existing automations have many limitations (e.g., complex templates, difficult to read and test, limited re-use due to huge lists of values that need setting), the ability to continuously reconcile systems with an intent-based approach stands out as a significant advantage, particularly in comparison to imperative tools, as it enhances robustness at scale. In the context of large-scale edge deployments, distributed actuation of intent emerges as a requisite feature. The traditional approach of triggering all actions from a centralized location is unreliable and impractical in such scenarios, where scalability is paramount. The intentbased nature of the orchestrator allows to put emphasis on uniformity in systems management enabling the management of deployment, repairs, and configuration through common components and standardized workflows streamlines operations. Focusing on the initial intent, allows to reduce cognitive load for the operations team, while the distributed actuation takes care of the downstream complexities for the edge locations. This approach not only facilitates more rapid responses but also minimizes the likelihood of human errors in the orchestration process. Figure 4-15 Intent-based orchestration and how the intent can impact the live environment Different network operators may have different requirements and challenges for their network design, performance, and integration which may lead to the adoption of different architectures to suit their specific needs and scenarios. The architectural heterogeneity of different network operators poses significant challenges for the integration and interoperability of heterogeneous networks. On top of this, the cloud continuum represents an additional layer of complexity that can differ from one operator to the other. An orchestrator that leverages an extendable intent-based approach can be used to comply with regulatory and SLA implications by providing a high-level and abstract way of expressing the desired outcomes and goals for the network and the services, without exposing the low-level details and configurations of the network devices and functions. By using intent-based orchestration, network operators can specify KPIs and service level objectives that they want to achieve, such as availability, reliability, security, latency, bandwidth, etc. The intent-based orchestrator can then automatically translate these intents into network policies and configurations and enforce them across the network. The policies are dynamically applied only on those network function that are affected based on the requested KPIs. Moreover, network modules can be serverless functions in order to provide a flexible and scalable way to implement and execute network functions and services. Serverless modules can enable a more efficient, reliable, and adaptable network service delivery and management since they are intrinsically more suitable for a highly dynamic environment. This dynamic orchestration necessitates an enhanced synchronization of different modules and distributed cloud functions. This synchronization is crucial for seamless communication and efficient resource management across various network components [UKK+23]. With the increasing demand on high data rates, Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 80 / 195 Regarding the role of the NTN and TN BSs as either MN or SN, there are two options. The first option is that the UE is connected to the TN as MN and to the NTN BS as SN. In this case, on the UE side, the NTN SN could be in deactivated state, or the NTN RF chain could be switched off for power saving purposes when the UE has a stable connection via the TN BS. If the NTN RF chain is switched off in the UE, then one benefit is that the UE will not perform any NTN measurements but will keep the RRC configuration for NTN. This RRC configuration will be ready for use once the UE moves away from the coverage of TN cell (and reconnects to the NTN). To detect the TN coverage loss and to switch on the NTN RF chain well in time, the TN network may configure UE measurements such that the UE triggers a measurement report when the UE is about to lose TN coverage. This can be achieved by either configuring triggering of measurement events earlier or by defining a new measurement event when UE is about to lose TN coverage. Once the UE triggers this measurement report, it may switch on the NTN receiver at the same time. After receiving this measurement report, a TN BS provides a notification to the NTN BS to take the control of the UE and perform uplink/downlink transmissions. While the UE is out of coverage of the TN BS, the NTN cell could either continue to assume the role of SN or switch to the role of MN. 5G already supports procedures for connection via SN when MN connection has failed, therefore the same procedures and signalling can be reused. Therefore, it is not necessary that the NTN BS always takes the role of MN. The second option is that the UE connects to the NTN cell as MN and to the TN cell as SN. In this case, any mobility between the TN cells can be handled as SN change or Primary Secondary Cell (PSCell) change, which are well defined procedures in 5G. However, one disadvantage of this approach will be that UE handover will take place very frequently due to the movement of satellites and not necessarily due to UE mobility. In this case, the UE will suffer user plane interruption due to handover every few seconds. This approach may have detrimental impact on UE power consumption and signalling load. Therefore, the first architectural TN-NTN option may be more beneficial than the second approach, at least in terms of reduced UE power consumption and lower service interruption time. However, selection amongst these two options depends on factors like agreement between NTN & TN MNO and the core network connectivity for each cell and UE. Finally, for the TN or NTN cell and mobile edge resource allocation, It may happen that a service is terminated at the TN edge and edge resources are located in a closer proximity to a TN cell/network infrastructure for service delivery. These resources may be deployed in different geographical locations compared to the NTN edge resources. For NTN-TN dual connectivity and when the switch happens from the TN to the NTN BS, the compute resources should also be moved accordingly in order to meet service requirements. So, when a UE is about to fall into a TN coverage hole then the compute resources at the edge of the TN BS are moved from a node closer to the TN BS to a node closer to the NTN base/earth station. 5.1.2.2 Subnetworks A Subnetwork (SubNW) is formed voluntarily by UEs based on mutual trust. A UE may assume the new role of a MgtN in a SubNW and become the SubNW’s primary node, which can communicate with the BS and other UEs. The MgtN would take over the coordination within a SubNW while still being in coordination with the global 6G Network (NW). Even though the SubNW is formed among UEs with limited NW configuration and awareness, the SubNW itself may be associated to the global 6G NW, along with all local UEs. In addition, the SubNW shall be well integrated in the 6G NW, so that local UEs are able to seamlessly move between the SubNW and the global NW. One architectural option is that a SubNW is unknown (i.e., transparent) to the global NW, where the NW is unaware of the MgtN acting as man-in-the-middle (e.g., impersonating multiple UE IDs). The BS communicates virtually with individual UEs, but the physical connection is always with the MgtN. The challenges of this architecture are two-fold. On the device side, the MgtN needs to be capable of instantiating multiple UP/CP entities for all devices, as well as of sending and receiving on behalf of every device in parallel. At the same time, the MgtN needs to be ensured that its own device capabilities will not be exceeded. On the NW side, the BS shall unnecessarily maintain individual procedures per UE (e.g., L3, L1, CSImeasurements/reporting, link adaptation, timing advance), which will have the same results, since all links will in practice be the same BS-MgtN link. Finally, since the NW is not aware that a SubNW exists, licensed spectrum cannot be used in the SubNW, and the NW cannot provide any new SubNW-specific functionality. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 81 / 195 Another architectural option is that a SubNW is known (i.e., non-transparent) to the global NW, where the NW is aware of the MgtN and its role, which is to manage local devices and aid other UEs in the SubNW with CP procedures. The UP for all UEs should be handled through the physical MgtN-BS connection, where the MgtN transfers data between a UE and the BS. Even though the challenges of the previous architectural option are resolved here, the aforementioned responsibilities of the MgtN have to be defined. The CP entities of local devices can be flexibly deployed on the MgtN. The global NW does not have to be aware of this CP offloading. Figure 5-3 depicts an example of such an architecture, where the whole or part of UE3’s CP is deployed at the MgtN. The SubNW may then use a new lightweight SubNW CP (snCP) between the MgtN and the UE. Note that the snCP is transparent to the NW, since it includes the configuration and the procedures that take place within the SubNW, as well as the offloaded UE CP information. An alternative architectural option could be that each UE’s CP is terminated at that UE. In that case, the MgtN forwards the RAN-BS’s CP data to each UE. Another option would be for a UE to offload its CP to another UE and not to the MgtN. This is similar to Figure 5-3, with the difference that the MgtN forwards the RAN-BS’s CP data to UE3 and then UE3 forwards that to UE2 in order to decode it and send a simplified version back to UE3. Still referring to Figure 5-3, the MgtN-BS interface would be the 6G Uu-equivalent interface, while the MgtN-UE3 interface may be any access network (e.g., WiFi, Sidelink, Uu). The higher layers of the UP terminate at the local device (e.g., PDU session, IP addresses). The lower layers may terminate at the MgtN, making the MgtNUE interface SubNW-specific. Regarding the relevant use case families, the immersive experience and cobots use case families [HEXD223D12] can be enabled by subnetworks, due to the locality, the density and the diversity (e.g., high-capability, low-capability) of the involved devices. Moreover, the subnetwork topology and the MgtN may also be used in the context of aiding energy-neutral devices with wireless power transfer by receiving those devices’ data from the base station and buffering it until they wake up to receive it [HEX224-D53]. Figure 5-3 UE CP deployed at the MgtN and use of snCP within the subnetwork. 5.1.2.3 Trust-related network functions Traditional mobile network operator (MNO) infrastructures, built on pre-deployed static frameworks, often face challenges in dynamically adapting to traffic bursts and varying user requirements. Rather than relying solely on central nodes, the system should lean towards a mesh-like structure, allowing for greater flexibility [EGG20]. The potential incorporation of terrestrial and aerial nodes, such as UAVs, into these flexible network formations further broadens the architectural horizon. Such integrations emphasize the need for a comprehensive trust framework that spans both terrestrial and aerial nodes, ensuring that the ConnectivityEverywhere paradigm is not just achieved, but is also trustworthy, which, within this context, refers to a multidimensional concept encompassing not just the security of the network, but also its capacity, energy efficiency, throughput, cost-effectiveness, and related aspects. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 82 / 195 The proposed solution follows this decentralized paradigm, with functions comprising node discovery and trust evaluation at its core. The node discovery function operates by continuously scanning the network environment to identify and register new nodes. This process involves collecting information such as node identification, location coordinates, operational status, and network capacity. By maintaining an up-to-date map of the network, the system can efficiently manage and optimize connectivity paths. Functions such as continuous trust evaluation introduce another layer of complexity. Architecturally, this means that the network must be capable of real-time assessment of node trustworthiness. The trust evaluation function, on the other hand, is built on a sophisticated algorithm that assesses various aspects of each node's performance and behaviour. Nodes are assigned trust scores, which are dynamically updated based on ongoing interactions and performance metrics. This evaluation is based on various parameters, including historical performance data, response times, data integrity, and security certifications. Trust between two nodes is established through a mutual evaluation process, where each node assesses the other based on the trust scores and recent interaction history. This ensures that connections are made only with nodes that meet the required trust threshold, fostering a secure and reliable network environment. The trust related functions draw data from the node discovery component and the AI/ML-driven resource optimization component, leading to a tight coupling between trust assessment and resource optimization in the 6G network ecosystem. As such, the communication between these components becomes crucial. Networks must be architected in a way that allows for seamless data flow and instant decision-making, ensuring that only the most trustworthy nodes are selected for data transfer, even in dynamically changing scenarios. Furthermore, the AI/ML resource optimization component, while essential for trust management, also holds broader architectural implications. Its integration means that 6G networks will inherently be smarter and more autonomous, capable of making real-time decisions based on a combination of node trustworthiness and resource availability. This reduces the need for manual oversight, but at the same time, necessitates robust AI/ML algorithms embedded at the core of the network design. 5.1.3 Evaluations 5.1.3.1 User-centric coverage models and prediction The concept of cellular coverage has been largely unchanged since the early days of cellular systems: A certain location is considered 'covered' when a connection with a device can be maintained with a sufficiently high probability. Thus, cellular coverage maps that are maintained by operators typically show coverage as a binary reality (either a geographical location is covered or not). Sometimes, operators introduce some nuances by showing a few distinct maps for certain phone conditions (indoor, outdoor, for instance) or data rates, but in essence, a location is considered covered when a connection can be maintained sustainably, regardless of other indicators of service quality. In the new 6G ecosystem, such a rudimentary assessment of coverage may not be adequate to describe the full nuances of 6G. In fact, in other parts of this document the notion of coverage has been related to other KPIs such as the latency, throughput, or robustness at a certain location. While network of networks and the integration of NTN will improve coverage, a clear understanding and evaluation must be adopted for what this implies. The notion of coverage needs to be tied to one or more certain service quality, e.g., throughput, latency. For instance, a geographical location may be covered under one latency requirement and not covered under another. In fact, because of the broad variety of 6G use cases, the actual support of a certain use case determines the coverage of that use case in a certain location. For instance, NTNs are likely to provide lower throughputs and higher latencies in certain locations compared to terrestrial networks. Certain use cases will therefore not be supported regions with only NTN support. Furthermore, a large-scale assessment of a region's coverage is currently often carried out by (1) assessing the percentage of the area that is covered and (2) by graphical coverage maps of a region. While the first notion (i.e., the coverage percentage) provides a simple, easy-to-compare, scalar value, the second (i.e., coverage maps) has limited use for comparing regions quantitatively [CB24]. Especially, there is a need to compare networks in terms of the coverage of rural regions versus urban regions. Currently, good quantitative measures are largely missing. When 6G introduces NTNs and multi-connectivity, which are expected to improve coverage in rural regions, measures (beyond mere areal percentage) are needed to assess and compare networks. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 83 / 195 A new notion of "coverage fairness" across a larger geographical region is needed that take into account other KPIs (such as local latency or throughput) and can be formulated based on the broadband inequality index proposed in [Fou16]. This notion then captures the extent and fairness by which a large geographical region is covered with 6G access. It provides quantitative understanding to what extent a new network-of-networks architecture improves coverage into the rural and remote parts of a large region. Hence it can be a useful tool for operators. The fairness indicator combines classical radio propagation models with analysis tools from economy and uses notions as the Lorentz Curve and the Gini Index and applies these to the detailed coverage maps of a network, see [Fou16]. The Gini index and related indices in the literature then provide a scalar value between 0 and 1, where 0 reflects complete uniform distribution across the region, while a value of a reflects a completely inequal scenario where distribution is concentrated in a single point. Figure 5-4 illustrates the development of the areal coverage across a large-scale network deployment region (typically a country) and shows the percentage of areal coverage (x-axis) along with a fairness indicator (yaxis), which represents the extent to which the coverage is equally distributed over urban and rural parts of the area. With such a graph, a network’s deployment in a large-scale region can be tracked over time: after a standard has been established the deployment of networks takes years and develops over time. Typically, upon initial deployment, the coverage percentage will be low, and since initial deployments typically appear in urban environments, the fairness indicator will reflect this through a high value (i.e., connectivity is concentrated to small regions in the region). In later stages of network development, typically not only the percentage of coverage increase, but also the coverage fairness with a more even distribution of coverage in urban and rural regions. Figure 5-4 Anticipated development of a network's coverage over time in terms of 'percentage areal coverage' (xaxis) and 'coverage fairness' (y-axis). This notion is proposed specifically for application to the introduction of NTNs, such that in a later stage the value of NTNs in terms of large-area coverage improvement can be related to existing coverage by terrestrial networks. Due to the distinctly different nature of NTNs’ deployment compared to terrestrial networks, the development of these networks over time will likely not follow the above-described trajectory of terrestrial networks in this diagram. 5.1.3.2 Flexible topology instantiation and evaluation The foundational principle of this study lies in the development of a robust framework that facilitates the ondemand realization of scalable, resilient, and flexible network topologies. This framework, which relies on procedures such as node discovery, trust assessment, and resource optimization, employs AI/ML techniques to ensure seamless connectivity associations based on parameters such as cost, trust, resource availability, and specific traffic source demands. A significant step in this research is the introduction of a methodology for dynamic drone placement. This strategy, which is shown in Figure 5-5, unlike traditional brute-force or gridbased approaches, leverages an innovative isometric grid to approximate drone positions, significantly enhancing the search process. Coupled with the utilization of Minimum Spanning Trees (MST) that minimize edge weights, and thus propose drone positions, the process of on-demand topology creation is streamlined. A Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 84 / 195 pruning algorithm further optimizes the network by iteratively eliminating non-essential leaf nodes, while redundancy elimination ensures that only those drones that enhance connectivity remain part of the network. Figure 5-5 Flexible Network Topology Using Systematic Drone Positioning Figure 5-6 AI-Enhanced Flexible Network Topology Using Dynamic Drone Positioning. The described procedural algorithmic approach outlines the steps to ensure optimal drone placement, in terms of coverage, for seamless network connectivity. Through the efficient use of isometric grid points, distance matrices, MSTs, novel pruning and redundancy elimination techniques, the solution ensures that all the traffic Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 85 / 195 sources will be served from at least one access point. Moving into the heuristic aspect of this research, the study expands its scope by using Genetic Algorithms (GA) for the serving of multiple traffic sources (TS), as shown in Figure 5-6. This approach, more than just a test but a thought-out exploration, complements the systematic methods mentioned before. The GA, designed to handle various challenges like trust levels, energy needs, bandwidth limits, and related costs, provides a flexible solution, since it can smoothly adjust to changing environments and capture the core ideas of scalability and adaptability. In essence, flexible topologies not only address the need for dynamic connectivity in remote areas, but also ensures that considerations such as infrastructure costs, energy consumption, trustworthiness, and sustainability are at the forefront of the solution. This paradigm-shift in network design and implementation, with its systematic and heuristic approaches, stands as a testament to the transformative potential of 6G telecommunications in addressing the plethora of use cases of the next generation of networks. 5.1.4 Summary Apart from enabling a seamless and ubiquitous communication system, this enabler can aid the reduction of complexity in certain devices and network nodes, by offloading some of their responsibilities to other more capable nodes, and also improve the connection reliability (e.g., via using NTN as anchor cell in certain scenarios). As shown in Figure 5-1, this has implications on the RAN signalling and procedures and hence new or enhanced methods for the communication between non-terrestrial nodes, as well as between a device, a terrestrial cell and a non-terrestrial cell. Similarly, the roles and responsibilities of each node in a subnetwork will also affect the RAN procedures on both the devices and the BSs. Moreover, a methodology for a trustdriven intelligent selection of nodes for formulating a flexible topology, which could either be network-centric or device-driven, should also be proposed. Table 5-1 summarizes the main benefits and implications of the network of networks enabler. The use case families “fully connected world”, “immersive experience” and “collaborative robots” [HEX223D12] may be enabled by the network of networks. The fully connected world use case family aims to provide network access everywhere, including remote areas with difficult access, airspace, and oceans [HEX223-D12]. The service coverage extension offered mainly with the use of NTN and the NTN-TN dual connectivity, as well as with subnetworks, may enable certain services of this use case. Immersive experience may involve multiple devices of various capabilities and/or owners, which may be co-located. Coordination between the devices and with the overlay 6G network can be achieved via the use of subnetworks. Collaborative robots will require the ad-hoc formation of trustworthy flexible topologies, which is part of the network of networks enabler. Table 5-1 Benefits and implications of "Network of networks" enabler Description Design of terrestrial subnetworks and NTN to create a seamless and ubiquitous communication system. Benefits KPI improvement Increased coverage, reduced interruption time and time in outage, increased availability and reliability, reduced complexity, enabling service continuity. Design principles [HEX223-D21] The flexible topologies of the terrestrial subnetworks and NTNs contribute to the flexibility to different network scenarios (#3 design principle in [HEX223-D21]). Moreover, NTN and terrestrial subnetworks will improve coverage, which fits in higher availability and resilience (#5 design principle in [HEX223-D21]) Dependencies / Basis for another enabler NTN-related mobility and subnetwork-related mobility and flexible radio protocol depend on this enabler. Both low-cost and highcapability devices could use the framework of subnetworks, where certain functionalities are offloaded to the MgtN. Implications Requirements The ISL solution for 6G NTN, e.g., modified IAB or any other multihop solutions depends on which NTN RAN architecture is selected for 6G NTN (e.g., gNB onboard, RU onboard). Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 86 / 195 A TN-NTN dual connectivity procedure should be introduced in order to switch faster to NTN in the cases where there is no good TN coverage. The new UE roles and their responsibilities in a subnetwork, as well as the coordination with the overlay network, will have an impact on both the UE and the RAN functions. Two architectural options are foreseen for subnetwork: transparent to the NW and non-transparent to the NW. The latter resolves the challenges of the former but requires the definition of the MgtN-NW procedures. A new lightweight subnetwork CP between the MgtN and the UEs in a subnetwork can be introduced. The trust of diverse network nodes may be device-centric or networkcentric. Standard relations & regulations NTNs should improve coverage. However, the notion of coverage needs to be revisited for 6G with the new diverse services and requirements and other related KPIs. Regulators will have to develop new measures to follow up and regulate national coverage. Various new architectures imply new needs to measure coverage. On the topic of TN-NTN DC, the measurements reported by the UE or evaluated by the network should provide sufficient information such that TN to NTN switchover is done in a timely manner. These procedures would impact the RAN and could be defined in 3GPP RAN2. The architecture of the subnetworks, the new UE roles and their responsibilities in a subnetwork could be defined in 3GPP RAN2. Required resources For NTN-TN DC for coverage enhancement, edge compute resources will need a migration from a node closer to a TN base station to an NTN base station/earth station in order to reduce latency. 5.2 Multi-connectivity 5.2.1 Introduction The previous deliverable [HEX223-D32] stated that Multi-Connectivity (MC) enables the aggregation of different carriers and Radio Access Technologies (RAT). CA is used for increasing both the user and the system throughput. DC usually employs either different Frequency Ranges (FR) or different RATs to increase either the user and system throughput or the robustness and reliability of the system, by offering multiple OverThe-Air (OTA) paths for transmitting data. The evolution of CA/DC from 5G to 6G aims to combine the best features of DC and CA to avoid the complexity of having two similar solutions, and instead focus on one 6G MCV solution. This enabler also targets to decrease the complexity of activating carriers of different FRs, which have an inherently high-power penalty because of monitoring the different FRs. In parallel, the enabler also investigates the aggregation of different access networks, such as Wireless Local Area Network (WLAN), to enhance coverage while still providing the increased reliability of having multiple OTA connections. The benefit of a device having simultaneous connections with multiple network nodes is evaluated with a study on how to manage communication and computation resources more efficiently in such scenarios. 5.2.2 Architectural implications 5.2.2.1 CA/DC evolution In [HEX23-D53], a new 6G multi-connectivity solution was proposed, cf., Figure 5-6 for a high-level view. The aim with this 6G MC proposal was to combine the best features from CA and DC to provide both extreme reliability and excellent flexibility, as well as simplify the solution by reducing the number of architecture options. The proposed solution combines the best features from CA and DC. The new solution aims to decouple Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 87 / 195 Downlink (DL) and Uplink (UL) (e.g., two DL connections and one UL connection, see Figure 5-6) and inherent use of in-active connections. For the in-active connections, the UE only need to sparsely monitor the control signalling from the network. In addition, the in-active connections should be able to be activated on a short notice. To increase the robustness of the system, there is a need for a more flexible use of the UL so that the SCell may take over the role of control signalling in the UL. The CA/DC evolution should also take into consideration the introduction of flexible topologies (e.g., NTN and subnetworks) of the network of networks enabler. As part of the CA/DC evolution, faster addition of cells compared to 5G would be beneficial. Early Measurements Reports (EMR) was introduced in 5G by 3GPP to allow for proactive UE measurements on NW preconfigured LTE/FR1/FR2 frequency layers, which would be ready for potential reporting once it transitions from Idle to Connected mode. The NW can use the measurement results provided by the UE to configure it with an SCG (i.e., active PSCell) and/or one or more activated/deactivated SCells, resulting in reducing the CA/DC setup delay significantly (i.e., from >500 ms to ~50-70 ms). However, this would require the UE to keep measuring the configured frequency layers (i.e., at least for 10s-300s) in Idle mode, deeming the EMR feature too power hungry (i.e., especially for FR2 layers if FR2 PCell is not supported), hence not gaining much traction. Figure 5-6 Proposed 6G multi-connectivity solutions overview [HEX223-D32]. An enhanced mechanism for PSCell/SCell addition when transitioning from Idle mode to Connected mode could be introduced. With this mechanism, the UE may perform measurements during Idle mode of specific, pre-configured PSCells and not on all frequency layers. Note that both the decision to perform measurements and to report them are based on the UE’s internal logic. During the random access procedure, the NW could then directly setup an SCG based on reported measurements by the UE, allowing for faster DC setup. Section 11.3.1.1 details an example of a fast addition of a PSCell or SCell when transitioning from Idle mode to Connected mode. This mechanism could be combined with the feature where the NW configures the UE with a set of PSCells during RRC Setup, where only one of them is activated right away (e.g., the first in the list, or indicated via a MAC CE) and the rest are deactivated. The concept of multi-server offloading (cf., Section 11.3.1.2 for details) can be particularly benefited from the advancements in next-generation multiple access techniques and the support of radio resource sharing among multiple concurrent user transmissions [LZM+22]. To effectively treat interference under a multi-user multiserver communication scenario, different users can utilize different frequency bands to accommodate their computation task offloading. However, a single mobile user’s concurrent transmissions to multiple Multiaccess Edge Computing (MEC) servers can be performed over the same time and frequency resources by employing a NOMA technique, such as the power-domain NOMA and the emerging Rate-Splitting Multiple Access (RSMA) techniques [LZM+22]. At the same time, however, this implies that MEC's performance is majorly interwoven with the radio resource allocation, and thus should be studied jointly. Typical optimization Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 88 / 195 objectives constitute the minimization of the total multi-user multi-server network’s time and energy overheads due to both communications (i.e., offloading) and processing locally at the user devices and remotely at the offloading MEC servers. Also, typical optimization variables and controllable parameters comprise the users' task offloading decision, subchannel assignment, uplink transmission power, and allocated computing resource by the respective MEC server. 5.2.2.2 Multi-connectivity with different access networks Different access networks could be integrated with 6G cellular networks in the RAN by using adaptation protocol layers or in the transport layer by using network virtualization and programmability. In this section, architectural options for each option are described. Figure 5-7 WLAN-Cellular Aggregation (WCA) where the UE and the WLAN terminal are connected to different BSs with different frequency ranges. Focusing on integration in the RAN and using WLAN as an example of a different access network, one of the main motivations for an integrated WLAN and 6G solution is the indoor use case (e.g., in-home), where a trusted WLAN Terminal (WT) is within cellular coverage, which could be leveraged to support the UE. Both the UE and the WT could have a wireless connection with the BS, while the UE and the WT also have a wireless connection with each other. The WT-BS link may use a different FR than the UE-BS link, therefore different resources may be assigned to the UE and to the WT. An example of such a system is depicted in Figure 5-7. The proposed WLAN-Cellular Aggregation (WCA) framework can aggregate the direct UE-BS path as well as the UE-WT-BS paths on RAN level at the BS where the radio bearer originated from. Another use case of interest is when a UE has no cellular coverage (i.e., remote UE), but is within WiFi range of a WT, which in turn is within cellular coverage and may act as a relay for the remote UE. The current and predicted usage of various QoS flows and Radio Bearers (RB) necessitate the exploration of how this framework would be integrated in a WT. The BS may configure the UE to transmit the same packet (i.e., packet duplication) or different packets (i.e., packet split) over the UE-BS and the UE-WT-BS paths. Packet duplication would increase the reliability of the connection, while packet split increases the throughput. Aggregation takes place below the IP layer, which means that the WT does not have an IP connection with the cellular network for the sake of relaying the UE’s packets from/to the cellular network, hence ensuring privacy. In addition, the UE would only utilize the trusted WT and involve the cellular network, only when the UE decides to enable this feature. There are benefits for the cellular NW to share radio resources between the WT and the UE instead of allocating all radio resources to the UE. For example, the WT may have more powerful cellular capabilities than the UE, it may be connected to power, and it may be more stationary, allowing for a better connection in higher frequency ranges, where beam management is required. At the same time, the UE could save more power by offloading data to the WT instead of directly transmitting it to the BS. Note that data transmitted via a public Uu SCG ngNB FR2 WLAN Terminal (WT) WLAN Interface Uu UE CN MCG ngNB FR1 Xn Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 89 / 195 (i.e., not trusted) WT shall be protected via cellular security. Nevertheless, it should be investigated if and how the UE may save power when transmitting data only via a private (i.e., trusted) WT. At the same time, the UEBS link would still be required for achieving higher reliability via packet duplication and for controlling and configuring control plane procedures such as mobility and WCA-related configurations. The LTE-WLAN Aggregation (LWA) [36.300] and the LTE-WLAN Radio Level Integration using IPSec Tunnel (LWIP) [36.300] features aggregated and integrated, respectively, LTE with WLAN. In the next deliverable, more details on WCA will be provided, as well as a comparison with LWA and LWIP. Shifting the focus to integrating different access networks in the transport layer, the concept of a dynamic federation of loosely coupled domains (technological or administrative ones) in the context of 6G is introduced. The proposed approach uses network virtualization and programmability to dynamically connect self-managed and self-operated domains with embedded intelligence that supports this operation. The main idea lies in using inter-domain gateways to provide uniform domain interactions. The approach is an evolution of the 6G-LEGO concept [KTK+21]. The main idea is to use abstracted protocols (intent-like) between different networking domains. To use this approach, each domain has CP and UP gateways responsible for translating the abstracted messages into domain-specific ones. Each domain service and respective interacting protocols must be defined. In addition, a set of functions responsible for inter-domain level operations, i.e., discovering, adding, or removing a domain, has to be specified to implement the approach. Figure 5-8 Simplified view of the dynamic federation of loosely coupled domains concept. The main features of the proposed framework, outlined in Figure 5-8, are described below. More details can be found in Section 11.3.1.3. • The solution is composed of autonomous modules called Functional Domain (FD). An FD can be a complete networking solution, a network segment or a set of networking functions that can be used for building a complete networking solution, e.g., WiFi or any other type of network. • FDs can be dynamically federated, creating a Federation of Functional Domains (FDF). Depending on the goal, FDs can be aggregated (federation of FDs of the same technology), integrated (federation of FDs of different technologies) and chained. • The interactions between FDs use high-level abstractions (intents and KPIs) to enable flexible FDF reconfiguration and to avoid complex end-to-end protocols. • The proposed framework uses the UP, CP, Management Plane (MP), Resource Plane (RP) and Federation Plane (FP). A set of high-level services should be defined for each plane, as the Federation of FDs is made on each plane and its services level. FD does not have to have all planes, similarly to the concept presented in [KTO+18]. The FP is a concept-specific plane used for federation-related operations only. • The framework includes framework-specific functions (FD internal functions, Federation Dedicated Functions and System Common Functions), plane-dedicated functions, and message buses used for interactions between FDs. Such an approach simplifies intra-FD and inter-FD interactions and enables the easy addition of necessary functions to FD or FDF. Functional Domain #2 Management Plane G-CP G-UP G-UP MP FP RP G-CP Functional Domain #3 G-CP: Control Plane G-UP: User Plane MP FP RP Functional Domain #1 G-UP MP FP RP G-CP Federation Plane Resource Plane Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 96 / 195 5.3.2.2 Semantic end-to-end system optimization Flexibility and semantic context of the applications are crucial to minimize network congestion and optimize end-to-end performance. The concept of Semantic RAN introduces the idea of flexible application allocation, allowing applications to be processed either locally on the user equipment, offloaded to the edge server, or a combination of both. This leads to the development of an orchestration algorithm that optimizes offloading policies, radio, and compute resources jointly to meet performance requirements, considering factors such as data compression, accuracy, network latency, battery life, and time-sensitive resource allocation to meet endto-end latency bounds. An example of such applications is the collaborative mobile robots use case, where the semantic context of the robotic task can be used in order to meet the strength QoS/QoE requirements while optimizing the available resources in the end-to-end system. The architectural modifications for having such a semantic solution includes integrating two main components in the ORAN architecture: the Semantic Deep Learning Analyzer (SDLA) and the Semantic Edge Slicing Module (SESM). The SDLA enhances network resource allocation by learning and interpreting semantic descriptions of application tasks. Meanwhile, the SESM solves the slicing algorithm, which allows for managing the edge and radio slicing at the edge and choosing the best fitting offloading policy. To ensure the proper functioning of these components, the radio, edge, and device status is monitored. The radio status involves measurements used to calculate the latency, e.g., signal strength or interference levels, the edge status regards context about the available resources to the SESM, e.g., CPU and memory usage, or computational load, and the device status checks the battery levels, current workload, and sensor information to select the optimal offloading policy. In the following, the O-RAN standard architecture is covered, exploring, and showcasing how these two components should be integrated to ensure efficient performance of mobile robot applications. A similar study regarding the ETSI-MEC architecture can be found in Section 11.3.2.2, along with a special case focused on the harmonization of ETSI MEC with the 3GPP-5G architecture. The integration of the SDLA and SESM components into the ORAN architecture is presented in Figure 5-14, while the specific steps that take place are as follows: 1. Petition for an instantiation of a robot task. The corresponding petition comprises a task descriptor of the robot task, which includes the deep learning model to be executed and its execution requirements, such as latency and accuracy, using an O-RAN Slice Request (OSR). The OSR is sent via a Human Machine interface from a Virtual Network Operator (VNO). 2. Semantic Analysis of the task. The SDLA uses the Non-Real-Time RAN Intelligent Controller (RIC) infrastructure to compute the corresponding latency and accuracy functions with the information gathered from the Radio Status (via O1 interface) and the task descriptor. These functions are then shared with the SESM (through A1 interface) at the Near-Real-Time RIC. 3. Semantic Edge Slicing of the task. The SESM determines how to accomplish the robot task by choosing between all available policies and identifying the necessary slicing requirements for radio and compute resources. For this purpose, the SESM uses as inputs the latency, accuracy, and battery function, the requirements of the tasks, and the metrics gathered from the Radio, Edge, and Robot status services (using interface E2). The computation and radio slicing are shared with RAN Edge and the CU using E2 interface. Additionally, if the task is offloaded, the Policy and a Compression Factor for data stream are shared with the VNO, which then communicates them to the robots. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 97 / 195 Figure 5-14 Integration of the SDLA and the SESM components in the O-RAN Architecture 5.3.3 Evaluations 5.3.3.1 Multi-domain SDN scalability evaluation The split of the SDN transport into multiple domains improves SDN scalability and makes the problem of the SDN controller placement less critical. However, such an approach raises inter-domain issues. In the multiSDN approach, path calculation (algorithmic delay) is shorter due to a smaller number of domain switches compared to the number of all switches in the network and path execution (deployment) – can be done in parallel in each SDN domain. The path computation time can be further improved using efficient path computation algorithms (see complexity details in Section 11.3.2.3) at domain and network levels. Figure 5-15 Path setup time dependency on the number of SDN domains To evaluate the improvement caused by the multi-domain SDN, simulations have been carried out concerning path setup time (i.e., path calculation and path deployment) for the same number of nodes but different numbers of SDN domains. In the experiment, 80 OpenFlow switches were divided into 1 to 20 equally sized domains. The network had 118 links and 20 hosts. The basic Dijkstra algorithm was used for route computation, and 100 x 0.5 Mbps data flows were generated using iperf3. As Figure 5-15 shows, the multi-domain approach provides a shorter path setup than a single-domain approach - in the case of 4 domains (20 switches per domain), the gain is about 60%. In the case of 16 domains (5 switches per domain), the average path setup time is about five times shorter than in the case of a single domain. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 98 / 195 Further reduction of domain size due to the small number of switches provides no significant improvement. The results show that the multi-SDN approach shortens the path setup time. 5.3.3.2 Autonomous robots As described in [HEX223-D32], the Semantic RAN concept is built upon the following three assumptions: (a) different Compression Factors can be used for efficient data transmission without sacrificing the performance of the task, (b) different offloading policies can meet the accuracy and latency bounds while considering the robot status, and (c) different Radio Resource Block configurations can meet the latency bounds. Figure 5-16 (a) Compression Factor of 100 using YOLOX-Tiny Figure 5-16 (b) Compression Factor of 1 using YOLOX-Tiny Figure 5-17 Number of detections per compression factor using YOLOX-S To validate the latter assumptions, an object detection experiment is performed using two YOLOX [YOL+21] models: YOLOX-S and YOLOX-Tiny. The YOLOX-S is a larger and more detailed model, designated to operate in a server equipped with a GPU, representing a high-capacity computational scenario. In contrast, the YOLOX-Tiny is a more compact model that runs on a laptop without a dedicated GPU, simulating a more resource-constrained environment as the case of a robot. Both models were connected to a camera. This input video flow is modulated using a Compression Factor. Adjusting this value from 100 down to 1, allows to gradually decrease the camera resolution. The same evaluation is performed in two different contexts: in a laboratory full of different classes to be inferred and in an empty corridor, where typically there are no classes to be inferred, rather than people sometimes crossing by. A more detailed explanation of the experimental analysis can be found in Section 11.3.2.4. In the following, the first two assumptions are covered, while the Radio Resource assumption will be considered in future work. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 99 / 195 In Figure 5-16 (a) and Figure 5-16 (b), different values for the Compression Factor are examined. As can be seen, the YOLOX-Tiny model with a large Compression Factor, effectively distinguished little objects like a cup and bigger ones such as a person. In contrast, when the Compression Factor was lowered to 1, the model failed to detect the cup but still identified the person. It is noteworthy that the different classes detected depend not only on the Compression Factor but also on the model itself, the training dataset, and the distance from the object. In this experiment, the model was trained with the COCO Dataset, which contains many person samples, and the picture was taken no more than a meter away from the inferred objects. It is therefore validated that different Compression Factors can lead to similar accuracy bounds without sacrificing the performance of the task. Figure 5-18 CPU Usage under the two different contexts Figure 5-17 demonstrates that for different Compression Factor values, the model can detect a varied number of objects, which decreases drastically from 50 down to 1. In this experiment, the YOLOX-S model was used, but the case is the same happens if the YOLOX-Tiny model is selected. Figure 5-18 shows the CPU Usage at the robot laptop for the two different contexts, as previously described. The box plots lead to the conclusion that in a room full of objects, there will be higher CPU Usage, which will result in more battery consumption for the robot. Thus, it is confirmed that different offloading strategies must be considered to achieve the accuracy and latency bounds while being energy efficient. In contexts where a detailed inference is needed, the semantic RAN should choose the most appropriate model for the use case. 5.3.3.3 Delayed computing Consider a network architecture where multiple computing layers—namely, edge, fog, and cloud—collaborate seamlessly to provide transparent computing services to subscribed users. The computing layers differ in computational power, resulting in different response times for the subscribed users. In this context, the variety of tasks and user application requirements create an opportunity to employ different computing options. Motivated by the latter and aiming to optimize resource utilization, the delayed computing paradigm suggests users leverage their delay tolerance and price sensitivity. This flexibility enables intelligent task orchestration across the computing continuum, reducing service costs or subscription fees through incentive mechanisms. In the studied delayed computing scenario, incompleteness of information arises from the a priori unknown characteristics of the offloaded task to the service provider, specifically in terms of computing intensity and service delay requirements. To address this issue, contract-theoretic modelling can be adopted for effective incentive mechanism design. Specifically, Contract Theory [BD04], belonging to the domain of Labor Economics, provides the theoretical foundations to construct mutually agreeable contracts or arrangements among economic players in the presence of incomplete information between them. Following the principles of contract-theoretic modelling, the service provider utilizes existing datasets about the potential characteristics of the offloaded tasks and distinguishes the users into different types. Then, based on this statistical knowledge, the service provider designs a menu of contracts tailored to each prospective type. The contract comprises a subscription fee to the service and a corresponding response time approximation. Subsequently, each user Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 100 / 195 autonomously selects the one contract out of the menu that best fits its type. By intelligently managing the trade-off between payment and service response time, users can be incentivized to choose a less restrictive contract in terms of response time, allowing for the network’s flexibility to orchestrate across the continuum. The preferences and motivations of both the service provider and users during the design and selection of contracts, respectively, are captured by a utility function. A common representation of the utility function is the difference between gain and cost. The service provider's gain can quantify resource utilization efficiency across the network, while the cost can be associated with the subscription discount offered to users. Conversely, the monetary reward can represent users' gain, with the cost corresponding to the experienced service delay. To determine the menu of contracts, the service provider solves an optimization problem to maximize its utility while securing the users’ participation in the contract, i.e., by guaranteeing their non-negative utility. 5.3.3.4 Context-aware connectivity for maritime ports This section encapsulates key findings from a study on enhancing connectivity for smart maritime ports. Data can be harmonised/standardised in various aspects of port operations, such as traffic flow, cargo handling, and supply chain management. The integration of 6G and 5G as well as Wi-Fi technologies can ensure the medium for the exchange of information, and the use of IoT can integrate other technologies that will allow an answer to other challenges for port management. Protocols like TSN can help combine different types of traffic in the same network, making best-effort networks more predictable. The combination of TSN and AI is studied to perform predictive maintenance of the network, anticipating possible faults and degradations, taking into consideration the historical context and status of the network. This can be an important tool for enhancing the performance and competitiveness of the port networks, allowing them to better meet the needs of shippers, carriers, and other stakeholders in the global logistics industry. For the TSN network simulation, available simulators such as NS-3 will be used. The demanding connectivity requirements from some applications in maritime ports, like real-time monitoring of vessel movements, automated cargo handling, and supply chain optimization, put additional stress on local networks covering the port location. Key performance indicators have been identified, such as AI/ML-related capabilities, E2E latency, and reliability to gauge the success of connectivity enhancements in maritime ports. 5.3.4 Summary The implications of the mechanisms involved in this enabler are that different system components e.g., RAN, transport, management and orchestration modules should interact with each other and become aware of the context. This means that new signalling and synchronization is required. Moreover, the novel resource allocation and orchestration mechanisms should also be operational and effective even when incomplete or partial context awareness is available. The mapping of this enabler to the 6G system blueprint is illustrated in Figure 5-1. Table 5-4 summarizes the main benefits and implications of the E2E context awareness management enabler. The use case families “cooperative robots” and “physical awareness” [HEX223-D12] may be enabled by E2E context awareness management. With the aim to improve both the applications and the communication’s performance, cobots are expected to take advantage of contextual information (e.g., RAN measurements, delay requirements, computation resources). Based on the aforementioned context, appropriate task allocation, as well as communication and compute resource management may enable this use case family. Physical awareness would require context-aware communication in the form of path selection and would benefit from the transport network abstraction. Table 5-4 Benefits and implications of "E2E context awareness management" enabler Description Mechanisms to allow network components to dynamically adapt to the context to ensure the expected E2E QoS. Benefits KPI improvement Reduction of the network overhead and flexible allocation of edge resources, ultimately improving the system performance by allowing multiple edge allocations and RAN slices. Optimal energy consumption of the end devices Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 101 / 195 Design principles [HEX223-D21 Effective and optimized use of the network infrastructure resources, as well as personalized and dynamic resource allocation improves the flexibility to different network scenarios (#3 design principle in [HEX223-D21]) Dependencies / Basis for another enabler In the existence of sensitive datasets, privacy preserving techniques and split neural networks should be used to perform cross-network function distributed ML model training to fuse the context model with other network models without necessitating raw data transfer. Implications Requirements Different network components e.g., RAN, CN, transport, should become aware of the context and need to interact, implying the need for signalling and synchronisation. A resource orchestrator would have to create an abstracted view of transport resources and to employ a software defined transport controller for resource management, ensuring that the required QoS is met. Effective resource allocation and orchestration mechanisms that operate even when incomplete or partial context awareness is available should be designed. In a mobile robotic context, diverse system elements must be exposed for the effective resource allocation and system orchestration. Thus, a correct resource exposure of the robot status should be ensured. Automatic translation of requirements from one component to another, according to the peculiar characteristics of each component would be required. In addition, harmonization of optimization mechanisms in each component is needed, to guarantee an efficient E2E interworking. Finally, a suitable abstraction and exposition of the capability of each component to facilitate the E2E managing of the network components according to the context is a prerequisite. Standard relations & regulations In the context of task allocation in semantic RAN, the ETSI MEC architecture [MEC035], the harmonization with 3GPP [ETSI36] and the O-RAN architecture [OAD24][OSA24] may be impacted. Required resources Cloud computing, edge computing and extreme edge devices Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 102 / 195 6 Architectural enablers for network beyond communications As stated in [HEX223-D32], the evolution of the network is pushing the boundaries, beyond conventional connectivity, into accommodating and supporting novel services, expanding the network’s scope by processing data, generating insights, and delivering added value from societal, innovation, and business perspectives. Examples of new services comprise sensing, enhanced localization and tracking, compute-as-a-service, etc [HEX223-D12]. In the initial analysis provided by [HEX223-D32], four enablers have been identified towards introducing Beyond Communications Services (BCS), namely Enabler #1 - Exposure and data management aspects; Enabler #2 - protocols, signalling and procedural aspects; Enabler #3 - Applicationand device-driven optimisations, and Enabler #4 - Enhancements of BCS capabilities, such as JCAS. In the context of this chapter, those enablers are further analysed; the sub-chapters have followed the above structure with the Enabler #2 split into two sub-chapters to further elaborate on the two primary directions of this enablers, i.e., JCAS and Compute offloading services (into sub-sections 6.2 and 6.3 respectively, while Enablers #3 and #4 are presented under the common sub-section 6.4 in order to better illustrate the architectural implications). This chapter examines the requirements, architectural implications, and foreseen benefits in relation to novel data exposure and management, protocols, signalling procedures, as well as network and application function placement towards service and QoE enhancement for users. In order to provide further insights, each architectural enabler is mapped to the 6G E2E system blueprint of Figure 6-1 [HEX223-D22]. This mapping aims to suggest the implications of the proposed enablers being part of the E2E system, in terms of layer functionality and respective interfaces. Figure 6-1 Beyond communication enablers potential mapping to the E2E System Blueprint 6.1 Exposure and data management 6.1.1 Introduction As we move towards the establishment of the next generation networks, efficient data management and strategic exposure are fundamental for leveraging the network's capabilities and for optimizing network functionality and enhancing capabilities, with particular attention to the secure and efficient exposure of data. This is essential for handling the extensive information that may be generated by the network's diverse devices, comprising RAN infrastructure, user equipment, radio sensing nodes, or other IoT devices. Importantly, the integration of JCAS represents a critical evolution, differentiating by its dual-purpose technology that enhances both data transmission and environmental sensing, thus optimizing the network's operational efficiency and situational awareness. The focus of this enabler is on data generated both by the RAN infrastructure (radio Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 103 / 195 sensing-related data), as well as on application layer data generated by various devices such as wireless sensors, cameras, robotic platform modules, etc., along with the interfaces towards network functions or service components that may be residing in other segments of the network or compute continuum. The placement of such functions and components is thus critical, as numerous aspects related to data privacy, communication and processing latency, reliability and other critical KPIs are affected. Placement of the various application components, e.g., micro-services in (multi-domain) cloud-native environments, and/or PNFs/VNFs, comprising end-to-end (vertical) application functionality involves strategically positioning those components to optimize performance, balance network load, and ensure data privacy and security. This placement is connected with KPIs such as data throughput, reliability, and cost, which are vital for assessing the efficacy of the network's data management strategies. 6.1.2 Architectural implications The architectural landscape of 6G will most probably be influenced by the exponential rise of IoT devices and the emergence of novel applications which leverage BCS capabilities, such as sensing, precision positioning, compute offload, etc. As the network evolves to accommodate a wider range of devices, from sensors to broadband devices, the challenges in such beyond communications data management and beyond communication services/data exposure become more pronounced. Such challenges mainly include (a) interfaces that support data collection, (b) data processing, (c) data distribution & scaling of interfaces, (d) trust differentiation when exposing to 3rd party applications, (e) network overload on the exposed Application Programming Interfaces (APIs), (f) privacy risks, and (g) latency/performance challenges. Devices with BCS capabilities introduce a unique set of data streams, distinct from the conventional user plane (UP) and CP data. These streams necessitate an architectural paradigm shift, creating an entirely new stream of data which, while resembling UP data, is not tied necessarily to a specific user. From an architectural standpoint, the core-RAN continuum's role in aggregating, processing, and exposing data becomes pivotal. A careful balance must be struck between ensuring data integrity and meeting QoS requirements for both small data packets from power-limited sensors and high-volume data streams for UEs and broadband services. This balance is further complicated by the trustworthiness of data exposure. In the 6G context, data is not only collected but also cleaned and labelled at various network locations. This process necessitates robust architectural enablers that prioritize security, privacy and trust, ensuring data protection throughout its lifecycle. Furthermore, the exposure and data management enablers are critical to the effective implementation of the services such as JCAS. These enablers facilitate the interaction between network functions, services, and thirdparty applications. Architecturally, this means a more intricate data flow, with data being exposed to both innetwork and external entities. Such exposure poses challenges, including trust differentiation when exposing to different network sub-systems (e.g., RAN), when exposing to third-party applications, potential network overload on exposed APIs, and the strategic placement of applications leveraging sensing functionalities to meet stringent QoS targets. Sensing solutions are essential for an authentic digital representation of the physical world (i.e., digital twin DT) enabling new human experiences through immersive mixed reality digital worlds. DT covers a complete view and the full lifecycle of the physical system, unlike simulators, which only focus on parts or several parts of it. The utilization of real-time data from diverse sources is a fundamental aspect of DT [HEX223-D12]. In the context of 6G, the integration of JCAS mechanisms serves to enhance the digital twinning capability. In this regard, data collection, data fusion, and data analysis are essential tasks for the generation of DTs. More specifically, identifying the sets of 6G nodes that are the most pertinent and provide the highest level of contribution towards the establishment of a precise DT would help avoiding unnecessary data collection and redundant sensing, saving network resources. 6G will play an important role in acquisition and fusion of data as well as rendering the digital twin. The network should assure that the latency of the data flow between the real network and its DT is kept in check to ensure timely ML model training and inference. However, due to the expected large amount of data that needs to be communicated between the 6G network DT and the physical 6G system (in real-time in some cases), how to efficiently integrate/interconnect these two entities is a big challenge [NMS+22]. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 104 / 195 Platform requirements for data collection operations shall be provided and dynamic adaptation of platform resources (e.g., storage) and capabilities (e.g., data formats compatible to different compute platforms) to changing requirements shall be supported. Data discovery/ exposure capabilities, enabling data advertisement to network entities that may be interested in consuming the said data together with the determining the said entities shall be supported. Any network entity with proper access rights should be able to access data or model(s), stored from another entity. A form of authorization and/or authentication should be performed when a network entity is trying to access, update or share data, analytics and model from another entity. Security should be enabled E2E for any operation of the data collection services, including access, exposure, storage, cleaning, processing and encoding. The network should be able to identify energy-aware data collection services and facilitate their operations. Data collection and exposure can be based on a local configuration or a configuration received from the requester. Different data consumers exist in the network (defined as general network entity), such as UEs, RAN nodes, CN NFs, AFs, 3rd party applications, OAM, etc. Discovery, configuration and in some cases evaluations of such data sources are among the functionalities to cover in 6G. The network's storage capabilities must ensure that data regarding its operations and services are stored according to quality standards set between data producers and consumers. Storage should be efficiently distributed for various network functions, considering factors like data longevity, sharing limitations, ownership, and technical constraints. Different data types may require separate databases but are generally categorized under a common term "database." Separating control and data planes seems reasonable since the control requests and actual data flows have different characteristics (e.g., small vs. massive data flows). Also, the measurement data may have characteristics that resemble actual user data, however, there are important differences. One possible solution to deal with such large data collection is to take advantages from a data plan, i.e., a data plane to carry sensing measurement data and other large data sets within the network has a number of benefits, e.g., dedicated hardware can be used to improve performance of the data plane. 6.1.3 Preliminary workflows and evaluation As long as the position of gNBs and UEs are known to the operator, this information can be used for sensing and enable some kind of QoS-based sensing. However, if this is not the case, it is likely that sensing services will have to be provided by the network on a best-effort basis. One way to improve sensing quality is obviously to carry out more radio measurements prior to exposure of the measurement report to the requesting application, preferably measurements that are geographically distributed. Involving for instance more UEs in the sensing and measurement process is likely to improve sensing quality. Involving more network nodes (UEs or base stations) naturally leads to architectural challenges of centralized vs distributed inference and processing of the measurements. Sensing in the context of Hexa-X-II WP3 refers to the network inferring information about the physical environment (for instance detect an object in a specific area of interest) from radio measurements. This inference is likely to be carried out in a staged process within the architecture. The outermost measurement node will acquire raw radio measurements which, on one hand, it could transfer directly to a more centralized network node or, alternatively, it could pre-process and compress and already carry out initial stages of the inference. As stated in the previous section, the core-RAN continuum's role in aggregating, processing, and exposing data becomes pivotal. Not only from an architectural standpoint (as addressed above) but also from a signal flow and data volumes, and sensing performance standpoint, as addressed here. Figure 6-2 shows two canonical architectures that clearly illustrate the architectural challenges (Here, we address challenges related to the signal flow and volumes, and sensing performance, but also challenges related to privacy and robustness against attacks are relevant in these canonical architectures). In a first architecture, the measurement node provides the final (exposed) measurement report directly from the radio measurements it carries out. Inference of the requested geographical information fully carried out locally in the measurement device: signalling compression to the deeper network layers is maximal, signalling overhead is minimal, and the sensing performance is likely to be modest at best (in some measure). In a second architecture, the measurement node Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 105 / 195 merely transfers its measurements to a centralized network node, where inference of the geographical information is carried out based on measurements of a massive number of nodes. In this architecture, no data compression to the deeper network layers is done, signalling overhead is maximal and often prohibitively large, but the sensing performance is likely to be best. Figure 6-2 Canonical forms of sensing. Top: Inference of geographical sensing information from radio measurements is carried out in the node where the radio measurement takes place (UE or BS). Information transfer to the Application Function (AF) occurs in finally exposed form. Bottom: Radio measurements are transferred as raw radio data to the AF, where inference of geographical sensing information takes place. 6G is likely to comprise of a design where some trade-off of the above canonical ways is done in a staged process of inference of sensing information. While sensing is done on a best-effort basis, the network will be able to trade off sensing performance against data signalling overhead. A more centralized inference has to be compared to a distributed design in terms of (primarily) data overhead and performance. One way to address this challenge is to first, define a number of sensing information classes, where the sensing requests from the applications are required into a certain exposure format for each class. In conclusion, the architectural implications of introducing device BCS data and capabilities exposure mechanisms are manifold. They require a holistic approach that not only addresses the increased data volume and computational load but also ensures data integrity, security, privacy, and trustworthiness. As 6G networks evolve, these architectural considerations will play a pivotal role in ensuring the seamless integration of a wide range of devices and applications, driving the next wave of digital transformation. 6.1.4 Summary To enable effective beyond communication service delivery, exposure of data and beyond communication service characteristics and capabilities should be enabled in a secure, privacy-preserving and efficient manner. Also, authorisation and authentication should be provided for such data consumer functions. Architectural enhancements to the E2E system to support E2E data management both in-network, but also towards 3rd parties should be provided. Table 6-1 Exposure and data management summary table Description Exposure and data management: This enabler encompasses various mechanisms that allow the exposure of data generated by various producers (including the RAN and sensing nodes) towards network-centric and application layers; besides exposure - storage, processing, trust are in focus Benefits KPI improvement Controlled data traffic/overhead via efficient data management, higher number of vertical services consuming (B)CS data, higher data availability, low latency (serving the data consumers from the nearest data provider), Novel services exploiting BCS data, 6G contributing to social aspects such as safety, trust, and sustainability increase due to novel services beyond communications (e.g., via JCAS in the connected mobility domain, environmental sensing for urban areas, or applications in the agriculture domain), scalability Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 112 / 195 When a device, acting as an ON, decides to offload a computation, it will have to discover and select the candidate CompNs, capable of performing the requested computation while satisfying the associated KPIs. Each CompN should estimate the task(s) execution complexity and resources demand (i.e., computation and storage) based on a common characterization of the offloaded compute workloads (i.e., compute tasks) determined by the following requirements: • Traffic class (i.e., one-time vs multi-iteration, one-node vs multi-collaborative-nodes) • Computation complexity (e.g., number of FLOPS and memory) • Communication requirements (e.g., size of compute payload to be transferred) • Precision requirement (e.g., quantization level of the compute data and operations) • Quality of compute service classes (QoCS), such as latency sensitive, precision sensitive, … These characterizations serve as the basis for the capability exchange and compute offloading coordination between the different nodes. The general architecture for computational offloading is introduced in [HEX223-D32]. Using it as a foundation for further study, the computation offloading procedure is foreseen as a staged procedure, as illustrated in Figure 6-5. In the first stage (i.e., Node Discovery Phase 1), the compute node capabilities are identified. It is followed by the second stage (i.e., Node Discovery Phase 2) where the request for computation offloading is performed. Finally, the third stage (i.e., Computational Offload Procedure) comprises sending of computing tasks and receiving of compute results. Figure 6-5 Computational Offloading Procedure - High Level Flow. Device offloading enabled via Compute-as-a-Service (CaaS) moves computation from a mobile device to a network node with more suitable compute and storage capabilities, see Figure 6-6. This may reduce the mobile device’s computational needs, heat generations, and power consumption, however at a cost of increased usage of network communication resources. Additionally, CaaS may enable better application scalability, new CPU hungry services (e.g., XR and digital twins), and improved operation times for battery-driven devices. This entails the need for a connection between the device offload functions and the network offload functions. Further analysis is however needed, to conclude on whether such network functionality needs to be standardized or not. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 113 / 195 Figure 6-6 Overview of computational offloading of a common coordination task, e.g., to realize collaborative perception or to save overall bandwidth 6.3.3 Preliminary workflows and evaluation In the architectural implications section above, two different architectural options are presented, with one focused on device offloading enabled via CaaS moving computation from a mobile device to a network node, while the other is focused on a general architecture and procedural flow for compute offload without emphasis on deployment. In [HEX223-D32], dynamic offloading with the aim to offload application functionality dynamically from a mobile device to network embedded computing is proposed. As a continuation to this work, a demonstration on how to perform dynamic offloading and the resulting benefits and challenges are in focus here. For this study, a scenario is of robot cars, trying to map an outdoor or a disaster area, where the device (i.e., the car) needs to analyse the environment with an onboard camera or radar, is used as a reference use-case. For efficient dynamic computational offloading, the runtime execution and the time-critical modules of the application running in the device must be replicated in the network. However, it is not expected that software and hardware platforms will be the same in the devices and the hosts that are handling the offload. To counter this, there are several technologies to package and deliver applications, for instance Virtual Machines (VM), like Java VM, or containers. In this demonstration, the emerging WebAssembly runtimes is used, which is selected due to being specifically portable, lightweight, secure, and polyglot. Furthermore, it is assumed that the application is built in a modular fashion, with well-defined modules representing microservices, objects, or even functions. These offload-able modules, perform a specific task, preferably, with as little interaction with the rest of the application on the device as possible, to avoid the unnecessary use of network communication resources. The offload-able modules are dynamically scheduled for offloading depending on situational changes, during application runtime. This is handled by an offload manager in the network and an offload handler in the device. In this work it is assumed that the device (UE) and offloading cluster (i.e., the network) have a pre-established connection between the offload handler in the device and the offload manager in the network (e.g., via RRC control plane functions), but eventually this should be supported by a discovery service. Additionally, it is assumed that there is a single network node, so there is no need for any compute offloading mobility here. However, in reality CaaS mobility must be supported within the PLMN network, but this is not treated here. In the experiment, the offloading is manually triggered. In a real service, there would be an automated trigger based on relevant status and context from the network, the device, and applications. The purpose of the demonstration is to examine the performance characteristics of offloading, especially the power consumption, execution time and network utilization. Figure 6-7 (left) shows the device power consumption for offloading (remote – in light grey) vs no offloading case (local – dark grey). As can be seen, the power consumption in the mobile device is sharply reduced for the offloading periods, with roughly 50% less power consumption in the remote offloading case. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 114 / 195 Figure 6-7 Dynamic offloading of a CPU demanding operation from the device to the network. The left figure shows the consumed power when there is local (when there is no offloading) and remote (offloading to the network). The right figure shows the device network transmission (i.e., uplink) due to offloading (remote) vs no offloading (local). However, this comes at the cost of the use of network communication resources, highlighted in Figure 6-7 (right), to ensure that the module running in the execution environment in the network is synchronized with the remaining application running on the device. Moreover, in [HEX223-D32], the general architecture for computational offloading is introduced, which is further expanded upon in the above section with Figure 6-6 illustrating the computation offloading as a staged procedure. The messaging exchange foreseen in the node Discovery Phase 1 and Phase 2, are shown in Figure 6-8 and Figure 6-9, respectively. These messages are protocol agnostic, meaning that they just illustrate the messaging exchange flow among nodes, with their exact implementation depending on where the different nodes would be deployed (i.e., NAS, RAN, etc.). Node Discovery Phase 1, illustrated in Figure 6-8, starts with the initial registration of the Computing Node (CompN) and ON within the CCN. Then, CompNs update the CCN with their available compute capabilities. Based on this information, CCN sends the Compute Capabilities Update message (e.g., memory size, computation capabilities, etc.) to the ON. The offloading can be either controlled by the CCN (i.e., the message contains the overall aggregated compute capability of all registered CompNs) or controlled by the ON (i.e., the message contains the compute capability per CompN). Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 115 / 195 Figure 6-8 Node Discovery Phase 1 The messaging exchange in the Node Discovery Phase 2 is depicted in Figure 6-9. In CCN controlled procedure, ON requests a general compute offload request that can be mapped by the CCN to any of its registered CompN(s). Then, the CCN evaluates the requested computation tasks, and, if approved, sends it to one or more CompNs accordingly. In the ON controlled case, ON requests a specific CompN, and CCN only forwards this request to the specified CompN. The CompN then evaluates the compute request by the CCN and sends a Computation Offload Response to the ON. If one or more CompN(s) reject the Computation Offload Request, depending on the latency requirements of the task(s), CCN may request these task(s) from other available CompN(s), or sends a reject in the Computation Offload Response message to the requesting ON. Based on this response, ON follows the Computation Offload Procedure in the third stage, which will be further described in the next deliverable. ON Alt#2: ON Controlled CCN Computing Node(s) Initial Registration Compute Capabilities Update [computation (e.g., FLOPs/s), memory, etc.] Compute Capabilities Update [ CompNID , computation (e.g., FLOPs/s), memory, etc.] CCN aggregates/consolidates all CompN(s) compute capabilities Alt#1: CCN Controlled Compute Capabilities Update [ CompNID , computation (e.g., FLOPs/s), memory, etc.] Initial Registration can be in parallel Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 116 / 195 Figure 6-9 Node Discovery Phase 2 6.3.4 Summary Novel architectural components for compute offloading may have an implication on the network signalling and procedures. This includes discovery of computing nodes, synchronization and coordination of offloading and computing nodes and novel data and compute offloading procedure. Table 6-3 summarizes the main benefits and implications of the Compute offloading protocols, signalling and procedures enabler. Table 6-3 Compute offloading protocols, signalling, and procedures summary table. Description Compute offloading protocols, signalling, and procedures: Architecture enhancements and protocols to support compute offloading from mobile devices to network nodes, without increasing the complexity of the communication protocol and satisfy latency, trustworthiness, power consumption, resilience Benefits KPI improvement Energy savings in the device Increasing application scalability and availability (Extended Reality (XR), digital twin, etc.) Communication and computation latency Privacy/security Power consumption Computation Resiliency Quality of computation/ data accuracy Device complexity Design principles [HEX223-D21] (#5) Resilience and availability (#1) Support and exposure of 6G services IF CCN Confirm: Computation Node Evaluation Computing Node ON CCN Alt#2: ON Controlled Computation Offload Request [ CompNID , computation, memory, latency requirements, etc.] Alt#1: CCN Controlled Request for Computation Offload Computation Offload Request [computation, memory, latency requirements, etc.] CCN choses the “CompN(s)” to which the “Compute Task(s)” are to be offloaded ON based on the available information choses the “CompN(s)” to which the “Compute Task(s)” are to be offloaded IF “response” is “confirm”: Stage#3 Computation Offload Procedure Computation Offload Response [ response , optional capability update] Computation Offload Response [response , optional capability update] Computation Offload Request [computation, memory, latency requirements, etc.] [Mandatory for Alt#1& Optional for Alt#2] CCN evaluates the compute request (e.g., due to updated information on available compute capabilities) evaluates the compute request CCN aggregates CompN(s) responses Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 117 / 195 Dependencies / Basis for another enabler Depends on Exposure and data management enabler, Architectural means and protocols, Integration and orchestration of extreme edge resources in the computing continuum, Multi-domain/Multi-cloud federation Present basis for Application-/Device-specific BCS data consuming functions and Distributed Compute-as-a-Service enabler Implications Requirements There is a need to support a connection between the device offload functions and the network offload functions. If the network functionality has to be standardized need to be further analysed. Identify (RAN) sensing procedure (incl. best-effort procedures, managing sensing QoS, sensing scheduling procedures, sensing admission management, etc.) Identify network signalling needs to support procedures New roles of network components (UE, core network, RAN) Standard relations & regulations 38.331 (RRC), 3GPP TS 23.501 System architecture for the 5G System (5GS) Required resources Compute resources are needed. Can be placed in the edge network (gNB), far edge (UE) or can also be centralised compute resources if the application is not delay sensitive. 6.4 Application-/Device-specific BCS optimisation architectural enablers 6.4.1 Introduction In the broader context of 6G network capabilities, the focus on Application-/Device-specific BCS data consuming functions signifies a major shift in network functionality and service provision. This section examines initially the complexities of these functions, which are essential in efficiently managing and utilizing the substantial data generated by a variety of devices within the network. A key aspect is the transition from conventional data handling to a more refined, applicationand device-specific method. This change is critical for maximizing the capabilities of 6G networks, facilitating them to effectively support a diverse array of applications. At the core of this development is the ability to selectively manage and process data in a manner that is precisely aligned with the specific requirements of different applications and devices. This approach not only improves network efficiency and performance but also elevates data privacy and security, which are of utmost importance in the rapidly expanding digital age. The discussion emphasizes the need to develop advanced algorithms and innovative technologies adept at navigating the complexity and volume of BCS data in 6G networks. These technologies are expected to not only optimize data throughput and reduce latency but also to underscore the importance of sustainability and energy efficiency in network operations. As 6G networks evolve, the establishment of robust frameworks for data exposure and processing becomes essential, ensuring that the network remains resilient, secure, and adaptable to the ever-evolving challenges of digital communication and data management. At the same time, incorporating sensing capabilities into a communication network represents a highly promising domain for the BCS that brings forth numerous possibilities and challenges. There are practical applications aimed at enhancing the network's own performance, as well as captivating scenarios where spatial sensing can be extended as a service to improve the performance of existing applications. Architectural enablers should be introduced that contribute to JCAS as BCS concept, which are essential in efficient sensing of the surrounding environment and in the introduction of the quantum sensing technologies. A key concept that is stressed out in regarding the JCAS are the self-sensing capabilities of the UEs together with the quantumenhanced communication and sensing capabilities incorporated into the network architecture. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 118 / 195 6.4.2 Architectural implications 6.4.2.1 BCS data-consumer application placement optimisations Optimizing BCS data-consumer application function placement requires a flexible architecture adaptable to various service needs. This necessitates a network infrastructure that can dynamically assign application functions, supported by intelligent algorithms. These algorithms must consider computational load, network resources, latency, and privacy to optimize system performance and energy efficiency efficiently. The architecture must ensure energy-efficient registration and connectivity checks for BCS, essential for devices with limited energy. An innovative registration mechanism is needed to minimize overhead and conserve energy, while keeping the network responsive, especially for devices in BCS activities like JCAS. These devices require rapid response in varying states. The architecture should balance effective registration protocols with resource efficiency, addressing both connectivity challenges and seamless mobility. Furthermore, as BCS extends to various data-intensive applications like massive twinning and Compute/AIas-a-Service, the architecture must support vast, seamless data exchanges while maintaining strict privacy and security standards. This requires the integration of advanced data management frameworks within the architecture that not only regulate data flow but also ensure that the data is processed and stored in a trusted environment. The incorporation of new network functions aimed at supporting trust and security is essential. These functions must be embedded within the architectural fabric, possibly through a layered approach that separates critical data paths from general network traffic, thus establishing clear boundaries for data processing and exposure. Further to the above, sustainability of operations is also critical. The architecture should not only be robust and secure but also energy-conscious, promoting sustainability in line with 6G directives for energy efficiency. To achieve this, the architectural design must prioritize not just the operational efficiency of data transmission and processing, but also the lifecycle management of the network infrastructure itself. To this end, it should facilitate the seamless introduction, scaling, and retirement of application functions in response to the fluctuating demands of BCS application/UE requests. Different JCAS applications are associated with stringent requirements. On the one hand, heavy computations need to be performed on the sensed data, coming from different sources, to provide with the information on their localization. On the other hand, JCAS applications could also be associated with delay-strict requirements, where the results are expected to be received in real-time (e.g., stopping a robot machine after detecting a human). In addition, although a far edge cloud is located near the end user and comes with promise of reduced communication delay, it can suffer from scarcity of compute resources which induces high processing delay. The trade-off between network metrics and compute metrics would call for a new approach that enables the Integration of Network and Compute (INC) to perform coordinated optimization. Figure 6-10 illustrates an architecture, where network communication is provided by a CSP (Communication Service Provider), whereas compute resources are offered by cloud providers (edge cloud and far edge clouds). Note that the CSP and the cloud providers could belong to different organizations. The user application (e.g., JCAS application) should therefore be deployed in a compute site with the required compute resource, while ensuring the associated network performance. The general idea of the INC approach is that instead of controlling or optimizing the use of network and compute resources separately, both are considered as part of the same system and governed by common processes. To this end, an INC server is considered to ensure coordinated optimization between network and compute processes. It therefore collects network metrics (e.g., maximum delay and minimum throughput to an edge cloud) from CSP as well as compute metrics from cloud providers (number of available CPUs/GPUs and memory). This would require standard interface to expose such metrics to an INC server. The availability of such metrics to the INC would allow performing an optimized decision on the placement of user applications in a way to reach the requested network and compute needs (note that usually it is enough to just select the compute site, which can select the target compute node locally). The decision would therefore be followed by a request to the selected compute site to deploy user application, and by another one to the CSP to steer the traffic to the selected compute site while enforcing the request QoS. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 119 / 195 Figure 6-10 Integration of Network and Compute (INC) server collects network and compute metric to decide optimized placement of application. From architectural point of view the JCAS can be categorized in two operating modes: bistatic and monostatic. In the traditional bistatic mode, the transmitter and receiver are separated, while in the monostatic mode they are located on the same device. For example, in the traditional bistatic localization and tracking, the base station can accurately localize and track the UE. On the other hand, in the monostatic mode the UE is capable to sense the environment and track moving objects in a so-called joint radar and communications. In addition to the joint radar and communication mode, the monostatic sensing can be performed using a combination of Time-of-Flight (ToF) and Angle-of-Arrival (AoA) measurements. For example, in IEEE 802.1 extracting the Channel State Information (CSI) of the path between an Access Point (AP) can produce accurate AoA estimations. The Fine Time Measurement (FTM) protocol [FTM+16] can offer accurate distance estimations by using the Time of Departure (ToD) and Time of Arrival (ToA) sent between the AP and the client. Combining these two techniques together can give us very accurate estimations of the surrounding environment. A possible architectural approach in monostatic sensing is the concept of self-sensing where the mmWave radios that are integrated in the device primarily for communication purposes can be re-used to enhance the environmental sensing when the optical sensors experience performance degradation (i.e., in case of LiDARs with translucent material.). Figure 6-11 illustrates the concept of self-sensing where a device (i.e., robot) moves and scans freely the indoor environment using dedicated sensors, while the mmWave radios periodically exchange AoA and ToF estimates between themselves (self-sensing) by bouncing the signal in the environment, thus enabling accurate estimates of the target object/material surface. Figure 6-11 Key intuition behind self-sensing The architectural implications of introducing the monostatic self-sensing concept and sensing data exposure are twofold. On one side, they require an architecture and protocols where the radios that perform self-sensing can exchange the sensing information with the infrastructure in a secure and efficient way, not degrading the communication performance. On the other hand, the same radios that are primarily designed and optimized for communication will need to evolve so they can support frequent sensing and addition to the communication. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 120 / 195 6.4.2.2 Quantum-based network architecture optimisations Key aspects of the quantum-based network architecture are quantum sensors for increased precision and sensitivity and quantum techniques such as Quantum Key Distribution (QKD) for secure communication. These enhancements are important for maximizing the applicability of JCAS, facilitating them to efficiently support a various types of use cases. At the same time, focusing on the JCAS as BCS, the integration of quantum-based technologies into the network architecture introduces unique aspects that exploit the principles of quantum mechanics for enhanced communication and sensing capabilities. The network will also benefit from advances in quantum computing to meet the increasing demand for computing power. The architecture may include hybrid layers that integrate classical and quantum communications. Classical layers will handle conventional data transmission, while quantum layers will support quantum communication for specific applications, creating a two-tier network. 6.4.3 Preliminary workflows and evaluation 6.4.3.1 BCS data consumer application-driven optimization In the context of 6G, the optimization of BCS data consuming application function placement considering certain data exposure interfaces from the network side is critical in order to satisfy various QoS (communication service and beyond) requirements. The focus of this evaluation is to analyse and compare different application placement strategies within a simulated network environment. The evaluation leverages a Python-based simulation environment incorporating network graphs, application requirements, and node capabilities, to explore the efficacy of these strategies under varying conditions. Figure 6-12 Simulated environment of a multi-layer network for application placement optimization The simulation environment, as shown in Figure 6-12, models a multi-layer network with nodes representing different layers such as Extreme Edge, Far Edge, Edge, and Core. Each node possesses attributes like CPU, memory, and storage, reflecting a realistic network scenario. Each layer is configured to mirror the increasing resource availability observed as we delve deeper into the network. Starting at the Extreme Edge, node capabilities are limited (CPU capabilities start modestly, ranging from 10 to 200 MHz, paired with memory capacities that vary from 0.5 to 2 GB, and storage options extending from 100 MB to 4 GB), encapsulating the constraints of edge computing devices. As we progress towards the Core layer, resources expand significantly, using nodes with CPUs operating at the GHz level and memory scaling to several GBs, and storage capabilities ranging from a few GBs to multiple TBs, reflecting the robust processing environment of central data centres. Application’s instances with specific requirements (CPU, memory, storage, latency sensitivity) are placed within this network, and their performance is evaluated based on several metrics. Two main application placement strategies are evaluated: 1. Genetic Algorithm: This strategy employs a genetic algorithm to optimize the placement of application instances across the network. The algorithm considers factors such as latency, energy consumption, data exposure, and resource utilization to determine the optimal node for each application instance. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 121 / 195 2. Random Placement: As a baseline comparison, a random placement strategy is also evaluated. Here, application instances are randomly assigned to nodes within the network, without considering the optimization factors. Evaluation Metrics The performance of each placement strategy is assessed based on the following metrics: • Latency: Measures the time delay experienced in the network, a crucial factor for latency-sensitive applications. • Energy Consumption: Evaluates the energy efficiency of the application placement, an important consideration for sustainable network operations. • Data Exposure: Assesses the extent of data exposure across the network, i.e., the distance from the data source, which has implications for security and privacy. • Resource Utilization: Indicates how effectively network resources are utilized by the placed applications. The results of the evaluation scenarios (Figure 6-13) reveal significant differences between the GA and random placement strategies. The GA consistently achieves lower latency and energy consumption, suggesting a more efficient utilization of network resources. In contrast, random placement, leads to suboptimal performance, particularly in terms of latency and energy efficiency. Data exposure and resource utilization metrics also show the GA's ability to balance network load and manage data more securely and efficiently. This is crucial in 6G networks where data management and security are paramount. (a) (b) (c) (d) Figure 6-13 Application component placement optimisation evaluation: E2E Latency (a), Energy Consumption (b), Data Exposure (c), Resource Utilisation (d) 6.4.3.2 Applicationand Device-driven optimization for BCS Figure 6-14 shows a generic call flow of the INC approach for a coordinated of network and compute optimization. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 128 / 195 option to share various resources with different tMECs by leveraging its orchestrating abilities, or even with peer cMECs. Dependency on a tMEC System: In instances where the cMEC lacks the implementation of a particular MEC function, it must depend on the upper layer tMEC system to provide the lacking functionalities. New workflows, guidelines for MEC application development, and specific interfaces must be established to address the deficiency in functions. • End-User Device Co-location and Awareness: The cMEC system has the flexibility to either operate within the same end-user device as the MEC application or to run on a constrained device in its immediate vicinity. The end-user device can engage in the cMEC integration in the following manner: 1. cMEC-aware: the end-user device and cMEC are either within the same local network or have knowledge of each other's identity (e.g., the cMEC operates on that specific end-user device). The end-user device has the capability to examine the available cMEC systems and request the deployment of a MEC application, leading to the establishment of a connection between the cMEC and a tMEC. 2. cMEC-unaware: The end-user device lacks awareness of any nearby cMEC and consequently requests the deployment of a MEC application directly to the tMEC. However, upon recognizing the presence of a nearby cMEC deployment, the tMEC opts to instantiate the application on the interconnected cMEC. Given the aforementioned points, Figure 7-3 details the architectural implications to interconnect the cMEC with the tMEC, without the MEO being present in the cMEC system. The main components of the system are the virtualization infrastructure that hosts the MEC applications (MEC App) and the Multi-access Edge Platform (MEP). The virtualization infrastructure furnishes computing, storage, and networking resources for MEC applications, encompassing a data plane. for routing the traffic among applications, services, local/external networks, and mobile edge platform. The mobile edge platform offers a setting in which MEC applications can discover, advertise, consume, and offer mobile edge services. Finally, MEC applications run on top of the virtualization infrastructure provided by the mobile edge host and can interact with the mobile edge platform to consume and publish mobile edge services via the Mp1 reference point. How these MEC applications retrieve the data to be published as a service is left unspecified by current ETSI MEC specifications. ETSI MEC defines a set of exemplary services. For example, the Radio Network Information service (RNIS) provides radio network-related information, such as up-to-date radio network status, metrics, and statistical data pertaining to the user plane, and information related to users served by the radio nodes. At the system level, an operational support systems (OSS) tool usually oversees the initiation and cessation of MEC applications as requested by a user application lifecycle management (UALCMP) proxy. These requests are received from either an end user or a customized portal. The presence of a MEC orchestrator (MEO) affords a comprehensive view of the entire MEC system, facilitating package onboarding and determining the most appropriate host for deploying the application. At the host level, the MEC platform manager (MEPM) directly manages the lifecycle of applications while configuring traffic, security, and DNS rules based on the application's requirements. Meanwhile, the MEC Execution Platform (MEP) provides the environment for MEC services to MEC applications and implements DNS and traffic control rules for these applications. Eventually, the computational, network, and memory resources of the platform are, managed by the virtual infrastructure manager (VIM). The architectural implications of interconnecting the cMEC and the tMEC include a cross-system reference points inter Mm2 and inter-Mm3 that are primarily introduced to facilitate the setup of the cMEC-tMEC interconnection. The Mx2 reference point is expanded to enable users to initiate lifecycle management actions (such as instantiation, deletion, or updates) of MEC applications within a cMEC or even a tMEC. Consequently, the inter-Mx2 interface, linking the cMEC application proxy to that of the tMEC, can ensure a certain level of consistency between cross-system applications (i.e., those spanning multiple layers) and enable any request to be propagated from cMEC to tMEC. Lastly, the Mp1 reference point, connecting MEC applications and services with their respective platforms, should be extended as an inter-Mp1 reference point for service consumption and app-to-app communications between different systems. The Operational Support System (OSS is a tool(from the service provider) working at the MEC level and may not always be associated with a subordinate local cMEC for application onboarding and instantiation. These tasks, typically carried out by a network manager working within the MEC through the Operations Support System (OSS), might require initiation by the end user (for example, requesting a specific application for their Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 129 / 195 home or vehicle) and managed by the cloud and the remote OSS and MEO of the tMEC, utilizing alternative workflows that support a new range of cross-system MEC interfaces. This includes interfaces such as Mx2 and Mm8 from the traditional MEC framework should be enhanced to allow users to trigger new instantiations. Figure 7-3 Architectural scheme of constrained MEC together with traditional MEC [RGO+23] The extended cloud framework that includes the constrained devices is enhancing the traditional cloud capabilities and could be used to run not only vertical applications, but also network functions. Figure 7-4 illustrates a scenario of an industrial factory, where robots execute different tasks, and are connected to a local network. The factory is equipped with local compute servers (extreme-edge, Far edge and Cloud) and is also connected to a CSP. The underlying applications are associated with different requirements, ranging from very strict delay requirements to medium delay requirements. As the extreme-edge cloud could suffer from scarcity of compute resources, some vertical applications could dynamically be deployed at the edge cloud and the extreme-edge cloud. Note that optimal placement of such applications would require coordinated processes between the M&O framework of the local virtual network and the CSP, by considering their network and compute capabilities. Consequently, this would require functionalities at vertical site to handle user traffic and to steer it locally or to the CSP. Handling user traffic is performed by a network module, which is UPF (User Plane Function). Different UPF types with specific functionalities are being standardized, including Uplink Classifier (UL-CL) and Intermediate UPF (I-UPF). These network modules are deployed in a chained fashion to ensure a common objective. For instance, an UL-CL can be inserted to steer a specific traffic to a local edge, while routing the remaining traffic to a central UPF anchor. In order to handle user traffic at the vertical site, a UPF type network function that is called Sub-UPF (as it operates in a sub-network environment) could therefore be deployed for this purpose. Sub-UPF (which could be owned by the vertical) can therefore be customized and configured to steer user traffic as per the location of the target application. Ensuring efficient user traffic steering in a way to reach the target QoS would require optimized placement of the chained user plane functions across the cloud continuum. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 130 / 195 Figure 7-4 Scenario for handling user traffic in cloud continuum environment The management and orchestration of the network functions like the Sub-UPF at the vertical side needs to (i) take into account heterogeneous virtualization platforms (e.g., Kubernetes, K3s, Microk8s, OpenStack, etc.) and extreme edge devices (e.g., IoT devices, Sensors and Actuators, Robots and Cobots, etc.), (ii) improve the placement mechanisms to efficiently deploy, migrate and distribute network functions that have proximity constraints (e.g., applications components that need to be executed near data sources to fast react to related events) and/or requirements exposing a unified interface, and (iii) to automate the discovery mechanisms of virtualization platforms and extreme edge devices information. To support such M&O framework a driverbased approach can allow the orchestrator to retrieve the platform specific computing capabilities and device specific information through per-platform and per-device drivers. These drivers, implemented as specific resource management software components that execute workflows within the M&O and share the same interface, are managed by a platform and device agnostic manager to perform the dynamic discovery, continuous monitoring, and inventory of the resources across the Extreme-Edge, Edge, and Cloud Continuum. The driver-based approach designed for the compute continuum M&O allows to plug and unplug specific drivers, even dynamically at runtime, depending on the scenario and on the types of the devices involved. Adding a new compute resource (i.e., that it could be hosted in a device) does not introduce any signalling cost to the discovery and monitoring processes since the new compute node join operation (e.g., to a Kubernetes cluster) is performed by the legacy virtualization platform-specific procedures (e.g., the Kubernetes ones), with no impact to the above-mentioned workflows. Figure 7-5 depicts a possible architecture of the compute continuum M&O that highlights the extreme edge, edge and cloud resources discovery, monitoring, and inventory workflows as well as the virtualized services orchestration workflow. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 131 / 195 Figure 7-5 High level software architecture of the compute continuum M&O The platform has been designed to be a multi-module architecture where each sub-module (in grey in the Figure 7-5) contributes to the functionalities of the orchestrator. • The Platform Manager enables the dynamic discovery and continuous monitoring of the resources across the extreme edge, edge, and Cloud Continuum through the management of platform-specific and device-specific drivers, embedded in an agnostic-designed skeleton, which can be develop, plugged, and unplugged depending on the underlying virtualization infrastructures and the involved devices. It also takes care of the virtualized applications orchestration operations by the means of platform-specific drivers. • The Resource Manager module registers to the Platform Manager to receive updates on the discovered and monitored virtualization platforms (along with device-specific information) to expose a unified interface to retrieve information about the extreme edge, edge and cloud continuum resources and thus enable the inventory functionality of the platform making use of different information models. • The Service Deployer module exposes a unified interface to manage platform-agnostic virtualized service applications orchestration requests, leveraging the different information model, and translates them in the platform-specific orchestration requests that can be handled by the platform-specific drivers available in the Platform Manager to perform the orchestration operations requested. The Service Deployer works alongside with the Service Template Catalogue to retrieve the platformspecific orchestration templates to be used to fulfil specific orchestration operations. The introduction of such a new M&O platform is focused on going beyond monolithic approaches of standards solutions like ETSI NFV MANO and aims at clearly separating the orchestration of the (legacy) network services from the orchestration of the vertical applications. In particular, it introduces a fully cloud-native Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 132 / 195 approach, with transparent integration of extreme-edge resources, to address the strict requirements of the application layer domain (in terms of deployment constraints, dynamicity, communication patterns, etc.). If we consider standards solution like ETSI Network Functions Virtualization Management And Orchestration (NFV M&O) the introduced platform can play the role (i.e., be mapped in the ETSI NFV M&O architecture) of a particular enhanced VIM due to the fact that its functionalities are mainly focused on the management of the resources of the extreme-edge, edge and cloud continuum layer (infrastructure layer) yet, at the same time, it offers advanced functions like the management of multiple domains, the dynamic discovery of the infrastructure layer resources and their continuous monitoring, and the possibility to orchestrate service application over the managed resources. In the context of the Hexa-X-I M&O framework [HEX22-D62] the highlighted M&O platform could fit in as infrastructure layer M&O since it covers all the requirements needed by such entity (functionalities of the infrastructure layer M&O) providing, at the same time, advanced functionalities (i.e., multi-domain support, dynamic discovery of continuum resources, etc.) In the same line of going beyond the legacy monolithic MNO-centric M&O approaches, another alternative architectural modification towards 6G would involve the development of the decentralized M&O concept. This approach is considered to have the potential to solve the new set of challenges with respect to the aggregation of devices beyond the MNO own premises mentioned above, i.e., the diversity of stakeholders in this domain (vertical industries, hyperscalers, end-users…), the high heterogeneity of devices (including reduced capability or battery power devices,), the potential high volatility of those devices (they could unexpectedly move, drastically change their available computing or processing resources or even be unexpectedly disconnected), and the size of this domain, that can be huge in scale. In this regard, Appendix 11.4 describes how a decentralized M&O approach could be implemented. This approach relies on the deployment of multiple instances of a reduced set of network elements that would be distributed through the entire network continuum, so addressing the M&O problem considering the network as a diverse ecosystem of resources and stakeholders (as it is envisaged towards 6G), rather than approaching it only from a single MNO perspective. 7.1.3 Preliminary workflows and evaluation This section presents some preliminary workflows and interactions of some of the features of the integration and orchestration of extreme edge resources in the compute continuum. The general idea that the architectural implications have been presented in the previous section are elaborated and evaluated in detail here. The driverbased M&O approach has three main functions, namely: (i) the dynamic discovery, (ii) continuous monitoring and inventory of the compute continuum resources and (iii) the virtualized service applications orchestration. The dynamic discovery, continuous monitoring, and inventory of (i.e., extreme-edge, edge and cloud) continuum resources functionalities, depicted in Figure 7-6, are fulfilled by the Platform Manager and Resource Manager modules of the M&O Platform as described in the previous section. In the first step of the workflow the Resource Manager performs a subscription to the Platform Manager to keep track of the new clusters (i.e., and their resources) that could be onboarded to be monitored in the future. Next, the Resource Manager queries all the available platforms from the Platform Manager to start receiving updates for them; the updates are produced by platform-specific and device-specific drivers of the Platform Manager that collect the platform and device information and characteristics from the target platforms. The workflow in Figure also details the operations executed by the Platform Manager when a new platform is onboarded to be monitored. Upon receiving a new platform, the Platform Manager spawns dedicated drivers to monitor the specific information of the underlying resources. As the last step the Platform Manager notifies the Resource Manager of the newly added platform so (i.e., the Resource Manager) it can keep track also of these resources in its inventory. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 133 / 195 Figure 7-6 Dynamic discovery, continuous monitoring, and inventory workflows The virtualized service applications orchestration workflows are depicted in Figure 7-7. The involved modules of the M&O Platform that fulfil these functionalities are the Platform Manager, with its platform-specific deployment drivers, the Service Deployer, the Resource Manager, to retrieve the deployment target platform information, and the Service Template Catalogue to retrieve the templates to be used to deploy or update a given virtualized service application. In the deployment workflow the Service Deployer module of the M&O Platform inspects the service components of the received deployment request, on its Northbound interface, and, for each of the requests, retrieve the specified orchestration template from the Service Template Catalogue. Next, it proceeds to customise the template and produces the platform-specific service component deployment request that will be handled by a platform-specific deployer (driver) of the Platform Manager to execute the deployment of the application component in the target virtualization platform. The service application update and delete workflows are very similar to the deployment one. In the update workflow the request includes the identifiers of the service components that need to be updated and the new configuration that need to be applied: the involved modules of the M&O Platform are the same as in the deployment one. In the delete workflow only the identifiers of the service components that need to be terminated are specified. It is worth mentioning that all the components of the M&O Platform and workflows highlighted in these paragraphs have been tested and integrated in an internal lab environment and in the PoC B testbed. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 134 / 195 Figure 7-7 Platform-agnostic service applications orchestration workflow Regarding the decentralized M&O approach also introduced in the previous Section 7.1.2, Appendix 11.4 introduces the high-level workflow showcasing the overall operation of the concept. Regarding its evaluation, works targeting some of the technical challenges associated to this concept are being addressed in the context of the System-PoC B, and are planned to be reported in Deliverables D2.4 and D2.6, reporting the interim and final end-to-end system evaluation results, respectively. 7.1.4 Summary As stated in [HEX223-D32], integration and orchestration of the extreme edge resources in the compute continuum enables the M&O capabilities of compute resources beyond the radio access part of the network. It provides the needed architectural components, interfaces, and mechanisms to orchestrate and manage the volatile and resource constrained extreme edge devices as part of the compute continuum. This enabler directly contributes towards the fulfilment of the improved QoE, seamless service continuity, efficient M&O, and privacy protection requirements of all the six use case families defined in WP1 [HEX223-D12]. For example, in the Immersive Experience, Trusted Environments and Fully Connected World Use Case Families this enabler contributes towards the improvement of the privacy and security protection. Different applications from the above-mentioned use cases will be allowed to handle their sensitive information (e.g., eHealth) in the device where it has been generated. In addition, in the Collaborative Robots and Digital Twins Use Case Families this use case contribute towards improved M&O and service continuity capabilities by providing mechanisms for flexible resource inclusion and allocation in the compute continuum. This enabler has implications on the infrastructure layer of the 6G E2E system blueprint where new interfaces and mechanisms need to be defined for: 1) exposing the capabilities of the extreme edge devices and 2) for interactions between the extreme edge, edge, and cloud resources. Moreover, implications on the Management and Orchestration component of the 6G E2E system blueprint are expected by defining new end-to-end multitechnology resource orchestration concepts to orchestrate resources over the volatile and constrained extreme edge devices. Table 7-1 shows a summary of the main benefits and implications of the Integration and orchestration of extreme edge resources enabler in the E2E 6G system blueprint that is under development in the project. Table 7-1: Benefits and implications of "Integration and orchestration of extreme edge resources" enabler Description Interfaces, architectural components, and mechanisms to allow the integration, orchestration and management of the extreme edge devices into the compute continuum. Benefits KPI improvement Latency, energy consumption, service availability, increase data privacy. Design principles [HEX223-D21] Full automation and optimization (#2) Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 135 / 195 Flexibility to different scenarios (#3) Internal interfaces are cloud optimized (#7) Dependencies / Basis for another enabler Programable network monitoring and telemetry Orchestration mechanisms for the computing continuum Programmable and flexible network configuration Implications Requirements New UE roles and responsibilities, coordination and interaction of the UE with the edge and cloud, communication between functions deployed in the edge/cloud and the UE, UE resource orchestration and management. Standard relations & regulations 3GPP TS 23.558 [23.558] ETSI GR MEC 036 [MEC035] Computing resources Cloud computing, edge computing and UE computing resources will jointly automated and optimized. 7.2 Multi-domain/multi-cloud federation 7.2.1 Introduction Multi-domain/Multi-cloud federation is the capability to aggregate cloud services provided by multiple domains and providers into a single, coherent cloud. The federation concept is not only about the capability of spanning across the administrative domains of different legal entities (e.g., network operators, nations), but it is also strictly related to the multi-cloud capability, i.e., the aggregation of underlying cloud resources built on different cloud technologies, including private and public cloud. In 5G, federation is mainly focused on sharing cloud/compute resources. The key concept is that in a 6G system, federation should extend the basic 5G federation concept, providing a joint offering of cloud and network services alongside with beyond communication services (e.g., sensing). From an operator point of view, this facilitates the seamless deployment of telco and non-telco applications, aligning with the "Beyond communications" paradigm. From a customer point of view, users can expect a consistent level of service across different operators and countries within the federated cloud. This can be achieved via network services pairing across federated domains allowing easy portability of applications between these domains. The architectural solution has three key innovations points that give value to a future implementation: • Native federation: In 5G system there is no native concept of federation. Federation mechanisms (e.g., GSMA OPG) in 5G are standardized in objects that are outside the 5G system M&O layer. This leads to higher costs of integration and maintenance and loose coupling between the 5G system and those external objects. The requirement is to have a direct management of the federation in the M&O layer of the 6G system (possibly by means of a dedicated module of a modular M&O layer). • Broader resource sharing via federation: 5G federation concept is focused on aggregation of cloud resources. The whole federation and related brokering systems are designed to offer cloud services. The proposal it to redesign federation concepts by including network services and beyond communication services (e.g., sensing), the latter of which is a key concept in HEXA-X-II project (see Chapter 6), allowing the federation customers to benefit from an aggregated cloud, network and beyond communication service. As an example, the integration can lead to optimized and coordinated services such as coordinated edge roaming: local breakout of data, intrinsic to network technology should be coordinated with application mobility, in order to grant seamless service to users across the federations (coupling of network local breakout and mobility of application is not natively synchronized in 5G system). The requirement is the evolution of federation mechanisms and interfaces towards the integrated cloud, network and beyond communication services. • Intent-based interfaces: Intend-based management is one key enabler of Hexa-X-II [HEX223-D22]. We are depicting an architecture in which all federation interfaces (including the interfaces to customer Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 136 / 195 and the interfaces to lower layer services to be shared in the federation are intent based. This leads to a federation of services rather than a federation of bare cloud resources. Section 7.2.2 categorizes the federation approaches that can be followed in a 6G architecture and describes architectural implications of the enabler for a specific approach. Section 7.2.3 offers a look on the initial evaluations obtained for this enabler and finally, Section 7.2.4 summarizes the concepts of this enabler organizing them in a table. 7.2.2 Architectural modifications Multi-domain/Multi-cloud federation greatly impacts the Orchestration functionality since 6G M&O layer needs to encompass a dedicated module to manage federation. Federation models can vary depending on how the relation between M&O Layer and services is implemented. The following models have been identified: • Direct model: Cloud services directly communicate with M&O layers of different Network Operators, regardless of the domain in which they are deployed. This model is the simplest one, but it has limited scalability. • Cloud service broker model: A common resource layer acts as a single Cloud service broker for all federated network operators. All Operators can exploit Cloud services from all the federated domains. This model has higher scalability than the first one, but it intrinsically has a governance problem (i.e., who owns the service broker) • Peering model: A dedicated peering interface is established between 6G M&O layers of Network Operators. Federation interfaces manage both business and technical aspects. Beside Cloud Service from federated operators, other cloud resources from external cloud entities can be aggregated and exposed. This model offers good level of scalability without relying on a centralized service. Regarding the interfaces, intent based APIs layer should be able to aggregate cloud services, network services, and beyond communication services provided by the operator and by all federated partners. The intent of services shall encompass the combination of all three types of services, i.e., with one intent-based API call, interfaces of the different services can be called. Finally, security and privacy network functions and policies should be extended to also consider the security related aspects of federation interfaces and federated resources. The federation multi-domain aspect of data centres is a critical factor in the evolution of the current state of the deployment architectures for large-scale city-wide or even country-wide deployments. However, we must look at more than just data centres for their traditional factor, i.e., a large pool of computing resources allowing big on-prem implementations. We are to look at them as a federation of nodes that also includes the edge nodes scattered around the city and, with the advent of 6G, the extreme edge nodes that can also be powerful computational units. Hence, the workloads that will be running shall be cloud-native and be able to bootstrap themselves on different nodes, regardless of their nature (e.g., CPU architecture). On the other hand, nature will be of utmost importance when choosing where to run a specific workload to benefit from the nature of the data centre. The critical factor is ensuring that the underlying infrastructure is homogeneous and based on cloud-native technologies so that an orchestrator can reach it and choose to deploy specific workloads there. Therefore, the concepts that we discuss here, in a 6G environment, will entail the extension of the cloud principles until the edge and the extreme edge. In this way this extension aims to foster a base configuration for the Advanced Orchestration and Management of the resources in every node of the federation ensuring an efficient resource allocation and utilization. The architecture should also amplify the lifecycle management of the applications using Life Cycle Assessment tools and solve the 3GPP premises while adding a flavour for the correct placement of the workload or resource. Finally, access to such a robust network of computing nodes will lower the overall cost of running an application, since the necessary resources to perform the tasks can be deployed across the different nodes. The 6G ecosystem needs to evolve towards amplifying the M&O capabilities of the 5G ecosystem (and previous ones), making sure every node (cloud, edge, extreme edge, onprem, etc.) has the same bootstrap recipe and the exact implementation principles for it to be orchestrated seamlessly. Such aspects are directly related to the heterogeneous integration of diverse technologies while optimizing the latency requirements (critical vs sensitive should be one of the placement decisions for the app). Moreover, the federation allows for “free” distributed intelligence with by design privacy since all nodes are on a Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 137 / 195 federation and (should) trust each other. Lastly, it touches on massive connectivity since all nodes must expose the same standard APIs for better interoperability while fostering new topologies for in situ processing or running concurrent tasks on energy-optimized nodes. The direction of the 6G ecosystem for the Multi-Domain Data Centre should treat it as another federated cloud node with specific characteristics that can confer its advantages over other nodes on the network. The app developer or resource holder should be on the lookout for making the best product possible and then tell the 6G ecosystem the network requirements for their developments. The network should decide where to place the workloads that makeup such a product. The following paragraphs present a modification of the ETSI NFV MANO framework using the Cloud service broker federation model. Network virtualization typically applies ETSI NFV specifications to orchestrate Network Services (NS) [NFV002] and assumes that the orchestrator operator is also owner of the infrastructure therefore no business interfaces to the NFVI, composed of NFVI-PoPs [INF001] are defined. The multi-provider approach, however, provides numerous benefits, such as reducing the cost of the resources by selecting the cheapest provider, extending the geographic infrastructure span, increasing infrastructure capacity, and contributing to higher infrastructure reliability. Moreover, the NS security can be improved, as each provider may handle only a part of the NS. The concept called Cloud Continuum [HEX22-D13] assumes integration and uniform exposure to applications of all available resources. It includes the unreliable and constrained devices, i.e., the resources of IoT nodes or smartphones, and the resources of NTNs, of which some, like Low Earth Orbit satellites (LEOs) or UAVs, may be short-lived. In this approach, a modification of the ETSI NFV M&O framework allows for dynamic interconnection of data centres of various infrastructure providers and exposing the aggregated resources to services. The key component of the concept is the resource layer that handles all infrastructure-related operations and acts as a proxy between Modified NFVOs (M-NFVO) and Modified VIMs (M-VIM). The modified ETSI NFV M&O components, ecosystem actors, and infrastructure components interact with resource layer via secure interfaces, as illustrated in Figure 7-7. For simplicity, it is assumed that Edge Cloud data centres and the Central Cloud data centre provide computing, storage and intra-data centre networking virtualization, as well as mechanisms to cope with faults and load balancing. Each data centre is described by the Data Centre Features (DCF), which consists of, among others, parameters concerning data centres’ geographic location, total capacity, delay of links between this data centre and other data centres, and resources cost. Some data centre parameters such as resource consumption, energy consumption, power status (in the case of battery-powered data centres), and estimated reliability are updated by data centres or resource layer in real time. In the case of mobile data centres (LEOs, UAVs, cars, pedestrians), details concerning data centre mobility patterns are also provided or calculated by the resource layer; therefore, DCF supports the Far-Edge resources usage. The main features of the proposed approach are the following: • Resource layer has mechanisms for dynamic adding and removal of data centres of multiple Infrastructure Providers using secure interfaces. The dynamically updated DCF describes each data centre. • Exposing all resources to orchestrators would make the orchestration not scalable; therefore, the resource layer exposes to orchestrators only a Partition of Infrastructure resources (NS-PoI) with its topology. The NS-PoI is created using NS requirements that may include data centre location, cost, energy efficiency, reliability, inter-data centre delay, etc. Please note that the resource layer computes the partition topology; this is no more the role of the orchestrator. • Based on the requirement of each NS (for example, location, inter-data centre delay), the resource layer may group several data centres to create an Aggregated Data Centre (ADC). The ADCs simplify NS-PoI topology and stabilize it (in the case of Infrastructure dynamic changes), improving that way orchestration scalability. • The resource layer supports multiple orchestrators that are no longer linked with a dedicated Infrastructure domain; each handle dynamically created, NS-specific NS-PoI. The orchestrators for specific NS or a set of NSs can be orchestrated. • The resource layer can autonomously trigger the VM migration within an ADC in a way invisible to orchestrators. The operation can be triggered by ADC modification (adding or removing a data centre Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 144 / 195 approach that enables the expansion of services across diverse national domains. Overall, the federated model emerges as a comprehensive solution that not only benefits Network Operators and customers but also fosters a more cost-effective and integrated environment for technology providers. Table 7-4 shows a summary of the main benefits and implications of the Multi-cloud/multi-domain federation enabler in the E2E 6G system blueprint that is under development in the project. Table 7-4 Benefits and implications of “Multi-cloud/multi-domain federation” enabler 7.3 Cloud transformation with quantum technologies 7.3.1 Introduction Quantum computing (QC) represents a ground-breaking computational approach that leverages the principles of quantum mechanics to perform specific calculations much more efficiently than traditional computers, such as laptops and desktops used in our daily lives [VCL24]. Unlike classical computers, which process information using bits in states of 0 or 1, quantum computers utilize quantum bits, or qubits, capable of existing in multiple states simultaneously due to the quantum property known as superposition. This empowers quantum computers with the capability to concurrently explore numerous potential solutions to a given problem. Quantum parallelism, a crucial aspect of QC, enables the simultaneous exploration of extensive solution spaces, offering the potential for more efficient resolution of complex problems compared to classical computers. Additionally, the entanglement phenomenon establishes correlations between the states of multiple qubits, allowing them to be interconnected in a way that modifications to one qubit instantaneously influence others, regardless of spatial separation. This property is harnessed in quantum algorithms to execute complex calculations with heightened efficiency. The convergence of 6G and QC not only represents a significant advancement in technological evolution but also heralds the emergence of transformative applications poised to revolutionize various sectors, including telecommunications, healthcare, finance, and AI. The synergy created by combining the ultra-fast, low-latency Description The main aim of the federation model is to enhance the efficiency, reliability, and flexibility of computing infrastructure by fostering interconnectivity among diverse infrastructures. High level concept design and placement algorithms are studied to allow organizations to leverage a distributed network of computing power, storage, and services, transcending the limitations of individual cloud providers. Benefits KPI improvement Latency, network load, reduce complexity for cross-domain deployment, improve QoE Design principles [HEX223-D21] (#1) Support and exposure of 6G services and capabilities (#2) Network Scalability (#7) Internal interfaces are cloud optimized Dependencies / Basis for another enabler Cloud continuum should provide intent-based interfaces for cloud services. 6G Core network should provide intent-based interfaces for network services. Beyond communication functions module should provide the intent-based interfaces for BYC services Implications Requirements Intent-based interfaces for every component M&O layer that is able to harmonize all the different parts. Federation interfaces Standard relations & regulations This enabler will impact the orchestration of network functions. Impacts includes but are not limited to [23.501], [23.502]. Required resources Cloud resources Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 145 / 195 communication capabilities of 6G with the unparalleled processing abilities of quantum computing is set to redefine the possibilities and applications of both technologies. 7.3.2 Architectural modifications The realization of a softwarized continuum, in which network operations are realized as microservices or intelligent agents, will require unprecedented resources for in-network computing, which will be a pillar of communication networks. In parallel, the employment of AI in the future 6G architecture will also create an explosion for computing at the control plane in order to target multiple objectives. First, it could be used for management and orchestration, traffic classification, and time-series forecasting. In particular, referred to management and orchestration, some examples of use cases could be traffic forecasts, automated Virtual Network Function (VNF) placement and network slicing, network self-healing, and hidden patterns discovery, to proactively assist network planning and sizing. With this immense computing load associated with softwarization and AI, QC may become a suitable candidate for core processing components within O-RAN, particularly for computations reliant on the RIC [LBS+22]. A QC-enabled core RIC holds the potential to deliver real-time outcomes for Near-Real-Time (Near-RT) services. Further, O-RAN can integrate quantum communication to bolster its security measures [AAU+23]. Critical components such as DUs and CUs within O-RAN infrastructure need to be both easily deployable and securely connected. Failure to ensure this may expose the network to eavesdropping, man-inthe-middle attacks, and various other security threats that could compromise user traffic flowing through ORAN nodes. QKD offers a solution by establishing secure keys between O-RAN components, effectively mitigating these risks. Figure 7-12 O-RAN quantum edge architecture Quantum-enabled Open RAN networks can also support innovative applications such as quantum optimal algorithms, quantum machine learning, and quantum-assisted blockchain technology. These applications can unlock new possibilities for network services and functionalities. Leveraging quantum computing demands the development of unique models tailored to its operations. In certain cases, standard computations like radio resource management and allocation may need to conform to QC models or algorithms for problem-solving. Specifically related to the RAN and the edge of the network, open software has obtained significant attention in order to realize customized solutions for specific businesses and use cases (for example industrial IoT in campus networks with focus on low latency). Furthermore, integrating QC hardware into the network architecture requires careful consideration of the limited capabilities of today’s quantum hardware and the complexity of multiple computing layers. As compute and storage requirements increase, the efficiency of QC integration decreases in comparison to classical HPCs. Therefore, a phased approach is critical, with quantum capabilities added incrementally, starting at the edge (the computing resources closest to end-user devices with limited capabilities of state-of-the-art noisy quantum computers) and eventually extending to the cloud (centralised cloud data centres providing massive computing and storage resources). In the above context of necessary unprecedented in-network computing resources, research and standardization are ongoing to create an open source High-Performance Computing (HPC) stack [SRK+22] including open Hardware and Software. This can play a pivotal role in future 6G open source MEC. This open-source approach Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 146 / 195 can help in ensuring Europe’s global competitiveness and data sovereignty, since it creates the capability to handle the entire data life cycle, which in turn relies on the underlying in-network edge computing infrastructure. Here, we aim to investigate the integration of quantum communication and computing technologies into ORAN and open source HPC stack for MEC (see Figure 7-12). This to improve the performance (in terms of, for example, latency, energy usage, connectivity, resilience) of the classical 5G and upcoming 6G RAN-MEC. Specifically, the idea is to integrate current available quantum computing platforms (with limited number of qubits), that are available today, with MEC classical platforms. Centralized and distributed quantum classical MEC can be considered. Furthermore, we target designing and developing open-source solutions for hybrid quantum-classical computing that have to support RAN and MEC network functionalities/microservices/agents. This is an important pillar in future 6G automated industrial networks, in which thousands of devices (robots, sensors, etc.) will significantly stress the computing at the RAN-edge of the network. Evaluations will consist of emulation of computing environments, exploiting the benefits of an extended quantum-classical RAN-MEC open source stack (socket to application) together with some specific experimental evaluations using available computing (HPC and quantum) platforms in the consortium. The hybrid classical model is expected to exploit the unique properties of quantum mechanics. In the midterm (upcoming 3-5 years), quantum capabilities may be used for secure information distribution, while heavy computing tasks remain within the domain of classical computers due to their current superiority over quantum counterparts. We will assess chosen Open RAN functionalities, comparing power consumption and latency in both centralized and distributed hybrid quantum-classical architectures. We will assess chosen Open RAN functionalities, comparing power consumption and latency in both centralized and distributed hybrid quantumclassical architectures. The hybrid classical model is expected to exploit the unique properties of quantum mechanics. In the midterm (upcoming 3-5 years), quantum capabilities may be used for secure information distribution, while heavy computing tasks remain within the domain of classical computers due to their current superiority over quantum counterparts. We will assess chosen Open RAN functionalities, comparing power consumption and latency in both centralized and distributed hybrid quantum-classical architectures. We will assess chosen Open RAN functionalities, comparing power consumption and latency in both centralized and distributed hybrid quantumclassical architectures. 7.3.3 Summary The impact of quantum technology on cloud transformation in 6G networks depends on the maturity of quantum technologies and their seamless integration into the network architecture. The incorporation of quantum technologies will involve their partial integration as sub-modules or in a hybrid-classical combination. With an emphasis on KPIs such as latency, security and robustness, quantum technologies will only be strategically integrated where their inclusion offers advantages over purely classical scenarios. The imperative for a hybrid architecture is to seamlessly integrate quantum capabilities into existing network infrastructures, particularly at the edge of the cloud. The hybrid classical model is expected to exploit the unique properties of quantum mechanics. In the midterm, quantum capabilities will be used for secure information distribution, while heavy computing tasks will remain within the domain of classical computers due to their current superiority over quantum counterparts. We will assess chosen Open RAN functionalities, comparing power consumption and latency in both centralized and distributed hybrid quantum-classical architectures. We will assess chosen Open RAN functionalities, comparing power consumption and latency in both centralized and distributed hybrid quantum-classical architectures. Table 7-5 shows a summary of the main benefits and implications of the Cloud transformation with quantum technologies enabler in the E2E 6G system blueprint that is under development in the project. Table 7-5:Benefits and implications of “Cloud transformation with quantum technologies” enabler Description Cloud Transformation with quantum technologies Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 147 / 195 Benefits KPI improvement Reduce latency, increase robustness of the network Hybrid (classical-quantum) open source Design principles [HEX223-D21] Integration of small-scale quantum computing platforms Hybrid quantum-classical centralized / distributed computing Dependencies / Basis for another enabler Interfaces for the entanglement distribution Modular architecture Implications Requirements Quantum hardware (computers, routers, repeaters,..) Interfaces for hybrid integration of classical and quantum components Quantum open sources Standard relations & regulations 3GPP specification [TS 38.300] Required resources Quantum hardware (s.a.) 8 Proof of Concepts In this section, we present the component proof of concepts (Component-PoCs) that are studied within WP3. 8.1 Component-PoC #B.2: Distributed Model Training and Inference 8.1.1 Remote controlled robot use case The PoC dataset is collected from an internal network simulator. The simulation consists of a remote-controlled robot, with a camera attached to it, in a mining use case. An actuator robot with a UE attached to it is located at a distant location and is controlled by a remote controller that has another UE. The actuator robot streams out video from the environment over its UE and the communication link to the controller UE located at the other end. The UE receives the video packets, renders them, and based on the observed video sequences; it sends control signals back to the remote actuator robot to operate the robot. Both UEs are connected to a base station, where the base station transmission power can be configured. The data logs related to data input Physical, MAC, IP, and video application (data output labels) are recoded during a simulation of video streaming. The input attributes are as follows. Physical entity had attributes related to SINR, received power, received information size; MAC entity had uplink transmission data volume; IP entity had uplink and downlink throughput as input features. The number of input attributes were not too many. To stress the computation and communication of neural network training with higher input feature dimensionality, additional dummy input features (all were set to zero) were added as input to the neural network. Therefore, at the end, Physical, MAC and IP entities had 259, 258, and 257 input attributes in total. The two use cases are deployed at two entities, as two NN models, in the output node. The use case entities in the output node collected dataset at the application level related to the video quality indicators: • videobitrate: measurement of the video bitrate of the received video frames at the control terminal client, • videodelay: measurement of the inter-frame time of the decoded video frames received at the control terminal. The goal is to estimate videobitrate and videodelay with the decentralized input attributes in the input node. In the inference phase, these two NN models are transferred from output consumer to UPF such that UPF can assess the QoE. Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 148 / 195 8.1.2 Distributed AI-Enabled Technology: Split Learning This document on the proof of concept (PoC) presents an AI-enabled technology called split learning and showcases benefits and trade-offs on different scenarios. The implementation is performed as a multi-headed, multi-tailed parallel split learning in Kubernetes. Neural networks that belong to different entities are placed in distributed manner to multiple Kubernetes pods. The PoC consists of three main types of nodes: input node, generalization node, and output node. The global NN model is split into these node types, and every node type contains portion of the global NN model. A node type in this demo is defined as the group of at least one neural network model. The neural network models that are deployed at the input and output of the global neural network are called input and output nodes, respectively. The input node contains the input dataset, i.e., ML features; and the output node contains the target labels. The intermediate NN model is deployed in the generalization node. These nodes are illustrated in Figure 8-1. Figure 8-1 Illustration of cross-network function training and model generalization via joint optimization for multiple use cases in a split learning setting. Blue trapezoids represent Kubernetes pods, and each pod hosts an NN model. Input node consists of three entities that each contains input data and compute capability. The three input entities are Physical, MAC, and IP, and each entity has one local NN model that trains on the local input data. The generalization node does not have access to raw dataset; it is rather an intermediate entity with a NN model (serving as a compute element) with the goal of taking high compute tasks related to the training of a split learning model. It also serves with the goal of generalizing the intermediate learned representations from input nodes, so called encodings or activations, to multiple use cases deployed in the output node. The output node consists of two NN models serving for two use case tasks, i.e., video bitrate estimation and video delay estimation. The Physical and MAC entities can be co-located in gNB; the IP entity and the generalization node can be co-located in UPF; and the output consumer entities that contains video bitrate and video delay data can be co-located in output consumer such as external application or some other external entity. The PoC demonstrates three AI enablers as follows. 8.1.2.1 Cross-network function training This enables model fusion with models that are trained on datasets obtained from different network functions. 8.1.2.2 Model generalization Model generalization is studied in two scenarios: Scenario 1 This enables part of the neural network ML model to generalize to multiple use cases (depicted as output consumers), i.e., reusing substantial portion of the neural network for video bitrate and video delay Hexa-X-II Deliverable D3.3 Dissemination level: Public Page 149 / 195 estimation. This then is expected to yield less models to manage. The split learning model architecture of scenario 1 is illustrated in Figure 8-2. The scenario consists of training jointly two use cases with the datasets obtained from multiple different network functions such as gNB, UPF, and other consumers that are potentially AFs or may be something else, e.g., PCF. The input ML features are obtained from the gNB and UPF in the input node; and the labels are obtained at the consumers at the output node. The datasets obtained at the gNB, UPF and other consumers (e.g., applications) are not shared in between, instead they train local NN models jointly in a split learning setting. The split learning allows these distributed and split NN models to be trained collaboratively by means of only exchanging activations and gradients. Scenario 2 Moreover, the modularization of NN model enables data domain adaptation to reuse a model between data domains or to jointly train high performing models when there are distributional differences between domains. This also allows us to train a model for a domain with missing labels when there exists a different domain for the same task with existing labels. There may be different data domains potentially caused by factors including base station reconfiguration, upgrade in the system, or to transfer models between heterogeneous networks, etc. This approach enables adaptation of an ML model in a data drift scenario to sustain model efficacy. This scenario is a collaborative NN model training consisting of only one use case, video bitrate estimation. In this scenario, the datasets are obtained via network simulation where there was a data drift during a video streaming session. This typically occurs since at the beginning of a video streaming, there is a video initialization phase where video packets are downloaded into the video playout buffer. The video client increases the playout bitrate until it reaches an optimum point, from then on the streaming is in steady-state. In data domain 1, the measurement dataset is obtained from the initial phase of the video streaming, i.e., between the duration when the video streaming is initiated and the time when it reached a steady-state. In data domain 2, the measurement datasets are obtained after a steady state has been reached. The data distributions between data domains 1 and 2 are different. Conventionally, an ML model that is trained in a dataset with a certain distribution under-performs for another dataset of different distribution. Therefore, conventionally additional model fine-tuning needs to be performed at the target domain. Instead, the goal with this scenario is to learn jointly a good representation of the two data domains via the generalization node with the assistance of additional domain classifier NN model. This way, distributed ML model training for multiple data domains can be performed once, and one model can serve for two data domains, simultaneously. The split learning model architecture of scenario 2 is illustrated in Figure 8-2. Data domain ids (1-data collected at video initialization phase; or 2data collected at steady state video streaming) are the target labels that the domain classifier trains on. In training, it learns to differentiate these two different data distributions, by reversing the sign of the gradients in the backward propagation after the domain classifier node. This entity needs to obtain these target variables in training phase. [Document text truncated for crawler view.]