Enhancing Holonic Architecture with Natural Language Processing for System of Systems

This is a self-archived version of an original article. This version may differ from the original in pagination and typographic details. Author(s): Title: Year: Version: Copyright: Rights: Rights url: Please cite the original version: CC BY-NC-ND 4.0 https://creativecommons.org/licenses/by-nc-nd/4.0/ Enhancing Holonic Architecture with Natural Language Processing for System of Systems © 2024 the Authors Accepted version (Final draft) Ashfaq, Muhammad; Sadik, Ahmed; Mikkonen, Tommi; Waseem, Muhammad; Mäkitalo, Niko Ashfaq, M., Sadik, A., Mikkonen, T., Waseem, M., & Mäkitalo, N. (2024). Enhancing Holonic Architecture with Natural Language Processing for System of Systems. In H.-G. Fill, F. J. Domínguez Mayo, Sinderen, Marten van, & L. Maciaszek (Eds.), ICSOFT 2024 : Proceedings of the 19th International Conference on Software Technologies - Volume 1 (pp. 427-433). SCITEPRESS Science And Technology Publications. https://doi.org/10.5220/0012787300003753 2024 Enhancing Holonic Architecture with Natural Language Processing for System of Systems Muhammad Ashfaq1∗, Ahmed R. Sadik2, Tommi Mikkonen1, Muhammad Waseem1, and Niko M¨ akitalo1∗ 1University of Jyv¨ askyl¨ a, Jyv¨ askyl¨ a, Finland 2Honda Research Institute Europe, Germany ∗Corresponding author: [email protected] Keywords: System of Systems, Holon Communication, Holonic architecture, Natural Language Processing, Conversational Generative Intelligence, Interoperability, Human-System Interaction. Abstract: The ever-growing complexity and dynamic nature of modern System of Systems (SoS) necessitate efficient communication mechanisms to ensure interoperability and collaborative functioning among constituent systems (CS), referred to as holons in the holonic architecture of SoS. This paper proposes a novel approach to enhance humand-to-holon and holon-to-holon communication within the holonic architecture through the integration of Natural Language Processing (NLP) techniques. Our proposed framework utilizes advancements in NLP, specifically Large Language Models (LLMs), enabling holons to understand and act on natural language instructions. This enables more intuitive holon-to-holon and human-to-holon interactions, leading to better coordination among diverse systems. The framework’s practical application is demonstrated through an Unmanned Vehicle Fleet (UVF) case study, showcasing its potential in enhancing communication and coordination in complex SoS. Additionally, we propose evaluation strategies to assess the efficiency and effectiveness of this framework, and identify areas for improvement. This work sets the stage for future exploration and prototype implementation, paving the way for further advancements in SoS communication and collaboration. 1 INTRODUCTION A System of System (SoS) is a collection of systems functioning together to achieve a common goal (Nielsen et al., 2015). These SoSs are comprised of multiple Constituent Systems (CS), each functioning independently with its management structure. CSs within an SoS can be geographically dispersed, further highlighting the need for effective communication and coordination. When integrated, the overall SoS capabilities are far more than that of the individual CSs forming the SoS. Moreover, an SoS should support evolutionary development allowing CSs to join or leave the SoS at runtime to meet the desired needs. SoS finds numerous applications in real-life domains, including energy grids, air traffic management, defense, and robotics swarms (Jamshidi, 2008). Managing the complexity of SoS is a major challenge. Traditional approaches often struggle due to the inherent autonomy and heterogeneous nature of CSs. Holonic architectures (Blair et al., 2015) offer a promising solution by decomposing the SoS into smaller, self-governing entities called holons (Koestler, 1968). This duality facilitates a recursive system architecture, allowing for self-reliance alongside cooperation with other holons forming a holarchy—a hierarchy of holons operating autonomously yet in coordination to achieve common objectives. The holonic approach aligns with key SoS architectural principles, such as interoperability, scalability, and adaptability. These principles motivate researchers to represent the CS of SoS as ‘holons’. This representation enables functionalities such as CS discovery and dynamic SoS composition (Elhabbash et al., 2024). While the holonic architecture offers a promising approach for SoS engineering, it can face substantial interoperability challenges. The heterogeneous CSs, or holons, often adhere to distinct data formats, communication protocols, and interaction patterns. This diversity creates interoperability hurdles, hindering seamless information sharing, command interpretation, and task collaboration requiring specialized knowledge for SoS understanding and implementation. Moreover, the dynamic nature of SoS, where CSs can join or leave, necessitates adaptive and flexible communication mechanisms to handle evolving SoS compositions. Finally, while in operation, SoS should have the capability to interact with humans, necessitating communication to expand towards a form that is immediately understood by humans. Empowering holons with Natural Language Processing (NLP) capabilities presents a transformative approach to overcoming the aforementioned communication hurdles. Such capabilities can enable holons to interpret and respond to natural language instructions, thereby simplifying the interaction between holons and humans and reducing the reliance on internal system knowledge. Furthermore, this approach facilitates holon-to-holon communication by encoding and decoding machine-executable commands into natural language. Thus, the NLP layer acts as a communication layer among humans and holons, agnostic to underlying CS heterogeneity and protocols, thereby enhancing overall collaboration and adaptability in SoS. Recent research explores integrating NLP technologies, especially Large Language Models (LLMs), into robots (Koubaa, 2023), leading to improved human-robot collaboration. However, these studies do not incorporate multi-robot functionality, which is essential for robot-to-robot communication. Furthermore, their scope is limited to robotics rather than to SoS and holonic architecture. In this paper, we propose extending the holonic architecture by integrating the NLP capabilities directly into the holons. We present a conceptual framework for NLP-enhanced SoS, enabling natural language interaction and decision-making. Through the utilization of advanced NLP technologies, such as LLMs, our approach facilitates seamless communication and collaboration within SoS. The framework’s practical application is demonstrated through an Unmanned Vehicle Fleet (UVF) case study. To the best of our knowledge, this is the first work of its kind to explore NLP-enhanced holonic architectures within the SoS domain. The remainder of this paper is organized as follows. Section 2 provides background information on NLP and holonic architecture. Section 3 reviews the state of the art in this topic. Section 4 details the proposed conceptual framework incorporating NLP into the holonic architecture. Section 5 demonstrates the application of the framework to the UVF case study. Section 6 presents ideas for evaluating the framework. Section 7 discusses our findings and their implications. Finally, Section 8 concludes the paper by drawing final conclusions and outlining potential avenues for future work. 2 BACKGROUND 2.1 Natural Language Processing Natural Language Processing (NLP) has become a cornerstone of artificial intelligence, facilitating communication between humans and computers (Khurana et al., 2023). It encompasses various techniques for enabling computers to understand, interpret, and generate human language. Among these techniques, Large Language Models (LLMs) have revolutionized the capabilities of machines in processing and generating human-like text. These LLMs, such as BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pre-trained Transformer), are typically based on complex neural networks trained on extensive text datasets (Zhao et al., 2023). Through training, these neural networks intricate language features, including patterns, structures, context, and semantics, enabling them to perform advanced such as text classification, sentiment analysis, translation, and question-answering (Radford et al., 2019; Brown et al., 2020). LLMs have found applications in diverse domains, including software development tasks like programming and code generation (Sadik et al., 2023b). However, their integration into broader human-system, inter-system, and intrasystem interactions is still in its early stages. This paper focuses on exploring this potential for broader integration. 2.2 Holonic Architecture for System of Systems Holons are autonomous yet connected entities that possess independent functionalities while contributing to a larger system (Koestler, 1968). Due to their dual nature, holons are excellent for modeling the heterogeneous CSs, accurately reflecting their independent functions and contributions to the overall SoS (Blair et al., 2015). 3 RELATED WORK Nundloll et al. (2020) utilized this concept of using holons to model IoT systems and introduced a framework that describes holons using ontologies. Elhabbash et al. (2024) adapted this framework to the SoS domain and proposed an SoS architecture where CSs are modeled as ontological holons. This architecture allows CSs to reason about and understand each other, facilitating CS discovery, ad-hoc scalability, and dynamic SoS composition. 2 However, this architecture assumes that the ontological descriptions of the holons are manually provided by vendors or systems engineers. Addressing this limitation, Zhang et al. (2023) propose an NLPbased approach that automatically extracts ontological descriptions of IoT devices by scraping web data. While this approach offers automation, the holonic architecture for SoS still lacks capabilities for holonto-environment communication (Halba et al., 2021), human-to-holon interaction, and communication with unknown holons. 4 CONCEPTUAL FRAMEWORK Our proposed framework aims to address the communication challenges inherent in SoS by leveraging NLP techniques with a particular focus on enabling holon-to-holon and holon-to-human communication. Figure 1 illustrates the framework depicting its components and their interactions. 4.1 Overview The framework involves a human operator who provides natural language instructions to the holons representing the CS of an SoS. The instructions can range from broad, high-level goals to specific tasks. The CSs of the SoS are represented by holons (e.g., Holon A and Holon B in Figure 1). Each holon possesses specific capabilities or services, which are the resources integrated into them. In addition, each holon is also equipped with a localized NLP module. These goals usually exceed the capabilities of a single holon and require the collaboration of multiple holons. The system then identifies relevant holons based on their capabilities and orchestrates their collaboration to achieve the operator’s objectives. 4.2 NLP Integration Module This module is the core of the framework. This module comprises three components: ontology-based prompt engineering, an LLM, and an NLP communication interface. 4.2.1 Ontology-based Prompt Engineering This component is responsible for crafting precise and contextually relevant prompts by incorporating domain-specific ontologies. This ensures that the operator’s inputs are tailored accurately to the specific needs of the SoS domain, thereby enhancing the effectiveness and precision of the LLM’s response (Section 4.2.2). 4.2.2 Large Language Model (LLM) This component utilizes NLP capabilities to understand and generate natural language interactions. It interprets the refined input from the prompt engineering component (Section 4.2.1), processes it, and converts it into a syntax comprehensible to other heterogeneous holons of the SoS. 4.2.3 NLP Communication Interface This interface translates the processed instructions from the LLM into actionable commands for the holons. These commands can activate or deactivate the holons’ capabilities or services as needed to achieve the overarching goal. The NLP module provides feedback to the human operator in the form of confirmation messages, clarifying questions, or summaries of the intended actions. This allows the operator to verify the system’s understanding and make any necessary corrections or refinements. 4.2.4 Holon-level NLP Each holon is equipped with a localized NLP module, including a domain-specific LLM, enabling it to understand and generate natural language instructions for both human-holon and holon-holon interactions. The human-holon communication occurs when the NLP Integration Module interacts with the localized NLP module of holons. The holon-to-holon interaction facilitates the holons to coordinate and exchange system descriptions of their encapsulated system with other holons using natural language (NL). As apparent in the Figure 1, the holon-to-holon interaction is possible even without the NLP Integration module. This distributed approach enhances autonomy and reduces reliance on a centralized module. 4.3 Holon Composition and Collaboration Holons, utilizing their embedded NLP capabilities, interact with each other and the central NLP module to determine their capabilities and relevance to the goal. Through an iterative negotiation process, holons commit to providing specific services, while others may be deemed irrelevant. The relevant holons then form a holon composition, collaborating to fulfill the given mission. 3 Natural Language Interaction Human Operator Refined prompt Ontology-based Prompt engineering NLP Communication Interface Capabilities / Services System of Systems Holon A Capabilities / Services Holon B Translates to/from actionable commands and holon descriptions NL Communication Large Language Model LLM Generate structured commands Refine Instructions (if needed) NLP Integration Module Figure 1: The conceptual framework showing NLP integration with the Holonic Architecture This approach streamlines the process of achieving complex goals within an SoS. Instead of requiring in-depth knowledge of each constituent system, the operator can simply communicate their intent in natural language, and the framework handles the rest. 5 CASE STUDY: NLP-INTEGRATED UVF FOR URBAN MOBILITY 5.1 Overview This case study explores the integration of the proposed NLP-enhanced holon communication framework (Section 4) into smart city transportation using Unmanned Vehicle Fleets (UVF). The UVF, operating within a dynamic environment, exemplifies an SoS with complex interaction dynamics and scalability challenges such as evolving missions, expanding range and capacity demands, vehicle failures, and battery limitations (Sadik et al., 2023a). It comprises several autonomous entities, including Unmanned Ground Vehicles (UGVs) and Unmanned Aerial Vehicles (UAVs), each functioning as a holon, operating independently as well as collaborating to achieve the fleet’s overall objectives (Tchappi et al., 2020). 5.2 Scenario Consider a scenario where a resident in a smart city needs transportation from Position A to Position B. The goal is to navigate the complex cityscape quickly and efficiently, considering no-fly zones, traffic conditions, and road layouts. This scenario presents several challenges: •Complex Urban Environment: Navigating through a densely populated urban area with varying altitudes and no-fly zones for UAVs. •Dynamic Routing: Adapting in real-time to traffic and environmental conditions to ensure the fastest and safest route. •Vehicle Coordination: Seamlessly transitioning between UAVs and UGVs while maintaining a consistent and comfortable experience. •Communication: Ensuring clear and efficient communication between the user, UVs, and the control center to manage expectations and adapt to any changes in the mission. 5.3 NLP-enhanced UVF Communication Figure 2 illustrates the integration of NLP module (Section 4.2) into the UVF communication framework. This integration enhances both human-to-fleet and intra-fleet communications. 5.3.1 Human-UV Interaction The user communicates their destination to the urban mobility service using a natural language interface. The NLP module’s ontology-based prompt engineering component (Section 4.2.1) refines this input, ensuring it’s tailored to the UVF domain. The LLM (Section 4.2.2) then interprets the request and initiates mission planning. 4 NLP Integration Figure 2: Human-UV Interaction and Planning 5.3.2 Intra-Fleet Negotiation To optimize efficiency, representative UVs from both the UGV and UAV swarms are selected to interact with the user and the urban mobility service. These representatives use their localized NLP modules (Section 4.2.3) to communicate with the rest of the fleet, negotiating roles, paths, and timing based on individual capabilities, current status, and environmental factors. This negotiation process aligns with Section 4.2.4 and Section 4.3 parts of the framework. For example, the UAV representative communicates its estimated time of arrival at the no-fly zone and available landing zones to the UGV representative. The UGV representative then analyzes potential routes, considering traffic conditions and road layouts, and proposes a suitable landing zone and rendezvous time. The UAV and UGV representatives iterate on this process until a mutually agreeable plan is reached. 5.4 Resulting UVF Composition Following the negotiation process, a UVF is formed to accommodate the no-fly zone (P1– P2in Figure 3). This composition includes two UAVs for aerial segments and one UGV for ground transportation, showcasing the dynamic SoS composition capability enabled by the framework. The UAV, upon reaching the no-fly zone, communicates with the UGV in natural language, I am approaching the no-fly zone. Please prepare to receive the passenger at the designated landing zone (Section 4.2.4). The UVs coordinate seamlessly, ensuring a smooth transition for the user at the landing and launch zones. Upon reaching the predetermined landing zone close to the user’s location (Position A), the first UAV communicates with the UGV to prepare for a smooth transition to the no-fly zone. The user is then transported by the UGV to a launch zone closer to his destination, where the second UAV takes over to complete the final leg of the journey. A B P1 P2 Figure 3: Resulting UVF of two UAVs and one UGV (P1— P2is no-fly zone) 5.5 Mission Success Criteria The success of the mission will be evaluated based on the time taken to complete the mission, the number of passengers successfully transported, the number of successful negotiations among UVs, and the overall satisfaction of the human operator with the system’s performance. These metrics will provide valuable insights into the effectiveness and usability of the NLPenhanced holonic architecture in real-world scenar5 ios. 6 EVALUATION While this paper primarily focuses on presenting a conceptual framework, we acknowledge the importance of evaluating its effectiveness in real-world scenarios. Future work will involve a rigorous evaluation of the proposed NLP-enhanced holonic architecture in a simulated environment, similar to the approach used by (Sadik et al., 2023a). The implementation can be done in by developing a multi-agent simulation using a suitable framework (e.g., JADE), or a multirobot environment using ROS 2 and Gazebo to model the interactions between the human operator, the NLP module, and the holons. Potential evaluation metrics could include: •Task Completion Rate: The percentage of tasks completed by the SoS using the NLP interface compared to traditional methods. •Communication Efficiency: The reduction in communication overhead (e.g., message volume, bandwidth usage) and time to complete tasks achieved through NLP-based interaction. •Communication Effectiveness: The percentage of correctly interpreted user requests and holongenerated instructions. •Adaptability: The ability of the SoS to dynamically reconfigure and adapt to changes in the environment or mission objectives, facilitated by NLP-based negotiation and coordination. •Usability: Qualitative assessment of the ease of use, intuitiveness, and user satisfaction of the NLP interface for both human operators and holons. By quantifying these metrics, we can assess the impact of NLP integration on SoS performance and identify areas for further improvement. 7 DISCUSSION The integration of NLP within the holonic architecture offers several advantages that contribute to the improved efficiency, adaptability, and usability of SoS. Firstly, by enabling natural language communication between the human operator and the holons, the framework reduces the cognitive load on the operator, who no longer needs to be familiar with the specific syntax or protocols of each constituent system. This streamlines the interaction process and allows for more efficient task assignment and coordination. Traditionally, achieving a goal in an SoS would require the operator to be familiar with the internal workings of each CS. The operator would then need to orchestrate these CSs together to design an SoS that accomplishes the goal. Secondly, the use of NLP enables the holons to dynamically negotiate and adapt their roles based on the given goal and the capabilities of other holons. This adaptability is crucial in complex and dynamic SoS environments where the composition of holons may change over time. Finally, the intuitive nature of natural language interaction enhances the overall usability of the SoS, making it more accessible to a wider range of users, including those without specialized technical knowledge. 8 CONCLUSION AND FUTURE WORK This paper introduces a novel approach to enhance holon communication within SoS through Natural Language Processing, aiming to bridge the communication gap between human operators and holons, and among holons themselves. The proposed framework demonstrates the potential for NLP to improve the efficiency, adaptability, and usability of SoS, paving the way for more intuitive and effective system-level collaboration. Overall, this field represents a promising area of ongoing research, with future developments expected to further refine and validate the proposed model in practical SoS applications. In the short term, leveraging the ROS2 platform (Daubaris et al., 2023), we are currently implementing the proposed framework using the Holon Programing Model (Ashfaq et al., 2024) and system architectures for autonomous robots (M¨ akitalo et al., 2021). While the integration of LLMs into the holonic architecture holds promise for enhancing interoperability and adaptability, future research should also address potential ethical concerns, such as privacy, safety, and conflicts of interest (Rousi et al., 2023; Levinson et al., 2024). Future work should also focus on addressing the challenges associated with natural language ambiguity, such as implementing clarification dialogs. This can involve exploring the use of domain-specific ontologies, controlled natural languages, clarification dialogs, and context-aware interpretation. Additionally, there is a need to integrate feedback mechanisms and validation techniques to 6 ensure robust and reliable communication within the SoS, enabling continuous learning and improvement of the NLP module. REFERENCES Ashfaq, M., Sadik, A. R., Mikkonen, T., Waseem, M., and M¨ akitalo, N. (2024). The holon programming model for heterogeneous and adaptive system of systems. TechRxiv preprint 10.36227/techrxiv.171387729.91629072/v2. Blair, G., Bromberg, Y.-D., Coulson, G., Elkhatib, Y., R´ eveill` ere, L., Ribeiro, H. B., Rivi` ere, E., and Ta¨ ıani, F. (2015). Holons: Towards a systematic approach to composing systems of systems. In 14th International Workshop on Adaptive and Reflective Middleware, pages 1–6. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. Daubaris, P., Linkola, S., Kantosalo, A., and M¨ akitalo, N. (2023). Getting started with ROS2 development: A case study of software development challenges. In 2023 IEEE/ACM 5th International Workshop on Robotics Software Engineering, pages 37–44. Elhabbash, A., Elkhatib, Y., Nundloll, V., Marco, V. S., and Blair, G. S. (2024). Principled and automated system of systems composition using an ontological architecture. Future Generation Computer Systems, 157:499–515. Halba, K., Griffor, E., Lbath, A., and Dahbura, A. (2021). A framework for the composition of IoT and CPS capabilities. In IEEE 45th Annual Computers, Software, and Applications Conference, pages 1265–1272. Jamshidi, M. (2008). Systems of systems engineering: Principles and applications. CRC press. Khurana, D., Koli, A., Khatter, K., and Singh, S. (2023). Natural language processing: State of the art, current trends and challenges. Multimedia tools and applications, 82(3):3713–3744. Koestler, A. (1968). The ghost in the machine. Macmillan. Koubaa, A. (2023). ROSGPT: Next-generation humanrobot interaction with ChatGPT and ROS. Preprints.org, 2023:2023040827. Levinson, L., Dietrich, M., Sarkisian, A., Sabanovic, S., and Smart, W. D. (2024). Privacy aware robotics. In Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction, pages 1335–1337. M¨ akitalo, N., Linkola, S., Laurinen, T., and M¨ annist¨ o, T. (2021). Towards novel and intentional cooperation of diverse autonomous robots: An architectural approach. In ECSA 2021 Companion, volume 2978 of CEUR Workshop Proceedings. Nielsen, C. B., Larsen, P. G., Fitzgerald, J., Woodcock, J., and Peleska, J. (2015). Systems of systems engineering: Basic concepts, model-based techniques, and research directions. ACM Computing Surveys, 48(2):1–41. Nundloll, V., Elkhatib, Y., Elhabbash, A., and Blair, G. S. (2020). An ontological framework for opportunistic composition of IoT systems. In IEEE international conference on informatics, IoT, and enabling technologies, pages 614–621. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. (2019). Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. Rousi, R., Samani, H., M¨ akitalo, N., Vakkuri, V., Linkola, S., Kemell, K.-K., Daubaris, P., Fronza, I., Mikkonen, T., and Abrahamsson, P. (2023). Business and ethical concerns in domestic conversational generative aiempowered multi-robot systems. In International Conference on Software Business, pages 173–189. Springer Nature Switzerland Cham. Sadik, A. R., Bolder, B., and Subasic, P. (2023a). A selfadaptive system of systems architecture to enable its adhoc scalability: Unmanned vehicle fleet-mission control center case study. In 7th International Conference on Intelligent Systems, Metaheuristics & Swarm Intelligence, pages 111–118. Sadik, A. R., Ceravola, A., Joublin, F., and Patra, J. (2023b). Analysis of ChatGPT on source code. arXiv preprint arXiv:2306.00597. Tchappi, I. H., Galland, S., Kamla, V. C., Kamgang, J. C., Mualla, Y., Najjar, A., and Hilaire, V. (2020). A critical review of the use of holonic paradigm in traffic and transportation systems. Engineering Applications of Artificial Intelligence, 90:103503. Zhang, Z., Elkhatib, Y., and Elhabbash, A. (2023). NLPbased generation of ontological system descriptions for composition of smart home devices. In IEEE International Conference on Web Services, pages 360–370. Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., Du, Y., Yang, C., Chen, Y., Chen, Z., Jiang, J., Ren, R., Li, Y., Tang, X., Liu, Z., Liu, P., Nie, J.-Y., and Wen, J.- R. (2023). A survey of large language models. arXiv preprint arXiv:2303.18223. 7