D2.1 - HARTU Architectural specification and integration plan

This project has received funding from the European Union’s Horizon Europe - Research and Innovation program under the grant agreement No 101092100. This report reflects only the author’s view and the Commission is not responsible for any use that may be made of the information it contains. D2.1 - HARTU Architectural specification and integration plan Deliverable ID: D2.1 Project Acronym: HARTU Grant: 101092100 Call: HORIZON-CL4-2022-TWIN-TRANSITION-01 Project Coordinator: TEKNIKER Work Package WP2 Deliverable Type Document, report Responsible Partner: Engineering Ingegneria Informatica S.p.a. (ENG) Contributors TEK, AIMEN, DFKI, ITRI Edition date: 22 December 2023 Version: 10 Status: Final Classification: PU D2.1 – HARTU Architectural specification and integration plan 2 HARTU Consortium HARTU “Handling with AI-enhanced Robotic Technologies for flexible manufactUring” (Contract No. 101092100) is a collaborative project within the Horizon Europe – Research and Innovation program (HORIZON-CL4-2022-TWIN-TRANSITION-01-04). The consortium members are: 1 FUNDACION TEKNIKER (TEK) Contact: Iñaki Maurtua [email protected] 2 DEUTSCHES FORSCHUNGSZENTRUM FUER KUENSTLICHE INTELLIGENZ GMBH (DFKI) Contact: Dennis Mronga [email protected] 3 ASOCIACIÓN DE INVESTIGACIÓN METALÚRGICA DEL NOROESTE (AIMEN) Contact: Jawad Masood jawad.maso[email protected] 4 ENGINEERING INGEGNERIA INFORMATICA S.P.A. (ENG) Contact: Riccardo Zanetti riccardo.zanet[email protected] 5 TOFAS TURK OTOMOBIL FABRIKASI ANONIM SIRKETI (TOFAS) Contact: Nuri Ertekin [email protected] 6 PHILIPS CONSUMER LIFESTYLE BV (PCL) Contact: Erik Koehorst [email protected] 7 ULMA MANUTENCION S. COOP. (ULMA) Contact: Leire Zubia [email protected] 8 DEEP BLUE Srl (DBL) Contact: Erica Vannucci [email protected] 9 FMI HTS DRACHTEN B.V. (FMI) Contact: Floris goet [email protected] 10 TECNOALIMENTI S.C.p.A (TCA) Contact: Marianna Faraldi [email protected] 11 POLITECNICO DI BARI (POLIBA) Contact: Giuseppe Carbone giuseppe.carb[email protected]t 12 OMNIGRASP S.r.l. (OMNI) Contact: Vito Cacucciolo [email protected] 13 INDUSTRIAL TECHNOLOGY RESEARCH INSTITUTE INCORPORATED (ITRI) Contact: Curtis Kuan [email protected].tw 14 INFAR INDUSTRIAL Co., Ltd (INFAR) Contact: Simon Chen [email protected] D2.1 – HARTU Architectural specification and integration plan 3 Document history Date Version Status Author(s) Description 31/07/2023 01 Draft ENG Table of Content definition 30/09/2023 02 Draft ALL Table of content finalization 09/10/2023 03 Draft ENG Added further descriptions in the sections; Added draft content in Sections 4 and 5; Added Section 6.2 23/10/2023 04 Draft ENG Started filling Abbreviations and Glossary of terms sections. Main responsibilities assignment 25/10/2023 05 Draft AIMEN Contribution to section 3.1 14/11/2023 06 Draft ENG Contribution to sections 2 and 5 11/12/2023 07 Draft TEK, AIMEN, DFKI, ITRI Contribution to 4.6 sub-sections, section 5 (Integration plan) and 4.7 (Requirements Traceability Matrix) 15/12/2023 08 Draft ENG Reviewing the entire document and addressing comments 18/12/2023 09 Draft ENG Quality check and content review 22/12/2023 10 Final ENG Version submitted D2.1 – HARTU Architectural specification and integration plan 4 Executive Summary The current deliverable (i.e., D2.1 - HARTU Architectural specification and integration plan) aims to report the main results of tasks T2.1 and provides relevant information about Reference Architecture or high-level specification of the HARTU software solutions that will be provided by technical partners (TEK, AIMEN, ITRI and DFKI) to address pilot needs and requirements. Tasks T2.1 covers the overall project approach on Reference Architecture model, reporting the results on technical drawings and software components specifications identifying key technical challenges to be addressed in HARTU, in terms of implementation and architecture definition. Considering this, the main inputs that contributed to the definition of the architecture and the specifications of the software are: • “D1.1 - Real world scenarios and metrics for validation definition”: an exploratory investigation of ‘requirements’ for the HARTU solution and its results, capturing both user level requirements and high-level requirements (i.e., functional, and non-functional) important to understand and extract the basic functionalities of the system. • “D1.2 Initial setup of real-world scenarios”: helps to extract ‘technical requirements’ considering the different prototypes corresponding to the 8 use cases defined by the project. The approach followed to complete deliverable D2.1 included: • The summary (ToC) and scope of the document agreed between the partners, and a detailed template including a description of each section and what was required to detail it, were disseminated to guide section leaders in collecting the required information. • A general overview of the manufacturing context and the reference architectures in Industry 4.0 is presented. • Specifications, definition of the reference model (Reference Architecture) in terms of technical drawings (HARTU Blueprint), definition of the architectural layers and FBB specification: starting from the requirements (D1.1) and the use case descriptions as well as prototypes (D1.2), the HARTU approach is represented using UML diagrams. • Definition of the integration plan that clearly shows “when” all functional blocks within each layer of the HARTU architecture will be delivered for testing and validation within industrial environments or test labs. • Support for the software development plan through a CI/CD enabled tool that allows rapid updating of the source codes of HARTU components, as well as code quality control and providing a valid repository for technical documentation and problem management. • Follow-up activities to monitor the progress of the work through different types of meetings: (1) General project follow-up web meetings (every three weeks) involving all partner representatives; (2) WP2 conference specific meetings; (3) Overall project F2F meeting. D2.1 – HARTU Architectural specification and integration plan 5 1 Table of contents 1 Introduction ................................................................................................................................. 9 1.1 Scope of this deliverable ..................................................................................................... 11 1.2 Relationship with other tasks .............................................................................................. 11 1.3 Structure of the document .................................................................................................. 12 2 Methodology .............................................................................................................................. 13 2.1 Layered Architecture ........................................................................................................... 13 2.2 Functional Building Blocks Specification ............................................................................. 14 3 The Context ................................................................................................................................ 15 3.1 Industry 4.0 reference architectures ................................................................................... 15 3.2 Background on ROS2 (Robot Operating System) ................................................................ 17 3.2.1 ROS2 Nodes .................................................................................................................. 18 3.2.2 ROS2 Messages ............................................................................................................ 18 3.2.3 ROS2 Services ............................................................................................................... 19 3.2.4 ROS2 Actions ................................................................................................................ 19 3.2.5 ROS2 Command Line Interface .................................................................................... 20 4 HARTU Adaptive Platform for Robotic Orchestration (APRO) ................................................... 21 4.1 Frontend Layer .................................................................................................................... 23 4.2 Application Layer ................................................................................................................. 24 4.3 Information Layer ................................................................................................................ 25 4.4 Middleware Layer ................................................................................................................ 25 4.5 Physical Layer ...................................................................................................................... 25 4.6 FBB Specification ................................................................................................................. 26 4.6.1 Frontend Layer FBBs Specification ............................................................................... 27 4.6.2 Application Layer FBBs specification ........................................................................... 29 4.6.3 Information Layer FBBs Specification .......................................................................... 34 4.6.4 Middleware Layer FBBs Specification .......................................................................... 37 4.7 Requirements Traceability Matrix ....................................................................................... 41 5 HARTU Integration Plan ............................................................................................................. 43 5.1 General Integration Status .................................................................................................. 43 5.2 Component Detailed Plan ................................................................................................... 43 5.3 Software Configuration Management ................................................................................ 45 D2.1 – HARTU Architectural specification and integration plan 6 5.3.1 Source code repository ................................................................................................ 45 5.3.2 CI/CD setup and automation ....................................................................................... 47 5.3.3 Quality control ............................................................................................................. 50 5.3.4 Guidelines to ease collaboration ................................................................................. 54 6 Conclusion .................................................................................................................................. 55 7 References.................................................................................................................................. 57 List of figures Figure 1: Relationships with other WPs and Tasks ............................................................................ 12 Figure 2. Component representing a Robot-asset and its Asset ....................................................... 17 Figure 3: ROS 2 Architecture Overview.............................................................................................. 18 Figure 4: Service communication for nodes in the ROS graph .......................................................... 19 Figure 5: ROS2 Action ......................................................................................................................... 20 Figure 6: HARTU Reference Architecture (Latest version) ................................................................. 22 Figure 7: HARTU FBB – Frontend Layer – AppManager (builder) Component Diagram ................... 27 Figure 8: HARTU FBB – Frontend Layer – AppManager (Control) Component Diagram .................. 28 Figure 9: HARTU FBB – Frontend Layer – SimEnv Component Diagram ........................................... 29 Figure 10: HARTU FBB - Application Layer – ImageAcquisition Component Diagram ...................... 30 Figure 11: HARTU FBB - Application Layer – ImageSegmentation Component Diagram .................. 30 Figure 12: HARTU FBB - Application Layer – GraspPlanner Component Diagram ............................ 31 Figure 13: HARTU FBB - Application Layer – PoseEstimation Component Diagram ......................... 32 Figure 14; HARTU FBB - Application Layer – ReleasePlanner Component Diagram ......................... 33 Figure 15: HARTU FBB - Application Layer – Adaptive MPC Component Diagram ........................... 33 Figure 16: HARTU FBB - Application Layer – Imitation Learning Component Diagram ..................... 34 Figure 17: HARTU FBB – Information Layer – Components Diagram ................................................ 35 Figure 18: HARTU FBB - Information Layer – HARTU_SegmentModel .............................................. 35 Figure 19: HARTU FBB - Information Layer – LocalGraspModel ........................................................ 35 Figure 20: HARTU FBB - Information Layer – GlobalGraspModel...................................................... 36 Figure 21: HARTU FBB - Information Layer – HARTU_PoseEstimationModel ................................... 36 Figure 22: HARTU FBB - Information Layer – DMPModel .................................................................. 36 Figure 23: HARTU FBB - Information Layer – ART Model .................................................................. 37 Figure 24: HARTU_GMMModel ......................................................................................................... 37 Figure 25: HARTU FBB - Middleware Layer – ART Classifier Component Diagram ........................... 37 Figure 26: HARTU FBB - Middleware Layer – Dynamic Movement Primitives Component Diagram38 Figure 27: HARTU FBB - Middleware Layer – Model Predictive Control Component Diagram ........ 39 Figure 28: HARTU FBB - Middleware Layer – BehaviorTree Component Diagram ........................... 40 Figure 29: HARTU FBB - Middleware Layer – DeepReinforcementLearning Component Diagram .. 41 Figure 30: GitLab "Issues" feature example ....................................................................................... 46 D2.1 – HARTU Architectural specification and integration plan 7 Figure 31: Docker based build process automated with GitLab in HARTU ....................................... 48 Figure 32: GitLab Runner setup for HARTU repositories ................................................................... 49 Figure 33: GitLab pipeline basic representation ................................................................................ 49 Figure 34: .gitlab-ci.yml -> pipeline definition to automate Docker build process within GitLab ..... 50 Figure 35: excerpt of Google Style Guides ......................................................................................... 51 Figure 36: excerpt of Common Weakness Enumeration (CWE) ........................................................ 51 Figure 37: extract from gitlab-ci.yml showing code-review setup in to a pipeline ........................... 53 Figure 38: extract from codeclimate.yml ........................................................................................... 54 Figure 39: HARTU repositories naming convention ........................................................................... 55 List of tables Table 1. ROS2 Commands .................................................................................................................. 20 Table 2. Main Functional Building Blocks (FBB) for HARTU Reference Architecture ........................ 26 Table 3. Requirements – Functional Building Blocks (FBB) traceability matrix ................................. 41 Table 4 HARTU FBB General Integration Status. ................................................................................ 43 Table 5. HARTU FBB Component, Detailed Integration Plan ............................................................. 44 Table 6. APRO SW component - programming language map .......................................................... 52 D2.1 – HARTU Architectural specification and integration plan 8 Acronyms List of the acronyms CAD Computer Aided Design CI/CD Continuous Integration/Continuous Delivery DDS Data Distribution Service DoA Description of Action FBB Functional Building Blocks GUI Graphical User Interface HARTU Handling with AI-enhanced Robotic Technologies for flexible manufactUring H2M Human to Machine ONNX Open Neural Network Exchange LMM Learning Movement Model GMM Gaussian Mixture Model WP Work Package PM Project Month SAM Segment Anything Model RA Reference Architecture ROS Robot Operating System SCM Software Control Management XML Extensible Markup Language YAML Yet Another Markup Language ToC Table of Contents UI User Interface D2.1 – HARTU Architectural specification and integration plan 9 1 Introduction Grasp, assembly, and release planning in manufacturing environments involves optimizing the sequence in which a robot or automated system picks up and place objects or assembles them. It aims to enhance efficiency by minimizing configuration time, improving process control and maximizing utilisation of resources, ultimately improving the overall production process. In grasp and release planning, considerations include the geometry, material and weight of objects, their pose, as well as the robot’s capabilities. The goal is to develop a strategy that allows the robot to efficiently grasp items, move them to desired locations, assemble them and release them appropriately. This planning often integrates with larger production planning systems to streamline operations and enhance productivity in manufacturing settings. Some common characteristics to build a reliable and efficient system to support these features, also from a hardware optimisation point of view, involve the use of some components such as the ones presented below: • Object Recognition: Implement a robust system for recognizing and identifying objects in the manufacturing environment. This can involve computer vision techniques or other sensing techniques, e.g., barcode readers. • Grasping Algorithms: Develop artificial intelligence algorithms that determine the optimal way for a robot to grasp an object based on its shape, size, and weight. This may involve predefined grasping strategies or machine learning approaches. • Path Planning: Integrate path planning algorithms to determine the most efficient route for the robot to move between different locations while carrying out manipulation tasks. • Robot Control Interface: Create an interface that allows seamless communication between the grasp and release planning system and the robot’s control system. This ensures the planned actions are executed accurately. • Sensors and Feedback: Implement sensors that provide real-time feedback on the success of grasping and releasing actions. This information can be used to adjust the plan if unexpected events arise. • Integration with manufacturing Workflow: Ensure that the manipulation planning system integrate smoothly with the broader manufacturing workflow, coordinating with other systems and processes. • Adaptability: Design the system to be adaptable to changes in the manufacturing environment, such as variations in object types, production volumes, or spatial configurations. • Test and Validation: Rigorously test the system under various conditions to validate its reliability, accuracy, and efficiency in different manufacturing scenarios. HARTU solution proposes to combine these elements, to create a comprehensive system that optimizes manipulation tasks in a manufacturing context, providing an architecture model (objective of this deliverable and of WP2) that will provide a structured framework for system design and integration. D2.1 – HARTU Architectural specification and integration plan 16 RAMI [1] presents a cubic model that provides a framework for a common understanding among the entities of an I4.0 System with respect to three axes: Layers axis: It describes the six functional levels into which manufacturing systems for Industry 4.0 can be divided. In each of these layers, service definitions abstractly describe the functionality provided to a layer N by a layer N-1. • Business layer: It orchestrates the high-level services to determine the status of the processes at a factory level. • Functional layer: It provides a definition of the high-level services offered by an asset and manages their access remotely. • Information layer: It acquires, processes and adapts the data from assets while ensuring its integrity and persistence. • Communication layer: It establishes architectural styles, message patterns and data formats to ensure interoperability. • Integration layer: It offers low-level services that enable access to the data and functionalities of the asset. • Asset layer: It represents the physical or logical entities with value for a company, including human beings. Life Cycle Value Stream axis: Supported by IEC 62890, it describes the operational status of the product, differentiating between product type and product instance: • Product type: A product in development stage constitutes a product type. • Product instance: A manufactured product constitutes an instance of a product type. Hierarchy Levels axis: It adopts some of the factory hierarchy levels of ISA 95 and ISA 88 (such as Enterprise, Work centres, Stations and Control device), while adding additional levels to make the factory hierarchy consistent with Industry 4.0: • Connected world: It represents a group of companies collaborating above the Enterprise level. • Field device: It represents devices that are directly involved in the manufacturing process below the Control device level. • Product: It represents the product in the factory hierarchy. Together with this cubic model, RAMI 4.0 also introduces the I4.0 Component as a participant of a manufacturing system [2]. An I4.0 Component consists of an asset (physical part) and an Asset Administration Shell or AAS (virtual part). I4.0 Components are service-oriented: the AAS provides the asset an interface through which service requests from other I4.0 Components are channelled. Within the AAS, service requests are handled by a Component Manager that manages the AAS submodels (also called Manifest), made up by the set of properties that describe the data and functionalities of the asset. Services provided by a I4.0 component can be grouped into two categories as seen Figure 2: D2.1 – HARTU Architectural specification and integration plan 17 • Submodel Services: They provide access to the information submodels of an I4.0 component without interacting with the asset. • Asset Related Services: They involve an interaction with the asset to execute some functionality or operation, or to manage its state. Services provided by I4.0 Components can be combined to compose manufacturing applications that are offered to other components, resulting in Application Relevant Services [2]. Figure 2. Component representing a Robot-asset and its Asset 3.2 Background on ROS2 (Robot Operating System) ROS2 is a middleware, a software system dedicated to handling connections and data exchange between users and applications. It is based on an anonymous publish/subscribe mechanism that allows for message passing between different processes. ROS2 includes a set of software libraries and tools to build robot applications. It contains a great number of drivers, state-of-the-art algorithms, and developer tools. Among these key tools, the ROS core includes tools that can be helpful to visualize data, navigate package structures, and create scripts that automate complex configurations. Examples include “rviz”, a 3D visualizer used to present robots, sensor data, and their environment. Furthermore, additional packages are available for SLAM (Simultaneous Localization and Mapping), navigation, perception, and simulation, to name a few. ROS2 handles a network of nodes in a system and the connections that allow them to communicate, this is known as the ROS2 graph. A general overview of the architecture behind ROS2 is shown in the following image: D2.1 – HARTU Architectural specification and integration plan 18 Figure 3: ROS 2 Architecture Overview Overall, the main improvements from ROS to ROS2 include an improved communication stack with real-time data distribution service (DDS) protocol, logging improvements, the ability to configure Quality of Service in an easier fashion, provide improvements to the command-line interface, improved support for continuous integration and deployment, among others. ROS2 provides a more robust, flexible, and powerful solution for robotic applications. A brief introduction of nodes, messages, services, actions, command line interface, and launch tools in ROS2 follow in this section of the document. 3.2.1 ROS2 Nodes ROS2 nodes are the basic computation unit of the network. Each node represents a single process running in the system. They use the ROS2 Client Library to communicate with other nodes. ROS2 applications typically communicate through interfaces of one of three types: messages, services, or actions. Nodes can send and receive messages via buses known as topics, which can be user-defined and include anything from sensor data to actuator commands. On the other hand, services and actions provide a single result for each call. These nodes also possess a shared database that allows for communal access to static and dynamic information. This is known as a parameter server. 3.2.2 ROS2 Messages Nodes can publish messages to named topics to transmit data to other nodes or subscribe to topics to receive messages from other nodes. For example, a node can be focused on reading and handling D2.1 – HARTU Architectural specification and integration plan 19 information from a camera and publish an image message on a specific topic so the image can be accessed by the whole system. Messages are written in the ROS Message Description Language (.msg) and are used to describe how the data being sent is structured. Messages can contain many different types of data, from simple types like integers to more complicated ones such as images and custom messages. Figure 4: Service communication for nodes in the ROS graph 3.2.3 ROS2 Services Nodes can also act as services. Services handle a specific computation or data on request from another node. These are optimized for handling quick computations since the node requesting is usually on hold until it receives a response from the service request. A node that wants to request a computation or data (client) sends a request message to another node (server), which fulfills the request and sends a response to the client. A service is defined by a service server and a service client. The service server's external behavior is defined by “.srv” files in the “srv/” directory, where the input and output of the service is defined. Service clients are the nodes that send the input information and request an output. 3.2.4 ROS2 Actions If there is a need for handling longer computation requests, ROS2 provides a solution: actions. Actions can send feedback on the status of the long-running computation and can be interrupted. Actions are defined by “.action” files in the “action/” directory. D2.1 – HARTU Architectural specification and integration plan 20 Figure 5: ROS2 Action 3.2.5 ROS2 Command Line Interface ROS2 also provides a set of commands for introspecting and working with nodes, topics, services, and more. These commands are accessible through the main command “ros2”. A list of some commands is shown in the following table: Table 1. ROS2 Commands Command Description action Introspect/interact with ROS2 actions. bag Record/play a rosbag. launch Run/introspect a launch file. node Introspect ROS2 nodes. param Introspect/configure parameters on a node. run Run ROS2 nodes. service Introspect/call ROS2 services. topic Introspect/publish ROS2 topics. The use of these commands should be done in the following way: ros2 <command> <verb> <flags> Where “<command>” can be one of those listed in the previous table, “<verb>” further specifies the action of the command (list, echo, pub), and “<flags>” help define the behavior of the command (e.g., list of topics or loop flag for rosbags). For example, the topic command can be used to display all the existing topics: ros2 topic list For more information on the command line interface, the “--help” flag can be used in different scenarios: ros2 –help D2.1 – HARTU Architectural specification and integration plan 21 ros2 <command> --help ros2 <command> <verb> --help ROS2 Launch Since a ROS2 system typically consists of many nodes running across different processes, the launch system provides a way to automate the running of many nodes with a single command. This system utilizes a “.launch” file, which can be written in Python, XML, or YAML, to describe the configuration of the system. This configuration may include instructions to use the ROS2 commands, run programs, and define arguments. These instructions can then be run using the “ros2 launch” command. 4 HARTU Adaptive Platform for Robotic Orchestration (APRO) HARTU Reference Architecture (RA) is designed as multi-layered architecture and built on top of functional building blocks methodology. Inspired by the requirements collected in WP1, the technical design of the architecture was also started in WP2 with the collaboration of the main technological partners involved (i.e., ENG, TEK, AIMEN, DFKI, ITRI). Figure 6 shows the final version of the HARTU RA, obtained after several iterations with the partners involved, through brainstorming meetings and resolution sessions which allowed the technical group to release substantial parts of the architecture: D2.1 – HARTU Architectural specification and integration plan 22 Figure 6: HARTU Reference Architecture (Latest version) D2.1 – HARTU Architectural specification and integration plan 23 Each layer in the architecture communicates with the adjacent through well-defined interfaces or APIs, enabling loose coupling and facilitating independent development and testing of each layer. This separation of concerns allows for easier maintenance, scalability, and the ability to replace or modify individual layers without affecting the entire system. Interoperability between the layers will be guaranteed by the interfaces offered by the ROS2 in HARTU, its architecture being based on this technology; however, any other interaction mechanisms (standard protocols, third-party libraries) are not excluded should there be a need to solve specific integration problems using custom solutions. Furthermore, to address complex problems in software solution design, the Functional Building Blocks (FBB) approach was applied to better specify the technical structure of HARTU RA layers (see section 2.2). Valid graphics and physic engine tools (e.g., Gazebo 5 , Mujoco 6 , Unity 7 , etc.) will be provided to test machine learning or deep learning models to build software applications without compromising real data (e.g., production data). Functional building blocks will implement added value services able to cover all HARTU use cases and meet the needs of the end user. The HARTU RA is composed of the following layers: • Frontend Layer • Application Layer • Information Layer • Middleware Layer • Physical Layer Some common features between the layers can be summarized as follow: • Each layer can include one or more functional (e.g., Perception in the Application Layer) or technological block (HARTU UI Block in the Frontend Layer). • Each main block can include sub-blocks (and so on), which extend the main one (not in the current version of the HARTU RA). • Each block (and sub-block) can contain components/modules/services/software artifacts and technologies. Components (generic term to express the list of solutions) expose a set of interfaces (APIs) that allows them to communicate with other blocks (in the same layer) or externally with other layers of the architecture. Internally, each component can implement interfaces that perform operations useful for carrying out activities (e.g., tasks, operations, etc.) common to the functional or technological area to which it belongs. 4.1 Frontend Layer The Frontend Layer is about Human-to-Machine (H2M) interaction with software systems exposing high-level functionality or interfaces (UI) for HARTU operators and system integrators. GUI and simulation tools are the main components of this layer, which provide visual representations of information about the process to be monitored/controlled and business functions that will provide 5 https://gazebosim.org/home 6 https://mujoco.org/ 7 https://unity.com/ D2.1 – HARTU Architectural specification and integration plan 24 outputs to the end user. Simulation functionality to design new physical solutions in realistic environments with synthetic or real-time data flows will be also provided. HARTU end users will interact with this layer using dashboards (graphical user interfaces) to perform operations such as to learn new assembly skills or start the execution of an application. The main interaction based on ROS 2 communication protocols (mostly) will be with the Application layer, which will provide the highest-level business functions of the architecture (detail below). 4.2 Application Layer The Application Layer contains interfaces (APIs) and provides services for frontend application processes; transmission forward the requests to the level of presentation (Frontend Layer), while receiving them. Communication with the ROS2 layer typically occurs via interfaces of the following three types: • messages (topics), • services • and actions. Topics are used for data streams (sensor data, robot state), services to execute remote procedures fast, while actions are used for any discrete behaviour that moves a robot or runs for a longer time but provides feedback during execution. Communication between the Frontend and Application layers in HARTU will be provided by ROS2, although other communication protocols can be used if there is a real need. The core components for robotic interaction will be exposed in this layer; heterogeneity in the technologies that express the functional blocks in this layer will represent one of the main characteristics of this layer that will host ROS 2 compliant applications, but also components developed ah-hoc in any programming language to meet a specific end user need. However, this level supports the following functional application areas of robotics applications: • Perception, • Robot Application Planning & Control • and Grasp/Release Planning. The Perception block will contain components and technologies capable of providing functions to support operators in perception (through the Frontend layer tools) of static and dynamic objects to build a reliable representation and detailed robot environment using computer vision and machine learning techniques. The perception block in a robot is therefore responsible for object detection, segmentation, and tracking. The component APIs of this block will interact at a lower level mainly with the Grasp/Release Planning block components, and internally with the components of the same block (e.g., the Image Segmentation component will identify and separate parts to be handled by the robotic system within an image provided by the Image Acquisition component). The Robot Application Planning & Control block will provide components for the execution of tasks, such as controlling the trajectory of the robot (Planning), and position movement (Control). In all robotic applications, completing a generic task requires execution of a specific action prescribed to D2.1 – HARTU Architectural specification and integration plan 25 the robot. The correct execution of this action is entrusted to the Robot Application Planning & Control Block, that should take care of it with commands consistent with the desired action. The Grasp/Release Planning block contains components and services for grasp detection and planning. The technologies of this block are particularly important because allow a robot to support humans in daily work. 4.3 Information Layer The Information Layer encompasses dedicated data stores to persist information related to various aspects of the system. This support will be of different nature, and will manage heterogeneous information, serving the applications and services in the Application layer that will need to consume data (e.g., analytics data, time series etc), or instantiate a specific data model to train AI algorithms, or model information for portability of applications between different execution environments. Furthermore, The HARTU reference models are also included in this layer (e.g., ONNX, SAM, DMP, GMM, etc.) like “placeholders” to make understandable the plethora of modelling solutions that the project will make available for system integrators. 4.4 Middleware Layer The Middleware Layer is an abstraction software between an operating system and the applications running on it. It essentially functions as a hidden translation layer, allowing communication between the Application layer and Operating System libraries regardless of their implementation. It effectively abstracts the high-level functions of the HARTU architecture from the dependency of the main OS like Linux, Windows, or Mac. For ROS 2 the decision was made to build it on top of an existing middleware solution (i.e., DDS or Data Distribution Service). The main advantage of this approach is that ROS 2 can leverage an existing, well-developed implementation of that standard. There are many different implementations available, and each has advantages and disadvantages in terms of supported platforms, programming languages, performance characteristics, memory space, dependencies, and licenses. To abstract from the specifics of these APIs, an abstract interface has been introduced that can be implemented for different DDS implementations. This middleware interface defines the API between ROS 2 client library and any specific implementation. Each interface implementation will usually be a thin adapter that maps the generic middleware interface to the middleware implementation-specific API. The HARTU Client Library block contains a series of AI and non-AI libraries, ROS 2 compliant or simply third-party libraries, which will provide functionality and algorithms to the Application layer blocks. AI algorithms use machine learning, deep learning and so on techniques to train training models, therefore, to make accurate predictions. 4.5 Physical Layer The Physical Layer represents the lowest layer of the architecture where the partners/end-user facilities (laboratories, shop floor, etc.) are located. The layer also defines the physical components of the system, including production equipment, product parts, sensors, and machinery (e.g., robots). Here, the data relating to the assets will be produced in real-time which will be used to feed the decision-making process, the AI algorithms, and the business functions in the Application layer. D2.1 – HARTU Architectural specification and integration plan 32 • Receives the SegmentationMask data from the ImageSegmentation module. • Estimates and publishes the position and orientation of each object. If the request is performed on an object with a known CAD, this module uses the HARTU_PoseEstimationModel for estimation. If the request is performed on planes or primitive shapes, the pose is calculated with classic algorithms. Figure 13: HARTU FBB - Application Layer – PoseEstimation Component Diagram 4.6.2.5 ReleasePlanner This component is responsible for generating the robot motion to place an object in the destination position. It provides different decision alternatives depending on the application. • In some cases it needs to monitor the way the object has been picked (to create a mosaic), information provided by the GraspedPartMonitor component. • It uses the information provided by the ReleaseZoneMonitor component to know the free space available at the target area, e.g., the free space in a container where we need to create a mosaic with the picked parts. • Sometimes it needs to identify the reference picked by means of the Barcode label, information provided by the BarCodeReader Component. • It calls the MosaicGenerator component to estimate the best pose of the part in the target container (in the case of mosaic generation). In some particular cases, it is an external system who calculates it (e.g., a warehouse system). D2.1 – HARTU Architectural specification and integration plan 33 Figure 14; HARTU FBB - Application Layer – ReleasePlanner Component Diagram 4.6.2.6 AdaptiveMPC Adaptive MPC is a torque-based robot controller that adapts its control parameters to the current situation using Gaussian Mixture Regression (GMR) 11 . The current situation can be deduced from the given task and the perceived contact forces using ART-based contact classification. Figure 15: HARTU FBB - Application Layer – Adaptive MPC Component Diagram Interfaces: • [In] Reference Pose: Desired position/orientation of the robot as continuous data stream • [In] Task: The current assembly task at hand. Will be used to select the appropriate set of control parameters. 11 https://github.com/AlexanderFabisch/gmr D2.1 – HARTU Architectural specification and integration plan 34 4.6.2.7 ImitationLearning Imitation learning provides an architecture for intuitive specification and execution of assembly tasks on robotic manipulators (singleor dual-arm), i.e., tasks that are subject to complex contact forces with the environment. It can be operated in two modes: Recording (Learning) Mode, where the operator teaches a task, and the system generates a dynamic movement primitive (DMP), and Execution Mode where a pre-learned DMP is executed given a task specific goal pose, which typically is provided by the perception system of the robot. Figure 16: HARTU FBB - Application Layer – Imitation Learning Component Diagram Interfaces: • [In] Goal Pose: Target pose of the DMP, provided by the perception system. • [In] Record/Store: Trigger to start recording of the trajectory. • [In] Execute: Trigger to start execution of an assembly task. • [In] Activate Hand Guiding: Trigger to activate hand guiding mode. 4.6.3 Information Layer FBBs Specification The Information Layer contains one functional block or Data & Models (MOD.IL.DEM). This block interacts with other Application Layer’s functional blocks and with the components of each block using its data models as input to perform certain operations or for configuring the component to start its operational phase within the system. Figure 17 shows the internal structure of the component and the main relationships with the components of the Application Layer blocks: D2.1 – HARTU Architectural specification and integration plan 35 Figure 17: HARTU FBB – Information Layer – Components Diagram 4.6.3.1 HARTU_SegmentModel It is the result of training YOLOv5 with the set of images created for a specific part reference, by the SegmentModeller component. Later, the ImageSegmentation component uses this model as input for the SAM model that, finally, provides the segmented image. Figure 18: HARTU FBB - Information Layer – HARTU_SegmentModel 4.6.3.2 LocalGraspModel The LocalGraspModel contains the information of the valid grasping points for a given part reference and gripper. It is created by the LocalGraspModeller offline, once per object. Figure 19: HARTU FBB - Information Layer – LocalGraspModel 4.6.3.3 GlobalGraspModel This model includes the configuration for two different Global grasping strategies: • The weights for the heuristic version • The weights of the trained DRL model for the AI-based version Additionally, the model includes the Identifier of the end effector and others that will be defined during the development of the GlobalGraspModeller (scheduled by M24). D2.1 – HARTU Architectural specification and integration plan 36 Figure 20: HARTU FBB - Information Layer – GlobalGraspModel 4.6.3.4 HARTU_PoseEstimationModel This includes the configuration of model parameters and the weights of the trained models. The configurations are included in YAML files, while the model weights are saved as an ONNX model. Figure 21: HARTU FBB - Information Layer – HARTU_PoseEstimationModel 4.6.3.5 HARTU_DMPModel Represents the parameters of a dynamic movement primitive, e.g., dimension, weights of the forcing term, execution time, etc. YAML files are used for disk I/O. Figure 22: HARTU FBB - Information Layer – DMPModel 4.6.3.6 HARTU_ARTModel Represents the parameters of the neural networks used inside the ART classifier, in principle only the weights of the neurons. MessagePack 12 is used for disk I/O. 12 https://msgpack.org/ D2.1 – HARTU Architectural specification and integration plan 37 Figure 23: HARTU FBB - Information Layer – ART Model 4.6.3.7 HARTU_GMMModel Represents the parameters of a Gaussian Mixture Model, i.e., priors, means, covariances of the Gaussians. According to the implementation by Fabisch 13 . Pickle 14 is used for disk I/O. Figure 24: HARTU_GMMModel 4.6.4 Middleware Layer FBBs Specification The Middleware Layer contains one functional block or HARTU Client Library (MOD.ML.CLL). Mainly this block contains low-level components native to the ROS operating system, which interact directly with the robot and in general with the physical part of the system. 4.6.4.1 Adaptive Resonance Theory (ART) Classifier ROS2 component for classification of contact situations based on proprioceptive robot sensors. Figure 25: HARTU FBB - Middleware Layer – ART Classifier Component Diagram Interfaces: 13 https://github.com/AlexanderFabisch/gmr/ 14 https://docs.python.org/3/library/pickle.html D2.1 – HARTU Architectural specification and integration plan 38 • [In] Joint torques: Measured motor torques of the manipulator. • [In] Contact wrenches: Measured force/torque at the end effector. • [Out] Category: Predicted label of the contact situation. 4.6.4.2 Inverse Reinforcement Learning The component for inverse reinforcement learning module has not yet been specified, as the corresponding work has not started yet. 4.6.4.3 Dynamic Movement Primitives ROS2 component for recording, storing, and executing singleor dual-arm Dynamic Movement Primitives (DMP). It uses an open-source movement primitives library provided by DFKI 15 Figure 26: HARTU FBB - Middleware Layer – Dynamic Movement Primitives Component Diagram Interfaces: • [In] Record Trajectory: ROS2 action server which records the pose of the robot end effector(s) and stores it as a time-dependent trajectories. • [In] Generate DMP: ROS2 action server which generates a DMP from the recorded data and stores it as yaml-file. • [In] Execute DMP: ROS2 action server which executes the generated DMP point-by-point given the initial and desired final pose. • [In] Transforms: The ROS2 robot frame transformation tree. • [In] Contact Wrench: The force/torque measured at the robot end effector. 15 https://github.com/dfki-ric/movement_primitives D2.1 – HARTU Architectural specification and integration plan 39 • [Out] Commanded Pose, Commanded Pose left/right: Next end effector pose(s) computed by the DMP. 4.6.4.4 Model Predictive Controller A robot controller which uses optimization to regulate one or multiple tasks while respecting the physical constraints of the robot and the environment. It is based on the Crocoddyl library 16 for optimal multi-contact point control. Figure 27: HARTU FBB - Middleware Layer – Model Predictive Control Component Diagram Interfaces: • [In] Reference Pose: Desired end effector position/orientation • [In] Constraints: Constraints of the physical system to control. • [In] Costs: Cost functions to optimize. • [In] Control parameters: E.g., stiffness and damping. • [Out] Commanded Torques: Joint torques to achieve all desired tasks while respecting the physical constraints. 16 https://github.com/loco-3d/crocoddyl D2.1 – HARTU Architectural specification and integration plan 40 4.6.4.5 Behaviour Trees ROS2 component that allows the execution of a behaviour tree previously created through the AppManager (Builder). • It makes use of the behaviortreecpp_v3 library. • It takes as input the AppConfigurationFile generated with the AppManager (Builder). • The BTExecutor is the component in charge of loading our custom nodes using the behaviortreecpp_v3 library and the structure of the tree defined in the AppConfigurationFile. • Then, the BTExecutor sends the ticks to the tree in a main loop and verifies the result (Success or Failure). Figure 28: HARTU FBB - Middleware Layer – BehaviorTree Component Diagram 4.6.4.6 Deep Reinforcement Learning The component is used by the GlobalGraspPlanner to define the model to select the part to be picked in a cluttered scene. D2.1 – HARTU Architectural specification and integration plan 41 Figure 29: HARTU FBB - Middleware Layer – DeepReinforcementLearning Component Diagram 4.7 Requirements Traceability Matrix The traceability matrix maps in a table the identified Functional Building Blocks (FBB) and presented in section 4.6 of this deliverable with the requirements identified in section 9 of D1.1, highlighting which block implements and satisfy a specific requirement, as well as to give a clear understanding of each block objectives. Some requirements contained in D1.1 are not contained in this table as they will not have an impact in terms of functionality in the demonstrators (thus, they will not be included in the prototypes). Furthermore, MOD.ML.CLL functional block has not been included in the table because it is a purely back-end and not directly connected to the requirements which are mostly satisfied in the application layer. The following table also contains different types of requirements like User, Functional and NonFunctional ones: Table 3. Requirements – Functional Building Blocks (FBB) traceability matrix Requirements MOD.FL.UIT MOD.AL.PER MOD.AL.APC MOD.AL.GRP MOD.IL.DEM FR-01 X X X X FR-02 X X FR-03 X X X FR-04 X X FR-05 X X FR-06 X X FR-07 X X FR-08 X X X FR-09 X FR-10 X FR-11 X FR-12 X X X FR-13 X D2.1 – HARTU Architectural specification and integration plan 48 Figure 31: Docker based build process automated with GitLab in HARTU Starting from the left, when a developer pushes to the 'develop' branch of the HARTU repository, this action triggers a job execution in GitLab CI/CD pipeline. The job is pre-configured and relies on GitLab Runner, as introduced previously in §5.3.1, a component dedicated to executing jobs for different stages, such as the “build” one, dependent on project-specific configurations. In this workflow, the Docker-based Runner, configured at the group level to be accessible across all HARTU repositories within the project (Figure 32), automatically downloads the latest source code version from the GitLab server. Subsequently, it initiates the building of the Docker image for the specific component. If errors occur during the build, the job is halted, and an automatic notification is sent back to the developer via the GitLab server. D2.1 – HARTU Architectural specification and integration plan 49 Figure 32: GitLab Runner setup for HARTU repositories Upon the successful completion of image construction, the Docker image is pushed to the internal private registry 22 , a service provided with the GitLab server which significantly simplifies the continuous delivery (CD) in the early stages, at least. In fact, this internal registry makes the Docker image of a HARTU component accessible for download and testing potentially in any remote locations, such as the different HARTU laboratories. Figure 33: GitLab pipeline basic representation The picture above shows the basics of a GitLab pipeline, which are: • Jobs are the fundamental elements of a CI/CD pipeline as contain the instructions of the actions to be performed. • Jobs are grouped in stages determining when they must be executed. • One stage can start only if all the jobs of the previous stage have been completed successfully. • Jobs belonging to one stage can be executed in parallel. • Each job is executed by a GitlLabRunner executor running outside the GitLab server. The following picture illustrates the pipeline definition configured for a SW component that can be Dockerised for the deployment. 22 https://docs.docker.com/registry/ D2.1 – HARTU Architectural specification and integration plan 50 Figure 34: .gitlab-ci.yml -> pipeline definition to automate Docker build process within GitLab 5.3.3 Quality control The quality control of the source code, a good practice also known as code-review, consists of verifying lines of code written against rules which are defined by coding style conventions; different conventions are available for any programming language, such as those listed and made publicly available by Google style guides 23 . On one hand, this good practice primarily ensures the seamless maintainability 24 of the source code, encompassing changeability, modularity, understandability, testability, and reusability. On the other hand, by proactively identifying bugs, undefined behaviour, and risky coding constructs, this type of test contributes to improving program execution and, consequently, enhances quality at the runtime level. In the contemporary landscape, developers have access to various tools and platforms that facilitate the automatic verification of coding standards and conventions, ensuring readability and monitoring code complexity within acceptable bounds. Consequently, significant attention has been directed toward determining the coding standards, conventions, and best practices for the APRO modules, along with the supporting tools necessary for implementation. These tools predominantly function as linting utilities, conducting a form of static analysis aimed at identifying problematic patterns or weaknesses, such as those outlined in the Common Weakness Enumeration (CWE) 25 , and ensuring code adherence to specific style guidelines. 23 https://github.com/google/styleguide#google-style-guides 24 https://www.it-cisq.org/standards/code-quality-standards/ 25 https://www.it-cisq.org/pdf/cisq-weaknesses-in-ascqm.pdf D2.1 – HARTU Architectural specification and integration plan 51 Figure 35: excerpt of Google Style Guides Figure 36: excerpt of Common Weakness Enumeration (CWE) A linter, as a static code analysis tool, plays a crucial role in scrutinizing source code to flag potential issues, including programming errors, bugs, stylistic errors, and suspicious constructs. In the followings are briefly outlined relevant linters for the most widely used programming and scripting languages within the APRO components. Additionally, a table is included below (¡Error! No se encuentra el origen de la referencia.), mapping these languages to the respective components where they are going to be adopted. • C++: Cppcheck 26 is a static analysis tool tailored for C/C++ code, offering distinctive code analysis capabilities to identify bugs. Its emphasis lies in the detection of undefined behaviour and hazardous coding constructs, with the aim of minimizing false positives. Additionally, Cppcheck is engineered to analyze C/C++ code, even when it exhibits non-standard syntax, a feature particularly relevant in embedded projects. • Python: PEP8 Style Guide for Python Code 27 establishes coding conventions for Python code, encompassing the standard library included in the primary Python distribution. • JavaScript: ESLint 28 is a static code analysis tool designed to detect problematic patterns present in JavaScript code. The rules within ESLint are configurable, allowing for the definition and loading of customized rules. ESLint addresses both code quality and coding style issues. 26 https://cppcheck.sourceforge.io/ 27 https://peps.python.org/pep-0008/ 28 https://eslint.org/ D2.1 – HARTU Architectural specification and integration plan 52 Table 6. APRO SW component - programming language map Programming language APRO component Python C++ C# APP Builder X ImageAcquisition X ImageSegmentation X PoseEstimation X X ReleasePlanner X X SegmentModeller X GlobalGraspPlanner X X GraspPlanner X ART Classifier X Dynamic Movement Primitives X X Adaptive MPC X X SimEnv X X As previously mentioned, a linter is a tool used for coding reviews with the goal of enhancing code quality. Numerous linters are available for various programming languages, and many developers currently integrate these tools into their preferred IDEs. Some have automated them as an additional step in their CI processes, while others employ them in both ways, a practice being adopted and in preparation for APRO components. To automate these checks within the APRO CI/CD process outlined in §5.3.2, the "Code Quality" feature of GitLab has been explored. This feature, along with its widget, facilitates the seamless integration of linters by executing them through the GitLab Runner. The GitLab Runner, as previously discussed, is a GitLab component dedicated to running code for building and testing stages based on specific configurations. In this case, existing configurations for the APRO CI/CD pipelines can be extended to allow linter execution. Utilizing the GitLab "Code Quality" feature and its widget, linters based on Code Climate 29 engines 30 are employed. These components are modular plugins available for nearly any programming language, offering extensibility and open-source freedom. In essence, a Code Climate Engine is a Docker Image that invokes a program to parse a configuration file and analyse source code files, potentially generating formatted output indicating detected issues. Through a wrapper within the GitLab Code Quality project 31 , these engines can be easily integrated and run within pipelines, starting from a Docker image built within the project itself. This image includes default Code Climate configurations, and only the Docker image(s) for the specific linter(s) need to be pulled. Default configurations can be overridden to meet the specific needs of each APRO 29 https://codeclimate.com/quality 30 https://docs.codeclimate.com/docs/list-of-engines 31 https://gitlab.com/gitlab-org/ci-cd/codequality D2.1 – HARTU Architectural specification and integration plan 53 component (languages, types of quality checks) by creating dedicated config files for each APRO module repository. The benefits of this solution are numerous, including: • Integration into GitLab's pipelines 32 in consistent alignment with the CI/CD workflow described in §5.3.2 and the Docker-based approach for deployments. • Displaying a comprehensive list of code quality violations generated by a pipeline at the end of the process. • Utilizing Code Climate Engines, which are free and open source. • No subscription requirement. • Extensibility with new plugins. From a practical standpoint, to configure these engines for automatic execution in GitLab, in addition to having a pre-configured GitLab Runner for job execution, two files need setup. These files are the .gitlab-ci.yml (already present if a pipeline has been defined, such as for automating testing and building, as introduced in §5.3.2) and the .codeclimate.yml. Both of them must be placed in the root folder of the GitLab repository, and their content is briefly outlined as follows. .gitlab-ci.yml: The excerpt displayed in Figure 37 illustrates a section of this configuration file. To execute the job involving the linters execution, three key configuration lines have been added, highlighted by as many blue bullets in the figure. The first step involves incorporating the template, where a significant portion of the necessary work is already predefined. Following this, it's essential to add a "test" stage and a job named "code_quality" to seamlessly integrate with the template. Within the "code_quality" job, settings can be configured to override defaults set in the template. For instance, to enhance security and performance 33 , Docker-in-Docker is deactivated, and a custom tag (named "linter") is established to ensure this job runs exclusively on a private runner with a matching tag. Finally, the configuration specifies the file name and format 32 https://docs.gitlab.com/ee/ci/pipelines/ 33 https://docs.gitlab.com/ee/ci/testing/code_quality.html#improve-code-quality-performance-with-private-runners Figure 37: extract from gitlab-ci.yml showing code-review setup in to a pipeline D2.1 – HARTU Architectural specification and integration plan 54 for collecting job output, making it accessible to users and developers for download as an artifact from the GitLab GUI. .codeclimate.yml: This file 34 , as shown in Figure 38, allows to specify the linters that will be applied to the source code, determining which quality checks will be carried out. Each enabled plugin in the configuration corresponds to a dedicated Docker image that is automatically fetched and executed as needed in the running job configured in the .gitlab-ci.yml. These specific plugins complement a set of general checks—currently, there are ten maintainability checks 35 — which are enabled by default (and therefore not displayed in this file excerpt). These checks cover various types of measures, as follows: 1. Argument count (argument-count): methods or functions defined with a high number of arguments. 2. Complex logic (complex-logic): boolean logic that may be hard to understand. 3. File length (file-lines): excessive lines of code within a single file. 4. Identical blocks of code (identical-code): duplicate code which is syntactically identical (but may be formatted differently). 5. Method complexity (method-complexity): functions or methods that may be hard to understand (Cognitive Complexity). 6. Method count (method-count): classes defined with a high number of functions or methods. 7. Method length(method-lines): excessive lines of code within a single function or method. 8. Nested control flow (nested-control-flow): deeply nested control structures like if or case. 9. Return statements (return-statements): functions or methods with a high number of return statements. 10. Similar blocks of code (similar-code): duplicate code which is not identical but shares the same structure (e.g., variable names may differ). 5.3.4 Guidelines to ease collaboration Some basic practical rules have been shared among the partners contributing to the development of APRO platform modules and their components. These rules are intended to ensure standardized and cohesive work between the different teams and include: 1. One single GitLab repo is dedicated to each HARTU module (set of components e.g. a FBB) in the HARTU Group. 34 https://docs.codeclimate.com/docs/advanced-configuration#section-configuration-formats 35 https://docs.codeclimate.com/docs/maintainability#section-checks Figure 38: extract from codeclimate.yml D2.1 – HARTU Architectural specification and integration plan 55 2. Each HARTU repo shall have mainly two branches, ’master’ and ’develop’ (developers can create further branches of 'develop’, where needed). 3. Developers shall always git push to the ’develop’ branch (default choice for HARTU Group). 4. The ’master’ branch should be aligned (e.g. through a merge request) upon the release of a new version, adhering to the following rule. 5. The master branch should contain code that is stable and includes tagged version numbers. 6. Naming convention: The internal organization of each HARTU module repository should adhere to the structure illustrated in the screenshot below. Figure 39: HARTU repositories naming convention It's important to note that these simple rules should be considered a preliminary example for the stated purpose and are subject to change during the project as needed. 6 Conclusion This deliverable contains the HARTU RA providing a complete and exhaustive design of the HARTU software solution, outlining also its implantation (to be further refined during the implementation phase in the technical WPs (i.e., in WP2, WP3 WP4 and WP5) before starting the piloting and validation activities within the scope of WP1. These results have been driven by the previous work and results of WP1, notably the analysis of requirements and reference applicative business scenario. Key highlights include: HARTU will provide automated AI based grasp and release planning, electro-active soft grippers and contact-rich assembly results for manufacturing scenarios ensuring compliance with ethics and legal requirements providing guidance on Human-AI teaming and defining user profile and skills for the new working scenario in Industry 5.0. D2.1 – HARTU Architectural specification and integration plan 56 HARTU RA covers and specifies the functionalities that the software solution addresses in three different but interconnected areas: shopfloor, functional and technological. HARTU RA provides simulation and adaptation capabilities in real and physical factory environments, including real-time operations considering the variability of the process that an operator is performing, considering the main variability that can affect the correct execution of a routine or the execution of a process pipeline. The HARTU RA presents all the functional and technical features that the system will have to implement. The HARTU solution enables the execution of a wide range of business scenarios, building its foundation on principles of modularity, interoperability, scalability, flexibility, and adaptability. This document also provides the basis for the implementation of the HARTU software solution and integration activities, that will be performed as part of the technical work packages, in particular WP2, WP3 and WP4. In particular, it defines the main components and structuring principles of the HARTU solution, also in terms of implementation, ensuring that interdependent activities can be streamlined in the best possible way, and in any case will provides a framework to cover the functional and technical requirements defined by the project partners. Without any doubt, the design and implementation of HARTU RA will also guide the deployment and validation of use cases within WP1. As a rule, use cases may require some customizations, without however affecting the components of the solution described in this document. The identified components could be extended during solution development, as long as they remain backward compatible according to the specifications described in this document, thus not affecting the systems using the interfaces they expose. However, design revisions may occur as a result of development and integration activities: • new technologies selected to further improve the solution or a specific activity. • given some changes in requirements and use cases. • general evolution of the project. This document will be kept alive, for future refinements and/or improvements (in terms of revisions) of the overall HARTU design solution and specifications, in order to report (if any) a continuous update of the internal structure of the blocks, their communication with other blocks, the necessity to add new components or new features for the whole system (e.g. if new requirements arise for use cases or during the testing phase to integrate new technologies to fully satisfy the results). D2.1 – HARTU Architectural specification and integration plan 57 7 References [1] P. Adolphs and U. Epple, “Status Report: Reference Architecture Model Industrie 4.0 (RAMI4.0),” accessed: 2023-11-13. [Online]. Available: GMA-Status-Report-RAMI-40-July2015.pdf (zvei.org). [2] [29] X. Ye and S. H. Hong, “Toward Industry 4.0 Components: Insights Into and Implementation of Asset Administration Shells,” IEEE Industrial Electronics Magazine, vol. 13, no. 1, pp. 13–25, Mar. 2019, doi: 10.1109/MIE.2019.2893397. [3] T. Miny, G. Stephan, T. Usländer, and J. Vialkowitsch, “Functional View of the Asset Administration Shell in an Industrie 4.0 System Environment,” 2021, accessed: 2023-11-13. [Online].Available:https://www.plattformi40.de/IP/Redaktion/DE/Downloads/Publikation/Fu nctional-View.html