An autonomous edge box system architecture for Industrial IoT applications

An autonomous edge box system architecture for Industrial IoT applications 1st Cheng Guo, 2nd Shun Yang, 3rd Sebastian Thiede Chair of Manufacturing Systems, Department of Design, Production and Management University of Twente Enschede, The Netherlands [email protected], [email protected], [email protected] Abstract—Digitalization has been developing in manufacturing for decades. Many concepts and technologies related to digitalization have been proposed and have brought significant benefits to enterprises. The Industrial Internet of Things (IIoT) is an important means to realize digital manufacturing. However, there are many barriers when integrating and implementing digitalization for industrial companies, especially for small and mediumsized enterprises (SMEs). These barriers mainly include high cost sensitivity, difficulty in system installation and deployment, lacking of unified methods and platforms, and threats to data and network security. Therefore, a lightweight IIoT architecture is proposed in this paper to support SMEs to overcome the barriers and obtain the practical values brought by IIoT with a lower cost and effort. Finally, a case study has been conducted to validate the proposed approach. Index Terms—IIoT, Cyber-Physical Production Systems, autonomous, Industry 4.0, OPC UA, Edge Computing I. INTRODUCTION As increasingly important trend, digitalization plays a vital role in modern manufacturing industry [1]. With the development of information communication technology (ICT), emerging enabling technologies, such as Industrial Internet of Things (IIoT), Cyber-Physical Production Systems (CPPS), Digital Twin (DT) are all concrete manifestations of digitalization in manufacturing [2] [3]. All of these technologies hold great promise for helping manufacturing companies continuously improve production efficiency, quality, and sustainability [4] [5]. The small and medium-sized enterprises (SMEs) make up the vast majority in society and the market and are a huge contributor to employment. However, SMEs have significant barriers as they have limited resources and expertise to implement digitalization and apply the corresponding technologies such as IIoT applications [6] [7]. An overarching framework is summarized in Fig. 1. Therefore, it is necessary to have a system architecture which enables the IIoT applications with low-cost, easy-to-deploy and autonomous. In particular, the autonomous features in the proposed architecture allow the SMEs put less effort to get the actual benefits from the IIoT technology. II. RELATED WORK This paper focuses on the research of system architecture and technologies related to the implementation of digitalizaFig. 1. The overarching framework of proposed IIoT system architecture tion manufacturing for SMEs. The relevant literature reviews are listed in Table I. The selection and customization of cloud platforms are key factors and play a critical role in IIoT architecture design. For cloud platforms, the existing researches in Table I discussed the cloud platform and compared FIWARE open platform with others. Literature reviews gave comprehensive analysis about FIWARE on requirements [3] and performance [8]. A number of systems have been implemented using FIWARE in a variety of fields, such as agriculture irrigation [9], digital twins [10], smart campus [11], energy management [12]. Moreover, it clarifies that the characteristics of FIWARE are suitable for the needs of the corresponding system. Additionally, there are also some studies that discuss the security of FIWARE [13] [14]. Moreover, the system structure is another important factor in IIoT architecture design. Based on the 5C architecture for implementation of Cyber-Physical System [15], the IIoT architecture has been further developed by integrating the edge/fog computing into the system structure. Several studies discussed already the advantages of the edge/fog computing in terms of data processing, offload, data privacy, latency and security [16] [17]. OPC UA is already a common communication interface protocol and data model standard in the industry. Numerous studies have applied OPC UA to implement corresponding digital systems and discussed the features and corresponding advantages [18] [19] [20]. OPC UA Publish-Subscribe mechanisms is suitable for cloud communication, which is This is the author accepted manuscript. The published version is available via IEEE Xplore. https:// ieeexplore.ieee.org/abstract/document/10774355 TABLE I RELATED WORK CATEGORIZATION Paper Topics covered Description Cloud platform Edge / Cloud Protocol [3] ✓Review about platforms of the IIoT with 6 requirements. [8] ✓A comprehensive evaluation of the FIWARE performance on vertical (increasing capacity) and horizontal (increasing quantity) way. [9] ✓ ✓ An IoT architecture use in agriculture with FIWARE as a PaaS platform and The scenarios involved the fog/edge layer. An evaluation for FIWARE platform. [10] ✓Using FIWARE components for general Digital Twin and discussing features of FIWARE. [11] ✓ ✓ Using FIWARE platform for a smart campus IoT architecture and with edge computing and AI. [12] ✓ ✓ Using FIWARE platform and MQTT protocol for energy management system. [13] ✓Security on FIWARE platform was discussed from different aspect, and introduced TLS and DTLS into the MQTT and LWM2M/CoAP protocol. [14] ✓Analyzing the security of FIWARE. Discussing the authentication and identity management of FIWARE. [15] ✓Proposing an 5 layer framework (Connection, Conversion, Cyber, Cognition and Configure) for the general CPS. [16] ✓A research to compare the IoT architecture regarding efficient data processing. [17] ✓An study about edge computing for intelligent manufacturing regarding the network data efficiency, latency and security. [18] ✓Using OPC UA protocol for an automation system. [19] ✓✓Proposing an architecture with OPC UA serer and also to consider the Pub/Sub mechanism. [20] ✓Using an open source open62541 OPC UA library to designed CPS system. [4] ✓Discussing the Pub/Sub mechanisms of OPC UA in cloud communication. one-to-many mechanism, and its lightweight features ensure communication efficiency [4]. The related work discussed in this section involves various aspects and components of IIoT. However, these studies usually only focus on only a few components of the overall IIoT system. Therefore, it is particularly necessary to select and integrate various advanced and appropriate components to form a comprehensive IIoT architecture, and to consider potential software applications that can bring practical benefits to manufacturing SMEs. This paper discusses the various components in the IIoT architecture and software design, and the integration of advanced and appropriate components to overcome the barriers encountered by SMEs in the digitalization of manufacturing. Fig. 2. Overview of the architecture. III. IIOT ARCHITECTURE A. System Architecture Overview This paper presents the design of a comprehensive IIoT architecture which includes the data acquisition on the edge device to data storage and applications in the cloud. The IIoT system architecture and components proposed in this paper are shown in Fig. 2 and will be explained in the following. With the development of digital technologies, the fourth industrial revolution has been going on for years and is gradually maturing. Reference Architectural Model Industrie (RAMI) 4.0 [21] was proposed by Germany and RAMI 4.0 has three dimensions which are Hierarchy levels, life cycle & Value Stream and Layers. The Asset Administration Shell (AAS) is an important key element in RAMI 4.0. AAS was introduced in 2016 [22] [23]. The Virtual Representation, Technical Functionality, Component Manager and Manifest are the core concepts of AAS. AAS is a major concept and component for the implementation of IIoT based on RAMI 4.0. The implementation of the architecture proposed in this paper partially considers the concept of AAS. The edge box is designed as the role of AAS for the machines in the whole system, In this architecture, the edge box acquires machine-related data through sensors and the machine data interface. The features, such as low cost and easy deployment, are also considered in realizing the edge box. In the Fig. 2, for the components of (1) edge box, the edge box is a role of OPC UA server to present. The OPC UA server of the edge box as AAS represent the machine which the edge box install on. The edge box can support a wide variety of sensors in the (1.1) Sensors module. Then the sensor data are collected by the module of (1.3) Data collection. Additionally, the edge box is capable of collecting data from the (1.2) Machine data interface. The northbound data interface of edge box can communicate with third-party systems in the local network through OPC UA client-server communication model The client-server communication is the original communication model of OPC UA [24]. OPC UA server address space can be accessed through a OPC UA client in local network in this proposed architecture. In the (2) Cloud, the architecture use FIWARE open platform which is supported by the European Commission [25] as the core components. The components of the (2.1) Data adaption and (2.2) Data storage layer in the cloud are designed based on the architecture and mechanisms of the FIWARE platform. The communication between the cloud and the edge box is implemented using the OPC UA pub/sub mechanism. OPC UA has enhanced its communication mechanism by adding pub/sub mode. The features of this mode are more suitable for cloud communication scenarios in this IIoT architecture [4] [26]. The development of virtualization technology has greatly improved the development efficiency and reduced the deployment difficulty of software systems. This is because virtualization provides the basic feature of development and deployment independence from the host platform. Docker is a lightweight virtualization container technology. Docker are more efficient than kernel-based virtualization technology in terms of performance, memory, communication [27]. In the field of industrial manufacturing, Docker technology can also be applied and meet real-time application scenarios [28]. In the architecture proposed in this paper, Docker containers are used to host the execution of software applications. With the development of ICT technology and machine learning, edge computing has become more indispensable in IIoT systems. Edge computing performs computing close to the data source, which reduces communication transmission and data latency and ease the need for cloud computing. Latency issues are also reduced, which enables real-time decision-making since there is no need to wait for the remote site to process the data. Especially for large-scale applications of big data and machine learning, edge computing is even more important. Applying machine learning algorithms at the edge instead of in the cloud can bring many advantages which include reducing communication volume, increasing energy efficiency, and improving real-time performance [29] [30]. Corresponding machine learning at the edge for manufacturing is also widely discussed and applied [29] [31]. The proposed IIoT architecture also provides support and applications for edge computing and machine learning in terms of hardware and software. In addition, edge computing can offload computing task from the cloud to the edge side and this enables the large-scale application of the device to be improved [17] [29]. In terms of data security, the platforms, protocols and components selected for this architecture also fully consider their security strategies. FIWARE and OPC UA have comprehensive data security strategies which have been discussed and evaluated in studies [13] [14]. Based on the integration of selected components, the system architecture supports the protection of data security and privacy. Reducing the complexity of IIoT system installation and Fig. 3. Networking diagram of the architecture. deployment is a key consideration of the overall architecture. In Fig. 3 shows an exemplary system setup which consists of two parts which are edge box and cloud. In principle, one machine corresponds to one edge box which is the virtual representation as AAS of machines. The hardware and software modules can be easily configured for different requirements and scenarios. Users can easily deploy and select sensors according to different application scenarios. Due to the heterogeneity of machines in the plant, especially for SMEs. A large amount of time is spent on different machines for connectivity and data acquisition. Thus, this Plug-and-Play strategy and edge computing feature can greatly reduce the deployment effort and support scalability and flexibility of this system. The autonomous features are reflected through software, hardware and the entire architecture. The versatility of this architecture for machines, plug-and-play and easy-to-install structure, and combined with corresponding machine learning applications, this IIoT architecture can systematize more autonomous features. B. Edge box modules design The detailed modules of the edge box are shown in Fig. 4. The edge box is designed as three layers in software structure. It consists of Device connection layer, Data layer and Communication layer. The Device connection layer is responsible for connectivity and data collection with various sensors and also includes communication with the machine’s own data interface. The main task of the programs in the module is to configure the corresponding bus protocols and collect data from sensor interfaces including I2C, SPI, UART, etc. The data obtained through the Device connection layer will be further processed in the Data layer according to requirements. The system functions of the edge computing role are also implemented in this layer. Machine learning algorithms have been widely used in manufacturing in recent years [32] [33]. Running machine learning algorithms on the edge rather than in the cloud brings many benefits [29]. In this architecture, the data processing algorithm modules which also includes Fig. 4. Edge box struct and modules the processing of multimedia data streams such as audio and image at the edge can be located at this layer. The edge box support for tflite-runtime and scikit-learn libraries also greatly facilitates the operation of artificial neural networks and traditional machine learning algorithms models on the edge. For communication layer, OPC UA is the main communication protocol for edge box in this architecture. Because OPC UA is a safe, reliable, platform-independent and widely adopted interoperable information exchange standard. Through the server/client mode of OPC UA, the client can access this address space through Ethernet communication. In the communication between the edge box and the cloud, use the broker-based pub/sub model to publish and subscribe for message exchange. C. Cloud modules design The design of the cloud software components in this architecture is based on FIWARE in terms of data transmission, storage and management. The whole cloud service design is generally divided into three layers as shown in Fig. 5. Orion Broker is the core component for the FIWARE framework. It communicates with other modules using Next Generation Service Interfaces (NGSI). In the communication layer, OPC UA Pub/Sub Agent converts the OPC UA Pub/Sub messages via MQTT broker to NGSI-v2 message which is sent to the Orion Broker. MongoDB is used by the Orion Broker to store the context data. Quantum Leap is a FIWARE component (Generic Enabler) which is used to persist time-series data. This GE is configured to use the CrateDB to store the time-series data. Various applications can be developed at the Application layer to implement functions facing end users. Some appliFig. 5. IIoT cloud struct and modules cations may be the base platform for most applications, such as visualization platforms and data algorithm platforms. Some terminal applications may be based on other applications, such as OEE prediction and other applications that rely on machine states module. IV. CASE STUDY For demonstration, testing and further development, an edge based system based on the presented architecture was implemented on a milling machine. This system implements basic functions including data collection, processing, transmission to the cloud, storage and end-user visualization. First, sensor data is collected and processed at the edge box which is configured with 3 types of sensor data (temperature, current, vibration). Then, the data is transferred to the cloud in realtime via OPC UA pub/sub which was based on MQTT for broker-based pub/sub model. Finally, the data and information are presented to the end-user through an application interface based on Grafana. The edge box consists of the main control board and sensors. The hardware cost of the entire system is relatively low. The detailed hardware components of the edge box are shown in the table II. The deployment of the edge box is non-intrusive in terms of necessary connections to e.g. machine control and does not require advanced expertise. The current clamp can be simply clamped on the power input cables of the milling machine. The edge box body and sensor module only need to be fixed directly to the cabinet of the milling machine. The deployment of the edge box for a milling machine is shown in Figure 6 Fig. 7 shows the dashboard of the real-time monitoring for the milling machine. The dashboard contains the temperature, electric current and vibration information. The gauges display TABLE II EDGE BOX HARDWARE COMPONENTS Components Technical Specifications Interface Function in system Raspberry Pi BCM2711 Cortex-A72 64-bit 1.5GHz, 4GB LPDDR4, Ethernet, Wi-Fi, I2C, SPI, UART, etc. Main control platform Accelerometer (MPU6050) 16-bit data output, Accelerometer: ±2g to ±16g I2C Collect vibration data Temperature (MPU6050) 16-bit signed value, -40 °C to +85 °C I2C Collect temperature data ADC (ADS1115) 2.0V to 5.5V, 4 single-ended input channels,16-bit precision I2C Convert analog voltage signal to digital signal Current sensor (RS CP03B) Maximum AC/DC Current:400A, Resolution ±1.0 percent Analog voltage Collect current data Fig. 6. Edge box deployment on a milling machine real-time data and the time-series graphs present historical data for a selected period. Based on the data obtained by this system, some potentially useful information can be obtained automatically. The autonomy of this architecture can be further reflected through some unsupervised learning algorithms. Here the electric current data and vibration data are combined to represent. The edge box processes the vibration data with FFT and the amplitude and frequency are uploaded to the cloud. One crucial functionality is the automatic detection of machine states which is the necessary base for further advanced use cases. Against this background, Fig. 8 shows a three-dimensional scatter plot which use K-means algorithm. The main purpose of this data clustering is to present further data information that can be generated by combining multiple types of sensor data based on this IIoT architecture. The red points indicate that the milling machine is engaged in a high-power milling Fig. 7. Real time dashboard Fig. 8. Sensor data clustering of milling machines operation. This operation results in the generation of relatively high current and amplitude. The blue points area represents a cluster where the milling machine performs other non-highpower operations. The green and purple points area should be a mixture of low amplitude vibrations generated by the milling machine during standby and vibrations caused by other reasons. The data also include vibrations caused by other reasons, such as noisy data caused by accidental hits or sensor hardware performance. To demonstrate the concept of AI at the edge, further applications can be developed based on the proposed IIoT architecture. For instance, vibration or sound sensor data can be analyzed using variant algorithms such as MLP, CNN, and Autoencoder, which support predicting the overall equipment effectiveness (OEE). In addition, there is obvious potential to combine multi-variant sensors and algorithms to support more complex applications such as predictive maintenance, abnormal detection in quality assurance, remote planning and operation, etc. V. CONCLUSIONS AND FUTURE WORK This study proposes an IIoT architecture to reduce the barriers that SMEs face when implementing digital manufacturing. This paper comprehensively investigated the components of implementing an IIoT system from sensors and edge box to cloud platforms and corresponding applications. Case studies which a system based on this architecture applied to a milling machine is presented. Overall, the features of the IIoT architecture are analyzed, including low cost, plug-and-play deployment, module-based design, open platform, data security, and edge computing. Furthermore, autonomous features are reflected in the combination of architecture and applications. Thereby, the common barriers encountered by SMEs in implementing digital manufacturing could be solved by means of the proposed IIoT architecture with concrete applications. Future work will focus more on data privacy and cyber threats to the architecture, as well as the evaluation and improvement of the overall performance of the system. In addition, more use cases will be explored based on this architecture. ACKNOWLEDGMENT This work is supported by AI Redgio 5.0 (Project No. 101092069) and content of this study could partially be used in the project. REFERENCES [1] M. Ghobakhloo, “Industry 4.0, digitization, and opportunities for sustainability,” Journal of cleaner production, vol. 252, no. 119869, 2020. [2] S. Thiede, “Cyber-physical production systems (CPPS): introduction,” Journal of Manufacturing and Materials Processing, vol. 5, no. 1, p. 24, 2021. [3] S. Yangui, “A panorama of cloud platforms for IoT applications across industries,” Sensors, vol. 20, no. 9, p. 2701, 2020. [4] D. G. Pivoto, L. F. de Almeida, R. da Rosa Righi, J. J. Rodrigues, A. B. Lugli, and A. M. Alberti, “Cyber-physical systems architectures for industrial internet of things applications in industry 4.0: A literature review,” Journal of manufacturing systems, vol. 58, pp. 176–192, 2021. [5] S. Thiede, “Digital technologies, methods and tools towards sustainable manufacturing: Does industry 4.0 support to reach environmental targets?” Procedia CIRP, vol. 98, pp. 1–6, 2021. [6] F. Hein-Pensel, H. Winkler, A. Br¨ uckner, M. W¨ olke, I. Jabs, I. J. Mayan, A. Kirschenbaum, J. Friedrich, and C. Zinke-Wehlmann, “Maturity assessment for industry 5.0: A review of existing maturity models,” Journal of Manufacturing Systems, vol. 66, pp. 200–210, 2023. [7] C. Schr¨ oder, “The challenges of industry 4.0 for small and mediumsized enterprises,” Friedrich-Ebert-Stiftung: Bonn, Germany, vol. 7, pp. 1–28, 2016. [8] V. Araujo, K. Mitra, S. Saguna, and C. ˚ Ahlund, “Performance evaluation of FIWARE: A cloud-based IoT platform for smart cities,” Journal of Parallel and Distributed Computing, vol. 132, pp. 250–261, 2019. [9] C. Kamienski, J.-P. Soininen, M. Taumberger, R. Dantas, A. Toscano, T. Salmon Cinotti, R. Filev Maia, and A. Torre Neto, “Smart water management platform: IoT-based precision irrigation for agriculture,” Sensors, vol. 19, no. 2, p. 276, 2019. [10] J. Conde, A. Munoz-Arcentales, A. Alonso, S. L´ opez-Pernas, and J. Salvachua, “Modeling digital twin data and architecture: A building guide with FIWARE as enabling technology,” IEEE Internet Computing, vol. 26, no. 3, pp. 7–14, 2021. [11] P. Martins, S. I. Lopes, and A. Curado, “Designing a FIWARE-Based smart campus with IoT edge-enabled intelligence,” in Trends and Applications in Information Systems and Technologies: Volume 3 9. Springer, 2021, pp. 557–569. [12] T. Storek, J. Lohm¨ oller, A. K¨ umpel, M. Baranski, and D. M¨ uller, “Application of the open-source cloud platform FIWARE for future building energy management systems,” in Journal of Physics: Conference Series, vol. 1343, no. 012063. IOP Publishing, 2019. [13] C. T. Oliveira, R. Moreira, F. de Oliveira Silva, R. S. Miani, and P. F. Rosa, “Improving security on IoT applications based on the FIWARE platform,” in 2018 IEEE 32nd International Conference on Advanced Information Networking and Applications (AINA). IEEE, 2018, pp. 686–693. [14] P. R. Sousa, L. Magalh˜ aes, J. S. Resende, R. Martins, and L. Antunes, “Provisioning, authentication and secure communications for IoT devices on FIWARE,” Sensors, vol. 21, no. 17, 2021. [15] J. Lee, B. Bagheri, and H.-A. Kao, “A cyber-physical systems architecture for industry 4.0-based manufacturing systems,” Manufacturing letters, vol. 3, pp. 18–23, 2015. [16] H. Chegini, R. K. Naha, A. Mahanti, and P. Thulasiraman, “Process automation in an IoT–fog–cloud ecosystem: A survey and taxonomy,” IoT, vol. 2, no. 1, pp. 92–118, 2021. [17] G. Nain, K. Pattanaik, and G. Sharma, “Towards edge computing in intelligent manufacturing: Past, present and future,” Journal of Manufacturing Systems, vol. 62, pp. 588–611, 2022. [18] M. V. Garc´ ıa, E. Irisarri, F. P´ erez, E. Est´ evez, and M. Marcos, “An open cpps automation architecture based on iec-61499 over opc-ua for flexible manufacturing in oil&gas industry,” IFAC-PapersOnLine, vol. 50, no. 1, pp. 1231–1238, 2017. [19] G. Jo, S.-H. Jang, and J. Jeong, “Design and implementation of cpps and edge computing architecture based on opc ua server,” Procedia Computer Science, vol. 155, pp. 97–104, 2019. [20] M. M¨ uller, E. Wings, and L. Bergmann, “Developing open source cyber-physical systems for service-oriented architectures using opc ua,” in 2017 IEEE 15th international conference on industrial informatics (INDIN). IEEE, 2017, pp. 83–88. [21] P. Adolphs, H. Bedenbender, D. Dirzus, M. Ehlich, U. Epple, M. Hankel, R. Heidel, M. Hoffmeister, H. Huhle, B. K¨ archer et al., “Status report-reference architecture model industrie 4.0 (rami4. 0),” VDI-Verein Deutscher Ingenieure eV and ZVEI-German Electrical and Electronic Manufacturers Association, Tech. Rep, 2015. [22] P. Adolphs, S. Auer, H. Bedenbender, M. Billmann, and M. Hankel, “Structure of the administration shell. continuation of the development of the reference model for the industrie 4.0 component,” ZVEI and VDI, status report, 2016. [23] T. Bangemann, C. Bauer, H. Bedenbender, M. Diesner, U. Epple, F. Elmas, J. Friedrich, T. Goldschmidt, F. G¨ obe, S. Gr¨ uner et al., “Industrie 4.0-technical assets: Basic terminology concepts life cycles and administration models,” VDI/VDE and ZVEI, 2016. [24] OPC UA specifications and information models. [Online]. Available: https://reference.opcfoundation.org/ [25] The FIWARE website. [Online]. Available: https://www.fiware.org/ [26] A. Burger, H. Koziolek, J. R¨ uckert, M. Platenius-Mohr, and G. Stomberg, “Bottleneck identification and performance modeling of opc ua communication models,” in Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering, 2019, pp. 231– 242. [27] A. M. Potdar, D. Narayan, S. Kengond, and M. M. Mulla, “Performance evaluation of docker container and virtual machine,” Procedia Computer Science, vol. 171, pp. 1419–1428, 2020. [28] M. Sollfrank, F. Loch, S. Denteneer, and B. Vogel-Heuser, “Evaluating docker for lightweight virtualization of distributed and time-sensitive applications in industrial automation,” IEEE Transactions on Industrial Informatics, vol. 17, no. 5, pp. 3566–3576, 2020. [29] M. Merenda, C. Porcaro, and D. Iero, “Edge machine learning for aienabled IoT devices: A review,” Sensors, vol. 20, no. 9, p. 2533, 2020. [30] J. Zietsch, M. Vogt, B. D. Lee, C. Herrmann, and S. Thiede, “Enabling smart manufacturing through a systematic planning framework for edge computing,” CIRP Journal of manufacturing science and technology, vol. 31, pp. 351–369, 2020. [31] G. Plastiras, M. Terzi, C. Kyrkou, and T. Theocharidcs, “Edge intelligence: Challenges and opportunities of near-sensor machine learning applications,” in 2018 IEEE 29th international conference on applicationspecific systems, architectures and processors. IEEE, 2018, pp. 1–7. [32] T. Wuest, D. Weimer, C. Irgens, and K.-D. Thoben, “Machine learning in manufacturing: advantages, challenges, and applications,” Production & Manufacturing Research, vol. 4, no. 1, pp. 23–45, 2016. [33] R. Rai, M. K. Tiwari, D. Ivanov, and A. Dolgui, “Machine learning in manufacturing and industry 4.0 applications,” International Journal of Production Research, vol. 59, no. 16, pp. 4773–4778, 2021.