Advancing human-robot collaboration: Predicting operator trajectories through AI and infrared imaging

Journal of Manufacturing Systems 74 (2024) 980–994 Available online 23 May 2024 0278-6125/© 2024 The Authors. Published by Elsevier Ltd on behalf of The Society of Manufacturing Engineers. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/). Technical paper Advancing human-robot collaboration: Predicting operator trajectories through AI and infrared imaging Konstantinos Katsampiris-Salgado, Nikos Dimitropoulos , Christos Gkrizis , George Michalos , Sotiris Makris * Laboratory for Manufacturing Systems and Automation, University of Patras, 26504 Patras, Greece ARTICLE INFO Keywords: Human - robot collaboration Artificial intelligence Synthetic dataset Infrared imaging Human detection ABSTRACT In industrial applications demanding close interaction between humans and robots, accurate perception of the robot’s surroundings is vital to adjust its behavior appropriately. This need becomes especially pertinent when manipulators operate autonomously and execute both separate and common tasks with human operators. To ensure operator safety in such Human-Robot Collaboration (HRC) environments, the present paper proposes an innovative approach that leverages advanced collaborative robotic systems capable of sensing the operator’s position to effectively prevent or mitigate potential contact. Our solution combines infrared-thermal imaging technology with a workspace monitoring system, utilizing thermal and depth cameras, to track the operator’s current location and predict their future trajectory using a multi-layer Long Short-Term Memory (LSTM) neural network. We also introduce a technique for simulating both human and robot movements to generate a synthetic dataset. This data is shared with the motion planning system to generate trajectories that do not intersect with the operator’s future paths, minimizing downtime due to safety stoppages. The solution is integrated and evaluated within an industrial high payload collaborative robot context, and the results have shown improvements in cycle time efficiency and operator approval. This approach offers a promising direction for enhancing safety and productivity in HRC workspaces. 1. Introduction Over the last years the need for the implementation of flexible and reconfigurable production systems, that are capable of producing highly customized products of different production sizes [1], has driven the research attention into designing and deploying human-robot collaborative (HRC) cells. The seamless, tight existence and collaboration between robotic manipulators and human operators, achieves the combination of the advantages of both aspects; the speed, accuracy, and facile weightlifting of robots with the flexibility, creativity, and dexterity of humans [2]. The integration of HRC (Human-Robot Collaboration) systems introduces significant challenges, particularly concerning safety aspects to ensure the operators’ protection. [3,4]. When humans and robots closely interact, strict safety measures are applied to limit the robot’s speed and force, especially in power and force-limiting applications. To prevent operators from entering hazardous areas, fail-safe sensors such as laser scanners, safety mats, light barriers, and safety camera systems monitor the workspace [4]. These sensors establish either a static safety zone layout or, in some cases, dynamically changing zones based on relevant robot orientation [5]. However, the discretization level allows only predefined robot motion programs. Non-fail-safe approaches using conventional RGB cameras or RGBD sensors [6] have been proposed to track operators’ positions in the workspace with good accuracy. However, these approaches require higher computational power and perform poorly in low-light conditions. In contrast, thermal cameras form the environment’s imagery based solely on emitted thermal energy, independent of light conditions, offering a robust solution for varying light and weather conditions [7,8]. To maximize the potential of both human and robotic resources, robots also need to possess cognitive abilities to comprehend their surroundings and human actions. Beyond this, predicting human intentions will allow robots to adapt their actions to optimize collaborative efficiency. * Corresponding author. E-mail address: [email protected] (S. Makris). Contents lists available at ScienceDirect Journal of Manufacturing Systems journal homepage: www.elsevier.com/locate/jmansys https://doi.org/10.1016/j.jmsy.2024.05.015 Received 11 December 2023; Received in revised form 14 May 2024; Accepted 14 May 2024 Journal of Manufacturing Systems 74 (2024) 980–994 981 1.1. Human detection using IR technology This advantage of thermal cameras has led to their application in numerous sectors where there is need for detecting humans [9]. Such examples may be found in automotive related safety applications for pedestrians and vehicles [10–12], in military applications for detecting e.g., snipers [13], in search and rescue applications where UAVs (Unmanned Aerial Vehicle) and rescue robots are deployed with Infra-Red (IR) cameras searching for people during natural disasters or terrorist occurrences [14]. Applications of IR cameras may be found in supervision systems that perform human detection, tracking and analyze his/her behavior acting proactively in cases of suspicious actions preventing potential intrusions [9]. General human activity analysis has also been studied, e.g., for separating walking and running spatio-temporal information [15] analysis of human posture in case somebody is lying down [16]. 1.2. Operator motion prediction Human intention recognition is a focal point of research in multiple fields. A significant hurdle faced by researchers is the stochastic nature of human behavior. Most methodologies employ Artificial Intelligence, using neural networks trained on real, synthetic, or mixed data. Application examples include pedestrian avoidance that are used for autonomous vehicles [17], task identification through visual observation of hand-object interaction [18,19], gesture-based command recognition used for AGVs controls [20], EEG-based recognition of human intentions [21], and context-based human motion recognition [22–24]. They also cover synchronization of HRC tasks by estimating human hand motion and syncing it with robot end-effector motion [25]. Specifically, for the estimation of humans’ future trajectory Rudenko et al. [26] have made an extensive study where they cover in depth the motion prediction problem and decompose it in higher lever, in two main directions; one based in modelling approach, and one based on contextual cues. The categorization of human motion prediction approaches is based on how they represent human movement and formulate the underlying causes of that movement. Physics-based approaches may include single or multiple dynamic, physics-based models. Pattern-based approaches learn movement patterns from the observed agent paths, using sequential or non-sequential predictive models. Planning-based approaches reason on motion intents of sensible agents and include forward or inverse planning methods in a sense to compute a path hypothesis [26]. 2. Approach and architecture 2.1. IR-based human tracking In the present study, human position detection was performed using various sensors placed in different layout, as this information I very important to be shared among the safety related and path planning modules. The approach followed, consists of depth sensors with integrated software SDK for human pose detection (Azure Kinect + Azure Kinect Body Tracking SDK) which is plug and play from Microsoft, and a custom-made IR camera based human tracking software which is analyzed in the manuscript. The overall detection pipeline, which is summarized in Fig. 1, consists of 4 main pillars. The first pillar is related to the sensing layer which corresponds to the data acquisition from the thermal camera, using the relevant SDKs. The second pillar corresponds to the raw data preprocessing and filtering to a format acceptable for the third pillar which is in charge of extracting human shapes from the image, leading to the last pillar which acquires the segmented human shapes from the images and returns an estimate for the cartesian position of the operator. An infrared camera placed on top of the ceiling of the workspace is used to detect and track the position of the operator within the workplace shared with the robotic manipulator. It is important, especially for soft safety HRC applications, to capture operator’s position with high frequency rates, and with adequate accuracy. The utilization of such a sensor was made as the radiometric images captured offer the opportunity to easily extract the features of the human presence due to the temperature differences of the operator with his/her surroundings. Initially, during the designing phase the cell dimensions were evaluated to choose the proper sensor layout (infrared camera and its lens) that will be capable of supervising the whole desired area. Afterwards, due to the intrinsic difficulty of thermal cameras to estimate their extrinsic and intrinsic parameters, as in contrary to the conventional RGB cameras were the applied algorithms use visual patterns to calculate the camera matrix, an emulation of the visual patterns with thermal ones was implemented. Afterwards, the radiometric image received was processed to remove thermal noise from the field of view (scaling temperature values and using Gaussian blur implemented in OpenCV), while in addition small heat emitters such as devices were also removed. The binarization of the image follows, to create thermal blobs within the image (inside threshold values that correspond to human temperature each pixel will have a value 255 (white), otherwise the value would be 0 (black). The operator detection follows, based on the size estimation of the thermal blobs viewed in the captured frames, as the number of pixels per blob is compared with specified values that represent humans; after finding thermal blob that represent humans. Then the calculation of their centroids is performed to acquire their position in the pixel space, while finally their values are transformed from pixel to cartesian space. The overall approach is summarized in Fig. 1, categorizing it in 4 main steps: •Acquisition of radiometric data •Image processing •Human shape extraction and detection •Human position estimation More information is presented in Section 3.1. Fig. 1. Approach of system’s IR based human position detection. K. Katsampiris-Salgado et al. Journal of Manufacturing Systems 74 (2024) 980–994 982 2.2. Operator future trajectory forecasting To deal with the uncertain nature of human motion, an approach employing Recurrent Neural Networks (RNNs) was devised to predict the operator’s future trajectory, taking into account multiple causative factors. Through the collection of several datasets that record both the operator’s movements and the robot’s paths, a correlation was observed with the operator’s future position. During multiple cycles of operation, the movement of the operator displays regular or even recurring behavior. Therefore, past human positions, in conjunction with previous process variables, might be used to construct a prediction model strategy that performs time series forecasting. This multivariate model accepts multiple variables as input over multiple timesteps, creating a solution for a sequence-to-sequence problem. Notably, the proposed method is adaptable and may be applied with any type of sensor, as long as it can provide the operator’s torso coordinates. In this study, input data from thermal and RGBD cameras were utilized. Initially, a simulation environment named Shopfloor Digital Representation (SDR) was created, which allowed the simulation of both human and robot movements. A synthetic dataset was generated, incorporating human motion data from numerous iterations of a specific HRC scenario. This scenario was then replicated several times in the real cell, producing a real-world dataset of human motion. Both datasets contain data about human movements and robot paths, correlated with the tasks being performed (Fig. 2 – data collection - training). In the execution phase, the Future Trajectory Forecasting (FTF) module is used in tandem with the other software modules of the robotic cell. The process begins with the SDR module estimating the duration of the next robot task and asking the FTF to provide a prediction of the operator’s trajectory for a slightly longer duration (to accommodate potential delays in the new robot movement). The FTF receives data about the robot’s pose and the operator’s position to make the forecast. The SDR then computes a robot trajectory that does not intersect with the operator’s future path and transmits it to the robot. FTF approach is illustrated in Fig. 2. More specifically, Fig. 2 captures the following: i) Orange color: indicates the actions done for generation of a dataset using simulation data. ii) Green color: indicates the actions done for generation of dataset using a real-time execution data iii) Gray color: indicates the actions done for training of the model iv) Blue color: indicates the execution of the process using the FTF model. More information is presented in Section 3.2 and Section 3.3. 2.3. Overall approach The present work blends the outputs from the IR-based human tracking, the depth data tracking and the prediction model to achieve robot motion plans that aim to prevent collisions. The overall approach as depicted in Fig. 3 in three core layers; the first layers aims to detect the operator using multiple sensors. The motivation under thus choice is to guarantee that the operator is at all times detected (infrared cameras as in detail described in Section 3.1, and depth sensors as described in Section 3.3.2.). The second layer is in charge of evaluation operator positional data, and via deep neural network, to perform a prediction on what will be operator’s future motion. This data are shared via Robot Operating System (ROS) to the third layer, which is the digital twin and the integrated motion planners as obstacles, thus facilitating the generation of plans that prevent collision with human operators. Fig. 2. Flowchart illustrating data acquisition, training, and FTF-enabled human-robot collaborative assembly. K. Katsampiris-Salgado et al. Journal of Manufacturing Systems 74 (2024) 980–994 983 3. Implementation 3.1. IR-based human tracking 3.1.1. Camera placement and calibration The thermal camera chosen for the application, is the FLIR Boson 320 [27]. The considerations for choosing the proper sensor for the application were mainly based on the following parameters: - Ability to capture the infrared radiation that human body emits. - Given the camera positioning, the Field of View (FOV) of the chosen sensor should be capable of capturing the whole area required to be monitored. As every object does emit electromagnetic energy and based on a spectrophotometrical examination, the maximum of the broad band of wavelengths that the energy is found to lie may be calculated by the Wien’s Displacement law (1): λmax =b Τ(1) Where λmax is peak wavelength the Τ is the absolute temperature and b is a constant of proportionality called Wien’s displacement constant. According to Barnes [28], the human skin IR emission wavelength emit by the human body has a maximum of λ=10 µm. Given this, the Boson320 is capable of capturing sufficient infrared wavelengths from 7.5 to 12.5 µm, meaning it fulfilled the detection of human related temperatures for the specific application. After determining the area that should be monitored by the camera, the required field of view can be calculated by the following Eq. (2) (3), with the HFOV and VFOV being acquired by the manufacturer’s available lens options. (see Fig. 4) LHFOV =2Htan(HFOV/2)(2) LVFOV =2Htan(VFOV/2)(3) HFOV, is the angle for horizontal field of view, the VFOV is the angle vertical field of view, LVFOV is the length of the vertical field of view and LHFOV is the is the length of the horizontal field of view. Besides, the estimation of the camera parameters, both their extrinsics and intrinsics, are calculated as this process is of major importance for finally making accurate positional estimations. The extrinsic parameters are related to the cartesian pose of the sensor relative to the 3D scene and consists of a rotation R and a translation T matrix while the intrinsic parameters represent the optical and the focal length of the camera and consists of a matrix K. For the calibration process, the Zhengyou Zhang’s [29] calibration algorithm was utilized and implemented in OpenCV [30], and a chessboard specific for the thermal camera heated in the sunlight was used. The chessboard includes temperature differences within the checkerboard pattern as the black squares absorb more heat from the sunlight or from IR lamps and thus make their temperature higher than that of the white blocks. In pixel space the coordinates of the pixels are described as [u, v, 1] −1 and the feature points on the calibration board which are in the 3D scene have the coordinates [X, Y, Z, 1] −1 , though since the board’s surface is flat X =0. Therefore, the relation between the pixels of the projected image and the calibration checkboard is given in Eq. (4): s⎡ ⎣ &u &v &1⎤ ⎦=K[R|T]⎡ ⎢ ⎢ ⎣ &X &Y &0 &1 ⎤ ⎥ ⎥ ⎦ (4) Where RT is the rotation matrix and translation matrix from worldframe to pixel-frame, and K is the camera matrix. The radial distortion to correct the output image can be represented as per Eqs. (5), (6): xdistorted =x(1+k1r2+k2r4+k3r6)(5) ydistorted =y(1+k1r2+k2r4+k3r6)(6) The tangential distortion to correct the output image can be represented as per Eqs. (7), (8): xdistorted =x+ [2p1xy +p2(r2+2x2)] (7) ydistorted =y+ [p1(r2+2y2) + 2p2xy](8) Where k1, k2, k3, p1, p2 are the distortion coefficients. After the calibration process, the distortion coefficients (k1,k2,p1,p2,k3,) are estimated along with the camera matrix, that includes information like focal length (fx, fy)and optical centers (cx,cy)as per Eq. (9): Fig. 3. Human detection - Motion prediction - Motion planning approach to collision prevented plans. Fig. 4. Field of View (FOV) of a camera placed on the ceiling. K. Katsampiris-Salgado et al. Journal of Manufacturing Systems 74 (2024) 980–994 984 camera matrix =⎡ ⎣ fx0cx 0fycy 0 0 1 ⎤ ⎦(9) In Fig. 5 the distorted and corrected/undistorted images are depicted. 3.1.2. Operator detection and cartesian position estimation There are many difficulties while estimating the position of the operator with a 2D low resolution camera, as the pixel density is low, meaning that the camera discretizes the 2D plane in less big-sized surfaces. Taking all those into allowance, the process presented in Fig. 5 was implemented to return operator’s position with sufficient accuracy for our system. Firstly, after the acquisition of the radiometric image data, the normalization of temperature values follows. To prevent potential high temperature differences from too cold or too hot objects that might be within the field of view of the camera, both a minimum and a maximum temperature threshold are set, thus keeping a stable imagery after the thresholding process. The acquired data as far as they represent temperature values are normalized in the range 0–255, taking as extreme values the threshold values described above, instead of the maximum and minimum temperatures within the sensor capture. The normalization Eq. (10) applied is the following: pixel_valuenormal =Tvalue_per_pixel −Tmin Tmax −Tmin (10) Where pixel_valuenormal is the normalized pixel intensity Tmin is the minimum temperature value and Tmax is the maximum temperature value and Tvalue_per_pixel is the temperature value per pixel from the IR image. In Fig. 6 an example of different threshold limits for the same image is depicted. Following the image thresholding, a human temperature isolation (HTI) function was developed that aims to extract human temperatures from the captured frame, returning a threshold-ed, binary image. Prior to the thresholding, we tried to apply smoothing filters to achieve less noisy binary frames, though no significant improvement was noticed. Instead, it delayed the final detection having as a result to reject it. The HTI function scans through the pixels, checking for whether each one of them is between the accepted thresholds, as shown in Eq. (11). Fig. 7. pixel_valuebin ={0∀pixel_valuenormal ∈ [min,max] 255 ∀pixel_valuenormal ∕∈ (min,max)(11) Where pixel_valuebin represents pixel intensity value. For smaller sized capture sections that may be heated or reflect radiation, further processing is performed, that based on the “blobs statistics” from the binary image, the operator’s shape can be extracted. A function scans through the detected heat blobs, and compares their size with a reference value, which represents an average of human thermal blob; finally, the function returns a list of human sized shapes, composing a binary image without thermal noise. An example of the described image binarization is presented in Fig. 8, where thermal blobs from hot, electrical devices (such as monitors, computers etc.) that resemble human temperature in the thermal imagery are removed. In the binary image where the human shape has been extracted and isolated, as the point of detection we regard the centroid of his/her amorphous heat shape. To perform this calculation, we utilized the calculation of the moments of the 2D blobs, which is a particular weighted average of image pixel intensities (12). Mij =∑ x∑ y xiyjI(x,y)(12) Where Mij represents the moment of the blob, x,y refers to the row and column index and I(x,y)refers to the intensity at that location (x, y)[50]. Afterwards, the pixel coordinates of the centroid are calculated using the following Eq. (13). {x,y} = {M10 M00 ,M01 M00}(13) To find the correlation of the pixels and cartesian coordinates, we apply the following Eqs. (14), (15) to acquire the relation between pixels and cartesian distances: lx=number of pixels inxdirection foradistance inxdirection measure distance inxdirection (14) ly=number of pixels inydirection foradistance inydirection measure distance inydirection (15) By knowing the relationship between pixels and actual dimensions, we convert the pixel coordinates having as reference the center of the image by applying the following Eqs. (16), (17). projection_pixelsx=imagewidth ∗0.5−x(16) projection_pixelsy=imageheight ∗0.5−y(17) Afterwards, the pixel coordinates are transformed to cartesian coordinates as per (18), (19) equations: projection_cartesianx=projection_pixelsx∗lx(18) projection_cartesiany=projection_pixelsy∗ly(19) Due to the fact that those coordinates do not represent the actual position of the operator in the floor, but the projection of his/her chest in the 2D plane, we added an additional error variable that has a polynomial format, since the error increases in a non-linear manner as the detected operator steps away from the center of the camera. To estimate the factors of the polynomial, a depth sensor was utilized. The final position is estimated as per Eqs. (20), (21): correct_positionx=projection_cartesianx+errorx(20) Fig. 5. Distorted (left) and undistorted image (right). K. Katsampiris-Salgado et al. Journal of Manufacturing Systems 74 (2024) 980–994 985 correct_positiony=projection_cartesiany+errory(21) The results from the detection are presented in Fig. 8 and Fig. 9. 3.1.3. Comparison with YOLOv7 The presented methodology was also compared with YOLOv7 [31], not only to evaluate the detection capabilities, as YOLOv7, is one of the latest and best performing models for object detection, but also to check the run time processing power required in terms of CPU. To perform the Fig. 6. Human position detection steps using an IR sensor. Fig. 7. IR captures for different threshold values. Fig. 8. Blobs filtering in the binary image after threshold application. Fig. 9. Detection from paper’s proposed method (left) compared to YOLOv7 (right). K. Katsampiris-Salgado et al. Journal of Manufacturing Systems 74 (2024) 980–994 986 afore mentioned comparison, a dataset consisting of 700 images was collected and split for train and test purposes in an analogy of 80–20%. The GPU was only used during the training phase of the model, to minimize delays. After the image’s annotation, the dataset was augmented as proposed from the available literature [32,33], to enhance the training process. As per the detection capabilities, in the test dataset both approaches demonstrated good results, performing accurate human detections. It is though important to point that the processing power required in the YOLO v7 approach, was significant larger, as it scored values ranging from 96.5% to 99.9% in CPU usage. In the contrary, the proposed thermal detection method scored values ranging from 9.9% to 12.4% for the same test video stream. The processor where the experiments were done was the Intel i7–1075 H. In Fig. 9 the detections from the two approaches are displayed. It is important to mention that the boundary box from the left image does not correspond to the detection boundary box, as its size is fixed based on the white dot which indeed is the actual detection placed onto the operator’s body, indicating the centroid of the human shape in the binary image. 3.2. Synthetic dataset generation in a simulation environment using A* 3.2.1. Shopfloor digital representation – human motion simulation The Shopfloor Digital Representation (SDR) primarily concentrates on the simulation and creation of robot movement, alongside replicating the operator’s motions within a virtual setting. This setting is an exact mirror of the actual environment. The SDR framework incorporates various tools, including the MoveIt! planning framework [34]. In this paper, our primary focus is on the SDR subcomponent responsible for emulating human movement. 3.2.2. Two-Dimensional (2D) projection of cell – search grid formation The planning system that generates the operator’s path conducts a heuristic search within a 2D grid, as will be further analyzed in Section 3.2. The objects projected onto the 2D map might be either static or dynamic. Static objects remain stationary, and their projections are made only once during the initialization phase; such objects are worktables, columns etc. On the other hand, dynamic objects can change multiple positions or/and poses during a cycle, like robots. The grid is an “m x n” matrix with values ranging from 0 to 100. Accessible or unoccupied grid cells have a value of 0, while restricted or occupied cells carry a value of 100. Moreover, the map is generated with respect to a predefined reference frame, specifically the base of the robot. This means that the elements of the 2D grid matrix incorporate two known values - the x and y coordinates of the reference frame. The procedure for projecting the robot onto the 2D map is outlined below: Primarily, the robot’s link positions serve as a reference to construct a rectangular volume around the links (see trasnforms in Fig. 10). The coordinates, transposed from the robot’s base link, corresponding to the 2D plane that parallels the structured map, are associated with a matrix element that has a smaller distance, as shown in Fig. 10. For simplicity, the distance comparison is executed separately for rows and columns, reducing computational complexity from O(n 2 ) to O(n) +O(n). After projecting the points, a sorting algorithm organizes the points within the matrix map in a clockwise pattern. Subsequently, a polygon-filling algorithm is employed, facilitating a conceptual representation of the robot on the 2D grid (Fig. 10). People do exercise caution and prefer maintaining a certain distance from robots [35], so the robot’s projection is expanded using a technique commonly used in image processing. This expansion of the robot’s projection is achieved by applying a 2D Gaussian blur filter [36], where each element of the array is convolved with a Gaussian kernel, then all the elements are summed to generate the output array. To mitigate computational complexity in a 2D matrix, the 2D Gaussian blur can be implemented using two separate 1D Gaussian blurs (for rows and columns), resulting in a computational complexity of 2 *O(n) rather than O (n 2 ). Fig. 11 represents the aforementioned application of the Gaussian blur. The rationale in using the Gaussian filter is the utilization of a computationally efficient expansion of robot project with a reducing occupancy from the actual obstacle onto the 2D map, preventing human simulated paths from being too close to the robot projection. 3.2.3. Human trajectory planner and controller The human trajectory planner utilized in the simulation environment, is based on the A* algorithm to generate human trajectories. The heuristic function is based on the Euclidean distance estimation between neighboring nodes [37]. It uses a normalized input from the map values, ranging from [0100] to [0,1], to assess the transit difficulty from the parent node (n p ) to the child nodes (n c ). Each transition along an axis is valued at 1 during the shift from np to n c . The role of the human motion controller is to carry out the trajectory generated by the A* planner. Given the dynamic changes in the map, objects may intersect or come near the preplanned path. Consequently, the controller’s task is to execute one node movement at a time, ensuring that no obstacles interfere with the planned trajectory. To counter the memory complexity challenges stemming from the A* algorithm, the search grid discretization value was subjected to numerous experiments, and areas of the grid that would not contribute to a path solution were designated as obstacles. The Chebyshev distance [38] was chosen for the cost function, as movement along the 2D map has eight possible transitions (horizontal, vertical and diagonal). The human trajectory planner aims to provide the controller solely with valid, unimpeded paths as illustrated in Fig. 12, in less than ~100 ms. Fig. 10. Frames correlation (left) and frame projection in the 2D map (right). K. Katsampiris-Salgado et al. Journal of Manufacturing Systems 74 (2024) 980–994 987 3.2.4. Synthetic dataset generation The simulation execution is launched initializing all the resources that are involved in the process (poses, process stage etc.). The tasks assigned to the robot correspond to specific poses and tool actuations, similar to those for the human operator. Upon completing a task, the human operator is directed to the next position. In alignment with reallife situations, this position will not intersect with the robot’s path and will not always remain constant. To introduce variability in both the x and y directions, a normal distribution is applied. It’s crucial to eliminate irregularities in the dataset, specifically "zigzag" movements due to the 2D grid discretization and mimic the sensor noise encountered in real-time human position estimation. For this purpose, a polynomial interpolation was implemented in the synthetic dataset to smoothen human trajectories. Specifically, the Lagrange interpolating polynomial P(x) of degree ≤(n −1) that passes through the n points (x1, y1 =f (x1)), (x2, y2 =f(x2)),⋅⋅⋅, (xn, yn =f(xn)) was employed. Additionally, stochastic noise following a normal distribution was introduced to the human dataset, mimicking the acquisition deviations of the Azure Kinect sensor in narrow field of view mode, as demonstrated by experimental results in [39]. With the induction of this noise, the dataset closely approximates the real sensor acquisitions, enhancing the realism of the simulation. 3.3. Implementation of a deep learning network for human future trajectory prediction 3.3.1. Modelling approach The proposed A* implementation could be utilized for the prediction model in real-time as a planning-based method, though it was not selected due to several issues, such as memory complexity, lack of guaranteed response time and strict task-oriented execution, that are not ideal for this soft safety application. In contrast, the proposed RNN model, which belongs to the pattern-based category, is more suitable for complex environments with complicated, unspecified dynamics, enabling predictions without strict task-oriented inputs. Pattern-based methods are highly dependent on the specific case, as they depend on statistical behavioral patterns of the acquired operator’s trajectories, though with ample data collection, good results may be achieved [26]. The adequacy of the data is very important as apart from the common patterns, the function approximator (in the specific case the RNN) may also learn infrequent patterns too, as those might occur from rare robot planning. Data adequacy also corroborates the utilization of synthetic datasets derived from simulations, as they may run a great number of iterations in a small amount of time. Fig. 11. Robot’s projection in the 2D map: without gaussian filter (left) with gaussian filter (right). Fig. 12. Human trajectory simulation using A* pathfinding algorithm, within the RVIZ [48] environment. K. Katsampiris-Salgado et al. Journal of Manufacturing Systems 74 (2024) 980–994 988 3.3.2. Human tracking using Azure Kinect To perform accurate human detection that will guarantee that operators are tracked consistently within the workspace and their position is shared redundantly to the prediction model, an additional sensor layout was added to the thermal camera detection system, including two Azure Kinect depth cameras. The sensing network is designed to detect human operators by analyzing both 2D image and depth data collected. Each point cloud undergoes processing via the body tracking algorithm, leading to the evaluation of the human body index map for various human joints. Subsequently, the positions and orientations of human articulations are selected based on the confidence values determined by the algorithm. The approach described above is illustrated in Fig. 13. To begin with, the efficient placement of sensors is crucial, as it facilitates more precise detection and evaluation of the operator’s joint positions and orientations. Given the availability of two RGBD sensors, they were strategically positioned at a fixed 90 degrees relative rotation. This orientation mitigates the problem of occlusions that may occur when the operator’s body blocks the view of certain joints from the camera, such as when the operator stands sideways. Since multi-sensor support through the official Microsoft Azure Kinect ROS driver was under development at the time of the study, we have modified the driver’s source code for our research to enable simultaneous utilization of both sensors. The node responsible for publishing the raw sensor data and body tracking evaluations shares the information from each sensor to two separate topics. The driver initiates both cameras using their unique serial numbers, allowing for data segregation based on these numbers. To access the data shared by the sensors, the translational and rotational transformations between the camera frames should be applied to the published data. This data is then further transformed to the necessary world/global frame, as demonstrated in Fig. 14, using human joints as an example. Ensuring the precise identification of the translational and rotational placement of the sensors is crucial. This is achieved through the extrinsic cameras’ calibration, resulting in a uniform point cloud. The calibration process, which identifies the transformation of the camera’s coordinate system relative to a predefined frame, involves two steps: •Camera position estimation using a fiducial marker. •Estimation refinement via the Iterative Closest Point (ICP) [49]. A fiducial marker, which possesses a specific pattern easily identifiable by computer vision applications, was used in this case. We employed an AprilTag [40] marker, recognizable by the OpenCV vision library. Iterative Closest Point (ICP) is an algorithm that iteratively minimizes the geometric difference between two sets of points by estimating the optimal transformation. The Open3D [41] open-source library was utilized, as it already implements ICP. After applying the ICP algorithm, the errors are minimized, resulting in a uniform point cloud. The Azure Kinect Body Tracking SDK was used for 3D detection of the human operator. This software can accurately detect and track multiple humans using deep learning techniques. The two-sensor layout is beneficial as it broadens the monitorable area and, with proper placement, yields more accurate human joint detections in instances of occlusions. The accuracy of the depth sensor’s human body tracking depends on various factors. As highlighted in this study [39], the camera’s field of view settings significantly impact detection accuracy. WFOV has higher error values compared to NFOV within the common reachable FOV. Moreover, the distance from the sensor is a critical factor. If the human is too close to the camera or near the limits of the FOV, the detection may be unachievable or compromised in terms of accuracy [39]. Taking these factors into account, we selected a combination of the Azure Kinect sensors’ available FOVs to cover the entire workspace. The WFOV monitors the wide length of the orthogonal area, while the NFOV monitors the smaller side. The operator’s joint positions are captured using the Azure Kinect Body Tracking SDK and the Azure Kinect ROS driver. The data is returned as a visualization_msgs/MarkerArray.msg ROS message (each marker in the MarkerArray represents one of the 32 joints detected by the AKBT SDK). This ROS object contains all the information regarding the joints’ orientation and transformation relative to the RGDB sensor. A slight modification was made within the driver to add the sensor’s confidence values to the "alpha" instance of each Marker. Fig. 13. Body tracking approach using depth sensors. K. Katsampiris-Salgado et al.