Thermal Imaging Detection System: A Case Study for Indoor Environments

Citation: Drahanský, M.; Charvát, M.; Macek, I.; Mohelníková, J. Thermal Imaging Detection System: A Case Study for Indoor Environments. Sensors 2023,23, 7822. https://doi.org/10.3390/s23187822 Received: 10 April 2023 Revised: 19 August 2023 Accepted: 23 August 2023 Published: 12 September 2023 Copyright: © 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/). sensors Article Thermal Imaging Detection System: A Case Study for Indoor Environments Martin Drahanský1,*, Michal Charvát2, Ivo Macek 3and Jitka Mohelníková2 1TrendBit s.r.o., 616 00 Brno, Czech Republic 2Faculty of Civil Engineering, Brno University of Technology, 612 00 Brno, Czech Republic; [email protected] (M.C.); [email protected].cz (J.M.) 3National Museum Prague, 110 00 Prague, Czech Republic; [email protected] *Correspondence: [email protected] Abstract: Currently, there is an increasing need for reliable mechanisms for automatically detecting and localizing people—from performing a people-flow analysis in museums and controlling smart homes to guarding hazardous areas like railway platforms. A method for detecting people using FLIR Lepton 3.5 thermal cameras and Raspberry Pi 3B+ computers was developed. The method creates a control and capture library for the Lepton 3.5 and a new person-detection technique that uses the state-of-the-art YOLO (You Only Look Once) real-time object detector based on deep neural networks. A thermal unit with an automated configuration using Ansible encapsulated in a custom 3D-printed enclosure was used. The unit has applications in simple thermal detection based on the modeling of complex scenes with polygonal boundaries and multiple thermal camera monitoring. An easily deployable person-detection and -localization system based on thermal imaging that supports multiple cameras and can serve as an input for other systems that take actions by knowing the positions of people in monitored environments was created. The thermal detection system was tested on a people-flow analysis performed in the Czech National Museum in Prague. The contribution of the presented method is the development of a small and simple detection system that is easily mountable with wide indoor as well as outdoor applications. The novelty of the system is in the utilization of the YOLO model for thermal data. Keywords: artificial intelligence; infrared imaging; infrared sensors; localization and mapping; neural networks; building security management 1. Introduction 1.1. Topicality of the Problem The human detection and localization problem represents one of the contemporary topical research tasks [ 1 – 5 ]. Detection technology has found applications in many areas of everyday life. It is often used for queue management in shops, people-flow analyses in museums and exhibition places, marketing for determining the best product placement, and people-detecting mechanisms in smart homes with controlled indoor environments. They can help to ensure safety in heavy-machinery workplaces of industry halls or at railway stations by guarding hazard zones. The presented research focuses on a system of people-detection using thermal imaging. Thermal imaging systems are performed on the basis of infrared radiation detection. Thermal imaging sensors can obtain an image of the human body from thermal emissions in the spectral range of long wavelength infrared radiation, from 8 to 14 µ m. These systems do not require any illumination from sunlight or moonlight or the use of an infrared illuminator [6,7]. Sensors 2023,23, 7822. https://doi.org/10.3390/s23187822 https://www.mdpi.com/journal/sensors Sensors 2023,23, 7822 2 of 19 1.2. Related Technologies The following overview of the detection technologies shows the current possibilities of systems used for people-detection. The methods behind these detection algorithms are summarized below. 1.2.1. Detection Technologies Some of the current technologies being used for counting, detecting, and locating people are as follows: • Infrared/laser beam interruption: Such a system counts people passing through a narrow passage, for example, a doorway. A transmitter device is installed on one side of the passage, and a receiver is installed on the other [ 8 ]. The two devices are connected together and form an invisible barrier of light. When an object breaks the connection between the transmitter and the receiver, the system registers a plus-one count. • Light Detection and Ranging (LIDAR) is a method for measuring distances (ranges) by illuminating the target with a laser light and measuring the reflection with a sensor. It consists of a single sensor, usually placed above a passage. The sensor acts as a transmitter and a receiver at the same time. The device casts laser beams in several directions and precisely measures the time required for each reflected beam to return to the sensor. It is possible to calculate the distance of each laser beam and, therefore, to create a depth map revealing objects in its field of view [9,10]. • GPS/Wi-Fi/Bluetooth tracking is a method of tracking people using wireless technologies such as Wi-Fi or Bluetooth. These smart technologies make it easy to triangulate the position of people and to create movement maps [11]. • Projecting structured light: Such a system usually consists of two parts—a camera and a projector. The projector casts structured light on the scene. The structured light is usually a horizontal black-and-white-lined pattern or a checkerboard pattern. The camera is then used to view the scene, and by analyzing deformations in the projected pattern, a depth map is constructed. This method is not directly used to detect or locate people, but is more often used for obtaining 3D models of relatively small objects or continuous depth maps [12]. • Three-dimensional stereo video analysis: Such a system consists of two precisely calibrated cameras viewing a scene. The technology is somewhat similar to human vision—two eyes viewing a scene, with the brain extracting depth information from differences in the two shifted images. • Monocular video analysis is quite similar to 3D stereoscopic vision; however, for monocular video analysis, no depth map is used, as the monocular video analysis uses only a single camera [13]. 1.2.2. Detection Algorithms People-detection is a task of placing boxes around objects in an image. The objectdetection problem has been studied since 1960. Facial detectors appeared about twenty years ago. Then, detection algorithms were developed. The Viola–Jones algorithm [ 14 ], released in 2001, uses hand-coded features that are fed into a support vector machine (SVM) classifier. The hand-coded features for facial detection are the positions of the eyes, nose, and mouth and their relation with respect to each other. The algorithm performs correctly when detecting faces, matched with the hand-coded features; however, a problem with detecting rotated faces or faces in any other orientation has appeared [15]. In 2005, a new detection algorithm was released. A histogram of oriented gradients (HOG) [ 16 ] is used as a feature descriptor, where each pixel of the image is replaced by a gradient showing the direction of decreasing pixel intensity with respect to the surrounding pixels. The image is then divided into squares, and all gradients inside each square are merged into a single most-dominant gradient. During this process, an image is replaced by a simple representation of the essence of the image using gradients. The algorithm uses a Sensors 2023,23, 7822 3 of 19 similarity metric to determine how close an image is to the object we are looking for by comparing their gradient patterns [17]. Progress came in 2012 when the CNN-based system AlexNet was developed [ 18 ]. Convolutional neural networks have been known since the 1990s. Recent increases in processing power and the amount of data are because of the neural network potential. The convolutional neural network, in essence, learns the feature descriptors on its own during the training process, as opposed to the previous two methods in which they had to be crafted by hand. These three algorithms, as described, correspond more to object classifiers, meaning that they can tell what the object in an image is if there is nothing but one object in it. It cannot detect and classify multiple objects in a single image. This has, however, been proven to be possible by repurposing any image classifier. The classifier can sequentially classify every part of an image through a sliding window, and detections with the highest confidence score, in that case, represent the output of the detector. This approach is, however, extremely computationally expensive. In 2014, the R-CNN object detector [ 19 ] was released, followed by the Fast RCNN [ 20 ] and Faster R-CNN [ 21 ] in 2015. They were used for the selective search technique instead of a sliding window to create sets of bounding boxes that were later completed in the classifier to cut down the number of operations. In 2015, the new YOLO algorithm was introduced. You Only Look Once (YOLO) is a state-of-the-art real-time object-detection system [ 22 ]. When compared with all other detectors, it takes a completely different approach. Instead of repurposing an image classifier and using it to classify different regions in the image, this approach uses a neural network that takes an image to the input and, in one single pass, outputs the regions with detected classes and scores for every region. This new approach is required to redefine the parametrization of the object detection. Every image is split into a grid, where each grid-cell is responsible for predicting several bounding boxes and confidence scores for each bounding box, saying how sure the detector is that a certain bounding box actually contains an object and, if there is an object, what kind of object it is. In 2016, the second version of YOLO was released [ 23 ], featuring important improvements that increased its overall accuracy. The detector finished training on images with higher resolutions and changed the way of representing bounding boxes. YOLOv2 uses dimension clusters to represent bounding boxes. Using unsupervised learning, the creators extracted the five most common shapes and sizes of bounding boxes occurring in the VOC 2007 image dataset, and used them as templates for bounding boxes that each cell in the YOLOv2 detector can detect. YOLOv2 also uses multi-scale training, meaning that the input image size is not fixed throughout the training process, but changes on the fly, resulting in a more robust detector, as it works better on differently sized images. The third version, YOLOv3 [ 24 ], brought even more improvements in 2018. One of them is the support for multiple labels. More importantly, YOLOv3 uses a new backbone (or feature extractor part of the network), Darknet-53. The network has 53 convolutional layers with short-cut connections, allowing for the extraction of finer-grained information from the image. This significantly improves the detection of accuracy of small objects. Unlike the previous versions, YOLOv3 makes bounding box predictions at three different scales, improving the accuracy of the detector. The fourth version of YOLO [ 25 ] was released in 2020. This version promised even better accuracy and speed, effectively dominating every other solution in the field of realtime object detection. For this version, the creators performed an ablation study to test and select the most effective training optimization methods, which lead to improvements in accuracy with minimum additional computational cost. The tested methods were mostly data augmentation techniques that could potentially increase the descriptive power of the feature-extracting part of the network. Some of the data augmentation methods are the following: edge map, flip, rotate, detexture, cutmix, mosaic, dropblock regularization, and so on. A new activation function has been tested, as well as other specialized techniques like cross-stage partial connections or multi-input Sensors 2023,23, 7822 4 of 19 weighted residual connections. The optimizations also covered selecting the optimal hyperparameters of the model, like the number of training steps, batch size, learning rate, momentum, weight decay, or minibatch size. YOLOv4 is superior to all other object detectors in terms of both its speed and accuracy. 2. Materials and Methods The presented system is focused on people-detection applications. Using a thermal imaging camera module to solve the problem of people-detection, it belongs to the monocular video analysis section and brings several advantages when compared with other approaches. Only a single camera is needed, so there is no need for extremely precise hardware calibration of the system, as with the stereo vision or structured-light projection. It is possible to detect and also locate individuals, in contrast with infrared/laser beam or light travel techniques, which can only count objects entering and leaving an area. Advantage of the system is in privacy. Facial recognition is not provided. This makes this approach more suitable for places where privacy plays an important role, e.g., at workplaces or homes. The system is not influenced by light conditions. Disadvantage is in infrared radiation reflections. Thermal images are not dependent on lighting conditions of the scene, which makes the system very effective during the night. The largest disadvantage of monocular image analysis is the missing depth dimension. The lower camera resolution with the missing depth dimension causes locations of the detected objects to be only a rough approximate, as the missing dimension has to be estimated based on some assumption, like if the object is touching the ground. The idea of the detection and localization of people due to the thermal camera is to capture a thermal image, detect objects corresponding to people in the image, and estimate locations of each object in a model of the scene using the perspective projection. The method of detection of people using thermal imaging camera deals with utilizing a single thermal-camera module, a single-board computer, and image processing. Using a small thermal camera module eliminates the possibility of person and/or face recognition while preserving the functionality of detecting and even locating people. The solution to the problem of people-detection based on thermal imaging is, therefore, a viable option for places where privacy plays an important role. This method was developed within the frame of a research project. Main tasks of the project were: • Thermal capture system composed of a Lepton 3.5 camera and a Raspberry Pi 3B+ single-board computer; • C++ capture library, allowing one to read thermal images from the camera in both raw format and false color over SPI interface; • Python control script, allowing one to issue commands to the camera over I 2 C in order to change a color palette, format, control automatic gain (AGC), perform flat-field correction (FFC), and other functions; • Python scripts for person-detection and single-camera rectangular scene abstraction, allowing for reverse-projecting image points of detected people into a 3D scene model. 2.1. Thermal Capture Unit The thermal capture unit, which can be used for standalone or remote capture of thermal data, was completed. The unit can be placed anywhere with electric and network connection and consists of FLIR’s Lepton 3.5 [ 26 ] thermal camera module (Lepton 3/3.5 with breakout board is shown in Figure 1) and a custom PCB with a circuit controlling the thermal module and the Raspberry Pi 3B+ single-board computer, which communicates directly with the camera. All these three parts are enclosed in a custom-designed 3D-printed enclosure box. Sensors 2023,23, 7822 5 of 19 Sensors 2023, 23, x FOR PEER REVIEW 5 of 20 directly with the camera. All these three parts are enclosed in a custom-designed 3Dprinted enclosure box. Figure 1. Lepton 3/3.5 with breakout board [27]. The thermal unit uses Raspbian Buster Lite, which is a minimal operating system with only 435 MB in size. Since it is the minimal system, it is necessary to install all dependencies and libraries manually. The exact same steps would have to be performed on every thermal unit and repeated for every new unit. Therefore, it only makes sense to use a tool to automate the steps of preparing the environment on thermal units for running the detection and localization system. The Raspberry Pi 3B+ computer is used to directly communicate with the thermal camera, and is therefore the “brain” of the thermal unit. The Raspberry Pi 3B+ is the last revision of the third generation single-board computer. It is a low-cost, credit-card sized computer capable of performing everything one might expect from a regular desktop computer. The Raspberry Pi runs a Debian-based operating system, Raspbian Bustler Lite. The computer has a built-in hardware support for SPI, I 2 C, UART, Bluetooth, and Wi-Fi communication. In general, input/output (GPIO) pins are also extremely important for interfacing with other electronic devices. The Raspberry Pi 3B+ single-board computer is shown in Figure 2. Figure 2. Raspberry Pi 3B+ single-board computer [28]. The Ansible [29] (software intended to do provisioning, configuration management, and application-deployment) was used by us for the automatic deployment. It is an agentless tool which temporarily connects to its targets via ssh to perform tasks specified in so-called Ansible playbooks. Figure 1. Lepton 3/3.5 with breakout board [27]. The thermal unit uses Raspbian Buster Lite, which is a minimal operating system with only 435 MB in size. Since it is the minimal system, it is necessary to install all dependencies and libraries manually. The exact same steps would have to be performed on every thermal unit and repeated for every new unit. Therefore, it only makes sense to use a tool to automate the steps of preparing the environment on thermal units for running the detection and localization system. The Raspberry Pi 3B+ computer is used to directly communicate with the thermal camera, and is therefore the “brain” of the thermal unit. The Raspberry Pi 3B+ is the last revision of the third generation single-board computer. It is a low-cost, credit-card sized computer capable of performing everything one might expect from a regular desktop computer. The Raspberry Pi runs a Debian-based operating system, Raspbian Bustler Lite. The computer has a built-in hardware support for SPI, I 2 C, UART, Bluetooth, and Wi-Fi communication. In general, input/output (GPIO) pins are also extremely important for interfacing with other electronic devices. The Raspberry Pi 3B+ single-board computer is shown in Figure 2. Sensors 2023, 23, x FOR PEER REVIEW 5 of 20 directly with the camera. All these three parts are enclosed in a custom-designed 3Dprinted enclosure box. Figure 1. Lepton 3/3.5 with breakout board [27]. The thermal unit uses Raspbian Buster Lite, which is a minimal operating system with only 435 MB in size. Since it is the minimal system, it is necessary to install all dependencies and libraries manually. The exact same steps would have to be performed on every thermal unit and repeated for every new unit. Therefore, it only makes sense to use a tool to automate the steps of preparing the environment on thermal units for running the detection and localization system. The Raspberry Pi 3B+ computer is used to directly communicate with the thermal camera, and is therefore the “brain” of the thermal unit. The Raspberry Pi 3B+ is the last revision of the third generation single-board computer. It is a low-cost, credit-card sized computer capable of performing everything one might expect from a regular desktop computer. The Raspberry Pi runs a Debian-based operating system, Raspbian Bustler Lite. The computer has a built-in hardware support for SPI, I 2 C, UART, Bluetooth, and Wi-Fi communication. In general, input/output (GPIO) pins are also extremely important for interfacing with other electronic devices. The Raspberry Pi 3B+ single-board computer is shown in Figure 2. Figure 2. Raspberry Pi 3B+ single-board computer [28]. The Ansible [29] (software intended to do provisioning, configuration management, and application-deployment) was used by us for the automatic deployment. It is an agentless tool which temporarily connects to its targets via ssh to perform tasks specified in so-called Ansible playbooks. Figure 2. Raspberry Pi 3B+ single-board computer [28]. The Ansible [ 29 ] (software intended to do provisioning, configuration management, and application-deployment) was used by us for the automatic deployment. It is an agentless tool which temporarily connects to its targets via ssh to perform tasks specified in so-called Ansible playbooks. A configuration on an SD card is due to writing the Raspbian Bustler Lite image to the SD card using, for example, the balenaEtcher tool—free open-source utility for creating live SD cards and USB flash drives [ 30 ], it can be inserted into the Raspberry Pi. The computer should boot up and connect to the network according to the configuration. After that, the Ansible can take over. From the master computer, the thermal units are Sensors 2023,23, 7822 6 of 19 configured by running the Ansible playbook. The thermal unit is ready to operate. In order to communicate with the camera using SPI and I2C hardware modules on the Raspberry Pi, they need to be enabled at the kernel level. The v4l2lepton3 library is the main software part. It is a library that takes care of controlling the camera and retrieving thermal video feed from it. It was designed and implemented from scratch. The library consists of two parts: the C++ application for thermal video manipulation and a Python3 package for the camera control and single-frame manipulation. The control software is implemented in the v4l2lepton3.control Python3 module as a part of the v4l2lepton3 Python3 package available in the git repository. In the implementation, each command has exactly one definition, which automatically generates allowed methods and contains a translation map for each option that the command can set. For people-detection, the real YOLO detector is used based on DNN module on thermal dataset. The DNN module implements forward pass (inferencing) with deep networks, and is pre-trained using deep learning frameworks like Darknet. The Lepton camera sends video frames over the SPI interface on 20 MHz, which implies that the length of wires connecting the Lepton camera to the Raspberry Pi needs to be as short as possible—maximum of 20 cm in order to provide a stable connection without interference and transmission errors. This condition enforces the need for the Lepton camera and the Raspberry Pi to be physically close to each other. Together, they form a thermal unit. The camera contains a sensor sensitive to long-wavelength infrared radiation in spectral range from 8 to 14 µ m [ 31 ]. The camera module is smaller than a dime and provides images with decent resolution of 160 by 120 pixels. The effective frame rate of the camera is only 8.7 Hz. The camera only requires a low voltage supply and has a small power consumption of about 160 mW. For better manipulation with the camera module, a breakout board was used. The power supply provides the camera module with three voltages: 1.2, 2.8, and 2.8–3.1 V. The breakout board also supplies the camera with the master clock signal. The camera uses two interfaces for the communication: •SPI, for transferring the video frames from the camera to the SPI master device; •I2C, for receiving control commands from the I2C master device. In order to make the whole thermal unit transferable, protected, and professionally looking for quick demonstrations or real-life deployment, an enclosure has been designed to fit and mount all of its components—the Raspberry Pi 3B+, the custom camera switch circuit board, and the Lepton 3.5 camera. The thermal unit case is composed of two parts— an enclosure box for the Raspberry Pi with the power switch and a camera chassis that is mounted to the top of the first part with a bit of slack that allows the camera chassis to be moved along the horizontal axis. The enclosure box was designed in Sketchup15 software and exported into.stl format for 3D-printing. The Lepton 3.5 camera chassis model was created using the official Lepton 15Sketchup—Trimble design software [ 32 ]. Breakout board [ 27 ] was taken from portal Thingiverse [33]. Two pins were added in the lower part of the back side of the chassis to serve as pivot points around which the camera could move. The camera chassis with the Lepton camera is inserted into the hole on the right side of the top piece of the enclosure. The SPI and I 2 C interfaces are connected directly to the Raspberry Pi via approximately 10 cm long jumper wires. Two power wires are connected to the custom PCB with the switch, which is mounted in the enclosure underneath the camera, right next to the Raspberry Pi. From the custom PCB, there are three wires going to the Raspberry Pi directly—to GND, 5 V, and a virtual GPIO-15 pin. An assembled thermal unit is shown in Figure 3. Sensors 2023,23, 7822 7 of 19 Sensors 2023, 23, x FOR PEER REVIEW 7 of 20 custom PCB, there are three wires going to the Raspberry Pi directly—to GND, 5 V, and a virtual GPIO-15 pin. An assembled thermal unit is shown in Figure 3. Figure 3. Finished thermal unit with Raspberry Pi 3B+, custom power switch, and the Lepton 3.5 thermal camera in the chassis. 2.2. Capture and Control Library The v4l2lepton3 library is the main software part of the project. It is a library that takes care of controlling the camera and retrieving thermal video feed from it. The library consists of two parts: the C++ application for thermal video manipulation and a Python3 package for camera control and single-frame manipulation. The Lepton camera provides a command and control interface (CCI) via a two-wire interface almost identical to that of I2C, with the only difference being that all transactions must be 16 bits in length. All Lepton’s registers are 16 bits wide. Lepton camera offers 4 control registers and 16 data registers, which are all 16 bits wide and are used by the host (master) device to issue commands to the camera. A command is issued by writing and reading particular registers in the camera via I2C. The exact process is described in the CCI documentation [34]. For implementation, each command has exactly one definition, which automatically generates allowed methods and contains a translation map for each option that the command can set. The implementation of the capture software follows the server–client model. It is written in C++ and forms a single process that runs on the Raspberry Pi computer to achieve maximum speed. The thermal unit behaves like a synchronous server. Its server process listens on a port and waits for a client to connect. Once a client is connected, it initializes the SPI interface and starts pulling frames from the camera and sending them over the open socket. When the connection to a client is lost, the thermal unit stops communicating with the Lepton camera and starts listening for another client. The sequential graph of the server–client model is presented in Figure 4. Figure 3. Finished thermal unit with Raspberry Pi 3B+, custom power switch, and the Lepton 3.5 thermal camera in the chassis. 2.2. Capture and Control Library The v4l2lepton3 library is the main software part of the project. It is a library that takes care of controlling the camera and retrieving thermal video feed from it. The library consists of two parts: the C++ application for thermal video manipulation and a Python3 package for camera control and single-frame manipulation. The Lepton camera provides a command and control interface (CCI) via a two-wire interface almost identical to that of I 2 C, with the only difference being that all transactions must be 16 bits in length. All Lepton’s registers are 16 bits wide. Lepton camera offers 4 control registers and 16 data registers, which are all 16 bits wide and are used by the host (master) device to issue commands to the camera. A command is issued by writing and reading particular registers in the camera via I2C. The exact process is described in the CCI documentation [ 34 ]. For implementation, each command has exactly one definition, which automatically generates allowed methods and contains a translation map for each option that the command can set. The implementation of the capture software follows the server–client model. It is written in C++ and forms a single process that runs on the Raspberry Pi computer to achieve maximum speed. The thermal unit behaves like a synchronous server. Its server process listens on a port and waits for a client to connect. Once a client is connected, it initializes the SPI interface and starts pulling frames from the camera and sending them over the open socket. When the connection to a client is lost, the thermal unit stops communicating with the Lepton camera and starts listening for another client. The sequential graph of the server–client model is presented in Figure 4. The connection between the server and the client is realized using a TCP connection. The TCP transport protocol has been chosen because it ensures in-order delivery of every packet. If packets get lost or arrive out of order, it would not be possible to assure proper reconstruction of each frame. The server is sending the data frame by frame, pixel by pixel. The client keeps receiving bytes until 160 × 120 × 2 bytes are obtained. From these data, the client reconstructs the thermal frame in its raw format (Y16). Since the stream may be compressed by the zlib stream compressor, it is important to receive the whole frame and then decompress it. Sensors 2023,23, 7822 8 of 19 Sensors 2023, 23, x FOR PEER REVIEW 8 of 20 Figure 4. Sequential graph of the server–client model. The connection between the server and the client is realized using a TCP connection. The TCP transport protocol has been chosen because it ensures in-order delivery of every packet. If packets get lost or arrive out of order, it would not be possible to assure proper reconstruction of each frame. The server is sending the data frame by frame, pixel by pixel. The client keeps receiving bytes until 160 × 120 × 2 bytes are obtained. From these data, the client reconstructs the thermal frame in its raw format (Y16). Since the stream may be compressed by the zlib stream compressor, it is important to receive the whole frame and then decompress it. 2.3. Scene Reconstruction The scene reconstruction and point protection from the camera to the 3D scene represents projecting objects from the image to the 3D scene model. In order to approximate coordinates of an image object in world space, it is necessary to understand the camera’s location and orientation in space. Knowing the pose of the camera allows us to reconstruct the 3D scene and display the camera and detected objects in it. The implemented scene abstraction and the problem of computation of correspondence between image coordinates of a bounding box and 3D coordinates in the scene model was solved. The detection process yields bounding boxes around detected people represented by image coordinates [35–44]. The next step is to create an abstraction of the environment monitored by the camera, and then translate each detected object into an approximate location in the model of the environment. In order to approximate coordinates of an image object in space, it is necessary to understand the camera’s location and orientation. The pose of the camera allows one to reconstruct the 3D scene and display the camera and detected objects. The camera-pose estimation problem is often referred as the Perspective-n-point problem (PnP). The PnP is a problem of estimating the pose of a calibrated camera. By pose, the camera position and orientation is determined with respect to another coordinate system. The camera pose can be determined via rotation matrix and translation vector. Solving the PnP problem requires pairs of corresponding 3D to 2D mapping points [37,40,44,45]. Given Figure 4. Sequential graph of the server–client model. 2.3. Scene Reconstruction The scene reconstruction and point protection from the camera to the 3D scene represents projecting objects from the image to the 3D scene model. In order to approximate coordinates of an image object in world space, it is necessary to understand the camera’s location and orientation in space. Knowing the pose of the camera allows us to reconstruct the 3D scene and display the camera and detected objects in it. The implemented scene abstraction and the problem of computation of correspondence between image coordinates of a bounding box and 3D coordinates in the scene model was solved. The detection process yields bounding boxes around detected people represented by image coordinates [35–44]. The next step is to create an abstraction of the environment monitored by the camera, and then translate each detected object into an approximate location in the model of the environment. In order to approximate coordinates of an image object in space, it is necessary to understand the camera’s location and orientation. The pose of the camera allows one to reconstruct the 3D scene and display the camera and detected objects. The camera-pose estimation problem is often referred as the Perspective-n-point problem (PnP). The PnP is a problem of estimating the pose of a calibrated camera. By pose, the camera position and orientation is determined with respect to another coordinate system. The camera pose can be determined via rotation matrix and translation vector. Solving the PnP problem requires pairs of corresponding 3D to 2D mapping points [ 37 , 40 , 44 , 45 ]. Given those mapping points, estimating the pose is a matter of solving a system of linear equations. At least four pairs of points are required to find a solution. It represents a scene-abstraction script, allowing one to model a rectangular scene with a single camera and the reverse-project detected people into it. The PnP problem can be expressed by Equation (1), which comes from the perspective projection (world to screen or world to image transformation): Pi=K[R|t]Pw(1) Sensors 2023,23, 7822 9 of 19 where Piis an image point (2D); Kis a matrix of intrinsic camera parameters; Ris a rotation matrix; tis a translation vector; Pwis a world point (3D). The expanded form of Equation (1) can be found in Equation (2):   xi yi 1  =  fxγcx 0fycy 0 0 1    r00 r01 r02 r10 r11 r12 r20 r21 r22       tx ty tz       xw yw zw 1     (2) where fxand fyare focal lengths; cxand cyare center-point coordinates of the image (principal point); γis axis skew (usually assumed 0). The [ R | t ] matrix is usually extended into a single 4 × 4 matrix for the sake of convenience—as seen in Equation (3). This matrix allows one to project points from the world to camera space (coordinate system), and, thus, is sometimes referred to as world to camera, world to view, or simply view matrix. [R|t]=     r00 r01 r02 tx r10 r11 r12 ty r20 0 r21 0 r22 0 tz 1     (3) The matrix of intrinsic camera parameters K represents the transformation of a point from camera to screen (or, alternatively, image) space. The matrix can be assembled from known camera parameters, such as resolution and field of view or focal lengths (more on that later). By plugging image points (2D) and corresponding world points (3D) into Equation (2), it is possible to compute rotation and translation vectors, and therefore to construct the world to camera or view matrix (3), which can be used to transform points from world into camera space. In the new implementation, the scene model is stored in a JSON (JavaScript Object Notation) configuration file, which contains all other positions of cameras and a list of boundaries with their corresponding names and displayed colors. The boundaries are stored as a list of vertices that are connected one by one. The mapping points are then used to calculate the screen to world transformation matrix for every camera when the scene is loaded. The transformation matrix is then used to project detected objects into the scene model. The result of the scene calibration is a configuration file of a scene model in a singlecoordinate system with cameras with known projection matrices. A calibrated camera shows its field of view using visible arms on the ground plan. With the screen to projection matrices, it is possible to assign a line in the same 3D coordinate system to each image pixel of every camera. These lines are then used for localizing detected people and placing them into the scene. In order to model a multi-room exposition or any larger complex environment with multiple cameras, the scene abstraction had to be rewritten almost from scratch and improved significantly. 3. Results The simple detector—YOLO pre-trained for thermal tasks based on the FLIR thermal dataset was used. The approach required us to redefine the parametrization of the detection. Every image is split into a grid, where each grid-cell is responsible for predicting several Sensors 2023,23, 7822 16 of 19 thermal dataset and use it to retrain a YOLO detection model which has been originally trained on a large image dataset. The process of configuring the thermal unit was automated using Ansible. The v4l2lepton3 control and capture library was designed from scratch. The control part supports many more commands with all methods and translated options. The capture part was split into two parts—a server and a client. The C++ multithreaded implementation of the server was sped-up using double-segment buffering, double-frame buffering, and a reduced number of system calls. It does not lose synchronization with the camera. It can recover from any kind of error and allows for zlib compression. There are two client implementations available. The C++ one uses a virtual video device to bring the remote thermal feed into the local machine for generic processing; the Python implementation is simple and easy to use or include in other projects. It is used in the detection software, but can also be used for quick previews. The scene-abstraction software has been redesigned so that, now, a scene is abstracted in a JSON configuration file and supports multiple polygonal boundaries and multiple cameras, which can be calibrated visually using a visual calibrator tool. Finally, the comparative thermal detector showing poor results in larger scenes with more people has been replaced by the new state-of-the-art YOLOv4 real-time object detector trained on a custom thermal dataset that was created by merging the FLIR’s public thermal dataset and a custom one created in the Czech National Museum within the scope of an ongoing cooperation. The new detector is far superior to the comparative detector and can reliably detect people even in some of the most challenging situations. The final detection system loads a preconfigured scene, connects to all cameras, displays their real-time thermal feeds, and, after the detection is performed, the detected persons are marked in the ground plan representation of the scene. The new detector was applied to the captured data from the National Museum with the aim of constructing a heatmap of visitors’ behavior. The built heatmap proves the capabilities of the detection system and may be beneficial for building new exhibitions in the museum in the future. By having the new detector, the process of estimating the locations of detected objects from their bounding boxes becomes an area for possible improvement. The accuracy of an estimated location rapidly decreases with distance because of the low resolution of the thermal camera. About 16 m away from the camera, a difference of one image pixel can easily translate into a 1 m difference in the scene model. If a bounding box around a person is moved even by a few image pixels, its estimated location can change significantly. In order to reverse-project an image point into a single point in the scene model, it is necessary to provide some additional information—for example, the zcoordinate of the searched point. For reverse-projecting the feet of a detected person, the zcoordinate would be set to 0; alternatively, 170 cm would be used for the head position (to represent an average person’s height). Using the head position is usually less accurate than the feet, as the height of people varies naturally. The position of feet works well when the camera is located high above ground or there are not many people in the scene. If neither condition is met, there is a higher possibility that a person would have their feet occluded by a different object. In that case, the system would assume that the person is further away and misplace him/her completely. When a bounding box is touching the bottom of an image, the system expects that the feet of the detected person are not visible and uses the head position instead. This, however, does not solve the issue with occlusions. One solution might be to train the detector to detect two classes—a torso and a whole person. That would require reannotating the whole dataset and longer training with unsure results, because the detector would then detect both the torso and the whole person and the system would have to identify that those two detections belong to the same person, which adds more room for error. Both problems could be solved by adding another camera to observe the same scene from a different angle. The camera would have a priority to localize objects closer to it and both cameras could agree on the same objects. The additional coordinate required for Sensors 2023,23, 7822 17 of 19 placing the detected object into the scene would be provided from the two cameras using stereo vision. This feature shall remain on the top of the list of future upgrades. Another possible improvement could be implementing allowed and blocked area concepts for localization. In the current implementation, there are no rules saying which section of the scene is marked for the possible occurrence of people and there is no way to tell which part of the scene is actually observable from which camera. By being able to determine which area of the scene is observable, it would become possible to exclude incorrect locations of detected people that lie outside the observable part of the scene. These outliers are often caused by thermal reflections, large occlusions, or the small resolution of the camera. The future improvements might also tackle the lens distortion of Lepton cameras, as it becomes apparent for some particular modules. Another interesting feature to implement in the system could be person tracking. Each detected person would obtain an ID, and their movement through the scene would be stored in a database. This kind of data could be used for a more specific type of people-flow analysis where we could, for example, calculate the most typical direction of the movement of people. The candidate technology for object tracking could be the new DeepSORT [ 48 ]. The development of the system could be for 3D thermograms with a porTable 3D measurement system based on geometric calibration and data structure adaptation [49]. Author Contributions: Conceptualization and methodology, M.D.; methodology and data analysis, M.C.; validation, I.M.; formal analysis, J.M. All authors have read and agreed to the published version of the manuscript. Funding: This research received no external funding. Informed Consent Statement: Informed consent was obtained from all subjects involved in the study. Data Availability Statement: Not applicable. Acknowledgments: The thermal imaging method was completed within the frame of research activities of the STRaDe research group at Brno University of Technology, Faculty of Information Technology and in collaboration with the National Museum, Prague. This work was financially supported by the Ministry of Culture of the Czech Republic (DKRVO 2019–2023/26.II.c, National Museum, 00023272); further by Reliable, Secure, and Efficient Computer Systems, IGA—FIT-S-206427; and in collaboration with project No. LO1408 “AdMaS UP—Advanced Materials, Structures and Technologies”, supported by the Ministry of Education, Youth and Sports under the “National Sustainability Programme I”. Conflicts of Interest: The authors declare no conflict of interest. References 1. Herrmann, C.; Ruf, M.; Beyerer, J. CNN-based thermal infrared person detection by domain adaptation. SPIE Defense + Security, Proceedings. In Autonomous Systems: Sensors, Vehicles, Security, and the Internet of Everything; SPIE: Orlando, FL, USA, 2018; Volume 10643, p. 1064308. 2. Davis, J.W.; Keck, M.A. A two-stage template approach to person detection in thermal imagery in Applications of Computer Vision. In Proceedings of the Seventh IEEE Workshops on Applications of Computer Vision (WACV/MOTION’05), Breckenridge CO, USA, 5–7 January 2005; Volume 1, pp. 364–369. [CrossRef] 3. Haider, A.; Shaukat, F.; Mir, J. Human detection in aerial thermal imaging using a fully convolutional regression network. Infrared Phys. Technol. 2021,116, 103796. [CrossRef] 4. Nguyen, D.T.; Li, W.; Ogunbona, P.O. Human detection from images and videos: A survey. Pattern Recogn. 2016 ,51, 148–175. [CrossRef] 5. Dai, C.; Zheng, Y.; Li, X. Pedestrian detection and tracking in infrared imagery using shape and appearance. Comput. Vis. Image Underst. 2007,106, 288–299. [CrossRef] 6. Byrnes, J. Unexploded Ordnance Detection and Mitigation; Springer: Berlin/Heidelberg, Germany, 2009; ISBN 978-1-4020-9252-7. 7. Minkina, W. Theoretical basics of radiant heat transfer—Practical examples of calculation for the infrared (IR) used in infrared thermography measurements. Quant. InfraRed Thermogr. J. 2021,18, 269–282. [CrossRef] 8. Ruser, H.; Pavlov, V. People counter based on fusion of reflected light intensities from an infrared sensor array. In Proceedings of the Informatik 2006—Informatik für Menschen, Dresden, Germany, 2–6 October 2006; Volume 1, pp. 379–383. Sensors 2023,23, 7822 18 of 19 9. Airport Suppliers, Acorel, Automatic High Accuracy Passenger Counting Systems, Saint Peray, France. Available online: https://www.airport-suppliers.com/supplier/acorel/ (accessed on 21 June 2020). 10. Bansal, M.; Southall, B.; Matei, B.; Eledath, J.; Sawhney, H. LIDAR-based Door and Stair Detection from a Mobile Robot. In Proceedings of the Unnambed Systems Technology XII, SPIE Defense, Security, and Sensing, Orlando, FL, USA, 5–9 April 2010; Volume 7692. [CrossRef] 11. Datatilsynet, Tracking in Public Spaces The use of WiFi, Bluetooth, Beacons and Intelligent Video Analytics, Oslo, Norway. 2016. Available online: https://www.datatilsynet.no (accessed on 25 September 2021). 12. McDonald, K. Structured Light 3D Scanning, Instructables. 2009. Available online: http://www.instructables.com/id/Structure d-Light-3D-Scanning/ (accessed on 21 June 2020). 13. Negied, N.K.; Hemayed, E.E.; Fayek, M.B. Pedestrians’ detection in thermal bands—Critical survey. J. Electr. Syst. Inform. Technol. 2015,2, 141–148. [CrossRef] 14. Viola, P.; Jones, M. Rapid Object Detection using a Boosted Cascade of Simple Features. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, Kauai, HI, USA, 8–14 December 2001; Volume 1, pp. 511–518. 15. YOLO-Object Detection. Available online: https://opencv-tutorial.readthedocs.io/en/latest/yolo/yolo.html (accessed on 10 January 2022). 16. Dalal, N.; Triggs, B. Histograms of oriented gradients for human detection. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), San Diego, CA, USA, 20–25 June 2005; Volume 1, pp. 886–893 . 17. Raval, S. YOLO Object Detection, (TensorFlow Tutorial), January 2022. Available online: https://theengineeringofconsciousexper ience.com/yolo-object-detection-tensorflow-tutorial/ (accessed on 10 January 2022). 18. Krizhevsky, A.; Sutskever, I.; Hinton, G.E. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems 25; Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q., Eds.; Curran Associates, Inc.: Lake Tahoe, NV, USA, 2012; pp. 1097–1105. Available online: http://papers.nips.cc/paper/4824-imagenet-classification-with-d eepconvolutional-neural-networks.pdf (accessed on 10 January 2022). 19. Girshick, R.B.; Donahue, J.; Darrell, T.; Malik, J. Rich feature hierarchies for accurate object detection and semantic segmentation, Cornell University, abs/1311.2524. 2014. Available online: http://arxiv.org/abs/1311.2524 (accessed on 10 January 2022). 20. Girshick, R.B. Fast R-CNN, Cornell University, abs/1504.08083. 2015. Available online: http://arxiv.org/abs/1504.08083 (accessed on 10 January 2022). 21. Ren, S.; He, K.; Girshick, R.B.; Sun, J. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. Cornell University, abs/1506.01497. 2016. Available online: http://arxiv.org/abs/1506.01497 (accessed on 10 January 2022). 22. Redmon, J.; Divvala, S.; Girshick, R.; Farhadi, A. You Only Look Once: Unified, Real-Time Object Detection. Computer Vision and Pattern Recognition, Cornell University. 2016. Available online: http://arxiv.org/abs/1506.02640 (accessed on 10 January 2022). 23. Redmon, J.; Farhadi, A. YOLO9000: Better, Faster, Stronger, Cornell University, abs/1612.08242. 2016. Available online: http://arxiv.org/abs/1612.08242 (accessed on 10 January 2022). 24. Redmon, J.; Farhadi, A. YOLOv3: An Incremental Improvement, Cornell University, abs/1804.02767. 2018. Available online: http://arxiv.org/abs/1804.02767 (accessed on 10 January 2022). 25. Bochkovskiy, A.; Wang, C.Y.; Liao, H.Y.M. YOLOv4: Optimal Speed and Accuracy of Object Detection, Cornell University. 2020. Available online: https://arxiv.org/abs/2004.10934 (accessed on 10 January 2022). 26. Flir Lepton ® 3.5 Long Wave Infrared (LWIR) Datasheet; FLIR Commercial Systems, Inc.: Wilsonville, Oregon, USA, 2018; Document number: 500-0659-00-09 Rev: 203. 27. GroupGets LLC. FLIR Lepton Breakout Board Official Page, Schematics, Specifications. 2018. Available online: https://groupget s.com/manufacturers/getlab/products/flir-lepton-breakout-board-v1-4 (accessed on 25 May 2018). 28. Raspberry Pi Foundation. Raspberry Pi 3 Model B+ Product Description. 2016. Available online: https://www.raspberrypi.org/ products/raspberry-pi-3-model-b-plus/ (accessed on 26 May 2020). 29. Ansible. Available online: https://www.ansible.com/ (accessed on 15 February 2022). 30. Balena. Available online: https://www.balena.io/etcher/ (accessed on 15 February 2022). 31. Lepton. Available online: https://www.flir.eu/support/products/lepton/?vertical=microcam&segment=oem#Documents% 20datasheet (accessed on 15 February 2022). 32. SketchUp. Available online: https://www.sketchup.com/ (accessed on 15 February 2022). 33. Thingiverse. Available online: https://www.thingiverse.com/ (accessed on 15 February 2022). 34. Flir Lepton ® Software Interface Description Document (IDD); FLIR Commercial Systems, Inc.: Wilsonville, Oregon, USA, 2018; Document number: 110-0144-04 Rev: 303. 35. Hartley, R. Multiple View Geometry in Computer Vision; Cambridge University Press: Cambridge, UK, 2011; ISBN 9780511811685. 36. Zhang, Z. A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell. 2000 ,11, 1330–1334. [CrossRef] 37. Solem, J.E. Programming Computer Vision with Python: Tools and Algorithms for Analyzing Images; O’Reilly Media, Inc.: Sebastopol, CA, USA, 2012; ISBN 1449316549. 38. Szeliski, R. Computer Vision: Algorithms and Applications; Springer: Berlin/Heidelberg, Germany, 2010; ISBN 1848829345. 39. Mordvintsev, A.; Abid, K. OpenCV-Python Tutorials. Thesis for Computer Science, Can-Tho University. 2017. Available online: https://www.docsity.com/en/opencv-ddddddddddddddddddddddddddddddddd/2539298/ (accessed on 15 February 2022). Sensors 2023,23, 7822 19 of 19 40. DeMenthon, D.F.; Davis, L.S. Model-Based Object Pose in 25 Lines of Code. Int. J. Comput. Vis. 1998,15, 123–141. [CrossRef] 41. Mallick, S. Head Pose Estimation using OpenCV and Dlib [Learn OpenCV blog], 26. 2016. Available online: https://www.learno pencv.com/head-pose-estimation-using-opencv-and-dlib/ (accessed on 25 May 2018). 42. OpenCV. OpenCV Modules. 2022. Available online: https://docs.opencv.org/3.4.1/ (accessed on 10 January 2022). 43. Bradski, G.; Kaehler, A. Learning OpenCV, 1st ed.; O’Reilly Media, Inc.: Farnham, UK, 2008; ISBN 978-0-596-51613-0. 44. Umbaugh, S.E. Digital Image Processing and Analysis: Human and Computer Vision Applications with CVIP Tools, 2nd ed.; CRC Press Taylor & Francis Group: Boca Raton, FL, USA, 2010; ISBN 9780429196324. 45. Hartley, R. Multiple View Geometry in Computer Vision, 2nd ed.; Cambridge University Press: Cambridge, UK, 2004; ISBN 354049698X. 46. Charvát, M. System for People Detection and Localization Using Thermal Imaging Cameras. Master’s Thesis, Brno University of Technology, Brno, Czech Republic, 2020. 47. Mircea, M.P.; Nedevschi, S.; Danescu, R. Robust Data Association Using Fusion of Data-Driven and Engineered Features for Real-Time Pedestrian Tracking in Thermal Images. Sensors 2021,21, 8005. [CrossRef] 48. Wojke, N.; Bewley, A.; Paulus, D. Simple Online and Realtime Tracking with a Deep Association Metric. In Proceedings of the IEEE International Conference on Image Processing (ICIP), Beijing, China, 17–20 September 2017; pp. 3645–3649. [CrossRef] 49. Schramm, S.; Osterhold, P.; Schmoll, R.; Kroll, A. Combining modern 3D reconstruction and thermal imaging: Generation of large-scale 3D thermograms in real-time. Quant. InfraRed Thermogr. J. 2022,19, 295–311. [CrossRef] Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.