On Automatic Person-in-Water Detection for Marine Search and Rescue Operations

This is a self-archived version of an original article. This version may differ from the original in pagination and typographic details. Author(s): Title: Year: Version: Copyright: Rights: Rights url: Please cite the original version: CC BY 4.0 https://creativecommons.org/licenses/by/4.0/ On Automatic Person-in-Water Detection for Marine Search and Rescue Operations © 2024 The Authors Published version Taipalmaa, Jussi; Raitoharju, Jenni; Queralta, Jorge Peña; Westerlund, Tomi; Gabbouj, Moncef Taipalmaa, J., Raitoharju, J., Queralta, J. P., Westerlund, T., & Gabbouj, M. (2024). On Automatic Person-in-Water Detection for Marine Search and Rescue Operations. IEEE Access, 12, 5242852438. https://doi.org/10.1109/access.2024.3386640 2024 Received 19 January 2024, accepted 7 March 2024, date of publication 9 April 2024, date of current version 17 April 2024. Digital Object Identifier 10.1109/ACCESS.2024.3386640 On Automatic Person-in-Water Detection for Marine Search and Rescue Operations JUSSI TAIPALMAA 1, JENNI RAITOHARJU 2, (Senior Member, IEEE), JORGE PEÑA QUERALTA 3,4, (Member, IEEE), TOMI WESTERLUND3, (Senior Member, IEEE), AND MONCEF GABBOUJ 1, (Fellow, IEEE) 1Department of Computing Sciences, Tampere University, 33014 Tampere, Finland 2Faculty of Information Technology, University of Jyväskylä, 40014 Jyväskylä, Finland 3Turku Intelligent Embedded and Robotic Systems Group, University of Turku, 20014 Turku, Finland 4SCAI Laboratory, SPZ, Swiss Federal School of Technology in Zürich (ETH Zürich), 8092 Zürich, Switzerland Corresponding author: Jussi Taipalmaa ([email protected]) This work was supported by the Academy of Finland’s AutoSOS Project under Grant 328755. ABSTRACT In marine search and rescue missions, the objective is to find a missing person in the water. Time is a critical factor in the identification of the missing person, as any delay in locating them can have life-threatening consequences. Autonomous unmanned aerial vehicles (UAVs) possess the potential to help in the search task by providing a bird’s-eye view helping to cover larger areas faster. Therefore, it is very important that UAVs can efficiently and accurately detect persons in the water. This work studies automatic person detection in the water from a UAV. We performed experiments on both lakes and sea near Turku, Finland, and captured videos of people in the water from various altitudes and different viewing angles. Our person-in-water detection tests focus on important factors that have not received sufficient attention in prior studies: evaluation metrics and detection thresholds, the impact and use of different bounding box sizes, multi-frame detection and performance in unseen environments. We provide analysis of the suitability of different approaches for the person detection task and we also publish our training and testing data that includes over 72000 frames. To the best of our knowledge, this is the largest publicly available person-inwater detection dataset. INDEX TERMS Search and rescue (SAR), person-in-water, unmanned aerial vehicle (UAV), object detection, deep learning (DL), dataset. I. INTRODUCTION Marine search and rescue (SAR) missions consist of finding the emergency site on sea or lake, locating the target, and performing the rescue operation. SAR operations in the marine environment can be challenging due to the vast and often unpredictable nature of the sea or lake area. The operations are also highly time-critical because a person overboard can stay above the surface for only a limited time. The associate editor coordinating the review of this manuscript and approving it for publication was Yangmin Li . The actual time depends on different factors, such as weather or fitness, current condition or equipment of the target person, but ultimately hypothermia begins when the human body core temperature drops below 35◦C[1]. Depending on the water temperature and clothing (e.g., drysuit vs. ordinary clothes), the onset of hypothermia may differ from hours to only minutes [2]. Recent studies have shown that SAR operations can significantly benefit from supporting autonomous or teleoperated robots and multi-robot systems [3],[4],[5]. For marine SAR operations, unmanned aerial vehicles (UAVs) 52428 2024 The Authors. This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ VOLUME 12, 2024 J. Taipalmaa et al.: On Automatic Person-in-Water Detection for Marine SAR Operations can offer an efficient solution by covering the search area and helping to detect the target to be rescued. In the future, an efficient solution could be that a UAV or a swarm of UAVs autonomously perform area coverage above the target area and use computer vision to find the target; then, the rescue personnel can perform the actual rescue mission. During the last decade, the number of applications using computer vision algorithms and also the performance of the algorithms have advanced significantly. With the rapid development of deep learning networks for object detection tasks, the performance of object detectors has greatly improved [6]. This suggests that deep learning-based object detection can be used in SAR missions. While some object detection methods can already surpass human performance in accuracy [7], some solutions can be rather slow and computationally heavy and the performance measured in run-time can differ a lot. In SAR operations, it is vital that the selected algorithms can run as close to real-time as possible on portable devices while still working with a high level of accuracy. The faster the algorithm can operate, the faster the UAV can search the area, which in turn can lead to a faster rescue of the persons in distress. A high level of performance of the algorithm assures that no important information is missed. In this paper, we study different aspects of deep learning-based object detection in SAR missions. We focus on important factors that have not received sufficient attention in prior studies: (i) evaluation metrics and detection thresholds, (ii) the impact and use of different bounding box sizes, (iii) multi-frame detection, and (iv) performance in unseen environments. Based on our experiments we provide recommendations for further studies on the topic. For this purpose, we have collected, annotated, and processed a dataset of over 72000 frames which contains images of a person swimming or floating in a water body, taken from a UAV flying above the area. To the best of our knowledge, this is currently the largest dataset on this topic and we will make it publicly available. The rest of the paper is organized as follows. Section II briefly reviews related research. Section III describes the data acquisition process and the collected dataset. In Section IV, we introduce the study setup and the design factors considered in this study. Experimental results are presented in Section V. Finally, Section VI concludes the paper. II. RELATED WORK In recent years, a lot of research has been conducted towards enabling autonomous drones to assist in SAR operations as shown by recent reviews on related topics [3],[8],[9],[10], [11],[12]. In this paper, we focus on the perception needed in marine search and rescue operations, specifically when searching for a person in the water. The person-in-water detection in SAR missions can be performed either using object detection or semantic segmentation. A survey of available deep learning semantic segmentation techniques [13], provides an extensive view of different methods to perform the segmentation. For autonomous driving, semantic segmentation has been studied fairly well [14], but for marine environments, the studies have been less common and have been focusing on a view from a surface vessel. In [15], three commonly used state-of-the-art deep learning semantic segmentation methods (U-Net [16], PSP-Net [17], and DeepLabv2 [18]) were benchmarked for obstacle detection on a marine environment. The leaderboard for a publicly available dataset, Modd2 [19], which contains images of swimmers and rowers among other obstacles, lists semantic segmentation methods capable of performing in marine environments [17],[18],[19],[20],[21],[22],[23], [24]. In [25], a dataset for surface vessel water segmentation was introduced, and in [26], the same data was used to train a segmentation model for a UAV view with swimmers and other obstacles in the water. While excellent results can be obtained when the algorithm is applied in conditions and environments that resemble the training images, it was observed that the performance decreases notably in different conditions. This further highlights the need for diverse training images and domain adaption techniques that help to adjust to unseen conditions [27]. A drawback of semantic segmentation is that it requires pixel-level analysis of the images and may waste precious time for analysing areas of lower interest, such as ground locations. Object detection, on the other hand, only requires region-level image analysis and can lead to faster detection of the person in water. A survey of deep learning-based object detection [6] has been published quite recently. These tasks usually still require high computing power and memory for real-time applications. Therefore, cloud computing or small-sized object detection methods have been used for UAV applications. In [28], cloud computing was used for object detection while performing low-level object detection and navigation on a UAV. Cloud computing assists the system with high computing power and memory, but communication with a cloud server can bring unpredictable delays. An alternative to cloud computing is to rely on specific object detection models, specifically designed for limited computational power and memory. In [29], frame rates of 5-18fps were obtained on different lightweight embedded processing platforms when running a lightweight object detection model for vehicle detection. In [30], an approach for learning efficient deep object detectors for real-time UAV applications through channel pruning of convolutional layers was proposed in [30]. The approach was tested on different generic object classes. In [31], another lightweight object detection model was tested on a dataset including people and objects in water. In [32], the authors proposed an adaptive submodularity and deep learning-based spatial search method for detecting humans with a UAV. In [33], the authors performed human detection for SAR operations using images from different in-land environments. VOLUME 12, 2024 52429 J. Taipalmaa et al.: On Automatic Person-in-Water Detection for Marine SAR Operations Recently, also person-in-water detection has received more attention. SeaDroneSee dataset [34] includes 54,000 frames with humans in open water captured with drones from various altitudes and viewing angles ranging from 5 to 260 meters and 0 to 90 degrees while providing the respective meta information for altitude, viewing angle and other metadata. A Maritime Computer Vision Challenge [35] included tasks for UAV-based maritime object detection and tracking. An approach for obtaining fast region-of-interest proposals in a video stream on an embedded GPU for maritime human detection tasks was proposed in [36]. III. DATA DESCRIPTION We collected our dataset at four different water bodies in the Turku area, Southwest Finland, in summer and early autumn weather. Specifically, the data was collected between early June and mid-September 2020. The dataset consists of videos from different environments, sea and lakes, recorded with a drone flying over the area at various heights. All dataset images/frames contain a similar structure, an image of a water area from above, with at least one person swimming in the water. The camera angle is set to face straight downwards, i.e., the camera pitch is always -90◦. The only changing factor in the setup is the flight altitude of the drone. All people appearing in videos are part of the study group and their informed consent was obtained. The data collection was carried out with a small drone equipped with an RGB camera with a rolling shutter. The camera has a fixed focal length of 24mm (35mm format equivalent) with a field of view of 83◦and an aperture f/2.8. The still images are taken with a resolution of 9MP (4000 ×2250 pixels), while the videos in the datasets are recorded in FullHD resolution (1920 ×1080 pixels) at a rate of 60fps. From the videos, we extract each frame. The different locations, their environment type, the number of recorded videos, and the overall length of the videos are shown in Table 1. The dataset contains videos of different types of water bodies with various turbidity levels and swimmers with different swimwear. The weather varies from sunny to overcast, which affects the illumination,e.g., the color of the water, and different wind speeds affect to the texture of the water. The maximum height recorded is 120 m owing to the limitations imposed by EU-level drone regulations. TABLE 1. Dataset collection sites, environment types, number of video clips, and combined length of the videos collected at each site. The different subsets, listed in Table 1, contain the captured frames and locations of the bounding boxes of the swimmer. The annotations were made by using the Labelbox-annotation tool [37]. The tool allows the interpolation of the bounding boxes in frames that are between hand-annotated frames. The videos for training and testing were selected from Maaria, Masku and Mustfinn subsets. The Littoinen subset was left outside, so it could be used later for testing as a completely unseen and different environment. The training and testing splits were determined in a way that we use the frames of a single video clip either for training or testing. This way we can ensure that consecutive frames are not used for both training and testing, and we can also reduce the possibility of over-fitting. The splits are described in Table 2 and Table 3. TABLE 2. List of subsets, number of videos and images used for training. TABLE 3. List of subsets, number of videos and images used for testing. This gives us an overall training/testing split of 67/33. All the images contain one object that we are trying to detect, a swimmer in the water. There are also other objects such as rock, vegetation, beach equipment and birds in the water, but we are not trying to detect them and they have not been annotated. This setup makes this a one-class detection problem. We released the dataset: SAR-HumanDetection-FinlandProper, along with the partitions used for training and testing. To the best of our knowledge, our dataset is the largest publicly available dataset for person-in-water detection in terms of the number of frames. The dataset can be found in the following link: https://doi.org/10.23729/9b3fcb5d-9655-4c629762-7442040f7579 IV. STUDY DESIGN Our purpose is to study different design factors in deep learning-based UAV person-in-water detection in SAR missions. The considered factors include evaluation metrics and detection thresholds (Section IV-B), the impact and use of different bounding box sizes (Section IV-C1), multiframe detection (Section IV-D), and performance in unseen environments (Section IV-E). In this study, we do not compare different network models or try to find a perfect hyperparameter setup, but focus 52430 VOLUME 12, 2024 J. Taipalmaa et al.: On Automatic Person-in-Water Detection for Marine SAR Operations on the above-mentioned factors that are important for the task at hand, but typically receive less attention. Therefore, we selected a commonly-used baseline model and setup that is used in all the experiments as described in Section IV-A. A. BASELINE MODEL AND SETUP As our baseline model we selected the widely-used YOLOv4model [38] with MobileNetV3Small backbone. We used a general YOLOv4/v3/v2 object detection pipeline, implemented with keras and tensorflow.1YOLOv3 -loss function was used, and the Adam algorithm was used for optimization. All the experiments were performed using NVIDIA GeForce RTX 2080 GPU with 8GB own memory. The drone equipment we used allowed us to gather high-quality (1080 ×1920 pixels) videos. YOLO models require the input size to be a multiple of 32. Therefore, we first downsampled our original data from 1080 × 1920 pixels resolution initially to 1024 ×1920 pixels. We made experiments with this input size and input size downsampled by a factor of 2, but these models turned out to be computationally too heavy for our use case and equipment. As the problem is highly time-critical and the model is planned to be used in an edge device, we decided to compromise between the output frame rate and input size. Therefore, we decided to use an input size that corresponds to downsampling by a factor of 4, i.e., 256 ×480 pixels. With this reduced input size, it is still possible to detect objects in the water. But since a portion of the videos are captured from a relatively high altitude, up to 120m, the loss of detail during downsampling is quite significant as can be noticed in Figure 1. This makes detection very difficult from high altitudes and other methods for solving this problem need to be considered as will be further discussed in Section IV-C. FIGURE 1. Images from low and high altitude, before (upper row) and after (lower row) downsampling. The low resolution images have been scaled back to the same size for easier comparison, and all the images have been cropped for better representation. The training was done in the batches of 8 images and the training/validation split was 90/10. The Model was pre-trained with Imagenet [39] and transfer learning was then 1github.com/david8862/keras-YOLOv3-model-set used to train the model with our train dataset described in Table 2. The output layers were first trained for 10 epochs, after which the backbone was unfrozen and the model continued to train for a maximum of 240 more epochs. The learning rate of 0.001 was used. For most experiments, testing was performed using the test set described in Table 3. Only when experimenting with unseen environments, the Littoinen subset was used instead. B. EVALUATION CRITERIA AND DETECTION THRESHOLDS Object detection algorithms output bounding boxes with different confidence scores. A detection is considered true positive (TP) when the confidence is high enough and the detected bounding box coordinates are close enough to the ground-truth bounding box coordinates. To this end, thresholds for the confidence level and for intersection over union (IoU) between the detected and ground-truth bounding boxes are typically used. If a detected bounding box does not correspond to any ground-truth bounding box, it is a false positive (FP) detection. A ground truth bounding box that is not detected at all is a false negative (FN). Different object detection evaluation metrics can be computed based on the numbers of TPs, FPs, and FNs. For given IoU and confidence thresholds, precision (P) and recall (R) are defined as P=TP TP +FP,R=TP TP +FN .(1) Too strict confidence thresholds lead to a situation where only a few bounding boxes are detected. If the detections are correct, this can yield perfect precision, while recall is low due to many FNs. Lower thresholds lead to many detected bounding boxes. The number of FNs becomes low, but also the number of FPs typically increases, which yields high recall values but low precision. Object detection algorithms need to balance between optimizing precision and recall by selecting suitable thresholds. Object detection algorithms are most commonly evaluated using different variants of Average Precision (AP) metric [40],[41]. The most common variant selects an IoU threshold (typically 0.5) and then computes precision and recall values for detections arranged using their confidence values. Average Precision (AP) with the all-point interpolation [41] is computed as mAP =X n (Rn+1−Rn)Pinterp(Rn+1) Pinterp(Rn+1)=max ˆ R:ˆ R≥Rn+1 P(ˆ R),(2) where Pinterp(Rn+1) denotes the maximum precision for which the corresponding recall value is greater than Rn+1, and thus the metric is computed for a single IoU threshold but all possible confidence thresholds. Another commonly used variant, AP@50:5:95, computes the average over 10 different IoU metrics ranging from 0.5 to 0.95. Computing the results over multiple thresholds measures the ability of VOLUME 12, 2024 52431 J. Taipalmaa et al.: On Automatic Person-in-Water Detection for Marine SAR Operations the algorithms to yield good results for a range of user requirements instead of studying which approach is optimal for specific requirements. In SAR missions, the most critical errors occur if a person in the water is completely missed. Therefore, the number of FNs should be as close to zero as possible. False detections (FPs) can be relatively easily checked by the rescue personnel assuming that the number of FPs is not overwhelmingly large. Therefore, compared to many other object detection tasks,the importance of minimizing the number of FNs is emphasized, while FPs are less critical. Here, it should be noted that the roles of IoU and confidence thresholds are significantly different. The IoU threshold does not impact the detected bounding boxes, only whether they are considered TPs or FPs. If a detected bounding box partially overlaps with a ground truth bounding box is considered FP, and the corresponding object is considered undetected (FN). Thus, a smaller IoU threshold increases the number of TPs and typically also decreases the number of FNs. Nevertheless, the actual detection output (bounding boxes) do not change. The main question is which labels (TP/FP) for the detected bounding boxes lead to the most meaningful performance evaluation for the task at hand. In person-in-water detection, bounding boxes are typically small compared to the overall image. Furthermore, minor inaccuracy in the bounding box coordinates is not a problem as the people are expected to float around anyway and the main goal is to help the rescue personnel to find the person in distress. Therefore, we argue that smaller IoU thresholds can lead to more meaningful evaluations for person-in-water detection. The impact of the confidence threshold can be much more dramatic. The threshold defines, which bounding boxes are given as the detection output. Tightening the confidence threshold can change a TP detection to an FN (or erase an FP detection). As mentioned above, in person-in-water detection tasks the FN can be deadly, while FPs can be handled to some extent. Therefore, we believe that focusing on low confidence thresholds is reasonable. While existing approaches in object detection for SAR use AP or AP@50:5:95 metrics [34],[35] for performance evaluation, we argue that this may not be the most meaningful choice. Instead of averaging over different confidence and IoU thresholds, we are more interested in thresholds minimizing the number of FNs. In this article, we will study the impact of these thresholds (along with other factors) in person-in-water detection. As an additional metric in our studies, we use F1-score which considers both precision and recall for single IoU and confidence thresholds as F1=2·P·R P+R.(3) C. OBJECT SIZE A critical factor in the design of a SAR mission is the flying altitude (distance to the target), which directly affects the perceived size of the person in the water. By flying too high, the risk of not detecting the person in the water is increased, while lower altitudes mean that more critical time is needed to cover the search area. Naturally, the optimal altitude is affected by the camera equipment, resolution, environment, weather, clothing and other possible equipment of the person in the water. As described above, the models’ input size may require downscaling, which further affects the detection probabilities. No generic rules can be given, but study the importance of the perceived target size in our use case in different ways as described in Sections IV-C1-IV-C3. 1) BOUNDING BOX SIZE Our data does not contain information on the distance to the object to be detected. However, the target bounding box sizes serve as a rough proxy. We compare detection rates for bounding box sizes 1-25, 24-50, 51-100, and 100+pixels. Furthermore, we study the impact of using larger (zoomedin) bounding box sizes in training and testing as described below. 2) ZOOMING-BASED DATA AUGMENTATION IN TRAINING PHASE Even though a lot of information is lost because of the need for downsampling from 1080 ×1920 pixels to 256 × 480 pixels, it is still possible to obtain more accurate images of the swimmers for training the model. To this end, we do the following: First, the original image is converted into the size of 1024×1920 pixels. After the conversion, we use a 4×4 grid of 256 ×480 pixels to obtain images that have the same size as the input of the model (Figure 2). From this grid, we select the ones containing an object for training. This allows us to use the portion of the original high-quality images with the model’s input size (later called zoomed images) in addition to the downsampled images for training the model. Without using any additional data augmentation or pre-processing, such as flipping images or adding noise to images, we will study how including larger bounding box sizes affects the final performance. FIGURE 2. Image split into smaller regions during the training phase. 3) GRID ZOOMING IN TEST PHASE In the test phase, the images can also be separated into smaller images using a 4 ×4 grid (Figure 3), similarly as in the data augmentation phase. The original image is split into 16 subimages and all the sub-images are fed into the network and 52432 VOLUME 12, 2024 J. Taipalmaa et al.: On Automatic Person-in-Water Detection for Marine SAR Operations then combined back to the original size. This allows us to use higher-quality images in testing, but the disadvantage is that the network needs to process 16 times the amount of images compared to performing the detection with downsampled images. FIGURE 3. Illustration of the original image, and a smaller image obtained with grid-zooming. In the post-processing phase, the smaller 256 ×480 pixel images need to be stitched back together into the original size (1024 ×1920 pixels). Since the detection can be at the edge of two or more smaller images, there can be multiple detections of a single object. This could be avoided by using more small images, some of which are overlapping, but since the problem is time-critical, adding more input regions would further slow down the overall process. Instead, we merge the multiple detections of a single object using Algorithm 1. Algorithm 1 Algorithm for Merging Detections Made With Grid-Zooming Method: Two Bounding Boxes are Considered Overlapping if they are Within the dist of Each Other. Overlapping Bounding Boxes are Merged. Algorithm Continues Until there are No Overlapping Bounding Boxes for all found bounding boxes: Data: bbox(xmin,ymin,xmax,ymax) B1←bbox1; B2←bbox2; dist ←10 pixels; if overlap(B1,B2, dist) then Bnew.xmin ←min(B1.xmin,B2.xmin); Bnew.ymin ←min(B1.ymin,B2.ymin); Bnew.xmax ←max(B1.xmax,B2.xmax ); Bnew.ymax ←max(B1.ymax,B2.ymax ); save Bnew elsesave B1; save B2; end D. MULTI-FRAME DETECTION In SAR tasks, the goal is to find the missing person in the water. It does not matter if the person is detected in every frame where he/she appears or only in sufficiently many frames to alert the rescue personnel. Furthermore, as the drones are taking videos, it is reasonable to assume that despite the movement, any person in water would appear in several frames. We will study a multi-frame detection scheme, where windows of Nframes are used. We consider that we have a TP detection if a person-in-water appearing in the window is detected at least in 1/2 of the frames. We hope that this approach can remove FPs appearing in single frames, while not causing additional FNs. Using the multi-frame scenario we will also study if the frame rate can be lowered without significantly harming the results to make the computations faster. E. UNSEEN ENVIRONMENT While all the test images used in our experiments are unseen and selected from different video clips than the training data, the test set used in most experiments still comes from the same water bodies. We will study how the results are affected when the test frames are from a completely unseen environment. V. EXPERIMENTAL RESULTS A. EVALUATION METRICS AND DETECTION THRESHOLDS -RESULTS We first studied the impacts of IoU and confidence thresholds on person-in-water detection. We trained our baseline approach described in Section IV-A and evaluated the model with different IoU-thresholds (0.10, 0.25, 0.50, 0.75) using the confidence threshold 0.1. The AP results are computed using the all-point interpolation over all confidence thresholds. Thus, they are not directly based on the given numbers of TPs, FPs and FNs. The results are given in Table 4. A couple of examples of detection results are shown in Figure 4. Similarly, we evaluated the model with different confidence thresholds (0.1, 0.3, 0.5, 0.7, 0.9) using the IoU threshold 0.1. The results are given in Table 5. Here, we do not give AP values as they are computed over all confidences, not separately for different confidence thresholds. As expected, in Table 4, the numbers of TPs, FPs, and FNs are all best for the lowest threshold and, thus, also all the other evaluation metrics show the best results for this threshold. The IoU examples in Figure 4show that it is meaningful to consider the detected bounding boxes as TPs because despite the low IoU they can lead rescue personnel to find the person-in-water. The confidence threshold has a larger impact on the number of FNs as shown in Table 5. The difference between confidence thresholds 0.1 and 0.9 is over 3-fold. This can be a critical life-saving difference in a SAR task. Furthermore, it can be seen that the precision for the lowest confidence threshold remains relatively high at 0.8880 showing that the number of FPs does not explode too high. VOLUME 12, 2024 52433 J. Taipalmaa et al.: On Automatic Person-in-Water Detection for Marine SAR Operations FIGURE 4. Three images from different environments. The green bounding box is the ground truth and the blue bounding box is the detection. All the images have been cropped for better representation. In all images 0.1≤IoU <0.25. TABLE 4. Impact of IoU threshold. TABLE 5. Impact of confidence threshold. As a conclusion, these experiments validate our expectations from Section IV-B. Evaluating the person-in-water detection performance in SAR tasks using low IoU thresholds is justified. Due to the high variation of FNs for different confidence thresholds, it is important to focus on the lowconfidence results. Using the commonly used AP metric overall confidence can potentially lead to choosing a model that performs better on high confidence, but fails to optimize the most important result with the lowest thresholds. Due to these observations, we will use the IoU and confidence threshold of 0.1 in the following experiments and report only evaluation metrics that are computed for these thresholds leaving out the AP considerations. B. SIZE OF THE OBJECT Next, we study the impact of the bounding box sizes in different ways. We first study the different combinations of zooming-based data augmentation in the training phase (Section IV-C2) and the grid-zooming-based evaluation in the test phase (Section IV-C3). The results are given in Table 6. The non-augmented training set and the original test set denote unmodified data that has been only scaled to the input size of 256 ×480 pixels. The augmented training set denotes training data that has been augmented by using the 4 ×4 grid-zoomed images in training in addition to the original downsampled image. The grid-zoom test data denotes that the images have gone through the grid-zoom procedure during the test phase. As we can see from Table 6, using augmented training data gives better results than the original training data for both test set types. The data augmentation does not affect the model’s inference speed and is therefore clearly a recommendable approach for person-in-water detection tasks. The described approach will be used in the remaining experiments of this paper. It should be noted that there is potential for further improvement using more varied downscaling factors, 52434 VOLUME 12, 2024 J. Taipalmaa et al.: On Automatic Person-in-Water Detection for Marine SAR Operations TABLE 6. Impact of zooming-based data augmentation and grid-zooming in test phase. TABLE 7. Impact of object size (bounding box area). TABLE 8. Using multiple consecutive frames in detection. TABLE 9. Impact of dropping frame rate for 300 frame window. but we leave further studies on this topic for future work. Using grid-zooming does not give desirable results when using non-augmented training data. This can be explained by the fact that such an arrangement requires the model to detect objects that are clearly larger than the training objects. Grid-zooming with the augmented training data gives the best results among all options. In particular, the number of FNs is significantly lower than for the other options. Nevertheless, it should remembered that using the grid-zooming option makes the inference 16 times slower, which may not be doable on the edge device. Therefore, studying other ways to improve the inference speed is important. Next, we study the direct impact of the target bounding box size, which can be interpreted as a proxy for the distance to the target. The higher the altitude, the smaller the target bounding box. The results categorized by the bounding box size are given in Table 7. Unsurprisingly, the results suggest that detection is more difficult from the higher altitudes. The model that does not use grid-zooming performs better at the VOLUME 12, 2024 52435