SimForest: RGBD Instance Segmentation Dataset

SimForest: RGBD Instance Segmentation Dataset Ramana Reddy Avula Dependable Transport Systems RISE Research Institutes of Sweden Bor˚ as, Sweden ramana.reddy[email protected] 0000-0001-9672-2689 Aleksi Narkilahti FrostBit Software Lab Lapland University of Applied Sciences Rovaniemi, Finland [email protected] 0009-0008-3937-1139 Krzysztof Wołk DAC.digital, SA Al.Grunwaldzka 472 80-309 Gda´ nsk, Poland [email protected] 0000-0001-5030-334X Abstract—Autonomous perception in forest environments requires accurate detection and segmentation of complex natural objects such as trees, rocks, and terrain features. However, the scarcity of large-scale, annotated forest datasets, especially those with depth and instance segmentation labels, hinders progress in deploying robust deep learning models for forestry applications. In this paper, we present SimForest, a 4K-resolution synthetic RGBD dataset generated using a photorealistic forestry simulator built on Unreal Engine 5. SimForest comprises 5,000 images, each annotated with aligned RGB data, depth maps, instance segmentation masks, and detailed metadata including object poses, terrain depth, camera parameters, and environmental conditions such as season, time, and cloudiness. The virtual scenes are geo-located and seasonally matched to a real forest near Ume˚ a, Sweden. To demonstrate the utility of SimForest, we conduct an experimental study involving the detection and segmentation of tree trunks using YOLOv11-based models trained on SimForest data. The evaluation shows strong detection accuracy (mAP@50 of 0.92) and solid segmentation performance (mAP@50 of 0.74). These findings highlight the potential of SimForest as a valuable resource for near-field RGBD perception in forestry and related outdoor robotics applications. Index Terms—Instance segmentation, Synthetic dataset, Forest perception, RGBD dataset, Tree trunk detection I. INTRODUCTION Autonomous perception in forest environments is a critical capability for a wide range of applications, including autonomous navigation, precision forestry, ecological monitoring, and disaster response. These tasks require the accurate detection, localization, and understanding of complex natural objects, such as trees, rocks, and terrain features, in unstructured outdoor scenes. Training effective deep learning models for these applications requires access to large-scale, high-quality annotated datasets. Unlike urban or indoor environments, where datasets such as COCO [1] and ImageNet [2] have driven significant advancements in object detection and segmentation, forest environments lack comparable resources. Existing real-world forest datasets [3] are typically limited in resolution, lack aligned depth information, or fail to capture the seasonal and environmental variability that characterizes natural forest ecosystems. Collecting large-scale real-world datasets in forest environments presents substantial challenges. Field campaigns are expensive, time-consuming, and often constrained by weather conditions, accessibility, and safety concerns. Obtaining precise ground truth annotations for 3D object poses, spatial relationships, and depth information requires specialized equipment and expertise that may not be readily available. Moreover, the manual annotation of forest imagery is particularly challenging due to the irregular shapes of natural objects, occlusions caused by dense vegetation, and the difficulty in precisely delineating boundaries between overlapping foliage and branches, often requiring several minutes per image and introducing unavoidable errors in boundary and occlusion handling [4]. Addressing these challenges, synthetic data generation has emerged as a promising solution. Simulation environments can generate unlimited amounts of data with perfect ground truth annotations, including precise object poses, depth maps, and instance segmentation masks. Additionally, synthetic datasets can be generated rapidly and cost-effectively, allowing for the exploration of different scenarios and edge cases that might be rare or hazardous to capture in real-world settings. In this paper, we introduce SimForest1, a comprehensive synthetic RGBD dataset specifically designed for forest perception tasks. SimForest leverages a high-fidelity forestry simulator [5] built on Unreal Engine 5 to generate high-resolution images captured across all four seasons under varying lighting conditions, cloudiness levels, and camera viewpoints. Each image is accompanied by an aligned depth map, instance segmentation masks, and detailed scene metadata including objectlevel 3D transforms, terrain depth map, camera intrinsics, camera pose, and environmental parameters such as season, time of day, and cloudiness. Instance segmentation annotations are provided for all visible objects within a 15-meter radius from the camera, focusing on near-field perception tasks such as obstacle avoidance and selective logging. The virtual scenes are geo-located and rendered to match the appearance and structure of a real-world forest near Ume˚ a, Sweden. A recent related work, SPREAD [6], presents a large-scale synthetic forest dataset built also based on Unreal Engine 5, but covering multiple forest biomes and providing RGB, depth, point clouds, segmentation labels, and tree-level metadata such as trunk and canopy diameter and height. However, unlike SPREAD’s extensive yet generic synthetic dataset, SimForest is geo-located to a specific real forest and includes fine-grained near-field RGBD segmentation data and rich terrain and envi1https://doi.org/10.5281/zenodo.15911876 ronmental metadata. Furthermore, the SPREAD dataset itself provides only RGB images at 960×540 resolution, although the provided scripts support rendering at 4K-resolution. In contrast, SimForest offers full 4K-resolution RGBD frames, making it a plug-and-play resource for high-fidelity perception tasks. To evaluate the usefulness of SimForest, we conduct an experimental study focused on the detection of tree logs suitable for harvesting, a key task in precision forestry. We train and benchmark YOLOv11 [7] based object detection and instance segmentation models exclusively on the SimForest dataset. The results provide empirical validation of the dataset’s quality and its suitability for developing and benchmarking RGBD perception models in challenging forestry and other outdoor robotic domains. In summary, the key contributions of this paper are as follows: •We introduce SimForest, a novel synthetic RGBD dataset for forest perception, comprising 5,000 high-resolution images with aligned depth maps, instance segmentation masks, and extensive metadata. •We benchmark SimForest on a practical tree trunk harvesting task using YOLO11x-based object detection and instance segmentation models. •We release SimForest as an open-source benchmarking resource, complete with its rich metadata, to support research in RGBD perception for forestry and outdoor robotics. II. SIMFOREST DATASET A. Dataset Overview The SimForest dataset comprises a total of 5,000 annotated frames, each containing aligned RGB images, depth maps, and instance segmentation masks. These frames are distributed across a diverse set of environmental conditions, covering all four seasons, varying times of day, cloudiness levels, and intraseasonal changes. Each data sample includes the following components: •RGB images: High-resolution JPEG images rendered with photorealistic lighting and textures. •Scene depth maps: 32-bit RGB-encoded PNG images where each pixel encodes a depth value in meters using three color channels. The depth Dcan be decoded using: D=R+ 256 ·G+ 2562 ·B 2563−1·1000 where R,G, and Bare the red, green, and blue channel values, respectively. •Instance annotations: Instance segmentation masks in COCO format for each visible object within 15 meters. In addition, the dataset includes the following metadata to enable detailed scene understanding: •Camera intrinsics and pose: Includes the camera intrinsics and full 6-DoF camera pose (position and orientation) relative to the world coordinate frame. •Instance metadata: For each annotated instance, metadata includes the instance segmentation ID, category ID, spatial location, orientation, and physical size in 3D space. •Environmental conditions: The simulated environment is characterized with season, hour of day, month, and a cloudiness score (0 to 1), enabling controlled experiments involving lighting and weather variability. •Terrain depth map: A separate depth map is included for each frame, providing pixel-wise terrain depth values that are useful for elevation-aware perception. The SimForest dataset is publicly available on Zenodo [8] under the CC BY 4.0 license. Figure 1 illustrates representative samples from the dataset, showing aligned RGB images, depth maps, and instance segmentation masks under varied environmental conditions. The dataset contains a total of 40,554 annotations across 11 distinct categories, with pine trees and their trunks comprising the majority of annotations (64%), as shown in Figure 2. B. Data Generation Process The SimForest dataset was created using a high-fidelity forestry simulation environment developed as part of the AGRARSENSE project [9]. Built on Unreal Engine 5, the simulator utilizes real geospatial data to replicate the structural and visual characteristics of boreal forests accurately. The virtual scenes are based on a terrain map near Ume˚ a in northern Sweden, with vegetation assets representing key Nordic species, including birch, pine, and spruce trees. The environment supports photorealistic rendering of seasonal variation, dynamic lighting, and atmospheric conditions. Data generation involved capturing aligned images from three virtual cameras: RGB, depth, and instance segmentation. The cameras were configured with a resolution of 3840×2160 pixels and a 90° horizontal field of view. A 200m × 200m georeferenced forest area was selected as the simulation region, and this area was divided into a regular grid to guide the sampling of camera positions. At each sampled grid cell, a camera pose was defined at a fixed height of 2 meters above the terrain surface, emulating the perspective of a low-flying drone navigating through the forest. The yaw angle of the camera was randomly selected from 16 discrete bins spaced uniformly over 360◦. This approach ensured broad coverage of the scene with diverse viewpoints while maintaining structured sampling density across the terrain. To further enhance diversity and realism, each image was rendered with randomized environmental parameters such as season (winter, spring, summer, autumn), time of day, month, and cloudiness (ranging from clear to overcast). Sun position and lighting conditions were simulated based on the real-world solar angles corresponding to the geo-location of the forest in Ume˚ a, Sweden, and selected time/month. This diversity enables systematic evaluation of perception models under a wide range of conditions, including challenging scenarios such as low sunlight and solar glare. The captured instance segmentation images were postprocessed to extract masks for objects whose nearest point RGB Scene depth Annotations Terrain depth Fig. 1: Samples from the SimForest dataset showing aligned RGB images, scene depth maps, instance segmentation masks, and terrain depth maps under diverse environmental conditions. Each column represents a different data sample. 0 2,000 4,000 6,000 8,000 10,000 12,000 14,000 Number of Annotations Terrain Foliage Birch Pine Spruce Sky Rock Snow Birch_Trunk Pine_Trunk Spruce_Trunk Category 3,711 1,565 793 14,012 879 5,000 202 1,632 594 11,816 350 Fig. 2: Distribution of annotations across categories in the SimForest dataset. to the camera falls within a 15-meter radius. This distancebased filtering ensures that the annotations remain relevant for near-field perception tasks, such as obstacle avoidance and selective logging. For each retained instance, a 2D bounding box was derived from the segmentation mask, while a 3D bounding box was estimated using the corresponding pixelwise depth map. A quality control procedure was applied to filter out frames with excessive occlusion or poor visibility, ensuring the dataset only includes scenes with meaningful and usable annotations. The final annotations, including object categories, segmentation masks, and bounding boxes, were saved in the COCO format, ensuring compatibility with widely used training and evaluation pipelines. III. EXPERIMENTAL EVALUATION We benchmark the SimForest dataset using state-of-the-art object detection and instance segmentation models to assess its suitability for perception tasks in forest environments. Specifically, we focus on the detection of tree trunks suitable for precision harvesting. For this purpose, we employ the YOLOv11x and YOLOv11x-seg models for object detection and instance segmentation, respectively. All training and evaluation data were drawn exclusively from the SimForest dataset, which spans all four seasons and a diverse range of environmental conditions. This controlled setup isolates the impact of synthetic data quality on model performance, providing a baseline for future sim-to-real transfer studies. For this study, we derived a dedicated Tree Trunk dataset from SimForest containing 3,086 images and 11,872 annotated tree trunks, retaining only those with a minimum diameter of 10 cm and a minimum height of 2 m. The data were split randomly in an 80–20 ratio, resulting in 2,468 images with 9,567 annotations for training and 618 images with 2,305 annotations for validation. Training was initialized from the default YOLOv11x and YOLOv11x-seg weights, leveraging transfer learning from pretrained 640×640 pixel models. The models were trained at an image size of 2,560×2,560 pixels to fit efficiently within the 24 GB VRAM of an NVIDIA RTX 4090 GPU. Standard YOLO augmentations such as random scaling, horizontal flipping, translation, and color jitter were retained, while more complex augmentations including mosaic, mixup, cutmix, and copy-paste were disabled. Training was configured for up to 300 epochs with early stopping enabled with a patience of 100, which triggered at epoch 277 for object detection, while instance segmentation completed all 300 epochs. The training (a) Object detection (YOLOv11x) training loss (b) Instance segmentation (YOLOv11x-seg) training loss Fig. 3: Training loss curves for object detection and instance segmentation models, showing stable convergence. process showed a consistent decrease in training losses over epochs for both object detection and instance segmentation, indicating stable convergence. The individual loss components for object detection (box, classification, and distribution focal losses) and for instance segmentation (including an additional segmentation loss) are illustrated in Fig. 3a and Fig. 3b. To maintain consistency with the dataset’s annotation policy, only objects within 15 meters of the camera were considered during post-processing, excluding predicted instances beyond this range. While this is less precise for bounding boxes, it provides a reasonable approximation of the annotation constraints. All files required for creating the derived dataset in YOLO format, as well as for training and validation of the models, are available in a supplementary GitHub repository2 As shown in Table I, the object detection model achieved strong performance, with a precision of 0.86 and an mAP@50 of 0.92. Although performance decreased under the stricter mAP@50–95 metric to 0.75, the results remain robust, highlighting the dataset’s suitability for near-field detection tasks. Instance segmentation performance was slightly lower, with 2https://github.com/RISE-Dependable-Transport-Systems/ SimForest-YOLO-Toolkit TABLE I: Performance of YOLOv11x and YOLOv11x-seg models for tree trunk detection and segmentation. Task Precision Recall mAP@50 mAP@50–95 Object detection 0.8616 0.8685 0.9208 0.7466 Instance segmentation 0.6854 0.7514 0.7379 0.5753 a precision of 0.69, an mAP@50 of 0.74, and an mAP@50– 95 of 0.58, reflecting the increased challenge of fine-grained pixel-level localization in dense forest scenes. These results demonstrate the value of SimForest as a high-quality benchmark for RGBD perception in unstructured outdoor environments, particularly for object detection. IV. CONCLUSION This paper introduces SimForest, a high-resolution synthetic RGBD dataset developed for near-field forest perception tasks such as object detection and instance segmentation. Built using a high-fidelity Unreal Engine 5-based simulator and geolocated to a real forest in northern Sweden, SimForest provides 5,000 images with aligned RGB, depth, and instance segmentation data, along with rich environmental and geometric metadata. The experimental results demonstrate strong detection performance (mAP@50 of 0.92) and solid segmentation results (mAP@50 of 0.74), despite the complexity of natural forest environments. While performance decreases under the stricter mAP@50–95 metric (dropping to 0.75 for detection and 0.58 for segmentation), this is expected given the finegrained localization and mask precision it requires. The dataset’s photorealistic seasonal rendering make it particularly well-suited for perception systems operating in boreal forests, while its synthetic nature enables scalable data generation without the logistical challenges of real-world collection. By releasing SimForest as an open-access resource, we aim to support research in forestry robotics, ecological monitoring, and other outdoor applications where high-fidelity perception is essential. Its 4K-resolution and rich multimodal annotations offer a valuable foundation for advancing RGBD perception in natural environments, complementing existing datasets that lack depth information or high-resolution imagery. In future work, we will explore how SimForest can support transfer learning and domain adaptation in forestry applications. A key objective is to benchmark sim-to-real generalization by testing models trained on synthetic data against realworld forest imagery. This evaluation will provide actionable insights into narrowing the simulation-to-reality gap and enhancing the practical effectiveness of synthetic training for real-world forestry perception systems. ACKNOWLEDGMENT This work was carried out within AGRARSENSE project (Grant Agreement No. 101095835), supported by the Chips JU and its members, including top-up funding from Sweden, Czechia, Finland, Ireland, Italy, Latvia, Netherlands, Norway, Spain, and the National Centre for Research and Development of Poland. REFERENCES [1] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll´ ar, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in European conf. on computer vision, Springer, 2014, pp. 740–755. [2] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” in CVPR09, 2009. [3] J. Lagos, U. Lempi¨ o, and E. Rahtu, “Finnwoodlands dataset,” in Scandinavian Conference on Image Analysis, Springer, 2023, pp. 95–110. [4] Y. Lu, Y. Huang, S. Sun, S. Fei, and V. Chen, “Putree: A photorealistic large-scale virtual benchmark for forest training,” in 2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), 2024, pp. 687–688. DOI: 10.1109/VRW62533.2024.00140. [5] FrostBit Software Lab (Lapland UAS), Agrarsense simulator, Accessed: 2025-07-21. [Online]. Available: https://dev.azure.com/AMKFrostBit/ AGRARSENSE. [6] Z. Feng, Y. She, and S. Keshav, “Spread: A large-scale, high-fidelity synthetic dataset for multiple forest vision tasks,” Ecological Informatics, vol. 87, p. 103 085, 2025, ISSN: 1574-9541. DOI: https://doi.org/ 10.1016/j.ecoinf.2025.103085. [7] G. Jocher and J. Qiu, Ultralytics YOLO11, version 11.0.0, 2024. [Online]. Available: https://github.com/ultralytics/ultralytics. [8] R. R. Avula and A. Narkilahti, Simforest: Rgbd instance segmentation dataset, Zenodo, Jul. 2025. DOI: 10.5281/zenodo.15911876. [9] CORDIS, AGRARSENSE - Smart, digitalized components and systems for data-based Agriculture and Forestry. [Online]. Available: https:// cordis.europa.eu/project/id/101095835.