RangeSAM: Promptable Segmentation of Sparse 3D Point Clouds in Outdoor Driving Scenes

RangeSAM: Promptable Segmentation of Sparse 3D Point Clouds in Outdoor Driving Scenes by Arya Araban (1439683) Supervised by: Kourosh Khoshelham, Cipher Zhang A thesis submitted in partial fulfillment of the Master of Computer Science (MC-CS) degree in the School of Computing and Information Systems The University of Melbourne October 2024 Abstract Point cloud segmentation stands as a critical component in 3D scene understanding, with particular significance in autonomous driving applications. While semantic segmentation has seen substantial progress, class-agnostic segmentation remains challenging due to its inherent need for adaptability across diverse object types. This challenge has sparked interest in promptable segmentation approaches, which offer promising solutions for handling arbitrary objects through guided inputs. However, existing 3D promptable segmentation methods primarily target dense point clouds, overlooking the critical challenge of sparsity prevalent in real-world LiDAR data, especially in outdoor driving scenes. We introduce RangeSAM, a novel architecture for promptable point cloud segmentation specifically designed to address this limitation. Our approach seamlessly integrates range-based processing techniques with principles from state-of-the-art 2D promptable segmentation models, enabling effective handling of sparse and irregular LiDAR data while maintaining high segmentation fidelity. RangeSAM incorporates a unique point prompt mechanism with and leverages transformer-based components to capture longrange dependencies, allowing for intuitive user guidance and robust segmentation in complex scenes. Through extensive evaluation on multiple datasets, including KITTI360, SemanticKITTI, and ApolloScape, we demonstrate that RangeSAM achieves competitive performance against existing promptable segmentation methods while effectively handling sparse point clouds. Our model achieves a score of 56.05% IoU with 20 point prompts on KITTI360, with experiments conducted confirming the effective utilization of prompt information. Notably, RangeSAM maintains strong performance across datasets unseen during training, achieving over 46% IoU on ApolloScape despite significant domain differences, and demonstrates competitive results against non-promptable approaches on SemanticKITTI with a competetive F1 score (74.4) and segmentation association metric (50.4). These results, combined with comprehensive ablation studies validating our distinctive prompt-centric design, establish RangeSAM as a promising solution for real-world applications requiring adaptive, user-guided segmentation of sparse point cloud data. Declaration of Authorship I, Arya Araban, declare that this thesis and the work presented in it are my own. I confirm that: ■This thesis does not incorporate without acknowledgement any material previously submitted for degree or diploma in any university; and that to the best of my knowledge and belief it does not contain any material previously published or written by another person where due reference is not made in the text. ■Clearance for this research from the University’s Ethics Committee was not required. ■The thesis is 24,704 words in length (excluding text in images, tables, bibliographies, and appendices). Signed: Arya Araban Date: October 28, 2024 ii Acknowledgements I would like to express my heartfelt gratitude to everyone who provided guidance, encouragement, and support throughout the journey of completing this thesis. Firstly, I wish to extend my appreciation to my supervisors, A/Prof. Kourosh Khoshelham and PhD candidate Cipher Zhang, for their invaluable insights and feedback which contributed greatly to the development of this work. I extend my sincere thanks to PhD candidate Zexian Huang, whose perspective was instrumental in helping me gain confidence and clarity in my project direction at crucial stages. I am also immensely thankful to my friends and family, whose unwavering support and encouragement during challenging times enabled me to persevere and remain focused. Each of these contributions has played a meaningful role in bringing this thesis to fruition, and for that, I am truly grateful. iii Contents Abstract i Declaration of Authorship ii Acknowledgements iii List of Figures vi List of Tables vii 1 Introduction 1 1.1 Background ................................... 1 1.2 Problem Statement ............................... 3 1.3 Research Objectives and Scope ........................ 4 1.4 Thesis Outline ................................. 6 2 Literature Review 8 2.1 Introduction to Point Clouds ......................... 8 2.1.1 Definition and Characteristics ..................... 8 2.1.2 Acquisition Methods .......................... 9 2.1.2.1 LiDAR Technology ..................... 9 2.1.2.2 Other Acquisition Techniques ................ 11 2.2 3D Semantic Segmentation .......................... 13 2.2.1 Image-Based Segmentation ...................... 14 2.2.2 Voxel-Based Segmentation ....................... 15 2.2.3 Point-Based Segmentation ....................... 16 2.2.4 Range-Based Segmentation ...................... 19 2.3 3D Class-Agnostic Segmentation ....................... 21 2.3.1 Evolution from Semantic to Class-Agnostic Segmentation ..... 23 2.3.2 Promptable Segmentation ....................... 24 2.3.2.1 Segment Anything Model (SAM) and its Utilization in 3D 25 2.3.2.2 Native 3D Promptable Architectures ............ 28 3 Methodology 31 3.1 Data Collection and Processing ........................ 31 3.2 RangeSAM ................................... 35 iv v 3.2.1 Instance Selection and Range Projection ............... 36 3.2.2 Image Encoder: Stem and ViT Encoder ............... 39 3.2.3 Sampling and Encoding Prompts ................... 41 3.2.4 Bi-Directional Transformer Integrated Decoder ........... 44 3.2.5 KNN-Based 3D Refinement ...................... 47 3.3 Implementation Details ............................ 49 3.3.1 Training Setup ............................. 50 3.3.2 Evaluation Metrics ........................... 53 3.3.3 Hyperparameter Tuning ........................ 55 4 Performance Evaluation 59 4.1 Evaluation Preliminaries ............................ 59 4.2 Quantitative Analysis ............................. 62 4.2.1 Prompt Influence ............................ 62 4.2.2 Cross-dataset Generalization ..................... 65 4.2.3 Comparative Prompt-Driven Segmentation ............. 67 4.2.4 Class-Agnostic Performance Comparison ............... 69 4.3 Qualitative Analysis .............................. 71 4.4 Additional Experiments ............................ 76 4.4.1 Dataset Influence on Generalization ................. 76 4.4.2 Ablation Study of Novel Architecture Components ......... 77 5 Conclusion 80 5.1 Overview .................................... 80 5.2 Contribution .................................. 81 5.3 Future Work .................................. 82 Bibliography 84 List of Figures 2.1 Comparison of Dense and Sparse Point Clouds ............... 9 2.2 LiDAR Principle of Operation ......................... 10 2.3 Semantic Segmentation of an Urban Point Cloud Scene .......... 13 2.4 General Structure of Point-Based Networks ................. 18 2.5 Architecture of RangeVIT ........................... 20 2.6 Segment Anything Model Architecture .................... 25 2.7 Overview of Point-SAM Architecture ..................... 29 3.1 SemanticKITTI Scan With Object Instances Color-Coded ......... 33 3.2 Preprocessed KITTI-360 Scan With Instance Highlight ........... 34 3.3 RangeSAM Architecture ............................ 36 3.4 Range Image With Instance Highlight and Crop Boundaries ........ 39 3.5 Random Prompt Sampling on Range Image ................. 43 3.6 Reprojection of 2D Prompts onto the 3D Point Cloud ........... 43 3.7 RangeSAM Decoder Architecture ....................... 44 3.8 KNN Algorithm Illustration .......................... 48 3.9 IoU Values Across Tuning Trials ....................... 56 3.10 Hyperparameter Relationships of Top 10 Performing Trials ........ 57 3.11 KNN Refiner Hyperparameter IoU Score Heatmap ............. 57 4.1 IoU Performance Across Varying Prompt Counts on KITTI360 ...... 63 4.2 Cross-Dataset IoU@K Comparison ...................... 66 4.3 3D Promptable Models IoU@K Comparison ................. 68 4.4 Object Instance Segmentation Example in SemanticKITTI ........ 72 4.5 Prediction Visualization of a Complete KITTI360 Scan ........... 73 4.6 Prediction Visualization of a Complete ApolloScape Scan ......... 74 4.7 Prompt Distribution Containing Background and Foreground Artifacts . . 75 vi List of Tables 3.1 Dataset Comparison - SemanticKITTI, KITTI-360, and ApolloScape . . . 32 4.1 Effect of Prompt Label Inversion ....................... 64 4.2 Class-Agnostic Model Performance on SemanticKITTI ........... 70 4.3 Training Source Impact on Model Generalization .............. 76 4.4 Object Category Performance by Training Source on ApolloScape ..... 77 4.5 RangeSAM Component Ablation Results .................. 78 4.6 Cross-Dataset Performance of Ablation Variants .............. 79 vii Chapter 1 Introduction 1.1 Background The advent of three-dimensional (3D) sensing technologies has revolutionized spatial data acquisition and analysis, with point clouds emerging as a powerful representation for capturing intricate geometric structures of real-world environments. Point clouds, composed of sets of points in three-dimensional space, offer a direct and detailed depiction of 3D geometry, preserving crucial depth information and enabling precise localization and mapping of objects and surfaces [1]. This rich representation has found widespread applications across various domains, with autonomous driving standing out as one of the most prominent examples of its impact [2]. Acquiring point cloud data has become increasingly accessible and prevalent due to the widespread adoption of Light Detection and Ranging (LiDAR) sensors, stereo cameras, and other depth-sensing devices. LiDAR, in particular, has become a cornerstone technology in autonomous driving systems, providing high-resolution 3D scans of the vehicle’s surroundings [3]. These scans capture the spatial relationships between various objects in the environment, forming a crucial input for perception, localization, and decision-making algorithms. While point clouds contain detailed spatial information, they do not inherently hold semantic meaning. To extract actionable insights, it is necessary to segment the point cloud into meaningful regions or objects. This process, known as point cloud segmentation, is a fundamental task in 3D computer vision and plays a pivotal role in scene understanding. Segmentation enables the identification and delineation of various objects within the point cloud, facilitating downstream tasks such as object detection, classification, and tracking [4]. 1 Chapter 2 Literature Review 2.1 Introduction to Point Clouds Understanding the intricacies of point cloud data is crucial for developing effective segmentation methods, particularly in the context of outdoor environments. This section explores point cloud characteristics and acquisition techniques, providing essential context for the subsequent discussion on segmentation approaches. 2.1.1 Definition and Characteristics Point clouds are a fundamental representation of 3D data, consisting of a set of points in three-dimensional space. Each point is typically defined by its spatial coordinates (X, Y, Z) and can include additional attributes such as color (R, G, B), intensity, or surface normal information [14]. This versatile and efficient way to represent complex 3D structures and environments captures intricate geometric details that might be lost in other 3D representations. Their unstructured nature allows for a more nuanced representation of spatial relationships and surface details, making them particularly valuable in applications ranging from autonomous driving and robotics to architecture and archaeology. [15]. While point clouds excel at capturing geometric information, they also present unique challenges for processing and analysis. Unlike the grid-like structure of 2D images, point clouds are inherently unordered and irregular. This characteristic, while allowing for flexible representation, complicates the application of traditional computer vision techniques and necessitates specialized algorithms for tasks such as segmentation and object detection [16]. 8 Chapter 2 - Literature Review 9 Another important characteristic of point clouds, particularly relevant to outdoor environments, is that they may have varying point density across different regions. Point clouds can range from dense representations with high spatial resolution to sparse representations where points are more dispersed, with the density often decreasing at greater distances from the sensing device. This uneven distribution of points poses significant challenges for segmentation algorithms and is a key focus of current research. Figure 2.1 illustrates the contrast between dense and sparse point cloud representations. Figure 2.1: Comparison of Dense and Sparse Point Clouds [17,18] In this figure, the left image visualizes an indoor scene with uniformly high point density throughout the captured space, enabling precise geometric detail of objects and surfaces. In contrast, the right image demonstrates an outdoor scene where the point density varies considerably - objects and surfaces at increasing distances exhibit progressively lower spatial resolution, presenting fundamental challenges for consistent object recognition and segmentation across the scene. 2.1.2 Acquisition Methods Point clouds can be acquired through various methods, each with its own strengths and limitations. The choice of acquisition method often depends on the specific application requirements, such as accuracy, scale, and speed of data collection. 2.1.2.1 LiDAR Technology Light Detection and Ranging (LiDAR) has become one of the most prominent technologies for point cloud acquisition, especially in applications like autonomous driving and urban mapping [19]. LiDAR systems emit laser pulses and measure the time it Chapter 2 - Literature Review 10 takes for the light to return after reflecting off surfaces in the environment. This timeof-flight (ToF) measurement is then employed to construct a precise 3D point cloud representation of the environment. To expand on this, a typical LiDAR system consists of three main components: 1. A laser emitter that sends out rapid pulses of light 2. A receiver that detects the reflected pulses 3. A precise timing system to measure the time between emission and reception The distance to each point in the environment is calculated using the formula: Distance = Speed of Light ×Time of Flight 2(2.1) By rapidly scanning in multiple directions, a LiDAR system can build up detailed point clouds of its surroundings. Modern LiDAR systems can emit hundreds of thousands of pulses per second, allowing for the rapid acquisition of dense point clouds. Figure 2.2 illustrates the basic principle of LiDAR operation. Figure 2.2: LiDAR Principle of Operation [20] LiDAR systems can be broadly categorized into three types based on their deployment: •Terrestrial LiDAR: Stationary systems typically used for high-resolution scanning of buildings, infrastructure, or archaeological sites. These systems offer high accuracy but are limited in their coverage area [21]. Chapter 2 - Literature Review 11 •Airborne LiDAR: Mounted on aircraft, these systems are used for large-scale mapping of terrain, forests, and urban areas. They are well-suited for topographic mapping and landscape modeling due to their elevated vantage point [22]. •Mobile LiDAR: Mounted on vehicles or handheld devices, these systems are particularly useful for mapping roads, urban environments, and indoor spaces. They offer the advantage of rapid data collection over large areas [23]. Each of these LiDAR types has its own characteristics in terms of point density, accuracy, and coverage area. Terrestrial LiDAR typically provides the highest point density but is limited in coverage area, while airborne LiDAR can cover vast areas but with lower point density. Mobile LiDAR provides an effective balance between point density and coverage area, ensuring that detailed data is captured across a wide geographical range. One of the key advantages of LiDAR technology is its ability to penetrate vegetation to some extent, allowing for the mapping of ground surfaces even in forested areas. This capability, combined with its high accuracy and rapid data acquisition, has made LiDAR a preferred choice for many 3D mapping applications [24]. However, LiDAR also has limitations. The technology can be affected by atmospheric conditions such as rain or fog, which can scatter or absorb the laser pulses. Additionally, highly reflective or absorbent surfaces can pose challenges for accurate distance measurement [25]. 2.1.2.2 Other Acquisition Techniques While LiDAR is a dominant technology for point cloud acquisition, especially in outdoor environments, several other techniques are worth noting: •Photogrammetry: this technique reconstructs 3D geometry from multiple 2D images by relying on the principle of triangulation, where the 3D position of a point can be determined by analyzing its projection in multiple images taken from different viewpoints [26]. Modern photogrammetry techniques, such as Structure from Motion (SfM), can automatically detect and match features across hundreds or thousands of images to reconstruct detailed 3D point clouds [27]. While photogrammetry can produce dense point clouds with accurate color information, it typically struggles with uniform or reflective surfaces and can be computationally intensive for large datasets. Chapter 2 - Literature Review 12 •Structured Light Scanning: This technique projects a known pattern of light onto an object and analyzes the deformation of this pattern to reconstruct the 3D shape. Structured light scanners can produce high-resolution point clouds and are often used for small to medium-sized objects in controlled environments [28]. •Time-of-Flight (ToF) Cameras: These devices emit infrared light pulses and measure the time taken for the light to return to the sensor for each pixel. While similar in principle to LiDAR, ToF cameras capture depth information for an entire scene simultaneously, resulting in a depth image that can be converted to a point cloud. ToF cameras are often used in indoor environments and for applications requiring real-time 3D sensing [29]. •Stereo Vision: This technique mimics human binocular vision by using two cameras separated by a known distance. By analyzing the disparities between the two images, depth information can be calculated and used to generate a point cloud. Stereo vision systems are often used in robotics and autonomous vehicles, complementing other sensing modalities [30]. Each of these methods has its strengths and limitations in terms of accuracy, range, speed, and cost. The choice of acquisition method often depends on the specific requirements of the application, the scale of the environment, and the desired level of detail. Photogrammetry excels in capturing detailed color information and can cover large areas, but struggles with uniform or reflective surfaces. Structured light scanning offers high resolution for small objects but is limited to controlled environments where lighting and movement can be managed effectively. ToF cameras provide real-time depth sensing but have limited range and resolution compared to LiDAR. Finally, Stereo vision is passive and can work in various lighting conditions, but its accuracy decreases with distance. Understanding the characteristics of these acquisition techniques is valuable for developing robust segmentation algorithms that can potentially generalize across different types of point cloud data. In the context of this research, which focuses on outdoor environments and particularly driving scenes, LiDAR remains the primary technology of interest due to its ability to rapidly acquire accurate, long-range 3D data in dynamic environments. Chapter 2 - Literature Review 13 2.2 3D Semantic Segmentation While our research endeavors to address the challenge of promptable class-agnostic segmentation of arbitrary objects in sparse 3D point clouds, it is important to understand the substantial and relevant body of work dedicated to semantic segmentation, which involves assigning class labels to individual points or groups of points within a 3D point cloud. While our task differs from semantic segmentation in that we do not rely on predefined object classes, understanding the foundations and advancements of this segmentation approach is essential, as it provides valuable insights and techniques that inform our approach. Figure 2.3 illustrates an example of semantic segmentation applied to a point cloud of an urban scene. Different colors represent distinct object classes, such as vehicles, roads, and vegetation. Figure 2.3: Semantic Segmentation of an Urban Point Cloud Scene [31] The approaches to 3D semantic segmentation can be broadly categorized into three main paradigms: image-based, voxel-based, and point-based methods [32]. Additionally, range-based methods have emerged as a hybrid approach [33], combining elements of image-based and point-based techniques. Each of these paradigms addresses the challenges of processing unstructured point cloud data in different ways. Understanding the strengths and limitations of each is crucial for developing effective segmentation techniques. Chapter 2 - Literature Review 14 2.2.1 Image-Based Segmentation Image-based methods address the challenge of unstructured point cloud data by projecting three-dimensional information onto two-dimensional image planes. This approach facilitates the application of established two-dimensional convolutional neural network (CNN) architectures, thereby leveraging the substantial advancements achieved in 2D image processing. One of the initial works in this domain is Multi-View CNN (MVCNN) [34]. MVCNN projects the point cloud from multiple viewpoints, generating a set of 2D images. These images are then processed through a shared CNN to extract features, which are subsequently pooled to create a global descriptor. This approach effectively captures the 3D structure of objects through multiple 2D representations. Building upon MVCNN, MVCNN-MultiRes [35] extends the multi-view concept by incorporating multi-resolution analysis. This approach uses sphere rendering of 3D shapes at different volume resolutions, alongside standard rendering. Features are extracted from these multi-resolution renderings and concatenated. The combined features are then used to train a classifier, allowing the model to leverage both detailed local information and broader contextual cues. Another notable image-based method is SnapNet [36] which introduced a more sophisticated approach for semantic segmentation. SnapNet uses an MVCNN-like architecture to project the point cloud onto multiple virtual camera views, generating both depth and RGB images. These images are then processed by separate CNN streams, and the resulting features are fused for semantic segmentation. This multi-modal approach allows SnapNet to leverage both geometric and color information for improved segmentation accuracy. While image-based methods benefit from the rich feature representations learned by 2D CNNs and the extensive research in 2D computer vision, they face several challenges: 1. Information loss: The projection from 3D to 2D inevitably leads to some loss of spatial information, particularly in complex scenes with occlusions. 2. View dependency: The segmentation results can be sensitive to the chosen projection views, potentially leading to inconsistencies across different viewpoints. 3. Empty Regions: In scenarios with sparse point clouds, common in long-range LiDAR scans, the projected images may contain significant empty regions, making feature extraction challenging. Chapter 2 - Literature Review 15 Despite these limitations, image-based methods remain popular due to their computational efficiency and the ability to leverage pre-trained 2D CNN models. They are particularly effective in scenarios where the point cloud can be consistently projected onto a 2D plane. 2.2.2 Voxel-Based Segmentation Voxel-based methods represent the 3D point cloud as a structured 3D grid of volumetric elements (voxels). Each voxel encodes information about the points falling within its volume, such as occupancy or point density. This representation allows the direct application of convolutional neural networks, enabling their success in the 3D domain. One of the early works in voxel-based segmentation is VoxNet [37], which introduced a 3D CNN architecture for object detection and classification using voxelized point cloud data. VoxNet demonstrates the effectiveness of 3D convolutions in capturing spatial relationships within point clouds. However, it also highlighted the computational challenges associated with processing large 3D grids, especially for high-resolution or large-scale point clouds. To address the computational limitations of dense 3D grids, OctNet [38] proposed an innovative octree-based encoding scheme. OctNet uses a hierarchical data structure that efficiently represents sparse 3D data by allocating computational resources to non-empty voxels while bypassing empty regions. This adaptive partitioning scheme allows OctNet to capture features at multiple scales, enhancing its ability to segment complex scenes. SEGCloud [39] is another technique which further advanced voxel-based methods. It introduces an end-to-end trainable pipeline that combines voxelization, 3D CNN processing, and a novel trilinear interpolation step to map CNN outputs back to the original point cloud. This approach addresses the challenge of information loss during voxelization and ensures fine-grained segmentation results. While voxel-based methods offer several advantages, including the ability to capture spatial context and the direct application of 3D CNNs, they also face significant challenges: 1. Quantization artifacts: The voxelization process can introduce quantization artifacts, particularly at voxel boundaries, potentially affecting segmentation accuracy. 2. Computational complexity: Processing dense 3D grids can be computationally expensive, especially for large-scale or high-resolution point clouds. Chapter 2 - Literature Review 16 3. Memory requirements: Voxel representations, particularly at high resolutions, can have substantial memory requirements, limiting their applicability to large scenes. Despite these challenges, voxel-based methods continue to be an active area of research, with ongoing efforts to develop more efficient and accurate architectures for processing 3D volumetric data. 2.2.3 Point-Based Segmentation Point-based methods aim to process point clouds directly in their native, unstructured format. These approaches avoid the need for intermediate representations like 2D projections or voxel grids, preserving the inherent spatial relationships within the point cloud data. Point-based methods can be further categorized into three main subcategories: Multi-Layer Perceptron (MLP) based methods, Graph Convolutional Network (GCN) based methods, and Transformer-based methods. •Multi-Layer Perceptron (MLP) based Methods: MLP-based methods use multi-layer perceptron networks to process individual points or local neighborhoods within the point cloud. These methods aim to learn point-wise features that can be aggregated to perform segmentation tasks. The seminal work in this category is PointNet [40], which revolutionized point cloud processing by directly operating on raw point cloud data. PointNet uses a series of MLPs to extract features for each point independently, followed by an operation which aggregates these features into a global feature vector. This architecture ensures permutation invariance, meaning the order in which points are processed does not affect the final result. Building on the success of PointNet, PointNet++ [41] introduced a hierarchical feature learning approach. PointNet++ addresses the limitation of PointNet in capturing local structures by applying PointNet recursively on nested partitions of the input point set. This allows the network to capture both fine-grained patterns and global context, leading to improved segmentation performance. RandLA-Net [42] further advanced MLP-based methods by introducing an efficient random sampling strategy coupled with local feature aggregation. This approach enables the processing of large-scale point clouds with millions of points, addressing the scalability issues faced by previous methods. •Graph Convolutional Network (GCN) based Methods: GCN-based methods represent the point cloud as a graph, where each point is treated as a node, Chapter 2 - Literature Review 17 and the relationships between points are encoded as edges. These methods apply graph convolutional operations to aggregate and propagate features across the graph, capturing both local and global contextual information. A notable example of GCN-based methods is the Dynamic Graph CNN (DGCNN) [43]. DGCNN introduces the concept of EdgeConv, a novel convolution-like operation on graphs. Unlike fixed graph convolutions, DGCNN dynamically computes the graph at each layer of the network, allowing it to capture semantic as well as geometric neighborhoods. This dynamic graph updating mechanism enables DGCNN to adapt to varying structural patterns within the point cloud and learn more discriminative features. Another significant contribution in this category is the Hierarchical Point-Edge Interaction Network (HPEIN) [44]. HPEIN employs a hierarchical graph convolutional network structure to process point clouds by progressively constructing a graph from coarse to fine layers. The network integrates point and edge features across multiple scales, effectively capturing geometric relationships throughout the scene. This approach incorporates point-edge interactions and edge feature supervision, leading to improved segmentation performance on multiple benchmarks. •Transformer-Based Methods: Transformer-based methods, inspired by their success in natural language processing and 2D computer vision, have gained traction in point cloud segmentation due to utilizing attention mechanisms. These mechanisms enable models to dynamically focus on relevant parts of the input data, weighing the importance of different points or features. Self-attention is a key component of transformers which allows each point to interact with all other points, capturing global relationships and long-range dependencies, which is crucial for understanding overall structure and context in point cloud segmentation. Point Transformer [45] is a pioneering work in this category, adapting the transformer architecture to point cloud data. It introduces a vector self-attention layer that adaptively modulates feature channels by computing attention weights based on point-wise geometric relationships. The multi-head attention mechanism enables Point Transformer to effectively process geometric features at varying scales, leading to robust segmentation performance. Point Cloud Transformer (PCT) [46] further refines the transformer-based approach for point clouds. PCT introduces a novel offset-attention module that explicitly models geometric relationships between points. This module, combined with a neighbor embedding layer, allows PCT to better capture detailed structural Chapter 2 - Literature Review 24 or removal of points, the Region Transformer is guided by learned attention weights, allowing for more flexible and adaptive segmentation. This approach bridges the gap between traditional region growing methods and modern transformer architectures, capturing complex spatial relationships within the point cloud and leading to more accurate and flexible segmentation results. These methods mark key steps in 3D class-agnostic segmentation. SGPN introduced similarity-based grouping, LRGNet developed adaptive region growing, 3DUIS pioneered unsupervised graph-based segmentation, and Region Transformer applied attention mechanisms. Each approach tackled different aspects of the challenge, advancing the field’s capabilities and pushing towards more robust and versatile class-agnostic segmentation methods. Despite these advances, these approaches still face limitations in terms of generalization to truly arbitrary objects and the ability to handle guided interactions, which are particularly critical in dynamic environments. This led to increased interest in promptable segmentation methods, which aim to provide more flexible and interactive ways of specifying segmentation targets. 2.3.2 Promptable Segmentation Promptable segmentation represents a significant advancement in the field of computer vision, offering a more flexible and interactive approach to image and point cloud segmentation. This method allows users to guide the segmentation process through various forms of input prompts, such as points, bounding boxes, or even natural language descriptions [8]. The ability to interactively specify objects of interest makes promptable segmentation particularly valuable in scenarios where the target objects may not be known in advance or where fine-grained control over the segmentation process is desired. The key motivation behind promptable segmentation is to create more versatile and userfriendly segmentation systems while reducing annotation burden. Traditional semantic segmentation models are limited to predefined object categories and struggle with novel or ambiguous objects. In contrast, promptable segmentation models can adapt to user intent on-the-fly, segmenting arbitrary objects based on minimal input. This approach often allows for training with weaker supervision, potentially reducing the need for exhaustive per-point labels in training data. It is important to note that while user input is a key feature of promptable segmentation, these models can also be designed to operate autonomously. By simulating user prompts Chapter 2 - Literature Review 25 or leveraging other contextual information, promptable models can function seamlessly without direct human guidance. This capability integrates promptable segmentation into automated systems, combining flexibility and efficiency. Promptable segmentation works by encoding user-provided or simulated prompts into a format that can be processed alongside the input image or point cloud. The model then uses this information to guide its attention towards the relevant regions or objects. This approach allows for a more dynamic and interactive segmentation process, allowing for refining results in real-time by providing additional prompts. The success of promptable segmentation in 2D image processing, has sparked significant interest in extending these capabilities to 3D point cloud data. However, this extension presents unique challenges due to the unstructured nature of point clouds and the increased complexity of 3D spatial relationships. Despite these challenges, the potential benefits of promptable segmentation in 3D are substantial, particularly for applications in robotics, autonomous driving systems, and virtual reality. 2.3.2.1 Segment Anything Model (SAM) and its Utilization in 3D The Segment Anything Model (SAM), introduced by Meta AI Research [8], represents a significant breakthrough in the field of promptable, class-agnostic segmentation. While SAM was primarily designed for 2D image segmentation, with it leveraging 2D transformer-based architectures and self-attention mechanisms, its innovative approach and impressive performance have inspired numerous adaptations and extensions to 3D point cloud segmentation. To fully grasp SAM’s potential for point cloud segmentation, it is essential to understand its inner workings and how it is being utilized for this purpose. Figure 2.6 illustrates an overview of the model’s architecture. Figure 2.6: Segment Anything Model Architecture The architecture of SAM consists of three main components: the image encoder, prompt encoder, and mask decoder. The image encoder, utilizes a Vision Transformer (ViT) to Chapter 2 - Literature Review 26 process the entire input image to create a rich, high-dimensional feature representation. This process, while computationally intensive, allows for a comprehensive understanding of the image content that can be flexibly queried by various prompts. The prompt encoder transforms user prompts (e.g. points and bounding boxes) into vector embeddings, enabling the model to comprehend and identify the designated object of interest. For instance, a point is represented as the sum of a positional encoding of its location and a learned embedding indicating whether it belongs to the foreground (object of interest) or background. This process is carried out by passing the prompt through a small neural network that maps the prompt elements to their corresponding vector representations. Finally, The mask decoder is a module that efficiently combines the image embedding from the encoder and the prompt embeddings to produce an output mask. It takes inspiration from Transformer segmentation models [53], featuring a modified Transformer decoder with cross-attention mechanisms. The decoder updates both the image embedding and prompt tokens through cross-attention, allowing for effective integration of the two sources of information. After upscaling the updated image embedding, the output token is used to dynamically predict the final mask through a small multi-layer perceptron. SAM has been trained on a large and diverse dataset of 11 million images and 1.1 billion masks, generated using its own data engine, which enables it to demonstrate strong zero-shot performance on various segmentation tasks, accurately segmenting objects in new images it has never seen before. It is important to note that user prompts, such as points and boxes, are not part of the dataset, and are dynamically created based on ground truth instance masks while training the model. To elaborate further, during training, SAM simulates an interactive setup by sampling prompts in multiple rounds per mask, allowing it to learn from increasingly refined inputs that mimic user interactions. For inference, users can input their own prompts to segment specific objects of interest. However, if the goal is to segment everything in the image, SAM overlays a grid of point prompts on the image to attempt to segment all existing objects. The success of SAM in 2D image segmentation has naturally led researchers to explore its potential applications in 3D point cloud segmentation. Several innovative approaches have emerged, each attempting to leverage SAM’s capabilities in the 3D domain while addressing the unique challenges of point cloud data. One notable approach, SAM3D [54], extends SAM to 3D point cloud segmentation. In the context of a 3D scene, complete with a collection of RGB images and corresponding Chapter 2 - Literature Review 27 pose data, SAM3D leverages the base SAM model to generate segmentation masks for each individual RGB image. These 2D masks are then projected onto the 3D point cloud, resulting in a series of 3D masks. In the final step, these 3D masks are merged together using a bottom-up approach, iteratively combining the masks to produce a comprehensive, final mask for the entire point cloud. SAMPro3D [55] presents another approach which also leverages the base SAM model for 3D point cloud segmentation without further training. It initializes 3D prompts within the input scene and projects them onto 2D frames to generate pixel prompts for SAM. This is followed by employing a 2D-guided prompt filtering mechanism to retain high-quality prompts based on their performance across all frames. A prompt consolidation strategy further refines the results by merging prompts likely segmenting the same object. Finally, all input points are projected onto the segmented frames, and their mask predictions are aggregated to produce the final 3D segmentation. Segment3D [56] takes a different approach from the previous two by incorporating SAM into its training pipeline rather than using it directly for inference. It employs a two-stage training approach to generate class-agnostic 3D segmentation masks without manual annotations. Initially, Segment3D pre-trains a 3D segmentation model on partial RGBD point clouds, using SAM-generated 2D masks as pseudo ground-truth labels. Then, the pre-trained model is fine-tuned on complete 3D point clouds in a self-supervised manner, using its own high-confidence mask predictions as the training signal. This approach avoids the merging process and directly infers segmentation masks for entire 3D scenes using a native 3D model. These adaptations of SAM to 3D point cloud segmentation demonstrate the model’s potential impact beyond its original 2D domain. However, they also highlight some of the challenges in directly utilizing pre-trained 2D methods for 3D data. These challenges include handling occlusions, ensuring consistency across multiple views, capturing the full 3D structure of objects, and bridging the gap between 2D image-based training and the unique properties of 3D point clouds. Additionally, These methods typically require more input than just the point clouds, such as RGB images or camera pose data, which can be a significant limitation in scenarios where only point cloud data is available. These limitations motivate the development of 3D promptable segmentation methods that directly operate on point cloud data while maintaining the flexibility and generalizability demonstrated by SAM in the 2D domain. Chapter 2 - Literature Review 28 2.3.2.2 Native 3D Promptable Architectures Recognizing the limitations of adapting 2D models such as SAM to 3D data, researchers have begun developing native 3D promptable segmentation methods. These approaches aim to combine the flexibility and generalization capabilities of promptable models with architectures specifically designed for point cloud data. Generally, these methods use points as prompts, as points are the natural atomic unit of 3D point cloud data and provide a simple, intuitive way for users to indicate regions of interest by clicking in the 3D space. InterObject3D [13] is a 3D promptable segmentation approach that refines results through direct interaction with 3D point clouds. It encodes user clicks as 3D volumes, representing positive and negative feedback as additional input channels. This enables seamless integration with existing 3D segmentation architectures, such as convolutional networks based on the Minkowski Engine [57]. InterObject3D generalizes well to new environments and object classes not seen during training, making it suitable for annotating new datasets or adapting to novel scenarios. In InterObject3D, click simulation is utilized for training and evaluation, mimicking realistic user behavior. During inference, the model targets the largest error region, ensuring reproducible evaluation. Additionally, InterObject3D features online adaptation, treating user corrections as training examples to update the model during testing, further improving generalization to unseen classes and datasets. Agile3D [10] takes a different approach to 3D promptable segmentation by focusing on efficient and adaptive processing of point cloud data. Unlike previous methods that segment objects sequentially, Agile3D can segment multiple objects simultaneously within a 3D scene. The model employs a click-as-query module that converts user clicks into spatial-temporal query vectors, encoding both the spatial location and temporal order of user interactions. This is combined with a click attention module that enables explicit interactions between click queries and the 3D scene features. A key innovation of Agile3D is its ability to pre-compute backbone features for a scene, allowing for faster inference during interactive segmentation. The model only needs to run a lightweight decoder for each iteration of user input, significantly reducing computation time compared to methods that require a full forward pass for each interaction. Agile3D also introduces an iterative training strategy that more closely mimics real user behavior, improving the model’s performance in practical scenarios. PointSAM [9] is another 3D promptable segmentation method that draws significant inspiration from the SAM architecture, with it being specifically tailored for 3D point Chapter 2 - Literature Review 29 clouds. Unlike methods that rely on projecting 2D SAM outputs to 3D, PointSAM operates directly on point cloud data. Similar to SAM, its architecture consists of three main components, as illustrated in Figure 2.7. Figure 2.7: Overview of Point-SAM Architecture In PointSAM, The Point Cloud Encoder first samples a fixed number of centers using Farthest Point Sampling (FPS) and groups k-nearest neighbors into patches. These patches are processed through a PointNet [40] and a pre-trained transformer from Uni3D [58] to generate the point-cloud embedding. The Prompt Encoder handles both point and mask prompts, encoding them into consistent representations. The Mask Decoder, a key component, uses transformer decoder blocks with prompt self-attention and crossattention in both directions between prompts and point-cloud embeddings. It then upsamples the point-cloud embedding and applies a dynamic linear classifier to generate the final segmentation mask. PointSAM can handle variable input sizes and produce multiple output masks, addressing many limitations of 2D projection methods. This approach allows for more accurate and consistent 3D segmentation results, including the ability to segment internal structures of 3D objects. While these native 3D promptable segmentation methods represent significant advancements in the field, they generally focus on dense point clouds and often use datasets that feature relatively complete object representations. This focus on dense data is understandable given the prevalence of such datasets in indoor scenarios and many computer Chapter 2 - Literature Review 30 vision benchmarks. However, in real-world applications, particularly in outdoor driving scenes, point clouds are often sparse due to the nature of LiDAR sensor data acquisition. This sparsity poses unique challenges that are not fully addressed by current approaches. The varying point density, especially at long ranges, and the potential for occlusions in LiDAR scans create additional complexities for segmentation tasks. As we transition to our methodology, we propose a novel approach that specifically targets these challenges associated with sparse outdoor LiDAR point clouds. By adapting promptable segmentation techniques to work effectively with sparse data, we aim to bridge a critical gap in the field and provide a more robust solution for real-world autonomous driving applications. Chapter 3 Methodology Our research introduces a novel approach to promptable segmentation for sparse outdoor LiDAR point clouds, addressing key challenges and building on insights identified in the literature review. At the core of our methodology is RangeSAM, a framework that combines the efficiency of range-based processing with the flexibility of promptable segmentation, specifically designed to address the unique challenges posed by sparse data in autonomous driving scenarios. RangeSAM leverages insights from the RangeViT architecture while introducing novel promptable segmentation capabilities inspired by SAM for interactive and class-agnostic segmentation. Through a unique adaptation of these concepts to the 3D domain and specialized handling of sparse LiDAR data, RangeSAM addresses a critical gap in 3D point cloud segmentation. In this chapter, we detail our methodological approach, beginning with how the data used in RangeSAM is gathered and processed, followed by a comprehensive description of RangeSAM’s architecture and functionality. We will then discuss the implementation details, including our training setup, the metrics utilized for evaluation, and the hyperparameter tuning process. 3.1 Data Collection and Processing The development of a promptable, class-agnostic segmentation model for 3D point clouds necessitates the use of outdoor LiDAR datasets with instance-level annotations. These instance labels are crucial for training the model, as they allow for the creation of point cloud object-instance pairs where the object of interest is designated as the ”foreground” and the rest of the scene as the ”background”. Additionally, point prompts, categorized 31 Chapter 3 - Methodology 32 as either foreground points or background points, serve as an essential input to guide the segmentation process. The specifics of how these prompts are utilized and encoded within the RangeSAM model will be elaborated upon in subsequent sections. To facilitate this research, we conducted a comprehensive analysis of available outdoor LiDAR datasets, focusing on those that provide instance-level annotations. Table 3.1 presents a comparison of three prominent datasets in this domain: SemanticKITTI [18], KITTI-360 [31], and ApolloScape[59]. Dataset SemanticKITTI KITTI-360 ApolloScape Location Karlsruhe, Germany Karlsruhe, Germany Multiple cities, China Total number of scans 43,552 80,000 10,995 Provides raw scan labels Yes No Yes Number of labeled scans 20,351 - 5,594 Number of semantic classes 28 45 6 Provides instance labels Yes Yes Yes Number of instance classes 8 21 6 Total driving distance (km) 39.2 73.7 - 360°coverage Yes Yes Yes Temporal consistency Yes Yes No Table 3.1: Dataset Comparison - SemanticKITTI, KITTI-360, and ApolloScape This table offers a comparative overview of key characteristics for each dataset, including the total number of scans, availability of raw scan labels, number of semantic and instance classes, and other relevant features. While all three datasets provide valuable information, they each possess unique attributes that influence their suitability for our specific research objectives. SemanticKITTI, derived from the original KITTI dataset [3], captures semi-urban environments in Karlsruhe, Germany. It offers a substantial number of labeled scans (20,351) and provides instance labels for 8 classes. The dataset’s strength lies in its temporal consistency and raw scan labeling, making it particularly useful for tasks requiring frame-to-frame coherence. Figure 3.1 illustrates a typical SemanticKITTI scan, where different colors represent distinct object instances. This visualization underscores the dataset’s capability to differentiate between individual objects within the same class, a crucial feature for instance-level segmentation tasks. ApolloScape, collected across multiple cities in China, offers a diverse range of urban environments. While it provides fewer labeled scans (5,594) compared to SemanticKITTI, it still offers instance labels for 6 classes. The dataset’s strength lies in its geographical diversity, potentially enhancing the robustness of models across different urban layouts. Chapter 3 - Methodology 33 Figure 3.1: SemanticKITTI Scan With Object Instances Color-Coded KITTI-360, which can be considered a successor to the original KITTI dataset, stands out for its comprehensive and detailed annotations. Captured in suburban environments of Karlsruhe, Germany, it provides the largest number of scans (80,000) among the three datasets. KITTI-360 offers instance labels for an impressive 21 classes, significantly more than SemanticKITTI or ApolloScape. This extensive class coverage includes detailed instance annotations for common objects found in urban datasets, such as cars and pedestrians, while also incorporating labels for less frequently represented classes such as poles, traffic lights, and buildings. This diversity makes it particularly suitable for fine-grained segmentation tasks in complex urban scenes. However, KITTI-360 presents a unique challenge: it creates dense scenes from raw LiDAR scans rather than providing labels for individual raw frames. The dataset aggregates multiple LiDAR scans to create denser point clouds, which are then annotated. While this approach results in more comprehensive and detailed annotations, it also means that KITTI-360 does not directly provide labels for raw LiDAR scans, as reflected in our dataset comparison table. This characteristic necessitates an additional step to recover labels for individual raw LiDAR frames. To address this challenge, we utilized a partial re-implementation of the raw 3D scan accumulation algorithm from the KITTI-360 development kit [60]. This tool facilitates the recovery of instance labels for individual raw LiDAR scans from the aggregated and annotated point clouds. By leveraging this method, we successfully extracted labels for 64,640 individual LiDAR scans, significantly exceeding the number of labeled scans available in SemanticKITTI or ApolloScape datasets. This approach allows us to harness the comprehensive annotations of KITTI-360 while retaining the granularity of individual LiDAR frames. Chapter 3 - Methodology 40 features (tc) has dimensions (Dh×H×W), where Hand Ware the height and width of the range image, respectively. It is important to note that these features are also passed to the Decoder (detailed later) via a skip connection, helping to preserve fine-grained spatial information throughout the network. 2. Projection Layer: To produce tokens compatible with the ViT input, an average pooling layer is applied which is followed by a 1x1 convolutional layer. This process reduces the spatial dimensions of tcfrom H×Wto H PH×W PW, where PHand PWare respectively the patch height and patch width. The final 1x1 convolution produces Dioutput channels. The convolutional stem thus yields M=HW PHPWvisual tokens (v1, . . . , vM), each of dimension Di, matching the input requirements of a standard ViT. To capture global contextual information, the ViT encoder processes an input sequence, t0, which is prepared by stacking all visual tokens (v1, . . . , vM) with a special learnable classification token (vclass ∈RDi) which aggregates information from the entire sequence. Positional embeddings (Epos ∈R(M+1)×Di) are then added to the stacked tokens, providing spatial context to the encoder: t0= [vclass, v1, ..., vM] + Epos (3.2) This input sequence is then processed through Ltransformer blocks, resulting in an updated sequence of tokens (tL∈R(M+1)×Di). After removing the classification token, we obtain the final deep patch representations (t′ L∈RM×Di). Following RangeVIT, We use pre-trained VIT weights. Specifically, we initialize our ViT encoder with weights from a model pre-trained on supervised image segmentation on the 2D urban driving Cityscapes dataset [65]. This model, in turn, was initialized with weights from a model pre-trained on ImageNet-21k [66]. Despite the apparent domain shift between natural images and range images, this transfer learning approach has been shown to significantly improve performance and training convergence in LiDAR segmentation tasks [49]. The benefits of this pre-training strategy are twofold: 1. Feature Transferability: The pre-trained model has learned to extract general visual features that are, to some extent, transferable to range image data. This provides a strong starting point for our task-specific fine-tuning. Chapter 3 - Methodology 41 2. Optimization Stability: The pre-trained weights offer a favorable initialization point in the parameter space, potentially leading to faster convergence and better local optima during fine-tuning. By leveraging this image encoder architecture, RangeSAM effectively processes the range image representation of the point cloud, capturing both local geometric details through the convolutional stem and global contextual information via the ViT encoder. This rich representation forms the foundation for the subsequent prompt-guided segmentation process, enabling our model to effectively isolate and segment objects of interest in sparse LiDAR point clouds. 3.2.3 Sampling and Encoding Prompts Building upon the previous two components, RangeSAM incorporates a prompt encoder, enabling interactive segmentation. This component facilitates the transformation of user-provided or dynamically generated prompts into a format that effectively guides the segmentation process. RangeSAM’s Prompt Encoder is inspired by the Prompt Encoder in SAM, but is tailored specifically for point cloud data and the unique challenges of LiDAR-based segmentation. Unlike SAM, which supports multiple prompt types including points, bounding boxes and text, RangeSAM focuses exclusively on point prompts (hereafter referred to simply as ”prompts”). This design choice is motivated by the natural compatibility of points with 3D data and the intuitive nature of point-based interaction in 3D space, building on the successes of previous work in 3D promptable segmentation [10,13]. Our prompt encoder consists of two main components: a positional encoding layer and a set of learnable embeddings. The positional encoding layer employs a technique based on random Fourier features to encode the spatial information of each prompt [67]. This encoding is crucial for capturing the relative positions of prompts within the range image. The positional encoding for a given prompt at location (x, y) is computed as follows: P E(x, y) = [ sin(f1x),cos(f1x),...,sin(fdx),cos(fdx), sin(f1y),cos(f1y),...,sin(fdy),cos(fdy)] (3.3) In this formulation, the frequencies fiare generated by randomly sampling from a standard normal distribution, followed by scaling the sampled values by 2π. This random sampling strategy allows the positional encoding to capture a wide range of spatial relationships by incorporating a diverse set of frequencies. These frequencies are used to Chapter 3 - Methodology 42 generate the features that comprise the positional encoding of a prompt, allowing the model to learn to associate spatial relationships with semantic meaning effectively. The resulting positional encoding P E(x, y) has a dimension of Dpe = 2d, where dis the number of frequency components used. In addition to the positional encoding, we employ learnable embeddings to differentiate between foreground and background prompts. Foreground prompts specify the area of the object of interest, while background prompts are used to exclude irrelevant areas from the segmentation mask. Inspired by SAM, We define two learnable embedding vectors, Efg for foreground prompts and Ebg for background prompts, each with a dimension Dpmatching the positional encoding dimension Dpe. These embeddings are initialized randomly and are fine-tuned during training to learn optimal representations for each of the foreground and background prompt types. The final embedding for a prompt is computed as: Ep=P E(x, y) + Etype (3.4) where Etype is the learnable prompt embedding corresponding to either foreground or background. Another key aspect of our prompt encoder is its ability to handle both 2D and 3D prompts seamlessly. While the prompt encoding process occurs in the 2D space of the range image, the model maintains compatibility with 3D interaction. This is achieved by projecting 3D prompts onto the range image, employing the projection method specified in equation 3.1 of the range projection section. This approach allows users to interact with the point cloud in 3D space while the model operates on the range image representation internally. To facilitate training and evaluation without relying on user interactions, which is impractical, we employ a random sampling strategy, a commonly used technique in interactive segmentation within the 2D domain [13,68]. In this strategy, within the range image, we randomly select between 1 and 12 foreground prompts from the target object and between 0 and 8 background prompts from an expanded bounding box around the target object. The expanded bounding box is created by enlarging the object’s bounding box by 20% in each dimension, ensuring that background points are sampled from relevant areas close to the object. In contrast to our method, SAM utilizes a more complex iterative refinement process for prompt generation during training. This approach simulates a series of user interactions, progressively refining the segmentation mask by adding new points based on the current Chapter 3 - Methodology 43 prediction’s errors. While this method allows for more realistic simulation of user interactions and enhances prompt emphasis due to repeated prompt-feature interactions, it is very computationally intensive. Considering our limited computational resources (which will be detailed later), we chose a simpler, non-iterative prompt sampling approach in RangeSAM. Our method strikes a good balance between training efficiency and the ability to learn from diverse prompt configurations while ensuring the model is exposed to a wide range of prompts during training. Figure 3.5 illustrates the result of our prompt sampling strategy on a range image. The red points represent foreground prompts sampled from the green target object, while the blue points indicate background prompts sampled from the surrounding area. Figure 3.5: Random Prompt Sampling on Range Image Figure 3.6 shows the same prompts re-projected into 3D space, demonstrating how 2D range image prompts correspond to locations in the original point cloud. This visualization highlights the model’s ability to bridge 2D and 3D representations effectively. Figure 3.6: Reprojection of 2D Prompts onto the 3D Point Cloud The prompt encoder’s output is a set of embedded points that encapsulate both spatial and semantic information. These embeddings are then passed to the subsequent stages of the RangeSAM architecture, guiding the segmentation process by providing information about the object of interest and its surroundings. Chapter 3 - Methodology 44 3.2.4 Bi-Directional Transformer Integrated Decoder The decoder component is the most crucial aspect of RangeSAM’s architecture, as it enables transforming the encoded representations into a segmentation mask, while effectively integrating the information contained in the prompts. Our decoder incorporates elements from the decoders used in both SAM and RangeVIT, optimizing it for our 3D promptable segmentation task. The adapted components include an advanced upsampling mechanism, skip connections, and a bi-directional transformer for processing prompts. Figure 3.7: RangeSAM Decoder Architecture As illustrated in Figure 3.7, the RangeSAM decoder integrates several key components to process the image embeddings and prompt embeddings effectively. The decoder takes as input the image embeddings x∈RM×Di, where Mis the number of visual tokens and Diis the image embedding dimension, and the prompt embeddings p∈RN×Dp, where Nis the number of prompts and Dpis the prompt embedding dimension. It produces an output mask m∈RH×W, where Hand Ware the height and width of the cropped range image, respectively. Following RangeViT’s design, the working dimension of our decoder is set to Dh, matching the dimension of the context features from the convolutional stem. The architecture can be broadly divided into three main stages: (1) the input projection and upsampling, (2) the bi-directional transformer for prompt-image interaction, and (3) the final mask prediction and refinement. Each of these components will be examined in detail. Chapter 3 - Methodology 45 Our decoder begins by projecting the encoded features from the ViT encoder to the decoder’s working dimension Dh. This is accomplished through a learnable linear projection, ensuring that the encoder’s output is compatible with the decoder’s architecture while allowing for adaptive feature transformation. Following the projection, the decoder employs an upsampling mechanism that combines bilinear interpolation with convolutional refinement. This approach, adapted from RangeViT, allows for efficient upscaling of the feature maps while preserving spatial information. The upsampling process can be expressed as: xu=fu(xp, tc) (3.5) where furepresents the upsampling function, xp∈RDh×Hp×Wpis the projected input, tcis the context features tensor passed through a skip connection, and xu∈RDh×H×W is the resulting upsampled output. The function fuinternally uses bilinear interpolation followed by convolutional layers to refine the upsampled features. As seen in RangeViT, The skip connection from the convolutional stem enhances spatial accuracy by directly supplying the pixel-wise context features tc. These features are concatenated with the upsampled features along the channel dimension and refined through convolutional layers. This approach allows the decoder to utilize both highlevel semantic information from the encoder and fine-grained spatial details from earlier layers, thereby improving segmentation quality. While the upsampling process play crucial roles in refining spatial details, the most significant aspect of the RangeSAM decoder is the integration of a bi-directional transformer to process prompts alongside image features. This transformer, inspired by SAM but adapted for range image data, allows for effective interaction between the prompt embeddings and image features. More specifically, the transformer processes the prompt embeddings, image features, and their positional information, and produces refined outputs that have been enriched through multiple layers of interaction. The bi-directional transformer has a depth of nlayers, each containing self-attention mechanisms for the prompts, cross-attention between prompts and image features in both directions, and feed-forward networks. This bi-directional attention flow allows for rich interaction between the prompts and image features, enabling the model to effectively utilize the provided prompts for segmentation. For each layer of the bi-directional transformer, we perform the following operations: Chapter 3 - Methodology 46 1. Self-attention on prompt embeddings: p′ i= (pi+ MHA(pi, pi, pi)) (3.6) 2. Cross-attention from prompts to image features: p′′ i= (p′ i+ MHA(p′ i, xi, xi)) (3.7) 3. Cross-attention from image features to prompts: x′ i= (xi+ MHA(xi, p′′ i, p′′ i)) (3.8) 4. Feed-forward networks for both prompt and image embeddings: pi+1 = (p′′ i+ FFN(p′′ i)) xi+1 = (x′ i+ FFN(x′ i)) (3.9) In these operations, xi∈RM×Dhrepresents the input image features to layer i,pi∈ RN×Dpare the prompt embeddings, MHA is multi-head attention, and FFN is a feedforward network. To further refine the decoder’s output, we introduce a learnable scaling factor α∈ Rfor prompt embeddings. This scaling factor is applied to the image features xi∈ RM×Dhbefore their interaction with prompt embeddings pi∈RN×Dpin the crossattention mechanism, modifying the operation to MHA(p′ i, αxi, αxi). This allows the model to learn to balance the influence of the prompt information and the visual data, ensuring that neither source dominates and both contribute meaningfully to the final segmentation. This is particularly valuable in RangeSAM, as it compensates for the absence of an iterative refinement process found in SAM, helping to maintain sufficient emphasis on prompts throughout the decoding process. After these bi-directional transformer operations, the updated image features and prompt embeddings are processed through an additional attention mechanism, following SAM’s design. This enables a prompt-guided refinement of the image features, enhancing the segmentation based on the provided prompts. The attended features are then passed through a convolutional layer that projects them into the output space, generating the segmentation mask. Unlike SAM, which produces multiple mask hypotheses and uses an IoU prediction head, RangeSAM opts for a simpler approach that generates a single mask output. This design Chapter 3 - Methodology 47 choice is motivated by the need for computational efficiency and the specific requirements of range image segmentation. The final output of the decoder is a segmentation prediction at the same resolution as the cropped range image. However, To ensure that the prompts are accurately reflected in the final prediction, RangeSAM includes a logit modification step we title as ”prompt label enforcement”. This step adjusts the logits—the model’s raw prediction values before conversion to probabilities—by modifying them at each spatial location (i, j) containing a prompt as follows: M′(i, j) =          +∞for the logit of the class corresponding to the prompt label −∞ for the logit of the class not corresponding to the prompt label M(i, j) if location (i, j) contains no prompt (3.10) This enforcement step ensures that the network’s output matches the prompt labels, either foreground or background, at the specified locations. During training, this modification influences both the loss computation and backpropagation. The prompted locations will always produce minimal loss values, therefore the network learns to be more confident in its predictions around prompt locations while relying on these points as anchors for segmenting ambiguous regions. This effectively acts as a hard constraint that ensures the model’s output respects the user-provided prompts. Ultimately, the decoder of RangeSAM effectively processes range image data, producing a segmentation mask aligned with the LiDAR point cloud’s range image representation. This completes the trainable portion of the network. While the output at this stage is capable of segmenting objects in the range image space, the segmentation mask is still in the 2D range image domain. This will be addressed through a final 3D refinement stage, which will create an entirely 3D mask. 3.2.5 KNN-Based 3D Refinement The final component of RangeSAM is the KNN-based 3D refiner, designed to address potential information loss during the range projection process and refine the segmentation results in 3D space. This component builds upon the KNN post-processing technique introduced in RangeViT, adapting it for binary segmentation rather than multi-class segmentation. Chapter 3 - Methodology 48 The primary purpose of the KNN 3D Refiner is to overcome the limitations of range image representation, particularly the occurrence of ”holes” or missing points in the projected 2D space. While the range image effectively preserves most of the 3D spatial information, some points may be lost due to occlusions or varying point densities. The KNN refiner helps recover these points and ensures a more accurate 3D segmentation. Before the KNN refinement begins, RangeSAM projects the 2D segmentation results back onto the original 3D point cloud. This projection is achieved using stored indices that map each pixel in the range image to its corresponding 3D point. However few 3D points may not have a corresponding 2D pixel and thus lack a segmentation label. These unlabeled points become the focus of the KNN refinement. For these points, the KNN algorithm operates by examining the Knearest neighbors of each point in the 3D space. Figure 3.8 provides a general illustration of the KNN algorithm, demonstrating how neighboring points are selected based on distance. The green point represents an unlabeled point, with K= 3 selecting the three nearest neighbors, classifying it as red, and K= 5 expanding the neighborhood to five points, classifying it as blue. This visual helps clarify how KNN aggregates information from nearby labeled points, a key aspect of the refinement process in RangeSAM. Figure 3.8: KNN Algorithm Illustration The KNN refinement process of RangeSAM can be broken down into several steps: 1. Identification of Unlabeled Points: Points in the 3D point cloud that did not receive a label through the initial 2D-to-3D mapping are identified. 2. Neighborhood Sampling: For each unlabeled point in the 3D point cloud, the algorithm identifies its Knearest neighbors within the specified search radius. This step utilizes the original 3D coordinates of the points, allowing for accurate spatial relationships to be considered. Chapter 3 - Methodology 49 3. Distance Weighting: The algorithm applies a Gaussian weighting to the distances between the unlabeled point and its neighbors. This weighting ensures that closer points have a stronger influence on the final classification. The Gaussian kernel is defined as W(d) = exp −d2 2σ2, with dbeing the distance between points and σ being the standard deviation. 4. Label Aggregation: The algorithm aggregates the binary labels (0 for background, 1 for foreground) from the labeled neighbors. These labels are weighted by their corresponding distance weights. 5. Final Classification: The final 3D label for each unlabeled point is determined by thresholding the weighted average of its neighbors’ labels. If the average exceeds 0.5, the point is classified as foreground; otherwise, it is classified as background. This process can be formulated as: L3D(p) =      1 if PK i=1 W(di)·L(ni) PK i=1 W(di)>0.5 0 otherwise (3.11) where L3D(p) is the final 3D label for the unlabeled point p,L(ni) is the label of the i-th neighbor (either from the initial projection or previously refined), and W(di) is the Gaussian weight based on the distance to the i-th neighbor. Overall, by considering neighboring points, the KNN 3D Refiner recovers information for occluded points and smooths out noise and inconsistencies in the segmentation. Additionally, it helps preserve fine geometric details that might have been lost in the 2D projection process. By incorporating this final refinement step, RangeSAM effectively bridges the gap between 2D range image processing and 3D point cloud segmentation, aiming to produce high-quality, and spatially consistent results by leveraging the strengths of both representations. 3.3 Implementation Details The implementation of RangeSAM involves careful consideration of various components, from data preprocessing to model architecture and training strategies. This section outlines the key aspects of our implementation, including the training setup, hyperparameter tuning process, and evaluation metrics. Chapter 3 - Methodology 56 do not reflect the fully-trained model’s performance in 3D promptable segmentation and are used solely for comparative purposes. The hyperparameters we tuned and their respective search spaces were as follows: learning rate (1e-5 to 1e-2), transformer depth (1 to 4 layers), number of attention heads (2, 4, 8, or 16), MLP dimension in the transformer (512, 1024, 1536, or 2048), and prompt embedding dimension (64, 128, 256, or 384). These ranges were chosen based on common practices in transformer architectures and our hardware requirements. Figure 3.9: IoU Values Across Tuning Trials Figure 3.9 presents the distribution of IoU values across our hyperparameter tuning trials. The box plot, complemented by a swarm overlay, reveals a performance range from 12.82 to 20.55, with a median of 15.90. This substantial spread of nearly 8% between the minimum and maximum IoU values underscores the critical impact of hyperparameter tuning on our model’s performance. The concentration of points around the median and upper quartile indicates that many configurations achieved moderate success, while the outliers near 20.0 highlight the potential for significant improvement through optimal parameter selection. Following this, in Figure 3.10 we illustrate the relationships between hyperparameters in the top 10 performing trials using a parallel coordinates plot. Each line represents a trial, with its color indicating the IoU value achieved. This visualization allows us to observe how different hyperparameter combinations contribute to model performance. Notably, the best-performing configurations cluster around a learning rate of 0.0004 , with a consistent transformer depth of 3 layers. While both 8 and 4 attention heads Chapter 3 - Methodology 57 Figure 3.10: Hyperparameter Relationships of Top 10 Performing Trials are present, the majority of top trials favor 8 heads. The MLP dimension is predominantly set at 2048, indicating a preference for higher model capacity. Interestingly, the prompt embedding dimension is consistently low (mostly 128) compared to the highest possible value (384), suggesting that smaller prompt representations are sufficient. The top-performing trial (IoU 20.55) uses 8 attention heads, the largest MLP dimension (2048), and a prompt embedding size of 128. Most other high-performing trials differ only slightly from this configuration. Based on these insights, we selected the hyperparameters from the top-performing trial for our final model, with the learning rate set to 0.0004, as it represents the optimal value in the continuous range observed across trials. As previously seen, the selection of these values is reflected in the training setup. Following the optimization of our main model architecture, we conducted a separate hyperparameter tuning process for the KNN-based 3D refiner. This post-processing step is not part of the trainable network but plays a crucial role in refining the final 3D segmentation output. We evaluated various combinations of the number of neighbors (k), search radius, and kernel standard deviation (σ) on our fully trained model. Given the relatively few parameters, we performed a grid search over the parameter space: number of neighbors k(1, 3, 5, 7, 9), search radius (3, 5, 7), and σ(0.5, 1, 2). These combinations were assessed to identify the optimal configuration. Figure 3.11: KNN Refiner Hyperparameter IoU Score Heatmap Figure 3.11 presents a heatmap of the IoU scores achieved with different hyperparameter combinations for the KNN refiner at the points which it is applied to, with darker cells Chapter 3 - Methodology 58 representing higher IoU. Interestingly, the best results were obtained with k=1 and a search radius of 3, which effectively renders the kernel standard deviation parameter irrelevant. This configuration achieved a high IoU score, significantly outperforming other combinations. A smaller neighborhood and moderate search radius indicate that relying on the nearest neighbor within a reasonable range offers the most precise refinement. This result can be explained by the prevalence of background points in LiDAR scans. Including more neighbors can inadvertently cause foreground points to be misclassified as background, particularly in areas with sparse object representation. Ultimately, by carefully exploring the hyperparameter space for both our main architecture and the 3D refiner, we have ensured that our model is well-configured to tackle the challenges of promptable, class-agnostic segmentation in sparse LiDAR point clouds. Chapter 4 Performance Evaluation Building upon the methodological framework outlined in the previous chapter, we now present a comprehensive evaluation of RangeSAM’s performance in promptable, classagnostic segmentation of LiDAR point clouds. Since our approach uniquely addresses the challenge of sparsity which is overlooked in existing research, direct comparison with prior methods is challenging. To contextualize our results, we have devised appropriate baselines. Our evaluation strategy is designed to provide a thorough understanding of RangeSAM’s capabilities, strengths, and limitations. The analysis encompasses quantitative metrics for assessing segmentation accuracy and efficiency, qualitative examinations to visualize the model’s output in various scenarios, and additional experiments which help understand the contribution of training datasets and key architectural components. This multi-faceted approach offers meaningful insights into RangeSAM’s performance both in absolute terms and relative to the established baselines, highlighting its potential impact in the field of point cloud segmentation. 4.1 Evaluation Preliminaries Prior to discussing our specific experiments, it is essential to establish some preliminaries for our evaluations. This section outlines key aspects of our evaluation methodology, ensuring clarity and consistency across subsequent analyses. Prompt Sampling Strategy. Central to our evaluation process is the point sampling strategy used to simulate user prompts. While our training process used flexible 59 Chapter 4 - Performance Evaluation 60 random sampling, our evaluation requires a more controlled approach to ensure fair comparisons. For quantitative evaluations, we employ a consistent point sampling strategy that mirrors our training methodology with a fixed number of prompts. When sampling Kpoints for evaluation, we target a ratio of approximately 65% foreground to 35% background points. This ratio, determined through empirical testing, provides a balance between object information and context. Points are sampled from within an expanded bounding box around the target object, enlarged by 20% in each dimension. For K= 1, we select a single foreground point; for K > 1, at least one background point is always included. If the number of available points is insufficient, we adjust the sampling while maintaining the total count of Kpoints. This strategy enables us to evaluate our model’s performance consistently across experiments, providing a standardized framework for assessing RangeSAM’s capabilities. In the following experiments, this strategy will be utilized as our prompt selection strategy unless explicitly stated otherwise. Promptable Segmentation Comparisons. The novel nature of RangeSAM, particularly its focus on sparse LiDAR scans, presents unique challenges in comparative evaluation with other existing 3D promptable segmentation models. Ideally, we would evaluate these models on sparse data similar to ours. However, practical constraints such as unavailable model weights, proprietary code, and the substantial modifications required make such direct comparisons infeasible within our research timeline. To bridge this gap and facilitate a more meaningful comparison, we’ve introduced a distinction between ”sparse” and ”dense” objects within our evaluation framework. This categorization is based on the observation that points closer to the LiDAR sensor tend to be denser, while those farther away are sparser. Specifically, we define objects as ”dense” if they meet two criteria: (1) their center point is within 15 meters of the sensor, and (2) they contain at least 50 foreground points. Objects that don’t meet both criteria are classified as ”sparse.” These thresholds were determined through visual examination of the point clouds, striking a balance that reasonably differentiates between noticeably denser and sparser object representations. While this approach doesn’t fully equate our task’s difficulty with that of existing methods, it allows us to contextualize RangeSAM’s performance more effectively. Importantly, this strategy ensures we don’t overestimate our model’s capabilities relative to others. Instead, it provides a conservative basis for comparison, acknowledging that even our ”dense” objects may present more challenges than the data typically used in prior work, due to them in actuality still being sparse. Chapter 4 - Performance Evaluation 61 Evaluation Models. For the majority of our evaluations, including all experiments in the Quantitative and Qualitative Analysis sections, we employ the model outlined in Section 3.3.1, which is trained on the KITTI360 dataset over 200 epochs. This model serves as our primary benchmark for assessing RangeSAM’s performance. Additionally, we will explore specific scenarios requiring models trained on alternative datasets or under varied conditions, detailed in a dedicated section containing additional experiments. These experiments are designed to examine specific aspects of RangeSAM’s architecture and cross-dataset performance. In each case, the exact model configurations will be specified, allowing us to maintain consistency in the main analysis while providing insights into the model’s behavior across diverse training regimes. Metric Prioritization. While our methodology introduced a comprehensive set of evaluation metrics, we prioritize the Intersection over Union (IoU) of the foreground mask as our primary performance indicator. IoU effectively captures many aspects of segmentation quality that other metrics individually represent. It inherently balances precision and recall, as it considers both false positives and false negatives in its denominator. Moreover, IoU’s sensitivity to spatial overlap aligns well with the Sassoc metric’s goal of assessing instance-level segmentation quality. For our experiments which focus on RangeSAM’s promptable segmentation capabilities, we will predominantly use the IoU variant we introduced as IoU@K, as is the standard practice across both 2D and 3D Promptable Segmentation techniques. However, to facilitate meaningful comparisons with non-promptable, class-agnostic segmentation models, and to gain further insights into the model’s behavior, we will employ the full range of metrics introduced. This approach ensures a fair and comprehensive evaluation across different segmentation paradigms, allowing us to contextualize RangeSAM’s performance within the broader landscape of 3D point cloud segmentation techniques. Object Coverage. Unless stated otherwise, our evaluation experiments process all object instances present in each processed point cloud scan, in contrast to our training methodology where one random object is sampled per scan. This comprehensive evaluation protocol allows for thoroughly assessing model performance across a full range of object characteristics, providing robust insights into the model’s segmentation capabilities under diverse operating conditions. Chapter 4 - Performance Evaluation 62 4.2 Quantitative Analysis 4.2.1 Prompt Influence Setup. To evaluate the impact of prompts on RangeSAM’s segmentation performance, we conducted an experiment consisting of two complementary components. The first component assessed the model’s segmentation quality as a function of the number of prompts provided, while the second examined the model’s sensitivity to prompt labels. In the first component, we evaluated RangeSAM’s performance using the IoU@K metric, where Krepresents the number of prompts provided. We varied Kfrom 1 to 100, applying our consistent point sampling strategy to select prompts for object instances in a representative subset (approximately a quarter) of the KITTI360 test dataset. This sampling approach allowed us to efficiently explore a wide range of Kvalues while maintaining statistical significance, providing insights into the model’s prompt utilization efficiency and the potential trade-offs between user interaction and segmentation quality. Additionally, we conducted separate evaluations using only foreground prompts and only background prompts to understand their individual contributions to the segmentation process. This comprehensive analysis facilitates in revealing the model’s ability to leverage different types of prompt information and the synergistic effects of combining foreground and background cues. The second component aimed to verify the model’s reliance on prompt labels by deliberately inverting the prompt assignments. We selected K= 20 as it represents a practical upper limit for user input in real-world scenarios, balancing between segmentation accuracy and user effort. For each object instance, we assigned background labels to prompts within the object and foreground labels to prompts in the background, effectively creating an ”inside-out” prompting scenario. This test serves to confirm whether RangeSAM genuinely learns to leverage the semantic information provided by the prompts or if it relies more heavily on the spatial distribution of the points of the point cloud. These components collectively provide a detailed view of RangeSAM’s prompt-driven segmentation capabilities, assessing both its performance scaling with prompt quantity and its ability to correctly interpret prompt semantics. Results and Discussion. The results of the first component, illustrating RangeSAM’s segmentation performance as a function of the number of prompts, are presented in Figure 4.1. Chapter 4 - Performance Evaluation 63 Figure 4.1: IoU Performance Across Varying Prompt Counts on KITTI360 This figure reveals several key insights into RangeSAM’s behavior. As represented, the performance curve can be divided into three distinct regions, each characterizing a different phase of the model’s response to increasing prompt inputs. In the initial phase (K= 1 to K≈10), we observe rapid improvement, with IoU rising sharply from 5.9 at K= 1 to about 50.5 at K= 10, demonstrating the model’s ability to quickly leverage even a small set of prompts. The second phase (K≈10 to K≈21) shows steady improvement, with IoU reaching approximately 57.0 at K= 21, aligning with our training process which used a maximum of 20 prompts. Beyond K= 21, performance gradually improves, despite minor fluctuations potentially caused by the model encountering prompt configurations outside its training range. Overall, the trend remains positive, with the IoU reaching approximately 66.9 at K= 100. Notably, the figure also illustrates the model’s performance when provided with only foreground or only background prompts. For the first few prompts (K≤5), the foregroundonly and all-prompts curves are nearly identical, while the background-only curve shows a rapid initial increase from near-zero to about 15.0 IoU. Beyond this point, the curves diverge into a more consistent pattern. The background-only curve, while consistently lower than the all-prompts curve, shows non-trivial segmentation capability, maintaining IoU values close to 20.0. This suggests that background prompts alone provide some context for object delineation, possibly by helping the model identify object boundaries through contrast. The foreground-only curve shows a pattern similar to the all-prompts curve, albeit with consistently lower IoU values. This indicates that while foreground prompts are crucial for object delineation, the inclusion of background prompts provides valuable additional context for refinement, enabling more accurate segmentation. The Chapter 4 - Performance Evaluation 64 relatively constant gap between the all-prompts and foreground-only curves beyond the initial prompts suggests that the contribution of background prompts, while significant, remains fairly consistent as the number of prompts increases. RangeSAM’s performance with a single foreground prompt, while limited, shows some capability in object segmentation. The steep initial improvement with just a few additional prompts demonstrates the model’s efficient use of minimal user input, potentially streamlining real-world manual segmentation tasks. This rapid improvement suggests RangeSAM quickly builds a robust understanding of object boundaries and spatial relationships within point clouds, even with sparse input. The continued, gradual improvement beyond the training range showcases RangeSAM’s ability to generalize and handle more complex segmentation scenarios. This indicates the model has learned flexible, generalizable features and strategies rather than simply memorizing specific prompt configurations. However, diminishing returns at higher prompt counts point to limitations in the current architecture and the nature of the task itself. As more prompts are added, they likely provide increasingly redundant information, suggesting a practical limit to the useful data extractable from prompts alone. Additionally, the inherent sparsity of LiDAR point clouds may impose a ceiling on achievable accuracy, regardless of the number of prompts provided. This observation highlights the challenging nature of segmentation in sparse 3D data and suggests that future improvements might require novel approaches to information integration or alternative input modalities to complement prompts. Proceeding to the next analysis, Table 4.1 presents the results of the second component, which compares normal and inverted prompt labeling at K= 20. Prompt Labeling IoU Normal 56.05 Inverted 17.53 Table 4.1: Effect of Prompt Label Inversion The comparison between normal and inverted prompt labeling reveals interesting insights into RangeSAM’s behavior. While the performance drop with inverted prompts is substantial, the resulting IoU of 17.53 is not negligible, suggesting that the model still extracts useful information even from incorrectly labeled prompts. This resilience is noteworthy, as it indicates that RangeSAM’s segmentation capability surpasses merely relying on prompt labels in their entirety. Chapter 4 - Performance Evaluation 65 The persisting performance with inverted prompts can be attributed to several factors. Firstly, even with incorrect labels, the prompts still provide valuable spatial information about the object’s location and extent within the point cloud. This suggests that RangeSAM leverages both the semantic labels and the spatial distribution of prompts in its segmentation process. Secondly, the model may be identifying patterns or features in the point cloud that correlate with object boundaries, allowing it to partially overcome the label inversion. However, the significant performance gap between normal and inverted prompts (56.05 vs 17.53) underscores the importance of correct labeling. It demonstrates that RangeSAM does indeed rely heavily on the semantic information provided by the prompts, using it to guide its attention and refine its segmentation decisions. The fact that performance with inverted prompts falls below that achieved with just two correctly labeled prompts (IoU@2 = 20.94) further emphasizes the value of accurate prompt information. Overall, the two components of this experiment yield several key insights into RangeSAM’s capabilities and limitations. It demonstrates the model’s ability to effectively utilize prompts for guided segmentation, with performance improving as more prompts are provided. The model shows robustness in handling prompts beyond its training range and exhibits a degree of resilience to mislabeled inputs. However, it also highlights areas for potential improvement, such as enhancing single-prompt performance, improving performance with a higher number of prompts, and developing strategies to better manage noisy or incorrect user inputs. 4.2.2 Cross-dataset Generalization Setup. This evaluation assesses RangeSAM’s ability to generalize across different datasets and urban environments. We evaluate the model’s performance on three distinct datasets: the KITTI360 test set (Karlsruhe, Germany, semi-urban environments), a subset of SemanticKITTI (Karlsruhe, suburban environments), and the entire ApolloScape dataset (various cities in China). We use IoU@K as our primary metric, evaluating performance at various Kvalues. The KITTI360 test set serves as our baseline, while we select a random subset of SemanticKITTI with an equivalent number of scans to ensure a balanced comparison. Due to its smaller size, we use the entire ApolloScape dataset. This analysis aims to provide insights into RangeSAM’s adaptability to different geographical contexts and urban layouts, demonstrating the model’s potential for real-world applications across diverse global settings. Chapter 4 - Performance Evaluation 72 count on segmentation quality, assess the model’s performance on different datasets, and investigate potential areas for improvement in our prompt sampling strategy. We begin by examining the impact of increasing prompt counts on segmentation quality for a typical object in the SemanticKITTI dataset. Figure 4.4 illustrates this progression, showcasing segmentation results for K= 3, K= 5, and K= 10 prompts, with red spheres representing foreground prompts and blue spheres representing background prompts. Figure 4.4: Object Instance Segmentation Example in SemanticKITTI As evident in this figure, the segmentation quality improves markedly as the number of prompts increases. At K= 3 (IoU 65.1), a tendency for over-segmentation can be observed, with the predicted mask extending beyond the ground truth boundaries. This suggests that with limited prompts, RangeSAM exhibits an over-inclusive bias, potentially capturing contextual information or nearby objects. As the prompt count increases to K= 5 (IoU 74.3), the segmentation becomes more refined, with fewer extraneous points included. However, some points in close proximity to the object are still incorrectly classified as foreground. At K= 10 (IoU 90.3), the model produces a highly accurate segmentation that closely aligns with the ground truth, with minimal false positives or negatives. This progression aligns with our quantitative findings, demonstrating RangeSAM’s ability to effectively utilize additional prompt information to refine its segmentations. The significant improvement from K= 3 to K= 10 in the example underscores the value of relevant prompts in achieving high-quality segmentations, particularly in challenging or ambiguous scenarios. It also highlights the model’s capacity to integrate multiple spatial cues to form a better understanding of object boundaries. Chapter 4 - Performance Evaluation 73 Moving beyond individual objects, Figure 4.5 presents a comparison between ground truth and RangeSAM’s predictions for an entire KITTI360 scan. Each object instance is segmented using 20 prompts (K= 20) and assigned a unique color. The overall IoU averaged across all instances in this scan is 59.8, which is impressive considering the scene’s complexity and the challenges posed by varying object sizes and point densities. Figure 4.5: Prediction Visualization of a Complete KITTI360 Scan In this figure, the predicted segmentation generally mirrors the ground truth. However, upon closer inspection, we notice that some objects in the prediction are missing points that are present in the ground truth, while others include extra points, with them being scattered onto other objects. This discrepancy could be attributed to several factors. For distant or partially occluded objects, the model may tend to struggle with accurately defining boundaries due to sparser point representation. Conversely, for larger or closer objects, the model can include nearby points not present in the ground truth, likely due to it aiming to mimic the contextual understanding gathered from training. Nonetheless, the overall coherence of the predicted segmentation demonstrates RangeSAM’s ability to maintain consistent performance across diverse object types and spatial arrangements within a scene. This is particularly noteworthy given the varying point densities and occlusions present in LiDAR scans. To assess RangeSAM’s generalization capabilities, we also examined its performance on the ApolloScape dataset, testing the model on a distinct urban environment with Chapter 4 - Performance Evaluation 74 Figure 4.6: Prediction Visualization of a Complete ApolloScape Scan different object characteristics than what the model has been trained on. Figure 4.6 illustrates the model’s segmentation results on an ApolloScape scan. The ApolloScape results reveal more pronounced challenges in segmentation accuracy compared to the KITTI360 scan. The average IoU across instances in this scan is 48.6, indicating a noticeable drop in performance. We observe a higher incidence of both false positives and false negatives in the predicted segmentations. Some objects in the ground truth have very few corresponding points in the prediction, suggesting difficulties in detecting certain objects entirely. Conversely, other predicted objects, while generally resembling their ground truth counterparts, exhibit scattered points extending beyond the true object boundaries. The performance gap between KITTI360 and ApolloScape aligns with our cross-dataset evaluation findings, which could be attributed to several factors. One factor is the differing urban environments in ApolloScape compared to those in training, presenting new architectural styles, vehicle types, and road layouts. Additionally, variations in LiDAR sensor specifications or data collection methods may lead to differences in point cloud density or distribution. These nuances, though subtle, could significantly impact results. Despite these challenges, the model still demonstrates a degree of effectiveness in segmenting many objects within the ApolloScape scan, suggesting that while RangeSAM’s Chapter 4 - Performance Evaluation 75 performance does degrade when applied to significantly different environments, it retains some ability to generalize its learned segmentation strategies. Finally, our qualitative analysis revealed an important insight into the prompt sampling strategy employed in RangeSAM. Figure 4.7 illustrates a recurring issue we observed in the spatial distribution of prompts. Figure 4.7: Prompt Distribution Containing Background and Foreground Artifacts This visualization highlights that background prompts may possibly be located far from the object of interest, and in rare cases, even foreground prompts can be misplaced outside the object’s true boundaries. This suboptimal prompt distribution can lead to several issues: 1. Reduced segmentation accuracy: Distant prompts prompts may cause the model to focus on irrelevant areas of the point cloud, potentially leading to over-segmentation or misidentification of object boundaries, reducing overall segementation quality. 2. Inconsistent instance grouping: Misplaced prompts prompts might result in the inclusion of foreground points from adjacent objects or background regions, contributing to the lower Sassoc scores observed in our quantitative analysis. 3. Inefficient use of prompt information: Prompts placed far from the object of interest provide less valuable information for refining the segmentation, potentially explaining why performance improvements diminish with higher prompt counts. The root cause of this observation lies in our current prompt sampling strategy, which operates in the 2D space of the range image without considering 3D spatial relationships. As touched upon in our methodology, prompts are sampled from a 2D expanded bounding box on the object of interest. This approach simplifies implementation but fails to account for the complex 3D structure of the point cloud. While the range image preserves depth information, our sampling strategy treats the space as a flat 2D Chapter 4 - Performance Evaluation 76 projection when selecting prompts. Consequently, when these 2D-sampled prompts are projected back into 3D space, they can end up in unexpected locations relative to the object of interest, leading to suboptimal prompt placements. Overall, our qualitative analysis offers deeper insights into RangeSAM’s behavior, corroborating and elucidating our quantitative findings. We observe the model’s ability to refine segmentations with increasing prompt counts, its effectiveness in handling complex scenes with multiple objects, and its challenges in generalizing to significantly different environments. The analysis also reveals opportunities for improvement, particularly in our prompt sampling strategy. 4.4 Additional Experiments 4.4.1 Dataset Influence on Generalization To further investigate the impact of training data diversity on RangeSAM’s generalization capabilities, we conducted an additional experiment comparing models trained on different datasets. This analysis aimed to determine whether the broader range of object types in KITTI360 enhances the model’s ability to generalize to significantly different urban environments. We trained two versions of RangeSAM: one on KITTI360 and another on SemanticKITTI, both for 50 epochs under similar training conditions trained over 15,000 scans each. We then evaluated both models on the ApolloScape dataset, which represents urban environments distinctly different from the training data. Table 4.3 presents the IoU@K values for these models. Training Dataset IoU@5 IoU@10 IoU@15 IoU@20 KITTI360 22.92 32.19 37.81 41.89 SemanticKITTI 21.07 29.47 33.57 36.35 Table 4.3: Training Source Impact on Model Generalization The results show that the model trained on KITTI360 consistently outperforms the one trained on SemanticKITTI when evaluated on ApolloScape, with the performance gap widening to over 5% at K= 20. These findings suggest that training on a dataset with a broader range of object types and urban scenarios enhances the model’s ability to generalize to different environments. KITTI360’s diverse semi-urban scenes likely expose the model to a wider variety of object shapes, sizes, and spatial relationships, fostering more robust and adaptable feature representations for novel urban layouts Chapter 4 - Performance Evaluation 77 and object appearances in ApolloScape. Conversely, the more homogeneous suburban environments of SemanticKITTI may limit the model’s exposure to the full spectrum of urban complexity, potentially constraining its generalization capabilities. To gain a more granular understanding of the models’ performance across different object types, we conducted a class-wise evaluation on the ApolloScape dataset. Table 4.4 presents the IoU@20 values for each object class for both the KITTI360-trained and SemanticKITTI-trained models. Model Ped. Traffic Cyclist S. Vehicle B. Vehicle Other KITTI360-trained 47.04 47.83 44.00 35.76 36.61 41.32 SemanticKITTI-trained 35.45 40.81 33.22 43.84 27.07 32.16 Table 4.4: Object Category Performance by Training Source on ApolloScape The class-wise evaluation reveals interesting patterns in the models’ generalization capabilities. The KITTI360-trained model significantly outperforms the SemanticKITTItrained model in most classes, particularly for pedestrians, cyclists, and big vehicles. This suggests that KITTI360’s more diverse urban scenes provide better training for these object types. The SemanticKITTI-trained model, however, shows a noticeable advantage on small vehicles, likely due to its focus on a narrower range of object types, allowing for more exposure to this specific object type during training. Overall, these results underscore the significant impact of training data diversity on the model’s ability to generalize across different object classes and urban environments. The experiment highlights the importance of exposing our model to a wide range of scenarios during training to enhance its adaptability to novel environments. While both datasets are valuable, the broader diversity of KITTI360 appears to confer advantages in cross-dataset generalization, particularly when dealing with significantly different urban layouts and object appearances. 4.4.2 Ablation Study of Novel Architecture Components To assess the impact of RangeSAM’s key innovations, we conducted an extensive ablation study. This analysis centers on three critical components we introduced in the architecture: the learnable scaling factor, the prompt loss function, and prompt label enforcement. These elements were designed to optimize the model’s ability to effectively leverage prompts during segmentation. As discussed in the methodology, the bi-directional transformer’s learnable scaling factor enables dynamic adjustments between prompt and image embeddings, allowing the Chapter 4 - Performance Evaluation 78 model to learn a balance that improves context-aware segmentations. The prompt loss function integrates a binary cross-entropy signal within a kernel around each prompt, encouraging the model to focus on nearby regions for more responsive prompt-guided segmentation. Finally, prompt label enforcement aligns predictions with the provided prompts’ labels, enhancing consistency and accuracy in segmenting ambiguous regions by reinforcing the importance of prompt locations. To assess the individual and combined effects of these components, we trained five variants of RangeSAM. Each model was trained for 15 epochs on a subset of 3,000 training scans from the KITTI360 dataset and validated on an additional 1,000 scans. To facilitate a controlled comparison, we employed a fixed set of 20 prompts for the evaluation, while also ensuring that the prompt locations remain consistent across all model variants in each scan. This setup allows us to isolate the impact of each component while controlling for other variables. The results of our ablation study are presented in Table 4.5, demonstrating that each of our novel components contributes positively to the model’s performance, with the combination of all three yielding the best outcome. Prompt Loss Label Enforcement Learnable Scaling IoU × × × 26.35 ×✓ ✓ 29.81 ✓×✓30.72 ✓ ✓ ×32.02 ✓ ✓ ✓ 34.32 Table 4.5: RangeSAM Component Ablation Results To understand each component’s contribution, we systematically removed one component while keeping the other two active. The prompt loss function proved to be the most crucial component, as its removal resulted in the largest performance drop, with IoU decreasing by 4.51% (from 34.32 to 29.81). Label enforcement showed the second-highest impact, with its removal leading to a decrease of 3.6% (to 30.72 IoU). Finally, while still beneficial, the learnable scaling factor had the smallest individual impact, with its removal resulting in a performance drop of 2.3% (to 32.02 IoU). When all components work together, we observe a synergistic effect, with the full model achieving 34.32 IoU, which is 7.97% higher than the baseline in which all components are disabled. This demonstrates that while each component makes a valuable individual contribution, their combination leads to the most effective promptable segmentation Chapter 4 - Performance Evaluation 79 system. The prompt loss provides the foundation for effective prompt utilization, enhanced by label enforcement’s consistency constraints and the learnable scaling factor’s dynamic information balancing. To further validate our findings, we assessed the cross-dataset generalization of the two extremes of the KITTI360-trained models: the fully equipped RangeSAM model with all three components, and the baseline version not containing any of these components. Both models were tested on the ApolloScape dataset using a consistent setup of 1000 test scans, each with 20 prompts. The results are detailed in Table 4.6. RangeSAM Variant IoU All Components 18.95 No Components 16.71 Table 4.6: Cross-Dataset Performance of Ablation Variants The results demonstrate that our architectural improvements enhance not only performance but also generalization capabilities. When evaluated on the previously unseen ApolloScape dataset, our full RangeSAM model maintains its advantage, achieving an IoU of 18.95 compared to 16.71 for the baseline model. While the overall performance is lower than on KITTI360 due to the domain shift, the relative improvement of 2.24% indicates that our proposed components contribute to better generalization across different LiDAR datasets. This suggests that the architectural benefits of our components are not dataset-specific but represent genuine improvements in the model’s ability to handle point cloud segmentation tasks. To conclude, the ablation study and the subsequent analysis reveal the individual and synergistic contributions of RangeSAM’s novel components, underscoring the effectiveness of our multi-faceted approach to integrating prompt information, which enhances segmentation performance in sparse outdoor LiDAR point clouds. However, this study was limited to a subset of the full dataset and a restricted number of training epochs, indicating that further exploration with extended training and larger datasets may uncover deeper insights into the long-term effects and interactions of these components. Chapter 5 Conclusion 5.1 Overview This research set out to address a critical gap in 3D point cloud segmentation: the lack of promptable segmentation models capable of handling the varying point densities characteristic of LiDAR scans. Our central research question asked whether integrating range projection with attention mechanisms could mitigate the challenge of sparsity in outdoor LiDAR scans, enabling promptable segmentation in them. The findings suggest a qualified ”yes” to this question, albeit with important nuances that merit discussion. RangeSAM demonstrates robust performance across varying point densities, achieving competitive results against both promptable and class-agnostic segmentation models. The model’s ability to maintain strong performance on sparse data, particularly evident in our experiments, indicates that our approach successfully adapts to sparse data. The rapid improvement in segmentation quality with increasing prompt counts, coupled with strong point-level accuracy metrics, suggests that RangeSAM leverages point prompts effectively to guide segmentation in challenging outdoor environments. Furthermore, the cross-dataset evaluation results, particularly on ApolloScape, indicate that our approach generalizes reasonably well to unseen urban environments, albeit with some performance degradation. This underscores the ongoing challenge of domain adaptation in 3D point cloud segmentation. Additionally, the lower instance-level coherence scores compared to a state-of-the-art conventional class-agnostic model suggest that while RangeSAM excels at point-level classification, it sometimes struggles with grouping points into coherent object instances. Our findings both align with and extend existing literature. While previous works have demonstrated the effectiveness of promptable segmentation in dense point clouds, our 80 Chapter 5 - Conclusion 81 research shows that these principles can be adapted to sparse LiDAR data through careful architectural design. However, the challenge of evaluating our work against existing methods highlighted a broader issue in the field: the lack of standardized benchmarks for promptable 3D segmentation in sparse environments. Despite our attempts to bridge this gap through careful experimental design, the difficulty in making direct comparisons underscores the nascent nature of this research direction. 5.2 Contribution This research makes several significant contributions to the field of 3D computer vision and point cloud processing. First and foremost, it introduces a novel architecture that bridges the gap between promptable segmentation and sparse LiDAR processing, representing the first dedicated attempt to address the challenges of varying point densities in 3D promptable segmentation. This opens new possibilities for interactive scene understanding in outdoor environments. The integration of range projection and attention mechanisms in RangeSAM introduces a novel paradigm for handling sparse 3D data. Our approach demonstrates that effective promptable segmentation is achievable even in sparse outdoor scenarios, paving the way for future advancements. Furthermore, the three architectural innovations we introduced—the learnable scaling factor, prompt loss function, and prompt label enforcement—significantly improved promptable segmentation performance as shown in our ablation study and could be valuable additions to other promptable segmentation approaches in both 2D and 3D domains. From a practical perspective, our work has immediate implications for applications in autonomous driving and urban scene understanding. RangeSAM’s ability to handle sparse LiDAR data while maintaining strong segmentation performance makes it particularly valuable for real-world scenarios where point density varies significantly with distance and environmental conditions. The model’s prompt-driven nature offers flexibility in deployment. It supports user-guided refinement for efficient outdoor data annotation, and although not explored in this research, its architecture could accommodate automated prompt generation processes to enable scene-wide segmentation. Furthermore, our research contributes to the ongoing discourse about the role of user interaction in 3D scene understanding. By demonstrating the effectiveness of point prompts in guiding segmentation, even in challenging sparse environments, we provide evidence for the viability of interactive approaches in practical 3D vision systems. This has implications beyond autonomous driving, potentially influencing applications in Bibliography 88 [43] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. ACM Transactions on Graphics (tog), 38(5):1–12, 2019. [44] Li Jiang, Hengshuang Zhao, Shu Liu, Xiaoyong Shen, Chi-Wing Fu, and Jiaya Jia. Hierarchical point-edge interaction network for point cloud semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10433–10441, 2019. [45] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 16259–16268, 2021. [46] Meng-Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R Martin, and Shi-Min Hu. Pct: Point cloud transformer. Computational Visual Media, 7: 187–199, 2021. [47] Bichen Wu, Alvin Wan, Xiangyu Yue, and Kurt Keutzer. Squeezeseg: Convolutional neural nets with recurrent crf for real-time road-object segmentation from 3d lidar point cloud. In 2018 IEEE international conference on robotics and automation (ICRA), pages 1887–1893. IEEE, 2018. [48] Andres Milioto, Ignacio Vizzo, Jens Behley, and Cyrill Stachniss. Rangenet++: Fast and accurate lidar semantic segmentation. In 2019 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 4213–4220. IEEE, 2019. [49] Angelika Ando, Spyros Gidaris, Andrei Bursuc, Gilles Puy, Alexandre Boulch, and Renaud Marlet. Rangevit: Towards vision transformers for 3d semantic segmentation in autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5240–5250, 2023. [50] Weiyue Wang, Ronald Yu, Qiangui Huang, and Ulrich Neumann. Sgpn: Similarity group proposal network for 3d point cloud instance segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2569–2578, 2018. [51] Jingdao Chen, Zsolt Kira, and Yong K Cho. Lrgnet: Learnable region growing for class-agnostic point cloud segmentation. IEEE Robotics and Automation Letters, 6 (2):2799–2806, 2021. [52] Dipesh Gyawali, Jian Zhang, and Bijaya B Karki. Region-transformer: Selfattention region based class-agnostic point cloud segmentation. arXiv preprint arXiv:2403.01407, 2024. Bibliography 89 [53] A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017. [54] Yunhan Yang, Xiaoyang Wu, Tong He, Hengshuang Zhao, and Xihui Liu. Sam3d: Segment anything in 3d scenes. arXiv preprint arXiv:2306.03908, 2023. [55] Mutian Xu, Xingyilang Yin, Lingteng Qiu, Yang Liu, Xin Tong, and Xiaoguang Han. Sampro3d: Locating sam prompts in 3d for zero-shot scene segmentation. arXiv preprint arXiv:2311.17707, 2023. [56] Rui Huang, Songyou Peng, Ayca Takmaz, Federico Tombari, Marc Pollefeys, Shiji Song, Gao Huang, and Francis Engelmann. Segment3d: Learning fine-grained classagnostic 3d segmentation without manual labels. arXiv preprint arXiv:2312.17232, 2023. [57] Christopher Choy, JunYoung Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3075–3084, 2019. [58] Junsheng Zhou, Jinsheng Wang, Baorui Ma, Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. Uni3d: Exploring unified 3d representation at scale. arXiv preprint arXiv:2310.06773, 2023. [59] Xinyu Huang, Xinjing Cheng, Qichuan Geng, Binbin Cao, Dingfu Zhou, Peng Wang, Yuanqing Lin, and Ruigang Yang. The apolloscape dataset for autonomous driving. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 954–960, 2018. [60] Marcus Schilling. recover-kitti360-label. https://github.com/MarcusSchilling/ recoverKITTI360label, 2023. [61] Yoshua Bengio, Ian Goodfellow, and Aaron Courville. Deep learning, volume 1. MIT press Cambridge, MA, USA, 2017. [62] Haibo He and Edwardo A Garcia. Learning from imbalanced data. IEEE Transactions on knowledge and data engineering, 21(9):1263–1284, 2009. [63] Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Doll´ar, and Ross Girshick. Early convolutions help transformers see better. Advances in neural information processing systems, 34:30392–30400, 2021. [64] Tiago Cortinhal, George Tzelepis, and Eren Erdal Aksoy. Salsanext: Fast, uncertainty-aware semantic segmentation of lidar point clouds. In Advances in Visual Computing: 15th International Symposium, ISVC 2020, San Diego, CA, USA, October 5–7, 2020, Proceedings, Part II 15, pages 207–222. Springer, 2020. Bibliography 90 [65] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213–3223, 2016. [66] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. [67] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in neural information processing systems, 33:7537–7547, 2020. [68] Won-Dong Jang and Chang-Su Kim. Interactive image segmentation via backpropagating refinement scheme. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5297–5306, 2019.