A wearable stereo vision-based obstacle detection system for visually impaired individuals

A wearable stereo vision-based obstacle detection system for visually impaired individuals Andrea Ruo[0009−0007−2183−2884], Dario Napolitano[0009−0001−6506−9709], and Valeria Villani[0000−0001−7619−0101] Department of Sciences and Methods for Engineering (DISMI), University of Modena and Reggio Emilia, Italy {andrea.ruo,valeria.villani}@unimore.it Abstract. Blind and visually impaired individuals face daily challenges in navigating autonomously in unfamiliar urban or domestic environments. Navigation poses critical challenges as obstacles and dangerous objects can compromise their safety and autonomy. Assistive technologies for visually impaired users have evolved to facilitate interaction with a complex and dynamic world, offering increasingly promising solutions. This work presents a wearable computer vision system implemented on a Raspberry Pi 4, utilizing an Intel RealSense D415 camera, capable of detecting and classifying objects and obstacles while estimating and verbally indicating their distance in real time, thus enhancing user autonomy and safety. Keywords: Computer Vision ·Assistive Technology ·Object Detection. 1 Introduction In everyday life, human eyes are the primary means of acquiring information from the surrounding environment. According to the World Health Organization, 285 million people worldwide have visual impairments, including 39 million who are blind [14]. Blind and visually impaired individuals, therefore, face daily challenges in navigating autonomously in complex environments, such as urban or domestic settings. Traditionally, blind individuals rely on white canes or guide dogs for navigation. White canes help detect obstacles, while guide dogs assist the user in reaching their destination. However, both solutions provide limited environmental details, making navigation in complex environments a significant challenge. Indeed, tasks such as crossing streets, moving through crowds, and recognizing objects in unfamiliar places can be risky due to the inability to perceive depth, motion, and spatial relationships. These challenges affect personal safety, independence, and overall quality of life, emphasizing the need for more advanced assistive technologies. Artificial intelligence (AI) and computer vision (CV) offer promising solutions by enabling real-time interpretation of the environment. By combining image processing algorithms with AI models, CV can accurately detect, recognize, and classify objects. These technologies have applications across various 2 A. Ruo et al. USB 3.2 power cable (3A) microHDMI cable optional external display external battery or line power INTEL RealSense D415 camera RGB + depthmap streaming optional bluetooth speaker/earphones Raspberry PI 4 Fig. 1: Hardware architecture layout of the proposed system. fields, such as healthcare [16], automotive [13] and industrial automation [8], enhancing accessibility and increasing efficiency. In robotics, CV plays a crucial role in enabling robots to perceive and understand their surroundings, leading to advancements in tasks like autonomous navigation, object recognition, and robotic assistance [5]. By integrating CV, robots become more intuitive, responsive, and capable of executing complex tasks with enhanced precision. For example, in [15], the authors proposed an architecture for user identification and social navigation in mobile robotics. Their approach leverages CV algorithms for gesture and facial recognition to identify users and facilitate navigation in social contexts. Similarly, in [18], a YOLO-based model was introduced to enhance object detection for service robots, improving their efficiency in navigating indoor environments. Beyond navigation, CV is also useful in improving robotic manipulation and motion planning. In [6], the authors proposed a layered architecture that incorporates visual foundation models to enhance task execution. Moreover, CV technologies can be integrated in human-robot collaboration, as demonstrated in [2], where the AFFDEX 2.0 module was employed to monitor real-time facial expressions during such interactions. In the context of accessibility, CV offers a significant opportunity for the development of assistive systems for blind and visually impaired individuals, allowing the detection and classification of obstacles in their environment. Current solutions implement diverse approaches to enhance spatial perception. AIris [4] integrates high-resolution cameras and natural language processing (NLP) to provide real-time audio scene descriptions, achieving great performance in object recognition and text reading. SOMAVIP [11] employs depth cameras and IoT frameworks to deliver semantic feedback in urban environments. RealSensebased systems [21], using stereo depth sensors such as R200 and RS410, enable efficient obstacle detection and path generation. YOLO-based systems [7] focus Title Suppressed Due to Excessive Length 3 on light RGB object detection, ensuring high speed at the cost of depth accuracy, while simpler configurations with ultrasonic sensors [17] prioritize cost efficiency. Despite these innovations, assistive technologies still face persistent challenges. Many solutions rely on centralized processing or cloud infrastructures, introducing latency and limiting usability in offline environments or areas with poor connectivity. Others provide minimal or non-semantic feedback, reducing the user’s ability to gain a detailed awareness of their surroundings. Another challenge is the integration of machine learning models into embedded systems due to the inherent limitations of these devices. Therefore, achieving a balance between advanced functionality, portability, cost efficiency, and adaptability to dynamic and unstructured scenarios remains an open challenge. In this work, we propose a wearable computer vision system, schematized in Fig. 1, capable of detecting and classifying objects and obstacles while estimating and verbally indicating their distance in real time, thus enhancing users’ autonomous mobility and safety. Specifically, the system employs a lightweight object detection model, EfficientDet-Lite0, in conjunction with the Intel RealSense D415 RGB-D sensor, which utilizes stereo vision technology. The system processes RGB and depth data in parallel on a Raspberry Pi 4, delivering real-time semantic feedback without relying on cloud infrastructure. In addition, portability and affordability enhance the system’s usability for real-world deployment. By addressing challenges related to depth accuracy, real-time feedback, and user-friendly experience, this work contributes to the development of scalable, reliable, and accessible assistive technologies for individuals with visual impairments. 2 Hardware and software components This section introduces the components used in the development of the proposed system. The hardware consists of the Intel RealSense D415 depth camera and the Raspberry Pi 4 single-board computer, which together provide real-time depth perception and computational capabilities. The Intel RealSense D415 camera, shown in Fig. 1, is an advanced depthsensing device designed for precise 3D perception. It employs an active stereo vision system that integrates two infrared sensors with an infrared projector, enabling the generation of high-resolution depth maps in real time. Its RGB sensor further allows the acquisition of color images synchronized with depth data, making it highly suitable for object recognition and 3D modeling applications. Complementing the camera, the Raspberry Pi 4, also depicted in Fig. 1, is a powerful single-board computer designed for embedded applications, featuring a compact size and a flexible design. It is well-suited for projects that require a balance between functionality and low power consumption. The following sections will delve into the software methodologies implemented in the system, including stereo vision techniques for depth estimation and the EfficientDet-Lite0 algorithm for object detection and classification. 4 A. Ruo et al. 2.1 Stereo vision Stereo vision is the process of estimating the depth of scene points based on their positional variation between two image sensors placed at a known relative distance. In this work, stereo vision is employed to acquire depth data, which is then used for obstacle detection. This process relies on identifying correspondences between elements (objects or recognizable details) present in the input images. When an object appears in both images, its position differs slightly between the two; this difference, known as disparity, is used to compute the object’s distance from the observation point. By iterating this operation for all identifiable points in both images, a disparity map is generated [12]. Specifically, the stereo vision process consists of four main stages: i) Offline calibration: to obtain the intrinsic parameters of the two cameras (such as focal length, image center, and lens distortion parameters) and the extrinsic parameters (i.e., the relative position of the two cameras, referred to as the baseline, and their relative rotation); ii) Rectification: which removes lens distortions and reduces the correspondence search from a two-dimensional to a one-dimensional operation, thus accelerating disparity computation; iii) Stereo matching: which aims at identifying homologous points in the stereo image pair; iv) Triangulation: given the disparity map, baseline, and focal length, triangulation determines the 3D position of the corresponding points in space. 2.2 Object detection Object detection is a fundamental task that involves identifying instances of objects within an image and classifying them into predefined categories. Numerous high-speed algorithms have been developed for object detection. Currently, object detection models can be divided into two main categories [22]: two-stage and one-stage detectors. The first consists of models that break down the task into multiple stages, following a coarse-to-fine approach. In contrast, the latter comprises models designed to complete the detection process in a single step using a single neural network. In this work, to ensure the real-time performance of the proposed system on a single-board computer, we adopt the EfficientDet-Lite0 model [20]. EfficientDetLite0 is a lightweight and optimized one-stage detector model designed for devices with limited resources and belongs to the EfficientDet family, which uses the EfficientNet convolutional neural network (CNN) as its backbone [19]. 3 Proposed architecture As introduced above, this work proposes a CV system equipped with an object detection model and stereo vision technology for depth estimation. The intended application scenario is assistive technology for visually impaired or blind individuals, providing them with audio feedback on obstacles in their surroundings. Title Suppressed Due to Excessive Length 5 Object detection process Obstacle detection process RealSense D415 camera Depth queueRGB queue Interpolation Binary filter Closing morphological transformation Contour detection Centroids Pre-processing Obstacle update and clean up Obstacle and object matching EfficientDet-Lite0 model Bounding boxes prediction and classification Features extraction Features fusion Obstacle queue Speech generation Output audio messages Messages queue Fig. 2: General overview of the image processing stages involved in the system, which consists of two main processing pipelines: the obstacle detection process (green), and the object detection process (blue). The objective is to integrate these functionalities into a wearable platform and to leverage the combined information to detect and classify objects and obstacles in real time. Specifically, in the context of this work, the term object refers to an identified element in the scene that could potentially become an obstacle. In contrast, the term obstacle refers to any surface within the captured scene that, even if not associated with a specific object, poses a risk because its proximity falls below a critical distance threshold relative to the user. 3.1 System overview The system was developed within a Python 3.7 virtual environment, utilizing key libraries such as NumPy [9], OpenCV [3], multiprocessing, TensorFlow [1], and PyRealSense2 [10], the latter being necessary for camera integration. As schematized in Fig. 2, the system consists of two main processing pipelines that form its core: – Obstacle detection process: by processing depth maps collected by the RGBD stereo camera, the system identifies areas in the scene that are at a critical distance, defined as below a certain threshold; – Object classification process: utilizing the EfficientDet-Lite0 model, the system detects objects in RGB frames and associates their position with that of the obstacles to provide a combined visualization and communication. 6 A. Ruo et al. The detailed execution flow of the proposed architecture is illustrated in Fig. 3. Specifically, the process begins with a main execution thread, which launches three parallel processes to handle data acquisition from the camera, obstacle detection, and object detection, respectively. 3.2 Stream reader process Given the computational demands of object detection and obstacle detection, which represent the primary bottlenecks in the pipeline, a multiprocessing-based approach was implemented. This design choice, preferred over multithreading, enables parallel execution of independent processes, optimizing performance, particularly in embedded environments. The stream reader process, shown in yellow in Fig. 3, corresponds to Algorithm 1, which is highlighted in red within the yellow block. This process is responsible for acquiring depth and RGB frames from the RealSense camera. Specifically, it initializes a pipeline to capture frames, processes the incoming data, and inserts the frames into two separate queues: the Depth queue for depth data and the RGB queue for RGB data. The process runs continuously until a stopEvent is triggered, as detailed in Algorithm 1. 3.3 Obstacle detection process The obstacle detection process, shown in green in Fig. 3, is responsible for realtime depth data processing and acts as a consumer of the Depth queue. It retrieves frames from the queue and applies pre-processing and contour extraction, followed by obstacle contours sorting and filtering. Through these operations, the process identifies a set of detected obstacles computing their corresponding distances. Pre-processing and contour extraction Given an RGB frame (Fig. 4a), a corresponding depth map can be derived, as illustrated in Fig. 4b. Considering a depth map as a two-dimensional matrix of numerical values representing the distance of projected points from the observation point, the pre-processing and contour extraction process aims to identify, sort, and filter the contours of obstacles in an image. As outlined in Algorithm 2, highlighted in red in Fig. 3 within the green block, the process begins with the generation of a binary mask by applying a depth threshold between 400 mm and 2500 mm (Fig. 4c). The lower bound corresponds to the camera’s minimum detection range, while the upper bound is empirically determined to define the region where obstacles are considered potentially dangerous. Next, morphological operations are applied to the mask to reduce noise and enhance image quality. Finally, contour detection is performed, and only the nmost significant obstacles (i.e., those with the largest contours) are retained. An example result is shown in Fig. 4d, where the extracted contour is highlighted in green. Title Suppressed Due to Excessive Length 7 Parallel independent processes LOOP - Object detection and pairing process Main process .run() .run() .run() Stream reader Obstacle detection Object detection Depth queue RGB queue Obstacle queue LOOP - Stream reader process(Alg.1) 1) .put(depth_frame) 2) .put(RGB_frame) LOOP - Obstacle detection process 1) .get() 2) obstacle = Obstacle(...) Update and create new obstacles(Alg.3, Alg.4) 3) obstacle_list.append(obstacle) 4) .put(obstacle_list) Obstacle Pre-processing and contour extraction process(Alg.2) 1) .get() 3) .get() 2) Update object locations Image pre-processing and object detection(Alg.5) 4) Detected objects and obstacle matching(Alg.6) Output:object position + distance +class OR Output: obstacle position + distance Fig. 3: Architecture execution flow diagram consisting of three parallel processes for handling data acquisition from the camera (yellow), obstacle detection (green), and object detection (blue). 8 A. Ruo et al. (a) (b) (c) down-center tmall.es EYo (d) Fig. 4: Output of the different stages of the obstacle detection process in an outdoor environment: (a) an RGB frame capturing a car; (b) the corresponding depth map; (c) the generated binary mask; and (d) the output of Algorithm 2 and Algorithm 3, where the detected contour is shown in green and the centroid in purple. Centroid computation After detecting obstacles, their centroids are computed using image moments to estimate their geometric centers. These centroids are then stored for tracking purposes. In particular, to ensure temporal consistency across frames, newly detected centroids are compared with previously identified obstacles following the procedure outlined in Algorithm 3, highlighted in red in Fig. 3 within the green block. The comparison is based on the following criteria: –Proximity matching: obstacles detected within a threshold of 50 pixels across consecutive frames are considered the same object, and their attributes (e.g., position and distance) are updated accordingly; –Merging overlapping detections: if a newly detected centroid falls within an extended radius proportional to an existing obstacle’s area, it is merged with that obstacle. Otherwise, it is registered as a new obstacle; –Filtering transient detections: only obstacles corresponding to newly detected centroids are retained, while transient or noise-induced detections are discarded. Specifically, the distance of each obstacle is calculated using the compute_distance method of the Obstacle class, as described in Algorithm 4. This method identifies the closest point within the obstacle and stores the minimum distance as an internal attribute when the obstacle is created or updated. Title Suppressed Due to Excessive Length 9 Algorithm 1: Stream reader process Require: Depth queue, RGB queue, stopEvent Ensure: Frames inserted into the queues 1begin 2pipeline ←New pipeline; 3config ←New configuration; 4pipelineWrapper ←PipelineWrapper(pipeline); 5Start(pipeline, config); 6while ¬IsSet(stopEvent)do 7frames ←WaitForFrames(pipeline); 8depthFrame, colorFrame ←GetFrames(frames); 9if depthFrame =NULL or colorFrame =NULL then 10 continue; 11 depthImage ←AsArray(GetData(depthFrame)); 12 colorImage ←AsArray(GetData(colorFrame)); 13 if IsFull(Depth queue)then 14 Get(Depth queue); 15 Put(Depth queue, depthImage); 16 if IsFull(RGB queue)then 17 Get(RGB queue); 18 Put(RGB queue, colorImage); 19 Stop(pipeline); Furthermore, based on the centroid’s position, each obstacle is assigned a label indicating its approximate location within the frame. The frame is divided into six regions, defined by the combination of two vertical sections (top and bottom) and three horizontal sections (left,center, and right). An example is shown in Fig. 4d, where the computed centroid is highlighted in purple. This centroid computation approach enhances robustness by ensuring that only persistent obstacles are tracked over time, reducing the impact of spurious detections. Algorithm 2: Pre-processing and contour extraction process Require: Image frame, n Ensure: n sorted and filtered contours by ContourArea 1begin 2maskObstacle ←InRange(frame, 400, 2000); 3closedMask ←MorphologyEx(maskObstacle, MORPHCLOSE, (7, 7)); 4contours ←FindContours(closedMask, ...); 5sortedContours ←Sort(contours, key ←ContourArea, reverse); 6filteredContours ←sortedContours[:n]; 7return filteredContours; 16 A. Ruo et al. 12. Kok, K.Y., Rajendran, P.: A review on stereo vision algorithm: Challenges and solutions. ECTI Transactions on Comput. and Inf. Technol. (2019) 13. Nguyen, V., Kim, H., Jun, S., Boo, K.: A study on real-time detection method of lane and vehicle for lane change assistant system using vision system on highway. Eng. science and technol., an Int. journal (2018) 14. Ni, D., Song, A., Tian, L., Xu, X., Chen, D.: A walking assistant robotic system for the visually impaired based on computer vision and tactile perception. Int. Journal of Social Robot. (2015) 15. Ruo, A., Sabattini, L., Villani, V.: Follow me: an architecture for user identification and social navigation with a mobile robot. In: Eur. Robot. Forum. Springer (2024) 16. Saba, T.: Computer vision for microscopic skin cancer diagnosis using handcrafted and non-handcrafted features. Microscopy Res. and Tech. (2021) 17. Shahira, K., Tripathy, S., Lijiya, A.: Obstacle detection, depth estimation and warning system for visually impaired people. In: TENCON IEEE Region 10 Conf. IEEE (2019) 18. Singh, K.J., Kapoor, D.S., Thakur, K., Sharma, A., Gao, X.: Computer-vision based object detection and recognition for service robot in indoor environment. Comput., Mater. & Continua (2022) 19. Tan, M., Le, Q.: Efficientnet: Rethinking model scaling for convolutional neural networks. In: Int. Conf. on Mach. Learning. PMLR (2019) 20. Tan, M., Pang, R., Le, Q.: Efficientdet: Scalable and efficient object detection. In: IEEE/CVF Conf. on comput. vision and pattern recog. (2020) 21. Yang, K., Wang, K., Lin, S., Bai, J., Bergasa, L., Arroyo, R.: Long-range traversability awareness and low-lying obstacle negotiation with realsense for the visually impaired. In: 1st Int. Conf. on Inf. Science and Syst. (2018) 22. Zou, Z., Chen, K., Shi, Z., Guo, Y., Ye, J.: Object detection in 20 years: A survey. Proceedings of the IEEE (2023)