Monocular visual SLAM, visual odometry, and structure from motion methods applied to 3D reconstruction: A comprehensive survey

Review article Monocular visual SLAM, visual odometry, and structure from motion methods applied to 3D reconstruction: A comprehensive survey Erick P. Herrera-Granda a , b , c , 1 , * , Juan C. Torres-Cantero b , Diego H. Peluffo-Ord´ o˜ nez c , d a Department of Mathematics, Escuela Polit´ ecnica Nacional, Ladr´ on de Guevara E11-235, Quito, 170525, Ecuador b Virtual Reality Laboratory, ETSIIT, Department of Computer Languages and Systems, University of Granada, c/Periodista Manuel Saucedo Aranda, s/n, 18071, Granada, Spain c SDAS Research Group, Ben Guerir, 43150, Morocco d College of Computing, Mohammed VI Polytechnic University, Lot 660, Hay Moulay Rachid Ben Guerir, 43150, Morocco ARTICLE INFO Keywords: Monocular SLAM Monocular visual odometry Monocular structure from motion Pure visual 3D reconstruction Monocular RGB three-dimensional reconstruction ABSTRACT Monocular Simultaneous Localization and Mapping (SLAM), Visual Odometry (VO), and Structure from Motion (SFM) are techniques that have emerged recently to address the problem of reconstructing objects or environments using monocular cameras. Monocular pure visual techniques have become attractive solutions for 3D reconstruction tasks due to their affordability, lightweight, easy deployment, good outdoor performance, and availability in most handheld devices without requiring additional input devices. In this work, we comprehensively overview the SLAM, VO, and SFM solutions for the 3D reconstruction problem that uses a monocular RGB camera as the only source of information to gather basic knowledge of this ill-posed problem and classify the existing techniques following a taxonomy. To achieve this goal, we extended the existing taxonomy to cover all the current classifications in the literature, comprising classic, machine learning, direct, indirect, dense, and sparse methods. We performed a detailed overview of 42 methods, considering 18 classic and 24 machine learning methods according to the ten categories defined in our extended taxonomy, comprehensively systematizing their algorithms and providing their basic formulations. Relevant information about each algorithm was summarized in nine criteria for classic methods and eleven criteria for machine learning methods to provide the reader with decision components to implement, select or design a 3D reconstruction system. Finally, an analysis of the temporal evolution of each category was performed, which determined that the classical-sparse-indirect and classical-dense-indirect categories have been the most accepted solutions to the monocular 3D reconstruction problem over the last 18 years. * Corresponding author. Department of Mathematics, Escuela Polit´ ecnica Nacional, Ladr´ on de Guevara E11-235, Quito, 170525, Ecuador. E-mail addresses: [email protected], [email protected] (E.P. Herrera-Granda), [email protected] (J.C. Torres-Cantero), [email protected], [email protected] (D.H. Peluffo-Ord´ o˜ nez). 1 The author’s affiliation at the time the article was submitted was: Unidad de Educaci´ on en Línea, Universidad de Otavalo, Av. de los Sarances y Pendoneros, Otavalo 100202, Ecuador. Affiliation changed to a since 2024. Contents lists available at ScienceDirect Heliyon journal homepage: www.cell.com/heliyon https://doi.org/10.1016/j.heliyon.2024.e37356 Received 13 June 2023; Received in revised form 29 August 2024; Accepted 2 September 2024 Heliyon 10 (2024) e37356 Available online 6 September 2024 2405-8440/© 2024 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY license ( http://creativecommons.org/licenses/by/4.0/ ). 1. Introduction In recent years, computer vision has witnessed a significant surge in the development of technologies focused on 3D reconstruction using monocular cameras. This growing interest is primarily driven by the widespread availability of affordable and lightweight monocular cameras commonly found in handheld devices. Despite their simplicity, these devices have shown remarkable capability in outdoor performance for 3D reconstruction tasks, attracting considerable attention from the research community [1]. The application of monocular 3D reconstruction spans a diverse range of fields. In robotics, it has contributed to scene understanding, allowing the advancement of autonomous driving and aiding in unmanned aerial vehicle navigation [2–9]. The augmented reality sector has also benefited, particularly in scene reconstruction and body mapping [2,7–13]. The film industry has utilized these techniques for facial puppetry and reenactment [1,14,15], and the medical field has seen advancements in the 3D reconstruction of surgical cavities and virtual endoscopy [16–19]. Thus, several solutions have been proposed to address the 3D reconstruction problem, notably monocular Simultaneous Localization and Mapping (SLAM), Visual Odometry (VO), and Structure from Motion (SFM). Each technique brings a unique perspective and methodology to the challenge of reconstructing environments and objects using monocular cameras. VO is a discipline focused on ego-motion estimation, developed over the need to determine the robot’s position and orientation using camera images, where 3D reconstruction is commonly used as a map-generating step from where vehicle location is estimated. There exists a wide range of solutions that can be used to perform odometry tasks, like wheel odometry, global position systems (GPS), global navigation satellite systems (GNSS), inertial navigation systems (INS), laser sensors, ultrasonic sensors, and VO [20]. Wheel odometry typically uses encoders, which are low-cost solutions; however, it produces position drift due to wheel slippage. INS are used to calculate position and orientation over three axes through accelerometers and gyroscopes, as the position is obtained by solving second-order integrals causing position drift due to minor errors in acceleration that can produce significant errors in position. GPS/GNSS obtain position information by trilateration of information obtained from three or more satellites, avoiding error accumulation over time; however, position measures obtained by GPS/GNSS can present meter-range errors and cannot be used for indoor or underwater applications. Similarly, laser and ultrasonic sensors provide scalar distance measurements from the sensor to the target using time-of-flight or phase shift principles, but they also present reflection problems depending on the material or orientation of the target surface. In contrast, VO is a low-cost solution more accurate than GPS, INS, ultrasonic sensors, and wheel odometry due to its low position error range from 0.1 to 2 % [21]. As an odometry solution, VO is characterized by a satisfactory balance between cost, reliability, and implementation complexity [22]. Furthermore, studies like [23,24] established that VO obtains better results by generating geometric maps of the environment from which the robot could be localized. SLAM is the ability developed for a mobile robot to start in an unknown environment and incrementally build a map using the information obtained from camera observations and simultaneously compute the estimated trajectory using single or multiple cameras [25,26]. Moreover, SFM algorithms have been referred to as Monocular SLAM [27] due to their similarity in obtaining a geometric representation by generating and tracking sparse or dense world models using direct or indirect techniques. The primary distinction between Visual Odometry (VO) and Simultaneous Localization and Mapping (SLAM) methods lies in the characteristic that in VO, the points are typically not reused once they exit the field of view. In contrast, SLAM typically enables the recycling of previously triangulated points as they reenter the field of view in loop-closure steps [20]. Another major difference between these disciplines is that SLAM performs both tasks simultaneously, generating a map of the environment and using this map to estimate the camera pose. SFM focuses more on generating a scene map, while VO focuses more on estimating the pose, also called ego-motion (translation and orientation of an agent) [21]. It must be mentioned that SLAM methods consist of larger systems developed to obtain a geometric representation and provide a sense of orientation (typically as a camera pose), trajectory, and loop closure, among others. In SLAM systems, this information is used for navigation tasks and scene comprehension, being complex systems, and sometimes, this information is employed for geometric map optimization tasks. In addition, the development of multiple SLAM systems was inspired by VO and SFM formulations [3,28–30] extending their pipelines to estimate ego-motion and geometry simultaneously. Thus, irrespectively of the denomination that may be given to every SLAM, VO, and SFM approach, all of them hold the common goal of both 3D scene reconstruction and camera pose estimation using imaging sensors, but with further specialization in the main task for which each was designed [2,20,24]. Despite their strengths, these methodologies face several limitations. Monocular pure visual 3D reconstruction involves some critical and well-known problems related to the lack of scale information and the computational complexity of recovering 3D models from sequences of monocular images without geometric priors. Nowadays, available monocular methods still lag behind controlled multi-view setups in quality, accuracy, and completeness [1]. Simplifications in real-world image formation processes, such as assuming low-frequency, distant illumination and neglecting shadows and global illumination effects, limit the accuracy of reconstructions. Additionally, a significant challenge arises in the presence of purely rotational movements in the absence of a stereoscopic baseline, which complicates the accurate capture and modelling of three-dimensional structures, as the lack of a baseline hinders the effective differentiation of depth and spatial relationships. Another well-identified challenge across SLAM, VO, and SFM is the absence of a comprehensive taxonomy that categorizes the vast array of existing solutions. This lack of classification and joint analysis poses a significant barrier to advancing the field. Moreover, each discipline has its own set of limitations. For instance, while producing superior 3D reconstruction results, SLAM systems often struggle with long-term operation and large loop closures, especially in dynamic environments [31]. VO methods, on the other hand, are sensitive to varying operating conditions, such as lighting and textures, affecting their robustness [20]. SFM techniques, while excelling in structure recovery, do not fully leverage the advantages of simultaneous pose and geometry recovery [32]. To address these challenges, our research conducted a comprehensive overview of the SLAM, VO, and SFM solutions for monocular E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 2 3D reconstruction, with a particular focus on those based on a monocular RGB camera as the sole source of information. In contrast to previous surveys like [20,31,33–36], our work extends the existing taxonomy, including a broader range of classifications, such as classic, machine learning, direct, indirect, dense, and sparse methods, which allows to cover multiple new proposals like hybrid, classic, and machine-learning based systems and their variants. This new taxonomy represents a significant advancement in organizing and understanding the field, considering techniques’ rapid evolution and diversity. Our detailed review of 42 methods is more extensive than most existing surveys, covering classic and machine-learning approaches. We provide systematic categorization and essential formulations for implementing, selecting, or designing a 3D reconstruction system. Furthermore, our work includes a time citation evolution analysis, a novel aspect not commonly found in other reviews, offering insights into the acceptance and evolution of these methods over the past 18 years. The here-presented analysis and comparative study provide insights into algorithm selection, which makes this work a valuable resource for researchers and practitioners. As a notable contribution, our review work presents a more exhaustive effort in categorising these methods compared to previous works because it comprehensively covers the main 42 monocular methods, allowing to gather their foundations, formulations, algorithm design, and extract categorical selection criteria. For instance, we not only focus on SLAM, VO, and SFM individually but also explore their interrelations and implications for monocular 3D reconstruction. This holistic view is absent in many existing surveys, which often concentrate on isolated aspects of the problem. This way, our research extends beyond existing literature by providing an updated, detailed, and comprehensive overview that integrates various aspects of monocular 3D reconstruction. Our novel taxonomy, extensive methodological review, and time citation analysis constitute the main contributions, setting our work apart from existing surveys and reviews in the field. This work was specifically undertaken to delve into the methodologies for creating 3D geometric representations (reconstructions) of an environment using a single moving monocular RGB camera. As such, the investigation excluded techniques beyond the scope of this study, such as Visual-Inertial (utilizing Inertial Navigation Systems for enhanced tracking), RGB-D (employing additional sensors to capture depth information from the surroundings), omnidirectional, and stereo techniques. 2. Related works In the literature, since monocular 3D reconstruction is an ill-posed problem that can be solved by combining different techniques and algorithms from disciplines like computer vision, robotics, and machine learning, only a few works can currently be related to this overview. In 2016, Aqel et al. [20] developed a comprehensive review of visual odometry types, approaches, challenges, and applications. In this work, authors described the different input modalities and motivations for studying the VO problem, so a first taxonomy was established to define the difference between featureand appearance-based approaches. The differentiating factor with our approach is that we covered SLAM, VO, and SFM problems. In contrast, only the VO problem was explored in the work mentioned above, not focusing on the monocular input mode. In 2021, Servi` eres et al. [31] proposed a detailed review of the state-of-the-art classification and experimental benchmarking of visual and visual-inertial SLAM techniques, detailing the basic structure of SLAM system, SLAM evolution problem beginning from its appearance, then introducing the visual SLAM and visual-inertial SLAM, addressed using direct and indirect classification. In addition, an experimental benchmark was carried out using DSO [24], LSD-SLAM [37], ORB-SLAM2 [38], ROVIO [39], and Vins-Mono [40] systems. In contrast to Ref. [31], we are not only focused on the SLAM approach but are especially interested in 3D reconstruction, so SLAM, VO, and SFM are suitable for our work. In addition, we are not only interested in the description and historical appearance of each method; instead, we are primarily focused on providing an appropriate taxonomy and describing each of the most important algorithms to give the reader an appropriate overview that can help them to correctly select the most appropriate method for projects and research. In the survey articles category, a few works address the pure visual monocular approach. In the Taketomi et al. study [33], a survey focused on the 2010 to 2016 period, developed specifically for the visual SLAM (vSLAM) approach. This study described the main elements of the vSLAM problem and adopted the feature-based and direct classification to briefly overview the main SLAM systems from 2010 to 2016. In this way, MonoSLAM [28], PTAM [41], and ORB-SLAM [11] were addressed as feature-based techniques, whereas DTAM [27], LSD-SLAM [3], SVO [23], and DSO [24] were described as direct formulations. In addition, the Taketomi et al. survey detailed the KinectFusion [42], Dense visual SLAM [43,44], Elastic Fusion [44], and SLAM++ [45] as RGB-D techniques. Another important survey that might be considered is the work of [36], which presents a survey in monocular SLAM algorithms for outdoor applications, mainly focused on natural environments requiring long-range capable techniques that are not affected by sunlight and can manage textured images with plenty of vegetation. The authors selected the most prominent algorithms available until 2018, so DSO [24], ORB-SLAM [11], and LSD-SLAM [3] were selected. These algorithms were reviewed and compared using the RMSE of the translational error over eight sequences recorded by the authors. Results evidenced that DSO was the method that performed better for outdoor applications. However, there was a lot of work that could be done to enhance reconstruction and tracking quality. In addition, the work [34] presents a survey of the main visual SLAM systems, focused on the methods that made considerable contributions before 2018. The survey summarizes the SLAM history, its relationship with the SFM and VO problems, the SLAM classification, the key issues, and improvement trends. The study adopted feature-based and direct classification, where many existing SLAM systems were mentioned. However, the authors reviewed the PTAM [41] and ORB-SLAM [11] as feature-based approaches and the DTAM [27], SVO [23], LSD-SLAM [3], and DSO [24] as direct approaches. One of the most recent surveys performed for monocular pure visual systems is the study of [46], which mainly focused on the monocular visual odometry problem, where authors described the VO problem, its formulation, and its similarities with the most recent SLAM proposals. In addition, the authors overviewed the MonoSLAM, PTAM, DTAM, KinectFusion, DVO, SVO, LSD-SLAM, DSO, and eight additional RGB-D methods. In contrast with the mentioned surveys, we performed a comprehensive and consistent survey of 42 algorithms, not only focused on a type of problem E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 3 (SLAM, SFM, or VO) or a specific category of the classification. Instead, we worked with a completely extended classification considering all the monocular pure visual systems contributing to the monocular 3D reconstruction problem. The work that we can refer to as closest to our approach is the article developed by Macario et al. [35], which is a comprehensive survey developed for visual SLAM algorithms where authors provide the reader with a set of initial concepts, a taxonomy based on direct and indirect classification, a review of eight visual-only SLAM methods, six visual-inertial SLAM methods, five RGB-D SLAM methods, and finally the discussion of open problems and open directions. The work was comprehensively explained, and each algorithm addressed was adequately described and systematized. We followed a similar direction, focusing our investigation on the monocular input mode, so we further analyzed this modality, properly defining and applying a taxonomy that better characterizes each algorithm. In addition, 42 of the most representative algorithms available are analyzed in depth, including an exhaustive overview of the newest Machine Learning-based (ML) approaches that had not been discussed in any of the cited related works. 3. Contributions and outline To the best of our knowledge, this is the first overview specialized in the monocular 3D reconstruction problem that considers SLAM, VO, and SFM solutions, integrating them into a complete taxonomy. To summarize, the key contributions of our work are: 1) A taxonomy designed to contain all possible current existing literature approaches built considering three classifications and all possible combinations defining ten categories overviewed in detail. 2) A comprehensive overview of the most representative 42 monocular SLAM, VO, and SFM algorithms comprised by 18 classic monocular methods and 24 methods that integrate machine learning techniques. Each examined method includes its algorithm systematization, including mathematic principles for those algorithms that made innovations in their formulations, especially regarding depth map estimation or optimization, since this work is focused on 3D reconstruction. 3) We defined 11 criteria to provide the reader with the components to implement, select or design a 3D reconstruction system, nine applicable to classic systems and two additional criteria only applicable to machine learning approaches. Information was gathered for each algorithm and is presented in Tables 2 and 3, where the criteria are: type of algorithm (SLAM, VO, or SFM), tracking method (direct vs. indirect), map density (dense vs. sparse), pixels used (the technique to extract pixel information), estimation method (depth map estimation technique), global optimization, relocalization, loop closure (whether or not the algorithm includes optimization, relocalization or loop closure steps), availability (open source repository where the algorithm is available), CNN architecture (commonly known employed CNN architecture) and the main tasks for which a CNN was employed. 4) A discussion of open questions, available solutions, future directions for each classification and temporal analysis of the citation scores achieved by each taxonomy category give the reader an intuition of the impact and acceptance that each classification, method and category has generated in this research field. This paper is organized as follows: Section 4 discusses input modalities, section 5 introduces the basics, notation, and taxonomy, section 6 presents an overview of the most representative classic methods, section 7 introduces an overview of the most representative methods that integrate machine learning, section 8 provides a discussion made over the entire taxonomy. Finally, conclusions are presented in section 9. Table 1 Input modalities used in 3D reconstruction. Type of camera Pros Cons Stereo Depth information and image scale are computed instantaneously. Depth information is easily obtained Provide 3D information Require more calibration effort and are more expensive than monocular ones. Difficult shutter synchronization It gets degraded to a monocular device when the stereo baseline is smaller than the camera-to-object distances. Omnidirectional Present a larger FOV close to 360◦, and images provide more information. Features remain longer in the image, helping to obtain well-refined models. More expensive than the rest Not compatible with mobile devices Some may not work in dynamic environments. Present distortions that come from the equirectangular representation Monocular RGB-D Provide depth measurements with each image in realtime rates Easy deployment Compatible with mobile devices Suitable for small robotics and indoor environments Produce erroneous measurements under sunlight. Limited by the range of the active sensor and the size of the projected pattern More expensive than monocular RGB sensors Monocular RGB Lowest cost Ubiquitous Easy deployment Available on most mobile devices Simple calibration Not limited by sensor range Suitable for small robotics in indoor and outdoor environments Can work under sunlight Presents image scale uncertainty. Does not provide depth measurements. Reconstruction tasks may consume more computational resources. E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 4 4. Input modalities The goal of obtaining a 3D geometric representation of the scene is a complex task that can be achieved using camera sensors. In the past, some state-of-the-art systems were built using complex camera arrays and lightning setups, primarily for indoor applications. Nowadays, different capture devices range from costly multi-view and stereo setups to cheap monocular sensors. Next, a brief overview of several input modalities used for 3D reconstruction is introduced. 4.1. Stereo setups Multi-view setups consist of an array or a set of pairwise stereo cameras distributed in a configuration that allows the capture of simultaneous views of the same object at the same time [1]. Remarkably, stereo systems that use a pair of cameras are called stereo systems [20]. Binocular cameras have two separate camera sensors, so in-depth information and image scale can be computed instantaneously by triangulation due to their fixed and known stereo baseline size. However, these cameras are generally more expensive than monocular sensors and require more calibration effort. In addition, these sensors must acquire images at the same time interval, which can be achieved by synchronizing shutter speed using an external trigger signal [47]. Generally, maintaining a calibrated constant baseline between two cameras requires much more effort than in the monocular case stereo as setups are degraded to monocular when the baseline is much smaller than the distance from the scene to the camera [48]. Hence, they are limited to working in small and indoor environments. Table 2 Input modalities used in 3D reconstruction. Method SLAM, VO or SFM Tracking method Map density Pixels used Estimation Global optimization Relocalization Loop closure Availability Jin et al. (2000) [58] SFM Featurebased Sparse Hi.grad. EKF – – – – MonoSLAM (2007) [28] SLAM Featurebased Sparse Shi Tomasi EKF – – – [113] PTAM (2007) [71] SLAM Featurebased Sparse Hi.grad. BA –[114] OpenMVG (2013) [73] SFM Featurebased Sparse Hi.grad. BA –[115] ORB-SLAM (2015) [11] SLAM Featurebased Sparse Hi.grad. Local BA [116] COLMAP (2016) [77] SFM Featurebased Sparse PnP matches Local and Global BA – – [117] ORB-SLAM2 (2017) [38] SLAM Featurebased Sparse Hi.grad. Local BA [118] ORB-SLAM3 (2021) [81] SLAM Featurebased Sparse Hi.grad. Local BA [119] Valgaerts et al. (2011) [85] SFM Optical flow Dense 8-point matches Robust 8-point algorithm – – – – Ranftl et al. (2016) [88] SFM Optical flow Dense FlowFields Superpixel graph minimization – – – – Stühmer et al. (2010) [91] SLAM Direct Dense Hi.grad. Cost volume refinement – – DTAM (2011) [27] SLAM Direct Dense Hi.grad. Cost volume refinement –[120] REMODE (2014) [4] SLAM Direct Dense Hi.grad. Bayesian estimation – – – [121] LSD-SLAM (2014) [37] SLAM Direct SemiDense Edgelets Pose graph optimization –[122] DSO (2017) [24] VO Direct Sparse Hi.grad. Local BA – – – [123] LDSO (2018) [64] VO Direct Sparse Hi.grad. Local BA GPGO –[124] DSM (2020) [30] SLAM Direct Sparse Hi. grad. Photometric BA – – [125] SVO (2014) [23] VO Hybrid Sparse FAST +Hi. grad. Local BA – – – [126] Hi.grad. is used to abbreviate a set of pixels with high-intensity gradient. EKF is used to abbreviate the Extended Kalman Filter technique. BA is used to abbreviate the Bundle Adjustment technique. GPGO is used to abbreviate the Global Pose Graph Optimization technique. E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 5 Table 3 Summary of the most representative ml monocular SLAM, VO, and SFM systems. Method SLAM, VO or SFM Tracking method Map density Pixels used Estimation CNN architecture CNN’s main estimation tasks Global optimization Relocalization Loop closure Availability DynaSLAM (2018) [102] SLAM Featurebased Sparse Hi.grad. Local BA Mask R-CNN Instance segmentation [192] BA-Net (2019) [59] SFM Featurebased Sparse Hi.grad. BA DRN-54 Depth Damping factor – – – [138] Steenbeek et al. (2022) [6] SLAM Featurebased Sparse Hi.grad. BA ResNet-50 Enc.dec. Scale Depth map densify [193] Sun et al. (2022) [60] SLAM Featurebased Sparse Hi.grad. BA ResNetXt-50 Enc.dec. Scale Relative depth Depth – Lee et al. (2022) [159] SLAM Featurebased Sparse Hi.grad. BA Enc. dec. Scale Semantic segmentation Feature refinement – SVR-Net (2023) [161] SLAM Featurebased Sparse Learned features Optimal match recurrent network ScanNet Local map Relative pose TSDF values – – – DeMoN (2017) [141] SFM Optical flow Dense SIFT keypoints matching FlowFields 8-point algorithm RANSAC Chain Enc.dec. Optical flow Depth Pose Surface normals – – – [108] DeepV2D (2020) [144] SLAM Optical flow Featurebased Dense Learned features 3D Stereo matching over cost volumes Residual Flow Hourglass Enc.dec. Depth Pose 3D stereo matching – – [106] VOLDOR (2020) [163] VO Optical flow residuals Dense Learned features Generalized ExpectationMaximization PWC-Net Optical flow – – – [104] DROID-SLAM (2021) [167] SLAM Optical flow Dense Learned features Between keyframes edges BA Residual blocks Feature extraction Optical flow Estate estimation –[194] SDF-SLAM [146] SLAM Featurebased Dense Learned features and descriptors BA Enc.dec. Feature and descriptor extraction Semantic segmentation – NeRF-SLAM (2022) [169] SLAM Optical-flow Dense Learned features Between keyframes edges BA Radiance field optimization Residual blocks Neural Radiance Fields Feature extraction Optical flow Estate estimation –[195] Rosinol et al. (2023) [170] SLAM Optical-flow Dense Learned features Between keyframes edges BA Probabilistic volumetric fusion Residual blocks Feature extraction Optical flow Estate estimation – – CNN-SLAM (2017) [2] SLAM Direct Semidense Hi.grad. Pose Graph optimization ResNet-50 FCN Depth Semantic segmentation –[105], a DeepTAM (2018) [171] SLAM Direct Optical flow Dense Hi. grad. Cost volume refinement Enc.dec. Pose hypotheses Optical flow Depth Depth refinement –[111] DeepFusion [174] SLAM Direct Semidense Hi. grad. Opt framework U-Net Log-depth gradients and uncertainties Scale – – – (continued on next page) E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 6 Table 3 (continued) Method SLAM, VO or SFM Tracking method Map density Pixels used Estimation CNN architecture CNN’s main estimation tasks Global optimization Relocalization Loop closure Availability CodeSLAM (2018) [134] SLAM Direct Dense Hi. grad. BA U-Net Variational Enc.dec. Code Compact depth – – [107], b DeepFactors (2020) [10] SLAM Direct Dense Hi. grad. Multiview BA U-Net Variational Enc.dec. Code Compact depth Uncertainty [112] DVSO (2018) [140] VO Direct Sparse Hi. grad. BA ResNet-50 Encoderdecoder Disparity maps – – – [110], c CNN-DVO (2020) [103] SLAM Direct Sparse Hi. grad. Dynamic upsampling and downsampling BA U-Net Encoderdecoder Depth –[191] D3VO (2020) [139] VO Direct Sparse Hi.grad. BA U-Net Encoderdecoder Depth, Pose, Uncertainty – – – – MonoRec (2021) [182] SFM Direct Sparse Hi. grad. Mask filter BA U-Net ResNet-18 features Encoderdecoder Depth Mask Moving objects – – – [109] DDSO (2022) [145] VO Direct Sparse Hi. grad. BA ResNet-50 Encoderdecoder Depth Pose Transformations – – – – CNN-SVO (2019) [66] VO Hybrid Sparse FAST +Hi.grad. Local BA ResNet-50 EncoderDecoder Depth – – – [196] Hi.grad. is used to abbreviate a set of pixels with a high-intensity gradient. Enc.dec. is used to abbreviate the Encoder-decoder CNN architecture. EKF is used to abbreviate the Extended Kalman Filter technique. BA is used to abbreviate the Bundle Adjustment technique. a Unofficial implementation of the CNN-SLAM method. There is no official implementation of this method yet. b Unofficial implementation of the Code-SLAM method. There is no official implementation of this method yet. c Unofficial implementation of the DVSO method. There is no official implementation of this method yet. E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 7 4.2. Omni-directional cameras Many authors decided to utilize omnidirectional cameras, like [49–52], due to their wide field of vision (FOV). According to Ref. [51], Omnidirectional cameras provide more information than regular cameras, and the features found in the images stay longer, helping to obtain well-refined 3D scenery models. However, Omnidirectional cameras are expensive, require a lot of setup effort, and are incompatible with mobile devices. Furthermore, some of these devices incrementally scan the scene using mechanical rotation, so these devices are intended to work in static conditions and may not work in dynamic environments [53]. One of the main problems these devices have for the 3D reconstruction task is the generation of distortions in the image that comes from the equirectangular representation, which happens when the obtained spherical pixels projected to a plane get significantly distorted, which can cause depth prediction errors [53]. 4.3. Monocular RGB-D RGB-D sensors have an additional active or passive sensor that allows the system to get depth measurements of the environment associated with each image pixel in real-time. Such depth measurements help solve the monocular reconstruction’s depth and scale ambiguity because they can be used as an estimate of the environment geometry. These devices can be classified as passive or active. In contrast to stereo sensors, passive RGB-D cameras typically present a projector instead of a second camera, projecting a pattern in the image to find coincident points. To prevent this pattern from being distinguished by the human eye, these sensors commonly work in the infrared domain (IR). RGB-D cameras using infrared projectors known as active. It must be mentioned that active RGB-D cameras can produce erroneous measurements under sunlight [2] due to sunlight IR radiation that overpowers the projector. For this reason, some RGB-D cameras present a combination of active and passive sensors that are activated whether under sunlight or not, which can significantly increase the cost of this kind of device. Another common limitation of these light devices is that they cannot reconstruct objects smaller than the projected pattern [1]. Time of Flight (ToF) is another active RGB-D camera that gets the depth measurements by emitting a light pulse and computing the time it takes to reach the object. This time measurement is challenging because this event occurs at the speed of light. That is why ToF RGB-D cameras typically perform worse than RGB-D light ones [1]. 4.4. Monocular RGB RGB cameras capture the intensity of the received light in three channels: red, green, and blue. RGB devices are designed to work in different configurations, ranging from CCD sensors acquiring each signal in a separate sensor to Bayer pattern-based sensors where colour filters are interleaved in front of a single sensor [1]. Monocular cameras are known for reducing the effect of calibration errors. They are ubiquitous, low cost, easy to deploy, and available in most portable devices, which is considered a significant incentive that has kept the attention of researchers, who usually prefer this type of input to perform reconstruction tasks involved in SLAM, SFM, and VO. Nevertheless, this input involves an ill-posed problem because monocular vision suffers from scale uncertainty [54,55], and the techniques used to achieve the goal of 3D reconstruction typically require a lot of computational resources. Table 1 summarizes the features and drawbacks of using the different input modalities for 3D reconstruction. As noticed in Table 1, although monocular RGB cameras present some important challenges due to their sensor less nature, not allowing them to provide direct depth measurements, they present the most attractive set of pros, especially for small robotics applications. Monocular RGB sensors present the lowest price among all the available camera types, are easy to deploy, and are compatible with almost all the existing portable devices and processors, like Single Board Computers (SBC) and Field Programmable Fig. 1. Bibliometric co-authorship analysis results were executed in VOSviewer using the association strength method, full-count strategy, and at least two documents per author. E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 8 Gate Arrays (FPGA), which makes this input modality especially attractive for researchers. This is why the research was developed, focusing on the monocular RGB input modality. 5. Basics and notation 5.1. Literature review process Scene3D reconstruction using monocular cameras is a complex challenge that has kept the interest of many researchers. However, there are few studies in this discipline, and as an ill-posed problem, each approach is considerably different from the rest, so the terminology could be used in varied ways. For that reason, the main objective of this paper was to establish a taxonomy and provide the reader with some basics about the discipline. In this way, we used the following search criteria in Scopus and Scholar Google, applied to the time range from 2000 to April of 2023, performing the search only for the English language: TITLE-ABS-KEY (("SLAM" OR "VO" OR "SFM" OR "Simultaneous Localization and Mapping" OR "Visual Odometry" OR "Structure from Motion")) AND ("Monocular" OR "Visual" OR "RGB") AND NOT ("RGB-D" OR "Stereo" OR "omnidirectional" OR "Visual Inertial" OR "VI")). After complete and independent reading, we set apart works that applied RGB-D, stereo, or omnidirectional cameras, obtaining a total of 137 studies focused on monocular RGB 3D reconstruction. The bibliometric information of each article was extracted using Scopus and Mendeley, so a .ris file was extracted, and its information was processed using VOSviewer software [56]. Bibliometric co-authorship analysis was set under selected bibliometric data, association strength method, full-count strategy, and a minimum of two documents to consider an author, so 64 authors met the threshold. Results obtained on bibliometric analysis are illustrated in Fig. 1. As can be seen, the bibliometric analysis helped us to find the authors with the greatest link strength based on their contribution to the state of the art, so the authors with the highest overall link strength were Cremers, D., Czarnowski, J., Davison, A., Clark, R. and Leuteneger, S., with link strengths of 28, 13, 12, 11 and 11 respectively. In this way, our literature review strategy was guided by an exhaustive reading, starting with the most representative authors, in order to gather basic knowledge. Then, we kept extending the bibliographic database according to the studies cited in each study as previous contributions and studies that were used for comparisons. 5.2. Notation Throughout the paper, bold lowercase letters (x) represent vectors for statements and formulations. Bold uppercase letters (R) represent matrixes. Scalars are represented by light lowercase letters (c). Functions and images are represented by upper-case light letters (I). Let us define image I containing a set of pixels. For each q pixel in the image, let us assume that there is a d depth value allowing for the projection of its corresponding 3D coordinates x= (x,y,z)T. In this way, camera poses are represented as transformation matrixes Ti∈SE(3), transforming a point from the real-world frame to the camera frame. R represents rotation matrixes, while Π and Π−1 are projection and back projection functions. Additionally, d∗represents inverse depth values, so D and D∗correspond to depth and inverse depth maps. 5.3. Initial words and approaches One of the main motivations of visual SLAM, SFM, and VO is Camera pose estimation, which initially used to be approached in three ways: using a feature-based approach, an appearance-based approach, or by a hybrid featureand appearance-based approach [20,21, 51]. Feature-based approach, as implemented in Refs. [10,11,38,57–59], typically performs image feature extraction (corner, lines, curves, among others), representative features matching, and motion estimation tasks. Matching is performed using the Euclidean distance of the feature vectors of both images to find matching candidates. In this way, given two images of the same scene at different poses, the first image features are matched with their corresponding features of the second image, thus providing a 3D position of the points related to those features, as depicted in Fig. 2. Motion is typically estimated by observing the feature displacement, where the Fig. 2. Feature matching and 3D position triangulation from multiple views. E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 9 combined and improved allowed the creation of a new system whose contributions include the use of the same tracking, mapping, relocalization, and loop closing features in real-time operation and satisfactory invariance to changes in viewpoint and illumination; use of a local co-visibility graph for tracking and mapping providing map size independence allowing real-time operation in large sceneries; use of a loop closing technique based on pose graph optimization; the ability to recover from tracking failure; a model selection based automatic initialization; and the implementation of map point and keyframe selection by the survival of the fittest technique. Similarly, PTAM ORB-SLAM uses three parallel threads for tracking, local mapping, and loop closing where the tracking thread performs the task of localizing the camera with every frame and decides when a keyframe should be inserted; the local mapping thread processes each new keyframe and executes bundle adjustment for 3D reconstruction of elements near to camera pose; and loop closing thread in charge of searching loops with each new keyframe and computing a similarity transformation measuring accumulated drift when a loop is detected. ORB-SLAM also implemented a non-prohibitive policy for spawning and culling keyframes for flexible map expansion, which can detect and delete redundant keyframes. Results showed that the main benefit of indirect techniques is that they can match features even with a wide baseline. According to Ref. [11], accuracy can still be improved by including points at infinity in the tracking containing primary rotation information. Also, it can be upgraded by using a denser map, or the system could be used as a backbone from which an accurate dense map may be built. Fig. 11 presents the ORB-SLAM algorithm inspired by the article [11]. COLMAP (2016). COLMAP is a method proposed for photogrammetry and computer vision, which aims to reconstruct a 3D scene from a set of 2D images. The main strategy of the method is to estimate camera poses and 3D points in the scene by optimizing a bundle adjustment problem. COLMAP employs feature-based matching to establish correspondences among images and then refines the matches based on geometric consistency to reject outliers. The method also introduces several improvements in the optimization process, such as a novel parameterization for rotation and a more robust optimization technique based on the Levenberg-Marquardt algorithm. SFM techniques are typically built over two stages: correspondence search and incremental reconstruction. The correspondence search is responsible for identifying overlaps in the input images and projections of the same points in such overlapping images to build a graph of image projections for each point. For this purpose, COLMAP performs feature extraction to select sets of local features invariant to radiometric and geometric changes to allow SFM to recognize them across multiple images; then, matching is executed to discover images viewing the same scene part by searching feature correspondences that are the most similar features in each image; finally, a geometric verification step must be addressed because matching is only based in appearance, and it is not guaranteed that feature correspondences are mapping the same scene point, so SFM verifies this matches by estimating transformations and if a transformation maps a sufficient number of features it is considered as verified. The second stage in COLMAP is incremental reconstruction, which takes the scene graph from the correspondence search to recover pose estimates and the scene structure as a point cloud. The incremental reconstruction starts with the initialization process that carefully selects an initial two-view reconstruction, which is crucial to prevent trajectory loss issues. Then, the image registration allows the system to register new images by solving the Perspective-n-Point (PnP) problem using the previously identified feature correspondences of triangulated points; then, the triangulation allows the extension of the scene point cloud representation by adding a set of points coming from each new image. Finally, since SFM tends to drift quickly to non-recoverable states, BA is applied to refine the camera and point parameters by minimizing the reprojection error. The main contributions of COLMAP to the SFM pipeline are incorporating improvements in image registration, triangulation, and BA procedures. COLMAP introduces a robust next-best image selection method to improve pose estimation and recover a reliable triangulation, which is an uncertainty-driven approach that uses a multi-resolution analysis by using a score S,which is higher if more points are visible and if the distribution of those points is uniform. In addition, COLMAP introduces a novel, robust, efficient triangulation procedure that aims to be more robust to outliers and recover independent points merged into one track by using the RANSAC approach to handle different levels of outlier contamination in the multi-view triangulation process. Finally, in COLMAP, BA was implemented for image registration and triangulation, allowing a local BA to be performed on the set of Fig. 12. Diagram of COLMAP algorithm. Adapted from Ref. [77]. E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 16 connected images for each image registration, and it performs a global BA each time the model grows a certain percentage. Compared to similar methods, COLMAP shows better performance in terms of accuracy, efficiency and scalability. For example, COLMAP can handle large image collections with hundreds of thousands of images while maintaining reconstruction quality without the limitations of handling large datasets. Moreover, COLMAP shows superior accuracy in reconstructing the scene’s geometry, which is important for virtual and augmented reality applications. Another advantage of COLMAP is its flexibility in handling different types of input data, such as unordered image sets, image sequences, and video frames. These features make COLMAP suitable for various applications, from 3D modelling of cultural heritage to robotic vision and autonomous driving. Additionally, COLMAP provides a comprehensive set of tools for visualizing and analyzing the reconstruction results, including point cloud rendering, texture mapping, and error analysis. Fig. 12 presents the COLMAP algorithm inspired by the article [77]. ORB-SLAM2 (2017). Next, Mur. Artal and Tard´ os continued the work of ORB-SLAM, developing ORB-SLAM2 [38], a system that extended the capabilities of its predecessor, allowing the system to operate with monocular, stereo, and RGB-D sensors. It was mainly developed based on the previous system but included stereo matching procedures for stereo cameras and stereo coordinate generation for RGB-D sensors. The main innovation in the monocular scenario was implementing a fourth thread responsible for performing a full Bundle Adjustment after loop closure pose graph optimization to compute the optimal structure and motion solution. This optimization procedure implies high computational cost because it is performed over all the points and features, so it is executed in a separate thread, allowing the system to continue expanding the map and detecting loops simultaneously. Like ORB-SLAM, this system incorporated an embedded DBoW2 place recognition module for relocalization and a co-visibility graph for large environments. For stereo and RGB-D implementation, monocular key points are also used to contribute rotation and translation estimation but do not provide scale information because this prior information can be measured or triangulated by the sensor. This method uses Levenberg-Marquard optimization by the implemented module g2o, optimizing the camera pose in the tracking thread, the local window of keyframes, points in the mapping thread, and all keyframes and points after loop closure. Therefore, full Bundle Adjustment is the case of local Bundle Adjustment, where all map points and keyframes are optimized except the origin keyframe. ORB-SLAM2 also includes a localization mode that disables local mapping and loop closing threads for known areas as long as there are no significant changes in the landscape, allowing for long-term and lightweight localization and functionality. This method was compared with many of the existent Stereo, RGB-D, and monocular systems of the time, outperforming most of them in the EuRoC [78], TUM [79], and KITTI [7] datasets, proving its functionality in a large variety of environments. In addition, this study proposes that its application could be extended to a wide variety of new sensors like omnidirectional, fisheye cameras, and large-scale dense fusion. Fig. 13 illustrates the ORB-SLAM2 algorithm inspired by the article [38]. ORB-SLAM3 (2021). It is well known that ORB-SLAM has been the gold standard of feature-based monocular SLAM systems for many years and has inspired the development of many other systems like DF-ORB-SLAM, ORB-SLAM-VI and ORB-SLAM3 [80–82]. ORB-SLAM3, proposed by Campos et al. [81], is the latest proposal to integrate and improve the developments of its predecessors. This system can perform visual, visual-inertial, and multimap SLAM with monocular, stereo, and RGB-D cameras for pinhole and fisheye devices. One of the most considerable contributions made in this work is its capability of performing four types of data association: short-term, enabling the system to match map elements of the last few seconds; mid-term data association, to match map elements close to the camera that present slight accumulated drift; long-term data association, to match map elements of previously visited areas using a place recognition technique regardless of accumulated drift; finally, multimap data association, to match and use BA map elements from previous mapping sessions creating a map from where the system can perform an accurate localization. The main Fig. 13. Diagram of ORB-SLAM2 algorithm. Adapted from Ref. [38]. E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 17 contributions of this method compared to previous versions are: a monocular and stereo visual-inertial system based on Maximum-a-Posteriori estimation (MAP), improving ORB-VI using the initialization technique of [83]; a new place recognition technique to improve its recall by checking geometrical and local keyframes consistency with three co-visible map keyframes, slightly incrementing computational cost; ORB-SLAM atlas, which is a multimap SLAM system, inspired in Ref. [84], to represent a set of disconnected maps that can be used in every map operation of place recognition, camera localization, loop closure, and map merging, enabling the combination of maps built at different times performing an incremental multi-session SLAM or even creating new maps when tracking is lost; an abstract camera representation, bringing the capability of use to any camera model by providing its projection and un-projection Jacobian functions. Moreover, a significant improvement is the ATLAS system, providing an active map where the tracking thread localizes incoming frames, continuously optimizing and increasing the Atlas with new keyframes, where even non-active maps await to be connected. The tracking thread computes the pose of a current frame concerning the active map, minimizes reprojection error, and selects which frames become keyframes. The local mapping thread adds keyframes and points to the active map, removes redundant points, and refines the map by bundle adjustment using a window of neighbouring keyframes. Finally, the loop and map merging thread detects common regions in the Atlas, so if common areas are found, loop correction is performed, and then an independent thread performs full BA to refine the whole map. ORB-SLAM3 was tested against a large set of monocular, stereo, monocular inertial, and stereo inertial methods in the EuRoC and TUM-VI datasets using RMSE (Root Mean Square Error) and ATE (Absolute Trajectory Error) metrics where the system proved to outperform all the methods in most sequences. Still, it was reported that ORB-SLAM3 fails in texture-less environments, slow motion, or pure rotational applications. Fig. 14 illustrates the ORB-SLAM3 algorithm inspired by the article [81]. 6.1.2. Classic +dense +indirect methods This formulation mainly estimates the 3D geometry from or in conjunction with the regularized optical flow field; combining the geometric error (deviation from the flow field) with the obtained geometric prior (explained by the smoothness of the flow field) [24] Fig. 14. Diagram of ORB-SLAM3 algorithm. Adapted from Ref. [81]. E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 18 is common. As mentioned above, this category of monocular methods requires a large amount of input information using most pixel values to perform its constituting processes. For instance, dense monocular methods do not require extracting a subset of features because they work directly over the entire input. So, these methods do not require discrete features but involve major computational costs due to the larger amount of data that will be processed. This category of methods is considered indirect because most depend on optical flow information obtained in a preprocessing step. Valgaerts et al. (2011). In 2012, Valgaerts et al. [85] performed a comparative study of dense and sparse methods, where the authors proposed a variational dense 3D reconstruction model to recover the fundamental matrix and optical flow by minimizing a single energy function. The authors also explored the difference between sparse feature-based methods for estimating epipolar geometry and dense energy-based methods commonly used for estimating correspondences within the image sequence. The investigation aimed to demonstrate that dense optical flow methods can also be used to estimate epipolar geometry, suggesting exploring a joint variational approach to jointly estimating epipolar geometry and optical flow. They compared feature-based techniques systems using Scale Invariant Feature Transform (SIFT) and Kanade-Lucas-Tomasi tracker (KLT) feature matching algorithms. Moreover, the use of Random Sampling Consensus (RANSAC) and Least Median of Squares (LMedS) techniques for estimating the fundamental matrix were evaluated. RANSAC extensions LORANSAC (local optimization RANSAC) of [86] and DEGENSAC (degenerate configurations RANSAC) [87] were analyzed as well. Several tests determined that dense estimation applied to epipolar geometry presents advantages over sparse methods when features are not well localized or when a small number of out-of-plane correspondences must be included to overcome degeneracy problems. They also tested their variational model on many applications. One was automatic 3D reconstruction by extracting the camera projection matrices for the estimated basic matrix, triangulating the back-projected ray for each pixel, or applying a projective transformation when no additional camera or scene information is available. This method achieved 3D reconstruction simultaneously, solving dense epipolar geometry and two-image optical flow by associating a 3D point in space with each image pixel. Therefore, authors achieved higher accuracy and stability than separate epipolar and optical flow estimations. However, as this method relies on image sequence enabling stable estimation, the system was limited to rigid applications without moving objects. Fig. 15 illustrates Valgaerts et al. algorithm inspired by the article [85]. Ranftl et al. (2016). One of the most successful methods in this category is the work of Ranftl et al. [88], called "Dense Monocular Depth Estimation in Complex Dynamic Scenes", which is a system able to obtain a dense depth map for both static and dynamic objects using only two consecutive frames. It uses a segmentation algorithm that segments optical flow, obtaining a set of motion models, each with its epipolar geometry, and then the scene can be reconstructed, optimizing a convex problem. In this method, depth estimation comprises two stages. First is motion segmentation, where a dynamic scene is divided into a set of moving models with their epipolar geometry performed in an optical flow field formulated as a variational labelling problem. Second is reconstruction, where the scene is assembled by jointly reasoning the scale and location of different components relative to the camera by object triangulation and reconstruction of all its constitutive objects. In the Motion segmentation stage, the dynamic scene is decomposed into an independent rigid motion set described by their fundamental matrix and a per-pixel assignment. This process, formulated as a joint estimation labelling problem, requires a dense optical flow field F=(fx,fy)between the I1 and I2 images generating a soft ul assignment of each pixel to either one l label for distinct Ml motion models or an additional l+1 outlier label, so the formulation is: (u∗ l,M∗ l)=arg minul,Fl∑ L+1 l=1 ul⋅g(Fl) + ‖Wl∇ul‖2,1,subject to ∑ L+1 l=1 ui l=1,ui l≥0,∀l.rank(Fl) = 2,(3) Fig. 15. Diagram of Valgaerts et al., algorithm. Adapted from Ref. [85]. E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 19 gi(Ml) = d(xi 1,Mlxi 2)2+d(xi 2,MT lxi 1)2,(4) where gi(Ml)corresponds to the symmetric distance to the epipolar lines for every l∈ {1. . . L}model, xi 1=[xiyi,1]T and xi 2=[xi− fi x,yi−fi y,1]are the homogeneous coordinates in the first image and their corresponding homogeneous coordinates of the second image, respectively. ‖Wl∇ul‖2,1 is the smoothness term, ∇the linear operator represents the discrete difference between x and y, and Wl is the diagonal weighting matrix applied to enable edge-preserving regularization. Then, the energy described in the equation is optimized using a variant of the primal-dual algorithm employing entropy proximal terms for implicitly representing simplex constraints to solve for labelling efficiently. So, fundamental matrices Ml are decomposed in parallel for all the L models, and soft assignments ui are used to reweight the individual correspondence: M∗ l=arg minFl∑ M i=1 ui l((xi 1)TMl(xi 2))2,subject to rank(Ml) = 2=2,(5) Then the subproblems are approximately solved using a reweighted version of an 8-point algorithm proposed in Ref. [89]. To obtain the number of dynamic models, the process performs the following steps: Apply the 8-point algorithm to mine a small set of candidates; Solve the energy equation by expanding the pool of motion candidates; add new models by robustly estimating motion from pixels with outlier label; expand the pool by splitting labels with disconnected regions; perform alternating minimization again and repeat the process until no further energy minimization can be made. Following this procedure, a set of epipolar geometries F∗ l and membership probabilities u∗ l is obtained for each pixel. Finally, a robust reconstruction is performed using these epipolar models and optical flow information using a super-pixel-based formulation. This system was tested over KITTI [7] and MPI Sintel [90] datasets, outperforming most of the dynamic scene geometry from monocular video techniques of the time, using different techniques for optical flow computation like Large Displacement Optical Flow (LDOF), EpicFlow, and FlowFields. Thus, it was found that since the proposed method relies on optic flow information, the whole system will fail if the optic flow estimation fails. In addition, this process has the limitations of a purely geometric approach that does not use prior information about shape and scene size. Hence, the authors suggest that the proposed method can be complemented by ML techniques to estimate the absolute scale. Fig. 16 illustrates the Ranftl et al. algorithm, inspired by the article [88]. 6.2. Classic +direct methods Direct methods are developed to recover scene geometry and agent ego-motion using direct pixel intensity information, so unlike indirect methods, these formulations do not require pre-processing steps such as feature extraction (which drastically reduces the amount of information available to the SLAM system, saving computational resources and time, but limiting the final 3D reconstruction density). In contrast, a direct approach works with each pixel intensity value on the image or at least most of them, so the final 3D reconstruction quality is commonly higher than indirect approaches. However, direct formulations rely on the brightness constancy assumption, which establishes that brightness over an object’s position must be the same over different angles. Unfortunately, this is not always true, so these formulations are known to fail in scenes that display motion blur, moving objects, or non-Lambertian surfaces. This category of methods can also be divided according to the final density of the 3D map so that there are dense and sparse formulations, which will be discussed in sections 6.2.1 and 6.2.2. Fig. 17 shows a timeline of the most representative contributions in Fig. 16. Diagram of Ranftl et al., algorithm. Adapted from Ref. [88]. E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 20 this area. As shown in Fig. 17, three of the most representative monocular systems of all time belong to the category DTAM [27], LSD-SLAM [37], and DSO [24]. It must be noted that although DSO has recently been introduced, it has attracted the interest of many researchers, achieving impressive citation scores. 6.2.1. Classic +dense +direct methods This category of slam approaches is dense because most input information is used to perform the reconstruction. In addition, they are classified as direct techniques because of the lack of preprocessing stages, so the input image is directly employed to obtain environment geometry. This formulation commonly applies photometric error and geometric priors to estimate dense or semi-dense geometry because they work directly on pixel intensity information. Stühmer et al. (2010). Stümer, Gumhold, and Cremers [68] suggested one of the first approaches to these techniques [91], called "Real-Time Dense Geometry from a Handheld Camera," a real-time variational system to estimate dense depth maps using multiple images directly. Given a set of images, this method uses data terms relative to the coordinate system of a specific view and uses its perspective projection to map such coordinates in a second camera frame; thus [91], the following energy function was proposed to estimate depth maps using multiple images: E(h) = λ∫Y∑ i∈T(x) | ρ i(x,D)|d2x+∫Y |∇D|d2x,(6) where D is the depth map, x= (x1,x2,1)T homogeneous 2D coordinates, d(x,D)is the depth value for each pixel, Yi is the image plane, and Ti is the camera pose. In this case, T(x)contains all image indexes where perspective projection π (exp( Ti)⋅d(x,D)) is contained in the image boundaries, and ρ i(x,D)is the linearized residual data term for an Ii image: ρ i(x,D) = Ii(x,D0) + (D−D0)ID i(x) − I0(x),(7) where ID i(x)represents the derivative d dDIi(x,D)|D0. As can be seen, the formulation takes direct pixel information into the energy function, and the D depth map is obtained after a minimization procedure, so this is a direct method. In addition, it is noticeable that this energy function is quite complex because the data term consists of the sum of absolute values for linear functions, and simple thresholding techniques cannot minimize it. Hence, the authors provide a proposal for generalized thresholding. The advantage of this dense multi-view proposal over two-image techniques is that different views should help estimate disparity information in areas occluded by other views because this system can add information from images where the object is not occluded. Another benefit is an increase in the signal-to-noise ratio, which improves results when the input images are affected by noise, which is common in video captured by most webcams or handheld devices. To summarize, in this method, instead of using real-time pose estimation measurements, the depth map is estimated using the current input image and the N closest keyframes to the current pose, so by using keyframe camera pose estimates, the amount of noise is minimized. Finally, this proposal was embedded with the camera tracking module of a PTAM implementation [41], which can store keyframes, and each camera pose associated with each keyframe is Fig. 17. Timeline for the most representative monocular SLAM, VO, or SFM classic direct systems. Fig. 18. Diagram Stühmer et al. algorithm. Adapted from Ref. [91]. E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 21 refined iteratively. Similarly, the depth map related to its corresponding keyframe is refined using the N closest keyframes. Fig. 18 illustrates the Stühmer et al. algorithm inspired by the article [91]. DTAM (2011). Another direct dense approach was proposed by Newcombe et al. [27], a dense per-pixel method that aims to create a dense 3D model by performing a dense and sub-pixel reconstruction for accurate camera tracking. This dense model was used to perform a full image alignment with respect to the current model so that camera motion could be estimated. Immediately, the model is extended and updated using the tracked images by creating and refining dense depth maps. In this method, the dense model corresponds to overlapping keyframes, where depth values d are back projected from every pixel, resulting in a direct method. Here an r keyframe comprises an Ir image, a camera pose Tr ω , and a Cr cost. Every qr pixel has an associated cost error Cr(q,d)for each d depth value. Then a large amount of m∈T(r)video frames are used to compute the cost volume, being T(r)a set of nearby frames. Then, the photometric error is computed by projecting each point of the volume into all the overlapping images summing L1 norms of each photometric error: Cr(q,d) = 1 T(r)∑ m∈T(r) ‖ ρ r(Im,q,d)‖1,(8) ρ r(Im,q,d) = Ir(q) − Im(Λ((ζTmrΛ−1(q,d))),(9) where ρ r is the photometric error for every overlapped image, ζ is the intrinsic matrix, Λ(xc) = (x/z,y/z)T is de-homogenization for a 3D point xc= (x,y,z)T. The inverse depth map is obtained by minimizing the energy functional Ed∗, which comprises non-convex photometric error cost as a data term and a convex regularizer term: ED∗=∫Ω{g(q)‖∇d∗(q)‖ϵ+λC(q,d∗(q))}dq,(10) where g(q) = e− α ‖∇Ir(q)‖β 2 is per pixel weight, D∗is the inverse depth map, ∇d∗(q)is the inverse depth map gradient, and Ω is the image domain, ϵ is set to a small value to reduce the stair-casing effect, λ=1/(1+0.5d)reflects data term quality and α and β are auxiliary variables. Thus, DTAM runs over an energy minimization framework using a photometric error data term and a robust spatial regularization term, so it starts defining a projective photometric cost volume (disparity space image in stereo matching) regularized using a weighted Huber norm on the inverse depth map gradient. Then, it is discretized and solved using duality principles to get the primaldual form, where the weighted Huber regulator is replaced by its conjugate by applying the Legendre-Frenchel transform, so the inverse depth map can be extracted by iteratively minimizing the cost volume for every pixel from a reference frame. DTAM uses the PTAM point-feature-based method for initialization until the first keyframe is captured. It then switches to its own fully dense tracking and mapping pipeline. Next, a new keyframe is added when the number of pixels without visible surface information from the previous predicted image is below a threshold. In brief, this method estimates camera pose in real-time by finding motion parameters generating a synthetic view that best matches a live video frame. Fig. 19 illustrates the DTAM algorithm inspired by the article [27]. REMODE (2014). In 2014 Pizzoli, F¨ orster, and Scaramuzza [4] proposed another dense direct method called REMODE, an abbreviation of "Regularized Monocular Depth Estimation", which is able to perform a dense depth map computation using Bayesian estimation and a novel complex optimization, developing a platform that acts like a depth sensor able to handle a large depth range. In REMODE, each pixel depth was estimated independently using a probabilistic approach and a new smoothing proposal. This per-pixel Bayesian depth estimation was built over the method proposed in Ref. [92], extending it by adding an optimization step consisting of regularization using a weighted Huber norm. However, unlike DTAM, depth uncertainty was used to exploit a convex formulation, avoiding the effect of noisy camera localizations. Thus, in REMODE, depth is computed as a Bayesian estimation problem by triangulating from a reference view and the last view acquired. Then, the pixel’s depth is handled as a parametric model updated for every observation, and smoothness is achieved by minimizing a regularized energy function. Furthermore, REMODE includes a probabilistic approach where a depth hypothesis dk is generated using the {Ik,Tk, ω }observation by triangulating r and k views, where Tk, ω is a rigid body transformation that describes the camera pose for every image. The depth sensor was modelled as a distribution that combines a good measurement, normally distributed around the real depth d, and an outlier measure containing the depth of the desired structure. Fig. 19. Diagram of DTAM algorithm. Adapted from Ref. [27]. E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 22 p(dk|d, ρ ) = ρ N(dk|d, σ 2 k)+ (1− ρ )U(dk|dmin,dmax),(11) where ρ and σ 2 k are the probability and variance of good measurement, p(d,p)is the prior (knowledge of uncertainty) on true depth and the ratio of measurements supporting it. Then, the corresponding posterior is approximated as a product of Gaussian distribution for depth and Beta distribution for the inlier ratio. q(d, ρ |ak,bk, μ k, τ 2 k)=Beta( ρ |ak,bk)N(d| μ k, τ 2 k),(12) ak,bk are parameters controlling Beta distribution. Then, for every q pixel μ k and τ 2 k are the mean depth estimation, its confidence for each observation and the denoised depth map F(q)is obtained by the following energy minimization: min F∫Ω{G(q)‖∇F(q)‖ϵ+λ‖F(q) − D(q)‖1}dq,(13) D(q)is the depth map, and G(q)is the “G-Weighted Total Variation” weighting function [93]. These equations are basic methods and examples of how a probabilistic approach can be incorporated to obtain a denoised depth map directly from pixel information. REMODE uses a tracking thread inspired by the odometry system of SVO [23], using an image alignment formulation to estimate the pose but working only with pixel intensity information. After that, the mapping thread triangulates the depth using each frame and the reference view, so the depth of each pixel is formulated as a parametric model computed as a Bayesian estimation problem that includes a regularizer based on the gradient Huber norm of the gradient, so the solution is obtained iteratively by minimization, exploiting a primal-dual formulation and a gradient descent-ascent technique. Fig. 20 illustrates the REMODE algorithm inspired by the article [4]. LSD-SLAM (2014). In the study by Engel et al. [37], a real-time monocular SLAM and 3D reconstruction system was developed that not only tracks the camera motion locally but also constructs consistent, large-scale, environment-dense maps by using a semi-dense representation that tracks depth values only on gradient surfaces. The method uses direct image alignment and filter-based estimation of semi-dense depth maps based on the proposal [65]. The global depth map is represented as a pose graph consisting of keyframes as vertices and 3D similarity transforms as edges, with the ability to detect environmental scale changes and correct for accumulated drift. LSD-SLAM uses an appearance-only loop detection algorithm, FAB-MAP [94], to propose candidates for large loop closures and extract their features without reusing any information acquired in the visual odometry front end. Contributions of the LSD-SLAM method consist of a direct method to perform alignment of two keyframes on ξ ϵ sim(3), and a probabilistically consistent incorporation of noisy uncertainty of the estimated depth into tracking. The novel image alignment is performed by Gauss-Newton minimization of the photometric error: E(T) = ∑ i(Iref (qi) − I( ω (qi,D∗ref (qi),T)))2,(14) where I are images, D∗is the per-pixel inverse depth map, qi is the image point, ω is a 3D projective warped function, and T is the camera pose. The complete method requires tracking, depth map estimation, and map initialization. The tracking module continuously tracks new images estimating the rigid body pose on the current keyframe, so the relative pose is calculated by minimizing the variance-normalized photometric error: Fig. 20. Diagram of REMODE algorithm. Adapted from Ref. [4]. E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 23 min T∈SE(3)∑ p∈ΩDi r2 q(q,Tij) σ 2 rq(q,Tij)δ ,(15) where r2 q and σ 2 rq are the photometric residual and variance, respectively. Also, for adding a keyframe to the map, the closest keyframes are found, and the edges are estimated by SE(3), so minimization is performed by the equation: min T∈SE(3)∑ q∈ΩD∗ i r2 q(q,Tij) σ 2 rq(q,Tij)+r2 d(q,Tij) σ 2 rd(q,Tij)δ .(16) Summarizing LSD-SLAM can obtain dense depth maps by estimating the rigid body pose from camera frames with respect to the current keyframe, using the previous frame for initialization (tracking). Next, the tracked frames are used to refine or replace the current keyframe so that depth is refined by multiple per-pixel small baseline comparisons (depth map estimation). Finally, when a new keyframe is replaced as a tracking reference and no further refinement is executed, it is added to the global map (map optimization). Fig. 21 shows the LSD-SLAM algorithm inspired by the article [37]. 6.2.2. Classic +sparse +direct methods This category of formulations typically uses photometric error optimized directly from the input frames without requiring the introduction of geometric priors and preprocessing steps. A crucial benefit from the direct formulation is that it commonly uses pixelwise inverse depth, which does not require a point to be recognized by itself, allowing a more grained and detailed geometry representation. Furthermore, this process allows for the sampling from every pixel, including edges and weak intensity variations, adding robustness to textured environments, whereas sparse methods do not require geometric priors, avoiding their limitations. Such priors involve the introduction of correlations between geometry parameters in which real-time statistically consistent joint optimization is typically impossible. These priors can also introduce bias, resulting in the tendency to reduce large-scale accuracy. Direct methods do not rely on a set of points’ repeatability, so they can operate on low-texture surfaces with contours. Many methods from this category apply photometric bundle adjustment to minimize the photometric error of mapped point observations on a local sliding window of active keyframes, where points are sampled across pixels with high gradients, such as edges and intensity variations. In this context, VO systems typically use sliding windows that select close-in-time active keyframes, marginalizing map points far from the field of view. This situation could represent a disadvantage since VO systems cannot benefit from map point reobservations. On the other hand, VSLAM sparse direct systems typically build persistent maps of the scene representing a network of keyframes connected through observing the same region at different times. DSO (2017). The work of [24] introduced a sparse formulation of direct visual odometry called Direct Sparse Odometry (DSO). This approach was designed to combine the advantages of direct methods (such as the ability to reconstruct most points, not just corners) with those of the sparse approach (such as the flexibility and efficiency of joint optimization). This monocular visual odometry algorithm can track even scenes with little texture, where indirect approaches often fail. DSO performs a continuous optimization of the photometric error applied to a set of recent frames, taking into account a photometrically calibrated model in the image formation process inspired by the proposal of [95]. At the same time, it optimizes the full likelihood for all the parameters involved in the model, such as camera poses, intrinsics, extrinsics, and inverse depth values in a photometric process equivalent to a windowed sparse bundle adjustment. Such processes require enough accurate initializations in the front end to carry out a non-convex optimization in the back end. The minimization process has to be carried out for the photometric error of a point p in a reference frame Ii over a small pixel neighbourhood, where experimentally, it was revealed that 8 pixels arranged in a residual spread pattern provide sufficient inforFig. 21. Diagram of LSD-SLAM algorithm. Adapted from Ref. [37]. E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 24 mation for computation. The formulation for photometric error to be minimized is as follows: Eqj:=∑ i∈Nq ω q(Ij[qʹ] − bj)−tjeaj tieai (Ii[q] − bi)γ ,(17) where Nq is the set of pixels, ti, and tj are exposure times for the Ii and Ij images, ai, bi, aj, and bj are the brightness transfer function parameters for Ti, Tj poses of the involved frames. The gradient-dependent weighting ω q and the projected point position qʹ, are given by: qʹ=Πc(RΠ−1 c(q,dq)+t)with [R t 0 1]:=TjT−1 i, ω q:=ζ2 ζ2+ ‖∇Ii(q)‖2 2 ,(18) where ζ represents the camera intrinsics matrix, Πζ, Π−1 ζ are projection and the back-projection functions, and dq is the inverse depth for a projected point position. Finally, i runs for all F frames, q runs for all Pi points of the image, and j runs over all obs(q)frames, where the point is visible, so the full photometric error for DSO was: Ephoto =∑ i∈F∑ q∈Pi∑ j∈obs(q) Eqj.(19) DSO algorithm performance comprises two modules in control of frame and point management. Frame management is intended to work over a set of active frames where each new frame is tracked from the current keyframe, two-frame direct alignment, a multiscale image pyramid, and a constant motion model track all its points. Then, a new keyframe is created when the field of view changes, on occlusions and disocclusions, and when camera exposure time changes. Consequently, old keyframes are marginalized when they are not visible enough, and the farthest keyframe is marginalized when the maximum number of active keyframes is exceeded. Finally, the point management module oversees the selection of candidate points within an optimization window, point tracking and candidate point activation as required for windowed optimization. The algorithm was evaluated adequately over three datasets, proving that using a large amount of data does not necessarily increase accuracy; however, using a sparse set of points improves accuracy and robustness. Fig. 22 introduces the DSO algorithm inspired by the article [24]. LDSO (2018). An extension of the DSO system was developed in the work of Gao et al. [64]. Since DSO allows using any pixel with a significant intensity gradient, it ensures repeatability points favouring corner features to detect loop closure candidates using the bag-of-words approach. Depth estimates of matched feature points were used to compute Sim (3) constraints; thus, in combination with pose-only bundle adjustment and point cloud alignment, they are fused with a co-visibility graph of relative poses extracted from the DSO sliding window optimization stage. Point selection is still needed in direct methods, yet a noticeable difference between direct and indirect methods is that point repeatability is not required in direct methods. In LDSO, corners and high gradient pixels are used where corners are used for generating BoW models, and the other pixels are used for tracking. LDSO proposes loop candidates for a keyframe by querying the database (using only those outside the window). Then, the method tries to match each corresponding feature, and the initial guess of SE (3) is computed by performing a RANSAC PnP. Next, a Sim (3) transformation is optimized using the Gauss-Newton method, minimizing the following cost function: Eloop =∑ mi∈Q1 ω 1ScrΠ−1(qi,d∗ qi)−Π−1(mi,d∗ qi)2+∑ mj∈Q2 ω 2Π(ScrΠ−1(qj,d∗ qj))−mj2,(20) Fig. 22. Diagram of DSO algorithm. Adapted from Ref. [24]. E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 25 sparse depth map is conveniently added from the SLAM system. To extend network generalization capabilities, the authors used diverse training data coming from five different datasets: Taskonomy [156], DIML [157], ApolloScape [8], DiverseDepth [154], and RedWeb [158], which were recorded using high and medium accuracy annotation devices like LiDAR, Laser and Stereo setups having indoors and outdoors dense sequences. In the training stage, a FAST corner detector was used to sample sparse depth points for dense depth datasets to match the behaviour of the sparse depth coming from the SLAM system, and the two input modes were selected randomly, so two different loss functions were set for each mode. Experimental results showed that the ORB-SLAM system improved considerably in terms of generalization, strengthening indoor and outdoor scenes performance, improving depth estimation capabilities, and reducing the absolute trajectory error evaluated on the KITTI dataset. Despite such excellent results in visual odometry evaluation, it must be mentioned that the obtained depth maps were denser than ORB-SLAM, but in our opinion, compared to the reconstruction results of other dense proposals, they are not dense enough to be considered a dense indirect approach. Fig. 31 exhibits the Sun et al. algorithm inspired by the article [60]. Lee et al. (2022). One of the most recent approaches made on the sparse indirect category that incorporates deep learning in its framework is the work of [159], which aims to improve the accuracy and robustness of existing SLAM methods for autonomous driving. SLAM systems are essential for self-driving vehicles, as they allow the estimation of their position and the mapping of the environment using a single camera. The authors have formulated a deep neural network-based monocular SLAM system that integrates semantic segmentation and 3D geometry estimation to improve the system’s accuracy. The proposed system was built upon ORB-SLAM, which was extended by implementing a deep neural network for estimating 3D geometry and adding a semantic segmentation module to improve the quality of the generated point cloud. The semantic segmentation module was integrated to help differentiate objects with similar geometries, like cars, pedestrians, and trees, using a labelling technique to improve the accuracy of the point cloud construction. It must be pointed out that the proposed monocular SLAM system’s main innovation is its ability to leverage semantic segmentation for more accurate 3D environment reconstruction and mapping, where the system’s deep neural network-based formulation allows it to learn and adapt to different environments and lighting conditions, making it robust to real-world scenarios. Moreover, the system uses a novel loss function that penalizes the translation and rotation errors differently to stabilize the estimated pose. In brief, the Lee et al. method was designed using three modules: localization mapping and segmentation. The localization module is responsible for selecting keyframes when mapping and segmentation modules complete each keyframe processing; this allows the extraction of corner features and the camera pose of each keyframe, estimated from the points of connected keyframes. The mapping module triangulates current and connected corner features to generate new 3D points. Then, it estimates a ground plane only using the points labelled as ground by the CNN, which also recovers the appropriate scale for camera poses. In this way, the mapping module estimates scale-corrected camera poses and 3D points. In contrast, the segmentation module performs deep-learning semantic segmentation over each down-sampled keyframe and refines the corner features existing on the keyframe by removing moving objects and areas with low-parallax, using the ERFNet CNN proposed in Ref. [160]. Thus, some additional improvements made in the Lee et al. approach are its scale correction in 3D mapping and its novel technique to remove factors that can lead to inappropriate mapping in each keyframe. To evaluate the proposed monocular SLAM system’s performance, the authors conducted experiments on the KITTI benchmark dataset, which allowed them to demonstrate that their system outperforms the existing state-of-the-art monocular SLAM methods like ORB-SLAM, ORB-SLAM2, and Mask-SLAM. The proposed system achieved an average translational error of 0.19 %, significantly lower than the 0.40 % error achieved by the closest competitor. Additionally, it was observed that the system’s semantic segmentation module improved the 3D reconstruction accuracy and enabled the generation of a detailed semantic map. Fig. 32 illustrates the Lee et al., algorithm inspired by the article [159]. SVR-Net (2023). The SVR-Net SLAM system [161] is a technique for simultaneous localization and mapping that was developed for both pure visual and visual +range sensors. Its purpose is to provide accurate 3D maps of unknown environments. A key contribution of this system is the incorporation of a Sparse Voxelized Recurrent (SVR) network, which is used to estimate the 3D location of key points in the scene. This improvement enables the system to achieve robust feature tracking, even in challenging lighting conditions or in the presence of occlusions. The SVR-Net SLAM system incorporates several other novel features, including an online learning algorithm that continually adapts to changing environmental conditions and graph optimization to refine the estimated camera poses and map structure. The system also includes loop-closure detection, which helps improve the map’s overall accuracy and consistency. Compared to other SLAM systems that rely solely on visual or range sensors, the SVR-Net SLAM system offers several key Fig. 31. Diagram of Sun et al., algorithm. Adapted from Ref. [60]. E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 32 advantages, such as its ability to produce highly detailed and accurate maps of complex environments, including structures with multiple levels and non-planar surfaces. Additionally, the system is computationally efficient, making it well-suited for real-time applications. In brief, SVR-Net SLAM utilises a coarse-to-fine approach to achieve efficient tracking and dense global mapping. It comprises two modules for processing raw data and refining the results. At the first stage, the system receives a pair of frames from where the raw pose and local map are estimated using the SVR network, so it starts representing the map as sparse voxels with TSDF (truncated signed distance function) values. Then, the map is extended using the first stage, the global map. In the second stage, SVR-Net SLAM performs voxel up-sampling, followed by pose and map refinement; then, the global map is extended by fusing it with the resulting fine local map. Specifically, the SVR-Net module was trained over the ScanNet(V2) proposal [162] as an end-to-end tracking and mapping network that takes a pair of RGB frames along with a set of voxel coordinates and outputs their local map, relative pose, and the TSDF values for the set of voxels. For this purpose, SVR-Net begins extracting feature maps for the images. Then, it transforms the feature map into feature voxels for each keyframe (first frame), and it estimates the correlations with the features of the second frame to provide similarity information for matching. Next, after sampling according to the currently estimated pose, a matching network Fig. 32. Diagram of Lee et al., algorithm. Adapted from Ref. [159]. Fig. 33. Diagram of SVR-Net SLAM algorithm. Adapted from Ref. [161]. E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 33 matches the features (optimal match search in the correlation field), updates the pose, and maps iteratively. This way, voxels are projected to the second frame using the estimated pose from the previous iteration. The correlation values obtained from the sample are inputted into the matching network to predict the updated TSDF values and corrected projection coordinates. Then, the feature-matching step outputs are utilized to update the pose and map estimations of SVR-Net. Finally, the SLAM pipeline, built over the Kinect-Fusion method [42], uses the local map information to extend the global map and enhance its global consistency. Fig. 33 illustrates the SVR-Net algorithm inspired by the article [161]. 7.1.2. ML +dense +indirect methods Indirect methods usually employ optical flow for depth prediction. It is important to note that a challenging task in VO frameworks is characterising feature location errors, which motion blur, occlusions, and viewpoint variations can corrupt. Specifically, the efficacy of direct methods typically relies on small motion and appearance-constancy assumptions, representing a limitation to providing robustness to scene variability, which reduces their applicability [163]. Recently, machine learning optical flow estimation, which can be described as a combination of rigid flow and an unconstrained flow describing general object motion, has reached state-of-the-art performance [164,165] demonstrating a superb level of accuracy, robustness, and generalization becoming an excellent solution, especially under challenging conditions like texture-less surfaces, motion blur, and large occlusions. DeMoN (2017). Ummenhofer et al. [141] introduced a noteworthy example of this practice with their work on DeMoN, a motion system structure. DeMoN features the first CNN that jointly computes depth and camera motion using unconstrained image pairs, improving its predictions. The network also estimates surface normals, optical flow, and matching confidence. Furthermore, DeMoN can exploit motion parallax information, a powerful clue to generalizing new scenarios enabling egomotion estimation. For this purpose, the system alternates optical flow estimation with depth and camera motion estimation. The system includes an adapted version of FlowNet [166] to solve optical flow using a pair of images. The network outputs, depth maps, and motion vectors could be very different, so they must be balanced using loss functions; hence, L1 loss L for inverse depth values is: Ldepth =∑ i,j |sd∗(i,j) −  d∗(i,j)|,(22) where d∗=1 z is the inverse depth,  d∗is the ground truth, and s is the predicted scale. The normal and optical flow use L2 norm to penalize deviation from n and  w ground truths: Lnormal =∑ i,j ‖n(i,j) −  n(i,j)‖2,(23) Lflow =∑ i,j ‖w(i,j) −  w(i,j)‖2,(24) then the loss for motion vectors is: Lrotation = ‖r−r‖2,(25) Ltraslation = ‖t−t‖2,(26) where r=θv is a minimal parameterization of a rotation with angle θ and v axis, and t is the translation vector. Finally, the scaleinvariant loss for a discrete scale invariant gradient g is: Fig. 34. Diagram of DeMoN algorithm. Adapted from Ref. [141]. E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 34 gh[f](i,j) = (f(i+h,j) − f(i,j) |f(i+h,j)| + |f(i,j)|,f(i,j+h) − f(i,j) |f(i,j+h)| + |f(i,j)|)T ,(27) Lgrad d∗=∑ h∈{1,2,4,8,16}∑ i,j ‖gh[d∗](i,j) − gh[ d∗](i,j)‖2,(28) to cover gradients at different scales, five different h spacings are used so the network can compare within a neighbourhood for every pixel. Finally, the system applies scale invariant gradient loss to each optical flow component to enhance the smoothness of the estimated flow fields, enhancing sharpness for motion discontinuities. Summarizing, DeMoN takes both images as input to predict the depth map using the first image and the relative pose using the second image. The CNN comprises a chain of encoder-decoder networks that iterate over optical flow, the depth map, and egomotion estimation. Therefore, the system is constituted of three main components: the bootstrap net that takes an image pair as input and outputs the initial depth and motion estimates, using encoder-decoder networks that calculate optical flow and confidence maps of the flow; the iterative net casting to sharpen discontinuities, improve the scale of the depth values, correct wrong estimates of initial bootstrapping network and improve depth, normal and motion estimates; and a final refinement net that increases the resolution of the final depth map. Fig. 34 illustrates the DeMoN algorithm inspired by the article [141]. DeepV2D (2020). Teed & Deng [144] proposed DeepV2D, an end-to-end learned pipeline that alternates depth and motion modules to predict depth and camera pose. The motion module uses depth estimates to predict camera pose, which becomes more accurate as depth estimates converge. Both modules were built over neural networks ruled by geometric principles but combined in an end-to-end differentiable architecture to perform structure from motion. The depth module takes camera motion as input and calculates depth prediction, whereas the motion module takes depth as input, estimating a motion correction term as output. The deepV2D depth module builds a cost volume using learned features where information is added to multiple viewpoints using a pooling layer. In this way, the depth module is composed of: a 2D feature extractor, which is a 2D encoder using two stacked hourglass networks that map each frame to a dense feature map; a cost volume back-projection module that reprojects coordinates in each frame for each possible depth; and a 3D stereo matching network performing stereo matching over a defined set of cost volumes. The motion module outputs a set of perturbations as error terms used to update the camera pose. Motion module requires of: initialization, where a frame is chosen as a keyframe, and relative motion is predicted; feature extraction, where learned features map every frame to a feature map; error term, taking two frames and an hourglass network predicting residual flow between their feature maps; and optimization layer, where pose increments are solved applying a Gauss-Newton update. Although DeepV2D was mainly developed and focused on depth estimation, it can be turned into an SLAM system by training the neural network to directly map optical flow to camera motion without requiring optical flow supervision. As a contribution, this work differs from DeMoN [141] because its motion module can be used on a variable number of frames. DeepV2D features a new motion estimation architecture called Flow-SE3, which sets it apart from other works like DeMoN and DeepTAM. This architecture allows the system to apply geometric constraints on camera motion, reducing reprojection error and benefiting from end-to-end training. Fig. 35 illustrates the DeepV2D algorithm inspired by the article [144]. VOLDOR (2021). Another recent work worth mentioning is the VOLDOR VO system developed by Min & Dunn [163]. This system utilises log-logistic depth residuals to perform visual odometry. It takes an externally estimated optical flow input, computed by machine learning estimators, and a probabilistic model to develop a VO pipeline that does not require feature extraction, RANSAC estimation, or local bundle adjustment processes to output camera pose and depth maps. It was observed that optical flow residuals tend to conform to a log-logistic distribution, which is the base of its probabilistic framework ruled by a Fisk-distributed residual model that jointly estimates camera motion, pixel depth, and motion-track confidence using a generalized expectation maximization (EM) formulation. Similarly, the VOLDOR inference framework uses a generalized expectation-maximization formulation to infer depth and rigidness, a maximum likelihood estimator (MLE) to bootstrap subsequent camera pose, a maximum inlier estimation to mitigate MLE criteria, and a forward-backwards algorithm to infer rigidness from the image reduced to hidden Markov chains. VOLDOR is agnostic to the optical flow input estimator, so PWC-net [165] was used to estimate the external dense optical flow input. PWC-net is a CNN that uses pyramidal processing, warped features, and cost volume to warp the CNN features of a second image using the optical flow Fig. 35. Diagram of DeepV2D algorithm. Adapted from Ref. [144]. E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 35 estimate of the current image, being 17 times smaller and easier to train than FlowNet2 [164] and comprised of a feature extractor, an optical flow estimator and context networks. In summary, VOLDOR uses a Fisk residual model to perform inference from a sequence of optical flows extracted by an external estimator (PWC-net system in this case), where the first camera pose is initialized from epipolar geometry from the first optical flow using the least median-square estimator. After that, depths are triangulated by two-view triangulation using optical flow and the first camera pose. Finally, the update is performed by inference over the Fisk residual model, where a generalized expectation maximization model updates depth and rigidness. Then, a maximum likelihood estimator is used for sequentially bootstrapping subsequent camera poses, while a maximum inlier estimation criterion is used to reduce bias generated at initialization and update processes. In conjunction with the Fisk residual model and maximum inlier estimation, this CNN system achieved optimum results in an ablation study and outperformed similar optical flow systems in the KITTY and TUM RGB-D datasets. Fig. 36 describes the VOLDOR algorithm inspired by the article [163]. DROID-SLAM (2021). One of the recent notable works is DROID-SLAM, which stands for ’Differentiable Recurrent OptimizationInspired Design’ [167]. This deep learning-based system performs iterative camera pose updates and estimates depth maps through a dense bundle adjustment layer. DROID-SLAM predicts updates in the dense flow fields domain using a Gated Recurrent Unit (GRU) to produce an error correction term in the dense correspondence field and a depth map. DROID-SLAM can perform real-time localization and mapping by using frontend and backend threads. The front-end inputs new frames, extracts features, selects keyframes and executes local bundle adjustment. At the same time, the backend thread simultaneously performs global bundle adjustment over the entire set of historic keyframes. DROID-SLAM is a differentiable architecture built over RAFT [168] (Recurrent all-pairs field transforms for optical flow), facilitating the system to work with optical flow, performing recurrent iterative updates so instead of updating the optical flow, the system updates depth maps and camera poses obtained through a differentiable bundle adjustment layer calculating the Gauss-Newton update to maximize compatibility with current optical flow estimate. DROID-SLAM can work with monocular, RGB-D, and stereo input, demonstrating an exceptional performance on the TartanAir, EuRoC, TUM-RGB-D, and ETH3D-SLAM benchmarks, outperforming acknowledged classic and learned-based monocular systems in most sequences. The authors Fig. 36. Diagram of VOLDOR algorithm. Adapted from Ref. [163]. Fig. 37. Diagram of DROID-SLAM algorithm. Adapted from Ref. [167]. E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 36 noted that the monocular system has a major limitation in its high computational cost. During the experimental stage, 24 GB of GPU memory was required to run the EuRoC, TartanAir, and ETH3D sequences. Fig. 37 presents the DROID-SLAM algorithm inspired by the article [167]. SDF-SLAM (2022). Another important and relevant work in this category is SDF-SLAM [146]. It is a monocular system based on the ideas of ORB-SLAM but with a novel feature extraction method and a dense semantic network for estimating dense depth labels. This places SDF-SLAM in the dense category of the taxonomy. As previously stated, the original ORB-SLAM system was designed to extract texture information, such as edges or corners, by using a combination of algorithms, including SIFT, SURF, ORB, and others, to extract feature points from each pair of adjacent frames based on their similarity. Feature points of each frame are projected to the next frame using the change in camera pose, where PnP is used to minimize errors between the projected point and actual matching. Next, PnP converts those feature points into 3D coordinates using the camera pose, fusing all the features and creating a depth map. Nevertheless, the map created by ORB-SLAM is not dense enough for many applications and lacks semantic information which a machine can recognize. This way, the proposal aims for accurate camera trajectory estimation and recovery of a three-dimensional scene semantic reconstruction map. Therefore, the semantic and depth fusion proposal fuses camera poses with depth and semantic information at the frame level to obtain a 3D semantic reconstruction achieved by implementing three key components: a feature point and CNN feature description (FPFDCNN) trained to perform feature extraction from a pair of images calculating vectors as descriptors for every feature point; a deep semantic fusion CNN (SDFCNN) trained to perform simultaneous semantic segmentation and depth prediction from RGB images, allowing the system to drastically reduce the number of estimated parameters; a monocular visual SLAM system embedding a deep learning system adding a data correction module to globally optimize the point cloud to obtain consistent point clouds. For such purposes, SDF-SLAM uses two neural networks where FPFDCNN is used to extract feature points from a pair of adjacent frames, matching them to obtain feature-matching pairs. Then, a minimization process is applied to obtain camera rotation and displacement matrixes, fed along with the image to SDFCNN to recover a dense map and a semantic segmentation. Finally, the data calibration module is fed with dense map, pose, and semantic segmentation information to perform global and local optimization tasks, creating a three-dimensional semantic map. The FPFDCNN was built over the encoder-decoder architecture constituted by an input layer, encoder layers (to extract the feature map), decoder layers (to restore the original image size and feature descriptors), output layers, and concatenate layers. For one thing, SDFCNN was implemented using a unified semantic down-sampling layer to process information in feature maps and an up-sampling layer for feature extraction and restoration that uses discriminative layers for feature classification, probability estimation, and depth regression estimation. Experiments carried out using SDF-SLAM on the TUM dataset demonstrated a clear improvement over the classic ORB-SLAM [11] and LSD-SLAM [37] methods, also demonstrating a significant improvement in semantic segmentation quality compared with CNN-SLAM [2], achieving 90 % accuracy for point cloud prediction and 67 % in semantic labelling. Another advantage of SDF-SLAM is that this technique can be fused with many classic SLAM approaches, which authors demonstrated with reported results running the same network architectures over the DSO framework. Fig. 38 illustrates the SDF-SLAM algorithm inspired by the article [146]. NeRF-SLAM (2022). Article [169] presents NeRF-SLAM, a monocular method for indoor scene reconstruction using normal priors, to enhance reconstruction accuracy and detail. The method combines the advantages of Neural Radiance Fields (NeRF) and the DROID-SLAM [167] approach as a tracking module to recover real-time accurate and detailed reconstructions of indoor scenes. NeRF-SLAM was developed to address limitations such as challenges in handling dynamic environments and occlusions and limited Fig. 38. Diagram of SDF-SLAM algorithm. Adapted from Ref. [146]. E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 37 scalability in many monocular systems. NeRF-SLAM framework aims to overcome these limitations by using an implicit neural representation to model the scene’s geometry and appearance, incorporating normal priors to improve reconstruction detail and accuracy. The framework consists of three components: a frontend that estimates the camera poses and generates sparse 3D point clouds, a backend that fuses the sparse point clouds, generating the final implicit representation of the scene, and a normal prediction module trained to predict surface normals from the implicit representation. The tracking front was built over DROID-SLAM, gathering dense depth maps and poses for each keyframe taken from an eight-keyframe sliding window. For this purpose, DROID-SLAM estimates the optical flow of each pair of frames using an approach inspired by Raft, which is a Convolutional GRU that computes the flow and weight using the correlation of each two-frames and a guess of the current optical flow. Then, DROID-SLAM solves the dense BA problem using the flows and weights representing the 3D geometry as a parameterized set of inverse depth maps, which helps solve the BA problem efficiently using a linear least squares formulation. Then, the method computes marginal covariances for the depth maps and poses, which are used along with the depth, poses, and input RGB images to optimize the radiance field’s parameters and refine the camera poses. As a result, the mapping backend uses the information obtained by the tracking front end to supervise the radiance field, which leads the system to obtain biased reconstructions. In summary, in NeRF-SLAM, the tracking thread continuously minimizes the BA reprojection error over an active window of keyframes. At the same time, the mapping thread optimizes all the keyframes obtained from the tracking thread, where the tracking thread only generates a new keyframe each time the mean of the optical flow between the previous keyframe and the current frame is higher than a threshold. From extensive experiments on several datasets comparing NeRF-SLAM with state-of-the-art methods, it was observed that NeRFSLAM outperformed existing methods in terms of reconstruction accuracy, detail, and robustness to dynamic scenes and occlusions, making NeRF-SLAM a promising approach for indoor scene reconstruction that combines the strengths of neural networks and SLAM. Fig. 39 illustrates the NeRF-SLAM algorithm inspired by the article [169]. Rosinol et al. (2023). The proposed probabilistic volumetric fusion method is a novel approach for dense mapping and reconstructing 3D environments from RGB-D (colour and depth) or monocular RGB data. It outperformed most existing state-of-the-art methods in accuracy, robustness, and speed. The proposed method extends the traditional volumetric fusion method by introducing a probabilistic framework based on Gaussian Process Regression (GPR). This probabilistic volumetric fusion method presents a novel approach to 3D environment reconstruction from RGB data that allows a more robust and accurate reconstruction by incorporating uncertainty in the depth measurements, which is crucial for robust decision-making in uncertain environments. Its probabilistic framework and real-time performance provide superior accuracy and robustness over existing state-of-the-art methods. The main improvement of this method over existing techniques is the ability to handle many frames in real-time. In brief, this method pursues the goal of fusing dense, noisy depth maps weighted by probabilistic uncertainty estimations into a volumetric map. For this purpose, the system uses the Droid-SLAM frontend to recover pose estimates and dense depth maps, which were also adapted to recover dense uncertainty maps. The Droid-SLAM framework is applied to obtain a set of inverse depths per keyframe to solve the BA problem. Then, the inverse depth uncertainties are computed from the information matrix of the underlying bundle adjustment problem, using the marginal covariances for the per-pixel depth variables, which allows recovering sparse depth maps that are upsampled using the Raft upsampling operator, and the depth variances are computed considering a nonlinear uncertainty propagation. Next, a fusion strategy is applied to produce a volumetric map using an uncertainty-aware volumetric mapping technique which, in contrast to Droid-SLAM (which uses an ad-hoc depth filter), uses the estimated uncertainties of each depth map on a probabilistic volumetric fusion model to provide a robust and mathematically sound alternative to reconstruct the scene geometry. Finally, the 3D mesh is extracted from the volume based on a maximum uncertainty bound. Only the voxels that present an uncertainty below this Fig. 39. Diagram of NeRF-SLAM algorithm. Adapted from Ref. [169]. E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 38 bound are meshed to extract the surfaces. The Rosinol et al. system was evaluated on different datasets, including synthetic and real-world data, and it was compared with several state-of-the-art methods, including fusion-based methods, such as KinectFusion and ElasticFusion, and deep-learning-based methods, such as MVSNet and NERF, demonstrating its robustness and generalization capability. The accuracy and quality of the proposed method make it suitable for use in augmented/virtual reality and computer graphics, including virtual game environments and movie production. This makes it a promising area for future research and development. In addition, its outstanding real-time performance makes it suitable for robotics applications, such as autonomous navigation in unknown environments. Fig. 40 illustrates the Rosinol et al. algorithm inspired by the article [170]. 7.2. ML +direct methods In recent years, machine learning has emerged as a promising avenue for research in monocular 3D reconstruction. This is due to its ability to address well-known limitations of monocular systems, such as scale ambiguity, motion blur, texture-less surfaces, and repetitive patterns. Classic direct systems face significant challenges, including reliance on a good initialization, the brightness constancy assumption, their performance in low-illumination environments, and their ability to generalize to unseen environments. Accordingly, in the past decade, researchers have made interesting contributions to overcome these challenges by embedding neural network architectures in classic SLAM, VO, or SFM proposals, enhancing performance and, in most cases, have demonstrated to outperform their classic versions. ML +Direct methods, like the rest of the taxonomy categories, may be divided into dense and sparse, depending on the density of recovered 3D reconstruction, so both of these alternatives are discussed in sections 7.2.1 and 7.2.2. Fig. 41 describes a timeline for the appearance of the most representative ML +direct methods in the last decade. Two of the most representative ML approaches belong to this category: CNN-SLAM and CodeSLAM, significantly contributing to the state-of-the-art, yielding impressive citation scores providing two interesting paths for researchers: the integration of semantic segmentation and the use of encoder-decoder architectures, respectively. 7.2.1. ML +dense +direct methods Similar to classic SLAM methods, machine learning dense formulations can be classified as dense or sparse depending on the sparsity of the final reconstruction. Some authors have proposed improving pose and depth estimation and estimating parameters such as scale factors or initialization terms. In addition, as shown in the previous section, the possibility of training networks to densify the map output of classic SLAM systems exists; thus, classic sparse methods were added to this category with their densified machine learning versions. As shown in Fig. 41, most ML direct approaches belong to the dense category due to the excellent results machine learning has achieved in densifying their 3D reconstructions. CNN-SLAM (2017). An exciting work that achieved the goal of fusing deep convolutional depth prediction with a dense direct system is the work of [2]. It uses CNN-predicted depth maps in monocular SLAM to overcome the issue of estimating the absolute reconstruction scale by learning from examples and getting rid of assumptions and geometric constraints. The obtained CNN depth map is used as an initial guess to perform a dense reconstruction, which is then refined by the LSD-SLAM direct approach. Most monocular SLAM systems have restraint scene absolute scale, so the final reconstruction scale is ambiguous. In contrast, the CNN-predicted depth map provides absolute scale information, aiding the pose estimation process for more Fig. 40. Diagram of Rosinol et al. algorithm. Adapted from Ref. [170]. E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 39 accurate pose trajectory and scene reconstruction. Also, most monocular systems fail under pure rotational camera movements due to the lack of a stereo baseline in contrast to a CNN-predicted depth map which does not face these problems because each frame is estimated individually. The system was built over the depth prediction method of [152] with the direct SLAM system by using the CNN-predicted depth map of every new keyframe as prior information for the SLAM system. The network architecture presents a first stage based on ResNet-50 initialized with pre-trained weights on ImageNet to estimate the environment scale. Then, a second part of the network replaces the last pooling and fully connected layers with a sequence of residual up-sampling blocks. Finally, drop-out is applied, and a convolutional layer outputs the predicted depth map. The CNN depth prediction and direct SLAM are fused using the uncertainty map. Uncertainty map Uki is the elementwise distance between the depth map of the current keyframe ki and the closest keyframe kj: Uki(u) = (Dki(u) − Dkj(Π(ζTki kjVki(u))))2,(29) where ζ is the camera intrinsics matrix, Vki(u) = ζ−1˙ uDki(u)represents a 3D element of the vertex map computed from the depth map of the current keyframe, u is a generic depth map element, with ˙ u being its homogeneous representation, v=Π(ζTki kjVki(u)), and  Ukj is the uncertainty related to the CNN estimation. The depth and uncertainty map of a frame are fused with those of the closest keyframe to improve the accuracy of every initialized keyframe where the uncertainty of the nearest keyframe is:  Ukj(v) = Dkj(v) Dki(u)Ukj(v) + σ 2 p,(30) next, the two uncertainty and depth maps are fused using the following weighted expressions: Dki(u) =  Ukj(v) • Dki(u) + Uki(u) • Dkj(v) Uki(u) +  Ukj(v),(31) Uki(u) =  Ukj(v) • Uki(u) Uki(u) +  Ukj(v).(32) In CNN-SLAM, the authors performed semantic segmentation by relying on the idea that the same network can be used to perform high-dimensional regression tasks. Therefore, CNN-SLAM could be considered the first example that jointly performed semantic segmentation and 3D reconstruction, opening up a new research avenue where multiple 3D regression tasks can be performed along Fig. 41. Timeline for the most representative monocular SLAM, VO or SFM, ML direct systems. Fig. 42. Diagram of CNN-SLAM algorithm. Adapted from Ref. [2]. E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 40 with depth prediction. To summarize, the CNN-SLAM system collects keyframes as a subset of visually distinct frames whose pose is refined by pose graph optimization. Simultaneously, camera pose estimation is performed by estimating the transformation of the current frame to the closest keyframe. The CNN-predicted depth map is obtained only for keyframes with high frame rates. An uncertainty map is created by measuring pixel-wise confidence for every prediction. At the same time, a second convolutional network conducts semantic label fusion to predict a semantic segmentation for each frame. Finally, the relative pose is optimized through a pose graph on keyframes. Fig. 42 illustrates the CNN-SLAM algorithm inspired by the article [2]. DeepTAM (2018). Zhou et al.’s DeepTAM [171] is an improved version of DTAM [27], formulated as a deep learning problem using two CNN architectures for tracking and mapping tasks. The proposal’s main contributions include a tracking network designed for incremental frame-to-keyframe tracking, a multiple hypothesis proposal for camera pose estimation, a mapping network that combines depth estimation with image priors, and a depth refinement strategy that combines CNN architectures with a narrow band technique. The tracking network of DeepTAM aligns the current image with a keyframe containing depth and the colour image to infer the camera pose, essentially computing 2D to 3D correspondence between the image and the keyframe. For this purpose, the authors used an encoder-decoder-based architecture to learn 6 DOF estimation poses related to a keyframe and use the optical flow, ensuring the network learns to exploit the relation between a pair of images. Hence, the decoder of the tracking network performs two tasks: optical flow prediction and pose hypothesis generation. They also used a coarse-to-fine strategy to learn to track large and small camera motion, so the authors trained three tracking networks to generate the pose hypothesis at different resolutions. Consequently, the tracking network performs an incremental pose estimation where each network is specialized for a distinct resolution level. Hence, each network computes a pose estimate where the latest pose guess is taken as a virtual keyframe at each resolution, thereby tracking the camera pose, and the final pose estimate is the product of all the incremental pose updates. The DeepTAM mapping network was developed based on the plane sweep stereo concept. It accumulates multiple-image information in a cost volume and extracts the depth map through a CNN using image-based priors and the accumulated depth information. To improve depth prediction, the authors used a network that iteratively refines the depth prediction using the cost volume within a narrow band around the geometry estimate of previous frames. In DeepTAM, the mapping architecture consists of fixedand narrowband modules. The fixed band module takes the input image and the cost volume to output an interpolation factor and depth estimation. The narrow band module runs iteratively to build a learned cost volume; finally, the depth map is obtained by applying a differentiable soft argmin operator, and a second encoder-decoder takes the depth estimation and the keyframe image producing a refined depth map. It is no secret that machine learning techniques tend to suffer from overfitting, so authors took special care in designing DeepTAM architecture and its configuration for the learning problem so that the network cannot learn simple shortcuts that may result in generalization problems. In addition, the authors used data augmentation techniques applied in the training process and used train datasets like SUN3D [172] and SUNCG [173] to learn to track 6 DOF motion. Thus, some of the key advantages of DeepTAM are its strong tracking capabilities, which, on experimental evaluations, have proven to achieve better results than its predecessor CNN-SLAM and its ability to process more than two images to refine the depth map, avoiding drift. Fig. 43 presents the DeepTAM algorithm inspired by the article [171]. DeepFusion (2019). In Ref. [174], the authors proposed a 3D reconstruction system named DeepFusion. Using a monocular SLAM system, the system aims to obtain dense scaled depth maps and poses in real time. Predicted depth gradients are used as a constraint on the neighbourhood pixels to ensure global reconstruction consistency and learned uncertainties are used to fuse the different Fig. 43. Diagram of DeepTAM algorithm. Adapted from Ref. [171]. E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 41 DVSO system. In contrast, MaskModule is trained using a mask loss corresponding to the binary cross entropy between the predicted mask and ground truth. In the MaskModule refinement stage, a supervised mask loss is added as a regularizer to stabilize training. New gradients are added to their corresponding structure in the cost volume to improve mask prediction capabilities and prevent overfitting. Finally, the DepthModule refinement stage was incorporated to make the DepthModule able to predict depths for moving objects by performing further stereo forward passes using the resulting depth map as prior for moving objects. Experimental results performed over the KITTI, TUM-Mono, and Oxford Robot-Car datasets proved the impressive capabilities of MonoRec regarding obtaining scene semidense point clouds. Ablation studies proved the importance of using MaskModule and DepthModule refinements to improve moving object detection and mapping. Fig. 50 describes the MonoRec algorithm inspired by the article [182]. DDSO (2022). Zhao et al.’s [145] DDSO is a monocular SLAM proposal that builds upon the DSO visual odometry framework [24]. The DDSO system incorporates deep neural networks trained in an unsupervised manner to enhance the accuracy and robustness of the DSO system. DSO estimates the pose from photometric information, eliminating the need to calculate feature descriptors. However, it has been demonstrated that direct methods are sensitive to photometric changes between frames and highly dependent on good initialization, which can be challenging in complex environments. In DSO, pose accuracy depends highly on the image alignment algorithm, which obtains inter-frame poses, optimizing the initial pose from a constant motion model. However, this constant motion model assumes that the current inter-frame pose is the same as the last, unsuitable for strong movements, motion blur, or highly repetitive textures. The constant motion model does not have prior camera pose information in DSO initialization, so a unit matrix is used empirically. In this way, DDSO improves the performance of DSO by enhancing this constant motion model using deep-learning-based pose estimation for initialization, replacing the unit matrix, and, for tracking, adding inter-frame pose information. This novel initialization is achieved using a CNN called TrajNet, which is trained unsupervised using four geometric constraints. One of the main contributions of this study was the novel pose-to-trajectory constraint. TrajNet is formulated and trained along with a depth estimation network called DepthNet, where the geometry constraints between the outputs of each pair of deep models serve as a training monitor. So the key supervisory signal for TrajNet is integrated of: the view reconstruction constraint to take into account the view reconstruction errors coming from pairs of consecutive frames based on the same depth map; the smoothness constraint, which is a loss term that promotes the representation of geometric details; the depth alignment constraint, to promote scale consistency in the outputs of DepthNet by aligning the scale of adjacent depth maps; and the novel pose-to-trajectory constraint that aims to improve the trajectory generation ability of the network promoting the scale consistency for three consecutive poses taking into account inter-frame poses. In DSO, the Gauss-Newton algorithm optimizes the total photometric error of a sliding window of 5–7 keyframes. Then, the whole process can be considered a nonlinear optimization problem, so an initial transformation should be provided, and it will be iteratively optimized; hence, DSO’s original proposal uses a unit matrix as the initial transformation. Furthermore, when tracking is lost, DSO initializes three motion models and 27 small rotations when image alignment fails, which is a complex process. DDSO TrajNet provides initial transformations and models for failure modes to overcome these issues. Experimental results proved that TrajNet integration allowed DDSO to outperform the original DSO system, allowing it to obtain robust and accurate trajectories and depth maps without complex calibration. Fig. 51 presents the DDSO algorithm inspired by the article [145]. 7.3. ML +hybrid methods Like traditional methods, there is a hybrid category for techniques that effectively combine direct and indirect principles. In this case, we considered the SVO [23] system, which combines direct pixel information extraction with feature extraction to perform both camera tracking and sparse depth map creation. In recent years, this system has been updated, integrating a CNN to enhance its generalization and initialization capabilities discussed in the next section. Despite the reduced number of works in this category, the Fig. 51. Diagram of MonoRec algorithm. Adapted from Ref. [182]. E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 48 scientific community has received this category well, being widely used for robotic implementations, thus reaching high citation scores. CNN-SVO (2019). The implementation of depth filters in the SVO mapping thread and the proposed use of direct pixel matching in the semi-direct framework of SVO have enabled the system to achieve efficient camera motion estimation at high frame rates. However, this proposal still has shortcomings, especially regarding the high-depth uncertainty in the map point initialization process. This problem is addressed in the study of [66], where authors incorporated a CNN to overcome this depth uncertainty limitation. The system was built entirely over the SVO framework [23], adding prior depth knowledge obtained by the single-image CNN MonoDepth of [187] for reducing the uncertainty in identifying feature correspondences, which was built over the Resnet50 backbone using a variant of its encoder-decoder architecture. SVO’s original proposal divided the system into mapping and tracking threads. Depth values for each feature are obtained by finding feature correspondence over the epipolar line, recovering depth by triangulation. The mapping thread runs the initialization of new map points with high depth uncertainty, updating this depth uncertainty using depth filters created to approximate the mean and variance of current depth values to separate inliers from outliers; hence, a depth filter converges when a point depth uncertainty is small. However, in the original SVO, depth uncertainty tends to be large, leading to two problems: erroneous feature correspondence on the epipolar line and many depth estimations far from converging to their true depth. Consequently, the CNN depth prediction is used to estimate better the mean and variance used in each depth filter, allowing for faster and more accurate convergence. As shown in Fig. 52, MonoDepth is added as a depth estimation module in the mapping thread, providing strong depth priors in the map points initialization process to initialize the depth filters, whereas the original SVO proposal initialized depth filters as average depth measurements of current image μ n=1/davg and variance were set as a function of the minimum depth of the image σ 2 n=1/ (6dmin )2; CNN-SVO replaced these simple values with more precise information of the depth estimation coming from the CNN for each filter position as μ n=1/dCNN, σ 2 n=1/(6dCNN )2. Experimental results demonstrated that adding prior CNN depth information improved the system’s performance for overexposed and underexposed images thanks to its illumination invariance properties, facilitating feature correspondence between views and overcoming key illumination issues of original SVO. Fig. 52 introduces the CNN-SVO algorithm inspired by the article [66]. 7.4. General comments for ML approaches As previously mentioned, machine learning techniques, particularly convolutional neural networks, have been extensively incorporated into various traditional proposals to address the issues and failure modes identified in each taxonomy category. It is worth noting that, as cited in works [104,163], some researchers believe that classic geometric approaches still outperform ML methods regarding reconstruction quality or tracking. However, as analyzed in this article, machine learning researchers have attempted to address this issue by developing and implementing new convolutional neural network architectures and creating more robust datasets to improve the training process. On the other hand, as demonstrated in many of the reviewed systems above, CNN can be used not only to recover scene depth or camera Fig. 52. Diagram of CNN-SVO algorithm. Adapted from Ref. [66]. E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 49 pose but also to densify depth maps, estimate initialization parameters, perform preprocessing steps such as feature extraction or optical flow estimation, and perform additional tasks such as semantic segmentation. Thus, from diverse perspectives, CNN can positively contribute to SLAM, VO, and SFM. Another broadly cited problem of machine learning proposals is overfitting [171], as the authors mentioned above keep contributing to the strengthening of different datasets suggesting new datasets like [8,79,154,156–158, 188] constantly facilitating researchers to perform more robust training processes and to incorporate more sophisticated training to their systems in the form of semi-supervised [182] and self-supervised approaches [5,132,180,189]. Similarly, some authors have reported problems related to the generalization of ML methods [60], which has been addressed by many researchers combining the use of datasets recorded in different situations like indoors [79], outdoors [127], autonomous driving [7] and even MAV [78] flying sequences. Thus, we can conclude that ML methods have explored various alternatives to solve the 3D reconstruction problem, making outstanding contributions to this research field. Furthermore, depending on the application that a SLAM, VO, or SFM system will have, we have gathered the same selection criteria described in classic approaches, adding specific criteria only applicable to ML systems, which are the CNN architectures that were integrated and the main estimation tasks that motivated the use of CNN for each system. Table 3 illustrates information gathered for 11 criteria reviewed in each ML system. Additionally, we have run some implementations of the ML methods available as open-source code. Some examples of the implemented algorithm executions are shown in Fig. 53, where: Fig. 53.a. represents the input image, Fig. 53.b. presents results obtained using the DynaSLAM (indirect +sparse) algorithm, Fig. 53.c. presents results obtained using the CNN-DSO (direct +sparse) algorithm, and Fig. 53.d. presents results obtained using the CNN-SVO (hybrid) algorithm. The implemented examples were selected due to their availability and the capacity to run depending on the monocular RGB input modality as the unique source of information. It must be mentioned that methods like [2,10,59,134,139,141,145,163,171,182,190,191] were made publicly available by their open-source codes, but their open-source published version did not include their monocular RGB pipeline and they depend on additional information, like externally estimated optical flow or depth prior information; thus they were not included in Fig. 53 examples. Fig. 53 illustrates results obtained by implementing ML monocular SLAM, VO, and SFM systems over publicly available datasets. 8. Discussion SLAM, VO, and SFM are problems that have been extensively studied over the last three decades, particularly in the last two decades, due to the limited computational power available to perform their constituent tasks. However, it should be noted that there are still unresolved issues that have motivated the development of the previously described systems. These include reinforcing existing systems, optimizing computational resources, adding scene comprehension capabilities, tracking and reconstructing moving objects, increasing map density, improving generalization capabilities for unseen environments, achieving per-pixel dense scaled reconstruction, and others. For instance, reinforcement of currently available methods has been extensively studied by authors like [27,30, Fig. 53. Examples of results obtained by ML approach implementations. (a) represents the input image, (b) presents results obtained using the DynaSLAM (indirect +sparse) algorithm [192], (c) presents results obtained using the CNN-DSO (direct +sparse) algorithm [197], (d) presents results obtained using the CNN-SVO (hybrid) algorithm [196]. Top row results correspond to the indoor example sequence seq_01, and bottom row results correspond to the outdoor sequence seq_29 of the TUM-MONO dataset [127]. The examples were obtained through multiple executions of each algorithm in our previous comparative work [107]. For further information on the implementation and performance of each algorithm and category, we encourage the reader to address the paper [128] and the repository: https://github.com/erickherreraresearch/ MonocularPureVisualSLAMComparison [129]. E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 50 38,64,81] working on original systems still implementing classic techniques and implementing neural networks over classic implementations [2,6,66,110,138,139,146,171]. Optimization of computational resources has been studied mainly for those implementations made over embedded devices with little computation capacity, like UAVs or drones, where authors had to implement unique feature, point, and frame selection strategies to work on a reduced amount of information [6,23]. The addition of scene comprehension capabilities is addressed in classic methods like [38,81] adding scene comprehension modules, or using machine learning to label point sets belonging to that the network was previously trained to recognize [2,146]. Moving object detection and depth prediction have also been studied by authors like [182,198], who have used artificial intelligence to create masks to determine the probability of each pixel belonging to moving objects, favoring object detection, removal, and prediction. The increase of map density has also been addressed by using classic dense techniques based on the optical flow [85,91] or pixel intensities [4,27,37], even using neural networks trained for this purpose [6,60]. Authors like [58] proposed using multiple datasets in the training stage to increment generalization capabilities. In contrast, others, like [182], used multistage training techniques, and others applied ML regularization and data augmentation techniques. Moreover, the goal of recovering scene scale from monocular images had been studied using neural networks especially trained for such [60,174,193] or by including this parameter on the CNN inference model. Despite the numerous proposals made by authors to overcome the failure of monocular 3D reconstruction modalities and open problems, many possibilities and combinations still require further study. Additionally, there are many more possibilities beyond this overview’s scope. Therefore, we have gathered cite score information of the methods reviewed in the previous sections, providing the reader with a sense of evolution in research fields. The information presented and discussed in the following sections was gathered using the Scopus database, per-year citation metrics, and the taxonomy proposed in this paper. The data was structured in a data frame considering the categoric variables: method name, whether it was a Classic or ML method, its dense or sparse classification, its direct or indirect classification, and its taxonomy final category. The data frame also considered numeric variables for the year, and the number of cites the article has received each year since its publication. The citation score variable was normalized using the preProcess function of the caret R package, which uses the Min-Max Scaling method to allow the data transform to the scale from 0 to 1, which is compatible with the selected trend identifying technique. To identify the trend of each time series, we employed the prophet function of Facebook’s Prophet package in R which took the normalized citing score and the year as inputs, giving the trend and yearly components as outputs, from where we captured the trend, which was finally rescaled and plotted using Splines with Loess method for ease of visualization. Considering the limited amount of existing data (from 2005 to 2023) and the discrete nature of the dependent variable, trends and patterns were easy to visualize since there were no observed considerable fluctuations or randomness in the data; thus, the additive decomposition assumption was accepted. Nevertheless, due to the small sample size available, these observed patterns and trends cannot be considered for inferential purposes; thus, their use is limited to an interval from 2005 to 2023. In addition, it must be mentioned that forecasting and an in-depth time series analysis of the presented data are out of the scope of this study. 8.1. Classic vs. machine learning Beginning with the primary classification of classic and machine learning (ML) methods, Fig. 54 displays citation scores for each category from 2005 to 2022. It is important to note that metrics from 2000 to 2005 (the year of the first cited system release [56]) were not provided due to their low citation score. Additionally, we did not include data for incomplete years, so metrics for 2023 were not considered for this analysis, as the year had not yet ended at the time of article submission. Ther citation scores presented in Figs. 54–57 correspond to the number of citations each study has gathered since its year of publication. Fig. 54. Citation score evolution for classic and ML methods since 2005. Dashed lines represent the trend line for each classification obtained, setting a 99 % confidence interval and the Loess method. E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 51 As evidenced in Fig. 54, over the last 18 years, classic methods have achieved higher citation scores attributed to the extensive time interval that these methods have been available, most of them as open-source implementations. It can be noticed that ML methods started to attract the research community’s attention in 2017 as pertaining citation scores kept increasing over time. We believe this trend will continue, so we expect ML citation scores to be close to the impressive values achieved by classic methods shortly. 8.2. Direct vs. indirect Regarding the direct vs. indirect classification, common issues can be highlighted in both categories. Indirect methods mostly rely on geometric Bundle Adjustment based on reprojection error. However, some authors like [24,59] have pointed out that this procedure has drawbacks, such as relying only on the information corresponding to their feature types (such as corners, blobs, or line segments). Feature matching between frames can also introduce a significant number of outliers. As mentioned in Refs. [11,24,59], direct formulations are sensitive to initialization due to the photometric effect of increasing non-convexity. Direct formulations are also sensitive to camera exposure and changes in white balance, being more sensitive to outliers (like motion blur or moving objects). In this way, many formulations have emerged to overcome the issues of each modality getting benefits from both. For example, some indirect implementations, like [59], introduced direct pixel information to densify the recovered depth maps while preserving an indirect backend. Indeed, there are direct formulations like [30] that implemented additional feature extraction techniques enhancing tracking Fig. 55. Citation score evolution for direct, indirect, and hybrid methods since 2005. Dashed lines represent the trend line for each classification obtained, setting a 99 % confidence interval and the Loess method. Fig. 56. Citation score evolution for dense and sparse methods since 2005. Dashed lines represent the trend line for each classification obtained, setting a 99 % confidence interval and the Loess method. E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 52 capabilities. Again, this was formally addressed by hybrid self-denominated semi-direct approaches [30,138], where the main difference with proposals like [30,59] resides in the ability to fuse both modalities in their optimization backend completely. The categories were processed with data from the Scopus database, so a citation progress analysis was performed over the years. Fig. 55 represents the direct vs. indirect evolution classification from 2005 to 2022. Fig. 55 shows that indirect methods are the most cited of the three taxonomy categories, closely followed by direct and hybrid approaches in the third position. This behaviour may be attributed to the ample time interval that indirect methods have been studied, being the first type of proposals that appeared in this research field. In contrast, the first direct processes were released in 2010, and hybrid techniques were addressed in 2014. It must be pointed out that, in this study, we have addressed 16 indirect, 17 direct, and two hybrid methods, so it is noticeable that hybrid approaches have reached an impressive number of citations despite the low number of works available in this category. 8.3. Dense vs. sparse Next are dense and sparse categories for the third component of the proposed taxonomy, which relies on the number of points that conform to the final 3D reconstruction. As mentioned in works like [4,10,23,27,65,66,85,88,99,193,199], the selection process for each of them strongly depends on the final application for which some 3D reconstruction systems may be used or created. For instance, if the application requires maps as dense as possible for navigation [7,182], exploration [5,6,16,18] or augmented reality applications [41,114], then a dense approach is appropriate. However, if the application does not require high 3D reconstruction definition or is required for fast movement [11,23] or little computer capabilities processors applications [6,23,60], then a sparse system is suitable. Following the same process used for the remaining classifications, the database was processed using dense vs. sparse classification as a categorical variable. The results are presented in Fig. 56. 8.4. Complete taxonomy Finally, for the sake of completeness, we analyzed the dataset using the complete taxonomy as a categorical variable. As established in section 5, this taxonomy comprises ten levels produced by combining the three classifications in every possible configuration. Classic +Dense +Direct, Classic +Sparse +Direct, Classic +Dense +Indirect, Classic +Sparse +Indirect, Classic +Hybrid, ML + Dense +Direct, ML +Sparse +Direct, ML +Dense +Indirect, ML +Classic +Sparse +Indirect and ML +Hybrid. Fig. 57 describes the evolution of the taxonomy in each category of citations over time. As evidenced in Fig. 57, the classic sparse indirect category achieved the highest citation scores, which remains the category that has most attracted the attention of researchers and inspired the most representative formulations. This behaviour may be attributed to the early availability of impressive methods like MonoSlam, PTAM, and ORB-SLAM [11,28,41] and the availability of their open-source codes, enabling their use in many implementations or for comparison with new proposals. In addition, the second most cited category is classic dense direct again, the high citation scores achieved may be caused by its capability to overcome one of the most critical issues of sparse indirect systems, which is the ability to recover well-defined depth maps that can be used for a large variety of applications. Finally, the third place is occupied by a relatively recent classic sparse direct approach (2018) that has attracted considerable attention from the scientific community. This behaviour could be motivated by the impressive results obtained by the systems derived from DSO [24], which significantly overcame the issue of noise introduced by direct pixel information extraction while keeping the set of points small enough to be processed with low computational resources. Thus, the effect observed in the citation Fig. 57. Citation score evolution for taxonomy proposed categories over time. Dashed lines represent the trend line for each classification obtained, setting a 99 % confidence interval and the Loess method. E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 53 scores of the different categories in taxonomy can be explained because of outstanding works that have significantly contributed to the state-of-the-art like MonoSlam, PTAM, DTAM, LSD-SLAM, REMODE, SVO, ORB-SLAM, DSO, among others. Fig. 58 lists systems reviewed as yearly released in this article with their corresponding citation scores. 8.5. Comparative analysis In our previous study [128], we compared the most important publicly available monocular SLAM, VO, and SFM methods suitable for 3D reconstruction tasks. For that comparison, we selected the most representative works for each category: indirect +sparse [38], indirect +dense [82], direct +dense [3], direct +sparse [24,30,64], hybrid, and three machine learning extensions of the classic approaches [66,102,197]. We extensively tested each algorithm in the most complete monocular pure visual dataset, the TUM-mono dataset [127], which includes 50 sequences for outdoor and indoor environments, covering a wide range of motion patterns. Following the recommendations of [25,127,200], we performed ten executions of each sequence forwards and backwards, giving 1000 executions for each algorithm and a total of 10000 observations that allowed us to build a large database which was used to perform a statistical analysis of the metrics proposed in the benchmark. The metrics considered the translation error et, rotation error er, scale error es, alignment error ealign for the start and end segments and the translational RMSE ermse. The gathered data was evaluated using the accumulated error and motion bias plots proposed by Ref. [127], and then each variable average behaviour was tested using the Kruskal-Wallis and the pairwise Wilcoxon signed-rank tests. Table 4 presents an extract of the results obtained in the study [128]. As shown in Table 4, the sparse direct methods present clear advantages over the other evaluated methods, registering a significant error reduction compared to the rest of the classic taxonomy. Thus, it is evident that the introduction of the DSO method represents one of the most important contributions in this research field, allowing for impressive SLAM development and VO systems and a considerable error reduction, which directly contributes to enhancing the precision of these systems. Moreover, it can be noticed that the addition of ML modules in classic pipelines significantly improved the performance of the classic methods, which is evidenced by a critical error reduction in the CNN-SVO and DynaSLAM methods when compared with their classic versions. For further details on the experimental protocol and the comparative results, we encourage the reader to address the paper [107] and the repository [108]. From all the information detailed in this work, we can provide the reader with a summary of the main advantages and disadvantages that have been identified for each category of the proposed taxonomy, according to what the authors detailed in their publications and with the experience that we gathered by implementing more than ten publicly available methods. The main benefits Fig. 58. Reviewed methods citation score. The right legend depicts a colour for each taxonomy classification ordered from the highest cited category to the lowest. The methods are ordered on the vertical axis according to their year of appearance. E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 54 and shortcomings of each category are summarized in Table 5. Table 5 shows that each category has its own set of advantages and issues inherent to the framework’s nature. This indicates a tradeoff between sparsity and preprocessing. Sparser methods result in faster and lighter performance due to the limited information, but the final reconstruction may lack crucial details which are significant for certain applications. Likewise, preprocessing entails additional steps, varying in computational workload. If preprocessing is focused on reducing the amount of information, like feature extraction, then the method will become lighter and faster, but it may present issues in texture-less areas due to the lack of corners and edges commonly tracked as features. However, optical flow can provide more information at the cost of high computational requirements. Therefore, many authors have contributed to ML extensions, employed CNNs to densify final reconstructions, or used lighter ML alternatives to recover optical flow. Finally, it must be mentioned that a light advantage still exists in using classic techniques due to the large number of publicly available systems for implementation and testing. This also represents a scalability advantage for the classic methods, considering their availability and multiple researchers’ reports of their ease of integration with existing ML modules to contribute to multiple tasks. 9. Conclusions Scene 3D reconstruction is a complex problem that can be addressed using various techniques and technologies. This work provides an overview of one of the most attractive alternatives to 3D reconstruction: the visual reconstruction of an environment using a monocular RGB camera as the sole source of information. We discussed several input modes and their advantages and disadvantages, considering three main techniques applicable to this task (SLAM, VO, and SFM) and establishing a taxonomy containing the most likely system configurations in the literature. Three classifications were defined, providing ten possible combinations comprising an extended taxonomy. Also, 42 of the most representative monocular systems were reviewed comprehensively. In order to guide the selection and application of such systems, we have gathered nine criteria for each classic system being decision components for the implementation of a 3D reconstruction system, which are: type of algorithm, tracking method, map density, pixels used, estimation method, global optimization, relocalization, loop closure, and availability. We have gathered eleven criteria for ML methods, the same as classic procedures, although introducing two additional criteria for the network: CNN architecture and CNN main estimation tasks. Furthermore, researchers can use the comprehensive information gathered to select the most suitable algorithm or taxonomy category for their projects. Additionally, we have discussed the main strengths and shortcomings of each of the classifications that comprise the taxonomy, and we have analyzed the progress of each classification and category of the taxonomy in the last 18 years according to their citation scores that give an intuition of the impact and acceptance that each of them has produced in this research field. Finally, for future work, we will use the proposed taxonomy to compare each category’s most representative open-source algorithms to determine their advantages and disadvantages to select the more suitable methodology for monocular 3D reconstruction of indoor scenes. Data availability statement Comparative data and 20 video examples of multiple algorithm executions are publicly available in the repository: https://github. com/erickherreraresearch/MonocularPureVisualSLAMComparison. Table 4 Medians and Kruskal-Walli’s comparisons for the error metrics of the TUM-mono benchmark. Adapted from Ref. [128]. Method Translation error Rotation error Scale error Start-segment alignment error End-segment alignment error RMSE Kruskal-Wallis general test χ 2=3582.9 pval.= 2.2e−16 χ 2=2278.4 pval.= 2.2e−16 χ 2=2419.1 pval.= 2.2e−16 χ 2=4575.7 pval.= 2.2e−16 χ 2=3718 pval.= 2.2e−16 χ 2=530.78 pval.= 2.2e−16 DSO (Dir. +Spa.) 0.8064585 a 0.8800369 b 1.064086 ab 0.003974759 a 0.004184367 a 0.1950799 ab LDSO (Dir. +Spa.) 0.7892125 a 0.9135608 ab 1.061302 ab 0.007925665 b 0.008009198 b 0.1944492 a CNN-DSO (Dir. + Spa. +ML) 0.7980411 a 0.9618528 a 1.058849 a 0.008987173 b 0.006199582 c 0.2083872 ab DSM (Dir. +Spa.) 0.8519143 b 1.1117710 c 1.064615 b 0.015794222 c 0.015537213 d 0.2167750 b DynaSLAM (Ind. + Spa. +ML) 1.7473504 c 1.5730542 d 1.126499 c 0.004286919 a 0.005516179 e 0.2389837 cd ORB-SLAM2 (Ind. +Spa.) 2.8738313 d 2.3585843 e 1.260155 d 0.004311949 a 0.005102672 e 0.3165024 e CNN-SVO (Hybrid +ML) 1.6248001 c 1.4159545 d 1.086399 e 0.067201999 d 0.062036008 f 0.2373532 c DF-ORB-SLAM (Ind. +Den.) 3.6423921 e 3.4940400 f 1.238232 f 0.053360456 e 0.084420570 g 0.3643844 e SVO (Hybrid) 5.4819407 f 3.3772024 f 1.343603 g 0.108150349 f 0.117753996 h 0.3642558 e LSD-SLAM (Dir. + Den.) 9.1403348 g 14.9621188 g 2.044298 h 0.158469383 g 0.190127787 i 0.3507099 d Means with different letters in the same column differ significantly according to the Kruskal-Walli’s test, and pairwise Wilcoxon signed rank test for pvalue ≤0.05. E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 55 Disclosure statement Part of this article is part of the work presented in Erick Herrera-Granda’s Ph.D. thesis entitled "Real-time Monocular 3D Reconstruction of scenarios using artificial intelligence techniques", submitted to University of Granada in 2024. CRediT authorship contribution statement Erick P. Herrera-Granda: Writing – review & editing, Writing – original draft, Investigation, Formal analysis. Juan C. TorresCantero: Supervision, Project administration, Formal analysis. Diego H. Peluffo-Ord´ o˜ nez: Visualization, Validation, Resources, Project administration, Funding acquisition. Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to Table 5 Pros and shortcomings for each category of the proposed taxonomy. Category Pros Cons Classic + Indirect + Sparse Computationally efficient by using only a subset of features. Feature extraction enhances matching across frames. Preprocessing refinement produces high accuracy. Sparse reconstructions can suffer from a lack of detail critical for some applications. Dependent on repeatability and invariance of extracted features. Fail in low texture environments lacking features. Some of these methods may produce localization errors when visiting lowtextured areas. Classic + Indirect + Dense Commonly utilises optical flow to maximize pixel data for dense reconstructions. Optical flow provides motion cues to aid tracking and mapping. Enhanced refinement through optimization techniques using optical flow information. Optical flow estimation is a computationally expensive preprocessing step. Performance relies heavily on optical flow accuracy. Sensitivity to scenarios with independently moving objects. Classic + Direct + Dense Able to utilize most pixel data, leading to more detailed reconstructions. Bypass the need for feature extraction or optical flow estimation steps. Can operate directly on photometric data for estimating geometry. Rely on brightness constancy assumption, prone to failure with illumination changes. Require precise initialization due to non-convex optimization. Computationally intensive due to large data volumes. A large amount of information implies large outlier volumes, so robust and efficient filtering techniques are required. Classic + Sparse + Direct Avoid relying on geometric priors that may introduce bias. Able to sample pixels with various intensity gradients, capturing edges, corners, textures, and many other types of information. Can operate directly on photometric data for estimating geometry. More efficient computation compared to dense methods. Sparse reconstructions can suffer from a lack of detail critical for some applications. Susceptible to noise without dense pixel information. Struggle with texture-less regions lacking gradients. Classic + Hybrid Combines the benefits of both direct pixel data and feature extraction. Proficient tracking from direct alignment and feature matching. Computationally efficient compared to purely dense methods. Complexity in integrating both modalities into a unified backend. Potentially suboptimal performance compared to dedicated direct/ indirect methods. Very few exemplar methods exist currently. Algorithms may present relocalization and loop closure errors in texture-less scenarios. ML +Indirect +Sparse Augments feature extraction, description and matching. Provides strong priors to aid SLAM initialization and scale recovery. Maintains efficiency of sparse optimization backend. Sparse reconstructions can suffer from a lack of detail critical for some applications. Risk of overfitting with limited training data complexity. Generalization remains a challenge. A sparse set of points based on feature matching can be insufficient to recover accurate scales and reconstructions. ML +Indirect +Dense Integration of learning enhances optical flow accuracy. CNNs boost feature extraction, matching, and pose estimation. More robust to challenging lighting conditions and motion patterns compared to classical flow. Enhanced refinement through optimization techniques using optical flow information. Computationally expensive due to dense flow estimation. Complexity in integrating CNN outputs into optimization backend. Generalization can be poor when exploring environments or motion patterns considerably different from the training data. ML +Direct + Dense CNNs can enhance initialization and handle photometric variations. End-to-end learning is feasible for pose and depth estimation. They can recover denser reconstructions. CNNs can be used to overcome scale ambiguity and densify the reconstructions based on pixel information. Still rely on brightness constancy assumption. Susceptible to overfitting due to limited training data complexity. Generalization can be poor in novel environments. High computational cost due to large data volumes. Still susceptible to large amounts of outliers. ML +Direct + Sparse CNN integration improves initialization and tracking robustness. Depth priors from CNNs mitigate scale ambiguity. Maintain computational efficiency of sparse methods. A reduced set of filtered pixel information reduces computational complexity. Less susceptible to outliers compared to dense methods. Reconstruction quality and CNN performance benefit considerably from a good point selection strategy. Restricted by sparsity, lacking detail in reconstructions. Generalization remains a challenge. Access to training data with ground truth depth is limited. Susceptible to noisy information coming from the CNNs. CNN integrations can only contribute in scenarios similar to training data. ML +Hybrid Combines the benefits of learning and direct alignment techniques. Improves initialization while retaining tracking precision. Efficient compared to dense learned methods. More stable performance compared to classic hybrid techniques. Very few exemplar methods exist currently. Complexity in integrating modalities into a unified backend. Generalization can be poorer than dedicated learned methods. Algorithms may present relocalization and loop closure errors in textureless scenarios. Considerably limited to producing sparse reconstructions due to their indirect modules. E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 56 influence the work reported in this paper. References [1] M. Zollh¨ ofer, et al., State of the art on monocular 3D face reconstruction, tracking, and applications, Comput. Graph. Forum 37 (2) (May 2018) 523–550, https://doi.org/10.1111/cgf.13382. [2] K. Tateno, F. Tombari, I. Laina, N. Navab, CNN-SLAM: real-time dense monocular SLAM with learned depth prediction, in: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Jul. 2017, pp. 6565–6574, https://doi.org/10.1109/CVPR.2017.695. [3] J. Engel, T. Sch¨ ops, D. Cremers, LSD-SLAM: large-scale direct monocular SLAM, in: Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 8690 LNCS, Springer Verlag, 2014, pp. 834–849, https://doi.org/10.1007/978-3-319-10605-2_54. PART 2. [4] M. Pizzoli, C. Forster, D. Scaramuzza, REMODE: probabilistic, monocular dense reconstruction in real time, in: 2014 IEEE International Conference on Robotics and Automation (ICRA), IEEE, May 2014, pp. 2609–2616, https://doi.org/10.1109/ICRA.2014.6907233. [5] L. Madhuanand, F. Nex, M.Y. Yang, Self-supervised monocular depth estimation from oblique UAV videos, ISPRS J. Photogrammetry Remote Sens. 176 (2021) 1–14, https://doi.org/10.1016/j.isprsjprs.2021.03.024. [6] A. Steenbeek, F. Nex, CNN-based dense monocular visual SLAM for real-time UAV exploration in emergency conditions, Drones 6 (3) (Mar. 2022) 79, https:// doi.org/10.3390/drones6030079. [7] A. Geiger, P. Lenz, R. Urtasun, Are we ready for autonomous driving? The KITTI vision benchmark suite, in: 2012 IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Jun. 2012, pp. 3354–3361, https://doi.org/10.1109/CVPR.2012.6248074. [8] X. Huang, et al., The ApolloScape dataset for autonomous driving, in: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), IEEE, Jun. 2018, pp. 1067–10676, https://doi.org/10.1109/CVPRW.2018.00141. [9] T. Chen, F. Pu, H. Chen, Z. Liu, WHUVID: a large-scale stereo-IMU dataset for visual-inertial odometry and autonomous driving in Chinese urban scenarios, Rem. Sens. 14 (9) (Apr. 2022) 2033, https://doi.org/10.3390/rs14092033. [10] J. Czarnowski, T. Laidlow, R. Clark, A.J. Davison, DeepFactors: real-time probabilistic dense monocular SLAM, IEEE Rob. Autom. Lett. 5 (2) (Apr. 2020) 721–728, https://doi.org/10.1109/LRA.2020.2965415. [11] R. Mur-Artal, J.M.M. Montiel, J.D. Tardos, ORB-SLAM: a versatile and accurate monocular SLAM system, IEEE Trans. Robot. 31 (5) (Oct. 2015) 1147–1163, https://doi.org/10.1109/TRO.2015.2463671. [12] R. Wang, M. Schworer, D. Cremers, Stereo DSO: large-scale direct sparse visual odometry with stereo cameras, in: 2017 IEEE International Conference on Computer Vision (ICCV), IEEE, Oct. 2017, pp. 3923–3931, https://doi.org/10.1109/ICCV.2017.421. [13] H. Strasdat, J.M.M. Montiel, A. Davison, Scale drift-aware large scale monocular SLAM, in: Robotics: Science and Systems VI, Robotics: Science and Systems Foundation, Jun. 2010, pp. 73–80, https://doi.org/10.15607/RSS.2010.VI.010. [14] E. Tretschk, et al., State of the art in dense monocular non-rigid 3D reconstruction, Comput. Graph. Forum 42 (2) (May 2023) 485–520, https://doi.org/ 10.1111/cgf.14774. [15] R. Khilar, S. Chitrakala, S. SelvamParvathy, 3D image reconstruction: techniques, applications and challenges, in: 2013 International Conference on Optical Imaging Sensor and Security (ICOSS), IEEE, Jul. 2013, pp. 1–6, https://doi.org/10.1109/ICOISS.2013.6678395. [16] N.A. Shallik, A.H. Moustafa, M.A.E. Marcus, Virtual Endoscopy and 3D Reconstruction in the Airways, Springer International Publishing, Cham, 2019, https:// doi.org/10.1007/978-3-030-23253-5. [17] Y.-H. Su, K. Lindgren, K. Huang, B. Hannaford, A comparison of surgical cavity 3D reconstruction methods, in: 2020 IEEE/SICE International Symposium on System Integration (SII), IEEE, Jan. 2020, pp. 329–336, https://doi.org/10.1109/SII46433.2020.9026289. [18] N. Mahmoud, T. Collins, A. Hostettler, L. Soler, C. Doignon, J.M.M. Montiel, Live tracking and dense reconstruction for handheld monocular endoscopy, IEEE Trans. Med. Imag. 38 (1) (Jan. 2019) 79–89, https://doi.org/10.1109/TMI.2018.2856109. [19] T. K¨ ohler, et al., ToF meets RGB: novel multi-sensor super-resolution for hybrid 3-D endoscopy, in: Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 8149 LNCS, Springer, Berlin, Heidelberg, 2013, pp. 139–146, https://doi.org/ 10.1007/978-3-642-40811-3_18. PART 1. [20] M.O.A. Aqel, M.H. Marhaban, M.I. Saripan, N. Bt. Ismail, Review of visual odometry: types, approaches, challenges, and applications, SpringerPlus 5 (1) (Dec. 2016) 1897, https://doi.org/10.1186/s40064-016-3573-7. [21] D. Scaramuzza, F. Fraundorfer, Visual odometry [tutorial], IEEE Robot. Autom. Mag. 18 (4) (Dec. 2011) 80–92, https://doi.org/10.1109/MRA.2011.943233. [22] D. Nist´ er, O. Naroditsky, J. Bergen, Visual odometry for ground vehicle applications, J. Field Robot. 23 (1) (Jan. 2006) 3–20, https://doi.org/10.1002/ rob.20103. [23] C. Forster, M. Pizzoli, D. Scaramuzza, SVO: fast semi-direct monocular visual odometry, in: 2014 IEEE International Conference on Robotics and Automation (ICRA), IEEE, May 2014, pp. 15–22, https://doi.org/10.1109/ICRA.2014.6906584. [24] J. Engel, V. Koltun, D. Cremers, Direct sparse odometry, IEEE Trans. Pattern Anal. Mach. Intell. 40 (3) (Mar. 2018) 611–625, https://doi.org/10.1109/ TPAMI.2017.2658577. [25] E. Mingachev, R. Lavrenov, E. Magid, M. Svinin, Comparative analysis of monocular SLAM algorithms using TUM and EuRoC benchmarks, in: Smart Innovation, Systems and Technologies, vol. 187, Springer Science and Business Media Deutschland GmbH, 2021, pp. 343–355, https://doi.org/10.1007/978981-15-5580-0_28. [26] M.W.M.G. Dissanayake, P. Newman, S. Clark, H.F. Durrant-Whyte, M. Csorba, A solution to the simultaneous localization and map building (SLAM) problem, IEEE Trans. Robot. Autom. 17 (3) (Jun. 2001) 229–241, https://doi.org/10.1109/70.938381. [27] R.A. Newcombe, S.J. Lovegrove, A.J. Davison, DTAM: dense tracking and mapping in real-time, in: 2011 International Conference on Computer Vision, IEEE, Nov. 2011, pp. 2320–2327, https://doi.org/10.1109/ICCV.2011.6126513. [28] A.J. Davison, I.D. Reid, N.D. Molton, O. Stasse, MonoSLAM: real-time single camera SLAM, IEEE Trans. Pattern Anal. Mach. Intell. 29 (6) (Jun. 2007) 1052–1067, https://doi.org/10.1109/TPAMI.2007.1049. [29] B. Williams, G. Klein, I. Reid, Real-time SLAM relocalisation, in: Proceedings of the IEEE International Conference on Computer Vision, IEEE, 2007, pp. 1–8, https://doi.org/10.1109/ICCV.2007.4409115. [30] J. Zubizarreta, I. Aguinaga, J.M.M. Montiel, Direct sparse mapping, IEEE Trans. Robot. 36 (4) (2020) 1363–1370, https://doi.org/10.1109/ TRO.2020.2991614. [31] M. Servi` eres, V. Renaudin, A. Dupuis, N. Antigny, Visual and visual-inertial SLAM: state of the art, classification, and experimental benchmarking, J. Sens. 2021 (Feb. 2021) 1–26, https://doi.org/10.1155/2021/2054828. [32] E.P. Herrera-Granda, “A Comparison of Monocular Visual SLAM and Visual Odometry Methods Applied to 3D Reconstruction,” GitHub repository, 2023 [Online]. Available: https://github.com/erickherreraresearch/MonocularPureVisualSLAMComparison. (Accessed 28 April 2023). [33] T. Taketomi, H. Uchiyama, S. Ikeda, Visual SLAM algorithms: a survey from 2010 to 2016, IPSJ Transactions on Computer Vision and Applications 9 (1) (Dec. 2017) 16, https://doi.org/10.1186/s41074-017-0027-2. [34] Y. Chen, Y. Zhou, Q. Lv, K.K. Deveerasetty, A review of V-SLAM, in: 2018 IEEE International Conference on Information and Automation (ICIA), IEEE, Aug. 2018, pp. 603–608, https://doi.org/10.1109/ICInfA.2018.8812387. [35] A. Macario Barros, M. Michel, Y. Moline, G. Corre, F. Carrel, A comprehensive survey of visual SLAM algorithms, Robotics 11 (1) (Feb. 2022) 24, https://doi. org/10.3390/robotics11010024. [36] G. Chahine, C. Pradalier, Survey of monocular SLAM algorithms in natural environments, in: 2018 15th Conference on Computer and Robot Vision (CRV), IEEE, May 2018, pp. 345–352, https://doi.org/10.1109/CRV.2018.00055. E.P. Herrera-Granda et al. Heliyon 10 (2024) e37356 57