3D Computer Vision: Stereo, SfM, NeRF, and Depth Estimation

Computer Vision Computer Vision 10 min read 2028 words Advanced ExcellentWiki Editorial Team

Understanding 3D Computer Vision

Three-dimensional computer vision is the branch of computer science that recovers the 3D structure of a scene from 2D image data. Unlike 2D vision, which captures only the projection of a scene onto a sensor plane, 3D vision aims to estimate depth, surface geometry, and spatial relationships. The fundamental challenge is that a single 2D image is an ambiguous projection — many different 3D scenes could produce the same 2D photograph. Overcoming this ambiguity requires either multiple viewpoints, active sensing, or learned priors from large datasets.

The applications of 3D vision span nearly every industry. Autonomous vehicles use stereo cameras and LiDAR for obstacle detection and path planning. Robotics relies on 3D perception for grasping and navigation. Augmented reality systems need accurate 3D scene understanding to place virtual objects convincingly. Cultural heritage preservation uses photogrammetry to create digital twins of historical artifacts. The global 3D imaging market was valued at approximately $30 billion in 2024 and continues to grow at over 20% annually, driven by advances in both hardware and deep learning.

Stereo Vision and Disparity Estimation

Stereo vision mimics human binocular perception by using two cameras separated by a known baseline distance. The core principle is triangulation: a point in the scene projects to different pixel locations in the left and right images, and the horizontal shift — called disparity — is inversely proportional to depth. The relationship is given by depth = (focal length × baseline) / disparity, so objects closer to the cameras have larger disparity values.

The stereo pipeline begins with rectification, which warps both images so that epipolar lines are horizontal. This reduces the correspondence search from two dimensions to one. Disparity computation then finds matching pixels along each horizontal line using cost metrics such as sum of absolute differences, normalized cross-correlation, or more recently, learned features from deep networks. Semi-Global Matching (SGM), introduced by Hirschmuller in 2008, remains a widely used optimization method that enforces smoothness constraints along multiple paths. Modern learning-based methods like RAFT-Stereo and STTR achieve significantly higher accuracy on challenging regions such as textureless surfaces and reflective objects.

Challenges in Stereo Matching

Despite decades of research, stereo matching remains difficult in several scenarios. Textureless regions — blank walls, clear skies, unmarked roads — provide no features for correspondence algorithms to match. Occluded regions are visible in only one camera, creating holes in the disparity map. Reflective and transparent surfaces, including windows and water, violate the Lambertian reflectance assumption that underlies most matching algorithms. Specular highlights produce intensity patterns that differ between viewpoints. According to the Middlebury stereo benchmark, even state-of-the-art methods achieve error rates above 10% on the most challenging test cases, highlighting the ongoing difficulty of this problem.

LiDAR and Active Depth Sensing

LiDAR (Light Detection and Ranging) provides depth measurements by emitting laser pulses and measuring their time of flight. Mechanical LiDAR systems, such as those from Velodyne and Ouster, use rotating mirrors to scan a 360-degree horizontal field of view. Solid-state LiDAR systems, including those from Luminar and Aeva, use phased array or flash illumination with no moving parts, offering lower cost and higher reliability. The Waymo Driver platform uses a custom LiDAR system with 360-degree coverage, range up to 300 meters, and wavelength of 1550 nanometers, which is eye-safe and performs well in fog compared to the more common 905 nanometer wavelength.

Depth cameras from Intel RealSense and Microsoft Azure Kinect use structured light or time-of-flight principles for short-range indoor depth sensing. The iPhone Pro’s LiDAR scanner enables 5-meter depth sensing for AR applications. Each active sensing technology has trade-offs in range, resolution, power consumption, and cost. LiDAR provides accurate absolute depth but produces sparse point clouds compared to the dense depth maps from stereo. Fusion approaches that combine LiDAR with camera data leverage the strengths of both sensors.

Structure from Motion and COLMAP

Structure from Motion (SfM) estimates both the 3D structure of a scene and the camera poses from a collection of unordered images. The process begins with feature detection and matching — typically using SIFT (Scale-Invariant Feature Transform) descriptors, which are invariant to scale, rotation, and illumination changes. Matches across image pairs establish geometric relationships that are refined through bundle adjustment, a non-linear optimization that simultaneously minimizes reprojection error for all 3D points and camera parameters.

COLMAP, developed by Johannes Schonberger at ETH Zurich, is the de facto standard SfM pipeline. It incorporates incremental reconstruction with robust verification, leveraging RANSAC to filter outlier matches. COLMAP can reconstruct scenes from hundreds or thousands of unordered internet photos, as demonstrated by its use in creating the Rome in a Day reconstruction. For large-scale scenes, COLMAP uses scene graph partitioning and hierarchical reconstruction to manage computational complexity. Incremental SfM adds images one at a time, verifying each new camera for geometric consistency before incorporating it into the reconstruction.

Multi-View Stereo

After SfM recovers sparse 3D points and camera poses, Multi-View Stereo (MVS) densifies the reconstruction by estimating depth for every pixel in each image. MVS methods start from the known camera poses and match pixels across multiple views, producing dense point clouds or depth maps. COLMAP includes a built-in MVS pipeline that has been successfully applied to archaeological site reconstruction, film visual effects, and architectural documentation.

Neural Radiance Fields and 3D Gaussian Splatting

Neural Radiance Fields (NeRF), introduced by Mildenhall et al. at UC Berkeley in 2020, represent a scene as a continuous 5D function that outputs color and density for any 3D location and viewing direction. A multi-layer perceptron (MLP) is trained on a sparse set of input images using gradient descent, learning to synthesize novel views from arbitrary camera positions. NeRF achieves photorealistic novel view synthesis, capturing complex effects like reflections, translucency, and anisotropic materials that traditional MVS methods struggle with.

The original NeRF required hours to train and seconds per frame for rendering, limiting its practical application. Subsequent work has dramatically improved efficiency. Instant Neural Graphics Primitives (Instant NGP) from NVIDIA uses a multi-resolution hash encoding to reduce training to minutes. 3D Gaussian Splatting, introduced by Kerbl et al. in 2023, represents scenes as millions of anisotropic 3D Gaussians rendered via differentiable rasterization, achieving real-time frame rates at high visual quality. As of 2025, 3D Gaussian Splatting has become the preferred method for applications requiring interactive visualization, including virtual reality, digital twins, and visual effects production.

Point Cloud Processing with PointNet

Point clouds store 3D geometry as unordered sets of points with (x, y, z) coordinates, optionally including attributes like color, normal vectors, and intensity. Processing point clouds is challenging because they are irregular — unlike images, which have a fixed grid structure. PointNet, introduced by Qi et al. at Stanford in 2017, was the first deep learning architecture to process raw point clouds directly. The key innovation is the use of symmetric functions (max pooling) to aggregate per-point features into a global descriptor, ensuring permutation invariance.

PointNet++ extends this with hierarchical feature learning, capturing local structures at multiple scales. Applications of 3D deep learning include autonomous driving perception (detecting vehicles and pedestrians in LiDAR point clouds), industrial inspection (measuring surface defects), and building information modeling (segmenting architectural elements). Open3D provides comprehensive tools for point cloud processing, including registration with ICP (Iterative Closest Point), surface reconstruction, and visualization.

Visual SLAM and Odometry

Visual Simultaneous Localization and Mapping (SLAM) builds a 3D map of an unknown environment while simultaneously tracking the camera’s position within that map. Feature-based SLAM methods, exemplified by the ORB-SLAM family developed by Mur-Artal and Tardos, detect and match sparse features (ORB descriptors) across frames, perform local bundle adjustment to refine recent poses and map points, and maintain a covisibility graph for loop closure detection. ORB-SLAM3, released in 2020, supports monocular, stereo, and RGB-D camera configurations and incorporates visual-inertial fusion for improved robustness.

Direct methods bypass feature extraction by optimizing over pixel intensities directly. DSO (Direct Sparse Odometry), developed by Engel et al., minimizes photometric error over a sliding window of recent frames. Semi-dense methods like LSD-SLAM reconstruct edges and gradient regions. Modern SLAM systems increasingly incorporate deep learning for depth prediction, feature matching with SuperPoint and SuperGlue, and end-to-end learned odometry. The KITTI odometry benchmark tracks state-of-the-art performance, with top methods achieving translation errors below 1% of distance traveled.

Volumetric Reconstruction and TSDF

Volumetric reconstruction fuses depth measurements into a 3D voxel grid using Truncated Signed Distance Functions (TSDF). Each voxel stores the signed distance to the nearest surface: positive values in front of the surface, negative values behind, and zero at the surface. New depth measurements are fused by averaging weighted TSDF values, allowing the reconstruction to improve incrementally. KinectFusion, introduced by Newcombe et al. at Microsoft Research in 2011, popularized real-time volumetric reconstruction using a consumer depth camera.

Voxel hashing, implemented in systems like InfiniTAM and Voxblox, stores only voxels near surfaces in a hash map, enabling large-scale reconstruction of entire buildings and city blocks. The extracted TSDF surface mesh can be obtained through Marching Cubes, which generates triangle meshes at the zero-crossing of the signed distance field. Modern reconstruction systems like NeuralRecon learn mesh reconstruction directly from video sequences using recurrent neural networks.

Depth Estimation from Single Images

Monocular depth estimation predicts a dense depth map from a single RGB image — a fundamentally ambiguous problem. A single image of a small object could be a pebble close to the camera or a boulder far away. Deep learning models overcome this ambiguity by learning statistical regularities from large training datasets. The MiDaS model, trained on multiple diverse datasets with a scale-and-shift-invariant loss, provides robust monocular depth estimates across indoor and outdoor scenes. DPT (Dense Prediction Transformer) uses a Vision Transformer backbone for high-resolution depth maps.

The standard benchmark is the NYU Depth v2 dataset for indoor scenes and KITTI for outdoor driving scenes. State-of-the-art methods achieve relative errors below 8% on KITTI, approaching the accuracy of stereo systems. Applications of monocular depth include smartphone portrait mode bokeh effects, AR object placement, and 2D-to-3D video conversion.

Frequently Asked Questions

How accurate is stereo vision compared to LiDAR? Stereo vision provides dense depth maps at every pixel but is less accurate than LiDAR, especially at long range. LiDAR achieves millimeter-level accuracy at short range and centimeter-level at 100 meters, while stereo typically achieves centimeter to decimeter accuracy depending on baseline and resolution. LiDAR also works in darkness, while stereo requires ambient illumination or an active IR projector.

What hardware do I need to get started with 3D vision? A good starting point is a stereo camera pair like the Intel RealSense D435 ($300) or an OAK-D camera ($250) that combines stereo with a neural processing unit. For software, OpenCV provides stereo calibration and disparity computation functions. COLMAP runs on any computer with reasonable GPU and can work with photos from a standard smartphone.

What is the difference between SfM and visual SLAM? Structure from Motion processes unordered images offline, optimizing all camera poses and 3D points simultaneously. Visual SLAM operates on a video stream in real time, tracking camera pose incrementally and building a map of the environment. SfM can handle arbitrary image collections but requires offline batch processing, while SLAM runs online but produces less globally consistent reconstructions.

How does NeRF compare to traditional photogrammetry? NeRF produces photorealistic novel views with correct handling of reflections, transparency, and complex lighting — things traditional MVS struggles with. However, NeRF requires dense viewpoint coverage and does not produce a clean geometric surface mesh without additional post-processing. Traditional photogrammetry produces explicit 3D geometry suitable for CAD and 3D printing. The two approaches are complementary and are increasingly combined in production pipelines.

What are the main challenges in 3D reconstruction from images? Reflective surfaces, textureless regions, thin structures, repetitive patterns, and dynamic objects all pose significant challenges. Transparent objects like glass are particularly difficult because they reveal the background behind them. Motion blur from camera movement or object motion degrades reconstruction quality, and specular highlights that move with viewpoint violate the Lambertian assumption underlying most MVS methods.