NOVEL DENSE STEREO ALGORITHMS FOR HIGH-QUALITY DEPTH ESTIMATION FROM IMAGES

This dissertation addresses the problem of inferring scene depth information from a collection of calibrated images taken from different viewpoints via stereo matching. Although it has been heavily investigated for decades, depth from stereo remains a long-standing challenge and popular research topic for several reasons. First of all, in order to be of practical use for many real-time applications such as autonomous driving, accurate depth estimation in real-time is of great importance and one of the core challenges in stereo. Second, for applications such as 3D reconstruction and view synthesis, high-quality depth estimation is crucial to achieve photo realistic results. However, due to the matching ambiguities, accurate dense depth estimates are difficult to achieve. Last but not least, most stereo algorithms rely on identification of corresponding points among images and only work effectively when scenes are Lambertian. For non-Lambertian surfaces, the brightness constancy assumption is no longer valid. This dissertation contributes three novel stereo algorithms that are motivated by the specific requirements and limitations imposed by different applications. In addressing high speed depth estimation from images, we present a stereo algorithm that achieves high quality results while maintaining real-time performance. We introduce an adaptive aggregation step in a dynamic-programming framework. Matching costs are aggregated in the vertical direction using a computationally expensive weighting scheme based on color and distance proximity. We utilize the vector processing capability and parallelism in commodity graphics hardware to speed up this process over two orders of magnitude. In addressing high accuracy depth estimation, we present a stereo model that makes use of constraints from points with known depths - the Ground Control Points (GCPs) as referred to in stereo literature. Our formulation explicitly models the influences of GCPs in a Markov Random Field. A novel regularization prior is naturally integrated into a global inference framework in a principled way using the Bayes rule. Our probabilistic framework allows GCPs to be obtained from various modalities and provides a natural way to integrate information from various sensors. In addressing non-Lambertian reflectance, we introduce a new invariant for stereo correspondence which allows completely arbitrary scene reflectance (bidirectional reflectance distribution functions - BRDFs). This invariant can be used to formulate a rank constraint on stereo matching when the scene is observed by several lighting configurations in which only the lighting intensity varies

Stereo image matching using wavelet scale-space representation

A multi-resolution technique for matching a stereo pair of images based on translation invariant discrete multi-wavelet transform is presented. The technique uses the well known coarse to fine strategy, involving the calculation of matching points at the coarsest level with consequent refinement up to the finest level. Vector coefficients of the wavelet transform modulus are used as matching features, where modulus maxima defines the shift invariant high-level features (multiscale edges) with phase pointing to the normal of the feature surface. The technique addresses the estimation of optimal corresponding points and the corresponding 2D disparity maps. Illuminative variation that can exist between the perspective views of the same scene is controlled using scale normalization at each decomposition level by dividing the details space coefficients with approximation space and then using normalized correlation. The problem of ambiguity, explicitly, and occlusion, implicitly, is addressed by using a geometric topological refinement procedure and symbolic tagging.<br /

Contribution towards a fast stereo dense matching.

Stereo matching is important in the area of computer vision as it is the basis of the reconstruction process. Many applications require 3D reconstruction such as view synthesis, robotics... The main task of matching uncalibrated images is to determine the corresponding pixels and other features where the motion between these images and the camera parameters is unknown. Although some methods have been carried out over the past two decades on the matching problem, most of these methods are not practical and difficult to implement. Our approach considers a reliable image edge features in order to develop a fast and practical method. Therefore, we propose a fast stereo matching algorithm combining two different approaches for matching as the image is segmented into two sets of regions: edge regions and non-edge regions. We have used an algebraic method that preserves disparity continuity at the object continuous surfaces. Our results demonstrate that we gain a speed dense matching while the implementation is kept simple and straightforward.Dept. of Computer Science. Paper copy at Leddy Library: Theses & Major Papers - Basement, West Bldg. / Call Number: Thesis2005 .Z42. Source: Masters Abstracts International, Volume: 44-03, page: 1420. Thesis (M.Sc.)--University of Windsor (Canada), 2005

Disparity Map Generation from Illumination Variant Stereo Images Using Efficient Hierarchical Dynamic Programming

A novel hierarchical stereo matching algorithm is presented which gives disparity map as output from illumination variant stereo pair. Illumination difference between two stereo images can lead to undesirable output. Stereo image pair often experience illumination variations due to many factors like real and practical situation, spatially and temporally separated camera positions, environmental illumination fluctuation, and the change in the strength or position of the light sources. Window matching and dynamic programming techniques are employed for disparity map estimation. Good quality disparity map is obtained with the optimized path. Homomorphic filtering is used as a preprocessing step to lessen illumination variation between the stereo images. Anisotropic diffusion is used to refine disparity map to give high quality disparity map as a final output. The robust performance of the proposed approach is suitable for real life circumstances where there will be always illumination variation between the images. The matching is carried out in a sequence of images representing the same scene, however in different resolutions. The hierarchical approach adopted decreases the computation time of the stereo matching problem. This algorithm can be helpful in applications like robot navigation, extraction of information from aerial surveys, 3D scene reconstruction, and military and security applications. Similarity measure SAD is often sensitive to illumination variation. It produces unacceptable disparity map results for illumination variant left and right images. Experimental results show that our proposed algorithm produces quality disparity maps for both wide range of illumination variant and invariant stereo image pair

A variational technique for three-dimensional reconstruction of local structure

Thesis (S.M.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, February 1999.Includes bibliographical references (leaves 66-70).by Eric Raphaël Amram.S.M

On the Synergies between Machine Learning and Binocular Stereo for Depth Estimation from Images: a Survey

Stereo matching is one of the longest-standing problems in computer vision with close to 40 years of studies and research. Throughout the years the paradigm has shifted from local, pixel-level decision to various forms of discrete and continuous optimization to data-driven, learning-based methods. Recently, the rise of machine learning and the rapid proliferation of deep learning enhanced stereo matching with new exciting trends and applications unthinkable until a few years ago. Interestingly, the relationship between these two worlds is two-way. While machine, and especially deep, learning advanced the state-of-the-art in stereo matching, stereo itself enabled new ground-breaking methodologies such as self-supervised monocular depth estimation based on deep networks. In this paper, we review recent research in the field of learning-based depth estimation from single and binocular images highlighting the synergies, the successes achieved so far and the open challenges the community is going to face in the immediate future.Comment: Accepted to TPAMI. Paper version of our CVPR 2019 tutorial: "Learning-based depth estimation from stereo and monocular images: successes, limitations and future challenges" (https://sites.google.com/view/cvpr-2019-depth-from-image/home

Robotic Mapping and Localization with Real-Time Dense Stereo on Reconfigurable Hardware

A reconfigurable architecture for dense stereo is presented as an observation framework for a real-time implementation of the simultaneous localization and mapping problem in robotics. The reconfigurable sensor detects point features from stereo image pairs to use at the measurement update stage of the procedure. The main hardware blocks are a dense depth stereo accelerator, a left and right image corner detector, and a stage performing left-right consistency check. For the stereo-processor stage, we have implemented and tested a global-matching component based on a maximum-likelihood dynamic programming technique. The system includes a Nios II processor for data control and a USB 2.0 interface for host communication. Remote control is used to guide a vehicle equipped with a stereo head in an indoor environment. The FastSLAM Bayesian algorithm is applied in order to track and update observations and the robot path in real time. The system is assessed using real scene depth detection and public reference data sets. The paper also reports resource usage and a comparison of mapping and localization results with ground truth

A computer stereo vision system: using horizontal intensity line segments bounded by edges.

by Chor-Tung Yau.Thesis (M.Phil.)--Chinese University of Hong Kong, 1996.Includes bibliographical references (leaves 106-110).Chapter 1 --- Introduction --- p.1Chapter 1.1 --- Objectives --- p.1Chapter 1.2 --- Factors of Depth Perception in Human Visual System --- p.2Chapter 1.2.1 --- Oculomotor Cues --- p.2Chapter 1.2.2 --- Pictorial Cues --- p.3Chapter 1.2.3 --- Movement-Produced Cues --- p.4Chapter 1.2.4 --- Binocular Disparity --- p.5Chapter 1.3 --- What Cues to Use in Computer Vision? --- p.6Chapter 1.4 --- The Process of Stereo Vision --- p.8Chapter 1.4.1 --- Depth and Disparity --- p.8Chapter 1.4.2 --- The Stereo Correspondence Problem --- p.10Chapter 1.4.3 --- Parallel and Nonparallel Axis Stereo Geometry --- p.11Chapter 1.4.4 --- Feature-based and Area-based Stereo Matching --- p.12Chapter 1.4.5 --- Constraints --- p.13Chapter 1.5 --- Organization of this thesis --- p.16Chapter 2 --- Related Work --- p.18Chapter 2.1 --- Marr and Poggio's Computational Theory --- p.18Chapter 2.2 --- Cooperative Methods --- p.19Chapter 2.3 --- Dynamic Programming --- p.21Chapter 2.4 --- Feature-based Methods --- p.24Chapter 2.5 --- Area-based Methods --- p.26Chapter 3 --- Overview of the Method --- p.30Chapter 3.1 --- Considerations --- p.31Chapter 3.2 --- Brief Description of the Method --- p.33Chapter 4 --- Preprocessing of Images --- p.35Chapter 4.1 --- Edge Detection --- p.35Chapter 4.1.1 --- The Laplacian of Gaussian (∇2G) operator --- p.37Chapter 4.1.2 --- The Canny edge detector --- p.40Chapter 4.2 --- Extraction of Horizontal Line Segments for Matching --- p.42Chapter 5 --- The Matching Process --- p.45Chapter 5.1 --- Reducing the Search Space --- p.45Chapter 5.2 --- Similarity Measure --- p.47Chapter 5.3 --- Treating Inclined Surfaces --- p.49Chapter 5.4 --- Ambiguity Caused By Occlusion --- p.51Chapter 5.5 --- Matching Segments of Different Length --- p.53Chapter 5.5.1 --- Cases Without Partial Occlusion --- p.53Chapter 5.5.2 --- Cases With Partial Occlusion --- p.55Chapter 5.5.3 --- Matching Scheme To Handle All the Cases --- p.56Chapter 5.5.4 --- Matching Scheme for Segments of same length --- p.57Chapter 5.6 --- Assigning Disparity Values --- p.58Chapter 5.7 --- Another Case of Partial Occlusion Not Handled --- p.60Chapter 5.8 --- Matching in Two passes --- p.61Chapter 5.8.1 --- Problems encountered in the First pass --- p.61Chapter 5.8.2 --- Second pass of matching --- p.63Chapter 5.9 --- Refinement of Disparity Map --- p.64Chapter 6 --- Coarse-to-fine Matching --- p.67Chapter 6.1 --- The Wavelet Representation --- p.67Chapter 6.2 --- Coarse-to-fine Matching --- p.71Chapter 7 --- Experimental Results and Analysis --- p.74Chapter 7.1 --- Experimental Results --- p.74Chapter 7.1.1 --- Image Pair 1 - The Pentagon Images --- p.74Chapter 7.1.2 --- Image Pair 2 - Random dot stereograms --- p.79Chapter 7.1.3 --- Image Pair 3 ´ؤ The Rubik Block Images --- p.81Chapter 7.1.4 --- Image Pair 4 - The Stack of Books Images --- p.85Chapter 7.1.5 --- Image Pair 5 - The Staple Box Images --- p.87Chapter 7.1.6 --- Image Pair 6 - Circuit Board Image --- p.91Chapter 8 --- Conclusion --- p.94Chapter A --- The Wavelet Transform --- p.96Chapter A.l --- Fourier Transform and Wavelet Transform --- p.96Chapter A.2 --- Continuous wavelet Transform --- p.97Chapter A.3 --- Discrete Time Wavelet Transform --- p.99Chapter B --- Acknowledgements to Testing Images --- p.100Chapter B.l --- The Circuit Board Image --- p.100Chapter B.2 --- The Stack of Books Image --- p.101Chapter B.3 --- The Rubik Block Images --- p.104Bibliography --- p.10