7 research outputs found
Improving disparity estimation based on residual cost volume and reconstruction error volume
Recently, great progress has been made in formulating dense disparity estimation as a pixel-wise learning task to be solved by deep convolutional neural networks. However, most resulting pixel-wise disparity maps only show little detail for small structures. In this paper, we propose a two-stage architecture: we first learn initial disparities using an initial network, and then employ a disparity refinement network, guided by the initial results, which directly learns disparity corrections. Based on the initial disparities, we construct a residual cost volume between shared left and right feature maps in a potential disparity residual interval, which can capture more detailed context information. Then, the right feature map is warped with the initial disparity and a reconstruction error volume is constructed between the warped right feature map and the original left feature map, which provides a measure of correctness of the initial disparities. The main contribution of this paper is to combine the residual cost volume and the reconstruction error volume to guide training of the refinement network. We use a shallow encoder-decoder module in the refinement network and do learning from coarse to fine, which simplifies the learning problem. We evaluate our method on several challenging stereo datasets. Experimental results demonstrate that our refinement network can significantly improve the overall accuracy by reducing the estimation error by 30% compared with our initial network. Moreover, our network also achieves competitive performance compared with other CNN-based methods. © 2020 International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives
Robust super-resolution depth imaging via a multi-feature fusion deep network
Three-dimensional imaging plays an important role in imaging applications
where it is necessary to record depth. The number of applications that use
depth imaging is increasing rapidly, and examples include self-driving
autonomous vehicles and auto-focus assist on smartphone cameras. Light
detection and ranging (LIDAR) via single-photon sensitive detector (SPAD)
arrays is an emerging technology that enables the acquisition of depth images
at high frame rates. However, the spatial resolution of this technology is
typically low in comparison to the intensity images recorded by conventional
cameras. To increase the native resolution of depth images from a SPAD camera,
we develop a deep network built specifically to take advantage of the multiple
features that can be extracted from a camera's histogram data. The network is
designed for a SPAD camera operating in a dual-mode such that it captures
alternate low resolution depth and high resolution intensity images at high
frame rates, thus the system does not require any additional sensor to provide
intensity images. The network then uses the intensity images and multiple
features extracted from downsampled histograms to guide the upsampling of the
depth. Our network provides significant image resolution enhancement and image
denoising across a wide range of signal-to-noise ratios and photon levels. We
apply the network to a range of 3D data, demonstrating denoising and a
four-fold resolution enhancement of depth
On the Synergies between Machine Learning and Binocular Stereo for Depth Estimation from Images: a Survey
Stereo matching is one of the longest-standing problems in computer vision
with close to 40 years of studies and research. Throughout the years the
paradigm has shifted from local, pixel-level decision to various forms of
discrete and continuous optimization to data-driven, learning-based methods.
Recently, the rise of machine learning and the rapid proliferation of deep
learning enhanced stereo matching with new exciting trends and applications
unthinkable until a few years ago. Interestingly, the relationship between
these two worlds is two-way. While machine, and especially deep, learning
advanced the state-of-the-art in stereo matching, stereo itself enabled new
ground-breaking methodologies such as self-supervised monocular depth
estimation based on deep networks. In this paper, we review recent research in
the field of learning-based depth estimation from single and binocular images
highlighting the synergies, the successes achieved so far and the open
challenges the community is going to face in the immediate future.Comment: Accepted to TPAMI. Paper version of our CVPR 2019 tutorial:
"Learning-based depth estimation from stereo and monocular images: successes,
limitations and future challenges"
(https://sites.google.com/view/cvpr-2019-depth-from-image/home