3D Scene Reconstruction by Stereo Methods for Analysis and Visualization of Sports Scenes

The 3D reconstruction of image and video scenes by stereo analysis is an important topic in computer vision research. In this talk, we first present some principles of stereo algorithms and recent developments. We then demonstrate two applications of stereo reconstruction for the analysis and visualization of human movement: (a) We employ depth maps derived from sport scenes for novel view synthesis, and (b) we show how stereo processing can be used for expressive visualization of human motion in a comic-like style

CHOSEN: Contrastive Hypothesis Selection for Multi-View Depth Refinement

We propose CHOSEN, a simple yet flexible, robust and effective multi-view depth refinement framework. It can be employed in any existing multi-view stereo pipeline, with straightforward generalization capability for different multi-view capture systems such as camera relative positioning and lenses. Given an initial depth estimation, CHOSEN iteratively re-samples and selects the best hypotheses, and automatically adapts to different metric or intrinsic scales determined by the capture system. The key to our approach is the application of contrastive learning in an appropriate solution space and a carefully designed hypothesis feature, based on which positive and negative hypotheses can be effectively distinguished. Integrated in a simple baseline multi-view stereo pipeline, CHOSEN delivers impressive quality in terms of depth and normal accuracy compared to many current deep learning based multi-view stereo pipelines

The Global Patch Collider

Abstract This paper proposes a novel extremely efficient, fullyparallelizable, task-specific algorithm for the computation of global point-wise correspondences in images and videos. Our algorithm, the Global Patch Collider, is based on detecting unique collisions between image points using a collection of learned tree structures that act as conditional hash functions. In contrast to conventional approaches that rely on pairwise distance computation, our algorithm isolates distinctive pixel pairs that hit the same leaf during traversal through multiple learned tree structures. The split functions stored at the intermediate nodes of the trees are trained to ensure that only visually similar patches or their geometric or photometric transformed versions fall into the same leaf node. The matching process involves passing all pixel positions in the images under analysis through the tree structures. We then compute matches by isolating points that uniquely collide with each other ie. fell in the same empty leaf in multiple trees. Our algorithm is linear in the number of pixels but can be made constant time on a parallel computation architecture as the tree traversal for individual image points is decoupled. We demonstrate the efficacy of our method by using it to perform optical flow matching and stereo matching on some challenging benchmarks. Experimental results show that not only is our method extremely computationally efficient, but it is also able to match or outperform state of the art methods that are much more complex

Fusion4D: Real-time Performance Capture of Challenging Scenes

We contribute a new pipeline for live multi-view performance capture, generating temporally coherent high-quality reconstructions in real-time. Our algorithm supports both incremental reconstruction, improving the surface estimation over time, as well as parameterizing the nonrigid scene motion. Our approach is highly robust to both large frame-to-frame motion and topology changes, allowing us to reconstruct extremely challenging scenes. We demonstrate advantages over related real-time techniques that either deform an online generated template or continually fuse depth data nonrigidly into a single reference model. Finally, we show geometric reconstruction results on par with offline methods which require orders of magnitude more processing time and many more RGBD cameras

Region-Based Optical Flow Estimation with Treatment of Occlusions

The estimation of optical flow plays a key-role in several computer vision problems, including motion detection and segmentation, frame interpolation, three-dimensional scene reconstruction, robot navigation, video shot detection, mosaicking and video compression. In this work we propose a new algorithm for computing a dense optical flow field between two or more images of a video sequence, which tackles the inherent problems of conventional optical flow algorithms. These algorithms usually show a bad performance in regions of low texture as well as near motion boundaries. We try to overcome these problems by segmenting the reference frame into regions of homogeneous color. The color segmentation incorporates the assumption that the motion inside regions of homogeneous color varies smoothly and motion discontinuities coincide with the borders of those regions. The affine motion model is used to describe the motion inside a segment. To initializ