DeMoN: Depth and Motion Network for Learning Monocular Stereo

In this paper we formulate structure from motion as a learning problem. We train a convolutional network end-to-end to compute depth and camera motion from successive, unconstrained image pairs. The architecture is composed of multiple stacked encoder-decoder networks, the core part being an iterative network that is able to improve its own predictions. The network estimates not only depth and motion, but additionally surface normals, optical flow between the images and confidence of the matching. A crucial component of the approach is a training loss based on spatial relative differences. Compared to traditional two-frame structure from motion methods, results are more accurate and more robust. In contrast to the popular depth-from-single-image networks, DeMoN learns the concept of matching and, thus, better generalizes to structures not seen during training.Comment: Camera ready version for CVPR 2017. Supplementary material included. Project page: http://lmb.informatik.uni-freiburg.de/people/ummenhof/depthmotionnet

Fast, Accurate Thin-Structure Obstacle Detection for Autonomous Mobile Robots

Safety is paramount for mobile robotic platforms such as self-driving cars and unmanned aerial vehicles. This work is devoted to a task that is indispensable for safety yet was largely overlooked in the past -- detecting obstacles that are of very thin structures, such as wires, cables and tree branches. This is a challenging problem, as thin objects can be problematic for active sensors such as lidar and sonar and even for stereo cameras. In this work, we propose to use video sequences for thin obstacle detection. We represent obstacles with edges in the video frames, and reconstruct them in 3D using efficient edge-based visual odometry techniques. We provide both a monocular camera solution and a stereo camera solution. The former incorporates Inertial Measurement Unit (IMU) data to solve scale ambiguity, while the latter enjoys a novel, purely vision-based solution. Experiments demonstrated that the proposed methods are fast and able to detect thin obstacles robustly and accurately under various conditions.Comment: Appeared at IEEE CVPR 2017 Workshop on Embedded Visio

Projector Self-Calibration using the Dual Absolute Quadric

The applications for projectors have increased dramatically since their origins in cinema. These include augmented reality, information displays, 3D scanning, and even archiving and surgical intervention. One common thread between all of these applications is the nec- essary step of projector calibration. Projector calibration can be a challenging task, and requires significant effort and preparation to ensure accuracy and fidelity. This is especially true in large scale, multi-projector installations used for projection mapping. Generally, the cameras for projector-camera systems are calibrated off-site, and then used in-field un- der the assumption that the intrinsics have remained constant. However, the assumption of off-site calibration imposes several hard restrictions. Among these, is that the intrinsics remain invariant between the off-site calibration process and the projector calibration site. This assumption is easily invalidated upon physical impact, or changing of lenses. To ad- dress this, camera self-calibration has been proposed for the projector calibration problem. However, current proposed methods suffer from degenerate conditions that are easily en- countered in practical projector calibration setups, resulting in undesirable variability and a distinct lack of robustness. In particular, the condition of near-intersecting optical axes of the camera positions used to capture the scene resulted in high variability and significant error in the recovered camera focal lengths. As such, a more robust method was required. To address this issue, an alternative camera self-calibration method is proposed. In this thesis we demonstrate our method of projector calibration with unknown and uncalibrated cameras via autocalibration using the Dual Absolute Quadric (DAQ). This method results in a significantly more robust projector calibration process, especially in the presence of correspondence noise when compared with previous methods. We use the DAQ method to calibrate the cameras using projector-generated correspondences, by upgrading an ini- tial projective calibration to metric, and subsequently calibrating the projector using the recovered metric structure of the scene. Our experiments provide strong evidence of the brittle behaviour of existing methods of projector self-calibration by evaluating them in near-degenerate conditions using both synthetic and real data. Further, they also show that the DAQ can be used successfully to calibrate a projector-camera system and reconstruct the surface used for projection mapping robustly, where previous methods fail

Accurately scaled 3-D scene reconstruction using a moving monocular camera and a single-point depth sensor

Abstract: A 3-D reconstruction produced using only a single camera and Structure from Motion (SfM) is always up to scale i.e. without real world dimensions. Real-world dimensions are necessary for many applications that require 3-D reconstruction since decisions are made based on the accuracy of the reconstruction and the estimated camera poses. Current solutions to the absence of scale require prior knowledge of or access to the imaged environment in order to provide absolute scale to a reconstruction. It is often necessary to obtain a 3-D reconstruction of an inaccessible or unknown enviroment. This research proposes the use of a basic SfM pipeline for 3-D reconstruction with a single camera while augmenting the camera with a depth measurement for each image by way of a laser point marker. The marker is identified in the image and projected such that its location is determined as the point with highest point density along the projection in the up to scale reconstruction. The known distance to this point provides a scale factor that can be applied to the up to scale reconstruction. The results obtained show that the proposed augmentation does provide better scale accuracy. The SfM pipeline has room for improvement especially in terms of two-view geometry and structure estimations. A proof of concept is achieved that may open the door to improved algorithms for more demanding applications.M.Ing. (Electrical and Electronic Engineering

Deformable 3-D Modelling from Uncalibrated Video Sequences

Submitted for the degree of Doctor of Philosophy, Queen Mary, University of Londo

Self-Calibration of Multi-Camera Systems for Vehicle Surround Sensing

Multi-camera systems are being deployed in a variety of vehicles and mobile robots today. To eliminate the need for cost and labor intensive maintenance and calibration, continuous self-calibration is highly desirable. In this book we present such an approach for self-calibration of multi-Camera systems for vehicle surround sensing. In an extensive evaluation we assess our algorithm quantitatively using real-world data

Multi-camera simultaneous localization and mapping

In this thesis, we study two aspects of simultaneous localization and mapping (SLAM) for multi-camera systems: minimal solution methods for the scaled motion of non-overlapping and partially overlapping two camera systems and enabling online, real-time mapping of large areas using the parallelism inherent in the visual simultaneous localization and mapping (VSLAM) problem. We present the only existing minimal solution method for six degree of freedom structure and motion estimation using a non-overlapping, rigid two camera system with known intrinsic and extrinsic calibration. One example application of our method is the three-dimensional reconstruction of urban scenes from video. Because our method does not require the cameras' fields-of-view to overlap, we are able to maximize coverage of the scene and avoid processing redundant, overlapping imagery. Additionally, we developed a minimal solution method for partially overlapping stereo camera systems to overcome degeneracies inherent to non-overlapping two-camera systems but still have a wide total field of view. The method takes two stereo images as its input. It uses one feature visible in all four views and three features visible across two temporal view pairs to constrain the system camera's motion. We show in synthetic experiments that our method creates rotation and translation estimates that are more accurate than the perspective three-point method as the overlap in the stereo camera's fields-of-view is reduced. A final part of this thesis is the development of an online, real-time visual SLAM system that achieves real-time speed by exploiting the parallelism inherent in the VSLAM problem. We show that feature tracking, relative pose estimation, and global mapping operations such as loop detection and loop correction can be effectively parallelized. Additionally, we demonstrate that a combination of short baseline, differentially tracked corner features, which can be tracked at high frame rates and wide baseline matchable but slower to compute features such as the scale-invariant feature transform can facilitate high speed visual odometry and at the same time support location recognition for loop detection and global geometric error correction

Cavlectometry: Towards Holistic Reconstruction of Large Mirror Objects

We introduce a method based on the deflectometry principle for the reconstruction of specular objects exhibiting significant size and geometric complexity. A key feature of our approach is the deployment of an Automatic Virtual Environment (CAVE) as pattern generator. To unfold the full power of this extraordinary experimental setup, an optical encoding scheme is developed which accounts for the distinctive topology of the CAVE. Furthermore, we devise an algorithm for detecting the object of interest in raw deflectometric images. The segmented foreground is used for single-view reconstruction, the background for estimation of the camera pose, necessary for calibrating the sensor system. Experiments suggest a significant gain of coverage in single measurements compared to previous methods. To facilitate research on specular surface reconstruction, we will make our data set publicly available