Learning by Analogy: Reliable Supervision from Transformations for Unsupervised Optical Flow Estimation

Unsupervised learning of optical flow, which leverages the supervision from view synthesis, has emerged as a promising alternative to supervised methods. However, the objective of unsupervised learning is likely to be unreliable in challenging scenes. In this work, we present a framework to use more reliable supervision from transformations. It simply twists the general unsupervised learning pipeline by running another forward pass with transformed data from augmentation, along with using transformed predictions of original data as the self-supervision signal. Besides, we further introduce a lightweight network with multiple frames by a highly-shared flow decoder. Our method consistently gets a leap of performance on several benchmarks with the best accuracy among deep unsupervised methods. Also, our method achieves competitive results to recent fully supervised methods while with much fewer parameters.Comment: Accepted to CVPR 2020, https://github.com/lliuz/ARFlo

Lidar with Velocity: Correcting Moving Objects Point Cloud Distortion from Oscillating Scanning Lidars by Fusion with Camera

Lidar point cloud distortion from moving object is an important problem in autonomous driving, and recently becomes even more demanding with the emerging of newer lidars, which feature back-and-forth scanning patterns. Accurately estimating moving object velocity would not only provide a tracking capability but also correct the point cloud distortion with more accurate description of the moving object. Since lidar measures the time-of-flight distance but with a sparse angular resolution, the measurement is precise in the radial measurement but lacks angularly. Camera on the other hand provides a dense angular resolution. In this paper, Gaussian-based lidar and camera fusion is proposed to estimate the full velocity and correct the lidar distortion. A probabilistic Kalman-filter framework is provided to track the moving objects, estimate their velocities and simultaneously correct the point clouds distortions. The framework is evaluated on real road data and the fusion method outperforms the traditional ICP-based and point-cloud only method. The complete working framework is open-sourced (https://github.com/ISEE-Technology/lidar-with-velocity) to accelerate the adoption of the emerging lidars

Linear Quasi-Parallax SfM for various classes of biological eyes

Ph.DDOCTOR OF PHILOSOPH

Epiflow Based Stereo Fusion

3-D reconstruction from images sequences has been the center topic of computer vision. Real-time applications call for causal processing of stereo sequences, as they are acquired, covering different regions of the scene. The first step is to compute the current stereo disparity, and recursive map building often requires fusing with the previous estimate. In this paper, the epiflow framework [1], originally proposed for establishing matches among stereo feature pairs is generalized to devise an iterative causal algorithm for stereo disparity map fusion. In the context of disparity fusion, quadruplet correspondence of the epiflow tracking algorithm becomes reminiscent of the “closest point” of the 3-D ICP algorithm. Unlike ICP, the 2-D epiflow framework permits incorporating both photometric and geometrical constraints, estimation of the stereo rig motion as supplementary information, as well as identifying local inconsistencies between the two disparity maps. Experiments with real data validate the proposed approach, and improved converge compared to the ICP algorithm

Automatic sensor platform positioning and three-dimensional target modeling from underwater stereo sequences

This dissertation explores various problems for recovering the trajectory of a mobile platform, and reconstructing the 3-D models of objects of interest from stereo data. It is primarily targeted to develop a robust, efficient and unified solution for autonomous navigation and 3-D reconstruction for applications in uncontrolled environments, including the mapping of benthic objects in underwater applications. To achieve these goals, the topics of research become considerably broader in scope, encompassing feature tracking, stereo matching, stereo fusion, recursive motion and structure estimation, and 3-D reconstruction. The unknown, unstructured and uncontrollable nature of environmental conditions dictate relaxing many restrictive assumptions of most earlier techniques, predominantly developed for operations within more friendly terrestrial environments. The critical issue is how to improve robustness while enhancing generality. While robust statistical measures have been introduced to improve the performance in some existing methods, the computational complexity and requirements prohibit real-time performance. To meet these objectives, a framework is envisioned that enables effective use of online real-time recursive estimates for offline dense optimal reconstruction based on global solutions. More precisely, some of our techniques are targeted directly to address real-time performance needs and computational constraints. Yet, their solutions provide estimates of certain parameters that are instrumental for the application of offline global reconstruction techniques.The epiflow framework has been proposed as the central foundation to regulate the infamous ill-posed problem of establishing image correspondence through motion and stereo cooperation, by fully exploiting the geometric and photometric constraints embedded in two pairs of stereo views. The epiflow framework has been successfully applied to stereo feature tracking and stereo fusion. The dual recursive estimation based on EKF aims to address the computation efficiency needs in uncontrolled environments. These provide solutions for online processing, resulting in an accurate estimate of the platform trajectory, scene structure in terms of the 3-D positions of certain prominent features, as well as dense local reconstructions by the application of well-known match propagation. Two improvements to match propagation enables robust dense stereo disparity computation in near real time. Our global method comprises stereo matching by improved belief propagation, where our revised data term provides a more solution to combat local brightness variations due to shading effects that are quite common particularly in underwater video but generally ignored by previous techniques. Employing these new developments, we have formalized an integrated automatic in situ 3-D reconstruction solution, to build 3-D target model from stereo sequences robustly and efficiently in the presence of significant visual artifacts and disturbances. Additionally, other immediate benefits include scalability, flexibility and adaptivity. Extensive experiments with underwater data demonstrate the efficacy of the proposed system