Video alignment to a common reference

2015 Spring.Includes bibliographical references.Handheld videos often include unintentional motion (jitter) and intentional motion (pan and/or zoom). Human viewers prefer to see jitter removed, creating a smoothly moving camera. For video analysis, in contrast, aligning to a fixed stable background is sometimes preferable. This paper presents an algorithm that removes both forms of motion using a novel and efficient way of tracking background points while ignoring moving foreground points. The approach is related to image mosaicing, but the result is a video rather than an enlarged still image. It is also related to multiple object tracking approaches, but simpler since moving objects need not be explicitly tracked. The algorithm presented takes as input a video and returns one or several stabilized videos. Videos are broken into parts when the algorithm detects background change and it becomes necessary to fix upon a new background. We present two techniques in this thesis. One technique stabilizes the video with respect to the first available frame. Another technique stabilizes the videos with respect to a best frame. Our approach assumes the person holding the camera is standing in one place and that objects in motion do not dominate the image. Our algorithm performs better than previously published approaches when compared on 1,401 handheld videos from the recently released Point-and-Shoot Face Recognition Challenge (PASC)

동적 환경에 강인한 모션 분류 기반의 영상 항법 알고리즘 개발

학위논문 (석사)-- 서울대학교 대학원 공과대학 기계항공공학부, 2017. 8. 김현진.In the paper, we propose a robust visual odometry algorithm for dynamic environments via rigid motion segmentation using a grid-based optical flow. The algorithm first divides image frame by a fixed-size grid, then calculates the three-dimensional motion of grids for light computational load and uniformly distributed optical flow vectors. Next, it selects several adjacent points among grid-based optical flow vectors based on a so-called entropy and generates motion hypotheses formed by three-dimensional rigid transformation. These processes for a spatial motion segmentation utilizes the principle of randomized hypothesis generation and the existing clustering algorithm, thus separating objects that move independently of each other. Moreover, we use a dual-mode simple Gaussian model in order to differentiate static and dynamic parts persistently. The model measures the output of the spatial motion segmentation algorithm and updates a probability vector consisting of the likelihood of representing specific label. For the evaluation of the proposed algorithm, we use a self-made dataset captured by ASUS Xtion Pro live RGB-D camera and Vicon motion capture system. We compare our algorithm with the existing motion segmentation algorithm and the current state-of-the-art visual odometry algorithm respectively, and the proposed algorithm estimates the ego-motion robustly and accurately in dynamic environments while showing the competitive performance of the motion segmentation.기존 대다수의 영상 항법 알고리즘은 정적인 환경을 가정하여 개발되어 왔으며, 잘 정의된 데이터셋에서 성능이 검증되어 왔다. 하지만 무인 로봇이 영상 항법을 활용하여 임무를 수행하여야 하는 장소는, 실제 사람이나 차량이 왕래하는 등 동적인 환경일 가능성이 크다. 비록 RANSAC을 활용하여 영상 항법을 수행하는 일부 알고리즘들은 프레임 내의 비정상적인 움직임을 위치 추정 과정에서 배제할 수 있지만, 이는 동적 물체가 영상 프레임의 작은 부분을 차지하는 경우에만 적용이 가능하다. 따라서 불확실성이 존재하는 동적 환경에서 자기 위치를 강인하게 추정하기 위해, 본 논문에서는 동적 환경에 강인한 영상 기반 주행 기록계 알고리즘을 제안한다. 제안한 알고리즘은 원활한 수행 속도와 이미지 내에 균일하게 분포된 모션을 계산하기 위해, 격자 기반 옵티컬 플로우를 이용한다. 그리고 격자 단위 그리드의 모션을 통해 단일 프레임 내에서 3차원 공간 모션 분할을 수행하고, 다수의 동적 물체 및 정적 요소를 지속적으로 구분 및 구별하기 위해 시간적 모션 분할을 수행한다. 특히 지속적으로 동적 및 정적 요소를 구별하기 위해, 우리는 이미지 내의 각 그리드에 이중 모드 가우시안 모델을 적용하여 알고리즘이 공간적 모션 분할의 일시적 노이즈에 강인하게 하고, 확률 벡터를 구성하여 그리드가 서로 구별되는 각각의 요소로 발현할 확률을 계산하게 한다. 개발한 알고리즘의 성능 검증을 위해 ASUS Xtion RGB-D 카메라와 Vicon 모션 캡쳐 시스템을 통해 구성한 데이터셋을 이용하였으며, 기존 모션 분할 알고리즘과의 재현율 (recall), 정밀도 (precision) 비교 및 기존 영상 기반 주행 기록계 알고리즘과의 추정 오차 비교를 통해 타 알고리즘 대비 우수한 모션 검출 및 위치 추정 성능을 확인하였다.Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii Chapter 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 Literature review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.2 Thesis contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.3 Thesis outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2 Background Knowledge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.1 Rigid transformation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.2 Grid-based optical flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 3 Motion Spatial Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3.1 Motion hypothesis search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3.2 Motion hypothesis refinement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3.3 Motion hypothesis clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 4 Motion Temporal Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 4.1 Label matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 4.2 Dual-mode simple Gaussian model . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 4.2.1 Update model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 4.2.2 Compensate model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 5 Evaluation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 5.1 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 5.2 Motion segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 5.3 Visual odometry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34Maste

Robust motion segmentation with subspace constraints

Motion segmentation is an important task in computer vision with many applications such as dynamic scene understanding and multi-body structure from motion. When the point correspondences across frames are given, motion segmentation can be addressed as a subspace clustering problem under an affine camera model. In the first two parts of this thesis, we target the general subspace clustering problem and propose two novel methods, namely Efficient Dense Subspace Clustering (EDSC) and the Robust Shape Interaction Matrix (RSIM) method. Instead of following the standard compressive sensing approach, in EDSC we formulate subspace clustering as a Frobenius norm minimization problem, which inherently yields denser connections between data points. While in the noise-free case we rely on the self-expressiveness of the observations, in the presence of noise we recover a clean dictionary to represent the data. Our formulation lets us solve the subspace clustering problem efficiently. More specifically, for outlier-free observations, the solution can be obtained in closed-form, and in the presence of outliers, we solve the problem by performing a series of linear operations. Furthermore, we show that our Frobenius norm formulation shares the same solution as the popular nuclear norm minimization approach when the data is free of any noise. In RSIM, we revisit the Shape Interaction Matrix (SIM) method, one of the earliest approaches for motion segmentation (or subspace clustering), and reveal its connections to several recent subspace clustering methods. We derive a simple, yet effective algorithm to robustify the SIM method and make it applicable to real-world scenarios where the data is corrupted by noise. We validate the proposed method by intuitive examples and justify it with the matrix perturbation theory. Moreover, we show that RSIM can be extended to handle missing data with a Grassmannian gradient descent method. The above subspace clustering methods work well for motion segmentation, yet they require that point trajectories across frames are known {\it a priori}. However, finding point correspondences is in itself a challenging task. Existing approaches tackle the correspondence estimation and motion segmentation problems separately. In the third part of this thesis, given a set of feature points detected in each frame of the sequence, we develop an approach which simultaneously performs motion segmentation and finds point correspondences across the frames. We formulate this problem in terms of Partial Permutation Matrices (PPMs) and aim to match feature descriptors while simultaneously encouraging point trajectories to satisfy subspace constraints. This lets us handle outliers in both point locations and feature appearance. The resulting optimization problem is solved via the Alternating Direction Method of Multipliers (ADMM), where each subproblem has an efficient solution. In particular, we show that most of the subproblems can be solved in closed-form, and one binary assignment subproblem can be solved by the Hungarian algorithm. Obtaining reliable feature tracks in a frame-by-frame manner is desirable in applications such as online motion segmentation. In the final part of the thesis, we introduce a novel multi-body feature tracker that exploits a multi-body rigidity assumption to improve tracking robustness under a general perspective camera model. A conventional approach to addressing this problem would consist of alternating between solving two subtasks: motion segmentation and feature tracking under rigidity constraints for each segment. This approach, however, requires knowing the number of motions, as well as assigning points to motion groups, which is typically sensitive to motion estimates. By contrast, we introduce a segmentation-free solution to multi-body feature tracking that bypasses the motion assignment step and reduces to solving a series of subproblems with closed-form solutions. In summary, in this thesis, we exploit the powerful subspace constraints and develop robust motion segmentation methods in different challenging scenarios where the trajectories are either given as input, or unknown beforehand. We also present a general robust multi-body feature tracker which can be used as the first step of motion segmentation to get reliable trajectories