Unsupervised Learning of Complex Articulated Kinematic Structures combining Motion and Skeleton Information

In this paper we present a novel framework for unsupervised kinematic structure learning of complex articulated objects from a single-view image sequence. In contrast to prior motion information based methods, which estimate relatively simple articulations, our method can generate arbitrarily complex kinematic structures with skeletal topology by a successive iterative merge process. The iterative merge process is guided by a skeleton distance function which is generated from a novel object boundary generation method from sparse points. Our main contributions can be summarised as follows: (i) Unsupervised complex articulated kinematic structure learning by combining motion and skeleton information. (ii) Iterative fine-to-coarse merging strategy for adaptive motion segmentation and structure smoothing. (iii) Skeleton estimation from sparse feature points. (iv) A new highly articulated object dataset containing multi-stage complexity with ground truth. Our experiments show that the proposed method out-performs state-of-the-art methods both quantitatively and qualitatively

Robust Real-time RGB-D Visual Odometry in Dynamic Environments via Rigid Motion Model

In the paper, we propose a robust real-time visual odometry in dynamic environments via rigid-motion model updated by scene flow. The proposed algorithm consists of spatial motion segmentation and temporal motion tracking. The spatial segmentation first generates several motion hypotheses by using a grid-based scene flow and clusters the extracted motion hypotheses, separating objects that move independently of one another. Further, we use a dual-mode motion model to consistently distinguish between the static and dynamic parts in the temporal motion tracking stage. Finally, the proposed algorithm estimates the pose of a camera by taking advantage of the region classified as static parts. In order to evaluate the performance of visual odometry under the existence of dynamic rigid objects, we use self-collected dataset containing RGB-D images and motion capture data for ground-truth. We compare our algorithm with state-of-the-art visual odometry algorithms. The validation results suggest that the proposed algorithm can estimate the pose of a camera robustly and accurately in dynamic environments

Median K-flats for hybrid linear modeling with many outliers

We describe the Median K-Flats (MKF) algorithm, a simple online method for hybrid linear modeling, i.e., for approximating data by a mixture of flats. This algorithm simultaneously partitions the data into clusters while finding their corresponding best approximating l1 d-flats, so that the cumulative l1 error is minimized. The current implementation restricts d-flats to be d-dimensional linear subspaces. It requires a negligible amount of storage, and its complexity, when modeling data consisting of N points in D-dimensional Euclidean space with K d-dimensional linear subspaces, is of order O(n K d D+n d^2 D), where n is the number of iterations required for convergence (empirically on the order of 10^4). Since it is an online algorithm, data can be supplied to it incrementally and it can incrementally produce the corresponding output. The performance of the algorithm is carefully evaluated using synthetic and real data

LEARNING TO RIG CHARACTERS

With the emergence of 3D virtual worlds, 3D social media, and massive online games, the need for diverse, high-quality, animation-ready characters and avatars is greater than ever. To animate characters, artists hand-craft articulation structures, such as animation skeletons and part deformers, which require significant amount of manual and laborious interaction with 2D/3D modeling interfaces. This thesis presents deep learning methods that are able to significantly automate the process of character rigging. First, the thesis introduces RigNet, a method capable of predicting an animation skeleton for an input static 3D shape in the form of a polygon mesh. The predicted skeletons match the animator expectations in joint placement and topology. RigNet also estimates surface skin weights which determine how the mesh is animated given the different skeletal poses. In contrast to prior work that fits pre-defined skeletal templates with hand-tuned objectives, RigNet is able to automatically rig diverse characters, such as humanoids, quadrupeds, toys, birds, with varying articulation structure and geometry. RigNet is based on a deep neural architecture that directly operates on the mesh representation. The architecture is trained on a diverse dataset of rigged models that we mined online and curated. The dataset includes 2.7K polygon meshes, along with their associated skeletons and corresponding skin weights. Second, the thesis introduces Morig, a method that automatically rigs character meshes driven by single-view point cloud streams capturing the motion of performing characters. Compared to RigNet, MoRig\u27s rigging is \emph{motion-aware}: its neural network encodes motion cues from the point clouds into compact feature representations that are informative about the articulated parts of the performing character. These motion-aware features guide the inference of an appropriate skeletal rig for the input mesh. Furthermore, Morig is able to animate the rig according to the captured point cloud motion. Morig can handle diverse characters with different morphologies (e.g., humanoids, quadrupeds, toy characters). It also accounts for occluded regions in the point clouds and mismatches in the part proportions between the input mesh and captured character. Third, the thesis introduces APES, a method that takes as input 2D raster images depicting a small set of poses of a character shown in a sprite sheet, and identifies articulated parts useful for rigging the character. APES uses a combination of neural network inference and integer linear programming to identify a compact set of articulated body parts, e.g. head, torso and limbs, that best reconstruct the input poses. Compared to Morig and RigNet that require a large collection of training models with associated skeletons and skinning weights, APES\u27 neural architecture relies on less effortful supervision from (i) pixel correspondences readily available in existing large cartoon image datasets (e.g., Creative Flow), (ii) a relatively small dataset of 57 cartoon characters segmented into moving parts. Finally, the thesis discusses future research directions related to combining neural rigging with 3D and 4D reconstruction of characters from point cloud data and 2D video as well as automating the process of motion synthesis for 3D characters

동적 환경에 강인한 모션 분류 기반의 영상 항법 알고리즘 개발

학위논문 (석사)-- 서울대학교 대학원 공과대학 기계항공공학부, 2017. 8. 김현진.In the paper, we propose a robust visual odometry algorithm for dynamic environments via rigid motion segmentation using a grid-based optical flow. The algorithm first divides image frame by a fixed-size grid, then calculates the three-dimensional motion of grids for light computational load and uniformly distributed optical flow vectors. Next, it selects several adjacent points among grid-based optical flow vectors based on a so-called entropy and generates motion hypotheses formed by three-dimensional rigid transformation. These processes for a spatial motion segmentation utilizes the principle of randomized hypothesis generation and the existing clustering algorithm, thus separating objects that move independently of each other. Moreover, we use a dual-mode simple Gaussian model in order to differentiate static and dynamic parts persistently. The model measures the output of the spatial motion segmentation algorithm and updates a probability vector consisting of the likelihood of representing specific label. For the evaluation of the proposed algorithm, we use a self-made dataset captured by ASUS Xtion Pro live RGB-D camera and Vicon motion capture system. We compare our algorithm with the existing motion segmentation algorithm and the current state-of-the-art visual odometry algorithm respectively, and the proposed algorithm estimates the ego-motion robustly and accurately in dynamic environments while showing the competitive performance of the motion segmentation.기존 대다수의 영상 항법 알고리즘은 정적인 환경을 가정하여 개발되어 왔으며, 잘 정의된 데이터셋에서 성능이 검증되어 왔다. 하지만 무인 로봇이 영상 항법을 활용하여 임무를 수행하여야 하는 장소는, 실제 사람이나 차량이 왕래하는 등 동적인 환경일 가능성이 크다. 비록 RANSAC을 활용하여 영상 항법을 수행하는 일부 알고리즘들은 프레임 내의 비정상적인 움직임을 위치 추정 과정에서 배제할 수 있지만, 이는 동적 물체가 영상 프레임의 작은 부분을 차지하는 경우에만 적용이 가능하다. 따라서 불확실성이 존재하는 동적 환경에서 자기 위치를 강인하게 추정하기 위해, 본 논문에서는 동적 환경에 강인한 영상 기반 주행 기록계 알고리즘을 제안한다. 제안한 알고리즘은 원활한 수행 속도와 이미지 내에 균일하게 분포된 모션을 계산하기 위해, 격자 기반 옵티컬 플로우를 이용한다. 그리고 격자 단위 그리드의 모션을 통해 단일 프레임 내에서 3차원 공간 모션 분할을 수행하고, 다수의 동적 물체 및 정적 요소를 지속적으로 구분 및 구별하기 위해 시간적 모션 분할을 수행한다. 특히 지속적으로 동적 및 정적 요소를 구별하기 위해, 우리는 이미지 내의 각 그리드에 이중 모드 가우시안 모델을 적용하여 알고리즘이 공간적 모션 분할의 일시적 노이즈에 강인하게 하고, 확률 벡터를 구성하여 그리드가 서로 구별되는 각각의 요소로 발현할 확률을 계산하게 한다. 개발한 알고리즘의 성능 검증을 위해 ASUS Xtion RGB-D 카메라와 Vicon 모션 캡쳐 시스템을 통해 구성한 데이터셋을 이용하였으며, 기존 모션 분할 알고리즘과의 재현율 (recall), 정밀도 (precision) 비교 및 기존 영상 기반 주행 기록계 알고리즘과의 추정 오차 비교를 통해 타 알고리즘 대비 우수한 모션 검출 및 위치 추정 성능을 확인하였다.Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii Chapter 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 Literature review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.2 Thesis contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.3 Thesis outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2 Background Knowledge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.1 Rigid transformation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.2 Grid-based optical flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 3 Motion Spatial Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3.1 Motion hypothesis search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3.2 Motion hypothesis refinement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3.3 Motion hypothesis clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 4 Motion Temporal Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 4.1 Label matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 4.2 Dual-mode simple Gaussian model . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 4.2.1 Update model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 4.2.2 Compensate model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 5 Evaluation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 5.1 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 5.2 Motion segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 5.3 Visual odometry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34Maste