Sequential non-rigid structure from motion using physical priors

© 20xx IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.We propose a new approach to simultaneously recover camera pose and 3D shape of non-rigid and potentially extensible surfaces from a monocular image sequence. For this purpose, we make use of the Extended Kalman Filter based Simultaneous Localization And Mapping (EKF-SLAM) formulation, a Bayesian optimization framework traditionally used in mobile robotics for estimating camera pose and reconstructing rigid scenarios. In order to extend the problem to a deformable domain we represent the object's surface mechanics by means of Navier's equations, which are solved using a Finite Element Method (FEM). With these main ingredients, we can further model the material's stretching, allowing us to go a step further than most of current techniques, typically constrained to surfaces undergoing isometric deformations. We extensively validate our approach in both real and synthetic experiments, and demonstrate its advantages with respect to competing methods. More specifically, we show that besides simultaneously retrieving camera pose and non-rigid shape, our approach is adequate for both isometric and extensible surfaces, does not require neither batch processing all the frames nor tracking points over the whole sequence and runs at several frames per second.Peer ReviewedPostprint (author's final draft

Nonrigid reconstruction of 3D breast surfaces with a low-cost RGBD camera for surgical planning and aesthetic evaluation

Accounting for 26% of all new cancer cases worldwide, breast cancer remains the most common form of cancer in women. Although early breast cancer has a favourable long-term prognosis, roughly a third of patients suffer from a suboptimal aesthetic outcome despite breast conserving cancer treatment. Clinical-quality 3D modelling of the breast surface therefore assumes an increasingly important role in advancing treatment planning, prediction and evaluation of breast cosmesis. Yet, existing 3D torso scanners are expensive and either infrastructure-heavy or subject to motion artefacts. In this paper we employ a single consumer-grade RGBD camera with an ICP-based registration approach to jointly align all points from a sequence of depth images non-rigidly. Subtle body deformation due to postural sway and respiration is successfully mitigated leading to a higher geometric accuracy through regularised locally affine transformations. We present results from 6 clinical cases where our method compares well with the gold standard and outperforms a previous approach. We show that our method produces better reconstructions qualitatively by visual assessment and quantitatively by consistently obtaining lower landmark error scores and yielding more accurate breast volume estimates

Single View Reconstruction for Human Face and Motion with Priors

Single view reconstruction is fundamentally an under-constrained problem. We aim to develop new approaches to model human face and motion with model priors that restrict the space of possible solutions. First, we develop a novel approach to recover the 3D shape from a single view image under challenging conditions, such as large variations in illumination and pose. The problem is addressed by employing the techniques of non-linear manifold embedding and alignment. Specifically, the local image models for each patch of facial images and the local surface models for each patch of 3D shape are learned using a non-linear dimensionality reduction technique, and the correspondences between these local models are then learned by a manifold alignment method. Local models successfully remove the dependency of large training databases for human face modeling. By combining the local shapes, the global shape of a face can be reconstructed directly from a single linear system of equations via least square. Unfortunately, this learning-based approach cannot be successfully applied to the problem of human motion modeling due to the internal and external variations in single view video-based marker-less motion capture. Therefore, we introduce a new model-based approach for capturing human motion using a stream of depth images from a single depth sensor. While a depth sensor provides metric 3D information, using a single sensor, instead of a camera array, results in a view-dependent and incomplete measurement of object motion. We develop a novel two-stage template fitting algorithm that is invariant to subject size and view-point variations, and robust to occlusions. Starting from a known pose, our algorithm first estimates a body configuration through temporal registration, which is used to search the template motion database for a best match. The best match body configuration as well as its corresponding surface mesh model are deformed to fit the input depth map, filling in the part that is occluded from the input and compensating for differences in pose and body-size between the input image and the template. Our approach does not require any makers, user-interaction, or appearance-based tracking. Experiments show that our approaches can achieve good modeling results for human face and motion, and are capable of dealing with variety of challenges in single view reconstruction, e.g., occlusion

Open and closed contours tracking based on shape priors and training

학위논문 (박사)-- 서울대학교 대학원 : 전기·컴퓨터공학부, 2016. 8. 조남익.This dissertation presents a new open and closed contours tracking algorithm using shape prior and its training based on a Bayesian framework, where the contour is a part (open contour) or the whole (closed contour) of the object's boundary. The shape of an object is a very important feature for many vision tasks such as object recognition and tracking. Specifically, the tracking performance can be increased if the target is determined and the tracker utilizes its shape information. The proposed method provides a new state space model for the representation of contours, which can reflect the shape information to the contour and handle rigid and non-rigid motions of contours independently. This model enables us to focus on the non-rigid motion during the tracking, and the model works for challenging rigid motion scenarios. In addition, for the robust tracking of contours, a measurement function that considers the contrast on object boundaries, target appearance, and temporal coherence is proposed. The proposed method is tested for various cases of contours such as open contour, closed contour and multi-contours. The state space model and measurement functions are modified a little bit in consideration of each contour model. First, an open contour is modeled and tracked by the proposed method, which has received little attention during several decades compared with the closed contour or bounding box shape tracking. The proposed state space model can represent an open contour that is moved by the dynamic model where rigid and non-rigid motions are absolutely separated. The measurement is designed with contrast, local track and appearance terms that indicate the proper position of the target and make the tracking more robust. The proposed method is applied to two examples of open contours targets (human omega shape and a cheetah profile), and experimental results show that the proposed method achieves superior performance to the conventional contour tracking methods. The proposed method is also compared with recent bounding box tracking methods for the object tracking purposes, and the comparison shows that the proposed method works robustly to fast motions and yields more accurate estimate of object's location than the conventional bounding box tracking methods. Second, the proposed method is tested for the closed contour tracking which is usually carried out by segmentation algorithms or level set methods. A closed contour is described by the proposed model and deformed by the dynamic model. Measurement function is the same to the case of open contour tracking except the local track term, which is calculated with partial object appearances that are denoted by some local patches and their relative positions. As an application example, automobiles in blackbox video sequences are tracked by the proposed method. Experimental results show that the proposed method accomplishes higher performance than conventional tracking methods where some of them presents the target as a bounding box and others extract the object boundary using segmentation methods. Moreover, the document capture and tracking algorithm is also proposed, which is suitable for applying the proposed method because the shape of document is well known (a quadrilateral) and its boundary can be estimated by the proposed method. This system is based on building quadrilaterals as document proposals using line segment detector and tests all proposals to find the best one with measurement terms. The proposed algorithm makes good marks at 2015 ICDAR competition. Finally, multi-contours tracking algorithm is devised based on the contour tracking method. It is assumed that targets belong to the same category and their appearances, colors and shapes are similar to each other. Thus, the proposed method trains only one shape model to track multi-contours. The state space vector is amended such that all contours can be represented by one state vector. In order to consider interactions between targets, the interaction term is attached to the existing dynamic model. As an example, human legs are tracked by the proposed method which may help to recognize the gaits. Experimental results show that conventional algorithms have troubles in tracking and distinguishing between the two legs, whereas all targets are well estimated accurately by the proposed method.Chapter 1 Introduction 1 1.1 Open contour tracking based on a nonrigid shape training 6 1.2 Target-based closed contour tracking 7 1.3 Multi-contours tracking for objects that belong in the same category 8 1.4 Structure of the dissertation 10 Chapter 2 Related work 11 2.1 Bounding box tracking 11 2.2 Contour tracking 12 2.3 Multi-objects tracking 14 Chapter 3 Open contour tracking based on a nonrigid shape training 15 3.1 Proposed state space model 15 3.1.1 Reviews on the active contour model 15 3.1.2 Proposed state vector 16 3.1.3 Proposed stochastic dynamic model 18 3.2 Training of the proposed state space model 20 3.2.1 Training criterion 22 3.2.2 Optimization method 23 3.2.3 Proof for solving the training problem 24 3.3 Measurement 27 3.3.1 Contrast term 27 3.3.2 Local track term 29 3.3.3 Appearance term 29 3.3.4 Model update 31 3.3.5 Weights of three measurement terms 31 3.4 Experimental results 34 3.4.1 Parameter selection 34 3.4.2 Label map construction 36 3.4.3 Comparison with existing contour-based methods 37 3.4.4 Comparison with bounding box based methods 42 3.4.5 Comparison with tracking methods for nonrigid objects 48 Chapter 4 Target-dependent closed contour tracking 51 4.1 The proposed model 51 4.1.1 Active contour model 51 4.1.2 Contour dynamics 52 4.2 Measurement 54 4.2.1 Local track term 54 4.3 Experimental results 57 4.3.1 Label map construction 59 4.3.2 Comparison to conventional tracking methods 59 4.4 Special case : document capture 65 4.4.1 Document model 65 4.4.2 Document proposals 66 4.4.3 Measurement 67 4.4.4 Renement . 69 4.4.5 Experimental results 70 Chapter 5 Multi-contours tracking for objects that belong to the same category 83 5.1 Proposed multi-contours tracking 83 5.1.1 State space model 84 5.1.2 Dynamics and measurement . 86 5.1.3 Particle sampling . 88 5.2 Experimental results 89 5.2.1 Comparison with other multi-objects tracking methods . 92 5.2.2 Comparison with tracking methods for a single object . 98 Chapter 6 Conclusions 101 Bibliography 103 Abstract (Korean) 109Docto

Intelligent visual media processing: when graphics meets vision

The computer graphics and computer vision communities have been working closely together in recent years, and a variety of algorithms and applications have been developed to analyze and manipulate the visual media around us. There are three major driving forces behind this phenomenon: i) the availability of big data from the Internet has created a demand for dealing with the ever increasing, vast amount of resources; ii) powerful processing tools, such as deep neural networks, provide e�ective ways for learning how to deal with heterogeneous visual data; iii) new data capture devices, such as the Kinect, bridge between algorithms for 2D image understanding and 3D model analysis. These driving forces have emerged only recently, and we believe that the computer graphics and computer vision communities are still in the beginning of their honeymoon phase. In this work we survey recent research on how computer vision techniques bene�t computer graphics techniques and vice versa, and cover research on analysis, manipulation, synthesis, and interaction. We also discuss existing problems and suggest possible further research directions

A Benchmark and Evaluation of Non-Rigid Structure from Motion

Non-Rigid structure from motion (NRSfM), is a long standing and central problem in computer vision, allowing us to obtain 3D information from multiple images when the scene is dynamic. A main issue regarding the further development of this important computer vision topic, is the lack of high quality data sets. We here address this issue by presenting of data set compiled for this purpose, which is made publicly available, and considerably larger than previous state of the art. To validate the applicability of this data set, and provide and investigation into the state of the art of NRSfM, including potential directions forward, we here present a benchmark and a scrupulous evaluation using this data set. This benchmark evaluates 16 different methods with available code, which we argue reasonably spans the state of the art in NRSfM. We also hope, that the presented and public data set and evaluation, will provide benchmark tools for further development in this field