Data-Driven Shape Analysis and Processing

Data-driven methods play an increasingly important role in discovering geometric, structural, and semantic relationships between 3D shapes in collections, and applying this analysis to support intelligent modeling, editing, and visualization of geometric data. In contrast to traditional approaches, a key feature of data-driven approaches is that they aggregate information from a collection of shapes to improve the analysis and processing of individual shapes. In addition, they are able to learn models that reason about properties and relationships of shapes without relying on hard-coded rules or explicitly programmed instructions. We provide an overview of the main concepts and components of these techniques, and discuss their application to shape classification, segmentation, matching, reconstruction, modeling and exploration, as well as scene analysis and synthesis, through reviewing the literature and relating the existing works with both qualitative and numerical comparisons. We conclude our report with ideas that can inspire future research in data-driven shape analysis and processing.Comment: 10 pages, 19 figure

The prognostic value of white-matter selective double inversion recovery mri sequence in multiple sclerosis: an exploratory study

Using a white-matter selective double inversion recovery sequence (WM-DIR) that suppresses both grey matter (GM) and cerebrospinal fluid (CSF) signals, some white matter (WM) lesions appear surrounded by a dark rim. These dark rim lesions (DRLs) seem to be specific for multiple sclerosis (MS). They could be of great usefulness in clinical practice, proving to increase the MRI diagnostic criteria specificity. The aims of this study are the identification of DRLs on 1.5 T MRI, the exploration of the relationship between DRLs and disease course, the characterization of DRLs with respect to perilesional normal-appearing WM using magnetization transfer imaging, and the investigation of possible differences in the underlying tissue properties by assessing WM-DIR images obtained at 3.0 T MRI. DRLs are frequent in primary progressive MS (PPMS) patients. Amongst relapsing-remitting MS (RRMS) patients, DRLs are associated with a high risk of the disease worsening and secondary progressive MS (SPMS) conversion after 15 years. The mean magnetization transfer ratio (MTR) of DRLs is significantly different from the lesion without the dark rim, suggesting that DRLs correspond to more destructive lesions

Depth-Assisted Semantic Segmentation, Image Enhancement and Parametric Modeling

This dissertation addresses the problem of employing 3D depth information on solving a number of traditional challenging computer vision/graphics problems. Humans have the abilities of perceiving the depth information in 3D world, which enable humans to reconstruct layouts, recognize objects and understand the geometric space and semantic meanings of the visual world. Therefore it is significant to explore how the 3D depth information can be utilized by computer vision systems to mimic such abilities of humans. This dissertation aims at employing 3D depth information to solve vision/graphics problems in the following aspects: scene understanding, image enhancements and 3D reconstruction and modeling. In addressing scene understanding problem, we present a framework for semantic segmentation and object recognition on urban video sequence only using dense depth maps recovered from the video. Five view-independent 3D features that vary with object class are extracted from dense depth maps and used for segmenting and recognizing different object classes in street scene images. We demonstrate a scene parsing algorithm that uses only dense 3D depth information to outperform using sparse 3D or 2D appearance features. In addressing image enhancement problem, we present a framework to overcome the imperfections of personal photographs of tourist sites using the rich information provided by large-scale internet photo collections (IPCs). By augmenting personal 2D images with 3D information reconstructed from IPCs, we address a number of traditionally challenging image enhancement techniques and achieve high-quality results using simple and robust algorithms. In addressing 3D reconstruction and modeling problem, we focus on parametric modeling of flower petals, the most distinctive part of a plant. The complex structure, severe occlusions and wide variations make the reconstruction of their 3D models a challenging task. We overcome these challenges by combining data driven modeling techniques with domain knowledge from botany. Taking a 3D point cloud of an input flower scanned from a single view, each segmented petal is fitted with a scale-invariant morphable petal shape model, which is constructed from individually scanned 3D exemplar petals. Novel constraints based on botany studies are incorporated into the fitting process for realistically reconstructing occluded regions and maintaining correct 3D spatial relations. The main contribution of the dissertation is in the intelligent usage of 3D depth information on solving traditional challenging vision/graphics problems. By developing some advanced algorithms either automatically or with minimum user interaction, the goal of this dissertation is to demonstrate that computed 3D depth behind the multiple images contains rich information of the visual world and therefore can be intelligently utilized to recognize/ understand semantic meanings of scenes, efficiently enhance and augment single 2D images, and reconstruct high-quality 3D models

Past, Present, and Future of Simultaneous Localization And Mapping: Towards the Robust-Perception Age

Simultaneous Localization and Mapping (SLAM)consists in the concurrent construction of a model of the environment (the map), and the estimation of the state of the robot moving within it. The SLAM community has made astonishing progress over the last 30 years, enabling large-scale real-world applications, and witnessing a steady transition of this technology to industry. We survey the current state of SLAM. We start by presenting what is now the de-facto standard formulation for SLAM. We then review related work, covering a broad set of topics including robustness and scalability in long-term mapping, metric and semantic representations for mapping, theoretical performance guarantees, active SLAM and exploration, and other new frontiers. This paper simultaneously serves as a position paper and tutorial to those who are users of SLAM. By looking at the published research with a critical eye, we delineate open challenges and new research issues, that still deserve careful scientific investigation. The paper also contains the authors' take on two questions that often animate discussions during robotics conferences: Do robots need SLAM? and Is SLAM solved

Data-driven shape analysis and processing

Data-driven methods serve an increasingly important role in discovering geometric, structural, and semantic relationships between shapes. In contrast to traditional approaches that process shapes in isolation of each other, data-driven methods aggregate information from 3D model collections to improve the analysis, modeling and editing of shapes. Through reviewing the literature, we provide an overview of the main concepts and components of these methods, as well as discuss their application to classification, segmentation, matching, reconstruction, modeling and exploration, as well as scene analysis and synthesis. We conclude our report with ideas that can inspire future research in data-driven shape analysis and processing

Single View Modeling and View Synthesis

This thesis develops new algorithms to produce 3D content from a single camera. Today, amateurs can use hand-held camcorders to capture and display the 3D world in 2D, using mature technologies. However, there is always a strong desire to record and re-explore the 3D world in 3D. To achieve this goal, current approaches usually make use of a camera array, which suffers from tedious setup and calibration processes, as well as lack of portability, limiting its application to lab experiments. In this thesis, I try to produce the 3D contents using a single camera, making it as simple as shooting pictures. It requires a new front end capturing device rather than a regular camcorder, as well as more sophisticated algorithms. First, in order to capture the highly detailed object surfaces, I designed and developed a depth camera based on a novel technique called light fall-off stereo (LFS). The LFS depth camera outputs color+depth image sequences and achieves 30 fps, which is necessary for capturing dynamic scenes. Based on the output color+depth images, I developed a new approach that builds 3D models of dynamic and deformable objects. While the camera can only capture part of a whole object at any instance, partial surfaces are assembled together to form a complete 3D model by a novel warping algorithm. Inspired by the success of single view 3D modeling, I extended my exploration into 2D-3D video conversion that does not utilize a depth camera. I developed a semi-automatic system that converts monocular videos into stereoscopic videos, via view synthesis. It combines motion analysis with user interaction, aiming to transfer as much depth inferring work from the user to the computer. I developed two new methods that analyze the optical flow in order to provide additional qualitative depth constraints. The automatically extracted depth information is presented in the user interface to assist with user labeling work. In this thesis, I developed new algorithms to produce 3D contents from a single camera. Depending on the input data, my algorithm can build high fidelity 3D models for dynamic and deformable objects if depth maps are provided. Otherwise, it can turn the video clips into stereoscopic video

3차원 사람 자세 추정을 위한 3차원 복원, 약지도학습, 지도학습 방법

학위논문 (박사)-- 서울대학교 대학원 : 융합과학기술대학원 융합과학부(지능형융합시스템전공), 2019. 2. 곽노준.Estimating human poses from images is one of the fundamental tasks in computer vision, which leads to lots of applications such as action recognition, human-computer interaction, and virtual reality. Especially, estimating 3D human poses from 2D inputs is a challenging problem since it is inherently under-constrained. In addition, obtaining 3D ground truth data for human poses is only possible under the limited and restricted environments. In this dissertation, 3D human pose estimation is studied in different aspects focusing on various types of the availability of the data. To this end, three different methods to retrieve 3D human poses from 2D observations or from RGB images---algorithms of 3D reconstruction, weakly-supervised learning, and supervised learning---are proposed. First, a non-rigid structure from motion (NRSfM) algorithm that reconstructs 3D structures of non-rigid objects such as human bodies from 2D observations is proposed. In the proposed framework which is named as Procrustean Regression, the 3D shapes are regularized based on their aligned shapes. We show that the cost function of the Procrustean Regression can be casted into an unconstrained problem or a problem with simple bound constraints, which can be efficiently solved by existing gradient descent solvers. This framework can be easily integrated with numerous existing models and assumptions, which makes it more practical for various real situations. The experimental results show that the proposed method gives competitive result to the state-of-the-art methods for orthographic projection with much less time complexity and memory requirement, and outperforms the existing methods for perspective projection. Second, a weakly-supervised learning method that is capable of learning 3D structures when only 2D ground truth data is available as a training set is presented. Extending the Procrustean Regression framework, we suggest Procrustean Regression Network, a learning method that trains neural networks to learn 3D structures using training data with 2D ground truths. This is the first attempt that directly integrates an NRSfM algorithm into neural network training. The cost function that contains a low-rank function is also firstly used as a cost function of neural networks that reconstructs 3D shapes. During the test phase, 3D structures of human bodies can be obtained via a feed-forward operation, which enables the framework to have much faster inference time compared to the 3D reconstruction algorithms. Third, a supervised learning method that infers 3D poses from 2D inputs using neural networks is suggested. The method exploits a relational unit which captures the relations between different body parts. In the method, each pair of different body parts generates relational features, and the average of the features from all the pairs are used for 3D pose estimation. We also suggest a dropout method called relational dropout, which can be used in relational modules to impose robustness to the occlusions. The experimental results validate that the performance of the proposed algorithm does not degrade much when missing points exist while maintaining state-of-the-art performance when every point is visible.RGB 영상에서의 사람 자세 추정 방법은 컴퓨터 비전 분야에서 중요하며 여러 어플리케이션의 기본이 되는 기술이다. 사람 자세 추정은 동작 인식, 인간-컴퓨터 상호작용, 가상 현실, 증강 현실 등 광범위한 분야에서 기반 기술로 사용될 수 있다. 특히, 2차원 입력으로부터 3차원 사람 자세를 추정하는 문제는 무수히 많은 해를 가질 수 있는 문제이기 때문에 풀기 어려운 문제로 알려져 있다. 또한, 3차원 실제 데이터의 습득은 모션캡처 스튜디오 등 제한된 환경하에서만 가능하기 때문에 얻을 수 있는 데이터의 양이 한정적이다. 본 논문에서는, 얻을 수 있는 학습 데이터의 종류에 따라 여러 방면으로 3차원 사람 자세를 추정하는 방법을 연구하였다. 구체적으로, 2차원 관측값 또는 RGB 영상을 바탕으로 3차원 사람 자세를 추정, 복원하는 세 가지 방법--3차원 복원, 약지도학습, 지도학습--을 제시하였다. 첫 번째로, 사람의 신체와 같이 비정형 객체의 2차원 관측값으로부터 3차원 구조를 복원하는 비정형 움직임 기반 구조 (Non-rigid structure from motion) 알고리즘을 제안하였다. 프로크루스테스 회귀 (Procrustean regression)으로 명명한 제안된 프레임워크에서, 3차원 형태들은 그들의 정렬된 형태에 대한 함수로 정규화된다. 제안된 프로크루스테스 회귀의 비용 함수는 3차원 형태 정렬과 관련된 제약을 비용 함수에 포함시켜 경사 하강법을 이용한 최적화가 가능하다. 제안된 방법은 다양한 모델과 가정을 포함시킬 수 있어 실용적이고 유연한 프레임워크이다. 다양한 실험을 통해 제안된 방법은 세계 최고 수준의 방법들과 비교해 유사한 성능을 보이면서, 동시에 시간, 공간 복잡도 면에서 기존 방법에 비해 우수함을 보였다. 두 번째로 제안된 방법은, 2차원 학습 데이터만 주어졌을 때 2차원 입력에서 3차원 구조를 복원하는 약지도학습 방법이다. 프로크루스테스 회귀 신경망 (Procrustean regression network)로 명명한 제안된 학습 방법은 신경망 또는 컨볼루션 신경망을 통해 사람의 2차원 자세로부터 3차원 자세를 추정하는 방법을 학습한다. 프로크루스테스 회귀에 사용된 비용 함수를 수정하여 신경망을 학습시키는 본 방법은, 비정형 움직임 기반 구조에 사용된 비용 함수를 신경망 학습에 적용한 최초의 시도이다. 또한 비용함수에 사용된 저계수 함수 (low-rank function)를 신경망 학습에 처음으로 사용하였다. 테스트 데이터에 대해서 3차원 사람 자세는 신경망의 전방전달(feed forward)연산에 의해 얻어지므로, 3차원 복원 방법에 비해 훨씬 빠른 3차원 자세 추정이 가능하다. 마지막으로, 신경망을 이용해 2차원 입력으로부터 3차원 사람 자세를 추정하는 지도학습 방법을 제시하였다. 본 방법은 관계 신경망 모듈(relational modules)을 활용해 신체의 다른 부위간의 관계를 학습한다. 서로 다른 부위의 쌍마다 관계 특징을 추출해 모든 관계 특징의 평균을 최종 3차원 자세 추정에 사용한다. 또한 관계형 드랍아웃(relational dropout)이라는 새로운 학습 방법을 제시해 가려짐에 의해 나타나지 않은 2차원 관측값이 있는 상황에서, 강인하게 동작할 수 있는 3차원 자세 추정 방법을 제시하였다. 실험을 통해 해당 방법이 2차원 관측값이 일부만 주어진 상황에서도 큰 성능 하락이 없이 효과적으로 3차원 자세를 추정함을 증명하였다.Abstract i Contents iii List of Tables vi List of Figures viii 1 Introduction 1 1.1 Problem Definition . . . . . . . . . . . . . . . . . . . . . . . . 4 1.2 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.3 Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.4 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.4.1 3D Reconstruction of Human Bodies . . . . . . . . . . 9 1.4.2 Weakly-Supervised Learning for 3D HPE . . . . . . . . 11 1.4.3 Supervised Learning for 3D HPE . . . . . . . . . . . . 11 1.5 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2 Related Works 14 2.1 2D Human Pose Estimation . . . . . . . . . . . . . . . . . . . . 14 2.2 3D Human Pose Estimation . . . . . . . . . . . . . . . . . . . . 16 2.3 Non-rigid Structure from Motion . . . . . . . . . . . . . . . . . 18 2.4 Learning to Reconstruct 3D Structures via Neural Networks . . 23 3 3D Reconstruction of Human Bodies via Procrustean Regression 25 3.1 Formalization of NRSfM . . . . . . . . . . . . . . . . . . . . . 27 3.2 Procrustean Regression . . . . . . . . . . . . . . . . . . . . . . 28 3.2.1 The Cost Function of Procrustean Regression . . . . . . 29 3.2.2 Derivatives of the Cost Function . . . . . . . . . . . . . 32 3.2.3 Example Functions for f and g . . . . . . . . . . . . . . 38 3.2.4 Handling Missing Points . . . . . . . . . . . . . . . . . 43 3.2.5 Optimization . . . . . . . . . . . . . . . . . . . . . . . 44 3.2.6 Initialization . . . . . . . . . . . . . . . . . . . . . . . 44 3.3 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . 45 3.3.1 Orthographic Projection . . . . . . . . . . . . . . . . . 46 3.3.2 Perspective Projection . . . . . . . . . . . . . . . . . . 56 3.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 3.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 4 Weakly-Supervised Learning of 3D Human Pose via Procrustean Regression Networks 69 4.1 The Cost Function for Procrustean Regression Network . . . . . 70 4.2 Choosing f and g for Procrustean Regression Network . . . . . 74 4.3 Implementation Details . . . . . . . . . . . . . . . . . . . . . . 75 4.4 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . 77 4.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 5 Supervised Learning of 3D Human Pose via Relational Networks 86 5.1 Relational Networks . . . . . . . . . . . . . . . . . . . . . . . 88 5.2 Relational Networks for 3D HPE . . . . . . . . . . . . . . . . . 88 5.3 Extensions to Multi-Frame Inputs . . . . . . . . . . . . . . . . 91 5.4 Relational Dropout . . . . . . . . . . . . . . . . . . . . . . . . 93 5.5 Implementation Details . . . . . . . . . . . . . . . . . . . . . . 94 5.6 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . 95 5.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 6 Concluding Remarks 105 6.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 6.2 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 6.3 Future Directions . . . . . . . . . . . . . . . . . . . . . . . . . 108 Abstract (In Korean) 128Docto

Direct medical image-based Finite Element modelling for patient-specific simulation of future implants

[EN] In patient specific biomedical simulation, the numerical model is usually created after cumbersome, time consuming procedures which often require highly specialized human work and a great amount of man-hours to be carried out. In order to make numerical simulation available for medical practice, it is of primary importance to reduce the cost associated to these procedures by making them automatic. In this paper a method for the automatic creation of Finite Element (FE) models from medical images is presented. This method is based on the use of a hierarchical structure of nested Cartesian grids in which the medical image is immersed. An efficient h-adaptive procedure conforms the FE model to the image characteristics by refining the mesh on the basis of the distribution of elastic properties associated to the pixel values. As a result, a problem with a reasonable number of degrees of freedom is obtained, skipping the geometry creation stage. All the image information is taken into account during the calculation of the element stiffness matrix, therefore it is straightforward to include the material heterogeneity in the simulation. The proposed method is an adapted version of the Cartesian grid Finite Element Method (cgFEM) for the FE analysis of objects defined by images. cgFEM is an immersed boundary method that uses h-adaptive Cartesian meshes non-conforming to the boundary of the object to be analysed. The proposed methodology, used together with the original geometry-based cgFEM, allows prosthesis geometries to be easily introduced in the model providing a useful tool for evaluating the effect of future implants in a preoperative framework. The potential of this kind of technology is presented by mean of an initial implementation in 2D and 3D for linear elasticity problems.With the support of the European Union Framework Programme (FP7) under grant agreement No. 289361 'Integrating Numerical Simulation and Geometric Design Technology (INSIST)', the Ministerio de Economia y Competitividad of Spain (DPI2010-20542) and the Generalitat Valenciana (PROMETEO/2016/007).Giovannelli, L.; Ródenas, J.; Navarro-Jiménez, J.; Tur Valiente, M. (2017). Direct medical image-based Finite Element modelling for patient-specific simulation of future implants. Finite Elements in Analysis and Design. 136:37-57. https://doi.org/10.1016/j.finel.2017.07.010S375713