A Low-Dimensional Representation for Robust Partial Isometric Correspondences Computation

Intrinsic isometric shape matching has become the standard approach for pose invariant correspondence estimation among deformable shapes. Most existing approaches assume global consistency, i.e., the metric structure of the whole manifold must not change significantly. While global isometric matching is well understood, only a few heuristic solutions are known for partial matching. Partial matching is particularly important for robustness to topological noise (incomplete data and contacts), which is a common problem in real-world 3D scanner data. In this paper, we introduce a new approach to partial, intrinsic isometric matching. Our method is based on the observation that isometries are fully determined by purely local information: a map of a single point and its tangent space fixes an isometry for both global and the partial maps. From this idea, we develop a new representation for partial isometric maps based on equivalence classes of correspondences between pairs of points and their tangent spaces. From this, we derive a local propagation algorithm that find such mappings efficiently. In contrast to previous heuristics based on RANSAC or expectation maximization, our method is based on a simple and sound theoretical model and fully deterministic. We apply our approach to register partial point clouds and compare it to the state-of-the-art methods, where we obtain significant improvements over global methods for real-world data and stronger guarantees than previous heuristic partial matching algorithms.Comment: 17 pages, 12 figure

Slippage Features

In this report, we present a novel feature detection technique for unstructured point clouds. We introduce a generalized concept of geometric features that detects locally uniquely identifiable keypoints as centroids of area with locally minimal slippage. We extend the concept to multiple scales and extract features using multi-scale mean shift clustering. In order to validate matches between feature points, we employ a two stage technique that first sorts out unlikely matches, followed by an approximate alignment between remaining features by a rotational cross-correlation analysis and a local iterative closest point (ICP) registration. The resulting residuals are then used as final similarity measure. The proposed combination of techniques results in a robust and reliable correspondence detection technique that yields registration results in situations where previous techniques are not able to detect usable feature correspondences. We provide a detailed empirical analysis of the method, and apply the technique to global registration, symmetry detection and deformable matching problems

Surface Registration for Pharyngeal Radiation Treatment Planning

Endoscopy is an in-body examination procedure that enables direct visualization of tumor spread on tissue surfaces. In the context of radiation treatment planning for throat cancer, there have been attempts to fuse this endoscopic information into the planning CT space for better tumor localization. One way to achieve this CT/Endoscope fusion is to first reconstruct a full 3D surface model from the endoscopic video and then register that surface into the CT space. These two steps both require an algorithm that can accurately register two or more surfaces. In this dissertation, I present a surface registration method I have developed, called Thin Shell Demons (TSD), for achieving the two goals mentioned above. There are two key aspects in TSD: geometry and mechanics. First, I develop a novel surface geometric feature descriptor based on multi-scale curvatures that can accurately capture local shape information. I show that the descriptor can be effectively used in TSD and other surface registration frameworks, such as spectral graph matching. Second, I adopt a physical thin shell model in TSD to produce realistic surface deformation in the registration process. I also extend this physical model for orthotropic thin shells and propose a probabilistic framework to learn orthotropic stiffness parameters from a group of known deformations. The anisotropic stiffness learning opens up a new perspective to shape analysis and allows more accurate surface deformation and registration in the TSD framework. Finally, I show that TSD can also be extended into a novel groupwise registration framework. The advantages of Thin Shell Demons allow us to build a complete 3D model of the throat, called an endoscopogram, from a group of single-frame-based reconstructions. It also allows us to register an endoscopogram to a CT segmentation surface, thereby allowing information transfer for treatment planning.Doctor of Philosoph

Geometric modeling of non-rigid 3D shapes : theory and application to object recognition.

One of the major goals of computer vision is the development of flexible and efficient methods for shape representation. This is true, especially for non-rigid 3D shapes where a great variety of shapes are produced as a result of deformations of a non-rigid object. Modeling these non-rigid shapes is a very challenging problem. Being able to analyze the properties of such shapes and describe their behavior is the key issue in research. Also, considering photometric features can play an important role in many shape analysis applications, such as shape matching and correspondence because it contains rich information about the visual appearance of real objects. This new information (contained in photometric features) and its important applications add another, new dimension to the problem\u27s difficulty. Two main approaches have been adopted in the literature for shape modeling for the matching and retrieval problem, local and global approaches. Local matching is performed between sparse points or regions of the shape, while the global shape approaches similarity is measured among entire models. These methods have an underlying assumption that shapes are rigidly transformed. And Most descriptors proposed so far are confined to shape, that is, they analyze only geometric and/or topological properties of 3D models. A shape descriptor or model should be isometry invariant, scale invariant, be able to capture the fine details of the shape, computationally efficient, and have many other good properties. A shape descriptor or model is needed. This shape descriptor should be: able to deal with the non-rigid shape deformation, able to handle the scale variation problem with less sensitivity to noise, able to match shapes related to the same class even if these shapes have missing parts, and able to encode both the photometric, and geometric information in one descriptor. This dissertation will address the problem of 3D non-rigid shape representation and textured 3D non-rigid shapes based on local features. Two approaches will be proposed for non-rigid shape matching and retrieval based on Heat Kernel (HK), and Scale-Invariant Heat Kernel (SI-HK) and one approach for modeling textured 3D non-rigid shapes based on scale-invariant Weighted Heat Kernel Signature (WHKS). For the first approach, the Laplace-Beltrami eigenfunctions is used to detect a small number of critical points on the shape surface. Then a shape descriptor is formed based on the heat kernels at the detected critical points for different scales. Sparse representation is used to reduce the dimensionality of the calculated descriptor. The proposed descriptor is used for classification via the Collaborative Representation-based Classification with a Regularized Least Square (CRC-RLS) algorithm. The experimental results have shown that the proposed descriptor can achieve state-of-the-art results on two benchmark data sets. For the second approach, an improved method to introduce scale-invariance has been also proposed to avoid noise-sensitive operations in the original transformation method. Then a new 3D shape descriptor is formed based on the histograms of the scale-invariant HK for a number of critical points on the shape at different time scales. A Collaborative Classification (CC) scheme is then employed for object classification. The experimental results have shown that the proposed descriptor can achieve high performance on the two benchmark data sets. An important observation from the experiments is that the proposed approach is more able to handle data under several distortion scenarios (noise, shot-noise, scale, and under missing parts) than the well-known approaches. For modeling textured 3D non-rigid shapes, this dissertation introduces, for the first time, a mathematical framework for the diffusion geometry on textured shapes. This dissertation presents an approach for shape matching and retrieval based on a weighted heat kernel signature. It shows how to include photometric information as a weight over the shape manifold, and it also propose a novel formulation for heat diffusion over weighted manifolds. Then this dissertation presents a new discretization method for the weighted heat kernel induced by the linear FEM weights. Finally, the weighted heat kernel signature is used as a shape descriptor. The proposed descriptor encodes both the photometric, and geometric information based on the solution of one equation. Finally, this dissertation proposes an approach for 3D face recognition based on the front contours of heat propagation over the face surface. The front contours are extracted automatically as heat is propagating starting from a detected set of landmarks. The propagation contours are used to successfully discriminate the various faces. The proposed approach is evaluated on the largest publicly available database of 3D facial images and successfully compared to the state-of-the-art approaches in the literature. This work can be extended to the problem of dense correspondence between non-rigid shapes. The proposed approaches with the properties of the Laplace-Beltrami eigenfunction can be utilized for 3D mesh segmentation. Another possible application of the proposed approach is the view point selection for 3D objects by selecting the most informative views that collectively provide the most descriptive presentation of the surface

Learning with Latent Representations of 3D Data: from Classical Methods to 3D Deep Learning

3D data contain rich information about the full geometry of objects or scenes. Learning tasks on them have always been considered as hard ones in the computer vision community due to their extreme high dimensionality. Hence, latent representations of 3D geometries are often used to lower the data dimensionality for better parameterization and easier computation. In this report, we make a brief review on those latent representations obtained via different methods including classical ones and the emerging neural learning-based ones. Furthermore, the nowadays widely used deep learning methods have also been more closely investigated regarding their applications on various 3D data formats. The possibility of combing those two kinds of methods has also been addressed

Consistent Correspondences for Shape and Image Problems

Establish consistent correspondences between different objects is a classic problem in computer science/vision. It helps to match highly similar objects in both 3D and 2D domain. Inthe 3D domain, finding consistent correspondences has been studying for more than 20 yearsand it is still a hot topic. In 2D domain, consistent correspondences can also help in puzzlesolving. However, only a few works are focused on this approach. In this thesis, we focuson finding consistent correspondences and extend to develop robust matching techniques inboth 3D shape segments and 2D puzzle solving. In the 3D domain, segment-wise matching isan important research problem that supports higher-level understanding of shapes in geometryprocessing. Many existing segment-wise matching techniques assume perfect input segmentation and would suffer from imperfect or over-segmented input. To handle this shortcoming,we propose multi-layer graphs (MLGs) to represent possible arrangements of partially mergedsegments of input shapes. We then adapt the diffusion pruning technique on the MLGs to findconsistent segment-wise matching. To obtain high-quality matching, we develop our own voting step which is able to remove inconsistent results, for finding hierarchically consistent correspondences as final output. We evaluate our technique with both quantitative and qualitativeexperiments on both man-made and deformable shapes. Experimental results demonstrate theeffectiveness of our technique when compared to two state-of-art methods. In the 2D domain,solving jigsaw puzzles is also a classic problem in computer vision with various applications.Over the past decades, many useful approaches have been introduced. Most existing worksuse edge-wise similarity measures for assembling puzzles with square pieces of the same size, and recent work innovates to use the loop constraint to improve efficiency and accuracy. Weobserve that most existing techniques cannot be easily extended to puzzles with rectangularpieces of arbitrary sizes, and no existing loop constraints can be used to model such challenging scenarios. We propose new matching approaches based on sub-edges/corners, modelledusing the MatchLift or diffusion framework to solve square puzzles with cycle consistency.We demonstrate the robustness of our approaches by comparing our methods with state-of-artmethods. We also show how puzzles with rectangular pieces of arbitrary sizes, or puzzles withtriangular and square pieces can be solved by our techniques