Surface Modeling and Analysis Using Range Images: Smoothing, Registration, Integration, and Segmentation

This dissertation presents a framework for 3D reconstruction and scene analysis, using a set of range images. The motivation for developing this framework came from the needs to reconstruct the surfaces of small mechanical parts in reverse engineering tasks, build a virtual environment of indoor and outdoor scenes, and understand 3D images. The input of the framework is a set of range images of an object or a scene captured by range scanners. The output is a triangulated surface that can be segmented into meaningful parts. A textured surface can be reconstructed if color images are provided. The framework consists of surface smoothing, registration, integration, and segmentation. Surface smoothing eliminates the noise present in raw measurements from range scanners. This research proposes area-decreasing flow that is theoretically identical to the mean curvature flow. Using area-decreasing flow, there is no need to estimate the curvature value and an optimal step size of the flow can be obtained. Crease edges and sharp corners are preserved by an adaptive scheme. Surface registration aligns measurements from different viewpoints in a common coordinate system. This research proposes a new surface representation scheme named point fingerprint. Surfaces are registered by finding corresponding point pairs in an overlapping region based on fingerprint comparison. Surface integration merges registered surface patches into a whole surface. This research employs an implicit surface-based integration technique. The proposed algorithm can generate watertight models by space carving or filling the holes based on volumetric interpolation. Textures from different views are integrated inside a volumetric grid. Surface segmentation is useful to decompose CAD models in reverse engineering tasks and help object recognition in a 3D scene. This research proposes a watershed-based surface mesh segmentation approach. The new algorithm accurately segments the plateaus by geodesic erosion using fast marching method. The performance of the framework is presented using both synthetic and real world data from different range scanners. The dissertation concludes by summarizing the development of the framework and then suggests future research topics

TetraSphere: A Neural Descriptor for O(3)-Invariant Point Cloud Analysis

Rotation invariance is an important requirement for the analysis of 3D point clouds. In this paper, we present a learnable descriptor for rotation- and reflection-invariant 3D point cloud analysis based on recently introduced steerable 3D spherical neurons and vector neurons. Specifically, we show the compatibility of the two approaches and apply steerable neurons in an end-to-end method, which both constitute the technical novelty. In our approach, we perform TetraTransform -- which lifts the 3D input to an equivariant 4D representation, constructed by the steerable neurons -- and extract deeper rotation-equivariant features using vector neurons. This integration of the TetraTransform into the VN-DGCNN framework, termed TetraSphere, inexpensively increases the number of parameters by less than 0.0007%. Taking only points as input, TetraSphere sets a new state-of-the-art performance classifying randomly rotated real-world object scans of the hardest subset of ScanObjectNN, even when trained on data without additional rotation augmentation. Additionally, TetraSphere demonstrates the second-best performance segmenting parts of the synthetic ShapeNet, consistently outperforming the baseline VN-DGCNN. All in all, our results reveal the practical value of steerable 3D spherical neurons for learning in 3D Euclidean space

Real-Time Segmentation of 4D Ultrasound by Active Geometric Functions

Four-dimensional ultrasound based on matrix phased array transducers can capture the complex 4D cardiac motion in a complete and real-time fashion. However, the large amount of information residing in 4D ultrasound scans and novel applications under interventional settings pose a big challenge in efficiency for workflow and computer-aided diagnostic algorithms such as segmentation. In this context, a novel formulation framework of the minimal surface problem, called active geometric functions (AGF), is proposed to reach truly real-time performance in segmenting 4D ultrasound data. A specific instance of AGF based on finite element modeling and Hermite surface descriptors was implemented and evaluated on 35 4D ultrasound data sets with a total of 425 time frames. Quantitative comparison to manual tracing showed that the proposed method provides LV contours close to manual segmentation and that the discrepancy was comparable to inter-observer tracing variability. The ability of such realtime segmentation will not only facilitate the diagnoses and workflow, but also enables novel applications such as interventional guidance and interactive image acquisition with online segmentation

Geometry of the ergodic quotient reveals coherent structures in flows

Dynamical systems that exhibit diverse behaviors can rarely be completely understood using a single approach. However, by identifying coherent structures in their state spaces, i.e., regions of uniform and simpler behavior, we could hope to study each of the structures separately and then form the understanding of the system as a whole. The method we present in this paper uses trajectory averages of scalar functions on the state space to: (a) identify invariant sets in the state space, (b) form coherent structures by aggregating invariant sets that are similar across multiple spatial scales. First, we construct the ergodic quotient, the object obtained by mapping trajectories to the space of trajectory averages of a function basis on the state space. Second, we endow the ergodic quotient with a metric structure that successfully captures how similar the invariant sets are in the state space. Finally, we parametrize the ergodic quotient using intrinsic diffusion modes on it. By segmenting the ergodic quotient based on the diffusion modes, we extract coherent features in the state space of the dynamical system. The algorithm is validated by analyzing the Arnold-Beltrami-Childress flow, which was the test-bed for alternative approaches: the Ulam's approximation of the transfer operator and the computation of Lagrangian Coherent Structures. Furthermore, we explain how the method extends the Poincar\'e map analysis for periodic flows. As a demonstration, we apply the method to a periodically-driven three-dimensional Hill's vortex flow, discovering unknown coherent structures in its state space. In the end, we discuss differences between the ergodic quotient and alternatives, propose a generalization to analysis of (quasi-)periodic structures, and lay out future research directions.Comment: Submitted to Elsevier Physica D: Nonlinear Phenomen

Deep representations of structures in the 3D-world

This thesis demonstrates a collection of neural network tools that leverage the structures and symmetries of the 3D-world. We have explored various aspects of a vision system ranging from relative pose estimation to 3D-part decomposition from 2D images. For any vision system, it is crucially important to understand and to resolve visual ambiguities in 3D arising from imaging methods. This thesis has shown that leveraging prior knowledge about the structures and the symmetries of the 3D-world in neural network architectures brings about better representations for ambiguous situations. It helps solve problems which are inherently ill-posed

Semi-Automatic segmentation of multiple mouse embryos in MR images

<p>Abstract</p> <p>Background</p> <p>The motivation behind this paper is to aid the automatic phenotyping of mouse embryos, wherein multiple embryos embedded within a single tube were scanned using Magnetic Resonance Imaging (MRI).</p> <p>Results</p> <p>Our algorithm, a modified version of the simplex deformable model of Delingette, addresses various issues with deformable models including initialization and inability to adapt to boundary concavities. In addition, it proposes a novel technique for automatic collision detection of multiple objects which are being segmented simultaneously, hence avoiding major leaks into adjacent neighbouring structures. We address the initialization problem by introducing balloon forces which expand the initial spherical models close to the true boundaries of the embryos. This results in models which are less sensitive to initial minimum of two fold after each stage of deformation. To determine collision during segmentation, our unique collision detection algorithm finds the intersection between binary masks created from the deformed models after every few iterations of the deformation and modifies the segmentation parameters accordingly hence avoiding collision.</p> <p>We have segmented six tubes of three dimensional MR images of multiple mouse embryos using our modified deformable model algorithm. We have then validated the results of the our semi-automatic segmentation versus manual segmentation of the same embryos. Our Validation shows that except paws and tails we have been able to segment the mouse embryos with minor error.</p> <p>Conclusions</p> <p>This paper describes our novel multiple object segmentation technique with collision detection using a modified deformable model algorithm. Further, it presents the results of segmenting magnetic resonance images of up to 32 mouse embryos stacked in one gel filled test tube and creating 32 individual masks.</p

Consistent Density Scanning and Information Extraction From Point Clouds of Building Interiors

Over the last decade, 3D range scanning systems have improved considerably enabling the designers to capture large and complex domains such as building interiors. The captured point cloud is processed to extract specific Building Information Models, where the main research challenge is to simultaneously handle huge and cohesive point clouds representing multiple objects, occluded features and vast geometric diversity. These domain characteristics increase the data complexities and thus make it difficult to extract accurate information models from the captured point clouds. The research work presented in this thesis improves the information extraction pipeline with the development of novel algorithms for consistent density scanning and information extraction automation for building interiors. A restricted density-based, scan planning methodology computes the number of scans to cover large linear domains while ensuring desired data density and reducing rigorous post-processing of data sets. The research work further develops effective algorithms to transform the captured data into information models in terms of domain features (layouts), meaningful data clusters (segmented data) and specific shape attributes (occluded boundaries) having better practical utility. Initially, a direct point-based simplification and layout extraction algorithm is presented that can handle the cohesive point clouds by adaptive simplification and an accurate layout extraction approach without generating an intermediate model. Further, three information extraction algorithms are presented that transforms point clouds into meaningful clusters. The novelty of these algorithms lies in the fact that they work directly on point clouds by exploiting their inherent characteristic. First a rapid data clustering algorithm is presented to quickly identify objects in the scanned scene using a robust hue, saturation and value (H S V) color model for better scene understanding. A hierarchical clustering algorithm is developed to handle the vast geometric diversity ranging from planar walls to complex freeform objects. The shape adaptive parameters help to segment planar as well as complex interiors whereas combining color and geometry based segmentation criterion improves clustering reliability and identifies unique clusters from geometrically similar regions. Finally, a progressive scan line based, side-ratio constraint algorithm is presented to identify occluded boundary data points by investigating their spatial discontinuity