Superquadric representation of scenes from multi-view range data

Object representation denotes representing three-dimensional (3D) real-world objects with known graphic or mathematic primitives recognizable to computers. This research has numerous applications for object-related tasks in areas including computer vision, computer graphics, reverse engineering, etc. Superquadrics, as volumetric and parametric models, have been selected to be the representation primitives throughout this research. Superquadrics are able to represent a large family of solid shapes by a single equation with only a few parameters. This dissertation addresses superquadric representation of multi-part objects and multiobject scenes. Two issues motivate this research. First, superquadric representation of multipart objects or multi-object scenes has been an unsolved problem due to the complex geometry of objects. Second, superquadrics recovered from single-view range data tend to have low confidence and accuracy due to partially scanned object surfaces caused by inherent occlusions. To address these two problems, this dissertation proposes a multi-view superquadric representation algorithm. By incorporating both part decomposition and multi-view range data, the proposed algorithm is able to not only represent multi-part objects or multi-object scenes, but also achieve high confidence and accuracy of recovered superquadrics. The multi-view superquadric representation algorithm consists of (i) initial superquadric model recovery from single-view range data, (ii) pairwise view registration based on recovered superquadric models, (iii) view integration, (iv) part decomposition, and (v) final superquadric fitting for each decomposed part. Within the multi-view superquadric representation framework, this dissertation proposes a 3D part decomposition algorithm to automatically decompose multi-part objects or multiobject scenes into their constituent single parts consistent with human visual perception. Superquadrics can then be recovered for each decomposed single-part object. The proposed part decomposition algorithm is based on curvature analysis, and includes (i) Gaussian curvature estimation, (ii) boundary labeling, (iii) part growing and labeling, and (iv) post-processing. In addition, this dissertation proposes an extended view registration algorithm based on superquadrics. The proposed view registration algorithm is able to handle deformable superquadrics as well as 3D unstructured data sets. For superquadric fitting, two objective functions primarily used in the literature have been comprehensively investigated with respect to noise, viewpoints, sample resolutions, etc. The objective function proved to have better performance has been used throughout this dissertation. In summary, the three algorithms (contributions) proposed in this dissertation are generic and flexible in the sense of handling triangle meshes, which are standard surface primitives in computer vision and graphics. For each proposed algorithm, the dissertation presents both theory and experimental results. The results demonstrate the efficiency of the algorithms using both synthetic and real range data of a large variety of objects and scenes. In addition, the experimental results include comparisons with previous methods from the literature. Finally, the dissertation concludes with a summary of the contributions to the state of the art in superquadric representation, and presents possible future extensions to this research

Superquadric Library, User Manual and Utility Programs

Superquadrics are a family of parametric shapes that have been used as primitives for shape representation in computer vision and computer graphics. They can be used for modeling tapering and bending deformations and are recovered efficiently by a stable numerical procedure. This document introduces the superquadric library, SQ_lib, developed at the GRASP Lab at the University of Pennsylvania. The manual is organized into three parts. The first part provides the reader with a description of superquadrics models and deformations that can be performed. Furthermore, it introduces the coordinate systems conventions which are used in the library. The second part presents some examples of applications on how one can use the functions defined in the library. It also lists utility programs which have been developed while conducting research. They provide a good source of examples for the application of the library. Finally, the last part describes the datatypes and each of the functions which are supported in the library. The library itself is organized in two sets Fundamental and Auxiliary functions. A quick reference to all the functions and an index is provided. Some of the functions and examples supplied perform data preprocessing and are connected to the PM image description also available from the GRASP Lab. These functions are provided in isolation from the remaining body of the library and can easily be excluded in the actual compilation of the library. Furthermore, routines for the visualization of the data, using X11, are also provided

Physics-Based Modeling of Nonrigid Objects for Vision and Graphics (Dissertation)

This thesis develops a physics-based framework for 3D shape and nonrigid motion modeling for computer vision and computer graphics. In computer vision it addresses the problems of complex 3D shape representation, shape reconstruction, quantitative model extraction from biomedical data for analysis and visualization, shape estimation, and motion tracking. In computer graphics it demonstrates the generative power of our framework to synthesize constrained shapes, nonrigid object motions and object interactions for the purposes of computer animation. Our framework is based on the use of a new class of dynamically deformable primitives which allow the combination of global and local deformations. It incorporates physical constraints to compose articulated models from deformable primitives and provides force-based techniques for fitting such models to sparse, noise-corrupted 2D and 3D visual data. The framework leads to shape and nonrigid motion estimators that exploit dynamically deformable models to track moving 3D objects from time-varying observations. We develop models with global deformation parameters which represent the salient shape features of natural parts, and local deformation parameters which capture shape details. In the context of computer graphics, these models represent the physics-based marriage of the parameterized and free-form modeling paradigms. An important benefit of their global/local descriptive power in the context of computer vision is that it can potentially satisfy the often conflicting requirements of shape reconstruction and shape recognition. The Lagrange equations of motion that govern our models, augmented by constraints, make them responsive to externally applied forces derived from input data or applied by the user. This system of differential equations is discretized using finite element methods and simulated through time using standard numerical techniques. We employ these equations to formulate a shape and nonrigid motion estimator. The estimator is a continuous extended Kalman filter that recursively transforms the discrepancy between the sensory data and the estimated model state into generalized forces. These adjust the translational, rotational, and deformational degrees of freedom such that the model evolves in a consistent fashion with the noisy data. We demonstrate the interactive time performance of our techniques in a series of experiments in computer vision, graphics, and visualization

Surface and Volumetric Segmentation of Complex 3-D Objects Using Parametric Shape Models

The problem of part definition, description, and decomposition is central to the shape recognition systems. In this dissertation, we develop an integrated framework for segmenting dense range data of complex 3-D scenes into their constituent parts in terms of surface and volumetric primitives. Unlike previous approaches, we use geometric properties derived from surface, as well as volumetric models, to recover structured descriptions of complex objects without a priori domain knowledge or stored models. To recover shape descriptions, we use bi-quadric models for surface representation and superquadric models for object-centered volumetric representation. The surface segmentation uses a novel approach of searching for the best piecewise description of the image in terms of bi-quadric (z = f(x,y)) models. It is used to generate the region adjacency graphs, to localize surface discontinuities, and to derive global shape properties of the surfaces. A superquadric model is recovered for the entire data set and residuals are computed to evaluate the fit. The goodness-of-fit value based on the inside-outside function, and the mean-squared distance of data from the model provide quantitative evaluation of the model. The qualitative evaluation criteria check the local consistency of the model in the form of residual maps of overestimated and underestimated data regions. The control structure invokes the models in a systematic manner, evaluates the intermediate descriptions, and integrates them to achieve final segmentation. Superquadric and bi-quadric models are recovered in parallel to incorporate the best of the coarse-to-fine and fine-to-coarse segmentation strategies. The model evaluation criteria determine the dimensionality of the scene, and decide whether to terminate the procedure, or selectively refine the segmentation by following a global-to-local part segmentation approach. The control module generates hypotheses about superquadric models at clusters of underestimated data and performs controlled extrapolation of the part-model by shrinking the global model. As the global model shrinks and the local models grow, they are evaluated and tested for termination or further segmentation. We present results on real range images of scenes of varying complexity, including objects with occluding parts, and scenes where surface segmentation is not sufficient to guide the volumetric segmentation. We analyze the issue of segmentation of complex scenes thoroughly by studying the effect of missing data on volumetric model recovery, generating object-centered descriptions, and presenting a complete set of criteria for the evaluation of the superquadric models. We conclude by discussing the applications of our approach in data reduction, 3-D object recognition, geometric modeling, automatic model generation. object manipulation, and active vision

Iterative Superquadric Recomposition of 3D Objects from Multiple Views

Humans are good at recomposing novel objects, i.e. they can identify commonalities between unknown objects from general structure to finer detail, an ability difficult to replicate by machines. We propose a framework, ISCO, to recompose an object using 3D superquadrics as semantic parts directly from 2D views without training a model that uses 3D supervision. To achieve this, we optimize the superquadric parameters that compose a specific instance of the object, comparing its rendered 3D view and 2D image silhouette. Our ISCO framework iteratively adds new superquadrics wherever the reconstruction error is high, abstracting first coarse regions and then finer details of the target object. With this simple coarse-to-fine inductive bias, ISCO provides consistent superquadrics for related object parts, despite not having any semantic supervision. Since ISCO does not train any neural network, it is also inherently robust to out-of-distribution objects. Experiments show that, compared to recent single instance superquadrics reconstruction approaches, ISCO provides consistently more accurate 3D reconstructions, even from images in the wild. Code available at https://github.com/ExplainableML/ISCO .Comment: Accepted at ICCV 202

Generalized ellipsoids and anisotropic filtering for segmentation improvement in 3D medical imaging

Deformable models have demonstrated to be very useful techniques for image segmentation. However, they present several weak points. Two of the main problems with deformable models are the following: (1) results are often dependent on the initial model location, and (2) the generation of image potentials is very sensitive to noise. Modeling and preprocessing methods presented in this paper contribute to solve these problems. We propose an initialization tool to obtain a good approximation to global shape and location of a given object into a 3D image. We also introduce a novel technique for corner preserving anisotropic diffusion filtering to improve contrast and corner measures. This is useful for both guiding initialization (global shape) and subsequent deformation for fine tuning (local shape).This work was supported by the Spanish Government and the Xunta de Galicia by projects TIC2000-0399-C02-02 and PGIDT99PXI20606B, respectively.2005-04-01S

Geometry-Aware Network for Non-Rigid Shape Prediction from a Single View

We propose a method for predicting the 3D shape of a deformable surface from a single view. By contrast with previous approaches, we do not need a pre-registered template of the surface, and our method is robust to the lack of texture and partial occlusions. At the core of our approach is a {\it geometry-aware} deep architecture that tackles the problem as usually done in analytic solutions: first perform 2D detection of the mesh and then estimate a 3D shape that is geometrically consistent with the image. We train this architecture in an end-to-end manner using a large dataset of synthetic renderings of shapes under different levels of deformation, material properties, textures and lighting conditions. We evaluate our approach on a test split of this dataset and available real benchmarks, consistently improving state-of-the-art solutions with a significantly lower computational time.Comment: Accepted at CVPR 201

The mitral valve computational anatomy and geometry analysis

We present a novel methodology to characterize and quantify the Mitral Valve (MV) geometry and physical attributes in a multi-resolution framework. A multi-scale decomposition was implemented to model the MV geometry by using superquadric shape primitives and spectral reconstruction of the finer-scale geometric details. Superquadrics provide a basis to normalize the size and approximate a basic model of the MV geometry. The point-wise difference between the original geometry and the superquadric model denotes the finer-scale geometric details, which can be modeled as a scalar attribute for the MV model development. The additive decomposition of the basic MV geometry from geometric details (attributes) allows recovering the actual geometry by superposition of the superquadric approximation and the finer-details model. We implemented a lasso optimization algorithm to perform spectral analysis and develop the Fourier reconstruction of the geometric details. The spectral modeling enabled us to resample the geometric details or use spectral filters in order to adjust the spatial resolution in the model reconstruction. It also provides the basis to control the level of detail in the final model reconstruction by applying low-pass filters in the frequency domain. The higher-order attributes such as internal fiber architecture can be integrated with the geometric models using the same framework. We applied our pipeline to create models of three ovine MVs based on computed-tomography 3D images with micrometer resolution. We were able to quantify the MV leaflet geometry, reconstruct models with custom level of geometric details, and develop medial representation of the MV leaflet structure. The results show that our methodology for geometry analysis provides a basis for assessing patient-specific geometries and facilitates developing population-averaged models. Ultimately, this approach allows building personalized image-based computational models for medical device design and surgical treatment simulations.Mechanical Engineerin

Part-Based Segmentation and Modeling of Range Data by Moving Target

n/