Surface Reconstruction from Noisy and Sparse Data

We introduce a set of algorithms for registering, filtering and measuring the similarity of unorganized 3d point clouds, usually obtained from multiple views. We contribute a method for computing the similarity between point clouds that represent closed surfaces, specifically segmented tumors from CT scans. We obtain watertight surfaces and utilize volumetric overlap to determine similarity in a volumetric way. This similarity measure is used to quantify treatment variability based on target volume segmentation both prior to and following radiotherapy planning stages. We also contribute an algorithm for the drift-free registration of thin, non- rigid scans, where drift is the build-up of error caused by sequential pairwise registration, which is the alignment of each scan to its neighbor. We construct an average scan using mutual nearest neighbors, each scan is registered to this average scan, after which we update the average scan and continue this process until convergence. The use case herein is for merging scans of plants from multiple views and registering vascular scans together. Our final contribution is a method for filtering noisy point clouds, specif- ically those constructed from merged depth maps as obtained from a range scanner or multiple view stereo (MVS), applying techniques that have been utilized in finding outliers in clustered data, but not in MVS. We utilize ker- nel density estimation to obtain a probability density function over the space of observed points, utilizing variable bandwidths based on the nature of the neighboring points, Mahalanobis and reachability distances that is more dis- criminative than a classical Mahalanobis distance-based metric

A Survey of Surface Reconstruction from Point Clouds

International audienceThe area of surface reconstruction has seen substantial progress in the past two decades. The traditional problem addressed by surface reconstruction is to recover the digital representation of a physical shape that has been scanned, where the scanned data contains a wide variety of defects. While much of the earlier work has been focused on reconstructing a piece-wise smooth representation of the original shape, recent work has taken on more specialized priors to address significantly challenging data imperfections, where the reconstruction can take on different representations – not necessarily the explicit geometry. We survey the field of surface reconstruction, and provide a categorization with respect to priors, data imperfections, and reconstruction output. By considering a holistic view of surface reconstruction, we show a detailed characterization of the field, highlight similarities between diverse reconstruction techniques, and provide directions for future work in surface reconstruction

Biharmonic fields and mesh completion

We discuss bi-harmonic fields which approximate signed distance fields. We conclude that the bi-harmonic field approximation can be a powerful tool for mesh completion in general and complex cases. We present an adaptive, multigrid algorithm to extrapolate signed distance fields. By defining a volume mask in a closed region bounding the area that must be repaired, the algorithm computes a signed distance field in well-defined regions and uses it as an over-determined boundary condition constraint for the biharmonic field computation in the remaining regions. The algorithm operates locally, within an expanded bounding box of each hole, and therefore scales well with the number of holes in a single, complex model. We discuss this approximation in practical examples in the case of triangular meshes resulting from laser scan acquisitions which require massive hole repair. We conclude that the proposed algorithm is robust and general, and is able to deal with complex topological casesPeer ReviewedPostprint (author's final draft

Modelling 3D humans : pose, shape, clothing and interactions

Digital humans are increasingly becoming a part of our lives with applications like animation, gaming, virtual try-on, Metaverse and much more. In recent years there has been a great push to make our models of digital humans as real as possible. In this thesis we present methodologies to model two key characteristics of real humans, their appearance and actions. This thesis covers four innovations: (i) MGN, the first approach to reconstruct 3D garments and body shape underneath, as separate meshes, from a few RGB images of a person. This allows, for the first time, real world applications like texture transfer, garment transfer and virtual try-on in 3D, using just images. (ii) IPNet, a neural network, that leverages implicit functions for detailed reconstruction and registers the reconstructed mesh with the parametric SMPL model to make it controllable for real world tasks like animation and editing. (iii) LoopReg, a novel formulation that makes 3D registration task end-to-end differentiable for the first time. Semi-supervised LoopReg outperforms contemporary supervised methods using ∼100x less supervised data. (iv) BEHAVE the first dataset and method to track full body real interactions between humans and movable objects. All our code, MGN digital wardrobe and BEHAVE dataset are publicly available for further research.Digital humans are increasingly becoming a part of our lives with applications like animation, gaming, virtual try-on, Metaverse and much more. In recent years there has been a great push to make our models of digital humans as real as possible. In this thesis we present methodologies to model two key characteristics of real humans, their appearance and actions. This thesis covers four innovations: (i) MGN, the first approach to reconstruct 3D garments and body shape underneath, as separate meshes, from a few RGB images of a person. This allows, for the first time, real world applications like texture transfer, garment transfer and virtual try-on in 3D, using just images. (ii) IPNet, a neural network, that leverages implicit functions for detailed reconstruction and registers the reconstructed mesh with the parametric SMPL model to make it controllable for real world tasks like animation and editing. (iii) LoopReg, a novel formulation that makes 3D registration task end-to-end differentiable for the first time. Semi-supervised LoopReg outperforms contemporary supervised methods using ∼100x less supervised data. (iv) BEHAVE the first dataset and method to track full body real interactions between humans and movable objects. All our code, MGN digital wardrobe and BEHAVE dataset are publicly available for further research.Der digitale Mensch wird immer mehr zu einem Teil unseres Lebens mit Anwendungen wie Animation, Spielen, virtuellem Ausprobieren, Metaverse und vielem mehr. In den letzten Jahren wurden große Anstrengungen unternommen, um unsere Modelle digitaler Menschen so real wie möglich zu gestalten. In dieser Arbeit stellen wir Methoden zur Modellierung von zwei Schlüsseleigenschaften echter Menschen vor: ihr Aussehen und ihre Handlungen. Wir schlagen MGN vor, den ersten Ansatz zur Rekonstruktion von 3D-Kleidungsstücken und der darunter liegenden Körperform als separate Netze aus einigen wenigen RGB-Bildern einer Person. Wir erweitern das weit verbreitete SMPL-Körpermodell, das nur unbekleidete Formen darstellt, um auch Kleidungsstücke zu erfassen (SMPL+G). SMPL+G kann mit Kleidungsstücken bekleidet werden, die entsprechend dem SMPL-Modell posiert und geformt werden können. Dies ermöglicht zum ersten Mal reale Anwendungen wie Texturübertragung, Kleidungsübertragung und virtuelle Anprobe in 3D, wobei nur Bilder verwendet werden. Wir unterstreichen auch die entscheidende Einschränkung der netzbasierten Darstellung für digitale Menschen, nämlich die Fähigkeit, hochfrequente Details darzustellen. Daher untersuchen wir die neue implizite funktionsbasierte Darstellung als Alternative zur netzbasierten Darstellung (einschließlich parametrischer Modelle wie SMPL) für digitale Menschen. Typischerweise mangelt es den Methoden, die auf letzteren basieren, an Details, während ersteren die Kontrolle fehlt. Wir schlagen IPNet vor, ein neuronales Netzwerk, das implizite Funktionen für eine detaillierte Rekonstruktion nutzt und das rekonstruierte Netz mit dem parametrischen SMPL-Modell registriert, um es kontrollierbar zu machen. Auf diese Weise wird das Beste aus beiden Welten genutzt. Wir untersuchen den Prozess der Registrierung eines parametrischen Modells, wie z. B. SMPL, auf ein 3D-Netz. Dieses jahrzehntealte Problem im Bereich der Computer Vision und der Graphik erfordert in der Regel einen zweistufigen Prozess: i) Herstellung von Korrespondenzen zwischen dem Modell und dem Netz, und ii) Optimierung des Modells, um den Abstand zwischen den entsprechenden Punkten zu minimieren. Dieser zweistufige Prozess ist nicht durchgängig differenzierbar. Wir schlagen LoopReg vor, das eine neue, auf impliziten Funktionen basierende Darstellung des Modells verwendet und die Registrierung differenzierbar macht. Semi-überwachtes LoopReg übertrifft aktuelle überwachte Methoden mit ∼100x weniger überwachten Daten. Die Modellierung des menschlichen Aussehens ist notwendig, aber nicht ausreichend, um realistische digitale Menschen zu schaffen. Wir müssen nicht nur modellieren, wie Menschen aussehen, sondern auch, wie sie mit ihren umgebenden Objekten interagieren. Zu diesem Zweck präsentieren wir mit BEHAVE den ersten Datensatz von realen Ganzkörper-Interaktionen zwischen Menschen und beweglichen Objekten. Wir stellen segmentierte Multiview-RGBDFrames zusammen mit registrierten SMPL- und Objekt-Fits sowie Kontaktannotationen in 3D zur Verfügung. Der BEHAVE-Datensatz enthält ∼15k Frames und seine Erweiterung enthält ∼400k Frames mit Pseudo-Ground-Truth-Annotationen. Unsere BEHAVE-Methode verwendet diesen Datensatz, um ein neuronales Netz zu trainieren, das die Person, das Objekt und die Kontakte zwischen ihnen gemeinsam verfolgt. In dieser Arbeit untersuchen wir die oben genannten Ideen und bieten eine eingehende Analyse unserer Schlüsselideen und Designentscheidungen. Wir erörtern auch die Grenzen unserer Ideen und schlagen künftige Arbeiten vor, um nicht nur diese Grenzen anzugehen, sondern auch die Forschung weiter auszubauen. Unser gesamter Code, die digitale Garderobe und der Datensatz sind für weitere Forschungen öffentlich zugänglich

Posture optimization algorithm for large structure assemblies based on skin model

Geometric deviations inevitably occur in product manufacturing and seriously affect the assembly quality and product functionality. Assembly simulations on the basis of computer-aided design (CAD) package could imitate the assembly process and thus find out the design deficiencies and detect the assemblability of the components. Although lots of researches have been done on the prediction of assembly variation considering the geometric errors, most of them only simplify the geometric variation as orientation and position deviation rather than the manufacturing deformation. However, in machinery manufacturing, even if the manufacturing defects are limited, they could propagate and accumulate through components and lead to a noncompliant assembly. Recently, many point-based models have been applied to assembly simulation; however they are mainly interested in simulating the resulting positions of the assembled parts and lack the consideration of the postprocessing after positioning. This paper enriches the complete assembly simulation process based on skin model and presents a simple and effective posture evaluation and optimization method. The studied approach includes a software algorithm applied to evaluate the contact state of the assembly parts and a mathematical model based on the particle swarm optimization to acquire the optimal assembly posture. To verify the efficiency and feasibility of the proposed method, a case study on the aircraft wing box scaling model assembly is performed

A Surface Reconstruction Method for In-Detail Underwater 3D Optical Mapping

International audienceUnderwater range scanning techniques are starting to gain interest in underwater exploration, providing new tools to represent the seafloor. These scans (often) acquired by underwater robots usually result in an unstructured point cloud, but given the common downward-looking or forward-looking configuration of these sensors with respect to the scene, the problem of recovering a piecewise linear approximation representing the scene is normally solved by approximating these 3D points using a heightmap (2.5D). Nevertheless, this representation is not able to correctly represent complex structures, especially those presenting arbitrary concavities normally exhibited in underwater objects. We present a method devoted to full 3D surface reconstruction that does not assume any specific sensor configuration. The method presented is robust to common defects in raw scanned data such as outliers and noise often present in extreme environments such as underwater, both for sonar and optical surveys. Moreover, the proposed method does not need a manual preprocessing step. It is also generic as it does not need any information other than the points themselves to work. This property leads to its wide application to any kind of range scanning technologies and we demonstrate its versatility by using it on synthetic data, controlled laser-scans, and multibeam sonar surveys. Finally, and given the unbeatable level of detail that optical methods can provide, we analyze the application of this method on optical datasets related to biology, geology and archeology

Noise-Adaptive Shape Reconstruction from Raw Point Sets

International audienceWe propose a noise-adaptive shape reconstruction method specialized to smooth, closed shapes. Our algorithm takes as input a defect-laden point set with variable noise and outliers, and comprises three main steps. First, we compute a novel noise-adaptive distance function to the inferred shape, which relies on the assumption that the inferred shape is a smooth submanifold of known dimension. Second, we estimate the sign and confidence of the function at a set of seed points, through minimizing a quadratic energy expressed on the edges of a uniform random graph. Third, we compute a signed implicit function through a random walker approach with soft constraints chosen as the most confident seed points computed in previous step

Acquisition of Surface Light Fields from Videos

La tesi presenta un nuovo approccio per la stima di Surface Light Field di oggetti reali, a partire da sequenze video acquisite in condizioni di illuminazione fisse e non controllate. Il metodo proposto si basa sulla separazione delle due componenti principali dell'apparenza superficiale dell'oggetto: la componente diffusiva, modellata come colore RGB, e la componente speculare, approssimata mediante un modello parametrico funzione della posizione dell'osservatore. L'apparenza superficiale ricostruita permette una visualizzazione fotorealistica e in real-time dell'oggetto al variare della posizione dell'osservatore, consentendo una navigazione 3D interattiva

Sparse Non-rigid Registration of 3D Shapes

Non-rigid registration of 3D shapes is an essential task of increasing importance as commodity depth sensors become more widely available for scanning dynamic scenes. Non-rigid registration is much more challenging than rigid registration as it estimates a set of local transformations instead of a single global transformation, and hence is prone to the overfitting issue due to underdetermination. The common wisdom in previous methods is to impose an ℓ2-norm regularization on the local transformation differences. However, the ℓ2-norm regularization tends to bias the solution towards outliers and noise with heavy-tailed distribution, which is verified by the poor goodness-of-fit of the Gaussian distribution over transformation differences. On the contrary, Laplacian distribution fits well with the transformation differences, suggesting the use of a sparsity prior. We propose a sparse non-rigid registration (SNR) method with an ℓ1-norm regularized model for transformation estimation, which is effectively solved by an alternate direction method (ADM) under the augmented Lagrangian framework. We also devise a multi-resolution scheme for robust and progressive registration. Results on both public datasets and our scanned datasets show the superiority of our method, particularly in handling large-scale deformations as well as outliers and noise