SPARF: Neural Radiance Fields from Sparse and Noisy Poses

Neural Radiance Field (NeRF) has recently emerged as a powerful representation to synthesize photorealistic novel views. While showing impressive performance, it relies on the availability of dense input views with highly accurate camera poses, thus limiting its application in real-world scenarios. In this work, we introduce Sparse Pose Adjusting Radiance Field (SPARF), to address the challenge of novel-view synthesis given only few wide-baseline input images (as low as 3) with noisy camera poses. Our approach exploits multi-view geometry constraints in order to jointly learn the NeRF and refine the camera poses. By relying on pixel matches extracted between the input views, our multi-view correspondence objective enforces the optimized scene and camera poses to converge to a global and geometrically accurate solution. Our depth consistency loss further encourages the reconstructed scene to be consistent from any viewpoint. Our approach sets a new state of the art in the sparse-view regime on multiple challenging datasets.Comment: Code will be released upon publicatio

Shape, Pose, and Appearance from a Single Image via Bootstrapped Radiance Field Inversion

Neural Radiance Fields (NeRF) coupled with GANs represent a promising direction in the area of 3D reconstruction from a single view, owing to their ability to efficiently model arbitrary topologies. Recent work in this area, however, has mostly focused on synthetic datasets where exact ground-truth poses are known, and has overlooked pose estimation, which is important for certain downstream applications such as augmented reality (AR) and robotics. We introduce a principled end-to-end reconstruction framework for natural images, where accurate ground-truth poses are not available. Our approach recovers an SDF-parameterized 3D shape, pose, and appearance from a single image of an object, without exploiting multiple views during training. More specifically, we leverage an unconditional 3D-aware generator, to which we apply a hybrid inversion scheme where a model produces a first guess of the solution which is then refined via optimization. Our framework can de-render an image in as few as 10 steps, enabling its use in practical scenarios. We demonstrate state-of-the-art results on a variety of real and synthetic benchmarks

Learning Delaunay Surface Elements for Mesh Reconstruction

We present a method for reconstructing triangle meshes from point clouds. Existing learning-based methods for mesh reconstruction mostly generate triangles individually, making it hard to create manifold meshes. We leverage the properties of 2D Delaunay triangulations to construct a mesh from manifold surface elements. Our method first estimates local geodesic neighborhoods around each point. We then perform a 2D projection of these neighborhoods using a learned logarithmic map. A Delaunay triangulation in this 2D domain is guaranteed to produce a manifold patch, which we call a Delaunay surface element. We synchronize the local 2D projections of neighboring elements to maximize the manifoldness of the reconstructed mesh. Our results show that we achieve better overall manifoldness of our reconstructed meshes than current methods to reconstruct meshes with arbitrary topology

Differentiable Surface Triangulation

Triangle meshes remain the most popular data representation for surface geometry. This ubiquitous representation is essentially a hybrid one that decouples continuous vertex locations from the discrete topological triangulation. Unfortunately, the combinatorial nature of the triangulation prevents taking derivatives over the space of possible meshings of any given surface. As a result, to date, mesh processing and optimization techniques have been unable to truly take advantage of modular gradient descent components of modern optimization frameworks. In this work, we present a differentiable surface triangulation that enables optimization for any per-vertex or per-face differentiable objective function over the space of underlying surface triangulations. Our method builds on the result that any 2D triangulation can be achieved by a suitably perturbed weighted Delaunay triangulation. We translate this result into a computational algorithm by proposing a soft relaxation of the classical weighted Delaunay triangulation and optimizing over vertex weights and vertex locations. We extend the algorithm to 3D by decomposing shapes into developable sets and differentiably meshing each set with suitable boundary constraints. We demonstrate the efficacy of our method on various planar and surface meshes on a range of difficult-to-optimize objective functions. Our code can be found online: https://github.com/mrakotosaon/diff-surface-triangulation

PointCleanNet: Learning to Denoise and Remove Outliers from Dense Point Clouds

Point clouds obtained with 3D scanners or by image-based reconstruction techniques are often corrupted with significant amount of noise and outliers. Traditional methods for point cloud denoising largely rely on local surface fitting (e.g., jets or MLS surfaces), local or non-local averaging, or on statistical assumptions about the underlying noise model. In contrast, we develop a simple data-driven method for removing outliers and reducing noise in unordered point clouds. We base our approach on a deep learning architecture adapted from PCPNet, which was recently proposed for estimating local 3D shape properties in point clouds. Our method first classifies and discards outlier samples, and then estimates correction vectors that project noisy points onto the original clean surfaces. The approach is efficient and robust to varying amounts of noise and outliers, while being able to handle large densely-sampled point clouds. In our extensive evaluation, both on synthesic and real data, we show an increased robustness to strong noise levels compared to various state-of-the-art methods, enabling accurate surface reconstruction from extremely noisy real data obtained by range scans. Finally, the simplicity and universality of our approach makes it very easy to integrate in any existing geometry processing pipeline

Représentations et méthodes basées sur l’apprentissage pour l’analyse, la manipulation et la reconstruction de formes en 3D

Traiter et analyser efficacement les données 3D est un défi crucial dans les applications modernes, car les formes 3D sont de plus en plus répandues avec la prolifération des dispositifs d'acquisition et des outils de modélisation. Alors que les succès de l'apprentissage profond en 2D sont devenus monnaie courante et entourent notre vie quotidienne, les applications qui impliquent des données 3D sont à la traîne. En raison de la structure non uniforme plus complexe des formes 3D, les méthodes d'apprentissage profond en 2D ne peuvent pas être facilement étendues et il existe une forte demande pour de nouvelles approches qui peuvent à la fois exploiter et permettre l'apprentissage en utilisant la structure géométrique. De plus, être capable de gérer les différentes représentations existantes des formes 3D telles que les nuages de points et les maillages, ainsi que les artefacts produits par les dispositifs d'acquisition 3D augmente la difficulté de la tâche. Dans cette thèse, nous proposons des approches systématiques qui exploitent pleinement les informations géométriques des données 3D dans des architectures d'apprentissage profond. Nous contribuons aux méthodes de débruitage de nuages de points, d'interpolation de formes et de reconstruction de formes. Nous observons que les architectures d'apprentissage profond facilitent l'apprentissage de la structure de surface sous-jacente des nuages de points, qui peut ensuite être utilisée pour le débruitage et l'interpolation de formes. L'encodage de prieurs appris basés sur des patchs locaux, ainsi que d'informations géométriques complémentaires telles que la longueur des arrêtes, permet de créer des pipelines puissants qui génèrent des formes réalistes. Le principal fil conducteur de nos contributions est de faciliter la conversion entre différentes représentations de formes. En particulier, alors que l'utilisation de l'apprentissage profond sur des mailles triangulaires est complexe en raison de leur nature combinatoire, nous introduisons des méthodes inspirées du traitement de la géométrie qui permettent la création et la manipulation de faces de triangles. Nos méthodes sont robustes et se généralisent bien aux données inconnues malgré des jeux d'entraînement limités. Notre travail ouvre donc la voie à une manipulation plus générale, robuste et universellement utile des données 3D.Efficiently processing and analysing 3D data is a crucial challenge in modern applications as 3D shapes are becoming more and more widespread with the proliferation of acquisition devices and modeling tools. While successes of 2D deep learning have become commonplace and surround our daily life, applications that involve 3D data are lagging behind. Due to the more complex non-uniform structure of 3D shapes, successful methods from 2D deep learning cannot be easily extended and there is a strong demand for novel approaches that can both exploit and enable learning using geometric structure. Moreover, being able to handle the various existing representations of 3D shapes such as point clouds and meshes, as well as the artefacts produced from 3D acquisition devices increases the difficulty of the task. In this thesis, we propose systematic approaches that fully exploit geometric information of 3D data in deep learning architectures. We contribute to point cloud denoising, shape interpolation and shape reconstruction methods. We observe that deep learning architectures facilitate learning the underlying surface structure on point clouds that can then be used for denoising as well as shape interpolation. Encoding local patch-based learned priors, as well as complementary geometric information such as edge lengths, leads to powerful pipelines that generate realistic shapes. The key common thread throughout our contributions is facilitating seamless conversion between different representations of shapes. In particular, while using deep learning on triangle meshes is highly challenging due to their combinatorial nature we introduce methods inspired from geometry processing that enable the creation and manipulation of triangle faces. Our methods are robust and generalize well to unseen data despite limited training sets. Our work, therefore, paves the way towards more general, robust and universally useful manipulation of 3D data

Représentations et méthodes basées sur l’apprentissage pour l’analyse, la manipulation et la reconstruction de formes en 3D

Efficiently processing and analysing 3D data is a crucial challenge in modern applications as 3D shapes are becoming more and more widespread with the proliferation of acquisition devices and modeling tools. While successes of 2D deep learning have become commonplace and surround our daily life, applications that involve 3D data are lagging behind. Due to the more complex non-uniform structure of 3D shapes, successful methods from 2D deep learning cannot be easily extended and there is a strong demand for novel approaches that can both exploit and enable learning using geometric structure. Moreover, being able to handle the various existing representations of 3D shapes such as point clouds and meshes, as well as the artefacts produced from 3D acquisition devices increases the difficulty of the task. In this thesis, we propose systematic approaches that fully exploit geometric information of 3D data in deep learning architectures. We contribute to point cloud denoising, shape interpolation and shape reconstruction methods. We observe that deep learning architectures facilitate learning the underlying surface structure on point clouds that can then be used for denoising as well as shape interpolation. Encoding local patch-based learned priors, as well as complementary geometric information such as edge lengths, leads to powerful pipelines that generate realistic shapes. The key common thread throughout our contributions is facilitating seamless conversion between different representations of shapes. In particular, while using deep learning on triangle meshes is highly challenging due to their combinatorial nature we introduce methods inspired from geometry processing that enable the creation and manipulation of triangle faces. Our methods are robust and generalize well to unseen data despite limited training sets. Our work, therefore, paves the way towards more general, robust and universally useful manipulation of 3D data.Traiter et analyser efficacement les données 3D est un défi crucial dans les applications modernes, car les formes 3D sont de plus en plus répandues avec la prolifération des dispositifs d'acquisition et des outils de modélisation. Alors que les succès de l'apprentissage profond en 2D sont devenus monnaie courante et entourent notre vie quotidienne, les applications qui impliquent des données 3D sont à la traîne. En raison de la structure non uniforme plus complexe des formes 3D, les méthodes d'apprentissage profond en 2D ne peuvent pas être facilement étendues et il existe une forte demande pour de nouvelles approches qui peuvent à la fois exploiter et permettre l'apprentissage en utilisant la structure géométrique. De plus, être capable de gérer les différentes représentations existantes des formes 3D telles que les nuages de points et les maillages, ainsi que les artefacts produits par les dispositifs d'acquisition 3D augmente la difficulté de la tâche. Dans cette thèse, nous proposons des approches systématiques qui exploitent pleinement les informations géométriques des données 3D dans des architectures d'apprentissage profond. Nous contribuons aux méthodes de débruitage de nuages de points, d'interpolation de formes et de reconstruction de formes. Nous observons que les architectures d'apprentissage profond facilitent l'apprentissage de la structure de surface sous-jacente des nuages de points, qui peut ensuite être utilisée pour le débruitage et l'interpolation de formes. L'encodage de prieurs appris basés sur des patchs locaux, ainsi que d'informations géométriques complémentaires telles que la longueur des arrêtes, permet de créer des pipelines puissants qui génèrent des formes réalistes. Le principal fil conducteur de nos contributions est de faciliter la conversion entre différentes représentations de formes. En particulier, alors que l'utilisation de l'apprentissage profond sur des mailles triangulaires est complexe en raison de leur nature combinatoire, nous introduisons des méthodes inspirées du traitement de la géométrie qui permettent la création et la manipulation de faces de triangles. Nos méthodes sont robustes et se généralisent bien aux données inconnues malgré des jeux d'entraînement limités. Notre travail ouvre donc la voie à une manipulation plus générale, robuste et universellement utile des données 3D

Représentations et méthodes basées sur l’apprentissage pour l’analyse, la manipulation et la reconstruction de formes en 3D

Efficiently processing and analysing 3D data is a crucial challenge in modern applications as 3D shapes are becoming more and more widespread with the proliferation of acquisition devices and modeling tools. While successes of 2D deep learning have become commonplace and surround our daily life, applications that involve 3D data are lagging behind. Due to the more complex non-uniform structure of 3D shapes, successful methods from 2D deep learning cannot be easily extended and there is a strong demand for novel approaches that can both exploit and enable learning using geometric structure. Moreover, being able to handle the various existing representations of 3D shapes such as point clouds and meshes, as well as the artefacts produced from 3D acquisition devices increases the difficulty of the task. In this thesis, we propose systematic approaches that fully exploit geometric information of 3D data in deep learning architectures. We contribute to point cloud denoising, shape interpolation and shape reconstruction methods. We observe that deep learning architectures facilitate learning the underlying surface structure on point clouds that can then be used for denoising as well as shape interpolation. Encoding local patch-based learned priors, as well as complementary geometric information such as edge lengths, leads to powerful pipelines that generate realistic shapes. The key common thread throughout our contributions is facilitating seamless conversion between different representations of shapes. In particular, while using deep learning on triangle meshes is highly challenging due to their combinatorial nature we introduce methods inspired from geometry processing that enable the creation and manipulation of triangle faces. Our methods are robust and generalize well to unseen data despite limited training sets. Our work, therefore, paves the way towards more general, robust and universally useful manipulation of 3D data.Traiter et analyser efficacement les données 3D est un défi crucial dans les applications modernes, car les formes 3D sont de plus en plus répandues avec la prolifération des dispositifs d'acquisition et des outils de modélisation. Alors que les succès de l'apprentissage profond en 2D sont devenus monnaie courante et entourent notre vie quotidienne, les applications qui impliquent des données 3D sont à la traîne. En raison de la structure non uniforme plus complexe des formes 3D, les méthodes d'apprentissage profond en 2D ne peuvent pas être facilement étendues et il existe une forte demande pour de nouvelles approches qui peuvent à la fois exploiter et permettre l'apprentissage en utilisant la structure géométrique. De plus, être capable de gérer les différentes représentations existantes des formes 3D telles que les nuages de points et les maillages, ainsi que les artefacts produits par les dispositifs d'acquisition 3D augmente la difficulté de la tâche. Dans cette thèse, nous proposons des approches systématiques qui exploitent pleinement les informations géométriques des données 3D dans des architectures d'apprentissage profond. Nous contribuons aux méthodes de débruitage de nuages de points, d'interpolation de formes et de reconstruction de formes. Nous observons que les architectures d'apprentissage profond facilitent l'apprentissage de la structure de surface sous-jacente des nuages de points, qui peut ensuite être utilisée pour le débruitage et l'interpolation de formes. L'encodage de prieurs appris basés sur des patchs locaux, ainsi que d'informations géométriques complémentaires telles que la longueur des arrêtes, permet de créer des pipelines puissants qui génèrent des formes réalistes. Le principal fil conducteur de nos contributions est de faciliter la conversion entre différentes représentations de formes. En particulier, alors que l'utilisation de l'apprentissage profond sur des mailles triangulaires est complexe en raison de leur nature combinatoire, nous introduisons des méthodes inspirées du traitement de la géométrie qui permettent la création et la manipulation de faces de triangles. Nos méthodes sont robustes et se généralisent bien aux données inconnues malgré des jeux d'entraînement limités. Notre travail ouvre donc la voie à une manipulation plus générale, robuste et universellement utile des données 3D

Correspondence Learning via Linearly-invariant Embedding

In this paper, we propose a fully differentiable pipeline for estimating accurate dense correspondences between 3D point clouds. The proposed pipeline is an extension and a generalization of the functional maps framework. However, instead of using the Laplace-Beltrami eigenfunctions as done in virtually all previous works in this domain, we demonstrate that learning the basis from data can both improve robustness and lead to better accuracy in challenging settings. We interpret the basis as a learned embedding into a higher dimensional space. Following the functional map paradigm the optimal transformation in this embedding space must be linear and we propose a separate architecture aimed at estimating the transformation by learning optimal descriptor functions. This leads to the first end-to-end trainable functional map-based correspondence approach in which both the basis and the descriptors are learned from data. Interestingly, we also observe that learning a canonical embedding leads to worse results, suggesting that leaving an extra linear degree of freedom to the embedding network gives it more robustness, thereby also shedding light onto the success of previous methods. Finally, we demonstrate that our approach achieves state-of-the-art results in challenging non-rigid 3D point cloud correspondence applications