LivePose: Online 3D Reconstruction from Monocular Video with Dynamic Camera Poses

Dense 3D reconstruction from RGB images traditionally assumes static camera pose estimates. This assumption has endured, even as recent works have increasingly focused on real-time methods for mobile devices. However, the assumption of a fixed pose for each image does not hold for online execution: poses from real-time SLAM are dynamic and may be updated following events such as bundle adjustment and loop closure. This has been addressed in the RGB-D setting, by de-integrating past views and re-integrating them with updated poses, but it remains largely untreated in the RGB-only setting. We formalize this problem to define the new task of dense online reconstruction from dynamically-posed images. To support further research, we introduce a dataset called LivePose containing the dynamic poses from a SLAM system running on ScanNet. We select three recent reconstruction systems and apply a framework based on de-integration to adapt each one to the dynamic-pose setting. In addition, we propose a novel, non-linear de-integration module that learns to remove stale scene content. We show that responding to pose updates is critical for high-quality reconstruction, and that our de-integration framework is an effective solution.Comment: ICCV 202

FineRecon: Depth-aware Feed-forward Network for Detailed 3D Reconstruction

Recent works on 3D reconstruction from posed images have demonstrated that direct inference of scene-level 3D geometry without test-time optimization is feasible using deep neural networks, showing remarkable promise and high efficiency. However, the reconstructed geometry, typically represented as a 3D truncated signed distance function (TSDF), is often coarse without fine geometric details. To address this problem, we propose three effective solutions for improving the fidelity of inference-based 3D reconstructions. We first present a resolution-agnostic TSDF supervision strategy to provide the network with a more accurate learning signal during training, avoiding the pitfalls of TSDF interpolation seen in previous work. We then introduce a depth guidance strategy using multi-view depth estimates to enhance the scene representation and recover more accurate surfaces. Finally, we develop a novel architecture for the final layers of the network, conditioning the output TSDF prediction on high-resolution image features in addition to coarse voxel features, enabling sharper reconstruction of fine details. Our method, FineRecon, produces smooth and highly accurate reconstructions, showing significant improvements across multiple depth and 3D reconstruction metrics.Comment: ICCV 202

Modélisation géométrique par primitives

Both man-made or natural objects contain repeated geometric elements that can be interpreted as primitive shapes. Plants, trees, living organisms or even crystals, showcase primitives that repeat themselves. Primitives are also commonly found in man-made environments because architects tend to reuse the same patterns over a building and typically employ simple shapes, such as rectangular windows and doors. During my PhD I studied geometric primitives from three points of view: their composition, simulation and autonomous discovery. In the first part I present a method to reverse-engineer the function by which some primitives are combined. Our system is based on a composition function template that is represented by a parametric surface. The parametric surface is deformed via a non-rigid alignment of a surface that, once converged, represents the desired operator. This enables the interactive modeling of operators via a simple sketch, solving a major usability gap of composition modeling. In the second part I introduce the use of a novel primitive for real-time physics simulations. This primitive is suitable to efficiently model volume-preserving deformations of rods but also of more complex structures such as muscles. One of the core advantages of our approach is that our primitive can serve as a unified representation to do collision detection, simulation, and surface skinning. In the third part I present an unsupervised deep learning framework to learn and detect primitives. In a signal containing a repetition of elements, the method is able to automatically identify the structure of these elements (i.e. primitives) with minimal supervision. In order to train the network that contains a non-differentiable operation, a novel multi-step training process is presented.Les objets naturels ou artificiels contiennent des éléments géométriques répétés pouvantêtre interprétés comme des formes primitives. Les plantes, les arbres, les organismes vivants ou même les cristaux représentent des primitives qui se répètent. Les primitives se retrouvent aussi couramment dans les environnements créés par l'homme, car les architectes ont tendance à réutiliser les mêmes modèles sur un bâtiment et utilisent généralement des formes simples, telles que des fenêtres et des portes rectangulaires. Au cours de ma thèse, j'ai étudié les primitives géométriques sous trois angles: leur composition, leur simulation et leur découverte autonome. Dans la première partie, je présente une méthode pour retrouver la fonction par laquelle certaines primitives sont combinées. Notre système est basé sur un modèle de fonction de composition représenté par une surface paramétrique. La surface paramétrique est déformée via un alignement non rigide d'une surface qui, une fois convergée, représente l'opérateur souhaité. Cela permet de modéliser de manière interactive les opérateurs via une simple esquisse, ce qui résout un problème majeur de facilité de modélisation de la composition. Dans la deuxième partie, je présente l'utilisation d'une nouvelle primitive pour les simulations de physique en temps réel. Cette primitive convient pour modéliser efficacement les déformations des cordes préservant le volume, mais également celles de structures plus complexes telles que les muscles. L'un des principaux avantages de notre approche est que notre primitive peut servir de représentation unifiée pour la détection de collision, la simulation et la déformation de peau. Dans la troisième partie, je présente un cadre d'apprentissage en profondeur non supervisé pour apprendre et détecter les primitives. Dans un signal contenant une répétition d'éléments, le procédé est capable d'identifier automatiquement la structure de ces éléments (c'est-à-dire des primitives) avec une supervision minimale. Afin de former le réseau qui contient une opération non différenciable, un nouveau processus d'apprentissage en plusieurs étapes est présenté

Geometric modeling with primitives

Les objets naturels ou artificiels contiennent des éléments géométriques répétés pouvantêtre interprétés comme des formes primitives. Les plantes, les arbres, les organismes vivants ou même les cristaux représentent des primitives qui se répètent. Les primitives se retrouvent aussi couramment dans les environnements créés par l'homme, car les architectes ont tendance à réutiliser les mêmes modèles sur un bâtiment et utilisent généralement des formes simples, telles que des fenêtres et des portes rectangulaires. Au cours de ma thèse, j'ai étudié les primitives géométriques sous trois angles: leur composition, leur simulation et leur découverte autonome. Dans la première partie, je présente une méthode pour retrouver la fonction par laquelle certaines primitives sont combinées. Notre système est basé sur un modèle de fonction de composition représenté par une surface paramétrique. La surface paramétrique est déformée via un alignement non rigide d'une surface qui, une fois convergée, représente l'opérateur souhaité. Cela permet de modéliser de manière interactive les opérateurs via une simple esquisse, ce qui résout un problème majeur de facilité de modélisation de la composition. Dans la deuxième partie, je présente l'utilisation d'une nouvelle primitive pour les simulations de physique en temps réel. Cette primitive convient pour modéliser efficacement les déformations des cordes préservant le volume, mais également celles de structures plus complexes telles que les muscles. L'un des principaux avantages de notre approche est que notre primitive peut servir de représentation unifiée pour la détection de collision, la simulation et la déformation de peau. Dans la troisième partie, je présente un cadre d'apprentissage en profondeur non supervisé pour apprendre et détecter les primitives. Dans un signal contenant une répétition d'éléments, le procédé est capable d'identifier automatiquement la structure de ces éléments (c'est-à-dire des primitives) avec une supervision minimale. Afin de former le réseau qui contient une opération non différenciable, un nouveau processus d'apprentissage en plusieurs étapes est présenté.Both man-made or natural objects contain repeated geometric elements that can be interpreted as primitive shapes. Plants, trees, living organisms or even crystals, showcase primitives that repeat themselves. Primitives are also commonly found in man-made environments because architects tend to reuse the same patterns over a building and typically employ simple shapes, such as rectangular windows and doors. During my PhD I studied geometric primitives from three points of view: their composition, simulation and autonomous discovery. In the first part I present a method to reverse-engineer the function by which some primitives are combined. Our system is based on a composition function template that is represented by a parametric surface. The parametric surface is deformed via a non-rigid alignment of a surface that, once converged, represents the desired operator. This enables the interactive modeling of operators via a simple sketch, solving a major usability gap of composition modeling. In the second part I introduce the use of a novel primitive for real-time physics simulations. This primitive is suitable to efficiently model volume-preserving deformations of rods but also of more complex structures such as muscles. One of the core advantages of our approach is that our primitive can serve as a unified representation to do collision detection, simulation, and surface skinning. In the third part I present an unsupervised deep learning framework to learn and detect primitives. In a signal containing a repetition of elements, the method is able to automatically identify the structure of these elements (i.e. primitives) with minimal supervision. In order to train the network that contains a non-differentiable operation, a novel multi-step training process is presented

Towards soundpainting gesture recognition

In this article, we describe our recent research activities on gesture recognition for soundpainting applications. Soundpainting is a multidisciplinary live composing sign language for musicians, actors, dancers, and visual artists. These gestures are produced by a soundpainter, which plays the role of a conductor, in order to lead a live performance. Soundpainting gestures are normalized and well defined, thus they are a very interesting case study in automatic gesture recognition. We describe a first gesture recognition system based on hidden Markov Models. We also report on the creation of a pilot corpus of soundpainting RGB/depth videos. The use of a computer could have many interesting applications listed in the paper. These applications are not limited to live performance, in which the computer would act as a performer. It could also help to investigate the balance between improvisation and planned creation in the particular context of soundpainting

Sketch-Based Implicit Blending

International audienceImplicit models can be combined by using composition operators; functions that determine the resulting shape. Recently, gradient-based composition operators have been used to express a variety of behaviours including smooth transitions, sharp edges, contact surfaces, bulging, or any combinations. The problem for designers is that building new operators is a complex task that requires specialized technical knowledge. In this work, we introduce an automatic method for deriving a gradient-based implicit operator from 2D drawings that prototype the intended visual behaviour. To solve this inverse problem, in which a shape defines a function, we introduce a general template for implicit operators. A user’s sketch is interpreted as samples in the 3D operator’s domain. We fit the template to the samples with a non-rigid registration approach. The process works at interactive rates and can accommodate successive refinements by the user. The final result can be applied to 3D surfaces as well as to 2D shapes. Our method is able to replicate the effect of any blending operator presented in the literature, as well as generating new ones such as non-commutative operators. We demonstrate the usability of our method with examples in font-design, collision-response modeling, implicit skinning, and complex shape design

VIPER: Volume Invariant Position-based Elastic Rods

International audienceWe extend the formulation of position-based rods to include elastic volumetric deformations. We achieve this by introducing an additional degree of freedom per vertex-isotropic scale (and its velocity). Including scale enriches the space of possible deformations, allowing the simulation of volumetric effects, such as a reduction in cross-sectional area when a rod is stretched. We rigorously derive the continuous formulation of its elastic energy potentials, and hence its associated position-based dynamics (PBD) updates to realize this model, enabling the simulation of up to 26000 DOFs at 140 Hz in our GPU implementation. We further show how rods can provide a compact alternative to tetrahedral meshes for the representation of complex muscle deformations, as well as providing a convenient representation for collision detection. This is achieved by modeling a muscle as a bundle of rods, for which we also introduce a technique to automatically convert a muscle surface mesh into a rods-bundle. Finally, we show how rods and/or bundles can be skinned to a surface mesh to drive its deformation, resulting in an alternative to cages for real-time volumetric deformation. The source code of our physics engine will be openly available