Mapping, Localization and Path Planning for Image-based Navigation using Visual Features and Map

Building on progress in feature representations for image retrieval, image-based localization has seen a surge of research interest. Image-based localization has the advantage of being inexpensive and efficient, often avoiding the use of 3D metric maps altogether. That said, the need to maintain a large number of reference images as an effective support of localization in a scene, nonetheless calls for them to be organized in a map structure of some kind. The problem of localization often arises as part of a navigation process. We are, therefore, interested in summarizing the reference images as a set of landmarks, which meet the requirements for image-based navigation. A contribution of this paper is to formulate such a set of requirements for the two sub-tasks involved: map construction and self-localization. These requirements are then exploited for compact map representation and accurate self-localization, using the framework of a network flow problem. During this process, we formulate the map construction and self-localization problems as convex quadratic and second-order cone programs, respectively. We evaluate our methods on publicly available indoor and outdoor datasets, where they outperform existing methods significantly.Comment: CVPR 2019, for implementation see https://github.com/janinethom

Deformable Neural Radiance Fields using RGB and Event Cameras

Modeling Neural Radiance Fields for fast-moving deformable objects from visual data alone is a challenging problem. A major issue arises due to the high deformation and low acquisition rates. To address this problem, we propose to use event cameras that offer very fast acquisition of visual change in an asynchronous manner. In this work, we develop a novel method to model the deformable neural radiance fields using RGB and event cameras. The proposed method uses the asynchronous stream of events and calibrated sparse RGB frames. In our setup, the camera pose at the individual events required to integrate them into the radiance fields remains unknown. Our method jointly optimizes these poses and the radiance field. This happens efficiently by leveraging the collection of events at once and actively sampling the events during learning. Experiments conducted on both realistically rendered graphics and real-world datasets demonstrate a significant benefit of the proposed method over the state-of-the-art and the compared baseline. This shows a promising direction for modeling deformable neural radiance fields in real-world dynamic scenes

Unsupervised Monocular Depth Reconstruction of Non-Rigid Scenes

Monocular depth reconstruction of complex and dynamic scenes is a highly challenging problem. While for rigid scenes learning-based methods have been offering promising results even in unsupervised cases, there exists little to no literature addressing the same for dynamic and deformable scenes. In this work, we present an unsupervised monocular framework for dense depth estimation of dynamic scenes, which jointly reconstructs rigid and non-rigid parts without explicitly modelling the camera motion. Using dense correspondences, we derive a training objective that aims to opportunistically preserve pairwise distances between reconstructed 3D points. In this process, the dense depth map is learned implicitly using the as-rigid-as-possible hypothesis. Our method provides promising results, demonstrating its capability of reconstructing 3D from challenging videos of non-rigid scenes. Furthermore, the proposed method also provides unsupervised motion segmentation results as an auxiliary output

Méthodes Analytiques Locales et Méthodes Globales Convexes pour la Reconstruction 3D de Surfaces Isométriquement Déformables

This thesis contributes to the problem of 3D reconstruction for deformable surfaces using a singlecamera. In order to model surface deformation, we use the isometric prior because many real objectdeformations are near-isometric. Isometry implies that the surface cannot stretch or compress. Wetackle two different problems.The first is called Shape-from-Template where the object’s deformed shape is computed from asingle image and a texture-mapped 3D template of the object surface. Previous methods propose adifferential model of the problem and compute the local analytic solutions. In the methods the solutionrelated to the depth-gradient is discarded and only the depth solution is used. We demonstrate thatthe depth solution lacks stability as the projection geometry tends to affine. We provide alternativemethods based on the local analytic solutions of first-order quantities, such as the depth-gradient orsurface normals. Our methods are stable in all projection geometries.The second type of problem, called Non-Rigid Shape-from-Motion is the more general templatefreereconstruction scenario. In this case one obtains the object’s shapes from a set of images whereit appears deformed. We contribute to this problem for both local and global solutions using the perspectivecamera. In the local or point-wise method, we solve for the surface normal at each pointassuming infinitesimal planarity of the surface. We then compute the surface by integration. In theglobal method we find a convex relaxation of the problem. This is based on relaxing isometry to inextensibilityand maximizing the surface’s average depth. This solution combines all constraints intoa single convex optimization program to compute depth and works for a sparse point representationof the surface.We detail the extensive experiments that were used to demonstrate the effectiveness of each ofthe proposed methods. The experiments show that our local template-free solution performs betterthan most of the previous methods. Our local template-based method and our global template-freemethod performs better than the state-of-the-art methods with robustness to correspondence noise.In particular, we are able to reconstruct difficult, non-smooth and articulating deformations with thelatter; while with the former we can accurately reconstruct large deformations with images taken atvery long focal lengths.Cette thèse contribue au problème de la reconstruction 3D pour les surfaces déformables avec uneseule caméra. Afin de modéliser la déformation de la surface, nous considérons l’isométrie puisquede nombreuses déformations d’objets réels sont quasi-isométriques. L’isométrie implique que, lorsde sa déformation, la surface ne peut pas être étirée ou compressée. Nous étudions deux problèmes.Le premier est le problème basé sur une modèle 3D de référence et une seule image. L’étatde l’art propose une méthode locale et analytique de calcul direct de profondeur sous l’hypothèsed’isométrie. Dans cette méthode, la solution pour le gradient de la profondeur n’est pas utilisée.Nous prouvons que cette méthode s’avère instable lorsque la géométrie de la caméra tend à êtreaffine. Nous fournissons des méthodes alternatives basées sur les solutions analytiques locales desquantités de premier ordre, telles que les gradients de profondeur ou les normales de la surface. Nosméthodes sont stables dans toutes les géométries de projection.Dans le deuxième type de problème de reconstruction sans modèle 3D de référence, on obtientles formes de l’objet à partir d’un ensemble d’images où il apparaît déformé. Nous fournissons dessolutions locales et globales basées sur le modéle de la caméra perspective. Dans la méthode locale oupar point, nous résolvons pour la normale de la surface en chaque point en supposant que la surfaceest infinitésimalement plane. Nous calculons ensuite la surface par intégration. Dans la méthodeglobale, nous trouvons une relaxation convexe du problème. Celle-ci est basée sur la relaxation del’isométrie en constrainte d’inextensibilité et sur la maximisation de la profondeur en chaque pointde la surface. Cette solution combine toutes les contraintes en un seul programme d’optimisationconvexe qui calcule la profondeur et utilise une représentation éparse de la surface.Nous détaillons les expériences approfondies qui ont été réalisées pour démontrer l’efficacitéde chacune des méthodes. Les expériences montrent que notre solution libre de modèle de réferencelocal fonctionne mieux que la plupart des méthodes précédentes. Notre méthode local avec un modèle3D de réference et notre méthode globale sans modèle 3D apportent de meilleurs résultats que lesméthodes de l’état de l’art en etant robuste au bruit de la correspondance. En particulier, nous sommesen mesure de reconstruire des déformations complexes, non-lisses et d’articulations avec la secondeméthode; alors qu’avec la première, nous pouvons reconstruire avec précision de déformations largesà partir d’images prises avec des très longues focales