Fast Image and LiDAR alignment based on 3D rendering in sensor topology

Mobile Mapping Systems are now commonly used in large urban acquisition campaigns. They are often equiped with LiDAR sensors and optical cameras, providing very large multimodal datasets. The fusion of both modalities serves different purposes such as point cloud colorization, geometry enhancement or object detection. However, this fusion task cannot be done directly as both modalities are only coarsely registered. This paper presents a fully automatic approach for LiDAR projection and optical image registration refinement based on LiDAR point cloud 3D renderings. First, a coarse 3D mesh is generated from the LiDAR point cloud using the sensor topology. Then, the mesh is rendered in the image domain. After that, a variational approach is used to align the rendering with the optical image. This method achieves high quality results while performing in very low computational time. Results on real data demonstrate the efficiency of the model for aligning LiDAR projections and optical images

2D Image Processing Applied to 3D LiDAR Point Clouds

L'intérêt toujours grandissant pour les données cartographiques fiables, notamment en milieu urbain, a motivé le développement de systèmes de cartographie mobiles terrestres. Ces systèmes sont conçus pour l'acquisition de données de très haute précision, telles que des nuages de points LiDAR 3D et des images optiques. La multitude de données, ainsi que leur diversité, rendent complexe le traitement des données issues de ce type de systèmes. Cette thèse se place dans le contexte du traitement de l'image appliqué au nuages de points LiDAR 3D issus de ce type de système.Premièrement, nous nous intéressons à des images issues de la projection de nuages de points LiDAR dans des grilles de pixels 2D régulières. Ces projections créent généralement des images éparses, dans lesquelles l'information de certains pixels n'est pas connue. Nous proposons alors différentes méthodes pour des applications telles que la génération d'orthoimages haute résolution, l'imagerie RGB-D et l'estimation de la visibilité des points d'un nuage.De plus, nous proposons d'exploiter la topologie d'acquisition des capteurs LiDAR pour produire des images de faible résolution: les range-images. Ces images offrent une représentation efficace et canonique du nuage de points, tout en étant directement accessibles à partir du nuage de points. Nous montrons comment ces images peuvent être utilisées pour simplifier, voire améliorer, des méthodes pour le recalage multi-modal, la segmentation, la désoccultation et la détection 3D.The ever growing demand for reliable mapping data, especially in urban environments, has motivated the development of "close-range" Mobile Mapping Systems (MMS). These systems acquire high precision data, and in particular 3D LiDAR point clouds and optical images. The large amount of data, along with their diversity, make MMS data processing a very complex task. This thesis lies in the context of 2D image processing applied to 3D LiDAR point clouds acquired with MMS.First, we focus on the projection of the LiDAR point clouds onto 2D pixel grids to create images. Such projections are often sparse because some pixels do not carry any information. We use these projections for different applications such as high resolution orthoimage generation, RGB-D imaging and visibility estimation in point clouds.Moreover, we exploit the topology of LiDAR sensors in order to create low resolution images, named range-images. These images offer an efficient and canonical representation of the point cloud, while being directly accessible from the point cloud. We show how range-images can be used to simplify, and sometimes outperform, methods for multi-modal registration, segmentation, desocclusion and 3D detection

Traitement d’image 2D appliqué à des nuages de points LiDAR 3D

The ever growing demand for reliable mapping data, especially in urban environments, has motivated the development of "close-range" Mobile Mapping Systems (MMS). These systems acquire high precision data, and in particular 3D LiDAR point clouds and optical images. The large amount of data, along with their diversity, make MMS data processing a very complex task. This thesis lies in the context of 2D image processing applied to 3D LiDAR point clouds acquired with MMS.First, we focus on the projection of the LiDAR point clouds onto 2D pixel grids to create images. Such projections are often sparse because some pixels do not carry any information. We use these projections for different applications such as high resolution orthoimage generation, RGB-D imaging and visibility estimation in point clouds.Moreover, we exploit the topology of LiDAR sensors in order to create low resolution images, named range-images. These images offer an efficient and canonical representation of the point cloud, while being directly accessible from the point cloud. We show how range-images can be used to simplify, and sometimes outperform, methods for multi-modal registration, segmentation, desocclusion and 3D detection.L'intérêt toujours grandissant pour les données cartographiques fiables, notamment en milieu urbain, a motivé le développement de systèmes de cartographie mobiles terrestres. Ces systèmes sont conçus pour l'acquisition de données de très haute précision, telles que des nuages de points LiDAR 3D et des images optiques. La multitude de données, ainsi que leur diversité, rendent complexe le traitement des données issues de ce type de systèmes. Cette thèse se place dans le contexte du traitement de l'image appliqué au nuages de points LiDAR 3D issus de ce type de système.Premièrement, nous nous intéressons à des images issues de la projection de nuages de points LiDAR dans des grilles de pixels 2D régulières. Ces projections créent généralement des images éparses, dans lesquelles l'information de certains pixels n'est pas connue. Nous proposons alors différentes méthodes pour des applications telles que la génération d'orthoimages haute résolution, l'imagerie RGB-D et l'estimation de la visibilité des points d'un nuage.De plus, nous proposons d'exploiter la topologie d'acquisition des capteurs LiDAR pour produire des images de faible résolution: les range-images. Ces images offrent une représentation efficace et canonique du nuage de points, tout en étant directement accessibles à partir du nuage de points. Nous montrons comment ces images peuvent être utilisées pour simplifier, voire améliorer, des méthodes pour le recalage multi-modal, la segmentation, la désoccultation et la détection 3D

Visibility estimation in point clouds with variable density

Estimating visibility in point clouds has many applications such as visualization, surface reconstruction and scene analysis through fusion of LiDAR point clouds and images. However, most current works rely on methods that require strong assumptions on the point cloud density, which are not valid for LiDAR point clouds acquired from mobile mapping systems, leading to low quality of point visibility estimations. This work presents a novel approach for the estimation of the visibility of a point cloud from a viewpoint. The method is designed to be fully automatic and it makes no assumption on the point cloud density. The visibility of each point is estimated by considering its screen-space neighborhood from the given viewpoint. Our resultsshow that our approach succeeds better in estimating the visibility on real-world data acquired using LiDAR scanners. We evaluate our approach by comparing its results to a new manually annotated dataset, which we make available online

RIU-Net: Embarrassingly simple semantic segmentation of 3D LiDAR point cloud

This paper proposes RIU-Net (for Range-Image U-Net), the adaptation of a popular semantic segmentation network for the semantic segmentation of a 3D LiDAR point cloud. The point cloud is turned into a 2D range-image by exploiting the topology of the sensor. This image is then used as input to a U-net. This architecture has already proved its efficiency for the task of semantic segmentation of medical images. We propose to demonstrate how it can also be used for the accurate semantic segmentation of a 3D LiDAR point cloud. Our model is trained on range-images built from KITTI 3D object detection dataset. Experiments show that RIU-Net, despite being very simple, outperforms the state-of-the-art of range-image based methods. Finally, we demonstrate that this architecture is able to operate at 90fps on a single GPU, which enables deployment on low computational power systems such as robots

Détection et localisation d'objets 3D par apprentissage profond en topologie capteur

Ce travail présente une nouvelle méthode pour la détection et la localisation d'objets dans des scènes 3D LiDAR acquises par des systèmes de cartographie mobile. Ce problème est généralement traité en discrétisant l'espace 3D en une fine grille de voxels. Nous introduisons une approche alternative ne nécessitant pas de discrétisation. Elle est basée sur la représentation en 2D du nuage de points en topologie capteur. Cette image sert d'entrée à un réseau de neurones convolutionnels qui en extrait les informations 3D des objets. La répresentation en topologie capteur présentant des ambiguïtés dans le fond de la scène, nous améliorerons les résultats de détection en couplant ce modèle avec un réseau de détection 2D d'objets sur une image optique. Les prédictions des deux réseaux sont finalement fusionnées pour obtenir les détections finales.This work proposes a novel approach for detection and localisation of objects in 3D LiDAR scenes aquired via Mobile Mapping Systems. While this task is often treated on a voxel grid representations of the point cloud, our method offers to use the point cloud in sensor topology, thus avoiding a discretisation step. This representation of the point cloud is used as an input for a CNN that extracts 3D positions and dimensions of objects in the scene. As far objects in the scene tends to be mixed with the background when seen in the sensor topology, we offer to enhance the 3D detection by fusing the 3D predictions with 2D object detections performed on optical images

Désoccultation de nuage de points LiDAR en topologie capteur

Ce travail présente une nouvelle méthode pour la désoccultation d'objets mobiles dans des scènes 3D LIDAR acquises par des dispositifs de scan urbains. Ce problème est généralement traité directement dans l'espace 3D. Cet article propose une approche alternative qui repose sur une représentation 2D en topologie capteur du nuage de points 3D. Cette image est ensuite segmentée par une méthode basée histogramme afin d'extraire le masque des objets à supprimer. Enfin, une méthode d'inpainting variationnel est utilisée pour supprimer ces objets.This work proposes a novel approach for the disoccultation of mobile objects in 3D LIDAR scenes aquired via MMS. Although this work is often treated in the 3D space, our method offers to deal with it using a simplified representation of the point cloud known as range image. The range image is segmented using histograms. Then, it is inpainted using a variational method with an horizontal prior in the scene

Diffusion and inpainting of reflectance and height LiDAR orthoimages

This paper presents a fully automatic framework for the generation of so-called LiDAR orthoimages (i.e. 2D raster maps of the reflectance and height LiDAR samples) from ground-level LiDAR scans. Beyond the Digital Surface Model (DSM or heightmap) provided by the height orthoimage, the pro- posed method cost-effectively generates a reflectance channel that is easily interpretable by human operators without relying on any optical acquisition, calibration and registration. Moreover, it com- monly achieves very high resolutions (1cm2 per pixel), thanks to the typical sampling density of static or mobile LiDAR scans.Compared to orthoimages generated from aerial datasets, the proposed LiDAR orthoimages are ac- quired from the ground level and thus do not suffer occlusions from hovering objects (trees, tunnels, bridges ...), enabling their use in a number of urban applications such as road network monitoring and management, as well as precise mapping of the public space e.g. for accessibility applications or management of underground networks.Its generation and usability however faces two issues : (i) the inhomogeneous sampling density of LiDAR point clouds and (ii) the presence of masked areas (holes) behind occluders, which include, in a urban context, cars, tree trunks, poles, pedestrians... (i) is addressed by first projecting the point cloud on a 2D-pixel grid so as to generate sparse and noisy reflectance and height images from which dense images estimated using a joint anisotropic diffusion of the height and reflectance channels. (ii) LiDAR shadow areas are detected by analysing the diffusion results so that they can be inpainted using an examplar-based method, guided by an alignment prior.Results on real mobile and static acquisition data demonstrate the effectiveness of the proposed pipeline in generating a very high resolution LiDAR orthoimage of reflectance and height while filling holes of various sizes in a visually satisfying way

Range-Image: Incorporating sensor topology for LiDAR point cloud processing

This paper proposes a novel methodology for LiDAR point cloud processing that takes advantage of the implicit topology of various LiDAR sensors to derive 2D images from the point cloud while bringing spatial structure to each point. The interest of such a methodology is then proved by addressing the problems of segmentation and disocclusion of mobile objects in 3D LiDAR scenes acquired via street-based Mobile Mapping Systems (MMS). Most of the existing lines of research tackle those problems directly in the 3D space. This work promotes an alternative approach by using this image representation of the 3D point cloud, taking advantage of the fact that the problem of disocclusion has been intensively studied in the 2D image processing community over the past decade. Using the image derived from the sensor data by exploiting the sensor topology, a semi-automatic segmentation procedure based on depth histograms is presented. Then, a variational image inpainting technique is introduced to reconstruct the areas that are occluded by objects. Experiments and validation on real data prove the effectiveness of this methodology both in terms of accuracy and speed