LIDAR-Camera Fusion for Road Detection Using Fully Convolutional Neural Networks

In this work, a deep learning approach has been developed to carry out road detection by fusing LIDAR point clouds and camera images. An unstructured and sparse point cloud is first projected onto the camera image plane and then upsampled to obtain a set of dense 2D images encoding spatial information. Several fully convolutional neural networks (FCNs) are then trained to carry out road detection, either by using data from a single sensor, or by using three fusion strategies: early, late, and the newly proposed cross fusion. Whereas in the former two fusion approaches, the integration of multimodal information is carried out at a predefined depth level, the cross fusion FCN is designed to directly learn from data where to integrate information; this is accomplished by using trainable cross connections between the LIDAR and the camera processing branches. To further highlight the benefits of using a multimodal system for road detection, a data set consisting of visually challenging scenes was extracted from driving sequences of the KITTI raw data set. It was then demonstrated that, as expected, a purely camera-based FCN severely underperforms on this data set. A multimodal system, on the other hand, is still able to provide high accuracy. Finally, the proposed cross fusion FCN was evaluated on the KITTI road benchmark where it achieved excellent performance, with a MaxF score of 96.03%, ranking it among the top-performing approaches

Integrating data from 3D CAD and 3D cameras for Real-Time Modeling

In a reversal of historic trends, the capital facilities industry is expressing an increasing desire for automation of equipment and construction processes. Simultaneously, the industry has become conscious that higher levels of interoperability are a key towards higher productivity and safer projects. In complex, dynamic, and rapidly changing three-dimensional (3D) environments such as facilities sites, cutting-edge 3D sensing technologies and processing algorithms are one area of development that can dramatically impact those projects factors. New 3D technologies are now being developed, with among them 3D camera. The main focus here is an investigation of the feasibility of rapidly combining and comparing – integrating – 3D sensed data (from a 3D camera) and 3D CAD data. Such a capability could improve construction quality assessment, facility aging assessment, as well as rapid environment reconstruction and construction automation. Some preliminary results are presented here. They deal with the challenge of fusing sensed and CAD data that are completely different in nature

3D modeling of indoor environments by a mobile platform with a laser scanner and panoramic camera

One major challenge of 3DTV is content acquisition. Here, we present a method to acquire a realistic, visually convincing D model of indoor environments based on a mobile platform that is equipped with a laser range scanner and a panoramic camera. The data of the 2D laser scans are used to solve the simultaneous lo- calization and mapping problem and to extract walls. Textures for walls and floor are built from the images of a calibrated panoramic camera. Multiresolution blending is used to hide seams in the gen- erated textures. The scene is further enriched by 3D-geometry cal- culated from a graph cut stereo technique. We present experimental results from a moderately large real environment.