Material Recognition Meets 3D Reconstruction : Novel Tools for Efficient, Automatic Acquisition Systems

For decades, the accurate acquisition of geometry and reflectance properties has represented one of the major objectives in computer vision and computer graphics with many applications in industry, entertainment and cultural heritage. Reproducing even the finest details of surface geometry and surface reflectance has become a ubiquitous prerequisite in visual prototyping, advertisement or digital preservation of objects. However, today's acquisition methods are typically designed for only a rather small range of material types. Furthermore, there is still a lack of accurate reconstruction methods for objects with a more complex surface reflectance behavior beyond diffuse reflectance. In addition to accurate acquisition techniques, the demand for creating large quantities of digital contents also pushes the focus towards fully automatic and highly efficient solutions that allow for masses of objects to be acquired as fast as possible. This thesis is dedicated to the investigation of basic components that allow an efficient, automatic acquisition process. We argue that such an efficient, automatic acquisition can be realized when material recognition "meets" 3D reconstruction and we will demonstrate that reliably recognizing the materials of the considered object allows a more efficient geometry acquisition. Therefore, the main objectives of this thesis are given by the development of novel, robust geometry acquisition techniques for surface materials beyond diffuse surface reflectance, and the development of novel, robust techniques for material recognition. In the context of 3D geometry acquisition, we introduce an improvement of structured light systems, which are capable of robustly acquiring objects ranging from diffuse surface reflectance to even specular surface reflectance with a sufficient diffuse component. We demonstrate that the resolution of the reconstruction can be increased significantly for multi-camera, multi-projector structured light systems by using overlappings of patterns that have been projected under different projector poses. As the reconstructions obtained by applying such triangulation-based techniques still contain high-frequency noise due to inaccurately localized correspondences established for images acquired under different viewpoints, we furthermore introduce a novel geometry acquisition technique that complements the structured light system with additional photometric normals and results in significantly more accurate reconstructions. In addition, we also present a novel method to acquire the 3D shape of mirroring objects with complex surface geometry. The aforementioned investigations on 3D reconstruction are accompanied by the development of novel tools for reliable material recognition which can be used in an initial step to recognize the present surface materials and, hence, to efficiently select the subsequently applied appropriate acquisition techniques based on these classified materials. In the scope of this thesis, we therefore focus on material recognition for scenarios with controlled illumination as given in lab environments as well as scenarios with natural illumination that are given in photographs of typical daily life scenes. Finally, based on the techniques developed in this thesis, we provide novel concepts towards efficient, automatic acquisition systems

Municipal Road Infrastructure Assessment Using Street View Images

Road quality assessment is a crucial part in Municipalities' work to maintain their infrastructure, plan upgrades, and manage their budgets. Properly maintaining this infrastructure relies heavily on consistently monitoring its condition and deterioration over time. This can be a challenge, especially in larger towns and cities where there is a lot of city property to keep an eye on. Municipalities rely on surveyors to keep them up to date on the condition of their infrastructure to prevent this failure before it happens. This is both to prevent injuries and further damage from occurring as a result of infrastructure failure, and since it is can be more cost effective to maintain property rather than have to replace it. Surveying can either be done manually or automatically, but it is not done frequently as it is expensive and also time consuming. Manual surveying can be inaccurate, while a large portion of automatic surveying techniques rely on expensive equipment. To solve this problem, we propose an automated infrastructure assessment method that relies on Street View images for its input and uses various computer vision and pattern recognition methods to generate its assessments. First, we segment the image into 'road' and 'background' regions. We propose a road segmentation algorithm specifically aimed at segmenting roads from street view images. We use Fisher vectors calculated on SIFT descriptors to encode small windows extracted from the main image at multiple scales. Then we classify these patches using an SVM and utilize a Gaussian voting scheme to obtain a segmentation. We additionally utilize a spatial prior to improve this segmentation. Optionally, we improve the segmentation further by making use of a weighted contour map calculated on a shadow-free intrinsic image, and a find an optimal segmentation by utilizing a purity tree. Our algorithm performs well and outputs a good segmentation for further use in road evaluation. We test our method on the KITTI road dataset, and compare it to the state-of-the-art on this dataset, along with a manually annotated subset of Google Street View. After segmenting the road, we describe an algorithm aimed at identifying distressed road regions and pinpointing cracks within them. We predict distressed regions by re-using the computed Fisher vectors and classifying them with a different SVM trained to distinguish between road qualities. We follow this step with a comparison to the weighed contour map within these distressed regions to identify exact crack and defect locations, and use the contour weights to predict the crack severity. Promising results are obtained on our manually annotated dataset, which indicate the viability of using this cost-effective system to perform road quality assessment at a municipal level