Synthesizing Training Data for Object Detection in Indoor Scenes

Detection of objects in cluttered indoor environments is one of the key enabling functionalities for service robots. The best performing object detection approaches in computer vision exploit deep Convolutional Neural Networks (CNN) to simultaneously detect and categorize the objects of interest in cluttered scenes. Training of such models typically requires large amounts of annotated training data which is time consuming and costly to obtain. In this work we explore the ability of using synthetically generated composite images for training state-of-the-art object detectors, especially for object instance detection. We superimpose 2D images of textured object models into images of real environments at variety of locations and scales. Our experiments evaluate different superimposition strategies ranging from purely image-based blending all the way to depth and semantics informed positioning of the object models into real scenes. We demonstrate the effectiveness of these object detector training strategies on two publicly available datasets, the GMU-Kitchens and the Washington RGB-D Scenes v2. As one observation, augmenting some hand-labeled training data with synthetic examples carefully composed onto scenes yields object detectors with comparable performance to using much more hand-labeled data. Broadly, this work charts new opportunities for training detectors for new objects by exploiting existing object model repositories in either a purely automatic fashion or with only a very small number of human-annotated examples.Comment: Added more experiments and link to project webpag

InLoc: Indoor Visual Localization with Dense Matching and View Synthesis

We seek to predict the 6 degree-of-freedom (6DoF) pose of a query photograph with respect to a large indoor 3D map. The contributions of this work are three-fold. First, we develop a new large-scale visual localization method targeted for indoor environments. The method proceeds along three steps: (i) efficient retrieval of candidate poses that ensures scalability to large-scale environments, (ii) pose estimation using dense matching rather than local features to deal with textureless indoor scenes, and (iii) pose verification by virtual view synthesis to cope with significant changes in viewpoint, scene layout, and occluders. Second, we collect a new dataset with reference 6DoF poses for large-scale indoor localization. Query photographs are captured by mobile phones at a different time than the reference 3D map, thus presenting a realistic indoor localization scenario. Third, we demonstrate that our method significantly outperforms current state-of-the-art indoor localization approaches on this new challenging data

Current State of the Art Historic Building Information Modelling

In an extensive review of existing literature a number of observations were made in relation to the current approaches for recording and modelling existing buildings and environments: Data collection and pre-processing techniques are becoming increasingly automated to allow for near real-time data capture and fast processing of this data for later modelling applications. Current BIM software is almost completely focused on new buildings and has very limited tools and pre-defined libraries for modelling existing and historic buildings. The development of reusable parametric library objects for existing and historic buildings supports modelling with high levels of detail while decreasing the modelling time. Mapping these parametric objects to survey data, however, is still a time-consuming task that requires further research. Promising developments have been made towards automatic object recognition and feature extraction from point clouds for as-built BIM. However, results are currently limited to simple and planar features. Further work is required for automatic accurate and reliable reconstruction of complex geometries from point cloud data. Procedural modelling can provide an automated solution for generating 3D geometries but lacks the detail and accuracy required for most as-built applications in AEC and heritage fields

Automated 3D model generation for urban environments [online]

Abstract In this thesis, we present a fast approach to automated generation of textured 3D city models with both high details at ground level and complete coverage for birds-eye view. A ground-based facade model is acquired by driving a vehicle equipped with two 2D laser scanners and a digital camera under normal traffic conditions on public roads. One scanner is mounted horizontally and is used to determine the approximate component of relative motion along the movement of the acquisition vehicle via scan matching; the obtained relative motion estimates are concatenated to form an initial path. Assuming that features such as buildings are visible from both ground-based and airborne view, this initial path is globally corrected by Monte-Carlo Localization techniques using an aerial photograph or a Digital Surface Model as a global map. The second scanner is mounted vertically and is used to capture the 3D shape of the building facades. Applying a series of automated processing steps, a texture-mapped 3D facade model is reconstructed from the vertical laser scans and the camera images. In order to obtain an airborne model containing the roof and terrain shape complementary to the facade model, a Digital Surface Model is created from airborne laser scans, then triangulated, and finally texturemapped with aerial imagery. Finally, the facade model and the airborne model are fused to one single model usable for both walk- and fly-thrus. The developed algorithms are evaluated on a large data set acquired in downtown Berkeley, and the results are shown and discussed

Recent advances in monocular model-based tracking: a systematic literature review

In this paper, we review the advances of monocular model-based tracking for last ten years period until 2014. In 2005, Lepetit, et. al, [19] reviewed the status of monocular model based rigid body tracking. Since then, direct 3D tracking has become quite popular research area, but monocular model-based tracking should still not be forgotten. We mainly focus on tracking, which could be applied to aug- mented reality, but also some other applications are covered. Given the wide subject area this paper tries to give a broad view on the research that has been conducted, giving the reader an introduction to the different disciplines that are tightly related to model-based tracking. The work has been conducted by searching through well known academic search databases in a systematic manner, and by selecting certain publications for closer examination. We analyze the results by dividing the found papers into different categories by their way of implementation. The issues which have not yet been solved are discussed. We also discuss on emerging model-based methods such as fusing different types of features and region-based pose estimation which could show the way for future research in this subject.Siirretty Doriast

Automatic reconstruction of parametric building models from indoor point clouds

AbstractWe present an automatic approach for the reconstruction of parametric 3D building models from indoor point clouds. While recently developed methods in this domain focus on mere local surface reconstructions which enable e.g. efficient visualization, our approach aims for a volumetric, parametric building model that additionally incorporates contextual information such as global wall connectivity. In contrast to pure surface reconstructions, our representation thereby allows more comprehensive use: first, it enables efficient high-level editing operations in terms of e.g. wall removal or room reshaping which always result in a topologically consistent representation. Second, it enables easy taking of measurements like e.g. determining wall thickness or room areas. These properties render our reconstruction method especially beneficial to architects or engineers for planning renovation or retrofitting. Following the idea of previous approaches, the reconstruction task is cast as a labeling problem which is solved by an energy minimization. This global optimization approach allows for the reconstruction of wall elements shared between rooms while simultaneously maintaining plausible connectivity between all wall elements. An automatic prior segmentation of the point clouds into rooms and outside area filters large-scale outliers and yields priors for the definition of labeling costs for the energy minimization. The reconstructed model is further enriched by detected doors and windows. We demonstrate the applicability and reconstruction power of our new approach on a variety of complex real-world datasets requiring little or no parameter adjustment

EXTENDING A MOBILE DEVICE WITH LOW-COST 3D MODELING AND BUILDING-SCALE MAPPING CAPABILITIES, FOR APPLICATION IN ARCHITECTURE AND ARCHAEOLOGY

One of the most challenging problem in architecture is the automated construction of 3D (and 4D) digital models of cultural objects with the aim of implementing open data repositories, scientifically authenticated and responding to well accepted standards of validation, evaluation, preservation, publication, updating and dissemination. The realization of such an ambitious objective requires the adoption of special technological instruments. In this paper we plan to use portable devices (i.e. smartphones, tablets or PDAs eventually extended to wearable ones), extended with a small plug-in, for automatically extracting 3D models of single objects and building-scale mapping of the surrounding environment. At the same time, the device will provide the capability of inserting notes and observations. Where the instrument cannot be directly applied, for example for exploring the top of a complex building, we consider mounting our device, or using equivalent existing equipment, on a drone, in a modular approach for obtaining data de-facto interchangeable. The approach based on the expansion packs has the advantage of anticipating (or even promoting) future extensions of new mobile devices, when the spectrum of possible applications justify the corresponding increased costs. In order to experiment and verify this approach we plan to test it in two specific scenarios of the cultural heritage domain in which such devices seem particularly promising: Strada Nuova in Genoa and Palazzo Ducale in Urbino, both located in Italy

Extending a mobile device with low-cost 3D modeling and building-scale mapping capabilities, for application in architecture and archaeology

One of the most challenging problem in architecture is the automated construction of 3D (and 4D) digital models of cultural objects with the aim of implementing open data repositories, scientifically authenticated and responding to well accepted standards of validation, evaluation, preservation, publication, updating and dissemination. The realization of such an ambitious objective requires the adoption of special technological instruments. In this paper we plan to use portable devices (i.e. smartphones, tablets or PDAs eventually extended to wearable ones), extended with a small plug-in, for automatically extracting 3D models of single objects and building-scale mapping of the surrounding environment. At the same time, the device will provide the capability of inserting notes and observations. Where the instrument cannot be directly applied, for example for exploring the top of a complex building, we consider mounting our device, or using equivalent existing equipment, on a drone, in a modular approach for obtaining data de-facto interchangeable. The approach based on the expansion packs has the advantage of anticipating (or even promoting) future extensions of new mobile devices, when the spectrum of possible applications justify the corresponding increased costs. In order to experiment and verify this approach we plan to test it in two specific scenarios of the cultural heritage domain in which such devices seem particularly promising: Strada Nuova in Genoa and Palazzo Ducale in Urbino, both located in Italy