Image-based window detection: an overview

Automated segmentation of buildings’ façade and detection of its elements is of high relevance in various fields of research as it, e. g., reduces the effort of 3 D reconstructing existing buildings and even entire cities or may be used for navigation and localization tasks. In recent years, several approaches were made concerning this issue. These can be mainly classified by their input data which are either images or 3 D point clouds. This paper provides a survey of image-based approaches. Particularly, this paper focuses on window detection and therefore groups related papers into the three major detection strategies. We juxtapose grammar based methods, pattern recognition and machine learning and contrast them referring to their generality of application. As we found out machine learning approaches seem most promising for window detection on generic façades and thus we will pursue these in future work

Incremental map refinement of building information using lidar point clouds

For autonomous systems, an accurate and precise map of the environment is of importance. Mapping from LiDAR point clouds is one of the promising ways to generate 3D environment models. However, there are many problems caused by inaccurate data, missing areas, low density of points and sensor noise. Also, it is often not possible or accurate enough to generate a map from only one measurement campaign. In this paper, we propose a method to incrementally refine the map by several measurements from different campaigns and represent the map in a hierarchical way with a measure indicating uncertainty and the level of detail for objects. The idea is thus to store all captured information with a tentative semantics and uncertainty - even when it is not yet complete. Hence, occulated areas are presented as well, which can be possibly improved by the supplemental observation from the next measurement campaign. The proposed 3D environment model framework and the incremental update method are evaluated using LiDAR scans obtained from Riegl Mobile Mapping System

Automatic Registration of Optical Aerial Imagery to a LiDAR Point Cloud for Generation of City Models

This paper presents a framework for automatic registration of both the optical and 3D structural information extracted from oblique aerial imagery to a Light Detection and Ranging (LiDAR) point cloud without prior knowledge of an initial alignment. The framework employs a coarse to fine strategy in the estimation of the registration parameters. First, a dense 3D point cloud and the associated relative camera parameters are extracted from the optical aerial imagery using a state-of-the-art 3D reconstruction algorithm. Next, a digital surface model (DSM) is generated from both the LiDAR and the optical imagery-derived point clouds. Coarse registration parameters are then computed from salient features extracted from the LiDAR and optical imagery-derived DSMs. The registration parameters are further refined using the iterative closest point (ICP) algorithm to minimize global error between the registered point clouds. The novelty of the proposed approach is in the computation of salient features from the DSMs, and the selection of matching salient features using geometric invariants coupled with Normalized Cross Correlation (NCC) match validation. The feature extraction and matching process enables the automatic estimation of the coarse registration parameters required for initializing the fine registration process. The registration framework is tested on a simulated scene and aerial datasets acquired in real urban environments. Results demonstrates the robustness of the framework for registering optical and 3D structural information extracted from aerial imagery to a LiDAR point cloud, when co-existing initial registration parameters are unavailable

AUTOMATIC IMAGE TO MODEL ALIGNMENT FOR PHOTO-REALISTIC URBAN MODEL RECONSTRUCTION

We introduce a hybrid approach in which images of an urban scene are automatically alignedwith a base geometry of the scene to determine model-relative external camera parameters. Thealgorithm takes as input a model of the scene and images with approximate external cameraparameters and aligns the images to the model by extracting the facades from the images andaligning the facades with the model by minimizing over a multivariate objective function. Theresulting image-pose pairs can be used to render photo-realistic views of the model via texturemapping.Several natural extensions to the base hybrid reconstruction technique are also introduced. Theseextensions, which include vanishing point based calibration refinement and video stream basedreconstruction, increase the accuracy of the base algorithm, reduce the amount of data that mustbe provided by the user as input to the algorithm, and provide a mechanism for automaticallycalibrating a large set of images for post processing steps such as automatic model enhancementand fly-through model visualization.Traditionally, photo-realistic urban reconstruction has been approached from purely image-basedor model-based approaches. Recently, research has been conducted on hybrid approaches, whichcombine the use of images and models. Such approaches typically require user assistance forcamera calibration. Our approach is an improvement over these methods because it does notrequire user assistance for camera calibration

Building structural characterization using mobile terrestrial point cloud for flood risk anticipation

Compte tenu de la fréquence élevée et de l'impact majeur des inondations, les décideurs, les acteurs des municipalités et le ministère de la sécurité publique ont un besoin urgent de disposer d'outils permettant de prédire ou d'évaluer l'importance des inondations et leur impact sur la population. D'après les statistiques, le premier étage des bâtiments, ainsi que les ouvertures inférieures, sont plus susceptibles de subir des dommages lors d'une inondation. Ainsi, dans le cadre de l'évaluation de l'impact des inondations, il serait nécessaire d'identifier l'emplacement de l'ouverture la plus basse des bâtiments et surtout sa hauteur par rapport au sol. Le système de balayage laser mobile (MLS) monté sur un véhicule s'est avéré être l'une des sources les plus fiables pour caractériser les bâtiments. Il peut produire des millions de points géoréférencés en 3D avec un niveau de détail suffisant, grâce à son point de vue depuis la rue et sa proximité. De plus, l'augmentation du nombre de jeux de données, issues des MLS acquis dans les villes et les environnements ruraux, permet de développer des approches pour caractériser les maisons résidentielles à l'échelle provinciale. Plusieurs défis sont associés à l'extraction d'informations descriptives des façades de bâtiments à l'aide de données MLS. Ainsi, les occlusions devant une façade rendent impossible l'obtention de points 3D sur ces parties de la façade. Aussi, comme les fenêtres sont principalement constituées de verre, qui ne réfléchit pas les signaux laser, les points disponibles pour celles-ci sont généralement limités. De plus, les approches de détection exploitent la répétitivité et les positions symétriques des ouvertures sur la façade. Mais ces caractéristiques sont absentes pour des maisons rurales et résidentielles. Finalement, la variabilité de la densité de points dans les données MLS rend difficile le processus de détection lorsqu'on travaille à l'échelle d'une ville. Par conséquent, l'objectif principal de cette recherche est de concevoir et de développer une approche globale d'extraction efficace des ouvertures présentes sur une façade. La solution proposée se compose de trois phases: l'extraction des façades, la détection des ouvertures et l'identification des occlusions. La première phase utilise une approche de segmentation adaptative par croissance de régions pour extraire la boîte englobante 3D de la façade. La deuxième phase combine la détection de trous avec une technique de maillage pour extraire les boîtes englobantes 2D des ouvertures. La dernière phase, qui vise à discriminer les occlusions des ouvertures, est en cours d'achèvement. Des évaluations qualitatives et quantitatives ont été réalisées à l'aide d'un jeu de données réelles, fourni par Jakarto Cartographie 3D Inc., de la province de Québec, au Canada. Les statistiques ont révélé que l'approche proposée pouvait obtenir de bons taux de performance malgré la complexité du jeu de données, représentatif des données acquises en situation réelle. Les défis concernant l'auto-occlusion de certaines façades et la présence de grandes occlusions environnantes seront à étudier plus en profondeur afin d'obtenir des informations plus précises sur les ouvertures des façades.Given the high frequency and major impact of floods, decision-makers, stakeholders in municipalities and public security ministry are in the urgent need to have tools allowing to predict or assess the significance of flood events and their impact on the population. Based on statistics, the first floor of the buildings, as well as the lower openings, are more likely subject to potential damage during a flood event. Thus, in the context of flood impact assessment, it would be required identifying the location of the buildings' lowest opening and especially its height above the ground. The capacity to characterize building with a relevant level of detail depends on the data sources used for the modeling. Different sources of data have been employed to characterize buildings' façade and openings. Mobile Laser Scanning (MLS) system mounted on a vehicle has proved to be one of the most reliable sources in this domain. It can produce millions of 3D georeferenced points with sufficient level of detail of the building facades and its openings, due to its street-view and close-range distance. Moreover, the increase of MLS providers and acquisitions in towns and rural environments, makes it possible to develop approaches to characterize residential houses at a provincial scale. Although being effective, several challenges are associated with extracting descriptive information of building facades using MLS data. The presence of occlusion in front of a facade makes it impossible to obtain the 3D points of the covered parts of the facade. Given the fact that windows mostly consist of glass and laser signals could not be reflected from the glass, limited points are usually available for windows. While the repetitive pattern and symmetrical positions of the openings on the facade makes it easier for the detection system to extract them, this characteristic is missing on the facade on rural and residential houses. The inconsistency of the point density in MLS data make the detection process even harder when working at city scale. Accordingly, the main objective of this research is to design and develop a comprehensive approach that effectively extracts facade openings. In order to meet the research project objective, the proposed solution consists of three phases including facade extraction, opening detection, and occlusion recognition. The first phase employs an adaptive region growing segmentation approach to extract the 3D bounding box of the facade. The second phase combines a hole-based assumption with an XZ gridding technique to extract 2D bounding boxes of the openings. The last phase which recognizes holes related to the occlusion from the openings is currently being completed. Qualitative and quantitative evaluations were performed using a real-word dataset provided by Jakarto Cartographie 3D inc. of the Quebec Province, Canada. Statistics revealed that the proposed approach could obtain good performance rates despite the complexity of the dataset, representative of the data acquired in real situations. Challenges regarding facade's self-occlusion and the presence of large surrounding occlusions should be further investigated for obtaining more accurate opening information on the facade