Documentation of heritage buildings using close-range UAV images: dense matching issues, comparison and case studies

International audienceThree-dimensional (3D) documentation of heritage buildings has long employed both image-based and range-based techniques. Unmanned aerial vehicles (UAVs) provide a particular advantage for image-based techniques in acquiring aerial views, which are difficult to attain using classical terrestrial-based methods. The technological development of optical sensors and dense matching algorithms also complement existing photogrammetric workflows for the documentation of heritage objects. In this paper, fundamental concepts in photogrammetry and 3D reconstruction based on structure from motion (SfM) will be briefly reviewed. Two case studies were performed using two types of UAVs, one being a state-of-the-art platform dedicated to obtaining close-range images. Comparisons with laser scanning data were performed and several issues regarding the aerial triangulation and dense matching results were assessed. The results show that although the dense matching of these UAV images may generate centimetre-level precision, a further increase in precision is often hampered by the quality of the onboard sensor

Development of inventory datasets through remote sensing and direct observation data for earthquake loss estimation

This report summarizes the lessons learnt in extracting exposure information for the three study sites, Thessaloniki, Vienna and Messina that were addressed in SYNER-G. Fine scale information on exposed elements that for SYNER-G include buildings, civil engineering works and population, is one of the variables used to quantify risk. Collecting data and creating exposure inventories is a very time-demanding job and all possible data-gathering techniques should be used to address the data shortcoming problem. This report focuses on combining direct observation and remote sensing data for the development of exposure models for seismic risk assessment. In this report a summary of the methods for collecting, processing and archiving inventory datasets is provided in Chapter 2. Chapter 3 deals with the integration of different data sources for optimum inventory datasets, whilst Chapters 4, 5 and 6 provide some case studies where combinations between direct observation and remote sensing have been used. The cities of Vienna (Austria), Thessaloniki (Greece) and Messina (Italy) have been chosen to test the proposed approaches.JRC.G.5-European laboratory for structural assessmen

Building structural characterization using mobile terrestrial point cloud for flood risk anticipation

Compte tenu de la fréquence élevée et de l'impact majeur des inondations, les décideurs, les acteurs des municipalités et le ministère de la sécurité publique ont un besoin urgent de disposer d'outils permettant de prédire ou d'évaluer l'importance des inondations et leur impact sur la population. D'après les statistiques, le premier étage des bâtiments, ainsi que les ouvertures inférieures, sont plus susceptibles de subir des dommages lors d'une inondation. Ainsi, dans le cadre de l'évaluation de l'impact des inondations, il serait nécessaire d'identifier l'emplacement de l'ouverture la plus basse des bâtiments et surtout sa hauteur par rapport au sol. Le système de balayage laser mobile (MLS) monté sur un véhicule s'est avéré être l'une des sources les plus fiables pour caractériser les bâtiments. Il peut produire des millions de points géoréférencés en 3D avec un niveau de détail suffisant, grâce à son point de vue depuis la rue et sa proximité. De plus, l'augmentation du nombre de jeux de données, issues des MLS acquis dans les villes et les environnements ruraux, permet de développer des approches pour caractériser les maisons résidentielles à l'échelle provinciale. Plusieurs défis sont associés à l'extraction d'informations descriptives des façades de bâtiments à l'aide de données MLS. Ainsi, les occlusions devant une façade rendent impossible l'obtention de points 3D sur ces parties de la façade. Aussi, comme les fenêtres sont principalement constituées de verre, qui ne réfléchit pas les signaux laser, les points disponibles pour celles-ci sont généralement limités. De plus, les approches de détection exploitent la répétitivité et les positions symétriques des ouvertures sur la façade. Mais ces caractéristiques sont absentes pour des maisons rurales et résidentielles. Finalement, la variabilité de la densité de points dans les données MLS rend difficile le processus de détection lorsqu'on travaille à l'échelle d'une ville. Par conséquent, l'objectif principal de cette recherche est de concevoir et de développer une approche globale d'extraction efficace des ouvertures présentes sur une façade. La solution proposée se compose de trois phases: l'extraction des façades, la détection des ouvertures et l'identification des occlusions. La première phase utilise une approche de segmentation adaptative par croissance de régions pour extraire la boîte englobante 3D de la façade. La deuxième phase combine la détection de trous avec une technique de maillage pour extraire les boîtes englobantes 2D des ouvertures. La dernière phase, qui vise à discriminer les occlusions des ouvertures, est en cours d'achèvement. Des évaluations qualitatives et quantitatives ont été réalisées à l'aide d'un jeu de données réelles, fourni par Jakarto Cartographie 3D Inc., de la province de Québec, au Canada. Les statistiques ont révélé que l'approche proposée pouvait obtenir de bons taux de performance malgré la complexité du jeu de données, représentatif des données acquises en situation réelle. Les défis concernant l'auto-occlusion de certaines façades et la présence de grandes occlusions environnantes seront à étudier plus en profondeur afin d'obtenir des informations plus précises sur les ouvertures des façades.Given the high frequency and major impact of floods, decision-makers, stakeholders in municipalities and public security ministry are in the urgent need to have tools allowing to predict or assess the significance of flood events and their impact on the population. Based on statistics, the first floor of the buildings, as well as the lower openings, are more likely subject to potential damage during a flood event. Thus, in the context of flood impact assessment, it would be required identifying the location of the buildings' lowest opening and especially its height above the ground. The capacity to characterize building with a relevant level of detail depends on the data sources used for the modeling. Different sources of data have been employed to characterize buildings' façade and openings. Mobile Laser Scanning (MLS) system mounted on a vehicle has proved to be one of the most reliable sources in this domain. It can produce millions of 3D georeferenced points with sufficient level of detail of the building facades and its openings, due to its street-view and close-range distance. Moreover, the increase of MLS providers and acquisitions in towns and rural environments, makes it possible to develop approaches to characterize residential houses at a provincial scale. Although being effective, several challenges are associated with extracting descriptive information of building facades using MLS data. The presence of occlusion in front of a facade makes it impossible to obtain the 3D points of the covered parts of the facade. Given the fact that windows mostly consist of glass and laser signals could not be reflected from the glass, limited points are usually available for windows. While the repetitive pattern and symmetrical positions of the openings on the facade makes it easier for the detection system to extract them, this characteristic is missing on the facade on rural and residential houses. The inconsistency of the point density in MLS data make the detection process even harder when working at city scale. Accordingly, the main objective of this research is to design and develop a comprehensive approach that effectively extracts facade openings. In order to meet the research project objective, the proposed solution consists of three phases including facade extraction, opening detection, and occlusion recognition. The first phase employs an adaptive region growing segmentation approach to extract the 3D bounding box of the facade. The second phase combines a hole-based assumption with an XZ gridding technique to extract 2D bounding boxes of the openings. The last phase which recognizes holes related to the occlusion from the openings is currently being completed. Qualitative and quantitative evaluations were performed using a real-word dataset provided by Jakarto Cartographie 3D inc. of the Quebec Province, Canada. Statistics revealed that the proposed approach could obtain good performance rates despite the complexity of the dataset, representative of the data acquired in real situations. Challenges regarding facade's self-occlusion and the presence of large surrounding occlusions should be further investigated for obtaining more accurate opening information on the facade

Structural Building Damage Detection with Deep Learning: Assessment of a State-of-the-Art CNN in Operational Conditions

Remotely sensed data can provide the basis for timely and efficient building damage maps that are of fundamental importance to support the response activities following disaster events. However, the generation of these maps continues to be mainly based on the manual extraction of relevant information in operational frameworks. Considering the identification of visible structural damages caused by earthquakes and explosions, several recent works have shown that Convolutional Neural Networks (CNN) outperform traditional methods. However, the limited availability of publicly available image datasets depicting structural disaster damages, and the wide variety of sensors and spatial resolution used for these acquisitions (from space, aerial and UAV platforms), have limited the clarity of how these networks can effectively serve First Responder needs and emergency mapping service requirements. In this paper, an advanced CNN for visible structural damage detection is tested to shed some light on what deep learning networks can currently deliver, and its adoption in realistic operational conditions after earthquakes and explosions is critically discussed. The heterogeneous and large datasets collected by the authors covering different locations, spatial resolutions and platforms were used to assess the network performances in terms of transfer learning with specific regard to geographical transferability of the trained network to imagery acquired in different locations. The computational time needed to deliver these maps is also assessed. Results show that quality metrics are influenced by the composition of training samples used in the network. To promote their wider use, three pre-trained networks—optimized for satellite, airborne and UAV image spatial resolutions and viewing angles—are made freely available to the scientific community

UAV strategies validation and remote sensing data for damage assessment in post-disaster scenarios

The recent seismic swarms, occurred in Italy since August 2016, outlined the importance of deepen Geomatics researches for the validation of new strategies aimed at rapid-mapping and documenting differently accessible and complex environments, as in urban contexts and damaged built heritage. In the emergency response, the crucial exploitation of technological advances should obtain and efficiently organize high-scale reliable geospatial data for the early warning, impact, and recovery phases. Fulfilling these issues, among others, the Copernicus EMS, has played by now an important role in immediate and extensive damage reconnaissance, as in the case of Centre Italy. Nevertheless, the use of remote sensing data is still affected by a problem of point-of-view, scale and detectable detail. Nadir images, airborne or satellite, in fact, strongly limited the confidence level of these products. The subjectivity of the operator involvement is still an open issue, both in the first fieldwork assessment, and in the following operational approach of interpretative damage detection and rapid mapping production. To overcome these limits, the introduction of UAV platforms for photogrammetric purposes, has proven to be a sustainable approach in terms of time savings, operators’ safety, reliability and accuracy of results: the nadir and oblique integration can provide large multiscale models, with the fundamental information related to the façades conditions. The presented research, conducted within the Central Italy earthquakes events, will focus on potentialities and limits of UAV photogrammetry in the two documented sites: Pescara del Tronto and Accumoli. Here, the aim is not limited to describe a series of strategies for georeferencing, blocks orientation and multitemporal co-registration solutions, but also to validate the implemented pipelines as a workflow that could be integrated in the operative intervention for emergency response in early impact activities. Thus, it would be possible to use this 3D metric products as a reference-data for significative improvements of reliability in typical visual inspection and mapping, flanking the traditional nadir airborne- or satellite-based products. The UAV acquisitions performed in two damaged villages are displayed, in order to underline the implication of the spatial information embedded in DSM reconstruction and 3D models, supporting more reliable damage assessments

Investigation of rapid remote sensing techniques for forensic wind analyses

Perishable damage data resulting from severe windstorms require efficient and rapid field collection techniques. Such datasets permit forensic damage investigations and characterization of civil infrastructure. Ultimately, observed structural damage serves as a proxy approach to estimate wind speeds for storms that include hurricanes, tornadoes, straight-line winds, etc. One of the more common methods to collect, preserve, and reconstruct three-dimensional damage scenes is the use of an unmanned aerial system (UAS), commonly known as a drone. Onboard photographic payloads permit scene reconstruction via structure-from-motion; however, such approaches often require direct site access and survey points for accurate results, which limit its efficiency. In this presentation, the use of UAS platforms with and without surveyed ground control points is investigated to understand the accuracy if site access is not possible. UAS datasets will be compared to lidar data of various structures collected following the 2017 Hurricane Harvey near Rockport, TX

Concepts and tools to improve the thermal energy performance of buildings and urban districts - diagnosis, assessment, improvement strategies and cost-benefit analyses

Retrofitting existing buildings to optimize their thermal energy performance is a key factor in achieving climate neutrality by 2045 in Germany. Analyzing buildings in their current condition is the first step toward preparing effective and efficient energy retrofit measures. A high-quality building analysis helps to evaluate whether a building or its components are suitable for retrofitting or replacement. Subsequently, appropriate combinations of retrofit measures that create financial and environmental synergies can be determined. This dissertation is a cumulative work based on nine papers on the thermal analysis of existing buildings. The focus of this work and related papers is on thermography with drones for building audits, intelligent processing of thermographic images to detect and assess thermal weaknesses, and building modeling approaches to evaluate thermal retrofit options. While individual buildings are usually the focus of retrofit planning, this dissertation also examines the role of buildings in the urban context, particularly on a district level. Multiple adjacent buildings offer numerous possibilities for further improving retrofits, such as the economies of scale for planning services and material procurement, neighborhood dynamics, and exchange of experiences between familiar building owners. This work reveals the opportunities and obstacles for panorama drone thermography for building audits. It shows that drones can contribute to a quick and structured data collection, particularly for large building stocks, and thus complement current approaches for district-scale analysis. However, the significant distance between the drone camera and building, which is necessary for automated flight routes, and varying recording angles limit the quantitative interpretability of thermographic images. Therefore, innovative approaches were developed to process image datasets generated using drones. A newly designed AI-based approach can automate the detection of thermal bridges on rooftops. Using generalizations about certain building classes as demonstrated by buildings from the 1950s and 1960s, a novel interpretation method for drone images is suggested. It enables decision-making regarding the need to retrofit thermal bridges of recorded buildings. A novel optimization model for German single-family houses was developed and applied in a case study to investigate the financial and ecological benefits of different thermal retrofit measures. The results showed that the retrofitting of building façades can significantly save energy. However, they also revealed that replacing the heating systems turns out to be more cost-effective for carbon dioxide savings. Small datasets, limited availability of technical equipment, and the need for simplified assumptions for building characteristics without any information were the main challenges of the approaches in this dissertation

A Review

Structural disaster damage detection and characterization is one of the oldest remote sensing challenges, and the utility of virtually every type of active and passive sensor deployed on various air- and spaceborne platforms has been assessed. The proliferation and growing sophistication of unmanned aerial vehicles (UAVs) in recent years has opened up many new opportunities for damage mapping, due to the high spatial resolution, the resulting stereo images and derivatives, and the flexibility of the platform. This study provides a comprehensive review of how UAV-based damage mapping has evolved from providing simple descriptive overviews of a disaster science, to more sophisticated texture and segmentation-based approaches, and finally to studies using advanced deep learning approaches, as well as multi-temporal and multi-perspective imagery to provide comprehensive damage descriptions. The paper further reviews studies on the utility of the developed mapping strategies and image processing pipelines for first responders, focusing especially on outcomes of two recent European research projects, RECONASS (Reconstruction and Recovery Planning: Rapid and Continuously Updated Construction Damage, and Related Needs Assessment) and INACHUS (Technological and Methodological Solutions for Integrated Wide Area Situation Awareness and Survivor Localization to Support Search and Rescue Teams). Finally, recent and emerging developments are reviewed, such as recent improvements in machine learning, increasing mapping autonomy, damage mapping in interior, GPS-denied environments, the utility of UAVs for infrastructure mapping and maintenance, as well as the emergence of UAVs with robotic abilities. Document type: Articl