Machine and deep learning implementations for heritage building information modelling : a critical review of theoretical and applied research

Research domain and Problem: HBIM modelling from point cloud data has become a crucial research topic in the last decade since it is potentially considered as the central data model paving the way for the digital heritage practice beyond digitization. Reality Capture technologies such as terrestrial laser scanning, drone-mounted LiDAR sensors and photogrammetry enable the reality capture with a sub-millimetre accurate point cloud file that can be used as a reference file for Heritage Building Information Modelling (HBIM). However, HBIM modelling from the point cloud data of heritage buildings is mainly manual, error-prone, and time-consuming. Furthermore, image processing techniques are insufficient for classification and segmentation of point cloud data to speed up and enhance the current workflow for HBIM modelling. Due to the challenges and bottlenecks in the scan-to-HBIM process, which is commonly criticized as complex with its bespoke requirements, semantic segmentation of point clouds is gaining popularity in the literature. Research Aim and Methodology: Therefore, this paper aims to provide a thorough critical review of Machine Learning and Deep Learning methods for point cloud segmentation, classification, and BIM geometry automation for cultural heritage case study applications. Research findings: This paper files the challenges of HBIM practice and the opportunities for semantic point cloud segmentation found across academic literature in the last decade. Beyond definitions and basic occurrence statistics, this paper discusses the success rates and implementation challenges of machine and deep learning classification methods. Research value and contribution: This paper provides a holistic review of point cloud segmentation and its potential for further development and application in the Cultural Heritage sector. The critical analysis provides insight into the current state-of-the-art methods and advises on their suitability for HBIM projects. The review has identified highly original threads of research, which hold the potential to significantly influence practice and further applied research

Extracting Physical and Environmental Information of Irish Roads Using Airborne and Mobile Sensors

Airborne sensors including LiDAR and digital cameras are now used extensively for capturing topographical information as these are often more economical and efficient as compared to the traditional photogrammetric and land surveying techniques. Data captured using airborne sensors can be used to extract 3D information important for, inter alia, city modelling, land use classification and urban planning. According to the EU noise directive (2002/49/EC), the National Road Authority (NRA) in Ireland is responsible for generating noise models for all roads which are used by more than 8,000 vehicles per day. Accordingly, the NRA has to cover approximately 4,000 km of road, 500m on each side. These noise models have to be updated every 5 years. Important inputs to noise model are digital terrain model (DTM), 3D building data, road width, road centre line, ground surface type and noise barriers. The objective of this research was to extract these objects and topographical information using nationally available datasets acquired from the Ordnance Survey of Ireland (OSI). The OSI uses ALS50-II LiDAR and ADS40 digital sensors for capturing ground information. Both sensors rely on direct georeferencing, minimizing the need for ground control points. Before exploiting the complementary nature of both datasets for information extraction, their planimetric and vertical accuracies were evaluated using independent ground control points. A new method was also developed for registration in case of any mismatch. DSMs from LiDAR and aerial images were used to find common points to determine the parameters of 2D conformal transformation. The developed method was also evaluated by the EuroSDR in a project which involved a number of partners. These measures were taken to ensure that the inputs to the noise model were of acceptable accuracy as recommended in the report (Assessment of Exposure to Noise, 2006) by the European Working Group. A combination of image classification techniques was used to extract information by the fusion of LiDAR and aerial images. The developed method has two phases, viz. object classification and object reconstruction. Buildings and vegetation were classified based on Normalized Difference Vegetation Index (NDVI) and a normalized digital surface model (nDSM). Holes in building segments were filled by object-oriented multiresolution segmentation. Vegetation that remained amongst buildings was classified using cues obtained from LiDAR. The short comings there in were overcome by developing an additional classification cue using multiple returns. The building extents were extracted and assigned a single height value generated from LiDAR nDSM. The extracted height was verified against the ground truth data acquired using terrestrial survey techniques. Vegetation was further classified into three categories, viz. trees, hedges and tree clusters based on shape parameter (for hedges) and distance from neighbouring trees (for clusters). The ground was classified into three surface types i.e. roads and parking area, exposed surface and grass. This was done using LiDAR intensity, NDVI and nDSM. Mobile Laser Scanning (MLS) data was used to extract walls and purpose built noise barriers, since these objects were not extractable from the available airborne sensor data. Principal Component Analysis (PCA) was used to filter points belonging to such objects. A line was then fitted to these points using robust least square fitting. The developed object extraction method was tested objectively in two independent areas namely the Test Area-1 and the Test Area-2. The results were thoroughly investigated by three different accuracy assessment methods using the OSI vector data. The acceptance of any developed method for commercial applications requires completeness and correctness values of 85% and 70% respectively. Accuracy measures obtained using the developed method of object extraction recommend its applicability for noise modellin

Remote sensing and data fusion of cultural and physical landscapes

This dissertation is written as part of the three-article option offered by the Geography Department at UNC Greensboro. Each article addresses specific research issues within Remote Sensing, Photogrammetry, and three-dimensional modeling related structural and subsurface remote sensing of historic cultural landscapes. The articles submitted in this dissertation are both separate study sites and research questions, but the unifying theme of geographic research methods applies throughout. The first article is titled Terrestrial Lidar and GPR Investigations into the Third Line of Battle at Guilford Courthouse National Military Park, Guilford County, North Carolina is published in the book Digital Methods and Remote Sensing in Archaeology: Archaeology in the Age of Sensing. Forte, Maurizio, Campana, Stefano R.L. (Eds.) 2016. The results of the research demonstrate the successful exportation of GPR data into three-dimensional point clouds. Subsequently, the converted GPR points in conjunction with the TLS were explored to aid in the identification of the colonial subsurface. The second article submitted for consideration is titled “Three-Dimensional Modeling using Terrestrial LiDAR, Unmanned Aerial Vehicles, and Digital Cameras at House in the Horseshoe State Historic Site, Sanford, North Carolina.” There are two different research components to this study, modeling a structure and the landscape. The structure modeling section compares three different remote sensing approaches to the capture and three-dimensional model creation of a historic building. A detailed comparison is made between the photogrammetric models generated from digital camera photography, a terrestrial laser scanner (TLS) and an unmanned aerial vehicle (UAS). The final article, “Geophysical Investigations at the Harper House Bentonville Battlefield, NC State Historic Site” submitted focuses on the Harper House located in at the Bentonville Civil War battlefield. UNCG conducted a geophysical survey using a ground penetrating radar and gradiometer. The findings from the data were used to determine and pinpoint areas of interest for subsequent excavation

UAV data modeling for geoinformation update

A dissertação visa avaliar a relevância e o desempenho dos dados obtidos por Veículos Aéreos Não Tripulados (VANT) na atualização de Geoinformação. Os dados obtidos por VANT serão utilizados quer em conjunto com outros dados – obtidos por plataformas tradicionais de deteção remota –, quer isoladamente, recorrendo à técnica de Structure from Motion (SfM), para gerar o modelo digital de superfície e os ortomosaicos de alta precisão em diferentes momentos. Para a avaliação da precisão dos dados, os modelos digitais de terreno serão comparados. Por outro lado, os dados e informação gerados permitirão atualizar Geoinformação e quantificar as mudanças ocorridas no uso e ocupação do solo. Os resultados irão alimentar a discussão crítica da ação antrópica nos aglomerados urbanos e as propostas de intervenção.The dissertation aims to assess the relevance and performance of data obtained by Unmanned Aerial Vehicles (UAVs) in updating Geoinformation. The data obtained by UAVs will be used either in conjunction with other data – obtained by traditional remote sensing platforms – or on its own, using the Structure from Motion (SfM) technique, to generate high-precision digital surface models and orthomosaics at different times. For the accuracy assessment of the data, the digital terrain models will be compared. On the other hand, the data and information generated will make it possible to update Geoinformation and quantify changes in land use and occupation. The results will feed the critical discussion of anthropic action in urban areas and intervention proposals

Photogrammetric suite to manage the survey workflow in challenging environments and conditions

The present work is intended in providing new and innovative instruments to support the photogrammetric survey workflow during all its phases. A suite of tools has been conceived in order to manage the planning, the acquisition, the post-processing and the restitution steps, with particular attention to the rigorousness of the approach and to the final precision. The main focus of the research has been the implementation of the tool MAGO, standing for Adaptive Mesh for Orthophoto Generation. Its novelty consists in the possibility to automatically reconstruct \u201cunrolled\u201d orthophotos of adjacent fa\ue7ades of a building using the point cloud, instead of the mesh, as input source for the orthophoto reconstruction. The second tool has been conceived as a photogrammetric procedure based on Bundle Block Adjustment. The same issue is analysed from two mirrored perspectives: on the one hand, the use of moving cameras in a static scenario in order to manage real-time indoor navigation; on the other hand, the use of static cameras in a moving scenario in order to achieve the simultaneously reconstruction of the 3D model of the changing object. A third tool named U.Ph.O., standing for Unmanned Photogrammetric Office, has been integrated with a new module. The general aim is on the one hand to plan the photogrammetric survey considering the expected precision, computed on the basis of a network simulation, and on the other hand to check if the achieved survey has been collected compatibly with the planned conditions. The provided integration concerns the treatment of surfaces with a generic orientation further than the ones with a planimetric development. After a brief introduction, a general description about the photogrammetric principles is given in the first chapter of the dissertation; a chapter follows about the parallelism between Photogrammetry and Computer Vision and the contribution of this last in the development of the described tools. The third chapter specifically regards, indeed, the implemented software and tools, while the fourth contains the training test and the validation. Finally, conclusions and future perspectives are reported

Remote Sensing and Geosciences for Archaeology

This book collects more than 20 papers, written by renowned experts and scientists from across the globe, that showcase the state-of-the-art and forefront research in archaeological remote sensing and the use of geoscientific techniques to investigate archaeological records and cultural heritage. Very high resolution satellite images from optical and radar space-borne sensors, airborne multi-spectral images, ground penetrating radar, terrestrial laser scanning, 3D modelling, Geographyc Information Systems (GIS) are among the techniques used in the archaeological studies published in this book. The reader can learn how to use these instruments and sensors, also in combination, to investigate cultural landscapes, discover new sites, reconstruct paleo-landscapes, augment the knowledge of monuments, and assess the condition of heritage at risk. Case studies scattered across Europe, Asia and America are presented: from the World UNESCO World Heritage Site of Lines and Geoglyphs of Nasca and Palpa to heritage under threat in the Middle East and North Africa, from coastal heritage in the intertidal flats of the German North Sea to Early and Neolithic settlements in Thessaly. Beginners will learn robust research methodologies and take inspiration; mature scholars will for sure derive inputs for new research and applications

Extracting structured information from 2D images

Convolutional neural networks can handle an impressive array of supervised learning tasks while relying on a single backbone architecture, suggesting that one solution fits all vision problems. But for many tasks, we can directly make use of the problem structure within neural networks to deliver more accurate predictions. In this thesis, we propose novel deep learning components that exploit the structured output space of an increasingly complex set of problems. We start from Optical Character Recognition (OCR) in natural scenes and leverage the constraints imposed by a spatial outline of letters and language requirements. Conventional OCR systems do not work well in natural scenes due to distortions, blur, or letter variability. We introduce a new attention-based model, equipped with extra information about the neuron positions to guide its focus across characters sequentially. It beats the previous state-of-the-art benchmark by a significant margin. We then turn to dense labeling tasks employing encoder-decoder architectures. We start with an experimental study that documents the drastic impact that decoder design can have on task performance. Rather than optimizing one decoder per task separately, we propose new robust layers for the upsampling of high-dimensional encodings. We show that these better suit the structured per pixel output across the board of all tasks. Finally, we turn to the problem of urban scene understanding. There is an elaborate structure in both the input space (multi-view recordings, aerial and street-view scenes) and the output space (multiple fine-grained attributes for holistic building understanding). We design new models that benefit from a relatively simple cuboidal-like geometry of buildings to create a single unified representation from multiple views. To benchmark our model, we build a new multi-view large-scale dataset of buildings images and fine-grained attributes and show systematic improvements when compared to a broad range of strong CNN-based baselines

Interferometric Synthetic Aperture RADAR and Radargrammetry towards the Categorization of Building Changes

The purpose of this work is the investigation of SAR techniques relying on multi image acquisition for fully automatic and rapid change detection analysis at building level. In particular, the benefits and limitations of a complementary use of two specific SAR techniques, InSAR and radargrammetry, in an emergency context are examined in term of quickness, globality and accuracy. The analysis is performed using spaceborne SAR data

Verbesserte Dokumentation des kulturellen Erbes mithilfe digitaler Photogrammetrie mit sichtbaren und thermischen Bildern von unbemannten Luftfahrzeugen (UAV)

There is always need for reliable and accurate data for documentation of cultural heritage including archaeological areas. The development in 3D data acquisition has let some technologies use for getting a complete documentation. Close range photogrammetry and terrestrial laser scanning are among the most common used techniques which help to get 3D data acquisition, with high level of detail, accuracy and effective results. However, these techniques are not always the most suitable ones for large archaeological areas, yet aerial images may help to provide a general overview of the area which is fundamental for interpretation and documentation of archaeological sites. Because of the limitations in aerial photogrammetry, UAVs (Unmanned Aerial Vehicles) has become an optimal solution for archaeological areas documentation with its potentials in the context of costs and abilities. To cover large areas at different altitudes, to be able to fly at different altitudes, under different weather conditions, to acquire image with high resolution are among the main advantages of this technology which make it usable and preferable for archaeological documentation. Since UAVs have been rapidly improving in sophistication and reliability, its possibilities aid in archaeological research have recently generated much interest, particularly for documenting sites, monuments and excavations. In this case study several aerial surveys will be conducted with a UAV mounted thermal camera on an archaeological area. After acquiring aerial images, they will be processed for producing both color and thermal-imagery in related software. Next step will be the alignment of the images in order to build an accurate and georeferenced 3D and mesh model of surveyed area. Then colored and thermal orthophoto mosaics as well as digital surface model (DSM) will be obtained for the documentation. The datasets of thermal images and color images will be collected and compared in order to detect archaeological remains on and under the ground.Es besteht immer Bedarf an zuverlässigen und genauen Daten für die Dokumentation des kulturellen Erbes, einschließlich archäologischer Gebiete. Die technischen Entwicklungen in der 3D-Datenerfassung haben erst die vollständige Dokumentation ermöglicht. Nahbereichsphotogrammetrie und terrestrisches Laserscanning gehören zu den am häufigsten verwendeten Techniken, die 3D-Datenerfassung mit hohem Detaillierungsgrad, Genauigkeit und effektive Ergebnissen ermöglichen. Diese Techniken sind jedoch nicht immer die am besten geeigneten für große archäologische Gebiete, dennoch können Luftbilder helfen, einen allgemeinen Überblick über das Gebiet zu geben, was für die Interpretation und Dokumentation archäologischer Stätten von grundlegender Bedeutung ist. Aufgrund der Einschränkungen in der Luftbildvermessung sind UAVs (Unmanned Aerial Vehicles) zu einer optimalen Lösung für die archäologische Geländedokumentation mit ihren Potenzialen im Kontext von Kosten und Fähigkeiten geworden. Hauptvorteile dieser Technologie sind u.a. große Gebiete in verschiedenen Höhen abzudecken und unter verschiedenen Wetterbedingungen fliegen zu können, Bilder mit hoher Auflösung aufzunehmen, die dann auch für die archäologische Dokumentation nutzbar und damit auch anderen Verfahren vorzuziehen sind. Da sich die UAVs in Bezug auf Entwicklungsgrad und Zuverlässigkeit rasant verbessert haben, haben ihre Möglichkeiten zur Unterstützung der archäologischen Forschung in jüngster Zeit großes Interesse geweckt, insbesondere bei der Dokumentation von Stätten, Denkmälern und Ausgrabungen. In dieser Fallstudie werden mehrere Kampagnen von Luftaufnahmen mit einer UAV-Wärmebildkamera auf einem archäologischen Gebiet durchgeführt. Nach der Bildaufufnahme die Farb- und Wärmebilder in einer entsprechenden Software verarbeitet. Der nächste Schritt wird die Verknüpfung der Bilder sein, um ein genaues und georeferenziertes 3D- und Netzmodell des vermessenden Bereichs zu erstellen. Anschließend werden farbige und thermische Orthophoto-Mosaike sowie digitale Oberflächenmodelle (DSM) für die Dokumentation abgeleitet. Die Datensätze von Wärme- und Farbbildern werden gesammelt und verglichen, um archäologische Überreste auf und unter dem Boden zu erkennen