Mapping and Real-Time Navigation With Application to Small UAS Urgent Landing

Small Unmanned Aircraft Systems (sUAS) operating in low-altitude airspace require flight near buildings and over people. Robust urgent landing capabilities including landing site selection are needed. However, conventional fixed-wing emergency landing sites such as open fields and empty roadways are rare in cities. This motivates our work to uniquely consider unoccupied flat rooftops as possible nearby landing sites. We propose novel methods to identify flat rooftop buildings, isolate their flat surfaces, and find touchdown points that maximize distance to obstacles. We model flat rooftop surfaces as polygons that capture their boundaries and possible obstructions on them. This thesis offers five specific contributions to support urgent rooftop landing. First, the Polylidar algorithm is developed which enables efficient non-convex polygon extraction with interior holes from 2D point sets. A key insight of this work is a novel boundary following method that contrasts computationally expensive geometric unions of triangles. Results from real-world and synthetic benchmarks show comparable accuracy and more than four times speedup compared to other state-of-the-art methods. Second, we extend polygon extraction from 2D to 3D data where polygons represent flat surfaces and interior holes representing obstacles. Our Polylidar3D algorithm transforms point clouds into a triangular mesh where dominant plane normals are identified and used to parallelize and regularize planar segmentation and polygon extraction. The result is a versatile and extremely fast algorithm for non-convex polygon extraction of 3D data. Third, we propose a framework for classifying roof shape (e.g., flat) within a city. We process satellite images, airborne LiDAR point clouds, and building outlines to generate both a satellite and depth image of each building. Convolutional neural networks are trained for each modality to extract high level features and sent to a random forest classifier for roof shape prediction. This research contributes the largest multi-city annotated dataset with over 4,500 rooftops used to train and test models. Our results show flat-like rooftops are identified with > 90% precision and recall. Fourth, we integrate Polylidar3D and our roof shape prediction model to extract flat rooftop surfaces from archived data sources. We uniquely identify optimal touchdown points for all landing sites. We model risk as an innovative combination of landing site and path risk metrics and conduct a multi-objective Pareto front analysis for sUAS urgent landing in cities. Our proposed emergency planning framework guarantees a risk-optimal landing site and flight plan is selected. Fifth, we verify a chosen rooftop landing site on real-time vertical approach with on-board LiDAR and camera sensors. Our method contributes an innovative fusion of semantic segmentation using neural networks with computational geometry that is robust to individual sensor and method failure. We construct a high-fidelity simulated city in the Unreal game engine with a statistically-accurate representation of rooftop obstacles. We show our method leads to greater than 4% improvement in accuracy for landing site identification compared to using LiDAR only. This work has broad impact for the safety of sUAS in cities as well as Urban Air Mobility (UAM). Our methods identify thousands of additional rooftop landing sites in cities which can provide safe landing zones in the event of emergencies. However, the maps we create are limited by the availability, accuracy, and resolution of archived data. Methods for quantifying data uncertainty or performing real-time map updates from a fleet of sUAS are left for future work.PHDRoboticsUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/170026/1/jdcasta_1.pd

Development of Mining Sector Applications for Emerging Remote Sensing and Deep Learning Technologies

This thesis uses neural networks and deep learning to address practical, real-world problems in the mining sector. The main focus is on developing novel applications in the area of object detection from remotely sensed data. This area has many potential mining applications and is an important part of moving towards data driven strategic decision making across the mining sector. The scientific contributions of this research are twofold; firstly, each of the three case studies demonstrate new applications which couple remote sensing and neural network based technologies for improved data driven decision making. Secondly, the thesis presents a framework to guide implementation of these technologies in the mining sector, providing a guide for researchers and professionals undertaking further studies of this type. The first case study builds a fully connected neural network method to locate supporting rock bolts from 3D laser scan data. This method combines input features from the remote sensing and mobile robotics research communities, generating accuracy scores up to 22% higher than those found using either feature set in isolation. The neural network approach also is compared to the widely used random forest classifier and is shown to outperform this classifier on the test datasets. Additionally, the algorithms’ performance is enhanced by adding a confusion class to the training data and by grouping the output predictions using density based spatial clustering. The method is tested on two datasets, gathered using different laser scanners, in different types of underground mines which have different rock bolting patterns. In both cases the method is found to be highly capable of detecting the rock bolts with recall scores of 0.87-0.96. The second case study investigates modern deep learning for LiDAR data. Here, multiple transfer learning strategies and LiDAR data representations are examined for the task of identifying historic mining remains. A transfer learning approach based on a Lunar crater detection model is used, due to the task similarities between both the underlying data structures and the geometries of the objects to be detected. The relationship between dataset resolution and detection accuracy is also examined, with the results showing that the approach is capable of detecting pits and shafts to a high degree of accuracy with precision and recall scores between 0.80-0.92, provided the input data is of sufficient quality and resolution. Alongside resolution, different LiDAR data representations are explored, showing that the precision-recall balance varies depending on the input LiDAR data representation. The third case study creates a deep convolutional neural network model to detect artisanal scale mining from multispectral satellite data. This model is trained from initialisation without transfer learning and demonstrates that accurate multispectral models can be built from a smaller training dataset when appropriate design and data augmentation strategies are adopted. Alongside the deep learning model, novel mosaicing algorithms are developed both to improve cloud cover penetration and to decrease noise in the final prediction maps. When applied to the study area, the results from this model provide valuable information about the expansion, migration and forest encroachment of artisanal scale mining in southwestern Ghana over the last four years. Finally, this thesis presents an implementation framework for these neural network based object detection models, to generalise the findings from this research to new mining sector deep learning tasks. This framework can be used to identify applications which would benefit from neural network approaches; to build the models; and to apply these algorithms in a real world environment. The case study chapters confirm that the neural network models are capable of interpreting remotely sensed data to a high degree of accuracy on real world mining problems, while the framework guides the development of new models to solve a wide range of related challenges

A Systematic Literature Survey of Unmanned Aerial Vehicle Based Structural Health Monitoring

Unmanned Aerial Vehicles (UAVs) are being employed in a multitude of civil applications owing to their ease of use, low maintenance, affordability, high-mobility, and ability to hover. UAVs are being utilized for real-time monitoring of road traffic, providing wireless coverage, remote sensing, search and rescue operations, delivery of goods, security and surveillance, precision agriculture, and civil infrastructure inspection. They are the next big revolution in technology and civil infrastructure, and it is expected to dominate more than $45 billion market value. The thesis surveys the UAV assisted Structural Health Monitoring or SHM literature over the last decade and categorize UAVs based on their aerodynamics, payload, design of build, and its applications. Further, the thesis presents the payload product line to facilitate the SHM tasks, details the different applications of UAVs exploited in the last decade to support civil structures, and discusses the critical challenges faced in UASHM applications across various domains. Finally, the thesis presents two artificial neural network-based structural damage detection models and conducts a detailed performance evaluation on multiple platforms like edge computing and cloud computing

Deep Learning Methods for Remote Sensing

Remote sensing is a field where important physical characteristics of an area are exacted using emitted radiation generally captured by satellite cameras, sensors onboard aerial vehicles, etc. Captured data help researchers develop solutions to sense and detect various characteristics such as forest fires, flooding, changes in urban areas, crop diseases, soil moisture, etc. The recent impressive progress in artificial intelligence (AI) and deep learning has sparked innovations in technologies, algorithms, and approaches and led to results that were unachievable until recently in multiple areas, among them remote sensing. This book consists of sixteen peer-reviewed papers covering new advances in the use of AI for remote sensing

Machine learning, classification of 3D UAV-SFM point clouds in the University of KwaZulu-Natal (Howard College)

Masters Degrees. University of KwaZulu- Natal, Durban.Three-dimensional (3D) point clouds derived using cost-effective and time-efficient photogrammetric technologies can provide information that can be utilized for decision-making in engineering, built environment and other related fields. This study focuses on the use of machine learning to automate the classification of points in a heterogeneous 3D scene situated in the University of KwaZulu-Natal, Howard College Campus sports field. The state of the camera mounted on the unmanned aerial vehicle (UAV) was evaluated through the process of camera calibration. Nadir aerial images captured using a UAV were used to generate a 3D point cloud employing the structure-from-motion (SfM) photogrammetric technique. The generated point cloud was georeferenced using natural ground control points (GCPs). Supervised and unsupervised classification approaches were used to classify points into three classes: ground, high vegetation and building. The supervised classification algorithm used a multi-scale dimensionality analysis to classify points. A georeferenced orthomosaic was used to generate random points for cross-validation. The accuracy of classification was evaluated, employing both qualitative and quantitative analysis. The camera calibration results showed negligible discrepancies when a comparison was made between the results obtained and the manufacturer’s specifications in parameters of the camera lens; hence the camera was in the excellent state of being used as a measuring device. Site visits and ground truth surveys were conducted to validate the classified point cloud. An overall root-mean-square (RMS) error of 0.053m was achieved from georeferencing the 3D point cloud. A root-mean-square error of 0.032m was achieved from georeferencing the orthomosaic. The multi-scale dimensionality analysis classified a point cloud and achieved an accuracy of 81.3% and a Kappa coefficient of 0.70. Good results were also achieved from the qualitative analysis. The classification results obtained indicated that a 3D heterogeneous scene can be classified into different land cover categories. These results show that the classification of 3D UAV-SfM point clouds provides a helpful tool for mapping and monitoring complex 3D environments

Creation and Spatial Analysis of 3D City Modeling based on GIS Data

The 3D city model is one of the crucial topics that are still under analysis by many engineers and programmers because of the great advancements in data acquisition technologies and 3D computer graphics programming. It is one of the best visualization methods for representing reality. This paper presents different techniques for the creation and spatial analysis of 3D city modeling based on Geographical Information System (GIS) technology using free data sources. To achieve that goal, the Mansoura University campus, located in Mansoura city, Egypt, was chosen as a case study. The minimum data requirements to generate a 3D city model are the terrain, 2D spatial features such as buildings, landscape area and street networks. Moreover, building height is an important attribute in the 3D extrusion process. The main challenge during the creation process is the dearth of accurate free datasets, and the time-consuming editing. Therefore, different data sources are used in this study to evaluate their accuracy and find suitable applications which can use the generated 3D model. Meanwhile, an accurate data source obtained using the traditional survey methods is used for the validation purpose. First, the terrain was obtained from a digital elevation model (DEM) and compared with grid leveling measurements. Second, 2D data were obtained from: the manual digitization from (30 cm) high-resolution imagery, and deep learning structure algorithms to detect the 2D features automatically using an object instance segmentation model and compared the results with the total station survey observations. Different techniques are used to investigate and evaluate the accuracy of these data sources. The procedural modeling technique is applied to generate the 3D city model. TensorFlow & Keras frameworks (Python APIs) were used in this paper; moreover, global mapper, ArcGIS Pro, QGIS and CityEngine software were used. The precision metrics from the trained deep learning model were 0.78 for buildings, 0.62 for streets and 0.89 for landscape areas. Despite, the manual digitizing results are better than the results from deep learning, but the extracted features accuracy is accepted and can be used in the creation process in the cases not require a highly accurate 3D model. The flood impact scenario is simulated as an application of spatial analysis on the generated 3D city model. Doi: 10.28991/CEJ-2022-08-01-08 Full Text: PD

VGC 2023 - Unveiling the dynamic Earth with digital methods: 5th Virtual Geoscience Conference: Book of Abstracts

Conference proceedings of the 5th Virtual Geoscience Conference, 21-22 September 2023, held in Dresden. The VGC is a multidisciplinary forum for researchers in geoscience, geomatics and related disciplines to share their latest developments and applications.:Short Courses 9 Workshops Stream 1 10 Workshop Stream 2 11 Workshop Stream 3 12 Session 1 – Point Cloud Processing: Workflows, Geometry & Semantics 14 Session 2 – Visualisation, communication & Teaching 27 Session 3 – Applying Machine Learning in Geosciences 36 Session 4 – Digital Outcrop Characterisation & Analysis 49 Session 5 – Airborne & Remote Mapping 58 Session 6 – Recent Developments in Geomorphic Process and Hazard Monitoring 69 Session 7 – Applications in Hydrology & Ecology 82 Poster Contributions 9

Benchmark aplicado à Deteção de Objetos de Mamoas Arqueológicas a partir de dados LiDAR

Human history and its archaeological evidence are priceless and should be preserved, esteemed and respected. However, the traditional work of an archaeologist is mainly manual labour, sluggish and requires specialized knowledge as well as considerable experience, which represents quite a limitation due to the available community of archaeologists. Besides this fact, concerns about global warming, the generalized rise of sea levels or destruction due to human activities, among others, contribute to a growing fear of losing some archaeological sites as the traditional method of identification and preservation of these sites can’t keep up with the propagation speed of such problems. Because of this, a growing willingness to implement Artificial Intelligence techniques has been evidenced, which allows some help to the archaeologist in several tasks, with particular focus to archaeological sitting identification, through remote detection. Currently, there are no applications or tools that can execute such work, however, there has been a growing effort in studies and work on a scientific and academic level. This thesis aims to implement a tool that, through LiDAR data readings, gathered from some geographical area, can perform object detection to specific archaeological findings (such as mounds), testing a variety of machine learning models to, assigning a classification, determine if it’s in the presence of an archaeological mound. The input of the work done for this thesis consists of a Digital Terrain Model (DTM), a Local Relief Model (LRM) and a Slope obtained from drone flights over Viana do Castelo, with the use of LiDAR sensors. The combination of these three images was processed to achieve a single image with higher identification of certain features for future model training. For comparison reasons, two datasets were built with different margin sizes around each archaeological mound. The goal of the thesis is to perform tests on some object detection architectures, compare the efficiency of their evaluations and be able to determine which of the tested models performs a better prediction result on detecting the presence of an archaeological mound. This study was able to perform the comparison of a total of nine Deep Learning (DL) architectures, testing four two-stage detectors and five one-stage detectors. As expected, most of the two-stage detectors outperformed the one-stage detectors in terms of mean average precision for the detection of archaeological mounds, except for the one stage detector Fully Convolutional One-Stage (FCOS), which achieved the highest mean average precision from all, showing results between 68.1% to 78.6% for both size dataset.A história da humanidade e as suas evidências arqueológicas são inestimáveis e devem ser preservadas, respeitadas e valorizadas. No entanto, o trabalho tradicional de um arqueólogo é principalmente uma tarefa manual, lenta e requer conhecimento especializado, bem como considerável experiência, o que representa uma limitação significativa devido à disponibilidade limitada de arqueólogos. Além disso, preocupações com o aquecimento global, o aumento generalizado do nível do mar ou a destruição devido a atividades humanas, entre outras, contribuem para um crescente receio de perder alguns sítios arqueológicos, já que o método tradicional de identificação e preservação desses sítios não consegue acompanhar a velocidade de propagação de tais problemas. Decorrente destes factos, aliado a uma tendência generalizada e com sucesso no recurso a técnicas de Inteligência Artificial em outras especialidades, também na Arqueologia tem-se vindo a verificar uma adesão significativa. A adoção de técnicas de Inteligência Artificial tem permitido alguma ajuda aos arqueólogos em várias tarefas, com especial foco na identificação de sítios arqueológicos através do recurso a métodos de deteção remota. Atualmente, não existem aplicações ou ferramentas que possam executar este trabalho, no entanto, tem-se verificado um esforço crescente de estudo e desenvolvimento de trabalho nesse sentido, quer ao nível académico quer científico. Esta tese tem como objetivo implementar uma ferramenta que, através da leitura de dados LiDAR, coletados de uma determinada área geográfica, consiga efetuar uma deteção de objetos referentes a vestígios arqueológicos específicos (mamoas), recorrendo a uma variedade de modelos de machine learning, atribuindo uma classificação para determinar se identificou ou não com sucesso a presença de uma mamoa. O ponto de partida do trabalho realizado nesta tese inicia-se com o acesso e trabalho realizado sobre três técnicas de visualização aplicada sobre dados LiDAR, nomeadamente consiste no acesso a ficheiros como Digital Terrain Model (DTM), Local Relief Model (LRM) e Slope. Estes dados LiDAR e consequente conversão nas técnicas de visualização anteriormente citadas ocorreram a partir de voos de drones, equipados com sensores LiDAR que, sobrevoando a zona de Viana do Castelo, proporcionou a obtenção de tais dados. Adicionalmente aos três ficheiros de técnicas de visualização, foi também disponibilizado um ficheiro shape que fornece informação georreferenciada da localização de mamoas na área sobrevoada pelos drones. Com recurso ao software QGIS, foi possível identificar que, as localizações das mamoas encontravam-se relativamente concentradas numa parte específica das imagens. Desta forma, e considerando o tamanho dos ficheiros em questão, efetuou-se uma seleção nas imagens, cortando áreas que já apresentassem uma distância considerável da mamoa mais próxima, de forma a tornar mais ágil o processo de trabalho e treino dos modelos escolhidos. Posteriormente, e com as imagens em tamanho mais reduzido, efetuou-se uma operação de combinação entre as três tipologias de imagens, obtendo uma única imagem onde, incorporando as características destas, permitiu realçar determinados aspetos com intuito de, posteriormente, auxiliar nas tarefas de treino e teste dos modelos de aprendizagem profunda a que foram aplicados. Seguiu-se o processo de pré-processamento de dados tendo sido definido e trabalhado um programa que executasse a mesma tarefa, fornecendo como output um dataset em formato COCO, formato escolhido dada popularidade e sucesso verificado na aplicação a métodos de deteção de objetos. A construção deste dataset foi igualmente realizada de forma a criar estrutura de ficheiros que, respeitando na mesma o formato COCO, proporcionasse a aplicação da técnica de leave-one-out cross-validation, uma vez que, foi considerado a melhor opção dada existência de apenas 77 mamoas, de forma a evitar cenários de enviesamento de dados ou até overfitting. Para diversificar e enriquecer esta análise comparativa, foram criados dois datasets diferentes, cujas bounding boxes em volta das mamoas apresentavam tamanhos diferentes, nomeadamente 15x15 metros e 30x30 metros. Como o objetivo da tese é a realização de testes em algumas arquiteturas de deteção de objeto, foi utilizada um projeto que, está precisamente preparado e desenvolvido para a realização de análises de benchmark, de várias metodologias de classificação de imagem, nas quais estão incluídas as de deteção de objeto. Esta biblioteca permitiu a realização do estudo comparativo não só entre as arquiteturas analisadas e identificadas como as mais promissoras e populares na análise de estado de arte, como ainda permitiu a comparação com outras arquiteturas dada a variedade de oferta de modelos que a mesma proporcionava. Este estudo conseguiu realizar a comparação com um total de nove arquiteturas de aprendizagem profunda, testando quatro detetores two-stage e cinco detetores one-stage. Como era esperado, a maioria dos detetores two-stage superou os detetores one-stage em termos de precisão média de deteção de mamoas, com exceção do modelo Fully Convolutional One-Stage (FCOS), que alcançou a maior precisão média de todos os modelos testados, apresentando resultados entre 68,1% e 78,6% em ambos os datasets. Igualmente esperado foi a confirmação do modelo one-stage Single Shot Detector (SSD) como sendo o modelo com mais rápido tempo de processamento de treino, apesar de, entre os restantes modelos, a diferença de tempo já ser menos significativa e não se notar uma supremacia dos modelos one-stage como seria inicialmente esperado