Label Efficient 3D Scene Understanding

3D scene understanding models are becoming increasingly integrated into modern society. With applications ranging from autonomous driving, Augmented Real- ity, Virtual Reality, robotics and mapping, the demand for well-behaved models is rapidly increasing. A key requirement for training modern 3D models is high- quality manually labelled training data. Collecting training data is often the time and monetary bottleneck, limiting the size of datasets. As modern data-driven neu- ral networks require very large datasets to achieve good generalisation, finding al- ternative strategies to manual labelling is sought after for many industries. In this thesis, we present a comprehensive study on achieving 3D scene under- standing with fewer labels. Specifically, we evaluate 4 approaches: existing data, synthetic data, weakly-supervised and self-supervised. Existing data looks at the potential of using readily available national mapping data as coarse labels for train- ing a building segmentation model. We further introduce an energy-based active contour snake algorithm to improve label quality by utilising co-registered LiDAR data. This is attractive as whilst the models may still require manual labels, these labels already exist. Synthetic data also exploits already existing data which was not originally designed for training neural networks. We demonstrate a pipeline for generating a synthetic Mobile Laser Scanner dataset. We experimentally evalu- ate if such a synthetic dataset can be used to pre-train smaller real-world datasets, increasing the generalisation with less data. A weakly-supervised approach is presented which allows for competitive per- formance on challenging real-world benchmark 3D scene understanding datasets with up to 95% less data. We propose a novel learning approach where the loss function is learnt. Our key insight is that the loss function is a local function and therefore can be trained with less data on a simpler task. Once trained our loss function can be used to train a 3D object detector using only unlabelled scenes. Our method is both flexible and very scalable, even performing well across datasets. Finally, we propose a method which only requires a single geometric represen- tation of each object class as supervision for 3D monocular object detection. We discuss why typical L2-like losses do not work for 3D object detection when us- ing differentiable renderer-based optimisation. We show that the undesirable local- minimas that the L2-like losses fall into can be avoided with the inclusion of a Generative Adversarial Network-like loss. We achieve state-of-the-art performance on the challenging 6DoF LineMOD dataset, without any scene level labels

Development of Mining Sector Applications for Emerging Remote Sensing and Deep Learning Technologies

This thesis uses neural networks and deep learning to address practical, real-world problems in the mining sector. The main focus is on developing novel applications in the area of object detection from remotely sensed data. This area has many potential mining applications and is an important part of moving towards data driven strategic decision making across the mining sector. The scientific contributions of this research are twofold; firstly, each of the three case studies demonstrate new applications which couple remote sensing and neural network based technologies for improved data driven decision making. Secondly, the thesis presents a framework to guide implementation of these technologies in the mining sector, providing a guide for researchers and professionals undertaking further studies of this type. The first case study builds a fully connected neural network method to locate supporting rock bolts from 3D laser scan data. This method combines input features from the remote sensing and mobile robotics research communities, generating accuracy scores up to 22% higher than those found using either feature set in isolation. The neural network approach also is compared to the widely used random forest classifier and is shown to outperform this classifier on the test datasets. Additionally, the algorithms’ performance is enhanced by adding a confusion class to the training data and by grouping the output predictions using density based spatial clustering. The method is tested on two datasets, gathered using different laser scanners, in different types of underground mines which have different rock bolting patterns. In both cases the method is found to be highly capable of detecting the rock bolts with recall scores of 0.87-0.96. The second case study investigates modern deep learning for LiDAR data. Here, multiple transfer learning strategies and LiDAR data representations are examined for the task of identifying historic mining remains. A transfer learning approach based on a Lunar crater detection model is used, due to the task similarities between both the underlying data structures and the geometries of the objects to be detected. The relationship between dataset resolution and detection accuracy is also examined, with the results showing that the approach is capable of detecting pits and shafts to a high degree of accuracy with precision and recall scores between 0.80-0.92, provided the input data is of sufficient quality and resolution. Alongside resolution, different LiDAR data representations are explored, showing that the precision-recall balance varies depending on the input LiDAR data representation. The third case study creates a deep convolutional neural network model to detect artisanal scale mining from multispectral satellite data. This model is trained from initialisation without transfer learning and demonstrates that accurate multispectral models can be built from a smaller training dataset when appropriate design and data augmentation strategies are adopted. Alongside the deep learning model, novel mosaicing algorithms are developed both to improve cloud cover penetration and to decrease noise in the final prediction maps. When applied to the study area, the results from this model provide valuable information about the expansion, migration and forest encroachment of artisanal scale mining in southwestern Ghana over the last four years. Finally, this thesis presents an implementation framework for these neural network based object detection models, to generalise the findings from this research to new mining sector deep learning tasks. This framework can be used to identify applications which would benefit from neural network approaches; to build the models; and to apply these algorithms in a real world environment. The case study chapters confirm that the neural network models are capable of interpreting remotely sensed data to a high degree of accuracy on real world mining problems, while the framework guides the development of new models to solve a wide range of related challenges

Review of Automatic Processing of Topography and Surface Feature Identification LiDAR Data Using Machine Learning Techniques

Machine Learning (ML) applications on Light Detection And Ranging (LiDAR) data have provided promising results and thus this topic has been widely addressed in the literature during the last few years. This paper reviews the essential and the more recent completed studies in the topography and surface feature identification domain. Four areas, with respect to the suggested approaches, have been analyzed and discussed: the input data, the concepts of point cloud structure for applying ML, the ML techniques used, and the applications of ML on LiDAR data. Then, an overview is provided to underline the advantages and the disadvantages of this research axis. Despite the training data labelling problem, the calculation cost, and the undesirable shortcutting due to data downsampling, most of the proposed methods use supervised ML concepts to classify the downsampled LiDAR data. Furthermore, despite the occasional highly accurate results, in most cases the results still require filtering. In fact, a considerable number of adopted approaches use the same data structure concepts employed in image processing to profit from available informatics tools. Knowing that the LiDAR point clouds represent rich 3D data, more effort is needed to develop specialized processing tools

Delving Deep into the Sketch and Photo Relation

"Sketches drawn by humans can play a similar role to photos in terms of conveying shape, posture as well as fine-grained information, and this fact has stimulated one line of cross-domain research that is related to sketch and photo, including sketch-based photo synthesis and retrieval. In this thesis, we aim to further investigate the relationship between sketch and photo. More specifically, we study certain under- explored traits in this relationship, and propose novel applications to reinforce the understanding of sketch and photo relation.Our exploration starts with the problem of sketch-based photo synthesis, where the unique trait of non-rigid alignment between sketch and photo is overlooked in existing research. We then carry on with our investigation from a new angle to study whether sketch can facilitate photo classifier generation. Building upon this, we continue to explore how sketch and photo are linked together on a more fine-grained level by tackling with the sketch-based photo segmenter prediction. Furthermore, we address the data scarcity issue identified in nearly all sketch-photo-related applications by examining their inherent correlation in the semantic aspect using sketch-based image retrieval (SBIR) as a test-bed. In general, we make four main contributions to the research on relationship between sketch and photo.Firstly, to mitigate the effect of deformation in sketch-based photo synthesis, we introduce the spatial transformer network to our image-image regression framework, which subtly deals with non-rigid alignment between the sketches and photos. The qualitative and quantitative experiments consistently reveal the superior quality of our synthesised photos over those generated by existing approaches.Secondly, sketch-based photo classifier generation is achieved with a novel model regression network, which maps the sketch to the parameters of photo classification model. It is shown that our model regression network is able to generalise across categories and photo classifiers for novel classes not involved in training are just a sketch away. Comprehensive experiments illustrate the promising performance of the generated binary and multi-class photo classifiers, and demonstrate that sketches can also be employed to enhance the granularity of existing photo classifiers.Thirdly, to achieve the goal of sketch-based photo segmentation, we propose a photo segmentation model generation algorithm that predicts the weights of a deep photo segmentation network according to the input sketch. The results confirm that one single sketch is the only prerequisite for unseen category photo segmentation, and the segmentation performance can be further improved by utilising sketch that is aligned with the object to be segmented in shape and position.Finally, we present an unsupervised representation learning framework for SBIR, the purpose of which is to eliminate the barrier imposed by data annotation scarcity. Prototype and memory bank reinforced joint distribution optimal transport is integrated into the unsupervised representation learning framework, so that the mapping between the sketches and photos could be automatically detected to learn a semantically meaningful yet domain-agnostic feature space. Extensive experiments and feature visualisation validate the efficacy of our proposed algorithm.

A review of technical factors to consider when designing neural networks for semantic segmentation of Earth Observation imagery

Semantic segmentation (classification) of Earth Observation imagery is a crucial task in remote sensing. This paper presents a comprehensive review of technical factors to consider when designing neural networks for this purpose. The review focuses on Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Generative Adversarial Networks (GANs), and transformer models, discussing prominent design patterns for these ANN families and their implications for semantic segmentation. Common pre-processing techniques for ensuring optimal data preparation are also covered. These include methods for image normalization and chipping, as well as strategies for addressing data imbalance in training samples, and techniques for overcoming limited data, including augmentation techniques, transfer learning, and domain adaptation. By encompassing both the technical aspects of neural network design and the data-related considerations, this review provides researchers and practitioners with a comprehensive and up-to-date understanding of the factors involved in designing effective neural networks for semantic segmentation of Earth Observation imagery.Comment: 145 pages with 32 figure

Drone-based Integration of Hyperspectral Imaging and Magnetics for Mineral Exploration

The advent of unoccupied aerial systems (UAS) as disruptive technology has a lasting impact on remote sensing, geophysics and most geosciences. Small, lightweight, and low-cost UAS enable researchers and surveyors to acquire earth observation data in higher spatial and spectral resolution as compared to airborne and satellite data. UAS-based applications range from rapid topographic mapping using photogrammetric techniques to hyperspectral and geophysical measurements of surface and subsurface geology. UAS surveys contribute to identifying metal deposits, monitoring of mine sites and can reveal arising environmental issues associated with mining. Further, affordable UAS technology will boost exploration data availability and expertise in the global south. This thesis investigates the application of UAS-based multi-sensor data for mineral exploration, in particular the integration of hyperspectral imagers, magnetometers and digital cameras (covering the visible red, green, blue light spectrum). UAS-based research is maturing, however the aforementioned methods are not unified effectively. RGB-based photogrammetry is used to investigate topography and surface texture. Image spectrometers measure mineral-specific surface signatures. Magnetometers detect geomagnetic field changes caused by magnetic minerals at surface and depth. The integration of such UAS sensor-based methods in this thesis augments exploration potential with non-invasive, high-resolution, safe, rapid and practical survey methods. UAS-based surveying acquired, processed and integrated data from three distinct test sites. The sites are located in Finland (Fe-Ti-V at Otanmäki; apatite at Siilinjärvi) and Greenland (Ni-Cu-PGE at Qullissat, Disko Island) and were chosen as geologically diverse areas in subarctic to arctic environments. Restricted accessibility, unfavourable atmospheric conditions, dark rocks, debris and vegetation cover and low solar illumination were common features. While the topography in Finland was moderately flat, a steep landscape challenged the Greenland field work. These restraints meant that acquisitions varied from site to site and how data was integrated and interpreted is dependent on the commodity of interest. Iron-based spectral absorption and magnetic mineral response were detected using hyperspectral and magnetic surveying in Otanmäki. Multi-sensor-based image feature detection and classification combined with magnetic forward modelling enabled seamless geologic mapping in Siilinjärvi. Detailed magnetic inversion and multispectral photogrammetry led to the construction of a comprehensive 3D model of magmatic exploration targets in Greenland. Ground truth at different intensity was employed to verify UAS-based data interpretations during all case studies. Laboratory analysis was applied when deemed necessary to acquire geologic-mineralogic validation (e.g., X-ray diffraction and optical microscopy for mineral identification to establish lithologic domains, magnetic susceptibility measurements for subsurface modelling), for example for trace amounts of magnetite in carbonatite (Siilinjärvi) and native iron occurrence in basalt (Qullissat). Technical achievements were the integration of a multicopter-based prototype fluxgate-magnetometer data from different survey altitudes with ground truth, and a feasibility study with a high-speed multispectral image system for fixed-wing UAS. The employed case studies transfer the experiences made towards general recommendations for UAS application-based multi-sensor integration. This thesis highlights the feasibility of UAS-based surveying at target scale (1–50 km2) and solidifies versatile survey approaches for multi-sensor integration.Ziel dieser Arbeit war es, das Potenzial einer Drohnen-basierten Mineralexploration mit Multisensor-Datenintegration unter Verwendung optisch-spektroskopischer und magnetischer Methoden zu untersuchen, um u. a. übertragbare Arbeitsabläufe zu erstellen. Die untersuchte Literatur legt nahe, dass Drohnen-basierte Bildspektroskopie und magnetische Sensoren ein ausgereiftes technologisches Niveau erreichen und erhebliches Potenzial für die Anwendungsentwicklung bieten, aber es noch keine ausreichende Synergie von hyperspektralen und magnetischen Methoden gibt. Diese Arbeit umfasste drei Fallstudien, bei denen die Drohnengestützte Vermessung von geologischen Zielen in subarktischen bis arktischen Regionen angewendet wurde. Eine Kombination von Drohnen-Technologie mit RGB, Multi- und Hyperspektralkameras und Magnetometern ist vorteilhaft und schuf die Grundlage für eine integrierte Modellierung in den Fallstudien. Die Untersuchungen wurden in einem Gelände mit flacher und zerklüfteter Topografie, verdeckten Zielen und unter oft schlechten Lichtverhältnissen durchgeführt. Unter diesen Bedingungen war es das Ziel, die Anwendbarkeit von Drohnen-basierten Multisensordaten in verschiedenen Explorationsumgebungen zu bewerten. Hochauflösende Oberflächenbilder und Untergrundinformationen aus der Magnetik wurden fusioniert und gemeinsam interpretiert, dabei war eine selektive Gesteinsprobennahme und Analyse ein wesentlicher Bestandteil dieser Arbeit und für die Validierung notwendig. Für eine Eisenerzlagerstätte wurde eine einfache Ressourcenschätzung durchgeführt, indem Magnetik, bildspektroskopisch-basierte Indizes und 2D-Strukturinterpretation integriert wurden. Fotogrammetrische 3D-Modellierung, magnetisches forward-modelling und hyperspektrale Klassifizierungen wurden für eine Karbonatit-Intrusion angewendet, um einen kompletten Explorationsabschnitt zu erfassen. Eine Vektorinversion von magnetischen Daten von Disko Island, Grönland, wurden genutzt, um großräumige 3D-Modelle von undifferenzierten Erdrutschblöcken zu erstellen, sowie diese zu identifizieren und zu vermessen. Die integrierte spektrale und magnetische Kartierung in komplexen Gebieten verbesserte die Erkennungsrate und räumliche Auflösung von Erkundungszielen und reduzierte Zeit, Aufwand und benötigtes Probenmaterial für eine komplexe Interpretation. Der Prototyp einer Multispektralkamera, gebaut für eine Starrflügler-Drohne für die schnelle Vermessung, wurde entwickelt, erfolgreich getestet und zum Teil ausgewertet. Die vorgelegte Arbeit zeigt die Vorteile und Potenziale von Multisensor-Drohnen als praktisches, leichtes, sicheres, schnelles und komfortabel einsetzbares geowissenschaftliches Werkzeug, um digitale Modelle für präzise Rohstofferkundung und geologische Kartierung zu erstellen

La Détection des changements tridimensionnels à l'aide de nuages de points : Une revue

peer reviewedChange detection is an important step for the characterization of object dynamics at the earth’s surface. In multi-temporal point clouds, the main challenge is to detect true changes at different granularities in a scene subject to significant noise and occlusion. To better understand new research perspectives in this field, a deep review of recent advances in 3D change detection methods is needed. To this end, we present a comprehensive review of the state of the art of 3D change detection approaches, mainly those using 3D point clouds. We review standard methods and recent advances in the use of machine and deep learning for change detection. In addition, the paper presents a summary of 3D point cloud benchmark datasets from different sensors (aerial, mobile, and static), together with associated information. We also investigate representative evaluation metrics for this task. To finish, we present open questions and research perspectives. By reviewing the relevant papers in the field, we highlight the potential of bi- and multi-temporal point clouds for better monitoring analysis for various applications.11. Sustainable cities and communitie

FCP-Net: A Feature-Compression-Pyramid Network Guided by Game-Theoretic Interactions for Medical Image Segmentation

Medical image segmentation is a crucial step in diagnosis and analysis of diseases for clinical applications. Deep neural network methods such as DeepLabv3+ have successfully been applied for medical image segmentation, but multi-level features are seldom integrated seamlessly into different attention mechanisms, and few studies have explored the interactions between medical image segmentation and classification tasks. Herein, we propose a feature-compression-pyramid network (FCP-Net) guided by game-theoretic interactions with a hybrid loss function (HLF) for the medical image segmentation. The proposed approach consists of segmentation branch, classification branch and interaction branch. In the encoding stage, a new strategy is developed for the segmentation branch by applying three modules, e.g., embedded feature ensemble, dilated spatial mapping and channel attention (DSMCA), and branch layer fusion. These modules allow effective extraction of spatial information, efficient identification of spatial correlation among various features, and fully integration of multireceptive field features from different branches. In the decoding stage, a DSMCA module and a multi-scale feature fusion module are used to establish multiple skip connections for enhancing fusion features. Classification and interaction branches are introduced to explore the potential benefits of the classification information task to the segmentation task. We further explore the interactions of segmentation and classification branches from a game theoretic view, and design an HLF. Based on this HLF, the segmentation, classification and interaction branches can collaboratively learn and teach each other throughout the training process, thus applying the conjoint information between the segmentation and classification tasks and improving the generalization performance. The proposed model has been evaluated using several datasets, including ISIC2017, ISIC2018, REFUGE, Kvasir-SEG, BUSI, and PH2, and the results prove its competitiveness compared with other state-of-the-art techniques

UAVs for the Environmental Sciences

This book gives an overview of the usage of UAVs in environmental sciences covering technical basics, data acquisition with different sensors, data processing schemes and illustrating various examples of application