Development of Mining Sector Applications for Emerging Remote Sensing and Deep Learning Technologies

This thesis uses neural networks and deep learning to address practical, real-world problems in the mining sector. The main focus is on developing novel applications in the area of object detection from remotely sensed data. This area has many potential mining applications and is an important part of moving towards data driven strategic decision making across the mining sector. The scientific contributions of this research are twofold; firstly, each of the three case studies demonstrate new applications which couple remote sensing and neural network based technologies for improved data driven decision making. Secondly, the thesis presents a framework to guide implementation of these technologies in the mining sector, providing a guide for researchers and professionals undertaking further studies of this type. The first case study builds a fully connected neural network method to locate supporting rock bolts from 3D laser scan data. This method combines input features from the remote sensing and mobile robotics research communities, generating accuracy scores up to 22% higher than those found using either feature set in isolation. The neural network approach also is compared to the widely used random forest classifier and is shown to outperform this classifier on the test datasets. Additionally, the algorithms’ performance is enhanced by adding a confusion class to the training data and by grouping the output predictions using density based spatial clustering. The method is tested on two datasets, gathered using different laser scanners, in different types of underground mines which have different rock bolting patterns. In both cases the method is found to be highly capable of detecting the rock bolts with recall scores of 0.87-0.96. The second case study investigates modern deep learning for LiDAR data. Here, multiple transfer learning strategies and LiDAR data representations are examined for the task of identifying historic mining remains. A transfer learning approach based on a Lunar crater detection model is used, due to the task similarities between both the underlying data structures and the geometries of the objects to be detected. The relationship between dataset resolution and detection accuracy is also examined, with the results showing that the approach is capable of detecting pits and shafts to a high degree of accuracy with precision and recall scores between 0.80-0.92, provided the input data is of sufficient quality and resolution. Alongside resolution, different LiDAR data representations are explored, showing that the precision-recall balance varies depending on the input LiDAR data representation. The third case study creates a deep convolutional neural network model to detect artisanal scale mining from multispectral satellite data. This model is trained from initialisation without transfer learning and demonstrates that accurate multispectral models can be built from a smaller training dataset when appropriate design and data augmentation strategies are adopted. Alongside the deep learning model, novel mosaicing algorithms are developed both to improve cloud cover penetration and to decrease noise in the final prediction maps. When applied to the study area, the results from this model provide valuable information about the expansion, migration and forest encroachment of artisanal scale mining in southwestern Ghana over the last four years. Finally, this thesis presents an implementation framework for these neural network based object detection models, to generalise the findings from this research to new mining sector deep learning tasks. This framework can be used to identify applications which would benefit from neural network approaches; to build the models; and to apply these algorithms in a real world environment. The case study chapters confirm that the neural network models are capable of interpreting remotely sensed data to a high degree of accuracy on real world mining problems, while the framework guides the development of new models to solve a wide range of related challenges

INTELLIGENT ROAD MAINTENANCE: A MACHINE LEARNING APPROACH FOR SURFACE DEFECT DETECTION

The emergence of increased sources for Big Data through consumer recording devices gives rise to a new basis for the management and governance of public infrastructures and policy de-sign. Road maintenance and detection of road surface defects, such as cracks, have traditionally been a time consuming and manual process. Lately, increased automation using easily acquirable front-view digital natural scene images is seen to be an alternative for taking timely maintenance decisions; reducing accidents and operating cost and increasing public safety. In this paper, we propose a machine learning based approach to handle the challenge of crack and related defect detection on road surfaces using front-view images captured from driver’s viewpoint under diverse conditions. We use a superpixel based method to first process the road images into smaller coherent image regions. These superpixels are then classified into crack and non-crack regions. Various texture-based features are combined for the classification mod-el. Classifiers such as Gradient Boosting, Artificial Neural Network, Random Forest and Linear Support Vector Machines are evaluated for the task. Evaluations on real datasets show that the approach successfully handles different road surface conditions and crack-types, while locating the defective regions in the scene images

A Systematic Review of Convolutional Neural Network-Based Structural Condition Assessment Techniques

With recent advances in non-contact sensing technology such as cameras, unmanned aerial and ground vehicles, the structural health monitoring (SHM) community has witnessed a prominent growth in deep learning-based condition assessment techniques of structural systems. These deep learning methods rely primarily on convolutional neural networks (CNNs). The CNN networks are trained using a large number of datasets for various types of damage and anomaly detection and post-disaster reconnaissance. The trained networks are then utilized to analyze newer data to detect the type and severity of the damage, enhancing the capabilities of non-contact sensors in developing autonomous SHM systems. In recent years, a broad range of CNN architectures has been developed by researchers to accommodate the extent of lighting and weather conditions, the quality of images, the amount of background and foreground noise, and multiclass damage in the structures. This paper presents a detailed literature review of existing CNN-based techniques in the context of infrastructure monitoring and maintenance. The review is categorized into multiple classes depending on the specific application and development of CNNs applied to data obtained from a wide range of structures. The challenges and limitations of the existing literature are discussed in detail at the end, followed by a brief conclusion on potential future research directions of CNN in structural condition assessment

Intelligent Industrial Cleaning: A Multi-Sensor Approach Utilising Machine Learning-Based Regression

Effectively cleaning equipment is essential for the safe production of food but requires a significant amount of time and resources such as water, energy, and chemicals. To optimize the cleaning of food production equipment, there is the need for innovative technologies to monitor the removal of fouling from equipment surfaces. In this work, optical and ultrasonic sensors are used to monitor the fouling removal of food materials with different physicochemical properties from a benchtop rig. Tailored signal and image processing procedures are developed to monitor the cleaning process, and a neural network regression model is developed to predict the amount of fouling remaining on the surface. The results show that the three dissimilar food fouling materials investigated were removed from the test section via different cleaning mechanisms, and the neural network models were able to predict the area and volume of fouling present during cleaning with accuracies as high as 98% and 97%, respectively. This work demonstrates that sensors and machine learning methods can be effectively combined to monitor cleaning processes

Model based 3D vision synthesis and analysis for production audit of installations.

PhDOne of the challenging problems in the aerospace industry is to design an automated 3D vision system that can sense the installation components in an assembly environment and check certain safety constraints are duly respected. This thesis describes a concept application to aid a safety engineer to perform an audit of a production aircraft against safety driven installation requirements such as segregation, proximity, orientation and trajectory. The capability is achieved using the following steps. The initial step is to perform image capture of a product and measurement of distance between datum points within the product with/without reference to a planar surface. This provides the safety engineer a means to perform measurements on a set of captured images of the equipment they are interested in. The next step is to reconstruct the digital model of fabricated product by using multiple captured images to reposition parts according to the actual model. Then, the projection onto the 3D digital reconstruction of the safety related installation constraints, respecting the original intent of the constraints that are defined in the digital mock up is done. The differences between the 3D reconstruction of the actual product and the design time digital mockup of the product are identified. Finally, the differences/non conformances that have a relevance to safety driven installation requirements with reference to the original safety requirement intent are identified. The above steps together give the safety engineer the ability to overlay a digital reconstruction that should be as true to the fabricated product as possible so that they can see how the product conforms or doesn't conform to the safety driven installation requirements. The work has produced a concept demonstrator that will be further developed in future work to address accuracy, work flow and process efficiency. A new depth based segmentation technique GrabcutD which is an improvement to existing Grabcut, a graph cut based segmentation method is proposed. Conventional Grabcut relies only on color information to achieve segmentation. However, in stereo or multiview analysis, there is additional information that could be also used to improve segmentation. Clearly, depth based approaches bear the potential discriminative power of ascertaining whether the object is nearer of farer. We show the usefulness of the approach when stereo information is available and evaluate it using standard datasets against state of the art result

Desarrollo de geotecnologías aplicadas a la inspección y monitorización de entornos industriales

Tesis por compendio de publicaciones[ES]El desarrollo tecnológico de las últimas dos décadas ha supuesto un cambio radical que está llevando a un nuevo paradigma en el que se entremezclan el mundo físico y el digital. Estos cambios han influido enormemente en la sociedad, modificando las formas de comunicación, acceso a información, ocio, trabajo, etc. Asimismo, la industria ha adoptado estas tecnologías disruptivas, las cuales están contribuyendo a lograr un mayor control y automatización del proceso productivo. En el ámbito industrial, las tareas de mantenimiento son críticas para garantizar el correcto funcionamiento de una planta o instalación, ya que influyen directamente en la productividad y pueden suponer un elevado costo adicional. Las nuevas tecnologías están posibilitando la monitorización continua y a la inspección automatizada, proporcionando herramientas auxiliares a los inspectores que mejoran la detección de fallos y permiten anticipar y optimizar la planificación de las tareas de mantenimiento. Con el objetivo de desarrollar herramientas que aporten mejoras en las tareas de mantenimiento en industria, la presente tesis doctoral se basa en el estudio de como las geotecnologías pueden aportar soluciones óptimas en la monitorización e inspección. Debido a la gran variedad de entornos industriales, las herramientas de apoyo al mantenimiento deben adaptarse a cada caso en concreto. En este aspecto, y con el fin de demostrar la adaptabilidad de la geomática y las geotecnologías, se han estudiado instalaciones industriales de ámbitos muy diversos, como una sala de máquinas (escenario interior), plantas fotovoltaicas (escenario exterior) y soldaduras (interior y exterior). La escala de los escenarios objeto de estudio ha sido muy variada, desde las escalas más pequeñas, para el estudio de las soldaduras y la sala de máquinas, a las escalas más grandes, en los estudios de evolución de la vegetación y presencia de masas de agua en plantas fotovoltaicas. Las geotecnologías demuestran su versatilidad para trabajar a distintas escalas, con soluciones que permiten un gran detalle y precisión, como la fotogrametría de rango cercano y el sistema de escaneado portátil (Portable Mobile Mapping System - PMMS), y otras que pueden abarcar zonas más amplias del territorio, como es el caso de la teledetección o la fotogrametría con drones. Según lo expuesto anteriormente, el enfoque de la tesis ha sido el estudio de elementos o instalaciones industriales a diferentes escalas. En el primer caso se desarrolló una herramienta para el control de calidad externo de soldaduras utilizando fotogrametría de rango cercano y algoritmos para la detección automática de defectos. En el segundo caso se propuso el uso de un PMMS para optimizar la toma de datos en las tareas de inspección en instalaciones fluidomecánicas. En el tercer caso se utilizó la fotogrametría con drones y la combinación de imágenes RGB y térmicas con algoritmos de visión computacional para la detección de patologías en paneles fotovoltaicos. Finalmente, para la monitorización de la vegetación y la detección de masas de agua en el entorno de plantas fotovoltaicas, se empleó la teledetección mediante el cálculo de índices espectrales. [EN]The technological development of the last two decades has brought about a radical change that is leading to a new paradigm in which the physical and digital worlds are intertwined. These changes have had a great impact on society, modifying communication methods, access to information, leisure, work, etc. In addition, the industry has adopted these disruptive technologies, which are contributing to achieving greater control and automation of the production process. In the industrial sector, maintenance tasks are critical to ensuring the proper operation of a plant or facility, as they directly influence productivity and can involve high additional costs. New technologies are making continuous monitoring and automated inspection possible, providing auxiliary tools to inspectors that improve fault detection and allow for the anticipation and optimization of maintenance task planning. With the aim of developing tools that provide improvements in maintenance tasks in industry, this doctoral thesis is based on the study of how geotechnologies can provide optimal solutions in monitoring and inspection. Due to the great variety of industrial environments, maintenance support tools must adapt to each specific case. In this regard, and in order to demonstrate the adaptability of geomatics and geotechnologies, industrial installations from very diverse areas have been studied, such as a machine room (indoor scenario), photovoltaic plants (outdoor scenario), and welding (indoor and outdoor scenarios). The scale of the studied scenarios has been very varied, ranging from smaller scales for the study of welds and machine rooms, to larger scales in the studies of vegetation evolution and the presence of bodies of water in photovoltaic plants. Geotechnologies demonstrate their versatility to work at different scales, with solutions that allow for great detail and precision, such as close-range photogrammetry and the Portable Mobile Mapping System (PMMS), as well as others that can cover larger areas of the territory, such as remote sensing or photogrammetry with drones. The focus of the thesis has been the study of industrial elements or installations at different scales. In the first case, a tool was developed for external quality control of welding, using close-range photogrammetry and algorithms for automatic defect detection. In the second case, the use of a PMMS is proposed to optimize data collection in fluid-mechanical installation inspection tasks. In the third case, drone photogrammetry and the combination of RGB and thermal images with computer vision algorithms were used for the detection of pathologies in photovoltaic panels. Finally, for the monitoring of vegetation and the detection of water masses in the environment of photovoltaic plants, remote sensing was employed through the calculation of spectral indices

Image-based 3-D reconstruction of constrained environments

Nuclear power plays a important role to the United Kingdom electricity generation infrastructure, providing a reliable baseload of low carbon electricity. The Advanced Gas-cooled Reactor (AGR) design makes up approximately 50% of the existing fleet, however, many of the operating reactors have exceeding their original design lifetimes.To ensure safe reactor operation, engineers perform periodic in-core visual inspections of reactor components to monitor the structural health of the core as it ages. However, current inspection mechanisms deployed provide limited structural information about the fuel channel or defects.;This thesis investigates the suitability of image-based 3-D reconstruction techniques to acquire 3-D structural geometry to enable improved diagnostic and prognostic abilities for inspection engineers. The application of image-based 3-D reconstruction to in-core inspection footage highlights significant challenges, most predominantly that the image saliency proves insuffcient for general reconstruction frameworks. The contribution of the thesis is threefold. Firstly, a novel semi-dense matching scheme which exploits sparse and dense image correspondence in combination with a novel intra-image region strength approach to improve the stability of the correspondence between images.;This results in a percentage increase of 138.53% of correct feature matches over similar state-of-the-art image matching paradigms. Secondly, a bespoke incremental Structure-from-Motion (SfM) framework called the Constrained Homogeneous SfM (CH-SfM) which is able to derive structure from deficient feature spaces and constrained environments. Thirdly, the application of the CH-SfM framework to remote visual inspection footage gathered within AGR fuel channels, outperforming other state-of-the-art reconstruction approaches and extracting representative 3-D structural geometry of orientational scans and fully circumferential reconstructions.;This is demonstrated on in-core and laboratory footage, achieving an approximate 3-D point density of 2.785 - 23.8025NX/cm² for real in-core inspection footage and high quality laboratory footage respectively. The demonstrated novelties have applicability to other constrained or feature-poor environments, with future work looking to producing fully dense, photo-realistic 3-D reconstructions.Nuclear power plays a important role to the United Kingdom electricity generation infrastructure, providing a reliable baseload of low carbon electricity. The Advanced Gas-cooled Reactor (AGR) design makes up approximately 50% of the existing fleet, however, many of the operating reactors have exceeding their original design lifetimes.To ensure safe reactor operation, engineers perform periodic in-core visual inspections of reactor components to monitor the structural health of the core as it ages. However, current inspection mechanisms deployed provide limited structural information about the fuel channel or defects.;This thesis investigates the suitability of image-based 3-D reconstruction techniques to acquire 3-D structural geometry to enable improved diagnostic and prognostic abilities for inspection engineers. The application of image-based 3-D reconstruction to in-core inspection footage highlights significant challenges, most predominantly that the image saliency proves insuffcient for general reconstruction frameworks. The contribution of the thesis is threefold. Firstly, a novel semi-dense matching scheme which exploits sparse and dense image correspondence in combination with a novel intra-image region strength approach to improve the stability of the correspondence between images.;This results in a percentage increase of 138.53% of correct feature matches over similar state-of-the-art image matching paradigms. Secondly, a bespoke incremental Structure-from-Motion (SfM) framework called the Constrained Homogeneous SfM (CH-SfM) which is able to derive structure from deficient feature spaces and constrained environments. Thirdly, the application of the CH-SfM framework to remote visual inspection footage gathered within AGR fuel channels, outperforming other state-of-the-art reconstruction approaches and extracting representative 3-D structural geometry of orientational scans and fully circumferential reconstructions.;This is demonstrated on in-core and laboratory footage, achieving an approximate 3-D point density of 2.785 - 23.8025NX/cm² for real in-core inspection footage and high quality laboratory footage respectively. The demonstrated novelties have applicability to other constrained or feature-poor environments, with future work looking to producing fully dense, photo-realistic 3-D reconstructions