A Comprehensive Survey of Deep Learning in Remote Sensing: Theories, Tools and Challenges for the Community

In recent years, deep learning (DL), a re-branding of neural networks (NNs), has risen to the top in numerous areas, namely computer vision (CV), speech recognition, natural language processing, etc. Whereas remote sensing (RS) possesses a number of unique challenges, primarily related to sensors and applications, inevitably RS draws from many of the same theories as CV; e.g., statistics, fusion, and machine learning, to name a few. This means that the RS community should be aware of, if not at the leading edge of, of advancements like DL. Herein, we provide the most comprehensive survey of state-of-the-art RS DL research. We also review recent new developments in the DL field that can be used in DL for RS. Namely, we focus on theories, tools and challenges for the RS community. Specifically, we focus on unsolved challenges and opportunities as it relates to (i) inadequate data sets, (ii) human-understandable solutions for modelling physical phenomena, (iii) Big Data, (iv) non-traditional heterogeneous data sources, (v) DL architectures and learning algorithms for spectral, spatial and temporal data, (vi) transfer learning, (vii) an improved theoretical understanding of DL systems, (viii) high barriers to entry, and (ix) training and optimizing the DL.Comment: 64 pages, 411 references. To appear in Journal of Applied Remote Sensin

BRUISE DETECTION IN APPLES USING 3D INFRARED IMAGING AND MACHINE LEARNING TECHNOLOGIES

Bruise detection plays an important role in fruit grading. A bruise detection system capable of finding and removing damaged products on the production lines will distinctly improve the quality of fruits for sale, and consequently improve the fruit economy. This dissertation presents a novel automatic detection system based on surface information obtained from 3D near-infrared imaging technique for bruised apple identification. The proposed 3D bruise detection system is expected to provide better performance in bruise detection than the existing 2D systems. We first propose a mesh denoising filter to reduce noise effect while preserving the geometric features of the meshes. Compared with several existing mesh denoising filters, the proposed filter achieves better performance in reducing noise effect as well as preserving bruised regions in 3D meshes of bruised apples. Next, we investigate two different machine learning techniques for the identification of bruised apples. The first technique is to extract hand-crafted feature from 3D meshes, and train a predictive classifier based on hand-crafted features. It is shown that the predictive model trained on the proposed hand-crafted features outperforms the same models trained on several other local shape descriptors. The second technique is to apply deep learning to learn the feature representation automatically from the mesh data, and then use the deep learning model or a new predictive model for the classification. The optimized deep learning model achieves very high classification accuracy, and it outperforms the performance of the detection system based on the proposed hand-crafted features. At last, we investigate GPU techniques for accelerating the proposed apple bruise detection system. Specifically, the dissertation proposes a GPU framework, implemented in CUDA, for the acceleration of the algorithm that extracts vertex-based local binary patterns. Experimental results show that the proposed GPU program speeds up the process of extracting local binary patterns by 5 times compared to a single-core CPU program

Three-dimensional modeling of natural heterogeneous objects

En la medicina y otros campos relacionados cuando se va a estudiar un objeto natural, se toman imágenes de tomografía computarizada a través de varios cortes paralelos. Estos cortes se apilan en datos de volumen y se reconstruyen en modelos computacionales con el fin de estudiar la estructura de dicho objeto. Para construir con éxito modelos tridimensionales es importante la identificación y extracción precisa de todas las regiones que comprenden el objeto heterogéneo natural. Sin embargo, la construcción de modelos tridimensionales por medio del computador a partir de imágenes médicas sigue siendo un problema difícil y plantea dos problemas relacionados con las inexactitudes que surgen de, y son inherentes al proceso de adquisición de datos. El primer problema es la aparición de artefactos que distorsionan el límite entre las regiones. Este es un problema común en la generación de mallas a partir de imágenes médicas, también conocido como efecto de escalón. El segundo problema es la extracción de mallas suaves 3D que se ajustan a los límites de las región que conforman los objetos heterogéneos naturales descritos en las imágenes médicas. Para resolver estos problemas, se propone el método CAREM y el método RAM. El énfasis de esta investigación está puesto en la exactitud y fidelidad a la forma de las regiones necesaria en las aplicaciones biomédicas. Todas las regiones representadas de forma implícita que componen el objeto heterogéneo natural se utilizan para generar mallas adaptadas a los requisitos de los métodos de elementos finitos a través de un enfoque de modelado de ingeniería reversa, por lo tanto, estas regiones se consideran como un todo en lugar de piezas aisladas ensambladas.In medicine and other related fields when a natural object is going to be studied, computed tomography images are taken through several parallel slices. These slices are then stacked in volume data and reconstructed into 3D computer models. In order to successfully build 3D computer models of natural heterogeneous objects, accurate identification and extraction of all regions comprising the natural heterogeneous object is important. However, building 3D computer models of natural heterogeneous objects from medical images is still a challenging problem, and poses two issues related to the inaccuracies which arise from and are inherent to the data acquisition process. The first issue is the appearance of aliasing artifacts in the boundary between regions, a common issue in mesh generation from medical images, also known as stair-stepped artifacts. The second issue is the extraction of smooth 3D multi-region meshes that conform to the region boundaries of natural heterogeneous objects described in the medical images. To solve these issues, the CAREM method and the RAM method are proposed. The emphasis of this research is placed on accuracy and shape fidelity needed for biomedical applications. All implicitly represented regions composing the natural heterogeneous object are used to generate meshes adapted to the requirements of finite element methods through a reverse engineering modeling approach, thus these regions are considered as whole rather than loosely assembled parts.Doctor en IngenieríaDoctorad

On the Real-Time Performance, Robustness and Accuracy of Medical Image Non-Rigid Registration

Three critical issues about medical image non-rigid registration are performance, robustness and accuracy. A registration method, which is capable of responding timely with an accurate alignment, robust against the variation of the image intensity and the missing data, is desirable for its clinical use. This work addresses all three of these issues. Unacceptable execution time of Non-rigid registration (NRR) often presents a major obstacle to its routine clinical use. We present a hybrid data partitioning method to parallelize a NRR method on a cooperative architecture, which enables us to get closer to the goal: accelerating using architecture rather than designing a parallel algorithm from scratch. to further accelerate the performance for the GPU part, a GPU optimization tool is provided to automatically optimize GPU execution configuration.;Missing data and variation of the intensity are two severe challenges for the robustness of the registration method. A novel point-based NRR method is presented to resolve mapping function (deformation field) with the point correspondence missing. The novelty of this method lies in incorporating a finite element biomechanical model into an Expectation and Maximization (EM) framework to resolve the correspondence and mapping function simultaneously. This method is extended to deal with the deformation induced by tumor resection, which imposes another challenge, i.e. incomplete intra-operative MRI. The registration is formulated as a three variable (Correspondence, Deformation Field, and Resection Region) functional minimization problem and resolved by a Nested Expectation and Maximization framework. The experimental results show the effectiveness of this method in correcting the deformation in the vicinity of the tumor. to deal with the variation of the intensity, two different methods are developed depending on the specific application. For the mono-modality registration on delayed enhanced cardiac MRI and cine MRI, a hybrid registration method is designed by unifying both intensity- and feature point-based metrics into one cost function. The experiment on the moving propagation of suspicious myocardial infarction shows effectiveness of this hybrid method. For the multi-modality registration on MRI and CT, a Mutual Information (MI)-based NRR is developed by modeling the underlying deformation as a Free-Form Deformation (FFD). MI is sensitive to the variation of the intensity due to equidistant bins. We overcome this disadvantage by designing a Top-to-Down K-means clustering method to naturally group similar intensities into one bin. The experiment shows this method can increase the accuracy of the MI-based registration.;In image registration, a finite element biomechanical model is usually employed to simulate the underlying movement of the soft tissue. We develop a multi-tissue mesh generation method to build a heterogeneous biomechanical model to realistically simulate the underlying movement of the brain. We focus on the following four critical mesh properties: tissue-dependent resolution, fidelity to tissue boundaries, smoothness of mesh surfaces, and element quality. Each mesh property can be controlled on a tissue level. The experiments on comparing the homogeneous model with the heterogeneous model demonstrate the effectiveness of the heterogeneous model in improving the registration accuracy

Heterogeneous volumetric data mapping and its medical applications

With the advance of data acquisition techniques, massive solid geometries are being collected routinely in scientific tasks, these complex and unstructured data need to be effectively correlated for various processing and analysis. Volumetric mapping solves bijective low-distortion correspondence between/among 3D geometric data, and can serve as an important preprocessing step in many tasks in compute-aided design and analysis, industrial manufacturing, medical image analysis, to name a few. This dissertation studied two important volumetric mapping problems: the mapping of heterogeneous volumes (with nonuniform inner structures/layers) and the mapping of sequential dynamic volumes. To effectively handle heterogeneous volumes, first, we studied the feature-aligned harmonic volumetric mapping. Compared to previous harmonic mapping, it supports the point, curve, and iso-surface alignment, which are important low-dimensional structures in heterogeneous volumetric data. Second, we proposed a biharmonic model for volumetric mapping. Unlike the conventional harmonic volumetric mapping that only supports positional continuity on the boundary, this new model allows us to have higher order continuity $C^1$ along the boundary surface. This suggests a potential model to solve the volumetric mapping of complex and big geometries through divide-and-conquer. We also studied the medical applications of our volumetric mapping in lung tumor respiratory motion modeling. We were building an effective digital platform for lung tumor radiotherapy based on effective volumetric CT/MRI image matching and analysis. We developed and integrated in this platform a set of geometric/image processing techniques including advanced image segmentation, finite element meshing, volumetric registration and interpolation. The lung organ/tumor and surrounding tissues are treated as a heterogeneous region and a dynamic 4D registration framework is developed for lung tumor motion modeling and tracking. Compared to the previous 3D pairwise registration, our new 4D parameterization model leads to a significantly improved registration accuracy. The constructed deforming model can hence approximate the deformation of the tissues and tumor

Visual Techniques for Geological Fieldwork Using Mobile Devices

Visual techniques in general and 3D visualisation in particular have seen considerable adoption within the last 30 years in the geosciences and geology. Techniques such as volume visualisation, for analysing subsurface processes, and photo-coloured LiDAR point-based rendering, to digitally explore rock exposures at the earth’s surface, were applied within geology as one of the first adopting branches of science. A large amount of digital, geological surface- and volume data is nowadays available to desktop-based workflows for geological applications such as hydrocarbon reservoir exploration, groundwater modelling, CO2 sequestration and, in the future, geothermal energy planning. On the other hand, the analysis and data collection during fieldwork has yet to embrace this ”digital revolution”: sedimentary logs, geological maps and stratigraphic sketches are still captured in each geologist’s individual fieldbook, and physical rocks samples are still transported to the lab for subsequent analysis. Is this still necessary, or are there extended digital means of data collection and exploration in the field ? Are modern digital interpretation techniques accurate and intuitive enough to relevantly support fieldwork in geology and other geoscience disciplines ? This dissertation aims to address these questions and, by doing so, close the technological gap between geological fieldwork and office workflows in geology. The emergence of mobile devices and their vast array of physical sensors, combined with touch-based user interfaces, high-resolution screens and digital cameras provide a possible digital platform that can be used by field geologists. Their ubiquitous availability increases the chances to adopt digital workflows in the field without additional, expensive equipment. The use of 3D data on mobile devices in the field is furthered by the availability of 3D digital outcrop models and the increasing ease of their acquisition. This dissertation assesses the prospects of adopting 3D visual techniques and mobile devices within field geology. The research of this dissertation uses previously acquired and processed digital outcrop models in the form of textured surfaces from optical remote sensing and photogrammetry. The scientific papers in this thesis present visual techniques and algorithms to map outcrop photographs in the field directly onto the surface models. Automatic mapping allows the projection of photo interpretations of stratigraphy and sedimentary facies on the 3D textured surface while providing the domain expert with simple-touse, intuitive tools for the photo interpretation itself. The developed visual approach, combining insight from all across the computer sciences dealing with visual information, merits into the mobile device Geological Registration and Interpretation Toolset (GRIT) app, which is assessed on an outcrop analogue study of the Saltwick Formation exposed at Whitby, North Yorkshire, UK. Although being applicable to a diversity of study scenarios within petroleum geology and the geosciences, the particular target application of the visual techniques is to easily provide field-based outcrop interpretations for subsequent construction of training images for multiple point statistics reservoir modelling, as envisaged within the VOM2MPS project. Despite the success and applicability of the visual approach, numerous drawbacks and probable future extensions are discussed in the thesis based on the conducted studies. Apart from elaborating on more obvious limitations originating from the use of mobile devices and their limited computing capabilities and sensor accuracies, a major contribution of this thesis is the careful analysis of conceptual drawbacks of established procedures in modelling, representing, constructing and disseminating the available surface geometry. A more mathematically-accurate geometric description of the underlying algebraic surfaces yields improvements and future applications unaddressed within the literature of geology and the computational geosciences to this date. Also, future extensions to the visual techniques proposed in this thesis allow for expanded analysis, 3D exploration and improved geological subsurface modelling in general.publishedVersio

Towards BIM/GIS interoperability: A theoretical framework and practical generation of spaces to support infrastructure Asset Management

The past ten years have seen the widespread adoption of Building Information Modelling (BIM) among both the Architectural, Engineering and Construction (AEC) and the Asset Management/ Facilities Management (AM/FM) communities. This has been driven by the use of digital information to support collaborative working and a vision for more efficient reuse of data. Within this context, spatial information is either held in a Geographic Information Systems (GIS) or as Computer-Aided Design (CAD) models in a Common Data Environment (CDE). However, these being heterogeneous systems, there are inevitable interoperability issues that result in poor integration. For this thesis, the interoperability challenges were investigated within a case study to ask: Can a better understanding of the conceptual and technical challenges to the integration of BIM and GIS provide improved support for the management of asset information in the context of a major infrastructure project? Within their respective fields, the terms BIM and GIS have acquired a range of accepted meanings, that do not align well with each other. A seven-level socio-technical framework is developed to harmonise concepts in spatial information systems. This framework is used to explore the interoperability gaps that must be resolved to enable design and construction information to be joined up with operational asset information. The Crossrail GIS and BIM systems were used to investigate some of the interoperability challenges that arise during the design, construction and operation of an infrastructure asset. One particular challenge concerns a missing link between AM-based information and CAD-based geometry which hinders engineering assets from being located within the geometric model and preventing geospatial analysis. A process is developed to link these CAD-based elements with AM-based assets using defined 3D spaces to locate assets. However, other interoperability challenges must first be overcome; firstly, the extraction, transformation and loading of geometry from CAD to GIS; secondly, the creation of an explicit representation of each 3D space from the implicit enclosing geometry. This thesis develops an implementation of the watershed transform algorithm to use real-world Crossrail geometry to generate voxelated interior spaces that can then be converted into a B-Rep mesh for use in 3D GIS. The issues faced at the technical level in this case study provide insight into the differences that must also be addressed at the conceptual level. With this in mind, this thesis develops a Spatial Information System Framework to classify the nature of differences between BIM, GIS and other spatial information systems