Pixel-Wise Classification Method for High Resolution Remote Sensing Imagery Using Deep Neural Networks

Considering the classification of high spatial resolution remote sensing imagery, this paper presents a novel classification method for such imagery using deep neural networks. Deep learning methods, such as a fully convolutional network (FCN) model, achieve state-of-the-art performance in natural image semantic segmentation when provided with large-scale datasets and respective labels. To use data efficiently in the training stage, we first pre-segment training images and their labels into small patches as supplements of training data using graph-based segmentation and the selective search method. Subsequently, FCN with atrous convolution is used to perform pixel-wise classification. In the testing stage, post-processing with fully connected conditional random fields (CRFs) is used to refine results. Extensive experiments based on the Vaihingen dataset demonstrate that our method performs better than the reference state-of-the-art networks when applied to high-resolution remote sensing imagery classification

Image-based time series representations for satellite images classification

Orientador: Ricardo da Silva TorresTese (doutorado) - Universidade Estadual de Campinas, Instituto de ComputaçãoResumo: A classificação de imagens de sensoriamento remoto por pixel com base no perfil temporal desempenha um papel importante em várias aplicações, tais como: reconhecimento de culturas, estudos fenológicos e monitoramento de mudanças na cobertura do solo. Avanços sensores captura de imagem aumentaram a necessidade de criação de metodologias para analisar o perfil temporal das informações coletadas. Nós investigamos dados coletados em dois tipos de sensores: (i) sensores em plataformas orbitais, esse tipo de imagem sofre interferências de nuvens e fatores atmosféricos; e (ii) sensores fixados em campo, mais especificamente, uma câmera digital no alto de uma torre, cujas imagens capturadas podem conter dezenas de espécies, dificultando a identificação de padrões de interesse. Devido às particularidades dos dados detectados remotamente, torna-se custoso enviar a imagem capturada pelo sensor diretamente para métodos de aprendizado de máquina sem realizar um pré-processamento. Para algumas aplicações de sensoriamento remoto, comumente não se utiliza as imagens brutas oriundas dos sensores, mas os índices de vegetação extraídos das regiões de interesse ao longo do tempo. Assim, o perfil temporal pode ser caracterizado como uma série de observações dos índices de vegetação dos pixels de interesse. Métodos baseados em aprendizado profundo obtiveram bons resultados em aplicações de sensoriamento remoto relacionadas à classificação de imagens. Contudo, em consequência da natureza dos dados, nem sempre é possível realizar o treinamento adequado das redes de aprendizado profundo pela limitação causada por dados faltantes. Entretanto, podemos nos beneficiar de redes previamente treinadas para detecção de objetos para extrair características e padrões de imagens. O problema alvo deste trabalho é classificar séries temporais extraídas de imagens de sensoriamento remoto representando as características temporais como imagens 2D. Este trabalho investiga abordagens que codificam séries temporais como representação de imagem para propor metodologias de classificação binária e multiclasse no contexto de sensoriamento remoto, se beneficiando de redes extratoras de características profundas. Os experimentos conduzidos para classificação binária foram realizados em dados de satélite para identificar plantações de eucalipto. Os resultados superaram métodos baseline propostos recentemente. Os experimentos realizados para classificação multiclasse focaram em imagens capturadas com câmera digital para detectar o padrão fenológico de regiões de interesse. Os resultados mostram que a acurácia aumenta se consideramos conjuntos de pixelsAbstract: Pixelwise remote sensing image classification based on temporal profile plays an important role in several applications, such as crop recognition, phenological studies, and land cover change monitoring. Advances in image capture sensors have increased the need for methodologies to analyze the temporal profile of collected information. We investigate data collected by two types of sensors: (i) sensors on orbital platforms, this type of image suffers from cloud interference and atmospheric factors; and (ii) field-mounted sensors, in particular, a digital camera on top of a tower, where captured images may contain dozens of species, making it difficult to identify patterns of interest. Due to the particularities of remotely detected data, it is prohibitive to send sensor captured images directly to machine learning methods without preprocessing. In some remote sensing applications, it is not commonly used the raw images from the sensors, but the vegetation indices extracted from regions of interest over time. Thus, the temporal profile can be characterized as a series of observations of vegetative indices of pixels of interest. Deep learning methods have been successfully in remote sensing applications related to image classification. However, due to the nature of the data, it is not always possible to properly train deep learning networks because of the lack of enough labeled data. However, we can benefit from previously trained 2D object detection networks to extract features and patterns from images. The target problem of this work is to classify remote sensing images, based on pixel time series represented as 2D representations. This work investigates approaches that encode time series into image representations to propose binary and multiclass classification methodologies in the context of remote sensing, taking advantage of data-driven feature extractor approaches. The experiments conducted for binary classification were performed on satellite data to identify eucalyptus plantations. The results surpassed the ones of recently proposed baseline methods. The experiments performed for multiclass classification focused on detecting regions of interest within images captured by a digital camera. The results show that the accuracy increases if we consider a set of pixelsDoutoradoCiência da ComputaçãoDoutora em Ciência da ComputaçãoCAPE

An Explainable AI Approach to Process Data in Mixed Reality Environments for Field Service Operations

Digital Twins is a concept that describes how physical objects can be represented and connected to the virtual world, the main goal of a Digital Twin is to centralise all the available information of an object of interest in a single virtual model. The Digital Twin consist of three main components: the physical object, a virtual representation of that object (typically a 3D model), and a real-time connection between both objects so that any change can be communicated to the other part. The possibility of understanding, visualising, and interacting with physical objects through a virtual environment is, at a very high level, the main benefit of using Digital Twins. The adoption of this concept has grown a lot in the recent years in industries such as the manufacturing, construction, health, and energy. Utility companies in the telecommunication industry, water services, and gas services are still falling behind in the adoption of these new concepts. The potential benefit for these sectors is huge where some of these benefits are real-time remote monitoring, predictive maintenance, scenario and risk assessment, better collaboration between stakeholders (internal and external), and better documentation. Existing Mixed Reality, Virtual Reality and Augmented Reality technologies can help with the interaction and visualisation of the virtual twin. The different levels of reality in combination with the digital twins will help with different tasks, for example, Virtual Reality is useful for remote tasks were most of the interaction happens with the virtual twin and Augmented Reality will help bringing the virtual twin and all its information to onsite tasks to help field engineers. However, there are different challenges when trying to connect all the different components and some of these challenges did slow down the adoption of these technologies by the utility companies. The research work in this thesis will focus on two main challenges: the cost of creating these digital twins from existing sources of information and the lack of an explainable AI approach that can be used as an enabler for the interaction between human and Digital Twin in the mixed reality environment. To address the challenge of automating the creation of digital representations at a low cost, two interval type-2 Fuzzy Rule-based Systems are presented as the best explainable AI alternatives to the opaque AI models for processing images and extracting information of the objects of interest. One of them was used to extract information about trees in a satellite image and generate a 3D representation of the geographic area combined with terrain data. This will be used for remote scenario and risk assessment and prediction of the telecommunication equipment getting damaged by natural elements like trees. The proposed approach achieved an 86.90% of accuracy, 3.5% better than the type-1 but 3.0% worse than the opaque Multilayer Perceptron model. The second interval type-2 Fuzzy Rule-based System is an explainable AI model that incorporates the use of context information in its rule to process 2D floor plan images, identify elements of interest and create a 3D digital representation of the building floors. This will benefit the telecommunication company by automating, at a low cost, the process of creating a more detailed in-building map with the telecommunication assets and improve the collaboration with external stakeholders like contractors for maintenance tasks or construction companies for any works in the building. The proposed method achieved a 97.5% Intersection over Union metric value which was comparable to the 99.3% Intersection over Union of the opaque Convolutional Neural Network model, however our proposed solution is highly interpretable and augmentable by human experts which cannot be achieved via opaque box AI models. Additionally, another interval type-2 Fuzzy Rule-based System for hand gesture classification is also presented in this thesis. This rule-based system achieved a 96.4% accuracy, and it is an easily adjustable model that can be modified to include more hand gestures, the opaque model alternative, a K-Nearest Neighbour algorithm achieved a 98.9% accuracy, however, this model cannot be easily modified by a human expert and re-training is needed which results in a cost of time. This hand gesture recognition model, alongside another fuzzy rule-based system, will help to address the challenge of the interaction between human and digital twin objects in Mixed Reality environments

A review of technical factors to consider when designing neural networks for semantic segmentation of Earth Observation imagery

Semantic segmentation (classification) of Earth Observation imagery is a crucial task in remote sensing. This paper presents a comprehensive review of technical factors to consider when designing neural networks for this purpose. The review focuses on Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Generative Adversarial Networks (GANs), and transformer models, discussing prominent design patterns for these ANN families and their implications for semantic segmentation. Common pre-processing techniques for ensuring optimal data preparation are also covered. These include methods for image normalization and chipping, as well as strategies for addressing data imbalance in training samples, and techniques for overcoming limited data, including augmentation techniques, transfer learning, and domain adaptation. By encompassing both the technical aspects of neural network design and the data-related considerations, this review provides researchers and practitioners with a comprehensive and up-to-date understanding of the factors involved in designing effective neural networks for semantic segmentation of Earth Observation imagery.Comment: 145 pages with 32 figure