Deep learning in remote sensing: a review

Standing at the paradigm shift towards data-intensive science, machine learning techniques are becoming increasingly important. In particular, as a major breakthrough in the field, deep learning has proven as an extremely powerful tool in many fields. Shall we embrace deep learning as the key to all? Or, should we resist a 'black-box' solution? There are controversial opinions in the remote sensing community. In this article, we analyze the challenges of using deep learning for remote sensing data analysis, review the recent advances, and provide resources to make deep learning in remote sensing ridiculously simple to start with. More importantly, we advocate remote sensing scientists to bring their expertise into deep learning, and use it as an implicit general model to tackle unprecedented large-scale influential challenges, such as climate change and urbanization.Comment: Accepted for publication IEEE Geoscience and Remote Sensing Magazin

Aprendizado ativo baseado em atributos contextuais de superpixel para classificação de imagem de sensoriamento remoto

Orientadores: Alexandre Xavier Falcão, Jefersson Alex dos SantosDissertação (mestrado) - Universidade Estadual de Campinas, Instituto de ComputaçãoResumo: Recentemente, técnicas de aprendizado de máquina têm sido propostas para criar mapas temáticos a partir de imagens de sensoriamento remoto. Estas técnicas podem ser divididas em métodos de classificação baseados em pixels ou regiões. Este trabalho concentra-se na segunda abordagem, uma vez que estamos interessados em imagens com milhões de pixels e a segmentação da imagem em regiões (superpixels) pode reduzir consideravelmente o número de amostras a serem classificadas. Porém, mesmo utilizando superpixels, o número de amostras ainda é grande para anotá-las manualmente e treinar o classificador. As técnicas de aprendizado ativo propostas resolvem este problema começando pela seleção de um conjunto pequeno de amostras selecionadas aleatoriamente. Tais amostras são anotadas manualmente e utilizadas para treinar a primeira instância do classificador. Em cada iteração do ciclo de aprendizagem, o classificador atribui rótulos e seleciona as amostras mais informativas para a correção/confirmação pelo usuário, aumentando o tamanho do conjunto de treinamento. A instância do classificador é melhorada no final de cada iteração pelo seu treinamento e utilizada na iteração seguinte até que o usuário esteja satisfeito com o classificador. Observamos que a maior parte dos métodos reclassificam o conjunto inteiro de dados em cada iteração do ciclo de aprendizagem, tornando este processo inviável para interação com o usuário. Portanto, enderaçamos dois problemas importantes em classificação baseada em regiões de imagens de sensoriamento remoto: (a) a descrição efetiva de superpixels e (b) a redução do tempo requerido para seleção de amostras em aprendizado ativo. Primeiro, propusemos um descritor contextual de superpixels baseado na técnica de sacola de palavras, que melhora o resultado de descritores de cor e textura amplamente utilizados. Posteriormente, propusemos um método supervisionado de redução do conjunto de dados que é baseado em um método do estado da arte em aprendizado ativo chamado Multi-Class Level Uncertainty (MCLU). Nosso método mostrou-se tão eficaz quanto o MCLU e ao mesmo tempo consideravelmente mais eficiente. Adicionalmente, melhoramos seu desempenho por meio da aplicação de um processo de relaxação no mapa de classificação, utilizando Campos Aleatórios de MarkovAbstract: In recent years, machine learning techniques have been proposed to create classification maps from remote sensing images. These techniques can be divided into pixel- and region-based image classification methods. This work concentrates on the second approach, since we are interested in images with millions of pixels and the segmentation of the image into regions (superpixels) can considerably reduce the number of samples for classification. However, even using superpixels the number of samples is still large for manual annotation of samples to train the classifier. Active learning techniques have been proposed to address the problem by starting from a small set of randomly selected samples, which are manually labeled and used to train a first instance of the classifier. At each learning iteration, the classifier assigns labels and selects the most informative samples for user correction/confirmation, increasing the size of the training set. An improved instance of the classifier is created by training, after each iteration, and used in the next iteration until the user is satisfied with the classifier. We observed that most methods reclassify the entire pool of unlabeled samples at every learning iteration, making the process unfeasible for user interaction. Therefore, we address two important problems in region-based classification of remote sensing images: (a) the effective superpixel description and (b) the reduction of the time required for sample selection in active learning. First, we propose a contextual superpixel descriptor, based on bag of visual words, that outperforms widely used color and texture descriptors. Second, we propose a supervised method for dataset reduction that is based on a state-of-art active learning technique, called Multi-Class Level Uncertainty (MCLU). Our method has shown to be as effective as MCLU, while being considerably more efficient. Additionally, we further improve its performance by applying a relaxation process on the classification map by using Markov Random FieldsMestradoCiência da ComputaçãoMestre em Ciência da Computaçã

Impact of Feature Representation on Remote Sensing Image Retrieval

Remote sensing images are acquired using special platforms, sensors and are classified as aerial, multispectral and hyperspectral images. Multispectral and hyperspectral images are represented using large spectral vectors as compared to normal Red, Green, Blue (RGB) images. Hence, remote sensing image retrieval process from large archives is a challenging task.  Remote sensing image retrieval mainly consist of feature representation as first step and finding out similar images to a query image as second step. Feature representation plays important part in the performance of remote sensing image retrieval process. Research work focuses on impact of feature representation of remote sensing images on the performance of remote sensing image retrieval. This study shows that more discriminative features of remote sensing images are needed to improve performance of remote sensing image retrieval process

Fine-Grained Object Recognition and Zero-Shot Learning in Remote Sensing Imagery

Fine-grained object recognition that aims to identify the type of an object among a large number of subcategories is an emerging application with the increasing resolution that exposes new details in image data. Traditional fully supervised algorithms fail to handle this problem where there is low between-class variance and high within-class variance for the classes of interest with small sample sizes. We study an even more extreme scenario named zero-shot learning (ZSL) in which no training example exists for some of the classes. ZSL aims to build a recognition model for new unseen categories by relating them to seen classes that were previously learned. We establish this relation by learning a compatibility function between image features extracted via a convolutional neural network and auxiliary information that describes the semantics of the classes of interest by using training samples from the seen classes. Then, we show how knowledge transfer can be performed for the unseen classes by maximizing this function during inference. We introduce a new data set that contains 40 different types of street trees in 1-ft spatial resolution aerial data, and evaluate the performance of this model with manually annotated attributes, a natural language model, and a scientific taxonomy as auxiliary information. The experiments show that the proposed model achieves 14.3% recognition accuracy for the classes with no training examples, which is significantly better than a random guess accuracy of 6.3% for 16 test classes, and three other ZSL algorithms.Comment: G. Sumbul, R. G. Cinbis, S. Aksoy, "Fine-Grained Object Recognition and Zero-Shot Learning in Remote Sensing Imagery", IEEE Transactions on Geoscience and Remote Sensing (TGRS), in press, 201

Dense semantic labeling of sub-decimeter resolution images with convolutional neural networks

Semantic labeling (or pixel-level land-cover classification) in ultra-high resolution imagery (< 10cm) requires statistical models able to learn high level concepts from spatial data, with large appearance variations. Convolutional Neural Networks (CNNs) achieve this goal by learning discriminatively a hierarchy of representations of increasing abstraction. In this paper we present a CNN-based system relying on an downsample-then-upsample architecture. Specifically, it first learns a rough spatial map of high-level representations by means of convolutions and then learns to upsample them back to the original resolution by deconvolutions. By doing so, the CNN learns to densely label every pixel at the original resolution of the image. This results in many advantages, including i) state-of-the-art numerical accuracy, ii) improved geometric accuracy of predictions and iii) high efficiency at inference time. We test the proposed system on the Vaihingen and Potsdam sub-decimeter resolution datasets, involving semantic labeling of aerial images of 9cm and 5cm resolution, respectively. These datasets are composed by many large and fully annotated tiles allowing an unbiased evaluation of models making use of spatial information. We do so by comparing two standard CNN architectures to the proposed one: standard patch classification, prediction of local label patches by employing only convolutions and full patch labeling by employing deconvolutions. All the systems compare favorably or outperform a state-of-the-art baseline relying on superpixels and powerful appearance descriptors. The proposed full patch labeling CNN outperforms these models by a large margin, also showing a very appealing inference time.Comment: Accepted in IEEE Transactions on Geoscience and Remote Sensing, 201

Could Multimedia approaches help Remote Sensing Analysis?

International audienceThe paper explores how multimedia approaches used in image understanding tasks could be adapted and used in remote sensing image analysis. Two approaches are investigated: the classical Bag of Visual Words (BoVW) approach and the Deep Learning approach. Tests are performed for the classification of the UC Merced Land Use Dataset which provide better results than the state of the art