Semantic Interleaving Global Channel Attention for Multilabel Remote Sensing Image Classification

Multi-Label Remote Sensing Image Classification (MLRSIC) has received increasing research interest. Taking the cooccurrence relationship of multiple labels as additional information helps to improve the performance of this task. Current methods focus on using it to constrain the final feature output of a Convolutional Neural Network (CNN). On the one hand, these methods do not make full use of label correlation to form feature representation. On the other hand, they increase the label noise sensitivity of the system, resulting in poor robustness. In this paper, a novel method called Semantic Interleaving Global Channel Attention (SIGNA) is proposed for MLRSIC. First, the label co-occurrence graph is obtained according to the statistical information of the data set. The label co-occurrence graph is used as the input of the Graph Neural Network (GNN) to generate optimal feature representations. Then, the semantic features and visual features are interleaved, to guide the feature expression of the image from the original feature space to the semantic feature space with embedded label relations. SIGNA triggers global attention of feature maps channels in a new semantic feature space to extract more important visual features. Multihead SIGNA based feature adaptive weighting networks are proposed to act on any layer of CNN in a plug-and-play manner. For remote sensing images, better classification performance can be achieved by inserting CNN into the shallow layer. We conduct extensive experimental comparisons on three data sets: UCM data set, AID data set, and DFC15 data set. Experimental results demonstrate that the proposed SIGNA achieves superior classification performance compared to state-of-the-art (SOTA) methods. It is worth mentioning that the codes of this paper will be open to the community for reproducibility research. Our codes are available at https://github.com/kyle-one/SIGNA.Comment: 14 pages, 13 figure

Informative and Representative Triplet Selection for Multilabel Remote Sensing Image Retrieval

Learning the similarity between remote sensing (RS) images forms the foundation for content-based RS image retrieval (CBIR). Recently, deep metric learning approaches that map the semantic similarity of images into an embedding (metric) space have been found very popular in RS. A common approach for learning the metric space relies on the selection of triplets of similar (positive) and dissimilar (negative) images to a reference image called an anchor. Choosing triplets is a difficult task particularly for multilabel RS CBIR, where each training image is annotated by multiple class labels. To address this problem, in this article, we propose a novel triplet sampling method in the framework of deep neural networks (DNNs) defined for multilabel RS CBIR problems. The proposed method selects a small set of the most representative and informative triplets based on two main steps. In the first step, a set of anchors that are diverse to each other in the embedding space is selected from the current minibatch using an iterative algorithm. In the second step, different sets of positive and negative images are chosen for each anchor by evaluating the relevancy, hardness, and diversity of the images among each other based on a novel strategy. Experimental results obtained on two multilabel benchmark archives show that the selection of the most informative and representative triplets in the context of DNNs results in: 1) reducing the computational complexity of the training phase of the DNNs without any significant loss on the performance and 2) an increase in learning speed since informative triplets allow fast convergence. The code of the proposed method is publicly available at https://git.tu-berlin.de/rsim/image-retrieval-from-triplets .EC/H2020/759764/EU/Accurate and Scalable Processing of Big Data in Earth Observation/BigEarthDFG, 273827070, SPP 1894: Volunteered Geographic Information: Interpretation, Visualisierung und Social Computin

A Survey of Semantic Construction and Application of Satellite Remote Sensing Images and Data

With the rapid development of satellite technology, remote sensing data has entered the era of big data, and the intelligent processing of remote sensing image has been paid more and more attention. Through the semantic research of remote sensing data, the processing ability of remote sensing data is greatly improved. This paper aims to introduce and analyze the research and application progress of remote sensing image satellite data processing from the perspective of semantic. Firstly, it introduces the characteristics and semantic knowledge of remote sensing big data; Secondly, the semantic concept, semantic construction and application fields are introduced in detail; then, for remote sensing big data, the technical progress in the study field of semantic construction is analyzed from four aspects: semantic description and understanding, semantic segmentation, semantic classification and semantic search, focusing on deep learning technology; Finally, the problems and challenges in the four aspects are discussed in detail, in order to find more directions to explore

Deep Learning for Aerial Scene Understanding in High Resolution Remote Sensing Imagery from the Lab to the Wild

Diese Arbeit präsentiert die Anwendung von Deep Learning beim Verständnis von Luftszenen, z. B. Luftszenenerkennung, Multi-Label-Objektklassifizierung und semantische Segmentierung. Abgesehen vom Training tiefer Netzwerke unter Laborbedingungen bietet diese Arbeit auch Lernstrategien für praktische Szenarien, z. B. werden Daten ohne Einschränkungen gesammelt oder Annotationen sind knapp

On Creating Benchmark Dataset for Aerial Image Interpretation: Reviews, Guidances and Million-AID

The past years have witnessed great progress on remote sensing (RS) image interpretation and its wide applications. With RS images becoming more accessible than ever before, there is an increasing demand for the automatic interpretation of these images. In this context, the benchmark datasets serve as essential prerequisites for developing and testing intelligent interpretation algorithms. After reviewing existing benchmark datasets in the research community of RS image interpretation, this article discusses the problem of how to efficiently prepare a suitable benchmark dataset for RS image interpretation. Specifically, we first analyze the current challenges of developing intelligent algorithms for RS image interpretation with bibliometric investigations. We then present the general guidances on creating benchmark datasets in efficient manners. Following the presented guidances, we also provide an example on building RS image dataset, i.e., Million-AID, a new large-scale benchmark dataset containing a million instances for RS image scene classification. Several challenges and perspectives in RS image annotation are finally discussed to facilitate the research in benchmark dataset construction. We do hope this paper will provide the RS community an overall perspective on constructing large-scale and practical image datasets for further research, especially data-driven ones

Automatic image annotation based on deep learning models: A systematic review and future challenges

Recently, much attention has been given to image annotation due to the massive increase in image data volume. One of the image retrieval methods which guarantees the retrieval of images in the same way as texts are automatic image annotation (AIA). Consequently, numerous studies have been conducted on AIA, particularly on the classification-based and probabilistic modeling techniques. Several image annotation techniques that performed reasonably on standard datasets have been developed over the last decade. In this paper, a review of the image annotation method was conducted, focusing more on deep learning models. Automatic image annotation (AIA) methods were also classified into five categories, including i) Convolutional Neural Network (CNN) based on AIA, ii) Recurrent Neural Network (RNN) based on AIA, iii) Deep Neural Networks (DNN) based on AIA, iv) Long-Short-Term Memory (LSTM) based on AIA, and v) Stacked auto-encoder (SAE) based on AIA. An assessment of the five varieties of AIA methods was also offered based on their principal notion, feature mining technique, explanation precision, computational density, and examined aggregated data. Moreover, the evaluation metrics used to evaluate AIA methods were reviewed and discussed. The need for careful consideration of methods throughout the improvement of novel procedures and datasets for image annotation assignment was highly demanded. From the analysis of the achievements so far, it is certain that more attention should be paid to automatic image annotation

Deep Unsupervised Embedding for Remotely Sensed Images Based on Spatially Augmented Momentum Contrast

Convolutional neural networks (CNNs) have achieved great success when characterizing remote sensing (RS) images. However, the lack of sufficient annotated data (together with the high complexity of the RS image domain) often makes supervised and transfer learning schemes limited from an operational perspective. Despite the fact that unsupervised methods can potentially relieve these limitations, they are frequently unable to effectively exploit relevant prior knowledge about the RS domain, which may eventually constrain their final performance. In order to address these challenges, this article presents a new unsupervised deep metric learning model, called spatially augmented momentum contrast (SauMoCo), which has been specially designed to characterize unlabeled RS scenes. Based on the first law of geography, the proposed approach defines spatial augmentation criteria to uncover semantic relationships among land cover tiles. Then, a queue of deep embeddings is constructed to enhance the semantic variety of RS tiles within the considered contrastive learning process, where an auxiliary CNN model serves as an updating mechanism. Our experimental comparison, including different state-of-the-art techniques and benchmark RS image archives, reveals that the proposed approach obtains remarkable performance gains when characterizing unlabeled scenes since it is able to substantially enhance the discrimination ability among complex land cover categories. The source codes of this article will be made available to the RS community for reproducible research

A Comprehensive Survey of Deep Learning in Remote Sensing: Theories, Tools and Challenges for the Community

In recent years, deep learning (DL), a re-branding of neural networks (NNs), has risen to the top in numerous areas, namely computer vision (CV), speech recognition, natural language processing, etc. Whereas remote sensing (RS) possesses a number of unique challenges, primarily related to sensors and applications, inevitably RS draws from many of the same theories as CV; e.g., statistics, fusion, and machine learning, to name a few. This means that the RS community should be aware of, if not at the leading edge of, of advancements like DL. Herein, we provide the most comprehensive survey of state-of-the-art RS DL research. We also review recent new developments in the DL field that can be used in DL for RS. Namely, we focus on theories, tools and challenges for the RS community. Specifically, we focus on unsolved challenges and opportunities as it relates to (i) inadequate data sets, (ii) human-understandable solutions for modelling physical phenomena, (iii) Big Data, (iv) non-traditional heterogeneous data sources, (v) DL architectures and learning algorithms for spectral, spatial and temporal data, (vi) transfer learning, (vii) an improved theoretical understanding of DL systems, (viii) high barriers to entry, and (ix) training and optimizing the DL.Comment: 64 pages, 411 references. To appear in Journal of Applied Remote Sensin