Multitemporal Relearning with Convolutional LSTM Models for Land Use Classification

In this article, we present a novel hybrid framework, which integrates spatial–temporal semantic segmentation with postclassification relearning, for multitemporal land use and land cover (LULC) classification based on very high resolution (VHR) satellite imagery. To efficiently obtain optimal multitemporal LULC classification maps, the hybrid framework utilizes a spatial–temporal semantic segmentation model to harness temporal dependency for extracting high-level spatial–temporal features. In addition, the principle of postclassification relearning is adopted to efficiently optimize model output. Thereby, the initial outcome of a semantic segmentation model is provided to a subsequent model via an extended input space to guide the learning of discriminative feature representations in an end-to-end fashion. Last, object-based voting is coupled with postclassification relearning for coping with the high intraclass and low interclass variances. The framework was tested with two different postclassification relearning strategies (i.e., pixel-based relearning and object-based relearning) and three convolutional neural network models, i.e., UNet, a simple Convolutional LSTM, and a UNet Convolutional-LSTM. The experiments were conducted on two datasets with LULC labels that contain rich semantic information and variant building morphologic features (e.g., informal settlements). Each dataset contains four time steps from WorldView-2 and Quickbird imagery. The experimental results unambiguously underline that the proposed framework is efficient in terms of classifying complex LULC maps with multitemporal VHR images

Image super-resolution with dense-sampling residual channel-spatial attention networks for multi-temporal remote sensing image classification

Image super-resolution (SR) techniques can benefit a wide range of applications in the remote sensing (RS) community, including image classification. This issue is particularly relevant for image classification on time series data, considering RS datasets that feature long temporal coverage generally have a limited spatial resolution. Recent advances in deep learning brought new opportunities for enhancing the spatial resolution of historic RS data. Numerous convolutional neural network (CNN)-based methods showed superior performance in terms of developing efficient end-to-end SR models for natural images. However, such models were rarely exploited for promoting image classification based on multispectral RS data. This paper proposes a novel CNNbased framework to enhance the spatial resolution of time series multispectral RS images. Thereby, the proposed SR model employs Residual Channel Attention Networks (RCAN) as a backbone structure, whereas based on this structure the proposed models uniquely integrate tailored channel-spatial attention and dense-sampling mechanisms for performance improvement. Subsequently, state-of-the-art CNN-based classifiers are incorporated to produce classification maps based on the enhanced time series data. The experiments proved that the proposed SR model can enable unambiguously better performance compared to RCAN and other (deep learning-based) SR techniques, especially in a domain adaptation context, i.e., leveraging Sentinel-2 images for generating SR Landsat images. Furthermore, the experimental results confirmed that the enhanced multi-temporal RS images can bring substantial improvement on fine-grained multi-temporal land use classification

A Comprehensive Survey of Deep Learning in Remote Sensing: Theories, Tools and Challenges for the Community

In recent years, deep learning (DL), a re-branding of neural networks (NNs), has risen to the top in numerous areas, namely computer vision (CV), speech recognition, natural language processing, etc. Whereas remote sensing (RS) possesses a number of unique challenges, primarily related to sensors and applications, inevitably RS draws from many of the same theories as CV; e.g., statistics, fusion, and machine learning, to name a few. This means that the RS community should be aware of, if not at the leading edge of, of advancements like DL. Herein, we provide the most comprehensive survey of state-of-the-art RS DL research. We also review recent new developments in the DL field that can be used in DL for RS. Namely, we focus on theories, tools and challenges for the RS community. Specifically, we focus on unsolved challenges and opportunities as it relates to (i) inadequate data sets, (ii) human-understandable solutions for modelling physical phenomena, (iii) Big Data, (iv) non-traditional heterogeneous data sources, (v) DL architectures and learning algorithms for spectral, spatial and temporal data, (vi) transfer learning, (vii) an improved theoretical understanding of DL systems, (viii) high barriers to entry, and (ix) training and optimizing the DL.Comment: 64 pages, 411 references. To appear in Journal of Applied Remote Sensin