SSL4EO-S12: A Large-Scale Multi-Modal, Multi-Temporal Dataset for Self-Supervised Learning in Earth Observation

Self-supervised pre-training bears potential to generate expressive representations without human annotation. Most pre-training in Earth observation (EO) are based on ImageNet or medium-size, labeled remote sensing (RS) datasets. We share an unlabeled RS dataset SSL4EO-S12 (Self-Supervised Learning for Earth Observation - Sentinel-1/2) to assemble a large-scale, global, multimodal, and multi-seasonal corpus of satellite imagery from the ESA Sentinel-1 \& -2 satellite missions. For EO applications we demonstrate SSL4EO-S12 to succeed in self-supervised pre-training for a set of methods: MoCo-v2, DINO, MAE, and data2vec. Resulting models yield downstream performance close to, or surpassing accuracy measures of supervised learning. In addition, pre-training on SSL4EO-S12 excels compared to existing datasets. We make openly available the dataset, related source code, and pre-trained models at https://github.com/zhu-xlab/SSL4EO-S12.Comment: Accepted by IEEE Geoscience and Remote Sensing Magazine. 18 page

Self-supervised Learning in Remote Sensing: A Review

In deep learning research, self-supervised learning (SSL) has received great attention triggering interest within both the computer vision and remote sensing communities. While there has been a big success in computer vision, most of the potential of SSL in the domain of earth observation remains locked. In this paper, we provide an introduction to, and a review of the concepts and latest developments in SSL for computer vision in the context of remote sensing. Further, we provide a preliminary benchmark of modern SSL algorithms on popular remote sensing datasets, verifying the potential of SSL in remote sensing and providing an extended study on data augmentations. Finally, we identify a list of promising directions of future research in SSL for earth observation (SSL4EO) to pave the way for fruitful interaction of both domains.Comment: Accepted by IEEE Geoscience and Remote Sensing Magazine. 32 pages, 22 content page

From natural images to spaceborne imagery: an empirical study of self-supervised learning for Earth observation

In this work, we provide an empirical study on the performance of self-supervised learning for spaceborne imagery. Specifically, we conduct extensive experiments on three well-known remote sensing datasets BigEarthNet, SEN12MS and LCZ42 using four representative state-of-the-art SSL algorithms MoCo, SwAV, SimSiam and Barlow Twins. We analyze the performance of SSL algorithms under different data regimes and compare them to vanilla supervised learning. In addition, we explore the impact of data augmentation, which is known to be a key component in the design and tuning of modern SSL methods

Semi-Supervised Learning for Hyperspectral Images by Non-Parametrically Predicting View Assignment

Hyperspectral image (HSI) classification is gaining a lot of momentum in present time because of high inherent spectral information within the images. However, these images suffer from the problem of curse of dimensionality and usually require a large number samples for tasks such as classification, especially in supervised setting. Recently, to effectively train the deep learning models with minimal labelled samples, the unlabeled samples are also being leveraged in self-supervised and semi-supervised setting. In this work, we leverage the idea of semi-supervised learning to assist the discriminative self-supervised pretraining of the models. The proposed method takes different augmented views of the unlabeled samples as input and assigns them the same pseudo-label corresponding to the labelled sample from the downstream task. We train our model on two HSI datasets, anemly Houston dataset (from data fusion contest, 2013) and Pavia university dataset, and show that the proposed approach performs better than self-supervised approach and supervised training

SSL4EO-L: Datasets and Foundation Models for Landsat Imagery

The Landsat program is the longest-running Earth observation program in history, with 50+ years of data acquisition by 8 satellites. The multispectral imagery captured by sensors onboard these satellites is critical for a wide range of scientific fields. Despite the increasing popularity of deep learning and remote sensing, the majority of researchers still use decision trees and random forests for Landsat image analysis due to the prevalence of small labeled datasets and lack of foundation models. In this paper, we introduce SSL4EO-L, the first ever dataset designed for Self-Supervised Learning for Earth Observation for the Landsat family of satellites (including 3 sensors and 2 product levels) and the largest Landsat dataset in history (5M image patches). Additionally, we modernize and re-release the L7 Irish and L8 Biome cloud detection datasets, and introduce the first ML benchmark datasets for Landsats 4-5 TM and Landsat 7 ETM+ SR. Finally, we pre-train the first foundation models for Landsat imagery using SSL4EO-L and evaluate their performance on multiple semantic segmentation tasks. All datasets and model weights are available via the TorchGeo (https://github.com/microsoft/torchgeo) library, making reproducibility and experimentation easy, and enabling scientific advancements in the burgeoning field of remote sensing for a multitude of downstream applications

SSL4EO-L: Datasets and Foundation Models for Landsat Imagery

The Landsat program is the longest-running Earth observation program in history, with 50+ years of data acquisition by 8 satellites. The multispectral imagery captured by sensors onboard these satellites is critical for a wide range of scientific fields. Despite the increasing popularity of deep learning and remote sensing, the majority of researchers still use decision trees and random forests for Landsat image analysis due to the prevalence of small labeled datasets and lack of foundation models. In this paper, we introduce SSL4EO-L, the first ever dataset designed for Self-Supervised Learning for Earth Observation for the Landsat family of satellites (including 3 sensors and 2 product levels) and the largest Landsat dataset in history (5M image patches). Additionally, we modernize and re-release the L7 Irish and L8 Biome cloud detection datasets, and introduce the first ML benchmark datasets for Landsats 4–5 TM and Landsat 7 ETM+ SR. Finally, we pre-train the first foundation models for Landsat imagery using SSL4EO-L and evaluate their performance on multiple semantic segmentation tasks. All datasets and model weights are available via the TorchGeo library, making reproducibility and experimentation easy, and enabling scientific advancements in the burgeoning field of remote sensing for a multitude of downstream applications

Automatic Detection of Building Displacements Through Unsupervised Learning From InSAR Data

We introduce an unsupervised learning method that aims to identify building anomalies using Interferometric Synthetic Aperture Radar (InSAR) time-series data. Specifically, we leverage data obtained from the European Ground Motion Service to develop our proposed approach, which employs a long short-term memory autoencoder model and a reconstruction loss function based on a soft variant of the dynamic time warping, namely “soft-DTW”. We deliberately utilize this loss function for its ability to compare time-series that are not aligned in time, unlike the other conventional reconstruction losses that do not account for time shifts. Moreover, we enhance the performance of anomaly detection by smoothing inputs with a Hann window and defining the learning objective to reconstruct the time order of randomly permuted input series. Our experimental findings, based on persistent scatterer data from Rome, Italy, reveal that our method outperforms several unsupervised machine learning and deep learning methods in detecting various types of building displacement, such as trend, noise, and step anomalies. Additionally, quantitative and qualitative evaluations validate the efficacy of our approach in identifying potentially anomalous buildings. Thus, our method offers a promising solution for detecting anomalies in PS-InSAR time-series, which could have substantial implications in the fields of urban monitoring and infrastructure management

Self Supervised Learning for Few Shot Hyperspectral Image Classification

Deep learning has proven to be a very effective approach for Hyperspectral Image (HSI) classification. However, deep neural networks require large annotated datasets to generalize well. This limits the applicability of deep learning for HSI classification, where manually labelling thousands of pixels for every scene is impractical. In this paper, we propose to leverage Self Supervised Learning (SSL) for HSI classification. We show that by pre-training an encoder on unlabeled pixels using Barlow-Twins, a state-of-the-art SSL algorithm, we can obtain accurate models with a handful of labels. Experimental results demonstrate that this approach significantly outperforms vanilla supervised learning

Automatic Detection of Building Displacements Through Unsupervised Learning From InSAR Data

We introduce an unsupervised learning method that aims to identify building anomalies using Interferometric Synthetic Aperture Radar (InSAR) time-series data. Specifically, we leverage data obtained from the European Ground Motion Service to develop our proposed approach, which employs a long short-term memory autoencoder model and a reconstruction loss function based on a soft variant of the dynamic time warping, namely “soft-DTW”. We deliberately utilize this loss function for its ability to compare time-series that are not aligned in time, unlike the other conventional reconstruction losses that do not account for time shifts. Moreover, we enhance the performance of anomaly detection by smoothing inputs with a Hann window and defining the learning objective to reconstruct the time order of randomly permuted input series. Our experimental findings, based on persistent scatterer data from Rome, Italy, reveal that our method outperforms several unsupervised machine learning and deep learning methods in detecting various types of building displacement, such as trend, noise, and step anomalies. Additionally, quantitative and qualitative evaluations validate the efficacy of our approach in identifying potentially anomalous buildings. Thus, our method offers a promising solution for detecting anomalies in PS-InSAR time-series, which could have substantial implications in the fields of urban monitoring and infrastructure management