22 research outputs found
Filmy Cloud Removal on Satellite Imagery with Multispectral Conditional Generative Adversarial Nets
In this paper, we propose a method for cloud removal from visible light RGB
satellite images by extending the conditional Generative Adversarial Networks
(cGANs) from RGB images to multispectral images. Satellite images have been
widely utilized for various purposes, such as natural environment monitoring
(pollution, forest or rivers), transportation improvement and prompt emergency
response to disasters. However, the obscurity caused by clouds makes it
unstable to monitor the situation on the ground with the visible light camera.
Images captured by a longer wavelength are introduced to reduce the effects of
clouds. Synthetic Aperture Radar (SAR) is such an example that improves
visibility even the clouds exist. On the other hand, the spatial resolution
decreases as the wavelength increases. Furthermore, the images captured by long
wavelengths differs considerably from those captured by visible light in terms
of their appearance. Therefore, we propose a network that can remove clouds and
generate visible light images from the multispectral images taken as inputs.
This is achieved by extending the input channels of cGANs to be compatible with
multispectral images. The networks are trained to output images that are close
to the ground truth using the images synthesized with clouds over the ground
truth as inputs. In the available dataset, the proportion of images of the
forest or the sea is very high, which will introduce bias in the training
dataset if uniformly sampled from the original dataset. Thus, we utilize the
t-Distributed Stochastic Neighbor Embedding (t-SNE) to improve the problem of
bias in the training dataset. Finally, we confirm the feasibility of the
proposed network on the dataset of four bands images, which include three
visible light bands and one near-infrared (NIR) band
Cloud Removal in Satellite Images Using Spatiotemporal Generative Networks
Satellite images hold great promise for continuous environmental monitoring
and earth observation. Occlusions cast by clouds, however, can severely limit
coverage, making ground information extraction more difficult. Existing
pipelines typically perform cloud removal with simple temporal composites and
hand-crafted filters. In contrast, we cast the problem of cloud removal as a
conditional image synthesis challenge, and we propose a trainable
spatiotemporal generator network (STGAN) to remove clouds. We train our model
on a new large-scale spatiotemporal dataset that we construct, containing 97640
image pairs covering all continents. We demonstrate experimentally that the
proposed STGAN model outperforms standard models and can generate realistic
cloud-free images with high PSNR and SSIM values across a variety of
atmospheric conditions, leading to improved performance in downstream tasks
such as land cover classification.Comment: Accepted to WACV 202
Cloud Removal in Sentinel-2 Imagery using a Deep Residual Neural Network and SAR-Optical Data Fusion
Optical remote sensing imagery is at the core of many Earth observation activities. The regular, consistent and global-scale nature of the satellite data is exploited in many applications, such as cropland monitoring, climate change assessment, land-cover and land-use classification, and disaster assessment. However, one main problem severely affects the temporal and spatial availability of surface observations, namely cloud cover. The task of removing clouds from optical images has been subject of studies since decades. The advent of the Big Data era in satellite remote sensing opens new possibilities for tackling the problem using powerful data-driven deep learning methods. In this paper, a deep residual neural network architecture is designed to remove clouds from multispectral Sentinel-2 imagery. SAR-optical data fusion is used to exploit the synergistic properties of the two imaging systems to guide the image reconstruction. Additionally, a novel cloud-adaptive loss is proposed to maximize the retainment of original information. The network is trained and tested on a globally sampled dataset comprising real cloudy and cloud-free images. The proposed setup allows to remove even optically thick clouds by reconstructing an optical representation of the underlying land surface structure
Multi-Sensor Data Fusion for Cloud Removal in Global and All-Season Sentinel-2 Imagery
This work has been accepted by IEEE TGRS for publication. The majority of
optical observations acquired via spaceborne earth imagery are affected by
clouds. While there is numerous prior work on reconstructing cloud-covered
information, previous studies are oftentimes confined to narrowly-defined
regions of interest, raising the question of whether an approach can generalize
to a diverse set of observations acquired at variable cloud coverage or in
different regions and seasons. We target the challenge of generalization by
curating a large novel data set for training new cloud removal approaches and
evaluate on two recently proposed performance metrics of image quality and
diversity. Our data set is the first publically available to contain a global
sample of co-registered radar and optical observations, cloudy as well as
cloud-free. Based on the observation that cloud coverage varies widely between
clear skies and absolute coverage, we propose a novel model that can deal with
either extremes and evaluate its performance on our proposed data set. Finally,
we demonstrate the superiority of training models on real over synthetic data,
underlining the need for a carefully curated data set of real observations. To
facilitate future research, our data set is made available onlineComment: This work has been accepted by IEEE TGRS for publicatio
SAR TO OPTICAL IMAGE SYNTHESIS FOR CLOUD REMOVAL WITH GENERATIVE ADVERSARIAL NETWORKS
Optical imagery is often affected by the presence of clouds. Aiming to reduce their effects, different reconstruction techniques have been proposed in the last years. A common alternative is to extract data from active sensors, like Synthetic Aperture Radar (SAR), because they are almost independent on the atmospheric conditions and solar illumination. On the other hand, SAR images are more complex to interpret than optical images requiring particular handling. Recently, Conditional Generative Adversarial Networks (cGANs) have been widely used in different image generation tasks presenting state-of-the-art results. One application of cGANs is learning a nonlinear mapping function from two images of different domains. In this work, we combine the fact that SAR images are hardly affected by clouds with the ability of cGANS for image translation in order to map optical images from SAR ones so as to recover regions that are covered by clouds. Experimental results indicate that the proposed solution achieves better classification accuracy than SAR based classification
An Overview on the Generation and Detection of Synthetic and Manipulated Satellite Images
Due to the reduction of technological costs and the increase of satellites
launches, satellite images are becoming more popular and easier to obtain.
Besides serving benevolent purposes, satellite data can also be used for
malicious reasons such as misinformation. As a matter of fact, satellite images
can be easily manipulated relying on general image editing tools. Moreover,
with the surge of Deep Neural Networks (DNNs) that can generate realistic
synthetic imagery belonging to various domains, additional threats related to
the diffusion of synthetically generated satellite images are emerging. In this
paper, we review the State of the Art (SOTA) on the generation and manipulation
of satellite images. In particular, we focus on both the generation of
synthetic satellite imagery from scratch, and the semantic manipulation of
satellite images by means of image-transfer technologies, including the
transformation of images obtained from one type of sensor to another one. We
also describe forensic detection techniques that have been researched so far to
classify and detect synthetic image forgeries. While we focus mostly on
forensic techniques explicitly tailored to the detection of AI-generated
synthetic contents, we also review some methods designed for general splicing
detection, which can in principle also be used to spot AI manipulate imagesComment: 25 pages, 17 figures, 5 tables, APSIPA 202
Hybrid GAN and Spectral Angular Distance for Cloud Removal
This paper aims to present a new algorithm to remove thin clouds and retain information in corrupted images without the use of auxiliary data. By injecting physical properties into the cycle consistent generative adversarial network (GAN), we were able to convert a cloudy multispectral image to a cloudless image. To recover information beneath clouds and shadows we create a synthetic multispectral space to obtain illumination invariant features. Multispectral vectors were transformed from Cartesian coordinates to Polar coordinates to obtain spectral angular distance (SAD) then we employed them as input to train the deep neural network (DNN). Afterward, the outputs of DNN were transformed to Cartesian coordinates to obtain shadow and cloud-free multispectral images. The proposed method, Hybrid GAN-SAD yields trustworthy reconstructed results because of exploiting transparent information from certain multispectral bands to recover uncorrupted images
Diffusion Enhancement for Cloud Removal in Ultra-Resolution Remote Sensing Imagery
The presence of cloud layers severely compromises the quality and
effectiveness of optical remote sensing (RS) images. However, existing
deep-learning (DL)-based Cloud Removal (CR) techniques encounter difficulties
in accurately reconstructing the original visual authenticity and detailed
semantic content of the images. To tackle this challenge, this work proposes to
encompass enhancements at the data and methodology fronts. On the data side, an
ultra-resolution benchmark named CUHK Cloud Removal (CUHK-CR) of 0.5m spatial
resolution is established. This benchmark incorporates rich detailed textures
and diverse cloud coverage, serving as a robust foundation for designing and
assessing CR models. From the methodology perspective, a novel diffusion-based
framework for CR called Diffusion Enhancement (DE) is proposed to perform
progressive texture detail recovery, which mitigates the training difficulty
with improved inference accuracy. Additionally, a Weight Allocation (WA)
network is developed to dynamically adjust the weights for feature fusion,
thereby further improving performance, particularly in the context of
ultra-resolution image generation. Furthermore, a coarse-to-fine training
strategy is applied to effectively expedite training convergence while reducing
the computational complexity required to handle ultra-resolution images.
Extensive experiments on the newly established CUHK-CR and existing datasets
such as RICE confirm that the proposed DE framework outperforms existing
DL-based methods in terms of both perceptual quality and signal fidelity