377 research outputs found
Recent Advances in Image Restoration with Applications to Real World Problems
In the past few decades, imaging hardware has improved tremendously in terms of resolution, making widespread usage of images in many diverse applications on Earth and planetary missions. However, practical issues associated with image acquisition are still affecting image quality. Some of these issues such as blurring, measurement noise, mosaicing artifacts, low spatial or spectral resolution, etc. can seriously affect the accuracy of the aforementioned applications. This book intends to provide the reader with a glimpse of the latest developments and recent advances in image restoration, which includes image super-resolution, image fusion to enhance spatial, spectral resolution, and temporal resolutions, and the generation of synthetic images using deep learning techniques. Some practical applications are also included
SpaceNet MVOI: a Multi-View Overhead Imagery Dataset
Detection and segmentation of objects in overheard imagery is a challenging
task. The variable density, random orientation, small size, and
instance-to-instance heterogeneity of objects in overhead imagery calls for
approaches distinct from existing models designed for natural scene datasets.
Though new overhead imagery datasets are being developed, they almost
universally comprise a single view taken from directly overhead ("at nadir"),
failing to address a critical variable: look angle. By contrast, views vary in
real-world overhead imagery, particularly in dynamic scenarios such as natural
disasters where first looks are often over 40 degrees off-nadir. This
represents an important challenge to computer vision methods, as changing view
angle adds distortions, alters resolution, and changes lighting. At present,
the impact of these perturbations for algorithmic detection and segmentation of
objects is untested. To address this problem, we present an open source
Multi-View Overhead Imagery dataset, termed SpaceNet MVOI, with 27 unique looks
from a broad range of viewing angles (-32.5 degrees to 54.0 degrees). Each of
these images cover the same 665 square km geographic extent and are annotated
with 126,747 building footprint labels, enabling direct assessment of the
impact of viewpoint perturbation on model performance. We benchmark multiple
leading segmentation and object detection models on: (1) building detection,
(2) generalization to unseen viewing angles and resolutions, and (3)
sensitivity of building footprint extraction to changes in resolution. We find
that state of the art segmentation and object detection models struggle to
identify buildings in off-nadir imagery and generalize poorly to unseen views,
presenting an important benchmark to explore the broadly relevant challenge of
detecting small, heterogeneous target objects in visually dynamic contexts.Comment: Accepted into IEEE International Conference on Computer Vision (ICCV)
201
Unsupervised Hyperspectral and Multispectral Images Fusion Based on the Cycle Consistency
Hyperspectral images (HSI) with abundant spectral information reflected
materials property usually perform low spatial resolution due to the hardware
limits. Meanwhile, multispectral images (MSI), e.g., RGB images, have a high
spatial resolution but deficient spectral signatures. Hyperspectral and
multispectral image fusion can be cost-effective and efficient for acquiring
both high spatial resolution and high spectral resolution images. Many of the
conventional HSI and MSI fusion algorithms rely on known spatial degradation
parameters, i.e., point spread function, spectral degradation parameters,
spectral response function, or both of them. Another class of deep
learning-based models relies on the ground truth of high spatial resolution HSI
and needs large amounts of paired training images when working in a supervised
manner. Both of these models are limited in practical fusion scenarios. In this
paper, we propose an unsupervised HSI and MSI fusion model based on the cycle
consistency, called CycFusion. The CycFusion learns the domain transformation
between low spatial resolution HSI (LrHSI) and high spatial resolution MSI
(HrMSI), and the desired high spatial resolution HSI (HrHSI) are considered to
be intermediate feature maps in the transformation networks. The CycFusion can
be trained with the objective functions of marginal matching in single
transform and cycle consistency in double transforms. Moreover, the estimated
PSF and SRF are embedded in the model as the pre-training weights, which
further enhances the practicality of our proposed model. Experiments conducted
on several datasets show that our proposed model outperforms all compared
unsupervised fusion methods. The codes of this paper will be available at this
address: https: //github.com/shuaikaishi/CycFusion for reproducibility
Learnable Reconstruction Methods from RGB Images to Hyperspectral Imaging: A Survey
Hyperspectral imaging enables versatile applications due to its competence in
capturing abundant spatial and spectral information, which are crucial for
identifying substances. However, the devices for acquiring hyperspectral images
are expensive and complicated. Therefore, many alternative spectral imaging
methods have been proposed by directly reconstructing the hyperspectral
information from lower-cost, more available RGB images. We present a thorough
investigation of these state-of-the-art spectral reconstruction methods from
the widespread RGB images. A systematic study and comparison of more than 25
methods has revealed that most of the data-driven deep learning methods are
superior to prior-based methods in terms of reconstruction accuracy and quality
despite lower speeds. This comprehensive review can serve as a fruitful
reference source for peer researchers, thus further inspiring future
development directions in related domains
A Latent Encoder Coupled Generative Adversarial Network (LE-GAN) for Efficient Hyperspectral Image Super-resolution
Realistic hyperspectral image (HSI) super-resolution (SR) techniques aim to
generate a high-resolution (HR) HSI with higher spectral and spatial fidelity
from its low-resolution (LR) counterpart. The generative adversarial network
(GAN) has proven to be an effective deep learning framework for image
super-resolution. However, the optimisation process of existing GAN-based
models frequently suffers from the problem of mode collapse, leading to the
limited capacity of spectral-spatial invariant reconstruction. This may cause
the spectral-spatial distortion on the generated HSI, especially with a large
upscaling factor. To alleviate the problem of mode collapse, this work has
proposed a novel GAN model coupled with a latent encoder (LE-GAN), which can
map the generated spectral-spatial features from the image space to the latent
space and produce a coupling component to regularise the generated samples.
Essentially, we treat an HSI as a high-dimensional manifold embedded in a
latent space. Thus, the optimisation of GAN models is converted to the problem
of learning the distributions of high-resolution HSI samples in the latent
space, making the distributions of the generated super-resolution HSIs closer
to those of their original high-resolution counterparts. We have conducted
experimental evaluations on the model performance of super-resolution and its
capability in alleviating mode collapse. The proposed approach has been tested
and validated based on two real HSI datasets with different sensors (i.e.
AVIRIS and UHD-185) for various upscaling factors and added noise levels, and
compared with the state-of-the-art super-resolution models (i.e. HyCoNet, LTTR,
BAGAN, SR- GAN, WGAN).Comment: 18 pages, 10 figure
- …