65 research outputs found
Unsupervised Sparse Dirichlet-Net for Hyperspectral Image Super-Resolution
In many computer vision applications, obtaining images of high resolution in
both the spatial and spectral domains are equally important. However, due to
hardware limitations, one can only expect to acquire images of high resolution
in either the spatial or spectral domains. This paper focuses on hyperspectral
image super-resolution (HSI-SR), where a hyperspectral image (HSI) with low
spatial resolution (LR) but high spectral resolution is fused with a
multispectral image (MSI) with high spatial resolution (HR) but low spectral
resolution to obtain HR HSI. Existing deep learning-based solutions are all
supervised that would need a large training set and the availability of HR HSI,
which is unrealistic. Here, we make the first attempt to solving the HSI-SR
problem using an unsupervised encoder-decoder architecture that carries the
following uniquenesses. First, it is composed of two encoder-decoder networks,
coupled through a shared decoder, in order to preserve the rich spectral
information from the HSI network. Second, the network encourages the
representations from both modalities to follow a sparse Dirichlet distribution
which naturally incorporates the two physical constraints of HSI and MSI.
Third, the angular difference between representations are minimized in order to
reduce the spectral distortion. We refer to the proposed architecture as
unsupervised Sparse Dirichlet-Net, or uSDN. Extensive experimental results
demonstrate the superior performance of uSDN as compared to the
state-of-the-art.Comment: Accepted by The IEEE Conference on Computer Vision and Pattern
Recognition (CVPR 2018, Spotlight
Guided Deep Decoder: Unsupervised Image Pair Fusion
The fusion of input and guidance images that have a tradeoff in their
information (e.g., hyperspectral and RGB image fusion or pansharpening) can be
interpreted as one general problem. However, previous studies applied a
task-specific handcrafted prior and did not address the problems with a unified
approach. To address this limitation, in this study, we propose a guided deep
decoder network as a general prior. The proposed network is composed of an
encoder-decoder network that exploits multi-scale features of a guidance image
and a deep decoder network that generates an output image. The two networks are
connected by feature refinement units to embed the multi-scale features of the
guidance image into the deep decoder network. The proposed network allows the
network parameters to be optimized in an unsupervised way without training
data. Our results show that the proposed network can achieve state-of-the-art
performance in various image fusion problems.Comment: ECCV 202
A convex formulation for hyperspectral image superresolution via subspace-based regularization
Hyperspectral remote sensing images (HSIs) usually have high spectral
resolution and low spatial resolution. Conversely, multispectral images (MSIs)
usually have low spectral and high spatial resolutions. The problem of
inferring images which combine the high spectral and high spatial resolutions
of HSIs and MSIs, respectively, is a data fusion problem that has been the
focus of recent active research due to the increasing availability of HSIs and
MSIs retrieved from the same geographical area.
We formulate this problem as the minimization of a convex objective function
containing two quadratic data-fitting terms and an edge-preserving regularizer.
The data-fitting terms account for blur, different resolutions, and additive
noise. The regularizer, a form of vector Total Variation, promotes
piecewise-smooth solutions with discontinuities aligned across the
hyperspectral bands.
The downsampling operator accounting for the different spatial resolutions,
the non-quadratic and non-smooth nature of the regularizer, and the very large
size of the HSI to be estimated lead to a hard optimization problem. We deal
with these difficulties by exploiting the fact that HSIs generally "live" in a
low-dimensional subspace and by tailoring the Split Augmented Lagrangian
Shrinkage Algorithm (SALSA), which is an instance of the Alternating Direction
Method of Multipliers (ADMM), to this optimization problem, by means of a
convenient variable splitting. The spatial blur and the spectral linear
operators linked, respectively, with the HSI and MSI acquisition processes are
also estimated, and we obtain an effective algorithm that outperforms the
state-of-the-art, as illustrated in a series of experiments with simulated and
real-life data.Comment: IEEE Trans. Geosci. Remote Sens., to be publishe
Hyperspectral and Multispectral Image Fusion using Optimized Twin Dictionaries
Spectral or spatial dictionary has been widely used in fusing low-spatial-resolution hyperspectral (LH) images and high-spatial-resolution multispectral (HM) images. However, only using spectral dictionary is insufficient for preserving spatial information, and vice versa. To address this problem, a new LH and HM image fusion method termed OTD using optimized twin dictionaries is proposed in this paper. The fusion problem of OTD is formulated analytically in the framework of sparse representation, as an optimization of twin spectral-spatial dictionaries and their corresponding sparse coefficients. More specifically, the spectral dictionary representing the generalized spectrums and its spectral sparse coefficients are optimized by utilizing the observed LH and HM images in the spectral domain; and the spatial dictionary representing the spatial information and its spatial sparse coefficients are optimized by modeling the rest of high-frequency information in the spatial domain. In addition, without non-negative constraints, the alternating direction methods of multipliers (ADMM) are employed to implement the above optimization process. Comparison results with the related state-of-the-art fusion methods on various datasets demonstrate that our proposed OTD method achieves a better fusion performance in both spatial and spectral domains
Panchromatic and multispectral image fusion for remote sensing and earth observation: Concepts, taxonomy, literature review, evaluation methodologies and challenges ahead
Panchromatic and multispectral image fusion, termed pan-sharpening, is to merge the spatial and spectral information of the source images into a fused one, which has a higher spatial and spectral resolution and is more reliable for downstream tasks compared with any of the source images. It has been widely applied to image interpretation and pre-processing of various applications. A large number of methods have been proposed to achieve better fusion results by considering the spatial and spectral relationships among panchromatic and multispectral images. In recent years, the fast development of artificial intelligence (AI) and deep learning (DL) has significantly enhanced the development of pan-sharpening techniques. However, this field lacks a comprehensive overview of recent advances boosted by the rise of AI and DL. This paper provides a comprehensive review of a variety of pan-sharpening methods that adopt four different paradigms, i.e., component substitution, multiresolution analysis, degradation model, and deep neural networks. As an important aspect of pan-sharpening, the evaluation of the fused image is also outlined to present various assessment methods in terms of reduced-resolution and full-resolution quality measurement. Then, we conclude this paper by discussing the existing limitations, difficulties, and challenges of pan-sharpening techniques, datasets, and quality assessment. In addition, the survey summarizes the development trends in these areas, which provide useful methodological practices for researchers and professionals. Finally, the developments in pan-sharpening are summarized in the conclusion part. The aim of the survey is to serve as a referential starting point for newcomers and a common point of agreement around the research directions to be followed in this exciting area
- …