301 research outputs found
A convex formulation for hyperspectral image superresolution via subspace-based regularization
Hyperspectral remote sensing images (HSIs) usually have high spectral
resolution and low spatial resolution. Conversely, multispectral images (MSIs)
usually have low spectral and high spatial resolutions. The problem of
inferring images which combine the high spectral and high spatial resolutions
of HSIs and MSIs, respectively, is a data fusion problem that has been the
focus of recent active research due to the increasing availability of HSIs and
MSIs retrieved from the same geographical area.
We formulate this problem as the minimization of a convex objective function
containing two quadratic data-fitting terms and an edge-preserving regularizer.
The data-fitting terms account for blur, different resolutions, and additive
noise. The regularizer, a form of vector Total Variation, promotes
piecewise-smooth solutions with discontinuities aligned across the
hyperspectral bands.
The downsampling operator accounting for the different spatial resolutions,
the non-quadratic and non-smooth nature of the regularizer, and the very large
size of the HSI to be estimated lead to a hard optimization problem. We deal
with these difficulties by exploiting the fact that HSIs generally "live" in a
low-dimensional subspace and by tailoring the Split Augmented Lagrangian
Shrinkage Algorithm (SALSA), which is an instance of the Alternating Direction
Method of Multipliers (ADMM), to this optimization problem, by means of a
convenient variable splitting. The spatial blur and the spectral linear
operators linked, respectively, with the HSI and MSI acquisition processes are
also estimated, and we obtain an effective algorithm that outperforms the
state-of-the-art, as illustrated in a series of experiments with simulated and
real-life data.Comment: IEEE Trans. Geosci. Remote Sens., to be publishe
Multi-scale spatial fusion and regularization induced unsupervised auxiliary task CNN model for deep super-resolution of hyperspectral image.
Hyperspectral images (HSI) features rich spectral information in many narrow bands but at a cost of a relatively low spatial resolution. As such, various methods have been developed for enhancing the spatial resolution of the low-resolution HSI (Lr-HSI) by fusing it with high-resolution multispectral images (Hr-MSI). The difference in spectrum range and spatial dimensions between the Lr-HSI and Hr-SI have been fundamental but challenging for multispectral/hyperspectral (MS/HS) fusion. In this paper, a multi-scale spatial fusion and regularization induced auxiliary task (MSAT) based CNN model is proposed for deep super-resolution of HSI, where a Lr-HSI is fused with a Hr-MSI to reconstruct a high-resolution HSI (Hr-HSI) counterpart. The multi-scale fusion is used to efficiently address the discrepancy in spatial resolutions between two inputs. Based on the general assumption that the acquired Hr-MSI and the reconstructed Hr-HSI share similar underlying characteristics, the auxiliary task is proposed to learn a representation for improved generality of the model and reduced overfitting. Experimental results on three public datasets have validated the effectiveness of our approach in comparison with several state-of-the-art methods
A Spectral Diffusion Prior for Hyperspectral Image Super-Resolution
Fusion-based hyperspectral image (HSI) super-resolution aims to produce a
high-spatial-resolution HSI by fusing a low-spatial-resolution HSI and a
high-spatial-resolution multispectral image. Such a HSI super-resolution
process can be modeled as an inverse problem, where the prior knowledge is
essential for obtaining the desired solution. Motivated by the success of
diffusion models, we propose a novel spectral diffusion prior for fusion-based
HSI super-resolution. Specifically, we first investigate the spectrum
generation problem and design a spectral diffusion model to model the spectral
data distribution. Then, in the framework of maximum a posteriori, we keep the
transition information between every two neighboring states during the reverse
generative process, and thereby embed the knowledge of trained spectral
diffusion model into the fusion problem in the form of a regularization term.
At last, we treat each generation step of the final optimization problem as its
subproblem, and employ the Adam to solve these subproblems in a reverse
sequence. Experimental results conducted on both synthetic and real datasets
demonstrate the effectiveness of the proposed approach. The code of the
proposed approach will be available on https://github.com/liuofficial/SDP
Hyperspectral and Multispectral Image Fusion Using the Conditional Denoising Diffusion Probabilistic Model
Hyperspectral images (HSI) have a large amount of spectral information
reflecting the characteristics of matter, while their spatial resolution is low
due to the limitations of imaging technology. Complementary to this are
multispectral images (MSI), e.g., RGB images, with high spatial resolution but
insufficient spectral bands. Hyperspectral and multispectral image fusion is a
technique for acquiring ideal images that have both high spatial and high
spectral resolution cost-effectively. Many existing HSI and MSI fusion
algorithms rely on known imaging degradation models, which are often not
available in practice. In this paper, we propose a deep fusion method based on
the conditional denoising diffusion probabilistic model, called DDPM-Fus.
Specifically, the DDPM-Fus contains the forward diffusion process which
gradually adds Gaussian noise to the high spatial resolution HSI (HrHSI) and
another reverse denoising process which learns to predict the desired HrHSI
from its noisy version conditioning on the corresponding high spatial
resolution MSI (HrMSI) and low spatial resolution HSI (LrHSI). Once the
training is completes, the proposed DDPM-Fus implements the reverse process on
the test HrMSI and LrHSI to generate the fused HrHSI. Experiments conducted on
one indoor and two remote sensing datasets show the superiority of the proposed
model when compared with other advanced deep learningbased fusion methods. The
codes of this work will be opensourced at this address:
https://github.com/shuaikaishi/DDPMFus for reproducibility
Deep learning in remote sensing: a review
Standing at the paradigm shift towards data-intensive science, machine
learning techniques are becoming increasingly important. In particular, as a
major breakthrough in the field, deep learning has proven as an extremely
powerful tool in many fields. Shall we embrace deep learning as the key to all?
Or, should we resist a 'black-box' solution? There are controversial opinions
in the remote sensing community. In this article, we analyze the challenges of
using deep learning for remote sensing data analysis, review the recent
advances, and provide resources to make deep learning in remote sensing
ridiculously simple to start with. More importantly, we advocate remote sensing
scientists to bring their expertise into deep learning, and use it as an
implicit general model to tackle unprecedented large-scale influential
challenges, such as climate change and urbanization.Comment: Accepted for publication IEEE Geoscience and Remote Sensing Magazin
Hyperspectral and Multispectral Image Fusion using Optimized Twin Dictionaries
Spectral or spatial dictionary has been widely used in fusing low-spatial-resolution hyperspectral (LH) images and high-spatial-resolution multispectral (HM) images. However, only using spectral dictionary is insufficient for preserving spatial information, and vice versa. To address this problem, a new LH and HM image fusion method termed OTD using optimized twin dictionaries is proposed in this paper. The fusion problem of OTD is formulated analytically in the framework of sparse representation, as an optimization of twin spectral-spatial dictionaries and their corresponding sparse coefficients. More specifically, the spectral dictionary representing the generalized spectrums and its spectral sparse coefficients are optimized by utilizing the observed LH and HM images in the spectral domain; and the spatial dictionary representing the spatial information and its spatial sparse coefficients are optimized by modeling the rest of high-frequency information in the spatial domain. In addition, without non-negative constraints, the alternating direction methods of multipliers (ADMM) are employed to implement the above optimization process. Comparison results with the related state-of-the-art fusion methods on various datasets demonstrate that our proposed OTD method achieves a better fusion performance in both spatial and spectral domains
Multi-scale Adaptive Fusion Network for Hyperspectral Image Denoising
Removing the noise and improving the visual quality of hyperspectral images
(HSIs) is challenging in academia and industry. Great efforts have been made to
leverage local, global or spectral context information for HSI denoising.
However, existing methods still have limitations in feature interaction
exploitation among multiple scales and rich spectral structure preservation. In
view of this, we propose a novel solution to investigate the HSI denoising
using a Multi-scale Adaptive Fusion Network (MAFNet), which can learn the
complex nonlinear mapping between clean and noisy HSI. Two key components
contribute to improving the hyperspectral image denoising: A progressively
multiscale information aggregation network and a co-attention fusion module.
Specifically, we first generate a set of multiscale images and feed them into a
coarse-fusion network to exploit the contextual texture correlation.
Thereafter, a fine fusion network is followed to exchange the information
across the parallel multiscale subnetworks. Furthermore, we design a
co-attention fusion module to adaptively emphasize informative features from
different scales, and thereby enhance the discriminative learning capability
for denoising. Extensive experiments on synthetic and real HSI datasets
demonstrate that the proposed MAFNet has achieved better denoising performance
than other state-of-the-art techniques. Our codes are available at
\verb'https://github.com/summitgao/MAFNet'.Comment: IEEE JSTASRS 2023, code at: https://github.com/summitgao/MAFNe
- …