23 research outputs found
Spatial-Spectral Transformer for Hyperspectral Image Denoising
Hyperspectral image (HSI) denoising is a crucial preprocessing procedure for
the subsequent HSI applications. Unfortunately, though witnessing the
development of deep learning in HSI denoising area, existing convolution-based
methods face the trade-off between computational efficiency and capability to
model non-local characteristics of HSI. In this paper, we propose a
Spatial-Spectral Transformer (SST) to alleviate this problem. To fully explore
intrinsic similarity characteristics in both spatial dimension and spectral
dimension, we conduct non-local spatial self-attention and global spectral
self-attention with Transformer architecture. The window-based spatial
self-attention focuses on the spatial similarity beyond the neighboring region.
While, spectral self-attention exploits the long-range dependencies between
highly correlative bands. Experimental results show that our proposed method
outperforms the state-of-the-art HSI denoising methods in quantitative quality
and visual results
CBANet: an end-to-end cross band 2-D attention network for hyperspectral change detection in remote sensing.
As a fundamental task in remote sensing observation of the earth, change detection using hyperspectral images (HSI) features high accuracy due to the combination of the rich spectral and spatial information, especially for identifying land-cover variations in bi-temporal HSIs. Relying on the image difference, existing HSI change detection methods fail to preserve the spectral characteristics and suffer from high data dimensionality, making them extremely challenging to deal with changing areas of various sizes. To tackle these challenges, we propose a cross-band 2-D self-attention Network (CBANet) for end-to-end HSI change detection. By embedding a cross-band feature extraction module into a 2-D spatial-spectral self-attention module, CBANet is highly capable of extracting the spectral difference of matching pixels by considering the correlation between adjacent pixels. The CBANet has shown three key advantages: 1) less parameters and high efficiency; 2) high efficacy of extracting representative spectral information from bi-temporal images; and 3) high stability and accuracy for identifying both sparse sporadic changing pixels and large changing areas whilst preserving the edges. Comprehensive experiments on three publicly available datasets have fully validated the efficacy and efficiency of the proposed methodology
Pixel Adaptive Deep Unfolding Transformer for Hyperspectral Image Reconstruction
Hyperspectral Image (HSI) reconstruction has made gratifying progress with
the deep unfolding framework by formulating the problem into a data module and
a prior module. Nevertheless, existing methods still face the problem of
insufficient matching with HSI data. The issues lie in three aspects: 1) fixed
gradient descent step in the data module while the degradation of HSI is
agnostic in the pixel-level. 2) inadequate prior module for 3D HSI cube. 3)
stage interaction ignoring the differences in features at different stages. To
address these issues, in this work, we propose a Pixel Adaptive Deep Unfolding
Transformer (PADUT) for HSI reconstruction. In the data module, a pixel
adaptive descent step is employed to focus on pixel-level agnostic degradation.
In the prior module, we introduce the Non-local Spectral Transformer (NST) to
emphasize the 3D characteristics of HSI for recovering. Moreover, inspired by
the diverse expression of features in different stages and depths, the stage
interaction is improved by the Fast Fourier Transform (FFT). Experimental
results on both simulated and real scenes exhibit the superior performance of
our method compared to state-of-the-art HSI reconstruction methods. The code is
released at: https://github.com/MyuLi/PADUT.Comment: ICCV 202
Masked Spatial-Spectral Autoencoders Are Excellent Hyperspectral Defenders
Deep learning methodology contributes a lot to the development of
hyperspectral image (HSI) analysis community. However, it also makes HSI
analysis systems vulnerable to adversarial attacks. To this end, we propose a
masked spatial-spectral autoencoder (MSSA) in this paper under self-supervised
learning theory, for enhancing the robustness of HSI analysis systems. First, a
masked sequence attention learning module is conducted to promote the inherent
robustness of HSI analysis systems along spectral channel. Then, we develop a
graph convolutional network with learnable graph structure to establish global
pixel-wise combinations.In this way, the attack effect would be dispersed by
all the related pixels among each combination, and a better defense performance
is achievable in spatial aspect.Finally, to improve the defense transferability
and address the problem of limited labelled samples, MSSA employs spectra
reconstruction as a pretext task and fits the datasets in a self-supervised
manner.Comprehensive experiments over three benchmarks verify the effectiveness
of MSSA in comparison with the state-of-the-art hyperspectral classification
methods and representative adversarial defense strategies.Comment: 14 pages, 9 figure
Binarized Spectral Compressive Imaging
Existing deep learning models for hyperspectral image (HSI) reconstruction
achieve good performance but require powerful hardwares with enormous memory
and computational resources. Consequently, these methods can hardly be deployed
on resource-limited mobile devices. In this paper, we propose a novel method,
Binarized Spectral-Redistribution Network (BiSRNet), for efficient and
practical HSI restoration from compressed measurement in snapshot compressive
imaging (SCI) systems. Firstly, we redesign a compact and easy-to-deploy base
model to be binarized. Then we present the basic unit, Binarized
Spectral-Redistribution Convolution (BiSR-Conv). BiSR-Conv can adaptively
redistribute the HSI representations before binarizing activation and uses a
scalable hyperbolic tangent function to closer approximate the Sign function in
backpropagation. Based on our BiSR-Conv, we customize four binarized
convolutional modules to address the dimension mismatch and propagate
full-precision information throughout the whole network. Finally, our BiSRNet
is derived by using the proposed techniques to binarize the base model.
Comprehensive quantitative and qualitative experiments manifest that our
proposed BiSRNet outperforms state-of-the-art binarization methods and achieves
comparable performance with full-precision algorithms. Code and models are
publicly available at https://github.com/caiyuanhao1998/BiSCI and
https://github.com/caiyuanhao1998/MSTComment: NeurIPS 2023; The first work to study binarized spectral compressive
imaging reconstruction proble
Aperture Diffraction for Compact Snapshot Spectral Imaging
We demonstrate a compact, cost-effective snapshot spectral imaging system
named Aperture Diffraction Imaging Spectrometer (ADIS), which consists only of
an imaging lens with an ultra-thin orthogonal aperture mask and a mosaic filter
sensor, requiring no additional physical footprint compared to common RGB
cameras. Then we introduce a new optical design that each point in the object
space is multiplexed to discrete encoding locations on the mosaic filter sensor
by diffraction-based spatial-spectral projection engineering generated from the
orthogonal mask. The orthogonal projection is uniformly accepted to obtain a
weakly calibration-dependent data form to enhance modulation robustness.
Meanwhile, the Cascade Shift-Shuffle Spectral Transformer (CSST) with strong
perception of the diffraction degeneration is designed to solve a
sparsity-constrained inverse problem, realizing the volume reconstruction from
2D measurements with Large amount of aliasing. Our system is evaluated by
elaborating the imaging optical theory and reconstruction algorithm with
demonstrating the experimental imaging under a single exposure. Ultimately, we
achieve the sub-super-pixel spatial resolution and high spectral resolution
imaging. The code will be available at: https://github.com/Krito-ex/CSST.Comment: accepted by International Conference on Computer Vision (ICCV) 202
DiffSCI: Zero-Shot Snapshot Compressive Imaging via Iterative Spectral Diffusion Model
This paper endeavors to advance the precision of snapshot compressive imaging
(SCI) reconstruction for multispectral image (MSI). To achieve this, we
integrate the advantageous attributes of established SCI techniques and an
image generative model, propose a novel structured zero-shot diffusion model,
dubbed DiffSCI. DiffSCI leverages the structural insights from the deep prior
and optimization-based methodologies, complemented by the generative
capabilities offered by the contemporary denoising diffusion model.
Specifically, firstly, we employ a pre-trained diffusion model, which has been
trained on a substantial corpus of RGB images, as the generative denoiser
within the Plug-and-Play framework for the first time. This integration allows
for the successful completion of SCI reconstruction, especially in the case
that current methods struggle to address effectively. Secondly, we
systematically account for spectral band correlations and introduce a robust
methodology to mitigate wavelength mismatch, thus enabling seamless adaptation
of the RGB diffusion model to MSIs. Thirdly, an accelerated algorithm is
implemented to expedite the resolution of the data subproblem. This
augmentation not only accelerates the convergence rate but also elevates the
quality of the reconstruction process. We present extensive testing to show
that DiffSCI exhibits discernible performance enhancements over prevailing
self-supervised and zero-shot approaches, surpassing even supervised
transformer counterparts across both simulated and real datasets. Our code will
be available