Interpretable Hyperspectral AI: When Non-Convex Modeling meets Hyperspectral Remote Sensing

Hyperspectral imaging, also known as image spectrometry, is a landmark technique in geoscience and remote sensing (RS). In the past decade, enormous efforts have been made to process and analyze these hyperspectral (HS) products mainly by means of seasoned experts. However, with the ever-growing volume of data, the bulk of costs in manpower and material resources poses new challenges on reducing the burden of manual labor and improving efficiency. For this reason, it is, therefore, urgent to develop more intelligent and automatic approaches for various HS RS applications. Machine learning (ML) tools with convex optimization have successfully undertaken the tasks of numerous artificial intelligence (AI)-related applications. However, their ability in handling complex practical problems remains limited, particularly for HS data, due to the effects of various spectral variabilities in the process of HS imaging and the complexity and redundancy of higher dimensional HS signals. Compared to the convex models, non-convex modeling, which is capable of characterizing more complex real scenes and providing the model interpretability technically and theoretically, has been proven to be a feasible solution to reduce the gap between challenging HS vision tasks and currently advanced intelligent data processing models

Deep Hyperspectral and Multispectral Image Fusion with Inter-image Variability

Hyperspectral and multispectral image fusion allows us to overcome the hardware limitations of hyperspectral imaging systems inherent to their lower spatial resolution. Nevertheless, existing algorithms usually fail to consider realistic image acquisition conditions. This paper presents a general imaging model that considers inter-image variability of data from heterogeneous sources and flexible image priors. The fusion problem is stated as an optimization problem in the maximum a posteriori framework. We introduce an original image fusion method that, on the one hand, solves the optimization problem accounting for inter-image variability with an iteratively reweighted scheme and, on the other hand, that leverages light-weight CNN-based networks to learn realistic image priors from data. In addition, we propose a zero-shot strategy to directly learn the image-specific prior of the latent images in an unsupervised manner. The performance of the algorithm is illustrated with real data subject to inter-image variability.Comment: IEEE Trans. Geosci. Remote sens., to be published. Manuscript submitted August 23, 2022; revised Dec. 15, 2022, and Mar. 13, 2023; and accepted Apr. 07, 202

Joint Nonlocal, Spectral, and Similarity Low-Rank Priors for Hyperspectral-Multispectral Image Fusion

The fusion of a low-spatial-and-high-spectral resolution hyperspectral image (HSI) with a high-spatial-and-low-spectral resolution multispectral image (MSI) allows synthesizing a high-resolution image (HRI), supporting remote sensing applications, such as disaster management, material identification, and precision agriculture. Unlike existing variational methods using low-rank regularizations separately, we present an HSI-MSI fusion method promoting various low-rank regularizations jointly. Our method refines the HRI spatial and spectral correlations from the individual HSI and MSI data through the proper plug-and-play (PnP) of a nonlocal patch-based denoiser in the alternating direction method of multipliers (ADMM). Notably, we consider the nonlocal self-similarity, the spectral low-rank, and introduce a rank-one similarity prior. Furthermore, we demonstrate via an extensive empirical study that the rank-one similarity prior is an inherent characteristic of the HRI. Simulations over standard benchmark datasets show the effectiveness of the proposed HSI-MSI fusion outperforming state-of-the-art methods, particularly in recovering low-contrast areas.acceptedVersionPeer reviewe

Super-Resolution for Hyperspectral and Multispectral Image Fusion Accounting for Seasonal Spectral Variability

Image fusion combines data from different heterogeneous sources to obtain more precise information about an underlying scene. Hyperspectral-multispectral (HS-MS) image fusion is currently attracting great interest in remote sensing since it allows the generation of high spatial resolution HS images, circumventing the main limitation of this imaging modality. Existing HS-MS fusion algorithms, however, neglect the spectral variability often existing between images acquired at different time instants. This time difference causes variations in spectral signatures of the underlying constituent materials due to different acquisition and seasonal conditions. This paper introduces a novel HS-MS image fusion strategy that combines an unmixing-based formulation with an explicit parametric model for typical spectral variability between the two images. Simulations with synthetic and real data show that the proposed strategy leads to a significant performance improvement under spectral variability and state-of-the-art performance otherwise