Self-supervised machine learning algorithms for hyperspectral image inpainting

Abstract

Hyperspectral images (HSIs) can be captured by the satellite sensors at a very narrow band of the electromagnetic spectrum, each presenting the samples at different timeslots, by for example, using push-broom strategies, which are the state-of-the-art practical technologies. Different from RGB images, the nature of the HSI acquisition system makes HSI a 3D data cube that covers hundreds or thousands of narrow spectral bands, conveying a wealth of spatial and spectral information. However, due to instrumental errors and atmospheric changes, the HSI images obtained in practice are often contaminated by noise and dark pixels, which may severely compromise the subsequent processing. It is thus, of vital importance, to improve the quality of the image in the first place. This PhD thesis focuses on the design and analysis of HSI inpainting algorithms to accurately recover images from incomplete observations. Since the existing solutions either failed or behaved badly in the most challenging scenarios where all the spectral bands are missing, which may happen in practice due to instrumental or downlink failures. This study aims to solve this issue through exploiting the recent deep learning techniques. We hope this thesis is able to encourage fruitful discussions and stimulate future research on the exploration of more powerful deep models for solving HSI inpainting problems. Firstly, we introduce here a novel HSI missing pixel prediction algorithm, called Low Rank and Sparsity Constraint Plug-and-Play (LRS-PnP). It is shown that LRS-PnP can effectively cope with the aforementioned difficulties found by traditional methods. The proposed LRS-PnP algorithm is further extended to a self-supervised model by combining the LRS-PnP with the Deep Image Prior (DIP), called LRS-PnP-DIP. We show that the proposed LRS-PnP-DIP algorithm enjoys the specific learning capability of deep networks, called inductive bias, but without needing any external training data, \textit{i,e.} self-supervised learning. In a series of experiments with real data, we show that the LRS-PnP-DIP either achieves state-of-the-art inpainting performance compared to other learning-based methods or outperforms them. However, it is found that the instability inherited from the conventional DIP model makes the LRS-PnP-DIP algorithm sometimes diverge. This observation motivate us to conduct a theoretical analysis of the convergence of the proposed method. Secondly, we explore LRS-PnP and LRS-PnP-DIP in more depth by showing that their potential instability can be solved by slightly modifying both deep hyperspectral prior and plug-and-play denoiser. Under some mild assumptions, we give a fixed-point convergence proof for the LRS-PnP-DIP algorithm and introduce a variant to the LRS-PnP-DIP. We show through extensive experiments that the proposed solution can produce visually and qualitatively superior inpainting results, which achieves competitive performance compared to the original algorithm. Thirdly, we present a powerful HSI inpainting algorithm that dynamically combines self-supervised learning with the recent popular Diffusion model. The proposed Hyerspectral Diffusion based on the Equivariant Imaging (HyDiff-EI) algorithm exploits the strong learning capability of the neural network prior and leverages the high-level hierarchical information of the diffusion models. We empirically demonstrate the effectiveness of the proposed method on the HSI datasets, showing a big performance gap over existing methods based on deep priors/existing diffusion models, and established new state-of-the-art on the self-supervised HSI inpainting task