Superresolution of Hyperspectral Image Using Advanced Nonlocal Means Filter and Iterative Back Projection

We introduce an efficient superresolution algorithm based on advanced nonlocal means (NLM) filter and iterative back projection for hyperspectral image. The nonlocal means method achieves the to-be-interpolated pixel by the weighted average of all pixels within an image, and the unrelated neighborhoods are automatically eliminated by the trivial weights. However, spatial location distance is also an important issue to reconstruct the missing pixel. Therefore, we proposed an advanced NLM (ANLM) filter considering both neighborhood similarity and patch distance. In the conventional NLM method, the search region was the whole image, while the proposed ANLM utilizes the limited search to reduce the complexity. The iterative back projection (IBP) is a very famous method to deal with the image restoration. In the superresolution issue, IBP is able to recover the high-resolution image iteratively from the given low-resolution image which is blurred due to the noise by minimizing the reconstruction error, while, because the reconstruction error of IBP is back projection and isotropic, the conventional IBP suffers from jaggy and ringing artifacts. Introducing the ANLM method to improve the visual quality is necessary

A review of spatial enhancement of hyperspectral remote sensing imaging techniques

Remote sensing technology has undeniable importance in various industrial applications, such as mineral exploration, plant detection, defect detection in aerospace and shipbuilding, and optical gas imaging, to name a few. Remote sensing technology has been continuously evolving, offering a range of image modalities that can facilitate the aforementioned applications. One such modality is Hyperspectral Imaging (HSI). Unlike Multispectral Images (MSI) and natural images, HSI consist of hundreds of bands. Despite their high spectral resolution, HSI suffer from low spatial resolution in comparison to their MSI counterpart, which hinders the utilization of their full potential. Therefore, spatial enhancement, or Super Resolution (SR), of HSI is a classical problem that has been gaining rapid attention over the past two decades. The literature is rich with various SR algorithms that enhance the spatial resolution of HSI while preserving their spectral fidelity. This paper reviews and discusses the most important algorithms relevant to this area of research between 2002-2022, along with the most frequently used datasets, HSI sensors, and quality metrics. Meta-analysis are drawn based on the aforementioned information, which is used as a foundation that summarizes the state of the field in a way that bridges the past and the present, identifies the current gap in it, and recommends possible future directions

Super Resolution of Wavelet-Encoded Images and Videos

In this dissertation, we address the multiframe super resolution reconstruction problem for wavelet-encoded images and videos. The goal of multiframe super resolution is to obtain one or more high resolution images by fusing a sequence of degraded or aliased low resolution images of the same scene. Since the low resolution images may be unaligned, a registration step is required before super resolution reconstruction. Therefore, we first explore in-band (i.e. in the wavelet-domain) image registration; then, investigate super resolution. Our motivation for analyzing the image registration and super resolution problems in the wavelet domain is the growing trend in wavelet-encoded imaging, and wavelet-encoding for image/video compression. Due to drawbacks of widely used discrete cosine transform in image and video compression, a considerable amount of literature is devoted to wavelet-based methods. However, since wavelets are shift-variant, existing methods cannot utilize wavelet subbands efficiently. In order to overcome this drawback, we establish and explore the direct relationship between the subbands under a translational shift, for image registration and super resolution. We then employ our devised in-band methodology, in a motion compensated video compression framework, to demonstrate the effective usage of wavelet subbands. Super resolution can also be used as a post-processing step in video compression in order to decrease the size of the video files to be compressed, with downsampling added as a pre-processing step. Therefore, we present a video compression scheme that utilizes super resolution to reconstruct the high frequency information lost during downsampling. In addition, super resolution is a crucial post-processing step for satellite imagery, due to the fact that it is hard to update imaging devices after a satellite is launched. Thus, we also demonstrate the usage of our devised methods in enhancing resolution of pansharpened multispectral images

Single image super resolution for spatial enhancement of hyperspectral remote sensing imagery

Hyperspectral Imaging (HSI) has emerged as a powerful tool for capturing detailed spectral information across various applications, such as remote sensing, medical imaging, and material identification. However, the limited spatial resolution of acquired HSI data poses a challenge due to hardware and acquisition constraints. Enhancing the spatial resolution of HSI is crucial for improving image processing tasks, such as object detection and classification. This research focuses on utilizing Single Image Super Resolution (SISR) techniques to enhance HSI, addressing four key challenges: the efficiency of 3D Deep Convolutional Neural Networks (3D-DCNNs) in HSI enhancement, minimizing spectral distortions, tackling data scarcity, and improving state-of-the-art performance. The thesis establishes a solid theoretical foundation and conducts an in-depth literature review to identify trends, gaps, and future directions in the field of HSI enhancement. Four chapters present novel research targeting each of the aforementioned challenges. All experiments are performed using publicly available datasets, and the results are evaluated both qualitatively and quantitatively using various commonly used metrics. The findings of this research contribute to the development of a novel 3D-CNN architecture known as 3D Super Resolution CNN 333 (3D-SRCNN333). This architecture demonstrates the capability to enhance HSI with minimal spectral distortions while maintaining acceptable computational cost and training time. Furthermore, a Bayesian-optimized hybrid spectral spatial loss function is devised to improve the spatial quality and minimize spectral distortions, combining the best characteristics of both domains. Addressing the challenge of data scarcity, this thesis conducts a thorough study on Data Augmentation techniques and their impact on the spectral signature of HSI. A new Data Augmentation technique called CutMixBlur is proposed, and various combinations of Data Augmentation techniques are evaluated to address the data scarcity challenge, leading to notable enhancements in performance. Lastly, the 3D-SRCNN333 architecture is extended to the frequency domain and wavelet domain to explore their advantages over the spatial domain. The experiments reveal promising results with the 3D Complex Residual SRCNN (3D-CRSRCNN), surpassing the performance of 3D-SRCNN333. The findings presented in this thesis have been published in reputable conferences and journals, indicating their contribution to the field of HSI enhancement. Overall, this thesis provides valuable insights into the field of HSI-SISR, offering a thorough understanding of the advancements, challenges, and potential applications. The developed algorithms and methodologies contribute to the broader goal of improving the spatial resolution and spectral fidelity of HSI, paving the way for further advancements in scientific research and practical implementations.Hyperspectral Imaging (HSI) has emerged as a powerful tool for capturing detailed spectral information across various applications, such as remote sensing, medical imaging, and material identification. However, the limited spatial resolution of acquired HSI data poses a challenge due to hardware and acquisition constraints. Enhancing the spatial resolution of HSI is crucial for improving image processing tasks, such as object detection and classification. This research focuses on utilizing Single Image Super Resolution (SISR) techniques to enhance HSI, addressing four key challenges: the efficiency of 3D Deep Convolutional Neural Networks (3D-DCNNs) in HSI enhancement, minimizing spectral distortions, tackling data scarcity, and improving state-of-the-art performance. The thesis establishes a solid theoretical foundation and conducts an in-depth literature review to identify trends, gaps, and future directions in the field of HSI enhancement. Four chapters present novel research targeting each of the aforementioned challenges. All experiments are performed using publicly available datasets, and the results are evaluated both qualitatively and quantitatively using various commonly used metrics. The findings of this research contribute to the development of a novel 3D-CNN architecture known as 3D Super Resolution CNN 333 (3D-SRCNN333). This architecture demonstrates the capability to enhance HSI with minimal spectral distortions while maintaining acceptable computational cost and training time. Furthermore, a Bayesian-optimized hybrid spectral spatial loss function is devised to improve the spatial quality and minimize spectral distortions, combining the best characteristics of both domains. Addressing the challenge of data scarcity, this thesis conducts a thorough study on Data Augmentation techniques and their impact on the spectral signature of HSI. A new Data Augmentation technique called CutMixBlur is proposed, and various combinations of Data Augmentation techniques are evaluated to address the data scarcity challenge, leading to notable enhancements in performance. Lastly, the 3D-SRCNN333 architecture is extended to the frequency domain and wavelet domain to explore their advantages over the spatial domain. The experiments reveal promising results with the 3D Complex Residual SRCNN (3D-CRSRCNN), surpassing the performance of 3D-SRCNN333. The findings presented in this thesis have been published in reputable conferences and journals, indicating their contribution to the field of HSI enhancement. Overall, this thesis provides valuable insights into the field of HSI-SISR, offering a thorough understanding of the advancements, challenges, and potential applications. The developed algorithms and methodologies contribute to the broader goal of improving the spatial resolution and spectral fidelity of HSI, paving the way for further advancements in scientific research and practical implementations

Adaptive Wiener Filter Super-Resolution of Color Filter Array Images

Digital color cameras using a single detector array with a Bayer color filter array (CFA) require interpolation or demosaicing to estimate missing color information and provide full-color images. However, demosaicing does not specifically address fundamental undersampling and aliasing inherent in typical camera designs. Fast non-uniform interpolation based super-resolution (SR) is an attractive approach to reduce or eliminate aliasing and its relatively low computational load is amenable to real-time applications. The adaptive Wiener filter (AWF) SR algorithm was initially developed for grayscale imaging and has not previously been applied to color SR demosaicing. Here, we develop a novel fast SR method for CFA cameras that is based on the AWF SR algorithm and uses global channel-to-channel statistical models. We apply this new method as a stand-alone algorithm and also as an initialization image for a variational SR algorithm. This paper presents the theoretical development of the color AWF SR approach and applies it in performance comparisons to other SR techniques for both simulated and real data

Bayesian fusion of multi-band images : A powerful tool for super-resolution

Hyperspectral (HS) imaging, which consists of acquiring a same scene in several hundreds of contiguous spectral bands (a three dimensional data cube), has opened a new range of relevant applications, such as target detection [MS02], classification [C.-03] and spectral unmixing [BDPD+12]. However, while HS sensors provide abundant spectral information, their spatial resolution is generally more limited. Thus, fusing the HS image with other highly resolved images of the same scene, such as multispectral (MS) or panchromatic (PAN) images is an interesting problem. The problem of fusing a high spectral and low spatial resolution image with an auxiliary image of higher spatial but lower spectral resolution, also known as multi-resolution image fusion, has been explored for many years [AMV+11]. From an application point of view, this problem is also important as motivated by recent national programs, e.g., the Japanese next-generation space-borne hyperspectral image suite (HISUI), which fuses co-registered MS and HS images acquired over the same scene under the same conditions [YI13]. Bayesian fusion allows for an intuitive interpretation of the fusion process via the posterior distribution. Since the fusion problem is usually ill-posed, the Bayesian methodology offers a convenient way to regularize the problem by defining appropriate prior distribution for the scene of interest. The aim of this thesis is to study new multi-band image fusion algorithms to enhance the resolution of hyperspectral image. In the first chapter, a hierarchical Bayesian framework is proposed for multi-band image fusion by incorporating forward model, statistical assumptions and Gaussian prior for the target image to be restored. To derive Bayesian estimators associated with the resulting posterior distribution, two algorithms based on Monte Carlo sampling and optimization strategy have been developed. In the second chapter, a sparse regularization using dictionaries learned from the observed images is introduced as an alternative of the naive Gaussian prior proposed in Chapter 1. instead of Gaussian prior is introduced to regularize the ill-posed problem. Identifying the supports jointly with the dictionaries circumvented the difficulty inherent to sparse coding. To minimize the target function, an alternate optimization algorithm has been designed, which accelerates the fusion process magnificently comparing with the simulation-based method. In the third chapter, by exploiting intrinsic properties of the blurring and downsampling matrices, a much more efficient fusion method is proposed thanks to a closed-form solution for the Sylvester matrix equation associated with maximizing the likelihood. The proposed solution can be embedded into an alternating direction method of multipliers or a block coordinate descent method to incorporate different priors or hyper-priors for the fusion problem, allowing for Bayesian estimators. In the last chapter, a joint multi-band image fusion and unmixing scheme is proposed by combining the well admitted linear spectral mixture model and the forward model. The joint fusion and unmixing problem is solved in an alternating optimization framework, mainly consisting of solving a Sylvester equation and projecting onto a simplex resulting from the non-negativity and sum-to-one constraints. The simulation results conducted on synthetic and semi-synthetic images illustrate the advantages of the developed Bayesian estimators, both qualitatively and quantitatively

Super-resolution mapping

Super-resolution mapping is becoming an increasing important technique in remote sensing for land cover mapping at a sub-pixel scale from coarse spatial resolution imagery. The potential of this technique could increase the value of the low cost coarse spatial resolution imagery. Among many types of land cover patches that can be represented by the super-resolution mapping, the prediction of patches smaller than an image pixel is one of the most difficult. This is because of the lack of information on the existence and spatial extend of the small land cover patches. Another difficult problem is to represent the location of small patches accurately. This thesis focuses on the potential of super-resolution mapping for accurate land cover mapping, with particular emphasis on the mapping of small patches. Popular super-resolution mapping techniques such as pixel swapping and the Hopfield neural network are used as well as a new method proposed. Using a Hopfield neural network (HNN) for super-resolution mapping, the best parameters and configuration to represent land cover patches of different sizes, shapes and mosaics are investigated. In addition, it also shown how a fusion of time series coarse spatial resolution imagery, such as daily MODIS 250 m images, can aid the determination of small land cover patch locations, thus reducing the spatial variability of the representation of such patches. Results of the improved HNN using a time series images are evaluated in a series of assessments, and demonstrated to be superior in terms of mapping accuracy than that of the standard techniques. A novel super-resolution mapping technique based on halftoning concept is presented as an alternative solution for the super-resolution mapping. This new technique is able to represent more land cover patches than the standard techniques

Super-resolution mapping

Super-resolution mapping is becoming an increasing important technique in remote sensing for land cover mapping at a sub-pixel scale from coarse spatial resolution imagery. The potential of this technique could increase the value of the low cost coarse spatial resolution imagery. Among many types of land cover patches that can be represented by the super-resolution mapping, the prediction of patches smaller than an image pixel is one of the most difficult. This is because of the lack of information on the existence and spatial extend of the small land cover patches. Another difficult problem is to represent the location of small patches accurately. This thesis focuses on the potential of super-resolution mapping for accurate land cover mapping, with particular emphasis on the mapping of small patches. Popular super-resolution mapping techniques such as pixel swapping and the Hopfield neural network are used as well as a new method proposed. Using a Hopfield neural network (HNN) for super-resolution mapping, the best parameters and configuration to represent land cover patches of different sizes, shapes and mosaics are investigated. In addition, it also shown how a fusion of time series coarse spatial resolution imagery, such as daily MODIS 250 m images, can aid the determination of small land cover patch locations, thus reducing the spatial variability of the representation of such patches. Results of the improved HNN using a time series images are evaluated in a series of assessments, and demonstrated to be superior in terms of mapping accuracy than that of the standard techniques. A novel super-resolution mapping technique based on halftoning concept is presented as an alternative solution for the super-resolution mapping. This new technique is able to represent more land cover patches than the standard techniques