35 research outputs found
The effect of the color filter array layout choice on state-of-the-art demosaicing
Interpolation from a Color Filter Array (CFA) is the most common method for obtaining full color image data. Its success relies on the smart combination of a CFA and a demosaicing algorithm. Demosaicing on the one hand has been extensively studied. Algorithmic development in the past 20 years ranges from simple linear interpolation to modern neural-network-based (NN) approaches that encode the prior knowledge of millions of training images to fill in missing data in an inconspicious way. CFA design, on the other hand, is less well studied, although still recognized to strongly impact demosaicing performance. This is because demosaicing algorithms are typically limited to one particular CFA pattern, impeding straightforward CFA comparison. This is starting to change with newer classes of demosaicing that may be considered generic or CFA-agnostic. In this study, by comparing performance of two state-of-the-art generic algorithms, we evaluate the potential of modern CFA-demosaicing. We test the hypothesis that, with the increasing power of NN-based demosaicing, the influence of optimal CFA design on system performance decreases. This hypothesis is supported with the experimental results. Such a finding would herald the possibility of relaxing CFA requirements, providing more freedom in the CFA design choice and producing high-quality cameras
Hyperspectral Demosaicing of Snapshot Camera Images Using Deep Learning
Spectral imaging technologies have rapidly evolved during the past decades.
The recent development of single-camera-one-shot techniques for hyperspectral
imaging allows multiple spectral bands to be captured simultaneously (3x3, 4x4
or 5x5 mosaic), opening up a wide range of applications. Examples include
intraoperative imaging, agricultural field inspection and food quality
assessment. To capture images across a wide spectrum range, i.e. to achieve
high spectral resolution, the sensor design sacrifices spatial resolution. With
increasing mosaic size, this effect becomes increasingly detrimental.
Furthermore, demosaicing is challenging. Without incorporating edge, shape, and
object information during interpolation, chromatic artifacts are likely to
appear in the obtained images. Recent approaches use neural networks for
demosaicing, enabling direct information extraction from image data. However,
obtaining training data for these approaches poses a challenge as well. This
work proposes a parallel neural network based demosaicing procedure trained on
a new ground truth dataset captured in a controlled environment by a
hyperspectral snapshot camera with a 4x4 mosaic pattern. The dataset is a
combination of real captured scenes with images from publicly available data
adapted to the 4x4 mosaic pattern. To obtain real world ground-truth data, we
performed multiple camera captures with 1-pixel shifts in order to compose the
entire data cube. Experiments show that the proposed network outperforms
state-of-art networks.Comment: German Conference on Pattern Recognition (GCPR) 202
Efficient training procedures for multi-spectral demosaicing
The simultaneous acquisition of multi-spectral images on a single sensor can be efficiently performed by single shot capture using a mutli-spectral filter array. This paper focused on the demosaicing of color and near-infrared bands and relied on a convolutional neural network (CNN). To train the deep learning model robustly and accurately, it is necessary to provide enough training data, with sufficient variability. We focused on the design of an efficient training procedure by discovering an optimal training dataset. We propose two data selection strategies, motivated by slightly different concepts. The general term that will be used for the proposed models trained using data selection is data selection-based multi-spectral demosaicing (DSMD). The first idea is clustering-based data selection (DSMD-C), with the goal to discover a representative subset with a high variance so as to train a robust model. The second is an adaptive-based data selection (DSMD-A), a self-guided approach that selects new data based on the current model accuracy. We performed a controlled experimental evaluation of the proposed training strategies and the results show that a careful selection of data does benefit the speed and accuracy of training. We are still able to achieve high reconstruction accuracy with a lightweight model
Demosaicking of Color Image Using Residual Interpolation
Demosaicking of color image by residual interpolation aims at reconstructing a full color image from the unfinished color sample output of a picture device. Because of the high value and maintenance, most of the colour device cameras are organized with CFA (Color Filter Array), it produces the mosaicked image. The colour filter array accommodates 3 primary colours red inexperienced and blue and it samples just one color element at every picture element location. The method of estimating the opposite 2 missing color parts at every picture element location is understood as demosaicking. The planned algorithmic program uses the foremost wide accepted technique, residual interpolation for image demosaicking. This technique involves generating the tentative estimates of red and blue pictures and conniving their residuals, that are the distinction between the determined and tentatively calculable picture element values. This technique produces higher correct results. Then the reconstructed image is evaluated to seek out the performance
Saliency Detection Gradient Preservation for Bayer Image Color Reconstruction
Image color reconstruction is a necessary process to recover high quality full color images from Bayer images. In view of the existence of image texture and edge blurring in color reconstruction algorithms, a four-direction joint gradient weighted residual interpolation algorithm is proposed, which uses four-direction weights obtained from RGB pixel gradients and residual gradients in Bayer images, linearly combined with the color difference estimation to effectively obtain the full G image. Aiming at the color cast phenomenon of the image after color interpolation, a saliency detection gradient-preserving color correction algorithm is proposed based on the RGB image captured under natural light. Firstly, the saliency detection method is used to segment the interpolated image and the RGB image into two regions, then carrying out the region correspondence for gradient-preserving color correction, and finally the weighted fusion method is used to obtain the final color reconstructed image. The experimental results show that the reconstructed image texture and edges are clearer and the colors are closer to RGB images
InSPECtor: an end-to-end design framework for compressive pixelated hyperspectral instruments
Classic designs of hyperspectral instrumentation densely sample the spatial
and spectral information of the scene of interest. Data may be compressed after
the acquisition. In this paper we introduce a framework for the design of an
optimized, micro-patterned snapshot hyperspectral imager that acquires an
optimized subset of the spatial and spectral information in the scene. The data
is thereby compressed already at the sensor level, but can be restored to the
full hyperspectral data cube by the jointly optimized reconstructor. This
framework is implemented with TensorFlow and makes use of its automatic
differentiation for the joint optimization of the layout of the micro-patterned
filter array as well as the reconstructor. We explore the achievable
compression ratio for different numbers of filter passbands, number of scanning
frames, and filter layouts using data collected by the Hyperscout instrument.
We show resulting instrument designs that take snapshot measurements without
losing significant information while reducing the data volume, acquisition
time, or detector space by a factor of 40 as compared to classic, dense
sampling. The joint optimization of a compressive hyperspectral imager design
and the accompanying reconstructor provides an avenue to substantially reduce
the data volume from hyperspectral imagers.Comment: 23 pages, 12 figures, published in Applied Optic
Spatial gradient consistency for unsupervised learning of hyperspectral demosaicking: Application to surgical imaging
Hyperspectral imaging has the potential to improve intraoperative decision
making if tissue characterisation is performed in real-time and with
high-resolution. Hyperspectral snapshot mosaic sensors offer a promising
approach due to their fast acquisition speed and compact size. However, a
demosaicking algorithm is required to fully recover the spatial and spectral
information of the snapshot images. Most state-of-the-art demosaicking
algorithms require ground-truth training data with paired snapshot and
high-resolution hyperspectral images, but such imagery pairs with the exact
same scene are physically impossible to acquire in intraoperative settings. In
this work, we present a fully unsupervised hyperspectral image demosaicking
algorithm which only requires exemplar snapshot images for training purposes.
We regard hyperspectral demosaicking as an ill-posed linear inverse problem
which we solve using a deep neural network. We take advantage of the spectral
correlation occurring in natural scenes to design a novel inter spectral band
regularisation term based on spatial gradient consistency. By combining our
proposed term with standard regularisation techniques and exploiting a standard
data fidelity term, we obtain an unsupervised loss function for training deep
neural networks, which allows us to achieve real-time hyperspectral image
demosaicking. Quantitative results on hyperspetral image datasets show that our
unsupervised demosaicking approach can achieve similar performance to its
supervised counter-part, and significantly outperform linear demosaicking. A
qualitative user study on real snapshot hyperspectral surgical images confirms
the results from the quantitative analysis. Our results suggest that the
proposed unsupervised algorithm can achieve promising hyperspectral
demosaicking in real-time thus advancing the suitability of the modality for
intraoperative use