629 research outputs found
Generalized Inpainting Method for Hyperspectral Image Acquisition
A recently designed hyperspectral imaging device enables multiplexed
acquisition of an entire data volume in a single snapshot thanks to
monolithically-integrated spectral filters. Such an agile imaging technique
comes at the cost of a reduced spatial resolution and the need for a
demosaicing procedure on its interleaved data. In this work, we address both
issues and propose an approach inspired by recent developments in compressed
sensing and analysis sparse models. We formulate our superresolution and
demosaicing task as a 3-D generalized inpainting problem. Interestingly, the
target spatial resolution can be adjusted for mitigating the compression level
of our sensing. The reconstruction procedure uses a fast greedy method called
Pseudo-inverse IHT. We also show on simulations that a random arrangement of
the spectral filters on the sensor is preferable to regular mosaic layout as it
improves the quality of the reconstruction. The efficiency of our technique is
demonstrated through numerical experiments on both synthetic and real data as
acquired by the snapshot imager.Comment: Keywords: Hyperspectral, inpainting, iterative hard thresholding,
sparse models, CMOS, Fabry-P\'ero
Hyperspectral Demosaicing of Snapshot Camera Images Using Deep Learning
Spectral imaging technologies have rapidly evolved during the past decades.
The recent development of single-camera-one-shot techniques for hyperspectral
imaging allows multiple spectral bands to be captured simultaneously (3x3, 4x4
or 5x5 mosaic), opening up a wide range of applications. Examples include
intraoperative imaging, agricultural field inspection and food quality
assessment. To capture images across a wide spectrum range, i.e. to achieve
high spectral resolution, the sensor design sacrifices spatial resolution. With
increasing mosaic size, this effect becomes increasingly detrimental.
Furthermore, demosaicing is challenging. Without incorporating edge, shape, and
object information during interpolation, chromatic artifacts are likely to
appear in the obtained images. Recent approaches use neural networks for
demosaicing, enabling direct information extraction from image data. However,
obtaining training data for these approaches poses a challenge as well. This
work proposes a parallel neural network based demosaicing procedure trained on
a new ground truth dataset captured in a controlled environment by a
hyperspectral snapshot camera with a 4x4 mosaic pattern. The dataset is a
combination of real captured scenes with images from publicly available data
adapted to the 4x4 mosaic pattern. To obtain real world ground-truth data, we
performed multiple camera captures with 1-pixel shifts in order to compose the
entire data cube. Experiments show that the proposed network outperforms
state-of-art networks.Comment: German Conference on Pattern Recognition (GCPR) 202
Spatial gradient consistency for unsupervised learning of hyperspectral demosaicking: Application to surgical imaging
Hyperspectral imaging has the potential to improve intraoperative decision
making if tissue characterisation is performed in real-time and with
high-resolution. Hyperspectral snapshot mosaic sensors offer a promising
approach due to their fast acquisition speed and compact size. However, a
demosaicking algorithm is required to fully recover the spatial and spectral
information of the snapshot images. Most state-of-the-art demosaicking
algorithms require ground-truth training data with paired snapshot and
high-resolution hyperspectral images, but such imagery pairs with the exact
same scene are physically impossible to acquire in intraoperative settings. In
this work, we present a fully unsupervised hyperspectral image demosaicking
algorithm which only requires exemplar snapshot images for training purposes.
We regard hyperspectral demosaicking as an ill-posed linear inverse problem
which we solve using a deep neural network. We take advantage of the spectral
correlation occurring in natural scenes to design a novel inter spectral band
regularisation term based on spatial gradient consistency. By combining our
proposed term with standard regularisation techniques and exploiting a standard
data fidelity term, we obtain an unsupervised loss function for training deep
neural networks, which allows us to achieve real-time hyperspectral image
demosaicking. Quantitative results on hyperspetral image datasets show that our
unsupervised demosaicking approach can achieve similar performance to its
supervised counter-part, and significantly outperform linear demosaicking. A
qualitative user study on real snapshot hyperspectral surgical images confirms
the results from the quantitative analysis. Our results suggest that the
proposed unsupervised algorithm can achieve promising hyperspectral
demosaicking in real-time thus advancing the suitability of the modality for
intraoperative use
Detection of irrigation inhomogeneities in an olive grove using the NDRE vegetation index obtained from UAV images
We have developed a simple photogrammetric method to identify heterogeneous areas of irrigated olive groves and vineyard crops using a commercial multispectral camera mounted on an unmanned aerial vehicle (UAV). By comparing NDVI, GNDVI, SAVI, and NDRE vegetation indices, we find that the latter shows irrigation irregularities in an olive grove not discernible with the other indices. This may render the NDRE as particularly useful to identify growth inhomogeneities in crops. Given the fact that few satellite detectors are sensible in the red-edge (RE) band and none with the spatial resolution offered by UAVs, this finding has the potential of turning UAVs into a local farmer’s favourite aid tool.Peer ReviewedPostprint (published version
Aperture Diffraction for Compact Snapshot Spectral Imaging
We demonstrate a compact, cost-effective snapshot spectral imaging system
named Aperture Diffraction Imaging Spectrometer (ADIS), which consists only of
an imaging lens with an ultra-thin orthogonal aperture mask and a mosaic filter
sensor, requiring no additional physical footprint compared to common RGB
cameras. Then we introduce a new optical design that each point in the object
space is multiplexed to discrete encoding locations on the mosaic filter sensor
by diffraction-based spatial-spectral projection engineering generated from the
orthogonal mask. The orthogonal projection is uniformly accepted to obtain a
weakly calibration-dependent data form to enhance modulation robustness.
Meanwhile, the Cascade Shift-Shuffle Spectral Transformer (CSST) with strong
perception of the diffraction degeneration is designed to solve a
sparsity-constrained inverse problem, realizing the volume reconstruction from
2D measurements with Large amount of aliasing. Our system is evaluated by
elaborating the imaging optical theory and reconstruction algorithm with
demonstrating the experimental imaging under a single exposure. Ultimately, we
achieve the sub-super-pixel spatial resolution and high spectral resolution
imaging. The code will be available at: https://github.com/Krito-ex/CSST.Comment: accepted by International Conference on Computer Vision (ICCV) 202
SpaceNet MVOI: a Multi-View Overhead Imagery Dataset
Detection and segmentation of objects in overheard imagery is a challenging
task. The variable density, random orientation, small size, and
instance-to-instance heterogeneity of objects in overhead imagery calls for
approaches distinct from existing models designed for natural scene datasets.
Though new overhead imagery datasets are being developed, they almost
universally comprise a single view taken from directly overhead ("at nadir"),
failing to address a critical variable: look angle. By contrast, views vary in
real-world overhead imagery, particularly in dynamic scenarios such as natural
disasters where first looks are often over 40 degrees off-nadir. This
represents an important challenge to computer vision methods, as changing view
angle adds distortions, alters resolution, and changes lighting. At present,
the impact of these perturbations for algorithmic detection and segmentation of
objects is untested. To address this problem, we present an open source
Multi-View Overhead Imagery dataset, termed SpaceNet MVOI, with 27 unique looks
from a broad range of viewing angles (-32.5 degrees to 54.0 degrees). Each of
these images cover the same 665 square km geographic extent and are annotated
with 126,747 building footprint labels, enabling direct assessment of the
impact of viewpoint perturbation on model performance. We benchmark multiple
leading segmentation and object detection models on: (1) building detection,
(2) generalization to unseen viewing angles and resolutions, and (3)
sensitivity of building footprint extraction to changes in resolution. We find
that state of the art segmentation and object detection models struggle to
identify buildings in off-nadir imagery and generalize poorly to unseen views,
presenting an important benchmark to explore the broadly relevant challenge of
detecting small, heterogeneous target objects in visually dynamic contexts.Comment: Accepted into IEEE International Conference on Computer Vision (ICCV)
201
- …