154 research outputs found
NTIRE 2020 Challenge on Spectral Reconstruction from an RGB Image
This paper reviews the second challenge on spectral reconstruction from RGB
images, i.e., the recovery of whole-scene hyperspectral (HS) information from a
3-channel RGB image. As in the previous challenge, two tracks were provided:
(i) a "Clean" track where HS images are estimated from noise-free RGBs, the RGB
images are themselves calculated numerically using the ground-truth HS images
and supplied spectral sensitivity functions (ii) a "Real World" track,
simulating capture by an uncalibrated and unknown camera, where the HS images
are recovered from noisy JPEG-compressed RGB images. A new, larger-than-ever,
natural hyperspectral image data set is presented, containing a total of 510 HS
images. The Clean and Real World tracks had 103 and 78 registered participants
respectively, with 14 teams competing in the final testing phase. A description
of the proposed methods, alongside their challenge scores and an extensive
evaluation of top performing methods is also provided. They gauge the
state-of-the-art in spectral reconstruction from an RGB image
Hierarchical Regression Network for Spectral Reconstruction from RGB Images
Capturing visual image with a hyperspectral camera has been successfully
applied to many areas due to its narrow-band imaging technology. Hyperspectral
reconstruction from RGB images denotes a reverse process of hyperspectral
imaging by discovering an inverse response function. Current works mainly map
RGB images directly to corresponding spectrum but do not consider context
information explicitly. Moreover, the use of encoder-decoder pair in current
algorithms leads to loss of information. To address these problems, we propose
a 4-level Hierarchical Regression Network (HRNet) with PixelShuffle layer as
inter-level interaction. Furthermore, we adopt a residual dense block to remove
artifacts of real world RGB images and a residual global block to build
attention mechanism for enlarging perceptive field. We evaluate proposed HRNet
with other architectures and techniques by participating in NTIRE 2020
Challenge on Spectral Reconstruction from RGB Images. The HRNet is the winning
method of track 2 - real world images and ranks 3rd on track 1 - clean images.
Please visit the project web page
https://github.com/zhaoyuzhi/Hierarchical-Regression-Network-for-Spectral-Reconstruction-from-RGB-Images
to try our codes and pre-trained models.Comment: 1st Place in CVPRW 2020 NTIRE Spectral Reconstruction Challeng
Towards Spectral Estimation from a Single RGB Image in the Wild
In contrast to the current literature, we address the problem of estimating
the spectrum from a single common trichromatic RGB image obtained under
unconstrained settings (e.g. unknown camera parameters, unknown scene radiance,
unknown scene contents). For this we use a reference spectrum as provided by a
hyperspectral image camera, and propose efficient deep learning solutions for
sensitivity function estimation and spectral reconstruction from a single RGB
image. We further expand the concept of spectral reconstruction such that to
work for RGB images taken in the wild and propose a solution based on a
convolutional network conditioned on the estimated sensitivity function.
Besides the proposed solutions, we study also generic and sensitivity
specialized models and discuss their limitations. We achieve state-of-the-art
competitive results on the standard example-based spectral reconstruction
benchmarks: ICVL, CAVE, NUS and NTIRE. Moreover, our experiments show that, for
the first time, accurate spectral estimation from a single RGB image in the
wild is within our reach
AdaptiveWeighted Attention Network with Camera Spectral Sensitivity Prior for Spectral Reconstruction from RGB Images
Recent promising effort for spectral reconstruction (SR) focuses on learning
a complicated mapping through using a deeper and wider convolutional neural
networks (CNNs). Nevertheless, most CNN-based SR algorithms neglect to explore
the camera spectral sensitivity (CSS) prior and interdependencies among
intermediate features, thus limiting the representation ability of the network
and performance of SR. To conquer these issues, we propose a novel adaptive
weighted attention network (AWAN) for SR, whose backbone is stacked with
multiple dual residual attention blocks (DRAB) decorating with long and short
skip connections to form the dual residual learning. Concretely, we investigate
an adaptive weighted channel attention (AWCA) module to reallocate channel-wise
feature responses via integrating correlations between channels. Furthermore, a
patch-level second-order non-local (PSNL) module is developed to capture
long-range spatial contextual information by second-order non-local operations
for more powerful feature representations. Based on the fact that the recovered
RGB images can be projected by the reconstructed hyperspectral image (HSI) and
the given CSS function, we incorporate the discrepancies of the RGB images and
HSIs as a finer constraint for more accurate reconstruction. Experimental
results demonstrate the effectiveness of our proposed AWAN network in terms of
quantitative comparison and perceptual quality over other state-of-the-art SR
methods. In the NTIRE 2020 Spectral Reconstruction Challenge, our entries
obtain the 1st ranking on the Clean track and the 3rd place on the Real World
track. Codes are available at https://github.com/Deep-imagelab/AWAN.Comment: The 1st ranking on the Clean track and the 3rd place only 1.59106e-4
more than the 1st on the Real World track of the NTIRE 2020 Spectral
Reconstruction Challeng
NTIRE 2020 Challenge on Image Demoireing: Methods and Results
This paper reviews the Challenge on Image Demoireing that was part of the New
Trends in Image Restoration and Enhancement (NTIRE) workshop, held in
conjunction with CVPR 2020. Demoireing is a difficult task of removing moire
patterns from an image to reveal an underlying clean image. The challenge was
divided into two tracks. Track 1 targeted the single image demoireing problem,
which seeks to remove moire patterns from a single image. Track 2 focused on
the burst demoireing problem, where a set of degraded moire images of the same
scene were provided as input, with the goal of producing a single demoired
image as output. The methods were ranked in terms of their fidelity, measured
using the peak signal-to-noise ratio (PSNR) between the ground truth clean
images and the restored images produced by the participants' methods. The
tracks had 142 and 99 registered participants, respectively, with a total of 14
and 6 submissions in the final testing stage. The entries span the current
state-of-the-art in image and burst image demoireing problems
NTIRE 2020 Challenge on Real Image Denoising: Dataset, Methods and Results
This paper reviews the NTIRE 2020 challenge on real image denoising with
focus on the newly introduced dataset, the proposed methods and their results.
The challenge is a new version of the previous NTIRE 2019 challenge on real
image denoising that was based on the SIDD benchmark. This challenge is based
on a newly collected validation and testing image datasets, and hence, named
SIDD+. This challenge has two tracks for quantitatively evaluating image
denoising performance in (1) the Bayer-pattern rawRGB and (2) the standard RGB
(sRGB) color spaces. Each track ~250 registered participants. A total of 22
teams, proposing 24 methods, competed in the final phase of the challenge. The
proposed methods by the participating teams represent the current
state-of-the-art performance in image denoising targeting real noisy images.
The newly collected SIDD+ datasets are publicly available at:
https://bit.ly/siddplus_data
MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction
Existing leading methods for spectral reconstruction (SR) focus on designing
deeper or wider convolutional neural networks (CNNs) to learn the end-to-end
mapping from the RGB image to its hyperspectral image (HSI). These CNN-based
methods achieve impressive restoration performance while showing limitations in
capturing the long-range dependencies and self-similarity prior. To cope with
this problem, we propose a novel Transformer-based method, Multi-stage
Spectral-wise Transformer (MST++), for efficient spectral reconstruction. In
particular, we employ Spectral-wise Multi-head Self-attention (S-MSA) that is
based on the HSI spatially sparse while spectrally self-similar nature to
compose the basic unit, Spectral-wise Attention Block (SAB). Then SABs build up
Single-stage Spectral-wise Transformer (SST) that exploits a U-shaped structure
to extract multi-resolution contextual information. Finally, our MST++,
cascaded by several SSTs, progressively improves the reconstruction quality
from coarse to fine. Comprehensive experiments show that our MST++
significantly outperforms other state-of-the-art methods. In the NTIRE 2022
Spectral Reconstruction Challenge, our approach won the First place. Code and
pre-trained models are publicly available at
https://github.com/caiyuanhao1998/MST-plus-plus.Comment: Winner of NTIRE 2022 Challenge on Spectral Reconstruction from RGB;
The First Transformer-based Method for Spectral Reconstructio
Light Weight Residual Dense Attention Net for Spectral Reconstruction from RGB Images
Hyperspectral Imaging is the acquisition of spectral and spatial information
of a particular scene. Capturing such information from a specialized
hyperspectral camera remains costly. Reconstructing such information from the
RGB image achieves a better solution in both classification and object
recognition tasks. This work proposes a novel light weight network with very
less number of parameters about 233,059 parameters based on Residual dense
model with attention mechanism to obtain this solution. This network uses
Coordination Convolutional Block to get the spatial information. The weights
from this block are shared by two independent feature extraction mechanisms,
one by dense feature extraction and the other by the multiscale hierarchical
feature extraction. Finally, the features from both the feature extraction
mechanisms are globally fused to produce the 31 spectral bands. The network is
trained with NTIRE 2020 challenge dataset and thus achieved 0.0457 MRAE metric
value with less computational complexity.Comment: 6pages,4 figure
MXR-U-Nets for Real Time Hyperspectral Reconstruction
In recent times, CNNs have made significant contributions to applications in
image generation, super-resolution and style transfer. In this paper, we build
upon the work of Howard and Gugger, He et al. and Misra, D. and propose a CNN
architecture that accurately reconstructs hyperspectral images from their RGB
counterparts. We also propose a much shallower version of our best model with a
10% relative memory footprint and 3x faster inference, thus enabling real-time
video applications while still experiencing only about a 0.5% decrease in
performance
NTIRE 2020 Challenge on Image and Video Deblurring
Motion blur is one of the most common degradation artifacts in dynamic scene
photography. This paper reviews the NTIRE 2020 Challenge on Image and Video
Deblurring. In this challenge, we present the evaluation results from 3
competition tracks as well as the proposed solutions. Track 1 aims to develop
single-image deblurring methods focusing on restoration quality. On Track 2,
the image deblurring methods are executed on a mobile platform to find the
balance of the running speed and the restoration accuracy. Track 3 targets
developing video deblurring methods that exploit the temporal relation between
input frames. In each competition, there were 163, 135, and 102 registered
participants and in the final testing phase, 9, 4, and 7 teams competed. The
winning methods demonstrate the state-ofthe-art performance on image and video
deblurring tasks.Comment: To be published in CVPR 2020 Workshop (New Trends in Image
Restoration and Enhancement
- …