60,841 research outputs found
Deep Learning-based Image Super-Resolution Considering Quantitative and Perceptual Quality
Recently, it has been shown that in super-resolution, there exists a tradeoff
relationship between the quantitative and perceptual quality of super-resolved
images, which correspond to the similarity to the ground-truth images and the
naturalness, respectively. In this paper, we propose a novel super-resolution
method that can improve the perceptual quality of the upscaled images while
preserving the conventional quantitative performance. The proposed method
employs a deep network for multi-pass upscaling in company with a discriminator
network and two quantitative score predictor networks. Experimental results
demonstrate that the proposed method achieves a good balance of the
quantitative and perceptual quality, showing more satisfactory results than
existing methods.Comment: Won the 2nd place for Region 2 in the PIRM Challenge on Perceptual
Super Resolution at ECCV 2018. GitHub at
https://github.com/idearibosome/tf-perceptual-eus
Hallucinated-IQA: No-Reference Image Quality Assessment via Adversarial Learning
No-reference image quality assessment (NR-IQA) is a fundamental yet
challenging task in low-level computer vision community. The difficulty is
particularly pronounced for the limited information, for which the
corresponding reference for comparison is typically absent. Although various
feature extraction mechanisms have been leveraged from natural scene statistics
to deep neural networks in previous methods, the performance bottleneck still
exists. In this work, we propose a hallucination-guided quality regression
network to address the issue. We firstly generate a hallucinated reference
constrained on the distorted image, to compensate the absence of the true
reference. Then, we pair the information of hallucinated reference with the
distorted image, and forward them to the regressor to learn the perceptual
discrepancy with the guidance of an implicit ranking relationship within the
generator, and therefore produce the precise quality prediction. To demonstrate
the effectiveness of our approach, comprehensive experiments are evaluated on
four popular image quality assessment benchmarks. Our method significantly
outperforms all the previous state-of-the-art methods by large margins. The
code and model will be publicly available on the project page
https://kwanyeelin.github.io/projects/HIQA/HIQA.html.Comment: Accepted to CVPR201
Deep Learning for Multiple-Image Super-Resolution
Super-resolution reconstruction (SRR) is a process aimed at enhancing spatial
resolution of images, either from a single observation, based on the learned
relation between low and high resolution, or from multiple images presenting
the same scene. SRR is particularly valuable, if it is infeasible to acquire
images at desired resolution, but many images of the same scene are available
at lower resolution---this is inherent to a variety of remote sensing
scenarios. Recently, we have witnessed substantial improvement in single-image
SRR attributed to the use of deep neural networks for learning the relation
between low and high resolution. Importantly, deep learning has not been
exploited for multiple-image SRR, which benefits from information fusion and in
general allows for achieving higher reconstruction accuracy. In this letter, we
introduce a new method which combines the advantages of multiple-image fusion
with learning the low-to-high resolution mapping using deep networks. The
reported experimental results indicate that our algorithm outperforms the
state-of-the-art SRR methods, including these that operate from a single image,
as well as those that perform multiple-image fusion.Comment: Submitted to IEEE Geoscience and Remote Sensing Letter
The 2018 PIRM Challenge on Perceptual Image Super-resolution
This paper reports on the 2018 PIRM challenge on perceptual super-resolution
(SR), held in conjunction with the Perceptual Image Restoration and
Manipulation (PIRM) workshop at ECCV 2018. In contrast to previous SR
challenges, our evaluation methodology jointly quantifies accuracy and
perceptual quality, therefore enabling perceptual-driven methods to compete
alongside algorithms that target PSNR maximization. Twenty-one participating
teams introduced algorithms which well-improved upon the existing
state-of-the-art methods in perceptual SR, as confirmed by a human opinion
study. We also analyze popular image quality measures and draw conclusions
regarding which of them correlates best with human opinion scores. We conclude
with an analysis of the current trends in perceptual SR, as reflected from the
leading submissions.Comment: Workshop and Challenge on Perceptual Image Restoration and
Manipulation in conjunction with ECCV 2018 webpage: https://www.pirm2018.org
A Deep Journey into Super-resolution: A survey
Deep convolutional networks based super-resolution is a fast-growing field
with numerous practical applications. In this exposition, we extensively
compare 30+ state-of-the-art super-resolution Convolutional Neural Networks
(CNNs) over three classical and three recently introduced challenging datasets
to benchmark single image super-resolution. We introduce a taxonomy for
deep-learning based super-resolution networks that groups existing methods into
nine categories including linear, residual, multi-branch, recursive,
progressive, attention-based and adversarial designs. We also provide
comparisons between the models in terms of network complexity, memory
footprint, model input and output, learning details, the type of network losses
and important architectural differences (e.g., depth, skip-connections,
filters). The extensive evaluation performed, shows the consistent and rapid
growth in the accuracy in the past few years along with a corresponding boost
in model complexity and the availability of large-scale datasets. It is also
observed that the pioneering methods identified as the benchmark have been
significantly outperformed by the current contenders. Despite the progress in
recent years, we identify several shortcomings of existing techniques and
provide future research directions towards the solution of these open problems.Comment: Accepted in ACM Computing Survey
Super-Resolution via Deep Learning
The recent phenomenal interest in convolutional neural networks (CNNs) must
have made it inevitable for the super-resolution (SR) community to explore its
potential. The response has been immense and in the last three years, since the
advent of the pioneering work, there appeared too many works not to warrant a
comprehensive survey. This paper surveys the SR literature in the context of
deep learning. We focus on the three important aspects of multimedia - namely
image, video and multi-dimensions, especially depth maps. In each case, first
relevant benchmarks are introduced in the form of datasets and state of the art
SR methods, excluding deep learning. Next is a detailed analysis of the
individual works, each including a short description of the method and a
critique of the results with special reference to the benchmarking done. This
is followed by minimum overall benchmarking in the form of comparison on some
common dataset, while relying on the results reported in various works
CISRDCNN: Super-resolution of compressed images using deep convolutional neural networks
In recent years, much research has been conducted on image super-resolution
(SR). To the best of our knowledge, however, few SR methods were concerned with
compressed images. The SR of compressed images is a challenging task due to the
complicated compression artifacts, while many images suffer from them in
practice. The intuitive solution for this difficult task is to decouple it into
two sequential but independent subproblems, i.e., compression artifacts
reduction (CAR) and SR. Nevertheless, some useful details may be removed in CAR
stage, which is contrary to the goal of SR and makes the SR stage more
challenging. In this paper, an end-to-end trainable deep convolutional neural
network is designed to perform SR on compressed images (CISRDCNN), which
reduces compression artifacts and improves image resolution jointly.
Experiments on compressed images produced by JPEG (we take the JPEG as an
example in this paper) demonstrate that the proposed CISRDCNN yields
state-of-the-art SR performance on commonly used test images and imagesets. The
results of CISRDCNN on real low quality web images are also very impressive,
with obvious quality enhancement. Further, we explore the application of the
proposed SR method in low bit-rate image coding, leading to better
rate-distortion performance than JPEG.Comment: 32 pages, 17 figures, 5 tables, preprint submitted to Neurocomputin
Bridging the Gap Between Computational Photography and Visual Recognition
What is the current state-of-the-art for image restoration and enhancement
applied to degraded images acquired under less than ideal circumstances? Can
the application of such algorithms as a pre-processing step to improve image
interpretability for manual analysis or automatic visual recognition to
classify scene content? While there have been important advances in the area of
computational photography to restore or enhance the visual quality of an image,
the capabilities of such techniques have not always translated in a useful way
to visual recognition tasks. Consequently, there is a pressing need for the
development of algorithms that are designed for the joint problem of improving
visual appearance and recognition, which will be an enabling factor for the
deployment of visual recognition tools in many real-world scenarios. To address
this, we introduce the UG^2 dataset as a large-scale benchmark composed of
video imagery captured under challenging conditions, and two enhancement tasks
designed to test algorithmic impact on visual quality and automatic object
recognition. Furthermore, we propose a set of metrics to evaluate the joint
improvement of such tasks as well as individual algorithmic advances, including
a novel psychophysics-based evaluation regime for human assessment and a
realistic set of quantitative measures for object recognition performance. We
introduce six new algorithms for image restoration or enhancement, which were
created as part of the IARPA sponsored UG^2 Challenge workshop held at CVPR
2018. Under the proposed evaluation regime, we present an in-depth analysis of
these algorithms and a host of deep learning-based and classic baseline
approaches. From the observed results, it is evident that we are in the early
days of building a bridge between computational photography and visual
recognition, leaving many opportunities for innovation in this area.Comment: CVPR Prize Challenge: http://www.ug2challenge.or
CFSNet: Toward a Controllable Feature Space for Image Restoration
Deep learning methods have witnessed the great progress in image restoration
with specific metrics (e.g., PSNR, SSIM). However, the perceptual quality of
the restored image is relatively subjective, and it is necessary for users to
control the reconstruction result according to personal preferences or image
characteristics, which cannot be done using existing deterministic networks.
This motivates us to exquisitely design a unified interactive framework for
general image restoration tasks. Under this framework, users can control
continuous transition of different objectives, e.g., the perception-distortion
trade-off of image super-resolution, the trade-off between noise reduction and
detail preservation. We achieve this goal by controlling the latent features of
the designed network. To be specific, our proposed framework, named
Controllable Feature Space Network (CFSNet), is entangled by two branches based
on different objectives. Our framework can adaptively learn the coupling
coefficients of different layers and channels, which provides finer control of
the restored image quality. Experiments on several typical image restoration
tasks fully validate the effective benefits of the proposed method. Code is
available at https://github.com/qibao77/CFSNet.Comment: Accepted by ICCV 201
A HVS-inspired Attention to Improve Loss Metrics for CNN-based Perception-Oriented Super-Resolution
Deep Convolutional Neural Network (CNN) features have been demonstrated to be
effective perceptual quality features. The perceptual loss, based on feature
maps of pre-trained CNN's has proven to be remarkably effective for CNN based
perceptual image restoration problems. In this work, taking inspiration from
the the Human Visual System (HVS) and visual perception, we propose a spatial
attention mechanism based on the dependency human contrast sensitivity on
spatial frequency. We identify regions in input images, based on the underlying
spatial frequency, which are not generally well reconstructed during
Super-Resolution but are most important in terms of visual sensitivity. Based
on this prior, we design a spatial attention map that is applied to feature
maps in the perceptual loss and its variants, helping them to identify regions
that are of more perceptual importance. The results demonstrate the our
technique improves the ability of the perceptual loss and contextual loss to
deliver more natural images in CNN based super-resolution
- …