32 research outputs found
Single Image Super-Resolution Using Multi-Scale Convolutional Neural Network
Methods based on convolutional neural network (CNN) have demonstrated
tremendous improvements on single image super-resolution. However, the previous
methods mainly restore images from one single area in the low resolution (LR)
input, which limits the flexibility of models to infer various scales of
details for high resolution (HR) output. Moreover, most of them train a
specific model for each up-scale factor. In this paper, we propose a
multi-scale super resolution (MSSR) network. Our network consists of
multi-scale paths to make the HR inference, which can learn to synthesize
features from different scales. This property helps reconstruct various kinds
of regions in HR images. In addition, only one single model is needed for
multiple up-scale factors, which is more efficient without loss of restoration
quality. Experiments on four public datasets demonstrate that the proposed
method achieved state-of-the-art performance with fast speed
Learning a Mixture of Deep Networks for Single Image Super-Resolution
Single image super-resolution (SR) is an ill-posed problem which aims to
recover high-resolution (HR) images from their low-resolution (LR)
observations. The crux of this problem lies in learning the complex mapping
between low-resolution patches and the corresponding high-resolution patches.
Prior arts have used either a mixture of simple regression models or a single
non-linear neural network for this propose. This paper proposes the method of
learning a mixture of SR inference modules in a unified framework to tackle
this problem. Specifically, a number of SR inference modules specialized in
different image local patterns are first independently applied on the LR image
to obtain various HR estimates, and the resultant HR estimates are adaptively
aggregated to form the final HR image. By selecting neural networks as the SR
inference module, the whole procedure can be incorporated into a unified
network and be optimized jointly. Extensive experiments are conducted to
investigate the relation between restoration performance and different network
architectures. Compared with other current image SR approaches, our proposed
method achieves state-of-the-arts restoration results on a wide range of images
consistently while allowing more flexible design choices. The source codes are
available in http://www.ifp.illinois.edu/~dingliu2/accv2016
Hypernetwork functional image representation
Motivated by the human way of memorizing images we introduce their functional
representation, where an image is represented by a neural network. For this
purpose, we construct a hypernetwork which takes an image and returns weights
to the target network, which maps point from the plane (representing positions
of the pixel) into its corresponding color in the image. Since the obtained
representation is continuous, one can easily inspect the image at various
resolutions and perform on it arbitrary continuous operations. Moreover, by
inspecting interpolations we show that such representation has some properties
characteristic to generative models. To evaluate the proposed mechanism
experimentally, we apply it to image super-resolution problem. Despite using a
single model for various scaling factors, we obtained results comparable to
existing super-resolution methods
Single Image Super-Resolution Using Lightweight CNN with Maxout Units
Rectified linear units (ReLU) are well-known to be helpful in obtaining
faster convergence and thus higher performance for many deep-learning-based
applications. However, networks with ReLU tend to perform poorly when the
number of filter parameters is constrained to a small number. To overcome it,
in this paper, we propose a novel network utilizing maxout units (MU), and show
its effectiveness on super-resolution (SR) applications. In general, the MU has
been known to make the filter sizes doubled in generating the feature maps of
the same sizes in classification problems. In this paper, we first reveal that
the MU can even make the filter sizes halved in restoration problems thus
leading to compaction of the network sizes. To show this, our SR network is
designed without increasing the filter sizes with MU, which outperforms the
state of the art SR methods with a smaller number of filter parameters. To the
best of our knowledge, we are the first to incorporate MU into SR applications
and show promising performance results. In MU, feature maps from a previous
convolutional layer are divided into two parts along channels, which are then
compared element-wise and only their max values are passed to a next layer.
Along with some interesting properties of MU to be analyzed, we further
investigate other variants of MU and their effects. In addition, while ReLU
have a trouble for learning in networks with a very small number of
convolutional filter parameters, MU do not. For SR applications, our MU-based
network reconstructs high-resolution images with comparable quality compared to
previous deep-learning-based SR methods, with lower filter parameters.Comment: ACCV201
LIRA: Lifelong Image Restoration from Unknown Blended Distortions
Most existing image restoration networks are designed in a disposable way and
catastrophically forget previously learned distortions when trained on a new
distortion removal task. To alleviate this problem, we raise the novel lifelong
image restoration problem for blended distortions. We first design a base
fork-join model in which multiple pre-trained expert models specializing in
individual distortion removal task work cooperatively and adaptively to handle
blended distortions. When the input is degraded by a new distortion, inspired
by adult neurogenesis in human memory system, we develop a neural growing
strategy where the previously trained model can incorporate a new expert branch
and continually accumulate new knowledge without interfering with learned
knowledge. Experimental results show that the proposed approach can not only
achieve state-of-the-art performance on blended distortions removal tasks in
both PSNR/SSIM metrics, but also maintain old expertise while learning new
restoration tasks.Comment: ECCV2020 accepte
Simple, Accurate, and Robust Nonparametric Blind Super-Resolution
This paper proposes a simple, accurate, and robust approach to single image
nonparametric blind Super-Resolution (SR). This task is formulated as a
functional to be minimized with respect to both an intermediate super-resolved
image and a nonparametric blur-kernel. The proposed approach includes a
convolution consistency constraint which uses a non-blind learning-based SR
result to better guide the estimation process. Another key component is the
unnatural bi-l0-l2-norm regularization imposed on the super-resolved, sharp
image and the blur-kernel, which is shown to be quite beneficial for estimating
the blur-kernel accurately. The numerical optimization is implemented by
coupling the splitting augmented Lagrangian and the conjugate gradient (CG).
Using the pre-estimated blur-kernel, we finally reconstruct the SR image by a
very simple non-blind SR method that uses a natural image prior. The proposed
approach is demonstrated to achieve better performance than the recent method
by Michaeli and Irani [2] in both terms of the kernel estimation accuracy and
image SR quality
W2S: Microscopy Data with Joint Denoising and Super-Resolution for Widefield to SIM Mapping
In fluorescence microscopy live-cell imaging, there is a critical trade-off
between the signal-to-noise ratio and spatial resolution on one side, and the
integrity of the biological sample on the other side. To obtain clean
high-resolution (HR) images, one can either use microscopy techniques, such as
structured-illumination microscopy (SIM), or apply denoising and
super-resolution (SR) algorithms. However, the former option requires multiple
shots that can damage the samples, and although efficient deep learning based
algorithms exist for the latter option, no benchmark exists to evaluate these
algorithms on the joint denoising and SR (JDSR) tasks. To study JDSR on
microscopy data, we propose such a novel JDSR dataset, Widefield2SIM (W2S),
acquired using a conventional fluorescence widefield and SIM imaging. W2S
includes 144,000 real fluorescence microscopy images, resulting in a total of
360 sets of images. A set is comprised of noisy low-resolution (LR) widefield
images with different noise levels, a noise-free LR image, and a corresponding
high-quality HR SIM image. W2S allows us to benchmark the combinations of 6
denoising methods and 6 SR methods. We show that state-of-the-art SR networks
perform very poorly on noisy inputs. Our evaluation also reveals that applying
the best denoiser in terms of reconstruction error followed by the best SR
method does not necessarily yield the best final result. Both quantitative and
qualitative results show that SR networks are sensitive to noise and the
sequential application of denoising and SR algorithms is sub-optimal. Lastly,
we demonstrate that SR networks retrained end-to-end for JDSR outperform any
combination of state-of-the-art deep denoising and SR networksComment: ECCVW 2020. Project page: \<https://github.com/ivrl/w2s
eCNN: A Block-Based and Highly-Parallel CNN Accelerator for Edge Inference
Convolutional neural networks (CNNs) have recently demonstrated superior
quality for computational imaging applications. Therefore, they have great
potential to revolutionize the image pipelines on cameras and displays.
However, it is difficult for conventional CNN accelerators to support
ultra-high-resolution videos at the edge due to their considerable DRAM
bandwidth and power consumption. Therefore, finding a further memory- and
computation-efficient microarchitecture is crucial to speed up this coming
revolution.
In this paper, we approach this goal by considering the inference flow,
network model, instruction set, and processor design jointly to optimize
hardware performance and image quality. We apply a block-based inference flow
which can eliminate all the DRAM bandwidth for feature maps and accordingly
propose a hardware-oriented network model, ERNet, to optimize image quality
based on hardware constraints. Then we devise a coarse-grained instruction set
architecture, FBISA, to support power-hungry convolution by massive
parallelism. Finally,we implement an embedded processor---eCNN---which
accommodates to ERNet and FBISA with a flexible processing architecture. Layout
results show that it can support high-quality ERNets for super-resolution and
denoising at up to 4K Ultra-HD 30 fps while using only DDR-400 and consuming
6.94W on average. By comparison, the state-of-the-art Diffy uses dual-channel
DDR3-2133 and consumes 54.3W to support lower-quality VDSR at Full HD 30 fps.
Lastly, we will also present application examples of high-performance style
transfer and object recognition to demonstrate the flexibility of eCNN.Comment: 14 pages; appearing in IEEE/ACM International Symposium on
Microarchitecture (MICRO), 201
Perceptual Losses for Real-Time Style Transfer and Super-Resolution
We consider image transformation problems, where an input image is transformed into an output image. Recent methods for such problems typically train feed-forward convolutional neural networks using a per-pixel loss between the output and ground-truth images. Parallel work has shown that high-quality images can be generated by defining and optimizing perceptual loss functions based on high-level features extracted from pretrained networks. We combine the benefits of both approaches, and propose the use of perceptual loss functions for training feed-forward networks for image transformation tasks. We show results on image style transfer, where a feed-forward network is trained to solve the optimization problem proposed by Gatys et al. in real-time. Compared to the optimization-based method, our network gives similar qualitative results but is three orders of magnitude faster. We also experiment with single-image super-resolution, where replacing a per-pixel loss with a perceptual loss gives visually pleasing results