402 research outputs found
Adversarial Spatio-Temporal Learning for Video Deblurring
Camera shake or target movement often leads to undesired blur effects in
videos captured by a hand-held camera. Despite significant efforts having been
devoted to video-deblur research, two major challenges remain: 1) how to model
the spatio-temporal characteristics across both the spatial domain (i.e., image
plane) and temporal domain (i.e., neighboring frames), and 2) how to restore
sharp image details w.r.t. the conventionally adopted metric of pixel-wise
errors. In this paper, to address the first challenge, we propose a DeBLuRring
Network (DBLRNet) for spatial-temporal learning by applying a modified 3D
convolution to both spatial and temporal domains. Our DBLRNet is able to
capture jointly spatial and temporal information encoded in neighboring frames,
which directly contributes to improved video deblur performance. To tackle the
second challenge, we leverage the developed DBLRNet as a generator in the GAN
(generative adversarial network) architecture, and employ a content loss in
addition to an adversarial loss for efficient adversarial training. The
developed network, which we name as DeBLuRring Generative Adversarial Network
(DBLRGAN), is tested on two standard benchmarks and achieves the
state-of-the-art performance.Comment: To appear in IEEE Transactions on Image Processing (TIP
Blind Image Deconvolution using Deep Generative Priors
This paper proposes a novel approach to regularize the \textit{ill-posed} and
\textit{non-linear} blind image deconvolution (blind deblurring) using deep
generative networks as priors. We employ two separate generative models --- one
trained to produce sharp images while the other trained to generate blur
kernels from lower-dimensional parameters. To deblur, we propose an alternating
gradient descent scheme operating in the latent lower-dimensional space of each
of the pretrained generative models. Our experiments show promising deblurring
results on images even under large blurs, and heavy noise. To address the
shortcomings of generative models such as mode collapse, we augment our
generative priors with classical image priors and report improved performance
on complex image datasets. The deblurring performance depends on how well the
range of the generator spans the image class. Interestingly, our experiments
show that even an untrained structured (convolutional) generative networks acts
as an image prior in the image deblurring context allowing us to extend our
results to more diverse natural image datasets
When Autonomous Systems Meet Accuracy and Transferability through AI: A Survey
With widespread applications of artificial intelligence (AI), the
capabilities of the perception, understanding, decision-making and control for
autonomous systems have improved significantly in the past years. When
autonomous systems consider the performance of accuracy and transferability,
several AI methods, like adversarial learning, reinforcement learning (RL) and
meta-learning, show their powerful performance. Here, we review the
learning-based approaches in autonomous systems from the perspectives of
accuracy and transferability. Accuracy means that a well-trained model shows
good results during the testing phase, in which the testing set shares a same
task or a data distribution with the training set. Transferability means that
when a well-trained model is transferred to other testing domains, the accuracy
is still good. Firstly, we introduce some basic concepts of transfer learning
and then present some preliminaries of adversarial learning, RL and
meta-learning. Secondly, we focus on reviewing the accuracy or transferability
or both of them to show the advantages of adversarial learning, like generative
adversarial networks (GANs), in typical computer vision tasks in autonomous
systems, including image style transfer, image superresolution, image
deblurring/dehazing/rain removal, semantic segmentation, depth estimation,
pedestrian detection and person re-identification (re-ID). Then, we further
review the performance of RL and meta-learning from the aspects of accuracy or
transferability or both of them in autonomous systems, involving pedestrian
tracking, robot navigation and robotic manipulation. Finally, we discuss
several challenges and future topics for using adversarial learning, RL and
meta-learning in autonomous systems
InverseNet: Solving Inverse Problems with Splitting Networks
We propose a new method that uses deep learning techniques to solve the
inverse problems. The inverse problem is cast in the form of learning an
end-to-end mapping from observed data to the ground-truth. Inspired by the
splitting strategy widely used in regularized iterative algorithm to tackle
inverse problems, the mapping is decomposed into two networks, with one
handling the inversion of the physical forward model associated with the data
term and one handling the denoising of the output from the former network,
i.e., the inverted version, associated with the prior/regularization term. The
two networks are trained jointly to learn the end-to-end mapping, getting rid
of a two-step training. The training is annealing as the intermediate variable
between these two networks bridges the gap between the input (the degraded
version of output) and output and progressively approaches to the ground-truth.
The proposed network, referred to as InverseNet, is flexible in the sense that
most of the existing end-to-end network structure can be leveraged in the first
network and most of the existing denoising network structure can be used in the
second one. Extensive experiments on both synthetic data and real datasets on
the tasks, motion deblurring, super-resolution, and colorization, demonstrate
the efficiency and accuracy of the proposed method compared with other image
processing algorithms
A Deep Optimization Approach for Image Deconvolution
In blind image deconvolution, priors are often leveraged to constrain the
solution space, so as to alleviate the under-determinacy. Priors which are
trained separately from the task of deconvolution tend to be instable, or
ineffective. We propose the Golf Optimizer, a novel but simple form of network
that learns deep priors from data with better propagation behavior. Like
playing golf, our method first estimates an aggressive propagation towards
optimum using one network, and recurrently applies a residual CNN to learn the
gradient of prior for delicate correction on restoration. Experiments show that
our network achieves competitive performance on GoPro dataset, and our model is
extremely lightweight compared with the state-of-art works.Comment: 12 pages, 16 figure
Gyroscope-Aided Motion Deblurring with Deep Networks
We propose a deblurring method that incorporates gyroscope measurements into
a convolutional neural network (CNN). With the help of such measurements, it
can handle extremely strong and spatially-variant motion blur. At the same
time, the image data is used to overcome the limitations of gyro-based blur
estimation. To train our network, we also introduce a novel way of generating
realistic training data using the gyroscope. The evaluation shows a clear
improvement in visual quality over the state-of-the-art while achieving
real-time performance. Furthermore, the method is shown to improve the
performance of existing feature detectors and descriptors against the motion
blur
Generative Imaging and Image Processing via Generative Encoder
This paper introduces a novel generative encoder (GE) model for generative
imaging and image processing with applications in compressed sensing and
imaging, image compression, denoising, inpainting, deblurring, and
super-resolution. The GE model consists of a pre-training phase and a solving
phase. In the pre-training phase, we separately train two deep neural networks:
a generative adversarial network (GAN) with a generator \G that captures the
data distribution of a given image set, and an auto-encoder (AE) network with
an encoder \EN that compresses images following the estimated distribution by
GAN. In the solving phase, given a noisy image , where
is the target unknown image, is an operator adding an
addictive, or multiplicative, or convolutional noise, or equivalently given
such an image in the compressed domain, i.e., given m=\EN(x), we solve
the optimization problem
z^*=\underset{z}{\mathrm{argmin}} \|\EN(\G(z))-m\|_2^2+\lambda\|z\|_2^2
to recover the image in a generative way via
\hat{x}:=\G(z^*)\approx x^*, where is a hyperparameter. The GE
model unifies the generative capacity of GANs and the stability of AEs in an
optimization framework above instead of stacking GANs and AEs into a single
network or combining their loss functions into one as in existing literature.
Numerical experiments show that the proposed model outperforms several
state-of-the-art algorithms
Super-resolution MRI through Deep Learning
Magnetic resonance imaging (MRI) is extensively used for diagnosis and
image-guided therapeutics. Due to hardware, physical and physiological
limitations, acquisition of high-resolution MRI data takes long scan time at
high system cost, and could be limited to low spatial coverage and also subject
to motion artifacts. Super-resolution MRI can be achieved with deep learning,
which is a promising approach and has a great potential for preclinical and
clinical imaging. Compared with polynomial interpolation or sparse-coding
algorithms, deep learning extracts prior knowledge from big data and produces
superior MRI images from a low-resolution counterpart. In this paper, we adapt
two state-of-the-art neural network models for CT denoising and deblurring,
transfer them for super-resolution MRI, and demonstrate encouraging
super-resolution MRI results toward two-fold resolution enhancement
NTIRE 2020 Challenge on Image and Video Deblurring
Motion blur is one of the most common degradation artifacts in dynamic scene
photography. This paper reviews the NTIRE 2020 Challenge on Image and Video
Deblurring. In this challenge, we present the evaluation results from 3
competition tracks as well as the proposed solutions. Track 1 aims to develop
single-image deblurring methods focusing on restoration quality. On Track 2,
the image deblurring methods are executed on a mobile platform to find the
balance of the running speed and the restoration accuracy. Track 3 targets
developing video deblurring methods that exploit the temporal relation between
input frames. In each competition, there were 163, 135, and 102 registered
participants and in the final testing phase, 9, 4, and 7 teams competed. The
winning methods demonstrate the state-ofthe-art performance on image and video
deblurring tasks.Comment: To be published in CVPR 2020 Workshop (New Trends in Image
Restoration and Enhancement
Model Adaptation for Inverse Problems in Imaging
Deep neural networks have been applied successfully to a wide variety of
inverse problems arising in computational imaging. These networks are typically
trained using a forward model that describes the measurement process to be
inverted, which is often incorporated directly into the network itself.
However, these approaches lack robustness to drift of the forward model: if at
test time the forward model varies (even slightly) from the one the network was
trained for, the reconstruction performance can degrade substantially. Given a
network trained to solve an initial inverse problem with a known forward model,
we propose two novel procedures that adapt the network to a perturbed forward
model, even without full knowledge of the perturbation. Our approaches do not
require access to more labeled data (i.e., ground truth images), but only a
small set of calibration measurements. We show these simple model adaptation
procedures empirically achieve robustness to changes in the forward model in a
variety of settings, including deblurring, super-resolution, and undersampled
image reconstruction in magnetic resonance imaging
- …