10,164 research outputs found
An Information-theoretic analysis of generative adversarial networks for image restoration in physics-based vision
Image restoration in physics-based vision (such as image denoising, dehazing, and deraining) are fundamental tasks in computer vision that attach great significance to the processing of visual data as well as subsequent applications in different fields. Existing methods mainly focus on exploring the physical properties and mechanisms of the imaging process, and tend to use a deconstructive idea in describing how the visual degradations (like noise, haze, and rain) are integrated with the background scenes. This idea, however, relies heavily on manually engineered features and handcrafted composition models, which can be theories only in ideal conditions or hypothetical models that may involve human bias or fail in simulating true situations in actual practices. With the progress of representation learning, generative methods, especially generative adversarial networks (GANs), are considered a more promising solution for image restoration tasks. It directly learns the restorations as end-to-end generation processes using large amounts of data without understanding their physical mechanisms, and it also allows completing missing details damaged information by involving external knowledge and generating plausible results with intelligent-level interpretation and semantics-level understanding of the input images. Nevertheless, existing studies that try to apply GAN models to image restoration tasks dose not achieve satisfactory performances compared with the traditional deconstructive methods. And there is scarcely any study or theory to explain how deep generative models work in relevant tasks.
In this study, we analyzed the learning dynamics of different deep generative models based on the information bottleneck principle and propose an information-theoretic framework to explain the generative methods for image restoration tasks. In which, we study the information flow in the image restoration models and point out three sources of information involved in generating the restoration results: (i) high-level information extracted by the encoder network, (ii) low-level information from the source inputs that retained, or pass directed through the skip connections, and, (iii) external information introduced by the learned parameters of the decoder network during the generation process.
Based on this theory, we pointed out that conventional GAN models may not be directly applicable to the tasks of image restoration, and we identify three key issues leading to their performance gaps in the image restoration tasks: (i) over-invested abstraction processes, (ii) inherent details loss, and (iii) imbalance optimization with vanishing gradient. We formulate these problems with corresponding theoretical analyses and provide empirical evidence to verify our hypotheses and prove the existence of these problems respectively.
To address these problems, we then proposed solutions and suggestions including optimizing network structure, enhancing details extraction and accumulation with network modules, as well as replacing measures of training objectives, to improve the performances of GAN models on the image restoration tasks. Ultimately, we verify our solutions on bench-marking datasets and achieve significant improvement on the baseline models
Latent Convolutional Models
We present a new latent model of natural images that can be learned on
large-scale datasets. The learning process provides a latent embedding for
every image in the training dataset, as well as a deep convolutional network
that maps the latent space to the image space. After training, the new model
provides a strong and universal image prior for a variety of image restoration
tasks such as large-hole inpainting, superresolution, and colorization. To
model high-resolution natural images, our approach uses latent spaces of very
high dimensionality (one to two orders of magnitude higher than previous latent
image models). To tackle this high dimensionality, we use latent spaces with a
special manifold structure (convolutional manifolds) parameterized by a ConvNet
of a certain architecture. In the experiments, we compare the learned latent
models with latent models learned by autoencoders, advanced variants of
generative adversarial networks, and a strong baseline system using simpler
parameterization of the latent space. Our model outperforms the competing
approaches over a range of restoration tasks.Comment: Updated with more recent experiment
Physics-Based Generative Adversarial Models for Image Restoration and Beyond
We present an algorithm to directly solve numerous image restoration problems
(e.g., image deblurring, image dehazing, image deraining, etc.). These problems
are highly ill-posed, and the common assumptions for existing methods are
usually based on heuristic image priors. In this paper, we find that these
problems can be solved by generative models with adversarial learning. However,
the basic formulation of generative adversarial networks (GANs) does not
generate realistic images, and some structures of the estimated images are
usually not preserved well. Motivated by an interesting observation that the
estimated results should be consistent with the observed inputs under the
physics models, we propose a physics model constrained learning algorithm so
that it can guide the estimation of the specific task in the conventional GAN
framework. The proposed algorithm is trained in an end-to-end fashion and can
be applied to a variety of image restoration and related low-level vision
problems. Extensive experiments demonstrate that our method performs favorably
against the state-of-the-art algorithms.Comment: IEEE TPAM
Image Restoration from Parametric Transformations using Generative Models
When images are statistically described by a generative model we can use this
information to develop optimum techniques for various image restoration
problems as inpainting, super-resolution, image coloring, generative model
inversion, etc. With the help of the generative model it is possible to
formulate, in a natural way, these restoration problems as Statistical
estimation problems. Our approach, by combining maximum a-posteriori
probability with maximum likelihood estimation, is capable of restoring images
that are distorted by transformations even when the latter contain unknown
parameters. The resulting optimization is completely defined with no parameters
requiring tuning. This must be compared with the current state of the art which
requires exact knowledge of the transformations and contains regularizer terms
with weights that must be properly defined. Finally, we must mention that we
extend our method to accommodate mixtures of multiple images where each image
is described by its own generative model and we are able of successfully
separating each participating image from a single mixture
Generative Adversarial Network based on Resnet for Conditional Image Restoration
The GANs promote an adversarive game to approximate complex and jointed
example probability. The networks driven by noise generate fake examples to
approximate realistic data distributions. Later the conditional GAN merges
prior-conditions as input in order to transfer attribute vectors to the
corresponding data. However, the CGAN is not designed to deal with the high
dimension conditions since indirect guide of the learning is inefficiency. In
this paper, we proposed a network ResGAN to generate fine images in terms of
extremely degenerated images. The coarse images aligned to attributes are
embedded as the generator inputs and classifier labels. In generative network,
a straight path similar to the Resnet is cohered to directly transfer the
coarse images to the higher layers. And adversarial training is circularly
implemented to prevent degeneration of the generated images. Experimental
results of applying the ResGAN to datasets MNIST, CIFAR10/100 and CELEBA show
its higher accuracy to the state-of-art GANs.Comment: 6 pages, 15 figures, conferenc
Deep Likelihood Network for Image Restoration with Multiple Degradation Levels
Convolutional neural networks have been proven effective in a variety of
image restoration tasks. Most state-of-the-art solutions, however, are trained
using images with a single particular degradation level, and their performance
deteriorates drastically when applied to other degradation settings. In this
paper, we propose deep likelihood network (DL-Net), aiming at generalizing
off-the-shelf image restoration networks to succeed over a spectrum of
degradation levels. We slightly modify an off-the-shelf network by appending a
simple recursive module, which is derived from a fidelity term, for
disentangling the computation for multiple degradation levels. Extensive
experimental results on image inpainting, interpolation, and super-resolution
show the effectiveness of our DL-Net.Comment: Accepted by IEEE Transactions on Image Processing; 13 pages, 6
figure
A Deep Optimization Approach for Image Deconvolution
In blind image deconvolution, priors are often leveraged to constrain the
solution space, so as to alleviate the under-determinacy. Priors which are
trained separately from the task of deconvolution tend to be instable, or
ineffective. We propose the Golf Optimizer, a novel but simple form of network
that learns deep priors from data with better propagation behavior. Like
playing golf, our method first estimates an aggressive propagation towards
optimum using one network, and recurrently applies a residual CNN to learn the
gradient of prior for delicate correction on restoration. Experiments show that
our network achieves competitive performance on GoPro dataset, and our model is
extremely lightweight compared with the state-of-art works.Comment: 12 pages, 16 figure
Generative Reversible Data Hiding by Image to Image Translation via GANs
The traditional reversible data hiding technique is based on cover image
modification which inevitably leaves some traces of rewriting that can be more
easily analyzed and attacked by the warder. Inspired by the cover synthesis
steganography based generative adversarial networks, in this paper, a novel
generative reversible data hiding scheme (GRDH) by image translation is
proposed. First, an image generator is used to obtain a realistic image, which
is used as an input to the image-to-image translation model with CycleGAN.
After image translation, a stego image with different semantic information will
be obtained. The secret message and the original input image can be recovered
separately by a well-trained message extractor and the inverse transform of the
image translation. Experimental results have verified the effectiveness of the
scheme
Multi-Scale Face Restoration with Sequential Gating Ensemble Network
Restoring face images from distortions is important in face recognition
applications and is challenged by multiple scale issues, which is still not
well-solved in research area. In this paper, we present a Sequential Gating
Ensemble Network (SGEN) for multi-scale face restoration issue. We first employ
the principle of ensemble learning into SGEN architecture design to reinforce
predictive performance of the network. The SGEN aggregates multi-level
base-encoders and base-decoders into the network, which enables the network to
contain multiple scales of receptive field. Instead of combining these
base-en/decoders directly with non-sequential operations, the SGEN takes
base-en/decoders from different levels as sequential data. Specifically, the
SGEN learns to sequentially extract high level information from base-encoders
in bottom-up manner and restore low level information from base-decoders in
top-down manner. Besides, we propose to realize bottom-up and top-down
information combination and selection with Sequential Gating Unit (SGU). The
SGU sequentially takes two inputs from different levels and decides the output
based on one active input. Experiment results demonstrate that our SGEN is more
effective at multi-scale human face restoration with more image details and
less noise than state-of-the-art image restoration models. By using adversarial
training, SGEN also produces more visually preferred results than other models
through subjective evaluation.Comment: 8 pages, 7 figures, Thirty-Second AAAI Conference on Artificial
Intelligence (AAAI-18
Underwater Color Restoration Using U-Net Denoising Autoencoder
Visual inspection of underwater structures by vehicles, e.g. remotely
operated vehicles (ROVs), plays an important role in scientific, military, and
commercial sectors. However, the automatic extraction of information using
software tools is hindered by the characteristics of water which degrade the
quality of captured videos. As a contribution for restoring the color of
underwater images, Underwater Denoising Autoencoder (UDAE) model is developed
using a denoising autoencoder with U-Net architecture. The proposed network
takes into consideration the accuracy and the computation cost to enable
real-time implementation on underwater visual tasks using end-to-end
autoencoder network. Underwater vehicles perception is improved by
reconstructing captured frames; hence obtaining better performance in
underwater tasks. Related learning methods use generative adversarial networks
(GANs) to generate color corrected underwater images, and to our knowledge this
paper is the first to deal with a single autoencoder capable of producing same
or better results. Moreover, image pairs are constructed for training the
proposed network, where it is hard to obtain such dataset from underwater
scenery. At the end, the proposed model is compared to a state-of-the-art
method.Comment: 6 pages, 8 figure
- …