14,017 research outputs found
Generative Adversarial Networks (GANs): Challenges, Solutions, and Future Directions
Generative Adversarial Networks (GANs) is a novel class of deep generative
models which has recently gained significant attention. GANs learns complex and
high-dimensional distributions implicitly over images, audio, and data.
However, there exists major challenges in training of GANs, i.e., mode
collapse, non-convergence and instability, due to inappropriate design of
network architecture, use of objective function and selection of optimization
algorithm. Recently, to address these challenges, several solutions for better
design and optimization of GANs have been investigated based on techniques of
re-engineered network architectures, new objective functions and alternative
optimization algorithms. To the best of our knowledge, there is no existing
survey that has particularly focused on broad and systematic developments of
these solutions. In this study, we perform a comprehensive survey of the
advancements in GANs design and optimization solutions proposed to handle GANs
challenges. We first identify key research issues within each design and
optimization technique and then propose a new taxonomy to structure solutions
by key research issues. In accordance with the taxonomy, we provide a detailed
discussion on different GANs variants proposed within each solution and their
relationships. Finally, based on the insights gained, we present the promising
research directions in this rapidly growing field.Comment: 42 pages, Figure 13, Table
Gradient-free activation maximization for identifying effective stimuli
A fundamental question for understanding brain function is what types of
stimuli drive neurons to fire. In visual neuroscience, this question has also
been posted as characterizing the receptive field of a neuron. The search for
effective stimuli has traditionally been based on a combination of insights
from previous studies, intuition, and luck. Recently, the same question has
emerged in the study of units in convolutional neural networks (ConvNets), and
together with this question a family of solutions were developed that are
generally referred to as "feature visualization by activation maximization."
We sought to bring in tools and techniques developed for studying ConvNets to
the study of biological neural networks. However, one key difference that
impedes direct translation of tools is that gradients can be obtained from
ConvNets using backpropagation, but such gradients are not available from the
brain. To circumvent this problem, we developed a method for gradient-free
activation maximization by combining a generative neural network with a genetic
algorithm. We termed this method XDream (EXtending DeepDream with real-time
evolution for activation maximization), and we have shown that this method can
reliably create strong stimuli for neurons in the macaque visual cortex (Ponce
et al., 2019). In this paper, we describe extensive experiments characterizing
the XDream method by using ConvNet units as in silico models of neurons. We
show that XDream is applicable across network layers, architectures, and
training sets; examine design choices in the algorithm; and provide practical
guides for choosing hyperparameters in the algorithm. XDream is an efficient
algorithm for uncovering neuronal tuning preferences in black-box networks
using a vast and diverse stimulus space.Comment: 16 pages, 8 figures, 3 table
Iterative Amortized Inference
Inference models are a key component in scaling variational inference to deep
latent variable models, most notably as encoder networks in variational
auto-encoders (VAEs). By replacing conventional optimization-based inference
with a learned model, inference is amortized over data examples and therefore
more computationally efficient. However, standard inference models are
restricted to direct mappings from data to approximate posterior estimates. The
failure of these models to reach fully optimized approximate posterior
estimates results in an amortization gap. We aim toward closing this gap by
proposing iterative inference models, which learn to perform inference
optimization through repeatedly encoding gradients. Our approach generalizes
standard inference models in VAEs and provides insight into several empirical
findings, including top-down inference techniques. We demonstrate the inference
optimization capabilities of iterative inference models and show that they
outperform standard inference models on several benchmark data sets of images
and text.Comment: International Conference on Machine Learning (ICML) 201
SGAN: An Alternative Training of Generative Adversarial Networks
The Generative Adversarial Networks (GANs) have demonstrated impressive
performance for data synthesis, and are now used in a wide range of computer
vision tasks. In spite of this success, they gained a reputation for being
difficult to train, what results in a time-consuming and human-involved
development process to use them.
We consider an alternative training process, named SGAN, in which several
adversarial "local" pairs of networks are trained independently so that a
"global" supervising pair of networks can be trained against them. The goal is
to train the global pair with the corresponding ensemble opponent for improved
performances in terms of mode coverage. This approach aims at increasing the
chances that learning will not stop for the global pair, preventing both to be
trapped in an unsatisfactory local minimum, or to face oscillations often
observed in practice. To guarantee the latter, the global pair never affects
the local ones.
The rules of SGAN training are thus as follows: the global generator and
discriminator are trained using the local discriminators and generators,
respectively, whereas the local networks are trained with their fixed local
opponent.
Experimental results on both toy and real-world problems demonstrate that
this approach outperforms standard training in terms of better mitigating mode
collapse, stability while converging and that it surprisingly, increases the
convergence speed as well
- …