24,006 research outputs found
Learning Implicit Generative Models with the Method of Learned Moments
We propose a method of moments (MoM) algorithm for training large-scale
implicit generative models. Moment estimation in this setting encounters two
problems: it is often difficult to define the millions of moments needed to
learn the model parameters, and it is hard to determine which properties are
useful when specifying moments. To address the first issue, we introduce a
moment network, and define the moments as the network's hidden units and the
gradient of the network's output with the respect to its parameters. To tackle
the second problem, we use asymptotic theory to highlight desiderata for
moments -- namely they should minimize the asymptotic variance of estimated
model parameters -- and introduce an objective to learn better moments. The
sequence of objectives created by this Method of Learned Moments (MoLM) can
train high-quality neural image samplers. On CIFAR-10, we demonstrate that
MoLM-trained generators achieve significantly higher Inception Scores and lower
Frechet Inception Distances than those trained with gradient
penalty-regularized and spectrally-normalized adversarial objectives. These
generators also achieve nearly perfect Multi-Scale Structural Similarity Scores
on CelebA, and can create high-quality samples of 128x128 images.Comment: ICML 2018, 6 figures, 17 page
Learning Implicit Generative Models by Matching Perceptual Features
Perceptual features (PFs) have been used with great success in tasks such as
transfer learning, style transfer, and super-resolution. However, the efficacy
of PFs as key source of information for learning generative models is not well
studied. We investigate here the use of PFs in the context of learning implicit
generative models through moment matching (MM). More specifically, we propose a
new effective MM approach that learns implicit generative models by performing
mean and covariance matching of features extracted from pretrained ConvNets.
Our proposed approach improves upon existing MM methods by: (1) breaking away
from the problematic min/max game of adversarial learning; (2) avoiding online
learning of kernel functions; and (3) being efficient with respect to both
number of used moments and required minibatch size. Our experimental results
demonstrate that, due to the expressiveness of PFs from pretrained deep
ConvNets, our method achieves state-of-the-art results for challenging
benchmarks.Comment: 16 page
Learning Implicit Generative Models Using Differentiable Graph Tests
Recently, there has been a growing interest in the problem of learning rich
implicit models - those from which we can sample, but can not evaluate their
density. These models apply some parametric function, such as a deep network,
to a base measure, and are learned end-to-end using stochastic optimization.
One strategy of devising a loss function is through the statistics of two
sample tests - if we can fool a statistical test, the learned distribution
should be a good model of the true data. However, not all tests can easily fit
into this framework, as they might not be differentiable with respect to the
data points, and hence with respect to the parameters of the implicit model.
Motivated by this problem, in this paper we show how two such classical tests,
the Friedman-Rafsky and k-nearest neighbour tests, can be effectively smoothed
using ideas from undirected graphical models - the matrix tree theorem and
cardinality potentials. Moreover, as we show experimentally, smoothing can
significantly increase the power of the test, which might of of independent
interest. Finally, we apply our method to learn implicit models
Approximate Inference with Amortised MCMC
We propose a novel approximate inference algorithm that approximates a target
distribution by amortising the dynamics of a user-selected MCMC sampler. The
idea is to initialise MCMC using samples from an approximation network, apply
the MCMC operator to improve these samples, and finally use the samples to
update the approximation network thereby improving its quality. This provides a
new generic framework for approximate inference, allowing us to deploy highly
complex, or implicitly defined approximation families with intractable
densities, including approximations produced by warping a source of randomness
through a deep neural network. Experiments consider image modelling with deep
generative models as a challenging test for the method. Deep models trained
using amortised MCMC are shown to generate realistic looking samples as well as
producing diverse imputations for images with regions of missing pixels
Asymmetric Variational Autoencoders
Variational inference for latent variable models is prevalent in various
machine learning problems, typically solved by maximizing the Evidence Lower
Bound (ELBO) of the true data likelihood with respect to a variational
distribution. However, freely enriching the family of variational distribution
is challenging since the ELBO requires variational likelihood evaluations of
the latent variables. In this paper, we propose a novel framework to enrich the
variational family by incorporating auxiliary variables to the variational
family. The resulting inference network doesn't require density evaluations for
the auxiliary variables and thus complex implicit densities over the auxiliary
variables can be constructed by neural networks. It can be shown that the
actual variational posterior of the proposed approach is essentially modeling a
rich probabilistic mixture of simple variational posterior indexed by auxiliary
variables, thus a flexible inference model can be built. Empirical evaluations
on several density estimation tasks demonstrates the effectiveness of the
proposed method.Comment: ICML 2018 Workshop on Theoretical Foundations and Applications of
Deep Generative Model
Parametric Adversarial Divergences are Good Task Losses for Generative Modeling
Generative modeling of high dimensional data like images is a notoriously
difficult and ill-defined problem. In particular, how to evaluate a learned
generative model is unclear. In this position paper, we argue that adversarial
learning, pioneered with generative adversarial networks (GANs), provides an
interesting framework to implicitly define more meaningful task losses for
generative modeling tasks, such as for generating "visually realistic" images.
We refer to those task losses as parametric adversarial divergences and we give
two main reasons why we think parametric divergences are good learning
objectives for generative modeling. Additionally, we unify the processes of
choosing a good structured loss (in structured prediction) and choosing a
discriminator architecture (in generative modeling) using statistical decision
theory; we are then able to formalize and quantify the intuition that "weaker"
losses are easier to learn from, in a specific setting. Finally, we propose two
new challenging tasks to evaluate parametric and nonparametric divergences: a
qualitative task of generating very high-resolution digits, and a quantitative
task of learning data that satisfies high-level algebraic constraints. We use
two common divergences to train a generator and show that the parametric
divergence outperforms the nonparametric divergence on both the qualitative and
the quantitative task.Comment: 22 page
Generative Adversarial Networks (GANs): What it can generate and What it cannot?
In recent years, Generative Adversarial Networks (GANs) have received
significant attention from the research community. With a straightforward
implementation and outstanding results, GANs have been used for numerous
applications. Despite the success, GANs lack a proper theoretical explanation.
These models suffer from issues like mode collapse, non-convergence, and
instability during training. To address these issues, researchers have proposed
theoretically rigorous frameworks inspired by varied fields of Game theory,
Statistical theory, Dynamical systems, etc.
In this paper, we propose to give an appropriate structure to study these
contributions systematically. We essentially categorize the papers based on the
issues they raise and the kind of novelty they introduce to address them.
Besides, we provide insight into how each of the discussed articles solves the
concerned problems. We compare and contrast different results and put forth a
summary of theoretical contributions about GANs with focus on image/visual
applications. We expect this summary paper to give a bird's eye view to a
person wishing to understand the theoretical progress in GANs so far
Variational Inference of Disentangled Latent Concepts from Unlabeled Observations
Disentangled representations, where the higher level data generative factors
are reflected in disjoint latent dimensions, offer several benefits such as
ease of deriving invariant representations, transferability to other tasks,
interpretability, etc. We consider the problem of unsupervised learning of
disentangled representations from large pool of unlabeled observations, and
propose a variational inference based approach to infer disentangled latent
factors. We introduce a regularizer on the expectation of the approximate
posterior over observed data that encourages the disentanglement. We also
propose a new disentanglement metric which is better aligned with the
qualitative disentanglement observed in the decoder's output. We empirically
observe significant improvement over existing methods in terms of both
disentanglement and data likelihood (reconstruction quality).Comment: ICLR 2018 Versio
Visually-Aware Fashion Recommendation and Design with Generative Image Models
Building effective recommender systems for domains like fashion is
challenging due to the high level of subjectivity and the semantic complexity
of the features involved (i.e., fashion styles). Recent work has shown that
approaches to `visual' recommendation (e.g.~clothing, art, etc.) can be made
more accurate by incorporating visual signals directly into the recommendation
objective, using `off-the-shelf' feature representations derived from deep
networks. Here, we seek to extend this contribution by showing that
recommendation performance can be significantly improved by learning `fashion
aware' image representations directly, i.e., by training the image
representation (from the pixel level) and the recommender system jointly; this
contribution is related to recent work using Siamese CNNs, though we are able
to show improvements over state-of-the-art recommendation techniques such as
BPR and variants that make use of pre-trained visual features. Furthermore, we
show that our model can be used \emph{generatively}, i.e., given a user and a
product category, we can generate new images (i.e., clothing items) that are
most consistent with their personal taste. This represents a first step towards
building systems that go beyond recommending existing items from a product
corpus, but which can be used to suggest styles and aid the design of new
products.Comment: 10 pages, 6 figures. Accepted by ICDM'17 as a long pape
Recent Advances in Autoencoder-Based Representation Learning
Learning useful representations with little or no supervision is a key
challenge in artificial intelligence. We provide an in-depth review of recent
advances in representation learning with a focus on autoencoder-based models.
To organize these results we make use of meta-priors believed useful for
downstream tasks, such as disentanglement and hierarchical organization of
features. In particular, we uncover three main mechanisms to enforce such
properties, namely (i) regularizing the (approximate or aggregate) posterior
distribution, (ii) factorizing the encoding and decoding distribution, or (iii)
introducing a structured prior distribution. While there are some promising
results, implicit or explicit supervision remains a key enabler and all current
methods use strong inductive biases and modeling assumptions. Finally, we
provide an analysis of autoencoder-based representation learning through the
lens of rate-distortion theory and identify a clear tradeoff between the amount
of prior knowledge available about the downstream tasks, and how useful the
representation is for this task.Comment: Presented at the third workshop on Bayesian Deep Learning (NeurIPS
2018
- …