46 research outputs found
On Unifying Deep Generative Models
Deep generative models have achieved impressive success in recent years.
Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs), as
emerging families for generative model learning, have largely been considered
as two distinct paradigms and received extensive independent studies
respectively. This paper aims to establish formal connections between GANs and
VAEs through a new formulation of them. We interpret sample generation in GANs
as performing posterior inference, and show that GANs and VAEs involve
minimizing KL divergences of respective posterior and inference distributions
with opposite directions, extending the two learning phases of classic
wake-sleep algorithm, respectively. The unified view provides a powerful tool
to analyze a diverse set of existing model variants, and enables to transfer
techniques across research lines in a principled way. For example, we apply the
importance weighting method in VAE literatures for improved GAN learning, and
enhance VAEs with an adversarial mechanism that leverages generated samples.
Experiments show generality and effectiveness of the transferred techniques.Comment: Polished and extended content over the ICLR conference version:
https://openreview.net/pdf?id=rylSzl-R
Training Generative Adversarial Networks with Weights
The impressive success of Generative Adversarial Networks (GANs) is often
overshadowed by the difficulties in their training. Despite the continuous
efforts and improvements, there are still open issues regarding their
convergence properties. In this paper, we propose a simple training variation
where suitable weights are defined and assist the training of the Generator. We
provide theoretical arguments why the proposed algorithm is better than the
baseline training in the sense of speeding up the training process and of
creating a stronger Generator. Performance results showed that the new
algorithm is more accurate in both synthetic and image datasets resulting in
improvements ranging between 5% and 50%.Comment: 6 pages, 3 figures, submitted to Icassp201
GLSR-VAE: Geodesic Latent Space Regularization for Variational AutoEncoder Architectures
VAEs (Variational AutoEncoders) have proved to be powerful in the context of
density modeling and have been used in a variety of contexts for creative
purposes. In many settings, the data we model possesses continuous attributes
that we would like to take into account at generation time. We propose in this
paper GLSR-VAE, a Geodesic Latent Space Regularization for the Variational
AutoEncoder architecture and its generalizations which allows a fine control on
the embedding of the data into the latent space. When augmenting the VAE loss
with this regularization, changes in the learned latent space reflects changes
of the attributes of the data. This deeper understanding of the VAE latent
space structure offers the possibility to modulate the attributes of the
generated data in a continuous way. We demonstrate its efficiency on a
monophonic music generation task where we manage to generate variations of
discrete sequences in an intended and playful way.Comment: 11 page
Concept-Oriented Deep Learning: Generative Concept Representations
Generative concept representations have three major advantages over
discriminative ones: they can represent uncertainty, they support integration
of learning and reasoning, and they are good for unsupervised and
semi-supervised learning. We discuss probabilistic and generative deep
learning, which generative concept representations are based on, and the use of
variational autoencoders and generative adversarial networks for learning
generative concept representations, particularly for concepts whose data are
sequences, structured data or graphs
Nonparametric Variational Auto-encoders for Hierarchical Representation Learning
The recently developed variational autoencoders (VAEs) have proved to be an
effective confluence of the rich representational power of neural networks with
Bayesian methods. However, most work on VAEs use a rather simple prior over the
latent variables such as standard normal distribution, thereby restricting its
applications to relatively simple phenomena. In this work, we propose
hierarchical nonparametric variational autoencoders, which combines
tree-structured Bayesian nonparametric priors with VAEs, to enable infinite
flexibility of the latent representation space. Both the neural parameters and
Bayesian priors are learned jointly using tailored variational inference. The
resulting model induces a hierarchical structure of latent semantic concepts
underlying the data corpus, and infers accurate representations of data
instances. We apply our model in video representation learning. Our method is
able to discover highly interpretable activity hierarchies, and obtain improved
clustering accuracy and generalization capacity based on the learned rich
representations.Comment: Accepted in ICCV 201
Deep Generative Models with Learnable Knowledge Constraints
The broad set of deep generative models (DGMs) has achieved remarkable
advances. However, it is often difficult to incorporate rich structured domain
knowledge with the end-to-end DGMs. Posterior regularization (PR) offers a
principled framework to impose structured constraints on probabilistic models,
but has limited applicability to the diverse DGMs that can lack a Bayesian
formulation or even explicit density evaluation. PR also requires constraints
to be fully specified a priori, which is impractical or suboptimal for complex
knowledge with learnable uncertain parts. In this paper, we establish
mathematical correspondence between PR and reinforcement learning (RL), and,
based on the connection, expand PR to learn constraints as the extrinsic reward
in RL. The resulting algorithm is model-agnostic to apply to any DGMs, and is
flexible to adapt arbitrary constraints with the model jointly. Experiments on
human image generation and templated sentence generation show models with
learned knowledge constraints by our algorithm greatly improve over base
generative models.Comment: Neural Information Processing Systems (NeurIPS) 201
Texar: A Modularized, Versatile, and Extensible Toolkit for Text Generation
We introduce Texar, an open-source toolkit aiming to support the broad set of
text generation tasks that transform any inputs into natural language, such as
machine translation, summarization, dialog, content manipulation, and so forth.
With the design goals of modularity, versatility, and extensibility in mind,
Texar extracts common patterns underlying the diverse tasks and methodologies,
creates a library of highly reusable modules, and allows arbitrary model
architectures and algorithmic paradigms. In Texar, model architecture,
inference, and learning processes are properly decomposed. Modules at a high
concept level can be freely assembled and plugged in/swapped out. The toolkit
also supports a rich set of large-scale pretrained models. Texar is thus
particularly suitable for researchers and practitioners to do fast prototyping
and experimentation. The versatile toolkit also fosters technique sharing
across different text generation tasks. Texar supports both TensorFlow and
PyTorch, and is released under Apache License 2.0 at https://www.texar.io.Comment: ACL 2019 demo, expanded versio
Generative Latent Flow
In this work, we propose the Generative Latent Flow (GLF), an algorithm for
generative modeling of the data distribution. GLF uses an Auto-encoder (AE) to
learn latent representations of the data, and a normalizing flow to map the
distribution of the latent variables to that of simple i.i.d noise. In contrast
to some other Auto-encoder based generative models, which use various
regularizers that encourage the encoded latent distribution to match the prior
distribution, our model explicitly constructs a mapping between these two
distributions, leading to better density matching while avoiding over
regularizing the latent variables. We compare our model with several related
techniques, and show that it has many relative advantages including fast
convergence, single stage training and minimal reconstruction trade-off. We
also study the relationship between our model and its stochastic counterpart,
and show that our model can be viewed as a vanishing noise limit of VAEs with
flow prior. Quantitatively, under standardized evaluations, our method achieves
state-of-the-art sample quality among AE based models on commonly used
datasets, and is competitive with GANs' benchmarks
Toward Controlled Generation of Text
Generic generation and manipulation of text is challenging and has limited
success compared to recent deep generative modeling in visual domain. This
paper aims at generating plausible natural language sentences, whose attributes
are dynamically controlled by learning disentangled latent representations with
designated semantics. We propose a new neural generative model which combines
variational auto-encoders and holistic attribute discriminators for effective
imposition of semantic structures. With differentiable approximation to
discrete text samples, explicit constraints on independent attribute controls,
and efficient collaborative learning of generator and discriminators, our model
learns highly interpretable representations from even only word annotations,
and produces realistic sentences with desired attributes. Quantitative
evaluation validates the accuracy of sentence and attribute generation.Comment: Code adapted for text style transfer is released at:
https://github.com/asyml/texar/tree/master/examples/text_style_transfe
Constrained Generation of Semantically Valid Graphs via Regularizing Variational Autoencoders
Deep generative models have achieved remarkable success in various data
domains, including images, time series, and natural languages. There remain,
however, substantial challenges for combinatorial structures, including graphs.
One of the key challenges lies in the difficulty of ensuring semantic validity
in context. For examples, in molecular graphs, the number of bonding-electron
pairs must not exceed the valence of an atom; whereas in protein interaction
networks, two proteins may be connected only when they belong to the same or
correlated gene ontology terms. These constraints are not easy to be
incorporated into a generative model. In this work, we propose a regularization
framework for variational autoencoders as a step toward semantic validity. We
focus on the matrix representation of graphs and formulate penalty terms that
regularize the output distribution of the decoder to encourage the satisfaction
of validity constraints. Experimental results confirm a much higher likelihood
of sampling valid graphs in our approach, compared with others reported in the
literature.Comment: NIPS 201