2,088 research outputs found
A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues
Sequential data often possesses a hierarchical structure with complex
dependencies between subsequences, such as found between the utterances in a
dialogue. In an effort to model this kind of generative process, we propose a
neural network-based generative architecture, with latent stochastic variables
that span a variable number of time steps. We apply the proposed model to the
task of dialogue response generation and compare it with recent neural network
architectures. We evaluate the model performance through automatic evaluation
metrics and by carrying out a human evaluation. The experiments demonstrate
that our model improves upon recently proposed models and that the latent
variables facilitate the generation of long outputs and maintain the context.Comment: 15 pages, 5 tables, 4 figure
Improving Variational Encoder-Decoders in Dialogue Generation
Variational encoder-decoders (VEDs) have shown promising results in dialogue
generation. However, the latent variable distributions are usually approximated
by a much simpler model than the powerful RNN structure used for encoding and
decoding, yielding the KL-vanishing problem and inconsistent training
objective. In this paper, we separate the training step into two phases: The
first phase learns to autoencode discrete texts into continuous embeddings,
from which the second phase learns to generalize latent representations by
reconstructing the encoded embedding. In this case, latent variables are
sampled by transforming Gaussian noise through multi-layer perceptrons and are
trained with a separate VED model, which has the potential of realizing a much
more flexible distribution. We compare our model with current popular models
and the experiment demonstrates substantial improvement in both metric-based
and human evaluations.Comment: Accepted by AAAI201
Multi-level Memory for Task Oriented Dialogs
Recent end-to-end task oriented dialog systems use memory architectures to
incorporate external knowledge in their dialogs. Current work makes simplifying
assumptions about the structure of the knowledge base, such as the use of
triples to represent knowledge, and combines dialog utterances (context) as
well as knowledge base (KB) results as part of the same memory. This causes an
explosion in the memory size, and makes the reasoning over memory harder. In
addition, such a memory design forces hierarchical properties of the data to be
fit into a triple structure of memory. This requires the memory reader to infer
relationships across otherwise connected attributes. In this paper we relax the
strong assumptions made by existing architectures and separate memories used
for modeling dialog context and KB results. Instead of using triples to store
KB results, we introduce a novel multi-level memory architecture consisting of
cells for each query and their corresponding results. The multi-level memory
first addresses queries, followed by results and finally each key-value pair
within a result. We conduct detailed experiments on three publicly available
task oriented dialog data sets and we find that our method conclusively
outperforms current state-of-the-art models. We report a 15-25% increase in
both entity F1 and BLEU scores.Comment: Accepted as full paper at NAACL 201
Text Generation Based on Generative Adversarial Nets with Latent Variable
In this paper, we propose a model using generative adversarial net (GAN) to
generate realistic text. Instead of using standard GAN, we combine variational
autoencoder (VAE) with generative adversarial net. The use of high-level latent
random variables is helpful to learn the data distribution and solve the
problem that generative adversarial net always emits the similar data. We
propose the VGAN model where the generative model is composed of recurrent
neural network and VAE. The discriminative model is a convolutional neural
network. We train the model via policy gradient. We apply the proposed model to
the task of text generation and compare it to other recent neural network based
models, such as recurrent neural network language model and SeqGAN. We evaluate
the performance of the model by calculating negative log-likelihood and the
BLEU score. We conduct experiments on three benchmark datasets, and results
show that our model outperforms other previous models
- …