419 research outputs found
Learning Disentangled Representations for Natural Language Definitions
Disentangling the encodings of neural models is a fundamental aspect for
improving interpretability, semantic control and downstream task performance in
Natural Language Processing. Currently, most disentanglement methods are
unsupervised or rely on synthetic datasets with known generative factors. We
argue that recurrent syntactic and semantic regularities in textual data can be
used to provide the models with both structural biases and generative factors.
We leverage the semantic structures present in a representative and
semantically dense category of sentence types, definitional sentences, for
training a Variational Autoencoder to learn disentangled representations. Our
experimental results show that the proposed model outperforms unsupervised
baselines on several qualitative and quantitative benchmarks for
disentanglement, and it also improves the results in the downstream task of
definition modeling.Comment: Findings of EACL 202
Unsupervised Syntactically Controlled Paraphrase Generation with Abstract Meaning Representations
Syntactically controlled paraphrase generation has become an emerging
research direction in recent years. Most existing approaches require annotated
paraphrase pairs for training and are thus costly to extend to new domains.
Unsupervised approaches, on the other hand, do not need paraphrase pairs but
suffer from relatively poor performance in terms of syntactic control and
quality of generated paraphrases. In this paper, we demonstrate that leveraging
Abstract Meaning Representations (AMR) can greatly improve the performance of
unsupervised syntactically controlled paraphrase generation. Our proposed
model, AMR-enhanced Paraphrase Generator (AMRPG), separately encodes the AMR
graph and the constituency parse of the input sentence into two disentangled
semantic and syntactic embeddings. A decoder is then learned to reconstruct the
input sentence from the semantic and syntactic embeddings. Our experiments show
that AMRPG generates more accurate syntactically controlled paraphrases, both
quantitatively and qualitatively, compared to the existing unsupervised
approaches. We also demonstrate that the paraphrases generated by AMRPG can be
used for data augmentation to improve the robustness of NLP models.Comment: Paper accepted by EMNLP 2022 Findings. The first two authors
contribute equall
Unsupervised Opinion Summarisation in the Wasserstein Space
Opinion summarisation synthesises opinions expressed in a group of documents
discussing the same topic to produce a single summary. Recent work has looked
at opinion summarisation of clusters of social media posts. Such posts are
noisy and have unpredictable structure, posing additional challenges for the
construction of the summary distribution and the preservation of meaning
compared to online reviews, which has been so far the focus of opinion
summarisation. To address these challenges we present \textit{WassOS}, an
unsupervised abstractive summarization model which makes use of the Wasserstein
distance. A Variational Autoencoder is used to get the distribution of
documents/posts, and the distributions are disentangled into separate semantic
and syntactic spaces. The summary distribution is obtained using the
Wasserstein barycenter of the semantic and syntactic distributions. A latent
variable sampled from the summary distribution is fed into a GRU decoder with a
transformer layer to produce the final summary. Our experiments on multiple
datasets including Twitter clusters, Reddit threads, and reviews show that
WassOS almost always outperforms the state-of-the-art on ROUGE metrics and
consistently produces the best summaries with respect to meaning preservation
according to human evaluations
Controllable Paraphrase Generation with a Syntactic Exemplar
Prior work on controllable text generation usually assumes that the
controlled attribute can take on one of a small set of values known a priori.
In this work, we propose a novel task, where the syntax of a generated sentence
is controlled rather by a sentential exemplar. To evaluate quantitatively with
standard metrics, we create a novel dataset with human annotations. We also
develop a variational model with a neural module specifically designed for
capturing syntactic knowledge and several multitask training objectives to
promote disentangled representation learning. Empirically, the proposed model
is observed to achieve improvements over baselines and learn to capture
desirable characteristics.Comment: ACL 2019 Lon
Graph-Induced Syntactic-Semantic Spaces in Transformer-Based Variational AutoEncoders
The injection of syntactic information in Variational AutoEncoders (VAEs) has
been shown to result in an overall improvement of performances and
generalisation. An effective strategy to achieve such a goal is to separate the
encoding of distributional semantic features and syntactic structures into
heterogeneous latent spaces via multi-task learning or dual encoder
architectures. However, existing works employing such techniques are limited to
LSTM-based VAEs. In this paper, we investigate latent space separation methods
for structural syntactic injection in Transformer-based VAE architectures
(i.e., Optimus). Specifically, we explore how syntactic structures can be
leveraged in the encoding stage through the integration of graph-based and
sequential models, and how multiple, specialised latent representations can be
injected into the decoder's attention mechanism via low-rank operators. Our
empirical evaluation, carried out on natural language sentences and
mathematical expressions, reveals that the proposed end-to-end VAE architecture
can result in a better overall organisation of the latent space, alleviating
the information loss occurring in standard VAE setups, resulting in enhanced
performances on language modelling and downstream generation tasks
- …