19 research outputs found
Recommended from our members
Learning meaning representations for text generation with deep generative models
This thesis explores conditioning a language generation model with auxiliary variables. By doing so, we hope to be able to better control the output of the language generator. We explore several kinds of auxiliary variables in this thesis, from unstructured continuous, to discrete, to structured discrete auxiliary variables, and evaluate their advantages and disadvantages. We consider three primary axes of variation: how interpretable the auxiliary variables are, how much control they provide over the generated text, and whether the variables can be induced from unlabelled data. The latter consideration is particularly interesting: if we can show that induced latent variables correspond to the semantics of the generated utterance, then by manipulating the variables, we have fine-grained control over the meaning of the generated utterance, thereby learning simple meaning representations for text generation.
We investigate three language generation tasks: open domain conversational response generation, sentence generation from a semantic topic, and generating surface form realisations of meaning representations. We use a different type of auxiliary variable for each task, describe the reasons for choosing that type of variable, and critically discuss how much the task benefited from an auxiliary variable decomposition. All of the models that we use combine a high-level graphical model with a neural language model text generator. The graphical model lets us specify the structure of the text generating process, while the neural text generator can learn how to generate fluent text from a large corpus of examples. We aim to show the utility of such \textit{deep generative models} of text for text generation in the following work
Graph Transformer for Graph-to-Sequence Learning
The dominant graph-to-sequence transduction models employ graph neural
networks for graph representation learning, where the structural information is
reflected by the receptive field of neurons. Unlike graph neural networks that
restrict the information exchange between immediate neighborhood, we propose a
new model, known as Graph Transformer, that uses explicit relation encoding and
allows direct communication between two distant nodes. It provides a more
efficient way for global graph structure modeling. Experiments on the
applications of text generation from Abstract Meaning Representation (AMR) and
syntax-based neural machine translation show the superiority of our proposed
model. Specifically, our model achieves 27.4 BLEU on LDC2015E86 and 29.7 BLEU
on LDC2017T10 for AMR-to-text generation, outperforming the state-of-the-art
results by up to 2.2 points. On the syntax-based translation tasks, our model
establishes new single-model state-of-the-art BLEU scores, 21.3 for
English-to-German and 14.1 for English-to-Czech, improving over the existing
best results, including ensembles, by over 1 BLEU.Comment: accepted by AAAI202
Generating Text from Anonymised Structures
International audienceSurface realisation maps a meaning representation (MR) to a text, usually a single sentence. In this paper, we introduce a new parallel dataset of deep meaning representations and French sentences and we present a novel method for MR-to-text generation which seeks to generalise by abstracting away from lexical content. Most current work on natural language generation focuses on generating text that matches a reference using BLEU as evaluation criteria. In this paper, we additionally consider the model's ability to reintroduce the function words that are absent from the deep input meaning representations. We show that our approach increases both BLEU score and the scores used to assess function words generation
Generating Text from Anonymised Structures
International audienceSurface realisation maps a meaning representation (MR) to a text, usually a single sentence. In this paper, we introduce a new parallel dataset of deep meaning representations and French sentences and we present a novel method for MR-to-text generation which seeks to generalise by abstracting away from lexical content. Most current work on natural language generation focuses on generating text that matches a reference using BLEU as evaluation criteria. In this paper, we additionally consider the model's ability to reintroduce the function words that are absent from the deep input meaning representations. We show that our approach increases both BLEU score and the scores used to assess function words generation
Understanding and generating language with abstract meaning representation
Abstract Meaning Representation (AMR) is a semantic representation for natural
language that encompasses annotations related to traditional tasks such as
Named Entity Recognition (NER), Semantic Role Labeling (SRL), word sense
disambiguation (WSD), and Coreference Resolution. AMR represents sentences
as graphs, where nodes represent concepts and edges represent semantic
relations between them.
Sentences are represented as graphs and not trees because nodes can have
multiple incoming edges, called reentrancies. This thesis investigates the impact
of reentrancies for parsing (from text to AMR) and generation (from AMR
to text). For the parsing task, we showed that it is possible to use techniques
from tree parsing and adapt them to deal with reentrancies. To better analyze
the quality of AMR parsers, we developed a set of fine-grained metrics
and found that state-of-the-art parsers predict reentrancies poorly. Hence we
provided a classification of linguistic phenomena causing reentrancies, categorized
the type of errors parsers do with respect to reentrancies, and proved
that correcting these errors can lead to significant improvements. For the generation
task, we showed that neural encoders that have access to reentrancies
outperform those who do not, demonstrating the importance of reentrancies
also for generation.
This thesis also discusses the problem of using AMR for languages other
than English. Annotating new AMR datasets for other languages is an expensive
process and requires defining annotation guidelines for each new language.
It is therefore reasonable to ask whether we can share AMR annotations
across languages. We provided evidence that AMR datasets for English
can be successfully transferred to other languages: we trained parsers for Italian,
Spanish, German, and Chinese to investigate the cross-linguality of AMR.
We showed cases where translational divergences between languages pose a
problem and cases where they do not. In summary, this thesis demonstrates
the impact of reentrancies in AMR as well as providing insights on AMR for
languages that do not yet have AMR datasets