1,591 research outputs found
Few-Shot NLG with Pre-Trained Language Model
Neural-based end-to-end approaches to natural language generation (NLG) from
structured data or knowledge are data-hungry, making their adoption for
real-world applications difficult with limited data. In this work, we propose
the new task of \textit{few-shot natural language generation}. Motivated by how
humans tend to summarize tabular data, we propose a simple yet effective
approach and show that it not only demonstrates strong performance but also
provides good generalization across domains. The design of the model
architecture is based on two aspects: content selection from input data and
language modeling to compose coherent sentences, which can be acquired from
prior knowledge. With just 200 training examples, across multiple domains, we
show that our approach achieves very reasonable performances and outperforms
the strongest baseline by an average of over 8.0 BLEU points improvement. Our
code and data can be found at \url{https://github.com/czyssrs/Few-Shot-NLG}Comment: ACL 202
Language Models as Few-Shot Learner for Task-Oriented Dialogue Systems
Task-oriented dialogue systems use four connected modules, namely, Natural
Language Understanding (NLU), a Dialogue State Tracking (DST), Dialogue Policy
(DP) and Natural Language Generation (NLG). A research challenge is to learn
each module with the least amount of samples (i.e., few-shots) given the high
cost related to the data collection. The most common and effective technique to
solve this problem is transfer learning, where large language models, either
pre-trained on text or task-specific data, are fine-tuned on the few samples.
These methods require fine-tuning steps and a set of parameters for each task.
Differently, language models, such as GPT-2 (Radford et al., 2019) and GPT-3
(Brown et al., 2020), allow few-shot learning by priming the model with few
examples. In this paper, we evaluate the priming few-shot ability of language
models in the NLU, DST, DP and NLG tasks. Importantly, we highlight the current
limitations of this approach, and we discuss the possible implication for
future work.Comment: Blog (https://andreamad8.github.io/few-shot-gpt/), Medium
(https://medium.com/@madottoandrea/language-model-as-few-shot-learner-for-task-oriented-dialogue-systems-db4765796744)
and Code (https://github.com/andreamad8/TASK-ORIENTED-LM-FEWSHOT
Few-shot Natural Language Generation for Task-Oriented Dialog
As a crucial component in task-oriented dialog systems, the Natural Language
Generation (NLG) module converts a dialog act represented in a semantic form
into a response in natural language. The success of traditional template-based
or statistical models typically relies on heavily annotated data, which is
infeasible for new domains. Therefore, it is pivotal for an NLG system to
generalize well with limited labelled data in real applications. To this end,
we present FewShotWoz, the first NLG benchmark to simulate the few-shot
learning setting in task-oriented dialog systems. Further, we develop the
SC-GPT model. It is pre-trained on a large set of annotated NLG corpus to
acquire the controllable generation ability, and fine-tuned with only a few
domain-specific labels to adapt to new domains. Experiments on FewShotWoz and
the large Multi-Domain-WOZ datasets show that the proposed SC-GPT significantly
outperforms existing methods, measured by various automatic metrics and human
evaluations.Comment: Project website: https://aka.ms/scgpt ; Code and data:
https://github.com/pengbaolin/SC-GP
Template Guided Text Generation for Task-Oriented Dialogue
Virtual assistants such as Google Assistant, Amazon Alexa, and Apple Siri
enable users to interact with a large number of services and APIs on the web
using natural language. In this work, we investigate two methods for Natural
Language Generation (NLG) using a single domain-independent model across a
large number of APIs. First, we propose a schema-guided approach which
conditions the generation on a schema describing the API in natural language.
Our second method investigates the use of a small number of templates, growing
linearly in number of slots, to convey the semantics of the API. To generate
utterances for an arbitrary slot combination, a few simple templates are first
concatenated to give a semantically correct, but possibly incoherent and
ungrammatical utterance. A pre-trained language model is subsequently employed
to rewrite it into coherent, natural sounding text. Through automatic metrics
and human evaluation, we show that our method improves over strong baselines,
is robust to out-of-domain inputs and shows improved sample efficiency
Logic2Text: High-Fidelity Natural Language Generation from Logical Forms
Previous works on Natural Language Generation (NLG) from structured data have
primarily focused on surface-level descriptions of record sequences. However,
for complex structured data, e.g., multi-row tables, it is often desirable for
an NLG system to describe interesting facts from logical inferences across
records. If only provided with the table, it is hard for existing models to
produce controllable and high-fidelity logical generations. In this work, we
formulate logical level NLG as generation from logical forms in order to obtain
controllable, high-fidelity, and faithful generations. We present a new
large-scale dataset, \textsc{Logic2Text}, with 10,753 descriptions involving
common logic types paired with the underlying logical forms. The logical forms
show diversified graph structure of free schema, which poses great challenges
on the model's ability to understand the semantics. We experiment on (1)
Fully-supervised training with the full datasets, and (2) Few-shot setting,
provided with hundreds of paired examples; We compare several popular
generation models and analyze their performances. We hope our dataset can
encourage research towards building an advanced NLG system capable of natural,
faithful, and human-like generation. The dataset and code are available at
https://github.com/czyssrs/Logic2Text.Comment: Findings of EMNLP 2020, 9 pages, 6 figure
Meta-Learning for Low-resource Natural Language Generation in Task-oriented Dialogue Systems
Natural language generation (NLG) is an essential component of task-oriented
dialogue systems. Despite the recent success of neural approaches for NLG, they
are typically developed for particular domains with rich annotated training
examples. In this paper, we study NLG in a low-resource setting to generate
sentences in new scenarios with handful training examples. We formulate the
problem from a meta-learning perspective, and propose a generalized
optimization-based approach (Meta-NLG) based on the well-recognized
model-agnostic meta-learning (MAML) algorithm. Meta-NLG defines a set of meta
tasks, and directly incorporates the objective of adapting to new low-resource
NLG tasks into the meta-learning optimization process. Extensive experiments
are conducted on a large multi-domain dataset (MultiWoz) with diverse
linguistic variations. We show that Meta-NLG significantly outperforms other
training procedures in various low-resource configurations. We analyze the
results, and demonstrate that Meta-NLG adapts extremely fast and well to
low-resource situations.Comment: Accepted as a full paper at IJCAI 201
SOLOIST: Building Task Bots at Scale with Transfer Learning and Machine Teaching
We present a new method SOLOIST that uses transfer learning and machine
teaching to build task bots at scale. We parameterize classical modular
task-oriented dialog systems using a Transformer-based auto-regressive language
model, which subsumes different dialog modules into a single neural model. We
pre-train, on heterogeneous dialog corpora, a task-grounded response generation
model, which can generate dialog responses grounded in user goals and
real-world knowledge for task completion. The pre-trained model can be
efficiently adapted to accomplish new tasks with a handful of task-specific
dialogs via machine teaching, where training samples are generated by human
teachers interacting with the system. Experiments show that (i) SOLOIST creates
new state-of-the-art on well-studied task-oriented dialog benchmarks, including
CamRest676 and MultiWOZ; (ii) in the few-shot fine-tuning settings, SOLOIST
significantly outperforms existing methods, and (iii) the use of machine
teaching substantially reduces the labeling cost of fine-tuning. The
pre-trained models and codes are available at https://aka.ms/soloist.Comment: 18 pages; To appear at TACL; Project Website: https://aka.ms/solois
Data Augmentation for Spoken Language Understanding via Pretrained Models
The training of spoken language understanding (SLU) models often faces the
problem of data scarcity. In this paper, we put forward a data augmentation
method with pretrained language models to boost the variability and accuracy of
generated utterances. Furthermore, we investigate and propose solutions to two
previously overlooked scenarios of data scarcity in SLU: i) Rich-in-Ontology:
ontology information with numerous valid dialogue acts are given; ii)
Rich-in-Utterance: a large number of unlabelled utterances are available.
Empirical results show that our method can produce synthetic training data that
boosts the performance of language understanding models in various scenarios.Comment: 6 pages, 1 figur
Zero-Shot Dialog Generation with Cross-Domain Latent Actions
This paper introduces zero-shot dialog generation (ZSDG), as a step towards
neural dialog systems that can instantly generalize to new situations with
minimal data. ZSDG enables an end-to-end generative dialog system to generalize
to a new domain for which only a domain description is provided and no training
dialogs are available. Then a novel learning framework, Action Matching, is
proposed. This algorithm can learn a cross-domain embedding space that models
the semantics of dialog responses which, in turn, lets a neural dialog
generation model generalize to new domains. We evaluate our methods on a new
synthetic dialog dataset, and an existing human-human dialog dataset. Results
show that our method has superior performance in learning dialog models that
rapidly adapt their behavior to new domains and suggests promising future
research.Comment: Accepted as a long paper in SIGDIAL 201
Schema-Guided Natural Language Generation
Neural network based approaches to data-to-text natural language generation
(NLG) have gained popularity in recent years, with the goal of generating a
natural language prompt that accurately realizes an input meaning
representation. To facilitate the training of neural network models,
researchers created large datasets of paired utterances and their meaning
representations. However, the creation of such datasets is an arduous task and
they mostly consist of simple meaning representations composed of slot and
value tokens to be realized. These representations do not include any
contextual information that an NLG system can use when trying to generalize,
such as domain information and descriptions of slots and values. In this paper,
we present the novel task of Schema-Guided Natural Language Generation
(SG-NLG). Here, the goal is still to generate a natural language prompt, but in
SG-NLG, the input MRs are paired with rich schemata providing contextual
information. To generate a dataset for SG-NLG we re-purpose an existing dataset
for another task: dialog state tracking, which includes a large and rich schema
spanning multiple different attributes, including information about the domain,
user intent, and slot descriptions. We train different state-of-the-art models
for neural natural language generation on this dataset and show that in many
cases, including rich schema information allows our models to produce higher
quality outputs both in terms of semantics and diversity. We also conduct
experiments comparing model performance on seen versus unseen domains, and
present a human evaluation demonstrating high ratings for overall output
quality.Comment: Accepted as a long paper at INLG 202
- …