5,422 research outputs found
FlowEval: A Consensus-Based Dialogue Evaluation Framework Using Segment Act Flows
Despite recent progress in open-domain dialogue evaluation, how to develop
automatic metrics remains an open problem. We explore the potential of dialogue
evaluation featuring dialog act information, which was hardly explicitly
modeled in previous methods. However, defined at the utterance level in
general, dialog act is of coarse granularity, as an utterance can contain
multiple segments possessing different functions. Hence, we propose segment
act, an extension of dialog act from utterance level to segment level, and
crowdsource a large-scale dataset for it. To utilize segment act flows,
sequences of segment acts, for evaluation, we develop the first consensus-based
dialogue evaluation framework, FlowEval. This framework provides a
reference-free approach for dialog evaluation by finding pseudo-references.
Extensive experiments against strong baselines on three benchmark datasets
demonstrate the effectiveness and other desirable characteristics of our
FlowEval, pointing out a potential path for better dialogue evaluation.Comment: EMNLP 2022 camera-ready versio
Turning Flowchart into Dialog: Plan-based Data Augmentation for Low-Resource Flowchart-grounded Troubleshooting Dialogs
Flowchart-grounded troubleshooting dialogue (FTD) systems, which follow the
instructions of a flowchart to diagnose users' problems in specific domains
(eg., vehicle, laptop), have been gaining research interest in recent years.
However, collecting sufficient dialogues that are naturally grounded on
flowcharts is costly, thus FTD systems are impeded by scarce training data. To
mitigate the data sparsity issue, we propose a plan-based data augmentation
(PlanDA) approach that generates diverse synthetic dialog data at scale by
transforming concise flowchart into dialogues. Specifically, its generative
model employs a variational-base framework with a hierarchical planning
strategy that includes global and local latent planning variables. Experiments
on the FloDial dataset show that synthetic dialogue produced by PlanDA improves
the performance of downstream tasks, including flowchart path retrieval and
response generation, in particular on the Out-of-Flowchart settings. In
addition, further analysis demonstrate the quality of synthetic data generated
by PlanDA in paths that are covered by current sample dialogues and paths that
are not covered
A Survey of Available Corpora For Building Data-Driven Dialogue Systems: The Journal Version
During the past decade, several areas of speech and language understanding have witnessed substantial breakthroughs from the use of data-driven models. In the area of dialogue systems, the trend is less obvious, and most practical systems are still built through significant engineering and expert knowledge. Nevertheless, several recent results suggest that data-driven approaches are feasible and quite promising. To facilitate research in this area, we have carried out a wide survey of publicly available datasets suitable for data-driven learning of dialogue systems. We discuss important characteristics of these datasets, how they can be used to learn diverse dialogue strategies, and their other potential uses. We also examine methods for transfer learning between datasets and the use of external knowledge. Finally, we discuss appropriate choice of evaluation metrics for the learning objective
- …