5 research outputs found
EQG-RACE: Examination-Type Question Generation
Question Generation (QG) is an essential component of the automatic
intelligent tutoring systems, which aims to generate high-quality questions for
facilitating the reading practice and assessments. However, existing QG
technologies encounter several key issues concerning the biased and unnatural
language sources of datasets which are mainly obtained from the Web (e.g.
SQuAD). In this paper, we propose an innovative Examination-type Question
Generation approach (EQG-RACE) to generate exam-like questions based on a
dataset extracted from RACE. Two main strategies are employed in EQG-RACE for
dealing with discrete answer information and reasoning among long contexts. A
Rough Answer and Key Sentence Tagging scheme is utilized to enhance the
representations of input. An Answer-guided Graph Convolutional Network (AG-GCN)
is designed to capture structure information in revealing the inter-sentences
and intra-sentence relations. Experimental results show a state-of-the-art
performance of EQG-RACE, which is apparently superior to the baselines. In
addition, our work has established a new QG prototype with a reshaped dataset
and QG method, which provides an important benchmark for related research in
future work. We will make our data and code publicly available for further
research.Comment: Accepted by AAAI-202
A Comparative Evaluation Methodology for NLG in Interactive Systems
Interactive systems have become an increasingly important type of application for deployment of NLG technology over recent years. At present, we do not yet have commonly agreed terminology or methodology for evaluating NLG within interactive systems. In this paper, we take steps towards addressing this gap by presenting a set of principles for designing new evaluations in our comparative evaluation methodology. We start with presenting a categorisation framework, giving an overview of different categories of evaluation measures, in order to provide standard terminology for categorising existing and new evaluation techniques. Background on existing evaluation methodologies for NLG and interactive systems is presented. The comparative evaluation methodology is presented. Finally, a methodology for comparative evaluation of NLG components embedded within interactive systems is presented in terms of the comparative evaluation methodology, using a specific task for illustrative purposes
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation
This paper surveys the current state of the art in Natural Language
Generation (NLG), defined as the task of generating text or speech from
non-linguistic input. A survey of NLG is timely in view of the changes that the
field has undergone over the past decade or so, especially in relation to new
(usually data-driven) methods, as well as new applications of NLG technology.
This survey therefore aims to (a) give an up-to-date synthesis of research on
the core tasks in NLG and the architectures adopted in which such tasks are
organised; (b) highlight a number of relatively recent research topics that
have arisen partly as a result of growing synergies between NLG and other areas
of artificial intelligence; (c) draw attention to the challenges in NLG
evaluation, relating them to similar challenges faced in other areas of Natural
Language Processing, with an emphasis on different evaluation methods and the
relationships between them.Comment: Published in Journal of AI Research (JAIR), volume 61, pp 75-170. 118
pages, 8 figures, 1 tabl
Question Generation Shared Task and Evaluation Challenge - Status report
The First Shared Task Evaluation Challenge on Question Generation took place in 2010 as part of the 3 rd workshop on Question Generation. The campaign included two tasks: Question Generation from Sentences and Question Generation from Paragraphs. This status report briefly summarizes the motivation, tasks and results. Lessons learned relevant to future QG-STECs are also offered. © 2011 Association for Computational Linguistics