Creation of a New Domain and Evaluation of Comparison Generation in a Natural Language Generation System

We describe the creation of a new domain for the Methodius Natural Language Generation System, and an evaluation of Methodius ’ parameterized comparison generation algorithm. The new domain was based around music and performers, and texts about the domain were generated using Methodius. Our evaluation showed that test subjects learned more from texts that contained comparisons than from those that did not. We also established that the comparison generation algorithm could generalize to the music domain.

Evaluating evaluation methods for generation in the presence of variation

Abstract. Recent years have seen increasing interest in automatic metrics for the evaluation of generation systems. When a system can generate syntactic variation, automatic evaluation becomes more difficult. In this paper, we compare the performance of several automatic evaluation metrics using a corpus of automatically generated paraphrases. We show that these evaluation metrics can at least partially measure adequacy (similarity in meaning), but are not good measures of fluency (syntactic correctness). We make several proposals for improving the evaluation of generation systems that produce variation.