A Gold Standard Methodology for Evaluating Accuracy in Data-To-Text Systems

Abstract

Acknowledgements: Many thanks to the Mechanical Turk annotators who participated in our experiment, and also to David Reiter, Tim Daniels, Rodrigo de Oliveira, and Andrew Smith for serving as pilot annotators when we were developing the methodology described in this paper. We would also like to thank Moray Greig for being our basketball domain expert during development. We are also grateful for the very helpful comments on this paper from the anonymous reviewers, the Aberdeen CLAN group, David Howcroft, Clement Rebuffel, and Chris van ´ der Lee. We would also like to thank Sam Wiseman, Ratish Puduppully, and Clement Rebuffel for pro- viding the generated texts from their respective systems. The work presented here is partially funded by the Engineering and Physical Sciences Research Council (EPSRC), which funds Craig Thomson under a National Productivity Investment Fund Doctoral Studentship (EP/R512412/1).Peer reviewedPublisher PD

    Similar works