61,854 research outputs found
The attribute selection for GRE challenge : overview and evaluation results
The Attribute Selection for Generating Referring
Expressions (ASGRE) Challenge was
the first shared-task evaluation challenge in
the field of Natural Language Generation.
Six teams submitted a total of 22 systems.
All submitted systems were tested automatically
for minimality, uniqueness and ‘humanlikeness’.
In addition, the output of 15
systems was tested in a task-based experiment
where subjects were asked to identify
referents, and the speed and accuracy of
identification was measured. This report describes
the ASGRE task and the five evaluation
methods, gives brief overviews of the
participating systems, and presents the evaluation
results.peer-reviewe
Content determination in GRE : evaluating the evaluator
In this paper, we discuss the evaluation mea- sures proposed in a number of recent papers as- sociated with the TUNA project1, and which have become an important component of the First NLG Shared Task and Evaluation Cam- paign (STEC) on attribute selection for referring expressions generation. Focusing on reference to individual objects, we discuss what such evalu- ation measures should be expected to achieve, and what alternative measures merit considera- tion.peer-reviewe
A Shared Task on Bandit Learning for Machine Translation
We introduce and describe the results of a novel shared task on bandit
learning for machine translation. The task was organized jointly by Amazon and
Heidelberg University for the first time at the Second Conference on Machine
Translation (WMT 2017). The goal of the task is to encourage research on
learning machine translation from weak user feedback instead of human
references or post-edits. On each of a sequence of rounds, a machine
translation system is required to propose a translation for an input, and
receives a real-valued estimate of the quality of the proposed translation for
learning. This paper describes the shared task's learning and evaluation setup,
using services hosted on Amazon Web Services (AWS), the data and evaluation
metrics, and the results of various machine translation architectures and
learning protocols.Comment: Conference on Machine Translation (WMT) 201
- …