The attribute selection for GRE challenge : overview and evaluation results

The Attribute Selection for Generating Referring Expressions (ASGRE) Challenge was the first shared-task evaluation challenge in the field of Natural Language Generation. Six teams submitted a total of 22 systems. All submitted systems were tested automatically for minimality, uniqueness and ‘humanlikeness’. In addition, the output of 15 systems was tested in a task-based experiment where subjects were asked to identify referents, and the speed and accuracy of identification was measured. This report describes the ASGRE task and the five evaluation methods, gives brief overviews of the participating systems, and presents the evaluation results.peer-reviewe

Content determination in GRE : evaluating the evaluator

In this paper, we discuss the evaluation mea- sures proposed in a number of recent papers as- sociated with the TUNA project1, and which have become an important component of the First NLG Shared Task and Evaluation Cam- paign (STEC) on attribute selection for referring expressions generation. Focusing on reference to individual objects, we discuss what such evalu- ation measures should be expected to achieve, and what alternative measures merit considera- tion.peer-reviewe

A Shared Task on Bandit Learning for Machine Translation

We introduce and describe the results of a novel shared task on bandit learning for machine translation. The task was organized jointly by Amazon and Heidelberg University for the first time at the Second Conference on Machine Translation (WMT 2017). The goal of the task is to encourage research on learning machine translation from weak user feedback instead of human references or post-edits. On each of a sequence of rounds, a machine translation system is required to propose a translation for an input, and receives a real-valued estimate of the quality of the proposed translation for learning. This paper describes the shared task's learning and evaluation setup, using services hosted on Amazon Web Services (AWS), the data and evaluation metrics, and the results of various machine translation architectures and learning protocols.Comment: Conference on Machine Translation (WMT) 201

The ethics of machine translation

In this paper I first describe the two main branches in machine translation research. I then go to discuss why the second of these, statistical machine translation, can cause some malaise among translation scholars. As some of the issues that arise are ethical in nature, I stop to ponder what an ethics of machine translation might involve, before considering the ethical stance adopted by some of the main protagonists in the development and popularisation of statistical machine translation, and in the teaching of translation