97 research outputs found
Overview of the IWSLT 2012 Evaluation Campaign
open5siWe report on the ninth evaluation campaign organized by
the IWSLT workshop. This year, the evaluation offered multiple
tracks on lecture translation based on the TED corpus, and
one track on dialog translation from Chinese
to English based on the Olympic trilingual corpus.
In particular, the TED tracks included a speech transcription
track in English, a speech translation track from English to French,
and text translation tracks from English to French and from Arabic
to English. In addition to the official tracks, ten unofficial
MT tracks were offered that required translating TED talks into English
from either Chinese, Dutch, German, Polish, Portuguese (Brazilian), Romanian, Russian, Slovak,
Slovene, or Turkish.
16 teams participated in the evaluation and submitted a total of 48 primary runs.
All runs were evaluated with objective metrics, while runs of the official translation
tracks were also ranked by crowd-sourced judges.
In particular, subjective ranking for the TED task was performed on a progress test which permitted
direct comparison of the results from this year against the best results from the 2011 round of the evaluation campaign.Marcello Federico; Mauro Cettolo; Luisa Bentivogli; Michael Paul; Sebastian StükerFederico, Marcello; Cettolo, Mauro; Bentivogli, Luisa; Michael, Paul; Sebastian, Stüke
Domain-independent Punctuation and Segmentation Insertion
Punctuation and segmentation is crucial in spoken language translation, as it has a strong impact to translation performance. However, the impact of rare or unknown words in the performance of punctuation and segmentation insertion has not been thoroughly studied. In this work, we simulate various degrees of domain-match in testing scenario and investigate their impact to the punctuation insertion task. We explore three rare word generalizing schemes using part-of-speech (POS) tokens. Experiments show that generalizing rare and unknown words greatly improves the punctuation insertion performance, reaching up to 8.8 points of improvement in F-score when applied to the out-of-domain test scenario. We show that this improvement in punctuation quality has a positive impact on a following machine translation (MT) performance, improving it by 2 BLEU points
Overview of the IWSLT 2017 Evaluation Campaign
The IWSLT 2017 evaluation campaign has organised three tasks. The Multilingual task, which is about training machine translation systems handling many-to-many language directions, including so-called zero-shot directions. The Dialogue task, which calls for the integration of context information in machine translation, in order to resolve anaphoric references that typically occur in human-human dialogue turns. And, finally, the Lecture task, which offers the challenge of automatically transcribing and translating real-life university lectures. Following the tradition of these reports, we will described all tasks in detail and present the results of all runs submitted by their participants
English → Russian MT evaluation campaign
This paper presents the settings and the result of the ROMIP 2013 MT shared task for the English→Russian language direction. The quality of generated translations was assessed using automatic metrics and human evaluation. We also discuss ways to reduce human evaluation efforts using pairwise sentence comparisons by human judges to simulate sort operations
- …