Towards predicting post-editing productivity

Machine translation (MT) quality is generally measured via automatic metrics, producing scores that have no meaning for translators who are required to post-edit MT output or for project managers who have to plan and budget for transla- tion projects. This paper investigates correlations between two such automatic metrics (general text matcher and translation edit rate) and post-editing productivity. For the purposes of this paper, productivity is measured via processing speed and cognitive measures of effort using eye tracking as a tool. Processing speed, average fixation time and count are found to correlate well with the scores for groups of segments. Segments with high GTM and TER scores require substantially less time and cognitive effort than medium or low-scoring segments. Future research involving score thresholds and confidence estimation is suggested

Identifying the machine translation error types with the greatest impact on post-editing effort

Translation Environment Tools make translators' work easier by providing them with term lists, translation memories and machine translation output. Ideally, such tools automatically predict whether it is more effortful to post-edit than to translate from scratch, and determine whether or not to provide translators with machine translation output. Current machine translation quality estimation systems heavily rely on automatic metrics, even though they do not accurately capture actual post-editing effort. In addition, these systems do not take translator experience into account, even though novices' translation processes are different from those of professional translators. In this paper, we report on the impact of machine translation errors on various types of post-editing effort indicators, for professional translators as well as student translators. We compare the impact of MT quality on a product effort indicator (HTER) with that on various process effort indicators. The translation and post-editing process of student translators and professional translators was logged with a combination of keystroke logging and eye-tracking, and the MT output was analyzed with a fine-grained translation quality assessment approach. We find that most post-editing effort indicators (product as well as process) are influenced by machine translation quality, but that different error types affect different post-editing effort indicators, confirming that a more fine-grained MT quality analysis is needed to correctly estimate actual post-editing effort. Coherence, meaning shifts, and structural issues are shown to be good indicators of post-editing effort. The additional impact of experience on these interactions between MT quality and post-editing effort is smaller than expected

Systematic evaluation of design choices for software development tools

[Abstract]: Most design and evaluation of software tools is based on the intuition and experience of the designers. Software tool designers consider themselves typical users of the tools that they build and tend to subjectively evaluate their products rather than objectively evaluate them using established usability methods. This subjective approach is inadequate if the quality of software tools is to improve and the use of more systematic methods is advocated. This paper summarises a sequence of studies that show how user interface design choices for software development tools can be evaluated using established usability engineering techniques. The techniques used included guideline review, predictive modelling and experimental studies with users

The impact of machine translation error types on post-editing effort indicators

In this paper, we report on a post-editing study for general text types from English into Dutch conducted with master's students of translation. We used a fine-grained machine translation (MT) quality assessment method with error weights that correspond to severity levels and are related to cognitive load. Linear mixed effects models are applied to analyze the impact of MT quality on potential post-editing effort indicators. The impact of MT quality is evaluated on three different levels, each with an increasing granularity. We find that MT quality is a significant predictor of all different types of post-editing effort indicators and that different types of MT errors predict different post-editing effort indicators

Taking statistical machine translation to the student translator

Despite the growth of statistical machine translation (SMT) research and development in recent years, it remains somewhat out of reach for the translation community where programming expertise and knowledge of statistics tend not to be commonplace. While the concept of SMT is relatively straightforward, its implementation in functioning systems remains difficult for most, regardless of expertise. More recently, however, developments such as SmartMATE have emerged which aim to assist users in creating their own customized SMT systems and thus reduce the learning curve associated with SMT. In addition to commercial uses, translator training stands to benefit from such increased levels of inclusion and access to state-of-the-art approaches to MT. In this paper we draw on experience in developing and evaluating a new syllabus in SMT for a cohort of post-graduate student translators: we identify several issues encountered in the introduction of student translators to SMT, and report on data derived from repeated measures questionnaires that aim to capture data on students’ self-efficacy in the use of SMT. Overall, results show that participants report significant increases in their levels of confidence and knowledge of MT in general, and of SMT in particular. Additional benefits – such as increased technical competence and confidence – and future refinements are also discussed

Measuring post-editing time and effort for different types of machine translation errors

Post-editing (PE) of machine translation (MT) is becoming more and more common in the professional translation setting. However, many users refuse to employ MT due to bad quality of the output it provides and even reject post-editing job offers. This can change by improving MT quality from the point of view of the PE process. This article investigates different types of MT errors and the difficulties they pose for PE in terms of post-editing time and technical effort. For the experiment we used English to German translations performed by MT engines. The errors were previously annotated using the MQM scheme for error annotation. The sentences were post-edited by students in translation. The experiment allowed us to make observations about the relation between technical and temporal PE effort, as well as to discover the types of errors that are more challenging for PE

Skills and Profile of the New Role of the Translator as MT Post-editor

This paper explores the skills and profile of the new role of the translator as MT post-editor in view of the rising interest and use of MT in the translation industry. After a brief review of the relevant literature declaring post-editing (PE) as a profession on its own, the paper goes on to identify the different tasks involved in PE processes, following the work of Krings (Krings, 2001). Then, a series of competences are defined and grouped into three main categories: core competences, linguistic skills and instrumental competences. Finally, a description of the controlled translation scenario of MT PE is advanced taking into account the overall scenario of any translation project, including client description, text domain, text description, use of glossaries, MT engine, MT output quality and purpose of the translated text.Aquest article aborda les habilitats i les característiques del perfil del nou rol del traductor com a posteditor de traducció automàtica, tot i tenint en compte l'augment de l'interès en i l'ús de la traducció automàtica per part de la industria de la traducció. Després d'una breu revisió de la literatura més rellevant sobre postedició (PE) en tant que professió per ella mateixa, l'article identifica les diferents tasques implicades en els processos de PE, segons la proposta de Krings (2001). A continuació es defineix una sèrie de competències que s'agrupen en tres categories principals: competències nuclears, habilitats lingüístiques i competències instrumentals. Finalment el artículo proposa una descripció de l'escenari de traducció controlada propi de la PE de traducció automàtica, sense perdre de vista l'escenari general de qualsevol projecte de traducció, que inclou la descripció del client, el domini del text, la descripció del text, l'ús de glossaris, el motor de traducció automàtica, la qualitat de la traducció automàtica resultant i el propòsit del text traduït.Este artículo aborda las habilidades y las características del perfil del nuevo rol del traductor como poseditor de traducción automática, a la luz del aumento del interés en y del uso de la traducción automática por parte de la industria de la traducción. Después de una breve revisión de la literatura más relevante sobre posedición (PE) en tanto que profesión por sí misma, en el artículo se identifican las diferentes tareas implicadas en los procesos de PE, según la propuesta de Krings (2001). A continuación se define una serie de competencias que se agrupan en tres categorías principales: competencias nucleares, habilidades lingüísticas y competencias instrumentales. Finalmente el artículo propone una descripción del escenario de traducción controlada propio de la PE de traducción automática, sin perder de vista el marco general de cualquier proyecto de traducción, que incluye la descripción del cliente, el dominio del texto, la descripción del texto, el uso de glosarios, el motor de traducción automática, la calidad de la traducción automática resultante y el propósito del texto traducido

Assessing the usability of raw machine translation output: A user-centered study using eye tracking

This paper reports on the results of a project that aimed to investigate the usability of raw machine translated technical support documentation for a commercial online file storage service. Adopting a user-centred approach, we utilize the ISO/TR 16982 definition of usability - goal completion, satisfaction, effectiveness, and efficiency – and apply eye-tracking measures shown to be reliable indicators of cognitive effort, along with a post-task questionnaire. We investigated these measures for the original user documentation written in English and in four target languages: Spanish, French, German and Japanese, all of which were translated using a freely available online statistical machine translation engine. Using native speakers for each language, we found several significant differences between the source and MT output, a finding that indicates a difference in usability between well-formed content and raw machine translated content. One target language in particular, Japanese, was found to have a considerably lower usability level when compared with the original English

Comparing post-editing difficulty of different machine translation errors in Spanish and German translations from English

Post-editing (PE) of Machine Translation (MT) is an increasingly popular way to integrate MT in the professional translation workflow, as it increases productivity and income. However, the quality of MT is not always good enough to blindly choose PE over translation from scratch. This article studies the PE of different error types and compares indicators of PE difficulty in English-to-Spanish and English-to-German translations. The results show that the indicators in question 1) do not correlate between each other for all error types, and 2) differ between languages