Measuring post-editing time and effort for different types of machine translation errors

Post-editing (PE) of machine translation (MT) is becoming more and more common in the professional translation setting. However, many users refuse to employ MT due to bad quality of the output it provides and even reject post-editing job offers. This can change by improving MT quality from the point of view of the PE process. This article investigates different types of MT errors and the difficulties they pose for PE in terms of post-editing time and technical effort. For the experiment we used English to German translations performed by MT engines. The errors were previously annotated using the MQM scheme for error annotation. The sentences were post-edited by students in translation. The experiment allowed us to make observations about the relation between technical and temporal PE effort, as well as to discover the types of errors that are more challenging for PE

Traducción literal vs. traducción por defecto. Un desafío para los dos constructos mediante la traducción del egipcio clásico como caso extremo

This paper presents the results of a study that compares the constructs of literal translation (Schaeffer & Carl, 2014) and default translation (Halverson, 2019) by means of an observational, exploratory study with Middle Egyptian translation as an extreme case in point. Two MA students in Egyptology at Universitat Autònoma de Barcelona and three recent graduates of the same MA programme took part in the study. They translated two excerpts from two Middle Egyptian literary texts into Spanish. InputLog was used to collect translation-process data and derive word-level indicators of cognitive effort from them: typos per word, word typing speed, and within-word pause. Results showed a clear link between default translations and cognitive effort (low number of typos, low number of respites, and fast writing speed). However, the assumption that deviations from literality cause greater cognitive effort was not observed. Hence, default translation may serve as a more adequate construct to describe the regular way translators perform.En este artículo se exponen los hallazgos de un estudio observacional y exploratorio en el cual se cotejan los constructos de traducción literal, según Schaeffer y Carl (2014), y el de traducción por defecto, propuesto por Halverson (2019), mediante la traducción del egipcio clásico como un caso extremo. En la investigación participaron dos alumnos del Máster Universitario en Egiptología de la Universitat Autònoma de Barcelona junto con tres egresados del mismo programa. La tarea consistió en traducir al español dos fragmentos procedentes de dos textos literarios escritos en egipcio clásico. Se empleó InputLog como herramienta para recoger datos referentes al proceso de traducción y para derivar de ellos indicadores de esfuerzo cognitivo a nivel de palabra, tales como errores tipográficos por palabra, velocidad de tecleo y pausas intra-palabra. Los resultados evidenciaron una clara vinculación entre las traducciones por defecto y el esfuerzo cognitivo, manifestado por un bajo número de errores tipográficos, escasas pausas y una elevada velocidad de escritura. No obstante, el supuesto de que las desviaciones de la traducción literal conllevan un esfuerzo cognitivo mayor no pudo ser verificado. Así, la traducción por defecto podría presentarse como un constructo más apropiado para describir la forma habitual de traducir de los/las traductores/as

Why Translation Is Difficult: A Corpus-Based Study of Non-Literality in Post-Editing and From-Scratch Translation

The paper develops a definition of translation literality that is based on the syntactic and semantic similarity of the source and the target texts. We provide theoretical and empirical evidence that absolute literal translations are easy to produce. Based on a multilingual corpus of alternative translations we investigate the effects of cross-lingual syntactic and semantic distance on translation production times and find that non-literality makes from-scratch translation and post-editing difficult. We show that statistical machine translation systems encounter even more difficulties with non-literality

Syntactic difficulties in translation

Even though machine translation (MT) systems such as Google Translate and DeepL have improved significantly over the last years, a continuous rise in globalisation and linguistic diversity requires increasing amounts of professional, error-free translation. One can imagine, for instance, that mistakes in medical leaflets can lead to disastrous consequences. Less catastrophic, but equally significant, is the lack of a consistent and creative style of MT systems in literary genres. In such cases, a human translation is preferred. Translating a text is a complex procedure that involves a variety of mental processes such as understanding the original message and its context, finding a fitting translation, and verifying that the translation is grammatical, contextually sound, and generally adequate and acceptable. From an educational perspective, it would be helpful if the translation difficulty of a given text can be predicted, for instance to ensure that texts of objectively appropriate difficulty levels are used in exams and assignments for translators. Also in the translation industry it may prove useful, for example to direct more difficult texts to more experienced translators. During this PhD project, my coauthors and I investigated which linguistic properties contribute to such difficulties. Specifically, we put our attention to syntactic differences between a source text and its translation, that is to say their (dis)similarities in terms of linguistic structure. To this end we developed new measures that can quantify such differences and made the implementation publicly available for other researchers to use. These metrics include word (group) movement (how does the order in the original text differ from that in a given translation), changes in the linguistic properties of words, and a comparison of the underlying abstract structure of a sentence and a translation. Translation difficulty cannot be directly measured but process information can help. Particularly, keystroke logging and eye-tracking data can be recorded during translation and used as a proxy for the required cognitive effort. An example: the longer a translator looks at a word, the more time and effort they likely need to process it. We investigated the effect that specific measures of syntactic similarity have on these behavioural processing features to get an indication of what their effect is on the translation difficulty. In short: how does the syntactic (dis)similarity between a source text and a possible translation impact the translation difficulty? In our experiments, we show that different syntactic properties indeed have an effect, and that differences in syntax between a source text and its translation affect the cognitive effort required to translate that text. These effects are not identical between syntactic properties, though, suggesting that individual syntactic properties affect the translation process in different ways and that not all syntactic dissimilarities contribute to translation difficulty equally.De kwaliteit van machinevertaalsystemen (MT) zoals Google Translate en DeepL is de afgelopen jaren sterk verbeterd. Door alsmaar meer globalisering en taalkundige diversiteit is er echter meer dan ooit nood aan professionele vertalingen waar geen fouten in staan. In zekere communicatievormen zouden vertaalfouten namelijk tot desastreuse gevolgen kunnen leiden, bijvoorbeeld in medische bijsluiters. Ook in minder levensbedreigende situaties verkiezen we nog steeds menselijke vertalingen, bijvoorbeeld daar waar een creatieve en consistente stijl noodzakelijk is, zoals in boeken en poëzie. Een tekst vertalen is een complex karwei waarin verschillende mentale processen een rol spelen. Zo moet bijvoorbeeld de brontekst gelezen en begrepen worden, moet er naar een vertaling gezocht worden, en daarbovenop moet tijdens het vertaalproces de vertaling continu gecontroleerd worden om te zorgen dat het ten eerste een juiste vertaling is en ten tweede dat de tekst ook grammaticaal correct is in de doeltaal. Vanuit een pedagogisch standpunt zou het nuttig zijn om de vertaalmoeilijkheid van een tekst te voorspellen. Zo wordt ervoor gezorgd dat de taken en examens van vertaalstudenten tot een objectief bepaald moeilijkheidsniveau behoren. Ook in de vertaalindustrie zou zo’n systeem van toepassing zijn; moeilijkere teksten kunnen aan de meest ervaren vertalers worden bezorgd. Samen met mijn medeauteurs heb ik tijdens dit doctoraatsproject onderzocht welke eigenschappen van een tekst bijdragen tot vertaalmoeilijkheden. We legden daarbij de nadruk op taalkundige, structurele verschillen tussen de brontekst en diens vertaling, en ontwikkelden verscheidene metrieken om dit soort syntactische verschillen te kunnen meten. Zo kan bijvoorbeeld een verschillende woord(groep)volgorde worden gekwantificeerd, kunnen verschillen in taalkundige labels worden geteld, en kunnen de abstracte, onderliggende structuren van een bronzin en een vertaling vergeleken worden. We maakten de implementatie van deze metrieken openbaar beschikbaar. De vertaalmoeilijkheid van een tekst kan niet zomaar gemeten worden, maar door naar gedragsdata van een vertaler te kijken, krijgen we wel een goed idee van de moeilijkheden waarmee ze geconfronteerd werden. De bewegingen en focuspunten van de ogen van de vertaler en hun toetsaanslagen kunnen worden geregistreerd en nadien gebruikt in een experimentele analyse. Ze geven ons nuttig informatie en kunnen zelfs dienen als een benadering van de nodige inspanning die geleverd moest worden tijdens het vertaalproces. Daarmee leidt het ons ook naar de elementen (woorden, woordgroepen) waar de vertaler moeilijkheden mee had. Als een vertaler lang naar een woord kijkt, dan kunnen we aannemen dat de verwerking ervan veel inspanning vergt. We kunnen deze gedragsdata dus gebruiken als een maat voor moeilijkheid. In ons onderzoek waren we voornamelijk benieuwd naar het effect van syntactische verschillen tussen een bronzin en een doelzin op dit soort gedragsdata. Onze resultaten tonen aan dat de voorgestelde metrieken inderdaad een effect hebben en dat taalkundige verschillen tussen een bron- en doeltekst leiden tot een hogere cognitieve belasting tijdens het vertalen van een tekst. Deze effecten verschillen per metriek, wat duidt op het belang van (onderzoek naar) individuele syntactische metrieken; niet elke metriek draagt even veel bij aan vertaalmoeilijkheden

Comparing post-editing difficulty of different machine translation errors in Spanish and German translations from English

Post-editing (PE) of Machine Translation (MT) is an increasingly popular way to integrate MT in the professional translation workflow, as it increases productivity and income. However, the quality of MT is not always good enough to blindly choose PE over translation from scratch. This article studies the PE of different error types and compares indicators of PE difficulty in English-to-Spanish and English-to-German translations. The results show that the indicators in question 1) do not correlate between each other for all error types, and 2) differ between languages

Metalinguistic Knowledge/Awareness/Ability in Cognitive Translation Studies: Some Questions

 Throughout the history of contemporary Translation Studies, theoretical, empirical and pedagogically oriented work has made use of a range of notions that assume a translator’s metalinguistic knowledge, or knowledge about language, rather than knowledge of a language or languages. Examples include ideas such as ‘translation strategies’, translational ‘problem-solving’, ‘the monitor model’ and models of translator competence. Issues related to learning, automatization, and consciousness also figure in many of the discussions. At the same time, studies in bi- and multilingualism and second (and third) language acquisition have also developed a range of related ideas and concepts to deal with some of the same issues and concerns in bi- and multilingual language production more broadly (see e.g. Jessner 2006: 40-43). Some recent translation process studies have begun to target questions related to metalinguistic awareness (e.g. Ehrensberger-Dow/Künzli 2010, Ehrensberger-Dow/Perrin 2009) while the underlying assumptions of some of the commonly used ideas are also being questioned (e.g. Muñoz Martín 2016a). The range of available ideas, the significant differences between them, and the increasingly important role these ideas are playing in cognitive translation research mandate a critical look at this conceptual field. In this paper, I present some current views on metalinguistic knowledge/awareness/ability within the bi- and multilingualism and second language acquisition (SLA) literature, and compare these to some of the most widely used constructs in Cognitive Translation Studies (CTS). The aim is to clear the conceptual ground and to single out some of the most pressing questions to be addressed regarding this particular aspect of translational cognition. 

Measuring the difficulty of text translation: The combination of text-focused and translator-oriented approaches

This paper explores the impact of text complexity on translators’ subjective perception of translation difficulty and on their cognitive load. Twenty-six MA translation students from a UK university were asked to translate three English texts with different complexity into Chinese. Their eye movements were recorded by an eye-tracker, and their cognitive load was self-assessed with a Likert scale before translation and NASA-TLX scales after translation. The results show that: (i) the intrinsic complexity measured by readability, word frequency and non-literalness was in line with the results received from informants’ subjective assessment of translation difficulty; (ii) moderate and positive correlations existed between most items in the self-assessments and the indicator (fixation and saccade durations) obtained by the eye-tracking measurements; and (iii) the informants’ cognitive load as indicated by fixation and saccade durations (but not for pupil size) increased significantly in two of the three texts along with the increase in source text complexity

Chengyu in Chinese Language Teaching: A preliminary analysis of Italian learners’ data

Chengyu, also known as Chinese four-character idioms, are a type of traditional Chinese idiom, mostly consisting of four characters. They commonly derive from classic Chinese literary sources, including those of the three great philosophical and religious traditions that influenced the entire East Asia cultural sphere: Confucianism, Daoism and Buddhism. Chengyu, therefore, possess a wide range of cultural references, and, from Chinese, spread to the languages of the other countries of the sinosphere, such as Japan and Korea. Although many scholars have emphasized the importance of the acquisition of chengyu, not much attention has been paid to chengyu learning in Chinese Language Teaching research so far. As a preliminary attempt to address this gap, this paper reports the results of two small-scale, exploratory experiments, aimed at investigating Italian learners’ general knowledge of chengyu and their main interpretation strategies, as well as comparing the effectiveness of direct and indirect instruction in chengyu teaching. The experiments involved participants from Bachelor and Master programs of Roma Tre University. The results show a predominant effect of negative transfer from Italian, as well as a better performance of the participants who received indirect instruction