Quantifying the effect of machine translation in a high-quality human translation production process

This paper studies the impact of machine translation (MT) on the translation workflow at the Directorate-General for Translation (DGT), focusing on two language pairs and two MT paradigms: English-into-French with statistical MT and English-into-Finnish with neural MT. We collected data from 20 professional translators at DGT while they carried out real translation tasks in normal working conditions. The participants enabled/disabled MT for half of the segments in each document. They filled in a survey at the end of the logging period. We measured the productivity gains (or losses) resulting from the use of MT and examined the relationship between technical effort and temporal effort. The results show that while the usage of MT leads to productivity gains on average, this is not the case for all translators. Moreover, the two technical effort indicators used in this study show weak correlations with post-editing time. The translators' perception of their speed gains was more or less in line with the actual results. Reduction of typing effort is the most frequently mentioned reason why participants preferred working with MT, but also the psychological benefits of not having to start from scratch were often mentioned

Towards Predicting Post-editing Effort with Source Text Readability

This paper investigates the impact of source text readability on the effort of post-editing English-Chinese Neural Machine Translation (NMT) output. Six readability formulas, including both traditional and newer ones, were employed to measure readability, and their predictive power towards post-editing effort was evaluated. Keystroke logging, self-report questionnaires, and retrospective protocols were applied to collect the data of post-editing for general text type from thirty-four student translators. The results reveal that: 1) readability has a significant yet weak effect on cognitive effort, while its impact on temporal and technical effort is less pronounced; 2) high NMT quality may alleviate the effect of readability; 3) readability formulas have the ability to predict post-editing effort to a certain extent, and newer formulas such as the Crowdsourced Algorithm of Reading Comprehension (CAREC) outperformed traditional formulas in most cases. Apart from readability formulas, the study shows that some fine-grained reading-related linguistic features are good predictors of post-editing time. Finally, this paper provides implications for automatic effort estimation in the translation industry

Exploring metrics for post-editing effort: and their ability to detect errors in machine translated output

As more companies integrate machine translation (MT) systems into their translation workflows, it becomes increasingly relevant to accurately measure post-editing (PE) effort. In this paper we explore how different types of errors in the MT output may affect PE effort, and take a closer look at the techniques used to measure it. For our experiment we curated a test suite of 60 EN > ES sentence pairs controlling certain features (sentence length, error frequency, topic, etc.) and had a group of 7 translators post-edit them using the PET tool, which helped collect temporal, technical and cognitive effort metrics. The results seem to challenge some previous error difficulty rankings; they also imply that, once other sentence features are controlled, the type of error to be addressed might not be as influential on effort as previously assumed. The low correlation values between the metrics for the different effort aspects may indicate that they do not reliably account for the full PE effort if not used in combination of one another

iMind: Uma ferramenta inteligente para suporte de compreensão de conteúdo

Usually while reading, content comprehension difficulty affects individual performance. Comprehension difficulties, e. g., could lead to a slow learning process, lower work quality, and inefficient decision-making. This thesis introduces an intelligent tool called “iMind” which uses wearable devices (e.g., smartwatches) to evaluate user comprehension difficulties and engagement levels while reading digital content. Comprehension difficulty can occur when there are not enough mental resources available for mental processing. The mental resource for mental processing is the cognitive load (CL). Fluctuations of CL lead to physiological manifestation of the autonomic nervous system (ANS), which can be measured by wearables, like smartwatches. ANS manifestations are, e. g., an increase in heart rate. With low-cost eye trackers, it is possible to correlate content regions to the measurements of ANS manifestation. In this sense, iMind uses a smartwatch and an eye tracker to identify comprehension difficulty at content regions level (where the user is looking). The tool uses machine learning techniques to classify content regions as difficult or non-difficult based on biometric and non-biometric features. The tool classified regions with a 75% accuracy and 80% f-score with Linear regression (LR). With the classified regions, it will be possible, in the future, to create contextual support for the reader in real-time by, e.g., translating the sentences that induced comprehension difficulty.Normalmente durante a leitura, a dificuldade de compreensão pode afetar o desempenho da leitura. A dificuldade de compreensão pode levar a um processo de aprendizagem mais lento, menor qualidade de trabalho ou uma ineficiente tomada de decisão. Esta tese apresenta uma ferramenta inteligente chamada “iMind” que usa dispositivos vestíveis (por exemplo, smartwatches) para avaliar a dificuldade de compreensão do utilizador durante a leitura de conteúdo digital. A dificuldade de compreensão pode ocorrer quando não há recursos mentais disponíveis suficientes para o processamento mental. O recurso usado para o processamento mental é a carga cognitiva (CL). As flutuações de CL levam a manifestações fisiológicas do sistema nervoso autônomo (ANS), manifestações essas, que pode ser medido por dispositivos vestíveis, como smartwatches. As manifestações do ANS são, por exemplo, um aumento da frequência cardíaca. Com eye trackers de baixo custo, é possível correlacionar manifestação do ANS com regiões do texto, por exemplo. Neste sentido, a ferramenta iMind utiliza um smartwatch e um eye tracker para identificar dificuldades de compreensão em regiões de conteúdo (para onde o utilizador está a olhar). Adicionalmente a ferramenta usa técnicas de machine learning para classificar regiões de conteúdo como difíceis ou não difíceis com base em features biométricos e não biométricos. A ferramenta classificou regiões com uma precisão de 75% e f-score de 80% usando regressão linear (LR). Com a classificação das regiões em tempo real, será possível, no futuro, criar suporte contextual para o leitor em tempo real onde, por exemplo, as frases que induzem dificuldade de compreensão são traduzidas

Exploring metrics for post-editing effort: and their ability to detect errors in machine translated output

As more companies integrate machine translation (MT) systems into their translation workflows, it becomes increasingly relevant to accurately measure post-editing (PE) effort. In this paper we explore how different types of errors in the MT output may affect PE effort, and take a closer look at the techniques used to measure it. For our experiment we curated a test suite of 60 EN > ES sentence pairs controlling certain features (sentence length, error frequency, topic, etc.) and had a group of 7 translators post-edit them using the PET tool, which helped collect temporal, technical and cognitive effort metrics. The results seem to challenge some previous error difficulty rankings; they also imply that, once other sentence features are controlled, the type of error to be addressed might not be as influential on effort as previously assumed. The low correlation values between the metrics for the different effort aspects may indicate that they do not reliably account for the full PE effort if not used in combination of one another

Translation, interpreting, cognition

Cognitive aspects of the translation process have become central in Translation and Interpreting Studies in recent years, further establishing the field of Cognitive Translatology. Empirical and interdisciplinary studies investigating translation and interpreting processes promise a hitherto unprecedented predictive and explanatory power. This collection contains such studies which observe behaviour during translation and interpreting. The contributions cover a vast area and investigate behaviour during translation and interpreting – with a focus on training of future professionals, on language processing more generally, on the role of technology in the practice of translation and interpreting, on translation of multimodal media texts, on aspects of ergonomics and usability, on emotions, self-concept and psychological factors, and finally also on revision and post-editing. For the present publication, we selected a number of contributions presented at the Second International Congress on Translation, Interpreting and Cognition hosted by the Tra&Co Lab at the Johannes Gutenberg University of Mainz. Most of the papers in this volume are formulated in a particular constraint-based grammar framework, Head-driven Phrase Structure Grammar. The contributions investigate how the lexical and constructional aspects of this theory can be combined to provide an answer to this question across different linguistic sub-theories

The way out of the box

Synopsis: Cognitive aspects of the translation process have become central in Translation and Interpreting Studies in recent years, further establishing the field of Cognitive Translatology. Empirical and interdisciplinary studies investigating translation and interpreting processes promise a hitherto unprecedented predictive and explanatory power. This collection contains such studies which observe behaviour during translation and interpreting. The contributions cover a vast area and investigate behaviour during translation and interpreting – with a focus on training of future professionals, on language processing more generally, on the role of technology in the practice of translation and interpreting, on translation of multimodal media texts, on aspects of ergonomics and usability, on emotions, self-concept and psychological factors, and finally also on revision and post-editing. For the present publication, we selected a number of contributions presented at the Second International Congress on Translation, Interpreting and Cognition hosted by the Tra&Co Lab at the Johannes Gutenberg University of Mainz

Poszukiwanie informacji w procesie post-edycji i tłumaczenia

Wydział AnglistykiRozprawa miała na celu zbadanie, jak osoby studiujące tłumaczenie i filologię angielską korzystają z tłumaczenia maszynowego i źródeł internetowych podczas tłumaczenia oraz post-edycji dwóch typów tekstu. Rozdział 1 wprowadza pojęcia tłumaczenia maszynowego, post-edycji i stosunku tłumaczy do nich. Rozdział 2 dotyczy poszukiwania informacji oraz modeli kompetencji tłumaczy. Rozdział 3 przedstawia wysiłek w procesie tłumaczenia i post-edycji, skupiając się na metodologii w okulografii i badaniach wykorzystujących keylogging. Rozdział 4 opisuje eksperymentalne badanie efektów oraz korelacji między aspektami nauczania tłumaczenia oraz poszukiwania informacji w tłumaczeniu i post-edycji. Zbadano siedem hipotez dotyczących efektu typu tekstu oraz grupy na wskaźniki wysiłku, zakres wykorzystanych źródeł oraz poprawność tłumaczenia. Analizy korelacyjne dotyczyły poprawności i procentu sprawdzonych wybranych słów/fraz, jak również procentowo oszacowanego czasu w źródłach i stosunku do tłumaczenia maszynowego oraz postrzeganego wysiłku, który to również skorelowano z zakresem źródeł. Niektóre z hipotez częściowo potwierdzono, ale relacje między wysiłkiem, poprawnością i stosunkiem do tłumaczenia maszynowego w poszukiwaniu informacji w tłumaczeniu i post-edycji są bardzo złożone. Opisane zależności mogą być szczególnie przydatne w nauczaniu tłumaczenia i badaniach nad procesem przekładu.This dissertation investigated translation trainees and EFL students interacting with machine translation and online resources (OR) in translation and post-editing tasks for two text types. Chapter 1 introduces the concepts of machine translation, post-editing, and translators’ attitudes towards them. Chapter 2 details information behaviour and translator competence models. Chapter 3 presents effort in both translation and post-editing process, with the focus on methodology in eye-tracking and keylogging studies. Chapter 4 is a detailed report on the experimental study on the effects and correlations between aspects of translation training and information behaviour in translation and post-editing. The experimental study tested seven hypotheses about effects of task type and group membership on effort indicators, range of consulted OR, and translation accuracy. Correlational analyses were also made between accuracy and percentage of researched rich points, as well as percentage of time spent in OR with attitude and perceived effort which was also correlated with the range of consulted OR. Some of the hypotheses were partially confirmed, but the relationship between effort, accuracy, and attitude in information searching during translation and post-editing is intensely nuanced. The findings may be particularly valuable for translation trainers and translation process researchers

Multi-modal post-editing of machine translation

As MT quality continues to improve, more and more translators switch from traditional translation from scratch to PE of MT output, which has been shown to save time and reduce errors. Instead of mainly generating text, translators are now asked to correct errors within otherwise helpful translation proposals, where repetitive MT errors make the process tiresome, while hard-to-spot errors make PE a cognitively demanding activity. Our contribution is three-fold: first, we explore whether interaction modalities other than mouse and keyboard could well support PE by creating and testing the MMPE translation environment. MMPE allows translators to cross out or hand-write text, drag and drop words for reordering, use spoken commands or hand gestures to manipulate text, or to combine any of these input modalities. Second, our interviews revealed that translators see value in automatically receiving additional translation support when a high CL is detected during PE. We therefore developed a sensor framework using a wide range of physiological and behavioral data to estimate perceived CL and tested it in three studies, showing that multi-modal, eye, heart, and skin measures can be used to make translation environments cognition-aware. Third, we present two multi-encoder Transformer architectures for APE and discuss how these can adapt MT output to a domain and thereby avoid correcting repetitive MT errors.Angesichts der stetig steigenden Qualität maschineller Übersetzungssysteme (MÜ) post-editieren (PE) immer mehr Übersetzer die MÜ-Ausgabe, was im Vergleich zur herkömmlichen Übersetzung Zeit spart und Fehler reduziert. Anstatt primär Text zu generieren, müssen Übersetzer nun Fehler in ansonsten hilfreichen Übersetzungsvorschlägen korrigieren. Dennoch bleibt die Arbeit durch wiederkehrende MÜ-Fehler mühsam und schwer zu erkennende Fehler fordern die Übersetzer kognitiv. Wir tragen auf drei Ebenen zur Verbesserung des PE bei: Erstens untersuchen wir, ob andere Interaktionsmodalitäten als Maus und Tastatur das PE unterstützen können, indem wir die Übersetzungsumgebung MMPE entwickeln und testen. MMPE ermöglicht es, Text handschriftlich, per Sprache oder über Handgesten zu verändern, Wörter per Drag & Drop neu anzuordnen oder all diese Eingabemodalitäten zu kombinieren. Zweitens stellen wir ein Sensor-Framework vor, das eine Vielzahl physiologischer und verhaltensbezogener Messwerte verwendet, um die kognitive Last (KL) abzuschätzen. In drei Studien konnten wir zeigen, dass multimodale Messung von Augen-, Herz- und Hautmerkmalen verwendet werden kann, um Übersetzungsumgebungen an die KL der Übersetzer anzupassen. Drittens stellen wir zwei Multi-Encoder-Transformer-Architekturen für das automatische Post-Editieren (APE) vor und erörtern, wie diese die MÜ-Ausgabe an eine Domäne anpassen und dadurch die Korrektur von sich wiederholenden MÜ-Fehlern vermeiden können.Deutsche Forschungsgemeinschaft (DFG), Projekt MMP