Comparing Intrinsic and Extrinsic Evaluation of MT Output in a Dialogue System

We present an exploratory study to assess machine translationoutput for application in a dialogue system using an intrinsicand an extrinsic evaluation method. For the intrinsicevaluation we developed an annotation scheme to determinethe quality of the translated utterances in isolation. For theextrinsic evaluation we employed theWizard of Oz techniqueto assess the quality of the translations in the context of a dialogueapplication. Results differ and we discuss the possiblereasons for this outcome

Automatic interpretation or automatic translation of speech: concepts, definitions and software architecture

A interpretação automática (IA), ou tradução automática de fala, é uma tecnologia que traduz discurso oral de uma língua para outra através de três funcionalidades acopladas em um único sistema computacional: reconhecimento automático de fala, tradução automática e síntese de fala. Apresentado pela primeira vez em 1983, durante a convenção ITU Telecom, em Genebra, o conceito de IA veicula a ideia de sistemas capazes de promover a comunicação entre pessoas que falam línguas diferentes de forma espontânea e eficaz (Lee 2015). A IA e seus sistemas, todavia, ainda são escassamente estudados, especialmente na área de Estudos de Interpretação, sendo difundidos, em sua maioria, na área de Ciências da Computação (Pöchhacker 2004). Desse modo, o objetivo geral deste trabalho é apresentar os resultados de uma pesquisa bibliográfico-documental, que teve como proposta investigar como a IA é concebida pelos teóricos que a estudam.Machine Interpreting (MI) or Speech-to-Speech Translation (S2S) is a new technology that converts spoken utterances from one language into another through three functionalities grouped in only one software: Automatic Speech Recognition (ASR), Machine Translation (MT) and Speech Synthesis (or TTS - Text-To-Speech). Firstly presented in 1983, during ITU Telecom Convention, in Geneva, MI consists in a software that enables natural and efficient communication between people from different languages (Lee, 2015). Nevertheless, MI and its components are scarcely studied, especially in Interpreting Studies, generally investigated by the areas of Computer Sciences (Pöchhacker 2004). Following this trend, this paper aims to present the result of a bibliographical research whose proposal was to investigate how MI is conceived by scholars

Lessons Learned from a Task-Based Evaluation of Speech-to-Speech Machine Translation

For several years we have been conducting Accuracy Based Evaluations (ABE) of the JANUS speech-to-speech MT system (Gates et al., 1997) which measure quality and fidelity of translation. Recently we have begun to design a Task Based Evaluation for JANUS (Thomas, 1999) which measures goal completion. This paper describes what we have learned by comparing the two types of evaluation. Both evaluations (ABE and TBE) were conducted on a common set of user studies in the semantic domain of travel planning. 1. Introduction For several years we have been conducting Accuracy Based Evaluations (ABE) (Gates et al., 1997) of the JANUS speech-to-speech machine translation system (Waibel, 1996; Levin et al., ). Our ABE focuses on whether the meaning of a source language segment is totally and accurately conveyed in the target language, and also includes a separate measure of fluency. This type of evaluation was useful in the early stages of system development for tracking our improvement over time..

Culturally-relevant augmented user interfaces for illiterate and semi-literate users

This thesis discusses guidelines for developers of Augmented User Interfaces that can be used by illiterate and semi-literate users. To discover how illiterate and semi-literate users intuitively understand interaction with a computer, a series of Wizard of Oz experiments were conducted. In the first Wizard of Oz study, users were presented with a standard desktop computer, fitted with a number of input devices to determine how they assume interaction should occur. This study found that the users preferred the use of speech and gestures which mirrored findings from other researchers. The study also found that users struggled to understand the tab metaphor which is used frequently in applications. From these findings, a localised culturally-relevant tab interface was developed to determine the feasibility of localised Graphical User Interface components. A second study was undertaken to compare the localised tab interface with the traditional tabbed interface. This study collected both quantitative and qualitative data from the participants. It found that users could interact with a localised tabbed interface faster and more accurately than with the traditional counterparts. More importantly, users stated that they intuitively understood the localised interface component, whereas they did not understand the traditional tab metaphor. These user studies have shown that the use of self-explanatory animations, video feedback, localised tabbed interface metaphors and voice output have a positive impact on enabling illiterate and semi-literate users to access information.TeXAdobe Acrobat 9.51 Paper Capture Plug-i

MULTIMODALITY IN COMPUTER MEDIATED COMMUNICATION

2002/2003XVI Ciclo1974Versione digitalizzata della tesi di dottorato cartacea

Hybrid discourse modeling and summarization for a speech-to-speech translation system

The thesis discusses two parts of the speech-to-speech translation system VerbMobil: the dialogue model and one of its applications, multilingual summary generation. In connection with the dialogue model, two topics are of special interest: (a) the use of a default unification operation called overlay as the fundamental operation for dialogue management; and (b) an intentional model that is able to describe intentions in dialogue on five levels in a language-independent way. Besides the actual generation algorithm developed, we present a comprehensive evaluation of the summarization functionality. In addition to precision and recall, a new characterization - confabulation - is defined that provides a more precise understanding of the performance of complex natural language processing systems.Die vorliegende Arbeit behandelt hauptsächlich zwei Themen, die für das VerbMobil-System, ein Übersetzungssystem gesprochener Spontansprache, entwickelt wurden: das Dialogmodell und als Applikation die multilinguale Generierung von Ergebnissprotokollen. Für die Dialogmodellierung sind zwei Themen von besonderem Interesse. Das erste behandelt eine in der vorliegenden Arbeit formalisierte Default-Unifikations-Operation namens Overlay, die als fundamentale Operation für Diskursverarbeitung dient. Das zweite besteht aus einem intentionalen Modell, das Intentionen eines Dialogs auf fünf Ebenen in einer sprachunabhängigen Repräsentation darstellt. Neben dem für die Protokollgenerierung entwickelten Generierungsalgorithmus wird eine umfassende Evaluation zur Protokollgenerierungsfunktionalität vorgestellt. Zusätzlich zu "precision" und "recall" wird ein neues Maß - Konfabulation (Engl.: "confabulation") - vorgestellt, das eine präzisere Charakterisierung der Qualität eines komplexen Sprachverarbeitungssystems ermöglicht

Use and Evaluation of Controlled Languages in Industrial Environments and Feasibility Study for the Implementation of Machine Translation

El presente trabajo de investigación se enmarca en los estudios de doctorado en traducción y la sociedad del conocimiento de la Universidad de Valencia y, en concreto, en la línea de investigación en tecnologías de la traducción, terminología y localización. En este sentido, esta disertación surge por la necesidad de establecer una metodología de investigación y ofrecer resultados empíricos sobre el desarrollo, implementación y evaluación de lenguajes controlados en la documentación técnica y su efecto tanto en los textos originales como en las traducciones de estos documentos. Así pues, el objetivo ha sido desarrollar una metodología para evaluar el impacto de los lenguajes controlados en la producción de documentación técnica dentro de contextos industriales y, más en concreto, en la elaboración de documentación técnica para el vehículo. El impacto se ha concretado en la mejora de la traducibilidad automática, un concepto que hemos discutido ampliamente en el capítulo 4, así como de la calidad de los textos meta.This research is part of the doctoral studies program "La traducción y la sociedad del conocimiento" at the University of Valencia. In particular the area of ​​research is translation technology, terminology and localisation. In this sense, this dissertation arises from the need to establish a research methodology and to provide empirical results on the development, implementation and evaluation of controlled languages ​​in the technical documentation and its effect on both original texts and the translations of these documents. Thus, the aim has been to develop a methodology to assess the impact of controlled languages ​​in the production of technical documentation in industrial contexts, and more specifically in the technical documentation for the vehicle. The impact has resulted in improved automatic translatability, a concept we have discussed at length in Chapter 4, as well as in the quality of the target texts