Traducció automàtica de la parla : creació i avaluació de sis motors de TAE

La traducció automàtica (TA) ha millorat notablement aquests darrers anys. No obstant això, la traducció de la parla i el processament del llenguatge natural encara són un desafiament per als sistemes de TA. Aquest treball sorgeix amb la motivació d'aportar una possible solució a la manca de naturalitat de la traducció automàtica per veu. La hipòtesi és que es pot millorar l'oralitat de les traduccions mitjançant corpus orals transcrits i optimitzacions en l'entrenament dels sistemes de TA. Per demostrar aquesta hipòtesi, es creen amb KantanMT (després d'haver provat amb MTradumàtica) sis motors de traducció automàtica estadística entrenats amb diferents corpus orals transcrits i escrits per després, evaluar-los.La traducción automática (TA) ha mejorado notablemente en los últimos años; sin embargo, la traducción del habla y el procesamiento del lenguaje natural siguen siendo todo un reto para los sistemas de TA. Este trabajo surge con la motivación de aportar una posible solución a la falta de naturalidad en la traducción automática del habla. Se parte de la hipótesis de que se puede mejorar la oralidad de las traducciones introduciendo corpus orales transcritos y optimizaciones en el entrenamiento de los sistemas de TA. Para probar esta hipótesis, se crean con KantanMT -tras probar MTradumàtica- seis motores de traducción automática estadística entrenados con distintos corpus orales transcritos y escritos y, después, se evalúan.Machine Translation (MT) has been greatly improved in recent years. Nevertheless, Spoken Language Translation (SLT) and natural language processing remain a major challenge for MT engines. The purpose of this work is to provide a possible solution to the lack of naturalness in SLT. The work is based on the hypothesis that it is possible to improve the orality of translations by introducing transcribed oral corpus and optimizations in the training process of MT systems. To test this hypothesis, six statistical machine translation engines, trained with different transcribed oral and written corpora, were created with KantanMT, after trying MTradumàtica, and then evaluated

Human evaluation and statistical analyses on machine reading comprehension, question generation and open-domain dialogue

Evaluation is a critical element in the development process of many natural language based systems. In this thesis, we will present critical analyses of standard evaluation methodologies applied in the following Natural Language Processing (NLP) domains: machine reading comprehension (MRC), question generation (QG), and open-domain dialogue. Generally speaking, systems from tasks like MRC are usually evaluated by comparing the similarity between hand-crafted references and system generated outputs using automatic evaluation metrics, thus these metrics are mainly borrowed from other NLP tasks that have been well-developed, such as machine translation and text summarization. Meanwhile, the evaluation of QG and dialogues is even a known open problem as such tasks do not have the corresponding references for computing the similarity, and human evaluation is indispensable when assessing the performance of the systems from these tasks. However, human evaluation is unfortunately not always valid because: i) human evaluation may cost too much and be hard to deploy when experts are involved; ii) human assessors can lack reliability in the crowd-sourcing environment. To overcome the challenges from both automatic metrics and human evaluation, we first design specific crowdsourcing human evaluation methods for these three target tasks, respectively. We then show that these human evaluation methods are reproducible, highly reliable, easy to deploy, and cost-effective. Additionally, with the data collected from our experiments, we measure the accuracy of existing automatic metrics and analyse the potential limitations and disadvantages of the direct application of these metrics. Furthermore, in allusion to the specific features of different tasks, we provide detailed statistical analyses on the collected data to discover their underlying trends, and further give suggestions about the directions to improving systems on different aspects