Unsupervised neural machine translation between the Portuguese language and the Chinese and Korean languages

Tese de Mestrado, Informática, 2023, Universidade de Lisboa, Faculdade de CiênciasO propósito desta dissertação é apresentar um estudo comparativo e de reprodução sobre técnicas de Tradução Automática Neuronal Não-Supervisionada (Unsupervised Neural Machine Translation) para o par de línguas Português (PT) →Chinês (ZH) e Português (PT) → Coreano (KR) tirando partido de ferramentas e recursos online. A escolha destes pares de línguas prende-se com duas grandes razões. A primeira refere-se à importância no panorama global das línguas asiáticas, nomeadamente do chinês, e também pela infuência que a língua portuguesa desempenha no mundo especialmente no hemisfério sul. A segunda razão é puramente académica. Como há escassez de estudos na área de Processamento Natural de Linguagem (NLP) com línguas não-germânicas (devido à hegemonia da língua inglesa), procurou-se desenvolver um trabalho que estude a infuência das técnicas de tradução não supervisionada em par de línguas poucos estudadas, a fm de testar a sua robustez. Falada por um quarto da população mundial, a língua chinesa é o“Ás”no baralho de cartas da China. De acordo com o International Chinese Language Education Week, em 2020 estimava-se que 200 milhões pessoas não-nativas já tinham aprendido chinês e que no ano corrente se encontravam mais de 25 milhões a estudá-la. Com a infuência que a língua chinesa desempenha, torna-se imperativo desenvolver ferramentas que preencham as falhas de comunicação. Assim, nesta conjuntura global surge a tradução automática como ponte de comunicação entre várias culturas e a China. A Coreia do Sul, também conhecida como um dos quatro tigres asiáticos, concretizou um feito extraordinário ao levantar-se da pobreza extrema para ser um dos países mais desenvolvidos do mundo em duas gerações. Apesar de não possuir a hegemonia económica da China, a Coreia do Sul exerce bastante infuência devido ao seu soft power na área de entretenimento, designado por hallyu. Esta“onda”de cultura pop coreana atraí multidões para a aprendizagem da cultura. De forma a desvanecer a barreira comunicativa entre os amantes da cultura coreana e os nativos, a tradução automática é um forte aliado porque permite a interação entre pessoas instantaneamente sem a necessidade de aprender uma língua nova. Apesar de Portugal não ter ligações culturais com a Coreia, há uma forte ligação com a região administrativa especial de Macau (RAEM) onde o português é uma das línguas ofciais, sendo que a Tradução Automática entre ambas as línguas ofciais é uma das áreas estratégicas do governo local tendo sido estabelecido um laboratório de Tradução Automática no Instituto Politécnico de Macau que visa construir um sistema que possa ser usado na função pública de auxílio aos tradutores. Neste trabalho foram realizadas duas abordagens: (i) Tradução Automática Neuronal Não Supervisionada (Unsupervised Neural Machine Translation) e; (ii) abordagem pivô (pivot approach). Como o foco da dissertação é em técnicas nãosupervisionadas, nenhuma das arquiteturas fez uso de dados paralelos entre os pares de línguas em questão. Nomeadamente, na primeira abordagem usou-se dados monolingues. Na segunda introduziu-se uma terceira língua pivô que é utilizada para estabelecer a ponte entre a língua de partida e a de chegada. Esta abordagem à tradução automática surgiu com a necessidade de criar sistemas de tradução para pares de línguas onde existem poucos ou nenhuns dados paralelos. Como demonstrado por Koehn and Knowles [2017a], a tradução automática neuronal precisa de grandes quantidades de dados a fm de ter um desempenho melhor que a Tradução Automática Estatística (SMT). No entanto, em pares de línguas com poucos recursos linguísticos isso não é exequível. Para tal, a arquitetura de tradução automática não supervisionada somente requer dados monolingues. A implementação escolhida foi a de Artetxe et al. [2018d] que é constituída por uma arquitetura encoder-decoder. Como contém um double-encoder, para esta abordagem foram consideradas ambas direções: Português ↔ Chinês e Português ↔ Coreano. Para além da reprodução para línguas dissimilares com poucos recursos, também foi elaborado um estudo de replicação do artigo original usando os dados de um dos pares de línguas estudados pelos autores: Inglês ↔ Francês. Outra alternativa para a falta de corpora paralelos é a abordagem pivô. Nesta abordagem, o sistema faz uso de uma terceira língua, designada por pivô, que liga a língua de partida à de chegada. Esta opção é tida em conta quando há existência de dados paralelos em abundância entre as duas línguas. A motivação deste método é fazer jus ao desempenho que as redes neuronais têm quando são alimentadas com grandes volumes de dados. Com a existência de grandes quantidades de corpora paralelos entre todas as línguas em questão e a pivô, o desempenho das redes compensa a propagação de erro introduzida pela língua intermediária. No nosso caso, a língua pivô escolhida foi o inglês pela forte presença de dados paralelos entre o pivô e as restantes três línguas. O sistema começa por traduzir de português para inglês e depois traduz a pivô para coreano ou chinês. Ao contrário da primeira abordagem, só foi considerada uma direção de Português → Chinês e Português → Coreano. Para implementar esta abordagem foi considerada a framework OpenNMT desenvolvida por [Klein et al., 2017]. Os resultados foram avaliados usando a métrica BLEU [Papineni et al., 2002b]. Com esta métrica foi possível comparar o desempenho entre as duas arquiteturas e aferir qual é o método mais efcaz para pares de línguas dissimilares com poucos recursos. Na direção Português → Chinês e Português → Coreano a abordagem pivô foi superior tendo obtido um BLEU de 13,37 pontos para a direção Português → Chinês e um BLEU de 17,28 pontos na direção Português → Coreano. Já com a abordagem de tradução automática neural não supervisionada o valor mais alto obtido na direção Português → Coreano foi de um BLEU de 0,69, enquanto na direção de Português → Chinês foi de 0,32 BLEU (num total de 100). Os valores da tradução não supervisionada vão estão alinhados com os obtidos por [Guzmán et al., 2019], [Kim et al., 2020]. A explicação dada para estes valores baixos prende-se com a qualidade dos cross-lingual embeddings. O desempenho dos cross-lingual embeddings tende a degradar-se quando mapeia pares de línguas distantes e, sendo que modelo de tradução automática não supervisionado é inicializado com os cross-lingual embeddings, caso estes sejam de baixa qualidade, o modelo não converge para um ótimo local, resultando nos valores obtidos na dissertação. Dos dois métodos testados, verifica-se que a abordagem pivô é a que tem melhor performance. Tal como foi possível averiguar pela literatura corrente e também pelos resultados obtidos nesta dissertação, o método neuronal não-supervisionado proposto por Artetxe et al. [2018d] não é sufcientemente robusto para inicializar um sistema de tradução suportado por textos monolingues em línguas distantes. Porém é uma abordagem promissora porque permitiria colmatar uma das grandes lacunas na área de Tradução Automática que se cinge à falta de dados paralelos de boa qualidade. No entanto seria necessário dar mais atenção ao problema dos cross-lingual embeddings em mapear línguas distantes. Este trabalho fornece uma visão sobre o estudo de técnicas não supervisionadas para pares de línguas distantes e providencia uma solução para a construção de sistemas de tradução automática para os pares de língua português-chinês e português-coreano usando dados monolingues.This dissertation presents a comparative and reproduction study on Unsupervised Neural Machine Translation techniques in the pair of languages Portuguese (PT) → Chinese (ZH) and Portuguese (PT) → Korean(KR). We chose these language-pairs for two main reasons. The frst one refers to the importance that Asian languages play in the global panorama and the infuence that Portuguese has in the southern hemisphere. The second reason is purely academic. Since there is a lack of studies in the area of Natural Language Processing (NLP) regarding non-Germanic languages, we focused on studying the infuence of nonsupervised techniques in under-studied languages. In this dissertation, we worked on two approaches: (i) Unsupervised Neural Machine Translation; (ii) the Pivot approach. The frst approach uses only monolingual corpora. As for the second, it uses parallel corpora between the pivot and the non-pivot languages. The unsupervised approach was devised to mitigate the problem of low-resource languages where training traditional Neural Machine Translations was unfeasible due to requiring large amounts of data to achieve promising results. As such, the unsupervised machine translation only requires monolingual corpora. In this dissertation we chose the mplementation of Artetxe et al. [2018d] to develop our work. Another alternative to the lack of parallel corpora is the pivot approach. In this approach, the system uses a third language (called pivot) that connects the source language to the target language. The reasoning behind this is to take advantage of the performance of the neural networks when being fed with large amounts of data, making it enough to counterbalance the error propagation which is introduced when adding a third language. The results were evaluated using the BLEU metric and showed that for both language pairs Portuguese → Chinese and Portuguese → Korean, the pivot approach had a better performance making it a more suitable choice for these dissimilar low resource language pairs

CHORUS Deliverable 2.1: State of the Art on Multimedia Search Engines

Based on the information provided by European projects and national initiatives related to multimedia search as well as domains experts that participated in the CHORUS Think-thanks and workshops, this document reports on the state of the art related to multimedia content search from, a technical, and socio-economic perspective. The technical perspective includes an up to date view on content based indexing and retrieval technologies, multimedia search in the context of mobile devices and peer-to-peer networks, and an overview of current evaluation and benchmark inititiatives to measure the performance of multimedia search engines. From a socio-economic perspective we inventorize the impact and legal consequences of these technical advances and point out future directions of research

Named Entity Recognition in Chinese Clinical Text

Objective: Named entity recognition (NER) is one of the fundamental tasks in natural language processing (NLP). In the medical domain, there have been a number of studies on NER in English clinical notes; however, very limited NER research has been done on clinical notes written in Chinese. The goal of this study is to develop corpora, methods, and systems for NER in Chinese clinical text. Materials and methods: To study entities in Chinese clinical text, we started with building annotated clinical corpora in Chinese. We developed an NER annotation guideline in Chinese by extending the one used in the 2010 i2b2 NLP challenge. We randomly selected 400 admission notes and 400 discharge summaries from Peking Union Medical College Hospital (PUMCH) in China. For each note, four types of entities including clinical problems, procedures, labs, and medications were annotated according to the developed guideline. In addition, an annotation tool was developed to assist two MD students to annotate Chinese clinical documents. A comparison of entity distribution between Chinese and English clinical notes (646 English and 400 Chinese discharge summaries) was performed using the annotated corpora, to identify the important features for NER. In the NER study, two-thirds of the 400 notes were used for training the NER systems and one-third were used for testing. We investigated the effects of different types of features including bag-of-characters, word segmentation, part-of-speech, and section information, with different machine learning (ML) algorithms including Conditional Random Fields (CRF), Support Vector Machines (SVM), Maximum Entropy (ME), and Structural Support Vector Machines (SSVM) on the Chinese clinical NER task. All classifiers were trained on the training dataset, evaluated on the test set, and microaveraged precision, recall, and F-measure were reported. Results: Our evaluation on the independent test set showed that most types of features were beneficial to Chinese NER systems, although the improvements were limited. By combining word segmentation and section information, the system achieved the highest performance, indicating that these two types of features are complementary to each other. When the same types of optimized features were used, CRF and SSVM outperformed SVM and ME. More specifically, SSVM reached the highest performance among the four algorithms, with F-measures of 93.51% and 90.01% for admission notes and discharge summaries respectively. Conclusions: In this study, we created large annotated datasets of Chinese admission notes and discharge summaries and then systematically evaluated different types of features (e.g., syntactic, semantic, and segmentation information) and four ML algorithms including CRF, SVM, SSVM, and ME for clinical NER in Chinese. To the best of our knowledge, this is one of the earliest comprehensive effort in Chinese clinical NER research and we believe it will provide valuable insights to NLP research in Chinese clinical text. Our results suggest that both word segmentation and section information improves NER in Chinese clinical text, and SSVM, a recent sequential labelling algorithm, outperformed CRF and other classification algorithms. Our best system achieved F-measures of 90.01% and 93.52% on Chinese discharge summaries and admission notes, respectively, indicating a promising start on Chinese NLP research

Design of a Controlled Language for Critical Infrastructures Protection

We describe a project for the construction of controlled language for critical infrastructures protection (CIP). This project originates from the need to coordinate and categorize the communications on CIP at the European level. These communications can be physically represented by official documents, reports on incidents, informal communications and plain e-mail. We explore the application of traditional library science tools for the construction of controlled languages in order to achieve our goal. Our starting point is an analogous work done during the sixties in the field of nuclear science known as the Euratom Thesaurus.JRC.G.6-Security technology assessmen

Language-Learner Computer Interactions

This book focuses on learner-computer interactions (LCI) in second language learning environments drawing largely on sociocultural theories of language development. It brings together a rich and varied range of theoretical discussions and applications in order to illustrate the way in which LCI can enrich our comprehension of technology-mediated communication, hence enhancing learners’ digital literacy skills. The book is based on the premise that, in order to fully understand the nature of language and literacy development in digital spaces, researchers and practitioners in linguistics, sciences and engineering need to borrow from each others’ theoretical and practical toolkits. In light of this premise, themes include such aspects as educational ergonomics, affordances, complex systems learning, learner personas and corpora, while also describing such data collecting tools as video screen capture devices, eye-tracking or intelligent learning tutoring systems

Tense-aspect processing in second language learners

This dissertation provides a language processing perspective on the study of second language acquisition (SLA) of tense and aspect. Of special interest are the universal vis-à-vis language- specific dimensions of temporal and aspectual semantics involved. According to the Aspect Hypothesis (AH, e.g. Andersen & Shirai, 1994), the initial acquisition and subsequent emergence of (perfective) past tense and progressive aspect morphology follow a semantic-driven, universal sequence. The AH appeals to a cognitive-based prototype account (Shirai & Andersen, 1995), and has gained ample empirical support from offline data in the past two decades. Mounting evidence of transfer, however, has begun to emerge in recent psycholinguistic research, suggesting that grammatical aspectual categories such as the English progressive have non-trivial influence on principles of information organization in language comprehension among L2 learners and bilingual speakers (Stutterheim & Carroll, 2006). This dissertation undertakes a psycholinguistic investigation of L2 learners’ processing of English past and progressive morphology. Participants included native English speakers as well as English L2 learners from Korean, German, and Mandarin Chinese backgrounds, whose L1s differ systematically with respect to past and progressive morphology. This cross-linguistic design enabled a systematic testing of both the prototype and transfer hypotheses in one single study. Three word-by-word self-paced reading experiments examined L2 learners’ automaticity in morphological processing, the universality of tense-aspect prototypes, and aspectual coercion. Experiment I generated evidence that L2 learners were generally capable of detecting tense- aspect morphosyntactic errors online. Reading time results from Experiment II revealed that L2 learners did not show uniform processing advantages afforded by tense-aspect prototypes. Instead, there exist L1 effects in prototypes, at least from evidence in processing L2 tense-aspect distinctions. Experiment III investigated the processing consequences of aspectual coercion in L2 learners, and results indicated strong L1 influence. The most robust finding across the three experiments is that the L2 learners showed clear L1-based variations in their performance, reflecting a strong tendency for transfer. Notably, these results were obtained after controlling for L2 proficiency and inflected verb form frequencies. A more prominent role of L1 influence is implicated in L2 learners’ representation of tense-aspect prototypes than previously assumed

Designing, implementing, and evaluating an automated writing evaluation tool for improving EFL graduate students’ abstract writing: a case in Taiwan

Writing English research article (RA) abstracts is a difficult but mandatory task for Taiwanese engineering graduate students (Feng, 2013). Understanding the current situation and needs of Taiwanese engineering graduate students, this dissertation aimed to develop and evaluate an automated writing evaluation (AWE) tool to assist their research article (RA) abstract writing in English by following a Design-Based Research (DBR) approach as the methodological framework. DBR was chosen because it strives to solve real-world problems through multiple iterations of development and building on results from each iteration to advance the project. Six design iterations were undertaken to develop and to evaluate the AWE tool in this dissertation, including (1) corpus compilation of engineering RAs, (2) genre analysis of engineering abstracts, (3) machine learning of move classification in abstracts, (4) analysis of lexical bundles used to express moves, (5) analysis of the choice of verb categories associated with moves, and finally, (6) AWE tool development based on previous findings, classroom implementation, and evaluation of the AWE tool following Chapelle’s (2001) computer-assisted language learning (CALL) framework. To begin with, I collected a corpus of 480 engineering RAs (Corpus-480) to extract appropriate linguistic properties as pedagogical materials to be implemented in the AWE tool. A sub-corpus (Corpus-72) was compiled with 72 RAs randomly chosen from Corpus-480 for manual and automated analyses. Next, to seek the best descriptive framework for the structure of engineering RA abstracts, two move schemata were compared: (1) IMRD (Introduction, Methodology, Results, and Discussion) and (2) CARS (Create-A-Research-Space, Swales, 1990). Abstracts in Corpus-72 were annotated and these two schemas were evaluated according to three quantitative metrics devised specifically for this comparison. Applying a statistical natural language processing (StatNLP) approach, a Support Vector Machine (SVM) was trained for automated move classification in abstracts. Formulaic language in engineering RA sections was used as linguistic features to automatically classify moves in abstracts. Additionally, four-word lexical bundles and verb categories were identified from Corpus-480 and Corpus-72, respectively. Four-word lexical bundles associated with moves in abstracts were extracted automatically. Additionally, verb categories (i.e., tense, aspect, and voice) in moves of abstracts were identified using CyWrite::Analyzer, a hybrid (statistical and rule-based) NLP software. Finally, the AWE tool was developed, based on the findings from the previous iterations, and implemented in an English-as-a-foreign-language (EFL) classroom setting. Through analyzing students’ drafts before and after using the tool, and responses to a questionnaire and a semi-structured interview, the AWE tool was evaluated based on Chapelle’s (2001) CALL evaluation framework. The findings showed that students attempted to improve their abstracts by adding, deleting, or changing the sequences of their sentences, lexical bundles, and verb categories in their abstracts. Their attitudes toward the effectiveness and appropriateness of the tool were quite positive. Overall, the AWE tool drew students’ attention to the use of lexical bundles and verb categories to achieve the communicative purposes of each move in their abstracts. In conclusion, this dissertation started from Taiwanese engineering students’ needs to improve their English abstract writing, and attempted to develop and evaluate an AWE tool for assisting them. Following DBR, the findings from this dissertation are discussed to improve the next generation of the AWE tools. Having these iterations in place, future studies can focus on developing pedagogical materials from genre-based analysis in different disciplines to fulfill learners’ needs

Representation Learning for Words and Entities

This thesis presents new methods for unsupervised learning of distributed representations of words and entities from text and knowledge bases. The first algorithm presented in the thesis is a multi-view algorithm for learning representations of words called Multiview Latent Semantic Analysis (MVLSA). By incorporating up to 46 different types of co-occurrence statistics for the same vocabulary of english words, I show that MVLSA outperforms other state-of-the-art word embedding models. Next, I focus on learning entity representations for search and recommendation and present the second method of this thesis, Neural Variational Set Expansion (NVSE). NVSE is also an unsupervised learning method, but it is based on the Variational Autoencoder framework. Evaluations with human annotators show that NVSE can facilitate better search and recommendation of information gathered from noisy, automatic annotation of unstructured natural language corpora. Finally, I move from unstructured data and focus on structured knowledge graphs. I present novel approaches for learning embeddings of vertices and edges in a knowledge graph that obey logical constraints.Comment: phd thesis, Machine Learning, Natural Language Processing, Representation Learning, Knowledge Graphs, Entities, Word Embeddings, Entity Embedding