A Survey of Word Reordering in Statistical Machine Translation: Computational Models and Language Phenomena

Word reordering is one of the most difficult aspects of statistical machine translation (SMT), and an important factor of its quality and efficiency. Despite the vast amount of research published to date, the interest of the community in this problem has not decreased, and no single method appears to be strongly dominant across language pairs. Instead, the choice of the optimal approach for a new translation task still seems to be mostly driven by empirical trials. To orientate the reader in this vast and complex research area, we present a comprehensive survey of word reordering viewed as a statistical modeling challenge and as a natural language phenomenon. The survey describes in detail how word reordering is modeled within different string-based and tree-based SMT frameworks and as a stand-alone task, including systematic overviews of the literature in advanced reordering modeling. We then question why some approaches are more successful than others in different language pairs. We argue that, besides measuring the amount of reordering, it is important to understand which kinds of reordering occur in a given language pair. To this end, we conduct a qualitative analysis of word reordering phenomena in a diverse sample of language pairs, based on a large collection of linguistic knowledge. Empirical results in the SMT literature are shown to support the hypothesis that a few linguistic facts can be very useful to anticipate the reordering characteristics of a language pair and to select the SMT framework that best suits them.Comment: 44 pages, to appear in Computational Linguistic

Loan Phonology

For many different reasons, speakers borrow words from other languages to fill gaps in their own lexical inventory. The past ten years have been characterized by a great interest among phonologists in the issue of how the nativization of loanwords occurs. The general feeling is that loanword nativization provides a direct window for observing how acoustic cues are categorized in terms of the distinctive features relevant to the L1 phonological system as well as for studying L1 phonological processes in action and thus to the true synchronic phonology of L1. The collection of essays presented in this volume provides an overview of the complex issues phonologists face when investigating this phenomenon and, more generally, the ways in which unfamiliar sounds and sound sequences are adapted to converge with the native language’s sound pattern. This book is of interest to theoretical phonologists as well as to linguists interested in language contact phenomena

Affinity Prober를 활용한 BERT 언어 지식의 레이어 별 탐침 연구

학위논문(석사) -- 서울대학교대학원 : 인문대학 언어학과, 2023. 8. 신효필.Transformer(Vawsani et al., 2017)의 등장 이후 Self Attention 기제를 사용한 다양한 사전학습 언어모델(Pre-trained Language Model)이 제안되었다. 이러한 사전학습 언어 모델은 일반적으로 미세조정(fine-tuning)을 통해 다양한 자연어 처리 문제에서 높은 성능을 보여왔다. 언어학 분야에서는 언어 모델의 내재적 언어 지식을 탐구하기 위해 통사론, 의미론, 언어 습득 등의 이론 및 실험 언어학 접근법을 기반으로 활발히 연구되고 있다. 본 논문은 Jang et al (2022)에서 제안한 언어 지식 탐침 방법론인 Affinity Prober의 사용 범주를 확장시키는 것을 목표로 한다. 이를 위해 self-attention mechanism에서 어텐션 스코어 값을 보존하며 토큰 간의 관계를 해석하는 알고리즘인 ADTRAS 알고리즘 (An Algorithm for Decrypting Token Relationships within Attention Scores)을 제안한다. 본 논문은 ADTRAS 알고리즘을 활용하여 첫 번째 실험에서 GLUE 벤치마크 내의 통사-의미적 기능을 요구하는 6가지 태스크에 각각 훈련된 BERT 모형의 레이어 패턴을 분석한다. 이를 통해 BERT 모형이 토큰 관계의 유의미한 변화를 포착하고, ADTRAS 알고리즘을 활용하여 BERT 어텐션 변화를 기반으로 BERT 모델이 스스로 어휘 범주(Lexical Category)를 활용하여 품사 정보를 학습한다는 실증적인 증거를 제시한다. 또한 어휘 범주를 중심으로 BERT 레이어의 분명한 언어학적 특징을 일반화한다. 두 번째 실험으로는 Affinity Prober를 활용하여 통사적 언어현상에서의 최소쌍 문장을 처리하는 BERT의 특징을 분석한다. 이 실험은 사용된 15가지의 통사적 언어현상이 BERT 모델에서 처리되는 과정을 Affinity Prober를 활용하여 탐구하여 레이어 별 패턴을 분석하는 것을 목적으로 한다. 이러한 실험 결과로 총 네 가지의 패턴이 관찰되었는데, 본 논문은 관찰된 패턴이 각각 유사한 언어현상 별로 묶인다고 주장한다. 첫 번째 패턴은 Passive와 Ellipsis N-bar와 관련된 언어현상들이 주를 이루며, 두 번째 패턴은 Island Effects, 세 번째 패턴은 Movement에서의 Syntactic Constraints에서의 언어현상, 마지막으로 네 번째 패턴에서는 Verb Predicate Types과 논항 구조에서의 언어현상들로 나타난다. 이러한 각 레이어 별 패턴이 ADTRAS 알고리즘에서의 결과와 일치한다는 점에서 본 실험을 통해 도출된 결과를 뒷받침한다. 요약하자면, 본 논문은 ADTRAS 알고리즘을 제안하고, Jang et al (2022)에서 제안한 Affinity Prober를 확장하여 연구에 활용하였다. 이 과정에서 통사적 언어현상의 BERT 레이어 별 패턴을 성공적으로 추출하여 결과를 설명하고자 노력하였다.This paper presents a comprehensive investigation into the linguistic knowledge embedded within BERT, a pre-trained language model based on the Transformer architecture. We reinforce and expand upon the methodology proposed by Jang et al (2022) by introducing the ADTRAS algorithm (An Algorithm for Decrypting Token Relationships within Attention Scores), which decrypts token relationships within BERT's attention scores to analyze patterns at each layer. Our experiments using ADTRAS algorithm demonstrate that BERT autonomously learns part-of-speech information by leveraging lexical categories. We also provide insights into the general tendencies of BERT's layers when processing content words and function words. Additionally, we introduce the Classification of Sentence Sequencing (CSS) as a Finetuning Strategy, enabling indirect learning from minimal pairs, and leverage the Affinity Prober to examine syntactic linguistic phenomena using the BLiMP dataset. By tracing patterns and clustering similar phenomena, we enhance our understanding of BERT's interpretation of linguistic structures. Furthermore, we establish in detail the attributes of BERT layers related to lexical categories by connecting the general tendencies of the layers generalized by the ADTRAS algorithm with the results obtained through the Affinity Prober. Our study makes several contributions. First, we introduce the ADTRAS algorithm, which enables a comprehensive analysis of BERT's linguistic knowledge. Second, we provide experimental evidence demonstrating BERT's ability to learn part-of-speech information. Third, we offer insights into the tendencies observed in different layers of BERT. Fourth, we propose the CSS Finetuning Strategy, which allows for indirect learning from minimal pairs. Fifth, we successfully cluster syntactic phenomena using the Affinity Prober. Finally, we uncover the general attention tendency of BERT towards lexical categories.Chapter 1. Introduction 1 Chapter 2. Related Work 5 2.1. Unveiling Linguistic Insights: The Probing Classifier Framework 6 2.2. The Interplay of Syntactic Tree and Neural Networks 7 2.3. BERT and Linguistics 7 Chapter 3. Generalization of Layer-Wise Attention Using ADTRAS Algorithm 10 3.1. Binary Categorization of Part-of-Speech in Sentences: Content Words and Function Words 10 3.2. ADTRAS Algorithm 13 3.3. The General Language Understanding Evaluation (GLUE) Benchmark 16 3.3.1 The Corpus of Linguistic Acceptability (CoLA) 16 3.3.2 The Microsoft Research Paraphrase Corpus (MRPC) 17 3.3.3 The Stanford Sentiment Treebank 2.0 (SST-2) 17 3.3.4 The Quora Question Pairs (QQP) 18 3.3.5 The Multi-Genre Natural Language Inference (MNLI) 18 3.3.6 The Words in Context (WiC) 19 3.3.7 Summary 19 3.4. Evaluating Attention Variations in Lexical Categories on NLU tasks 20 3.4.1 Intrinsic Learning of Lexical Categories in BERT for Downstream Tasks 21 3.4.2 Generalization of Layer-Wise Attention in Fine-Tuned BERT Models 23 Chapter 4. Probing Intrinsic Linguistic Knowledges of Deep Learning-based Language Model using Affinity Prober 23 4.1. Jang et al (2022) 24 4.2. Affinity Prober 26 4.2.1 Multi-Head Attention on Transformer Architecture 26 4.2.2 Affinity Relationship 27 4.2.3 Probabilistic Distribution of Categorized Affnity Relationships 28 4.2.4 The Algorithm of Affinity Prober on BERT 29 Chapter 5. The Benchmark of Linguistic Minimal Pairs (BLiMP) 33 5.1. Adjunct Island 34 5.2. Animate Subject 35 5.2.1. Animate Subject Passive 35 5.2.2. Animate Subject Trans 35 5.3. Causative 36 5.4. Complex NP Island 37 5.5. Coordinate Structure Constraint 38 5.5.1. Left Branch 38 5.5.2. Object Extraction 39 5.6. Drop Argument 40 5.7. Ellipsis N-bar 41 5.8. Inchoative 42 5.9. Intransitive 43 5.10. Transitive 44 5.11. Left Branch Island 44 5.11.1. Echo Question 44 5.11.2. Simple Question 45 5.12. Passive 46 5.13. Sentential Subject Island 47 5.14. Wh Island 48 5.15. Wh Questions 49 5.15.1. Object Gap 49 5.15.2. Subject Gap 50 Chapter 6. Experiment 51 6.1. Finetuning Strategy 51 6.2. Result: Clustering of Similar Linguistic Phenomena 53 6.2.1. Group I: Passive and Ellipsis N-bar 54 6.2.2. Group II: Island Effects 58 6.2.3. Group III: Syntactic Constraints on Movement 62 6.2.4. Group IV: Verbal Predicate Types and Argument Structures 66 6.3. Summary 72 Chapter 7. Conclusion 73 Reference 76 Appendix. 79 Abstract (In Korean) 98석

Biased learning of phonological alternations

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Linguistics and Philosophy, 2013.Cataloged from PDF version of thesis.Includes bibliographical references (pages 198-203).What is the initial state of the grammar when children begin to figure out patterns of phonological alternations? This thesis documents the developmental stages of children acquiring Korean verb and noun paradigms, and provides a unified account using the initial Output-Output correspondence (OO-CORR) bias. It is claimed that OO-CORR constraints are undominated in the initial state but the empirical evidence for this claim is primarily anecdotal. I provide experimental support for the claim. In the acquisition literature, children's innovative forms are normally attributed to imperfect or incomplete mastery of the adult grammar. The evidence presented in this thesis suggests that children in some intermediate stages are able to produce adult forms when the syntactic context demands it, but otherwise they avoid doing so if the forms involve phonological alternations. The specific way that children satisfy OO-CORR constraints is to inflect forms so that the output is faithful to one specific slot of the paradigm, the base form. I show that children select the base form by considering both phonological informativeness and the availability of the surface forms, and try to achieve paradigm uniformity on the basis of the base form in their production. Assuming an initial bias to output-output correspondence constraints, I model the attested learning trajectories. The observed intermediate stages are correctly predicted, showing that the learning of certain alternations is facilitated when they share structural changes with other alternations. Artificial grammar learning experiments confirm that featural overlap of alternations facilitates the learning of the patterns. I propose a learning model that incorporates the attested learning biases.by Young Ah Do.Ph.D

The Development Of Glide Deletion In Seoul Korean: A Corpus And Articulatory Study

This dissertation investigates the pathways and causes of the development of glide deletion in Seoul Korean. Seoul provides fertile ground for studies of linguistic innovation in an urban setting since it has seen rapid historical, social and demographic changes in the twentieth century. The phenomenon under investigation is the variable deletion of the labiovelar glide /w/ found to be on the rise in Seoul Korean (Silva, 1991; Kang, 1997). I present two studies addressing variation and change at two different levels: a corpus study tracking the development of /w/-deletion at the phonological level and an articulatory study examining the phonetic aspect of this change. The corpus data are drawn from the sociolinguistic interviews with 48 native Seoul Koreans between 2015 and 2017. A trend comparison with the data from an earlier study of /w/- deletion (Kang, 1997) reveals that /w/-deletion in postconsonantal position has begun to retreat, while non-postconsonantal /w/-deletion has been rising vigorously. More importantly, the effect of preceding segment that used to be the strongest constraint on /w/-deletion has weakened over time. I conclude that /w/-deletion in Seoul Korean is being reanalyzed with the structural details being diluted over time. I analyze this weakening of the original pattern as the result of linguistic diffusion induced by a great influx of migrants into Seoul after the Korean War (1950-1953). In an articulatory study, ultrasound data of tongue movements and video data of lip rounding for the production of /w/ for three native Seoul Koreans in their 20s, 30s and 50s were analyzed using Optical Flow Analysis. I find that /w/ in Seoul Korean is subject to both gradient reduction and categorical deletion and that younger speakers exhibit a significantly larger articulatory gestures for /w/ after a bilabial than older generation, which is consistent with the pattern of phonological change found in the corpus study. This dissertation demonstrates the importance of using both corpus and articulatory data in the investigation of a change, finding the coexistence of gradient and categorical effects in segmental deletion processes. Finally, it advances our understanding of the outcome of migration-induced dialect contact in contemporary urban settings

Getting Past the Language Gap: Innovations in Machine Translation

In this chapter, we will be reviewing state of the art machine translation systems, and will discuss innovative methods for machine translation, highlighting the most promising techniques and applications. Machine translation (MT) has benefited from a revitalization in the last 10 years or so, after a period of relatively slow activity. In 2005 the field received a jumpstart when a powerful complete experimental package for building MT systems from scratch became freely available as a result of the unified efforts of the MOSES international consortium. Around the same time, hierarchical methods had been introduced by Chinese researchers, which allowed the introduction and use of syntactic information in translation modeling. Furthermore, the advances in the related field of computational linguistics, making off-the-shelf taggers and parsers readily available, helped give MT an additional boost. Yet there is still more progress to be made. For example, MT will be enhanced greatly when both syntax and semantics are on board: this still presents a major challenge though many advanced research groups are currently pursuing ways to meet this challenge head-on. The next generation of MT will consist of a collection of hybrid systems. It also augurs well for the mobile environment, as we look forward to more advanced and improved technologies that enable the working of Speech-To-Speech machine translation on hand-held devices, i.e. speech recognition and speech synthesis. We review all of these developments and point out in the final section some of the most promising research avenues for the future of MT

Research in the Language, Information and Computation Laboratory of the University of Pennsylvania

This report takes its name from the Computational Linguistics Feedback Forum (CLiFF), an informal discussion group for students and faculty. However the scope of the research covered in this report is broader than the title might suggest; this is the yearly report of the LINC Lab, the Language, Information and Computation Laboratory of the University of Pennsylvania. It may at first be hard to see the threads that bind together the work presented here, work by faculty, graduate students and postdocs in the Computer Science and Linguistics Departments, and the Institute for Research in Cognitive Science. It includes prototypical Natural Language fields such as: Combinatorial Categorial Grammars, Tree Adjoining Grammars, syntactic parsing and the syntax-semantics interface; but it extends to statistical methods, plan inference, instruction understanding, intonation, causal reasoning, free word order languages, geometric reasoning, medical informatics, connectionism, and language acquisition. Naturally, this introduction cannot spell out all the connections between these abstracts; we invite you to explore them on your own. In fact, with this issue it’s easier than ever to do so: this document is accessible on the “information superhighway”. Just call up http://www.cis.upenn.edu/~cliff-group/94/cliffnotes.html In addition, you can find many of the papers referenced in the CLiFF Notes on the net. Most can be obtained by following links from the authors’ abstracts in the web version of this report. The abstracts describe the researchers’ many areas of investigation, explain their shared concerns, and present some interesting work in Cognitive Science. We hope its new online format makes the CLiFF Notes a more useful and interesting guide to Computational Linguistics activity at Penn