Automatic Pronunciation Assessment -- A Review

Pronunciation assessment and its application in computer-aided pronunciation training (CAPT) have seen impressive progress in recent years. With the rapid growth in language processing and deep learning over the past few years, there is a need for an updated review. In this paper, we review methods employed in pronunciation assessment for both phonemic and prosodic. We categorize the main challenges observed in prominent research trends, and highlight existing limitations, and available resources. This is followed by a discussion of the remaining challenges and possible directions for future work.Comment: 9 pages, accepted to EMNLP Finding

CAPT를 위한 발음 변이 분석 및 CycleGAN 기반 피드백 생성

학위논문(박사)--서울대학교 대학원 :인문대학 협동과정 인지과학전공,2020. 2. 정민화.Despite the growing popularity in learning Korean as a foreign language and the rapid development in language learning applications, the existing computer-assisted pronunciation training (CAPT) systems in Korean do not utilize linguistic characteristics of non-native Korean speech. Pronunciation variations in non-native speech are far more diverse than those observed in native speech, which may pose a difficulty in combining such knowledge in an automatic system. Moreover, most of the existing methods rely on feature extraction results from signal processing, prosodic analysis, and natural language processing techniques. Such methods entail limitations since they necessarily depend on finding the right features for the task and the extraction accuracies. This thesis presents a new approach for corrective feedback generation in a CAPT system, in which pronunciation variation patterns and linguistic correlates with accentedness are analyzed and combined with a deep neural network approach, so that feature engineering efforts are minimized while maintaining the linguistically important factors for the corrective feedback generation task. Investigations on non-native Korean speech characteristics in contrast with those of native speakers, and their correlation with accentedness judgement show that both segmental and prosodic variations are important factors in a Korean CAPT system. The present thesis argues that the feedback generation task can be interpreted as a style transfer problem, and proposes to evaluate the idea using generative adversarial network. A corrective feedback generation model is trained on 65,100 read utterances by 217 non-native speakers of 27 mother tongue backgrounds. The features are automatically learnt in an unsupervised way in an auxiliary classifier CycleGAN setting, in which the generator learns to map a foreign accented speech to native speech distributions. In order to inject linguistic knowledge into the network, an auxiliary classifier is trained so that the feedback also identifies the linguistic error types that were defined in the first half of the thesis. The proposed approach generates a corrected version the speech using the learners own voice, outperforming the conventional Pitch-Synchronous Overlap-and-Add method.외국어로서의 한국어 교육에 대한 관심이 고조되어 한국어 학습자의 수가 크게 증가하고 있으며, 음성언어처리 기술을 적용한 컴퓨터 기반 발음 교육(Computer-Assisted Pronunciation Training; CAPT) 어플리케이션에 대한 연구 또한 적극적으로 이루어지고 있다. 그럼에도 불구하고 현존하는 한국어 말하기 교육 시스템은 외국인의 한국어에 대한 언어학적 특징을 충분히 활용하지 않고 있으며, 최신 언어처리 기술 또한 적용되지 않고 있는 실정이다. 가능한 원인으로써는 외국인 발화 한국어 현상에 대한 분석이 충분하게 이루어지지 않았다는 점, 그리고 관련 연구가 있어도 이를 자동화된 시스템에 반영하기에는 고도화된 연구가 필요하다는 점이 있다. 뿐만 아니라 CAPT 기술 전반적으로는 신호처리, 운율 분석, 자연어처리 기법과 같은 특징 추출에 의존하고 있어서 적합한 특징을 찾고 이를 정확하게 추출하는 데에 많은 시간과 노력이 필요한 실정이다. 이는 최신 딥러닝 기반 언어처리 기술을 활용함으로써 이 과정 또한 발전의 여지가 많다는 바를 시사한다. 따라서 본 연구는 먼저 CAPT 시스템 개발에 있어 발음 변이 양상과 언어학적 상관관계를 분석하였다. 외국인 화자들의 낭독체 변이 양상과 한국어 원어민 화자들의 낭독체 변이 양상을 대조하고 주요한 변이를 확인한 후, 상관관계 분석을 통하여 의사소통에 영향을 미치는 중요도를 파악하였다. 그 결과, 종성 삭제와 3중 대립의 혼동, 초분절 관련 오류가 발생할 경우 피드백 생성에 우선적으로 반영하는 것이 필요하다는 것이 확인되었다. 교정된 피드백을 자동으로 생성하는 것은 CAPT 시스템의 중요한 과제 중 하나이다. 본 연구는 이 과제가 발화의 스타일 변화의 문제로 해석이 가능하다고 보았으며, 생성적 적대 신경망 (Cycle-consistent Generative Adversarial Network; CycleGAN) 구조에서 모델링하는 것을 제안하였다. GAN 네트워크의 생성모델은 비원어민 발화의 분포와 원어민 발화 분포의 매핑을 학습하며, Cycle consistency 손실함수를 사용함으로써 발화간 전반적인 구조를 유지함과 동시에 과도한 교정을 방지하였다. 별도의 특징 추출 과정이 없이 필요한 특징들이 CycleGAN 프레임워크에서 무감독 방법으로 스스로 학습되는 방법으로, 언어 확장이 용이한 방법이다. 언어학적 분석에서 드러난 주요한 변이들 간의 우선순위는 Auxiliary Classifier CycleGAN 구조에서 모델링하는 것을 제안하였다. 이 방법은 기존의 CycleGAN에 지식을 접목시켜 피드백 음성을 생성함과 동시에 해당 피드백이 어떤 유형의 오류인지 분류하는 문제를 수행한다. 이는 도메인 지식이 교정 피드백 생성 단계까지 유지되고 통제가 가능하다는 장점이 있다는 데에 그 의의가 있다. 본 연구에서 제안한 방법을 평가하기 위해서 27개의 모국어를 갖는 217명의 유의미 어휘 발화 65,100개로 피드백 자동 생성 모델을 훈련하고, 개선 여부 및 정도에 대한 지각 평가를 수행하였다. 제안된 방법을 사용하였을 때 학습자 본인의 목소리를 유지한 채 교정된 발음으로 변환하는 것이 가능하며, 전통적인 방법인 음높이 동기식 중첩가산 (Pitch-Synchronous Overlap-and-Add) 알고리즘을 사용하는 방법에 비해 상대 개선률 16.67%이 확인되었다.Chapter 1. Introduction 1 1.1. Motivation 1 1.1.1. An Overview of CAPT Systems 3 1.1.2. Survey of existing Korean CAPT Systems 5 1.2. Problem Statement 7 1.3. Thesis Structure 7 Chapter 2. Pronunciation Analysis of Korean Produced by Chinese 9 2.1. Comparison between Korean and Chinese 11 2.1.1. Phonetic and Syllable Structure Comparisons 11 2.1.2. Phonological Comparisons 14 2.2. Related Works 16 2.3. Proposed Analysis Method 19 2.3.1. Corpus 19 2.3.2. Transcribers and Agreement Rates 22 2.4. Salient Pronunciation Variations 22 2.4.1. Segmental Variation Patterns 22 2.4.1.1. Discussions 25 2.4.2. Phonological Variation Patterns 26 2.4.1.2. Discussions 27 2.5. Summary 29 Chapter 3. Correlation Analysis of Pronunciation Variations and Human Evaluation 30 3.1. Related Works 31 3.1.1. Criteria used in L2 Speech 31 3.1.2. Criteria used in L2 Korean Speech 32 3.2. Proposed Human Evaluation Method 36 3.2.1. Reading Prompt Design 36 3.2.2. Evaluation Criteria Design 37 3.2.3. Raters and Agreement Rates 40 3.3. Linguistic Factors Affecting L2 Korean Accentedness 41 3.3.1. Pearsons Correlation Analysis 41 3.3.2. Discussions 42 3.3.3. Implications for Automatic Feedback Generation 44 3.4. Summary 45 Chapter 4. Corrective Feedback Generation for CAPT 46 4.1. Related Works 46 4.1.1. Prosody Transplantation 47 4.1.2. Recent Speech Conversion Methods 49 4.1.3. Evaluation of Corrective Feedback 50 4.2. Proposed Method: Corrective Feedback as a Style Transfer 51 4.2.1. Speech Analysis at Spectral Domain 53 4.2.2. Self-imitative Learning 55 4.2.3. An Analogy: CAPT System and GAN Architecture 57 4.3. Generative Adversarial Networks 59 4.3.1. Conditional GAN 61 4.3.2. CycleGAN 62 4.4. Experiment 63 4.4.1. Corpus 64 4.4.2. Baseline Implementation 65 4.4.3. Adversarial Training Implementation 65 4.4.4. Spectrogram-to-Spectrogram Training 66 4.5. Results and Evaluation 69 4.5.1. Spectrogram Generation Results 69 4.5.2. Perceptual Evaluation 70 4.5.3. Discussions 72 4.6. Summary 74 Chapter 5. Integration of Linguistic Knowledge in an Auxiliary Classifier CycleGAN for Feedback Generation 75 5.1. Linguistic Class Selection 75 5.2. Auxiliary Classifier CycleGAN Design 77 5.3. Experiment and Results 80 5.3.1. Corpus 80 5.3.2. Feature Annotations 81 5.3.3. Experiment Setup 81 5.3.4. Results 82 5.4. Summary 84 Chapter 6. Conclusion 86 6.1. Thesis Results 86 6.2. Thesis Contributions 88 6.3. Recommendations for Future Work 89 Bibliography 91 Appendix 107 Abstract in Korean 117 Acknowledgments 120Docto

Methods and Effects of Shadowing Using Online Authentic Videos on L2 Acquisition of Mandarin Chinese Tones

Mandarin Chinese tones are notoriously difficult for second language (L2) learners. Previous research focuses on tone training methods that can help learners produce monosyllabic lexical tones, and studies about the production of multisyllabic lexical tones at the sentence level in spontaneous speech are limited. This study applies shadowing—a method where the learners repeat what they heard with as little delay as possible—to tone training and compares the effects of using authentic videos and textbook audios as shadowing materials for beginner L2 Mandarin learners’ tone improvement at the sentence level. Fourteen students in elementary Chinese classes at an American university participated in the tone training activity for four weeks. The participants in the “authentic video” group received authentic videos as their training materials, while the “textbook audio” group was trained with textbook audios. The participants shadowed the materials twice a week, six times per session, at home in their free time. Tone accuracy was rated by Mandarin native speakers according to the pre-test and the post-test consisting of a read-aloud task and a one-on-one conversation. Qualitative and quantitative surveys were conducted to analyze learners’ attitudes toward the shadowing activity and the materials. The results indicate that learners in both groups showed significant improvements in their accuracy in spontaneous speech with no significant differences between the two groups. As for learners’ attitudes, although the participants reported overall positive feedback on the shadowing activity regardless of the materials, authentic materials generated great interest from the participants and were more appealing to the learners. A strong correlation between learners’ confidence in speaking and flexibility of the activity was also found. Based on the finding, pedagogical implications are discussed, including how to select suitable materials and shadowing instructions. For example, educators could introduce textbook audios first and gradually add authentic materials. The findings provide Mandarin Chinese instructors an effective and engaging way to improve learners’ tone production in spontaneous speaking. Incorporating shadowing activities into class has great potential to encourage learners’ autonomy without occupying precious class time. The findings not only contribute to research on teaching Chinese as a second language and the related pedagogy but also shed light on the use of authentic materials in second language teaching and learning

An Error Analysis in Chinese Language Tones Pronunciation by Indonesian Students

This research aims to find out the Chinese language error pronunciation and the factors that cause pronunciation errors by Indonesian students. The paper shows some errors in pronouncing Chinese language tones by Indonesian students who study at Jiangxi Nanchang University China. This research focuses on Error Analysis in Chinese Finals Tone Pronunciations. As an Indonesian, the writer thinks of the need to analyze this error and hopes it will become a solution for some tone errors that appear in Indonesian Students' pronunciation. The tone in the Chinese language is very important, it determines the meaning of the words. The data sources of this research are Indonesian students who studied at Nanchang University, Jiangxi Province, China. The result shows that there are errors in pronouncing the Chinese tone. Some errors that appear when pronouncing a single tone are fewer than when pronouncing one word with a double tone or two-syllabic. Some causal factors of these errors came from some conditions, such as the influence of the mother language of the students, culture, social environment, etc

Etymology-based instructional system: using hypermedia to teach the recognition of Chinese characters to beginning learners of Chinese as a second language

As a logographic script, written Chinese conveys meaning directly from its graphic forms. Numerous researchers have suggested the pedagogical strategy of teaching Chinese writing by emphasizing the pictorial features of Chinese characters. This study was conducted to develop and to evaluate an instructional system in which instructional materials are based upon Chinese etymology and presented by hypermedia system;Thirty characters were selected, and pertinent information such as original forms, evolution of shapes, meanings, pronunciations,and applications were included. Presentation of the material, stored on a videodisc, was controlled by a HyperCard program;Twenty-eight beginning learner of Chinese as a second language in Taiwan, Republic of China, participated in the study as subjects; seven Chinese-language/literature instructors volunteered to evaluate the system. The relations of subjects\u27 linguistic backgrounds and attitudes towards the system, as well as their interests in learning Chinese characters before and after using the system, were investigated;Two significant results were found. Native speakers of alphabetic languages showed significantly more positive attitudes towards computer use than did native speakers of syllabic languages. Subjects\u27 interest in learning Chinese characters after using the system was significantly greater than was their interest before. But relations between attitude towards the system and a number of linguistic variables, with the exception of the native-language variable, were not significant;Verbal comments from subjects and Chinese language/literature instructors indicated the program\u27s emphasis on pictorial features and etymological analyses of Chinese characters was quite valuable in instruction of the Chinese writing system

Heritage and non-heritage Chinese students' perceptions of language learning strategy use

PhD ThesisEvidence has accumulated that learningChinese as a Foreign/Second Language (CFL/CSL) andas a Heritage Language (CHL) are not the same–it has been proposed, therefore, that different learning strategies and different pedagogy are needed for these two groups of learners. However, knowledge of CHL-specific learning strategies is incomplete, andTCHL-specific pedagogy is under development (Lynch, 2003a). The current comparative study investigates heritage and non-heritage Chinese students’ perceptions of language learning strategy use. It also explores correlations among learners’ individual variables, for example: their language proficiency level, motivations on learning Chinese, and language learning beliefs, together with other variables suchas gender, age, mother tongue, andtheir perceptions of learning strategy use. The study was conducted using both quantitative and qualitative methods. In the quantitative part, the variables of Chinese language learners and language learning strategy use were examined to explore the statistical relationship between the heritage and non-heritage Chinese language learners. Afterinformal pre-interviews and a pilot study, a sample of 278 Chinese language students (142 non-heritages students, 117 heritage students) from eighteen mainstream British universities, colleges, andsenior high schools weresurveyed. The survey was carried out through three structured pencil-and-paper questionnaires, including: a background questionnaire, Horwitz’s Beliefs about Language Learning Inventory (BALLI), and Oxford’s Strategy Inventory for Language Learning (SILL). Statistical analyses wereperformed using SPSS version 17 for Windows, such as Descriptive Analyses, Independent T-Tests, Pearson Product-moment Correlations, Factor Analyses, ANOVAs, Post-hoc Multiple Comparisons,ANCOVA, and Multiple Regressions. In the qualitative phase, the variableswere examined further within a semi-structured interview paradigm. The study addresses questions of both theoreticaland pedagogical significance, and makes specific proposals regarding ways to improve pedagogy forUK heritage and non-heritage Chinesestudents

A Sound Approach to Language Matters: In Honor of Ocke-Schwen Bohn

The contributions in this Festschrift were written by Ocke’s current and former PhD-students, colleagues and research collaborators. The Festschrift is divided into six sections, moving from the smallest building blocks of language, through gradually expanding objects of linguistic inquiry to the highest levels of description - all of which have formed a part of Ocke’s career, in connection with his teaching and/or his academic productions: “Segments”, “Perception of Accent”, “Between Sounds and Graphemes”, “Prosody”, “Morphology and Syntax” and “Second Language Acquisition”. Each one of these illustrates a sound approach to language matters

A conjecture of singing chinese in italian

This research intends to show how Chinese contemporary vocal works can be sung with the western lyrical singing technique, focusing on the pronunciation of the Italian language: The way of dealing with Chinese vowels and consonants in the pronunciation of articulation refers to the rules/principles of that presented in Italian language. The subject was inspired by Dr. A. Hirt’s lecture about singing English like Italians in 2011. In terms of rationality, to convey a sense yet also to approach the maximization of the rules of phonation (vowels) and articulation (consonants), researchers hypothesize that Chinese language (Mandarin) can be pronounced like the Italian language but in the setting of singing. This study will take into consideration from pieces of literature about singing technique to teaching, from viewpoints about articulation (in singing) of performers, to recordings and videos. We Believe it’s necessary to import(impart) knowledge about the singing of Chinese phonetics and linguistics, compared to Italian, the most traditional language for singing and the original language of a considerable number of masterpieces on what regards vocal repertoire,since they have been evolving from two completely families of languages

Using Translation Tools for L2-Learning in a Self-Regulated Environment

The goal of this dissertation is to investigate the use of Google Translate (GT), a free online translation software that includes translation, Text-to-Speech (TTS), and Automatic Speech Recognition (ASR) functionalities, for online self-regulated learning of Mandarin Chinese within an interactionist approach. It also explores how GT can be used for pedagogical purposes within a complex learning environment that combines computer-assisted language learning (CALL), online self-regulated learning (SRL), and informed by interactionist theories. This dissertation begins with a review of the literature around the importance of interaction in language learning, followed by how a fully online interactionist approach using GT as a language partner and interlocutor allows learners to practice language use whenever and wherever they please (Chapter 1). Next, the dissertation contains three manuscript-based chapters (Chapters 2, 3, and 4) and a concluding chapter (Chapter 5). Each manuscript explores one aspect of using GT for pedagogical purposes by addressing the following overarching research questions: 1) Can GT provide the necessary computer-assisted interaction including input, output, and feedback to promote second language (L2) learning (Manuscripts 1 and 2), and 2) Are learners willing and able to use GT in an online, self-regulated environment for the learning of Mandarin and its associated tones? (Manuscript 3). The first manuscript investigates the use of GT’s TTS as a source of Mandarin Chinese input when compared with a native speaker in terms of Intelligibility (how much is understood), Comprehensibility (how challenging something is to be understood), and Naturalness (how much does a synthesized voice differ from a human speaker). The second manuscript further explores GT’s ability to interact with a human interlocutor by investigating how much Mandarin Chinese speech can GT recognize at various language levels (intermediate, advanced, and native) and whether it can provide transcriptions accurate enough to be used as feedback by the language learner. The third and final manuscript investigate the pedagogical feasibility of using GT and its built-in features (translation, TTS, ASR) in an online, self-regulated environment by exploring how a small group of participants acquire language features, develop self-regulated learning strategies, and perceive the GT-enhanced pedagogical environment as a venue for language learning. This dissertation will contribute to the literature around using translation, TTS, and ASR software for language learning, as well as interaction theories, SRL, CALL, and the acquisition of Mandarin Chinese in general. This research innovates on existing online language learning research and interactionist approaches by positioning GT as an interlocutor despite its intrinsic limitations (it is after all, not a human). This dissertation will further help guide future research into how human beings can interact with computers for language learning and will only become more relevant as translation, TTS, and ASR software becomes more intelligent, capable, and life-like