MISPRONUNCIATION DETECTION AND DIAGNOSIS IN MANDARIN ACCENTED ENGLISH SPEECH

This work presents the development, implementation, and evaluation of a Mispronunciation Detection and Diagnosis (MDD) system, with application to pronunciation evaluation of Mandarin-accented English speech. A comprehensive detection and diagnosis of errors in the Electromagnetic Articulography corpus of Mandarin-Accented English (EMA-MAE) was performed by using the expert phonetic transcripts and an Automatic Speech Recognition (ASR) system. Articulatory features derived from the parallel kinematic data available in the EMA-MAE corpus were used to identify the most significant articulatory error patterns seen in L2 speakers during common mispronunciations. Using both acoustic and articulatory information, an ASR based Mispronunciation Detection and Diagnosis (MDD) system was built and evaluated across different feature combinations and Deep Neural Network (DNN) architectures. The MDD system captured mispronunciation errors with a detection accuracy of 82.4%, a diagnostic accuracy of 75.8% and a false rejection rate of 17.2%. The results demonstrate the advantage of using articulatory features in revealing the significant contributors of mispronunciation as well as improving the performance of MDD systems

CAPT를 위한 발음 변이 분석 및 CycleGAN 기반 피드백 생성

학위논문(박사)--서울대학교 대학원 :인문대학 협동과정 인지과학전공,2020. 2. 정민화.Despite the growing popularity in learning Korean as a foreign language and the rapid development in language learning applications, the existing computer-assisted pronunciation training (CAPT) systems in Korean do not utilize linguistic characteristics of non-native Korean speech. Pronunciation variations in non-native speech are far more diverse than those observed in native speech, which may pose a difficulty in combining such knowledge in an automatic system. Moreover, most of the existing methods rely on feature extraction results from signal processing, prosodic analysis, and natural language processing techniques. Such methods entail limitations since they necessarily depend on finding the right features for the task and the extraction accuracies. This thesis presents a new approach for corrective feedback generation in a CAPT system, in which pronunciation variation patterns and linguistic correlates with accentedness are analyzed and combined with a deep neural network approach, so that feature engineering efforts are minimized while maintaining the linguistically important factors for the corrective feedback generation task. Investigations on non-native Korean speech characteristics in contrast with those of native speakers, and their correlation with accentedness judgement show that both segmental and prosodic variations are important factors in a Korean CAPT system. The present thesis argues that the feedback generation task can be interpreted as a style transfer problem, and proposes to evaluate the idea using generative adversarial network. A corrective feedback generation model is trained on 65,100 read utterances by 217 non-native speakers of 27 mother tongue backgrounds. The features are automatically learnt in an unsupervised way in an auxiliary classifier CycleGAN setting, in which the generator learns to map a foreign accented speech to native speech distributions. In order to inject linguistic knowledge into the network, an auxiliary classifier is trained so that the feedback also identifies the linguistic error types that were defined in the first half of the thesis. The proposed approach generates a corrected version the speech using the learners own voice, outperforming the conventional Pitch-Synchronous Overlap-and-Add method.외국어로서의 한국어 교육에 대한 관심이 고조되어 한국어 학습자의 수가 크게 증가하고 있으며, 음성언어처리 기술을 적용한 컴퓨터 기반 발음 교육(Computer-Assisted Pronunciation Training; CAPT) 어플리케이션에 대한 연구 또한 적극적으로 이루어지고 있다. 그럼에도 불구하고 현존하는 한국어 말하기 교육 시스템은 외국인의 한국어에 대한 언어학적 특징을 충분히 활용하지 않고 있으며, 최신 언어처리 기술 또한 적용되지 않고 있는 실정이다. 가능한 원인으로써는 외국인 발화 한국어 현상에 대한 분석이 충분하게 이루어지지 않았다는 점, 그리고 관련 연구가 있어도 이를 자동화된 시스템에 반영하기에는 고도화된 연구가 필요하다는 점이 있다. 뿐만 아니라 CAPT 기술 전반적으로는 신호처리, 운율 분석, 자연어처리 기법과 같은 특징 추출에 의존하고 있어서 적합한 특징을 찾고 이를 정확하게 추출하는 데에 많은 시간과 노력이 필요한 실정이다. 이는 최신 딥러닝 기반 언어처리 기술을 활용함으로써 이 과정 또한 발전의 여지가 많다는 바를 시사한다. 따라서 본 연구는 먼저 CAPT 시스템 개발에 있어 발음 변이 양상과 언어학적 상관관계를 분석하였다. 외국인 화자들의 낭독체 변이 양상과 한국어 원어민 화자들의 낭독체 변이 양상을 대조하고 주요한 변이를 확인한 후, 상관관계 분석을 통하여 의사소통에 영향을 미치는 중요도를 파악하였다. 그 결과, 종성 삭제와 3중 대립의 혼동, 초분절 관련 오류가 발생할 경우 피드백 생성에 우선적으로 반영하는 것이 필요하다는 것이 확인되었다. 교정된 피드백을 자동으로 생성하는 것은 CAPT 시스템의 중요한 과제 중 하나이다. 본 연구는 이 과제가 발화의 스타일 변화의 문제로 해석이 가능하다고 보았으며, 생성적 적대 신경망 (Cycle-consistent Generative Adversarial Network; CycleGAN) 구조에서 모델링하는 것을 제안하였다. GAN 네트워크의 생성모델은 비원어민 발화의 분포와 원어민 발화 분포의 매핑을 학습하며, Cycle consistency 손실함수를 사용함으로써 발화간 전반적인 구조를 유지함과 동시에 과도한 교정을 방지하였다. 별도의 특징 추출 과정이 없이 필요한 특징들이 CycleGAN 프레임워크에서 무감독 방법으로 스스로 학습되는 방법으로, 언어 확장이 용이한 방법이다. 언어학적 분석에서 드러난 주요한 변이들 간의 우선순위는 Auxiliary Classifier CycleGAN 구조에서 모델링하는 것을 제안하였다. 이 방법은 기존의 CycleGAN에 지식을 접목시켜 피드백 음성을 생성함과 동시에 해당 피드백이 어떤 유형의 오류인지 분류하는 문제를 수행한다. 이는 도메인 지식이 교정 피드백 생성 단계까지 유지되고 통제가 가능하다는 장점이 있다는 데에 그 의의가 있다. 본 연구에서 제안한 방법을 평가하기 위해서 27개의 모국어를 갖는 217명의 유의미 어휘 발화 65,100개로 피드백 자동 생성 모델을 훈련하고, 개선 여부 및 정도에 대한 지각 평가를 수행하였다. 제안된 방법을 사용하였을 때 학습자 본인의 목소리를 유지한 채 교정된 발음으로 변환하는 것이 가능하며, 전통적인 방법인 음높이 동기식 중첩가산 (Pitch-Synchronous Overlap-and-Add) 알고리즘을 사용하는 방법에 비해 상대 개선률 16.67%이 확인되었다.Chapter 1. Introduction 1 1.1. Motivation 1 1.1.1. An Overview of CAPT Systems 3 1.1.2. Survey of existing Korean CAPT Systems 5 1.2. Problem Statement 7 1.3. Thesis Structure 7 Chapter 2. Pronunciation Analysis of Korean Produced by Chinese 9 2.1. Comparison between Korean and Chinese 11 2.1.1. Phonetic and Syllable Structure Comparisons 11 2.1.2. Phonological Comparisons 14 2.2. Related Works 16 2.3. Proposed Analysis Method 19 2.3.1. Corpus 19 2.3.2. Transcribers and Agreement Rates 22 2.4. Salient Pronunciation Variations 22 2.4.1. Segmental Variation Patterns 22 2.4.1.1. Discussions 25 2.4.2. Phonological Variation Patterns 26 2.4.1.2. Discussions 27 2.5. Summary 29 Chapter 3. Correlation Analysis of Pronunciation Variations and Human Evaluation 30 3.1. Related Works 31 3.1.1. Criteria used in L2 Speech 31 3.1.2. Criteria used in L2 Korean Speech 32 3.2. Proposed Human Evaluation Method 36 3.2.1. Reading Prompt Design 36 3.2.2. Evaluation Criteria Design 37 3.2.3. Raters and Agreement Rates 40 3.3. Linguistic Factors Affecting L2 Korean Accentedness 41 3.3.1. Pearsons Correlation Analysis 41 3.3.2. Discussions 42 3.3.3. Implications for Automatic Feedback Generation 44 3.4. Summary 45 Chapter 4. Corrective Feedback Generation for CAPT 46 4.1. Related Works 46 4.1.1. Prosody Transplantation 47 4.1.2. Recent Speech Conversion Methods 49 4.1.3. Evaluation of Corrective Feedback 50 4.2. Proposed Method: Corrective Feedback as a Style Transfer 51 4.2.1. Speech Analysis at Spectral Domain 53 4.2.2. Self-imitative Learning 55 4.2.3. An Analogy: CAPT System and GAN Architecture 57 4.3. Generative Adversarial Networks 59 4.3.1. Conditional GAN 61 4.3.2. CycleGAN 62 4.4. Experiment 63 4.4.1. Corpus 64 4.4.2. Baseline Implementation 65 4.4.3. Adversarial Training Implementation 65 4.4.4. Spectrogram-to-Spectrogram Training 66 4.5. Results and Evaluation 69 4.5.1. Spectrogram Generation Results 69 4.5.2. Perceptual Evaluation 70 4.5.3. Discussions 72 4.6. Summary 74 Chapter 5. Integration of Linguistic Knowledge in an Auxiliary Classifier CycleGAN for Feedback Generation 75 5.1. Linguistic Class Selection 75 5.2. Auxiliary Classifier CycleGAN Design 77 5.3. Experiment and Results 80 5.3.1. Corpus 80 5.3.2. Feature Annotations 81 5.3.3. Experiment Setup 81 5.3.4. Results 82 5.4. Summary 84 Chapter 6. Conclusion 86 6.1. Thesis Results 86 6.2. Thesis Contributions 88 6.3. Recommendations for Future Work 89 Bibliography 91 Appendix 107 Abstract in Korean 117 Acknowledgments 120Docto

The Processing of Accented Speech

This thesis examines the processing of accented speech in both infants and adults. Accents provide a natural and reasonably consistent form of inter-speaker variation in the speech signal, but it is not yet clear exactly what processes are used to normalise this form of variation, or when and how those processes develop. Two adult studies use ERP data to examine differences between the online processing of regional- and foreign-accented speech as compared to a baseline consisting of the listeners’ home accent. These studies demonstrate that the two types of accents recruit normalisation processes which are qualitatively, and not just quantitatively, different. This provided support for the hypothesis that foreign and regional accents require different mechanisms to normalise accent-based variation (Adank et al., 2009, Floccia et al., 2009), rather than for the hypothesis that different types of accents are normalised according to their perceptual distance from the listener’s own accent (Clarke & Garrett, 2004). They also provide support for the Abstract entry approach to lexical storage of variant forms, which suggests that variant forms undergo a process of prelexical normalisation, allowing access to a canonical lexical entry (Pallier et al., 2001), rather than for the Exemplar-based approach, which suggests that variant word-forms are individually represented in the lexicon (Johnson, 1997). Two further studies examined how infants segment words from continuous speech when presented with accented speakers. The first of these includes a set of behavioural experiments, which highlight some methodological issues in the existing literature and offer some potential explanations for conflicting evidence about the age at which infants are able to segment speech. The second uses ERP data to investigate segmentation within and across accents, and provides neurophysiological evidence that 11-month-olds are able to distinguish newly-segmented words at the auditory level even within a foreign accent, or across accents, but that they are more able to treat new word-forms as word-like in a familiar accent than a foreign accent

The Role of Socioindexical Expectation in Speech Perception.

Listeners extract cues to speaker identity from the speech stream. Recent evidence suggests that listeners will also perceive these socioindexical cues, even if absent, when primed to expect them. Most researchers interpret these findings as evidence for exemplar models of speech perception (Niedzielski, 1999; Hay et al., 2006b; Staum Casasanto, 2009a). At least one early line of research, however, attributes the influence of socioindexical knowledge on speech perception to listeners’ negative bias (Rubin, 1992). A series of three experiments with experienced and inexperienced listeners investigates the use of socioindexical expectation during speech perception. The first experiment, a yes/no accent identification task, reveals that listeners, whether experienced or inexperienced with Chinese-accented English, are capable of judging the authenticity of a non-native accent. Experienced listeners are significantly more accurate and inexperienced listeners are significantly more likely to rate an imitated accent as authentic –suggesting they depend more heavily on stereotypical features. Experiment 2 addresses whether listeners can use socioindexical expectations to enhance speech perception. Both experienced and inexperienced listeners were significantly better at transcribing Chinese-accented sentences in noise when presented with an Asian face than when presented either with a silhouette or a Caucasian face. This result suggests that the negative bias hypothesis can not be correct; listeners can use socioindexical cues to enhance speech perception. Experiment 3 used eye-tracking to investigate the time course of the influence of socioindexical expectation. Inexperienced listeners hearing a Standard American English voice showed no significant difference in fixation latencies when presented either with an Asian or Caucasian face. Listeners shown an Asian face showed significantly longer dwell times to the target image late in the trial, however. This result reinforces the finding that the negative bias hypothesis can not be correct but is also not consistent with an exemplar based account in which socioindexical expectation pre-activates groups of socially labeled exemplars (Johnson, 2006). These results point to the need for more natural –more linguistic– tasks in the investigation of socioindexical speech perception and the need to look closely at time course to better understand the role of socioindexical expectation in speech perception.Ph.D.LinguisticsUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/89684/1/clunis_1.pd

Accent intelligibility across native and non-native accent pairings:investigating links with electrophysiological measures of word recognition

The intelligibility of accented speech in noise depends on the interaction of the accents of the talker and the listener. However, it is not yet clear how this influence arises. Accent familiarity is commonly proposed to be a major contributor to accent intelligibility, but recent evidence suggests that the similarity between talker and listener accents may also be able to account for accent intelligibility across talker-listener pairings. In addition, differences in accent intelligibility are also often only found in the presence of other adverse conditions, so it is not clear if the talker-listener pairing also influences speech processing in quiet conditions. This research had two main aims; to further investigate the relationship between accent similarity and intelligibility, and to use online EEG methods to explore the possible presence of talker-listener pairing related differences on speech perception in quiet conditions. English and Spanish listeners listened to Standard Southern British English (SSBE), Glaswegian English (GE) and Spanish-accented English (SpE) in a speech-in-noise recognition task, and also completed an event-related potential (ERP) task to elicit the PMN and N400 responses. Accent similarity was measured using the ACCDIST metric. Results showed the same (or extremely similar) patterns in accent intelligibility and accent similarity for both listener groups, giving further support to the hypothesis that accent similarity can contribute to the level of intelligibility of an accent within a talker-listener pairing. ERP data also suggest that speech processing in quiet is influenced by the talker-listener pairing. The PMN, which relates to phonological processing, seems particularly dependent on a match between talker and listener accent, but the more semantic N400 showed some flexibility in the ability to process accented speech

Adaptation Algorithms for Neural Network-Based Speech Recognition: An Overview

We present a structured overview of adaptation algorithms for neural network-based speech recognition, considering both hybrid hidden Markov model / neural network systems and end-to-end neural network systems, with a focus on speaker adaptation, domain adaptation, and accent adaptation. The overview characterizes adaptation algorithms as based on embeddings, model parameter adaptation, or data augmentation. We present a meta-analysis of the performance of speech recognition adaptation algorithms, based on relative error rate reductions as reported in the literature.Comment: Submitted to IEEE Open Journal of Signal Processing. 30 pages, 27 figure

Exploring social attitudes toward second language speakers of English across Canada

Nearly a fifth of Canada’s population is represented by people from other countries, most of whom speak languages other than English or French. Though there has been some exploration of social attitudes toward these ethnolinguistic groups, there have been no multi-city investigations of social attitudes, particularly toward second language (L2) speakers, in major cities where immigrant populations are most concentrated. Furthermore, there have been few studies that present speech samples alongside images suggesting speaker ethnicity and/or religious affiliation. Therefore, this dissertation explores ratings of L2 speakers across multiple dimensions and considers the role of social attitudes and social network exposure in formation of those judgments. Study 1 explored how residents of Calgary and Montreal judge the comprehensibility and accentedness of L2 speech in audio-only and audiovisual conditions and whether those judgments are associated with residents’ overall social attitudes toward immigrants. There were no context or image effects, but differences emerged among ratings of certain language groups, as well as raters’ general attitudes toward immigrants. Ultimately, raters’ attitudes were not associated with their ratings of L2 speech. Study 2 explored how native-born residents of Canada judge L2 speakers’ intelligence, friendliness, and trustworthiness and investigated how those judgments might be related to residents’ general social attitudes toward immigrants. No significant differences emerged between speech samples presented as audio-only versus with a nonreligious or religious image. However, there was a clear hierarchy in how specific language groups were evaluated. Social attitudes questionnaire responses revealed generally positive attitudes toward immigrants. Ultimately, those attitudes had weak relationships with rater judgments of L2 speakers’ intelligence, friendliness, and trustworthiness. Study 3 explored how native-born residents of Montreal and Calgary compared in judgments of L2 speaker citizenship and how those judgments might be related to raters’ L2 social networks. No differences based on image condition or context surfaced in judgments of L2 speaker citizenship. However, specific language groups differed in how their citizenship status was perceived. There were also between-context differences in native-born residents’ interactions with L2 speakers. Ultimately, social network characteristics had no influence on citizenship ratings in either context