37 research outputs found
MISPRONUNCIATION DETECTION AND DIAGNOSIS IN MANDARIN ACCENTED ENGLISH SPEECH
This work presents the development, implementation, and evaluation of a Mispronunciation Detection and Diagnosis (MDD) system, with application to pronunciation evaluation of Mandarin-accented English speech. A comprehensive detection and diagnosis of errors in the Electromagnetic Articulography corpus of Mandarin-Accented English (EMA-MAE) was performed by using the expert phonetic transcripts and an Automatic Speech Recognition (ASR) system. Articulatory features derived from the parallel kinematic data available in the EMA-MAE corpus were used to identify the most significant articulatory error patterns seen in L2 speakers during common mispronunciations. Using both acoustic and articulatory information, an ASR based Mispronunciation Detection and Diagnosis (MDD) system was built and evaluated across different feature combinations and Deep Neural Network (DNN) architectures. The MDD system captured mispronunciation errors with a detection accuracy of 82.4%, a diagnostic accuracy of 75.8% and a false rejection rate of 17.2%. The results demonstrate the advantage of using articulatory features in revealing the significant contributors of mispronunciation as well as improving the performance of MDD systems
Multi-View Multi-Task Representation Learning for Mispronunciation Detection
The disparity in phonology between learner's native (L1) and target (L2)
language poses a significant challenge for mispronunciation detection and
diagnosis (MDD) systems. This challenge is further intensified by lack of
annotated L2 data. This paper proposes a novel MDD architecture that exploits
multiple `views' of the same input data assisted by auxiliary tasks to learn
more distinctive phonetic representation in a low-resource setting. Using the
mono- and multilingual encoders, the model learn multiple views of the input,
and capture the sound properties across diverse languages and accents. These
encoded representations are further enriched by learning articulatory features
in a multi-task setup. Our reported results using the L2-ARCTIC data
outperformed the SOTA models, with a phoneme error rate reduction of 11.13% and
8.60% and absolute F1 score increase of 5.89%, and 2.49% compared to the
single-view mono- and multilingual systems, with a limited L2 dataset.Comment: 5 page
์๋๋ฐ์ํ๊ฐ-๋ฐ์์ค๋ฅ๊ฒ์ถ ํตํฉ ๋ชจ๋ธ
ํ์๋
ผ๋ฌธ(์์ฌ) -- ์์ธ๋ํ๊ต๋ํ์ : ์ธ๋ฌธ๋ํ ์ธ์ดํ๊ณผ, 2023. 8. ์ ๋ฏผํ.์ค์ฆ ์ฐ๊ตฌ์ ์ํ๋ฉด ๋น์์ด๋ฏผ ๋ฐ์ ํ๊ฐ์ ์์ด ์ ๋ฌธ ํ๊ฐ์๊ฐ ์ฑ์ ํ๋ ๋ฐ์ ์ ์์ ์์ ์ค๋ฅ ์ฌ์ด์ ์๊ด๊ด๊ณ๋ ๋งค์ฐ ๋๋ค. ๊ทธ๋ฌ๋ ๊ธฐ์กด์ ์ปดํจํฐ๊ธฐ๋ฐ๋ฐ์ํ๋ จ (Computer-assisted Pronunciation Training; CAPT) ์์คํ
์ ์๋๋ฐ์ํ๊ฐ (Automatic Pronunciation Assessment; APA) ๊ณผ์ ๋ฐ ๋ฐ์์ค๋ฅ๊ฒ์ถ (Mispronunciation Detection and Diagnosis; MDD) ๊ณผ์ ๋ฅผ ๋
๋ฆฝ์ ์ธ ๊ณผ์ ๋ก ์ทจ๊ธํ๋ฉฐ ๊ฐ ๋ชจ๋ธ์ ์ฑ๋ฅ์ ๊ฐ๋ณ์ ์ผ๋ก ํฅ์์ํค๋ ๊ฒ์๋ง ์ด์ ์ ๋์๋ค. ๋ณธ ์ฐ๊ตฌ์์๋ ๋ ๊ณผ์ ์ฌ์ด์ ๋์ ์๊ด๊ด๊ณ์ ์ฃผ๋ชฉ, ๋ค์ค์์
ํ์ต ๊ธฐ๋ฒ์ ํ์ฉํ์ฌ ์๋๋ฐ์ํ๊ฐ์ ๋ฐ์์ค๋ฅ๊ฒ์ถ ๊ณผ์ ๋ฅผ ๋์์ ํ๋ จํ๋ ์๋ก์ด ์ํคํ
์ฒ๋ฅผ ์ ์ํ๋ค. ๊ตฌ์ฒด์ ์ผ๋ก๋ APA ๊ณผ์ ๋ฅผ ์ํด ๊ต์ฐจ ์ํธ๋กํผ ์์คํจ์ ๋ฐ RMSE ์์คํจ์๋ฅผ ์คํํ๋ฉฐ, MDD ์์คํจ์๋ CTC ์์คํจ์๋ก ๊ณ ์ ๋๋ค. ๊ทผ๊ฐ ์ํฅ ๋ชจ๋ธ์ ์ฌ์ ํ๋ จ๋ ์๊ธฐ์ง๋ํ์ต๊ธฐ๋ฐ ๋ชจ๋ธ๋ก ํ๋ฉฐ, ์ด๋ ๋์ฑ ํ๋ถํ ์ํฅ ์ ๋ณด๋ฅผ ์ํด ๋ค์ค์์
ํ์ต์ ๊ฑฐ์น๊ธฐ ์ ์ ๋ถ์์ ์ผ๋ก ์์์ธ์์ ๋ํ์ฌ ๋ฏธ์ธ์กฐ์ ๋๊ธฐ๋ ํ๋ค. ์ํฅ ๋ชจ๋ธ๊ณผ ํจ๊ป ๋ฐ์์ ํฉ์ ์(Goodness-of-Pronunciation; GOP)๊ฐ ์ถ๊ฐ์ ์ธ ์
๋ ฅ์ผ๋ก ์ฌ์ฉ๋๋ค.
์คํ ๊ฒฐ๊ณผ, ํตํฉ ๋ชจ๋ธ์ด ๋จ์ผ ์๋๋ฐ์ํ๊ฐ ๋ฐ ๋ฐ์์ค๋ฅ๊ฒ์ถ ๋ชจ๋ธ๋ณด๋ค ๋งค์ฐ ๋์ ์ฑ๋ฅ์ ๋ณด์๋ค. ๊ตฌ์ฒด์ ์ผ๋ก๋ Speechocean762 ๋ฐ์ดํฐ์
์์ ์๋๋ฐ์ํ๊ฐ ๊ณผ์ ์ ์ฌ์ฉ๋ ๋ค ํญ๋ชฉ์ ์ ์๋ค์ ํ๊ท ํผ์ด์จ์๊ด๊ณ์๊ฐ 0.041 ์ฆ๊ฐํ์์ผ๋ฉฐ, ๋ฐ์์ค๋ฅ๊ฒ์ถ ๊ณผ์ ์ ๋ํด F1 ์ ์๊ฐ 0.003 ์ฆ๊ฐํ์๋ค. ํตํฉ ๋ชจ๋ธ์ ๋ํด ์๋๋ ์ํคํ
์ฒ ์ค์์๋, Robust Wav2vec2.0 ์ํฅ๋ชจ๋ธ๊ณผ ๋ฐ์์ ํฉ์ ์๋ฅผ ํ์ฉํ์ฌ RMSE/CTC ์์คํจ์๋ก ํ๋ จํ ๋ชจ๋ธ์ ์ฑ๋ฅ์ด ๊ฐ์ฅ ์ข์๋ค. ๋ชจ๋ธ์ ๋ถ์ํ ๊ฒฐ๊ณผ, ํตํฉ ๋ชจ๋ธ์ด ๊ฐ๋ณ ๋ชจ๋ธ์ ๋นํด ๋ถํฌ๊ฐ ๋ฎ์ ์ ์ ๋ฐ ๋ฐ์์ค๋ฅ๋ฅผ ๋ ์ ํํ๊ฒ ๊ตฌ๋ถํ์์์ ํ์ธํ ์ ์์๋ค.
ํฅ๋ฏธ๋กญ๊ฒ๋ ํตํฉ ๋ชจ๋ธ์ ์์ด ๊ฐ ํ์ ๊ณผ์ ๋ค์ ์ฑ๋ฅ ํฅ์ ์ ๋๋ ๊ฐ ๋ฐ์ ์ ์์ ๋ฐ์ ์ค๋ฅ ๋ ์ด๋ธ ์ฌ์ด์ ์๊ด๊ณ์ ํฌ๊ธฐ์ ๋น๋กํ์๋ค. ๋ ํตํฉ ๋ชจ๋ธ์ ์ฑ๋ฅ์ด ๊ฐ์ ๋ ์๋ก ๋ชจ๋ธ์ ์์ธก ๋ฐ์์ ์, ๊ทธ๋ฆฌ๊ณ ๋ชจ๋ธ์ ์์ธก ๋ฐ์์ค๋ฅ์ ๋ํ ์๊ด์ฑ์ด ๋์์ก๋ค. ๋ณธ ์ฐ๊ตฌ ๊ฒฐ๊ณผ๋ ํตํฉ ๋ชจ๋ธ์ด ๋ฐ์ ์ ์ ๋ฐ ์์ ์ค๋ฅ ์ฌ์ด์ ์ธ์ดํ์ ์๊ด์ฑ์ ํ์ฉํ์ฌ ์๋๋ฐ์ํ๊ฐ ๋ฐ ๋ฐ์์ค๋ฅ๊ฒ์ถ ๊ณผ์ ์ ์ฑ๋ฅ์ ํฅ์์์ผฐ์ผ๋ฉฐ, ๊ทธ ๊ฒฐ๊ณผ ํตํฉ ๋ชจ๋ธ์ด ์ ๋ฌธ ํ๊ฐ์๋ค์ ์ค์ ๋น์์ด๋ฏผ ํ๊ฐ์ ๋น์ทํ ์์์ ๋ค๋ค๋ ๊ฒ์ ๋ณด์ฌ์ค๋ค.Empirical studies report a strong correlation between pronunciation scores and mispronunciations in non-native speech assessments of human evaluators. However, the existing system of computer-assisted pronunciation training (CAPT) regards automatic pronunciation assessment (APA) and mispronunciation detection and diagnosis (MDD) as independent and focuses on individual performance improvement. Motivated by the correlation between two tasks, this study proposes a novel architecture that jointly tackles APA and MDD with a multi-task learning scheme to benefit both tasks. Specifically, APA loss is examined between cross-entropy and root mean square error (RMSE) criteria, and MDD loss is fixed to Connectionist Temporal Classification (CTC) criteria. For the backbone acoustic model, self-supervised model is used with an auxiliary fine-tuning on phone recognition before multi-task learning to leverage extra knowledge transfer. Goodness-of-Pronunciation (GOP) measure is given as an additional input along with the acoustic model.
The joint model significantly outperformed single-task learning counterparts, with a mean of 0.041 PCC increase for APA task on four multi-aspect scores and 0.003 F1 increase for MDD task on Speechocean762 dataset. For the joint model architecture, multi-task learning with RMSE and CTC criteria with raw Robust Wav2vec2.0 and GOP measure achieved the best performance. Analysis indicates that the joint model learned to distinguish scores with low distribution, and to better recognize mispronunciations as mispronunciations compared to single-task learning models.
Interestingly, the degree of the performance increase in each subtask for the joint model was proportional to the strength of the correlation between respective pronunciation score and mispronunciation labels, and the strength of the correlation between the model predictions also increased as the joint model achieved higher performances. The findings reveal that the joint model leveraged the linguistic correlation between pronunciation scores and mispronunciations to improve performances for APA and MDD tasks, and to show behaviors that follow the assessments of human experts.Chapter 1, Introduction 1
Chapter 2. Related work 5
Chapter 3. Methodology 17
Chapter 4. Results 28
Chapter 5. Discussion 47
Chapter 6. Conclusion 52
References 53
Appendix 60
๊ตญ๋ฌธ ์ด๋ก 65์
Phonological Level wav2vec2-based Mispronunciation Detection and Diagnosis Method
The automatic identification and analysis of pronunciation errors, known as
Mispronunciation Detection and Diagnosis (MDD) plays a crucial role in Computer
Aided Pronunciation Learning (CAPL) tools such as Second-Language (L2) learning
or speech therapy applications. Existing MDD methods relying on analysing
phonemes can only detect categorical errors of phonemes that have an adequate
amount of training data to be modelled. With the unpredictable nature of the
pronunciation errors of non-native or disordered speakers and the scarcity of
training datasets, it is unfeasible to model all types of mispronunciations.
Moreover, phoneme-level MDD approaches have a limited ability to provide
detailed diagnostic information about the error made. In this paper, we propose
a low-level MDD approach based on the detection of speech attribute features.
Speech attribute features break down phoneme production into elementary
components that are directly related to the articulatory system leading to more
formative feedback to the learner. We further propose a multi-label variant of
the Connectionist Temporal Classification (CTC) approach to jointly model the
non-mutually exclusive speech attributes using a single model. The pre-trained
wav2vec2 model was employed as a core model for the speech attribute detector.
The proposed method was applied to L2 speech corpora collected from English
learners from different native languages. The proposed speech attribute MDD
method was further compared to the traditional phoneme-level MDD and achieved a
significantly lower False Acceptance Rate (FAR), False Rejection Rate (FRR),
and Diagnostic Error Rate (DER) over all speech attributes compared to the
phoneme-level equivalent
Automatic Pronunciation Assessment -- A Review
Pronunciation assessment and its application in computer-aided pronunciation
training (CAPT) have seen impressive progress in recent years. With the rapid
growth in language processing and deep learning over the past few years, there
is a need for an updated review. In this paper, we review methods employed in
pronunciation assessment for both phonemic and prosodic. We categorize the main
challenges observed in prominent research trends, and highlight existing
limitations, and available resources. This is followed by a discussion of the
remaining challenges and possible directions for future work.Comment: 9 pages, accepted to EMNLP Finding
Automatic Screening of Childhood Speech Sound Disorders and Detection of Associated Pronunciation Errors
Speech disorders in children can affect their fluency and intelligibility. Delay in their diagnosis and treatment increases the risk of social impairment and learning disabilities. With the significant shortage of Speech and Language Pathologists (SLPs), there is an increasing interest in Computer-Aided Speech Therapy tools with automatic detection and diagnosis capability.
However, the scarcity and unreliable annotation of disordered child speech corpora along with the high acoustic variations in the child speech data has impeded the development of reliable automatic detection and diagnosis of childhood speech sound disorders. Therefore, this thesis investigates two types of detection systems that can be achieved with minimum dependency on annotated mispronounced speech data.
First, a novel approach that adopts paralinguistic features which represent the prosodic, spectral, and voice quality characteristics of the speech was proposed to perform segment- and subject-level classification of Typically Developing (TD) and Speech Sound Disordered (SSD) child speech using a binary Support Vector Machine (SVM) classifier. As paralinguistic features are both language- and content-independent, they can be extracted from an unannotated speech signal.
Second, a novel Mispronunciation Detection and Diagnosis (MDD) approach was introduced to detect the pronunciation errors made due to SSDs and provide low-level diagnostic information that can be used in constructing formative feedback and a detailed diagnostic report. Unlike existing MDD methods where detection and diagnosis are performed at the phoneme level, the proposed method achieved MDD at the speech attribute level, namely the manners and places of articulations. The speech attribute features describe the involved articulators and their interactions when making a speech sound allowing a low-level description of the pronunciation error to be provided. Two novel methods to model speech attributes are further proposed in this thesis, a frame-based (phoneme-alignment) method leveraging the Multi-Task Learning (MTL) criterion and training a separate model for each attribute, and an alignment-free jointly-learnt method based on the Connectionist Temporal Classification (CTC) sequence to sequence criterion.
The proposed techniques have been evaluated using standard and publicly accessible adult and child speech corpora, while the MDD method has been validated using L2 speech corpora
Automatic detection of accent and lexical pronunciation errors in spontaneous non-native English speech
Detecting individual pronunciation errors and diagnosing pronunciation error tendencies in a language learner based on their speech are important components of computer-aided language learning (CALL). The tasks of error detection and error tendency diagnosis become particularly challenging when the speech in question is spontaneous and particularly given the challenges posed by the inconsistency of human annotation of pronunciation errors. This paper presents an approach to these tasks by distinguishing between lexical errors, wherein the speaker does not know how a particular word is pronounced, and accent errors, wherein the candidate's speech exhibits consistent patterns of phone substitution, deletion and insertion. Three annotated corpora of non-native English speech by speakers of multiple L1s are analysed, the consistency of human annotation investigated and a method presented for detecting individual accent and lexical errors and diagnosing accent error tendencies at the speaker level