305 research outputs found
A computational model for studying L1โs effect on L2 speech learning
abstract: Much evidence has shown that first language (L1) plays an important role in the formation of L2 phonological system during second language (L2) learning process. This combines with the fact that different L1s have distinct phonological patterns to indicate the diverse L2 speech learning outcomes for speakers from different L1 backgrounds. This dissertation hypothesizes that phonological distances between accented speech and speakers' L1 speech are also correlated with perceived accentedness, and the correlations are negative for some phonological properties. Moreover, contrastive phonological distinctions between L1s and L2 will manifest themselves in the accented speech produced by speaker from these L1s. To test the hypotheses, this study comes up with a computational model to analyze the accented speech properties in both segmental (short-term speech measurements on short-segment or phoneme level) and suprasegmental (long-term speech measurements on word, long-segment, or sentence level) feature space. The benefit of using a computational model is that it enables quantitative analysis of L1's effect on accent in terms of different phonological properties. The core parts of this computational model are feature extraction schemes to extract pronunciation and prosody representation of accented speech based on existing techniques in speech processing field. Correlation analysis on both segmental and suprasegmental feature space is conducted to look into the relationship between acoustic measurements related to L1s and perceived accentedness across several L1s. Multiple regression analysis is employed to investigate how the L1's effect impacts the perception of foreign accent, and how accented speech produced by speakers from different L1s behaves distinctly on segmental and suprasegmental feature spaces. Results unveil the potential application of the methodology in this study to provide quantitative analysis of accented speech, and extend current studies in L2 speech learning theory to large scale. Practically, this study further shows that the computational model proposed in this study can benefit automatic accentedness evaluation system by adding features related to speakers' L1s.Dissertation/ThesisDoctoral Dissertation Speech and Hearing Science 201
Phonological Level wav2vec2-based Mispronunciation Detection and Diagnosis Method
The automatic identification and analysis of pronunciation errors, known as
Mispronunciation Detection and Diagnosis (MDD) plays a crucial role in Computer
Aided Pronunciation Learning (CAPL) tools such as Second-Language (L2) learning
or speech therapy applications. Existing MDD methods relying on analysing
phonemes can only detect categorical errors of phonemes that have an adequate
amount of training data to be modelled. With the unpredictable nature of the
pronunciation errors of non-native or disordered speakers and the scarcity of
training datasets, it is unfeasible to model all types of mispronunciations.
Moreover, phoneme-level MDD approaches have a limited ability to provide
detailed diagnostic information about the error made. In this paper, we propose
a low-level MDD approach based on the detection of speech attribute features.
Speech attribute features break down phoneme production into elementary
components that are directly related to the articulatory system leading to more
formative feedback to the learner. We further propose a multi-label variant of
the Connectionist Temporal Classification (CTC) approach to jointly model the
non-mutually exclusive speech attributes using a single model. The pre-trained
wav2vec2 model was employed as a core model for the speech attribute detector.
The proposed method was applied to L2 speech corpora collected from English
learners from different native languages. The proposed speech attribute MDD
method was further compared to the traditional phoneme-level MDD and achieved a
significantly lower False Acceptance Rate (FAR), False Rejection Rate (FRR),
and Diagnostic Error Rate (DER) over all speech attributes compared to the
phoneme-level equivalent
Automatic Pronunciation Assessment -- A Review
Pronunciation assessment and its application in computer-aided pronunciation
training (CAPT) have seen impressive progress in recent years. With the rapid
growth in language processing and deep learning over the past few years, there
is a need for an updated review. In this paper, we review methods employed in
pronunciation assessment for both phonemic and prosodic. We categorize the main
challenges observed in prominent research trends, and highlight existing
limitations, and available resources. This is followed by a discussion of the
remaining challenges and possible directions for future work.Comment: 9 pages, accepted to EMNLP Finding
์๋๋ฐ์ํ๊ฐ-๋ฐ์์ค๋ฅ๊ฒ์ถ ํตํฉ ๋ชจ๋ธ
ํ์๋
ผ๋ฌธ(์์ฌ) -- ์์ธ๋ํ๊ต๋ํ์ : ์ธ๋ฌธ๋ํ ์ธ์ดํ๊ณผ, 2023. 8. ์ ๋ฏผํ.์ค์ฆ ์ฐ๊ตฌ์ ์ํ๋ฉด ๋น์์ด๋ฏผ ๋ฐ์ ํ๊ฐ์ ์์ด ์ ๋ฌธ ํ๊ฐ์๊ฐ ์ฑ์ ํ๋ ๋ฐ์ ์ ์์ ์์ ์ค๋ฅ ์ฌ์ด์ ์๊ด๊ด๊ณ๋ ๋งค์ฐ ๋๋ค. ๊ทธ๋ฌ๋ ๊ธฐ์กด์ ์ปดํจํฐ๊ธฐ๋ฐ๋ฐ์ํ๋ จ (Computer-assisted Pronunciation Training; CAPT) ์์คํ
์ ์๋๋ฐ์ํ๊ฐ (Automatic Pronunciation Assessment; APA) ๊ณผ์ ๋ฐ ๋ฐ์์ค๋ฅ๊ฒ์ถ (Mispronunciation Detection and Diagnosis; MDD) ๊ณผ์ ๋ฅผ ๋
๋ฆฝ์ ์ธ ๊ณผ์ ๋ก ์ทจ๊ธํ๋ฉฐ ๊ฐ ๋ชจ๋ธ์ ์ฑ๋ฅ์ ๊ฐ๋ณ์ ์ผ๋ก ํฅ์์ํค๋ ๊ฒ์๋ง ์ด์ ์ ๋์๋ค. ๋ณธ ์ฐ๊ตฌ์์๋ ๋ ๊ณผ์ ์ฌ์ด์ ๋์ ์๊ด๊ด๊ณ์ ์ฃผ๋ชฉ, ๋ค์ค์์
ํ์ต ๊ธฐ๋ฒ์ ํ์ฉํ์ฌ ์๋๋ฐ์ํ๊ฐ์ ๋ฐ์์ค๋ฅ๊ฒ์ถ ๊ณผ์ ๋ฅผ ๋์์ ํ๋ จํ๋ ์๋ก์ด ์ํคํ
์ฒ๋ฅผ ์ ์ํ๋ค. ๊ตฌ์ฒด์ ์ผ๋ก๋ APA ๊ณผ์ ๋ฅผ ์ํด ๊ต์ฐจ ์ํธ๋กํผ ์์คํจ์ ๋ฐ RMSE ์์คํจ์๋ฅผ ์คํํ๋ฉฐ, MDD ์์คํจ์๋ CTC ์์คํจ์๋ก ๊ณ ์ ๋๋ค. ๊ทผ๊ฐ ์ํฅ ๋ชจ๋ธ์ ์ฌ์ ํ๋ จ๋ ์๊ธฐ์ง๋ํ์ต๊ธฐ๋ฐ ๋ชจ๋ธ๋ก ํ๋ฉฐ, ์ด๋ ๋์ฑ ํ๋ถํ ์ํฅ ์ ๋ณด๋ฅผ ์ํด ๋ค์ค์์
ํ์ต์ ๊ฑฐ์น๊ธฐ ์ ์ ๋ถ์์ ์ผ๋ก ์์์ธ์์ ๋ํ์ฌ ๋ฏธ์ธ์กฐ์ ๋๊ธฐ๋ ํ๋ค. ์ํฅ ๋ชจ๋ธ๊ณผ ํจ๊ป ๋ฐ์์ ํฉ์ ์(Goodness-of-Pronunciation; GOP)๊ฐ ์ถ๊ฐ์ ์ธ ์
๋ ฅ์ผ๋ก ์ฌ์ฉ๋๋ค.
์คํ ๊ฒฐ๊ณผ, ํตํฉ ๋ชจ๋ธ์ด ๋จ์ผ ์๋๋ฐ์ํ๊ฐ ๋ฐ ๋ฐ์์ค๋ฅ๊ฒ์ถ ๋ชจ๋ธ๋ณด๋ค ๋งค์ฐ ๋์ ์ฑ๋ฅ์ ๋ณด์๋ค. ๊ตฌ์ฒด์ ์ผ๋ก๋ Speechocean762 ๋ฐ์ดํฐ์
์์ ์๋๋ฐ์ํ๊ฐ ๊ณผ์ ์ ์ฌ์ฉ๋ ๋ค ํญ๋ชฉ์ ์ ์๋ค์ ํ๊ท ํผ์ด์จ์๊ด๊ณ์๊ฐ 0.041 ์ฆ๊ฐํ์์ผ๋ฉฐ, ๋ฐ์์ค๋ฅ๊ฒ์ถ ๊ณผ์ ์ ๋ํด F1 ์ ์๊ฐ 0.003 ์ฆ๊ฐํ์๋ค. ํตํฉ ๋ชจ๋ธ์ ๋ํด ์๋๋ ์ํคํ
์ฒ ์ค์์๋, Robust Wav2vec2.0 ์ํฅ๋ชจ๋ธ๊ณผ ๋ฐ์์ ํฉ์ ์๋ฅผ ํ์ฉํ์ฌ RMSE/CTC ์์คํจ์๋ก ํ๋ จํ ๋ชจ๋ธ์ ์ฑ๋ฅ์ด ๊ฐ์ฅ ์ข์๋ค. ๋ชจ๋ธ์ ๋ถ์ํ ๊ฒฐ๊ณผ, ํตํฉ ๋ชจ๋ธ์ด ๊ฐ๋ณ ๋ชจ๋ธ์ ๋นํด ๋ถํฌ๊ฐ ๋ฎ์ ์ ์ ๋ฐ ๋ฐ์์ค๋ฅ๋ฅผ ๋ ์ ํํ๊ฒ ๊ตฌ๋ถํ์์์ ํ์ธํ ์ ์์๋ค.
ํฅ๋ฏธ๋กญ๊ฒ๋ ํตํฉ ๋ชจ๋ธ์ ์์ด ๊ฐ ํ์ ๊ณผ์ ๋ค์ ์ฑ๋ฅ ํฅ์ ์ ๋๋ ๊ฐ ๋ฐ์ ์ ์์ ๋ฐ์ ์ค๋ฅ ๋ ์ด๋ธ ์ฌ์ด์ ์๊ด๊ณ์ ํฌ๊ธฐ์ ๋น๋กํ์๋ค. ๋ ํตํฉ ๋ชจ๋ธ์ ์ฑ๋ฅ์ด ๊ฐ์ ๋ ์๋ก ๋ชจ๋ธ์ ์์ธก ๋ฐ์์ ์, ๊ทธ๋ฆฌ๊ณ ๋ชจ๋ธ์ ์์ธก ๋ฐ์์ค๋ฅ์ ๋ํ ์๊ด์ฑ์ด ๋์์ก๋ค. ๋ณธ ์ฐ๊ตฌ ๊ฒฐ๊ณผ๋ ํตํฉ ๋ชจ๋ธ์ด ๋ฐ์ ์ ์ ๋ฐ ์์ ์ค๋ฅ ์ฌ์ด์ ์ธ์ดํ์ ์๊ด์ฑ์ ํ์ฉํ์ฌ ์๋๋ฐ์ํ๊ฐ ๋ฐ ๋ฐ์์ค๋ฅ๊ฒ์ถ ๊ณผ์ ์ ์ฑ๋ฅ์ ํฅ์์์ผฐ์ผ๋ฉฐ, ๊ทธ ๊ฒฐ๊ณผ ํตํฉ ๋ชจ๋ธ์ด ์ ๋ฌธ ํ๊ฐ์๋ค์ ์ค์ ๋น์์ด๋ฏผ ํ๊ฐ์ ๋น์ทํ ์์์ ๋ค๋ค๋ ๊ฒ์ ๋ณด์ฌ์ค๋ค.Empirical studies report a strong correlation between pronunciation scores and mispronunciations in non-native speech assessments of human evaluators. However, the existing system of computer-assisted pronunciation training (CAPT) regards automatic pronunciation assessment (APA) and mispronunciation detection and diagnosis (MDD) as independent and focuses on individual performance improvement. Motivated by the correlation between two tasks, this study proposes a novel architecture that jointly tackles APA and MDD with a multi-task learning scheme to benefit both tasks. Specifically, APA loss is examined between cross-entropy and root mean square error (RMSE) criteria, and MDD loss is fixed to Connectionist Temporal Classification (CTC) criteria. For the backbone acoustic model, self-supervised model is used with an auxiliary fine-tuning on phone recognition before multi-task learning to leverage extra knowledge transfer. Goodness-of-Pronunciation (GOP) measure is given as an additional input along with the acoustic model.
The joint model significantly outperformed single-task learning counterparts, with a mean of 0.041 PCC increase for APA task on four multi-aspect scores and 0.003 F1 increase for MDD task on Speechocean762 dataset. For the joint model architecture, multi-task learning with RMSE and CTC criteria with raw Robust Wav2vec2.0 and GOP measure achieved the best performance. Analysis indicates that the joint model learned to distinguish scores with low distribution, and to better recognize mispronunciations as mispronunciations compared to single-task learning models.
Interestingly, the degree of the performance increase in each subtask for the joint model was proportional to the strength of the correlation between respective pronunciation score and mispronunciation labels, and the strength of the correlation between the model predictions also increased as the joint model achieved higher performances. The findings reveal that the joint model leveraged the linguistic correlation between pronunciation scores and mispronunciations to improve performances for APA and MDD tasks, and to show behaviors that follow the assessments of human experts.Chapter 1, Introduction 1
Chapter 2. Related work 5
Chapter 3. Methodology 17
Chapter 4. Results 28
Chapter 5. Discussion 47
Chapter 6. Conclusion 52
References 53
Appendix 60
๊ตญ๋ฌธ ์ด๋ก 65์
Leveraging phone-level linguistic-acoustic similarity for utterance-level pronunciation scoring
Recent studies on pronunciation scoring have explored the effect of
introducing phone embeddings as reference pronunciation, but mostly in an
implicit manner, i.e., addition or concatenation of reference phone embedding
and actual pronunciation of the target phone as the phone-level pronunciation
quality representation. In this paper, we propose to use linguistic-acoustic
similarity to explicitly measure the deviation of non-native production from
its native reference for pronunciation assessment. Specifically, the deviation
is first estimated by the cosine similarity between reference phone embedding
and corresponding acoustic embedding. Next, a phone-level Goodness of
pronunciation (GOP) pre-training stage is introduced to guide this
similarity-based learning for better initialization of the aforementioned two
embeddings. Finally, a transformer-based hierarchical pronunciation scorer is
used to map a sequence of phone embeddings, acoustic embeddings along with
their similarity measures to predict the final utterance-level score.
Experimental results on the non-native databases suggest that the proposed
system significantly outperforms the baselines, where the acoustic and phone
embeddings are simply added or concatenated. A further examination shows that
the phone embeddings learned in the proposed approach are able to capture
linguistic-acoustic attributes of native pronunciation as reference.Comment: Accepted by ICASSP 202
Methods for pronunciation assessment in computer aided language learning
Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2011.Cataloged from PDF version of thesis.Includes bibliographical references (p. 149-176).Learning a foreign language is a challenging endeavor that entails acquiring a wide range of new knowledge including words, grammar, gestures, sounds, etc. Mastering these skills all require extensive practice by the learner and opportunities may not always be available. Computer Aided Language Learning (CALL) systems provide non-threatening environments where foreign language skills can be practiced where ever and whenever a student desires. These systems often have several technologies to identify the different types of errors made by a student. This thesis focuses on the problem of identifying mispronunciations made by a foreign language student using a CALL system. We make several assumptions about the nature of the learning activity: it takes place using a dialogue system, it is a task- or game-oriented activity, the student should not be interrupted by the pronunciation feedback system, and that the goal of the feedback system is to identify severe mispronunciations with high reliability. Detecting mispronunciations requires a corpus of speech with human judgements of pronunciation quality. Typical approaches to collecting such a corpus use an expert phonetician to both phonetically transcribe and assign judgements of quality to each phone in a corpus. This is time consuming and expensive. It also places an extra burden on the transcriber. We describe a novel method for obtaining phone level judgements of pronunciation quality by utilizing non-expert, crowd-sourced, word level judgements of pronunciation. Foreign language learners typically exhibit high variation and pronunciation shapes distinct from native speakers that make analysis for mispronunciation difficult. We detail a simple, but effective method for transforming the vowel space of non-native speakers to make mispronunciation detection more robust and accurate. We show that this transformation not only enhances performance on a simple classification task, but also results in distributions that can be better exploited for mispronunciation detection. This transformation of the vowel is exploited to train a mispronunciation detector using a variety of features derived from acoustic model scores and vowel class distributions. We confirm that the transformation technique results in a more robust and accurate identification of mispronunciations than traditional acoustic models.by Mitchell A. Peabody.Ph.D
Artificial Neural Network (ANN) in a Small Dataset to determine Neutrality in the Pronunciation of English as a Foreign Language in Filipino Call Center Agents
Artificial Neural Networks (ANNs) have continued to be efficient models in solving classification problems. In this paper, we explore the use of an ANN with a small dataset to accurately classify whether Filipino call center agentsโ pronunciations are neutral or not based on their employerโs standards. Isolated utterances of the ten most commonly used words in the call center were recorded from eleven agents creating a dataset of 110 utterances. Two learning specialists were consulted to establish ground truths and Cohenโs Kappa was computed as 0.82, validating the reliability of the dataset. The first thirteen Mel-Frequency Cepstral Coefficients (MFCCs) were then extracted from each word and an ANN was trained with Ten-fold Stratified Cross Validation. Experimental results on the model recorded a classification accuracy of 89.60% supported by an overall F-Score of 0.92
MISPRONUNCIATION DETECTION AND DIAGNOSIS IN MANDARIN ACCENTED ENGLISH SPEECH
This work presents the development, implementation, and evaluation of a Mispronunciation Detection and Diagnosis (MDD) system, with application to pronunciation evaluation of Mandarin-accented English speech. A comprehensive detection and diagnosis of errors in the Electromagnetic Articulography corpus of Mandarin-Accented English (EMA-MAE) was performed by using the expert phonetic transcripts and an Automatic Speech Recognition (ASR) system. Articulatory features derived from the parallel kinematic data available in the EMA-MAE corpus were used to identify the most significant articulatory error patterns seen in L2 speakers during common mispronunciations. Using both acoustic and articulatory information, an ASR based Mispronunciation Detection and Diagnosis (MDD) system was built and evaluated across different feature combinations and Deep Neural Network (DNN) architectures. The MDD system captured mispronunciation errors with a detection accuracy of 82.4%, a diagnostic accuracy of 75.8% and a false rejection rate of 17.2%. The results demonstrate the advantage of using articulatory features in revealing the significant contributors of mispronunciation as well as improving the performance of MDD systems
- โฆ