Optimizing Vocabulary Modeling for Dysarthric Voice User Interface

학위논문 (박사)-- 서울대학교 대학원 : 협동과정 인지과학전공, 2016. 2. 정민화.마비말장애 화자의 음성 인터페이스 활용에서 빈번한 조음오류, 눌변, 빈번하고 불규칙적으로 발생하는 발화중단, 느린 발화속도 등은 오인식을 유발하는 요인이 된다. 선행연구에서는 장애발화의 음향 및 음운적 특성을 분석하고 이를 기반으로 장애발화 수정, 음향모델 적응, 발음변이 모델링, 문법 및 어휘 모델링 등을 통해 인식오류의 문제를 보완하였다. 본 논문에서는 장애발화의 특징을 반영하여 음향모델을 최적화했다. 또한 어휘모델의 구성에서 음소범주 기반의 조음특징과 인식오류와의 관계를 모형화하여 단어의 선택기준으로 도입했고 단어 간 유사도를 줄임으로써 장애화자의 음성 인터페이스 시 발생하는 인식오류를 줄였다. 마비말장애 화자를 위한 음향모델의 구축을 위해 첫째로 장애화자의 느린 발화속도에 따라 특징추출의 윈도우 크기와 HMM를 구성하는 state 개수를 조정하여 오류를 낮췄다. 둘째로 HMM의 출력확률 모델로서 GMM, Subspace GMM, DNN 등을 도입하여 인식오류를 비교했다. 셋째로 학습데이터 부족문제에 대한 대응방법으로 정상발화 도입의 효율성을 인식실험으로 확인했다. 조음특징과 인식오류율의 혼합선형모델 분석에서 자음범주 중 오류율과 유의수준 0.05 이하에서 상관관계를 보인 범주는 마찰음과 비음이고 모든 모음범주는 오류율과 유의수준 0.05 이하에서 상관관계를 보였다. 또한 혼합모델은 자음을 조음방법으로 범주화할 때가 조음위치로 범주화할 때에 비해 낮은 AIC를 보였고 모음을 혀의 위치로 범주화할 때가 낮은 AIC를 보였다. 음소를 조음방법과 혀의 위치로 범주화했을 때 마찰음은 단어에 포함될수록 인식오류를 높이는 결과를 보였고 비음은 인식정확도를 높였다. 폐쇄음, 파찰음, 유음 등은 인식결과에 대한 영향이 0에 가까웠다. 모든 모음 범주는 인식정확도를 높였다. 그 중 중설모음의 영향력이 가장 컸고 후설모음, 전설모음, 이중모음 순으로 작아졌다. 단어 간 인식오류의 유발 가능성을 Levenshtein 거리와 Cosine 거리 등 열 거리 기준의 단어 간 유사도로 모델링했고 유사도의 최소-최대 비교와 N-best 추정으로 정의된 단어간 유사도가 음성인식에 영향을 주는 요인이 될 수 있음을 확인했다. 장애발화 인식을 위한 어휘모델의 구성에서 먼저 조음특징 기반의 혼합모델로 단어의 조음점수를 계산하고 점수를 최대화하는 단어로 인식단어 리스트를 만들었다. 또한 단어 간 유사도를 최소화하도록 어휘모델을 수정하여 실험한 결과 기존 통화표 방식에 비해 절대적으로 5.7%, 상대적으로 34.6%의 인식오류가 감소하였다.Articulation errors, disfluency, impulsive pause, low speaking rate have been suggested as factors of recognition error for dysarthric speakers using voice user interface. In related works, methods for correcting dysarthric speech, AM adaptation, pronunciation variation modeling, grammar modeling and vocabulary modeling based on acoustic and phonetic analyses on dysarthric speech were proposed to compensate those factors. In this paper, acoustic model was optimized. And words in the vocabulary were selected by the GLMM which had modeled relationship between recognition errors and articulatory features for phonetic class and optimized by lowering similarity between words. Problems in training AM for dysarthric speech recognition were addressed: firstly low speaking rate was compensated by varying the window length of FFT and the number of states of HMM. Secondly the efficiency of models for emission probability of HMM was compared. Thirdly AM trained using large amount of non-dysarthric speech was experimented. Fricative and nasal were statistically significant in the analysis of relation between recognition error and consonant classes. And all vowel classes were significant. AIC was lower by classifying consonants based on manner of articulation than based on place and by classifying vowels based on position of tongue than based on height. Fricatives increased WER and nasals increased accuracy of recognition. Estimates of plosive, affricate, liquid were close to zero. All vowel classes increased accuracy. Estimate of central vowel was the largest followed by back vowel, front vowel and diphthong. Triggering recognition error by competitive words was modeled by similarity between words based on Levenshtein and cosine distance respectively. Effect of similarity between words on recognition result was confirmed by the minimum-maximum similarity contrast and the N-best prediction. Prior to model vocabulary, articulation score for each word was calculated. Firstly the vocabulary was composed of the words with the maximum articulation scores. Secondly the words in the vocabulary with high similarity were replaced by the word with less similarity and large articulation score. In dysarthric speech recognitions, the optimized vocabulary lowered WER by 5.72% (34.60% ERR).제 1 장 서론 1 제 2 장 관련연구 5 제 1 절 마비말장애 발화 조음오류 분석 5 제 2 절 음성인식 시스템의 구조와 모델 9 2.1 특징추출 12 2.2 음향모델 12 2.3 발음모델 16 2.4 언어모델 18 제 3 절 음성인식시스템의 활용분야 18 제 4 절 마비말장애 발화 인식 22 제 3 장 특징추출 및 음향모델 베이스라인 구축 27 제 1 절 접근방법 28 제 2 절 개발 환경 29 2.1 음성 데이터 29 2.2 음성인식 환경 및 음향모델 학습절차 30 제 3 절 인식실험 32 3.1 발화속도 모델링 32 3.2 출력확률 모델 파라미터 최적화 38 3.3 음향모델 학습 데이터의 구성 44 3.4 실험결과 분석 및 요약 46 제 4 절 결론 48 제 4 장 조음오류 특성 기반 인식단어 선택기준 49 제 1 절 한국어 음소 정의와 조음 특징에 의한 범주화 50 제 2 절 연구 목표 53 제 3 절 분석 데이터 53 3.1 음성 데이터 53 3.2 음소의 범주 및 통계 54 3.3 음성인식 56 3.4 데이터 분석 57 제 4 절 분석 결과 59 제 5 절 결론 62 제 5 장 단어 간 유사도 최소화 기준 인식단어 최적화 63 제 1 절 열 거리 기반의 단어 간 유사도 65 제 2 절 인식률에 대한 단어 간 유사도의 영향 67 제 3 절 N-best 추정 68 제 6 장 인식실험 74 제 1 절 단어 리스트 구성 74 1.1 베이스라인 인식 단어 리스트 76 1.2 조음점수 최대화 단어 리스트 77 1.3 단어 간 유사도 최소화 단어 리스트 77 제 2 절 기초 실험 78 제 3 절 실험 환경 81 3.1 음성 코퍼스 구성 81 제 4 절 인식 결과 84 4.1 베이스라인 모델 84 4.2 조음점수 최대화 모델 84 4.3 단어 간 유사도 최소화 모델 86 제 7 장 결론 88 제 1 절 연구결과 요약 및 평가 88 제 2 절 기여도 요약 90 제 3 절 향후 연구 91 참고 문헌 94 부록 101 Abstract 114Docto

Deep Transfer Learning for Automatic Speech Recognition: Towards Better Generalization

Automatic speech recognition (ASR) has recently become an important challenge when using deep learning (DL). It requires large-scale training datasets and high computational and storage resources. Moreover, DL techniques and machine learning (ML) approaches in general, hypothesize that training and testing data come from the same domain, with the same input feature space and data distribution characteristics. This assumption, however, is not applicable in some real-world artificial intelligence (AI) applications. Moreover, there are situations where gathering real data is challenging, expensive, or rarely occurring, which can not meet the data requirements of DL models. deep transfer learning (DTL) has been introduced to overcome these issues, which helps develop high-performing models using real datasets that are small or slightly different but related to the training data. This paper presents a comprehensive survey of DTL-based ASR frameworks to shed light on the latest developments and helps academics and professionals understand current challenges. Specifically, after presenting the DTL background, a well-designed taxonomy is adopted to inform the state-of-the-art. A critical analysis is then conducted to identify the limitations and advantages of each framework. Moving on, a comparative study is introduced to highlight the current challenges before deriving opportunities for future research

Towards Automatic Speech-Language Assessment for Aphasia Rehabilitation

Speech-based technology has the potential to reinforce traditional aphasia therapy through the development of automatic speech-language assessment systems. Such systems can provide clinicians with supplementary information to assist with progress monitoring and treatment planning, and can provide support for on-demand auxiliary treatment. However, current technology cannot support this type of application due to the difficulties associated with aphasic speech processing. The focus of this dissertation is on the development of computational methods that can accurately assess aphasic speech across a range of clinically-relevant dimensions. The first part of the dissertation focuses on novel techniques for assessing aphasic speech intelligibility in constrained contexts. The second part investigates acoustic modeling methods that lead to significant improvement in aphasic speech recognition and allow the system to work with unconstrained speech samples. The final part demonstrates the efficacy of speech recognition-based analysis in automatic paraphasia detection, extraction of clinically-motivated quantitative measures, and estimation of aphasia severity. The methods and results presented in this work will enable robust technologies for accurately recognizing and assessing aphasic speech, and will provide insights into the link between computational methods and clinical understanding of aphasia.PHDComputer Science & EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/140840/1/ducle_1.pd

Personalised Dialogue Management for Users with Speech Disorders

Many electronic devices are beginning to include Voice User Interfaces (VUIs) as an alternative to conventional interfaces. VUIs are especially useful for users with restricted upper limb mobility, because they cannot use keyboards and mice. These users, however, often suffer from speech disorders (e.g. dysarthria), making Automatic Speech Recognition (ASR) challenging, thus degrading the performance of the VUI. Partially Observable Markov Decision Process (POMDP) based Dialogue Management (DM) has been shown to improve the interaction performance in challenging ASR environments, but most of the research in this area has focused on Spoken Dialogue Systems (SDSs) developed to provide information, where the users interact with the system only a few times. In contrast, most VUIs are likely to be used by a single speaker over a long period of time, but very little research has been carried out on adaptation of DM models to specific speakers. This thesis explores methods to adapt DM models (in particular dialogue state tracking models and policy models) to a specific user during a longitudinal interaction. The main differences between personalised VUIs and typical SDSs are identified and studied. Then, state-of-the-art DM models are modified to be used in scenarios which are unique to long-term personalised VUIs, such as personalised models initialised with data from different speakers or scenarios where the dialogue environment (e.g. the ASR) changes over time. In addition, several speaker and environment related features are shown to be useful to improve the interaction performance. This study is done in the context of homeService, a VUI developed to help users with dysarthria to control their home devices. The study shows that personalisation of the POMDP-DM framework can greatly improve the performance of these interfaces

Graph-based Estimation of Information Divergence Functions

abstract: Information divergence functions, such as the Kullback-Leibler divergence or the Hellinger distance, play a critical role in statistical signal processing and information theory; however estimating them can be challenge. Most often, parametric assumptions are made about the two distributions to estimate the divergence of interest. In cases where no parametric model fits the data, non-parametric density estimation is used. In statistical signal processing applications, Gaussianity is usually assumed since closed-form expressions for common divergence measures have been derived for this family of distributions. Parametric assumptions are preferred when it is known that the data follows the model, however this is rarely the case in real-word scenarios. Non-parametric density estimators are characterized by a very large number of parameters that have to be tuned with costly cross-validation. In this dissertation we focus on a specific family of non-parametric estimators, called direct estimators, that bypass density estimation completely and directly estimate the quantity of interest from the data. We introduce a new divergence measure, the $D_p$-divergence, that can be estimated directly from samples without parametric assumptions on the distribution. We show that the $D_p$-divergence bounds the binary, cross-domain, and multi-class Bayes error rates and, in certain cases, provides provably tighter bounds than the Hellinger divergence. In addition, we also propose a new methodology that allows the experimenter to construct direct estimators for existing divergence measures or to construct new divergence measures with custom properties that are tailored to the application. To examine the practical efficacy of these new methods, we evaluate them in a statistical learning framework on a series of real-world data science problems involving speech-based monitoring of neuro-motor disorders.Dissertation/ThesisDoctoral Dissertation Electrical Engineering 201

Factors influencing the efficacy of delayed auditory feedback in treating dysarthria associated with Parkinson\u27s disease

Parkinson\u27s disease patients exhibit a high prevalence of speech deficits including excessive speech rate, reduced intelligibility, and disfluencies. The present study examined the effects of delayed auditory feedback (DAF) as a rate control intervention for dysarthric speakers with Parkinson\u27s disease. Adverse reactions to relatively long delay intervals are commonly observed during clinical use of DAF, and seem to result from improper matching of the delayed signal. To facilitate optimal use of DAF, therefore, clinicians must provide instruction, modeling, and feedback. Clinician instruction is frequently used in speech-language therapy, but has not been evaluated during use of DAF-based interventions. Therefore, the primary purpose of the present study was to evaluate the impact of clinician instruction on the effectiveness of DAF in treating speech deficits. A related purpose was to compare the effects of different delay intervals on speech behaviors. An A-B-A-B single-subject design was utilized. The A phases consisted of a sentence reading task using DAF, while the B phases incorporated clinician instruction into the DAF protocol. During each of the 16 experimental sessions, speakers read with four different delay intervals (0 ms, 50 ms, 100 ms, and 150 ms). During the B phases, the experimenter provided verbal feedback and modeling pertaining to how precisely the speaker matched the delayed signal. Dependent variables measured were speech rate, percent intelligible syllables, and percent disfluencies. Three males with Parkinson\u27s disease and an associated dysarthria participated in the study. Results revealed that for all three speakers, DAF significantly reduced reading rate and produced significant improvements in either intelligibility (for Speaker 3) or fluency (for Speakers 1 and 2). A delay interval of 150 ms produced the greatest reductions in reading rates for all three speakers, although any of the DAF settings used was sufficient to produce significant improvements in either intelligibility or fluency. In addition, supplementing the DAF intervention with clinician instruction resulted in significantly greater gains achieved with DAF. These findings confirmed the effectiveness of various intervals of DAF in improving speech deficits in Parkinson\u27s disease speakers, particular when patients are provided with instruction and modeling from the clinician

Automatic analysis of pathological speech

De ernst van een spraakstoornis wordt vaak gemeten a.d.h.v. spraakverstaanbaarheid. Deze maat wordt in de klinische praktijk vaak bepaald met een perceptuele test. Zo’n test is van nature subjectief vermits de therapeut die de test afneemt de (stoornis van de) patiënt vaak kent en ook vertrouwd is met het gebruikte testmateriaal. Daarom is het interessant te onderzoeken of men met spraakherkenning een objectieve beoordelaar van verstaanbaarheid kan creëren. In deze thesis wordt een methodologie uitgewerkt om een gestandaardiseerde perceptuele test, het Nederlandstalig Spraakverstaanbaarheidsonderzoek (NSVO), te automatiseren. Hiervoor wordt gebruik gemaakt van spraakherkenning om de patiënt fonologisch en fonemisch te karakteriseren en uit deze karakterisering een spraakverstaanbaarheidsscore af te leiden. Experimenten hebben aangetoond dat de berekende scores zeer betrouwbaar zijn. Vermits het NSVO met nonsenswoorden werkt, kunnen vooral kinderen hierdoor leesfouten maken. Daarom werden nieuwe methodes ontwikkeld, gebaseerd op betekenisdragende lopende spraak, die hiertegen robuust zijn en tegelijk ook in verschillende talen gebruikt kunnen worden. Met deze nieuwe modellen bleek het mogelijk te zijn om betrouwbare verstaanbaarheidsscores te berekenen voor Vlaamse, Nederlandse en Duitse spraak. Tenslotte heeft het onderzoek ook belangrijke stappen gezet in de richting van een automatische karakterisering van andere aspecten van de spraakstoornis, zoals articulatie en stemgeving

Multinomial logistic regression probability ratio-based feature vectors for Malay vowel recognition

Vowel Recognition is a part of automatic speech recognition (ASR) systems that classifies speech signals into groups of vowels. The performance of Malay vowel recognition (MVR) like any multiclass classification problem depends largely on Feature Vectors (FVs). FVs such as Mel-frequency Cepstral Coefficients (MFCC) have produced high error rates due to poor phoneme information. Classifier transformed probabilistic features have proved a better alternative in conveying phoneme information. However, the high dimensionality of the probabilistic features introduces additional complexity that deteriorates ASR performance. This study aims to improve MVR performance by proposing an algorithm that transforms MFCC FVs into a new set of features using Multinomial Logistic Regression (MLR) to reduce the dimensionality of the probabilistic features. This study was carried out in four phases which are pre-processing and feature extraction, best regression coefficients generation, feature transformation, and performance evaluation. The speech corpus consists of 1953 samples of five Malay vowels of /a/, /e/, /i/, /o/ and /u/ recorded from students of two public universities in Malaysia. Two sets of algorithms were developed which are DBRCs and FELT. DBRCs algorithm determines the best regression coefficients (DBRCs) to obtain the best set of regression coefficients (RCs) from the extracted 39-MFCC FVs through resampling and data swapping approach. FELT algorithm transforms 39-MFCC FVs using logistic transformation method into FELT FVs. Vowel recognition rates of FELT and 39-MFCC FVs were compared using four different classification techniques of Artificial Neural Network, MLR, Linear Discriminant Analysis, and k-Nearest Neighbour. Classification results showed that FELT FVs surpass the performance of 39-MFCC FVs in MVR. Depending on the classifiers used, the improved performance of 1.48% - 11.70% was attained by FELT over MFCC. Furthermore, FELT significantly improved the recognition accuracy of vowels /o/ and /u/ by 5.13% and 8.04% respectively. This study contributes two algorithms for determining the best set of RCs and generating FELT FVs from MFCC. The FELT FVs eliminate the need for dimensionality reduction with comparable performances. Furthermore, FELT FVs improved MVR for all the five vowels especially /o/ and /u/. The improved MVR performance will spur the development of Malay speech-based systems, especially for the Malaysian community

Exploring and Evaluating Personalized Models for Code Generation

Large Transformer models achieved the state-of-the-art status for Natural Language Understanding tasks and are increasingly becoming the baseline model architecture for modeling source code. Transformers are usually pre-trained on large unsupervised corpora, learning token representations and transformations relevant to modeling generally available text, and are then fine-tuned on a particular downstream task of interest. While fine-tuning is a tried-and-true method for adapting a model to a new domain -- for example, question-answering on a given topic -- generalization remains an on-going challenge. In this paper, we explore and evaluate transformer model fine-tuning for personalization. In the context of generating unit tests for Java methods, we evaluate learning to personalize to a specific software project using several personalization techniques. We consider three key approaches: (i) custom fine-tuning, which allows all the model parameters to be tuned; (ii) lightweight fine-tuning, which freezes most of the model's parameters, allowing tuning of the token embeddings and softmax layer only or the final layer alone; (iii) prefix tuning, which keeps model parameters frozen, but optimizes a small project-specific prefix vector. Each of these techniques offers a trade-off in total compute cost and predictive performance, which we evaluate by code and task-specific metrics, training time, and total computational operations. We compare these fine-tuning strategies for code generation and discuss the potential generalization and cost benefits of each in various deployment scenarios.Comment: Accepted to the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2022), Industry Track - Singapore, November 14-18, 2022, to appear 9 page

Improving multilingual speech recognition systems

End-to-end trainable deep neural networks have become the state-of-the-art architecture for automatic speech recognition (ASR), provided that the network is trained with a sufficiently large dataset. However, many existing languages are too sparsely resourced for deep learning networks to achieve as high accuracy as their resource-abundant counterparts. Multilingual recognition systems mitigate data sparsity issues by training models on data from multiple language resources to learn a speech-to-text or speech-to-phone model universal to all languages. The resulting multilingual ASR models usually have better recognition accuracy than the models trained on the individual dataset. In this work, we propose that two limitations exist for multilingual systems, and resolving the two limitations could result in improved recognition accuracy: (1) existing corpora are of the considerably varied form (spontaneous or read speech), corpus size, noise level, and phoneme distribution and the ASR models trained on the joint multilingual dataset have large performance disparities over different languages. We present an optimizable loss function, equal accuracy ratio (EAR), that measures the sequence-level performance disparity between different user groups and we show that explicitly optimizing this objective reduces the performance gap and improves the multilingual recognition accuracy. (2) While having good accuracy on the seen training language, the multilingual systems do not generalize well to unseen testing languages, which we refer to as cross-lingual recognition accuracy. We introduce language embedding using external linguistic typologies and show that such embedding can significantly increase both multilingual and cross-lingual accuracy. We illustrate the effectiveness of the proposed methods with experiments on multilingual and multi-user and multi-dialect corpora