Strategies for Handling Out-of-Vocabulary Words in Automatic Speech Recognition

Nowadays, most ASR (automatic speech recognition) systems deployed in industry are closed-vocabulary systems, meaning we have a limited vocabulary of words the system can recognize, and where pronunciations are provided to the system. Words out of this vocabulary are called out-of-vocabulary (OOV) words, for which either pronunciations or both spellings and pronunciations are not known to the system. The basic motivations of developing strategies to handle OOV words are: First, in the training phase, missing or wrong pronunciations of words in training data results in poor acoustic models. Second, in the test phase, words out of the vocabulary cannot be recognized at all, and mis-recognition of OOV words may affect recognition performance of its in-vocabulary neighbors as well. Therefore, this dissertation is dedicated to exploring strategies of handling OOV words in closed-vocabulary ASR. First, we investigate dealing with OOV words in ASR training data, by introducing an acoustic-data driven pronunciation learning framework using a likelihood-reduction based criterion for selecting pronunciation candidates from multiple sources, i.e. standard grapheme-to-phoneme algorithms (G2P) and phonetic decoding, in a greedy fashion. This framework effectively expands a small hand-crafted pronunciation lexicon to cover OOV words, for which the learned pronunciations have higher quality than approaches using G2P alone or using other baseline pruning criteria. Furthermore, applying the proposed framework to generate alternative pronunciations for in-vocabulary (IV) words improves both recognition performance on relevant words and overall acoustic model performance. Second, we investigate dealing with OOV words in ASR test data, i.e. OOV detection and recovery. We first conduct a comparative study of a hybrid lexical model (HLM) approach for OOV detection, and several baseline approaches, with the conclusion that the HLM approach outperforms others in both OOV detection and first pass OOV recovery performance. Next, we introduce a grammar-decoding framework for efficient second pass OOV recovery, showing that with properly designed schemes of estimating OOV unigram probabilities, the framework significantly improves OOV recovery and overall decoding performance compared to first pass decoding. Finally we propose an open-vocabulary word-level recurrent neural network language model (RNNLM) re-scoring framework, making it possible to re-score lattices containing recovered OOVs using a single word-level RNNLM, that was ignorant of OOVs when it was trained. Above all, the whole OOV recovery pipeline shows the potential of a highly efficient open-vocabulary word-level ASR decoding framework, tightly integrated into a standard WFST decoding pipeline

An enhanced automatic speech recognition system for Arabic

International audienceAutomatic speech recognition for Arabic is a very challenging task. Despite all the classical techniques for Automatic Speech Recognition (ASR), which can be efficiently applied to Arabic speech recognition , it is essential to take into consideration the language specificities to improve the system performance. In this article, we focus on Modern Standard Arabic (MSA) speech recognition. We introduce the challenges related to Arabic language, namely the complex morphology nature of the language and the absence of the short vowels in written text, which leads to several potential vowelization for each graphemes, which is often conflicting. We develop an ASR system for MSA by using Kaldi toolkit. Several acoustic and language models are trained. We obtain a Word Error Rate (WER) of 14.42 for the baseline system and 12.2 relative improvement by rescoring the lattice and by rewriting the output with the right hamoza above or below Alif

Speech Recognition

Chapters in the first part of the book cover all the essential speech processing techniques for building robust, automatic speech recognition systems: the representation for speech signals and the methods for speech-features extraction, acoustic and language modeling, efficient algorithms for searching the hypothesis space, and multimodal approaches to speech recognition. The last part of the book is devoted to other speech processing applications that can use the information from automatic speech recognition for speaker identification and tracking, for prosody modeling in emotion-detection systems and in other speech processing applications that are able to operate in real-world environments, like mobile communication services and smart homes

Deriving and Exploiting Situational Information in Speech: Investigations in a Simulated Search and Rescue Scenario

The need for automatic recognition and understanding of speech is emerging in tasks involving the processing of large volumes of natural conversations. In application domains such as Search and Rescue, exploiting automated systems for extracting mission-critical information from speech communications has the potential to make a real difference. Spoken language understanding has commonly been approached by identifying units of meaning (such as sentences, named entities, and dialogue acts) for providing a basis for further discourse analysis. However, this fine-grained identification of fundamental units of meaning is sensitive to high error rates in the automatic transcription of noisy speech. This thesis demonstrates that topic segmentation and identification techniques can be employed for information extraction from spoken conversations by being robust to such errors. Two novel topic-based approaches are presented for extracting situational information within the search and rescue context. The first approach shows that identifying the changes in the context and content of first responders' report over time can provide an estimation of their location. The second approach presents a speech-based topological map estimation technique that is inspired, in part, by automatic mapping algorithms commonly used in robotics. The proposed approaches are evaluated on a goal-oriented conversational speech corpus, which has been designed and collected based on an abstract communication model between a first responder and a task leader during a search process. Results have confirmed that a highly imperfect transcription of noisy speech has limited impact on the information extraction performance compared with that obtained on the transcription of clean speech data. This thesis also shows that speech recognition accuracy can benefit from rescoring its initial transcription hypotheses based on the derived high-level location information. A new two-pass speech decoding architecture is presented. In this architecture, the location estimation from a first decoding pass is used to dynamically adapt a general language model which is used for rescoring the initial recognition hypotheses. This decoding strategy has resulted in a statistically significant gain in the recognition accuracy of the spoken conversations in high background noise. It is concluded that the techniques developed in this thesis can be extended to more application domains that deal with large volumes of natural spoken conversations

Teaching Community College Students Strategies For Learning Unknown Words As They Read Expository Text

The study reported here investigated methods that enable college students to learn the meaning of unknown words as they read discipline specific academic text. The ability to read and comprehend text is known to be positively correlated with academic success. However that ability is challenging to college students in part because of the sophisticated vocabulary encountered in academic text. The study reported here utilized an experimental design. Forty one participants read specific passages aloud during three sessions. Participants were randomly assigned to one of four intervention groups to investigate alternative methods of learning the meaning of unknown words. In the Strategies group, participants learned the use of context cues, morphological cues, and syntactic cues. Participants in the Definition group learned to use researcher supplied definitions. In the Strategies plus Definition group, participants learned to use both the strategies and definitions. Using a constructivist framework to create meaning while interacting with text, these three groups had time for practice and received feedback. Participants in the fourth group, the Control group, engaged in discussion of the passages. Intervention and outcome measures examined word learning and comprehension. All participants completed a transfer task to investigate the effects of treatment on independent text reading. It was expected that participants in the intervention groups would outperform participants in the Control group, and that participants in the Strategies plus Definition group would outperform participants in the other two intervention groups. Results were mixed. Analyses of data revealed that participants in all three intervention groups demonstrated significantly better word learning and comprehension as measured by definition recall, CLOZE and response to comprehension questions than participants in the Control group. Other measures did not support these hypotheses. There were also interaction effects involving time with treatment groups performing differently on intervention and outcome measures than on transfer task measures. In general participants in the intervention groups performed better during the first three training texts than during the final transfer task. Additionally, participants in the intervention groups did not perform significantly better on the transfer task than participants in the control group. Thus the word learning treatments and their impact on comprehension did not generalize to a novel task as was hypothesized. Results of this study contribute to the research by helping us understand the benefit of methods that enable college students to access academic text. Use of definitions and to a lesser extent, use of strategies, appear to have a positive impact on word learning and comprehension. The use of a combination of strategies and definitions also appears to have a positive impact but, with mixed results, this awaits further study