8 research outputs found
Constrained Output Embeddings for End-to-End Code-Switching Speech Recognition with Only Monolingual Data
The lack of code-switch training data is one of the major concerns in the
development of end-to-end code-switching automatic speech recognition (ASR)
models. In this work, we propose a method to train an improved end-to-end
code-switching ASR using only monolingual data. Our method encourages the
distributions of output token embeddings of monolingual languages to be
similar, and hence, promotes the ASR model to easily code-switch between
languages. Specifically, we propose to use Jensen-Shannon divergence and cosine
distance based constraints. The former will enforce output embeddings of
monolingual languages to possess similar distributions, while the later simply
brings the centroids of two distributions to be close to each other.
Experimental results demonstrate high effectiveness of the proposed method,
yielding up to 4.5% absolute mixed error rate improvement on Mandarin-English
code-switching ASR task.Comment: 5 pages, 3 figures, accepted to INTERSPEECH 201
Language-specific Acoustic Boundary Learning for Mandarin-English Code-switching Speech Recognition
Code-switching speech recognition (CSSR) transcribes speech that switches
between multiple languages or dialects within a single sentence. The main
challenge in this task is that different languages often have similar
pronunciations, making it difficult for models to distinguish between them. In
this paper, we propose a method for solving the CSSR task from the perspective
of language-specific acoustic boundary learning. We introduce language-specific
weight estimators (LSWE) to model acoustic boundary learning in different
languages separately. Additionally, a non-autoregressive (NAR) decoder and a
language change detection (LCD) module are employed to assist in training.
Evaluated on the SEAME corpus, our method achieves a state-of-the-art mixed
error rate (MER) of 16.29% and 22.81% on the test_man and test_sge sets. We
also demonstrate the effectiveness of our method on a 9000-hour in-house
meeting code-switching dataset, where our method achieves a relatively 7.9% MER
reduction
Adaptive Contextual Biasing for Transducer Based Streaming Speech Recognition
By incorporating additional contextual information, deep biasing methods have
emerged as a promising solution for speech recognition of personalized words.
However, for real-world voice assistants, always biasing on such personalized
words with high prediction scores can significantly degrade the performance of
recognizing common words. To address this issue, we propose an adaptive
contextual biasing method based on Context-Aware Transformer Transducer (CATT)
that utilizes the biased encoder and predictor embeddings to perform streaming
prediction of contextual phrase occurrences. Such prediction is then used to
dynamically switch the bias list on and off, enabling the model to adapt to
both personalized and common scenarios. Experiments on Librispeech and internal
voice assistant datasets show that our approach can achieve up to 6.7% and
20.7% relative reduction in WER and CER compared to the baseline respectively,
mitigating up to 96.7% and 84.9% of the relative WER and CER increase for
common cases. Furthermore, our approach has a minimal performance impact in
personalized scenarios while maintaining a streaming inference pipeline with
negligible RTF increase
Neural Natural Language Generation: A Survey on Multilinguality, Multimodality, Controllability and Learning
Developing artificial learning systems that can understand and generate natural language has been one of the long-standing goals of artificial intelligence. Recent decades have witnessed an impressive progress on both of these problems, giving rise to a new family of approaches. Especially, the advances in deep learning over the past couple of years have led to neural approaches to natural language generation (NLG). These methods combine generative language learning techniques with neural-networks based frameworks. With a wide range of applications in natural language processing, neural NLG (NNLG) is a new and fast growing field of research. In this state-of-the-art report, we investigate the recent developments and applications of NNLG in its full extent from a multidimensional view, covering critical perspectives such as multimodality, multilinguality, controllability and learning strategies. We summarize the fundamental building blocks of NNLG approaches from these aspects and provide detailed reviews of commonly used preprocessing steps and basic neural architectures. This report also focuses on the seminal applications of these NNLG models such as machine translation, description generation, automatic speech recognition, abstractive summarization, text simplification, question answering and generation, and dialogue generation. Finally, we conclude with a thorough discussion of the described frameworks by pointing out some open research directions.This work has been partially supported by the European Commission ICT COST Action âMulti-task, Multilingual, Multi-modal Language Generationâ (CA18231). AE was supported by BAGEP 2021 Award of the Science Academy. EE was supported in part by TUBA GEBIP 2018 Award. BP is in in part funded by Independent Research Fund Denmark (DFF) grant 9063-00077B. IC has received funding from the European Unionâs Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 838188. EL is partly funded by Generalitat Valenciana and the Spanish Government throught projects PROMETEU/2018/089 and RTI2018-094649-B-I00, respectively. SMI is partly funded by UNIRI project uniri-drustv-18-20. GB is partly supported by the Ministry of Innovation and the National Research, Development and Innovation Office within the framework of the Hungarian Artificial Intelligence National Laboratory Programme. COT is partially funded by the Romanian Ministry of European Investments and Projects through the Competitiveness Operational Program (POC) project âHOLOTRAINâ (grant no. 29/221 ap2/07.04.2020, SMIS code: 129077) and by the German Academic Exchange Service (DAAD) through the project âAWAKEN: content-Aware and netWork-Aware faKE News mitigationâ (grant no. 91809005). ESA is partially funded by the German Academic Exchange Service (DAAD) through the project âDeep-Learning Anomaly Detection for Human and Automated Users Behaviorâ (grant no. 91809358)