Tutorial: End-to-End Speech Translation

Speech translation is the translation of speech in one language typically to text in another, traditionally accomplished through a combination of automatic speech recognition and machine translation. Speech translation has attracted interest for many years, but the recent successful applications of deep learning to both individual tasks have enabled new opportunities through joint modeling, in what we today call 'end-to-end speech translation.' In this tutorial we will introduce the techniques used in cutting-edge research on speech translation. Starting from the traditional cascaded approach, we will given an overview on data sources and model architectures to achieve state-of-the art performance with end-to-end speech translation for both high- and low-resource languages. In addition, we will discuss methods to evaluate analyze the proposed solutions, as well as the challenges faced when applying speech translation models for real-world applications

Speech and translation technologies for voice-over and audio description: final results of the ALST project

The ALST project (FFI-201231024, funded by the Spanish Ministry of Economy) started in January 2013 aiming to research the implementation of speech technologies (speech recognition and speech synthesis) and translation technologies (machine translation) in two audiovisual transfer modes: audio description, and voice-over. The project will end in December 2015, and the presentation will give an overview of the main objectives of the project as well as its main results. Although limited in its funding and scope, the project has carried out innovative research in the following aspects: a) Implementation of automatic speech recognition and respeaking to generate faster transcripts: experiments have been carried out with professional transcribers to compare various working scenarios (manual transcription/automatic speech recognition/respeaking). b) Implementation of machine translation plus post-editing as an alternative to traditional audiovisual translation: experiments have been carried out using audio descriptions (English into Catalan) and also wildlife documentaries to be voiced over (English into Spanish). c) Implementation of text-to-speech systems instead of human voices, both in audio descriptions and documentaries. All experiments have in common the willingness to obtain objective data (generally linked to the productivity gain in terms of time) but also subjective data on user experience and perceived quality. The presentation will not get into a detailed analysis of all experiments for obvious time constraints, but will offer a general structured overview that will allow the audience to understand the main results of the project as well as possible future research venues

재귀형 인공신경망을 이용한 온라인 음성인식

학위논문 (박사)-- 서울대학교 대학원 : 전기·컴퓨터공학부, 2017. 2. 성원용.재귀형 인공신경망(recurrent neural network, RNN)은 최근 시퀀스-투-시퀀스(sequence-to-sequence) 방식의 여러 모델에서 좋은 성능을 보여 왔다. 최근의 음성인식에서 사용하는 종단간(end-to-end) 훈련 방식의 발전으로 인해, RNN은 일련의 오디오 특징(feature)을 입력으로 하고 일련의 글자(character) 혹은 단어들을 출력으로 하는 단일한 함수를 학습할 수 있게 되었다. 이 함수는 중간에 음소 단위 혹은 발음 사전(lexicon) 단위의 변환을 거치지 않는다. 지금까지, 대부분의 종단간 음성인식은 기존 방식으로 얻은 높은 정확도를 따라가는 데 초점이 맞춰져 있었다. 하지만, 비록 종단간 음성인식 모델이 기존 음성인식 모델만큼의 정확도를 달성했음에도, 이 모델은 보통 미리 잘라진 오디오 데이터를 사용하는 발화 단위의 음성인식에 사용되었고, 실시간으로 연속적인 오디오 데이터를 받아 사용하는 음성인식에는 잘 사용되지 않았다. 이것은 미리 잘라진 데이터로 학습한 RNN은 매우 긴 오디오 입력에 대해서도 잘 동작하도록 일반화(generalization)가 되기 어려웠기 때문이다. 위 문제를 해결하기 위해, 본 논문에서는 무한히 긴 시퀀스를 사용하는 RNN 훈련 방법을 제안한다. 먼저, 이를 위한 효과적인 그래픽 프로세서(graphics processing unit, GPU) 기반 RNN 훈련 프레임워크(framework)를 설명한다. 이 프레임워크는 제한된 시간축 역전파(truncated backpropagation through time, truncated BPTT)를 사용해 훈련되며, 덕분에 실시간으로 들어오는 연속적인 데이터를 사용하여 훈련할 수 있다. 다음으로, 연결성 시계열 분류기(connectionist temporal classification, CTC) 알고리즘의 손실(loss) 계산 방식을 변형한 실시간 CTC 학습 알고리즘을 선보인다. 새롭게 선보인 CTC 손실 계산 알고리즘은 truncated BPTT 기반의 RNN 훈련에 바로 적용될 수 있다. 다음으로, RNN만으로 구성된 종단간 실시간 음성인식 모델을 소개한다. 이 모델은 크게 CTC 출력을 사용하는 음향(acoustic) RNN과 글자 단위 RNN 언어 모델(language model)로 구성된다. 그리고, 접두사 트리(prefix-tree) 기반의 새로운 빔 탐색(beam search)이 사용되어 무한한 입력 오디오에 대해 디코딩(decoding)을 수행할 수 있다. 이 디코딩 방식에는 새로운 빔 가지치기(beam pruning) 알고리즘이 도입되어 트리 구조의 크기가 지수적으로 증가하는 것을 방지한다. 위 음성인식 모델에는 별도의 음소 모델이나 발음 사전이 포함되어 있지 않고, 무한히 긴 일련의 오디오에 대해 디코딩을 수행할 수 있다는 특징이 있다. 위 모델은 또한 다른 종단간 모델에 비해 매우 적은 메모리를 사용하면서도 비견될 만한 정확도를 보인다. 마지막으로, 본 논문에서는 계층형 구조(hierarchical structure)를 이용해 글자 단위 RNN 언어 모델의 성능을 향상시켰다. 특히, 이 글자 단위 RNN 모델은 비슷한 파라미터 수를 갖는 단어 단위 RNN 언어 모델보다 개선된 예측 복잡도(perplexity)를 달성하였다. 또한, 이 글자 단위 RNN 언어 모델을 앞서 설명한 글자 단위 실시간 음성인식 시스템에 적용하여 더욱 적은 연산을 사용하면서도 음성인식 정확도를 향상시킬 수 있었다.Recurrent neural networks (RNNs) have shown outstanding sequence to sequence modeling performance. Thanks to recent advances in end-to-end training approaches for automatic speech recognition (ASR), RNNs can learn direct mapping functions from the sequence of audio features to the sequence of output characters or words without any intermediate phoneme or lexicon layers. So far, majority of studies on end-to-end ASR have been focused on increasing the accuracy of speech recognition to the level of traditional state-of-the-art models. However, although the end-to-end ASR models have reached the accuracy of the traditional systems, their application has usually been limited to utterance-level speech recognition with pre-segmented audio instead of online speech recognition with continuous audio. This is because the RNNs cannot be easily generalized to very long streams of audio when they are trained with segmented audio. To address this problem, we propose an RNN training approach on training sequences with virtually infinite length. Specifically, we describe an efficient GPU-based RNN training framework for the truncated backpropagation through time (BPTT) algorithm, which is suitable for online (continuous) training. Then, we present an online version of the connectionist temporal classification (CTC) loss computation algorithm, where the original CTC loss is estimated with partial sliding window. This modified CTC algorithm can be directly employed for truncated BPTT based RNN training. In addition, a fully RNN based end-to-end online ASR model is proposed. The model is composed of an acoustic RNN with CTC output and a character-level RNN language model that is augmented with a hierarchical structure. Prefix-tree based beam search decoding is employed with a new beam pruning algorithm to prevent exponential growth of the tree. The model is free from phoneme or lexicon models, and can be used for decoding infinitely long audio sequences. Also, this model has very small memory footprint compared to the other end-to-end systems while showing the competitive accuracy. Furthermore, we propose an improved character-level RNN LM with a hierarchical structure. This character-level RNN LM shows improved perplexity compared to the lightweight word-level RNN LM with a comparable size. When this RNN LM is applied to the proposed character-level online ASR, better speech recognition accuracy can be achieved with reduced amount of computation.1 Introduction 1 1.1 Automatic Speech Recognition 1 1.1.1 Traditional ASR 2 1.1.2 End-to-End ASR with Recurrent Neural Networks 3 1.1.3 Offline and Online ASR 3 1.2 Scope of the Dissertation 4 1.2.1 End-to-End Online ASR with RNNs 4 1.2.2 Challenges and Contributions 5 2 Flexible and Efficient RNN Training on GPUs 7 2.1 Introduction 7 2.2 Generalization 9 2.2.1 Generalized RNN Structure 9 2.2.2 Training 11 2.3 Parallelization 15 2.3.1 Intra-Stream Parallelism 15 2.3.2 Inter-Stream Parallelism 17 2.4 Experiments 18 2.5 Concluding Remarks 21 3 Online Sequence Training with Connectionist Temporal Classification 22 3.1 Introduction 22 3.2 Connectionist Temporal Classification 25 3.3 Online Sequence Training 28 3.3.1 Problem Definition 28 3.3.2 Overview of the Proposed Approach 29 3.3.3 CTC-TR: Standard CTC with Truncation 31 3.3.4 CTC-EM: EM-Based Online CTC 32 3.4 Training Continuously Running RNNs 37 3.5 Parallel Training 38 3.6 Experiments 39 3.6.1 End-to-End Speech Recognition with RNNs 39 3.6.2 Phoneme Recognition on TIMIT 46 3.7 Concluding Remarks 51 4 Character-Level Incremental Speech Recognition 52 4.1 Introduction 52 4.2 Models 54 4.2.1 Acoustic Model 54 4.2.2 Language Model 56 4.3 Character-Level Beam Search 57 4.3.1 Prefix-Tree-Based CTC Beam Search 57 4.3.2 Pruning 60 4.4 Experiments 62 4.5 Concluding Remarks 65 5 Character-Level Language Modeling with Hierarchical RNNs 66 5.1 Introduction 66 5.2 Related Work 68 5.2.1 Character-Level Language Modeling with RNNs 68 5.2.2 Character-Aware Word-Level Language Modeling 69 5.3 RNNs with External Clock and Reset Signals 70 5.4 Character-Level Language Modeling with a Hierarchical RNN 72 5.5 Experiments 75 5.5.1 Perplexity 76 5.5.2 End-to-End Automatic Speech Recognition (ASR) 79 5.6 Concluding Remarks 81 6 Conclusion 83 Bibliography 85 Abstract in Korean 98Docto

Robust speech recognition under noisy environments.

Lee Siu Wa.Thesis (M.Phil.)--Chinese University of Hong Kong, 2004.Includes bibliographical references (leaves 116-121).Abstracts in English and Chinese.Abstract --- p.vChapter 1 --- Introduction --- p.1Chapter 1.1 --- An Overview on Automatic Speech Recognition --- p.2Chapter 1.2 --- Thesis Outline --- p.6Chapter 2 --- Baseline Speech Recognition System --- p.8Chapter 2.1 --- Baseline Speech Recognition Framework --- p.8Chapter 2.2 --- Acoustic Feature Extraction --- p.11Chapter 2.2.1 --- Speech Production and Source-Filter Model --- p.12Chapter 2.2.2 --- Review of Feature Representations --- p.14Chapter 2.2.3 --- Mel-frequency Cepstral Coefficients --- p.20Chapter 2.2.4 --- Energy and Dynamic Features --- p.24Chapter 2.3 --- Back-end Decoder --- p.26Chapter 2.4 --- English Digit String Corpus ´ؤ AURORA2 --- p.28Chapter 2.5 --- Baseline Recognition Experiment --- p.31Chapter 3 --- A Simple Recognition Framework with Model Selection --- p.34Chapter 3.1 --- Mismatch between Training and Testing Conditions --- p.34Chapter 3.2 --- Matched Training and Testing Conditions --- p.38Chapter 3.2.1 --- Noise type-Matching --- p.38Chapter 3.2.2 --- SNR-Matching --- p.43Chapter 3.2.3 --- Noise Type and SNR-Matching --- p.44Chapter 3.3 --- Recognition Framework with Model Selection --- p.48Chapter 4 --- Noise Spectral Estimation --- p.53Chapter 4.1 --- Introduction to Statistical Estimation Methods --- p.53Chapter 4.1.1 --- Conventional Estimation Methods --- p.54Chapter 4.1.2 --- Histogram Technique --- p.55Chapter 4.2 --- Quantile-based Noise Estimation (QBNE) --- p.57Chapter 4.2.1 --- Overview of Quantile-based Noise Estimation (QBNE) --- p.58Chapter 4.2.2 --- Time-Frequency Quantile-based Noise Estimation (T-F QBNE) --- p.62Chapter 4.2.3 --- Mainlobe-Resilient Time-Frequency Quantile-based Noise Estimation (M-R T-F QBNE) --- p.65Chapter 4.3 --- Estimation Performance Analysis --- p.72Chapter 4.4 --- Recognition Experiment with Model Selection --- p.74Chapter 5 --- Feature Compensation: Algorithm and Experiment --- p.81Chapter 5.1 --- Feature Deviation from Clean Speech --- p.81Chapter 5.1.1 --- Deviation in MFCC Features --- p.82Chapter 5.1.2 --- Implications for Feature Compensation --- p.84Chapter 5.2 --- Overview of Conventional Compensation Methods --- p.86Chapter 5.3 --- Feature Compensation by In-phase Feature Induction --- p.94Chapter 5.3.1 --- Motivation --- p.94Chapter 5.3.2 --- Methodology --- p.97Chapter 5.4 --- Compensation Framework for Magnitude Spectrum and Segmen- tal Energy --- p.102Chapter 5.5 --- Recognition -Experiments --- p.103Chapter 6 --- Conclusions --- p.112Chapter 6.1 --- Summary and Discussions --- p.112Chapter 6.2 --- Future Directions --- p.114Bibliography --- p.11

Deep Learning for Environmentally Robust Speech Recognition: An Overview of Recent Developments

Eliminating the negative effect of non-stationary environmental noise is a long-standing research topic for automatic speech recognition that stills remains an important challenge. Data-driven supervised approaches, including ones based on deep neural networks, have recently emerged as potential alternatives to traditional unsupervised approaches and with sufficient training, can alleviate the shortcomings of the unsupervised methods in various real-life acoustic environments. In this light, we review recently developed, representative deep learning approaches for tackling non-stationary additive and convolutional degradation of speech with the aim of providing guidelines for those involved in the development of environmentally robust speech recognition systems. We separately discuss single- and multi-channel techniques developed for the front-end and back-end of speech recognition systems, as well as joint front-end and back-end training frameworks

Towards Affordable Disclosure of Spoken Word Archives

This paper presents and discusses ongoing work aiming at affordable disclosure of real-world spoken word archives in general, and in particular of a collection of recorded interviews with Dutch survivors of World War II concentration camp Buchenwald. Given such collections, the least we want to be able to provide is search at different levels and a flexible way of presenting results. Strategies for automatic annotation based on speech recognition – supporting e.g., within-document search– are outlined and discussed with respect to the Buchenwald interview collection. In addition, usability aspects of the spoken word search are discussed on the basis of our experiences with the online Buchenwald web portal. It is concluded that, although user feedback is generally fairly positive, automatic annotation performance is still far from satisfactory, and requires additional research