We present a method for training bottleneck MLPs for use in tandem ASR. Experiments on meetings data show that this approach leads to improved performance compared with training MLPs from a random initialization

Frankel, Joe

King, Simon

Wang, Dong

English

Edinburgh Research Explorer

     Edinburgh Research Explorer                                      Growing bottleneck features for tandem ASRCitation for published version:Frankel, J, Wang, D & King, S 2008, 'Growing bottleneck features for tandem ASR'. in Proc. Interspeech.pp. 1549.Link:Link to publication record in Edinburgh Research ExplorerDocument Version:Author final version (often known as postprint)Published In:Proc. InterspeechPublisher Rights Statement:Frankel, J., Wang, D., & King, S. (2008). Growing bottleneck features for tandem ASR. In Proc. Interspeech. (pp.1549).General rightsCopyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s)and / or other copyright owners and it is a condition of accessing these publications that users recognise andabide by the legal requirements associated with these rights.Take down policyThe University of Edinburgh has made every reasonable effort to ensure that Edinburgh Research Explorercontent complies with UK legislation. If you believe that the public display of this file breaches copyright pleasecontact openaccess@ed.ac.uk providing details, and we will remove access to the work immediately andinvestigate your claim.Download date: 20. Feb. 2015Growing bottleneck features for tandem ASRJoe Frankel, Dong Wang, Simon KingCentre for Speech Technology Research, University of Edinburgh, UKjoe@cstr.ed.ac.uk; dwang2@inf.ed.ac.uk; Simon.King@ed.ac.ukAbstractWe present a method for training bottleneck MLPs for use intandem ASR. Experiments on meetings data show that this ap-proach leads to improved performance compared with trainingMLPs from a random initialization.Index Terms: tandem ASR, bottleneck MLP1. IntroductionTandem automatic speech recognition (ASR) [1] extends acous-tic parameters such as Mel frequency cepstral coefficients(MFCCs) to include features derived from phone-classificationmulti-layer perceptrons (MLPs). Rather than interpreting theoutputs as phone posteriors, they are subject to a logarithmtransformation and dimensionality reduction, then used as fea-tures in a standard hidden Markov model (HMM) ASR system.A common configuration is to use 9 frames of perceptuallinear prediction (PLP) cepstra as input to a 3-layer MLP. Prin-cipal components analysis (PCA) is used for dimensionality re-duction, and whilst there is no clear optimal final dimension,work at ICSI has found that retaining around 95% of the vari-ance, which yields a feature vector of dimension 25 works well.An alternative approach is to use a bottleneck MLP [2] (a.k.a.autoencoder network), and make the dimensionality reductionintegral to the network. Multiple hidden layers are used, one ofwhich is low-dimension and is known as the ‘bottleneck layer’.Hinton and Salakhutdinov [3] suggest that autoencoder net-work training is particularly sensitive to initial conditions, andthat when initial weights close to a good solution, gradient de-scent works well. In order to improve the initial estimates, theypropose “pretraining” in which the learned activations from onelayer are used as the input to the following layer. We apply sim-ilar ideas in the context of training of MLPs for tandem ASR.2. Tandem MLPsWe present the results of experiments using 3 different MLPs.In all cases, the input layers have 351 input units (9 frames of39 PLPs) and 46 output units (one for each phone class) with asoftmax activation function. The baseline 3-layer MLP has size351, 5000, 46. We then trained a 5-layer MLP of size 351, 5000,25, 5000, 46 units from a random initialization. The activationsof the bottleneck layer of 25 units are subjected to a logarithmtransform, and used as the tandem features.The third MLP also had 5 layers, though rather than beingtrained from scratch the MLP was “grown”. A 4-layer MLPof size 351, 5000, 25, 46 was first trained using the weightsand biases of the input to first hidden layer of the 3-layer MLPabove, with the remainder of weights randomly initialized. A5-layer MLP of size 351, 5000, 25, 5000, 46 was then trained1This work is supported by EPSRC Advanced Research Fellowand Marie Curie Early Stage Training Site EdSST (MEST-CT-2005-020568).with the weights and biases connecting the input and first twohidden layers taken from the 4-layer MLP, and the remainderof weights randomly initialized. The cross validation accuracyfor the grown 5-layer MLP was higher than that of either therandomly-initialized 5-layer and the 3-layer MLP.3. Experiments and discussionExperiments use data from the meetings domain, with speechrecorded on headset microphones. HMMs and MLPs weretrained on a little over 100 hours of data, and results are pre-sented on test data from the NIST Spring 2004 RT04s evalua-tion. A trigram language model was used during decoding.The acoustic waveform was parameterized as 12 Mel-frequency cepstral coefficients (MFCCs) with energy, and 1stand 2nd order derivatives. The final feature dimension was 64once the 25 MLP features were added. The 5-layer MLPs canbe used to provide tandem features either from the bottlenecklayer, or from the output layer (in which case they are subjectedto dimensionality reduction via PCA.Features WERMFCC baseline 40.8%MFCC + 3-layer tandem 38.8%MFCC + 5-layer tandem (output) 41.6%MFCC + 5-layer tandem (bottleneck) 40.9%MFCC + 5-layer tandem (bottleneck), grown 37.5%Table 1: ASR results for each of the 3 and 5-layer MLPsThe results presented in Table 1 show that using tandemfeatures from the 3-layer MLP leads to improved performancecompared to the MFCC baseline. The 5-layer MLP which wastrained from a random initialization gives an increase or nochange in WER compared with the MFCC baseline when thetandem features are taken from the output and bottleneck layersrespectively. However, the bottleneck features from the tandemMLP which has been grown leads to a reduction in WER com-pared with the 3-layer MLP result. A paired t-test shows thatthis result is significant with p < 0.001.We conclude that the integral dimensionality reduction ofbottleneck MLPs leads to improved performance over 3-layertandem MLPs and PCA, and that bottleneck MLPs should begrown rather than trained from a random initialization.4. References[1] H. Hermansky, D. Ellis, and S. Sharma, “Tandem connectionistfeature stream extraction for conventional hmm systems,” in ProcICASSP, vol. III, Istanbul, Turkey, 2000, pp. 1635–1638.[2] F. Grezl, M. Karafiat, S. Kontar, and J. Cernocky, “Probabilisticand bottle-neck features for lvcsr of meetings,” in Proc. ICASSP,Honolulu, April 2007.[3] G. Hinton and R. Salakhutdinov, “Reducing the dimensionality ofdata with neural networks,” Science, vol. 313, no. 28, pp. 504 –507, July 2006.

Probabilistic and bottle-neck features for lvcsr of meetings,” in

Reducing the dimensionality of data with neural networks,”

Tandem connectionist feature stream extraction for conventional hmm systems,”

Edinburgh Research Archive

Growing bottleneck features for tandem ASRJoe Frankel, Dong Wang, Simon KingCentre for Speech Technology Research, University of Edinburgh, UKjoe@cstr.ed.ac.uk; dwang2@inf.ed.ac.uk; Simon.King@ed.ac.ukAbstractWe present a method for training bottleneck MLPs for use intandem ASR. Experiments on meetings data show that this ap-proach leads to improved performance compared with trainingMLPs from a random initialization.Index Terms: tandem ASR, bottleneck MLP1. IntroductionTandem automatic speech recognition (ASR) [1] extends acous-tic parameters such as Mel frequency cepstral coefficients(MFCCs) to include features derived from phone-classificationmulti-layer perceptrons (MLPs). Rather than interpreting theoutputs as phone posteriors, they are subject to a logarithmtransformation and dimensionality reduction, then used as fea-tures in a standard hidden Markov model (HMM) ASR system.A common configuration is to use 9 frames of perceptuallinear prediction (PLP) cepstra as input to a 3-layer MLP. Prin-cipal components analysis (PCA) is used for dimensionality re-duction, and whilst there is no clear optimal final dimension,work at ICSI has found that retaining around 95% of the vari-ance, which yields a feature vector of dimension 25 works well.An alternative approach is to use a bottleneck MLP [2] (a.k.a.autoencoder network), and make the dimensionality reductionintegral to the network. Multiple hidden layers are used, one ofwhich is low-dimension and is known as the ‘bottleneck layer’.Hinton and Salakhutdinov [3] suggest that autoencoder net-work training is particularly sensitive to initial conditions, andthat when initial weights close to a good solution, gradient de-scent works well. In order to improve the initial estimates, theypropose “pretraining” in which the learned activations from onelayer are used as the input to the following layer. We apply sim-ilar ideas in the context of training of MLPs for tandem ASR.2. Tandem MLPsWe present the results of experiments using 3 different MLPs.In all cases, the input layers have 351 input units (9 frames of39 PLPs) and 46 output units (one for each phone class) with asoftmax activation function. The baseline 3-layer MLP has size351, 5000, 46. We then trained a 5-layer MLP of size 351, 5000,25, 5000, 46 units from a random initialization. The activationsof the bottleneck layer of 25 units are subjected to a logarithmtransform, and used as the tandem features.The third MLP also had 5 layers, though rather than beingtrained from scratch the MLP was “grown”. A 4-layer MLPof size 351, 5000, 25, 46 was first trained using the weightsand biases of the input to first hidden layer of the 3-layer MLPabove, with the remainder of weights randomly initialized. A5-layer MLP of size 351, 5000, 25, 5000, 46 was then trained1This work is supported by EPSRC Advanced Research Fellowand Marie Curie Early Stage Training Site EdSST (MEST-CT-2005-020568).with the weights and biases connecting the input and first twohidden layers taken from the 4-layer MLP, and the remainderof weights randomly initialized. The cross validation accuracyfor the grown 5-layer MLP was higher than that of either therandomly-initialized 5-layer and the 3-layer MLP.3. Experiments and discussionExperiments use data from the meetings domain, with speechrecorded on headset microphones. HMMs and MLPs weretrained on a little over 100 hours of data, and results are pre-sented on test data from the NIST Spring 2004 RT04s evalua-tion. A trigram language model was used during decoding.The acoustic waveform was parameterized as 12 Mel-frequency cepstral coefficients (MFCCs) with energy, and 1stand 2nd order derivatives. The final feature dimension was 64once the 25 MLP features were added. The 5-layer MLPs canbe used to provide tandem features either from the bottlenecklayer, or from the output layer (in which case they are subjectedto dimensionality reduction via PCA.Features WERMFCC baseline 40.8%MFCC + 3-layer tandem 38.8%MFCC + 5-layer tandem (output) 41.6%MFCC + 5-layer tandem (bottleneck) 40.9%MFCC + 5-layer tandem (bottleneck), grown 37.5%Table 1: ASR results for each of the 3 and 5-layer MLPsThe results presented in Table 1 show that using tandemfeatures from the 3-layer MLP leads to improved performancecompared to the MFCC baseline. The 5-layer MLP which wastrained from a random initialization gives an increase or nochange in WER compared with the MFCC baseline when thetandem features are taken from the output and bottleneck layersrespectively. However, the bottleneck features from the tandemMLP which has been grown leads to a reduction in WER com-pared with the 3-layer MLP result. A paired t-test shows thatthis result is significant with p < 0.001.We conclude that the integral dimensionality reduction ofbottleneck MLPs leads to improved performance over 3-layertandem MLPs and PCA, and that bottleneck MLPs should begrown rather than trained from a random initialization.4. References[1] H. Hermansky, D. Ellis, and S. Sharma, “Tandem connectionistfeature stream extraction for conventional hmm systems,” in ProcICASSP, vol. III, Istanbul, Turkey, 2000, pp. 1635–1638.[2] F. Grezl, M. Karafiat, S. Kontar, and J. Cernocky, “Probabilisticand bottle-neck features for lvcsr of meetings,” in Proc. ICASSP,Honolulu, April 2007.[3] G. Hinton and R. Salakhutdinov, “Reducing the dimensionality ofdata with neural networks,” Science, vol. 313, no. 28, pp. 504 –507, July 2006.

Growing bottleneck features for tandem ASR

https://era.ed.ac.uk/bitstream/1842/3898/1/bottlenet.a.pdf

Growing bottleneck features for tandem ASR

Abstract

Similar works

Full text

Available Versions

Edinburgh Research Explorer

Edinburgh Research Archive