Bergmann, J

Schuller, B

Weninger, F

Journal of Machine Learning Research

English

Spiral - Imperial College Digital Repository

Journal of Machine Learning Research 16 (2015) 547-551 Submitted 7/13; Published 3/15Introducing CURRENNT: The Munich Open-SourceCUDA RecurREnt Neural Network ToolkitFelix Weninger weninger@tum.deJohannes Bergmann privat@johannes-bergmann.deBjörn Schuller∗ schuller@tum.deMachine Learning & Signal Processing, Technische Universität München, 80290 Munich, GermanyEditor: Mikio BraunAbstractIn this article, we introduce CURRENNT, an open-source parallel implementation of deeprecurrent neural networks (RNNs) supporting graphics processing units (GPUs) throughNVIDIA’s Computed Unified Device Architecture (CUDA). CURRENNT supports uni- andbidirectional RNNs with Long Short-Term Memory (LSTM) memory cells which overcomethe vanishing gradient problem. To our knowledge, CURRENNT is the first publiclyavailable parallel implementation of deep LSTM-RNNs. Benchmarks are given on a noisyspeech recognition task from the 2013 2nd CHiME Speech Separation and RecognitionChallenge, where LSTM-RNNs have been shown to deliver best performance. In the result,double digit speedups in bidirectional LSTM training are achieved with respect to a referencesingle-threaded CPU implementation. CURRENNT is available under the GNU GeneralPublic License from http://sourceforge.net/p/currennt.Keywords: parallel computing, deep neural networks, recurrent neural networks, LongShort-Term Memory1. IntroductionRecurrent neural networks (RNNs) are known as powerful sequence learners. In particular,the Long Short-Term Memory (LSTM) architecture has been proven to provide excellentmodeling of language (Sundermeyer et al., 2012), music (Eck and Schmidhuber, 2002), speech(Graves et al., 2013), and facial expressions (Wöllmer et al., 2012). LSTM units overcome thevanishing gradient problem of traditional RNNs by the introduction of a memory cell whichcan be controlled by input, output and reset operations (Gers et al., 2000). In particular,recent research demonstrates that deep LSTM-RNNs exhibit superior performance in speechrecognition in comparison to state-of-the-art deep feed forward networks (Graves et al.,2013). However, in contrast to the widespread usage of the latter (Hinton et al., 2012),RNNs are still not adopted by the research community at large. One of the major barriersis the lack of high-performance implementations for training RNNs; at the same time, suchimplementations are non-trivial due to the limited parallelism caused by time dependencies.To the best of our knowledge, there is no publicly available software dedicated to parallelLSTM-RNN training. Thus, we introduce our CUda RecurREnt Neural Network Toolkit(CURRENNT) which exploits a mini-batch learning scheme performing parallel weight∗. B. Schuller is also with the Department of Computing, Imperial College London, UK.c©2015 Felix Weninger, Johannes Bergmann and Björn Schuller.Weninger, Bergmann and SchullerMulticlass-ClassificationLayerBinary-ClassificationLayerSse-PostOutputLayerFeedForwardLayerLstmLayerSoftmaxLayerPostOutputLayerTrainableLayerInputLayerLayerNeuralNetwork OptimizerSteepestDecent-OptimizerDataSetDataSetFraction11N1K1 1 MFigure 1: CURRENNT’s C++ classes for deep feedforward and LSTM-RNN modeling.updates from multiple sequences. CURRENNT implements learning from big data sets thatdo not fit into memory by means of a random access data format. GPUs are supportedthrough NVIDIA’s Computed Unified Device Architecture (CUDA). The RNN structureimplemented in CURRENNT is based on LSTM units and in addition, feedforward networktraining is supported. Besides simple regression, CURRENNT also includes logistic andsoftmax output layers for training of binary and multi-way classification.To briefly refer some related studies and freely available implementations: A ‘reference’CPU implementation of LSTM-RNNs as used by Graves (2008) is available as open-sourceC++ code (Graves, 2013). A Python library for many machine learning algorithms includingLSTM-RNN has been introduced by Schaul et al. (2010); however, it does not directlysupport parallel processing. Multi-core training of (standard) RNNs has been investigatedby Cernanský (2009), but the source code is not available. Pascanu et al. have recentlyreleased a Python implementation (‘GroundHog’) of various RNN types described in theirstudy (Pascanu et al., 2014), exploiting GPU-accelerated training through Theano; yet, itdoes not provide LSTM-RNNs, and at the moment there is no user-friendly interface.2. DesignCURRENNT provides a C++ class library for deep LSTM-RNN modeling (cf. Figure 1) anda command line application for network training and evaluation. The network architecturecan be specified by the user in JavaScript Object Notation (JSON), and trained parametersare saved in the same format, allowing, e.g., for deep learning with pre-training. For efficiencyreasons, features for training and evaluation are given in binary format, adhering to theNetCDF standard, but network outputs can also be saved in CSV format to facilitatepost-processing. All C++ code is designed to be platform independent and has been testedon Windows and various Linux distributions. The required CUDA compute capability is 1.3(2008), allowing usage on virtually all of the consumer grade NVIDIA GPUs deployed intoday’s desktop PCs. The behavior of the gradient descent training algorithm is controlledby various switches of the command line application, allowing, e.g., for on-line or batchlearning and fine-tuning of the training algorithm such as adding Gaussian noise to the inputactivations and randomly shuffling training data in on-line learning to improve generalization(Graves et al., 2013). The interested reader is referred to the documentation for more details.548CURRENNT: Open-Source CUDA RecurREnt Neural Network Toolkit3. ImplementationDeep LSTM-RNNs with N layers are implemented as follows. An input sequence xt ismapped to the output sequence yt, t = 1, . . . , T through the iteration (forward pass):h(0)t := xt,h(n)t := L(n)t(h(n−1)t ,h(n)t−1),yt := S(W(N),(N+1)h(N)t + b(N+1)).In the above and the ongoing, W denotes weight matrices and b stands for bias vectors(with superscripts denoting layer indices). h(n)t denotes the hidden feature representationof time frame t in the level n units, n = 1, . . . , N . The 0-th layer is the input layer andthe N + 1-th layer the output layer. S is the (vector valued) output layer function, e.g.,a softmax function for multi-way classification (cf. Figure 1). L(n)t denotes the compositeLSTM activation function which is used instead of the common simple sigmoid-shapedfunctions. The crucial point is to augment each unit with a state variable ct, resulting in anautomaton-like structure. The hidden layer activations correspond to the state variables(‘memory cells’) scaled by the activations of the ‘output gates’ o(n)t ,h(n)t = o(n)t ⊗ tanh(c(n)t ),c(n)t = f(n)t ⊗ c(n)t−1 + i(n)t ⊗ tanh(W(n−1),(n)h(n−1)t + W(n),(n)h(n)t−1 + b(n)c), (1)where ⊗ denotes element-wise multiplication and tanh is also applied element-wise. Thus,the state is scaled by a ‘forget’ gate (Gers et al., 2000) with dynamic activation f(n)t insteadof a recurrent connection with static weight. i(n)t is the activation of the input gate thatregulates the ‘influx’ from the feedforward and recurrent connections. The activations of theinput, output and forget gates are calculated in a similar fashion as c(n)t (Graves et al., 2013).From the dependencies between layers (n− 1 n) and time steps (t− 1 t) in the above,it is obvious that parallel computation of feedforward activations cannot be performed acrosslayers; further, parallel computation of recurrent activations is not possible across time steps.Thus, to increase the degree of parallelization, we consider data fractions (cf. Figure 1) ofsize P out of S sequences in parallel, each having exactly T time steps (creating ‘dummy’time steps for shorter sequences which are neglected in the error calculation). For instance,we consider a state matrix C(n) for the n-th layer,C(n) = [c(n)1,p · · · c(n)1,p+P−1 · · · c(n)T,p · · · c(n)T,p+P−1], (2)where c(n)t,p is the state for sequence p in layer n at time t. To realize the update equation(1) we can now compute the feedforward part for all time steps and P sequences in parallelsimply by pre-multiplication with W(n−1),(n). For the recurrent part, we can update C(n)from ‘left to right’ using W(n),(n). Input, output and forget gate activations are calculatedin an analogous fashion. In this process, the matrix structure (2) ensures memory locality ofthe data corresponding to one time step (matrices are stored in column-major order). Forbidirectional layers the above matrix structure is replicated at each layer; in the ‘backward’549Weninger, Bergmann and SchullerRNNLIB (Graves, 2013) CURRENNTParallel sequences (P ) 1 1 10 50 200Validation set error (10 ep.) 0.138 0.138 0.135 0.137 0.144Validation set error (50 ep.) 0.120 0.119 0.116 0.118 0.119Training time / epoch [s] 7 420 3 805 580 392 334Speedup (1.0) 2.0 12.8 18.9 22.2Table 1: Performance (error / speedup) on CHiME 2013 noisy speech recognition task.part, the recurrent parts are updated from ‘right to left’, and activations are collected in asingle vector before passing them to the subsequent layer (Graves et al., 2013).During network training, the backward pass for the hidden layers is realized similarly, bysplitting the matrix of weight changes into a part propagated to the preceding layer and arecurrent part propagated to the previous time step, resulting in a parallel implementationof the backpropagation through time (BPTT) algorithm. The weight changes are appliedfor all sequences (batch learning) or for each data fraction. Thus, if 1 < P < S we performmini-batch learning. In this case, only P sequences have to be kept in memory at once,allowing for learning from large data sets.4. BenchmarkWe conclude with a benchmark on a word recognition task with convolutive non-stationarynoise from the 2013 2nd CHiME Challenge’s track 1 (Vincent et al., 2013), where bidirectionalLSTM decoding has been shown to deliver best performance (Geiger et al., 2013). Weconsider frame-wise word error rate as well as computation speedup in training with respectto the open source C++ reference implementation by Graves (2013) running in a singleCPU thread on an Intel Core2Quad PC with 4 GB of RAM. The GPU is an NVIDIAGTX 560 with 2 GB of RAM. We compare results for different values of P while fixing theother training parameters. The corresponding NetCDF, network configuration, and trainingparameter files are distributed with CURRENNT. Results (Figure 1) show that the errorrate after 50 epochs is not heavily influenced by the batch size for parallel processing, whilespeedups of up to 22.2 can be achieved.5. ConclusionsCURRENNT, our GPU implementation of deep LSTM-RNN for labeling sequential data,has been shown to deliver double digit training speedups at equal accuracy in a noisy speechrecognition task. Future work will be concentrated on discriminative training objectives andcost functions for transcription tasks (Graves, 2008).AcknowledgmentsThis research has been supported by DFG grant SCHU 2502/4-1. The authors would like tothank Alex Graves for helpful discussions.550CURRENNT: Open-Source CUDA RecurREnt Neural Network ToolkitReferencesM. Cernanský. Training recurrent neural network using multistream extended Kalman filteron multicore processor and CUDA enabled graphic processor unit. In Proc. of ICANN,volume 1, pages 381–390, 2009.D. Eck and J. Schmidhuber. Learning the long-term structure of the blues. In Proc. ofICANN, pages 284–289, 2002.J. T. Geiger, F. Weninger, A. Hurmalainen, J. F. Gemmeke, M. Wöllmer, B. Schuller,G. Rigoll, and T. Virtanen. The TUM+TUT+KUL approach to the CHiME Challenge2013: Multi-stream ASR exploiting BLSTM networks and sparse NMF. In Proc. 2ndCHiME Workshop, pages 25–30, Vancouver, Canada, 2013.F. Gers, J. Schmidhuber, and F. Cummins. Learning to forget: Continual prediction withLSTM. Neural Computation, 12(10):2451–2471, 2000.A. Graves. Supervised Sequence Labelling with Recurrent Neural Networks. PhD thesis,Technische Universität München, 2008.A. Graves. RNNLIB: A recurrent neural network library for sequence learning problems.http://sourceforge.net/projects/rnnl/, 2013.A. Graves, A. Mohamed, and G. Hinton. Speech recognition with deep recurrent neuralnetworks. In Proc. of ICASSP, pages 6645–6649, Vancouver, Canada, 2013.G. Hinton, L. Deng, D. Yu, G. E. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke,P. Nguyen, T. N. Sainath, and B. Kingsbury. Deep neural networks for acoustic modelingin speech recognition. IEEE Signal Processing Magazine, 29(6):82–97, 2012.R. Pascanu, C. Gulcehre, K. Cho, and Y. Bengio. How to construct deep recurrent neuralnetworks. In Proc. of ICLR, 2014.T. Schaul, J. Bayer, D. Wierstra, Y. Sun, M. Felder, F. Sehnke, T. Rückstieß, and J. Schmid-huber. PyBrain. Journal of Machine Learning Research, 11:743–746, 2010.M. Sundermeyer, R. Schlüter, and H. Ney. LSTM neural networks for language modeling.In Proc. of INTERSPEECH, Portland, OR, USA, 2012.E. Vincent, J. Barker, S. Watanabe, J. Le Roux, F. Nesta, and M. Matassoni. The second‘CHiME’ speech separation and recognition challenge: Datasets, tasks and baselines. InProc. of ICASSP, pages 126–130, Vancouver, Canada, 2013.M. Wöllmer, M. Kaiser, F. Eyben, F. Weninger, B. Schuller, and G. Rigoll. Fully automaticaudiovisual emotion recognition – voice, words, and the face. In T. Fingscheidt andW. Kellermann, editors, Proceedings of Speech Communication; 10. ITG Symposium,pages 1–4, Braunschweig, Germany, 2012.551

Introducing CURRENNT: the Munich Open-Source CUDA RecurREnt Neural Network Toolkit

http://spiral.imperial.ac.uk/bitstream/10044/1/33191/4/weninger15a.pdf

Introducing CURRENNT: the Munich Open-Source CUDA RecurREnt Neural Network Toolkit

Abstract

Similar works

Full text

Available Versions

Spiral - Imperial College Digital Repository