3,353 research outputs found
Single stream parallelization of generalized LSTM-like RNNs on a GPU
Recurrent neural networks (RNNs) have shown outstanding performance on
processing sequence data. However, they suffer from long training time, which
demands parallel implementations of the training procedure. Parallelization of
the training algorithms for RNNs are very challenging because internal
recurrent paths form dependencies between two different time frames. In this
paper, we first propose a generalized graph-based RNN structure that covers the
most popular long short-term memory (LSTM) network. Then, we present a
parallelization approach that automatically explores parallelisms of arbitrary
RNNs by analyzing the graph structure. The experimental results show that the
proposed approach shows great speed-up even with a single training stream, and
further accelerates the training when combined with multiple parallel training
streams.Comment: Accepted by the 40th IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP) 201
Accelerating recurrent neural network training using sequence bucketing and multi-GPU data parallelization
An efficient algorithm for recurrent neural network training is presented.
The approach increases the training speed for tasks where a length of the input
sequence may vary significantly. The proposed approach is based on the optimal
batch bucketing by input sequence length and data parallelization on multiple
graphical processing units. The baseline training performance without sequence
bucketing is compared with the proposed solution for a different number of
buckets. An example is given for the online handwriting recognition task using
an LSTM recurrent neural network. The evaluation is performed in terms of the
wall clock time, number of epochs, and validation loss value.Comment: 4 pages, 5 figures, Comments, 2016 IEEE First International
Conference on Data Stream Mining & Processing (DSMP), Lviv, 201
Recurrent Neural Networks with Top-k Gains for Session-based Recommendations
RNNs have been shown to be excellent models for sequential data and in
particular for data that is generated by users in an session-based manner. The
use of RNNs provides impressive performance benefits over classical methods in
session-based recommendations. In this work we introduce novel ranking loss
functions tailored to RNNs in the recommendation setting. The improved
performance of these losses over alternatives, along with further tricks and
refinements described in this work, allow for an overall improvement of up to
35% in terms of MRR and Recall@20 over previous session-based RNN solutions and
up to 53% over classical collaborative filtering approaches. Unlike data
augmentation-based improvements, our method does not increase training times
significantly. We further demonstrate the performance gain of the RNN over
baselines in an online A/B test.Comment: CIKM'18, authors' versio
- …