292 research outputs found
Model Accuracy and Runtime Tradeoff in Distributed Deep Learning:A Systematic Study
This paper presents Rudra, a parameter server based distributed computing
framework tuned for training large-scale deep neural networks. Using variants
of the asynchronous stochastic gradient descent algorithm we study the impact
of synchronization protocol, stale gradient updates, minibatch size, learning
rates, and number of learners on runtime performance and model accuracy. We
introduce a new learning rate modulation strategy to counter the effect of
stale gradients and propose a new synchronization protocol that can effectively
bound the staleness in gradients, improve runtime performance and achieve good
model accuracy. Our empirical investigation reveals a principled approach for
distributed training of neural networks: the mini-batch size per learner should
be reduced as more learners are added to the system to preserve the model
accuracy. We validate this approach using commonly-used image classification
benchmarks: CIFAR10 and ImageNet.Comment: Accepted by The IEEE International Conference on Data Mining 2016
(ICDM 2016
Making Asynchronous Stochastic Gradient Descent Work for Transformers
Asynchronous stochastic gradient descent (SGD) is attractive from a speed
perspective because workers do not wait for synchronization. However, the
Transformer model converges poorly with asynchronous SGD, resulting in
substantially lower quality compared to synchronous SGD. To investigate why
this is the case, we isolate differences between asynchronous and synchronous
methods to investigate batch size and staleness effects. We find that summing
several asynchronous updates, rather than applying them immediately, restores
convergence behavior. With this hybrid method, Transformer training for neural
machine translation task reaches a near-convergence level 1.36x faster in
single-node multi-GPU training with no impact on model quality
- …