57 research outputs found
The Case for Full-Matrix Adaptive Regularization
Adaptive regularization methods come in diagonal and full-matrix variants.
However, only the former have enjoyed widespread adoption in training
large-scale deep models. This is due to the computational overhead of
manipulating a full matrix in high dimension. In this paper, we show how to
make full-matrix adaptive regularization practical and useful. We present GGT,
a truly scalable full-matrix adaptive optimizer. At the heart of our algorithm
is an efficient method for computing the inverse square root of a low-rank
matrix. We show that GGT converges to first-order local minima, providing the
first rigorous theoretical analysis of adaptive regularization in non-convex
optimization. In preliminary experiments, GGT trains faster across a variety of
synthetic tasks and standard deep learning benchmarks
Exploring efficient neural architectures for linguistic-acoustic mapping in text-to-speech
Conversion from text to speech relies on the accurate mapping from linguistic to acoustic symbol sequences, for which current practice employs recurrent statistical models such as recurrent neural networks. Despite the good performance of such models (in terms of low distortion in the generated speech), their recursive structure with intermediate affine transformations tends to make them slow to train and to sample from. In this work, we explore two different mechanisms that enhance the operational efficiency of recurrent neural networks, and study their performance–speed trade-off. The first mechanism is based on the quasi-recurrent neural network, where expensive affine transformations are removed from temporal connections and placed only on feed-forward computational directions. The second mechanism includes a module based on the transformer decoder network, designed without recurrent connections but emulating them with attention and positioning codes. Our results show that the proposed decoder networks are competitive in terms of distortion when compared to a recurrent baseline, whilst being significantly faster in terms of CPU and GPU inference time. The best performing model is the one based on the quasi-recurrent mechanism, reaching the same level of naturalness as the recurrent neural network based model with a speedup of 11.2 on CPU and 3.3 on GPU.Peer ReviewedPostprint (published version
- …