44 research outputs found
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
In deep learning, models typically reuse the same parameters for all inputs.
Mixture of Experts (MoE) defies this and instead selects different parameters
for each incoming example. The result is a sparsely-activated model -- with
outrageous numbers of parameters -- but a constant computational cost. However,
despite several notable successes of MoE, widespread adoption has been hindered
by complexity, communication costs and training instability -- we address these
with the Switch Transformer. We simplify the MoE routing algorithm and design
intuitive improved models with reduced communication and computational costs.
Our proposed training techniques help wrangle the instabilities and we show
large sparse models may be trained, for the first time, with lower precision
(bfloat16) formats. We design models based off T5-Base and T5-Large to obtain
up to 7x increases in pre-training speed with the same computational resources.
These improvements extend into multilingual settings where we measure gains
over the mT5-Base version across all 101 languages. Finally, we advance the
current scale of language models by pre-training up to trillion parameter
models on the "Colossal Clean Crawled Corpus" and achieve a 4x speedup over the
T5-XXL model
Skip-gram Language Modeling Using Sparse Non-negative Matrix Probability Estimation
We present a novel family of language model (LM) estimation techniques named
Sparse Non-negative Matrix (SNM) estimation. A first set of experiments
empirically evaluating it on the One Billion Word Benchmark shows that SNM
-gram LMs perform almost as well as the well-established Kneser-Ney (KN)
models. When using skip-gram features the models are able to match the
state-of-the-art recurrent neural network (RNN) LMs; combining the two modeling
techniques yields the best known result on the benchmark. The computational
advantages of SNM over both maximum entropy and RNN LM estimation are probably
its main strength, promising an approach that has the same flexibility in
combining arbitrary features effectively and yet should scale to very large
amounts of data as gracefully as -gram LMs do