8 research outputs found
Spatial Mixture-of-Experts
Many data have an underlying dependence on spatial location; it may be
weather on the Earth, a simulation on a mesh, or a registered image. Yet this
feature is rarely taken advantage of, and violates common assumptions made by
many neural network layers, such as translation equivariance. Further, many
works that do incorporate locality fail to capture fine-grained structure. To
address this, we introduce the Spatial Mixture-of-Experts (SMoE) layer, a
sparsely-gated layer that learns spatial structure in the input domain and
routes experts at a fine-grained level to utilize it. We also develop new
techniques to train SMoEs, including a self-supervised routing loss and damping
expert errors. Finally, we show strong results for SMoEs on numerous tasks, and
set new state-of-the-art results for medium-range weather prediction and
post-processing ensemble weather forecasts.Comment: 20 pages, 3 figures; NeurIPS 202
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
In deep learning, models typically reuse the same parameters for all inputs.
Mixture of Experts (MoE) defies this and instead selects different parameters
for each incoming example. The result is a sparsely-activated model -- with
outrageous numbers of parameters -- but a constant computational cost. However,
despite several notable successes of MoE, widespread adoption has been hindered
by complexity, communication costs and training instability -- we address these
with the Switch Transformer. We simplify the MoE routing algorithm and design
intuitive improved models with reduced communication and computational costs.
Our proposed training techniques help wrangle the instabilities and we show
large sparse models may be trained, for the first time, with lower precision
(bfloat16) formats. We design models based off T5-Base and T5-Large to obtain
up to 7x increases in pre-training speed with the same computational resources.
These improvements extend into multilingual settings where we measure gains
over the mT5-Base version across all 101 languages. Finally, we advance the
current scale of language models by pre-training up to trillion parameter
models on the "Colossal Clean Crawled Corpus" and achieve a 4x speedup over the
T5-XXL model
Towards Continual Reinforcement Learning: A Review and Perspectives
In this article, we aim to provide a literature review of different
formulations and approaches to continual reinforcement learning (RL), also
known as lifelong or non-stationary RL. We begin by discussing our perspective
on why RL is a natural fit for studying continual learning. We then provide a
taxonomy of different continual RL formulations and mathematically characterize
the non-stationary dynamics of each setting. We go on to discuss evaluation of
continual RL agents, providing an overview of benchmarks used in the literature
and important metrics for understanding agent performance. Finally, we
highlight open problems and challenges in bridging the gap between the current
state of continual RL and findings in neuroscience. While still in its early
days, the study of continual RL has the promise to develop better incremental
reinforcement learners that can function in increasingly realistic applications
where non-stationarity plays a vital role. These include applications such as
those in the fields of healthcare, education, logistics, and robotics.Comment: Preprint, 52 pages, 8 figure