102 research outputs found
A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biases
Learned optimizers -- neural networks that are trained to act as optimizers
-- have the potential to dramatically accelerate training of machine learning
models. However, even when meta-trained across thousands of tasks at huge
computational expense, blackbox learned optimizers often struggle with
stability and generalization when applied to tasks unlike those in their
meta-training set. In this paper, we use tools from dynamical systems to
investigate the inductive biases and stability properties of optimization
algorithms, and apply the resulting insights to designing inductive biases for
blackbox optimizers. Our investigation begins with a noisy quadratic model,
where we characterize conditions in which optimization is stable, in terms of
eigenvalues of the training dynamics. We then introduce simple modifications to
a learned optimizer's architecture and meta-training procedure which lead to
improved stability, and improve the optimizer's inductive bias. We apply the
resulting learned optimizer to a variety of neural network training tasks,
where it outperforms the current state of the art learned optimizer -- at
matched optimizer computational overhead -- with regard to optimization
performance and meta-training speed, and is capable of generalization to tasks
far different from those it was meta-trained on.Comment: NeurIPS 202
- …