40,253 research outputs found
Efficient Regret Minimization in Non-Convex Games
We consider regret minimization in repeated games with non-convex loss
functions. Minimizing the standard notion of regret is computationally
intractable. Thus, we define a natural notion of regret which permits efficient
optimization and generalizes offline guarantees for convergence to an
approximate local optimum. We give gradient-based methods that achieve optimal
regret, which in turn guarantee convergence to equilibrium in this framework.Comment: Published as a conference paper at ICML 201
The Case for Full-Matrix Adaptive Regularization
Adaptive regularization methods come in diagonal and full-matrix variants.
However, only the former have enjoyed widespread adoption in training
large-scale deep models. This is due to the computational overhead of
manipulating a full matrix in high dimension. In this paper, we show how to
make full-matrix adaptive regularization practical and useful. We present GGT,
a truly scalable full-matrix adaptive optimizer. At the heart of our algorithm
is an efficient method for computing the inverse square root of a low-rank
matrix. We show that GGT converges to first-order local minima, providing the
first rigorous theoretical analysis of adaptive regularization in non-convex
optimization. In preliminary experiments, GGT trains faster across a variety of
synthetic tasks and standard deep learning benchmarks
- …