14 research outputs found
Smooth markets: A basic mechanism for organizing gradient-based learners
With the success of modern machine learning, it is becoming increasingly
important to understand and control how learning algorithms interact.
Unfortunately, negative results from game theory show there is little hope of
understanding or controlling general n-player games. We therefore introduce
smooth markets (SM-games), a class of n-player games with pairwise zero sum
interactions. SM-games codify a common design pattern in machine learning that
includes (some) GANs, adversarial training, and other recent algorithms. We
show that SM-games are amenable to analysis and optimization using first-order
methods.Comment: 18 pages, 3 figure
Taming GANs with Lookahead
Generative Adversarial Networks are notoriously challenging to train. The
underlying minimax optimization is highly susceptible to the variance of the
stochastic gradient and the rotational component of the associated game vector
field. We empirically demonstrate the effectiveness of the Lookahead
meta-optimization method for optimizing games, originally proposed for standard
minimization. The backtracking step of Lookahead naturally handles the
rotational game dynamics, which in turn enables the gradient ascent descent
method to converge on challenging toy games often analyzed in the literature.
Moreover, it implicitly handles high variance without using large mini-batches,
known to be essential for reaching state of the art performance. Experimental
results on MNIST, SVHN, and CIFAR-10, demonstrate a clear advantage of
combining Lookahead with Adam or extragradient, in terms of performance, memory
footprint, and improved stability. Using 30-fold fewer parameters and 16-fold
smaller minibatches we outperform the reported performance of the
class-dependent BigGAN on CIFAR-10 by obtaining FID of \emph{without}
using the class labels, bringing state-of-the-art GAN training within reach of
common computational resources