37 research outputs found
Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting
We introduce the Kronecker factored online Laplace approximation for
overcoming catastrophic forgetting in neural networks. The method is grounded
in a Bayesian online learning framework, where we recursively approximate the
posterior after every task with a Gaussian, leading to a quadratic penalty on
changes to the weights. The Laplace approximation requires calculating the
Hessian around a mode, which is typically intractable for modern architectures.
In order to make our method scalable, we leverage recent block-diagonal
Kronecker factored approximations to the curvature. Our algorithm achieves over
90% test accuracy across a sequence of 50 instantiations of the permuted MNIST
dataset, substantially outperforming related methods for overcoming
catastrophic forgetting.Comment: 13 pages, 6 figure
Online Structured Laplace Approximations for Overcoming Catastrophic Forgetting
We introduce the Kronecker factored online Laplace approximation for overcoming
catastrophic forgetting in neural networks. The method is grounded in a Bayesian
online learning framework, where we recursively approximate the posterior after
every task with a Gaussian, leading to a quadratic penalty on changes to the weights.
The Laplace approximation requires calculating the Hessian around a mode, which
is typically intractable for modern architectures. In order to make our method
scalable, we leverage recent block-diagonal Kronecker factored approximations to
the curvature. Our algorithm achieves over 90% test accuracy across a sequence
of 50 instantiations of the permuted MNIST dataset, substantially outperforming
related methods for overcoming catastrophic forgetting