4,472 research outputs found
Limitations of the Empirical Fisher Approximation for Natural Gradient Descent
Natural gradient descent, which preconditions a gradient descent update with
the Fisher information matrix of the underlying statistical model, is a way to
capture partial second-order information. Several highly visible works have
advocated an approximation known as the empirical Fisher, drawing connections
between approximate second-order methods and heuristics like Adam. We dispute
this argument by showing that the empirical Fisher---unlike the Fisher---does
not generally capture second-order information. We further argue that the
conditions under which the empirical Fisher approaches the Fisher (and the
Hessian) are unlikely to be met in practice, and that, even on simple
optimization problems, the pathologies of the empirical Fisher can have
undesirable effects.Comment: V3: Minor corrections (typographic errors
A Unified Framework for Compositional Fitting of Active Appearance Models
Active Appearance Models (AAMs) are one of the most popular and
well-established techniques for modeling deformable objects in computer vision.
In this paper, we study the problem of fitting AAMs using Compositional
Gradient Descent (CGD) algorithms. We present a unified and complete view of
these algorithms and classify them with respect to three main characteristics:
i) cost function; ii) type of composition; and iii) optimization method.
Furthermore, we extend the previous view by: a) proposing a novel Bayesian cost
function that can be interpreted as a general probabilistic formulation of the
well-known project-out loss; b) introducing two new types of composition,
asymmetric and bidirectional, that combine the gradients of both image and
appearance model to derive better conver- gent and more robust CGD algorithms;
and c) providing new valuable insights into existent CGD algorithms by
reinterpreting them as direct applications of the Schur complement and the
Wiberg method. Finally, in order to encourage open research and facilitate
future comparisons with our work, we make the implementa- tion of the
algorithms studied in this paper publicly available as part of the Menpo
Project.Comment: 39 page
Optimization Methods for Inverse Problems
Optimization plays an important role in solving many inverse problems.
Indeed, the task of inversion often either involves or is fully cast as a
solution of an optimization problem. In this light, the mere non-linear,
non-convex, and large-scale nature of many of these inversions gives rise to
some very challenging optimization problems. The inverse problem community has
long been developing various techniques for solving such optimization tasks.
However, other, seemingly disjoint communities, such as that of machine
learning, have developed, almost in parallel, interesting alternative methods
which might have stayed under the radar of the inverse problem community. In
this survey, we aim to change that. In doing so, we first discuss current
state-of-the-art optimization methods widely used in inverse problems. We then
survey recent related advances in addressing similar challenges in problems
faced by the machine learning community, and discuss their potential advantages
for solving inverse problems. By highlighting the similarities among the
optimization challenges faced by the inverse problem and the machine learning
communities, we hope that this survey can serve as a bridge in bringing
together these two communities and encourage cross fertilization of ideas.Comment: 13 page
Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting
We introduce the Kronecker factored online Laplace approximation for
overcoming catastrophic forgetting in neural networks. The method is grounded
in a Bayesian online learning framework, where we recursively approximate the
posterior after every task with a Gaussian, leading to a quadratic penalty on
changes to the weights. The Laplace approximation requires calculating the
Hessian around a mode, which is typically intractable for modern architectures.
In order to make our method scalable, we leverage recent block-diagonal
Kronecker factored approximations to the curvature. Our algorithm achieves over
90% test accuracy across a sequence of 50 instantiations of the permuted MNIST
dataset, substantially outperforming related methods for overcoming
catastrophic forgetting.Comment: 13 pages, 6 figure
- …