2,035 research outputs found
Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting
We introduce the Kronecker factored online Laplace approximation for
overcoming catastrophic forgetting in neural networks. The method is grounded
in a Bayesian online learning framework, where we recursively approximate the
posterior after every task with a Gaussian, leading to a quadratic penalty on
changes to the weights. The Laplace approximation requires calculating the
Hessian around a mode, which is typically intractable for modern architectures.
In order to make our method scalable, we leverage recent block-diagonal
Kronecker factored approximations to the curvature. Our algorithm achieves over
90% test accuracy across a sequence of 50 instantiations of the permuted MNIST
dataset, substantially outperforming related methods for overcoming
catastrophic forgetting.Comment: 13 pages, 6 figure
Adaptive Control of Robotic Manipulators using Deep Neural Networks
In this paper, we present a lifelong deep learning-based control of robotic manipulators with nonstandard adaptive laws using singular value decomposition (SVD) based direct tracking error driven (DTED) approach. Moreover, we incorporate concurrent learning (CL) to relax persistency of excitation condition and elastic weight consolidation (EWC) for lifelong learning on different tasks in the adaptive laws. Simulation results confirm theoretical conclusions
Bayesian Dark Knowledge
We consider the problem of Bayesian parameter estimation for deep neural
networks, which is important in problem settings where we may have little data,
and/ or where we need accurate posterior predictive densities, e.g., for
applications involving bandits or active learning. One simple approach to this
is to use online Monte Carlo methods, such as SGLD (stochastic gradient
Langevin dynamics). Unfortunately, such a method needs to store many copies of
the parameters (which wastes memory), and needs to make predictions using many
versions of the model (which wastes time).
We describe a method for "distilling" a Monte Carlo approximation to the
posterior predictive density into a more compact form, namely a single deep
neural network. We compare to two very recent approaches to Bayesian neural
networks, namely an approach based on expectation propagation [Hernandez-Lobato
and Adams, 2015] and an approach based on variational Bayes [Blundell et al.,
2015]. Our method performs better than both of these, is much simpler to
implement, and uses less computation at test time.Comment: final version submitted to NIPS 201
A Multi Hidden Recurrent Neural Network with a Modified Grey Wolf Optimizer
Identifying university students' weaknesses results in better learning and
can function as an early warning system to enable students to improve. However,
the satisfaction level of existing systems is not promising. New and dynamic
hybrid systems are needed to imitate this mechanism. A hybrid system (a
modified Recurrent Neural Network with an adapted Grey Wolf Optimizer) is used
to forecast students' outcomes. This proposed system would improve instruction
by the faculty and enhance the students' learning experiences. The results show
that a modified recurrent neural network with an adapted Grey Wolf Optimizer
has the best accuracy when compared with other models.Comment: 34 pages, published in PLoS ON
- …