651 research outputs found
On Quadratic Penalties in Elastic Weight Consolidation
Elastic weight consolidation (EWC, Kirkpatrick et al, 2017) is a novel
algorithm designed to safeguard against catastrophic forgetting in neural
networks. EWC can be seen as an approximation to Laplace propagation (Eskin et
al, 2004), and this view is consistent with the motivation given by Kirkpatrick
et al (2017). In this note, I present an extended derivation that covers the
case when there are more than two tasks. I show that the quadratic penalties in
EWC are inconsistent with this derivation and might lead to double-counting
data from earlier tasks
- …