It is often useful to compactly summarize important properties of model
parameters and training data so that they can be used later without storing
and/or iterating over the entire dataset. As a specific case, we consider
estimating the Function Space Distance (FSD) over a training set, i.e. the
average discrepancy between the outputs of two neural networks. We propose a
Linearized Activation Function TRick (LAFTR) and derive an efficient
approximation to FSD for ReLU neural networks. The key idea is to approximate
the architecture as a linear network with stochastic gating. Despite requiring
only one parameter per unit of the network, our approach outcompetes other
parametric approximations with larger memory requirements. Applied to continual
learning, our parametric approximation is competitive with state-of-the-art
nonparametric approximations, which require storing many training examples.
Furthermore, we show its efficacy in estimating influence functions accurately
and detecting mislabeled examples without expensive iterations over the entire
dataset.Comment: 18 pages, 5 figures, ICML 202