Universal Distribution of Saliencies for Pruning in Layered Neural Networks

A better understanding of pruning methods based on a ranking of weights according to their saliency in a trained network requires further information on the statistical properties of such saliencies. We focus on two-layer networks with either a linear or nonlinear output unit, and obtain analytic expressions for the distribution of saliencies and their logarithms. Our results reveal unexpected universal properties of the log-saliency distribution and suggest a novel algorithm for saliency-based weight ranking that avoids the numerical cost of second derivative evaluations. 1 Introduction The problem of supervised learning in layered neural networks is a two stage process. A choice of architecture leads to the implicit definition of an associated parameter space f~wg, which represents the ensemble of weights whose values need to be determined in order to fully specify the network. This parameter space is then searched so as to identify specific parameter values ~ w . The goal is to..