47 research outputs found
Information Losses in Neural Classifiers from Sampling
This paper considers the subject of information losses arising from the
finite datasets used in the training of neural classifiers. It proves a
relationship between such losses as the product of the expected total variation
of the estimated neural model with the information about the feature space
contained in the hidden representation of that model. It then bounds this
expected total variation as a function of the size of randomly sampled datasets
in a fairly general setting, and without bringing in any additional dependence
on model complexity. It ultimately obtains bounds on information losses that
are less sensitive to input compression and in general much smaller than
existing bounds. The paper then uses these bounds to explain some recent
experimental findings of information compression in neural networks which
cannot be explained by previous work. Finally, the paper shows that not only
are these bounds much smaller than existing ones, but that they also correspond
well with experiments.Comment: To be published in IEEE TNNL