The ability of deep neural networks to generalise well even when they
interpolate their training data has been explained using various "simplicity
biases". These theories postulate that neural networks avoid overfitting by
first learning simple functions, say a linear classifier, before learning more
complex, non-linear functions. Meanwhile, data structure is also recognised as
a key ingredient for good generalisation, yet its role in simplicity biases is
not yet understood. Here, we show that neural networks trained using stochastic
gradient descent initially classify their inputs using lower-order input
statistics, like mean and covariance, and exploit higher-order statistics only
later during training. We first demonstrate this distributional simplicity bias
(DSB) in a solvable model of a neural network trained on synthetic data. We
empirically demonstrate DSB in a range of deep convolutional networks and
visual transformers trained on CIFAR10, and show that it even holds in networks
pre-trained on ImageNet. We discuss the relation of DSB to other simplicity
biases and consider its implications for the principle of Gaussian universality
in learning.Comment: Source code available at https://github.com/sgoldt/dist_inc_com