1 research outputs found
Gradient descent in Gaussian random fields as a toy model for high-dimensional optimisation in deep learning
In this paper we model the loss function of high-dimensional optimization
problems by a Gaussian random field, or equivalently a Gaussian process. Our
aim is to study gradient descent in such loss functions or energy landscapes
and compare it to results obtained from real high-dimensional optimization
problems such as encountered in deep learning. In particular, we analyze the
distribution of the improved loss function after a step of gradient descent,
provide analytic expressions for the moments as well as prove asymptotic
normality as the dimension of the parameter space becomes large. Moreover, we
compare this with the expectation of the global minimum of the landscape
obtained by means of the Euler characteristic of excursion sets. Besides
complementing our analytical findings with numerical results from simulated
Gaussian random fields, we also compare it to loss functions obtained from
optimisation problems on synthetic and real data sets by proposing a "black
box" random field toy-model for a deep neural network loss function.Comment: 10 pages, 10 figure