Identifying and attacking the saddle point problem in high-dimensional non-convex optimization

A central challenge to many fields of science and engineering involves minimizing non-convex error functions over continuous, high dimensional spaces. Gradient descent or quasi-Newton methods are almost ubiquitously used to perform such minimizations, and it is often thought that a main source of difficulty for these local methods to find the global minimum is the proliferation of local minima with much higher error than the global minimum. Here we argue, based on results from statistical physics, random matrix theory, neural network theory, and empirical evidence, that a deeper and more profound difficulty originates from the proliferation of saddle points, not local minima, especially in high dimensional problems of practical interest. Such saddle points are surrounded by high error plateaus that can dramatically slow down learning, and give the illusory impression of the existence of a local minimum. Motivated by these arguments, we propose a new approach to second-order optimization, the saddle-free Newton method, that can rapidly escape high dimensional saddle points, unlike gradient descent and quasi-Newton methods. We apply this algorithm to deep or recurrent neural network training, and provide numerical evidence for its superior optimization performance.Comment: The theoretical review and analysis in this article draw heavily from arXiv:1405.4604 [cs.LG

Does face inversion change spatial frequency tuning?

International audienceThe authors examined spatial frequency (SF) tuning of upright and inverted face identification using an SF variant of the Bubbles technique (F. Gosselin & P. G. Schyns, 2001). In Experiment 1, they validated the SF Bubbles technique in a plaid detection task. In Experiments 2a-c, the SFs used for identifying upright and inverted inner facial features were investigated. Although a clear inversion effect was present (mean accuracy was 24% higher and response times 455 ms shorter for upright faces), SF tunings were remarkably similar in both orientation conditions (mean r = .98; an SF band of 1.9 octaves centered at 9.8 cycles per face width for faces of about 6 degrees ). In Experiments 3a and b, the authors demonstrated that their technique is sensitive to both subtle bottom-up and top-down induced changes in SF tuning, suggesting that the null results of Experiments 2a-c are real. The most parsimonious explanation of the findings is provided by the quantitative account of the face inversion effect: The same information is used for identifying upright and inverted inner facial features, but processing has greater sensitivity with the former

Sedimentation in a stratified ambient

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Mathematics, 2003.Includes bibliographical references (p. 147-156).We study the interaction between settling particles and a stratified ambient in a variety of contexts. We first study the generation of large scale fluid motions by the localised release of a finite mass of particles in the form of plumes or gravity currents. We present the results of a combined theoretical and experimental study describing the evolution of particle clouds formed by the release of heavy particles. In the early stages of motion, particle clouds behave as turbulent fluid thermals; however, their radial expansion eventually stops and particles settle from the base of the cloud at their individual settling speed. We focus on deducing a criterion for the various modes of particle deposition from particle clouds in a stratified ambient. We proceed to study the deposition patterns resulting from particle-laden gravity currents that spread horizontally when released in a particle-free ambient. Using a box-model, we focus on bidisperse gravity currents and examine the resulting particle distribution and maximal deposit length. We then turn to suspensions where particles are initially present throughout the fluid. The simultaneous presence of particles and of a stratified ambient may lead to behaviour analogous to double-diffusive systems, with particles playing the role of a diffusing component. We examine the linear stability of the settling of a particle concentration gradient in a stratified fluid. Numerical simulations allow us to determine the stability of the system for a broad range of particle settling speeds and diffusion coefficients. We then report on layering arising from sedimentation in a density stratified ambient beneath an inclined wall.(cont.) From our experimental study, we describe the series of horizontal intrusions formed by particle-free fluid intruding at its level of neutral buoyancy. We present numerical models describing the time evolution of the concentration of particles and the layer formation. Finally, we present an experimental and theoretical study of the combined influence of hindered settling and settling speed variations due to an ambient stratification. We develop a criterion for the stability of a suspension settling in a stratified ambient and experimental observations allow us to qualify the main features of this instability.by François Alain Blanchette.Ph.D