20,509 research outputs found
A Geometric Approach of Gradient Descent Algorithms in Neural Networks
In this paper, we present an original geometric framework to analyze the
convergence properties of gradient descent trajectories in the context of
linear neural networks. Built upon a key invariance property induced by the
network structure, we propose a conjecture called \emph{overfitting conjecture}
stating that, for almost every training data, the corresponding gradient
descent trajectory converges to a global minimum, for almost every initial
condition. This would imply that, for linear neural networks of an arbitrary
number of hidden layers, the solution achieved by simple gradient descent
algorithm is equivalent to that of least square estimation. Our first result
consists in establishing, in the case of linear networks of arbitrary depth,
convergence of gradient descent trajectories to critical points of the loss
function. Our second result is the proof of the \emph{overfitting conjecture}
in the case of single-hidden-layer linear networks with an argument based on
the notion of normal hyperbolicity and under a generic property on the training
data (i.e., holding for almost every training data).Comment: Preprint. Work in progres
Deep neural network for traffic sign recognition systems: An analysis of spatial transformers and stochastic optimisation methods
This paper presents a Deep Learning approach for traffic sign recognition systems. Several classification experiments are conducted over publicly available traffic sign datasets from Germany and Belgium using a Deep Neural Network which comprises Convolutional layers and Spatial Transformer Networks. Such trials are built to measure the impact of diverse factors with the end goal of designing a Convolutional Neural Network that can improve the state-of-the-art of traffic sign classification task. First, different adaptive and non-adaptive stochastic gradient descent optimisation algorithms such as SGD, SGD-Nesterov, RMSprop and Adam are evaluated. Subsequently, multiple combinations of Spatial Transformer Networks placed at distinct positions within the main neural network are analysed. The recognition rate of the proposed Convolutional Neural Network reports an accuracy of 99.71% in the German Traffic Sign Recognition Benchmark, outperforming previous state-of-the-art methods and also being more efficient in terms of memory requirements.Ministerio de Economía y Competitividad TIN2017-82113-C2-1-RMinisterio de Economía y Competitividad TIN2013-46801-C4-1-
Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
A central challenge to many fields of science and engineering involves
minimizing non-convex error functions over continuous, high dimensional spaces.
Gradient descent or quasi-Newton methods are almost ubiquitously used to
perform such minimizations, and it is often thought that a main source of
difficulty for these local methods to find the global minimum is the
proliferation of local minima with much higher error than the global minimum.
Here we argue, based on results from statistical physics, random matrix theory,
neural network theory, and empirical evidence, that a deeper and more profound
difficulty originates from the proliferation of saddle points, not local
minima, especially in high dimensional problems of practical interest. Such
saddle points are surrounded by high error plateaus that can dramatically slow
down learning, and give the illusory impression of the existence of a local
minimum. Motivated by these arguments, we propose a new approach to
second-order optimization, the saddle-free Newton method, that can rapidly
escape high dimensional saddle points, unlike gradient descent and quasi-Newton
methods. We apply this algorithm to deep or recurrent neural network training,
and provide numerical evidence for its superior optimization performance.Comment: The theoretical review and analysis in this article draw heavily from
arXiv:1405.4604 [cs.LG
- …