Regularization Tools for Training Large-Scale Neural Networks

Abstract

We present regularization tools for training small-and-medium as well as large-scale artificial feedforward neural networks. The determination of the weights leads to very ill-conditioned nonlinear least squares problems and regularization is often suggested to get control over the network complexity, small variance error, and nice optimization problems. The algorithms proposed solve explicitly a sequence of Tikhonov regularized nonlinear least squares problems. For small-and-medium size problems the Gauss-Newton method is applied to the regularized problem that is much more well-conditioned than the original problem, and exhibits far better convergence properties than a Levenberg-Marquardt method. Numerical results presented also confirm that the proposed implementations are more reliable and efficient than the Levenberg-Marquardt method. For large-scale problems, methods using new special purpose automatic differentiation combined with conjugate gradient methods are proposed. The alg..