Using the Nonlinear L-curve and its Dual

The L-curve has been used on linear inverse problems for a long time. We generalize the L-curve to the nonlinear finite dimensional setting and introduce another most useful dual curve. The analytic and geometrical properties of these curves are derived together with a discussion on their use in algorithms. Key words: Nonlinear least squares, optimization, L-curve, regularization, Gauss-Newton method 1 Introduction Inverse problems appear in many different engineering applications. An inverse problem consists of a direct problem and some unknown function(s) or parameter(s). In many cases the solution does not depend continuously on the unknown quantities and the problem is ill-posed. A typical ill-posed problem is when the task is to determine these unkowns given measured, inexact, data. Given such an ill-posed problem it is a good idea to reformulate the original problem into a well-posed problem giving a solution that is not too large and with a small residual. 1.1 An example from..

Algorithms for using the nonlinear L-curve

When using Tikhonov regularization for finite dimensional ill-posed problems there is a problem dependent choice of the regularization parameter. We present general tools for determining a proper regularization parameter that are based on the nonlinear L-curve and the associated (dual) a-curve. Given approximations of the solution of the Tikhonov problem we define upper and lower piecewise linear approximations of the L- and a-curve called shadow curves. These shadow curves are thouroughly investigated. Finally, we present ways to update the shadow curves and their use to identify good regularized solutions. AMS(MOS) subject classification: 62J05, 65U05 Key words: Nonlinear least squares, optimization, L-curve, regularization, Gauss-Newton method Contents 1 Introduction 1 2 Local results 3 3 Shadow curves 4 3.1 Basic ideas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 3.2 The polygon shadow curves . . . . . . . . . . . . . . . . . . . . . 4 3.3 The connection betwe..

Gauss-Newton Based Algorithms For Constrained Nonlinear Least Squares Problems

This paper has two main purposes: To discuss general principles for a reliable and efficient numerical method based on the linearizations given in (2) and (3) and to present algorithms for which we have developed software. Our practical experience of methods of this kind are closely related to and, to some extent, limited by the experimentation needed to derive that software. Still there are two features which, we think, should be included in most algorithms based on the linearizations (2) and (3): 1) If the least squares problem turns out to have a large relative residual close to the solution - see the definition in section 2.8 - there has to be a switch to a method that incorporates second order information. The algorithm should in this sense be a hybrid algorithm. 2) It seems to be both convenient and natural to use line search on a quadratic merit function for a constrained nonlinear least squares problem. The merit function will be described and investigated in section 4. 1.1 Equality constrained problem

Regularization Methods for Nonlinear Least Squares Problems. Part I: Exactly Rank-deficient Problems

In two papers, we develop theory and methods for regularization of nonlinear least squares problems to minimize the Euclidean norm of f(x) 2 ! m ; x 2 ! n . In this first paper, we consider the case where the Jacobian is exactly rank-deficient. Then due to the constant rank theorem the function f(x) = h(z(x)), where z 2 ! r and r ! min(m; n) has the property of being exactly rank-deficient almost everywhere. This composed function simplifies derivations and reveals that such a nonlinear least squares problem can be formulated as a nonlinear minimum norm problem. We propose two regularization methods to solve the nonlinear minimum norm problem: A Gauss-Newton minimum norm method and a Tikhonov regularization method. It is proved that both methods converge to a minimum norm solution. Local and asymptotic convergence rates are thoroughly investigated and it is shown that the convergence depends on the curvatures both in the function space and in the parameter space. Numerical result..

On condition numbers and algorithms for determining a rigid body movement. BIT

Abstract Using a set of landmarks to represent a rigid body, a rotation of the body can be determined in least squares sense as the solution of an orthogonal Procrustes problem. We discuss some geometrical properties of the condition number for the problem of determining the orthogonal matrix representing the rotation. It is shown that the condition number critically depends on the configuration of the landmarks. The problem is also reformulated as an unconstrained nonlinear least squares problem and the condition number is related to the geometry of such problems. In the common 3-D case, the movement can be represented by using a screw axis. Also the condition numbers for the problem of determining the screw axis representation are shown to closely depend on the configuration of the landmarks. The condition numbers are finally used to show that the used algorithms are stable

Regularization Methods for Nonlinear Least Squares Problems. Part II: Almost Rank-Deficiency

. A nonlinear least squares problem is almost rank deficient at a local minimum if there is a large gap in the singular values of the Jacobian and at least one singular value is small. We analyze the almost rank deficient problem giving the relevant KKT-conditions and propose two methods based on truncation and Tikhonov regularization. Our approach is based on techniques from linear algebra and nonlinear optimization. This enables us to develop a local and asymptotic convergence theory based on second order information for Gauss-Newton like methods applied to the nonlinear truncated and Tikhonov regularized problems with known regularization parameter. Finally, we test the methods on artificial problems where we are able to choose the singular values and the nonlinearity of the problem making it possible to show the different features of the problem and the methods. The method based on Tikhonov regularization is more generally applicable to illposed problems having no gap in the singul..

Analyzing the nonlinear L-curve

One way of solving an ill-posed or ill-conditioned problem is by regularization. We consider the regularization of the finite dimensional nonlinear system of equations f(x) = 0. The regularization is performed by formulating a Tikhonov problem with an unknnown regularization parameter. The nonlinear L-curve is the size of the solution considered as a function of the size of the residual (when the regularization parameter is changed). The dual curve connected to the nonlinear L-curve is defined as the value of the minimization function as a function of the regularization parameter. We show that the L-curve is a strictly decreasing convex function and its dual is strictly increasing and concave. The connection between these two curves and other natural regularization formulations is presented as well as a thourough analysis concerning logarithmic scales and corners of the L-curve

Regularization Tools for Training Large-Scale Neural Networks

We present regularization tools for training small-and-medium as well as large-scale artificial feedforward neural networks. The determination of the weights leads to very ill-conditioned nonlinear least squares problems and regularization is often suggested to get control over the network complexity, small variance error, and nice optimization problems. The algorithms proposed solve explicitly a sequence of Tikhonov regularized nonlinear least squares problems. For small-and-medium size problems the Gauss-Newton method is applied to the regularized problem that is much more well-conditioned than the original problem, and exhibits far better convergence properties than a Levenberg-Marquardt method. Numerical results presented also confirm that the proposed implementations are more reliable and efficient than the Levenberg-Marquardt method. For large-scale problems, methods using new special purpose automatic differentiation combined with conjugate gradient methods are proposed. The alg..

Interpretation and Practical Use of Error Propagation Matrices

Di#erent kinds of linear systems of equations, Ax = b where A , often occur when solving real-world problems. The singular value decomposition of A can then be used to construct error propagation matrices and by use of these it is possible to investigate how changes in both the matrix A and vector b a#ect the solution x. Theoretical error bounds based on condition numbers indicate the worst case but the use of experimental error analysis makes it possible to also have information about the e#ect of a more limited amount of perturbations and are in that sense more realistic. In this paper it is shown how the e#ect of perturbations can be analyzed by a semiexperimental analysis for the case m = n and m > n. The analysis combines the theory of the error propagation matrices with an experimental error analysis based on randomly generated perturbations that takes the structure of A into account. Keywords : pseudoinverse solution, perturbation theory, singular value decomposition, experimental error analysis Paper III Contents

Regularization Tools for Training Large Feed-Forward Neural Networks Using Automatic Differentiation

this paper, we propose optimization methods explicitly applied to the nonlinear regularized problem for large-scale problems. To be specific, we formulate and solve nonlinear Tikhonov regularized problems. In [12] it was shown theoretically and practically that this approach is superior to standard optimization regularizatio