On the global convergence of the inexact semi-smooth Newton method for absolute value equation

In this paper, we investigate global convergence properties of the inexact nonsmooth Newton method for solving the system of absolute value equations (AVE). Global $Q$-linear convergence is established under suitable assumptions. Moreover, we present some numerical experiments designed to investigate the practical viability of the proposed scheme

Inexact Newton method with feasible inexact projections for solving constrained smooth and nonsmooth equations

In this paper, we propose a new method that combines the inexact Newton method with a procedure to obtain a feasible inexact projection for solving constrained smooth and nonsmooth equations. The local convergence theorems are established under the assumption of smoothness or semismoothness of the function that defines the equation and its regularity at the solution. In particular, we show that a sequence generated by the method converges to a solution with linear, superlinear, or quadratic rate, under suitable conditions. Moreover, some numerical experiments are reported to illustrate the practical behavior of the proposed method

Level-set methods for convex optimization

Convex optimization problems arising in applications often have favorable objective functions and complicated constraints, thereby precluding first-order methods from being immediately applicable. We describe an approach that exchanges the roles of the objective and constraint functions, and instead approximately solves a sequence of parametric level-set problems. A zero-finding procedure, based on inexact function evaluations and possibly inexact derivative information, leads to an efficient solution scheme for the original problem. We describe the theoretical and practical properties of this approach for a broad range of problems, including low-rank semidefinite optimization, sparse optimization, and generalized linear models for inference.Comment: 38 page

A semi-smooth Newton method for projection equations and linear complementarity problems with respect to the second order cone

In this paper a special semi-smooth equation associated to the second order cone is studied. It is shown that, under mild assumptions, the semi-smooth Newton method applied to this equation is well-defined and the generated sequence is globally and Q-linearly convergent to a solution. As an application, the obtained results are used to study the linear second order cone complementarity problem, with special emphasis on the particular case of positive definite matrices. Moreover, some computational experiments designed to investigate the practical viability of the method are presented.Comment: 18 page

Inexact Newton Methods for Stochastic Nonconvex Optimization with Applications to Neural Network Training

We study stochastic inexact Newton methods and consider their application in nonconvex settings. Building on the work of [R. Bollapragada, R. H. Byrd, and J. Nocedal, IMA Journal of Numerical Analysis, 39 (2018), pp. 545--578] we derive bounds for convergence rates in expected value for stochastic low rank Newton methods, and stochastic inexact Newton Krylov methods. These bounds quantify the errors incurred in subsampling the Hessian and gradient, as well as in approximating the Newton linear solve, and in choosing regularization and step length parameters. We deploy these methods in training convolutional autoencoders for the MNIST and CIFAR10 data sets. Numerical results demonstrate that, relative to first order methods, these stochastic inexact Newton methods often converge faster, are more cost-effective, and generalize better

An inexact Douglas-Rachford splitting method for solving absolute value equations

The last two decades witnessed the increasing of the interests on the absolute value equations (AVE) of finding $x\in\mathbb{R}^n$ such that $Ax-|x|-b=0$, where $A\in \mathbb{R}^{n\times n}$ and $b\in \mathbb{R}^n$. In this paper, we pay our attention on designing efficient algorithms. To this end, we reformulate AVE to a generalized linear complementarity problem (GLCP), which, among the equivalent forms, is the most economical one in the sense that it does not increase the dimension of the variables. For solving the GLCP, we propose an inexact Douglas-Rachford splitting method which can adopt a relative error tolerance. As a consequence, in the inner iteration processes, we can employ the LSQR method ([C.C. Paige and M.A. Saunders, ACM Trans. Mathe. Softw. (TOMS), 8 (1982), pp. 43--71]) to find a qualified approximate solution for each subproblem, which makes the cost per iteration very low. We prove the convergence of the algorithm and establish its global linear rate of convergence. Comparing results with the popular algorithms such as the exact generalized Newton method [O.L. Mangasarian, Optim. Lett., 1 (2007), pp. 3--8], the inexact semi-smooth Newton method [J.Y.B. Cruz, O.P. Ferreira and L.F. Prudente, Comput. Optim. Appl., 65 (2016), pp. 93--108] and the exact SOR-like method [Y.-F. Ke and C.-F. Ma, Appl. Math. Comput., 311 (2017), pp. 195--202] are reported, which indicate that the proposed algorithm is very promising. Moreover, our method also extends the range of numerically solvable of the AVE; that is, it can deal with not only the case that $\|A^{-1}\|<1$, the commonly used in those existing literature, but also the case where $\|A^{-1}\|=1$.Comment: 25 pages, 3 figures, 3 table

Physarum Dynamics and Optimal Transport for Basis Pursuit

We study the connections between Physarum Dynamics and Dynamic Monge Kantorovich (DMK) Optimal Transport algorithms for the solution of Basis Pursuit problems. We show the equivalence between these two models and unveil their dynamic character by showing existence and uniqueness of the solution for all times and constructing a Lyapunov functional with negative Lie-derivative that drives the large-time convergence. We propose a discretization of the equation by means of a combination of implicit time-stepping and Newton method yielding an efficient and robust method for the solution of general basis pursuit problems. Several numerical experiments run on literature benchmark problems are used to show the accuracy, efficiency, and robustness of the proposed method

A class of inexact modified Newton-type iteration methods for solving the generalized absolute value equations

In Wang et al. (J. Optim. Theory Appl., \textbf{181}: 216--230, 2019), a class of effective modified Newton-tpye (MN) iteration methods are proposed for solving the generalized absolute value equations (GAVE) and it has been found that the MN iteration method involves the classical Picard iteration method as a special case. In the present paper, it will be claimed that a Douglas-Rachford splitting method for AVE is also a special case of the MN method. In addition, a class of inexact MN (IMN) iteration methods are developed to solve GAVE. Linear convergence of the IMN method is established and some specific sufficient conditions are presented for symmetric positive definite coefficient matrix. Numerical results are given to demonstrate the efficiency of the IMN iteration method.Comment: 14 pages, 4 table

An Optimal Block Diagonal Preconditioner for Heterogeneous Saddle Point Problems in Phase Separation

The phase separation processes are typically modeled by Cahn-Hilliard equations. This equation was originally introduced to model phase separation in binary alloys, where phase stands for concentration of different components in alloy. When the binary alloy under preparation is subjected to a rapid reduction in temperature below a critical temperature, it has been experimentally observed that the concentration changes from a mixed state to a visibly distinct spatially separated two phase for binary alloy. This rapid reduction in the temperature, the so-called "deep quench limit", is modeled effectively by obstacle potential. The discretization of Cahn-Hilliard equation with obstacle potential leads to a block $2 \times 2$ {\em non-linear} system, where the $(1,1)$ block has a non-linear and non-smooth term. Recently a globally convergent Newton Schur method was proposed for the non-linear Schur complement corresponding to this non-linear system. The proposed method is similar to an inexact active set method in the sense that the active sets are first approximately identified by solving a quadratic obstacle problem corresponding to the $(1,1)$ block of the block $2 \times 2$ system, and later solving a reduced linear system by annihilating the rows and columns corresponding to identified active sets. For solving the quadratic obstacle problem, various optimal multigrid like methods have been proposed. In this paper, we study a non-standard norm that is equivalent to applying a block diagonal preconditioner to the reduced linear systems. Numerical experiments confirm the optimality of the solver and convergence independent of problem parameters on sufficiently fine mesh.Comment: 2 figure

Implementation of the ADMM to Parabolic Optimal Control Problems with Control Constraints and Beyond

Parabolic optimal control problems with control constraints are generally challenging, from either theoretical analysis or algorithmic design perspectives. Conceptually, the well-known alternating direction method of multipliers (ADMM) can be directly applied to such a problem. An attractive advantage of this direct ADMM application is that the control constraint can be untied from the parabolic PDE constraint; these two inherently different constraints thus can be treated individually in iterations. At each iteration of the ADMM, the main computation is for solving an unconstrained parabolic optimal control problem. Because of its high dimensionality after discretization, the unconstrained parabolic optimal control problem at each iteration can be solved only inexactly by implementing certain numerical scheme internally and thus a two-layer nested iterative scheme is required. It then becomes important to find an easily implementable and efficient inexactness criterion to execute the internal iterations, and to prove the overall convergence rigorously for the resulting two-layer nested iterative scheme. To implement the ADMM efficiently, we propose an inexactness criterion that is independent of the mesh size of the involved discretization, and it can be executed automatically with no need to set empirically perceived constant accuracy a prior. The inexactness criterion turns out to allow us to solve the resulting unconstrained optimal control problems to medium or even low accuracy and thus saves computation significantly, yet convergence of the overall two-layer nested iterative scheme can be still guaranteed rigorously. Efficiency of this ADMM implementation is promisingly validated by preliminary numerical results. Our methodology can also be extended to a range of optimal control problems constrained by other linear PDEs such as elliptic equations and hyperbolic equations