Planar Conjugate Gradient Algorithm for Large-Scale Unconstrained Optimization, Part 1: Theory

Abstract. In this paper, we describe an application of the planar conjugate gradient method introduced in Part 1 (Ref. 1) and aimed at solving indefinite nonsingular sets of linear equations. We prove that it can be used fruitfully within optimization frameworks; in particular, we present a globally convergent truncated Newton scheme, which uses the above planar method for solving the Newton equation. Finally, our approach is tested over several problems from the CUTE collection (Ref. 2). Key Words. Large-scale unconstrained optimization, truncated Newto

Bridging the gap between Trust–Region Methods (TRMs) and Linesearch Based Methods (LBMs) for Nonlinear Programming: quadratic sub–problems

We consider the solution of a recurrent sub–problem within both constrained and unconstrained Nonlinear Programming: namely the minimization of a quadratic function subject to linear constraints. This problem appears in a number of LBM frameworks, and to some extent it reveals a close analogy with the solution of trust–region sub–problems. In particular, we refer to a structured quadratic problem where five linear inequality constraints are included. We show that our proposal retains an appreciable versatility, despite its particular structure, so that a number of different real instances may be reformulated following the pattern in our proposal. Moreover, we detail how to compute an exact global solution of our quadratic sub–problem, exploiting first order KKT conditions

A Framework of Conjugate Direction Methods for Symmetric Linear Systems in Optimization

In this paper, we introduce a parameter-dependent class of Krylov-based methods, namely Conjugate Directions (Formula Presented.), for the solution of symmetric linear systems. We give evidence that, in our proposal, we generate sequences of conjugate directions, extending some properties of the standard conjugate gradient (CG) method, in order to preserve the conjugacy. For specific values of the parameters in our framework, we obtain schemes equivalent to both the CG and the scaled-CG. We also prove the finite convergence of the algorithms in (Formula Presented.), and we provide some error analysis. Finally, preconditioning is introduced for (Formula Presented.), and we show that standard error bounds for the preconditioned CG also hold for the preconditioned (Formula Presented.).In this paper, we introduce a parameter-dependent class of Krylov-based methods, namely Conjugate Directions , for the solution of symmetric linear systems. We give evidence that, in our proposal, we generate sequences of conjugate directions, extending some properties of the standard conjugate gradient (CG) method, in order to preserve the conjugacy. For specific values of the parameters in our framework, we obtain schemes equivalent to both the CG and the scaled-CG. We also prove the finite convergence of the algorithms in , and we provide some error analysis. Finally, preconditioning is introduced for , and we show that standard error bounds for the preconditioned CG also hold for the preconditioned

Preconditioning Newton-Krylov Methods in Non-Convex Large Scale Optimization

We consider an iterative preconditioning technique for non-convex large scale optimization. First, we refer to the solution of large scale indefinite linear systems by using a Krylov subspace method, and describe the iterative construction of a preconditioner which does not involve matrices products or matrices storage. The set of directions generated by the Krylov subspace method is used, as by product, to provide an approximate inverse preconditioner. Then, we experience our preconditioner within Truncated Newton schemes for large scale unconstrained optimization, where we generalize the truncation rule by Nash–Sofer (Oper. Res. Lett. 9:219–221, 1990) to the indefinite case, too. We use a Krylov subspace method to both approximately solve the Newton equation and to construct the preconditioner to be used at the current outer iteration. An extensive numerical experience shows that the proposed preconditioning strategy, compared with the unpreconditioned strategy and PREQN (Morales and Nocedal in SIAM J. Optim. 10:1079–1096, 2000), may lead to a reduction of the overall inner iterations. Finally, we show that our proposal has some similarities with the Limited Memory Preconditioners (Gratton et al. in SIAM J. Optim. 21:912–935, 2011)

A Newton-MR algorithm with complexity guarantees for nonconvex smooth unconstrained optimization

In this paper, we consider variants of Newton-MR algorithm for solving unconstrained, smooth, but non-convex optimization problems. Unlike the overwhelming majority of Newton-type methods, which rely on conjugate gradient algorithm as the primary workhorse for their respective sub-problems, Newton-MR employs minimum residual (MINRES) method. Recently, it has been established that MINRES has inherent ability to detect non-positive curvature directions as soon as they arise and certain useful monotonicity properties will be satisfied before such detection. We leverage these recent results and show that our algorithms come with desirable properties including competitive first and second-order worst-case complexities. Numerical examples demonstrate the performance of our proposed algorithms

Data Filtering for Cluster Analysis by ℓ0\ell_0-Norm Regularization

A data filtering method for cluster analysis is proposed, based on minimizing a least squares function with a weighted $\ell_0$-norm penalty. To overcome the discontinuity of the objective function, smooth non-convex functions are employed to approximate the $\ell_0$-norm. The convergence of the global minimum points of the approximating problems towards global minimum points of the original problem is stated. The proposed method also exploits a suitable technique to choose the penalty parameter. Numerical results on synthetic and real data sets are finally provided, showing how some existing clustering methods can take advantages from the proposed filtering strategy.Comment: Optimization Letters (2017

Planar methods and grossone for the Conjugate Gradient breakdown in nonlinear programming

This paper deals with an analysis of the Conjugate Gradient (CG) method (Hestenes and Stiefel in J Res Nat Bur Stand 49:409-436, 1952), in the presence of degenerates on indefinite linear systems. Several approaches have been proposed in the literature to issue the latter drawback in optimization frameworks, including reformulating the original linear system or recurring to approximately solving it. All the proposed alternatives seem to rely on algebraic considerations, and basically pursue the idea of improving numerical efficiency. In this regard, here we sketch two separate analyses for the possible CG degeneracy. First, we start detailing a more standard algebraic viewpoint of the problem, suggested by planar methods. Then, another algebraic perspective is detailed, relying on a novel recently proposed theory, which includes an additional number, namely grossone. The use of grossone allows to work numerically with infinities and infinitesimals. The results obtained using the two proposed approaches perfectly match, showing that grossone may represent a fruitful and promising tool to be exploited within Nonlinear Programming

A Novel Gradient Methodology with Economical Objective Function Evaluations for Data Science Applications

Gradient methods are experiencing a growth in methodological and theoretical developments owing to the challenges of optimization problems arising in data science. Focusing on data science applications with expensive objective function evaluations yet inexpensive gradient function evaluations, gradient methods that never make objective function evaluations are either being rejuvenated or actively developed. However, as we show, such gradient methods are all susceptible to catastrophic divergence under realistic conditions for data science applications. In light of this, gradient methods which make use of objective function evaluations become more appealing, yet, as we show, can result in an exponential increase in objective evaluations between accepted iterates. As a result, existing gradient methods are poorly suited to the needs of optimization problems arising from data science. In this work, we address this gap by developing a generic methodology that economically uses objective function evaluations in a problem-driven manner to prevent catastrophic divergence and avoid an explosion in objective evaluations between accepted iterates. Our methodology allows for specific procedures that can make use of specific step size selection methodologies or search direction strategies, and we develop a novel step size selection methodology that is well-suited to data science applications. We show that a procedure resulting from our methodology is highly competitive with standard optimization methods on CUTEst test problems. We then show a procedure resulting from our methodology is highly favorable relative to standard optimization methods on optimization problems arising in our target data science applications. Thus, we provide a novel gradient methodology that is better suited to optimization problems arising in data science.Comment: 52 pages, 14 figures, 7 tables, 14 algorithm