Lagrange Multipliers, (Exact) Regularization and Error Bounds for Monotone Variational Inequalities

We examine two central regularization strategies for monotone variational inequalities, the first a direct regularization of the operative monotone mapping, and the second via regularization of the associated dual gap function. A key link in the relationship between the solution sets to these various regularized problems is the idea of exact regularization, which, in turn, is fundamentally associated with the existence of Lagrange multipliers for the regularized variational inequality. A regularization is said to be exact if a solution to the regularized problem is a solution to the unregularized problem for all parameters beyond a certain value. The Lagrange multipliers corresponding to a particular regularization of a variational inequality, on the other hand, are defined via the dual gap function. Our analysis suggests various conceptual, iteratively regularized numerical schemes, for which we provide error bounds, and hence stopping criteria, under the additional assumption that the solution set to the unregularized problem is what we call weakly sharp of order greater than one.Comment: Updated version after referee comments. 34 pages, 1 table, 20 reference

On stochastic and deterministic quasi-Newton methods for non-Strongly convex optimization: Asymptotic convergence and rate analysis

Motivated by applications arising from large scale optimization and machine learning, we consider stochastic quasi-Newton (SQN) methods for solving unconstrained convex optimization problems. The convergence analysis of the SQN methods, both full and limited-memory variants, require the objective function to be strongly convex. However, this assumption is fairly restrictive and does not hold for applications such as minimizing the logistic regression loss function. To the best of our knowledge, no rate statements currently exist for SQN methods in the absence of such an assumption. Also, among the existing first-order methods for addressing stochastic optimization problems with merely convex objectives, those equipped with provable convergence rates employ averaging. However, this averaging technique has a detrimental impact on inducing sparsity. Motivated by these gaps, the main contributions of the paper are as follows: (i) Addressing large scale stochastic optimization problems, we develop an iteratively regularized stochastic limited-memory BFGS (IRS-LBFGS) algorithm, where the stepsize, regularization parameter, and the Hessian inverse approximation matrix are updated iteratively. We establish the convergence to an optimal solution of the original problem both in an almost-sure and mean senses. We derive the convergence rate in terms of the objective function's values and show that it is of the order $\mathcal{O}\left(k^{-\left(\frac{1}{3}-\epsilon\right)}\right)$, where $\epsilon$ is an arbitrary small positive scalar; (ii) In deterministic regime, we show that the regularized limited-memory BFGS algorithm displays a rate of the order $\mathcal{O}\left(\frac{1}{k^{1 -\epsilon'}}\right)$, where $\epsilon'$ is an arbitrary small positive scalar. We present our numerical experiments performed on a large scale text classification problem

An iterative regularized mirror descent method for ill-posed nondifferentiable stochastic optimization

A wide range of applications arising in machine learning and signal processing can be cast as convex optimization problems. These problems are often ill-posed, i.e., the optimal solution lacks a desired property such as uniqueness or sparsity. In the literature, to address ill-posedness, a bilevel optimization problem is considered where the goal is to find among optimal solutions of the inner level optimization problem, a solution that minimizes a secondary metric, i.e., the outer level objective function. In addressing the resulting bilevel model, the convergence analysis of most existing methods is limited to the case where both inner and outer level objectives are differentiable deterministic functions. While these assumptions may not hold in big data applications, to the best of our knowledge, no solution method equipped with complexity analysis exists to address presence of uncertainty and nondifferentiability in both levels in this class of problems. Motivated by this gap, we develop a first-order method called Iterative Regularized Stochastic Mirror Descent (IR-SMD). We establish the global convergence of the iterate generated by the algorithm to the optimal solution of the bilevel problem in an almost sure and a mean sense. We derive a convergence rate of ${\cal O}\left(1/N^{0.5-\delta}\right)$ for the inner level problem, where $\delta>0$ is an arbitrary small scalar. Numerical experiments for solving two classes of bilevel problems, including a large scale binary text classification application, are presented