Stochastic Programming with Probability

In this work we study optimization problems subject to a failure constraint. This constraint is expressed in terms of a condition that causes failure, representing a physical or technical breakdown. We formulate the problem in terms of a probability constraint, where the level of "confidence" is a modelling parameter and has the interpretation that the probability of failure should not exceed that level. Application of the stochastic Arrow-Hurwicz algorithm poses two difficulties: one is structural and arises from the lack of convexity of the probability constraint, and the other is the estimation of the gradient of the probability constraint. We develop two gradient estimators with decreasing bias via a convolution method and a finite difference technique, respectively, and we provide a full analysis of convergence of the algorithms. Convergence results are used to tune the parameters of the numerical algorithms in order to achieve best convergence rates, and numerical results are included via an example of application in finance

Nonconvex Generalization of ADMM for Nonlinear Equality Constrained Problems

The ever-increasing demand for efficient and distributed optimization algorithms for large-scale data has led to the growing popularity of the Alternating Direction Method of Multipliers (ADMM). However, although the use of ADMM to solve linear equality constrained problems is well understood, we lacks a generic framework for solving problems with nonlinear equality constraints, which are common in practical applications (e.g., spherical constraints). To address this problem, we are proposing a new generic ADMM framework for handling nonlinear equality constraints, neADMM. After introducing the generalized problem formulation and the neADMM algorithm, the convergence properties of neADMM are discussed, along with its sublinear convergence rate $o(1/k)$, where $k$ is the number of iterations. Next, two important applications of neADMM are considered and the paper concludes by describing extensive experiments on several synthetic and real-world datasets to demonstrate the convergence and effectiveness of neADMM compared to existing state-of-the-art methods

Super-Linear Convergence of Dual Augmented-Lagrangian Algorithm for Sparsity Regularized Estimation

We analyze the convergence behaviour of a recently proposed algorithm for regularized estimation called Dual Augmented Lagrangian (DAL). Our analysis is based on a new interpretation of DAL as a proximal minimization algorithm. We theoretically show under some conditions that DAL converges super-linearly in a non-asymptotic and global sense. Due to a special modelling of sparse estimation problems in the context of machine learning, the assumptions we make are milder and more natural than those made in conventional analysis of augmented Lagrangian algorithms. In addition, the new interpretation enables us to generalize DAL to wide varieties of sparse estimation problems. We experimentally confirm our analysis in a large scale $\ell_1$-regularized logistic regression problem and extensively compare the efficiency of DAL algorithm to previously proposed algorithms on both synthetic and benchmark datasets.Comment: 51 pages, 9 figure

Projection methods in conic optimization

There exist efficient algorithms to project a point onto the intersection of a convex cone and an affine subspace. Those conic projections are in turn the work-horse of a range of algorithms in conic optimization, having a variety of applications in science, finance and engineering. This chapter reviews some of these algorithms, emphasizing the so-called regularization algorithms for linear conic optimization, and applications in polynomial optimization. This is a presentation of the material of several recent research articles; we aim here at clarifying the ideas, presenting them in a general framework, and pointing out important techniques