An Efficient Primal-Dual Prox Method for Non-Smooth Optimization

We study the non-smooth optimization problems in machine learning, where both the loss function and the regularizer are non-smooth functions. Previous studies on efficient empirical loss minimization assume either a smooth loss function or a strongly convex regularizer, making them unsuitable for non-smooth optimization. We develop a simple yet efficient method for a family of non-smooth optimization problems where the dual form of the loss function is bilinear in primal and dual variables. We cast a non-smooth optimization problem into a minimax optimization problem, and develop a primal dual prox method that solves the minimax optimization problem at a rate of $O(1/T)$ {assuming that the proximal step can be efficiently solved}, significantly faster than a standard subgradient descent method that has an $O(1/\sqrt{T})$ convergence rate. Our empirical study verifies the efficiency of the proposed method for various non-smooth optimization problems that arise ubiquitously in machine learning by comparing it to the state-of-the-art first order methods

Reformulations of mathematical programming problems as linear complementarity problems

A family of complementarity problems are defined as extensions of the well known Linear Complementarity Problem (LCP). These are (i.) Second Linear Complementarity Problem (SLCP) which is an LCP extended by introducing further equality restrictions and unrestricted variables, (ii.) Minimum Linear Complementarity Problem (MLCP) which is an LCP with additional variables not required to be complementary and with a linear objective function which is to be minimized, (iii.) Second Minimum Linear Complementarity Problem (SMLCP) which is an MLCP but the nonnegative restriction on one of each pair of complementary variables is relaxed so that it is allowed to be unrestricted in value. A number of well known mathematical programming problems, namely quadratic programming (convex, nonconvex, pseudoconvex nonconvex), bilinear programming, game theory, zero-one integer programming, the fixed charge problem, absolute value programming, variable separable programming are reformulated as members of this family of four complementarity problems

Bilinearity rank of the cone of positive polynomials and related cones

For a proper cone K ⊂ Rn and its dual cone K the complementary slackness condition xT s = 0 defines an n-dimensional manifold C(K) in the space { (x, s) | x ∈ K, s ∈ K^* }. When K is a symmetric cone, this manifold can be described by a set of n bilinear equalities. When K is a symmetric cone, this fact translates to a set of n linearly independent bilinear identities (optimality conditions) satisfied by every (x, s) ∈ C(K). This proves to be very useful when optimizing over such cones, therefore it is natural to look for similar optimality conditions for non-symmetric cones. In this paper we define the bilinearity rank of a cone, which is the number of linearly independent bilinear identities valid for the cone, and describe a linear algebraic technique to bound this quantity. We examine several well-known cones, in particular the cone of positive polynomials P2n+1 and its dual, the closure of the moment cone M2n+1, and compute their bilinearity ranks. We show that there are exactly four linearly independent bilinear identities which hold for all (x,s) ∈ C(P2n+1), regardless of the dimension of the cones. For nonnegative polynomials over an interval or half-line there are only two linearly independent bilinear identities. These results are extended to trigonometric and exponential polynomials

Bilinearity rank of the cone of positive polynomials and related cones

For a proper cone K ⊂ Rn and its dual cone K the complementary slackness condition xT s = 0 defines an n-dimensional manifold C(K) in the space { (x, s) | x ∈ K, s ∈ K^* }. When K is a symmetric cone, this manifold can be described by a set of n bilinear equalities. When K is a symmetric cone, this fact translates to a set of n linearly independent bilinear identities (optimality conditions) satisfied by every (x, s) ∈ C(K). This proves to be very useful when optimizing over such cones, therefore it is natural to look for similar optimality conditions for non-symmetric cones. In this paper we define the bilinearity rank of a cone, which is the number of linearly independent bilinear identities valid for the cone, and describe a linear algebraic technique to bound this quantity. We examine several well-known cones, in particular the cone of positive polynomials P2n+1 and its dual, the closure of the moment cone M2n+1, and compute their bilinearity ranks. We show that there are exactly four linearly independent bilinear identities which hold for all (x,s) ∈ C(P2n+1), regardless of the dimension of the cones. For nonnegative polynomials over an interval or half-line there are only two linearly independent bilinear identities. These results are extended to trigonometric and exponential polynomials

Nearly Optimal Linear Convergence of Stochastic Primal-Dual Methods for Linear Programming

There is a recent interest on first-order methods for linear programming (LP). In this paper,we propose a stochastic algorithm using variance reduction and restarts for solving sharp primal-dual problems such as LP. We show that the proposed stochastic method exhibits a linear convergence rate for solving sharp instances with a high probability. In addition, we propose an efficient coordinate-based stochastic oracle for unconstrained bilinear problems, which has $\mathcal O(1)$ per iteration cost and improves the complexity of the existing deterministic and stochastic algorithms. Finally, we show that the obtained linear convergence rate is nearly optimal (upto $\log$ terms) for a wide class of stochastic primal dual methods