4,867 research outputs found
Barrier subgradient method
In this paper we develop a new primal-dual subgradient method for nonsmooth convex optimization problems. This scheme is based on a self-concordant barrier for the basic feasible set. It is suitable for finding approximate solutions with certain relative accuracy. We discuss some applications of this technique including fractional covering problem, maximal concurrent flow problem, semidefinite relaxations and nonlinear online optimization.convex optimization, subgradient methods, non-smooth optimization, minimax problems, saddle points, variational inequalities, stochastic optimization, black-box methods, lower complexity bounds.
Some Primal-Dual Theory for Subgradient Methods for Strongly Convex Optimization
We consider (stochastic) subgradient methods for strongly convex but
potentially nonsmooth non-Lipschitz optimization. We provide new equivalent
dual descriptions (in the style of dual averaging) for the classic subgradient
method, the proximal subgradient method, and the switching subgradient method.
These equivalences enable convergence guarantees in terms of both
their classic primal gap and a not previously analyzed dual gap for strongly
convex optimization. Consequently, our theory provides these classic methods
with simple, optimal stopping criteria and optimality certificates at no added
computational cost. Our results apply under nearly any stepsize selection and
for a range of non-Lipschitz ill-conditioned problems where the early
iterations of the subgradient method may diverge exponentially quickly (a
phenomenon which, to the best of our knowledge, no prior works address). Even
in the presence of such undesirable behaviors, our theory still ensures and
bounds eventual convergence.Comment: 29 page
An Efficient Primal-Dual Prox Method for Non-Smooth Optimization
We study the non-smooth optimization problems in machine learning, where both
the loss function and the regularizer are non-smooth functions. Previous
studies on efficient empirical loss minimization assume either a smooth loss
function or a strongly convex regularizer, making them unsuitable for
non-smooth optimization. We develop a simple yet efficient method for a family
of non-smooth optimization problems where the dual form of the loss function is
bilinear in primal and dual variables. We cast a non-smooth optimization
problem into a minimax optimization problem, and develop a primal dual prox
method that solves the minimax optimization problem at a rate of
{assuming that the proximal step can be efficiently solved}, significantly
faster than a standard subgradient descent method that has an
convergence rate. Our empirical study verifies the efficiency of the proposed
method for various non-smooth optimization problems that arise ubiquitously in
machine learning by comparing it to the state-of-the-art first order methods
A duality-based approach for distributed min-max optimization with application to demand side management
In this paper we consider a distributed optimization scenario in which a set
of processors aims at minimizing the maximum of a collection of "separable
convex functions" subject to local constraints. This set-up is motivated by
peak-demand minimization problems in smart grids. Here, the goal is to minimize
the peak value over a finite horizon with: (i) the demand at each time instant
being the sum of contributions from different devices, and (ii) the local
states at different time instants being coupled through local dynamics. The
min-max structure and the double coupling (through the devices and over the
time horizon) makes this problem challenging in a distributed set-up (e.g.,
well-known distributed dual decomposition approaches cannot be applied). We
propose a distributed algorithm based on the combination of duality methods and
properties from min-max optimization. Specifically, we derive a series of
equivalent problems by introducing ad-hoc slack variables and by going back and
forth from primal and dual formulations. On the resulting problem we apply a
dual subgradient method, which turns out to be a distributed algorithm. We
prove the correctness of the proposed algorithm and show its effectiveness via
numerical computations.Comment: arXiv admin note: substantial text overlap with arXiv:1611.0916
Primal Recovery from Consensus-Based Dual Decomposition for Distributed Convex Optimization
Dual decomposition has been successfully employed in a variety of distributed
convex optimization problems solved by a network of computing and communicating
nodes. Often, when the cost function is separable but the constraints are
coupled, the dual decomposition scheme involves local parallel subgradient
calculations and a global subgradient update performed by a master node. In
this paper, we propose a consensus-based dual decomposition to remove the need
for such a master node and still enable the computing nodes to generate an
approximate dual solution for the underlying convex optimization problem. In
addition, we provide a primal recovery mechanism to allow the nodes to have
access to approximate near-optimal primal solutions. Our scheme is based on a
constant stepsize choice and the dual and primal objective convergence are
achieved up to a bounded error floor dependent on the stepsize and on the
number of consensus steps among the nodes
- …