Faster and Non-ergodic O(1/K) Stochastic Alternating Direction Method of Multipliers

We study stochastic convex optimization subjected to linear equality constraints. Traditional Stochastic Alternating Direction Method of Multipliers and its Nesterov's acceleration scheme can only achieve ergodic O(1/\sqrt{K}) convergence rates, where K is the number of iteration. By introducing Variance Reduction (VR) techniques, the convergence rates improve to ergodic O(1/K). In this paper, we propose a new stochastic ADMM which elaborately integrates Nesterov's extrapolation and VR techniques. We prove that our algorithm can achieve a non-ergodic O(1/K) convergence rate which is optimal for separable linearly constrained non-smooth convex problems, while the convergence rates of VR based ADMM methods are actually tight O(1/\sqrt{K}) in non-ergodic sense. To the best of our knowledge, this is the first work that achieves a truly accelerated, stochastic convergence rate for constrained convex problems. The experimental results demonstrate that our algorithm is significantly faster than the existing state-of-the-art stochastic ADMM methods

An Explicit Rate Bound for the Over-Relaxed ADMM

The framework of Integral Quadratic Constraints of Lessard et al. (2014) reduces the computation of upper bounds on the convergence rate of several optimization algorithms to semi-definite programming (SDP). Followup work by Nishihara et al. (2015) applies this technique to the entire family of over-relaxed Alternating Direction Method of Multipliers (ADMM). Unfortunately, they only provide an explicit error bound for sufficiently large values of some of the parameters of the problem, leaving the computation for the general case as a numerical optimization problem. In this paper we provide an exact analytical solution to this SDP and obtain a general and explicit upper bound on the convergence rate of the entire family of over-relaxed ADMM. Furthermore, we demonstrate that it is not possible to extract from this SDP a general bound better than ours. We end with a few numerical illustrations of our result and a comparison between the convergence rate we obtain for the ADMM with known convergence rates for the Gradient Descent.Comment: IEEE International Symposium on Information Theory (ISIT), 201

One Mirror Descent Algorithm for Convex Constrained Optimization Problems with non-standard growth properties

The paper is devoted to a special Mirror Descent algorithm for problems of convex minimization with functional constraints. The objective function may not satisfy the Lipschitz condition, but it must necessarily have the Lipshitz-continuous gradient. We assume, that the functional constraint can be non-smooth, but satisfying the Lipschitz condition. In particular, such functionals appear in the well-known Truss Topology Design problem. Also we have applied the technique of restarts in the mentioned version of Mirror Descent for strongly convex problems. Some estimations for a rate of convergence are investigated for considered Mirror Descent algorithms.Comment: 12 page

Potential-Function Proofs for First-Order Methods

This note discusses proofs for convergence of first-order methods based on simple potential-function arguments. We cover methods like gradient descent (for both smooth and non-smooth settings), mirror descent, and some accelerated variants

Adaptive Importance Sampling via Stochastic Convex Programming

We show that the variance of the Monte Carlo estimator that is importance sampled from an exponential family is a convex function of the natural parameter of the distribution. With this insight, we propose an adaptive importance sampling algorithm that simultaneously improves the choice of sampling distribution while accumulating a Monte Carlo estimate. Exploiting convexity, we prove that the method's unbiased estimator has variance that is asymptotically optimal over the exponential family

Accelerated First-Order Methods for Hyperbolic Programming

A framework is developed for applying accelerated methods to general hyperbolic programming, including linear, second-order cone, and semidefinite programming as special cases. The approach replaces a hyperbolic program with a convex optimization problem whose smooth objective function is explicit, and for which the only constraints are linear equations (one more linear equation than for the original problem). Virtually any first-order method can be applied. Iteration bounds for a representative accelerated method are derived.Comment: A (serious) typo in specifying the main algorithm has been corrected, and suggestions made by referees have been addressed (submitted to Mathematical Programming

Convergence of first-order methods via the convex conjugate

This paper gives a unified and succinct approach to the $O(1/\sqrt{k}), O(1/k),$ and $O(1/k^2)$ convergence rates of the subgradient, gradient, and accelerated gradient methods for unconstrained convex minimization. In the three cases the proof of convergence follows from a generic bound defined by the convex conjugate of the objective function

Convex integer minimization in fixed dimension

We show that minimizing a convex function over the integer points of a bounded convex set is polynomial in fixed dimension.Comment: submitte

A note on alternating minimization algorithms: Bregman frame

In this paper, we propose a Bregman frame for several classical alternating minimization algorithms. In the frame, these algorithms have uniform mathematical formulation. We also present convergence analysis for the frame algorithm. Under the Kurdyka-Lojasiewicz property, stronger convergence is obtained

Massive MIMO Multicast Beamforming Via Accelerated Random Coordinate Descent

One key feature of massive multiple-input multiple-output systems is the large number of antennas and users. As a result, reducing the computational complexity of beamforming design becomes imperative. To this end, the goal of this paper is to achieve a lower complexity order than that of existing beamforming methods, via the parallel accelerated random coordinate descent (ARCD). However, it is known that ARCD is only applicable when the problem is convex, smooth, and separable. In contrast, the beamforming design problem is nonconvex, nonsmooth, and nonseparable. Despite these challenges, this paper shows that it is possible to incorporate ARCD for multicast beamforming by leveraging majorization minimization and strong duality. Numerical results show that the proposed method reduces the execution time by one order of magnitude compared to state-of-the-art methods.Comment: IEEE ICASSP'19, Brighton, UK, May 201