An Infeasible-Point Subgradient Method Using Adaptive Approximate Projections

We propose a new subgradient method for the minimization of nonsmooth convex functions over a convex set. To speed up computations we use adaptive approximate projections only requiring to move within a certain distance of the exact projections (which decreases in the course of the algorithm). In particular, the iterates in our method can be infeasible throughout the whole procedure. Nevertheless, we provide conditions which ensure convergence to an optimal feasible point under suitable assumptions. One convergence result deals with step size sequences that are fixed a priori. Two other results handle dynamic Polyak-type step sizes depending on a lower or upper estimate of the optimal objective function value, respectively. Additionally, we briefly sketch two applications: Optimization with convex chance constraints, and finding the minimum l1-norm solution to an underdetermined linear system, an important problem in Compressed Sensing.Comment: 36 pages, 3 figure

Subsampling Algorithms for Semidefinite Programming

We derive a stochastic gradient algorithm for semidefinite optimization using randomization techniques. The algorithm uses subsampling to reduce the computational cost of each iteration and the subsampling ratio explicitly controls granularity, i.e. the tradeoff between cost per iteration and total number of iterations. Furthermore, the total computational cost is directly proportional to the complexity (i.e. rank) of the solution. We study numerical performance on some large-scale problems arising in statistical learning.Comment: Final version, to appear in Stochastic System

Distributed Optimization for Coordinated Beamforming in Multi-Cell Multigroup Multicast Systems: Power Minimization and SINR Balancing

This paper considers coordinated multicast beamforming in a multi-cell multigroup multiple-input single-output system. Each base station (BS) serves multiple groups of users by forming a single beam with common information per group. We propose centralized and distributed beamforming algorithms for two different optimization targets. The first objective is to minimize the total transmission power of all the BSs while guaranteeing the user-specific minimum quality-of-service targets. The semidefinite relaxation (SDR) method is used to approximate the non-convex multicast problem as a semidefinite program (SDP), which is solvable via centralized processing. Subsequently, two alternative distributed methods are proposed. The first approach turns the SDP into a two-level optimization via primal decomposition. At the higher level, inter-cell interference powers are optimized for fixed beamformers while the lower level locally optimizes the beamformers by minimizing BS-specific transmit powers for the given inter-cell interference constraints. The second distributed solution is enabled via an alternating direction method of multipliers, where the inter-cell interference optimization is divided into a local and a global optimization by forcing the equality via consistency constraints. We further propose a centralized and a simple distributed beamforming design for the signal-to-interference-plus-noise ratio (SINR) balancing problem in which the minimum SINR among the users is maximized with given per-BS power constraints. This problem is solved via the bisection method as a series of SDP feasibility problems. The simulation results show the superiority of the proposed coordinated beamforming algorithms over traditional non-coordinated transmission schemes, and illustrate the fast convergence of the distributed methods.Comment: Accepted for publication in the IEEE Transactions on Signal Processing, 14 pages, 10 figure

Capacity Optimization through Sensing Threshold Adaptation for Cognitive Radio Networks

In this paper we propose the capacity optimization over sensing threshold for sensing-based cognitive radio networks. The objective function of the proposed optimization is to maximize the capacity at the secondary user subject to the constraints on the transmit power and the sensing threshold in order to protect the primary user. The defined optimization problem is a convex optimization over the transmit power and the sensing threshold where the concavity on sensing threshold is proved. The problem is solved by using Lagrange duality decomposition method in conjunction with a subgradient iterative algorithm and the numerical results show that the proposed optimization can lead to significant capacity maximization for the secondary user as long as the primary user can afford

Adaptive CSMA under the SINR Model: Efficient Approximation Algorithms for Throughput and Utility Maximization

We consider a Carrier Sense Multiple Access (CSMA) based scheduling algorithm for a single-hop wireless network under a realistic Signal-to-interference-plus-noise ratio (SINR) model for the interference. We propose two local optimization based approximation algorithms to efficiently estimate certain attempt rate parameters of CSMA called fugacities. It is known that adaptive CSMA can achieve throughput optimality by sampling feasible schedules from a Gibbs distribution, with appropriate fugacities. Unfortunately, obtaining these optimal fugacities is an NP-hard problem. Further, the existing adaptive CSMA algorithms use a stochastic gradient descent based method, which usually entails an impractically slow (exponential in the size of the network) convergence to the optimal fugacities. To address this issue, we first propose an algorithm to estimate the fugacities, that can support a given set of desired service rates. The convergence rate and the complexity of this algorithm are independent of the network size, and depend only on the neighborhood size of a link. Further, we show that the proposed algorithm corresponds exactly to performing the well-known Bethe approximation to the underlying Gibbs distribution. Then, we propose another local algorithm to estimate the optimal fugacities under a utility maximization framework, and characterize its accuracy. Numerical results indicate that the proposed methods have a good degree of accuracy, and achieve extremely fast convergence to near-optimal fugacities, and often outperform the convergence rate of the stochastic gradient descent by a few orders of magnitude.Comment: Accepted for publication in the IEEE/ACM Transactions on Networkin

Cross-Layer Designs in Coded Wireless Fading Networks with Multicast

A cross-layer design along with an optimal resource allocation framework is formulated for wireless fading networks, where the nodes are allowed to perform network coding. The aim is to jointly optimize end-to-end transport layer rates, network code design variables, broadcast link flows, link capacities, average power consumption, and short-term power allocation policies. As in the routing paradigm where nodes simply forward packets, the cross-layer optimization problem with network coding is non-convex in general. It is proved however, that with network coding, dual decomposition for multicast is optimal so long as the fading at each wireless link is a continuous random variable. This lends itself to provably convergent subgradient algorithms, which not only admit a layered-architecture interpretation but also optimally integrate network coding in the protocol stack. The dual algorithm is also paired with a scheme that yields near-optimal network design variables, namely multicast end-to-end rates, network code design quantities, flows over the broadcast links, link capacities, and average power consumption. Finally, an asynchronous subgradient method is developed, whereby the dual updates at the physical layer can be affordably performed with a certain delay with respect to the resource allocation tasks in upper layers. This attractive feature is motivated by the complexity of the physical layer subproblem, and is an adaptation of the subgradient method suitable for network control.Comment: Accepted in IEEE/ACM Transactions on Networking; revision pendin

Medians and means in Riemannian geometry: existence, uniqueness and computation

This paper is a short summary of our recent work on the medians and means of probability measures in Riemannian manifolds. Firstly, the existence and uniqueness results of local medians are given. In order to compute medians in practical cases, we propose a subgradient algorithm and prove its convergence. After that, Fr\'echet medians are considered. We prove their statistical consistency and give some quantitative estimations of their robustness with the aid of upper curvature bounds. We also show that, in compact Riemannian manifolds, the Fr\'echet medians of generic data points are always unique. Stochastic and deterministic algorithms are proposed for computing Riemannian p-means. The rate of convergence and error estimates of these algorithms are also obtained. Finally, we apply the medians and the Riemannian geometry of Toeplitz covariance matrices to radar target detection

Proximally Guided Stochastic Subgradient Method for Nonsmooth, Nonconvex Problems

In this paper, we introduce a stochastic projected subgradient method for weakly convex (i.e., uniformly prox-regular) nonsmooth, nonconvex functions---a wide class of functions which includes the additive and convex composite classes. At a high-level, the method is an inexact proximal point iteration in which the strongly convex proximal subproblems are quickly solved with a specialized stochastic projected subgradient method. The primary contribution of this paper is a simple proof that the proposed algorithm converges at the same rate as the stochastic gradient method for smooth nonconvex problems. This result appears to be the first convergence rate analysis of a stochastic (or even deterministic) subgradient method for the class of weakly convex functions.Comment: Updated 9/17/2018: Major Revision -added high probability bounds, improved convergence analysis in general, new experimental results. Updated 7/26/2017: Added references to introduction and a couple simple extensions as Sections 3.2 and 4. Updated 8/23/2017: Added NSF acknowledgements. Updated 10/16/2017: Added experimental result

MAGIC: a general, powerful and tractable method for selective inference

Selective inference is a recent research topic that tries to perform valid inference after using the data to select a reasonable statistical model. We propose MAGIC, a new method for selective inference that is general, powerful and tractable. MAGIC is a method for selective inference after solving a convex optimization problem with smooth loss and $\ell_1$ penalty. Randomization is incorporated into the optimization problem to boost statistical power. Through reparametrization, MAGIC reduces the problem into a sampling problem with simple constraints. MAGIC applies to many $\ell_1$ penalized optimization problem including the Lasso, logistic Lasso and neighborhood selection in graphical models, all of which we consider in this paper

Smooth Structured Prediction Using Quantum and Classical Gibbs Samplers

We introduce two quantum algorithms for solving structured prediction problems. We show that a stochastic subgradient descent method that uses the quantum minimum finding algorithm and takes its probabilistic failure into account solves the structured prediction problem with a runtime that scales with the square root of the size of the label space, and in $\widetilde O\left(1/\epsilon\right)$ with respect to the precision, $\epsilon$, of the solution. Motivated by robust inference techniques in machine learning, we introduce another quantum algorithm that solves a smooth approximation of the structured prediction problem with a similar quantum speedup in the size of the label space and a similar scaling in the precision parameter. In doing so, we analyze a stochastic gradient algorithm for convex optimization in the presence of an additive error in the calculation of the gradients, and show that its convergence rate does not deteriorate if the additive errors are of the order $O(\sqrt\epsilon)$. This algorithm uses quantum Gibbs sampling at temperature $\Omega (\epsilon)$ as a subroutine. Based on these theoretical observations, we propose a method for using quantum Gibbs samplers to combine feedforward neural networks with probabilistic graphical models for quantum machine learning. Our numerical results using Monte Carlo simulations on an image tagging task demonstrate the benefit of the approach