A Scalable and Extensible Framework for Superposition-Structured Models

In many learning tasks, structural models usually lead to better interpretability and higher generalization performance. In recent years, however, the simple structural models such as lasso are frequently proved to be insufficient. Accordingly, there has been a lot of work on "superposition-structured" models where multiple structural constraints are imposed. To efficiently solve these "superposition-structured" statistical models, we develop a framework based on a proximal Newton-type method. Employing the smoothed conic dual approach with the LBFGS updating formula, we propose a scalable and extensible proximal quasi-Newton (SEP-QN) framework. Empirical analysis on various datasets shows that our framework is potentially powerful, and achieves super-linear convergence rate for optimizing some popular "superposition-structured" statistical models such as the fused sparse group lasso

Subsampling Algorithms for Semidefinite Programming

We derive a stochastic gradient algorithm for semidefinite optimization using randomization techniques. The algorithm uses subsampling to reduce the computational cost of each iteration and the subsampling ratio explicitly controls granularity, i.e. the tradeoff between cost per iteration and total number of iterations. Furthermore, the total computational cost is directly proportional to the complexity (i.e. rank) of the solution. We study numerical performance on some large-scale problems arising in statistical learning.Comment: Final version, to appear in Stochastic System

Submodular Optimization with Submodular Cover and Submodular Knapsack Constraints

We investigate two new optimization problems -- minimizing a submodular function subject to a submodular lower bound constraint (submodular cover) and maximizing a submodular function subject to a submodular upper bound constraint (submodular knapsack). We are motivated by a number of real-world applications in machine learning including sensor placement and data subset selection, which require maximizing a certain submodular function (like coverage or diversity) while simultaneously minimizing another (like cooperative cost). These problems are often posed as minimizing the difference between submodular functions [14, 35] which is in the worst case inapproximable. We show, however, that by phrasing these problems as constrained optimization, which is more natural for many applications, we achieve a number of bounded approximation guarantees. We also show that both these problems are closely related and an approximation algorithm solving one can be used to obtain an approximation guarantee for the other. We provide hardness results for both problems thus showing that our approximation factors are tight up to log-factors. Finally, we empirically demonstrate the performance and good scalability properties of our algorithms.Comment: 23 pages. A short version of this appeared in Advances of NIPS-201

L2L_2-Box Optimization for Green Cloud-RAN via Network Adaptation

In this paper, we propose a reformulation for the Mixed Integer Programming (MIP) problem into an exact and continuous model through using the $\ell_2$-box technique to recast the binary constraints into a box with an $\ell_2$ sphere constraint. The reformulated problem can be tackled by a dual ascent algorithm combined with a Majorization-Minimization (MM) method for the subproblems to solve the network power consumption problem of the Cloud Radio Access Network (Cloud-RAN), and which leads to solving a sequence of Difference of Convex (DC) subproblems handled by an inexact MM algorithm. After obtaining the final solution, we use it as the initial result of the bi-section Group Sparse Beamforming (GSBF) algorithm to promote the group-sparsity of beamformers, rather than using the weighted $\ell_1 / \ell_2$-norm. Simulation results indicate that the new method outperforms the bi-section GSBF algorithm by achieving smaller network power consumption, especially in sparser cases, i.e., Cloud-RANs with a lot of Remote Radio Heads (RRHs) but fewer users.Comment: 4 pages, 4 figure

A Simple Iterative Algorithm for Parsimonious Binary Kernel Fisher Discrimination

By applying recent results in optimization theory variously known as optimization transfer or majorize/minimize algorithms, an algorithm for binary, kernel, Fisher discriminant analysis is introduced that makes use of a non-smooth penalty on the coefficients to provide a parsimonious solution. The problem is converted into a smooth optimization that can be solved iteratively with no greater overhead than iteratively re-weighted least-squares. The result is simple, easily programmed and is shown to perform, in terms of both accuracy and parsimony, as well as or better than a number of leading machine learning algorithms on two well-studied and substantial benchmarks