Regularization vs. Relaxation: A conic optimization perspective of statistical variable selection

Variable selection is a fundamental task in statistical data analysis. Sparsity-inducing regularization methods are a popular class of methods that simultaneously perform variable selection and model estimation. The central problem is a quadratic optimization problem with an l0-norm penalty. Exactly enforcing the l0-norm penalty is computationally intractable for larger scale problems, so dif- ferent sparsity-inducing penalty functions that approximate the l0-norm have been introduced. In this paper, we show that viewing the problem from a convex relaxation perspective offers new insights. In particular, we show that a popular sparsity-inducing concave penalty function known as the Minimax Concave Penalty (MCP), and the reverse Huber penalty derived in a recent work by Pilanci, Wainwright and Ghaoui, can both be derived as special cases of a lifted convex relaxation called the perspective relaxation. The optimal perspective relaxation is a related minimax problem that balances the overall convexity and tightness of approximation to the l0 norm. We show it can be solved by a semidefinite relaxation. Moreover, a probabilistic interpretation of the semidefinite relaxation reveals connections with the boolean quadric polytope in combinatorial optimization. Finally by reformulating the l0-norm pe- nalized problem as a two-level problem, with the inner level being a Max-Cut problem, our proposed semidefinite relaxation can be realized by replacing the inner level problem with its semidefinite relaxation studied by Goemans and Williamson. This interpretation suggests using the Goemans-Williamson rounding procedure to find approximate solutions to the l0-norm penalized problem. Numerical experiments demonstrate the tightness of our proposed semidefinite relaxation, and the effectiveness of finding approximate solutions by Goemans-Williamson rounding.Comment: Also available on optimization online {http://www.optimization-online.org/DB_HTML/2015/05/4932.html

ADMM for the SDP relaxation of the QAP

The semidefinite programming (SDP) relaxation has proven to be extremely strong for many hard discrete optimization problems. This is in particular true for the quadratic assignment problem (QAP), arguably one of the hardest NP-hard discrete optimization problems. There are several difficulties that arise in efficiently solving the SDP relaxation, e.g.,~increased dimension; inefficiency of the current primal-dual interior point solvers in terms of both time and accuracy; and difficulty and high expense in adding cutting plane constraints. We propose using the alternating direction method of multipliers (ADMM) to solve the SDP relaxation. This first order approach allows for inexpensive iterations, a method of cheaply obtaining low rank solutions, as well a trivial way of adding cutting plane inequalities. When compared to current approaches and current best available bounds we obtain remarkable robustness, efficiency and improved bounds.Comment: 12 pages, 1 tabl

Semidefinite relaxation based branch-and-bound method for nonconvex quadratic programming

Thesis (S.M.)--Massachusetts Institute of Technology, Computation for Design and Optimization Program, 2006.Includes bibliographical references (leaves 73-75).In this thesis, we use a semidefinite relaxation based branch-and-bound method to solve nonconvex quadratic programming problems. Firstly, we show an interval branch-and-bound method to calculate the bounds for the minimum of bounded polynomials. Then we demonstrate four SDP relaxation methods to solve nonconvex Box constrained Quadratic Programming (BoxQP) problems and the comparison of the four methods. For some lower dimensional problems, SDP relaxation methods can achieve tight bounds for the BoxQP problem; whereas for higher dimensional cases (more than 20 dimensions), the bounds achieved by the four Semidefinite programming (SDP) relaxation methods are always loose. To achieve tight bounds for higher dimensional BoxQP problems, we combine the branch-and-bound method and SDP relaxation method to develop an SDP relaxation based branch-and-bound (SDPBB) method. We introduce a sensitivity analysis method for the branching process of SDPBB. This sensitivity analysis method can improve the convergence speed significantly.(cont.) Compared to the interval branch-and-bound method and the global optimization software BARON, SDPBB can achieve better bounds and is also much more efficient. Additionally, we have developed a multisection algorithm for SDPBB and the multisection algorithm has been parallelized using Message Passing Interface (MPI). By parallelizing the program, we can significantly improve the speed of solving higher dimensional BoxQP problems.by Sha Hu.S.M

Concave Quadratic Cuts for Mixed-Integer Quadratic Problems

The technique of semidefinite programming (SDP) relaxation can be used to obtain a nontrivial bound on the optimal value of a nonconvex quadratically constrained quadratic program (QCQP). We explore concave quadratic inequalities that hold for any vector in the integer lattice ${\bf Z}^n$, and show that adding these inequalities to a mixed-integer nonconvex QCQP can improve the SDP-based bound on the optimal value. This scheme is tested using several numerical problem instances of the max-cut problem and the integer least squares problem.Comment: 24 pages, 1 figur

Penalized Semidefinite Programming for Quadratically-Constrained Quadratic Optimization

In this paper, we give a new penalized semidefinite programming approach for non-convex quadratically-constrained quadratic programs (QCQPs). We incorporate penalty terms into the objective of convex relaxations in order to retrieve feasible and near-optimal solutions for non-convex QCQPs. We introduce a generalized linear independence constraint qualification (GLICQ) criterion and prove that any GLICQ regular point that is sufficiently close to the feasible set can be used to construct an appropriate penalty term and recover a feasible solution. Inspired by these results, we develop a heuristic sequential procedure that preserves feasibility and aims to improve the objective value at each iteration. Numerical experiments on large-scale system identification problems as well as benchmark instances from the library of quadratic programming (QPLIB) demonstrate the ability of the proposed penalized semidefinite programs in finding near-optimal solutions for non-convex QCQP

Using a conic bundle method to accelerate both phases of a quadratic convex reformulation

We present algorithm MIQCR-CB that is an advancement of method MIQCR~(Billionnet, Elloumi and Lambert, 2012). MIQCR is a method for solving mixed-integer quadratic programs and works in two phases: the first phase determines an equivalent quadratic formulation with a convex objective function by solving a semidefinite problem $(SDP)$, and, in the second phase, the equivalent formulation is solved by a standard solver. As the reformulation relies on the solution of a large-scale semidefinite program, it is not tractable by existing semidefinite solvers, already for medium sized problems. To surmount this difficulty, we present in MIQCR-CB a subgradient algorithm within a Lagrangian duality framework for solving $(SDP)$ that substantially speeds up the first phase. Moreover, this algorithm leads to a reformulated problem of smaller size than the one obtained by the original MIQCR method which results in a shorter time for solving the second phase. We present extensive computational results to show the efficiency of our algorithm

Conic relaxation approaches for equal deployment problems

An important problem in the breeding of livestock, crops, and forest trees is the optimum of selection of genotypes that maximizes genetic gain. The key constraint in the optimal selection is a convex quadratic constraint that ensures genetic diversity, therefore, the optimal selection can be cast as a second-order cone programming (SOCP) problem. Yamashita et al. (2015) exploits the structural sparsity of the quadratic constraints and reduces the computation time drastically while attaining the same optimal solution. This paper is concerned with the special case of equal deployment (ED), in which we solve the optimal selection problem with the constraint that contribution of genotypes must either be a fixed size or zero. This involves a nature of combinatorial optimization, and the ED problem can be described as a mixed-integer SOCP problem. In this paper, we discuss conic relaxation approaches for the ED problem based on LP (linear programming), SOCP, and SDP (semidefinite programming). We analyze theoretical bounds derivedfrom the SDP relaxation approaches using the work of Tseng (2003) and show that the theoretical bounds are not quite sharp for tree breeding problems. We propose a steepest-ascent method that combines the solution obtained from the conic relaxation problems with a concept from discrete convex optimization in order to acquire an approximate solution for the ED problem in a practical time. From numerical tests, we observed that among the LP, SOCP, and SDP relaxation problems, SOCP gave a suitable solution from the viewpoints of the optimal values and the computation time. The steepest-ascent method starting from the SOCP solution provides high-quality solutions much faster than an existing method that has been widely used for the optimal selection problems and a branch-and-bound method

A Scalable Semidefinite Relaxation Approach to Grid Scheduling

Determination of the most economic strategies for supply and transmission of electricity is a daunting computational challenge. Due to theoretical barriers, so-called NP-hardness, the amount of effort to optimize the schedule of generating units and route of power, can grow exponentially with the number of decision variables. Practical approaches to this problem involve legacy approximations and ad-hoc heuristics that may undermine the efficiency and reliability of power system operations, that are ever growing in scale and complexity. Therefore, the development of powerful optimization methods for detailed power system scheduling is critical to the realization of smart grids and has received significant attention recently. In this paper, we propose for the first time a computational method, which is capable of solving large-scale power system scheduling problems with thousands of generating units, while accurately incorporating the nonlinear equations that govern the flow of electricity on the grid. The utilization of this accurate nonlinear model, as opposed to its linear approximations, results in a more efficient and transparent market design, as well as improvements in the reliability of power system operations. We design a polynomial-time solvable third-order semidefinite programming (TSDP) relaxation, with the aim of finding a near globally optimal solution for the original NP-hard problem. The proposed method is demonstrated on the largest available benchmark instances from real-world European grid data, for which provably optimal or near-optimal solutions are obtained

A Semidefinite Relaxation for Air Traffic Flow Scheduling

We first formulate the problem of optimally scheduling air traffic low with sector capacity constraints as a mixed integer linear program. We then use semidefinite relaxation techniques to form a convex relaxation of that problem. Finally, we present a randomization algorithm to further improve the quality of the solution. Because of the specific structure of the air traffic flow problem, the relaxation has a single semidefinite constraint of size dn where d is the maximum delay and n the number of flights.Comment: Submitted to RIVF 200

Solution to the Inverse Wulff Problem by Means of the Enhanced Semidefinite Relaxation Method

We propose a novel method of resolving the optimal anisotropy function. The idea is to construct the optimal anisotropy function as a solution to the inverse Wulff problem, i.e. as a minimizer for the anisoperimetric ratio for a given Jordan curve in the plane. It leads to a nonconvex quadratic optimization problem with linear matrix inequalities. In order to solve it we propose the so-called enhanced semidefinite relaxation method which is based on a solution to a convex semidefinite problem obtained by a semidefinite relaxation of the original problem augmented by quadratic-linear constraints. We show that the sequence of finite dimensional approximations of the optimal anisoperimetric ratio converges to the optimal anisoperimetric ratio which is a solution to the inverse Wulff problem. Several computational examples, including those corresponding to boundaries of real snowflakes and discussion on the rate of convergence of numerical method are also presented in this paper.Comment: 5 figure