Statistical Limits of Convex Relaxations

Many high dimensional sparse learning problems are formulated as nonconvex optimization. A popular approach to solve these nonconvex optimization problems is through convex relaxations such as linear and semidefinite programming. In this paper, we study the statistical limits of convex relaxations. Particularly, we consider two problems: Mean estimation for sparse principal submatrix and edge probability estimation for stochastic block model. We exploit the sum-of-squares relaxation hierarchy to sharply characterize the limits of a broad class of convex relaxations. Our result shows statistical optimality needs to be compromised for achieving computational tractability using convex relaxations. Compared with existing results on computational lower bounds for statistical problems, which consider general polynomial-time algorithms and rely on computational hardness hypotheses on problems like planted clique detection, our theory focuses on a broad class of convex relaxations and does not rely on unproven hypotheses.Comment: 22 page

Conic Programming Approaches for Polynomial Optimization: Theory and Applications

Historically, polynomials are among the most popular class of functions used for empirical modeling in science and engineering. Polynomials are easy to evaluate, appear naturally in many physical (real-world) systems, and can be used to accurately approximate any smooth function. It is not surprising then, that the task of solving polynomial optimization problems; that is, problems where both the objective function and constraints are multivariate polynomials, is ubiquitous and of enormous interest in these fields. Clearly, polynomial op- timization problems encompass a very general class of non-convex optimization problems, including key combinatorial optimization problems.The focus of the first three chapters of this document is to address the solution of polynomial optimization problems in theory and in practice, using a conic optimization approach. Convex optimization has been well studied to solve quadratic constrained quadratic problems. In the first part, convex relaxations for general polynomial optimization problems are discussed. Instead of using the matrix space to study quadratic programs, we study the convex relaxations for POPs through a lifted tensor space, more specifically, using the completely positive tensor cone and the completely positive semidefinite tensor cone. We show that tensor relaxations theoretically yield no-worse global bounds for a class of polynomial optimization problems than relaxation for a QCQP reformulation of the POPs. We also propose an approximation strategy for tensor cones and show empirically the advantage of the tensor relaxation.In the second part, we propose an alternative SDP and SOCP hierarchy to obtain global bounds for general polynomial optimization problems. Comparing with other existing SDP and SOCP hierarchies that uses higher degree sum of square (SOS) polynomials and scaled diagonally sum of square polynomials (SDSOS) when the hierarchy level increases, these proposed hierarchies, using fixed degree SOS and SDSOS polynomials but more of these polynomials, perform numerically better. Numerical results show that the hierarchies we proposed have better performance in terms of tightness of the bound and solution time compared with other hierarchies in the literature.The third chapter deals with Alternating Current Optimal Power Flow problem via a polynomial optimization approach. The Alternating Current Optimal Power Flow (ACOPF) problem is a challenging non-convex optimization problem in power systems. Prior research mainly focuses on using SDP relaxations and SDP-based hierarchies to address the solution of ACOPF problem. In this Chapter, we apply existing SOCP hierarchies to this problem and explore the structure of the network to propose simplified hierarchies for ACOPF problems. Compared with SDP approaches, SOCP approaches are easier to solve and can be used to approximate large scale ACOPF problems.The last chapter also relates to the use of conic optimization techniques, but in this case to pricing in markets with non-convexities. Indeed, it is an application of conic optimization approach to solve a pricing problem in energy systems. Prior research in energy market pricing mainly focus on linear costs in the objective function. Due to the penetration of renewable energies into the current electricity grid, it is important to consider quadratic costs in the objective function, which reflects the ramping costs for traditional generators. This study address the issue how to find the market clearing prices when considering quadratic costs in the objective function

A Bounded Degree Lasserre Hierarchy with SOCP Relaxations for Global Polynomial Optimization and Applications

In this paper, we propose a new convergent conic programming hierarchy of relaxations involving both semi-definite cone and second-order cone constraints for solving nonconvex polynomial optimization problems to global optimality. The significance of this hierarchy is that the size and number of the semi-definite and second-order cone constraints of the relaxations are fixed and independent of the step or level of the approximation in the hierarchy. Using the Krivine-Stengle's certificate of positivity in real algebraic geometry, we establish the convergence of the hierarchy of relaxations, extending the very recent so-called bounded degree Lasserre hierarchy. In particular, we also provide a convergent bounded degree second-order cone programming (SOCP) hierarchy for solving polynomial optimization problems. We then present finite convergence at step one of the SOCP hierarchy for two classes of polynomial optimization problems: a subclass of convex polynomial optimization problems where the objective and constraint functions are SOCP-convex polynomials, defined in terms of specially structured sum of squares polynomials, and a class of polynomial optimization problems, involving polynomials with essentially non-positive coefficients. In the case of one-step convergence for problems with SOCP-convex polynomials, we show how a global solution is recovered from the relaxation via Jensen's inequality of SOCP-convex polynomials. As an application, we derive a corresponding convergent conic linear programming hierarchy for conic-convex semi-algebraic programs. Whenever the semi-algebraic set of the conic-convex program is described by convex polynomial inequalities, we show further that the values of the relaxation problems converge to the common value of the convex program and its Lagrangian dual under a constraint qualification.Comment: This is an updated and improved version of the previous paper entitled with "A New Hierarchy of Second-order Cone Programming Relaxations for Polynomial Optimization: Asymptotic and Finite Convergence

Alternative SDP and SOCP Approximations for Polynomial Optimization

In theory, hierarchies of semidefinite programming (SDP) relaxations based on sum of squares (SOS) polynomials have been shown to provide arbitrarily close approximations for a general polynomial optimization problem (POP). However, due to the computational challenge of solving SDPs, it becomes difficult to use SDP hierarchies for large-scale problems. To address this, hierarchies of second-order cone programming (SOCP) relaxations resulting from a restriction of the SOS polynomial condition have been recently proposed to approximate POPs. Here, we consider alternative ways to use this SOCP restrictions of the SOS condition. In particular, we show that SOCP hierarchies can be effectively used to strengthen hierarchies of linear programming (LP) relaxations for POPs. Specifically, we show that this solution approach is substantially more effective in finding solutions of certain POPs for which the more common hierarchies of SDP relaxations are known to perform poorly. Furthermore, when the feasible set of the POP is compact, these SOCP hierarchies converge to the POP's optimal value

Semidefinite Programming for Min-Max Problems and Games

We introduce two min-max problems: the first problem is to minimize the supremum of finitely many rational functions over a compact basic semi-algebraic set whereas the second problem is a 2-player zero-sum polynomial game in randomized strategies and with compact basic semi-algebraic pure strategy sets. It is proved that their optimal solution can be approximated by solving a hierarchy of semidefinite relaxations, in the spirit of the moment approach developed in Lasserre. This provides a unified approach and a class of algorithms to approximate all Nash equilibria and min-max strategies of many static and dynamic games. Each semidefinite relaxation can be solved in time which is polynomial in its input size and practice from global optimization suggests that very often few relaxations are needed for a good approximation (and sometimes even finite convergence).Comment: 21 page

An improved semidefinite programming hierarchy for testing entanglement

We present a stronger version of the Doherty-Parrilo-Spedalieri (DPS) hierarchy of approximations for the set of separable states. Unlike DPS, our hierarchy converges exactly at a finite number of rounds for any fixed input dimension. This yields an algorithm for separability testing which is singly exponential in dimension and polylogarithmic in accuracy. Our analysis makes use of tools from algebraic geometry, but our algorithm is elementary and differs from DPS only by one simple additional collection of constraints.Comment: 22 pages. v2: published version, adds numerical results. Matlab code available at https://github.com/isobovine/dpsplus

On the Complexity of Testing Attainment of the Optimal Value in Nonlinear Optimization

We prove that unless P=NP, there exists no polynomial time (or even pseudo-polynomial time) algorithm that can test whether the optimal value of a nonlinear optimization problem where the objective and constraints are given by low-degree polynomials is attained. If the degrees of these polynomials are fixed, our results along with previously-known "Frank-Wolfe type" theorems imply that exactly one of two cases can occur: either the optimal value is attained on every instance, or it is strongly NP-hard to distinguish attainment from non-attainment. We also show that testing for some well-known sufficient conditions for attainment of the optimal value, such as coercivity of the objective function and closedness and boundedness of the feasible set, is strongly NP-hard. As a byproduct, our proofs imply that testing the Archimedean property of a quadratic module is strongly NP-hard, a property that is of independent interest to the convergence of the Lasserre hierarchy. Finally, we give semidefinite programming (SDP)-based sufficient conditions for attainment of the optimal value, in particular a new characterization of coercive polynomials that lends itself to an SDP hierarchy.Comment: 18 page

Guaranteed Globally Optimal Planar Pose Graph and Landmark SLAM via Sparse-Bounded Sums-of-Squares Programming

Autonomous navigation requires an accurate model or map of the environment. While dramatic progress in the prior two decades has enabled large-scale simultaneous localization and mapping (SLAM), the majority of existing methods rely on non-linear optimization techniques to find the maximum likelihood estimate (MLE) of the robot trajectory and surrounding environment. These methods are prone to local minima and are thus sensitive to initialization. Several recent papers have developed optimization algorithms for the Pose-Graph SLAM problem that can certify the optimality of a computed solution. Though this does not guarantee a priori that this approach generates an optimal solution, a recent extension has shown that when the noise lies within a critical threshold that the solution to the optimization algorithm is guaranteed to be optimal. To address the limitations of existing approaches, this paper illustrates that the Pose-Graph SLAM and Landmark SLAM can be formulated as polynomial optimization programs that are sum-of-squares (SOS) convex. This paper then describes how the Pose-Graph and Landmark SLAM problems can be solved to a global minimum without initialization regardless of noise level using the sparse bounded degree sum-of-squares (Sparse-BSOS) optimization method. Finally, the superior performance of the proposed approach when compared to existing SLAM methods is illustrated on graphs with several hundred nodes.Comment: 7 pages, 5 figure

A bounded degree SOS hierarchy for polynomial optimization

We consider a new hierarchy of semidefinite relaxations for the general polynomial optimization problem $(P):\:f^{\ast}=\min \{\,f(x):x\in K\,\}$ on a compact basic semi-algebraic set $K\subset\R^n$. This hierarchy combines some advantages of the standard LP-relaxations associated with Krivine's positivity certificate and some advantages of the standard SOS-hierarchy. In particular it has the following attractive features: (a) In contrast to the standard SOS-hierarchy, for each relaxation in the hierarchy, the size of the matrix associated with the semidefinite constraint is the same and fixed in advance by the user. (b) In contrast to the LP-hierarchy, finite convergence occurs at the first step of the hierarchy for an important class of convex problems. Finally (c) some important techniques related to the use of point evaluations for declaring a polynomial to be zero and to the use of rank-one matrices make an efficient implementation possible. Preliminary results on a sample of non convex problems are encouraging

Approximation Algorithms for PSPACE-Hard Hierarchically and Periodically Specified Problems

We study the efficient approximability of basic graph and logic problems in the literature when instances are specified hierarchically as in \cite{Le89} or are specified by 1-dimensional finite narrow periodic specifications as in \cite{Wa93}. We show that, for most of the problems $\Pi$ considered when specified using {\bf k-level-restricted} hierarchical specifications or $k$-narrow periodic specifications the following holds: \item Let $\rho$ be any performance guarantee of a polynomial time approximation algorithm for $\Pi$, when instances are specified using standard specifications. Then $\forall \epsilon > 0$, $\Pi$ has a polynomial time approximation algorithm with performance guarantee $(1 + \epsilon) \rho$. \item $\Pi$ has a polynomial time approximation scheme when restricted to planar instances. \end{romannum} These are the first polynomial time approximation schemes for PSPACE-hard hierarchically or periodically specified problems. Since several of the problems considered are PSPACE-hard, our results provide the first examples of natural PSPACE-hard optimization problems that have polynomial time approximation schemes. This answers an open question in Condon et. al. \cite{CF+93}.Comment: 5 Figures, 24 page