Learning with Clustering Structure

We study supervised learning problems using clustering constraints to impose structure on either features or samples, seeking to help both prediction and interpretation. The problem of clustering features arises naturally in text classification for instance, to reduce dimensionality by grouping words together and identify synonyms. The sample clustering problem on the other hand, applies to multiclass problems where we are allowed to make multiple predictions and the performance of the best answer is recorded. We derive a unified optimization formulation highlighting the common structure of these problems and produce algorithms whose core iteration complexity amounts to a k-means clustering step, which can be approximated efficiently. We extend these results to combine sparsity and clustering constraints, and develop a new projection algorithm on the set of clustered sparse vectors. We prove convergence of our algorithms on random instances, based on a union of subspaces interpretation of the clustering structure. Finally, we test the robustness of our methods on artificial data sets as well as real data extracted from movie reviews.Comment: Completely rewritten. New convergence proofs in the clustered and sparse clustered case. New projection algorithm on sparse clustered vector

Polyhedral Approximations of Quadratic Semi-Assignment Problems, Disjunctive Programs, and Base-2 Expansions of Integer Variables

This research is concerned with developing improved representations for special families of mixed-discrete programming problems. Such problems can typically be modeled using different mathematical forms, and the representation employed can greatly influence the problem\u27s ability to be solved. Generally speaking, it is desired to obtain mixed 0-1 linear forms whose continuous relaxations provide tight polyhedral outer-approximations to the convex hulls of feasible solutions. This dissertation makes contributions to three distinct problems, providing new forms that improve upon published works. The first emphasis is on devising solution procedures for the classical quadratic semi-assignment problem(QSAP), which is an NP-hard 0-1 quadratic program. The effort begins by using a reformulation-linearization technique to recast the problem as a mixed 0-1 linear program. The resulting form provides insight into identifying special instances that are readily solvable. For the general case, the form is shown to have a tight continuous relaxation, as well as to possess a decomposable structure. Specifically, a Hamiltonian decomposition of a graph interpretation is devised to motivate a Lagrangian dual whose subproblems consist of families of separable acyclic minimum-cost network flows. The result is an efficient approach for computing tight lower bounds on the optimal objective value to the original discrete program. Extensive computational experience is reported to evaluate the tightness of the representation and the expedience of the algorithm. The second contribution uses disjunctive programming arguments to model the convex hull of the union of a finite collection of polytopes. It is well known that the convex hull of the union of n polytopes can be obtained by lifting the problem into a higher-dimensional space using n auxiliary continuous (scaling) variables. When placed within a larger optimization problem, these variables must be restricted to be binary. This work examines an approach that uses fewer binary variables. The same scaling technique is employed, but the variables are treated as continuous by introducing a logarithmic number of new binary variables and constraints. The scaling variables can now be substituted from the problem. Moreover, an emphasis of this work, is that specially structured polytopes lead to well-defined projection operations that yield more concise forms. These special polytopes consist of knapsack problems having SOS-1 and SOS-2 type restrictions. Different projections are defined for the SOS-2 case, leading to forms that serve to both explain and unify alternative representations for piecewise-linear functions, as well as to promote favorable computational experience. The third contribution uses minimal cover and set covering inequalities to define the previously unknown convex hulls of special sets of binary vectors that are lexicographically lower and upper bounded by given vectors. These convex hulls are used to obtain ideal representations for base-2 expansions of bounded integer variables, and also afford a new perspective on, and extend convex hull results for, binary knapsack polytopes having weakly super-decreasing coefficients. Computational experience for base-2 expansions of integer variables exhibits a reduction in effort

Quantum walk speedup of backtracking algorithms

We describe a general method to obtain quantum speedups of classical algorithms which are based on the technique of backtracking, a standard approach for solving constraint satisfaction problems (CSPs). Backtracking algorithms explore a tree whose vertices are partial solutions to a CSP in an attempt to find a complete solution. Assume there is a classical backtracking algorithm which finds a solution to a CSP on n variables, or outputs that none exists, and whose corresponding tree contains T vertices, each vertex corresponding to a test of a partial solution. Then we show that there is a bounded-error quantum algorithm which completes the same task using O(sqrt(T) n^(3/2) log n) tests. In particular, this quantum algorithm can be used to speed up the DPLL algorithm, which is the basis of many of the most efficient SAT solvers used in practice. The quantum algorithm is based on the use of a quantum walk algorithm of Belovs to search in the backtracking tree. We also discuss how, for certain distributions on the inputs, the algorithm can lead to an exponential reduction in expected runtime.Comment: 23 pages; v2: minor changes to presentatio

A Global Method for a Two-Dimensional Cutting Stock Problem in the Manufacturing Industry

A two-dimensional cutting stock problem (2DCSP) needs to cut a set of given rectangular items from standard-sized rectangular materials with the objective of minimizing the number of materials used. This problem frequently arises in different manufacturing industries such as glass, wood, paper, plastic, etc. However, the current literatures lack a deterministic method for solving the 2DCSP. However, this study proposes a global method to solve the 2DCSP. It aims to reduce the number of binary variables for the proposed model to speed up the solving time and obtain the optimal solution. Our experiments demonstrate that the proposed method is superior to current reference methods for solving the 2DCSP

Stabbing Planes

We introduce and develop a new semi-algebraic proof system, called Stabbing Planes that is in the style of DPLL-based modern SAT solvers. As with DPLL, there is only one rule: the current polytope can be subdivided by branching on an inequality and its "integer negation." That is, we can (nondeterministically choose) a hyperplane a x >= b with integer coefficients, which partitions the polytope into three pieces: the points in the polytope satisfying a x >= b, the points satisfying a x <= b-1, and the middle slab b-1 < a x < b. Since the middle slab contains no integer points it can be safely discarded, and the algorithm proceeds recursively on the other two branches. Each path terminates when the current polytope is empty, which is polynomial-time checkable. Among our results, we show somewhat surprisingly that Stabbing Planes can efficiently simulate Cutting Planes, and moreover, is strictly stronger than Cutting Planes under a reasonable conjecture. We prove linear lower bounds on the rank of Stabbing Planes refutations, by adapting a lifting argument in communication complexity

Solving Set Constraint Satisfaction Problems using ROBDDs

In this paper we present a new approach to modeling finite set domain constraint problems using Reduced Ordered Binary Decision Diagrams (ROBDDs). We show that it is possible to construct an efficient set domain propagator which compactly represents many set domains and set constraints using ROBDDs. We demonstrate that the ROBDD-based approach provides unprecedented flexibility in modeling constraint satisfaction problems, leading to performance improvements. We also show that the ROBDD-based modeling approach can be extended to the modeling of integer and multiset constraint problems in a straightforward manner. Since domain propagation is not always practical, we also show how to incorporate less strict consistency notions into the ROBDD framework, such as set bounds, cardinality bounds and lexicographic bounds consistency. Finally, we present experimental results that demonstrate the ROBDD-based solver performs better than various more conventional constraint solvers on several standard set constraint problems