Approximation Algorithms for Distributionally Robust Stochastic Optimization

Two-stage stochastic optimization is a widely used framework for modeling uncertainty, where we have a probability distribution over possible realizations of the data, called scenarios, and decisions are taken in two stages: we take first-stage actions knowing only the underlying distribution and before a scenario is realized, and may take additional second-stage recourse actions after a scenario is realized. The goal is typically to minimize the total expected cost. A common criticism levied at this model is that the underlying probability distribution is itself often imprecise. To address this, an approach that is quite versatile and has gained popularity in the stochastic-optimization literature is the two-stage distributionally robust stochastic model: given a collection D of probability distributions, our goal now is to minimize the maximum expected total cost with respect to a distribution in D. There has been almost no prior work however on developing approximation algorithms for distributionally robust problems where the underlying scenario collection is discrete, as is the case with discrete-optimization problems. We provide frameworks for designing approximation algorithms in such settings when the collection D is a ball around a central distribution, defined relative to two notions of distance between probability distributions: Wasserstein metrics (which include the L_1 metric) and the L_infinity metric. Our frameworks yield efficient algorithms even in settings with an exponential number of scenarios, where the central distribution may only be accessed via a sampling oracle. For distributionally robust optimization under a Wasserstein ball, we first show that one can utilize the sample average approximation (SAA) method (solve the distributionally robust problem with an empirical estimate of the central distribution) to reduce the problem to the case where the central distribution has a polynomial-size support, and is represented explicitly. This follows because we argue that a distributionally robust problem can be reduced in a novel way to a standard two-stage stochastic problem with bounded inflation factor, which enables one to use the SAA machinery developed for two-stage stochastic problems. Complementing this, we show how to approximately solve a fractional relaxation of the SAA problem (i.e., the distributionally robust problem obtained by replacing the original central distribution with its empirical estimate). Unlike in two-stage {stochastic, robust} optimization with polynomially many scenarios, this turns out to be quite challenging. We utilize a variant of the ellipsoid method for convex optimization in conjunction with several new ideas to show that the SAA problem can be approximately solved provided that we have an (approximation) algorithm for a certain max-min problem that is akin to, and generalizes, the k-max-min problem (find the worst-case scenario consisting of at most k elements) encountered in two-stage robust optimization. We obtain such an algorithm for various discrete-optimization problems; by complementing this via rounding algorithms that provide local (i.e., per-scenario) approximation guarantees, we obtain the first approximation algorithms for the distributionally robust versions of a variety of discrete-optimization problems including set cover, vertex cover, edge cover, facility location, and Steiner tree, with guarantees that are, except for set cover, within O(1)-factors of the guarantees known for the deterministic version of the problem. For distributionally robust optimization under an L_infinity ball, we consider a fractional relaxation of the problem, and replace its objective function with a proxy function that is pointwise close to the true objective function (within a factor of 2). We then show that we can efficiently compute approximate subgradients of the proxy function, provided that we have an algorithm for the problem of computing the t worst scenarios under a given first-stage decision, given an integer t. We can then approximately minimize the proxy function via a variant of the ellipsoid method, and thus obtain an approximate solution for the fractional relaxation of the distributionally robust problem. Complementing this via rounding algorithms with local guarantees, we obtain approximation algorithms for distributionally robust versions of various covering problems, including set cover, vertex cover, edge cover, and facility location, with guarantees that are within O(1)-factors of the guarantees known for their deterministic versions

Algorithms as Mechanisms: The Price of Anarchy of Relax-and-Round

Many algorithms that are originally designed without explicitly considering incentive properties are later combined with simple pricing rules and used as mechanisms. The resulting mechanisms are often natural and simple to understand. But how good are these algorithms as mechanisms? Truthful reporting of valuations is typically not a dominant strategy (certainly not with a pay-your-bid, first-price rule, but it is likely not a good strategy even with a critical value, or second-price style rule either). Our goal is to show that a wide class of approximation algorithms yields this way mechanisms with low Price of Anarchy. The seminal result of Lucier and Borodin [SODA 2010] shows that combining a greedy algorithm that is an $\alpha$-approximation algorithm with a pay-your-bid payment rule yields a mechanism whose Price of Anarchy is $O(\alpha)$. In this paper we significantly extend the class of algorithms for which such a result is available by showing that this close connection between approximation ratio on the one hand and Price of Anarchy on the other also holds for the design principle of relaxation and rounding provided that the relaxation is smooth and the rounding is oblivious. We demonstrate the far-reaching consequences of our result by showing its implications for sparse packing integer programs, such as multi-unit auctions and generalized matching, for the maximum traveling salesman problem, for combinatorial auctions, and for single source unsplittable flow problems. In all these problems our approach leads to novel simple, near-optimal mechanisms whose Price of Anarchy either matches or beats the performance guarantees of known mechanisms.Comment: Extended abstract appeared in Proc. of 16th ACM Conference on Economics and Computation (EC'15

Discrete Optimization in Early Vision - Model Tractability Versus Fidelity

Early vision is the process occurring before any semantic interpretation of an image takes place. Motion estimation, object segmentation and detection are all parts of early vision, but recognition is not. Some models in early vision are easy to perform inference with---they are tractable. Others describe the reality well---they have high fidelity. This thesis improves the tractability-fidelity trade-off of the current state of the art by introducing new discrete methods for image segmentation and other problems of early vision. The first part studies pseudo-boolean optimization, both from a theoretical perspective as well as a practical one by introducing new algorithms. The main result is the generalization of the roof duality concept to polynomials of higher degree than two. Another focus is parallelization; discrete optimization methods for multi-core processors, computer clusters, and graphical processing units are presented. Remaining in an image segmentation context, the second part studies parametric problems where a set of model parameters and a segmentation are estimated simultaneously. For a small number of parameters these problems can still be optimally solved. One application is an optimal method for solving the two-phase Mumford-Shah functional. The third part shifts the focus to curvature regularization---where the commonly used length and area penalization is replaced by curvature in two and three dimensions. These problems can be discretized over a mesh and special attention is given to the mesh geometry. Specifically, hexagonal meshes in the plane are compared to square ones and a method for generating adaptive meshes is introduced and evaluated. The framework is then extended to curvature regularization of surfaces. Finally, the thesis is concluded by three applications to early vision problems: cardiac MRI segmentation, image registration, and cell classification

Maximum Persistency via Iterative Relaxed Inference with Graphical Models

We consider the NP-hard problem of MAP-inference for undirected discrete graphical models. We propose a polynomial time and practically efficient algorithm for finding a part of its optimal solution. Specifically, our algorithm marks some labels of the considered graphical model either as (i) optimal, meaning that they belong to all optimal solutions of the inference problem; (ii) non-optimal if they provably do not belong to any solution. With access to an exact solver of a linear programming relaxation to the MAP-inference problem, our algorithm marks the maximal possible (in a specified sense) number of labels. We also present a version of the algorithm, which has access to a suboptimal dual solver only and still can ensure the (non-)optimality for the marked labels, although the overall number of the marked labels may decrease. We propose an efficient implementation, which runs in time comparable to a single run of a suboptimal dual solver. Our method is well-scalable and shows state-of-the-art results on computational benchmarks from machine learning and computer vision.Comment: Reworked version, submitted to PAM

Near-Optimal and Robust Mechanism Design for Covering Problems with Correlated Players

We consider the problem of designing incentive-compatible, ex-post individually rational (IR) mechanisms for covering problems in the Bayesian setting, where players' types are drawn from an underlying distribution and may be correlated, and the goal is to minimize the expected total payment made by the mechanism. We formulate a notion of incentive compatibility (IC) that we call {\em support-based IC} that is substantially more robust than Bayesian IC, and develop black-box reductions from support-based-IC mechanism design to algorithm design. For single-dimensional settings, this black-box reduction applies even when we only have an LP-relative {\em approximation algorithm} for the algorithmic problem. Thus, we obtain near-optimal mechanisms for various covering settings including single-dimensional covering problems, multi-item procurement auctions, and multidimensional facility location.Comment: Major changes compared to the previous version. Please consult this versio

Higher-order inference in conditional random fields using submodular functions

Higher-order and dense conditional random fields (CRFs) are expressive graphical models which have been very successful in low-level computer vision applications such as semantic segmentation, and stereo matching. These models are able to capture long-range interactions and higher-order image statistics much better than pairwise CRFs. This expressive power comes at a price though - inference problems in these models are computationally very demanding. This is a particular challenge in computer vision, where fast inference is important and the problem involves millions of pixels. In this thesis, we look at how submodular functions can help us designing efficient inference methods for higher-order and dense CRFs. Submodular functions are special discrete functions that have important properties from an optimisation perspective, and are closely related to convex functions. We use submodularity in a two-fold manner: (a) to design efficient MAP inference algorithm for a robust higher-order model that generalises the widely-used truncated convex models, and (b) to glean insights into a recently proposed variational inference algorithm which give us a principled approach for applying it efficiently to higher-order and dense CRFs

Computational Methods for Discrete Conic Optimization Problems

This thesis addresses computational aspects of discrete conic optimization. Westudy two well-known classes of optimization problems closely related to mixedinteger linear optimization problems. The case of mixed integer second-ordercone optimization problems (MISOCP) is a generalization in which therequirement that solutions be in the non-negative orthant is replaced by arequirement that they be in a second-order cone. Inverse MILP, on the otherhand, is the problem of determining the objective function that makes a givensolution to a given MILP optimal.Although these classes seem unrelated on the surface, the proposedsolution methodology for both classes involves outer approximation of a conicfeasible region by linear inequalities. In both cases, an iterative algorithmin which a separation problem is solved to generate the approximation isemployed. From a complexity standpoint, both MISOCP and inverse MILP areNP--hard. As in the case of MILPs, the usual decision version ofMISOCP is NP-complete, whereas in contrast to MILP, we provide the firstproof that a certain decision version of inverse MILP is rathercoNP-complete.With respect to MISOCP, we first introduce a basic outer approximationalgorithm to solve SOCPs based on a cutting-plane approach. As expected, theperformance of our implementation of such an algorithm is shown to lag behindthe well-known interior point method. Despite this, such a cutting-planeapproach does have promise as a method of producing bounds when embedded withina state-of-the-art branch-and-cut implementation due to its superior ability towarm-start the bound computation after imposing branching constraints. Ourouter-approximation-based branch-and-cut algorithm relaxes both integrality andconic constraints to obtain a linear relaxation. This linear relaxation isstrengthened by the addition of valid inequalities obtained by separatinginfeasible points. Valid inequalities may be obtained by separation from theconvex hull of integer solution lying within the relaxed feasible region or byseparation from the feasible region described by the (relaxed) conicconstraints. Solutions are stored when both integer and conic feasibility isachieved. We review the literature on cutting-plane procedures for MISOCP andmixed integer convex optimization problems.With respect to inverse MILP, we formulate this problem as a conicproblem and derive a cutting-plane algorithm for it. The separation problem inthis algorithm is a modified version of the original MILP. We show that thereis a close relationship between this algorithm and a similar iterativealgorithm for separating infeasible points from the convex hull of solutions tothe original MILP that forms part of the basis for the well-known result ofGrotschel-Lovasz-Schrijver that demonstrates the complexity-wiseequivalence of separation and optimization.In order to test our ideas, we implement a number of software librariesthat together constitute DisCO, a full-featured solver for MISOCP. Thefirst of the supporting libraries is OsiConic, an abstract base classin C++ for interfacing to SOCP solvers. We provide interfaces using thislibrary for widely used commercial and open source SOCP/nonlinear problemsolvers. We also introduce CglConic, a library that implements cuttingprocedures for MISOCP feasible set. We perform extensive computationalexperiments with DisCO comparing a wide range of variants of our proposedalgorithm, as well as other approaches. As DisCO is built on top of a libraryfor distributed parallel tree search algorithms, we also perform experimentsshowing that our algorithm is effective and scalable when parallelized