A Decomposition Algorithm for Nested Resource Allocation Problems

We propose an exact polynomial algorithm for a resource allocation problem with convex costs and constraints on partial sums of resource consumptions, in the presence of either continuous or integer variables. No assumption of strict convexity or differentiability is needed. The method solves a hierarchy of resource allocation subproblems, whose solutions are used to convert constraints on sums of resources into bounds for separate variables at higher levels. The resulting time complexity for the integer problem is $O(n \log m \log (B/n))$, and the complexity of obtaining an $\epsilon$-approximate solution for the continuous case is $O(n \log m \log (B/\epsilon))$, $n$ being the number of variables, $m$ the number of ascending constraints (such that $m < n$), $\epsilon$ a desired precision, and $B$ the total resource. This algorithm attains the best-known complexity when $m = n$, and improves it when $\log m = o(\log n)$. Extensive experimental analyses are conducted with four recent algorithms on various continuous problems issued from theory and practice. The proposed method achieves a higher performance than previous algorithms, addressing all problems with up to one million variables in less than one minute on a modern computer.Comment: Working Paper -- MIT, 23 page

Reflection methods for user-friendly submodular optimization

Recently, it has become evident that submodularity naturally captures widely occurring concepts in machine learning, signal processing and computer vision. Consequently, there is need for efficient optimization procedures for submodular functions, especially for minimization problems. While general submodular minimization is challenging, we propose a new method that exploits existing decomposability of submodular functions. In contrast to previous approaches, our method is neither approximate, nor impractical, nor does it need any cumbersome parameter tuning. Moreover, it is easy to implement and parallelize. A key component of our method is a formulation of the discrete submodular minimization problem as a continuous best approximation problem that is solved through a sequence of reflections, and its solution can be easily thresholded to obtain an optimal discrete solution. This method solves both the continuous and discrete formulations of the problem, and therefore has applications in learning, inference, and reconstruction. In our experiments, we illustrate the benefits of our method on two image segmentation tasks.Comment: Neural Information Processing Systems (NIPS), \'Etats-Unis (2013

Approximation schemes for scheduling on a single machine subject to cumulative deterioration and maintenance

We consider a scheduling problem on a single machine to minimize the makespan. The processing conditions are subject to cumulative deterioration, but can be restored by a single maintenance. We link the problem to the Subset-sum problem (if the duration of maintenance is constant) and to the Half-Product Problem (if the duration of maintenance depends on its start time). For both versions of the problem, we adapt the existing fully polynomial-time approximation schemes to our problems by handling the additive constants

Network Flow Algorithms for Structured Sparsity

We consider a class of learning problems that involve a structured sparsity-inducing norm defined as the sum of $\ell_\infty$-norms over groups of variables. Whereas a lot of effort has been put in developing fast optimization methods when the groups are disjoint or embedded in a specific hierarchical structure, we address here the case of general overlapping groups. To this end, we show that the corresponding optimization problem is related to network flow optimization. More precisely, the proximal problem associated with the norm we consider is dual to a quadratic min-cost flow problem. We propose an efficient procedure which computes its solution exactly in polynomial time. Our algorithm scales up to millions of variables, and opens up a whole new range of applications for structured sparse models. We present several experiments on image and video data, demonstrating the applicability and scalability of our approach for various problems.Comment: accepted for publication in Adv. Neural Information Processing Systems, 201

A Duality Theory with Zero Duality Gap for Nonlinear Programming

Duality is an important notion for constrained optimization which provides a theoretical foundation for a number of constraint decomposition schemes such as separable programming and for deriving lower bounds in space decomposition algorithms such as branch and bound. However, the conventional duality theory has the fundamental limit that it leads to duality gaps for nonconvex optimization problems, especially discrete and mixed-integer problems where the feasible sets are nonconvex. In this paper, we propose a novel extended duality theory for nonlinear optimization that overcomes some limitations of previous dual methods. Based on a new dual function, the extended duality theory leads to zero duality gap for general nonconvex problems defined in discrete, continuous, and mixed-integer spaces under mild conditions

Convex Analysis and Optimization with Submodular Functions: a Tutorial

Set-functions appear in many areas of computer science and applied mathematics, such as machine learning, computer vision, operations research or electrical networks. Among these set-functions, submodular functions play an important role, similar to convex functions on vector spaces. In this tutorial, the theory of submodular functions is presented, in a self-contained way, with all results shown from first principles. A good knowledge of convex analysis is assumed

About strongly polynomial time algorithms for quadratic optimization over submodular constraints

We present new strongly polynomial algorithms for special cases of convex separable quadratic minimization over submodular constraints. The main results are: an O(NM log(N2/M)) algorithm for the problem Network defined on a network on M arcs and N nodes; an O(n log n) algorithm for the tree problem on n variables; an O(n log n) algorithm for the Nested problem, and a linear time algorithm for the Generalized Upper Bound problem. These algorithms are the best known so rar for these problems. The status of the general problem and open questions are presented as weil.This research has been supported in part by ONR grant N00014-91-J-1241

An Algorithmic Theory of Dependent Regularizers, Part 1: Submodular Structure

We present an exploration of the rich theoretical connections between several classes of regularized models, network flows, and recent results in submodular function theory. This work unifies key aspects of these problems under a common theory, leading to novel methods for working with several important models of interest in statistics, machine learning and computer vision. In Part 1, we review the concepts of network flows and submodular function optimization theory foundational to our results. We then examine the connections between network flows and the minimum-norm algorithm from submodular optimization, extending and improving several current results. This leads to a concise representation of the structure of a large class of pairwise regularized models important in machine learning, statistics and computer vision. In Part 2, we describe the full regularization path of a class of penalized regression problems with dependent variables that includes the graph-guided LASSO and total variation constrained models. This description also motivates a practical algorithm. This allows us to efficiently find the regularization path of the discretized version of TV penalized models. Ultimately, our new algorithms scale up to high-dimensional problems with millions of variables

A Class of Convex Quadratic Nonseparable Resource Allocation Problems with Generalized Bound Constraints

We study a convex quadratic nonseparable resource allocation problem that arises in the area of decentralized energy management (DEM), where unbalance in electricity networks has to be minimized. In this problem, the given resource is allocated over a set of activities that is divided into subsets, and a cost is assigned to the overall allocated amount of resources to activities within the same subset. We derive two efficient algorithms with $O(n \log  n)$ worst-case time complexity to solve this problem. For the special case where all subsets have the same size, one of these algorithms even runs in linear time given the subset size. Both algorithms are inspired by well-studied breakpoint search methods for separable convex resource allocation problems. Numerical evaluations on both real and synthetic data confirm the theoretical efficiency of both algorithms and demonstrate their suitability for integration in DEM systems