827 research outputs found
Efficient Minimization of Decomposable Submodular Functions
Many combinatorial problems arising in machine learning can be reduced to the
problem of minimizing a submodular function. Submodular functions are a natural
discrete analog of convex functions, and can be minimized in strongly
polynomial time. Unfortunately, state-of-the-art algorithms for general
submodular minimization are intractable for larger problems. In this paper, we
introduce a novel subclass of submodular minimization problems that we call
decomposable. Decomposable submodular functions are those that can be
represented as sums of concave functions applied to modular functions. We
develop an algorithm, SLG, that can efficiently minimize decomposable
submodular functions with tens of thousands of variables. Our algorithm
exploits recent results in smoothed convex minimization. We apply SLG to
synthetic benchmarks and a joint classification-and-segmentation task, and show
that it outperforms the state-of-the-art general purpose submodular
minimization algorithms by several orders of magnitude.Comment: Expanded version of paper for Neural Information Processing Systems
201
Structured Sparsity: Discrete and Convex approaches
Compressive sensing (CS) exploits sparsity to recover sparse or compressible
signals from dimensionality reducing, non-adaptive sensing mechanisms. Sparsity
is also used to enhance interpretability in machine learning and statistics
applications: While the ambient dimension is vast in modern data analysis
problems, the relevant information therein typically resides in a much lower
dimensional space. However, many solutions proposed nowadays do not leverage
the true underlying structure. Recent results in CS extend the simple sparsity
idea to more sophisticated {\em structured} sparsity models, which describe the
interdependency between the nonzero components of a signal, allowing to
increase the interpretability of the results and lead to better recovery
performance. In order to better understand the impact of structured sparsity,
in this chapter we analyze the connections between the discrete models and
their convex relaxations, highlighting their relative advantages. We start with
the general group sparse model and then elaborate on two important special
cases: the dispersive and the hierarchical models. For each, we present the
models in their discrete nature, discuss how to solve the ensuing discrete
problems and then describe convex relaxations. We also consider more general
structures as defined by set functions and present their convex proxies.
Further, we discuss efficient optimization solutions for structured sparsity
problems and illustrate structured sparsity in action via three applications.Comment: 30 pages, 18 figure
Submodular Optimization with Submodular Cover and Submodular Knapsack Constraints
We investigate two new optimization problems -- minimizing a submodular
function subject to a submodular lower bound constraint (submodular cover) and
maximizing a submodular function subject to a submodular upper bound constraint
(submodular knapsack). We are motivated by a number of real-world applications
in machine learning including sensor placement and data subset selection, which
require maximizing a certain submodular function (like coverage or diversity)
while simultaneously minimizing another (like cooperative cost). These problems
are often posed as minimizing the difference between submodular functions [14,
35] which is in the worst case inapproximable. We show, however, that by
phrasing these problems as constrained optimization, which is more natural for
many applications, we achieve a number of bounded approximation guarantees. We
also show that both these problems are closely related and an approximation
algorithm solving one can be used to obtain an approximation guarantee for the
other. We provide hardness results for both problems thus showing that our
approximation factors are tight up to log-factors. Finally, we empirically
demonstrate the performance and good scalability properties of our algorithms.Comment: 23 pages. A short version of this appeared in Advances of NIPS-201
Curvature and Optimal Algorithms for Learning and Minimizing Submodular Functions
We investigate three related and important problems connected to machine
learning: approximating a submodular function everywhere, learning a submodular
function (in a PAC-like setting [53]), and constrained minimization of
submodular functions. We show that the complexity of all three problems depends
on the 'curvature' of the submodular function, and provide lower and upper
bounds that refine and improve previous results [3, 16, 18, 52]. Our proof
techniques are fairly generic. We either use a black-box transformation of the
function (for approximation and learning), or a transformation of algorithms to
use an appropriate surrogate function (for minimization). Curiously, curvature
has been known to influence approximations for submodular maximization [7, 55],
but its effect on minimization, approximation and learning has hitherto been
open. We complete this picture, and also support our theoretical claims by
empirical results.Comment: 21 pages. A shorter version appeared in Advances of NIPS-201
Near-Optimal Sensor Scheduling for Batch State Estimation: Complexity, Algorithms, and Limits
In this paper, we focus on batch state estimation for linear systems. This
problem is important in applications such as environmental field estimation,
robotic navigation, and target tracking. Its difficulty lies on that limited
operational resources among the sensors, e.g., shared communication bandwidth
or battery power, constrain the number of sensors that can be active at each
measurement step. As a result, sensor scheduling algorithms must be employed.
Notwithstanding, current sensor scheduling algorithms for batch state
estimation scale poorly with the system size and the time horizon. In addition,
current sensor scheduling algorithms for Kalman filtering, although they scale
better, provide no performance guarantees or approximation bounds for the
minimization of the batch state estimation error. In this paper, one of our
main contributions is to provide an algorithm that enjoys both the estimation
accuracy of the batch state scheduling algorithms and the low time complexity
of the Kalman filtering scheduling algorithms. In particular: 1) our algorithm
is near-optimal: it achieves a solution up to a multiplicative factor 1/2 from
the optimal solution, and this factor is close to the best approximation factor
1/e one can achieve in polynomial time for this problem; 2) our algorithm has
(polynomial) time complexity that is not only lower than that of the current
algorithms for batch state estimation; it is also lower than, or similar to,
that of the current algorithms for Kalman filtering. We achieve these results
by proving two properties for our batch state estimation error metric, which
quantifies the square error of the minimum variance linear estimator of the
batch state vector: a) it is supermodular in the choice of the sensors; b) it
has a sparsity pattern (it involves matrices that are block tri-diagonal) that
facilitates its evaluation at each sensor set.Comment: Correction of typos in proof
- …