868 research outputs found
SFO: A Toolbox for Submodular Function Optimization
In recent years, a fundamental problem structure has emerged as very useful in a variety of machine learning applications: Submodularity is an intuitive diminishing returns property, stating that adding an element to a smaller set helps more than adding it to a larger set. Similarly to convexity, submodularity allows one to efficiently find provably (near-) optimal solutions for large problems. We present SFO, a toolbox for use in MATLAB or Octave that implements algorithms for minimization and maximization of submodular functions. A tutorial script illustrates the application of submodularity to machine learning and AI problems such as feature selection, clustering, inference and optimized information gathering
Structured sparsity-inducing norms through submodular functions
Sparse methods for supervised learning aim at finding good linear predictors
from as few variables as possible, i.e., with small cardinality of their
supports. This combinatorial selection problem is often turned into a convex
optimization problem by replacing the cardinality function by its convex
envelope (tightest convex lower bound), in this case the L1-norm. In this
paper, we investigate more general set-functions than the cardinality, that may
incorporate prior knowledge or structural constraints which are common in many
applications: namely, we show that for nondecreasing submodular set-functions,
the corresponding convex envelope can be obtained from its \lova extension, a
common tool in submodular analysis. This defines a family of polyhedral norms,
for which we provide generic algorithmic tools (subgradients and proximal
operators) and theoretical results (conditions for support recovery or
high-dimensional inference). By selecting specific submodular functions, we can
give a new interpretation to known norms, such as those based on
rank-statistics or grouped norms with potentially overlapping groups; we also
define new norms, in particular ones that can be used as non-factorial priors
for supervised learning
Efficient Algorithms for Searching the Minimum Information Partition in Integrated Information Theory
The ability to integrate information in the brain is considered to be an
essential property for cognition and consciousness. Integrated Information
Theory (IIT) hypothesizes that the amount of integrated information () in
the brain is related to the level of consciousness. IIT proposes that to
quantify information integration in a system as a whole, integrated information
should be measured across the partition of the system at which information loss
caused by partitioning is minimized, called the Minimum Information Partition
(MIP). The computational cost for exhaustively searching for the MIP grows
exponentially with system size, making it difficult to apply IIT to real neural
data. It has been previously shown that if a measure of satisfies a
mathematical property, submodularity, the MIP can be found in a polynomial
order by an optimization algorithm. However, although the first version of
is submodular, the later versions are not. In this study, we empirically
explore to what extent the algorithm can be applied to the non-submodular
measures of by evaluating the accuracy of the algorithm in simulated
data and real neural data. We find that the algorithm identifies the MIP in a
nearly perfect manner even for the non-submodular measures. Our results show
that the algorithm allows us to measure in large systems within a
practical amount of time
Duality between Feature Selection and Data Clustering
The feature-selection problem is formulated from an information-theoretic
perspective. We show that the problem can be efficiently solved by an extension
of the recently proposed info-clustering paradigm. This reveals the fundamental
duality between feature selection and data clustering,which is a consequence of
the more general duality between the principal partition and the principal
lattice of partitions in combinatorial optimization
Sampling and Reconstruction of Graph Signals via Weak Submodularity and Semidefinite Relaxation
We study the problem of sampling a bandlimited graph signal in the presence
of noise, where the objective is to select a node subset of prescribed
cardinality that minimizes the signal reconstruction mean squared error (MSE).
To that end, we formulate the task at hand as the minimization of MSE subject
to binary constraints, and approximate the resulting NP-hard problem via
semidefinite programming (SDP) relaxation. Moreover, we provide an alternative
formulation based on maximizing a monotone weak submodular function and propose
a randomized-greedy algorithm to find a sub-optimal subset. We then derive a
worst-case performance guarantee on the MSE returned by the randomized greedy
algorithm for general non-stationary graph signals. The efficacy of the
proposed methods is illustrated through numerical simulations on synthetic and
real-world graphs. Notably, the randomized greedy algorithm yields an
order-of-magnitude speedup over state-of-the-art greedy sampling schemes, while
incurring only a marginal MSE performance loss
- …