13,625 research outputs found
Optimal Sparse Decision Trees
Decision tree algorithms have been among the most popular algorithms for
interpretable (transparent) machine learning since the early 1980's. The
problem that has plagued decision tree algorithms since their inception is
their lack of optimality, or lack of guarantees of closeness to optimality:
decision tree algorithms are often greedy or myopic, and sometimes produce
unquestionably suboptimal models. Hardness of decision tree optimization is
both a theoretical and practical obstacle, and even careful mathematical
programming approaches have not been able to solve these problems efficiently.
This work introduces the first practical algorithm for optimal decision trees
for binary variables. The algorithm is a co-design of analytical bounds that
reduce the search space and modern systems techniques, including data
structures and a custom bit-vector library. Our experiments highlight
advantages in scalability, speed, and proof of optimality.Comment: 33rd Conference on Neural Information Processing Systems (NeurIPS
2019), Vancouver, Canad
Incremental Sampling-based Algorithms for Optimal Motion Planning
During the last decade, incremental sampling-based motion planning
algorithms, such as the Rapidly-exploring Random Trees (RRTs) have been shown
to work well in practice and to possess theoretical guarantees such as
probabilistic completeness. However, no theoretical bounds on the quality of
the solution obtained by these algorithms have been established so far. The
first contribution of this paper is a negative result: it is proven that, under
mild technical conditions, the cost of the best path in the RRT converges
almost surely to a non-optimal value. Second, a new algorithm is considered,
called the Rapidly-exploring Random Graph (RRG), and it is shown that the cost
of the best path in the RRG converges to the optimum almost surely. Third, a
tree version of RRG is introduced, called the RRT algorithm, which
preserves the asymptotic optimality of RRG while maintaining a tree structure
like RRT. The analysis of the new algorithms hinges on novel connections
between sampling-based motion planning algorithms and the theory of random
geometric graphs. In terms of computational complexity, it is shown that the
number of simple operations required by both the RRG and RRT algorithms is
asymptotically within a constant factor of that required by RRT.Comment: 20 pages, 10 figures, this manuscript is submitted to the
International Journal of Robotics Research, a short version is to appear at
the 2010 Robotics: Science and Systems Conference
Melding the Data-Decisions Pipeline: Decision-Focused Learning for Combinatorial Optimization
Creating impact in real-world settings requires artificial intelligence
techniques to span the full pipeline from data, to predictive models, to
decisions. These components are typically approached separately: a machine
learning model is first trained via a measure of predictive accuracy, and then
its predictions are used as input into an optimization algorithm which produces
a decision. However, the loss function used to train the model may easily be
misaligned with the end goal, which is to make the best decisions possible.
Hand-tuning the loss function to align with optimization is a difficult and
error-prone process (which is often skipped entirely).
We focus on combinatorial optimization problems and introduce a general
framework for decision-focused learning, where the machine learning model is
directly trained in conjunction with the optimization algorithm to produce
high-quality decisions. Technically, our contribution is a means of integrating
common classes of discrete optimization problems into deep learning or other
predictive models, which are typically trained via gradient descent. The main
idea is to use a continuous relaxation of the discrete problem to propagate
gradients through the optimization procedure. We instantiate this framework for
two broad classes of combinatorial problems: linear programs and submodular
maximization. Experimental results across a variety of domains show that
decision-focused learning often leads to improved optimization performance
compared to traditional methods. We find that standard measures of accuracy are
not a reliable proxy for a predictive model's utility in optimization, and our
method's ability to specify the true goal as the model's training objective
yields substantial dividends across a range of decision problems.Comment: Full version of paper accepted at AAAI 201
A Fast Quartet Tree Heuristic for Hierarchical Clustering
The Minimum Quartet Tree Cost problem is to construct an optimal weight tree
from the weighted quartet topologies on objects, where
optimality means that the summed weight of the embedded quartet topologies is
optimal (so it can be the case that the optimal tree embeds all quartets as
nonoptimal topologies). We present a Monte Carlo heuristic, based on randomized
hill climbing, for approximating the optimal weight tree, given the quartet
topology weights. The method repeatedly transforms a dendrogram, with all
objects involved as leaves, achieving a monotonic approximation to the exact
single globally optimal tree. The problem and the solution heuristic has been
extensively used for general hierarchical clustering of nontree-like
(non-phylogeny) data in various domains and across domains with heterogeneous
data. We also present a greatly improved heuristic, reducing the running time
by a factor of order a thousand to ten thousand. All this is implemented and
available, as part of the CompLearn package. We compare performance and running
time of the original and improved versions with those of UPGMA, BioNJ, and NJ,
as implemented in the SplitsTree package on genomic data for which the latter
are optimized.
Keywords: Data and knowledge visualization, Pattern
matching--Clustering--Algorithms/Similarity measures, Hierarchical clustering,
Global optimization, Quartet tree, Randomized hill-climbing,Comment: LaTeX, 40 pages, 11 figures; this paper has substantial overlap with
arXiv:cs/0606048 in cs.D
A Static Optimality Transformation with Applications to Planar Point Location
Over the last decade, there have been several data structures that, given a
planar subdivision and a probability distribution over the plane, provide a way
for answering point location queries that is fine-tuned for the distribution.
All these methods suffer from the requirement that the query distribution must
be known in advance.
We present a new data structure for point location queries in planar
triangulations. Our structure is asymptotically as fast as the optimal
structures, but it requires no prior information about the queries. This is a
2D analogue of the jump from Knuth's optimum binary search trees (discovered in
1971) to the splay trees of Sleator and Tarjan in 1985. While the former need
to know the query distribution, the latter are statically optimal. This means
that we can adapt to the query sequence and achieve the same asymptotic
performance as an optimum static structure, without needing any additional
information.Comment: 13 pages, 1 figure, a preliminary version appeared at SoCG 201
- …