114 research outputs found
Optimal Sparse Decision Trees
Decision tree algorithms have been among the most popular algorithms for
interpretable (transparent) machine learning since the early 1980's. The
problem that has plagued decision tree algorithms since their inception is
their lack of optimality, or lack of guarantees of closeness to optimality:
decision tree algorithms are often greedy or myopic, and sometimes produce
unquestionably suboptimal models. Hardness of decision tree optimization is
both a theoretical and practical obstacle, and even careful mathematical
programming approaches have not been able to solve these problems efficiently.
This work introduces the first practical algorithm for optimal decision trees
for binary variables. The algorithm is a co-design of analytical bounds that
reduce the search space and modern systems techniques, including data
structures and a custom bit-vector library. Our experiments highlight
advantages in scalability, speed, and proof of optimality.Comment: 33rd Conference on Neural Information Processing Systems (NeurIPS
2019), Vancouver, Canad
Probabilistic Dataset Reconstruction from Interpretable Models
Interpretability is often pointed out as a key requirement for trustworthy
machine learning. However, learning and releasing models that are inherently
interpretable leaks information regarding the underlying training data. As such
disclosure may directly conflict with privacy, a precise quantification of the
privacy impact of such breach is a fundamental problem. For instance, previous
work have shown that the structure of a decision tree can be leveraged to build
a probabilistic reconstruction of its training dataset, with the uncertainty of
the reconstruction being a relevant metric for the information leak. In this
paper, we propose of a novel framework generalizing these probabilistic
reconstructions in the sense that it can handle other forms of interpretable
models and more generic types of knowledge. In addition, we demonstrate that
under realistic assumptions regarding the interpretable models' structure, the
uncertainty of the reconstruction can be computed efficiently. Finally, we
illustrate the applicability of our approach on both decision trees and rule
lists, by comparing the theoretical information leak associated to either exact
or heuristic learning algorithms. Our results suggest that optimal
interpretable models are often more compact and leak less information regarding
their training data than greedily-built ones, for a given accuracy level
Interpretable multiclass classification by MDL-based rule lists
Interpretable classifiers have recently witnessed an increase in attention
from the data mining community because they are inherently easier to understand
and explain than their more complex counterparts. Examples of interpretable
classification models include decision trees, rule sets, and rule lists.
Learning such models often involves optimizing hyperparameters, which typically
requires substantial amounts of data and may result in relatively large models.
In this paper, we consider the problem of learning compact yet accurate
probabilistic rule lists for multiclass classification. Specifically, we
propose a novel formalization based on probabilistic rule lists and the minimum
description length (MDL) principle. This results in virtually parameter-free
model selection that naturally allows to trade-off model complexity with
goodness of fit, by which overfitting and the need for hyperparameter tuning
are effectively avoided. Finally, we introduce the Classy algorithm, which
greedily finds rule lists according to the proposed criterion. We empirically
demonstrate that Classy selects small probabilistic rule lists that outperform
state-of-the-art classifiers when it comes to the combination of predictive
performance and interpretability. We show that Classy is insensitive to its
only parameter, i.e., the candidate set, and that compression on the training
set correlates with classification performance, validating our MDL-based
selection criterion
Landmarks in Case-Based Reasoning:From Theory to Data
Widespread application of uninterpretable machine learning systems for sensitive purposes has spurred research into elucidating the decision making process of these systems. These efforts have their background in many different disciplines, one of which is the field of AI & law. In particular, recent works have observed that machine learning training data can be interpreted as legal cases. Under this interpretation the formalism developed to study case law, called the theory of precedential constraint, can be used to analyze the way in which machine learning systems draw on training data - or should draw on them - to make decisions. These works predominantly stay on the theoretical level, hence in the present work the formalism is evaluated on a real world dataset. Through this analysis we identify a significant new concept which we call landmark cases, and use it to characterize the types of datasets that are more or less suitable to be described by the theory.</p
- …