46 research outputs found
F-measure Maximization in Multi-Label Classification with Conditionally Independent Label Subsets
We discuss a method to improve the exact F-measure maximization algorithm
called GFM, proposed in (Dembczynski et al. 2011) for multi-label
classification, assuming the label set can be can partitioned into
conditionally independent subsets given the input features. If the labels were
all independent, the estimation of only parameters ( denoting the number
of labels) would suffice to derive Bayes-optimal predictions in
operations. In the general case, parameters are required by GFM, to
solve the problem in operations. In this work, we show that the number
of parameters can be reduced further to , in the best case, assuming the
label set can be partitioned into conditionally independent subsets. As
this label partition needs to be estimated from the data beforehand, we use
first the procedure proposed in (Gasse et al. 2015) that finds such partition
and then infer the required parameters locally in each label subset. The latter
are aggregated and serve as input to GFM to form the Bayes-optimal prediction.
We show on a synthetic experiment that the reduction in the number of
parameters brings about significant benefits in terms of performance
A theory of unstructured bargaining using distribution-valued solution concepts
In experiments it is typically found that many joint utility outcomes arise in any given unstructured bargaining game. This suggests using a positive unstructured bargaining concept that maps a bargaining game to a probability distribution over outcomes rather than to a single outcome. We show how to "translate" Nash's bargaining axioms to apply to such distributional bargaining concepts. We then prove that a subset of those axioms forces the distribution over outcomes to be a power-law. Unlike Nash's original result, our result holds even if the feasible set is finite. When the feasible set is convex and comprehensive, the mode of the power law distribution is the Harsanyi bargaining solution, and if we require symmetry it is the Nash bargaining solution. However in general these modes of the joint utility distribution are not Bayes-optimal predictions for the joint uitlity, nor are the bargains corresponding to those outcomes the most likely bargains. We then show how an external regulator can use distributional solution concepts to optimally design an unstructured bargaining scenario. Throughout we demonstrate our analysis in computational experiments involving flight rerouting negotiations in the National Airspace System.JEL Codes:
A Sandbox Tool to Bias(Stress)-Test Fairness Algorithms
Motivated by the growing importance of reducing unfairness in ML predictions,
Fair-ML researchers have presented an extensive suite of algorithmic
"fairness-enhancing" remedies. Most existing algorithms, however, are agnostic
to the sources of the observed unfairness. As a result, the literature
currently lacks guiding frameworks to specify conditions under which each
algorithmic intervention can potentially alleviate the underpinning cause of
unfairness. To close this gap, we scrutinize the underlying biases (e.g., in
the training data or design choices) that cause observational unfairness. We
present a bias-injection sandbox tool to investigate fairness consequences of
various biases and assess the effectiveness of algorithmic remedies in the
presence of specific types of bias. We call this process the
bias(stress)-testing of algorithmic interventions. Unlike existing toolkits,
ours provides a controlled environment to counterfactually inject biases in the
ML pipeline. This stylized setup offers the distinct capability of testing
fairness interventions beyond observational data and against an unbiased
benchmark. In particular, we can test whether a given remedy can alleviate the
injected bias by comparing the predictions resulting after the intervention in
the biased setting with true labels in the unbiased regime -- that is, before
any bias injection. We illustrate the utility of our toolkit via a
proof-of-concept case study on synthetic data. Our empirical analysis showcases
the type of insights that can be obtained through our simulations
Hedging predictions in machine learning
Recent advances in machine learning make it possible to design efficient
prediction algorithms for data sets with huge numbers of parameters. This paper
describes a new technique for "hedging" the predictions output by many such
algorithms, including support vector machines, kernel ridge regression, kernel
nearest neighbours, and by many other state-of-the-art methods. The hedged
predictions for the labels of new objects include quantitative measures of
their own accuracy and reliability. These measures are provably valid under the
assumption of randomness, traditional in machine learning: the objects and
their labels are assumed to be generated independently from the same
probability distribution. In particular, it becomes possible to control (up to
statistical fluctuations) the number of erroneous predictions by selecting a
suitable confidence level. Validity being achieved automatically, the remaining
goal of hedged prediction is efficiency: taking full account of the new
objects' features and other available information to produce as accurate
predictions as possible. This can be done successfully using the powerful
machinery of modern machine learning.Comment: 24 pages; 9 figures; 2 tables; a version of this paper (with
discussion and rejoinder) is to appear in "The Computer Journal
On Aggregation in Ensembles of Multilabel Classifiers
While a variety of ensemble methods for multilabel classification have been
proposed in the literature, the question of how to aggregate the predictions of
the individual members of the ensemble has received little attention so far. In
this paper, we introduce a formal framework of ensemble multilabel
classification, in which we distinguish two principal approaches: "predict then
combine" (PTC), where the ensemble members first make loss minimizing
predictions which are subsequently combined, and "combine then predict" (CTP),
which first aggregates information such as marginal label probabilities from
the individual ensemble members, and then derives a prediction from this
aggregation. While both approaches generalize voting techniques commonly used
for multilabel ensembles, they allow to explicitly take the target performance
measure into account. Therefore, concrete instantiations of CTP and PTC can be
tailored to concrete loss functions. Experimentally, we show that standard
voting techniques are indeed outperformed by suitable instantiations of CTP and
PTC, and provide some evidence that CTP performs well for decomposable loss
functions, whereas PTC is the better choice for non-decomposable losses.Comment: 14 pages, 2 figure