3,625 research outputs found
On the Bayes-optimality of F-measure maximizers
The F-measure, which has originally been introduced in information retrieval,
is nowadays routinely used as a performance metric for problems such as binary
classification, multi-label classification, and structured output prediction.
Optimizing this measure is a statistically and computationally challenging
problem, since no closed-form solution exists. Adopting a decision-theoretic
perspective, this article provides a formal and experimental analysis of
different approaches for maximizing the F-measure. We start with a Bayes-risk
analysis of related loss functions, such as Hamming loss and subset zero-one
loss, showing that optimizing such losses as a surrogate of the F-measure leads
to a high worst-case regret. Subsequently, we perform a similar type of
analysis for F-measure maximizing algorithms, showing that such algorithms are
approximate, while relying on additional assumptions regarding the statistical
distribution of the binary response variables. Furthermore, we present a new
algorithm which is not only computationally efficient but also Bayes-optimal,
regardless of the underlying distribution. To this end, the algorithm requires
only a quadratic (with respect to the number of binary responses) number of
parameters of the joint distribution. We illustrate the practical performance
of all analyzed methods by means of experiments with multi-label classification
problems
Distributed Online Big Data Classification Using Context Information
Distributed, online data mining systems have emerged as a result of
applications requiring analysis of large amounts of correlated and
high-dimensional data produced by multiple distributed data sources. We propose
a distributed online data classification framework where data is gathered by
distributed data sources and processed by a heterogeneous set of distributed
learners which learn online, at run-time, how to classify the different data
streams either by using their locally available classification functions or by
helping each other by classifying each other's data. Importantly, since the
data is gathered at different locations, sending the data to another learner to
process incurs additional costs such as delays, and hence this will be only
beneficial if the benefits obtained from a better classification will exceed
the costs. We model the problem of joint classification by the distributed and
heterogeneous learners from multiple data sources as a distributed contextual
bandit problem where each data is characterized by a specific context. We
develop a distributed online learning algorithm for which we can prove
sublinear regret. Compared to prior work in distributed online data mining, our
work is the first to provide analytic regret results characterizing the
performance of the proposed algorithm
A sparse multinomial probit model for classification
A recent development in penalized probit modelling using a hierarchical Bayesian approach has led to a sparse binomial (two-class) probit classifier that can be trained via an EM algorithm. A key advantage of the formulation is that no tuning of hyperparameters relating to the penalty is needed thus simplifying the model selection process. The resulting model demonstrates excellent classification performance and a high degree of sparsity when used as a kernel machine. It is, however, restricted to the binary classification problem and can only be used in the multinomial situation via a one-against-all or one-against-many strategy. To overcome this, we apply the idea to the multinomial probit model. This leads to a direct multi-classification approach and is shown to give a sparse solution with accuracy and sparsity comparable with the current state-of-the-art. Comparative numerical benchmark examples are used to demonstrate the method
The Measure and Mismeasure of Fairness: A Critical Review of Fair Machine Learning
The nascent field of fair machine learning aims to ensure that decisions
guided by algorithms are equitable. Over the last several years, three formal
definitions of fairness have gained prominence: (1) anti-classification,
meaning that protected attributes---like race, gender, and their proxies---are
not explicitly used to make decisions; (2) classification parity, meaning that
common measures of predictive performance (e.g., false positive and false
negative rates) are equal across groups defined by the protected attributes;
and (3) calibration, meaning that conditional on risk estimates, outcomes are
independent of protected attributes. Here we show that all three of these
fairness definitions suffer from significant statistical limitations. Requiring
anti-classification or classification parity can, perversely, harm the very
groups they were designed to protect; and calibration, though generally
desirable, provides little guarantee that decisions are equitable. In contrast
to these formal fairness criteria, we argue that it is often preferable to
treat similarly risky people similarly, based on the most statistically
accurate estimates of risk that one can produce. Such a strategy, while not
universally applicable, often aligns well with policy objectives; notably, this
strategy will typically violate both anti-classification and classification
parity. In practice, it requires significant effort to construct suitable risk
estimates. One must carefully define and measure the targets of prediction to
avoid retrenching biases in the data. But, importantly, one cannot generally
address these difficulties by requiring that algorithms satisfy popular
mathematical formalizations of fairness. By highlighting these challenges in
the foundation of fair machine learning, we hope to help researchers and
practitioners productively advance the area
Ethical Adversaries: Towards Mitigating Unfairness with Adversarial Machine Learning
Machine learning is being integrated into a growing number of critical
systems with far-reaching impacts on society. Unexpected behaviour and unfair
decision processes are coming under increasing scrutiny due to this widespread
use and its theoretical considerations. Individuals, as well as organisations,
notice, test, and criticize unfair results to hold model designers and
deployers accountable. We offer a framework that assists these groups in
mitigating unfair representations stemming from the training datasets. Our
framework relies on two inter-operating adversaries to improve fairness. First,
a model is trained with the goal of preventing the guessing of protected
attributes' values while limiting utility losses. This first step optimizes the
model's parameters for fairness. Second, the framework leverages evasion
attacks from adversarial machine learning to generate new examples that will be
misclassified. These new examples are then used to retrain and improve the
model in the first step. These two steps are iteratively applied until a
significant improvement in fairness is obtained. We evaluated our framework on
well-studied datasets in the fairness literature -- including COMPAS -- where
it can surpass other approaches concerning demographic parity, equality of
opportunity and also the model's utility. We also illustrate our findings on
the subtle difficulties when mitigating unfairness and highlight how our
framework can assist model designers.Comment: 15 pages, 3 figures, 1 tabl
The Utility of Text: The Case of Amicus Briefs and the Supreme Court
We explore the idea that authoring a piece of text is an act of maximizing
one's expected utility. To make this idea concrete, we consider the societally
important decisions of the Supreme Court of the United States. Extensive past
work in quantitative political science provides a framework for empirically
modeling the decisions of justices and how they relate to text. We incorporate
into such a model texts authored by amici curiae ("friends of the court"
separate from the litigants) who seek to weigh in on the decision, then
explicitly model their goals in a random utility model. We demonstrate the
benefits of this approach in improved vote prediction and the ability to
perform counterfactual analysis.Comment: Working draf
- …