3 research outputs found
Calibrated Surrogate Maximization of Linear-fractional Utility in Binary Classification
Complex classification performance metrics such as the F-measure
and Jaccard index are often used, in order to handle class-imbalanced cases
such as information retrieval and image segmentation. These performance metrics
are not decomposable, that is, they cannot be expressed in a per-example
manner, which hinders a straightforward application of M-estimation widely used
in supervised learning. In this paper, we consider linear-fractional metrics,
which are a family of classification performance metrics that encompasses many
standard ones such as the F-measure and Jaccard index, and propose
methods to directly maximize performances under those metrics. A clue to tackle
their direct optimization is a calibrated surrogate utility, which is a
tractable lower bound of the true utility function representing a given metric.
We characterize sufficient conditions which make the surrogate maximization
coincide with the maximization of the true utility. Simulation results on
benchmark datasets validate the effectiveness of our calibrated surrogate
maximization especially if the sample sizes are extremely small.Comment: AISTATS2020 camera ready submissio
Calibrated Surrogate Losses for Adversarially Robust Classification
Adversarially robust classification seeks a classifier that is insensitive to
adversarial perturbations of test patterns. This problem is often formulated
via a minimax objective, where the target loss is the worst-case value of the
0-1 loss subject to a bound on the size of perturbation. Recent work has
proposed convex surrogates for the adversarial 0-1 loss, in an effort to make
optimization more tractable. In this work, we consider the question of which
surrogate losses are calibrated with respect to the adversarial 0-1 loss,
meaning that minimization of the former implies minimization of the latter. We
show that no convex surrogate loss is calibrated with respect to the
adversarial 0-1 loss when restricted to the class of linear models. We further
introduce a class of nonconvex losses and offer necessary and sufficient
conditions for losses in this class to be calibrated.Comment: COLT2020 (to appear); 43 pages, 21 figure
Classification with Rejection Based on Cost-sensitive Classification
The goal of classification with rejection is to avoid risky misclassification
in error-critical applications such as medical diagnosis and product
inspection. In this paper, based on the relationship between classification
with rejection and cost-sensitive classification, we propose a novel method of
classification with rejection by learning an ensemble of cost-sensitive
classifiers, which satisfies all the following properties: (i) it can avoid
estimating class-posterior probabilities, resulting in improved classification
accuracy, (ii) it allows a flexible choice of losses including non-convex ones,
(iii) it does not require complicated modifications when using different
losses, (iv) it is applicable to both binary and multiclass cases, and (v) it
is theoretically justifiable for any classification-calibrated loss.
Experimental results demonstrate the usefulness of our proposed approach in
clean-labeled, noisy-labeled, and positive-unlabeled classification.Comment: 40 pages. Added the discussion of the recent work by Gangrade et al.
(2021) at the end of Section 3.4, where the idea of constructing
cost-sensitive classifiers for classification with rejection has also been
explored in a different framework of classification with rejection (where the
goal is not minimizing the 0-1-c risk as in our paper