6 research outputs found
Who Should Predict? Exact Algorithms For Learning to Defer to Humans
Automated AI classifiers should be able to defer the prediction to a human
decision maker to ensure more accurate predictions. In this work, we jointly
train a classifier with a rejector, which decides on each data point whether
the classifier or the human should predict. We show that prior approaches can
fail to find a human-AI system with low misclassification error even when there
exists a linear classifier and rejector that have zero error (the realizable
setting). We prove that obtaining a linear pair with low error is NP-hard even
when the problem is realizable. To complement this negative result, we give a
mixed-integer-linear-programming (MILP) formulation that can optimally solve
the problem in the linear setting. However, the MILP only scales to
moderately-sized problems. Therefore, we provide a novel surrogate loss
function that is realizable-consistent and performs well empirically. We test
our approaches on a comprehensive set of datasets and compare to a wide range
of baselines.Comment: AISTATS 202
Revisiting Discriminative vs. Generative Classifiers: Theory and Implications
A large-scale deep model pre-trained on massive labeled or unlabeled data
transfers well to downstream tasks. Linear evaluation freezes parameters in the
pre-trained model and trains a linear classifier separately, which is efficient
and attractive for transfer. However, little work has investigated the
classifier in linear evaluation except for the default logistic regression.
Inspired by the statistical efficiency of naive Bayes, the paper revisits the
classical topic on discriminative vs. generative classifiers. Theoretically,
the paper considers the surrogate loss instead of the zero-one loss in analyses
and generalizes the classical results from binary cases to multiclass ones. We
show that, under mild assumptions, multiclass naive Bayes requires
samples to approach its asymptotic error while the corresponding multiclass
logistic regression requires samples, where is the feature
dimension. To establish it, we present a multiclass -consistency
bound framework and an explicit bound for logistic loss, which are of
independent interests. Simulation results on a mixture of Gaussian validate our
theoretical findings. Experiments on various pre-trained deep vision models
show that naive Bayes consistently converges faster as the number of data
increases. Besides, naive Bayes shows promise in few-shot cases and we observe
the "two regimes" phenomenon in pre-trained supervised models. Our code is
available at https://github.com/ML-GSAI/Revisiting-Dis-vs-Gen-Classifiers.Comment: Accepted by ICML 2023, 58 page