A Flexible and Adaptive Framework for Abstention Under Class Imbalance

In practical applications of machine learning, it is often desirable to identify and abstain on examples where the model's predictions are likely to be incorrect. Much of the prior work on this topic focused on out-of-distribution detection or performance metrics such as top-k accuracy. Comparatively little attention was given to metrics such as area-under-the-curve or Cohen's Kappa, which are extremely relevant for imbalanced datasets. Abstention strategies aimed at top-k accuracy can produce poor results on these metrics when applied to imbalanced datasets, even when all examples are in-distribution. We propose a framework to address this gap. Our framework leverages the insight that calibrated probability estimates can be used as a proxy for the true class labels, thereby allowing us to estimate the change in an arbitrary metric if an example were abstained on. Using this framework, we derive computationally efficient metric-specific abstention algorithms for optimizing the sensitivity at a target specificity level, the area under the ROC, and the weighted Cohen's Kappa. Because our method relies only on calibrated probability estimates, we further show that by leveraging recent work on domain adaptation under label shift, we can generalize to test-set distributions that may have a different class imbalance compared to the training set distribution. On various experiments involving medical imaging, natural language processing, computer vision and genomics, we demonstrate the effectiveness of our approach. Source code available at https://github.com/blindauth/abstention. Colab notebooks reproducing results available at https://github.com/blindauth/abstention_experiments

AUC-based Selective Classification

Selective classification (or classification with a reject option) pairs a classifier with a selection function to determine whether or not a prediction should be accepted. This framework trades off coverage (probability of accepting a prediction) with predictive performance, typically measured by distributive loss functions. In many application scenarios, such as credit scoring, performance is instead measured by ranking metrics, such as the Area Under the ROC Curve (AUC). We propose a model-agnostic approach to associate a selection function to a given probabilistic binary classifier. The approach is specifically targeted at optimizing the AUC. We provide both theoretical justifications and a novel algorithm, called AUCROSS, to achieve such a goal. Experiments show that our method succeeds in trading-off coverage for AUC, improving over existing selective classification methods targeted at optimizing accuracy

Information-Theoretic Measures for Objective Evaluation of Classifications

This work presents a systematic study of objective evaluations of abstaining classifications using Information-Theoretic Measures (ITMs). First, we define objective measures for which they do not depend on any free parameter. This definition provides technical simplicity for examining "objectivity" or "subjectivity" directly to classification evaluations. Second, we propose twenty four normalized ITMs, derived from either mutual information, divergence, or cross-entropy, for investigation. Contrary to conventional performance measures that apply empirical formulas based on users' intuitions or preferences, the ITMs are theoretically more sound for realizing objective evaluations of classifications. We apply them to distinguish "error types" and "reject types" in binary classifications without the need for input data of cost terms. Third, to better understand and select the ITMs, we suggest three desirable features for classification assessment measures, which appear more crucial and appealing from the viewpoint of classification applications. Using these features as "meta-measures", we can reveal the advantages and limitations of ITMs from a higher level of evaluation knowledge. Numerical examples are given to corroborate our claims and compare the differences among the proposed measures. The best measure is selected in terms of the meta-measures, and its specific properties regarding error types and reject types are analytically derived.Comment: 25 Pages, 1 Figure, 10 Table

Localized Randomized Smoothing for Collective Robustness Certification

Models for image segmentation, node classification and many other tasks map a single input to multiple labels. By perturbing this single shared input (e.g. the image) an adversary can manipulate several predictions (e.g. misclassify several pixels). Collective robustness certification is the task of provably bounding the number of robust predictions under this threat model. The only dedicated method that goes beyond certifying each output independently is limited to strictly local models, where each prediction is associated with a small receptive field. We propose a more general collective robustness certificate for all types of models. We further show that this approach is beneficial for the larger class of softly local models, where each output is dependent on the entire input but assigns different levels of importance to different input regions (e.g. based on their proximity in the image). The certificate is based on our novel localized randomized smoothing approach, where the random perturbation strength for different input regions is proportional to their importance for the outputs. Localized smoothing Pareto-dominates existing certificates on both image segmentation and node classification tasks, simultaneously offering higher accuracy and stronger certificates.Comment: Accepted at ICLR 202

The Role of the Physical Therapist in Health Promotion as Perceived by Patients with Neurological Pathologies: A Descriptive Study

Background and Purpose: Patients with neurological pathologies have been found to be less likely to engage in personal health behaviors than the general population. This predisposes them to acquire secondary chronic conditions such as obesity, diabetes, and cardiovascular disease. Studies suggest that this population may be underserved in regards to the promotion of health behaviors. Literature is lacking regarding neurological patients’ perspectives of the physical therapist’s role in promoting personal health behaviors. The purpose of this study was to investigate the perceptions of patients with neurological disability regarding the physical therapist’s role in promoting the personal health behaviors of physical activity, healthy weight management, smoking cessation, and fruit and vegetable consumption. Methods: A convenience sample of patients from a Minneapolis area outpatient rehabilitation center was obtained by physical therapist referral. Surveys were distributed to patients who met the inclusion criteria. The survey obtained information regarding the patient’s perception of what the role of the physical therapist should be for each personal health behavior. Data were analyzed using Microsoft Excel 2013. Results: Thirty-five surveys met inclusion criteria and were analyzed. Respondents’ demographics were as follows: mean age of 52.3±16.7years, 62.9% were male, average BMI of 28.1 ± 6.6 and 73.5% reported having a neurological condition for at least 3 years. A key finding was that 76% of respondents believe that physical therapists should suggest ways to maintain a healthy weight, however it was only addressed with 37% of the sample. The majority of respondents believed physical therapists should advise them about physical activity (88.6%), smoking cessation (65%), and weight management (83%). Conclusion: Overall, respondents with chronic neurological conditions in an outpatient setting who were surveyed expressed the belief that physical therapists should advise them in the personal health behaviors of physical activity, weight management, smoking cessation, and fruit and vegetable intake. Although the majority of respondents believed weight management should be discussed in their therapy sessions, only 37% reported their physical therapist addressed their weight. This finding suggests a potential opportunity for physical therapists to have conversation with their patients on healthy weight management

Survey on Leveraging Uncertainty Estimation Towards Trustworthy Deep Neural Networks: The Case of Reject Option and Post-training Processing

Although neural networks (especially deep neural networks) have achieved \textit{better-than-human} performance in many fields, their real-world deployment is still questionable due to the lack of awareness about the limitation in their knowledge. To incorporate such awareness in the machine learning model, prediction with reject option (also known as selective classification or classification with abstention) has been proposed in literature. In this paper, we present a systematic review of the prediction with the reject option in the context of various neural networks. To the best of our knowledge, this is the first study focusing on this aspect of neural networks. Moreover, we discuss different novel loss functions related to the reject option and post-training processing (if any) of network output for generating suitable measurements for knowledge awareness of the model. Finally, we address the application of the rejection option in reducing the prediction time for the real-time problems and present a comprehensive summary of the techniques related to the reject option in the context of extensive variety of neural networks. Our code is available on GitHub: \url{https://github.com/MehediHasanTutul/Reject_option

An Application of Out-of-Distribution Detection for Two-Stage Object Detection Networks

Recently, much research has been published for detecting when a classification neural network is presented with data that does not fit into one of the class labels the network learned at train time. These so-called out-of-distribution (OOD) detection techniques hold promise for improving safety in systems where unusual or novel inputs may results in errors that endanger human lives. Autonomous vehicles could specifically benefit from the use of these techniques if they could be adapted to detect and localize unusual objects in a driving environment, allowing for such objects to be treated with a high degree of caution. This thesis explores the modification of a selection of existing OOD detection methods from the image classification literature for use in a two-stage object detection network. It is found that the task of detecting objects as being OOD is difficult to define for object detection networks that include a high-variance background class label, but that these methods can instead be adapted for detecting when background regions are incorrectly classified as foreground and when foreground objects of interest are incorrectly classified as background in the final layers of the network. It is found that some methods provide a slight improvement over the baseline method that uses softmax confidence scores for detecting these kinds of errors

MaskTune: Mitigating Spurious Correlations by Forcing to Explore

A fundamental challenge of over-parameterized deep learning models is learning meaningful data representations that yield good performance on a downstream task without over-fitting spurious input features. This work proposes MaskTune, a masking strategy that prevents over-reliance on spurious (or a limited number of) features. MaskTune forces the trained model to explore new features during a single epoch finetuning by masking previously discovered features. MaskTune, unlike earlier approaches for mitigating shortcut learning, does not require any supervision, such as annotating spurious features or labels for subgroup samples in a dataset. Our empirical results on biased MNIST, CelebA, Waterbirds, and ImagenNet-9L datasets show that MaskTune is effective on tasks that often suffer from the existence of spurious correlations. Finally, we show that MaskTune outperforms or achieves similar performance to the competing methods when applied to the selective classification (classification with rejection option) task. Code for MaskTune is available at https://github.com/aliasgharkhani/Masktune.Comment: Accepted to NeurIPS 202