Cross-Entropy Loss Functions: Theoretical Analysis and Applications

Cross-entropy is a widely used loss function in applications. It coincides with the logistic loss applied to the outputs of a neural network, when the softmax is used. But, what guarantees can we rely on when using cross-entropy as a surrogate loss? We present a theoretical analysis of a broad family of losses, comp-sum losses, that includes cross-entropy (or logistic loss), generalized cross-entropy, the mean absolute error and other loss cross-entropy-like functions. We give the first $H$-consistency bounds for these loss functions. These are non-asymptotic guarantees that upper bound the zero-one loss estimation error in terms of the estimation error of a surrogate loss, for the specific hypothesis set $H$ used. We further show that our bounds are tight. These bounds depend on quantities called minimizability gaps, which only depend on the loss function and the hypothesis set. To make them more explicit, we give a specific analysis of these gaps for comp-sum losses. We also introduce a new family of loss functions, smooth adversarial comp-sum losses, derived from their comp-sum counterparts by adding in a related smooth term. We show that these loss functions are beneficial in the adversarial setting by proving that they admit $H$-consistency bounds. This leads to new adversarial robustness algorithms that consist of minimizing a regularized smooth adversarial comp-sum loss. While our main purpose is a theoretical analysis, we also present an extensive empirical analysis comparing comp-sum losses. We further report the results of a series of experiments demonstrating that our adversarial robustness algorithms outperform the current state-of-the-art, while also achieving a superior non-adversarial accuracy

Predictor-Rejector Multi-Class Abstention: Theoretical Analysis and Algorithms

We study the key framework of learning with abstention in the multi-class classification setting. In this setting, the learner can choose to abstain from making a prediction with some pre-defined cost. We present a series of new theoretical and algorithmic results for this learning problem in the predictor-rejector framework. We introduce several new families of surrogate losses for which we prove strong non-asymptotic and hypothesis set-specific consistency guarantees, thereby resolving positively two existing open questions. These guarantees provide upper bounds on the estimation error of the abstention loss function in terms of that of the surrogate loss. We analyze both a single-stage setting where the predictor and rejector are learned simultaneously and a two-stage setting crucial in applications, where the predictor is learned in a first stage using a standard surrogate loss such as cross-entropy. These guarantees suggest new multi-class abstention algorithms based on minimizing these surrogate losses. We also report the results of extensive experiments comparing these algorithms to the current state-of-the-art algorithms on CIFAR-10, CIFAR-100 and SVHN datasets. Our results demonstrate empirically the benefit of our new surrogate losses and show the remarkable performance of our broadly applicable two-stage abstention algorithm

Regression with Multi-Expert Deferral

Learning to defer with multiple experts is a framework where the learner can choose to defer the prediction to several experts. While this problem has received significant attention in classification contexts, it presents unique challenges in regression due to the infinite and continuous nature of the label space. In this work, we introduce a novel framework of regression with deferral, which involves deferring the prediction to multiple experts. We present a comprehensive analysis for both the single-stage scenario, where there is simultaneous learning of predictor and deferral functions, and the two-stage scenario, which involves a pre-trained predictor with a learned deferral function. We introduce new surrogate loss functions for both scenarios and prove that they are supported by $H$-consistency bounds. These bounds provide consistency guarantees that are stronger than Bayes consistency, as they are non-asymptotic and hypothesis set-specific. Our framework is versatile, applying to multiple experts, accommodating any bounded regression losses, addressing both instance-dependent and label-dependent costs, and supporting both single-stage and two-stage methods. A by-product is that our single-stage formulation includes the recent regression with abstention framework (Cheng et al., 2023) as a special case, where only a single expert, the squared loss and a label-independent cost are considered. Minimizing our proposed loss functions directly leads to novel algorithms for regression with deferral. We report the results of extensive experiments showing the effectiveness of our proposed algorithms

Miss Rate Prediction across All Program Inputs

Improving cache performance requires understanding cache behavior. However, measuring cache performance for one or two data input sets provides little insight into how cache behavior varies across all data input sets. This paper uses our recently published locality analysis to generate a parameterized model of program cache behavior. Given a cache size and associativity, this model predicts the miss rate for arbitrary data input set sizes. This model also identifies critical data input sizes where cache behavior exhibits marked changes. Experiments show this technique is within 2% of the hit rate for set associative caches on a set of integer and floating-point programs

Self-healing WS2 tribofilms: An in-situ appraisal of mechanisms

Self-healing tribocoatings are being developed for aerospace applications to improve the lifetime and reduce the surface maintenance of components in motion. Here the tribo-induced self-healing behaviour of a WS2/a-C tribocoating has been evaluated for the first time by in-situ scanning electron microscopy (SEM) to evaluate the mechanisms of damage and self-recovery. In-situ SEM imaging reveals that scratch damage results in coating brittle fracture and spalling, and that Hertzian pressure affects healing rate at early stages of sliding. WS2 nanocrystallites, formed via atomic rearrangement at flexural interfaces, enable the healing of irregular damages and congruently offer superlubrication in vacuum. Such damage control in tribo-service may make flawless coatings an unnecessary prerequisite in tribo-applications

Generalisability of and lessons learned from a mixed-methods study conducted in three low- and middle-income countries to identify care pathways for atrial fibrillation

BackgroundIdentifying existing care pathways is the first step for understanding how services can be improved to enable early diagnosis and effective follow-up care for non-communicable diseases (NCDs); however, evidence on how care pathways can and should be identified in low- and middle-income countries (LMICs) is lacking.ObjectiveTo describe generalisability and lessons learned from recruitment and data collection for the quantitative component of a mixed methods study designed to determine the care pathway for atrial fibrillation (AF) in Brazil, China and Sri Lanka.MethodsAdults (≥18 years) that spoke the local language and with an AF diagnosis were eligible. We excluded anyone with a hearing or cognitive impairment or ineligible address. Eligible participants were identified using electronic records in Brazil and China; in Sri Lanka, researchers attended the outpatient clinics to identify eligible participants. Data were collected using two quantitative questionnaires administered at least 2-months apart. A minimum sample size of 238 was required for each country.ResultsThe required sample size was met in Brazil (n = 267) and China (n = 298), but a large proportion of AF patients could not be contacted (47% and 27%, respectively) or refused to participate (36% and 38%, respectively). In Sri Lanka, recruitment was challenging, resulting in a reduced sample (n = 151). Mean age of participants from Brazil, China and Sri Lanka was 69 (SD = 11.3), 65 (SD = 12.8) and 58 (SD = 11.7), respectively. Females accounted for 49% of the Brazil sample, 62% in China and 70% in Sri Lanka.ConclusionsGeneralisability was an issue in Brazil and China, as was selection bias. Recruitment bias was highlighted in Sri Lanka. Additional or alternative recruitment methods may be required to ensure generalisability and reduce bias in future studies aimed at identifying NCD care pathways in LMICs