TorchCP: A Library for Conformal Prediction based on PyTorch

TorchCP is a Python toolbox for conformal prediction research on deep learning models. It contains various implementations for posthoc and training methods for classification and regression tasks (including multi-dimension output). TorchCP is built on PyTorch (Paszke et al., 2019) and leverages the advantages of matrix computation to provide concise and efficient inference implementations. The code is licensed under the LGPL license and is open-sourced at $\href{https://github.com/ml-stat-Sustech/TorchCP}{\text{this https URL}}$

Shot noise of spin current and spin transfer torque

We report the theoretical investigation of noise spectrum of spin current and spin transfer torque for non-colinear spin polarized transport in a spin-valve device which consists of normal scattering region connected by two ferromagnetic electrodes. Our theory was developed using non-equilibrium Green's function method and general non-linear $S^\sigma-V$ and $S^\tau-V$ relations were derived as a function of angle $\theta$ between magnetization of two leads. We have applied our theory to a quantum dot system with a resonant level coupled with two ferromagnetic electrodes. It was found that for the MNM system, the auto-correlation of spin current is enough to characterize the fluctuation of spin current. For a system with three ferromagnetic layers, however, both auto-correlation and cross-correlation of spin current are needed to characterize the noise spectrum of spin current. Furthermore, the spin transfer torque and the torque noise were studied for the MNM system. For a quantum dot with a resonant level, the derivative of spin torque with respect to bias voltage is proportional to $\sin\theta$ when the system is far away from the resonance. When the system is near the resonance, the spin transfer torque becomes non-sinusoidal function of $\theta$. The derivative of noise spectrum of spin transfer torque with respect to the bias voltage $N_\tau$ behaves differently when the system is near or far away from the resonance. Specifically, the differential shot noise of spin transfer torque $N_\tau$ is a concave function of $\theta$ near the resonance while it becomes convex function of $\theta$ far away from resonance. For certain bias voltages, the period $N_\tau(\theta)$ becomes $\pi$ instead of $2\pi$. For small $\theta$, it was found that the differential shot noise of spin transfer torque is very sensitive to the bias voltage and the other system parameters.Comment: 15pages, 6figure

Does Confidence Calibration Help Conformal Prediction?

Conformal prediction, as an emerging uncertainty qualification technique, constructs prediction sets that are guaranteed to contain the true label with high probability. Previous works usually employ temperature scaling to calibrate the classifier, assuming that confidence calibration can benefit conformal prediction. In this work, we first show that post-hoc calibration methods surprisingly lead to larger prediction sets with improved calibration, while over-confidence with small temperatures benefits the conformal prediction performance instead. Theoretically, we prove that high confidence reduces the probability of appending a new class in the prediction set. Inspired by the analysis, we propose a novel method, $\textbf{Conformal Temperature Scaling}$ (ConfTS), which rectifies the objective through the gap between the threshold and the non-conformity score of the ground-truth label. In this way, the new objective of ConfTS will optimize the temperature value toward an optimal set that satisfies the $\textit{marginal coverage}$. Experiments demonstrate that our method can effectively improve widely-used conformal prediction methods

In Defense of Softmax Parametrization for Calibrated and Consistent Learning to Defer

Enabling machine learning classifiers to defer their decision to a downstream expert when the expert is more accurate will ensure improved safety and performance. This objective can be achieved with the learning-to-defer framework which aims to jointly learn how to classify and how to defer to the expert. In recent studies, it has been theoretically shown that popular estimators for learning to defer parameterized with softmax provide unbounded estimates for the likelihood of deferring which makes them uncalibrated. However, it remains unknown whether this is due to the widely used softmax parameterization and if we can find a softmax-based estimator that is both statistically consistent and possesses a valid probability estimator. In this work, we first show that the cause of the miscalibrated and unbounded estimator in prior literature is due to the symmetric nature of the surrogate losses used and not due to softmax. We then propose a novel statistically consistent asymmetric softmax-based surrogate loss that can produce valid estimates without the issue of unboundedness. We further analyze the non-asymptotic properties of our method and empirically validate its performance and calibration on benchmark datasets.Comment: NeurIPS 202

On the Importance of Feature Separability in Predicting Out-Of-Distribution Error

Estimating the generalization performance is practically challenging on out-of-distribution (OOD) data without ground truth labels. While previous methods emphasize the connection between distribution difference and OOD accuracy, we show that a large domain gap not necessarily leads to a low test accuracy. In this paper, we investigate this problem from the perspective of feature separability, and propose a dataset-level score based upon feature dispersion to estimate the test accuracy under distribution shift. Our method is inspired by desirable properties of features in representation learning: high inter-class dispersion and high intra-class compactness. Our analysis shows that inter-class dispersion is strongly correlated with the model accuracy, while intra-class compactness does not reflect the generalization performance on OOD data. Extensive experiments demonstrate the superiority of our method in both prediction performance and computational efficiency