5 research outputs found
Parameterized Temperature Scaling for Boosting the Expressive Power in Post-Hoc Uncertainty Calibration
We address the problem of uncertainty calibration and introduce a novel
calibration method, Parametrized Temperature Scaling (PTS). Standard deep
neural networks typically yield uncalibrated predictions, which can be
transformed into calibrated confidence scores using post-hoc calibration
methods. In this contribution, we demonstrate that the performance of
accuracy-preserving state-of-the-art post-hoc calibrators is limited by their
intrinsic expressive power. We generalize temperature scaling by computing
prediction-specific temperatures, parameterized by a neural network. We show
with extensive experiments that our novel accuracy-preserving approach
consistently outperforms existing algorithms across a large number of model
architectures, datasets and metrics.Comment: Technical repor
Quality Control at Your Fingertips: Quality-Aware Translation Models
Maximum-a-posteriori (MAP) decoding is the most widely used decoding strategy
for neural machine translation (NMT) models. The underlying assumption is that
model probability correlates well with human judgment, with better translations
being more likely. However, research has shown that this assumption does not
always hold, and decoding strategies which directly optimize a utility
function, like Minimum Bayes Risk (MBR) or Quality-Aware decoding can
significantly improve translation quality over standard MAP decoding. The main
disadvantage of these methods is that they require an additional model to
predict the utility, and additional steps during decoding, which makes the
entire process computationally demanding. In this paper, we propose to make the
NMT models themselves quality-aware by training them to estimate the quality of
their own output. During decoding, we can use the model's own quality estimates
to guide the generation process and produce the highest-quality translations
possible. We demonstrate that the model can self-evaluate its own output during
translation, eliminating the need for a separate quality estimation model.
Moreover, we show that using this quality signal as a prompt during MAP
decoding can significantly improve translation quality. When using the internal
quality estimate to prune the hypothesis space during MBR decoding, we can not
only further improve translation quality, but also reduce inference speed by
two orders of magnitude
Towards Trustworthy Predictions from Deep Neural Networks with Fast Adversarial Calibration
To facilitate a wide-spread acceptance of AI systems guiding decision making
in real-world applications, trustworthiness of deployed models is key. That is,
it is crucial for predictive models to be uncertainty-aware and yield
well-calibrated (and thus trustworthy) predictions for both in-domain samples
as well as under domain shift. Recent efforts to account for predictive
uncertainty include post-processing steps for trained neural networks, Bayesian
neural networks as well as alternative non-Bayesian approaches such as ensemble
approaches and evidential deep learning. Here, we propose an efficient yet
general modelling approach for obtaining well-calibrated, trustworthy
probabilities for samples obtained after a domain shift. We introduce a new
training strategy combining an entropy-encouraging loss term with an
adversarial calibration loss term and demonstrate that this results in
well-calibrated and technically trustworthy predictions for a wide range of
domain drifts. We comprehensively evaluate previously proposed approaches on
different data modalities, a large range of data sets including sequence data,
network architectures and perturbation strategies. We observe that our
modelling approach substantially outperforms existing state-of-the-art
approaches, yielding well-calibrated predictions under domain drift.Comment: In Thirty-Fifth AAAI Conference on Artificial Intelligence
(AAAI-2021). Code available at https://github.com/tochris/falco
Beyond In-Domain Scenarios: Robust Density-Aware Calibration
Calibrating deep learning models to yield uncertainty-aware predictions is
crucial as deep neural networks get increasingly deployed in safety-critical
applications. While existing post-hoc calibration methods achieve impressive
results on in-domain test datasets, they are limited by their inability to
yield reliable uncertainty estimates in domain-shift and out-of-domain (OOD)
scenarios. We aim to bridge this gap by proposing DAC, an accuracy-preserving
as well as Density-Aware Calibration method based on k-nearest-neighbors (KNN).
In contrast to existing post-hoc methods, we utilize hidden layers of
classifiers as a source for uncertainty-related information and study their
importance. We show that DAC is a generic method that can readily be combined
with state-of-the-art post-hoc methods. DAC boosts the robustness of
calibration performance in domain-shift and OOD, while maintaining excellent
in-domain predictive uncertainty estimates. We demonstrate that DAC leads to
consistently better calibration across a large number of model architectures,
datasets, and metrics. Additionally, we show that DAC improves calibration
substantially on recent large-scale neural networks pre-trained on vast amounts
of data.Comment: In Proceedings of the International Conference on Machine Learning
(ICML), 2023. Code available at
https://github.com/futakw/DensityAwareCalibratio