145 research outputs found
Channel selection for test-time adaptation under distribution shift
To ensure robustness and generalization to real-world scenarios, test-time adaptation has been recently studied as an approach to adjust models to a new data
distribution during inference. Test-time batch normalization is a simple and popular
method that achieved compelling performance on domain shift benchmarks by
recalculating batch normalization statistics on test batches. However, in many
practical applications this technique is vulnerable to label distribution shifts. We
propose to tackle this challenge by only selectively adapting channels in a deep
network, minimizing drastic adaptation that is sensitive to label shifts. We find that
adapted models significantly improve the performance compared to the baseline
models and counteract unknown label shifts
ASI: Accuracy-Stability Index for Evaluating Deep Learning Models
In the context of deep learning research, where model introductions
continually occur, the need for effective and efficient evaluation remains
paramount. Existing methods often emphasize accuracy metrics, overlooking
stability. To address this, the paper introduces the Accuracy-Stability Index
(ASI), a quantitative measure incorporating both accuracy and stability for
assessing deep learning models. Experimental results demonstrate the
application of ASI, and a 3D surface model is presented for visualizing ASI,
mean accuracy, and coefficient of variation. This paper addresses the important
issue of quantitative benchmarking metrics for deep learning models, providing
a new approach for accurately evaluating accuracy and stability of deep
learning models. The paper concludes with discussions on potential weaknesses
and outlines future research directions.Comment: 6 pages, 3 figure
- …