2 research outputs found
Evaluating natural language processing models with generalization metrics that do not need access to any training or testing data
The search for effective and robust metrics has been the focus of recent
theoretical and empirical work on generalization of deep neural networks (NNs).
In this paper, we discuss the performance of natural language processing (NLP)
models, and we evaluate various existing and novel generalization metrics.
Compared to prior studies, we (i) focus on NLP instead of computer vision (CV),
(ii) focus on generalization metrics that predict test error instead of the
generalization gap, (iii) focus on generalization metrics that do not need the
access to data, and (iv) focus on the heavy-tail (HT) phenomenon that has
received comparatively less attention in the study of NNs. We extend recent
HT-based work which focuses on power law (PL) distributions, and we study
exponential and exponentially truncated power law (E-TPL) fitting to the
empirical spectral densities (ESDs) of weight matrices. Our empirical studies
are carried on (i) hundreds of Transformers trained in different settings, in
which we systematically vary different hyperparameters, (ii) a total of 51
pretrained Transformers from eight families of Huggingface NLP models,
including BERT, GPT2, etc., and (iii) a total of 28 existing and novel
generalization metrics. From our empirical analyses, we show that shape
metrics, or the metrics obtained from fitting the shape of the ESDs, perform
uniformly better at predicting generalization performance than scale metrics
commonly studied in the literature, as measured by the rank correlations with
the generalization performance. We also show that among the three HT
distributions considered in our paper, the E-TPL fitting of ESDs performs the
most robustly when the models are trained in experimental settings, while the
PL fitting achieves the best performance on well-trained Huggingface models,
and that both E-TPL and PL metrics (which are both shape metrics) outperform
scale metrics