Peer assessment or promotion by numbers? A comparative study of different measures of researcher performance within the UK Library and Information Science research community
Hirsch’s h-index, Egghe’s g-index, total citation and publication counts, and five proposed new metrics were correlated with one another using Spearman’s Rank Correlation for one hundred randomly selected academics and researchers working in UK Library and Information Science departments. Metrics were compared for individuals of different genders and at institutions awarded different RAE (2001) grades. Individuals’ metrics were rank-correlated against academic ranks and RAE (2001) grades of their employing departments. Metrics calculated using Web of Science and Google Scholar data were compared. Peer- and h-index metric-ranked orders of researchers were rank-correlated. Citation behaviour and attitudes towards peer and citation-based assessment of 263 academics and researchers were investigated by factor analysis of online attitudinal survey responses.
h increased curvilinearly with total citation and publication counts, suggesting that h was constrained by the activity in the field preventing individuals producing enough heavily cited publications to increase their h-index scores. Most individuals therefore shared similar h-index scores, making interpersonal comparisons difficult. Total citation counts and Bihui’s a-index scores distinguished between more individuals, though whether they could confidently identify differences between individuals is uncertain. Both databases arbitrarily omitted individuals and publications, systematically biasing citation metrics calculated using them.
In contrast to studies of larger fields, no citation metrics correlated with RAE grade, academic rank, or direct peer-assessment, suggesting that citation-based assessment is unsuitable for research fields with relatively little research activity. No gender bias was evident in academic rank, esteem or citedness.
At least nine independent factors influence citation behaviour. Mertonian factors dominated. The independence of the factors suggested different individuals have different combinations of non-Mertonian motivations. The overriding meaning of citations was confirmed as signals of relevance and reward.
Recommendations for future research include a need to develop simple, robust methods to identify subfields and normalise citations across subfields, to quantify the impact of random bias and to determine whether it varies across subfields, and to study the rate of accumulation of citations and citation distribution changes for individuals (and departments) over time to determine whether career age can be controlled for, in particular
