The metric tide: report of the independent review of the role of metrics in research assessment and management

This report presents the findings and recommendations of the Independent Review of the Role of Metrics in Research Assessment and Management. The review was chaired by Professor James Wilsdon, supported by an independent and multidisciplinary group of experts in scientometrics, research funding, research policy, publishing, university management and administration. This review has gone beyond earlier studies to take a deeper look at potential uses and limitations of research metrics and indicators. It has explored the use of metrics across different disciplines, and assessed their potential contribution to the development of research excellence and impact. It has analysed their role in processes of research assessment, including the next cycle of the Research Excellence Framework (REF). It has considered the changing ways in which universities are using quantitative indicators in their management systems, and the growing power of league tables and rankings. And it has considered the negative or unintended effects of metrics on various aspects of research culture. The report starts by tracing the history of metrics in research management and assessment, in the UK and internationally. It looks at the applicability of metrics within different research cultures, compares the peer review system with metric-based alternatives, and considers what balance might be struck between the two. It charts the development of research management systems within institutions, and examines the effects of the growing use of quantitative indicators on different aspects of research culture, including performance management, equality, diversity, interdisciplinarity, and the ‘gaming’ of assessment systems. The review looks at how different funders are using quantitative indicators, and considers their potential role in research and innovation policy. Finally, it examines the role that metrics played in REF2014, and outlines scenarios for their contribution to future exercises

Should Top Universities Be Led By Top Researchers and Are They? A Citations Analysis

[Excerpt] This paper addresses the question: should the world’s top universities be led by top researchers, and are they? The lifetime citations are counted by hand of the leaders of the world’s top 100 universities identified in a global university ranking. These numbers are then normalized by adjusting for the different citation conventions across academic disciplines. Two statistical measures are used -- Pearson\u27s correlation coefficient and Spearman\u27s rho. This study documents a positive correlation between the lifetime citations of a University’s president and the position of that university in the global ranking. Better universities are run by better researchers. The results are not driven by outliers. That the top universities in the world -- who have the widest choice of candidates -- systematically appoint top researchers as their vice chancellors and presidents seems important to understand. This paper also shows that the pattern of presidents life-time citations follows a version of Lotka’s power law. There are two main areas of contribution. First, this paper attempts to use bibliometric data to address a performance- related question of a type not seen before (to the author’s knowledge). Second, despite the importance of research to research universities -- as described in many mission-statements -- no studies currently exist that ask whether it matters if the head of a research university is himself or herself a committed researcher. Given the importance of universities in the world, and the difficulty that many have in appointing leaders, this question seems pertinent

Evaluating academic research in Germany: patterns and policies

In this country study of Germany the patterns and policies of academic research as well as the evaluation of academic research are analyzed, through applying the following approach: first of all, a bibliometric survey is carried out that investigates the publication output and publication efficiency of Germany’s academic research within international journals; we further investigate whether the results of a bibliometric survey appear compatible with the performance of other indicators. Secondly, discourse and policies of the evaluation of Germany’s university research are investigated by addressing issues such as: the current situation; the structural and cultural constraints against evaluations; the general reasons why evaluations of university research will play an increasingly important role in the future; and an overview of specific evaluation initiatives. Thirdly and finally, also the discourse and policies of the evaluation of Germany’s university-related research are examined. [author's abstract

Ranking institutions within a university based on their scientific performance: A percentile-based approach

Over the recent years, the subject of university rankings has attracted a significant amount of attention and sparked a scientific debate. However, few studies on this topic focus on elaborating the scientific performance of universities’ institutions, such as institutes, schools, and faculties. For this reason, the aim of this study is to design an appropriate framework for evaluating and ranking institutions within a university. The devised methodology ranks institutions based on the number of published papers, mean normalized citation score (MNCS), and four percentile-based indicators using the I-distance method. We applied the proposed framework and scrutinized the University of Belgrade (UB) as the biggest and the best-ranked university in Serbia. Thus, 31 faculties and 11 institutes were compared. Namely, an in-depth percentile-based analysis of the UB papers indexed in the Science Citation Index Expanded (SCIe) and the Social Science Citation Index (SSCI) for the period 2008-2011 is provided. The results clearly show considerable discrepancies in two occasions: first, when it comes to the question of leading author, and second, when it comes to analyzing the percentile rank classes (PRs) of groups of faculties