The metric tide: report of the independent review of the role of metrics in research assessment and management

This report presents the findings and recommendations of the Independent Review of the Role of Metrics in Research Assessment and Management. The review was chaired by Professor James Wilsdon, supported by an independent and multidisciplinary group of experts in scientometrics, research funding, research policy, publishing, university management and administration. This review has gone beyond earlier studies to take a deeper look at potential uses and limitations of research metrics and indicators. It has explored the use of metrics across different disciplines, and assessed their potential contribution to the development of research excellence and impact. It has analysed their role in processes of research assessment, including the next cycle of the Research Excellence Framework (REF). It has considered the changing ways in which universities are using quantitative indicators in their management systems, and the growing power of league tables and rankings. And it has considered the negative or unintended effects of metrics on various aspects of research culture. The report starts by tracing the history of metrics in research management and assessment, in the UK and internationally. It looks at the applicability of metrics within different research cultures, compares the peer review system with metric-based alternatives, and considers what balance might be struck between the two. It charts the development of research management systems within institutions, and examines the effects of the growing use of quantitative indicators on different aspects of research culture, including performance management, equality, diversity, interdisciplinarity, and the ‘gaming’ of assessment systems. The review looks at how different funders are using quantitative indicators, and considers their potential role in research and innovation policy. Finally, it examines the role that metrics played in REF2014, and outlines scenarios for their contribution to future exercises

Do prestigious Spanish scholarly book publishers have more teaching impact?

Purpose The purpose of this paper is to assess the educational value of prestigious and productive Spanish scholarly publishers based on mentions of their books in online scholarly syllabi. Design/methodology/approach Syllabus mentions of 15,117 books from 27 publishers were searched for, manually checked and compared with Microsoft Academic (MA) citations. Findings Most books published by Ariel, Síntesis, Tecnos and Cátedra have been mentioned in at least one online syllabus, indicating that their books have consistently high educational value. In contrast, few books published by the most productive publishers were mentioned in online syllabi. Prestigious publishers have both the highest educational impact based on syllabus mentions and the highest research impact based on MA citations. Research limitations/implications The results might be different for other publishers. The online syllabus mentions found may be a small fraction of the syllabus mentions of the sampled books. Practical implications Authors of Spanish-language social sciences and humanities books should consider general prestige when selecting a publisher if they want educational uptake for their work. Originality/value This is the first study assessing book publishers based on syllabus mentions

Do prestigious Spanish scholarly book publishers have more teaching impact?

Purpose The purpose of this paper is to assess the educational value of prestigious and productive Spanish scholarly publishers based on mentions of their books in online scholarly syllabi. Design/methodology/approach Syllabus mentions of 15,117 books from 27 publishers were searched for, manually checked and compared with Microsoft Academic (MA) citations. Findings Most books published by Ariel, Síntesis, Tecnos and Cátedra have been mentioned in at least one online syllabus, indicating that their books have consistently high educational value. In contrast, few books published by the most productive publishers were mentioned in online syllabi. Prestigious publishers have both the highest educational impact based on syllabus mentions and the highest research impact based on MA citations. Research limitations/implications The results might be different for other publishers. The online syllabus mentions found may be a small fraction of the syllabus mentions of the sampled books. Practical implications Authors of Spanish-language social sciences and humanities books should consider general prestige when selecting a publisher if they want educational uptake for their work. Originality/value This is the first study assessing book publishers based on syllabus mentions

Normative versus strategic accounts of acknowledgment data: the case of the top-five journals of economics

Two alternative accounts can be given of the information contained in the acknowledgments of academic publications. According to the mainstream normative account, the acknowledgments serve to repay debts towards formal or informal collaborators. According to the strategic account, by contrast, the acknowledgments serve to increase the perceived quality of papers by associating the authors with influential scholars. The two accounts are assessed by analyzing the acknowledgments indexed in Web of Science of 1218 articles published in the "top-five journals" of economics for the years 2015-2019. The analysis is focused on six dimensions: (i) the style of acknowledging texts, (ii) the distribution of mentions, (iii) the identity of the most mentioned acknowledgees, (iv) the shares of highly and lowly mentioned acknowledgees, (v) the hierarchy of the acknowledgment network, and (vi) the correlation at a paper level between intellectual similarity, measured by common references, and social similarity, measured by common acknowledges. Results show that the normative and the strategic account should be considered as valid but partial explanations of acknowledging behavior. Hence, acknowledgments should be used with extreme caution for investigating collaboration practices and they should not be used to produce acknowledgments-based metrics of scholars for evaluative purposes.Comment: 46 pages, 5 figures, 5 table

Mining Scholarly Publications for Research Evaluation

Scientific research can lead to breakthroughs that revolutionise society by solving long-standing problems. However, investment of public funds into research requires the ability to clearly demonstrate beneficial returns, accountability, and good management. At the same time, with the amount of scholarly literature rapidly expanding, recognising key research that presents the most important contributions to science is becoming increasingly difficult and time-consuming. This creates a need for effective and appropriate research evaluation methods. However, the question of how to evaluate the quality of research outcomes is very difficult to answer and despite decades of research, there is still no standard solution to this problem. Given this growing need for research evaluation, it is increasingly important to understand how research should be evaluated, and whether the existing methods meet this need. However, the current solutions, which are predominantly based on counting the number of interactions in the scholarly communication network, are insufficient for a number of reasons. In particular, they struggle in capturing many aspects of the academic culture and often significantly lag behind current developments. This work focuses on the evaluation of research publications and aims at creating new methods which utilise publication content. It studies the concept of research publication quality, methods assessing the performance of new research publication evaluation methods, analyses and extends the existing methods, and, most importantly, presents a new class of metrics which are based on publication manuscripts. By bridging the fields of research evaluation and text- and data-mining, this work provides tools for analysing the outcomes of research, and for relieving information overload in scholarly publishing

The Social Systems Citation Theory (SSCT): A proposal to use the social systems theory for conceptualizing publications and their citations links

The normative theory of citing considers citations as rewarding tools to acknowledge the influence of scientific works, while the social constructivist theory of citing considers citations, for example, as persuasion tools used by authors to support their claims, and convince the scientific community that those claims are valid. Other citation theories and models have been proposed in recent years to overcome the limitations of the normative and social constructivist theories. Nevertheless, they have not been able to fully explain all citation motives of scientists (but have a certain focus). This study proposes a new theory (which we call “social systems citation theory”, SSCT) that integrates previous theories and models on publications and their citation links and is mainly based on Niklas Luhmann’s “social systems theory”. Luhmann’s social systems theory focuses on “communications” as the basic constituting elements of a social science system and not on humans and their motives. Humans are not part of social systems but are connected with them and irritate them. Thus, the social systems theory does not have the problem of integrating various and different motives of humans to cite in the science system. In the SSCT, authors’ motives to cite belong to psychic systems while publications and their citation links belong to the social science system. The systems operate autonomously but interact with each other: the social system operates recursively with publications and citation links. Although psychic systems stimulate or irritate the science system, they do not determine communications in the science system. In this study, we explain the SSCT and demonstrate how the theory can be used to underlie empirical bibliometric studies

Interdisciplinarity and research on local issues: evidence from a developing country

This paper explores the relationship between interdisciplinarity and research pertaining to local issues. Using Colombian publications from 1991 until 2011 in the Web of Science, we investigate the relationship between the degree of interdisciplinarity and the local orientation of the articles. We find that a higher degree of interdisciplinarity in a publication is associated with a greater emphasis on Colombian issues. In particular, our results suggest that research that combines cognitively disparate disciplines, what we refer to as distal interdisciplinarity, tends to be associated with more local focus of research. We discuss the implications of these results in the context of policies aiming to foster the local socio-economic impact of research in developing countries.Comment: 24 page