The metric tide: report of the independent review of the role of metrics in research assessment and management

This report presents the findings and recommendations of the Independent Review of the Role of Metrics in Research Assessment and Management. The review was chaired by Professor James Wilsdon, supported by an independent and multidisciplinary group of experts in scientometrics, research funding, research policy, publishing, university management and administration. This review has gone beyond earlier studies to take a deeper look at potential uses and limitations of research metrics and indicators. It has explored the use of metrics across different disciplines, and assessed their potential contribution to the development of research excellence and impact. It has analysed their role in processes of research assessment, including the next cycle of the Research Excellence Framework (REF). It has considered the changing ways in which universities are using quantitative indicators in their management systems, and the growing power of league tables and rankings. And it has considered the negative or unintended effects of metrics on various aspects of research culture. The report starts by tracing the history of metrics in research management and assessment, in the UK and internationally. It looks at the applicability of metrics within different research cultures, compares the peer review system with metric-based alternatives, and considers what balance might be struck between the two. It charts the development of research management systems within institutions, and examines the effects of the growing use of quantitative indicators on different aspects of research culture, including performance management, equality, diversity, interdisciplinarity, and the ‘gaming’ of assessment systems. The review looks at how different funders are using quantitative indicators, and considers their potential role in research and innovation policy. Finally, it examines the role that metrics played in REF2014, and outlines scenarios for their contribution to future exercises

Genericity versus expressivity - an exercise in semantic interoperable research information systems for Web Science

The web does not only enable new forms of science, it also creates new possibilities to study science and new digital scholarship. This paper brings together multiple perspectives: from individual researchers seeking the best options to display their activities and market their skills on the academic job market; to academic institutions, national funding agencies, and countries needing to monitor the science system and account for public money spending. We also address the research interests aimed at better understanding the self-organising and complex nature of the science system through researcher tracing, the identification of the emergence of new fields, and knowledge discovery using large-data mining and non-linear dynamics. In particular this paper draws attention to the need for standardisation and data interoperability in the area of research information as an indispensable pre-condition for any science modelling. We discuss which levels of complexity are needed to provide a globally, interoperable, and expressive data infrastructure for research information. With possible dynamic science model applications in mind, we introduce the need for a "middle-range" level of complexity for data representation and propose a conceptual model for research data based on a core international ontology with national and local extensions.Comment: Long version of a paper submitted to the WebScience 201

A framework for monitoring scientific production behavior in Research evaluation systems based on journal ranking lists the brazilian case

A extensão e a qualidade dos resultados de pesquisa tornaram-se fatores-chave para a avaliação do desempenho da universidade. Vários países introduziram sistemas de avaliação de pesquisas que vinculam financiamento a indicadores de desempenho como forma de aumentar a prestação de contas. Em geral, classificações de periódicos são parte integrante desses sistemas. Esta tese aborda o desenvolvimento e a avaliação de um framework para monitorar o comportamento da produção científica em contextos em que as listas de classificação de periódicos estão no centro da avaliação. O principal objetivo do framework é permitir a identificação de padrões desejáveis e adversos na produção acadêmica. Considerando que o Brasil utiliza um sistema de classificação de periódicos (QUALIS) há mais de duas décadas, o framework foi aplicado numa análise de dez anos da produção científica brasileira em oito áreas distintas, tomando o banco de dados da Scopus como referência. Os resultados mostraram um declínio na proporção de artigos indexados na Scopus nas áreas de Ciências Sociais e Humanas (SSH). Um número restrito de periódicos, que permaneceu no sistema durante o período estudado, concentrou um número maior de artigos. No geral, esses periódicos tiveram sua classificação QUALIS inalterada ou melhorada ao longo das avaliações periódicas. Entretanto, na maioria, houve uma diminuição significativa no impacto de citação desses. Além disso, aqueles de menor impacto passaram para as categorias QUALIS mais altas ao longo dos anos, o que ocorreu simultaneamente com um aumento no número de artigos em periódicos de baixo impacto em todos as áreas. Esses resultados mostraram que o uso de listas de classificação de periódicos pode levar professores e alunos a publicarem em periódicos de alta classificação, apesar de terem um baixo impacto de citação. Quando os periódicos de baixo impacto alcançam uma classificação alta, eles também podem concentrar uma grande quantidade de artigos publicados. De certa forma, esses padrões são semelhantes a outros resultados encontrados na literatura, nos quais um aumento significativo de publicações foi seguido por um declínio no impacto. O efeito potencial desses modelos de avaliação é que eles podem incitar as pessoas a selecionar meios de publicação com uma pontuação mais alta de acordo com os critérios estabelecidos, independentemente de sua visibilidade. Além disso, esses efeitos podem intensificar-se quando os resultados da avaliação são vinculados a financiamento.The extent and quality of research output have become key factors for university performance evaluation. Several countries introduced research evaluation systems that link funding to performance indicators as a way to enhance accountability. In general, journal rankings are an integral part of these systems. This thesis approaches the development and evaluation of a framework for monitoring scientific production behavior in settings where journal-ranking lists are at the center of research assessment. The main goal of the framework is to enable the identification of desirable and adverse patterns in academic production. Considering that Brazil has been using a specific journal ranking system (QUALIS) for more than two decades, the framework was applied in the ten-year analysis of Brazilian scientific production in eight distinct subject fields and taking the Scopus database as a reference. Results showed a decline in the proportion of Scopus-indexed articles in the areas of Social Sciences and Humanities (SSH). A few journals that remained in the system during the whole evaluation period concentrated a larger number of published articles. Overall, these journals had their QUALIS classification unchanged or improved in the ranking over the periodic evaluations. However, in general, there was a significant decrease in their citation impact. Moreover, lower-impact journals moved to the highest QUALIS categories over the years, what happened simultaneously with an increase in the number of articles in low-impact journals in all fields. These results have shown that the use of journal ranking lists may lead faculty and students to submit their papers to highly ranked journals, even though may have a low citation impact. When low-impact journals reach a high rank, they may also concentrate a high amount of published articles. In a certain way, these patterns are similar to other results found in literature, in which a significant increase in publication productivity has been followed by an impact decline. The potential effect of these evaluation models is that they may incite people to select publication venues that make them score higher according to the established criteria, regardless of their publications’ visibility. Besides, this effect can be intensified once the evaluation results are linked to funds

Assessment, Evaluations and Definitions of Research Impact: A Review

This article aims to explore what is understood by the term ‘research impact’ and to provide a comprehensive assimilation of available literature and information, drawing on global experiences to understand the potential for methods and frameworks of impact assessment being implemented for UK impact assessment. We take a more focused look at the impact component of the UK Research Excellence Framework taking place in 2014 and some of the challenges to evaluating impact and the role that systems might play in the future for capturing the links between research and impact and the requirements we have for these systems.Jisc [DIINN10

Data Quality Management

Data quality is crucial in measuring and analyzing science, technology and innovation adequately, which allows for the proper monitoring of research efficiency, productivity and even strategic decision making. In this chapter, the concept of data quality will be defined in terms of the different dimensions that together determine the quality of data. Next, methods will be discussed to measure these dimensions using objective and subjective methods. Specific attention will be paid to the management of data quality through the discussion of critical success factors in operational, managerial and governance processes including training that affect data quality. The chapter will be concluded with a section on data quality improvement, which examines data quality issues and provides roadmaps in order to improve and follow-up on data quality, in order to obtain data that can be used as a reliable source for quantitative and qualitative measurements of research

7 Essays on Impact

Edited by Dr Andrew Dean, Dr Michael Wykes & Hilary Stevens, University of ExeterThrough the Jisc-funded DESCRIBE Project we have sought to undertake a rigorous assessment of current standards relating to the evidence of impacts arising from Higher Education research. This document contains seven valuable essays each exploring the topic of Impact. Each essay is distinct and we have sought to enable selected thought-leaders and Impact experts to both review the status quo, and to look to the future, making suggestions and recommendations for the development of Impact in the sector. DESCRIBE has been managed by the University of Exeter’s Research and Knowledge Transfer team in partnership with the Marchmont Observatory. We have sought to combine the latest thinking on research Impact with examples and recommendations which are practical and rooted in the art of the possible.JISC DIINN1

IRIS: Supporting & Managing the Research Life-cycle

IRIS is a new Current Research Information System (CRIS) developed by Cineca to upgrade and replace two previous solutions that have been used by Italian universities in the last 10 years. At the end of 2015, sixty-three Italian institutions are using IRIS. One of the main components of IRIS is DSpace-CRIS, an open source solution that can also be used as a standalone system, and as such is now installed in seven institutions outside Italy. The many solutions that IRIS provides include support for national research assessment exercises, institutional repositories (IR) and an expert finder. Both IRIS and DSpace-CRIS are compliant with the Common European Research Information Format (CERIF), allowing easy interchange and benchmarking with other CERIF compliant systems. New IRIS functionality includes Open Researcher and Contributor ID (ORCID) push / pull interoperability. IRIS has five different modules integrated together through standard protocols and interfaces.published_or_final_versio

The current state and future perspectives of the research information infrastructure in Croatia

The purpose of this paper is to analyze the existing Croatian research information infrastructure and to outline a new model of the Croatian Current Research Information System (CroRIS), required for the systematical monitoring and evaluation of the research processes and output of the Croatian public research and higher education institutions, as well as for the increasing of their international visibility. Based on the results of the analysis of the existing research information infrastructure in Croatia, the general outline of the new CERIF-based CroRIS model is proposed. The analysis of the existing research information infrastructure showed that there are some functional parts which could be reused and/or used as a basis for the implementation of the proposed CroRIS model. The proposed model represents a concept which would allow a quality decision and science-policy making in Croatia

Research Management Systems: Systematic Mapping of Literature (2007-2017)

Research management in relation to Research and Development (R&D) has found a comprehensive and powerful tool in the Current Research Information System (CRIS). Publications on the subject are still emerging, so knowing what research has been done and its contributions presents an opportunity to build theoretical and reference frameworks, and to identify gaps and potential opportunities for future developments in information technology. Various articles covering these systems were analyzed with the objective of identifying the research areas, countries and institutions in which they were published, as well as their research approaches, contributions and topics covered. A systematic mapping of literature was carried out, which included 33 articles published on Web of Science (WoS) and Scopus, from 2007 to 2017. To ensure valid results, the evaluation guide for researchers was used during for the design and review phases of the systematic mapping studies, as well as the corresponding evaluation rubric; in addition, a strategy to define direct decision rules on how to classify an article based on the results of evaluations of multiple researchers was adopted. The findings show that the main research areas are Information Science & Library Science and Computer Science; the largest concentration of publications is found in Europe; research focuses on Evaluation Research; contributions are oriented to create processes and models, and to describe the applications and uses of CRIS. It is concluded that there are areas of opportunity for the development of research in this area, such as the expansion of the research areas in which it is used, the possibilities of collaboration, sharing and exchange at a global level, and the coverage towards integration issues with open access