A framework for monitoring scientific production behavior in Research evaluation systems based on journal ranking lists the brazilian case

A extensão e a qualidade dos resultados de pesquisa tornaram-se fatores-chave para a avaliação do desempenho da universidade. Vários países introduziram sistemas de avaliação de pesquisas que vinculam financiamento a indicadores de desempenho como forma de aumentar a prestação de contas. Em geral, classificações de periódicos são parte integrante desses sistemas. Esta tese aborda o desenvolvimento e a avaliação de um framework para monitorar o comportamento da produção científica em contextos em que as listas de classificação de periódicos estão no centro da avaliação. O principal objetivo do framework é permitir a identificação de padrões desejáveis e adversos na produção acadêmica. Considerando que o Brasil utiliza um sistema de classificação de periódicos (QUALIS) há mais de duas décadas, o framework foi aplicado numa análise de dez anos da produção científica brasileira em oito áreas distintas, tomando o banco de dados da Scopus como referência. Os resultados mostraram um declínio na proporção de artigos indexados na Scopus nas áreas de Ciências Sociais e Humanas (SSH). Um número restrito de periódicos, que permaneceu no sistema durante o período estudado, concentrou um número maior de artigos. No geral, esses periódicos tiveram sua classificação QUALIS inalterada ou melhorada ao longo das avaliações periódicas. Entretanto, na maioria, houve uma diminuição significativa no impacto de citação desses. Além disso, aqueles de menor impacto passaram para as categorias QUALIS mais altas ao longo dos anos, o que ocorreu simultaneamente com um aumento no número de artigos em periódicos de baixo impacto em todos as áreas. Esses resultados mostraram que o uso de listas de classificação de periódicos pode levar professores e alunos a publicarem em periódicos de alta classificação, apesar de terem um baixo impacto de citação. Quando os periódicos de baixo impacto alcançam uma classificação alta, eles também podem concentrar uma grande quantidade de artigos publicados. De certa forma, esses padrões são semelhantes a outros resultados encontrados na literatura, nos quais um aumento significativo de publicações foi seguido por um declínio no impacto. O efeito potencial desses modelos de avaliação é que eles podem incitar as pessoas a selecionar meios de publicação com uma pontuação mais alta de acordo com os critérios estabelecidos, independentemente de sua visibilidade. Além disso, esses efeitos podem intensificar-se quando os resultados da avaliação são vinculados a financiamento.The extent and quality of research output have become key factors for university performance evaluation. Several countries introduced research evaluation systems that link funding to performance indicators as a way to enhance accountability. In general, journal rankings are an integral part of these systems. This thesis approaches the development and evaluation of a framework for monitoring scientific production behavior in settings where journal-ranking lists are at the center of research assessment. The main goal of the framework is to enable the identification of desirable and adverse patterns in academic production. Considering that Brazil has been using a specific journal ranking system (QUALIS) for more than two decades, the framework was applied in the ten-year analysis of Brazilian scientific production in eight distinct subject fields and taking the Scopus database as a reference. Results showed a decline in the proportion of Scopus-indexed articles in the areas of Social Sciences and Humanities (SSH). A few journals that remained in the system during the whole evaluation period concentrated a larger number of published articles. Overall, these journals had their QUALIS classification unchanged or improved in the ranking over the periodic evaluations. However, in general, there was a significant decrease in their citation impact. Moreover, lower-impact journals moved to the highest QUALIS categories over the years, what happened simultaneously with an increase in the number of articles in low-impact journals in all fields. These results have shown that the use of journal ranking lists may lead faculty and students to submit their papers to highly ranked journals, even though may have a low citation impact. When low-impact journals reach a high rank, they may also concentrate a high amount of published articles. In a certain way, these patterns are similar to other results found in literature, in which a significant increase in publication productivity has been followed by an impact decline. The potential effect of these evaluation models is that they may incite people to select publication venues that make them score higher according to the established criteria, regardless of their publications’ visibility. Besides, this effect can be intensified once the evaluation results are linked to funds

7 Essays on Impact

Edited by Dr Andrew Dean, Dr Michael Wykes & Hilary Stevens, University of ExeterThrough the Jisc-funded DESCRIBE Project we have sought to undertake a rigorous assessment of current standards relating to the evidence of impacts arising from Higher Education research. This document contains seven valuable essays each exploring the topic of Impact. Each essay is distinct and we have sought to enable selected thought-leaders and Impact experts to both review the status quo, and to look to the future, making suggestions and recommendations for the development of Impact in the sector. DESCRIBE has been managed by the University of Exeter’s Research and Knowledge Transfer team in partnership with the Marchmont Observatory. We have sought to combine the latest thinking on research Impact with examples and recommendations which are practical and rooted in the art of the possible.JISC DIINN1

Assessing the Benefits of Public Research Within an Economic Framework: The Case of USDA's Agricultural Research Service

Evaluation of publicly funded research can help provide accountability and prioritize programs. In addition, Federal intramural research planning generally involves an institutional assessment of the appropriate Federal role, if any, and whether the research should be left to others, such as universities or the private sector. Many methods of evaluation are available, peer review—used primarily for establishing scientific merit—being the most common. Economic analysis focuses on quantifying ultimate research outcomes, whether measured in goods with market prices or in nonmarket goods such as environmental quality or human health. However, standard economic techniques may not be amenable for evaluating some important public research priorities or for institutional assessments. This report reviews quantitative methods and applies qualitative economic reasoning and stakeholder interviewing methods to the evaluation of economic benefits of Federal intramural research using three case studies of research conducted by USDA’s Agricultural Research Service (ARS). Differences among the case studies highlight the need to select suitable assessment techniques from available methodologies, the limited scope for comparing assessment results across programs, and the inherent difficulty in quantifying benefits in some research areas. When measurement and attribution issues make it difficult to quantify these benefits, the report discusses how qualitative insights based on economic concepts can help research prioritization.Agricultural Research Service, Federal intramural research, publicly funded research, Environmental Economics and Policy, Food Consumption/Nutrition/Food Safety, Livestock Production/Industries, Productivity Analysis,

The metric tide: report of the independent review of the role of metrics in research assessment and management

This report presents the findings and recommendations of the Independent Review of the Role of Metrics in Research Assessment and Management. The review was chaired by Professor James Wilsdon, supported by an independent and multidisciplinary group of experts in scientometrics, research funding, research policy, publishing, university management and administration. This review has gone beyond earlier studies to take a deeper look at potential uses and limitations of research metrics and indicators. It has explored the use of metrics across different disciplines, and assessed their potential contribution to the development of research excellence and impact. It has analysed their role in processes of research assessment, including the next cycle of the Research Excellence Framework (REF). It has considered the changing ways in which universities are using quantitative indicators in their management systems, and the growing power of league tables and rankings. And it has considered the negative or unintended effects of metrics on various aspects of research culture. The report starts by tracing the history of metrics in research management and assessment, in the UK and internationally. It looks at the applicability of metrics within different research cultures, compares the peer review system with metric-based alternatives, and considers what balance might be struck between the two. It charts the development of research management systems within institutions, and examines the effects of the growing use of quantitative indicators on different aspects of research culture, including performance management, equality, diversity, interdisciplinarity, and the ‘gaming’ of assessment systems. The review looks at how different funders are using quantitative indicators, and considers their potential role in research and innovation policy. Finally, it examines the role that metrics played in REF2014, and outlines scenarios for their contribution to future exercises

Research Performance Based Funding Systems: a Comparative Assessment

Performance based funding is used by most EU Member States to increase the performance of their public research system. This report analyses the different nature of systems in EU Member States, selected associated and third countries. It aims to inform Member States which are in a process of mutual learning to improve the design of their allocation systems. • Research Performance based funding systems provide incentives to increase scientific performance and concentrates resources in well performing organisations • The nature of systems in place differs widely • The specific features of RPBF assessment designs can generate unintended consequences • The assessment suggests RPBF as a potential avenue for several Member States • The choice for specific designs, taking into accounts costs and potential benefits, should take into account the national contextJRC.J.6-Innovation Systems Analysi

Agricultural research: a growing global divide?

"Sustained, well-targeted, and effectively used investments in R&D have reaped handsome rewards from improved agricultural productivity and cheaper, higher quality foods and fibers. As we begin a new millennium, the global patterns of investments in agricultural R&D are changing in ways that may have profound consequences for the structure of agriculture worldwide and the ability of poor people in poor counties to feed themselves. This report documents and discusses these changing investment patterns, highlighting developments in the public and private sectors. It revises and carries forward to 2000 data that were previously reported in the 2001 IFPRI Food Policy Report Slow Magic: Agricultural R&D a Century After Mendel. Some past trends are continuing or have come into sharper focus, while others are moving in new directions not apparent in the previous series. In addition, this report illustrates the use of spatial data to analyze spillover prospects among countries or agroecologies and the targeting of R&D to address specific production problems like drought-induced production risks." Authors' PrefaceResearch and development, Agricultural productivity, Investments, Agricultural research, Poverty, Public investment, Private sector, Spatial analysis (Statistics),

Avian Community Responses to Bison Grazing in North American Intermountain Grasslands

Grassland and shrubland songbird species are a guild of conservation concern in North America. Many of these species have experienced severe population declines, due to habitat loss and land use change. This makes the conservation and management of remaining habitat of crucial importance for this guild. Grazing by large herbivores is an ecosystem process in grassland systems, and in North America, one of the major historic grazers was the Plains bison (Bison bison). Bison are considered ecosystem engineers, because they modify habitat to be more or less suitable for other species, such as grassland and shrubland songbirds. Bison grazing can affect avian habitat by altering the vegetation structure. In turn, birds respond to these changes in vegetation structure. Bison are becoming more common due to recent reintroductions. From 2009-2017, at least eight bison reintroductions have occurred in North America. Lands where bison exist are good candidates for songbird conservation because bison are typically present in protected areas with a large grass and shrub component. Despite this potential, there is limited research about the effects of bison grazing on grassland and shrubland songbirds. Further research on this subject will inform bison management for songbird conservation. I investigate the relationship between bison grazing and songbird responses in two intermountain grasslands: the National Bison Range and Yellowstone National Park. In Chapter 1, I explore two ecological processes that may maintain species richness of grassland and shrubland songbirds: habitat heterogeneity from bison grazing, and productivity, a measure of the resources available to individuals. I analyzed the relationship between these variables and the occupancy of 10 avian species. I conclude that bison grazing has a stronger influence on bird occupancy and species richness than site productivity. In Chapter 2, I test whether differences between the study sites influence the abundance responses of vesper sparrow (Pooecetes gramineus) and western meadowlark (Sturnella neglecta) to bison grazing intensity. The differences, such as scale of bison grazing, herd size and density, and vegetation type show minimal influence on these species’ responses to bison grazing, but may be worth considering in conservation applications