A framework for monitoring scientific production behavior in Research evaluation systems based on journal ranking lists the brazilian case

A extensão e a qualidade dos resultados de pesquisa tornaram-se fatores-chave para a avaliação do desempenho da universidade. Vários países introduziram sistemas de avaliação de pesquisas que vinculam financiamento a indicadores de desempenho como forma de aumentar a prestação de contas. Em geral, classificações de periódicos são parte integrante desses sistemas. Esta tese aborda o desenvolvimento e a avaliação de um framework para monitorar o comportamento da produção científica em contextos em que as listas de classificação de periódicos estão no centro da avaliação. O principal objetivo do framework é permitir a identificação de padrões desejáveis e adversos na produção acadêmica. Considerando que o Brasil utiliza um sistema de classificação de periódicos (QUALIS) há mais de duas décadas, o framework foi aplicado numa análise de dez anos da produção científica brasileira em oito áreas distintas, tomando o banco de dados da Scopus como referência. Os resultados mostraram um declínio na proporção de artigos indexados na Scopus nas áreas de Ciências Sociais e Humanas (SSH). Um número restrito de periódicos, que permaneceu no sistema durante o período estudado, concentrou um número maior de artigos. No geral, esses periódicos tiveram sua classificação QUALIS inalterada ou melhorada ao longo das avaliações periódicas. Entretanto, na maioria, houve uma diminuição significativa no impacto de citação desses. Além disso, aqueles de menor impacto passaram para as categorias QUALIS mais altas ao longo dos anos, o que ocorreu simultaneamente com um aumento no número de artigos em periódicos de baixo impacto em todos as áreas. Esses resultados mostraram que o uso de listas de classificação de periódicos pode levar professores e alunos a publicarem em periódicos de alta classificação, apesar de terem um baixo impacto de citação. Quando os periódicos de baixo impacto alcançam uma classificação alta, eles também podem concentrar uma grande quantidade de artigos publicados. De certa forma, esses padrões são semelhantes a outros resultados encontrados na literatura, nos quais um aumento significativo de publicações foi seguido por um declínio no impacto. O efeito potencial desses modelos de avaliação é que eles podem incitar as pessoas a selecionar meios de publicação com uma pontuação mais alta de acordo com os critérios estabelecidos, independentemente de sua visibilidade. Além disso, esses efeitos podem intensificar-se quando os resultados da avaliação são vinculados a financiamento.The extent and quality of research output have become key factors for university performance evaluation. Several countries introduced research evaluation systems that link funding to performance indicators as a way to enhance accountability. In general, journal rankings are an integral part of these systems. This thesis approaches the development and evaluation of a framework for monitoring scientific production behavior in settings where journal-ranking lists are at the center of research assessment. The main goal of the framework is to enable the identification of desirable and adverse patterns in academic production. Considering that Brazil has been using a specific journal ranking system (QUALIS) for more than two decades, the framework was applied in the ten-year analysis of Brazilian scientific production in eight distinct subject fields and taking the Scopus database as a reference. Results showed a decline in the proportion of Scopus-indexed articles in the areas of Social Sciences and Humanities (SSH). A few journals that remained in the system during the whole evaluation period concentrated a larger number of published articles. Overall, these journals had their QUALIS classification unchanged or improved in the ranking over the periodic evaluations. However, in general, there was a significant decrease in their citation impact. Moreover, lower-impact journals moved to the highest QUALIS categories over the years, what happened simultaneously with an increase in the number of articles in low-impact journals in all fields. These results have shown that the use of journal ranking lists may lead faculty and students to submit their papers to highly ranked journals, even though may have a low citation impact. When low-impact journals reach a high rank, they may also concentrate a high amount of published articles. In a certain way, these patterns are similar to other results found in literature, in which a significant increase in publication productivity has been followed by an impact decline. The potential effect of these evaluation models is that they may incite people to select publication venues that make them score higher according to the established criteria, regardless of their publications’ visibility. Besides, this effect can be intensified once the evaluation results are linked to funds

Publication practices in the Humanities: An in-depth case study of a Swedish Arts and Humanities Faculty 2010-2018

This paper is a case study of research publication practices at the Faculty of Arts and Humanities at Linnaeus University, a young, mid-sized university in the south-east of Sweden. Research output was measured from publications in the local institutional repository following the guidelines of local research policy as defined in university documentation. The data collection comprised 3,316 metadata records of publications self-registered by authors affiliated with the faculty during the period of 2010–2018. A statistical analysis of research output was conducted, focusing on preferred publication types, disciplinary specificity, level of co-authorship, and the language of the publication as registered in the local repository. The analysis focused on two main research questions: 1) how do the local research practices stand in relation to traditional publication patterns in the humanities? 2) how do the observed publication patterns relate to local university policy on publication and research evaluation? The empirical results suggest a limited correlation between publication practices and research incentives from university management, a finding that is corroborated by previous research on the scholarly character of the humanities and university policies. Overall, traditional humanities publication patterns were largely maintained throughout the period under investigation

Impact and Visibility of Norwegian, Finnish and Spanish Journals in the fields of humanities

This article analyses the impact and visibility of scholarly journals in the humanities that are publishing in the national languages in Finland, Norway and Spain. Three types of publishers are considered: commercial publishers, scholarly society as publisher, and research organizations as publishers. Indicators of visibility and impact were obtained from Web of Science, SCOPUS, Google Metrics, Scimago Journal Rank and Journal Citation Report. The findings compiled show that in Spain the categories “History and Archaeology” and “Language and Literature” account for almost 70% of the journals analysed, while the other countries offer a more homogeneous distribution. In Finland, the scholarly society publisher is predominant, in Spain, research organization as publishers, mostly universities, have a greater weighting, while in Norway, the commercial publishers take centre stage. The results show that journals from Finland and Norway will have reduced possibilities in terms of impact and visibility, since the vernacular language appeals to a smaller readership. Conversely, the Spanish journals are more attractive for indexing in commercial databases. Distribution in open access ranges from 64 to 70% in Norwegian and Finish journals, and to 91% in Spanish journals. The existence of DOI range from 31 to 41% in Nordic journals to 60% in Spanish journals and has a more widespread bearing on the citations received in all three countries (journals with DOI and open access are cited more frequently)

Past performance does not guarantee future results: lessons from the evaluation of research units in Portugal

Research units in Portugal undergo a formal evaluation process based on peer review which is the basis for distributing funding from the national research council. This article analyzes the evaluation results and asks how good they are at predicting future research performance. Better research evaluations mean the institution receives more funding, so the key question is to what extent research evaluations are able to predict future performance as measured by bibliometric indicators. We use data from the peer evaluation of units in 2007–08, and analyze how well it is able to predict the results of a bibliometric study of the units’ Web of Science publications in the period 2007–10. We found that, in general, units that had better peer ratings, and thus more funding, as well as an increased capacity to attract extra funding, were not necessarily those that ended up producing more excellent research. The results provide an empirical contribution to the discussion regarding whether science can be measured and how, and reinforce the importance of evaluations where the use of quantitative data is defined and the differences between areas are accounted for. This analysis provides a snapshot of Portugal's recent scientific performance. Chemistry and physics are among the subfields with higher output and impact, which agrees with a traditional preferential funding of these areas. Institutions also excel in areas that may be assuming an increased relevance (Plant Sciences, Food Science and Technology, Neurosciences and other health-related subfields), which should be taken into account when implementing future science policies.info:eu-repo/semantics/publishedVersio

Research Performance Based Funding Systems: a Comparative Assessment

Performance based funding is used by most EU Member States to increase the performance of their public research system. This report analyses the different nature of systems in EU Member States, selected associated and third countries. It aims to inform Member States which are in a process of mutual learning to improve the design of their allocation systems. • Research Performance based funding systems provide incentives to increase scientific performance and concentrates resources in well performing organisations • The nature of systems in place differs widely • The specific features of RPBF assessment designs can generate unintended consequences • The assessment suggests RPBF as a potential avenue for several Member States • The choice for specific designs, taking into accounts costs and potential benefits, should take into account the national contextJRC.J.6-Innovation Systems Analysi

The effect of academic mobility on research performance: The case of Italy

Abstract This work investigates the effects of researchers' mobility on their research performance. The reference context is that of national intrasector mobility, in a country, Italy, characterized by a research system lacking the typical elements of an academic labor market. In particular, the analysis was conducted on 568 academics working at national universities and affected by mobility in the period 2009–2014. The effect of mobility on the variation of performance at the turn of the transfer was analyzed considering the interplay of demographic/sociological characteristics of the researchers, as well as contextual factors related to both the organization of origin and destination. Results show that it is the less productive academics who represent the larger share of those who move, and more than half of the mobile academics worsen their performance after the transfer