How to identify peer-reviewed publications: Open-identity labels in scholarly book publishing

This work was supported by the Ministry of Science and Higher Education in Poland (https://www.gov.pl/nauka/) within the DIALOG Programme: the project title ‘Research into Excellence Patterns in Science and Art’. Tim Engels and Raf Guns thank the Flemish government for its funding of the Center for R&D Monitoring (ECOOM). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.This article discusses the open-identity label, i.e., the practice of disclosing reviewers’ names in published scholarly books, a common practice in Central and Eastern European countries. This study’s objective is to verify whether the open-identity label is a type of peer-review label (like those used in Finland and Flanders, i.e., the Flemish part of Belgium), and as such, whether it can be used as a delineation criterion in various systems used to evaluate scholarly publications. We have conducted a two-phase sequential explanatory study. In the first phase, interviews with 20 of the 40 largest Polish publishers of scholarly books were conducted to investigate how Polish publishers control peer reviews and whether the open-identity label can be used to identify peer-reviewed books. In the other phase, two questionnaires were used to analyze perceptions of peer-review and open-identity labelling among authors (n = 600) and reviewers (n = 875) of books published by these 20 publishers. Integrated results allowed us to verify publishers’ claims concerning their peer-review practices. Our findings reveal that publishers actually control peer reviews by providing assessment criteria to reviewers and sending reviews to authors. Publishers rarely ask for permission to disclose reviewers’ names, but it is obvious to reviewers that this practice of disclosing names is part of peer reviewing. This study also shows that only the names of reviewers who accepted manuscripts for publication are disclosed. Thus, most importantly, our analysis shows that the open-identity label that Polish publishers use is a type of peer-review label like those used in Flanders and Finland, and as such, it can be used to identify peer-reviewed scholarly books

Methods for the generation of normalized citation impact scores in bibliometrics: Which method best reflects the judgements of experts?

Evaluative bibliometrics compares the citation impact of researchers, research groups and institutions with each other across time scales and disciplines. Both factors - discipline and period - have an influence on the citation count which is independent of the quality of the publication. Normalizing the citation impact of papers for these two factors started in the mid-1980s. Since then, a range of different methods have been presented for producing normalized citation impact scores. The current study uses a data set of over 50,000 records to test which of the methods so far presented correlate better with the assessment of papers by peers. The peer assessments come from F1000Prime - a post-publication peer review system of the biomedical literature. Of the normalized indicators, the current study involves not only cited-side indicators, such as the mean normalized citation score, but also citing-side indicators. As the results show, the correlations of the indicators with the peer assessments all turn out to be very similar. Since F1000 focuses on biomedicine, it is important that the results of this study are validated by other studies based on datasets from other disciplines or (ideally) based on multi-disciplinary datasets.Comment: Accepted for publication in the Journal of Informetric

A review of the literature on citation impact indicators

Citation impact indicators nowadays play an important role in research evaluation, and consequently these indicators have received a lot of attention in the bibliometric and scientometric literature. This paper provides an in-depth review of the literature on citation impact indicators. First, an overview is given of the literature on bibliographic databases that can be used to calculate citation impact indicators (Web of Science, Scopus, and Google Scholar). Next, selected topics in the literature on citation impact indicators are reviewed in detail. The first topic is the selection of publications and citations to be included in the calculation of citation impact indicators. The second topic is the normalization of citation impact indicators, in particular normalization for field differences. Counting methods for dealing with co-authored publications are the third topic, and citation impact indicators for journals are the last topic. The paper concludes by offering some recommendations for future research

Unveiling the path towards sustainability: scientific interest at HEIs from a scientometric approach in the period 2008-2017

Mención Internacional en el título de doctorLa humanidad ha experimentado el impacto de un modelo económico insostenible a todos los niveles. Este tema se ha cristalizado en diferentes cumbres y conferencias durante el siglo XX. Como resultado de esta preocupación, surgió el concepto de Desarrollo Sostenible (DS). Sin embargo, este concepto ha recibido muchas críticas por ser altamente antropocéntrico y compartimentado, carente de coherencia conceptual o interconexión entre todos los aspectos involucrados. Más tarde, la aparición de los Objetivos de Desarrollo del Milenio (ODM) en 2000 y los recientes Objetivos de Desarrollo Sostenible (ODS) en 2015 constituyen una nueva era. Este es el plan para lograr un futuro mejor y más sostenible para todos, en el que todos los agentes involucrados deben participar. En este punto, las instituciones de educación superior (IES) tienen un papel central y la sostenibilidad se ha convertido en una prioridad política para la ciencia. El objetivo de este estudio es conocer los patrones de la investigación llevada a cabo en investigación de sostenibilidad, incluido el flujo de actividad científica, así como la colaboración o el impacto que genera dicha investigación. Este estudio de doctorado explora cómo se puede delinear este concepto desde un enfoque bibliométrico, lo cual conduce a la ‘ciencia de la sostenibilidad’. La producción científica de artículos fue identificada y analizada en el período 2008-2017 en la Web of Science (WoS). Además, este estudio explora las instituciones de educación superior (IES) y su papel en el fomento de la sostenibilidad, mediante la evaluación de su investigación y la implementación de prácticas de sostenibilidad en las IES españolas. Además, presenta una delineación de los Objetivos de Desarrollo Sostenible (ODS) y propone una metodología para clasificar la producción científica en cada uno e los objetivos. El análisis de esta producción se realiza a través de indicadores bibliométricos unidimensionales y multidimensionales. Estos indicadores se han dividido y analizado en diferentes niveles de agregación, desde el más general hasta el más específico, comenzando con las características generales de investigación y descendiendo al nivel de país, instituciones o temática, entre otros. Los resultados muestran un interés creciente en la investigación de sostenibilidad y se observa una fuerte influencia del pilar medioambiental. Además, hay países con una alta producción científica pero no tan especializados en el tema como otros con una menor producción. En cuanto a las instituciones, los resultados obtenidos muestran que las IES realizaron un importante esfuerzo de investigación para el desarrollo sostenible y son las que producen un mayor número de documentos. Además, se observa que las instituciones tienden a colaborar con centros geográficamente próximos. Al analizar las Prácticas de sostenibilidad en las IES españolas, se encuentran asociaciones altas entre variables como la presencia de un Plan de Sostenibilidad y de una Oficina Verde. Sin embargo, este estudio demuestra claramente que, aunque se reconoce que el desarrollo sostenible es muy importante para las IES y la sociedad, todavía no está integrado en las estrategias, actividades y políticas de todo el sistema. Como conclusión, se afirma que es esencial identificar estrategias de sostenibilidad e introducir desarrollo sostenible en todas las actividades en el entorno de las IES. Finalmente, esta tesis contribuye a la literatura sobre instituciones de educación superior sostenibles, así como al análisis y la mejora de educación superior para el desarrollo sostenible, especialmente en el sistema de educación superior español. Además, este estudio contribuye al análisis bibliométrico al ofrecer dos propuestas de delineación científica para la ciencia de la sostenibilidad y los objetivos de desarrollo sostenible, así como metodologías para clasificar la producción científica. Este análisis denota la importancia de los estudios bibliométricos para el estudio y la caracterización de la producción científica en un campo transdisciplinario que, además, se puede extrapolar a otros campos de estudio.Humanity has experienced the impact of an unsustainable economic model at all levels. This topic has crystallized in different summits and conferences during the 20th century. As a result of this concern, the concept of sustainable development (SD) emerged. However, it has received much criticism for being highly anthropocentric and compartmentalized, and lacking conceptual coherence or interconnectedness among all the aspects involved. The introduction of the Millennium Development Goals (MDGs) in 2000 and the recent Sustainable Development Goals (SDGs) in 2015 heralded a new era. They represent a blueprint to achieve a better and more sustainable future for all, in which all stakeholders need to be involved. At this point, higher education institutions (HEIs) have a central role to play and sustainability has emerged as a policy priority for science. The objective of this study is to investigate the patterns of sustainability research, including the flow of scientific activity, as well as the collaboration or impact that such research generates. This doctoral study explores how can sustainability can be delineated from a bibliometric approach, leading to a new approach of “sustainability science”. The scientific production of articles was identified and analysed for the period 2008–2017 using the Web of Science (WoS). Moreover, this research study explores HEIs and their role in fostering sustainability, by assessing their research and the implementation of sustainability practices in Spanish HEIs. As well, it presents a delineation of the Sustainable Development Goals (SDGs) and proposes a methodology for classifying the output on each SDG. This analysis is done through unidimensional and multidimensional bibliometric indicators. These indicators have been divided and analysed in different levels of aggregation, from the most general to the most specific, starting with general research features and progressing to country, institutional, and thematic levels, among others. The results indicate a growing interest in sustainability research and a strong influence on the environmental pillar. Moreover, some countries with the highest scientific output are not as specialized in terms of topics as others with a lower output. Regarding institutions, the results obtained indicate that HEIs made an important research contribution to SD and are the ones that produce a higher number of documents. It was found that institutions tend to collaborate with other institutions that are close. By analysing sustainability practices in Spanish HEIs, it was found that there are more associations between variables such as having a sustainability plan and having a green office. However, this study clearly demonstrates that although SD is recognized as being very important to HEIs and society, it is not yet embedded in the whole system’s strategies, activities, and policies. In conclusion, this research study reveals that it is essential to identify sustainability strategies and introduce SD in all activities in the HEI environment. Finally, this thesis contributes to the literature on sustainable HEIs, as well as to how higher education for SD is understood and can be improved, especially in the Spanish higher education system. Moreover, this contributes to bibliometric study by offering two delineation approach to sustainability science and sustainable development goals as well as methodologies for classifying scientific output. This denotes the importante of bibliometric studies for the study and characterization of scientific output in a transdisciplinary field that can be extrapolated to other fields of study.Programa de Doctorado en Documentación: Archivos y Bibliotecas en el Entorno Digital por la Universidad Carlos III de MadridPresidente: Carlos Balaguer Bernaldo de Quirós.- Secretario: Birger Larsen.- Vocal: Sandra Sofía Ferreira Da Silva C

The rate of growth in scientific publication and the decline in coverage provided by Science Citation Index

The growth rate of scientific publication has been studied from 1907 to 2007 using available data from a number of literature databases, including Science Citation Index (SCI) and Social Sciences Citation Index (SSCI). Traditional scientific publishing, that is publication in peer-reviewed journals, is still increasing although there are big differences between fields. There are no indications that the growth rate has decreased in the last 50 years. At the same time publication using new channels, for example conference proceedings, open archives and home pages, is growing fast. The growth rate for SCI up to 2007 is smaller than for comparable databases. This means that SCI was covering a decreasing part of the traditional scientific literature. There are also clear indications that the coverage by SCI is especially low in some of the scientific areas with the highest growth rate, including computer science and engineering sciences. The role of conference proceedings, open access archives and publications published on the net is increasing, especially in scientific fields with high growth rates, but this has only partially been reflected in the databases. The new publication channels challenge the use of the big databases in measurements of scientific productivity or output and of the growth rate of science. Because of the declining coverage and this challenge it is problematic that SCI has been used and is used as the dominant source for science indicators based on publication and citation numbers. The limited data available for social sciences show that the growth rate in SSCI was remarkably low and indicate that the coverage by SSCI was declining over time. National Science Indicators from Thomson Reuters is based solely on SCI, SSCI and Arts and Humanities Citation Index (AHCI). Therefore the declining coverage of the citation databases problematizes the use of this source

Study on open science: The general state of the play in Open Science principles and practices at European life sciences institutes

Nowadays, open science is a hot topic on all levels and also is one of the priorities of the European Research Area. Components that are commonly associated with open science are open access, open data, open methodology, open source, open peer review, open science policies and citizen science. Open science may a great potential to connect and influence the practices of researchers, funding institutions and the public. In this paper, we evaluate the level of openness based on public surveys at four European life sciences institute

Congress UPV Proceedings of the 21ST International Conference on Science and Technology Indicators

This is the book of proceedings of the 21st Science and Technology Indicators Conference that took place in València (Spain) from 14th to 16th of September 2016. The conference theme for this year, ‘Peripheries, frontiers and beyond’ aimed to study the development and use of Science, Technology and Innovation indicators in spaces that have not been the focus of current indicator development, for example, in the Global South, or the Social Sciences and Humanities. The exploration to the margins and beyond proposed by the theme has brought to the STI Conference an interesting array of new contributors from a variety of fields and geographies. This year’s conference had a record 382 registered participants from 40 different countries, including 23 European, 9 American, 4 Asia-Pacific, 4 Africa and Near East. About 26% of participants came from outside of Europe. There were also many participants (17%) from organisations outside academia including governments (8%), businesses (5%), foundations (2%) and international organisations (2%). This is particularly important in a field that is practice-oriented. The chapters of the proceedings attest to the breadth of issues discussed. Infrastructure, benchmarking and use of innovation indicators, societal impact and mission oriented-research, mobility and careers, social sciences and the humanities, participation and culture, gender, and altmetrics, among others. We hope that the diversity of this Conference has fostered productive dialogues and synergistic ideas and made a contribution, small as it may be, to the development and use of indicators that, being more inclusive, will foster a more inclusive and fair world

A framework for monitoring scientific production behavior in Research evaluation systems based on journal ranking lists the brazilian case

A extensão e a qualidade dos resultados de pesquisa tornaram-se fatores-chave para a avaliação do desempenho da universidade. Vários países introduziram sistemas de avaliação de pesquisas que vinculam financiamento a indicadores de desempenho como forma de aumentar a prestação de contas. Em geral, classificações de periódicos são parte integrante desses sistemas. Esta tese aborda o desenvolvimento e a avaliação de um framework para monitorar o comportamento da produção científica em contextos em que as listas de classificação de periódicos estão no centro da avaliação. O principal objetivo do framework é permitir a identificação de padrões desejáveis e adversos na produção acadêmica. Considerando que o Brasil utiliza um sistema de classificação de periódicos (QUALIS) há mais de duas décadas, o framework foi aplicado numa análise de dez anos da produção científica brasileira em oito áreas distintas, tomando o banco de dados da Scopus como referência. Os resultados mostraram um declínio na proporção de artigos indexados na Scopus nas áreas de Ciências Sociais e Humanas (SSH). Um número restrito de periódicos, que permaneceu no sistema durante o período estudado, concentrou um número maior de artigos. No geral, esses periódicos tiveram sua classificação QUALIS inalterada ou melhorada ao longo das avaliações periódicas. Entretanto, na maioria, houve uma diminuição significativa no impacto de citação desses. Além disso, aqueles de menor impacto passaram para as categorias QUALIS mais altas ao longo dos anos, o que ocorreu simultaneamente com um aumento no número de artigos em periódicos de baixo impacto em todos as áreas. Esses resultados mostraram que o uso de listas de classificação de periódicos pode levar professores e alunos a publicarem em periódicos de alta classificação, apesar de terem um baixo impacto de citação. Quando os periódicos de baixo impacto alcançam uma classificação alta, eles também podem concentrar uma grande quantidade de artigos publicados. De certa forma, esses padrões são semelhantes a outros resultados encontrados na literatura, nos quais um aumento significativo de publicações foi seguido por um declínio no impacto. O efeito potencial desses modelos de avaliação é que eles podem incitar as pessoas a selecionar meios de publicação com uma pontuação mais alta de acordo com os critérios estabelecidos, independentemente de sua visibilidade. Além disso, esses efeitos podem intensificar-se quando os resultados da avaliação são vinculados a financiamento.The extent and quality of research output have become key factors for university performance evaluation. Several countries introduced research evaluation systems that link funding to performance indicators as a way to enhance accountability. In general, journal rankings are an integral part of these systems. This thesis approaches the development and evaluation of a framework for monitoring scientific production behavior in settings where journal-ranking lists are at the center of research assessment. The main goal of the framework is to enable the identification of desirable and adverse patterns in academic production. Considering that Brazil has been using a specific journal ranking system (QUALIS) for more than two decades, the framework was applied in the ten-year analysis of Brazilian scientific production in eight distinct subject fields and taking the Scopus database as a reference. Results showed a decline in the proportion of Scopus-indexed articles in the areas of Social Sciences and Humanities (SSH). A few journals that remained in the system during the whole evaluation period concentrated a larger number of published articles. Overall, these journals had their QUALIS classification unchanged or improved in the ranking over the periodic evaluations. However, in general, there was a significant decrease in their citation impact. Moreover, lower-impact journals moved to the highest QUALIS categories over the years, what happened simultaneously with an increase in the number of articles in low-impact journals in all fields. These results have shown that the use of journal ranking lists may lead faculty and students to submit their papers to highly ranked journals, even though may have a low citation impact. When low-impact journals reach a high rank, they may also concentrate a high amount of published articles. In a certain way, these patterns are similar to other results found in literature, in which a significant increase in publication productivity has been followed by an impact decline. The potential effect of these evaluation models is that they may incite people to select publication venues that make them score higher according to the established criteria, regardless of their publications’ visibility. Besides, this effect can be intensified once the evaluation results are linked to funds