Reviewing, indicating, and counting books for modern research evaluation systems

In this chapter, we focus on the specialists who have helped to improve the conditions for book assessments in research evaluation exercises, with empirically based data and insights supporting their greater integration. Our review highlights the research carried out by four types of expert communities, referred to as the monitors, the subject classifiers, the indexers and the indicator constructionists. Many challenges lie ahead for scholars affiliated with these communities, particularly the latter three. By acknowledging their unique, yet interrelated roles, we show where the greatest potential is for both quantitative and qualitative indicator advancements in book-inclusive evaluation systems.Comment: Forthcoming in Glanzel, W., Moed, H.F., Schmoch U., Thelwall, M. (2018). Springer Handbook of Science and Technology Indicators. Springer Some corrections made in subsection 'Publisher prestige or quality

Social media metrics for new research evaluation

This chapter approaches, both from a theoretical and practical perspective, the most important principles and conceptual frameworks that can be considered in the application of social media metrics for scientific evaluation. We propose conceptually valid uses for social media metrics in research evaluation. The chapter discusses frameworks and uses of these metrics as well as principles and recommendations for the consideration and application of current (and potentially new) metrics in research evaluation.Comment: Forthcoming in Glanzel, W., Moed, H.F., Schmoch U., Thelwall, M. (2018). Springer Handbook of Science and Technology Indicators. Springe

Disentangling Gold Open Access

This chapter focuses on the analysis of current publication trends in gold Open Access (OA). The purpose of the chapter is to develop a full understanding on country patterns, OA journals characteristics and citation differences between gold OA and non-gold OA publications. For this, we will first review current literature regarding Open Access and its relation with its so-called citation advantage. Starting with a chronological perspective we will describe its development, how different countries are promoting OA publishing, and its effects on the journal publishing industry. We will deepen the analysis by investigating the research output produced by different units of analysis. First, we will focus on the production of countries with a special emphasis on citation and disciplinary differences. A point of interest will be identification of national idiosyncrasies and the relation between OA publication and research of local interest. This will lead to our second unit of analysis, OA journals indexed in Web of Science. Here we will deepen on journals characteristics and publisher types to clearly identify factors which may affect citation differences between OA and traditional journals which may not necessarily be derived from the OA factor. Gold OA publishing is being encouraged in many countries as opposed to Green OA. This chapter aims at fully understanding how it affects researchers’ publication patterns and whether it ensures an alleged citation advantage as opposed to non-gold OA publications.Esta publicación ha sido financiado por el Vicerrectorado de Investigación y Transferencia de la Universidad de Granada en el Marco del Plan Propio de Investigación y su programa Visiting Scholar

Exploring the Potentialities of Automatic Extraction of University Webometric Information

The main objective of this work is to show the potentialities of recently developed approaches for automatic knowledge extraction directly from the universities’ websites. The information automatically extracted can be potentially updated with a frequency higher than once per year, and be safe from manipulations or misinterpretations. Moreover, this approach allows us flexibility in collecting indicators about the efficiency of universities’ websites and their effectiveness in disseminating key contents. These new indicators can complement traditional indicators of scientific research (e.g. number of articles and number of citations) and teaching (e.g. number of students and graduates) by introducing further dimensions to allow new insights for “profiling” the analyzed universities. The main findings of this study concern the evaluation of the potential in digitalization of universities, in particular by presenting techniques for the automatic extraction of information from the web to build indicators of quality and impact of universities’ websites. These indicators can complement traditional indicators and can be used to identify groups of universities with common features using clustering techniques working with the above indicators

Altmetrics as an Answer to the Need for Democratization of Research and Its Evaluation

In the evaluation of research, the same unequal structure present in the production of research is reproduced. Despite a few very productive researchers (in terms of papers and citations received), there are also few researchers who are involved in the research evaluation process (in terms of being editorial board members of journals or reviewers). To produce a high number of papers and receive many citations and to be involved in the evaluation of research papers, you need to be in the minority of giants who have high productivity and more scientific success. As editorial board members and reviewers, we often find the same minority of giants. In this paper, we apply an economic approach to interpret recent trends in research evaluation and derive a new interpretation of Altmetrics as a response to the need for democratization of research and its evaluation. In this context, the majority of pygmies can participate in evaluation with Altmetrics, whose use is more democratic, that is, much wider and open to all

Same-gender citations do not indicate a substantial gender homophily bias

Can the male citation advantage (more citations for papers written by male than female scientists) be explained by gender homophily bias, i.e., the preference of scientists to cite other scientists of the same gender category? Previous studies report much evidence that this is the case. However, the observed gender homophily bias may be overestimated by overlooking structural aspects such as the gender composition of research topics in which scientists specialize. When controlling for research topics at a high level of granularity, there is only little evidence for a gender homophily bias in citation decisions. Our study points out the importance of controlling structural aspects such as gendered specialization in research topics when investigating gender bias in science

Scrambling for higher metrics in the Journal Impact Factor bubble period: a real-world problem in science management and its implications

Universities and funders in many countries have been using Journal Impact Factor (JIF) as an indicator for research and grant assessment despite its controversial nature as a statistical representation of scientific quality. This study investigates how the changes of JIF over the years can affect its role in research evaluation and science management by using JIF data from annual Journal Citation Reports (JCR) to illustrate the changes. The descriptive statistics find out an increase in the median JIF for the top 50 journals in the JCR, from 29.300 in 2017 to 33.162 in 2019. Moreover, on average, elite journal families have up to 27 journals in the top 50. In the group of journals with a JIF of lower than 1, the proportion has shrunk by 14.53% in the 2015–2019 period. The findings suggest a potential ‘JIF bubble period’ that science policymaker, university, public fund managers, and other stakeholders should pay more attention to JIF as a criterion for quality assessment to ensure more efficient science management

Identifying emerging topics in the peer‑reviewed literature to facilitate curriculum renewal and development

This article reports a bibliometric analysis of emerging topics in the psychiatric literature indexed in the MEDLINE database as a technique for renewal of clinical training curricula. Summary data of English-language articles indexed in the MEDLINE database between 1971-2018 were downloaded. Emerging topics in nine demi-decades between 1972-1976 and 2012-2016 were identified by the incremental incidence of individual Medical Subject Headings (MeSH) compared with previous years. Co-word analysis was used to investigate and visualise the relationships between emerging topics in each demi-decade. Summaries of 18 million articles annotated with psychiatric/psychological MeSH were retrieved and used to identify emerging topics. Peaks in the number of articles annotated by the top 20 emerging topics in 9 demi-decades coincided with release of the third and fourth editions of the Diagnostic and Statistical Manual which codifies psychiatric diagnoses. Themes emerging from network visualisations of the most common emerging MeSH in each demi-decade were consistent with movements in psychiatric/psychological theory and practice since the 1970s, including the recent focus on psychological and social factors implicated in suicide and suicide prevention. The identification of emerging topics within the published medical literature is a viable technique for use in curriculum renewal projects as a counterweight to biases driven by expert judgement. While indices like MEDLINE make the published literature an appealing initial step in building an empirical basis for curriculum development, it also demonstrates the potential value of less public and less structured data, such as health service electronic medical record