Peer assessment or promotion by numbers? A comparative study of different measures of researcher performance within the UK Library and Information Science research community

Hirsch’s h-index, Egghe’s g-index, total citation and publication counts, and five proposed new metrics were correlated with one another using Spearman’s Rank Correlation for one hundred randomly selected academics and researchers working in UK Library and Information Science departments. Metrics were compared for individuals of different genders and at institutions awarded different RAE (2001) grades. Individuals’ metrics were rank-correlated against academic ranks and RAE (2001) grades of their employing departments. Metrics calculated using Web of Science and Google Scholar data were compared. Peer- and h-index metric-ranked orders of researchers were rank-correlated. Citation behaviour and attitudes towards peer and citation-based assessment of 263 academics and researchers were investigated by factor analysis of online attitudinal survey responses. h increased curvilinearly with total citation and publication counts, suggesting that h was constrained by the activity in the field preventing individuals producing enough heavily cited publications to increase their h-index scores. Most individuals therefore shared similar h-index scores, making interpersonal comparisons difficult. Total citation counts and Bihui’s a-index scores distinguished between more individuals, though whether they could confidently identify differences between individuals is uncertain. Both databases arbitrarily omitted individuals and publications, systematically biasing citation metrics calculated using them. In contrast to studies of larger fields, no citation metrics correlated with RAE grade, academic rank, or direct peer-assessment, suggesting that citation-based assessment is unsuitable for research fields with relatively little research activity. No gender bias was evident in academic rank, esteem or citedness. At least nine independent factors influence citation behaviour. Mertonian factors dominated. The independence of the factors suggested different individuals have different combinations of non-Mertonian motivations. The overriding meaning of citations was confirmed as signals of relevance and reward. Recommendations for future research include a need to develop simple, robust methods to identify subfields and normalise citations across subfields, to quantify the impact of random bias and to determine whether it varies across subfields, and to study the rate of accumulation of citations and citation distribution changes for individuals (and departments) over time to determine whether career age can be controlled for, in particular

A bibliometric view on the internationalization of European educational research

Is there a trend towards internationalization of educational research in Europe? Educational research is said to follow a tradition of nationally oriented studies and interventions supported by a national publication culture. Publications are a suitable source of empirical analysis of research output, as they reflect results, emergence and impact of research. This study focuses on publication based bibliometric indicators, which represent measurable characteristics of international orientation of research publications and which can be surveyed in time course. Being aware that the Web of Science (WoS) databases cover a crucial but rather limited proportion of the worldwide educational research output, this study provides bibliometric insights into the development of national publication outputs in educational research in the WoS and what idiosyncrasies are revealed for European countries, into the role of English as a publication language, into the trend towards transnational co-authorship as an indicator of international cooperation, and into citation frequencies as a measurement of research communication or research impact. (DIPF/Orig.

The effect of the rapid growth of covid-19 publications on citation indicators

A concern has been raised that "covidization" of research would cause an overemphasizes on covid-19 and pandemics at the expense of other research. The rapid growth of publications related to the Covid-19 pandemic renders a vast amount of citations from this literature. This growth may affect bibliometric indicators. In this paper I explored how the growth of covid-19 publications influences bibliometric indicators commonly used in university rankings, research evaluation and research allocation, namely the field normalized citation score and the journal impact factor. I found that the burst of publications in the early stage of the covid-19 pandemic affects field-normalized citation scores and will affect the journal impact factor. Publications unrelated to covid-19 are also heavily affected. I conclude that there is a considerable risk to draw misleading conclusions from citation indicators spanning over the beginning of the covid-19 pandemic, in particular when time series are used and when the biomedical literature is assessed

Interdisciplinarity in Translation Medicine: A Bibliometric Case Study.

Ph.D. Thesis. University of Hawaiʻi at Mānoa 2017

What is a complex innovation system?

Innovation systems are sometimes referred to as complex systems, something that is intuitively understood but poorly defined. A complex system dynamically evolves in non-linear ways giving it unique properties that distinguish it from other systems. In particular, a common signature of complex systems is scale-invariant emergent properties. A scale-invariant property can be identified because it is solely described by a power law function, f(x) = kxα, where the exponent, α, is a measure of scale-invariance. The focus of this paper is to describe and illustrate that innovation systems have properties of a complex adaptive system. In particular scale-invariant emergent properties indicative of their complex nature that can be quantified and used to inform public policy. The global research system is an example of an innovation system. Peer-reviewed publications containing knowledge are a characteristic output. Citations or references to these articles are an indirect measure of the impact the knowledge has on the research community. Peer-reviewed papers indexed in Scopus and in the Web of Science were used as data sources to produce measures of sizes and impact. These measures are used to illustrate how scale-invariant properties can be identified and quantified. It is demonstrated that the distribution of impact has a reasonable likelihood of being scale-invariant with scaling exponents that tended toward a value of less than 3.0 with the passage of time and decreasing group sizes. Scale-invariant correlations are shown between the evolution of impact and size with time and between field impact and sizes at points in time. The recursive or self-similar nature of scale-invariance suggests that any smaller innovation system within the global research system is likely to be complex with scale-invariant properties too

Citation models and research evaluation

Citations in science are being studied from several perspectives. On the one hand, there are approaches such as scientometrics and the science of science, which take a more quantitative perspective. In this chapter I briefly review some of the literature on citations, citation distributions and models of citations. These citations feature prominently in another part of the literature which is dealing with research evaluation and the role of metrics and indicators in that process. Here I briefly review part of the discussion in research evaluation. This also touches on the subject of how citations relate to peer review. Finally, I try to integrate the two literatures with the aim of clarifying what I believe the two can learn from each other. The fundamental problem in research evaluation is that research quality is unobservable. This has consequences for conclusions that we can draw from quantitative studies of citations and citation models. The term "indicators" is a relevant concept in this context, which I try to clarify. Causality is important for properly understanding indicators, especially when indicators are used in practice: when we act on indicators, we enter causal territory. Even when an indicator might have been valid, through its very use, the consequences of its use may invalidate it. By combining citation models with proper causal reasoning and acknowledging the fundamental problem about unobservable research quality, we may hope to make progress.Comment: This is a draft. The final version will be available in Handbook of Computational Social Science edited by Taha Yasseri, forthcoming 2023, Edward Elgar Publishing Lt

The Impact of Research Data Sharing and Reuse on Data Citation in STEM Fields

Despite the open science movement and mandates for the sharing of research data by major funding agencies and influential journals, the citation of data sharing and reuse has not become standard practice in the various science, technology, engineering and mathematics (STEM) fields. Advances in technology have lowered some barriers to data sharing, but it is a socio-technical phenomenon and the impact of the ongoing evolution in scholarly communication practices has yet to be quantified. Furthermore, there is need for a deeper and more nuanced understanding of author self-citation and recitation, the most often cited types of data, disciplinary differences regarding data citation and the extent of interdisciplinarity in data citation. This study employed a mixed methods approach that combined coding with semi-automatic text-searching techniques in order to assess the impact of data sharing and reuse on data citation in STEM fields. The research considered over 500,000 open research data entities, such as datasets, software and data studies, from over 350 repositories worldwide. I also examined 705 bibliographic publications with a total of 15,261 instances of data sharing, reuse, and citation the data, article, discipline and interdisciplinary levels. More specifically, I measured the phenomenon of data sharing in terms of formal data citation, frequently cited data types, and author self-citation, and I explored recitation at the levels of both data- and bibliography-level, and data reuse practices in bibliographies, associations of disciplines, and interdisciplinary contexts. The results of this research revealed, to begin with, disciplinary differences with regard to the impact of data sharing and reuse on data citation in STEM fields. This research also yielded the following additional findings regarding the citation of data by STEM researchers; 1) data sharing practices were diverse across disciplines: 2) data sharing has been increasing in recent years; 3) each discipline made use of major digital repositories; 4) these repositories took various forms depending on the discipline; 5) certain data types were more often cited in each discipline, so that the frequency distribution of the data types was highly skewed; 6) author self-citation and recitation followed similar trends at the data and bibliographic levels, but specific practices varied within each discipline; 7) associations between and across data and author self-citation and recitation at the bibliographic level were observed, with the self-citation rate differing significantly among disciplines;8) data reuse in bibliographies was rare yet diverse; 9) informal citation of data sharing and reuse at the bibliographic level was more common in certain fields, with astronomy/physics showing the highest amount (98%) and technology the lowest (69%); 10) within bibliographic publications, the documentation of data sharing and reuse occurred mainly in the main text; 11) publications in certain disciplines, such as chemistry, computing and engineering, did not attract citations from more than one field (i.e., showed no diversity); and, on the other hand,12) publications in other fields attracted a wide range of interdisciplinary data citations. This dissertation, then, contributes to the understanding of two key areas aspects of the current citation systems. First, the findings have practical implications for individual researchers, decision makers, funding agencies and publishers with regard to giving due credits to those who share their data. Second, this research has methodological implications in terms of reducing the labor required to analyze the full text of associated articles in order to identify evidence of data citation

The Usefulness of Peer Review for Selecting Manuscripts for Publication: A Utility Analysis Taking as an Example a High-Impact Journal

Background: High predictive validity – that is, a strong association between the outcome of peer review (usually, reviewers’ ratings) and the scientific quality of a manuscript submitted to a journal (measured as citations of the later published paper) – does not as a rule suffice to demonstrate the usefulness of peer review for the selection of manuscripts. To assess usefulness, it is important to include in addition the base rate (proportion of submissions that are fundamentally suitable for publication) and the selection rate (the proportion of submissions accepted). Methodology/Principal Findings: Taking the example of the high-impact journal Angewandte Chemie International Edition (AC-IE), we present a general approach for determining the usefulness of peer reviews for the selection of manuscripts for publication. The results of our study show that peer review is useful: 78 % of the submissions accepted by AC-IE are correctly accepted for publication when the editor’s decision is based on one review, 69 % of the submissions are correctly accepted for publication when the editor’s decision is based on two reviews, and 65 % of the submissions are correctly accepted for publication when the editor’s decision is based on three reviews. Conclusions/Significance: The paper points out through what changes in the selection rate, base rate or validity coefficient a higher success rate (utility) in the AC-IE selection process could be achieved