492 research outputs found
Inter-rater reliability and convergent validity of F1000Prime peer review
Peer review is the backbone of modern science. F1000Prime is a
post-publication peer review system of the biomedical literature (papers from
medical and biological journals). This study is concerned with the inter-rater
reliability and convergent validity of the peer recommendations formulated in
the F1000Prime peer review system. The study is based on around 100,000 papers
with recommendations from Faculty members. Even if intersubjectivity plays a
fundamental role in science, the analyses of the reliability of the F1000Prime
peer review system show a rather low level of agreement between Faculty
members. This result is in agreement with most other studies which have been
published on the journal peer review system. Logistic regression models are
used to investigate the convergent validity of the F1000Prime peer review
system. As the results show, the proportion of highly cited papers among those
selected by the Faculty members is significantly higher than expected. In
addition, better recommendation scores are also connected with better
performance of the papers.Comment: Accepted for publication in the Journal of the Association for
Information Science and Technolog
How to analyse percentile impact data meaningfully in bibliometrics: The statistical analysis of distributions, percentile rank classes and top-cited papers
According to current research in bibliometrics, percentiles (or percentile
rank classes) are the most suitable method for normalising the citation counts
of individual publications in terms of the subject area, the document type and
the publication year. Up to now, bibliometric research has concerned itself
primarily with the calculation of percentiles. This study suggests how
percentiles can be analysed meaningfully for an evaluation study. Publication
sets from four universities are compared with each other to provide sample
data. These suggestions take into account on the one hand the distribution of
percentiles over the publications in the sets (here: universities) and on the
other hand concentrate on the range of publications with the highest citation
impact - that is, the range which is usually of most interest in the evaluation
of scientific performance
Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics
Today, it is not clear how the impact of research on other areas of society
than science should be measured. While peer review and bibliometrics have
become standard methods for measuring the impact of research in science, there
is not yet an accepted framework within which to measure societal impact.
Alternative metrics (called altmetrics to distinguish them from bibliometrics)
are considered an interesting option for assessing the societal impact of
research, as they offer new ways to measure (public) engagement with research
output. Altmetrics is a term to describe web-based metrics for the impact of
publications and other scholarly material by using data from social media
platforms (e.g. Twitter or Mendeley). This overview of studies explores the
potential of altmetrics for measuring societal impact. It deals with the
definition and classification of altmetrics. Furthermore, their benefits and
disadvantages for measuring impact are discussed.Comment: Accepted for publication in the Journal of Informetric
Is there currently a scientific revolution in scientometrics?
The author of this letter to the editor would like to set forth the argument
that scientometrics is currently in a phase in which a taxonomic change, and
hence a revolution, is taking place. One of the key terms in scientometrics is
scientific impact which nowadays is understood to mean not only the impact on
science but the impact on every area of society.Comment: Accepted for publication in the Journal of the American Society for
Information Science and Technolog
Which cities' paper output and citation impact are above expectation in information science? Some improvements of our previous mapping approaches
Bornmann and Leydesdorff (in press) proposed methods based on Web-of-Science
data to identify field-specific excellence in cities where highly-cited papers
were published more frequently than can be expected. Top performers in output
are cities in which authors are located who publish a number of highly-cited
papers that is statistically significantly higher than can be expected for
these cities. Using papers published between 1989 and 2009 in information
science improvements to the methods of Bornmann and Leydesdorff (in press) are
presented and an alternative mapping approach based on the indicator I3 is
introduced here. The I3 indicator was introduced by Leydesdorff and Bornmann
(in press)
How to evaluate individual researchers working in the natural and life sciences meaningfully? A proposal of methods based on percentiles of citations
Although bibliometrics has been a separate research field for many years,
there is still no uniformity in the way bibliometric analyses are applied to
individual researchers. Therefore, this study aims to set up proposals how to
evaluate individual researchers working in the natural and life sciences. 2005
saw the introduction of the h index, which gives information about a
researcher's productivity and the impact of his or her publications in a single
number (h is the number of publications with at least h citations); however, it
is not possible to cover the multidimensional complexity of research
performance and to undertake inter-personal comparisons with this number. This
study therefore includes recommendations for a set of indicators to be used for
evaluating researchers. Our proposals relate to the selection of data on which
an evaluation is based, the analysis of the data and the presentation of the
results.Comment: Accepted for publication in Scientometric
Field- and time-normalization of data with many zeros: An empirical analysis using citation and Twitter data
Thelwall (2017a, 2017b) proposed a new family of field- and time-normalized
indicators, which is intended for sparse data. These indicators are based on
units of analysis (e.g., institutions) rather than on the paper level. They
compare the proportion of mentioned papers (e.g., on Twitter) of a unit with
the proportion of mentioned papers in the corresponding fields and publication
years (the expected values). We propose a new indicator (Mantel-Haenszel
quotient, MHq) for the indicator family. The MHq goes back to the MH analysis.
This analysis is an established method, which can be used to pool the data from
several 2x2 cross tables based on different subgroups. We investigate (using
citations and assessments by peers, i.e., F1000Prime recommendations) whether
the indicator family (including the MHq) can distinguish between quality levels
defined by the assessments of peers. Thus, we test the convergent validity. We
find that the MHq is able to distinguish between quality levels (in most cases)
while other indicators of the family are not. Since our study approves the MHq
as a convergent valid indicator, we apply the MHq to four different Twitter
groups as defined by the company Altmetric (e.g., science communicators). Our
results show that there is a weak relationship between all four Twitter groups
and scientific quality, much weaker than between citations and scientific
quality. Therefore, our results discourage the use of Twitter counts in
research evaluation.Comment: This is a substantially extended version of a conference paper which
has been presented at the 16th International Conference on Scientometrics &
Informetrics (ISSI) 2017. 18 pages, 2 tables, 5 figures, and 20 equations.
Accepted for publication in the Scientometrics special issue for the ISSI
2017. arXiv admin note: text overlap with arXiv:1704.02211, arXiv:1712.0222
Tracing the origin of a scientific legend by Reference Publication Year Spectroscopy (RPYS): the legend of the Darwin finches
In a previews paper we introduced the quantitative method named Reference
Publication Year Spectroscopy (RPYS). With this method one can determine the
historical roots of research fields and quantify their impact on current
research. RPYS is based on the analysis of the frequency with which references
are cited in the publications of a specific research field in terms of the
publication years of these cited references. In this study, we illustrate that
RPYS can also be used to reveal the origin of scientific legends. We selected
Darwin finches as an example for illustration. Charles Darwin, the originator
of evolutionary theory, was given credit for finches he did not see and for
observations and insights about the finches he never made. We have shown that a
book published in 1947 is the most-highly cited early reference cited within
the relevant literature. This book had already been revealed as the origin of
the term Darwin finches by Sulloway through careful historical analysis.Comment: Accepted for publication in Scientometric
Allegation of scientific misconduct increases Twitter attention
The web-based microblogging system Twitter is a very popular altmetrics
source for measuring the broader impact of science. In this case study, we
demonstrate how problematic the use of Twitter data for research evaluation can
be, even though the aspiration of measurement is degraded from impact to
attention measurement. We collected the Twitter data for the paper published by
Yamamizu et al. (2017). An investigative committee found that the main figures
in the paper are fraudulent
Which differences can be expected when two universities in the Leiden Ranking are compared? Some benchmarks for institutional research evaluations
The comparison of two universities in terms of bibliometric indicators
frequently faces the problem of assessing the differences as meaningful or not.
This Letter to the Editor proposes some benchmarks which can be used for
supporting the interpretation of institutional differences
- …