5 research outputs found
Quantifying the consistency of scientific databases
Science is a social process with far-reaching impact on our modern society.
In the recent years, for the first time we are able to scientifically study the
science itself. This is enabled by massive amounts of data on scientific
publications that is increasingly becoming available. The data is contained in
several databases such as Web of Science or PubMed, maintained by various
public and private entities. Unfortunately, these databases are not always
consistent, which considerably hinders this study. Relying on the powerful
framework of complex networks, we conduct a systematic analysis of the
consistency among six major scientific databases. We found that identifying a
single "best" database is far from easy. Nevertheless, our results indicate
appreciable differences in mutual consistency of different databases, which we
interpret as recipes for future bibliometric studies.Comment: 20 pages, 5 figures, 4 table
Do PageRank-based author rankings outperform simple citation counts?
The basic indicators of a researcher's productivity and impact are still the
number of publications and their citation counts. These metrics are clear,
straightforward, and easy to obtain. When a ranking of scholars is needed, for
instance in grant, award, or promotion procedures, their use is the fastest and
cheapest way of prioritizing some scientists over others. However, due to their
nature, there is a danger of oversimplifying scientific achievements.
Therefore, many other indicators have been proposed including the usage of the
PageRank algorithm known for the ranking of webpages and its modifications
suited to citation networks. Nevertheless, this recursive method is
computationally expensive and even if it has the advantage of favouring
prestige over popularity, its application should be well justified,
particularly when compared to the standard citation counts. In this study, we
analyze three large datasets of computer science papers in the categories of
artificial intelligence, software engineering, and theory and methods and apply
12 different ranking methods to the citation networks of authors. We compare
the resulting rankings with self-compiled lists of outstanding researchers
selected as frequent editorial board members of prestigious journals in the
field and conclude that there is no evidence of PageRank-based methods
outperforming simple citation counts.Comment: 28 pages, 5 figures, 6 table
Recommended from our members
Mining Scholarly Publications for Research Evaluation
Scientific research can lead to breakthroughs that revolutionise society by solving long-standing problems. However, investment of public funds into research requires the ability to clearly demonstrate beneficial returns, accountability, and good management. At the same time, with the amount of scholarly literature rapidly expanding, recognising key research that presents the most important contributions to science is becoming increasingly difficult and time-consuming. This creates a need for effective and appropriate research evaluation methods. However, the question of how to evaluate the quality of research outcomes is very difficult to answer and despite decades of research, there is still no standard solution to this problem.
Given this growing need for research evaluation, it is increasingly important to understand how research should be evaluated, and whether the existing methods meet this need. However, the current solutions, which are predominantly based on counting the number of interactions in the scholarly communication network, are insufficient for a number of reasons. In particular, they struggle in capturing many aspects of the academic culture and often significantly lag behind current developments.
This work focuses on the evaluation of research publications and aims at creating new methods which utilise publication content. It studies the concept of research publication quality, methods assessing the performance of new research publication evaluation methods, analyses and extends the existing methods, and, most importantly, presents a new class of metrics which are based on publication manuscripts. By bridging the fields of research evaluation and text- and data-mining, this work provides tools for analysing the outcomes of research, and for relieving information overload in scholarly publishing