Do Rankings Reflect Research Quality?

Publication and citation rankings have become major indicators of the scientific worth of universities and countries, and determine to a large extent the career of individual scholars. We argue that such rankings do not effectively measure research quality, which should be the essence of evaluation. For that reason, an alternative ranking is developed as a quality indicator, based on membership on academic editorial boards of professional journals. It turns out that especially the ranking of individual scholars is far from objective. The results differ markedly, depending on whether research quantity or research quality is considered. Even quantity rankings are not objective; two citation rankings, based on different samples, produce entirely different results. It follows that any career decisions based on rankings are dominatedby chance and do not reflect research quality. Instead of propagating a ranking based on board membership as the gold standard, we suggest that committees make use of this quality indicator to find members who, in turn, evaluate the research quality of individual scholars.rankings, universities, scholars, publications, citations

Zipf's law and log-normal distributions in measures of scientific output across fields and institutions: 40 years of Slovenia's research as an example

Slovenia's Current Research Information System (SICRIS) currently hosts 86,443 publications with citation data from 8,359 researchers working on the whole plethora of social and natural sciences from 1970 till present. Using these data, we show that the citation distributions derived from individual publications have Zipfian properties in that they can be fitted by a power law $P(x) \sim x^{-\alpha}$, with $\alpha$ between 2.4 and 3.1 depending on the institution and field of research. Distributions of indexes that quantify the success of researchers rather than individual publications, on the other hand, cannot be associated with a power law. We find that for Egghe's g-index and Hirsch's h-index the log-normal form $P(x) \sim \exp[-a\ln x -b(\ln x)^2]$ applies best, with $a$ and $b$ depending moderately on the underlying set of researchers. In special cases, particularly for institutions with a strongly hierarchical constitution and research fields with high self-citation rates, exponential distributions can be observed as well. Both indexes yield distributions with equivalent statistical properties, which is a strong indicator for their consistency and logical connectedness. At the same time, differences in the assessment of citation histories of individual researchers strengthen their importance for properly evaluating the quality and impact of scientific output.Comment: 8 pages, 3 figures; accepted for publication in Journal of Informetrics [supplementary material available at http://www.matjazperc.com/sicris/stats.html

Do rankings reflect research quality?

Publication and citation rankings have become major indicators of the scientific worth of universities and determine to a large extent the career of individual scholars. Such rankings do not effectively measure research quality, which should be the essence of any evaluation. These quantity rankings are not objective; two citation rankings, based on different samples, produce entirely different results. For that reason, an alternative ranking is developed as a quality indicator, based on membership on academic editorial boards of professional journals. It turns out that the ranking of individual scholars based on that measure is far from objective. Furthermore, the results differ markedly, depending on whether research quantity or quality is considered. Thus, career decisions based on rankings are dominated by chance and do not reflect research quality. We suggest that evaluations should rely on multiple criteria. Public management should return to approved methods such as engaging independent experts who in turn provide measurements of research quality for their research communities

Diferencias y evolución del impacto académico en los perfiles de Google Scholar Citations: Una aplicación de árboles de decisión

The aim of this paper is to analyse the research performance of more than 3,000 profiles from Google Scholar Citations to define which groups (by gender, academic positions and disciplines) bring together more successful profiles. This analysis was faced both from a static and a longitudinal point of view. Decision trees were used to detect the most important variables in order to distinguish winning profiles and to observe which categories bring together more authors with high number of citations and h-indexes. Results show that the career is the most relevant aspect to achieve citations and improve the h-index. Senior researchers are thus ranked in the best positions, while young scholars describe nascent curricula. Otherwise, this distribution changes when growth rates are computed. It is concluded that researchers with a stable career from life sciences have better research impact than young researchers from humanities and social sciences, despite that the fastest growing profiles belong to young scholars.El propósito de este artículo es analizar la producción e impacto de más de 3000 perfiles tomados de Google Scholar Citations con el fin de identificar qué segmentos (por género, puestos académicos y disciplinas) son más exitosos en términos de impacto científico. Este análisis se afrontó tanto desde una perspectiva estática como longitudinal. Los árboles de decisión fueron usados para detectar las variables más importantes para agrupar perfiles con un mayor número de citas por artículo e índice h. Resultados muestran que la carrera académica es el factor más importante para conseguir citas y mejorar el índice h. Los investigadores más veteranos son así los que ocupan las primeras posiciones, mientras que los jóvenes investigadores describen curriculums en ciernes. Por el contrario, estos resultados cambian cuando el crecimiento de los perfiles es observado. Así los curriculums más jóvenes son los que experimentan un crecimiento más fuerte, mientras que los más veteranos muestran signos de estabilización y estancamiento. Se concluye que los investigadores con una carrera estable pertenecientes a las ciencias de la vida tienen mejor impacto que los jóvenes investigadores de humanidades y ciencias sociales, a pesar de que estos últimos son los que más rápido crecen en número de citas por documento

Congress UPV Proceedings of the 21ST International Conference on Science and Technology Indicators

This is the book of proceedings of the 21st Science and Technology Indicators Conference that took place in València (Spain) from 14th to 16th of September 2016. The conference theme for this year, ‘Peripheries, frontiers and beyond’ aimed to study the development and use of Science, Technology and Innovation indicators in spaces that have not been the focus of current indicator development, for example, in the Global South, or the Social Sciences and Humanities. The exploration to the margins and beyond proposed by the theme has brought to the STI Conference an interesting array of new contributors from a variety of fields and geographies. This year’s conference had a record 382 registered participants from 40 different countries, including 23 European, 9 American, 4 Asia-Pacific, 4 Africa and Near East. About 26% of participants came from outside of Europe. There were also many participants (17%) from organisations outside academia including governments (8%), businesses (5%), foundations (2%) and international organisations (2%). This is particularly important in a field that is practice-oriented. The chapters of the proceedings attest to the breadth of issues discussed. Infrastructure, benchmarking and use of innovation indicators, societal impact and mission oriented-research, mobility and careers, social sciences and the humanities, participation and culture, gender, and altmetrics, among others. We hope that the diversity of this Conference has fostered productive dialogues and synergistic ideas and made a contribution, small as it may be, to the development and use of indicators that, being more inclusive, will foster a more inclusive and fair world

Study on open science: The general state of the play in Open Science principles and practices at European life sciences institutes

Nowadays, open science is a hot topic on all levels and also is one of the priorities of the European Research Area. Components that are commonly associated with open science are open access, open data, open methodology, open source, open peer review, open science policies and citizen science. Open science may a great potential to connect and influence the practices of researchers, funding institutions and the public. In this paper, we evaluate the level of openness based on public surveys at four European life sciences institute

International collaboration and research quality: Evidence from the Us-China collaboration in nanotechnology

Atlanta Conference on Science and Innovation Policy 2011This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder. ©2011 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.Based on a longitudinal publication data of 77 Chinese nanoscientists, this study empirically examines the impact of US-China collaboration on the research performance of Chinese researchers. The study found that US-China collaboration has a positive effect on China s nano research quality. And such impact demonstrates a time-decaying pattern at the level of individual paper, but not at the level of journal

AN AUTOMATIC METHOD FOR CLASSIFYING MEDICAL RESEARCHERS INTO DOMAIN SPECIFIC SUBGROUPS

Objective:This dissertation developed an automatic classification procedure, as an example of a novel tool for an informationist, which extracts information from published abstracts, classifies abstracts into their "fields of study," and then determines the researcher's "field of study" and "level of activity." Method: This dissertation compared a domain expert's method of classification and an automatic classification procedure on a random sample of 101 medical researchers (derived from a potential list of 305 medical researchers) and their associated abstracts. Design: The study design is a retrospective, cross-sectional, inter-rater agreement study, designed to compare two classification methods (i.e., automatic classification procedure and domain expert). The study population consists of University of Pittsburgh, School of Medicine, Department of Medicine (DOM) professionals who (1) have published at least one article listed in PubMed® as first or last author and/or (2) are the primary investigator for at least one grant listed in CRISP.Main outcome measures: Three outcome measures were derived from the domain expert's versus automatic categorization procedure: (1) an abstract's "field of study," (2) a researcher's "field of study" and (3) a researcher's "level of activity and field of study." Results: Kappa showed moderate agreement between automatic and domain expert classification for the abstracts' "field of study" (Kappa = 0.535, n = 504, p < .000). Kappa showed moderate agreement between automatic and domain expert classification of the researcher's "field of study" (Kappa = 0.535, n = 101, p < .000). Kappa showed good agreement between automatic and domain expert classification of the researcher's "level of activity and field of study" (Kappa = 0.634, n = 101, p < .000). Conclusion: The study suggests that an automatic library classification procedure can provide rapid classification of medical research abstracts into their "fields of study." The classification procedure can also process multiple abstracts' "fields of study" and classify their associated medical researchers into their "field of study" and "level of activity and field of study." The classification procedure, used as a tool by an informationist, can be used as the basis for new services