Metrics to evaluate research performance in academic institutions: A critique of ERA 2010 as applied in forestry and the indirect H2 index as a possible alternative

Excellence for Research in Australia (ERA) is an attempt by the Australian Research Council to rate Australian universities on a 5-point scale within 180 Fields of Research using metrics and peer evaluation by an evaluation committee. Some of the bibliometric data contributing to this ranking suffer statistical issues associated with skewed distributions. Other data are standardised year-by-year, placing undue emphasis on the most recent publications which may not yet have reliable citation patterns. The bibliometric data offered to the evaluation committees is extensive, but lacks effective syntheses such as the h-index and its variants. The indirect H2 index is objective, can be computed automatically and efficiently, is resistant to manipulation, and a good indicator of impact to assist the ERA evaluation committees and to similar evaluations internationally.Comment: 19 pages, 6 figures, 7 tables, appendice

Does the Committee Peer Review Select the Best Applicants for Funding? An Investigation of the Selection Process for Two European Molecular Biology Organization Programmes

Does peer review fulfill its declared objective of identifying the best science and the best scientists? In order to answer this question we analyzed the Long-Term Fellowship and the Young Investigator programmes of the European Molecular Biology Organization. Both programmes aim to identify and support the best post doctoral fellows and young group leaders in the life sciences. We checked the association between the selection decisions and the scientific performance of the applicants. Our study involved publication and citation data for 668 applicants to the Long-Term Fellowship programme from the year 1998 (130 approved, 538 rejected) and 297 applicants to the Young Investigator programme (39 approved and 258 rejected applicants) from the years 2001 and 2002. If quantity and impact of research publications are used as a criterion for scientific achievement, the results of (zero-truncated) negative binomial models show that the peer review process indeed selects scientists who perform on a higher level than the rejected ones subsequent to application. We determined the extent of errors due to over-estimation (type I errors) and under-estimation (type 2 errors) of future scientific performance. Our statistical analyses point out that between 26% and 48% of the decisions made to award or reject an application show one of both error types. Even though for a part of the applicants, the selection committee did not correctly estimate the applicant's future performance, the results show a statistically significant association between selection decisions and the applicants' scientific achievements, if quantity and impact of research publications are used as a criterion for scientific achievement

What Makes a Great Journal Great in the Sciences? Which Came First, the Chicken or the Egg?

The paper is concerned with analysing what makes a great journal great in the sciences, based on quantifiable Research Assessment Measures (RAM). Alternative RAM are discussed, with an emphasis on the Thomson Reuters ISI Web of Science database (hereafter ISI). Various ISI RAM that are calculated annually or updated daily are defined and analysed, including the classic 2-year impact factor (2YIF), 5-year impact factor (5YIF), Immediacy (or zero-year impact factor (0YIF)), Eigenfactor, Article Influence, C3PO (Citation Performance Per Paper Online), h-index, Zinfluence, PI-BETA (Papers Ignored - By Even The Authors), Impact Factor Inflation (IFI), and three new RAM, namely Historical Self-citation Threshold Approval Rating (H-STAR), 2 Year Self-citation Threshold Approval Rating (2Y-STAR), and Cited Article Influence (CAI). The RAM data are analysed for the 6 most highly cited journals in 20 highly-varied and well-known ISI categories in the sciences, where the journals are chosen on the basis of 2YIF. The application to these 20 ISI categories could be used as a template for other ISI categories in the sciences and social sciences, and as a benchmark for newer journals in a range of ISI disciplines. In addition to evaluating the 6 most highly cited journals in each of 20 ISI categories, the paper also highlights the similarities and differences in alternative RAM, finds that several RAM capture similar performance characteristics for the most highly cited scientific journals, determines that PI-BETA is not highly correlated with the other RAM, and hence conveys additional information regarding research performance. In order to provide a meta analysis summary of the RAM, which are predominantly ratios, harmonic mean rankings are presented of the 13 RAM for the 6 most highly cited journals in each of the 20 ISI categories. It is shown that emphasizing THE impact factor, specifically the 2-year impact factor, of a journal to the exclusion of other informative RAM can lead to a distorted evaluation of journal performance and influence on different disciplines, especially in view of inflated journal self citations

Benchmarking scientific performance by decomposing leadership of Cuban and Latin American institutions in Public Health

This is a post-peer-review, pre-copyedit version of an article published in Scientometrics. The final authenticated version is available online at: http://dx.doi.org/10.1007/s11192-015-1831-z”.Comparative benchmarking with bibliometric indicators can be an aid in decision-making with regard to research management. This study aims to characterize scientific performance in a domain (Public Health) by the institutions of a country (Cuba), taking as reference world output and regional output (other Latin American centers) during the period 2003–2012. A new approach is used here to assess to what extent the leadership of a specific institution can change its citation impact. Cuba was found to have a high level of specialization and scientific leadership that does not match the low international visibility of Cuban institutions. This leading output appears mainly in non-collaborative papers, in national journals; publication in English is very scarce and the rate of international collaboration is very low. The Instituto de Medicina Tropical Pedro Kouri stands out, alone, as a national reference. Meanwhile, at the regional level, Latin American institutions deserving mention for their high autonomy in normalized citation would include Universidad de Buenos Aires (ARG), Universidade Federal de Pelotas (BRA), Consejo Nacional de Investigaciones Cientı´ficas y Te´cnicas (ARG), Instituto Oswaldo Cruz (BRA) and the Centro de Pesquisas Rene Rachou (BRA). We identify a crucial aspect that can give rise to misinterpretations of data: a high share of leadership cannot be considered positive for institutions when it is mainly associated with a high proportion of non-collaborative papers and a very low level of performance. Because leadership might be questionable in some cases, we propose future studies to ensure a better interpretation of findings.This work was made possible through financing by the scholarship funds for international mobility between Andalusian and IberoAmerican Universities and the SCImago GroupPeer reviewe

An index to quantify an individual's scientific research output that takes into account the effect of multiple coauthorship

I propose the index $\hbar$ ("hbar"), defined as the number of papers of an individual that have citation count larger than or equal to the $\hbar$ of all coauthors of each paper, as a useful index to characterize the scientific output of a researcher that takes into account the effect of multiple coauthorship. The bar is higher for $\hbar$.Comment: A few minor changes from v1. To be published in Scientometric

Flare energetics

In this investigation of flare energetics, researchers sought to establish a comprehensive and self-consistent picture of the sources and transport of energy within a flare. To achieve this goal, they chose five flares in 1980 that were well observed with instruments on the Solar Maximum Mission, and with other space-borne and ground-based instruments. The events were chosen to represent various types of flares. Details of the observations available for them and the corresponding physical parameters derived from these data are presented. The flares were studied from two perspectives, the impulsive and gradual phases, and then the results were compared to obtain the overall picture of the energics of these flares. The role that modeling can play in estimating the total energy of a flare when the observationally determined parameters are used as the input to a numerical model is discussed. Finally, a critique of the current understanding of flare energetics and the methods used to determine various energetics terms is outlined, and possible future directions of research in this area are suggested

The flaring and quiescent components of the solar corona

The solar corona is a template to understand stellar activity. The Sun is a moderately active star, and its corona differs from active stars: active stellar coronae have a double-peaked EM(T) with the hot peak at 8-20 MK, while the non flaring solar corona has one peak at 1-2 MK. We study the average contribution of flares to the solar EM(T) to investigate indirectly the hypothesis that the hot peak of the EM(T) of active stellar coronae is due to a large number of unresolved solar-like flares, and to infer properties on the flare distribution from nano- to macro-flares. We measure the disk-integrated time-averaged emission measure, EM_F(T), of an unbiased sample of solar flares analyzing uninterrupted GOES/XRS light curves over time intervals of one month. We obtain the EM_Q(T) of quiescent corona for the same time intervals from the Yohkoh/SXT data. To investigate how EM_F(T) and EM_Q(T) vary with the solar cycle, we evaluate them at different phases of the cycle (from Dec. 1991 to Apr. 1998). Irrespective of the solar cycle phase, EM_F(T) appears like a peak of the distribution significantly larger than the values of EM_Q(T) for T~5-10 MK. As a result the time-averaged EM(T) of the whole solar corona is double-peaked, with the hot peak, due to time-averaged flares, located at temperature similar of that of active stars, but less enhanced. The EM_F(T) shape supports the hypothesis that the hot EM(T) peak of active coronae is due to unresolved solar-like flares. If this is the case, quiescent and flare components should follow different scaling laws for increasing stellar activity. In the assumption that the heating of the corona is entirely due to flares, from nano- to macro-flares, then either the flare distribution or the confined plasma response to flares, or both, are bimodal.Comment: 8 pages, 7 postscript figures, accepted for publication in Astronomy and Astrophysic

Which aspects of the open science agenda are most relevant to scientometric research and publishing? An opinion paper

© 2021 The Authors. Published by MIT Press. This is an open access article available under a Creative Commons licence. The published version can be accessed at the following link on the publisher’s website: https://doi.org/10.1162/qss_e_00121Open Science is an umbrella term that encompasses many recommendations for possible changes in research practices, management, and publishing with the objective to increase transparency and accessibility. This has become an important science policy issue that all disciplines should consider. Many Open Science recommendations may be valuable for the further development of research and publishing but not all are relevant to all fields. This opinion paper considers the aspects of Open Science that are most relevant for scientometricians, discussing how they can be usefully applied.The work of R.G. was supported by the Flemish Government through its funding of the Flemish Centre for R&D Monitoring (ECOOM

Publication and patent analysis of European researchers in the field of production technology and manufacturing systems

This paper develops a structured comparison among a sample of European researchers in the field of Production Technology and Manufacturing Systems, on the basis of scientific publications and patents. Researchers are evaluated and compared by a variegated set of indicators concerning (1) the output of individual researchers and (2) that of groups of researchers from the same country. While not claiming to be exhaustive, the results of this preliminary study provide a rough indication of the publishing and patenting activity of researchers in the field of interest, identifying (dis)similarities between different countries. Of particular interest is a proposal for aggregating analysis results by means of maps based on publication and patent indicators. A large amount of empirical data are presented and discusse