Confidence intervals for normalised citation counts: Can they delimit underlying research capability?

This is an accepted manuscript of an article published by Elsevier in Journal of Informetrics on 24/10/2017, available online: https://doi.org/10.1016/j.joi.2017.09.002 The accepted version of the publication may differ from the final published version.Normalised citation counts are routinely used to assess the average impact of research groups or nations. There is controversy over whether confidence intervals for them are theoretically valid or practically useful. In response, this article introduces the concept of a group’s underlying research capability to produce impactful research. It then investigates whether confidence intervals could delimit the underlying capability of a group in practice. From 123120 confidence interval comparisons for the average citation impact of the national outputs of ten countries within 36 individual large monodisciplinary journals, moderately fewer than 95% of subsequent indicator values fall within 95% confidence intervals from prior years, with the percentage declining over time. This is consistent with confidence intervals effectively delimiting the research capability of a group, although it does not prove that this is the cause of the results. The results are unaffected by whether internationally collaborative articles are included

The research production of nations and departments: A statistical model for the share of publications

Policy makers and managers sometimes assess the share of research produced by a group (country, department, institution). This takes the form of the percentage of publications in a journal, field or broad area that has been published by the group. This quantity is affected by essentially random influences that obscure underlying changes over time and differences between groups. A model of research production is needed to help identify whether differences between two shares indicate underlying differences. This article introduces a simple production model for indicators that report the share of the world’s output in a journal or subject category, assuming that every new article has the same probability to be authored by a given group. With this assumption, confidence limits can be calculated for the underlying production capability (i.e., probability to publish). The results of a time series analysis of national contributions to 36 large monodisciplinary journals 1996-2016 are broadly consistent with this hypothesis. Follow up tests of countries and institutions in 26 Scopus subject categories support the conclusions but highlight the importance of ensuring consistent subject category coverage

科学家评价方法述评

［目的 / 意义］ 梳理并评述国内外有关科学家评价的方法， 归纳好的做法和经验， 为相关研究机构制定科学家评价方法提供参考。［方法 / 过程］ 从定性、定量和新型评价范式等角度进行梳理， 其中定量研究方法主要包括: 基于论文数和引文数、标准化指标、h 指数及其衍生指数等单指标定量评价方法，基于链接的方法， 替代计量学以及多指标综合评价方法等。定性方法主要从同行评议角度展开， 新型评价范式 主 要 介 绍 了 大 数 据 时 代 基 于 人 工 智 能 和机器学习的科学家评价新范式。［结果 / 结论］ 科学家评价工作还 有 很 大 的 空 间 有 待 探 索， 包 括: 评 价 理 论 基 础、时间维度的考虑、强调多指标综合的方法、指标优化、评价实践等多个方面。 &nbsp;</p

Strategic Hiring and Research Productivity at U.S. Research Universities

Research universities provide new knowledge that advance technology and links the United States’ science and knowledge system with the world (Altbach, 2011). Public research universities play a significant role in regional and national economic development. Universities rely on funding from multiple sources to operate effectively. However, in recent years, public research universities in the United States have experienced declines in state funding (Hyman & Jacobs, 2010; Webber, 2017). To offset declines in funding, public research universities may seek to increase research productivity through the hiring of faculty to address strategic institutional research goals. Strategic hiring is one approach used by institutions to increase research productivity. However, little is known about the effectiveness of strategic faculty hiring endeavors to increase research productivity. In order to better understand the effectiveness of strategic faculty hiring endeavors on research productivity at U.S. public research universities, this study surveyed Chief Research Officers to determine the characteristics of the institutions (Carnegie class, number of faculty and research staff, number of proposals submitted, and strategic faculty hiring status), the characteristics of strategic faculty hiring endeavors (alignment with strategic plan, distinct measure of research productivity, and institutional area responsible) and the perceived effectiveness of strategic faculty hiring endeavors on research productivity. A researcher designed study, based on the relevant literature, was used to survey respondents. All respondents expressed the perception of strategic faculty hiring as an effective way to increase research productivity. More specifically, the Chief Research Officers at each of the responding institutions indicated that the strategic faculty hiring endeavor at their university was, on average, moderate to very effective in increasing research productivity. The study found no statistically significant relationship between the characteristics of strategic faculty hiring endeavors and the perceived effectiveness on research productivit