 There is a greater need to evaluate the research, and yet it is increasingly more complex to do so. The main criticisms that the inadequate use of informetric indicators receive are exposed. Alternative ways to use them to measure the research performance in an informed and responsible way, exploiting its potential adequately, are suggested

Moed, Henk

English

Scipedia

El profesional de la información, 2018, marzo-abril, v. 27, n. 2. eISSN: 1699-2407     237ALTERNATIVE APPROACHES TO THE QUANTITATIVE ASSESSMENT OF ACADEMIC RESEARCHEnfoques alternativos a la evaluación cuantitativa de la investigación académicaHenk F. MoedHenk F. Moed was a senior staff member at the Centre for Science and Technology Studies (CWTS) at Leiden University between 1981 and 2010. During 2010-2014 he was a senior scientific advisor at Elsevier, Amsterdam. He obtained a Ph.D. degree in Science Studies at the University of Leiden in 1989. He has been active in numerous research topics, including: the creation of bibliometric data-bases from raw data from Thomson Scientific’s Web of Science and Elsevier’s Scopus; development of a new journal impact measure (SNIP). He published over 100 research articles in international, peer reviewed journals. Was program chair of numerous international conferences and is editorial board member of several journals. He is a winner of the Derek de Solla Price Award in 1999. He edited in 2004, jointly with W. Glänzel and U. Schmoch, a Handbook on quantitative science and technology research (Kluwer, 800 pp.), and published in 2005 a monograph, Citation analysis in re-search evaluation (Springer, 346 pp.), which is one of the very few textbooks in the field. He is currently an independent sci-entific advisor, and visiting professor at the Sapienza University of Rome and University of Granada. In September 2017 he published a second monograph with Springer, entitled Applied evaluative informetrics, and he is currently editing a second Handbook on S&T indicators (with W. Glänzel, U. Schmoch and M. Thelwall), to be published in 2018. He is program chair of the upcoming conference of the International Society for Scientometrics and Informetrics (ISSI) in September 2019 in Rome. https://orcid.org/0000-0003-2446-905XUniversità La Sapienza, Roma, Italyhf.moed@gmail.comAbstractThere is a greater need to evaluate the research, and yet it is increasingly more complex to do so. The main criticisms that the inadequate use of informetric indicators receive are exposed. Alternative ways to use them to measure the research performance in an informed and responsible way, exploiting its potential adequately, are suggested.KeywordsInformetrics; Indicators; Criticisms; Alternative approaches; Research assessment; Scientific-scholarly progress; Applied evaluative informetrics; Impact.ResumenExiste una mayor necesidad de evaluar la investigación, y sin embargo cada vez es más complejo hacerlo. Se exponen las principales críticas que recibe el uso inadecuado de indicadores informétricos, y se sugieren formas alternativas de utilizar-los para medir el rendimiento de la investigación de una manera informada y responsable, explotando adecuadamente su potencial.Palabras claveInformetría; Indicadores; Criticas; Alternativas; Evaluación de la investigación; Progreso científico-académico; Informetría evaluativa aplicada; Impacto.Moed, Henk F. (2018). “Alternative approaches to the quantitative assessment of academic research”. El profesional de la información, v. 27, n. 2, pp. 237-239.https://doi.org/10.3145/epi.2018.mar.02Article received on 16-11-2017Henk F. Moed238     El profesional de la información, 2018, marzo-abril, v. 27, n. 2. eISSN: 1699-24071. IntroductionDuring the past decade, in the domain of science policy an increasing emphasis was placed on societal value and value for money, performance-based funding and on globalization of academic research, and a growing need for internal re-search assessment and research information systems. At the same time, due to the computerization of the re-search process and the digitization of scholarly communi-cation, research assessment is more and more becoming a ‘big data’ activity, involving multiple comprehensive citation indexes, electronic full text databases, large publications re-positories, usage data from publishers’ sites, and altmetric, webometric and other new data sources.These trends created an increasing interest in the develop-ment, availability and application of new indicators for re-search assessment. Many new indicators were developed and have become available on a large scale. Desktop biblio-metrics is becoming a common assessment practice.But more and more critique is articulated on the way biblio-metric –or, more general, informetric – indicators are used in research assessment. Indicators may be biased and not measure what they are supposed to measure; most studies adopt a limited time horizon; indicators can be manipulated and may have constitutive effects; measuring societal im-pact is problematic; and when they are used, an evaluative framework and assessment model are often lacking.In my book Applied evaluative informetrics (Moed, 2017a), I discuss the various criticisms in detail. I reflect upon their implications for the actual use of informetric indicators in research assessment, and for future indicator development. The central question in this book is: How can we use re-search performance indicators in an informed and responsi-ble manner, taking into account the critique on the way they are currently used, and properly exploiting their potential? Section 2 of this paper discusses the main criticisms against the use of bibliometric or informetric indicators in research assessment. Next, Section 3 sketches a series of alternative approaches to the use of informetric indicators in research assessment. Finally, concluding remarks are made in Section 4. These sections are largely based on the executive sum-mary in Moed (2017a), and a blog posted at The Bibliomagi-cian (Moed, 2017b).2. Main criticisms against the use of informetric indicators in research assessmentThe following views are expressed, partly supportive, and partly as a counter-critique towards the criticisms of current practices in the use of research performance indicators. - Calculating indicators at the level of an individual and claiming they measure by themselves the individual’s performance, suggests a façade of exactness that cannot be justified. A valid and fair assessment of individual re-search performance can be conducted properly only on the basis of sufficient background knowledge on the par-ticular role they played in the research presented in their publications, and by taking into account also other types on information on their performance.- The notion of making a contribution to scientific-scholar-ly progress, does have a basis in reality, that can best be illustrated by referring to an historical viewpoint. History will show which contributions to scholarly knowledge are valuable and sustainable. In this sense, informetric indi-cators do not measure contribution to scientific-scholarly progress, but rather indicate attention, visibility or short term impact.- Societal value cannot be assessed in a politically neutral manner. The foundation of the criteria for assessing so-cietal value is not a matter in which scientific experts have qualitate qua a preferred status, but should eventually take place in the policy domain. One possible option is moving away from the objective to evaluate an activity’s societal value, towards measuring in a neutral manner researchers’ orientation towards any articulated, lawful need in society.- Studies on changes in editorial and author practices under the influence of assessment exercises are most relevant and illuminative. But the issue at stake is not whether scholars’ practices change under the influence of the use of informetric indicators, but rather whether or not the application of such measures enhances research perfor-mance. Although this is in some cases difficult to assess without extra study, other cases clearly show traces of mere indicator manipulation with no positive effect on performance at all.- A typical example of a constitutive effect is that research quality is more and more conceived as what citations measure. More empirical research on the size of consti-tutive effects is needed. If there is a genuine constituti-ve effect of informetric indicators in quality assessment, one should not point the critique on current assessment practices merely towards informetric indicators as such, but rather towards any claim for an absolute status of a particular way to assess research quality. Research quality is not what citations measure, but at the same time peers may assess it wrongly. - If the role of informetric indicators has become too do-minant, it does not follow that the notion to intelligently combine peer judgments and indicators is fundamentally flawed and that indicators should be banned from the as-sessment arena. But it does show that the combination Due to the computerization of the re-search process and the digitization of scholarly communication, research as-sessment is more and more becoming a ‘big data’ activityInformetric indicators do not measure contribution to scientific-scholarly pro-gress, but rather indicate attention, visi-bility or short term impactAlternative approaches to the quantitative assessment of academic researchEl profesional de la información, 2018, marzo-abril, v. 27, n. 2. eISSN: 1699-2407     239of the two methodologies has to be organized in a more balanced manner. - In the proper use of informetric tools an evaluative fra-mework and an assessment model are indispensable. To the extent that in a practical application an evaluative fra-mework is absent or implicit, there is a vacuum, that may be easily filled either with ad-hoc arguments of evaluators and policy makers, or with un-reflected assumptions un-derlying informetric tools. Perhaps the role of such ad hoc arguments and assumptions has nowadays become too dominant. It can be reduced only if evaluative framewor-ks become stronger, and more actively determine which tools are to be used, and how.3. Alternative approaches to the assessment of academic researchThe following alternative approaches to the assessment of academic research are proposed. - A key assumption in the assessment of academic research has been that it is not the potential influence or impor-tance of research, but the actual influence or impact that is of primary interest to policy makers and evaluators. But an academic assessment policy is conceivable that rejects this assumption. It embodies a shift in focus from the measurement of performance itself to the assessment of preconditions for performance.- Rather than using citations as indicator of research im-portance or quality, they could provide a tool in the as-sessment of communication effectiveness, and express the extent to which researchers bring their work to the attention of a broad, potentially interested audience. This extent can in principle be measured with informetric tools. It discourages the use of citation data as a principal indicator of importance. - The functions of publications and other forms of scien-tific-scholarly output, as well as their target audiences should be taken into account more explicitly than they have been in the past. Scientific-scholarly journals could be systematically categorized according to their function and target audience, and separate indicators could be cal-culated for each category. More sophisticated indicators of internationality of communication sources can be cal-culated than the journal impact factor and its variants.- One possible approach to the use of informetric indicators in research assessment is a systematic exploration of in-dicators as tools to set minimum performance standards. Using baseline indicators, researchers will most probably change their research practices as they are stimulated to meet the standards, but if the standards are appropriate and fair, this behavior will actually increase their perfor-mance and that of their institutions.- At the upper part of the quality distribution, it is perhaps feasible to distinguish entities which are ‘hors catégorie’, or ‘at Nobel Prize level’. Assessment processes focusing on the very top of the quality distributions could further ope-rationalize the criteria for this qualification.- Realistically speaking, rankings of world universities are here to stay. Academic institutions could, individually or collectively, seek to influence the various systems by for-mally sending to their creators a request to consider the implementation of a series of new features: more advan-ced analytical tools; more insight into how the methodo-logical decisions influence rankings; and more informa-tion in the system about additional, relevant factors, such as teaching course language. - In response to major criticisms towards current national research assessment exercises and performance-based funding formula, an alternative model would require less efforts, be more transparent, stimulate new research li-nes and reduce to some extent the Matthew effect. The basic unit of assessment in such a model is the emerging research group rather than the individual researcher. Institutions submit emerging groups and their research programs, which are assessed in a combined peer re-view-based and informetric approach, applying minimum performance criteria. A funding formula is partly based on an institution’s number of acknowledged emerging groups.4. Concluding remarksThe practical realization of these proposals requires a lar-ge amount of informetric research and development. The book proposes several new directions for indicator de-velopment. They constitute important elements of a wi-der R&D program of applied evaluative informetrics. The further exploration of measures of communication effec-tiveness, minimum performance standards, new functio-nalities in research information systems, and tools to facili-tate alternative funding formula, should be conducted in a close collaboration between informetricians and external stakeholders, each with their own domain of expertise and responsibilities.Key referencesMoed, Henk F. (2017a). Applied evaluative informetrics, Springer, XXI + 312 pp. ISBN: 978 3 319 60521 0 https://doi.org/10.1007/978-3-319-60522-7Moed, Henk F. (2017b). How can we use research perfor-mance indicators in an informed and responsible manner?https://thebibliomagician.wordpress.com/2017/11/03Rather than using citations as indicator of research importance or quality, they could provide a tool in the assessment of communication effectivenesA typical example of a constitutive effect is that research quality is more and more conceived as what citations measure

Alternative approaches to the quantitative assessment of academic research

https://www.scipedia.com/public/Moed_2018a

Alternative approaches to the quantitative assessment of academic research

Abstract

Similar works

Full text

Available Versions

Scipedia