Publishing in a high-impact journal carries the implicit promise that the article will also be highly cited. But the proof of this logic remains unsubstantiated. By combining more accurate citation metrics, like the hIa-index and the citation-per-author-per-year metric, Anne-Wil Harzing and Wilfred Mijnhardt provide a more substantial alternative to the narrow journal-based metric. This combined metric provides a more reliable comparison between academics in different disciplines and at different career stages

Harzing, Anne-Wil

Mijnhardt, Wilfred

LSE Research Online

9/17/2014Proof over promise: Moving citation metric systems beyondjournal impact towards a career impact approach.blogs.lse.ac.uk /impactofsocialsciences/2014/09/17/proof-over-promise-career-impact-approach-harzing/Publishing in a high-impact journal carries the implicit promise that the article will also be highly cited. Butthe proof of this logic remains unsubstantiated. By combining more accurate citation metrics, likethe hIa-index and the citation-per-author-per-year metric, Anne-Wil Harzing and WilfredMijnhardt provide a more substantial alternative to the narrow journal-based metric. This combinedmetric provides a more reliable comparison between academics in different disciplines and atdifferent career stages.Economists love to rank. Even a casual five-minute literature search reveals literally hundreds ofpublications on rankings of academic productivity and impact. Dutch economists are no exception.In fact, they produced what, to our best knowledge, is the oldest ranking in the field: a nation-wideranking of Economists (the Economics top-40) that has entered its fourth decade. However, thisranking is based on publication volume in ISI-listed journals weighted by the journal’s impact factoror article influence score, rather than on the actual impact of the publications in question.In this blog we explain the added value of a career-impact focus and envision some practicalimplications for business schools.  We will show that an impact driven approach using GoogleScholar creates a more democratic and inclusive assessment.Proof (impact) and Promise (journals)On average, publications in high-impact journals by definition get cited more frequently than publications in low-impact journals as the journal impact factor is based on average citations. This still is the fundamental logic behindthe creation and maintenance of Journals list by business schools and associations such as ABS. We call thisprinciple “promise”, i.e. publishing in a high-impact journal carries the implicit promise that the article will also behighly cited. However, not all individual papers published in these high-impact journals will fulfil this promise. In ourpaper Proof over promise: Towards a more inclusive ranking of Dutch academics in Economics & Business , wetherefore set out to create a ranking based on “proof”, i.e. rather than looking at the promised number of citationsimplied by the journal impact factor or article influence score, we look at actual citations to an author’s work.1/4Image credit: George Peabody Library by Matthew Petroff  (Wikimedia, CC BY-SA 3.0)Introducing two new impact metrics with individualized career focusIn our paper we use two relatively new metrics, the citations per author per year (CAY) metric and the individualannual h-index (hIa, see Harzing, Alakangas & Adams, 2014 for details). These two metrics are the individualisedand annualised equivalents of respectively total citations and the traditional h-index, two important bibliometricmeasures. We correct our metrics for the number of co-authors as the number of co-authors positively influencesboth the number of publications an academic is able to publish and the number of citations. A correction for thenumber of years an academic has been active is important as citations continue to increase over an academiccareer.hIa career impact: field independent focus on contributionIn this blog, we focus on the hIa index, which is calculated by dividing the individual h-index (an h-index corrected forthe number of co-authors) by the number of years an academic has been active, i.e. the number of years that havelapsed since their first publication. The metric thus represents the average number of single-author-equivalent“impactful” articles that an academic has published per year and hence permits an intuitive interpretation. Based onan empirical example of 146 academics in five major disciplines at different career stages, Harzing, Alakangas &Adams (2014) showed that the hIa index attenuates h-index differences that are purely attributable to (disciplinary)co-authorship practices and career lengths. As such, this metric provides a more reliable comparison betweenacademics in different disciplines and at different career stages than the h-index.A second aspect of our focus on proof over promise is the use of Google Scholar rather than ISI as a data source.Although ISI listing is seen by many to imply a quality stamp, in our view it should not matter where research ispublished. If a particular research output is highly cited, it clearly influences the field and that should be moreimportant than the journal in which it is published. As Google Scholar on its own is not very suitable for bibliometricanalyses, Publish or Perish (Harzing 2007) was used to collect citation data from Google Scholar. There are nowmore than 500 published articles referring to the Publish or Perish program. This indicates that—in spite of itslimitations—Google Scholar is perceived to be a useful source of bibliometric data.Resulting ranking for Economics & Business in the Netherlands2/4The resulting hIa-based ranking of academics in Economics & Business in the Netherlands (see Harzing &Mijnhardt, in press for details) is substantially different from the original ranking based on publications and is moreinclusive in terms of disciplines, age and affiliation.Whereas in the original publication-based Economics top-40 more than two thirds of the listed academicsworks in Economics or Management Science, this proportion is reduced to 43% in the hIa ranking. Thechange in disciplinary composition is particularly striking in the top-20. In the publication-based Economicstop-40, three quarters of the academics in the top-20 were economists, in the PoP hIa top-40 this is reducedto just over a third.At 47, the average age in the hIa top-40 is lower than in the original ranking (50), but most noticeable is theaddition an additional five academics under 40 and another five aged between 40 and 45. In fact, all but threeof the newly listed academics in this top-40 are aged 45 or under.In the original top-40 academics affiliated with Erasmus and Tilburg made up just over or just under half ofthe list, in the hIa top-40, this is reduced to a third. Apart from the University of Utrecht, every university isrepresented in the hIa top-40. Maastricht, Eindhoven and the UvA (University of Amsterdam) in particular domuch better in citation-based rankings than in the original publication-based top-40.We argue that our “proof over promise” approach is more “democratic”/inclusive than the original Economics top-40.First, by expanding the type of research outputs considered beyond the narrow scope of publications in ISI-listedjournals, we remove the disciplinary bias against Management, Marketing and Accounting & Finance, disciplines inwhich a smaller proportion of high-quality journals is ISI-listed than in Economics and Management Science(Harzing & van der Wal, 2009).Second, citation-based performance metrics can be argued to be more democratic as their “verdict” is based on thereception of the paper by the academic community as a whole, whereas acceptance in a high-impact journal isdependent on only a handful of gatekeepers (the editor and reviewers).Third, our ranking was conducted with a free software program (Publish or Perish) and a publicly available database(Google Scholar). Hence, any reader can easily replicate the ranking without the need for subscription-baseddatabases. This also means that any academic can look up their own citation record and easily find out where theyscore in the current rankingFourth, citation-based rankings and in particular the hIa ranking are likely to provide more dynamic rankings interms of changes over the years. Younger academics can more easily enter into the hIa ranking if they perform wellrelative to their career length in terms of single-author equivalent impactful papers.As for other research metrics, the CAY or hIa-index should never be used as the sole criterion to evaluateacademics. Another crucial question that should always be asked is: “Has the scholar asked an important questionand investigated it in such a way that it has the potential to advance societal understanding and well-being?” (seee.g. Adler and Harzing, 2009). However, we argue that the hIa-index provides an important additional perspectiveover and above a ranking based purely on publications in high impact journals alone.Implications for Business schoolsUsing metrics such as the CAY and hIa, business schools can move away from the journal-based logic and startthinking from an impact -driven logic. This could have important consequences for incentive policies, mainly forsenior faculty. We advice deans of business schools to start assessing their senior faculty from an impactperspective, instead of the narrow journal-based assessment. This approach is not only more inclusive, it is alsomore democratic and gives more opportunity for diversity to flourish, especially in business schools with a diversesocial sciences background.Further details: Harzing, A.W.; Mijnhardt, W. Proof over promise: Towards a more inclusive ranking of Dutch3/4academics in Economics & Business, in press for Scientometrics, DOI: 10.1007/s11192-014-1370-z, prepublicationversion available at http://www.harzing.com/papers.htm#top40Note: This article gives the views of the author, and not the position of the Impact of Social Science blog, nor of theLondon School of Economics. Please review our Comments Policy if you have any concerns on posting a commentbelow.About the AuthorsAnne-Wil Harzing is Research Professor and Research Development Advisor at ESCP Europe and Professor ofInternational Management at Middlesex University. Her research interests include international HRM, HQ-subsidiaryrelationships, the role of language in international business, and the quality and impact of academic research. Since1999 she maintains an extensive website (www.harzing.com) with resources for international management andacademic publishing.Wilfred Mijnhardt is Policy Director at Rotterdam School of Management and Executive Director of the ErasmusResearch Institute of Management (ERIM) at Erasmus University Rotterdam, one of the prominent researchinstitutes in Research in Management in Europe. He publishes a weblog “Research Management in ManagementResearch” and shares his ideas on social media @wmijnhardtCopyright © The Author (or The Authors) - Unless otherwise stated, this work is licensed under a CreativeCommons Attribution Unported 3.0 License.4/4

Proof over promise: Moving citation metric systems beyond journal impact towards a career impact approach.

Abstract

Similar works

Full text

Available Versions

LSE Research Online