This paper proposes the creation of a dashboard consisting of five metrics that could be used to replace the journal impact factor.  It should be especially useful in circumstances, like promotion and tenure committees, where the evaluators do not share the authors subject expertise and where they are working under time constraints

Lewis, David W.

IUPUIScholarWorks

 1 Proposal for a Standard Article Metrics Dashboard to Replace the Journal Impact Factor  David W. Lewis July 2019   © 2019 David W. Lewis. This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.   Introduction  As I noted in an earlier working paper, the journal impact factor, though it is widely understood to be deeply flawed, remains in use because it is a simple way on passing judgement on the quality of a recently published article.1  This is especially true in the cases, like promotion and tenure committees where the individuals passing judgement have no expertise in the subject the article speaks to and are pressed for time.  In this paper I will propose a simple set of metrics, displayed in a dashboard, to be used to evaluate articles in contexts like promotion and tenure committees.    Metrics  There are many metrics to choose from.2  I would propose the following metrics: 1. Availability 2. Citations 3. Downloads 4. Altmetric Attention Score or Impactstory blog counts                                                1 David W. Lewis, “Why Do P&T Committees Keep Hiring the Journal Impact Factor?” June 2019, http://hdl.handle.net/1805/19707 2 See for example: Metrics Toolkit, developed by Robin Champieux, Heather Coates and Stacy Konkiel, https://www.metrics-toolkit.org    2 5. Journal Impact Factor vs Article Performance  Availability.  This is not a performance metric per se, rather, it is an indication of how freely and easily the article can be read.  There would be four possible conditions. 1. Gold — the version of record was published in open access journal or the article was open at the time of publication in a hybrid journal. 2. Green Open — The article was published in a paywalled journal, but the version of record is available in a disciplinary or institutional repository. 3. Light Green — The article was published in a paywalled journal, but an authoritative version, but not the version of record, is available in a repository.  In addition to the light green indicator, the price of the article from the publisher will be listed. 4. Purchase Only — The article is available only by purchase.  The price of the article from the publisher will be listed. The options might be show as follows:             Citations.  Citations would be taken from Google Scholar, Web of Science and SCOPUS.  The display might look something like this:  Google Scholar 188 Web of Science 60 SCOPUS 71   Downloads.  Downloads would be taken from the publisher.   Altmetric Attention Score.  Although the Altmetric Attention Score is not without issues, it is a generally sound and consistent message on the attention and article has received.  A supplement or alternative would be to take the Impactstory blog and Google+ mentions.  The display would be:  $37.50 $41.95  3          Journal Impact Factor vs Article Performance.  It might seem curious to include journal impact factor as a metric to evaluated articles.  This inclusion has two purposes.  First, it is familiar.  The purpose here is educational. Its inclusion will along with other metrics will provide context and hopefully show that the journal impact factor is not always as useful as it might seem.  I would not use the journal impact factor per se.  Rather the metric would be the number of citations to an article in the first two years after publication divided by the journal impact factor in the year the article was published.  This would show the extent to which an article outperformed or underperformed other articles published in that journal in that year.  So, if the journal impact factor in the year of publication was 1.016 and the article received 15 citations in the following two years, it would have significantly outperformed the other articles published that year in that journal being cited14.764 times more than the average article (in the spirit of the JIF this number would be carried out to the third decimal, hopefully showing the absurdity of three decimals in the JIF).  The variance in these number would expose one of the flaws of the straight journal impact factor.  It could be display as follows:  Citations 15 14.764 JIF 1.016   Example  So, the result would end up looking something like the following (since some of the metrics change it would need to be dated):  David W. Lewis, “The Inevitability of Open Access,” College & Research Libraries 73(5):493-506 September 2012, DOI: https://doi.org/10.5860/crl-299                                                                                                                            July 5, 2019   4  Or,  David W. Lewis, “From Stacks to the Web: The Transformation of Academic Library Collecting,” College & Research Libraries 74(2):159-176 March 2013, DOI: https://doi.org/10.5860/crl-309      July 5, 2019   How It Would Work   The proposed system would have absolutely no chance of working if faculty preparing vitas or dossiers had to compiled the data to create this metrics dashboard for each article.  Therefore, if the system is to work it will require a service that would create the article dashboard simply and cheaply.  Such a service would not seem that difficult to create.  A DOI or similar identifier should be able to be used to pull the required data, which all comes from standard sources — Unpaywall or 1Science, Google Scholar, Clarivate, Altmetrics, and Impactstory — and should be straight forward to extract.  The download statistics from publishers would likely be the most difficult to get comprehensively in an automated way.     The service provider could charge individuals a nominal fee, say 10 cents an article, or to an institution for an annual fee that then could be used by all use by associated individuals.  An free open service would be even better.  One could imagine it tied into annual reporting or CRIS systems.   Conclusion  The journal impact factor, while broadly understood to be a deeply flawed measure, will continue to be used because it gives some sense of the potential influence of a recently published article.  It is easy to find and provides a simple, if inappropriate, means of comparison.  The mix of metrics proposed here, if they can be collected and displayed simply and cheaply, provides much more context and a much richer sense of impact than the journal impact factor.  It also can be scanned and absorbed 0  5 without undue effort, which is necessary if we expect it to be used by evaluators who need to review large quantities of information in a limited amount of time.  I would hope someone takes on the task of making this, or something like it, real. 

Proposal for a Standard Article Metrics Dashboard to Replace the Journal Impact Factor

https://scholarworks.iupui.edu/bitstream/1805/19850/1/Proposal%20for%20a%20Standard%20Article%20Metrics%20Dashboard%20to%20Replace%20the%20Journal%20Impact%20Factor.pdf

Proposal for a Standard Article Metrics Dashboard to Replace the Journal Impact Factor

Abstract

Similar works

Full text

Available Versions

IUPUIScholarWorks