Conversations on impact tend to revolve around technical issue of measurement and finding appropriate metrics. To widen the conversation J. Britt Holbrook presents a list of 56 indicators of impact developed by the Center for the Study of Interdisciplinarity to help simplify the question of impact. By moving beyond technical aspects there is a greater opportunity for academics to embrace and explore other facets of impact

Holbrook, J. Britt

LSE Research Online

6/12/2013Developing indicators of the impact of scholarlycommunication is a massive technical challenge – but it’salso much simpler than thatblogs.lse.ac.uk /impactofsocialsciences/2013/06/12/56-indicators-of-impact/Conversations on impact tend to revolve around technical issue of measurement and finding appropriatemetrics. To widen the conversation J. Britt Holbrook presents a list of 56 indicators of impactdeveloped by the Center for the Study of Interdisciplinarity to help simplify the question of impact. Bymoving beyond technical aspects there is a greater opportunity for academics to embrace andexplore other facets of impact.Perhaps the most frequently asked question regarding impact is logistical: how can we measure theimpact of our work? A recent story in the Chronicle of Higher Education suggests: “The largerconversation about how to measure scholarly impact is probably as old as scholarship itself.” Today,the question ranges from the development of article level metrics to building a shared infrastructure for all ofscholarly communication.Why should we develop new ways to measure the impact of our work? Jason Priem et al. and Heather Piwowaranswer that current measures of impact don’t work; since traditional approaches to measuring scholarlycommunication don’t reward impact, we need a way to measure and reward other approaches that do. Similarviews are shared by some and contested by others.What’s most striking about answers to the ‘why’ question is how quickly they turn toward the ‘how’ question.Altmetrics developers are doers and inventors – they take action and try to figure things out. We can’t figure outimpact just by thinking about it; we have to do research, warns Piwowar in the post linked above. After noting thatthe system is broken, Priem and Piwowar quickly ask, “How can we fix it?” Instead of answering the ‘why’ question,we ask a different question: Do altmetrics work?Should we resist attempts to measure the impact of our work? Philip Moriarty offers sophisticated arguments againstthe ‘impact agenda’, while others are more demonstrative. Robert Frodeman and I argue that embracing impact in away that preserves autonomy is a better strategy than mere resistance. Despite contrary answers to the question ofresistance, many of us agree that not everything that counts can be counted. Our disagreement rests on differentconceptions of freedom. ‘The resistance’ tends toward a negative concept of freedom that sees all forms ofinterference as evils. Advocates of owning impact, however, embrace a positive view of freedom, emphasizing self-determination as fundamental.Infrequently asked questionsAlthough these frequently asked questions are interesting and important, it’s ultimately simpler than that. Colleaguesat the Center for the Study of Interdisciplinarity (CSID), Kelli Barr, Keith Brown, and I, submitted something simpleand quite messy to Nature to see whether they might join us in catalyzing a conversation on the question of impact.Frankly, we doubted it. Nevertheless, we submitted the following list, along with a brief discussion of how and whywe generated it.1/4Of course, the editors at Nature declined to publish it. Instead, they asked us to clean up the list to make it moregenerally applicable and less CSID-specific. We did so, and the officially published version of our correspondence isavailable here. It can also be viewed free of charge here. The editors of the LSE Impact blog had a similar urge toclean up our mess. This is the table they proposed:Comparing the original and edited versions illustrates the value of the simpler questions about impact I propose we2/4begin to ask more frequently. I realize that people have already been asking some of these questions – but notfrequently enough.What do we mean by ‘impact’?  Does it make a difference if we speak of ‘scholarly impact’ rather than ‘the impact ofscholarly communication’? Does ‘scholarly communication’ include teaching, as well as publishing (whether intraditional or alternative venues)? I’d be interested to develop an account of why we ought to revive one feature ofthe Humboldtian idea of the university: research and teaching as mutually reinforcing activities essential to what itmeans to be a professor. Our list includes both research and teaching activities, as well as activities that might fallunder the rubric of ‘outreach’ or policy engagement.Why do we use the term ‘impact’?! The impact horse has left the barn. But we ought to question what sorts ofactivities fall within its scope, even if we would prefer a different term. Claire Donovan’s reaction to CSID’s list ispriceless: “Only one on the list has anything to do with impact.” Before you click on this link to the twitter exchange,see if you can guess which one Claire had picked out. I couldn’t.No question could be simpler: if we want to have an impact, then who will/should be impacted?  The answer to thisquestion should inform and be informed by research on how to measure impact. But asking the question ofaudience takes us beyond thinking of the ‘how’ question as susceptible only to technical answers.Not all academics are interested in putting a number on their impact. The more frequently asked question ofmeasurement favors technical answers. Technical answers paradoxically foster expertise in impact measurementand take the question of impact out of the hands of most academics, while placing a ready-made answer into thehands of managers.  Technical answers, in other words, foster technocratic domination over the question of impact.Imagine if HEFCE or the US NIH merely had to grab and drag the Altmetric bookmarklet onto their bookmarks bars,visit someone’s CV to find an article, then ‘Altmetric it!’ to see its score. There’s a danger that push-button evaluationby non-experts could replace peer review. This could be the end of the data-driven impact story.Despite my hyperbole, I’m not arithmophobic. CSID’s list incorporates numbers, and we include things captured byaltmetrics. I love altmetrics and spend (too much) time ‘altmetricing’ my own products, using both Altmetric andImpact Story. But, along with the thrill that comes from knowing that people are paying attention to my work, whatmotivates me to explore altmetrics is a drive to write my own impact story. I don’t want ‘the data’ driving that story forme. Nor do I want someone else at the wheel. I want to tell my own story and to appeal to numbers when I decidethey can help correct my course or bolster my impact claims. Numbers themselves – or metrics – are not thedanger. The danger is numbers being arbitrarily imposed on us.This is the simplest, yet most difficult, question of all: where do we want impact to go? For now, we still have somesay in the matter. We might even be able to impact the question at a policy level. I’ve offered some of mysuggestions, above. I’m very interested to hear yours.Note: This article gives the views of the author, and not the position of the Impact of Social Science blog, nor of theLondon School of Economics.  About the AuthorJ. Britt Holbrook is Assistant Director of the Center for the Study of Interdisciplinarity at the University of NorthTexas (UNT), where he has served as Research Assistant Professor within the Department of Philosophy andReligion Studies since 2005. He has also held teaching positions in philosophy at Emory University and at GeorgiaState University. Holbrook’s current research focuses on interdisciplinarity, peer review, scholarly communication,and the relationship between science, technology, and society. He is especially interested in the incorporation ofsocietal impacts considerations into the peer review process of publicly supported funding agencies. Link to ORCIDiD. Follow him on twitter @jbrittholbrook.  3/4Copyright © The Author (or The Authors) - Unless otherwise stated, this work is licensed under a CreativeCommons Attribution Unported 3.0 License.4/4

Developing indicators of the impact of scholarly communication is a massive technical challenge – but it’s also much simpler than that

Article on developing indicators of the impact of scholarly communication and the technical challenges

Developing indicators of the impact of scholarly communication is a massive technical challenge – but it’s also much simpler than that

Abstract

Similar works

Full text

Available Versions

LSE Research Online

UNT Digital Library