What does COVID-19 mean for the evaluation of the impact criterion in REF2021?

The concept of research impact represents, to a degree, a formal way of understanding the productive relationships forged between academic research and the wider world. Unsurprisingly, these relationships took on entirely new dimensions as the COVID-19 pandemic changed the world. In this post, Gemma Derrick and Julie Bayley consider how COVID-19 is potentially influencing the ongoing 2021 Research Excellence Framework cycle and its analysis of UK research impact. Pointing to the ways in which the pandemic may have differentially influenced research and impact activities, they suggest this cycle could also hold lessons for the future in relation to longstanding inequalities within the REF

When it comes to gender inequality in academia, we know more than what can be measured

In academia gender bias is often figured in terms of research productivity and differentials surrounding the academic work of men and women. Alesia Zuccala and Gemma Derrick posit that this outlook inherently ignores a wider set of variables impacting women, and that attempts to achieve cultural change in academia can only be realised, by acknowledging variables that are ultimately difficult to quantify

The evaluation scale:exploring decisions about societal impact in peer review panels

Realising the societal gains from publicly funded health and medical research requires a model for a reflexive evaluation precedent for the societal impact of research. This research explores UK Research Excellence Framework evaluators’ values and opinions and assessing societal impact, prior to the assessment taking place. Specifically, we discuss the characteristics of two different impact assessment extremes – the “quality-focused” evaluation and “societal impact-focused” evaluation. We show the wide range of evaluator views about impact, and that these views could be conceptually reflected in a range of different positions along a conceptual evaluation scale. We describe the characteristics of these extremes in detail, and discuss the different beliefs evaluators had which could influence where they positioned themselves along the scale. These decisions, we argue, when considered together, form a dominant definition of societal impact that influences the direction of its evaluation by the panel

The Corona-Eye: Exploring the risks of COVID-19 on fair assessments of impact for REF 2021

This paper assesses the risk of two COVID-19 related changes necessary for the expert-review of the REF2021’s Impact criterion: the move from F2F to virtual deliberation; and the changing research landscape caused by the COVID-19 crisis requiring an extension of deadlines, and accommodation of COVID-19 related mitigation. Peer review in its basic form requires expert debate, where dissenting opinions and non-verbal cues are absorbed into a groups deliberative practice and therefore inform outcomes. With a move to deliberations in virtual settings, the most likely current outcome for REF2021 evaluations, the extent that negotiation dynamics necessary in F2F evaluations are diminished and how this limits panelists’ ability to sensitively assess COVID-19 mitigation statements is questioned. This article explores the nature of, and associated capabilities to undertake, complex decision making in virtual settings around the Impact criterion as well the consequences of COVID-19 on normal Impact trajectories. It examines the risks these changes present for evaluation of the Impact criterion and provides recommendations to offset these risks to enhance discussion and safeguard the legitimacy of evaluation outcomes. This paper is also relevant for evaluation processes of academic criteria that require both a shift to virtual, and/or guidance of how to sensitively assess the effect of COVID-19 on narratives of individual, group or organisational performance

The Association between Four Citation Metrics and Peer Rankings of Research Influence of Australian Researchers in Six Fields of Public Health

Doubt about the relevance, appropriateness and transparency of peer review has promoted the use of citation metrics as a viable adjunct or alternative in the assessment of research impact. It is also commonly acknowledged that research metrics will not replace peer review unless they are shown to correspond with the assessment of peers. This paper evaluates the relationship between researchers' influence as evaluated by their peers and various citation metrics representing different aspects of research output in 6 fields of public health in Australia. For four fields, the results showed a modest positive correlation between different research metrics and peer assessments of research influence. However, for two fields, tobacco and injury, negative or no correlations were found. This suggests a peer understanding of research influence within these fields differed from visibility in the mainstream, peer-reviewed scientific literature. This research therefore recommends the use of both peer review and metrics in a combined approach in assessing research influence. Future research evaluation frameworks intent on incorporating metrics should first analyse each field closely to determine what measures of research influence are valued highly by members of that research community. This will aid the development of comprehensive and relevant frameworks with which to fairly and transparently distribute research funds or approve promotion applications

Towards characterising negative impact:Introducing Grimpact

This paper categorises the nature of what we have labelled as the potential for ‘GrImpact’, in the evaluation of the wider influence of research, beyond academia. As the impact agenda broadly defined grows to include more formally criteria that consider the value of research beyond academia, so too does the pressure to ensure that these assessments of public value are conducted with the public’s best interests in mind. In many cases, any negative impact from research cannot be foreseen at the time of the evaluation, making it vital that any kinds of rewards for impact stimulate only positive public benefits (aka “the right type” of impact). Using a series of case studies of identified “negative impact” this paper explores the concept of Grimpact, as well as creates typologies of its characteristics and precursors

From “a fair game” to “a form of covert research”:Research Ethics Committee Members differing notions of consent and potential risk to participants within social media research.

Social media (SM) research presents new challenges for research ethics committees (RECs) who must balance familiar ethical principles with new notions of public availability. This article qualitatively examines how U.K. REC members view this balance in terms of risk and consent. While it found significant variance overall, there were discernible experience-based trends. REC members with less experience of reviewing SM held inflexible notions of consent and risk that could be categorized as either relying on traditional notions of requiring direct consent, or viewing publicly available data as “fair game.” More experienced REC members took a more nuanced approach to data use and consent. We conclude that the more nuanced approach should be best practice during ethical review of SM research

Reaching “an audience that you would never dream of speaking to”: influential public health researchers’ views on the role of news media in influencing policy and public understanding.

While governments and academic institutions urge researchers to engage with news media, traditional academic values of public disengagement have inhibited many from giving high priority to media activity. In this interview-based study, we report on the views about news media engagement and strategies used by 36 peer-voted leading Australian public health researchers in six fields. We consider their views about the role and importance of media in influencing policy; their reflections on effective or ineffective media communicators; and strategies used by these researchers about how to best retain their credibility and influence while engaging with the news media. A willingness and capacity to engage with the mass media was seen as an essential attribute of influential public health researchers.NHMR

Reaching “an audience that you would never dream of speaking to”: influential public health researchers’ views on the role of news media in influencing policy and public understanding.

While governments and academic institutions urge researchers to engage with news media, traditional academic values of public disengagement have inhibited many from giving high priority to media activity. In this interview-based study, we report on the views about news media engagement and strategies used by 36 peer-voted leading Australian public health researchers in six fields. We consider their views about the role and importance of media in influencing policy; their reflections on effective or ineffective media communicators; and strategies used by these researchers about how to best retain their credibility and influence while engaging with the news media. A willingness and capacity to engage with the mass media was seen as an essential attribute of influential public health researchers.NHMR