The craft of evaluative practice: Negotiating legitimate methodologies within complex interventions

Evaluations of complex interventions are likely to encounter tensions between different methodological principles, and between the inherent causal rationality of evaluation and the messy complexity of real institutional contexts. Conceptualizing evaluation as producing putatively authoritative evidence, we show how ‘legitimacy’ is a useful concept for unpacking evaluation design in practice. A case study of service integration shows how different approaches may have unpredictable levels of legitimacy, based in contrasting assessments of their methodological acceptability and actual utility. Through showing how practitioners resolved the tensions, we suggest that crafting a patchwork of different methodologies may be legitimate and effective, and can be seen as underpinned by its own pragmatic rationality. However, we also conclude that the explanatory power of theory-driven evaluation can be embedded in such an approach, both in elements of the patchwork and as an overarching guiding principle for the crafting process

Translating research for policy: the importance of equivalence, function, and loyalty

The question of how to make academic research more useful to government, and frustration over its lack of obvious use, have long been the subject of policy makers’ and scholars’ attention. These have driven the global development of institutionalised links between the two communities, while also leading to a broad consensus as to why the goal is often not realised. In order to better explain the barriers, this paper takes the concept of “translation” very literally, and proposes an innovative approach, which analyses academic and policy practices using ideas from the humanities-based discipline of Translation Studies. This enables an exploration of what constitutes good translation, and in particular of the tension between keeping faith with the original material and users’ understandable emphasis on functionality. The conclusion is that while some aspect of original research content must be maintained, what this is cannot be prescribed: the appropriate equivalence between original and translation is always context-dependent. This throws the emphasis on the relational aspects of translatorial action for promoting “good translation”. The argument follows Christiane Nord in seeing the core issue as the moral one of a translator’s loyalty to original author and user, and so also of mutual trust between academics and civil servants. This raises important questions about how such trust can be cultivated, and so finally leads to an emphasis on the importance of an endeavour shared by researchers and policy makers, which recognises and respects their different environments and the work involved in creating useful meaning from scholarly research

Everyday stories of impact: interpreting knowledge exchange in the contemporary university

Research into the barriers of getting evidence produced by academics into policymaking processes has often highlighted the lack of research on academics and what they do, as compared to what policymakers do. This was most recently highlighted in a systematic review of the literature (Oliver et al, 2014). This paper reports on research carried out with academics who were tasked with producing evidence reviews for the UK Department for Communities and Local Government based on research funded by the UK Arts and Humanities Research Council. Using a novel co-produced methodology the academics were interviewed by an academic and a UK civil servant, with the analysis carried out by both. Using an interpretive approach, the findings identify specific meaning-making stories or practices that were enablers or barriers to producing evidence suitable for policymakers. The paper identifies three areas that affect academic behaviour at the nexus with policymaking: career biographies; disciplinary background; and the contradictory institutional pressures on academics. We conclude by arguing for a more collaborative approach between academics and policymakers. The co-produced approach also allowed us to identify the need for policymakers and civil servants to learn more about the different drivers of academics and the ways in which they work

Using structured eradication feasibility assessment to prioritize the management of new and emerging invasive alien species in Europe

Prioritizing the management of invasive alien species (IAS) is of global importance and within Europe integral to the EU IAS regulation. To prioritize management effectively, the risks posed by IAS need to be assessed, but so too does the feasibility of their management. While the risk of IAS to the EU has been assessed, the feasibility of management has not. We assessed the feasibility of eradicating 60 new (not yet established) and 35 emerging (established with limited distribution) species that pose a threat to the EU, as identified by horizon scanning. The assessment was carried out by 34 experts in invasion management from across Europe, applying the Non‐Native Risk Management scheme to defined invasion scenarios and eradication strategies for each species, assessing the feasibility of eradication using seven key risk management criteria. Management priorities were identified by combining scores for risk (derived from horizon scanning) and feasibility of eradication. The results show eradication feasibility score and risk score were not correlated, indicating that risk management criteria evaluate different information than risk assessment. In all, 17 new species were identified as particularly high priorities for eradication should they establish in the future, whereas 14 emerging species were identified as priorities for eradication now. A number of species considered highest priority for eradication were terrestrial vertebrates, a group that has been the focus of a number of eradication attempts in Europe. However, eradication priorities also included a diverse range of other taxa (plants, invertebrates and fish) suggesting there is scope to broaden the taxonomic range of attempted eradication in Europe. We demonstrate that broad scale structured assessments of management feasibility can help prioritize IAS for management. Such frameworks are needed to support evidence‐based decision‐making

Translation for Policy

This report summarises the learning from a project funded by the AHRC which worked at the interface between central government policy and academic research. The project’s primary purpose was to enhance the policy impact of the Connected Communities (CC) programme, both directly and by exploring ways to overcome barriers to impact. It started from the assumption that collectively the CC projects have generated much policy-relevant knowledge, including from projects which did not have an explicit focus on or interest in policy

Translation across borders: Connecting the academic and policy communities

In this chapter we look at the legacy of three projects which connected the research and policy communities, through the development of ‘policy briefs’ for the UK Department of Communities and Local Government (DCLG). These were to be short and accessible reviews of research – in particular from the arts and humanities - relevant to policy on localism. Starting from an understanding of policy-making as meaning-making, and of translation as situated and purposeful action, we explored through ethnographic and action research how academics and government analysts translate research into ideas useful for policy makers.  We conclude that the legacy of researching for policy can be understood not just in terms of ‘things left behind’ and their direct impact on policy, but also more broadly in terms of the purposes of all those involved being met, and influences on academic and civil service norms and subsequent practice. Co-production is central to leaving such a legacy, in particular to break down mutual misunderstanding across the policy/academia border and to enable research to be used. In contrast interdisciplinarity seems less important, though broadening the disciplinary base of research used by government is certainly valuable. Underpinning everything else, the development of relationships of trust through collaboration and mutual learning is paramount