Bruce, Ann

Lyall, Catherine

Marsden, Wendy

Meagher, L.

Tait, Joyce

Edinburgh Research Explorer

     Edinburgh Research Explorer                                      A Short Guide to Evaluating Interdisciplinary ResearchCitation for published version:Lyall, C, Tait, J, Meagher, L, Bruce, A & Marsden, W A Short Guide to Evaluating InterdisciplinaryResearch.Link:Link to publication record in Edinburgh Research ExplorerDocument Version:Publisher's PDF, also known as Version of recordPublisher Rights Statement:© Lyall, C., Tait, J., Meagher, L., Bruce, A., & Marsden, W. A Short Guide to Evaluating InterdisciplinaryResearchGeneral rightsCopyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s)and / or other copyright owners and it is a condition of accessing these publications that users recognise andabide by the legal requirements associated with these rights.Take down policyThe University of Edinburgh has made every reasonable effort to ensure that Edinburgh Research Explorercontent complies with UK legislation. If you believe that the public display of this file breaches copyright pleasecontact openaccess@ed.ac.uk providing details, and we will remove access to the work immediately andinvestigate your claim.Download date: 05. Apr. 2019 ISSTI Briefing Note (Number 9) March 2011 A Short Guide to Evaluating Interdisciplinary Research  Catherine Lyall1, Joyce Tait1, Laura Meagher2, Ann Bruce1 and Wendy Marsden1  Judging quality in interdisciplinarity..................................................................................................... 1 Improving interdisciplinary evaluation processes................................................................................ 2 A new vision for interdisciplinarity ....................................................................................................... 3 Evaluation plays a critical role in blocking or facilitating interdisciplinary research. Research funders play an important role in shaping investments and on their longer term impacts: the effective and appropriate evaluation of interdisciplinary investments is a key area where funders can provide strong leadership. This note offers practical suggestions for judging interdisciplinary work fairly, particularly when it is in competition with single discipline work. Improved evaluation criteria and processes are the key to achieving a more stable and consistent role for interdisciplinary research and for improving its intellectual status in academia. Sensitive evaluation of interdisciplinary research can also play a role in delivering improved value for money for the investments being made in this area. Evaluation of interdisciplinarity occurs in a variety of situations (e.g. review of grant proposals, manuscripts for publication, or end-of-research impact evaluation). The criteria appropriate to evaluation of academically-oriented interdisciplinary research may often be different from those used for problem-focused projects and programmes3. Whatever the evaluation situation, interdisciplinary work overall is done no favours if evaluation is not rigorous. However, achieving shared definitions of rigour and quality across a range of settings takes extra effort. The distinctiveness of the evaluation challenges posed by interdisciplinarity should be recognised, planned for, and accommodated. Judging quality in interdisciplinarity Peer review is an essential component of evaluation of discipline-based projects and must also be the cornerstone of evaluation of the quality of interdisciplinary research proposals. However, the criteria adopted by disciplines do not translate well across to interdisciplinary initiatives. Questions to ask when assessing interdisciplinary quality include: • Does the topic/problem posed require an interdisciplinary approach? • Does the topic/problem and approach lend itself to robust, high quality research? • Does the work show rigorous problem framing, data collection and analysis? • Is the work consonant with/grounded in its source disciplines/methodologies or is it likely to develop novel methodological approaches? • Has the work added or will it add to knowledge, albeit in a non-conventional way? The disciplines that serve as academic homes for most evaluators are subjected to a coherent set of discipline-specific, and agreed, criteria. This relative clarity may also                                                      1ESRC Innogen Centre, University of Edinburgh; 2 Technology Development Group; 3 For a discussion of these terms see Short Guide No. 1 (incorrectly, we and many others would argue) reinforce the convictions held by some that monodisciplinary work is necessarily more rigorous than interdisciplinary work. It also explains why discipline-based evaluators may find it less problematic to evaluate academically-oriented interdisciplinary research where the contribution of individual disciplines to an overall academic objective may be easier to specify and to accommodate within conventional frames of discipline-based thinking.   The lack of agreed indicators may be one reason why a question mark often hangs over the academic value of interdisciplinarity. This may leave evaluators in the uncomfortable position of judging something that is, in part, unknowable through their own expertise. We maintain that it is indeed possible to assess quality of interdisciplinary research, through appropriate criteria and processes. The following quality criteria are relevant for both academically-oriented and problem focused interdisciplinary initiatives: Quality criteria for interdisciplinary research • The proposal should indicate the expected synergistic outcomes from the combination of disciplines/approaches, the likely benefits for disciplines (in the case of academically-oriented interdisciplinary research) or the societal or business benefits (in the case of problem focused interdisciplinary research). Elements of both may be incorporated in the same project.  • Do not expect a problem-focused interdisciplinary initiative to contribute to enhancement of the knowledge base of any of the individual disciplines involved. A single project is unlikely to deliver discipline-related breakthroughs as well as the other synergistic benefits of integrating disciplines. To expect to find both in a single proposal is to place unrealistic demands on the researchers. • Look for a good understanding of the disciplines involved, and of their limitations, and a clear justification for the choice of disciplines based on the needs of the research questions. • Look for evidence that the researcher or the research team have understood the challenges of interdisciplinary integration, including methodological integration, and the ‘human’ side of fostering interactions and communication, and have developed an effective strategy to deal with this. • More than for monodisciplinary projects, interdisciplinary ones may need to develop and change as they proceed. The proposal might therefore set out a flexible timetable: while the end goal should be clear, the routes to achieving it might be subject to revision as the project progresses. • In evaluating published outcomes of interdisciplinary research do not include journal prestige or citation patterns as criteria as both actively disadvantage interdisciplinary research outputs. • In evaluating researchers, links to excellent discipline-based research can be an advantage, but much more important is evidence of past success in conducting or leading interdisciplinary research. Where young, inexperienced researchers are involved, an integrative mind-set is important and this can often be judged from the style of writing. The kind of focused mind-set that can excel in a discipline-based context can be a disadvantage for interdisciplinary research.  • Well before the event, make it clear to those being evaluated the quality criteria by which their work will be judged and encourage them to explain why the proposed research needs to be interdisciplinary; what disciplines are involved and why; how they will be integrated, and how the quality of the interdisciplinary outcomes might be assessed.  Improving interdisciplinary evaluation processes The choice of evaluators, their disciplinary and interdisciplinary backgrounds, and their roles in the evaluation process need to be considered carefully. Interdisciplinary researchers often lack a fixed peer community which means that researchers who are not well known to referees may be disadvantaged by the review process. The problem is acute for proposals that are trying to put forward a novel interdisciplinary project where there may not be a recognised set of peers who are individually qualified to referee it.  It is common for interdisciplinary review panels to tackle this problem by including an appropriate range of disciplinary experts and one or two token interdisciplinary reviewers. By definition, any one reviewer is unlikely to encompass appreciation for the entirety of the ‘package’ put forward in an interdisciplinary proposal. Furthermore, individuals affiliated with different disciplines may weight various factors differently. Challenges in handling disparate input into the review panel process can damage the chances of interdisciplinary proposals. For example, one of the authors was involved in a review panel for a UK research council and was the only interdisciplinary expert on the panel. As lead evaluator for an interdisciplinary proposal she judged that it met all the quality criteria outlined in the previous section and met the requirements of the call and should be funded. However, each of the disciplinary experts on the panel counselled rejection because the project, although competent in their respective disciplines, did not contribute to their advancement. The numerical weight of these comments (perhaps reinforced by the disciplinary prestige of the commentators) prevailed and the project was rejected. This kind of outcome can be avoided by giving clear guidance to panels as to how they should weight disciplinary and interdisciplinary contributions and by ensuring that the panel chair is alert to these biases.  Panels evaluating problem-focused interdisciplinary initiatives often also include non-academic evaluators representing policy, business or citizen stakeholders and this can be seen as contributing to the objectivity of the evaluation process since they do not share professional networks with the applicants. However, these non-academic experts are also usually in a minority and they are often allocated specific roles in the process, for example to judge the quality of the proposed stakeholder engagement. In our experience, they generally defer to academic evaluators over questions of research quality and so have a limited influence on the overall grade allocated to an interdisciplinary initiative. The process of finding appropriate peers to review interdisciplinary work is thus a frequently cited challenge for those managing evaluations and often a source of deep frustration for interdisciplinary researchers subjecting themselves to such judgment (as well as for interdisciplinary evaluators appointed as token members of a predominantly discipline-based team). Clearly, one issue is that of expertise – what range of skills and experience should be represented on a review panel, or among individual evaluators. Another issue is the process by which consensus about the quality of interdisciplinary proposals is achieved. The management of the process of evaluation is critically important, with informed staff giving clear guidance to panels on how to evaluate interdisciplinary initiatives and appointing a panel chair with a good understanding of what is required and a strong enough control over the process to ensure that the guidance is followed.  The following guidance builds on the insights of others and our own experience: Tips for effective interdisciplinary review panels For those managing interdisciplinary peer review processes • The make-up of an evaluation panel is probably the most important factor in ensuring maximum potential fairness in the process, so that better quality interdisciplinary projects are funded and poorer ones rejected. The choice of panel members will depend on the context of the evaluation: a set of individual small-scale projects or a major interdisciplinary programme; ex ante or ex post evaluation; academically-oriented or problem-focused interdisciplinary research. • Discipline-based experts should be chosen on the basis of the breadth of their disciplinary understanding rather than their expertise (no matter how prestigious) in one narrow area.  • In all cases it is desirable to have at least one third of the panel members with a successful track record in interdisciplinary research. • Ensure that evaluation panel members are provided with guidance on how to evaluate interdisciplinary proposals, including clearly specified criteria, as outlined above. The panel should also be advised on the processes to be adopted, including how to deal with disagreements on the value of different disciplinary contributions and what weight to give to disciplinary contributions in relation to overall interdisciplinary quality. • The role of a panel chair will be crucial in ensuring that any such guidelines are implemented by the panel, and not sidelined in favour of traditional disciplinary criteria as is so often the case. For individual reviewers • Consider personal biases and the implications this might have for evaluation  • Be willing to engage in dialogue and respond to others’ views.  A new vision for interdisciplinarity Research funding organisations have invested major sums of money in academically-oriented interdisciplinary research with a view to stimulating the emergence of new interdisciplinary research areas in subjects such as nanotechnology and synthetic biology. For problem-focused interdisciplinary research, this has taken the form of encouraging individuals, teams or new centres to invest their careers and their organisation’s resources in contemporary, complex issues important to society.  Experience with these major programmes and other funding initiatives dedicated explicitly to interdisciplinary research may help to shape research evaluation but, with an increasing number of initiatives that cut across the remits of individual research councils in the UK, a new vision is required to promote organisational learning for interdisciplinarity. Research Councils UK might consider:  • the establishment of an interdisciplinary reviewers’ college (consisting of individuals expert in a range of interdisciplinary areas) to address the common challenge of finding reviewers who are sympathetic to interdisciplinary research and understand how to evaluate it both rigorously and appropriately • establishing shared administrative resources for interdisciplinary investments with dedicated administrators experienced in the particular requirements of interdisciplinary research and research training • facilitating the development of a cadre of early career and more senior interdisciplinary researchers by hosting community-building events across different interdisciplinary capacity-building schemes and investments. An Interdisciplinary Funders Forum similar to the UK Strategic Forum for the Social Sciences  or the former Environmental Research Funders Forum could promote shared learning  • developing an Interdisciplinary Portal analogous to the current RCUK Knowledge Transfer Portal to co-ordinate and consolidate access by the research community to information about funding, training and other forms of support dedicated to interdisciplinarity and its evaluation.   This Short Guide draws on a chapter from our book Interdisciplinary Research Journeys. Practical Strategies for Capturing Creativity (Bloomsbury Academic, 2011) and a recent study for the Natural Environment Research Council ‘QUEST: Capturing Lessons for Interdisciplinarity’ (NERC grant reference: NE/H012001/1).  Other notes in this series can be downloaded from www.tinyurl.com/idwiki   For further information contact c.lyall@ed.ac.uk                                    www.tinyurl.com/idwiki  

https://www.pure.ed.ac.uk/ws/files/12623490/A_Short_Guide_to_Evaluating_Interdisciplinary_Research.pdf

A Short Guide to Evaluating Interdisciplinary Research

Abstract

Similar works

Full text

Available Versions

Edinburgh Research Explorer