A systems approach for optimizing implementation to impact: meeting report and proceedings of the 2019 In the Trenches: Implementation to Impact International Summit

Background The In the Trenches series of cutting-edge knowledge sharing events on impact for front-line experts and practitioners provides an engagement platform for diverse stakeholders across government, research funding organizations, industry, and academia to share emerging knowledge and practical experiences. The second event of the series In the Trenches: Implementation to Impact International Summit was held in Banff, Alberta, Canada, on June 7–8, 2019. The overarching vision for the Summit was to create an engagement platform for addressing key challenges and finding practical solutions to move from implementation (i.e. putting findings into effect) to impact (i.e. creating benefits to society and the economy). Processes and proceedings The Summit used diverse approaches to facilitate active engagement and knowledge sharing between 80 delegates across sectors and jurisdictions. Summit sessions mostly consisted of short talks and moderated panels grouped into eight thematic sessions. Each presentation included a summary of Key Messages, along with a summary of the Actionable Insights which concluded each session. The presentations and discussions are analysed, synthesized and described in this proceedings paper using a systems approach. This demonstrates how the Summit focused on each of the necessary functions (and associated components) that should be undertaken, and combined, for effective research and innovation: stewardship and governance, securing finance, creating capacity, and producing and using research. The approach also identifies relevant challenges. Conclusions There is increased interest globally in the benefits that can accrue from adopting a systems approach to research and innovation. Various organizations in Canada and internationally have made considerable progress on Implementation to Impact, often as a result of well-planned initiatives. The Summit highlights the value of 1) collaboration between researchers and potential users, and 2) the adoption by funders of approaches involving an increasing range of responsibilities and activities. The Summit website (https://inthetrenchessummit.com/) will be periodically updated with new resources and information about future In the Trenches events.Alberta Innovates; European Union’s Horizon 2020 research and innovation programme award STARBIOS2 (grant agreement No. 709517); National Institute for Health Research (grant NIHR Oxford Biomedical Research Centre (BRC-1215-20008) to the Oxford University Hospitals NHS Foundation Trust and the University of Oxford)

Assessing Health Research and Innovation Impact: Evolution of a Framework and Tools in Alberta, Canada

Publicly funded research and innovation (R&I) organizations around the world are facing increasing demands to demonstrate the impacts of their investments. In most cases, these demands are shifting from academically based outputs to impacts that benefit society. Funders and other organizations are grappling to understand and demonstrate how their investments and activities are achieving impact. This is compounded with challenges that are inherent to impact assessment, such as having an agreed understanding of impact, the time lag from research to impact, establishing attribution and contribution, and consideration of diverse stakeholder needs and values. In response, many organizations are implementing frameworks and using web-based tools to track and assess academic and societal impact. This conceptual analysis begins with an overview of international research impact frameworks and emerging tools that are used by an increasing number of public R&I funders to demonstrate the value of their investments. From concept to real-world, this paper illustrates how one organization, Alberta Innovates, used the Canadian Academy of Health Sciences (CAHS) impact framework to guide implementation of its fit-for-purpose impact framework with an agnostic international six-block protocol. The implementation of the impact framework at Alberta Innovates is also supported by adopting emerging web-based tools. Drawing on the lessons learned from this continuous organizational endeavor to assess and measure R&I impact, we present preliminary plans for developing an impact strategy for Alberta Innovates that can be applied across sectors, including energy, environment and agriculture, and may possibly be adopted by other international funders

ISRIA statement: ten-point guidelines for an effective process of research impact assessment

As governments, funding agencies and research organisations worldwide seek to maximise both the financial and non-financial returns on investment in research, the way the research process is organised and funded is becoming increasingly under scrutiny. There are growing demands and aspirations to measure research impact (beyond academic publications), to understand how science works, and to optimise its societal and economic impact. In response, a multidisciplinary practice called research impact assessment is rapidly developing. Given that the practice is still in its formative stage, systematised recommendations or accepted standards for practitioners (such as funders and those responsible for managing research projects) across countries or disciplines to guide research impact assessment are not yet available. In this statement, we propose initial guidelines for a rigorous and effective process of research impact assessment applicable to all research disciplines and oriented towards practice. This statement systematises expert knowledge and practitioner experience from designing and delivering the International School on Research Impact Assessment (ISRIA). It brings together insights from over 450 experts and practitioners from 34 countries, who participated in the school during its 5-year run (from 2013 to 2017) and shares a set of core values from the school’s learning programme. These insights are distilled into ten-point guidelines, which relate to (1) context, (2) purpose, (3) stakeholders’ needs, (4) stakeholder engagement, (5) conceptual frameworks, (6) methods and data sources, (7) indicators and metrics, (8) ethics and conflicts of interest, (9) communication, and (10) community of practice. The guidelines can help practitioners improve and standardise the process of research impact assessment, but they are by no means exhaustive and require evaluation and continuous improvement. The prima facie effectiveness of the guidelines is based on the systematised expert and practitioner knowledge of the school’s faculty and participants derived from their practical experience and research evidence. The current knowledge base has gaps in terms of the geographical and scientific discipline as well as stakeholder coverage and representation. The guidelines can be further strengthened through evaluation and continuous improvement by the global research impact assessment community

ISRIA statement: ten-point guidelines for an effective process of research impact assessment

As governments, funding agencies and research organisations worldwide seek to maximise both the financial and non-financial returns on investment in research, the way the research process is organised and funded is becoming increasingly under scrutiny. There are growing demands and aspirations to measure research impact (beyond academic publications), to understand how science works, and to optimise its societal and economic impact. In response, a multidisciplinary practice called research impact assessment is rapidly developing. Given that the practice is still in its formative stage, systematised recommendations or accepted standards for practitioners (such as funders and those responsible for managing research projects) across countries or disciplines to guide research impact assessment are not yet available. In this statement, we propose initial guidelines for a rigorous and effective process of research impact assessment applicable to all research disciplines and oriented towards practice. This statement systematises expert knowledge and practitioner experience from designing and delivering the International School on Research Impact Assessment (ISRIA). It brings together insights from over 450 experts and practitioners from 34 countries, who participated in the school during its 5-year run (from 2013 to 2017) and shares a set of core values from the school’s learning programme. These insights are distilled into ten-point guidelines, which relate to (1) context, (2) purpose, (3) stakeholders’ needs, (4) stakeholder engagement, (5) conceptual frameworks, (6) methods and data sources, (7) indicators and metrics, (8) ethics and conflicts of interest, (9) communication, and (10) community of practice. The guidelines can help practitioners improve and standardise the process of research impact assessment, but they are by no means exhaustive and require evaluation and continuous improvement. The prima facie effectiveness of the guidelines is based on the systematised expert and practitioner knowledge of the school’s faculty and participants derived from their practical experience and research evidence. The current knowledge base has gaps in terms of the geographical and scientific discipline as well as stakeholder coverage and representation. The guidelines can be further strengthened through evaluation and continuous improvement by the global research impact assessment community