Evaluating the 'return on patient engagement initiatives' in medicines research and development: A literature review

Search strategy and inclusion criteria. We undertook a scoping literature review using a systematic search, including academic and grey literature with a focus on evaluation approaches or outcomes associated with patient engagement. No date limits were applied other than a cut-off of publications after July 2018. Data extraction and synthesis. Data were extracted from 91 publications, coded and thematically analysed. Main results. A total of 18 benefits and 5 costs of patient engagement were identified, mapped with 28 possible indicators for their evaluation. Several quantitative and qualitative methods were found for evaluation of benefits and costs of patient engagement. Discussion and conclusions. Currently available indicators and methods are of some use in measuring impact but are not sufficient to understand the pathway to impact, nor whether interaction between researchers and patients leads to change. We suggest that the impacts of patient engagement can best be determined not by applying single indicators, but a coherent set of measures

Abstracts from the NIHR INVOLVE Conference 2017

n/

Recruiting patients as partners in health research: a qualitative descriptive study

Plain English summary Increasingly, funders and researchers want to partner with patients in health research, but it can be challenging for researchers to find patient partners. More than taking part in research as participants, patient partners help design, carry out and manage research projects. The goal of this study was to describe ways that patient partners have been recruited by researchers and patient engagement leads (individuals within organizations responsible for promoting and supporting patients as research partners). We talked with researchers and patient engagement leads in Canada and the United Kingdom, as well as a patient representative. We found three ways that could help researchers and patients find each other. One way is a case-by-case basis, where patients are often sought with experience of a health condition that is the focus of the research. The other ways involved directories where projects were posted and could be found by patients and researchers, or a third party matched patients with research projects. We found four recruitment strategies: 1. Social marketing 2. Community outreach 3. Health system 4. Partnering with other organizations (e.g., advocacy groups) There are many influences on finding, selecting and retaining patient partners: patient characteristics, the local setting, the opportunity, work climate, education and support. We hope study results will provide a useful starting point for research teams in recruiting their patient partners. Abstract Background Patient engagement in clinical trials and other health research continues to gain momentum. While the benefits of patient engagement in research are emerging, relatively little is known about recruiting patients as research partners. The purpose of this study was to describe recruitment strategies and models of recruiting patients as partners in health research. Methods Qualitative descriptive study. Thirteen patient engagement leads and health researchers from Canada and the United Kingdom, as well as one patient representative from a national patient organization (7 female) completed semi-structured interviews. Results Recruitment infrastructures available to respondents varied, but could be categorized into three models including the traditional, third-party and directory models. Four categories of recruitment strategies were identified, representing multiple ways of recruiting patient partners: social marketing recruitment, community outreach recruitment, health system recruitment, and partnering recruitment. Conclusions Multiple recruitment strategies were identified for engaging patient partners in research, and some common factors influenced recruitment. Study findings contribute to the evidence base in patient engagement and provide guidance for research teams to help identify potential recruitment methods for their patient partners

Determining the ‘value’ of patient engagement

“What is in it for me?” A question often asked by people involved in patient engagement practices. Evaluating whether engagement adds value for different stakeholder groups can be an effective tool to further support patient engagement. The PARADIGM project is a European public-private partnership that is developing ways to ensure that patients are meaningfully involved in medicines development. One of the aims of the project is to develop metrics to measure the “return on engagement”. To contribute to this system innovation, the PARADIGM workstream co-led by MSD and the Athena Institute (VU Amsterdam) aims to enrich good quality engagement practices with process indicators that can be used to monitor and evaluate whether the ongoing process of developing a new medicine leads to valuable outcomes for all stakeholders involved. This includes patient representatives or patient partners, Health Technology Assessment (HTA) organisations and industry. It will deliver a coherent and flexible framework that can be used as a toolbox. This implies that the framework can be used by participants in a wide variety of engagement practices, like patient engagement in protocol development, in agenda setting or in approval or reimbursement procedures of new medicines. At the end of the 30-month project, the monitoring and evaluation framework can be used as a toolbox to open the floor for full dialogue, containing helpful evaluation tools for everyone involved. This undertaking has not been done before and addresses the need for a consensus-based monitoring and evaluation framework. Current scientific and non-scientific literature suggests several indicators to measure the value of patient engagement, but these are scattered over decision points and do not always link up to clear outcomes. Furthermore, less is published on methods to actually measure the value of patient engagement, and validated tools for evaluating impact are not available. The explicit goal of our work is to create this framework in a co-construction process with all relevant stakeholders, where PARADIGM partners and others are invited to discuss whether the most relevant endpoints are included and how the framework can be as practical as possible. We are applying the Reflexive Monitoring in Action (RMA) approach, an integrated methodology to encourage learning within multi-actor groups or networks as well as institutional change, in order to not only build a framework but work on successful implementation by co-developing it with the future end-users. In the first six months of PARADIGM, we worked on a literature review to gain insights on existing tools, indicators and the like, leading to a preliminary framework that conceptualizes the (observable) outcomes and impact for medicines development across the lifecycle. Furthermore, we explored what constitutes value of patient engagement for all parties involved, and how this can be measured; what are appropriate and feasible indicators, what is the return on engagement? In parallel, we started to collect examples of patient engagement practices from within and outside the PARADIGM consortium. We concluded these efforts by organising a workshop, kindly hosted by the Association of the British Pharmaceutical Industry (ABPI) in London, where participants provided valuable input on how to best present and categorise the results of the literature review and how we should approach which examples of existing patient engagement practices. We are now embarking on, first, the retrospective analysis of examples of patient engagement practices based on the preliminary framework together with the respective partners. We will apply, test and refine the framework by evaluating the examples. Second, we will apply the improved framework to prospective cases and use it to help the stakeholders enhance the value and impact of the engagement. This will lead us to produce two distinct, but interrelated products for all parties: 1) a tailor-made advice on how valuable impact can be reached in their own context 2) an accompanying evaluation framework that can be used by everyone who wants to enrich good quality engagement practices and improve the value and impact for all involved. We strongly believe that the co-creation of this actionable framework will greatly support the cultural shift that is needed to move the needle towards greater patient engagement in medicines R&D

Determining the ‘value’ of patient engagement

“What is in it for me?” A question often asked by people involved in patient engagement practices. Evaluating whether engagement adds value for different stakeholder groups can be an effective tool to further support patient engagement. The PARADIGM project is a European public-private partnership that is developing ways to ensure that patients are meaningfully involved in medicines development. One of the aims of the project is to develop metrics to measure the “return on engagement”. To contribute to this system innovation, the PARADIGM workstream co-led by MSD and the Athena Institute (VU Amsterdam) aims to enrich good quality engagement practices with process indicators that can be used to monitor and evaluate whether the ongoing process of developing a new medicine leads to valuable outcomes for all stakeholders involved. This includes patient representatives or patient partners, Health Technology Assessment (HTA) organisations and industry. It will deliver a coherent and flexible framework that can be used as a toolbox. This implies that the framework can be used by participants in a wide variety of engagement practices, like patient engagement in protocol development, in agenda setting or in approval or reimbursement procedures of new medicines. At the end of the 30-month project, the monitoring and evaluation framework can be used as a toolbox to open the floor for full dialogue, containing helpful evaluation tools for everyone involved. This undertaking has not been done before and addresses the need for a consensus-based monitoring and evaluation framework. Current scientific and non-scientific literature suggests several indicators to measure the value of patient engagement, but these are scattered over decision points and do not always link up to clear outcomes. Furthermore, less is published on methods to actually measure the value of patient engagement, and validated tools for evaluating impact are not available. The explicit goal of our work is to create this framework in a co-construction process with all relevant stakeholders, where PARADIGM partners and others are invited to discuss whether the most relevant endpoints are included and how the framework can be as practical as possible. We are applying the Reflexive Monitoring in Action (RMA) approach, an integrated methodology to encourage learning within multi-actor groups or networks as well as institutional change, in order to not only build a framework but work on successful implementation by co-developing it with the future end-users. In the first six months of PARADIGM, we worked on a literature review to gain insights on existing tools, indicators and the like, leading to a preliminary framework that conceptualizes the (observable) outcomes and impact for medicines development across the lifecycle. Furthermore, we explored what constitutes value of patient engagement for all parties involved, and how this can be measured; what are appropriate and feasible indicators, what is the return on engagement? In parallel, we started to collect examples of patient engagement practices from within and outside the PARADIGM consortium. We concluded these efforts by organising a workshop, kindly hosted by the Association of the British Pharmaceutical Industry (ABPI) in London, where participants provided valuable input on how to best present and categorise the results of the literature review and how we should approach which examples of existing patient engagement practices. We are now embarking on, first, the retrospective analysis of examples of patient engagement practices based on the preliminary framework together with the respective partners. We will apply, test and refine the framework by evaluating the examples. Second, we will apply the improved framework to prospective cases and use it to help the stakeholders enhance the value and impact of the engagement. This will lead us to produce two distinct, but interrelated products for all parties: 1) a tailor-made advice on how valuable impact can be reached in their own context 2) an accompanying evaluation framework that can be used by everyone who wants to enrich good quality engagement practices and improve the value and impact for all involved. We strongly believe that the co-creation of this actionable framework will greatly support the cultural shift that is needed to move the needle towards greater patient engagement in medicines R&D

Co-building a patient-oriented research curriculum in Canada

Plain English summary Foundations in Patient-Oriented Research is a course designed and piloted in Canada to help patients, researchers, health care professionals and health system decision-makers gain an introductory understanding of patient-oriented research, the research enterprise, and how to work in a team. The course curriculum was co-developed by a diverse group of people with different lived experiences and relevant expertise. The course is meant to be delivered in a ‘co-learning format’ with classes comprised of all the above stakeholder groups learning together. The purpose of this study was to explore the experiences of the project leaders, developers, facilitators and patient co-facilitators who were involved in the process of co-developing, piloting and revising the curriculum. Our findings suggest that co-developing a patient-oriented research curriculum increases its quality, uptake and credibility. The co-development process not only resulted in training that benefited the target learners, but it provided valuable learning experiences about patient-oriented research for the project leaders, developers, facilitators and patient co-facilitators. These findings and the resulting recommendations may provide guidance for other learning and development groups wishing to undertake a similar project. Abstract Background Foundations in Patient-Oriented Research is a course designed and piloted in Canada to build mutually beneficial relationships for conducting patient-oriented research by ensuring that relevant stakeholders – patients, researchers, health care professionals and health system decision-makers – have a common foundational understanding of patient-oriented research, the research enterprise, and team dynamics. The curriculum was co-developed by a group of patients, researchers, patient engagement experts and curriculum development experts and involved consultations with broader groups of the relevant stakeholders mentioned above. It was designed to be delivered in a ‘co-learning format’ with classes comprised of all stakeholder groups learning together. The purpose of this study was to explore the experiences of individuals involved in the process of co-developing, piloting and revising Foundations in Patient-Oriented Research. Methods An embedded case study was conducted with individuals who were involved in the co-development, pilot and revision of Foundations in Patient-Oriented Research. These individuals took on different roles during the curriculum development process, including project co-lead, developer, facilitator, and patient co-facilitator. The constant comparison method was used to inductively develop themes from the two focus group sessions. Results Discussions from the focus groups revealed the value of co-building the content, co-facilitating the course sessions, and the importance of the co-learning format. The training itself was perceived as valuable and the systematic approach to co-development was perceived as a success. Several barriers were identified, including the amount of resources, time and commitment required to complete the project. There was a notable tension between maintaining the integrity of the content and having the freedom to adapt it to local contexts. Over the course of the project, the project co-leads, developers and facilitators found that their own understanding of patient-oriented research deepened. Conclusions The findings of this study suggest that co-developing a patient-oriented research curriculum increases its quality, uptake and credibility. The co-development process not only resulted in training that benefited the target learners, but also built capacity for patient-oriented research within the project co-leads, developers, facilitators and patient co-facilitators. Our findings and recommendations may provide guidance for other learning and development groups wishing to undertake a similar project

Co-design and its consequences: developing a shared patient engagement framework in the IMI-PARADIGM project

Whilst patient engagement (PE) activities have become increasingly prevalent in medicines development, collaborating actors have different perspectives of its goals and its added value. In the development of PE standards and frameworks, the significance of these differences tends to be minimized. Boundary objects have been shown to mediate knowledge exchange between multiple social worlds, thereby playing an important role in participatory technology governance processes. In this paper, we draw on boundary objects to learn from the process of co-designing a PE monitoring and evaluation (M&E) framework within the IMI-PARADIGM consortium (2018 – 2020). As facilitators of PARADIGM’s co-design process, we report on the challenges encountered in designing a practicable M&E framework that serves different needs and interests. We argue these challenges of co-design reflect a negotiation of different frames throughout, thereby providing insight into how such work may contribute to addressing the challenge of knowledge integration in institutional medicines development settings

Giving patients a voice:A participatory evaluation of patient engagement in Newfoundland and Labrador Health Research

Background: Efforts to engage patients as partners in health research have grown and thereby the need for feedback and evaluation. In this pilot evaluation study, we aimed to 1) evaluate patient engagement in health research projects in Newfoundland and Labrador, Canada, and 2) learn more about how to best monitor and evaluate patient engagement. This paper presents the results of our participatory evaluation study and the lessons learned. The evaluation of the projects was driven by questions patients wanted answered. Methods: We conducted a formative evaluation of patient engagement in health research projects. Projects spanned a variety of topics, target groups, research designs and methods of patient engagement. Participants included principal investigators (n = 6) and their patient partners (n = 14). Furthermore, graduate students (n = 13) working on their own research projects participated. Participants completed an online survey with closed and open-ended questions about their patient engagement efforts, experiences and preliminary outcomes. Patients were involved as co-investigators in the entire evaluation study. We used qualitative methods to evaluate our participatory process. Results: The evaluation study results show that most patients and researchers felt prepared and worked together in various phases of the research process. Both groups felt that the insights and comments of patients influenced research decisions. They believed that patient engagement improved the quality and uptake of research. Students felt less prepared and were less satisfied with their patient engagement experience compared to researchers and their patient partners. Involvement of patient co-investigators in this evaluation resulted in learnings, transparency, validation of findings and increased applicability. Challenges were to select evaluation questions relevant to all stakeholders and to adapt evaluation tools to local needs. Conclusions: Our findings show that researchers, patient partners and students value patient engagement in health research. Capacity building at the supervisor level in academic institutions is needed to better support students. Sufficient time is also needed to permit observable outcomes. Participatory evaluation may increase the relevance and usefulness of information, but it also raises issues such as who defines and designs the content of evaluation tools. A co-creation process is required to develop appropriate monitoring and evaluation strategies

Evaluation of patient engagement in medicine development: A multi-stakeholder framework with metrics

Background: Patient engagement is becoming more customary in medicine development. However, embedding it in organizational decision-making remains challenging, partly due to lack of agreement on its value and the means to evaluate it. The objective of this project was to develop a monitoring and evaluation framework, with metrics, to demonstrate impact and enhance learning. Methods: A consortium of five patient groups, 15 biopharmaceutical companies and two academic groups iteratively created a framework in a multi-phase participatory process, including analysis of its application in 24 cases. Results: The framework includes six components, with 87 metrics and 15 context factors distributed among (sub)components: (a) Input: expectations, preparations, resources, representativeness of stakeholders; (b) Activities/process: structure, management, interactions, satisfaction; (c) Learnings and changes; (d) Impacts: research relevance, study ethics and inclusiveness, study quality and efficiency, quality of evidence and uptake of products, empowerment, reputation and trust, embedding of patient engagement; (e) Context: policy, institutional, community, decision-making contextual factors. Case study findings show a wide variation in use of metrics. There is no ‘one size fits all’ set of metrics appropriate for every initiative or organization. Presented sample sets of metrics can be tailored to individual situations. Conclusion: Introducing change into any process is best done when the value of that change is clear. This framework allows participants to select what metrics they value and assess to what extent patient engagement has contributed. Patient contribution: Five patient groups were involved in all phases of the study (design, conduct, interpretation of data) and in writing the manuscript