Methodological challenges of mixed methods intervention evaluations

This paper addresses the methodological challenges that accompany the use of a combination of research methods to evaluate complex interventions. In evaluating complex interventions, the question about effectiveness is not the only question that needs to be answered. Of equal interest are questions about acceptability, feasibility, and implementation of the intervention and the evaluation study itself. Using qualitative research in conjunction with trials enables us to address this diversity of questions. The combination of methods results in a mixed methods intervention evaluation (MMIE). In this article we demonstrate the relevance of mixed methods evaluation studies and provide case studies from health care. Methodological challenges that need our attention are, among others, choosing appropriate designs for MMIEs, determining realistic expectations of both components, and assigning adequate resources to both components. Solving these methodological issues will improve our research designs and provide further insights into complex interventions

Objective nebuliser adherence data as “proof” of adherence in the management of Cystic Fibrosis : a qualitative interview study

Purpose: Low adherence to medication via nebulisers is linked to poor clinical outcomes for people with Cystic Fibrosis (PWCF). Advances in technology allow electronic monitoring of nebuliser usage and feedback of objective adherence data to PWCF and clinical teams caring for them. CFHealthHub is a new intervention that collects and displays objective adherence data in easy-to-read formats with the aim of improving nebuliser adherence and health. There is little understanding of how objective adherence data is perceived by PWCF and healthcare professionals (HCPs). Patients and Methods: A qualitative study using semi-structured interviews with 22 PWCF and 31 HCPs who had used the CFHealthHub intervention. Results: Objective adherence data was welcomed by the majority of PWCF in the sample, and HCP delivering the intervention, because the data allowed PWCF to reflect on patterns of adherence or non-adherence. Ease of use and characteristics of data display were important, particularly the use of a “traffic light” system to allow PWCF to easily see if they were meeting their adherence targets. For PWCF objective adherence data was used as “proof to self”, offering reassurance to high adherers, and a wake-up call to those with lower levels of adherence. It could also provide ‘proof to others’ where PWCF had higher levels of adherence than HCP or family members believed. The data could sometimes change HCP perceptions of PWCF’s identities as poor adherers. Where adherence was not high, data was used to facilitate honest discussions between PWCF and HCPs about how to increase adherence. HCPs perceived that it was important to use the data positively to motivate, rather than criticise, PWCF. Conclusion: Objective nebuliser adherence data in CFHealthHub can offer proof of adherence to PWCF and HCPs. It is important to use it constructively to facilitate discussions on how to improve adherence

Maximising the value of combining qualitative research and randomised controlled trials in health research: the QUAlitative Research in Trials (QUART) study--a mixed methods study.

BACKGROUND: Researchers sometimes undertake qualitative research with randomised controlled trials (RCTs) of health interventions. OBJECTIVES: To systematically explore how qualitative research is being used with trials and identify ways of maximising its value to the trial aim of providing evidence of effectiveness of health interventions. DESIGN: A sequential mixed methods study with four components. METHODS: (1) Database search of peer-reviewed journals between January 2008 and September 2010 for articles reporting the qualitative research undertaken with specific trials, (2) systematic search of database of registered trials to identify studies combining qualitative research and trials, (3) survey of 200 lead investigators of trials with no apparent qualitative research and (4) semistructured telephone interviews with 18 researchers purposively sampled from the first three methods. RESULTS: Qualitative research was undertaken with at least 12% of trials. A large number of articles reporting qualitative research undertaken with trials (n=296) were published between 2008 and 2010. A total of 28% (82/296) of articles reported qualitative research undertaken at the pre-trial stage and around one-quarter concerned drugs or devices. The articles focused on 22 aspects of the trial within five broad categories. Some focused on more than one aspect of the trial, totalling 356 examples. The qualitative research focused on the intervention being trialled (71%, 254/356), the design and conduct of the trial (15%, 54/356), the outcomes of the trial (1%, 5/356), the measures used in the trial (3%, 10/356), and the health condition in the trial (9%, 33/356). The potential value of the qualitative research to the trial endeavour included improving the external validity of trials and facilitating interpretation of trial findings. This value could be maximised by using qualitative research more at the pre-trial stage and reporting findings with explicit attention to the implications for the trial endeavour. During interviews, three models of study were identified: qualitative research as peripheral to the trial, qualitative research as an 'add-on' to the trial and a study with qualitative research and trial as essential components, with the third model offering more opportunity to maximise the value of the qualitative research. Interviewees valued the use of qualitative research with trials and identified team structures and wider structural issues which gave more value to the trial than the qualitative research as barriers to maximising the value of the qualitative research. CONCLUSION: A large number of articles were published between 2008 and 2010, addressing a wide range of aspects of trials. There were examples of this research affecting the trial by facilitating interpretation of trial findings, developing and refining interventions for testing in the trial and changing the measures used in the trial. However, researchers were not necessarily maximising the value of qualitative research undertaken with trials to the endeavour of generating evidence of effectiveness of health interventions. Researchers can maximise value by promoting its use at the pre-trial stage to ensure that the intervention and trial conduct is optimised at the main trial stage, being explicit about the conclusions for the trial endeavour in peer-reviewed journal articles reporting the qualitative research and valuing the contribution of the qualitative research as much as the trial. Future recommendations for researchers include: plan the qualitative research, design and implement studies not trials, use qualitative research at the feasibility and pilot stage of trials, be explicit in publications about the impact of the qualitative research on the trial and implications for the trial endeavour, undertake in-depth qualitative research, allow qualitative research to take a challenging role and develop a learning environment around the use of qualitative research and trials. FUNDING: This project was funded by the Medical Research Council (MRC) as part of the MRC-National Institute for Health Research Methodology Research programme

Beyond interviews and focus groups: a framework for integrating innovative qualitative methods into randomised controlled trials of complex public health interventions

Background Randomised controlled trials (RCTs) are widely used for establishing evidence of the effectiveness of interventions, yet public health interventions are often complex, posing specific challenges for RCTs. Although there is increasing recognition that qualitative methods can and should be integrated into RCTs, few frameworks and practical guidance highlight which qualitative methods should be integrated and for what purposes. As a result, qualitative methods are often poorly or haphazardly integrated into existing trials, and researchers rely heavily on interviews and focus group discussions. To improve current practice, we propose a framework for innovative qualitative research methods that can help address the challenges of RCTs for complex public health interventions. Methods We used a stepped approach to develop a practical framework for researchers. This consisted of (1) a systematic review of the innovative qualitative methods mentioned in the health literature, (2) in-depth interviews with 23 academics from different methodological backgrounds working on RCTs of public health interventions in 11 different countries, and (3) a framework development and group consensus-building process. Results The findings are presented in accordance with the CONSORT (Consolidated Standards of Reporting Trials) Statement categories for ease of use. We identify the main challenges of RCTs for public health interventions alongside each of the CONSORT categories, and potential innovative qualitative methods that overcome each challenge are listed as part of a Framework for the Integration of Innovative Qualitative Methods into RCTs of Complex Health Interventions. Innovative qualitative methods described in the interviews include rapid ethnographic appraisals, document analysis, diary methods, interactive voice responses and short message service, community mapping, spiral walks, pair interviews and visual participatory analysis. Conclusions The findings of this study point to the usefulness of observational and participatory methods for trials of complex public health interventions, offering a novel contribution to the broader literature about the need for mixed methods approaches. Integrating a diverse toolkit of qualitative methods can enable appropriate adjustments to the intervention or process (or both) of data collection during RCTs, which in turn can create more sustainable and effective interventions. However, such integration will require a cultural shift towards the adoption of method-neutral research approaches, transdisciplinary collaborations, and publishing regimes

An intervention to support adherence to inhaled medication in adults with cystic fibrosis : the ACtiF research programme including RCT

Background People with cystic fibrosis frequently have low levels of adherence to inhaled medications. Objectives The objectives were to develop and evaluate an intervention for adults with cystic fibrosis to improve adherence to their inhaled medication. Design We used agile software methods to develop an online platform. We used mixed methods to develop a behaviour change intervention for delivery by an interventionist. These were integrated to become the CFHealthHub intervention. We undertook a feasibility study consisting of a pilot randomised controlled trial and process evaluation in two cystic fibrosis centres. We evaluated the intervention using an open-label, parallel-group randomised controlled trial with usual care as the control. Participants were randomised in a 1 : 1 ratio to intervention or usual care. Usual care consisted of clinic visits every 3 months. We undertook a process evaluation alongside the randomised controlled trial, including a fidelity study, a qualitative interview study and a mediation analysis. We undertook a health economic analysis using both a within-trial and model-based analysis. Setting The randomised controlled trial took place in 19 UK cystic fibrosis centres. Participants Participants were people aged ≥ 16 years with cystic fibrosis, on the cystic fibrosis registry, not post lung transplant or on the active transplant list, who were able to consent and not using dry-powder inhalers. Intervention People with cystic fibrosis used a nebuliser with electronic monitoring capabilities. This transferred data automatically to a digital platform. People with cystic fibrosis and clinicians could monitor adherence using these data, including through a mobile application (app). CFHealthHub displayed graphs of adherence data as well as educational and problem-solving information. A trained interventionist helped people with cystic fibrosis to address their adherence. Main outcome measures Randomised controlled trial – adjusted incidence rate ratio of pulmonary exacerbations meeting the modified Fuchs criteria over a 12-month follow-up period (primary outcome); change in percentage adherence; and per cent predicted forced expiratory volume in 1 second (key secondary outcomes). Process evaluation – percentage fidelity to intervention delivery, and participant and interventionist perceptions of the intervention. Economic modelling – incremental cost per quality-adjusted life-year gained. Results Randomised controlled trial – 608 participants were randomised to the intervention (n = 305) or usual care (n = 303). To our knowledge, this was the largest randomised controlled trial in cystic fibrosis undertaken in the UK. The adjusted rate of exacerbations per year (primary outcome) was 1.63 in the intervention and 1.77 in the usual-care arm (incidence rate ratio 0.96, 95% confidence interval 0.83 to 1.12; p = 0.638) after adjustment for covariates. The adjusted difference in mean weekly normative adherence was 9.5% (95% confidence interval 8.6% to 10.4%) across 1 year, favouring the intervention. Adjusted mean difference in forced expiratory volume in 1 second (per cent) predicted at 12 months was 1.4% (95% confidence interval –0.2% to 3.0%). No adverse events were related to the intervention. Process evaluation – fidelity of intervention delivery was high, the intervention was acceptable to people with cystic fibrosis, participants engaged with the intervention [287/305 (94%) attended the first intervention visit], expected mechanisms of action were identified and contextual factors varied between randomised controlled trial sites. Qualitative interviews with 22 people with cystic fibrosis and 26 interventionists identified that people with cystic fibrosis welcomed the objective adherence data as proof of actions to self and others, and valued the relationship that they built with the interventionists. Economic modelling – the within-trial analysis suggests that the intervention generated 0.01 additional quality-adjusted life-years at an additional cost of £865.91 per patient, leading to an incremental cost-effectiveness ratio of £71,136 per quality-adjusted life-year gained. This should be interpreted with caution owing to the short time horizon. The health economic model suggests that the intervention is expected to generate 0.17 additional quality-adjusted life-years and cost savings of £1790 over a lifetime (70-year) horizon; hence, the intervention is expected to dominate usual care. Assuming a willingness-to-pay threshold of £20,000 per quality-adjusted life-year gained, the probability that the intervention generates more net benefit than usual care is 0.89. The model results are dependent on assumptions regarding the duration over which costs and effects of the intervention apply, the impact of the intervention on forced expiratory volume in 1 second (per cent) predicted and the relationship between increased adherence and drug-prescribing levels. Limitations Number of exacerbations is a sensitive and valid measure of clinical change used in many trials. However, data collection of this outcome in this context was challenging and could have been subject to bias. It was not possible to measure baseline adherence accurately. It was not possible to quantify the impact of the intervention on the number of packs of medicines prescribed. Conclusions We developed a feasible and acceptable intervention that was delivered to fidelity in the randomised controlled trial. We observed no statistically significant difference in the primary outcome of exacerbation rates over 12 months. We observed an increase in normative adherence levels in a disease where adherence levels are low. The magnitude of the increase in adherence may not have been large enough to affect exacerbations. Future work Given the non-significant difference in the primary outcome, further research is required to explore why an increase in objective normative adherence did not reduce exacerbations and to develop interventions that reduce exacerbations. Trial registration Work package 3.1: Current Controlled Trials ISRCTN13076797. Work packages 3.2 and 3.3: Current Controlled Trials ISRCTN55504164

Moving from randomized controlled trials to mixed methods intervention evaluations

This chapter explores why mixed methods intervention evaluations are needed when undertaking randomized controlled trials in order to address a wide range of questions relevant to understanding the effectiveness of an intervention. It describes three frameworks that illustrate different ways in which mixed methods intervention evaluations may be undertaken within the context of a randomized controlled trial: the temporal framework; the process-outcome framework, which includes process evaluations; and the “aspects of a trial” framework. The chapter considers how the language used to describe qualitative research undertaken with trials can represent different underlying assumptions about the relative value of the qualitative research in relation to the trial. The chapter concludes with a discussion of some of the challenges that arise when undertaking mixed method intervention evaluations and the value of including qualitative research in systematic reviews of trials via evidence synthesis

Getting added value from using qualitative research with randomized controlled trials: a qualitative interview study.

Qualitative research is undertaken with randomized controlled trials of health interventions. Our aim was to explore the perceptions of researchers with experience of this endeavour to understand the added value of qualitative research to the trial in practice

Modelling successful self-management in adults with cystic fibrosis: Vicarious self-efficacy from videos of ‘people like me’

Background Self-efficacy is an important determinant of treatment adherence, and peer modelling of success can provide vicarious self-efficacy. A series of patient stories (‘talking heads’ videos) were developed with people with cystic fibrosis (CF) as part of the CFHealthHub multi-component adherence intervention, aiming to demonstrate success with daily therapy in ‘people like me’. Methodology One-to-one semi-structured interviews exploring patients’ experiences, barriers and facilitators of nebuliser adherence were audio and video-recorded between October 2015 and August 2016. Interview transcripts were reviewed to identify descriptions of problem-solving and sustained treatment success. Positive stories potentially providing vicarious descriptions of success were selected as video clips. Results In total, 14 adults with CF were recruited from five UK CF centres. Each participant contributed a median of five (interquartile range: 3-6) video clips, and a total of 57 unique clips were uploaded onto the CFHealthHub digital platform. Nine of those clips spanned two categories, hence, there were 66 clips across 16 categories. Conclusions The videos were well received though some adults were concerned that comparisons with peers might create anxiety by highlighting the possibility of future decline or current relative underperformance. It is important to sensitively support choice when providing resources aiming to increase vicarious self-efficacy. Our experience may guide the development of similar videos for people with other long-term conditions

Development and evaluation of an intervention to support adherence to treatment in adults with Cystic Fibrosis : a randomised controlled trial and parallel process evaluation protocol

Cystic Fibrosis (CF) is a genetic disease that affects around 10,000 individuals in the UK. Individuals with CF are prone to lung infections; inhaled medications are required to stay healthy, costing £30 million annually, yet average adherence has been estimated at 36%. Patients with 80% of prescribed medication). This is both costly to the healthcare system and distressing for patients with CF and their families. This protocol details the procedures of a two-armed, superiority, open-labelled randomised controlled trial with a parallel, mixed methods process evaluation. This study aims to examine the efficacy of a complex intervention designed to help patients with CF to adhere to therapy, compared to usual care, on clinical and participant related outcomes. In addition, this trial aims to identify the best way to deliver the complex intervention