Developing and evaluating complex interventions: the new Medical Research Council guidance

<p><i>Evaluating complex interventions is complicated. The Medical Research Council's evaluation framework (2000) brought welcome clarity to the task. Now the council has updated its guidance</i></p> <p>Complex interventions are widely used in the health service, in public health practice, and in areas of social policy that have important health consequences, such as education, transport, and housing. They present various problems for evaluators, in addition to the practical and methodological difficulties that any successful evaluation must overcome. In 2000, the Medical Research Council (MRC) published a framework<sup>1</sup> to help researchers and research funders to recognise and adopt appropriate methods. The framework has been highly influential, and the accompanying BMJ paper is widely cited.<sup>2</sup> However, much valuable experience has since accumulated of both conventional and more innovative methods. This has now been incorporated in comprehensively revised and updated guidance recently released by the MRC (<a href="www.mrc.ac.uk/complexinterventionsguidance">www.mrc.ac.uk/complexinterventionsguidance</a>). In this article we summarise the issues that prompted the revision and the key messages of the new guidance. </p&gt

Evaluating Digital Math Tools in the Field

Many school districts have adopted digital tools to supplement or replace teacher-led instruction, usually based on the premise that these tools can provide more personalized or individualized experiences for students and at lower cost. Rigorously evaluating whether such initiatives promote better student outcomes in the field is difficult as most schools and teachers are unwilling to enforce rigorous study designs such as randomized control trials. We used study designs that were feasible in practice to assess whether two digital math tools, eSpark and IXL, were associated with improvements in 3rd – 6th grade student test scores in math. We also investigated the resource requirements and costs of implementing eSpark and IXL to assess whether these tools represent a valuable use of resources. We find that while IXL is substantially less costly to implement than eSpark, its use is not significantly associated with students’ math performance

Synthetic Controls: A New Approach to Evaluating Interventions

Synthetic control methods are a novel approach to comparative case study research using observational data. Though developed within political science, the methods can potentially be applied to a wide range of evaluation problems in economics, public health, social policy and other disciplines. In the traditional approach, an area in which a new or redesigned service is being implemented is compared with another ‘control’ area (in which there is no change) and statistical adjustment used to account for any differences between areas that might bias the comparison. In the new approach, a synthetic control is derived using data on past trends in all potentially comparable areas, providing a more robust basis for identifying the impact of the service change. Synthetic control methods may be a valuable addition to the range of techniques available for non-randomised evaluations of social, economic and public health interventions. To date there have been few applications in a UK context, and none in Scotland. Published evidence suggests considerable potential to apply synthetic controls to public service innovations at NHS Board, local authority or Community Planning Partnership level, and may widen the range of policy and practice changes that can usefully be evaluated

Development of a Delirium Educational Program for Hospital Medicine Providers

Hospital medicine providers were surveyed to evaluate baseline delirium attitudes and behaviors. An educational program was then shared, and a follow up survey was given to determine if their delirium attitudes and practices were impacted by the educational intervention. Follow up survey results indicated that providers perceived more of a change in their attitudes than practices, and overall found the education to be useful and felt more confident in treating delirium as a result

Evidence-based practice educational intervention studies: A systematic review of what is taught and how it is measured

Abstract Background Despite the established interest in evidence-based practice (EBP) as a core competence for clinicians, evidence for how best to teach and evaluate EBP remains weak. We sought to systematically assess coverage of the five EBP steps, review the outcome domains measured, and assess the properties of the instruments used in studies evaluating EBP educational interventions. Methods We conducted a systematic review of controlled studies (i.e. studies with a separate control group) which had investigated the effect of EBP educational interventions. We used citation analysis technique and tracked the forward and backward citations of the index articles (i.e. the systematic reviews and primary studies included in an overview of the effect of EBP teaching) using Web of Science until May 2017. We extracted information on intervention content (grouped into the five EBP steps), and the outcome domains assessed. We also searched the literature for published reliability and validity data of the EBP instruments used. Results Of 1831 records identified, 302 full-text articles were screened, and 85 included. Of these, 46 (54%) studies were randomised trials, 51 (60%) included postgraduate level participants, and 63 (75%) taught medical professionals. EBP Step 3 (critical appraisal) was the most frequently taught step (63 studies; 74%). Only 10 (12%) of the studies taught content which addressed all five EBP steps. Of the 85 studies, 52 (61%) evaluated EBP skills, 39 (46%) knowledge, 35 (41%) attitudes, 19 (22%) behaviours, 15 (18%) self-efficacy, and 7 (8%) measured reactions to EBP teaching delivery. Of the 24 instruments used in the included studies, 6 were high-quality (achieved ≥3 types of established validity evidence) and these were used in 14 (29%) of the 52 studies that measured EBP skills; 14 (41%) of the 39 studies that measured EBP knowledge; and 8 (26%) of the 35 studies that measured EBP attitude. Conclusions Most EBP educational interventions which have been evaluated in controlled studies focus on teaching only some of the EBP steps (predominantly critically appraisal of evidence) and did not use high-quality instruments to measure outcomes. Educational packages and instruments which address all EBP steps are needed to improve EBP teaching

Educational Games in Geriatric Medicine Education: A Systematic Review

OBJECTIVE: To systematically review the medical literature to assess the effect of geriatric educational games on the satisfaction, knowledge, beliefs, attitudes and behaviors of health care professionals. METHODS: We conducted a systematic review following the Cochrane Collaboration methodology including an electronic search of 10 electronic databases. We included randomized controlled trials (RCT) and controlled clinical trials (CCT) and excluded single arm studies. Population of interests included members (practitioners or students) of the health care professions. Outcomes of interests were participants' satisfaction, knowledge, beliefs, attitude, and behaviors. RESULTS: We included 8 studies evaluating 5 geriatric role playing games, all conducted in United States. All studies suffered from one or more methodological limitations but the overall quality of evidence was acceptable. None of the studies assessed the effects of the games on beliefs or behaviors. None of the 8 studies reported a statistically significant difference between the 2 groups in terms of change in attitude. One study assessed the impact on knowledge and found non-statistically significant difference between the 2 groups. Two studies found levels of satisfaction among participants to be high. We did not conduct a planned meta-analysis because the included studies either reported no statistical data or reported different summary statistics. CONCLUSION: The available evidence does not support the use of role playing interventions in geriatric medical education with the aim of improving the attitudes towards the elderly.Society of General Internal Medicin