Ahmad, R

Castro Sanchez, EM

Holmes, A

Iwami, M

English

Spiral - Imperial College Digital Repository

Application of novel outcome evaluation criteria to UK social marketing campaigns focused on infectionsCastro-Sánchez E, Iwami M, Ahmad R, Holmes AH.IntroductionSocial marketing (SM) interventions have been valuable in addressing communicablediseases1, and the government in the United Kingdom (UK) spent about £55 million on socialmarketing in 2013-42. Whilst evaluation of public health activities is critical to determine if anintervention achieves its desired results, is cost effective, or requires modification, SM hasreceived scrutiny for insufficient standardisation of practice and lack of outcome evaluation.Previous systematic reviews of SM campaigns noted increased attention to processevaluation (e.g., segmentation, implementation), but only small improvements in the qualityof outcome evaluation3, or significant lack of rigorous outcome evaluation techniques4.We developed and applied a novel set of evaluation criteria to UK social marketingcampaigns focused on infections.MethodsCampaigns active between 2001-2013 were identified from literature and internet searches.Two researchers (ECS, MI) independently extracted data on 13 evaluation indicatorspreviously proposed (Box 1) from information publicly available about the campaigns andinterventions.ResultsWe identified 17 social marketing campaigns focusing on chlamydia (7), screening of sexuallytransmitted infections (5), immunisations (2), hand hygiene (2), and healthcare-associatedinfections (1). (TABLE 1)Median score for all campaigns was 14.2 +/- 2 (26 points maximum possible). Indicators 1‘Developed with thorough understanding of the problem & key stakeholders’ and 5 ‘Outcomemeasures that meet the needs of the stakeholders’ were best described (score 26; 34 pointsmaximum possible). Two-thirds (14/22, 66.1%) of interventions provided insufficient orambiguous information towards each indicator (in amber, Table 1).Indicator 8 ‘Include measurements indicating the way the intervention led to desired outcomebetween intervention and outcome’ received the lowest score (11/34), and was effectivelynot included in 7/17 interventions. Other indicators such as 9 ‘Include measurement ofcompeting factors that could influence the change’ (13/34 points) and 10 ‘Identification ofunpredicted effects changes from intervention’ (15/34 points) included poor or incompletereporting. Indicators 3, 7, 10 and 12 were insufficiently or inadequately reported in 88% ofcampaigns.ConclusionsUK social marketing campaigns on infections could improve their reporting of key outcomeevaluation aspects. Sustainability, economic evaluation and unintended effects of campaignsrequire increased attention.Your thoughts? Use post-its and add below, contact e.castro-sanchez@imperial.ac.uk or @castrocloudDiscussionWe recommended initial standards for outcome measures to improve value assessment ofinterventions. This framework is a flexible tool that can be used in numerous, as audit tool,intervention development, benchmarking, kaizen, scaling-up, dissemination, and/orresearch. As with most quality indicators, these criteria could be considered subjective. Otherlimitations include the limited information available across the interventions and that mighthave led to over-estimation of partial scores.Intervention 1. Understanding Problem & Stakeholders2. Definition of Outcome3. Long term Change4. Clear Theory / Plausibility5. Meet Needs6. Measured Impact7. Acceptability / Sustainability8. Link between intervention & outcome9. CompetingFactors10. UnpredictedChange11. Strategy 12. EconAnalysis13. Peer-ReviewScore‘Are you getting it?’5 2 0 1 2 1 1 1 1 1 1 2 1 1 15CaSH6 2 2 1 2 2 1 1 0 1 1 1 1 1 16‘Chlamydia get tested’72 2 1 1 2 1 1 0 0 1 0 1 1 13‘Checkin’ it out’8 / ‘It’s down to you’92 1 1 1 1 1 1 1 1 1 1 1 0 13Clean safe care10 1 1 1 1 1 1 1 1 1 1 1 1 1 13Cleanyourhands11,12 1 2 1 2 1 2 1 0 0 1 1 1 2 15COAST13 2 2 2 2 2 1 1 0 0 1 0 1 1 15Coronation Street14 1 2 1 2 1 1 2 1 1 0 2 1 2 17DaSH15 1 1 1 2 2 1 1 0 1 1 0 1 1 13‘Easy-peasy’ chlamydia testing16,172 2 1 1 2 1 1 0 1 1 1 1 1 15Increasing the uptake of MMR in London182 0 1 1 2 1 1 0 1 1 2 1 1 14Judah et al.19 1 1 0 2 1 0 0 2 0 0 2 0 2 11‘Just4You’20 1 0 1 1 1 1 1 1 1 1 1 1 0 11Love is Infectious21 1 1 1 1 1 1 1 1 1 1 1 1 1 13‘My60seconds’22-23 2 2 1 2 2 0 1 1 1 1 0 1 1 15NHSCSP – Mass media242 2 1 1 2 1 1 1 1 1 2 2 2 19‘Spend a Penny, Gain a Fiver’251 1 1 1 2 1 1 1 1 1 1 1 1 14Score 26 22 17 25 26 16 17 11 13 15 18 17 19FundingNIHR HPRU in Healthcare Associated Infection and Antimicrobial Resistance in partnershipwith Public Health England (PHE). The views are those of the author(s) and not necessarilythose of the NHS, the NIHR, Department of Health or Public Health England.Quality indicator Description1. Developed with thorough understanding of the problem & key stakeholdersLack of understanding can lead to ineffective interventions, iatrogenic effects and financial waste2. Clear definition of desired outcome Prior to intervention design and implementation, including  what changes will be measured and how3. ‘A priori’ indication of how much change is needed to achieve desired effectsTo differentiate between statistical and ‘on the ground’ changes due to an intervention4. Clear theory/plausibility of how intervention results in outcomesStrict adherence to existing theories not necessary, but logical plausibility for why intervention effective should be present5. Include outcome measures that meet the needs of the stakeholdersMeeting the needs and interests of stakeholders is ethical and essential for continuing a successful intervention6. Measure impact Often not measured, but helpful in establishing sustainability and efficiency7. Ensure acceptability and sustainability ofintervention for different participantsInterventions must be acceptable to the target population in order to be ethical and effective. 8. Include measurements indicating the way the intervention led to desired outcomePre- and post- intervention changes not enough to determine that the changes can be attributed to the intervention9. Include measurement of competing factors that could influence the changeWith complex interventions environmental determinants also measured to ensure outcomes due to intervention10. Identification of unpredicted changes, including 'iatrogenic' effectsAll changes induced by intervention are critical to measure11. Application of most robust outcome evaluation allowed by intervention designMinimum outcome measurements required to determine the outcome changed after the intervention was implemented are essential, however the most robust strategy is bestEach reviewer independently scored the quality indicators for validity. Scores were given from0 points (lowest quality, indicator definition not met, red), to 1 point (indicator definitionpartially met, or not reported, amber) or 2 points (highest quality, indicator definition metand/or reported, green). Therefore, intervention scores could range between 0 and 26points. Final scores were discussed and agreed, with the assistance of a third rater toresolve discrepancies (intraclass correlation coefficient 0.9453).Box 1. Outcome evaluation indicators, Castro et al [in press] Table 1. Social marketing campaigns on infection, 2001-2013, UK NIHR Health Protection Research Unit in Healthcare Associated Infection and Antimicrobial Resistance at Imperial College London, London (United Kingdom)ReferencesReference list available at

Application of novel outcome evaluation criteria to UK social marketing campaigns focused on infections

http://spiral.imperial.ac.uk/bitstream/10044/1/42602/2/Application%20of%20novel%20outcome%20evaluation%20criteria%20to%20UK.pdf

Application of novel outcome evaluation criteria to UK social marketing campaigns focused on infections

Abstract

Similar works

Full text

Available Versions

Spiral - Imperial College Digital Repository