Broadening horizons in impact evaluation for water, sanitation and hygiene planning: recycling and reinterpreting evidence

To meet universal Sustainable Development Goal targets, decision-makers need evidence about the effectiveness policy and programmes. Impact evaluations aim to provide that evidence, by quantifying the magnitude of changes in outcomes caused by WASH interventions in particular contexts for particular groups. However, there are concerns about the findings of single studies like randomised controlled trials (RCTs) and non-randomised studies (NRS), due to biases inherent in each approach. The best way to inform decisions is to use evidence from a variety of methodologies and contexts. An evidence census shows that, while the quantity and quality of WASH impact evaluations has increased, there are important ethical concerns about relevance, reporting and representativeness. Drawing on the census, a critical appraisal tool was developed to evaluate consistently biases in RCTs and NRS. The tool was piloted in systematic reviews of internal and external replications on international development topics. The results of systematic reviews and meta-analyses that applied the tool in external replications were analysed. The findings showed that NRS with relatively low risk-of-bias produced the same pooled effects on average as RCTs (standardised mean difference (SMD)=0.00; 95% confidence interval (CI)=-0.06, 0.06), but NRS with high risk-of-bias over-estimated effects (SMD=0.17; 95% CI=0.07, 0.28). A systematic review of internal replication studies also found well-designed NRS produced effects that were statistically indistinguishable from RCTs (mean squared error=0.00). Lack of access to and use of safe water, sanitation and hygiene (WASH) are thought to kill 300,000 children annually. RCTs are often considered the best causal evidence, but they cannot usually assess mortality due to power and ethical reasons. Existing systematic reviews assume diarrhoea morbidity is closely correlated with mortality. Meta-analysis of mortality impacts from the evidence census found 15 percent reduction in the odds of all-cause mortality in childhood, and 50 percent reduction in odds of diarrhoea mortality. WASH interventions reduce more deaths when they include hygiene and total sanitation