Using Bayes Factors to evaluate evidence for no effect: examples from the SIPS project

Aims: To illustrate how Bayes Factors are important for determining the effectiveness of interventions. Method: We consider a case where inappropriate conclusions were publicly drawn based on significance testing, namely the SIPS Project (Screening and Intervention Programme for Sensible drinking), a pragmatic, cluster-randomized controlled trial in each of two healthcare settings and in the criminal justice system. We showhow Bayes Factors can disambiguate the non-significant findings from the SIPS Project and thus determine whether the findings represent evidence of absence or absence of evidence. We show how to model the sort of effects that could be expected, and how to check the robustness of the Bayes Factors. Results: The findings from the three SIPS trials taken individually are largely uninformative but, when data from these trials are combined, there is moderate evidence for a null hypothesis (H0) and thus for a lack of effect of brief intervention compared with simple clinical feedback and an alcohol information leaflet (B = 0.24, p = 0.43). Conclusion: Scientists who find non-significant results should suspend judgment – unless they calculate a Bayes Factor to indicate either that there is evidence for a null hypothesis (H0) over a (welljustified) alternative hypothesis (H1), or else that more data are needed

Baseline demographic and clinical values at randomisation according to study group.

<p>Baseline demographic and clinical values at randomisation according to study group.</p

Baseline, 6 and 12 week means and mean differences of memantine versus placebo using last outcome carried forward adjusted for baseline (except CGI as no baseline).

<p>The effect at weeks 6 & 12 uses lme and adjusts for baseline for CMAI, NPI, SIB and SMMSE.* p<0.05 ** p <0.01 *** P<0.001. NPI =  Neuropsychiatric Inventory; CGIC = Clinical Global Impression of change; SMMSE =  Standard Mini Mental State examination; CMAI =  Cohen Mansfield Agitation Index; SIB =  Severe Impairment Battery.</p

CONSORT STATEMENT.

<p>CONSORT STATEMENT.</p

Adverse events.

<p>Adverse events.</p

Mean (+/−95% Confidence intervals) total NPI by time and by group.

<p>Mean (+/−95% Confidence intervals) total NPI by time and by group.</p

Mean (+/−95% confidence intervals) total CMAI by time and by group.

<p>Circles are individual data points; red: memantine, blue: placebo.</p

Baseline AUDIT score.

<p>Baseline AUDIT score.</p

Outcome measures at 6 and 12 months.

a<p>adjusted for mean baseline AUDIT score and screening instrument.</p>d<p>adjusted for screening instrument.</p><p>*estimates of differences are produced from weighted regression models.</p

Consort Diagram.

<p>Consort Diagram.</p