Kapur, Shitij

Munafò, Marcus

English

Munafo, Marcus

Explore Bristol Research

                          Kapur, S., & Munafò, M. (2019). Small Sample Sizes and a FalseEconomy for Psychiatric Clinical Trials. JAMA Psychiatry, 76(7), 676-677. https://doi.org/10.1001/jamapsychiatry.2019.0095Publisher's PDF, also known as Version of recordLink to published version (if available):10.1001/jamapsychiatry.2019.0095Link to publication record in Explore Bristol ResearchPDF-documentThis is the final published version of the article (version of record). It first appeared online via AMA athttps://jamanetwork.com/journals/jamapsychiatry/fullarticle/2729439 . Please refer to any applicable terms of useof the publisher.University of Bristol - Explore Bristol ResearchGeneral rightsThis document is made available in accordance with publisher policies. Please cite only thepublished version using the reference above. Full terms of use are available:http://www.bristol.ac.uk/red/research-policy/pure/user-guides/ebr-terms/Small Sample Sizes and a False Economyfor Psychiatric Clinical TrialsShitij Kapur, MBBS, PhD, FRCPC; Marcus Munafò, PhDIn 2013, JAMA Psychiatry published an exciting new findingby Hallak et al1: patients with schizophrenia who were treatedwith a single infusion of the antihypertensive agent sodiumnitroprusside showed a dramatic, instantaneous, and sus-tained improvement in psy-chotic and negative symp-toms. The study by Hallaket al1 was not only double-blind, randomized, and placebo-controlled, but the authors had also presented a prospectivepower analysis, ensured good interrater reliability, andreported all the relevant aspects of the trial. Understandably,this study generated tremendous excitement and severalattempts were made to replicate the finding. In this issue ofJAMA Psychiatry, Brown et al2 report an equally systematicstudy that fails to replicate the original finding. In fact, thefindings by Brown et al2 join those of 2 other prior studies3,4that also failed to replicate the original finding. A carefulanalysis of the competing studies fails to reveal an obviousexplanation for this discrepancy—except that the originalfinding was based on a rather small sample of just 10 patientsper treatment arm.Failure to reproduce such dramatic findings is neither new,nor limited to psychiatry. As provocatively claimed byIoannidis5 in 2005, perhaps most published studies are in-deed false. Alert to this possibility, the Academy of Medical Sci-ences in the United Kingdom published a report on the repro-ducibility and reliability of biomedical research, and identifiedthe many causes for the low replicability of published stud-ies, from deficiencies in the conduct and reporting of studiesto incentive structures that promote publication over rigour.6,7However, in the matter of the sodium nitroprusside studies themethods of the studies seem to be strong and the reportingseems to be thorough. But rigor of conduct and reporting can-not overcome the challenge of starting with a small numberof patients.A review of 83 psychiatric intervention studies indicatedthat only half had been subject to any attempt at replication.8Of those that had been subject to an attempt at replication, only37% replicated the original effect size, while the others eithercontradicted the original finding or observed a much smallereffect. The review clearly showed that, the larger the originalstudy, the more likely it was to be replicated.It is a statistical inevitability that (all other things beingequal) smaller studies will provide more imprecise estimatesof any true effect. Therefore, if a number of small studies areperformed, there will be a wider variation of findings (some-times described as vibration of effects). If many small studiesare conducted, only those that generate a large effect size (andtherefore reach a P value below the conventional 5% thresh-old) will be published and these findings from studies with asmall sample size are likely to represent inflated effect sizeestimates, and at worst, false positives.Based on this finding it would be easy to recommend largersample sizes in general, given evidence that studies in the bio-medical sciences are usually too small to detect credible orlikely effect sizes.9 But any such general commandment wouldbe too simplistic. Ultimately, studies should be large enoughto give a sufficiently precise estimate, but not so large as to bewasteful. Funders and journals increasingly require a justifi-cation of sample size. However, calculation of sample size isnot a perfect science and requires an estimation of the ex-pected outcomes in the drug and placebo groups and an esti-mation of the expected variance. Getting these estimates rightand realistic is at the heart of the matter.Hallak et al1 predicted an effect size of d = 1.5. How doesthis size compare with effect sizes in this field? We provide not2, but 3 comparators. A study of clinical interventions in medi-cine and psychiatry across a range of domains indicates a me-dian observed an effect size of 0.37 for general medicine and0.41 for psychiatry.10 A review of more than 7000 patientstreated with antipsychotics in randomized clinical trialsshowed that the effect size of the newer atypical antipsychot-ics, compared with placebo, was 0.48.11 Finally, studies ofschizophrenia clinicians show that they consider an improve-ment from baseline (which includes placebo response plus drugresponse) equivalent to an approximate effect size of d = 1.0to be clinically significant.12 Thus, seen from these 3 perspec-tives, an expectation of d = 1.5 vs placebo seems rather opti-mistic. The temptation is understandable. One would like todiscover the big findings, and such an assumption comes withthe payoff that it decreases the number of patients requiredby sample size calculators. However, such over-optimismcomes with 2 wasteful consequences: a higher than appropri-ate chance of a false negative and, in the event of a positivefinding, a higher chance that the positive finding might be afalse positive with an inflated effect.So, what is the right effect size to aim for? There is not, andcannot be, a single answer. What seems prudent is that trialsof any new treatment should assume the median observed inthe field (which usually in the range of 0.3-0.5), and those whohope for a much larger effect size should be required to pro-vide a strong justification for such optimism. Because, over-optimism that leads to too small a sample may just be a falseeconomy.Related articleOpinionEDITORIALjamapsychiatry.com (Reprinted) JAMA Psychiatry Published online March 27, 2019 E1© 2019 American Medical Association. All rights reserved.Downloaded From: https://jamanetwork.com/ by a University of Bristol User  on 05/14/2019ARTICLE INFORMATIONAuthor Affiliations: Faculty of Medicine, Dentistry,and Health Sciences, University of Melbourne,Melbourne, Australia (Kapur); School ofPsychological Science, University of Bristol, Bristol,United Kingdom (Munafò); Medical ResearchCouncil of UK Integrative Epidemiology Unit,University of Bristol, Bristol, United Kingdom(Munafò).Corresponding Author: Shitij Kapur, MBBS, PhD,FRCPC, Faculty of Medicine, Dentistry, and HealthSciences, University of Melbourne, Melbourne3010, Australia (shitij.kapur@unimelb.edu.au).Published Online: March 27, 2019.doi:10.1001/jamapsychiatry.2019.0095Conflict of Interest Disclosures: Dr Kapur reportedbeing coauthor on one of the previously publishedstudies that failed to replicate the sodiumnitroprusside effect. No other disclosures werereported.REFERENCES1. Hallak JE, Maia-de-Oliveira JP, Abrao J, et al.Rapid improvement of acute schizophreniasymptoms after intravenous sodium nitroprusside:a randomized, double-blind, placebo-controlledtrial. JAMA Psychiatry. 2013;70(7):668-676.2. Brown HE, Freudenreich O, Fan X, et al. Efficacyand tolerability of adjunctive intravenous sodiumnitroprusside treatment for outpatients withschizophrenia: a randomized clinical trial [publishedonline March 27, 2019]. JAMA Psychiatry. doi:10.1001/jamapsychiatry.2019.01513. Stone JM, Morrison PD, Koychev I, et al. Theeffect of sodium nitroprusside on psychoticsymptoms and spatial working memory in patientswith schizophrenia: a randomized, double-blind,placebo-controlled trial. Psychol Med. 2016;46(16):3443-3450. doi:10.1017/S00332917160022454. Wang X, Zhao J, Hu Y, et al. Sodiumnitroprusside treatment for psychotic symptomsand cognitive deficits of schizophrenia:a randomized, double-blind, placebo-controlledtrial. Psychiatry Res. 2018;269:271-277. doi:10.1016/j.psychres.2018.08.0795. Ioannidis JP. Why most published researchfindings are false. PLoS Med. 2005;2(8):e124.6. Academy of Medical Sciences. Reproducibilityand Reliability of Biomedical Research: ImprovingResearch Practice. London, UK: Academy of MedicalSciences; 2015.7. Higginson AD, Munafò MR. Current incentivesfor scientists lead to underpowered studies witherroneous conclusions. PLoS Biol. 2016;14(11):e2000995. doi:10.1371/journal.pbio.20009958. Tajika A, Ogawa Y, Takeshima N, Hayasaka Y,Furukawa TA. Replication and contradiction ofhighly cited research papers in psychiatry: 10-yearfollow-up. Br J Psychiatry. 2015;207(4):357-362.doi:10.1192/bjp.bp.113.1437019. Dumas-Mallet E, Button KS, Boraud T, Gonon F,Munafò MR. Low statistical power in biomedicalscience: a review of three human research domains.R Soc Open Sci. 2017;4(2):160254. doi:10.1098/rsos.16025410. Leucht S, Hierl S, Kissling W, Dold M, Davis JM.Putting the efficacy of psychiatric and generalmedicine medication into perspective: review ofmeta-analyses. Br J Psychiatry. 2012;200(2):97-106.doi:10.1192/bjp.bp.111.09659411. Leucht S, Arbter D, Engel RR, Kissling W, DavisJM. How effective are second-generationantipsychotic drugs? a meta-analysis ofplacebo-controlled trials. Mol Psychiatry. 2009;14(4):429-447. doi:10.1038/sj.mp.400213612. Hermes ED, Sokoloff D, Stroup TS, RosenheckRA. Minimum clinically important difference in thePositive and Negative Syndrome Scale with datafrom the Clinical Antipsychotic Trials of InterventionEffectiveness (CATIE). J Clin Psychiatry. 2012;73(4):526-532. doi:10.4088/JCP.11m07162Opinion EditorialE2 JAMA Psychiatry Published online March 27, 2019 (Reprinted) jamapsychiatry.com© 2019 American Medical Association. All rights reserved.Downloaded From: https://jamanetwork.com/ by a University of Bristol User  on 05/14/2019

Small Sample Sizes and a False Economy for Psychiatric Clinical Trials

https://research-information.bris.ac.uk/files/194244736/munafo.pdf

Small Sample Sizes and a False Economy for Psychiatric Clinical Trials

Abstract

Similar works

Full text

Available Versions

Explore Bristol Research