Anderson, Geoffrey M.

Garfinkel, Susan

Gurwitz, Jerry H.

Mamdani, Muhammad M.

Normand, Sharon-Lise T.

Rochon, Paula A.

Streiner, David L.

Sykora, Kathy

English

eScholarship@UMMS

University of Massachusetts Medical School eScholarship@UMMS Meyers Primary Care Institute Publications and Presentations Meyers Primary Care Institute 2005-04-16 Reader's guide to critical appraisal of cohort studies: 1. Role and design. Paula A. Rochon Baycrest Centre for Geriatric Care Et al. Let us know how access to this document benefits you. Follow this and additional works at: https://escholarship.umassmed.edu/meyers_pp  Part of the Health Services Research Commons Repository Citation Rochon PA, Gurwitz JH, Sykora K, Mamdani MM, Streiner DL, Garfinkel S, Normand ST, Anderson GM. (2005). Reader's guide to critical appraisal of cohort studies: 1. Role and design.. Meyers Primary Care Institute Publications and Presentations. https://doi.org/10.1136/bmj.330.7496.895. Retrieved from https://escholarship.umassmed.edu/meyers_pp/62 This material is brought to you by eScholarship@UMMS. It has been accepted for inclusion in Meyers Primary Care Institute Publications and Presentations by an authorized administrator of eScholarship@UMMS. For more information, please contact Lisa.Palmer@umassmed.edu. Education and debateReader’s guide to critical appraisal of cohort studies: 1.Role and designPaula A Rochon, Jerry H Gurwitz, Kathy Sykora, Muhammad Mamdani, David L Streiner,Susan Garfinkel, Sharon-Lise T Normand, Geoffrey M AndersonCohort studies can provide valuable information unavailable from randomised trials, but readersneed to be alert to possible flawsValid evidence on the benefits and risks of healthcareinterventions is essential to rational decision making.Randomised controlled trials are considered the bestmethod for providing evidence on efficacy. However,they face important ethical and logistical constraintsand have been criticised for focusing on highly selectedpopulations and outcomes.1 2 Some of these problemscan be overcome by cohort studies. Cohort studies canbe thought of as natural experiments in whichoutcomes are measured in real world rather thanexperimental settings. They can evaluate large groupsof diverse individuals, follow them for long periods,and provide information on a range of outcomes,including rare adverse events. However, the promise ofcohort studies as a useful source of evidence needs tobe balanced against concerns about the validity of thatevidence.3 4In this three paper series we will provide anapproach to the critical appraisal of cohort studies.This article describes the role and design of cohortstudies and explains how selection bias can confoundthe relation between the intervention and the outcome.The second article will outline strategies for identifica-tion and assessment of the potential for confounding,and the third article describes statistical techniques thatcan be used to deal with confounding. Each paperdefines a set of questions that, taken together, can pro-vide readers with a systematic approach to criticallyassessing evidence from cohort studies.Randomised trial or cohort study?Cohort studies are similar to randomised controlledtrials in that they compare outcomes in groups that didand did not receive an intervention. The maindifference is that allocation of individuals is not bychance. Table 1 gives some important similarities anddifferences between the two types of study. Becausethey are expensive and recruiting patients can be diffi-cult, randomised controlled trials are generally shortterm and used to determine efficacy in selected popu-lations under strict conditions. Cohort studies can beused to determine if the efficacy observed inrandomised trials translates into effectiveness inbroader populations and more realistic settings and toprovide information on adverse events and risks.5Selection bias as a threat to validityThe internal validity of a study is defined as the extentto which the observed difference in outcomes betweenthe two comparison groups can be attributed to theintervention rather than other factors. The biggestadvantage of randomised controlled trials comparedwith cohort studies is that the random allocation pro-cess enhances the internal validity of a study byminimising selection bias and confounding.6 Thispaper relies on the definitions provided by CONSORT(box 1).7Allocation by chance in a randomised controlledtrial should mean that the groups being compared aresimilar in terms of both measured and unmeasuredbaseline factors.8 This is not so in cohort studies, andtherefore cohort studies are vulnerable to selectionbias. In cohort studies, factors that determined whethera person received the intervention could result in thegroups differing in factors related to the outcome,either because people were preferentially selected toreceive one treatment or because of choices that theymade. These baseline differences in prognosis couldconfound the assessment of the effect of the interven-tion.In cohort studies care must be taken to minimise,assess, and deal with selection bias. A comprehensiveCohort studies can use diverse populationsPHOTOALTO/PHOTONICAThis is the firstof three articleson appraisingcohort studiesKunin-LunenfeldApplied ResearchUnit, BaycrestCentre for GeriatricCare, Toronto, ON,CanadaPaula A Rochonsenior scientistMeyers PrimaryCare Institute,Worcester, MA01605, USAJerry H Gurwitzexecutive directorInstitute for ClinicalEvaluative Sciences,Toronto, ON,CanadaKathy Sykorasenior biostatisticianMuhammadMamdanisenior scientistSusan Garfinkelresearch coordinatorDepartment ofPsychiatry,University ofToronto, Toronto,ON, CanadaDavid L StreinerprofessorDepartment ofHealth Care Policy,Harvard MedicalSchool, Boston,USASharon-Lise TNormandprofessor of healthcare policy(biostatistics)continued overBMJ 2005;330:895–7895BMJ VOLUME 330 16 APRIL 2005 bmj.comapproach is needed that includes the selection ofappropriate comparison groups, the identification andassessment of the comparability of potential confound-ers between those comparison groups, and the use ofsophisticated statistical techniques in the analysis.Comparison groups in cohort studiesThe essence of any cohort study is the comparison ofoutcomes between people who received the interven-tion and those who did not. For example, to answer thequestion, “Do patients who receive an atypical antipsy-chotic drug have an increased risk of hip fracture?” acohort study must ask: “What would have happened tothese patients if they had not received the atypicalantipsychotic drug?”Ideally, the comparison group in the cohort studyshould be identical to the intervention group, apartfrom the fact that they did not receive the intervention.This ideal comparison group is described bymethodologists as providing the “counterfactual” or“potential outcome.”9 In reality, this ideal comparisongroup does not exist. Part of the art of designing acohort study is choosing comparison groups thatapproach this ideal in order to minimise selection biaswhile maintaining clinically relevance.The analysis of the association between antipsy-chotic drugs and hip fracture can be used to define thetypes of comparisons that could be found in cohortstudies. For any specific intervention (such as exposureto atypical antipsychotics) two factors—the exposureexperience of the comparison group and thepopulation from which the intervention and compari-son groups are selected—define the types of compari-sons that are possible (box 2). People taking atypicalantipsychotics can be compared with either peopletaking an alternative antipsychotic or with thoseprescribed no antipsychotic drugs. These comparisonscould be made in a general population (all elderlypeople) or in a restricted population (elderly peoplewith dementia).Questions to ask when assessing acohort study designWhat comparison is being made?Published studies may include more than one type ofcomparison, but the focus of any appraisal of a cohortstudy is on an individual comparison between an inter-vention group and a comparison group in a definedpopulation. A well written study should contain a cleardefinition of why the two groups were selected andhow they were defined. This information is essentialfor assessment of clinical relevance and potential forselection bias.Does the comparison make clinical sense?The clinical relevance of comparisons needs to beassessed for each case. In the analysis of antipsychoticuse and hip fracture, for instance, all four types of com-parison might be relevant. However, this might not betrue in other analyses. For example, although it wouldbe possible for a cohort study to compare HIV positivepatients receiving antiretroviral therapy with thosereceiving no intervention,10 this comparison would beirrelevant to many clinicians. A more relevant cohortstudy would compare patients receiving one antiretro-viral therapy with patients receiving another interven-tion.11 In contrast, a clinically relevant study of theadverse effects of a commonly used treatment such asa non-steroidal anti-inflammatory drug might includea comparison with a no intervention population sinceno drug treatment could be a realistic option for somepeople.12Cohort studies should not only describe the popu-lations being compared but also include a discussion ofthe clinical context for that comparison and provide ajustification for the comparison. Readers of thesestudies should determine if the study makes acomparison that is realistic and relevant to theirdecision needs.Table 1 Comparison of cohort studies and randomised controlled trialsItem Cohort studies Randomised controlled trialsPopulations studied Diverse populations of patients who are observed in a range ofsettingsHighly selected populations recruited on the basis of detailedcriteria and treated at selected sitesAllocation to the intervention Based on decisions made by providers or patients Based on chance and controlled by investigatorsOutcomes Can be defined after the intervention and can include rare orunexpected eventsPrimary outcomes are determined before patients are enteredinto study and are focused on predicted benefits and risksFollow-up Many cohort studies rely on existing experience (retrospectivestudies) and can provide an opportunity for long follow-upProspective studies; often have short follow-up because ofcosts and pressure to produce timely evidenceAnalysis Sophisticated multivariate techniques may be required to dealwith confoundingAnalysis is straightforwardBox 1: CONSORT definitions of selection biasand confounding7Selection bias—a systematic error in creatingintervention groups, causing them to differ withrespect to prognosis. The groups differ in measured orunmeasured baseline characteristics because of theway in which participants were selected for the studyor assigned to their study groupsConfounding—a situation in which the estimatedintervention effect is biased because of somedifference between the comparison groups apart fromthe planned interventions such as baselinecharacteristics, prognostic factors, or concomitantinterventions. For a factor to be a confounder, it mustdiffer between the comparison groups and predict theoutcome of interestBox 2: Possible types of comparisons in cohortstudyGeneral population1 Intervention v alternative intervention2 Intervention v no interventionRestricted population3 Intervention v alternative intervention4 Intervention v no interventionEducation and debateDepartment ofHealth Policy,Management, andEvaluation, Facultyof Medicine,University ofToronto, Toronto,ON, CanadaGeoffrey MAndersonchair in healthmanagementstrategiesCorrespondence to:G M Anderson,Institute for ClinicalEvaluative Sciences,2075 BayviewAvenue, Toronto,ON M4N 3M5,Canadageoff.anderson@utoronto.ca896 BMJ VOLUME 330 16 APRIL 2005 bmj.comWhat are the potential selection biases?Selection bias occurs when there is somethinginherently different between the groups being com-pared that could explain differences in the observedoutcomes. One powerful strategy to minimise selectionbias is to restrict inclusion in the study to those with adefined diagnosis or specific characteristics.3 Restrict-ing the groups to a specific characteristic removes thepotential for bias related to that characteristic and canreduce differences in related characteristics. Table 2presents data from a cohort of older adults given atypi-cal antipsychotics and a no intervention comparisongroup. Patients taking atypical antipsychotics were over12 times more likely (63.1% v 4.7%) to have dementia.Dementia is related to the risk of hip fracture, and thisimbalance may be an important source of confound-ing. Restricting the study to people with dementiaeliminates this source of confounding and reducesselection related to age as the mean age differencebetween the groups dropped from years to months.An inevitable consequence of restriction is reducedsample size. In the example, the sample decreasedfrom 1.3 million to about 80 000 when the dementiarestriction was applied. When smaller databases arebeing used, restriction can greatly limit the power ofthe study. Restriction on the basis of clinical character-istics limits the generalisability of the findings. Themore restrictive the population, the less generalisablethe results.It is important to keep in mind the effect the choiceof comparison groups will have on potential selectionbias when evaluating a cohort study. Some sources ofselection bias are clear—for example, if access to atypicalantipsychotics was limited to patients of specialists thiscould result in patients who received these drugs beingdifferent from those who did not. Some sources of biasmay bemore subtle. For example, if doctors thought thatatypical antipsychotics had fewer side effects than typicalantipsychotics, they might preferentially use the atypicalantipsychotics in frailer patients. This form of selectionbias, referred to as channelling bias or confounding byindication,13 occurs when patients are assigned to oneintervention or another on the basis of prognosticfactors and is key issue in cohort studies.Readers should recognise the potential forselection bias in all cohort studies and carefullyconsider possible sources of bias. In the next article wewill outline the link between selection bias andconfounding and describe a strategy for identifyingand assessing the potential for confounding.We thank Andreas Laupacis for his comments and JenniferGold, Michelle Laxer, and Monica Lee for help in preparing themanuscript.Contributors and sources: The series is based on discussionsthat took place at regular meetings of the Canadian Institute forHealth Research chronic disease new emerging team. PAR is ageriatrician with extensive research experience in cohort studiesof prescription drugs who wrote the first draft of this article andis the guarantor. JHG and MM are clinicians and researchersand SLTN and DLS are statisticians who commented on draftsof this paper. KS programmed and conducted analyses and SGconducted literature searches and reviews. PAR and GMA con-ceived the idea for the series and GMA worked on drafts of thisarticle and coordinated the development of the series.Funding: This work was supported by a CIHR operating grant(CIHR No. MOP 53124) and a CIHR chronic disease newemerging team programme (NET-54010).Competing interests: None declared.1 Gurwitz JH, Col NF, Avorn J. The exclusion of the elderly and womenfrom clinical trials in acute myocardial infarction. JAMA 1992;268:1417-22.2 Murray MD, Callahan CM. Improving medication use for older adults: anintegrated research agenda. Ann Intern Med 2003;139:425-9.3 McKee M, Britton A, Black N, McPherson K, Sanderson C, Bain C. Inter-preting the evidence: choosing between randomised and non-randomised studies. BMJ 1991;1999:312-5.4 Benson K, Hartz AJ. A comparison of observational studies andrandomized, controlled trials. N Engl J Med 2000:342:1878-86.5 Black N. Why we need observational studies to evaluate the effectivenessof health care. BMJ 1996;312:1215-8.6 Grimes DA, Schulz KF. Bias and causal associations in observationalresearch. Lancet 2002;359:248-52.7 Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, et al.The revised CONSORT statement for reporting randomized trials:explanation and elaboration. Ann Intern Med 2001;134:663-94.8 Altman DG, Bland JM. Treatment allocation in controlled trials: why ran-domize. BMJ 1999;318:1209.9 Greenland S, Morgenstern H. Confounding in health research. Annu RevPublic Health 2001;22:189-212.10 Badri M, Wilson D, Wood R. Effect of highly active antiretroviral therapyon the incidence of tuberculosis in South Africa: a cohort study. Lancet2002;359:2059-64.11 Fellay J, Boubaker K, Ledergerber B, Bernasconi E, Furrer H, Battegay M.Prevalence of adverse events associated with potent antiretroviraltreatment: Swiss HIV cohort study. Lancet 2001;358:1322-7.12 Mamdani M, Rochon PA, Juurlink DN, Kopp A, Anderson GM, Naglie G,et al. Observational study of upper gastrointestinal haemorrhage in eld-erly patients given selective cyclo-oxygenase-2 inhibitors or conventionalnon-steroidal anti-inflammatory drugs. BMJ 2002;325:1-6.13 Psaty BM, Koepsell TD, Lin D, Weiss NS, Siscovick DS, Rosendaal FR, etal. Assessment and control for confounding by indication inobservational studies. J Am Geriatr Soc 1999;47:749-54.(Accepted 18 February 2005)Table 2 Effect on age distribution and sample size of restricting comparison of atypical antipsychotic with no intervention toindividuals with dementiaAll older people Older people with dementiaAtypical antipsychotic (n=34 960) No Intervention (n=1 251 435) Atypical antipsychotic (n=21 427) No intervention (n=58 754)Mean (SD) age 80.46 (7.63) 74.50 (6.58) 81.69 (7.11) 80.95 (7.64)No (%) withdementia21 427 (61.3) 58 754 (4.7) 21 427 (100) 58 754 (100)Key questionsWhat comparison is being made?Does the comparison make clinical sense?What are the potential selection biases?EndpieceGood adviceBetter to hunt in fields, for health unbought,Than fee the doctor for a nauseous draught.The wise, for cure, on exercise depend;God never made his work for man to mend.John Dryden (1631-1700) in Epistle to JohnDriden of Chesterton (1700)Fred Charatan, retired geriatric physician, FloridaEducation and debate897BMJ VOLUME 330 16 APRIL 2005 bmj.com

Reader\u27s guide to critical appraisal of cohort studies: 1. Role and design.

https://escholarship.umassmed.edu/cgi/viewcontent.cgi?article=1071&amp;context=meyers_pp

Reader\u27s guide to critical appraisal of cohort studies: 1. Role and design.

Abstract

Similar works

Full text

Available Versions

eScholarship@UMMS