Use of Risk Models to Predict Death in the Next Year Among Individual Ambulatory Patients With Heart Failure

Importance: The clinical practice guidelines for heart failure recommend the use of validated risk models to estimate prognosis. Understanding how well models identify individuals who will die in the next year informs decision making for advanced treatments and hospice. Objective: To quantify how risk models calculated in routine practice estimate more than 50% 1-year mortality among ambulatory patients with heart failure who die in the subsequent year. Design, Setting, and Participants: Ambulatory adults with heart failure from 3 integrated health systems were enrolled between 2005 and 2008. The probability of death was estimated using the Seattle Heart Failure Model (SHFM) and the Meta-Analysis Global Group in Chronic Heart Failure (MAGGIC) risk calculator. Baseline covariates were collected from electronic health records. Missing covariates were imputed. Estimated mortality was compared with actual mortality at both population and individual levels. Main Outcomes and Measures: One-year mortality. Results: Among 10930 patients with heart failure, the median age was 77 years, and 48.0% of these patients were female. In the year after study enrollment, 1661 patients died (15.9% by life-table analysis). At the population level, 1-year predicted mortality among the cohort was 9.7% for the SHFM (C statistic of 0.66) and 17.5% for the MAGGIC risk calculator (C statistic of 0.69). At the individual level, the SHFM predicted a more than 50% probability of dying in the next year for 8 of the 1661 patients who died (sensitivity for 1-year death was 0.5%) and for 5 patients who lived at least a year (positive predictive value, 61.5%). The MAGGIC risk calculator predicted a more than 50% probability of dying in the next year for 52 of the 1661 patients who died (sensitivity, 3.1%) and for 63 patients who lived at least a year (positive predictive value, 45.2%). Conversely, the SHFM estimated that 8496 patients (77.8%) had a less than 15% probability of dying at 1 year, yet this lower-risk end of the score range captured nearly two-thirds of deaths (n = 997); similarly, the MAGGIC risk calculator estimated a probability of dying of less than 25% for the majority of patients who died at 1 year (n = 914). Conclusions and Relevance: Although heart failure risk models perform reasonably well at the population level, they do not reliably predict which individual patients will die in the next year

Extent of non-publication in cohorts of studies approved by research ethics committees or included in trial registries

BACKGROUND: The synthesis of published research in systematic reviews is essential when providing evidence to inform clinical and health policy decision-making. However, the validity of systematic reviews is threatened if journal publications represent a biased selection of all studies that have been conducted (dissemination bias). To investigate the extent of dissemination bias we conducted a systematic review that determined the proportion of studies published as peer-reviewed journal articles and investigated factors associated with full publication in cohorts of studies (i) approved by research ethics committees (RECs) or (ii) included in trial registries. METHODS AND FINDINGS: Four bibliographic databases were searched for methodological research projects (MRPs) without limitations for publication year, language or study location. The searches were supplemented by handsearching the references of included MRPs. We estimated the proportion of studies published using prediction intervals (PI) and a random effects meta-analysis. Pooled odds ratios (OR) were used to express associations between study characteristics and journal publication. Seventeen MRPs (23 publications) evaluated cohorts of studies approved by RECs; the proportion of published studies had a PI between 22% and 72% and the weighted pooled proportion when combining estimates would be 46.2% (95% CI 40.2%-52.4%, I2 = 94.4%). Twenty-two MRPs (22 publications) evaluated cohorts of studies included in trial registries; the PI of the proportion published ranged from 13% to 90% and the weighted pooled proportion would be 54.2% (95% CI 42.0%-65.9%, I2 = 98.9%). REC-approved studies with statistically significant results (compared with those without statistically significant results) were more likely to be published (pooled OR 2.8; 95% CI 2.2-3.5). Phase-III trials were also more likely to be published than phase II trials (pooled OR 2.0; 95% CI 1.6-2.5). The probability of publication within two years after study completion ranged from 7% to 30%. CONCLUSIONS: A substantial part of the studies approved by RECs or included in trial registries remains unpublished. Due to the large heterogeneity a prediction of the publication probability for a future study is very uncertain. Non-publication of research is not a random process, e.g., it is associated with the direction of study findings. Our findings suggest that the dissemination of research findings is biased

Protocol for a systematic review on the extent of non-publication of research studies and associated study characteristics

Abstract Background Methodological research has found that non-published studies often have different results than those that are published, a phenomenon known as publication bias. When results are not published, or are published selectively based on the direction or the strength of the findings, healthcare professionals and consumers of healthcare cannot base their decision-making on the full body of current evidence. Methods As part of the OPEN project (http://www.open-project.eu) we will conduct a systematic review with the following objectives: 1. To determine the proportion and/or rate of non-publication of studies by systematically reviewing methodological research projects that followed up a cohort of studies that a. received research ethics committee (REC) approval, b. were registered in trial registries, or c. were presented as abstracts at conferences. 2. To assess the association of study characteristics (for example, direction and/or strength of findings) with likelihood of full publication. To identify reports of relevant methodological research projects we will conduct electronic database searches, check reference lists, and contact experts. Published and unpublished projects will be included. The inclusion criteria are as follows: a. RECs: methodological research projects that examined the subsequent proportion and/or rate of publication of studies that received approval from RECs; b. Trial registries: methodological research projects that examine the subsequent proportion and/or rate of publication of studies registered in trial registries; c. Conference abstracts: methodological research projects that examine the subsequent proportion and/or rate of full publication of studies which were initially presented at conferences as abstracts. Primary outcomes: Proportion/rate of published studies; time to full publication (mean/median; cumulative publication rate by time). Secondary outcomes: Association of study characteristics with full publication. The different questions (a, b, and c) will be investigated separately. Data synthesis will involve a combination of descriptive and statistical summaries of the included methodological research projects. Discussion Results are expected to be publicly available in mid 2013.</p

Weighted proportion of published studies for 17 MRPs following studies after REC approval.

<p>Weighted proportion of published studies for 17 MRPs following studies after REC approval.</p

Factors associated with journal publication.

<p>Nr: Not reported, MRP: Methodological research project, No: Number, OR: Odds ratio, REC: Research ethics committee.</p><p>Factors associated with journal publication.</p

Main characteristics of 17 methodological research projects following studies after approval by a research ethics committee.

<p>*No definite follow-up time predictable, but more than 24 months follow-up from study approval or completion to search for full publication fulfilled.</p><p>RCT: randomised controlled trial, REC: research ethic committee.</p><p>Main characteristics of 17 methodological research projects following studies after approval by a research ethics committee.</p

Time to publication after ethics committee approval.

<p>Time to publication after ethics committee approval.</p

Main characteristics of 22 methodological research projects following studies included in trial registries.

<p>*No definite follow-up time predictable, but more than 24 months follow-up from study completion to search for full publication fulfilled.</p><p>FDA: US Food and Drug Administration, GSK: Glaxo Smith Kline, ISRCTN: International Standard Randomised Controlled Trial Number, NIH: National Institutes of Health, RCT: randomised controlled trial, WHO: World Health Organisation.</p><p>Main characteristics of 22 methodological research projects following studies included in trial registries.</p

Risk of bias table for MRPs following studies after approval by a REC.

<p>*Easterbrook 1992 reported publication rates for completed and approved studies separately. We just refer to the completed sample in this review. NA: Not applicable.</p><p><b>+</b> means low risk of bias; <b>-</b> means high risk of bias;<b>?</b> means unclear risk of bias.</p><p><b>Follow-up time:</b>>24 months after study completion: <b>+</b>. <24 months follow-up after study completion: <b>-</b>. MRPs which judged their follow- up rather on approved than completed studies. Although these MRPs fulfilled the 24 month follow-up criteria, we judged them to have an unclear risk of bias:<b>?</b>.</p><p><b>Methodology used to identify journal publication:</b> electronic search and author contact: <b>+</b>. only author contact (with response rate of ≥80%): <b>+</b>. only author contact (with response rate <80%): <b>-</b>; only database search (in 1 database): <b>-</b>. only database search (in>1 database): <b>+</b>. methodology not given:<b>?</b>.</p><p><b>Adjustment for confounders:</b> if an analysis for factors associated with journal publication was carried out: <b>+</b>. if no analysis was carried out: <b>-</b>.</p><p><b>Matching criteria:</b> if ≥2 matching criteria given: <b>+</b>. if <2 matching criteria given: <b>-</b>. matching criteria not given in MRP:<b>?</b>. if only author contact was used to identify journal publication: <b>NA</b> (not applicable).</p><p><b>Research status:</b> completed protocols: <b>+</b>. approved protocols: <b>-</b>.</p><p><b>Sampling method:</b> all trials, random sample or consecutive trials: <b>+</b>. if only investigator responded to questionnaire for this sample: <b>-</b>. sampling method not given (e.g., without the word “all”):<b>?</b>.</p><p>Risk of bias table for MRPs following studies after approval by a REC.</p

Risk of bias table for MRPs following studies included in trial registries.

<p>*The research status of Shamliyan 2012a refers to completed and terminated trials.</p><p>**The MRP of Tfelt-Hansen 2011 is based on the GKS registry only.</p><p><b>+</b> means low risk of bias; <b>-</b> means high risk of bias;<b>?</b> means unclear risk of bias.</p><p><b>Follow-up time:</b>>24 months: <b>+</b>. <24 months: <b>-</b>. follow-up time not given/or could not be estimated:<b>?</b>.</p><p><b>Methodology used to identify journal publication:</b> electronic search and author contact and/or search in trial registry: <b>+</b>. only author contact (with a response rate of ≥80%): <b>+</b>. only author contact (with a response rate <80%): <b>-</b>. only search in trial registry or only 1 database: <b>-</b>. methodology not given:<b>?</b>.</p><p><b>Adjustment for confounders:</b> if an analysis for factors associated with the journal publication was carried out: <b>+</b>. if no analyses were carried out: <b>-</b>.</p><p><b>Matching criteria:</b> if ≥2 matching criteria given: <b>+</b>. if <2 matching criteria given: <b>-</b>. matching criteria not given in MRP:<b>?</b>.</p><p><b>Research status:</b> completed registry entries: <b>+</b>. completed and initiated mixed: <b>-</b>. not mentioned:<b>?</b>.</p><p><b>Sampling method:</b> all trials, random sample or consecutive trials: <b>+</b>. if only investigator responded to questionnaire for this sample: <b>-</b>. sampling method not given (e.g., without the word “all”):<b>?</b>.</p><p>Risk of bias table for MRPs following studies included in trial registries.</p