Design Characteristics Influence Performance of Clinical Prediction Rules in Validation: A Meta-Epidemiological Study

BACKGROUND: Many new clinical prediction rules are derived and validated. But the design and reporting quality of clinical prediction research has been less than optimal. We aimed to assess whether design characteristics of validation studies were associated with the overestimation of clinical prediction rules' performance. We also aimed to evaluate whether validation studies clearly reported important methodological characteristics. METHODS: Electronic databases were searched for systematic reviews of clinical prediction rule studies published between 2006 and 2010. Data were extracted from the eligible validation studies included in the systematic reviews. A meta-analytic meta-epidemiological approach was used to assess the influence of design characteristics on predictive performance. From each validation study, it was assessed whether 7 design and 7 reporting characteristics were properly described. RESULTS: A total of 287 validation studies of clinical prediction rule were collected from 15 systematic reviews (31 meta-analyses). Validation studies using case-control design produced a summary diagnostic odds ratio (DOR) 2.2 times (95% CI: 1.2-4.3) larger than validation studies using cohort design and unclear design. When differential verification was used, the summary DOR was overestimated by twofold (95% CI: 1.2 -3.1) compared to complete, partial and unclear verification. The summary RDOR of validation studies with inadequate sample size was 1.9 (95% CI: 1.2 -3.1) compared to studies with adequate sample size. Study site, reliability, and clinical prediction rule was adequately described in 10.1%, 9.4%, and 7.0% of validation studies respectively. CONCLUSION: Validation studies with design shortcomings may overestimate the performance of clinical prediction rules. The quality of reporting among studies validating clinical prediction rules needs to be improved

Research waste and inefficiency in cardiovascular prediction rule development: exploration of causes and implications

Background: Research waste and inefficiency occur when research studies do not contribute to the decisions of patients, clinicians, and policymakers. In this thesis, I examine research waste and inefficiency throughout the stages of cardiovascular clinical prediction rule (CPRs) development and explore their causes and implications. Methods: This thesis includes six research projects. A mixed-methods study, which consists of a review of derivation studies, thematic content analysis, and author survey, assesses justifications for deriving a new cardiovascular CPR. Using survival analysis, I evaluate the probability for a CPR for cardiovascular disease risk to have an independent external validation study after its derivation. I assess whether external validation studies update the Framingham Wilson coronary heart disease (CHD) risk rule when it performs poorly, and external validation studies contribute to understanding the performance in meta- analyses. A systematic review evaluates the design, methods, and reporting of studies assessing the impact of cardiovascular CPRs. I conduct an online survey to understand which cardiovascular CPRs are recognised and used by GPs in the UK. Lastly, an exploratory qualitative interview study with thematic analysis explores how perceptions of and experiences with cardiovascular CPR research influence primary care providers’ (PCPs) decisions about using cardiovascular CPRs. Results: I found that the authors of derivation studies do not cite existing evidence when justifying a new cardiovascular CPR even though they believe it is important to do so. Most of cardiovascular CPRs do not have a timely conducted independent external validation study. External validation studies of Framingham Wilson CHD risk rule fail to generate the evidence of generalizability to their full potential because they do not update the CPR when it performs poorly, or do not contribute to understanding the CPR’s overall performance. The vast majority of impact studies had flaws in study design, methods or reporting. GPs’ use of cardiovascular CPRs in the UK has evolved and is focused on a few CPRs, such as the QRISK scores. PCPs’ decisions about using cardiovascular CPRs seem to be influenced by their familiarity, trust, and confidence in using research evidence. Conclusions: To reduce research waste and inefficiency in cardiovascular CPR development, it is necessary to shift the paradigm from deriving new CPRs to externally validating, assessing the impact of, and promoting the adoption of high-quality existing cardiovascular CPRs. Also, systematic efforts are needed to reduce the flaws in design, methods, and reporting of CPR studies, and strengthen evidence-base to support the use of cardiovascular CPRs

Predictors for independent external validation of cardiovascular risk clinical prediction rules: Cox proportional hazards regression analyses

Abstract Background Clinical prediction rules (CPRs) should be externally validated by independent researchers. Although there are many cardiovascular CPRs, most have not been externally validated. It is not known why some CPRs are externally validated by independent researchers and others are not. Methods We analyzed cardiovascular risk CPRs included in a systematic review. Independent external validations were identified by forward citation searches of derivation studies. Time between the publication of a cardiovascular CPR and the first independent external validation was calculated. We assessed Kaplan-Meier estimates of the probability to have an independent external validation. Using univariable Cox regression, we explored whether characteristics of derivation (design, location, sample size, number of predictors, presentation format, validation in derivation), reporting (participants, predictors, outcomes, performance measure, information for risk calculation), and publication (journal impact factor) are associated with time to the first independent external validation. Results Of 125 cardiovascular risk CPRs, 29 had an independent external validation. The median follow-up was 118 months (95% CI, 99–130). The 25th percentile of event time was 122 months (95% CI, 91–299). Cardiovascular risk CPRs from the USA were 4.15 times (95% CI, 1.89–9.13) more likely to have an independent external validation. Increasing the sample size of derivation by ten times was associated with a 2.32-fold (95% CI, 1.37–3.91) increase in the probability of having an independent external validation. CPRs presented with an internal validation tend to get an independent external validation sooner (HR = 1.73, 95% CI, 0.77–3.93). CPRs reporting all the information necessary for calculating individual risk were 2.65 (95% CI, 1.01–6.96) times more likely to have an independent external validation. Publishing a cardiovascular risk CPR in a journal that has one unit higher impact factor was associated with a 6% (95% CI, 3–9) higher likelihood of an independent external validation. Conclusions The probability for cardiovascular risk CPRs to get an independent external validation was low even many years after their derivations. Authors of new cardiovascular risk CPRs should consider using adequate sample size, conducting an internal validation, and reporting all the information needed for risk calculation as these features were associated with an independent external validation

Influence of research evidence on the use of cardiovascular clinical prediction rules in primary care: an exploratory qualitative interview study

Abstract Background Cardiovascular clinical prediction rules (CPRs) are widely used in primary care. They accumulate research evidence through derivation, external validation, and impact studies. However, existing knowledge about the influence of research evidence on the use of CPRs is limited. Therefore, we explored how primary care clinicians’ perceptions of and experiences with research influence their use of cardiovascular CPRs. Methods We conducted an exploratory qualitative interview study with thematic analysis. Primary care clinicians were recruited from the WWAMI (Washington, Wyoming, Alaska, Montana and Idaho) region Practice and Research Network (WPRN). We used purposeful sampling to ensure maximum variation within the participant group. Data were collected by conducting semi-structured online interviews. We analyzed data using inductive thematic analysis to identify commonalities and differences within themes. Results Of 29 primary care clinicians who completed the questionnaire, 15 participated in the interview. We identified two main themes relating to the influence of clinicians’ perceptions of and experiences with cardiovascular CPR research on their decisions about using cardiovascular CPRs: “Seek and judge” and “be acquainted and assume.” When clinicians are familiar with, trust, and feel confident in using research evidence, they might actively search and assess the evidence, which may then influence their decisions about using cardiovascular CPRs. However, clinicians, who are unfamiliar with, distrust, or find it challenging to use research evidence, might be passively acquainted with evidence but do not make their own judgment on the trustworthiness of such evidence. Therefore, these clinicians might not rely on research evidence when making decisions about using cardiovascular CPRs. Conclusions Clinicians’ perceptions and experiences could influence how they use research evidence in decisions about using cardiovascular CPRs. This implies, when promoting evidence-based decisions, it might be useful to target clinicians’ unfamiliarity, distrust, and challenges regarding the use of research evidence rather than focusing only on their knowledge and skills. Further, because clinicians often rely on evidence-unrelated factors, guideline developers and policymakers should recommend cardiovascular CPRs supported by high-quality evidence

Additional file 2 of Influence of research evidence on the use of cardiovascular clinical prediction rules in primary care: an exploratory qualitative interview study

Additional file 2. Interview topic guide

Additional file 1 of Influence of research evidence on the use of cardiovascular clinical prediction rules in primary care: an exploratory qualitative interview study

Additional file 1. Search strategy to identify studies that evaluated the influence of research evidence on the uptake of clinical prediction rules

Additional file 2: of Predictors for independent external validation of cardiovascular risk clinical prediction rules: Cox proportional hazards regression analyses

Journals that published derivation and independent external validation studies. (DOCX 121 kb

Why do authors derive new cardiovascular clinical prediction rules in the presence of existing rules? A mixed methods study

<div><p>Background</p><p>Researchers should examine existing evidence to determine the need for a new study. It is unknown whether developers evaluate existing evidence to justify new cardiovascular clinical prediction rules (CPRs).</p><p>Objective</p><p>We aimed to assess whether authors of cardiovascular CPRs cited existing CPRs, why some authors did not cite existing CPRs, and why they thought existing CPRs were insufficient.</p><p>Method</p><p>Derivation studies of cardiovascular CPRs from the International Register of Clinical Prediction Rules for Primary Care were evaluated. We reviewed the introduction sections to determine whether existing CPRs were cited. Using thematic content analysis, the stated reasons for determining existing cardiovascular CPRs insufficient were explored. Study authors were surveyed via e-mail and post. We asked whether they were aware of any existing cardiovascular CPRs at the time of derivation, how they searched for existing CPRs, and whether they thought it was important to cite existing CPRs.</p><p>Results</p><p>Of 85 derivation studies included, 48 (56.5%) cited existing CPRs, 33 (38.8%) did not cite any CPR, and four (4.7%) declared there was none to cite. Content analysis identified five categories of existing CPRs insufficiency related to: (1) derivation (5 studies; 11.4% of 44), (2) construct (31 studies; 70.5%), (3) performance (10 studies; 22.7%), (4) transferability (13 studies; 29.5%), and (5) evidence (8 studies; 18.2%). Authors of 54 derivation studies (71.1% of 76 authors contacted) responded to the survey. Twenty-five authors (46.3%) reported they were aware of existing CPR at the time of derivation. Twenty-nine authors (53.7%) declared they conducted a systematic search to identify existing CPRs. Most authors (90.7%) indicated citing existing CPRs was important.</p><p>Conclusion</p><p>Cardiovascular CPRs are often developed without citing existing CPRs although most authors agree it is important. Common justifications for new CPRs concerned construct, including choice of predictor variables or relevance of outcomes. Developers should clearly justify why new CPRs are needed with reference to existing CPRs to avoid unnecessary duplication.</p></div

Flow diagram.

<p>Flow diagram.</p