Systematic Review and Meta-Analysis of Validation Studies on a Diabetes Case Definition from Health Administrative Records

<div><p>Objectives</p><p>Health administrative data are frequently used for diabetes surveillance. We aimed to determine the sensitivity and specificity of a commonly-used diabetes case definition (two physician claims or one hospital discharge abstract record within a two-year period) and their potential effect on prevalence estimation.</p><p>Methods</p><p>Following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, we searched Medline (from 1950) and Embase (from 1980) databases for validation studies through August 2012 (keywords: “diabetes mellitus”; “administrative databases”; “validation studies”). Reviewers abstracted data with standardized forms and assessed quality using Quality Assessment of Diagnostic Accuracy Studies (QUADAS) criteria. A generalized linear model approach to random-effects bivariate regression meta-analysis was used to pool sensitivity and specificity estimates. We applied correction factors derived from pooled sensitivity and specificity estimates to prevalence estimates from national surveillance reports and projected prevalence estimates over 10 years (to 2018).</p><p>Results</p><p>The search strategy identified 1423 abstracts among which 11 studies were deemed relevant and reviewed; 6 of these reported sensitivity and specificity allowing pooling in a meta-analysis. Compared to surveys or medical records, sensitivity was 82.3% (95%CI 75.8, 87.4) and specificity was 97.9% (95%CI 96.5, 98.8). The diabetes case definition underestimated prevalence when it was ≤10.6% and overestimated prevalence otherwise.</p><p>Conclusion</p><p>The diabetes case definition examined misses up to one fifth of diabetes cases and wrongly identifies diabetes in approximately 2% of the population. This may be sufficiently sensitive and specific for surveillance purposes, in particular monitoring prevalence trends. Applying correction factors to adjust prevalence estimates from this definition may be helpful to increase accuracy of estimates.</p></div

Flow diagram of selection strategy and article reviews.

<p>Flow diagram is in accordance with Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA).</p

Test properties of the NDSS case definition.

<p>AHS: Alberta Health Services; CCHS: Canadian Community Health Survey; CLS: Calgary Laboratory Services; EMR: Electronic Medical Records; GP: General Practitioner; HEDIS: Health Plan Employer Data and Information Set; HMO: Health Maintenance Organization; Kappa: Kappa statistic; MCBS: Medicare Current Beneficiary Survey; MHSC: Manitoba Health Services Commission; MHHP: Manitoba Heart Health Project; MHSIP: Manitoba Health Services Insurance Plan; NDSS: National Diabetes Surveillance System; NPHS: National Population Health Survey; NPV: Negative predictive value; ODB: Ontario Drug Benefit; ODD: Ontario Diabetes Database; OHI: Ontario health Insurance Plan; Sens: Sensitivity; SHB: Saskatchewan health beneficiaries; Spec: Specificity; PPV: Positive predictive value.</p>|<p>Included in meta-analysis.</p>◊<p>Test estimates and 95% confidence intervals which were not reported in the original paper were manually calculated from available raw data.</p>||<p>95% confidence intervals were estimated based on a diabetes prevalence of 6.8% via the ODD and a sample size of 4691 from the NPHS cycle 3 1998/9.</p>*<p>Test properties were recalculated to designate self-reported diabetes from survey as the reference standard.</p><p>The sample size of the ODB were not explicitly stated in the paper. Hence, the 95%CIs could not be calculated.</p><p>-Demographic data was not reported in the study.</p>∫<p>the authors validated one physician claim or one hospitalization which is similar but not identical to the NDSS criteria.</p><p>@test measures of the case algorithm closet to the NDSS criteria is displayed in the table.</p>¢<p>Test measures were calculated using the reference standard closest to current diabetes diagnosis criteria (i.e. CDA guidelines for diagnostic criteria of diabetes OR glycated hemoglobin ≥6.7%) was displayed in the table.</p><p>^△The sensitivity was calculated by projecting the case-control design (n = 145 696) to the entire Saskatchewan population where 625 994 individuals would not have met the NDSS criteria of which 3443 would have been identified as having diabetes based in medication alone.</p

Random-effects bivariate regression analysis of the pooled test accuracies from 6 studies.

<p>The Hierarchical Summary Receiver Operator Characteristics (HSROC) curve displays the 95% confidence interval of the summary operating point and the 95% prediction region, which is the confidence region for a forecast of the true sensitivity and specificity in a future study. The shape of the prediction region is generated based on the assumption of a bivariate normal distribution for the random effects model. The Empirical Bayes estimate gives the best estimate of the true sensitivity and specificity of each study and these estimates will be shrunk towards the summary point compared with the study-specific estimates. The stronger the shrinkage, the greater the precision of the test estimate. The random-effects bivariate regression analysis could not be done for the subgroups stratified by validation method because the small number of studies.</p

Crude and adjusted prevalence of diabetes in Canada.

<p>Crude prevalence: prevalence of diabetes in Canada for fiscal years 2002/3 through 2006/7 obtained from the NDSS 2009 report <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0075256#pone.0075256-Svensen1" target="_blank">[25]</a>; Adjusted prevalence: prevalence after applying correction factors [(Prevalence(%) - 2.1)/0.802)]; The margins of error for all adjusted prevalence and crude prevalence estimates were ∼0.01% (n∼25 000 000). Projected crude prevalence: future prevalence assuming an increase of 0.4% per year; Projected adjusted prevalence: future prevalence after applying correction factors; Total diabetes: Estimated prevalence of physician-diagnosed and undiagnosed diabetes assuming 1/3 of total diabetes is undiagnosed. The crossover point of the crude and adjusted prevalence lines is ∼10.6% around year 2013.</p

Forest plots of sensitivities and specificities of the NDSS case definition reported by included validation studies.

<p>ES (95%CI): Summary estimate (95% confidence interval); Charts: Reference standard by medical chart review; Survey: Reference standard by patient self-report from population-based survey.</p

Schematic representation of Mendelian randomization analysis.

<p>The leftmost box lists SNPs that were genome-wide significant for 25OHD level in SUNLIGHT (<i>n</i> = 33,996). The blue arrow represents the effect of SNPs on multiply adjusted natural-log-transformed 25OHD level using data from CaMos (<i>n</i> = 2,347). The green arrow represents the causal association of decreased 25OHD level with the risk of MS using data from the largest genetic association study to date for MS (the IMSGC Immunochip study, up to 14,498 cases and 24,091 healthy controls).</p

Mendelian randomization estimate of the association of 25OHD level with risk of multiple sclerosis.

<p>Estimates obtained using a fixed-effects model.</p

Mendelian randomization estimate of the association of 25OHD level with risk of multiple sclerosis excluding the <i>DHCR7</i> locus.

<p>Estimates obtained using a fixed-effects model.</p

Mendelian randomization estimate of the association of decreased 25OHD with the risk of multiple sclerosis stratified by SNPs near genes involved in 25OHD synthesis versus metabolism using a fixed-effects model.

<p>^aOR is expressed as the odds of MS for a 1-SD decrease in natural-log-transformed 25OHD level. Note that the 95% CI for the <i>I</i>² cannot be properly estimated given that there are only two SNPs per model.</p><p>Mendelian randomization estimate of the association of decreased 25OHD with the risk of multiple sclerosis stratified by SNPs near genes involved in 25OHD synthesis versus metabolism using a fixed-effects model.</p