Poster presentation from the 38th International Conference on 
Pharmacoepidemiology & Therapeutic Risk 
Managemen

Bloemers, Sarah

Cooper, Toska

Dasgupta, Nabarun

Diprete, Bethany L.

Gibson, Teresa

Irwin, Debra E

Pack, Kenneth

Pence, Brian W

Young, Jessica C

Carolina Digital Repository

External validation of a machine learning algorithm to distinguish death from disenrollment in claims dataBethany L. DiPrete1, Jessica C. Young2, Nabarun Dasgupta3, Kenneth Pack4, Toska Cooper3, Sarah Bloemers4, Brian W. Pence1, Debra E Irwin4, Teresa Gibson41Department of Epidemiology, University of North Carolina at Chapel Hill2Cecil G. Sheps Center for Health Services Research, University of North Carolina at Chapel Hill3Injury Prevention Research Center, University of North Carolina at Chapel Hill4IBM Watson HealthBACKGROUND• A common threat to validity in claims-based research is the inability to distinguish deaths from disenrollment events. • An algorithm was developed using a national administrative claims database with linked date of death to predict death within 61 days of disenrollment• Algorithm performance in national administrative claims database: ROC = 0.933 in beneficiaries <65 years). OBJECTIVE• To conduct an external validation of an algorithm to predict death at disenrollment in order to understand performance of the algorithm in different study populationsMETHODS• Validation data came from a large private health insurance provider in North Carolina (NC) from 2006-2018. • Claims were linked to death records from the NC Department of Health and Human Services using a hierarchical matching algorithm • We identified a sub-population of beneficiaries that met the following criteria:• aged 18-64 years • with a pain diagnosis, surgery, or opioid analgesic, and • one year of continuous enrollment prior to disenrollment. • Individuals were excluded if still enrolled at the end of the study period or 64th year of age. • We evaluated model performance using C-statistics, sensitivity, and specificity overall and in subgroups of interest (age, sex, calendar year, opioid prescriptions). RESULTS• We identified 1,192,813 individuals who disenrolled during the study period, of whom 14,190 (1.2%) died within 61 days of disenrollment.• Overall model performance in the validation data was high (ROC: 0.90) and comparable to performance in the Marketscan <65 population• At a predicted probability threshold of 80%, the algorithm classified 1.3% of disenrollments as deaths with sensitivity of 0.45 and specificity of 0.99• Sensitivity improved at lower probability thresholds. • In subgroup analyses, model performance was a little higher among females (sensitivity: 0.490, specificity: 0.995, ROC: 0.926) than males (sensitivity: 0.422, specificity: 0.990, ROC: 0.870) at the 80% prediction probability threshold• Performance was highest among ages 55-64 (sensitivity: 0.591, specificity: 0.973, ROC: 0.890) and lowest among ages 18-34 (sensitivity: 0.042, specificity: 1.00, ROC: 0.815).• Performance was comparable across calendar time.Table 1. Model performance in the external NC claims cohort (1.2% death rate) vs performance in the MarketScan<65 population (1.6% death rate)• Our external validation demonstrated that the algorithm performed similarly to performance in the original dataset used for algorithm development, with high specificity and varying sensitivity. • This work indicates that this algorithm is generalizable to other commercial claims databases and populations of interest. DISCLOSURESFunding for this project was supplied by the following sources• US Food and Drug Administration (HHSF223201810183C)• National Institute for Drug Abuse (grant No. R21DA046048)The following personal or financial relationships relevant to this presentation existed during the past 12 months/during the conduct of the study:• DiPrete: personal fees from Target RWE/NoviSci outside the submitted work. • Dasgupta: served on an advisory board for the RADARS System of Denver Health outside the submitted workTable 2a. Model performance among individuals who filled an opioid prescription within 30 days of disenrollmentPoster 148Abstract 1193Presented at the 38th International Conference on Pharmacoepidemiology & Therapeutic Risk ManagementCopenhagen, Denmark, August 24-28, 2022CONCLUSIONSContact: diprete@email.unc.edu#ICPE22  |  @UNC_RxEpi  |  @bethanydipretePredicted Probability Threshold% of disenrollmentsabove the threshold PPV NPV Sensitivity Specificity ROC0.1 43.16% 0.131 0.992 0.930 0.6010.9130.2 29.37% 0.184 0.991 0.890 0.7450.3 21.66% 0.238 0.988 0.850 0.8240.4 16.76% 0.293 0.986 0.810 0.8740.5 13.47% 0.350 0.984 0.776 0.9070.6 10.90% 0.413 0.982 0.741 0.9320.7 8.78% 0.480 0.980 0.694 0.9510.8 6.82% 0.557 0.976 0.626 0.9680.9 4.59% 0.668 0.969 0.505 0.9840.95 2.96% 0.740 0.960 0.361 0.992NC Claims Data Cohort MarketScan <65 PopulationPredicted Probability Threshold% of disenrollmentsabove the thresholdPPV NPV Sensitivity Specificity ROC% of disenrollmentsabove the thresholdPPV NPV Sensitivity Specificity ROC0.1 23.15% 0.043 0.997 0.835 0.7760.89924.08% 0.063 0.998 0.900 0.7710.9330.2 12.97% 0.069 0.997 0.750 0.878 9.94% 0.137 0.996 0.809 0.9130.3 7.82% 0.104 0.996 0.685 0.929 4.80% 0.260 0.995 0.739 0.9640.4 4.99% 0.150 0.995 0.631 0.957 3.09% 0.377 0.995 0.690 0.9800.5 3.42% 0.206 0.995 0.591 0.973 2.36% 0.464 0.994 0.650 0.9870.6 2.49% 0.263 0.995 0.552 0.981 1.89% 0.543 0.993 0.609 0.9910.7 1.82% 0.331 0.994 0.508 0.988 1.53% 0.620 0.992 0.561 0.9940.8 1.29% 0.412 0.993 0.448 0.992 1.19% 0.703 0.991 0.498 0.9960.9 0.77% 0.546 0.992 0.352 0.996 0.87% 0.795 0.990 0.410 0.9980.95 0.45% 0.653 0.991 0.247 0.998 0.65% 0.844 0.989 0.325 0.999Predicted Probability Threshold% of disenrollmentsabove the threshold PPV NPV Sensitivity Specificity ROC0.1 21.92% 0.032 0.998 0.796 0.7860.8810.2 11.96% 0.051 0.997 0.691 0.8860.3 6.97% 0.079 0.996 0.616 0.9350.4 4.27% 0.116 0.996 0.556 0.9620.5 2.80% 0.163 0.996 0.513 0.9760.6 1.98% 0.213 0.995 0.473 0.9840.7 1.40% 0.275 0.995 0.431 0.9900.8 0.95% 0.349 0.994 0.374 0.9940.9 0.53% 0.482 0.994 0.288 0.9970.95 0.30% 0.600 0.993 0.200 0.999• Model performance was a higher among individuals who filled a prescription for opioid analgesics within 30 days of disenrollment (sensitivity: 0.626, specificity: 0.968, ROC: 0.913) than those who did not (sensitivity: 0.374, specificity: 0.994, ROC: 0.881) at the 80% prediction probability thresholdTable 2b. Model performance among individuals without an opioid prescription within 30 days of disenrollment

External validation of a machine learning algorithm to distinguish  death from disenrollment in claims data

https://cdr.lib.unc.edu/downloads/k3569d935

External validation of a machine learning algorithm to distinguish death from disenrollment in claims data

Abstract

Similar works

Full text

Available Versions

Carolina Digital Repository