On the dependence of the critical success index (CSI) on prevalence

The critical success index (CSI) is an established metric used in meteorology to verify the accuracy of weather forecasts. It is defined as the ratio of hits to the sum of hits, false alarms, and misses. Translationally, CSI has gained popularity as a unitary outcome measure in various clinical situations where large numbers of true negatives may influence the interpretation of other, more traditional, outcome measures, such as specificity (Spec) and negative predictive value (NPV), or when unified interpretation of positive predictive value (PPV) and sensitivity (Sens) is needed. The derivation of CSI from measures including PPV has prompted questions as to whether and how CSI values may vary with disease prevalence (P), just as PPV estimates are dependent on P, and hence whether CSI values are generalizable between studies with differing prevalences. As no detailed study of the relation of CSI to prevalence has been undertaken hitherto, the dataset of a previously published test accuracy study of a cognitive screening instrument was interrogated to address this question. Three different methods were used to examine the change in CSI across a range of prevalences, using both the Bayes formula and equations directly relating CSI to Sens, PPV, P, and the test threshold (Q). These approaches showed that, as expected, CSI does vary with prevalence, but the dependence differs according to the method of calculation that is adopted. Bayesian rescaling of both Sens and PPV generates a concave curve, suggesting that CSI will be maximal at a particular prevalence, which may vary according to the particular dataset

Case-control study developing Scottish Epilepsy Deaths Study score to predict epilepsy-related death

This study aims to develop a risk prediction model for epilepsy-related death in adults. In this age- and sex-matched case-control study, we compared adults (aged ≥16 years) who had epilepsy-related death between 2009-2016 to living adults with epilepsy in Scotland. Cases were identified from validated administrative national datasets linked to mortality records. ICD-10 cause-of-death coding was used to define epilepsy-related death. Controls were recruited from a research database and epilepsy clinics. Clinical data from medical records were abstracted and used to undertake univariable and multivariable conditional logistic regression to develop a risk prediction model consisting of four variables chosen a priori. A weighted sum of the factors present was taken to create a risk index - the Scottish Epilepsy Deaths Study Score (SEDS Score). Odds ratios (OR) were estimated with 95% confidence intervals (CIs). 224 deceased cases (mean age 48 years, 114 male) and 224 matched living controls were compared. In univariable analysis, predictors of epilepsy-related death were recent epilepsy-related accident and emergency (A&E) attendance (OR 5.1, 95% CI 3.2-8.3), living in deprived areas (OR 2.5, 95% CI 1.6-4.0), developmental epilepsy (OR 3.1, 95% CI 1.7-5.7), raised Charlson Comorbidity Index (CCI) score (OR 2.5, 95% CI 1.2-5.2), alcohol abuse (OR 4.4, 95% CI 2.2-9.2), absent recent neurology review (OR 3.8, 95% CI 2.4-6.1), and generalised epilepsy (OR 1.9, 95% CI 1.2-3.0). SEDS Score model variables were derived from the first four listed above, with CCI ≥2 given 1 point, living in the two most deprived areas given 2 points, having an inherited or congenital aetiology or risk factor for developing epilepsy given 2 points, and recent epilepsy-related A&E attendance given 3 points. Compared to having a SEDS Score of 0, those with a SEDS Score of 1 remained low risk, with OR 1.6 (95% CI 0.5-4.8). Those with a SEDS Score of 2-3 had moderate risk, with OR 2.8 (95% CI 1.3-6.2). Those with a SEDS Score of 4-5 and 6-8 were high risk, with OR 14.4 (95% CI 5.9-35.2) and 24.0 (95% CI 8.1-71.2), respectively. The SEDS Score may be a helpful tool for identifying adults at high risk of epilepsy-related death and requires external validation

Critical success index or F measure to validate the accuracy of administrative healthcare data identifying epilepsy in deceased adults in Scotland

Background: Methods to undertake diagnostic accuracy studies of administrative epilepsy data are challenged bylack of a way to reliably rank case-ascertainment algorithms in order of their accuracy. This is because it isdifficult to know how to prioritise positive predictive value (PPV) and sensitivity (Sens). Large numbers of truenegative (TN) instances frequently found in epilepsy studies make it difficult to discriminate algorithm accuracyon the basis of negative predictive value (NPV) and specificity (Spec) as these become inflated (usually >90%).This study demonstrates the complementary value of using weather forecasting or machine learning metricscritical success index (CSI) or F measure, respectively, as unitary metrics combining PPV and sensitivity. Wereanalyse data published in a diagnostic accuracy study of administrative epilepsy mortality data in Scotland.Method: CSI was calculated as 1/[(1/PPV) + (1/Sens) – 1]. F measure was calculated as 2.PPV.Sens/(PPV +Sens). CSI and F values range from 0 to 1, interpreted as 0 = inaccurate prediction and 1 = perfect accuracy. Thepublished algorithms were reanalysed using these and their accuracy re-ranked according to CSI in order to allowcomparison to the original rankings.Results: CSI scores were conservative (range 0.02–0.826), always less than or equal to the lower of the correspondingPPV (range 39–100%) and sensitivity (range 2–93%). F values were less conservative (range0.039–0.905), sometimes higher than either PPV or sensitivity, but were always higher than CSI. Low CSI and Fvalues occurred when there was a large difference between PPV and sensitivity, e.g. CSI was 0.02 and F was0.039 in an instance when PPV was 100% and sensitivity was 2%. Algorithms with both high PPV and sensitivityperformed best in terms of CSI and F measure, e.g. CSI was 0.826 and F was 0.905 in an instance when PPV was90% and sensitivity was 91%.Conclusion: CSI or F measure can combine PPV and sensitivity values into a convenient single metric that is easierto interpret and rank in terms of diagnostic accuracy than trying to rank diagnostic accuracy according to the twomeasures themselves. CSI or F prioritise instances where both PPV and sensitivity are high over instances wherethere are large differences between PPV and sensitivity (even if one of these is very high), allowing diagnosticaccuracy thresholds based on combined PPV and sensitivity to be determined. Therefore, CSI or F measures maybe helpful complementary metrics to report alongside PPV and sensitivity in diagnostic accuracy studies ofadministrative epilepsy data

Association of antiseizure medications and adverse cardiovascular events: A global health federated network analysis

AbstractObjectiveA diagnosis of epilepsy has been associated with adverse cardiovascular events (CEs), but the extent to which antiseizure medications (ASMs) may contribute to this is not well understood. The aim of this study was to compare the risk of adverse CEs associated with ASM in patients with epilepsy (PWE).MethodsA retrospective case–control cohort study was conducted using TriNetX, a global health federated network of anonymized patient records. Patients older than 18 years, with a diagnosis of epilepsy (International Classification of Diseases, 10th Revision code G40) and a medication code of carbamazepine, lamotrigine, or valproate were compared. Patients with cardiovascular disease prior to the diagnosis of epilepsy were excluded. Cohorts were 1:1 propensity score matched (PSM) according to age, sex, ethnicity, hypertension, heart failure, atherosclerotic heart disease, atrial and cardiac arrythmias, diabetes, disorders of lipoprotein metabolism, obesity, schizophrenia and bipolar disorder, medications, and epilepsy classification. The primary outcome was a composite of adverse CEs (ischemic stroke, acute ischemic heart disease, and heart failure) at 10 years. Cox regression analyses were used to calculate hazard ratios (HRs) and 95% confidence intervals (CIs) following 1:1 PSM.ResultsOf 374 950 PWE included; three cohorts were established after PSM: (1) carbamazepine compared to lamotrigine, n = 4722, mean age 37.4 years; (2) valproate compared to lamotrigine, n = 5478, mean age 33.9 years; and (3) valproate compared to carbamazepine, n = 4544, mean age 37.0 years. Carbamazepine and valproate use were associated with significantly higher risk of composite cardiovascular outcome compared to lamotrigine (HR = 1.390, 95% CI = 1.160–1.665 and HR = 1.264, 95% CI = 1.050–1.521, respectively). Valproate was associated with a 10‐year higher risk of all‐cause death than carbamazepine (HR = 1.226, 95% CI = 1.017–1.478), but risk of other events was not significantly different.SignificanceCarbamazepine and valproate were associated with increased CE risks compared to lamotrigine. Cardiovascular risk factor monitoring and careful follow‐up should be considered for these patients.</jats:sec