Bias in sensitivity and specificity caused by data-driven selection of optimal cutoff values:Mechanisms, magnitude, and solutions

BACKGROUND: Optimal cutoff values for tests results involving continuous variables are often derived in a data-driven way. This approach, however, may lead to overly optimistic measures of diagnostic accuracy. We evaluated the magnitude of the bias in sensitivity and specificity associated with data-driven selection of cutoff values and examined potential solutions to reduce this bias. METHODS: Different sample sizes, distributions, and prevalences were used in a simulation study. We compared data-driven estimates of accuracy based on the Youden index with the true values and calculated the median bias. Three alternative approaches (assuming a specific distribution, leave-one-out, smoothed ROC curve) were examined for their ability to reduce this bias. RESULTS: The magnitude of bias caused by data-driven optimization of cutoff values was inversely related to sample size. If the true values for sensitivity and specificity are both 84%, the estimates in studies with a sample size of 40 will be approximately 90%. If the sample size increases to 200, the estimates will be 86%. The distribution of the test results had little impact on the amount of bias when sample size was held constant. More robust methods of optimizing cutoff values were less prone to bias, but the performance deteriorated if the underlying assumptions were not met. CONCLUSIONS: Data-driven selection of the optimal cutoff value can lead to overly optimistic estimates of sensitivity and specificity, especially in small studies. Alternative methods can reduce this bias, but finding robust estimates for cutoff values and accuracy requires considerable sample size

The flare in alkaline phosphatase activity post-orchidectomy predicts which patient may benefit from early chemotherapy in metastatic prostate cancer

BACKGROUND: A flare in serum alkaline phosphatase (ALP) activity post-orchidectomy has been shown to be of negative prognostic value for progression-free survival (PFS) in patients with prostate cancer. The aim of this study was to investigate whether a flare in ALP may help identify patients in whom prognosis could be positively influenced by early chemotherapy. METHODS: A retrospective analysis of the database of a Dutch multicenter study was conducted to evaluate the prognostic value of the flare in ALP post-orchidectomy for survival and PFS in 112 patients treated with orchidectomy (previously reported) compared to 121 age- and stage-matched patients additionally treated with estramustine-phosphate (EMP) as first line therapy. RESULTS: There was no overall difference in PFS and survival between the two treatment regimen. Subgroup analysis of patients demonstrating a greater than 50% increase in ALP post-orchidectomy showed, however, a significant increase in PFS in patients additionally treated with EMP. CONCLUSIONS: Our data suggest that the simple measurement of ALP activity within 4 weeks of castration represents a useful adjunct in assessing which patients with prostate cancer undergoing androgen ablation may benefit from additional early chemotherap

Contemporary prevalence of pulmonary arterial hypertension in adult congenital heart disease following the updated clinical classification

The aging congenital heart disease (CHD) population is prone to develop a variety of sequelae, including pulmonary arterial hypertension (PAH). Previous prevalence estimates are limited in applicability due to the use of tertiary centers, or database encoding only. We aimed to investigate the contemporary prevalence of PAH in adult CHD patients, using a nationwide population. A cross-sectional study was performed, using the population-based Dutch CONgenital CORvitia (CONCOR) registry. All patients born with a systemic-to-pulmonary shunt, thereby at risk of developing PAH, were identified. From this cohort, a random sample was obtained and carefully reviewed. Of 12,624 registered adults with CHD alive in 2011, 5,487 (44%) were at risk of PAH. The random sample consisted of 1,814 patients (mean age 40 ± 15 years) and 135 PAH cases were observed. PAH prevalence in patients born with a systemic-to-pulmonary shunt was 7.4%. The prevalence of PAH after corrective cardiac surgery was remarkably high (5.7%). Furthermore, PAH prevalence increased with age, from 2.5% under 30 years until 35% in the eldest. PAH prevalence in the entire CHD population was 3.2%. Based on 3000 per million adult CHD patients in the general population, we can assume that PAH-CHD is present in 100 per million. This new approach using a nationwide CHD population reports a PAH prevalence of 3.2% in CHD patients, and 100 per million in the general adult population. Especially in patients after shunt closure and the elderly, physicians should be aware of PAH-CHD, to provide optimal therapeutic and clinical car