Avoiding Biased Clinical Machine Learning Model Performance Estimates in the Presence of Label Selection

When evaluating the performance of clinical machine learning models, one must consider the deployment population. When the population of patients with observed labels is only a subset of the deployment population (label selection), standard model performance estimates on the observed population may be misleading. In this study we describe three classes of label selection and simulate five causally distinct scenarios to assess how particular selection mechanisms bias a suite of commonly reported binary machine learning model performance metrics. Simulations reveal that when selection is affected by observed features, naive estimates of model discrimination may be misleading. When selection is affected by labels, naive estimates of calibration fail to reflect reality. We borrow traditional weighting estimators from causal inference literature and find that when selection probabilities are properly specified, they recover full population estimates. We then tackle the real-world task of monitoring the performance of deployed machine learning models whose interactions with clinicians feed-back and affect the selection mechanism of the labels. We train three machine learning models to flag low-yield laboratory diagnostics, and simulate their intended consequence of reducing wasteful laboratory utilization. We find that naive estimates of AUROC on the observed population undershoot actual performance by up to 20%. Such a disparity could be large enough to lead to the wrongful termination of a successful clinical decision support tool. We propose an altered deployment procedure, one that combines injected randomization with traditional weighted estimates, and find it recovers true model performance

DEPLOYR: A technical framework for deploying custom real-time machine learning models into the electronic medical record

Machine learning (ML) applications in healthcare are extensively researched, but successful translations to the bedside are scant. Healthcare institutions are establishing frameworks to govern and promote the implementation of accurate, actionable and reliable models that integrate with clinical workflow. Such governance frameworks require an accompanying technical framework to deploy models in a resource efficient manner. Here we present DEPLOYR, a technical framework for enabling real-time deployment and monitoring of researcher created clinical ML models into a widely used electronic medical record (EMR) system. We discuss core functionality and design decisions, including mechanisms to trigger inference based on actions within EMR software, modules that collect real-time data to make inferences, mechanisms that close-the-loop by displaying inferences back to end-users within their workflow, monitoring modules that track performance of deployed models over time, silent deployment capabilities, and mechanisms to prospectively evaluate a deployed model's impact. We demonstrate the use of DEPLOYR by silently deploying and prospectively evaluating twelve ML models triggered by clinician button-clicks in Stanford Health Care's production instance of Epic. Our study highlights the need and feasibility for such silent deployment, because prospectively measured performance varies from retrospective estimates. By describing DEPLOYR, we aim to inform ML deployment best practices and help bridge the model implementation gap

Reformacija kao proces uspostavljanja i obnavljanja odnosa s Bogom

22q11.2 deletion syndrome (22q11DS)—a neurodevelopmental condition caused by a hemizygous deletion on chromosome 22—is associated with an elevated risk of psychosis and other developmental brain disorders. Prior single-site diffusion magnetic resonance imaging (dMRI) studies have reported altered white matter (WM) microstructure in 22q11DS, but small samples and variable methods have led to contradictory results. Here we present the largest study ever conducted of dMRI-derived measures of WM microstructure in 22q11DS (334 22q11.2 deletion carriers and 260 healthy age- and sex-matched controls; age range 6–52 years). Using harmonization protocols developed by the ENIGMA-DTI working group, we identified widespread reductions in mean, axial and radial diffusivities in 22q11DS, most pronounced in regions with major cortico-cortical and cortico-thalamic fibers: the corona radiata, corpus callosum, superior longitudinal fasciculus, posterior thalamic radiations, and sagittal stratum (Cohen’s d’s ranging from −0.9 to −1.3). Only the posterior limb of the internal capsule (IC), comprised primarily of corticofugal fibers, showed higher axial diffusivity in 22q11DS. 22q11DS patients showed higher mean fractional anisotropy (FA) in callosal and projection fibers (IC and corona radiata) relative to controls, but lower FA than controls in regions with predominantly association fibers. Psychotic illness in 22q11DS was associated with more substantial diffusivity reductions in multiple regions. Overall, these findings indicate large effects of the 22q11.2 deletion on WM microstructure, especially in major cortico-cortical connections. Taken together with findings from animal models, this pattern of abnormalities may reflect disrupted neurogenesis of projection neurons in outer cortical layers

Estimating the efficacy of symptom-based screening for COVID-19

Abstract There is substantial interest in using presenting symptoms to prioritize testing for COVID-19 and establish symptom-based surveillance. However, little is currently known about the specificity of COVID-19 symptoms. To assess the feasibility of symptom-based screening for COVID-19, we used data from tests for common respiratory viruses and SARS-CoV-2 in our health system to measure the ability to correctly classify virus test results based on presenting symptoms. Based on these results, symptom-based screening may not be an effective strategy to identify individuals who should be tested for SARS-CoV-2 infection or to obtain a leading indicator of new COVID-19 cases

Altered white matter microstructure in 22q11.2 deletion syndrome: A multisite diffusion tensor imaging study

22q11.2 deletion syndrome (22q11DS)—a neurodevelopmental condition caused by a hemizygous deletion on chromosome 22—is associated with an elevated risk of psychosis and other developmental brain disorders. Prior single-site diffusion magnetic resonance imaging (dMRI) studies have reported altered white matter (WM) microstructure in 22q11DS, but small samples and variable methods have led to contradictory results. Here we present the largest study ever conducted of dMRI-derived measures of WM microstructure in 22q11DS (334 22q11.2 deletion carriers and 260 healthy age- and sex-matched controls; age range 6–52 years). Using harmonization protocols developed by the ENIGMA-DTI working group, we identified widespread reductions in mean, axial and radial diffusivities in 22q11DS, most pronounced in regions with major cortico-cortical and cortico-thalamic fibers: the corona radiata, corpus callosum, superior longitudinal fasciculus, posterior thalamic radiations, and sagittal stratum (Cohen’s d’s ranging from −0.9 to −1.3). Only the posterior limb of the internal capsule (IC), comprised primarily of corticofugal fibers, showed higher axial diffusivity in 22q11DS. 22q11DS patients showed higher mean fractional anisotropy (FA) in callosal and projection fibers (IC and corona radiata) relative to controls, but lower FA than controls in regions with predominantly association fibers. Psychotic illness in 22q11DS was associated with more substantial diffusivity reductions in multiple regions. Overall, these findings indicate large effects of the 22q11.2 deletion on WM microstructure, especially in major cortico-cortical connections. Taken together with findings from animal models, this pattern of abnormalities may reflect disrupted neurogenesis of projection neurons in outer cortical layers.Sin financiación15.992 JCR (2020) Q1, 10/295 Biochemistry & Molecular Biology5.071 SJR (2020) Q1, 5/85 Cellular and Molecular NeuroscienceNo data IDR 2020UE

Altered white matter microstructure in 22q11.2 deletion syndrome: a multisite diffusion tensor imaging study

22q11.2 deletion syndrome (22q11DS)-a neurodevelopmental condition caused by a hemizygous deletion on chromosome 22-is associated with an elevated risk of psychosis and other developmental brain disorders. Prior single-site diffusion magnetic resonance imaging (dMRI) studies have reported altered white matter (WM) microstructure in 22q11DS, but small samples and variable methods have led to contradictory results. Here we present the largest study ever conducted of dMRI-derived measures of WM microstructure in 22q11DS (334 22q11.2 deletion carriers and 260 healthy age- and sex-matched controls; age range 6-52 years). Using harmonization protocols developed by the ENIGMA-DTI working group, we identified widespread reductions in mean, axial and radial diffusivities in 22q11DS, most pronounced in regions with major cortico-cortical and cortico-thalamic fibers: the corona radiata, corpus callosum, superior longitudinal fasciculus, posterior thalamic radiations, and sagittal stratum (Cohen's d's ranging from -0.9 to -1.3). Only the posterior limb of the internal capsule (IC), comprised primarily of corticofugal fibers, showed higher axial diffusivity in 22q11DS. 22q11DS patients showed higher mean fractional anisotropy (FA) in callosal and projection fibers (IC and corona radiata) relative to controls, but lower FA than controls in regions with predominantly association fibers. Psychotic illness in 22q11DS was associated with more substantial diffusivity reductions in multiple regions. Overall, these findings indicate large effects of the 22q11.2 deletion on WM microstructure, especially in major cortico-cortical connections. Taken together with findings from animal models, this pattern of abnormalities may reflect disrupted neurogenesis of projection neurons in outer cortical layers

Altered white matter microstructure in 22q11.2 deletion syndrome: a multisite diffusion tensor imaging study.

22q11.2 deletion syndrome (22q11DS)-a neurodevelopmental condition caused by a hemizygous deletion on chromosome 22-is associated with an elevated risk of psychosis and other developmental brain disorders. Prior single-site diffusion magnetic resonance imaging (dMRI) studies have reported altered white matter (WM) microstructure in 22q11DS, but small samples and variable methods have led to contradictory results. Here we present the largest study ever conducted of dMRI-derived measures of WM microstructure in 22q11DS (334 22q11.2 deletion carriers and 260 healthy age- and sex-matched controls; age range 6-52 years). Using harmonization protocols developed by the ENIGMA-DTI working group, we identified widespread reductions in mean, axial and radial diffusivities in 22q11DS, most pronounced in regions with major cortico-cortical and cortico-thalamic fibers: the corona radiata, corpus callosum, superior longitudinal fasciculus, posterior thalamic radiations, and sagittal stratum (Cohen's d's ranging from -0.9 to -1.3). Only the posterior limb of the internal capsule (IC), comprised primarily of corticofugal fibers, showed higher axial diffusivity in 22q11DS. 22q11DS patients showed higher mean fractional anisotropy (FA) in callosal and projection fibers (IC and corona radiata) relative to controls, but lower FA than controls in regions with predominantly association fibers. Psychotic illness in 22q11DS was associated with more substantial diffusivity reductions in multiple regions. Overall, these findings indicate large effects of the 22q11.2 deletion on WM microstructure, especially in major cortico-cortical connections. Taken together with findings from animal models, this pattern of abnormalities may reflect disrupted neurogenesis of projection neurons in outer cortical layers

Elective cancer surgery in COVID-19-free surgical pathways during the SARS-CoV-2 pandemic : an international, multicenter, comparative cohort study

PURPOSE As cancer surgery restarts after the first COVID-19 wave, health care providers urgently require data to determine where elective surgery is best performed. This study aimed to determine whether COVID-19-free surgical pathways were associated with lower postoperative pulmonary complication rates compared with hospitals with no defined pathway. PATIENTS AND METHODS This international, multicenter cohort study included patients who underwent elective surgery for 10 solid cancer types without preoperative suspicion of SARS-CoV-2. Participating hospitals included patients from local emergence of SARS-CoV-2 until April 19, 2020. At the time of surgery, hospitals were defined as having a COVID-19-free surgical pathway (complete segregation of the operating theater, critical care, and inpatient ward areas) or no defined pathway (incomplete or no segregation, areas shared with patients with COVID-19). The primary outcome was 30-day postoperative pulmonary complications (pneumonia, acute respiratory distress syndrome, unexpected ventilation). RESULTS Of 9,171 patients from 447 hospitals in 55 countries, 2,481 were operated on in COVID-19-free surgical pathways. Patients who underwent surgery within COVID-19-free surgical pathways were younger with fewer comorbidities than those in hospitals with no defined pathway but with similar proportions of major surgery. After adjustment, pulmonary complication rates were lower with COVID-19-free surgical pathways (2.2% v 4.9%; adjusted odds ratio [aOR], 0.62; 95% CI, 0.44 to 0.86). This was consistent in sensitivity analyses for low-risk patients (American Society of Anesthesiologists grade 1/2), propensity score-matched models, and patients with negative SARS-CoV-2 preoperative tests. The postoperative SARS-CoV-2 infection rate was also lower in COVID-19-free surgical pathways (2.1% v 3.6%; aOR, 0.53; 95% CI, 0.36 to 0.76). CONCLUSION Within available resources, dedicated COVID-19-free surgical pathways should be established to provide safe elective cancer surgery during current and before future SARS-CoV-2 outbreaks

Delaying surgery for patients with a previous SARS-CoV-2 infection

Not availabl