Perspective Toward Machine Learning Implementation in Pediatric Medicine: Mixed Methods Study

BackgroundGiven the costs of machine learning implementation, a systematic approach to prioritizing which models to implement into clinical practice may be valuable. ObjectiveThe primary objective was to determine the health care attributes respondents at 2 pediatric institutions rate as important when prioritizing machine learning model implementation. The secondary objective was to describe their perspectives on implementation using a qualitative approach. MethodsIn this mixed methods study, we distributed a survey to health system leaders, physicians, and data scientists at 2 pediatric institutions. We asked respondents to rank the following 5 attributes in terms of implementation usefulness: the clinical problem was common, the clinical problem caused substantial morbidity and mortality, risk stratification led to different actions that could reasonably improve patient outcomes, reducing physician workload, and saving money. Important attributes were those ranked as first or second most important. Individual qualitative interviews were conducted with a subsample of respondents. ResultsAmong 613 eligible respondents, 275 (44.9%) responded. Qualitative interviews were conducted with 17 respondents. The most common important attributes were risk stratification leading to different actions (205/275, 74.5%) and clinical problem causing substantial morbidity or mortality (177/275, 64.4%). The attributes considered least important were reducing physician workload and saving money. Qualitative interviews consistently prioritized implementations that improved patient outcomes. ConclusionsRespondents prioritized machine learning model implementation where risk stratification would lead to different actions and clinical problems that caused substantial morbidity and mortality. Implementations that improved patient outcomes were prioritized. These results can help provide a framework for machine learning model implementation

Characterizing the limitations of using diagnosis codes in the context of machine learning for healthcare

Abstract Background Diagnostic codes are commonly used as inputs for clinical prediction models, to create labels for prediction tasks, and to identify cohorts for multicenter network studies. However, the coverage rates of diagnostic codes and their variability across institutions are underexplored. The primary objective was to describe lab- and diagnosis-based labels for 7 selected outcomes at three institutions. Secondary objectives were to describe agreement, sensitivity, and specificity of diagnosis-based labels against lab-based labels. Methods This study included three cohorts: SickKids from The Hospital for Sick Children, and StanfordPeds and StanfordAdults from Stanford Medicine. We included seven clinical outcomes with lab-based definitions: acute kidney injury, hyperkalemia, hypoglycemia, hyponatremia, anemia, neutropenia and thrombocytopenia. For each outcome, we created four lab-based labels (abnormal, mild, moderate and severe) based on test result and one diagnosis-based label. Proportion of admissions with a positive label were presented for each outcome stratified by cohort. Using lab-based labels as the gold standard, agreement using Cohen’s Kappa, sensitivity and specificity were calculated for each lab-based severity level. Results The number of admissions included were: SickKids (n = 59,298), StanfordPeds (n = 24,639) and StanfordAdults (n = 159,985). The proportion of admissions with a positive diagnosis-based label was significantly higher for StanfordPeds compared to SickKids across all outcomes, with odds ratio (99.9% confidence interval) for abnormal diagnosis-based label ranging from 2.2 (1.7–2.7) for neutropenia to 18.4 (10.1–33.4) for hyperkalemia. Lab-based labels were more similar by institution. When using lab-based labels as the gold standard, Cohen’s Kappa and sensitivity were lower at SickKids for all severity levels compared to StanfordPeds. Conclusions Across multiple outcomes, diagnosis codes were consistently different between the two pediatric institutions. This difference was not explained by differences in test results. These results may have implications for machine learning model development and deployment

Clostridioides difficile infection in paediatric patients with cancer and haematopoietic stem cell transplant recipients

BACKGROUND: Epidemiology of Clostridioides difficile infection (CDI) in paediatric cancer patients is uncertain. The primary objective was to describe the prevalence of CDI outcomes among paediatric patients receiving cancer treatments. Secondary objectives were to describe clinical features of CDI, propose a definition of severe CDI and to determine risk factors for CDI clinical outcomes. METHODS: A multi-centre retrospective cohort study that included paediatric patients (1-18 years of age) receiving cancer treatments with CDI. Severe CDI definition was achieved by consensus. Univariable and multivariable regression was conducted to evaluate risk factors for CDI outcomes. RESULTS: There were 627 eligible patients who experienced 721 CDI episodes. The prevalence of clinical cure was 82.9%, recurrence was 9.6%, global cure was 75.0% and repeated new CDI episode was 12.8%. The proposed definition of severe CDI was the presence of colitis, pneumatosis intestinalis, pseudomembranous colitis, ileus or surgery for CDI, occurring in 70 (9.7%) episodes. In univariable regression, initial oral metronidazole or initial oral vancomycin were not significantly associated with failure to achieve clinical cure or CDI recurrence. In multiple regression, oral metronidazole was significantly associated with higher odds (odds ratio (OR) 1.7, 95% confidence interval (CI) 1.0-2.7) and oral vancomycin was significantly associated with lower odds (OR 0.4, 95% CI 0.2-0.8) of repeated new episodes. CONCLUSION: The prevalence of clinical cure was 82.9% and recurrence was 9.6% in pediatric patients receiving cancer treatments. Severe CDI, as per our proposed definition, occurred in 9.7% episodes. Initial oral vancomycin was significantly associated with a reduction in repeated new CDI episodes

Additional file 1 of Characterizing the limitations of using diagnosis codes in the context of machine learning for healthcare

Supplementary Material

Multisite external validation of a risk prediction model for the diagnosis of blood stream infections in febrile pediatric oncology patients without severe neutropenia

BACKGROUND: Pediatric oncology patients are at an increased risk of invasive bacterial infection due to immunosuppression. The risk of such infection in the absence of severe neutropenia (absolute neutrophil count ≥ 500/μL) is not well established and a validated prediction model for blood stream infection (BSI) risk offers clinical usefulness. METHODS: A 6-site retrospective external validation was conducted using a previously published risk prediction model for BSI in febrile pediatric oncology patients without severe neutropenia: the Esbenshade/Vanderbilt (EsVan) model. A reduced model (EsVan2) excluding 2 less clinically reliable variables also was created using the initial EsVan model derivative cohort, and was validated using all 5 external validation cohorts. One data set was used only in sensitivity analyses due to missing some variables. RESULTS: From the 5 primary data sets, there were a total of 1197 febrile episodes and 76 episodes of bacteremia. The overall C statistic for predicting bacteremia was 0.695, with a calibration slope of 0.50 for the original model and a calibration slope of 1.0 when recalibration was applied to the model. The model performed better in predicting high-risk bacteremia (gram-negative or Staphylococcus aureus infection) versus BSI alone, with a C statistic of 0.801 and a calibration slope of 0.65. The EsVan2 model outperformed the EsVan model across data sets with a C statistic of 0.733 for predicting BSI and a C statistic of 0.841 for high-risk BSI. CONCLUSIONS: The results of this external validation demonstrated that the EsVan and EsVan2 models are able to predict BSI across multiple performance sites and, once validated and implemented prospectively, could assist in decision making in clinical practice. Cancer 2017;123:3781–3790

Medical Outcomes, Quality of Life, and Family Perceptions for Outpatient vs Inpatient Neutropenia Management After Chemotherapy for Pediatric Acute Myeloid Leukemia

Importance: Pediatric acute myeloid leukemia (AML) requires multiple courses of intensive chemotherapy that result in neutropenia, with significant risk for infectious complications. Supportive care guidelines recommend hospitalization until neutrophil recovery. However, there are little data to support inpatient over outpatient management. Objective: To evaluate outpatient vs inpatient neutropenia management for pediatric AML. Design, setting, and participants: This cohort study used qualitative and quantitative methods to compare medical outcomes, patient health-related quality of life (HRQOL), and patient and family perceptions between outpatient and inpatient neutropenia management. The study included patients from 17 US pediatric hospitals with frontline chemotherapy start dates ranging from January 2011 to July 2019, although the specific date ranges differed for the individual analyses by design and relative timing. Data were analyzed from August 2019 to February 2020. Exposures: Discharge to outpatient vs inpatient neutropenia management. Main outcomes and measures: The primary outcomes of interest were course-specific bacteremia incidence, times to next course, and patient HRQOL. Course-specific mortality was a secondary medical outcome. Results: Primary quantitative analyses included 554 patients (272 [49.1%] girls and 282 [50.9%] boys; mean [SD] age, 8.2 [6.1] years). Bacteremia incidence was not significantly different during outpatient vs inpatient management (67 courses [23.8%] vs 265 courses [29.0%]; adjusted rate ratio, 0.73; 95% CI, 0.56 to 1.06; P = .08). Outpatient management was not associated with delays to the next course compared with inpatient management (mean [SD] 30.7 [12.2] days vs 32.8 [9.7] days; adjusted mean difference, -2.2; 95% CI, -4.1 to -0.2, P = .03). Mortality during intensification II was higher for patients who received outpatient management compared with those who received inpatient management (3 patients [5.4%] vs 1 patient [0.5%]; P = .03), but comparable with inpatient management at other courses (eg, 0 patients vs 5 patients [1.3%] during induction I; P = .59). Among 97 patients evaluated for HRQOL, outcomes did not differ between outpatient and inpatient management (mean [SD] Pediatric Quality of Life Inventory total score, 70.1 [18.9] vs 68.7 [19.4]; adjusted mean difference, -2.8; 95% CI, -11.2 to 5.6). A total of 86 respondents (20 [23.3%] in outpatient management, 66 [76.7%] in inpatient management) completed qualitative interviews. Independent of management strategy received, 74 respondents (86.0%) expressed satisfaction with their experience. Concerns for hospital-associated infections among caregivers (6 of 7 caregiver respondents [85.7%] who were dissatisfied with inpatient management) and family separation (2 of 2 patient respondents [100%] who were dissatisfied with inpatient management) drove dissatisfaction with inpatient management. Stress of caring for a neutropenic child at home (3 of 3 respondents [100%] who were dissatisfied with outpatient management) drove dissatisfaction with outpatient management. Conclusions and relevance: This cohort study found that outpatient neutropenia management was not associated with higher bacteremia incidence, treatment delays, or worse HRQOL compared with inpatient neutropenia management among pediatric patients with AML. While outpatient management may be safe for many patients, course-specific mortality differences suggest that outpatient management in intensification II should be approached with caution. Patient and family experiences varied, suggesting that outpatient management may be preferred by some but may not be feasible for all families. Further studies to refine and standardize safe outpatient management practices are warranted

Additional file 3 of Symptom management care pathway adaptation process and specific adaptation decisions

Additional file 3: Clinical Practice Guideline and Care Pathway Utilization for Symptom Management at Baseline (N=10

Additional file 2 of Symptom management care pathway adaptation process and specific adaptation decisions

Additional file 2: Options for Adaptations Considering Degree of Bothersome Symptom