Development and evaluation of uncertainty quantifying machine learning models to predict piperacillin plasma concentrations in critically ill patients

Background: Beta-lactam antimicrobial concentrations are frequently suboptimal in critically ill patients. Population pharmacokinetic (PopPK) modeling is the golden standard to predict drug concentrations. However, currently available PopPK models often lack predictive accuracy, making them less suited to guide dosing regimen adaptations. Furthermore, many currently developed models for clinical applications often lack uncertainty quantification. We, therefore, aimed to develop machine learning (ML) models for the prediction of piperacillin plasma concentrations while also providing uncertainty quantification with the aim of clinical practice. Methods: Blood samples for piperacillin analysis were prospectively collected from critically ill patients receiving continuous infusion of piperacillin/tazobactam. Interpretable ML models for the prediction of piperacillin concentrations were designed using CatBoost and Gaussian processes. Distribution-based Uncertainty Quantification was added to the CatBoost model using a proposed Quantile Ensemble method, useable for any model optimizing a quantile function. These models are subsequently evaluated using the distribution coverage error, a proposed interpretable uncertainty quantification calibration metric. Development and internal evaluation of the ML models were performed on the Ghent University Hospital database (752 piperacillin concentrations from 282 patients). Ensuing, ML models were compared with a published PopPK model on a database from the University Medical Centre of Groningen where a different dosing regimen is used (46 piperacillin concentrations from 15 patients.). Results: The best performing model was the Catboost model with an RMSE and R-2 of 31.94-0.64 and 33.53-0.60 for internal evaluation with and without previous concentration. Furthermore, the results prove the added value of the proposed Quantile Ensemble model in providing clinically useful individualized uncertainty predictions and show the limits of homoscedastic methods like Gaussian Processes in clinical applications. Conclusions: Our results show that ML models can consistently estimate piperacillin concentrations with acceptable and high predictive accuracy when identical dosing regimens as in the training data are used while providing highly relevant uncertainty predictions. However, generalization capabilities to other dosing schemes are limited. Notwithstanding, incorporating ML models in therapeutic drug monitoring programs seems definitely promising and the current work provides a basis for validating the model in clinical practice

Powershap: A Power-full Shapley Feature Selection Method

Feature selection is a crucial step in developing robust and powerful machine learning models. Feature selection techniques can be divided into two categories: filter and wrapper methods. While wrapper methods commonly result in strong predictive performances, they suffer from a large computational complexity and therefore take a significant amount of time to complete, especially when dealing with high-dimensional feature sets. Alternatively, filter methods are considerably faster, but suffer from several other disadvantages, such as (i) requiring a threshold value, (ii) not taking into account intercorrelation between features, and (iii) ignoring feature interactions with the model. To this end, we present powershap, a novel wrapper feature selection method, which leverages statistical hypothesis testing and power calculations in combination with Shapley values for quick and intuitive feature selection. Powershap is built on the core assumption that an informative feature will have a larger impact on the prediction compared to a known random feature. Benchmarks and simulations show that powershap outperforms other filter methods with predictive performances on par with wrapper methods while being significantly faster, often even reaching half or a third of the execution time. As such, powershap provides a competitive and quick algorithm that can be used by various models in different domains. Furthermore, powershap is implemented as a plug-and-play and open-source sklearn component, enabling easy integration in conventional data science pipelines. User experience is even further enhanced by also providing an automatic mode that automatically tunes the hyper-parameters of the powershap algorithm, allowing to use the algorithm without any configuration needed

PowerShap : a power-full Shapley feature selection method

Feature selection is a crucial step in developing robust and powerful machine learning models. Feature selection techniques can be divided into two categories: filter and wrapper methods. While wrapper methods commonly result in strong predictive performances, they suffer from a large computational complexity and therefore take a significant amount of time to complete, especially when dealing with high-dimensional feature sets. Alternatively, filter methods are considerably faster, but suffer from several other disadvantages, such as (i) requiring a threshold value, (ii) many filter methods not taking into account intercorrelation between features, and (iii) ignoring feature interactions with the model. To this end, we present PowerShap, a novel wrapper feature selection method, which leverages statistical hypothesis testing and power calculations in combination with Shapley values for quick and intuitive feature selection. PowerShap is built on the core assumption that an informative feature will have a larger impact on the prediction compared to a known random feature. Benchmarks and simulations show that PowerShap outperforms other filter methods with predictive performances on par with wrapper methods while being significantly faster, often even reaching half or a third of the execution time. As such, PowerShap provides a competitive and quick algorithm that can be used by various models in different domains. Furthermore, PowerShap is implemented as a plug-and-play and open-source sklearn component, enabling easy integration in conventional data science pipelines. User experience is even further enhanced by also providing an automatic mode that automatically tunes the hyper-parameters of the PowerShap algorithm, allowing to use the algorithm without any configuration needed

Generalizable calibrated machine learning models for real-time atrial fibrillation risk prediction in ICU patients

Background: Atrial Fibrillation (AF) is the most common arrhythmia in the intensive care unit (ICU) and is associated with increased morbidity and mortality. Identification of patients at risk for AF is not routinely performed as AF prediction models are almost solely developed for the general population or for particular ICU populations. However, early AF risk identification could help to take targeted preemptive actions and possibly reduce morbidity and mortality. Predictive models need to be validated across hospitals with different standards of care and convey their predictions in a clinically useful manner. Therefore, we designed AF risk models for ICU patients using uncertainty quantification to provide a risk score and evaluated them on multiple ICU datasets. Methods: Three CatBoost models, utilizing feature windows comprising data 1.5-13.5, 6-18, or 12-24 hours before AF occurrence, were built using 2-repeat-10-fold cross-validation on AmsterdamUMCdb, the first freely available European ICU database. Furthermore, AF Patients were matched with no-AF patients for training. Transferability was validated using a direct and a recalibration evaluation on two independent external datasets, MIMIC-IV and GUH. The calibration of the predicted probability, used as an AF risk score, was measured using the Expected Calibration Error (ECE) and the presented Expected Signed Calibration Error (ESCE). Additionally, all models were evaluated across time during the ICU stay. Results: The model performance reached Areas Under the Curve (AUCs) of 0.81 at internal validation. Direct external validation showed partial generalizability with AUCs reaching 0.77. However, recalibration resulted in performances matching or exceeding that of the internal validation. All models furthermore showed calibration capabilities demonstrating adequate risk prediction competence. Conclusion: Ultimately, recalibrating models reduces the challenge of generalization to unseen datasets. Moreover, utilizing the patient-matching methodology together with the assessment of uncertainty calibration can serve as a step toward the development of clinical AF prediction models

Pathogen-based target attainment of optimized continuous infusion dosing regimens of piperacillin-tazobactam and meropenem in surgical ICU patients: a prospective single center observational study

Abstract Background Several studies have indicated that commonly used piperacillin-tazobactam (TZP) and meropenem (MEM) dosing regimens lead to suboptimal plasma concentrations for a range of pharmacokinetic/pharmacodynamic (PK/PD) targets in intensive care unit (ICU) patients. These targets are often based on a hypothetical worst-case scenario, possibly overestimating the percentage of suboptimal concentrations. We aimed to evaluate the pathogen-based clinically relevant target attainment (CRTA) and therapeutic range attainment (TRA) of optimized continuous infusion dosing regimens of TZP and MEM in surgical ICU patients. Methods A single center prospective observational study was conducted between March 2016 and April 2019. Free plasma concentrations were calculated by correcting total plasma concentrations, determined on remnants of blood gas samples by ultra-performance liquid chromatography with tandem mass spectrometry, for their protein binding. Break points (BP) of identified pathogens were derived from epidemiological cut-off values. CRTA was defined as a corrected measured total serum concentration above the BP and calculated for increasing BP multiplications up to 6 × BP. The upper limit of the therapeutic range was set at 157.2 mg/L for TZP and 45 mg/L for MEM. As a worst-case scenario, a BP of 16 mg/L for TZP and 2 mg/L for MEM was used. Results 781 unique patients were included with 1036 distinctive beta-lactam antimicrobial prescriptions (731 TZP, 305 MEM) for 1003 unique infections/prophylactic regimens (750 TZP, 323 MEM). 2810 samples were available (1892 TZP, 918 MEM). The median corrected plasma concentration for TZP was 86.4 mg/L [IQR 56.2–148] and 16.2 mg/L [10.2–25.5] for MEM. CRTA and TRA was consistently higher for the pathogen-based scenario than for the worst-case scenario, but nonetheless, a substantial proportion of samples did not attain commonly used PK/PD targets. Conclusion Despite these pathogen-based data demonstrating that CRTA and TRA is higher than in the often-used theoretical worst-case scenario, a substantial proportion of samples did not attain commonly used PK/PD targets when using optimised continuous infusion dosing regimens. Therefore, more dosing optimization research seems warranted. At the same time, a ‘pathogen-based analysis’ approach might prove to be more sensible than a worst-case scenario approach when evaluating target attainment and linked clinical outcomes