Prediction models for clustered data: comparison of a random intercept and standard regression model

BACKGROUND: When study data are clustered, standard regression analysis is considered inappropriate and analytical techniques for clustered data need to be used. For prediction research in which the interest of predictor effects is on the patient level, random effect regression models are probably preferred over standard regression analysis. It is well known that the random effect parameter estimates and the standard logistic regression parameter estimates are different. Here, we compared random effect and standard logistic regression models for their ability to provide accurate predictions. METHODS: Using an empirical study on 1642 surgical patients at risk of postoperative nausea and vomiting, who were treated by one of 19 anesthesiologists (clusters), we developed prognostic models either with standard or random intercept logistic regression. External validity of these models was assessed in new patients from other anesthesiologists. We supported our results with simulation studies using intra-class correlation coefficients (ICC) of 5%, 15%, or 30%. Standard performance measures and measures adapted for the clustered data structure were estimated. RESULTS: The model developed with random effect analysis showed better discrimination than the standard approach, if the cluster effects were used for risk prediction (standard c-index of 0.69 versus 0.66). In the external validation set, both models showed similar discrimination (standard c-index 0.68 versus 0.67). The simulation study confirmed these results. For datasets with a high ICC (≥15%), model calibration was only adequate in external subjects, if the used performance measure assumed the same data structure as the model development method: standard calibration measures showed good calibration for the standard developed model, calibration measures adapting the clustered data structure showed good calibration for the prediction model with random intercept. CONCLUSION: The models with random intercept discriminate better than the standard model only if the cluster effect is used for predictions. The prediction model with random intercept had good calibration within clusters

The predictive performance and impact of pediatric early warning systems in hospitalized pediatric oncology patients-A systematic review

Pediatric early warning systems (PEWS) arewidely used to identify clinically deteriorating patients. Hospitalized pediatric oncology patients are particularly prone to clinical deterioration. We assessed the PEWS performance to predict early clinical deterioration and the effect of PEWS implementation on patient outcomes in pediatric oncology patients. PubMED, EMBASE, and CINAHL databases were systematically searched from inception up to March 2020. Quality assessment was performed using the Prediction model study Risk-Of-Bias Assessment Tool (PROBAST) and the Cochrane Risk-of-Bias Tool. Nine studies were included. Due to heterogeneity of study designs, outcome measures, and diversity of PEWS, it was not possible to conduct a meta-analysis. Although the studies reported high sensitivity, specificity, and area under the receiver operating characteristics curve (AUROC) of PEWS detecting inpatient deterioration, overall risk of bias of the studies was high. This review highlights limited evidence on the predictive performance of PEWS for clinical deterioration and the effect of PEWS implementation

Barriers and facilitators perceived by physicians when using prediction models in practice

Objectives Prediction models may facilitate risk-based management of health care conditions. In a large cluster-randomized trial, presenting calculated risks of postoperative nausea and vomiting (PONV) to physicians (assistive approach) increased risk-based management of PONV. This increase did not improve patient outcome - that is, PONV incidence. This prompted us to explore how prediction tools guide the decision-making process of physicians. Study Design and Setting Using mixed methods, we interviewed eight physicians to understand how predicted risks were perceived by the physicians and how they influenced decision making. Subsequently, all 57 physicians of the trial were surveyed for how the presented risks influenced their perceptions. Results Although the prediction tool made physicians more aware of PONV prevention, the physicians reported three barriers to use predicted risks in their decision making. PONV was not considered an outcome of utmost importance; decision making on PONV prophylaxis was mostly intuitive rather than risk based; prediction models do not weigh benefits and risks of prophylactic drugs. Conclusion Combining probabilistic output of the model with their clinical experience may be difficult for physicians, especially when their decision-making process is mostly intuitive. Adding recommendations to predicted risks (directive approach) was considered an important step to facilitate the uptake of a prediction tool

Whetting the SWORD: Detecting Workarounds by Using Active Learning and Logistic Regression

In many organizations, especially in healthcare, workers may work around prescribed procedures. Detecting these workarounds can give insights into difficulties concerning the procedures, which in turn can be used to improve them. Previous studies have shown that workarounds may be discovered from an event log using a set of predefined patterns such as the duration of a trace or the number of resources involved in one. However, domain experts may find it difficult to evaluate and monitor results if there are multiple patterns that indicate workarounds. Training a model that merges the features is often difficult because there are no available datasets covering workarounds. Labeling traces generally requires a lot of time from domain experts. In addition, this would have to be repeated for every new domain, company, or even department since the types of workarounds that occur may differ strongly between them. In this work, we propose to combine the features using a Logistic Regression model and train through Active Learning. In a case study at a hospital, we find that after training the model on only 10 to 15 traces, it stabilizes with an approximate F1 score of .75. This shows that we create and train a model that can detect workarounds well without requiring a large amount of labeled data or a lot of time from a domain expert

Autonomous patient consent for anaesthesia without preoperative consultation: a qualitative feasibility study including low-risk procedures

Background: Informed consent for anaesthesia is mandatory and requires provision of information and subsequent consent during consultation between anaesthesiologist and patient. Although information can be provided in an electronic format, it is unknown whether this a valid substitute for a consultation. We explored whether provision of digital information is equivalent to oral consultation and whether it enables patients to give electronic informed consent (e-consent) for anaesthesia. Methods: Qualitative feasibility study using semi-structured interviews in 20 low-risk adults scheduled for minor surgery under general anaesthesia or procedural sedation at a university hospital. Data were analysed using a thematic content analysis approach. During the interviews, patients followed an application that provides information and subsequent e-consenting. Results: The mean age was 50 yr and patients had good digital skills. Fifteen patients (75%) had previous experience of anaesthesia. The digital application provided enough information for all patients, but eight (40%) preferred consultation with an anaesthesiologist, mainly for personal contact. Patients had different information needs, with previous experiences leading to lower information needs. Nineteen patients had sufficient information to consent autonomously. Most patients considered separate anaesthesia consent superfluous to the surgical consent. Conclusion: The digital application provided sufficient information and patients valued the information offered and the advantage of processing information at their own pace. This information made patients feel empowered to autonomously consent to anaesthesia without consultation. Remarkably, consent for anaesthesia was considered unimportant, because patients felt they had ‘no choice’ if they wanted to undergo surgery

Whetting the SWORD: Detecting Workarounds by Using Active Learning and Logistic Regression

In many organizations, especially in healthcare, workers may work around prescribed procedures. Detecting these workarounds can give insights into difficulties concerning the procedures, which in turn can be used to improve them. Previous studies have shown that workarounds may be discovered from an event log using a set of predefined patterns such as the duration of a trace or the number of resources involved in one. However, domain experts may find it difficult to evaluate and monitor results if there are multiple patterns that indicate workarounds. Training a model that merges the features is often difficult because there are no available datasets covering workarounds. Labeling traces generally requires a lot of time from domain experts. In addition, this would have to be repeated for every new domain, company, or even department since the types of workarounds that occur may differ strongly between them. In this work, we propose to combine the features using a Logistic Regression model and train through Active Learning. In a case study at a hospital, we find that after training the model on only 10 to 15 traces, it stabilizes with an approximate F1 score of.75. This shows that we create and train a model that can detect workarounds well without requiring a large amount of labeled data or a lot of time from a domain expert