Using automated text classification to explore uncertainty in NICE appraisals for drugs for rare diseases

Objective: This study examined the application, feasibility, and validity of supervised learning models for text classification in appraisals for rare disease treatments (RDTs) in relation to uncertainty, and analyzed differences between appraisals based on the classification results. Methods: We analyzed appraisals for RDTs (n = 94) published by the National Institute for Health and Care Excellence (NICE) between January 2011 and May 2023. We used Naïve Bayes, Lasso, and Support Vector Machine models in a binary text classification task (classifying paragraphs as either referencing uncertainty in the evidence base or not). To illustrate the results, we tested hypotheses in relation to the appraisal guidance, advanced therapy medicinal product (ATMP) status, disease area, and age group. Results: The best performing (Lasso) model achieved 83.6 percent classification accuracy (sensitivity = 74.4 percent, specificity = 92.6 percent). Paragraphs classified as referencing uncertainty were significantly more likely to arise in highly specialized technology (HST) appraisals compared to appraisals from the technology appraisal (TA) guidance (adjusted odds ratio = 1.44, 95 percent CI 1.09, 1.90, p = 0.004). There was no significant association between paragraphs classified as referencing uncertainty and appraisals for ATMPs, non-oncology RDTs, and RDTs indicated for children only or adults and children. These results were robust to the threshold value used for classifying paragraphs but were sensitive to the choice of classification model. Conclusion: Using supervised learning models for text classification in NICE appraisals for RDTs is feasible, but the results of downstream analyses may be sensitive to the choice of classification model

How are missing data in covariates handled in observational time-to-event studies in oncology? A systematic review.

BACKGROUND: Missing data in covariates can result in biased estimates and loss of power to detect associations. It can also lead to other challenges in time-to-event analyses including the handling of time-varying effects of covariates, selection of covariates and their flexible modelling. This review aims to describe how researchers approach time-to-event analyses with missing data. METHODS: Medline and Embase were searched for observational time-to-event studies in oncology published from January 2012 to January 2018. The review focused on proportional hazards models or extended Cox models. We investigated the extent and reporting of missing data and how it was addressed in the analysis. Covariate modelling and selection, and assessment of the proportional hazards assumption were also investigated, alongside the treatment of missing data in these procedures. RESULTS: 148 studies were included. The mean proportion of individuals with missingness in any covariate was 32%. 53% of studies used complete-case analysis, and 22% used multiple imputation. In total, 14% of studies stated an assumption concerning missing data and only 34% stated missingness as a limitation. The proportional hazards assumption was checked in 28% of studies, of which, 17% did not state the assessment method. 58% of 144 multivariable models stated their covariate selection procedure with use of a pre-selected set of covariates being the most popular followed by stepwise methods and univariable analyses. Of 69 studies that included continuous covariates, 81% did not assess the appropriateness of the functional form. CONCLUSION: While guidelines for handling missing data in epidemiological studies are in place, this review indicates that few report implementing recommendations in practice. Although missing data are present in many studies, we found that few state clearly how they handled it or the assumptions they have made. Easy-to-implement but potentially biased approaches such as complete-case analysis are most commonly used despite these relying on strong assumptions and where often more appropriate methods should be employed. Authors should be encouraged to follow existing guidelines to address missing data, and increased levels of expectation from journals and editors could be used to improve practice

Comparative effectiveness of second line oral antidiabetic treatments among people with type 2 diabetes mellitus: emulation of a target trial using routinely collected health data.

OBJECTIVE: To compare the effectiveness of three commonly prescribed oral antidiabetic drugs added to metformin for people with type 2 diabetes mellitus requiring second line treatment in routine clinical practice. DESIGN: Cohort study emulating a comparative effectiveness trial (target trial). SETTING: Linked primary care, hospital, and death data in England, 2015-21. PARTICIPANTS: 75 739 adults with type 2 diabetes mellitus who initiated second line oral antidiabetic treatment with a sulfonylurea, DPP-4 inhibitor, or SGLT-2 inhibitor added to metformin. MAIN OUTCOME MEASURES: Primary outcome was absolute change in glycated haemoglobin A1c (HbA1c) between baseline and one year follow-up. Secondary outcomes were change in body mass index (BMI), systolic blood pressure, and estimated glomerular filtration rate (eGFR) at one year and two years, change in HbA1c at two years, and time to ≥40% decline in eGFR, major adverse kidney event, hospital admission for heart failure, major adverse cardiovascular event (MACE), and all cause mortality. Instrumental variable analysis was used to reduce the risk of confounding due to unobserved baseline measures. RESULTS: 75 739 people initiated second line oral antidiabetic treatment with sulfonylureas (n=25 693, 33.9%), DPP-4 inhibitors (n=34 464 ,45.5%), or SGLT-2 inhibitors (n=15 582, 20.6%). SGLT-2 inhibitors were more effective than DPP-4 inhibitors or sulfonylureas in reducing mean HbA1c values between baseline and one year. After the instrumental variable analysis, the mean differences in HbA1c change between baseline and one year were -2.5 mmol/mol (95% confidence interval (CI) -3.7 to -1.3) for SGLT-2 inhibitors versus sulfonylureas and -3.2 mmol/mol (-4.6 to -1.8) for SGLT-2 inhibitors versus DPP-4 inhibitors. SGLT-2 inhibitors were more effective than sulfonylureas or DPP-4 inhibitors in reducing BMI and systolic blood pressure. For some secondary endpoints, evidence for SGLT-2 inhibitors being more effective was lacking-the hazard ratio for MACE, for example, was 0.99 (95% CI 0.61 to 1.62) versus sulfonylureas and 0.91 (0.51 to 1.63) versus DPP-4 inhibitors. SGLT-2 inhibitors had reduced hazards of hospital admission for heart failure compared with DPP-4 inhibitors (0.32, 0.12 to 0.90) and sulfonylureas (0.46, 0.20 to 1.05). The hazard ratio for a ≥40% decline in eGFR indicated a protective effect versus sulfonylureas (0.42, 0.22 to 0.82), with high uncertainty in the estimated hazard ratio versus DPP-4 inhibitors (0.64, 0.29 to 1.43). CONCLUSIONS: This emulation study of a target trial found that SGLT-2 inhibitors were more effective than sulfonylureas or DPP-4 inhibitors in lowering mean HbA1c, BMI, and systolic blood pressure and in reducing the hazards of hospital admission for heart failure (v DPP-4 inhibitors) and kidney disease progression (v sulfonylureas), with no evidence of differences in other clinical endpoints

How are missing data handled in observational time-to-event studies A systematic review.

Missing data in covariates are known to result in biased estimates of association with the outcome and loss of power to detect associations. Missing data can also lead to other challenges in time-to-event analyses including the handling of time-varying effects of covariates, selection of covariates and their flexible modelling. This review aimed to understand how researchers are approaching time-to-event analyses when missing data are present. Medline and Embase were searched for observational time-to-event studies published from January 2011 to January 2018. We assessed the covariate selection procedure, assumptions of proportional hazards models, if functional forms were considered and how missing data affected this. We recorded the extent of missing data and how it was addressed in the analysis, for example using a complete-case analysis or multiple imputation. 148 studies were included in the review. On average, 15% of data were discarded due to missingness while determining the study population and 32% during the analysis stage. In total, 86% did not state any missing data assumptions. Complete-case analysis was common (56%) while 22% used multiple imputation. While guidelines are in place, few studies are implementing their recommendations in practice. Missing data are present in many studies but few state clearly how they handled it or the assumptions they have made. (Presentation 15 min. + Discussion 5 min.)Non UBCUnreviewedAuthor affiliation: London School of Hygiene and Tropical MedicineGraduat

Can the application of machine learning to electronic health records guide antibiotic prescribing decisions for suspected urinary tract infection in the Emergency Department?

Urinary tract infections (UTIs) are a major cause of emergency hospital admissions, but it remains challenging to diagnose them reliably. Application of machine learning (ML) to routine patient data could support clinical decision-making. We developed a ML model predicting bacteriuria in the ED and evaluated its performance in key patient groups to determine scope for its future use to improve UTI diagnosis and thus guide antibiotic prescribing decisions in clinical practice. We used retrospective electronic health records from a large UK hospital (2011-2019). Non-pregnant adults who attended the ED and had a urine sample cultured were eligible for inclusion. The primary outcome was predominant bacterial growth ≥104 cfu/mL in urine. Predictors included demography, medical history, ED diagnoses, blood tests, and urine flow cytometry. Linear and tree-based models were trained via repeated cross-validation, re-calibrated, and validated on data from 2018/19. Changes in performance were investigated by age, sex, ethnicity, and suspected ED diagnosis, and compared to clinical judgement. Among 12,680 included samples, 4,677 (36.9%) showed bacterial growth. Relying primarily on flow cytometry parameters, our best model achieved an area under the ROC curve (AUC) of 0.813 (95% CI 0.792-0.834) in the test data, and achieved both higher sensitivity and specificity compared to proxies of clinician's judgement. Performance remained stable for white and non-white patients but was lower during a period of laboratory procedure change in 2015, in patients ≥65 years (AUC 0.783, 95% CI 0.752-0.815), and in men (AUC 0.758, 95% CI 0.717-0.798). Performance was also slightly reduced in patients with recorded suspicion of UTI (AUC 0.797, 95% CI 0.765-0.828). Our results suggest scope for use of ML to inform antibiotic prescribing decisions by improving diagnosis of suspected UTI in the ED, but performance varied with patient characteristics. Clinical utility of predictive models for UTI is therefore likely to differ for important patient subgroups including women <65 years, women ≥65 years, and men. Tailored models and decision thresholds may be required that account for differences in achievable performance, background incidence, and risks of infectious complications in these groups

Can the application of machine learning to electronic health records guide antibiotic prescribing decisions for suspected urinary tract infection in the Emergency Department?

Urinary tract infections (UTIs) are a major cause of emergency hospital admissions, but it remains challenging to diagnose them reliably. Application of machine learning (ML) to routine patient data could support clinical decision-making. We developed a ML model predicting bacteriuria in the ED and evaluated its performance in key patient groups to determine scope for its future use to improve UTI diagnosis and thus guide antibiotic prescribing decisions in clinical practice. We used retrospective electronic health records from a large UK hospital (2011-2019). Non-pregnant adults who attended the ED and had a urine sample cultured were eligible for inclusion. The primary outcome was predominant bacterial growth ≥104 cfu/mL in urine. Predictors included demography, medical history, ED diagnoses, blood tests, and urine flow cytometry. Linear and tree-based models were trained via repeated cross-validation, re-calibrated, and validated on data from 2018/19. Changes in performance were investigated by age, sex, ethnicity, and suspected ED diagnosis, and compared to clinical judgement. Among 12,680 included samples, 4,677 (36.9%) showed bacterial growth. Relying primarily on flow cytometry parameters, our best model achieved an area under the ROC curve (AUC) of 0.813 (95% CI 0.792-0.834) in the test data, and achieved both higher sensitivity and specificity compared to proxies of clinician's judgement. Performance remained stable for white and non-white patients but was lower during a period of laboratory procedure change in 2015, in patients ≥65 years (AUC 0.783, 95% CI 0.752-0.815), and in men (AUC 0.758, 95% CI 0.717-0.798). Performance was also slightly reduced in patients with recorded suspicion of UTI (AUC 0.797, 95% CI 0.765-0.828). Our results suggest scope for use of ML to inform antibiotic prescribing decisions by improving diagnosis of suspected UTI in the ED, but performance varied with patient characteristics. Clinical utility of predictive models for UTI is therefore likely to differ for important patient subgroups including women <65 years, women ≥65 years, and men. Tailored models and decision thresholds may be required that account for differences in achievable performance, background incidence, and risks of infectious complications in these groups

Physical activity, exercise capacity and sedentary behavior in people with alpha-1 antitrypsin deficiency: a scoping review

Alpha-1 antitrypsin deficiency (AATD) is a hereditary disorder and a genetic risk factor for chronic obstructive pulmonary disease (COPD). Physical activity (PA) is important for the prevention and treatment of chronic disease. Little is known about PA in people with AATD. Therefore, we aimed to map the research undertaken to improve and/or measure PA, sedentary behaviour (SB) or exercise in people with AATD. Searches were conducted in CINAHL, Medline, EMBASE and clinical trial databases for studies published in 2021. Databases were searched for keywords (physical activity, AATD, exercise, sedentary behavior) as well as synonyms of these terms, which were connected using Boolean operators. The search yielded 360 records; 37 records were included for review. All included studies (n = 37) assessed exercise capacity; 22 studies reported the use of the six-minute walk test, the incremental shuttle walk test and cardiopulmonary exercise testing were reported in three studies each. Other objective measures of exercise capacity included a submaximal treadmill test, the Naughton protocol treadmill test, cycle ergometer maximal test, endurance shuttle walk test, constant cycle work rate test, a peak work rate test and the number of flights of stairs a participant was able to walk without stopping. A number of participant self-reported measures of exercise capacity were noted. Only one study aimed to analyze the effects of an intensive fitness intervention on daily PA. One further study reported on an exercise intervention and objectively measured PA at baseline. No studies measured SB. The assessment of PA and use of PA as an intervention in AATD is limited, and research into SB absent. Future research should measure PA and SB levels in people with AATD and explore interventions to enhance PA in this susceptible population. </p

Physical activity, exercise capacity and sedentary behavior in people with alpha-1 antitrypsin deficiency: a scoping review

Alpha-1 antitrypsin deficiency (AATD) is a hereditary disorder and a genetic risk factor for chronic obstructive pulmonary disease (COPD). Physical activity (PA) is important for the prevention and treatment of chronic disease. Little is known about PA in people with AATD. Therefore, we aimed to map the research undertaken to improve and/or measure PA, sedentary behaviour (SB) or exercise in people with AATD. Searches were conducted in CINAHL, Medline, EMBASE and clinical trial databases for studies published in 2021. Databases were searched for keywords (physical activity, AATD, exercise, sedentary behavior) as well as synonyms of these terms, which were connected using Boolean operators. The search yielded 360 records; 37 records were included for review. All included studies (n = 37) assessed exercise capacity; 22 studies reported the use of the six-minute walk test, the incremental shuttle walk test and cardiopulmonary exercise testing were reported in three studies each. Other objective measures of exercise capacity included a submaximal treadmill test, the Naughton protocol treadmill test, cycle ergometer maximal test, endurance shuttle walk test, constant cycle work rate test, a peak work rate test and the number of flights of stairs a participant was able to walk without stopping. A number of participant self-reported measures of exercise capacity were noted. Only one study aimed to analyze the effects of an intensive fitness intervention on daily PA. One further study reported on an exercise intervention and objectively measured PA at baseline. No studies measured SB. The assessment of PA and use of PA as an intervention in AATD is limited, and research into SB absent. Future research should measure PA and SB levels in people with AATD and explore interventions to enhance PA in this susceptible population. </p