Machine Learning and External Validation of the IDENTIFY Risk Calculator for Patients with Haematuria Referred to Secondary Care for Suspected Urinary Tract Cancer

BACKGROUND: The IDENTIFY study developed a model to predict urinary tract cancer using patient characteristics from a large multicentre, international cohort of patients referred with haematuria. In addition to calculating an individual's cancer risk, it proposes thresholds to stratify them into very-low-risk (<1%), low-risk (1-<5%), intermediate-risk (5-<20%), and high-risk (≥20%) groups. OBJECTIVE: To externally validate the IDENTIFY haematuria risk calculator and compare traditional regression with machine learning algorithms. DESIGN, SETTING, AND PARTICIPANTS: Prospective data were collected on patients referred to secondary care with new haematuria. Data were collected for patient variables included in the IDENTIFY risk calculator, cancer outcome, and TNM staging. Machine learning methods were used to evaluate whether better models than those developed with traditional regression methods existed. OUTCOME MEASUREMENTS AND STATISTICAL ANALYSIS: The area under the receiver operating characteristic curve (AUC) for the detection of urinary tract cancer, calibration coefficient, calibration in the large (CITL), and Brier score were determined. RESULTS AND LIMITATIONS: There were 3582 patients in the validation cohort. The development and validation cohorts were well matched. The AUC of the IDENTIFY risk calculator on the validation cohort was 0.78. This improved to 0.80 on a subanalysis of urothelial cancer prevalent countries alone, with a calibration slope of 1.04, CITL of 0.24, and Brier score of 0.14. The best machine learning model was Random Forest, which achieved an AUC of 0.76 on the validation cohort. There were no cancers stratified to the very-low-risk group in the validation cohort. Most cancers were stratified to the intermediate- and high-risk groups, with more aggressive cancers in higher-risk groups. CONCLUSIONS: The IDENTIFY risk calculator performed well at predicting cancer in patients referred with haematuria on external validation. This tool can be used by urologists to better counsel patients on their cancer risks, to prioritise diagnostic resources on appropriate patients, and to avoid unnecessary invasive procedures in those with a very low risk of cancer. PATIENT SUMMARY: We previously developed a calculator that predicts patients' risk of cancer when they have blood in their urine, based on their personal characteristics. We have validated this risk calculator, by testing it on a separate group of patients to ensure that it works as expected. Most patients found to have cancer tended to be in the higher-risk groups and had more aggressive types of cancer with a higher risk. This tool can be used by clinicians to fast-track high-risk patients based on the calculator and investigate them more thoroughly

The IDENTIFY study: the investigation and detection of urological neoplasia in patients referred with suspected urinary tract cancer - a multicentre observational study.

Funder: Action Bladder Cancer UKFunder: Rosetrees Trust; Id: http://dx.doi.org/10.13039/501100000833Funder: Urology Care Foundation; Id: http://dx.doi.org/10.13039/100006280OBJECTIVE: To evaluate the contemporary prevalence of urinary tract cancer (bladder cancer, upper tract urothelial cancer [UTUC] and renal cancer) in patients referred to secondary care with haematuria, adjusted for established patient risk markers and geographical variation. PATIENTS AND METHODS: This was an international multicentre prospective observational study. We included patients aged ≥16 years, referred to secondary care with suspected urinary tract cancer. Patients with a known or previous urological malignancy were excluded. We estimated the prevalence of bladder cancer, UTUC, renal cancer and prostate cancer; stratified by age, type of haematuria, sex, and smoking. We used a multivariable mixed-effects logistic regression to adjust cancer prevalence for age, type of haematuria, sex, smoking, hospitals, and countries. RESULTS: Of the 11 059 patients assessed for eligibility, 10 896 were included from 110 hospitals across 26 countries. The overall adjusted cancer prevalence (n = 2257) was 28.2% (95% confidence interval [CI] 22.3-34.1), bladder cancer (n = 1951) 24.7% (95% CI 19.1-30.2), UTUC (n = 128) 1.14% (95% CI 0.77-1.52), renal cancer (n = 107) 1.05% (95% CI 0.80-1.29), and prostate cancer (n = 124) 1.75% (95% CI 1.32-2.18). The odds ratios for patient risk markers in the model for all cancers were: age 1.04 (95% CI 1.03-1.05; P < 0.001), visible haematuria 3.47 (95% CI 2.90-4.15; P < 0.001), male sex 1.30 (95% CI 1.14-1.50; P < 0.001), and smoking 2.70 (95% CI 2.30-3.18; P < 0.001). CONCLUSIONS: A better understanding of cancer prevalence across an international population is required to inform clinical guidelines. We are the first to report urinary tract cancer prevalence across an international population in patients referred to secondary care, adjusted for patient risk markers and geographical variation. Bladder cancer was the most prevalent disease. Visible haematuria was the strongest predictor for urinary tract cancer

Machine Learning and External Validation of the IDENTIFY Risk Calculator for Patients with Haematuria Referred to Secondary Care for Suspected Urinary Tract Cancer

Background: The IDENTIFY study developed a model to predict urinary tract cancer using patient characteristics from a large multicentre, international cohort of patients referred with haematuria. In addition to calculating an individual's cancer risk, it proposes thresholds to stratify them into very-low-risk (<1%), low-risk (1–<5%), intermediate-risk (5–<20%), and high-risk (≥20%) groups. Objective: To externally validate the IDENTIFY haematuria risk calculator and compare traditional regression with machine learning algorithms. Design, setting, and participants: Prospective data were collected on patients referred to secondary care with new haematuria. Data were collected for patient variables included in the IDENTIFY risk calculator, cancer outcome, and TNM staging. Machine learning methods were used to evaluate whether better models than those developed with traditional regression methods existed. Outcome measurements and statistical analysis: The area under the receiver operating characteristic curve (AUC) for the detection of urinary tract cancer, calibration coefficient, calibration in the large (CITL), and Brier score were determined. Results and limitations: There were 3582 patients in the validation cohort. The development and validation cohorts were well matched. The AUC of the IDENTIFY risk calculator on the validation cohort was 0.78. This improved to 0.80 on a subanalysis of urothelial cancer prevalent countries alone, with a calibration slope of 1.04, CITL of 0.24, and Brier score of 0.14. The best machine learning model was Random Forest, which achieved an AUC of 0.76 on the validation cohort. There were no cancers stratified to the very-low-risk group in the validation cohort. Most cancers were stratified to the intermediate- and high-risk groups, with more aggressive cancers in higher-risk groups. Conclusions: The IDENTIFY risk calculator performed well at predicting cancer in patients referred with haematuria on external validation. This tool can be used by urologists to better counsel patients on their cancer risks, to prioritise diagnostic resources on appropriate patients, and to avoid unnecessary invasive procedures in those with a very low risk of cancer. Patient summary: We previously developed a calculator that predicts patients’ risk of cancer when they have blood in their urine, based on their personal characteristics. We have validated this risk calculator, by testing it on a separate group of patients to ensure that it works as expected. Most patients found to have cancer tended to be in the higher-risk groups and had more aggressive types of cancer with a higher risk. This tool can be used by clinicians to fast-track high-risk patients based on the calculator and investigate them more thoroughly

Prevalencia del cáncer del tracto urinario. Análisis de la cohorte española del estudio IDENTIFY // Prevalence of urinary tract cancer in the Spanish cohort of the IDENTIFY study

Introducción: Los tumores malignos del tracto urinario están asociados a gran morbimortalidad siendo su prevalencia variable a nivel global. Recientemente el estudio IDENTIFY ha publicado resultados sobre la prevalencia del cáncer del tracto urinario a nivel internacional. Este estudio evalúa la prevalencia de cáncer dentro de la cohorte española del estudio IDENTIFY para determinar si los resultados publicados son extrapolables a nuestra población. // Introduction: Malignant tumors of the urinary tract are associated with high morbidity and mortality, and their prevalence can vary worldwide. Recently, the IDENTIFY study has published results on the prevalence of urinary tract cancer at a global level. This study evaluates the prevalence of cancer within the Spanish cohort of the IDENTIFY study to determine whether the published results can be extrapolated to our population

Prevalence of urinary tract cancer in the Spanish cohort of the IDENTIFY study // Prevalencia del cáncer del tracto urinario. Análisis de la cohorte española del estudio IDENTIFY

Introduction: Malignant tumors of the urinary tract are associated with high morbidity and mortality, and their prevalence can vary worldwide. Recently, the IDENTIFY study has published results on the prevalence of urinary tract cancer at a global level. This study evaluates the prevalence of cancer within the Spanish cohort of the IDENTIFY study to determine whether the published results can be extrapolated to our population. // Patients and methods: An analysis of the data from the Spanish cohort of patients in the IDENTIFY study was performed. This is a prospective cohort of patients referred to secondary care with suspected cancer, predominantly due to hematuria. Patients were recruited between December 2017 and December 2018. // Results: A total of 706 patients from 9 Spanish centers were analyzed. Of these, 277 (39.2%) were diagnosed with cancer: 259 (36.7%) bladder cancer, 10 (1.4%) upper tract urothelial carcinoma, 9 (1.2%) renal cancer and 5 (0.7%) prostate cancer. Increasing age (OR 1.05 (95% CI 1.03−1.06; P < 0.001)), visible hematuria (VH) OR 2.19 (95% CI 1.13–4.24; P = 0.02)) and smoking (ex-smokers: OR 2.11(95% CI 1.30–3.40; P = 0.002); smokers: OR 2.36 (95% CI 1.40–3.95; P = 0.001)) were associated with higher probability of bladder cancer. // Conclusion: This study highlights the risk of bladder cancer in patients with VH and smoking habits. Bladder cancer presented the highest prevalence; higher than the prevalence reported in previous series and presented in the IDENTIFY study. Future work should evaluate other associated factors that allow us to create cancer prediction models to improve the detection of cancer in our patients

The significance of isolated de novo red patches in the bladder in patients referred with suspected urinary tract cancer:results from the IDENTIFY study

Objectives To assess the contemporary malignancy rate in isolated de novo red patches in the bladder and associated risk factors for better selection of red patch biopsy. Patients Patients from the IDENTIFY dataset; Patients referred to secondary care with suspected urinary tract cancer and found to have isolated de novo red patches on cystoscopy.Methods We reported the unadjusted cancer prevalence in isolated de novo red patches that were biopsied; multivariable logistic regression was used to explore cancer-associated risk factors including age, sex, smoking, type of haematuria, LUTS, UTIs and a suspicious-looking red patch (as reported by the cystoscopist). Sub-analysis of these by clinical role and experience was performed.Results A total of 1110 patients with isolated de novo red patches were included. 41.5% (n = 461) were biopsied, with a malignancy rate of 12.8% (59/461), which was significantly higher in suspicious versus non-suspicious red patches (19.1% vs. 2.81%, p &lt; 0.01). There was a significant association between bladder cancer and age (OR 1.04, 95% CI 1.01–1.07, p = 0.01), smoking history (OR 2.62, 95% CI 1.09–6.27, p = 0.03) and suspicious-looking patch (OR 6.50, 95% CI 2.47–17.1, p &lt; 0.01). The majority of malignancies were in over 60-year-olds. Malignancy rates in suspicious versus non-suspicious red patches did not differ significantly between clinical roles or experiences. Limitations included subjectivity in classifying a suspicious patch and selection bias as not all patches were biopsied.Conclusions Many patients still undergo unnecessary biopsies under general anaesthetic for isolated de novo red patches. Clinicians should consider the patient's age, smoking status and how suspicious-looking the patch is, before deciding on surveillance versus biopsy to improve cancer diagnostic yield

Machine learning and external validation of the IDENTIFY risk calculator for patients with haematuria referred to secondary care for suspected urinary tract cancer

Background The IDENTIFY study developed a model to predict urinary tract cancer using patient characteristics from a large multicentre, international cohort of patients referred with haematuria. In addition to calculating an individual’s cancer risk, it proposes thresholds to stratify them into very-low-risk (&lt;1%), low-risk (1–&lt;5%), intermediate-risk (5–&lt;20%), and high-risk (≥20%) groups.Objective To externally validate the IDENTIFY haematuria risk calculator and compare traditional regression with machine learning algorithms.Design, setting, and participants Prospective data were collected on patients referred to secondary care with new haematuria. Data were collected for patient variables included in the IDENTIFY risk calculator, cancer outcome, and TNM staging. Machine learning methods were used to evaluate whether better models than those developed with traditional regression methods existed. Outcome measurements and statistical analysis The area under the receiver operating characteristic curve (AUC) for the detection of urinary tract cancer, calibration coefficient, calibration in the large (CITL), and Brier score were determined. Results and limitations There were 3582 patients in the validation cohort. The development and validation cohorts were well matched. The AUC of the IDENTIFY risk calculator on the validation cohort was 0.78. This improved to 0.80 on a subanalysis of urothelial cancer prevalent countries alone, with a calibration slope of 1.04, CITL of 0.24, and Brier score of 0.14. The best machine learning model was Random Forest, which achieved an AUC of 0.76 on the validation cohort. There were no cancers stratified to the very-low-risk group in the validation cohort. Most cancers were stratified to the intermediate- and high-risk groups, with more aggressive cancers in higher-risk groups. Conclusions The IDENTIFY risk calculator performed well at predicting cancer in patients referred with haematuria on external validation. This tool can be used by urologists to better counsel patients on their cancer risks, to prioritise diagnostic resources on appropriate patients, and to avoid unnecessary invasive procedures in those with a very low risk of cancer. Patient summary We previously developed a calculator that predicts patients’ risk of cancer when they have blood in their urine, based on their personal characteristics. We have validated this risk calculator, by testing it on a separate group of patients to ensure that it works as expected. Most patients found to have cancer tended to be in the higher-risk groups and had more aggressive types of cancer with a higher risk. This tool can be used by clinicians to fast-track high-risk patients based on the calculator and investigate them more thoroughly

Developing a Diagnostic Multivariable Prediction Model for Urinary Tract Cancer in Patients Referred with Haematuria : results from the IDENTIFY Collaborative Study

open access via Elsevier agreement Acknowledgments: We would like to thank all the BURST research collaborators for taking part in this study, Max Peters for his support and advice regarding the methods and Jonathan Deeks for his support from the Test Evaluation Research Group. Though unrelated to this study, the BURST Research Collaborative would like to acknowledge funding from the BJU International, the British Association of Urological Surgeons, Ferring Pharmaceuticals Ltd, and Dominvs Group.Peer reviewedPublisher PD

Prevalence of urinary tract cancer in the Spanish cohort of the IDENTIFY study

Introduction: Malignant tumors of the urinary tract are associated with high morbidity and mortality, and their prevalence can vary worldwide. Recently, the IDENTIFY study has published results on the prevalence of urinary tract cancer at a global level. This study evaluates the prevalence of cancer within the Spanish cohort of the IDENTIFY study to determine whether the published results can be extrapolated to our population. Patients and methods: An analysis of the data from the Spanish cohort of patients in the IDENTIFY study was performed. This is a prospective cohort of patients referred to secondary care with suspected cancer, predominantly due to hematuria. Patients were recruited between December 2017 and December 2018. Results: A total of 706 patients from 9 Spanish centers were analyzed. Of these, 277 (39.2%) were diagnosed with cancer: 259 (36.7%) bladder cancer, 10 (1.4%) upper tract urothelial carcinoma, 9 (1.2%) renal cancer and 5 (0.7%) prostate cancer. Increasing age (OR 1.05 (95% CI 1.03-1.06; P < 0.001)), visible hematuria (VH) OR 2.19 (95% CI 1.13-4.24; P = 0.02)) and smoking (ex-smokers: OR 2.11(95% CI 1.30-3.40; P = 0.002); smokers: OR 2.36 (95% CI 1.40-3.95; P = 0.001)) were associated with higher probability of bladder cancer. Conclusion: This study highlights the risk of bladder cancer in patients with VH and smoking habits. Bladder cancer presented the highest prevalence; higher than the prevalence reported in previous series and presented in the IDENTIFY study. Future work should evaluate other associated factors that allow us to create cancer prediction models to improve the detection of cancer in our patients