Multicenter external validation of the liverpool uveal melanoma prognosticator online: An OOG collaborative study

Uveal melanoma (UM) is fatal in ~50% of patients as a result of disseminated disease. This study aims to externally validate the Liverpool Uveal Melanoma Prognosticator Online V3 (LUMPO3) to determine its reliability in predicting survival after treatment for choroidal melanoma when utilizing external data from other ocular oncology centers. Anonymized data of 1836 UM patients from seven international ocular oncology centers were analyzed with LUMPO3 to predict the 10-year survival for each patient in each external dataset. The analysts were masked to the patient outcomes. Model predictions were sent to an independent statistician to evaluate LUMPO3’s performance using discrimination and calibration methods. LUMPO3’s ability to discriminate between UM patients who died of metastatic UM and those who were still alive was fair-to-good, with C-statistics ranging from 0.64 to 0.85 at year 1. The pooled estimate for all external centers was 0.72 (95% confidence interval: 0.68 to 0.75). Agreement between observed and predicted survival probabilities was generally good given differences in case mix and survival rates between different centers. Despite the differences between the international cohorts of patients with primary UM, LUMPO3 is a valuable tool for predicting all-cause mortality in this disease when using data from external centers

Prognostic modelling of breast cancer patients: a benchmark of predictive models with external validation

Dissertação apresentada para obtenção do Grau de Doutor em Engenharia Electrotécnica e de Computadores – Sistemas Digitais e Percepcionais pela Universidade Nova de Lisboa, Faculdade de Ciências e TecnologiaThere are several clinical prognostic models in the medical field. Prior to clinical use, the outcome models of longitudinal cohort data need to undergo a multi-centre evaluation of their predictive accuracy. This thesis evaluates the possible gain in predictive accuracy in multicentre evaluation of a flexible model with Bayesian regularisation, the (PLANN-ARD), using a reference data set for breast cancer, which comprises 4016 records from patients diagnosed during 1989-93 and reported by the BCCA, Canada, with follow-up of 10 years. The method is compared with the widely used Cox regression model. Both methods were fitted to routinely acquired data from 743 patients diagnosed during 1990-94 at the Christie Hospital, UK, with follow-up of 5 years following surgery. Methodological advances developed to support the external validation of this neural network with clinical data include: imputation of missing data in both the training and validation data sets; and a prognostic index for stratification of patients into risk groups that can be extended to non-linear models. Predictive accuracy was measured empirically with a standard discrimination index, Ctd, and with a calibration measure, using the Hosmer-Lemeshow test statistic. Both Cox regression and the PLANN-ARD model are found to have similar discrimination but the neural network showed marginally better predictive accuracy over the 5-year followup period. In addition, the regularised neural network has the substantial advantage of being suited for making predictions of hazard rates and survival for individual patients. Four different approaches to stratify patients into risk groups are also proposed, each with a different foundation. While it was found that the four methodologies broadly agree, there are important differences between them. Rules sets were extracted and compared for the two stratification methods, the log-rank bootstrap and by direct application of regression trees, and with two rule extraction methodologies, OSRE and CART, respectively. In addition, widely used clinical breast cancer prognostic indexes such as the NPI, TNM and St. Gallen consensus rules, were compared with the proposed prognostic models expressed as regression trees, concluding that the suggested approaches may enhance current practice. Finally, a Web clinical decision support system is proposed for clinical oncologists and for breast cancer patients making prognostic assessments, which is tailored to the particular characteristics of the individual patient. This system comprises three different prognostic modelling methodologies: the NPI, Cox regression modelling and PLANN-ARD. For a given patient, all three models yield a generally consistent but not identical set of prognostic indices that can be analysed together in order to obtain a consensus and so achieve a more robust prognostic assessment of the expected patient outcome

Double-blind evaluation and benchmarking of survival models in a multi-centre study

Accurate modelling of time-to-event data is of particular importance for both exploratory and predictive analysis in cancer, and can have a direct impact on clinical care. This study presents a detailed double-blind evaluation of the accuracy in out-of-sample prediction of mortality from two generic non-linear models, using artificial neural networks benchmarked against a partial logistic spline, log-normal and COX regression models. A data set containing 2880 samples was shared over the Internet using a purpose-built secure environment called GEOCONDA (www.geoconda.com). The evaluation was carried out in three parts. The first was a comparison between the predicted survival estimates for each of the four survival groups defined by the TNM staging system, against the empirical estimates derived by the Kaplan-Meier method. The second approach focused on the accurate prediction of survival over time, quantified with the time dependent C index (C(td)). Finally, calibration plots were obtained over the range of follow-up and tested using a generalization of the Hosmer-Lemeshow test. All models showed satisfactory performance, with values of C(td) of about 0.7. None of the models showed a systematic tendency towards over/under estimation of the observed survival at tau=3 and 5 years. At tau=10 years, all models underestimated the observed survival, except for COX regression which returned an overestimate. The study presents a robust and unbiased benchmarking methodology using a bespoke web facility. It was concluded that powerful, recent flexible modelling algorithms show a comparative predictive performance to that of more established methods from the medical and biological literature, for the reference data set

What is the added value of using non-linear models to explore complex healthcare datasets?

Health care is a complex system and it is therefore expected to behave in a non-linear manner. It is important for the delivery of health interventions to patients that the best possible analysis of available data is undertaken. Many of the conventional models used for health care data are linear. This research compares the performance of linear models with non-linear models for two health care data sets of complex interventions. Logistic regression, latent class analysis and a classification artificial neural network were each used to model outcomes for patients using data from a randomised controlled trial of a cognitive behavioural complex intervention for non-specific low back pain. A Cox proportional hazards model and an artificial neural network were used to model survival and the hazards for different sub-groups of patients using an observational study of a cardiovascular rehabilitation complex intervention. The artificial neural network and an ordinary logistic regression were more accurate in classifying patient recovery from back pain than a logistic regression on latent class membership. The most sensitive models were the artificial neural network and the latent class logistic regression. The best overall performance was the artificial neural network, providing both sensitivity and accuracy. Survival was modelled equally well by the Cox model and the artificial neural network, when compared to the empirical Kaplan-Meier survival curve. Long term survival for the cardiovascular patients was strongly associated with secondary prevention medications, and fitness was also important. Moreover, improvement in fitness during the rehabilitation period to a fairly modest 'high fitness' category was as advantageous for long-term survival as having achieved that same level of fitness by the beginning of the rehabilitation period. Having adjusted for fitness, BMI was not a predictor of long term survival after a cardiac event or procedure. The Cox proportional hazards model was constrained by its assumptions to produce hazard trajectories proportional to the baseline hazard. The artificial neural network model produced hazard trajectories that vary, giving rise to hypotheses about how the predictors of survival interact in their influence on the hazard. The artificial neural network, an exemplar non-linear model, has been shown to match or exceed the capability of conventional models in the analysis of complex health care data sets