Double-blind evaluation and benchmarking of survival models in a multi-centre study

Accurate modelling of time-to-event data is of particular importance for both exploratory and predictive analysis in cancer, and can have a direct impact on clinical care. This study presents a detailed double-blind evaluation of the accuracy in out-of-sample prediction of mortality from two generic non-linear models, using artificial neural networks benchmarked against a partial logistic spline, log-normal and COX regression models. A data set containing 2880 samples was shared over the Internet using a purpose-built secure environment called GEOCONDA (www.geoconda.com). The evaluation was carried out in three parts. The first was a comparison between the predicted survival estimates for each of the four survival groups defined by the TNM staging system, against the empirical estimates derived by the Kaplan-Meier method. The second approach focused on the accurate prediction of survival over time, quantified with the time dependent C index (C(td)). Finally, calibration plots were obtained over the range of follow-up and tested using a generalization of the Hosmer-Lemeshow test. All models showed satisfactory performance, with values of C(td) of about 0.7. None of the models showed a systematic tendency towards over/under estimation of the observed survival at tau=3 and 5 years. At tau=10 years, all models underestimated the observed survival, except for COX regression which returned an overestimate. The study presents a robust and unbiased benchmarking methodology using a bespoke web facility. It was concluded that powerful, recent flexible modelling algorithms show a comparative predictive performance to that of more established methods from the medical and biological literature, for the reference data set

The Major Acute-Phase Protein, Serum Amyloid P Component, in Mice Is Not Involved in Endogenous Resistance against Tumor Necrosis Factor Alpha-Induced Lethal Hepatitis, Shock, and Skin Necrosis

The proinflammatory cytokine tumor necrosis factor alpha (TNF-α) induces lethal hepatitis when injected into d-(+)-galactosamine-sensitized mice on the one hand or systemic inflammatory response syndrome (SIRS) in normal mice on the other hand. We studied whether serum amyloid P component (SAP), the major acute-phase protein in mice, plays a protective role in both lethal models. For this purpose, we used SAP(0/0) mice generated by gene targeting. We studied the lethal response of SAP(0/0) or SAP(+/+) mice to both lethal triggers but found no differences in the sensitivity of both types of mice. We also investigated whether SAP is involved in establishing two types of endogenous protection: one using a single injection of interleukin-1β (IL-1β) for desensitization and clearly involving a liver protein, the other by tolerizing mice for 5 days using small doses of human TNF-α. Although after IL-1β or after tolerization the SAP levels in the serum had risen fourfold in the control mice and not in the SAP(0/0) mice, the same extents of desensitization and tolerization were achieved. Finally, we observed that the induction of hemorrhagic necrosis in the skin of mice by two consecutive local injections with TNF-α was not altered in SAP(0/0) mice. We conclude that the presence or absence of SAP has no influence on the sensitivity of mice to TNF-α-induced hepatitis, SIRS, and hemorrhagic necrosis or on the endogenous protective mechanisms of desensitization or tolerization