Predicting sample size required for classification performance

<p>Abstract</p> <p>Background</p> <p>Supervised learning methods need annotated data in order to generate efficient models. Annotated data, however, is a relatively scarce resource and can be expensive to obtain. For both passive and active learning methods, there is a need to estimate the size of the annotated sample required to reach a performance target.</p> <p>Methods</p> <p>We designed and implemented a method that fits an inverse power law model to points of a given learning curve created using a small annotated training set. Fitting is carried out using nonlinear weighted least squares optimization. The fitted model is then used to predict the classifier's performance and confidence interval for larger sample sizes. For evaluation, the nonlinear weighted curve fitting method was applied to a set of learning curves generated using clinical text and waveform classification tasks with active and passive sampling methods, and predictions were validated using standard goodness of fit measures. As control we used an un-weighted fitting method.</p> <p>Results</p> <p>A total of 568 models were fitted and the model predictions were compared with the observed performances. Depending on the data set and sampling method, it took between 80 to 560 annotated samples to achieve mean average and root mean squared error below 0.01. Results also show that our weighted fitting method outperformed the baseline un-weighted method (p < 0.05).</p> <p>Conclusions</p> <p>This paper describes a simple and effective sample size prediction algorithm that conducts weighted fitting of learning curves. The algorithm outperformed an un-weighted algorithm described in previous literature. It can help researchers determine annotation sample size for supervised machine learning.</p

Double-blind evaluation and benchmarking of survival models in a multi-centre study

Accurate modelling of time-to-event data is of particular importance for both exploratory and predictive analysis in cancer, and can have a direct impact on clinical care. This study presents a detailed double-blind evaluation of the accuracy in out-of-sample prediction of mortality from two generic non-linear models, using artificial neural networks benchmarked against a partial logistic spline, log-normal and COX regression models. A data set containing 2880 samples was shared over the Internet using a purpose-built secure environment called GEOCONDA (www.geoconda.com). The evaluation was carried out in three parts. The first was a comparison between the predicted survival estimates for each of the four survival groups defined by the TNM staging system, against the empirical estimates derived by the Kaplan-Meier method. The second approach focused on the accurate prediction of survival over time, quantified with the time dependent C index (C(td)). Finally, calibration plots were obtained over the range of follow-up and tested using a generalization of the Hosmer-Lemeshow test. All models showed satisfactory performance, with values of C(td) of about 0.7. None of the models showed a systematic tendency towards over/under estimation of the observed survival at tau=3 and 5 years. At tau=10 years, all models underestimated the observed survival, except for COX regression which returned an overestimate. The study presents a robust and unbiased benchmarking methodology using a bespoke web facility. It was concluded that powerful, recent flexible modelling algorithms show a comparative predictive performance to that of more established methods from the medical and biological literature, for the reference data set

Impact of mutational status on outcomes in myelofibrosis patients treated with ruxolitinib in the COMFORT-II study

The JAK1/JAK2 inhibitor ruxolitinib produced significant reductions in splenomegaly and symptomatic burden and improved survival in patients with myelofibrosis (MF), irrespective of their JAK2 mutation status, in 2 phase III studies against placebo (COMFORT-I) and best available therapy (COMFORT-II). We performed a comprehensive mutation analysis to evaluate the impact of 14 MF-associated mutations on clinical outcomes in 166 patients included in COMFORT-II. We found that responses in splenomegaly and symptoms, as well as the risk of developing ruxolitinib-associated anemia and thrombocytopenia, occurred at similar frequencies across different mutation profiles. Ruxolitinib improved survival independent of mutation profile and reduced the risk of death in patients harboring a set of prognostically detrimental mutations (ASXL1, EZH2, SRSF2, IDH1/2) with an hazard ratio of 0.57 (95% confidence interval: 0.30-1.08) vs best available therapy. These data indicate that clinical efficacy and survival improvement may occur across different molecular subsets of patients with MF treated with ruxolitinib

Three-year efficacy, safety, and survival findings from COMFORT-II, a phase 3 study comparing ruxolitinib with best available therapy for myelofibrosis

Ruxolitinib is a potent Janus kinase (JAK)1/JAK2 inhibitor that has demonstrated rapid reductions in splenomegaly and marked improvement in disease-related symptoms and quality of life in patients with myelofibrosis (MF). The present analysis reports the 3-year follow-up (median, 151 weeks) of the efficacy and safety of Controlled Myelofibrosis Study With Oral Janus-associated Kinase (JAK) Inhibitor Treatment-II (the COMFORT-II Trial), comparing ruxolitinib with the best available therapy (BAT) in 219 patients with intermediate-2 and high-risk MF. In the ruxolitinib arm, with continued therapy, spleen volume reductions of 6535% by magnetic resonance imaging (equivalent to approximately 50% reduction by palpation) were sustained for at least 144 weeks, with the probability of 50% (95% confidence interval [CI], 36-63) among patients achieving such degree of response. At the time of this analysis, 45% of the patients randomized to ruxolitinib remained on treatment. Ruxolitinib continues to be well tolerated. Anemia and thrombocytopenia were the main toxicities, but they were generally manageable, improved over time, and rarely led to treatment discontinuation (1% and 3.6% of patients, respectively). No single nonhematologic adverse event led to definitive ruxolitinib discontinuation in more than 1 patient. Additionally, patients randomized to ruxolitinib showed longer overall survival than those randomized to BAT (hazard ratio, 0.48; 95% CI, 0.28-0.85; log-rank test, P = .009)

Experimental comparison of machine learning approaches to medical domains: a case study of genotype influence on oral cancer development

Summarization: Research in medical domains is facing new challenges as the available information increases in quantity and quality. In this context, Machine Learning methodologies can provide the right tools for data analysis, which can cope with recurring problems in medical research, such as the integration of clinical and genetic data. In this study we provide an experimental comparison of an heterogeneous subset of Machine Learning methods. For such a purpose, a representative dataset for medical analysis was chosen which regards Head and Neck Squamous Cell Carcinoma (HNSCC). HNSCC is a kind of oral cancer associated with smoking and alcohol drinking habits; however the individual risk could be modified by genetic polymorphisms of enzymes involved in the metabolism of tobacco carcinogens and in the DNA repair mechanisms. To study this relationship, the data set comprised demographic and life-style (age, gender, smoke and alcohol), and genetic data (the individual genotype of 11 polymorphic genes), with the information on 124 HNSCC patients and 231 healthy controls. Strengths and weaknesses of the different algorithms when applied to medical datasets, such as the one considered, will be analyzed, with particular attention to the issue of missing values.Presented on