Post-Selection Confidence Bounds for Prediction Performance

In machine learning, the selection of a promising model from a potentially large number of competing models and the assessment of its generalization performance are critical tasks that need careful consideration. Typically, model selection and evaluation are strictly separated endeavors, splitting the sample at hand into a training, validation, and evaluation set, and only compute a single confidence interval for the prediction performance of the final selected model. We however propose an algorithm how to compute valid lower confidence bounds for multiple models that have been selected based on their prediction performances in the evaluation set by interpreting the selection problem as a simultaneous inference problem. We use bootstrap tilting and a maxT-type multiplicity correction. The approach is universally applicable for any combination of prediction models, any model selection strategy, and any prediction performance measure that accepts weights. We conducted various simulation experiments which show that our proposed approach yields lower confidence bounds that are at least comparably good as bounds from standard approaches, and that reliably reach the nominal coverage probability. In addition, especially when sample size is small, our proposed approach yields better performing prediction models than the default selection of only one model for evaluation does.Comment: Changed title, changed layout of figures, added a number to equation 4, added more info to figure captions, added keyword

Model selection based on combined penalties for biomarker identification

The growing role of targeted medicine has led to an increased focus on the development of actionable biomarkers. Current penalized selection methods that are used to identify biomarker panels for classification in high-dimensional data, however, often result in highly complex panels that need careful pruning for practical use. In the framework of regularization methods, a penalty that is a weighted sum of the L1 and L0 norm has been proposed to account for the complexity of the resulting model. In practice, the limitation of this penalty is that the objective function is non-convex, non-smooth, the optimization is computationally intensive and the application to high-dimensional settings is challenging. In this paper, we propose a stepwise forward variable selection method which combines the L0 with L1 or L2 norms. The penalized likelihood criterion that is used in the stepwise selection procedure results in more parsimonious models, keeping only the most relevant features. Simulation results and a real application show that our approach exhibits a comparable performance with common selection methods with respect to the prediction performance while minimizing the number of variables in the selected model resulting in a more parsimonious model as desired

A Bayesian model to estimate the cutoff and the clinical utility of a biomarker assay

To enable targeted therapies and enhance medical decision-making, biomarkers are increasingly used as screening and diagnostic tests. When using quantitative biomarkers for classification purposes, this often implies that an appropriate cutoff for the biomarker has to be determined and its clinical utility must be assessed. In the context of drug development, it is of interest how the probability of response changes with increasing values of the biomarker. Unlike sensitivity and specificity, predictive values are functions of the accuracy of the test, depend on the prevalence of the disease and therefore are a useful tool in this setting. In this paper, we propose a Bayesian method to not only estimate the cutoff value using the negative and positive predictive values, but also estimate the uncertainty around this estimate. Using Bayesian inference allows us to incorporate prior information, and obtain posterior estimates and credible intervals for the cut-off and associated predictive values. The performance of the Bayesian approach is compared with alternative methods via simulation studies of bias, interval coverage and width and illustrations on real data with binary and time-to-event outcomes are provided

Efficient Adaptive Designs for Clinical Trials of Interventions for COVID-19.

The COVID-19 pandemic has led to an unprecedented response in terms of clinical research activity. An important part of this research has been focused on randomized controlled clinical trials to evaluate potential therapies for COVID-19. The results from this research need to be obtained as rapidly as possible. This presents a number of challenges associated with considerable uncertainty over the natural history of the disease and the number and characteristics of patients affected, and the emergence of new potential therapies. These challenges make adaptive designs for clinical trials a particularly attractive option. Such designs allow a trial to be modified on the basis of interim analysis data or stopped as soon as sufficiently strong evidence has been observed to answer the research question, without compromising the trial's scientific validity or integrity. In this article, we describe some of the adaptive design approaches that are available and discuss particular issues and challenges associated with their use in the pandemic setting. Our discussion is illustrated by details of four ongoing COVID-19 trials that have used adaptive designs

Heterogeneous Pattern of Retinal Nerve Fiber Layer in Multiple Sclerosis. High Resolution Optical Coherence Tomography: Potential and Limitations

BACKGROUND: Recently the reduction of the retinal nerve fibre layer (RNFL) was suggested to be associated with diffuse axonal damage in the whole CNS of multiple sclerosis (MS) patients. However, several points are still under discussion. (1) Is high resolution optical coherence tomography (OCT) required to detect the partly very subtle RNFL changes seen in MS patients? (2) Can a reduction of RNFL be detected in all MS patients, even in early disease courses and in all MS subtypes? (3) Does an optic neuritis (ON) or focal lesions along the visual pathways, which are both very common in MS, limit the predication of diffuse axonal degeneration in the whole CNS? The purpose of our study was to determine the baseline characteristics of clinical definite relapsing-remitting (RRMS) and secondary progressive (SPMS) MS patients with high resolution OCT technique. METHODOLOGY: Forty-two RRMS and 17 SPMS patients with and without history of uni- or bilateral ON, and 59 age- and sex-matched healthy controls were analysed prospectively with the high resolution spectral-domain OCT device (SD-OCT) using the Spectralis 3.5mm circle scan protocol with locked reference images and eye tracking mode. Furthermore we performed tests for visual and contrast acuity and sensitivity (ETDRS, Sloan and Pelli-Robson-charts), for color vision (Lanthony D-15), the Humphrey visual field and visual evoked potential testing (VEP). PRINCIPAL FINDINGS: All 4 groups (RRMS and SPMS with or without ON) showed significantly reduced RNFL globally, or at least in one of the peripapillary sectors compared to age-/sex-matched healthy controls. In patients with previous ON additional RNFL reduction was found. However, in many RRMS patients the RNFL was found within normal range. We found no correlation between RNFL reduction and disease duration (range 9-540 months). CONCLUSIONS: RNFL baseline characteristics of RRMS and SPMS are heterogeneous (range from normal to markedly reduced levels)

A proposal for a new PhD level curriculum on quantitative methods for drug development

This paper provides an overview of “Improving Design, Evaluation and Analysis of early drug development Studies” (IDEAS), a European Commission–funded network bringing together leading academic institutions and small‐ to large‐sized pharmaceutical companies to train a cohort of graduate‐level medical statisticians. The network is composed of a diverse mix of public and private sector partners spread across Europe, which will host 14 early‐stage researchers for 36 months. IDEAS training activities are composed of a well‐rounded mixture of specialist methodological components and generic transferable skills. Particular attention is paid to fostering collaborations between researchers and supervisors, which span academia and the private sector. Within this paper, we review existing medical statistics programmes (MSc and PhD) and highlight the training they provide on skills relevant to drug development. Motivated by this review and our experiences with the IDEAS project, we propose a concept for a joint, harmonised European PhD programme to train statisticians in quantitative methods for drug development

A bipolar theorem for L^0_+(\Om, \Cal F, \P)

A consequence of the Hahn-Banach theorem is the classical bipolar theorem which states that the bipolar of a subset of a locally convex vector pace equals its closed convex hull. The space \L of real-valued random variables on a probability space \OF equipped with the topology of convergence in measure fails to be locally convex so that - a priori - the classical bipolar theorem does not apply. In this note we show an analogue of the bipolar theorem for subsets of the positive orthant \LO, if we place \LO in duality with itself, the scalar product now taking values in $[0, \infty]$. In this setting the order structure of \L plays an important role and we obtain that the bipolar of a subset of \LO equals its closed, convex and solid hull. In the course of the proof we show a decomposition lemma for convex subsets of \LO into a "bounded" and "hereditarily unbounded" part, which seems interesting in its own right. (author's abstract)Series: Working Papers SFB "Adaptive Information Systems and Modelling in Economics and Management Science

Group sequential and confirmatory adaptive designs in clinical trials

This book provides an up-to-date review of the general principles of and techniques for confirmatory adaptive designs. Confirmatory adaptive designs are a generalization of group sequential designs. With these designs, interim analyses are performed in order to stop the trial prematurely under control of the Type I error rate. In adaptive designs, it is also permissible to perform a data-driven change of relevant aspects of the study design at interim stages. This includes, for example, a sample-size reassessment, a treatment-arm selection or a selection of a pre-specified sub-population. Essentially, this adaptive methodology was introduced in the 1990s. Since then, it has become popular and the object of intense discussion and still represents a rapidly growing field of statistical research. This book describes adaptive design methodology at an elementary level, while also considering designing and planning issues as well as methods for analyzing an adaptively planned trial. This includes estimation methods and methods for the determination of an overall p-value. Part I of the book provides the group sequential methods that are necessary for understanding and applying the adaptive design methodology supplied in Parts II and III of the book. The book contains many examples that illustrate use of the methods for practical application. The book is primarily written for applied statisticians from academia and industry who are interested in confirmatory adaptive designs. It is assumed that readers are familiar with the basic principles of descriptive statistics, parameter estimation and statistical testing. This book will also be suitable for an advanced statistical course for applied statisticians or clinicians with a sound statistical background