Concentration inequalities of the cross-validation estimate for stable predictors

In this article, we derive concentration inequalities for the cross-validation estimate of the generalization error for stable predictors in the context of risk assessment. The notion of stability has been first introduced by \cite{DEWA79} and extended by \cite{KEA95}, \cite{BE01} and \cite{KUNIY02} to characterize class of predictors with infinite VC dimension. In particular, this covers $k$-nearest neighbors rules, bayesian algorithm (\cite{KEA95}), boosting,... General loss functions and class of predictors are considered. We use the formalism introduced by \cite{DUD03} to cover a large variety of cross-validation procedures including leave-one-out cross-validation, $k$-fold cross-validation, hold-out cross-validation (or split sample), and the leave-$\upsilon$-out cross-validation. In particular, we give a simple rule on how to choose the cross-validation, depending on the stability of the class of predictors. In the special case of uniform stability, an interesting consequence is that the number of elements in the test set is not required to grow to infinity for the consistency of the cross-validation procedure. In this special case, the particular interest of leave-one-out cross-validation is emphasized

Multiplicative local linear hazard estimation and best one-sided cross-validation

This paper develops detailed mathematical statistical theory of a new class of cross-validation techniques of local linear kernel hazards and their multiplicative bias corrections. The new class of cross-validation combines principles of local information and recent advances in indirect cross-validation. A few applications of cross-validating multiplicative kernel hazard estimation do exist in the literature. However, detailed mathematical statistical theory and small sample performance are introduced via this paper and further upgraded to our new class of best one-sided cross-validation. Best one-sided cross-validation turns out to have excellent performance in its practical illustrations, in its small sample performance and in its mathematical statistical theoretical performance

Assessing behavioural changes in ALS: cross-validation of ALS-specific measures

Objective: The Beaumont Behavioural Inventory (BBI) is a behavioural proxy report for the assessment of behavioural changes in ALS. This tool has been validated against the FrSBe, a non-ALS specific behavioural assessment, and further comparison of the BBI against a disease-specific tool was considered. This study cross-validates the BBI against the ALS-FTD-Q. Methods: 60 ALS patients, 8% also meeting criteria for FTD, were recruited. All patients were evaluated using the BBI and the ALS-FTD-Q, completed by a carer. Correlational analysis was performed to assess construct validity. Precision, sensitivity, specificity and overall accuracy of the BBI, when compared to the ALS-FTD-Q, were obtained. Results: The mean score of the whole sample on the BBI was 11.45±13.06. ALS-FTD patients scored significantly higher than non-demented ALS patients (31.6±14.64, 9.62±11.38; p<.0001). A significant large positive correlation between the BBI and the ALS-FTD-Q was observed (r=.807, p<.0001), and no significant correlations between the BBI and other clinical/demographic characteristics, indicating good convergent and discriminant validity, respectively. 72% of overall concordance was observed. Precision, sensitivity and specificity for the classification of severely impaired patients were adequate. However, lower concordance in the classification of mild behavioural changes was observed, with higher sensitivity using the BBI, most likely secondary to BBI items which endorsed behavioural aspects not measured by the ALS-FTD-Q. Discussion: Good construct validity has been further confirmed when the BBI is compared to an ALS-specific tool. Furthermore, the BBI is a more comprehensive behavioural assessment for ALS, as it measures the whole behavioural spectrum in this condition