Assessing the performance of an allocation rule

AbstractThe problem of estimating the error rates of a sample-based rule on the basis of the same sample used in its construction is considered. The apparent error rate is an obvious nonparametric estimate of the conditional error rate of a sample rule, but unfortunately it provides too optimistic an assessment. Attention is focussed on the formation of improved estimates, mainly through appropriate bias correction of the apparent error rate. In this respect the role of the bootstrap, a computer-based methodology, is highlighted

Error Rate Estimation On the Basis of Posterior Probabilities

The so-called posterior probability estimator, e, formed by averaging the minimum of the posterior probabilities over a set of initial or additional observations (which need not be classified) is considered in the context of estimating the overall actual error rate for the linear discriminant function appropriate for two multivariate normal populations with a common covariance matrix. The bias of e is examined by deriving asymptotic approximations under three different models, the normal, logistic, and mixture models. The properties of e are investigated further by a series of simulation experiments for the logistic and mixture models for which there are few other available estimators