In M-open problems where no true model can be conceptualized, it is common to
back off from modeling and merely seek good prediction. Even in M-complete
problems, taking a predictive approach can be very useful. Stacking is a model
averaging procedure that gives a composite predictor by combining individual
predictors from a list of models using weights that optimize a cross-validation
criterion. We show that the stacking weights also asymptotically minimize a
posterior expected loss. Hence we formally provide a Bayesian justification for
cross-validation. Often the weights are constrained to be positive and sum to
one. For greater generality, we omit the positivity constraint and relax the
`sum to one' constraint.
A key question is `What predictors should be in the average?' We first verify
that the stacking error depends only on the span of the models. Then we propose
using bootstrap samples from the data to generate empirical basis elements that
can be used to form models. We use this in two computed examples to give
stacking predictors that are (i) data driven, (ii) optimal with respect to the
number of component predictors, and (iii) optimal with respect to the weight
each predictor gets.Comment: 37 pages, 2 figure
