54,314 research outputs found
Bootstrap-Based Inference for Cube Root Asymptotics
This paper proposes a valid bootstrap-based distributional approximation for
M-estimators exhibiting a Chernoff (1964)-type limiting distribution. For
estimators of this kind, the standard nonparametric bootstrap is inconsistent.
The method proposed herein is based on the nonparametric bootstrap, but
restores consistency by altering the shape of the criterion function defining
the estimator whose distribution we seek to approximate. This modification
leads to a generic and easy-to-implement resampling method for inference that
is conceptually distinct from other available distributional approximations. We
illustrate the applicability of our results with four examples in econometrics
and machine learning
Active Sampling for Class Probability Estimation and Ranking
In many cost-sensitive environments class probability estimates are used by decision
makers to evaluate the expected utility from a set of alternatives. Supervised
learning can be used to build class probability estimates; however, it often is very
costly to obtain training data with class labels. Active sampling acquires data incrementally,
at each phase identifying especially useful additional data for labeling,
and can be used to economize on examples needed for learning. We outline the
critical features for an active sampling approach and present an active sampling
method for estimating class probabilities and ranking. BOOTSTRAP-LV identifies particularly
informative new data for learning based on the variance in probability estimates,
and by accounting for a particular data item's informative value for the
rest of the input space. We show empirically that the method reduces the number
of data items that must be obtained and labeled, across a wide variety of domains.
We investigate the contribution of the components of the algorithm and show that
each provides valuable information to help identify informative examples. We also
compare BOOTSTRAP-LV with UNCERTAINTY SAMPLING,a n existing active sampling
method designed to maximize classification accuracy. The results show that BOOTSTRAP-LV uses fewer examples to exhibit a certain class probability estimation accuracy
and provide insights on the behavior of the algorithms. Finally, to further our
understanding of the contributions made by the elements of BOOTSTRAP-LV, we experiment
with a new active sampling algorithm drawing from both UNCERTAINIY
SAMPLING and BOOTSTRAP-LV and show that it is significantly more competitive
with BOOTSTRAP-LV compared to UNCERTAINTY SAMPLING. The analysis suggests
more general implications for improving existing active sampling algorithms for
classification.Information Systems Working Papers Serie
The Theory Behind Overfitting, Cross Validation, Regularization, Bagging, and Boosting: Tutorial
In this tutorial paper, we first define mean squared error, variance,
covariance, and bias of both random variables and classification/predictor
models. Then, we formulate the true and generalization errors of the model for
both training and validation/test instances where we make use of the Stein's
Unbiased Risk Estimator (SURE). We define overfitting, underfitting, and
generalization using the obtained true and generalization errors. We introduce
cross validation and two well-known examples which are -fold and
leave-one-out cross validations. We briefly introduce generalized cross
validation and then move on to regularization where we use the SURE again. We
work on both and norm regularizations. Then, we show that
bootstrap aggregating (bagging) reduces the variance of estimation. Boosting,
specifically AdaBoost, is introduced and it is explained as both an additive
model and a maximum margin model, i.e., Support Vector Machine (SVM). The upper
bound on the generalization error of boosting is also provided to show why
boosting prevents from overfitting. As examples of regularization, the theory
of ridge and lasso regressions, weight decay, noise injection to input/weights,
and early stopping are explained. Random forest, dropout, histogram of oriented
gradients, and single shot multi-box detector are explained as examples of
bagging in machine learning and computer vision. Finally, boosting tree and SVM
models are mentioned as examples of boosting.Comment: 23 pages, 9 figure
A bagging SVM to learn from positive and unlabeled examples
We consider the problem of learning a binary classifier from a training set
of positive and unlabeled examples, both in the inductive and in the
transductive setting. This problem, often referred to as \emph{PU learning},
differs from the standard supervised classification problem by the lack of
negative examples in the training set. It corresponds to an ubiquitous
situation in many applications such as information retrieval or gene ranking,
when we have identified a set of data of interest sharing a particular
property, and we wish to automatically retrieve additional data sharing the
same property among a large and easily available pool of unlabeled data. We
propose a conceptually simple method, akin to bagging, to approach both
inductive and transductive PU learning problems, by converting them into series
of supervised binary classification problems discriminating the known positive
examples from random subsamples of the unlabeled set. We empirically
demonstrate the relevance of the method on simulated and real data, where it
performs at least as well as existing methods while being faster
- …