9 research outputs found
Model Selection with the Loss Rank Principle
A key issue in statistics and machine learning is to automatically select the
"right" model complexity, e.g., the number of neighbors to be averaged over in
k nearest neighbor (kNN) regression or the polynomial degree in regression with
polynomials. We suggest a novel principle - the Loss Rank Principle (LoRP) -
for model selection in regression and classification. It is based on the loss
rank, which counts how many other (fictitious) data would be fitted better.
LoRP selects the model that has minimal loss rank. Unlike most penalized
maximum likelihood variants (AIC, BIC, MDL), LoRP depends only on the
regression functions and the loss function. It works without a stochastic noise
model, and is directly applicable to any non-parametric regressor, like kNN.Comment: 31 LaTeX pages, 1 figur
The Loss Rank Criterion for Variable Selection in Linear Regression Analysis
Lasso and other regularization procedures are attractive methods for variable
selection, subject to a proper choice of shrinkage parameter. Given a set of
potential subsets produced by a regularization algorithm, a consistent model
selection criterion is proposed to select the best one among this preselected
set. The approach leads to a fast and efficient procedure for variable
selection, especially in high-dimensional settings. Model selection consistency
of the suggested criterion is proven when the number of covariates d is fixed.
Simulation studies suggest that the criterion still enjoys model selection
consistency when d is much larger than the sample size. The simulations also
show that our approach for variable selection works surprisingly well in
comparison with existing competitors. The method is also applied to a real data
set.Comment: 18 pages, 1 figur
Model Selection by Loss Rank for Classification and Unsupervised Learning
Hutter (2007) recently introduced the loss rank principle (LoRP) as a general purpose principle for model selection. The LoRP enjoys many attractive properties and deserves further investigations. The LoRP has been well-studied for regression framework in Hutter and Tran (2010). In this paper, we study the LoRP for classification framework, and develop it further for model selection problems in unsupervised learning where the main interest is to describe the associations between input measurements, like cluster analysis or graphical modelling. Theoretical properties and simulation studies are presented
Model selection with the Loss Rank Principle
A key issue in statistics and machine learning is to automatically select the "right" model complexity, e.g., the number of neighbors to be averaged over in k nearest neighbor (k NN) regression or the polynomial degree in regression with polynomials. We suggest a novel principle-the Loss Rank Principle (LoRP)-for model selection in regression and classification. It is based on the loss rank, which counts how many other (fictitious) data would be fitted better. LoRP selects the model that has minimal loss rank. Unlike most penalized maximum likelihood variants (AIC, BIC, MDL), LoRP depends only on the regression functions and the loss function. It works without a stochastic noise model, and is directly applicable to any non-parametric regressor, like k NN
Model selection with the Loss Rank Principle
A key issue in statistics and machine learning is to automatically select the "right" model complexity, e.g., the number of neighbors to be averaged over in k nearest neighbor () regression or the polynomial degree in regression with polynomials. We suggest a novel principle-the Loss Rank Principle (LoRP)-for model selection in regression and classification. It is based on the loss rank, which counts how many other (fictitious) data would be fitted better. LoRP selects the model that has minimal loss rank. Unlike most penalized maximum likelihood variants (AIC, BIC, MDL), LoRP depends only on the regression functions and the loss function. It works without a stochastic noise model, and is directly applicable to any non-parametric regressor, like .