2,095 research outputs found
Ridge Estimation of Inverse Covariance Matrices from High-Dimensional Data
We study ridge estimation of the precision matrix in the high-dimensional
setting where the number of variables is large relative to the sample size. We
first review two archetypal ridge estimators and note that their utilized
penalties do not coincide with common ridge penalties. Subsequently, starting
from a common ridge penalty, analytic expressions are derived for two
alternative ridge estimators of the precision matrix. The alternative
estimators are compared to the archetypes with regard to eigenvalue shrinkage
and risk. The alternatives are also compared to the graphical lasso within the
context of graphical modeling. The comparisons may give reason to prefer the
proposed alternative estimators
Fixed effects selection in the linear mixed-effects model using adaptive ridge procedure for L0 penalty performance
This paper is concerned with the selection of fixed effects along with the
estimation of fixed effects, random effects and variance components in the
linear mixed-effects model. We introduce a selection procedure based on an
adaptive ridge (AR) penalty of the profiled likelihood, where the covariance
matrix of the random effects is Cholesky factorized. This selection procedure
is intended to both low and high-dimensional settings where the number of fixed
effects is allowed to grow exponentially with the total sample size, yielding
technical difficulties due to the non-convex optimization problem induced by L0
penalties. Through extensive simulation studies, the procedure is compared to
the LASSO selection and appears to enjoy the model selection consistency as
well as the estimation consistency
The Loss Rank Criterion for Variable Selection in Linear Regression Analysis
Lasso and other regularization procedures are attractive methods for variable
selection, subject to a proper choice of shrinkage parameter. Given a set of
potential subsets produced by a regularization algorithm, a consistent model
selection criterion is proposed to select the best one among this preselected
set. The approach leads to a fast and efficient procedure for variable
selection, especially in high-dimensional settings. Model selection consistency
of the suggested criterion is proven when the number of covariates d is fixed.
Simulation studies suggest that the criterion still enjoys model selection
consistency when d is much larger than the sample size. The simulations also
show that our approach for variable selection works surprisingly well in
comparison with existing competitors. The method is also applied to a real data
set.Comment: 18 pages, 1 figur
Regularized Ordinal Regression and the ordinalNet R Package
Regularization techniques such as the lasso (Tibshirani 1996) and elastic net
(Zou and Hastie 2005) can be used to improve regression model coefficient
estimation and prediction accuracy, as well as to perform variable selection.
Ordinal regression models are widely used in applications where the use of
regularization could be beneficial; however, these models are not included in
many popular software packages for regularized regression. We propose a
coordinate descent algorithm to fit a broad class of ordinal regression models
with an elastic net penalty. Furthermore, we demonstrate that each model in
this class generalizes to a more flexible form, for instance to accommodate
unordered categorical data. We introduce an elastic net penalty class that
applies to both model forms. Additionally, this penalty can be used to shrink a
non-ordinal model toward its ordinal counterpart. Finally, we introduce the R
package ordinalNet, which implements the algorithm for this model class
Lecture notes on ridge regression
The linear regression model cannot be fitted to high-dimensional data, as the
high-dimensionality brings about empirical non-identifiability. Penalized
regression overcomes this non-identifiability by augmentation of the loss
function by a penalty (i.e. a function of regression coefficients). The ridge
penalty is the sum of squared regression coefficients, giving rise to ridge
regression. Here many aspect of ridge regression are reviewed e.g. moments,
mean squared error, its equivalence to constrained estimation, and its relation
to Bayesian regression. Finally, its behaviour and use are illustrated in
simulation and on omics data. Subsequently, ridge regression is generalized to
allow for a more general penalty. The ridge penalization framework is then
translated to logistic regression and its properties are shown to carry over.
To contrast ridge penalized estimation, the final chapter introduces its lasso
counterpart
Model Selection with the Loss Rank Principle
A key issue in statistics and machine learning is to automatically select the
"right" model complexity, e.g., the number of neighbors to be averaged over in
k nearest neighbor (kNN) regression or the polynomial degree in regression with
polynomials. We suggest a novel principle - the Loss Rank Principle (LoRP) -
for model selection in regression and classification. It is based on the loss
rank, which counts how many other (fictitious) data would be fitted better.
LoRP selects the model that has minimal loss rank. Unlike most penalized
maximum likelihood variants (AIC, BIC, MDL), LoRP depends only on the
regression functions and the loss function. It works without a stochastic noise
model, and is directly applicable to any non-parametric regressor, like kNN.Comment: 31 LaTeX pages, 1 figur
Penalized Regression with Ordinal Predictors
Ordered categorial predictors are a common case in regression modeling. In contrast to the case of ordinal response variables, ordinal predictors have been largely neglected in the literature. In this article penalized regression techniques are proposed. Based on dummy coding two types of penalization are explicitly developed; the first imposes a difference penalty, the second is a ridge type refitting procedure. A Bayesian motivation as well as alternative ways of derivation are provided. Simulation studies and real world data serve for illustration and to
compare the approach to methods often seen in practice, namely linear regression on the group labels and pure dummy coding. The proposed regression techniques turn out to be highly competitive. On the basis of GLMs the concept is generalized to the case of non-normal outcomes by performing penalized likelihood estimation. The paper is a preprint of an article published in the International Statistical Review. Please use the journal version for citation
- …