24,249 research outputs found
Lecture notes on ridge regression
The linear regression model cannot be fitted to high-dimensional data, as the
high-dimensionality brings about empirical non-identifiability. Penalized
regression overcomes this non-identifiability by augmentation of the loss
function by a penalty (i.e. a function of regression coefficients). The ridge
penalty is the sum of squared regression coefficients, giving rise to ridge
regression. Here many aspect of ridge regression are reviewed e.g. moments,
mean squared error, its equivalence to constrained estimation, and its relation
to Bayesian regression. Finally, its behaviour and use are illustrated in
simulation and on omics data. Subsequently, ridge regression is generalized to
allow for a more general penalty. The ridge penalization framework is then
translated to logistic regression and its properties are shown to carry over.
To contrast ridge penalized estimation, the final chapter introduces its lasso
counterpart
Tensor-on-tensor regression
We propose a framework for the linear prediction of a multi-way array (i.e.,
a tensor) from another multi-way array of arbitrary dimension, using the
contracted tensor product. This framework generalizes several existing
approaches, including methods to predict a scalar outcome from a tensor, a
matrix from a matrix, or a tensor from a scalar. We describe an approach that
exploits the multiway structure of both the predictors and the outcomes by
restricting the coefficients to have reduced CP-rank. We propose a general and
efficient algorithm for penalized least-squares estimation, which allows for a
ridge (L_2) penalty on the coefficients. The objective is shown to give the
mode of a Bayesian posterior, which motivates a Gibbs sampling algorithm for
inference. We illustrate the approach with an application to facial image data.
An R package is available at https://github.com/lockEF/MultiwayRegression .Comment: 33 pages, 3 figure
Bayesian Regularisation in Structured Additive Regression Models for Survival Data
During recent years, penalized likelihood approaches have attracted a lot of interest both in the area of semiparametric regression and for the regularization of high-dimensional regression models. In this paper, we introduce a Bayesian formulation that allows to combine both aspects into a joint regression model with a focus on hazard regression for survival times. While Bayesian penalized splines form the basis for estimating nonparametric and flexible time-varying effects, regularization of high-dimensional covariate vectors is based on scale mixture of normals priors. This class of priors allows to keep a (conditional) Gaussian prior for regression coefficients on the predictor stage of the model but introduces suitable mixture distributions for the Gaussian variance to achieve regularization. This scale mixture property allows to device general and adaptive Markov chain Monte Carlo simulation algorithms for fitting a variety of hazard regression models. In particular, unifying algorithms based on iteratively weighted least squares proposals can be employed both for regularization and penalized semiparametric function estimation. Since sampling based estimates do no longer have the variable selection property well-known for the Lasso in frequentist analyses, we additionally consider spike and slab priors that introduce a further mixing stage that allows to separate between influential and redundant parameters. We demonstrate the different shrinkage properties with three simulation settings and apply the methods to the PBC Liver dataset
Fully Bayesian Penalized Regression with a Generalized Bridge Prior
We consider penalized regression models under a unified framework. The
particular method is determined by the form of the penalty term, which is
typically chosen by cross validation. We introduce a fully Bayesian approach
that incorporates both sparse and dense settings and show how to use a type of
model averaging approach to eliminate the nuisance penalty parameters and
perform inference through the marginal posterior distribution of the regression
coefficients. We establish tail robustness of the resulting estimator as well
as conditional and marginal posterior consistency for the Bayesian model. We
develop a component-wise Markov chain Monte Carlo algorithm for sampling.
Numerical results show that the method tends to select the optimal penalty and
performs well in both variable selection and prediction and is comparable to,
and often better than alternative methods. Both simulated and real data
examples are provided
- …