10,912 research outputs found
A Geometric View on Constrained M-Estimators
We study the estimation error of constrained M-estimators, and derive
explicit upper bounds on the expected estimation error determined by the
Gaussian width of the constraint set. Both of the cases where the true
parameter is on the boundary of the constraint set (matched constraint), and
where the true parameter is strictly in the constraint set (mismatched
constraint) are considered. For both cases, we derive novel universal
estimation error bounds for regression in a generalized linear model with the
canonical link function. Our error bound for the mismatched constraint case is
minimax optimal in terms of its dependence on the sample size, for Gaussian
linear regression by the Lasso
Estimation Error of the Constrained Lasso
This paper presents a non-asymptotic upper bound for the estimation error of the constrained lasso, under the high-dimensional () setting. In contrast to existing results, the error bound in this paper is sharp, is valid when the parameter to be estimated is not exactly sparse (e.g., when it is weakly sparse), and shows explicitly the effect of over-estimating the -norm of the parameter to be estimated on the estimation performance. The results of this paper show that the constrained lasso is minimax optimal for estimating a parameter with bounded -norm, and also for estimating a weakly sparse parameter if its -norm is accessible
Estimation with Norm Regularization
Analysis of non-asymptotic estimation error and structured statistical
recovery based on norm regularized regression, such as Lasso, needs to consider
four aspects: the norm, the loss function, the design matrix, and the noise
model. This paper presents generalizations of such estimation error analysis on
all four aspects compared to the existing literature. We characterize the
restricted error set where the estimation error vector lies, establish
relations between error sets for the constrained and regularized problems, and
present an estimation error bound applicable to any norm. Precise
characterizations of the bound is presented for isotropic as well as
anisotropic subGaussian design matrices, subGaussian noise models, and convex
loss functions, including least squares and generalized linear models. Generic
chaining and associated results play an important role in the analysis. A key
result from the analysis is that the sample complexity of all such estimators
depends on the Gaussian width of a spherical cap corresponding to the
restricted error set. Further, once the number of samples crosses the
required sample complexity, the estimation error decreases as
, where depends on the Gaussian width of the unit norm
ball.Comment: Fixed technical issues. Generalized some result
Lecture notes on ridge regression
The linear regression model cannot be fitted to high-dimensional data, as the
high-dimensionality brings about empirical non-identifiability. Penalized
regression overcomes this non-identifiability by augmentation of the loss
function by a penalty (i.e. a function of regression coefficients). The ridge
penalty is the sum of squared regression coefficients, giving rise to ridge
regression. Here many aspect of ridge regression are reviewed e.g. moments,
mean squared error, its equivalence to constrained estimation, and its relation
to Bayesian regression. Finally, its behaviour and use are illustrated in
simulation and on omics data. Subsequently, ridge regression is generalized to
allow for a more general penalty. The ridge penalization framework is then
translated to logistic regression and its properties are shown to carry over.
To contrast ridge penalized estimation, the final chapter introduces its lasso
counterpart
Denoising and change point localisation in piecewise-constant high-dimensional regression coefficients
We study the theoretical properties of the fused lasso procedure originally proposed by Tibshirani et al. (2005) in the context of a linear regression model in which the regression coefficient are totally ordered and assumed to be sparse and piecewise constant. Despite its popularity, to the best of our knowledge, estimation error bounds in high-dimensional settings have only been obtained for the simple case in which the design matrix is the identity matrix. We formulate a novel restricted isometry condition on the design matrix that is tailored to the fused lasso estimator and derive estimation bounds for both the constrained version of the fused lasso assuming dense coefficients and for its penalised version. We observe that the estimation error can be dominated by either the lasso or the fused lasso rate, depending on whether the number of non-zero coefficient is larger than the number of piece-wise constant segments. Finally, we devise a post-processing procedure to recover the piecewise-constant pattern of the coefficients. Extensive numerical experiments support our theoretical findings
Quantized Low-Rank Multivariate Regression with Random Dithering
Low-rank multivariate regression (LRMR) is an important statistical learning
model that combines highly correlated tasks as a multiresponse regression
problem with low-rank priori on the coefficient matrix. In this paper, we study
quantized LRMR, a practical setting where the responses and/or the covariates
are discretized to finite precision. We focus on the estimation of the
underlying coefficient matrix. To make consistent estimator that could achieve
arbitrarily small error possible, we employ uniform quantization with random
dithering, i.e., we add appropriate random noise to the data before
quantization. Specifically, uniform dither and triangular dither are used for
responses and covariates, respectively. Based on the quantized data, we propose
the constrained Lasso and regularized Lasso estimators, and derive the
non-asymptotic error bounds. With the aid of dithering, the estimators achieve
minimax optimal rate, while quantization only slightly worsens the
multiplicative factor in the error rate. Moreover, we extend our results to a
low-rank regression model with matrix responses. We corroborate and demonstrate
our theoretical results via simulations on synthetic data or image restoration.Comment: 16 pages (Submitted
- …