52,489 research outputs found
On Low-rank Trace Regression under General Sampling Distribution
A growing number of modern statistical learning problems involve estimating a
large number of parameters from a (smaller) number of noisy observations. In a
subset of these problems (matrix completion, matrix compressed sensing, and
multi-task learning) the unknown parameters form a high-dimensional matrix B*,
and two popular approaches for the estimation are convex relaxation of
rank-penalized regression or non-convex optimization. It is also known that
these estimators satisfy near optimal error bounds under assumptions on rank,
coherence, or spikiness of the unknown matrix.
In this paper, we introduce a unifying technique for analyzing all of these
problems via both estimators that leads to short proofs for the existing
results as well as new results. Specifically, first we introduce a general
notion of spikiness for B* and consider a general family of estimators and
prove non-asymptotic error bounds for the their estimation error. Our approach
relies on a generic recipe to prove restricted strong convexity for the
sampling operator of the trace regression. Second, and most notably, we prove
similar error bounds when the regularization parameter is chosen via K-fold
cross-validation. This result is significant in that existing theory on
cross-validated estimators do not apply to our setting since our estimators are
not known to satisfy their required notion of stability. Third, we study
applications of our general results to four subproblems of (1) matrix
completion, (2) multi-task learning, (3) compressed sensing with Gaussian
ensembles, and (4) compressed sensing with factored measurements. For (1), (3),
and (4) we recover matching error bounds as those found in the literature, and
for (2) we obtain (to the best of our knowledge) the first such error bound. We
also demonstrate how our frameworks applies to the exact recovery problem in
(3) and (4).Comment: 32 pages, 1 figur
Oracle Inequalities for Convex Loss Functions with Non-Linear Targets
This paper consider penalized empirical loss minimization of convex loss
functions with unknown non-linear target functions. Using the elastic net
penalty we establish a finite sample oracle inequality which bounds the loss of
our estimator from above with high probability. If the unknown target is linear
this inequality also provides an upper bound of the estimation error of the
estimated parameter vector. These are new results and they generalize the
econometrics and statistics literature. Next, we use the non-asymptotic results
to show that the excess loss of our estimator is asymptotically of the same
order as that of the oracle. If the target is linear we give sufficient
conditions for consistency of the estimated parameter vector. Next, we briefly
discuss how a thresholded version of our estimator can be used to perform
consistent variable selection. We give two examples of loss functions covered
by our framework and show how penalized nonparametric series estimation is
contained as a special case and provide a finite sample upper bound on the mean
square error of the elastic net series estimator.Comment: 44 page
A convexity approach to dynamic output feedback robust MPC for LPV systems with bounded disturbances
International audienceA convexity approach to dynamic output feedback robust model predictive control (OFRMPC) is proposed for linear parameter varying (LPV) systems with bounded disturbances. At each sampling time, the model parameters and disturbances are assumed to be unknown but bounded within pre-specified convex sets. Robust stability conditions on the augmented closed-loop system are derived using the techniques of robust positively invariant (RPI) set and the S-procedure. A convexity method reformulates the non-convex bilinear matrix inequalities (BMIs) problem as a convex optimization one such that the on-line computational burden is significantly reduced. The on-line optimized dynamic output feedback controller parameters steer the augmented states to converge within RPI sets and recursive feasibility of the optimization problem is guaranteed. Furthermore, bounds of the estimation error set are refreshed by updating the shape matrix of the future ellipsoidal estimation error set. The dynamic OFRMPC approach guarantees that the disturbance-free augmented closed-loop system (without consideration of disturbances) converges to the origin. In addition, when the system is subject to bounded disturbances, the augmented closed-loop system converges to a neighborhood of the origin. Two simulation examples are given to verify the effectiveness of the approach
A Geometric View on Constrained M-Estimators
We study the estimation error of constrained M-estimators, and derive
explicit upper bounds on the expected estimation error determined by the
Gaussian width of the constraint set. Both of the cases where the true
parameter is on the boundary of the constraint set (matched constraint), and
where the true parameter is strictly in the constraint set (mismatched
constraint) are considered. For both cases, we derive novel universal
estimation error bounds for regression in a generalized linear model with the
canonical link function. Our error bound for the mismatched constraint case is
minimax optimal in terms of its dependence on the sample size, for Gaussian
linear regression by the Lasso
Sparse Identification and Estimation of Large-Scale Vector AutoRegressive Moving Averages
The Vector AutoRegressive Moving Average (VARMA) model is fundamental to the
theory of multivariate time series; however, in practice, identifiability
issues have led many authors to abandon VARMA modeling in favor of the simpler
Vector AutoRegressive (VAR) model. Such a practice is unfortunate since even
very simple VARMA models can have quite complicated VAR representations. We
narrow this gap with a new optimization-based approach to VARMA identification
that is built upon the principle of parsimony. Among all equivalent
data-generating models, we seek the parameterization that is "simplest" in a
certain sense. A user-specified strongly convex penalty is used to measure
model simplicity, and that same penalty is then used to define an estimator
that can be efficiently computed. We show that our estimator converges to a
parsimonious element in the set of all equivalent data-generating models, in a
double asymptotic regime where the number of component time series is allowed
to grow with sample size. Further, we derive non-asymptotic upper bounds on the
estimation error of our method relative to our specially identified target.
Novel theoretical machinery includes non-asymptotic analysis of infinite-order
VAR, elastic net estimation under a singular covariance structure of
regressors, and new concentration inequalities for quadratic forms of random
variables from Gaussian time series. We illustrate the competitive performance
of our methods in simulation and several application domains, including
macro-economic forecasting, demand forecasting, and volatility forecasting
- …