8,604 research outputs found
Convergence analysis of generalized iteratively reweighted least squares algorithms on convex function spaces
The computation of robust regression estimates often relies on minimization of a convex functional on a convex set. In this paper we discuss a general technique for a large class of convex functionals to compute the minimizers iteratively which is closely related to majorization-minimization algorithms. Our approach is based on a quadratic approximation of the functional to be minimized and includes the iteratively reweighted least squares algorithm as a special case. We prove convergence on convex function spaces for general coercive and convex functionals F and derive geometric convergence in certain unconstrained settings. The algorithm is applied to TV penalized quantile regression and is compared with a step size corrected Newton-Raphson algorithm. It is found that typically in the first steps the iteratively reweighted least squares algorithm performs significantly better, whereas the Newton type method outpaces the former only after many iterations. Finally, in the setting of bivariate regression with unimodality constraints we illustrate how this algorithm allows to utilize highly efficient algorithms for special quadratic programs in more complex settings. --regression analysis,monotone regression,quantile regression,shape constraints,L1 regression,nonparametric regression,total variation semi-norm,reweighted least squares,Fermat's problem,convex approximation,quadratic approximation,pool adjacent violators algorithm
A Generic Path Algorithm for Regularized Statistical Estimation
Regularization is widely used in statistics and machine learning to prevent
overfitting and gear solution towards prior information. In general, a
regularized estimation problem minimizes the sum of a loss function and a
penalty term. The penalty term is usually weighted by a tuning parameter and
encourages certain constraints on the parameters to be estimated. Particular
choices of constraints lead to the popular lasso, fused-lasso, and other
generalized penalized regression methods. Although there has been a lot
of research in this area, developing efficient optimization methods for many
nonseparable penalties remains a challenge. In this article we propose an exact
path solver based on ordinary differential equations (EPSODE) that works for
any convex loss function and can deal with generalized penalties as well
as more complicated regularization such as inequality constraints encountered
in shape-restricted regressions and nonparametric density estimation. In the
path following process, the solution path hits, exits, and slides along the
various constraints and vividly illustrates the tradeoffs between goodness of
fit and model parsimony. In practice, the EPSODE can be coupled with AIC, BIC,
or cross-validation to select an optimal tuning parameter. Our
applications to generalized regularized generalized linear models,
shape-restricted regressions, Gaussian graphical models, and nonparametric
density estimation showcase the potential of the EPSODE algorithm.Comment: 28 pages, 5 figure
Bayesian nonparametric multivariate convex regression
In many applications, such as economics, operations research and
reinforcement learning, one often needs to estimate a multivariate regression
function f subject to a convexity constraint. For example, in sequential
decision processes the value of a state under optimal subsequent decisions may
be known to be convex or concave. We propose a new Bayesian nonparametric
multivariate approach based on characterizing the unknown regression function
as the max of a random collection of unknown hyperplanes. This specification
induces a prior with large support in a Kullback-Leibler sense on the space of
convex functions, while also leading to strong posterior consistency. Although
we assume that f is defined over R^p, we show that this model has a convergence
rate of log(n)^{-1} n^{-1/(d+2)} under the empirical L2 norm when f actually
maps a d dimensional linear subspace to R. We design an efficient reversible
jump MCMC algorithm for posterior computation and demonstrate the methods
through application to value function approximation
Robust Estimation and Wavelet Thresholding in Partial Linear Models
This paper is concerned with a semiparametric partially linear regression
model with unknown regression coefficients, an unknown nonparametric function
for the non-linear component, and unobservable Gaussian distributed random
errors. We present a wavelet thresholding based estimation procedure to
estimate the components of the partial linear model by establishing a
connection between an -penalty based wavelet estimator of the
nonparametric component and Huber's M-estimation of a standard linear model
with outliers. Some general results on the large sample properties of the
estimates of both the parametric and the nonparametric part of the model are
established. Simulations and a real example are used to illustrate the general
results and to compare the proposed methodology with other methods available in
the recent literature
Distributed Kernel Regression: An Algorithm for Training Collaboratively
This paper addresses the problem of distributed learning under communication
constraints, motivated by distributed signal processing in wireless sensor
networks and data mining with distributed databases. After formalizing a
general model for distributed learning, an algorithm for collaboratively
training regularized kernel least-squares regression estimators is derived.
Noting that the algorithm can be viewed as an application of successive
orthogonal projection algorithms, its convergence properties are investigated
and the statistical behavior of the estimator is discussed in a simplified
theoretical setting.Comment: To be presented at the 2006 IEEE Information Theory Workshop, Punta
del Este, Uruguay, March 13-17, 200
A Selective Review of Group Selection in High-Dimensional Models
Grouping structures arise naturally in many statistical modeling problems.
Several methods have been proposed for variable selection that respect grouping
structure in variables. Examples include the group LASSO and several concave
group selection methods. In this article, we give a selective review of group
selection concerning methodological developments, theoretical properties and
computational algorithms. We pay particular attention to group selection
methods involving concave penalties. We address both group selection and
bi-level selection methods. We describe several applications of these methods
in nonparametric additive models, semiparametric regression, seemingly
unrelated regressions, genomic data analysis and genome wide association
studies. We also highlight some issues that require further study.Comment: Published in at http://dx.doi.org/10.1214/12-STS392 the Statistical
Science (http://www.imstat.org/sts/) by the Institute of Mathematical
Statistics (http://www.imstat.org
Distributed Regression in Sensor Networks: Training Distributively with Alternating Projections
Wireless sensor networks (WSNs) have attracted considerable attention in
recent years and motivate a host of new challenges for distributed signal
processing. The problem of distributed or decentralized estimation has often
been considered in the context of parametric models. However, the success of
parametric methods is limited by the appropriateness of the strong statistical
assumptions made by the models. In this paper, a more flexible nonparametric
model for distributed regression is considered that is applicable in a variety
of WSN applications including field estimation. Here, starting with the
standard regularized kernel least-squares estimator, a message-passing
algorithm for distributed estimation in WSNs is derived. The algorithm can be
viewed as an instantiation of the successive orthogonal projection (SOP)
algorithm. Various practical aspects of the algorithm are discussed and several
numerical simulations validate the potential of the approach.Comment: To appear in the Proceedings of the SPIE Conference on Advanced
Signal Processing Algorithms, Architectures and Implementations XV, San
Diego, CA, July 31 - August 4, 200
- …