Ridge Fusion in Statistical Learning

We propose a penalized likelihood method to jointly estimate multiple precision matrices for use in quadratic discriminant analysis and model based clustering. A ridge penalty and a ridge fusion penalty are used to introduce shrinkage and promote similarity between precision matrix estimates. Block-wise coordinate descent is used for optimization, and validation likelihood is used for tuning parameter selection. Our method is applied in quadratic discriminant analysis and semi-supervised model based clustering.Comment: 24 pages and 9 tables, 3 figure

Estimating sufficient reductions of the predictors in abundant high-dimensional regressions

We study the asymptotic behavior of a class of methods for sufficient dimension reduction in high-dimension regressions, as the sample size and number of predictors grow in various alignments. It is demonstrated that these methods are consistent in a variety of settings, particularly in abundant regressions where most predictors contribute some information on the response, and oracle rates are possible. Simulation results are presented to support the theoretical conclusion.Comment: Published in at http://dx.doi.org/10.1214/11-AOS962 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org

Sparse permutation invariant covariance estimation

The paper proposes a method for constructing a sparse estimator for the inverse covariance (concentration) matrix in high-dimensional settings. The estimator uses a penalized normal likelihood approach and forces sparsity by using a lasso-type penalty. We establish a rate of convergence in the Frobenius norm as both data dimension $p$ and sample size $n$ are allowed to grow, and show that the rate depends explicitly on how sparse the true concentration matrix is. We also show that a correlation-based version of the method exhibits better rates in the operator norm. We also derive a fast iterative algorithm for computing the estimator, which relies on the popular Cholesky decomposition of the inverse but produces a permutation-invariant estimator. The method is compared to other estimators on simulated data and on a real data example of tumor tissue classification using gene expression data.Comment: Published in at http://dx.doi.org/10.1214/08-EJS176 the Electronic Journal of Statistics (http://www.i-journals.org/ejs/) by the Institute of Mathematical Statistics (http://www.imstat.org

Fitted value shrinkage

We propose a penalized least-squares method to fit the linear regression model with fitted values that are invariant to invertible linear transformations of the design matrix. This invariance is important, for example, when practitioners have categorical predictors and interactions. Our method has the same computational cost as ridge-penalized least squares, which lacks this invariance. We derive the expected squared distance between the vector of population fitted values and its shrinkage estimator as well as the tuning parameter value that minimizes this expectation. In addition to using cross validation, we construct two estimators of this optimal tuning parameter value and study their asymptotic properties. Our numerical experiments and data examples show that our method performs similarly to ridge-penalized least-squares

On the existence of the weighted bridge penalized Gaussian likelihood precision matrix estimator

We establish a necessary and sufficient condition for the existence of the precision matrix estimator obtained by minimizing the negative Gaussian log-likelihood plus a weighted bridge penalty. This condition enables us to connect the literature on Gaussian graphical models to the literature on penalized Gaussian likelihood.Fil: Rothman, Adam J.. University of Minnesota. Scool Of Statistics; Estados UnidosFil: Forzani, Liliana Maria. Universidad Nacional del Litoral. Facultad de Ingeniería Química; Argentina. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Santa Fe. Instituto de Matemática Aplicada "Litoral"; Argentin

On the existence of the weighted bridge penalized Gaussian likelihood precision matrix estimator

We establish a necessary and sufficient condition for the existence of the precision matrix estimator obtained by minimizing the negative Gaussian log-likelihood plus a weighted bridge penalty. This condition enables us to connect the literature on Gaussian graphical models to the literature on penalized Gaussian likelihood.Fil: Rothman, Adam J.. University of Minnesota. Scool Of Statistics; Estados UnidosFil: Forzani, Liliana Maria. Universidad Nacional del Litoral. Facultad de Ingeniería Química; Argentina. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Santa Fe. Instituto de Matemática Aplicada "Litoral"; Argentin

Sparse Estimation of High-Dimensional Covariance Matrices.

This thesis develops methodology and asymptotic analysis for sparse estimators of the covariance matrix and the inverse covariance (concentration) matrix in high-dimensional settings. We propose estimators that are invariant to the ordering of the variables and estimators that exploit variable ordering. For the estimators that are invariant to the ordering of the variables, estimation is based on both lasso-type penalized normal likelihood and a new proposed class of generalized thresholding operators which combine thresholding with shrinkage applied to the entries of the sample covariance matrix. For both approaches we obtain explicit convergence rates in matrix norms that show the trade-off between the sparsity of the true model, dimension, and the sample size. In addition, we show that the generalized thresholding approach estimates true zeros as zeros with probability tending to 1, and is sign consistent for non-zero elements. We also derive a fast iterative algorithm for computing the penalized likelihood estimator. To exploit a natural ordering of the variables to estimate the covariance matrix, we propose a new regression interpretation of the Cholesky factor of the covariance matrix, as opposed to the well known regression interpretation of the Cholesky factor of the inverse covariance, which leads to a new class of regularized covariance estimators suitable for high-dimensional problems. We also establish theoretical connections between banding Cholesky factors of the covariance matrix and its inverse and constrained maximum likelihood estimation under the banding constraint. These covariance estimators are compared to other estimators on simulated data and on real data examples from gene microarray experiments and remote sensing. Lastly, we propose a procedure for constructing a sparse estimator of a multivariate regression coefficient matrix that accounts for correlation of the response variables. An efficient optimization algorithm and a fast approximation are developed and we show that the proposed method outperforms relevant competitors when the responses are highly correlated. We also apply the new method to a finance example on predicting asset returns.Ph.D.StatisticsUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/75832/1/ajrothma_1.pd