Two Stage Dantzig Selector for High Dimensional Data

학위논문 (박사)-- 서울대학교 대학원 : 통계학과, 2014. 2. 김용대.Variable selection is important in high dimensional regression. The traditional variable selection methods such as stepwise selection are unstable which means that the set of the selected variables are varying according to the data sets. As an alternative to those methods, a series of penalized methods are used for estimation and variable selection simultaneously. The LASSO yields sparse solution, but it is not selection consistent and biased. Non-convex penalized methods such as the SCAD and the MCP are known to be selection consistent and yield unbiased estimator. However they suffer from multiple local minima and their computations are unstable for tuning parameter. Two stage methods based on the LASSO such as one step LLA and calibrated CCCP are developed which can obtain the oracle estimator as the unique local minimum. We propose a two stage method based on Dantzig selector. The motivation of our proposed method is that lessening the effect of the noise variables is important in the two stage method. The l1 norm of the Dantzig selector is always less than equal to that of the LASSO and the non-asymptotic error bounds of Dantzig selector tent to be lesser than those of LASSO for the same tuning parameter. Therefore we expect the improvement on the estimation using the Dantzig selector instead of the LASSO in the two stage method while this proposed method also satisfies the selection consistency. The results of the numerical experiments can support our contention. We also apply these two stage methods which are based on LASSO or Dantzig selector to estimation of inverse covariance matrix (a.k.a. precision matrix). Precision matrix estimation is essential not only because it can be used in various applications but also because it refers to the direct relationship between variables via the conditional dependence of variables under the normality assumption. Under some regularity conditions our methods hold selection consistency and obtain columnwise root n-consistent estimator for true nonzero precision matrix elements. The numerical analyses show that the proposed methods perform well in terms of variable selection and estimation.Abstract i 1 Introduction 1 1.1 Overview 1 1.2 Outline of the thesis 3 2 Two Stage Dantzig Selecotr for High Dimensional Linear Regression 5 2.1 Introduction 5 2.2 Sparse regularization methods 8 2.2.1 The l1 regularization 9 2.2.2 Nonconvex penalized methods 13 2.2.3 Two stage methods 21 2.3 Two Stage Dantzig Selector 25 2.3.1 Method 25 2.3.2 Motivation 25 2.3.3 Theoretical properties 29 2.3.4 Algorithm 36 2.3.5 Tuning regularization parameter 37 2.4 Numerical analyses 37 2.4.1 Simulations 39 2.4.2 Real data analysis 44 2.5 Conclusion 45 3 Two Stage Methods for Precision Matrix Estimation 46 3.1 Introduction 46 3.2 Estimation of precision matrix via columnwise two-stage methods 49 3.2.1 Two stage method based on LASSO 50 3.2.2 Two stage Dantzig selector 53 3.2.3 Theoretical results 55 3.3 Numerical analyses 64 3.3.1 Simulations 64 3.3.2 Real data analysis 83 3.4 Conclusion 87 4 Concluding remarks 88 5 Appendix 90 5.1 Algorithms for Dantzig selector 90 5.1.1 Primal-dual interior point algorithm (Candes and Romberg, 2005) 91 5.1.2 DASSO (James et al., 2009) 96 5.1.3 Alternating direction method (ADM) (Lu et al., 2012) 100 Abstract (in Korean) 111 감사의 글 114Docto