Cleaning large correlation matrices: tools from random matrix theory

This review covers recent results concerning the estimation of large covariance matrices using tools from Random Matrix Theory (RMT). We introduce several RMT methods and analytical techniques, such as the Replica formalism and Free Probability, with an emphasis on the Marchenko-Pastur equation that provides information on the resolvent of multiplicatively corrupted noisy matrices. Special care is devoted to the statistics of the eigenvectors of the empirical correlation matrix, which turn out to be crucial for many applications. We show in particular how these results can be used to build consistent "Rotationally Invariant" estimators (RIE) for large correlation matrices when there is no prior on the structure of the underlying process. The last part of this review is dedicated to some real-world applications within financial markets as a case in point. We establish empirically the efficacy of the RIE framework, which is found to be superior in this case to all previously proposed methods. The case of additively (rather than multiplicatively) corrupted noisy matrices is also dealt with in a special Appendix. Several open problems and interesting technical developments are discussed throughout the paper.Comment: 165 pages, article submitted to Physics Report

ESSAYS ON MISSPECIFICATION IN HIGH DIMENSIONAL ECONOMETRICS AND ASSET PRICING

This dissertation examines misspecification issues in two contexts: (i) signal (or equivalently factor) detection in high-dimensional factor models and (ii) the identification of the physical probability distribution of stock returns in the asset pricing literature. The first essay revisits the panel information criteria (IC) proposed by Bai and Ng (2002), which is a popular estimator for the number of factors in high-dimensional factor models, and studies its over-detection risk in finite samples. First, we analyze the finite sample performance of IC by computing the over-detection probability bound. In particular, we specify the asymptotic over-detection condition of IC in terms of eigenvalues coming from pure noise and then derive the computable formula for a non-asymptotic upper bound on the overestimation probability by adopting random matrix theory. We show that unless the sample size is sufficiently large, the overestimation probability is not negligible even for the case in which factors have strong explanatory power. Second, we show that for small sample sizes the over-detection risk of IC is significantly reduced by the degrees of freedom adjustment in the penalty of the original criteria. Finally, we propose modified information criteria (MIC) as a practical guide to improving the finite sample performance of IC. Simulations show that our MIC outperforms IC for the case with weakly serially or cross-sectionally correlated errors as well as i.i.d. errors. The second essay examines the misdetection risk of the panel information criteria (IC) proposed by Bai and Ng (2002) for detecting the number of factors in high-dimensional factor models and examines the optimal penalty to minimize an upper bound on the misdetection probability of the IC estimator in finite samples. This study extends the first chapter, which analyzed the finite sample performance of the IC estimator regarding its over-detection risk, to the comprehensive misdetection risk considering under-detection risk as well. We derive the computable formula for a non-asymptotic upper bound on the misdetection probability by employing recent results from random matrix theory. Using the formula, we analyze the misdetection risk of the IC estimator and achieve the minimum upper bound of the misdetection probability by finding the optimal weight for the penalty function. Our numerical examples suggest that modified criteria with the optimized penalization improve the finite sample performance of the original IC estimator. In my third essay, we revisit the Recovery theorem on the identification of the physical probability distribution of stock returns, proposed by Ross (2015). First, its applicability in fixed-income markets is considered. We suggest a new procedure for applying the Recovery theorem to the Gaussian affine term structure. As a result, we can recover a particular probability distribution and decompose forward rates into the investors\u27 short-rate expectations and term premia under this recovered probability measure. Next, the reliability of the Recovery theorem is examined. In particular, we study its misspecification issue in line with the claim of misspecified recovery by Borovička, Hansen, and Scheinkman (2015). Our empirical result verifies that what Ross really recovers is not the physical probability but the long-term risk-neutral probability which absorbs compensation for exposure to permanent shocks. In consequence, we can decompose forward term premia into nearly constant short-term risk premia associated with transitory shocks and highly volatile long-term risk premia corresponding to permanent shocks. Finally, we find that a secular decline in forward rates is mostly attributed to investors\u27 short-rate expectations under the long-term risk-neutral probability measure, and all important variations in term premia can be captured by long-term risk premia. Concisely, long-term risk matters for asset pricing

Numerical Computation of Wishart Eigenvalue Distributions for Multistatic Radar Detection

abstract: Eigenvalues of the Gram matrix formed from received data frequently appear in sufficient detection statistics for multi-channel detection with Generalized Likelihood Ratio (GLRT) and Bayesian tests. In a frequently presented model for passive radar, in which the null hypothesis is that the channels are independent and contain only complex white Gaussian noise and the alternative hypothesis is that the channels contain a common rank-one signal in the mean, the GLRT statistic is the largest eigenvalue $\lambda_1$ of the Gram matrix formed from data. This Gram matrix has a Wishart distribution. Although exact expressions for the distribution of $\lambda_1$ are known under both hypotheses, numerically calculating values of these distribution functions presents difficulties in cases where the dimension of the data vectors is large. This dissertation presents tractable methods for computing the distribution of $\lambda_1$ under both the null and alternative hypotheses through a technique of expanding known expressions for the distribution of $\lambda_1$ as inner products of orthogonal polynomials. These newly presented expressions for the distribution allow for computation of detection thresholds and receiver operating characteristic curves to arbitrary precision in floating point arithmetic. This represents a significant advancement over the state of the art in a problem that could previously only be addressed by Monte Carlo methods.Dissertation/ThesisDoctoral Dissertation Electrical Engineering 201

On the concentration of large deviations for fat tailed distributions, with application to financial data

Large deviations for fat tailed distributions, i.e. those that decay slower than exponential, are not only relatively likely, but they also occur in a rather peculiar way where a finite fraction of the whole sample deviation is concentrated on a single variable. The regime of large deviations is separated from the regime of typical fluctuations by a phase transition where the symmetry between the points in the sample is spontaneously broken. For stochastic processes with a fat tailed microscopic noise, this implies that while typical realizations are well described by a diffusion process with continuous sample paths, large deviation paths are typically discontinuous. For eigenvalues of random matrices with fat tailed distributed elements, a large deviation where the trace of the matrix is anomalously large concentrates on just a single eigenvalue, whereas in the thin tailed world the large deviation affects the whole distribution. These results find a natural application to finance. Since the price dynamics of financial stocks is characterized by fat tailed increments, large fluctuations of stock prices are expected to be realized by discrete jumps. Interestingly, we find that large excursions of prices are more likely realized by continuous drifts rather than by discontinuous jumps. Indeed, auto-correlations suppress the concentration of large deviations. Financial covariance matrices also exhibit an anomalously large eigenvalue, the market mode, as compared to the prediction of random matrix theory. We show that this is explained by a large deviation with excess covariance rather than by one with excess volatility.Comment: 38 pages, 12 figure

Extreme value analysis for the sample autocovariance matrices of heavy-tailed multivariate time series

We provide some asymptotic theory for the largest eigenvalues of a sample covariance matrix of a p-dimensional time series where the dimension p = p_n converges to infinity when the sample size n increases. We give a short overview of the literature on the topic both in the light- and heavy-tailed cases when the data have finite (infinite) fourth moment, respectively. Our main focus is on the heavytailed case. In this case, one has a theory for the point process of the normalized eigenvalues of the sample covariance matrix in the iid case but also when rows and columns of the data are linearly dependent. We provide limit results for the weak convergence of these point processes to Poisson or cluster Poisson processes. Based on this convergence we can also derive the limit laws of various function als of the ordered eigenvalues such as the joint convergence of a finite number of the largest order statistics, the joint limit law of the largest eigenvalue and the trace, limit laws for successive ratios of ordered eigenvalues, etc. We also develop some limit theory for the singular values of the sample autocovariance matrices and their sums of squares. The theory is illustrated for simulated data and for the components of the S&P 500 stock index.Comment: in Extremes; Statistical Theory and Applications in Science, Engineering and Economics; ISSN 1386-1999; (2016

Concentration Inequalities of Random Matrices and Solving Ptychography with a Convex Relaxation

Random matrix theory has seen rapid development in recent years. In particular, researchers have developed many non-asymptotic matrix concentration inequalities that parallel powerful scalar concentration inequalities. In this thesis, we focus on three topics: 1) estimating sparse covariance matrix using matrix concentration inequalities, 2) constructing the matrix phi-entropy to derive matrix concentration inequalities, 3) developing scalable algorithms to solve the phase recovery problem of ptychography based on low-rank matrix factorization. Estimation of covariance matrix is an important subject. In the setting of high dimensional statistics, the number of samples can be small in comparison to the dimension of the problem, thus estimating the complete covariance matrix is unfeasible. By assuming that the covariance matrix satisfies some sparsity assumptions, prior work has proved that it is feasible to estimate the sparse covariance matrix of Gaussian distribution using the masked sample covariance estimator. In this thesis, we use a new approach and apply non-asymptotic matrix concentration inequalities to obtain tight sample bounds for estimating the sparse covariance matrix of subgaussian distributions. The entropy method is a powerful approach in developing scalar concentration inequalities. The key ingredient is the subadditivity property that scalar entropy function exhibits. In this thesis, we construct a new concept of matrix phi-entropy and prove that matrix phi-entropy also satisfies a subadditivity property similar to the scalar form. We apply this new concept of matrix phi-entropy to derive non-asymptotic matrix concentration inequalities. Ptychography is a computational imaging technique which transforms low-resolution intensity-only images into a high-resolution complex recovery of the signal. Conventional algorithms are based on alternating projection, which lacks theoretical guarantees for their performance. In this thesis, we construct two new algorithms. The first algorithm relies on a convex formulation of the ptychography problem and on low-rank matrix recovery. This algorithm improves traditional approaches' performance but has high computational cost. The second algorithm achieves near-linear runtime and memory complexity by factorizing the objective matrix into its low-rank components and approximates the first algorithm's imaging quality.</p

Optimal Estimation and Rank Detection for Sparse Spiked Covariance Matrices

This paper considers sparse spiked covariance matrix models in the high-dimensional setting and studies the minimax estimation of the covariance matrix and the principal subspace as well as the minimax rank detection. The optimal rate of convergence for estimating the spiked covariance matrix under the spectral norm is established, which requires significantly different techniques from those for estimating other structured covariance matrices such as bandable or sparse covariance matrices. We also establish the minimax rate under the spectral norm for estimating the principal subspace, the primary object of interest in principal component analysis. In addition, the optimal rate for the rank detection boundary is obtained. This result also resolves the gap in a recent paper by Berthet and Rigollet [1] where the special case of rank one is considered

Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions

Low-rank matrix approximations, such as the truncated singular value decomposition and the rank-revealing QR decomposition, play a central role in data analysis and scientific computing. This work surveys and extends recent research which demonstrates that randomization offers a powerful tool for performing low-rank matrix approximation. These techniques exploit modern computational architectures more fully than classical methods and open the possibility of dealing with truly massive data sets. This paper presents a modular framework for constructing randomized algorithms that compute partial matrix decompositions. These methods use random sampling to identify a subspace that captures most of the action of a matrix. The input matrix is then compressed—either explicitly or implicitly—to this subspace, and the reduced matrix is manipulated deterministically to obtain the desired low-rank factorization. In many cases, this approach beats its classical competitors in terms of accuracy, robustness, and/or speed. These claims are supported by extensive numerical experiments and a detailed error analysis. The specific benefits of randomized techniques depend on the computational environment. Consider the model problem of finding the k dominant components of the singular value decomposition of an m × n matrix. (i) For a dense input matrix, randomized algorithms require O(mn log(k)) floating-point operations (flops) in contrast to O(mnk) for classical algorithms. (ii) For a sparse input matrix, the flop count matches classical Krylov subspace methods, but the randomized approach is more robust and can easily be reorganized to exploit multiprocessor architectures. (iii) For a matrix that is too large to fit in fast memory, the randomized techniques require only a constant number of passes over the data, as opposed to O(k) passes for classical algorithms. In fact, it is sometimes possible to perform matrix approximation with a single pass over the data