Spectrum Estimation: A Unified Framework for Covariance Matrix Estimation and PCA in Large Dimensions

Covariance matrix estimation and principal component analysis (PCA) are two cornerstones of multivariate analysis. Classic textbook solutions perform poorly when the dimension of the data is of a magnitude similar to the sample size, or even larger. In such settings, there is a common remedy for both statistical problems: nonlinear shrinkage of the eigenvalues of the sample covariance matrix. The optimal nonlinear shrinkage formula depends on unknown population quantities and is thus not available. It is, however, possible to consistently estimate an oracle nonlinear shrinkage, which is motivated on asymptotic grounds. A key tool to this end is consistent estimation of the set of eigenvalues of the population covariance matrix (also known as the spectrum), an interesting and challenging problem in its own right. Extensive Monte Carlo simulations demonstrate that our methods have desirable finite-sample properties and outperform previous proposals.Comment: 40 pages, 8 figures, 5 tables, University of Zurich, Department of Economics, Working Paper No. 105, Revised version, July 201

Robust Pilot Decontamination Based on Joint Angle and Power Domain Discrimination

We address the problem of noise and interference corrupted channel estimation in massive MIMO systems. Interference, which originates from pilot reuse (or contamination), can in principle be discriminated on the basis of the distributions of path angles and amplitudes. In this paper we propose novel robust channel estimation algorithms exploiting path diversity in both angle and power domains, relying on a suitable combination of the spatial filtering and amplitude based projection. The proposed approaches are able to cope with a wide range of system and topology scenarios, including those where, unlike in previous works, interference channel may overlap with desired channels in terms of multipath angles of arrival or exceed them in terms of received power. In particular we establish analytically the conditions under which the proposed channel estimator is fully decontaminated. Simulation results confirm the overall system gains when using the new methods.Comment: 14 pages, 5 figures, accepted for publication in IEEE Transactions on Signal Processin

Signal Processing in Large Systems: a New Paradigm

For a long time, detection and parameter estimation methods for signal processing have relied on asymptotic statistics as the number $n$ of observations of a population grows large comparatively to the population size $N$, i.e. $n/N\to \infty$. Modern technological and societal advances now demand the study of sometimes extremely large populations and simultaneously require fast signal processing due to accelerated system dynamics. This results in not-so-large practical ratios $n/N$, sometimes even smaller than one. A disruptive change in classical signal processing methods has therefore been initiated in the past ten years, mostly spurred by the field of large dimensional random matrix theory. The early works in random matrix theory for signal processing applications are however scarce and highly technical. This tutorial provides an accessible methodological introduction to the modern tools of random matrix theory and to the signal processing methods derived from them, with an emphasis on simple illustrative examples

When do improved covariance matrix estimators enhance portfolio optimization? An empirical comparative study of nine estimators

The use of improved covariance matrix estimators as an alternative to the sample estimator is considered an important approach for enhancing portfolio optimization. Here we empirically compare the performance of 9 improved covariance estimation procedures by using daily returns of 90 highly capitalized US stocks for the period 1997-2007. We find that the usefulness of covariance matrix estimators strongly depends on the ratio between estimation period T and number of stocks N, on the presence or absence of short selling, and on the performance metric considered. When short selling is allowed, several estimation methods achieve a realized risk that is significantly smaller than the one obtained with the sample covariance method. This is particularly true when T/N is close to one. Moreover many estimators reduce the fraction of negative portfolio weights, while little improvement is achieved in the degree of diversification. On the contrary when short selling is not allowed and T>N, the considered methods are unable to outperform the sample covariance in terms of realized risk but can give much more diversified portfolios than the one obtained with the sample covariance. When T<N the use of the sample covariance matrix and of the pseudoinverse gives portfolios with very poor performance.Comment: 30 page