We study empirical covariance matrices in finance. Due to the limited amount
of available input information, these objects incorporate a huge amount of
noise, so their naive use in optimization procedures, such as portfolio
selection, may be misleading. In this paper we investigate a recently
introduced filtering procedure, and demonstrate the applicability of this
method in a controlled, simulation environment.Comment: 9 pages with 3 EPS figure

Kondor, Imre

Nowak, Maciej A.

Pafka, Szilard

Papp, Gabor

Acta Physica Polonica B

English

arXiv

We study empirical covariance matrices in finance. Due to the limited amount of available input information, these objects incorporate a huge amount of noise, so their naive use in optimization procedures, such as portfolio selection, may be misleading. In this paper we investigate a recently introduced filtering procedure, and demonstrate the applicability of this method in a controlled, simulation environment

Papp, G.

Pafka, Sz.

Nowak, Maciej

Kondor, I.

Jagiellonian Univeristy Repository

Vol. 36 (2005) ACTA PHYSICA POLONICA B No 9RANDOM MATRIX FILTERINGIN PORTFOLIO OPTIMIZATION∗G. Pappa, Sz. Pafkaa,b, M.A. Nowakc, I. Kondora,daInstitute of Physics, Eötvös UniversityPázmány P. s. 1/a, H-1117 Budapest, HungarybCIB Bank, Medve u. 4-14, H-1027, Budapest, HungarycMark Kac Complex Systems Research CentreandMarian Smoluchowski Institute for Physics, Jagellonian UniversityReymonta 4, 30-059 Kraków, PolanddCollegium Budapest — Institute for Advanced StudySzentháromság u. 2, H-1014 Budapest, Hungary(Received August 5, 2005)We study empirical covariance matrices in finance. Due to the limitedamount of available input information, these objects incorporate a hugeamount of noise, so their naive use in optimization procedures, such asportfolio selection, may be misleading. In this paper we investigate a re-cently introduced filtering procedure, and demonstrate the applicability ofthis method in a controlled, simulation environment.PACS numbers: 87.23.Ge, 05.45.Tp, 05.40.–a1. IntroductionInvestment decisions are governed by weighing risk vs. reward, thatis possible loss against expected return. Markowitz’ classical portfolio the-ory [1] assumes that the underlying stochastic process is multivariate normalwith known returns and covariances. In practice, these parameters have tobe determined from observations on the market. Since the number of obser-vations is necessarily limited, empirically determined parameters will alwayscontain a certain measurement error. Even if we disregard the notoriouslyhard problem of estimating returns and concentrate solely on the covari-ances, we still run into a problem of serious information imbalance: the sizeN of typical bank portfolios is too large compared to the amount of infor-mation contained in the finite-length time series available for the assets in∗ Presented at the Conference on Applications of Random Matrices to Economy andOther Complex Systems, Kraków, Poland, May 25–28, 2005.(2757)2758 G. Papp et al.the portfolio. As the number of input data is N × T , where T is the lengthof the time series, whereas the number of data needed for the constructionof the covariance matrix is O(N2), we expect that the quality of the esti-mate depends essentially on the ratio N/T and that the error goes to zeroonly in the limit of very small N/T . Now the problem is that N/T is neversufficiently small in practice, in fact, it may easily become larger than unity,the threshold value where the covariance matrix becomes singular and theportfolio selection problem meaningless.Over the past decades a large number of different techniques have beendeveloped to tackle this problem and reduce the effective dimension of largebank portfolios [2]. Our purpose here is to apply a recently introduced filter-ing procedure [3] in a well controlled simulation setting where the efficiencyof the method can be reliably tested.In order to determine the optimal portfolio, one has to invert the co-variance matrix. Since this has, as a rule, a number of small eigenvalues,any measurement error will get amplified and the resulting portfolio will besensitive to the noise. In order to study the effect of noise, we start froma known correlation matrix, dress it with noise, and reestablish the resultsof [4]. Next, we apply the cleaning procedure of [3] to the empirical covari-ance matrix, and investigate the improvement of the result compared withthe original, unfiltered theory.2. Results and discussion2.1. Model correlation matrixLet us start from a known covariance matrix, C, of size N × N , repre-senting the true correlation between N instruments making up the portfolio.The portfolio weights wi , (i = 1, . . . ,N), satisfy the constraint∑i wi = 1,and we assume that short-selling is allowed, i.e. some of the weights can benegative. For the sake of simplicity, we do not impose any further condi-tions on the weights (like e.g. the usual constraint on expected returns, whichcannot be determined on a daily horizon with any reliability anyhow), andconcentrate on the minimal risk portfolio. In a Gaussian world the naturalmeasure of risk is the portfolio variance which is then our objective functionto be minimized,R2 =N∑i,j=1wi Cij wj . (1)After some trivial algebra one finds the optimal weights asw∗i =∑Nj=1 C−1ij∑Ni,j=1 C−1ij. (2)Random Matrix Filtering in Portfolio Optimization 2759It is natural to assume that stocks of companies belonging to a givenindustrial sector are more strongly correlated than those belonging to dif-ferent sectors. Accordingly, we expect that the covariance matrix displaysa block diagonal structure. For simplicity, we assume that the elementsoutside the diagonal blocks (that describe some general correlation with thewhole market) are all equal and non-negative, ̺0 ≥ 0, and the elements̺i, (i = 1, 2, . . .), describing intra-sector correlations in the diagonal blocksare constants within each block, and larger than those outside the blocks,̺i ≥ ̺0. The model just described is the same as the one introduced byNoh [5]. For the sake of simplicity again, we study a case when the cor-relation and covariance matrices are the same, i.e. we set the variance ofindividual instruments to unity. The structure of the correlation matrix,which we will refer to as the market-plus-sectors-model in the following, isthen given by the pattern shown in figure 1. Such a matrix, containing MFig. 1. Structure of the market-plus-sectors model correlation matrix. Correlationwith the market is ̺0, while the correlation inside sector i is ̺i (i = 1, 2, . . .).sectors, possesses M small eigenvalues given by 1 − ̺i < 1, i = 1, 2, . . ..The corresponding eigenvectors will be strongly localized, having only twononzero elements (of equal absolute value but opposite sign). Their multi-plicity is Ni−1, i=1, 2, . . . (where Ni is the number of entries within sector i),i.e. the total multiplicity of the small eigenvalues is N − M . In addition,there are M large eigenvalues (λ > 1), typically singlets, that depend on allthe parameters of the model: ̺0, ̺i, and Ni. That is, an M sector matrixhas 2M different eigenvalues. By virtue of the Frobenius–Perron theorem,2760 G. Papp et al.the largest eigenvalue will necessarily be a singlet of O(N), with an eigen-vector having all positive components. This mode can then be identifiedwith the whole market.On the whole, this simple model reproduces all the main features ob-served in the spectra of real-life empirical covariance matrices [6]. It willalso be useful to consider two special cases of the model: When all the ̺i’sare the same and equal to ̺0, the block structure disappears, and we areleft with a one-factor model, where only covariances with the market mat-ter. The spectrum of the corresponding covariance matrix consists of merelytwo values, a large eigenvalue of order N , and an (N − 1)-fold degeneratesmall eigenvalue, 1 − ̺0 < 1. As a further simplification, we can even drop̺0, and end up with the unit matrix as the most trivial covariance matrixconceivable.2.2. Empirical correlationAssuming that we have chosen one of the above models, the market-plus-sectors model, the one-factor model, or just the unit matrix, we can constructthe corresponding empirical correlation matrix from them as follows: wegenerate finite time series from the true correlation matrix C,xit =∑jAi,j yjt t = 1, . . . , T , (3)where A is the Cholesky decomposition of the true correlation matrixC = AAT , and yjt is a random Gaussian variable with mean zero andvariance 1 at time t.Then the empirical correlation matrix is given by the usual estimator asC(e) =1TT∑t=1xitxjt . (4)The resulting empirical covariances will fluctuate from sample to sample.The main effect of this noise will be to resolve the degeneracy of the smalleigenvalues, so that for a large enough matrix they form a quasi-continuousband. For N and T going to infinity so that r = N/T is fixed and smallerthan one, the spectral density of the small eigenvalues will be given by theMarchenko–Pastur spectrum [7]. (For r larger than one, an additional Dirac-delta appears at the origin.) For small enough r’s the large eigenvaluesremain relatively unaffected by the noise, but as r grows and approachesunity, the effect of noise becomes dramatic, as we demonstrate below.Random Matrix Filtering in Portfolio Optimization 2761The Markowitz-weights corresponding to the empirical covariance matrixarew(e)i =∑Nj=1 C(e)−1ij∑Ni,j=1 C(e)−1ij. (5)Now we can evaluate the risk associated with this choice of the portfolio.A possible way to characterize the effect of measurement error is to eval-uate the variance by using the true correlation matrix C with the weightscalculated from the empirical one, C(e),R(e)2=N∑i,j=1w(e)i Cij w(e)j . (6)Since the empirical weights are not optimal, we always have R(e)2 ≥ R2.In the following we will useq20 =R(e)R≥ 1 (7)as a measure of the effect of noise on portfolio selection. q0 can be easilyevaluated for the special case when the true covariance matrix is just the unitmatrix. Then the empirical covariance matrix will be a random matrix witha spectral density fast converging to the Marchenko–Pastur spectrum [7]:̺(λ) =12rπλ√(λ − λ<)(λ> − λ) , (8)where r = N/T , λ<,> = (1 ±√r)2. Evaluating q0 in the diagonal represen-tation, we getq20 =∫dλ 1λ2̺(λ)(∫dλ 1λ̺(λ))2 =11 − r . (9)This strikingly simple result, dating back to a discussion between thepresent authors, was first published in [4]. It remains valid up to O(1/N)corrections also for the one-factor model, and, within corrections controlledby the size of the sectors, also for the market-plus-sectors model. It tells usthat as the size N of the portfolio grows and approaches the length of thetime series T , the error in the portfolio diverges. While it is a commonplacethat at the threshold N = T the portfolio problem becomes meaningless (the2762 G. Papp et al.covariance matrix looses its positive definite character), it does not seem tohave been noticed in the quantitative finance literature earlier that the errorcan be given by such a simple exact formula.A comparison between the theoretical prediction and the simulation isdisplayed in figure 2: the agreement is perfect. Concerning the relevance ofour simple result for real markets, one has to realize that it has been derivedon the basis of idealized conditions: perfect stationarity of the process andGaussian distribution of returns. Neither of these holds true on real markets,therefore we believe that our formula is a lower bound for the error in real-lifeportfolios. 0.1 1 10 100 0.5  1  1.5  2q 0r=N/TOne-factor modelcleanedrawtheoryFig. 2. One factor model results for N = 100, q0 versus r: optimization withempirical correlation matrices (errorbars), random matrix result (solid line), andoptimization with the cleaned one-sector correlation matrix (stars). For r > 1 thestandard Markowitz theory is not applicable.Given the fact that r is never small in practice, and, in fact, it mayeven go beyond the critical value r = 1, it is imperative that some sort offiltering or cleaning procedure be applied, in order to reduce the effect ofnoise. A number of these techniques is available in the literature [2]. Eachof them corresponds to injecting some external information, additional tothe time series data, into the empirical covariance matrix. The procedureproposed recently in [3] requires that we make an educated guess concerningthe structure of the market. We are going to test its performance in thenext section.Random Matrix Filtering in Portfolio Optimization 27632.3. Cleaned correlationStudies of real financial empirical matrices [6] have revealed that theyonly have a relatively small number of large eigenvalues (in the case of theS&P500 less than 20), the rest are small, and conform rather well to theMarchenko–Pastur pattern (8). This must mean that the number of relevantsectors is fairly small compared with the size of the portfolio. Our task istherefore to reconstruct the true correlation matrix assuming a structurewith a few sectors.The general theory of the cleaning procedure of the empirical correlationmatrix dressed with Gaussian noise is described in [3]. Let G(Z) be theresolvent for the cleaned correlation matrixG(Z) =1NTr1Z − C , (10)with a similar formula for the resolvent g(z) of the empirical correlationmatrix. Then the relation between the two is expressed asz g(z) = Z G(Z), with z =Z1 − r + rZG(Z) . (11)This can be translated into a relation between the corresponding moments,and from the knowledge of 2M − 1 moments one is able to reconstruct Msectors for the true correlation matrix. We also note that this procedure,applied in the reverse direction, allows one to calculate the spectral densityfor more complicated scenarios, for correlated random matrices. As a result,the eigenvalue spectrum will slightly, but noticeably change, and come to acloser agreement with the one observed in finance. The procedure describedin [3] allows one to reconstruct the eigenvalues only, but not the eigenvectors.Our aim here is to reconstruct the true correlation matrix using the cleanedeigenvalues and the empirical eigenvectors. The question is whether such aprocedure can lead to any improvement?We present our result in figures 2, 3, for the one-factor model and themarket-plus-sectors model, respectively. The true correlation matrix is wellreconstructed in both cases, the portfolio built from the (cleaned) empiricaldata is suboptimal by only 5–10%. Thus, the cleaning procedure leads to avery substantial improvement compared with the naive use of the empiricalcovariance matrix, and allows the optimization to be performed even in therange N > T .The cleaning procedure does not determine the number of sectors, orof the eigenvalues to be searched for, it is a parameter of the fitting. Wetested the method with different numbers of eigenvalues (i.e. different num-bers of sectors), and observed the changes in performance depending on the2764 G. Papp et al. 1 1.5 2 2.5 3 0.5  1  1.5  2q 0r=N/T2 eig.4 eig.6 eig.8 eig.theoryFig. 3. Market-plus-sectors model results for N = 100, q0 versus r, for differentnumbers of eigenvalues reconstructed. Above 4 eigenvalues the result does notchange anymore.trial structure. We find that there is a clear saturation after one reachesthe number of eigenvalues corresponding the number of sectors in the truecorrelation matrix, and a further increase of the number of searched eigen-values does not changes the result. This allows one to determine the propernumber of sectors by looking for the signature of saturation.On the other hand, no problem arises if one sets the number of searchedeigenvalues higher than required: the cleaning procedure will return lessindependent eigenvalues, saturating at the maximal number allowed by thenumber of sectors.In conclusion, we have performed a preliminary study of the effect ofthe random matrix based cleaning described in [3] on the optimization offinancial portfolios. We have found that the method works very efficientlyin an artificial test environment, reproducing nearly perfectly a one-factormodel scenario, and leading to a huge improvement in a market-plus-sectorsmodel with a moderate number of sectors. It is obvious that before a finaljudgment can be passed on the method, a more detailed study of its variousaspects should be performed, extending also to a possible cleaning of theeigenvectors. Furthermore, a careful comparison of the efficiency of themethod with other filtering procedures proposed in the literature is clearlynecessary.Random Matrix Filtering in Portfolio Optimization 2765Two of us (I.K. and Sz.P.) are grateful for the hospitality of the Institutefor Theoretical Sciences — A Joint Institute of Argonne National Labo-ratory and University of Notre Dame, where part of this manuscript hasbeen prepared. G.P. acknowledges the support of Hungarian OTKA grantT047050.REFERENCES[1] H. Markowitz, J. Fin. 7, 91 (1952).[2] E.J. Elton, M.J. Gruber, Modern Portfolio Theory and Investment Analysis,J. Wiley and Sons, New York 1995.[3] Z. Burda, A. Goerlich, A. Jarosz, J. Jurkiewicz, Physica A 343, 295 (2004).[4] Sz. Pafka, I. Kondor, Physica A 319, 487 (2003).[5] J.D. Noh, Phys. Rev. E61, 5981 (2000).[6] L. Laloux, P. Cizeau, J.-P. Bouchaud, M. Potters, Phys. Rev. Lett. 83, 1467(1999); V. Plerou, P. Gopikrishnan, B. Rosenow, L.A.N. Amaral, H.E. Stanley,Phys. Rev. Lett. 83, 1471 (1999).[7] V.A. Marchenko, L.A. Pastur, Math. USSR-Sbornik, 1, 457 (1967).

Random matrix filtering in portfolio optimization

Random Matrix Filtering in Portfolio Optimization

Random Matrix Filtering in Portfolio Optimization

Abstract

Similar works

Full text

Available Versions

Jagiellonian Univeristy Repository