Let $X_1,..., X_N\in\R^n$ be independent centered random vectors with
log-concave distribution and with the identity as covariance matrix. We show
that with overwhelming probability at least $1 - 3 \exp(-c\sqrt{n}\r)$ one has
$
  \sup_{x\in S^{n-1}} \Big|\frac{1/N}\sum_{i=1}^N (||^2 - \E|<X_i,
x>|^2\r)\Big|
  \leq C \sqrt{\frac{n/N}},$ where $C$ is an absolute positive constant. This
result is valid in a more general framework when the linear forms
$()_{i\leq N, x\in S^{n-1}}$ and the Euclidean norms $(|X_i|/\sqrt
n)_{i\leq N}$ exhibit uniformly a sub-exponential decay. As a consequence, if
$A$ denotes the random matrix with columns $(X_i)$, then with overwhelming
probability, the extremal singular values $\lambda_{\rm min}$ and $\lambda_{\rm
max}$ of $AA^\top$ satisfy the inequalities $ 1 - C\sqrt{{n/N}} \le
{\lambda_{\rm min}/N} \le \frac{\lambda_{\rm max}/N} \le 1 + C\sqrt{{n/N}} $
which is a quantitative version of Bai-Yin theorem \cite{BY} known for random
matrices with i.i.d. entries

Adamczak, Radosław

Litvak, Alexander E.

Pajor, Alain

Tomczak-Jaegermann, Nicole

English

arXiv

Radosław Adamczak

Alexander E. Litvak

Alain Pajor

Nicole Tomczak-Jaegermann

Crossref

Comptes Rendus Mathematique

Sharp bounds on the rate of convergence of the empirical covariance matrix

Numérisation de Documents Anciens Mathématiques

C. R. Acad. Sci. Paris, Ser. I 349 (2011) 195–200Contents lists available at ScienceDirectC. R. Acad. Sci. Paris, Ser. Iwww.sciencedirect.comFunctional Analysis/Probability TheorySharp bounds on the rate of convergence of the empirical covariancematrix ✩Ordre asymptotique des valeurs singulières extrêmes de la matrice de covarianceempiriqueRadosław Adamczak a,1, Alexander E. Litvak b, Alain Pajor c, Nicole Tomczak-Jaegermann b,2a Institute of Mathematics, University of Warsaw, Banacha 2, 02-097 Warszawa, Polandb Department of Mathematical and Statistical Sciences, University of Alberta, Edmonton, Alberta, Canada T6G 2G1c Équipe d’analyse et mathématiques appliquées, université Paris Est, 5, boulevard Descartes, Champs-sur-Marne, 77454 Marne-la-Vallee cedex 2, Francea r t i c l e i n f o a b s t r a c tArticle history:Received 20 November 2010Accepted 21 December 2010Available online 6 January 2011Presented by Gilles PisierLet X1, . . . , XN ∈ Rn be independent centered random vectors with log-concave distributionand with the identity as covariance matrix. We show that with overwhelming probabilityone hassupx∈Sn−1∣∣∣∣∣ 1NN∑i=1(∣∣〈Xi, x〉∣∣2 − E∣∣〈Xi, x〉∣∣2)∣∣∣∣∣  C√nN,where C is an absolute positive constant. This result is valid in a more general frameworkwhen the linear forms (〈Xi, x〉)iN , x ∈ Sn−1 and the Euclidean norms (|Xi |/√n )iN exhibituniformly a sub-exponential decay. As a consequence, if A denotes the random matrixwith columns (Xi), then with overwhelming probability, the extremal singular values λminand λmax of A A satisfy the inequalities 1 − C√nN  λminN  λmaxN  1 + C√nN which is aquantitative version of Bai–Yin theorem (Z.D. Bai, Y.Q. Yin, 1993 [4]) known for randommatrices with i.i.d. entries.© 2010 Académie des sciences. Published by Elsevier Masson SAS. All rights reserved.r é s u m éSoient X1, . . . , XN ∈ Rn des vecteurs aléatoires indépendants centrés, de matrice decovariance l’identité et à densité log-concave. On démontre qu’avec une grande probabilité,on asupx∈Sn−1∣∣∣∣∣ 1NN∑i=1(∣∣〈Xi, x〉∣∣2 − E∣∣〈Xi, x〉∣∣2)∣∣∣∣∣  C√nN,où C > 0 est une constante numérique. Ce résultat reste vrai dans le cadre beaucoupplus général où les formes linéaires (〈Xi, x〉)iN , x ∈ Sn−1 et les normes euclidiennes✩ The research was conducted while the authors participated in the Thematic Program on Asymptotic Geometric Analysis at the Fields Institute in Torontoin Fall 2010.E-mail addresses: R.Adamczak@mimuw.edu.pl (R. Adamczak), alexandr@math.ualberta.ca (A.E. Litvak), alain.pajor@univ-mlv.fr (A. Pajor),nicole@ellpspace.math.ualberta.ca (N. Tomczak-Jaegermann).1 Research partially supported by MNiSW Grant no. N N201 397437 and the Foundation for Polish Science.2 This author holds the Canada Research Chair in Geometric Analysis.1631-073X/$ – see front matter © 2010 Académie des sciences. Published by Elsevier Masson SAS. All rights reserved.doi:10.1016/j.crma.2010.12.014196 R. Adamczak et al. / C. R. Acad. Sci. Paris, Ser. I 349 (2011) 195–200(|Xi |/√n )iN vérifient des inégalités de type sous-exponentiel. Il en résulte que si Adésigne la matrice dont les colonnes sont (Xi), alors avec grande probabilité, les valeurssingulières extrêmes λmin et λmax de A A vérifient 1 − C√nN  λminN  λmaxN  1 + C√nN ,ce qui est une version quantitative du théorème de Bai–Yin (Z.D. Bai, Y.Q. Yin, 1993 [4])bien connu pour les matrices aléatoires à coefficients i.i.d.© 2010 Académie des sciences. Published by Elsevier Masson SAS. All rights reserved.Version française abrégéeSoient X ∈ Rn un vecteur aléatoire centré de matrice de covariance l’identité et X1, . . . , XN ∈ Rn un échantillon detaille N de vecteurs aléatoires indépendants de même loi que X . Soit A la matrice aléatoire n × N dont les colonnessont les vecteurs (Xi). On note λmin (resp. λmax) la plus petite (resp. la plus grande) valeur singulière de la matrice decovariance empirique A A . Dans l’étude du régime local en théorie des matrices aléatoires, on s’intéresse particulièrementau comportement asymptotique des valeurs extrêmes du spectre de A A . Dans le cas de l’Ensemble de Wishart où levecteur X est à coordonnées i.i.d. le théorème de Bai–Yin assure la convergence de λmin/N et λmax/N lorsque n, N → ∞et n/N → β ∈ (0,1), sous des hypothèses de moment d’ordre 4. Dans cette Note, on se place d’un point de vue « non-asymptotique » et dans un cadre où les coordonnées des vecteurs colonnes sont non i.i.d. et l’on cherche à quantifier lecomportement des valeurs extrêmes lorsque n et N sont fixés. Dans le cas gaussien par exemple, on montre que les valeurssingulières extrêmes vérifient les inégalités1 − C√nN λminN λmaxN 1 + C√nN(1)avec grande probabilité, où C > 0 est une constante numérique.On considère l’espace Rn muni du produit scalaire naturel 〈·,·〉 ainsi que de la norme euclidienne associée | · |. Onutilisera cette même notation | · | pour désigner le cardinal d’un ensemble.Dans cette Note, on démontre que les inégalités (1) sont vérifiées sous les hypothèses suivantes :i) X1, . . . , XN sont des vecteurs centrés, indépendants, uniformément ψ1, c’est-à-dire qu’il existe ψ > 0 tel quesupiNsupy∈Sn−1∥∥∣∣〈Xi, y〉∣∣∥∥ψ1  ψ,où pour une variable aléatoire Y ∈ R on a posé ‖Y ‖ψ1 = inf{C > 0;E exp(|Y |/C)  2}.ii) Il existe K > 0 tel queP(maxiN|Xi |/√n > K max{1, (N/n)1/4}) exp(−√n ).Un échantillon de vecteurs uniformément distribués dans la boule euclidienne de rayon K√n vérifie ces hypothèses. Unexemple particulièrement intéressant de classe qui vérifie ces hypothèses est l’Ensemble log-concave (voir [1]) : on supposeque les vecteurs (Xi) sont indépendants, centrés et log-concaves isotropes. Rappelons qu’un vecteur est isotrope quand samatrice de covariance est l’identité et qu’il est log-concave si sa distribution est à densité log-concave. Les vecteurs (Xi)vérifient i) et ii) avec ψ et K des constantes universelles. La propriété ii) résulte d’un résultat de Paouris [7] (voir [1],Lemma 3.1).Des vecteurs aléatoires de Rn indépendants isotropes (Xi)iN qui vérifient une inégalité de Poincaré avec constante L,c’est-à-dire tels que Var( f (Xi))  L2E|∇ f (Xi)|2 pour toute fonction régulière f à support compact, remplissent les hypo-thèses i) et ii) avec ψ = C L et K = C L où C est une constante numérique.Dans le cadre qui précède, la matrice de covariance de X étant l’identité, la relation (1) est une conséquence du théorèmeplus général suivant :Théorème 1. Soient N,n  1 des entiers et ψ, K  1. Soient X1, . . . , XN ∈ Rn des vecteurs aléatoires indépendants vérifiant i) et ii)avec les paramètres ψ et K . Alors avec une probabilité supérieure à 1 − 2 exp(−c√n ), on asupx∈Sn−1∣∣∣∣∣ 1NN∑i=1(∣∣〈Xi, x〉∣∣2 − E∣∣〈Xi, x〉∣∣2)∣∣∣∣∣  C(ψ + K )2√nN,où C, c > 0 sont des constantes numériques.Ce résultat améliore l’encadrement obtenu dans [1] où le facteur√n/N du second membre était remplacé par√n/N ln(2N/n). Il est aussi optimal aux constantes numériques près puisque qu’il donne une estimation similaire au casR. Adamczak et al. / C. R. Acad. Sci. Paris, Ser. I 349 (2011) 195–200 197gaussien. L’estimation en√n/N ln(2N/n) de [1] était suffisante pour résoudre le problème de Kannan, Lovász et Simonovits[6] qui concernait le cas N proportionel à n.Outre que le théorème 1 est optimal, c’est une version quantitative du théorème de Bai–Yin valable dans un cadre noni.i.d. (les coordonnées de X ne sont pas a priori indépendantes). La question de certains principes universels de la théoriedes matrices aléatoires peut être posé. En particulier, le résultat asymptotique de Bai–Yin pour les valeurs extrêmes duspectre de la matrice de covariance empirique est-il vrai pour l’Ensemble log-concave. La version quantitative démontréedans cette Note permet à présent de supporter cette conjecture.La démonstration du théorème 1 s’appuie sur le résultat suivant de [1] (Theorem 3.13), un choix judicieux des paramètreset un argument d’approximation.Théorème 2. Soient A une matrice n × N et 1  m  N. On considère le paramètre Am défini plus loin par (4). Sous les mêmeshypothèses que le théorème 1, pour tout t  1 on aP(∃m  N, Am  Cψt max{√m ln2Nm,√n}+ 6 maxiN|Xi |) exp(−t√n ),où C > 0 est une constante numérique.1. IntroductionLet X ∈ Rn be a centered random vector whose covariance matrix is the identity and X1, . . . , XN be independent copiesof X . Let A be a random n × N matrix whose columns are (Xi). By λmin (resp. λmax) we denote the smallest (resp. thelargest) singular number of the matrix of empirical covariance A A . In the study of the local regime in the random matrixtheory of particular interest is the limit behaviour of extremal values of the spectrum of A A . In the case of WishartEnsemble when the coordinates of X are independent, the Bai–Yin theorem [4] establishes the convergence of λmin/N andλmax/N when n, N → ∞ and n/N → β ∈ (0,1), under the assumption of a finite fourth moment. In this Note we study theasymptotic non-limit behaviour (also called “non-asymptotic” in Statistics), i.e., we look for sharp upper and lower boundsfor singular values in terms of n and N , when n and N are sufficiently large. For example, for Gaussian matrices it is knownthat singular values satisfy inequalities (1) with probability close to 1. We obtain the same estimates for large class ofrandom matrices, which in particular do not require that entries of the matrix are independent or that Xi ’s are identicallydistributed. Note that the natural question about convergence of singular values in such a case is still open (see [2] for thecase of Xi having uniform distribution on a rescaled np ball).The natural scalar product and Euclidean norm on Rn are denoted by 〈·,·〉 and | · |. We also denote by the same notation| · | the cardinality of a set. By C , C1, c, etc. we will denote absolute positive constants.Let X1, . . . , XN be a sequence of random vectors in Rn (not necessarily identically distributed). We say that it is uniformlyψ1 if for some ψ > 0,supiNsupy∈Sn−1∥∥∣∣〈Xi, y〉∣∣∥∥ψ1  ψ, (2)where for a random variable Y ∈ R, ‖Y ‖ψ1 = inf{C > 0;E exp(|Y |/C)  2}. We say that it satisfies the boundedness conditionwith constant K (for some K  1) ifP(maxiN|Xi|/√n > K max{1, (N/n)1/4}) exp(−√n ). (3)2. Main resultThe main result of this Note is the following theorem:Theorem 1. Let N,n be positive integers and ψ, K  1. Let X1, . . . , XN be independent random vectors in Rn satisfying (2) and (3).Then with probability at least 1 − 2 exp(−c√n ) one hassupx∈Sn−1∣∣∣∣∣ 1NN∑i=1(∣∣〈Xi, x〉∣∣2 − E∣∣〈Xi, x〉∣∣2)∣∣∣∣∣  C(ψ + K )2√nN.Theorem 1 improves estimates obtained in [1] for log-concave isotropic vectors. There, we considered essentially thecase of N proportional to n, which was sufficient to answer the question of Kannan, Lovász and Simonovits [6], however,for bigger N , the results were off by a logarithmic factor. The theorem above removes this factor completely leading to thebest possible estimate for an arbitrary N , that is to an estimate of the same order as in the Gaussian case.As a consequence, we obtain in our setting, the following quantitative version of Bai–Yin theorem [4] known for randommatrices with i.i.d. entries.198 R. Adamczak et al. / C. R. Acad. Sci. Paris, Ser. I 349 (2011) 195–200Corollary 1. Let A be a random n × N matrix, whose columns X1, . . . , XN are isotropic random vectors satisfying the assumptions ofTheorem 1. Then with probability at least 1 − 2 exp(−c√n ),1 − C(ψ + K )2√nN λminN λmaxN 1 + C(ψ + K )2√nN.To emphasize the strength of the above results we observe that conditions (2) and (3) are valid for many classes ofdistributions.Example 1. Random vectors uniformly distributed on the Euclidean ball of radius K√n clearly satisfy (3). They also satisfy(2) with ψ = C K .Example 2. Log-concave isotropic random vectors in Rn . Recall that a random vector is isotropic if its covariance matrix isthe identity and it is log-concave if its distribution has a log-concave density. Such vectors satisfy (2) and (3) for appropriateabsolute constants ψ and K . The boundedness condition follows from Paouris’ theorem [7] and is explicitly written, e.g., in[1], Lemma 3.1. We would like to remark that a version of Theorem 1 with a weaker probability estimate was proved byAubrun in the case of isotropic log-concave random vectors under an additional assumption of unconditionality (see [3]).Example 3. Any isotropic random vectors (Xi)iN in Rn , satisfying the Poincaré inequality with constant L, i.e. such thatVar( f (Xi))  L2E|∇ f (Xi)|2 for all compactly supported smooth functions, satisfy (2) with ψ = C L and (3) with K = C L. Thequestion from [5] whether all log-concave isotropic random vectors satisfy the Poincaré inequality with an absolute constantis one of the major open problems in the theory of log-concave measures.3. Proof of Theorem 1The proof of Theorem 1 is close to arguments in Section 4.3 of [1], however it uses a choice of parameters moreappropriate for the case considered here, and a new approximation argument. We need additional notations. Let 1  m  N .By Um we denote the subset of all vectors in S N−1 having at most m non-zero coordinates. For an n × N matrix A we letAm = supz∈Um|Az|. (4)The main technical tool is the following result which is the “in particular” part of Theorem 3.13 from [1] in which oneneeds to adjust corresponding constants and to take a union bound.Theorem 2. Let X1, . . . , XN be as in Theorem 1, let A be a random n × N matrix whose columns are the Xi ’s. Then for every t  1 onehasP(∃m, Am  Cψt max{√m ln2Nm,√n}+ 6 maxiN|Xi |) exp(−t√n ).Proof of Theorem 1. We assume N  n, otherwise Theorem 2 implies Theorem 1. For x ∈ Sn−1 setS(x) =∣∣∣∣∣ 1NN∑i=1(∣∣〈Xi, x〉∣∣2 − E∣∣〈Xi, x〉∣∣2)∣∣∣∣∣.Let B > 0 be a parameter which we specify later and observe thatsupx∈Sn−1S(x)  supx∈Sn−1(∣∣∣∣∣ 1NN∑i=1((∣∣〈Xi, x〉∣∣ ∧ B)2 − E(∣∣〈Xi, x〉∣∣ ∧ B)2)∣∣∣∣∣+ 1NN∑i=1(∣∣〈Xi, x〉∣∣2 − B2)1{|〈Xi ,x〉|B} + 1N EN∑i=1(∣∣〈Xi, x〉∣∣2 − B2)1{|〈Xi ,x〉|B}).We denote the summands under the supremum by S1(x), S2(x), and S3(x), respectively.Estimate for S1: Given x ∈ Sn−1 and i  N let Zi = Zi(x) = (|〈Xi, x〉|∧ B)2 −E(|〈Xi, x〉|∧ B)2. Then |Zi |  B2. Moreover, sinceVar(Zi)  E(∣∣〈Xi, x〉∣∣ ∧ B)4  E∣∣〈Xi, x〉∣∣4  C1ψ4,we observe that σ 2 = 1 ∑Ni=1 Var(Zi)  C1ψ4. Thus, by Bernstein’s inequalityNR. Adamczak et al. / C. R. Acad. Sci. Paris, Ser. I 349 (2011) 195–200 199P(S1(x)  θ) = P(1NN∑i=1Zi  θ) exp(− θ2N2(C1ψ4 + B2θ/3)).It is well known that Sn−1 admits a (1/3)-net N in the Euclidean metric such that |N |  7n . Then by the union bound weobtain that ifθ2N > 8C1ψ4n ln 7 and θ N > (8/3)B2n ln 7 (5)thenP(supx∈NS1(x)  θ) exp(n ln 7 − θ2N2(C1ψ4 + B2θ/3)) exp(− θ2N4(C1ψ4 + B2θ/3)). (6)Estimates for S2 and S3: By Hölder’s inequality and (2) we have, for some absolute constant C2  1,supx∈Sn−1S3(x) 1NN∑i=1supx∈Sn−1∥∥〈Xi, x〉∥∥24P(∣∣〈Xi, x〉∣∣  B)1/2  C2ψ2 exp(−B/ψ). (7)To estimate S2, we will use the following notationM = max{ψ2n,maxiN|Xi |2}, E B = E B(x) ={i  N:∣∣〈Xi, x〉∣∣  B}, m = supx∈Sn−1∣∣E B(x)∣∣.By the definition of Am , we have for every x ∈ Sn−1B2|E B | ∑i∈E B∣∣〈Xi, x〉∣∣2  sup|E|m∑i∈E∣∣〈Xi, x〉∣∣2  A2m,which yields B2m  A2m and N S2(x)  A2m . Theorem 2 implies that for some absolute constant C  C2, with probability atleast 1 − exp(−√n ) one hasB2m  C(M + ψ2m ln2 2Nm)and supx∈Sn−1S2(x)  C(MN+ ψ2 mNln22Nm). (8)Now we choose the parameters. Let B = 2√2Cψ ln(5N/n). Then (7) gives S3(x)  Cψ2 nN  C MN for all x ∈ Sn−1 and togetherwith (8) it yields that with probability at least 1 − exp(−√n ) one hassupx∈Sn−1(S2(x) + S3(x)) C((2M/N) + ψ2(m/N) ln2(2N/m)).It is easy to check that M  ψ2m ln2(2N/m) on the set where (8) holds. Indeed, assume it is not so, thus M <ψ2m ln2(2N/m). Then by (8) we observe that B2  2Cψ2 ln2(2N/m), which impliesm  2N exp(−B/ψ√2C ) = 2n2/25N.By our hypothetical upper bound for M and since f (m) = m ln2(2N/m) increases on [1,2N/e2], we getψ2n  M  ψ2(8n2/25N)ln2(5N/n),which is impossible.It follows that P(supx∈Sn−1 (S2(x) + S3(x))  3C(M/N))  1 − exp(−√n ). Combining this estimate with (6), we getP(supx∈N S(x)  θ + 3C MN )  1 − exp(−√n ) − exp(− θ2 N4(C1ψ4+B2θ/3) ). We now set θ = C3ψ2√n/N , where C3 is a sufficientlylarge absolute positive constant so that (5) is satisfied. Then using boundedness condition with constant K we obtainP(supx∈NS(x) (C3ψ2 + 3C K 2)√n/N)  1 − exp(−√n ) − exp(−cn)  1 − 2 exp(−c√n ),where c is a sufficiently small positive constant. It proves the desired estimate on the (1/3)-net.To pass from N to the whole sphere note that S(x) can be written as |〈T x, x〉|, where T is a self-adjoint operator on Rn .Thus, writing for each x ∈ Sn−1, x = y + z with y ∈ N and |z|  1/3, we get‖T ‖ = supx∈Sn−1∣∣〈T x, x〉∣∣  supy∈N∣∣〈T y, y〉∣∣ + 23supy∈N|T y| + sup|z|1/3∣∣〈T z, z〉∣∣  supy∈NS(y) + 79‖T ‖,which implies the desired estimate on the whole sphere Sn−1. 200 R. Adamczak et al. / C. R. Acad. Sci. Paris, Ser. I 349 (2011) 195–200References[1] R. Adamczak, A.E. Litvak, A. Pajor, N. Tomczak-Jaegermann, Quantitative estimates of the convergence of the empirical covariance matrix in log-concaveensembles, J. Amer. Math. Soc. 234 (2010) 535–561.[2] G. Aubrun, Random points in the unit ball of np , Positivity 10 (2006) 755–759.[3] G. Aubrun, Sampling convex bodies: a random matrix approach, Proc. Amer. Math. Soc. 135 (2007) 1293–1303.[4] Z.D. Bai, Y.Q. Yin, Limit of the smallest eigenvalue of a large dimensional sample covariance matrix, Ann. Probab. 21 (1993) 1275–1294.[5] R. Kannan, L. Lovász, M. Simonovits, Isoperimetric problems for convex bodies and a localization lemma, Discrete Comput. Geom. 13 (3–4) (1995)541–559.[6] R. Kannan, L. Lovász, M. Simonovits, Random walks and O ∗(n5) volume algorithm for convex bodies, Random Structures Algorithms 2 (1) (1997) 1–50.[7] G. Paouris, Concentration of mass on convex bodies, Geom. Funct. Anal. 16 (5) (2006) 1021–1049.

Comptes Rendus Mathématique

International audienceLetX1,...,XN ∈Rn,n≤N,beindependentcenteredrandomvectorswithlog-concavedistribution and with the identity as covariance matrix. We show that with overwhelming probability one has , decay. As a consequence, if A denotes the random matrix with columns (Xi), then with overwhelming probability, the extremﰅal singular values λmin and λmax of AA⊤ satisfy the inequalities 1 − Cﰅ n ≤ N λmin ≤ λmax ≤ 1 + C n which is a quantitative version of Bai-Yin theorem [4] known for random NNN matrices with i.i.d. entries

Litvak, Alexander

HAL - UPEC / UPEM

Sharp bounds on the rate of convergence of the empirical covariance
  matrix

Sharp bounds on the rate of convergence of the empirical covariance matrix

Abstract

Similar works

Full text

Available Versions

Crossref

Numérisation de Documents Anciens Mathématiques

Comptes Rendus Mathématique

HAL - UPEC / UPEM