Charles Dossal

Comptes Rendus Mathématique

C. R. Acad. Sci. Paris, Ser. I 350 (2012) 117–120Contents lists available at SciVerse ScienceDirectC. R. Acad. Sci. Paris, Ser. Iwww.sciencedirect.comTheory of SignalsA necessary and sufficient condition for exact sparse recovery by 1minimizationUne condition nécessaire et suffisante d’identifiabilité parcimonieuse par minimisation 1Charles DossalIMB, Université Bordeaux 1, 351, Cours de la Libération, 33405 Talence cedex, Francea r t i c l e i n f o a b s t r a c tArticle history:Received 18 November 2011Accepted after revision 21 December 2011Available online 9 January 2012Presented by Yves MeyerIn this Note, a new sharp sufficient condition for exact sparse recovery by 1-penalizedminimization from linear measurements is proposed. The main contribution of this paperis to show that, for most matrices, this condition is also necessary. Moreover, whenthe 1 minimizer is unique, we investigate its sensitivity to the measurements and weestablish that the application associating the measurements to this minimizer is Lipschitz-continuous.© 2012 Published by Elsevier Masson SAS on behalf of Académie des sciences.r é s u m éDans cette Note, une nouvelle condition suffisante pour l’identifiabilité parcimonieuse parminimisation 1 pénalisée à partir de mesures linéaires est proposée. La contributionmajeure de ce travail est de prouver que pour la plupart des matrices, cette conditionest aussi nécessaire. Par ailleurs, lorsque le minimiseur du problème 1 est unique, sasensibilité aux mesures est étudiée et il est montré que l’application qui envoie les mesuressur ce minimiseur est Lipschitz-continue.© 2012 Published by Elsevier Masson SAS on behalf of Académie des sciences.1. IntroductionLet x0 ∈ RN be a vector and A be a real matrix with n rows and N columns. Let y0 be n linear measurements of x0, i.e.y0 = Ax0 ∈ Rn , where typically n < N . In this paper, we propose a necessary and sufficient condition ensuring that x0 can berecovered from y0 by solving the following optimization problem P1(y0)minx∈RN‖x‖1 such that y = Ax. P1(y)There is of course a huge literature on the subject, and covering it fairly is beyond the scope of this paper. We restrict ouroverview to those works pertaining to ours. For instance, our new condition can be seen as an extension of [6] and [7]. Thegeometric interpretations of results are related to the geometry of polytopes previously used by Donoho [2] and Donohoand Tanner [3] to study 1 minimization. Indeed, in [6], the following optimization problem (the so-called Lasso problem)minx∈RN12‖y − Ax‖22 + γ ‖x‖1 P1(y, γ )E-mail address: charles.dossal@math.u-bordeaux1.fr.1631-073X/$ – see front matter © 2012 Published by Elsevier Masson SAS on behalf of Académie des sciences.doi:10.1016/j.crma.2011.12.014118 C. Dossal / C. R. Acad. Sci. Paris, Ser. I 350 (2012) 117–120is studied. It is proved that any x0 belonging to the set F is the unique solution of P1(Ax0), whereF = {x such that rank(AI ) = |I| and ∀ j /∈ I,∣∣〈a j, AI(AtI AI)−1sign(xI )〉∣∣ < 1},where I is the support of x, AI the active matrix associated to x, whose columns are those of A indexed by I , xI are thenon-zero components of x and a j is the column of A indexed by j. In the sequel, Span(ai)i∈I will be denoted V I . Thesufficient condition developed in [6] plays a pivotal role in several papers which investigate support recovery in presence ofnoise by solving P1(y, γ ), e.g. [1,8,5] random vectors [1] or random measurements [5].In [7], the Exact Recovery Condition (ERC) is defined. This condition does not depend on the sign but only on the supportof x0 and provides results about the recovery of x0 and stability to noise.The goal of this Note is to show that the set of vectors that can be recovered by 1 minimization is exactly, for mostmatrices, F the closure of F . This result highlights the fact thatmaxj /∈I∣∣〈a j, AI(AtI AI)−1sign(xI )〉∣∣ and∥∥AI(AtI AI)−1sign(xI )∥∥2are good indicators of the identifiability, and brings arguments justifying the success of algorithms developed in [4] to findvery sparse but non-identifiable vectors.Before proceeding, let us fix some terminology and definitions. A vector x0 is said identifiable if and only if it is theunique solution of P1(Ax0).Definition 1.1. Let (xi)iN be N points of Rn . These points (xi)iN are said in general position (GP) if all affine subspacesof Rn which dimension k < n contain at most k + 1 points xi . A matrix A satisfies condition (GP) if for all sign vectorS ∈ {−1,1}N , points (S[i]ai)iN are in general position.It can be noticed that for any matrix A, the matrix A + E satisfies condition (GP) with probability 1 if E is any ran-dom perturbation with an absolutely continuous density with respect to the Lebesgue measure: with this definition mostmatrices satisfy condition (GP).2. ContributionsThe contributions of this paper are summarized as follows:Theorem 2.1. If x0 ∈ F , then x0 is identifiable.Theorem 2.2.(i) If x0 is identifiable, and if for all y in a neighborhood of y0 = Ax0 P1(y) has only one solution, then x0 ∈ F .(ii) If A satisfies (GP), for all y ∈ Im(A), P1(y) has a unique solution and x is identifiable if and only if x ∈ F .Theorem 2.3. If for all y ∈ Im(A) the solution of P1(y) is unique, the application φ associating y to this solution is Lipschitz.3. Preliminary lemmasThe two following lemmas can be found in [6].Lemma 3.1. A vector x∗ is a solution of P1(y, γ ) if and only ifAtI(y − Ax∗) = γ sign(x∗I)and ∀ j /∈ I ∣∣〈a j, y − Ax∗〉∣∣  1where the support of x∗ is denoted by I .Lemma 3.2. If for a vector x∗ the matrix AI satisfies rank(AI ) = |I| andAtI(y − Ax∗) = γ sign(x∗I)and ∀ j /∈ I, ∣∣〈a j, AI(AtI AI)−1sign(x∗I)〉∣∣ < 1then x∗ is the unique minimizer of P1(y, γ ).Lemma 3.3. If x0 is the unique solution of P1(Ax0) then rank(AI ) = |I|.Proof. If there exists h ∈ Ker(A) that is supported on I = I(x0), one has for all t ∈ R, A(x0 + th) = Ax0. Consequently theapplication t → ‖x0 + th‖1 is locally affine in a neighborhood of 0. It follows that there exists t = 0 such that ‖x0 + th‖1 ‖x0‖1. C. Dossal / C. R. Acad. Sci. Paris, Ser. I 350 (2012) 117–120 119Lemma 3.4. For all y ∈ Im(A) there exists a solution x to P1(y) such that rank(I) = |I|.Proof. Let x0 be a solution of P1(y). If rank(I) < |I|, there exists h ∈ Ker(A) that is supported on I . For a suitable t ,I(x0 + th)  I(x0) and x0 + th is a solution of P1(y). Lemma 3.5. If x0 ∈ F , then x0 is the unique solution of P1(Ax0).Proof. Lemma 3.2 shows that x0 is the unique solution of P1(AI (x0I + γ (AtI AI )−1sign(x0I )), γ ) if γ > 0 is small enough. Lemma 3.6. Let y ∈ Im(A) and (γn)n∈N a sequence of positive real numbers tending to 0. If (xn)n∈N are some solutions of P1(y, γn)such that limn→∞ xn = x0 one gets that x0 is a solution of P1(y).4. Proofs4.1. Proof of Theorem 2.1From Lemma 3.5 one has that vectors in F are identifiable. Since AI (AtI AI )−1sign(xI ) only depends on the sign and thesupport of x, F is a union of cones of various dimensions. It follows that the closure F of F is exactly equal to the set ofvectors x0 that can extended into a vector of F :F = {x0 such that ∃x1 such that I(x0) ∩ I(x1) = ∅ and x0 + x1 = x2 ∈ F }.Let us now suppose that x0 ∈ F , and that there exists x1 such that I(x0) ∩ I(x1) = ∅ and x2 = x0 + x1 belongs to F .Choose x3 ∈ RN such that Ax0 = Ax3 and define x4 := x3 + x1. We have Ax4 = Ax2 and since x2 ∈ F , x2 is the uniquesolution of P1(Ax2) which implies that ‖x2‖1 < ‖x4‖1. It follows that∥∥x2∥∥1 =∥∥x0∥∥1 +∥∥x1∥∥1 <∥∥x4∥∥1 ∥∥x3∥∥1 +∥∥x1∥∥1, (1)which implies ‖x0‖1 < ‖x3‖1. That is, x0 is the unique solution of P1(Ax0).4.2. Proof of Theorem 2.2Proof. Let y ∈ Im(A) and (γn)n∈N a sequence of positive real numbers tending to 0. From Lemma 3.4 we can always choosea solution x(γn) of P1(y, γ ) such that the associated active matrix AIn has a full rank. Since the number of possible supportsand signs is finite, up to an extraction of a subsequence, we can suppose that all x(γn) share the same support I and thesame sign S . From Lemma 3.1, we know that x(γn) satisfiesx(γn)I = x0I − γn(AtI AI)−1S and ∀ j /∈ I ∣∣〈a j, y − Ax(γn)〉∣∣  γ (2)where x0 is the vector supported on I such that x0I = (AtI AI )−1 AtI y. Using Lemma 3.6, x0 is a solution of P1(y). From (2)we deduce that I(x0) ⊂ I for γn small enough. It follows that y − Ax(γn) = γ AI (AtI AI )−1 S = γ dI,S and thus ∀ j /∈ I ,|〈a j,dI,S 〉|  1.Let us define J to be the set of all indices such that |〈a j,dI,S 〉| = 1. One can first notice that I ⊂ J .We now show that(i) either (a j) j∈ J are linearly dependent and one can build vectors x1 close to x0 such that P1(x1) has several solutions, inwhich case condition (GP) cannot be satisfied;(ii) or (a j) j∈ J are linearly independent and x(γn) ∈ F and x0 ∈ F and x0 is identifiable.(i) Suppose that (a j) j∈ J are linearly dependent and let ε > 0. Define K  J such that I ⊂ K and (ak)k∈K is a basis ofV J = Span(a j) j∈ J and j0 ∈ J ∩ K c . Define S K ∈ R|K | by S K = AtK dI,S . Then, S K is a sign vector and x1 the vector supportedon K defined byx1K = x0K +ε‖S K ‖2 S K .Now choose ε > 0 small enough to ensure that sign(x1I(x0)) = sign(x0I(x0)). From definition of x1, it follows that ‖x0 − x1‖2 = ε.For γ small enough the support and the sign of x1 and the vector x1(γ ) supported in K and defined by x1(γ )K = x1K −γ (AtK AK )−1 S K are identical.We shall prove that x1(γ ) is a solution of P1(Ax1, γ ) using Lemma 3.1. Indeed AtK (Ax1 − Ax1(γ )) = γ S K . The vector S =At dI,S is the common sign of vectors (x(γn))n∈N . Since limn→∞ x(γn) = x0 it follows that for all i ∈ I(x0) ⊂ I , sign(x0[i]) =I120 C. Dossal / C. R. Acad. Sci. Paris, Ser. I 350 (2012) 117–120S[i]. Since for all i ∈ I , S[i] = 〈ai,dI,S 〉 = S K [i] it follows that S K = sign(x1K ). Moreover sign(x1K ) = sign(x1(γ )K ) which yieldsthat AtK (Ax1 − Ax1(γ )) = γ sign(x1(γ )K ). Since dI,S ∈ V I ⊂ V K , one has dI,S = AK (AtK AK )−1 AtK dI,S = AK (AtK AK )−1 S K . Ifj /∈ K ,∣∣〈a j, Ax1 − Ax1(γ )〉∣∣ = γ ∣∣〈a j, AK(AtK AK)−1S K〉∣∣ = γ ∣∣〈a j,dI,S〉∣∣ < γ ,and by Lemma 3.1 it follows that x1(γ ) is a solution of P1(Ax1, γ ). Now Lemma 3.6 shows that x1 is a solution of P1(Ax1).Choose some h ∈ Ker(A) that is supported on K ∪{ j0} and such that h[ j0] = S[ j0] = atj0 dI,S . Denote hK the restriction of h onindices K . Since 0 = Ah = S[ j0]a j0 + AK hK and S K = AtK dI,S one has 〈sign(x1K ),hK 〉 = StK hk = dtI,S AK hK = −S[ j0]dtI,Sa j0 =−1. Moreover, for all t ∈ R, A(x1 + th) = Ax1 and for small non-negative t∥∥x1 + th∥∥1 =∥∥x1∥∥1 + t + t〈sign(x1),hK〉 = ∥∥x1∥∥1,which shows that x1 is not the unique minimizer of P1(Ax1).Notice that for each l ∈ K ∪ { j0}, S[l]al ∈ V K and S[l]al belongs to the affine hyperplan HdI,S = {u, such that〈u,dI,S 〉 = 1}. Hence at least |K | + 1 points S[l]al belong to an affine subspace V K ∩ HdI,S which dimension equals to|K | − 1. Therefore, (GP) is not satisfied.(ii) Let us now suppose that the (a j) j∈ J are linearly independent. Define S J = AtJ dI,S , ∀i ∈ I , S[i] = S J [i]. Let x1 be thevector supported on J such that for all i ∈ I(x0), x1[i] = x0[i] and for all j ∈ J ∩ I(x0)c , x1[ j] = S J [ j]. One has sign(x1J ) = S Jand since dI,S ∈ V I ⊂ V J , one has dI,S = A J (AtJ A J )−1 AtJ dI,S = A J (AtJ A J )−1 S J . It follows that for all l /∈ J ,∣∣〈al, A J(AtJ A J)−1S J〉∣∣ = ∣∣〈al,dI,S〉∣∣ < 1implying that x1 ∈ F and hence x0 ∈ F . 4.3. Proof of Theorem 2.3The proof relies on the following lemma.Lemma 4.1. For all y0 ∈ Im(A), there exists ε0 such that for all y ∈ Im(A) ∩ B(y0, ε0), one has I(φ(y0)) ⊂ I(φ(y)).Proof. To prove this lemma we will prove that for any y0 ∈ Im(A) and any sequence (yn)n∈N tending to y0, any subsequence(yun )n∈N such that I(φ(yun )) = Jun = J is constant, the support J satisfies J ⊃ I = I(φ(y0)).Denote by x0 = φ(y0) and I = I(x0). One has x0(I) = (AtI AI )−1 AtI y0 = A+I y0. Let (yn)n∈N be a sequence of elementsof Im(A) tending to y0 and xn = φ(yn). Up to an extraction of a subsequence, one can suppose that for all n ∈ N, I(xn) isconstant. Denote by J this common support. Then xnJ = (AtJ A J )−1 AtJ yn = A+J yn . Let x∞ = limn→∞ xn and x∞J = A+J y0. Let znbe the sequence of vectors supported on I defined by znI = A+I yn . By the definition of zn , Azn = P V I yn . Let K be a set suchthat (ak)k∈K is a basis of Im(A), and vn the vector supported on K such that vnK = A+K (P V I ⊥ (yn)). One has A(zn + vn) = ynand ‖zn + vn‖1  ‖zn‖1 + ‖vn‖1. Noticing that∥∥vn∥∥1 =∥∥A+K(P V I ⊥(yn − y0))∥∥1 −→n→∞ 0one deduces that limn→∞ ‖zn + vn‖1 = ‖x0‖1. Moreover limn→∞ ‖xn‖1 = ‖x∞‖1 and since ‖xn‖1  ‖zn + vn‖1 we deducethat ‖x∞‖1  ‖x0‖1 and finally x0 = x∞ since x0 = φ(y0). Thus one has I = I(x0) = I(x∞) ⊂ J which concludes the proof ofthe lemma. Let y0 ∈ Im(A). There exists ε0 > 0 such that for all y ∈ Im(A) ∩ B(y0, ε0), there exists J satisfying I ⊂ J . Moreoverx = φ(y) is supported in J and x J = A+J y. Since I ⊂ J , one also has x0J = A+J y0 and thus ‖x − x0‖2 = ‖A+J (y − y0)‖2. Onededuces that ∀y ∈ Im(A) ∩ B(y0, ε0), ‖φ(y0) − φ(y)‖2  C‖y0 − y‖2 with C = max J , rank(A J )=| J | λmax(A+J ) which concludesthe proof of the theorem. References[1] E.J. Candès, Y. Plan, Near ideal model selection by 1 minimization, Ann. Statist. 37 (2009) 2145–2177.[2] D. Donoho, Neighborly polytopes and sparse solution of underdetermined linear equations, Preprint, 2004.[3] D. Donoho, J. Tanner, Neighborliness of randomly-projected simplices in high dimensions, Preprint, 2005.[4] Ch. Dossal, G. Peyré, J. Fadili, A numerical exploration of compressed sampling recovery, Linear Algebra Appl. 432 (2010) 1663–1679.[5] Ch. Dossal, M.L. Chabanol, G. Peyré, J. Fadili, Sharp support recovery from noisy random measurements by 1 minimization, Appl. Comput. Harmon.Anal. (2011), doi:10.1016/j.acha.2011.09.003.[6] J.J. Fuchs, On sparse representations in arbitrary redundant bases, IEEE Trans. Inform. Theory 50 (2004) 1341–1344.[7] J.A. Tropp, Just relax: convex programming methods for identifying sparse signals in noise, IEEE Trans. Inform. Theory 52 (2006) 1030–1051.[8] Wainwright, Sharp thresholds for high-dimensional and noisy sparsity recovery using 1-constrained quadratic programming (lasso), IEEE Trans. Inform.Theory 55 (5) (2009) 2183–2202.

A necessary and sufficient condition for exact sparse recovery by $ {\ell }_{1}$ minimization

Dossal, Charles

Numérisation de Documents Anciens Mathématiques

https://comptes-rendus.academie-sciences.fr/mathematique/item/10.1016/j.crma.2011.12.014.pdf

A necessary and sufficient condition for exact sparse recovery by ${\ell }_{1}$ minimization

Abstract

Similar works

Full text

Available Versions

Comptes Rendus Mathématique

Numérisation de Documents Anciens Mathématiques

A necessary and sufficient condition for exact sparse recovery by ℓ1 {\ell }_{1}ℓ1​ minimization

Abstract

Similar works

Full text

Available Versions

Comptes Rendus Mathématique

Numérisation de Documents Anciens Mathématiques

A necessary and sufficient condition for exact sparse recovery by ${\ell }_{1}$ minimization