We consider the deterministic construction of a measurement

matrix and a recovery method for signals that are block

sparse. A signal that has dimension N = nd, which consists

of n blocks of size d, is called (s, d)-block sparse if

only s blocks out of n are nonzero. We construct an explicit

linear mapping Φ that maps the (s, d)-block sparse signal

to a measurement vector of dimension M, where s•d <N(1-(1-M/N)^(d/(d+1))-o(1).





We show that if the (s, d)-

block sparse signal is chosen uniformly at random then the

signal can almost surely be reconstructed from the measurement

vector in O(N^3) computations

Hassibi, Babak

Parvaresh, Farzad

English

Farzad Parvaresh

Babak Hassibi

Crossref

Explicit measurements with almost optimal thresholds for compressed sensing

Caltech Authors - Main

EXPLICIT MEASUREMENTS WITH ALMOST
OPTIMAL THRESHOLDS FOR COMPRESSED SENSING
Farzad Parvaresh ∗ and Babak Hassibi †
∗ Center for Mathematics of Information † Department of Electrical Engineering
California Institute of Technology,
1200 East California, Pasadena, CA, 91125, USA.
E-mail: {fparvaresh, hassibi}@caltech.edu
ABSTRACT
We consider the deterministic construction of a measure-
ment matrix and a recovery method for signals that are block
sparse. A signal that has dimension N = nd, which con-
sists of n blocks of size d, is called (s, d)-block sparse if
only s blocks out of n are nonzero. We construct an ex-
plicit linear mapping Φ that maps the (s, d)-block sparse sig-
nal to a measurement vector of dimension M , where s · d <
N
(
1− (1− MN
) d
d+1
)
− o(1). We show that if the (s, d)-
block sparse signal is chosen uniformly at random then the
signal can almost surely be reconstructed from the measure-
ment vector in O(N3) computations.
Index Terms— Convex optimization, sparse signals,
Reed-Solomon codes, decoding algorithms, compressed
sensing.
1. INTRODUCTION
Consider the set of signals of dimension N with at most s
nonzero element over CN . This set of signals spans the union
of
(
N
s
)
subspaces of dimension s overCN . If we project these
subspaces to a random subspace of dimension s+1, then with
high probability we get a one to one mapping between the
projected sparse signals and the original ones. The recent re-
sults of Cande´s, Donoho, Romberg, and Tao [2, 3, 5], applied
to applications such as tomography and digital photography,
have revealed the power of random sampling. Recently, many
other applications for compressed sensing have been devel-
oped in areas such as data mining, DNA microarrays, and A/D
converters.
Let ΦM,N denotes the linear measurement matrix, so that
the samples or the measurements of a sparse signal x∈CN
become y = Φ · x∈CM ,M  s + 1. To reconstruct the
This work was supported in parts by the National Science Foundation
under grants no. CCR-0133818 and CCR-0326554, by the David and Lucille
Packard Foundation, and by Caltechs Lee Center for Advanced Networking.
signal x from the measurement vector y one needs to solve
the underdetermined linear system of equations Φx = y, for
a given y, under the condition that x is a s-sparse signal. This
can be represented as the following optimization problem:
min
x
‖x‖0 subject to Φx = y (1)
Here the 0 norm or the Hamming norm is the number of
nonzero elements of x.
A naive exhaustive search checks all the possible
(
N
s
)
nonzero coordinates for the signal x to ﬁnd the minimum and
that takes an exponential time in N . However, one may try to
solve (1) by relaxing the 0 norm to 1 norm.
min
x
‖x‖1 subject to Φx = y (2)
Assume δ is equal to M/N and ρ is s/M and the measure-
ment matrix Φ is chosen uniformly at random from the set
of linear projections from CN to CM . Donoho and Tan-
ner [5, 11] determined the region (δ, ρ) for which the 1 opti-
mization and 0 coincide. They compute two different types
of “strong”, ρS(δ), and “weak”, ρW (δ), threshold functions.
The strong threshold function ensures that 1 and 0 are equiv-
alent for s < M ρS(M/N) with overwhelming probability in
the uniform selection of measurement matrix Φ. For the weak
threshold the equivalence between (1) and (2) holds for most
signals x when s < M ρW (M/N) with overwhelming prob-
ability in uniform selection of Φ. (cf. Figure 1)
How much do we pay by relaxing the 0 optimization to
1? Let’s deﬁne ρoptS (δ) and ρ
opt
W (δ) to be the supremum of all
the threshold functions over all the linear measurements. We
know that ρoptS = 1/2, ρ
opt
W = 1. There is a large gap between
the storing and week thresholds ρS(W )(δ) and ρ
opt
S(W )(δ).
How do we choose the measurement matrix Φ? In most of
the literature in compressed sampling the measurement ma-
trix is an instance of a class of random matrices. Then, with
overwhelming probability, Φ satisﬁes certain reconstruction
38531-4244-1484-9/08/$25.00 ©2008 IEEE ICASSP 2008
Authorized licensed use limited to: CALIFORNIA INSTITUTE OF TECHNOLOGY. Downloaded on April 14,2010 at 21:34:55 UTC from IEEE Xplore.  Restrictions apply. 
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
S
ρ
W
ρoptS
ρoptW
δ = M/N
ρ=
 s/
M d=1
d=2
ρ
d=6
Fig. 1. Thresholds for the recovery of sparse signals. ρS is
the strong threshold and ρW is the weak threshold for linear
program. The rest are thresholds for recovery of (s, d)-block
sparse signals. Using our proposed method the improvement
over ρW is clear. As d grows, our threshold approaches ρ
opt
W .
properties. However, there is no efﬁcient method for verify-
ing that a given matrix has these properties [12]. Recently,
a line of research on compressed sensing has been devoted to
the explicit construction of the measurement matrix, however,
the threshold functions of these explicit constructions are usu-
ally worst than those of 1 optimization.
1.1. Contributions
A connection between compressed sensing and Reed-Solomon
codes over the complex ﬁled is already implicit in various
works in the literature, often under different names such as
annihilator ﬁlters and recovery of a measure from its mo-
ments [1, 7]. In this paper, we make this connection explicit
by choosing the measurement vector to be essentially the syn-
drome of the code. The sparse signal can then be recovered
by any well known decoding algorithm such as Berlekamp-
Massey for RS codes.
However, recently, there have been many remarkable
breakthroughs in list-decoding of Reed-Solomon codes such
as Sudan [10] and Guruswami-Sudan [6] algorithms; To the
best of our knowledge we are not aware of any research on
using these classes of algorithms for compressed sampling.
In a Reed-Solomon code of length N and dimension K, the
Berlekamp-Massey decoder only needs the syndrome vector
of dimension N−K to ﬁnd the error locater polynomial.
The dimension of the syndrome vector is smaller than the
dimension of the received word; that is equivalent to having a
measurement vector with smaller dimension than the sparse
signal in compressed sampling. However, in the list-decoding
algorithms the whole received word is being used, and not
the syndromes, to perform the decoding. One contribution of
ours is to show that one can construct a “received word” out
of the syndrome vector to perform the list-decoding algorithm
for compressed sensing applications.
One of the crucial steps of all the Sudan-type list-decoding
algorithms is to factor a bivariate polynomial over the under-
lying ﬁeld. This factorization can be done efﬁciently over
ﬁnite ﬁelds but we are not aware of any efﬁcient algorithm
for factoring a bivariate polynomial over the complex ﬁeld.
Instead of list-decoding, we propose to use Coppersmith and
Sudan [4] decoding. This algorithm is probabilistic and de-
codes with probability of 1−O(N c/q) where, c is a constant,
q is the size of the ﬁnite ﬁeld, and N is the length of the code;
considering that errors are generated uniformly at random
in Fq. However, we show that over the complex ﬁeld, the
Coppersmith-Sudan algorithm will almost surely recover the
random sparse signal. In computer science, Reed-Solomon
codes have mostly been used at rates that approaches zero
and the authors in [4] basically give decoding bounds that are
suitable for these rates. We use more advance tools from al-
gebra, such as working with the Gro¨bner basis of certain ideal
of polynomials and we improve the decoding bound of [4].
The new bound shows improvement compare to conventional
decoding algorithms for all rates in [0, 1].
In addition, the Coppersmith-Sudan algorithm can be
used to recover curves in three and more dimensions. That
is tantamount to the possibility of recovering block sparse
signals with a small number of measurements. Consider
a signal of dimension N which consists of n blocks of
size d = N/n. We say the signal is (s, d)-block sparse
if only s blocks out of n is nonzero. We show that using
syndrome measurements one can almost surely recover an
(s, d)-block sparse signal from M measurements efﬁciently
if s · d < N
(
1− (1− MN
) d
d+1
)
− o(1) (Check Figure 1 for
the plot of thresholds).
2. THE MEASUREMENT MATRIX
Let C denote the complex ﬁeld. We use C[X1, X2, · · · , Xc]
to denote the rings of polynomials overC in several variables.
Reed-Solomon codes are obtained by evaluation of certain
function in C[X] in a set of points D = {ω0, ω1, · · · , ωN−1}
in C. Throughout this work we choose ωi = λi for i =
0, 1, · · · , N − 1, where λ is the N -th root of unity. 1 A Reed-
Solomon code RS (N,K) of length N and dimension K over
the complex ﬁeld is deﬁned as follows:
RS (N,K) def= {(f(ω0), f(ω1), · · · , f(ωN−1)) :
f(X)∈C[X],deg f < K}
Notice that RS (N,K) is a subspace of dimension K in
C
N . Deﬁne Synd (N,K) to be the orthogonal space to
1In the general case ωi’s can be any set of different numbers in C. We
choose them to be on the unit circle with equal distance so that the measure-
ment becomes the inverse Fourier transform.
3854
Authorized licensed use limited to: CALIFORNIA INSTITUTE OF TECHNOLOGY. Downloaded on April 14,2010 at 21:34:55 UTC from IEEE Xplore.  Restrictions apply. 
RS (N,K).
Synd (N,K) def=
{
(v1, v2, · · · , vN )∈CN :
〈v, c〉 = 0, for all c∈RS (N,K)}
We call Synd (N,K) the syndrome or the measurement
space.
Deﬁnition 1 Set K = N −M and consider the correspond-
ing linear projection Φ from CN to Synd (N,K). We deﬁne
the M -dimensional measurement of x to be
y = Φ · x. (3)
Lemma 1 Assume that the evaluation points of the RS code
are the consecutive powers of the N -th root of unity, i.e. ωi =
λi for i = 0, 1, · · · , N−1, then the measurement vector y
in (3) is the inverse Fourier transform of x at frequencies
ωK , ωK+1, · · · , ωN−1.
Lemma 2 Any vector x∈CN can be written uniquely as a
summation of vectors r∈Synd (N,K) and c∈RS (N,K):
y = Φx ∈CN−K
r = Φ†y ∈Synd (N,K)
c = x− r ∈RS (N,K)
(4)
where Φ† is the conjugate transpose of Φ.
For a given measurement vector y of the s-sparse signal x
we construct the “received vector” r = Φ†y. Now, from Lem-
ma 2, we know that r = x + c for some c∈RS (N,K). That
means, r is simply a RS codeword c that has been corrupted
at s positions. Thus, for example, if we use the Berlekamp-
Massey algorithm to decode r, as far as the number of cor-
rupted coordinates is smaller than half the minimum distance
of the code s < (N − K)/2 = M/2, the decoder outputs
the codeword c and sparse signal x. In the next section, we
explore the possibility of using other RS decoding algorithms
for compressed sensing.
3. RECOVERY FROM THE MEASUREMENTS
Now that we have established a connection between the re-
covery of the sparse signal x from the measurement vector
y = Φx and the RS decoding of r = Φ†y, we can use other
advanced decoding algorithms such as the list-decoding al-
gorithm of Guruswami-Sudan [6] for recovery. The bottle-
neck of the Guruswami-Sudan algorithm over complex ﬁelds
is the factorization part. We are not aware of any efﬁcient
factorization algorithm over the complex ﬁeld. Considering
the fact that there are many efﬁcient algorithms to factor uni-
variate polynomials over the complex ﬁeld, one can use the
Ruth-Ruckenstein [9] algorithm. However, the algorithm, in
principle, is sensitive to numerical inaccuracies.
Another elegant decoding algorithm with bounds compa-
rable to the list-decoding algorithm of GS was introduced by
Coppersmith and Sudan [4]. Their algorithm does not rely on
tools such as the factoring of multivariate polynomials. Basi-
cally, given a received word, they construct a matrix A such
that the right kernel of A with high probability consists of
vectors with support that is entirely on the “non-error” coor-
dinates of the received vector.
We show that, over the complex ﬁeld their algorithm al-
most surly recovers the codeword if the sparse signal is cho-
sen uniformly at random over CN , we further improve the
bounds [4] and show that the performance of the algorithm is
comparable to the list-decoding algorithm of GS at all rates in
[0, 1].
Notations and deﬁnitions. Given Δ, let MK,Δ be the set of
monomials XaY b with a + (K − 1)b  Δ. For a positive
integer p, let Sp = {(d, e) : d + e < p}. Given (d, e)∈Sp,
let f [d,e]α,β;M be the vector in C
|M| whose coordinates are in-
dexed by monomials M in M and whose M th coordinate is
∂d+e
∂Xd∂Y e
XaY b|(α,β) if M = XaY b.
Algorithm 1 Coppersmith-Sudan decoding algorithm
Input: Received vector r∈CN , multiplicity p, and codeword
dimension K.
Output: Codeword c∈CN or FAIL.
1: Parameters: Set Δ sufﬁciently large such that
|MK,Δ|  N · |Sp|.
2: Step 1: Let A be the matrix whose columns are in-
dexed by pairs (i, (d, e)) with i∈{0, 1, · · · , N − 1}
and(d, e)∈Sp where the (i, (e, d))th column is f [d,e]ωi,ri;M.
Let b be a non-zero vector such that A · b = 0.
3: Step 2: Let J be the set of all indices i∈{0, 1, · · · , N −
1} such that there exists a tuple (d, e)∈Sp for which the
(i, (d, e))th coordinate of b is nonzero.
4: if there exists a polynomial f(X) with deg f(X) < K
such that f(ωi) = ri for every i∈J then
5: return c = (f(ω0), f(ω1), · · · , f(ωN−1)).
6: else
7: return FAIL.
8: end if
4. ANALYSIS OF THE ALGORITHM
Due to lack of space we omit the proofs. For details, the
reader is referred to [8]. Let I denote the set of non-error posi-
tions of r and t = |I|. Let f(X) be the corresponding polyno-
mial of the RS codeword c. We prove the following Lemmas.
First, the matrix A does have a rank less than N · |Sp| and
thus a vector b as required in Step 1 does exist. Second, with
high probability the subset J found in Step 2 is a subset of
I . Third, the size of J is at least K and so there is at most
one polynomial f(X) of degree less than K that interpolates
3855
Authorized licensed use limited to: CALIFORNIA INSTITUTE OF TECHNOLOGY. Downloaded on April 14,2010 at 21:34:55 UTC from IEEE Xplore.  Restrictions apply. 
1 2 3 4 5 6 7 8 9 10 11
10−5
10−4
10−3
10−2
10−1
100
101
Sparsity
|x −
 x h
at
| 2
σ
σ
σ
σ
σ
= 10^−4
= 10^−3
= 10^−2
= 10^−1
= 10^−5
s
^
|x −
 x|
Fig. 2. Simulation results at different noise levels. N = 50,
M = 25,  is for the Berlekamp-Massey, × is for the
Coppersmith-Sudan, O is for the Lasso.
through points of J .
Let B be the |M| × (|Sp| · t) matrix consisting of those
columns of A that correspond to i∈ I .
Lemma 3 (i) If t > Δ/p, then the matrix B has column
dependency. (ii) There are no column dependencies in B
involving fewer that H blocks of columns, provided Δ >
pH + p(p + 1)/2 . (iii) Almost surely, the matrix A has no
linear dependencies involving any of the columns indexed by
(i, (e, d)) where i /∈ I , provided |MΔ−(2p+1)| > N · |Sp| .
Theorem 1 For every ﬁxed constant d, using the syndrome
measurement matrix with the Coppersmith-Sudan decoding
algorithm we can almost surely recover (s, d)-block sparse
signals from M = δN measurements if
S < M
1− (1− δ) dd+1
δ
− o(1) (5)
where S = s · d is the number of nonzero elements of the
sparse signal, and N is size of the sparse signal.
Remark. When d = 1, then (5) reduces to S < M(1 −√
1− δ)/δ, which is greater than 1/2 for all δ ∈ [0, 1].
5. ROBUSTNESS TO NOISE
In practice the measurement vector is usually corrupted by
noise. Let N (0, σ) be a complex Gaussian random variable
with zero mean and standard deviation σ. We assume that
yw = Φ · x + w (6)
where w∈NM (0, σ). We choose x at random, i.e. the sup-
port of x is chosen uniformly at random from all the possible
(
N
s
)
subsets and the values are drawn i.i.d. formN (0, 1). Due
to the noise, the matrix A is full rank, so in Algorithm 1 we
choose b to be the right singular vector with the smallest sin-
gular value. Figure 2 shows the median of the squared error
‖x−xˆ‖2 as a function of the sparsity s. From the Figure 2, Al-
gorithm 1 is more robust to noise than the BM algorithm. We
also compare the performance to the well known LASSO al-
gorithm [13] which minimizes ‖x−xˆ‖2 with an 1 constraint.
6. REFERENCES
[1] M.Akc¸akaya and V. Tarokh, Performance Bounds on Sparse
Representations Using Redundant Frames, arXiv:cs/0703045.
[2] E. Cande´s, J. Romberg, and T. Tao, Robust uncertainty princ-
ples: Exact signal reconstruction from highly incomplete fre-
quency information, IEEE Trans. on Inform. Theory, 52,
pp. 489–509, 2006.
[3] E. Cande´s and T. Tao, Near optimal singal recovery from ran-
dom projections: Universal encoding strategies, IEEE Trans.
on Inform. Theory, 2006.
[4] D. Coppersmith and M. Sudan, Reconstructing curves in three
(and higher) dimensional spaces from noisy data, Proc. 35 th
STOC , pp. 136–142, San Diego, CA., June 2003.
[5] D. L. Donoho, High-dimensional centrally-symmetric poly-
topes with neighborliness proportional to dimension, Disc.
Comput. Geometry, December 2005.
[6] V.Guruswami and M. Sudan, Improved decoding of Reed-
Solomon and algebraic-geometric codes, IEEE Trans. on In-
form. Theory, 45, pp. 1755–1764, Septebmer 1999.
[7] I.Maravic´ and M.Vetterli, Sampling and Reconstruction of
Signals With Finite Rate of Innovation in the Presence of
Noise, IEEE Trans. On Signal Proc., 53(8), pp. 2788–2805,
August 2005.
[8] F. Parvaresh and B.Hassibi, Compressed sensing with almost
optimal thresholds, in preperation.
[9] R.M.Roth and G. Ruckenstein, Efﬁcient decoding of Reed-
Solomon codes beyond half the minimum distance, IEEE
Trans. on Inform. Theory, 46, pp. 246–258, January 2000.
[10] M. Sudan, Decoding of Reed-Solomon codes beyond the
error-correction bound, Journal of Complexity, 13(1),
pp. 180–193, 1997.
[11] J. Tanner and DDohono, Thresholds for the Recovery of
Sparse Solutions via L1 Minimization, Proc. of the Conf. on
Inform. Sci. and Sys., March 2006.
[12] T. Tao, Open question: deterministic UUP matrices,
Weblog at http://terrytao.wordress.com/2007/07/02/open-
question-deterministic-uup-matrices 2007.
[13] R. Tibshirani, Regression shrinkage and selection via the lasso,
Journal of the Royal Stat. soc. B, 58(1), pp. 267–288, 1996.
3856
Authorized licensed use limited to: CALIFORNIA INSTITUTE OF TECHNOLOGY. Downloaded on April 14,2010 at 21:34:55 UTC from IEEE Xplore.  Restrictions apply. 


and T.Tao, Robust uncertainty princples: Exact signal reconstruction from highly incomplete frequency information,

Authorized licensed use limited to:

B.Hassibi, Compressed sensing with almost optimal thresholds, in preperation.

Compressed sensing with almost optimal thresholds, in preperation.

Decoding of Reed-Solomon codes beyond the error-correction bound,

E.Cand´ es and T.Tao, Near optimal singal recovery from random projections: Universal encoding strategies,

Efficient decoding of ReedSolomon codes beyond half the minimum distance,

G.Ruckenstein, Efﬁcient decoding of ReedSolomon codes beyond half the minimum distance,

High-dimensional centrally-symmetric polytopes with neighborliness proportional to dimension,

I.Maravi´ c and M.Vetterli, Sampling and Reconstruction of Signals With Finite Rate of Innovation in the Presence of Noise,

Improved decoding of ReedSolomon and algebraic-geometric codes,

M.Akc ¸akaya and V.Tarokh, Performance Bounds on Sparse Representations Using Redundant Frames,

M.Sudan, Improved decoding of ReedSolomon and algebraic-geometric codes,

Near optimal singal recovery from random projections: Universal encoding strategies,

Open question: deterministic UUP matrices, Weblog at http://terrytao.wordress.com/2007/07/02/openquestion-deterministic-uup-matrices

Performance Bounds on Sparse Representations Using Redundant Frames,

question: deterministic UUP matrices, Weblog at http://terrytao.wordress.com/2007/07/02/openquestion-deterministic-uup-matrices

Reconstructing curves in three (and higher) dimensional spaces from noisy data,

Regression shrinkage and selection via the lasso,

Robust uncertainty princples: Exact signal reconstruction from highly incomplete frequency information,

Sampling and Reconstruction of Signals With Finite Rate of Innovation in the Presence of Noise,

Thresholds for the Recovery of Sparse Solutions via L1 Minimization,

https://authors.library.caltech.edu/19150/1/Parvaresh2008p87442009_Ieee_International_Conference_On_Acoustics_Speech_And_Signal_Processing_Vols_1-_8_Proceedings.pdf

Explicit measurements with almost optimal thresholds for compressed sensing

Abstract

Similar works

Full text

Available Versions

Crossref

Caltech Authors - Main

Caltech Authors - Main