312 research outputs found
A performance analysis framework for SOCP algorithms in noisy compressed sensing
Solving under-determined systems of linear equations with sparse solutions
attracted enormous amount of attention in recent years, above all, due to work
of \cite{CRT,CanRomTao06,DonohoPol}. In \cite{CRT,CanRomTao06,DonohoPol} it was
rigorously shown for the first time that in a statistical and large dimensional
context a linear sparsity can be recovered from an under-determined system via
a simple polynomial -optimization algorithm. \cite{CanRomTao06} went
even further and established that in \emph{noisy} systems for any linear level
of under-determinedness there is again a linear sparsity that can be
\emph{approximately} recovered through an SOCP (second order cone programming)
noisy equivalent to . Moreover, the approximate solution is (in an
-norm sense) guaranteed to be no further from the sparse unknown vector
than a constant times the noise. In this paper we will also consider solving
\emph{noisy} linear systems and present an alternative statistical framework
that can be used for their analysis. To demonstrate how the framework works we
will show how one can use it to precisely characterize the approximation error
of a wide class of SOCP algorithms. We will also show that our theoretical
predictions are in a solid agrement with the results one can get through
numerical simulations.Comment: arXiv admin note: substantial text overlap with arXiv:1303.729
Upper-bounding -optimization weak thresholds
In our recent work \cite{StojnicCSetam09} we considered solving
under-determined systems of linear equations with sparse solutions. In a large
dimensional and statistical context we proved that if the number of equations
in the system is proportional to the length of the unknown vector then there is
a sparsity (number of non-zero elements of the unknown vector) also
proportional to the length of the unknown vector such that a polynomial
-optimization technique succeeds in solving the system. We provided
lower bounds on the proportionality constants that are in a solid numerical
agreement with what one can observe through numerical experiments. Here we
create a mechanism that can be used to derive the upper bounds on the
proportionality constants. Moreover, the upper bounds obtained through such a
mechanism match the lower bounds from \cite{StojnicCSetam09} and ultimately
make the latter ones optimal.Comment: arXiv admin note: text overlap with arXiv:0907.366
Negative spherical perceptron
In this paper we consider the classical spherical perceptron problem. This
problem and its variants have been studied in a great detail in a broad
literature ranging from statistical physics and neural networks to computer
science and pure geometry. Among the most well known results are those created
using the machinery of statistical physics in \cite{Gar88}. They typically
relate to various features ranging from the storage capacity to typical overlap
of the optimal configurations and the number of incorrectly stored patterns. In
\cite{SchTir02,SchTir03,TalBook} many of the predictions of the statistical
mechanics were rigorously shown to be correct. In our own work
\cite{StojnicGardGen13} we then presented an alternative way that can be used
to study the spherical perceptrons as well. Among other things we reaffirmed
many of the results obtained in \cite{SchTir02,SchTir03,TalBook} and thereby
confirmed many of the predictions established by the statistical mechanics.
Those mostly relate to spherical perceptrons with positive thresholds (which we
will typically refer to as the positive spherical perceptrons). In this paper
we go a step further and attack the negative counterpart, i.e. the perceptron
with negative thresholds. We present a mechanism that can be used to analyze
many features of such a model. As a concrete example, we specialize our results
for a particular feature, namely the storage capacity. The results we obtain
for the storage capacity seem to indicate that the negative case could be more
combinatorial in nature and as such a somewhat harder challenge than the
positive counterpart.Comment: arXiv admin note: substantial text overlap with arXiv:1306.3809,
arXiv:1306.397
Discrete perceptrons
Perceptrons have been known for a long time as a promising tool within the
neural networks theory. The analytical treatment for a special class of
perceptrons started in seminal work of Gardner \cite{Gar88}. Techniques
initially employed to characterize perceptrons relied on a statistical
mechanics approach. Many of such predictions obtained in \cite{Gar88} (and in a
follow-up \cite{GarDer88}) were later on established rigorously as mathematical
facts (see, e.g.
\cite{SchTir02,SchTir03,TalBook,StojnicGardGen13,StojnicGardSphNeg13,StojnicGardSphErr13}).
These typically related to spherical perceptrons. A lot of work has been done
related to various other types of perceptrons. Among the most challenging ones
are what we will refer to as the discrete perceptrons. An introductory
statistical mechanics treatment of such perceptrons was given in
\cite{GutSte90}. Relying on results of \cite{Gar88}, \cite{GutSte90}
characterized many of the features of several types of discrete perceptrons. We
in this paper, consider a similar subclass of discrete perceptrons and provide
a mathematically rigorous set of results related to their performance. As it
will turn out, many of the statistical mechanics predictions obtained for
discrete predictions will in fact appear as mathematically provable bounds.
This will in a way emulate a similar type of behavior we observed in
\cite{StojnicGardGen13,StojnicGardSphNeg13,StojnicGardSphErr13} when studying
spherical perceptrons.Comment: arXiv admin note: substantial text overlap with arXiv:1306.3809,
arXiv:1306.3980, arXiv:1306.397
Block-length dependent thresholds in block-sparse compressed sensing
One of the most basic problems in compressed sensing is solving an
under-determined system of linear equations. Although this problem seems rather
hard certain -optimization algorithm appears to be very successful in
solving it. The recent work of \cite{CRT,DonohoPol} rigorously proved (in a
large dimensional and statistical context) that if the number of equations
(measurements in the compressed sensing terminology) in the system is
proportional to the length of the unknown vector then there is a sparsity
(number of non-zero elements of the unknown vector) also proportional to the
length of the unknown vector such that -optimization algorithm succeeds
in solving the system. In more recent papers
\cite{StojnicICASSP09block,StojnicJSTSP09} we considered the setup of the
so-called \textbf{block}-sparse unknown vectors. In a large dimensional and
statistical context, we determined sharp lower bounds on the values of
allowable sparsity for any given number (proportional to the length of the
unknown vector) of equations such that an -optimization
algorithm succeeds in solving the system. The results established in
\cite{StojnicICASSP09block,StojnicJSTSP09} assumed a fairly large block-length
of the block-sparse vectors. In this paper we consider the block-length to be a
parameter of the system. Consequently, we then establish sharp lower bounds on
the values of the allowable block-sparsity as functions of the block-length
Spherical perceptron as a storage memory with limited errors
It has been known for a long time that the classical spherical perceptrons
can be used as storage memories. Seminal work of Gardner, \cite{Gar88}, started
an analytical study of perceptrons storage abilities. Many of the Gardner's
predictions obtained through statistical mechanics tools have been rigorously
justified. Among the most important ones are of course the storage capacities.
The first rigorous confirmations were obtained in \cite{SchTir02,SchTir03} for
the storage capacity of the so-called positive spherical perceptron. These were
later reestablished in \cite{TalBook} and a bit more recently in
\cite{StojnicGardGen13}. In this paper we consider a variant of the spherical
perceptron that operates as a storage memory but allows for a certain fraction
of errors. In Gardner's original work the statistical mechanics predictions in
this directions were presented sa well. Here, through a mathematically rigorous
analysis, we confirm that the Gardner's predictions in this direction are in
fact provable upper bounds on the true values of the storage capacity.
Moreover, we then present a mechanism that can be used to lower these bounds.
Numerical results that we present indicate that the Garnder's storage capacity
predictions may, in a fairly wide range of parameters, be not that far away
from the true values
Under-determined linear systems and -optimization thresholds
Recent studies of under-determined linear systems of equations with sparse
solutions showed a great practical and theoretical efficiency of a particular
technique called -optimization. Seminal works \cite{CRT,DOnoho06CS}
rigorously confirmed it for the first time. Namely, \cite{CRT,DOnoho06CS}
showed, in a statistical context, that technique can recover sparse
solutions of under-determined systems even when the sparsity is linearly
proportional to the dimension of the system. A followup \cite{DonohoPol} then
precisely characterized such a linearity through a geometric approach and a
series of work\cite{StojnicCSetam09,StojnicUpper10,StojnicEquiv10} reaffirmed
statements of \cite{DonohoPol} through a purely probabilistic approach. A
theoretically interesting alternative to is a more general version
called (with an essentially arbitrary ). While is
typically considered as a first available convex relaxation of sparsity norm
, , albeit non-convex, should technically be a
tighter relaxation of . Even though developing polynomial (or close to
be polynomial) algorithms for non-convex problems is still in its initial
phases one may wonder what would be the limits of an ,
relaxation even if at some point one can develop algorithms that could handle
its non-convexity. A collection of answers to this and a few realted questions
is precisely what we present in this paper
Meshes that trap random subspaces
In our recent work \cite{StojnicCSetam09,StojnicUpper10} we considered
solving under-determined systems of linear equations with sparse solutions. In
a large dimensional and statistical context we proved results related to
performance of a polynomial -optimization technique when used for
solving such systems. As one of the tools we used a probabilistic result of
Gordon \cite{Gordon88}. In this paper we revisit this classic result in its
core form and show how it can be reused to in a sense prove its own optimality
Lifting -optimization thresholds
In this paper we look at a connection between the ,
optimization and under-determined linear systems of equations with sparse
solutions. The case , or in other words optimization and its a
connection with linear systems has been thoroughly studied in last several
decades; in fact, especially so during the last decade after the seminal works
\cite{CRT,DOnoho06CS} appeared. While current understanding of
optimization-linear systems connection is fairly known, much less so is the
case with a general , optimization. In our recent work
\cite{StojnicLqThrBnds10} we provided a study in this direction. As a result we
were able to obtain a collection of lower bounds on various , optimization thresholds. In this paper, we provide a substantial conceptual
improvement of the methodology presented in \cite{StojnicLqThrBnds10}.
Moreover, the practical results in terms of achievable thresholds are also
encouraging. As is usually the case with these and similar problems, the
methodology we developed emphasizes their a combinatorial nature and attempts
to somehow handle it. Although our results' main contributions should be on a
conceptual level, they already give a very strong suggestion that
optimization can in fact provide a better performance than , a fact
long believed to be true due to a tighter optimization relaxation it provides
to the original sparsity finding oriented original problem
formulation. As such, they in a way give a solid boost to further exploration
of the design of the algorithms that would be able to handle ,
optimization in a reasonable (if not polynomial) time.Comment: arXiv admin note: substantial text overlap with arXiv:1306.3774,
arXiv:1306.377
Optimality of -optimization block-length dependent thresholds
The recent work of \cite{CRT,DonohoPol} rigorously proved (in a large
dimensional and statistical context) that if the number of equations
(measurements in the compressed sensing terminology) in the system is
proportional to the length of the unknown vector then there is a sparsity
(number of non-zero elements of the unknown vector) also proportional to the
length of the unknown vector such that -optimization algorithm succeeds
in solving the system. In more recent papers
\cite{StojnicCSetamBlock09,StojnicICASSP09block,StojnicJSTSP09} we considered
under-determined systems with the so-called \textbf{block}-sparse solutions. In
a large dimensional and statistical context in \cite{StojnicCSetamBlock09} we
determined lower bounds on the values of allowable sparsity for any given
number (proportional to the length of the unknown vector) of equations such
that an -optimization algorithm succeeds in solving the system.
These lower bounds happened to be in a solid numerical agreement with what one
can observe through numerical experiments. Here we derive the corresponding
upper bounds. Moreover, the upper bounds that we obtain in this paper match the
lower bounds from \cite{StojnicCSetamBlock09} and ultimately make them optimal.Comment: arXiv admin note: substantial text overlap with arXiv:1303.7289, and
text overlap with arXiv:0907.367
- β¦