201 research outputs found
The dynamics of message passing on dense graphs, with applications to compressed sensing
Approximate message passing algorithms proved to be extremely effective in
reconstructing sparse signals from a small number of incoherent linear
measurements. Extensive numerical experiments further showed that their
dynamics is accurately tracked by a simple one-dimensional iteration termed
state evolution. In this paper we provide the first rigorous foundation to
state evolution. We prove that indeed it holds asymptotically in the large
system limit for sensing matrices with independent and identically distributed
gaussian entries.
While our focus is on message passing algorithms for compressed sensing, the
analysis extends beyond this setting, to a general class of algorithms on dense
graphs. In this context, state evolution plays the role that density evolution
has for sparse graphs.
The proof technique is fundamentally different from the standard approach to
density evolution, in that it copes with large number of short loops in the
underlying factor graph. It relies instead on a conditioning technique recently
developed by Erwin Bolthausen in the context of spin glass theory.Comment: 41 page
On Low-rank Trace Regression under General Sampling Distribution
A growing number of modern statistical learning problems involve estimating a
large number of parameters from a (smaller) number of noisy observations. In a
subset of these problems (matrix completion, matrix compressed sensing, and
multi-task learning) the unknown parameters form a high-dimensional matrix B*,
and two popular approaches for the estimation are convex relaxation of
rank-penalized regression or non-convex optimization. It is also known that
these estimators satisfy near optimal error bounds under assumptions on rank,
coherence, or spikiness of the unknown matrix.
In this paper, we introduce a unifying technique for analyzing all of these
problems via both estimators that leads to short proofs for the existing
results as well as new results. Specifically, first we introduce a general
notion of spikiness for B* and consider a general family of estimators and
prove non-asymptotic error bounds for the their estimation error. Our approach
relies on a generic recipe to prove restricted strong convexity for the
sampling operator of the trace regression. Second, and most notably, we prove
similar error bounds when the regularization parameter is chosen via K-fold
cross-validation. This result is significant in that existing theory on
cross-validated estimators do not apply to our setting since our estimators are
not known to satisfy their required notion of stability. Third, we study
applications of our general results to four subproblems of (1) matrix
completion, (2) multi-task learning, (3) compressed sensing with Gaussian
ensembles, and (4) compressed sensing with factored measurements. For (1), (3),
and (4) we recover matching error bounds as those found in the literature, and
for (2) we obtain (to the best of our knowledge) the first such error bound. We
also demonstrate how our frameworks applies to the exact recovery problem in
(3) and (4).Comment: 32 pages, 1 figur
Combinatorial approach to the interpolation method and scaling limits in sparse random graphs
We establish the existence of free energy limits for several combinatorial
models on Erd\"{o}s-R\'{e}nyi graph and
random -regular graph . For a variety of models, including
independent sets, MAX-CUT, coloring and K-SAT, we prove that the free energy
both at a positive and zero temperature, appropriately rescaled, converges to a
limit as the size of the underlying graph diverges to infinity. In the zero
temperature case, this is interpreted as the existence of the scaling limit for
the corresponding combinatorial optimization problem. For example, as a special
case we prove that the size of a largest independent set in these graphs,
normalized by the number of nodes converges to a limit w.h.p. This resolves an
open problem which was proposed by Aldous (Some open problems) as one of his
six favorite open problems. It was also mentioned as an open problem in several
other places: Conjecture 2.20 in Wormald [In Surveys in Combinatorics, 1999
(Canterbury) (1999) 239-298 Cambridge Univ. Press]; Bollob\'{a}s and Riordan
[Random Structures Algorithms 39 (2011) 1-38]; Janson and Thomason [Combin.
Probab. Comput. 17 (2008) 259-264] and Aldous and Steele [In Probability on
Discrete Structures (2004) 1-72 Springer].Comment: Published in at http://dx.doi.org/10.1214/12-AOP816 the Annals of
Probability (http://www.imstat.org/aop/) by the Institute of Mathematical
Statistics (http://www.imstat.org
Maximum Weight Matching via Max-Product Belief Propagation
Max-product "belief propagation" is an iterative, local, message-passing
algorithm for finding the maximum a posteriori (MAP) assignment of a discrete
probability distribution specified by a graphical model. Despite the
spectacular success of the algorithm in many application areas such as
iterative decoding, computer vision and combinatorial optimization which
involve graphs with many cycles, theoretical results about both correctness and
convergence of the algorithm are known in few cases (Weiss-Freeman Wainwright,
Yeddidia-Weiss-Freeman, Richardson-Urbanke}.
In this paper we consider the problem of finding the Maximum Weight Matching
(MWM) in a weighted complete bipartite graph. We define a probability
distribution on the bipartite graph whose MAP assignment corresponds to the
MWM. We use the max-product algorithm for finding the MAP of this distribution
or equivalently, the MWM on the bipartite graph. Even though the underlying
bipartite graph has many short cycles, we find that surprisingly, the
max-product algorithm always converges to the correct MAP assignment as long as
the MAP assignment is unique. We provide a bound on the number of iterations
required by the algorithm and evaluate the computational cost of the algorithm.
We find that for a graph of size , the computational cost of the algorithm
scales as , which is the same as the computational cost of the best
known algorithm. Finally, we establish the precise relation between the
max-product algorithm and the celebrated {\em auction} algorithm proposed by
Bertsekas. This suggests possible connections between dual algorithm and
max-product algorithm for discrete optimization problems.Comment: In the proceedings of the 2005 IEEE International Symposium on
Information Theor
Causal Message Passing: A Method for Experiments with Unknown and General Network Interference
Randomized experiments are a powerful methodology for data-driven evaluation
of decisions or interventions. Yet, their validity may be undermined by network
interference. This occurs when the treatment of one unit impacts not only its
outcome but also that of connected units, biasing traditional treatment effect
estimations. Our study introduces a new framework to accommodate complex and
unknown network interference, moving beyond specialized models in the existing
literature. Our framework, which we term causal message-passing, is grounded in
a high-dimensional approximate message passing methodology and is specifically
tailored to experimental design settings with prevalent network interference.
Utilizing causal message-passing, we present a practical algorithm for
estimating the total treatment effect and demonstrate its efficacy in four
numerical scenarios, each with its unique interference structure
- β¦