10,273 research outputs found
Sketching for Large-Scale Learning of Mixture Models
Learning parameters from voluminous data can be prohibitive in terms of
memory and computational requirements. We propose a "compressive learning"
framework where we estimate model parameters from a sketch of the training
data. This sketch is a collection of generalized moments of the underlying
probability distribution of the data. It can be computed in a single pass on
the training set, and is easily computable on streams or distributed datasets.
The proposed framework shares similarities with compressive sensing, which aims
at drastically reducing the dimension of high-dimensional signals while
preserving the ability to reconstruct them. To perform the estimation task, we
derive an iterative algorithm analogous to sparse reconstruction algorithms in
the context of linear inverse problems. We exemplify our framework with the
compressive estimation of a Gaussian Mixture Model (GMM), providing heuristics
on the choice of the sketching procedure and theoretical guarantees of
reconstruction. We experimentally show on synthetic data that the proposed
algorithm yields results comparable to the classical Expectation-Maximization
(EM) technique while requiring significantly less memory and fewer computations
when the number of database elements is large. We further demonstrate the
potential of the approach on real large-scale data (over 10 8 training samples)
for the task of model-based speaker verification. Finally, we draw some
connections between the proposed framework and approximate Hilbert space
embedding of probability distributions using random features. We show that the
proposed sketching operator can be seen as an innovative method to design
translation-invariant kernels adapted to the analysis of GMMs. We also use this
theoretical framework to derive information preservation guarantees, in the
spirit of infinite-dimensional compressive sensing
A Max-Product EM Algorithm for Reconstructing Markov-tree Sparse Signals from Compressive Samples
We propose a Bayesian expectation-maximization (EM) algorithm for
reconstructing Markov-tree sparse signals via belief propagation. The
measurements follow an underdetermined linear model where the
regression-coefficient vector is the sum of an unknown approximately sparse
signal and a zero-mean white Gaussian noise with an unknown variance. The
signal is composed of large- and small-magnitude components identified by
binary state variables whose probabilistic dependence structure is described by
a Markov tree. Gaussian priors are assigned to the signal coefficients given
their state variables and the Jeffreys' noninformative prior is assigned to the
noise variance. Our signal reconstruction scheme is based on an EM iteration
that aims at maximizing the posterior distribution of the signal and its state
variables given the noise variance. We construct the missing data for the EM
iteration so that the complete-data posterior distribution corresponds to a
hidden Markov tree (HMT) probabilistic graphical model that contains no loops
and implement its maximization (M) step via a max-product algorithm. This EM
algorithm estimates the vector of state variables as well as solves iteratively
a linear system of equations to obtain the corresponding signal estimate. We
select the noise variance so that the corresponding estimated signal and state
variables obtained upon convergence of the EM iteration have the largest
marginal posterior distribution. We compare the proposed and existing
state-of-the-art reconstruction methods via signal and image reconstruction
experiments.Comment: To appear in IEEE Transactions on Signal Processin
Video Compressive Sensing for Dynamic MRI
We present a video compressive sensing framework, termed kt-CSLDS, to
accelerate the image acquisition process of dynamic magnetic resonance imaging
(MRI). We are inspired by a state-of-the-art model for video compressive
sensing that utilizes a linear dynamical system (LDS) to model the motion
manifold. Given compressive measurements, the state sequence of an LDS can be
first estimated using system identification techniques. We then reconstruct the
observation matrix using a joint structured sparsity assumption. In particular,
we minimize an objective function with a mixture of wavelet sparsity and joint
sparsity within the observation matrix. We derive an efficient convex
optimization algorithm through alternating direction method of multipliers
(ADMM), and provide a theoretical guarantee for global convergence. We
demonstrate the performance of our approach for video compressive sensing, in
terms of reconstruction accuracy. We also investigate the impact of various
sampling strategies. We apply this framework to accelerate the acquisition
process of dynamic MRI and show it achieves the best reconstruction accuracy
with the least computational time compared with existing algorithms in the
literature.Comment: 30 pages, 9 figure
Compressive Spectral Clustering
Spectral clustering has become a popular technique due to its high
performance in many contexts. It comprises three main steps: create a
similarity graph between N objects to cluster, compute the first k eigenvectors
of its Laplacian matrix to define a feature vector for each object, and run
k-means on these features to separate objects into k classes. Each of these
three steps becomes computationally intensive for large N and/or k. We propose
to speed up the last two steps based on recent results in the emerging field of
graph signal processing: graph filtering of random signals, and random sampling
of bandlimited graph signals. We prove that our method, with a gain in
computation time that can reach several orders of magnitude, is in fact an
approximation of spectral clustering, for which we are able to control the
error. We test the performance of our method on artificial and real-world
network data.Comment: 12 pages, 2 figure
Off-grid Direction of Arrival Estimation Using Sparse Bayesian Inference
Direction of arrival (DOA) estimation is a classical problem in signal
processing with many practical applications. Its research has recently been
advanced owing to the development of methods based on sparse signal
reconstruction. While these methods have shown advantages over conventional
ones, there are still difficulties in practical situations where true DOAs are
not on the discretized sampling grid. To deal with such an off-grid DOA
estimation problem, this paper studies an off-grid model that takes into
account effects of the off-grid DOAs and has a smaller modeling error. An
iterative algorithm is developed based on the off-grid model from a Bayesian
perspective while joint sparsity among different snapshots is exploited by
assuming a Laplace prior for signals at all snapshots. The new approach applies
to both single snapshot and multi-snapshot cases. Numerical simulations show
that the proposed algorithm has improved accuracy in terms of mean squared
estimation error. The algorithm can maintain high estimation accuracy even
under a very coarse sampling grid.Comment: To appear in the IEEE Trans. Signal Processing. This is a revised,
shortened version of version
Dynamic Compressive Sensing of Time-Varying Signals via Approximate Message Passing
In this work the dynamic compressive sensing (CS) problem of recovering
sparse, correlated, time-varying signals from sub-Nyquist, non-adaptive, linear
measurements is explored from a Bayesian perspective. While there has been a
handful of previously proposed Bayesian dynamic CS algorithms in the
literature, the ability to perform inference on high-dimensional problems in a
computationally efficient manner remains elusive. In response, we propose a
probabilistic dynamic CS signal model that captures both amplitude and support
correlation structure, and describe an approximate message passing algorithm
that performs soft signal estimation and support detection with a computational
complexity that is linear in all problem dimensions. The algorithm, DCS-AMP,
can perform either causal filtering or non-causal smoothing, and is capable of
learning model parameters adaptively from the data through an
expectation-maximization learning procedure. We provide numerical evidence that
DCS-AMP performs within 3 dB of oracle bounds on synthetic data under a variety
of operating conditions. We further describe the result of applying DCS-AMP to
two real dynamic CS datasets, as well as a frequency estimation task, to
bolster our claim that DCS-AMP is capable of offering state-of-the-art
performance and speed on real-world high-dimensional problems.Comment: 32 pages, 7 figure
- …