876 research outputs found
An Online Parallel and Distributed Algorithm for Recursive Estimation of Sparse Signals
In this paper, we consider a recursive estimation problem for linear
regression where the signal to be estimated admits a sparse representation and
measurement samples are only sequentially available. We propose a convergent
parallel estimation scheme that consists in solving a sequence of
-regularized least-square problems approximately. The proposed scheme
is novel in three aspects: i) all elements of the unknown vector variable are
updated in parallel at each time instance, and convergence speed is much faster
than state-of-the-art schemes which update the elements sequentially; ii) both
the update direction and stepsize of each element have simple closed-form
expressions, so the algorithm is suitable for online (real-time)
implementation; and iii) the stepsize is designed to accelerate the convergence
but it does not suffer from the common trouble of parameter tuning in
literature. Both centralized and distributed implementation schemes are
discussed. The attractive features of the proposed algorithm are also
numerically consolidated.Comment: Part of this work has been presented at The Asilomar Conference on
Signals, Systems, and Computers, Nov. 201
M-Power Regularized Least Squares Regression
Regularization is used to find a solution that both fits the data and is
sufficiently smooth, and thereby is very effective for designing and refining
learning algorithms. But the influence of its exponent remains poorly
understood. In particular, it is unclear how the exponent of the reproducing
kernel Hilbert space~(RKHS) regularization term affects the accuracy and the
efficiency of kernel-based learning algorithms. Here we consider regularized
least squares regression (RLSR) with an RKHS regularization raised to the power
of m, where m is a variable real exponent. We design an efficient algorithm for
solving the associated minimization problem, we provide a theoretical analysis
of its stability, and we compare its advantage with respect to computational
complexity, speed of convergence and prediction accuracy to the classical
kernel ridge regression algorithm where the regularization exponent m is fixed
at 2. Our results show that the m-power RLSR problem can be solved efficiently,
and support the suggestion that one can use a regularization term that grows
significantly slower than the standard quadratic growth in the RKHS norm
Adaptive Graph Signal Processing: Algorithms and Optimal Sampling Strategies
The goal of this paper is to propose novel strategies for adaptive learning
of signals defined over graphs, which are observed over a (randomly
time-varying) subset of vertices. We recast two classical adaptive algorithms
in the graph signal processing framework, namely, the least mean squares (LMS)
and the recursive least squares (RLS) adaptive estimation strategies. For both
methods, a detailed mean-square analysis illustrates the effect of random
sampling on the adaptive reconstruction capability and the steady-state
performance. Then, several probabilistic sampling strategies are proposed to
design the sampling probability at each node in the graph, with the aim of
optimizing the tradeoff between steady-state performance, graph sampling rate,
and convergence rate of the adaptive algorithms. Finally, a distributed RLS
strategy is derived and is shown to be convergent to its centralized
counterpart. Numerical simulations carried out over both synthetic and real
data illustrate the good performance of the proposed sampling and
reconstruction strategies for (possibly distributed) adaptive learning of
signals defined over graphs.Comment: Submitted to IEEE Transactions on Signal Processing, September 201
A Stochastic Majorize-Minimize Subspace Algorithm for Online Penalized Least Squares Estimation
Stochastic approximation techniques play an important role in solving many
problems encountered in machine learning or adaptive signal processing. In
these contexts, the statistics of the data are often unknown a priori or their
direct computation is too intensive, and they have thus to be estimated online
from the observed signals. For batch optimization of an objective function
being the sum of a data fidelity term and a penalization (e.g. a sparsity
promoting function), Majorize-Minimize (MM) methods have recently attracted
much interest since they are fast, highly flexible, and effective in ensuring
convergence. The goal of this paper is to show how these methods can be
successfully extended to the case when the data fidelity term corresponds to a
least squares criterion and the cost function is replaced by a sequence of
stochastic approximations of it. In this context, we propose an online version
of an MM subspace algorithm and we study its convergence by using suitable
probabilistic tools. Simulation results illustrate the good practical
performance of the proposed algorithm associated with a memory gradient
subspace, when applied to both non-adaptive and adaptive filter identification
problems
- …