9,936 research outputs found
Distributed Bayesian Learning with Stochastic Natural-gradient Expectation Propagation and the Posterior Server
This paper makes two contributions to Bayesian machine learning algorithms.
Firstly, we propose stochastic natural gradient expectation propagation (SNEP),
a novel alternative to expectation propagation (EP), a popular variational
inference algorithm. SNEP is a black box variational algorithm, in that it does
not require any simplifying assumptions on the distribution of interest, beyond
the existence of some Monte Carlo sampler for estimating the moments of the EP
tilted distributions. Further, as opposed to EP which has no guarantee of
convergence, SNEP can be shown to be convergent, even when using Monte Carlo
moment estimates. Secondly, we propose a novel architecture for distributed
Bayesian learning which we call the posterior server. The posterior server
allows scalable and robust Bayesian learning in cases where a data set is
stored in a distributed manner across a cluster, with each compute node
containing a disjoint subset of data. An independent Monte Carlo sampler is run
on each compute node, with direct access only to the local data subset, but
which targets an approximation to the global posterior distribution given all
data across the whole cluster. This is achieved by using a distributed
asynchronous implementation of SNEP to pass messages across the cluster. We
demonstrate SNEP and the posterior server on distributed Bayesian learning of
logistic regression and neural networks.
Keywords: Distributed Learning, Large Scale Learning, Deep Learning, Bayesian
Learn- ing, Variational Inference, Expectation Propagation, Stochastic
Approximation, Natural Gradient, Markov chain Monte Carlo, Parameter Server,
Posterior Server.Comment: 37 pages, 7 figure
Towards Efficient Maximum Likelihood Estimation of LPV-SS Models
How to efficiently identify multiple-input multiple-output (MIMO) linear
parameter-varying (LPV) discrete-time state-space (SS) models with affine
dependence on the scheduling variable still remains an open question, as
identification methods proposed in the literature suffer heavily from the curse
of dimensionality and/or depend on over-restrictive approximations of the
measured signal behaviors. However, obtaining an SS model of the targeted
system is crucial for many LPV control synthesis methods, as these synthesis
tools are almost exclusively formulated for the aforementioned representation
of the system dynamics. Therefore, in this paper, we tackle the problem by
combining state-of-the-art LPV input-output (IO) identification methods with an
LPV-IO to LPV-SS realization scheme and a maximum likelihood refinement step.
The resulting modular LPV-SS identification approach achieves statical
efficiency with a relatively low computational load. The method contains the
following three steps: 1) estimation of the Markov coefficient sequence of the
underlying system using correlation analysis or Bayesian impulse response
estimation, then 2) LPV-SS realization of the estimated coefficients by using a
basis reduced Ho-Kalman method, and 3) refinement of the LPV-SS model estimate
from a maximum-likelihood point of view by a gradient-based or an
expectation-maximization optimization methodology. The effectiveness of the
full identification scheme is demonstrated by a Monte Carlo study where our
proposed method is compared to existing schemes for identifying a MIMO LPV
system
Inertial Stochastic PALM (iSPALM) and Applications in Machine Learning
Inertial algorithms for minimizing nonsmooth and nonconvex functions as the
inertial proximal alternating linearized minimization algorithm (iPALM) have
demonstrated their superiority with respect to computation time over their non
inertial variants. In many problems in imaging and machine learning, the
objective functions have a special form involving huge data which encourage the
application of stochastic algorithms. While algorithms based on stochastic
gradient descent are still used in the majority of applications, recently also
stochastic algorithms for minimizing nonsmooth and nonconvex functions were
proposed. In this paper, we derive an inertial variant of a stochastic PALM
algorithm with variance-reduced gradient estimator, called iSPALM, and prove
linear convergence of the algorithm under certain assumptions. Our inertial
approach can be seen as generalization of momentum methods widely used to speed
up and stabilize optimization algorithms, in particular in machine learning, to
nonsmooth problems. Numerical experiments for learning the weights of a
so-called proximal neural network and the parameters of Student-t mixture
models show that our new algorithm outperforms both stochastic PALM and its
deterministic counterparts
- …