4,651 research outputs found
Distributed Machine Learning via Sufficient Factor Broadcasting
Matrix-parametrized models, including multiclass logistic regression and
sparse coding, are used in machine learning (ML) applications ranging from
computer vision to computational biology. When these models are applied to
large-scale ML problems starting at millions of samples and tens of thousands
of classes, their parameter matrix can grow at an unexpected rate, resulting in
high parameter synchronization costs that greatly slow down distributed
learning. To address this issue, we propose a Sufficient Factor Broadcasting
(SFB) computation model for efficient distributed learning of a large family of
matrix-parameterized models, which share the following property: the parameter
update computed on each data sample is a rank-1 matrix, i.e., the outer product
of two "sufficient factors" (SFs). By broadcasting the SFs among worker
machines and reconstructing the update matrices locally at each worker, SFB
improves communication efficiency --- communication costs are linear in the
parameter matrix's dimensions, rather than quadratic --- without affecting
computational correctness. We present a theoretical convergence analysis of
SFB, and empirically corroborate its efficiency on four different
matrix-parametrized ML models
A Distributed Frank-Wolfe Algorithm for Communication-Efficient Sparse Learning
Learning sparse combinations is a frequent theme in machine learning. In this
paper, we study its associated optimization problem in the distributed setting
where the elements to be combined are not centrally located but spread over a
network. We address the key challenges of balancing communication costs and
optimization errors. To this end, we propose a distributed Frank-Wolfe (dFW)
algorithm. We obtain theoretical guarantees on the optimization error
and communication cost that do not depend on the total number of
combining elements. We further show that the communication cost of dFW is
optimal by deriving a lower-bound on the communication cost required to
construct an -approximate solution. We validate our theoretical
analysis with empirical studies on synthetic and real-world data, which
demonstrate that dFW outperforms both baselines and competing methods. We also
study the performance of dFW when the conditions of our analysis are relaxed,
and show that dFW is fairly robust.Comment: Extended version of the SIAM Data Mining 2015 pape
A Stochastic Broadcast Pi-Calculus
In this paper we propose a stochastic broadcast PI-calculus which can be used
to model server-client based systems where synchronization is always governed
by only one participant. Therefore, there is no need to determine the joint
synchronization rates. We also take immediate transitions into account which is
useful to model behaviors with no impact on the temporal properties of a
system. Since immediate transitions may introduce non-determinism, we will show
how these non-determinism can be resolved, and as result a valid CTMC will be
obtained finally. Also some practical examples are given to show the
application of this calculus.Comment: In Proceedings QAPL 2011, arXiv:1107.074
The Stability and Control of Stochastically Switching Dynamical Systems
Inherent randomness and unpredictability is an underlying property in most realistic phenomena. In this work, we present a new framework for introducing stochasticity into dynamical systems via intermittently switching between deterministic regimes. Extending the work by Belykh, Belykh, and Hasler, we provide analytical insight into how randomly switching network topologies behave with respect to their averaged, static counterparts (obtained by replacing the stochastic variables with their expectation) when switching is fast. Beyond fast switching, we uncover a highly nontrivial phenomenon by which a network can switch between two asynchronous regimes and synchronize against all odds. Then, we establish rigorous theory for this framework in discrete-time systems for arbitrary switching periods (not limited to switching at each time step). Using stability and ergodic theories, we are able to provide analytical criteria for the stability of synchronization for two coupled maps and the ability of a single map to control an arbitrary network of maps. This work not only presents new phenomena in stochastically switching dynamical systems, but also provides the first rigorous analysis of switching dynamical systems with an arbitrary switching period
- …