4,651 research outputs found

    Distributed Machine Learning via Sufficient Factor Broadcasting

    Full text link
    Matrix-parametrized models, including multiclass logistic regression and sparse coding, are used in machine learning (ML) applications ranging from computer vision to computational biology. When these models are applied to large-scale ML problems starting at millions of samples and tens of thousands of classes, their parameter matrix can grow at an unexpected rate, resulting in high parameter synchronization costs that greatly slow down distributed learning. To address this issue, we propose a Sufficient Factor Broadcasting (SFB) computation model for efficient distributed learning of a large family of matrix-parameterized models, which share the following property: the parameter update computed on each data sample is a rank-1 matrix, i.e., the outer product of two "sufficient factors" (SFs). By broadcasting the SFs among worker machines and reconstructing the update matrices locally at each worker, SFB improves communication efficiency --- communication costs are linear in the parameter matrix's dimensions, rather than quadratic --- without affecting computational correctness. We present a theoretical convergence analysis of SFB, and empirically corroborate its efficiency on four different matrix-parametrized ML models

    A Distributed Frank-Wolfe Algorithm for Communication-Efficient Sparse Learning

    Full text link
    Learning sparse combinations is a frequent theme in machine learning. In this paper, we study its associated optimization problem in the distributed setting where the elements to be combined are not centrally located but spread over a network. We address the key challenges of balancing communication costs and optimization errors. To this end, we propose a distributed Frank-Wolfe (dFW) algorithm. We obtain theoretical guarantees on the optimization error ϵ\epsilon and communication cost that do not depend on the total number of combining elements. We further show that the communication cost of dFW is optimal by deriving a lower-bound on the communication cost required to construct an ϵ\epsilon-approximate solution. We validate our theoretical analysis with empirical studies on synthetic and real-world data, which demonstrate that dFW outperforms both baselines and competing methods. We also study the performance of dFW when the conditions of our analysis are relaxed, and show that dFW is fairly robust.Comment: Extended version of the SIAM Data Mining 2015 pape

    A Stochastic Broadcast Pi-Calculus

    Get PDF
    In this paper we propose a stochastic broadcast PI-calculus which can be used to model server-client based systems where synchronization is always governed by only one participant. Therefore, there is no need to determine the joint synchronization rates. We also take immediate transitions into account which is useful to model behaviors with no impact on the temporal properties of a system. Since immediate transitions may introduce non-determinism, we will show how these non-determinism can be resolved, and as result a valid CTMC will be obtained finally. Also some practical examples are given to show the application of this calculus.Comment: In Proceedings QAPL 2011, arXiv:1107.074

    The Stability and Control of Stochastically Switching Dynamical Systems

    Get PDF
    Inherent randomness and unpredictability is an underlying property in most realistic phenomena. In this work, we present a new framework for introducing stochasticity into dynamical systems via intermittently switching between deterministic regimes. Extending the work by Belykh, Belykh, and Hasler, we provide analytical insight into how randomly switching network topologies behave with respect to their averaged, static counterparts (obtained by replacing the stochastic variables with their expectation) when switching is fast. Beyond fast switching, we uncover a highly nontrivial phenomenon by which a network can switch between two asynchronous regimes and synchronize against all odds. Then, we establish rigorous theory for this framework in discrete-time systems for arbitrary switching periods (not limited to switching at each time step). Using stability and ergodic theories, we are able to provide analytical criteria for the stability of synchronization for two coupled maps and the ability of a single map to control an arbitrary network of maps. This work not only presents new phenomena in stochastically switching dynamical systems, but also provides the first rigorous analysis of switching dynamical systems with an arbitrary switching period
    corecore