7,172 research outputs found
Greedy Gossip with Eavesdropping
This paper presents greedy gossip with eavesdropping (GGE), a novel
randomized gossip algorithm for distributed computation of the average
consensus problem. In gossip algorithms, nodes in the network randomly
communicate with their neighbors and exchange information iteratively. The
algorithms are simple and decentralized, making them attractive for wireless
network applications. In general, gossip algorithms are robust to unreliable
wireless conditions and time varying network topologies. In this paper we
introduce GGE and demonstrate that greedy updates lead to rapid convergence. We
do not require nodes to have any location information. Instead, greedy updates
are made possible by exploiting the broadcast nature of wireless
communications. During the operation of GGE, when a node decides to gossip,
instead of choosing one of its neighbors at random, it makes a greedy
selection, choosing the node which has the value most different from its own.
In order to make this selection, nodes need to know their neighbors' values.
Therefore, we assume that all transmissions are wireless broadcasts and nodes
keep track of their neighbors' values by eavesdropping on their communications.
We show that the convergence of GGE is guaranteed for connected network
topologies. We also study the rates of convergence and illustrate, through
theoretical bounds and numerical simulations, that GGE consistently outperforms
randomized gossip and performs comparably to geographic gossip on
moderate-sized random geometric graph topologies.Comment: 25 pages, 7 figure
Distributed Learning over Unreliable Networks
Most of today's distributed machine learning systems assume {\em reliable
networks}: whenever two machines exchange information (e.g., gradients or
models), the network should guarantee the delivery of the message. At the same
time, recent work exhibits the impressive tolerance of machine learning
algorithms to errors or noise arising from relaxed communication or
synchronization. In this paper, we connect these two trends, and consider the
following question: {\em Can we design machine learning systems that are
tolerant to network unreliability during training?} With this motivation, we
focus on a theoretical problem of independent interest---given a standard
distributed parameter server architecture, if every communication between the
worker and the server has a non-zero probability of being dropped, does
there exist an algorithm that still converges, and at what speed? The technical
contribution of this paper is a novel theoretical analysis proving that
distributed learning over unreliable network can achieve comparable convergence
rate to centralized or distributed learning over reliable networks. Further, we
prove that the influence of the packet drop rate diminishes with the growth of
the number of \textcolor{black}{parameter servers}. We map this theoretical
result onto a real-world scenario, training deep neural networks over an
unreliable network layer, and conduct network simulation to validate the system
improvement by allowing the networks to be unreliable
A Partition-Based Implementation of the Relaxed ADMM for Distributed Convex Optimization over Lossy Networks
In this paper we propose a distributed implementation of the relaxed
Alternating Direction Method of Multipliers algorithm (R-ADMM) for optimization
of a separable convex cost function, whose terms are stored by a set of
interacting agents, one for each agent. Specifically the local cost stored by
each node is in general a function of both the state of the node and the states
of its neighbors, a framework that we refer to as `partition-based'
optimization. This framework presents a great flexibility and can be adapted to
a large number of different applications. We show that the partition-based
R-ADMM algorithm we introduce is linked to the relaxed Peaceman-Rachford
Splitting (R-PRS) operator which, historically, has been introduced in the
literature to find the zeros of sum of functions. Interestingly, making use of
non expansive operator theory, the proposed algorithm is shown to be provably
robust against random packet losses that might occur in the communication
between neighboring nodes. Finally, the effectiveness of the proposed algorithm
is confirmed by a set of compelling numerical simulations run over random
geometric graphs subject to i.i.d. random packet losses.Comment: Full version of the paper to be presented at Conference on Decision
and Control (CDC) 201
Gossip Algorithms for Distributed Signal Processing
Gossip algorithms are attractive for in-network processing in sensor networks
because they do not require any specialized routing, there is no bottleneck or
single point of failure, and they are robust to unreliable wireless network
conditions. Recently, there has been a surge of activity in the computer
science, control, signal processing, and information theory communities,
developing faster and more robust gossip algorithms and deriving theoretical
performance guarantees. This article presents an overview of recent work in the
area. We describe convergence rate results, which are related to the number of
transmitted messages and thus the amount of energy consumed in the network for
gossiping. We discuss issues related to gossiping over wireless links,
including the effects of quantization and noise, and we illustrate the use of
gossip algorithms for canonical signal processing tasks including distributed
estimation, source localization, and compression.Comment: Submitted to Proceedings of the IEEE, 29 page
- …