636 research outputs found
Genuinely Distributed Byzantine Machine Learning
Machine Learning (ML) solutions are nowadays distributed, according to the
so-called server/worker architecture. One server holds the model parameters
while several workers train the model. Clearly, such architecture is prone to
various types of component failures, which can be all encompassed within the
spectrum of a Byzantine behavior. Several approaches have been proposed
recently to tolerate Byzantine workers. Yet all require trusting a central
parameter server. We initiate in this paper the study of the ``general''
Byzantine-resilient distributed machine learning problem where no individual
component is trusted.
We show that this problem can be solved in an asynchronous system, despite
the presence of Byzantine parameter servers and
Byzantine workers (which is optimal). We present a new algorithm, ByzSGD, which
solves the general Byzantine-resilient distributed machine learning problem by
relying on three major schemes. The first, Scatter/Gather, is a communication
scheme whose goal is to bound the maximum drift among models on correct
servers. The second, Distributed Median Contraction (DMC), leverages the
geometric properties of the median in high dimensional spaces to bring
parameters within the correct servers back close to each other, ensuring
learning convergence. The third, Minimum-Diameter Averaging (MDA), is a
statistically-robust gradient aggregation rule whose goal is to tolerate
Byzantine workers. MDA requires loose bound on the variance of non-Byzantine
gradient estimates, compared to existing alternatives (e.g., Krum).
Interestingly, ByzSGD ensures Byzantine resilience without adding communication
rounds (on a normal path), compared to vanilla non-Byzantine alternatives.
ByzSGD requires, however, a larger number of messages which, we show, can be
reduced if we assume synchrony.Comment: This is a merge of arXiv:1905.03853 and arXiv:1911.07537;
arXiv:1911.07537 will be retracte
A Survey on Consortium Blockchain Consensus Mechanisms
Blockchain is a distributed ledger that is decentralized, immutable, and
transparent, which maintains a continuously growing list of transaction records
ordered into blocks. As the core of blockchain, the consensus algorithm is an
agreement to validate the correctness of blockchain transactions. For example,
Bitcoin is a public blockchain where each node in Bitcoin uses the Proof of
Work (PoW) algorithm to reach a consensus by competing to solve a puzzle.
Unlike a public blockchain, a consortium blockchain is an enterprise-level
blockchain that does not contend with the issues of creating a resource-saving
global consensus protocol. This paper highilights several state-of-the art
solutions in consensus algorithms for enterprise blockchain. For example, the
HyperLedger by Linux Foundation includes implementing Practical Byzantine Fault
Tolerance (PBFT) as the consensus algorithm. PBFT can tolerate a range of
malicious nodes and reach consensus with quadratic complexity. Another
consensus algorithm, HotStuff, implemented by Facebook Libra project, has
achieved linear complexity of the authenticator. This paper presents the
operational mechanisms of these and other consensus protocols, and analyzes and
compares their advantages and drawbacks.Comment: under submissio
- …