Genuinely Distributed Byzantine Machine Learning

Machine Learning (ML) solutions are nowadays distributed, according to the so-called server/worker architecture. One server holds the model parameters while several workers train the model. Clearly, such architecture is prone to various types of component failures, which can be all encompassed within the spectrum of a Byzantine behavior. Several approaches have been proposed recently to tolerate Byzantine workers. Yet all require trusting a central parameter server. We initiate in this paper the study of the ``general'' Byzantine-resilient distributed machine learning problem where no individual component is trusted. We show that this problem can be solved in an asynchronous system, despite the presence of $\frac{1}{3}$ Byzantine parameter servers and $\frac{1}{3}$ Byzantine workers (which is optimal). We present a new algorithm, ByzSGD, which solves the general Byzantine-resilient distributed machine learning problem by relying on three major schemes. The first, Scatter/Gather, is a communication scheme whose goal is to bound the maximum drift among models on correct servers. The second, Distributed Median Contraction (DMC), leverages the geometric properties of the median in high dimensional spaces to bring parameters within the correct servers back close to each other, ensuring learning convergence. The third, Minimum-Diameter Averaging (MDA), is a statistically-robust gradient aggregation rule whose goal is to tolerate Byzantine workers. MDA requires loose bound on the variance of non-Byzantine gradient estimates, compared to existing alternatives (e.g., Krum). Interestingly, ByzSGD ensures Byzantine resilience without adding communication rounds (on a normal path), compared to vanilla non-Byzantine alternatives. ByzSGD requires, however, a larger number of messages which, we show, can be reduced if we assume synchrony.Comment: This is a merge of arXiv:1905.03853 and arXiv:1911.07537; arXiv:1911.07537 will be retracte

Dynamic Resilient Containment Control in Multirobot Systems

In this article, we study the dynamic resilient containment control problem for continuous-time multirobot systems (MRSs), i.e., the problem of designing a local interaction protocol that drives a set of robots, namely the followers, toward a region delimited by the positions of another set of robots, namely the leaders, under the presence of adversarial robots in the network. In our setting, all robots are anonymous, i.e., they do not recognize the identity or class of other robots. We consider as adversarial all those robots that intentionally or accidentally try to disrupt the objective of the MRS, e.g., robots that are being hijacked by a cyber–physical attack or have experienced a fault. Under specific topological conditions defined by the notion of (r,s)-robustness, our control strategy is proven to be successful in driving the followers toward the target region, namely a hypercube, in finite time. It is also proven that the followers cannot escape the moving containment area despite the persistent influence of anonymous adversarial robots. Numerical results with a team of 44 robots are provided to corroborate the theoretical findings

Resilient Multi-Dimensional Consensus in Adversarial Environment

This paper considers the multi-dimensional consensus in networked systems, where some of the agents might be misbehaving (or faulty). Despite the influence of these misbehaviors, the healthy agents aim to reach an agreement within the convex hull of their initial states. Towards this end, this paper develops a resilient consensus algorithm, where each healthy agent sorts its received values on one dimension, computes two "middle points" based on the sorted values, and moves its state toward these middle points. We further show that the computation of middle points can be efficiently achieved by linear programming. Compared with the existing works, this approach has lower computational complexity. Assuming that the number of malicious agents is upper bounded, sufficient conditions on the network topology are then presented to guarantee the achievement of resilient consensus. Some numerical examples are finally provided to verify the theoretical results.Comment: arXiv admin note: substantial text overlap with arXiv:1911.1083

Byzantine-Resilient Distributed Optimization of Multi-Dimensional Functions

The problem of distributed optimization requires a group of agents to reach agreement on a parameter that minimizes the average of their local cost functions using information received from their neighbors. While there are a variety of distributed optimization algorithms that can solve this problem, they are typically vulnerable to malicious (or "Byzantine") agents that do not follow the algorithm. Recent attempts to address this issue focus on single dimensional functions, or provide analysis under certain assumptions on the statistical properties of the functions at the agents. In this paper, we propose a resilient distributed optimization algorithm for multi-dimensional convex functions. Our scheme involves two filtering steps at each iteration of the algorithm: (1) distance-based and (2) component-wise removal of extreme states. We show that this algorithm can mitigate the impact of up to F Byzantine agents in the neighborhood of each regular node, without knowing the identities of the Byzantine agents in advance. In particular, we show that if the network topology satisfies certain conditions, all of the regular states are guaranteed to asymptotically converge to a bounded region that contains the global minimizer.Comment: 10 pages, 1 figure. To appear in the Proceedings of the 2020 American Control Conference, 1-3 July 2020, Denver, CO, US