Online Distributed Optimization with Clipped Stochastic Gradients: High Probability Bound of Regrets

Abstract

In this paper, the problem of distributed optimization is studied via a network of agents. Each agent only has access to a stochastic gradient of its own objective function in the previous time, and can communicate with its neighbors via a network. To handle this problem, an online distributed clipped stochastic gradient descent algorithm is proposed. Dynamic regrets are used to capture the performance of the algorithm. Particularly, the high probability bounds of regrets are analyzed when the stochastic gradients satisfy the heavy-tailed noise condition. For the convex case, the offline benchmark of the dynamic regret is to seek the minimizer of the objective function each time. Under mild assumptions on the graph connectivity, we prove that the dynamic regret grows sublinearly with high probability under a certain clipping parameter. For the non-convex case, the offline benchmark of the dynamic regret is to find the stationary point of the objective function each time. We show that the dynamic regret grows sublinearly with high probability if the variation of the objective function grows within a certain rate. Finally, numerical simulations are provided to demonstrate the effectiveness of our theoretical results