Asynchronous Optimization Methods for Efficient Training of Deep Neural Networks with Guarantees

Asynchronous distributed algorithms are a popular way to reduce synchronization costs in large-scale optimization, and in particular for neural network training. However, for nonsmooth and nonconvex objectives, few convergence guarantees exist beyond cases where closed-form proximal operator solutions are available. As most popular contemporary deep neural networks lead to nonsmooth and nonconvex objectives, there is now a pressing need for such convergence guarantees. In this paper, we analyze for the first time the convergence of stochastic asynchronous optimization for this general class of objectives. In particular, we focus on stochastic subgradient methods allowing for block variable partitioning, where the shared-memory-based model is asynchronously updated by concurrent processes. To this end, we first introduce a probabilistic model which captures key features of real asynchronous scheduling between concurrent processes; under this model, we establish convergence with probability one to an invariant set for stochastic subgradient methods with momentum. From the practical perspective, one issue with the family of methods we consider is that it is not efficiently supported by machine learning frameworks, as they mostly focus on distributed data-parallel strategies. To address this, we propose a new implementation strategy for shared-memory based training of deep neural networks, whereby concurrent parameter servers are utilized to train a partitioned but shared model in single- and multi-GPU settings. Based on this implementation, we achieve on average 1.2x speed-up in comparison to state-of-the-art training methods for popular image classification tasks without compromising accuracy

Adaptive asynchronous time-stepping, stopping criteria, and a posteriori error estimates for fixed-stress iterative schemes for coupled poromechanics problems

In this paper we develop adaptive iterative coupling schemes for the Biot system modeling coupled poromechanics problems. We particularly consider the space-time formulation of the fixed-stress iterative scheme, in which we first solve the problem of flow over the whole space-time interval, then exploiting the space-time information for solving the mechanics. Two common discretizations of this algorithm are then introduced based on two coupled mixed finite element methods in-space and the backward Euler scheme in-time. Therefrom, adaptive fixed-stress algorithms are build on conforming reconstructions of the pressure and displacement together with equilibrated flux and stresses reconstructions. These ingredients are used to derive a posteriori error estimates for the fixed-stress algorithms, distinguishing the different error components, namely the spatial discretization, the temporal discretization, and the fixed-stress iteration components. Precisely, at the iteration $k\geq 1$ of the adaptive algorithm, we prove that our estimate gives a guaranteed and fully computable upper bound on the energy-type error measuring the difference between the exact and approximate pressure and displacement. These error components are efficiently used to design adaptive asynchronous time-stepping and adaptive stopping criteria for the fixed-stress algorithms. Numerical experiments illustrate the efficiency of our estimates and the performance of the adaptive iterative coupling algorithms

Reaching Approximate Byzantine Consensus in Partially-Connected Mobile Networks

We consider the problem of approximate consensus in mobile networks containing Byzantine nodes. We assume that each correct node can communicate only with its neighbors and has no knowledge of the global topology. As all nodes have moving ability, the topology is dynamic. The number of Byzantine nodes is bounded by f and known by all correct nodes. We first introduce an approximate Byzantine consensus protocol which is based on the linear iteration method. As nodes are allowed to collect information during several consecutive rounds, moving gives them the opportunity to gather more values. We propose a novel sufficient and necessary condition to guarantee the final convergence of the consensus protocol. The requirement expressed by our condition is not "universal": in each phase it affects only a single correct node. More precisely, at least one correct node among those that propose either the minimum or the maximum value which is present in the network, has to receive enough messages (quantity constraint) with either higher or lower values (quality constraint). Of course, nodes' motion should not prevent this requirement to be fulfilled. Our conclusion shows that the proposed condition can be satisfied if the total number of nodes is greater than 3f+1.Comment: No. RR-7985 (2012

Fault tolerant architectures for integrated aircraft electronics systems

Work into possible architectures for future flight control computer systems is described. Ada for Fault-Tolerant Systems, the NETS Network Error-Tolerant System architecture, and voting in asynchronous systems are covered