Parallel coordinate descent for the Adaboost problem

We design a randomised parallel version of Adaboost based on previous studies on parallel coordinate descent. The algorithm uses the fact that the logarithm of the exponential loss is a function with coordinate-wise Lipschitz continuous gradient, in order to define the step lengths. We provide the proof of convergence for this randomised Adaboost algorithm and a theoretical parallelisation speedup factor. We finally provide numerical examples on learning problems of various sizes that show that the algorithm is competitive with concurrent approaches, especially for large scale problems.Comment: 7 pages, 3 figures, extended version of the paper presented to ICMLA'1

Distributed Optimisation with Linear Equality and Inequality Constraints using PDMM

In this paper, we consider the problem of distributed optimisation of a separable convex cost function over a graph, where every edge and node in the graph could carry both linear equality and/or inequality constraints. We show how to modify the primal-dual method of multipliers (PDMM), originally designed for linear equality constraints, such that it can handle inequality constraints as well. In contrast to most existing algorithms for optimisation with inequality constraints, the proposed algorithm does not need any slack variables. Using convex analysis, monotone operator theory and fixed-point theory, we show how to derive the update equations of the modified PDMM algorithm by applying Peaceman-Rachford splitting to the monotonic inclusion related to the extended dual problem. To incorporate the inequality constraints, we impose a non-negativity constraint on the associated dual variables. This additional constraint results in the introduction of a reflection operator to model the data exchange in the network, instead of a permutation operator as derived for equality constraint PDMM. Convergence for both synchronous and stochastic update schemes of PDMM are provided. The latter includes asynchronous update schemes and update schemes with transmission losses.Comment: 9 page

On Distributed Nonconvex Optimisation Via Modified ADMM

This paper addresses the problem of nonconvex nonsmooth decentralised optimisation in multi-agent networks with undirected connected communication graphs. Our contribution lies in introducing an algorithmic framework designed for the distributed minimisation of the sum of a smooth (possibly nonconvex and non-separable) function and a convex (possibly nonsmooth and non-separable) regulariser. The proposed algorithm can be seen as a modified version of the ADMM algorithm where, at each step, an "inner loop" needs to be iterated for a number of iterations. The role of the inner loop is to aggregate and disseminate information across the network. We observe that a naive decentralised approach (one iteration of the inner loop) may not converge. We establish the asymptotic convergence of the proposed algorithm to the set of stationary points of the nonconvex problem where the number of iterations of the inner loop increases logarithmically with the step count of the ADMM algorithm. We present numerical results demonstrating the proposed method's correctness and performance.Comment: 6 pages, 1 Figur

Distributed Convex Optimisation using the Alternating Direction Method of Multipliers (ADMM) in Lossy Scenarios

The Alternating Direction Method of Multipliers (ADMM) is an extensively studied algorithm suitable for solving convex distributed optimisation problems. This Thesis presents a formulation of the ADMM that is guaranteed to converge if the communications among agents are faulty and the agents perform updates asynchronously. With strongly convex costs, the proposed algorithm is shown to converge exponentially fast. The further extension to partition-based problems is presented

Asynchronous Distributed Optimization over Lossy Networks via Relaxed ADMM: Stability and Linear Convergence

In this work we focus on the problem of minimizing the sum of convex cost functions in a distributed fashion over a peer-to-peer network. In particular, we are interested in the case in which communications between nodes are prone to failures and the agents are not synchronized among themselves. We address the problem proposing a modified version of the relaxed ADMM, which corresponds to the Peaceman-Rachford splitting method applied to the dual. By exploiting results from operator theory, we are able to prove the almost sure convergence of the proposed algorithm under general assumptions on the distribution of communication loss and node activation events. By further assuming the cost functions to be strongly convex, we prove the linear convergence of the algorithm in mean to a neighborhood of the optimal solution, and provide an upper bound to the convergence rate. Finally, we present numerical results testing the proposed method in different scenarios.Comment: To appear in IEEE Transactions on Automatic Contro

Cloud-Based Centralized/Decentralized Multi-Agent Optimization with Communication Delays

We present and analyze a computational hybrid architecture for performing multi-agent optimization. The optimization problems under consideration have convex objective and constraint functions with mild smoothness conditions imposed on them. For such problems, we provide a primal-dual algorithm implemented in the hybrid architecture, which consists of a decentralized network of agents into which centralized information is occasionally injected, and we establish its convergence properties. To accomplish this, a central cloud computer aggregates global information, carries out computations of the dual variables based on this information, and then distributes the updated dual variables to the agents. The agents update their (primal) state variables and also communicate among themselves with each agent sharing and receiving state information with some number of its neighbors. Throughout, communications with the cloud are not assumed to be synchronous or instantaneous, and communication delays are explicitly accounted for in the modeling and analysis of the system. Experimental results are presented to support the theoretical developments made.Comment: 8 pages, 4 figure