Stochastic Modified Equations for Continuous Limit of Stochastic ADMM

Stochastic version of alternating direction method of multiplier (ADMM) and its variants (linearized ADMM, gradient-based ADMM) plays a key role for modern large scale machine learning problems. One example is the regularized empirical risk minimization problem. In this work, we put different variants of stochastic ADMM into a unified form, which includes standard, linearized and gradient-based ADMM with relaxation, and study their dynamics via a continuous-time model approach. We adapt the mathematical framework of stochastic modified equation (SME), and show that the dynamics of stochastic ADMM is approximated by a class of stochastic differential equations with small noise parameters in the sense of weak approximation. The continuous-time analysis would uncover important analytical insights into the behaviors of the discrete-time algorithm, which are non-trivial to gain otherwise. For example, we could characterize the fluctuation of the solution paths precisely, and decide optimal stopping time to minimize the variance of solution paths

Connections between Mean-Field Game and Social Welfare Optimization

This paper studies the connection between a class of mean-field games and a social welfare optimization problem. We consider a mean-field game in function spaces with a large population of agents, and each agent seeks to minimize an individual cost function. The cost functions of different agents are coupled through a mean-field term that depends on the mean of the population states. We show that although the mean-field game is not a potential game, under some mild condition the $\epsilon$-Nash equilibrium of the mean-field game coincides with the optimal solution to a social welfare optimization problem, and this is true even when the individual cost functions are non-convex. The connection enables us to evaluate and promote the efficiency of the mean-field equilibrium. In addition, it also leads to several important implications on the existence, uniqueness, and computation of the mean-field equilibrium. Numerical results are presented to validate the solution, and examples are provided to show the applicability of the proposed approach

Parallel ADMM for robust quadratic optimal resource allocation problems

An alternating direction method of multipliers (ADMM) solver is described for optimal resource allocation problems with separable convex quadratic costs and constraints and linear coupling constraints. We describe a parallel implementation of the solver on a graphics processing unit (GPU) using a bespoke quartic function minimizer. An application to robust optimal energy management in hybrid electric vehicles is described, and the results of numerical simulations comparing the computation times of the parallel GPU implementation with those of an equivalent serial implementation are presented

A Proximal Zeroth-Order Algorithm for Nonconvex Nonsmooth Problems

In this paper, we focus on solving an important class of nonconvex optimization problems which includes many problems for example signal processing over a networked multi-agent system and distributed learning over networks. Motivated by many applications in which the local objective function is the sum of smooth but possibly nonconvex part, and non-smooth but convex part subject to a linear equality constraint, this paper proposes a proximal zeroth-order primal dual algorithm (PZO-PDA) that accounts for the information structure of the problem. This algorithm only utilize the zeroth-order information (i.e., the functional values) of smooth functions, yet the flexibility is achieved for applications that only noisy information of the objective function is accessible, where classical methods cannot be applied. We prove convergence and rate of convergence for PZO-PDA. Numerical experiments are provided to validate the theoretical results

Survey: Sixty Years of Douglas--Rachford

The Douglas--Rachford method is a splitting method frequently employed for finding zeroes of sums of maximally monotone operators. When the operators in question are normal cones operators, the iterated process may be used to solve feasibility problems of the form: Find $x \in \bigcap_{k=1}^N S_k.$ The success of the method in the context of closed, convex, nonempty sets $S_1,\dots,S_N$ is well-known and understood from a theoretical standpoint. However, its performance in the nonconvex context is less understood yet surprisingly impressive. This was particularly compelling to Jonathan M. Borwein who, intrigued by Elser, Rankenburg, and Thibault's success in applying the method for solving Sudoku Puzzles, began an investigation of his own. We survey the current body of literature on the subject, and we summarize its history. We especially commemorate Professor Borwein's celebrated contributions to the area

Dynamical convergence analysis for nonconvex linearized proximal ADMM algorithms

The convergence analysis of optimization algorithms using continuous-time dynamical systems has received much attention in recent years. In this paper, we investigate applications of these systems to analyze the convergence of linearized proximal ADMM algorithms for nonconvex composite optimization, whose objective function is the sum of a continuously differentiable function and a composition of a possibly nonconvex function with a linear operator. We first derive a first-order differential inclusion for the linearized proximal ADMM algorithm, LP-ADMM. Both the global convergence and the convergence rates of the generated trajectory are established with the use of Kurdyka-\L{}ojasiewicz (KL) property. Then, a stochastic variant, LP-SADMM, is delved into an investigation for finite-sum nonconvex composite problems. Under mild conditions, we obtain the stochastic differential equation corresponding to LP-SADMM, and demonstrate the almost sure global convergence of the generated trajectory by leveraging the KL property. Based on the almost sure convergence of trajectory, we construct a stochastic process that converges almost surely to an approximate critical point of objective function, and derive the expected convergence rates associated with this stochastic process. Moreover, we propose an accelerated LP-SADMM that incorporates Nesterov's acceleration technique. The continuous-time dynamical system of this algorithm is modeled as a second-order stochastic differential equation. Within the context of KL property, we explore the related almost sure convergence and expected convergence rates

Newton-Raphson Consensus under asynchronous and lossy communications for peer-to-peer networks

In this work we study the problem of unconstrained convex-optimization in a fully distributed multi-agent setting which includes asynchronous computation and lossy communication. In particular, we extend a recently proposed algorithm named Newton-Raphson Consensus by integrating it with a broadcast-based average consensus algorithm which is robust to packet losses. We show via the separation of time scales principle that under mild conditions (i.e., persistency of the agents activation and bounded consecutive communication failures) the proposed algorithm is proved to be locally exponentially stable with respect to the optimal global solution. Finally, we complement the theoretical analysis with numerical simulations that are based on real datasets

Communication-Efficient Distributed Optimization of Self-Concordant Empirical Loss

We consider distributed convex optimization problems originated from sample average approximation of stochastic optimization, or empirical risk minimization in machine learning. We assume that each machine in the distributed computing system has access to a local empirical loss function, constructed with i.i.d. data sampled from a common distribution. We propose a communication-efficient distributed algorithm to minimize the overall empirical loss, which is the average of the local empirical losses. The algorithm is based on an inexact damped Newton method, where the inexact Newton steps are computed by a distributed preconditioned conjugate gradient method. We analyze its iteration complexity and communication efficiency for minimizing self-concordant empirical loss functions, and discuss the results for distributed ridge regression, logistic regression and binary classification with a smoothed hinge loss. In a standard setting for supervised learning, the required number of communication rounds of the algorithm does not increase with the sample size, and only grows slowly with the number of machines

An FFT-based method for computing the effective crack energy of a heterogeneous material on a combinatorially consistent grid

We introduce an FFT-based solver for the combinatorial continuous maximum flow discretization applied to computing the minimum cut through heterogeneous microstructures. Recently, computational methods were introduced for computing the effective crack energy of periodic and random media. These were based on the continuous minimum cut-maximum flow duality of G. Strang, and made use of discretizations based on trigonometric polynomials and finite elements. For maximum flow problems on graphs, node-based discretization methods avoid metrication artifacts associated to edge-based discretizations. We discretize the minimum cut problem on heterogeneous microstructures by the combinatorial continuous maximum flow discretization introduced by Couprie et al. Furthermore, we introduce an associated FFT-based ADMM solver and provide several adaptive strategies for choosing numerical parameters. We demonstrate the salient features of the proposed approach on problems of industrial scale

Distributed Control Methods for Integrating Renewable Generations and ICT Systems

With increased energy demand and decreased fossil fuels usages, the penetration of distributed generators (DGs) attracts more and more attention. Currently centralized control approaches can no longer meet real-time requirements for future power system. A proper decentralized control strategy needs to be proposed in order to enhance system voltage stability, reduce system power loss and increase operational security. This thesis has three key contributions: Firstly, a decentralized coordinated reactive power control strategy is proposed to tackle voltage fluctuation issues due to the uncertainty of output of DG. Case study shows results of coordinated control methods which can regulate the voltage level effectively whilst also enlarging the total reactive power capability to reduce the possibility of active power curtailment. Subsequently, the communication system time-delay is considered when analyzing the impact of voltage regulation. Secondly, a consensus distributed alternating direction multiplier method (ADMM) algorithm is improved to solve the optimal power ow (OPF) problem. Both synchronous and asynchronous algorithms are proposed to study the performance of convergence rate. Four different strategies are proposed to mitigate the impact of time-delay. Simulation results show that the optimization of reactive power allocation can minimize system power loss effectively and the proposed weighted autoregressive (AR) strategies can achieve an effective convergence result. Thirdly, a neighboring monitoring scheme based on the reputation rating is proposed to detect and mitigate the potential false data injection attack. The simulation results show that the predictive value can effectively replace the manipulated data. The convergence results based on the predictive value can be very close to the results of normal case without cyber attack