Distributed optimization for multi-agent system over unbalanced graphs with linear convergence rate

summary:Distributed optimization over unbalanced graphs is an important problem in multi-agent systems. Most of literatures, by introducing some auxiliary variables, utilize the Push-Sum scheme to handle the widespread unbalance graph with row or column stochastic matrix only. But the introduced auxiliary dynamics bring more calculation and communication tasks. In this paper, based on the in-degree and out-degree information of each agent, we propose an innovative distributed optimization algorithm to reduce the calculation and communication complexity of the conventional Push-Sum scheme. Furthermore, with the aid of small gain theory, we prove the linear convergence rate of the proposed algorithm

CONSENSUS, PREDICTION AND OPTIMIZATION IN DIRECTED NETWORKS

This dissertation develops theory and algorithms for distributed consensus in multi-agent networks. The models considered are opinion dynamics models based on the well known DeGroot model. We study the following three related topics: consensus of networks with leaders, consensus prediction, and distributed optimization. First, we revisit the problem of agreement seeking in a weighted directed network in the presence of leaders. We develop new sufficient conditions that are weaker than existing conditions for guaranteeing consensus for both fixed and switching network topologies, emphasizing the importance not only of persistent connectivity between the leader and the followers but also of the strength of the connections. We then study the problem of a leader aiming to maximize its influence on the opinions of the network agents through targeted connection with a limited number of agents, possibly in the presence of another leader having a competing opinion. We reveal fundamental properties of leader influence defined in terms of either the transient behavior or the achieved steady state opinions of the network agents. In particular, not only is the degree of this influence a supermodular set function, but its continuous relaxation is also convex for any strongly connected directed network. These results pave the way for developing efficient approximation algorithms admitting certain quality certifications, which when combined can provide effective tools and better analysis for optimal influence spreading in large networks. Second, we introduce and investigate problems of network monitoring and consensus prediction. Here, an observer, without exact knowledge of the network, seeks to determine in the shortest possible time the asymptotic agreement value by monitoring a subset of the agents. We uncover a fundamental limit on the minimum required monitoring time for the case of a single observed node, and analyze the case of multiple observed nodes. We provide conditions for achieving the limit in the former case and develop algorithms toward achieving conjectured bounds in the latter through local observation and local computation. Third, we study a distributed optimization problem where a network of agents seeks to minimize the sum of the agents' individual objective functions while each agent may be associated with a separate local constraint. We develop new distributed algorithms for solving this problem. In these algorithms, consensus prediction is employed as a means to achieve fast convergence rates, possibly in finite time. An advantage of our distributed optimization algorithms is that they work under milder assumptions on the network weight matrix than are commonly assumed in the literature. Most distributed algorithms require undirected networks. Consensus-based algorithms can apply to directed networks under an assumption that the network weight matrix is doubly stochastic (i.e., both row stochastic and column stochastic), or in some recent literature only column stochastic. Our algorithms work for directed networks and only require row stochasticity, a mild assumption. Doubly stochastic or column stochastic weight matrices can be hard to arrange locally, especially in broadcast-based communication. We achieve the simplification to the row stochastic assumption through a distributed rescaling technique. Next, we develop a unified convergence analysis of a distributed projected subgradient algorithm and its variation that can be applied to both unconstrained and constrained problems without assuming boundedness or commonality of the local constraint sets

Quantized Consensus by the Alternating Direction Method of Multipliers: Algorithms and Applications

Collaborative in-network processing is a major tenet in the fields of control, signal processing, information theory, and computer science. Agents operating in a coordinated fashion can gain greater efficiency and operational capability than those perform solo missions. In many such applications the central task is to compute the global average of agents\u27 data in a distributed manner. Much recent attention has been devoted to quantized consensus, where, due to practical constraints, only quantized communications are allowed between neighboring nodes in order to achieve the average consensus. This dissertation aims to develop efficient quantized consensus algorithms based on the alternating direction method of multipliers (ADMM) for networked applications, and in particular, consensus based detection in large scale sensor networks. We study the effects of two commonly used uniform quantization schemes, dithered and deterministic quantizations, on an ADMM based distributed averaging algorithm. With dithered quantization, this algorithm yields linear convergence to the desired average in the mean sense with a bounded variance. When deterministic quantization is employed, the distributed ADMM either converges to a consensus or cycles with a finite period after a finite-time iteration. In the cyclic case, local quantized variables have the same sample mean over one period and hence each node can also reach a consensus. We then obtain an upper bound on the consensus error, which depends only on the quantization resolution and the average degree of the network. This is preferred in large scale networks where the range of agents\u27 data and the size of network may be large. Noticing that existing quantized consensus algorithms, including the above two, adopt infinite-bit quantizers unless a bound on agents\u27 data is known a priori, we further develop an ADMM based quantized consensus algorithm using finite-bit bounded quantizers for possibly unbounded agents\u27 data. By picking a small enough ADMM step size, this algorithm can obtain the same consensus result as using the unbounded deterministic quantizer. We then apply this algorithm to distributed detection in connected sensor networks where each node can only exchange information with its direct neighbors. We establish that, with each node employing an identical one-bit quantizer for local information exchange, our approach achieves the optimal asymptotic performance of centralized detection. The statement is true under three different detection frameworks: the Bayesian criterion where the maximum a posteriori detector is optimal, the Neyman-Pearson criterion with a constant type-I error constraint, and the Neyman-Pearson criterion with an exponential type-I error constraint. The key to achieving optimal asymptotic performance is the use of a one-bit deterministic quantizer with controllable threshold that results in desired consensus error bounds

A learning framework for higher-order consistency models in multi-class pixel labeling problems

Recently, higher-order Markov random field (MRF) models have been successfully applied to problems in computer vision, especially scene understanding problems. One successful higher-order MRF model for scene understanding is the consistency model [Kohli and Kumar, 2010; Kohli et al., 2009] and earlier work by Ladicky et al. [2009, 2013] which contain higher-order potentials composed of lower linear envelope functions. In semantic image segmentation problems, which seek to identify the pixels of images with pre-defined labels of objects and backgrounds, this model encourages consistent label assignments over segmented regions of images. However, solving this MRF problem exactly is generally NP-hard; instead, efficient approximate inference algorithms are used. Furthermore, the lower linear envelope functions involve a number of parameters to learn. But, the typical cross-validation used for pairwise MRF models is not a practical method for estimating such a large number of parameters. Nevertheless, few works have proposed efficient learning methods to deal with the large number of parameters in these consistency models. In this thesis, we propose a unified inference and learning framework for the consistency model. We investigate various issues and present solutions for inference and learning with this higher-order MRF model as follows. First, we derive two variants of the consistency model for multi-class pixel labeling tasks. Our model defines an energy function scoring any given label assignments over an image. In order to perform Maximum a posteriori (MAP) inference in this model, we minimize the energy function using move-making algorithms in which the higher-order problems are transformed into tractable pairwise problems. Then, we employ a max-margin framework for learning optimal parameters. This learning framework provides a generalized approach for searching the large parameter space. Second, we propose a novel use of the Gaussian mixture model (GMM) for encoding consistency constraints over a large set of pixels. Here, we use various oversegmentation methods to define coherent regions for the consistency potentials. In general, Mean shift (MS) produces locally coherent regions, and GMM provides globally coherent regions, which do not need to be contiguous. Our model exploits both local and global information together and improves the labeling accuracy on real data sets. Accordingly, we use multiple higher-order terms associated with each over-segmentation method. Our learning framework allows us to deal with the large number of parameters involved with multiple higher-order terms. Next, we explore a dual decomposition (DD) method for our multi-class consistency model. The dual decomposition MRF (DD-MRF) is an alternative method for optimizing the energy function. In dual decomposition, a complex MRF problem is decomposed into many easy subproblems and we optimize the relaxed dual problem using a projected subgradient method. At convergence, we expect a global optimum in the dual space because it is a concave maximization problem. To optimize our higher-order DD-MRF exactly, we propose an exact minimization algorithm for solving the higher-order subproblems. Moreover, the minimization algorithm is much more efficient than graph-cuts. The dual decomposition approach also solves the max-margin learning problem by minimizing the dual losses derived from DD-MRF. Here, our minimization algorithm allows us to optimize the DD learning exactly and efficiently, which in most cases finds better parameters than the previous learning approach. Last, we focus on improving labeling accuracies of our higher-order model by combining mid-level features, which we call region features. The region features help customize the general envelope functions for individual segmented regions. By assigning specified weights to the envelope functions, we can choose subsets of highly likely labels for each segmented region. We train multiple classifiers with region features and aggregate them to increase prediction performance of possible labels for each region. Importantly, introducing these region features does not change the previous inference and learning algorithms

Implementacija i analiza klase algoritama za distribuiranu konveksnuoptimizaciju: Evaluacija performansi i osobina na praktičnim HPCklasterima

The significance of distributed optimization algorithms manifests through growing interest in various application domains. It finds its use in Big Data analytics, distributed machine learning, distributed control, vehicle networks and smart grid, inter alia. This thesis focuses on primal and dual distributed convex optimization methods. First, it introduces a general algorithmic framework of first and second order methods of primal type, that utilize the concepts of sparsified communications and computations across a connected graph of working nodes. Besides several already existing methods, the framework also includes novel variants that utilize unidirectional communication. Although there have been a remarkable amount of theory and theoretical advances in this field, practical evaluations of methods on real data and practical large scale and High Performance Computing (HPC) systems are of much smaller volume. Therefore, we developed the implementations and performed various numerical evaluations of the proposed methods in an actual, parallel programming environment. The implementations were developed using the Message Passing Interface (MPI) and tested on a High Performance Computing cluster. These empirical evaluations result with very useful insights and guidelines regarding performance and highlights the most important communication-computational tradeoffs in a real execution environment. As there exists a wide set of machine learning algorithms that can be viewed as optimization problems, distributed optimization plays a significant role in this area. The thesis also presents an algorithm for convex clustering, based on the dual method Alternating Direction Method of Multipliers (ADMM), that relies on COMPS Superscalar (COMPSs) framework for parallelization. We provide results of extensive numerical evaluations of the algorithm on a HPC cluster environment, to demonstrate the high degree of scalability and efficiency of the method, compared to existing alternative solvers for convex clustering. The program code for the developed algorithms is open-source and available in the corresponding repositories.спарсификоване комуникације и израчунавања преко повезаног графа чворова. Поред неколико већ постојећих метода, појављују се и нове варијанте које користе једносмерну комуникацију. Иако у овој области постоји изузетно велика количина теорије и теоријског напретка, практичне евалуације метода над стварним подацима и практичним системима рачунара вискох перформанси (High Performance Computing — HPC) великих размера, су много мањег обима. Стога смо развили имплементације и извршили скуп различитих нумеричких евалуација предложених метода у стварном, паралелном програмском окружењу. Имплементације су развијене коришћењем технологије Message Passing Interface (MPI) и тестиране су на рачунарском кластеру високих перформанси. Ове емпиријске процене резултирају веома корисним увидима и смерницама у вези са перформансама и наглашавају најважније компромисе између комуникације и израчунавања, у стварном окружењу извршавања. С обзиром на постојање широког скупа алгоритама машинског учења који се могу посматрати као оптимизациони проблеми, дистрибуирана оптимизација има врло значајну улогу у овој области. У тези је такође представљен и алгоритам за конвексно кластеровање, заснован на дуалној методи Alternating Direction Method of Multipliers (ADMM), која се ослања на COMPS Superscalar (COMPSs) приступ за паралелизацију. Приказујемо резултате опсежних нумеричких евалуација алгоритма на HPC рачунарском кластеру, како бисмо демонстрирали висок степен скалабилности и ефикасности методе, у поређењу са постојећим алтернативним приступима за конвексно кластеровање. Програмски код за развијене алгоритме је софтвер отвореног кода, и доступан је у одговарајућим репозиторијумима. sparsifikovane komunikacije i izračunavanja preko povezanog grafa čvorova. Pored nekoliko već postojećih metoda, pojavljuju se i nove varijante koje koriste jednosmernu komunikaciju. Iako u ovoj oblasti postoji izuzetno velika količina teorije i teorijskog napretka, praktične evaluacije metoda nad stvarnim podacima i praktičnim sistemima računara viskoh performansi (High Performance Computing — HPC) velikih razmera, su mnogo manjeg obima. Stoga smo razvili implementacije i izvršili skup različitih numeričkih evaluacija predloženih metoda u stvarnom, paralelnom programskom okruženju. Implementacije su razvijene korišćenjem tehnologije Message Passing Interface (MPI) i testirane su na računarskom klasteru visokih performansi. Ove empirijske procene rezultiraju veoma korisnim uvidima i smernicama u vezi sa performansama i naglašavaju najvažnije kompromise između komunikacije i izračunavanja, u stvarnom okruženju izvršavanja. S obzirom na postojanje širokog skupa algoritama mašinskog učenja koji se mogu posmatrati kao optimizacioni problemi, distribuirana optimizacija ima vrlo značajnu ulogu u ovoj oblasti. U tezi je takođe predstavljen i algoritam za konveksno klasterovanje, zasnovan na dualnoj metodi Alternating Direction Method of Multipliers (ADMM), koja se oslanja na COMPS Superscalar (COMPSs) pristup za paralelizaciju. Prikazujemo rezultate opsežnih numeričkih evaluacija algoritma na HPC računarskom klasteru, kako bismo demonstrirali visok stepen skalabilnosti i efikasnosti metode, u poređenju sa postojećim alternativnim pristupima za konveksno klasterovanje. Programski kod za razvijene algoritme je softver otvorenog koda, i dostupan je u odgovarajućim repozitorijumima.

Conditional Gradient Methods

The purpose of this survey is to serve both as a gentle introduction and a coherent overview of state-of-the-art Frank--Wolfe algorithms, also called conditional gradient algorithms, for function minimization. These algorithms are especially useful in convex optimization when linear optimization is cheaper than projections. The selection of the material has been guided by the principle of highlighting crucial ideas as well as presenting new approaches that we believe might become important in the future, with ample citations even of old works imperative in the development of newer methods. Yet, our selection is sometimes biased, and need not reflect consensus of the research community, and we have certainly missed recent important contributions. After all the research area of Frank--Wolfe is very active, making it a moving target. We apologize sincerely in advance for any such distortions and we fully acknowledge: We stand on the shoulder of giants.Comment: 238 pages with many figures. The FrankWolfe.jl Julia package (https://github.com/ZIB-IOL/FrankWolfe.jl) providces state-of-the-art implementations of many Frank--Wolfe method