638 research outputs found
Distributed Dictionary Learning
The paper studies distributed Dictionary Learning (DL) problems where the
learning task is distributed over a multi-agent network with time-varying
(nonsymmetric) connectivity. This formulation is relevant, for instance, in
big-data scenarios where massive amounts of data are collected/stored in
different spatial locations and it is unfeasible to aggregate and/or process
all the data in a fusion center, due to resource limitations, communication
overhead or privacy considerations. We develop a general distributed
algorithmic framework for the (nonconvex) DL problem and establish its
asymptotic convergence. The new method hinges on Successive Convex
Approximation (SCA) techniques coupled with i) a gradient tracking mechanism
instrumental to locally estimate the missing global information; and ii) a
consensus step, as a mechanism to distribute the computations among the agents.
To the best of our knowledge, this is the first distributed algorithm with
provable convergence for the DL problem and, more in general, bi-convex
optimization problems over (time-varying) directed graphs
Fast ADMM Algorithm for Distributed Optimization with Adaptive Penalty
We propose new methods to speed up convergence of the Alternating Direction
Method of Multipliers (ADMM), a common optimization tool in the context of
large scale and distributed learning. The proposed method accelerates the speed
of convergence by automatically deciding the constraint penalty needed for
parameter consensus in each iteration. In addition, we also propose an
extension of the method that adaptively determines the maximum number of
iterations to update the penalty. We show that this approach effectively leads
to an adaptive, dynamic network topology underlying the distributed
optimization. The utility of the new penalty update schemes is demonstrated on
both synthetic and real data, including a computer vision application of
distributed structure from motion.Comment: 8 pages manuscript, 2 pages appendix, 5 figure
On the Convergence of Decentralized Gradient Descent
Consider the consensus problem of minimizing where
each is only known to one individual agent out of a connected network
of agents. All the agents shall collaboratively solve this problem and
obtain the solution subject to data exchanges restricted to between neighboring
agents. Such algorithms avoid the need of a fusion center, offer better network
load balance, and improve data privacy. We study the decentralized gradient
descent method in which each agent updates its variable , which is
a local approximate to the unknown variable , by combining the average of
its neighbors' with the negative gradient step .
The iteration is where the averaging coefficients form a symmetric doubly stochastic matrix
. We analyze the convergence of this
iteration and derive its converge rate, assuming that each is proper
closed convex and lower bounded, is Lipschitz continuous with
constant , and stepsize is fixed. Provided that where , the objective error at the averaged
solution, , reduces at a speed of
until it reaches . If are further (restricted) strongly
convex, then both and each converge
to the global minimizer at a linear rate until reaching an
-neighborhood of . We also develop an iteration for
decentralized basis pursuit and establish its linear convergence to an
-neighborhood of the true unknown sparse signal
Recommended from our members
Simultaneous Bayesian Sparse Approximation with Structured Sparse Models
Sparse approximation is key to many signal processing, image processing and machine learning applications. If multiple signals maintain some degree of dependency, for example the support sets are statistically related, then it will generally be advantageous to jointly estimate the sparse representation vectors from the measurements vectors as opposed to solving for each signal individually. In this paper, we propose simultaneous sparse Bayesian learning (SBL) for joint sparse approximation with two structured sparse models (SSMs), where one is row-sparse with embedded element-sparse, and the other one is row-sparse plus element-sparse. While SBL has attracted much attention as a means to deal with a single sparse approximation problem, it is not obvious how to extend SBL to SSMs. By capitalizing on a dual-space view of existing convex methods for SMs, we showcase the precision component model and covariance component model for SSMs, where both models involve a common hyperparameter and an innovation hyperparameter that together control the prior variance for each coefficient. The statistical perspective of precision component vs. covariance component models unfolds the intrinsic mechanism in SSMs, and also leads to our development of SBL-inspired cost functions for SSMs. Centralized algorithms, that include ℓ1 and ℓ2 reweighting algorithms, and consensus based decentralized algorithms are developed for simultaneous sparse approximation with SSMs. In addition, theoretical analysis is conducted to provide valuable insights into the proposed approach, which includes global minima analysis of the SBLinspired nonconvex cost functions and convergence analysis of the proposed ℓ1 reweighting algorithms for SSMs. Superior performance of the proposed algorithms is demonstrated by numerical experiments.This is the author accepted manuscript. The final version is available from IEEE at http://dx.doi.org/10.1109/TSP.2016.2605067
New and Provable Results for Network Inference Problems and Multi-agent Optimization Algorithms
abstract: Our ability to understand networks is important to many applications, from the analysis and modeling of biological networks to analyzing social networks. Unveiling network dynamics allows us to make predictions and decisions. Moreover, network dynamics models have inspired new ideas for computational methods involving multi-agent cooperation, offering effective solutions for optimization tasks. This dissertation presents new theoretical results on network inference and multi-agent optimization, split into two parts -
The first part deals with modeling and identification of network dynamics. I study two types of network dynamics arising from social and gene networks. Based on the network dynamics, the proposed network identification method works like a `network RADAR', meaning that interaction strengths between agents are inferred by injecting `signal' into the network and observing the resultant reverberation. In social networks, this is accomplished by stubborn agents whose opinions do not change throughout a discussion. In gene networks, genes are suppressed to create desired perturbations. The steady-states under these perturbations are characterized. In contrast to the common assumption of full rank input, I take a laxer assumption where low-rank input is used, to better model the empirical network data. Importantly, a network is proven to be identifiable from low rank data of rank that grows proportional to the network's sparsity. The proposed method is applied to synthetic and empirical data, and is shown to offer superior performance compared to prior work. The second part is concerned with algorithms on networks. I develop three consensus-based algorithms for multi-agent optimization. The first method is a decentralized Frank-Wolfe (DeFW) algorithm. The main advantage of DeFW lies on its projection-free nature, where we can replace the costly projection step in traditional algorithms by a low-cost linear optimization step. I prove the convergence rates of DeFW for convex and non-convex problems. I also develop two consensus-based alternating optimization algorithms --- one for least square problems and one for non-convex problems. These algorithms exploit the problem structure for faster convergence and their efficacy is demonstrated by numerical simulations.
I conclude this dissertation by describing future research directions.Dissertation/ThesisDoctoral Dissertation Electrical Engineering 201
- …