115 research outputs found
Decentralized Dictionary Learning Over Time-Varying Digraphs
This paper studies Dictionary Learning problems wherein the learning task is
distributed over a multi-agent network, modeled as a time-varying directed
graph. This formulation is relevant, for instance, in Big Data scenarios where
massive amounts of data are collected/stored in different locations (e.g.,
sensors, clouds) and aggregating and/or processing all data in a fusion center
might be inefficient or unfeasible, due to resource limitations, communication
overheads or privacy issues. We develop a unified decentralized algorithmic
framework for this class of nonconvex problems, which is proved to converge to
stationary solutions at a sublinear rate. The new method hinges on Successive
Convex Approximation techniques, coupled with a decentralized tracking
mechanism aiming at locally estimating the gradient of the smooth part of the
sum-utility. To the best of our knowledge, this is the first provably
convergent decentralized algorithm for Dictionary Learning and, more generally,
bi-convex problems over (time-varying) (di)graphs
Distributed Nonconvex Multiagent Optimization Over Time-Varying Networks
We study nonconvex distributed optimization in multiagent networks where the
communications between nodes is modeled as a time-varying sequence of arbitrary
digraphs. We introduce a novel broadcast-based distributed algorithmic
framework for the (constrained) minimization of the sum of a smooth (possibly
nonconvex and nonseparable) function, i.e., the agents' sum-utility, plus a
convex (possibly nonsmooth and nonseparable) regularizer. The latter is usually
employed to enforce some structure in the solution, typically sparsity. The
proposed method hinges on Successive Convex Approximation (SCA) techniques
coupled with i) a tracking mechanism instrumental to locally estimate the
gradients of agents' cost functions; and ii) a novel broadcast protocol to
disseminate information and distribute the computation among the agents.
Asymptotic convergence to stationary solutions is established. A key feature of
the proposed algorithm is that it neither requires the double-stochasticity of
the consensus matrices (but only column stochasticity) nor the knowledge of the
graph sequence to implement. To the best of our knowledge, the proposed
framework is the first broadcast-based distributed algorithm for convex and
nonconvex constrained optimization over arbitrary, time-varying digraphs.
Numerical results show that our algorithm outperforms current schemes on both
convex and nonconvex problems.Comment: Copyright 2001 SS&C. Published in the Proceedings of the 50th annual
Asilomar conference on signals, systems, and computers, Nov. 6-9, 2016, CA,
US
FROST -- Fast row-stochastic optimization with uncoordinated step-sizes
In this paper, we discuss distributed optimization over directed graphs,
where doubly-stochastic weights cannot be constructed. Most of the existing
algorithms overcome this issue by applying push-sum consensus, which utilizes
column-stochastic weights. The formulation of column-stochastic weights
requires each agent to know (at least) its out-degree, which may be impractical
in e.g., broadcast-based communication protocols. In contrast, we describe
FROST (Fast Row-stochastic-Optimization with uncoordinated STep-sizes), an
optimization algorithm applicable to directed graphs that does not require the
knowledge of out-degrees; the implementation of which is straightforward as
each agent locally assigns weights to the incoming information and locally
chooses a suitable step-size. We show that FROST converges linearly to the
optimal solution for smooth and strongly-convex functions given that the
largest step-size is positive and sufficiently small.Comment: Submitted for journal publication, currently under revie
Decentralized Complete Dictionary Learning via -Norm Maximization
With the rapid development of information technologies, centralized data
processing is subject to many limitations, such as computational overheads,
communication delays, and data privacy leakage. Decentralized data processing
over networked terminal nodes becomes an important technology in the era of big
data. Dictionary learning is a powerful representation learning method to
exploit the low-dimensional structure from the high-dimensional data. By
exploiting the low-dimensional structure, the storage and the processing
overhead of data can be effectively reduced. In this paper, we propose a novel
decentralized complete dictionary learning algorithm, which is based on
-norm maximization. Compared with existing decentralized dictionary
learning algorithms, comprehensive numerical experiments show that the novel
algorithm has significant advantages in terms of per-iteration computational
complexity, communication cost, and convergence rate in many scenarios.
Moreover, a rigorous theoretical analysis shows that the dictionaries learned
by the proposed algorithm can converge to the one learned by a centralized
dictionary learning algorithm at a linear rate with high probability under
certain conditions
Distributed Online Optimization via Gradient Tracking with Adaptive Momentum
This paper deals with a network of computing agents aiming to solve an online
optimization problem in a distributed fashion, i.e., by means of local
computation and communication, without any central coordinator. We propose the
gradient tracking with adaptive momentum estimation (GTAdam) distributed
algorithm, which combines a gradient tracking mechanism with first and second
order momentum estimates of the gradient. The algorithm is analyzed in the
online setting for strongly convex and smooth cost functions. We prove that the
average dynamic regret is bounded and that the convergence rate is linear. The
algorithm is tested on a time-varying classification problem, on a (moving)
target localization problem and in a stochastic optimization setup from image
classification. In these numerical experiments from multi-agent learning,
GTAdam outperforms state-of-the-art distributed optimization methods
- …