4,350 research outputs found
A Unified Contraction Analysis of a Class of Distributed Algorithms for Composite Optimization
We study distributed composite optimization over networks: agents minimize
the sum of a smooth (strongly) convex function, the agents' sum-utility, plus a
non-smooth (extended-valued) convex one. We propose a general algorithmic
framework for such a class of problems and provide a unified convergence
analysis leveraging the theory of operator splitting. Our results unify several
approaches proposed in the literature of distributed optimization for special
instances of our formulation. Distinguishing features of our scheme are: (i)
when the agents' functions are strongly convex, the algorithm converges at a
linear rate, whose dependencies on the agents' functions and the network
topology are decoupled, matching the typical rates of centralized optimization;
(ii) the step-size does not depend on the network parameters but only on the
optimization ones; and (iii) the algorithm can adjust the ratio between the
number of communications and computations to achieve the same rate of the
centralized proximal gradient scheme (in terms of computations). This is the
first time that a distributed algorithm applicable to composite optimization
enjoys such properties.Comment: To appear in the Proc. of the 2019 IEEE International Workshop on
Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP 19
Implicit Fixed-point Proximity Framework for Optimization Problems and Its Applications
A variety of optimization problems especially in the field of image processing are not differentiable in nature. The non-differentiability of the objective functions together with the large dimension of the underlying images makes minimizing the objective function theoretically challenging and numerically difficult. The fixed-point proximity framework that we will systematically study in this dissertation provides a direct and unified methodology for finding solutions to those optimization problems. The framework approaches the models arising from applications straightforwardly by using various fixed point techniques as well as convex analysis tools such as the subdifferential and proximity operator.
With the notion of proximity operator, we can convert those optimization problems into finding fixed points of nonlinear operators. Under the fixed-point proximity framework, these fixed point problems are often solved through iterative schemes in which each iteration can be computed in an explicit form. We further explore this fixed point formulation, and develop implicit iterative schemes for finding fixed points of nonlinear operators associated with the underlying problems, with the goal of relaxing restrictions in the development of solving the fixed point equations. Theoretical analysis is provided for the convergence of implicit algorithms proposed under the framework. The numerical experiments on image reconstruction models demonstrate that the proposed implicit fixed-point proximity algorithms work well in comparison with existing explicit fixed-point proximity algorithms in terms of the consumed computational time and accuracy of the solutions
High-dimensional regression with noisy and missing data: Provable guarantees with nonconvexity
Although the standard formulations of prediction problems involve
fully-observed and noiseless data drawn in an i.i.d. manner, many applications
involve noisy and/or missing data, possibly involving dependence, as well. We
study these issues in the context of high-dimensional sparse linear regression,
and propose novel estimators for the cases of noisy, missing and/or dependent
data. Many standard approaches to noisy or missing data, such as those using
the EM algorithm, lead to optimization problems that are inherently nonconvex,
and it is difficult to establish theoretical guarantees on practical
algorithms. While our approach also involves optimizing nonconvex programs, we
are able to both analyze the statistical error associated with any global
optimum, and more surprisingly, to prove that a simple algorithm based on
projected gradient descent will converge in polynomial time to a small
neighborhood of the set of all global minimizers. On the statistical side, we
provide nonasymptotic bounds that hold with high probability for the cases of
noisy, missing and/or dependent data. On the computational side, we prove that
under the same types of conditions required for statistical consistency, the
projected gradient descent algorithm is guaranteed to converge at a geometric
rate to a near-global minimizer. We illustrate these theoretical predictions
with simulations, showing close agreement with the predicted scalings.Comment: Published in at http://dx.doi.org/10.1214/12-AOS1018 the Annals of
Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical
Statistics (http://www.imstat.org
- …