517 research outputs found
Asynchronous Distributed ADMM for Large-Scale Optimization- Part I: Algorithm and Convergence Analysis
Aiming at solving large-scale learning problems, this paper studies
distributed optimization methods based on the alternating direction method of
multipliers (ADMM). By formulating the learning problem as a consensus problem,
the ADMM can be used to solve the consensus problem in a fully parallel fashion
over a computer network with a star topology. However, traditional synchronized
computation does not scale well with the problem size, as the speed of the
algorithm is limited by the slowest workers. This is particularly true in a
heterogeneous network where the computing nodes experience different
computation and communication delays. In this paper, we propose an asynchronous
distributed ADMM (AD-AMM) which can effectively improve the time efficiency of
distributed optimization. Our main interest lies in analyzing the convergence
conditions of the AD-ADMM, under the popular partially asynchronous model,
which is defined based on a maximum tolerable delay of the network.
Specifically, by considering general and possibly non-convex cost functions, we
show that the AD-ADMM is guaranteed to converge to the set of
Karush-Kuhn-Tucker (KKT) points as long as the algorithm parameters are chosen
appropriately according to the network delay. We further illustrate that the
asynchrony of the ADMM has to be handled with care, as slightly modifying the
implementation of the AD-ADMM can jeopardize the algorithm convergence, even
under a standard convex setting.Comment: 37 page
Distributed Quantile Regression Analysis and a Group Variable Selection Method
This dissertation develops novel methodologies for distributed quantile regression analysis
for big data by utilizing a distributed optimization algorithm called the alternating direction
method of multipliers (ADMM). Specifically, we first write the penalized quantile regression
into a specific form that can be solved by the ADMM and propose numerical algorithms
for solving the ADMM subproblems. This results in the distributed QR-ADMM
algorithm. Then, to further reduce the computational time, we formulate the penalized
quantile regression into another equivalent ADMM form in which all the subproblems have
exact closed-form solutions and hence avoid iterative numerical methods. This results in the
single-loop QPADM algorithm that further improve on the computational efficiency of the
QR-ADMM. Both QR-ADMM and QPADM enjoy flexible parallelization by enabling data
splitting across both sample space and feature space, which make them especially appealing
for the case when both sample size n and feature dimension p are large.
Besides the QR-ADMM and QPADM algorithms for penalized quantile regression, we
also develop a group variable selection method by approximating the Bayesian information
criterion. Unlike existing penalization methods for feature selection, our proposed gMIC
algorithm is free of parameter tuning and hence enjoys greater computational efficiency.
Although the current version of gMIC focuses on the generalized linear model, it can be
naturally extended to the quantile regression for feature selection.
We provide theoretical analysis for our proposed methods. Specifically, we conduct numerical
convergence analysis for the QR-ADMM and QPADM algorithms, and provide
asymptotical theories and oracle property of feature selection for the gMIC method. All
our methods are evaluated with simulation studies and real data analysis
- …