15,390 research outputs found
ReBoot: Distributed statistical learning via refitting Bootstrap samples
In this paper, we study a one-shot distributed learning algorithm via
refitting Bootstrap samples, which we refer to as ReBoot. Given the local
models that are fit on multiple independent subsamples, ReBoot refits a new
model on the union of the Bootstrap samples drawn from these local models. The
whole procedure requires only one round of communication of model parameters.
Theoretically, we analyze the statistical rate of ReBoot for generalized linear
models (GLM) and noisy phase retrieval, which represent convex and non-convex
problems respectively. In both cases, ReBoot provably achieves the full-sample
statistical rate whenever the subsample size is not too small. In particular,
we show that the systematic bias of ReBoot, the error that is independent of
the number of subsamples, is in GLM, where n is the subsample
size. This rate is sharper than that of model parameter averaging and its
variants, implying the higher tolerance of ReBoot with respect to data splits
to maintain the full-sample rate. Simulation study exhibits the statistical
advantage of ReBoot over competing methods including averaging and CSL
(Communication-efficient Surrogate Likelihood) with up to two rounds of
gradient communication. Finally, we propose FedReBoot, an iterative version of
ReBoot, to aggregate convolutional neural networks for image classification,
which exhibits substantial superiority over FedAve within early rounds of
communication
CoCoA: A General Framework for Communication-Efficient Distributed Optimization
The scale of modern datasets necessitates the development of efficient
distributed optimization methods for machine learning. We present a
general-purpose framework for distributed computing environments, CoCoA, that
has an efficient communication scheme and is applicable to a wide variety of
problems in machine learning and signal processing. We extend the framework to
cover general non-strongly-convex regularizers, including L1-regularized
problems like lasso, sparse logistic regression, and elastic net
regularization, and show how earlier work can be derived as a special case. We
provide convergence guarantees for the class of convex regularized loss
minimization objectives, leveraging a novel approach in handling
non-strongly-convex regularizers and non-smooth loss functions. The resulting
framework has markedly improved performance over state-of-the-art methods, as
we illustrate with an extensive set of experiments on real distributed
datasets
L1-Regularized Distributed Optimization: A Communication-Efficient Primal-Dual Framework
Despite the importance of sparsity in many large-scale applications, there
are few methods for distributed optimization of sparsity-inducing objectives.
In this paper, we present a communication-efficient framework for
L1-regularized optimization in the distributed environment. By viewing
classical objectives in a more general primal-dual setting, we develop a new
class of methods that can be efficiently distributed and applied to common
sparsity-inducing models, such as Lasso, sparse logistic regression, and
elastic net-regularized problems. We provide theoretical convergence guarantees
for our framework, and demonstrate its efficiency and flexibility with a
thorough experimental comparison on Amazon EC2. Our proposed framework yields
speedups of up to 50x as compared to current state-of-the-art methods for
distributed L1-regularized optimization
Accelerating federated learning via momentum gradient descent
Federated learning (FL) provides a communication-efficient approach to solve machine learning problems concerning distributed data, without sending raw data to a central server. However, existing works on FL only utilize first-order gradient descent (GD) and do not consider the preceding iterations to gradient update which can potentially accelerate convergence. In this article, we consider momentum term which relates to the last iteration. The proposed momentum federated learning (MFL) uses momentum gradient descent (MGD) in the local update step of FL system. We establish global convergence properties of MFL and derive an upper bound on MFL convergence rate. Comparing the upper bounds on MFL and FL convergence rates, we provide conditions in which MFL accelerates the convergence. For different machine learning models, the convergence performance of MFL is evaluated based on experiments with MNIST and CIFAR-10 datasets. Simulation results confirm that MFL is globally convergent and further reveal significant convergence improvement over FL
- …