650 research outputs found
Personalized Graph Federated Learning with Differential Privacy
This paper presents a personalized graph federated learning (PGFL) framework
in which distributedly connected servers and their respective edge devices
collaboratively learn device or cluster-specific models while maintaining the
privacy of every individual device. The proposed approach exploits similarities
among different models to provide a more relevant experience for each device,
even in situations with diverse data distributions and disproportionate
datasets. Furthermore, to ensure a secure and efficient approach to
collaborative personalized learning, we study a variant of the PGFL
implementation that utilizes differential privacy, specifically
zero-concentrated differential privacy, where a noise sequence perturbs model
exchanges. Our mathematical analysis shows that the proposed privacy-preserving
PGFL algorithm converges to the optimal cluster-specific solution for each
cluster in linear time. It also shows that exploiting similarities among
clusters leads to an alternative output whose distance to the original solution
is bounded, and that this bound can be adjusted by modifying the algorithm's
hyperparameters. Further, our analysis shows that the algorithm ensures local
differential privacy for all clients in terms of zero-concentrated differential
privacy. Finally, the performance of the proposed PGFL algorithm is examined by
performing numerical experiments in the context of regression and
classification using synthetic data and the MNIST dataset
Multitask Online Mirror Descent
We introduce and analyze MT-OMD, a multitask generalization of Online Mirror
Descent (OMD) which operates by sharing updates between tasks. We prove that
the regret of MT-OMD is of order , where
is the task variance according to the geometry induced by the
regularizer, is the number of tasks, and is the time horizon. Whenever
tasks are similar, that is , our method improves upon the
bound obtained by running independent OMDs on each task. We further
provide a matching lower bound, and show that our multitask extensions of
Online Gradient Descent and Exponentiated Gradient, two major instances of OMD,
enjoy closed-form updates, making them easy to use in practice. Finally, we
present experiments on both synthetic and real-world datasets supporting our
findings
ColD Fusion: Collaborative Descent for Distributed Multitask Finetuning
We propose a new paradigm to continually evolve pretrained models, denoted
ColD Fusion. It provides the benefits of multitask learning but leverages
distributed computation with limited communication and eliminates the need for
shared data. Consequentially, ColD Fusion can give rise to a synergistic loop,
where finetuned models can be recycled to continually improve the pretrained
model they are based upon. We show that ColD Fusion yields comparable benefits
to multitask training by producing a model that (a) attains strong performance
on all of the datasets it was trained on; and (b) is a better starting point
for finetuning on unseen datasets. We show that ColD Fusion outperforms RoBERTa
and even previous multitask models. Specifically, when training and testing on
35 diverse datasets, ColD Fusion-based model outperforms RoBERTa by 2.33 points
on average without any changes to the architecture.Comment: ACL 2
Tracking Performance of Online Stochastic Learners
The utilization of online stochastic algorithms is popular in large-scale
learning settings due to their ability to compute updates on the fly, without
the need to store and process data in large batches. When a constant step-size
is used, these algorithms also have the ability to adapt to drifts in problem
parameters, such as data or model properties, and track the optimal solution
with reasonable accuracy. Building on analogies with the study of adaptive
filters, we establish a link between steady-state performance derived under
stationarity assumptions and the tracking performance of online learners under
random walk models. The link allows us to infer the tracking performance from
steady-state expressions directly and almost by inspection
- …