23 research outputs found
Multi-center federated learning: clients clustering for better personalization
Personalized decision-making can be implemented in a Federated learning (FL) framework that can collaboratively train a decision model by extracting knowledge across intelligent clients, e.g. smartphones or enterprises. FL can mitigate the data privacy risk of collaborative training since it merely collects local gradients from users without access to their data. However, FL is fragile in the presence of statistical heterogeneity that is commonly encountered in personalized decision making, e.g., non-IID data over different clients. Existing FL approaches usually update a single global model to capture the shared knowledge of all users by aggregating their gradients, regardless of the discrepancy between their data distributions. By comparison, a mixture of multiple global models could capture the heterogeneity across various clients if assigning the client to different global models (i.e., centers) in FL. To this end, we propose a novel multi-center aggregation mechanism to cluster clients using their models’ parameters. It learns multiple global models from data as the cluster centers, and simultaneously derives the optimal matching between users and centers. We then formulate it as an optimization problem that can be efficiently solved by a stochastic expectation maximization (EM) algorithm. Experiments on multiple benchmark datasets of FL show that our method outperforms several popular baseline methods. The experimental source codes are publicly available on the Github repository (GitHub repository: https://github.com/mingxuts/multi-center-fed-learning)
Federated Learning with Intermediate Representation Regularization
In contrast to centralized model training that involves data collection,
federated learning (FL) enables remote clients to collaboratively train a model
without exposing their private data. However, model performance usually
degrades in FL due to the heterogeneous data generated by clients of diverse
characteristics. One promising strategy to maintain good performance is by
limiting the local training from drifting far away from the global model.
Previous studies accomplish this by regularizing the distance between the
representations learned by the local and global models. However, they only
consider representations from the early layers of a model or the layer
preceding the output layer. In this study, we introduce FedIntR, which provides
a more fine-grained regularization by integrating the representations of
intermediate layers into the local training process. Specifically, FedIntR
computes a regularization term that encourages the closeness between the
intermediate layer representations of the local and global models.
Additionally, FedIntR automatically determines the contribution of each layer's
representation to the regularization term based on the similarity between local
and global representations. We conduct extensive experiments on various
datasets to show that FedIntR can achieve equivalent or higher performance
compared to the state-of-the-art approaches. Our code is available at
https://github.com/YLTun/FedIntR.Comment: IEEE BigComp 202
Dynamic Heterogeneous Federated Learning with Multi-Level Prototypes
Federated learning shows promise as a privacy-preserving collaborative
learning technique. Existing heterogeneous federated learning mainly focuses on
skewing the label distribution across clients. However, most approaches suffer
from catastrophic forgetting and concept drift, mainly when the global
distribution of all classes is extremely unbalanced and the data distribution
of the client dynamically evolves over time. In this paper, we study the new
task, i.e., Dynamic Heterogeneous Federated Learning (DHFL), which addresses
the practical scenario where heterogeneous data distributions exist among
different clients and dynamic tasks within the client. Accordingly, we propose
a novel federated learning framework named Federated Multi-Level Prototypes
(FedMLP) and design federated multi-level regularizations. To mitigate concept
drift, we construct prototypes and semantic prototypes to provide fruitful
generalization knowledge and ensure the continuity of prototype spaces. To
maintain the model stability and consistency of convergence, three
regularizations are introduced as training losses, i.e., prototype-based
regularization, semantic prototype-based regularization, and federated
inter-task regularization. Extensive experiments show that the proposed method
achieves state-of-the-art performance in various settings
REPA: Client Clustering without Training and Data Labels for Improved Federated Learning in Non-IID Settings
Clustering clients into groups that exhibit relatively homogeneous data
distributions represents one of the major means of improving the performance of
federated learning (FL) in non-independent and identically distributed
(non-IID) data settings. Yet, the applicability of current state-of-the-art
approaches remains limited as these approaches cluster clients based on
information, such as the evolution of local model parameters, that is only
obtainable through actual on-client training. On the other hand, there is a
need to make FL models available to clients who are not able to perform the
training themselves, as they do not have the processing capabilities required
for training, or simply want to use the model without participating in the
training. Furthermore, the existing alternative approaches that avert the
training still require that individual clients have a sufficient amount of
labeled data upon which the clustering is based, essentially assuming that each
client is a data annotator. In this paper, we present REPA, an approach to
client clustering in non-IID FL settings that requires neither training nor
labeled data collection. REPA uses a novel supervised autoencoder-based method
to create embeddings that profile a client's underlying data-generating
processes without exposing the data to the server and without requiring local
training. Our experimental analysis over three different datasets demonstrates
that REPA delivers state-of-the-art model performance while expanding the
applicability of cluster-based FL to previously uncovered use cases
Speeding up Heterogeneous Federated Learning with Sequentially Trained Superclients
Federated Learning (FL) allows training machine learning models in
privacy-constrained scenarios by enabling the cooperation of edge devices
without requiring local data sharing. This approach raises several challenges
due to the different statistical distribution of the local datasets and the
clients' computational heterogeneity. In particular, the presence of highly
non-i.i.d. data severely impairs both the performance of the trained neural
network and its convergence rate, increasing the number of communication rounds
requested to reach a performance comparable to that of the centralized
scenario. As a solution, we propose FedSeq, a novel framework leveraging the
sequential training of subgroups of heterogeneous clients, i.e. superclients,
to emulate the centralized paradigm in a privacy-compliant way. Given a fixed
budget of communication rounds, we show that FedSeq outperforms or match
several state-of-the-art federated algorithms in terms of final performance and
speed of convergence. Finally, our method can be easily integrated with other
approaches available in the literature. Empirical results show that combining
existing algorithms with FedSeq further improves its final performance and
convergence speed. We test our method on CIFAR-10 and CIFAR-100 and prove its
effectiveness in both i.i.d. and non-i.i.d. scenarios.Comment: Published at the 26th International Conference on Pattern Recognition
(ICPR), 2022, pp. 3376-338
Cluster-driven Graph Federated Learning over Multiple Domains
Federated Learning (FL) deals with learning a central model (i.e. the server)
in privacy-constrained scenarios, where data are stored on multiple devices
(i.e. the clients). The central model has no direct access to the data, but
only to the updates of the parameters computed locally by each client. This
raises a problem, known as statistical heterogeneity, because the clients may
have different data distributions (i.e. domains). This is only partly
alleviated by clustering the clients. Clustering may reduce heterogeneity by
identifying the domains, but it deprives each cluster model of the data and
supervision of others. Here we propose a novel Cluster-driven Graph Federated
Learning (FedCG). In FedCG, clustering serves to address statistical
heterogeneity, while Graph Convolutional Networks (GCNs) enable sharing
knowledge across them. FedCG: i) identifies the domains via an FL-compliant
clustering and instantiates domain-specific modules (residual branches) for
each domain; ii) connects the domain-specific modules through a GCN at training
to learn the interactions among domains and share knowledge; and iii) learns to
cluster unsupervised via teacher-student classifier-training iterations and to
address novel unseen test domains via their domain soft-assignment scores.
Thanks to the unique interplay of GCN over clusters, FedCG achieves the
state-of-the-art on multiple FL benchmarks
When Federated Recommendation Meets Cold-Start Problem: Separating Item Attributes and User Interactions
Federated recommendation system usually trains a global model on the server
without direct access to users' private data on their own devices. However,
this separation of the recommendation model and users' private data poses a
challenge in providing quality service, particularly when it comes to new
items, namely cold-start recommendations in federated settings. This paper
introduces a novel method called Item-aligned Federated Aggregation (IFedRec)
to address this challenge. It is the first research work in federated
recommendation to specifically study the cold-start scenario. The proposed
method learns two sets of item representations by leveraging item attributes
and interaction records simultaneously. Additionally, an item representation
alignment mechanism is designed to align two item representations and learn the
meta attribute network at the server within a federated learning framework.
Experiments on four benchmark datasets demonstrate IFedRec's superior
performance for cold-start scenarios. Furthermore, we also verify IFedRec owns
good robustness when the system faces limited client participation and noise
injection, which brings promising practical application potential in
privacy-protection enhanced federated recommendation systems. The
implementation code is availableComment: Accepted as a regular paper of WWW'2