4 research outputs found
Contrastive encoder pre-training-based clustered federated learning for heterogeneous data
Federated learning (FL) is a promising approach that enables distributed
clients to collaboratively train a global model while preserving their data
privacy. However, FL often suffers from data heterogeneity problems, which can
significantly affect its performance. To address this, clustered federated
learning (CFL) has been proposed to construct personalized models for different
client clusters. One effective client clustering strategy is to allow clients
to choose their own local models from a model pool based on their performance.
However, without pre-trained model parameters, such a strategy is prone to
clustering failure, in which all clients choose the same model. Unfortunately,
collecting a large amount of labeled data for pre-training can be costly and
impractical in distributed environments. To overcome this challenge, we
leverage self-supervised contrastive learning to exploit unlabeled data for the
pre-training of FL systems. Together, self-supervised pre-training and client
clustering can be crucial components for tackling the data heterogeneity issues
of FL. Leveraging these two crucial strategies, we propose contrastive
pre-training-based clustered federated learning (CP-CFL) to improve the model
convergence and overall performance of FL systems. In this work, we demonstrate
the effectiveness of CP-CFL through extensive experiments in heterogeneous FL
settings, and present various interesting observations.Comment: Published in Neural Network
Federated Learning with Intermediate Representation Regularization
In contrast to centralized model training that involves data collection,
federated learning (FL) enables remote clients to collaboratively train a model
without exposing their private data. However, model performance usually
degrades in FL due to the heterogeneous data generated by clients of diverse
characteristics. One promising strategy to maintain good performance is by
limiting the local training from drifting far away from the global model.
Previous studies accomplish this by regularizing the distance between the
representations learned by the local and global models. However, they only
consider representations from the early layers of a model or the layer
preceding the output layer. In this study, we introduce FedIntR, which provides
a more fine-grained regularization by integrating the representations of
intermediate layers into the local training process. Specifically, FedIntR
computes a regularization term that encourages the closeness between the
intermediate layer representations of the local and global models.
Additionally, FedIntR automatically determines the contribution of each layer's
representation to the regularization term based on the similarity between local
and global representations. We conduct extensive experiments on various
datasets to show that FedIntR can achieve equivalent or higher performance
compared to the state-of-the-art approaches. Our code is available at
https://github.com/YLTun/FedIntR.Comment: IEEE BigComp 202
Federated Learning with Diffusion Models for Privacy-Sensitive Vision Tasks
Diffusion models have shown great potential for vision-related tasks,
particularly for image generation. However, their training is typically
conducted in a centralized manner, relying on data collected from publicly
available sources. This approach may not be feasible or practical in many
domains, such as the medical field, which involves privacy concerns over data
collection. Despite the challenges associated with privacy-sensitive data, such
domains could still benefit from valuable vision services provided by diffusion
models. Federated learning (FL) plays a crucial role in enabling decentralized
model training without compromising data privacy. Instead of collecting data,
an FL system gathers model parameters, effectively safeguarding the private
data of different parties involved. This makes FL systems vital for managing
decentralized learning tasks, especially in scenarios where privacy-sensitive
data is distributed across a network of clients. Nonetheless, FL presents its
own set of challenges due to its distributed nature and privacy-preserving
properties. Therefore, in this study, we explore the FL strategy to train
diffusion models, paving the way for the development of federated diffusion
models. We conduct experiments on various FL scenarios, and our findings
demonstrate that federated diffusion models have great potential to deliver
vision services to privacy-sensitive domains
FedMEKT: Distillation-based Embedding Knowledge Transfer for Multimodal Federated Learning
Federated learning (FL) enables a decentralized machine learning paradigm for
multiple clients to collaboratively train a generalized global model without
sharing their private data. Most existing works simply propose typical FL
systems for single-modal data, thus limiting its potential on exploiting
valuable multimodal data for future personalized applications. Furthermore, the
majority of FL approaches still rely on the labeled data at the client side,
which is limited in real-world applications due to the inability of
self-annotation from users. In light of these limitations, we propose a novel
multimodal FL framework that employs a semi-supervised learning approach to
leverage the representations from different modalities. Bringing this concept
into a system, we develop a distillation-based multimodal embedding knowledge
transfer mechanism, namely FedMEKT, which allows the server and clients to
exchange the joint knowledge of their learning models extracted from a small
multimodal proxy dataset. Our FedMEKT iteratively updates the generalized
global encoders with the joint embedding knowledge from the participating
clients. Thereby, to address the modality discrepancy and labeled data
constraint in existing FL systems, our proposed FedMEKT comprises local
multimodal autoencoder learning, generalized multimodal autoencoder
construction, and generalized classifier learning. Through extensive
experiments on three multimodal human activity recognition datasets, we
demonstrate that FedMEKT achieves superior global encoder performance on linear
evaluation and guarantees user privacy for personal data and model parameters
while demanding less communication cost than other baselines