3,220 research outputs found
Selective Knowledge Sharing for Privacy-Preserving Federated Distillation without A Good Teacher
While federated learning is promising for privacy-preserving collaborative
learning without revealing local data, it remains vulnerable to white-box
attacks and struggles to adapt to heterogeneous clients. Federated distillation
(FD), built upon knowledge distillation--an effective technique for
transferring knowledge from a teacher model to student models--emerges as an
alternative paradigm, which provides enhanced privacy guarantees and addresses
model heterogeneity. Nevertheless, challenges arise due to variations in local
data distributions and the absence of a well-trained teacher model, which leads
to misleading and ambiguous knowledge sharing that significantly degrades model
performance. To address these issues, this paper proposes a selective knowledge
sharing mechanism for FD, termed Selective-FD. It includes client-side
selectors and a server-side selector to accurately and precisely identify
knowledge from local and ensemble predictions, respectively. Empirical studies,
backed by theoretical insights, demonstrate that our approach enhances the
generalization capabilities of the FD framework and consistently outperforms
baseline methods. This study presents a promising direction for effective
knowledge transfer in privacy-preserving collaborative learning
pFedES: Model Heterogeneous Personalized Federated Learning with Feature Extractor Sharing
As a privacy-preserving collaborative machine learning paradigm, federated
learning (FL) has attracted significant interest from academia and the industry
alike. To allow each data owner (a.k.a., FL clients) to train a heterogeneous
and personalized local model based on its local data distribution, system
resources and requirements on model structure, the field of model-heterogeneous
personalized federated learning (MHPFL) has emerged. Existing MHPFL approaches
either rely on the availability of a public dataset with special
characteristics to facilitate knowledge transfer, incur high computation and
communication costs, or face potential model leakage risks. To address these
limitations, we propose a model-heterogeneous personalized Federated learning
approach based on feature Extractor Sharing (pFedES). It incorporates a small
homogeneous feature extractor into each client's heterogeneous local model.
Clients train them via the proposed iterative learning method to enable the
exchange of global generalized knowledge and local personalized knowledge. The
small local homogeneous extractors produced after local training are uploaded
to the FL server and for aggregation to facilitate easy knowledge sharing among
clients. We theoretically prove that pFedES can converge over wall-to-wall
time. Extensive experiments on two real-world datasets against six
state-of-the-art methods demonstrate that pFedES builds the most accurate
model, while incurring low communication and computation costs. Compared with
the best-performing baseline, it achieves 1.61% higher test accuracy, while
reducing communication and computation costs by 99.6% and 82.9%, respectively.Comment: 12 pages, 10 figures. arXiv admin note: text overlap with
arXiv:2310.1328
Federated Neural Architecture Search
To preserve user privacy while enabling mobile intelligence, techniques have
been proposed to train deep neural networks on decentralized data. However,
training over decentralized data makes the design of neural architecture quite
difficult as it already was. Such difficulty is further amplified when
designing and deploying different neural architectures for heterogeneous mobile
platforms. In this work, we propose an automatic neural architecture search
into the decentralized training, as a new DNN training paradigm called
Federated Neural Architecture Search, namely federated NAS. To deal with the
primary challenge of limited on-client computational and communication
resources, we present FedNAS, a highly optimized framework for efficient
federated NAS. FedNAS fully exploits the key opportunity of insufficient model
candidate re-training during the architecture search process, and incorporates
three key optimizations: parallel candidates training on partial clients, early
dropping candidates with inferior performance, and dynamic round numbers.
Tested on large-scale datasets and typical CNN architectures, FedNAS achieves
comparable model accuracy as state-of-the-art NAS algorithm that trains models
with centralized data, and also reduces the client cost by up to two orders of
magnitude compared to a straightforward design of federated NAS
Heterogeneous Federated Learning: State-of-the-art and Research Challenges
Federated learning (FL) has drawn increasing attention owing to its potential
use in large-scale industrial applications. Existing federated learning works
mainly focus on model homogeneous settings. However, practical federated
learning typically faces the heterogeneity of data distributions, model
architectures, network environments, and hardware devices among participant
clients. Heterogeneous Federated Learning (HFL) is much more challenging, and
corresponding solutions are diverse and complex. Therefore, a systematic survey
on this topic about the research challenges and state-of-the-art is essential.
In this survey, we firstly summarize the various research challenges in HFL
from five aspects: statistical heterogeneity, model heterogeneity,
communication heterogeneity, device heterogeneity, and additional challenges.
In addition, recent advances in HFL are reviewed and a new taxonomy of existing
HFL methods is proposed with an in-depth analysis of their pros and cons. We
classify existing methods from three different levels according to the HFL
procedure: data-level, model-level, and server-level. Finally, several critical
and promising future research directions in HFL are discussed, which may
facilitate further developments in this field. A periodically updated
collection on HFL is available at https://github.com/marswhu/HFL_Survey.Comment: 42 pages, 11 figures, and 4 table
Continual Local Training for Better Initialization of Federated Models
Federated learning (FL) refers to the learning paradigm that trains machine
learning models directly in the decentralized systems consisting of smart edge
devices without transmitting the raw data, which avoids the heavy communication
costs and privacy concerns. Given the typical heterogeneous data distributions
in such situations, the popular FL algorithm \emph{Federated Averaging}
(FedAvg) suffers from weight divergence and thus cannot achieve a competitive
performance for the global model (denoted as the \emph{initial performance} in
FL) compared to centralized methods. In this paper, we propose the local
continual training strategy to address this problem. Importance weights are
evaluated on a small proxy dataset on the central server and then used to
constrain the local training. With this additional term, we alleviate the
weight divergence and continually integrate the knowledge on different local
clients into the global model, which ensures a better generalization ability.
Experiments on various FL settings demonstrate that our method significantly
improves the initial performance of federated models with few extra
communication costs.Comment: This paper has been accepted to 2020 IEEE International Conference on
Image Processing (ICIP 2020
- …