381 research outputs found
Towards Efficient Communications in Federated Learning: A Contemporary Survey
In the traditional distributed machine learning scenario, the user's private
data is transmitted between nodes and a central server, which results in great
potential privacy risks. In order to balance the issues of data privacy and
joint training of models, federated learning (FL) is proposed as a special
distributed machine learning with a privacy protection mechanism, which can
realize multi-party collaborative computing without revealing the original
data. However, in practice, FL faces many challenging communication problems.
This review aims to clarify the relationship between these communication
problems, and focus on systematically analyzing the research progress of FL
communication work from three perspectives: communication efficiency,
communication environment, and communication resource allocation. Firstly, we
sort out the current challenges existing in the communications of FL. Secondly,
we have compiled articles related to FL communications, and then describe the
development trend of the entire field guided by the logical relationship
between them. Finally, we point out the future research directions for
communications in FL
Heterogeneous Federated Learning: State-of-the-art and Research Challenges
Federated learning (FL) has drawn increasing attention owing to its potential
use in large-scale industrial applications. Existing federated learning works
mainly focus on model homogeneous settings. However, practical federated
learning typically faces the heterogeneity of data distributions, model
architectures, network environments, and hardware devices among participant
clients. Heterogeneous Federated Learning (HFL) is much more challenging, and
corresponding solutions are diverse and complex. Therefore, a systematic survey
on this topic about the research challenges and state-of-the-art is essential.
In this survey, we firstly summarize the various research challenges in HFL
from five aspects: statistical heterogeneity, model heterogeneity,
communication heterogeneity, device heterogeneity, and additional challenges.
In addition, recent advances in HFL are reviewed and a new taxonomy of existing
HFL methods is proposed with an in-depth analysis of their pros and cons. We
classify existing methods from three different levels according to the HFL
procedure: data-level, model-level, and server-level. Finally, several critical
and promising future research directions in HFL are discussed, which may
facilitate further developments in this field. A periodically updated
collection on HFL is available at https://github.com/marswhu/HFL_Survey.Comment: 42 pages, 11 figures, and 4 table
Decentralized and Model-Free Federated Learning: Consensus-Based Distillation in Function Space
This paper proposes a fully decentralized federated learning (FL) scheme for
Internet of Everything (IoE) devices that are connected via multi-hop networks.
Because FL algorithms hardly converge the parameters of machine learning (ML)
models, this paper focuses on the convergence of ML models in function spaces.
Considering that the representative loss functions of ML tasks e.g, mean
squared error (MSE) and Kullback-Leibler (KL) divergence, are convex
functionals, algorithms that directly update functions in function spaces could
converge to the optimal solution. The key concept of this paper is to tailor a
consensus-based optimization algorithm to work in the function space and
achieve the global optimum in a distributed manner. This paper first analyzes
the convergence of the proposed algorithm in a function space, which is
referred to as a meta-algorithm, and shows that the spectral graph theory can
be applied to the function space in a manner similar to that of numerical
vectors. Then, consensus-based multi-hop federated distillation (CMFD) is
developed for a neural network (NN) to implement the meta-algorithm. CMFD
leverages knowledge distillation to realize function aggregation among adjacent
devices without parameter averaging. An advantage of CMFD is that it works even
with different NN models among the distributed learners. Although CMFD does not
perfectly reflect the behavior of the meta-algorithm, the discussion of the
meta-algorithm's convergence property promotes an intuitive understanding of
CMFD, and simulation evaluations show that NN models converge using CMFD for
several tasks. The simulation results also show that CMFD achieves higher
accuracy than parameter aggregation for weakly connected networks, and CMFD is
more stable than parameter aggregation methods.Comment: submitted to IEEE TSIP
Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Data
This study develops a federated learning (FL) framework overcoming largely incremental communication costs due to model sizes in typical frameworks without compromising model performance. To this end, based on the idea of leveraging an unlabeled open dataset, we propose a distillation-based semi-supervised FL (DS-FL) algorithm that exchanges the outputs of local models among mobile devices, instead of model parameter exchange employed by the typical frameworks. In DS-FL, the communication cost depends only on the output dimensions of the models and does not scale up according to the model size. The exchanged model outputs are used to label each sample of the open dataset, which creates an additionally labeled dataset. Based on the new dataset, local models are further trained, and model performance is enhanced owing to the data augmentation effect. We further highlight that in DS-FL, the heterogeneity of the devices’ dataset leads to ambiguous of each data sample and lowing of the training convergence. To prevent this, we propose entropy reduction averaging, where the aggregated model outputs are intentionally sharpened. Moreover, extensive experiments show that DS-FL reduces communication costs up to 99 percent relative to those of the FL benchmark while achieving similar or higher classification accuracy
- …