371 research outputs found
Sparse random networks for communication-efficient federated learning
One main challenge in federated learning is the large communication cost of ex-changing weight updates from clients to the server at each round. While prior work has made great progress in compressing the weight updates through gradient compression methods, we propose a radically different approach that does not update the weights at all. Instead, our method freezes the weights at their initial random values and learns how to sparsify the random network for the best performance. To this end, the clients collaborate in training a stochastic binary mask to find the optimal sparse random network within the original one. At the end of the training, the final model is a sparse network with random weights – or a sub-network inside the dense random network. We show improvements in accuracy, communication (less than 1 bit per parameter (bpp)), convergence speed, and final model size (less than 1 bpp) over relevant baselines on MNIST, EMNIST, CIFAR- 10, and CIFAR-100 datasets, in the low bitrate regime
A Comprehensive Survey On Client Selections in Federated Learning
Federated Learning (FL) is a rapidly growing field in machine learning that
allows data to be trained across multiple decentralized devices. The selection
of clients to participate in the training process is a critical factor for the
performance of the overall system. In this survey, we provide a comprehensive
overview of the state-of-the-art client selection techniques in FL, including
their strengths and limitations, as well as the challenges and open issues that
need to be addressed. We cover conventional selection techniques such as random
selection where all or partial random of clients is used for the trained. We
also cover performance-aware selections and as well as resource-aware
selections for resource-constrained networks and heterogeneous networks. We
also discuss the usage of client selection in model security enhancement.
Lastly, we discuss open issues and challenges related to clients selection in
dynamic constrained, and heterogeneous networks
Complement Sparsification: Low-Overhead Model Pruning for Federated Learning
Federated Learning (FL) is a privacy-preserving distributed deep learning
paradigm that involves substantial communication and computation effort, which
is a problem for resource-constrained mobile and IoT devices. Model
pruning/sparsification develops sparse models that could solve this problem,
but existing sparsification solutions cannot satisfy at the same time the
requirements for low bidirectional communication overhead between the server
and the clients, low computation overhead at the clients, and good model
accuracy, under the FL assumption that the server does not have access to raw
data to fine-tune the pruned models. We propose Complement Sparsification (CS),
a pruning mechanism that satisfies all these requirements through a
complementary and collaborative pruning done at the server and the clients. At
each round, CS creates a global sparse model that contains the weights that
capture the general data distribution of all clients, while the clients create
local sparse models with the weights pruned from the global model to capture
the local trends. For improved model performance, these two types of
complementary sparse models are aggregated into a dense model in each round,
which is subsequently pruned in an iterative process. CS requires little
computation overhead on the top of vanilla FL for both the server and the
clients. We demonstrate that CS is an approximation of vanilla FL and, thus,
its models perform well. We evaluate CS experimentally with two popular FL
benchmark datasets. CS achieves substantial reduction in bidirectional
communication, while achieving performance comparable with vanilla FL. In
addition, CS outperforms baseline pruning mechanisms for FL
PA-iMFL: Communication-Efficient Privacy Amplification Method against Data Reconstruction Attack in Improved Multi-Layer Federated Learning
Recently, big data has seen explosive growth in the Internet of Things (IoT).
Multi-layer FL (MFL) based on cloud-edge-end architecture can promote model
training efficiency and model accuracy while preserving IoT data privacy. This
paper considers an improved MFL, where edge layer devices own private data and
can join the training process. iMFL can improve edge resource utilization and
also alleviate the strict requirement of end devices, but suffers from the
issues of Data Reconstruction Attack (DRA) and unacceptable communication
overhead. This paper aims to address these issues with iMFL. We propose a
Privacy Amplification scheme on iMFL (PA-iMFL). Differing from standard MFL, we
design privacy operations in end and edge devices after local training,
including three sequential components, local differential privacy with Laplace
mechanism, privacy amplification subsample, and gradient sign reset.
Benefitting from privacy operations, PA-iMFL reduces communication overhead and
achieves privacy-preserving. Extensive results demonstrate that against
State-Of-The-Art (SOTA) DRAs, PA-iMFL can effectively mitigate private data
leakage and reach the same level of protection capability as the SOTA defense
model. Moreover, due to adopting privacy operations in edge devices, PA-iMFL
promotes up to 2.8 times communication efficiency than the SOTA compression
method without compromising model accuracy.Comment: 12 pages, 11 figure
Machine Unlearning: Solutions and Challenges
Machine learning models may inadvertently memorize sensitive, unauthorized,
or malicious data, posing risks of privacy violations, security breaches, and
performance deterioration. To address these issues, machine unlearning has
emerged as a critical technique to selectively remove specific training data
points' influence on trained models. This paper provides a comprehensive
taxonomy and analysis of machine unlearning research. We categorize existing
research into exact unlearning that algorithmically removes data influence
entirely and approximate unlearning that efficiently minimizes influence
through limited parameter updates. By reviewing the state-of-the-art solutions,
we critically discuss their advantages and limitations. Furthermore, we propose
future directions to advance machine unlearning and establish it as an
essential capability for trustworthy and adaptive machine learning. This paper
provides researchers with a roadmap of open problems, encouraging impactful
contributions to address real-world needs for selective data removal
Trustworthy Federated Learning: A Survey
Federated Learning (FL) has emerged as a significant advancement in the field
of Artificial Intelligence (AI), enabling collaborative model training across
distributed devices while maintaining data privacy. As the importance of FL
increases, addressing trustworthiness issues in its various aspects becomes
crucial. In this survey, we provide an extensive overview of the current state
of Trustworthy FL, exploring existing solutions and well-defined pillars
relevant to Trustworthy . Despite the growth in literature on trustworthy
centralized Machine Learning (ML)/Deep Learning (DL), further efforts are
necessary to identify trustworthiness pillars and evaluation metrics specific
to FL models, as well as to develop solutions for computing trustworthiness
levels. We propose a taxonomy that encompasses three main pillars:
Interpretability, Fairness, and Security & Privacy. Each pillar represents a
dimension of trust, further broken down into different notions. Our survey
covers trustworthiness challenges at every level in FL settings. We present a
comprehensive architecture of Trustworthy FL, addressing the fundamental
principles underlying the concept, and offer an in-depth analysis of trust
assessment mechanisms. In conclusion, we identify key research challenges
related to every aspect of Trustworthy FL and suggest future research
directions. This comprehensive survey serves as a valuable resource for
researchers and practitioners working on the development and implementation of
Trustworthy FL systems, contributing to a more secure and reliable AI
landscape.Comment: 45 Pages, 8 Figures, 9 Table
- …