2,393 research outputs found
Recommended from our members
Federated learning model complexity vs robustness to non-IID data and selective federated learning
Federated learning trains a global model using data distributed across local nodes, and differs from centralized machine learning by moving the computation to the data in order to address the challenges of data ownership, privacy, computational power, and data storage. Previous federated learning research has addressed the effect of non independent and identically distributed data on federated learning [6]. Meanwhile, local models may have better performance if the test set is also non-IID [7]. However, there may be insufficient data on a node to train a local model for every node; hence the purpose of federated learning.
This research is the first, to our knowledge, to consider model performance on both a global test set and non-IID test set. Our experiments provide a original finding in that federated learning is only robust to non-IID data with constraints on the width and depth of a neural network. There is a tradeoff, however, between model complexity and feasibility of training the model on edge devices. Thus, we propose selective federated learning algorithm which greatly allows simpler models that fit on edge devices to be robust to highly non-IID data. For non-IID test sets, we prove that a converged federated model may converge to weights which do not provide the optimal local loss for an arbitrary chosen number of training samples on each node. Additionally, this thesis discusses the experiments that were conducted to examine the effects of model complexity, percentage of unbalanced data, and the current modes of model aggregation on model accuracy. For the experiments, we deployed federated learning library for multiple devices, Jetson Nano, Raspberry Pi, Macbook Pro, and Linux server and provide hardware benchmarks.Electrical and Computer Engineerin
Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data
On-device machine learning (ML) enables the training process to exploit a
massive amount of user-generated private data samples. To enjoy this benefit,
inter-device communication overhead should be minimized. With this end, we
propose federated distillation (FD), a distributed model training algorithm
whose communication payload size is much smaller than a benchmark scheme,
federated learning (FL), particularly when the model size is large. Moreover,
user-generated data samples are likely to become non-IID across devices, which
commonly degrades the performance compared to the case with an IID dataset. To
cope with this, we propose federated augmentation (FAug), where each device
collectively trains a generative model, and thereby augments its local data
towards yielding an IID dataset. Empirical studies demonstrate that FD with
FAug yields around 26x less communication overhead while achieving 95-98% test
accuracy compared to FL.Comment: presented at the 32nd Conference on Neural Information Processing
Systems (NIPS 2018), 2nd Workshop on Machine Learning on the Phone and other
Consumer Devices (MLPCD 2), Montr\'eal, Canad
Learning From Drift: Federated Learning on Non-IID Data via Drift Regularization
Federated learning algorithms perform reasonably well on independent and
identically distributed (IID) data. They, on the other hand, suffer greatly
from heterogeneous environments, i.e., Non-IID data. Despite the fact that many
research projects have been done to address this issue, recent findings
indicate that they are still sub-optimal when compared to training on IID data.
In this work, we carefully analyze the existing methods in heterogeneous
environments. Interestingly, we find that regularizing the classifier's outputs
is quite effective in preventing performance degradation on Non-IID data.
Motivated by this, we propose Learning from Drift (LfD), a novel method for
effectively training the model in heterogeneous settings. Our scheme
encapsulates two key components: drift estimation and drift regularization.
Specifically, LfD first estimates how different the local model is from the
global model (i.e., drift). The local model is then regularized such that it
does not fall in the direction of the estimated drift. In the experiment, we
evaluate each method through the lens of the five aspects of federated
learning, i.e., Generalization, Heterogeneity, Scalability, Forgetting, and
Efficiency. Comprehensive evaluation results clearly support the superiority of
LfD in federated learning with Non-IID data
Exploiting Personalized Invariance for Better Out-of-distribution Generalization in Federated Learning
Recently, data heterogeneity among the training datasets on the local clients
(a.k.a., Non-IID data) has attracted intense interest in Federated Learning
(FL), and many personalized federated learning methods have been proposed to
handle it. However, the distribution shift between the training dataset and
testing dataset on each client is never considered in FL, despite it being
general in real-world scenarios. We notice that the distribution shift (a.k.a.,
out-of-distribution generalization) problem under Non-IID federated setting
becomes rather challenging due to the entanglement between personalized and
spurious information. To tackle the above problem, we elaborate a general
dual-regularized learning framework to explore the personalized invariance,
compared with the exsiting personalized federated learning methods which are
regularized by a single baseline (usually the global model). Utilizing the
personalized invariant features, the developed personalized models can
efficiently exploit the most relevant information and meanwhile eliminate
spurious information so as to enhance the out-of-distribution generalization
performance for each client. Both the theoretical analysis on convergence and
OOD generalization performance and the results of extensive experiments
demonstrate the superiority of our method over the existing federated learning
and invariant learning methods, in diverse out-of-distribution and Non-IID data
cases
Privacy is What We Care About: Experimental Investigation of Federated Learning on Edge Devices
Federated Learning enables training of a general model through edge devices
without sending raw data to the cloud. Hence, this approach is attractive for
digital health applications, where data is sourced through edge devices and
users care about privacy. Here, we report on the feasibility to train deep
neural networks on the Raspberry Pi4s as edge devices. A CNN, a LSTM and a MLP
were successfully trained on the MNIST data-set. Further, federated learning is
demonstrated experimentally on IID and non-IID samples in a parametric study,
to benchmark the model convergence. The weight updates from the workers are
shared with the cloud to train the general model through federated learning.
With the CNN and the non-IID samples a test-accuracy of up to 85% could be
achieved within a training time of 2 minutes, while exchanging less than
MB data per device. In addition, we discuss federated learning from an use-case
standpoint, elaborating on privacy risks and labeling requirements for the
application of emotion detection from sound. Based on the experimental
findings, we discuss possible research directions to improve model and system
performance. Finally, we provide best practices for a practitioner, considering
the implementation of federated learning.Comment: Accepted in ACM AIChallengeIoT 2019, New York, US
Knowledge-Aware Federated Active Learning with Non-IID Data
Federated learning enables multiple decentralized clients to learn
collaboratively without sharing the local training data. However, the expensive
annotation cost to acquire data labels on local clients remains an obstacle in
utilizing local data. In this paper, we propose a federated active learning
paradigm to efficiently learn a global model with limited annotation budget
while protecting data privacy in a decentralized learning way. The main
challenge faced by federated active learning is the mismatch between the active
sampling goal of the global model on the server and that of the asynchronous
local clients. This becomes even more significant when data is distributed
non-IID across local clients. To address the aforementioned challenge, we
propose Knowledge-Aware Federated Active Learning (KAFAL), which consists of
Knowledge-Specialized Active Sampling (KSAS) and Knowledge-Compensatory
Federated Update (KCFU). KSAS is a novel active sampling method tailored for
the federated active learning problem. It deals with the mismatch challenge by
sampling actively based on the discrepancies between local and global models.
KSAS intensifies specialized knowledge in local clients, ensuring the sampled
data to be informative for both the local clients and the global model. KCFU,
in the meantime, deals with the client heterogeneity caused by limited data and
non-IID data distributions. It compensates for each client's ability in weak
classes by the assistance of the global model. Extensive experiments and
analyses are conducted to show the superiority of KSAS over the
state-of-the-art active learning methods and the efficiency of KCFU under the
federated active learning framework.Comment: 14 pages, 12 figure
- …