337 research outputs found
Edge Intelligence Over the Air: Two Faces of Interference in Federated Learning
Federated edge learning is envisioned as the bedrock of enabling intelligence
in next-generation wireless networks, but the limited spectral resources often
constrain its scalability. In light of this challenge, a line of recent
research suggested integrating analog over-the-air computations into federated
edge learning systems, to exploit the superposition property of electromagnetic
waves for fast aggregation of intermediate parameters and achieve (almost)
unlimited scalability. Over-the-air computations also benefit the system in
other aspects, such as low hardware cost, reduced access latency, and enhanced
privacy protection. Despite these advantages, the interference introduced by
wireless communications also influences various aspects of the model training
process, while its importance is not well recognized yet. This article provides
a comprehensive overview of the positive and negative effects of interference
on over-the-air computation-based edge learning systems. The potential open
issues and research trends are also discussed.Comment: 7 pages, 6 figures. Accepted by IEEE Communications Magazin
Towards Scalable, Private and Practical Deep Learning
Deep Learning (DL) models have drastically improved the performance of Artificial Intelligence (AI) tasks such as image recognition, word prediction, translation, among many others, on which traditional Machine Learning (ML) models fall short. However, DL models are costly to design, train, and deploy due to their computing and memory demands. Designing DL models usually requires extensive expertise and significant manual tuning efforts. Even with the latest accelerators such as Graphics Processing Unit (GPU) and Tensor Processing Unit (TPU), training DL models can take prohibitively long time, therefore training large DL models in a distributed manner is a norm. Massive amount of data is made available thanks to the prevalence of mobile and internet-of-things (IoT) devices. However, regulations such as HIPAA and GDPR limit the access and transmission of personal data to protect security and privacy. Therefore, enabling DL model training in a decentralized but private fashion is urgent and critical. Deploying trained DL models in a real world environment usually requires meeting Quality of Service (QoS) standards, which makes adaptability of DL models an important yet challenging matter. In this dissertation, we aim to address the above challenges to make a step towards scalable, private, and practical deep learning. To simplify DL model design, we propose Efficient Progressive Neural-Architecture Search (EPNAS) and FedCust to automatically design model architectures and tune hyperparameters, respectively. To provide efficient and robust distributed training while preserving privacy, we design LEASGD, TiFL, and HDFL. We further conduct a study on the security aspect of distributed learning by focusing on how data heterogeneity affects backdoor attacks and how to mitigate such threats. Finally, we use super resolution (SR) as an example application to explore model adaptability for cross platform deployment and dynamic runtime environment. Specifically, we propose DySR and AdaSR frameworks which enable SR models to meet QoS by dynamically adapting to available resources instantly and seamlessly without excessive memory overheads
Computation and Communication Efficient Federated Learning over Wireless Networks
Federated learning (FL) allows model training from local data by edge devices
while preserving data privacy. However, the learning accuracy decreases due to
the heterogeneity of devices data, and the computation and communication
latency increase when updating large scale learning models on devices with
limited computational capability and wireless resources. To overcome these
challenges, we consider a novel FL framework with partial model pruning and
personalization. This framework splits the learning model into a global part
with model pruning shared with all devices to learn data representations and a
personalized part to be fine tuned for a specific device, which adapts the
model size during FL to reduce both computation and communication overhead and
minimize the overall training time, and increases the learning accuracy for the
device with non independent and identically distributed (non IID) data. Then,
the computation and communication latency and the convergence analysis of the
proposed FL framework are mathematically analyzed. Based on the convergence
analysis, an optimization problem is formulated to maximize the convergence
rate under a latency threshold by jointly optimizing the pruning ratio and
wireless resource allocation. By decoupling the optimization problem and
deploying Karush Kuhn Tucker (KKT) conditions, we derive the closed form
solutions of pruning ratio and wireless resource allocation. Finally,
experimental results demonstrate that the proposed FL framework achieves a
remarkable reduction of approximately 50 percents computation and communication
latency compared with the scheme only with model personalization.Comment: arXiv admin note: text overlap with arXiv:2305.0904
- …