Federated learning over wireless fading channels

We study federated machine learning at the wirelessnetwork edge, where limited power wireless devices, each withits own dataset, build a joint model with the help of a remoteparameter server (PS). We consider a bandwidth-limited fadingmultiple access channel (MAC) from the wireless devices to thePS, and propose various techniques to implement distributedstochastic gradient descent (DSGD) over this shared noisywireless channel. We first propose a digital DSGD (D-DSGD)scheme, in which one device is selected opportunistically fortransmission at each iteration based on the channel conditions;the scheduled device quantizes its gradient estimate to a finitenumber of bits imposed by the channel condition, and transmitsthese bits to the PS in a reliable manner. Next, motivated bythe additive nature of the wireless MAC, we propose a novelanalog communication scheme, referred to as thecompressedanalogDSGD (CA-DSGD), where the devices first sparsifytheir gradient estimates while accumulating error from previousiterations, and project the resultant sparse vector into a low-dimensional vector for bandwidth reduction. We also design apower allocation scheme to align the received gradient vectorsat the PS in an efficient manner. Numerical results show thatD-DSGD outperforms other digital approaches in the literature;however, in general the proposed CA-DSGD algorithm convergesfaster than the D-DSGD scheme, and reaches a higher level ofaccuracy. We have observed that the gap between the analogand digital schemes increases when the datasets of devices arenot independent and identically distributed (i.i.d.). Furthermore,the performance of the CA-DSGD scheme is shown to be robustagainst imperfect channel state information (CSI) at the devices.Overall these results show clear advantages for the proposedanalog over-the-air DSGD scheme, which suggests that learningand communication algorithms should be designed jointly toachieve the best end-to-end performance in machine learningapplications at the wireless edge

Wireless Data Acquisition for Edge Learning: Data-Importance Aware Retransmission

By deploying machine-learning algorithms at the network edge, edge learning can leverage the enormous real-time data generated by billions of mobile devices to train AI models, which enable intelligent mobile applications. In this emerging research area, one key direction is to efficiently utilize radio resources for wireless data acquisition to minimize the latency of executing a learning task at an edge server. Along this direction, we consider the specific problem of retransmission decision in each communication round to ensure both reliability and quantity of those training data for accelerating model convergence. To solve the problem, a new retransmission protocol called data-importance aware automatic-repeat-request (importance ARQ) is proposed. Unlike the classic ARQ focusing merely on reliability, importance ARQ selectively retransmits a data sample based on its uncertainty which helps learning and can be measured using the model under training. Underpinning the proposed protocol is a derived elegant communication-learning relation between two corresponding metrics, i.e., signal-to-noise ratio (SNR) and data uncertainty. This relation facilitates the design of a simple threshold based policy for importance ARQ. The policy is first derived based on the classic classifier model of support vector machine (SVM), where the uncertainty of a data sample is measured by its distance to the decision boundary. The policy is then extended to the more complex model of convolutional neural networks (CNN) where data uncertainty is measured by entropy. Extensive experiments have been conducted for both the SVM and CNN using real datasets with balanced and imbalanced distributions. Experimental results demonstrate that importance ARQ effectively copes with channel fading and noise in wireless data acquisition to achieve faster model convergence than the conventional channel-aware ARQ.Comment: This is an updated version: 1) extension to general classifiers; 2) consideration of imbalanced classification in the experiments. Submitted to IEEE Journal for possible publicatio

Wireless for Machine Learning

As data generation increasingly takes place on devices without a wired connection, Machine Learning over wireless networks becomes critical. Many studies have shown that traditional wireless protocols are highly inefficient or unsustainable to support Distributed Machine Learning. This is creating the need for new wireless communication methods. In this survey, we give an exhaustive review of the state of the art wireless methods that are specifically designed to support Machine Learning services. Namely, over-the-air computation and radio resource allocation optimized for Machine Learning. In the over-the-air approach, multiple devices communicate simultaneously over the same time slot and frequency band to exploit the superposition property of wireless channels for gradient averaging over-the-air. In radio resource allocation optimized for Machine Learning, Active Learning metrics allow for data evaluation to greatly optimize the assignment of radio resources. This paper gives a comprehensive introduction to these methods, reviews the most important works, and highlights crucial open problems.Comment: Corrected typo in author name. From the incorrect Maitron to the correct Mairto