30 research outputs found
Decentralized Differentially Private Without-Replacement Stochastic Gradient Descent
While machine learning has achieved remarkable results in a wide variety of
domains, the training of models often requires large datasets that may need to
be collected from different individuals. As sensitive information may be
contained in the individual's dataset, sharing training data may lead to severe
privacy concerns. Therefore, there is a compelling need to develop
privacy-aware machine learning methods, for which one effective approach is to
leverage the generic framework of differential privacy. Considering that
stochastic gradient descent (SGD) is one of the mostly adopted methods for
large-scale machine learning problems, two decentralized differentially private
SGD algorithms are proposed in this work. Particularly, we focus on SGD without
replacement due to its favorable structure for practical implementation. In
addition, both privacy and convergence analysis are provided for the proposed
algorithms. Finally, extensive experiments are performed to verify the
theoretical results and demonstrate the effectiveness of the proposed
algorithms
Resource Constrained Vehicular Edge Federated Learning with Highly Mobile Connected Vehicles
This paper proposes a vehicular edge federated learning (VEFL) solution,
where an edge server leverages highly mobile connected vehicles' (CVs') onboard
central processing units (CPUs) and local datasets to train a global model.
Convergence analysis reveals that the VEFL training loss depends on the
successful receptions of the CVs' trained models over the intermittent
vehicle-to-infrastructure (V2I) wireless links. Owing to high mobility, in the
full device participation case (FDPC), the edge server aggregates client model
parameters based on a weighted combination according to the CVs' dataset sizes
and sojourn periods, while it selects a subset of CVs in the partial device
participation case (PDPC). We then devise joint VEFL and radio access
technology (RAT) parameters optimization problems under delay, energy and cost
constraints to maximize the probability of successful reception of the locally
trained models. Considering that the optimization problem is NP-hard, we
decompose it into a VEFL parameter optimization sub-problem, given the
estimated worst-case sojourn period, delay and energy expense, and an online
RAT parameter optimization sub-problem. Finally, extensive simulations are
conducted to validate the effectiveness of the proposed solutions with a
practical 5G new radio (5G-NR) RAT under a realistic microscopic mobility
model
Distributed Learning over Networks with Graph-Attention-Based Personalization
In conventional distributed learning over a network, multiple agents
collaboratively build a common machine learning model. However, due to the
underlying non-i.i.d. data distribution among agents, the unified learning
model becomes inefficient for each agent to process its locally accessible
data. To address this problem, we propose a graph-attention-based personalized
training algorithm (GATTA) for distributed deep learning. The GATTA enables
each agent to train its local personalized model while exploiting its
correlation with neighboring nodes and utilizing their useful information for
aggregation. In particular, the personalized model in each agent is composed of
a global part and a node-specific part. By treating each agent as one node in a
graph and the node-specific parameters as its features, the benefits of the
graph attention mechanism can be inherited. Namely, instead of aggregation
based on averaging, it learns the specific weights for different neighboring
nodes without requiring prior knowledge about the graph structure or the
neighboring nodes' data distribution. Furthermore, relying on the
weight-learning procedure, we develop a communication-efficient GATTA by
skipping the transmission of information with small aggregation weights.
Additionally, we theoretically analyze the convergence properties of GATTA for
non-convex loss functions. Numerical results validate the excellent
performances of the proposed algorithms in terms of convergence and
communication cost.Comment: Accepted for publication in IEEE TSP; with supplementary details for
the derivation
Mobile MIMO Channel Prediction with ODE-RNN: a Physics-Inspired Adaptive Approach
Obtaining accurate channel state information (CSI) is crucial and challenging
for multiple-input multiple-output (MIMO) wireless communication systems.
Conventional channel estimation method cannot guarantee the accuracy of mobile
CSI while requires high signaling overhead. Through exploring the intrinsic
correlation among a set of historical CSI instances randomly obtained in a
certain communication environment, channel prediction can significantly
increase CSI accuracy and save signaling overhead. In this paper, we propose a
novel channel prediction method based on ordinary differential equation
(ODE)-recurrent neural network (RNN) for accurate and flexible mobile MIMO
channel prediction. Differing from existing works using sequential network
structures for exploring the numerical correlation between observed data, our
proposed method tries to represent the implicit physics process of path
responses changing by specially designed continuous learning network with ODE
structure. Due to the targeted design of learning network, our proposed method
fits the mathematics feature of CSI data better and enjoy higher network
interpretability. Experimental results show that the proposed learning approach
outperforms existing methods, especially for long time interval of the CSI
sequence and large channel measurement error.Comment: 7 pages, conferenc
Breaking the Communication-Privacy-Accuracy Tradeoff with -Differential Privacy
We consider a federated data analytics problem in which a server coordinates
the collaborative data analysis of multiple users with privacy concerns and
limited communication capability. The commonly adopted compression schemes
introduce information loss into local data while improving communication
efficiency, and it remains an open problem whether such discrete-valued
mechanisms provide any privacy protection. In this paper, we study the local
differential privacy guarantees of discrete-valued mechanisms with finite
output space through the lens of -differential privacy (DP). More
specifically, we advance the existing literature by deriving tight -DP
guarantees for a variety of discrete-valued mechanisms, including the binomial
noise and the binomial mechanisms that are proposed for privacy preservation,
and the sign-based methods that are proposed for data compression, in
closed-form expressions. We further investigate the amplification in privacy by
sparsification and propose a ternary stochastic compressor. By leveraging
compression for privacy amplification, we improve the existing methods by
removing the dependency of accuracy (in terms of mean square error) on
communication cost in the popular use case of distributed mean estimation,
therefore breaking the three-way tradeoff between privacy, communication, and
accuracy. Finally, we discuss the Byzantine resilience of the proposed
mechanism and its application in federated learning