371 research outputs found
Ternary Compression for Communication-Efficient Federated Learning
Learning over massive data stored in different locations is essential in many
real-world applications. However, sharing data is full of challenges due to the
increasing demands of privacy and security with the growing use of smart mobile
devices and IoT devices. Federated learning provides a potential solution to
privacy-preserving and secure machine learning, by means of jointly training a
global model without uploading data distributed on multiple devices to a
central server. However, most existing work on federated learning adopts
machine learning models with full-precision weights, and almost all these
models contain a large number of redundant parameters that do not need to be
transmitted to the server, consuming an excessive amount of communication
costs. To address this issue, we propose a federated trained ternary
quantization (FTTQ) algorithm, which optimizes the quantized networks on the
clients through a self-learning quantization factor. A convergence proof of the
quantization factor and the unbiasedness of FTTQ is given. In addition, we
propose a ternary federated averaging protocol (T-FedAvg) to reduce the
upstream and downstream communication of federated learning systems. Empirical
experiments are conducted to train widely used deep learning models on publicly
available datasets, and our results demonstrate the effectiveness of FTTQ and
T-FedAvg compared with the canonical federated learning algorithms in reducing
communication costs and maintaining the learning performance
Learning Sparse & Ternary Neural Networks with Entropy-Constrained Trained Ternarization (EC2T)
Deep neural networks (DNN) have shown remarkable success in a variety of
machine learning applications. The capacity of these models (i.e., number of
parameters), endows them with expressive power and allows them to reach the
desired performance. In recent years, there is an increasing interest in
deploying DNNs to resource-constrained devices (i.e., mobile devices) with
limited energy, memory, and computational budget. To address this problem, we
propose Entropy-Constrained Trained Ternarization (EC2T), a general framework
to create sparse and ternary neural networks which are efficient in terms of
storage (e.g., at most two binary-masks and two full-precision values are
required to save a weight matrix) and computation (e.g., MAC operations are
reduced to a few accumulations plus two multiplications). This approach
consists of two steps. First, a super-network is created by scaling the
dimensions of a pre-trained model (i.e., its width and depth). Subsequently,
this super-network is simultaneously pruned (using an entropy constraint) and
quantized (that is, ternary values are assigned layer-wise) in a training
process, resulting in a sparse and ternary network representation. We validate
the proposed approach in CIFAR-10, CIFAR-100, and ImageNet datasets, showing
its effectiveness in image classification tasks.Comment: Proceedings of the CVPR'20 Joint Workshop on Efficient Deep Learning
in Computer Vision. Code is available at
https://github.com/d-becking/efficientCNN
3DQ: Compact Quantized Neural Networks for Volumetric Whole Brain Segmentation
Model architectures have been dramatically increasing in size, improving
performance at the cost of resource requirements. In this paper we propose 3DQ,
a ternary quantization method, applied for the first time to 3D Fully
Convolutional Neural Networks (F-CNNs), enabling 16x model compression while
maintaining performance on par with full precision models. We extensively
evaluate 3DQ on two datasets for the challenging task of whole brain
segmentation. Additionally, we showcase our method's ability to generalize on
two common 3D architectures, namely 3D U-Net and V-Net. Outperforming a variety
of baselines, the proposed method is capable of compressing large 3D models to
a few MBytes, alleviating the storage needs in space critical applications.Comment: Accepted to MICCAI 201
Structured Sparse Ternary Compression for Convolutional Layers in Federated Learning
In Cross-device Federated Learning, communication efficiency is of paramount importance. Sparse Ternary Compression (STC) is one of the most effective techniques for considerably reducing the per-round communication cost of Federated Learning (FL) without significantly degrading the accuracy of the global model, by using ternary quantization in series to topk sparsification. In this paper, we propose an original variant of STC that is specifically designed and implemented for convolutional layers. Our variant is originally based on the experimental evidence that a pattern exists in the distribution of client updates, namely, the difference between the received global model and the locally trained model. In particular, we have experimentally found that the largest (in absolute value) updates for convolutional layers tend to form clusters in a kernel-wise fashion. Therefore, our primary novel idea is to a-priori restrict the elements of STC updates to lay on such a structured pattern, thus allowing us to further reduce the STC communication cost. We have designed, implemented, and evaluated our novel technique, called Structured Sparse Ternary Compression (SSTC). Reported experimental results show that SSTC shrinks compressed updates by a factor of x3 with respect to traditional STC and with a reduction up to x104 with respect to uncompressed FedAvg, at the expense of negligible degradation of the global model accuracy
Breaking the Communication-Privacy-Accuracy Tradeoff with -Differential Privacy
We consider a federated data analytics problem in which a server coordinates
the collaborative data analysis of multiple users with privacy concerns and
limited communication capability. The commonly adopted compression schemes
introduce information loss into local data while improving communication
efficiency, and it remains an open problem whether such discrete-valued
mechanisms provide any privacy protection. In this paper, we study the local
differential privacy guarantees of discrete-valued mechanisms with finite
output space through the lens of -differential privacy (DP). More
specifically, we advance the existing literature by deriving tight -DP
guarantees for a variety of discrete-valued mechanisms, including the binomial
noise and the binomial mechanisms that are proposed for privacy preservation,
and the sign-based methods that are proposed for data compression, in
closed-form expressions. We further investigate the amplification in privacy by
sparsification and propose a ternary stochastic compressor. By leveraging
compression for privacy amplification, we improve the existing methods by
removing the dependency of accuracy (in terms of mean square error) on
communication cost in the popular use case of distributed mean estimation,
therefore breaking the three-way tradeoff between privacy, communication, and
accuracy. Finally, we discuss the Byzantine resilience of the proposed
mechanism and its application in federated learning
- …