1,653 research outputs found
Learning Sparse & Ternary Neural Networks with Entropy-Constrained Trained Ternarization (EC2T)
Deep neural networks (DNN) have shown remarkable success in a variety of
machine learning applications. The capacity of these models (i.e., number of
parameters), endows them with expressive power and allows them to reach the
desired performance. In recent years, there is an increasing interest in
deploying DNNs to resource-constrained devices (i.e., mobile devices) with
limited energy, memory, and computational budget. To address this problem, we
propose Entropy-Constrained Trained Ternarization (EC2T), a general framework
to create sparse and ternary neural networks which are efficient in terms of
storage (e.g., at most two binary-masks and two full-precision values are
required to save a weight matrix) and computation (e.g., MAC operations are
reduced to a few accumulations plus two multiplications). This approach
consists of two steps. First, a super-network is created by scaling the
dimensions of a pre-trained model (i.e., its width and depth). Subsequently,
this super-network is simultaneously pruned (using an entropy constraint) and
quantized (that is, ternary values are assigned layer-wise) in a training
process, resulting in a sparse and ternary network representation. We validate
the proposed approach in CIFAR-10, CIFAR-100, and ImageNet datasets, showing
its effectiveness in image classification tasks.Comment: Proceedings of the CVPR'20 Joint Workshop on Efficient Deep Learning
in Computer Vision. Code is available at
https://github.com/d-becking/efficientCNN
Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edge
We envision a mobile edge computing (MEC) framework for machine learning (ML)
technologies, which leverages distributed client data and computation resources
for training high-performance ML models while preserving client privacy. Toward
this future goal, this work aims to extend Federated Learning (FL), a
decentralized learning framework that enables privacy-preserving training of
models, to work with heterogeneous clients in a practical cellular network. The
FL protocol iteratively asks random clients to download a trainable model from
a server, update it with own data, and upload the updated model to the server,
while asking the server to aggregate multiple client updates to further improve
the model. While clients in this protocol are free from disclosing own private
data, the overall training process can become inefficient when some clients are
with limited computational resources (i.e. requiring longer update time) or
under poor wireless channel conditions (longer upload time). Our new FL
protocol, which we refer to as FedCS, mitigates this problem and performs FL
efficiently while actively managing clients based on their resource conditions.
Specifically, FedCS solves a client selection problem with resource
constraints, which allows the server to aggregate as many client updates as
possible and to accelerate performance improvement in ML models. We conducted
an experimental evaluation using publicly-available large-scale image datasets
to train deep neural networks on MEC environment simulations. The experimental
results show that FedCS is able to complete its training process in a
significantly shorter time compared to the original FL protocol
- …