6 research outputs found
Incremental Learning Using a Grow-and-Prune Paradigm with Efficient Neural Networks
Deep neural networks (DNNs) have become a widely deployed model for numerous
machine learning applications. However, their fixed architecture, substantial
training cost, and significant model redundancy make it difficult to
efficiently update them to accommodate previously unseen data. To solve these
problems, we propose an incremental learning framework based on a
grow-and-prune neural network synthesis paradigm. When new data arrive, the
neural network first grows new connections based on the gradients to increase
the network capacity to accommodate new data. Then, the framework iteratively
prunes away connections based on the magnitude of weights to enhance network
compactness, and hence recover efficiency. Finally, the model rests at a
lightweight DNN that is both ready for inference and suitable for future
grow-and-prune updates. The proposed framework improves accuracy, shrinks
network size, and significantly reduces the additional training cost for
incoming data compared to conventional approaches, such as training from
scratch and network fine-tuning. For the LeNet-300-100 and LeNet-5 neural
network architectures derived for the MNIST dataset, the framework reduces
training cost by up to 64% (63%) and 67% (63%) compared to training from
scratch (network fine-tuning), respectively. For the ResNet-18 architecture
derived for the ImageNet dataset and DeepSpeech2 for the AN4 dataset, the
corresponding training cost reductions against training from scratch (network
fine-tunning) are 64% (60%) and 67% (62%), respectively. Our derived models
contain fewer network parameters but achieve higher accuracy relative to
conventional baselines
Input-dependent edge-cloud mapping of recurrent neural networks inference
6noGiven the computational complexity of Recurrent Neural Networks (RNNs) inference, IoT and mobile devices typically offload this task to the cloud. However, the execution time and energy consumption of RNN inference strongly depends on the length of the processed input. Therefore, considering also communication costs, it may be more convenient to process short input sequences locally and only offload long ones to the cloud. In this paper, we propose a low-overhead runtime tool that performs this choice automatically. Results based on real edge and cloud devices show that our method is able to simultaneously reduce the total execution time and energy consumption of the system compared to solutions that run RNN inference fully locally or fully in the cloud.partially_openopenJahier Pagliari D.; Chiaro R.; Chen Y.; Vinco S.; Macii E.; Poncino M.Jahier Pagliari, D.; Chiaro, R.; Chen, Y.; Vinco, S.; Macii, E.; Poncino, M
DiabDeep: Pervasive Diabetes Diagnosis based on Wearable Medical Sensors and Efficient Neural Networks
Diabetes impacts the quality of life of millions of people. However, diabetes
diagnosis is still an arduous process, given that the disease develops and gets
treated outside the clinic. The emergence of wearable medical sensors (WMSs)
and machine learning points to a way forward to address this challenge. WMSs
enable a continuous mechanism to collect and analyze physiological signals.
However, disease diagnosis based on WMS data and its effective deployment on
resource-constrained edge devices remain challenging due to inefficient feature
extraction and vast computation cost. In this work, we propose a framework
called DiabDeep that combines efficient neural networks (called DiabNNs) with
WMSs for pervasive diabetes diagnosis. DiabDeep bypasses the feature extraction
stage and acts directly on WMS data. It enables both an (i) accurate inference
on the server, e.g., a desktop, and (ii) efficient inference on an edge device,
e.g., a smartphone, based on varying design goals and resource budgets. On the
server, we stack sparsely connected layers to deliver high accuracy. On the
edge, we use a hidden-layer long short-term memory based recurrent layer to cut
down on computation and storage. At the core of DiabDeep lies a grow-and-prune
training flow: it leverages gradient-based growth and magnitude-based pruning
algorithms to learn both weights and connections for DiabNNs. We demonstrate
the effectiveness of DiabDeep through analyzing data from 52 participants. For
server (edge) side inference, we achieve a 96.3% (95.3%) accuracy in
classifying diabetics against healthy individuals, and a 95.7% (94.6%) accuracy
in distinguishing among type-1/type-2 diabetic, and healthy individuals.
Against conventional baselines, DiabNNs achieve higher accuracy, while reducing
the model size (FLOPs) by up to 454.5x (8.9x). Therefore, the system can be
viewed as pervasive and efficient, yet very accurate