32 research outputs found
EENED: End-to-End Neural Epilepsy Detection based on Convolutional Transformer
Recently Transformer and Convolution neural network (CNN) based models have
shown promising results in EEG signal processing. Transformer models can
capture the global dependencies in EEG signals through a self-attention
mechanism, while CNN models can capture local features such as sawtooth waves.
In this work, we propose an end-to-end neural epilepsy detection model, EENED,
that combines CNN and Transformer. Specifically, by introducing the convolution
module into the Transformer encoder, EENED can learn the time-dependent
relationship of the patient's EEG signal features and notice local EEG abnormal
mutations closely related to epilepsy, such as the appearance of spikes and the
sprinkling of sharp and slow waves. Our proposed framework combines the ability
of Transformer and CNN to capture different scale features of EEG signals and
holds promise for improving the accuracy and reliability of epilepsy detection.
Our source code will be released soon on GitHub.Comment: Accepted by IEEE CAI 202
Fast Training of NMT Model with Data Sorting
The Transformer model has revolutionized Natural Language Processing tasks
such as Neural Machine Translation, and many efforts have been made to study
the Transformer architecture, which increased its efficiency and accuracy. One
potential area for improvement is to address the computation of empty tokens
that the Transformer computes only to discard them later, leading to an
unnecessary computational burden. To tackle this, we propose an algorithm that
sorts translation sentence pairs based on their length before batching,
minimizing the waste of computing power. Since the amount of sorting could
violate the independent and identically distributed (i.i.d) data assumption, we
sort the data partially. In experiments, we apply the proposed method to
English-Korean and English-Luganda language pairs for machine translation and
show that there are gains in computational time while maintaining the
performance. Our method is independent of architectures, so that it can be
easily integrated into any training process with flexible data lengths
Quantized Transformer Language Model Implementations on Edge Devices
Large-scale transformer-based models like the Bidirectional Encoder
Representations from Transformers (BERT) are widely used for Natural Language
Processing (NLP) applications, wherein these models are initially pre-trained
with a large corpus with millions of parameters and then fine-tuned for a
downstream NLP task. One of the major limitations of these large-scale models
is that they cannot be deployed on resource-constrained devices due to their
large model size and increased inference latency. In order to overcome these
limitations, such large-scale models can be converted to an optimized
FlatBuffer format, tailored for deployment on resource-constrained edge
devices. Herein, we evaluate the performance of such FlatBuffer transformed
MobileBERT models on three different edge devices, fine-tuned for Reputation
analysis of English language tweets in the RepLab 2013 dataset. In addition,
this study encompassed an evaluation of the deployed models, wherein their
latency, performance, and resource efficiency were meticulously assessed. Our
experiment results show that, compared to the original BERT large model, the
converted and quantized MobileBERT models have 160 smaller footprints
for a 4.1% drop in accuracy while analyzing at least one tweet per second on
edge devices. Furthermore, our study highlights the privacy-preserving aspect
of TinyML systems as all data is processed locally within a serverless
environment.Comment: Accepted for publication on 22nd International Conference of Machine
Learning and Applications, ICMLA 202