With the rise of Artificial Intelligence (AI), it is becoming a significant phenomenon in our lives. As with many other powerful tools, AI brings many advantages but many risks as well. Predictions and automation can significantly help in our everyday lives. However, sending our data to servers for processing can severely hurt our privacy. In this paper, we describe experiments designed to find out whether we can enjoy the benefits of AI in the privacy of our mobile devices. We focus on text data since such data are easy to store in large quantities for mining by third parties. We measure the performance of deep learning methods in terms of accuracy (when compared to fully-fledged server models) and speed (number of text documents processed in a second). We conclude our paper with findings that with few relatively small modifications, mobile devices can process hundreds to thousands of documents while leveraging deep learning models

Konopík, Miloslav

Sido, Jakub

University of West Bohemia Digital Library

Deep Learning for Text Data on Mobile Devices

DSpace at University of West Bohemia

Deep Learning for Text Data on MobileDevices1st Jakub SidoDept. of Computer Science and EngineeringUniversity of West BohemiaPilsen, Czech Republicsidoj@kiv.zcu.cz2nd Miloslav Konopı´kDept. of Computer Science and EngineeringUniversity of West BohemiaPilsen, Czech Republickonopik@kiv.zcu.czAbstract—With the rise of Artificial Intelligence (AI),it is becoming a significant phenomenon in our lives.As with many other powerful tools, AI brings manyadvantages but many risks as well. Predictions andautomation can significantly help in our everyday lives.However, sending our data to servers for processing canseverely hurt our privacy. In this paper, we describeexperiments designed to find out whether we can enjoythe benefits of AI in the privacy of our mobile devices.We focus on text data since such data are easy tostore in large quantities for mining by third parties. Wemeasure the performance of deep learning methods interms of accuracy (when compared to fully-fledged servermodels) and speed (number of text documents processedin a second). We conclude our paper with findings thatwith few relatively small modifications, mobile devicescan process hundreds to thousands of documents whileleveraging deep learning models.Index Terms—Deep learning, neural networks, mobilecomputing, CNN, LSTM.I. INTRODUCTIONThis paper deals with topics of machine learning(part of AI) and privacy. We attempt to investigatewhether the current technical means enable users toprocess their data within the privacy of their mobiledevices. Current mobile devices are usually well pro-tected with various biometric security authenticationmethods. If the mobile devices are sufficiently power-ful and the software framework mature enough, theremight be the possibility to execute machine learningalgorithms directly on the devices. This way, users arenot forced to transfer their data out of the security oftheir mobile devices.We decided to focus on text data. There are alreadyquite a few benchmarks for image processing 1 [1].Text data are, however, entirely different. The sen-tences have variable lengths (images can be re-scaled).The models for text also require large matrices of wordembeddings 2.We design our experiments in the Tensorflow Lite,which is a mobile version of Tensorflow [2]. Tensor-flow is an open-source machine learning tool focusedmainly on artificial neural networks released by Googlein 2015. Tensorflow supports conversion of modelscomputed on powerful computers to mobile devices.1See e.g. http://ai-benchmark.com/.2Word embeddings are in principle large dictionaries of wordsand their vector representations. The models require vocabularies ofmore than 10,000 words.However, Tensorflow Lite supports only a subset of op-erations and the models have to be simplified in somecases. A quantization process can further optimize theTensorflow models (see section IV-C). It significantlyreduces the computation time of the models at the costsof small accuracy degradation.To summarize, in this paper, we attempt to reach thefollowing goals:• Benchmark the computational performance of thedeep learning models on mobile devices.• Measure the accuracy drop of quantization.• Examine the support of complex models in theTensorflow Lite tool on mobile devices. Discoverthe accuracy penalty of simpler models.II. SELECTED TASKFor our experiments, we opted for the task of Senti-ment Analysis. The goal of the task is to determine theoverall sentiment of a given document. For example,a sentence “A good film all round.” bears a positivesentiment and “The film, to put it even more bluntly, isa total bore and would appeal to no one but perhapsthose who made the film.” expresses clearly a negativesentiment.The task requires some degree of text understandingand the algorithm needs to deal with several NLP3challenges such as figurative language (irony, sarcasm,and metaphor), negation, ambiguity (some positivewords can be used in a negative context) and others.III. TRAINING AND EVALUATION DATASETSA. CSFD Sentiment DatasetThe CSFD dataset consists of 91,381 movie reviewswritten in Czech from the Czech Movie Database 4.The reviews are split into three categories accordingto their star rating (0-1 stars as negative, 2-3 starsas neutral, 4-5 stars as positive). The dataset contains30,897 positive, 30,768 neutral, and 29,716 negativereviews. 82,244 reviews are used for training and 9,137for testing. More details about the dataset can be foundin [3].3Natural Language Processing4https://www.csfd.cz/ISBN 978-80-261-0812-2, © University of West Bohemia, 2019B. IMDb Sentiment DatasetThe IMDb dataset [4] contains 25,000 training and25,000 evaluation movie reviews written in Englishfrom IMDb 5. Because some movies receive substan-tially more reviews than others, only at most 30 reviewsfrom any movie are included in the collection. Ratingson IMDb are given as star values (1, 2, ..., 10), whichare mapped to 0 and 1 (0: stars ≤ 5, 1 : stars > 6)to label sentiment categories.IV. DEEP LEARNING MODELSFor our experiments, we employ two populardeep learning architectures: Long Short-Term Memory(LSTM) – see section IV-A and Convolutional NeuralNetwork (CNN) – see section IV-B.A. LSTMLSTM [5] belongs in the recurrent neural network(RNN) class of artificial neural networks. RNNs pro-cess input data in sequences and encode them intoa hidden vector (memory). LSTMs improves vanillaRNNs by adding an ability to control storing anddeleting information from memory. With more precisecontrol of the memory, it can store substantially longersequences.In our architecture (see Figure 1), we first transformwords into their vector representations using an em-bedding layer. For the CSFD dataset, we use randomlyinitialized vectors with a dimension of 60. Since thetraining part of IMDb dataset is almost three timessmaller, we use pre-trained vectors from the FastTexttool [6] in this case.After the embedding layer, we attach two stackedLSTM layers and one fully connected softmax layer.During the implementation phase of our experi-ments, we struggled heavily with the conversion toTensorflow Lite models. Currently, the RNNs imple-mentation is highly experimental in Tensorflow Liteand it seems that it does not support variable sequencelengths. Support of variable sequences is, however,crucial for text data. If we pad sequences, e.g. withan end-of-sequence token, we get approximately 10%drop of accuracy. To solve this issue, we have firstopened an issue on GitHub and contacted the commu-nity. However, they have confirmed our findings thatTesorflow Lite does not support processing sequenceswith variable length. We have solved this issue bycomputing LSTMs for the whole padded sequences.Then we select appropriate end states of the LSTMby an advanced indexing tensor operation (gather-nd).Our solution is available on GitHub6.We use the maximum length of sequences set to 150words and the dimension of hidden states is set to 64.B. CNNConvolutional Neural Networks apply a convolutionoperation to the input followed by a pooling operation(computing maximum or average on a window ofconvolution output) [7].5https://www.IMDb.com/6https://github.com/konopik/tflite-lstm-textWord EmbedingTimeLSTM LSTM LSTM LSTM LSTM LSTM LSTM LSTM LSTM LSTM 0.080.790.13Fully-Connected Softmax 1st Layer  2nd LayerFig. 1. LSTM architecture. 32 x T + 32 x T + 16 x T  Convolution Fully-Connected Softmax Word Embeding Max-over-time pooling 32+32+16  Dropout50%Fully-Connected 9 TFig. 2. CNN architecture.In our architecture, we use the same embeddinglayer as in the LSTM architecture and add only oneconvolution layer followed by max pooling. This ap-proach is quite standard for text processing. We use2D convolution kernels where one dimension is set tothe embedding dimension and the other one is set to2, 3 and 4 respectively. This way the network looksfor patterns in 2, 3 and 4 consequent words. We apply32 2-word kernels, 32 3-word kernels and 16 4-wordkernels.In the end, we connect two fully connected layers,one with Sigmoid activation, the other one with soft-max (see Figure 2).C. Post-training quantizationTensorflow stores the weights in a 32-bit floatingpoint format by default. In order to increase the compu-tation speed, Tensorflow allows converting (quantize)the weights into 8-bit fixed point format. In the post-training quantization approach, Tensorflow uses hybridoperators, which compute the activations in an 8-bitfixed point format. However, the outputs are still storedusing floating point.The post-training quantization process increasesmodel computation speed, decreases inference latencyand model size. The costs of quantization consist ofreduced accuracy of the model. We measure bothcomputation speed difference and accuracy differencein our experiments.V. EXPERIMENTS AND DISCUSSIONWe conduct experiments where we measure thecomputation speed of the models (in terms of a numberof processed reviews per second), the CPU load ofmodel execution and the accuracy of the models. Inour experiments, we distinguish between original andquantized models. We measure both proposed architec-tures, LSTM based and convolutional neural networks.In Figures 3 – 8, we show the computation speed insolid lines and the CPU load in dotted lines.For the CSFD dataset, we measure the computationspeed on two devices, Huawei p9 lite and Google Pixel3XL. For the IMDb dataset, we show the results onlyon Huawei p9 lite.We can draw several conclusions from the obtainedresults. First, the CNN networks in Tensorflow Lite areby one order of magnitude faster then LSTMs. Second,the quantization has signification effect of reducingcomputation costs of the models. In CNNs, this showsin CPU load where the quantized models requireroughly half of the CPU allocation when compared tooriginal models. However, it does not transfer to theincreased speed of computation, which is comparablefor both quantized and original models. In LSTMs,the CPU load is almost identical, but the quantizedmodels are approximately twice as fast. Third, a high-performance mobile device (such as the Google Pixel3XML we tested) is capable of processing hundredsto thousands of reviews per second. More than enoughfor handling users’ data. Taking into account the rateof technological progress, we can safely expect thatprocessor in today’s high-performance models will bea norm soon.In Table I, we compare the accuracies of predict-ing the correct sentiment for both CSFD and IMDbdatasets and both LSTM and CNN architectures. Tocompare with state of the art, we add the best resultspublished so far. For the CSFD, we use results fromthe original dataset publication [3]. There is a newerpublication available, however, the authors use externalinformation (association of the reviews to particularmovies) and they obtain only a minor improvement(+1.5%). For this dataset, we can reach the accuracyof the state-of-the-art models.For the IMDb dataset, the best-published results[8] obtain astonishing accuracy of 95%. In this case,we reach significantly lower accuracies. However, webelieve that with careful fine-tuning, we would be ableto decrease the gap slightly. Nevertheless, we think thatthe obtained accuracy is sufficient enough for the endusers.TABLE IMODEL ACCURACIESDataset CSFD IMDbModel Original Quantized Original QuantizedLSTM 80 79 86 86CNN 77 76 83 81SoTA 81 – 95 –VI. CONCLUSIONOur experiments show that even an average mobilephone can process tens to hundreds of text documents2 4 8 16 32 64 128 5120200400600Batch sizeReviewspersecondplot 1plot 2020406080100CPU[%]Original QuantiziedOriginal CPU CPU Quant.Fig. 3. CNN – CSFD – Huawei p9 lite2 4 8 16 32 64 128 5120200400600800Batch sizeReviewspersecondplot 1plot 2020406080100CPU[%]Original QuantiziedOriginal CPU CPU Quant.Fig. 4. CNN – IMDB – Huawei p9 lite2 4 8 16 32 64 128 5120102030Batch sizeReviewspersecondplot 1plot 2020406080100CPU[%]Original QuantiziedOriginal CPU CPU Quant.Fig. 5. LSTM – CSFD – Huawei p9 liteper second. To put these numbers into a context, adesktop computer (6-core Intel CPU) with a GPUaccelerator (Nvidia GTX 1080Ti) can process up to 50times more requests per second than a powerful mobiledevice (such as Google Pixel 3XL). We estimate thatwith more complex models, the difference between amobile device and a server may increase even more.2 4 8 16 32 64 128 5120102030Batch sizeReviewspersecondplot 1plot 2020406080100CPU[%]Original QuantiziedOriginal CPU CPU Quant.Fig. 6. LSTM – IMDb – Huawei p9 lite2 4 8 16 32 64 128 51205001,0001,500Batch sizeReviewspersecondplot 1plot 2020406080100CPU[%]Original QuantiziedOriginal CPU CPU Quant.Fig. 7. CNN – CSFD – PIXEL 3XL2 4 8 16 32 64 128 512050100Batch sizeReviewspersecondplot 1plot 2020406080100CPU[%]Original QuantizedOriginal CPU CPU Quant.Fig. 8. LSTM – CSFD – PIXEL 3XLOur results indicate that the state-of-the-art modelsdeliver higher accuracy than our simplified ones. How-ever, the accuracy of simplified models is still sufficientaccording to our opinion. Therefore, we conclude thatit is technically possible to process user data directlyon mobile devices. Users can enjoy the possibilities topredict a match of their interests, analyze post on theirsocial network walls, extract information from their e-mails and more without the necessity to share theirdata with third parties.VII. ACKNOWLEDGEMENTThis work has been supported by Grant No. SGS-2019-018 Processing of heterogeneous data and itsspecialized applications, and was partly supportedfrom ERDF “Research and Development of IntelligentComponents of Advanced Technologies for the PilsenMetropolitan Area (InteCom)” and by the projectLO1506 of the Czech Ministry of Education, Youth andSports. Computational resources were supplied by theMinistry of Education, Youth and Sports of the CzechRepublic under the Projects CESNET (Project No.LM2015042) and CERIT-Scientific Cloud (Project No.LM2015085) provided within the program “Projects ofLarge Research, Development and Innovations Infras-tructures”.REFERENCES[1] Andrey Ignatov, Radu Timofte, William Chou, Ke Wang, MaxWu, Tim Hartley, and Luc Van Gool. AI benchmark: Run-ning deep neural networks on android smartphones. CoRR,abs/1810.01109, 2018.[2] Martı´n Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo,Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jef-frey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow,Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia,Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Lev-enberg, Dandelion Mane´, Rajat Monga, Sherry Moore, DerekMurray, Chris Olah, Mike Schuster, Jonathon Shlens, BenoitSteiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, VincentVanhoucke, Vijay Vasudevan, Fernanda Vie´gas, Oriol Vinyals,Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, andXiaoqiang Zheng. TensorFlow: Large-scale machine learningon heterogeneous systems, 2015. Software available fromtensorflow.org.[3] Ivan Habernal, Toma´sˇ Pta´cek, and Josef Steinberger. Sentimentanalysis in czech social media using supervised machine learn-ing. In Proceedings of the 4th Workshop on Computational Ap-proaches to Subjectivity, Sentiment and Social Media Analysis,pages 65–74, 2013.[4] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang,Andrew Y. Ng, and Christopher Potts. Learning word vectorsfor sentiment analysis. In Proceedings of the 49th Annual Meet-ing of the Association for Computational Linguistics: HumanLanguage Technologies - Volume 1, HLT ’11, pages 142–150,Stroudsburg, PA, USA, 2011. Association for ComputationalLinguistics.[5] Sepp Hochreiter and Ju¨rgen Schmidhuber. Lstm can solve hardlong time lag problems. In Advances in neural informationprocessing systems, pages 473–479, 1997.[6] Piotr Bojanowski, Edouard Grave, Armand Joulin, and TomasMikolov. Enriching word vectors with subword information.Transactions of the Association for Computational Linguistics,5:135–146, 2017.[7] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard,W. Hubbard, and L. D. Jackel. Backpropagation applied to hand-written zip code recognition. Neural Computation, 1(4):541–551, Dec 1989.[8] Jeremy Howard and Sebastian Ruder. Universal language modelfine-tuning for text classification. In Proceedings of the 56th An-nual Meeting of the Association for Computational Linguistics(Volume 1: Long Papers), pages 328–339, Melbourne, Australia,July 2018. Association for Computational Linguistics.

English

http://dspace5.zcu.cz/bitstream/11025/35532/1/Sido.pdf

Deep Learning for Text Data on Mobile Devices

Abstract

Similar works

Full text

Available Versions

University of West Bohemia Digital Library

DSpace at University of West Bohemia