Baysal Erez, Işıl

van Keulen, Maurice

English

Deep learning algorithms have became the state-of-the-art models for various tasks in a large area of applications. The most advanced deep learning models have many parameters, increasing costs, computational requirements, and memory footprints. Recently, the dynamic sparse training methods showed that it is possible to outperform the dense neural networks with sparse neural networks, while reducing the number of parameters (connections) quadratically. So far, all the proposed sparse training methods are tested on well-known benchmark datasets without data quality problems. However, in real-world data science applications, a lot of data quality challenges may appear, (e.g. missing data). Missing data can cause daunting challenges in determining the accuracy of models. Within this research, we intend to understand the interplay between dynamic sparse training methods and data sparsity for their mutual benefits

University of Twente Research Information

Understanding dynamic sparse trainingcapabilities in accommodating sparse dataIşıl Baysal Erez1[0000−0002−0636−5372], Elena Mocanu1[0000−0002−0856−579X], andMaurice van Keulen1[0000−0003−2436−1372]EEMCS, University of Twente, 7500 AE Enschede, The Netherlands{i.baysalerez, e.mocanu, m.vankeulen}@utwente.nlAbstract. Deep learning algorithms have became the state-of-the-artmodels for various tasks in a large area of applications. The most ad-vanced deep learning models have many parameters, increasing costs,computational requirements, and memory footprints. Recently, the dy-namic sparse training methods showed that it is possible to outperformthe dense neural networks with sparse neural networks, while reducingthe number of parameters (connections) quadratically. So far, all theproposed sparse training methods are tested on well-known benchmarkdatasets without data quality problems. However, in real-world data sci-ence applications, a lot of data quality challenges may appear, (e.g. miss-ing data). Missing data can cause daunting challenges in determining theaccuracy of models. Within this research, we intend to understand theinterplay between dynamic sparse training methods and data sparsityfor their mutual benefits.Keywords: Deep learning · Dynamic Sparse Training · Sparse Data1 IntroductionEnabled by the large amount of data that are continuously recorded, deep learn-ing has been proven successful for various tasks (e.g., classification, regression,clustering, feature selection, and dimensionality reduction) in a large area ofapplications. However, the most advanced deep learning models have many pa-rameters, increasing costs, computational requirements, and memory footprints.For example, Inception-V3 [14] a highly accurate object recognition network,requires 5.7 billion arithmetic operations and 27 million parameters to be eval-uated, while GPT-3 [2] an experimental natural language processing network,requires 175 billion parameters (350 GiB assuming 16 bits per parameter). Thelatest language model, GPT-4, is claimed to have 100 trillion parameters —500x the size of GPT-3. Training these huge models require extensive comput-ing facilities leading to an undesirable impact on our otherwise scarce resources(i.e. energy consumption and CO2 footprint [12]). Thus, deep learning models,in general, must be trained in the cloud and then moved on to the hardware forexploitation. This limits their flexibility drastically.2 Baysal Erez et al.2 Related workA plethora of methods have been proposed, ranging from more traditional com-pression and pruning methods [8] [11] to the most recently introduced sparsetraining methods [9] [6], aiming to obtain faster and cheaper training and in-ference for neural networks. Model pruning assumes the training of large neuralnetworks in the cloud. Then, it identifies the least meaningful connections basedon various criteria, e.g., magnitude or information metrics. After that, thoseunimportant connections are removed in order to obtain models with sparseconnectivity, which yield smaller memory footprints and faster computationaltimes during inference. The sparse models obtained following the above pro-cedures, also called dense-to-sparse models, can be as effective and even beexploited for inference purposes. Perhaps one of the most recent and populardense-to-sparse method is The Lottery Ticket Hypothesis [7]. The solutions dis-cussed above are limited by the initial need to train a very large neural networkin the cloud. Recently, it has been shown that sparse neural networks trainedfrom scratch (i.e., sparse training) can reach or even can outperform dense neu-ral networks while using much fewer computational resources, and consequently,small memory footprint, while having high representational power. The basicidea of sparse-to-sparse models is to use sparse connectivity before training inneural networks [1, 9, 10, 4, 6, 15]. This starts to appear as the “de facto” solu-tion to obtain sparse neural networks with a small number of parameters, whichmany times outperform even their very large dense counterparts. All in all, thismakes the adaptive sparse connectivity concept very suitable for the trainingand exploitation of neural networks.In this research, sparse data will be described as a subset of the data. It canbe distinguished between two cases. First one includes some missingness mecha-nisms which have been traditionally divided into three main categories, such asMissing Completely at Random (MCAR) — in which the probability of miss-ing a data depends neither of observed or unobserved data, Missing at Random(MAR) — in which the probability of missing a data depends just on observeddata, and Missing not at Random (MNAR) — in which the probability of miss-ing a data depends on observed data conditioned by observed measurements [13].Second case includes, when all data are present, but the artificial neural networkcan not model them correctly. For example, data which are recognized to beeasy-to-forget or hard-to-be-memorized.3 Sparse data and sparse training modelsWe envision a general framework that accounts for mutual benefits betweensparse data and sparse models. The main PhD research is split into two mainparts. In the first part, we have started investigating the learning capabilities ofdynamic sparse training methods to account for sparse data (e.g. missing data)in a given incomplete observability environment. In the second part, we aim toexplore and extend the dynamic sparse training methods to accommodate specialUnderstanding DST capabilities in accommodating sparse data 3types of data recognized as being hard to perceive by the neural network model(e.g. easy-to-forget or hard-to-memorize) and the main well-known challengesrelated to it. The general flow of information, including the connection betweenboth parts, can be seen in Figure 1.Part I  Can sparse training benefit from a data-efficient solution? How should the sparse training models & sparse data interact? Can they possibly be simultaneously used to drive training efficiency to the next level? Sparse modelsPart IIAre the dynamic sparse training algorithms able to accommodate the hard-to-memorize and easy-to-forget samples? To what extent can we find a subset of data that allows us to overcome th  catastrophic forgetting problem in sparse neural networks? Sparse dataWhen an artificial neural network consider to have as an input sparse data?DataSparsedataWhat are sparse data?• We refer to sparse data as a subset of the datai.baysalerez@utwente.nlWhat are sparse training models?Neural NetworksSparseNNs• We refer to sparse models as being artificial neural networks with a subset of connections missing.ResultsPart IPart IIConclusionsReferencesMissing dataEasy to forget dataIncomplete modelingIncomplete observabilityHard to memorizedataMCAR MAR MNARIterative pruningkTrain PruneTraining timeGradient regrowPruneRandom regrowTrain high sparsityTrain denseTrain low sparsityPruningStatic sparsetrainingDynamic sparsetrainingApproachxjxi……………………+Random regrowMagnitude removalSparse initializationSparse Evolutionary Training Sparse models meets all the data[1] I. Baysal Erez, The mutually reinforcing of sparse data and sparse training models, SIAM International Conference on Data Mining (SDM22), Doctoral Consortium, 28-30 April, 2022.[2] D. C. Mocanu, E. Mocanu, P. Stone, P.H. Nguyen, M. Gibescu, A. Liotta, Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science, Nature Communications, 9(1), 1-12, 2018.New dataContinual learning Hard to memorizedataMulti-task learningMemory replaySparseNNsSparse data The mutually reinforcing of sparse data and sparse training models[1]Işıl Baysal ErezPhD Advisers: Elena Mocanu and Maurice van KeulenUniversity of Twente, Enschede, the Netherlands• Our initial results using data in an incomplete observability environment showed a great potential towards an impressive computational reduction and an increase in the accuracy.• These results are yet limited to one dynamic sparse training algorithm (i.e., Sparse Evolutionary Training [1]) and one dataset where the missing completely at random features have been simulated (Breast Cancer Wisconsin). • Further theoretical considerations and empirical results are currently under development to generalize these results and support these impactful claims.Sparse tabular data• Missing completely at random (MCAR) features• Various levels of data and model sparsities• Static MLP has an Erdös-Rényi random sparse  initializationExtreme sparse regimeComparison between a dense MLP, a static MLP, and SET-MLPFig. 1. Schematic representation of main research components and their interrelations.Sparse data includes all three missing data mechanisms: MCAR, MAR and MNAR.Black hash represents a missing data point, and orange and green shadows representsthe conditional dependence to that specific missing point in case of missing at random.Part I consider the sparse data as input into a dynamic sparse model (i.e. SparseEvolutionary Training [9]). Part II focus on understanding the possible difficulties toperceive the given data and their reflection on online learning tasks.Part I: Incomplete data observability In real-world data science appli-cations, quite often, the neural networks models should be able to accommodatevarious data quality aspects (e.g., missing data) and take advantage in a data-efficient way of all of them. In this first part, we want to go one step beyond thestate-of-the-art and understand the interplay between “sparse data” and “sparsetraining models”. This lead us to the following research questions:4 Baysal Erez et al.Can sparse training benefit from a data-efficient solution? How should the“sparse training models” and “sparse data” interact? Can they possibly be si-multaneously used to drive training efficiency to the next level?Part II: Incomplete modeling In this second part, we aim to investigatethe learning capabilities of sparse neural networks in a fully-observable environ-ment, where all the data are present, but the neural network may not perceivethem. Recently, it has been observed that deep neural networks can have samplesthat are either hard-to-memorize during training or easy-to-forget during prun-ing. Prior work has shown that using a specifically selected subset of data ratherthan the all training set, a dense-to-sparse model may overcome these issuesand identify lottery tickets more effectively – leading to the general idea for thisparticular pruning models [16]. Based on our knowledge, the hard-to-memorizeand easy-to-forget sample problems have not been analyzed in the context ofany sparse-to-sparse models. Consequently, in the second part of this research,the following research questions will be investigated:To what extent can we find a subset of data that allows us to overcomethe catastrophic forgetting problem in sparse neural networks? Are the dynamicsparse training algorithms able to accommodate the hard-to-memorize and easy-to-forget samples?4 Preliminary ResultsWe are considering the Wisconsin Breast Cancer [5] dataset, a very simple multi-variate benchmark dataset with two classes, benign and malign, and 32 variables.We simulate the MCAR data using a set-aside masked values from the full datamatrix. The obtained sparse data are then fed into a multi-layer perceptron(MLP) and trained using a sparse evolutionary training (SET) procedure. Wecompare the learning capabilities and computational performance of SET-MLPwith a dense MLP and a static sparse MLP (SS-MLP).For all three models, we use similar hyperparameters and a 30× 100× 100×100 × 2 architecture. For the three hidden layers, we use a ReLU activationfunction, while for the output layer, we use Softmax. During training, we usea stochastic gradient descend optimizer and a cross-entropy loss function. Fur-thermore, we use a learning rate of 0.001, a momentum of 0.9, a batch size of10, and a dropout rate of 0.1. The initial sparsity level for SET-MLP and StaticSparse MLP varies, while for SET-MLP the evolution of the weights is consid-ered with a ζ = 0.1 rate. All models have been run three times using a trulysparse implementation, building on top of the Curci et al. [3] code.In Fig. 2 (left), we plot the accuracy of SET-MLP (solid lines) versus SS-MLP (dashed lines) and their dense MLP counterpart (black line) in the caseof low (i.e. 25%) and moderate (i.e. 50% and 75%) model sparsities. We canobserve that both resource-aware models, SET-MLP and SS-MLP, are oftenmore accurate than the dense network while having a quadratically reducednumber of parameters. Furthermore, as we increase the level of sparsity towardthe high sparsity regime, as can be seen in Fig.2 (right), all three methods showUnderstanding DST capabilities in accommodating sparse data 50 10 20 30 40 50 60 70 80 90MCAR - Data sparsity [%]92939495969798Wisconsin Breast CancerAccuracy [%]Static MLP (75% sparsity)Static MLP (50% sparsity)Static MLP (25% sparsity)SET MLP (75% sparsity)SET MLP (50% sparsity)SET MLP (25% sparsity)Dense MLP0 10 20 30 40 50 60 70 80 90MCAR - Data sparsity [%]65707580859095100Wisconsin Breast CancerAccuracy [%]Static MLP (97% sparsity)Static MLP (95% sparsity)Static MLP (92% sparsity)Static MLP (90% sparsity)SET MLP (97% sparsity)SET MLP (95% sparsity)SET MLP (92% sparsity)SET MLP (90% sparsity)Dense MLPFig. 2. Comparison over Wisconsin Breast Cancer dataset between a dense MLP, astatic sparse MLP, and SET-MLP in (left) a low to moderate sparse regime and (right)an extreme sparse training regime.good generalization capabilities and similar accuracy. Beyond a 90% reduction inboth the data input used and model parameters, the huge gain in computationalperformance is reflected in a slight decrease in accuracy.Our initial results using data in an incomplete observability environmentshow a great potential towards an impressive computational reduction and anincrease in accuracy. These results are yet limited to one dynamic sparse trainingalgorithm (i.e., Sparse Evolutionary Training [9] and one dataset where the miss-ing completely at random features have been simulated (Breast Cancer Wiscon-sin) [5]. Further theoretical considerations and empirical results are currently un-der development to generalize these results and support these impactful claims.5 AcknowledgmentsThis research has been partly funded by the Turkey’s Ministry of National Ed-ucation study-abroad program.References1. Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deeprewiring: Training very sparse deep networks. arXiv preprint arXiv:1711.05136,2017.2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Pra-fulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell,et al. Language models are few-shot learners. Advances in neural information pro-cessing systems, 33:1877–1901, 2020.3. Selima Curci, Decebal Constantin Mocanu, and Mykola Pechenizkiyi. Truly sparseneural networks at scale, 2021.4. Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster trainingwithout losing performance. arXiv preprint arXiv:1907.04840, 2019.5. Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.6. Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen.Rigging the lottery: Making all tickets winners. In International Conference onMachine Learning, pages 2943–2952. PMLR, 2020.6 Baysal Erez et al.7. Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Findingsparse, trainable neural networks. arXiv preprint arXiv:1803.03635, 2018.8. Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances inneural information processing systems, 2, 1989.9. Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen,Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural net-works with adaptive sparse connectivity inspired by network science. Nature com-munications, 9(1):1–12, 2018.10. Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolu-tional neural networks by dynamic sparse reparameterization. In InternationalConference on Machine Learning, pages 4646–4655. PMLR, 2019.11. Michael C Mozer and Paul Smolensky. Using relevance to reduce network sizeautomatically. Connection Science, 1(1):3–16, 1989.12. David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia,Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions andlarge neural network training. arXiv preprint arXiv:2104.10350, 2021.13. Donald B Rubin. Inference and missing data. Biometrika, 63(3):581–592, 1976.14. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and ZbigniewWojna. Rethinking the inception architecture for computer vision. In Proceedingsof the IEEE conference on computer vision and pattern recognition, pages 2818–2826, 2016.15. Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu, YifanGong, Zheng Zhan, Chaoyang He, Qing Jin, et al. Mest: Accurate and fast memory-economic sparse training framework on the edge. Advances in Neural InformationProcessing Systems, 34, 2021.16. Zhenyu Zhang, Xuxi Chen, Tianlong Chen, and Zhangyang Wang. Efficient lotteryticket finding: Less data is more. In International Conference on Machine Learning,pages 12380–12390. PMLR, 2021.

Understanding dynamic sparse training capabilities in accommodating sparse data

https://ris.utwente.nl/ws/files/287256562/Understanding_dynamic_sparse_training_capabilities_in_accommodating_sparse_data.pdf

Understanding dynamic sparse training capabilities in accommodating sparse data

Abstract

Similar works

Full text

Available Versions

University of Twente Research Information