Neural networks present big popularity and success in many fields. The large training
time process problem is a very important task nowadays. In this paper, a new
approach to get over this issue based on reducing dataset size is proposed. Two
algorithms covering two different shape notions are shown and experimental results
are given.Ministerio de Economía y Competitividad MTM2015-67072-

González Díaz, Rocío

Gutiérrez Naranjo, Miguel Ángel

Paluzo Hidalgo, Eduardo

English

idUS. Depósito de Investigación Universidad de Sevilla

Representative datasets for neural networksRocio Gonzalez-Diaz 1,2, Eduardo Paluzo-Hidalgo 3Dept. Applied Math IUniversity of Seville, Seville, SpainMiguel A. Gutiérrez-Naranjo 4Dept. Computer Sciences and Artificial IntelligenceUniversity of Seville, Seville, SpainAbstractNeural networks present big popularity and success in many fields. The large train-ing time process problem is a very important task nowadays. In this paper, a newapproach to get over this issue based on reducing dataset size is proposed. Twoalgorithms covering two different shape notions are shown and experimental resultsare given.Keywords: Algorithm, α-shape, proximity graph, neural networks.1 Introduction and backgroundReducing time in the training process in Machine Learning algorithms is avivid area nowadays. It is on the basis of many current tools as the use of1 Email: rogodi@us.es2 Partially supported by MINECO, FEDER/UE under grant MTM2015-67072-P. 3 Email: epaluzo@us.es4 Email: magutier@us.esContrastive Divergence in Restricted Boltzmann Machines or even in the mo-tivation for the use of Convolutional Neural Networks [6]. Many of the effortshave the aim of reducing the dimensionality of the data, as the classic Prin-cipal Component Analysis method [7]. In this paper, a different approach isproposed. The first key idea is that close point data provide similar infor-mation in the training process and therefore, training time can be reducedby removing some of such similar data without losing accuracy. In order toachieve such size reduction, the dataset is embedded in a metric space anda new pooling strategy based on proximity graphs density is proposed. Thesecond key in this paper is based on the α-shape of the point cloud [1]. Weshow that such subset of points has vital information for training a neural net.In this paper, the Hausdorff distance is considered to measure the distance be-tween sets in Rn with the Euclidean metric and a fully connected feedforwardmultilayer perceptron is used as the Machine Learning tool.Let G be an undirected graph, V (G) the set of vertices and E(G) theset of edges 5 . Let X ⊂ Rn be a point cloud, ε ∈ R and (X, d) a metricspace. An ε-distance proximity graph G(X, ε) is the graph whose vertices areV (G) = X and edges E(G) = {(x, y)| x, y ∈ X , d(x, y) ≤ ε}. Let x ∈ X bea sample point. x is called α-extreme if there exists an open ball of radius1α, B 1α, such that x lies on the boundary and B 1αcontains all the points ofX. Finding such α-extreme uses Delaunay triangulations [3], but a detaileddescription goes beyond the scope of this paper. Let X, Y ⊂ Rn two non-emtpy point clouds endowed with the same metric. Hausdorff distance isdefined by dH(X, Y ) = max{supx∈Xinfy∈Y d(x, y), supy∈Y infx∈Xd(x, y)}.A neural network (NN) [5] is a massively parallel distributed processormade up of simple processing units that has a natural propensity for storingexperiential knowledge and making it available for use. NNs can be describedas hierarchical directed weighted graphs which nodes are called neurons. Theseneurons are distributed in layers. NNs receive an input which is processedacross neurons by functions predefined, called activation functions, giving anumerical output. When NNs are just forward oriented and nodes betweenlayers are fully connected, they are called fully connected feedforward NNs 6 .A neural network learning algorithm modifies NN parameters in order to op-timize a cost function. Stochastic gradient descent is a NN learning algorithmthat performs a parameter update following the opposite direction of the gra-dient.5 The reader is assumed to be familiar with graph theory.6 For a further reading about neural networks [4].(a) Cloud of the dataset (b) NN output (c) NN classificationFig. 1. Neural Networks as a surface fitting problem2 Stating the problemGiven a dataset X ⊂ Rn × {0, . . . , k} with 0, . . . , k the classification labels,after training a neural network with X, Rn is splited in k regions accordingto the classification of the points. In such way, a manifold frontier of the pre-diction of a neural network is obtained as a process of local search, based onminimizing a given cost function (two dimension example in Fig. 1). Trainingsuch neural nets usually takes a long time, due mainly to the large amount ofpoints of the dataset. In this paper, we wonder about the possibility of reduc-ing the number of points of the dataset without losing accuracy. The chosensubset of significant points is called a representative subset of the dataset.The underlying idea on the selection of representative dataset is that closepoints provide similar information to the learning process and, in some sense,such information is redundant and therefore, some points of the dataset canbe removed. This way, the remaining points keep the same information andthe obtained neural nets are comparable as classification tools.3 Algorithms for computing representative datasetsTwo algorithms for obtaining representative datasets are proposed.Proximity Graph Based Algorithm (PGA) reduces the dataset sizeusing proximity as a criterion for deleting points. Given a point cloud datasetX ⊂ Rn (e.g. Fig. 2a). A distance d ∈ R is a hyperparameter used to con-struct a proximity graph G(X, d) which summarizes information about densityand neighborhood of X. In order to find clusters, a community detection al-gorithm is applied to G(X, d), encapsulating agglomeration notion in a set ofcommunities (e.g. Fig. 2b). Finally, site percolation is performed, followingany strategy as random sampling in each community or any stochastic processas deletion of high degree nodes. The result of the algorithm is a simplified(a) Original dataset (b) Set of communities (c) Simplified datasetFig. 2. Application of the proximity graph algorithm(a) Original dataset(3.000 points)(b) PGA dataset (1497points)(c) α-extremes dataset(867 points)Fig. 3. Original dataset and representative datasetsversion of X keeping its shape, as we see in Fig. 2c. In order to avoid gettingan imbalanced dataset, the algorithm is applied for each class. The complexityof the algorithm depends on the different choices of community algorithm andsampling method.Algorithm 1 Proximity graph representative dataset (PGA)Input: Point cloud dataset X ⊂ Rn × {1 . . . k} and d ∈ ROutput: A point cloud X̂ ⊂ X.for i = 1 to k doGi(Xi, d) where X ⊃ Xi ⊂ Rn × {i}ci = communityDetection(Gi)X̂i = sampling(ci)X̂ =⋃ki=1 X̂iThe second approach is based on the concept of α-shape. In order tocapture the intuitive notion of shape, we propose α-extreme representativedataset as a sufficient dataset for a neural network. An efficient algorithm toobtain α-shapes was firstly introduced in [3]. These α-shapes can be obtainedwith a O(n · log n) algorithm with O(n) space in two dimensions and withO(n|k−12|) space in k dimensions. To compute α-extremes, firstly Delaunaytriangulation is determined and after that, α-extremes are easily obtained.Algorithm 2 α-extreme representative datasetInput: Point cloud dataset X ⊂ Rn × {1 . . . k}Output: A point cloud X̂ ⊂ X.do Compute α-shapes (e.g. alphahull R package)X̂ = α-extremes4 ExperimentsIn this section, some experimental results are presented using proximity graphbased algorithm and α-extremes subset. A fully connected feedforward neuralnetwork was used. It has 100 neurons in the unique hidden layer, hyperbolictangent activation function and softmax output activation function. It wastrained with stochastic gradient descent with mini batch size 50, 100 epochiterations and momentum parameter 0.9. This neural network was trainedwith three different datasets: an original dataset, a simplified version of theoriginal (which is output of the PGA) and the α-extreme dataset. Fig. 3 showsthese datasets. Hausdorff distance between the original dataset and graphbased algorithm output dataset is 0.3227921 and, Hausdorff distance betweenthe original dataset and α-extremes dataset is 0.6030393. Furthermore, meansquare error is 0.16 for each neural network predicting over the original dataset.Accuracy predicting, i.e. percentage of well classified points, over the originaldataset is, for the three neural networks, on the range (95− 96%).5 Conclusions and Future WorkReducing dataset size can be a nice approach in order to understand NNsstability and functionality. We have shown two representative datasets stylesthat cover different shape notions. Furthermore, we also show that NNs cando nice predictions using simplified datasets. Future work would consist on:(1) formalizing that representative datasets can be a good strategy to achieveoptimization and accuracy; (2) the extension to higher dimensions, e.g., image(a) Original pred. (b) PGAD dataset (c) α-extremes datasetFig. 4. Neural network predictionsclassification problems (it could be done using the notion of α-complex whichextends the one of α-shape to higher dimension [2]).References[1] H. Edelsbrunner. Smooth surfaces for multi-scale shape representation. InP. S. Thiagarajan, editor, Foundations of Software Technology and TheoreticalComputer Science, pages 391–412, Berlin, Heidelberg, 1995. Springer BerlinHeidelberg.[2] H. Edelsbrunner and J. L. Harer. Computational Topology, An Introduction.American Mathematical Society, 2010.[3] H. Edelsbrunner, D. Kirkpatrick, and R. Seidel. On the shape of a set of pointsin the plane. IEEE Trans. Inf. Theor., 29(4):551–559, Sept. 2006.[4] I. Goodfellow, Y. Bengio, and A. Courville. Deep Learning. MIT Press, 2016.http://www.deeplearningbook.org.[5] S. Haykin. Neural Networks and Learning Machines. Number v. 10 in Neuralnetworks and learning machines. Prentice Hall, 2009.[6] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning appliedto document recognition. In Proceedings of the IEEE, volume 86, pages 2278–2324, 1998.[7] K. Pearson. On lines and planes of closest fit to systems of points in space.Philosophical Magazine, 2(6):559–572, 1901.

Representative datasets for neural networks

https://idus.us.es/bitstream/11441/97959/1/Representative%20datasets%20for%20neural%20networks..pdf

Representative datasets for neural networks

Abstract

Similar works

Full text

Available Versions

idUS. Depósito de Investigación Universidad de Sevilla