In this paper we present an optimization of the Optimum-Path Forest classifier training procedure, which is based on a theoretical relationship between minimum spanning forest and optimum-path forest for a specific path-cost function. Experiments on public datasets have shown that the proposed approach can obtain similar accuracy to the traditional one hut with faster data training

Adriana S. Iwashita

Alexandre X. Falcão

João Manuel R. S. Tavares

João P. Papa

Roberto A. Lotufo

Victor H. Costa de Albuquerque

Victor M. de Araújo

Repositório Aberto da Universidade do Porto

Speeding Up Optimum-Path Forest Training by Path-cost PropagationAdriana S. Iwashita1, Joa˜o P. Papa1, Alexandre X. Falca˜o2, Roberto A. Lotufo3Victor M. de Araujo Oliveira3, Victor H. Costa de Albuquerque4, Joa˜o Manuel R. S. Tavares51Sa˜o Paulo State University - Department of Computing, Bauru - Brazil2University of Campinas - Institute of Computing, Campinas - Brazil3University of Campinas - School of Electrical and Computer Engineering, Campinas - Brazil4Center of Technological Sciences - University of Fortaleza, Fortaleza - Brazil5Faculty of Engineering of the University of Porto, Porto - PortugalE-mail: papa@fc.unesp.brAbstractIn this paper we present an optimization of theOptimum-Path Forest classifier training procedure,which is based on a theoretical relationship betweenminimum spanning forest and optimum-path forest fora specific path-cost function. Experiments on publicdatasets have shown that the proposed approach canobtain similar accuracy to the traditional one but withfaster data training.1. IntroductionPattern recognition techniques attempt to learn afunction that maps the input data to some label basedon the samples’ behavior. Such learning process is themost time-consuming phase of several pattern recogni-tion techniques, since it can be composed by a trainingstep followed by parameters’ optimization, which mayrequire a retraining procure. The problem gets worstin interactive classification tools, in which the user isasked to label samples that will be used for training. Theremaining image is then classified and the results can berefined by another samples’ manual labeling. This situ-ation may be unacceptable for high resolution images.In the last decade, several pattern recognition tech-niques have been proposed and widely used in severalapplications. Among them, we may cite Support Vec-tor Machines (SVM) [3], Artificial Neural Networks(ANN) and Self-Organizing Maps (SOM) [5], and alsoThe first and second authors would like to thank Fapesp grants#2010/12697-8 and #2009/16206-1, and CNPq grant #303182/2011-3. The fourth author thanks CNPq and FUNCAP a DCR grant (projectnumber 35.0053/2011.1) from UNIFOR, in Brazil.Bayesian classifiers. The main problem of such tech-niques is the high computational burden for training.Although SVM have achieved high recognition ratesfor several applications, when the training set becomeslarge, its learning may be impracticable for real timeapplications with fast feedback responses.Recently, a novel pattern recognition techniquecalled Optimum Path Forest (OPF) was proposed in or-der to overcome such challenges. Papa et al. [7] showedthat OPF obtained similar effectiveness to SVM, butthat is faster in the training phase. The OPF models theproblem of pattern recognition as a graph partition intooptimum-path trees (OPTs), which are rooted by keysamples (prototypes) that try to compete among them-selves in order to conquer the remaining samples. Suchprototype nodes can be found by computing a MinimumSpanning Tree (MST) over the training set and markingthe connected nodes from different labels.Further, Papa et al. [6] proposed an optimization ofthe OPF classification algorithm, which lead such ap-proach to be faster than the original one, and with sim-ilar recognition rates. In this paper, we present an op-timization of the OPF training step based on a theoreti-cal relationship between the optimum-path forest gener-ated by OPF and the Minimum Spanning Forest (MSF),which is essentially a MST generated by OPF in thetraining phase with the edges between prototypes. Suchcontribution is very interesting in the context of OPF-based research, mainly in applications involving largedatasets. The remainder of this paper is organized asfollows. Sections 2 and 3 present the OPF backgroundand the proposed training step optimization algorithm,respectively. Section 4 discusses the experimental re-sults, and Section 5 states the conclusions.2. Pattern Classification using Optimum-Path ForestThe OPF classifier works by modeling the problemof pattern recognition as a graph partition in a given fea-ture space. The nodes are represented by the featurevectors and the edges connect all pairs of them, defininga full connectedness graph. The partition of the graph iscarried out by a competition process between some keysamples (prototypes), which offer optimum paths to theremaining nodes of the graph. Each prototype sampledefines its own OPT, and the collection of all OPTs de-fines an optimum-path forest, which gives the name tothe classifier [7, 6].2.1 Background TheoryLet Z = Z1∪Z2 be a dataset labeled with a functionλ, in which Z1 and Z2 are, respectively, a training andtest sets such that Z1 is used to train a given classifierand Z2 is used to assess its accuracy. Let S ⊆ Z1 aset of prototype samples. Essentially, the OPF classifiercreates a discrete optimal partition of the feature spacesuch that any sample s ∈ Z2 can be classified accordingto this partition.The OPF algorithm may be used with any smoothpath-cost function which can group samples with sim-ilar properties [4]. Particularly, we used the path-costfunction fmax, which is computed as follows:fmax(〈s〉) ={0 if s ∈ S,+∞ otherwisefmax(pi · 〈s, t〉) = max{fmax(pi), d(s, t)}, (1)in which d(s, t) means the distance between samples sand t, and a path pi is defined as a sequence of adja-cent samples. As such, we have that fmax(pi) computesthe maximum distance between adjacent samples in pi,when pi is not a trivial path.The OPF algorithm assigns one optimum path P ∗(s)from S to every sample s ∈ Z1, forming an optimumpath forest P (a function with no cycles that assignsto each s ∈ Z1\S its predecessor P (s) in P ∗(s) or amarker nil when s ∈ S. Let R(s) ∈ S be the rootof P ∗(s) that can be reached from P (s). The OPF al-gorithm computes for each s ∈ Z1, the cost C(s) ofP ∗(s), the label L(s) = λ(R(s)), and the predecessorP (s).2.2 TrainingWe say that S∗ is an optimum set of prototypes whenthe OPF algorithm minimizes the classification errorsfor every s ∈ Z1. S∗ can be found by exploiting the the-oretical relation between MST and OPT for fmax [2].The training essentially consists in finding S∗ and anOPF classifier rooted at S∗.By computing an MST in the complete graph(Z1, A), we obtain a connected acyclic graph whosenodes are all samples of Z1 and the arcs are undirectedand weighted by the distances d between adjacent sam-ples. The spanning tree is optimum in the sense thatthe sum of its arc weights is minimum as comparedto any other spanning tree in the complete graph. Inthe MST, every pair of samples is connected by a sin-gle path which is optimum according to fmax. That is,the minimum-spanning tree contains one optimum-pathtree for any selected root node. The optimum proto-types are the closest elements of the MST with differentlabels in Z1 (i.e., elements that fall in the frontier of theclasses). After finding prototypes, we run the competi-tion process in order to build the optimum-path forest.2.3 ClassificationFor any sample t ∈ Z2, we consider all arcs con-necting t with samples s ∈ Z1, as though t were part ofthe training graph. Considering all possible paths fromS∗ to t, we find the optimum path P ∗(t) from S∗ andlabel t with the class λ(R(t)) of its most strongly con-nected prototype R(t) ∈ S∗. This path can be identifiedincrementally by evaluating the optimum cost C(t) asC(t) = min{max{C(s), d(s, t)}}, ∀s ∈ Z1. (2)Let the node s∗ ∈ Z1 be the one that satisfies Equa-tion 2 (i.e., the predecessor P (t) in the optimum pathP ∗(t)). Given that L(s∗) = λ(R(t)), the classificationsimply assigns L(s∗) as the class of t. An error occurswhen L(s∗) 6= λ(t).3 Optimization by Path-cost PropagationAs aforementioned in Section 2.2, the optimum setof prototypes S∗, i.e., the one that minimizes the classi-fication errors in the training step, can be found by ex-ploiting the theoretical relation between MST and OPTfor fmax [2].Let G be a graph and F a minimum spanning forestrelative to G, which is essentially a collection of mini-mum spanning trees. Allene et al. [2] have proved thatif F is a MSF, F is also an optimum-path forest usingfmax as path-cost function. This theorem stands thatgiven a MST of G, we can obtain an optimum-path for-est of it by just removing the arcs between prototypesand updating the costs of all nodes, which we call path-cost propagation.The intuitive idea behind this theorem says that theoptimum-paths from prototypes to all nodes will fol-low the shape of MST, and an optimum-path from theprototype of the white class, for instance, will need topass through the prototype from the black class in orderto reach the black nodes. As we do not have negativedistances, one prototype can not conquer another one.Therefore, the black prototype works as a sentinel toprotect the black nodes from paths that come from thewhite ones. It should be noted that if we have an uniqueMST, the OPF training error would be zero.Figure 1. Example of a complete graph. Theweights of the edges stand for the distance be-tween nodes.Figure 2. Minimum Spanning Tree of the graphdisplayed in Figure 1.Figure 3. Optimum-Path forest of the graphdisplayed in Figure 1 using the prototypes ob-tained in Figure 2.In Figure 1, we can see a complete graph where theedges between nodes denote the distance between theircorresponding feature vectors. The next step of OPFtraining algorithm is to find the prototypes, which canbe done by computing a MST and then marking thenearest nodes from different classes (Figure 2 displaysthe MST and the bounded nodes stand for the proto-types). Finally, we run the OPF algorithm itself (seeSection 2.2) in order to begin the competition processand then to partition the graph into OPTs, which arerooted at each prototype (Figure 3).From Figure 2, we can also obtain the optimum-pathforest shown in Figure 3 by only removing the arc be-tween prototypes, directing the arcs and conducting thepath-cost propagation using fmax. Therefore, it is notneeded to run OPF algorithm, which makes the wholeprocedure faster for training. However, it is importantto emphasize that the proposed optimization approachbased on path propagation only works with fmax, asstated in [2]. Although other path-cost functions havebeen proposed, the ones which employs fmax is themost used.4 ExperimentsIn this section, we discuss the experiments con-ducted in order to show the efficiency of the proposedoptimization for OPF training step. We have used twopublic datasets with different sizes to assess the trainingefficiency and also the classification accuracy in distinctscenarios:• MPEG-7 Shape Dataset [1]: it comprises 1400 sil-houette images equally divided in 70 classes, i.e.,20 samples per class. As we have employed theFourier descriptor to each image, the feature vec-tor contains 126 dimensions; and• Covtype Dataset1 : it comprises 29048 images, di-vided in 7 classes with a feature vector containing54 dimensions. This version is a subset containing5% of the original one.In regard to training and test sets size, we employeda cross-validation procedure with different percentagesfor training and test sets, which range from 10% to 90%of training set size, with steps of 10. Figure 4 displaysthe mean execution times in seconds for training con-cerning the traditional and the proposed approach forMPEG-7 dataset. We not show the accuracy resultssince the recognition rates in the test set for both ap-proaches were the same. As one can see, the proposedapproach becomes still faster when the training set sizeincreases. On the average, the proposed approach hasbeen 1.403 times faster for training than traditional one,without loss of generalization over the test set.Figures 5 and 6 display, respectively, the mean ac-curacy and training times for the different percentagesof training and test sets for Covtype dataset. In orderto assess the results, we have executed an unpaired Stu-dent’s t-test with significance level equals to 0.05. The1http://archive.ics.uci.edu/ml 0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 10  20  30  40  50  60  70  80  90Mean execution time [s]Training set percentageTraditionalProposedFigure 4. Mean execution times in secondswith respect to training step for MPEG-7dataset.null hypothesis, i.e., the ones that guarantees that theproposed training algorithm and the traditional one havethe same accuracy was accepted for training set percent-ages at 10%, 20%, 30%, 70% and 90%. In regard to theremaining training set sizes, the traditional algorithmwas slightly better. However, the proposed approachhas been 1.878 times faster for training (on average).The speed up can be higher as the training set increases.It is important to highlight that such differences onthe accuracy are due to the presence of several MSTs,i.e., several optimum paths. Suppose we have a node sfrom the white class that offers an optimum path-costCs(t) to a black sample t. Now, assume that we havea black node p that also offers the optimum path-costCp(t), i.e., Cs(t) = Cp(t). In this case, if s came out ofthe queue before p, s will conquer t, and we will have amisclassification. 70 72 74 76 78 80 82 84 86 88 10  20  30  40  50  60  70  80  90Mean accuracyTraining set percentageTraditionalProposedFigure 5. Mean classification accuracy for Cov-type dataset. 0 5 10 15 20 25 30 35 40 45 10  20  30  40  50  60  70  80  90Mean execution time [s]Training set percentageTraditionalProposedFigure 6. Mean execution times in secondswith respect to training step for Covtypedataset.5 ConclusionsIn this paper, we have proposed an optimization ofthe OPF training step algorithm based on a theoret-ical relation between minimum spanning forests andoptimum-path forests. As the optimum-paths follow theshape of the MST, we can build an optimum-path forestby removing the arcs between prototypes, redirectingthe edges and propagating the path-costs using fmax.The proposed approach has shown to be up to 2 timesfaster for training with similar accuracy over the testsets.References[1] Mpeg-7: The generic multimedia content descriptionstandard, part 1. IEEE MultiMedia, 09(2):78–87, 2002.[2] C. Alle`ne, J.-Y. Audibert, M. Couprie, J. Cousty, andR. Keriven. Some links between min-cuts, optimal span-ning forests and watersheds. In Proceedings of the Inter-national Symposium on Mathematical Morphology, vol-ume 1, pages 253–264, 2007.[3] C. Cortes and V. Vapnik. Support-vector networks. Ma-chine Learning, 20(3):273–297, 1995.[4] A. Falca˜o, J. Stolfi, and R. Lotufo. The image forest-ing transform theory, algorithms, and applications. IEEETransactions on Pattern Analysis and Machine Intelli-gence, 26(1):19–29, 2004.[5] S. Haykin. Neural Networks: A comprehensive founda-tion. Prentice-Hall, 1998.[6] J. P. Papa, A. X. Falca˜o, V. H. C. Albuquerque, and J. M.R. S. Tavares. Efficient supervised optimum-path for-est classification for large datasets. Pattern Recognition,45(1):512–520, 2012.[7] J. P. Papa, A. X. Falca˜o, and C. T. N. Suzuki. Supervisedpattern classification based on optimum-path forest. In-ternational Journal of Imaging Systems and Technology,19(2):120–131, 2009.

Speeding up optimum-path forest training by path-cost propagation

In this paper we present an optimization of the Optimum-Path Forest classifier training procedure, which is based on a theoretical relationship between minimum spanning forest and optimum-path forest for a specific path-cost function. Experiments on public datasets have shown that the proposed approach can obtain similar accuracy to the traditional one but with faster data training. © 2012 ICPR Org Committee

Speeding up optimum-path forest training by path-cost propagation

Abstract

Similar works

Full text

Available Versions

Repositório Aberto da Universidade do Porto

Acervo Digital da Unesp