Dimauro, G.

Guerriero, A.

Impedovo, S.

Lecce Di, V.

Pirlo, G.

Salzo, A.

English

This paper presents a perturbation­based approach useful to select the best combination method for a multi­classifier system. The basic idea is to simulate small variations in the performance of the set of classifiers and to evaluate to what extent they influence the performance of the combined classifier. In the experimental phase, the Behavioural Knowledge Space and the Dempster­Shafer combination methods have been considered. The experimental results, carried out in the field of hand­written numeral recognition, demonstrate the effectiveness of the new approac

Proceedings - University of Groningen

   University of GroningenA PERTURBATION­BASED APPROACH FOR MULTI­CLASSIFIER SYSTEM DESIGNLecce Di, V.; Dimauro, G.; Guerriero, A.; Impedovo, S.; Pirlo, G.; Salzo, A.Published in:EPRINTS-BOOK-TITLEIMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite fromit. Please check the document version below.Document VersionPublisher's PDF, also known as Version of recordPublication date:2004Link to publication in University of Groningen/UMCG research databaseCitation for published version (APA):Lecce Di, V., Dimauro, G., Guerriero, A., Impedovo, S., Pirlo, G., & Salzo, A. (2004). APERTURBATION­BASED APPROACH FOR MULTI­CLASSIFIER SYSTEM DESIGN. In EPRINTS-BOOK-TITLE s.n..CopyrightOther than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of theauthor(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).The publication may also be distributed here under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license.More information can be found on the University of Groningen website: https://www.rug.nl/library/open-access/self-archiving-pure/taverne-amendment.Take-down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons thenumber of authors shown on this cover page is limited to 10 maximum.Download date: 03-06-2022A PERTURBATION-BASED APPROACH FOR MULTI-CLASSIFIERSYSTEM DESIGNV.DI LECCE1, G.DIMAURO2, A.GUERRIERO1, S.IMPEDOVO2, G.PIRLO2, A.SALZO2(1) Dipartimento di Ing. Elettronica - Politecnico di Bari-Via Re David - 70126 Bari - Italy(2) Dipartimento di Informatica - Università di Bari - Via Orabona, 4 - 70126 Bari –ItalyThis paper presents a perturbation-based approach useful to select the best combinationmethod for a multi-classifier system. The basic idea is to simulate small variations in theperformance of the set of classifiers and to evaluate to what extent they influence theperformance of the combined classifier. In the experimental phase, the BehaviouralKnowledge Space and the Dempster-Shafer combination methods have been considered. Theexperimental results, carried out in the field of  hand-written numeral recognition,demonstrate the  effectiveness of the new approach.1 IntroductionThe interest of researchers in multi -classifier systems is motivated by the consideration thatthey allow high performances also in the cases of difficult classification problems [1]. Sincetheoretical analysis of combination methods can be very difficult [2,3] the evaluation ofcomplex combination methods is generally carried out on experimental basis [4,5]. The netresult is that the design of multi -classifier systems still presents many unsolved issues [2,3].An important issue is related to the selection of the best method to combine theindividual classifier for a specific application. This aspect is generally evaluated bymeasuring the performance of  the methods by using data sets derived from the realapplication. In this case, there is no assurance that similar results can be obtained in theworking environment in which the performance of the individual classifiers can change [5].In this paper a perturbation-based approach is presented for the selection of the bestcombination method for a multi -classifier system. Small variations in the performance of theset of classifiers are simulated and their effect to the performance of the combined classifieris evaluated. Some indexes are also provided to obtain information on the performance of thecombined classifiers.The organisation of the paper is the following: Section 2 presents the problem ofselection of the combination method for multi -classifier system. The new perturbation-basedapproach is presented in Section 3. The experimental results are reported in Section 4.553In: L.R.B. Schomaker and L.G. Vuurpijl (Eds.), Proceedings of the Seventh International Workshop on Frontiers in Handwriting Recognition,September 11-13 2000, Amsterdam, ISBN 90-76942-01-3, Nijmegen: International Unipen Foundation, pp 553-5582 Multi-Classifier System DesignIn a multi -classifier system the final decision is obtained by combining the decisions ofseveral classifiers [1]. When a parallel-combination topology is considered [6], the inputpattern pt is provided to each classifier Ai, i=1,..,K which decides the membership of thepattern xt to the pattern classes ω1, ω2,..., ωm . Let Ai(t) be the response of the i-th classifierwhen the pattern xt is processed, the combination method M provides the final response bycombining the responses of the individual classifiers. From the outputs of the individualclassifiers, the method computes a confidence score S(ωi) for each class ωi, i=1,2,...,m.Successively a decision rule is used to produce the final classification response.The performance of a combination method is generally evaluated on experimental basisby considering measures as the recognition rate RM, the substitution rate EM and the rejectionrate LM, or other indexes and cost functions generally defined as a linear combination of RM ,EM and LM [1,2].Now, let be CM= LM + a⋅EM  the cost function considered for the evaluation of thecombination methods [6], the selection of the best method among M1 and M2 for a multi -classifier system follows the rules:   If CM1 < CM2 then use M1   If CM1 > CM2 then use M2   If CM1 = CM2 then use M1 or M2 equivalently.where CM1 and  CM2 are evaluated using a test data set derived from the real workingenvironment. Unfortunately, since the performance of the individual classifiers can change intime due to the learning capabilities of the classifiers or to the variabilit y in input data quality,there is no assurance that the combination method will provide acceptable results for thespecific application in the working environment. Therefore, it is important that theeffectiveness of methods for classifier combination M i is evaluated also when thecharacteristics of the set of individual classifiers change.3 A perturbation-based approach to combination method selectionLet A1,A2,…,AK be the individual classifiers combined by a combination method M. Letus consider the vector of responses obtained by inputting the patterns xt belonging to adatabase T. If T contains N patterns, the vector of responses consists of  N elements eachhaving K responses (one for each classifier) (A1(t), A2(t), …, AK(t)) ,  t=1,2,,N.From the multi -dimensional vector, two kinds of features are derived:   Features at the level of individual classifiers.The recognition rate of each classifier:RAi= ∑ttiR xtAQN)),((1where554î ==                                                                                 otherwise    0correct) is (t)A (i.e.   class  the tobelongs xand (t)A   if    1)),(( ijtjiωωtiR xtAQFeatures at the level of the set of classifiers.The correlation between classifiers [5]:∑⋅=<      1,...Kj=i,,21jijAiAAKρρwhere∑==NtjijAiA tAtAQN 1, ))(),((1ρρandî=          otherwise    0)(A=(t)A  if    1))(),((ji ttAtAQ jiρ .Therefore, the performance of the combination method M can be considered asdepending on the recognition rates of the individual classifiers R=(RA1,RA2,…,RAK) and onthe correlation ρ of the set of classifiersCE(R,ρ)= LE(R,ρ)+a⋅EE(R,ρ)                                 (1)where LM(R,ρ) and EM(R,ρ) denote the rejection rate and the substitution rate of thecombination method M as functions of R and ρ.The combination method M can be evaluated in different working conditions byperturbing the characteristics of the classifiers by mean of an automatic procedure. For thispurpose the vector of responses of the individual classifiers can be modified according to thefollowing considerations:•  Modification of features at the level of individual classifiers: For each classifier avariation is assumed in its recognition rate. For instance, for Ai whose recognition rate isoriginally of RAi , we have that the new recognition rate R*Ai  will  range in [RAi - δRAi ,RAi + δRAi], where δRAi is suitably defined.•  Modification of features at the level of the set of classifiers: A variation is assumed forthe correlation ρ of the entire set of classifiers. Specifically we have that the newcorrelation ρ* will range in [ρ-δρ, ρ+δρ]where δρ is suitably defined.For example, let us consider the vector of responses in Figure 1 for A1,A2,A3,A4 (N=10),where: R→Recognition; S1,S2,S3,S4→Substitution (rejections are not considered in thisexample). It results R=(0.7,0.8,0.7,0.6) (in fact: RA1=0.7, RA2=0.8, RA3=0.7, RA4=0.6), andρ=3.2/6. In this case, if we combine A1,A2,A3,A4 by the combination method M, we willachieve the value CM(R, ρ)= CM(0.7, 0.8, 0.7, 0.6, 3.2/6). In order to investigate thebehaviour of M, a perturbation can be performed on the vector of responses according to the555considerations (a) and (b). In Figure 2 a perturbed version of the vector of responses is shown(the modifications are marked with "^"). First, the output of A4 for pattern 3 has beenchanged (this modification changes the recognition rate of A4 and also the value of ρ for theentire set of classifiers). Second, the outputs of A2 for patterns 7 and 8 have been swapped(this modification does not change the recognition rate of A2 but only the value of ρ for theentire set of classifiers). It is easy to verify that, for the case in Figure 2 we haveR*=(0.7,0.8,0.7,0.7) (in fact: R*A1=0.6, R*A2=0.6, R*A3=0.6, R*A4=0.7), and ρ*=3.8/6.A1 A2 A3 A4Pattern 1 R R R RPattern 2 R R R S1Pattern 3 R R S3 S2Pattern 4 S1 S4 R S3Pattern 5 R R R RPattern 6 R R R RPattern 7 S3 R S2 RPattern 8 S2 R S1 S1Pattern 9 R S2 R RPattern 10 R R R RFigure 1: Vector of responses (0.7,0.8,0.7,0.6,3.2/6)A1 A2 A3 A4Pattern 1 R R R RPattern 2 R R R S1Pattern 3 R R S3 R(^)Pattern 4 S1 S4 R S3Pattern 5 R R R RPattern 6 R R R RPattern 7 S3 R(^) S2 RPattern 8 S2 S2(^) S1 S1Pattern 9 R R R RPattern 10 R R R RFigure 2: Vector of responses after perturbation (0.7,0.8,0.7,0.7,3.8/6).Modifications: A4(3)=R (Fig. 1: A4(3)=S2);A2(7)=R ; A2(8)=S2 (Fig. 1: A2(7)=S2 ; A2(8)=R).Now, let us consider the set of values assumed by the cost function for M when theentire set of  perturbed vector is considered M={ CM(R*, ρ*) |(R* , ρ*)∈  I*} where theperturbation range I* is defined as I* =[RA1-δRA1 ,RA1+δRA1]×[RA2-δRA2,RA2+δRA2]×..×[RAK-δRAK , RAK+δRAK]×[ρ-δρ,ρ+δρ].From the set M , useful information on the effectiveness of  M in different workingconditions can be derived:CM(R*, ρ*)CM(R, ρ)556 max(  M ): the maximum value in  M; min(  M ): the minimum value in  M; M(  M ): the mean value of  M; SD(  M ): the standard deviation of  M.These indexes allow a more accurate selection of the combination method for a multi -classifier system depending on the particular application field. Typical selection criteria canbe:  max(  M ) minimum (worst performance as good as possible); max(  M ) minimum (best performance as good as possible); M(  M ) minimum (best performance on average); SD(  M ) minimum (performance as stable as possible).4 Experimental ResultsThe proposed approach has been applied to a system for hand-written numeralrecognition that implements both the Behavioural Knowledge Space Method (BKS) [7] andthe Dempster-Shafer Method (DS) [8]. The system [9] combines four classifiers trained withabout 18468 numerals of the CEDAR Database (BR directory) [10]: Region (RA1=0.91),Contour (RA2=0.87), Loci (RA3=0.90), Histogram (RA4=0.87).Initially, from the vector of responses of the set of classifiers (tested with data of the BSdirectory of the CEDAR database) we derive the values R=(0.91,0.87,0.90,0.87) and ρ=0.84.From eq. (1), (with a=10), it results CBKS=0,485 and CDS =0,472. Hence DS outcomes BKS.Now, when the perturbation-based approach is applied to investigate the behaviour ofBKS(R*, ρ*) and DS(R*, ρ*) for (R*, ρ*)∈  I*, where δRAi=0.01 (a variation of 1% isconsidered for the recognition rate of the individual classifiers) and δρ=0.03, we obtain thesets  BKS and  DS as Table 1 shows.Table 1: Perturbation-based analysis of BKS and DS BKS  DSmax(  M ) 0.525 0.690min(  M ) 0.475 0.323M(  M ) 0.493 0.491SD(  M ) 0.019 0.156The result in Table 1 shows that, although DS and BKS have similar mean performance(M(  DS )≅ M(  BKS)), DS outperforms BKS in terms of best result ((min(  DS )<min(  BKS)),while BKS results more stable than DS ((SD(  DS )>SD(  BKS)) and it is also superior to DS interms of worst result ((max(  DS )>max(  BKS)).5575 ConclusionsThis paper presents a new perturbation-based approach for the selection of thecombination method best suited in a multi -classifier system for abstract-level classifiers. Theapproach provides useful information to evaluate the effectiveness of a combination methodin real environments, where modifications of  working  conditions can occur due to variationsin quality of input patterns or to learning capabili ties of classifiers.References 1.   L.Xu, A.Krzyzak, C.-Y Suen, “Methods of Combining Multiple Classifiers and TheirApplications to Handwriting Recognition” , IEEE SMC, Vol. 22, N.3, 1992, pp. 418-435. 2.   L. Lam, Y.S. Huang, C.Y. Suen, “Combination of Multiple Classifier Decisions forOptical Character Recognition” , in Handbook of Character Recognition and DocumentImage Analysis, Ed. H. Bunke and P.S.P. Wang, World Scientific, Singapore, 1997, pp.79-101. 3.   J. Kittler, M. Hatef, R.P.W. Duin, J. Matas, "On Combining Classifiers", IEEE PAMI,Vol. 20, n. 3, 1998, pp. 226-238. 4.   C. Nadal, R. Legault, and C.Y. Suen, “Complementary algorithms for the recognition oftotally un-constrained hand-written numerals” , Proc. 10th ICPR., Vol. A, June 1990, pp.434-449. 5.   G. Dimauro, S. Impedovo, G. Pirlo, “Multiple Experts: a new methodology for theevaluation of the combination processes” , Progress in Handwriting Recognition, ed. byA.C.Downton and S. Impedovo, World Scientific Publishing Co. Pte. Ltd., Singapore,1997, pp. 329-335. 6.   A.F.R. Rahman, M.C. Fairhurst, “ Introducing New Multiple Expert DecisionCombination Topologies: A Case Study using Recognition of Hand-written Characters” ,Proc. ICDAR’97, Ulm, Germany, 1997, pp. 886-891. 7.   Huang, C.Y. Suen, “An Optimal Method of Combining Multiple Classifiers forUnconstrained Handwritten Numeral Recognition” , Proc. of IWFHR-3, Buffalo, NY,1993, pp. 11-20. 8.   E. Mandler and J. Schuermann, “Combining the Classification Results of independentclassifiers based on the Dempster/Shafer theory of evidence”, in  Pattern Recognitionand Artifi cial Intell igence, eds. E.S.Gelsema and L.N.Kanal, North Holland,Amsterdam, 1988, pp. 381-393. 9.   G. Dimauro, S. Impedovo, G. Pirlo, A. Salzo, “Automatic Bankchecks Processing : ANew Engineered System”, International Journal of Pattern Recognition and Artifi cialIntelli gence, Vol. 11, N.4, World Scientific Publ., Singapore, 1997, pp. 1-38. 10.   J.J. Hull , “A database for handwritten text recognition research", IEEE PAMI, Vol. 16,N. 5, 1994, pp. 550-554.558

A PERTURBATION­BASED APPROACH FOR MULTI­CLASSIFIER SYSTEM DESIGN

https://pure.rug.nl/ws/files/2977234/poster-029-Impedovo.pdf

A PERTURBATIONBASED APPROACH FOR MULTICLASSIFIER SYSTEM DESIGN

Abstract

Similar works

Full text

Available Versions

Proceedings - University of Groningen

A PERTURBATION­BASED APPROACH FOR MULTI­CLASSIFIER SYSTEM DESIGN

Abstract

Similar works

Full text

Available Versions

Proceedings - University of Groningen

A PERTURBATIONBASED APPROACH FOR MULTICLASSIFIER SYSTEM DESIGN