Microsoft, Motorola, Siemens, Hitachi, IAPR, NICI, IUF
This paper presents a perturbation­based approach useful to select the best combination method for a multi­classifier system. The basic idea is to simulate small variations in the performance of the set of classifiers and to evaluate to what extent they influence the performance of the combined classifier. In the experimental phase, the Behavioural Knowledge Space and the Dempster­Shafer combination methods have been considered. The experimental results, carried out in the field of hand­written numeral recognition, demonstrate the effectiveness of the new approach

Dimauro, G.

Guerriero, A.

Impedovo, S.

Lecce Di, V.

Pirlo, G.

Salzo, A.

English

University of Groningen Research Database

  
 University of Groningen
A PERTURBATION­BASED APPROACH FOR MULTI­CLASSIFIER SYSTEM DESIGN
Lecce Di, V.; Dimauro, G.; Guerriero, A.; Impedovo, S.; Pirlo, G.; Salzo, A.
Published in:
EPRINTS-BOOK-TITLE
IMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite from
it. Please check the document version below.
Document Version
Publisher's PDF, also known as Version of record
Publication date:
2004
Link to publication in University of Groningen/UMCG research database
Citation for published version (APA):
Lecce Di, V., Dimauro, G., Guerriero, A., Impedovo, S., Pirlo, G., & Salzo, A. (2004). A
PERTURBATION­BASED APPROACH FOR MULTI­CLASSIFIER SYSTEM DESIGN. In EPRINTS-BOOK-
TITLE s.n..
Copyright
Other than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of the
author(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).
Take-down policy
If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately
and investigate your claim.
Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons the
number of authors shown on this cover page is limited to 10 maximum.
Download date: 10-02-2018
A PERTURBATION-BASED APPROACH FOR MULTI-CLASSIFIER
SYSTEM DESIGN
V.DI LECCE1, G.DIMAURO2, A.GUERRIERO1, S.IMPEDOVO2, G.PIRLO2, A.SALZO2
(1) Dipartimento di Ing. Elettronica - Politecnico di Bari-
Via Re David - 70126 Bari - Italy
(2) Dipartimento di Informatica - Università di Bari -
 Via Orabona, 4 - 70126 Bari –Italy
This paper presents a perturbation-based approach useful to select the best combination
method for a multi-classifier system. The basic idea is to simulate small variations in the
performance of the set of classifiers and to evaluate to what extent they influence the
performance of the combined classifier. In the experimental phase, the Behavioural
Knowledge Space and the Dempster-Shafer combination methods have been considered. The
experimental results, carried out in the field of  hand-written numeral recognition,
demonstrate the  effectiveness of the new approach.
1 Introduction
The interest of researchers in multi-classifier systems is motivated by the consideration that
they allow high performances also in the cases of difficult classification problems [1]. Since
theoretical analysis of combination methods can be very difficult [2,3] the evaluation of
complex combination methods is generally carried out on experimental basis [4,5]. The net
result is that the design of multi-classifier systems still presents many unsolved issues [2,3].
An important issue is related to the selection of the best method to combine the
individual classifier for a specific application. This aspect is generally evaluated by
measuring the performance of  the methods by using data sets derived from the real
application. In this case, there is no assurance that similar results can be obtained in the
working environment in which the performance of the individual classifiers can change [5].
In this paper a perturbation-based approach is presented for the selection of the best
combination method for a multi-classifier system. Small variations in the performance of the
set of classifiers are simulated and their effect to the performance of the combined classifier
is evaluated. Some indexes are also provided to obtain information on the performance of the
combined classifiers.
The organisation of the paper is the following: Section 2 presents the problem of
selection of the combination method for multi-classifier system. The new perturbation-based
approach is presented in Section 3. The experimental results are reported in Section 4.
553
In: L.R.B. Schomaker and L.G. Vuurpijl (Eds.), Proceedings of the Seventh International Workshop on Frontiers in Handwriting Recognition,
September 11-13 2000, Amsterdam, ISBN 90-76942-01-3, Nijmegen: International Unipen Foundation, pp 553-558
2 Multi-Classifier System Design
In a multi-classifier system the final decision is obtained by combining the decisions of
several classifiers [1]. When a parallel-combination topology is considered [6], the input
pattern pt is provided to each classifier Ai, i=1,..,K which decides the membership of the
pattern xt to the pattern classes w 1, w 2,..., w m . Let Ai(t) be the response of the i-th classifier
when the pattern xt is processed, the combination method M provides the final response by
combining the responses of the individual classifiers. From the outputs of the individual
classifiers, the method computes a confidence score S( w i) for each class w i, i=1,2,...,m.
Successively a decision rule is used to produce the final classification response.
The performance of a combination method is generally evaluated on experimental basis
by considering measures as the recognition rate RM, the substitution rate EM and the rejection
rate LM, or other indexes and cost functions generally defined as a linear combination of RM ,
EM and LM [1,2].
Now, let be CM= LM + a× EM  the cost function considered for the evaluation of the
combination methods [6], the selection of the best method among M1 and M2 for a multi-
classifier system follows the rules:
 
 If CM1 < CM2 then use M1
 
 If CM1 > CM2 then use M2
 
 If CM1 = CM2 then use M1 or M2 equivalently.
where CM1 and  CM2 are evaluated using a test data set derived from the real working
environment. Unfortunately, since the performance of the individual classifiers can change in
time due to the learning capabilities of the classifiers or to the variability in input data quality,
there is no assurance that the combination method will provide acceptable results for the
specific application in the working environment. Therefore, it is important that the
effectiveness of methods for classifier combination Mi is evaluated also when the
characteristics of the set of individual classifiers change.
3 A perturbation-based approach to combination method selection
Let A1,A2,…,AK be the individual classifiers combined by a combination method M. Let
us consider the vector of responses obtained by inputting the patterns xt belonging to a
database T. If T contains N patterns, the vector of responses consists of  N elements each
having K responses (one for each classifier) (A1(t), A2(t), …, AK(t)) ,  t=1,2,,N.
From the multi-dimensional vector, two kinds of features are derived:
 
 Features at the level of individual classifiers.
The recognition rate of each classifier:
RAi= å
t
tiR xtAQN )),((
1
where
554
îí
ì
=
=
                                                                                         otherwise    0
correct) is (t)A (i.e.   class  the tobelongs xand (t)A   if    1)),(( ijtji wwtiR xtAQ
Features at the level of the set of classifiers.
The correlation between classifiers [5]:
œ
œ
œ
œ
ß
ø
Œ
Œ
Œ
Œ
º
Ø
å
×
œ
œ
ß
ø
Œ
Œ
º
Ø
÷
÷
ł
ö
ç
ç
Ł
æ
=
<      
1,...Kj=i,
,2
1
ji
jAiAA
K
rr
where
å
=
=
N
t
jijAiA tAtAQN 1,
))(),((1
r
r
and
ï
î
ï
í
ì
=
          otherwise    0
)(A=(t)A  if    1
))(),((
ji t
tAtAQ jir .
Therefore, the performance of the combination method M can be considered as
depending on the recognition rates of the individual classifiers R=(RA1,RA2,…,RAK) and on
the correlation r  of the set of classifiers
CE(R, r )= LE(R, r )+a× EE(R, r )                                  (1)
where LM(R, r ) and EM(R, r ) denote the rejection rate and the substitution rate of the
combination method M as functions of R and r .
The combination method M can be evaluated in different working conditions by
perturbing the characteristics of the classifiers by mean of an automatic procedure. For this
purpose the vector of responses of the individual classifiers can be modified according to the
following considerations:
• Modification of features at the level of individual classifiers: For each classifier a
variation is assumed in its recognition rate. For instance, for Ai whose recognition rate is
originally of RAi , we have that the new recognition rate R*Ai  will  range in [RAi - d RAi ,
RAi + d RAi], where d RAi is suitably defined.
• Modification of features at the level of the set of classifiers: A variation is assumed for
the correlation r  of the entire set of classifiers. Specifically we have that the new
correlation r * will range in [ r - d r , r +d r ]where d r  is suitably defined.
For example, let us consider the vector of responses in Figure 1 for A1,A2,A3,A4 (N=10),
where: Rfi Recognition; S1,S2,S3,S4 fi Substitution (rejections are not considered in this
example). It results R=(0.7,0.8,0.7,0.6) (in fact: RA1=0.7, RA2=0.8, RA3=0.7, RA4=0.6), and
r =3.2/6. In this case, if we combine A1,A2,A3,A4 by the combination method M, we will
achieve the value CM(R, r )= CM(0.7, 0.8, 0.7, 0.6, 3.2/6). In order to investigate the
behaviour of M, a perturbation can be performed on the vector of responses according to the
555
considerations (a) and (b). In Figure 2 a perturbed version of the vector of responses is shown
(the modifications are marked with "^"). First, the output of A4 for pattern 3 has been
changed (this modification changes the recognition rate of A4 and also the value of r  for the
entire set of classifiers). Second, the outputs of A2 for patterns 7 and 8 have been swapped
(this modification does not change the recognition rate of A2 but only the value of r  for the
entire set of classifiers). It is easy to verify that, for the case in Figure 2 we have
R*=(0.7,0.8,0.7,0.7) (in fact: R*A1=0.6, R*A2=0.6, R*A3=0.6, R*A4=0.7), and r *=3.8/6.
A1 A2 A3 A4
Pattern 1 R R R R
Pattern 2 R R R S1
Pattern 3 R R S3 S2
Pattern 4 S1 S4 R S3
Pattern 5 R R R R
Pattern 6 R R R R
Pattern 7 S3 R S2 R
Pattern 8 S2 R S1 S1
Pattern 9 R S2 R R
Pattern 10 R R R R
Figure 1: Vector of responses (0.7,0.8,0.7,0.6,3.2/6)
A1 A2 A3 A4
Pattern 1 R R R R
Pattern 2 R R R S1
Pattern 3 R R S3 R(^)
Pattern 4 S1 S4 R S3
Pattern 5 R R R R
Pattern 6 R R R R
Pattern 7 S3 R(^) S2 R
Pattern 8 S2 S2(^) S1 S1
Pattern 9 R R R R
Pattern 10 R R R R
Figure 2: Vector of responses after perturbation (0.7,0.8,0.7,0.7,3.8/6).
Modifications: A4(3)=R (Fig. 1: A4(3)=S2);A2(7)=R ; A2(8)=S2 (Fig. 1: A2(7)=S2 ; A2(8)=R).
Now, let us consider the set of values assumed by the cost function for M when the
entire set of  perturbed vector is considered 

M={CM(R*, r *) |(R* , r *) ˛  I*} where the
perturbation range I* is defined as I* =[RA1- d RA1 ,RA1+d RA1] · [RA2- d RA2,RA2+d RA2] · .. · [RAK-
d RAK , RAK+d RAK] · [ r - d r , r +d r ].
From the set 

M , useful information on the effectiveness of  M in different working
conditions can be derived:
CM(R*, rr *)
CM(R, rr )
556
 max(  M ): the maximum value in  M;

 min(  M ): the minimum value in  M;

 M(  M ): the mean value of  M;

 SD(  M ): the standard deviation of  M.
These indexes allow a more accurate selection of the combination method for a multi-
classifier system depending on the particular application field. Typical selection criteria can
be: 
 max(  M ) minimum (worst performance as good as possible);
 max(  M ) minimum (best performance as good as possible);
 M(  M ) minimum (best performance on average);
 SD(  M ) minimum (performance as stable as possible).
4 Experimental Results
The proposed approach has been applied to a system for hand-written numeral
recognition that implements both the Behavioural Knowledge Space Method (BKS) [7] and
the Dempster-Shafer Method (DS) [8]. The system [9] combines four classifiers trained with
about 18468 numerals of the CEDAR Database (BR directory) [10]: Region (RA1=0.91),
Contour (RA2=0.87), Loci (RA3=0.90), Histogram (RA4=0.87).
Initially, from the vector of responses of the set of classifiers (tested with data of the BS
directory of the CEDAR database) we derive the values R=(0.91,0.87,0.90,0.87) and r =0.84.
From eq. (1), (with a=10), it results CBKS=0,485 and CDS =0,472. Hence DS outcomes BKS.
Now, when the perturbation-based approach is applied to investigate the behaviour of
BKS(R*, r *) and DS(R*, r *) for (R*, r *) ˛  I*, where d RAi=0.01 (a variation of 1% is
considered for the recognition rate of the individual classifiers) and d r =0.03, we obtain the
sets  BKS and  DS as Table 1 shows.
Table 1: Perturbation-based analysis of BKS and DS
 BKS  DS
max(  M ) 0.525 0.690
min(  M ) 0.475 0.323
M(  M ) 0.493 0.491
SD(  M ) 0.019 0.156
The result in Table 1 shows that, although DS and BKS have similar mean performance
(M(  DS ) @ M(  BKS)), DS outperforms BKS in terms of best result ((min(  DS )<min(  BKS)),
while BKS results more stable than DS ((SD(  DS )>SD(  BKS)) and it is also superior to DS in
terms of worst result ((max(  DS )>max(  BKS)).
557
5 Conclusions
This paper presents a new perturbation-based approach for the selection of the
combination method best suited in a multi-classifier system for abstract-level classifiers. The
approach provides useful information to evaluate the effectiveness of a combination method
in real environments, where modifications of  working  conditions can occur due to variations
in quality of input patterns or to learning capabilities of classifiers.
References
 1.   L.Xu, A.Krzyzak, C.-Y Suen, “Methods of Combining Multiple Classifiers and Their
Applications to Handwriting Recognition”, IEEE SMC, Vol. 22, N.3, 1992, pp. 418-435.
 2.   L. Lam, Y.S. Huang, C.Y. Suen, “Combination of Multiple Classifier Decisions for
Optical Character Recognition”, in Handbook of Character Recognition and Document
Image Analysis, Ed. H. Bunke and P.S.P. Wang, World Scientific, Singapore, 1997, pp.
79-101.
 3.   J. Kittler, M. Hatef, R.P.W. Duin, J. Matas, "On Combining Classifiers", IEEE PAMI,
Vol. 20, n. 3, 1998, pp. 226-238.
 4.   C. Nadal, R. Legault, and C.Y. Suen, “Complementary algorithms for the recognition of
totally un-constrained hand-written numerals”, Proc. 10th ICPR., Vol. A, June 1990, pp.
434-449.
 5.   G. Dimauro, S. Impedovo, G. Pirlo, “Multiple Experts: a new methodology for the
evaluation of the combination processes”, Progress in Handwriting Recognition, ed. by
A.C.Downton and S. Impedovo, World Scientific Publishing Co. Pte. Ltd., Singapore,
1997, pp. 329-335.
 6.   A.F.R. Rahman, M.C. Fairhurst, “Introducing New Multiple Expert Decision
Combination Topologies: A Case Study using Recognition of Hand-written Characters”,
Proc. ICDAR’97, Ulm, Germany, 1997, pp. 886-891.
 7.   Huang, C.Y. Suen, “An Optimal Method of Combining Multiple Classifiers for
Unconstrained Handwritten Numeral Recognition”, Proc. of IWFHR-3, Buffalo, NY,
1993, pp. 11-20.
 8.   E. Mandler and J. Schuermann, “Combining the Classification Results of independent
classifiers based on the Dempster/Shafer theory of evidence”, in  Pattern Recognition
and Artificial Intelligence, eds. E.S.Gelsema and L.N.Kanal, North Holland,
Amsterdam, 1988, pp. 381-393.
 9.   G. Dimauro, S. Impedovo, G. Pirlo, A. Salzo, “Automatic Bankchecks Processing : A
New Engineered System”, International Journal of Pattern Recognition and Artificial
Intelligence, Vol. 11, N.4, World Scientific Publ., Singapore, 1997, pp. 1-38.
 10.   J.J. Hull, “A database for handwritten text recognition research", IEEE PAMI, Vol. 16,
N. 5, 1994, pp. 550-554.
558


A PERTURBATION­BASED APPROACH FOR MULTI­CLASSIFIER SYSTEM DESIGN

This paper presents a perturbation­based approach useful to select the best combination method for a multi­classifier system. The basic idea is to simulate small variations in the performance of the set of classifiers and to evaluate to what extent they influence the performance of the combined classifier. In the experimental phase, the Behavioural Knowledge Space and the Dempster­Shafer combination methods have been considered. The experimental results, carried out in the field of hand­written numeral recognition, demonstrate the effectiveness of the new approac

Proceedings - University of Groningen

   University of GroningenA PERTURBATION­BASED APPROACH FOR MULTI­CLASSIFIER SYSTEM DESIGNLecce Di, V.; Dimauro, G.; Guerriero, A.; Impedovo, S.; Pirlo, G.; Salzo, A.Published in:EPRINTS-BOOK-TITLEIMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite fromit. Please check the document version below.Document VersionPublisher's PDF, also known as Version of recordPublication date:2004Link to publication in University of Groningen/UMCG research databaseCitation for published version (APA):Lecce Di, V., Dimauro, G., Guerriero, A., Impedovo, S., Pirlo, G., & Salzo, A. (2004). APERTURBATION­BASED APPROACH FOR MULTI­CLASSIFIER SYSTEM DESIGN. In EPRINTS-BOOK-TITLE s.n..CopyrightOther than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of theauthor(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).The publication may also be distributed here under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license.More information can be found on the University of Groningen website: https://www.rug.nl/library/open-access/self-archiving-pure/taverne-amendment.Take-down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons thenumber of authors shown on this cover page is limited to 10 maximum.Download date: 03-06-2022A PERTURBATION-BASED APPROACH FOR MULTI-CLASSIFIERSYSTEM DESIGNV.DI LECCE1, G.DIMAURO2, A.GUERRIERO1, S.IMPEDOVO2, G.PIRLO2, A.SALZO2(1) Dipartimento di Ing. Elettronica - Politecnico di Bari-Via Re David - 70126 Bari - Italy(2) Dipartimento di Informatica - Università di Bari - Via Orabona, 4 - 70126 Bari –ItalyThis paper presents a perturbation-based approach useful to select the best combinationmethod for a multi-classifier system. The basic idea is to simulate small variations in theperformance of the set of classifiers and to evaluate to what extent they influence theperformance of the combined classifier. In the experimental phase, the BehaviouralKnowledge Space and the Dempster-Shafer combination methods have been considered. Theexperimental results, carried out in the field of  hand-written numeral recognition,demonstrate the  effectiveness of the new approach.1 IntroductionThe interest of researchers in multi -classifier systems is motivated by the consideration thatthey allow high performances also in the cases of difficult classification problems [1]. Sincetheoretical analysis of combination methods can be very difficult [2,3] the evaluation ofcomplex combination methods is generally carried out on experimental basis [4,5]. The netresult is that the design of multi -classifier systems still presents many unsolved issues [2,3].An important issue is related to the selection of the best method to combine theindividual classifier for a specific application. This aspect is generally evaluated bymeasuring the performance of  the methods by using data sets derived from the realapplication. In this case, there is no assurance that similar results can be obtained in theworking environment in which the performance of the individual classifiers can change [5].In this paper a perturbation-based approach is presented for the selection of the bestcombination method for a multi -classifier system. Small variations in the performance of theset of classifiers are simulated and their effect to the performance of the combined classifieris evaluated. Some indexes are also provided to obtain information on the performance of thecombined classifiers.The organisation of the paper is the following: Section 2 presents the problem ofselection of the combination method for multi -classifier system. The new perturbation-basedapproach is presented in Section 3. The experimental results are reported in Section 4.553In: L.R.B. Schomaker and L.G. Vuurpijl (Eds.), Proceedings of the Seventh International Workshop on Frontiers in Handwriting Recognition,September 11-13 2000, Amsterdam, ISBN 90-76942-01-3, Nijmegen: International Unipen Foundation, pp 553-5582 Multi-Classifier System DesignIn a multi -classifier system the final decision is obtained by combining the decisions ofseveral classifiers [1]. When a parallel-combination topology is considered [6], the inputpattern pt is provided to each classifier Ai, i=1,..,K which decides the membership of thepattern xt to the pattern classes ω1, ω2,..., ωm . Let Ai(t) be the response of the i-th classifierwhen the pattern xt is processed, the combination method M provides the final response bycombining the responses of the individual classifiers. From the outputs of the individualclassifiers, the method computes a confidence score S(ωi) for each class ωi, i=1,2,...,m.Successively a decision rule is used to produce the final classification response.The performance of a combination method is generally evaluated on experimental basisby considering measures as the recognition rate RM, the substitution rate EM and the rejectionrate LM, or other indexes and cost functions generally defined as a linear combination of RM ,EM and LM [1,2].Now, let be CM= LM + a⋅EM  the cost function considered for the evaluation of thecombination methods [6], the selection of the best method among M1 and M2 for a multi -classifier system follows the rules:   If CM1 < CM2 then use M1   If CM1 > CM2 then use M2   If CM1 = CM2 then use M1 or M2 equivalently.where CM1 and  CM2 are evaluated using a test data set derived from the real workingenvironment. Unfortunately, since the performance of the individual classifiers can change intime due to the learning capabilities of the classifiers or to the variabilit y in input data quality,there is no assurance that the combination method will provide acceptable results for thespecific application in the working environment. Therefore, it is important that theeffectiveness of methods for classifier combination M i is evaluated also when thecharacteristics of the set of individual classifiers change.3 A perturbation-based approach to combination method selectionLet A1,A2,…,AK be the individual classifiers combined by a combination method M. Letus consider the vector of responses obtained by inputting the patterns xt belonging to adatabase T. If T contains N patterns, the vector of responses consists of  N elements eachhaving K responses (one for each classifier) (A1(t), A2(t), …, AK(t)) ,  t=1,2,,N.From the multi -dimensional vector, two kinds of features are derived:   Features at the level of individual classifiers.The recognition rate of each classifier:RAi= ∑ttiR xtAQN)),((1where554î ==                                                                                 otherwise    0correct) is (t)A (i.e.   class  the tobelongs xand (t)A   if    1)),(( ijtjiωωtiR xtAQFeatures at the level of the set of classifiers.The correlation between classifiers [5]:∑⋅=<      1,...Kj=i,,21jijAiAAKρρwhere∑==NtjijAiA tAtAQN 1, ))(),((1ρρandî=          otherwise    0)(A=(t)A  if    1))(),((ji ttAtAQ jiρ .Therefore, the performance of the combination method M can be considered asdepending on the recognition rates of the individual classifiers R=(RA1,RA2,…,RAK) and onthe correlation ρ of the set of classifiersCE(R,ρ)= LE(R,ρ)+a⋅EE(R,ρ)                                 (1)where LM(R,ρ) and EM(R,ρ) denote the rejection rate and the substitution rate of thecombination method M as functions of R and ρ.The combination method M can be evaluated in different working conditions byperturbing the characteristics of the classifiers by mean of an automatic procedure. For thispurpose the vector of responses of the individual classifiers can be modified according to thefollowing considerations:•  Modification of features at the level of individual classifiers: For each classifier avariation is assumed in its recognition rate. For instance, for Ai whose recognition rate isoriginally of RAi , we have that the new recognition rate R*Ai  will  range in [RAi - δRAi ,RAi + δRAi], where δRAi is suitably defined.•  Modification of features at the level of the set of classifiers: A variation is assumed forthe correlation ρ of the entire set of classifiers. Specifically we have that the newcorrelation ρ* will range in [ρ-δρ, ρ+δρ]where δρ is suitably defined.For example, let us consider the vector of responses in Figure 1 for A1,A2,A3,A4 (N=10),where: R→Recognition; S1,S2,S3,S4→Substitution (rejections are not considered in thisexample). It results R=(0.7,0.8,0.7,0.6) (in fact: RA1=0.7, RA2=0.8, RA3=0.7, RA4=0.6), andρ=3.2/6. In this case, if we combine A1,A2,A3,A4 by the combination method M, we willachieve the value CM(R, ρ)= CM(0.7, 0.8, 0.7, 0.6, 3.2/6). In order to investigate thebehaviour of M, a perturbation can be performed on the vector of responses according to the555considerations (a) and (b). In Figure 2 a perturbed version of the vector of responses is shown(the modifications are marked with "^"). First, the output of A4 for pattern 3 has beenchanged (this modification changes the recognition rate of A4 and also the value of ρ for theentire set of classifiers). Second, the outputs of A2 for patterns 7 and 8 have been swapped(this modification does not change the recognition rate of A2 but only the value of ρ for theentire set of classifiers). It is easy to verify that, for the case in Figure 2 we haveR*=(0.7,0.8,0.7,0.7) (in fact: R*A1=0.6, R*A2=0.6, R*A3=0.6, R*A4=0.7), and ρ*=3.8/6.A1 A2 A3 A4Pattern 1 R R R RPattern 2 R R R S1Pattern 3 R R S3 S2Pattern 4 S1 S4 R S3Pattern 5 R R R RPattern 6 R R R RPattern 7 S3 R S2 RPattern 8 S2 R S1 S1Pattern 9 R S2 R RPattern 10 R R R RFigure 1: Vector of responses (0.7,0.8,0.7,0.6,3.2/6)A1 A2 A3 A4Pattern 1 R R R RPattern 2 R R R S1Pattern 3 R R S3 R(^)Pattern 4 S1 S4 R S3Pattern 5 R R R RPattern 6 R R R RPattern 7 S3 R(^) S2 RPattern 8 S2 S2(^) S1 S1Pattern 9 R R R RPattern 10 R R R RFigure 2: Vector of responses after perturbation (0.7,0.8,0.7,0.7,3.8/6).Modifications: A4(3)=R (Fig. 1: A4(3)=S2);A2(7)=R ; A2(8)=S2 (Fig. 1: A2(7)=S2 ; A2(8)=R).Now, let us consider the set of values assumed by the cost function for M when theentire set of  perturbed vector is considered M={ CM(R*, ρ*) |(R* , ρ*)∈  I*} where theperturbation range I* is defined as I* =[RA1-δRA1 ,RA1+δRA1]×[RA2-δRA2,RA2+δRA2]×..×[RAK-δRAK , RAK+δRAK]×[ρ-δρ,ρ+δρ].From the set M , useful information on the effectiveness of  M in different workingconditions can be derived:CM(R*, ρ*)CM(R, ρ)556 max(  M ): the maximum value in  M; min(  M ): the minimum value in  M; M(  M ): the mean value of  M; SD(  M ): the standard deviation of  M.These indexes allow a more accurate selection of the combination method for a multi -classifier system depending on the particular application field. Typical selection criteria canbe:  max(  M ) minimum (worst performance as good as possible); max(  M ) minimum (best performance as good as possible); M(  M ) minimum (best performance on average); SD(  M ) minimum (performance as stable as possible).4 Experimental ResultsThe proposed approach has been applied to a system for hand-written numeralrecognition that implements both the Behavioural Knowledge Space Method (BKS) [7] andthe Dempster-Shafer Method (DS) [8]. The system [9] combines four classifiers trained withabout 18468 numerals of the CEDAR Database (BR directory) [10]: Region (RA1=0.91),Contour (RA2=0.87), Loci (RA3=0.90), Histogram (RA4=0.87).Initially, from the vector of responses of the set of classifiers (tested with data of the BSdirectory of the CEDAR database) we derive the values R=(0.91,0.87,0.90,0.87) and ρ=0.84.From eq. (1), (with a=10), it results CBKS=0,485 and CDS =0,472. Hence DS outcomes BKS.Now, when the perturbation-based approach is applied to investigate the behaviour ofBKS(R*, ρ*) and DS(R*, ρ*) for (R*, ρ*)∈  I*, where δRAi=0.01 (a variation of 1% isconsidered for the recognition rate of the individual classifiers) and δρ=0.03, we obtain thesets  BKS and  DS as Table 1 shows.Table 1: Perturbation-based analysis of BKS and DS BKS  DSmax(  M ) 0.525 0.690min(  M ) 0.475 0.323M(  M ) 0.493 0.491SD(  M ) 0.019 0.156The result in Table 1 shows that, although DS and BKS have similar mean performance(M(  DS )≅ M(  BKS)), DS outperforms BKS in terms of best result ((min(  DS )<min(  BKS)),while BKS results more stable than DS ((SD(  DS )>SD(  BKS)) and it is also superior to DS interms of worst result ((max(  DS )>max(  BKS)).5575 ConclusionsThis paper presents a new perturbation-based approach for the selection of thecombination method best suited in a multi -classifier system for abstract-level classifiers. Theapproach provides useful information to evaluate the effectiveness of a combination methodin real environments, where modifications of  working  conditions can occur due to variationsin quality of input patterns or to learning capabili ties of classifiers.References 1.   L.Xu, A.Krzyzak, C.-Y Suen, “Methods of Combining Multiple Classifiers and TheirApplications to Handwriting Recognition” , IEEE SMC, Vol. 22, N.3, 1992, pp. 418-435. 2.   L. Lam, Y.S. Huang, C.Y. Suen, “Combination of Multiple Classifier Decisions forOptical Character Recognition” , in Handbook of Character Recognition and DocumentImage Analysis, Ed. H. Bunke and P.S.P. Wang, World Scientific, Singapore, 1997, pp.79-101. 3.   J. Kittler, M. Hatef, R.P.W. Duin, J. Matas, "On Combining Classifiers", IEEE PAMI,Vol. 20, n. 3, 1998, pp. 226-238. 4.   C. Nadal, R. Legault, and C.Y. Suen, “Complementary algorithms for the recognition oftotally un-constrained hand-written numerals” , Proc. 10th ICPR., Vol. A, June 1990, pp.434-449. 5.   G. Dimauro, S. Impedovo, G. Pirlo, “Multiple Experts: a new methodology for theevaluation of the combination processes” , Progress in Handwriting Recognition, ed. byA.C.Downton and S. Impedovo, World Scientific Publishing Co. Pte. Ltd., Singapore,1997, pp. 329-335. 6.   A.F.R. Rahman, M.C. Fairhurst, “ Introducing New Multiple Expert DecisionCombination Topologies: A Case Study using Recognition of Hand-written Characters” ,Proc. ICDAR’97, Ulm, Germany, 1997, pp. 886-891. 7.   Huang, C.Y. Suen, “An Optimal Method of Combining Multiple Classifiers forUnconstrained Handwritten Numeral Recognition” , Proc. of IWFHR-3, Buffalo, NY,1993, pp. 11-20. 8.   E. Mandler and J. Schuermann, “Combining the Classification Results of independentclassifiers based on the Dempster/Shafer theory of evidence”, in  Pattern Recognitionand Artifi cial Intell igence, eds. E.S.Gelsema and L.N.Kanal, North Holland,Amsterdam, 1988, pp. 381-393. 9.   G. Dimauro, S. Impedovo, G. Pirlo, A. Salzo, “Automatic Bankchecks Processing : ANew Engineered System”, International Journal of Pattern Recognition and Artifi cialIntelli gence, Vol. 11, N.4, World Scientific Publ., Singapore, 1997, pp. 1-38. 10.   J.J. Hull , “A database for handwritten text recognition research", IEEE PAMI, Vol. 16,N. 5, 1994, pp. 550-554.558

   University of GroningenA PERTURBATION­BASED APPROACH FOR MULTI­CLASSIFIER SYSTEM DESIGNLecce Di, V.; Dimauro, G.; Guerriero, A.; Impedovo, S.; Pirlo, G.; Salzo, A.Published in:EPRINTS-BOOK-TITLEIMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite fromit. Please check the document version below.Document VersionPublisher's PDF, also known as Version of recordPublication date:2004Link to publication in University of Groningen/UMCG research databaseCitation for published version (APA):Lecce Di, V., Dimauro, G., Guerriero, A., Impedovo, S., Pirlo, G., & Salzo, A. (2004). APERTURBATION­BASED APPROACH FOR MULTI­CLASSIFIER SYSTEM DESIGN. In EPRINTS-BOOK-TITLE s.n..CopyrightOther than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of theauthor(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).The publication may also be distributed here under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license.More information can be found on the University of Groningen website: https://www.rug.nl/library/open-access/self-archiving-pure/taverne-amendment.Take-down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons thenumber of authors shown on this cover page is limited to 10 maximum.Download date: 16-07-2023A PERTURBATION-BASED APPROACH FOR MULTI-CLASSIFIERSYSTEM DESIGNV.DI LECCE1, G.DIMAURO2, A.GUERRIERO1, S.IMPEDOVO2, G.PIRLO2, A.SALZO2(1) Dipartimento di Ing. Elettronica - Politecnico di Bari-Via Re David - 70126 Bari - Italy(2) Dipartimento di Informatica - Università di Bari - Via Orabona, 4 - 70126 Bari –ItalyThis paper presents a perturbation-based approach useful to select the best combinationmethod for a multi-classifier system. The basic idea is to simulate small variations in theperformance of the set of classifiers and to evaluate to what extent they influence theperformance of the combined classifier. In the experimental phase, the BehaviouralKnowledge Space and the Dempster-Shafer combination methods have been considered. Theexperimental results, carried out in the field of  hand-written numeral recognition,demonstrate the  effectiveness of the new approach.1 IntroductionThe interest of researchers in multi -classifier systems is motivated by the consideration thatthey allow high performances also in the cases of difficult classification problems [1]. Sincetheoretical analysis of combination methods can be very difficult [2,3] the evaluation ofcomplex combination methods is generally carried out on experimental basis [4,5]. The netresult is that the design of multi -classifier systems still presents many unsolved issues [2,3].An important issue is related to the selection of the best method to combine theindividual classifier for a specific application. This aspect is generally evaluated bymeasuring the performance of  the methods by using data sets derived from the realapplication. In this case, there is no assurance that similar results can be obtained in theworking environment in which the performance of the individual classifiers can change [5].In this paper a perturbation-based approach is presented for the selection of the bestcombination method for a multi -classifier system. Small variations in the performance of theset of classifiers are simulated and their effect to the performance of the combined classifieris evaluated. Some indexes are also provided to obtain information on the performance of thecombined classifiers.The organisation of the paper is the following: Section 2 presents the problem ofselection of the combination method for multi -classifier system. The new perturbation-basedapproach is presented in Section 3. The experimental results are reported in Section 4.553In: L.R.B. Schomaker and L.G. Vuurpijl (Eds.), Proceedings of the Seventh International Workshop on Frontiers in Handwriting Recognition,September 11-13 2000, Amsterdam, ISBN 90-76942-01-3, Nijmegen: International Unipen Foundation, pp 553-5582 Multi-Classifier System DesignIn a multi -classifier system the final decision is obtained by combining the decisions ofseveral classifiers [1]. When a parallel-combination topology is considered [6], the inputpattern pt is provided to each classifier Ai, i=1,..,K which decides the membership of thepattern xt to the pattern classes ω1, ω2,..., ωm . Let Ai(t) be the response of the i-th classifierwhen the pattern xt is processed, the combination method M provides the final response bycombining the responses of the individual classifiers. From the outputs of the individualclassifiers, the method computes a confidence score S(ωi) for each class ωi, i=1,2,...,m.Successively a decision rule is used to produce the final classification response.The performance of a combination method is generally evaluated on experimental basisby considering measures as the recognition rate RM, the substitution rate EM and the rejectionrate LM, or other indexes and cost functions generally defined as a linear combination of RM ,EM and LM [1,2].Now, let be CM= LM + a⋅EM  the cost function considered for the evaluation of thecombination methods [6], the selection of the best method among M1 and M2 for a multi -classifier system follows the rules:   If CM1 < CM2 then use M1   If CM1 > CM2 then use M2   If CM1 = CM2 then use M1 or M2 equivalently.where CM1 and  CM2 are evaluated using a test data set derived from the real workingenvironment. Unfortunately, since the performance of the individual classifiers can change intime due to the learning capabilities of the classifiers or to the variabilit y in input data quality,there is no assurance that the combination method will provide acceptable results for thespecific application in the working environment. Therefore, it is important that theeffectiveness of methods for classifier combination M i is evaluated also when thecharacteristics of the set of individual classifiers change.3 A perturbation-based approach to combination method selectionLet A1,A2,…,AK be the individual classifiers combined by a combination method M. Letus consider the vector of responses obtained by inputting the patterns xt belonging to adatabase T. If T contains N patterns, the vector of responses consists of  N elements eachhaving K responses (one for each classifier) (A1(t), A2(t), …, AK(t)) ,  t=1,2,,N.From the multi -dimensional vector, two kinds of features are derived:   Features at the level of individual classifiers.The recognition rate of each classifier:RAi= ∑ttiR xtAQN)),((1where554î ==                                                                                 otherwise    0correct) is (t)A (i.e.   class  the tobelongs xand (t)A   if    1)),(( ijtjiωωtiR xtAQFeatures at the level of the set of classifiers.The correlation between classifiers [5]:∑⋅=<      1,...Kj=i,,21jijAiAAKρρwhere∑==NtjijAiA tAtAQN 1, ))(),((1ρρandî=          otherwise    0)(A=(t)A  if    1))(),((ji ttAtAQ jiρ .Therefore, the performance of the combination method M can be considered asdepending on the recognition rates of the individual classifiers R=(RA1,RA2,…,RAK) and onthe correlation ρ of the set of classifiersCE(R,ρ)= LE(R,ρ)+a⋅EE(R,ρ)                                 (1)where LM(R,ρ) and EM(R,ρ) denote the rejection rate and the substitution rate of thecombination method M as functions of R and ρ.The combination method M can be evaluated in different working conditions byperturbing the characteristics of the classifiers by mean of an automatic procedure. For thispurpose the vector of responses of the individual classifiers can be modified according to thefollowing considerations:•  Modification of features at the level of individual classifiers: For each classifier avariation is assumed in its recognition rate. For instance, for Ai whose recognition rate isoriginally of RAi , we have that the new recognition rate R*Ai  will  range in [RAi - δRAi ,RAi + δRAi], where δRAi is suitably defined.•  Modification of features at the level of the set of classifiers: A variation is assumed forthe correlation ρ of the entire set of classifiers. Specifically we have that the newcorrelation ρ* will range in [ρ-δρ, ρ+δρ]where δρ is suitably defined.For example, let us consider the vector of responses in Figure 1 for A1,A2,A3,A4 (N=10),where: R→Recognition; S1,S2,S3,S4→Substitution (rejections are not considered in thisexample). It results R=(0.7,0.8,0.7,0.6) (in fact: RA1=0.7, RA2=0.8, RA3=0.7, RA4=0.6), andρ=3.2/6. In this case, if we combine A1,A2,A3,A4 by the combination method M, we willachieve the value CM(R, ρ)= CM(0.7, 0.8, 0.7, 0.6, 3.2/6). In order to investigate thebehaviour of M, a perturbation can be performed on the vector of responses according to the555considerations (a) and (b). In Figure 2 a perturbed version of the vector of responses is shown(the modifications are marked with "^"). First, the output of A4 for pattern 3 has beenchanged (this modification changes the recognition rate of A4 and also the value of ρ for theentire set of classifiers). Second, the outputs of A2 for patterns 7 and 8 have been swapped(this modification does not change the recognition rate of A2 but only the value of ρ for theentire set of classifiers). It is easy to verify that, for the case in Figure 2 we haveR*=(0.7,0.8,0.7,0.7) (in fact: R*A1=0.6, R*A2=0.6, R*A3=0.6, R*A4=0.7), and ρ*=3.8/6.A1 A2 A3 A4Pattern 1 R R R RPattern 2 R R R S1Pattern 3 R R S3 S2Pattern 4 S1 S4 R S3Pattern 5 R R R RPattern 6 R R R RPattern 7 S3 R S2 RPattern 8 S2 R S1 S1Pattern 9 R S2 R RPattern 10 R R R RFigure 1: Vector of responses (0.7,0.8,0.7,0.6,3.2/6)A1 A2 A3 A4Pattern 1 R R R RPattern 2 R R R S1Pattern 3 R R S3 R(^)Pattern 4 S1 S4 R S3Pattern 5 R R R RPattern 6 R R R RPattern 7 S3 R(^) S2 RPattern 8 S2 S2(^) S1 S1Pattern 9 R R R RPattern 10 R R R RFigure 2: Vector of responses after perturbation (0.7,0.8,0.7,0.7,3.8/6).Modifications: A4(3)=R (Fig. 1: A4(3)=S2);A2(7)=R ; A2(8)=S2 (Fig. 1: A2(7)=S2 ; A2(8)=R).Now, let us consider the set of values assumed by the cost function for M when theentire set of  perturbed vector is considered M={ CM(R*, ρ*) |(R* , ρ*)∈  I*} where theperturbation range I* is defined as I* =[RA1-δRA1 ,RA1+δRA1]×[RA2-δRA2,RA2+δRA2]×..×[RAK-δRAK , RAK+δRAK]×[ρ-δρ,ρ+δρ].From the set M , useful information on the effectiveness of  M in different workingconditions can be derived:CM(R*, ρ*)CM(R, ρ)556 max(  M ): the maximum value in  M; min(  M ): the minimum value in  M; M(  M ): the mean value of  M; SD(  M ): the standard deviation of  M.These indexes allow a more accurate selection of the combination method for a multi -classifier system depending on the particular application field. Typical selection criteria canbe:  max(  M ) minimum (worst performance as good as possible); max(  M ) minimum (best performance as good as possible); M(  M ) minimum (best performance on average); SD(  M ) minimum (performance as stable as possible).4 Experimental ResultsThe proposed approach has been applied to a system for hand-written numeralrecognition that implements both the Behavioural Knowledge Space Method (BKS) [7] andthe Dempster-Shafer Method (DS) [8]. The system [9] combines four classifiers trained withabout 18468 numerals of the CEDAR Database (BR directory) [10]: Region (RA1=0.91),Contour (RA2=0.87), Loci (RA3=0.90), Histogram (RA4=0.87).Initially, from the vector of responses of the set of classifiers (tested with data of the BSdirectory of the CEDAR database) we derive the values R=(0.91,0.87,0.90,0.87) and ρ=0.84.From eq. (1), (with a=10), it results CBKS=0,485 and CDS =0,472. Hence DS outcomes BKS.Now, when the perturbation-based approach is applied to investigate the behaviour ofBKS(R*, ρ*) and DS(R*, ρ*) for (R*, ρ*)∈  I*, where δRAi=0.01 (a variation of 1% isconsidered for the recognition rate of the individual classifiers) and δρ=0.03, we obtain thesets  BKS and  DS as Table 1 shows.Table 1: Perturbation-based analysis of BKS and DS BKS  DSmax(  M ) 0.525 0.690min(  M ) 0.475 0.323M(  M ) 0.493 0.491SD(  M ) 0.019 0.156The result in Table 1 shows that, although DS and BKS have similar mean performance(M(  DS )≅ M(  BKS)), DS outperforms BKS in terms of best result ((min(  DS )<min(  BKS)),while BKS results more stable than DS ((SD(  DS )>SD(  BKS)) and it is also superior to DS interms of worst result ((max(  DS )>max(  BKS)).5575 ConclusionsThis paper presents a new perturbation-based approach for the selection of thecombination method best suited in a multi -classifier system for abstract-level classifiers. Theapproach provides useful information to evaluate the effectiveness of a combination methodin real environments, where modifications of  working  conditions can occur due to variationsin quality of input patterns or to learning capabili ties of classifiers.References 1.   L.Xu, A.Krzyzak, C.-Y Suen, “Methods of Combining Multiple Classifiers and TheirApplications to Handwriting Recognition” , IEEE SMC, Vol. 22, N.3, 1992, pp. 418-435. 2.   L. Lam, Y.S. Huang, C.Y. Suen, “Combination of Multiple Classifier Decisions forOptical Character Recognition” , in Handbook of Character Recognition and DocumentImage Analysis, Ed. H. Bunke and P.S.P. Wang, World Scientific, Singapore, 1997, pp.79-101. 3.   J. Kittler, M. Hatef, R.P.W. Duin, J. Matas, "On Combining Classifiers", IEEE PAMI,Vol. 20, n. 3, 1998, pp. 226-238. 4.   C. Nadal, R. Legault, and C.Y. Suen, “Complementary algorithms for the recognition oftotally un-constrained hand-written numerals” , Proc. 10th ICPR., Vol. A, June 1990, pp.434-449. 5.   G. Dimauro, S. Impedovo, G. Pirlo, “Multiple Experts: a new methodology for theevaluation of the combination processes” , Progress in Handwriting Recognition, ed. byA.C.Downton and S. Impedovo, World Scientific Publishing Co. Pte. Ltd., Singapore,1997, pp. 329-335. 6.   A.F.R. Rahman, M.C. Fairhurst, “ Introducing New Multiple Expert DecisionCombination Topologies: A Case Study using Recognition of Hand-written Characters” ,Proc. ICDAR’97, Ulm, Germany, 1997, pp. 886-891. 7.   Huang, C.Y. Suen, “An Optimal Method of Combining Multiple Classifiers forUnconstrained Handwritten Numeral Recognition” , Proc. of IWFHR-3, Buffalo, NY,1993, pp. 11-20. 8.   E. Mandler and J. Schuermann, “Combining the Classification Results of independentclassifiers based on the Dempster/Shafer theory of evidence”, in  Pattern Recognitionand Artifi cial Intell igence, eds. E.S.Gelsema and L.N.Kanal, North Holland,Amsterdam, 1988, pp. 381-393. 9.   G. Dimauro, S. Impedovo, G. Pirlo, A. Salzo, “Automatic Bankchecks Processing : ANew Engineered System”, International Journal of Pattern Recognition and Artifi cialIntelli gence, Vol. 11, N.4, World Scientific Publ., Singapore, 1997, pp. 1-38. 10.   J.J. Hull , “A database for handwritten text recognition research", IEEE PAMI, Vol. 16,N. 5, 1994, pp. 550-554.558

Dissertations of the University of Groningen

   University of GroningenA PERTURBATION­BASED APPROACH FOR MULTI­CLASSIFIER SYSTEM DESIGNLecce Di, V.; Dimauro, G.; Guerriero, A.; Impedovo, S.; Pirlo, G.; Salzo, A.Published in:EPRINTS-BOOK-TITLEIMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite fromit. Please check the document version below.Document VersionPublisher's PDF, also known as Version of recordPublication date:2004Link to publication in University of Groningen/UMCG research databaseCitation for published version (APA):Lecce Di, V., Dimauro, G., Guerriero, A., Impedovo, S., Pirlo, G., & Salzo, A. (2004). APERTURBATION­BASED APPROACH FOR MULTI­CLASSIFIER SYSTEM DESIGN. In EPRINTS-BOOK-TITLE s.n..CopyrightOther than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of theauthor(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).The publication may also be distributed here under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license.More information can be found on the University of Groningen website: https://www.rug.nl/library/open-access/self-archiving-pure/taverne-amendment.Take-down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons thenumber of authors shown on this cover page is limited to 10 maximum.Download date: 04-06-2022A PERTURBATION-BASED APPROACH FOR MULTI-CLASSIFIERSYSTEM DESIGNV.DI LECCE1, G.DIMAURO2, A.GUERRIERO1, S.IMPEDOVO2, G.PIRLO2, A.SALZO2(1) Dipartimento di Ing. Elettronica - Politecnico di Bari-Via Re David - 70126 Bari - Italy(2) Dipartimento di Informatica - Università di Bari - Via Orabona, 4 - 70126 Bari –ItalyThis paper presents a perturbation-based approach useful to select the best combinationmethod for a multi-classifier system. The basic idea is to simulate small variations in theperformance of the set of classifiers and to evaluate to what extent they influence theperformance of the combined classifier. In the experimental phase, the BehaviouralKnowledge Space and the Dempster-Shafer combination methods have been considered. Theexperimental results, carried out in the field of  hand-written numeral recognition,demonstrate the  effectiveness of the new approach.1 IntroductionThe interest of researchers in multi -classifier systems is motivated by the consideration thatthey allow high performances also in the cases of difficult classification problems [1]. Sincetheoretical analysis of combination methods can be very difficult [2,3] the evaluation ofcomplex combination methods is generally carried out on experimental basis [4,5]. The netresult is that the design of multi -classifier systems still presents many unsolved issues [2,3].An important issue is related to the selection of the best method to combine theindividual classifier for a specific application. This aspect is generally evaluated bymeasuring the performance of  the methods by using data sets derived from the realapplication. In this case, there is no assurance that similar results can be obtained in theworking environment in which the performance of the individual classifiers can change [5].In this paper a perturbation-based approach is presented for the selection of the bestcombination method for a multi -classifier system. Small variations in the performance of theset of classifiers are simulated and their effect to the performance of the combined classifieris evaluated. Some indexes are also provided to obtain information on the performance of thecombined classifiers.The organisation of the paper is the following: Section 2 presents the problem ofselection of the combination method for multi -classifier system. The new perturbation-basedapproach is presented in Section 3. The experimental results are reported in Section 4.553In: L.R.B. Schomaker and L.G. Vuurpijl (Eds.), Proceedings of the Seventh International Workshop on Frontiers in Handwriting Recognition,September 11-13 2000, Amsterdam, ISBN 90-76942-01-3, Nijmegen: International Unipen Foundation, pp 553-5582 Multi-Classifier System DesignIn a multi -classifier system the final decision is obtained by combining the decisions ofseveral classifiers [1]. When a parallel-combination topology is considered [6], the inputpattern pt is provided to each classifier Ai, i=1,..,K which decides the membership of thepattern xt to the pattern classes ω1, ω2,..., ωm . Let Ai(t) be the response of the i-th classifierwhen the pattern xt is processed, the combination method M provides the final response bycombining the responses of the individual classifiers. From the outputs of the individualclassifiers, the method computes a confidence score S(ωi) for each class ωi, i=1,2,...,m.Successively a decision rule is used to produce the final classification response.The performance of a combination method is generally evaluated on experimental basisby considering measures as the recognition rate RM, the substitution rate EM and the rejectionrate LM, or other indexes and cost functions generally defined as a linear combination of RM ,EM and LM [1,2].Now, let be CM= LM + a⋅EM  the cost function considered for the evaluation of thecombination methods [6], the selection of the best method among M1 and M2 for a multi -classifier system follows the rules:   If CM1 < CM2 then use M1   If CM1 > CM2 then use M2   If CM1 = CM2 then use M1 or M2 equivalently.where CM1 and  CM2 are evaluated using a test data set derived from the real workingenvironment. Unfortunately, since the performance of the individual classifiers can change intime due to the learning capabilities of the classifiers or to the variabilit y in input data quality,there is no assurance that the combination method will provide acceptable results for thespecific application in the working environment. Therefore, it is important that theeffectiveness of methods for classifier combination M i is evaluated also when thecharacteristics of the set of individual classifiers change.3 A perturbation-based approach to combination method selectionLet A1,A2,…,AK be the individual classifiers combined by a combination method M. Letus consider the vector of responses obtained by inputting the patterns xt belonging to adatabase T. If T contains N patterns, the vector of responses consists of  N elements eachhaving K responses (one for each classifier) (A1(t), A2(t), …, AK(t)) ,  t=1,2,,N.From the multi -dimensional vector, two kinds of features are derived:   Features at the level of individual classifiers.The recognition rate of each classifier:RAi= ∑ttiR xtAQN)),((1where554î ==                                                                                 otherwise    0correct) is (t)A (i.e.   class  the tobelongs xand (t)A   if    1)),(( ijtjiωωtiR xtAQFeatures at the level of the set of classifiers.The correlation between classifiers [5]:∑⋅=<      1,...Kj=i,,21jijAiAAKρρwhere∑==NtjijAiA tAtAQN 1, ))(),((1ρρandî=          otherwise    0)(A=(t)A  if    1))(),((ji ttAtAQ jiρ .Therefore, the performance of the combination method M can be considered asdepending on the recognition rates of the individual classifiers R=(RA1,RA2,…,RAK) and onthe correlation ρ of the set of classifiersCE(R,ρ)= LE(R,ρ)+a⋅EE(R,ρ)                                 (1)where LM(R,ρ) and EM(R,ρ) denote the rejection rate and the substitution rate of thecombination method M as functions of R and ρ.The combination method M can be evaluated in different working conditions byperturbing the characteristics of the classifiers by mean of an automatic procedure. For thispurpose the vector of responses of the individual classifiers can be modified according to thefollowing considerations:•  Modification of features at the level of individual classifiers: For each classifier avariation is assumed in its recognition rate. For instance, for Ai whose recognition rate isoriginally of RAi , we have that the new recognition rate R*Ai  will  range in [RAi - δRAi ,RAi + δRAi], where δRAi is suitably defined.•  Modification of features at the level of the set of classifiers: A variation is assumed forthe correlation ρ of the entire set of classifiers. Specifically we have that the newcorrelation ρ* will range in [ρ-δρ, ρ+δρ]where δρ is suitably defined.For example, let us consider the vector of responses in Figure 1 for A1,A2,A3,A4 (N=10),where: R→Recognition; S1,S2,S3,S4→Substitution (rejections are not considered in thisexample). It results R=(0.7,0.8,0.7,0.6) (in fact: RA1=0.7, RA2=0.8, RA3=0.7, RA4=0.6), andρ=3.2/6. In this case, if we combine A1,A2,A3,A4 by the combination method M, we willachieve the value CM(R, ρ)= CM(0.7, 0.8, 0.7, 0.6, 3.2/6). In order to investigate thebehaviour of M, a perturbation can be performed on the vector of responses according to the555considerations (a) and (b). In Figure 2 a perturbed version of the vector of responses is shown(the modifications are marked with "^"). First, the output of A4 for pattern 3 has beenchanged (this modification changes the recognition rate of A4 and also the value of ρ for theentire set of classifiers). Second, the outputs of A2 for patterns 7 and 8 have been swapped(this modification does not change the recognition rate of A2 but only the value of ρ for theentire set of classifiers). It is easy to verify that, for the case in Figure 2 we haveR*=(0.7,0.8,0.7,0.7) (in fact: R*A1=0.6, R*A2=0.6, R*A3=0.6, R*A4=0.7), and ρ*=3.8/6.A1 A2 A3 A4Pattern 1 R R R RPattern 2 R R R S1Pattern 3 R R S3 S2Pattern 4 S1 S4 R S3Pattern 5 R R R RPattern 6 R R R RPattern 7 S3 R S2 RPattern 8 S2 R S1 S1Pattern 9 R S2 R RPattern 10 R R R RFigure 1: Vector of responses (0.7,0.8,0.7,0.6,3.2/6)A1 A2 A3 A4Pattern 1 R R R RPattern 2 R R R S1Pattern 3 R R S3 R(^)Pattern 4 S1 S4 R S3Pattern 5 R R R RPattern 6 R R R RPattern 7 S3 R(^) S2 RPattern 8 S2 S2(^) S1 S1Pattern 9 R R R RPattern 10 R R R RFigure 2: Vector of responses after perturbation (0.7,0.8,0.7,0.7,3.8/6).Modifications: A4(3)=R (Fig. 1: A4(3)=S2);A2(7)=R ; A2(8)=S2 (Fig. 1: A2(7)=S2 ; A2(8)=R).Now, let us consider the set of values assumed by the cost function for M when theentire set of  perturbed vector is considered M={ CM(R*, ρ*) |(R* , ρ*)∈  I*} where theperturbation range I* is defined as I* =[RA1-δRA1 ,RA1+δRA1]×[RA2-δRA2,RA2+δRA2]×..×[RAK-δRAK , RAK+δRAK]×[ρ-δρ,ρ+δρ].From the set M , useful information on the effectiveness of  M in different workingconditions can be derived:CM(R*, ρ*)CM(R, ρ)556 max(  M ): the maximum value in  M; min(  M ): the minimum value in  M; M(  M ): the mean value of  M; SD(  M ): the standard deviation of  M.These indexes allow a more accurate selection of the combination method for a multi -classifier system depending on the particular application field. Typical selection criteria canbe:  max(  M ) minimum (worst performance as good as possible); max(  M ) minimum (best performance as good as possible); M(  M ) minimum (best performance on average); SD(  M ) minimum (performance as stable as possible).4 Experimental ResultsThe proposed approach has been applied to a system for hand-written numeralrecognition that implements both the Behavioural Knowledge Space Method (BKS) [7] andthe Dempster-Shafer Method (DS) [8]. The system [9] combines four classifiers trained withabout 18468 numerals of the CEDAR Database (BR directory) [10]: Region (RA1=0.91),Contour (RA2=0.87), Loci (RA3=0.90), Histogram (RA4=0.87).Initially, from the vector of responses of the set of classifiers (tested with data of the BSdirectory of the CEDAR database) we derive the values R=(0.91,0.87,0.90,0.87) and ρ=0.84.From eq. (1), (with a=10), it results CBKS=0,485 and CDS =0,472. Hence DS outcomes BKS.Now, when the perturbation-based approach is applied to investigate the behaviour ofBKS(R*, ρ*) and DS(R*, ρ*) for (R*, ρ*)∈  I*, where δRAi=0.01 (a variation of 1% isconsidered for the recognition rate of the individual classifiers) and δρ=0.03, we obtain thesets  BKS and  DS as Table 1 shows.Table 1: Perturbation-based analysis of BKS and DS BKS  DSmax(  M ) 0.525 0.690min(  M ) 0.475 0.323M(  M ) 0.493 0.491SD(  M ) 0.019 0.156The result in Table 1 shows that, although DS and BKS have similar mean performance(M(  DS )≅ M(  BKS)), DS outperforms BKS in terms of best result ((min(  DS )<min(  BKS)),while BKS results more stable than DS ((SD(  DS )>SD(  BKS)) and it is also superior to DS interms of worst result ((max(  DS )>max(  BKS)).5575 ConclusionsThis paper presents a new perturbation-based approach for the selection of thecombination method best suited in a multi -classifier system for abstract-level classifiers. Theapproach provides useful information to evaluate the effectiveness of a combination methodin real environments, where modifications of  working  conditions can occur due to variationsin quality of input patterns or to learning capabili ties of classifiers.References 1.   L.Xu, A.Krzyzak, C.-Y Suen, “Methods of Combining Multiple Classifiers and TheirApplications to Handwriting Recognition” , IEEE SMC, Vol. 22, N.3, 1992, pp. 418-435. 2.   L. Lam, Y.S. Huang, C.Y. Suen, “Combination of Multiple Classifier Decisions forOptical Character Recognition” , in Handbook of Character Recognition and DocumentImage Analysis, Ed. H. Bunke and P.S.P. Wang, World Scientific, Singapore, 1997, pp.79-101. 3.   J. Kittler, M. Hatef, R.P.W. Duin, J. Matas, "On Combining Classifiers", IEEE PAMI,Vol. 20, n. 3, 1998, pp. 226-238. 4.   C. Nadal, R. Legault, and C.Y. Suen, “Complementary algorithms for the recognition oftotally un-constrained hand-written numerals” , Proc. 10th ICPR., Vol. A, June 1990, pp.434-449. 5.   G. Dimauro, S. Impedovo, G. Pirlo, “Multiple Experts: a new methodology for theevaluation of the combination processes” , Progress in Handwriting Recognition, ed. byA.C.Downton and S. Impedovo, World Scientific Publishing Co. Pte. Ltd., Singapore,1997, pp. 329-335. 6.   A.F.R. Rahman, M.C. Fairhurst, “ Introducing New Multiple Expert DecisionCombination Topologies: A Case Study using Recognition of Hand-written Characters” ,Proc. ICDAR’97, Ulm, Germany, 1997, pp. 886-891. 7.   Huang, C.Y. Suen, “An Optimal Method of Combining Multiple Classifiers forUnconstrained Handwritten Numeral Recognition” , Proc. of IWFHR-3, Buffalo, NY,1993, pp. 11-20. 8.   E. Mandler and J. Schuermann, “Combining the Classification Results of independentclassifiers based on the Dempster/Shafer theory of evidence”, in  Pattern Recognitionand Artifi cial Intell igence, eds. E.S.Gelsema and L.N.Kanal, North Holland,Amsterdam, 1988, pp. 381-393. 9.   G. Dimauro, S. Impedovo, G. Pirlo, A. Salzo, “Automatic Bankchecks Processing : ANew Engineered System”, International Journal of Pattern Recognition and Artifi cialIntelli gence, Vol. 11, N.4, World Scientific Publ., Singapore, 1997, pp. 1-38. 10.   J.J. Hull , “A database for handwritten text recognition research", IEEE PAMI, Vol. 16,N. 5, 1994, pp. 550-554.558

ARTS repository - University of Groningen

   University of GroningenA PERTURBATION­BASED APPROACH FOR MULTI­CLASSIFIER SYSTEM DESIGNLecce Di, V.; Dimauro, G.; Guerriero, A.; Impedovo, S.; Pirlo, G.; Salzo, A.Published in:EPRINTS-BOOK-TITLEIMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite fromit. Please check the document version below.Document VersionPublisher's PDF, also known as Version of recordPublication date:2004Link to publication in University of Groningen/UMCG research databaseCitation for published version (APA):Lecce Di, V., Dimauro, G., Guerriero, A., Impedovo, S., Pirlo, G., & Salzo, A. (2004). APERTURBATION­BASED APPROACH FOR MULTI­CLASSIFIER SYSTEM DESIGN. In EPRINTS-BOOK-TITLE s.n..CopyrightOther than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of theauthor(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).The publication may also be distributed here under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license.More information can be found on the University of Groningen website: https://www.rug.nl/library/open-access/self-archiving-pure/taverne-amendment.Take-down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons thenumber of authors shown on this cover page is limited to 10 maximum.Download date: 06-06-2022A PERTURBATION-BASED APPROACH FOR MULTI-CLASSIFIERSYSTEM DESIGNV.DI LECCE1, G.DIMAURO2, A.GUERRIERO1, S.IMPEDOVO2, G.PIRLO2, A.SALZO2(1) Dipartimento di Ing. Elettronica - Politecnico di Bari-Via Re David - 70126 Bari - Italy(2) Dipartimento di Informatica - Università di Bari - Via Orabona, 4 - 70126 Bari –ItalyThis paper presents a perturbation-based approach useful to select the best combinationmethod for a multi-classifier system. The basic idea is to simulate small variations in theperformance of the set of classifiers and to evaluate to what extent they influence theperformance of the combined classifier. In the experimental phase, the BehaviouralKnowledge Space and the Dempster-Shafer combination methods have been considered. Theexperimental results, carried out in the field of  hand-written numeral recognition,demonstrate the  effectiveness of the new approach.1 IntroductionThe interest of researchers in multi -classifier systems is motivated by the consideration thatthey allow high performances also in the cases of difficult classification problems [1]. Sincetheoretical analysis of combination methods can be very difficult [2,3] the evaluation ofcomplex combination methods is generally carried out on experimental basis [4,5]. The netresult is that the design of multi -classifier systems still presents many unsolved issues [2,3].An important issue is related to the selection of the best method to combine theindividual classifier for a specific application. This aspect is generally evaluated bymeasuring the performance of  the methods by using data sets derived from the realapplication. In this case, there is no assurance that similar results can be obtained in theworking environment in which the performance of the individual classifiers can change [5].In this paper a perturbation-based approach is presented for the selection of the bestcombination method for a multi -classifier system. Small variations in the performance of theset of classifiers are simulated and their effect to the performance of the combined classifieris evaluated. Some indexes are also provided to obtain information on the performance of thecombined classifiers.The organisation of the paper is the following: Section 2 presents the problem ofselection of the combination method for multi -classifier system. The new perturbation-basedapproach is presented in Section 3. The experimental results are reported in Section 4.553In: L.R.B. Schomaker and L.G. Vuurpijl (Eds.), Proceedings of the Seventh International Workshop on Frontiers in Handwriting Recognition,September 11-13 2000, Amsterdam, ISBN 90-76942-01-3, Nijmegen: International Unipen Foundation, pp 553-5582 Multi-Classifier System DesignIn a multi -classifier system the final decision is obtained by combining the decisions ofseveral classifiers [1]. When a parallel-combination topology is considered [6], the inputpattern pt is provided to each classifier Ai, i=1,..,K which decides the membership of thepattern xt to the pattern classes ω1, ω2,..., ωm . Let Ai(t) be the response of the i-th classifierwhen the pattern xt is processed, the combination method M provides the final response bycombining the responses of the individual classifiers. From the outputs of the individualclassifiers, the method computes a confidence score S(ωi) for each class ωi, i=1,2,...,m.Successively a decision rule is used to produce the final classification response.The performance of a combination method is generally evaluated on experimental basisby considering measures as the recognition rate RM, the substitution rate EM and the rejectionrate LM, or other indexes and cost functions generally defined as a linear combination of RM ,EM and LM [1,2].Now, let be CM= LM + a⋅EM  the cost function considered for the evaluation of thecombination methods [6], the selection of the best method among M1 and M2 for a multi -classifier system follows the rules:   If CM1 < CM2 then use M1   If CM1 > CM2 then use M2   If CM1 = CM2 then use M1 or M2 equivalently.where CM1 and  CM2 are evaluated using a test data set derived from the real workingenvironment. Unfortunately, since the performance of the individual classifiers can change intime due to the learning capabilities of the classifiers or to the variabilit y in input data quality,there is no assurance that the combination method will provide acceptable results for thespecific application in the working environment. Therefore, it is important that theeffectiveness of methods for classifier combination M i is evaluated also when thecharacteristics of the set of individual classifiers change.3 A perturbation-based approach to combination method selectionLet A1,A2,…,AK be the individual classifiers combined by a combination method M. Letus consider the vector of responses obtained by inputting the patterns xt belonging to adatabase T. If T contains N patterns, the vector of responses consists of  N elements eachhaving K responses (one for each classifier) (A1(t), A2(t), …, AK(t)) ,  t=1,2,,N.From the multi -dimensional vector, two kinds of features are derived:   Features at the level of individual classifiers.The recognition rate of each classifier:RAi= ∑ttiR xtAQN)),((1where554î ==                                                                                 otherwise    0correct) is (t)A (i.e.   class  the tobelongs xand (t)A   if    1)),(( ijtjiωωtiR xtAQFeatures at the level of the set of classifiers.The correlation between classifiers [5]:∑⋅=<      1,...Kj=i,,21jijAiAAKρρwhere∑==NtjijAiA tAtAQN 1, ))(),((1ρρandî=          otherwise    0)(A=(t)A  if    1))(),((ji ttAtAQ jiρ .Therefore, the performance of the combination method M can be considered asdepending on the recognition rates of the individual classifiers R=(RA1,RA2,…,RAK) and onthe correlation ρ of the set of classifiersCE(R,ρ)= LE(R,ρ)+a⋅EE(R,ρ)                                 (1)where LM(R,ρ) and EM(R,ρ) denote the rejection rate and the substitution rate of thecombination method M as functions of R and ρ.The combination method M can be evaluated in different working conditions byperturbing the characteristics of the classifiers by mean of an automatic procedure. For thispurpose the vector of responses of the individual classifiers can be modified according to thefollowing considerations:•  Modification of features at the level of individual classifiers: For each classifier avariation is assumed in its recognition rate. For instance, for Ai whose recognition rate isoriginally of RAi , we have that the new recognition rate R*Ai  will  range in [RAi - δRAi ,RAi + δRAi], where δRAi is suitably defined.•  Modification of features at the level of the set of classifiers: A variation is assumed forthe correlation ρ of the entire set of classifiers. Specifically we have that the newcorrelation ρ* will range in [ρ-δρ, ρ+δρ]where δρ is suitably defined.For example, let us consider the vector of responses in Figure 1 for A1,A2,A3,A4 (N=10),where: R→Recognition; S1,S2,S3,S4→Substitution (rejections are not considered in thisexample). It results R=(0.7,0.8,0.7,0.6) (in fact: RA1=0.7, RA2=0.8, RA3=0.7, RA4=0.6), andρ=3.2/6. In this case, if we combine A1,A2,A3,A4 by the combination method M, we willachieve the value CM(R, ρ)= CM(0.7, 0.8, 0.7, 0.6, 3.2/6). In order to investigate thebehaviour of M, a perturbation can be performed on the vector of responses according to the555considerations (a) and (b). In Figure 2 a perturbed version of the vector of responses is shown(the modifications are marked with "^"). First, the output of A4 for pattern 3 has beenchanged (this modification changes the recognition rate of A4 and also the value of ρ for theentire set of classifiers). Second, the outputs of A2 for patterns 7 and 8 have been swapped(this modification does not change the recognition rate of A2 but only the value of ρ for theentire set of classifiers). It is easy to verify that, for the case in Figure 2 we haveR*=(0.7,0.8,0.7,0.7) (in fact: R*A1=0.6, R*A2=0.6, R*A3=0.6, R*A4=0.7), and ρ*=3.8/6.A1 A2 A3 A4Pattern 1 R R R RPattern 2 R R R S1Pattern 3 R R S3 S2Pattern 4 S1 S4 R S3Pattern 5 R R R RPattern 6 R R R RPattern 7 S3 R S2 RPattern 8 S2 R S1 S1Pattern 9 R S2 R RPattern 10 R R R RFigure 1: Vector of responses (0.7,0.8,0.7,0.6,3.2/6)A1 A2 A3 A4Pattern 1 R R R RPattern 2 R R R S1Pattern 3 R R S3 R(^)Pattern 4 S1 S4 R S3Pattern 5 R R R RPattern 6 R R R RPattern 7 S3 R(^) S2 RPattern 8 S2 S2(^) S1 S1Pattern 9 R R R RPattern 10 R R R RFigure 2: Vector of responses after perturbation (0.7,0.8,0.7,0.7,3.8/6).Modifications: A4(3)=R (Fig. 1: A4(3)=S2);A2(7)=R ; A2(8)=S2 (Fig. 1: A2(7)=S2 ; A2(8)=R).Now, let us consider the set of values assumed by the cost function for M when theentire set of  perturbed vector is considered M={ CM(R*, ρ*) |(R* , ρ*)∈  I*} where theperturbation range I* is defined as I* =[RA1-δRA1 ,RA1+δRA1]×[RA2-δRA2,RA2+δRA2]×..×[RAK-δRAK , RAK+δRAK]×[ρ-δρ,ρ+δρ].From the set M , useful information on the effectiveness of  M in different workingconditions can be derived:CM(R*, ρ*)CM(R, ρ)556 max(  M ): the maximum value in  M; min(  M ): the minimum value in  M; M(  M ): the mean value of  M; SD(  M ): the standard deviation of  M.These indexes allow a more accurate selection of the combination method for a multi -classifier system depending on the particular application field. Typical selection criteria canbe:  max(  M ) minimum (worst performance as good as possible); max(  M ) minimum (best performance as good as possible); M(  M ) minimum (best performance on average); SD(  M ) minimum (performance as stable as possible).4 Experimental ResultsThe proposed approach has been applied to a system for hand-written numeralrecognition that implements both the Behavioural Knowledge Space Method (BKS) [7] andthe Dempster-Shafer Method (DS) [8]. The system [9] combines four classifiers trained withabout 18468 numerals of the CEDAR Database (BR directory) [10]: Region (RA1=0.91),Contour (RA2=0.87), Loci (RA3=0.90), Histogram (RA4=0.87).Initially, from the vector of responses of the set of classifiers (tested with data of the BSdirectory of the CEDAR database) we derive the values R=(0.91,0.87,0.90,0.87) and ρ=0.84.From eq. (1), (with a=10), it results CBKS=0,485 and CDS =0,472. Hence DS outcomes BKS.Now, when the perturbation-based approach is applied to investigate the behaviour ofBKS(R*, ρ*) and DS(R*, ρ*) for (R*, ρ*)∈  I*, where δRAi=0.01 (a variation of 1% isconsidered for the recognition rate of the individual classifiers) and δρ=0.03, we obtain thesets  BKS and  DS as Table 1 shows.Table 1: Perturbation-based analysis of BKS and DS BKS  DSmax(  M ) 0.525 0.690min(  M ) 0.475 0.323M(  M ) 0.493 0.491SD(  M ) 0.019 0.156The result in Table 1 shows that, although DS and BKS have similar mean performance(M(  DS )≅ M(  BKS)), DS outperforms BKS in terms of best result ((min(  DS )<min(  BKS)),while BKS results more stable than DS ((SD(  DS )>SD(  BKS)) and it is also superior to DS interms of worst result ((max(  DS )>max(  BKS)).5575 ConclusionsThis paper presents a new perturbation-based approach for the selection of thecombination method best suited in a multi -classifier system for abstract-level classifiers. Theapproach provides useful information to evaluate the effectiveness of a combination methodin real environments, where modifications of  working  conditions can occur due to variationsin quality of input patterns or to learning capabili ties of classifiers.References 1.   L.Xu, A.Krzyzak, C.-Y Suen, “Methods of Combining Multiple Classifiers and TheirApplications to Handwriting Recognition” , IEEE SMC, Vol. 22, N.3, 1992, pp. 418-435. 2.   L. Lam, Y.S. Huang, C.Y. Suen, “Combination of Multiple Classifier Decisions forOptical Character Recognition” , in Handbook of Character Recognition and DocumentImage Analysis, Ed. H. Bunke and P.S.P. Wang, World Scientific, Singapore, 1997, pp.79-101. 3.   J. Kittler, M. Hatef, R.P.W. Duin, J. Matas, "On Combining Classifiers", IEEE PAMI,Vol. 20, n. 3, 1998, pp. 226-238. 4.   C. Nadal, R. Legault, and C.Y. Suen, “Complementary algorithms for the recognition oftotally un-constrained hand-written numerals” , Proc. 10th ICPR., Vol. A, June 1990, pp.434-449. 5.   G. Dimauro, S. Impedovo, G. Pirlo, “Multiple Experts: a new methodology for theevaluation of the combination processes” , Progress in Handwriting Recognition, ed. byA.C.Downton and S. Impedovo, World Scientific Publishing Co. Pte. Ltd., Singapore,1997, pp. 329-335. 6.   A.F.R. Rahman, M.C. Fairhurst, “ Introducing New Multiple Expert DecisionCombination Topologies: A Case Study using Recognition of Hand-written Characters” ,Proc. ICDAR’97, Ulm, Germany, 1997, pp. 886-891. 7.   Huang, C.Y. Suen, “An Optimal Method of Combining Multiple Classifiers forUnconstrained Handwritten Numeral Recognition” , Proc. of IWFHR-3, Buffalo, NY,1993, pp. 11-20. 8.   E. Mandler and J. Schuermann, “Combining the Classification Results of independentclassifiers based on the Dempster/Shafer theory of evidence”, in  Pattern Recognitionand Artifi cial Intell igence, eds. E.S.Gelsema and L.N.Kanal, North Holland,Amsterdam, 1988, pp. 381-393. 9.   G. Dimauro, S. Impedovo, G. Pirlo, A. Salzo, “Automatic Bankchecks Processing : ANew Engineered System”, International Journal of Pattern Recognition and Artifi cialIntelli gence, Vol. 11, N.4, World Scientific Publ., Singapore, 1997, pp. 1-38. 10.   J.J. Hull , “A database for handwritten text recognition research", IEEE PAMI, Vol. 16,N. 5, 1994, pp. 550-554.558

University of Groningen

   University of GroningenA PERTURBATION­BASED APPROACH FOR MULTI­CLASSIFIER SYSTEM DESIGNLecce Di, V.; Dimauro, G.; Guerriero, A.; Impedovo, S.; Pirlo, G.; Salzo, A.Published in:EPRINTS-BOOK-TITLEIMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite fromit. Please check the document version below.Document VersionPublisher's PDF, also known as Version of recordPublication date:2004Link to publication in University of Groningen/UMCG research databaseCitation for published version (APA):Lecce Di, V., Dimauro, G., Guerriero, A., Impedovo, S., Pirlo, G., & Salzo, A. (2004). APERTURBATION­BASED APPROACH FOR MULTI­CLASSIFIER SYSTEM DESIGN. In EPRINTS-BOOK-TITLE s.n..CopyrightOther than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of theauthor(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).Take-down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons thenumber of authors shown on this cover page is limited to 10 maximum.Download date: 12-11-2019A PERTURBATION-BASED APPROACH FOR MULTI-CLASSIFIERSYSTEM DESIGNV.DI LECCE1, G.DIMAURO2, A.GUERRIERO1, S.IMPEDOVO2, G.PIRLO2, A.SALZO2(1) Dipartimento di Ing. Elettronica - Politecnico di Bari-Via Re David - 70126 Bari - Italy(2) Dipartimento di Informatica - Università di Bari - Via Orabona, 4 - 70126 Bari –ItalyThis paper presents a perturbation-based approach useful to select the best combinationmethod for a multi-classifier system. The basic idea is to simulate small variations in theperformance of the set of classifiers and to evaluate to what extent they influence theperformance of the combined classifier. In the experimental phase, the BehaviouralKnowledge Space and the Dempster-Shafer combination methods have been considered. Theexperimental results, carried out in the field of  hand-written numeral recognition,demonstrate the  effectiveness of the new approach.1 IntroductionThe interest of researchers in multi-classifier systems is motivated by the consideration thatthey allow high performances also in the cases of difficult classification problems [1]. Sincetheoretical analysis of combination methods can be very difficult [2,3] the evaluation ofcomplex combination methods is generally carried out on experimental basis [4,5]. The netresult is that the design of multi-classifier systems still presents many unsolved issues [2,3].An important issue is related to the selection of the best method to combine theindividual classifier for a specific application. This aspect is generally evaluated bymeasuring the performance of  the methods by using data sets derived from the realapplication. In this case, there is no assurance that similar results can be obtained in theworking environment in which the performance of the individual classifiers can change [5].In this paper a perturbation-based approach is presented for the selection of the bestcombination method for a multi-classifier system. Small variations in the performance of theset of classifiers are simulated and their effect to the performance of the combined classifieris evaluated. Some indexes are also provided to obtain information on the performance of thecombined classifiers.The organisation of the paper is the following: Section 2 presents the problem ofselection of the combination method for multi-classifier system. The new perturbation-basedapproach is presented in Section 3. The experimental results are reported in Section 4.553In: L.R.B. Schomaker and L.G. Vuurpijl (Eds.), Proceedings of the Seventh International Workshop on Frontiers in Handwriting Recognition,September 11-13 2000, Amsterdam, ISBN 90-76942-01-3, Nijmegen: International Unipen Foundation, pp 553-5582 Multi-Classifier System DesignIn a multi-classifier system the final decision is obtained by combining the decisions ofseveral classifiers [1]. When a parallel-combination topology is considered [6], the inputpattern pt is provided to each classifier Ai, i=1,..,K which decides the membership of thepattern xt to the pattern classes w 1, w 2,..., w m . Let Ai(t) be the response of the i-th classifierwhen the pattern xt is processed, the combination method M provides the final response bycombining the responses of the individual classifiers. From the outputs of the individualclassifiers, the method computes a confidence score S( w i) for each class w i, i=1,2,...,m.Successively a decision rule is used to produce the final classification response.The performance of a combination method is generally evaluated on experimental basisby considering measures as the recognition rate RM, the substitution rate EM and the rejectionrate LM, or other indexes and cost functions generally defined as a linear combination of RM ,EM and LM [1,2].Now, let be CM= LM + a× EM  the cost function considered for the evaluation of thecombination methods [6], the selection of the best method among M1 and M2 for a multi-classifier system follows the rules:  If CM1 < CM2 then use M1  If CM1 > CM2 then use M2  If CM1 = CM2 then use M1 or M2 equivalently.where CM1 and  CM2 are evaluated using a test data set derived from the real workingenvironment. Unfortunately, since the performance of the individual classifiers can change intime due to the learning capabilities of the classifiers or to the variability in input data quality,there is no assurance that the combination method will provide acceptable results for thespecific application in the working environment. Therefore, it is important that theeffectiveness of methods for classifier combination Mi is evaluated also when thecharacteristics of the set of individual classifiers change.3 A perturbation-based approach to combination method selectionLet A1,A2,…,AK be the individual classifiers combined by a combination method M. Letus consider the vector of responses obtained by inputting the patterns xt belonging to adatabase T. If T contains N patterns, the vector of responses consists of  N elements eachhaving K responses (one for each classifier) (A1(t), A2(t), …, AK(t)) ,  t=1,2,,N.From the multi-dimensional vector, two kinds of features are derived:  Features at the level of individual classifiers.The recognition rate of each classifier:RAi= åttiR xtAQN )),((1where554îíì==                                                                                         otherwise    0correct) is (t)A (i.e.   class  the tobelongs xand (t)A   if    1)),(( ijtji wwtiR xtAQFeatures at the level of the set of classifiers.The correlation between classifiers [5]:œœœœßøŒŒŒŒºØå×œœßøŒŒºØ÷÷łöççŁæ=<      1,...Kj=i,,21jijAiAAKrrwhereå==NtjijAiA tAtAQN 1,))(),((1rrandïîïíì=          otherwise    0)(A=(t)A  if    1))(),((ji ttAtAQ jir .Therefore, the performance of the combination method M can be considered asdepending on the recognition rates of the individual classifiers R=(RA1,RA2,…,RAK) and onthe correlation r  of the set of classifiersCE(R, r )= LE(R, r )+a× EE(R, r )                                  (1)where LM(R, r ) and EM(R, r ) denote the rejection rate and the substitution rate of thecombination method M as functions of R and r .The combination method M can be evaluated in different working conditions byperturbing the characteristics of the classifiers by mean of an automatic procedure. For thispurpose the vector of responses of the individual classifiers can be modified according to thefollowing considerations:• Modification of features at the level of individual classifiers: For each classifier avariation is assumed in its recognition rate. For instance, for Ai whose recognition rate isoriginally of RAi , we have that the new recognition rate R*Ai  will  range in [RAi - d RAi ,RAi + d RAi], where d RAi is suitably defined.• Modification of features at the level of the set of classifiers: A variation is assumed forthe correlation r  of the entire set of classifiers. Specifically we have that the newcorrelation r * will range in [ r - d r , r +d r ]where d r  is suitably defined.For example, let us consider the vector of responses in Figure 1 for A1,A2,A3,A4 (N=10),where: Rfi Recognition; S1,S2,S3,S4 fi Substitution (rejections are not considered in thisexample). It results R=(0.7,0.8,0.7,0.6) (in fact: RA1=0.7, RA2=0.8, RA3=0.7, RA4=0.6), andr =3.2/6. In this case, if we combine A1,A2,A3,A4 by the combination method M, we willachieve the value CM(R, r )= CM(0.7, 0.8, 0.7, 0.6, 3.2/6). In order to investigate thebehaviour of M, a perturbation can be performed on the vector of responses according to the555considerations (a) and (b). In Figure 2 a perturbed version of the vector of responses is shown(the modifications are marked with "^"). First, the output of A4 for pattern 3 has beenchanged (this modification changes the recognition rate of A4 and also the value of r  for theentire set of classifiers). Second, the outputs of A2 for patterns 7 and 8 have been swapped(this modification does not change the recognition rate of A2 but only the value of r  for theentire set of classifiers). It is easy to verify that, for the case in Figure 2 we haveR*=(0.7,0.8,0.7,0.7) (in fact: R*A1=0.6, R*A2=0.6, R*A3=0.6, R*A4=0.7), and r *=3.8/6.A1 A2 A3 A4Pattern 1 R R R RPattern 2 R R R S1Pattern 3 R R S3 S2Pattern 4 S1 S4 R S3Pattern 5 R R R RPattern 6 R R R RPattern 7 S3 R S2 RPattern 8 S2 R S1 S1Pattern 9 R S2 R RPattern 10 R R R RFigure 1: Vector of responses (0.7,0.8,0.7,0.6,3.2/6)A1 A2 A3 A4Pattern 1 R R R RPattern 2 R R R S1Pattern 3 R R S3 R(^)Pattern 4 S1 S4 R S3Pattern 5 R R R RPattern 6 R R R RPattern 7 S3 R(^) S2 RPattern 8 S2 S2(^) S1 S1Pattern 9 R R R RPattern 10 R R R RFigure 2: Vector of responses after perturbation (0.7,0.8,0.7,0.7,3.8/6).Modifications: A4(3)=R (Fig. 1: A4(3)=S2);A2(7)=R ; A2(8)=S2 (Fig. 1: A2(7)=S2 ; A2(8)=R).Now, let us consider the set of values assumed by the cost function for M when theentire set of  perturbed vector is considered M={CM(R*, r *) |(R* , r *) ˛  I*} where theperturbation range I* is defined as I* =[RA1- d RA1 ,RA1+d RA1] · [RA2- d RA2,RA2+d RA2] · .. · [RAK-d RAK , RAK+d RAK] · [ r - d r , r +d r ].From the set M , useful information on the effectiveness of  M in different workingconditions can be derived:CM(R*, rr *)CM(R, rr )556 max(  M ): the maximum value in  M; min(  M ): the minimum value in  M; M(  M ): the mean value of  M; SD(  M ): the standard deviation of  M.These indexes allow a more accurate selection of the combination method for a multi-classifier system depending on the particular application field. Typical selection criteria canbe:  max(  M ) minimum (worst performance as good as possible); max(  M ) minimum (best performance as good as possible); M(  M ) minimum (best performance on average); SD(  M ) minimum (performance as stable as possible).4 Experimental ResultsThe proposed approach has been applied to a system for hand-written numeralrecognition that implements both the Behavioural Knowledge Space Method (BKS) [7] andthe Dempster-Shafer Method (DS) [8]. The system [9] combines four classifiers trained withabout 18468 numerals of the CEDAR Database (BR directory) [10]: Region (RA1=0.91),Contour (RA2=0.87), Loci (RA3=0.90), Histogram (RA4=0.87).Initially, from the vector of responses of the set of classifiers (tested with data of the BSdirectory of the CEDAR database) we derive the values R=(0.91,0.87,0.90,0.87) and r =0.84.From eq. (1), (with a=10), it results CBKS=0,485 and CDS =0,472. Hence DS outcomes BKS.Now, when the perturbation-based approach is applied to investigate the behaviour ofBKS(R*, r *) and DS(R*, r *) for (R*, r *) ˛  I*, where d RAi=0.01 (a variation of 1% isconsidered for the recognition rate of the individual classifiers) and d r =0.03, we obtain thesets  BKS and  DS as Table 1 shows.Table 1: Perturbation-based analysis of BKS and DS BKS  DSmax(  M ) 0.525 0.690min(  M ) 0.475 0.323M(  M ) 0.493 0.491SD(  M ) 0.019 0.156The result in Table 1 shows that, although DS and BKS have similar mean performance(M(  DS ) @ M(  BKS)), DS outperforms BKS in terms of best result ((min(  DS )<min(  BKS)),while BKS results more stable than DS ((SD(  DS )>SD(  BKS)) and it is also superior to DS interms of worst result ((max(  DS )>max(  BKS)).5575 ConclusionsThis paper presents a new perturbation-based approach for the selection of thecombination method best suited in a multi-classifier system for abstract-level classifiers. Theapproach provides useful information to evaluate the effectiveness of a combination methodin real environments, where modifications of  working  conditions can occur due to variationsin quality of input patterns or to learning capabilities of classifiers.References 1.   L.Xu, A.Krzyzak, C.-Y Suen, “Methods of Combining Multiple Classifiers and TheirApplications to Handwriting Recognition”, IEEE SMC, Vol. 22, N.3, 1992, pp. 418-435. 2.   L. Lam, Y.S. Huang, C.Y. Suen, “Combination of Multiple Classifier Decisions forOptical Character Recognition”, in Handbook of Character Recognition and DocumentImage Analysis, Ed. H. Bunke and P.S.P. Wang, World Scientific, Singapore, 1997, pp.79-101. 3.   J. Kittler, M. Hatef, R.P.W. Duin, J. Matas, "On Combining Classifiers", IEEE PAMI,Vol. 20, n. 3, 1998, pp. 226-238. 4.   C. Nadal, R. Legault, and C.Y. Suen, “Complementary algorithms for the recognition oftotally un-constrained hand-written numerals”, Proc. 10th ICPR., Vol. A, June 1990, pp.434-449. 5.   G. Dimauro, S. Impedovo, G. Pirlo, “Multiple Experts: a new methodology for theevaluation of the combination processes”, Progress in Handwriting Recognition, ed. byA.C.Downton and S. Impedovo, World Scientific Publishing Co. Pte. Ltd., Singapore,1997, pp. 329-335. 6.   A.F.R. Rahman, M.C. Fairhurst, “Introducing New Multiple Expert DecisionCombination Topologies: A Case Study using Recognition of Hand-written Characters”,Proc. ICDAR’97, Ulm, Germany, 1997, pp. 886-891. 7.   Huang, C.Y. Suen, “An Optimal Method of Combining Multiple Classifiers forUnconstrained Handwritten Numeral Recognition”, Proc. of IWFHR-3, Buffalo, NY,1993, pp. 11-20. 8.   E. Mandler and J. Schuermann, “Combining the Classification Results of independentclassifiers based on the Dempster/Shafer theory of evidence”, in  Pattern Recognitionand Artificial Intelligence, eds. E.S.Gelsema and L.N.Kanal, North Holland,Amsterdam, 1988, pp. 381-393. 9.   G. Dimauro, S. Impedovo, G. Pirlo, A. Salzo, “Automatic Bankchecks Processing : ANew Engineered System”, International Journal of Pattern Recognition and ArtificialIntelligence, Vol. 11, N.4, World Scientific Publ., Singapore, 1997, pp. 1-38. 10.   J.J. Hull, “A database for handwritten text recognition research", IEEE PAMI, Vol. 16,N. 5, 1994, pp. 550-554.558

   University of GroningenA PERTURBATION­BASED APPROACH FOR MULTI­CLASSIFIER SYSTEM DESIGNLecce Di, V.; Dimauro, G.; Guerriero, A.; Impedovo, S.; Pirlo, G.; Salzo, A.Published in:EPRINTS-BOOK-TITLEIMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite fromit. Please check the document version below.Document VersionPublisher's PDF, also known as Version of recordPublication date:2004Link to publication in University of Groningen/UMCG research databaseCitation for published version (APA):Lecce Di, V., Dimauro, G., Guerriero, A., Impedovo, S., Pirlo, G., & Salzo, A. (2004). APERTURBATION­BASED APPROACH FOR MULTI­CLASSIFIER SYSTEM DESIGN. In EPRINTS-BOOK-TITLE s.n..CopyrightOther than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of theauthor(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).The publication may also be distributed here under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license.More information can be found on the University of Groningen website: https://www.rug.nl/library/open-access/self-archiving-pure/taverne-amendment.Take-down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons thenumber of authors shown on this cover page is limited to 10 maximum.Download date: 09-09-2023A PERTURBATION-BASED APPROACH FOR MULTI-CLASSIFIERSYSTEM DESIGNV.DI LECCE1, G.DIMAURO2, A.GUERRIERO1, S.IMPEDOVO2, G.PIRLO2, A.SALZO2(1) Dipartimento di Ing. Elettronica - Politecnico di Bari-Via Re David - 70126 Bari - Italy(2) Dipartimento di Informatica - Università di Bari - Via Orabona, 4 - 70126 Bari –ItalyThis paper presents a perturbation-based approach useful to select the best combinationmethod for a multi-classifier system. The basic idea is to simulate small variations in theperformance of the set of classifiers and to evaluate to what extent they influence theperformance of the combined classifier. In the experimental phase, the BehaviouralKnowledge Space and the Dempster-Shafer combination methods have been considered. Theexperimental results, carried out in the field of  hand-written numeral recognition,demonstrate the  effectiveness of the new approach.1 IntroductionThe interest of researchers in multi -classifier systems is motivated by the consideration thatthey allow high performances also in the cases of difficult classification problems [1]. Sincetheoretical analysis of combination methods can be very difficult [2,3] the evaluation ofcomplex combination methods is generally carried out on experimental basis [4,5]. The netresult is that the design of multi -classifier systems still presents many unsolved issues [2,3].An important issue is related to the selection of the best method to combine theindividual classifier for a specific application. This aspect is generally evaluated bymeasuring the performance of  the methods by using data sets derived from the realapplication. In this case, there is no assurance that similar results can be obtained in theworking environment in which the performance of the individual classifiers can change [5].In this paper a perturbation-based approach is presented for the selection of the bestcombination method for a multi -classifier system. Small variations in the performance of theset of classifiers are simulated and their effect to the performance of the combined classifieris evaluated. Some indexes are also provided to obtain information on the performance of thecombined classifiers.The organisation of the paper is the following: Section 2 presents the problem ofselection of the combination method for multi -classifier system. The new perturbation-basedapproach is presented in Section 3. The experimental results are reported in Section 4.553In: L.R.B. Schomaker and L.G. Vuurpijl (Eds.), Proceedings of the Seventh International Workshop on Frontiers in Handwriting Recognition,September 11-13 2000, Amsterdam, ISBN 90-76942-01-3, Nijmegen: International Unipen Foundation, pp 553-5582 Multi-Classifier System DesignIn a multi -classifier system the final decision is obtained by combining the decisions ofseveral classifiers [1]. When a parallel-combination topology is considered [6], the inputpattern pt is provided to each classifier Ai, i=1,..,K which decides the membership of thepattern xt to the pattern classes ω1, ω2,..., ωm . Let Ai(t) be the response of the i-th classifierwhen the pattern xt is processed, the combination method M provides the final response bycombining the responses of the individual classifiers. From the outputs of the individualclassifiers, the method computes a confidence score S(ωi) for each class ωi, i=1,2,...,m.Successively a decision rule is used to produce the final classification response.The performance of a combination method is generally evaluated on experimental basisby considering measures as the recognition rate RM, the substitution rate EM and the rejectionrate LM, or other indexes and cost functions generally defined as a linear combination of RM ,EM and LM [1,2].Now, let be CM= LM + a⋅EM  the cost function considered for the evaluation of thecombination methods [6], the selection of the best method among M1 and M2 for a multi -classifier system follows the rules:   If CM1 < CM2 then use M1   If CM1 > CM2 then use M2   If CM1 = CM2 then use M1 or M2 equivalently.where CM1 and  CM2 are evaluated using a test data set derived from the real workingenvironment. Unfortunately, since the performance of the individual classifiers can change intime due to the learning capabilities of the classifiers or to the variabilit y in input data quality,there is no assurance that the combination method will provide acceptable results for thespecific application in the working environment. Therefore, it is important that theeffectiveness of methods for classifier combination M i is evaluated also when thecharacteristics of the set of individual classifiers change.3 A perturbation-based approach to combination method selectionLet A1,A2,…,AK be the individual classifiers combined by a combination method M. Letus consider the vector of responses obtained by inputting the patterns xt belonging to adatabase T. If T contains N patterns, the vector of responses consists of  N elements eachhaving K responses (one for each classifier) (A1(t), A2(t), …, AK(t)) ,  t=1,2,,N.From the multi -dimensional vector, two kinds of features are derived:   Features at the level of individual classifiers.The recognition rate of each classifier:RAi= ∑ttiR xtAQN)),((1where554î ==                                                                                 otherwise    0correct) is (t)A (i.e.   class  the tobelongs xand (t)A   if    1)),(( ijtjiωωtiR xtAQFeatures at the level of the set of classifiers.The correlation between classifiers [5]:∑⋅=<      1,...Kj=i,,21jijAiAAKρρwhere∑==NtjijAiA tAtAQN 1, ))(),((1ρρandî=          otherwise    0)(A=(t)A  if    1))(),((ji ttAtAQ jiρ .Therefore, the performance of the combination method M can be considered asdepending on the recognition rates of the individual classifiers R=(RA1,RA2,…,RAK) and onthe correlation ρ of the set of classifiersCE(R,ρ)= LE(R,ρ)+a⋅EE(R,ρ)                                 (1)where LM(R,ρ) and EM(R,ρ) denote the rejection rate and the substitution rate of thecombination method M as functions of R and ρ.The combination method M can be evaluated in different working conditions byperturbing the characteristics of the classifiers by mean of an automatic procedure. For thispurpose the vector of responses of the individual classifiers can be modified according to thefollowing considerations:•  Modification of features at the level of individual classifiers: For each classifier avariation is assumed in its recognition rate. For instance, for Ai whose recognition rate isoriginally of RAi , we have that the new recognition rate R*Ai  will  range in [RAi - δRAi ,RAi + δRAi], where δRAi is suitably defined.•  Modification of features at the level of the set of classifiers: A variation is assumed forthe correlation ρ of the entire set of classifiers. Specifically we have that the newcorrelation ρ* will range in [ρ-δρ, ρ+δρ]where δρ is suitably defined.For example, let us consider the vector of responses in Figure 1 for A1,A2,A3,A4 (N=10),where: R→Recognition; S1,S2,S3,S4→Substitution (rejections are not considered in thisexample). It results R=(0.7,0.8,0.7,0.6) (in fact: RA1=0.7, RA2=0.8, RA3=0.7, RA4=0.6), andρ=3.2/6. In this case, if we combine A1,A2,A3,A4 by the combination method M, we willachieve the value CM(R, ρ)= CM(0.7, 0.8, 0.7, 0.6, 3.2/6). In order to investigate thebehaviour of M, a perturbation can be performed on the vector of responses according to the555considerations (a) and (b). In Figure 2 a perturbed version of the vector of responses is shown(the modifications are marked with "^"). First, the output of A4 for pattern 3 has beenchanged (this modification changes the recognition rate of A4 and also the value of ρ for theentire set of classifiers). Second, the outputs of A2 for patterns 7 and 8 have been swapped(this modification does not change the recognition rate of A2 but only the value of ρ for theentire set of classifiers). It is easy to verify that, for the case in Figure 2 we haveR*=(0.7,0.8,0.7,0.7) (in fact: R*A1=0.6, R*A2=0.6, R*A3=0.6, R*A4=0.7), and ρ*=3.8/6.A1 A2 A3 A4Pattern 1 R R R RPattern 2 R R R S1Pattern 3 R R S3 S2Pattern 4 S1 S4 R S3Pattern 5 R R R RPattern 6 R R R RPattern 7 S3 R S2 RPattern 8 S2 R S1 S1Pattern 9 R S2 R RPattern 10 R R R RFigure 1: Vector of responses (0.7,0.8,0.7,0.6,3.2/6)A1 A2 A3 A4Pattern 1 R R R RPattern 2 R R R S1Pattern 3 R R S3 R(^)Pattern 4 S1 S4 R S3Pattern 5 R R R RPattern 6 R R R RPattern 7 S3 R(^) S2 RPattern 8 S2 S2(^) S1 S1Pattern 9 R R R RPattern 10 R R R RFigure 2: Vector of responses after perturbation (0.7,0.8,0.7,0.7,3.8/6).Modifications: A4(3)=R (Fig. 1: A4(3)=S2);A2(7)=R ; A2(8)=S2 (Fig. 1: A2(7)=S2 ; A2(8)=R).Now, let us consider the set of values assumed by the cost function for M when theentire set of  perturbed vector is considered M={ CM(R*, ρ*) |(R* , ρ*)∈  I*} where theperturbation range I* is defined as I* =[RA1-δRA1 ,RA1+δRA1]×[RA2-δRA2,RA2+δRA2]×..×[RAK-δRAK , RAK+δRAK]×[ρ-δρ,ρ+δρ].From the set M , useful information on the effectiveness of  M in different workingconditions can be derived:CM(R*, ρ*)CM(R, ρ)556 max(  M ): the maximum value in  M; min(  M ): the minimum value in  M; M(  M ): the mean value of  M; SD(  M ): the standard deviation of  M.These indexes allow a more accurate selection of the combination method for a multi -classifier system depending on the particular application field. Typical selection criteria canbe:  max(  M ) minimum (worst performance as good as possible); max(  M ) minimum (best performance as good as possible); M(  M ) minimum (best performance on average); SD(  M ) minimum (performance as stable as possible).4 Experimental ResultsThe proposed approach has been applied to a system for hand-written numeralrecognition that implements both the Behavioural Knowledge Space Method (BKS) [7] andthe Dempster-Shafer Method (DS) [8]. The system [9] combines four classifiers trained withabout 18468 numerals of the CEDAR Database (BR directory) [10]: Region (RA1=0.91),Contour (RA2=0.87), Loci (RA3=0.90), Histogram (RA4=0.87).Initially, from the vector of responses of the set of classifiers (tested with data of the BSdirectory of the CEDAR database) we derive the values R=(0.91,0.87,0.90,0.87) and ρ=0.84.From eq. (1), (with a=10), it results CBKS=0,485 and CDS =0,472. Hence DS outcomes BKS.Now, when the perturbation-based approach is applied to investigate the behaviour ofBKS(R*, ρ*) and DS(R*, ρ*) for (R*, ρ*)∈  I*, where δRAi=0.01 (a variation of 1% isconsidered for the recognition rate of the individual classifiers) and δρ=0.03, we obtain thesets  BKS and  DS as Table 1 shows.Table 1: Perturbation-based analysis of BKS and DS BKS  DSmax(  M ) 0.525 0.690min(  M ) 0.475 0.323M(  M ) 0.493 0.491SD(  M ) 0.019 0.156The result in Table 1 shows that, although DS and BKS have similar mean performance(M(  DS )≅ M(  BKS)), DS outperforms BKS in terms of best result ((min(  DS )<min(  BKS)),while BKS results more stable than DS ((SD(  DS )>SD(  BKS)) and it is also superior to DS interms of worst result ((max(  DS )>max(  BKS)).5575 ConclusionsThis paper presents a new perturbation-based approach for the selection of thecombination method best suited in a multi -classifier system for abstract-level classifiers. Theapproach provides useful information to evaluate the effectiveness of a combination methodin real environments, where modifications of  working  conditions can occur due to variationsin quality of input patterns or to learning capabili ties of classifiers.References 1.   L.Xu, A.Krzyzak, C.-Y Suen, “Methods of Combining Multiple Classifiers and TheirApplications to Handwriting Recognition” , IEEE SMC, Vol. 22, N.3, 1992, pp. 418-435. 2.   L. Lam, Y.S. Huang, C.Y. Suen, “Combination of Multiple Classifier Decisions forOptical Character Recognition” , in Handbook of Character Recognition and DocumentImage Analysis, Ed. H. Bunke and P.S.P. Wang, World Scientific, Singapore, 1997, pp.79-101. 3.   J. Kittler, M. Hatef, R.P.W. Duin, J. Matas, "On Combining Classifiers", IEEE PAMI,Vol. 20, n. 3, 1998, pp. 226-238. 4.   C. Nadal, R. Legault, and C.Y. Suen, “Complementary algorithms for the recognition oftotally un-constrained hand-written numerals” , Proc. 10th ICPR., Vol. A, June 1990, pp.434-449. 5.   G. Dimauro, S. Impedovo, G. Pirlo, “Multiple Experts: a new methodology for theevaluation of the combination processes” , Progress in Handwriting Recognition, ed. byA.C.Downton and S. Impedovo, World Scientific Publishing Co. Pte. Ltd., Singapore,1997, pp. 329-335. 6.   A.F.R. Rahman, M.C. Fairhurst, “ Introducing New Multiple Expert DecisionCombination Topologies: A Case Study using Recognition of Hand-written Characters” ,Proc. ICDAR’97, Ulm, Germany, 1997, pp. 886-891. 7.   Huang, C.Y. Suen, “An Optimal Method of Combining Multiple Classifiers forUnconstrained Handwritten Numeral Recognition” , Proc. of IWFHR-3, Buffalo, NY,1993, pp. 11-20. 8.   E. Mandler and J. Schuermann, “Combining the Classification Results of independentclassifiers based on the Dempster/Shafer theory of evidence”, in  Pattern Recognitionand Artifi cial Intell igence, eds. E.S.Gelsema and L.N.Kanal, North Holland,Amsterdam, 1988, pp. 381-393. 9.   G. Dimauro, S. Impedovo, G. Pirlo, A. Salzo, “Automatic Bankchecks Processing : ANew Engineered System”, International Journal of Pattern Recognition and Artifi cialIntelli gence, Vol. 11, N.4, World Scientific Publ., Singapore, 1997, pp. 1-38. 10.   J.J. Hull , “A database for handwritten text recognition research", IEEE PAMI, Vol. 16,N. 5, 1994, pp. 550-554.558

University of Groningen Digital Archive

A PERTURBATION-BASED APPROACH FOR MULTI-CLASSIFIER
SYSTEM DESIGN
V.DI LECCE
1, G.DIMAURO
2, A.GUERRIERO
1, S.IMPEDOVO
2, G.PIRLO
2, A.SALZO
2
(1) Dipartimento di Ing. Elettronica - Politecnico di Bari-
Via Re David - 70126 Bari - Italy
(2) Dipartimento di Informatica - Università di Bari -
 Via Orabona, 4 - 70126 Bari –Italy
This paper presents a perturbation-based approach useful to select the best combination
method for a multi-classifier system. The basic idea is to simulate small variations in the
performance of the set of classifiers and to evaluate to what extent they influence the
performance of the combined classifier. In the experimental phase, the Behavioural
Knowledge Space and the Dempster-Shafer combination methods have been considered. The
experimental results, carried out in the field of  hand-written numeral recognition,
demonstrate the  effectiveness of the new approach.
1 Introduction
The interest of researchers in multi-classifier systems is motivated by the consideration that
they allow high performances also in the cases of difficult classification problems [1]. Since
theoretical analysis of combination methods can be very difficult [2,3] the evaluation of
complex combination methods is generally carried out on experimental basis [4,5]. The net
result is that the design of multi-classifier systems still presents many unsolved issues [2,3].
An important issue is related to the selection of the best method to combine the
individual classifier for a specific application. This aspect is generally evaluated by
measuring the performance of  the methods by using data sets derived from the real
application. In this case, there is no assurance that similar results can be obtained in the
working environment in which the performance of the individual classifiers can change [5].
In this paper a perturbation-based approach is presented for the selection of the best
combination method for a multi-classifier system. Small variations in the performance of the
set of classifiers are simulated and their effect to the performance of the combined classifier
is evaluated. Some indexes are also provided to obtain information on the performance of the
combined classifiers.
The organisation of the paper is the following: Section 2 presents the problem of
selection of the combination method for multi-classifier system. The new perturbation-based
approach is presented in Section 3. The experimental results are reported in Section 4.
553
In: L.R.B. Schomaker and L.G. Vuurpijl (Eds.), Proceedings of the Seventh International Workshop on Frontiers in Handwriting Recognition,
September 11-13 2000, Amsterdam, ISBN 90-76942-01-3, Nijmegen: International Unipen Foundation, pp 553-5582 Multi-Classifier System Design
In a multi-classifier system the final decision is obtained by combining the decisions of
several classifiers [1]. When a parallel-combination topology is considered [6], the input
pattern pt is provided to each classifier Ai, i=1,..,K which decides the membership of the
pattern xt to the pattern classes w1, w2,..., wm . Let Ai(t) be the response of the i-th classifier
when the pattern xt is processed, the combination method M provides the final response by
combining the responses of the individual classifiers. From the outputs of the individual
classifiers, the method computes a confidence score S(wi) for each class wi, i=1,2,...,m.
Successively a decision rule is used to produce the final classification response.
The performance of a combination method is generally evaluated on experimental basis
by considering measures as the recognition rate RM, the substitution rate EM and the rejection
rate LM, or other indexes and cost functions generally defined as a linear combination of RM ,
EM and LM [1,2].
Now, let be CM= LM + a×EM  the cost function considered for the evaluation of the
combination methods [6], the selection of the best method among M1 and M2 for a multi-
classifier system follows the rules:
￿  If CM1 < CM2 then use M1
￿  If CM1 > CM2 then use M2
￿  If CM1 = CM2 then use M1 or M2 equivalently.
where CM1 and  CM2 are evaluated using a test data set derived from the real working
environment. Unfortunately, since the performance of the individual classifiers can change in
time due to the learning capabilities of the classifiers or to the variability in input data quality,
there is no assurance that the combination method will provide acceptable results for the
specific application in the working environment. Therefore, it is important that the
effectiveness of methods for classifier combination Mi is evaluated also when the
characteristics of the set of individual classifiers change.
3 A perturbation-based approach to combination method selection
Let A1,A2,…,AK be the individual classifiers combined by a combination method M. Let
us consider the vector of responses obtained by inputting the patterns xt belonging to a
database T. If T contains N patterns, the vector of responses consists of  N elements each
having K responses (one for each classifier) (A1(t), A2(t), …, AK(t)) ,  t=1,2,,N.
From the multi-dimensional vector, two kinds of features are derived:
￿  Features at the level of individual classifiers.
The recognition rate of each classifier:
RAi=  å
t
t i R x t A Q
N
) ), ( (
1
where
554î
í
ì =
=
                                                                                                  otherwise      0
correct)   is   (t) A   (i.e.      class    the  to belongs  x and   (t) A     if      1
) ), ( (
i j t j i w w
t i R x t A Q
Features at the level of the set of classifiers.
The correlation between classifiers [5]:
ú
ú
ú
ú
û
ù
ê
ê
ê
ê
ë
é
å ×
ú
ú
û
ù
ê
ê
ë
é
÷ ÷
ø
ö
ç ç
è
æ
=
<       
1,...K j= i,
, 2
1
j i
j A i A A
K
r r
where
å =
=
N
t
j i j A i A t A t A Q
N 1
, )) ( ), ( (
1
r r
and
ï î
ï
í
ì
=
           otherwise      0
) ( A = (t) A    if      1
)) ( ), ( (
j i t
t A t A Q j i r .
Therefore, the performance of the combination method M can be considered as
depending on the recognition rates of the individual classifiers R=(RA1,RA2,…,RAK) and on
the correlation r of the set of classifiers
CE(R,r)= LE(R,r)+a×EE(R,r)                                  (1)
where LM(R,r) and EM(R,r) denote the rejection rate and the substitution rate of the
combination method M as functions of R and r.
The combination method M can be evaluated in different working conditions by
perturbing the characteristics of the classifiers by mean of an automatic procedure. For this
purpose the vector of responses of the individual classifiers can be modified according to the
following considerations:
· Modification of features at the level of individual classifiers: For each classifier a
variation is assumed in its recognition rate. For instance, for Ai whose recognition rate is
originally of RAi , we have that the new recognition rate R
*
Ai  will  range in [RAi - dRAi ,
RAi + dRAi], where dRAi is suitably defined.
· Modification of features at the level of the set of classifiers: A variation is assumed for
the correlation r of the entire set of classifiers. Specifically we have that the new
correlation r
* will range in [r-dr, r+dr]where dr is suitably defined.
For example, let us consider the vector of responses in Figure 1 for A1,A2,A3,A4 (N=10),
where: R®Recognition; S1,S2,S3,S4®Substitution (rejections are not considered in this
example). It results R=(0.7,0.8,0.7,0.6) (in fact: RA1=0.7, RA2=0.8, RA3=0.7, RA4=0.6), and
r=3.2/6. In this case, if we combine A1,A2,A3,A4 by the combination method M, we will
achieve the value CM(R, r)= CM(0.7, 0.8, 0.7, 0.6, 3.2/6). In order to investigate the
behaviour of M, a perturbation can be performed on the vector of responses according to the
555considerations (a) and (b). In Figure 2 a perturbed version of the vector of responses is shown
(the modifications are marked with "^"). First, the output of A4 for pattern 3 has been
changed (this modification changes the recognition rate of A4 and also the value of r for the
entire set of classifiers). Second, the outputs of A2 for patterns 7 and 8 have been swapped
(this modification does not change the recognition rate of A2 but only the value of r for the
entire set of classifiers). It is easy to verify that, for the case in Figure 2 we have
R
*=(0.7,0.8,0.7,0.7) (in fact: R
*
A1=0.6, R
*
A2=0.6, R
*
A3=0.6, R
*
A4=0.7), and r
*=3.8/6.
A1 A2 A3 A4
Pattern 1 R R R R
Pattern 2 R R R S1
Pattern 3 R R S3 S2
Pattern 4 S1 S4 R S3
Pattern 5 R R R R
Pattern 6 R R R R
Pattern 7 S3 R S2 R
Pattern 8 S2 R S1 S1
Pattern 9 R S2 R R
Pattern 10 R R R R
Figure 1: Vector of responses (0.7,0.8,0.7,0.6,3.2/6)
A1 A2 A3 A4
Pattern 1 R R R R
Pattern 2 R R R S1
Pattern 3 R R S3 R(^)
Pattern 4 S1 S4 R S3
Pattern 5 R R R R
Pattern 6 R R R R
Pattern 7 S3 R(^) S2 R
Pattern 8 S2 S2(^) S1 S1
Pattern 9 R R R R
Pattern 10 R R R R
Figure 2: Vector of responses after perturbation (0.7,0.8,0.7,0.7,3.8/6).
Modifications: A4(3)=R (Fig. 1: A4(3)=S2);A2(7)=R ; A2(8)=S2 (Fig. 1: A2(7)=S2 ; A2(8)=R).
Now, let us consider the set of values assumed by the cost function for M when the
entire set of  perturbed vector is considered 
￿
M={CM(R
*, r
*) |(R
* , r
*)Î I
*} where the
perturbation range I
* is defined as I
* =[RA1-dRA1 ,RA1+dRA1]´[RA2-dRA2,RA2+dRA2]´..´[RAK-
dRAK , RAK+dRAK]´[r-dr,r+dr].
From the set 
￿
M , useful information on the effectiveness of  M in different working
conditions can be derived:
CM(R*, r r*)
CM(R, r r)
556￿
 max(
￿ M
 ): the maximum value in 
￿ M;
￿
 min(
￿ M
 ): the minimum value in 
￿ M;
￿
 M(
￿ M
 ): the mean value of 
￿ M;
￿
 SD(
￿ M
 ): the standard deviation of 
￿ M.
These indexes allow a more accurate selection of the combination method for a multi-
classifier system depending on the particular application field. Typical selection criteria can
be:
￿
 max(
￿ M
 ) minimum (worst performance as good as possible);
￿
 max(
￿ M
 ) minimum (best performance as good as possible);
￿
 M(
￿ M
 ) minimum (best performance on average);
￿
 SD(
￿ M
 ) minimum (performance as stable as possible).
4 Experimental Results
The proposed approach has been applied to a system for hand-written numeral
recognition that implements both the Behavioural Knowledge Space Method (BKS) [7] and
the Dempster-Shafer Method (DS) [8]. The system [9] combines four classifiers trained with
about 18468 numerals of the CEDAR Database (BR directory) [10]: Region (RA1=0.91),
Contour (RA2=0.87), Loci (RA3=0.90), Histogram (RA4=0.87).
Initially, from the vector of responses of the set of classifiers (tested with data of the BS
directory of the CEDAR database) we derive the values R=(0.91,0.87,0.90,0.87) and r=0.84.
From eq. (1), (with a=10), it results CBKS=0,485 and CDS =0,472. Hence DS outcomes BKS.
Now, when the perturbation-based approach is applied to investigate the behaviour of
BKS(R*, r*) and DS(R*, r*) for (R*, r*)Î I
*, where dRAi=0.01 (a variation of 1% is
considered for the recognition rate of the individual classifiers) and dr=0.03, we obtain the
sets 
￿ BKS and 
￿ DS as Table 1 shows.
Table 1: Perturbation-based analysis of BKS and DS
￿ BKS
￿ DS
max(
￿ M
 ) 0.525 0.690
min(
￿ M
 ) 0.475 0.323
M(
￿ M
 ) 0.493 0.491
SD(
￿ M
 ) 0.019 0.156
The result in Table 1 shows that, although DS and BKS have similar mean performance
(M(
￿ DS
 )@M(
￿ BKS)), DS outperforms BKS in terms of best result ((min(
￿ DS
 )<min(
￿ BKS)),
while BKS results more stable than DS ((SD(
￿ DS
 )>SD(
￿ BKS)) and it is also superior to DS in
terms of worst result ((max(
￿ DS
 )>max(
￿ BKS)).
5575 Conclusions
This paper presents a new perturbation-based approach for the selection of the
combination method best suited in a multi-classifier system for abstract-level classifiers. The
approach provides useful information to evaluate the effectiveness of a combination method
in real environments, where modifications of  working  conditions can occur due to variations
in quality of input patterns or to learning capabilities of classifiers.
References
 1.   L.Xu, A.Krzyzak, C.-Y Suen, “Methods of Combining Multiple Classifiers and Their
Applications to Handwriting Recognition”, IEEE SMC, Vol. 22, N.3, 1992, pp. 418-435.
 2.   L. Lam, Y.S. Huang, C.Y. Suen, “Combination of Multiple Classifier Decisions for
Optical Character Recognition”, in Handbook of Character Recognition and Document
Image Analysis, Ed. H. Bunke and P.S.P. Wang, World Scientific, Singapore, 1997, pp.
79-101.
 3.   J. Kittler, M. Hatef, R.P.W. Duin, J. Matas, "On Combining Classifiers", IEEE PAMI,
Vol. 20, n. 3, 1998, pp. 226-238.
 4.   C. Nadal, R. Legault, and C.Y. Suen, “Complementary algorithms for the recognition of
totally un-constrained hand-written numerals”, Proc. 10
th ICPR., Vol. A, June 1990, pp.
434-449.
 5.   G. Dimauro, S. Impedovo, G. Pirlo, “Multiple Experts: a new methodology for the
evaluation of the combination processes”, Progress in Handwriting Recognition, ed. by
A.C.Downton and S. Impedovo, World Scientific Publishing Co. Pte. Ltd., Singapore,
1997, pp. 329-335.
 6.   A.F.R. Rahman, M.C. Fairhurst, “Introducing New Multiple Expert Decision
Combination Topologies: A Case Study using Recognition of Hand-written Characters”,
Proc. ICDAR’97, Ulm, Germany, 1997, pp. 886-891.
 7.   Huang, C.Y. Suen, “An Optimal Method of Combining Multiple Classifiers for
Unconstrained Handwritten Numeral Recognition”, Proc. of IWFHR-3, Buffalo, NY,
1993, pp. 11-20.
 8.   E. Mandler and J. Schuermann, “Combining the Classification Results of independent
classifiers based on the Dempster/Shafer theory of evidence”, in  Pattern Recognition
and Artificial Intelligence, eds. E.S.Gelsema and L.N.Kanal, North Holland,
Amsterdam, 1988, pp. 381-393.
 9.   G. Dimauro, S. Impedovo, G. Pirlo, A. Salzo, “Automatic Bankchecks Processing : A
New Engineered System”, International Journal of Pattern Recognition and Artificial
Intelligence, Vol. 11, N.4, World Scientific Publ., Singapore, 1997, pp. 1-38.
 10.   J.J. Hull, “A database for handwritten text recognition research", IEEE PAMI, Vol. 16,
N. 5, 1994, pp. 550-554.
558

A database for handwritten text recognition research&quot;,

An Optimal Method of Combining Multiple Classifiers for Unconstrained Handwritten Numeral Recognition”,

Automatic Bankchecks Processing : A New Engineered System”,

Combination of Multiple Classifier Decisions for Optical Character Recognition”,

Combining the Classification Results of independent classifiers based on the Dempster/Shafer theory of evidence”,

Complementary algorithms for the recognition of totally un-constrained hand-written numerals”,

Introducing New Multiple Expert Decision Combination Topologies: A Case Study using Recognition of Hand-written Characters”,

Methods of Combining Multiple Classifiers and Their Applications to Handwriting Recognition”,

Multiple Experts: a new methodology for the evaluation of the combination processes”,

On Combining Classifiers&quot;,

   University of GroningenA PERTURBATION­BASED APPROACH FOR MULTI­CLASSIFIER SYSTEM DESIGNLecce Di, V.; Dimauro, G.; Guerriero, A.; Impedovo, S.; Pirlo, G.; Salzo, A.Published in:EPRINTS-BOOK-TITLEIMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite fromit. Please check the document version below.Document VersionPublisher's PDF, also known as Version of recordPublication date:2004Link to publication in University of Groningen/UMCG research databaseCitation for published version (APA):Lecce Di, V., Dimauro, G., Guerriero, A., Impedovo, S., Pirlo, G., & Salzo, A. (2004). APERTURBATION­BASED APPROACH FOR MULTI­CLASSIFIER SYSTEM DESIGN. In EPRINTS-BOOK-TITLE s.n..CopyrightOther than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of theauthor(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).The publication may also be distributed here under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license.More information can be found on the University of Groningen website: https://www.rug.nl/library/open-access/self-archiving-pure/taverne-amendment.Take-down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons thenumber of authors shown on this cover page is limited to 10 maximum.Download date: 05-12-2023A PERTURBATION-BASED APPROACH FOR MULTI-CLASSIFIERSYSTEM DESIGNV.DI LECCE1, G.DIMAURO2, A.GUERRIERO1, S.IMPEDOVO2, G.PIRLO2, A.SALZO2(1) Dipartimento di Ing. Elettronica - Politecnico di Bari-Via Re David - 70126 Bari - Italy(2) Dipartimento di Informatica - Università di Bari - Via Orabona, 4 - 70126 Bari –ItalyThis paper presents a perturbation-based approach useful to select the best combinationmethod for a multi-classifier system. The basic idea is to simulate small variations in theperformance of the set of classifiers and to evaluate to what extent they influence theperformance of the combined classifier. In the experimental phase, the BehaviouralKnowledge Space and the Dempster-Shafer combination methods have been considered. Theexperimental results, carried out in the field of  hand-written numeral recognition,demonstrate the  effectiveness of the new approach.1 IntroductionThe interest of researchers in multi -classifier systems is motivated by the consideration thatthey allow high performances also in the cases of difficult classification problems [1]. Sincetheoretical analysis of combination methods can be very difficult [2,3] the evaluation ofcomplex combination methods is generally carried out on experimental basis [4,5]. The netresult is that the design of multi -classifier systems still presents many unsolved issues [2,3].An important issue is related to the selection of the best method to combine theindividual classifier for a specific application. This aspect is generally evaluated bymeasuring the performance of  the methods by using data sets derived from the realapplication. In this case, there is no assurance that similar results can be obtained in theworking environment in which the performance of the individual classifiers can change [5].In this paper a perturbation-based approach is presented for the selection of the bestcombination method for a multi -classifier system. Small variations in the performance of theset of classifiers are simulated and their effect to the performance of the combined classifieris evaluated. Some indexes are also provided to obtain information on the performance of thecombined classifiers.The organisation of the paper is the following: Section 2 presents the problem ofselection of the combination method for multi -classifier system. The new perturbation-basedapproach is presented in Section 3. The experimental results are reported in Section 4.553In: L.R.B. Schomaker and L.G. Vuurpijl (Eds.), Proceedings of the Seventh International Workshop on Frontiers in Handwriting Recognition,September 11-13 2000, Amsterdam, ISBN 90-76942-01-3, Nijmegen: International Unipen Foundation, pp 553-5582 Multi-Classifier System DesignIn a multi -classifier system the final decision is obtained by combining the decisions ofseveral classifiers [1]. When a parallel-combination topology is considered [6], the inputpattern pt is provided to each classifier Ai, i=1,..,K which decides the membership of thepattern xt to the pattern classes ω1, ω2,..., ωm . Let Ai(t) be the response of the i-th classifierwhen the pattern xt is processed, the combination method M provides the final response bycombining the responses of the individual classifiers. From the outputs of the individualclassifiers, the method computes a confidence score S(ωi) for each class ωi, i=1,2,...,m.Successively a decision rule is used to produce the final classification response.The performance of a combination method is generally evaluated on experimental basisby considering measures as the recognition rate RM, the substitution rate EM and the rejectionrate LM, or other indexes and cost functions generally defined as a linear combination of RM ,EM and LM [1,2].Now, let be CM= LM + a⋅EM  the cost function considered for the evaluation of thecombination methods [6], the selection of the best method among M1 and M2 for a multi -classifier system follows the rules:   If CM1 < CM2 then use M1   If CM1 > CM2 then use M2   If CM1 = CM2 then use M1 or M2 equivalently.where CM1 and  CM2 are evaluated using a test data set derived from the real workingenvironment. Unfortunately, since the performance of the individual classifiers can change intime due to the learning capabilities of the classifiers or to the variabilit y in input data quality,there is no assurance that the combination method will provide acceptable results for thespecific application in the working environment. Therefore, it is important that theeffectiveness of methods for classifier combination M i is evaluated also when thecharacteristics of the set of individual classifiers change.3 A perturbation-based approach to combination method selectionLet A1,A2,…,AK be the individual classifiers combined by a combination method M. Letus consider the vector of responses obtained by inputting the patterns xt belonging to adatabase T. If T contains N patterns, the vector of responses consists of  N elements eachhaving K responses (one for each classifier) (A1(t), A2(t), …, AK(t)) ,  t=1,2,,N.From the multi -dimensional vector, two kinds of features are derived:   Features at the level of individual classifiers.The recognition rate of each classifier:RAi= ∑ttiR xtAQN)),((1where554î ==                                                                                 otherwise    0correct) is (t)A (i.e.   class  the tobelongs xand (t)A   if    1)),(( ijtjiωωtiR xtAQFeatures at the level of the set of classifiers.The correlation between classifiers [5]:∑⋅=<      1,...Kj=i,,21jijAiAAKρρwhere∑==NtjijAiA tAtAQN 1, ))(),((1ρρandî=          otherwise    0)(A=(t)A  if    1))(),((ji ttAtAQ jiρ .Therefore, the performance of the combination method M can be considered asdepending on the recognition rates of the individual classifiers R=(RA1,RA2,…,RAK) and onthe correlation ρ of the set of classifiersCE(R,ρ)= LE(R,ρ)+a⋅EE(R,ρ)                                 (1)where LM(R,ρ) and EM(R,ρ) denote the rejection rate and the substitution rate of thecombination method M as functions of R and ρ.The combination method M can be evaluated in different working conditions byperturbing the characteristics of the classifiers by mean of an automatic procedure. For thispurpose the vector of responses of the individual classifiers can be modified according to thefollowing considerations:•  Modification of features at the level of individual classifiers: For each classifier avariation is assumed in its recognition rate. For instance, for Ai whose recognition rate isoriginally of RAi , we have that the new recognition rate R*Ai  will  range in [RAi - δRAi ,RAi + δRAi], where δRAi is suitably defined.•  Modification of features at the level of the set of classifiers: A variation is assumed forthe correlation ρ of the entire set of classifiers. Specifically we have that the newcorrelation ρ* will range in [ρ-δρ, ρ+δρ]where δρ is suitably defined.For example, let us consider the vector of responses in Figure 1 for A1,A2,A3,A4 (N=10),where: R→Recognition; S1,S2,S3,S4→Substitution (rejections are not considered in thisexample). It results R=(0.7,0.8,0.7,0.6) (in fact: RA1=0.7, RA2=0.8, RA3=0.7, RA4=0.6), andρ=3.2/6. In this case, if we combine A1,A2,A3,A4 by the combination method M, we willachieve the value CM(R, ρ)= CM(0.7, 0.8, 0.7, 0.6, 3.2/6). In order to investigate thebehaviour of M, a perturbation can be performed on the vector of responses according to the555considerations (a) and (b). In Figure 2 a perturbed version of the vector of responses is shown(the modifications are marked with "^"). First, the output of A4 for pattern 3 has beenchanged (this modification changes the recognition rate of A4 and also the value of ρ for theentire set of classifiers). Second, the outputs of A2 for patterns 7 and 8 have been swapped(this modification does not change the recognition rate of A2 but only the value of ρ for theentire set of classifiers). It is easy to verify that, for the case in Figure 2 we haveR*=(0.7,0.8,0.7,0.7) (in fact: R*A1=0.6, R*A2=0.6, R*A3=0.6, R*A4=0.7), and ρ*=3.8/6.A1 A2 A3 A4Pattern 1 R R R RPattern 2 R R R S1Pattern 3 R R S3 S2Pattern 4 S1 S4 R S3Pattern 5 R R R RPattern 6 R R R RPattern 7 S3 R S2 RPattern 8 S2 R S1 S1Pattern 9 R S2 R RPattern 10 R R R RFigure 1: Vector of responses (0.7,0.8,0.7,0.6,3.2/6)A1 A2 A3 A4Pattern 1 R R R RPattern 2 R R R S1Pattern 3 R R S3 R(^)Pattern 4 S1 S4 R S3Pattern 5 R R R RPattern 6 R R R RPattern 7 S3 R(^) S2 RPattern 8 S2 S2(^) S1 S1Pattern 9 R R R RPattern 10 R R R RFigure 2: Vector of responses after perturbation (0.7,0.8,0.7,0.7,3.8/6).Modifications: A4(3)=R (Fig. 1: A4(3)=S2);A2(7)=R ; A2(8)=S2 (Fig. 1: A2(7)=S2 ; A2(8)=R).Now, let us consider the set of values assumed by the cost function for M when theentire set of  perturbed vector is considered M={ CM(R*, ρ*) |(R* , ρ*)∈  I*} where theperturbation range I* is defined as I* =[RA1-δRA1 ,RA1+δRA1]×[RA2-δRA2,RA2+δRA2]×..×[RAK-δRAK , RAK+δRAK]×[ρ-δρ,ρ+δρ].From the set M , useful information on the effectiveness of  M in different workingconditions can be derived:CM(R*, ρ*)CM(R, ρ)556 max(  M ): the maximum value in  M; min(  M ): the minimum value in  M; M(  M ): the mean value of  M; SD(  M ): the standard deviation of  M.These indexes allow a more accurate selection of the combination method for a multi -classifier system depending on the particular application field. Typical selection criteria canbe:  max(  M ) minimum (worst performance as good as possible); max(  M ) minimum (best performance as good as possible); M(  M ) minimum (best performance on average); SD(  M ) minimum (performance as stable as possible).4 Experimental ResultsThe proposed approach has been applied to a system for hand-written numeralrecognition that implements both the Behavioural Knowledge Space Method (BKS) [7] andthe Dempster-Shafer Method (DS) [8]. The system [9] combines four classifiers trained withabout 18468 numerals of the CEDAR Database (BR directory) [10]: Region (RA1=0.91),Contour (RA2=0.87), Loci (RA3=0.90), Histogram (RA4=0.87).Initially, from the vector of responses of the set of classifiers (tested with data of the BSdirectory of the CEDAR database) we derive the values R=(0.91,0.87,0.90,0.87) and ρ=0.84.From eq. (1), (with a=10), it results CBKS=0,485 and CDS =0,472. Hence DS outcomes BKS.Now, when the perturbation-based approach is applied to investigate the behaviour ofBKS(R*, ρ*) and DS(R*, ρ*) for (R*, ρ*)∈  I*, where δRAi=0.01 (a variation of 1% isconsidered for the recognition rate of the individual classifiers) and δρ=0.03, we obtain thesets  BKS and  DS as Table 1 shows.Table 1: Perturbation-based analysis of BKS and DS BKS  DSmax(  M ) 0.525 0.690min(  M ) 0.475 0.323M(  M ) 0.493 0.491SD(  M ) 0.019 0.156The result in Table 1 shows that, although DS and BKS have similar mean performance(M(  DS )≅ M(  BKS)), DS outperforms BKS in terms of best result ((min(  DS )<min(  BKS)),while BKS results more stable than DS ((SD(  DS )>SD(  BKS)) and it is also superior to DS interms of worst result ((max(  DS )>max(  BKS)).5575 ConclusionsThis paper presents a new perturbation-based approach for the selection of thecombination method best suited in a multi -classifier system for abstract-level classifiers. Theapproach provides useful information to evaluate the effectiveness of a combination methodin real environments, where modifications of  working  conditions can occur due to variationsin quality of input patterns or to learning capabili ties of classifiers.References 1.   L.Xu, A.Krzyzak, C.-Y Suen, “Methods of Combining Multiple Classifiers and TheirApplications to Handwriting Recognition” , IEEE SMC, Vol. 22, N.3, 1992, pp. 418-435. 2.   L. Lam, Y.S. Huang, C.Y. Suen, “Combination of Multiple Classifier Decisions forOptical Character Recognition” , in Handbook of Character Recognition and DocumentImage Analysis, Ed. H. Bunke and P.S.P. Wang, World Scientific, Singapore, 1997, pp.79-101. 3.   J. Kittler, M. Hatef, R.P.W. Duin, J. Matas, "On Combining Classifiers", IEEE PAMI,Vol. 20, n. 3, 1998, pp. 226-238. 4.   C. Nadal, R. Legault, and C.Y. Suen, “Complementary algorithms for the recognition oftotally un-constrained hand-written numerals” , Proc. 10th ICPR., Vol. A, June 1990, pp.434-449. 5.   G. Dimauro, S. Impedovo, G. Pirlo, “Multiple Experts: a new methodology for theevaluation of the combination processes” , Progress in Handwriting Recognition, ed. byA.C.Downton and S. Impedovo, World Scientific Publishing Co. Pte. Ltd., Singapore,1997, pp. 329-335. 6.   A.F.R. Rahman, M.C. Fairhurst, “ Introducing New Multiple Expert DecisionCombination Topologies: A Case Study using Recognition of Hand-written Characters” ,Proc. ICDAR’97, Ulm, Germany, 1997, pp. 886-891. 7.   Huang, C.Y. Suen, “An Optimal Method of Combining Multiple Classifiers forUnconstrained Handwritten Numeral Recognition” , Proc. of IWFHR-3, Buffalo, NY,1993, pp. 11-20. 8.   E. Mandler and J. Schuermann, “Combining the Classification Results of independentclassifiers based on the Dempster/Shafer theory of evidence”, in  Pattern Recognitionand Artifi cial Intell igence, eds. E.S.Gelsema and L.N.Kanal, North Holland,Amsterdam, 1988, pp. 381-393. 9.   G. Dimauro, S. Impedovo, G. Pirlo, A. Salzo, “Automatic Bankchecks Processing : ANew Engineered System”, International Journal of Pattern Recognition and Artifi cialIntelli gence, Vol. 11, N.4, World Scientific Publ., Singapore, 1997, pp. 1-38. 10.   J.J. Hull , “A database for handwritten text recognition research", IEEE PAMI, Vol. 16,N. 5, 1994, pp. 550-554.558

https://pure.rug.nl/ws/files/2977234/poster-029-Impedovo.pdf

A PERTURBATIONBASED APPROACH FOR MULTICLASSIFIER SYSTEM DESIGN

Abstract

Similar works

Full text

Available Versions

University of Groningen Research Database

Proceedings - University of Groningen

Proceedings - University of Groningen

Dissertations of the University of Groningen

ARTS repository - University of Groningen

University of Groningen

Dissertations of the University of Groningen

University of Groningen Digital Archive

ARTS repository - University of Groningen

A PERTURBATION­BASED APPROACH FOR MULTI­CLASSIFIER SYSTEM DESIGN

Abstract

Similar works

Full text

Available Versions

University of Groningen Research Database

Proceedings - University of Groningen

Proceedings - University of Groningen

Dissertations of the University of Groningen

ARTS repository - University of Groningen

University of Groningen

Dissertations of the University of Groningen

University of Groningen Digital Archive

ARTS repository - University of Groningen

A PERTURBATIONBASED APPROACH FOR MULTICLASSIFIER SYSTEM DESIGN