We compare in this paper several feature selection methods for the Naive
Bayes Classifier (NBC) when the data under study are described by a large
number of redundant binary indicators. Wrapper approaches guided by the NBC
estimation of the classification error probability out-perform filter
approaches while retaining a reasonable computational cost

Cottrell, Marie

Lacaille, Jérôme

Rabenoro, Tsirizo

Rossi, Fabrice

English

arXiv

International audienceWe compare in this paper several feature selection methods for the Naive Bayes Classifier (NBC) when the data under study are described by a large number of redundant binary indicators. Wrapper approaches guided by the NBC estimation of the classification error probability out-perform filter approaches while retaining a reasonable computational cost

Archive Ouverte en Sciences de l'Information et de la Communication

HAL Id: hal-01162981
https://hal.archives-ouvertes.fr/hal-01162981
Submitted on 11 Jun 2015
HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d’enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
Distributed under a Creative Commons Attribution| 4.0 International License
Search Strategies for Binary Feature Selection for a
Naive Bayes Classifier
Tsirizo Rabenoro, Jérôme Lacaille, Marie Cottrell, Fabrice Rossi
To cite this version:
Tsirizo Rabenoro, Jérôme Lacaille, Marie Cottrell, Fabrice Rossi. Search Strategies for Binary Feature
Selection for a Naive Bayes Classifier. European Symposium on Artificial Neural Networks, Com-
putational Intelligence and Machine Learning (ESANN), Apr 2015, Bruges, Belgium. pp.291-296.
￿hal-01162981￿
Search Strategies for Binary Feature Selection
for a Naive Bayes Classifier
Tsirizo Rabenoro1,2, Jérôme Lacaille1 , Marie Cottrell2, and Fabrice Rossi2 ∗
1- SAMM EA 4543, Université Paris 1 Panthéon-Sorbonne
90, rue de Tolbiac, 75634 Paris cedex 13, France
2- Snecma, Groupe Safran,77550 Moissy Cramayel, France
Abstract. We compare in this paper several feature selection methods for
the Naive Bayes Classifier (NBC) when the data under study are described
by a large number of redundant binary indicators. Wrapper approaches
guided by the NBC estimation of the classification error probability out-
perform filter approaches while retaining a reasonable computational cost.
1 Introduction
We consider in this paper application contexts in which a large body of expert
knowledge is available as a series of simple and low level parametric scores.
The goal is to build an interpretable classifier from this knowledge (and from a
learning set). Then the scores are assumed to be simple parametric functions
from the data space to R with the interpretation that a high value of a score
indicates that the datum submitted to the score belongs probably to the class
that the score has been designed to detect.
Let’s consider a concrete example from our main application domain, aircraft
engine monitoring (see [7] for details). We aim here at classifying some short time
series (around 150 time points, each series having its own specific length) into
different classes (normal signal and different types of anomalies corresponding
to some non stationarity in the signal). Domain experts have selected a set of
statistical tests as scores. For instance, the Mann-Whitney U test can be used to
reject the null hypothesis that two populations are identical. It can be applied
to a time series by selecting a potential break point in the series tb and a window
size w, and by considering the w/2 points before tb as the first population and
the w/2 points after tb as the second population. The score is the p-value of
the U test applied to those populations: a high value leads to not rejecting the
null hypothesis and thus is an indication that the time series belongs to “no
anomaly” class. Notice that the parameters of the score are here the potential
break point tb and the window size w. See [7] for other examples.
While the experts can design scores, they are seldom able to provide more
than hints about the parameters and the thresholds (i.e. when to consider that
the score is “high enough”). In addition, the scores are generally no sufficient
alone and several of them should be combined to achieve acceptable classification
rates. We proposed in [7] to address this problem via feature selection using the
filter mRMR approach [5]. The main idea, recalled in Section 2, consists in
∗T. Rabenoro is supported by a grant from Snecma, Safran Group.
turning the scores into a large set of redundant binary features and in using a
feature selection method to keep useful ones. This has the effect of finding good
parameters and thresholds for the scores while retaining a reasonable number
of scores, easing interpretation of the decision made by a Naive Bayes Classifier
(NBC) built on them. In [7], the selection is done by a filter method. We
investigate in the present paper wrapper based approaches.
2 From scores to binary indicators
We assume given a training set (Xi, Yi)1≤i≤N where the observations space X
can be arbitrary while the target space is a finite set of classes, Y = {1, . . . ,K}.
We are also given a set of Q parametric scores, (sq)1≤q≤Q. Each sq is a function
from X ×Wq to R, where Wq is the parameter space of the score.
The main constraint of our context is that experts only allow score results
to be used by the classifier. In addition, the semantic of the scores means that
only decisions of the form sq(Xi, wq) ≤ λq are really meaningful. We propose to
transform this set of scores into a much larger set of binary indicators. This is
done by choosing for each score a finite subset of Wq, {w
1
q , . . . , w
pq
q } and a finite
set of thresholds {λ1q, . . . , λ
tq
q }, and by defining pq × tq indicator functions by
I
p,t
q (X) = Isq(X,wpq )≤λtq . This can be seen as a form of grid search in the “score
space” in the sense that tuning the scores to the data set can be done indirectly
by selecting relevant binary indicators (a similar principle is used in e.g. [1]).
By feeding the training set (Xi, Yi)1≤i≤N through the indicators, we obtain a
new training set (Bi, Yi)1≤i≤N where the Bi take values into {0, 1}
P where P is
the total number of binary indicators generated from the scores.
Contrarily to arbitrary binary valued variables, those indicators are intrinsi-
cally highly redundant and correlated. For instance, if λtq < λ
t′
q , then I
p,t
q (X) = 1
implies Ip,t
′
q (X) = 1. This has adverse consequences on feature selection methods
and on the Naive Bayes Classifier.
3 Naive Bayes Classifier
The Naive Bayes Classifier (NBC, this e.g. [4]) is a very simple and robust
classifier based on the (unrealistic) assumption that the features used to describe
the objects to classify are conditionally independent given the class. In our
context, this translates into P (B = b|Y = k) =
∏P
p=1 P (B
p = bp|Y = k), where
Bp is the p-th indicator value in the indicator vector. This allows to estimate
easily the posterior probability P (Y = k|B = b) as
P (Y = k|B = b) =
P (Y = k)
∏P
p=1 P (B
p = bp|Y = k)
P (Bp = b)
,
using estimated values of the P (Bp = bp|Y = k). Those values are obtained by
simple class conditional counts.
In our context, the motivation for using the NBC is twofold. Its classical
properties (simplicity, robustness and good performances) are of course a first
motivation (it was used successfully in e.g. [1] with binary indicators). In
addition, the actual classification is performed in a way that is very easy to
interpret by a domain expert with limited machine learning expertise. It consists
indeed in comparing posterior probabilities, which can be done on an indicator
by indicator basis, by computing P (B
p=bp|Y =k)
P (Bp=bp|Y=k′) .
This allows to show to the user the indicators, and thus the underlying scores,
that are the most important in one decision, by being the more discriminant
between classes for a given observation (see [6] for a complete visual solution).
In our application context, a black box decision model is unacceptable, while
this kind of grey box decision is accepted by the domain experts, as long as the
NBC is constructed from their scores.
4 Feature selection for the NBC
Selecting good features is of utmost importance to get good performances with
a NBC. In addition, the type of decision analysis we mentioned in the previous
section is only realistic if the number of features remains relatively small.
Numerous feature selection methods have been investigated for NBC, ranging
from wrapper forward search [3] to mRMR like filter approach as in [1, 7], but
more sophisticated search strategies (such as forward backward methods, see
[2], chapter 4) have not. In addition, our specific context of highly redundant
binary indicators remains also unexplored. It finally be noted that the solution
recommended in text books remains a basic mutual information based filter
approach (see e.g. [4]).
4.1 Incremental calculation
The main motivation of filter approaches is generally the large computational
cost of wrapper solutions, as the latter tend to give better feature subsets than
the former. Fortunately, the NBC structure allows one to implement forward or
backward strategies in a rather efficient way. Indeed, the decision of a NBC is
done by comparing posterior probabilities, which can be done equivalently by
comparing the log likelihoods of the pair (B, k), for the different classes k:
logP (B = b, Y = k) = logP (Y = k) +
P∑
p=1
logP (Bp = bp|Y = k).
Given a feature subset of size m ≤ P , this can be computed in O(Nm) for all
the (Bi, Yi) provided the full conditional distribution P (B
p|Y = k) have been
already computed (this is done in O(NP ) if K is small compared to P ). Then
the effect of adding or removing a feature can be computed by simply adding or
subtracting to logP (B = b, Y = k) the contribution of the feature, that is in a
total time in O(N). Then evaluating all the features in a forward or backward
step is in O(NP ) and thus the total cost of a forward (or backward) search is in
O(NP 2). Notice that this does not apply to arbitrary search strategies where
one can move from a feature subset to a completely different one (such as in
genetic algorithms).
This is still an order of magnitude more expensive than e.g. a simple Mutual
Information (MI) based forward search which costs O(NP ) but it is comparable
to mRMR which costs also O(NP 2) when used to rank all the features. Com-
pared to classifiers for which no incremental solution exists, forward or backward
wrapper based approaches are then more affordable for the NBC. Indeed non
incremental classifiers have generally at least a training cost in O(Nm) for m
features, leading to a O(NP 3) total cost.
4.2 Other NBC specific aspects
In addition to the reasonable complexity of its wrapper solutions, the NBC has
some specific aspects that should be taken into account during feature selections.
Firstly, the NBC is non monotonic as adding features can degrade (at lot) its
performances (mostly because it cannot weight features). Thus branch-and-
bound solutions cannot be used reliably.
A second issue is the evaluation metric to be used during the search. In
wrapper approaches, one uses in general the risk under consideration, that is the
classification error in our case. However, because of the additive nature of the
NBC, many features have either no effect or an identical one when they are added
one by one to an existing set of features. In other words, the classification error
is not sensitive enough to distinguish between some of the features. We propose
therefore to use the error probability as estimated by the NBC itself as the quality
measure during feature selection. For a feature subset S, this is 1
N
∑N
i=1 P̂S(Y 6=
Yi|B = b) where P̂S(Y 6= Yi|B = b) is the conditional probability estimated with
the features from S by the NBC. Using this value amounts to taking into account
the uncertainty in the decision as estimated by the NBC itself. It should be noted
that the log conditional likelihood
∑N
i=1 log P̂S(Y = Yi|B = b) gives very poor
results in this NBC context because of the conditional independence assumption.
(results are not reported here for space reasons.)
5 Experimental evaluation
We compare in this section several feature selection scheme for the NBC used
on the binary indicators obtained as explained in Section 2.
5.1 Data sets
We use simulated data sets similar to the one used in [7]. The training set and the
test set have identical characteristics: they are made of 6000 times series, with
3000 normal examples (Gaussian white noise with σ = 1 standard deviation) and
3000 abnormal examples belong to three different classes (in equal proportion).
The mean change anomaly consists in switching from a µ = 0 mean white noise
to a µ ∈ [1, 5] white noise. The variance change anomaly consists in switching
from a σ = 1 standard deviation white noise to a σ ∈ [2, 6] white noise. Finally,
the trend shift anomaly adds a linear trend to the signal from the change point
to the end of the time series, with a final trend amplitude in [1, 5]. Signal lengths
are chosen uniformly at random in [100, 200] time steps, while the change point
happens in the 60 % central area of the signal (e.g. in [20, 80] for a length 100
signal).
The scores are based on sliding windows on which two population tests are
conducted (as explained in Section 1). We use the U test, the Kolmogorov-
Smirnov test and the F-test (variance test). The parameter is in all cases the
window length. We use also confirmatory scores based on successive windows.
Details on the parameter values and on the confirmation scores can be found in
[7]. After the binarization process described in Section 2 has been applied, we
obtain in this context 814 indicators.
5.2 General procedure and evaluation
For each feature selection method, the NBC is built on half of the training set
(keeping class proportions) and the best feature subset is selected using the
second half of the training set (by choosing the smallest subset among those
that have the lowest classification error). The feature subset is then evaluated
on the test set by reporting the classification error.
5.3 Feature selection techniques
We use two filter procedures as reference, namely a simple Mutual Information
(MI) feature ranking, and the mRMR ranking [5]. All the other methods are
wrapper approaches using either the classification error or the error probability
as performance measure. We compare a forward search (at each step, the best
feature is added to the feature set), a backward search (at each step, the worst
feature is removed from the feature set) and full forward/backward search (also
called floating search in [2]). In those algorithms, a forward phase is followed by
a backward phase (and vice versa) until the results do not improve. For instance,
one starts by a backward search to find a first optimal subset, then proceeds to
a forward search from this subset to get a better one (with more variables). In
case of improvement, the procedure is restarted from the last subset (backward,
then forward, etc.).
6 Results and discussion
Results are summarized in table 1. As expected, the wrapper approaches outper-
form the filter ones. In addition, the high redundancy of the binary indicators,
implied by their constructions, has strong adverse effects on the MI filter method
as it tends to select very redundant indicators. The mRMR ranking avoids this
effect but obtains sub optimal results.
Method Perf. Measure # of features test error
MI filter Error 422 0.1387
mRMR filter Error 19 0.1435
Forward search Error 136 0.1237
Forward search Probability 207 0.1225
Backward search Error 27 0.1308
Backward search Probability 86 0.1283
Forward–Backward Error 92 0.1238
Forward–Backward Probability 123 0.1237
Backward–Forward Error 112 0.1267
Backward–Forward Probability 122 0.1168
Table 1: Classification error obtained on the test set
Using the error probability rather than the classification error always im-
proves the results of the wrapper approaches. It allows a more accurate ordering
of the features than cannot be inferred by the search procedure alone.
Moving from a simple greedy search to a floating search improves the perfor-
mances in the case of the backward search as it tends to select too few variables.
In the case of the forward search, the performances are slightly degraded but
the number of features is strongly reduced.
Those results show that the filter approaches should be avoided for the NBC.
They also show that the error probability should be used to guide the greedy
search in order to get a more accurate ordering of the features at each step
of the search. The greedy search wrapper procedures give rather comparable
results with slightly increased performances for the floating search. In the highly
redundant binary indicators context, the backward floating search guided by
the error probability appears as the best solution, contrarily to the classical
recommendations for the NBC (namely, using a MI filter).
References
[1] F. Fleuret. Fast binary feature selection with conditional mutual information. Journal of
Machine Learning Research (JMLR), 5:1531–1555, 2004.
[2] I. Guyon, S. Gunn, M. Nikravesh, and L. A. Zadeh, editors. Feature Extraction: Founda-
tions and Applications. Springer, 2006.
[3] P. Langley and S. Sage. Induction of selective bayesian classifiers. In R. L. de Mántaras and
D. Poole, editors, UAI ’94: Proceedings of the Tenth Annual Conference on Uncertainty
in Artificial Intelligence, pages 399–406. Morgan Kaufmann, 1994.
[4] K. P. Murphy. Machine learning: a probabilistic perspective. MIT Press, 2012.
[5] H. Peng, F. Long, and C. Ding. Feature selection based on mutual information criteria
of max-dependency, max-relevance, and min-redundancy. Pattern Analysis and Machine
Intelligence, IEEE Transactions on, 27(8):1226–1238, 2005.
[6] B. Poulin, R. Eisner, D. Szafron, P. Lu, R. Greiner, D. S. Wishart, A. Fyshe, B. Pearcy,
C. MacDonell, and J. Anvik. Visual explanation of evidence with additive classifiers. In
Proceedings Of The National Conference On Artificial Intelligence, 2006.
[7] T. Rabenoro, J. Lacaille, M. Cottrell, and F. Rossi. Anomaly detection based on indicators
aggregation. In Proceedings of the International Joint Conference on Neural Networks
(IJCNN 2014), pages 2548–2555, Beijing (China), 7 2014. IEEE.


Search Strategies for Binary Feature Selection for a Naive Bayes Classifier

HAL-Paris1

Search Strategies for Binary Feature Selection for aNaive Bayes ClassifierTsirizo Rabenoro, Je´roˆme Lacaille, Marie Cottrell, Fabrice RossiTo cite this version:Tsirizo Rabenoro, Je´roˆme Lacaille, Marie Cottrell, Fabrice Rossi. Search Strategies for Bi-nary Feature Selection for a Naive Bayes Classifier. European Symposium on Artificial NeuralNetworks, Computational Intelligence and Machine Learning (ESANN), Apr 2015, Bruges, Bel-gium. pp.291-296, 2015, Proceedings of the 23-th European Symposium on Artificial NeuralNetworks, Computational Intelligence and Machine Learning (ESANN 2015). <hal-01162981>HAL Id: hal-01162981https://hal.archives-ouvertes.fr/hal-01162981Submitted on 11 Jun 2015HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestine´e au de´poˆt et a` la diffusion de documentsscientifiques de niveau recherche, publie´s ou non,e´manant des e´tablissements d’enseignement et derecherche franc¸ais ou e´trangers, des laboratoirespublics ou prive´s.Distributed under a Creative Commons Attribution 4.0 International LicenseSearch Strategies for Binary Feature Selectionfor a Naive Bayes ClassifierTsirizo Rabenoro1,2, Je´roˆme Lacaille1 , Marie Cottrell2, and Fabrice Rossi2 ∗1- SAMM EA 4543, Universite´ Paris 1 Panthe´on-Sorbonne90, rue de Tolbiac, 75634 Paris cedex 13, France2- Snecma, Groupe Safran,77550 Moissy Cramayel, FranceAbstract. We compare in this paper several feature selection methods forthe Naive Bayes Classifier (NBC) when the data under study are describedby a large number of redundant binary indicators. Wrapper approachesguided by the NBC estimation of the classification error probability out-perform filter approaches while retaining a reasonable computational cost.1 IntroductionWe consider in this paper application contexts in which a large body of expertknowledge is available as a series of simple and low level parametric scores.The goal is to build an interpretable classifier from this knowledge (and from alearning set). Then the scores are assumed to be simple parametric functionsfrom the data space to R with the interpretation that a high value of a scoreindicates that the datum submitted to the score belongs probably to the classthat the score has been designed to detect.Let’s consider a concrete example from our main application domain, aircraftengine monitoring (see [7] for details). We aim here at classifying some short timeseries (around 150 time points, each series having its own specific length) intodifferent classes (normal signal and different types of anomalies correspondingto some non stationarity in the signal). Domain experts have selected a set ofstatistical tests as scores. For instance, the Mann-Whitney U test can be used toreject the null hypothesis that two populations are identical. It can be appliedto a time series by selecting a potential break point in the series tb and a windowsize w, and by considering the w/2 points before tb as the first population andthe w/2 points after tb as the second population. The score is the p-value ofthe U test applied to those populations: a high value leads to not rejecting thenull hypothesis and thus is an indication that the time series belongs to “noanomaly” class. Notice that the parameters of the score are here the potentialbreak point tb and the window size w. See [7] for other examples.While the experts can design scores, they are seldom able to provide morethan hints about the parameters and the thresholds (i.e. when to consider thatthe score is “high enough”). In addition, the scores are generally no sufficientalone and several of them should be combined to achieve acceptable classificationrates. We proposed in [7] to address this problem via feature selection using thefilter mRMR approach [5]. The main idea, recalled in Section 2, consists in∗T. Rabenoro is supported by a grant from Snecma, Safran Group.turning the scores into a large set of redundant binary features and in using afeature selection method to keep useful ones. This has the effect of finding goodparameters and thresholds for the scores while retaining a reasonable numberof scores, easing interpretation of the decision made by a Naive Bayes Classifier(NBC) built on them. In [7], the selection is done by a filter method. Weinvestigate in the present paper wrapper based approaches.2 From scores to binary indicatorsWe assume given a training set (Xi, Yi)1≤i≤N where the observations space Xcan be arbitrary while the target space is a finite set of classes, Y = {1, . . . ,K}.We are also given a set of Q parametric scores, (sq)1≤q≤Q. Each sq is a functionfrom X ×Wq to R, where Wq is the parameter space of the score.The main constraint of our context is that experts only allow score resultsto be used by the classifier. In addition, the semantic of the scores means thatonly decisions of the form sq(Xi, wq) ≤ λq are really meaningful. We propose totransform this set of scores into a much larger set of binary indicators. This isdone by choosing for each score a finite subset of Wq, {w1q , . . . , wpqq } and a finiteset of thresholds {λ1q, . . . , λtqq }, and by defining pq × tq indicator functions byIp,tq (X) = Isq(X,wpq )≤λtq . This can be seen as a form of grid search in the “scorespace” in the sense that tuning the scores to the data set can be done indirectlyby selecting relevant binary indicators (a similar principle is used in e.g. [1]).By feeding the training set (Xi, Yi)1≤i≤N through the indicators, we obtain anew training set (Bi, Yi)1≤i≤N where the Bi take values into {0, 1}P where P isthe total number of binary indicators generated from the scores.Contrarily to arbitrary binary valued variables, those indicators are intrinsi-cally highly redundant and correlated. For instance, if λtq < λt′q , then Ip,tq (X) = 1implies Ip,t′q (X) = 1. This has adverse consequences on feature selection methodsand on the Naive Bayes Classifier.3 Naive Bayes ClassifierThe Naive Bayes Classifier (NBC, this e.g. [4]) is a very simple and robustclassifier based on the (unrealistic) assumption that the features used to describethe objects to classify are conditionally independent given the class. In ourcontext, this translates into P (B = b|Y = k) =∏Pp=1 P (Bp = bp|Y = k), whereBp is the p-th indicator value in the indicator vector. This allows to estimateeasily the posterior probability P (Y = k|B = b) asP (Y = k|B = b) =P (Y = k)∏Pp=1 P (Bp = bp|Y = k)P (Bp = b),using estimated values of the P (Bp = bp|Y = k). Those values are obtained bysimple class conditional counts.In our context, the motivation for using the NBC is twofold. Its classicalproperties (simplicity, robustness and good performances) are of course a firstmotivation (it was used successfully in e.g. [1] with binary indicators). Inaddition, the actual classification is performed in a way that is very easy tointerpret by a domain expert with limited machine learning expertise. It consistsindeed in comparing posterior probabilities, which can be done on an indicatorby indicator basis, by computing P (Bp=bp|Y=k)P (Bp=bp|Y=k′) .This allows to show to the user the indicators, and thus the underlying scores,that are the most important in one decision, by being the more discriminantbetween classes for a given observation (see [6] for a complete visual solution).In our application context, a black box decision model is unacceptable, whilethis kind of grey box decision is accepted by the domain experts, as long as theNBC is constructed from their scores.4 Feature selection for the NBCSelecting good features is of utmost importance to get good performances witha NBC. In addition, the type of decision analysis we mentioned in the previoussection is only realistic if the number of features remains relatively small.Numerous feature selection methods have been investigated for NBC, rangingfrom wrapper forward search [3] to mRMR like filter approach as in [1, 7], butmore sophisticated search strategies (such as forward backward methods, see[2], chapter 4) have not. In addition, our specific context of highly redundantbinary indicators remains also unexplored. It finally be noted that the solutionrecommended in text books remains a basic mutual information based filterapproach (see e.g. [4]).4.1 Incremental calculationThe main motivation of filter approaches is generally the large computationalcost of wrapper solutions, as the latter tend to give better feature subsets thanthe former. Fortunately, the NBC structure allows one to implement forward orbackward strategies in a rather efficient way. Indeed, the decision of a NBC isdone by comparing posterior probabilities, which can be done equivalently bycomparing the log likelihoods of the pair (B, k), for the different classes k:logP (B = b, Y = k) = logP (Y = k) +P∑p=1logP (Bp = bp|Y = k).Given a feature subset of size m ≤ P , this can be computed in O(Nm) for allthe (Bi, Yi) provided the full conditional distribution P (Bp|Y = k) have beenalready computed (this is done in O(NP ) if K is small compared to P ). Thenthe effect of adding or removing a feature can be computed by simply adding orsubtracting to logP (B = b, Y = k) the contribution of the feature, that is in atotal time in O(N). Then evaluating all the features in a forward or backwardstep is in O(NP ) and thus the total cost of a forward (or backward) search is inO(NP 2). Notice that this does not apply to arbitrary search strategies whereone can move from a feature subset to a completely different one (such as ingenetic algorithms).This is still an order of magnitude more expensive than e.g. a simple MutualInformation (MI) based forward search which costs O(NP ) but it is comparableto mRMR which costs also O(NP 2) when used to rank all the features. Com-pared to classifiers for which no incremental solution exists, forward or backwardwrapper based approaches are then more affordable for the NBC. Indeed nonincremental classifiers have generally at least a training cost in O(Nm) for mfeatures, leading to a O(NP 3) total cost.4.2 Other NBC specific aspectsIn addition to the reasonable complexity of its wrapper solutions, the NBC hassome specific aspects that should be taken into account during feature selections.Firstly, the NBC is non monotonic as adding features can degrade (at lot) itsperformances (mostly because it cannot weight features). Thus branch-and-bound solutions cannot be used reliably.A second issue is the evaluation metric to be used during the search. Inwrapper approaches, one uses in general the risk under consideration, that is theclassification error in our case. However, because of the additive nature of theNBC, many features have either no effect or an identical one when they are addedone by one to an existing set of features. In other words, the classification erroris not sensitive enough to distinguish between some of the features. We proposetherefore to use the error probability as estimated by the NBC itself as the qualitymeasure during feature selection. For a feature subset S, this is 1N∑Ni=1 PˆS(Y 6=Yi|B = b) where PˆS(Y 6= Yi|B = b) is the conditional probability estimated withthe features from S by the NBC. Using this value amounts to taking into accountthe uncertainty in the decision as estimated by the NBC itself. It should be notedthat the log conditional likelihood∑Ni=1 log PˆS(Y = Yi|B = b) gives very poorresults in this NBC context because of the conditional independence assumption.(results are not reported here for space reasons.)5 Experimental evaluationWe compare in this section several feature selection scheme for the NBC usedon the binary indicators obtained as explained in Section 2.5.1 Data setsWe use simulated data sets similar to the one used in [7]. The training set and thetest set have identical characteristics: they are made of 6000 times series, with3000 normal examples (Gaussian white noise with σ = 1 standard deviation) and3000 abnormal examples belong to three different classes (in equal proportion).The mean change anomaly consists in switching from a µ = 0 mean white noiseto a µ ∈ [1, 5] white noise. The variance change anomaly consists in switchingfrom a σ = 1 standard deviation white noise to a σ ∈ [2, 6] white noise. Finally,the trend shift anomaly adds a linear trend to the signal from the change pointto the end of the time series, with a final trend amplitude in [1, 5]. Signal lengthsare chosen uniformly at random in [100, 200] time steps, while the change pointhappens in the 60 % central area of the signal (e.g. in [20, 80] for a length 100signal).The scores are based on sliding windows on which two population tests areconducted (as explained in Section 1). We use the U test, the Kolmogorov-Smirnov test and the F-test (variance test). The parameter is in all cases thewindow length. We use also confirmatory scores based on successive windows.Details on the parameter values and on the confirmation scores can be found in[7]. After the binarization process described in Section 2 has been applied, weobtain in this context 814 indicators.5.2 General procedure and evaluationFor each feature selection method, the NBC is built on half of the training set(keeping class proportions) and the best feature subset is selected using thesecond half of the training set (by choosing the smallest subset among thosethat have the lowest classification error). The feature subset is then evaluatedon the test set by reporting the classification error.5.3 Feature selection techniquesWe use two filter procedures as reference, namely a simple Mutual Information(MI) feature ranking, and the mRMR ranking [5]. All the other methods arewrapper approaches using either the classification error or the error probabilityas performance measure. We compare a forward search (at each step, the bestfeature is added to the feature set), a backward search (at each step, the worstfeature is removed from the feature set) and full forward/backward search (alsocalled floating search in [2]). In those algorithms, a forward phase is followed bya backward phase (and vice versa) until the results do not improve. For instance,one starts by a backward search to find a first optimal subset, then proceeds toa forward search from this subset to get a better one (with more variables). Incase of improvement, the procedure is restarted from the last subset (backward,then forward, etc.).6 Results and discussionResults are summarized in table 1. As expected, the wrapper approaches outper-form the filter ones. In addition, the high redundancy of the binary indicators,implied by their constructions, has strong adverse effects on the MI filter methodas it tends to select very redundant indicators. The mRMR ranking avoids thiseffect but obtains sub optimal results.Method Perf. Measure # of features test errorMI filter Error 422 0.1387mRMR filter Error 19 0.1435Forward search Error 136 0.1237Forward search Probability 207 0.1225Backward search Error 27 0.1308Backward search Probability 86 0.1283Forward–Backward Error 92 0.1238Forward–Backward Probability 123 0.1237Backward–Forward Error 112 0.1267Backward–Forward Probability 122 0.1168Table 1: Classification error obtained on the test setUsing the error probability rather than the classification error always im-proves the results of the wrapper approaches. It allows a more accurate orderingof the features than cannot be inferred by the search procedure alone.Moving from a simple greedy search to a floating search improves the perfor-mances in the case of the backward search as it tends to select too few variables.In the case of the forward search, the performances are slightly degraded butthe number of features is strongly reduced.Those results show that the filter approaches should be avoided for the NBC.They also show that the error probability should be used to guide the greedysearch in order to get a more accurate ordering of the features at each stepof the search. The greedy search wrapper procedures give rather comparableresults with slightly increased performances for the floating search. In the highlyredundant binary indicators context, the backward floating search guided bythe error probability appears as the best solution, contrarily to the classicalrecommendations for the NBC (namely, using a MI filter).References[1] F. Fleuret. Fast binary feature selection with conditional mutual information. Journal ofMachine Learning Research (JMLR), 5:1531–1555, 2004.[2] I. Guyon, S. Gunn, M. Nikravesh, and L. A. Zadeh, editors. Feature Extraction: Founda-tions and Applications. Springer, 2006.[3] P. Langley and S. Sage. Induction of selective bayesian classifiers. In R. L. de Ma´ntaras andD. Poole, editors, UAI ’94: Proceedings of the Tenth Annual Conference on Uncertaintyin Artificial Intelligence, pages 399–406. Morgan Kaufmann, 1994.[4] K. P. Murphy. Machine learning: a probabilistic perspective. MIT Press, 2012.[5] H. Peng, F. Long, and C. Ding. Feature selection based on mutual information criteriaof max-dependency, max-relevance, and min-redundancy. Pattern Analysis and MachineIntelligence, IEEE Transactions on, 27(8):1226–1238, 2005.[6] B. Poulin, R. Eisner, D. Szafron, P. Lu, R. Greiner, D. S. Wishart, A. Fyshe, B. Pearcy,C. MacDonell, and J. Anvik. Visual explanation of evidence with additive classifiers. InProceedings Of The National Conference On Artificial Intelligence, 2006.[7] T. Rabenoro, J. Lacaille, M. Cottrell, and F. Rossi. Anomaly detection based on indicatorsaggregation. In Proceedings of the International Joint Conference on Neural Networks(IJCNN 2014), pages 2548–2555, Beijing (China), 7 2014. IEEE.

https://hal.archives-ouvertes.fr/hal-01162981/document

Search Strategies for Binary Feature Selection for a Naive Bayes Classifier

Abstract

Similar works

Full text

Available Versions

Archive Ouverte en Sciences de l'Information et de la Communication

HAL-Paris1