Includes bibliographical references.This paper presents a training algorithm for probabilistic neural networks (PNNs) using the minimum classification error (MCE) criterion. A comparison is made between the MCE training scheme and the widely used maximum likelihood (ML) learning on a cloud classification problem using satellite imagery data.This work was supported by the DoD Center for Geosciences/Atmospheric Research (CG/AR) under Contract DAAL01-98-2-0078

Azimi-Sadjadi, Mahmood R.

Tian, Bin

English

This paper presents a training algorithm for probabilistic neural networks (PNNs) using the minimum classification error (MCE) criterion. A comparison is made between the MCE training scheme and the widely used maximum likelihood (ML) learning on a cloud classification problem using satellite imagery data.This work was supported by the DoD Center for Geosciences/Atmospheric Research (CG/AR) under Contract DAAL01-98-2-0078

Azimi-Sadjadi, Mahmood R., author

Tian, Bin, author

IEEE, publisher

Mountain Scholar (Digital Collections of Colorado and Wyoming)

164 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 12, NO. 1, JANUARY 2001Comparison of Two Different PNN Training Approaches for Satellite CloudData ClassificationBin Tian and Mahmood R. Azimi-SadjadiAbstract—This paper presents a training algorithm for proba-bilistic neural networks (PNNs) using the minimum classificationerror (MCE) criterion. A comparison is made between the MCEtraining scheme and the widely used maximum likelihood (ML)learning on a cloud classification problem using satellite imagerydata.Index Terms—Cloud classification, maximum likelihood, min-imum classification error, probabilistic neural network.I. INTRODUCTIONProbabilistic neural network (PNN) is a kind of supervisedneural network that are widely used in the area of pattern recog-nition, nonlinear mapping, and estimation of probability of classmembership and likelihood ratios. The original PNN structure[1], is a direct neural-network implementation of the Parzennonparametric probability density function (PDF) estimation[2] and Bayes classification rule. Although its training schemeis very simple and fast, one major drawback is that potentiallya very large network will be formed since every training patternneeds to be stored. This leads to increased storage and compu-tational time requirements during the testing phase. One naturalidea to simplify the PNN is to reduce the number of neurons,i.e., use fewer kernels but place them at optimal places. In [3]Streitet al. improved the PNN by using finite Gaussian mixturemodels and maximum likelihood (ML) training scheme. How-ever, the ML-based training does not necessarily lead to a min-imum error performance for the classifier. This may be due tothe fact that the Gaussian mixture model may not be an accu-rate assumption for some of the feature space distribution andthe training data set is often inadequate. In [4], Juanget al.pro-posed a new learning scheme based upon the minimum classifi-cation error (MCE) criterion. In [5], Gish pointed out that min-imization of the number of errors is not the only benefit of theMCE. The MCE criterion is also inherently robust. The robust-ness stems from its counting misclassifications and ignoring themagnitude of the error, i.e., ignoring how far the misclassifiedevents are from the decision boundary. Owing to its robustness,MCE has been widely used in speech recognition applications[6], [7].In this study, ML and MCE are used to estimate the parametersets of the Gaussian mixture model. Their performances are ex-amined on the Geostationary Operational Environmental Satel-lite (GOES)-8 imagery data for cloud classification. The organ-ization of this paper is as follows. Section II briefly introducesManuscript received September 9, 1999; revised May 15, 2000. This workwas supported by the DoD Center for Geosciences/Atmospheric Research(CG/AR) under Contract DAAL01-98-2-0078.The authors are with the Department of Electrical and Computer Engineering,Colorado State University, Fort Collins, CO 80523-1373 USA.Publisher Item Identifier S 1045-9227(01)00536-7.the Gaussian mixture model. In Sections III and IV, ML, andMCE training schemes for PNN are discussed separately. Com-parisons of these training algorithms are presented in Section V.II. GAUSSIAN MIXTURE MODELConsider a -dimensional input feature vector which be-longs to one of the classes, . A classifiercan be regarded as a mapping, thatclassifies the given pattern to class . Suppose that theclass conditional distribution, , and thea priori classprobability are known, then the best classifier is given bythe fundamental Bayes decision rule(1)One main concern when implementing the above optimal Bayesclassifier is to estimate and from the training dataset. Generally, is highly dependent on the specific task andshould be decided by the physical knowledge of the problem.For the sake of convenience, uniform distribution assumptionfor is adopted in this study. Also, for any class, we as-ume that the can be represented by a Gaussian mixturemodel, i.e,(2)where is the number of Gaussian components in classand’s are the weights of the components which satisfy the con-straint denotes the multivariateGaussian density function of theth component in class andand are its mean vector and covariance matrix, respec-tively. This Gaussian mixture model can be easily mapped tohe PNN structure and the resultant PNN will need much fewern urons. The price paid for this simplification is that the simplenoniterative training procedure will no longer be applicable. In-tead, the weights of the PNN, i.e., the parameter sets of the mix-ture model for each class, need to be estimated from the trainingdata set.III. M AXIMUM LIKELIHOOD TRAINING FOR PNNLet denote the parameter set used todescribe the mixture model of classand denotethe whole parameter space for the PNN. The goal of training isto estimate the parameter spacefrom the training set. If weassume that the parameters inare unknown fixed quantities,the ML estimation method is a suitable choice.Now suppose that the training samples drawn independentlyfrom the feature space form the set, which can be further1045–9227/01$10.00 © 2001 IEEEIEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 12, NO. 1, JANUARY 2001 165separated into subsets , in which all thesamples belong to class. The ML estimation of parameter setis then given by(3)For the computational efficiency, generally we will maximizethe equivalent log-likelihood, i.e.,(4)The last step in (4) is arrived at based upon the assumption thatthe conditional probability of class is solely decided by theparameter set of that class,and not by the parameter set of theother classes. The maximization of the log-likelihood functioncan be done using a probability gradient descent (PGD) scheme[8]. Let(5)denote the log likelihood function and take the partial derivativeof this function with respect to each parameter in, then wehave(6)where represents either , or . Based on (5) and(6), we can further get(7)Based on the PGD scheme, the log-likelihood function can bemaximized using the following training equations:(8)where and are learning factors with values betweenzero and one.There is one important observation from (7) and (8). The up-dating of parameter set of class is only dependent on thetraining samples in this class, i.e., the optimization process canbe solved separately for each class without considering the ef-fect of the others. This is especially suitable for the cloud classi-fication application since a new cloud type can easily be addedto the system without affecting the other classes. Moreover, inthe updating process, we have the choice of updating only thoseclasses that are affected by the temporal changes in the cloudfeatures. Another benefit of this property is the reduced trainingtime due to the fact that each class can be trained separately, thusrequiring a small number of neurons and training samples.Due to the nature of the PGD scheme, the PNN trainingprocess using (7) and (8) generally needs a lot of iterationsbefore converging to an acceptable result. This incurs expensivecost in the training phase. Fortunately, there is an efficientapproach called expectation-maximization (EM) which solvesthis problem. The reader is referred to [9] for the detail treat-ment on this topic.IV. MCE TRAINING FOR PNNThe main idea behind using ML criterion for the PNN trainingwas to accurately estimate the class conditional probability fromthe training samples. However, since the number of trainingsamples is limited and the Gaussian mixture assumption maynot be correct, the estimated distribution may not be accurate.So the optimal performance of the Bayes classifier may not bereached in practice. If we reexamine the Bayes classificationrule in (1), it can be found that the actual value of isin fact not so important for decision. As long as the conditionalprobability is larger than the corresponding values for the otherclasses, the classifier can still make the right decision. There-fore, a natural approach to improve the performance of classi-fier is to estimate the functions , whichcan most successfully discriminate different classes. This is thebasic idea behind the discriminant analysis. In general, the dis-criminant function can be in any form and may not necessarilyrelate to the probability, but here we still use the same form of, i.e., Gaussian mixture models.Consider an input feature vectorbelonging to class . Ac-cording to the Bayes decision rule in (1),will be correctlyclassified ifor(9)which also can be rewritten asor(10)166 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 12, NO. 1, JANUARY 2001For the training set , we can define a cost function(11)where is a step function. It is very clear that the cost func-tion is nothing but the count of the number of incor-rectly classified samples. Minimizing this cost function thusleads to minimizing the classification error. As a result, this cri-terion is called MCE criterion [4], [10].Direct minimization of the cost function in (11) is almost im-possible since both the max (maximization) operation and thestep function are not differentiable. The maximization opera-tion is used to find the most critical rival class for. In [4], afunction was suggested to approximate the maximiza-tion operation(12)For large is generally a very good approximation ofthe unless several s are simulta-neously close to or equal to the maximum value. In most of thesituations, or is sufficient [4]. Moreover, we can use thesigmoid function to replace the step function. Sigmoid functioncan be considered as a smoothed version of step function and isdefined as(13)where is a parameter. Using these approximations, the costfunction in (10) becomes(14)This cost function in (14) is also called as “smooth count ofclassification error” since the sigmoid function is used [10].Now based on the MCE criterion, we want to find the pa-rameter sets of the PNN,, that can minimize the cost function(14). Again we can use the PGD scheme. For thet Gaussiancomponent in class , we take the derivative of the cost func-tion (14) with respect to the parameters. Using the chain ruleand after some manipulations, we can getand (15)where represents either parameter or . Thelast step is obtained since thea priori class probability,, is independent of the parameterss. The results of(a)(b)Fig. 1. GOES 8 satellite images obtained at 15:45 UTC, May 1st, 1995. (a)Visible. (b) IR.TABLE ICONFUSIONMATRIX FOR THE ML TRAINED PNN. (OVERALL CLASSIFICATIONRATE 84.9%)for different parameters were givenbefore in (7) in Section II. The function is defined as(16)Once the derivative is determined, a similar learning rule as in(8) can be used to minimize the cost function in (14).There are several observations from the training equations in(16). First, the decoupling propertydoes notexist for the min-imum error criterion. Each training sample contributes to theIEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 12, NO. 1, JANUARY 2001 167Fig. 2. Comparison of color-coded images. (a) Result of ML trained PNN. (b) Result of MCE trained PNN. (c) Expert labelled Image. (d) Colormap for tenclasses.estimation of parameters of class, no matter whether it be-longs to class or not. This is quite different from the situationin the ML training where the parameters of classare only de-cided by the training samples of that class. Moreover, the contri-bution of each training sample to the derivative is weighted by. The property of this function can be clearly demon-strated in the following cases. Letbelong to class , and as-sume , i.e, can be correctly identified bythe current parameter set with confidence, the contribution ofthis sample to the class will be weighted bysince for. Similarly, if , i.e, the input is too difficultto be correctly classified, its contribution to the parameter set isalso very small. On the other hand, those inputs located on thedecision boundary region will lead to comparable andvalues, thus contribute mostly to the final parameter es-timation. Overall, the training results of the MCE criterion aremainly decided by the samples around the decision boundariesformed by the current parameter sets. This is a very distinct char-acteristic of the MCE training.Unlike the ML criterion, it is difficult to find an efficienttraining approach for the MCE criterion to replace the PGDscheme. The PGD solution suffers from several drawbacks. Itnot only converges very slowly leading to extensive computa-tional cost, but also it is prone to local minimum problem. Basedon our experiments, the result of the MCE-PGD training is verysensitive to the initial values of the parameter set. This is partlydue to the fact that the MCE training is mainly decided by thedistribution of a small subset of the training samples (aroundthe initial boundaries) instead of the whole training set. It isquite common that the subset is not representing well the featurespace or its size is too small to lead to any meaningful trainingresult. In order to overcome this initialization problem, in ourapplication we always use the ML trained PNN as the startingpoint for the MCE-PGD training [4].V. RESULTS AND DISCUSSIONSThe performance of ML and MCE training algorithm was ex-amined using channel 1 (visible) and channel 4 (IR) of GOES 8atellite images. One typical image pair obtained at 15:45 uni-versal time code (UTC), May 1st, 1995 is shown in Fig. 1.After classification, the clouds were separated into tenclasses: Warm Land (Wl), Cold Land (Cl), Warm Water (Ww),Cold Water (Cw), Stratus (St), Cumulus (Cu), Altostratus (As),Cirrus (Ci), Cirrostratus (Cs), and Stratocumulus (Sc). Table Ipresents the classification confusion matrix of the ML trainingscheme. The numbers located on the diagonal indicate thecorrect classification rate for each class. The overall classifi-cation rate is 84.9%. The color-coded image based on SVD168 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 12, NO. 1, JANUARY 2001TABLE IICONFUSION MATRIX FOR THE MCE TRAINED PNN. (OVERALLCLASSIFICATION RATE 86.9%)features [11] and using the ML-based classifier is shown inFig. 2(a). For the ML training, the EM approach can help toachieve the maximum-likelihood estimation efficiently whenthe observations can be viewed as incomplete data.The confusion matrix of the MCE trained PNN is given inTable II. Comparing with the results of the ML-based PNN inTable I, the overall classification rate is improved by 2%, notas dramatic as we expected. This observation indicates that theGaussian mixture model may in fact be a good representationof the feature space. Among the ten classes, accuracy improve-ments were observed for six of them, with the exceptions ofWarm Water (Ww), Stratus (St), Altostratus (As) and Cirro-stratus (Cs). The color-coded classified image is provided in Fig.2(b). Visual inspection of Fig. 2(b) and (a) reveals that the twoimages are quite similar except for some minor isolated blocks.Fig. 2(c) and (d) shows the meteorological expert labelled imageand the colormap for ten different classes, respectively. Note thatin Fig. 2(c) only those areas for which the labeling results ofthe experts agreed were color coded and used for training andtesting of the PNNs.Overall, this study indicates that the MCE training can pro-vide some improvements in the classification rate when com-pared with the ML-type training. Clearly, the improvement ofMCE training is dependent on the feature space distribution.However, considering that the PGD approach used for the MCEtraining generally needs much more computational time thanthat of the EM approach for the ML training, the performanceimprovements may not be significant enough to justify the ad-ditional training cost.REFERENCES[1] D. F. Specht, “Probabilistic neural network,”Neural Networks, vol. 3,pp. 109–118, 1990.[2] E. Parzen, “On estimation of a probability density function and mode,”Ann. Math. Statist., vol. 33, pp. 1065–1076, 1962.[3] R. L. Streit and T. E. Luginbuhl, “Maximum likelihood training of prob-abilistic neural networks,”IEEE Trans. Neural Networks, vol. 5, pp.764–783, 1994.[4] B. H. Juang and S. Katagiri, “Discriminative learning for minimum errorclassification,”IEEE Trans. Signal Processing, vol. 40, pp. 3043–3053,Dec. 1992.[5] H. Gish, “A minimum classification error, maximum likelihood, neuralnetwork,” in Proc. 1992 IEEE Int. Conf. Acoust., Speech, Signal Pro-cessing, vol. 2, pp. 289–292.[6] R. Chengalvarayan and L. Deng, “Speech trajectory discriminationusing minimum classification error learning,” inIEEE Trans. SpeechAudio Processing, vol. 6, pp. 505–515.[7] D. Rainton and S. Sagayama, “A new minimum error classificationtraining technique for HMM-based speech recognition,” presented atthe Proc. 3rd Int. Symp. Signal Processing Applicat.. ISSPA-92.[8] S. Haykin, “Neural networks: A comprehensive foundation,”. Englewood Cliffs, NJ: Prentice-Hall, 1994.[9] A. P. Dempster, N. M. Laird, and D. B. Rubin, “Maximum likelihoodfrom incomplete data via the EM algorithm,”J. Roy. Statist. Soc., ser. B,vol. 39, pp. 1–38, 1977.[10] H. Ney, “On the probabilistic interpretation of neural network classifiersand discriminative training criteria,”IEEE Trans. Pattern Anal. MachineIntell., vol. 17, pp. 107–119, 1995.[11] B. Tian, M. A. Shaikh, M. R. Azimi-Sadjadi, T. H. Vonder-Haar, andD. L. Reinke, “A Study of Clo9ud Classification with Neural NetworksUsing Spectral and Textural Features,”IEEE Trans. Neural Networks,vol. 10, pp. 138–151, Jan. 1999.

Comparison of two different PNN training approaches for satellite cloud data classification

https://mountainscholar.org/bitstream/10217/926/1/ECEmra00008.pdf

Comparison of two different PNN training approaches for satellite cloud data classification

Abstract

Similar works

Full text

Available Versions

Mountain Scholar (Digital Collections of Colorado and Wyoming)