The use of unlabeled samples in improving the performance of classifiers is studied. When the number of training samples is fixed and small, additional feature measurements may reduce the performance of a statistical classifier. It is shown that by using unlabeled samples, estimates of the parameters can be improved and therefore this phenomenon may be mitigated. Various methods for using unlabeled samples are reviewed and experimental results are provided

Landgrebe, David A.

Shahshahani, Behzad M.

English

NASA Technical Reports Server

NASA-CR-Z04756
USE OF UNLABELED SAMPLES FOR MITIGATING THE HUGtlES PHENOMENON ] +
Behzad M. Shahshahani and David A. Landgrebe
School of Electrical Engineering
Purdue University
West Lafayette, IN 47906
Tel: (317) 494-1743; Fax: (317) 494-6440
behzad@ecn.purdue.edu landgreb@ecn.purdue.edu
++
Abstract
The use of unlabeled samples in improving the performance of
classifiers is studied. When the number of training samples is
fixed and small, additional feature measurements may reduce
the performance of a statistical classifier. It is shown that by
using unlabeled samples estimates of the parameters can be
improved and therefore this phenomenon may be mitigated.
Various methods for using unlabeled samples are reviewed and
experimental results are provided.
Key Words: Hughes Phenomenon, Small Sample Size,
Classifier Design
I. Introduction
Muhispectral sensor technology is advancing towards the
production of sensors with larger numbers of spectral bands.
The hope is that the detailed spectral responses produced by
these sensors could be used for discriminating between more
classes thus, providing better understanding of the nature of the
materials populating the scene. For example, high dimensional
spectra/ responses are considered very informative for
distinguishing among sub-classes of a particular ground cover
cJass. In order to design a classifier, training samples from each
sub-class are required. It becomes difficult and expensive to
obtain large training sample sets for each sub-class.
Consequently, the conventional problem of small sample size
versus high dimensionality arises. This problem, often referred
to as the Hughes phenomenon [1], is the loss of ctassifiability
that is observed when the dimensionality of the data increases
while the training sample size remains fixed.
II. Hughes Phenomenon
The minimum achievable error in a classification problem is
the Bayes error. A decision rule that assigns a sample to the
class that has the maximum a posterior probability (MAP
classifier) achieves the Bayes error. In order to design such a
classifier, knowledge of the posterior probabilities and thus, the
class conditional probability density functions is required. If
such knowledge is available then by increasing the
dimensionality one would expect to enhance the performance.
[n other words, the Bayes error is a decreasing function of the
dimensionality of the data. In practice, however, class
conditional probability density functions (pdf's) need to be
estimated from a set of training samples. When these estimates
are used in place of the true values of the pdf's the resulting
decision rule is sub-optimal and hence has a higher probability
of error. The expected value of the probability of error, taken
over all training sample sets of a particular size is, therefore,
larger than the Bayes error. When a new feature is added to the
data the Bayes error decreases, but at the same time the bias of
the classification error increases. This increase is due to the fact
that more parameters need to be estimated from the same
number of samples. If the increase in the bias of the
classification error is more than the decrease in the Baycs error,
then the use of the additional feature degrades the performance
of the decision rule. This phenomenon is called the Hughes
effect.
I This work was suppOrted in part by NASA under Grant NAGW-925.
III. Classification Error
Consider a classification problem involving m classes with
prior probabilities Pi and probability density functions re(x). By
e we denote the Bayes error achieved by using the MAP
classifier when Pi and fi(x) are known. Let 0 denote the vector
of parameters of the MAP classifier. Let 0 ° denote the true
value of 8. The error achieved by using 8" in the decision rule
is e*, the Bayes error. Now, let's assume that 0 is an estimate
of 0". If deviation of 0 from 0 ° is not large, one can
approximate the error corresponding to the decision rule
obtained using t9 by using a Taylor series expansion of up to
the second term:
at It.o" 2
where tr(A) denotes the trace of matrix A and I-I(0") is the
O:e(O)
Hessian evaluated at 0* ( ). The term
_o=o" ,90 Io---o"
is zero since 0" is an extreme point of e(O). If the bias of 0 is
zero or negligible (E(O} = 0*), then the expected value of
can be approximated as follows:
_{_} - e"+1-tr{ H(e" )cov(_)J
2
The second term in the right hand side is posidve since both the
covariance matrix and the Hessian at 0" are positive semi
definite (the Hessian is positive semi-definite at 0* since 0" is a
minimum of e(O) so e(O) is convex around 0"). Now, consider
another unbiased estimate 0. If coy(O) _< coy(O) (in other
words coy(O) - coy(O) is positive semi-definite) then one can
easily show that :
tr{n(O" )cov(O)) -_tr(H(e')cov(O)}
IV. Effect of Additional Unlabeled Samples
Let us assume that 0 is an estimate of 0 ° obtained by using the
training samples. Furthermore, assume that 0 is asymptotically
unbiased and efficient (for example, maximum likelihood
estimates always posses these properties). In other words, for
large sample sizes, E( O} - 0 ° and coy(O) = I_ t , where Is is
the Fisher information roan'ix. The subscript s denotes that the
Fisher information matrix corresponds to a supervised estimate
obtained by using training samples that are drawn from each
class separately. The Fisher information matrix is positive
definite and is defined as follows:
I= E{[_ologf(x)][ ff-_logf(x)]r }
*Proceedings of the International GeoscJence and Remote Sensing Symposium(IGARSS'93), Tokyo,
pp 1535-7, August 1993.
- 1535-
https://ntrs.nasa.gov/search.jsp?R=19980007187 2020-06-16T00:39:40+00:00Z
Now, let us assume that 0 is another estimate of Ov obtained by
using some unlabeled samples in addition to the training
samples. The unlabeled samples are drawn randomly from the
mixture of the m classes. If 0 possesses the same properties of
asymptotic unbiasedness and efficiency, one can approximate
cove 0) by Ic-I where lc is the Fisher information matrix
corresponding to the estimate that is obtained by combining
training and unlabeled samples. Provided that the unlabeled
and waining samples are independent, one can write:
Ic = I_ + I.
where Iu is another information matrix corresponding to the
information contained in the unlabeled samples for estimating
0e. Since all of the information matrices are positive definite
one can write Ic --2Is. Therefore, cov((_) S coy(0). From the
developments of section ll[, one can conclude that the expected
error of the decision rule that uses 0 is less than the one that is
obtained by using 0 :
E:g} -<E(d]
Therefore a decision rule that is obtained by incorporating the
unlabeled samples in the estimation process achieves a lower
error rate. Thus, by using such a decision rule the Hughes
phenomenon may be delayed to a higher dimension.
Consequently, more features can be used without sacrificing
the performance and in fact, the additional information in the
new features may cause an improvement in the classification
accuracy.
V. Methods of Incorporating Unlabeled Samples
parametric Ca_e
A particular case of interest is when individual classes are
multivariate Gaussian. In this case, the maximum likelihood
(ML) estimates of the parameters of the mixture density
consisting of the rn normal classes can be found by the EM
(Expectation-Maximization) algorithm [2]. Assuming that n
unlabeled samples denoted by xk are available from the mixture
,,"tl
density fix/O) = z_Pifi(x), and ni training samples denoted
i=l
by za are available from each class i, the ML estimates of the
parameters of the mixture can be found by using the following
iterafive equations [3]:
n_
--p:. = _=/ , 11i* =
k=l k=l
n _ pc(i/xk) 4- ni
k=l
?qi/x_)¢x_ -u:)¢xk _u:)r + _:z,k -u:)¢z_k -In:) r
k=l k=l
where p':i/_) = ?:fdxduf, _)
f(xd oc)
and/.ti and 2_. are the mean vector and the covariance matrix of
class i, superscripts + and c denote the next and current values
of the parameters respectively. The parameter set 0 contains all
the prior probabilities, mean vectors and covariance matrices.
Multi-Component Classes
Another interesting case that arises regularly in remote sensing
is when the individual classes have multiple components. If
each such component is assumed to be multivar-/ate Gaussian,
then the class conditional pdf's, fi(x), are themselves mixture
densities. Often training samples are known to belong to a
specific class without any reference to the particular
component within :hat class. In this case again the EM
algorithm can be used to obtain the ML estimates of the
parameters. The EM equations are presented in [4J.
Non Parame_: (_ase
Unlabeled samples can also be utilized in the nonparametric
estimation of the pdfs. One technique for doing so was
proposed in [5]. Let fi(x) be a nonparametric estimate offi(x)
obtained by using the training samples. Let f(x) be the
nonparametric estimate of the mixture densityf(x) based on the
training samples (obtained by a linear sum of .:i(x)), andf'(x)
be a nonparametric estimate of fix)obtained by using the
unlabeled samples alone. Then a nonparametric estimate fi(x)
of fi(x) based on both the training samples and unlabeled
samples can be constructed by the following formula:
:iex) = ]i(x)v., ,
:(x---S:
The estimate :i(x) was shown to have a lower variance than
_r_fx){5].
VI. Experimental Results
Exoeriment I (AVIRIS data)
A portion of an AVIRIS frame (consisting of 200 bands) taken
over Tippecanoe county in Indiana was considered in this
experiment. Four ground cover classes were determined by
using the ground truth map. The classes were bare soil (380
pixels), wheat (513 pixels), soybean (741 pixels), and corn
(836 pixels). Dimensionality of the data was changed from 1 to
18 by sequentially adding more bands. Selection of a new band
was based on the average pairwise Bhattacharrya distance
measure. Twenty training samples were drawn randomly from
each class. The statistics of each class were estimated once by
using only the twenty training samples and once using both the
twenty training samples and some unlabeled samples drawn
from the total field. The parametric equations for the single
component per class case were used to obtain the ML
estimates. Consequently, the rest of the samples were classified
and the total classification accuracy was computed. Each
experiment was repeated ten times independently and the
average of the ten u-ials was obtained. The results are shown in
Figure 1. In Figure 1, the supervised curve refers to the case
when training samples were used alone in the estimation
process and maximum likelihood classification was performed
consequently. The curves labeled combined 500 and combined
I000 refer to the cases when 500 and 1000 unlabeled samples
were used in addition to the training samples respectively and
MAP classification was subsequently performed.
It can be seen from Figure I that the Hughes phenomenon
starts to appear around dimension 8 when supervised learning
is used. It is delayed to dimension 14 when 500 unlabeled
samples were used and to 16 when 1000 unlabeled samples
were used. The minimum error for the supervised case was
5.93% and was achieved at dimension 8. For the cases with 500
and I000 unlabeled samples, the minimum errors were 3.78%
and 3.87% both at dimension 13. Therefore, the use of
unlabeled samples not only delayed the occurrence of the
- 1536-
Hughesphenomenonbut,alsomadethe information in the new
features usable for decreasing the error further.
3
¢J
1- T j
,o ...... ......... ....
i-Ti i l ii Ni i i i i,o :,.,.,,
: : : : : : : : : : : ix_ : : : i
,o _'- _---L-_.--_--i--_---i---_--_-_---L-_-_---i-- $-\ i
"1-- t ........6o --_--q..-_. _n,,sod __.;..._... _. _,_._... q...:
: : :l--"°-'--_°_"bi_'_s°° I : : : ',_i : !
,o --_.--_---_1 [-_---_--_--÷ -,--_ ....
'° -!--'-q' ...... ,_-@-:---!--i--:-_,--'---!
....... .....i .....
: : ', : : ', : : ', ,, ,, : : : : _,',
,o ] ] i ] I i i il i i _ J i I l
5 7 9 1_ 13 15 17
¢llmension
Figure h Classification performance versus dimensionality (AVIRIS data)
Exoeriment 2 (FLC1 Data)
The same kind of experiment was performed on a portion of
the Flight Line C1 data set which is a 12 band multispectral
image taken over Indiana. Four ground classes were
determined using the ground truth map: corn (2436 pixels),
soybean (2640 pixets), wheat (2365 pixels) and red clover
(2793 pixels), dimensionality was changed from I to 12 by
sequentially adding bands as described in experiment I.
w
o
=.
i
lOO _ ........ _ ....... f ......
-..P _-.r-- r • • I
,, __ i...-
i  i:i
.° ....°
t t75 1 I _ comblO0 l 1! ! -----4'---=om_soo : : :
i i i I i ! i I i i i
_s I I I _ I I I I I I I
2 3 4 $ S 7 8 9 10 11 12
dimension
Figure 2: Classification performance versus dimensionality for FLCI data.
From each class 15 training samples were drawn randomly and
the statistics of each class were estimated once using these
training samp}es alone, once using an additional 100 unlabeled
samples, and once using 500 additional unlabeled samples. The
rest of the samples were classified. Each experiment was
performed I0 times independently and the average of the ten
trials was obtained. The results are shown in Figure 2. In the
supervised learning case, the Hughes phenomenon appeared at
dimension 4, with I00 unlabeled sample it was delayed to
approximately dimension 7, and with 500 unlabeled samples
the Hughes phenomenon did not occur. The minimum error for
the supervised learning case was 2.89% at dimension 4. For the
combinedl00 and combined 500 cases it was 2.20% at
dimension 5 and 1.86% at dimension 8, respectively.
VII. Concluding Remarks
The use of unlabeled samples for mitigating the Hughes
phenomenon was investigated in this paper. Throughout tt'e
experiments it was seen that when the dimensionatity is small
(compared to the size of the training set) the estimates obtained
by waining samples alone are usually adequate for obtaining a
reasonable decision rule. In this case, additional unlabeled
samples did little to increase the accuracy, and in fact in our
experiments they sometimes reduced the accuracy slightly.
This result is most likely due to the fact that the unlabeled
samples may contain oudiers from unknown classes and/or
from boundaries of the fields. However, when the
dimensionality is high (compared to the number of training
samples), unlabeled samples can significantly improve the
accuracy. Since it is very difficult to guess the optimal number
of features related to the training sample size in advance, it
becomes important to reduce the effect of the Hughes
phenomenon. For example, recently a new feature extraction
method was proposed that is based on reducing the
dimensionality of the data at the same time that the
performance remains similar to the original space [6]. In such a
case, one would like to ensure that the performance in the
original higher dimensional space is not severely effected by
the Hughes phenomenon. Based on the material presented in
this paper, we suggest the following steps for designing
classifiers when training samples are limited:
1) If possible, estimate the Bayes error in order to have an
under'standing of the difficulty of the problem.
2) Design a classifier using the training samples alone.
3) Test the performance of the designed classifier (test
samples, resubstitution, leave-one-out, etc.).
4) If the performance of the classifier was not
satisfactory, draw a set of unlabeled samples and design a
new classifier using both training and unlabeled samples.
Test the classifier again and if necessary use more
unlabeled samples.
VIII. References
it] G.F. Hughes, "On The Mean Accuracy Of Statistical
Pattern Recognizers," IEEE Trans. In for. Theory, Vol.
IT-14, No. I, pp 55-63, 1968
[21 A.P. Dempster, N.M. Laird, D.B. Rubin, "Maximum
Likelihood Estimation from Incomplete Data via EM
Algorithm," J. R. Statist. Soc., B 39, pp 1-38, 1977.
[3] R.A. Redner, H.F. Walker, "Mixture Densities,
Maximum Likelihood and the EM Algorithm," SIAM
Review, Vo}. 26, No. 2, pp 195-239, 1984.
[4] B.M. Shahshahani , D.A. Landgrebe, "Using Partially
Labeled Data for Normal Mixture Identification with
Application to Class Definition," in Proc. IEEE
International Geoscience & Remote Sensing
Symposium, pp 1603-1605, 1992
[5] P. Hall, D.M. Titterington, "The Use of Uncategorized
Data to improve the Performance of a Nonparametric
Estimator of a Mixture Density," J.R. Statist. Soc. B 47,
pp 155-163, 1985
[6] C. Lee, D.A. Landgrebe, "Feature Extraction Based on
Decision Boundaries," IEEE Trans. Pattern Anal.
Mach. lntell., Vol 15, No 4, pp388-400, 1993
- 1537-


Use of Unlabeled Samples for Mitigating the Hughes Phenomenon

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server