Two classes of nonparametric density estimators, the histogram and the kernel estimator, both require a choice of smoothing parameter, or 'window width'. The optimum choice of this parameter is in general very difficult. An upper bound to the choices that depends only on the standard deviation of the distribution is described

Terrell, G. R.

English

NASA Technical Reports Server

  
 
 
N O T I C E 
 
THIS DOCUMENT HAS BEEN REPRODUCED FROM 
MICROFICHE. ALTHOUGH IT IS RECOGNIZED THAT 
CERTAIN PORTIONS ARE ILLEGIBLE, IT IS BEING RELEASED 
IN THE INTEREST OF MAKING AVAILABLE AS MUCH 
INFORMATION AS POSSIBLE 
https://ntrs.nasa.gov/search.jsp?R=19810003239 2020-03-21T16:09:00+00:00Z
^ :I ,	 —	 __...:	 a	 ... "	 ^ :^'•r:",^ ;aesw!n"yP!
G	 M1
LrOl.^lir/^/EEL^► 	 ^	 _	 , r
ENGINEERING AND MANAGEMENT SERVICES COMPANY, INC. ;	JSC-16669
^ 9 ^ NASA CR
Ref: 64-7947
Contract NAS 9-15800
Job Order 73-302
TECHNICAL MEMORANDUM
A BOUND FOR THE SMOOTHING PARAMETER IN CERTAIN
WELL-KNOWN NONPARAMETRIC DENSITY ESTIMATORS
By
George R. Terrell
(NkSA-CR-160657) B BOUND FOR THE S!lOOTHING	 581 -11747
PARAMETER IN CERTAIN WELL-KNONN
NONPkRAMET&IC DENSITY ESTIIIkTORS (Lockheed
Engineering and Hanagesent) 9 p	 Onclas
HC A02/11F A01	 CSCL 12A G3/65 38955
i
7
Approved By: ^I • ^^
•	 T. E. Pinter,
 Super—Mor
Techniques Development Section
p^^1980 /
,aE ^  Vic,
APRIL 1980
1830 NASA ROAD 1, Houston, Texas 77058 • TO (713) 3335411
LEMSCO-14850
A Subsidiary of Lockheed Corporation
k
l
t
I
1. Mont Na 2. Gownunent AepYion No. 3 Redpislt'a c o ft tw
SC-16669
d. Trf1/ and Mulle a Report Dew
A Bound For The Smoothing Parameter in Certain
Well-Known Nonparametric Density Estimators
April 1980
trrfatawelg 0C^ae
7. AudNW111 a. Performing Orwiation Ralsert No.
George R. Terrell LEMSCO- 14850
10. Work Unit No.
a ►arforming orgenintion Name end Addres
Lockheed Engineering and Management Services Company, Inc.
1830 NASA Road 1 11 Contract or cant No
Houston, Texas 77058 NAS 9-15800
13. Type of Report and Period Cwm)od
Technical Memorandum12. soo"ewing Agency Name and Addres
National Aeronautics and Space Administration
Lyndon B. Johnsr l Space Center 1a' S00fg0f	 Ag"'cy 0o0i
.Houston, Texas 77058
la supo mentery Notes
1a Abeam
Two classes of nonparametric density estimators, the histogram and the kernel
estimator, both require a choice of smoothing parameter, or "window width."
The optimum choice of this parameter is in general very difficult.
	
This
note proposes an upper bound to the choices that depends only on the standard
deviation of the distribution.
17. Key Words (Suggnted by Authoris ► 1 1e. Distribution Staten»nt
Density estimation, histogram, Parzen kernel,
nonparametric methods.
19. senr ity 08*f. lof this report) 20. security pasir. (of this page) 21. fro. of Pages 22. MOO*
Unclassified Unclassified 11
'For'ele by theAstional Tochnieal Information Senike, Springfield. Virginia 22161
= Rwm 1424 4114 ttev 751	 O&W -- JIC
ii
A-
,,
CONTENTS
Section	 Page
1.	 INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . 	 1
i '. NONPARAMETRIC DENSITY ESTIMATORS . . . . . . . . . . . . . . . . . . 2
I	
3. SMOOTHEST DISTRIBUTIONS . . . . . . . . . . . . . . . . . . . . . . 4
4.	 APPLICATIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . 	 5
i	 5. REFERENCES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 	 6
iii
I w
r
f	 1. INTRODUCTION
1
The most cohrionly used methods of clustering and discrimination in picture-
element (pixel) data sets from remote sensing problems involve the assumption
of normality of subsets of the data. Since the distributions are often very
complex, use of this assumption can introduce error. Alternative, so-called
nonparametric techniques may be preferable as a first step. This memorandum
suggests some simple methods for choosing an initial nonparametric description
of such data sets.
n w
2. NONPARAMETRIC DENSITY ESTIMATORS
For the reasons cited in the Introduction, it is often desirable to have an
estimate of the unknown density function f(x) of an absolutely continuous
random variable. This may be done by ascribing f(x) to a family of functions,
such as the normal family, and then estimating those parameters which distin-
guish the members of the farily from each other. If one is unwilling to
assert in advance that the underlying distribution for a data set belongs to
some parametric family, then one may use nonparametric estimators of density.
The most common of these is the histogram; but there are others such as
kernel estimators, k-th nearest neighbor estimators, and orthogonal series
estimators. We will concentrate on the first two, although similar principles
would seem to apply to the others mentioned.
The constant width histogram is obtained by partitioning the domain of the
random variable into intervals of a certain width h. Then, if x falls in
the interval (xi,xi+h),
fi(x) = nh (the number of data points which fall in (x i ,x i +h), out
of a sample of n)
Notice that the appropriate "window width" h remains to be determined. If it
is too large, features of the distribution may be obscured. If it is too
small, peculiarities of the particular sample may dominate the estimate.
Using the criterion of minimum Integrated Mean Squared Error, Scott (1979) has
determined that the asymptotically optimal value for h is
h* = (6/ff'(x)2dx)1/3n-1/3
The unknown quantity ff 
I 
W 2 dx remains to be estimated from the data. This
may be difficult for small n and time consuming for large n. Scott proposes
that for a quick choice of h, it be assumed that the distribution is normal,
giving
h  = 3.49sn-1/3
t
2
w
where s is the sample standard deviation. His empirical studies indicate
that ff 
I (X) 2 dx is often larger than for a normal distribution, that is,
many distributions of interest are less smooth than the normal.
The Parzen (1962) kernel density estimator is given by
x-x
?(x) - 
nh E k(--h !)
xi
where k is a standard density function, called the kernel, symmetric with
mean zero and variance one. h is once again called the window width, and
is a measure of how far away from each data point its influence is felt.
Once again, using the Integrated Mean Squared Error criterion, Epanechnikov
(1969) established that the optimum choice of h is
h* = n-1/5( ff If (x) 2dx) -1/5 (/k2(x)dx)1/5
He further established the optimum choice for k, and also noted that the
choice of kernel makes comparatively little difference to the integrated
'	 mean squared error. We cnce again face the problem of estimating a measure
 2
of smoothness of the underlying distribution, ff (x) dx, with only the data
to help us. This involves considerable computational and statistical
difficulty. We could follow the lead of the previous section and assume
that a reasonable value of h would be found by assuming the data is normal.
Thus, using the uniform kernel, we get
h * = 1.06412sn-1/5
n
This may be a good starting point for the exploratory adjustment of h. The
arbitrary nature of the choice is still disquieting.
3
it
)
,
.,^.	 ...
3. SMOOTHEST DISTRIBUTIONS
We will proceed to show that, under certain mild restrictions, there exists
a smoothest distribution from the point of view of the histogram and a
smoothest distribution from the point of view of the kernel estimators.
These will immediately provide us with an upper bound to the choice of
smoothing parameter h.
Consider distributions f with two continuous derivations on the whole real
line. We will construct an f with fixed variance that has minimum
ff I (x) 2 dx. We will use a variational argument very similar to that used by
Epanechnikov (1969) in constructing the optimal kernel. Assuming f has
variance one, we will c nsider a slightly varied distribution f+6 such
that = 0
 f6 = 0 x26 = 0. Thus fi f1+61)2 - fi'2 will be negligible
of
compared to 6, so J' 6 =0. Integrating by parts, this becomes ff 6 = 0.
Since 6 is arbitrary up to the given restrictions, f must be a quadratic
polynomial, and so f is a quartic polynomial. By an argument very close'to
Epanechnikov's, the distribution with minimal ff'(x)2 dx is
f(x)(16 (1-x')' on L-1,11
0 elsewhere
up to an affine change of variables. It may be noted thatff(x) 2dx for
the unit normal distribution is .14105 whereas for the smoothest distribution
with variance one it is .1151.
With the criterion ff If ( X )(x) dx for smoothness in the expression for the
optimal kernel width we can do a similar computation. Among all distributions
with four continuous derivatives, there is one for which f"	 2(x) dx is
minimal given a fixed variance. We can derive it just as above, to get
f(x) - I 32 (1-x2 ) 3 on L-1,1J
l 0 elsewhere
up to an affine change of variables. ff
o
	
2(x) dx for the unit normal is
.21157 and for the minimum valued function with variance one it is .14403.
4
4. APPLICATIONS
We can now construct an upper bound to the bin width h for a histogram in
terms of the sample standard deviation s.
hmax ° 3.71sn-1/3
Any histogram with bin width much wider thin this is presumably oversmoothed.
Notice that this is only seven percent wider than the normal optimum proposed
by Scott (1979). According to his results on sensitivity to optimum width,
the maximum width histogram would only be one half of one percent larger
than the optimum in integrated mean squared error. Thus, the maximum width
histogram is a good starting point for graphical and iterative methods of
seeking optimal representation. It carries with it the additional reassurance
that adjustments in bin width need proceed in only one direction, toward
smaller values.
Similarly, the upper bound for the optimal choice of smoothing parameter h
for a kernel density estimate is
hmax	 1.15sn-1/5
using the sample standard deviation, the uniform kernel, and fi 2(x) dx for
our smoothest density. Again, it is only slightly larger than the optimal
value for a normal distribution, and so is a plausible initial value in the
downward search for the optimum of an unknown density.
Clearly this variational approach may be generalized to other density
estimation procedures in which the optimum value of the smoothing parameter
depends on soar related measure of density smoothness. For example, Terrell
and Scott (1980) propose an estimator with a higher rate of convergence than
a Parzen estimator whose optimal ;;moothing parameter depends upon the
second and fourth derivative of the density. It should be noted that our
density-free choices for smoothing parameter may be less useful in the case
of higher-order methods than the kernel method because of the increasing
sensitivity of such methods to deviation from the optimal window width.
1
5
5. REFERENCES
1. Epanechnikov, V. A.: Nonparametric Estimates of a Multivariate
Probabili^y Density.	 Theory of Probability and Its Applications,
14, pp. 153-158, 1969.
It
	
2. Parzen, E.: On The Estimation of Probability Density and Mode.
Annals of Mathematical Statistics, 33, pp. 1065-1070, 1962.
3. Scott, D. W.: On Optimal and Data iiasud Histograms. Biometrika, 66,
pp. 605-610, 1919.
4. Terrell, G. R.; and Scott, D. W.: On Improving Convergence Rates For
Nonnegative Kernel Density Ls timators. Annals of Statistics (to appear),
1980.
U


A bound for the smoothing parameter in certain well-known nonparametric density estimators

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server