We present a result about stochastic boundedness of stable empirical processes on Vapnik-Cervonenkis classes of functions and we apply it to obtain a rate of convergence for the approximation between the sample and the populational variation in the k-centroids problem in clustering analysis.Clustering analysis; K-centroids; Empirical processes; Vapnik-Cervonenkis classes of functions;

Romo, Juan

English

Research Papers in Economics

Working Paper 92-36  Divisi6n de Economfa 
September 1992  Universidad Carlos ID de Madrid 
Calle Madrid,  126 
28903 Getafe (Spain) 
Fax (341) 624 9849 
A RATE OF CONVERGENCE IN CLUSTERING ANALYSIS 
Juan Romo· 
C 
r, 
'-J 
Abstract ---------------------------
We present a result about  stochastic  boundedness of stable empirical processes on 
Vapnik-Cervonenkis classes of functions and we apply it to obtain a rate of convergence for 
the  approximation  between  the  sample  and  the  populational  variation  in  the  k-centroids 
problem in clustering analysis. 
c 
Key words: 
Clustering  analysis; 
functions. 
k-centroids;  empirical  processes;  Vapnik-Cervonenkis  classes  of 
c 
·Departamento de Estadfstica y Econometrfa, Universidad Carlos ITI de Madrid, Spain. 1  INTRODUCTION 
The k-means  method  in  clustering  analysis  to partition n  points into k 
groups can be described in  the following  way:  given  points Xl, ••• ,Xn  in nd 
we find  centers bl , ..• ,ble  minimizing 
and then we assign Xi to the closest center  (11  .  11  is the euclidean norm).  The 
name of  the criterion is  due  to  the fact  that each  center b j  is  the mean of 
the points Xi in its cluster. This problem was already consider by W.D. Fisher 
(1958).  MacQueen (1967) presented the more general case ofk-centroids: given 
points Xll' .. ,Xn  in n
d
,  to find  centers al, ...  ,ale minimizing 
n 
~1~<?1e  11  Xi - aj liP,  1~ p ~  2. 
1=1  _3­
The statistical version of this problem is  as  follows:  let P be a probability 
measure on (nd, 8nd)  and let {Xn  : n E N} be a sequence ofindependent and 
identically distributed random variables with law P.  Let Pn  be the empirical 
measure associated to Xl, ... ,Xn .  For each set a =a(k) ={aI, ... ,ale}  C nd, 
we define 
fp(x,a) = min  11  X  - aj liP,  1 ~ p ~  2. 
l5,i9
 
Let
 
Pfp("a) ==  [  m~n  11  X  - aj  liP  dP(x) =I:
le 
[  11  X -aj liP  dP(x),
Jnd  l5,35,1e  j=lJAj 
where Aj is  the set of points closest to aj than to any other center.  Also,  we 
have 
1 n 
p.  ( (.  a) =- '"' min  11  X· - a' liP. nJp  ,  ~l< '<le  1  3  n  1=1  _3­
Let a = a(k)  = {all"" ad be the k-centroid minimizing Pfp("  a)  over all 
sets a C nd  having at most k  points, i.e., 
(1.1) 
We will  assume that a is  unique.  Let a(n) = a(n)(k) =  {a~n), ...  ,a~n)} be the 
corresponding sample value, i.e., 
c  (1.2) 
2 In clustering analysis is interesting to establish the convergence of the sam­
ple clusters to the populational ones and the convergence of the sample varia­
tion PnIp(·, a(n)) to PI p(·, a). The convergence of the sequence of sets must be 
understood in the sense of the Hausdorff's metric:  if A and Bare nonempty 
compact subsets of nd,  the Hausdorff metric is 
dH(A, B) =max{sup d(x, B), sup d(y, An, 
~EA  ~EB 
where d(x, C) = infllEc  11  x -y " for any Cc nd•  So dH(A, B) < b if, and only 
if, for any point in A there exists a point in B such that the distance between 
them is  less than b,  and conversely.  If  follows  that if A has k different points 
and b is  less  than half of the minimum distance between points in  A, B  is  a 
set with at most k points and such that dH(A, B) < b,  then B has to contain 
exactly k  different  points each of which  lies within a distance b of a unique 
point in A.  It follows  that the sequence of sets {a(n)  : n  E N} converges to 
the set a  if for  every n  there exists a labelling  (a~n), . .. , akn))  for  the points 
in a(n)  and a labelling (ab . .. ,ak) for  the points in a such that a~n)  ~ aj for 
1 =::;  i  =::;  k. 
Pollard (1981)  proved that if Ind  11  x  lid  dP(x) < 00, 1  =::;  p  =::;  2,  and for 
each j  EN, 1 =::;  j  =::;  k,  there exists a unique a(j) satisfying (1.1) then 
a(n)  ~  a almost surely as  n ~ 00  (1.3) 
and 
PnIp(·,a(n)) ~ PIp(·,a) almost surely as  n ~  00.  (1.4) 
This improves the result of Sverdrup-Thygeson (1981) who obtained the same 
conclusion assuming compactness of the sample space. 
For p = 2, Pollard (1982) established asymptotic normality of n 1/ 2(a(n) -a) 
assuming that Ind  11  x  11 2  dP(x) < 00, that P is  absolutely continuous and 
under conditions ensuring that the second order differential of the map a  ~ 
PICa) is  positive definite.  For p = 1 and d = 1,  Butler (1986)  has  shown 
asymptotic normality of n1/ 2(a(n)  - a)  assuming Ind  11  x  11  dP(x) < 00 and 
positive definite second order derivative.  See  also  Butler (1988)  for  central 
limit results for universally optimal and bounded locally optimal theory and 
for  techniques for assesing the number of groups in the data. 
Our main theorem will follow  from a result about stochastic boundedness 
for empirical processes on Vapnik-Cervonenkis classes of functions; so we recall 
some notation from the theory of empirical processes.  For a probability space 
(5, $, P),  Lp  (5, $, P),  0  < p  <  00,  will  be  the class  of  real  measurable 
functions I  on 5  such that Is II  IP  dP < 00. If I  E L1 (5, $, P), we will write 
PI instead of Is IdP. If;:  is a collection of measurable functions on 5, we will 
3 
'). 
.J. 
J 
.j 
) assume that the envolope of F, P(s) =sUPfE.1" 1f(s) I,  is finite for all  s E 5. 
Although P  is  not necessarily measurable, there always exists a  measurable 
F*  : 5  --. R  (see Dudley (1984), Theorem 3.1.1) such that P  ~  P*  and for all 
measurable functions h such that P  ~  h,  we have that P*  ~  h almost surely 
with respect to P.  For 1 ~ P ~  2,  we will write 
nEN. 
Since P  is  finite for  all  s  E  5, we  will  have that for  all  n  E N  and for  all 
pE [1,2], 
n 
{1I!!')(f): f  E F} =  {n-l/p~)f(Xi) - PI): f  E F} 
i=l 
is astochastic process with sample paths in the space l
OO (F) of all real bounded 
...... )	  functions  defined on F.  If H  is  a  real bounded function on F, we  will  put 
II  H  11.1"=  sUPfE.1" I H(f) I.  We say that F  is  supremum measurable (for P) if 
sup 1 Q{f(Xn )) : n EN} I 
fE.1" 
is p.N_ completion measurable for each function Qwhich is a linear or quadratic 
function of finitely many of the f(Xn )  (see Alexander (1987)). 
Let T be a non-empty set and let ACT. If  C is a class of subsets of T, we 
will write 
and 
mC(n) = max{.6c(A): A C T,UA =n},  nE N. 
Define  the  (Vapnik-Cervonenkis)  index  of C as  V(C)  = min{n EN: 
mC(n)  < 2n }  or V(C)  ==  +00 if mC(n)  = 2n  for all n  E N.  We  say that C 
is a  Vapnik-Cervonenkis class ofsets if V(C) < 00. It f  is a real function on 5, 
its graph is G(f) =  ((s,t) E 5  x R  : 0 ~ t ~  f(s) or f(8)  ~ t  ~  O}  . Fe R S 
is  a  Vapnik- Cervonenkis  class  of functions if {G(f)  : f  E F} is  a  Vapnik­
Cervonenkis class of sets in 5  x R. 
We  will  denote by N(c,p,F) the minimum number of balls  with radius 
e > 0 for the metric p and centers in F  needed to cover F. For any finite set 
T C 5, let 
and define o 
N(2)(c,F) =  sup  N(c,d¥),F). 
TES,T  finite 
4 
c J
 
If {Z(t) : t E T} is a subgaussian process, Marcus and Pisier (1978) proved 
that if 
{ITler
1  (logN(c,0',T)?/2dc < 00  J 
0 
then Z  has  a  version  Z with O'-uniformly  continuous sample paths and, for 
to  E T, 
N - Esup 1Z(t) I:::;  E I Z(to) I+c [llTler (logN(c,O',T))1/2dc + <p(1  T  lu)1 ,
) 
tET  0 
where  <p(6)  = 46(loglog4 I T  \u  6-1)1/2,0  < 6:::;1  T  lu  and I T  lu  is  the
 
diameter of T  for  0'  . If Z(t) =L:i:l cihi(t) with hi  continuous in (T, 0')  then 
)
 Z=Z.
 
2  RESULTS 
Our main result (Theorem 3)  gives a rate of convergence for  (1.4)  in the 
case p  = 1;  its proof relies on  a  technical lemma on  the differentiability of 
the application a .......  Pf(', a),  where f(x,a) =minl<j<k  11  x - aj  11,  and on a 
proposition about stochastic boundedness of the empirical process.  First, we 
present these results. 
) 
Lemma 1.  Let P be absolutely continuous with respect to Lebesgue measure 
on R d and such that fnd  11  x  W)  dP(x) < 00, 1 :::; p:::;  2.  The map a .......  f(', a) 
from Rkd  into ['p(Rd, Bnd, P)  is  11  .  II-differentiable.  As  a  consequence,  the 
map a .......  Pf (" a)  is differentiable. 
) 
Proof. We can write 
f(x, a + h) =f(x,a) + h' . 6(x, a)+  11  h  11  r(x, a, h)  (2.5) 
where 
and 
r(x, a, h)  -+ 0  as  h -+ 0 
for  P-almost every x  (in fact, except for x =aj, j = 1, ... , k).  Hence 
1r(x,a, h)  I:::;  [1~~l/11  x - aj - h 11- 11  x - aj  11  + I hi. 6(x,a) I]  11  h 11-1 
_3_ 
5  ) c 
c 
k 
~II  h 11-
1  ~):II  X  - aj -	 hj  11 - 11  x  - aj 11)+  11  ~(x,a)  11 
j=1 
o 
k 
.L(II x - aj - hj  11
2
-	 11  X  - aj  11 
2
) +1 
j=1 
~II  h 11-
1  (m~n  11  X  - aj - hj  11 +  11  X  - aj 11) t (11  hj  11 
2 +2hj (x - aj)) + 1 I<)<k	  . 
- -	 )=1 
~ C (1+t x ­ 11  Gj 11)  E l!('R',8,F). 
for small enough h.  Then 
Ilr(',a,h)llp-+O  as  h-+O	  (2.6) 
and the map a 1-+ P f(·, a)  is  I1  . lip-differentiable.  It follows  that 
11  r(·, a, h)  Ih-+ 0  as  h -+ 0; 
(2.5) and dominated convergence prove that the map a 1-+ Pf(·,a) is differen­
tiable with derivative P ~ (·,a).  This finishes the proof. 
Now  we establish a sufficient  condition for  stochastic boundedness of the 
stable empirical process. 
Proposition  1.  Let:F C  .c1(S,S,P)  be  a  supremum measurable Vapnik­
Cervonenkis class of functions with envelope P finite everywhere.  Let 1 < P < 
2.	 If 
SUPt>otPP{P* > t} = K  < 00 
then {II n-;  2::'=1 (f(Xi)	 - PJ)  II:F: n E N} is stochastically bounded. 
Proof.  From Andersen, Gine and Zinn  (1988),  it is enough to show that for 
some 6 > 0, 
n 
lim  limsupPr*{1I n-; Led!  1  II:F> M} = O.  (2.7) o	  M-oo  n-oo  i=1  {F·S6n  P (Xi) 
6 Let  0 = 1 and define :Fn = {fI{F":5n1/1'}}  and Fn = FI{F.:5n1 /1'}  . 
Xll ••• ,Xn  fixed,  consider the subgaussian process 
and its associated [,2 distance 
1 
u(J,g) = (l:~l  (~~pg)2(Xi))  2 • 
The diameter of :Fn  for  u  is 
1 
I:Fn lu~  2 (l:~ln~~2(Xi))  =Dn(F). 2 
Let Tn  =  {Xll ... ,X n }.  Since 
we get 
Now, 
n 
Pr*{11  n-; Le:i!(Xi )  lI.1"n>  M} 
i=l 
< E  [l/\.l....E  l:~l  e:i!(Xi )  11  ] - x  M  C
11  n1/ p  .1"n
 
(by Marcus and Pisier (1978))
 
C  [ [Dn(F)
  1  1]
~ ;;Ex  lo  (log N(e:, u, :Fn)) 2de: +4(1og log 4)2 Dn(F) 
7
 
J,
I 
J. 
I For each 
) 
) 
) 
) 
\J 
) 
') Since F is  a Vapnik-Cervonenkis class, the integral between brackets is  finite; 
so  to obtain (2.7), it is enough to show that 
·  l'  Ex Dn(F)  1tm  tmsup  M  =  O. 
M-oo  n-oo 
)  This will follow  if we show that Ex Dn(F) is bounded in n;  but 
ExD~(F) =2nEx  [~::I{F"~nl/P}l 
[00  F* 
=4 Jo  ntPr{nlh,I{F'~nl/P}  > t}dt 
:5 41
1 
ntPr{F* > tn
1
/ P}dt 
< 21
1 
I<  t1- Pdt = 2I<p  < 00 
- P 2 -p  '  o
 
because 1 < p < 2,  and the result follows.
 
Our final result gives the rate of convergence for the approximation of the 
sample variation to the populational one. 
Theorem 3. Let P be absolutely continuous with respect to Lebesgue measure 
on R d  and such that fnd  11  x  liP dP(x) < 00,  1 :5 p:5 2.  Then 
in  probability. 
Proof.  If a = {a}, ... ,an }  minimizes (1.1)  (for p=l), consider the class of 
functions 
{r(',a,a - a): a E U(a)} 
where U(a) is  a neighborhood of a such that 
k 
Ir(x,a,a - a) 1:5 C(l +L  11  x - aj  11) 
i=1 
for all x  E R and all a E U(a).  Let Bi be the set of points which are closer to 
(Xi  than to any other of the elements of a. Then 
o 
r(x,a,a - a) =11  a - a  11-
1 (f(x,a) - f(x, a) - (a - a)' l:::. (x,a)) = 
8 By Problem H.  28  in Pollard (1984), the class 
:F= {r(·,a,a-a) :aEU(a)} 
verifies the conditions of Proposition 1 (since the envelope 
k 
F(x) =C(l +:L 11  x-ai 11)  E £P 
i=l 
because fnd  11  x  liP dP(x) < 00)  ; so  Proposition 1 proves that {v![)  : n EN'} 
is stochastically bounded in  lOO(:F),  i.e., 
lim  lim sup Pr*  {II  v~)  11.1"> M} =O. 
M .....oo  n .....oo 
From (2.5), with a =a  and a +h =a(n), we get 
(2.8) 
) 
!(x,a(n)) =!(x,a)+(a(n)-a)' .6(x,a)+ 11 
Integrating with respect to v![), we obtain 
a(n)-a 11  r(x"a,a(n)-a).  (2.9) 
) 
The theorem of Pollard (1981)  proves that a(n)  -+ a  almost surely.  Hence 
(2.8) implies that 
m  probability. 
By the law  of large numbers in finite dimensions, the second term in the right 
hand side of (2.5) also tends to zero in probability.  Then, from (2.5) it follows 
that 
:J 
I.e. 
m  probability,  J 
9 
.) 
) in probability.  By  the weak law  of large numbers in 1(,d  and since the map 
a ~ Pf(·,a) is continuous, we obtain 
n1-;(Pnf(·,a(n)) - Pf(·,a)) --+ 0  m  probability. 
This concludes the proof. 
10 J, 
REFERENCES 
ALEXANDER, K. S.  (1987).  The central limit theorem for empirical processes 
on Vapnik- Cervonenkis classes of functions.  Ann. Probab.,  15,  178-203. 
ANDERSEN, N. T., GINE, E. and ZINN, J. (1988).  The central limit theorem 
for  empirical processes  under local  conditions:  the case  of Radon infinitely 
divisible limits without Gaussian component.  Trans.  Amer. Math.  Soc., 308, 
603- 635. 
BUTLER, R.W. (1986).  Optimal stratification and clustering on the line using 
the L1-norm.  J.  Mult.  Analysis 19,  142-155. 
BUTLER, R. W. (1988).  Optimal clustering on the real line.  J.  Mult.  Analysis 
24,  88-108. 
DUDLEY, R.M. (1984).  A course on empirical processes.  Leet.  Notes in Math. 
Springer-Verlag. 
) 
FISHER, W. D.  (1958).  On grouping for  maximum homogeneity. 
Statis.  Assoc.  53, 789-798. 
J.  Amer. 
MACQUEEN, J.B. (1967).  Some methods for  classification and analysis for 
multivariate observations.  Proc.  Fifth Berk.  Symp.  Math.  Stat.  Prob.  281-
296. 
MARCUS, M.  R. and PISIER, G.  (1978).  Random Fourier Series with Appli-
cations to Harmonic Analysis.  Princeton U.  Press, Princeton, New Jersey. 
) 
POLLARD, D. (1981).  Strong consistency ofk-means clustering.  Ann. Statist. 
9,  135-140. 
J 
POLLARD, D.  (1982).  A central limit theorem for k-means clustering.  Ann. 
Probab.,  10, 919- 926. 
POLLARD, D.  (1984).  Convergence of Stochastic Processes.  Springer-Verlag, 
Berlin. 
) 
SVERDRUP- THYGESON, H.  (1981).  Strong law of large numbers for mea-
sures of central tendency and dispersion of random variables in compact spaces. 
Ann. Statist.  9,  141-145. 
')I 
11 
). 

A central limit theorem for k-means clustering.

A course on empirical processes.

Convergence of Stochastic Processes.

On grouping for maximum homogeneity.

Optimal clustering on the real line.

Optimal stratification and clustering on the line using the L1-norm.

Random Fourier Series with Applications to Harmonic Analysis.

Some methods for classification and analysis for multivariate observations.

Strong consistency ofk-means clustering.

Strong law of large numbers for measures of central tendency and dispersion of random variables in compact spaces.

The central limit theorem for empirical processes on Vapnik- Cervonenkis classes of functions.

The central limit theorem for empirical processes under local conditions: the case of Radon infinitely divisible limits without Gaussian component.

A rate of convergence in clustering analysis.

http://e-archivo.uc3m.es/bitstream/10016/2884/1/we9236.pdf

A rate of convergence in clustering analysis.

Abstract

Similar works

Full text

Available Versions

Research Papers in Economics