In previous work the authors (Phys. Rev. Let., vol.65, p.945-8, 1990) proposed the concept of deterministic annealing for the problem of clustering and vector quantization. This approach is summarized. The authors extend the clustering method to the constraint clustering method. Adding constraints to the deterministic annealing mechanism expands the variety of optimization problems which can be solved by this method. A brief presentation of the clustering approach is given. Two examples to which the constraint clustering approach can be applied are included

Fox, G. C.

Gurewitz, E.

Rose, K.

Caltech Authors - Main

Deterministic Annealing, Constrained Clustering, and Opthiieation. 
K. Rose t , E. Gurewitz,, and G. C. FoxS 
Caltech Concurrent Computation Program 
California Institute of Technology 
J. Introduction 
Clustering is usually formulated as an optimization problem by defining a cost function to be minimized. 
Most of these cost functions are nonconvex and have several local minima Traditional techniques [1,2,3,4] are 
essentially decent algorithms, as the cost is reduced at each iteration. Therefore, they tend to get trapped in local 
minima. 
Simulated annealing, or stochastic relaxation [5], is a known technique for avoiding local minima of nonconvex 
optimization problems. However, the process to escape local minima requires very slow schedules [6] which are 
not realistic for many practical applications. 
In a previous work [7,8] we proposed the concept of deterministic annealing for the problem of clustering 
and vector quantization. Our approach is strongly motivated by the physical analogy and baeed on principals of 
statistical mechanics or information theory [9,10]. In this paper we extend our clustering method to the constraint 
clustering method. Sections 2 and 3 are a brief presentation of our clustering approach. Section 4 presents the 
”constraint clustering approach”, and sections 5 and 6 are two examples for which this approach can be applied. 
2. Clustering by Deterministic Annealing 
In previous works [7,8] we have suggested a deterministic annealing approach to clustering and vector quan- 
tization. This approach is briefly summarized in this section and the following one. 
The method uses a fuzzy formulation where each data point z is associated in probability with each cluster 
Cj. The cluster C, is represented by its ”cluster centroid” yj. The energy (cost or distortion) of associating the 
data point z to the cluster C, is d ( z , y , ) .  If the set Y = {yj} of the clusters representatives is given, the expected 
energy of the system is 
where P(z E Cj) is the probability that the data point z belongs to the cluster Cj. Since we do nof have any 
prior knowledge about the data probability distribution function, we apply the principle of maximum entropy. 
As is well known, the probability distribution which maximizes the entropy under the expectation constraint (1) 
is 
e-Dd(z,Vj) 
P(2 E Cj) = ~ 
z x  ’ (2) 
where Zx is the partition function Z, = cj e-pd(zJ’J). 
The Lagrange multiplier p is determined by a given value of E in (l), and is inversely proportional to the 
”temperature”. For a given set of cluster representatives {y,}, it is assumed that the probabilities relating different 
data point to their clusters are independent. Hence the total partition function is Z = n, Z,, and the free energy 
F is given by 
F = --logz(Y) = - C l o g  Ce-@d(Z,Yj) (3). 1 1 
B ’ z  j 
t Department of Electrical and Computer Engineering, University of California Santa Barbara. 
* Physics Department, Nuclear Research Center Negev, Israel 
$ University of Syracuse. 
CH 3065-0/91113000-2515 $1.00 OIEEE 
Instead of considering the association probability of a data point, we consider the probability of an entire 
instance. An instance is given by a set of representatives Y = {yj} and every data point is associated, in each 
instance, with one and only one representative. The energy of an instance is given by: 
Where V = {u,j} is the set of associations, u,j = 1 if z E Cj ,  and u,j = 0 for all the other clusters. In another 
study we are showing [ll] that for a given p the most probable instance is the one with the minimal energy D, 
and the set of the most probable representatives is the one with the minimal free energy. Thus, on one hand we 
have the energy D, which is minimized to obtain the optimal, hard clustering solution for the training set. On 
the other hand we have the free energy F which is minimized, at a given p, to obtain the set of the most probable 
representatives. Moreover, for p 4 03, both D and F are minimized by the same set Y. 
3. Deterministic Annealing and Phase ‘Pransitions 
The set Y that optimizes the free energy satisfies: 
Where we have used a shorthand notation for differentiation with respect to each component of y j  separately. 
After normalization (5) can be written as: 
a 
ayj 
(-42,Yj)) j  = 0. 
At p = 0 the association probabilities are uniform and 
This is the centroid condition for the entire data set viewed as one cluster. As proved in [ll] there is a unique 
solution to this equation. Thus at p = 0 our effective function is convex, and all the representatives converge 
to the same point, which is the global minimum. We shall interpret identical representatives as representing the 
same cluster and consider the set Y of representatives without repetitions as the set of natural clusters. Thus, at 
p = 0 we have one natural cluster. 
The single solution of (5) at p = 0 will be a solution for all /3 > 0, but at some positive p it will change 
from stable solution (local minimum) to unstable solution (saddle point). At this point the system can gain 
energy by splitting into subgroups of representatives. Each subgroup defines a newly formed natural cluster. 
Hence, the system undergoes a phase transition. This process results in a natural hierarchy of clustering solution. 
Recalling that /3 is the Lagrange multiplier related to the average energy (distortion), we have clustering solutions 
at decreasing levels of average energy. It can be regarded as clustering at different scales. 
An impotent example of a distortion measure is the v-th law dy(z,y) = xi Iz(i) - y(i)ly. The squared 
distance distortion, v = 2, is the most extensively used distortion measure. For this measure, we have shown 
[7,8] that the first transition occurs when (I - 2pC,,)Yx = 0, where C,, is the covariance matrix of the data, 
I is the identity matrix, and Yx is an eigenvector of CZz. The critical p is thus: pc = (2AmoZ)-’, where A,,, 
is the largest eigenvalue of C,,. Hence, the critical temperature is determined by the variance along the largest 
principal axis of the distortion. The split will be initialized in the direction of this principal axis. 
4. Constrained Clustering 
So far we did not put any constraint on the clusters’ representatives. It was only implicitly assumed that 
there are enough representatives to enable phase transitions. Adding constraints to the deterministic annealing 
mechanism expands the variety of optimization problems which can be solved by this method. 
2516 
.-- 
There is a large family of optimization problems which may be viewed as looking for the optimal association 
between two sets, one set of variables and one set of fixed data. In the clustering problem these two sets are 
the set of data point and the set of clusters representatives. Optimal association, in this regard, means that an 
association cost function is defined, and has to be minimized. 
Let us introduce constrains into our clustering formulation. In addition to the constraint of the average energy 
(D(Y, V)) = E, there is a constraint concerning only the cluster representatives (T(Y)) = L. The probability 
distribution which maximizes the entropy under these constraints is 
e-BD(Y,V)-XT(Y) 
Cy,,ve e-BD(Y‘,V‘)-XT(Y’) P ( Y , V )  = 
By summing over all possible associations V we obtain the marginal probability 
e-BF(Y,B)-XT(Y) 
Cy ,  e-BF(Y’,B)-AT(Y) ’ P ( Y )  = 
where F is given in (3). The most probable Y is the one which minimizes PF+AT. It can be viewed as minimizing 
F subjected to the constraint T(Y) = L’ (L’ = L/p). Hence, our procedure for constrained clustering is to 
minimize the free energy subjected to the appropriate constraint, or equivalently to minimize the Lagrangian: 
F’ = F + qT where q = A/p (6) 
5. Mass-Constrained Clustering 
In our clustering method the location of the clusters’ representatives depends on their number. Therefore, 
Let Ab denote the multiplicity of representatives in the k-th cluster. Then Z, = cj e-@d(zl!’j), is rewritten 
we introduce a constraint taking into account the mass, or the population of a natural cluster. 
as Z, = ck Ake-Bd(z ,v*) ,  the free energy (3) is now 
The free energy is to be minimized under the constraint of a fixed number of representatives. The Lagrangian to 
be minimized is thus 
F’ = F + q ( c A k  - M). 
k 
We do not require Ak to be integers, and M is the total mass of the natural clusters. The set {A,} which minimize 
F’ satisfies 
which yields 
Multiplying by the appropriate and summing over all natural clusters 
2517 
which by the mass constraint yields qp = N/M = l/p, where N is the number of data points and p is the mass 
of a data point. Fkom the last equation we get 
Note that although p is constant, it  could be made dependent on z to generalize the method to cases where the 
given data points do not have equal weights. 
In the mass-constrained formulation the process is independent of the number of representatives, as long as 
this number is greater than the number of natural clusters. Let the natural clusters be represented by the clusters 
centroids {yj} and masses { X i } .  The j-th natural cluster Cj is represented by its centroid yj and its mass X j .  If 
this natural cluster is represented by more than one centroid, i.e. mj and Cj = UkC,!k),  it  is represented by the 
two sets {y;”} and {Ay)}, k = 1, ..., m, and Ay) = Xj. From the definition of Zx it  is clear that Zx is 
invariant to any division of the natural cluster into clusters. Furthermore, the probability of associating a data 
point z to a natural cluster does not change 
It is therefore clear that these. representatives will satisfy (5), and will thus be obtained by our method regardless 
of the multiplicity m, or the mass division {Ay)}.  
6. The Travelling Salesman Problem 
We shall derive here the Elastic Net [12,13] approach to the travelling salesman problem (TSP) as an 
application of our constraint clustering procedure. 
In the deterministic annealing clustering algorithm, if we throw in enough representatives and let p + 00, 
then each data point becomes a natural cluster. This can be viewed as a process of mapping a set of data points 
to a set of representatives. 
The TSP is: Given a set of data points (cities), find the shortest close path which passes through all of them. 
Actually, like in the Elastic Net approach, instead of minimizing the length we shall minimize the sum L of the 
squared distances between consecutive cities on the path (L). Hence, we add the constraint to the free energy to 
obtain the Lagrangian (6) 
n 
F’ = F + A(C IYk - Yk-1I2 - L), YO = Yn. 
k l  
The optimal Y mast satisfy the condition &F‘ = 0, for all the j’s, which by (5) yields 
If we choose the square distance as our clustering distortion measure d(z,y), we obtain the EN formulation for 
the optimum. 
CP(Z E Cj)(yj - z) + ~ ( 2 y j  - yj+l -  yj-1) = 0. 
2 
This equation depends on p through the association probabilities P. When /3 -+ 00, each representative converge 
to one city. 
An important question at this point is how X should be varied with p. In [12] the relation was A IX 1 / d ,  
while in [14] it  was kept constant. 
The procedure suggested here is as follows: a) For a given 8, gradually increase L and optimize, until 
L reaches some appropriate value below the free (of constraint) length. b) Keeping L constant,update p and 
optimize, return to (a). Such an approach can be implemented directly using methods for nonlinear optimization, 
like for example Generalized Hopfield Networks. 
It is more convenient to control the Lagrange multiplier X rather than the length L. It was shown [ll] that 
one can use the following approximation: 
where AE/AL may be estimated using the last two iterations in A, before the moment to update p. 
Concluding €&marks 
Constrained clustering can be used to improve the deterministic annealing method for clustering. Moreover, 
it allows applying annealing to various other optimization problems which have a data association aspect, and can 
be formulated as constrained clustering. Annealing is obtained by gradually making the association less fuzzy, 
and helps avoiding local minima. 
References 
[l] G. Ball and D. Hall, ” a  clustering technique for summarizing multivariate data”,” Behavioral Science, vol. 
[2] J .  C. Dun ,  “ a  fuzzy relative of the isodata process and its use in detecting compact well-separated clusters,” 
[3] J. C. Bezdek, “A convergence theorem for the fuzzy isodata clustering algorithms,” IEEE hnsact ions  on 
12, pp. 153-155, 1967. 
J. Cybm, vol. 3, pp. 32-57,1974. 
Pattern Analysis and Machine Intelligence, vol. PAMI-2, pp. 1-8, 1980. 
Communications, vd. COM-28, pp. 8495 ,  1980. 
[4] Y. Linde, A. Buzo, and R. M. Gray, ”an algorithm for vector quantizer design,” IEEE Transactions on 
[5] S. Kirkpetrick, C. D. Gelatt, and M. P. Vecchi, ”Optimization by simulated annealing,” Science, vol. 220, 
[6] S. Giman and D. Giman, ”Stochastic relaxation, Gibbs distribution, and the Bayesian restoration of images,” 
[7] K. Rose, E. Gurewitz, and G. C. Fox, ”Statistical mechanics and phase transition in clustering,” Physical 
[8] K. Rose, E. Gurewitz, and G. C. Fox, ”Vector quantization by deterministic annealing,” to be published. 
[9] E. T .  Jaynes, “Information theory and statistical mechanics,” in Papers on probability, statistics and statistical 
pp. 671-680, 1983. 
IEEE hnsact ions  on Pattern Analysis and Machine Intelligence, vol. PAMI-6, pp. 721-741, 1984. 
Review Letfers, vol. 65, pp. 945-948, 1990. 
physics (R. D. Rosenkrantz, ed.), Dordrecht, The Netherlands: Kluer Academic Publishers, 1989. 
Illinois Press, 1949. 
nology, Pasadena California, 1991. 
method,” N a i m ,  vol. 326, pp. 689-691, 1987. 
[lo] C. Shannon and W. weaver, The mafhematical theory of communication. Urbana, Ill.: The University of 
[ll] K. Rose, Deterministic annealing, clustering and Optimization. Ph.D. Thesis, California Institute of Tech- 
[12] R. Durbin and D. Willshsw, ”An analogue approach to the travelling salesman problem using an elastic net 
2519 
1131 R. Durbin R. Szcliski, and A. L. Yuille, "An analysis of the elastic net approach to the travelling salesman 
[I41 A .  I,. Yuille, "genernlixed dcformnble riiodels, stalislical j>hysiur, and matching problem," Neural Compula- 
problem," Neural Computation, vol. 1, pp. 348-358, 1989. 
Iton, vol .  2, pp. 1-24, 1990. 
\ \  .. 
FIG. 1 Clustering at  different phases. The data is generated from four Gaussian distribution centered at 
the locations marked by x. The calculated centers are marked by 0 .  
FIG. 2 The effect of cluster mass (population) at  inlermediate p. The data is sampled from two normal 
distributions whose centers are niarked by x. The computed representatives are marked by 0.  
.FIG. 3 A constrained cruscering reqult for a 50 cities TSP. 


Deterministic annealing, constrained clustering, and optimization

English

https://authors.library.caltech.edu/78353/1/00170767.pdf

Deterministic annealing, constrained clustering, and optimization

Abstract

Similar works

Full text

Available Versions

Caltech Authors - Main

Caltech Authors - Main