Boltzmann Machines and the EM algorithm

Abstract

In this paper we formulate the Expectation Maximization (EM) algorithm for Boltzmann Machines and we prove that the Kullback distance is a Lyaponov function for the EM algorithm. As a result the EM algorithm yields the same solutions as the original learning rule of Ackley, Hinton and Sejnowski. We give an example of the EM algorithm applied to a special class of Boltzmann Machines (BM). This class of BM's includes feedforward networks, radial basis networks and unsupervised clustering and probability density estimation networks. For this Boltzmann Machine the EM algorithm gives a significant speed up compared to standard methods such as (conjugate) gradient descent. 1 email: [email protected] 1 Introduction Boltzmann Machines are an interesting class of Neural Networks, because they have explicit parallelism of neuron and weight dynamics. They describe a general class of networks of which feedforward networks and clustering networks are special cases. However, their practical us..