We introduce a new stochastic competitive learning algorithm (SCoLA) and apply it to vector quantization for image compression. In competitive learning, the training process involves presenting, simultaneously, an input vector to each of the competing neurons, which then compare the input vector to their own weight vectors and one of them is declared the winner based on some deterministic distortion measure. Here a stochastic criterion is used for selecting the winning neuron, whose weights are then updated to become more like the input vector. The performance of the new algorithm is compared to that of frequency-sensitive competitive learning (FSCL); it was found that SCoLA achieves higher peak signal-to-noise ratios (PSNR) than FSC

Bouzerdoum, Abdesselam

English

Research Online @ ECU

Edith Cowan University Research Online ECU Publications Pre. 2011 2001 Image compression using a stochastic competitive learning algorithm (scola) Abdesselam Bouzerdoum Edith Cowan University Follow this and additional works at: https://ro.ecu.edu.au/ecuworks  Part of the Engineering Commons 10.1109/ISSPA.2001.950200 This is an Author's Accepted Manuscript of: Bouzerdoum, A. (2001). Image compression using a stochastic competitive learning algorithm (scola). Proceedings of 6th Internatrional Symposium on Signal Processing and its Applications. (pp. 541-544). Malaysia. IEEE. Available here © 2001 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. This Conference Proceeding is posted at Research Online. https://ro.ecu.edu.au/ecuworks/4811 Intemational Symposium on Signal Processing and its Applications (ISSPA)). Kuala Lumpur, Malaysia, 13 - 16 Augusl. 2001. orgarlized by the Dept. of Microelectronics and Computer Engineering, UTM, Malaysia and Signal Processing Research Centre. QUT, Australia JMAGE COMPRESSION USING A STOCHASTIC COMPETITIVE LEARNING ALGORITHM (SCOLA) Abdesselam Bouzerdoum School of Engineering and Mathematics, Edith Cowan University 100 Joondalup Drive, Joondalup, WA 6027, Australia a. bouzerdoum@ecu.edu. au ABSTRACT In this article we introduce a new stochastic competitive learning algorithm (SCoLA) and apply it to vector quan- tization for image compression. In competitive learning, the training process involves presenting, simultaneously, an input vector to each of the competing neurons, which then compare the input vector to their own weight vectors and one of them is declared the winner based on some deterministic distortion measure. Here a stochastic crite- rion is used for  selecting the winning neuron, whose weights are then updated to become more like the input vector. The performance of the new algorithm is com- pared to that of frequency-sensitive competitive learning (FSCL); it was found that SCoLA achieves higher peak signal-to-noise ratios (PSNR) 6han FSCL. 1. INTRODUCTION Image compression is extremely important for efficient storage and transmission of image data. Applications include storing medical images, finger prints and draw- ings, transmission of TV and remote sensing images, telemedicine, video conferencing, etc. Image compres- sion also finds applications in the areas of pattern classi- fication and pattern recognition. Consequently fast image coding techniques with high visual fidelity are of major importance. Many techniques have been developed for image coding, some are lossy and some are lossless; they range from simple subsampling to statistical and trans- form coding. Vector quantization (VQ) is a lossy compression tec& nique that can achieve high compression rates with good visual fidelity, especially when combined with other tech- niques such as transform coding. Numerous VQ algo- rithms have been investigated for speech and image coding; for review see, e.g., [ l ,  2, 31. The algorithms of interest here are those based on competitive learning, where the vector quantizer is constructed adaptively using a competitive learning algorithm; by contrast to batch techniques, such as the LBG (Linde-Buzo-Gray) vector quantizer [4], which require the entire data set to construct the codebook vectors. The advantage over batch techniques is that much less storage is required. Two examples of competitive learning algorithms, which have been used extensively for vector quantiza- tion, are the Kohonen selj-organizing feature maps (KSFM) [5] and the frequency sensitive competitive learning (FSCL) algorithm [6]. These algorithms have been used for VQ of speech waveforms [5 ,  61 and for image compression [7-91. Both of them achieve compa- rable performances to that of the LBG algorithm. Their performance, however, depends on several factors such as the network size, the initial learning rate, and the learning rate function. In this study, we investigate the perform- ance of a new competitive learning algorithm, know as the stochastic competitive learning algorithm (SCoLA) [lo], in image compression using vector quantization. In general, competitive learning algorithms differ in two respects. The first is the criterion employed for deter- mining the winning neuron. For instance the standard competitive learning (SCL) and KSFM algorithms choose the neuron with maximum response as the winner [ l l ,  121; whereas the FSCL algorithm modifies the distortion measure to favor neurons which have not been winning as often as the others [13, 141. The second difference is the method used for updating the neurons. The FSCL and SCL algorithms update only the winning neuron. On the other hand the KSFM algorithm updates the weight vec- tors of the winning neuron and all the neurons residing in a ‘neighborhood’ surrounding the winning neuron. The stochastic competitive learning algorithm, referred to herein as SCoLA, differs from the conventional competi- tive learning algorithms in one important respect, that is the criterion employed for determining the winning neu- ron; SCoLA uses a stochastic criterion, whereas conven- tional algorithms use a deterministic one. The paper is organized as follows. In the next section, the application of competitive learning to VQ is briefly described. Then the stochastic competitive learning algo- rithm is introduced in the following section. The applica- tion of the new algorithm, SCoLA, to image compression using vector quantization is presented in Section 4, which is followed by some concluding remarks in Section 5. 0-7803-6703-0/01/%10.00~001 IEEE 54 1 International Symposium on Signal Processing and its Applications (ISSPA), Kuala Lumpur. Malaysia, 13 - 16 August, 2001. Organized by the Dept. of Microelectronics and Computer Engineering, UTM Malaysia and Signal Processing Research Centre. QUT, Australia 2. VECTOR QUANTIZATION USING COMPETITIVE LEARNING Many techniques have been developed for image coding. They range from simple subsampling to statistical and transform coding. Most of these techniques use scalar quantization, and Hence do not achieve very high reduc- tions in bit rate. Another coding technique, which achieves relatively high bit rate reductions, is vector quantization (VQ). The purpose of VQ is to represent a given set of vectors of some arbitrary dimension by a smaller number of code vectors of the same dimension. Each of the original vectors is replaced by an index to one of the code vectors. The quantized data consists of a codebook of vectors plus a code vector index for each of the original data vectors. Data compression is achieved since in general the code vector index takes only a frac- tion of the storage required for one of the original data vectors. The class of neural networks which use competitive learning algorithms are readily employable as vector quantizers. These networks consist of a number of neu- rons each of which has a weight vector of the same dimension as that of the input vectors. The training proc- ess involves presenting an input vector simultaneously to each of the neurons. The neurons compare the input vec- tor to their own weight vectors and one of them is declared the winner. The winning neuron then updates its weight vector to become more like the input vector. This process is repeated using all input vectors until some stopping criterion is reached. In terms of VQ the weight vectors are considered to be the code vectors. Consequently the process of finding a codebook becomes one of training the neural network and extracting the weight vectors. Furthermore the trained network can be used to map the input vectors to their corresponding code vectors, that is to carry out the encoding operation. The quantization process is carried out according to some criterion which determines the accuracy of the quantized data. A typical criterion is to minimize a given distortion measure between the original vectors and their corresponding code vectors. Several distortion measures have been suggested. The simplest and most commonly used distortion meqsure is the square error distortion given by d ( x , y )  = I ! X - Y ~ ~ ~  where x is the original vector and y is the reproduced vector. Other more sophisticated distortion measures have been used before [ 11 to obtain better distortion rates. 3. THE STOCHASTIC COMPETITIVE LEARNING ALGORITHM In the deterministic approach to competitive learning, the winner is selected based on some deterministic distortion measure, such as the Euclidean distance between the input vector and the weight vector. By contrast, in sto- chastic competitive learning the criterion for selecting the winning neuron consists of a deterministic component and a stochastic component. The deterministic compo- nent is a function of the distance between the input vector and the weight vector. The stochastic component, on the other hand, is a normally distributed zero-mean random variable. Let Xk be the input at time t ,  and let dik be some dis; tance. measure between Xk and the weight vector of the ith neuron, wi : Then the response of the ith neuron, in a stochastic com- petitive network, is given by where U i k  is given by 1 U i k  = - 1 + d i k  (3) and r i ( t )  is a normal random variable with zero-mean and standard deviation o i ( t ) :  Note that U .  is the deterministic component of the response and IS solely dependent on the distance between Xk and w i ,  whereas r i ( t )  is the random component of the response; its variance is a monotonically decreasing function of the number of times the neuron wins the com- petition. '1( From Eqs. (2) and (4) it is clear that the response of the ith output neuron is a normal random variable with mean uik  and standard deviation o;(t): . Y,(O = N(Uik9 o,(t>) The winner of the competition is determined in a similar manner to standard competitive learning; that is, the win- ning neuron i* is the one with the largest response: is = arg max { y i ( t ) }  I 542 International Symposium on Signal Processing and its Applications (ISSPA), Kuala Lumpur. Malaysia, I3 - 16 August, 2001. Organized by the Dept. of Microelectronics and Computer Engineering, CJTM, Malaysia and Signal Processing Research Centre, QlJT Australia A count is kept of the number of times each neuron has won the competition. Therefore, at any time instant t, each output neuron is characterized by three variables: its weight vector w i ,  the standard deviation of its random component oi(t), and a count of the number of times it has won the competitiqn ni( t ) .  All these three variables are updated upon the neuron winning the competition. The complete stochastic competitive learning algorithm (SCoLA) is presented below. Step 1: Initialize all weight vectors to some suitable ran- dom values, set the winning frequency count jor all neu- rons equal to 1, and set the standard deviations to o0: n i ( 0 )  = 1, V i  ~ ~ ( 0 )  = o,,, V i  Step 2: Select an input xk at random, present it to all neurons and compute the response of every output neuron using Eqs. (1) to (4). Step 3: Select the winning neuron according to Eq. (5) and update its weight vector, winning frequency count and standard deviation: Step 4: Repeat Steps 2 and 3 until convergence or a stop- ping criterion is reached. ' Note that each time a neuron wins the competition its standard deviation is reduced, thereby increasing the likeli- hood that the same neuron will win the competition in the hture for the same (or similar input) pattern. However, it may still loose the competition due to the stochastic nature of the response. Furthermore, as the variances of the fre- quently loosing neurons remain high, their chances of win- ning the competition with other input patterns remain high. As the training process progresses, the influence of the sto- chastic component on the response is expected to diminish; in the long term, the response would mainly be determined by the deterministic component. the learning rate T) deter- mines the extent to which the winning neuron's weight vector is modified or shifted towards the given input vec- tor; as the learning rate increases the weight vector moves closer to the input vector. Typically, the learning rate is chosen as a monotonically decreasing function of the training time. 4. SIMULATION RESULTS To investigate the performance of SCoLA several simula- tion experiments were conducted using three different images: MRI, Lenna and baboon image. First the images were partitioned into nonoverlapping 4 x 4 regions. Each region was then converted into a 16-dimensional column vector. The resulting vectors were used to train several neural networks; all trained networks had the same number on inputs (16 inputs) and different number of out- puts. For comparison purposes, the networks were also trained using FSCL. Figure 1 illustrates the perform- ances of SCoLA and FSCL. It is clearly obvious that in terms of PSNR (peak signal-to-noise ratio) SCoLA out- performs FSCL. Figure 2 illustrates the performance,dur- ing training of the two algorithms. I 32 64 128 256 Number a i  Codebook Vectors Figure 1. Peak signal-to-noise ratio (PSNR) for SCoLA (squares) and FSCL (circles). Top to bottom, the curves are for MRI, Lenna and baboon images, respectively. 32 , , , , , , , , . 30 - ~ Figure 2. PSNR as a function of training epochs using Lenna image. 543 International Symposium on Signal Processing and its Applications (ISSPA), Kuala Lumpur. Malaysia. 13 - 16 August, 2001. Organized by the Dept. of Microelectronics and Computer Engineering, UTM, Malaysia and Signal Processing Research Centre, QUT, Australia References Coded Figure 3. original images of Lenna and baboon and their counterparts coded using a network of size (16, 128), i.e., 16 inputs and 128 output units (BR = 0.4375 bitslpixel), trained on Lenna using SCoLA. Figure 3 shows the original images and the vector quan- tized images. Both images were vector quantized using a neural network which was trained on the Lenna image using SCoLA. Despite this, the network achieves good performance on the baboon image, which proves that the network can generalize well. 5. CONCLUSION A new stochastic competitive learning algorithm has been introduced. The main feature of the algorithm is that the response of the output neurons is of stochastic in nature, composed of a deterministic component and a random component. The deterministic component is a function of the distance between the input vector and the neuron’s weight vector, whereas the stochastic component is a ran- dom variable whose standard deviation decreases mono- tonically with the neuron’s frequency of winning the competition. The main advantage of the proposed algo- rithm is that it overcomes the neural underutilization prob- lem without the huge computational overhead of some other competitive learning algorithms. Results of several simulation experiments on image compression using vec- tor quantization clearly show that the proposed algorithm achieves better PSNR performance than FSCL. R. M. Gray. “Vector Quantization.” IEEE ASSP Magazine, Vol. 1, pp. 4-29, April 1984. Makhoul, J., S .  Roucos and H. Gish. “Vector quantization in speech coding.” Proc. of IEEE, Vol. 73,  pp. 1551-1558, Nov. 1985. Nasrabadi, N. M. and R. A. King. “Image coding using vec- tor quantization: A review,” IEEE Trans. on Communica- tions, Vol. 36, pp. 957-971, 1988. Linde, Y., A. Buzo and R. Gray. “An algorithm for vector quantizer design.” IEEE Trans. Communications, Vol. 28, pp. 8495,1980. T. Kohonen. Self-organizing Maps. Springer, Berlin, 1995. Ahalt, S .  C., A. K. Krishnamurthy, P. Chen and D.E. Melt- on. “Competitive leaming algorithms for vector qunatiza- tion.” Neural Networks, Vol. 3, pp 277-290, 1990. Nasrabadi, N. M. and Y. Feng. “Vector Quantization of im- ages based upon the Kohonen self-orgainzing feature maps.” Proc. IEEE Int. Conf: on Neural Networks, pp. Ahalt, S .  C. and A. K. Krishnamurthy. “Performance analy- sis of two image vector quantization techniques.” Proceed- ings ofIJCNN, Vol. I, pp. 169-175, 1989. Bouzerdoum, A. and T. J. Alton. “Image compression using frequency sensitive competitive leaming.” Proc. of IEEE Workshop on Visual Signal Processing & Communications, pp 89-92.21-22 September 1993, Melbourne, Australia. 1101-1 108, 1988. [ 101 A. Bouzerdoum. “A stochastic competitive learning algorithm .” Proceedings of Interantional Joint conference on Neural Networks (IJCNN-2001), Washington D.C., USA. Hertz, J., A. Krogh, and R. G. Palmer. Introduction to the Theory of Neural Computation. Addison Wesley, Red- wood City, CA., 1991. Rumelhart, E. E. and D. Zipser. “Feature discovery by com- petitive leaming.” Cognitive Science, Vol. 9, pp. 75-1 12, 1985. Also reprinted in: Parallel Distributed Processing, Vol. 1 : Foundation, (D. E. Rumelhart and J. L. McClelland, eds.), pp. 151-193, MIT Press, Cambridge, MA, 1986. S .  Grossberg. “Competitive leaming: from interactive acti- vation to adaptive resonance.” Cognitive Science, Vol. 11, DeSieno, D. (1988). “Adding a conscience to competitive leaming.” Proc. of IEEE International Conference on Neu- ral Networks, Vol. I, pp. 117-124, 1988, San Diego. pp. 23-63.1987, 544 

Image compression using a stochastic competitive learning algorithm (scola)

https://ro.ecu.edu.au/cgi/viewcontent.cgi?article=5810&amp;context=ecuworks

Image compression using a stochastic competitive learning algorithm (scola)

Abstract

Similar works

Full text

Available Versions

Research Online @ ECU