The Stability and Attractivity of Neural Associative Memories > Ph.D. Student: Han-bing JI Supervisors: Professor Kwong-sak LEUNG Professor Yee LEUNG Date: September 20，1996 Department of Computer Science and Engineering The Chinese University of Hong Kong .UMI Number: 9908164 UMI Microform 9908164 Copyright 1998, by UMI Company. All rights reserved. copying under Title 17, United States Code. Acknowledgment I would very like to take this opportunity to express my great gratitude to my two supervisors Professors Kwong-sak Leung and Yee Leung for their valuable supervision and help during my three years of research. Table of Contents Title Acknowledgment Abbreviations Nomenclature Diagram Captions Figure Captions Table Captions Abstract Chapter 1 Introduction Chapter 2 An Overview of Related Works in Neural Associative Memories 2.1 Capacities of the Hopfield-Type Associative Memories 2.2 Improvements Based Upon Global Distribution Storage 2.2.1 Learning Scheme 2.2.2 Network Structure 2.2.3 Nonmonotone Dynamics 2.3 Local Distribution Storage 2.4 Transformation Models 36 2.5 Concluding Remarks and Prospectus of this Thesis 37 Chapter 3 Adaptive Weighted Outer-Product Learning Associative Memory 40 3.1 Introduction 40 3.2 The Adaptive Weighted Outer-Product Learning Associative Memory 41 义 2.1 The Hopfield Associative memory • 41 3.2.2 The Weighted Outer-Product Learning Associative Memor\ 44 3.3 Sufficient Conditions for the Learning Weights and SNRGs 45 3.4 Search of the Learning Weights Through Adaptive Learning 49 3.4.1 Global-Error-Measure Algorithm 50 3.4.2 Local-Error-Measure A Igoriihm 52 3.4.3 Adjustments of Neuronal Parameters 53 3.5 Experimental Results 56 3.5.1 Representative Examples 56 3.5.2 Computer Simulations of the Adaptive Algorithms 61 3.6 Conclusion 69 Chapter 4 A Novel Neural Associative Memory with Maximum Stability 70 4.1 Introduction 70 4.2 A Novel Encoding Strategy Based Neural Associative Memory 71 4.3 Analysis of the Stability and Attractivity 73 4.3.1 Stability Analysis 73 4.3. /. I The Hopfield Model 73 4.3.1.2 First-Order Outer-Product Model with Self-Feedback Connections 74 4 .?. 1.3 The Novel Encoding Strategy Based Neurol Associative Mettu>r\ 75 4.3.2 Attractivity Analysis 77 4.3.2.1 The Hopfield Model 77 43.2.2 The First-Order Outer-Prixiuct Model wi//i Self-Feedback Connections 11 j The Novel Encoding Stralcf>\ Rased Neural Associative Memory 78 4.4 Computer Simulations 80 4.5 Conclusion 87 Chapter 5 Correlation-Type Associative Memory Using the Gaussian Function 88 5.1 Introduction 88 5.2 The Gaussian Correlation Associative Memory (GCAM) 89 5.2.1 The RCAM 89 5.2.2 The ECAM 92 5.2.3 The GCAM 92 5.3 Analyses of Several Correlation-Type Auto-Associative Memories 94 5.3.1 Linear Function Model 95 5.3.2 Exponential Function Model 97' 5.3.3 The Left-hand-side Gaussian Function Model 97 5.4 Simulation Results 100 5.5 Conclusion I l l Chapter 6 A Further Investigation into the Upper Limit of the Asymptotic Storage Capacity of the Hopfield Associative Memory ... 6.1 Introduction 6.2 Two Analysis Methods "… 6.2. / The Neural Dynamic Method 6.2.2 The SNRG Method 6.3 Investigation into the Upper Bound of the Asymptotic Storage Capacity of the Hopfield Network 116 6.3.1 By the Neural Dynamic Method 116 6.3.2 By the SNRG Method 118 6.4 Computer Simulations 121 6.5 Conclusion 133 Chapter 7 Concluding Remarks, Evaluation, and Outlook for Future Research 丨 34 Appendix I: The Proof of Theorem 3-1 139 Appendix n: The proof of Theorem 3-2 141 Appendix III: The Stability and Attractivity Analysis of the 6 Three Associative Memories 147 A3.1 Stability Analysis 147 A3.1.1 The Hopfield Model …… - 147 A3.1.2 The First-Order Outer-Product Model with Self-Feedback Connections 148 A3.1.3 The Novel Encoding Strategy Based Neural Associative Memory 150 A3.2 Attractivity Analysis 151 A3.2.1 The Hopfield Model •二 151 A.12.2 The First-Order Outer-Product Model with Self-Feedback Connections 152 A3.2.3 The Novel Encoding strategy Based Neural Associative Memon- 153 Appendix IV: Investigation into the Upper Bound of the Asymptotic Storage Capacity of the Hopfield Associative Memory from two Different Viewpoints 155 A4.1 The Neural Dynamic Approach 155 A4.2 By the SNRG Method 丨 56 References 160 Author's Paper List 164 Abbreviations BAM - Bidirectional Associative Memory ECAM - Exponential Correlation Associative Memory FM - Fundamental Memory FP - Fixed Point GCAM - Gaussian Correlation Associative Memory LHSGF - Left-Hand-Side Gaussian Function RCAM - Recurrent Correlation Associative Memory SNR - Signal to Noise Ratio SNRG - Signal to Noise Ratio Gain WOPLAM - Weighted Outer-Product Learning Associative Memory 8 Nomenclature a - base of the exponential function a⑴-learning weight directed to the recall of FM u - incremental change of a(rl Pi - parameter of the sigmoid function P： - parameter of the sigmoid function C(u,r)，11妒)-correlation between FMs u) - extensive distance between FMs u,rl and u F01 - associative recall error of FM u(n F - overall associative recall error of the network - function used for an optimization with a constraint 9 G(r) - SNRG ofFMu H(u(r>，u

) - Hamming distance between FMs u(r> and utp) L - length of a neighboring range of a hit of an FM m - number of FMs n - dimension of FMs N"11 - computational complexity of an algorithm o丨…-ilh element of the output vector corresponding to FM u q(p) - nonlinear monotonically decreasing function of the error percentage p p - error percentage of an input (compared to its corresponding FM) 9-equals d[Ilin(u, x> - correlation between utrl and x 10 Diagram Captions The Block diagram of the Hopfield network (in Chapter 3). The Block diagram of the RCAM (in Chapter 5). 11 Figure Captions Figure 3-1. Experimental results obtained by using the WOPLAM with different sets of learning weights (a'", a<2), an>, a'41). All sets of learning weights are chosen arbitrarily, (a) The four FMs (They are taken as the network's inputs). (bMm) Results of the WOPLAM by different sets of learning weights, (b) (1，1, I, 1). (The case of the Hopfield network), (c) (2，2, 2，2). (d) (1, 2，2, 2). (e) (2，3, 3，3). (f) (1，I, 1,2). (g) (2, 2, 3, 2). (h) (I, 1,2, 2). (i) (3, 4, 4, 3). (j) (2，3, 2, 2). (k) (4，6，6，4). (1) (4，5, 5, 4). (m) (5, 6, 6, 5). Figure 3-2. Empirical results obtained by efficiently utilizing the WOPLAM with different sets of learning weights (a⑴，al2\ a,4>). All sets of learning weights are empirically found and chosen so that as many SNRGs as possible are made to he larger than or equal to their corresponding thresholds. In this example, the maximum number of FPs is 3. (a) The four FMs (They are taken as the network's inputs), (h)-(e) Results of the WOPLAM with different sets of learning weights, (b) (4.2，5.7, 5.7, 3.0). (c) (0.7，3.4； 3.5,3.3). (d) (3.0,4.0, 0.7, 4.0). (e) (2.7，0.7, 3.4, 3.5). Figure 3-3. Simulation results of n=l(). Figure 3-4. Simulation results of n=20. Figure 3-5. Simulation results of n=50. • 12 Figure 3-6. Convergence speed comparison between different WOPLAMs (n=10). Figure 3-7. Convergence speed comparison between different WOPLAMs (n=20). Figure 3-8. Convergence speed comparison between different WOPLAMs (n=50). Figure 4-1. The effects of different neighboring ranges for the case of n= 10. Figure 4-2. The effects of different neighboring ranges for the case of n=20. Figure 4-3. The effects of different neighboring ranges for the ease of n=30. Figure 4-4. The effects of different neighboring ranges for the case of n=40. Figure 4-5. The overview of the effects of using different L for the case of n=30. Figure 5-1. The simulation results of n=20. The horizontal axis corresponds to the number of the FMs and the vertical axis corresponds to the number of the correctly recalled FMs. The results of the GCAM are plotted in solid lines while those of the EC AM are plotted in dotted lines. Figure 5-2. The simulation results of n=30. The horizontal axis corresponds to the number of the FMs and the vertical axis corresponds to the number of the correctly recalled FMs. The、 results of the GCAM are plotted in solid lines while those of the ECAM are plotted in dotted lines. v Figure 5-3. The simulation results of n=40. The horizontal axis corresponds to the number of the FMs and^he vertical axis corresponds t s^C number of the correctly recalled FMs. The results of the GCAM are plotted in solid lines while those of the ECAM are plotted in dotted lines. Figure 6-1. Simulation results of n=20. Figure 6-2. Simulation results of n=5(). Figure 6-3. Simulation results of、n=100. Figure 6-4. Simulation results of n= 150. Figure 6-5. Simulation results of n=200. Figure 6-6. Simulation results of n=250. 14 Figure 6-7. Simulation results of n=3(X). Figure 6-8. Simulation results of n=350. Figure 6-9. Simulation results of n=400. Table Captions Table 3-1. Learning weights and SNRGs of the cases in Fig.3- l(bHm). The single underlined SNRGs are the ones sufficient for their corresponding FMs to he FPs. The double underlined ones are the empirically obtained thresholds for their corresponding FMs to be FPs. Table 3-2. Learning weights and SNRGs of the cases in Fig.3-2(hMe). The "underlined" has the same meaning as that in Table 3-1. Table 5-1. The values of the parameters used in the simulations corresponding to Figs.5-l(a) to (g) with n = 20. p is the "error" percentage of the input. Table 5-2. The values of the parameters used in the simulations corresponding to Figs.5-2(a) to (g) with n = 30. p is the "error" percentage of the input. Table 5-3. The values of the parameters used in the simulations corresponding to Figs.5-3(a) to (g) with n = 40. p is the "error" percentage of the input. 16 Abstract In the Hopfield-type associative memories, all the fundamental memories (FMs) are stored in the manner of global distribution storage. One connection weight represents partial information of all the FMs, and thus cross-talk between different FMs exists in the storage process. More importantly, the storage capacity of the Hopfield model has been found to he severely constrained by the dimension of the FMs. Empirical [ 18] and theoretical [26] results show that the storage capacity of the Hopfield network, i.e., the number of stably stored FMs, is in general less than 15 percent of the FMs' dimension. In general, the quantity and quality of the associative recall of the FMs deteriorate as the number of FMs increases. Sometimes, even when this number is small, the associative recall of the FMs may not he all correct. The objective of this thesis is to find out and to improve the factors affecting the storage capacity and associative recall performance of the Hopfield-type associative memories. Approaches used include some new encoding/learning methods which can increase or greatly increase the signal to noise ratio (SNR), and local distribution storage with nonlinear function which not only has no cross-talk between different FMs but also enlarges the difference between auto- and mutual-correlations. Finally, a further investigation into the scheme of the Hopfield model is given which reveals some critical factors for the associative store/recall. There are four main topics of original results to be presented in this dissertation. They are an adaptive weighted outer-product learning associative memory, neural associative memory with maximum stability based on a novel encoding strategy, a Gaussian correlation associative memory, and a further investigation into the upper hound of the asymptotic storage capacity of the Hopfield associative memory. With respect to the adaptive weighted outer-product learning associative memory, for the correct recall of an FM, a corresponding learning weight is attached to this FM and a parameter called signal to noise ratio gain (SNRG) is devised. Sufficient conditions for the learning weights and the SNRGs are derived. It is found both empirically and theoretically that the SNRGs have I their own threshold values for the correct recall of the corresponding FMs. Based upon the gradient-descent-search approach, several algorithms are constructed to adaptively find out optimal learning weights with reference to global- or local-error measure. With respect to the novel encoding strategy based neural associative memory with * maximum stability, unlike the conventional point wise outer-product rule used in the Hopfield-type associative memories, the proposed encoding method computes the connection weight between any two neurons by summing up not only the products of the corresponding two bits of all FMs but also the products of their neighboring bits within certain range. Theoretical analysis is conducted to investigate the stability, attractivity and their interrelationship of the proposed model. It is found both theoretically and experimentally that the proposed encoding scheme is an ideal approach for making the FMs fixed points and thus maximizing the storage capacity. 、 Within the Gaussian correlation associative memory (GCAM) - a high-capacity correlation-type associative memory neural network, the FMs stored are locally represented by two layers of connection weights of the network. The left-hand side of the Gaussian function (LHSGF) is used as weighting functions. Using the LHSGF has the same 18 effectiveness in maximally discriminating auto-correlations from all mutual correlations as the exponential function used in the ECAM (Exponential Correlation Associative Memory), but has no limitation of dynamic ranges in real circuits implementation from which the ECAM suffers. The GCAM has the same exponentially-growing storage capacity as the ECAM. Besides, basins of attractions of the FMs in the GCAM can he controlled through adjusting two parameters of the LHSGF and thus can be larger than in the ECAM. In the further study of the upper hound of the asymptotic storage capacity of the Hopfield associative memory, the upper bound of the asymptotic storage capacity of the Hopfield network is further investigated from two different points of view - the neural dynamic and the SNRG approaches. Similar results are concurrently obtained, both of which show that the asymptotic storage capacity of the Hopfield network in order that all the FMs are exactly recoverable does not grow directly proportionally or proportionally to the dimension (n) of the FMs but is upper hounded as n approaches infinity. Such an upper hound is not decided or directly decided by n, hut is directly determined by the distribution of the elements of the FMs. Overall, in this thesis, neural associative memories are studied not only within the framework of the conventional global distribution storage hut also within that of the local distribution storage. Moreover, within the former framework, a novel group-wise strategy is proposed for encoding the FMs other than the conventional point-wise method. Finally, a further investigation into the Hopfield model reveals some intrinsic and critical factors which post some open problems on the storage capacity of neural associative memories. 19 Chapter 1 Introduction Since Hopfield's conception of the outer-product learning based neural associative memory in his seminal paper [ 18], much research has been done to theoretically investigate the storage capacity, information capacity, and performances of the Hopfield associative memory from various points of view (1], [4], [5], [6], [11], [13], [16], [20], [23】，[261，[44]. It has been found that the storage capacity of the Hopfield network is severely constrained by the dimension of the fundamental memories (FMs). Empirical [18] and theoretical [26] results show that the storage capacity of the Hopfield network, i.e., the number of stably stored FMs, is in general less than 15 percent of the FMs' dimension. In general, the quantity and quality of the associative recall of the FMs deteriorate as the number of FMs increases. Sometimes, even when } this number is small, the associative recall of the FMs may not be all correct. To alleviate such a limitation, various approaches have been attempted to increase the storage capacity over the years. For example, Lee [24】，Soffer [41J and, Bak and Little [3】raised the order of the correlation matrix; Venkatesh and Psoitis [48] utilized the spectrum of a linear operator; Chiueh and Goodman [8], [9] presented a general model of associative memory in which a function is applied to the inner products of an input and the FMs; an iterative projection rule [55] was constructed to update the connection weights initially defined by the Hebbian rule when the FMs are sequentially presented to the network; Wang et al. [50], [51 ] proposed a BAM (Bidirectional Associative Memory) with multiple training which was further extended to a generalized BAM. Linear programming/multiple training and sequential multiple training are applied to find the weights of a generalized correlation matrix; Wang and Lee [54] put additional network structure 20 into the BAM in order to cancel or suppress noise terms in a retrieval process; Morita, et al. [28】，[59] and，Nishimori and Opris [30] made use of nonmonotonic output functions; Xu, et al. [56] proposed an asymmetric BAM which can cater for the logical asymmetry of interconnections and non-Qrthogonality of patterns. It should be pointed out that the aforementioned work done on the BAM is in principle also effective for the Hopfield-type neural associative memories. In a word, the examples cited above can improve the performances in terms of the storage capacity and the associative recall of a network but usually at the expense of extra costs such as computation complexity, network structure and hardware implementation. The main thrust of this thesis is to analyze and improve the storage capacity of the Hopfield network by formulating some novel encoding schemes and learning algorithms. Stability and attractivity of the proposed associative memories are also examined. The basic approach of the thesis can be described as follows: Firstly, to find out the critical factors and fundamental mechanism for the associative store/recall of the Hopfield model. Secondly, to propose learning algorithms to improve the storage capacity and to make these algorithms practical. Thirdly, to propose a novel encoding strategy to achieve maximum or extreme stability of the network. However, such achievements are obtained on the expenses of attractivity to some extent. Fourthly, unlike the global distribution storage used in conventional studies, a local distribution storage model with nonlinear dynamics is presented. Extreme stability and attractivity are both achieved. Finally, based on the results obtained, a further study of the associative store/recall mechanism of the Hopfield mode! are conducted, and accordingly future research topics are clearly addressed. 21 Due to the nature of the FMs, we postulate that by assigning different learning weights to different FMs, the storage capacity of the Hopfield network can be substantially increased. To achieve this goal, we put forth a Hopfield-type weighted outer-product learning associative memory with several adaptive algorithms. For the correct recall of an FM, a corresponding learning weight is attached to this FM and a parameter called signal to noise ratio gain (SNRG) is devised. Sufficient conditions for the learning weights and the SNRGs are derived. It is found both empirically and theoretically that the SNRGs have their own threshold values for the correct recall of the corresponding FMs. Based on the gradient-descent-search approach, several algorithms are constructed to adaptively find the optimal learning weights with reference to a global- or local-error measure. We further propose a neural associative memory based on a novel encoding strategy. Unlike the conventional pointwise outer-product learning rule commonly used in the Hopfield-type associative memories, the novel encoding method computes the connection weight between any two neurons (including self-feedback connections) by summing up not only the products of the corresponding two bits of ail the FMs, but also the products of their neighboring bits within a certain range as well. The widely-used SNR (Signal to Noise Ratio) and the neural dynamic approaches are utilized to theoretically investigate the performances of the proposed model in terms of stability and attract ivity. Both theoretical and experimental results show that the proposed encoding strategy is very powerful in maximizing the stability (making as many FMs as possible be fixed points) or the capability. Both of the aforementioned studies lie within the framework of the global distribution storage of the FMs. Since it has been found out that local distribution storage of the FMs has 22 an advantage over the global distribution storage in the way that the former can greatly supress the cross-talks among the FMs, we then make an effort in encoding the FMs by the local distribution storage. Based on these considerations, we propose a correlation-based associative memory using the left-hand side of the Gaussian function (LHSGF) as weighting functions - the Gaussian correlation associative memory (GCAM). It is extended from the RCAM (Recurrent Correlation Associative Memory) and the ECAM (Exponential Correlation Associative Memory) proposed by Chiueh and Goodman [8], [9]. The ECAM is a special case of the RCAM. When the function used in the RCAM is the exponential function, the RCAM is reduced to the ECAM. The storage capacity of the ECAM was theoretically proven to be growing exponentially with the dimension of the FMs. However, from the viewpoint of real circuits implementation, the storage capacity of the ECAM is practically limited by the dynamic ranges of the real circuits [9]. Circuits cannot give an infinitely large output response to an input. They have limited ranges of output response. As a result, the storage capacity of the real ECAM circuits will be far below the theoretical result. In order to overcome such a shortcoming, we modify the ECAM by replacing the exponential function with the LHSGF and thus propose the GCAM. As the weighting functions, the LHSGF is to be used with which the required monotonically-nondecreasing property [9J still holds. Using the LHSGF has the same effectiveness in maximally discriminating an auto-correlation (between an input pattern and its corresponding FM) from all mutual correlations (between the input pattern and all the other FMs) as the exponential function is used in the ECAM, but it has no limitation on the dynamic ranges in the real 23 circuits implementation suffered by the ECAM. The GCAM has the same exponentially-growing storage capacity as the ECAM. Besides, basins of attractions of the FMs in the GCAM can be controlled through adjusting two parameters of the LHSGF and thus can be larger than that in the ECAM. In addition to the studies described above, we are still very interested in the problems like: why the Hopfield model has a very small storage capacity, and what factors affect its storage capacity. To answer these questions, a further investigation into the upper hound of the asymptotic storage capacity of the Hopfield network is conducted. In [26], it was shown that if m FMs are chosen at random (the elements of the FMs take +1 or -I with an equal probability of 0.5), the maximum asymptotic storage capacity in order that most of the m FMs are exactly recoverable, is n/(21ogn). By imposing an additional restriction that every one of the m FMs be exactly recoverable, tl\e maximum asymptotic storage capacity can be no more than n/(41ogn) as n approaches infinity. Some similar results concerning the storage capacity were provided in [23], [44] and [48]. The results in [20】showed that the Hopfield network has major limitations when applied to fixed pattern classification problems because of its sensitivity to the number of the FMs and the SNR of the input data. It was shown in [13] that for associative memories composed of n linear threshold functions without self-feedback connections, even when the Hamming distances between the desired memories are within yri and (l-y)n, there are sets of size (l-2y) 1 (for y < 1/2)，the FMs of which cannot be simultaneously stable. A similar phenomenon holds for the sum of outer-products connection matrix. It was shown in [6] that the Hopfield network can result in many spurious stable states -exponential in the number of the FMs - even in the case when the FMs are orthogonal. 24 We will theoretically investigate the upper bound of the asymptotic storage capacity of the Hopfield network from two different points of view. One is based on the neural dynamic approach while the other is based on the SNRG concept. Similar results are simultaneously obtained from both viewpoints. It is shown that the asymptotic storage capacity of the Hopfield network, in order that all the FMs are exactly recoverable, does not grow directly proportionally or proportionally to n (the dimension of the FMs) hut is upper bounded as n approaches infinity. Such an upper bound is not directly decided or decided by n, but is directly determined by the distribution of the elements of the FMs. The result by the neural dynamic approach shows that the upper bound of the asymptotic storage capacity of the Hopfield network is directly dependent upon the minimum extensive-distance between any two different FMs. We will see that as the number of the FMs grows larger, the Hopfield network has major limitation in stably storing all the FMs because of its sensitivity to the minimum extensive-distance between any two different FMs. A small minimum extensive-distance can greatly deteriorate the storage capacity of the Hopfield network. Finally, we make a summary of this study to conclude the whole thesis. It is worth mentioning here that the FMs will not be pre-processed for orthogonalization because using such an approach to improve the stability of the FMs is not common. The FMs used are just randomly generated from symmetric Bernoulli trials. The remaining part of this thesis is organized as follows: An overview on neural associative memories is made in Chapter 2. In Chapter 3, we propose a weighted outer-product learning associative memoryWOPLAM), including learning scheme, critical factors for associative store/recall, improvement of the storage capacity, and practical algorithms. A 25 novel encoding model with maximum or extreme stability is then constructed in Chapter 4. It stores the FMs with global distribution storage and lies within the framework of the Hopfield-type models. Its attractivity, however, still needs to be improved. In Chapter 5, the GCAM which adopts local distribution storage with nonlinear function is discussed. In the GCAM, cross-taiks among the FMs are completely eliminated and the differences between auto- and mutual-correlations are enlarged. A further investigation into and understanding of the Hopfield-type models is made in Chapter 6. Results obtained reveals the intrinsic mechanism and critical factors for the associative store/recall. In Chapter 7, we summarise the main points of the thesis and outline some directions for further research. 26 Chapter 2 An Overview of Related Works in Neural Associative Memories 2.1 Capacities of the Hopfield-Type Associative Memories Storage capacity has always been a major issue in the Hopfield network. Researchers with different backgrounds use different approaches to investigate the storage capacity of the Hopfield-type associative memories. In these studies, fundamental memories (FMs) are generally assumed to be randomly generated. Results discussed below on one hand can help us understand the Hopfield-type models, and on the other hand they reveal certain outstanding problems that need to be investigated. Some of the problems are addressed in this thesis. McEIiece, et ai. [26] theoretically deduced the asymptotic storage capacity of the Hopfield associative memory using the information theory. Their results told us that if m fundamental memories (FMs) are chosen at random, the maximum asymptotic value of m, in order that most of the m original memories are exactly recoverable, is n/(2logn) (n is the dimension of FMs). By further imposing the restriction that every one of the m FMs be exactly recoverable, m can reach no more than n/(4logn) asymptotically as n approaches infinity. The results obtained by Jacyna and Malaret [20] showed that the Hopfield network has major limitations when applied to fixed pattern classification problems because of its sensitivity to the number of coded vectors stored and the signal to noise ratio (SNR) of the input data. Some similar results concerning the storage capacity were provided by Kuh and Dickinson [23], Sussmann [44], and Dembo [13】，all of which used other approaches. It was shown in【13】that for * associative memories composed of n linear threshold functions without self-feedback connections, even when the Hamming distances between the FMs are within yn and (l-y)n, 27 there are sets of size (l-2y) 1 (for y < 1/2)，the FMs of which cannot be simultaneously stable. A similar phenomenon holds for the sum of outer-products connection matrix. Abu-Mostafa and Jacques [ 1 ] found out that the asymptotic information capacity of a Hopfield network of n neurons is of the order n' hit, and the number of arbitrary state vectors that can be made stable in a Hopfield network of n neurons was proven to be hounded above by n. i ¥ Bruck and Roychowdhury [6] showed that the Hopfield network can result in many spurious stable states - exponential in the number of the FMs - even in the case when the FMs are orthogonal. With respect to the convergence properties of the Hopfield network, the results obtained in Bruck (5] revealed that they are dependent on the structure of the interconnections matrix and the method by which the nodes are updated. Floreen [ 16] improved the worst-case upper bound on the convergence time of Hopfield associative memories to half of its previously known value. Concerning similar vvbtiwi corresponding to other kinds of neural associative memories, Chou [11] derived an asymptotic expression for the capacity of an associative memory proposed by Kanerva. Ryan and Andreae [36] experimentally demonstrated a discrepancy between Kanerva's theoretical calculation of capacity and the actual capacity, and offered a correct theory. They not only suggested a modified method of reading from memory which results in a capacity nearly the same as that originally predicted by Kanerva, but also a different method of writing to memory which increases the capacity by an order of magnitude. In Baram [4] and [37], networks of ternary neurons are considered whose connections are 28 formed by the Hebbian storage rule. The storage capacity is derived. The number of connections required by a ternary network depends on the number of stored patterns and can be considerably smaller than that of a fully connected network. 2.2 Improvements Based Upon Global Distribution Storage Global distribution storage means that each connection weight represents partial information of all FMs, or all the FMs are partially represented by each connection weight. 2.2.1 Learning Scheme Quite a lot of researchers try to decrease noise and/or increase signal to improve storage capacities within the framework of the Hopfield-type models. Venkatesh and Psaltis [48] proposed ari associative memory that utilizes the spectrum of a linear operator to store the FMs. Its storage capacity is linear in the dimension of the FMs. Recurrent strategies for this spectral method were developed for their computation. Wang, et al. [50] presented a concept of the multiple training for the BAM (Bidirectional Associative Memory) which can be guaranteed to achieve correct recall of a single trained pair. And, the dummy augmentation proposed can be guaranteed to achieve recall of all trained pairs if. attaching dummy data to the training pairs is allowed. Wang, et al. [51 ] derived the necessary and sufficient conditions for the weights of a generalized BAM which guarantees the recall of all training pairs. A linear programming/multiple training method was proposed to determine the weights that satisfy the condition when a solution is feasible. A sequential multiple training method yields integers for the weights, where the weights are multiplicities of training of the training pairs. Yen and Michel [58] developed a design technique for associative memories with learning and forgetting capabilities utilizing the eigenstructure method. In many respects, such a model 29 constitutes significant improvements over the outer-product method, the projection learning rule, and the pseudo-in verse method with stability constraints. Wang, et al. [52] applied the multiple training concept to the hackpropagation algorithm for use in associative memories. The algorithm assigns different weights to various pairs in the energy function. The pair weights are updated using the basic differential multiplier method. Oh and Kothari [31] described an iterative learning algorithm by which guaranteed recall of all training pairs is ensured. The algorithm is based on a novel adaptation from the well-known relaxation method for solving a system of linear inequalities. Sudharsanan and Sundareshan |43] obtained various « results characterizing the equilibrium conditions of a very useful dynamical neural network model and a systematic synthesis procedure for designing associative memories with non-symmetrical weight matrices. Perfetti [34] showed the applicability of the projection rule to the design of associative memories based on the neural networks with sigmoidal nonlinearities. The method exhibits the features of learning capability, computational efficiency, exact storage of binary patterns as asymptotically stable equilibrium points and global stability of the network. In Coombes and Taylor [ 12], it was shown how techniques in statistical mechanics may be used to structure connection weights which are capable of storing n patterns in a Hopfield network of n spins. Simulation results confirmed that n random patterns may indeed be stored in the Hopfield network of n spins using a set of weights that are proportional to the inverse of the pattern correlation matrix. Basins of attraction may he increased , up to some maximum, for a set of stored patterns at the expense of the stability of the remaining patterns. This is very similar to that proposed in the model of learning by selection. Parodi, et al. [33] showed that in a noise-like coding model of associative memories, the core of the information 30 -coding mechanism lies in the key-production process by which a given pattern is transformed into a corresponding noise-like key for both information storage and retrieval. An iterative projection rule [55] was constructed to update the connection weights initially defined by the Hebbian rule when the FMs are sequentially presented to the network. Nemoto and Kubono [29] proposed a complex associative memory in which input and output patterns are vectors of values 1 and -1，and weights take on complex values. The learning rule for the complex weights was developed. The complex associative memory yields better performance than would be predicted by the increase of the degree of freedom due to complexification. Gibson and Robinson [17] proposed a sparse autoassociative memory model which has its origins in the biologically motivated model for memory storage. The network consists of randomly connected excitatory neurons, together with an inhibitory interneuron that sets their thresholds; both the degree of connectivity between the neurons and the level of firing in the stored memories can he set arbitrarily. Conventional statistical methods were applied to analyze its dynamics. The network dynamics are contained in a set of four coupled difference equations. In {46], Gibson, et al. studied its stability and bifurcation structure. The equilibrium properties were investigated analytically in certain limiting cases and numerically in the general case. The regions of parameter space corresponding to stable and unstable behavior were mapped. It was shown that for suitable parameter choices, the network possesses stable fixed points which correspond to memory retrieval. Yen【57] proposed for the BAMs a computationally efficient synthesis procedure which possesses possibilities of exerting control over the number of spurious states, estimating the basins of attraction and effectively storing a nurrtber of desired stable memories. Schwenker, et al. [38] investigated the pattern completion performance of neural autoassociative memories composed of binary thresholded neurons for sparsely coded binary memory patterns. By focusing on iterative retrieval, they were able to introduce effective threshold control strategies. In a finite-size autoassociative memory, they showed that iterative retrieval achieves higher capacity and better error correction capability than one step-retrieval. 2.2.2 Network Structure Other than the learning scheme, a lot of researchers have tried to decrease noise and/or increase signal to improve storage capacities by means of modifying network structures of the Hopfield-type associative memories. Simpson [40] proposed a higher-order BAM and then a higher-order intraconnected BAMs. Tan [45] devised a data-driven approach by modifying the configuration of the Hopfield network to allow for hidden structures which are made by appended vectors. The correlation between the Hopfield and hidden networks is made hy the cross products of the FMs and appended vectors. The results showed that if the set of the FMs are mutually orthogonal, then all of them are fixed points (FPs). If the set of the FMs are not mutually orthogonal, they can be orthogonalized by appending to them the appropriate bipolar vectors. Wang and Lee [54] constructed a modified bidirectional decoding strategy so that all the given training pairs are guaranteed to be recalled successfully with the aid of the augmented correlation matrices, the long term memory traces and the activating functions. The model saves much more extra connecting weights than the dummy augmentation method [50]. Kang [21 ] proposed a triple-layered hybrid associative memory. The first synapse is a one-shot associative memory using the modified Kohonen's adaptive leaning algorithm with arbitrary input patterns. The second one is Kosko's BAM consisting of orthogonal input/output basis 32 vectors. The third one is a simple one-shot associative memory with arbitrary output images. Xu, et al. [56] proposed an asymmetric BAM. It not only can cater for the logical asymmetry of interconnections but also is capable of accommodating a lager number of non-orthogonal patterns. Buckingham and Willshaw [7] presented five different strategies for setting the thresholds of units in partially connected networks. Stiefvater, et a). [42] suggested a general design mechanism for the construction of a local neighborhood structure using a statistical analysis of an arbitrary given pattern set. Karlholm [22] gave a study of recurrent associative memories with exclusively short-range connections. Higher-order couplings were used to increase the capacity. Results showed that perfect learning of random patterns is difficult for very short coupling ranges. However, by choosing ranges longer than certain limiting sizes, depending on network size and order, the theoretical capacity limit can be closely reached. Those limiting sizes increase very slowly with the network size. It was found from simulations that even networks with coupling ranges below the limiting size, they are able to complete input patterns with more than 10% error. 2.2.3 Nonmonotone Dynamics The Hopfield model uses the monotone dynamics to update neuron states. Some researchers adopts nonmonotone dynamics for the evolution of networks. Morita [28] examined the dynamics of the auto-associative memory, and presented a novel neural dynamics which greatly enhances the ability of the auto-associative memory. This dynamics works in such a way that the output of some particular neurons is reversed (discrete) or the output function is not sigmoid but nonmonotonic (analog). It is because both processing methods keep the variances of the weighted summed inputs of neurons, and thus 33 the sum of squares of the overlappings with all the FMs except the target FM, from growing too large. Yoshizawa, et al. [59] derived two kinds of theoretical estimates of the absolute capacity of the network proposed in [28]. One gave the upper bound 0.5n and the other gave the average value 0.4n，which fit well with computer simulations. In Nishimori and Opris [30], it was found that a certain type of nonmonotonic input-output relation of a single neuron yields an enhanced memory capacity compared with the conventional monotonic relation. The maximum capacity obtained is 0.22n which is quite close to the prediction made in [28], [59] by simulations of the particular reverse method. De Felice, et al. [15] described the process of the activation of a generic neuron by a nonmonotone increasing output function. An iterative algorithm, called Edinburgh algorithm was chosen as a learning rule as a modification of the Hebbian rule. Wang [49] designed a simple learning approach to store all training patterns with basins of attraction as large as possible. In the recall phase, a dynamic threshold rule which aims at reducing the probability of converging to spurious states by using a gradually decreased threshold was applied. Araki and Saito [2] utilized time-variant self-feedback connections. The synthesis procedure guarantees the storage of any desired memory set. Englisch and Herrmann [ 14] proposed a model which can reach the theoretically maximal storage capacity in the limit of strong bias when an optimized external field is introduced. The information processing abilities were calculated in the strongly diluted approximation, and a version of the model with nonlinear synapses was dealt with. In Shirazi, et al. (39], Hopfield network with ^n < 1) malfunctioning neurons was considered, and the asymptotic storage capacity of such a network was derived as a function of the parameter ^ under stability and attractivity requirements. 34 2.3 Local Distribution Storage Local distribution storage means that each connection weight represents only one element of an FM, or one element of an FM is represented by one connection weight. Accordingly, no cross-talk exists in the storage of all the FMs. The storage capacity thus obtained is without doubt high. Besides, some researchers adopt nonlinear dynamics to enlarge differences 、 1 between auto- and mutual-correlations to greatly improve storage capacities. Lippman [25] pointed out that the Hamming network has a number of obvious advantages over the Hopfield network. The Hamming network implements the optimal minimum error classifier, requires much fewer connections, does not suffer from spurious output patterns which can produce a "no-match" result. The number of connections in the Hamming network grows linearly with n (the dimension of the FMs) while that of the Hopfield network grows as % the square of n. Meilijson, et al. [27] showed that the activation function of the memory neurons in the original Hamming network may be replaced by a simple threshold function. By judiciously determining the threshold value, the “winner-take-all” subnet of the Hamming network may be altogether discarded. The storage capacity of the resulting network can grow r exponentially with n. Chiueh and Goodman [8], [9] proposed a model called RCAM (Recurrent Correlation Associative Memory) for a class of high-capacity auto-associative memories. Its structure is based on a two-layer recurrent neural network and its operation depends on the correlation measure. It is asymptotically stable in both the synchronous and asynchronous modes as long as the weighting function is continuous and monotone nondecreasing. The asymptotical storage capacity of the RCAM's special case - ECAM (Exponential Correlation Associative Memory) scales exponentially with the length of the 35 FMs, and the ultimate upper bound for the capacity of associative memories is met. Wang and Don [53] proposed an exponential BAM which provides a significantly high storage capacity for pattern pairs. It utilizes an exponential scheme to magnify the SNR such that the enhancement of the similarity between an input pair and its nearest stored pattern is feasible. Zhang, et al. [60] discussed the BAM from the matched-filtering viewpoint. Sufficient and/or necessary conditions for the stability and attFactivity of equilibrium states were presented. The exponential function was also used to enhance the correlations between the binary vectors of the retrieval key and that of the stored pattern similar to the key to improve the BAM's performance. 2.4 Transformation Models Transformation approaches are more like those introduced in the field of signal, system and control. Transformation models arc conceptually different from the Hopfield-type models in storing the FMs. In Olivier [32】，a procedure for designing linear associative memories allows for exact data extraction from uncorrupted keys and minimizes the error between the data vector and the extracted vector when the key vector is corrupted hy noise. Zou and Lu [61 ] proposed an approach to design the linear associative memories which are optimal in rejection of general colored noise in the sense of the least mean square error. Hunt, et al. [19] proposed a model which is composed of a nonlinear transformation in the spectral domain followed by the association. The Moore-Pen rose pseudoinverst is employed to obtain the least-square optimal solution. 36 2.5 Concluding Remarks and Prospectus of this Thesis We have made an overall review of the related works in neural associative memories in the following aspects: capacities of the Hopfield-type associative memories, improvements based upon global distribution storage in terms of learning scheme, network structure and nonmonotone dynamics, local distribution storage, and transformation models. Roughly, most of the current studies on neural associative memories can be categorized into one of these aspects. The work of this thesis is also directed in these fields. As we have seen, in order to improve the storage capacity of the Hopfield network, much research has been done within its framework in the direction of improving the learning schemes and network structures. We consider the uniform encoding/learning of the FMs used in the Hopfield-type networks has room for improvement, and hence we propose weighted encoding/learning in which different learning weights are directed to the associative recall of different FMs. Thereafter, we empirically find and then theoretically prove that there exists a parameter, called signal to noise ratio gain (SNRG), which is critical to the successful store/recall of FMs. With respect to the learning scheme, several algorithms capable of adaptively finding learning weights and neuronal parameters are proposed to construct a weighted outer-product learning associative memory (WOPLAM). The WOPLAM is efficient in storing and recalling the FMs. Although much research has been conducted to improve the storage capacity of the Hopfield-type networks, their core processing part - the learning algorithm, still belongs to the simple outer-product rule. So, what these research can achieve is clearly limited. In this respect, we propose a novel encoding/learning strategy within the framework of the Hopfield-type associative memories. In the novel encoding/learning method, the connection weights are not just decided by the summed products of the corresponding two bits of all the FMs as in the Hopfield-type neural associative memories. Summed products of a number of the neighboring bits of the corresponding two bits in all the FMs are also taken into account. This is the significant difference in the encoding/learning mechanism between the proposed neural associative memory and the Hopfield-type models. Nevertheless, the network structure of the proposed model is the same as the Hopfield model except that it is with self-feedback connections. The retrieval process is also identical to that of the Hopfield model. With the novel encoding/learning method being applied, an extreme stability of the FMs can be reached such that all of the FMs are FPs. Both of the aforementioned studies lie in the framework of the global distribution storage of the FMs. Since it has been found that the local distribution storage of the FMs has an advantage over the global distribution storage in greatly depressing the cross-talk among the FMs as in the ECAM, we make an effort towards encoding the FMs by local distribution storage. A Gaussian correlation associative memory (GCAM) is proposed. It is capable of overcoming a shortage of limited output responses in circuits implementation suffered by the ECAM but still has attractive performances such as the exponentially-growing storage capacity, and the controllable radius of attractions through adjusting parameters of the Gaussian function. In addition to the studies described above, we are still very interested in the problems like: Why the Hopfield network has a very small storage capacity, and what factors affect its storage capacity. To answer these questions, a further investigation into the upper bound of 38 the asymptotic storage capacity of the Hopfield network is presented in chapter 6. The main result obtained is that the asymptotic storage capacity of the Hopfield network, in order that all the FMs can be exactly recoverable, does not grow directly proportionally or proportionally to the dimension (n) of the FMs but is upper hounded as n approaches infinity. The upper bound is not directly decided or decided hy n, hut is directly determined by the distribution of the elements of the FMs. 39 Chapter 3 Adaptive Weighted Outer-Product Learning Associative Memory 3.1 Introduction The storage capacity of the Hopfield network has been found to be severely constrained by the dimension of the ftindamental memories (FMs). Empirical【18] and theoretical [26] results show that its storage capacity, i.e., the number of stably stored FMs, is in general less than 15 percent of the FMs' dimension. Generally, the quantity and quality of the recall deteriorate as the number of the FMs increases. Sometimes, even when the number is small, the recalls may not be all correct (See section 3.5.1 )• To alleviate the limitation, various approaches have been attempted to increase the storage capacity over the years [3], [8】，[9], f24], [41 ], [48], [50], [51), [55]. Due to the nature of the FMs, we postulate that by assigning different learning weights to different FMs, the storage capacity of the Hopfield network can be substantially increased. To 、 achieve this goal, we put forth in this paper a Hopfield-type weighted outer-product learning associative memory (WOPLAM) with several adaptive algorithms. In this chapter, associative memory neural networks with adaptive weighted outer-product learning are proposed. For the correct recall of an FM, a corresponding learning weight is attached and a parameter called signal to noise ratio gain (SNRG) is devised. The sufficient conditions for the learning weights and the SNRGs are derived. It is found both empirically and theoretically that the SNRGs have their own threshold values for correct recalls of the corresponding FMs. Based on the gradient-descent-search approach, several algorithms are constructed to adaptively find optimal learning weights with reference to global- or local-error measure. 40 Section 3.2 briefly recapitulates the background of the Hopfield network and introduces the adaptive WOPLAM for performing associative memories. The sufficient conditions of the learning weights and the signal to noise ratio gains (SNRGs) are then derived in section 3.3. With respect to the WOPLAM, we present in section 3.4 several adaptive algorithms to dynam cally find the optimal learning weights on the basis of the gradient-descent approach. Computer simulation results are given in section 3.5 to substantiate the theoretical arguments. We then conclude the paper by a summary of the study. 3.2 The Adaptive Weighted Outer-Product Learning Associative Memory 3.2.1 The Hopfield Associative Memory To facilitate our discussion, we first outline in brief the Hopfield network as follows: Given a set of m FMs in a Hopfield network, the weight W„H (i, j = 1,2，••.，n) of the connection matrix is obtained as: in lu.^-u；0, if i n r=l W•丨H = { (3-1) where W,,H is the connection strength between neurons i and j in the Hopfield network; n is the dimension of the pattern vectors; u⑷，r = 1,2，…，m, are FMs. If sgn{ [W„H] u is a fixed point (FP) of the Hopfield network; [W„H] u(p) is a matrix-vector product; and sgn( . *) is a sign function applied to all elements in a vector resulting from the matrix-vector product. The block diagram of the Hopfield network is given below. 41 The block diagram of the Hopfield model. WH = [W./‘] is the connection W丨丨”i, j = I, 2,…，II，is the connection weight between neurons i and j. x,. i = the input of the network, o,. i = 1,2，“，11, is the output of the network. The retrieval process is as follows: y ^ l W . ^ x , , -j=i (3-2) o,= sgn (y,), where xJf j = 1, 2, •••, n, is the network's input; y, is the summed input of neuron i, and o, is the output of neuron i, i = 1 , 2 ， n ; and sgn( • ) is the sign function. Assume one of the FMs, u: E[|(n-l)u,,p,|] (n-l) ,/2 = (3-4) in n (Var( I I u,(r, u;r) ur)) , / 2 (m-l) ,/2 . r= I 冲 •沟 43 Relatively speaking, the larger is the SNR, the more stable the FMs become. 3.2.2 The Weighted Outer-Product Learning Associative Memory To improve the storage capacity of the Hopfield network, we consider it necessary to allocate different learning weights to different FMs because of their varying nature. The model thus constructed is called the weighted outer-product learning associative memory (WOPLAM) in which the connection weights are obtained as: • m I o T i T U广， if i 关 j, w„w 二 { (3-5) where a(rl is a learning weight associated with the recall of FM u 0 ) is a real number, r=l, 2,…，m; and W丨广 is the connection strength between neurons i and j (i, j = 1,2，…’ n) in the WOPLAM. Definitions for the FM and FP of the WOPLAM are the same as those of the Hopfield network. (In section 3.4, we will present adaptive algorithms to find the optimal learning weights.) To have an insight into the learning weights for successful recalls, we deduce a necessary condition for the learning weights as follows. Theorem 3-1: If FM u(s) is an FP in the WOPLAM, then its corresponding learning weight satisfies the following condition: in a ( s ) > ( D H - D H ' ) / [ 4 ( n - l ) ] + m - I a,r\ s = 1，2，"., m, (3-6) n n n n where D H = - I I u, 三 (3-7a) ( I (a,k>r )l/2 k=l ‘印 (m— l)l/2_ a ,p' > a印 ( L ) 2.(2t)l/2. q(p)- n (m- 1) (n - 2pn + I) > 1/2 (3-7b) or 2 (2t),/2 • q(p)- n I a' (n-2pn+ I) a(p) > ik) < p < m. (3-8) 45 the asymptotic storage capacity of the WOPLAM will grow at the greatest rate so that after one cycle, the bit-error probability (the probability that a hit in the next-state vector is different from the corresponding bit in the nearest FM) is less than (4m) l/2e 'as n approaches infinity (Here t is a fixed and large number, and q( • ) is a nonlinear monotonically decreasing function, 0 < q( • ) < 1). It should be noted that the right-hand side of ^quality (3-7b) can be treated as the thresholds of G's. They are definite and real-valued with respect to a given set of FMs. G, r = 1,2，...，m, are important parameters for associative store/recall. Their threshold values imply the degrees to which the associative store of the corresponding FMs are required. A larger (smaller) threshold value of an FM implies that it is more difficult (easier) to make this FM an FP (to be discussed in this section). The learning weights ex's can be varied and can take on different values to meet inequality (3-8). Based on the above theorem, the following four corollaries are obtained. Corollary 1: In a WOPLAM, any FM can be correctly recalled when its G satisfies inequality (3-7b). Corollary 2: In a WOPLAM, any FM can he correctly recalled when its learning weight satisfies inequality (3-8). Corollary 3: In a WOPLAM, as many FMs as possible or all of them can be correctly recalled when as many corresponding G's as possihjc or all of them satisfy inequality (3-7b). Corollary 4: In a WOPLAM, as many FMs as possible or all of them can be correctly recalled A >： when as many corresponding learning weights as possible or all of them satisfy inequality (3-8). Now, let us examine the learning weights and the parameters G's. Firstly, inequalities (3-7b) and (3-8) imply that thresholds of all G's are equal and so are all the learning weights. However, 46 this conclusion is dependent upon the assumption that all FMs are strictly generated by symmetric Bernoulli trials. Actually, neither the learning weights nor the G's thresholds are equal. Given a set of FMs, some of them have larger G's thresholds, and need relatively larger learning weights while the others have smaller G's thresholds and need relatively smaller learning weights. The larger (smaller) the G's threshold is, the greater (smaller) the degree of associative store for the corresponding FM is required. Secondly, by examining eqn.(3-7a) and, inequalities (3-7h) and (3-8) closely, it is clear that given any set of FMs, generally not all of the learning weights can satisfy their conditions simultaneously, neither can all of the G's. To achieve the maximum number of correctly recalled FMs, learning weights must he chosen properly—neither too small nor too large, so that the total number of learning weights or G's that can meet their sufficient conditions can be maximized. This can be accomplished by making sonie trade-offs in deciding on the learning weights. If the G's threshold of an FM is high (low) then its learning weight should he large (small) as well. The strategy basically is to find a set of optimal G's so that the WOPLAM will give the maximum number of correctly recalled FMs. In what follows, we analyze the G's from the perspective of SNR (Signal to Noise Ratio) analysis [26], [48], [9], Through such analysis, we can compare the differences between the Hopfield network and the WOPLAM in performing associative memories, and gain an insight into the physical meaning of the G's. We can also see the advantages of the WOPLAM and consequently the mechanism of the outer-product type associative memories. 47 Assume that the FMs are randomly generated by symmetric Bernoulli trials with -1 and 1 as possible outcomes. Suppose one of the FMs, u),=(n-"k + i i ^-ur-ur-ur. (3-9) With respect to the recall of the ith component of the pth FM, the first term in the right-hand side of eqn.(3-9) can be viewed as a "signal" while the second term as a "noise". Therefore, we obtain the SNR of the ith component of the pth FM as: E[|(n-l)a , pV p , l ] (Var( 1 I a(f,u:r,-u;r,-u;p,)),/2 (n-l)a , p ' ( n - D ^ a ^ 、 = = (3-10) m 111 ((n-l) I(a,r>)2),/2 ( I (aiy),c . r=l.*p r=l.*p In the case of the Hopfield network, with a uniform storage (all learning weights being set to 1) of all the FMs, the SNR of the ith component of the pth FM becomes E[|(n-l).url】（n-丨严 = (3-11) in n (Var( I Zu/ r).u,VP,” l /2 (m-l) , /2 . Considering that an FM cannot be correctly recalled by the Hopfield network but can be correctly recalled by the WOPLAM, then we hypothesize that there exists a gain of the SNR in the WOPLAM over the Hopfield network. Hence, from eqns.(3-10) and (3-11), the gain of the SNR, by the WOPLAM, of the ith component of the pth FM is 48 (n-l),/2a,p' ‘ m (I(a ,rV) l /2 ( m - l ^ . o T r=l.印 = (3-12) m (n-D,/2 ( I (a{y) i r2 . r=l.*p 4 / l、l/2 ( m - 1 ) With reference to eqn.(3-7a), we can see that the definition of the G's is the same as that in eqn.(3-12). Hence, we can consider the G's as the signal to noise ratio gain (SNRG). In the Hopfield network, all the FMs are equally stored. It is equivalent to the WOPLAM using a unit learning weight for each FM. If all the FMs are strictly generated by symmetric Bernoulli trials, then they will have the same SNR as in eqn.(3-l I) for their recall in a probabilistic sense. However, this rarely happens, especially in real applications. In fact, given any set of FMs, it might he easier to learn some but more difficult to learn the others. Easy-to-learn FMs might need smaller SNRs (might be smaller than that in eqn.(3-l I)) while difficult-to-Icarn FMs might need larger SNRs (might be larger than that in eqn.(3-11)). Therefore, storing the FMs equally in the Hopfield network is inadequate and ineffective when the FMs are not randomly generated. This problem, nevertheless, can be handled by the WOPLAM 3.4 Search of the Learning Weights Through Adaptive Learning In this section, we present several algorithms for adaptive search of the learning weights. They are based on the gradient-descent approach [35]. Although this approach is subject to al! the usual problems such as local trapping, initialization and numerical convergence, it can still locate the optimal learning weights. It is worth mentioning that there are some other alternatives for optimization computation. For example, by combining characteristics of the simulated annealing 49 algorithm arid neural network, Van Den Bout and Miller [47】developed an algorithm for graph partitioning called MFA (Mean Field Annealing) which exhibits rapid convergence resulted from the neural network while preserving the solution quality afforded hy simulated annealing. Initially, we have no information on the optima) learning weights. The adaptive algorithm is controlled by minimizing the sum of squared errors between the FMs and their corresponding temporal output vectors of the network when these FMs are used as the respective inputs of the WOPLAM. The elements of the temporal output vectors take on real values from -1.0 to 1.0 during the learning process. The incorporation of a continuous optimization technique into a bipolar design sometimes does yield a better design than methods simply based on a hard bipolar design process. s In the following subsections, we will describe several adaptive algorithms for optimizing the 9 learning weights based upon different error measures. We first present the glohal-error-measure algorithm which searches for the optimal learning weights based on the overall system error. To reduce computation complexity, we propose a local-error-rrjcasure algorithm in which the optimal learning weights are searched with reference to only the errors of the FMs concerned. To accelerate the adaptive process of both algorithms, we then construct two other algorithms by adaptively adjusting the key parameters in the global- and local-error-measure algorithms respectively. 3.4.1 Global-Error-Measure Algorithm Assume that one of the FMs, u

(I < p < m), is taken as an input to the network, then the corresponding summed input of the ith neuron (1 < i < n) is: m n y,(p)= ([W.H u^), = (n- 1)- oT. u,,p) + I I a(". u,⑴-u,m- u/p,. (3-13) 50 To adaptively update the learning weights, a sigmoid function is adopted to produce the neuron's output or state as: 2 o,,p) = f(y,

) = 1 . (3-14) 1 +exp(-p,(yi,p,-p2)) The parameter Pi controls the steepness of this function which is symmetrical around (3：. For the ith element of the FM, u 0) is a real-valued parameter controlling the convergence rate of the updating process, dF m n dF：0 m n dF：0 df dy,⑴ 一 = 1 1 一： X I da (p) ,=. aa (p) r=i,=i df dy, (r> da (p} m n = 1 I ( 一 2 ) (u,,r>- f(y,<")H-2H-pl> exp(-pl(y:"- P：)) iur-u; p ,-u,m (I +exp(-P丨(y:"- p2)))2 m n n (t) r / - . ( r K v / I f^/y.^w V . . = -r| (3-21) da!p>， F = i F r , 1=1 dF n ()F:P> n clfr df dyr 一 = I 一 = I aoT aa (pl ,=1 r)f cly, (p) doT n =-Pr(n-l)-1(u, , p )- f(y,- p:)(丨一 f2(y广>). (3-26) r=l i=l With respect to p2，we have 3F Ap2 = -r| ， (3-27) ap： d F m n 3 F , ' r > tn n O F , … ^ f ——=11 ——=11 邓2 r^ l i:l 丨:l 邓： m n = P, I I (u,m - f(y,<"))•( 1 - f2(y/")). (3-28) r^ l i=l Hence, the updating rule for Pi is: d F P,(t+l) = P , ( t ) - T l api(t) 54 =Pi(t) + Ti l I (u•⑴-f( W (y,‘" 一 (i2(t))- (1 - fV丨))； (3-29) r^ l i=l and the updating rule for (3： is dF p2(t+l) = P:U)-n it) n =(3:(t) - n P . ( t )-11 (u,,r'- f(y,,r>)) ( I - f2(y/")). (3-30) n=!i=l From eqns.(3-29) and (3-30), we can estimate that the computational complexity of adjusting pi and P： is lOmn. Adding this improvement only leads to a slight increase in computational complexity, but it greatly accelerates the adaptive process while maintaining the computation quality. It should be noted that in eqns.(3-19), (3-23), (3-29) and (3-30), we, for simplicity of description, use a uniform real-valued parameter r| to control the convergence rates of updating the learning weights, |5| and P:. Actually, r\ will be different for different cases. The new algorithm based on the glohal-error-nieasure including the adaptive adjustments of Pi and p2，is described as follows: (a) Initially set all learning weights to 1. Initialize P and 6. Construct a WOPLAM with eqn.(3-5). (b) Sequentially input the FMs and compute the neurons' summed inputs and temporal outputs with eqns.(3-13) and (3-14). (c) Adjust pi and p2 with eqns.(3-29) and (3-30) respectively. (d) Update the learning weights using eqn.(3-19). (e) Reconstruct the WOPLAM with eqn.(3-5) using newly obtained learning weights. (f) Repeat by going back to step (b) until all the learning weights, (3 and Bconverge. The modified algorithm based on the local-error-measure is the same except for cqn.(3-19) in step (d) being replaced by cqn.(3-23). 3.5 Experimental Results This section is divided into two parts. The first part involves the use of representative examples to visualize the importance of the SNRGs for associative store and recail. The second part shows computer simulation results of the WOPLAMs using adaptive algorithms to recall randomly generated FMs. 3.5.1 Representative Examples In this subsection, we shall demonstrate the importance of the learning weights and the SNRGs without using any adaptive algorithm. In the computer simulations, the computation of associative recall is conducted in a synchronous mode. We would like to observe whether or not the FMs are fixed points (FPs). Fig.3-1(a) shows the four digital images (in black and white) used. Each has 60 rows and 40 columns giving a total of 2400 pixels. In the simulations, the FMs are obtained by sampling every other pixel resulting in 1200 elements (-1 or +1) in each FM. Fig.3- 1(h) is the result using the Hopfield network which gives only one FP. Through a good number of tests, rt is found empirically that FMs 1, 2, 3 and 4 can become FPs when G , h , G<2), G(1) and G,4) are larger than or equal to 0.8115, 1.1882，1.2603 and 1.1547, « respectively. It because the distribution of the elements (—1 or +1) of the four FMs are different, their thresholds are naturally different. Although these thresholds are empirically obtained and may 56 be slightly different from the optimal ones, they can shed some light on the importance of the SNRGs' thresholds to the associative recall performance. The experiments are repeated with other sets of FMs and similar results are obtained. From these results, we conclude that the SNRGs have their own thresholds and the FMs can he correctly recalled when their corresponding SNRGs are larger than or equal to their thresholds. Next, we try to maximize the number of FPs by maximizing the number of the learning weights with their corresponding SNRGs being larger than or equal to their thresholds. By eqn.(3-12), we find that only three of the SNRGs can he made larger than or equal to their corresponding thresholds at any one time. Due to such a limitation, the maximum number of FPs is three in this case. We can choose any three of the FMs to be FPs by appropriately setting the learning weights as shown in Fig.3-2. Tables 3-1 and 3-2 give the learning weights and SNRGs for the cases shown in Figs.3-1 and 3-2 respectively. From the values given, we can see that the effect of using the set of learning weights (2，2, 2, 2) is the same as that of learning weights 11, 1, I，I). Actually, a set of learning weights can be normalized. 57 Figure 3-1. Expci iiiicmal results obtained hy using the WOPLAM with different sets of learning weights ((X111. (Xl2), (x<”,（x'“). All sets ol learning weights arc chosen arbitrarily, (a) The four FMs (Tlicy are taken as the network's inputs), (h)-(m) Results of the WOPLAM by different sets of learning weights, (h) (I, I, I, I). (The ease of the Hopfield network), (c) (2，2, 2，2). (d) (I, 2, 2, 2). (e) (2. 3, 3, 3). (0(1 , L 1,2). (g) (2, 2, 3, 2). (h) (1,丨，2, 2). (i) (3’ 4，4’ 3). (j) (2. 3, 2’ 2). (k) (4, 6, 6’ 4). (I) (4, 5, 5，4). (m) (5, 6, 6, 5). 58 (a) 里4，igure 3-2. limpirical results obtained hy clllcieiitly utilizing the WOPLAM with different sets of learning weights (« l h , (x<2), (/”，a"'). All sets of learning weights arc empirically found and chosen so that as many SNRGs as jxjssihlc aic nuidc lo he larger than or equal to their corresponding thresholds. In (his example, the maximum number of FFs is 3. (a) The four FMs (They are taken as the network's inputs), (h)-(e) Results of the WOPLAM with different sets of learning weights, (b) (4.2, 5.7, 5.7, 3.0). (c) (0.7, 3.4, 3.5, 3.3). (d) (3.0，4.0,0.7, 4.0). (e) (2.7, 0.7，3.4, 3.5). 59 a(1) a(2) a(3) (4i a ' G⑴ G(2) G(3) G(4) 1 1 1 1 1.0000 1.0000 1.0000 1.0000 2 2 2 2 1.0000 1.0000 1.0000 1.0000 1 2 2 2 0.5000 1.1547 1.1547 1.1547 2 3 3 3 0.6667 1.1078 1.1078 1.1078 1 1 1 2 0.7071 0.7071 0.7071 2.0000 2 2 3 2 0.8402 0.8402 1.5000 0.8402 1 1 2 2 0.5774 0.5774 1.4142 1.4142 3 . 4 4 3 0.8115 1.1882 1.1882 0.8115 2 3 2 2 0.8402 1.5000 0.8402 0.8402 4 6 6 4 0.7385 1.2603 1.2603 0.7385 4 5 5 4 0.8528 1.1471 1.1471 0.8528 5 6 6 5 0.8793 1.1206 1.1206 0.8793 Table 3-1. Learning weights and SNRGs of the cases in Fig.3-l(bHm). The single underlined SNRGs are the ones sufficient for their corresponding FMs to be FPs. The double underlined ones are the empirically obtained thresholds for their corresponding FMs to he FPs. AO) A(2) A(3) A(4) G(1) G(2) G⑶ G(4) 4.2 5.7 5.7 3.0 0.8458 1.2839 1.2839 0.5717 0.7 3.4 3.5 3.3 0.2058 1.2115 1.2657 1.1595 3.0 4.0 0.7 4.0 0.9116 1.3723 0.1894 1.3723 2.7 0.7 3.4 3.5 0.9487 0.2174 1.3158 1.3785 Table 3-2. Learning weights and SNRGs of the cases in Fig.3-2(bHe). The "underlined" has the same meaning as that in Table 3-1. 60 3,5.2 Computer Simulations of the Adaptive Algorithms In this subsection, we evaluate our adaptive WOPLAMs with randomly generated FMs. All experiments are conducted by varying the number of FMs (m) with the dimension (n) fixed. The recall results obtained by various adaptive algorithms are depicted in Figs.3-3 一 3-5. In these figures, "Local WOPLAM 1" and "Local WOPLAM 2" represent respectively the local-error-measure WOPLAMs without arid with the adjustments of Pi and p： in eqn.(3-14). On the other hand, "Global WOPLAM I" and "Global WOPLAM 2" represent respectively the globaJ-error-measure WOPLAMs without and with the adjustments of Pi and p2 in eqn.(3-14). Every point on all the curves in these figures is the mean of the results obtained from 10 different sets of data. Generally, the WOPLAMs perform better than the Hopfield network in terms of storage capacity and associative recall. From these figures, it can be observed that the storage capacities of the WOPLAMs with the adaptive algorithms vary approximately between 20 to 30 percent of the FMs' dimension. It should be mentioned that the results depend on the value of r| in eqns.(3-19), (3-23), (3-29) and (3-30)，and the initial values of pi and p： in eqn.(3-14). We did not fine tune these values in our simulations, otherwise better results could be obtained. When m is small there is almost no difference among the four kinds of WOPLAMs in terms of the storage capacity and associative recall. When m becomes large, the global WOPLAMs have larger storage capacities and better association abilities than the local ones. The reasons for these results are that in the "local" algorithm the updating of each learning weight is decided by its own behavior and the interactions between different learning weights are not taken into account. While in a "global" algorithm the updating of each learning weight is decided not only by its own behavior but also by that of all other learning weights. Hence, the interactions between different learning 61 weights are accounted for. When m is small those interactions are small and can almost be neglected, resulting in the global and local algorithms having almost equivalent performances. But when m is large, these interactions are great and cannot he neglected any more. Hence, the "global" algorithms provide better association performances than the "local" ones. The two global WOPLAMs，without and with the adjustments of pi and p:, have similar association performances. Similarly, the two local WOPLAMs also have similar association performances. Simulation results also reveal that the adjustments of Pi and p2 greatly accelerate the adaptation process without degenerating the overall association performances. Results concerning the convergence speeds are depicted in Figs.3-6 一 3-8. The computation speed can be improved by an order of magnitude. Despite the obvious higher complexity of the global model compared with that of the local one，the addition of the acceleration algorithm (by adjusting Pi and p：) to the global model results in a computation speed faster than that of local WOPLAM 1. We can summarize our simulation results of the WOPLAMs into two points as follows: (a) The accelerated global WOPLAM gains substantially in convergence speed without any loss in the storage capacity and hence is considered the best amongst the four new models. (b) Considering speed alone, the local WOPLAM with the adjustments of Pi and (3： has the fastest convergence but does not yield ideal storage capacity. 62 i r / x Global WOPLAM 1 \ \ \ ( o Global WOPLAM 2 \ + Local WOPLAM 1 # Local WOPLAM 2 Hopfield network 2 3 4 5 6 7 8 9 10 11 The number of FMs(m) Figure 3-3. Simulation results of n=10. u 4.5 .53 5 cri2. sdLL)0」① c j u u n u e q h -63 Figure 3-4. Simulation results of n=20. x Global WOPLAM 1 o Global WOPLAM 2 + Local WOPLAM 1 * Local WOPLAM 2 Hopfield network 5 6 7 The number of FMs(m) 8 9 10 S 0 . L L1 0 j ① q i u e l l H 64 Figure 3-5. Simulation results of n=50. x Global WOPLAM 1 o Global WOPLAM 2 + Local WOPLAM 1 * Local WOPLAM 2 Hopfield network 10 15 The number of FMs(m) sdLLoj ① < q i u n u ① qH 65 3500 Figure 3-6. Convergence speed comparison between different WOPLAMs(n=1Q). The number of FMs(m) 3000 2500 2000 500 000 500 0a 2 S S 6 J 9 A C 8 u J o l s u o ! m 」①二 1 0 J e q i u © l i h -66 x Global WOPLAM 1 + Local WOPLAM 1 o Global WOPLAM 2 * Local WOPLAM 2 3 4 -gT" 5 6 7 The number of FMs(m) 8 丄 9 10 67 > Figure 3-7. Convergence speed comparison between different WOPLAMs(n=20). i cnn 000 500 8 S 6 J ① >U8mc 一 )」 0一 suo!m」el j》0 J a ) q i u ① 1800 Figure 3-8. Convergence speed comparison between different WOPLAMs(n=5Q). The number of FMs(m) 30)0)000000 64208C6C4020 1111 e o u a o ) 」 a ) > u o o I E U ! ) 」 0 》 s u o ! l e 」 9 J j» • — 〇 JgqEnc0 68 1 Finally, it is worth mentioning that using the adaptive algorithms can achieve the minimization of the overall-error between network outputs and their corresponding FMs. When the number of FMs is small, minimizing such an overall-error is almost equivalent to having all of the FMs be FPs. However, when the number of FMs is large, the number of FPs achieved under such kind of error measure may only be a maximum one. In this case, a maximum number of FPs will he obtained by sacrificing a small (minimum) number of FMs (hy allocating very small learning weights to them) if needed. In other words, in the WOPLAM, given a set of FMs, if it is impossible to make all the FMs be FPs, then a maximum number ot FMs will he made to become FPs at the expense of the stabilities of the other remaining FMs. 3.6 Conclusion The concept of adaptive weighted outer-product learning has been proposed for the encoding of the FMs in Hopfield-type networks. The model - WOPLAM proposed improves the performances of associative storage and recall of the FMs. Sufficient conditions for the learning weights and SNRGs are deduced in a probabilistic context. It has been demonstrated that each SNRG has its own threshold and any FM can be correctly recalled when its SNRG is larger than or equal to its threshold. In principle, the asymptotic storage capacity of the WOPLAM will grow at the greatest rate when all the SNRGs or learning weights satisfy the corresponding sufficient conditions. Several gradient-descent algorithms capable of dynamically finding the optimal learning weights have also been implemented. In general, the WOPLAM, with the use of an adaptive algorithm, has the storage capacity of up to 0.2n to 0.3n or above. 69 Chapter 4 A Novel Neural Associative Memory with Maximum Stability 4.1 Introduction Hopfield proposed an outer-product based neural associative memory in his seminal paper • [18]. Since its storage capacity was found to be severely constrained both empirically [18] and theoretically [26], much research has been done for the improvements of the storage capacity and the associative recall by various approaches [3],【8], [9], [28j, [30], [48], |50], [51】，[54|, [56], [59], [60]. It should be pointed out that the examples listed on the BAM (Bidirectional Associative Memory) is in principle also effective for the Hopfield-type neural associative memories. In other words, the examples cited above can improve the performances in terms of the storage capacity and associative recall at the expense of very high extra costs, such as computation, network structure and hardware implementation complexities. In this chapter, we propose a novel neural associative memory based on a new encoding strategy. Unlike the pointwise outer-product rule commonly used in the Hopfield-type models, the novel encoding method computes the connection weight between any two neurons (including self-feedback connections) by summing up not only the products of the corresponding two bits of all the fundamental memories (FMs), but also the products of their neighboring bits as well. The widely-used signal to noise ratio (SNR) and dynamic approaches are utilized to theoretically investigate the performances in terms of stability and attractivity. Both theoretical and experimental results show that the proposed modeKs very^powerful in maximizing the stability and thus making as many FMs as possible be fixed points ^FPs). 70 In the following part of this chapter, section 4.2 presents the novel encoding strategy for the neural associative memory. The stability and attractivity of the proposed model are analyzed theoretically in detail in section 4.3. Simulation results are given in section 4.4， followed by some concluding remarks. 4.2 A Novel Encoding Strategy Based Neural Associative Memory Assume there are m FMs, u n, we set iV" = u卜”⑴.In eqn.(4-1 )，when L = 1, the proposed neural associative memory reduces to the first-order outer-product neural associative memory which is the same as the Hopfield model except for the self-feedback connections. Suppose one of the FMs, u(p), is taken as a retrieval key input, then we have y,= 口|丨.1]广 (4-2) J=« Replacing W丨丨 in eqn.(4-2) with eqn.(4-1 )，we have L m n y, = 1 1 l u ^ r - u ^ r - u r t= I r=l i=l m n L - l n in = (n + L m - l ) u r + 1 1 u , W + I I I (4-3) r=l.^ p j=l.*i |=丨 |=|,灼 r=l The first term in the right-hand side of eqn.(4-3) can he regarded as a "signal" for the associative recall of u, 0, i = 1, 2, n; and when x is an "error" version of one of the FMs, u 0, i = 1,2，…，n. Based on such a concept, we will derive some theoretical results to compare the performances among the Hopfield model, the first-order outer-product model with self-feedback connections, and our proposed model in terms of the stability and attractivity. 4.3.1 Stability Analysis First, we will analyze the stabilities of these three associative memories. Here, FM u(p) is taken as the network's input (The detailed proof is given in Appendix III). 4.3.1.1 The Hopfield Model We have the following respective results: n — m \C(u ir\ u,p>)|trett < (4-5) m — 1 ， 73 n(m - 2) + m ‘ « ) > 2(m - 1) (4-6) n -m < n-2dmin(u ( r ,,u (p>) I . where |C(u(r,,u(p>)U = l l ^ ' u r u = n-2cWn), r^ p j= I r^ p r^ p d(u⑷’ u(p>) = min[ H(u(p>, u(r>), H(u,p), -u)】， (4-7) (4-8) (4-9) and H stands for the Hamming distance. Hence to the Hopfield model, eqn.(4-5) or eqn.(4-6) is a necessary and sufficient condition for the FMs to be FPs. Eqn.(4-7) gives the maximum number of the FMs that can become FPs, given the minimum distance between any two FMs, dinin(u), =1,2,…， r ru 关 p 4.3.1.2 The First-Order Outer-Product Model with Self-Feedback Connections The connection weights of this model are computed as follows: m w „ = W ， U = 1,2, •，n. They are the same as those of the Hopfield model except the self-feedback connections being not set to zero. V/e have the following respective results: n |C(u⑷’ u ^ U x < (4-10) nep m — 1 , n(m - 2) ( W u⑴，⑶ > (4-11) 阿 2(m — U , 74 +1 n-2dmin(u) (4-12) Here, the definitions of |C(u,r), u(p>)|irax and d(u⑴，u

) are the same as in the Hopfield model. Hence, for this model, eqn.(4-10) or eqn.(4-l 1) is a necessary and sufficient condition for the FMs to be FPs. Eqn.(4-12) gives the maximum number of the FMs that can become FPs, given the minimum distance between any two FMs, u), r = I, 2,…，m , ^ p . 4.3.1.3 The Novel Encoding Strategy Based Neural Associative Memory The following theorem gives the theoretical relationship between the length (L) of the neighboring range of each bit and the requirement for the FMs to he FPs as well as the maximum number of the FMs that can be FPs. Theorem 4-1: (i) Assume u)Ux S (4-13) m — 1 ， n (m- 2) - m ( L - I) d,„n(u) > (4-14) r^ 2 ( m - 1) , where I C C u ^ ^ ^ U = 1 1 u；0• U j , p ) U = n - 2d瞧—"，u

), d(u⑴，！！明）=min[ H(u(p), u(r>), H(u)】，and H stands for the Hamming distance, (ii) Given the minimum distance between any two FMs, drnm(u, u,p>), r = 1,2，...，m，本 p，the maximum number of the FMs that can become FPs is 2n-2dmin(u ( r ) ,u (p>) m < (4-15) n _ ( L _ 1) - 2cU(uw，u themselves which means that they can be allowed to be quite mutually correlated or to be quite near in the extensive distance defined by d(u)’ r = 1,2，. •，m,本 p，and vice versa. Finally, 76 those equations also imply that the first-order outer-product model with the self-feedback connections performs better than the Hopfield model with respect to the stability. 4.3.2 Attractivity Analysis We will analyze the attractivities of these three associative memories in the following subsections. Here, the network's input x is an “error” version of FM u,p>. The Hamming distance between x and u(p) is di bits (The detailed proof is given in Appendix III). 4.3.2.1 The Hopfield Model We have the following respective results: n - m - (m - 1 )-Cinax d, < (4-16) 2m ， n + Ctnax m < (4-17) 1 + C„m + 2d I . where Cinax is the maximum correlation among ail the FMs. Hence to the Hopfield model, eqn.(4-16) is a necessary and sufficient condition of the radius of attraction for each FM. Eqn.(4-17) gives the maximum number of the FMs that have the radius of attraction (d|), given the maximum correlation among all the FMs. 4.3.2.2 The First-Order Outer-Product Model with Self-Feedback Connections We have the following respective results: n - ( m - D - C ^ d, < 丨 (4-18) 2m ，、 n + C,nax I m < 、 (4-19) 2d I + C , m * , 77 where Cinax is the maximum correlation among all the FMs. Hence to this model, eqn.(4-!8) is a necessary and sufficient condition of the radius of attraction of each FM. Eqn.(4-19) gives the maximum number of the FMs that have the radius of attraction (d丨)，given the maximum correlation among all the FMs. 4.3.2.3 The Novel Encoding Strategy Based Neural Associative Memory The following theorem gives the theoretical relationship between the length (L) of the neighboring range of each bit and the radius of attraction of each FM as well as the resulting maximum number of the FMs. Theorem 4-2: (i) Assume u(r', r = 1 ,2，•"，m , are m randomly generated FMs. Suppose the network's input x is di hits away from the corresponding FM utp>. FM u(p) can be correctly recalled from x if and only if n - m(L - l ) - ( m - 1 )C,IUX d, < (4-20) 2m ， where C„ m is the maximum correlation among all the FMs; (ii) Given the maximum correlation among all the FMs, the maximum number of the FMs that have the required radius of attraction is m < (4-21) 2d丨+ L - 1 + C 舰 . Comparing eqns.(4-16), (4-17), (4-18)，(4-19), (4-20) and (4-21), we can see that in the proposed model, the integer parameter L will decrease the radius of attraction of each FM, or will decrease the maximum number of the stably stored FMs that have the required radius of attraction. Eqn.(4-18) tells that the longer/shorter the L is, the smaller/greater the di will be allowed. In other 華 78 words, given a set of the FMs, the longer/shorter the L is, the smaller/greater the radius of attraction of each FM can be achieved. Eqn.(4-19) tells that given a radius of attraction of the FMs, d|, the longer/shorter the L is, the smaller/greater the maximum number of the stably stored FMs can be achieved which have the required radius of attraction. As a result, large length of the neighboring range (L) will decrease the radius of attraction of the FMs, or decrease the maximum number of the stably stored FMs that have the required radius of attraction of the FMs. Moreover, the equations also imply that the first-order outer-product model with the self-feedback connections performs better than the Hopfield model with respect to the attractivity. Finally, from the discussions of the above two subsections, we reach a conclusion that in the novel encoding strategy based neural associative memory, extreme stability of the FMs can be achieved at the cost of their attractivity. With the neighboring range being introduced into the learning/encoding of the FMs, the longer/shorter the length of the neighboring range is applied, the more/less stable the FMs are, and the less/more attractive the FMs are. This conforms to the known fact that the more/less stable the FMs are, the less/more attractive they are. Finally, let us check the stabilities of these three associative memories when all the FMs are mutually orthogonal: to the Hopfield model, from eqns.(3-l) and (3-2), we have in n y,• u,(p) = - m + n + I I u / r > . u,)u,,r>. u/p> = n - m > 0 , when n > m, which implies that all the mutually orthogonal FMs will be stable in the Hopfield model so long as n is not smaller than m; 79 to the first-order outer-product model with the self-feedback connections, from eqn.(3-2), we have m n y,.u,(p>= n + I l u ^ - u ^ u r - u r M•印 i=l m = n + l Q u ^ ^ ^ V - u r = n > 0， r=l.印 which implies that all the mutually orthogonal FMs are stable in the first-order outer-product model with the self-feedback connections; to the proposed model, from eqns.(3-2) and (4-! )，we have tn n y,. u,

= n + m ( L - 1)+ I I u/“. u , ’ . u,,r' • u:p> r=l j=l m n L a. V V V n .1 0. Hence, the proposed model can always make all the mutually orthogonal FMs be stable. 4.4 Computer Simulations The synchronous mode is adopted for the proposed network's evolution. All the FMs are randomly generated from symmetric Bernoulli trials, i.e., their elements take a value of +1 or 一 1 with an equal probability of 0.5. Without the loss of generality, the right-hand-side neighboring range of each bit is utilized here. The computer simulations procede by gradually increasing m (the number of the FMs) with n (the dimension of the FMs) being fixed. Each point on all the curves given is obtained by ten times average under the same n, m, and L but different sets of FMs. Here, 80 we will only emphasize on checking whether the FMs can become FPs in the proposed model. In other words, we will only examine the stability of the FMs. \ Figs.4-1 to 4-4^how results of how many FMs can be FPs. The results of using different L under certain n are given to demonstrate the effect of the neighboring ranges on the stability of the FMs in the proposed model. From the figures, we can see that the larger the L is, the more stable the FMs will be. When L is sufficiently large, the proposed model is capable of making all the FMs be FPs, i.e.，achieving the extreme stability of the FMs. For the cases of n = 10, 20，30 and 40, in order to achieve the extreme or maximum stabilities of the FMs, sufficiently large values of L needed are 4, 10，14 and 19，respectively. (It should be pointed out that these sufficiently large values of L may not be the smallest ones which are necessary for the proposed model to achieve the maximum or extreme stability in these cases). However, given an n, when L reaches a certain sufficient high value, the extreme or maximum stability of the FMs can be achieved and in this case any longer L is no longer necessarily needed. This effect can be observed from Fig.4-5 which gives results of using L = 1，2，3，6，9, 14 and 23，for the case of n = 30. It is shown that when L is not smaller than 9’ the extreme or maximum stability of the FMs can be reached. Results corresponding to the cases of other values of n demonstrate the same matter, though they are not listed here. In general, the neighboring range required for the extreme or maximum stability of the FMs is not very large comparing with the dimension of the FMs. 81 Figure 4-1. The effects of different neighbouring ranges for the case of n=10. o The neighbouring-range length L=1 The neighbouring-range length L=2 - T h e neighbouring-range length L=4 '〇 10 20 30 40 50 60 The number of FMs 70 80 90 100 _ _ _ _ l _ T J o )o)ooooooooo o 0987654321 SCI」j ① q i u n u e q h -82 、入 j Figure 4-2. The effects of different neighbouring ranges for the case of n=20. 40 50 60 The number of FMs 80 90 100 ^ i _ — — — _ _ _ 1 r L L r_u o )o)oiopoooooo o 10987654321 sCLL i .oJa )qu jnu ① q h -83 Figure 4-3. The effects of different neighbouring ranges for the case of n=30. o The neighbouring-range length L= The neighbouring-range length L=2 - T h e neighbouring-range length L=14 6b cP 00(5® qb o o cP cP o D O O OCP °CP oQD 0 0 ° 〇 ① 0 CCDCP 00 10 20 30 40 50 60 The number of FMs 70 80 90 100 ^ i _ 一 _ _ _ ^ J o 109 8-7 6 5 4 3 ^ u s d L i . o J e q E n u ① qH 84 Figure 4-4. The effects of different neighbouring ranges for the case of n=40. o The neighbouring-range length L=1 The neighbouring-range length L=2 - T h e neighbouring-range length L=19 20 40 60 80 100 120 The number of FMs 140 160 180 200 2 0 0 -1 8 0 -1 6 0 -140~ (0 u： 1 2 0 -| 100-艺 8 0 -6 0 -40 -2 0 -0 85 Figure 4-5. The overview of the effects of using different L for the case of n=30. In the upward direction, L = 1, 2’ 3, 6’ 9, 14 and 23, respectively. 10 20 30 40 50 60 The number of FMs 70 80 90 100 00 90 80 0 60 50 40 30 20 0 0 0 s d L L一 0 j e c j t u n u ( D L l l 86 4.5 Conclusion We have proposed a novel encoding strategy based neural associative memory. The proposed encoding method computes the connection weights by summing up not only the products of the corresponding two bits of all the FMs, but also the products of their neighboring bits within a certain neighboring range as well. Both the theoretical and experimental results show that the novel encoding strategy is an ideal approach for a neural associative memory to achieve the extreme or maximum stability of the FMs. It is worth mentioning that s«ch extreme or maximum stability of the FMs is achieved at the cost of their attractivity. The radius of attraction of each FM in the proposed model becomes smaller as the length of the neighboring range used grows larger. 87 Chapter 5 Correlation-Type Associative Memory Using the Gaussian Function 5.1 Introduction Since the seminal paper of Hopfield [18], much research has been done on associative memory neural networks [8], [9】，[26], [28], [48], [51], [54], [59), [60]. Chiueh and Goodman proposed a general form of auto-associative memory called the RCAM (Recurrent Correlation Associative Memory) [8】，[9] which is based on operations of correlations. When the exponential function is used, the RCAM reduces to the EC A M (Exponential Correlation Associative Memory) [8], [9]. The storage capacity of the ECAM is theoretically proved to be growing exponentially with the dimension of the fundamental memories (FMs). However, from the viewpoint of real circuits implementation, the storage capacity of the ECAM is practically limited by dynamic ranges of the real circuits [9]. The circuits cannot give an infinitely large output response to an input. They have a limited ranges of output responses. In order to overcome such a shortage, we modify the ECAM by replacing the exponential function with the left-hand side of the Gaussian function (LHSGF). In this chapter, a new 、； model - the Gaussian correlation associative memory (GCAM) is proposed. As weighting functions, the LHSGF is to be used with which the monotonically-nondecreasing property required is still held [9]. Using the LHSGF has the same effectiveness in maximally discriminating an auto-correlation between an input pattern and its corresponding FM from mutual correlations between the input pattern and all the other FMs as the exponential function used in the ECAM does, but has no limitation of the dynamic ranges in the real 88 circuits implementation from which the ECAM suffers. The GCAM has exponentially-growing storage capacity like the ECAM. Besides, basins of attractions of the FMs in the V, GCAM can be controlled through adjusting two parameters of the LHSGF and can be larger than those of the ECAM. The organization of this chapter is as follows. In section 5.2, the GCAM is presented. Section 5.3 delineates analyses of several auto-associative memories including the first-order outer-product associative memories, the ECAM and the GCAM. Section 5.4 gives computer simulation results, followed by concluding remarks. 5.2 The Gaussian Correlation Associative Memory (GCAM) First, let us briefly review the RCAM (Recurrent Correlation Associative Memory) and its variant ECAM (Exponential Correlation Associative Memory) [9]. 5.2.1 The RCAM The RCAM is composed ( f three layers of input, hidden and output neurons. Inter-connection weights exist between two neighboring layers of neurons. There are feedback links from output-layer neurons to the corresponding input-layer neurons. Its block diagram is given below. 89 The block diagram of the RCAM where from top to bottom are input, hidden, output layers, respectively. W = [Wjj], W" = Ui°\ i = I，2 ,…，n , j = I, 2, •••’ m, is the connection matrix between the input and the hidden layers; while W，= [Wj / ] , W / = Uj0), i = 1，2,..., n’ j = 1，2 ,…，m , is the connection matrix between the hidden and the output layers. \ i t i = 1，2,…，ii, is the input of the network, and o„ i = 1,2，...，n, is the output of the network, f denotes a monotonically non-decreasing function. Denote u(r), r = 1，2，...，m, to be m fundamental memories (FMs) stored in the network; x to be an input pattern of the network; u丨…to be the ith element of FM u…，i = 1，2，…’ n, j = 1， 2, m; W"(W丨丨=u,⑴)，i = 1,2,…，n, j = 1,2，...，m, to be the connection weights between the input-layer and the hidden-layer neurons; W丨丨’（W"’ = u,1"), j = 1,2,…，m, i = 1,2,…，n, to be the connection weights between the hidden-layer and the output-layer neurons; and o,, i = 1, 2, ••，n，to be an output of the network. From these notations, we can see that the FMs are locally represented by the two layers of the connection weights of the network. When it is input to the input-layer neurons, x passes through the subnet [W丨J which is equivalent to doing the correlation operations between x and u(r), r = I，2，…，m . The hidden-layer neurons receive the correlation values and apply functions on them. Then the functioned values at the hidden-layer neurons pass through the subnet [W,,1] which is equivalent to doing weighted summations of all the FMs where these functioned values are used as corresponding weights. The output-layer neurons receive the summation values and give out their signs as an output of the network. Summarizing the above described process involved in the RCAM, we o therefore have an evolution equation of the network as follows: m n o, = sgn{ I f d x . W,,)- W / ) j=l i=l m = sgn{ I f [ C ( x , u { I ) ) ] u ! n ) , i = l,2，〜，n, (5-1) r=l where f (•) is a monotonically-nondecreasing function used by the hidden-layer neurons to apply on the correlation values received and, C( .，.）and sgn( • ) stand for the correlation operation and the sign function, respectively. Finally, the network's output o (o„ i = 1,2，...’ 91 n) can be fed directly back to the corresponding input-layer neurons to cycly repeat the above evolving process for a final convergence. 5.2.2 The ECAM In the evolution process of the RCAM as in eqn.(5-1), it is implied that the function f ( • ) plays a critical role in achieving high-quality associative recalls by maximally discriminating the auto-correlation between the input pattern and its corresponding FM from the mutual correlations between the input pattern and all the other FMs. When the function used in eqn.(5-l) is the exponential function, the RCAM reduces to the ECAM (Exponential Correlation Associative Memory) [9】.The storage capacity of the ECAM grows exponentially with n (the dimension of the FMs), which is resulted from the exponential function's high capability of maximally discriminating the auto-correlation from all the mutual correlations. It is worth mentioning that from the viewpoint of the network structure, the exponentially-growing storage capacity of the ECAM is achieved p' , r t 'y at the cost of the structural complexity of the network. The number of the connection weights of the ECAM is 2mn which grows linearly with m (the number of the FMs). While the number of the connection weights of the Hopfield network is independent of m and equals to n2, constantly. However, the storage capacity of the Hopfield model is only less than 0.15n [18], [26]. Therefore, it is reasonable and acceptable for the ECAM to have the linearly-growing structural complexity with the exponentially-growing storage capacity being possessed. 5.2.3 The GCAM Although the storage capacity of the ECAM is theoretically proved to be growing exponentially with the dimension of the FMs, from the viewpoint of real circuits implementation, the storage capacity of the ECAM is practically limited by dynamic ranges of the real circuits [9]. The circuits cannot give an infinitely large output response to an input. They have a limited ranges of output responses. As a result, the storage capacity of the real ECAM circuits wil l be far below the theoretical result. In order to overcome such a shortage, we modify the ECAM by replacing the exponential function with the left-hand side of the Gaussian function (LHSGF). Using the LHSGF as the weighting functions in eqn.(5-l), we propose a Gaussian correlation associative memory (GCAM). The monotonically-nondecreasing property required for the function f ( • ) [9】is still held for the convergence of the network. The GCAM has the same network structure and evolution process as the RCAM. The LHSGF used for the GCAM is as follows: f(t) = exp{ } , t < n -卩， (27C),/2a 2cr % 、 where n - | i is a right-most peaking point of the LHSGF (or is a center of the Gaussian function), and a controls the slope of the LHSGF (or controls the width of the Gaussian function). Larger\smaller values of a make the LHSGF change more slowly\faster with the variable t (or make the central part of the Gaussian function wider\narrower). From results given in the next section, we can see that using the LHSGF has the same effectiveness in maximally discriminating the auto-correlation between the input pattern and its corresponding FM from all the mutual correlations between the input pattern and all the other FMs as the exponential function used in the ECAM does, but has no limitation of the dynamic ranges in the real circuits implementation from which the ECAM suffers. Besides, on t one hand, the GCAM has exponentially-growing storage capacity like the ECAM. On the 93 other hand, the basins of attractions of the FMs in the GCAM can be controlled through adjusting two parameters of the LHSGF, and can be larger than those of the EC AM. 5.3 Analyses of Several Correlation-Type Auto-Associative Memories In this section, we will use neural dynamic approach [60] to analyze several correlation-type auto-associative memories. First, we briefly review the neural dynamic approach. Assume one of the FMs, u(p丨，is taken as the input of the network. The output of the network is given as follows: m o/ p ,= sgn{ I f l Q u r u(n) ]• u,,r'} m = s g n { f [ C(u,p', u,p)) ] u,,p) + I f [ C(u'p', u,r>)]- u,,n } in = s g n { f(n)Ul ,p ) + I f [ C(u,p), u(n) ] u:" }， (5-2) r= I where C(u,p), u,p)) = n . If the output of the network, o,p>, is equal to the input of the network, u(p), i.e., o/p>.u/p> > 0 ， i = l，2，〜，n， (5-3) 礞 then FM u(p) is called a fixed point (FP). By eqns.(5-2) and (5-3), in order for eqn.(5-3) to be satisfied, we must have m { f(n)u,(p) + I f [C (u ( p ) , u ^ J u ^ J u r > 0， i=l，2 ,…’ n. r=l.印、 - * Hence, f ( n ) + I f [ C ( u < p ) , u(r>) ] u ( , r , u l , p ) >0， i = 1,2, - , n . r=l.*p 、 94 By the above inequality, we can reach a sufficient condition for FM u(p) to be an FP, or equivalently, a sufficient condition for the stability of the FMs, as follows: m f(n) > I |f[C(u

,u will be taken as the input of the network. 5.3.1 Linear Function Model This is the case of the first-order outer-product neural associative memory. From the analysis given below, we can see that it is mathematically the same as the Hopfield model except with self-feedback connections. By substituting the linear function f(t) = t into eqn.(5-2), we have an output of the network as follows: in o,(p) = sgn{ nu, ,p) + I C(u ( m - l)|C(u } r=l in = sgn{ (n - m)u1(p ' + l C ( u < n , u,p))- u,(r>}， i = 1,2,…，n. (5-7) r=l.*p Also by eqn.(5-4), we require n - m > ( m - 1)1 C(u l r ), u(p)) | I!iax , Therefore, a sufficient condition for an FM to be an FP in the Hopfield model is as follows: n+ |C(ii⑴， m < r = 1 ， 2 ， m , ^ p . ( 5 - 8 ) i + | c ( u ( r > ， u , L ， From eqns.(5-5) and (5-7), we can see that only their first terms in the large brackets are slightly different. This means that the RCAM in the case of f(t) = t is mathematically the same as the Hopfield model except with the self-feedback connections. From eqns.(5-6) and (5-8), we can see that the storage capacity of the RCAM with f(t) = t being applied is only a little higher than that of the Hopfield model, although the structure of the former model grows linearly with m while that of the latter one is independent of m and just equals to the constant n2. This is because the linear function f(t) = t used for the RCAM is not effective enough to efficiently discriminate the auto-correlation from the mutual-correlations ( f(t) = t used for the R C A M is not capable of maximally discriminating the auto-correlation from all the mutual correlations at all). So, using the linear function for the R(JAM cannot exploit the advantage of the RCAM in the storage capacity with certain class of nonlinear functions being used. 5.3.2 Exponential Function Model This is the case of the ECAM [9】.Applying the exponential function f(t) = a1, a > 1，into eqn.(5-2), we have an output of the network as follows: in (r) (p) o,(p) = sgn{ an u,(p) + I a C ( u , u >. u丨…1 , i=l，2,“.，n. (5-9) r= I ,*p By eqn.(5-4), we accordingly require (r) (p) n ^ / 1 � C ( u , u ) max , ^ a > (m - 1)- a , r = l，2 , "，m ,关p . '丨 I Hence, a sufficient condition for an FM to be an FP in the ECAM is obtained as follows: I ( r ) 卟 I ^ n - C ( u , u ) max , , � � , r v � m < a + 1 ， r = 1,2, • m , 关 p . (5-10) From the first term of the right-hand side of the above inequality, we can see that the storage capacity of the ECAM grows exponentially with n (9]. Such an exponentially-growing storage capacity is resulted from the effectiveness of using the exponential function for the RCAM to maximally discriminate the auto-correlation from all the mutual correlations. In the next subsection, we wi l l examine how about using the LHSGF for the RCAM. 5.3.3 Left-hand-side Gaussian Function Model With the left-hand side of the Gaussian function (LHSGF) / ( t - b ) 2 、 tt f⑴二 expl- / , t) - n ]2 / (2cr) } u,' r ), i=l，2，...，n. (5-11) r=l.)tp (2n)|/J By eqn.(5-4), we accordingly require 1 m - 1 > exp{ - [ |C(u , r ), u(p') | m a x - n ] 2 / ( 2 c r ) }. (27t),/2 (2TC),/2 Thus, a sufficient condition for an FM to be an FP in the GCAM is obtained as follows: m < exp{ [ n - |C(u(r), u ,p ') | 瞧 ] 2 / ( 2 < r ) } + I， r = 1 , 2 , 关 p . (5-12) From the first term of the right-hand side of the above inequality, we can also see that the storage capacity of the GCAM grows exponentially with n, like the ECAM. The reason for it is that using the LHSGF has the same effectiveness in maximally discriminating the auto-correlation from all the mutual correlations as the exponential function, and thus makes a great use of the linearly-growing structure of the RCAM. 98 Now, we will consider the case when the input of the network is not exactly one of the FMs but an “error” version of one of the FMs. If the input of the network is one of the FMs, u, r = 1，2，…，m，is still the auto-correlation between this input pattern and its corresponding FM u(p). But in this case, such an auto-correlation is smaller than n. Considering both of the two cases, in order to maximally discriminate the auto-correlation between the input pattern I and its corresponding FM from all the mutual correlations between this input pattern and all ^ ^ v ^ the other FMs no matter whether the input pattern is exactly one of the FMs or is only an 丨 ““error’，version of one of the FMs, we modify the LHSGF as follows: " 1 [ t - ( n - n ) ] J \ f(t) = exp{ }， t < n - n ， ^5-13) {2n)m a 2ar where ^ is a shift of the right-most peaking point of the LHSGF (or a shift of the center of the Gaussian function). Parameter \ i should be appropriately tuned so as to make the right-most peaking point of the LHSGF approximately catch the auto-correlation (the maximum correlation). In addition, parameter a should be selected to be somewhat small in order for almost all the mutual correlations to fall outside the range of the right-most peaking point. Such a range covers an area of the largest-changing slope of the LHSGF. With the parameters | i and a being so selected, the LHSGF will be capable of maximally discriminating the auto-correlation from all the mutual correlations as required. 99 It should be pointed out that in the computer simulations here, parameters and a are pre-selected empirically by random trials and errors. Theoretically, they can be optimally found out by certain nonlinear optimization techniques such as the adaptive mean-squared-error approaches proposed in section 3.4. The computational complexity can also be greatly reduced with the computational precision being kept (refer to section 3.4). Results obtained by those adaptive mean-squared-error approaches can be trade-offs between the stability and the attractivity of the FMs. In other words, basins of attraction of the FMs can be optimally maximized with the requirement on the stability of the FMs being satisfied. However, like most of the nonlinear optimization techniques, those adaptive methods proposed in section 3.4 also suffer from the local optimum problem. The local optimum problem can be tackled by some other techniques, such as those in [47】.Nevertheless, the issues concerning adaptively finding out i^s and as by certain nonlinear optimization technique will not be addressed in this paper. f 5.4 Simulation Results In this section, we will give computer simulation results to compare the GCAM with the ECAM. In the computer simulations, the synchronous mode is used, i.e., only one evolutionary cycle as of eqn.(5-l) is to be carried out. In other words, no feed-back connections from the output neurons to the corresponding input neurons are needed. The simulations are conducted by varying m (the number of the FMs) with n (the dimension of the FMs) fixed. Each point on ail the curves given corresponds to 100 different sets of the FMs which are randomly 100 generated from symmetric Bernoulli trials (i.e., elements of ajl the FMs take +1 or - 1 with equal probability). Figs.5-1, 5-2 and 5-3 show the results corresponding to the cases of n = 20’ 30 and 40’ respectively. In these figures, the horizontal axis corresponds to the number of the FMs stored in the network while the vertical axis corresponds to the number of the correctly recalled FMs. The results of the GCAM are plotted in solid lines while those of the EC A M are plotted in dotted lines. Figs.5-l(a), 5-2(a) ahd 5-3(a) show the results concerning how many FMs can become FPs while all the other figures show how many FMs can be correctly recovered when the inputs are the “error” versions of the corresponding FMs. The results of the ECAM are obtained under the conditions of a = 2 and without the limitation of the dynamic ranges. In Figs.5-l(a)(b)(d)(0, 5-2(a)(b)(d)(0 and 5-3(a)(b)(d)(f)， the shifts of the right-most peaking point of the LHSGF, are all set to zero. In the remaining figures, ^s are not zero-valued and, fis and as are empirically pre-selected with random trials and errors depending upon the given “error” percentages of the inputs. The objective of appropriately selecting \ i and a is to make the right-most peaking point of the LHSGF approach the auto-correlation and to make all the mutual correlations fall outside the range of the largest slope of the LHSGF, respectively. As such, the LHSGF can be fully exploited in maximally discriminating the auto-correlation from all the mutual correlations. The values of [ i and a used for the simulations are shown in Tables 5-1, 5-2 and 5-3. It is worth mentioning that although the “error” percentage of the input has been taken into the account of the selection of 卩 and a given in these tables for the computer simulations here, we only want to shed light on the effects of [ i and a on the associative recall performance of the 101 GCAM. In fact, the selection of | i and a can be independent of the “error，’ percentage of the input but can just be based upon such a requirement that the basins of attractions of the FMs are to be maximized with the constraint of all the FMs being FPs. In other words, the selection of | i and a is a trade-off between the extreme stability and the attractivity of the FMs. Since such a trade-off of the selection of and a can be empirically accomplished without great difficulties, and the selection of | i and a need not be very strict but can only be loose (loosely appropriate values of 卩 and a are effective enough for the GCAM to achieve ideal results), the 觚 proposed GCAM is thus practical. Finally, it should be noted here that empirically pre-selecting | i and a need not be separately optimized for each particular test set of the FMs (or is independent of an arbitrarily given test set of the FMs) since the FMs are randomly generated from the symmetric Bernoulli trials. The same values of pre-selected [ i and a are used at all the neurons in the hidden layer as well as for all the 100 sets of the FMs corresponding to a point on the graphs. Nevertheless, optimal values of ^ and a can be adaptively found out by some nonlinear optimization techniques based on some error criteria such as those in section 3.4. If so, jis and as obtained at the different neurons in the hidden layer can be different. Moreover, by the nonlinear techniques, \is and as for the different neurons can be adaptively found out, combiningly or separately, based upon different error functions applied (refer to section 3.4). The issues concerning adaptively searching jx and a wil l not be addressed here. Now, we will examine the simulation results. With respect to the stability of the FMs, Figs.5-l(a), 5-2(a) and 5-3(a) show that the GCAM has extreme capability of making the FMs 102 FPs like the EC AM. Both the EC A M and the GCAM have the extreme stability of the FMs, i.e., both of them are able to make (almost) all the FMs FPs. With respect to the attractivity of the FMs, it is shown in the remaining figures that the GCAM performs better than the EC AM, especially when | i and CT are appropriately selected. So, we can reach a conclusion that the GCAM has more powerful error-correcting ability than the ECAM with the extremely perfect stability being kept. 103 60 40 20 0 The shift of GF center is 0. 0 0 20 40 60 Fig.5-1(a)Error percent, of inputs is 0. The shift of GF center is 0. Figure 5-1. The simulation results of n=20. The horizontal axis corresponds to the number of the FMs and the vertical axis corresponds to the number of the correctly recalled FMs. The results of the GCAM are plotted in solid lines while those of the ECAM are plotted in dotted lines. The shift of GF center is not 0. 0 0 0 ‘ » 1 01 » » ' 0 20 40 60 0 20 40 60 Fig.5-3(b)Error percent, of inputs is 10%. Fig.5-3(c)Error percent, of inputs is 10%. 104 The shift of GF center is 0. The shift of GF center is not 0. 50 j 1 501 1 0 20 40 60 0 20 40 60 Fig.5~1(d)Error percent, of inputs is 20%. 6 5 4 3 2 The shift of GF center is 0. H Fig.5-1(e)Error percent, of inputs is 20%. The shift of GF center is not 0. 0 20 40 60 Fig.5-3(b)Error percent, of inputs is 10%. Fig.5-3(c)Error percent, of inputs is 10%. 105 The shift of GF center is 0. ；f71 I X 1 0 20 40 60 80 Fig.5-2(a)Error percent, of inputs is 0. The shift of GF center is 0. ；lZJ ；L/J 0 20 40 60 80 • 0 20 40 60 80 60 40 Figure 5-2. The simulation results of n=30. The horizontal axis corresponds to the numbe of the FMs and the vertical axis corresponds to the number of the correctly recalled FMs. The results of the GCAM are plotted in solid lines while those of the ECAM are plotted in dotted lines. The shift of GF center is not 0. 80 60 40 Fig.5-2(b)Error percent, of inputs is 10%. Fig.5-2(c)Error percent, of inputs is 10%. 106 乂， The shift of GF center is 0. The shift of GF center is not 0. Fig.5-2(d)Error percent, of inputs is 20%. The shift of GF center is 0. 801 j 801 0 20 40 60 80 0 20 40 60 80 60 40 Fig.5-2(e)Error percent, of inputs is 20%. The shift of GF center is not 0. 20 j 1 / 401 1 V I ：[/ 1 0 20 40 60 80 0 20 40 60 80 30 20 Fig.5-3(b)Error percent, of inputs is 10%. Fig.5-3(c)Error percent, of inputs is 10%. 107 , The shift o? GF center is 0. t , z 1 0 50 100 Fig.5~3(a)Error percent, of inputs is 0. The shift of GF center is 0. Figure 5-3. The simulation results of n=40. The horizontal axis corresponds to the number of the FMs and the vertical axis corresponds to the number of the correctly recalled FMs. The results of the GCAM are plotted in solid lines while those of the ECAM are plotted in dotted lines. The shift of GF center is not 0. j x . : l z I 0 50 100 0 50 100 Fig.5-3(b)Error percent, of inputs is %. Fig.5-3(c)Error percent, of inputs is 10%. 108 The shift of GF center is 0. The shift of GF center is not 0. ' 0 ' 0 0 50 100 0 50 100 Fig.5-3(d)Error percent, of inputs is 20%. The shift of GF center is 0. Fig.5-3(e)Error percent, of inputs is 20% The shift of GF center is not 0. 60 5°l 6 r 0 50 100 0 50 100 Fig.5-3(f)Error percent, of inputs is 30%. Fig.5-3(g)Error percent, of inputs is 30% (a) (b) (c) (d) (e) (0 (g) p 0.0 10% 20% im 遍 30% 0.0 0.0 3.9 0.0 7.5 0.0 11.5 a 3.0 3.0 0.2 3.0 0.4 3.0 0.4 nr Table 5-1. The values of the parameters used in the simulations corresponding to Figs.5-l(a) to (g) with n = 20. p is the "error” percentage of the input. (a) (b) (c) (d) (e) (0 (g) P 0.0 10% 10% 20% im 30% 307f 0.0 0.0 5.5 0.0 11.6 0.0 17.5 a 8.0 4.0 0.4 4.0 0.3 4.0 0.4 Table 5-2. The values of the parameters used in the simulations corresponding to Figs.5-2(a) to (g) with n = 30. p is the “error” percentage of the input. t A ‘ (a) (b) (c) (d) (e) (0 (g) P 0.0 10% 隱 im im 搬搬 0.0 0.0 7.5 0.0 15.5 0.0 23.5 a 12.0 5.0 0.4 5.0 0.4 5.0 0.4 Table 5-3. The values of the parameters used in the simulations corresponding to Figs.5-3(a) to (g) with n = 40. p is the “error” percentage of the input. 110 5.5 Conclusion We have proposed a correlation-type auto-associative memory - the GCAM (Gaussian Correlation Associative Memory) which is extended from the ECAM (Exponential Correlation Associative Memory) [9]. The LHSGF (Left-Hand Side of the Gaussian Function) is used as the weighting functions. The real GCAM circuits will not have the limitation of the dynamic ranges in the real circuits implementation from which the real ECAM circuits suffer. The GCAM not only has the extreme stability of the FMs like the ECAM, but also has more powerful error-correcting ability (or stronger attractivity of the FMs) than the ECAM so long as the two parameters of the LHSGF are (loosely) appropriately selected. Chapter 6 A Further Investigation into the Upper Bound of the Asymptotic Storage Capacity of the Hopfield Associative Memory 6.1 Introduction Hopfield empirically estimated the storage capacity of his model to be not greater than 0.15n [18]. Since then, many other investigation have been conducted to theoretically search for the storage capacity of the Hopfield network by various approaches. The typical results among them were those given in [26] in which the asymptotic storage capacity of the Hopfield network was theoretically derived through the information theory. It was shown that if m FMs are chosen at random (the elements of the FMs take +1 or -1 with an equal probability of 0.5), the maximum asymptotic storage capacity, in order that most of the m FMs are exactly recoverable, is n/(21ogn). Furthermore, under the additional restriction that every one of the m FMs be exactly recoverable, the maximum asymptotic storage capacity can be no more than n/(41ogn) as n approaches infinity. Most of the other results obtained in [1】，[6】，【13】，[20], [23], [44], [48】are also similar. The results of theoretical investigations in [20] showed that the Hopfield network has major limitations when applied to fixed pattern classification problems because of its sensitivity to the number of fundamental memories stored and the SNR (Signal to Noise Ratio) of the input data. It was shown in (13] that for associative memories composed of n linear threshold functions without self-feedback connections, even when the Hamming distances between the desired rqemories are within yn and ( l - y )n , there are sets of size (l-2y)_丨(for y < 1/2)，the elements of which cannot be simultaneously stable. A similar phenomenon holds for the sum of outer-products connection matrix. It was shown 112 in [6] that the Hopfield network can result in many spurious stable states-exponential in the number of the FMs - even in the case when the FMs are orthogonal. In this chapter, we wil l theoretically investigate the upper bound of the asymptotic storage capacity of the Hopfield network from two different points of view. One is based upon the neural dynamics approach while the other is based upon the signal to noise ratio gain (SNRG) concept (discussed in chapter 3). Similar result is obtained from both viewpoints which is further supported by simulation results. It is shown that the asymptotic storage capacity of the Hopfield network, in order that all FMs are exactly recoverable, does not grow directly proportional or proportional to the dimension (n) of the FMs as described in [18], [26] (Here, "directly proportional" means that the storage capacity is approximately equal to 0.12-0.15n [18] while “proportional，，means that the storage capacity is equal to those obtained in [26]), but is directly determined by the distribution of the elements of the FMs. The result by the neural dynamics approach even shows that the upper bound of the asymptotic storage capacity of the Hopfield network is directly determined by the minimum extensive-distance between any two different FMs. We wil l see that the Hopfield network has major limitations in stably storing all the FMs because of its sensitivity to the minimum extensive-distance between any two different FMs. A small minimum extensive-distance between any two different FMs wi l l greatly deteriorate the storage capacity of the Hopfield network. The organization of this chapter is as follows. Section 6.2 briefly describes the two analysis methods. Section 6.3 theoretically investigates the upper bound of the asymptotic storage capacity of the Hopfield network from two different points of view. Section 6.4 is a concluding remark. 113 6.2 Two Analysis Methods In this section, we will describe two methods for the purpose of analyzing the Hopfield network. 6.2.1 The Neural Dynamics Method (60] Assume vector x is the input of the network which corresponds to FM u,p). The output of the network is computed as follows: ( 6 - 1 ) o, = sgn (y丨）， where y, and o, are respectively the summed input and the output of neuron i, i = I，2，…，n. It is well known that FM u(pl can be correctly recalled from input x if and only if y, • u,(p) > 0 , i=l，2，“ ’ n ; (6-2) and when the input x is exactly FM u'p), FM u ))1/2 (m-”丨72 (6-3) 114 To the WOPLAM, the corresponding SNR for the associative recall of the ith element of u(p) in the WOPLAM is obtained as follows: E[ | (n_l>ot吶 u ,叫] (Var( I Zc r .u / r > . i i / r , . u , ' p’）严 e : ( n - l ) a (p) (n - l ) l / 2 -a(p> (6-4) ( ( n - l ) I ( a < p , ) 2 ) l / 2 ( I ( a ( p , ) 2 ) , / 2 Considering that an FM cannot become an FP in the Hopfield network but can become an FP in the WOPLAM, we hypothesize that there should exist a gain in the SNR in the WOPLAM over the Hopfield network. Hence, from eqns.(6-3) and (6-4)，the gain of the SNR, named the SNRG, by the WOPLAM, for the pth FM is as follows: ( n - l ) , / 2 a ( p ) I (a1

、2、l/2 1/2 ^(p) (n -1 ) 1/2 ,1/2 ( m - D ' ^ a 1 in ( K a ^ ) 2 ) ^ r=l.印三G卟’ (6-5) ( m - 1 ) 丨 0 where G ,p) is defined as the SNRG of FM u

, p = 1, 2, . . . ， m . The SNRGs play an important role in the perfect associative recall of the FMs. In the WOPLAM, when the SNRG of an FM is greater than or equal to its threshold value, this FM can be correctly recalled. Moreover, when the SNRGs of all the FMs are greater than or equal to their corresponding threshold values, these FNlS can be correctly recalled. This condition for the stability and attractivity of the FMs can be regarded as a limitation on the number of the FMs that can be stably stored in the Hopfield network. The 115 reason can be found in eqn.(6-5). An SNRG is dependent upon all the learning weights. All the It SNRGs mutually affect each other. We must avoid to get the SNRG of an FM to be so large that the other SNRGs will naturally be lower or much lower than their thresholds. In fact, the choice of all SNRG values is a compromise that as many FMs as possible or even all the FMs are to be correctly recalled. Consequently, the asymptotic storage capacity of the Hopfield network is upper bounded. In the following section, we will use the above two methods to investigate in details the upper bound of the asymptotic storage capacity of the Hopfield network. 6.3 Investigation into the Upper Bound of the Asymptotic Storage Capacity of the ；Hopfield Network 6.3.1 By the Neural Dynamics Method We have the following result (Refer to Appendix IV for its proof): n 一 1 m < + I 1 + n - 2dm m (u⑴，u , ntp n - 1 = + 丨，（6-6) 1 + n - 29 n where dmu^u⑴，u(p)) is the minimum extensive-distance between u), (6-7) r^ p j=l r^ p r^ p 116 d(u(r)，u(p)) = mintHCu^，u⑴)，H(u,p', -u,r,)J， where H stands for the Hamming distance. Here, d ( u " , u. From eqn.(6-7), we can see that 0 falls into the range of (0, 0.5). As n approaches infinity, the right-hand side of eqn.(6-6) approaches 1/(1-29) whose maximum value can be obtained as 0 reaches 0.5. It is implied in eqn.(6-6) that the value of m gets larger as 6 gets closer to 0.5, under the restriction that all of the m FMs should be FPs. Now, the question is what is the maximum value of 6 in the range of (0.0, 0.5) that can be reached which corresponds to the maximum value of m, under the restriction that all of the m FMs should be FPs. To find an answer to this question, a lot of simulation arc conducted. The results are given in section 6.4 (refer to that section for simulation details). From the results shown, we can see that the trend of the simulation conforms with the analysis that large values of m will be obtained when 9 becomes large, under the restriction that all of the m FMs should be FPs. Furthermore, it can obviously he observed that B will he smaller as m grows larger. When m grows relatively larger, the approximate value of 6 reached will not he greater than 0.46. But, we can also see that in order for all of the m FMs to be FPs, the approximate value of 0 cannot be less than 0.45. So, empirically, it is reasonable to have such an assumption that to a set of randomly generated FMs (the elements of the FMs take +1 or - 1 with an equal probability of 0.5), the number of which is not very small, 0.45 (on an average) can be regarded as a maximum value of 9. In other words, it means that the maximum value of the minimum extensive-distance between any two different FMs is empirically assumed to be not 117 greater than 0.45n, on an average. By such a consideration, we can reach a conclusion that the maximum value of m shall be 10 (on an average) in order that all the m FMs are to be exactly recoverable. Consequently, from the viewpoint of neural dynamics, we can see that the asymptotic storage capacity of the Hopfield model has a constant (on an average) as its upper bound and does not grow directly proportionally or proportionally to the dimension of the 6.3.2 By the SNRG Method In the weighted outer-product learning associative memory (WOPLAM) (refer to Chapter 3)， it is known that when the SNRG of an FM is greater than or equal to its threshold value, the FM can be correctly recalled; and when the SNRGs of a set of or even all of the FMs are greater than or equal to their thresholds, they can thus be correctly recalled. With respect to this, we try to find out a learning weight for an FM so that its SNRG is maximised under the constraint that the total sum of all learning weights is a constant. Or, the problem is to maximise G(p)，p = I，2, • •，m, subject to the constraint m I a(r) = a m , (6-8) where a is an average value of the learning weights. The introduction of the constraint reflects the fact that the choice of all SNRGs is a compromise among all of them which is implied by eqn.(6-5). Generally, it must be avoided that the SNRG of an FM is so large that the other SNRGs will not be greater than or equal to their thresholds. A procedure for finding out appropriate learning weights for the FMs under the above consideration is discussed below. Define a function O such that 118 )2 m = + T ( I a ( r , - a m ) , (6-9) where x is a real-valued parameter and eqn.(6-5) has been undertaken. The optimal solution of eqn.(6-9) (The optimisation procedure is given in Appendix IV) is obtained as follows: in I (a,r))2 m A in(l(a ir))2) ir' r=\.*{) = ( I (a ( rV ),/2 • ( A m ) , p = l,2，".,m, (6-10) r=l.*p 3 A m a ! p ] > A m - ( l ( a l t ) ) 2 )1/2 -r=l.*T> where A = + (6-11) 2 (m- l ) 3 'M(rT>-l) , / 2 . Consequently, we have (m- l ) , / 2 o T G(p, = m ( K a ( r V y a > (m-\) irl • (A in )， p = l，2, . " ,m . (6-12) The right-hand side of eqn.(6-12) can be viewed as a threshold of the SNRGs. It should be mentioned again that the result about the SNRGs in eqn.(6-12) is a consequence of the two 119 fundamental assumptions: one is that all the FMs are generated at random and the other is that the dimension of the FMs approaches infinity. Accordingly, from the viewpoint of the SNRG, we can see that when the dimension of the FMs is sufficiently large, the quality of the associative recall of the randomly generated FMs is not decided or directly decided by the dimension of the FMs but is dependent upon the number of the FMs. Intuitively, as n is sufficiently large, the maximum number of the FMs that can all be correctly recalled is definitely upper bounded. Depending upon the basic fact that a threshold value of the SNRG is greater than zero, the second term of the right-hand side of eqn.(6-12) is thus required to be greater than zero. As such, it is finally found that to have all the m FMs to be exactly recoverable, the maximum value of m should not be greater than the value of 10, on an average. (The meaning of “on an average，，lies in the fact that the upper-bound value (10) is optimally obtained under the specific constraint of eqn.(6-8). The upper-bound value may not exactly be 10 while imposing some other constraints). Therefore, we can see from the two different points of view, the neural dynamics and the SNRG concept, that the asymptotic storage capacity of the Hopfield network in order that all the FMs stored in the network are exactly recoverable, does not grow directly proportionally or proportionally to n but is bounded from above by a constant (on an average) as n approaches infinity. The constant (on an average) upper-bound is not decided or directly decided by n but is directly determined by the distribution of the elements of the FMs. The resulting upper bound is found out to be 10 on an average. In addition, by the first point of view, we further reach a conclusion that the upper bound of the asymptotic storage capacity of the Hopfield network is in effect determined by the minimum extensive-distance between any two different FMs. Since such a minimum extensive-distance actually cannot quite be able to approach 0.5n (the empirically 120 obtained optimal estimate is 0.45n, on an average), then the upper bound of the asymptotic storage capacity of the Hopfield network is thus very low (according to eqn.(6-6)). Besides, although the upper bound of the asymptotic storage capacity of the Hopfield network is not indicated in all related literature [20】，[48】，[49】，[50], [51 ]，[56], [60], (a lot of literatures are not listed), the trend of such an upper bound can clearly be observed in their simulation results. The existence of the upper bound of the asymptotic storage capacity and its dependence upon the minimum extensive-distance can be used to explain why the Hopfield network cannot perfectly store a very small number of FMs even with a large dimension (refer to the representative examples given in Chapter 3). The reason is that although the dimension of the FMs is very large and its number is very small, a small minimum extensive-distance is strong enough to severely deteriorate the associative recall performance of the Hopfield network [50], [54], (60] (there are a lot of literatures not listed here). Therefore, in addition to the dimension and the number of FMs, the distribution of the elements of FMs, or saying, different FMs, can be regarded as another performance of the Hopfield network. the minimum extensive-distance between any two very important factor that greatly affects the 6.4 Computer Simulation In this section, a lot of computer simulation results are given to show the relationships among the number of the FMs, the number of the FMs being FPs, 0 (in eqn.(6-7)), and the dimension of the FMs. In all the computer simulation, the synchronous mode is used. The simulation is conducted by varying m (the number of the FMs) with n (the dimension of the FMs) fixed. Each point on 121 all the curves given corresponds to 100 different sets of the FMs which are randomly generated from symmetric Bernoulli trials (i.e., elements of all the FMs take +1 or - 1 with equal probability). In all the figures, the horizontal axis, in both (a) and (b), corresponds to the number of the FMs. The vertical axis in figures (a) corresponds to the number of the FMs being FPs, while that in figures (b) corresponds to 0 (denoted by THETA). From the results shown, we can see that the trend of the simulations conforms with the above analysis that large value of m can be obtained when 8 is relatively large, under the restriction that all of the m FMs should be FPs. Furthermore, it can obviously be observed that 9 gets smaller as m grows larger. When m grows larger, the approximate value of 9 reached wi l l not be greater than 0.46. But, we can also see that in order for all of the m FMs to be FPs, the approximate value of B reached cannot be less than 0.45, on an average. So, empirically, it is reasonable to conclude that to a set of randomly generated FMs (the elements of the FMs take +1 or - 1 with an equal probability of 0.5), the number of which is not very small, 0.45 can be regarded as a maximum value of 9, on an average. In other words, the maximum value of the minimum extensive-distance between any two different FMs is empirically found to be not greater than 0.45n, on an average. We can also see from all of the simulation results that with the strict restriction of all of the m FMs being FPs, this number m does not conform with the well-known result of 0.12 ~ 0.15n. It is not directly decided or decided by n but is directly determined by 8 or the distribution of the elements of the FMs. Moreover, Figs. 6-1 to 6-4 show that the maximum value of m，with the strict restriction of all of the m FMs being FPs, is less than 10. Although such a maximum value of m in Figs. 6-5 to 6-9 is greater than 10, this phenomenon does not 122 contradict the theoretical analysis. Since the upper bound of 10 is based upon the condition of 6 beVpg 0.45 (by neural dynamics approach) or of imposing the constraint of eqn.(6-8) (by SNRG approach). I f the assumed condition is varied, the upper bound obtained will be slightly different. The neural dynamics approach tells us that the upper bound approaches l/( 1-20) as n approaches infinity (refer to section 6.3.1). This can be observed in Figs. 6-5 to 6-9. In these figures, the values of 6 are obviously greater than those in Figs. 6-1 to 6-4，and are above 0.45 which correspondingly increases the upper bound of the storage capacity. But, the upper bound of the storage capacity thus achieved is still much smaller than 0.12 - 0.15n. Consequently, we can see that the asymptotic storage capacity of the Hopfield model does not grow directly proportionally or proportionally to n but has a constant (on an average) as its upper bound. Such an upper hound is directly determined by the distribution of the elements of the FMs (the minimum extensive-distance between any two different FMs). 123 Figure 6-1. Simulation results of n=20. 10 20 30 40 50 60 (a) The number of the FMs 10 20 30 40 50 60 (b) The number of the FMs 4 3 2 1 o s d z l ①石一0 J e q L u n c e - c J _ 0 0.5 0.45 0.4 0.35 0.3 0.25 0.2 Q o u o a l s ^ e > j s u a ) p c e ① q h -124 Figure 6-2. Simulation results of n=50. 1 1 1 1 1 A I I J I I J 10 20 30 40 50 60 (a) The number of the FMs 10 20 30 40 50 60 (b) The number of the FMs 125 8 6 4 2 o s d L L0 ) 5 ) 0 J Q - Q E n u e q H 0 0.5 0.45 0 0.35 0.3 0 Q o U B l s j - o a ) > s u ① p < 9 ① Figure 6-3. Simulation results of n=100. 10 20 30 40 50 60 (a) The number of the FMs 10 20 30 40 50 60 (b) The number of the FMs 126 _ r 1 _ 1 1 1 o 2 o , 8 6 4 2 o 1 1 一 s d ^ ①Ml )0」① C I E n u e X I h -_ _ _ _ _ _ o 8 6 4 2 4 8 .4.4.4.40..3 o o o o o .① o u c o l w l p ① > l w u a ) p < a )①J Z H Figure 6-4. Simulation results of n=150. 10 20 30 40 50 60 (a) The number of the FMs 10 20 30 40 50 60 (b) The number of the FMs 1 _ 1 o o 5 o s d L L© £o」 ①J D E n u o q l 5 8 6 4 2 4 0.4.4.4.40. o o o o ① ① ① p { ①a ) q l -127 Figure 6-5. Simulation results of n=200. 10 20 30 40 50 60 (a) The number of the FMs 10 20 30 40 50 60 (b) The number of the FMs 20 5 0 5 0 0 s d L L a ) q j ) 0 J a ) < q a l n u 5 8 6 4 2 04 4 4 4 d do.0.e o u C T J J W I P e A l w u e p c Q )①J I 0.4 128 Figure 6-6. Simulation results of n=250. 1 1— 1 1 1 ^ x n 1 — — I I I I 10 20 30 40 50 60 (a) The number of the FMs (b) The number of the FMs 25 20 5 0 5 0 0 s d U L e J I J )0 J e J D E n u ① q h -8 7 6 5 4 3 2 .44.44.44 4 CJO..O..O.. e o u e l s - 6 a ) > s c a ) s < a ) o f 129 Figure 6-7. Simulation results of n=300. 20 30 40 (a) The number of the FMs 10 20 30 40 50 60 (b) The number of the FMs 30 25 20 5 0 5 0 0 s d L L q £1 0 s q i c Q ) X : h 8 -0/49 0 6 4 o . d do. a ) o u 扣t s l p ① A l w u e l x a ) a ) J Z 0.45 4 3 0 130 Figure 6-8. Simulation results of n=350. 10 20 30 40 50 60 (a) The number of the FMs 10 20 30 40 50 60 (b) The number of the FMs 30 25 20 5 0 5 0 0 sdLL①JIJ 1 0」 Q ) q E n u 9 J Z 1 0.5 0.49-0.48-4 o . o . o . d ① O U E J S - a a ) > s u e p < ①① £ 0.46 5 -4 0 131 Figure 6-9. Simulation results of n=400. 20 30 40 (a) The number of the FMs 10 20 30 40 50 60 (b) The number of the FMs 0 30 20 0 0 0 s d L J .①丢o J e d E n u0 1 1 1 0.5 {Q 46 45 44 0 o o o o o o e o c B l w p 9 > w u 9 p c e a ) q l 132 6.5 Conclusion From the viewpoints of the neural dynamics and SNRG, with the restriction that all the I FMs be exactly recoverable, it is simultaneously found that the asymptotic storage capacity of the Hopfield network does not grow directly proportionally or proportionally to n but has a constant (on an average) upper bound as n approaches infinity. Such an upper bound is not decided or directly decided by n, hut is directly determined by the distribution of the elements of the FMs. 133 Chapter 7 Concluding Remarks, Evaluation, and Outlook for Future Research In this dissertation, four main topics - the adaptive weighted outer-product learning associative memory, the novel encoding strategy based neural associative memory with maximum stability, the Gaussian correlation associative memory (GCAM), and the further investigation into the upper bound of the Hopfield associative memory have been discussed. With respect to the adaptive weighted outer-product learning associative memory, the concept of adaptive weighted outer-product learning is proposed for the encoding of the fundamental memories (FMs) in Hopfield-type networks. The weighted outer-product learning associative memories (WOPLAMs) proposed improve the performances of associative store and recall of the FMs. The sufficient conditions for the learning weights and the signal to noise ratio gains (SNRGs) are derived in a probabilistic context. It is demonstrated that each SNRG has its own threshold and that any FM can be correctly recalled when its corresponding SNRG is larger than or equal to its threshold. In principle, the asymptotic storage capacity of the WOPLAM will grow at the greatest rate when all the SNRGs or learning weights satisfy their sufficient conditions. It appears that the WOPLAM can achieve correct recall of all the FMs by the appropriate setting of the learning weights. In practice, however, given a set of FMs, one might only correctly recall as many FMs as it is possible to make the corresponding SNRGs or learning weights satisfy their sufficient conditions. Several gradient-descent-search algorithms capable of dynamically finding out the optimal learning weights are also proposed and implemented. The objective functions used for these algorithms are based upon the global- or local-error-measure. The algorithms can be further 134 accelerated by the adaptive adjustments of the neuronal parameters. Faster convergence is achieved without losing any computation quality and memory capacity. In general, the WOPLAMs, with the use of adaptive algorithms, have the storage capacity of up to 0.2n to 0.3n or above. It should be emphasized that to achieve these improvements, only the computation complexity of the learning phase is increased while that of the recall phase remains unchanged. It is worth mentioning that using the adaptive algorithms can achieve the minimization of the overall-error between network outputs and their corresponding FMs. When the number of FMs is small, minimizing such an overall-error can make all the FMs FPs. However, when the number of FMs is large, the number of FPs achieved using such kind of error measure can only be maximized. In this case, a maximum number of FPs can be obtained by sacrificing a small (minimum) number of FMs (by allocating very small learning weights to them) if needed. In other words, in the WOPLAM, given a set of FMs, if it is impossible to make all the FMs he FPs, then a maximum number of FMs will be made to become FPs at the expense of the stabilities of the other remaining FMs. In order to improve the performance, we may use some other alternatives for optimization computation. For example, by combining characteristics of the simulated annealing algorithm and neural network, Van Den Bout and Miller [47] developed an algorithm for graph partitioning called MFA (Mean Field Annealing) which exhibits rapid convergence resulted from the neural network while preserving the solution quality afforded by simulated annealing. With respect to the novel encoding strategy based neural associative memory with the extreme or maximum stability of the FMs, a novel encoding strategy for neural associative memories is proposed. Unlike the conventional outer-product learning rule used in the Hopfield-type memories, the proposed encoding method computes the connection weights by summing up not only the 135 products of the corresponding two bits of all the FMs, but also the products of their neighboring bits within a certain range. Theoretical analysis has been carried out to investigate the performances of the proposed model in terms of its stability and attractivity which are compared with those of the Hopfield-type models. Both the theoretical and experimental results show that the novel encoding strategy is an ideal approach for a neural associative memory to achieve the extreme or maximum stability of the FMs. It should be pointed out that such extreme or maximum stability of the FMs is achieved at the cost of their error-correcting ability. The radius of attraction of each FM in the proposed model becomes smaller as the length of the neighboring range used grows larger. How to deal with the contradiction between this extreme or maximum stability and degenerating radius of attraction is an interesting future research topic. Regarding the local storage distribution of the FMs, we have proposed the correlation-type autoassociative memory - the GCAM, which is extended from the ECAM (Exponential Correlation Associative Memory) [9]. The left-hand-side Gaussian function (LHSGF) is used as weighting functions. It is effective enough for the LHSGF to maximally discriminate the auto-correlation (between the input pattern and its corresponding FM) from all the mutual correlations (between the input pattern and all the other FMs), like the exponential function used in the ECAM. But the real GCAM circuits will not have the limitation of dynamic ranges in the real circuits implementation from which the real ECAM circuits suffer. Besides, the basins of attractions of the FMs in the GCAM can be controlled through adjusting two parameters of the LHSGF and thus can be larger than those of the ECAM. As a conclusion, the GCAM not only has the extreme stability of the FMs like the ECAM, but also has higher 136 error-correcting ability (or stronger attractivity of the FMs) than the ECAM as long as the two parameters of the LHSGF are (loosely) appropriately selected. How to adaptively find the two parameters of the LHSGF comprises our future work on this research. Theoretically, they can be optimally found out by certain nonlinear optimization techniques such as the adaptive mean-squared-error approaches proposed in section 3.4. The computational complexity can also be greatly reduced with the computational precision being kept (refer to section 3.4). Results obtained by those adaptive mean-squared-error approaches can be trade-offs between the stability and the attractivity of the FMs. In other words, basins of attraction of the FMs can be optimally maximized with the requirement on the stability of the FMs being satisfied. However, like most of the nonlinear optimization techniques, the adaptive methods in section 3.4 also suffer from the local optimum problem. The local optimum problem can be tackled by some other techniques, such as those in [47], Finally, we further investigate the upper bound of the asymptotic storage capacity of the Hopfield network from two different points of view - the neural dynamics approach and the SNRG concept. With the restriction that all the FMs be exactly recoverable, similar results are obtained simultaneously from both approaches showing that as n approaches infinity, the asymptotic storage capacity of the Hopfield network does not grow directly proportionally or proportionally to n but is bounded from above (on an average). Such an upper bound is not decided or directly decided by n, but is directly determined by the distribution of the elements of the FMs. The future effort to be directed on this research includes finding a closed form solution of the asymptotic storage capacity of the Hopfield network versus the distribution of the 137 elements of the FMs; on the basis of this solution, to construct a novel encodingMearning algorithm which is expected to be insensitive to the distribution of the elements of the FMs and thus to greatly increase the storage capacity; to make the encodingMearning algorithm additive, i.e., not thorough, but only additive encodingMearning is needed when a new FM i appears; and based on the encodingMearning algorithm obtained, to set up a novel network architecture for neural associative memories. Since the Hopfield-type networks have really been found to have severe limitation in their > storage capacities, we believe that in order to greatly improve the associative recall performance of the associative memories, some other novel nonlinear dynamics should be adopted and evaluated for their evolution. The most important thing we should do next is to propose some other frameworks or mechanisms for neural associative encodingMearning. 138 Appendix I The Proof of Theorem 3-Let u(s>, be a neighbor one Hamming distance away from u 0, one wishes to make it an FP by the WOPLAM, i.e., Dw - D w ' < 0, namely n n hi it n in t “ l ^ a UjFj u, u, 1 1 r= I I=|.*J r=| Drv • v V ••⑴〜⑴••⑴.•⑴••⑴ 1 V V V ..,sh .A.'" , . , r l . . w — Dw = - 2- L L u, a u, u, u, 2- L u, a u, u, n n in n � n in = - I I I (a , r ) - \ y u r \ { t ) u ! s ) u r + I I 5：( 0 ^ - " . 1 1 : % / % . % 1广+ D h - D h _ i = I I .右 r=丨 I •芍户 I •灼 n= • in n n = - K a ( r ) - D i i ur ur (ur ur — u r . u , ' ’ + Dm - Dh ' r=l 丨=丨.芍 j= I.^i m n = - K a ( r ) - 1 ) 1 2 u^ ur-2 u^-ur + D h - DH ' n=l j= 1,灼 m n = - 4 1 I (a , r ) - 1).<".11/"‘11广.11广 + D h - Dh ' < 0 , r=l thus. in n m n I I ^ u ^ - u r - u ^ u r > ( D „ - D H V 4 + I I up,r, u;r, u;s, u,,s) r= I j= I r = 1 j= ••沟 139 n - f - ( n - [ (a(r) - I) r=l in (Dh - D h ' ) / 4 = ( D H - D H ' ) / 4 > (D H - D H ' ) / 4 = (Dh - D „ ' ) / 4 therefore, we have o ^ 〉 ( D H - D H ' ) / [ 4 ( n - l ) l Cv 140 Appendix II The proof of Theorem 3-2 The proof is conducted by following the approach in [9]. For a given p, 0 < p < 1/2, suppose that the weighted outer-product learning associative memory (WOPLAM) is started with a bipolar vector x that is d| (= pn) bits away from the nearest fundamental memory (FM), say u(p), i.e.. where e has di nonzero (+2 or -2 ) components. Without loss of generality, assume that …明二 -1 . Furthermore, because the bit-error probability is larger for the case where x,关 u,lp) (x, = +1, e, = 2) than when x,= u,

(x, = —L e,= 0)，only the-former case will he studied here. Thus, we have n x,'= sgn{ I W „ W . x,) j=i n in = s g n { I W W } in in = sgn{ Ia (k ,--u:k>- I a , k , x,} k=l k=l in in = sgn{ -a(p, n(l - 2 p ) - I a ' k , x, + I a u；11} k-丨 K 邛 m = sgn{ -a denotes the correlation of u U , , ，m. Z = Ia ( k , - z k , (A2-2) m in v ^ I a ^ - z , - I a ( k ) x, k:l k=l f m m in = Ia , k , - z k - 1 a ,k) = -« 2"1 [9】.The expectation of Z| can be roughly bounded as follows: n-l n-1 E[z,] = I Prob[ z, = n - 2j] (n - 2j) - I Prob[ z, = - ( n - 2 j - 2)] (n - 2j - 2) J=di' J=di'-I n-1 n-1 S = (1/K) { l C r H , ( n - 2 j ) — l C n , ' ( n - 2 j - 2 ) } J=d." j= ) 2 ) , / 2 (m-l) l / 2-a , p> > I a ( k ) 2(2 t ) , / 2 q(p)n (m- ] ) l / 2 > n - 2pn + I where t is a fixed and large number. Thus (A2-9) 2(2t) l / 2q(p)n l a 1 k=l.3tp n 一 2pn + 1 o T > (A2-10) By inequalities (A2-8) and (A2-9), we have ( k ) , 2 Var[Z]<4q‘(p)rT k=l.*p 4q ' : (p )n- : [a , p , (n -2pn+ I)]' < [2(2t) , / 2q(p)n]2 [ a , p ) ( n - 2 p n + 1)]* 2t 144 [a(p, n (l -2p) + I a ( (kh <=\ < 2t Considering inequalities (A2-7) and (A2-9), we have E [ Z ] < 4 l a , k ) 4a ( p ) (n-2pn+ 1) < 2(2t) , / 2q(p)n (A2-I1) 4[a ,p) n (l -2p ) + Ia , L ) k=l < 2(2t) l /2q(p)n « a ( p ) n (l -2p ) + Ia , k > k=l (A2-12) as n —> oo, t is a fixed and large number. Therefore, the expectation of Z is significantly smaller than a 0 】 =Prob[ Z > a,p) n( 1 - 2 p ) + I a , k ' ] k=l m = P rob [Z -E [Z ] > ot(p、n(l-2p) + Xcx…】 k=l in a(p) n(l - 2 p ) + I a , k > k=l =Q{ } (Var[ Z ] 严 0 ] = Prob[ Z > oT n( I - 2p> + X a, we say that u,p) is an FP if and only if y, • u,^' > 0, i = 1,2，."’ n; and when x is an ‘‘error，，version of one of the FMs, u

, we say that u(p) can be correctly recalled if and only if y,. u,(pl > 0, i = 1, 2, • •，n. Based on such a concept，we will derive some theoretical results to compare the performances among the Hopfield model, the first-order outer-product model with self-feedback connections, and our proposed model in terms of the stability and attractivity. A3.1 Stability Analysis First, we will analyze the stabilities of these three kinds of associative memories. Here, FM u(p) is taken as the network's input. A3.1.1 The Hopfield Model By eqns.(4-1) and (4-2), we have n in n in n y , = i W(JH . Xj = I I = i l u r - u r - u r . j=l n：I j= 1 r=丨丨=1.沟 Therefore, m n r=l 147 = - I u,,r>. 11,⑴.u,(p)- u,,p) + I u r - u,^'. u;p) •u,"1' + I I u,,r)- U|<". u;p> • u,(p) 1=1 m n = - m + n + I K u / W i n=l.*p p i > n - m - (m - 1 ) |C(u⑷，u(p’)| = n - m ~ n ( m - l ) + 2 ( m - 1 )dIllin(u, u,p>) f^ p n - K . d m > 0 (A3-1) (A3-2) (A3-3) where |C(u(r u(p))U = IIu;r,-u,，-u,r>)], (A3-4) (A3-5) and H stands for the Hamming distance. From eqns.(A3-l), (A3-2) and (A3-3), we have the following respective results: n - m |C(u i^ p 2(m 一 1) n -m < n-2dm i n (u ( r , ,u ( p ) ) + I (A3-6) (A3-7) (A3-8) A3.1.2 The First-Order Outer-Product Model with Self-Feedback Connections The connection weights of this model are computed as follows: W I J = Iu, ( r ) - u; r ), i , j = 1’ 2，…，n. 148 They are the same as those of the Hopfield model except the self-feedback connections being not set to zero. From the above equation and eqn.(4-2), we have n m n m n y,= = i z u / w = z i u r - u r - u ^ . j = l r=l j = l r=l j = l Therefore, m n >',.u,

= z i u r - u ^ u ^ u r ( p ) u , PI u l u . m u , n z i = l I m n (r) (p) (r) (p) = n + I r=l.印 j=l > n - ( m - l ) |C(u< r > ,u ( p >)U (A3-9) = n - n ( m - l ) + 2 ( m - 1 ).d隱(u("’ u

) (A3-10) = 2 n - 2dm,n(u(r)，u(p)) - I n - 2dnnn(u,r), u,p>) ]m > 0 . (A3-11) Here, the definitions of |C(u…，u

)|inax and d(tT，u(p>) are the same as in the Hopfield model. From eqns.(A3-9), (A3-10) and (A3-11), we have the following respective results: n |C(u(r>，u

)|瞧 S (A3-12) m — 1 ’ n(m 一 2) d_(u…，！！糾）> (A3-13) 師 2 ( m - 1 )， m < + I . (A3-14) n-2dmin(u ( r t ,u ( p>) 149 A3.1.3 The Novel Encoding Strategy Based Neural Associative Memory Proof: From eqns.(4-2) and (4-5), we have n in n L in n L y,=： i w . - x , = 1 1 l u ^ u ^ r - x , = n l u ^ r - u ^ r - u ^ j=l r= 1 j= I i= I r=l j=l 1=1 Therefore, m n L v ,,

_ y V V ,, 'r> ,, …n , Y V V .. (D (r) n(p> ..

-2L Z u , • Uj • u, • u, + 2 - 2 - 2 - ul+1_i . u … • u, • u, ri l j=l n= I |= I t=2 n in n = I u,(p>- u/p)- u;p) •u1,p) + I I u;r)- u;p)- u,,r>- u；^ J=l r= I j= I m L m n L . • ‘ … v . ZY ..

n ip> . V v V ,, (r» .. "> ,, 'p> ,, n + m ( L - l ) - ( m - 1 )|C(u,r>, u , p , ) U (A3-15) = n + m(L - 1) - n(m - 1) + 2(m - I )山仙(11⑴，u

) (A3-16) = 2 n - 2 d m i n ( u ( r ) , u < p , ) - [ n - ( L - 1) - 2dnnn(u,r>, u

) ]m > 0 . (A3-17) From eqns.(A3-15), (A3-16) and (A3-17)，we have the following respective results: n + m(L - I) |C(u…，u

)|瞧 S (A3-18) m — 1 , n(m - 2) - m(L - 1) c W , , u , > ： (A3-19) n^p 2 ( m - !) , 150 m < (A3-20) n - ( L - l)-2d rn,n(u ( r t,u (p)) . # A3.2 Attractivity Analysts We will analyze the attractivities of these three kinds of associative memories in the following subsections. Here, the network's input x is an "error" version of FM u - m + n -2d, - 1 C(x，u(r)) r=l.*p > - m + n - 2d, - (m - " (C瞧+ 2d丨） = n - m - ( m - l ) C I i m - 2 m d , (A3-21) = n + C n m - [ 1 + Cnstt + 2d,】m 2 0， (A3-22) 151 where Cmax is the maximum correlation among all the FMs. In the above derivation, the relation of C(x, < C,nax + 2dI [60] has been applied. From eqns.(A3-2l) and (A3-22), we have the following respective results: n - m - (m - 1 ).C,mx d, < (A3-23) 2m , n m < (A3-24) 1 + C 隱 + 2d I . A3.2.2 The First-Order Outer-Product Model with Self-Feedback Connections From eqn.(4-2) and the connection weights rule given in section A3.1.2，we have n m n y,= iw.-x, = I W . x ’ . j = l r=l j = l Therefore, m n y , u r = i i u r - u r ^ - u r n in n = K x , + I K v W , in = n - 2d, + ! C ( x , u(r))- u,,r, u r m > n - 2 d , - l C ( x , u⑷） > n - 2 d , - ( m - l ) ( C n m + 2d,) = n - (m - 1 )Ctnax - 2 1 ^ (A3-25) = n + Crnax - (2d, + C瞧)m > 0 , (A3-26) 152 where 匸匪 is the maximum correlation among all the FMs. Also, in the above derivation, the relation of C(x, u(r>) < C u i a x + 2d| has been used. From eqns.(A3-25) and (A3-26), we have the following respective results: < n - ( m - 1 )C„ 2m (A3-27) n + C m < 2d I + C n m (A3-28) A3.2.3 The Novel Encoding Strategy Based Neural Associative Memory Proof: By eqns.(4-2) and (4-5), we have m n L y,= l w i r X j = 1 1 ( r ) “ (r) j-H-l … 户I j=l 1=1 Therefore, m n L y,-ur= I I K rW r ,. v u, ( p, r=l >=1 1=1 in n in n L = i i u ,⑴. ic . x,. u , ^ + 1 1 1 ii …—r). u … x ( - u r 1=1 ^ ‘ r=l )二1 1=2 = I u,(p) • < ) . X】• u, ( p ) + 1 1 u , , n - X, . u:°. u,,p) m L i l I ( = 2 in n L I l u ^ u ^ ^ - u r - x , + I I l u , ,r) " (f' 1 1=2 in L I • • Xj - U n rn = I u ; p ) • X, + IC (x ,u ( r ) ) . u, > n - 2 d , - m ( L - I) - 1 C ( x , u n - 2 d , - m ( L - l ) - ( m - l ) (C I i m + 2d,) 153 = n - (m - 1 )-Cirew - m(L - 1) - 2md| (A3-29) = n -»- C ^ - m[ 2d, + (L - I) -f CM„X 1 > 0 , (A3-30) where the relation of C(x, u) < C„m + 2d| has been used. From eqns.(A3-29) and (A3-30), we have the following respective results: n - m ( L - l ) - ( m - 1 )CIIUX < m < 2m n + C,磁 2d, + L - 1 + C, (A3-31) (A3-32) 154 Appendix IV Investigation into the Upper Limit of the Asymptotic Storage Capacity of Hopfield Associative Memory from two Different Viewpoints A4.1 By the Neural Dynamic Method Assume fundamental memory (FM) ulpl is taken as the network's input. By eqns.(6-l) and (6-2), we thus have the summed input of neuron i as follows: n in 1 n y ^ l w . - x ^ I Z C W . 户丨 r=l j=l,灼 Therefore, m n y, • U,,p> = 1 111.<" • 11,⑴• u r . U,…丨 z f u l u , I H II r + u , p H . u , p) u , + p> ^ u , u , u , I r I m n = - m + n + I I u广• u,n-m-(m-l)|C(u(r) ,u , p , ) l m a x = n - m - n ( m - l ) + 2(m—1 >dmm(u⑴，u,p') = 2 n - 2dlmn(u 0 , (A4-1) where eqn.(6-3) has been undertaken and. I C(u)]， (A4-2) r^ p j= I r^ p r^ p d(u(r)，u)]， (A4-3) 155 and H stands for the Hamming distance. Here, d(u l"，u lp )) is defined as an extensive distance between u) is the minimum extensive-distance between u'n and uip), r ^ p, and is equal to 0 n. A4.2 By the SNRG Method We try to find out a learning weight for an FM so as to make its signal to noise ratio gain (SNRG) be maximised subject to a constraint that a total summation of all the learning weights be a constant. Or, the problem is to maximise G ,p ' , p = I, 2, •••, m. with the constraint of m I a , r ) = a m , (A4-5) r=! r ” where a is implied to be an average value of the learning weights. The introduction of the constraint reflects the fact that all the SNRGs are a compromise between them implied in eqn.(6-9). All the SNRGs mutually affect themselves and not a single SNRG will be too large to make other SNRGs be greater than or equal to their thresholds. A procedure for finding out appropriate learning weights for the FMs under the above consideration is discussed below. Define a function such that in I ( a < r y T = 0 (A4-7) We thus have a , p , = I (a,r)) (A4-8) 2 ( m - l ) Replacing a) a,p> -m-1 r=l.*k I ( a , r r = 0 (A4-11) Its solution is obtained as follows: a lp) = q ]n-Q/qm (A4-12) in which I («) =i.印 Q = (A4-13) and I (a,rV = I.*T a m a m ((一)： m-1 4 I (a,r)) (r)x2 il/： m~l 27 r=i.*p I (a(r>)2 a m > m-1 a m 1 3>/3 r=l 邛 K ^ ' V ) (r)v2 J/2 il/2 m tn 川、 / V / J”、2 J/2 Since a m = S a' > ( Z (a1"厂）•，we accordingly have r^l n=l tn m 川、2 / v /^ ,(rK2J/2 K a ( l =1•印 q > ( K a ( r V ) ' m-1 川、2、l/2 , ( I ( a , r y ) ' 4 nH m ( I ( a , r > ) : ) 1 / 2 ] , / 2 } m—I i^i.^r m m ,(rK2 / v /〜⑴、2、l/2 , a (t -> ( I (a(' m-1 I ( a , r , r 4 3V3 ITh-1 ,1/2 158 I ( a ' =i.印 ⑴、2 ( I (a , r > ) 2 ) 1 / 2 m - I (rK2、l/2 •( I (a ( rV ) l V J ( m - l ) , / 2 = ( I (a⑴)2 严 • A (A4-14) where A = (A4-15) 2 (m- l ) 3 ^ ( m - l ) l / 2 From eqns.(A4-12), (A4-I3) and (A4-I4), we obtain a final solution to eqn.(A3-l I) as follows: I (a A i n - ( K a < r > ) 2 ) , / 2 -r=l,印 r=l.*p A1^ - ( I (a , ry ) r=l *P ,(r)x2 ,1/2 1/2 \ l / 2 K i / a a (邛 IL I (A…一 A \ r \ )， ni (A4-16) 159 References 1. Y.S.Abu-Mostafa and J.-M.St.Jacques, "Information capacity of the Hopfield modef,’， IEEE trans, on Information Theory, vol.IT-31, no.4, pp.461-464, July 1985. 2. K.Araki and T.Saito，“An associative memory including time-variant self-feedback,” Neural Networks, vol.7, no.8’ pp. 1267-1271, 1994. 3. C.S.Bak and M.J.Little, "Memory capacity of artificial neural network with high order node connections," Proc. of the IEEE International Conf. on Neural Networks, San Diego, CA, USA, vol.1, pp.207-216, 1988. 4. Y.Baram, “On the capacity of ternary Hebbian networks," IEEE Trans, on Information Theory, vol.37, no.3, pp.528-534, May 1991. . 5. J.Bruck, “On the convergence properties of the Hopfield model," Proceedings of the IEEE, vol.78, no. 10，pp. 1579-1585, Oct. 1990. 6. J. Bruck and V. P. Roychowdhury, “On the number of spurious memories in the Hopfield model," IEEE Trans, on Information Theory, vol.36, no.2, pp.393-397, March 1990. 7. J.Buckingham and D.Willshaw, “On setting unit thresholds in an incompletely connected associative net，” Network, vol.4, pp.441-459, 1993. 8. T.-D.Chiueh and R.M.Goodman, "High capacity exponential associative memories," Proc. of the IEEE International Conf. on Neural Networks, San Diego, CA, USA, vol. I，pp. 153-160, 1988. 9. T.-D.Chiueh and R.M.Goodman, “Recurrent correlation associative memories," IEEE Trans, on Neural Networks, vol.2，no.2，pp.275-284, March 1991. 10. T.-D.Chiueh and H.-K.Tsai, "Multivalued associative memories based on recurrent networks,” IEEE Trans, on Neural Networks, vol.4, no.2, pp.364-366, Mar, 1993 11. P.A.Chou, “The capacity of the Kanerva associative memory," IEEE Trans, on Information Theory, vol.35, no.2, pp.281-298, Mar. 1989. 12. S.Coombes and J.G.Taylor, "Using generalized principal component analysis to achieve associative memory in a Hopfield net,” Network, vol.5, pp.75-88, 1994. 13. A.Dembo, “On the capacity of associative memories with linear threshold functions," IEEE trans, on Information Theory, vol.35，no.4, pp.709-720, July 1989. 14. H.Englisch and M.Herrmann, “Associative memory for patterns with different bias," Network, vol.4, pp.223-242,丨993. 15. De Felice, C.Marangi, G.Nardulli, G.Pasquariello and L.Tedesco, "Dynamics of neural networks with non-monotone activation function," Network, vol.4, pp. 1-9, 1993. 16. P.Floreen, "Worst-case convergence times for Hopfield memories," IEEE Trans, on Neural Networks, vol.2, no.5, pp.533-535, Sept. 1991. 17. W.G.Gibson and J.Robinson, "Statistical analysis of the dynamics of a sparse associative memory," Neural Networks, vol.5, pp.645-661, 1992. 18. J.J.Hopfield, “Neural network and physical systems with emergent collective computational abilities," Proc. Nat. Acad. Sci” U.S., vol.79, pp.2554-2558, 1982. 19. B.Hunt, M.S.Nadar, P.Keller, E.Von Collu and A.Goyal, "Synthesis of a nonrecurrent associative memory model based on a nonlinear transformation in the spectral domain," IEEE Trans, on Neural Networks，vol.4, no.5, pp.873-878, Sept. 1993. 160 20. G.M.Jacyna and E.R.Malaret, "Classification performance of a Hopfield neural network based on a Hebbian-like learning rule," IEEE Trans, on Information Theory, vol.35, no.2, pp.263-280, Mar. 1989. 21. Hoon Kang, "Multilayer associative neural networks (MANN's): storage capacity versus perfect recall," IEEE Trans, on Neural Networks, vol.5, no.5, pp.812-822, Sept. 1994. 22. J.M.Karlholm, “Associative memories with short-range, higher order couplings," Neural Networks, vol.6, pp.409-421, 1993. 23. A.Kuh and B.W.Dickinson, "Information capacity of associative memories,” IEEE Trans, on Information Theory, vol.35, no. 1, pp.59-68, Jan. 1989. 24. Y.C.Lee, G.Doolen, H.H.Chen, G.Z.Sun, T.Maxwell, H.Y.Lee and C.L.Giles, "Machine learning using a higher-order correlation network," Physica D, vol.22, no. 1-3, pp.276-306, 1986. 25. R.P.Lippmann, "An introduction to computing with neural nets，" IEEE ASSP Magazine, pp.4-22, April 1987. 26. R.J.McEliece，E.C.Posner, E.Rodemich and S.Venkatesh, "The capacity of the Hopfield associative memory," IEEE Trans, on Information Theory, vol.33, pp.461-482, 1987. 27. I.Meilijson, E.Ruppin and M.Sipper, “A single-iteration threshold Hamming network," IEEE Trans, on Neural Networks, vol.6, no. 1, pp.261-266, Jan. 1995. 28. M.Mori ta, "Associative memory with nonmonotone dynamics," Neural Networks, vol.6, pp.115-126, 1993. 29. I.Nemoto and M.Kubono, "Complex associative memory," Neural Networks, vol.9，no.2， pp.253-261,丨996. 30. H.Nishimori and I.Opris, “Retrieval process of an associative memory with a general input-output function," Neural Networks, vol.6, pp. 1061-1067, 1993. 31. H.Oh and S.C.Kothari, “Adaptation of the relaxation method for learning in bidirectional associative memory," IEEE Trans, on Neural Networks, vol.5, no.4，pp.576-583, July 1994. 32. P.D.Olivier, “Optimal noise rejection in linear associative memories，” IEEE Trans, on Systems, Man, and Cybernetics, vol. 18, no.5，pp.814-815, Sept./Oct., 1988. 33. G.Parodi，S.Ridella and R.Zunino, “Using chaos to generate keys for associative noise-like coding memories’” Neural Networks, vol.6, pp.559-572, 1993. 34. R.Perfetti, "Mapping binary associative memories onto sigmoidal neural networks using a modified projection learning rule," IEEE Trans, on Circuits and Systems-II: Analog and Digital Signal Processing, vol.41, no.7，pp.474-477, July 1994. 35. D.E.Rumelhart, J.McClelland, and P.D.P. research group, Parallel Distributed Processing. Cambridge, MA, USA: MIT Press, 1986，vol.1, Explorations in the Microstructure of Cognition. 36. S.W.Ryan and J.H.Andreae, “Improving the performance of Kanerva's associative memory，” IEEE Trans, on Neural Networks, vol.6, no. 1, pp. 125-130, Jan. 1995. 37. D.Sal'ee and Y.Baram, “High-capacity Hebbian storage by sparse sampling/' IEEE Trans, on Neural Networks, vol.6, no.2，pp.349-356, March 1995. 38. F.Schwenker, F.T.Sommer and G.Palm, "Iterative retrieval of sparsely coded associative memory patterns,” Neural Networks, vol.9, no.3，pp.445-455, 1996. 39. Mahdad N.Shirazi, Mehdi N.Shirazi and S.Maekawa, “The capacity of associative memories with malfunctioning neurons," IEEE Trans, on Neural Networks, vol.4, no.4, pp.628-635, July 1993. 161 40. P.K.Simpson, “Higher-ordered and intraconnected bidirectional associative memories’” IEEE Trans, on Systems, Man, Cybernetics, vol.20, no.3，pp.637-653, May/June 1990. 41. B.Soffer，"Holographic associative memories," Proc. of Workshop on Neural Network Devices and Applications, Jet Propulsion Lab., CA, USA, pp. 125-146, 1987. 42. T.Stiefvater, K.-R.MuIler and H.Janssen, "Sparsely connected Hopfield networks for the recognition of correlated pattern sets," Network, vol.4, pp.313-336, 1993. 43. S.I.Sudharsanan and M.K.Sundareshan, "Equilibrium characterization of dynamical neural networks and a systematic synthesis procedure for associative memories,” IEEE Trans, on Neural Networks, vol.2, no.5, pp.509-521, Sept. 1991. 44. H.J.Sussmann, “On the number of memories that can be perfectly stored in a neural net with Hebb weights," IEEE Trans, on Information theory, vol.35, no. 1，pp. 174-178，Jan. 1989. 45. S.Tan，"Pattern storage and Hopfield neural associative memory with hidden structure," International Journal of Neural Systems, vol.3, no.3，pp.315-322, 1992. 46. C.M.Thomas, W.G.Gibson and J.Robinson, "Stability and bifurcations in an associative memory model," Neural Networks, vol.9, no. 1, pp.53-66, 1996. 47. D.E.Van Den Bout and T.K.Mil ler, “Graph partitioning using annealed neural networks," IEEE Trans, on Neural Networks, vol. 1，no.2，pp. 192-203, June 1990. 48. S.S.Venkatesh and D.Psaltis, “Linear and logarithmic capacities in associative neural networks，” IEEE Trans, on Information Theory, vol.35，pp.558-568, 1989. 49. T.Wang, “Improving recall in associative memories by dynamic threshold," Neural Networks, vol.7, no.9, pp. 1379-1385, 1994. 50. Y.-F.Wang, J.B.Cruz and J.H.Mulligan, "Two coding strategies for bidirectional associative memory," IEEE Trans, on Neural Networks, vol. 1，no. 1，pp.81 -92, 1990. 51. Y.-F.Wang, J.B.Cruz and J.H.Mulligan, “Guaranteed recall of all training pairs for bidirectional associative memory," IEEE Trans, on Neural Networks, vol.2, no.6，pp.559-567, 1991. 52. Y.-F.Wang, J.B.Cruz and J.H.Mulligan, "Multiple training concept for back-propagation neural networks for use in associative memories," Neural Networks, vol.6, pp. 1169-1175, 1993. 53. C.-C.Wang and H.-S.Don, "An analysis of high-capacity discrete exponential BAM，” IEEE Trans, on Neural networks, vol.6, no.2, pp.492-496, Mar. 1995. 54. W.-J.Wang and D.-L.Lee, “A modified bidirectional decoding strategy based on the B A M structure," IEEE Trans, on Neural Networks, vol.4, no.4, pp.710-717, July 1993. 55. J.-K.Wu’ Neural Networks and Simulation Methods. New York • Basel • Hong Kong: Marcel Dekker, Inc., 1994, Chapter 9-Associative Memory. 56. Z.-B.Xu, Y.Leung and X.-W.He，“Asymmetric bidirectional associative memories,” IEEE Trans, on Systems, Man and Cybernetics, vol.24, no. 10, pp. 1558-1564, Oct. 1994. 57. G.G.Yen，"Eigenstructure bidirectional associative memory: an effective synthesis procedure," IEEE Trans, on Neural Networks, vol.6, no.5，pp. 1293-1297, Sept. 1995. 58. G.Yen and A.N.Michel, " A learning and forgetting algorithm in associative memories: The eigenstructure method,” IEEE Trans, on Circuits and Systems-II: Analog and Digital signal processing, vol.39, no.4, pp.212-225, Apr. 1992. 59. S.Yoshizawa, M.Mori ta and S.-I.Amari, "Capacity of associative memory using a nonmonotonic neuron model’，，Neural Networks, vol.6，pp. 167-171，1993. 162 60. B.-L.Zhang, B.-Z.Xu and C.-P.Kwong, “Performance analysis of the bidirectional associative memory and an improved model from the matched-filtering viewpoint，” IEEE Trans, on Neural Networks, vol.4, no.5, pp.864-872, Sept. 1993. 61. LV-H.Zou and J.Lu, "Optimal linear associative memories for noise rejection," Proceedings of the 35th Midwest Symposium on Circuits and Systems, vol.2, pp. 1373-1376, Washington, DC, USA, 9-12 Aug., 1992. 163 Author's Paper List 1. Kwong-sak Leung, Han-bing Ji and Yee Leung, "Weighted outer-product learning associative memory with adaptive algorithms,” in Proceedings of the IASTED Internationa] Conference of Artificial Intelligence, Expert Systems and Neural Networks, Zurich, Switzerland, pp. 100-105, July 1994. 2. Han-bing Ji, Kwong-sak Leung and Yee Leung, “Gaussian correlation associative memory," in Proceedings of IEEE International Conference on Neural Networks, Perth, Australia, pp. 176卜 1766, November 1995. 3. Han-bing Ji, Kwong-sak Leung and Yee Leung, “A novel encoding strategy for associative memory," Artificial Neural Networks - ICANN96, C.v.d.Malsburg, W.v.Seelen, J.C.Vorbruggen, B.Sendhoff, eds., Springer Verlag, Berlin, Heidelberg, New York, vol.1112 of Lecture Notes in Computer Science, pp.21-27, 1996. 4. Kwong-sak Leung, Han-bing Ji and Yee Leung, "Adaptive weighted outer-product learning associative memory," to be published in IEEE Trans, on Systems, Man, and Cybernetics, 1997. t 5. Han-bing Ji, Kwong-sak Leung and Yee Leung, "Correlation-type associative memory using the Gaussian function," submitted to IEEE Trans, on Circuits arid Systems, Part I, 1996. 6. Han-bing Ji, Kwong-sak Leung and Yee Leung, “A novel encoding strategy based neural associative memory with maximum stability," submitted to IEEE Trans, on Systems, Man, and Cybernetics, 1996. 164

The stability and attractivity of neural associative memories.

Abstract

Similar works

Full text

Available Versions

CUHK Digital Repository