The mathematical neuron basic cells used as basic cells in popular neural network architectures and algorithms are discussed. The most popular neuron models (without training) used in neural network architectures and algorithms (NNA) are considered, focusing on hardware implementation of neuron models used in NAA, and in emulation of biological systems. Mathematical descriptions and block diagram representations are utilized in an independent approach. Nonoscillatory and oscillatory models are discusse

Linares Barranco, Bernabé

Rodríguez Vázquez, Ángel Benito

Sánchez Sinencio, Edgar

English

idUS. Depósito de Investigación Universidad de Sevilla

CMOS CIRCUIT IMPLEMENTATIONS FOR NEURON MODELS Bemabe' Linares-Barranco' *2, Edgar Sdnchez-Sinencio' , Angel Rodriguez- VcizqueZ 'Texas A&M University Department of Electrical Engineering College Station, Texas 77843-3128 2Dpto. Diseiio Analbgico Centro Nacional de Microelectrbica 4104 Sevilla, SPAIN ABSTRACT Discussion of the mathematical neuron basic cells employed in popular neural network architectures and algorithms are presented. From the fundamental Mc Culloch-Pitts model up to  the resultant model from Carpenter-Grossberg Neural Nets are discussed. We consider the viability of the implementation of these models in CMOS technology using transconductance and current-mode techniques. Simulations and exper- imental results from discrete and CMOS test-chips are presented. I. INTRODUCTION Artificial neural networks research was very active in the 60's. In the last few years, a reviving interest in this field [ I ]  has rapidly grown. One of the main reasons of this reno- vated interest is the potential of hardware im lementations of these nets. Artificial neural networks (ANN87 mainly consist of basic cells (neurons) heavily interconnected through variable weighted links (synapses). The neuron outputs are typically weighted (via synapses) and injected (through dendrites) to the neuron inputs. The ANNs can be trained to implement dif- ferent tasks (i.e., classifiers, image, speech recognition). This training is usually accomplished by changin the weights (of the synapses) using strategic learning algoritfms. In this paper, we specifically deal with the most popular neuron models (without training) used in neural network archi- tectures and algorithms (NNA). The focus will be on hardware implementation of neuron models used in NNA, and in emula- tion of biological systems. In this highly interdisciplinary field of NN, authors have not agreed in selecting one unique neuron model for all ap- plications. Therefore, regarding hardware implementations we should also consider the implementations of the different neu- ron models. Two attractive circuit implementation techniques [2,3 : current-mode and transconductance-based will be consider eJ . In some convenient cases, some extra current to voltage (or vice versa) converters will be included in the implementations. 11. NEURON MODELS We will discuss several common neuron models in this section. Mathematical descriptions and block diagram rep- resentations will be utilized. This approach is technology- independent. Non-oscillatory and oscillatory models are dis- cussed. Basic Neuron Cell One of the most conventional basic neuron cells 41 is shown in Fig. 1 .  The inputs x i k  and weights Wit  are re I ated to the h e a r  output by n Note that a threshold value might be introduced by making one of the inputs equal to one, and the corresponding weight equal to the threshold value. the nonlinear ouput yk is where in general fn(-)  is a limiting or threshold function. Sev- eral common limiting (also called activation) functions are shown in Fig. 2. The original McCulloch-Pitts neuron used only the binary (hard-limiting) function. Later development of learn- ing algorithms required having a differentiable limiting func- tion [1,13] such as Fig. 2 b and the sigmoid of Fig. 2 k ) .  additional dynamics is next discussed. Basic Dynamic Neuron Cell This cell is often used in solving optimization problems. Hopfield [5,6] has used it and made it popular in his optimiza- tion network solutions. The mathematical description of the dynamic neuron cell is given by yk = f n ( s k )  ( 2 )  One variation of the basic M cCulloch-Pitts neuron involvlng ( 3 )  where rink is an independent input signal and 1/Rk is the self- relaxation term which makes the output s k  zero for no input. w,k are the weights for the different inputs f ( S i ) ,  and fk) is a sigmoidal limiting function as shown in Fig. 2(c). T e corresponding block diagram of (3) is shown in Fig. 3. The summation term in (3) is sometimes split into two, to consider the excitatory and inhibitory inputs. This yields n E  sk n' dSk c k x  = l i n  - - Wikf(Si )  + wikf(si) ( 4 )  i=l i= 1 Note that the growth rate tion of Fig. 2(b) or 2(c) is not a The most general neural network topology is claimed by Grossberg [14]. According to this topology the differential equation describing the activity sk of each of the neuron cells in the system is given by: r n 1 L Gl where Ik, Jk are the external inputs, Dik and Gik describe the weight of the interconnections between neurons, f i ( . )  and si( .)  are nonlinear functions, A& is the self or forgetting term2, and the constants Bk, c k ,  uk and Fk are responsible for the shunting properties of this net that provide automatic gain control and total activity normalization [14]. Equation ( 5 )  can be rewritten in order to merge the posi- tive and negative terms as f o ~ ~ s  & is related to the previous models by & = 1/RkCk. -242lSl.00 Q 1990 IEEE This equation can be represented schematically by the block diagram of Fig. 4. Most of the known neural network archi- tectures and algorithms can be considered as special cases of this net (note that we are not considering the learning mech- anism but only the short term memory STM). For example, the ART1 [15] and ART2 [16] STM equations correspond pre- cisely to  the block dia ram in Fig. 4; additive models such as the ones by Hopfield f5], Anderson [17], McCulloch-Pitts [4] or Ackley [18] are simply obtained by eliminating the shunting terms in Fig. 4., i.e., H k  = 1, LI; = 0; feedforward nets like the back propagation algorithm [19] are obtained by properly making zero some of the weights ztJ .  As mentioned before, different shapes can be used for the nonlinear function ft(S,). So far, we have discussed only what we call non-oscillatory neurnn models, where the output of the neuron v k  is a non- linear monotonic function of the internal potential s k .  Let us consider the other general case. Analog Neural Oscillator Cell For biological oscillatory neurons, the output is a firing sequence of pulses whose frequency is a nonlinear monotonic function of the internal potential. In this case, we have to  substitute the function fk(.) in Fig. 4 by a voltage-controlled- oscillator (VCO) whose output frequency is controlled nonlin- early by S k ,  as shown in Fig. 5 .  One of the closest models resembling biological neurons is the Hodgkin and Huxley model [7]. We discuss two simplified versions of the Hodgkin and HLxley model. First the hysteretic neural type oscillator cell [8-101 is presented. In this case the VCO of Fig. 5 is implemented according to  the block diagram of Fig. 6. The corresponding characterization of Fig. 6 is given by C k L  d X  = -- X k  - G  H H ( X k ) - S k  (7) d t  Rk where H ( X k )  describes the hysteresis nature of the model and Fk is the oscillation frequency of the signal V k .  Thus F k  will have an oscillatory frequency dependent on S k .  Other alternatives to  feed SI, into the VCO instead of connecting it to the summing input, as shown in Fi . 6, are to either make Rk or H+ and H -  dependent on SI; [9f Also note that in this model for oscillating neurons all the signals VI to V, have equal amplitude but the oscillating frequencies are different. The next model presented is the FitzHugh-Nagumo, which is one of the most complex, with potential application in emu- lation of biological systems. This cell can be characterized by a second-order system of the following form: Where S k l  and S k Z  represent the summation and integration of inputs coming from other neurons and/or independent inputs, and f ( X k 2 )  was ori inally [11] suggested as a cubic polynomial. It has been shown f12] that f ( x k 2 )  can be approximated as a piecewise linear function as shown in Fig. 7(a). The block di- agram representing eqs. (9a) and (9b) is shown in Fig. 7. The solution of (9) yields a Van der Pol when the time constants associated with (9a) and (9b) are equal. 111. NEURON MODEL IMPLEMENTATIONS: In order to  exploit the advantages and to avoid the draw- backs of VLSI CMOS integration we will consider two differ- ent approaches of implementation, the transconductance-mode and the current-mode. Both avoid the use of resistors and allow use of tunable synapses in the network. Also, we will not consider any shunting mechanism in our future discussion, therefore H i  = 1 and L ;  = 0 in Figs. 4 and 5 .  A. Non-Oscillatory A. l .  Transconductance Mode: This mode allows a very simple implementation for the integration device of the neuron (see Fig. 3), namely a capacitor as shown in Fig. 8. The saturation of the output voltage is accomplished by a limiting device (a simple inverter), whose characteristic is also shown in Fig. 8. Observe that W i k  = G i k  can take any polarity [3]. A . & .  Current-Mode: The current-mode comparator [9] shown in Fig. 9 allows the implementation of the limiting function of Fig. 9b. This element is symbolically represented in Fig. 9c. The complete implementation of eq. (3) using this technique is given in Fig. 10. Note that the weights of the interconnections are determined by the (F) ratio of the synaptic transistors Mi. The resitance Rk that implements the self-term of eq. (3) can be made using an MOS transistor working in its triode region. Rk can also be assumed composed as the parallel combinaiton of the output resistors of the transistors. B. Oscillatory Neurons B. 1: Transconductance-Mode: The fundaments of this neural modeling that is based on a hysteretic element were presented elsewhere [9]. According to Fig. 5, the equation relating the input ( S k )  and output v k  of the VCO would be of the type d l i k  7= - g l x k  - H(Xk) - g Z S k r  v k  = H(Xk) (10) where g l  and g2 are constants and H ( . )  is a hysteresis func- tion. The frequency is controlled by s k .  We denote the value of this frequency by F k  = f ( S k ) .  A slight modification of this circuit in order to  obtain a suitable CMOS implementation is shown in Fig. 11 [13]. Here g2 = 0 and S k  is fed into the VCO through 91, which is made to depend nonlinearly on s k .  This hysteretic model can be included in the more general model proposed by FitzHugh-Nagumo [ll], by noting that in reality a hysteretic element is a simplification of a second order sys- tem [20]. An OTA-C implementation of FitzHugh-Nagumo’s differential equation using nonlinear OTA-C techniques [3] has been presented elsewhere [21]. The set of differential equations is, according to Fig. 12, (11) g m l s k  + g m l x k  - g m 3 V k  = C l v k  - g m Z V k  - f n ( - x k )  = C Z X k  Where g n 2 1 ,  g m 2 ,  c1 and CZ are constants, x k  is an internal variable (same as X k l  in eq. (9)) of the VCO, V k  is the output (same as xk2 in eq. (9)) and fn(.) is an N-shaped nonlin- ear function as shown in Fig. 12 together with the OTA-C implement ation. D.2: Current-Mode: Based on the current-mode comparator of Fig. 4, it is very simple, to implement a current-mode hys- teretic element [22] as shown in Fig. 13. A complete hysteretic- type current mode VCO model for Fig. 5 is shown in Fig. 14. It is essentially composed by a hysteretic element and a non- linear transconductor, and it works under the same principle as the previous circuit of Fig. 11. Iv. EXPERIMENTAL AND SIMULATED RESULTS The circuit of Fig. 11 was fabricated in a 3pm CMOS process (MOSIS). Fig. 15 shows the switching of the output between the on (oscillations) and off (no oscillations) states depending on the input, S k ,  being above or below threshold. At this time we built a discrete prototype for the FitzHugh- Nagumo circuit of Fig. 12. We can observe in Fig. 16 how the input (lower trace) makes the VCO to change between the fir- ing state (on) and the non-firing (off) state. Only simulation results are available for the current-mode circuits of Fig. 13 and 14. In Fig. 17, we can see the hysteresis loop generated by the circuit of Fig. 13, and in Fig. 18, we can see the on-off operation of the circuit of Fig. 14. V. CONCLUSIONS -4 discussion of neuron models using mathematical de- scriptions and block diagrams which are suitable for CMOS integration was presented. Two approaches have been used to 2422 address this problem: the transconductance and the current- mode. Both of these techniques are suitable for IC designs. Several experimental and simulated results of the prototypes for 3pm CMOS processes (MOSIS) were presented. f n  (*) Y k  Threshold Function x 2* REFERENCES [ 11 3. A. Anderson, and E. Rosenfeld, editors. Neuwcomputing: Founda- tions of Reseamh. The MIT Press, Cambridge, MA 1988. Heteroassociative Memory Using Current-Mode MOS Analog VLSI Circuits”. IEEE 7”s. Cimuits Swst., Vol. 36. pp. 747-755, (Special [ 21 K. A. Boahen, P. 0. Pouliquen, A. G. Andreou, and R. E. Jenkins, ”A X *  ~ i ~ .  1. ~i~ N~~~~~ cell [ 31 41 [ 51 [ 61 I 71 [ 81 I 91 . -  .. Issue on Neural Networks), May 1989. E. Sbcha-Sinencio, B. Linares-Barranco, J. Ram’rez-Angulo, A. Rodriguez-Vkquez, “OTA-Based Nonlinear Function Syntheses,” IEEE Joumal of Solid-State Circuits, December 1989. W. S. McCulloch, and W. Pitts, “A Logical Calculus of the Ideas Imminent in Nervous Activity”, Bulletin of Mathematical Biophysics, Vol. 5, pp. 115-133, 1943. D. W. Tank, J. J .  Hopfield, ”Simple ‘Neural’ Optimization Networks: An AID Converter, Signal Decision Circuit, and a Linear Programming Circuit“, IEEE Bans.  Cimuiis Sys., Vol. CAS-33, No. 5, pp. 533- 541, May 1986. J .  3. Hopfield, D. W. Tank, “ ‘Neural’ Computation of Decisions in Optimization Problems”, Eiol. Cybem, Vol. 52, pp. 141-152, 1985. A. L. Hodgkin, A. F. Huxley, “A Qualitative Description of Membrane Current and Its Application to Conduction and Excitation in Nerves”, J.  Physics, Vol. 77, pp. 500-544, 1952. G. Kiruthi, R. C. Ajmera, R. Newcomb, T. Yanis, H. Yazdain, “A Hysteretic Neural-Type Pulse Oscillator”, IEEE h i .  Symp. Cim. B. Linares-Barranco, E. Sbchez-Sinencio, A. Rodriguez-Vkquez, and J.L. Huertas, “A Programmable Neuron Oscillator Cell”, IEEE ’hns .  Circuits Sysi., (Special Issue on Neural Networks) Vol. 36, pp. 7576- 761, May 1989. Syst.,  pp. 1173-1175, 1983. [lo] B. Linares-Barranco, E. Sanchez-Sinencio, R.W. Newcomb, A. Fbdriguez- Vbquez, J.L. Huertas, “A Novel CMOS Analog Neural Oscillator Cell”, IEEE Int. Symp. Circ. Syst., pp. 794-797, May 1989. [ l l ]  R. FitzHugh, “Mathematical Models of Excitation and Propagation in Nerves”, National Institule of Health, Technical Repori, 1967. (121 J.P. Keener, “Analog Circuitry for the Van der Pol and FitzHugh- Nagumo Equations”, IEEE Zbuns. on Systems, Man and Cybernetics, Vol. SMC-13, Sept./Oct. 1933. [13] T. Kohonen, Self-Organization and Associative Memory, Springer- Verlag, 1988. [14] S. Grossberg, “Nonlinear Neural Networks: Principles, Mechanisms, and Architectures,” Neural Networks, Vol. 1, pp. 17-61, 1988. [15] G. A. Carpenter and S. Grossberg, “A Massively Parallel Architecture for a Self-organizing Neural Pattern Recognition Machine,” Computer Vision, Graphics and Image Processing, Vol. 37, pp. 54-115, 1987. [lG] G. A. Carpenter and S. Grossberg, “ART2: Self-organization of Sta- ble Category Recognition Codes for Analog Input Patters,” Applied Opiics, Vol. 26, pp. 4921-4930, December 1987. (171 J .  A. Anderson, J .  W. Silverstein, S. R. Ritz and R. S. Jones, “Distinc- tive Features, Categorical Perception, and Probability Learning: Some Applications of a Neural Model,” Psychological Review, Vol. 84, pp. 413-451, 1977. [18] D. H. Ackley, G. E. Hinton and T. J. Sejnowski, “A Learning Algorithm for Boltzmann Machines,” Cognitive Science, Vol. 9, pp. 147-169, 1985. [19] D. E. Rumelhart, G. E. Hinton and R. J. Williams, “Learning In- ternal Representatiions by Error Propagation,” E. E. Rumelhart and J.  C. McClelland (Eds.), Parallel Distribufed Pmcessing, Cambridge, MA:MIT Press. (201 L. 0. Chua, C. A. Desoes and E. S. Kuh, Linear and Nonlinear Cir- cuiis, McGraw-Hill, New York, 1987. P l ]  E. Sinchez-Sinencio and B. Linarea-Barranco, “Circuit Implementa- tion of Neural Fitzhugh-Nagumo Equations,” Proc. Midwest Symp. on Circuits and Systems, 1989. [22] Z. Wang and W. Guggeniihl, “Novel CMOS Current Schmitt Trigger,” Electronics Letters, 24, pp. 1514-1516, 1988. Fig. 2. Activation Functions: Binary, Hard-Limiting or Signum, (b) Threshold gic, (c) Signoidal Fig. 3. Dynamic Basic Neuron Cell Architecture Fig. 4. Block Diagram for General Non-Oscillatory Neuron Model I I T+ I Fig. 5. Block Diagram for General Oscillatory Neuron Model r-+ II Fig. 6. Hysteretic Neural Oscillator Cell 2423 t;'^'*' (b) Fig. 7. Fitzhugh-Nagumo Neuron; (a) Nonlinear Function - f ( X s k ) ,  (b) Block Diagram or Neuron Fig. 8. OTA-C Implementation of Non-Oscillatory Neural Network ( 8 )  (b) ( c l  Fig. 9. Current-Mode Comparator; (a) Circuit, (b) Transfer Function, (c) Symbol Fig. 10. Current-Mode Implementation of Nonoscillatory Neural Network Fig. 11. OTA-C CMOS Implementation of Oscillatory Neuron Fig. 12. OTA-C Implementation of Oscillatory FitzHugh- Nagumo Neuron I vss Fig. 13. Current-Mode Hysteresis Element Fig. 14. Current-Mode Oscillatory Neuron em, * m m  !sm. / ' " " " ' ~  Fig. Fig 15. Experimental Results for CMOS OTA-C Hysteretic Neuron 16. Experimental Results for OTA-C Fitzhugh-Nagumo Neuron Fig. 17. Simulated Current-Mode Hystersis Comparator Fig. 18. Simulated Results for Current-Mode Oscillatory Neu- ron 2424 

CMOS circuit implementations for neuron models

https://idus.us.es/bitstream/11441/89965/1/CMOS%20CIRCUIT%20IMPLEMENTATIONS.pdf

CMOS circuit implementations for neuron models

Abstract

Similar works

Full text

Available Versions

idUS. Depósito de Investigación Universidad de Sevilla