A novel CMOS analog neural oscillator cell

A NOVEL CMOS ANALOG NEURAL OSCILLATOR CELL B. Linares-Barranco',', E. S&chea-Sinencio1, R. W. Newcomb3, A. Rodriguez-V&quez2 and J. L. Huertad 'Dept. of Electrical Engineering, Texas A&M University, College Station, TX 77843-3128, U.S.A. 'Dpto. de Electronica y Electromagnetismo, Fac. de Fisica, Univ. de Sevilla, 41012-Sevilla, Spain 3Microsystem Laboratory, Elec. Engr. Dept., Univ.of Maryland, College Park, MD 20742, U.S.A. AbstractA very flexible programmable CMOS analog neural oscillator cell architecture is presented. The proposed neuron circuit architecture is a hysteretic neuraltype pulse oscillator. Its implementation consists of a transconductance comparator, a capacitor and two non-linear resistors. It has over nine decades of oscillation frequency range, i.e., 10-zHz 5 fo., 5 20MHz. This range has been experimentally versed. The oscillator cell in the test-chip was implemented in a standard 3pm (p-well), double-metal CMOS technology, and has a dimension of about 44,000pm' (without the capacitor). Preliminary measurements and simulated results agree very well. I. INTRODUCTION The interest in neural networks is old, one main reason has been the potential to solve difficult complex engineering problems [l] - [4] that could not be easily and practically solved with conventional approaches. Using synthetic neural networks, researchers try to mimic human-like performance to solve engineering problems such as control systems, including optimization, learning and adaptive systems, also strong interest exists for speech and image recognition. A renovated interest in neural networks has arisen in the engineering scientific community among others. One of the main driving forces behind this renovated interest lies in the fact that a very large number of highly interconnected arrays of basic cells (neurons) can be, in principle, efficiently fabricated BS integrated circuits. Several recent results [4] - 151 of implemented neural networks in CMOS technology show promising potential applications [2], [3], [lo] and [13]. Another attractive property of a neural network is its great fault tolerance and degree of robustness. A basic nerve cell is called a neuron. Biological neurons are permanently sending electrochemical signals back and forth to each other and modifying their communication lines with every new experience. Several researchers (see Refs. 7.1 and 7.3 in [l]) considered the brain as being associated with an energy surface whose valleys correspond to stable repeatable brain response. External stimulus determines the initial point of the energy surface. The brain, by means of firing sequency of pulses changing dynamically, reaches a stable pattern that corresponds to a valley floor on the surface. This energy surface question can be interpreted as the conventional minimization or optimization of a function often found in engineering problem (i.e., control system, circuit design, etc.), and where a stable pattern is the corresponding minimum of a function. A neuron has both on (excitatory) and of(inhibitory) connections, the output consists of positive' (or negative) packages of pulses for excitatory (or inhibitory) output signal. Neurons are interconnected through synapses. A synapse can be considered as a weighted relation between the output of a neuron and the input received by another one. This relation can be modeled as inhibitory or excitatory. Interconnections form neural networks, however, researchers [13] have not agreed on precisely which part of the nervous system to model and the exact degree of fidelity. The complexity and variety of inherent algorithm associated with the human nervous system is astonishing. Several crude mathematical models of neural nets have been reported in the literature [I] - [SI, [9] where the basic model consists of a neuron producing an output determined by a weighted sum of inputs coming from other neuron outputs and external (stimuli) signals. The mathematical description [4], [7], [ll] of the dynamic behavior of a neural networkZ with N neurons can be characterized by: 3 z; is the neural activity in neuron i, and can be interpreted as a voltage signal. In the above equation, Term 1 represents the passive decay of neural activity in the absence of all other signals (Terms 2 and 3), in which case z;(t) comes from a linear single time constant system and z;(t) = z;(o)e-Ait. Thus A; is the self term. 1/Ai has units of time and can be inversely related to a time constant. Term 2 corresponds to external stimuli input signals I,; which can be interpreted as input current sources. Term 3 involves the synaptic weighting functions T,j ' The positive (negative) packages of pulses are arbitrarily associated with the excitatory (inhibitory) output signal. Authors use different names for neural networks, such ar neuromorphic systems, connectionist models, parallel distributed processing models, artificial neural nets or synthetic neural systems. 794 ISCAS '89 CH2692-2/89/0000-0794 $1 .&I 0 1989 IEEE and ?;, and the neuron state Vj. Note that Ti, and ?,, model the excitatory and inhibitory functions, respectively. Vj is the neural state which is related to the neural activity z; by the nonlinear activation function I,(.), which bounds the values of Vj between a maximum and a minimum value as shown in Fig. 1, i.e., V, = fj(zj). Some authors use activation functions having a sharp transition as shown in Fig. l(a); others use a function with a defined transition region between V,,,, and Vjmin, (Fig. l(b)), and still others use functions with a soft transition as shown in Fig. l(c); these last two activation functions are also known as sigmoids. A variation of (1) where hysteretic elements [I], [4] - [8] are used instead of an activation function (i.e., Term 3 is modified) can be described' as: hi (24 -= dt -4%; + I,i - Ah, H;(zi, Vi) - S N-1 vi = (Tij - T;j)fj(zj) (2b) j=O where T;,, ?,j, fj(-), A, and I,i are defined as before, A,,, is a weighting non-negative constant and €I,(z;, Vi) describes a hysteresis type [SI - [8] of neuron model for a constant vl., i.e., if2 < -2if z 7 -z+ if -2- < z; < z+ ifz; = -2or z; =z+ H(z;, const) = {H+, -H-) [-H-, H+] (3) { H+ Fig. 2 illustrates the hysteresis loop characterized by (3). For the operation of a neuraldlator a line crossing the origin (in Fig. 2) in the 1" and 3rd quadrant is needed; then the oscillation or no'oscillation depends on the slop of the (linear) load line. Other equivalent cecillation mechanisms are possible for symmetric hysteresis loop and non-linear load resbtors. One of these mechanisms is introduced later. Note that the asymmetry (for linear resistor loads) of this loop is required to guarantee the existence of one stable point [8] as will later be needed. In this paper, we describe a CMOS analog neural oscillator cell that has some of the properties of a biological neuron. These inhibitory and excitatory properties are functions of the input sum exceeding (or not) a firing threshold level. The propceed cell is voltage programmable over a large oscillation frequency range. II. MATHEMATICAL APPROACH Let us rewrite (2) in a convenient form for our CMOS circuit implementation, that is for a single neuron characterized by the first-order state equations (for simplicity the subindice i is deleted): dz dt U, 2) + I&) (4) C- = -A2 - C( where U = input, z = internal state variable, C is a non-negative constant, I,[Z) is -AhH(z, V), and the input variable U controls the value of C(u, z) and since the U = I. term has been incorporated in G(u, z), it is no longer explicitly shown. The equilibrium points, neglecting the -Az term, are reached when % = 0, resulting in I.+.) = C(U, 2.) (5) C(u, z) is a nonlinear current which is defined as for z < 0 i~(z) = G(u, z) = ml(u)z for 0 < z < z1(u) (6) The solutions of (5) for two different G(u, 2)'s are pictorially shown in Fig. 3. Observe that the value of IC modifies the slope ml(u), thus IC must be a function of U. The mode of operation illustrated in Fig. 3(a) shows two equilibrium points, A and B, where A represents a stable point and B represents an unstable point. Thus, eventually the equilibrium point A is reached and z becomes 2.. Note that z, < V+. Thus Io(z,) = 4. The mode of operation illustrated in Fig. 3(b) has two unstable points A' and B', therefore, the circuit oscillates trying to reach a stable point. Next, we analyze (4) under this unstable mode of operation to determine the oscillating signal 2. Assume z(o) = 0 and I.(O+) = I&,, then (4) becomes' (" mz(u)z for z~(u) < z (7) dz Cdt = -m 1Z-k IS which solution yields This equation holds until z(t) becomes V+, (therefore, we assume & > V+) at which time the output of the hysteretic element changes to -b (we assume the ideal case where no time is taken in going from b to -16 on the hysteresis curve). Let tl be this time, with z(t1) = V+. Solving for tl we obtain: (9) -, after tlthe characterizing differential equation yields dz dt which solution is given by c- = -I& - mlz, with ~(0) = v+ (10) (11) This situation remains until t2, where z(t2) = 0. At this time iN(z) switches to 0 89 per (6). Therefore, solving for t; we obtain Now the differential equation becomes dz dt (13) C- = -1s but with z(0) = 0 the solution is time linearly dependent and given by z(t) = -&t C (14) The time at which IO(%) changes from -4 to Ib occurs when in (14) z(ts) = -V-, thus ts results cvtS = - The cycle is completed when the differential equation becomes (15) (16) Ib dz c-=I dt b with Z(0) = -vand the solution becomes ' The physical meaning of the term -&z, in (Za) is different from (1). 795 * For the sake of simplicity ml(u) U simply denoted as ml. - - rurtiicr imvrmarivn can ~e mraineu morn ut-. w. hennern JenKins. -~ (217) 333-4789 z(t) = -v- + Ft Ib (17) (18) Thus Z(t4) = 0, thus resulting cv - t4 = - The time diagram of the oscillation signals are illustrated in Fig. 4 where TI = C/ml and TZ = C/ml. 111. CMOS NEURAL OSCILLATOR CIRCUIT Ib The proposed neural oscillator cell architecture is shown in Fig. 5(a). This architecture consists of a hysteretic block, an integrator, and a non-linear resistor load RNL related to C(u, 2). A more detailed block diagram of the oscillator cell is illustrated in Fig. 5(b) where the hysteretic element is shown to consist of a comparator connected with positive feedback and a non-linear load RNH. The input (2) - output (Io) characteristics of the hysteretic element (consisting oTbhe comparator and non-linear resistor load) are given by Ib I 2 < v+ Io = { -IS , 2 > -v- (19) [-zb,zb] , 2=v+ , -vThe corresponding characteristics of i~ for the non-linear resistor RNL are given by (6). Observe that the output impedance of the comparator can be associated with the term 42, of (2a). The CMOS circuit implementation is shown in Fig. 5(c). Note that vb controls the value of Is through transistors M24, M21 and M9. M21 with M9 form a current mirror where the zb current is injected. Transistors Ml-M9 form the transconductance comparator, M10-Ml1 implements the non-linear resistor RNH. Observe that RNH consists of a diode connection of hw transistors. Current IOl, from Fig. 5(b), is obtained through MlZP-MlZN in Fig. 5(c). Transistors M22 and M23 allow us to have output currents of both polarities to be injected into other cells. V, determines the IC (of Fig. 3) injected through M20 and current mirror (M13 and M18). Vc consists of an inherent value Vc. (fixed) plus an input U. Furthermore, Vc determines the slope ml (see eqs (6) and (7)). The non-hear resistor load, RNL is implemented by transistors M13 to M19. When VN(= z) < 0, transistors M15 is off and the bias current, IC (also refer to Fig. 3), of the differential pair goes through M14. Thus the current, i~, through M17 and M19 is zero. When VN > 0, M14 is non-conducting and M15 is on, and the current through M17 and M19 becomes IC. That is i~ = IC. Since transistors M14 and M15 do not have large (W/L) ratios, there is a smooth transition between the on and off states of this nonlinear resistor. That is the reaSOn the resistor, RNL has been modeled by three piecewise linear segments. IV. EXPERIMENTAL RESULTS The circuit shown in Fig. 5(c) was implemented in a 3pm CMOS technology (through and thanks to MOSIS). A microphe tograph of the CMOS analog neural oscillator-cell is shown in Fig. 6. The cell occupies an area of about 44,000pm’ without including the capacitor. In our case, we used either a 1pF capacitor or the parasitic capacitance at that node. All the (W/L) ratios of the transistors of Fig 5(c) are 7pm/5pm. The experimental results for an oscillating frequency of 19.96 x loe Hz is shown in Fig. 7. The integrating capacitor for this result is the parasitic capacitance present at that output node. Fig. 8 shows simultaneously the voltages Vc and z = (VN),with vb = 3.98V for an oscillating frequency of about 92 KHz where vb is the voltage producing (see Fig. 5(b)). The oscillator-cell was capable of producing output pulses between 0.01 Hz and nearly 20 MHz. The experimental results for the dependance between the controlling voltage vb and the oscillating frequencies are shown in Fig. 9. V. CONCLUSIONS We have extended our preliminary work on a programmable neural oscillator cell [SI using discrete components. The pre posed monolithic cell is tunable over a range of 9 decades. It was built using ordinary doublemetal 3pm CMOS technology and has potential for fast speed applications. This neuron architecture has excellent voltage (or current) programmability properties. Extension of this neuron-oscillator cell to include the synaptic weighting functions as well as the nonlinear activition function is being considered. We are also currently investigating sound neural network architectures where the proposed cell can be fully exploited. VI. BIELIOGRAPHY 111 F. C. Hoppensteadt, An Introduction to the Mathcmoticr of Neurom, Cambridv University Pm, 1986. 121 C. Koch, J. Muroquin, and A. Yuille, ‘Andog ‘Neural’ Networks in Early Vision”, Proc. Natl. Acad. Sei, (Bwphyricr). Vol. 83, pp. 4263-4267. June 1986. 131 R. P. Lippman, ‘An Introduction to Computing with Neural Nets”, IEEE ASSP Mag., pp. 4-22, April 1987. [4] A. F. Murray and A. V. W. Smith, ‘A Synchronous VLSI Ned Networks Using PulrtStream Arithmetic“, LEEE 1. Solid-State Circuitr, Vol. 23, pp. 688-697, June 1988. [5] J. Hutchinson, C. Koch, J. Luo, and C. Mead, ‘Computing Motion Using Analog and Binary Resistive Networks”, Computer, pp. 52-63, March 1988. [6] N. El-Leithy and R. W. Newcomb, ‘Hysterisis in Neural-Type Circuits”, Proc. IEEE/ISCAS, Vol. 2, pp. 993-996, June 1988. [7] G. KiNthi, R. C. Ajmera, R. Nercomb, T. Yami and H. Yazdani, ‘A Hysteretic Neural Type Oscillator”, Proc. IEEE/ISCAS, Vol. 3.. pp. 11731175, May 1983. [a] B. Linares-Barranco. E. Shcher-Sinencio, A. Rodriguez-Vhquer, and J. L. Huertss, ‘A Programmable Neural Oscillator Cell’. to be published, IEEE Tram. on Circuitr and Syrtemr, May 1989. [9] A. Rodriguez-Vhquez, A. Rueda, J. L. HU&M and R. Domhguez-Csstro, ‘Switched Ca acitor ‘Neural’ Networks for Linear Programming’, Elmtron Letter#, $01. 24, pp. 49&498, April 1988. [lo] D. W. Tank and J. J. Hopfield, ‘Simple ‘Neural’ Optimization Networks: An A/D Converter, Signal Decision Circuit, and a Linear Programming Circuit”, IEEE Tram. Circuitr 8 Syrt., Vol. CAS-33, pp. 533-541, May 1986. [ll] J. J. Hopfield and D. W. Tank, “Neural’ Computation of Decisions Optimization Problems’, Biologicd Cybern.. Vol. 52, pp. 141-152, 1985. [12] C. Von Euler, ‘Centrd Pattern Generation During Breathing”, Trendr in Neurorciencer, pp. 275-277, November 1980. [13] J. I. Rahl, “Electronic Implementation of Neuromorphic Systems”, Proc. IEEE Cwtomed Integrated Circuitr, May 1988. Y V. Vj vjmax ymx fr ymax fr Fig. 1 Activation hnction. (a) Step Type, (b) Hard Type and (c) Soft Limiting (Sigmoid) Type. 7% H, (xl, Constant) %eFig. 2 Hysteresis Loop Characteristics of a Neuron Model. X X (a) (b) Fig. 3 Egnilibrium Point Characteriaticr. (a) Stable Mode, (b) Oscillating Mode. x(t> I Fig. 4 Time Diagram of the Output z(t) and Io1(t). I x<t> (b) Fig. 5 Neural-Oscillator Cell Architecture. (a) Block Diagram, (b) More Detailed Block Diagram, and 797 5 fcl MOS Circuit Imdementation VP = VP + U. .I. .I. I I Fig. 7 Neural Oscillator Output. Oscillating hqnency 19.6 MHz. I --._-._......- I Fig. 8 Vc shown in the Upper Ttace, and VN in the Lower Ttace. For an Oscillating Fkeqnency of 92.0028 KHs. VOltag* Fig. 9 Oscillating hquency vs. Controlling Voltage Vb for C = C, (parasitic capacitance) . __. - rurtiicr imvrmarivn can ~e mraineu morn ut-. w. hennern JenKins. -~ (217) 333-4789