Synaptic weight generation in VLSI stochastic neural networks

Synap ic Weigh Gene a ion in VLSI S ochas ic Neu al Ne wo ks. J.G. O ega, J.M. Que1.0, C.L. Jane and L.G. F anquelo Escuela Supe io de Ingenie os A da. F eina Me cedes s/n Se illa 41012 SPAIN .ABSTRACT Fully pa allel s ochas ic neu al ne wo k implemen a ions can be ealized nowdays. How- e e , in hese implemen a ions mos o he silicon a ea is consumed in he s ochas ic pulse sequence gene a ion ci cui s. In o de o imp o e hei e iciency in e ms o consumed silicon a ea, new echniques mus be de elopped. This is specially impo an in applica ions whe e a la ge numbe o synap ic weigh s a e needed. In his pape we p esen a new app oach ha can signi ican ly inc ease he e iciency os he 1. In oduc ion S ochas ic logic sys ems ealize pseudoanalog ope - a ions using s ochas ically coded pulse sequences, [l], [2]. In s ochas ic sys ems, he e ms ha a e o be p ocessed a e synch onous pulse sequences. In o ma ion is codi ied as he p obabili y, a a gi en clock cycle, o he pulse aking "high" alue. S ochas ic pulse sequences a e gene a ed in such a way ha all pulse s eams a e s ochas ically inde- penden . Conside now a se o n pulse s eams whose p obabili ies, a a gi en clock cycle, o being a "high" le el a e pl, p2, ..., p,. These p obabili ies a e mu ually independen . I hese sequences a e he inpu s o a n-inpu AND ga e, he p obabili y o he ga e ou pu , a a gi en clock cycle, o be- ing a "high" le el is equal o n::: pi. I is clea ha he p oduc ope a ion is achie ed by means o simple AND ga es, ha is, by a ex emely low a ea consuming ci cui . S ochas ic summa ion is a much mo e di icul ope a ion o pe o m, specially i he e ms o be added a e signed. Two ypes o ci cui s ha e been desc ibed in he bibliog aphy. One is he OR ga e and he o he is he up-down coun e . The up-down coun e echnique, al houglh is widely used in neu al ne wo k implemen a ion, [3]- [5], has a e y impo an d awback. Pulses coming om o he neu ons ha e o be mul iplexed in ime (i.e. se ialized) leading o high compu a ion imes. I wo pulse sequences a e he inpu s o an OR ga e and he pulse sequences o be added do no o e lap, he ou pu i ing p obabili y is equal o he addi ion o bo h i ing p obabili ies. This OR-based add unc ion is hus dis o ed by pulse o e lap. In echnique ha has been used up o now. ,._.__________.._.._.......... .... " Fig. 1: S ochas ic a chi ec u e. o de o achie e a guasy linea beha iou pulse den- si ies should s ay e y low, specially i many e ms a e o be added. This echnique does no pe mi he in eg a ion o neu ons wi h a e y high numbe o synap ic connec ions as i would lead o ex emely low maximum pulse densi y, [6]. I should be aken in o accoun ha he addi ion o wo numbe s ha ake alues anging om 0 o 1 may ake a alue bigge han one, which can no be ep esen ed by a p obabili y. In p e ious pape s we ha e p oposed a ully pa - allel s ochas ic compu a ion a chi ec u e sui able o neu al ne wo k implemen a ion, [7], [8]. I ci - cum en s one o he main d awbacks o s ochas ic compu a ion a chi ec u es ha ha e been used up o now: he absence o a space-e icien echnique o adding weigh ed inpu signals in pa allel. Howe e s ill emains an impo an p oblem o be sol ed: o ind a simple ci cui ha gene a es he s ochas ic signals. 0-7803-2768-3/95/$4.00 0 1995 IEEE 179 2. S ochas ic Pulse Gene a ion A ha dwa e implemen a ion o a mul ilaye neu al ne wo k based on his s ochas ic a chi ec u e was p esen ed in a p e ious pape , [9]. This pu ely digi al a chi ec u e is expandable, and he ci cui was designed so ha any mul ilaye ne wo k could be implemen ed by adding an app op ia e numbe o 1.C.s. Howe e mos o he silicon a ea o his implemen a ion is consumed in ci cui s in ol ed in s ochas ic pulse gene a ion o he synap ic weigh s (block M o Fig.(l)). In his implemen a ion, s ochas ic s eams o pulses a e gene a ed using a well known echnique ha has been b oadly used and desc ibed in he echnical li e a u e: Digi al codi ica ions o hese weigh s a e digi ally s o ed and hen compa ed wi h unco ela ed andom numbe s p oducing un- co ela ed s ochas ic signals, [2]-[5]. The e o e load egis e s, digi al compa a o s and a pseudo andom gene a o a e necessa y. Fig. 2: Pulse sequence co ela ion. To imp o e space e iciency we p oposed a new echnique o p oduce he andom pulse sequences ha codi y he synap ic weigh s, [lo]. The basic s ochas ic cell o his echnique is a high equency oscilla o whose T~ a e can be con olled, and a lowe equency sampling ci cui . The % a e is ixed equal o he synap ic weigh ha is mean o be s ochas ically codi ied. I he oscilla o ’s in- pu capaci o has a small alue, he oscilla o will be e y sensi i e o noise. Consequen ly, he e is a ce ain deg ee o unce ain y abou he ol age a oscila o ’s inpu , p oducing ime unce - ain y abou he momen in which he oscila o swi ches om ei he he on s a e o he o s a e o om he o s a e o he on s a e. I ollows ha oscilla o ’s phase canno be p edic ed a e i ai1 2.52 oc n T/dl 2ue Fig. 3: Phase noise. has swi ched se e al imes. In Fig.(2) i is shown he au oco ela ion o he pulse sequence, (A), as a uc ion o he noise in ensi y and he oscilla ion- sampling equencies a io ( / s). This igu e has been ob ained by simula ion conside ing ha he noise is whi e and gaussian. I he oscila o ’s ou pu is sampled wi h a lip lop a a slow enough a e, he sampled signal will be andom. The p obabili y o his signal aking he ’’high’’ le el will be equal In o de o p oduce comple e spa ial (be ween di e en pulse sequences) and ime (in each pulse sequence) andomness, phase unce ain y mus be g ea e o equal o 27 . Following hese ideas, we designed es boa d using disc e e componen s. Na u ally he maximum oscilla ing equency was a he modes , bu he measu ed c oss-co ela ion be ween di e en sampled oscilla o s and he au o- co ela ion we e e y encou aging. In Fig.(3) we show how noise acumula es p oducing phase unce - ain y. We decided o ace he VLSI implemen a- ion o he p oposed ci cui as we in end o de elop applica ions including a la ge numbe o neu ons and synap ic weigh s pe neu on. o +. 3. VLSI Implemen a ion The oscilla o consis s on i e consecu i e C-mos in- e e s. The ou pu o he i h in e e is conec ed o he i s in e e ’s inpu , leading o an uns able ci cui ha oscila es a a high equency. VI and V, a e ol age signals ha a e used o ix he To, and To alues. The ou pu is conec ed o a lip- lop ha samples i . The lip- lop clock signal is suplied ia an inpu digi al pad. The whole ci cui is plo ed in Fig.(4). The wo ol ages could be ei he in e nally ixed in applica ions whe e lea ning is no an esen ial ea- u e, o could also be adjus ed by addi ional ha d- wa e i lea ning is o be included in he conside ed applica ion. The o e all design consis s on: 180 I Fig. 4: Basic cell. Fig. 5: Oscilla o ’s layou . 1. An oscilla o whose VI and Vz ol ages can be ex e nally ixed. I pe mi s o gene a e pulse s eams in a wide ange o densi ies. The size o his oscilla o is 129 x 80 mic ons. The size o he lip- lop is 180 x 100mic ons. The oscilla o ’s layou can be seen in Fig.(5). 2. Th ee oscilla o whose VI and V, a e ixed by a Mos ansis o ol age di ide . 3. Eigh equal1 oscilla o s ha e been included in his design. In hese oscilla o s VI ol age is ixed o 5V, yielding a 2.5ns To, ime. V2 ol - ages a e conec ed o eigh analog pads so ha he eigh To imes can be ixed ex e nally. All oscilla o cells ha e gua d ings in o de o p e en , o , a leas , minimize, coupling be ween he di e en ci cui s. The ci cui has been designed using ES2 1.5pm echnology and he so wa e pack- age was MAGIC. The size o he whole ci cui , in- cluding he sampling lip- lops, is 743 x 736 mic ons. 4. Expe imen al Resul s. Fig. (6) shows he exis ing ela ionship be ween he con ol ol age Vz and he a io. VI has been ixed o OV. No ice ha akes only alues anging om 0 o 0.5 because 0.5 o 1 a ios can be ob ained by means o an in e e ga e. In o de o ind ou whe he spa ial p oximi y o he basic cells is ela ed o high c oss-co ela ion I 1 .o 2.0 3.0 4.0 5.0 0.00 1 0.0 Con ol ol age ( ) Fig. 6: egula ion plo . numbe s, we ha e calcula ed he c oss-co ela ion be ween he sampled s eam o pulses o one o he basic cells and he es o hem. The ob ained alues a e shown in Tab.(4). These calcula ions ha e been ca ied ou h ee imes (columns a, b and c). The sample equency is 100kHz, he pulse sequence leng h is 1000 and he a io is 0.5. I can be seen ha s ochas ically independen pulse sequences can be ob ained p o ided ha he basic cells a e no placed oo closed om each o he . The whole layou is shown in Fig.(7). The ’0’ cell is placed in he uppe -le co ne o he layou , ’l’, ’2’ and ’3’ cells a e placed below. The ’4’, ’5’’ ’6’ and ’7’ cells a e placed a he igh side o hem. An inc ease o he sampling equency does no lead o highe c oss-co ela ion alues. Howe e he ime- co ela ion numbe s inc ease as he swi ching e- quency becomes highe . In Fig. (8) i is shown he ime-co ela ion o pulse sequences sampled om he ’0’ and ’5’ basic cells. 0-6 0- 7 ~~~ -0.605882 I -0.424706 I -0.4082351 I I J -0.010588 0.010588 -0.005882 Table: 1: C oss-co ela ion esul s ( sample = 100kHz). s ochas ic applica- ions ime-unco ela ion is no an essen ial ea u e (i wo s eams a e o be mul iplied by means o an In many 181 0 5 1015202530354045Y) o.1L ' ' ' ' ' ' ' ' I Fig. 8: Time-co ela ion esul s ( sample = 100kHz). 5. Conclusions and Fu he Wo k In his pape we ha e p esen ed a new echnique o gene a e s ochas ic pulse sequences ha is sui able o VLSI implemen a ion pu poses. The basic cell consis s on a high equency oscilla o and a sam- pling lip- lop. The consumed silicon a ea is e y small, specially i i is compa ed wi h he s ic ly digi al app oach ha has been conside ed up o now. The spa ial co ela ion alues o di e en pulse sequences a e qui e sa is ac o y. Howe e he ime-co ela ion beha io should be imp o ed. In o de o achie e his goal, we a e going o include a noise sou ce in he ing oscilla o and use highe scales o in eg a ion which will lead o smalle inpu capaci o s. Re e ences ..- , . . . . . . , . . . . . . , . , . . . . . Fig. 7: Ci cui layou AND ga e only spa ial co ela ion is needed). In such cases his echnique may be used leading o space-e icien implemen a ions. We a e cu en ly imp o ing he empo al s a is ical beha io o he pulse s eams. ~ 182 [I] B.R. Gaines. S ochas ic Compu ing Sys ems. Ad ances in In o ma ion Sys ems Science, [2] Y. Kondo and Y. Sawada. Func ional Abili ies o a S ochas ic Logic Neu al Ne wo ks IEEE T ans. on Neu al Ne wo ks, o1.3, pp.434-443, 1992. [3] D.E. Van den Bou and T.K. Mille 111. A Dig- i al A chi ec u e Employing S ocha icism o he Simula ion o Hop ield Neu al Ne s. IEEE T ans. on Ci cui and Sys ems, ~01.36, pp. 732- 738. 1989 [4] W. Wike, D.E. Van den Bou and T.K. Mille I11 The VLSI Implemen a ion o STONN. IEEE In . Join Con . on Neu al Ne wo ks, [5] J.M. Que o, C.L. Jane and L.G. F an- quelo. Cons ained Hop ield Neu al Ne wo k o Real-Time P edic i e Con ol. P oc. o he 1994 IEEE In . Con . on Indus ial Elec on- acs, Con ol and Ins umen a ion, Iecon'94. Bologna, Sep . 1994. [6] Alan F. Mu ay, Dan e Del Co so and Lionel Ta assenko. Pulse-S eam VLSI Neu al Ne - wo ks Mixing Analog and Digi al Techniques IEEE T ans. on Neu al Ne wo ks, ~01.2910.2, [7] C.L. Jane , J.M. Que o and L.G. F anquelo. Fully Pa allel Summa ion in a New S ochas ic Neu al Ne wo k A chi ec u e. IEEE In . Con . on Neu al Ne wo ks, San F ancisco, pp. 1498- 1503. 1993 [8] C.L. Jane , J.M. Que o, J. Rios, J.G. O ega and L.G. F anquelo. Design C i e ia o Fully Pa allel S ochas ic Neu al Ne wo k. ICECS'94 El Cai o (Egyp ), Dec., 1994. [9] J.M. Que o, J.G. O ega, C.L. Jane and L.G. F anquelo. VLSI Implemen a ion o a ully pa allel s ochas ic Neu al Ne wo k. IEEE In . Con . on Neu al Ne wo ks, O lando (USA), July, 1994. [lo] J.G. O ega, J.M.Que o, C.L. Jane and L.G .F anquelo. In e aces o S ochas ic Logic: Applica ion o S ochas ic Neu al Ne wo k. ICECS'94 El Caa o (Egyp ), Dec., 1994. ~01.2, pp. 37-172. 1969 01.2, pp.593-598, 1990. pp.193-204, 1991.