Synap ic Weigh Gene a ion in
VLSI
S ochas ic Neu al
Ne wo ks.
J.G.
O ega,
J.M.
Que1.0,
C.L.
Jane and
L.G.
F anquelo
Escuela Supe io de Ingenie os
A da. F eina Me cedes s/n
Se illa
41012
SPAIN
.ABSTRACT
Fully pa allel s ochas ic neu al ne wo k implemen a ions can be ealized nowdays. How-
e e , in hese implemen a ions mos o he silicon a ea is consumed in he s ochas ic pulse
sequence gene a ion ci cui s. In o de o imp o e hei e iciency in e ms o consumed silicon
a ea, new echniques mus be de elopped. This is specially impo an in applica ions whe e
a la ge numbe o synap ic weigh s a e needed. In his pape we p esen a new app oach ha
can signi ican ly inc ease he e iciency
os
he
1.
In oduc ion
S ochas ic logic sys ems ealize pseudoanalog ope -
a ions using s ochas ically coded pulse sequences,
[l],
[2].
In s ochas ic sys ems, he e ms ha a e
o be p ocessed a e synch onous pulse sequences.
In o ma ion is codi ied
as
he p obabili y, a
a
gi en clock cycle,
o
he pulse aking "high" alue.
S ochas ic pulse sequences a e gene a ed in such a
way ha all pulse s eams a e s ochas ically inde-
penden .
Conside now a se
o
n pulse s eams whose
p obabili ies,
a
a gi en clock cycle, o being a
"high" le el a e
pl,
p2,
...,
p,.
These p obabili ies
a e mu ually independen . I hese sequences a e
he inpu s
o
a n-inpu
AND
ga e, he p obabili y
o he ga e ou pu , a a gi en clock cycle,
o
be-
ing a "high" le el
is
equal o
n:::
pi.
I is clea
ha he p oduc ope a ion
is
achie ed by means o
simple
AND
ga es, ha is, by a ex emely low a ea
consuming ci cui .
S ochas ic summa ion
is
a
much mo e di icul
ope a ion o pe o m, specially i he e ms o be
added a e signed. Two ypes o ci cui s ha e been
desc ibed in he bibliog aphy. One is he
OR
ga e
and he o he is he up-down coun e .
The up-down coun e echnique, al houglh is
widely used in neu al ne wo k implemen a ion,
[3]-
[5],
has a e y impo an d awback. Pulses coming
om o he neu ons ha e o be mul iplexed in ime
(i.e.
se ialized)
leading o
high compu a ion
imes.
I wo pulse sequences a e he inpu s o an
OR
ga e and he pulse sequences o be added do no
o e lap, he ou pu i ing p obabili y is equal o he
addi ion o bo h i ing p obabili ies. This OR-based
add unc ion is hus dis o ed by pulse o e lap. In
echnique ha
has
been used up o now.
,._.__________.._.._..........
....
"
Fig.
1:
S ochas ic a chi ec u e.
o de o achie e
a
guasy linea beha iou pulse den-
si ies should s ay e y low, specially i many e ms
a e o be added. This echnique does no pe mi he
in eg a ion
o
neu ons wi h
a
e y high numbe o
synap ic connec ions
as
i would lead o ex emely
low maximum pulse densi y,
[6].
I should be aken
in o accoun ha he addi ion o wo numbe s ha
ake alues anging om
0
o
1
may ake
a
alue
bigge han one, which can no be ep esen ed by a
p obabili y.
In p e ious pape s we ha e p oposed a ully pa -
allel s ochas ic compu a ion a chi ec u e sui able
o neu al ne wo k implemen a ion,
[7],
[8].
I ci -
cum en s one o he main d awbacks
o
s ochas ic
compu a ion a chi ec u es ha ha e been used up
o now: he absence o
a
space-e icien echnique o
adding weigh ed inpu signals in pa allel. Howe e
s ill emains an impo an p oblem o be sol ed: o
ind
a
simple ci cui ha gene a es he s ochas ic
signals.
0-7803-2768-3/95/$4.00
0
1995
IEEE
179
2.
S ochas ic Pulse Gene a ion
A
ha dwa e implemen a ion
o
a
mul ilaye neu al
ne wo k based on his s ochas ic a chi ec u e was
p esen ed in
a
p e ious pape ,
[9].
This pu ely
digi al a chi ec u e
is
expandable, and he ci cui
was designed
so
ha any mul ilaye ne wo k could
be implemen ed by adding an app op ia e numbe
o
1.C.s.
Howe e
mos
o
he silicon a ea
o
his implemen a ion is consumed in ci cui s
in ol ed in s ochas ic pulse gene a ion
o
he
synap ic weigh s
(block
M
o Fig.(l)).
In his implemen a ion, s ochas ic s eams o
pulses a e gene a ed using a well known echnique
ha has been b oadly used and desc ibed in he
echnical li e a u e: Digi al codi ica ions o hese
weigh s a e digi ally s o ed and hen compa ed
wi h unco ela ed andom numbe s p oducing un-
co ela ed s ochas ic signals,
[2]-[5].
The e o e load
egis e s, digi al compa a o s and a pseudo andom
gene a o a e necessa y.
Fig.
2:
Pulse sequence co ela ion.
To imp o e space e iciency
we
p oposed a new
echnique o p oduce he andom pulse sequences
ha codi y he synap ic weigh s,
[lo].
The basic
s ochas ic cell o his echnique is a high equency
oscilla o whose
T~
a e can be con olled, and a
lowe equency sampling ci cui . The
%
a e is
ixed equal o he synap ic weigh ha is mean
o
be s ochas ically codi ied.
I he oscilla o ’s
in-
pu capaci o has a small alue, he oscilla o
will be e y sensi i e o noise.
Consequen ly,
he e is a ce ain deg ee o unce ain y abou he
ol age a oscila o ’s inpu , p oducing ime unce -
ain y abou he momen in which he oscila o
swi ches om ei he he on s a e o he
o
s a e
o
om he
o
s a e o he on s a e. I ollows
ha oscilla o ’s phase canno be p edic ed a e i
ai1
2.52
oc
n
T/dl
2ue
Fig.
3:
Phase noise.
has swi ched se e al imes. In Fig.(2) i is shown
he au oco ela ion o he pulse sequence,
(A),
as
a uc ion o he noise in ensi y and he oscilla ion-
sampling equencies a io ( / s). This igu e has
been ob ained by simula ion conside ing ha he
noise is whi e and gaussian. I he oscila o ’s ou pu
is sampled wi h
a
lip lop
a
a
slow enough a e,
he sampled signal will be andom. The p obabili y
o his signal aking he ’’high’’ le el will be equal
In o de o p oduce comple e spa ial (be ween
di e en pulse sequences) and ime (in each pulse
sequence) andomness, phase unce ain y mus be
g ea e
o
equal o
27 .
Following hese ideas, we
designed es boa d using
disc e e componen s.
Na u ally he maximum oscilla ing equency was
a he modes , bu he measu ed c oss-co ela ion
be ween di e en sampled oscilla o s and he au o-
co ela ion we e e y encou aging. In Fig.(3) we
show how noise acumula es p oducing phase unce -
ain y. We decided o ace he
VLSI
implemen a-
ion
o he p oposed ci cui
as
we in end o de elop
applica ions including
a
la ge numbe o neu ons
and synap ic weigh s pe neu on.
o
+.
3.
VLSI
Implemen a ion
The oscilla o consis s on i e consecu i e C-mos in-
e e s. The ou pu o he i h in e e is conec ed
o he i s in e e ’s inpu , leading o an uns able
ci cui ha oscila es a a high equency.
VI
and
V,
a e ol age signals ha a e used o ix he
To,
and
To
alues.
The
ou pu
is
conec ed o a lip-
lop ha samples i . The lip- lop clock signal
is
suplied ia an inpu digi al pad. The whole ci cui
is plo ed in Fig.(4).
The wo ol ages could be ei he in e nally ixed
in applica ions whe e lea ning is no an esen ial ea-
u e,
o
could also be adjus ed by addi ional ha d-
wa e i lea ning is o be included in he conside ed
applica ion.
The o e all design consis s on:
180
I
Fig.
4:
Basic cell.
Fig.
5:
Oscilla o ’s layou .
1. An oscilla o whose
VI
and
Vz
ol ages can be
ex e nally ixed. I pe mi s o gene a e pulse
s eams in
a
wide ange o densi ies. The size
o his oscilla o is 129
x
80 mic ons. The
size o he lip- lop is 180
x
100mic ons. The
oscilla o ’s layou can be seen in Fig.(5).
2. Th ee oscilla o whose
VI
and
V,
a e ixed by
a
Mos
ansis o ol age di ide .
3. Eigh equal1 oscilla o s ha e been included in
his design. In hese oscilla o s
VI
ol age is
ixed o 5V, yielding a 2.5ns
To,
ime.
V2
ol -
ages a e conec ed
o
eigh analog pads
so
ha
he eigh
To
imes can be ixed ex e nally.
All oscilla o cells ha e gua d ings in o de o
p e en ,
o ,
a leas , minimize, coupling be ween
he di e en ci cui s. The ci cui has been designed
using ES2 1.5pm echnology and he so wa e pack-
age was MAGIC. The size o he whole ci cui , in-
cluding he sampling lip- lops, is 743
x
736 mic ons.
4.
Expe imen al Resul s.
Fig.
(6)
shows he exis ing ela ionship be ween
he con ol ol age
Vz
and he a io.
VI
has
been ixed
o
OV. No ice ha akes only alues
anging om
0
o 0.5 because 0.5 o
1
a ios can be
ob ained
by
means
o
an in e e ga e.
In o de o ind ou whe he spa ial p oximi y
o
he basic cells is ela ed o high c oss-co ela ion
I
1
.o
2.0
3.0
4.0
5.0
0.00
1
0.0
Con ol
ol age
( )
Fig.
6:
egula ion plo .
numbe s, we ha e calcula ed he c oss-co ela ion
be ween he sampled s eam o pulses o one o he
basic cells and he es
o
hem. The ob ained
alues a e shown in Tab.(4). These calcula ions
ha e been ca ied ou h ee imes (columns a, b
and c). The sample equency is 100kHz, he pulse
sequence leng h is
1000
and he a io is 0.5. I
can be seen ha s ochas ically independen pulse
sequences can be ob ained p o ided ha he basic
cells a e no placed oo closed om each o he . The
whole layou is shown in Fig.(7). The
’0’
cell is
placed in he uppe -le co ne o he layou ,
’l’,
’2’
and ’3’ cells a e placed below. The ’4’, ’5’’ ’6’ and
’7’ cells a e placed a he igh side
o
hem. An
inc ease o he sampling equency does no lead o
highe c oss-co ela ion alues. Howe e he ime-
co ela ion numbe s inc ease
as
he swi ching e-
quency becomes highe . In Fig.
(8)
i is shown he
ime-co ela ion o pulse sequences sampled om
he
’0’
and ’5’ basic cells.
0-6
0-
7
~~~
-0.605882
I
-0.424706
I
-0.4082351
I
I
J
-0.010588 0.010588 -0.005882
Table:
1:
C oss-co ela ion esul s
( sample
=
100kHz).
s ochas ic applica-
ions ime-unco ela ion is no an essen ial ea u e
(i wo s eams a e o be mul iplied
by
means o an
In
many
181
0
5
1015202530354045Y)
o.1L
'
'
'
'
'
'
'
'
I
Fig.
8:
Time-co ela ion
esul s
( sample
=
100kHz).
5.
Conclusions and Fu he
Wo k
In his pape we ha e p esen ed a new echnique o
gene a e s ochas ic pulse sequences ha is sui able
o
VLSI implemen a ion pu poses. The basic cell
consis s on a high equency oscilla o and a sam-
pling lip- lop. The consumed silicon a ea is e y
small, specially i i is compa ed wi h he s ic ly
digi al app oach ha has been conside ed up o
now.
The spa ial co ela ion alues o di e en
pulse sequences a e qui e sa is ac o y. Howe e he
ime-co ela ion beha io should be imp o ed. In
o de o achie e his goal, we a e going o include
a
noise sou ce in he ing oscilla o and use highe
scales
o
in eg a ion which will lead o smalle inpu
capaci o s.
Re e ences
..-
,
. . . . . .
,
. . .
.
. .
,
.
,
. . . . .
Fig.
7:
Ci cui layou
AND ga e only spa ial co ela ion is needed). In
such cases his echnique may be used leading o
space-e icien implemen a ions. We a e cu en ly
imp o ing he empo al s a is ical beha io o he
pulse s eams.
~
182
[I]
B.R.
Gaines. S ochas ic Compu ing Sys ems.
Ad ances in In o ma ion Sys ems Science,
[2]
Y.
Kondo and
Y.
Sawada. Func ional Abili ies
o
a
S ochas ic Logic Neu al Ne wo ks
IEEE
T ans. on Neu al Ne wo ks,
o1.3, pp.434-443,
1992.
[3] D.E. Van den Bou and T.K. Mille 111. A Dig-
i al A chi ec u e Employing S ocha icism
o
he Simula ion
o
Hop ield Neu al Ne s.
IEEE
T ans. on Ci cui and Sys ems,
~01.36, pp. 732-
738. 1989
[4] W. Wike, D.E. Van den
Bou
and T.K. Mille
I11 The VLSI Implemen a ion
o
STONN.
IEEE In . Join Con . on Neu al Ne wo ks,
[5]
J.M. Que o, C.L. Jane and L.G. F an-
quelo. Cons ained Hop ield Neu al Ne wo k
o
Real-Time P edic i e Con ol.
P oc.
o
he
1994
IEEE In . Con . on Indus ial Elec on-
acs,
Con ol and Ins umen a ion,
Iecon'94.
Bologna, Sep . 1994.
[6]
Alan
F.
Mu ay, Dan e Del Co so and Lionel
Ta assenko. Pulse-S eam VLSI Neu al Ne -
wo ks Mixing Analog and Digi al Techniques
IEEE T ans. on Neu al Ne wo ks,
~01.2910.2,
[7] C.L. Jane , J.M. Que o and L.G. F anquelo.
Fully Pa allel Summa ion in
a
New S ochas ic
Neu al Ne wo k A chi ec u e.
IEEE In . Con .
on
Neu al Ne wo ks,
San F ancisco, pp. 1498-
1503. 1993
[8] C.L. Jane , J.M. Que o, J.
Rios,
J.G. O ega
and L.G. F anquelo. Design C i e ia o Fully
Pa allel S ochas ic Neu al Ne wo k.
ICECS'94
El Cai o (Egyp ),
Dec., 1994.
[9] J.M. Que o, J.G. O ega, C.L. Jane and L.G.
F anquelo.
VLSI
Implemen a ion
o
a ully
pa allel s ochas ic Neu al Ne wo k.
IEEE In .
Con . on Neu al Ne wo ks,
O lando (USA),
July, 1994.
[lo]
J.G. O ega, J.M.Que o,
C.L.
Jane and L.G
.F anquelo. In e aces o S ochas ic Logic:
Applica ion
o
S ochas ic
Neu al Ne wo k.
ICECS'94 El
Caa o
(Egyp ),
Dec., 1994.
~01.2, pp. 37-172. 1969
01.2, pp.593-598, 1990.
pp.193-204, 1991.