i
VLSI
Implemen a ion
o
a
Fully Pa allel
S ochas ic Neu al Ne wo k
J.M.Que o, J.G.O ega, C.L.Jane and L.G.F anquelo
IEEE
h
EMBER
Dp o.
de Ingenie ia de Sis emas
y
Au omii ica
Uni . de
Se illa,
Spain
Abs ac -
In his pape we p esen
a
pu ely digi al s ochas ic implemen a ion o mul ilaye
neu al ne wo ks. We ha e de elopped his implemen a ion using
an
a chi ec u e ha pe mi s
he addi ion o
a
e y la ge numbe o synap ic connec ions, p o ided ha he neu on’s ans e
unc ion is he ha d limi ing unc ion. The exp ession ha ela es he design pa ame e , ha is,
he maximun pulse densi y, wi h he accu acy
o
he ope a ions has been used
as
design c i e ium.
The esul ing ci cui is easily con igu able and expandable.
I.
INTR.ODUCTION
S ochas ic neu al ne wo ks a chi ec mii es ha e ecen ly ecei ed much a en ion
[1],[2].
The use o
hese a chi ec u es dec eases he amoun o ha dwa e needed o ea?ize he ope a ions in ol ed in
neu ons. I we
look
a
he ans e unc ion o
a
neu on (3;
=
a(Cj;uijyj
Ii))
we ealize
ha
we ha e o e alua e
a
se o a i hme ic ope a ion, mo e p ecisely,
a
se
o
p oduc s and addi ions.
In o de o make e icien elec onic implemen a ions o neu al ne wo ks con aining
a
high numbe
o
neu ons,
i
woiild be desi a.ble ha, hese ope a ions could be pe o med by simple ci cui y.
S ochas ic logic sys ems ealize pseudoanalog ope a ions using s ochas ically coded pulse
sequences. These pulse sequences ca,n be gene a ed using
a
g oup o digi al compa a o s.
A
digi al
codi ica ions o a iables a e compa ed wi h unco ela ed andom numbe s p oducing unco ela ed
s ochas ic signals whose alues andomly ake alues
0
o
1.
The a e age o hese alues can be
<
1,
wi h Rmax
iewed
as
an analog alue in he ange
[-a,a]
o
[O,o],
whe e
a
=
andomma=
-
he maximum alue ha ca,n be s o ed in
R;
and
~undom~,,
is he maximum andom numbe
gene a ed.
Rmax
Mul iplica ion
o
wo s ochx ic pulse sequences should p oduce ano he s ochas ic s eam o
pulses whose i ing p obabili
is
he p oduc
o
he inpu i ing p obabili ies. This can be easily
achie ed i he inpu sequences a e s ochas ically independen . The ci cui ha implemen s his
ope a ion is
a
simple
AND
ga e.
0-7803-1901-x194
$4.00
01994
IEEE
2040
S ochas ic summa ion is
a
much mo e di icul ope a ion o pe o m, specially i he e ms
o be added a e signed. Two ypes o ci cui s ha e been desc ibed in he bibliog aphy. One
is
he
OR ga e and he o he is he up-down coun e .
The up-down coun e s echnique, al hough is widely used when implemen ing ’Hop ield’s
neu al ne wo k, has
a
e y impo an d awback. Pulses coming om o he neu ons ha e
o
be
mul iplexed in ime (i.e.
se zdized)
leading o
high
compu a ion,
i”
i he ne wo k has many
neu ons and many connec ions pe neu on. I should be also poin ed ou ha he se ializa ion ha
akes place is only e icien in neu d ne wo ks whose neu ons a e ully connec ed .
I wo
pulse
sequences
am e
ed o an OR ga e he ou pu i ing p obabili y would be equd
o he sum o bo h i ing p obabili ies
i
he pulse sequences o be added did no o e lap. This
OR-based add unc ion is hus dis o ed by pulse o e lap. I only posi i e e ms a e o be added
pulse o e lap is no
a
d awback because
i
p oduces he non linea ans e unc ion o
a
neu on.
I signed inpu s a e o be added linea beha io is an essen ial ea u e. Howe e his app oach is
ex emely in e eshing due o he simplici y o he equi ed ha dwa e which would pe mi he digi al
implemen a ion
o
a bi a y neu a.1 ne wo ks con aining
a
e y high numbe o neu ons.
11.
PROPOSED
STOCHASTIC: ARCHITECTURE
I has been p o ed ha he
OR.
ga e based summa ion is ex emely e icien i we es ic
ou sel es o neu al ne wo ks in which neu ons ake only wo disc e e alues. Due
o
he ac ha
he exponen ial Iiinc, ion is iiioiio oiiicl lly inc easing, acid akin in o accoun i s p ope ies, i can
be deduced ha
sign(Cizy
xi)
=
sign(niz;”-
e-”:
-
l-Iizy
e-“i
),
whe e supe sc ip s
+
and
-
a e
ex ended o posi i e and nega i e e ms espec i ely.
5
.
The las exp ession can
be
ega. ded
as
he compa. ison
o
wo pulse s eams gene a ed by wo
s ochas ic mul iplie s, he e o e no adde is needed. The only p oblem is
o
e alua e he exponen ial
ans o ma ion
in
a.
e icien
way.
I
he neu al ne wo k has been adimen ionalized in such
a
way ha all e ms o be agg ega ed
ake alues anging om ze o o
a
small numbe
a
close enough o ze o,
e-”%
can be app oxima ed
by
1
-
2;.
The esul ing s uc u e is shown in Fig.1. Inpu pulses a e in e ed by no ga es and hen
ed o he co esponding
AND
ga es
by
he i s se
o
logic blocks. I should be poin ed ou ha
hese logic blocks should main ain i s ou pu signal a high le el while no being d i en by he
inpu pulses. The ou pu logic Mock e alua es he neu on ou pu .
I
only he “posi i e” pulse is
a
high le el he coun e is inc emeii ed by one and i he pulse is “nega i e” he coun e is hen
deuemen ed
by
one. The sign bi is chmged i
a
ze o c ossing akes place. I
a
second o de
app oxima. ion
o
he exponen id , a.ns o ma. ,ion was made, we could la gely inc ease he alue o
U
a he expenses o ci cui ’s complexi y (see Fig.
2).
Due o he ac ha, he exponen ial ans o ma ion is no e alua ed exac ly, i ollows ha
he addi ion opem ion has
a.
limi ed accu acy.
I
he o d neu on’s exci a o y inpu and he o al
2041
i
‘I
neu on’s inhibi o y inpu a e e y closed numbe s, he neu on’s ou pu may be unco ec .
I
has
been p o ed ha he accu acy o he addi ion is bounded
by
he ollowing exp ession:
whe e
2
is he a e e o
o
he addi ion ope a ion.
I
he mos accu a e compa ison ha has
o be e alua ed is a known quan i y, i ollows ha
a
has o be chosen
so
ha
<
w,
whe e
DIFF
is he di e ence be ween he o al posi i e neu on’s inpu
and
he o al nega i e neu on’s
inpu
.
I
he ne inpu s can ake any alues, he limi ed accu acy
o
he addi ion de e mines
a
egion
whe e he neu on’s ou pu will be unce ain.
111.
IMPLEMENTATION
In his sec ion se e al design issues, such us andom numbe gene a ion, ne wo k con igu a ion,
ne wo k expansion a.nd ha dwa e implemen a ion,
a e
desc ibed.
a) R.a.ndom numbe gene a. ion
Linea Feedback Shi R.egis e (LFSR) is one o he mos s udied digi al echniques o gen-
e a e pseudo- andom numbe s
[4].
Howe e , all he esul s gi en in p e ious sec ions a e alid i
pulse secuences a e pu ely andom -specially hose in ol ed in he’same neu on. In ou design we
ha e used
a
9-bi
LFSR.
wi h
do
=
XOR(q0,
qd).
We
can ob ain he c oss-co ela ion
(C )
be ween
shi ed sequences
n(k)
and
n(k
-
p),
whe e
n,(k)
s ands o he o iginal no malized sequence and
n(k
+
p)
is i s p-s eep ahead shi sequence.
Fig
3
ep esen s he c oss-co ela ion be ween shi ed
mdom numbe sequences. I
is
clea ha we need
a,
8-s ep shi a leas o gene a e pulse se-
quences as unco ela ed
as
possible. These shi s ha e been macle using 8-XOR ga es. The numbe
o unco ela ed pseudo- andom sequences ha can
be
simul aneously gene a ed
is
2’/8
N
64.
This
numbe can
be
inc eased using
a
la ge
LFSR.
b) Ne wo k con igu a. ion
We
ha e iised pipeliiicd pa allel egis e s o allow neu al ne wo k p og ammabili y wi h a
minimun numbe o inpu s. The size o hese egis e s is ela ed o he size o he LFSR and
he maximum ole a ed e o in he exponen ial app oxima ion. In his case we ha e choosen
a
7-bi egis e o weigh s o age. I
we
conside inpu sequences o he ne wo k wi h a maximun
p obabili y o
0.5,
he esul ing niul ipli a ioii has a mean
o
$0.5
=
0.125.
Acco ding o Fig.
4,
i leads o an exponen ial app oxima, ion e o o
6%
(linea ampp oxima ion
o
he exponen ial
ans o ma ion).
c) Ne wo k expansion
One o he mos ema, kable ea u es
o
he
p oposed
aachi ec u e is ha he numbe o
2042
connec ions can be easily inc eased. Addi ions a e caIcula, ed like p oduc s,
so
ha se s o weigh s
can be added using
!P+
and
q-
signals wi hou addi iona.1 ha dwa e (see Fig.
1).
The numbe o
hidden laye s ca,n
also
he inc eased wi h addi ional ci cui s in cascade.
(1)
Hwdwa e implemen a ion
The p oposed s ochas ic a chi ec u e has been implemen ed using
EDGE
wi h
ES2
1.5~
S anda Cell lib a ies. The esul ing ci cui con ains wo laye s wi h i e inpu unsigned pulse
signals, i e hidden neu ons and one ou pu neu on. Ou pu neu on
Q+
and
Q-
signals a e
accessible,
so
ha. he numbe o hidden neu ons ma,y be inc eased.
IV.
RESULTS
In o de o es he beha iou o he
I.C.,
we ha e de elopped he con olled p esen ed in
[5].
This applica ion consis s on
a
pe cep on ha app oxima es
a
classi ica ion su ace desc ibed by
a
196-poin
a ay.
.4
wo laye pe cep on wi h
10
hidden neu ons ha e been con igu ed using Back
p opaga ion. Two
1.C.s
ha e been used o implemen his applica, ion.
All aining inpu pa e ns a e co ec ly classi ied
by
he ne wo k. The ansien esponse
o
he ou pu neu on o he inpu ec o (0.0867,0.42,0.39) is shown in Fig.
5,
whe e he clock a e
is
7.5MHz.
Fig.
6
shows ,he beha io
o
he ne wo k
as
he inpu ec o c osses he decision su ace. In
his igu e he desi ed ne wo k’s esponse and he ac ual ne wo k’s esponse ha e been ep esen ed.
Se e al easons explain he sligh , di e ences be ween he wo g a,phics: he disc e iza ion o he
synap ic weigh s, he limi ed ac,cu a.c.y
o
he s ochas ic a,ddi ions and he use
o
pseudo andom
sequences ins ead o eal andom numbe s.
V.
CONCLUSIONS
In his pa.pe we ha e p esen ed
a
ha. wa, e implemen a ion o
a.
mul ilaye neu al ne wo k
using he s ochas ic a. chi ec u e ha
was
p oposed
in
[3].
This a chi ec u e can be easily expanded,
and he ci cui has been designed
so
ha, any niul ila,ye ne wo k can be implemen ed by adding an
app op ia e numbe
o
1.C.s.
As a.n example we ha e implemen ed he con olle wi h pe cep on-
like s uc u e which was p esen ed
in
[5].
The esponse
o
he ne wo k ma ches wi h
a
high deg ee
o
accu acy he heo e ical con olle ’s esponse.
Re e ences
[l]
Y.
Kon lo and
Y.
Sawa. la. Func ional Ahili ies o
a
S ochas ic Logic Neu al Ne wo ks
IEEE
T ans.
on
Nein~nl
Ne wo ks,
o1.3, pp.434-443, 1992.
2043
[2]
D.E.
Van den Bou and
T.K.
Mille
111.
A
Digi al A chi ec u e
Employing
S ochas icism
o
he Simula ion
o
Hop ield Neu al Ne s.
IEEE T ans.
on
Ci cui
and
Sys ems,
o1.36, pp.
732-738. 1989.
[3]
C.L.
Jane ,
J.M.
Que o and
L.G.
F anquelo. Fully Pa allel Summa ion
in
a
New S ochas ic
Neu al Ne wo k A chi ec u e.
IEEE
In .
Con .
on
Neu al Ne wo ks,
San
F ancisco,
pp.
1498-
1503.
1993.
[4]
W.
Pe e son.
E o
Co ec ing Codes.
MIT
P ess.
1992.
[5]
.J.M. Que o, J.M.
Ca. a.sco
and
L.C. F a.nqnelo. Ada.p a i e Ene gy
Feed-Back
Con ol
o
Resonan Con e e s Using Neu al
Ne wo ks.
23 d Annual Powe Elec onics Specialis s
Con-
e ence,
Toledo,
Spain,
pp.
800-806.
1992.
Mpxiniun
puke
mi y:
a
(a<cl
Figu e
1:
Neu on's s uc u e
Figu e
2: E alua. ion c.i cui
o
he second o de app oxima ion
o
he exponen ial ans o ma ion.
2044

0.045
0.0.
0.035
0.03
Y.UI5
0.0)
0.015
0.01
0.005
0
-0
005
1
1-
Figu e
3:
C oss-co ela ion be ween p-s ep
ahead shi pseudo- andom numbe sequences
gene a ed
by
a
9-bi
LFSR)
01
0.05
I
-O."!
Figu e
4:
No malized exponen ial a.ns o ma.-
ioii e o , o he i s ,
second
and
hi d
o dc
app oxima ion.
DO
Figu e
5:
T ansien esponse
o
he ou pu neu-
on
o
he inpu ec o (0.0867,0.42,0.39)
Figu e
6:
Ne wo k's desi ed and ac ual esponse
as
he inpu ec o c osses he decision su ace.
2045