An effective way to generate neural network structures for function approximation

Math w areandSoftC om puting 1( 1994)139-161 AnE ec ti ve Wa ytoG ener ateN eu ral N et w or kS tru cturesforFunction A ppro xim ati on A ndreas Ba stian Lab.forIn t. F uzz yE nginee ri ngRes earc h Sib erHe gnerBl dg.4.Fl ., 89-1 Yam ashita-c ho, N ak a -ku,Y ok oham a,231JAPAN Submitte dbyL. T.K o czy Abs tr act Ones tillopenquestioninthear ea ofr es ea rc hofm ulti-la y er feed fo rw a rdneura lnet w orksis concer ningthen um berofneuronsinit sh iddenla y er (s).Especia llyi nreallifeapplicat ions, thisproblemiso ftensolv edb yheuristicmethods.Int hisw or k aneective wa yt od yna micallydetermine then um berofhidden unitsinathreela y erfeedfo rw ar dneuraln et w o rkforf unction appro ximationisp ropo sed. 1In trod uction Mult i-la yer fe edforw ard ne ur al ne tworks are e nj oying a rising p opularit yandt hei r appli ca ti on s ra nge f ro m medical a ppl ications [ 1] ,ha ndwriti ng re cog ni tion [2], the g am eofbac kga m mon [3], curre nc ye xchan ge rate predi cti on [ 4] , con trol and ide ntic ati o n of nonli near dyna m ical ly system s [5], to pig carcas e g rading i nsl augh terhouses [ 6]. De spi te thi sob vi ou s succe s s, one freq uen task ed fundam en ta l question is a b o ut the size of th e hidden layer(s). Esp eciallyinin dustrial applicatio ns where compa ct networks which require a minimal amount 139 140 A.B asti an of storagem em oryandcom puti ng tim eareneeded,theso-c al le d\ righ t siz e"ofahi ddenl ay er i sav e ryim portan tissue.I ft hene t w orkistoo sm all ,a go od re presen tationofthetraining datam i gh tnotbepossible .Ontheotherhand,ano ve rs ize dnet w ork,be si desrequiringl ong com putationa lt im e, m i gh tl ooseitsgene ra liza ti onabi li t y . It ha sb eensho wni n[ 7]|b yapplyin gt he Stone{W ei erstrassTh eo - rem |thatafee dforw ar dn euralne twi thonlyonehi dd en la y e rusing arbitrarysqua shingfunctionsi sc apableto appro xim atean yB ore lm e asurab le functionfromoneni ted im en si onspacetoanot he r.Th us,t his ki ndofnet w orkc anberega rde dasanuniv ersalapproxim ate.Ho w ev er, thisthe ore mdoes notpro v idesthen um be rofhiddenuni tsnecessary forappro xi mating afunction. Al thoughthereex istanumb erofapproac hestodetermine then um - berofhidde nunit s,e. g.[ 8-10] ,m ost of the mar etoocom pl ic ate dfor af astappl ication,orareonlysuitablefor so m espec iali zedappl ic ati on.F urthe rm ore, allapproac he sdonotdistinguishbet w ee nthetask ofpatternl earningandfunctionappro xim ati on.Y et,iti sw el l-kno wn thatt hoset w otask sarev e rydi eren t:patte rnle arni ngre quire sthe m e m oriz ingpo w erofth enet w ork,whil einf unc tiona ppro xim ati onthe generali zi ngabi li t yofanet w orki snee ded. Inthis w ork,w ewillm ai nlydealwith theproblemofndi ngthe optim alne tw orkstructureforf unc tion ap pro xi m ation.In[11-12]the boundsonthen um be ro fhiddenunitsinm ul ti-lay er fe edforw ardnetworksn ee de dtoappro xi m ateafuncti onwithan ydesiredacc urac y w e regiv en. I nthisstudy ,thosethe or eticalresultsarederi v e dto a sim pleandm orep ra cticalapproac htondtheo pt im aln um be rof hidde nuni ts.Itwil lb es ho wn,thatgiv enam odeling methodthat con vergestotheoutput,i ti sp os sibl etondthem i nimumn um be ro f hidde n units suc ie nttore presen t the o utput b ydy na m icall y a dding or de let ing hi dde nuni ts . The i ni tial s i ze o f a ne twork i sal so of major i mp ortance when d e termining its optim al st ructure. T h us , the usag e o f feature e xtraction of the data is pro p o sed to de term ine the initi al n um ber of hi dden units . For this purp ose, the w ell-kno wn F uzz y C-Means (F CM) cl ustering algo ri thm [21] is em pl oyed. Altho ugh a ny learning metho d applicable for feedforward neura l networks can be a pplied, in this study the backpro paga - AnEe ctiv eW a yt oG enerat eN euralNe t w ork. .. 14 1 tion-learningalgor ithm[ 13 ]i su se d,si nc ethi salgori thmi sb yno wquite w el lunderstooda st hereex istan um berofpublicationsc onc erningi ts le arningp ar am e ters[ 14-15] ,w ei ghts[ 16 ], im pro v edle arni ngalg orithm s [17],andopti m umshapeof thes ig m oi dfunction[18].Furtherm ore, thislearning al go ri thmistheprobabl ym osto ften applie done. Thi spa pe ri so rgani ze dasfoll o ws:t om ak et hisstudyself -c on t ai ne d, in thenextse ctionsom epreliminarydeni tionsareprovided,fol lo w e d b ythein troduc tion oftheproposedm ethodin Se ction3.T odem onstrat et hee ec tiv enessofthism etho d,i nSecti on4i tisappli ed to sev eralpoly nom i alfunctions andonereall ifeex am ple,n am elythe m ode li ngofah um anope ratorofac hem icalpl an t.Finall y ,c onc lusionsandoutlookaregi ve n. 2Prelim in arydiscuss ion 2.1Thebac kpropagati onal go ri thm Let'sconsideram ulti -la y ernet w orkofcon ti n uous-v al uedneurons.When apatt erni spresen tedtothenet w ork,thege neralacti v ati onofeac h neuroni nthenet w ork(exc epttheinputunitsw hoseacti v ati oni s cl am pedb ytheinputve ctor)i sgiv enb yusi ngasi gmo idalf unc tion suc htha t: o pj = 1 1+exp  0  P i w ij o pi +2 j  (1) where o pj i st he ac tiv ationoftheneuron j due to thepre se n tationof patte rn p ,and w ji isthew ei gh tf ro mn eu ro n i to ne uron j .2 j i sthe biasfortheneuron j . Withou t an ylossofge neralit y,i nthi sw or kwewil lusethebac kpropa gationl earning a l gorithm .This algorithm c an be considere dasa ge nerali za t ion of the l eastsquare metho d used for a ppro xi mating functions throu gh p o l ynom ial s . The net work learns to map a set o f inputs to a set of o utputs b yi terativ ely a djusting the w ei gh ts w of thenet work by: 1 w ji ( t +1) =  pj o pi +  1 w ji ( t )(2) where t ,  ,  pj ,  and w ji are the presentation number, the learning 142 A.B asti an rate,theerrorsi gnalforn euron j ,theacti v at ionofneuron i as a resul toftheinpu tp at t ern p ,them om en tumfactor andthew e igh t fromneuron i to neuron j ,re spec tiv el y. The errorsi gnalforanoutputn euroni sc al culatedasfoll ow s:  p j =  tar get pj 0 o p j  o pj (1 0 o pj )(3) where tar get i st hedesi redtar getv al ue. Th is erroristhenbac k propag atedtotheprevi ousla y era nd thee rrorsignalforahi ddenl ay er neuronisc al cul ateda sfoll o ws:  pj = o pj (1 0 o pj ) X k  pk w kj : (4) Al thoughi ti su nderstoodthatthesetti ng so fthelearningparam - ete rsi nthisw orkandshapeo fthesigm oidalfunc tionma yn otyie ld fast c on v ergenc e,theparam e ters ofthenet w orksarek eptconstan tto ensureafa i rc om pari sonbe t w ee nsev eralnet w orkstruc tures. 2.2Boundsonthen um berofhi dde nuni ts Inam ulti -l a y e rfee df orw ardneuralnet w orkev eryhi dden unitc onstructsah ype rplanede cisi onsurfacei ntheinputspace.In[12]i t hasbee nshow nthatan yarbi trarytrai ning setwi th p trai ni ngpatternsc anbem appedb yam ul ti-la y erfe edforw ar dne ural netw orkwith oneh iddenla y erand p 0 1hiddenu nits.Thefollo wi ngtheoremw as proofed: Theore m1[ 12] :L et S be a k -ele ment of E n wh er e k> 1 isan inte ger,and let f b eanar bi tr ar yr e al-val ue dfunctiond ene don S , i.e., f : S ! E .A nmulti -layer p er c ep tron w ith k 0 1 hidden un its is cap abl e of r ealizin g f . This t he ore mpro vide s the n um be r of hidden units n e eded for a correc tre presen tation o f the k elem ents. Yet, i ti sob vi ous t ha tthe num b er o f th e hidden units c a n b e reduce difsim ila r traini ng patterns are pre se nted to the ne twork. Therefore o ne m aysa ythatThe ore m1 gives us the sucientnumb er of neuron s needed fo r the representation of th e k elements. AnEe ctiv eW a yt oG enerat eN euralNe t w ork. .. 14 3 Co rol lary1 T hesucientnu mb er of hiddenunitso famulti-l ay er p er c eptr o nwitho ne hidd enlayerne e de dtorepr es enta k elem entnite set S is k 0 1 . Theproofofthis coro l laryfoll o wsdirec tlyf ro mT heorem1.Since Theo re m1 w asal readypro v eni n[ 12 ], thereaderisrefe rredto that public ati on. Them inim umn um be ro fhi dde nunitsneededtor epre se n ta k el - ement niteset S cannotbede ducedt heoreti call y ,si ncei tdep ends v e rym uc honthedataandthek indofappli cation.H o w ev er, am i ni - m umn um berofhi ddenuni tsexi stsforam ulti-l a y e rp erce pt ronwi th onehi ddenla y er,asi tw illb esho wninthefoll o wi ng. Le t X =( X 1 ;:::;X n )beav e ctor of n inputv ariable swi th x = ( x 1 ;:::;x n )be inganinstanc eo fthi sv ector,andl et Y betheo utput v ari abl ewi th y bei ngan instanceofit.Acc ordi ngtoT heorem1,a m ul ti-la yerperce ptr onwi thonehi ddenl a y er,consistingof k 0 1hidde n units,isablet or e presen t k instanc esoftheinputv ec tor.Onem a y sa ythat,giv enabounded subset Y oftheonedim ensiona lEucli dean sp aceandabounde dsubset X ofthen-dim ensionalEuc li deanspace, thereexistsafuncti on f : R n ! R wi th Y = f ( X ). If r unit sarerem o v edfrom thehi ddenla y er,thengi ve na bounde d subset Y oft heonedi m ensionalEucl idean spac eandaboundedsubse t X ofthen-dim ensionalEucl ideanspac e,theree xistsafunc tion f : R n ! R wi th Y r = f ( X ).Thei ndex r denotes then um berofrem o v e d hiddenuni ts.Gi v e nacri terion tode term i net here pre sentationer ror, the inuenc eofrem o ving r hidden unit sc anbedene das: Denit ion1 Thel ossofinfor mationduetoar emovalof r hidden un its is give n a s LI = Z j Y 0 Y r j dX; (5) wh er e the index r indicate s the nu m ber of r emov e dhi dden units. Le t's no w cons i der a nite s e tofpatte rn to b e represen ted. Th i s is the pro bl em of a ppro ximating a b ounded function f : A  R n ! R , from a b o unded subset A of n -dimension al E uclidea n space to a b ounded s ubset f ( A ) of o ne dimensiona l Eu clidean space, bymea ns of 144 A.B asti an ex am ples( x 1 ;y 1 ), ( x 2 ;y 2 ) ;:::; ( x k ;y k )w here y k = f ( x k ). Thelossof i nf orm ati onfort he di sc retec aseisdeneda s: De ni tion2 Given a ni tes etofinput/output data,t helossofinfor mation duetoth er emovalof r hidde nu nitsisdene dby: LI = k X i =0 j y i 0 y i r j dX; (6) wher etheindex r indic atesthenumb erofr emove dhi ddenunits. Thislossofi nform atio nf or theni tecasecanbeused to observ e thei nuenc eoftherem o v alof hidden units,th uspro vidi ng ap ossibil it y todeterminetheopti m umstruc tureof an et w orktore presen tagiv en i nput-output rel ation. Theore m2 Gi vena ninput-outputdatasetwith x k ve ctorsgener ate d r andomlyfr om A inac c or danc ewithaxe dpr ob ability densityfunction  ( x ) (  isassume dtob e0outside A ),andgivenamo delingmetho dthat c onver gestotheoutput Y andamethod tojudgether epr esentationo f thenetwor k, thenitisp ossibletondthem in im umnumb erofhi dden unitssucienttom odeltheoutput. Pr o of : Sinc eLIindi catesthel os so finform ation,the k trai ning patterns canb ere presen te db yusi ng at hree-la y ere dfe edforw ard neural net w ork with k 0 1hi dden uni ts.I nthef ol lo wi ng,oneh i ddenuni tafterthe other is rem oved whil ew atc hing the corresp onding LI v alues.  LI > 0 indi cates that the l ower bound of m ini mal n umber of hi dde n uni ts i s already ex ceede d.  LI =0 i ndicates that there stil le xist en ough rem aini ng h i dden units to re presen tthetraini n g pa tte rn. End of proof. AnEe ctiv eW a yt oG enerat eN euralNe t w ork. .. 14 5 Forp r acti calappli cations, Theorem2hastoberel ax edb yi ntroducingathresholdv alue  suc hthatwhileLI   ,hiddenuni tscan stil lbe re mo ved.A nothe rre laxation is to al lo wtheexi st enceoflo ca l m inim a.H o w ev er,usi ngse ve ra ln et w ork swi thdi ere nt initia lw e igh ts wil ll ik elyle ad us to thebe stpossi blesoluti on ,a lthoughi tcannotbe gu aran tee dthatthegl obalm ini mumi sf ound.Ne v e rthe le ss,f romt he appl icati onalpoi n to fv iewthereisnoneedtondther igh tnet w ork sizefortheloc al mini m um ,i ftheglobalm ini mumi sv eryhardtond. Suc hc asesarenotun usual .Infact, un ti l1988itw asnotev enknown forc ertainwhe therbac kpropagatione rrorsurfac es ha v el ocalm i nim a, whenM cIne rne yan dco-w ork ers[ 19]disco v ered al ocalm i nim umafte r a12-hourruno na Cr a y -2supercom puter. Iti sob vious,thatr em ovingonehi ddenunitafter theo therfrom ane t w orkw ithani niti alsi zeof k 0 1hiddenunitswil lrequi rel ong com putingtim es inm ostc ase s.I tistherefo re bette rtostartwi th asm all erinit ialnetw ork.Y e t,si ncethereno wexis tsthepossi bil it y thatthese lec tednet w orkis toosm al l,thei nuenceofadd i nghi dde n neuronsm ustalsobeobse rv e d. Denit ion3 Givenanitesetofin put/outputdata,thega inofi nform ationduetoad dingahiddenunitsi sdene dby: GI = k X i =0    y i 0 y i q    dX; (7) wh ere thei ndex q indic atesthenumb er of adde dhiddenunits. Gi v e nthi scri teriato ob serv ethei nuenceofaddingh iddenneurons,Th eo re m2 issti llv al idev e nift he initi alam oun tofhiddenun its isl essthan k 0 1, sinc eh idde nunitsc anbe ad dedw hi le watc hi ngthe ga i nofinform ation.  GI > 0i ndicates tha t the lo wer b ou nd of m inim al n um ber of hidden uni ts is no t y etexc eede d, si nc eaddi tiona l hidde nuni ts im pro ve the re presen tation cap abil it yofthene twork.  GI =0 i ndicates tha t the real ready exi sts e n ough hi ddenuni ts to re presen t the training pa tte rn. Like in the case of removing hidden units, a threshold value  can be intro duced fo r practica l a pplications. 146 A.B asti an 2.3 G e nerali z at i on Inthepr evi oussectio nam ethodol ogytodeterm inet he n um berof hidde nuni ts wa sg iv en.H o w ev er,i tw asa ss um ed thatacorrec tle arn- i ng sc hem eispro vided.Sinc ei nt hisstudyw em ai nl ydealw itht he problemofndin ga no pti m alne t w orkstruc turef or functionappro x- i m ation, th egenerali zationabil it yo fanet w orkm ustbee nsured, by whic htheabi li t yofanet work togene ral iz epatternso utsidethetrain- i ngsetism ean t. I no therw ords:i faninputv ector li esbet w ee nor cl oseto trai ni ngse texam pl es,thenets houldproduc eoutputswhic h arereasonablere latedtotheoutputsofthosetrai ni ngsets. M ore ov er, duringthel earni ngp hasethenet w orkshoul ddi sregardnoiseandoutl iers. Generall yspeaki ng,generali zationde pendsonthreep aram ete rs: thecom plexit yofthetr ai ni ngdata,then um beroftraini ngdata,and thenet w orksi ze. As m en ti one dinthei n troductio n,ano v e rsize dnetw orkwi lll ooseitsgene ra l iz ati onabi lit yandm em oriz esthetraining datainstead,i ncl udi ngitsnoiseandoutl iers.Ithasbee npo in tedout i n[ 20]thatane t w orkwhic hsi m pl ym emo ri ze sthetrai ni ngpatterns m a yf ai lwhe npre se n tedwithsim i larbutsl igh tlydi eren tpat terns. One issueofm em ori zati oni sthes o-c al ledo v ertti ngofthe data. Thisphenom e nonusuall yoc cursi fthet rai ningsetcon tainsnoise and outli ers.As ill ustratedinFi gure1,acurve tte dtoogoodtothedata setm i gh tbev ery po or forin te rpolationandextr apol ati on. Ov ertting is basi callyrelatedtothe structureofthenet w orkand thetrai ni ngti me .S incei nthisw orkonl yfeedforwa rdne t w orkswith one hi ddenl a y e rarec onsi dered, thestructureo ft he netw orki sdir ectl y rel atedt othen um berof neuronsi ntheh idde nl a y e r.O nem a yvi ew the n um b er o f the hi dden units a s the a vai lable degree so f freedom a net has in o rde rtore presen ta ce rta i ni np ut-outpu t rel ati on . T he second big inuenceon o vertti ng is the traini ng tim e, tha t is, the n um be rof ti mes a pattern is presente d to the net work duri n g the training. In this case, o vertting m igh tbeexpl ai ned a s the attem p t of the net work t o t all t ra i ning patterns, e ven after a go o d app ro x im ati o n of the inputoutpu t relation isachieved. This is often address edasovertra ining. One appro achto avoid overtraining is to stop the training when a 154 A.B asti an Notic ethatinthisexam ple then um berofc lustersi si den ti calw ith thenalsol uti on of theal gorithm .Ho w e v e r, if thec lusteri ngal gori thm i sc on tinued,a notherm ini m umcanbef ound fo r13cluster s.The correspondi ngMSEtothat netw or ksiz e(13hiddenunits)w as in de ed sm all erthant he onew i th11hiddenuni ts,as show ni ntabl e2.This resultsi ndicates,thattheclusteringalgori thmshoul dbec on ti nued ev enifam ini m umisrea ched.Thec auseofthisl iesin thef ac t,that thev ali dati on functi ononlyndsl oc alm ini mum. Ho w e v er,therst m inim um foundalsol eadstoagood represen tati on. cl uste rs/n um berofClusterMSE hiddenu nitsv al idit y 2 {1203.991 0. 002185 3 {1529.123 0. 00154 2 4 {21 15 .5470.0 01121 5 {25 37.0830.0 00038 6 {2674.5920. 000027 7 {2 717.5220. 000027 8 {3621.3820. 000025 9 {4600.3220. 000025 10{4723.9440. 000025 11 {4752.0030.0 00019 12{4721.0300. 0000 25 13 {4782.2690.0 00015 14{4752.3020. 000025 T abl e2.Res ul ts forthepoly nom i alfunction z = x 3 + y 3 . Ex am ple 3. Appro xi mat ing z = x 4 + y 4 . The results for a ppro xi mating the fourth o rder p olynom ia l function also i ndicate that the searc hfor the o pti mum n um ber of cl us te rs should be con tinued f or se veral steps. Assho wn i nT able 3 the c luster vali dit y funci to n re ac hedam inim um for 8 c lusters, and in thi scase , the net work structure i den ti cation a l gori th m term inated with 9 hidden units as the optim al solution. However, the clu stering pro cedure was continued a nd the validity functio n reached an other minimum at 14 clus ters, here the a lgo rithm AnEe ctiv eW a yt oG enerat eN euralNe t w ork. .. 15 5 term inatedwi th14hi dde nu ni ts as theoptim alsol ution. Note thati n thiscaseasm alle rM SE w asac hie v ed,assho wn in Ta bl e3. cluster s/n um berofCl usterMSE hi ddenunitsv al idit y 2 {47. 4300. 014685 3 {603.4530. 002632 4 { 1031.382 0. 001712 5 { 16 10.574 0.0 02 310 6 { 17 40.5450.0 00 120 7 { 1960.0010. 000111 8 { 1974.007 0. 000168 9 { 1960.532 0.0000 83 10{1942.9000. 000095 11{1921.1120. 000085 12{1904.7320. 000080 13{1905.2320. 000075 14 { 2012.5830.0000 60 15{1992.24 10. 000068 T able3.Resultsforthepo lynom i alf unc tion z = x 4 + y 4 . Exam pl e4. Appro xim ati ng z = x 5 + y 5 . Finall y, the algo r ithm w a s appli ed on da ta from a fth o rde r pol yno m ial function. In th i s case there exi sts onl yone m ini mum for the val idation function, as sho wn i nT able 4. The optimal n um ber of c luster was 7, whichalso turned out to b e the optimal numberofhidd en units. 156 A.B asti an cl uste rs/n um berof ClusterMSE hiddenunits v al idi t y 2 { 719. 2010. 001983 3 {1720.214 0. 00087 2 4 {2414.241 0. 00002 6 5 {3212 .231 0. 00002 0 6 {3444.954 0. 00002 1 7 {3721.328 0.0 00019 8 {3671.2930. 000020 9 {3674.2930. 000020 10{3420.2030. 000020 11{3523.0230. 000020 12{3600.3430. 000020 13{3670.3240. 000020 14{3669.3420. 000020 T abl e4.Re sul tsforthepolyn om i alfunction z = x 5 + y 5 . 4.2Hum anop er ati onofac he m i c alpl an t Inthela st ex am pletheproposedalgor ithm is testedusingdataofa h um anoperato r' sc on trol ofac he m ic alpl an tp ro duci ng apol ym e rb y thepol ym er izationofsom em onom e rs[23].Theo pe ratorde term i nes the s e t poi n t for the m ono m er o w rate and pas se sthisinform ati o n to a PID c on tro l ler, whic hc alculates the a c tua l m onom er  owrateinputfor the plan t. There are 7 0 data sets a vail abl ewi th e ac hsetc ons i sting of v e inputs (m ono m er concen tra ti on, c hangeofm ono m er conce ntra ti on , monom er o w rate, t em pe rature 1 and tem p erature 2 i ns ide the plan t) and one single outpu t, na m ely the set poi ntform onom er o wrate . Here, the initial network contained6 hidd en units. T he a lgo rithm terminated with 8 hidden units (see Tab le5). 158 A.B asti an e. g. atthev e ryearlybeginning of theproce ss. 5 C onclus ions Int hi sstudyasim pleal gorithmt od y nam i call ydete rm i nethen um ber ofhi ddenuni tsofathre e-la y erneuralnet w orkw asprop os e d.This algori thms ta rtsb ycl usteringtheoutputd at a.Then um berofc lusters is th ei ni tialn um berofhiddenu ni tsofthenet w ork .Inthefollo wi ng, thehidde nl a y erofthenet w orkgro wsorshrinksinsiz e, dependingon the resul tofthege nerate-and-evaluateproce dure . Ther es ul tsinse ction4i ndi cate, that theappli cationofthe FCM- algori thm to determ ineaniniti alne t work structurei sv e ryuseful .I n allc asesthec lusteri ngresultw ascl osetothe nalresultoftheal gori thm ,andthe reforesa v edm uc hcom putationalti m e .Y et,oneshould alw a y sk eepinm indthattheF CM-algorithmm a ygetstuc kinalocal m i nim um ,asithas be ensho wni nsom eofthepr esen te de xam ple s. Thec lusteri ngproc edureshouldther eforebec on ti nuedf oratl eastanotherthre ec lusteri ngst epsa fterthev ali dati onfunc tion foundarst m i nim um . Al thought heproposedal gori thmisma inlythough ttobeusedfor functi onap proxim ation,itcanal sobeus ed to so m eex tendforpattern lear ni ng.I nthi scase,allinputa ndoutputdatashouldbec lustere d. Theal gorit hmw asdev elop ed forpote n tialc on trol andm od el ing desi gnappli cations.Curren ta nd futurew orki stherefore dire ctedin thatw ay . Ac kno wl edgm en ts The autho r w oul dli keto t han k Pro f .T.T er a no, the d i recto r o f L IF E, Prof. L.T. K ocz y,who e nd o ws the LIFE cha i r of fuzzy theory at the Tokyo Institute of Techno logy, a nd esp ecially the reviewers fo r their kind help a nd valua ble su ggestions. AnEe ctiv eW a yt oG enerat eN euralNe t w ork. .. 15 9 Referen ces [ 1]Sai to,K. ,Nak ano,R ., Me dicaldi agnosti cexpert System base d onPDPm odel , Pro c. I nt.J ointConf.onNeu ralN et works ,pp. 25 5- 26 2,SanDi ego,1988. [ 2]LeC un,Y .,Boser,B .,Denk e r,J.S ., Henderson,D.,Ho w ard,R.E., Hubard,W. ,Jac k el ,L. D. ,B ac k propagati onappli edto Handwri ttenZ IPc odeR ecogni tion, Neur a lComputation ,V ol.1,pp.541- 551,198 9. [ 3]T e sauro,G.,Ne urogam m onWi nsCom puterO l ym pi ad, Neur al Computation ,V ol.1,pp.321-323,19 90 . [ 4]Refene s,A .N ., Azem a-Barac,M. ,Chen,L. ,Ka ro ussos,S.A ., Curre ncyEx c hangeRateP redi cti onandN euralN et w ork Design Strate gi es, Neur alComp ut.&A pplic. ,V ol.1(1),pp.46-58, 19 93. [ 5]Narendra ,K.S. ,P arthasarath y ,K.,Iden ticationandCon trolof Dy nam i calSystem sU singNeuralNe t w orks, I EEETr ans.onNN , V ol.1,N .1,Mar c h,1990. [ 6]Tho dberg,H .H ,N euralInform ation ProcessingSystemforPig CarcaseGradingi nDanishSl augh terhouses, Neur alComp ut.& App lic. ,V ol .1( 4),p p. 248-255,1993. [ 7] Hornik,K. ,Stinc hcom b, M.,W hi te,H .,Mul til a y e rfeedforw ard ne tw orks ar euni v e rsalappr ox ima tors, Neur alNetwo rks 2,pp.3 59 - 366,1989. [8]Si etsm a,J.,D o w,R .J.F .,Ne ur alnetprunin g-wh yandho w, Pr oc. IEEE Int. J oin t Conf. Neu r al N etw or ks , pp. 1 32 5-1 333 , San Di ego , 1 98 8. [9] Ka rni n, E . D., A S i mpl e Pro ce d ure fo r Pruning Bac k- Propa gation Trai ne dNe ura l Ne tworks, IEEE T rans. on Neur al Ne tworks ,Vol. 1 (2), June 1 990 . [10] No da, I., L earning Metho d byOverload, Proc. IJCNN'93 ,pp. 1 35 7-1 360 . Nagoya, 1993. 160 A.B asti an [1 1] Sartori ,A. S.,An tsakli s, P.J. ,Asim pl em ethodtod eriv eb ounds onthesiz eandt ra in mul til a y erneuralnet works, IEEET r ans.on Neur alN et s ,lette r,V ol .2(4),pp.46 7-47 1, 1991. [ 12]Huang ,S .C., H uang,Y. F.,B oundson Numb er ofHi ddenU nits inM ulti la y erP erc eptrons, IEEE Tr ans.Neur alNetworks ,Vol. 2 (1) ,pp.47-55,1991. [ 13]Rum e lhar t,D.E. ,H in to n, G.E.,Wil li am s,R. J.,Le ar ni ngrepesen tationb ybac k-prop agatingerrors, Natur e ,23 2, pp.533-536, 1986. [ 14]Och iai, K. ,Usui,S.,I m pro v edKi ck OutLearni ngAlgorithmwith Delta-Bar-De lta-B arR ul e, Pr o c.IEEEInt.Conf.onN N9 3 ,vol. 1, pp .269-274,SanF ranc isco,1 993. [ 15]V ogl,T.P . ,Mangis,J.K. ,Ri gl er, A .K. ,Zi nk,W. T. ,Al k on,D.L., Ac cel eratingthecon v ergenceoftheB ac k-Propagatio nMethod, Biol.Cyb ern. 59,pp.257-263,1 988. [ 16]Pirez ,Y. M., Sark ar,D. ,Bac k-Propagationwithcon troll edOsci llationofW eigh ts, Pr oc.I EEEInt.Conf.onNN93 ,v ol. 1, p. 21 -26,SanF ranci sco,1993. [ 17]v anOo y en,A ., Nie nh uis,B. ,I m pro vi ng theC on v ergenceofthe Bac k-PropagationA lg ori thm , Neur alNetworks ,V ol.5 ,p.465-471, 1992. [ 18]Yam ada,T.,Y abuta,T. ,N euralN et w orkCon troll erU singAu- to t uni ng M etho d for N onl inear F uncti ons, IEEE T rans. on NN , Vol. 3, N .4,July1992. [19] M cI nerney ,J. M, Haine s, K.G. ,Biafo re ,S. ,H ec ht-Ni elsen, R., Ca n ba c kpropa gation e rro r surfa c es ha ve no n-globa l m ini ma?, IJCN N'89 , I I, 627 , 198 9. [2 0] Siets ma , J., Dow, R.F.J., Creating Articial Neural Networks Tha t Generalize, Neural Ne tworks ,vol. 4, pp. 6 7-7 9, 19 91. AnEe ctiv eW a yt oG enerat eN euralNe t w ork. .. 16 1 [21] Bez dek,J.C.,P atte rn Re cogn i tionwithF uzz yObjecti ve Fu nc tion Algorithm ,New York:P le n umPress,1981. [22]Zade h,L. A.,F uzzyS ets, Infor m.Contr. ,V ol .8,p p. 33 8-353,1965. [23] Su geno,M. ,Y asuk a w a,T.,Afuzz y-Lo gic-B asedApproac hto Qual itativ eMode ling, IEEET r ans.onF uzzySys tems ,1 (1 ), 1993.