A ool o Mul i-S a egy Lea ning
F ancisco Reinaldo1,2, Ma cus Siquei a1,2, Rui Camacho1, and Lu´ıs Paulo Reis1
1LIACC, Facul y o Enginee ing, Uni e si y o Po o, Po ugal
2GIC, Depa men o Compu e Science, Uniles eMG, B azil
ei eup, camacho,lp eis{@ e.up.p },[email p o ec ed]
Abs ac . This pape p esen s he AFRANCI ool o he de elopmen
o Mul i-S a egy lea ning sys ems. AFRANCI allows use s o build, in
an in e ac i e and easy way, complex sys ems. Sys ems a e buil using a
wo s ep me hodology: design o he s uc u e o he sys em; and ill in
he modules. The s uc u e o he a ge sys em is a collec ion o in e con-
nec ed modules. The use may hen choose among a a ie y o lea ning
algo i hms o cons uc each module. The ool has se e al buil -in Ma-
chine Lea ning algo i hms and in e aces ha enable i o use ex e nal
lea ning ools like WEKA o CN2. AFRANCI uses he in e dependency
o he modules o de e mine he sequence o hei aining. To imp o e
usabili y, he ool uses a w appe ha hides om he use he pa am-
e e uning p ocedu e o each algo i hm. In a inal s ep o he design
sequence AFRANCI gene a es a compac and legible eady- o-use ANSI
C++ open-sou ce code o he inal sys em.
To illus a e he concep we ha e empi ically e alua ed he ool in he
con ex o he RoboCup Rescue domain. We ha e de eloped a small
sys em ha uses bo h neu al ne wo ks and ules in he same sys em.
The expe imen ha e shown ha a e y signi ican speed up is a ained
in he de elopmen o sys ems when using his ool.
1 In oduc ion
O e he pas ew yea s, he e has been a signi ican inc ease in he in e es
o wo king wi h he e ogeneous lea ning algo i hms in o de o achie e as and
complex beha iou s in au onomous agen s. An Au onomous Agen (AA) can
be seen as a collec ion o componen s o he e ogeneous lea ning modules wi h
well-de ined in e aces and ine uned beha iou s. I has been ecognised ha
di e en beha iou s may equi e di e en lea ning s a egies.
A popula AA a chi ec u e conside s se e al lea ning algo i hms/modules
a anged in ho izon al and/o e ical le els o compose beha iou s wi h di e en
le els o abs ac ion. This laye ed lea ning is specially adequa e o domains ha
a e oo complex o a di ec mapping om he inpu o he ou pu ep esen a ion
o wo k [1]. This app oach b ings some new challenges on he a angemen o
hese modules o achie e inne easoning, p edic ion and abs ac ion me hods
ins ead o classical planning esea ch. Addi ionally, he use o obus ools a e
necessa y o suppo his kind o app oach. Al hough no in he con ex o AA
© A. Gelbukh, C.A. Reyes-Ga cía. (Eds.)
Ad ances in A i icial In elligence.
Resea ch in Compu ing Science 26, 2006, pp. 51-59
Recei ed 02/06/06
Accep ed 03/10/06
Final e sion 11/10/06
cons uc ion, bu also using a hie a chical o ganisa ion o modules, Alan Shapi o
[2] p oposes he use o “s uc u ed induc ion” o add ess he p oblem o inducing
complex concep s.
Howe e , when using a con en ional p og amming language, like C o C++
o encode such modules, an expe p og amme is o en equi ed. Each new
use s a s a p ojec om sc a ch, and occasionally i esul s in a bad p og am
code s uc u e e ealing p oblems whene e he code needs o be ex ended o
upda ed. To o e come such di icul ies some ools p opose uni o m and easily
modelling acili ies o he lea ning modules. Such ools include Ma lab c
[3]
and SNNS (S u ga Neu al Ne wo k Simula o ) c
[4] and [5]. These ools a e
limi ed as a as he use in e ace aspec s a e conce ned since hese a e o en
neglec ed. In addi ion, he ools do no suppo he design o se e al modula ,
hie a chic and complex s uc u es o he e ogeneous lea ning modules in he same
en i onmen as AFRANCI does.
The AFRANCI ool [6] o e s solu ions o assembling and linking oge he
g aphically on he sc een he modules equi ed o c ea e la ge scale beha iou -
based sys ems. In he de elopmen o AA applica ions he e is he need o agg e-
ga e di e en kinds o beha iou s wo king oge he o inc ease he complexi y
o beha iou s. The use ’s choice is no es ic ed o low le els bu is ee o
choose he sui able abs ac ion le el. Ano he ad an age o AFRANCI is he
possibili y o using ex e nal lea ning ools/algo i hms such as he WEKA [7]
lib a y o CN2 induc ion algo i hm [8] o compose he e ogeneous lea ning mod-
ules. The ool makes easily adjus able and ex ensible connec ion wi h WEKA
lea ning modules and CN2 wi hou he use ’s pe cep ion. These aspec s will be
p esen ed in Sec ion 2.
Acco ding o he p oposed objec i es, his wo k has enabled he de elopmen
o a ool ha can: a) suppo g aphic designs o la ge and ex ensible beha iou -
based a chi ec u es composed o he e ogeneous clus e s wi h cogni i e p ocesses;
b) make a ailable a wide se o a ied unc ionali ies o con ol modules; c)
o e acili ies and esou ces o in e connec se e al s uc u es o ex e nal con ol
p ocesses so ha hey could communica e in a mul i-en i onmen ; d) a ange
lea ning modules in ho izon al o e ical le els o inse new o subs i u e he
p e ious agen cha ac e is ics by an adop ed echnique om so wa e enginee ing
[6] and [9]; e) o e pa allelised ne wo k aining; ) au oma ically gene a e a
eady- o-use ANSI C++ open-sou ce code om he sc een.
The con ibu ion o his pape goes beyond a objec -o ien ed implemen a ion
o ideas. I add esses abili ies, specialisa ions and policies o accomplish speci ic
asks in an Au onomous Agen se ing by using a mul i-s a egy lea ning in o -
de o achie e complex beha iou s. We o e a scalable modelling and pa allel
en i onmen s wi h he pu pose o speeding up beha iou agen simula ions. Fu -
he mo e, he ool is a aluable con ibu ion o use s wi hou knowledge abou
deep p og amming language and lea ning modules because i o e s high le el o
abs ac ion and au oma ic c ea ion o sou ce-code, as obse ed in Sec ion 5.
The es o he pape is o ganised as ollows. Sec ion 2 p esen s an o e iew
o AFRANCI a chi ec u es and how i handles he gene a ed sys em’s modules.
52 Reinaldo F., Siquei a M., Camacho R., Reis L.
Sec ion 3 desc ibes how o design beha iou s uc u es in he in e connec ed
a chi ec u e in o de o achie e he bes agen pe o mance. The use o w appe s
is desc ibed in Sec ion 4. Expe imen s, esul s and discussion a e p esen ed in
Sec ion 5. Rela ed wo k is p esen ed in Sec ion 6. Finally, some conclusions a e
d awn in he las sec ion.
2 AFRANCI Tool
The AFRANCI Tool [6] is a ool buil o e some classes o open-sou ce Py a-
midNe F amewo k [10] pla o m, which ex ends he use o ex e nal lib a ies
o lea ning algo i hms like he WEKA c
lib a y and CN2 induc ion algo i hm.
Py amidNe was es ic ed o a subse o A i icial Neu al Ne wo k (ANN) al-
go i hms whe eas wi h he new abili y o use WEKA c
lib a y and CN2 he
use has access o a use ul eposi o y o machine lea ning algo i hms o da a
p e-p ocessing, classi ica ion, eg ession, clus e ing, associa ion ules and isual-
isa ion.
The AFRANCI ool is composed o h ee main pa s, which a e he G aphic
Use In e ace (GUI), he Machine Lea ning Modules (MLM) and he Au o-
ma ic Open-Sou ce Code Gene a o (ACG). Fi s , GUI is a se o main classes
o handling and modelling lea ning modules, using g aphic elemen s ha will
in e ac wi h he use . MLM implemen s he cons uc ion o he modules using
he chosen lea ning algo i hms. Finally, ACG ecei es a desc ip ion o pic o-
ial ep esen a ion o he sys em s uc u e and p oduces he sho au oma ic
open-sou ce code.
The ool allows use s o design and implemen beha iou -based a chi ec u es
h ough he in e connec ion o elemen a y he e ogeneous con ol modules in he
o m o a ci cui diag am. New s uc u es can be cons uc ed by linking s anda d
s uc u es o modules oge he . This o m is e y amilia o enginee s in digi al
sys ems design and in model analysis sys ems ools (e.g. SIMULINK c
). I makes
some i ems a ailable, such as desk op cons uc o , senso , ac ua o , line links,
skins o lea ning modules, menus, dialogue box. Each g aphical elemen can be
d agged and d opped o a e-sizable sc een and i s ea u es (colou , label, o ma
and size) be adjus ed. He e ogeneous con ol modules a e s anda d boxes wi h
inpu and ou pu o da a. Fo ins ance, i he use is wo king wi h ANN, he/she
has access o a numbe o hidden neu ons, neu on ans e unc ions, aining
pa ame e s e c.
Wi h he pu pose o speeding up he implemen a ion o lea ning modules,
he ool o e s h ee o he impo an ea u es: he mul i pa allel en i onmen ,
he au oma ic ne wo k design o modules and he au oma ic de elopmen o
open-sou ce code. Fi s , each new en i onmen includes a powe ul pa allel e-
sou ce o aining he p oposed designed a chi ec u e. The use can link and
pu oge he se e al sub-p ojec s and un hem, a he same ime because i
is ee o manual schedule . In he design p ocess, once adjus men s ha e been
made, he module is unligh ed and he aining and simula ion can be un again
quickly. The second ea u e is he on sc een au oma ic/wiza d design o ne -
A ool o Mul i-S a egy Lea ning 53
wo ks o lea ning modules by impo ing o CSV (Comma Sepa a ed Values)
iles. When using ANN/WEKA/CN2, he use links s anda d lea ning modules
oge he ; each lea ning module can be simula ed a any le el o abs ac ion o
ine uning o he assembled sys em. The use needs li le knowledge abou i . As
a las s ep he ool p oduces clean and eady- o-use ANSI C++ open-sou ce o
in o ma ion usion, planning and coo dina ion wi h a ew mouse clicks. By using
a high-pe o mance in e p e a ion algo i hm, his unc ional bu compac ANSI
C++ execu able co e is c ea ed om he d awn p ojec . This ANSI C++ code
can be edi ed on sc een in o de o be modi ied and compiled easily in di e en
ope a ing sys ems.
3 Designing a S uc u e
In his sec ion we b ie ly analyse he simple s eps o design, ain and ob ain a
inished sys em composed o di e en lea ning modules.
Fi s , i is necessa y o design he agen ’s s uc u e. The use should plan
how many modules, how hey a e in e connec ed and wha algo i hms o use
in each module. In he nex s ep he use d aws a p ojec in he ool desk op
(sc een). The p ojec mus ha e inpu , con ol modules, ou pu s and links. A e
designing he p ojec , e e y module ecei es he da abase ile ha s o es da a
o eed i in he aining phase.
The in e connec ions a e es ablished be ween inpu s, con ol modules and
ou pu s o make a comple e wi ed ne wo k. The use can connec wo en i-
onmen s: om he ou pu o modules in he i s en i onmen o he inpu o
o he assembled o modules in a second en i onmen . This ool allows he use
o in e connec e e y hing wi hou loss o pe o mance because i uses a pa allel
en i onmen . The use can une each module by changing he de aul module
pa ame e s. Ca e ul is needed in his s ep because a badly planned se up may
lead o he eme gence o w ong easoning p ocesses. This s ep is he only one
ha equi es a li le mo e knowledge abou a anging o modules.
The aining p ocess is simple and consis s o aining and ine uning o
he beha iou -based a chi ec u e o achie e he agen ’s goal. The au oma ic
aining p ocess is esponsible o almos e e y hing and based on he da a
lux sequence o igge modules o aining. Independen ly o he ho izon al o
e ical a chi ec u e le el, he use can ollow he i e a ion aining p ocess o
each module by g aphics, da a windows and o he s.
Finally, he use can gene a e a eady- o-use ANSI C++ open-sou ce o plug
i in he agen . The sou ce code is a codi ica ion o he g aphic modelling. Inside
o his code, use will ind inpu and ou pu connec ions, weigh s, an ac i a ion
unc ions, in he case o ANN, a compac main execu able co e, and some inpu
and ou pu ma ixes. Summing up, hese quick s eps help use o subs i u e
la ge p og ams, ime consuming p ojec s and con using lines o code.
54 Reinaldo F., Siquei a M., Camacho R., Reis L.
4 W appe s
Almos all Machine Lea ning sys ems ha e pa ame e s ha mus be uned o
achie e a good quali y o he cons uc ed model. An expe ienced p ac i ione
knows ha changes in he pa ame e ’s alues may lead o qui e di e en esul s.
To une a sys em’s pa ame e equi es knowledge o he sys em. This is mos
o en a se e e obs acle o he wide sp ead use o such algo i hms.
As p oposed by John [11] one possible app oach o o e come such a si ua ion
is by he use o a w appe . A w appe p oduces se e al models using di e en
combina ions o he lea ning algo i hm and e u ns he “bes ” model. In ou ool
he w appe op imises he es se e o a e es ima ion. This au oma ic uning
o pa ame e comple ely hides he de ails o using he lea ning algo i hms om
he use . I is he e o e a way o make he ool usable by a wide ange o use s.
When he o al numbe o combina ions o pa ame e ’s alues is small3 he
ool ies all he combina ions and chooses he bes one. When he numbe o
combina ions is la ge he pa ame e s a e uned using a Gene ic Algo i hm (GA)
[12] wi h mu a ion and c oss-o e GA ope a o s. Fo he ANN we also choose
au oma ically hei s uc u e. A GA is used o choose he bes numbe o hidden
laye s and numbe o neu ons in each laye o each ANN. The ype o ANN is
also included in he au oma ic choice.
As u u e wo k we in end o ex end he use o w appe s o do ea u e (subse )-
selec ion as in [13] and [14]. This acili y would equi e howe e a igh e ela ion-
ship be ween he modules syn hesis and hei in e -connec ion. I he w appe
decided ha some ea u e is no ele an o he classi ie hen ha inpu would
ha e o be emo ed in he modules in e -connec ion design s age.
5 Expe imen s
To illus a e he ea u es o AFRANCI and acili a e he eade ’s unde s anding
we gene a ed a simple a i icial p oblem and da ase in he RoboCup Rescue
se ing. The p oblem we de ised is o decide i a ambulance o i eman should
escue o no a ci ilian o a nea es e uge. The ci ilian is somewhe e in a bu ning
building. The decision is commonly made based on he localisa ion and agen ’s
and ci ilian’s li e condi ions. The main independen a iables include: he posi-
ion (X, Y) o he ambulance, i eman, building on i e, i e b igade, he nea es
e uge ( escue building), and o he ci ilian; he li e condi ion measu e o he
i eman and ci ilian 4, he building ola ile in o ma ion is composed o ea lie
bu n , s a e and s uc u e; he s a e o he ambulance (busy/ ee) o ecei e he
ci ilian and o he i eman (busy/ ee) o ex inguish he i e o o escue he
ci ilian; a measu e ha assesses he di icul y o he ci ilian escue si ua ion.
The sys em de ised is composed by se e al modules ha encode he deci-
sion o he i eman and he ci ilian and pa icula ly a main module o combine
i eman and ambulance decisions. Figu e 1 shows he modula s uc u e o he
3In he cu en implemen a ion small is less han 20.
4A measu e be ween 0 and 100 o he ene gy he i eman can use.
A ool o Mul i-S a egy Lea ning 55

Fig. 1. A chi ec u al Design.
sys em cons uc ed in e ac i ely by he use . No pe cei ed in he igu e is he
he e ogenei y o he modules. Di e en modules use di e en ep esen a ions
sand whe e induced using di e en lea ning algo i hms. Re e ing o he mod-
ule’s labels in Figu e 1 he ollowing algo i hms we e used. The Ci ilian module
was cons uc ed using AFRANCI buil -in eed o wa d ANN. The Ambulance,
BuildingOnFi e, Fi eman and RescueFi eman modules we e cons uc ed using
CN2 ule lea ne . The o he modules, Building and Decision, we e cons uc ed
using WEKA’s J48 Decision T ee algo i hm. The use had only o selec he
numbe and place in he window o he modules, o connec hem and choose
inpu and ou pu a iable names. All his was done using d ag-and-d op ope -
a ions. He hen p o ided he da ase and he ool ained he modules in he
co ec sequence and gene a ed a C++ p og am ha encodes he sys em.
56 Reinaldo F., Siquei a M., Camacho R., Reis L.
J48 p uned ee
------------------
ambulance_ap = TRUE: RescueAmbulance
ambulance_ap = FALSE: RescueFi eman
IF occupied = occupied
THEN class = n [52 0]
ELSE
IF ci ilian = no ap
THEN class = n [20 0]
ELSE
IF Yamb > 8164.50
AND Xci < 9204.50
THEN class = y [0 7]
ELSE
IF Yamb > 340.00
AND 386.00 < Yci < 7731.50
AND X e uge > 3800.00
THEN class = n [9 0]
ELSE
IF Yamb < 5721.00
AND X e uge > 866.00
THEN class = y [0 8]
ELSE
(DEFAULT) class = n [4 0]
a) b)
Fig. 2. a) Decision T ee gene a ed by WEKA’s J48 lea ne . b) Rule se gene a ed by
CN2.
The ac ion o deciding which agen will be esponsible o escuing he inju ed
ci ilian is aken in he Decision module (see Figu e 1). The i eman agen will
pe o m a escue ac ion only i he ambulance agen will no be able o do so.
In he case o bo h, he i eman and ambulance, a e capable o escuing he
ci ilian, he module decides in a ou o he ambulance agen because a i eman
agen has o ex inguish i es in bu ning buildings wi h he aim o p ese ing he
ci y. In o de o check i he ambulance is en i ely ap o escue a ci ilian (see
Figu e 2) he ules induced by CN2 es ablish ha : ( ule 1) i he ambulance is
occupied hen i is useless o a emp he escue; ( ule 2) i he ci ilian has no
enough “ i ali y” hen i is also no escued; (o he ules) he ci ilian will be
escued i i has enough “ene gy” and he ambulance is be ween he ci ilian and
he escue place o he wise i will no be escued.
6 Rela ed Wo k
In o de o de elop a obus ool, we analysed wo ele an ones. The Ma -
Lab c
and SIMULINK c
 ools a e mainly conce ned wi h he o e o a de ailed
design o a con ol p ocess [15] and o au oma ically gene a e a eady- o-use code
o i . Howe e , ha ool has wo main limi a ions ha a ec he design o he
whole p ojec . Fi s , he use canno design a complex s uc u e composed o se -
e al he e ogeneous modules no a in e connec ed a chi ec u e because he ool
A ool o Mul i-S a egy Lea ning 57
does no o e acili ies no a speci ic mul i-en i onmen o wo k wi h. These
limi a ions inhibi he use ’s abili y o handle di e en le els o abs ac ion,
se e al p ocesses in he same en i onmen . Second, he eady- o-use code was
gene a ed om a simple lea ning algo i hm is la ge o be wo ked, complex o be
unde s ood and unp ac ical o be used. O he s udied so wa e simula o was
he SNNS c
(S u ga Neu al Ne wo k Simula o ) [4] [5] ha i exclusi ely
wo ks wi h ANN in o de o c ea e hei applica ions. Al hough he ool o e s a
good g aphic en i onmen , i s epe o y is limi ed o only a beha iou le el a a
ime, wi hou o e ing a pa allel en i onmen o ain he ANNs. Mo eo e , i is
limi ed conce ning expandabili y o new ways o g aphically p e iew he whole
s uc u e ha is being wo ked wi h.
In Sec ion 2, we p esen ed he ool which o e s a use iendly pa allel g aphic
en i onmen ha acili a es he de elopmen and aining o Modula and Hie -
a chic Sys ems by including se e al modules wi h di e en lea ning algo i hms.
The sys em a chi ec u e p oduced wi h he AFRANCI ool p omo ed he Mul i-
S a egy lea ning as long as a ious lea ning modules a e used o compose he
s uc u e.
7 Conclusions
In his pape we desc ibed a ool o he de elopmen o Mul i-S a egy lea ning
sys ems. Using a iendly g aphical in e ace he use may de ine he modula
s uc u e o he sys em and choose he lea ning algo i hms o cons uc each
module. He hen p o ides he da ase and le s he ool ain, in he co ec se-
quence, each module p oduces a comple e sel con aining p og am encoded in
he C++ language.
The ool p o ides se e al lea ning algo i hms and is able o call ex e nal
lea ning algo i hms o cons uc he modules. The deploymen o he ool con-
i med he assump ion ha i was easy and as o de elop Mul i-S a egy sys em
wi h AFRANCI.
As u u e wo k we in end o de elop w appe s (John, 1994b) o au oma ically
une he modules pa ame e s, op imising he es se e o a e es ima ion.
Re e ences
1. Pe e S one and Manuela M. Veloso. Laye ed lea ning. In Machine Lea ning:
ECML 2000, 11 h Eu opean Con e ence on Machine Lea ning, Ba celona, Ca -
alonia, Spain, May 31 - June 2, 2000, P oceedings, olume 1810, pages 369–381.
Sp inge , Be lin, 2000.
2. A.D. Shapi o. S uc u ed Induc ion in Expe Sys ems. Adison-Wesley, Woking-
ham, 1987.
3. Ma hwo ks. Ma lab. Ma hwo ks, Inc, Na ick, MA, 1999.
58 Reinaldo F., Siquei a M., Camacho R., Reis L.
4. A. Zell, N. Mache, T. Somme , and T. Ko b. Design o he snns neu al ne wo k
simula o . In H. Kaindl, edi o , 7. ¨
Os e eichische A i icial-In elligence-Tagung,
pages 93–102. Sp inge , Be lin, Heidelbe g, 1991.
5. And eas Zell, Niels Mache, Ral Huebne , Michael Schmalzl, Tilman Somme ,
and Thomas Ko b. SNNS: S u ga neu al ne wo k simula o . Technical epo ,
S u ga , 1992.
6. F ancisco Reinaldo, Rui Camacho, and Lu´ıs Paulo Reis. A anci: An a chi ec u e
o lea ning agen s. Phd epo , FEUP, Po o, Po ugal, Augus 2005.
7. Ian H. Wi en and Eibe F ank. Da a Mining: P ac ical machine lea ning ools and
echniques. Mo gan Kau mann, San F ancisco, 2nd. edi ion, 2005.
8. P. Cla k and T. Nible . The CN2 algo i hm. Machine Lea ning, (3):261–283,
1989.
9. Robe W ay, Ron Chong, Joseph Phillips, Se h Roge s, and Bill Walsh. A Su ey
o Cogni i e and Agen A chi ec u es. Uni e si y o Michigan, 1994.
10. F. A. F. Reinaldo. P ojec ing a amewo k and p og amming a sys em o de elop-
men o modula and he e ogeneous a i icial neu al ne wo ks. Dep . o compu e
science, Fede al Uni . o San a Ca a ina, Flo ianopolis, B azil, Feb 2003.
11. H. Geo ge John. C oss- alida ed c4.5: Using e o es ima ion o au oma ic pa-
ame e selec ion. Technical no e s an-cs- n-94-12, Compu e Science Depa men ,
S an o d Uni e si y, Cali o nia, Oc obe 1994.
12. J.H. Holland. Adap ion in Na u al and A i icial Sys ems. Uni e si y o Michigan
P ess, Ann A bo , Michigan, 1975.
13. Ron Koha i. W appe s o Pe o mance Enhancemen and Obli ious Decision
G aphs. PhD hesis, S an o d Uni e si y, 1995.
14. Ron Koha i and Dan Summe ield. Fea u es subse selec ion using he w appe
me hod:o e i ing and dynamic sea ch space opology. In Fi s In e na ional Con-
e ence on Knowledge Disco e y and Da a Mining (KDD-95), 1995.
15. Richa d C. Do and Robe H. Bishop. Mode n con ol sys ems. Ten h, 2004.
A ool o Mul i-S a egy Lea ning 59