i
Mas e ’s Deg ee P og am in
Da a Science and Ad anced Analy ics
EasyML
An Au oML Sys em using Me a Lea ning and Pa icle Swa m
Op imiza ion
Lyinde Nelson Swale
Disse a ion
p esen ed as pa ial equi emen o ob aining he Mas e ’s Deg ee P og am in Da a Science and Ad anced Analy ics
NOVA In o ma ion Managemen School
Ins i u o Supe io de Es a ís ica e Ges ão de In o mação
Uni e sidade No a de Lisboa
MDSAA
ii
i
NOVA In o ma ion Managemen School
Ins i u o Supe io de Es a ís ica e Ges ão de In o mação
Uni e sidade No a de Lisboa
EASYML
by
Lyinde Nelson Swale
Disse a ion p esen ed as pa ial equi emen o ob aining he Mas e ’s deg ee in Da a Science and
Ad anced Analy ics, wi h a Specializa ion in Da a Science
Supe iso / Co Supe iso : Leona do Vanneschi
Feb ua y 2023
ii
STATEMENT OF INTEGRITY
I he eby decla e ha ing conduc ed his academic wo k wi h in eg i y. I con i m ha I ha e no used
plagia ism o any o m o undue use o in o ma ion o alsi ica ion o esul s along he p ocess leading
o i s elabo a ion. I u he decla e ha I ha e ully acknowledge he Rules o Conduc and Code o
Hono om he NOVA In o ma ion Managemen School.
Lisbon, Feb ua y 2023
iii
DEDICATION
To my lo ing b o he Rodge s, R.I.P.

i
ACKNOWLEDGEMENTS
I wish o hank God o his g ace and s eng h while wo king on his esea ch wi hou which I wouldn’
be able o comple e success ully. I’m also e y g a e ul o my supe iso LV who encou aged me and
guided me h ough he igh di ec ion.
ABSTRACT
In ecen yea s machine lea ning has made g ea s ides in many applica ion a eas and an e e -g owing
numbe o disciplines ely on i . Howe e , machine lea ning modelling p ocess in ol es ying many
machine lea ning algo i hms wi h di e en pa ame e con igu a ions which is conside ed insu icien ,
edious, and ime-consuming. The challenge has b ough abou he need o o - he-shel solu ions
ha allow a da ase o choose i s bes modelling pipeline including da a p ep ocessing, model selec ion
and hype pa ame e op imiza ion wi hou o wi h e y li le human in e en ion in he p ocess.
Despi e he a ailabili y o nume ous Au oML sys ems ha can au oma e he machine lea ning
modeling p ocess, he e is s ill a need o a solu ion ha can achie e he same esul s using a
signi ican ly smalle space, while imp o ing e iciency. This hesis p oposes an Au oML sys em named
EasyML ha uses me a-lea ning o model selec ion and pa icle swa m op imiza ion o
hype pa ame e op imiza ion. The esea ch objec i es include conduc ing a comp ehensi e li e a u e
e iew on S a e-o - he-A echniques and exis ing Au oML sys ems, design, and de elopmen o
EasyML, e alua ing he sys em's pe o mance on benchma k da ase s, compa ing i s e iciency o
o he Au oML sys ems, and iden i ying i s limi a ions and sugges ing u u e esea ch di ec ions. The
esea ch me hodology combines Design Science Resea ch and CRISP-DM. EasyML ou pe o ms exis ing
solu ions like Sma ML and Au o-WEKA on all benchma k da ase s. EasyML has he po en ial o
con ibu e o he de elopmen o mo e e icien and e ec i e Au oML sys ems, he eby mee ing he
inc easing demand o da a scien is s wi h s ong knowledge o a ious machine lea ning algo i hms
and echniques.
KEYWORDS
Au oma ed Machine Lea ning; Me a Lea ning; Pa icle Swa m Op imiza ion; Hype pa ame e
Op imiza ion; OpenML
i
INDEX
1. In oduc ion ................................................................................................................ 1
1.1. P oblem De ini ion ............................................................................................... 1
1.2. Resea ch Objec i es ............................................................................................. 2
1.3. Resea ch Me hodology ........................................................................................ 2
1.4. Achie ed Resul s .................................................................................................. 4
1.5. Documen S uc u e ............................................................................................. 4
2. S a e-o - he-a ........................................................................................................... 5
2.1. T adi ional Machine Lea ning .............................................................................. 5
2.2. Au oma ed Machine Lea ning ............................................................................. 6
2.2.1. Au oML Techniques ...................................................................................... 6
2.3. Me a Lea ning ...................................................................................................... 7
2.3.1. Me a Lea ning Techniques ............................................................................ 8
2.3.2. Me a Lea ning F amewo k ............................................................................ 9
Knowledge-acquisi ion mode ................................................................................. 9
Ad iso y mode ...................................................................................................... 10
2.4. Hype pa ame e Op imiza ion ........................................................................... 11
2.5. Rela ed Wo k in Au oML .................................................................................... 12
2.5.1. Sma ML ...................................................................................................... 12
2.5.2. Au o-WEKA .................................................................................................. 12
2.5.3. TPOT ............................................................................................................ 13
3. Me hodology ............................................................................................................. 14
3.1. The Knowledge Base (KB) ................................................................................... 14
3.1.1. Da a collec ion ............................................................................................ 15
3.1.2. Da a analysis and p ep ocessing ................................................................. 17
3.2. The Ex ac o ...................................................................................................... 22
3.3. The Me a Lea ne ............................................................................................... 23
3.4. The Op imize ..................................................................................................... 24
4. Resul s and discussion ............................................................................................... 26
4.1. Benchma k Da a ................................................................................................. 26
4.1.1. Abalone da ase .......................................................................................... 26
4.1.2. Madelon da ase ......................................................................................... 27
4.1.3. Semeion da ase ......................................................................................... 28
4.1.4. Yeas da ase ............................................................................................... 28
ii
4.2. Compa ison o EasyML o o he Au oML Sys ems ............................................. 29
4.2.1. Pe o mance ............................................................................................... 29
4.2.2. Technologies ............................................................................................... 30
5. Conclusion ................................................................................................................. 31
6. Limi a ions and ecommenda ions o u u e wo ks ................................................ 32
7. Re e ences ................................................................................................................ 33
Appendix (op ional) ........................................................... E o ! Bookma k no de ined.
Annexes (op ional) ............................................................. E o ! Bookma k no de ined.
4
E alua ion
The ully unc ioning algo i hm unde wen es ing using ou benchma k da ase s o assess i s
pe o mance. Upon compa ison wi h some o he exis ing solu ions, i was ound ha he algo i hm
demons a ed e iciency and e ec i eness wi hin i s ield o applica ion. The algo i hm pe o med well
and was success ul in achie ing i s in ended goals when es ed agains s anda d da ase s commonly
used o benchma king.
1.4. ACHIEVED RESULTS
The hesis achie ed he goal o designing and de eloping an Au oML sys em, which can au oma ically
iden i y he bes model and op imize i s hype pa ame e s in an e icien and e ec i e manne ,
con ibu ing o he de elopmen o Au oML sys ems ha can mee he inc easing demand o da a
scien is s wi h s ong knowledge o a ious machine lea ning algo i hms and echniques. The sys em
u ilizes me a lea ning o model selec ion, le e aging p io knowledge ob ained om o e 600 di e se
da ase s and app oxima ely 8 million machine lea ning asks. To op imize he selec ed model, PSO is
u ilized. The sys em is cons uc ed on op o se en classi ie s and doesn' equi e p ep ocessing o he
da ase .
1.5. DOCUMENT STRUCTURE
The hesis is s uc u ed in o se en chap e s ha p o ide a comp ehensi e s udy o he p oposed
Au oML sys em. The In oduc ion chap e p o ides an o e iew o he esea ch p oblem, esea ch
objec i es, and esea ch signi icance. The S a e-o - he-A chap e e iews he cu en s a e-o - he-a
echniques and exis ing Au oML sys ems. The Me hodology chap e explains he esea ch
me hodology used o design and de elop he p oposed Au oML sys em. The Resul s and Discussion
chap e p esen s he expe imen al esul s and compa es he accu acy o he p oposed sys em wi h
exis ing Au oML amewo ks. The Conclusion chap e summa izes he esea ch indings and discusses
hei implica ions. The Limi a ions and Fu u e Wo k chap e iden i ies he limi a ions o he p oposed
sys em and sugges s a eas o u u e esea ch. The Re e ences chap e lis s all e e ences ci ed in he
hesis.

5
2. STATE-OF-THE-ART
2.1. TRADITIONAL MACHINE LEARNING
ML in ol es he use o algo i hms and s a is ical models o enable a compu e o imp o e i s
pe o mance on a speci ic ask h ough expe ience, wi hou explici p og amming (Simon, 2015).
Despi e his claim, machine lea ning s ill equi es conside able explici p og amming (Olson e al.,
2016). The e a e ou main s eps ha go in o building a ML model namely da a p ep ocessing, model
selec ion, hype pa ame e op imiza ion and model e alua ion.
Da a p ep ocessing is he p ocess o p epa ing da a o analysis by cleaning (Alasadi & Bhaya, 2017),
ans o ming, and o ganizing i . I ’s an impo an s ep in he machine lea ning p ocess because he
quali y and quan i y o he da a can a ec he pe o mance o he model signi ican ly. This p ocess
includes da a collec ion which in ol es ga he ing da a om a ious sou ces such as da abases o online
pla o ms; da a cleaning which in ol es iden i ying and co ec ing e o s o inconsis encies in he da a
like missing alues, duplica e en ies, o inco ec da a ypes; da a ans o ma ion which in ol es da a
modi ica ion o make i sui able o analysis including da a no maliza ion o encoding; and o ganizing
he da a in o a s uc u ed o ma ha is easy o use and analyze.
Once he da a is p ep ocessed, he nex s ep is o selec he app op ia e model o ain he da a. Model
selec ion is he p ocess o choosing he bes model o m a se o candida e models o a gi en ask.
The main goal is o build a model ha can make accu a e p edic ions on new, p e iously unseen da a.
The e a e se e al echniques ha can be used in model selec ion such as spli ing da a in o aining
and es se s, using c oss- alida ion and using a alida ion se o une he hype pa ame e s o each
model. This p ocess is highly i e a i e as he e isn’ an ML model ha pe o ms bes ac oss all asks
and he e o e he da a mus be es ed on di e en models manually un il he model wi h he bes
pe o mance is ound.
Once he model is selec ed, i ’s necessa y o ine une he model’s hype pa ame e s o achie e he bes
possible pe o mance. Hype pa ame e op imiza ion is he p ocess o inding he op imal alues o
he hype pa ame e s o a model. Hype pa ame e s a e he pa ame e s ha a e se be o e he model
is ained, and hey con ol he beha iou and pe o mance o he model. Examples o
hype pa ame e s include he lea ning a e, he numbe o laye s in a neu al ne wo k, o he
egula iza ion s eng h. Op imizing he hype pa ame e s o a model can signi ican ly imp o e i s
pe o mance, bu i can be ime-consuming and equi es a lo o ial and e o . The e a e se e al
app oaches o hype pa ame e op imiza ion, including manual uning, g id sea ch, and andom sea ch.
Model e alua ion is an impo an s ep in he machine lea ning p ocess, as i helps o iden i y a eas o
imp o emen and op imize he model o be e pe o mance. Model e alua ion is he p ocess o
e alua ing he pe o mance o a machine lea ning model on a gi en ask. This in ol es compa ing he
p edic ed ou pu s o he model wi h he ac ual ou pu s o he da ase and using a ious me ics and
echniques o measu e he accu acy and e ec i eness o he model. Some common me ics used in
model e alua ion include accu acy, p ecision, ecall, F1 sco e, and AUC (a ea unde he cu e).
6
T adi ionally, hese s eps ha e equi ed signi ican expe ise in ML and da a science. Howe e , Au oML
aims o au oma e as much o his p ocess as possible, allowing e en hose wi h limi ed ML knowledge
o build high-quali y models.
2.2. AUTOMATED MACHINE LEARNING
Au oma ed machine lea ning (Au oML) is a sub ield o a i icial in elligence (AI) and ML ha ocuses
on au oma ically selec ing and op imizing machine lea ning algo i hms o a gi en da ase and ask. I
is an eme ging ield ha aims o educe human e o , imp o e accu acy and pe o mance du ing he
p ocess o building machine lea ning models and p omo e he ep oducibili y and ai ness o scien i ic
s udies (Hu e e al., 2018). Au oML bene i s may be summa ized as ollows:
Au oma ion o he machine lea ning p ocess: Au oML allows use s o au oma e he p ocess o building,
es ing, and op imizing machine lea ning models, elimina ing he need o expe knowledge in
p og amming and da a science.
Imp o ed accu acy and pe o mance: Au oML algo i hms a e designed o au oma ically op imize
machine lea ning models o maximum accu acy and pe o mance. This can lead o be e esul s han
manually buil models.
Time and cos sa ings: Au oML can signi ican ly educe he ime and cos equi ed o build and
op imize machine lea ning models, as i emo es he need o manual coding and da a p epa a ion.
Inc eased e iciency: Au oML allows use s o build, es , and op imize mul iple quickly and easily
machine lea ning models, inc easing he e iciency o he machine lea ning p ocess.
G ea e accessibili y: Au oML makes machine lea ning mo e accessible o non-expe s, allowing
indi iduals wi h limi ed p og amming and da a science knowledge o le e age he powe o machine
lea ning in hei wo k.
2.2.1. Au oML Techniques
Au oML may be applied h ough di e en app oaches including:
i. Au oma ed da a p ep ocessing by au oma ing he p ocess o p epa ing da a asks such as
missing alues impu a ion, ou lie de ec ion and da a scaling.
ii. Au oma ed model selec ion by using algo i hms o sea ch h ough a space o possible models
and selec he one ha pe o ms he bes on a gi en ask.
iii. Ensemble lea ning which in ol es combining mul iple ML models o imp o e he o e all
pe o mance o he sys em.
7
i . Au oma ed hype pa ame e op imiza ion using algo i hms o sea ch h ough he space o
possible hype pa ame e alues and selec he ones ha esul in he bes model
pe o mance.
. Neu al a chi ec u e sea ch which in ol es using algo i hms o sea ch h ough he space o
possible neu al ne wo k a chi ec u es and selec he one ha pe o ms he bes on a gi en
ask.
i. Me a lea ning which uses knowledge om p io asks on new asks in a da a-d i en manne ;
and
ii. Full Au oML which in ol es using algo i hms o ully au oma e he en i e ML model building
p ocess, om da a p ep ocessing o model e alua ion.
Fo he scope o his esea ch, we will only discuss me a lea ning and hype pa ame e op imiza ion
as well as ela ed wo k done on he ield.
2.3. META LEARNING
One key echnique used in Au oML is me a lea ning, whose main goal is o de elop algo i hms ha
can adap o new asks and en i onmen s quickly using pas expe ience (Hu e e al., 2018) and
knowledge o in o m he lea ning p ocess. This allows he Au oML sys em o selec he mos
app op ia e model o a gi en p oblem based on i s pe o mance on simila asks in he pas .
In adi ional machine lea ning, he algo i hm is ypically p o ided wi h a la ge da ase and le o lea n
he pa e ns and ela ionships wi hin he da a on i s own. Howe e , in me a lea ning, he algo i hm is
o en p o ided wi h a smalle se o examples and mus quickly adap o a new ask using he
in o ma ion om hese examples.
I he Au oML sys em has p e iously seen a da ase wi h simila me a ea u es o he cu en one, i
can use he knowledge gained om ha expe ience o selec a model ha is likely o pe o m well on
he cu en ask. This app oach can sa e ime and esou ces by a oiding he need o ex ensi e ial-
and-e o in model selec ion.
Me a lea ning di e s om adi ional machine lea ning in he scope o he le el o adap a ion (Vilal a
e al., 2009). Me a lea ning models a e designed o lea n om a small amoun o da a while adi ional
ML models ypically equi e la ge amoun s o da a o be e ec i e. Ano he key di e ence is he abili y
o me a lea ning models o lea n om a wide a ie y o asks and da ase s whe eas adi ional ML
models a e specialized o a speci ic ask a hand.
E iciency is ano he ac o dis inguishes he wo models whe e Me a lea ning models can adap o
new asks mo e quickly and e icien ly han adi ional machine lea ning models, as hey a e able o
ans e knowledge lea ned om p e ious asks o new ones. Ano he di e ence is ha me a lea ning
models a e used in si ua ions whe e da a o asks change o e ime while adi ional ML models a e
used in s a ic da a.
8
2.3.1. Me a Lea ning Techniques
The e a e se e al app oaches ha can be aken in o accoun while building a me a leaning sys em.
(Vilal a e al., 2009) g ouped he app oaches in o 6 ca ego ies namely da ase cha ac e iza ion,
mapping da ase s o p edic i e models, lea ning om base-lea ne s, induc i e ans e and lea ning o
lea n, and dynamic-bias selec ion.
Da ase cha ac e iza ion may be de ined as a desc ip ion o he p ope ies o a da ase used o ain
an ML model o a pa icula ask. In o ma ion acqui ed in his echnique is ele an since high quali y
me a ea u es p o ide some in o ma ion o di e en ia e he pe o mance o a se o gi en lea ning
s a egies (Vilal a e al., 2009). The cha ac e iza ion can include in o ma ion abou he dis ibu ion o
he da a, he ypes o inpu s and ou pu s, he ela ionship be ween inpu s and ou pu s, and he
complexi y o he ask. Some o he da a ha can be ex ac ed h ough da ase cha ac e iza ion include
s a is ical and in o ma ion- heo e ic da a such as numbe o classes, numbe o ea u es, a io o
examples o ea u es, deg ee o co ela ion be ween ea u es and a ge concep , a e age class
en opy and class-condi ional en- opy, skewness, ku osis, and signal o noise a io.
Landma king and model-based cha ac e iza ion a e o he sou ces o da a in da ase cha ac e iza ion.
In landma king, simple lea ne s such as decision s umps o 1NN as used as models on a da ase and
he accu acy and e o a e o he models a e used o cha ac e ize he da ase while in model-based
cha ac e iza ion, a da ase is passed o a s anda d decision ee and he ee p ope ies such as
maximum ee dep h, numbe o nodes pe ea u e and ee imbalance a e used o cha ac e ize he
da ase .
A me a lea ning sys em has he abili y o mapping a da ase o a p edic i e model using a p ede ined
c i e ia such as accu acy, unning ime o s o age space. In his echnique, he e a e se e al
app oaches, bu he mos common ones a e lea ning a he me a-le el, mapping que y examples o
models and anking. Lea ning a he me a le el is when he lea ning sys em ma ches me a ea u es o
a new da ase o he exis ing ones usually con ained in a me a knowledge base whe e each da ase
has a a ge label associa ed wi h i and a p edic i e model is used o ma ch he new da ase om he
ones in he knowledge base by e u ning i s a ge label.
In mapping que y examples o models’ app oach, he me a lea ning sys ems uses he accu acy c i e ia
o selec models o exis in he knowledge base and once a new da ase en e s he sys em, K-nea es
neighbo app oach is used whe e k bes models o he da ase s mos simila o he new da ase s a e
p oposed. Ranking uses he same app oach as lea ning a he me a le el bu e u ns mo e han one
model o example i may e u n he i s o i h bes pe o ming models pe da ase . This echnique
is hough as mo e lexible and in o ma i e o use s.
O he echniques a e lea ning om base lea ne s whe e s acked gene aliza ion, boos ing, landma king
and me a-decision ees app oaches a e used, induc i e ans e whe e he new ask is no ma ched
o he exis ing da ase s in he KB bu a he he me a knowledge is inco po a ed in he new ask and
dynamic-bias selec ion which can be unde s ood as he sea ch o he igh hypo hesis space o
concep ep esen a ion as he lea ning sys em encoun e s new asks.
9
2.3.2. Me a Lea ning F amewo k
(Vilal a e al., 2009) p oposed a me a lea ning amewo k ha is di ided in o wo pa s namely
Knowledge-acquisi ion mode and ad iso y mode. In he Knowledge-acquisi ion mode, he sys em
collec s ele an da a o a ious machine lea ning asks om open-sou ce pla o ms such as OpenML
and Kaggle while ad iso y mode maps a new ask me ada a o he mos simila exis ing asks me ada a
using a simila i y measu e such as Euclidean dis ance
2
be ween he asks.
OpenML is a pla o m o sha ing machine lea ning da ase s, code, and expe imen s (Vanscho en e
al., 2014). I was de eloped o suppo esea che s, de elope s, and da a scien is s in he machine
lea ning communi y o easily collabo a e and sha e hei wo k. The pla o m allows use s o disco e
and use machine lea ning da ase s, algo i hms, and expe imen s (Feu e e al., 2021) sha ed by o he
use s, as well as o upload and sha e hei own da ase s, code, and expe imen s. I also p o ides ools
o acking and compa ing he pe o mance o di e en machine lea ning models on di e en
da ase s, and o inding he mos e ec i e models and echniques o a pa icula ask.
Knowledge-acquisi ion mode
(Vanscho en, 2018) explo es h ee app oaches me a ea u es can be used in he lea ning p ocess i.e.,
model e alua ions, ask ea u es and model con igu a ions ex ac ed om he asks pe o med on
di e en da ase s. The main model e alua ions me a ea u es a e pipeline and model con igu a ion,
p edic i e accu acy sco e, a ea unde he cu e and un ime. This p ocess is use ul o con igu a ion
ecommenda ions and un ime p edic ions.
Tasks me a ea u es may be used o ind simila i y be ween he exis ing asks and he new ask o he
pu pose o model o pipeline ecommenda ions. This app oach may also be used o ind he ele ance
o a ea u e o a ask’s pe o mance o e en p edic a model’s pe o mance. (Vanscho en, 2018) g oups
asks me a ea u es ha can be ex ac ed om a da ase in o 6 ca ego ies namely simple, s a is ical,
in o ma ion- heo e ic, complexi y, model-based and landma ks. The ca ego ies a e explained in de ail
on Table 2.1.
Model con igu a ion-based ea u es a e mos ly used o neu al ne wo k aining whe e echniques
such as ans e lea ning and ew-sho lea ning a e applied. In his app oach me ada a comes om he
machine lea ning models s uc u es and pa ame e s in such a way ha a me a lea ne L ha lea ns
how o ain a (base) lea ne lnew o a new ask new, gi en simila asks j ∈ T and he co esponding
op imized models lj ∈ L whe e L is he space o all possible models. The lea ne lj is ypically de ined by
i s model pa ame e s W = {wk}, k = 1...K and/o i s con igu a ion qi Î q.
Table 2.1 – Task me a ea u es and hei measu es.
Ca ego y
Fea u e
Measu e
2
Euclidean dis ance is he squa ed dis ance be ween poin s (Dokmanic e al., 2015)

10
Simple ea u es
Da ase ’s dimensions
( ows & columns)
Ta ge class dis ibu ion
Missing alues
Nume ic and ca ego ical
ea u es
Ou lie s
Speed & scalabili y
Cu se o dimensionali y
Complexi y
Da a impu a ion
S a is ical ea u es
Da a skewness
Ku osis
Fea u es co ela ion
Co a iance
Spa si y
PCA
Class p obabili y
Fea u e no mali y
Deg ee o disc e eness
Fea u e dependence
Class dis ibu ion
In o ma ion- heo e ic
ea u es
Class en opy
No mali y en opy
Mu ual in o ma ion
Unce ain y coe icien
Noise-signal a io
Ta ge class imbalance
Fea u e in o ma i eness
Fea u e impo ance
Noisiness o da a
Complexi y ea u es
Fishe ’s disc imina ion
Concep a ia ion
Da a consis ency
Class sepa abili y
Task complexi y
Da a quali y
Model-based ea u es
Numbe o nodes and
lea es
B anch leng h pe class
In o ma ion gain
Task complexi y
Fea u e impo ance
Ta ge class complexi y
Sepa abili y
Landma ke s ea u es
1NN
DecisionS ump
RandomT ee
Nai eBayes
Accu acy
E o a e
Da a spa si y
Sepa abili y
P obing pe o mance
Ad iso y mode
In ad iso y mode, knowledge base da a acqui ed in he acquisi ion mode is ma ched wi h he me a
ea u es o he new da ase by a me a lea ne o p oduce a ecommenda ion ega ding he bes
a ailable lea ning s a egy. A me a lea ne may be de ined as an algo i hm ha is ained how o lea n
om da a o he pu pose o selec ing machine lea ning models o a gi en ask.
The me a lea ne is ained on a la ge da ase o machine lea ning asks, whe e each ask is de ined by
a se o me a ea u es as inpu da a and a a ge ou pu ha can be a machine lea ning model, pipeline
con igu a ion, un ime e c.
11
A ained me a lea ne can be used o au oma e he p ocess o selec ing and aining machine lea ning
models o new asks. To do his, he me a lea ne is gi en he inpu da a and a ge ou pu o he
new ask, and i uses i s lea ned knowledge o how o selec and ain machine lea ning models o
selec he app op ia e model and ain i on he new da a.
The goal o using a me a lea ne in Au oML is o au oma e he p ocess o selec ing and aining
machine lea ning models, which can be ime-consuming and equi e a lo o domain expe ise. By using
a me a lea ne , he p ocess o selec ing and aining machine lea ning models can be au oma ed,
allowing use s o ocus on o he aspec s o hei machine lea ning p ojec .
2.4. HYPERPARAMETER OPTIMIZATION
Hype pa ame e op imiza ion (HPO) is a c i ical s ep in he de elopmen o machine lea ning models,
as he selec ion o hype pa ame e s can g ea ly impac he pe o mance o he model. The e a e
se e al app oaches o HPO, bu he mos popula ones a e g id sea ch, andom sea ch, Bayesian
op imiza ion, g adien -based op imiza ion, hype band, and e olu iona y op imiza ion.
E olu iona y op imiza ion uses popula ion-based me hods, such as gene ic algo i hms, e olu iona y
algo i hms, e olu iona y s a egies, and pa icle swa m op imiza ion. These a e op imiza ion
algo i hms ha main ain a popula ion, i.e., a se o con igu a ions, and imp o e his popula ion by
mu a ion and/o c osso e o ob ain a new gene a ion o be e con igu a ions. These me hods a e
concep ually simple, can handle di e en da a ypes, and a e ex emely pa allel since a popula ion o
N membe s can be e alua ed in pa allel on N machines (Hu e e al., 2018).
Pa icle swa m op imiza ion (PSO) is a me aheu is ic op imiza ion algo i hm inspi ed by he beha io
o swa ms in na u e. The algo i hm simula es animals’ social beha io , including insec s, he ds, bi ds,
and ishes (Wang e al., 2018). Each pa icle in he popula ion ep esen s di e en combina ions o
hype pa ame e alues o he machine lea ning model. The i ness o he pa icle is e alua ed by
aining he model wi h he co esponding hype pa ame e s and e alua ing i s pe o mance on a
alida ion se . The pe sonal bes and global bes solu ions a e upda ed as he pa icles explo e he
sea ch space.
This app oach allows he Au oML sys em o explo e he as space o possible hype pa ame e s quickly
and e icien ly and ind he bes se ings o a gi en p oblem. This can imp o e he pe o mance o ML
models and educe he need o manual uning o he hype pa ame e s.
The main ad an ages o he PSO algo i hm a e summa ized as: simple concep , easy implemen a ion,
obus ness o con ol pa ame e s and compu a ional e iciency when compa ed wi h ma h
ma hema ical algo i hms and o he heu is ic op imiza ion echniques (Lee & Pa k, 2006). This is
because i is a popula ion-based algo i hm, which means i can explo e he sea ch space mo e
e ec i ely han a single-pa icle algo i hm. This allows PSO o ind he op imal solu ion in a mo e
e icien and eliable way.
To use PSO o hype pa ame e op imiza ion, he sea ch space is de ined based on he ange o alues
ha he hype pa ame e s can ake. The ini ial popula ion o pa icles is hen gene a ed andomly
12
wi hin his sea ch space. The pa icles a e hen upda ed i e a i ely acco ding o he PSO ules, which
include a pe sonal bes and global bes upda e ule.
A each i e a ion, each pa icle is upda ed based on i s pe sonal and global bes posi ions and i s cu en
eloci y. The pe sonal bes posi ion is he bes solu ion ha he pa icle has ound so a while he global
bes posi ion is he bes solu ion ound by any pa icle in he popula ion. The pa icle’s eloci y is
upda ed acco ding o a se o equa ions designed o s ee he pa icle owa ds he global bes posi ion.
This p ocess con inues un il he algo i hm con e ges o a sa is ac o y solu ion, o un il a p ede e mined
numbe o i e a ions is eached.
2.5. RELATED WORK IN AUTOML
The e a e se e al Au oML sys ems in exis ence bu o he scope o his s udy, h ee di e se sys ems
ha use di e en app oaches a e s udied and analyzed.
2.5.1. Sma ML
Sma ML may be desc ibed as a me a lea ning-based amewo k o au oma ed selec ion and
hype pa ame e uning o machine lea ning algo i hms (Mahe & Sak , 2019) using me a lea ning
echnique o model selec ion and sequen ial model-based algo i hm con igu a ion (SMAC) o
op imiza ion. Sma ML a chi ec u e (Mahe & Sak , 2019) consis s o 5 phases namely inpu de ini ion
phase, da a p e-p ocessing phase, algo i hm selec ion phase, pa ame e uning and ou pu
compu a ion and knowledge base upda ing.
In he inpu phase he use uploads a da ase , chooses he ea u e p e-p ocessing echniques, and
indica es he a ge ea u e o he da ase . The selec ed p e-p ocessing echniques a e hen applied
on he da ase , he da a is andomly spli in ain and es da a and me a ea u e compu a ion is
pe o med on he p e-p ocessing phase. In he algo i hm selec ion phase, he compu ed me a ea u es
a e compa ed wi h he me a ea u es in he knowledge base using a nea es neighbou app oach hen
ollowed by measu ing Euclidean dis ance be ween he new da ase and exis ing da ase s ollowed by
selec ing n algo i hms based on hei pe o mance. The selec ed algo i hms a e he op imized in he
ollowing phase whe e he o iginal con igu a ions om he selec ed algo i hms a e used as a s a ing
poin . The esul s om he op imiza ion p ocess a e hen compa ed and he bes model con igu a ion
is e u ned o he use .
Sma ML con ains 15 machine lea ning classi ie s and can be used as a package in R o as a web
applica ion.
2.5.2. Au o-WEKA
Au o-WEKA is an Au oML sys em ha au oma ically and simul aneously chooses a lea ning algo i hm
and se s i s hype pa ame e s o op imize pe o mance using SMAC and ee s uc u ed pa zen
es ima o (TPE). The ool was i s ly c ea ed in 2013 o he pu pose o sol ing he combined algo i hm
selec ion and hype pa ame e op imiza ion (CASH) p oblem (Ko ho e al., 2017). Au o-WEKA
13
inco po a es ea u e selec ion echniques and all machine lea ning app oaches implemen ed in
WEKA’s s anda d dis ibu ion, spanning 2 ensemble me hods, 10 me a-me hods, 28 base lea ne s, and
hype pa ame e se ings o each lea ne .
The p ocess begins wi h p e-p ocessing he inpu da a, which includes asks such as handling missing
alues, no malizing he da a, and ea u e selec ion. WEKA classi ie s a e hen applied o he p e-
p ocessed da a using di e en hype pa ame e se ings. These me hods a e hen e alua ed using a
pe o mance me ic, such as accu acy o F1-sco e, and he bes -pe o ming combina ion o classi ie
and p e-p ocessing me hod is selec ed o he inal model.
Au o-WEKA can be un using he command line, GUI, o h ough a p og amma ic in e ace, and
suppo s se e al pe o mance me ics, sea ch s a egies and esampling me hods o e alua ing he
pe o mance o he models.
2.5.3. TPOT
(Olson e al., 2016) de ine T ee-based Pipeline Op imiza ion Tool (TPOT) as a gene ic p og amming
based Au oML sys em ha op imizes a se ies o ea u e p e-p ocesso s and machine lea ning models
wi h he goal o maximizing classi ica ion accu acy on a supe ised classi ica ion ask. TPOT is a
w appe o he Py hon machine lea ning package, sciki -lea n and uses 3 g oups o ope a o s namely
ea u e selec ion ope a o s, ea u e p e-p ocessing ope a o s and supe ised classi ica ion ope a o s.
Fea u e selec ion ope a o s consis s o Va ianceTh eshold, Selec KBes , Selec Pe cen ile, Selec Fwe,
and Recu si e Fea u e Elimina ion (RFE) while ea u e p e-p ocessing ope a o s a e S anda dScale ,
Robus Scale , MinMaxScale , MaxAbsScale , RandomizedPCA, Bina ize , and PolynomialFea u es and
supe ised classi ica ion ope a o s consis o DecisionT ee, RandomFo es , eX eme G adien Boos ing
Classi ie , Logis icReg ession, and KNea es Neighbo Classi ie . These ope a o s a e andomly
combined using gene ic p og amming and es ed on a da ase and he combina ion o pipeline wi h
he bes pe o mance is e u ned.
21
Table 3.1 – Me a Fea u es in KB
Ca ego y
Fea u e
Basic ea u es
'Numbe O Ins ances', 'Numbe O Fea u es', 'Numbe O Classes',
'Numbe O Bina yFea u es', 'Pe cen ageO Bina yFea u es',
'Numbe O SymbolicFea u es','Numbe O Nume icFea u es',
'Pe cen ageO Nume icFea u es','Pe cen ageO SymbolicFea u es',
'Numbe O MissingValues’, ‘Numbe O Ins ancesWi hMissingValues',
'Pe cen ageO MissingValues', 'MaxNominalA Dis inc Values',
'Pe cen ageO Ins ancesWi hMissingValues','Mino i yClassPe cen age',
'Majo i yClassPe cen age', 'Majo i yClassSize', 'Mino i yClassSize'
Complex ea u es
'MaxKu osisO Nume icA s', 'MinKu osisO Nume icA s',
'MeanKu osisO Nume icA s', 'Qua ile1Ku osisO Nume icA s',
'Qua ile2Ku osisO Nume icA s', 'Qua ile3Ku osisO Nume icA s',
'MaxSkewnessO Nume icA s', 'MinSkewnessO Nume icA s',
'MeanSkewnessO Nume icA s', 'Qua ile3MeansO Nume icA s'
'Qua ile2SkewnessO Nume icA s', 'Qua ile2MeansO Nume icA s',
'Qua ile3SkewnessO Nume icA s','Qua ile1SkewnessO Nume icA s',
'MaxS dDe O Nume icA s', 'MinS dDe O Nume icA s',
'MeanS dDe O Nume icA s', 'Qua ile1S dDe O Nume icA s',
'Qua ile2S dDe O Nume icA s', 'Qua ile3S dDe O Nume icA s',
'Qua ile1MeansO Nume icA s', 'MaxMeansO Nume icA s',
'MinMeansO Nume icA s', 'MeanMeansO Nume icA s', ‘ClassEn opy’,
'Nai eBayesE Ra e', 'Nai eBayesKappa'
'J48_00001_E Ra e', 'J48_00001_Kappa', 'J48_0001_E Ra e',
'J48_0001_Kappa', 'J48_001_E Ra e', 'J48_001_Kappa',
'REPT eeDep h1E Ra e', 'REPT eeDep h1Kappa', 'kNN1NE Ra e',
'kNN1NKappa','REPT eeDep h2E Ra e', 'REPT eeDep h2Kappa',
'REPT eeDep h3E Ra e', 'REPT eeDep h3Kappa',
'RandomT eeDep h1E Ra e', 'RandomT eeDep h1Kappa',
'RandomT eeDep h2E Ra e', 'RandomT eeDep h2Kappa',
'RandomT eeDep h3E Ra e', 'RandomT eeDep h3Kappa'

22
3.2. THE EXTRACTOR
A new da ase en e s he sys em o he pu pose o inding he model and hype pa ame e s ha will
gi e bes p edic i e esul s based on i s me a ea u es al hough i doesn’ ha e he ea u es ye . The
ex ac o is pa o he Au oML algo i hm ha akes as inpu he new da ase and calcula es 66 basic
and complex ea u es o ma ch he exis ing me a ea u es (see Table 2).
Figu e 3.14 – The Ex ac o
Figu e 3.15 is he one o he componen s o he main algo i hm ha ex ac s he landma king me a
ea u es. Weka classi ie s we e mainly used as landma ke s by PyWeka lib a y whe e he p og am
would ake a da ase , models and hype pa ame e s as an inpu and e alua ions, e o a e and Kappa
sco es we e e u ned as da ase s me a ea u es.
Figu e 3.15 – Landma ke s componen
A new da ase
in o m o CSV
is sen o he
ea u e
ex ac o
Nai e Bayes, 1NN,
Decision S ump,
Random T ees, Rep
T ee, J48
Desc ip i e
da a
S a is ical
da a
New da a
23
3.3. THE META LEARNER
Gi en 680 asks i, each wi h 66 me a ea u es
𝒇𝒊
𝒋
and a a ge ea u e con aining he ask’s bes
pe o ming model
b
, he main objec i e o he me a lea ne is o selec
b
ha may pe o m bes on
he ask new o he new da ase dnew. This is done by inding he ask ha is mos simila o he
new ask by hei me a ea u es using he Euclidean dis ance
E
as he simila i y unc ion, he mos
simila ask
s
in he knowledge base
K
such ha :
"
i
ÎK
E
(
s
, new)
£
E
( i, new)
whe e:
K
= { i, i = 1, 2, ..., 680}
E(s, new) =
"#(
s, new
)
2
n
i=1
b
con ains se en machine lea ning classi ie s whe e 5 o hem a e om Sciki Lea n lib a y, one om
XGBOOST and one GLMNET.
Table 3.2 – Classi ie s sea ch space
Classi ie
Lib a y
DecisionT eeClassi ie
Sciki Lea n
GLMNET
Glmne Vigne e
G adien Boos ingClassi ie
Sciki Lea n
Nai eBayesClassi ie
Sciki Lea n
RandomFo es Classi ie
Sciki Lea n
SVC
Sciki Lea n
XGBoos
XGBoos
24
3.4. THE OPTIMIZER
The op imize uses he global bes (Gbes ) PSO (Talukde , 2011) whe e he posi ion o each pa icle is
in luenced by he bes pa icle in he en i e popula ion. This app oach leads o as e con e gence as
each pa icle connec s wi h e e y o he pa icle. The model selec ed by he me a lea ne (
b
) is passed
o he op imize whe e i ’s hype pa ame e s a e uned and modi ied using PSO o ob ain he maximum
accu acy sco e.
The model
b
ep esen s a pa icle in he popula ion which con ains a ec o o h ee con inuous
hype pa ame e s
C
ha a e dynamic and a ec o o disc e e hype pa ame e s
V
ha a e s a ic as
hey s ay cons an h oughou he op imiza ion p ocess while
C
mu a es. In his s udy, ec o
C
ep esen s he posi ion o he pa icle
b
in he swa m. The op imize consis s o wo main aspec s
namely he sea ch space and he i ness unc ion which will be discussed in he ollowing sec ion.
The op imize ini ializes a popula ion o n pa icles andomly whe e each pa icle has a cu en posi ion
in sea ch space Xi, a cu en eloci y Vi and a pe sonal bes posi ion Pbes ,i whe e i
Î
[1, 2,...n]. The Pbes
co esponds o model
b
highes p edic i e accu acy when es ed on he new da ase dnew using he
i ness unc ion
ƒ
. The posi ion e u ning he highes accu acy among all Pbes , i becomes he Gbes .
The algo i hm upda es he posi ions o all pa icles depending on how he use se s he pa ame e s.
By de aul , EasyML c ea es 5 gene a ions o he popula ion o 5 pa icles. The numbe is se low o
ensu e he algo i hm’s e iciency. A he end o he algo i hm, he posi ion ha e u ned he highes
accu acy along wi h he accu acy a e e u ned by he op imize . The posi ions a e upda ed by al e ing
he pa icles’ eloci y using he ollowing o mula:
epea o 5 gene a ions
o each i = 1, 2, 3, 4, 5 do
Xi := Xi + Vi ;
Vi := w Vi + c1 1 * (Pbes ,i - Xi) + c2 2 * (Gbes - Xi) ;
i (
ƒ
(
b
(Xi)) > (
ƒ
(
b
(Pbes ,i)) hen Pbes ,i := Xi ;
i (
ƒ
(
b
(Xi)) > (
ƒ
(
b
(Gbes )) hen Gbes := Xi ;
end
end
The ine ia weigh w is picked andomly be ween he lowe bound o 0.4 and highe bound o 0.9, he
cogni i e and social componen s (c1 and c2 espec i ely) a e se cons an ly a 1 while 1 and 2 a e
picked andomly be ween 0 and 0.9.
This pa o he algo i hm pe o ms wo main asks namely da a p ep ocessing and p edic i e analy ics
on he new da ase dnew. Fo e iciency pu poses, da a p ep ocessing echniques a e limi ed o
emo ing eco ds wi h missing alues, spli ing he da a o ain and es da ase s, and s anda dizing
he da a. andom_s a e hype pa ame e o he ain_ es _spli me hod is one o he h ee
25
pa ame e s ha a e dynamically al e ed by he algo i hm. Fo da a s anda diza ion, MinMaxScale ()
and S anda dScale () a e andomly selec ed. dnew is hen ained by one o he selec ed 7 machine
lea ning models and he es accu acy sco e is e u ned. Fo he ee-based algo i hm, he o he wo
hype pa ame e s ep esen ed as pa icle’s posi ion a e max_dep h and andom_s a e while o SVC
a e cache_size and andom_s a e.
26
4. RESULTS AND DISCUSSION
This chap e discusses he ob ained esul s on he pe o mance o ou de eloped Au oML algo i hm
and i s compa ison o Sma ML, Au o-WEKA and TPOT in e ms o echniques and pe o mance. The
es s we e pe o med wi h espec o e iciency by exploi ing a small sea ch space bo h by using se en
ML models and op imizing as ew hype pa ame e s as possible while e u ning be e esul s han
s a e-o - he a exis ing solu ions.
The pe o mance was es ed on en di e en con igu a ions in o de o unde s and i ’s beha io and
wha con igu a ions would wo k be e based on he da ase dimensions like he numbe o eco ds,
numbe o ea u es and numbe o classes i con ains. We es ed he algo i hm by se ing he pa icles
popula ion and gene a ions o 5 o 10 and maximum posi ion o 10 o 20 o 50. The con igu a ions
we e combined sys ema ically, and he esul s we e eco ded and discussed below.
4.1. BENCHMARK DATA
We conduc ed expe imen s on ou benchma k da ase s o es ablish ha he de eloped Au oML
algo i hm EasyML can ind accu a e models on a ange o da ase s in an e icien way. Resul s we e
collec ed on expe imen s pe o med on ou da ase s namely, Abalone, Madelon, Semeion and Yeas .
4.1.1. Abalone da ase
ABALONE is a well-known da ase o 9 con inuous ea u es and 4177 ins ances wi h no missing alues
whe e he goal is o p edic abalones’ sex whe he i s M, F, o I (in an ). The da ase is modi ied da a
om esea ch on popula ion biology o abalones (1994), e ie ed om UCI Machine Lea ning
Reposi o y.
The selec ed model o he Abalone da ase was Suppo Vec o Machine whe e he op imized
hype pa ame e s we e cache_size and andom_s a e. Fo p e-p ocessing, he da ase was spli 70/30
o ain/ es da a and he andom_s a e pa ame e was op imized. The bes accu acy sco e was 59
when cache_size was 42, andom_s a e o he model was 22 and andom_s a e o he
ain_ es _spli was 26.
Table 4.1 – Abalone con igu a ions and pe o mance
Scale
Popula ion
Gene a ions
Max pos
Bes Pos
Max sco e
S anda dScale ()
10
10
50
[42,22,26]
59
S anda dScale ()
10
10
20
[6,13,15]
59
MinMaxScale ()
10
5
20
[1,11,15]
59
S anda dScale ()
10
5
50
[11,35,42]
58
MinMaxScale ()
5
5
50
[39,7,34]
58

27
S anda dScale ()
10
10
10
[6,9,8]
57
S anda dScale ()
10
5
10
[4,2,8]
57
MinMaxScale ()
5
10
20
[41,18,27]
57
S anda dScale ()
5
5
20
[7,4,18]
56
MinMaxScale ()
5
5
10
[2,4,10]
56
4.1.2. Madelon da ase
MADELON is an a i icial da ase ha is a wo-class classi ica ion p oblem wi h 500 con inuous inpu
a iables and 2600 ins ances. The da ase is a wo-class classi ica ion p oblem, and i s bigges
challenge is ha i ’s mul i a ia e and highly non-linea . I was also e ie ed om UCI Machine
Lea ning Reposi o y and was pa o he NIPS 2003 ea u e selec ion challenge.
The selec ed model o he Madelon da ase was Random Fo es whe e he op imized
hype pa ame e s we e max_dep h and andom_s a e. Fo p e-p ocessing, he da ase was spli 70/30
o ain/ es da a and he andom_s a e pa ame e was op imized. The bes accu acy sco e was 75
when max_dep h was 11, andom_s a e o he model was 10 and andom_s a e o he
ain_ es _spli was 22.
Table 4.2 – Madelon con igu a ions and pe o mance
Scale
Popula ion
Gene a ions
Max pos
Bes Pos
Max sco e
S anda dScale ()
10
5
20
[11,10,22]
75
MinMaxScale ()
5
5
50
[42,33,23]
75
MinMaxScale ()
10
10
50
[16,25,3]
74
S anda dScale ()
10
5
50
[34,8,3]
73
MinMaxScale ()
10
10
20
[10,23,23]
73
S anda dScale ()
5
5
20
[18,12,9]
72
S anda dScale ()
5
10
20
[12,11,15]
72
S anda dScale ()
5
5
10
[9,1,2]
72
MinMaxScale ()
10
10
10
[9,7,6]
72
S anda dScale ()
10
5
10
[4,5,9]
69
28
4.1.3. Semeion da ase
SEMEION is a mul i-class classi ica ion ask o a o al o 1593 handw i en digi s om a ound 80 pe sons
we e scanned, s e ched in a ec angula box 16x16 in a g ay scale o 256 alues. Then each pixel o
each image was scaled in o a Boolean (1/0) alue using a ixed h eshold. The da ase con ains 1593
eco ds, 257 Boolean ea u es and 10 a ge classes o p edic .
The selec ed model o he Semeion da ase was Suppo Vec o Machine whe e he op imized
hype pa ame e s we e cache_size and andom_s a e. Fo p e-p ocessing, he da ase was spli 70/30
o ain/ es da a and he andom_s a e pa ame e was op imized. The bes accu acy sco e was 97
when cache_size was 6, andom_s a e o he model was 4 and andom_s a e o he ain_ es _spli
was 3.
Table 4.3 – Semeion con igu a ions and pe o mance
Scale
Popula ion
Gene a ions
Max pos
Bes Pos
Max sco e
MinMaxScale ()
10
5
20
[6,4,3]
97
S anda dScale ()
5
10
20
[10,2,9]
97
MinMaxScale ()
10
5
10
[5,3,8]
97
S anda dScale ()
5
5
10
[5,10,9]
97
MinMaxScale ()
10
10
10
[10,1,9]
97
S anda dScale ()
5
5
50
[46,39,25]
97
S anda dScale ()
10
5
50
[21,24,9]
97
S anda dScale ()
10
10
50
[42,19,9]
97
S anda dScale ()
10
10
20
[4,7,9]
97
MinMaxScale ()
5
5
20
[5,1,17]
96
4.1.4. Yeas da ase
Yeas da ase consis s o a p o ein-p o ein in e ac ion ne wo k. The da ase has 1484 eco ds, 8
ea u es and 10 a ge classes which makes i a mul i a ia e p oblem. I is o p edic he cellula
localiza ion si es o p o eins and was dona ed o UCI eposi o y in 1996.
The selec ed model o he Yeas da ase was Random Fo es whe e he op imized hype pa ame e s
we e max_dep h and andom_s a e. Fo p e-p ocessing, he da ase was spli 70/30 o ain/ es da a
and he andom_s a e pa ame e was op imized. The bes accu acy sco e was 68 when max_dep h
was 13, andom_s a e o he model was 13 and andom_s a e o he ain_ es _spli was 13.
29
Table 4.4 – Yeas con igu a ions and esul s
Scale
Popula ion
Gene a ions
Max pos
Bes Pos
Max sco e
MinMaxScale ()
10
5
20
[13,13,13]
68
MinMaxScale ()
5
10
20
[14,39,13]
67
MinMaxScale ()
5
5
20
[27,39,13]
67
S anda dScale ()
10
5
10
[20,19,13]
66
S anda dScale ()
5
5
50
[28,52,13]
66
S anda dScale ()
10
5
50
[28,60,13]
65
S anda dScale ()
10
10
50
[44,47,13]
65
MinMaxScale ()
10
10
20
[16,16,13]
65
S anda dScale ()
10
10
10
[6,4,9]
64
S anda dScale ()
5
5
10
[7,5,9]
62
4.2. COMPARISON OF EASYML TO OTHER AUTOML SYSTEMS
4.2.1. Pe o mance
The esul s o he pe o mance e alua ion we e compa ed o hose o o he Au oML sys ems, namely
Au o-WEKA and Sma ML, which had also been es ed on he same benchma k da ase s. Fo he
Abalone da ase , EasyML pe o med signi ican ly be e whe e i was able o p edic wi h 59% on he
es da a while Au o-Weka and Sma ML had he sco e o 25% and 27% simul aneously.
Fo he Madelon da ase , EasyML pe o med sligh ly be e han Sma ML whe e he sys ems’ sco es
we e 75% and 74% espec i ely. Au o-WEKA ga e he p edic i e sco e o 56%. EasyML pe o med
be e han bo h sys ems whe e i had he accu acy o 97% while Sma ML pe o med a 94% and
Au o-Weka a 89%. EasyML also pe o med bes when es ed on he Yeas da ase by p edic ing he
a ge class by 68% while Sma ML p edic ed a 66% and Au o-WEKA a 52%.
The able below summa izes he dimensions o he benchma k da ase s used and pe o mance on he
h ee Au oML sys ems.
Table 4.5 - EasyML pe o mance compa ing o Au o-Weka and Sma ML
Da ase
# Fea u es
# Classes
# Ins ances
Au o-Weka
Accu acy
Sma ML
Accu acy
EasyML
Accu acy
abalone
9
3
8192
25
27
59
madelon
500
2
2600
56
74
75
semeion
256
10
1593
89
94
97
yeas
8
10
1484
52
66
68
30
4.2.2. Technologies
EasyML has se e al dependencies due o he echnologies used. Besides Py hon and i s popla lib a ies
such as Pandas and NumPy, he use mus ins all Ja a i ual machine which suppo s he Weka lib a y
and ins all OpenML API o ex ac he mos upda ed da a om he eposi o y. EasyML can be used
locally and cu en ly suppo s Py hon only.
In Table 4.6, a compa ison o ou di e en Au oML solu ions, including EasyML, TPOT, Au o-Weka,
and Sma ML, is p esen ed. The able displays in o ma ion on he p og amming language used by each
solu ion, he op imiza ion me hod, he numbe o classi ie s a ailable on op o each language, and
whe he he solu ions inco po a e me a-lea ning o ea u e p e-p ocessing. EasyML, which uses
Py hon as i s p og amming language, u ilizes Pa icle Swa m Op imiza ion o op imiza ion and o e s
se en classi ie s on op o Py hon.
On he o he hand, Sma ML, which is p og ammed in R, uses Bayesian op imiza ion (SMAC & TPE) and
o e s 15 classi ie s on op o R. Au o-Weka, p og ammed in Ja a, uses Bayesian op imiza ion (SMAC)
and o e s 15 classi ie s on op o WEKA. TPOT, which also uses Py hon, u ilizes Gene ic P og amming
o op imiza ion and o e s 27 classi ie s on op o Sklea n. Bo h EasyML and Sma ML inco po a e
me a-lea ning, while Au o-Weka and TPOT do no .
Table 4.6 EasyML echnologies compa ing o TPOT, Au o-Weka and Sma ML
EasyML
Sma ML
Au o-Weka
TPOT
P og amming
language
Py hon
R
Ja a
Py hon
Op imiza ion
me hod
Pa icle Swa m
Op imiza ion
Bayesian
(SMAC & TPE)
Bayesian (SMAC)
Gene ic
P og amming
Numbe o
classi ie s
7
On op o Py hon
15
On op o R
27
On op o WEKA
15
On op o Sklea n
Use me a
lea ning
Yes
Yes
No
No
Fea u e
p ep ocessing
No
Yes
No
No