Uni e sidade do Minho
Escola de Engenha ia
João Ped o Ba os de Sá
Au oma ion o Machine Lea ning
Models Benchma king
decembe , 2021
Uni e sidade do Minho
Escola de Engenha ia
João Ped o Ba os de Sá
Au oma ion o Machine Lea ning
Models Benchma king
Mas e Thesis
Mas e in In o ma ics Enginee ing
Wo k de eloped unde he supe ision o :
P o esso Doc o João Miguel Lobo Fe nandes
P o esso Doc o And é Lei e Fe ei a
decembe , 2021
COPYRIGHT AND TERMS OF USE OF THIS WORK BY A THIRD PARTY
This is academic wo k ha can be used by hi d pa ies as long as in e na ionally accep ed ules and good
p ac ices ega ding copy igh and ela ed igh s a e espec ed.
Acco dingly, his wo k may be used unde he license p o ided below.
I he use needs pe mission o make use o he wo k unde condi ions no p o ided o in he indica ed
licensing, hey should con ac he au ho h ough he Reposi o iUM o Uni e sidade do Minho.
License g an ed o he use s o his wo k
C ea i e Commons A ibu ion-NonComme cial-Sha eAlike 4.0 In e na ional
CC BY-NC-SA 4.0
h ps://c ea i ecommons.o g/licenses/by-nc-sa/4.0/deed.en
ii
Acknowledgemen s
This wo ds a e dedica ed o e e yone in ol ed in my academic jou ney so a , i ’s success is in pa hanks
o you.
To my pa en s, b o he and gi l iend o all he suppo p o ided du ing his long yea . I was hanks
o all o you I go he s eng h necessa y o o e come all he challenges, and inish ano he chap e in my
academic and pe sonal li e.
To P o esso João Miguel Lobo Fe nandes, hank you o p o iding all he suppo , guidance and
men o ship o e he yea o achie e his documen .
To my wo k supe iso , And é Fe ei a, hank you o being my i s p o essional leade and men o ,
you guidance and con idence in my wo k was c ucial owa ds he success o he wo k achie ed h oughou
he yea .
To he MlOps eam who oge he o med g ea momen s o iendship and collabo a ion, i was a
p i ilege o wo k wi h you and hank you o making his wo k possible in he end.
To Ped o Viei a, o being a g ea iend and gi ing he mos hones ad ice, always con ibu ing wi h
posi i e eedback leading o a highe quali y s anda d in he wo k pe o med.
To e e yone a Bosch, hank you o p o iding a g ea wo k en i onmen and con ibu ing o his wo k.
iii

STATEMENT OF INTEGRITY
I he eby decla e ha ing conduc ed his academic wo k wi h in eg i y. I con i m ha I ha e no used pla-
gia ism o any o m o undue use o in o ma ion o alsi ica ion o esul s along he p ocess leading o i s
elabo a ion.
I u he decla e ha I ha e ully acknowledged he Code o E hical Conduc o he Uni e sidade do
Minho.
i
Resumo
Au oma ion o Machine Lea ning Models Benchma king
Na á ea de ciência de dados, o machine lea ning es á-se a e ela uma e amen a essencial pa a esol e
p oblemas complexos. As emp esas es ão a in es i em equipas de ciência de dados e Machine Lea ning
pa a desen ol e modelos que ap esen em alo pa a os clien es. No en an o, es es modelos são uma
pequena pe cen agem de uma pipeline de p oje os de Machine Lea ning (ML) e, pa a en ega um p odu o
de ML comple o, é necessá io um núme o maio de componen es.
De Ops é uma men alidade de engenha ia e um conjun o de p á icas que isa uni ica o p ocesso
de desen ol imen o e o p ocesso de ope ações em um so wa e, MLOps é um concei o simila a De Ops
mas aplicado ao desen ol imen o e en ega de soluçoes de ML. O ní el de au oma ização das e apas em
uma pipeline de ML de ine a ma u idade do p ocesso de ML, que e le e a elocidade de eino de no os
modelos com no os dados ou de eino de no os modelos com di e en es implemen ações. Um sis ema
de ML é um sis ema de so wa e, desen ol imen o e a ualizações con inuas são necessá ias pa a ga an i
um sis ema que escale con o me as necessidades.
O p incipal obje i o des a ese é apoia a c iação de um sis ema in eg ado de ML com uma a qui e u a
que p opo cione a capacidade de se con inuamen e ope ada em um ambien e de p odução. Um concei o
pa a a aliação de desempenho de algo i mos de e se elabo ado e implemen ado. O p incipal obje i o é
melho a e acele a o ciclo de desen ol imen o de modelos de ML na emp esa.
Pa a a ingi es e obje i o su ge a necessidade de de ini uma a qui e u a com especi icações e a
implemen ação de p ocessos au oma izadas num pipeline de ML exis en e, es e p ocesso êm como
obje i o alcança uma e amen a de benchma k de modelos, com capacidade de analisa o desempenho
do modelo, um mo o de in e ência e um banco de dados pa a a mazena odas as mé icas compu adas.
Um sis ema baseado em IA em desen ol imen o o nece o caso de es udo pa a desen ol e e ali-
da a a qui ec u a. Os a anços a uais na á ea da condução semiau omá ica in oduz a necessidade de
sis emas de moni o amen o que podem localiza e de ec a e en os especí icos no eículo. Os conjun os
de senso es são ins alados den o da cabine pa a alimen a sis emas in eligen es que isam analisa e
sinaliza ce os compo amen os que podem impac a a segu ança e o con o o dos passagei os.
Pala as-cha e: Engenha ia So wa e, Ap endizagem Máquina, Ciência dados, De Ops, MlOps …
Abs ac
Au oma ion o Machine Lea ning Models Benchma king
In he ield o da a science, ML is p o ing o be a co e ea u e o sol e complex eal-wo ld p oblems.
Businesses a e in es ing in da a science and ML eams o de elop AI based models ha can deli e
business alue o hei use s. Howe e , hese models a e only a small ac ion o an ML p ojec pipeline,
and o deli e an end o end ML p oduc , a g ea e numbe o componen s a e needed.
De Ops is an enginee ing mindse and a se o p ac ices ha aims o uni y he de elopmen p ocess
and he ope a ion p ocess on so wa e. MlOps is a simila concep o De Ops bu applicable o he
de elopmen and deli e y o ML based solu ions. The au oma ion o he s eps in a ML pipeline de ines he
ma u i y o he ML p ocess, e lec ing he eloci y o aining new models gi en new da a o aining new
models gi en new implemen a ions. An ML sys em is a so wa e sys em ha can suppo de elopmen ,
p o ide con inuous in eg a ion and con inuous deli e y apply o help gua an ee ha one can eliably build
and ope a e ML sys ems a scale.
The main objec i e o his hesis a e o suppo he c ea ion o an in eg a ed ML sys em wi h an a chi-
ec u e ha p o ides he abili y o be con inuously ope a ed in a p oduc ion-like en i onmen . Fu he mo e,
a concep o e alua e he pe o mance o algo i hms shall be de ised and implemen ed. The end goal is
o imp o e and accele a e he ML de elopmen li ecycle. To achie e his goal su ges he need o de ine an
a chi ec u e alongside speci ica ions and he implemen a ion o se e al au oma ed s eps in o an exis ing
ML pipeline. To imp o e and accele a e model de elopmen an model engine benchma k ool is de ised
capable o se e al ea u es, including he abili y o ha e dashboa ds o model pe o mance e alua ion, an
au oma ic in e ence engine, pe o mance me ics o he model and a da abase o s o e all he compu ed
me ics and me ada a.
An AI-based sys em unde de elopmen p o ides he case s udy o de elop and alida e his a chi ec-
u e. The cu en ad ances o semi-au oma ed d i ing in oduce he need o moni o ing sys ems o scan
and de ec speci ic e en s in he ehicle. Senso clus e s a e ins alled inside he ehicle cabin o eed
da a o in elligen sys ems ha aim o analyze and ed lag ce ain beha iou s ha can po en ially impac
passenge s sa e y and com o while using he ehicle.
Keywo ds: Machine Lea ning, So wa e, MlOps, De Ops, Da a Science, Pipelines, Au oma ion
i
Con en s
Lis o Figu es ix
Ac onyms xi
1 In oduc ion 1
1.1 Con ex andmo i a ion .............................. 1
1.2 Challengea BOSCH ............................... 2
1.3 Thesisgoals ................................... 2
1.4 Documen s uc u e................................ 3
2 S a e o he a 5
2.1 So wa e de elopmen me hodologies . . . . . . . . . . . . . . . . . . . . . . . 5
2.1.1 The wa e all app oach . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.1.2 Theagileapp oach............................ 6
2.2 De Ops...................................... 6
2.2.1 Technicaldeb .............................. 7
2.2.2 De OpsCul u e ............................. 7
2.2.3 De Ops echniques............................ 8
2.3 De elopmen in machine lea ning . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.3.1 C isp-Dm ................................ 9
2.3.2 Hidden echnical deb in Machine Lea ning . . . . . . . . . . . . . . . . 10
2.4 MLOps...................................... 11
2.5 Bes p ac ices o model deploymen . . . . . . . . . . . . . . . . . . . . . . . 14
2.5.1 The p oblem o concep d i . . . . . . . . . . . . . . . . . . . . . . . 14
2.5.2 MlOps amewo ks o he da a engine . . . . . . . . . . . . . . . . . . 14
2.5.3 MlOps amewo ks o model benchma k engine . . . . . . . . . . . . . 15
2.5.4 Model benchma king me hodologies . . . . . . . . . . . . . . . . . . . 15
2.6 Model explainabili y me hodologies . . . . . . . . . . . . . . . . . . . . . . . . 16
2.6.1 Lime alues explainabili y . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.6.2 Shap alues ............................... 18
ii
1.4. DOCUMENT STRUCTURE
Figu e 2: Model engine benchma k componen s
Fi s , he Model pe o mance dashboa d is a ool o simpli y he p ocess o e alua ing an ML
model. The dashboa d will con ain key pe o mance indica o s o he model, hus acili a ing he compa -
ison p ocess be ween models.
The In e ence engine is a componen o he ML sys em pipeline ha applies logical ules o a
knowledge g aph o de i e new in o ma ion. In e ence engines ha e gained much ac ion due o he
cu en machine lea ning boom and a e ex emely use ul a p o iding new insigh and knowledge.
Las ly, in he cu en machine lea ning pano ama, many models algo i hms a e buil , ained, and
e alua ed, all wi h di e en a chi ec u es, da a pipelines, and pa ame e s, gene a ing an issue. The Model
pe o mance da abase is a componen designed o au oma e and acili a e model s o age. I s o es all
he in o ma ion ela ed o each model, including key pe o mance indica o s and da a pipelines used. I
also allows o a e sioning sys em o know exac ly e e y hing abou each model and i s e alua ion.
This solu ion aims o au oma e, simpli y and speed he whole p ocess o building and deploying models
in he ML sys ems de eloped.
1.4 Documen s uc u e
The emaining documen s uc u e is he ollowing:
•S a e o he a - This sec ion a ge s a comple e e iew o he li e a u e a ound he opic o
3

CHAPTER 1. INTRODUCTION
his hesis, including p oblems in so wa e de elopmen me hodologies and he solu ions ha we e
p oposed o e he yea s o sol e hem. Machine lea ning sys ems can be so wa e sys ems, so he
same challenges can also exis in he cu en ML sys em de elopmen . A comple e e iew o hese
opics will be p esen ed in his sec ion.
•Ea ly wo k - This chap e shows he ea ly wo k owa ds he de elopmen o he MlOps au oma ion
pipeline.
•Objec i es and esul s - This chap e ou lines he equi emen s de ined o he solu ion.
•Design and a chi ec u e - Full explana ion o he implemen a ion p ocess o achie e he a -
chi ec u e de ined in he p e ious s eps. All he decisions made, implemen a ion choices, and
ou comes.
•Conclusion - This chap e p o ides he expe imen se up explana ion and he en i e ou come o
hose se ups. I concludes he wo k achie ed so a .
4
Chap e
2
S a e o he a
This chap e p o ides an in-dep h li e a u e e iew on machine lea ning and so wa e sys ems de elopmen
app oaches. Wha was used in he pas , wha is used now, and he main ad an ages and p oblems
ha a ose om each. Since machine lea ning is a sub- ield o A i icial In elligence (AI), some so wa e
de elopmen p oblems ex apola ed om one o he o he , so be o e we ackle he app oaches cu en ly
used in Machine Lea ning (ML), a b ie look a he pas is needed o u he enhance he pe cep ion o he
cu en s a e o he a in de elopmen me hodologies.
2.1 So wa e de elopmen me hodologies
This sub-chap e p o ides he ounda ion on how he cu en s a e o he a came o be. I e iews some
o he echnical di icul ies and he solu ions p oposed in he pas o unde s and he p esen .
2.1.1 The wa e all app oach
Be o e he 21s cen u y, he wa e all app oach was a common me hodology used o deli e so wa e
inside an o ganiza ion. This app oach was di ided in o mul iple phases, as desc ibed in igu e 3. Each
phase had o be comple ed be o e p oceeding o he ollowing s age, and his mean ha changes equi ed
a e he ini ial equi emen s phase we e complex and cos ly o implemen , and he p ocess was e y igid,
main enance in he so wa e was also a di icul ask, (Van Cas e en, 2017).
A he beginning o he cen u y, so wa e was in mo e demand han e e , and he wa e all app oach
s a ed o show limi a ions dealing wi h he ola ili y in so wa e equi emen s, (Rahman, 2019). The
de elopmen cycle was long, igid, and could no longe keep up wi h he inc easing demand o so wa e
supply and main enance p ocesses.
5
CHAPTER 2. STATE OF THE ART
Figu e 3: Wa e all de elopmen app oach, by (Van Cas e en, 2017
).
2.1.2 The agile app oach
In 2001, due o he limi a ion o he wa e all app oach, a g oup o de elope s discussed a new me hodology
o so wa e de elopmen , which led o he c ea ion o he agile mani es o, (Ma in Fowle , 2001). This
mani es o in oduced a se o ules and me hods o imp o e in e ac ion and communica ion wi hin eams.
Agile app oaches equi ed ace- o- ace in e ac ion and sho i e a i e cycles wi h dynamic ea u e planning,
(Van Cas e en, 2017).
The agile app oach was mean o make so wa e deli e y as and con inuous. I in ends o in e-
g a e business people wi h de elope s, acili a e la e equi emen s change, mo i a e indi iduals, imp o e
he o e all eam pe o mance, and schedule egula mee ings o discuss how o become mo e e ec i e.
Simplici y in design choices was welcomed, (Kuma and Bha ia, 2012).
Mul iple pape s su ged o e he yea s on ackling agile de elopmen , being sc um one o he mos
popula . I named he de elopmen i e a ions cycles as sp in s, which usually las s be ween 1-4 weeks.
Team mee ings called daily sc ums happen e e y day and se e o discuss wha has been done, di icul ies,
and u u e wo k, bu he o iginal mani es o’s co e philosophy emained unchanged.
2.2 De Ops
De Ops
is a se o p ac ices ha combines so wa e de elopmen (De ) and IT ope a ions (Ops) wi h one
key di e ence wi h espec o he agile app oach, which while i was g ea in simpli ying he de elopmen
p ocess be ween business people and de elope s, he ope a ions p ocess was le ou . To sol e his limi a-
ion,
De Ops
was bo n, p oposing me hods o app oxima e and combine he de elopmen and ope a ions
p ocess in o one.
6
2.2. DEVOPS
2.2.1 Technical deb
Technical deb is a e m coined by Wa d Cunningham in 1992. The e m e e s o a me apho be ween
deb in he inancial sys em and he deb in so wa e de elopmen , (Cunningham, 1992).
This deb e e s no necessa ily o lousy coding. Mos o he ime, i ep esen s a ade-o be ween good
p ac ices and simpli ied implemen a ions o sa e ime and speed up he deli e y p ocess o new ea u es.
As he de elopmen p ocess ma u es o e ime, es ing eams epo mo e and mo e bugs esul ing om
hese cheap implemen a ions c ea ing a de ici o e ime ha needs o be paid. This phenomenon can
c ea e a p oblem whe e so wa e main enance is comp omised because as each si ua ion is deal wi h,
mo e issues a ise in he p ocess.
Wi h he in oduc ion o agile me hodologies, dealing wi h echnical deb became mo e s aigh o wa d,
bu he e s ill was li le connec ion be ween de elope s and ope a o s.
De Ops
su ged as a way o mi iga e
he p oblem o echnical deb , s eamlining he whole de elopmen p ocess, om so wa e equi emen s o
ope a ions. While his helped manage echnical deb , i has no elimina ed no was i expec ed o elimina e
he p oblem.
2.2.2 De Ops Cul u e
Gene Kim, one o he bigges con ibu o s o he opic s a es in his book,
The De Ops Handbook
ha h ee
p inciples a e sus aining a good
De Ops
p ac ice and implemen a ion, (Kim, Humble, Debois, and Willis,
2016).
The i s p inciple is abou enabling le - o- igh wo k low and communica ions om he de-
elopmen o ope a ions and cos ume s, emo ing he ba ie s be ween di e en eams o depa men s
and p omo ing coope a ion. The wo k is made isible and accessible o maximize low, he upda es a e
mino bu equen , and quali y is p e e able o e quan i y. This leads o a smalle ime equi emen o
ul il equi emen s, all while inc easing quali y. The esul o his p ac ice is con inuous build, con inuous
in eg a ion,con inuous es and con inuous deploymen .
The second p inciple is all abou enabling as eedback low om igh - o-le om he ope a-
ions and cos ume s o de elope s in all phases o he wo k. Adop ing his p inciple allows o much quicke
p oblem de ec ion and consequen ix, pe mi ing p oblems o be sol ed a he co e, c ea ing quali y in he
p oduc , and educing a al ailu es o occu down he line. This p inciple is o con as wi h he i s , see
igu e 4.
The hi d p inciple is all abou building he
De Ops
cul u e and mindse , o ensu e a high us
men ali y ha suppo s a dynamic, disciplined en i onmen . The scien i ic app oach o building p oduc s
wi h ole ance o isks, making knowledge gained om ailu e and success alike, o complemen his he
sho e eedback loops allow o sa e sys ems which inc ease he ole ance o aking isks and lea ning
wi h hem. A eam adop ing hese p inciples makes hem e ol e as e han he compe i ion.
7
CHAPTER 2. STATE OF THE ART
Figu e 4: De Ops communica ion and eedback low
2.2.3 De Ops echniques
To adop
De Ops
he e a e se e al echniques o apply, Manish Vi mani s a ed nume ous echniques,
(Vi mani, 2015), as ollows:
•Con inuous planning s a es ha business plans need o be agile o mee up wi h as ma ke
changes. F equen in e im checkpoin s o o e iew he cu en s a e and adjus as needed. Wi h-
ou De Ops, i ’s ha d o de elope s o keep up wi h change, bu wi h De Ops, he inc ease in
communica ion and eedback be ween de elope s and cus ome s allows o a be e o e haul low.
•Con inuous in eg a ion o Con inuous In eg a ion (CI) o sho is a p ocess whe e a eam au o-
ma es mos o he asks be ween de elopmen and ope a ions. I in ol es eams in eg a ing hei
wo k egula ly and au oma ing asks like building, es ing, and alida ing. In he end, his achie es
as e bug inding and ixing and a g ea e s anda d o quali y in he so wa e wi h lowe cos s. This
is wha makes CI a s one pilla o a good De Ops implemen a ion.
•Con inuous deploymen is he ac o ge ing all ypes o upda es in o he hands o use s as as
and sa ely as possible, wi hou wo ying abou wha ype o upda e i is, whe he a bug ix, a new
ea u e o e en a la ge scale upda e. This educes ic ion poin s inhe en in he mo e adi ional
deploymen o elease p ocesses. In o he wo ds, he applica ion is always in a p oduc ion- eady
s a e.
•Con inuous es ing is all abou au oma ing e e y s ep possible when es ing. I an ac ion is
epe i i e o e ime, i mus be au oma ed. Cu en ly, he e is a la ge numbe o echnologies in he
ma ke ha simpli y his p ocess.
8

2.3. DEVELOPMENT IN MACHINE LEARNING
•Con inuous moni o ing, due o es ing be e and ea ly, he e is an oppo uni y o obse e c i ical
pa ame e s and eac o unseen e en s as e .
2.3 De elopmen in machine lea ning
Some machine lea ning applica ions sha e a se o simila cha ac e is ics wi h mo e adi ional so wa e
applica ions. In many cases, hey a e almos iden ical, bu wi h an ex a ML model laye , (“MLOps”,
2020). Due o his, much o he same con en ional p oblems a e p esen , wi h he ML sys em in oducing
new conce ns ha need o be add essed. This sec ion p o ides a comple e analysis o ML de elopmen
me hodologies s a e o he a and he wo k being done o imp o e ML sys em pipelines. In igu e 5 a
ypical ML sys em s uc u e is p esen ed.
Figu e 5: Componen s in a ML sys em, (“MLOps”, 2020
)
2.3.1 C isp-Dm
Wi h he ise o machine lea ning, one o he me hodologies ha gained he mos popula i y is
C isp-
Dm
, sho o C oss Indus y S anda d P ocess o Da a Mining, This me hodology is an indus y
s anda d consis ing o six phases ha desc ibe mos machine lea ning pipelines, see 6.
Each s ep o he pipeline desc ibed is an impo an phase and a e explained as ollows:
•Business unde s anding is a c ucial s ep. I consis s o unde s anding he unde lying business
ha we wan o apply machine lea ning, wha cus ome s wan o accomplish and wha da a can
lead us he e. I we s a wo king on a p oblem wi hou p e ious knowledge, i won’ be easy o
know which da a o collec and he bes way o ea i .
•Da a unde s anding is he s ep whe e we gain in-dep h in o ma ion abou he p oblem. I usually
s a s wi h an ex ensi e explo a ion o he da a wi h s a is ic analysis and da a isualiza ion.
9
CHAPTER 2. STATE OF THE ART
Figu e 6: C isp-DM pipeline by Hassanien, 2019
•Da a p epa a ion is gene ally one o he mos ime-consuming pa s o he pipeline. I ’s in equen
o ob ain he da a eady o i he algo i hms. Mos o he ime, we ha e aw collec ed da a ha is
unsui ed o ou goals. Gene ally, we wan each eco d in a ow and each a ibu e in a column,
bu his is a gene al ule o humb as ideo o audio da a ha e di e en needs. In his phase o he
pipeline, we al eady ha e in o ma ion on he quali y o he da a, o ins ance, how many missing
alues, i he da ase is balanced, o simply i he e is enough da a o an ML algo i hm o lea n.
•Modelling is abou c ea ing he model. Fi s , he e is he need o choose he echnique ha bes
sui s ou p oblem and da a, hen a es design is gene a ed ha may in ol e spli ing he da ase
in o aining, alida ion, and es subse s. The nex s ep is o build he model and ine- une i o ge
he bes esul s possible. In his phase, we can go back o he da a p epa a ion phase and make
changes o bes sui ou cu en model.
•E alua ion is he phase whe e he model is e alua ed, in his s ep we check i he model pe o ms
well and answe s o he business needs, hen an e alua ion o he en i e pipeline is done. In his
s age a decision is made whe he o deploy he model o e u n o he i s phase o s a again and
change he app oach.
•Deploymen is he las s age o he pipeline since a model isn’ use ul i i canno be accessed by
i s cus ome s. In his s age, a deploymen plan is buil and a moni o ing sys em is c ea ed o help
u u e main enance in he model.
2.3.2 Hidden echnical deb in Machine Lea ning
In chap e 2.2.1 he e m echnical deb is in oduced, ollowing ha de ini ion, (Sculley e al., 2015)
in oduced in a published a icle in 2015 he p eeminence o ML sys ems o de elop a e y speci ic ype o
10
2.4. MLOPS
echnical deb , which he called Hidden echnical deb s a ing ha ML sys ems ha e all he adi ional
p oblems o adi ional coding plus speci ic p oblems ha a ise om he ML componen s.
Fu he mo e, in adi ional so wa e, echnical deb is code ela ed, bu in ML applica ions, Hidden
echnical deb is code and da a-dependen , since an ML model lea ns om da a, any change o he model
o i ’s da a changes e e y hing, (Alahdab and Çalıklı, 2019, Sculley e al., 2015 and Sculley e al., 2014)
exempli ies his phenomenon in an a icle s udying and inding pa e ns o hidden echnical deb in ML
sys ems.
2.4 MLOps
De Ops
e olu ionized adi ional so wa e de elopmen . ML sys em de elopmen is s ill in an in an s a e
compa ed o so wa e, bu he a ea has been ecei ing much mo e ocus in ecen yea s. In he academic
communi y, mos o he a en ion goes o he ea ly s ages o he ML pipeline, p epa ing and explo ing da a
and model de elopmen , bu in he eal wo ld, model deploymen and main enance a e as impo an i we
wan o gi e use o ou ML sys ems, so au oma ing ou sys em pipeline becomes a c ucial s ep o ha e
eliable esul s in eal-li e applica ions, (A nold e al., 2020).
MlOps came o sol e many o he issues o ML de elopmen by applying
De Ops
p ac ices in ML
sys ems. Acco ding o ((“MLOps”, 2020)), ML sys ems di e om adi ional so wa e and ail in di e en
a eas. The s eps in a adi ional da a science wo k low a e as ollows:
•Da a ex ac ion is a he beginning o e e y da a science wo k low, since ML isn’ possible wi hou
da a. In his phase, da a is collec ed om one o mul iple ele an sou ces.
•Da a analysis co esponds o he phase whe e he aw da a collec ed in he p e ious phase
is analyzed. Once his p ocess is comple ed, he eam ge s amilia wi h he da a a ailable, i s
cha ac e is ics, and he wo k ha will need o be pe o med in he nex phase.
•Da a P epa a ion is one o he s eps ha a e gene ally mo e ime-consuming. I in ol es cleaning
he aw da a, ea missing alues and pe o m ea u e enginee ing. In he end o his s ep we ha e
he da a eady o eed ou model.
•Model aining is whe e he eam implemen s and es s wi h di e en algo i hms and di e en
hype -pa ame e s une. This ou pu s he bes - ained ML model.
•Model e alua ion, he model is alida ed o p edic i e alue, and i i passes, i ’s eady o be
deployed in a eal-wo ld scena io.
•Model se ing is when he model is deployed in he eal wo ld in a speci ic en i onmen , his can
be a web API, mobile o desk op applica ion, among o he possibili ies.
11
CHAPTER 2. STATE OF THE ART
•Model moni o ing is he inal s ep o a usual Da a science wo k low, he model is moni o ed, and
i key pe o mance indexes s a d opping, main enance is equi ed, and an ML ask in his p ocess
is pe o med.
MlOps
is all abou au oma ing e e y single s ep ha is epe i i e along mul iple ML pipelines, and he le el
o au oma ion de ines he ma u i y o he MlOps implemen a ion. A bigge ma u i y means a as e end o
end p ocess.
An en y-le el ML sys em has no au oma ion in he pipeline. This is e y common in academic p ojec s
and companies s a ing hei i s ML p ojec s. The ypical pipeline in his ype o p ojec doesn’ include
con inuous deli e y o in eg a ion, no au onomous moni o ing, see igu e 7. This ype o sys em usually
doesn’ age well due o all he limi a ions ha will con ibu e o hidden echnical deb .
Figu e 7: Fi s le el o ma u i y, by (“MLOps”, 2020
)
In he nex le el, we ind a ype o ML sys ems ha ha e a mo e ma u e
MLOps
implemen a ion. These
ypes o sys ems ha e a signi ican le el o au oma ion implemen ed. This includes he da a inges ion
pipeline o eed he model ha ensu es con inuous aining, inally, he consequen con inuous se ing o
he model o he inal cus ome . A his le el, we can ain new models gi en new da a e y as . Howe e ,
he p ocess is s ill slow and no ideal o making changes in he ML algo i hm, see igu e 8.
The inal le el o ma u i y is he ideal
MLOps
ML sys em wi h ully- ledged CI/CD, (con inuous in eg a-
ion / con inuous deli e y), implemen a ions in his s age. The con inuous in eg a ion usually has ea u es
like au oma ic es ing o model pe o mance, con e gence and beha io checks be ween di e en pipeline
componen s. The con inuous deli e y implies e y as deli e y o changes in he pipeline o a p oduc ion
en i onmen , his le el o ma u i y is expec ed o ha e a e y as implemen a ion gi en new da a o gi en
a new algo i hm, see igu e 9.
One o he key pa s o e e y
MLOps
implemen a ion is o imp o e he sys em’s quali y in he long un,
as ML sys ems don’ age in he same way as adi ional so wa e sys ems. To comba his phenomenon,
12
2.8. SUMMARY
The Hellinge dis ance measu es he simila i y be ween wo dis ibu ions. The o mula o calcula e
he Hellinge dis ance is in igu e 12, he o mula e u ns alues close o ze o i bo h dis ibu ions a e e y
simila and close o one when hey a e e y di e en om each o he .
Figu e 12: Hellinge dis ance
The a e age pai wise dis ance i e a es o e all obse a ion in bo h dis ibu ion and e u ns he a e age
compu ed dis ance, he o mula is in igu e 13, euclidean dis ance is he dis ance o mula in his case, bu
any o mula ha can p edic he dis ance be ween obse a ion wo k wi h he a e age pai wise o mula.
Figu e 13: A e age Pai wise dis ance
In he li e a u e, (B eck, Cai, Nielsen, Salib, and Sculley, 2017) explo es in e es ing app oaches like
da a s a is ical analysis and da a schemes ha include he no mal ange expec ed o he ea u e, his
way i da a s a s de ia ing om he no mal s anda d a he ime he model was ained, a wa ning can
be p o ided o he de elope who can ac and a oid mal unc ion in he sys em due o da a d i ing, his
p ocess allows sco ing and quan i y changes in da a. Howe e , me ics analysis needs o be obse ed as
well, which p esen challenges. Fu he mo e, he e is no way o compa e he p edic ions wi h he g ound
u h as he da a is no labelled in eal- ime and equi es human in e en ion, so his p esen s challenges
ha need o be sol ed.
Da a d i is a no mal phenomenon ha will always p esen i sel o e ime, like o example consuming
habi s changing yea a e yea . Howe e , ce ain e en s can ende an ML algo i hm useless o e nigh ,
o example, a global pandemic ha o ced he en i e popula ion o use a ace mask, making a huge
shi owa ds online sales and led he global ma ke s o an as onishing ola ili y a e, his led o huge
implica ions in acial ecogni ion applica ions, ecommenda ion algo i hms unde pe o ming and s ock
ma ke algo i hms making huge mis akes.
2.8 Summa y
In conclusion, de eloping an end o end ML applica ion has many ac o s o conside . Adop ing he bes
MlOps p ac ices means a as e de elopmen cycle and pos -deploymen upda es, he gains ob ained
wi h MlOps depend on he ma u i y o he implemen a ion. To assis wi h he au oma ion o he pipeline,
many amewo ks a e p esen in he ma ke ha help he p ocess. Each has i s main ad an ages and
disad an ages o each use case.
19

CHAPTER 2. STATE OF THE ART
Choosing he bes algo i hm is complex, and good benchma k me hodologies in he ma ke allow o
compa isons be ween di e en models. Some me hodologies conside in e ence cos , aining cos , o
bo h, and depending on he use case all ha e hei ad an ages and disad an ages.
E en he bes algo i hm won’ s and he es o ime as da a d i can and will impac he model’s
pe o mance. Many moni o ing me hodologies a e being s udied o comba his phenomenon, which
applies echniques o con ol and signal po en ial da a d i .
20
C h a p e
3
Ea ly wo k
A ca has unique pa icula i ies o collec ing da a due o i s as pace mo emen , esul ing in po en ial
signi ican changes in he ai quali y and p ope ies o he du a ion o he ip. This ea ly wo k subsec ion
in ends on p o o yping a solu ion o collec ing as amoun s o da a om eal-wo ld ci cula ing ehicles,
and s udy hei p ope ies o ind alue in hem. This alue can be o de ec dange ous le els o smoke o
dis inc noise ha can indica e dange o he passenge o physical ha m o he ehicle.
Be o e he da a is eady o be ed in o Ml algo i hms, many s eps a e needed, like collec ing i , ea ing
i , and ensu ing eliabili y in he da a. This p ocess is aimed o be au oma ed, so a pipeline among he
algo i hms and he da a sou ce is cons uc ed.
A e he model is ained, a pipeline be ween he aining and deploymen will also be cons uc ed
o s eamline and acili a e deploymen , benchma king, and moni o ing new models. This au oma ion will
lead o a much as e end o end cycle and p o ide he abili y o upda e and main ain he Ml applica ion
easily.
3.1 Da a collec ion - Ha dwa e solu ion
3.1.1 Senso s and compu a ional de ice
To collec da a om he ehicle, a compu a ional de ice and senso s a e needed. Senso s collec aw da a
om he en i onmen , decode he da a, and eco d i o an app op ia e s o age solu ion. The ollowing
ha dwa e was selec ed
•Pm2.5 Senso is a pa icle senso capable o de ec ing mul iple sizes o pa icles in he ai .
•Uma8 mic ophone a ay is a high de ini ion 8 channel mic ophone a ay ha ha collec s audio
om he en i onmen .
21
CHAPTER 3. EARLY WORK
•Bme680 is a gas senso capable o measu ing a wide ange o gases, p essu e, humidi y, and
empe a u e.
•Raspbe y pi4 model B is he compu a ion de ice ha manages he senso s and collec he da a
om hem in a da abase
These senso s we e mainly selec ed because hey can cap u e he da a needed and we e a ailable in he
company s ock o senso s.
The compu a ional de ice is chosen because i o e s he compu a ional powe equi ed and can un
a Linux dis ibu ion ha uns he so wa e necessa y o manage he senso s.
3.1.2 Casing solu ion
The esul s ob ained in a gi en es ehicle needs o be ep oducible and gene alized o all ehicles.
The senso s mus be ins alled in iden ical loca ions wi h iden ical con igu a ions o a oid di e en noise
be ween di e en se ups and o minimize noise om he en i onmen . This equi es a gene alized se up.
To sol e he ins alla ion p oblem, a casing design was de eloped using a 3d P in e . This design
needs o allow o he co ec accommoda ion o all he senso s. Fo ins ance, he mic ophone canno
ha e sound blocked, and p ope ai low needs o be supplied o he gas and pa icle senso ; o he wise,
hey will no unc ion co ec ly. Fu he mo e, he ins alla ion loca ion is in he ca windshield, so i needs
o be a ached wi hou sc a ching o damaging he glass when ins alled.
Wi h all he equi emen s add essed, he design p ocess can s a , a ee CAD so wa e was used, and
a wo-le el design was concep ualized. This allows o ai low and sound o pass o he senso s while no
o e complica ing he manu ac u ing(p in ing) p ocess. The esul s can be seen in igu e 14
Figu e 14: Design’s le els design
This design is p in ed and he inal esul is displayed in igu e 15. The design p o ides a simple,
e ec i e, and pleasan casing solu ion ha won’ dis ac he d i e and can blend in wi h he na u al
aes he ics o he ca .
22
3.2. SOFTWARE SOLUTION
Figu e 15: P in ed mul i le el design
3.2 So wa e solu ion
To manage and coo dina e all senso s, Robo ope a ing sys em (Ros) is used. ROS allows us o ea each
senso as a node ha can communica e wi h a cen al node, usually called he lis ene ha subsc ibes
o channels and ecei es messages om he senso nodes publishing in hose channels. This app oach
allows o achie e ue sync be ween da a om di e en sou ces, a ime synch onize il e is used o
elimina e incohe en da a ha a i es in di e en ime sou ces, he da a is hen s o ed in a ile s uc u e
called
bag ile
ha allows s o ing aw da a wi h di e en o ma s which a e highly bene icial since each
senso has a unique da a schema, his a chi ec u e can be isualized in igu e 17.
Figu e 16: Componen diag am o da a collec ion ool
This applica ion uses a launch sc ip o ini ia e all nodes a he same ime. The node senso egis e s
hei channels in he ROS API, and he lis ene node subsc ibes o hem and s a s wai ing o messages.
The senso s s a collec ing da a and a e imed in p ede ined in e als o publish he da a in he speci ic
da a schema. The lis ene hea s he messages a i ing in di e en channels and e i ies he ime synch
be ween hem o check i consis ency is main ained; i e e y hing is good, hey a e s o ed in a da abase
o o he wise dele ed. This low can be obse ed in 17
23
CHAPTER 3. EARLY WORK
Figu e 17: Ac i i y diag am o da a collec ion ool
3.2.1 Da a isualiza ion ool
The da a collec ion ool is c ea ed and has a s anda dized ou pu meaning he da a ea men can be
au oma ized, o his eason, he da a isualiza ion ool has equi emen s speci ied o allow he au oma ion
o da a be ween he da abase and he isualiza ion ool. To achie e his he ool mus in e p e and decode
he
bag iles
ha a e he ou pu o he collec ion ool, hen ex ac and ea he da a by in e p e ing how
many senso s a e egis e ed and gi e he use he choice o selec he ones ha ha e in e es o he
cu en analysis, hen show he g aphs and s a is ical in o ma ion om he da a, see igu e 18.
Figu e 18: Componen diag am o da a isualiza ion ool
This ou componen s in igu e 18 wo k as ollows:
24

3.2. SOFTWARE SOLUTION
•Da aLoade Se ice is a collec ion o unc ions ha loads aw da a and execu es he necessa y
p ocesses o make i usable
•Da a is he componen ha ep esen s he ins ance o da a, p o iding unc ions and in o ma ion
on i ’s domain
•Da aCon olle is a comunica ion laye be ween he model class Da a and he Use In e ace,
his app ach in en s on cen alizing all he eques s.
•Use In e ace is he isual in e ace o he so wa e and makes in e ac ion wi h he ool simpli ied.
This applica ion is designed o be used h ough he use in e ace, all he unc ions a e explained in
he ac i i y diag am in igu e 19
Figu e 19: Ac i i y diag am o isualiza ion ool
25
Chap e
4
Objec i es and esul s
4.1 P oblem de ini ion
The de elopmen o an end o end Machine Lea ning (ML) pipeline is a e y complex p ocess. A concep
o e alua e he pe o mance o algo i hms shall be de ised and implemen ed o suppo he choice o he
bes model and an in e ence engine ha p o ide eedback o he p edic ions and assu e model eliabili y.
The end goal o he hesis is o accele a e and imp o e he company al eady exis ing ML de elopmen
pipeline by au oma ing and imp o ing he exis ing s eps. The esea ch ound ha he bes model is a highly
subjec i e ma e , wi h he use case has signi ican implica ions on benchma k me ics. In his sec ion, a
ew conside a ions a e made o achie e he inal goal.
4.1.1 Benchma k conside a ion
The ul ima e p oblem o his p ojec is o make impac de ec ion and classi ica ion, his means he model
ecei es da a om he senso s and p edic s i an impac happened and wha ype o impac o unde s and
damages done o he ca , based on his goal a be e decision o benchma k me ics can be made.
In many ML p ojec s in e ence imes a e an essen ial subjec , howe e , he goal o he cu en ML
implemen a ions is o de ec and classi y damage impac s on a ca ex e io , since we ha e no in en ion on
p edic ing upcoming e en s, in e ence imes a e no impo an . This means ha he co ec me ics a e
he single mos impo an ac o in e alua ing models.
Di e en models wi h di e en ole all low om he pipeline and a i e a he benchma king engine,
each mus ake in o conside a ion di e en e alua ion me ics o co ec ly assess i s use ulness and pe -
o mance. The da a scien is mus be he esponsible ac o who chooses he co ec me ics o e alua e
he model.
26
4.2. OBJECTIVES
The benchma k engine pla o m’s ole is o ha e a s anda d ha allows o he benchma k o e e y
me ic and o display he ele an in o ma ion on he dashboa d. The web dashboa d has he ole o
in ui i ely display model key pe o mance indexes and showcase he bes models o each model ype
p esen in he sys em.
4.1.2 Concep d i in he con ex
Da a d i is ce ain o occu in e e y da a- ela ed p oblem, bu i ’s essen ial o conside how i can a ec
he pe o mance in his speci ic use case. In his con ex , da a d i can occu due o se e al easons,
some can be easily iden i iable and some no . Fo ins ance, i a senso s a s o misbeha e, i can be
easily iden i ied wi h a moni o ing ool. Howe e , suppose he ca manu ac u ing p ocess o he ma e ials
i ’s made changes o e he yea s o make i sa e . In ha case, he da a o a simila inciden can change
d as ically, like he sound o he collision o less g- o ce egis e ing, making he model obsole e.
A co ec moni o ing app oach mus be deployed o de ec he occu ence o da a d i in he da a
ecei ed by he model, his can be done o ins ance by applying he Hellinge dis ance, igu e 12. I he
same ype o collision da a s a s di e en ia ing a lo om aining o eal-wo ld, hen his o mula will s a
o ou pu alues close o 1. A case s udy needs o be pe o med o disco e he bes me ic o measu e
such de ia ions o i such me ics can be help ul in he da ase used.
This sys em is also esponsible o keeping benchma king each model has new da a ha co esponds
o he o iginal que y is a ailable. This p ocess helps de ec model pe o mance o e ime and ensu e he
model is s ill pe o ming as in ended.
4.1.3 Con inuous deploymen and con inuous aining
Ensu ing a pipeline capable o con inuous deploymen and aining is essen ial. Due o he en e p ise
na u e o his applica ion, down ime mus be educed o he minimum, so i ’s impo an o design a
pipeline ha can pe o m upda es in eal- ime o he model. So con inuous aining ensu es concep
d i doesn’ make he model ou da ed, and con inuous deploymen ensu es he model is upda ed. The
benchma k engine in eg a ed in o he pipeline helps choose he bes model in he cu en se ing.
4.2 Objec i es
The end goal is o design and build a pipeline capable o imp o ing and accele a ing he cu en ML
de elopmen cycle in he o ganiza ion, his implies o acili a e, imp o e and au oma e mos o he s eps
and asks al eady exis ing. In he end, ime and human cos s mus be signi ican ly educed.
This hesis end goal is o p o ide a solu ion ha can ackle he inal s eps o he pipeline. A e he
model is ained, he solu ion mus au oma e e e y ac ion necessa y and display he inal key pe o mance
indexes and compa isons be ween di e en models in he web dashboa d. The model benchma king
27
CHAPTER 4. OBJECTIVES AND RESULTS
engine needs o wo k and communica e wi h he p e ious componen s om he pipeline and espec
he s anda d de ined by he eam. To ackle he p oblem, h ee-componen mus be de ised. These
componen s a e as ollows:
• Model benchma k engine, capable o indexing, e sioning and benchma king each model ha a -
i es while also pe o ming con inuous e alua ions on exis ing models.
• Web engine, capable o displaying all in o ma ion abou each model and benchma k while also
p o iding a compa ison ool
• Da abase solu ion, capable o s o ing and managing all he in o ma ion equi ed by he sys em.
4.3 Requi emen s
To sol e he p oblem de ined abo e, a combina ion o h ee componen s will be de eloped, each o sol e
a se o challenges ela ed o he end s age o he pipeline. Combined, hey o m he model benchma k
engine e e enced in his documen . These ools add ess he gene al complica ion in ML de elopmen ,
om model aining o deploymen and moni o ing.
The ollowing equi emen s a e de ined o he sys em.
4.3.1 Func ional equi emen s
The sys em mus allow o:
•Recei e new models om he pipeline. Models will low om he model aining engine
au oma ically, a s anda d is necessa y o send and ecei e models o communica ion and usabili y
pu poses.
•P obe i new Da a is a ailable. A model comes associa ed o a da ase using i s embedded
me ada a and a da ase Tag, new da a ele an o he model may be a ailable wi h ime making i
necessa y o index and p ocess i o he app op ia e models.
•T igge benchma k ac ion au oma ically o new models o new da a. The i s bench-
ma k needs o be igge ed when possible by he sys em, howe e his doesn’ mean he end o
he model e alua ion p ocess, each model is se o be con inuously benchma ked as new da a is
made a ailable.
•S o e all he ele an me ada a abou each model / benchma k / da ase . The end use
mus ha e access o all ele an model in o ma ion, his in o ma ion is p ocessed and s o ed in he
da abase o u he u iliza ion.
28
5.2. MODEL BENCHMARK ENGINE DESIGN AND ARCHITECTURE OVERVIEW
5.2.1 Sys em use cases
Conside ing he p oblem de ined, he use o he model benchma king engine, a da a scien is , does no
ha e any in e ac ion wi h he unde lying sys em and only consul s he in o ma ion ou pu ed and p o ided
by he web dashboa d. The use has he ollowing use cases:
•UC1: The use mus be able o manually inse o dele e models om he da abase.
•UC2: The use mus be able o access he da abase and pe o m CRUD ope a ions.
•UC3: The use mus be able o il e each in o ma ion able in he dashboa d.
•UC4: The use mus be able o o de each in o ma ion able by column.
•UC5: The use mus be able o compa e di e en model benchma ks.
•UC6: The use mus be able o ha e esul s il e ed by da ase .
•UC7: The use mus be able o see all he me ada a collec ed abou he model.
•UC8: The use mus be able o access a model explainabili y dashboa d.
•UC9: The use mus be able o access da ase explainabili y dashboa d.
•UC10: The use mus be able o access all logging in o ma ion in he sys em.
•UC11: The use mus be able o download indi idual models.
Mos o hese use cases equi e ha he au oma ed unde lying unc ion wo k as expec ed. The sys em
is esponsible o ecei ing new models, s o ing he me ada a and da ase in o ma ion in he da abase,
pe o ming benchma k ope a ions, and s o ing he me ics. Finally, i also uns ope a ions o compu e
SHAP
alues and cha s.
The main ac o o he sys em is he da a scien is , which is denomina ed as he main use o he
sys em. Mos ea u es and use cases implemen ed ha e he pu pose o imp o e he da a scien is abili y
o ga he knowledge abou each model and he decision-making p ocess.
Each Use Case (UC) is explained in he ollowing use case diag ams. Fo example, in igu e 23 he
UC1 is explained, he model is inse ed using he web dashboa d which uploads he ile o an objec
s o e solu ion, an in e nal scheduled p ocess e ches o new models in a speci ied in e al and i i ounds
one makes a eques o he ile, i hen pa ses he embedded me ada a and s o es i in he da abase
espec ing he da a schema de ined.
The sequence diag am in igu e 24 showcases manual CRUD ope a ions o he da abase, UC2 ha
may be necessa y o pe o m. The web dashboa d p o ides he in e ace o hese ope a ions, and when
con i med, he changes a e execu ed in he da abase engine, making he p ocess as use - iendly as
possible o he use .
All da a ables in he web dashboa d mus be dynamic. This means each column mus ha e he
possibili y o o de by ascending o descending and a cus om sea ch ba o il e he con en in each able.
UC3, UC4 is desc ibed in he sequence diag am in igu e igu e 25.
Ano he essen ial use case is UC5 and UC6. A ool mus be de ised ha allows compa ison be ween
benchma ks o he same model o di e en models, and i is also impo an o il e all esul s by da ase o
35

CHAPTER 5. DESIGN AND ARCHITECTURE
Figu e 23: UC1: Sequence diag am o manual model inse ion
Figu e 24: UC2: Sequence diag am o manual CRUD ope a ions
Figu e 25: UC3, UC4: Sequence diag am o da a il e ing
36
5.2. MODEL BENCHMARK ENGINE DESIGN AND ARCHITECTURE OVERVIEW
ind he bes pe o ming model o a gi en da ase o he bes pe o ming model o a p ojec use case like
a damage de ec o o damage loca o , he sequence diag am in igu e 26 explains he in ended sequence
o his p ocess.
Figu e 26: UC5, UC6: Sequence diag am o benchma k and esul s il e ing
UC7 desc ibes he p ocess o a use ge ing all he in o ma ion a ailable abou a model. This includes
all i s me ada a like he name, imes amp in oduced, and all he benchma ks in o ma ion pe o med o
he model like he me ics esul s, da ase s used, dis ibu ion ma ices and mo e. This in o ma ion can
be consul ed by eques ing he model page, and he p ocess is desc ibed in he diag am in igu e 27
UC8 is abou he use eques ing and accessing a model explainabili y page. Explainabili y e e s o
he model hype pa ame e s and uning in o ma ion and he SHAP explainabili y cha s explo ed in s a e
o he a . Each model should ha e SHAP explainabili y. The p ocess o displaying his in o ma ion is in
igu e 28. This in o ma ion should al eady be compu ed a e he benchma k is pe o med and he da a
s o ed in he da abase.
UC9 UC9 is a ea u e eques ed so he end-use can unde s and wha is inside a speci ic da ase
used o a benchma k. Al hough he p ocess is explained in he sequence diag am o igu e 29, his is
di e en om o he in o ma ion pages, all da a s a is ics a e compu ed ou side he
benchma king engine
componen and ins ead compu ed on he da a inges ion engine, due o in e ope abili y equi emen s
be ween he wo, a s anda d mus exis o gua an ee he co ec da ase is eques ed.
UC10 is a use case equi emen because he end-use will only access he web dashboa d. I an e o
happens in he backg ound componen s, i is impo an o ha e access o he logs. Logs include sys em
ailu es, excep ions and also c i ical pe o med ope a ions like benchma king being execu ed on a model.
37
CHAPTER 5. DESIGN AND ARCHITECTURE
Figu e 27: UC7: Sequence diag am o o de ing a model page.
Figu e 28: UC8: Sequence diag am o o de ing an explainabili y page.
38
5.2. MODEL BENCHMARK ENGINE DESIGN AND ARCHITECTURE OVERVIEW
Figu e 29: UC9: Sequence diag am o o de ing a da ase explainabili y page.
The sequence diag am in igu e 30 explains he log collec ion and s o ing p ocess, and he dashboa d
p o ides he in o ma ion.
Figu e 30: UC10: Sequence diag am o ob aining logging in o ma ion.
UC11 is a pa o eques ing he model page, so igu e 27 explains he p ocess and hen a download
bu on is a ailable in he model page.
39
CHAPTER 5. DESIGN AND ARCHITECTURE
To achie e his se o unc ionali ies desc ibed in he use cases, he sequence diag am in igu e 31
showcases he s eps needed and he o de o be execu ed by he sys em. All h ee main pipeline com-
ponen s a e p esen and ep esen ed inside a ec angle which is an en i y. The sequence s a s wi h he
model aining engine submi ing a ained ML model o he Model benchma king engine. This sys em
indexes he incoming model in a da abase o be picked by he benchma k engine o be e alua ed. The
co ec da ase ag is in he model da ase , so a eques o he e alua ion da ase co esponding o he
ag is sen o he da a inges ion engine, which e u ns he da a, he benchma k is pe o med, and he
esul ing me ics a e s o ed in he da abase. Finally, all he da a in he da abase is a ailable o consul in
he end-use ’s dashboa d.
Figu e 31: Model benchma king engine in e nal p ocess
Wi h all he au oma ed s eps desc ibed in igu e 31 a pipeline in oduced model is benchma ked.
Howe e , one o he use cases implies ha he use can also occasionally inse models ha we en’
p oduced by he pipeline. The sequence is desc ibed in igu e 32. I is impo an o unde s and ha
a model om he pipeline comes wi h p ede ined me ada a, some necessa y and some op ional, ha
should also be p esen in manual inpu ed models, o else he sys em may be incapable o pe o ming an
au oma ic benchma k.
40

5.3. SOLUTION ARCHITECTURE
Figu e 32: Model benchma king engine, manual inpu ed model
An impo an use case is o quickly p oduce alue om he da a s o ed on he da abase o he end-
use . The dashboa d mus p o ide he ools o pa se he da a and display i so he da a scien is can see
and pe cei e he in o ma ion as e icien ly as possible, o ins ance, inding he bes pe o ming algo i hm
o a speci ic use case o he bes algo i hm in a speci ic da ase .
5.3 Solu ion a chi ec u e
Inside he model benchma king engine, h ee main componen s a e designed o answe he p ojec e-
qui emen s and use cases. This sec ion speci ies and explains he design choices associa ed wi h each.
The i s componen equi emen is o ha e a s o age solu ion o he da a associa ed wi h he sys em,
he da abase componen is designed speci ically o his p oblem.
The second componen equi emen is o ha e a benchma king engine capable o managing all
he ope a ions ela ed o a machine lea ning p oblem, e ching, scheduling and upda ing benchma ks o
each model au oma ically wi hou use inpu .
The hi d componen equi emen is o ha e a ull web dashboa d capable o answe ing all he
ques ions abou he collec ed da a simply and e ec i ely. The dashboa d con aine accomplishes his.
To unde s and he gene al scope o each subcomponen in in eg a ion wi h he pipeline, he building
block diag am in igu e 33 explains he scope in h ee laye s, each di ing in mo e de ail.
In laye wo, we ha e a mo e de ailed look in o he h ee componen s ha oge he compose he model
benchma king engine. They a e he da abase solu ion, benchma k engine and he dashboa d. Finally, in
laye h ee, we ha e a mo e de ailed look in o each componen .
The ollowing subsec ions explain in de ail he unc ioning o each subcomponen .
41
CHAPTER 5. DESIGN AND ARCHITECTURE
Figu e 33: Model benchma king engine, building block diag am
42
5.3. SOLUTION ARCHITECTURE
5.3.1 Da abase solu ion, design, and a chi ec u e
The i s componen is esponsible o he sys em da a s o age solu ion. When conside ing a da abase
solu ion, many ac o s a e in ol ed in he ype and s uc u e o da a s o ed and he op imiza ion o each
da abase engine. The ollowing ques ion help o decide on which solu ion o choose:
• Is he da a scheme ixed o will i change?
• Is he e ela ions be ween di e en ables in he da abase?
• How much da a will he sys em s o e?
• Does he da abase engine scale as equi ed?
• Is he da abase engine open sou ce o does a licensing ag eemen s wi h he company exis s?
Following hese ques ions, he solu ion can s a o be o mula ed. This p oblem is de ined in an objec -
o ien ed language. The class diag am in igu e 34 helps o unde s and he da a s uc u e and ela ions
ha exis . The scheme is expec ed o be mos ly ixed so ha no signi ican mig a ions will happen o he
da abase wi h ime. Mos ables ha e ela ionships be ween each o he , meaning a ela ional da abase
engine will be ideal when only conside ing his ac . Full da a size isn’ expec ed o be an issue, as ac ual
model iles will be s o ed independen ly.
A e conside a ion, he Pos g eSQL da abase engine is chosen as i mee s all he equi emen s.
Howe e , models iles s ill need an app op ia e s o age solu ion. This solu ion is an independen objec
s o ing se ice called minio. Minio is an open-sou ce, high pe o mance and cloud-based s o ing solu ion
ha p o ides all he equi emen s o all pipeline componen s, so all he da a lake b u e iles will also
be s o ed in a minio bucke . Fu he mo e, since some da ase s can each mul iple gigaby es o size,
eques s need o be p ocessed as as as possible, minio can w i e/ ead a up o 183 GB/s and 171 GB/s
espec i ely.
43
CHAPTER 5. DESIGN AND ARCHITECTURE
Figu e 34: Benchma king engine class diag am
Following he diag am in igu e 34. Each model has associa ed a unique au ho and an ’
explainabili y
’,
which is whe e he di e en model hype pa ame e s a e s o ed. Las bu no leas a e he benchma ks
pe o med on he model, wi h each benchma k ha ing associa ed a da ase and i s SHAP explainabili y
cha s. Log ables a e also p esen o s o e sys em ope a ions, mal unc ions and excep ions. The da a
s uc u e is no e y complex bu achie es all equi emen s, and wi h a unique model ile, he componen
should be able o ill all his in o ma ion au oma ically.
The inal conside a ion is o he in e ope abili y s anda d be ween he da abase and he objec s o age
solu ion, he model name wo ks as he unique ID o each model and is s o ed in bo h s o age sys ems.
When a sys em wan s o pick a gi en model, i asks he ile s o age solu ion o he model name s o ed
in he ela ional da abase, and i e u ns he model ile. Since model names mus be unique he sys em
gua an ees he co ec unc ionali y.
Finally, a e a model is ained in he p e ious s eps o he pipeline, i is inse ed in o he objec s o age
solu ion. Then, an au oma ic p ocess is scheduled o un in a gi en ime in e al which checks he models
inside he objec solu ion bucke and he ones indexed in he da abase. I new models a e ound, hey a e
added o he sys em.
The da abase solu ion is made o wo dis inc componen s as discussed abo e and shown in igu e
33 on le el 2 o he diag am:
•Da abase engine - Pos g eSQL
44
5.3. SOLUTION ARCHITECTURE
The cha in igu e 40 showcases he key me ic o unde s and he aw ou pu o he sys em in e ms o
benchma ks, he me ics benchma ks pe minu e is e e enced in he cha as BPM, as can be obse ed,
benchma ks pe minu e a e a 0.51 in he single- h eaded es , using wo h eads he alue ises o 0.90
and hen 1.19, 1.39, 1.56 and inally 1.67 using all six h eads alloca ed o he p oblem. Fu he inc ease
in h eads is no sus ainable on he ha dwa e side, and e u ns a e al eady diminishing, indica ing ha he
pa alleliza ion p ocess is op imized o he ha dwa e a ailable.
Figu e 40: Benchma king engine scalabili y compa ison, h ead BPM compa ison
Using his scaling echnique, he sys em now has a speed up o 327 pe cen compa ing o he
baseline single- h eaded a chi ec u e, which signi ican ly imp o es he esponse capabili ies ha i p o ides
o con inue o e alua e machine lea ning models, u he in he u u e, i he sys em needs mo e scaling,
a ho izon al scaling app oach migh be equi ed, which implies adding compu a ional de ices.
5.3.3 Web engine a chi ec u e
The inal subcomponen is he web engine, which is esponsible o p o iding all he in o ma ion collec ed
by he sys em o he end-use in he simples and mos in o ma i e way possible. This goal is achie ed by
ha ing a web amewo k handling he backend da a and a powe ul on end dashboa d o showcase all
he in o ma ion.
All he da a is e ched in he da abase solu ion. This means ha no communica ions occu be ween he
benchma k and web engines, which simpli ies he a chi ec u e. The echnologies o de elop he solu ion
a e as ollows:
51

CHAPTER 5. DESIGN AND ARCHITECTURE
•Django: Django is a powe ul py hon web amewo ks ha allows he c ea ion o la ge scale web
applica ions, i also has a powe ul ORM da abase connec o ha wo ks na i ely wi h he chosen
SQL da abase solu ion and simpli ies he que ying p ocess.
•Boo s ap: Boo s ap is a powe ul HTML, CSS and JS lib a y ha allows he c ea ion o in e ac i e
and dynamic web pages, i is specially powe ul o c ea e dashboa d like applica ions which a e he
goal o his applica ion.
The a chi ec u al pa e n is he Model-View-Con olle , which sepa a es he applica ion in o h ee logical
componen s, allowing o g ea e modula i y, easie main enance and a bigge upg ade pa h o he u u e.
Figu e 41 showcases he componen s uc u e and he connec ion o he s o age solu ion.
Figu e 41: Web engine subcomponen explained
52
5.4. DEPLOYMENT OVERVIEW
5.4 Deploymen o e iew
The deploymen o he applica ion showcases he equi emen s and echniques used o deploy he ap-
plica ion and he ha dwa e u ilized and equi ed o gua an ee he co ec unc ionali y. The igu e 42
is a deploymen diag am and displays all h ee main componen s in he pipeline. TCP-IP connec s he
model benchma king engine wi h he da a inges ion engine and he model aining engine.
Inside he model benchma king engine, he e a e h ee main docke con aine s, each unning a subcom-
ponen . Communica ion be ween subcomponen s only happens wi h he s o age solu ion. The inal use s
o he sys em adminis a o in e ac s wi h he sys em using a web b owse and es ablishing an HTTPS
communica ion ha displays a on end dashboa d.
Figu e 42: Applica ion deploymen diag am
53
CHAPTER 5. DESIGN AND ARCHITECTURE
5.5 Summa y
Now ha he a chi ec u al design o he applica ion is buil , he e is a solid g ound o s a he de elopmen
p ocess, he a chi ec u e o he whole A las sys em is comple ed, he gene al o e iew can be seen in
igu e 43.
Figu e 43: A las inal a chi ec u e
54
C h a p e
6
Resul s demons a ion and discussion
The ocus o his chap e will be o demons a e he web dashboa d, all he in o ma ion displayed in he
dashboa d is compu ed in he o he componen s as shown in he p e ious sec ion wi hou addi ional use
inpu . This is key o he sys em au oma ion and ime-sa ing bene i s, as i allows he use o ha e i s
models con inuously benchma ked wi hou pe o ming any ac ion in he sys em.
The ini ial page and he i s in e ac ion wi h he sys em is he de aul on page. I is designed o show
key in o ma ion abou he model benchma king engine and he A las pipeline and mus be in ui i e
and s aigh o wa d, such as allowing he use o use he sys em wi hou ex e nal help.
Figu e 44 demons a es he ini ial page wi h mul iple numbe ed ec angles highligh ing di e en in o -
ma ion a ailable, A ea 1 ep esen s he side menu ha gi es he use quick access o all majo unc ions
a ailable. A ea 2con ains in o ma ion abou he sys em s a e like models loaded, benchma ks execu ed
and pending an execu ion, numbe o di e en da ase s p esen , and he numbe o he di e en au ho s
who submi ed models o he sys em. A ea 3a e quick access ables wi h he la es i e pe o med models
and he la es i e pe o med benchma ks. I con ains he name o he model, he au ho ha c ea ed
i , da e o submission and a benchma k s a e wi h he la e being ed i he benchma k is no ye pe -
o med o a g een bu on i i is a ailable, his allows he use o ind i s la es submi ed model wi h as
ew clicks as possible. Las , a ea 4 ep esen s he A las pipeline s a us, since he pipeline depends on
all componen s wo king p ope ly, i is impo an o ha e all componen s’ s a us eadily a ailable and easy
o ob ain.
In he ini ial page and ollowing he side menu, he use can choose he
models page, benchma ks,
da ase s, and he alchemy page
which will be explained sho ly. The
model page
is p o ided as shown
in igu e 45, mos columns a e sel -explana o y and con ain ele an model in o ma ion, he las wo
columns p o ide in o ma ion abou he benchma k s a e and explainabili y s a e, i he icon is g ey, he
sys em hasn’ compu ed he in o ma ion ye . A ed icon means ha some con en may no be possible
55
CHAPTER 6. RESULTS DEMONSTRATION AND DISCUSSION
Figu e 44: Ini ial sys em webpage
o ob ain, and a blue icon indica es ha all in o ma ion is a ailable. Clicking he bu on will ake he use
o he espec i e page. All in o ma ion in he able can be o de ed by column, sea chable by ex and
pagina ion cus omizable by he use .
Figu e 45: Model lis ing webpage
Simila o he model page, he
benchma k page
shown in igu e 46 shows he benchma ks pe o med
by he sys em, he main di e ence is ha a model can ha e one o mo e benchma ks associa ed so his
page will show each benchma k independen ly, he use can also access i s dashboa d and see all he
in o ma ion a ailable.
The
da ase ’s page
is demons a ed in igu e 47, and i con ains in o ma ion abou all he da ase s
indexed by he sys em, he que y ag, which is he human- eadable iden i ie , and all he e sions o he
da ase s p esen and he la es que y ID co esponden o he ag. Finally, he las column akes he use
56

Figu e 46: Benchma k lis ing webpage
Figu e 47: Da ase lis ing webpage
57
CHAPTER 6. RESULTS DEMONSTRATION AND DISCUSSION
o he da ase page, shown in igu e 48 and displays all a ailable e sions and a ull s a is ical epo abou
he gi en da ase .
The da ase de ail page is p o ided by he da a inges ion engine, possible due o he igh in eg a ion
be ween all he componen s in he pipeline and he achie ed in e ope abili y. i is o he highes ele ance
o p o ide da ase in o ma ion, as use s migh wan o consul he da ase used o aining and e alua ion
o each model o eplica e he esul s. The da ase in o ma ion is e y de ailed wi h s a is ic in o ma ion
abou a numbe o e en s, en i onmen condi ions and me ada a in o ma ion, gi ing he use a ully- ledged
epo on he da a.
Figu e 48: Da ase de ail webpage
One o he main issues o ha ing a ully au oma ed sys em ha compu es all in o ma ion in he backend
wi hou use inpu is he lack o knowledge abou he s a us o hose ope a ions. To ackle his issue, a
logging sys em was de ised ha ca ches checkpoin s in common ope a ions and s o es hei s a e; i
can be seen in igu e 49, he logging sys em also s o es excep ions ha happen in he sys em. To gi e
he use he logging in o ma ion, he admin dashboa d was c ea ed. I p o ides all he la es logs om
he sys em classi ied as in o o wa ning, i is also impo an ha new checkpoin s o logs can be added
wi h use eedback. This page also p o ides in o ma ion abou he sys em componen s a us, a manual
op ion o upload machine lea ning models ha do no come om he pipeline, bu he use s ill wan s
o in oduce in o he sys em, and a bu on o access he da abase managemen sys em which p o ides
CRUD ope a ions.
Accessing he
Manage da abase
bu on will ake he use o he page ep esen ed in igu e 50. This
page p o ides ull con ol o he ables in he da abase, allowing he sys em manage o sol e po en ial
excep ions o aul y models ha a i e in he sys em. I is expec ed ha mos hiccups can be ea ed by
consul ing he excep ion in he logs page and sol ing he unde lying p oblem in he da abase managemen
page.
58
Figu e 49: Admin dashboa d and log in o ma ion webpage
Figu e 50: Admin da ase managemen webpage
59
CHAPTER 6. RESULTS DEMONSTRATION AND DISCUSSION
The model benchma k esul s need an app op ia e dashboa d ha is easy o in e p e and deli e s
as much in o ma ion as possible wi hou o e whelming he use , i is also impo an o show he mul iple
benchma ks a model migh ha e and app op ia ely iden i y hem.
Figu e 51: Model benchma k page
The model benchma k page is shown in igu e 51. Each d awn ec angle in he pic u e e e ences
a di e en in o ma ional sec ion. A ea 1showcases model in o ma ion like he da ase ag i uses, da a
window o p edic ion and e en benchma k ime, i also has a sho cu o he model explainabili y which
will be explained la e . A ea 2is he me ics sec ion, i displays all he me ics conside ed ele an by he
da a scien is o he speci ic machine lea ning model wi h i s name and alue. A ea 3is he cha sec ion
ha displays benchma ks esul s in ime o each benchma k, his cha is c ucial o unde s and he model
e olu ion o each benchma k pe o med. A ea 4is he sec ion ha displays all he o he benchma ks
pe o med on he model o gi e he use a good e e ence, he da e o benchma k is he i s column which
allows o iden i y he ime ame o each benchma k.
Figu e 52 showcases he benchma k dis ibu ion sec ion o page 51, i displays he labels p edic ed
by he model and he ac ual g ound u h alue and is e y use ul o unde s and he model bo lenecks
and i he aining p ocess needs o be imp o ed.
The model hype pa ame e s page is an impo an ea u e o ha e since i allows o o he use s o
eplica e and imp o e upon al eady good models, i makes each machine lea ning model ep oducible,
his page is displayed in igu e 53, each box ep esen s one hype pa ame e , his in o ma ion is dynamic
and can wo k o any gi en compa ible machine lea ning algo i hm.
Finally, he las ea u e is he model explainabili y, o achie e his unc ionali y SHAP amewo k is used,
which is leading s a e o he a in algo i hm explainabili y. ML explainabili y is cu en ly a ho opic and
helps demys i y he
black box
eeling abou ml algo i hms, i is specially impo an o wo main pu poses:
60