Optimizing deep neural networks for resource-limited embedded systems

id192731


OPTIMIZING DEEP NEURAL NETWORKS FOR
RESOURCE-LIMITED EMBEDDED SYSTEMS
ESTEBAN GATEIN
Thesis supe iso
MARINAZAPATERSANCHO(HEIG-VDSchoolo Enginee ingandManagemen )
Tu o :JORDIDELGADOPIN(Depa men o Compu e Science)
Deg ee
Bachelo 'sDeg eeinA i icialIn elligence
Bachelo 's hesis
Facul a d'In o mà ica de Ba celona (FIB)
Uni e si a Poli ècnica de Ca alunya (UPC) - Ba celonaTech
22/01/2025

Op imizing Deep Neu al Ne wo ks o Resou ce-Limi ed Embedded Sys ems
Abs ac
In ecen yea s, applica ions based on a i icial in elligence (AI) echniques ha e be-
come inc easingly common, o he poin whe e deep neu al ne wo ks (DNNs) ha e been
applied ac oss almos any ield, p o iding solu ions o a wide ange o p oblems. Simul-
aneously, embedded sys ems ha e e ol ed o mee he g owing demand o AI in eg a-
ion. While mos con en ional mic op ocesso s ha e emained unable o p ocess DNNs
e icien ly, many esea che s ha e belie ed ha he u u e o embedded echnology lies in
inc easing he au onomy o edge de ices, enabling hem o un AI-based applica ions.
This bachelo hesis ocused on he implemen a ion o DNNs o au onomous d one
na iga ion, deploying hese ne wo ks on an ul a-low-powe mic op ocesso speci ically
designed o deep lea ning applica ions. Fu he mo e, i explo ed cu ing-edge model com-
p ession echniques o op imize he pe o mance o he DNNs and in es iga ed he limi a-
ions o his eme ging echnology in he con ex o embedded sys ems.
The esea ch examined a ious app oaches, including model quan iza ion, knowledge
dis illa ion, he implemen a ion o ea ly exi s, and edge- o-cloud s a egies, aiming o bal-
ance compu a ional e iciency and accu acy while espec ing he cons ain s o ul a-low-
powe ha dwa e.
2
Resum
En els da e s anys, les aplicacions basades en `
ecniques d’in el·lig`
encia a i icial (IA)
s’han o na cada egada m´
es habi uals, ins al pun que les xa xes neu onals p o undes
s’han aplica a gai eb´
e qualse ol camp, p opo cionan solucions a una `
amplia gamma de
p oblemes. Al ma eix emps, els sis emes in eg a s han e oluciona pe sa is e la c eixen
demanda d’in eg aci´
o de la IA. To i que la majo ia de mic op ocessado s con encionals
con inuen sen ine icien s pe p ocessa xa xes neu onals, mol s in es igado s c euen que
el u u de la ecnologia inc us ada passa pe augmen a l’au onomia dels disposi ius edge,
pe me en -los execu a aplicacions basades en IA.
Aques eball de i de g au s’ha cen a en la implemen aci´
o de xa xes neu onals pe a
la na egaci´
o au `
onoma de d ons, desplegan aques es xa xes en un mic op ocessado ul a
e icien dissenya espec´
ı icamen pe a aplicacions d’ap enen a ge p o und. A m´
es, s’han
explo a `
ecniques inno ado es de comp essi´
o de models pe op imi za el endimen de
les xa xes neu onals i s’han in es iga les limi acions d’aques a ecnologia eme gen en el
con ex dels sis emes inc us a s.
La in es igaci´
o ha examina di e sos en ocamen s, incloen -hi la quan i zaci´
o de mo-
dels, la des il·laci´
o de coneixemen , la implemen aci´
o de so ides an icipades i es a `
egies
edge- o-cloud, amb l’objec iu d’equilib a l’e ici`
encia compu acional i la p ecisi´
o espec-
an les limi acions del maquina i ul a e icien .
3
Op imizing Deep Neu al Ne wo ks o Resou ce-Limi ed Embedded Sys ems
Resumen
En los ´
ul imos a˜
nos, las aplicaciones basadas en ´
ecnicas de in eligencia a i icial (IA)
se han uel o cada ez m´
as comunes, has a el pun o de que las edes neu onales p o undas
se han aplicado en casi cualquie campo, p opo cionando soluciones a una amplia gama de
p oblemas. Al mismo iempo, los sis emas embebidos han e olucionado pa a sa is ace la
c ecien e demanda de in eg aci´
on de la IA. Aunque la mayo ´
ıa de los mic op ocesado es
con encionales siguen siendo ine icien es pa a p ocesa edes neu onales, muchos in es i-
gado es c een que el u u o de la ecnolog´
ıa embebida adica en aumen a la au onom´
ıa de
los disposi i os edge, pe mi i´
endoles ejecu a aplicaciones basadas en IA.
Es e abajo de in de g ado se ha cen ado en la implemen aci´
on de edes neu onales
pa a la na egaci´
on au ´
onoma de d ones, desplegando es as edes en un mic op ocesado de
ul a bajo consumo espec´
ı icamen e dise˜
nado pa a aplicaciones de ap endizaje p o undo.
Adem´
as, se han explo ado ´
ecnicas de angua dia de comp esi´
on de modelos pa a op i-
miza el endimien o de las edes neu onales y se han in es igado las limi aciones de es a
ecnolog´
ıa eme gen e en el con ex o de los sis emas embebidos.
La in es igaci´
on ha examinado a ios en oques, incluidos la cuan izaci´
on de modelos,
la des ilaci´
on de conocimien o, la implemen aci´
on de salidas emp anas y es a egias edge-
o-cloud, con el obje i o de equilib a la e iciencia compu acional y la p ecisi´
on espe ando
las limi aciones del ha dwa e de ul a bajo consumo.
4

CONTENTS
Con en s
1 In oduc ion 7
2 Con ex and S a e o he A 9
2.1 Low-Powe De ices o Edge Compu ing . . . . . . . . . . . . . . . . . . . . 9
2.2 Edge- o-Cloud S a egies o E iciency . . . . . . . . . . . . . . . . . . . . . 10
2.3 Mul i-Task Lea ning in Con olu ional Neu al Ne wo ks . . . . . . . . . . . . . 11
2.4 Op imiza ion o Low-Powe De ices: Model Comp ession . . . . . . . . . . . 12
2.5 Quan iza ion in Con olu ional Neu al Ne wo ks . . . . . . . . . . . . . . . . . 12
2.5.1 Pos -T aining Quan iza ion (PTQ): . . . . . . . . . . . . . . . . . . . . 13
2.5.2 Quan iza ion-Awa e T aining (QAT): . . . . . . . . . . . . . . . . . . 13
2.6 Ea ly-Exi S a egies in DNNs o Edge Compu ing . . . . . . . . . . . . . . . 14
2.7 Knowledge Dis illa ion in DNNs . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.8 Es ablished Wo k and P ojec Con ex . . . . . . . . . . . . . . . . . . . . . . 16
3 Da a Acquisi ion 18
3.1 CollisionDa ase ................................. 18
3.2 Udaci yDa ase .................................. 18
3.3 Fi e and Smoke De ec ion Da ase (FASDD) . . . . . . . . . . . . . . . . . . . 20
4 T aining o Resou ce-Limi ed Pla o ms 21
4.1 Quan iza ion-Awa e T aining (QAT) . . . . . . . . . . . . . . . . . . . . . . . 21
4.1.1 QAT o he D oNe model . . . . . . . . . . . . . . . . . . . . . . . . 21
4.1.2 Design o he adap ed model . . . . . . . . . . . . . . . . . . . . . . . 23
4.2 Smoke and Fi e De ec ion Neu al Ne wo k . . . . . . . . . . . . . . . . . . . . 26
4.3 Tes ing on he GAP9 Pla o m: Limi a ions and Deploymen P ocess . . . . . . 27
4.3.1 Se up o he GAP9 Pla o m . . . . . . . . . . . . . . . . . . . . . . . 27
4.3.2 NNTool and Au o ile o DNN Deploymen . . . . . . . . . . . . . . 27
5
Op imizing Deep Neu al Ne wo ks o Resou ce-Limi ed Embedded Sys ems
4.3.3 Challenges Du ing Deploymen . . . . . . . . . . . . . . . . . . . . . 28
4.3.4 T ansi ion o PyTo ch and ONNX . . . . . . . . . . . . . . . . . . . . 29
4.3.5 Adap a ions o GAP9 Deploymen . . . . . . . . . . . . . . . . . . . 29
5 Reducing DNNs o Resou ce-Limi ed Pla o ms 32
5.1 Ea ly Exi S a egies In es iga ion . . . . . . . . . . . . . . . . . . . . . . . . 32
5.2 Mul i-Objec i e DNNs wi h Ea ly Exi s . . . . . . . . . . . . . . . . . . . . . 33
5.2.1 Model wi h Ea ly Exi o Fi e and Smoke De ec ion . . . . . . . . . . 33
5.2.2 Model wi h Ea ly Exi s o Collision P edic ion and S ee ing Angle . . 35
5.3 Knowledge Dis illa ion o DNNs . . . . . . . . . . . . . . . . . . . . . . . . 36
5.4 Comp ession o he Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
6 Demons a o 40
6.1 ModelGene aliza ion............................... 40
6.1.1 Con ex Vec o s.............................. 40
6.1.2 Model T ans o ma ions . . . . . . . . . . . . . . . . . . . . . . . . . . 41
6.2 Deploymen wi h he Edge- o-Cloud se up . . . . . . . . . . . . . . . . . . . . 42
6.2.1 Se e Design and Implemen a ion . . . . . . . . . . . . . . . . . . . . 42
6.2.2 D oneBandi on GVSoC . . . . . . . . . . . . . . . . . . . . . . . . . 43
6.2.3 Deploymen o a Knowledge Dis illed Model . . . . . . . . . . . . . . 47
7 E alua ion and Resul s 48
8 Sus ainabili y Analysis and E hical Implica ions 53
9 Conclusions 55
Appendices 56
6
1 In oduc ion
1 In oduc ion
The in eg a ion o a i icial in elligence in o embedded sys ems has b ough impo an ad ance-
men s o many applica ions, making solu ions mo e e icien and sma e . Howe e , wo king
wi h esou ce-cons ained embedded sys ems makes i e y di icul o deploy complex deep
neu al ne wo ks. This hesis explo es me hods o o e coming hese challenges in op imizing
DNNs o ul a-low-powe de ices, wi h a ocus on au onomous na iga ion o d ones.
The p ojec se es as a con inua ion o he D oneBandi p ojec , which uses he GAP8 p oces-
so in eg a ed in o a nanod one o eal- ime neu al in e ence, p esen ing a he same ime an
algo i hm o de ine a dynamic edge- o-cloud s a egy o DNNs, wi h he objec i e o i e a i ely
choosing he op imal cu ing poin o he model. Based on his p ojec , his esea ch aims o
explo e he capabili ies o he GAP9 p ocesso o implemen a mul i- ask lea ning model capa-
ble o p edic ing collisions, s ee ing angles, and i e o smoke de ec ion. The p ojec has he
objec i e o comp essing such DNNs o make he deploymen easible while main aining a bal-
ance be ween op imiza ion and pe o mance loss. Addi ionally, he DNNs ha e o be designed
wi h he in en ion o wo king in a eal-wo ld use case de ined by a nanod one employing an
uns able edge- o-cloud se up, wi h he assump ion ha he GAP9 can be in eg a ed in o a d one
simila ly o he GAP8. The op imiza ion echniques conside ed and expe imen ed wi h ha e
he objec i e o sol ing he p oblem o he connec ion by he implemen a ion o an al e na i e
DNN wo king only on he edge de ice. The GAP9 p ocesso is o be pushed o i s limi s o
unde s and i s limi a ions and i s capabili ies, and he deploymen o DNNs on his chip has
as a goal demons a ing ha comp ession echniques a e a solu ion o esou ce-cons ained
embedded sys ems.
F om expe imen ing wi h he GAP8 p ocesso o he deploymen o models ained wi h cu ing-
edge comp ession echniques, his bachelo hesis will answe he ollowing ques ions:
• How can quan iza ion and ea ly-exi echniques imp o e he e iciency o neu al ne wo ks
o deploymen on he GAP9 mic op ocesso ?
• How e ec i ely can a comp essed neu al ne wo k p edic collision de ec ion, s ee ing
angles, and i e/smoke de ec ion on ul a-low-powe embedded sys ems?
• Wha is he maximum comp ession a io ha can be achie ed o neu al ne wo ks on
GAP9 wi hou comp omising ask-speci ic accu acy?
By he in eg a ion and he suppo o a esea ch g oup a he Recon igu able & Embedded
Digi al Sys ems (REDS) ins i u e a he HEIG-VD, his p ojec is ca ied ou in a esea ch
amewo k ha combines heo e ical explo a ion wi h p ac ical implemen a ion. This docu-
men s a s, he e o e, wi h a b ie exposi ion and de ini ion o he concep s used du ing he e-
sea ch p ojec . Then, he me hodology is de ined by exposing he design and decision-making
7
Op imizing Deep Neu al Ne wo ks o Resou ce-Limi ed Embedded Sys ems
p ocesses, ollowed by sec ions dedica ed o he p ac ical demons a ion and he e alua ion o
esul s. The esul s de ails, including he aining p ocesses o neu al ne wo k models, a e a ail-
able in he public Gi Hub eposi o y: Embedded & Op imized DNNs. This eposi o y p o ides
access o he codebase and sc ip s used o eplica e he esul s discussed in his hesis, showing
anspa ency and ep oducibili y o he wo k.
The p ac ical ou come o he p ojec includes he possibili y o using he GAP9 p ocesso by
he REDS ins i u e wi h he limi a ion o he p ocesso s al eady known, wi h he solu ions ha
can be applied o DNNs o make hei deploymen s easible. Ne e heless, as his p ojec is
a con inua ion o he D oneBandi p ojec , he mo e impo an ou come is he design o he
ools ha make he use o he algo i hm less complica ed. While he ange o ields in which
his algo i hm can be used o design inno a i e solu ions is as , i s implemen a ion can some-
imes be challenging due o esou ce limi a ions. The e o e, he solu ions p oposed in his wo k
mus be unde s ood as a way o minimizing he impac o hese challenges by making e ec i e
deploymen s mo e con enien .
8
2 Con ex and S a e o he A
Du ing he knowledge dis illa ion p ocess, he eache model is used o gene a e so a ge s o
p obabili y dis ibu ions o e he ou pu classes. These so a ge s encode addi ional in o ma-
ion abou he ela ionships be ween classes, which he s uden model lea ns du ing aining.
This auxilia y aining signal allows he s uden model o achie e highe accu acy han i would
using adi ional aining me hods based on g ound- u h labels. By using his app oach, i is
possible o ain models ha mee he s ic memo y and compu a ional equi emen s o edge
de ices like he GAP9.
Ano he c i ical ad an age o knowledge dis illa ion in edge scena ios is i s abili y o educe
he s uden model o speci ic asks o en i onmen s. Fo example, he dis illed model can be
ine- uned o p io i ize he mos ele an ea u es o he deploymen con ex , u he educing
compu a ional complexi y wi hou sac i icing pe o mance.
In p ac ice, he success o knowledge dis illa ion elies on e ec i ely combining he loss unc-
ions associa ed wi h he so a ge s gene a ed by he eache model and he ha d labels. The so
a ge s a e ypically ep esen ed as a p obabili y dis ibu ion o e he ou pu classes, smoo hed
by a empe a u e pa ame e T o ampli y he ela i e di e ences be ween class p obabili ies.
The Kullback-Leible (KL) di e gence is commonly used as he loss unc ion o aligning he
s uden model’s ou pu dis ibu ion qwi h he eache ’s so ened ou pu dis ibu ion p, de ined
as:
LKD =T2KL(p||q) = T2∑
i
pilog pi
qi
,
whe e piand qideno e he p obabili ies o class ip edic ed by he eache and s uden models,
espec i ely, and T2scales he loss o he e ec o empe a u e T.
This loss is combined wi h he s anda d c oss-en opy loss LCE calcula ed using he labels y o
guide he s uden model’s lea ning. The o e all objec i e unc ion is a weigh ed sum o he wo
componen s:
L o al =αLCE +(1−α)LKD,
whe e α∈[0,1]is a hype pa ame e ha con ols he balance be ween he ha d label supe ision
and he knowledge dis illa ion om he eache . By uning αand T, he s uden model can
lea n o imi a e he eache ’s beha io while also being es ic ed o he ue labels, esul ing
in a model ha achie es an op imal balance be ween accu acy, model size, and compu a ional
e iciency.
15

Op imizing Deep Neu al Ne wo ks o Resou ce-Limi ed Embedded Sys ems
Figu e 2: C azy lie d one compa ed o a 0.01 C coin. [Palossi e al., 2019]
2.8 Es ablished Wo k and P ojec Con ex
I was be o e s a ing his hesis wo k and joining he esea ch collec i e ha he main p ojec on
which his wo k is based was al eady in mo ion. The las a emp was di ec ed o he o ma ion
o a amewo k o using an unmanned ae ial ehicle (UAV) o apply eal- ime neu al image
p ocessing. I was mo e abou de eloping a s a egy o ge ing in o ma ion om he edge o
he cloud ha would make he d one a oid hese ba ie s dynamically and in eal ime. This ask
was o sa is y a ious cons ain s o limi ed compu a ional esou ces and ene gy a ailabili y,
which a e usually a ibu ed o nanod ones.
The p ojec unco e ed he ac ha he ligh weigh con olu ional neu al ne wo ks based on he
use o he GAP8 p ocesso , an SoC sys em, had been deployed. The p ocesso is a cu ing-
edge piece o ha dwa e specially de eloped o pu low-powe AI applica ions in o ac ion. The
chip was in eg a ed in o he C azy lie d one, which is a modula and e sa ile nanod one buil
by Bi c aze and o be used wi h he AI deck, he ca y-on module ha is speci ically made o
imp o e he d one’s p ocessing speed in machine lea ning asks. The size o he d one can be
app ecia ed in Figu e 2. This achie emen e ealed he possibili y o unning a neu al ne wo k
in e ence on a esou ce-cons ained nanod one, which is a ough p ojec because o he d one’s
s ic size, weigh , and powe limi a ions.
Rela ed o he echnical p oblems wi h he use o a CNN-based applica ion pe o med on a nan-
od one, p oblems wi h he edge- o-cloud in e ence p ocess we e also deal wi h. All o his was
made possible by he de elopmen o D oneBandi , which is an online decision-making algo-
i hm ope a ing in he a ea o mul i-a med con ex ual bandi s. The p ima y ocus o D oneBan-
di , as p esen ed in [Chacun e al., 2024], is o ind he mos op imal pa o cu so as he edge-
o-cloud s a egy can be ealized mos e icien ly du ing e e y in e ence i e a ion. To do his,
he algo i hm dynamically assumes he mos sui able dis ibu ion o he compu a ional asks
16
2 Con ex and S a e o he A
be ween he d one (edge) and he se e cloud in he p ocess by es ima ing he da a amoun and
ime o la ency o he connec ion. Delays caused by hese ac o s a e minimized.
The p edic i e modeling by D oneBandi es ima es in e ence ime on he edge and he cloud,
enabling i o dynamically adjus he s a egy in eal ime, based on each speci ic con ex and
adap ing o he WiFi condi ions. This no only ensu es la ency educ ion bu also ensu es ha
he limi ed d one esou ces a e e icien ly u ilized o p o ide smoo he and mo e eliable obs a-
cle a oidance du ing ligh . This pape has pu sued he in eg a ion o edge compu ing, cloud
p ocessing, and ad anced decision algo i hms in o de o es ablish a solid basis o u u e e-
sea ch on collabo a i e in e ence s a egies in esou ce-cons ained en i onmen s.
Gi en he desc ip ion o he es ablished wo k, his bachelo hesis will con inue he D oneBandi
p ojec by p oposing a mo e complex use case i ing o he es ic ions o he edge- o-cloud
amewo k and p opose solu ions o o e come he encoun e ed limi a ions.
17
Op imizing Deep Neu al Ne wo ks o Resou ce-Limi ed Embedded Sys ems
3 Da a Acquisi ion
The i s s ep o he p ojec has been ob aining he da a o he models. Th ee di e en da ase s
ha e been used, all o hem con aining housands o images wi h hei co esponding labels o
each ask. In his sec ion a desc ip ion o each da ase is gi en, as well as he p ocess ollowed
o ob ain hem.
Figu e 3: Image examples o he collision da ase . [Loque cio e al., 2018]
3.1 Collision Da ase
The i s da ase collec ed is he one c ea ed by he D oNe p ojec , he collision da ase , which
has also been used o he D oneBandi p ojec . The da a is composed o di e en olde s ha
con ain images o ideos aken by a came a moun ed on a bike ilming h ough a ci y. The da a
has been labeled o indica e whe he he e is an imminen collision o no in each ame and
has been s uc u ed o i he aining pipeline o D oNe . A da a sample wi h i s co esponding
labels can be obse ed in Figu e 3.
The da a can be downloaded h ough he webpage o he p ojec , wi h he s uc u e needed o
un he model, which can be ound in [Loque cio e al., 2018]. The quali y o he da a has no
been analyzed, as sa is ac o y esul s a e eached in bo h p ojec s using his da a.
3.2 Udaci y Da ase
The second da ase collec ed is he Udaci y Da ase , which con ains images and senso da a
om an au onomous ca , as i can be obse ed in Figu e 4. Conc e ely, he images a e collec ed
om h ee di e en came as disposed o in he cen e and on bo h sides o he ca and iles
con aining in o ma ion abou he con ol o he ca , including he angle o he s ee ing wheel.
The ins uc ions o ob ain his publicly a ailable da a a e desc ibed in he D oNe p ojec .
18
3 Da a Acquisi ion
Figu e 4: Sample images om he udaci y da ase . [Loque cio e al., 2018]
The i s s ep has been o download he da a iles, which a e a ailable in a o en o ma , he
iles o download can be ound as explained in [Loque cio e al., 2018]. Then, a o en clien
has been used o download he da a i sel ; in his case, open-sou ce so wa e has been used,
qBi o en .
A e downloading he da a, he o ma ob ained is comp essed images in he Robo Ope a ing
Sys em (ROS) bag2 o ma . These bags con ain he sequences o images and da a ex ac ed
di ec ly om he ca and need o be p ep ocessed in o de o be able o eed he aining pipeline
o a model. In he D oNe p ojec , i is ecommended o use a conc e e eposi o y, which can
be ound in [Rwigh man, 2019]. A e ying o ins all one e sion om ROS in he ope a ing
sys em ha was used, i has been obse ed ha he con aine ha has o be used acco ding
o Rwigh man was no able o wo k. A i ual machine has been ins alled unning Ubun u
18.04, as he con aine gi en by he eposi o y and he ROS e sion needed ha e been es ed
on his e sion o Ubun u. E en ha ing he compa ible so wa e, he con aine s needed o be
upda ed, conc e ely in he Docke ile3whe e he commands o download he equi emen s we e
ou da ed. Also, he s uc u e o he di ec o ies con aining he ROS bags has been changed, as
he con aine is sensi i e o he numbe o iles in each di ec o y. Then, execu ing he con aine
p oduced, o each ROS bag, he h ee se s o images o he came as o he ca and he iles
con aining he in o ma ion o he senso s.
Addi ionally, o ob ain he use ul da a o aining a model, ano he sc ip p o ided by D oNe
has o be execu ed, which maps da a om he ca senso s and ma ches he angle o he s ee ing
wheel wi h he images o he cen al came a, using he imes amps o he da a.
Finally, he di ec o y s uc u es o he da a ha e been o ganized in h ee main se s: aining,
alida ion, and es . In each one o hem, i can be seen ha di e en di ec o ies a e con-
ained, co esponding o di e en ideo sequences and e e ed o om now on as expe imen s,
ma ching wi h he s uc u e o he collision da ase . The e o e, bo h da ase s ha e been me ged
wi hou any u he analysis, o he same eason as in he collision da ase .
2Mo e in o ma ion can be ound on h p://wiki. os.o g/Bags.
3Docke iles a e iles used by Docke , an open-sou ce pla o m used o deploy applica ions in con aine s, ha
c ea es and ini ializes he en i onmen .
19
Op imizing Deep Neu al Ne wo ks o Resou ce-Limi ed Embedded Sys ems
3.3 Fi e and Smoke De ec ion Da ase (FASDD)
The i e and smoke de ec ion da ase (FASDD) con ains housands o images o i e, smoke, i e
and smoke, o nei he o hem in di e en se ups, as shown in he sample in Figu e 5. This open-
access da ase is cons uc ed o be able o ain e y accu a e i e de ec ion models and con ains
labels cons uc ed o conc e e models used in he s a e o he a compu e ision con ex , such
as YOLO 9.
No deep analysis has been ealized o measu e he quali y o he da a, as he au ho s o he
da ase p o ide all o he needed in o ma ion wi h some benchma ks es ing he da a and mak-
ing some expe imen s wi h i in [Wang e al., 2024]. E en so, he da a s uc u e has been eo -
ganized in o de o only keep he da a om he classes o each image and ma ch he s uc u e o
he o he wo da ase s, so gene a ing simula ed expe imen s wi h balanced classes by andomly
choosing he images om he di e en classes.
Figu e 5: Sample images om he FASDD. [Wang e al., 2024]
20

4 T aining o Resou ce-Limi ed Pla o ms
4 T aining o Resou ce-Limi ed Pla o ms
GAP8 and GAP9 a e examples o esou ce-cons ained pla o ms designed o wo k unde p o-
cessing cons ic ions, limi ed memo y, and low ene gy consump ion. These cons ain s igge
new app oaches o adap a ion o a model, such as quan iza ion, which makes low-bi o ma
ypes o model pa ame e s wi hou making much di e ence o accu acy. The pu pose is o c e-
a e ligh weigh and e icien models ha mee he edge- o-cloud equi emen s while achie ing
high-quali y pe o mance.
This sec ion i s elucida es he adap a ion o D oNe o quan iza ion-awa e aining o achie -
ing e ec i e in e ence using in ege s. A comp ehensi e o e iew is made on he s eps o de-
signing and adap ing he model o he la es amewo ks’ and he speci ic needs o esou ce-
cons ained en i onmen s. I also ou lines he p og ession owa ds de eloping a low-weigh
neu al smoke de ec ion ne wo k ha ocuses on edge de ices wi h less a chi ec u e and e i-
cien implemen a ion.
These challenges pa e he jou ney o model deploymen on he GAP9 pla o m, in addi ion o
pu ing in o pe spec i e some limi a ions wi h GAP9 as well as solu ions pu in place. The ou -
come o such an expe imen o e s an insigh on how he design o neu al ne wo ks op imized
o esou ce-cons ained pla o ms is p oduced. In he case o his p ojec , e en i i is p esen ed
as a con inua ion o he D oneBandi p ojec deployed on GAP8, only he GAP9 p ocesso will
be conside ed. The eason o his decision is based on he upg ades gi en by he GAP9 p o-
cesso compa ed o i s p e ious e sion, and s a ing a p ojec wi h his new p ocesso unlocks
he possibili y o ansi ioning he con inuing ongoing p ojec s wo king wi h he GAP8 o he
GAP9 p ocesso in o de o ake ad an age o he new ea u es o he p ocesso . The di e ence
be ween p ocesso s can be consul ed in [Sizo a, 2024].
4.1 Quan iza ion-Awa e T aining (QAT)
The ini ial pa o he p ojec has been o in es iga e he possibili y o aining he model ha
was deployed on he GAP8 using QAT in o de o a oid he loss o p ecision induced by he
PTQ. Addi ionally, wi h he objec i e o imp o ing he use case gi en by he p ojec , which was
he collision a oidance, one o he main objec i es has been adding a ask o be sol ed by he
d one o a gain o au onomy.
4.1.1 QAT o he D oNe model
The i s model conside ed in his wo k is he one designed in he D oNe p ojec , a esidual
neu al ne wo k o med by blocks o con olu ional laye s ha ou pu s wo di e en esul s: he
21
Op imizing Deep Neu al Ne wo ks o Resou ce-Limi ed Embedded Sys ems
p obabili y o collision o he d one wi h an obs acle and he s ee ing angle needed o a oid i ,
ained wi h he da ase s p e iously in oduced in 3.1 and 3.2.
Deploying a model in a quan ized o m is a es ic ion gi en by he GAP8, as i is impossible
o he p ocesso o handle loa ing poin ope a ions and numbe s. I is impo an o ema k
ha his is no a es ic ion o he GAP9, as i can wo k wi h loa ing poin ope a ions wi h
32-bi p ecision, bu his op ion limi s he possible size o he model and won’ be aligned
wi h he objec i e o maximizing he comp essions o he models o he maximum; he e o e,
his al e na i e is no conside ed in his wo k. Gi en he possibili y o imp o ing he esul s
obse ed in he D oNe p ojec and he ac ha he model could be deployed esol ing mo e
han one ask in he GAP8, he i s s ep has been abou p epa ing he o iginal model o a
aining pipeline in QAT.
As he D oNe p ojec con ains he equi emen s o he en i onmen needed o e alua e a p e-
ained e sion o he model o o ain he model again. The i s p oblem ha has been encoun-
e ed wi h he code o he p ojec is abou he e sion compa ibili y o he deep lea ning lib a y
used o he aining o he models. In ac , he Tenso Flow e sion used o he p ojec is 1.5.0,
eleased in 2018, which is no able o wo k wi h he QAT ools om Tenso Flow, as hey a e
way mo e ecen . O he p oblems ha ha e been obse ed a e he lack o possibili y o making
in e ences wi h a single image, which can be use ul o obse e he punc ual esul s and e alua e
models on a gi en se o images; his has been sol ed by c ea ing a sc ip ha would ead an
image, load a p e ained model wi h he equi ed a chi ec u e, and un he in e ence; and he
impossibili y o modi y he model easily o he in oduc ion o he al e na i e ne ha has o be
able o un on he edge exclusi ely. Gi en hese limi a ions, he decision has been aken o no
use he model as is bu o ec ea e a simila e sion using he la es e sion o Tenso Flow o
ain he model ia QAT and include he solu ion o he al e na i e ne wo k. E en hough he
use o he o iginal code o he p ojec is no conside ed, he model has been un o in e ence,
so a b ie desc ip ion o he a chi ec u e is p o ided, which can also be obse ed in Figu e 6.
The model is composed o h ee esidual blocks, each one o hem o med by wo con olu ional
laye s. Be o e he i s con olu ional block, he i s con olu ional laye can be ound, and he
wo objec i es o he ne a e p eceded by a single ully connec ed laye o each objec i e.
22
4 T aining o Resou ce-Limi ed Pla o ms
Figu e 6: A chi ec u e o he D oNe model. [Loque cio e al., 2018]
4.1.2 Design o he adap ed model
Wi h he objec i e o unning an al e na i e ne wo k exclusi ely on he edge ne wo k o he
momen s in which he connec ion wi h he se e is mal unc ioning, a solu ion has been p o-
posed. This solu ion combines adap ing he D oNe model o he new e sion o he amewo k
o ain he model wi h QAT and adding he p e iously men ioned al e na i e ne . A e an ini-
ial app oach no conside ed in his wo k exposed in Appendix A, he solu ion e ol ed owa ds
including ano he ou pu o he D oNe model. The idea elies on ha ing a model capable o
p edic ing he p obabili y o collision, he s ee ing angle o a oid i , and ano he classi ie , wi h
he objec i e o p edic ing whe he he zone is sa e o land in o no . This las classi ie has
o gi e he ou pu on he edge, so i s b anch has o be de ined a e he i s laye s, and he
algo i hm deciding whe e he cu ing poin is o he edge- o-cloud s a egy has o be es ic ed
o making decisions conside ing only he cu ing poin s a ailable a e his ou pu . The i s
p oblem ha has been conside ed o a ask like his is he da a, as no public da ase s we e
a ailable wi h landing zone images o d ones o UAV came a images wi h labels indica ing i
i could land o no , a oiding any dange . To o e come his, FASDD has been used, gi en ha
he images con ain ou di e en classes (as exposed in Sec ion 3.3), his da ase is conside ed
as an al e na i e e en mo e complex han a bina y classi ie . Addi ionally, he possibili y o
p edic ing whe he he e is smoke, i e, bo h, o nei he wi h an in e io nanod one is a use case
use ul enough o be conside ed.
The model designed is simila o he one p esen ed in he D oNe p ojec , emo ing he esidual
connec ions as i is a es ic ion om D oneBandi , he algo i hm used o he edge- o-cloud
s a egy. I is composed o h ee con olu ional laye s, ollowed by he b anch wi h he ou pu o
he i e and smoke de ec ion and a b anch wi h ou mo e con olu ions and he o he wo ou pu s.
A schema o he model can be obse ed in Figu e 7, howe e , no e ha some con olu ion laye s
include ba ch no maliza ion laye s, and ac i a ions a e no ep esen ed.
23
Op imizing Deep Neu al Ne wo ks o Resou ce-Limi ed Embedded Sys ems
Inpu
Con olu ion
Pooling Con olu ion
Con olu ion
Pooling
Fully Connec ed FASDD
Con olu ion Con olu ion
Con olu ion Con olu ion
Fully Connec ed
Collision
Fully Connec ed
S ee ing Angle
Figu e 7: A chi ec u e o he adap ed model.
The main challenge o ain a model like his one is abou he labels, as o each image, he
ne wo k ou pu s h ee labels (one o each ask) bu is ained wi h only one label pe image
a he ime. A i s , i was expec ed o c ea e a lo o p oblems du ing he aining, bu i has
been obse ed ha his also was a p oblem o e come in he D oNe p ojec by igno ing he loss
o he asks when he label was no a ailable. E en i he heo e ical solu ion was a ailable,
he cus omized loss unc ions had o be designed om sc a ch, as well as he da a loade s, due
o he dep eca ion o he code and he impossibili y o ain he model by using masks in he
aining p ocess o a Tenso Flow model.
To make e e y hing wo k, he da a loade s ha e had o be edesigned o ead and p ep ocess
he images om he di e en expe imen s. The p ep ocessing is only abou eading he images
in g ayscale, esizing he images o a 200x200 pixel de ini ion, as he came a o he d one
conside ed has his esolu ion, and scaling he alues be ween 0 and 1. The missing labels ha e
been eplaced by NaN (No a Numbe ) alues in o de o ha e he same s uc u e o each image
o eed he aine o he model.
In he case o he loss, i has been obse ed ha aining he model wi h NaN alues was in-
oducing e o s and p oducing NaN alues back, which would make he loss impossible o
compu e. Because o ha eason, hese alues ha e been eplaced by a nume ical symbolic
alue ha could be de ec ed and igno ed by he pe sonalized loss unc ions, in his case, 999.
Th ee di e en loss unc ions ha e been de ined: one o he bina y classi ie o he collision
ask compu ing he bina y c oss-en opy, one o he mul i-class classi ie o he FASDD, and
one o he eg ession ask co esponding o he s ee ing angle and compu ing he mean squa ed
e o (MSE). Each one o hem has been de ined using he Tenso Flow ope a o s, as hey ha e
o wo k on enso s. Find each loss unc ion de ined as ollows:
24
4 T aining o Resou ce-Limi ed Pla o ms
Cycles Time (ms) Ene gy (µJ) In e ences pe sec.
In e ence 1 6255513 16.90679189 0.1892103732 59.14782688
In e ence 2 6253756 16.90204324 0.1891572293 59.16444454
In e ence 3 6254906 16.90515135 0.1891920133 59.15356682
In e ence 4 6254092 16.90295135 0.1891673923 59.16126594
In e ence 5 6254313 16.90354865 0.1891740768 59.15917544
In e ence 6 6253601 16.90162432 0.189152541 59.16591097
In e ence 7 6254643 16.90444054 0.1891840583 59.15605415
In e ence 8 6253591 16.9015973 0.1891522385 59.16600558
In e ence 9 6253762 16.90205946 0.1891574108 59.16438777
In e ence 10 6253501 16.90135405 0.1891495163 59.16685709
A e age 6254167.8 16.90315622 0.189169685 59.16054891
Table 1: Time and ene gy consump ion on single in e ence o he FASDD ask.
he compu e is limi ed, bu hanks o he e alua ion ki , some da a abou he execu ion can be
e ie ed. The esul s a e shown in Table 1, whe e i can be seen ha he numbe o cycles is no
always he same, e en i he numbe o ope a ions o compu e is, o ealizing his in e ence,
exac ly 33913280 ope a ions a e ealized. The ime is calcula ed by di iding he numbe o
cycles by he equency o he boa d, in his case, 370 MHz, and he ene gy is calcula ed by
mul iplying he numbe o ope a ions by he e iciency and by he ime, and he e iciency o he
boa d is known o be abou 3.3×10−13 W/ope a ion, acco ding o [Sizo a, 2024]. Finally, i is
impo an o ema k on he las column o Table 1, whe e he in e ences pe second ha could
be eached a e shown; almos 60 in e ences pe second is an accep able alue o he use case
o a nanod one lying in an in e io se up.
31

Op imizing Deep Neu al Ne wo ks o Resou ce-Limi ed Embedded Sys ems
5 Reducing DNNs o Resou ce-Limi ed Pla o ms
5.1 Ea ly Exi S a egies In es iga ion
Inspi ed by he way he auxilia y ne wo k has been designed as an exi ha will always gi e
an ou pu in he edge de ice, ea ly exi s a e conside ed in his wo k o lowe he wo kload o
he da a ans e be ween he edge de ice and he se e . These ea ly exi s ha e he objec i e o
esol e one o mo e han one ask ha is al eady esol ed by he whole ne wo k. The ou pu o
hese b anches has o be gi en inside he edge de ice, so he cu ing poin will ha e o be de ined
a e hese exi s. The main idea is o ha e an ou pu on which he con idence o he p edic ion
can be es ima ed, gi en a h eshold o con idence. I he p edic ions gi en by he ea ly exi a e
con iden enough, he model has o s op he execu ion he e, so he e won’ be da a ans e o
ha i e a ion; o he wise, he model has o keep wi h he execu ion and send he in e media e
ou pu s o he op imal cu ing poin o he i e a ion o he se e in o de o end he execu ion.
Fo he con idence measu emen , i is easy o de ine o he classi ie s, as he p obabili y o
p edic ed classes will indica e how much he model belie es an image belongs o one o he
classes. I has o be no ed ha he ou pu s om he models on he GAP9 p ocesso s a e in a
ange om -128 o 127, and ha he so max unc ion is sensi i e o he scale o he ou pu s, so
he esul s will always show high con idence in he ou pu s. To o e come his, he ou pu s o he
classi ie s ha e o be no malized, which is no a ha d ask as he ange o he alues is known.
Once he ou pu s a e no malized, a so max unc ion can be applied (no usually p esen in he
model as PyTo ch applies i in e nally in he loss unc ions), and a decision can be aken.
Con idence measu e canno ely on class p obabili ies as in he case o classi ie s o eg ession
asks. Ins ead, an app oach has been designed o es ima e he con idence o he ea ly exi
p edic ions. A Lasso eg ession model is ained o p edic he e o o he main eg ession
model based on he in e media e ou pu s o he ea ly exi . This e o p edic ion is a p oxy o
con idence, in ha he lowe he p edic ed e o , he highe he con idence in he ea ly exi ’s
p edic ion. In his way, he sys em se s a h eshold on accep able e o and decides whe he he
ea ly exi is con iden enough o s op execu ion o needs o con inue he p ocessing o he inpu
u he h ough he main ne wo k. The aining and applica ion o he Lasso eg ession model
will be elabo a ed in la e sec ions.
A ew models we e de eloped o es he ea ly exi s a egy, using a numbe o di e en con-
igu a ions and s a egies: some by using di e en numbe s o ea ly exi s, o he s by a ying
hei placemen wi hin he ne wo k, and s ill o he s by es ing al e na e me hods o con idence
es ima ion. Besides, mul iple con idence h esholds ha e been ied o balance he ade-o be-
ween he accu acy o p edic ions and educ ion in da a ans e and compu a ional wo kloads.
In ac , all hese s a egies a e e alua ed agains hei impac on pe o mance me ics such as
32
5 Reducing DNNs o Resou ce-Limi ed Pla o ms
o e all accu acy, la ency, and ene gy consump ion. The esul s o hose expe imen s a e in-
cluded la e in his wo k and p esen se e al iews on how ea ly exi s could be implemen ed
and op imized.
5.2 Mul i-Objec i e DNNs wi h Ea ly Exi s
Th ee di e en DNNs ha e been de eloped wi h ea ly exi s, one o hem al eady p esen ed in
Sec ion 4.2, wo king wi h an ea ly exi o e alua e he impac o ha ing an ea ly ou pu like
he one p esen ed in Sec ion 4.1.2 o he GAP9 mic op ocesso . Two o he models ha e been
de eloped wi h ea ly exi s and in eg a ing one o mo e han one s a egy wi h he objec i e o
educing he compu a ional cha ge on he chip. Fo he model esol ing he FASDD ask, he
s a egy is no implemen ed no conside ed in his sec ion, as i is no esol ing he p ima y
asks ha ha e o be sol ed o make a d one au onomous. Consequen ly, he o he wo models
a e he ones ha a e p esen ed in his sec ion.
5.2.1 Model wi h Ea ly Exi o Fi e and Smoke De ec ion
The i s model de eloped wi h an ea ly exi s a egy has been a model esol ing he h ee asks
conside ed in his wo k: he objec collision p edic ion, he s ee ing angle eg ession o a oid
collision, and he classi ie o indica e whe he he e is i e, smoke, bo h, o nei he in he image.
The ea ly exi o his model is conside ed o be only on he i e and smoke de ec ion, as i is a
mul i-class classi ie and i is easy o ob ain a con idence measu e om i s ou pu s, as i can be
done conside ing he p obabili y o each class gi en by a so max unc ion applied o he ou pu s
as a o m o con idence. Conside ing he ou pu s o he so max unc ion is a i s app oach o
ob ain a o m o con idence o he model; in u u e implemen a ions, a deepe analysis on how
o ex ac con idence on quan ized ou pu logi s should be done.
Inpu
Con olu ion
Con olu ion
Pooling Con olu ion
Con olu ion
Pooling
Con olu ion
Fully Connec ed
FASDD
Con olu ion
Con olu ion Pooling
Fully Connec ed
Collision
Fully Connec ed
FASDD
Fully Connec ed
S ee ing Angle
Figu e 10: A chi ec u e o he model wi h an ea ly exi o he FASDD ask.
33
Op imizing Deep Neu al Ne wo ks o Resou ce-Limi ed Embedded Sys ems
The model is based on he one de eloped in Sec ion 4.1.2, as i has o esol e he same asks in
he same way bu add he inal ou pu o he FASDD ask. The esul ing model is o med by
ou con olu ional laye s, ollowed by wo b anches, one wi h he ea ly exi wi h one mo e con-
olu ional laye and a ully connec ed laye and he o he b anch wi h wo mo e con olu ional
laye s and he h ee exi s, as displayed in Figu e 10.
The design and he aining o he model a e pe o med using PyTo ch, by using simila ech-
niques o handling he labels as done o he model on which i is based. The da a loade s
ha e been designed o load he da a, scale i be ween -1 and 1, and c ea e he labels s uc u e by
illing he missing alues wi h a symbolic numbe ha can easily be de ec ed in he pe sonalized
loss unc ions, in his case -9999. In he case o he classi ie , ze os a e used on he ull ec o
o he labels, which is also easy o de ec by summing up all he alues o he ec o , esul ing
in one i he label is alid and ze o o he wise. Fo he aining loop, he losses om he Py-
To ch amewo k can be used by sending he alid labels and he co esponding p edic ions o
each pic u e. In his case, he model is ained by coun ing he loss o he alid labels in each
ba ch, wi hou conside ing ha no alid labels could be ound on a conc e e ba ch. Also, o he
FASDD ask, he loss is compu ed wo imes, as one will coun o he ea ly exi and one o
he inal ou pu , which can gi e wo di e en alues. The aining is also moni o ed by com-
pu ing some me ics ha a e p in ed and sa ed o obse e i s e olu ion, as i is compu a ionally
expensi e o ain a model as his one. The me ics using PyTo ch ha e o be compu ed manu-
ally, so di e en coun e s ha e been de ined o coun how many labels a e igh and how many
labels a e conside ed in o al, which allows o ex ac me ics as he accu acy o he collision
p edic ion and he FASDD ask, sepa a ing he ea ly exi om he inal ou pu , and he MSE o
he eg ession ask. The moni o ing o he model has been done bo h on he aining and he
alida ion pa i ions o he da a, allowing o obse e whe he o e i ing4is p esen o no .
The s a egy i sel is designed in o de o make he ea ly exi esol e he ask only when he
model is con iden enough o gi e an answe wi hou unning all o he ne wo k. To do ha , a
so max unc ion is applied o he logi s o he classi ie in he ea ly exi , and a h eshold has
o be de ined in o de o decide whe he he con idence is enough o s op he execu ion o his
ask o no . The de ini ion o he h eshold has been done by i e a ing o e possible h eshold
alues, making in e ences o e a es ing pa i ion o he da a, and obse ing he accu acy wi h
each alue, wi h he objec i e o maximizing he accu acy o he ea ly exi . By ollowing
his p ocess, he bes alue obse ed as a con idence (and he e o e, p obabili y o one o he
classes) h eshold is 0.95, which inc eases he o e all accu acy o he ask, as he only samples
ea ed by he inal ou pu o he model a e he ones whe e he ea ly exi is no su e. In Sec ion
7, a mo e de ailed analysis o he compu a ional sa ings is gi en when deploying his s a egy.
Addi ionally, he ea ly exi s a egy has o be implemen ed in he ONNX g aph o make i
4Obse ed du ing he aining o he model by wa ching he loss o he alida ion da ase , as an explosion o i s
alue would indica e ha he model is lea ning o esol e he conc e e samples o he aining pa i ion.
34
5 Reducing DNNs o Resou ce-Limi ed Pla o ms
wo k on he GAP9 p ocesso . As his ea ly exi is no s opping he ull execu ion o he DNN,
he execu ion g aph needs o be modi ied in o de o e i y i he con idence c i e ia a e me
by he ea ly exi and, in he case ha hey a e, execu e only he wo o he asks, sa ing he
compu a ional cha ge o he inal ou pu o he FASDD ask. To e ec his change, he ONNX
model is i s loaded h ough he ONNX Py hon lib a y. An ea ly exi s a egy is added by
inse ing an I node in he ONNX g aph ha checks he con idence c i e ia o he ea ly exi
ou pu . Con idence is calcula ed wi h he no malized ou pu s o he classi ie and hei so max
p obabili ies. This means ha i he con idence o he p edic ion is g ea e han he se h eshold,
hen he low o execu ion will skip he compu a ion o he inal ou pu o he FASDD ask
and p oceed wi h execu ing only he o he wo asks. The desc ibed condi ional logic was
p og amma ically in oduced in o he ONNX g aph. The con idence is compu ed using he
ou pu s o he ea ly exi classi ie , and he compa a o node checks whe he he con idence
exceeds he h eshold. The I node hen b anches he execu ion g aph in o wo pa hs, whe e one
pa h bypasses he inal laye s esponsible o he FASDD ask when con idence is enough and
ano he pa h i he con idence does no mee he ba . Adding he se e al ONNX ini ialize s and
enso s o main ain con idence e alua ion and condi ional execu ion logic ex ends his u he .
This modi ica ion allowed u ning he ea ly exi s a egy wi h he GAP9 p ocesso compa ible
and e icien .
5.2.2 Model wi h Ea ly Exi s o Collision P edic ion and S ee ing Angle
A second model has been de ined wi h he objec i e o implemen ing an ea ly exi s a egy ha
s ops he ull execu ion o he model. As i is no a complex ask o measu e con idence on a
mul i-class classi ie , i is no he same case o a eg ession esol ed wi h a DNN. The e o e,
a model esol ing only he collision p edic ion and he s ee ing angle eg ession, in eg a ing an
ea ly exi o each one o hem, and wi h he objec i e o ully s opping he execu ion o he
model a e he ea ly exi .
The model, as shown in Figu e 11, is o med by ou con olu ional laye s, ollowed by he wo
ea ly exi b anches ha include wo ully connec ed laye s, each one, and he main b anch wi h
wo mo e sha ed con olu ional laye s and he las bi u ca ion, including he wo ou pu s wi h a
ully connec ed laye o each ask.
The aining code has been implemen ed simila ly o he p e iously p esen ed model, by con-
side ing he loss o he alid labels only, using symbolic alues o iden i y hese missing alues
while espec ing he s uc u e o he da a, and c ea ing bina y masks o a oid calcula ing he
loss o e hese missing alues.
Two s a egies ha e been conside ed o he implemen a ion o he ea ly exi s. The i s one
is based on he con idence o he bina y classi ie indica ing whe he he model is con iden
35
Op imizing Deep Neu al Ne wo ks o Resou ce-Limi ed Embedded Sys ems
Inpu
Con olu ion
Con olu ion
Pooling Con olu ion
Con olu ion
Pooling Con olu ion Con olu ion
Pooling
Con olu ion
Fully Connec ed
Collision
Con olu ion
Fully Connec ed
S ee ing Angle
Fully Connec ed
Collision
Fully Connec ed
S ee ing Angle
Figu e 11: A chi ec u e o he model wi h wo ea ly exi s esol ing wo asks.
enough o ake he ou pu as an answe o no . This i s s a egy elies on s opping all o he
execu ion i he ea ly exi o he collision ask is aken, also aking he esul gi en by he ask o
he s ee ing angle collision, e en i he e a e no p esen clues ha he model is con iden abou
he gi en alue o he s ee ing angle. This could be a p oblem, as he s ee ing angle is needed
o a oid he collision, bu i is also no needed in mos o he cases, as i he bina y p edic ion
indica es no collision, i is no conside ed. Also, in o de o implemen a s a egy like his, he
pe o mance o he s ee ing angle eg ession has o mee accep able c i e ia, as i would no
make sense o ake he ou pu as a solu ion i i p oduces andom numbe s. The quali y o he
ea ly exi is analyzed wi h mo e de ail in Sec ion 7.
The second s a egy is based on he c ea ion o a second model based on he possibili y o
unde s anding he e o o he eg ession gi en by he DNN. The p incipal idea is o ake he
ea ly exi o he bina y classi ie only when he model is su e enough abou he quali y o he
answe , and hen, i he ou pu o he collision p edic ion indica es ha a collision is abou o
happen, an e alua ion o he quali y o he s ee ing ou pu is execu ed in o de o know i he
model is con iden enough abou he eg ession ou pu . In case i mee s a con idence h eshold,
he execu ion is s opped; o he wise, only he eg ession ask needs o be execu ed in i s ull
o m, so he execu ion o he b anch o he inal ou pu o he collision ask can be deac i a ed.
The design o a s a egy like ha allows a huge educ ion o he compu a ional complexi y o
he model, and he implemen a ion o i can be done using he ONNX lib a y, as seen in he
p e ious sec ion.
5.3 Knowledge Dis illa ion o DNNs
As knowledge dis illa ion is becoming a echnique used o educe DNNs size be o e deploying
a model, i is conside ed in his wo k as a way o educe he dimensionali y o a model o make i
36

5 Reducing DNNs o Resou ce-Limi ed Pla o ms
Inpu
Con olu ion
Pooling Con olu ion
Pooling
Fully Connec ed
FASDD
Con olu ion
Pooling Con olu ion
Pooling Pooling
Fully Connec ed
Collision
Fully Connec ed
FASDD
Fully Connec ed
S ee ing Angle
Figu e 12: A chi ec u e o he s uden model.
i in he GAP9, e en i i s o iginal o m would no be able o i . As when using quan iza ion o
educe he da a p ecision o he model, knowledge dis illa ion aims o educe he compu a ional
cos o a model while p ese ing i s pe o mance. In o de o apply knowledge dis illa ion and
demons a e i s powe , he DNNs de eloped un il his poin o his wo k ha e been designed
wi h millions o pa ame e s, conc e ely abou 150 million. I is aken in o accoun ha a model
his size could al eady ha e been educed wi hou losing accu acy, bu he goal o applying KD
in his case will be o educe he numbe o pa ame e s o app oxima ely one million.
The p ocess o dis illing knowledge s a s by de ining a eache model, so a la ge model wi h
esul s good enough ha will be used in o de o ain a s uden model by eaching i how o
lea n. Simila ly o he ea ly exi s and he con idence measu e, his is a p ocess e y well adap ed
o mul i-class classi ie s, as he aw ou pu s o he model ( he logi s) a e di ec ly used as he way
a ge o make he s uden lea n; hese logi s a e he so labels. O cou se, ha d labels, so he
ue ones co esponding o da a, a e also used in o de o ain he model as i would be usually
done. The eache model used o applying his echnique is he model wi h an ea ly exi o
i e and smoke de ec ion, p esen ed in Sec ion 5.2.1. As he p oblem wi h KD is o apply i o
o he ypes o asks han mul i-class classi ie s, his sec ion only conside s he ac o sol ing
he FASDD ask by adding KD and ha he consequence will be ha he p essu e o his ask
o e he ull DNN will be eleased, making i possible o esol e he o he wo asks wi h no
ha many weigh s.
The s uden model is o med by wo con olu ional laye s, ollowed by he ea ly exi wi h ewe
weigh s in i s ully connec ed laye , and wo mo e con olu ional laye s o he main b anch,
ending wi h he h ee ou pu s wi h high sa ings in he ully connec ed weigh s by using wo
pooling laye s be o e hem, as i can be obse ed in Figu e 12. Also, each one o he con o-
lu ional laye s is ollowed by ba ch no maliza ion and pooling laye s, always espec ing he
37
Op imizing Deep Neu al Ne wo ks o Resou ce-Limi ed Embedded Sys ems
sequence o laye s compa ible wi h he GAP9, which o ces he ba ch no maliza ion laye s o
be be ween he con olu ion and he ac i a ion laye .
The aining o he model is done wi h PyTo ch; he da aloade s a e he same as in he aining
o he model in 5.2.1, bu he loss unc ion has a li le di e ence, as i has o compu e he KD
loss o he so a ge s o he classi ie . This loss is based on he Kullback-Leible di e gence,
which de ines a dis ance be ween wo p obabili y dis ibu ions. The e o e, he loss o he
dis illa ion is de ined as ollows:
Ldis ill =T2·KL(PS,PT)
whe e:
KL(PS,PT) =
N
∑
i=1
pS(i)logpS(i)+ε
pT(i)+ε
He e:
•Tis he empe a u e scaling pa ame e .
•PSand PTa e he so ened p obabili y dis ibu ions o he s uden and eache ne wo ks,
espec i ely, gi en by:
PT=so maxzT
T,PS=so maxzS
T
whe e zTand zSa e he logi s o he eache and s uden models.
•pS(i)and pT(i)deno e he p obabili ies o class iin he s uden and eache dis ibu ions.
•εis a small cons an added o nume ical s abili y, se o 1 ×10−10 in he aining ase.
The T2 e m scales he KL di e gence o accoun o he e ec o empe a u e on he logi s.
Du ing he aining, i is de ined as T=2. The loss encou ages he s uden ne wo k o imi a e
he so ened ou pu s o he eache ne wo k, acili a ing knowledge dis illa ion.
I is impo an o ema k ha his loss needs o be combined wi h he c oss-en opy loss in o de
o ully ain he mul i-class classi ie . In his case, he dis illa ion loss is added o he o al loss
in he same way as he ha d a ge s loss. This could be adap ed depending on he impo ance
KD needs o ake in o de o ob ain be e esul s, as well as he losses o he o he wo asks,
which could be weigh ed wi h he objec i e o no losing impo ance when he model is being
ained.
38
5 Reducing DNNs o Resou ce-Limi ed Pla o ms
5.4 Comp ession o he Model
Finally, a e ha ing ied di e en comp ession and op imiza ion me hods, one inal model
could be de ined. As he pe ec model depends on he use case ha needs o be esol ed, he
comp ession o a model has as i s only limi he pe o mance loss ha one could assume o lose.
As he comp ession echniques p esen ed in his wo k a e all compa ible wi h each o he , a
model combining all o hem is no ha d o c ea e. Fo example, he model p esen ed wi h h ee
asks and one ea ly exi in Sec ion 5.2.1, has been quan ized ollowing PTQ and e alua ed, as
he numbe o ope a ions can be lowe ed hanks o he ea ly exi a he same ime as he size
o he weigh s hanks o quan iza ion. The esul s o such a combina ion o echniques a e
p esen ed wi h mo e de ail in Sec ion 7.
Ne e heless, comp essing DNNs has a limi . I is obse ed in his wo k and exposed wi h mo e
de ail in he e alua ion and esul s sec ions, Sec ion 7, ha when using KD o he FASDD ask,
he e is an impo an accu acy loss o he ea ly exi . This accu acy loss is o ally canceling he
e ec and bene i ha ha ing an ea ly exi ep esen s, and he eason why his happens is un-
known. One solu ion would be o un a deepe analysis o e he aining in o de o unde s and
i he dis illa ion loss is ha ing an impac on he ea ly exi o on he o e all loss o he model, he
one which is used by he s ochas ic g adien descen op imize o ain he model. The ac o
ha ing mul iple objec i es is des abilizing enough o he aining p ocess, so adding KD, ea ly
exi s, and hen applying quan iza ion o he ull model is in oducing a lo mo e pa ame e s ha
ha e o be aken in accoun be o e deploymen .
Addi ionally, he es ic ions gi en by he GAP9 p ocesso and he edge- o-cloud se up make
some o he p oblems impossible o esol e, o e en impossible o educe hei impac s. Fo
example, skip connec ions would ha e been an impo an ea u e o make he aining s able, as
ecen ly demons a ed in [MacDonald e al., 2022]. One o he issue ha could ha e been sol ed
wi h esidual connec ions is he esul s o QAT and he ac ha he ea ly exi was gi ing be e
esul s han he inal ou pu , e en i no explana ion can be gi en on why his happens. This is
conside ed o be a possible u he wo k o his p ojec .
39
Op imizing Deep Neu al Ne wo ks o Resou ce-Limi ed Embedded Sys ems
6 Demons a o
6.1 Model Gene aliza ion
As one o he objec i es o his bachelo hesis is deploying DNNs in a conc e e edge- o-cloud
se up, i is impo an o ha e all he necessa y ools o do so. In o de o imp o e he expe i-
ence o using D oneBandi , a ew sc ip s ha e been de eloped, wi h he aim o acili a ing he
in eg a ion o a model wi h D oneBandi .
6.1.1 Con ex Vec o s
Con ex ec o s a e needed by D oneBandi o calcula e he complexi y o he model depending
on he cu ing poin . Those ec o s a e simila o hose used in he ANS algo i hm, p esen ed
in [Zhang e al., 2021], bu D oneBandi no malizes he ec o s o ob ain be e esul s. These
ec o s de ine he numbe o mul iply–accumula e ope a ions (MACs) pe ype o laye , as well
as he numbe o laye s o each ype and he size o he in e media e ou pu gi en he cu ing
poin . One ec o can be de ined o he ull model bu also o a pa o he model, which
is in e es ing o hese op imiza ion-based algo i hms because o he compu a ional complex-
i y in o ma ion hey gi e. No e ha ReLU laye s do no eally ha e MAC ope a ions, so he
compu a ion ime is di e en han o he o he laye ypes; his is why he numbe o MACs is
conside ed depending on he laye ype. The size o he in e media e ou pu co esponds o he
numbe o by es occupied by he da a ha has o be sen o he se e , gi en he cu ing poin .
No e ha he models a e quan ized o he deploymen , so his numbe o by es co esponds
exac ly o he numbe o weigh s, as he da a ype is in 8.
To ex ac hese con ex ec o s, a code was p o ided ha was used o he D oneBandi p ojec ,
bu he in o ma ion ex ac ed depended oo much on he ype o he model and was unable o
calcula e he MACs o he ea ly exi s. Gi en ha issue, i has been added as an objec i e
he de elopmen o a unc ion capable o ex ac ing hose con ex ec o s o any model. The
de eloped code s a s by ex ac ing a summa y o he model, wi h simila in o ma ion o he
one gi en by he summa y unc ion o he o chin o lib a y, bu e u ning a dic iona y wi h he
in o ma ion. Gi en he summa y o he model, he unc ion i e a es o e he possible cu ing
poin s and calcula es he MACs and he in e media e ou pu size om he cu ing poin o he
end o he model. I also handles an ea ly exi in he case his ea ly exi does no p esen mo e
laye s han he ou pu one. This unc ion wo ks o models made ou o sequen ial blocks, as
hey a e he models compa ible wi h he D oneBandi algo i hm. The MACs a e calcula ed as
ollows:
40
6 Demons a o
Cycles Time (ms) Ene gy (µJ) In e ences pe sec.
In e ence 1 24392955 65.92690541 0.7378132088 15.16831397
In e ence 2 24394882 65.93211351 0.7378714948 15.16711579
In e ence 3 24393346 65.92796216 0.7378250354 15.16807083
In e ence 4 24393202 65.92757297 0.7378206798 15.16816038
In e ence 5 24392439 65.92551081 0.7377976014 15.16863484
In e ence 6 24388213 65.91408919 0.7376697777 15.17126327
In e ence 7 24393120 65.92735135 0.7378181996 15.16821136
In e ence 8 24391777 65.92372162 0.7377775779 15.16904652
In e ence 9 24393292 65.92781622 0.7378234021 15.16810441
In e ence 10 24395841 65.93470541 0.7379005016 15.16651957
A e age 24392906.7 65.92677486 0,7378117479 15.168344
Table 2: Time and ene gy consump ion on single in e ence o he model esol ing he h ee
asks, one o hem ained using KD.
be obse ed as D oneBandi inds a balance by cu ing he model on he second cu ing poin ,
minimizing bo h la encies and sending he mos li le in e media e ou pu possible.
Gi en ha all he ea u es o D oneBandi ha e been es ed and success ully deployed on he
simula ion o he GAP9, he p ojec is conside ed unc ional. On one hand, he se e is able o
ead and ob ain in o ma ion unning he in e ence om a gi en poin , while on he o he hand,
D oneBandi is able o co ec ly choose a cu ing poin , adap ing he decision on he la ency and
he con ex ec o s. The limi a ions o he edge de ice made i impossible o join he wo pa s
o he demons a ion, bu ha ing all o i s elemen s wo king also demons a es ha he use case
can be sol ed in a ic ional case in which he GAP9 was in eg a ed in o a d one wi h a WiFi
module and a compa ible came a.
6.2.3 Deploymen o a Knowledge Dis illed Model
As all o he deploymen has been made wi h he model esol ing he FASDD ask, and gi en
ha ime was oo sho in o de o ollow he ull p ocess o he deploymen on he model using
KD, a deploymen o his model has been done sepa a ely in o de o demons a e ha he ull
model can un on he edge. This is impo an as D oneBandi needs o ha e he possibili y o
making he ull in e ence on edge o e alua e whe e he model needs o be cu . A e seeing
ha he model is able o i in he GAP9, a simila ex ac o me ics has been done, exac ly
as he one done o he model esol ing he FASDD ask. The esul s, shown in Table 2, show
ha he numbe o in e ences pe second d ops signi ican ly. The ac ha he ini ial numbe o
in e ences pe second was almos 60 and i d opped o 15 indica es ha he mo e complex he
model is o he d one, he ewe images i can p ocess. This can be a p oblem depending on he
use case, and gi en ha his is de eloped o a d one, he speed o he UAV has o be adap ed
o i s p ocessing ime, as o he wise i may collide o change o di ec ion mis akenly.
47

Op imizing Deep Neu al Ne wo ks o Resou ce-Limi ed Embedded Sys ems
7 E alua ion and Resul s
This sec ion has he objec i e o exposing he pe o mance o he models de eloped, ela ing
he pe o mance loss o each comp ession echnique ha has been conside ed.
The i s model ained o his wo k is he one esol ing h ee asks, wi h a single ou pu
o each one o hem, p esen ed in Sec ion 4.1.2. In he ollowing plo , Figu e 15, i can be
obse ed he di e ence o accu acies be ween asks. Fo he FASDD ask, he QAT model
p esen s a dec ease in accu acy o a bi mo e han 5%, while he MSE is also sligh ly inc easing.
These could be accep able esul s conside ing he loss o pe o mance o he quan ized model,
i i we e no o he p oblem ound on he collision ask. I is obse ed ha he collision
ask is unde i ing on he o iginal model, p obably because o he numbe o epochs ha is
au oma ically chosen by an Ea ly S op objec , which has a condi ion o s op he aining
a e seeing ha he loss is no imp o ing o e he alida ion pa i ion o he da a. As his is a
p oblem o he e alua ion o he models and compa ing hem be ween hem, his is he only
model ained wi h an ea ly s op condi ion.
Figu e 15: Compa ison o pe o mance me ics be ween he O iginal Model and QAT Model
o he model esol ing h ee asks wi hou ea ly exi s.
Then, he size o he model and he ime needed o make an in e ence ha e been e ie ed in
o de o be able o unde s and he powe o quan iza ion. The o iginal model weigh s 119.30
MB in TFLi e o ma , sa ing his way he weigh s and he a chi ec u e in he same ile, while
he QAT model weigh s 9.91 MB, also in TFLi e o ma . I is imp essi e o see how he same
model wi h no big pe o mance di e ences is capable o being comp essed un il less han one
48
7 E alua ion and Resul s
en h o i s o iginal o m by educing he p ecision o he weigh s and ac i a ions. Addi ionally,
bo h models ha e been execu ed on a compu e using CPUs; his is no ep esen a i e o he
ime needed o a QAT model o do an in e ence, as edge de ices a e op imized o execu ing
ope a ions o his kind. Ne e heless, compa ing he compu ing imes on he same compu e
can gi e an idea o he sa ings ha can be achie ed: in his case, he in e ence wi h he o iginal
model needed 30.58 ms, while he quan ized one only needed 13.99 ms.
Jus a e he de elopmen o his model, he one esol ing he FASDD ask has been designed.
As his model in eg a ed an ea ly exi , he e alua ion o he accu acy aims o unde s and how
he ype o quan iza ion can a ec his one. The esul s o he pe o mance o each exi a e
shown depending on he quan iza ion ype in Table 3. As i can be obse ed in he able, he
model is no p esen ing highe accu acy wi h he QAT p ocess, and he esul s show a be e
pe o mance on he ea ly exi han wi h he ull model in QAT. These esul s a e no sa is ying
as he e is no logical explana ion o he accu acy di e ence. The main eason ha can be
conside ed o his o happen is he ins abili y o aining a li le model like he one used o
FASDD and in oduced by he ake nodes o simula e he quan iza ion. Gi en hese esul s, he
decision aken has been o p oceed wi h PTQ o he deploymen wi hou wo ying abou a big
loss o pe o mance wi h QAT. The model weigh s 0.42 MB, bo h in PTQ and in QAT, which
makes sense as he execu ion g aph esul ing om bo h quan iza ion p ocesses is he same.
QAT PTQ
Ea ly Exi 69.80 73.69
Full model 68.71 76.38
Table 3: Pe cen age o accu acy depending on he quan iza ion me hod.
When coming o he analysis o he ea ly exi s, wo models ha e been p esen ed, one esol ing
he o iginal asks o D oNe , he collision p edic ion and he s ee ing angle eg ession, wi h an
ea ly exi o each one o hem. The o he one p esen ed esol ed h ee asks, he wo same as
he p e ious one, and he FASDD ask, wi h an ea ly exi on his las ask. As he second o
hese wo models has been implemen ed wi h KD also, he i s one e alua ed he e is he one
esol ing wo asks.
As he i s s a egy elied on choosing he bes con idence h eshold o he collision ask,
di e en alues o he possible h eshold we e ied in o de o maximize he accu acy o he
ea ly exi , esul ing in a h eshold o 0.95. In Table 4, i can be obse ed how he accu acy
o he collision ask is changing when calcula ing i based on he s a egy, so s opping he
execu ion when he ea ly exi con idence exceeds 0.95. E en i o he inal ou pu he accu acy
dec eases, he o e all accu acy esul ing om he e alua ion is 0.9274, gi en ha he model
akes he ou pu o he ea ly exi 89.48% o he ime. Also, he MSE is no conside ed because
i is no a pa ame e o he decision o he s a egy, bu gi en he imes he ea ly exi is aken,
49
Op imizing Deep Neu al Ne wo ks o Resou ce-Limi ed Embedded Sys ems
he o e all MSE can be compu ed o he s ee ing angle decision, esul ing in 0.0088, which is
a be e esul han aking he inal ou pu o he o iginal model as a decision.
O iginal Model Applying he S a egy
Accu acy ea ly exi 0.9036 0.9315
Accu acy inal ou pu 0.9287 0.8928
MSE ea ly exi 0.0082 -
MSE inal ou pu 0.0141 -
Table 4: Pe o mance me ics on he model esol ing wo asks wi h wo ea ly exi s, compa ison
be ween model wi h and wi hou he ea ly exi s a egy, wi h he s a egy conside ing only he
collision ask accu acy.
As almos hal o he model is no compu ed when he ea ly exi is aken as he esul o an
in e ence, he numbe o MACs ha can be sa ed has been compu ed. In ac , he expe imen
has been done o e a hund ed in e ences, and i has been obse ed ha abou 1759721592.99
MACs a e sa ed pe in e ence. This ep esen s a 89.59% o he MACs o he model.
Then, a second s a egy has been de ined o he same model, aking also he decision o he
s ee ing angle in o accoun o he ea ly exi s a egy. The s a egy elies on p edic ing he
e o ha can be gi en by he DNN wi h a p e ained and alida ed lasso eg ession o de ine
a measu e o con idence simila o he one o he collision bina y ask. The lasso eg ession
is ained in o de o p edic he e o o he DNN, and he ini ial hypo hesis explaining why i
migh wo k is because DNN e o s a e o en no andom; hey migh depend on ce ain inpu
ea u es, edge cases, o egions whe e he model’s capaci y is insu icien . The e o e, he lasso
eg ession is ained wi h he same inpu gi en o he DNN, he ou pu o he DNN o he
eg ession, and has he objec i e o p edic ing he e o done by he DNN. Then, he con idence
is compu ed o any sample as ollows:
X=hFla en(Image)ˆyp edi
σ= LassoP edic ion(X)
100
Con idence =1−σ
MaxE o
whe e:
• Image ep esen s he inpu images o es ing he DNN, la ened in o a ec o .
• ˆyp ed is he DNN’s p edic ed ou pu o he co esponding inpu images.
50
7 E alua ion and Resul s
•Xis he inpu o he Lasso eg ession model, which is a conca ena ion o he la ened es
images and he co esponding DNN p edic ions.
• LassoP edic (X)is he Lasso eg ession model’s p edic ion o he e o a iance, gi en
X, di ided by 100 o be e pe o mance and esul s be ween 0 and 1.
•σis he s anda d de ia ion o he p edic ed e o .
• MaxE o is he maximum obse ed e o in he da ase .
This model has been es ed on a es ing pa i ion o he da ase , and he esul s gi en by he
model a e an MSE o 0.0029300626 and an R2o 0.9979, which a e excellen esul s and allow
o s a e ha he lasso eg ession is explaining he e o o he DNN. Gi en his eg ession, and
applying i o he s a egy by calcula ing he con idence o he eg ession ea ly exi , i has been
seen ha he bes h eshold o he eg ession ask is 0.99, which is appa en ly no ha d o each
as when he con idence is high enough o he collision ask, he eg ession ask is exceeding his
con idence 100% o he ime, esul ing in he esul s exposed in Table 5. The pe o mance o he
collision ask is he same as in he p e ious ask, as i is he i s pa ame e o he decision aken
in o accoun ; only i he s a egy is s ill being e alua ed is he eg ession conside ed. Applying
his s a egy on eal da a esul s in aking he decision in he same cases as in he p e ious case,
so no u he esul s a e gi en abou he MACs sa ings, as hey a e he same as be o e.
O iginal Model Applying he S a egy
Accu acy ea ly exi 0.9036 0.9315
Accu acy inal ou pu 0.9287 0.8928
MSE ea ly exi 0.0082 0.0013
MSE inal ou pu 0.0141 0.0148
Table 5: Pe o mance me ics on he model esol ing wo asks wi h wo ea ly exi s, compa ison
be ween model wi h and wi hou he ea ly exi s a egy, wi h he s a egy conside ing bo h asks.
S ill in he opic o ea ly exi s, he second model ha had an implemen a ion o an ea ly exi
is he one esol ing h ee asks wi h he ea ly exi on he FASDD ask, p esen ed in Sec ion
5.2.1. In Table 6, i can be obse ed how he accu acy o he ea ly exi is inc easing, wi h a
h eshold o 0.95, chosen he same way as o he p e ious case. E en i he accu acy o he
inal ou pu is conside ably d opping, he h eshold is exceeded 77.77% o he ime, making he
o e all accu acy o he FASDD ask 80.29%, which is be e han wha i could be done wi hou
he s a egy, indica ing ha he e is a good balance be ween he esolu ion o he asks on he
ea ly exi when he model is con iden wi h he asks esol ed by he ull model when he model
needs mo e compu a ion.
Fo his case and o he imes he h eshold is exceeded, i has been obse ed ha 28456464.77
MACs can be sa ed o each in e ence, way less han o he p e ious model, bu he eason is
51
Op imizing Deep Neu al Ne wo ks o Resou ce-Limi ed Embedded Sys ems
O iginal Model Applying he S a egy
Accu acy FASDD ea ly exi 0.7819 0.8608
Accu acy FASDD inal ou pu 0.8083 0.6
Accu acy collision ask 0.9499 0.9499
MSE s ee ing angle ask 0.0017 0.0017
Table 6: Pe o mance me ics on he model esol ing h ee asks wi h one ea ly exi on he
FASDD ask, compa ison be ween model wi h and wi hou he ea ly exi s a egy.
ha in his s a egy only he FASDD ask is deac i a ed, bu he ull ne s ill has o be un o he
o he wo asks. E en so, conside ing only he FASDD ask, 44.76% o he MACs a e sa ed.
Ending wi h he same model, he las echnique conside ed o comp ess a neu al ne wo k is
knowledge dis illa ion, which has been applied o bo h he ea ly exi and he inal exi o he
FASDD b anch o he p e ious model. The s uden model, p esen ed in Sec ion 5.3, was shown
o be de ined wi h only one million pa ame e s, compa ed o he 160 million o he eache
model. This way, he s uden model size is abou 3.916 MB, compa ed o he 650.883 MB o
he o iginal model, and can e en be educed o 0.9965 MB i he model is quan ized a e he
aining.
KD Model KD + PTQ model
Accu acy FASDD ea ly exi 0.1977 0.2065
Accu acy FASDD inal ou pu 0.8112 0.8154
Accu acy collision ask 0.8455 0.8374
MSE s ee ing angle ask 0.0145 0.0155
Table 7: Pe o mance me ics on he model esol ing h ee asks wi h one ea ly exi on he
FASDD ask, wi h knowledge dis illa ion in eg a ed o he aining and compa ed o he quan-
ized e sion o he same model.
In Table 7, he pe o mance esul s a e shown, whe e i can be obse ed ha he ea ly exi is
gi ing o ally andom esul s, p obably due o he comp ession le el achie ed. Wha is eally
in e es ing o ema k is ha he esul s o he PTQ model a e eally close o he ones o he
KD model, and he esul s a e mo e han accep able o he asks hey ha e o esol e, ha ing
achie ed an inc edible comp ession o he size o he model.
52

8 Sus ainabili y Analysis and E hical Implica ions
8 Sus ainabili y Analysis and E hical Implica ions
Wi h he quick de elopmen in he ield o echnology, he e is a need o hink abou he e hical
and sus ainable implica ions o a i icial in elligence. While he e’s huge po en ial wi h AI ech-
nologies, hei de elopmen s also b ing ques ions o p i acy, misuse, and esou ce consump ion
o he on . This sec ion is impo an in aming he limi ed e hical conside a ions ele an o
his p ojec and highligh ing i s alignmen wi h sus ainable p ac ices.
The e hical isks o his p ojec a e e y minimal. Compa ed o applica ions handling sensi-
i e o pe sonal da a, he da ase s used in his s udy a e publicly a ailable and do no con ain
iden i iable in o ma ion. The case a hand deals mainly wi h he use o d ones ha can analyze
isual in o ma ion in eal ime o pe o m au onomous decision-making ac ions, like collision
a oidance o i e de ec ion, wi h he objec i e o minimizing da a ans e s be ween se e s.
This ul ima ely ensu es ha decisions made a e only based on incoming da a and con ex , no
on any unde lying da a misuse o bias.
Fu he mo e, he p ojec explici ly a oids any misuse o e en enabling i ; he ocus is i mly
on sa e and anspa en applica ions, wi h no ole ance o deploymen in con ex s ha could
comp omise e hical s anda ds, such as su eillance o ha m ul au oma ion.
The ocus o his p ojec has been on he deploymen o AI models in mic op ocesso s o
asks such as i e and smoke de ec ion o collision a oidance wi h he use o d ones. These
applica ions a e hough o inc ease sa e y and e iciency in esou ce-cons ained en i onmen s,
like disas e scena ios. By limi ing he scope o his p ojec o his bene icial use, misuse is
being minimized while maximizing he alue o he echnology.
This will be a ounda ion o sus ainabili y in he p ojec o AI model comp ession. Op imizing
models o esou ce-cons ained de ices no only leads o ene gy e iciency bu also lowe s he
en i onmen al impac o deploying AI in embedded sys ems. The equi emen o compac and
compu a ionally e icien models becomes e y c i ical when AI sys ems s a wo king in as ly
di e en , esou ce-cons ained en i onmen s.
This p ojec used 60.5 hou s o GPU aining wi h an N idia GeFo ce RTX 3090. This is a
signi ican ly long aining ime, bu in ac , many model comp ession me hods ha e been pe -
o med o educe compu a ion equi emen s o bo h aining and in e ence, which helped in
educing he po en ial en i onmen al impac ha a model like he ones de eloped could ha e.
These e o s align wi h sus ainable AI p ac ices by p io i izing ene gy e iciency. Model com-
p ession also ul ills he goal o he esponsible use o AI esou ces, as smalle and op imized
models use ewe compu a ional esou ces when deployed. This app oach can also be consid-
e ed a way o minimize he ca bon oo p in gene a ed by he aining p ocesses o hese models,
as i deployed and used o long pe iods, hei en i onmen al impac is way lowe han i could
be wi h non-op imized models in he long e m.
53
Op imizing Deep Neu al Ne wo ks o Resou ce-Limi ed Embedded Sys ems
Las ly, i is impo an o highligh he alignmen o he p ojec wi h he Eu opean Union’s Sus-
ainable De elopmen Goals (SDGs), pa icula ly he ollowing:
•SDG 9: Indus y, Inno a ion, and In as uc u e
–By de eloping and op imizing AI models o deploymen in esou ce-cons ained
en i onmen s, he p ojec p omo es inno a ion and he sus ainable use o ad anced
echnologies in indus ial and ope a ional con ex s.
•SDG 11: Sus ainable Ci ies and Communi ies
–The use case o au onomous d ones o i e and smoke de ec ion di ec ly con ibu es
o sa e communi ies by o e ing he possibili y o esponding o en i onmen al
eme gencies in eal-wo ld scena ios.
•SDG 12: Responsible Consump ion and P oduc ion
–The p ojec exempli ies esponsible esou ce usage by p io i izing e iciency in AI
model aining and in e ence.
•SDG 13: Clima e Ac ion
–The ene gy-e icien design o AI models suppo s clima e ac ion by educing he
en i onmen al impac o deploying and aining AI sys ems, pa icula ly in ene gy-
in ensi e scena ios.
54
9 Conclusions
9 Conclusions
Du ing his p ojec , di e en DNNs ha e been designed in o de o ul ill he gi en objec i es.
The bachelo hesis explo ed he di e en possibili ies o comp essing models in o de o de-
ploy hem on an embedded sys em using a GAP p ocesso as he edge de ice. The echniques
ha ha e been expe imen ed wi h include quan iza ion, ea ly exi s, and knowledge dis illa ion,
all o hem espec ing he es ic ions o he gi en edge- o-cloud se up. The use case o an
au onomous nanod one has been he base o he p ojec , making he goal o each de eloped
model o sol e he basic asks usually ope a ed by humans when wo king wi h UAVs. The
comp ession echniques demons a ed ha DNNs can be op imized wi h minimal impac s on
hei pe o mance, while he success ul deploymen s on he edge de ice show ha he imple-
men a ion on a d one is pe ec ly possible.
This esea ch has been ca ied ou in he con ex o a esea ch g oup o he REDs Ins i u e
a he HEIG-VD, so he ou come o he p ojec will allow a democ a iza ion o he use o
he GAP9 mic op ocesso in he ins i u e, also opening a doo o he use o such embedded
sys ems o he labo a o ies o he subjec s ca ied by he p o esso s wo king a his ins i u e.
Mo e impo an ly, his wo k included he de elopmen o many ools ha make he use o
D oneBandi , he p ojec o which his hesis is linked, easie . By p o iding he necessa y ools
o sequen ialize a complex model and he au oma ic ex ac ion o con ex ec o s, D oneBandi
becomes he pe ec algo i hm o be chosen when de ining a dynamic edge- o-cloud s a egy o
DNNs.
On he o he hand, he limi a ions encoun e ed wi h he GAP9 p ocesso limi ed he possibili y
o deploymen and demons a ion o he p ojec . The challenges p esen ed in his hesis ha e
been he main pain poin s o he wo k and ha e been o e come by p oposing al e na i es ha
we e no planned on a i s ime. Using a cu ing-edge de ice like he GAP9 allowed expe imen-
a ion wi h uncommon and inaccessible echnology, bu a he same ime mean an inc ease in
he wo kload o he p ojec . The u u e wo k o his hesis elies on he in eg a ion o he GAP9
in a eal nanod one and he es ing o he models in a eal-wo ld scena io by expe imen ing on
he e ec ha he di e en p esen ed comp ession echniques can ha e.
On a inal no e, a e lec ion abou he mode n o m o wo king wi h AI is made. The un-
s oppable pu sui o aining la ge and la ge models has led us o a c i ical poin whe e he
sus ainabili y o AI mus ake p ecedence o e i s scale. As demons a ed in his hesis, he
deploymen o DNNs canno , and should no , mi o hei aining phase. The a ailabili y o
ad anced comp ession echniques has p o en ha ema kable e iciency can be achie ed wi h-
ou comp omising pe o mance. I is ime o change om a mindse ha glo i ies size and
complexi y o one ha alues inno a ion, esou ces, and esponsibili y. The ue po en ial o AI
lies no in how big i can become bu in how hough ully i can be de eloped and deployed.
55
Op imizing Deep Neu al Ne wo ks o Resou ce-Limi ed Embedded Sys ems
Appendices
A Hop ield Ne wo k as a Solu ion o he Image Classi ie
One o he p oposed goals o ini ia e he p ojec was o ind a new use case o he d one o he
han i s capabili y o ci cum en obs acles h ough he edge- o-cloud p ocessing app oach. The
concep concen a ed on making he in e ence p ocess au onomous on he p ocesso o he d one
wi hou he equi emen o se e connec i i y. This hen led o explo ing objec de ec ion as a
po en ial new applica ion, which is one o he classic asks alling unde compu e ision.
The app oach en isaged he use o a Hop ield Ne wo k, a ne wo k well known as capable o
econs uc ing pa e ns om bina y images. In he p oposal, ha mean aining he ne wo k on
he bina y images o an objec and hen unning in e ence con inuously o de e mine how he
ne wo k ene gy s a e was beha ing. I he ne wo k con e ged o a ained pa e n based on a p e-
de ined h eshold, ha would signal ha an objec had been de ec ed. Howe e , implemen ing
his solu ion e ealed some challenges:
1. Bina y Inpu Requi emen : The classical Hop ield Ne wo k equi es bina y inpu da a.
To sa is y his equi emen , a Canny edge de ec ion il e was applied o p ep ocess im-
ages, p oducing bina y con ou s o objec s.
2. High Dimensionali y: The numbe o neu ons in he Hop ield Ne wo k mus ma ch he
dimensionali y o he inpu da a. Fo a 200x200-pixel image, his mean 40,000 neu ons,
leading o a ne wo k wi h o e 1.6 billion pa ame e s due o i s ull connec i i y. This
made he model compu a ionally impossible o ain o deploy.
3. Dimensionali y Reduc ion A emp s: To mi iga e he high dimensionali y, se e al s a e-
gies we e es ed:
• Applying PCA a e bina iza ion educed he dimensionali y bu p oduced non-
bina y ou pu s, which could no se e as inpu s o he Hop ield Ne wo k.
• Applying PCA be o e bina iza ion ailed o p ese e local ea u es, making edge
de ec ion ine ec i e.
• C opping egions wi h maximum in o ma ion succeeded in educing image size by
hal while e aining c i ical objec ea u es.
S ill, he esul s we e limi ed. Expe imen s on he CIFAR-10, CIFAR-100, Imagene e, and
Himax da ase s demons a ed ha he ne wo k was only able o iden i y objec s when he inpu
image was e y close o he aining da a. Fu he mo e, because o he high dimensionali y, he
ne wo k would o en con e ge o he same ene gy s a e o he objec and wi hou . Wi h hese
limi a ions in mind, he Hop ield Ne wo k was no sui ed o his ask.
56