Beyond Horizons - The Rise of the Edge AI Processing Paradigm

Beyond Ho izons –
The Rise o he Edge AI P ocessing
Pa adigm
RIVER PUBLISHERS SERIES IN COMMUNICATIONS
AND NETWORKING
Se ies Edi o s:
ABBAS JAMALIPOUR MARINA RUGGIERI
The Uni e si y o Sydney Uni e si y o Rome To Ve ga a
Aus alia I aly
MARKO JURCEVIC
Uni e si y o Zag eb
C oa ia
The “Ri e Publishe s Se ies in Communica ions and Ne wo king” is a
se ies o comp ehensi e academic and p o essional books which ocus on
communica ion and ne wo k sys ems. Topics ange om he heo y and use
o sys ems in ol ing all e minals, compu e s, and in o ma ion p ocesso s
o wi ed and wi eless ne wo ks and ne wo k layou s, p o ocols, a chi ec-
u es, and implemen a ions. Also co e ed a e de elopmen s s emming om
new ma ke demands in sys ems, p oduc s, and echnologies such as pe -
sonal communica ions se ices, mul imedia sys ems, en e p ise ne wo ks,
and op ical communica ions.
The se ies includes esea ch monog aphs, edi ed olumes, handbooks
and ex books, p o iding p o essionals, esea che s, educa o s, and ad anced
s uden s in he ield wi h an in aluable insigh in o he la es esea ch and
de elopmen s.
Topics included in his se ies include:
• Communica ion heo y
• Mul imedia sys ems
• Ne wo k a chi ec u e
• Op ical communica ions
• Pe sonal communica ion se ices
• Telecoms ne wo ks
• Wi i ne wo k p o ocols
Fo a lis o o he books in his se ies, isi www. i e publishe s.com
Beyond Ho izons –
The Rise o he Edge AI P ocessing
Pa adigm
Edi o s
O idiu Ve mesan
SINTEF, No way
Ma cello Coppola
STMic oelec onics, F ance
Fabian Che si
CEA, F ance
Ri e Publishe s
Published, sold and dis ibu ed by:
Ri e Publishe s
B oage ej 10
9260 Gis up
Denma k
www. i e publishe s.com
ISBN: 978-87-4380-863-3 (Ha dback)
978-87-4380-862-6 (Ebook)
©The Edi o (s) and The Au ho (s) 2025. This book is published open access.
Open Access
This book is dis ibu ed unde he e ms o he C ea i e Commons A ibu ion-Non-
Comme cial 4.0 In e na ional License, CC-BY-NC 4.0 (h p://c ea i ecommons.o g/
licenses/by/4.0/), which pe mi s use, duplica ion, adap a ion, dis ibu ion and ep o-
duc ion in any medium o o ma , as long as you gi e app op ia e c edi o he
o iginal au ho (s) and he sou ce, a link is p o ided o he C ea i e Commons license
and any changes made a e indica ed. The images o o he hi d pa y ma e ial in
his book a e included in he wo k’s C ea i e Commons license, unless indica ed
o he wise in he c edi line; i such ma e ial is no included in he wo k’s C ea i e
Commons license and he espec i e ac ion is no pe mi ed by s a u o y egula ion,
use s will need o ob ain pe mission om he license holde o duplica e, adap , o
ep oduce he ma e ial.
The use o gene al desc ip i e names, egis e ed names, adema ks, se ice ma ks,
e c. in his publica ion does no imply, e en in he absence o a speci ic s a emen ,
ha such names a e exemp om he ele an p o ec i e laws and egula ions and
he e o e ee o gene al use.
The publishe , he au ho s and he edi o s a e sa e o assume ha he ad ice and
in o ma ion in his book a e belie ed o be ue and accu a e a he da e o publica-
ion. Nei he he publishe no he au ho s o he edi o s gi e a wa an y, exp ess o
implied, wi h espec o he ma e ial con ained he ein o o any e o s o omissions
ha may ha e been made.
P in ed on acid- ee pape .

Dedica ion
“I a machine is expec ed o be in allible, i canno also be in elligen .”
– Alan Tu ing
“Wisdom is he daugh e o expe ience.”
– Leona do da Vinci
“Wonde is he beginning o wisdom.”
– Soc a es
“Educa ion is no he lea ning o he ac s, bu he aining o he mind
o hink.”
– Albe Eins ein
Acknowledgemen
The edi o s would like o hank all he con ibu o s o hei suppo in he
planning and p epa a ion o his book. The ecommenda ions and opinions
exp essed in he book a e hose o he edi o s, au ho s, and con ibu o s
and do no necessa ily ep esen hose o any o ganiza ions, employe s, o
companies.
O idiu Ve mesan
Ma cello Coppola
Fabian Che si
Con en s
P e ace xi
Lis o Figu es xiii
Lis o Tables x ii
Lis o Con ibu o s xix
1 Ad ancing Edge AI Pe cep ion Pla o ms and Senso
Fusion o Las -Mile Deli e y Au onomous Vehicles 1
O idiu Ve mesan, Roy Bah , Hans-E ik Sand,
Simen Ma en ius Saxegaa d, Helge B udeli,
Pe e Emanuelsson, and Ma in Fø isdahl
1.1 In oduc ion and Backg ound . . . . . . . . . . . . . . . . . 2
1.2 Senso Fusion in Las -Mile Con ex . . . . . . . . . . . . . 7
1.3 Au onomous Vehicle A chi ec u e o Las -Mile Deli e y . . 15
1.3.1 Localisa ion and High-De ini ion Map . . . . . . . . 15
1.3.2 Pe cep ion Implemen a ion . . . . . . . . . . . . . . 15
1.3.3 P edic ion, Decision-Making, Planning and Rou e
Op imisa ion ..................... 23
1.3.3.1 Odome y and pa h planning . . . . . . . 23
1.4 Edge AI Pla o ms . . . . . . . . . . . . . . . . . . . . . . 26
1.4.1 Robo Ope a ing Sys em . . . . . . . . . . . . . . . 27
1.5 Fu u e Conside a ions and Resea ch . . . . . . . . . . . . . 31
1.5.1 Deploymen Conside a ions . . . . . . . . . . . . . 31
1.5.2 Fu u e esea ch . . . . . . . . . . . . . . . . . . . . 32
1.6 Conclusion .......................... 34
ii
iii Con en s
2 AIDGE: A F amewo k o Deep Neu al Ne wo k De elopmen ,
T aining and Deploymen on he Edge 41
Fabian Che si, Oli ie Bichle , Cy il Moineau, Maxence Naud,
Lau en Sou ie , Vincen Templie , Thibaul Allene , Inna Kuche ,
and Vincen Lo ain
2.1 In oduc ion and Backg ound . . . . . . . . . . . . . . . . . 42
2.1.1 Rela edWo k..................... 43
2.2 Ou F amewo k O e iew . . . . . . . . . . . . . . . . . . 44
2.2.1 In e nal G aph Rep esen a ion . . . . . . . . . . . . 47
2.2.2 Pla o m in e ope abili y . . . . . . . . . . . . . . . 48
2.2.3 G aph Regula Exp ession (G aphRegex) . . . . . . 48
2.2.4 Ne wo k op imiza ion . . . . . . . . . . . . . . . . 49
2.2.5 Expo phase ..................... 51
2.3 Conclusion and u u e wo k . . . . . . . . . . . . . . . . . . 52
3 A scalable and lexible in e connec -based da a low
a chi ec u e o Edge AI In e ence 55
Rohi P asad and Hana K ichene
3.1 In oduc ion.......................... 56
3.2 Rela edWo k ......................... 57
3.3 Backg ound: da a low execu ion models . . . . . . . . . . . 57
3.4 In e connec -based da a low a chi ec u e . . . . . . . . . . . 58
3.4.1 NGC: Neu al Global Con olle . . . . . . . . . . . 59
3.4.2 NPE: Neu al P ocessing Elemen . . . . . . . . . . 59
3.4.3 AINoC: A i icial In elligence Ne wo k-on-Chip . . 61
3.4.4 Global Bu e s . . . . . . . . . . . . . . . . . . . . 63
3.5 Execu ionModel ....................... 63
3.6 Expe imen s and Resul s . . . . . . . . . . . . . . . . . . . 64
3.6.1 E alua ion Me hodology . . . . . . . . . . . . . . . 64
3.6.2 FPGA Implemen a ion Resul s . . . . . . . . . . . . 65
3.6.2.1 A ea .................... 65
3.6.2.2 La ency . . . . . . . . . . . . . . . . . . 67
3.6.2.3 Ene gy consump ion . . . . . . . . . . . . 67
3.6.2.4 Ene gy e iciency . . . . . . . . . . . . . 69
3.7 Conclusion .......................... 70
Lis o Figu es x
Figu e 6.2 Le : BIPBIP weeding sys em behind a obo ized
ac o . Righ : Inside BIPBIP, he came a and he
ligh ing sys em [2]. . . . . . . . . . . . . . . . . . 102
Figu e 6.3 BIPBIP weeding module. The mechanical in a- ow
hoeing ool is ep esen ed by he od on he le , he
compu ing sys em in yellow, he wo LED panels
and he came a in black inside he ision chambe
(in g ay) which allows o isola e he ision sys em
om changing ligh condi ions [2]. . . . . . . . . . 103
Figu e 6.4 Example o anno a ions on he image da abase.
Maize c ops a e anno a ed in blue and he s ems in
cyan, bean c ops in ed and he s ems in
o ange[2]....................... 104
Figu e 6.5 Schema ic ep esen a ion o he BIPBIP ision sys-
em wi h bo h ha dwa e accele a o possible: a GPU
o he NVIDIA Je son case o an ASIC o he
pla o m4.1a..................... 105
Figu e 6.6 The adap ed ne wo k a chi ec u e used o his
applica ion. The igu e p esen s how he duplica ed
Mobilene laye s and he SSD head a e connec ed o
he Neu oCo gi backbone. . . . . . . . . . . . . . 107
Figu e 6.7 Resul s om he Yolo V4 ne wo k (le ) and he
p oposed SSD ne wo k ( igh ) on maize. Blue ec -
angles show he plan s. G een ec angles show he
s em loca ions. . . . . . . . . . . . . . . . . . . . 108

Lis o Tables
Table 1.1 Senso Modali y Compa ison o Las -Mile Deli e y
AVs........................... 9
Table 1.2 Summa y o ROS 2 Fea u es Compa ed
oROS1[11] ..................... 27
Table 3.1 CNNLaye s ype ................... 64
Table 3.2 B eakdown o Ve sal ACAP VCK190 FPGA esou ces
used by he modules o he p oposed a chi ec u e a e
syn hesis........................ 66
Table 3.3 Di e en execu ion phases in he p oposed
a chi ec u e....................... 68
Table 4.1 Accu acy on es se o FL model wi h 2 clien s o
mul iple lea ning s eps. . . . . . . . . . . . . . . . . 80
Table 4.2 Accu acy on es se o FL model o di e en numbe
o clien s o 2 lea ning s eps. . . . . . . . . . . . . . 81
Table 4.3 Accu acy on es se o FL models o di e en da ase
o e laps o 2 clien s. . . . . . . . . . . . . . . . . . 82
Table 4.4 Accu acy on es se o FL models o di e en da ase
o e laps o 10 clien s. . . . . . . . . . . . . . . . . . 82
Table 4.5 Accu acy on es se o cen alized models o di e en
da ase sizes....................... 84
Table 5.1 KPIs o Measu ing Image A i ac s. . . . . . . . . . 92
Table 6.1 Numbe o images and anno a ions o each c op. . . 104
Table 6.2 De ec ion pe o mance (%) and in e ence speed ( ps)
o Yolo 4 on he NVIDIA Je son Xa ie including
ideo acquisi ion and pos -p ocessing o each c op. . 106
Table 6.3 De ec ion pe o mance (loss unc ion) using he new
a chi ec u e. ...................... 108
x ii
Lis o Con ibu o s
Allene , Thibaul , CEA, F ance
Bah , Roy, SINTEF AS, No way
Bichle , Oli ie , CEA, F ance
Bijani, Sepeh , NXP Semiconduc o s, Ge many
B udeli, Helge, Paxs e AS, No way
Che si, Fabian, CEA, F ance
Da Cos a, Jean-Pie e, Uni e si y o Bo deaux, CNRS, Bo deaux Sciences
Ag o, F ance
Deshayes, Ayme ic, Uni e si y o Bo deaux, CNRS, F ance
Emanuelsson, Pe e , Paxs e AS, No way
Fø isdahl, Ma in, Paxs e AS, No way
Ge main, Ch is ian, Uni e si y o Bo deaux, CNRS, Bo deaux Sciences
Ag o, F ance
Ke esz es, Ba na, Uni e si y o Bo deaux, CNRS, Bo deaux Sciences Ag o,
F ance
K ichene, Hana, Uni e si é Pa is-Saclay, CEA-Lis , F ance
Kuche , Inna, CEA, F ance
Lo ain, Vincen , CEA, F ance
Moineau, Cy il, CEA, F ance
Naud, Maxence, CEA, F ance
P asad, Rohi Uni e si é Pa is-Saclay, CEA-Lis , F ance
Sand, Hans-E ik, NxTECH AS, No way
Saxegaa d, Simen Ma en ius, NxTECH AS, No way
xix
xx Lis o Con ibu o s
Se panos, Dimi ios, Uni e si y o Pa as, G eece
Sou ie , Lau en , CEA, F ance
Templie , Vincen , CEA, F ance
Ve mesan, O idiu, SINTEF AS, No way
Xenos, Geo gios, Uni e si y o Pa as, G eece

1
Ad ancing Edge AI Pe cep ion Pla o ms
and Senso Fusion o Las -Mile Deli e y
Au onomous Vehicles
O idiu Ve mesan1, Roy Bah 1, Hans-E ik Sand2,
Simen Ma en ius Saxegaa d2, Helge B udeli3, Pe e Emanuelsson3,
and Ma in Fø isdahl3
1SINTEF AS, No way
2NxTECH AS, No way
3Paxs e AS, No way
Abs ac
In he apidly e ol ing landscape o anspo a ion, mobili y, and logis ics, he
las -mile ep esen s he inal and c ucial leg o he deli e y jou ney. I in ol es
goods a elling om a anspo a ion hub o he ul ima e des ina ion. This
is he mos expensi e and ime-sensi i e pa o he supply chain business
model. Challenges include na iga ing dense u ban en i onmen s wi h a i-
ous ypes o a ic pa icipan s (e.g., pedes ians, bicycles, animals, elec ic
scoo e s, mo o cycles, e c.), dealing wi h a ic conges ion, loca ing speci ic
deli e y poin s, managing a high densi y o s ops, and handling ailed deli e y
a emp s. In his con ex , he in e sec ion o edge a i icial in elligence (AI),
au onomous sys ems, obo ics, and senso usion in pe cep ion and na i-
ga ion ad ances he de elopmen o las -mile deli e y au onomous ehicle
(AV) pla o ms ha e ol e owa ds so wa e-de ined and AI-de ined ehi-
cles (SDVs and ADVs). The ad ancemen s include mul iple senso sys ems
o pe cep ion and communica ion (e.g., ul asound, ine ial, LiDAR, ada ,
came a, V2X, e c.), eal- ime da a p ocessing o localisa ion, and obus
algo i hms o na iga ion and in e ac ion wi h di e se a ic en i onmen s.
This chap e p esen s he concep and he implemen a ion o an AI-based
1
2Ad ancing Edge AI Pe cep ion Pla o ms and Senso Fusion
pe cep ion and senso usion pla o m echnical solu ion o au onomous las -
mile deli e y in con olled a ic en i onmen s.
Keywo ds: edge AI, pe cep ion, au onomous ehicle, senso usion, objec
ecogni ion, las -mile deli e y.
1.1 In oduc ion and Backg ound
The u u e o mobili y is in elligen , elec ical, au onomous, connec ed, and
sha ed, a ec ing all h ee b oad ypes o mobili y: pe sonal mobili y (mo ing
indi iduals o small g oups o people), mass ansi (mo ing la ge numbe s
o people), and he mo emen o goods.
The las -mile o logis ics e e s o he inal leg o he deli e y jou ney,
ypically om a local dis ibu ion hub o e ail cen e o he end ecipien ’s
loca ion, such as a home o business [1]. This segmen is no o iously he
mos complex, ine icien , and expensi e pa o he en i e supply chain, o en
accoun ing o o e 50% o o al deli e y cos s [3].
Las -mile logis ics a e inc easingly au oma ed, and companies ha a e
p epa ed o his shi a e in a s onge posi ion o compe e and ake he
lead. Au onomous las -mile deli e y, u ilising ehicles anging om small
sidewalk obo s o au oma ed ans, p omises signi ican e iciency gains
and cos educ ions in logis ics. As a esul , a ious ypes o au onomous
ehicles o las -mile deli e y ha e eme ged as ollows [10] and illus a ed in
Figu e 1.1:
• Pedes ian sidewalk ehicles. These a e slow ehicles designed o a el
a a pedes ian speed o 4-6 km pe hou . This low speed o e s imp o ed
sa e y and allows he ope a o s o con ol he ehicle in an eme gency.
• Bicycle sidewalk ehicles. These a e ehicles designed o a el up o a
bicycle speed o 12-15 km pe hou .
Figu e 1.1 Types o ehicles o las -mile deli e y.
1.1 In oduc ion and Backg ound 3
• On- oad deli e y ehicles. These ehicles a e buil o on- oad deli e y
a up o 45-50 km pe hou . Thei so wa e algo i hms and senso
sys ems esemble hose o au onomous ehicles.
D i e less echnology use s u ilise au onomous deli e ies o se e al
pu poses:
• Deli e y o goods om wa ehouses o s o es and ou le s o es ocking
in en o y and shel es.
• Deli e y o goods om s o es o end consume s.
• Deli e y o goods and pa s be ween he wa ehouses and p oduc ion
acili ies.
Au onomous ehicles (AVs), encompassing oad-going ans and smalle
sidewalk au onomous deli e y ehicles, a e eme ging as a po en ially ans-
o ma i e solu ion. By elimina ing he need o a human d i e , AVs o e he
po en ial o 24/7 ope a ion, educed labou cos s (a signi ican componen
o las -mile expense), op imised ou ing, and po en ially lowe emissions,
p ima ily i elec ic. They can na iga e na ow s ee s o pedes ian zones
inaccessible o la ge ehicles and imp o e deli e y imes by a oiding
human- ela ed delays.
The de elopmen o las -mile deli e y au onomous ehicle lee s is linked
o he e olu ion o In e ne o Robo ic Things (IoRT) pla o ms. IoRT se es
as he echnological backbone, in eg a ing indi idual au onomous obo ic
ehicles in o an in e connec ed sys em o sys ems. IoRT combines IoT ech-
nologies wi h obo ics, edge compu ing and AI, allowing o he coo dina ion
o la ge-scale lee s o deli e y obo s ha migh o he wise ope a e alone.
Connec ing and in eg a ing he las -mile deli e y au onomous ehicle in o
lee s ha a e coo dina ed using dis ibu ed ne wo ks o IoRT pla o ms
enables unc ions such as emo e moni o ing, in elligen communica ion,
and he managemen o he en i e deli e y p ocess, which can be managed
om he dis ibu ion cen e o he cus ome ’s doo s ep. The in elligence
and eal- ime esponsi eness o he au onomous deli e y lee s and he
IoRT pla o ms a e signi ican ly enhanced by edge AI. Edge AI embeds
da a p ocessing and decision-making capabili ies di ec ly on o he ehicles
hemsel es, s eng hening he p ocessing capabili ies and he analy ics o
each ehicle in he lee o he IoRT pla o m. The use o edge AI enables
con inuous ou e op imisa ion, obs acles and a ic pa icipan s a oidance,
and eal- ime adap a ion o changing en i onmen al condi ions, which a e
c i ical o na iga ing pedes ian spaces and complex u ban en i onmen s
sa ely and e icien ly. Edge AI-powe ed obo ics wi hin he IoRT amewo k
4Ad ancing Edge AI Pe cep ion Pla o ms and Senso Fusion
calcula e he mos e icien pa hs in eal- ime, ensu ing ha las -mile deli e y
is no only au oma ed bu also in elligen , as , and eliable [5, 6].
Fo au onomous las -mile deli e y o become a eali y, he co e enabling
echnology is obus pe cep ion – he AV’s abili y o sense, in e p e , and
unde s and i s complex and dynamic su oundings. Las -mile en i onmen s,
whe he sidewalks o u ban s ee s, p esen unique pe cep ion challenges:
close-qua e s manoeu ing a ound pedes ians, cyclis s, pe s, pa ked ca s,
s ee u ni u e, and unp edic able obs acles; na iga ing a ied e ain includ-
ing cu bs and une en su aces; in e p e ing complex a ic signals and signs
a in e sec ions; p ecisely iden i ying he inal deli e y loca ion (e.g., a spe-
ci ic doo way o po ch); managing a high densi y o s ops, and handling ailed
deli e y a emp s [2]. Failu es in pe cep ion can lead di ec ly o collisions,
inco ec deli e ies, o mission ailu e.
No single senso can eliably cap u e all necessa y en i onmen al in o -
ma ion unde all condi ions. Came as s uggle in poo ligh ing o wea he ,
LiDAR can be expensi e and has limi a ions in ad e se wea he , ada has
lowe esolu ion, and ul asound has a e y sho ange. The e o e, senso
usion – he in elligen combina ion o da a om mul iple, di e se senso s
is key [14]. By in eg a ing complemen a y da a s eams, senso usion aims
o c ea e a uni ied, comp ehensi e, and eliable en i onmen al model ha is
mo e accu a e and obus han wha could be achie ed wi h indi idual senso s
alone.
Las -mile deli e y au onomous ehicles can ope a e in lee s wi h indi id-
ual ehicles ac ing as cogni i e agen s using pe cep ion modules o p ocess
images, GNSS posi ions o LiDAR scans o au onomous sys em decision-
making, esul ing in ac ions, such as ac ua o commands o V2X messages.
The high deg ee o in e dependencies be ween many unc ional componen s
o au onomous ehicles equi es he implemen a ion o new sys em a chi ec-
u es and new unde lying so wa e amewo ks. The concep s o de eloping
AI-based las -mile au onomous deli e y ehicles a e embedding Robo Ope -
a ing Sys em (ROS) in o compac , scalable, AI-based pe cep ion, localisa ion
and senso usion pla o ms ad ancing he solu ions and applica ions o
au onomous anspo o goods.
An essen ial aspec o he sa e use o las -mile deli e y au onomous
ehicle echnology is de e mining i s capabili ies and limi a ions and com-
munica ing hese o end use s, leading o a s a e o “in o med sa e y”. The
i s s age in es ablishing he capabili y o an au onomous ehicle is de ining
i s Ope a ional Design Domain (ODD). The ODD is de ined in [17] as he
1.2 Senso Fusion in Las -Mile Con ex 11
pa icula ly on ehicles ope a ing o e bumpy e ain, such as sidewalks.
LiDAR is c ucial o p ecise localisa ion and mapping wi hin complex u ban
canyons o sidewalk en i onmen s whe e GPS may be un eliable [21]. I
excels a de ec ing low-lying obs acles, cu bs, po holes, o changes in e ain
ha migh be missed by came as alone. The high cos emains a signi ican
challenge o he las -mile business case [4]. The ypically lowe speeds
and sho e ope a ional anges in las -mile deli e y may allow o he use
o lowe -cos , sho e - ange LiDAR senso s compa ed o hose needed o
high-speed highway au onomy [14]. Fusion wi h came as is essen ial o add
seman ic unde s anding o LiDAR’s geome ic da a.
Rada senso s [19] a e one o he key elemen s o he au onomous ehicle’s
pe cep ion sys em due o hei esilience o ad e se en i onmen al condi ions.
Rada can see h ough da kness and og, and o a ce ain ex en h ough ain,
and snow, condi ions ha se e ely challenge o blind o he senso s, such as
came as. This capabili y ensu es a baseline o ope a ional sa e y and unc-
ionali y, ega dless o he ime o day o wea he condi ions, p o iding da a
on he ange, eloci y, and angle o o he objec s wi h a high deg ee o accu-
acy. The syne gis ic in eg a ion o high- equency ada , 5G communica ion,
and mul imodal ada echnologies g ea ly enhances he sensing capabili y
and en i onmen al adap abili y o au onomous ehicle pe cep ion sys ems
[20]. Howe e , ada s p esen a ew challenges. In hea y ain alls, he adio
signals can su e om a enua ion, sligh ly educing hei e ec i e ange.
Addi ionally, in dense u ban en i onmen s, he adio wa es can bounce o
mul iple su aces be o e e u ning o he senso . This mul i-pa h e lec ion, o
clu e , can c ea e "ghos " a ge s, misleading he ehicle’s pe cep ion sys em
in o “ hinking” an objec is p esen whe e he e is none. The esolu ion o
ada makes i challenging o classi y objec s wi h ce ain y, as i o examples
s uggles o dis inguish be ween a pedes ian, a cyclis , o a s a iona y objec ,
such as a signpos , based on i s signa u e alone.
These challenges a e pa icula ly ampli ied in he con ex o las -mile
deli e y o au onomous ehicles. The ODD o hese ehicles in ol es
na iga ing complex and clu e ed en i onmen s such as esiden ial s ee s,
sidewalks, and loading zones. S anda d au omo i e ada s a e op imised
o de ec ing la ge me allic objec s, such as o he ehicles, and may ail
o eliably de ec smalle , low-p o ile, o non-me allic i ems in hese a eas,
including deli e y packages, cu bs, child en’s oys, o pe s. The p oximi y o
buildings, pa ked ehicles, and o he s ee u ni u e exace ba es he mul i-
pa h e lec ion p oblem, making i mo e challenging o main ain a clea and
accu a e pe cep ion o he immedia e su oundings.

12 Ad ancing Edge AI Pe cep ion Pla o ms and Senso Fusion
IMUs measu e he ehicle’s linea accele a ion and angula eloci y using
accele ome e s and gy oscopes [16]. This da a is in eg a ed o e ime o es i-
ma e changes in eloci y, posi ion, and o ien a ion ( oll, pi ch, yaw). They a e
undamen al o s a e es ima ion and enable dead eckoning na iga ion du ing
pe iods when ex e nal posi ioning signals, such as GPS, a e una ailable [16].
IMUs p o ide high- equency mo ion da a (o en 100 Hz o highe ), com-
ple ely independen o ex e nal signals o en i onmen al condi ions, allowing
con inuous ope a ion in unnels, u ban canyons, dense oliage, o indoo s
[16]. They a e ela i ely low-cos , especially Mic o-Elec o-Mechanical Sys-
ems (MEMS) based uni s, compac , and consume li le powe [16]. IMU da a
is c i ical o s abilising pe cep ion da a om o he senso s (compensa ing o
ehicle mo ion) and o p o iding he mo ion inpu s needed o senso usion
algo i hms, such as Kalman il e s [24]. The p ima y limi a ion o IMUs is
d i , mino e o s in accele a ion and angula eloci y measu emen s accu-
mula e o e ime, leading o apidly inc easing e o s in he es ima ed posi ion
and o ien a ion [16]. This necessi a es equen co ec ions using absolu e
posi ioning senso s (such as GPS) o ela i e posi ioning de i ed om o he
senso s (e.g., LiDAR/came a-based SLAM). IMUs a e sensi i e o empe a-
u e changes and ib a ions, which can a ec hei accu acy [16]. Accu a e
calib a ion is c ucial, bu i can be complex [16]. Magne ome e s, some imes
included o heading e e ence, a e un eliable in u ban en i onmen s due
o magne ic in e e ence om buildings, ehicles, and in as uc u e [25].
IMUs a e indispensable o las -mile na iga ion due o equen GPS signal
deg ada ion o loss in u ban canyons, unde passes, o nea all buildings [16].
The high- equency da a helps main ain a smoo h es ima e o he ehicle’s
s a e, which is c ucial o con olling obo s na iga ing po en ially une en
sidewalks o making equen s ops and s a s. Cos -e ec i e MEMS IMUs
a e gene ally su icien , bu obus usion wi h GPS, LiDAR-SLAM, o isual
odome y is essen ial o bind he inhe en d i [16].
Ul asonic senso s use high- equency sound wa es o de ec he p esence
and dis ance o objec s a e y sho anges [26]. They ope a e on he
p inciple o measu ing he ime-o - ligh o emi ed sound pulses e lec ing
o nea by objec s. They a e ela i ely inexpensi e and easy o in eg a e.
They can de ec objec s close o he ehicle (wi hin a ew me e s), e ec i ely
co e ing blind spo s o en missed by came as o LiDAR [23]. Thei pe o -
mance is la gely una ec ed by ligh ing condi ions (wo k in da kness) o he
colou / anspa ency o he objec [27]. They a e ela i ely obus in some
ad e se wea he condi ions [27]. Ul asound senso s ha e a minimal de ec ion
ange, o up o 4-5 me e s depending on he senso and condi ions [26]. Thei
1.2 Senso Fusion in Las -Mile Con ex 13
angula esolu ion is poo due o b oad beam pa e ns, making i di icul
o dis inguish be ween closely spaced objec s, de e mine objec shape, o
p ecisely loca e small objec s [27]. Pe o mance deg ades signi ican ly a
highe ehicle speeds [23]. They can be suscep ible o in e e ence om
ex e nal ul asonic noise sou ces [28]. They may s uggle o de ec so ,
sound-abso bing ma e ials [27]. Thei p ima y u ili y in las -mile deli e y
is o low-speed, close-qua e s manoeu ing, such as pa king assis ance o
ans, docking a a speci ic deli e y poin , na iga ing e y na ow passages, o
de ec ing immedia e low-lying obs acles like cu bs igh nex o he ehicle
o obo [26]. They can se e as a sa e y senso o de ec ing he p esence
o people nea loading doo s [29]. Due o hei limi ed ange and esolu ion,
hey a e unsui able o p ima y na iga ion o obs acle a oidance a ypical
ope a ional speeds bu se e as a aluable, cos -e ec i e complemen a y
senso o nea - ield sa e y and p ecision manoeu ing.
V2X encompasses echnologies (p ima ily DSRC/IEEE 802.11p and C-
V2X/cellula ) ha enable ehicles o communica e wi elessly wi h o he
ehicles (V2V), oadside in as uc u e (V2I), pedes ians (V2P, o en ia
sma phones), and he ne wo k o cloud (V2N) [30]. I s key ole in pe -
cep ion is enabling coope a i e pe cep ion, whe e senso da a and de i ed
in o ma ion a e sha ed among connec ed en i ies [30]. V2X can d ama ically
ex end a ehicle’s pe cep ion ange and awa eness beyond he line-o -sigh
limi a ions o i s onboa d senso s [30]. By sha ing da a ( aw senso da a,
p ocessed objec lis s, o in en in o ma ion), ehicles can “see” a ound
co ne s o h ough obs uc ions ia he senso s o o he connec ed agen s. V2I
communica ion can p o ide c i ical in o ma ion, such as a ic signal phase
and iming (SPaT), oad haza d wa nings, and wo k zone ale s [30]. This
enhanced si ua ional awa eness can signi ican ly imp o e sa e y and a ic
e iciency, enabling coo dina ed manoeu es such as pla ooning [30]. C-V2X
o e s he po en ial ad an age o le e aging exis ing cellula in as uc u e o
V2N, po en ially p o iding mo e exhaus i e co e age compa ed o DSRC’s
eliance on dedica ed Roadside Uni s (RSUs) [31]. The e ec i eness o V2X,
pa icula ly V2V and V2I o coope a i e pe cep ion, hea ily depends on
widesp ead adop ion and deploymen – a signi ican ne wo k e ec challenge.
Communica ion channels ha e limi a ions in e ms o la ency, eliabili y,
bandwid h, and ange, which can a ec he imeliness and quali y o sha ed
pe cep ion da a. Ensu ing he secu i y and au hen ici y o V2X messages
is pa amoun o p e en malicious a acks (e.g., alse haza d wa nings,
Sybil a acks) [30]. P i acy conce ns exis ega ding he sha ing o ehicle
da a. S anda disa ion is s ill e ol ing, wi h ongoing deba e and egional
14 Ad ancing Edge AI Pe cep ion Pla o ms and Senso Fusion
di e ences be ween DSRC (IEEE 802.11p/ITS-G5) and C-V2X (LTE-V2X,
5G-V2X) hinde ing global in e ope abili y [31]. Deploying he necessa y
in as uc u e (RSUs o DSRC, po en ially upg aded cellula ne wo ks o
C-V2X) in ol es signi ican cos and e o . Coope a i e pe cep ion ia V2X
is po en ially e y aluable in dense, occluded u ban en i onmen s ypical
o las -mile ou es, allowing a deli e y obo o an o pe cei e pedes ians
o ehicles hidden om i s di ec iew [30]. V2I communica ion p o iding
a ic ligh s a us is c ucial o sa e in e sec ion nego ia ion. V2N connec i -
i y can be used o eal- ime upda es o deli e y ou es, ecei ing cus ome
ins uc ions, emo e moni o ing, o po en ially eleope a ion unde challeng-
ing si ua ions. Howe e , eliable connec i i y (cellula o RSU co e age)
migh be inconsis en ac oss all deli e y zones, including dense u ban a eas
o mo e emo e subu ban neighbou hoods. Secu i y is especially c i ical o
au onomous deli e y ehicles, as i p e en s he , hijacking, o dis up ion o
se ice.
GNSS plays a key ole in au onomous las -mile deli e y ehicles by p o id-
ing essen ial posi ioning, na iga ion, and iming in o ma ion. I enables hese
ehicles o accu a ely de e mine hei loca ion and na iga e o deli e y des-
ina ions, acili a ing e icien ou e op imisa ion and eal- ime acking. One
o he p ima y s eng hs o GNSS is i s global co e age, enabling posi ioning
da a o be a ailable i ually anywhe e. I s high accu acy, especially when
complemen ed by augmen a ion sys ems like Real-Time Kinema ic (RTK),
can achie e cen ime e-le el p ecision, which is essen ial o ope a ing in
complex u ban en i onmen s. Addi ionally, GNSS p o ides eal- ime da a
upda es ha suppo con inuous adjus men s du ing deli e ies, making i a
cos -e ec i e solu ion widely a ailable o implemen a ion. GNSS has limi a-
ions as signal in e e ence can occu in u ban en i onmen s, whe e buildings
o unnels obs uc sa elli e signals, leading o deg aded pe o mance. Mul-
ipa h e ec s, whe e signals bounce o su aces, can u he comp omise
accu acy. La ency issues may a ise, a ec ing he sys em’s esponsi eness
in dynamic a ic si ua ions, and ex eme wea he condi ions o sa elli e
ou ages may challenge he eliabili y o GNSS. To e ec i ely u ilise GNSS
in las -mile deli e y ehicles, speci ic equi emen s mus be me . In eg a ing
GNSS wi h o he echnologies, such as ine ial na iga ion sys ems (INS),
LiDAR, and came as, is i al o enhancing accu acy and eliabili y. Robus
so wa e algo i hms a e needed o p ocess GNSS da a and compensa e
o en i onmen al e o s. Ene gy-e icien solu ions a e essen ial o ensu e
con inuous ope a ion wi hou o e bu dening he ehicle’s powe esou ces.
1.3 Au onomous Vehicle A chi ec u e o Las -Mile Deli e y 15
Real- ime da a exchange mus be es ablished o op imise ou es and ensu e
sa e y while also adhe ing o sa e y s anda ds compliance.
1.3 Au onomous Vehicle A chi ec u e o Las -Mile
Deli e y
Au onomous ehicle o anspo o goods conside s he use o scalable
p ocessing capabili ies a he edge wi h AI-based unc ions implemen ed in o
he pe cep ion domain and co e ing he edge compu ing capabili ies imple-
men ed in o ehicles o di e en sizes using he same gene ic a chi ec u e as
illus a ed in Figu e 1.4 [7].
1.3.1 Localisa ion and High-De ini ion Map
Localisa ion is c i ical o he sa e and e icien ope a ion o las -mile deli e y
au onomous ehicles. By employing high-de ini ion (HD) maps ha de ail
u ban in as uc u e, such as oad ypes, cu b loca ions, and a ic signals,
hese ehicles can na iga e complex en i onmen s mo e e ec i ely.
The maps inco po a e eal- ime da a upda es o e lec changing oad
condi ions, enhancing na iga ional accu acy and sa e y.
1.3.2 Pe cep ion Implemen a ion
Pe cep ion in las -mile deli e y au onomous ehicles in eg a es AI o iden-
i y and classi y a ious objec s wi hin he ehicle’s icini y. This in ol es
Figu e 1.4 Vehicles and scalabili y. Sou ce: [7].
16 Ad ancing Edge AI Pe cep ion Pla o ms and Senso Fusion
Figu e 1.5 Pe cep ion and senso s usion. Sou ce: [8].
a usion o da a om mul iple sou ces, whe e machine lea ning (ML)
algo i hms help p edic po en ial obs acles and dynamic changes in he
en i onmen .
The esul an da a enhances si ua ional awa eness and in o ms decision-
making p ocesses. An o e iew o he senso s used in he pe cep ion and
senso usion pla o ms o au onomous ehicles o las -mile deli e y o
goods is illus a ed in Figu e 1.5.
Senso usion, AI p ocessing and decision-making
The o e all sys em a chi ec u e o he au onomous ehicle comp ises
con olle s o he pe cep ion, senso usion and ac ions o he ehicle’s
ac ua o s based on he sensed en i onmen , objec i es, and cons ain s. I is
di ided in o h ee p ima y blocks: de ec ion, pe cep ion, and decision policy,
as illus a ed in Figu e 1.6 [9].
The main sys em managemen componen s consis o he Ope a ing
Sys em (OS) based on Linux Ubun u, and he middlewa e based on he ROS.
ROS1 is a high-le el API o e alua ing senso da a and con olling
ac ua o s.
The in eg a ion ac i i ies on senso usion, combines homogeneous
and he e ogeneous da a om di e en sou ces like he pe cep ion sen-
so s (LiDAR, came as, ul asonic senso s, e c.) o acili a e AI p ocessing,
decision-making, and planning.
The pe cep ion wo k low o image ecogni ion, objec de ec ion and
acking a e illus a ed in Figu e 1.7 [9]. Technology w appe s a e used o
in eg a e di e en p o ocols, da a o ma s, and in e aces seamlessly.

1.3 Au onomous Vehicle A chi ec u e o Las -Mile Deli e y 17
AI p ocessing and he au onomous sys ems a e in eg a ed using pe -
cep ion senso s o mapping he en i onmen , HW/SW componen s o he
acquisi ion, p ocessing, agg ega ion, analysis, and in e p e a ion o da a, AI-
based algo i hms and me hods o si ua ion assessmen , ac ion planning,
cogni i e decision-making, and ac ua o s o ac ing on he s ee ing, b aking
and p opulsion sys ems.
Va ious AI amewo ks, such as Py hon, PyTo ch, Ke as, and Tenso Flow,
as well as se e al machine ision lib a ies like OpenCV, SimpleCV, he Poin
Figu e 1.6 Au onomous ehicle o las -mile deli e y – Pla o m in eg a ion componen s.
Sou ce: Adap ed om [9].
Figu e 1.7 Pe cep ion wo k low o image ecogni ion, objec de ec ion and acking.
Sou ce: Adap ed om [9].
18 Ad ancing Edge AI Pe cep ion Pla o ms and Senso Fusion
Cloud lib a y, and YOLO, and AI-based compu ing pla o ms like NVIDIA
Je son AGX O in, we e e alua ed o in eg a ion in o di e en laye s o
au onomous ehicle a chi ec u e.
These AI-based amewo ks, algo i hms, lib a ies, and pla o ms we e
u ilised du ing a ious phases o he au onomous ehicle pla o m demon-
s a o ’s de elopmen .
The decision-making elies on senso usion and AI p ocessing.
Fu he mo e, he ehicle pla o m ea u es se e al ac ua o s and con-
ol uni s, including he elec ic s ee ing se o and h o le/speed con ol
o au onomous d i ing, speech in o ma ion/ ecogni ion o use-case se -
ice/secu i y pu poses, NFC/mobile locking/unlocking sys ems, and wi e-
less/wi ed eme gency s ops o sa e y easons.
Pe cep ion senso s and na iga ion
The pla o m was in eg a ed wi h he NVIDIA Je son AGC O in, which
p o ides AI-based pe cep ion and senso usion capabili ies [9]. To abs ac
he senso b and and in e ace om O in, pa sing o he ul asonic sen-
so elec onic con ol uni (ECU) da a we e implemen ed and an in e ace
p o ided (independen o ul asonic sys em used).
Ha ing he ehicle con ol uni (VCU) in e p e he senso da a also
enables i o ha e an eme gency b ake unc ion. This unc ion is se so ha i
he ehicle is au onomous mode, he ehicle’s VCU sends a signal o disable
d i e o he CAN elay, which in u n igge s engaging o he elec onic pa k
b ake.
To ge he ul asonic senso s o ha e an impac on he ehicle mo ion
he e needs o be se e al in e aces de ined whe e he da a and in o ma ion
can low. The e a e wo dis inc ypes o in e aces in o m o CAN and ROS
opics. An illus a ion o he in e aces is gi en in Figu e 1.8.
The ul asonic senso ha dwa e abs ac ion laye (HAL) p o ides an in e -
ace ha includes he CAN ou pu om he VCU in e ace bu p o ides i as
an ROS opic ha o he nodes inside NVIDIA Je son AGX O in pla o m can
make use o . The opic is desc ibed in a message and p o ides in o ma ion
o each senso in a sepa a e a iable.
T acking Sys em o Pla ooning
Pla ooning o au onomous ehicles e e s o a o ma ion o mul iple
au onomous ehicles a elling closely oge he in a single- ile line, wi h he
lead ehicle con olling he speed and di ec ion o he ollowing ehicles.
The g oup o connec ed au onomous ehicles exchange in o ma ion, allowing
1.3 Au onomous Vehicle A chi ec u e o Las -Mile Deli e y 19
Figu e 1.8 Sys em o e iew. Sou ce: Adap ed om [9].
hem o d i e in a coo dina ed manne , wi h e y small spacings, while s ill
a elling sa ely a ela i ely high speeds [36].
The lee s o las -mile deli e y au onomous ehicles u ilise pla ooning
coo dina ed d i ing s yle o enhance logis ics e iciency, inc ease oad capac-
i y, imp o e a ic low, educe deli e y imes, and load goods om common
wa ehouse hubs.
Pla ooning equi es he de elopmen o obus and eliable V2V
communica ion, mul i-senso pe cep ion sys ems, con ol algo i hms, and
in as uc u e, including dedica ed lanes o speci ic oad con igu a ions, o
unc ion op imally.
In o ma ion on he ehicles’ speeds, posi ions, accele a ions, decele a-
ions, and o he ele an da a o he ehicles in he pla oon, as well as o
hose joining o lea ing he pla oon, is c ucial, as all he ehicles in he
pla oon need o eac e icien ly and sa ely in eal- ime.
Se e al in o ma ion low opologies (IFTs) ha e been adi ionally used in
he li e a u e, such as p edecesso - ollowing (PF), wo-p edecesso - ollowing
(TPF), and bidi ec ional (BDL). The ad ancemen o communica ion sys-
ems inc eased he use o mo e gene al schemes such as -p edecesso
ollowing ( PLF). The dynamic pla oon na u e, wi h ehicles changing
hei ela i e posi ion o e ime, also adds complexi y o he opology o
communica ions [36].
The pla ooning s yle o d i ing can be implemen ed o au onomous ehi-
cles equipped wi h V2X connec i i y by con eying a ic in o ma ion (e.g.,
20 Ad ancing Edge AI Pe cep ion Pla o ms and Senso Fusion
GNSS, speed, o signal iming) and using ei he unlicensed V2X (ITS-G5) o
cellula V2X (LTE-V2X/NR-V2X) [37].
The pla ooning o las -mile deli e y au onomous ehicles can be imple-
men ed using he ehicle’s pe cep ion senso s (e.g., came as, ul asound) o
a eas wi h good isibili y, and an al e na i e solu ion can complemen he
V2X sys em.
The ollowing sec ion p esen s he implemen a ion o a acking sys em
o pla ooning, as demons a ed in he ECSEL JU AI4CSM p ojec [9],
u ilising an NVIDIA Je son AGX O in p ocessing pla o m unning he
Ubun u Linux ope a ing sys em, which o e s AI capabili ies o he ehicle’s
pe cep ion domain. The ehicle’s onboa d uni communica es wi h i s senso s
h ough he ROS ope a ing sys em as middlewa e.
The au onomous ehicle, illus a ed in he Figu e 1.15, is equipped
wi h mul iple pe cep ion senso s, including LiDAR, a dep h came a, and
ul asound senso s.
Fo he implemen a ion, he In el RealSense D455 RGB-D dep h came a
was used as a senso o de ec he logo ma k placed on he ea o he lead
ehicle, se ing as a a ge o he ollowe ehicle o ollow.
The logo in Figu e 1.9 is a ached o he ea o he ehicle. The logo
design can make i di icul o dis inguish i om o he ci cula objec s o
signs wi h s aigh lines wi hin he ci cle, such as no s opping/pa king signs.
To de ec he logo, wo di e en de ec ion model amewo ks we e es ed,
YOLO 5 [38] and YOLO 8 [39]. YOLO is a compu e ision model de el-
oped by Ul aly ics and is pa o he "You Only Look Once" (YOLO) amily
o models, known o hei high in e ence speed, making hem sui able o
Figu e 1.9 T acking sys em logo and no s opping/pa king sign.
1.4 Edge AI Pla o ms 27
1.4.1 Robo Ope a ing Sys em
ROS is no a adi ional ope a ing sys em in he sense o Windows o Linux.
Ins ead, i is a lexible amewo k o so wa e lib a ies and ools ha simpli y
he c ea ion o complex obo applica ions [45, 46].
ROS is an open-sou ce amewo k o w i ing obo so wa e, p o iding a
collec ion o ools, lib a ies, and con en ions ha aim o simpli y he ask o
c ea ing complex and obus obo beha iou ac oss a wide a ie y o obo ic
pla o ms. I has also been ecen ly used in au onomous ehicles, as seen in
he case o his implemen a ion o las -mile deli e y au onomous ehicles
and o he au onomous sys ems. ROS is a he a middlewa e, a se o so wa e
amewo ks o obo so wa e de elopmen [35, 40, 47, 48].
The e a e wo e sions o ROS: ROS 1, which e ol ed wi h communi y
con ibu ions, and ROS 2, eleased in 2017. ROS 2 inco po a es eal- ime
capabili ies, imp o ed secu i y, and be e suppo o dis ibu ed sys ems by
le e aging he Da a Dis ibu ion Se ice (DDS) s anda d [45]. A able o key
di e ences be ween ROS 1 and ROS 2 can be seen in Table 1.2 [11].
ROS p o ides se ices expec ed om an ope a ing sys em, including
ha dwa e abs ac ion, low-le el de ice con ol, implemen a ion o commonly
Table 1.2 Summa y o ROS 2 Fea u es Compa ed o ROS 1 [11]
Ca ego y ROS 1 ROS 2
Ne wo k T anspo Tailo ed p o ocol buil
on TCP/UDP
Exis ing s anda d (DDS), wi h
abs ac ion suppo ing addi ion o
o he s
Ne wo k A chi ec u e Cen al name se e
( osco e)
Pee - o-pee disco e y
Pla o m Suppo Linux Linux, Windows, macOS
Clien Lib a ies W i en independen ly
in each language
Sha ing a common unde lying C
lib a y ( cl)
Node s. P ocess Single node pe
p ocess
Mul iple nodes pe p ocess
Th eading Model Callback queues and
handle s
Swappable execu o
Node S a e Managemen None Li ecycle nodes
Embedded Sys ems The ROSSe ial clien
lib a y used o small,
embedded de ices
The mic o-ROS s ack in eg a es
mic ocon olle s wi h s anda d
ROS 2
Pa ame e Access Auxilia y p o ocol
buil on XMLRPC
Implemen ed using se ice calls
Pa ame e Types Type in e ed when
assigned
Type decla ed and en o ced

28 Ad ancing Edge AI Pe cep ion Pla o ms and Senso Fusion
used unc ionali y, message passing be ween p ocesses, and package man-
agemen .
The co e o ROS is i s anonymous publish/ subsc ibe messaging sys em.
A p ocess (called a “node”) ha has in o ma ion o sha e can publish i o
a speci ic “ opic”. O he nodes in e es ed in ha ype o in o ma ion can
subsc ibe o he opic o ecei e he messages, which c ea es a modula ,
decoupled a chi ec u e whe e di e en pa s o he sys em can be de eloped
and es ed independen ly [42, 43] .
The in eg a ion o ROS in o au onomous ehicles in ol es se e al key
aspec s, such as [41]:
•Ha dwa e abs ac ion, whe e ROS p o ides a s anda dised in e ace
o a wide a ie y o senso s and ac ua o s, meaning ha a high-le el
au onomous d i ing algo i hm can be de eloped independen ly o he
speci ic ha dwa e being used. In his con ex , a “LiDAR d i e ” node
could publish da a om a pa icula b and o LiDAR o a s anda dised
opic, and a pe cep ion node could subsc ibe o ha opic wi hou
needing o know he speci ics o he LiDAR ha dwa e.
•In e -p ocess communica ion: Au onomous ehicles ha e a mul i ude
o p ocesses unning concu en ly: pe cep ion, localisa ion, planning,
and con ol. ROS’s messaging sys em allows hese p ocesses o com-
munica e wi h each o he in a eliable and ime-synch onised manne ,
e en i hey a e unning on di e en compu e s wi hin he ehicle.
•Ecosys em and ools: ROS has an ecosys em o ools o isualisa ion,
simula ion, and da a logging. Tools like RViz enable de elope s o isu-
alise senso da a and he ehicle’s s a e in 3D, while Gazebo p o ides a
ealis ic simula ion en i onmen o es ing algo i hms wi hou equi ing
a physical ehicle.
ROS is used as a pla o m o de eloping pe cep ion and senso usion
sys ems [49] in he implemen a ion o he las -mile deli e y au onomous
ehicle p esen ed in his chap e .
As a esul , se e al ea u es o he pla o m we e analysed, e alua ed and
in eg a ed in o he ehicle a chi ec u e. These elemen s a e desc ibed below:
•AI in eg a ion: The modula na u e o ROS makes i easy o in eg a e
AI and machine lea ning lib a ies. The ypical app oach used was o
ha e a ROS node ha u ilises a lib a y such as Tenso Flow o PyTo ch
o pe o m objec de ec ion o seman ic segmen a ion on came a images.
The esul s o his p ocess (e.g., he loca ions o o he ehicles and
pedes ians) we e hen published o a ROS opic o o he nodes o use.
1.4 Edge AI Pla o ms 29
•Senso usion: The implemen ed au onomous ehicles ely on a a ie y
o senso s, including came as, LiDAR, ul asound, IMUs, e c. ROS
p o ided a amewo k o using he da a om hese di e en senso s
o c ea e a mo e accu a e and obus unde s anding o he en i onmen .
A “senso usion” node could subsc ibe o opics con aining da a om
he came a, LiDAR, and IMU, and hen use a il e (like a Kalman
il e ) o combine his da a and p oduce a uni ied ep esen a ion o he
en i onmen .
The o iginal implemen a ion o he unc ions o he las -mile deli e y
au onomous ehicles was implemen ed in he o iginal e sion o ROS (ROS
1). The es esul s show ha he sys em has some limi a ions in e ms o
eal- ime pe o mance and secu i y, and he newe e sion, ROS 2 [44, 45], is
in eg a ed in o he new ehicle design due o he ollowing [35]:
•Real- ime capabili ies: ROS 2 is buil on op o he DDS s anda d,
which p o ides eal- ime, eliable, and scalable communica ion. The
capabili ies a e necessa y o au onomous ehicle implemen a ion o
educe delays and inc ease he sys em’s obus ness.
•Secu i y: ROS 2 includes a be e secu i y amewo k ha p o ides
ea u es like au hen ica ion, enc yp ion, and access con ol, which a e
essen ial o p o ec ing he ehicle om cybe a acks.
•Quali y o Se ice (QoS): ROS 2 allows speci ying QoS policies o
each publishe and subsc ibe , which enables he con ol o aspec s like
eliabili y, du abili y, and la ency, ensu ing ha c i ical da a is deli e ed
in a imely and eliable manne .
A se o p inciples and speci ic equi emen s guides he design o ROS
2, including dis ibu ion, abs ac ion, asynch ony, and modula i y, as well
as se e al design equi emen s such as secu i y, in eg a ion o embedded
sys ems, use o di e se communica ion ne wo ks, eal- ime compu ing, and
p oduc eadiness.
The ROS 2 APIs p o ide access o communica ion pa e ns, such as
se ices and ac ions, which a e o ganised unde he concep o a node.
ROS 2 also p o ides APIs o pa ame e s, ime s, launch, and o he auxil-
ia y ools, which can be used o design a obo ic sys em. ROS 2 issues a
eques - esponse s yle pa e n, known as se ices. Reques - esponse commu-
nica ion p o ides a clea associa ion be ween a eques and i s co esponding
esponse, which can be help ul when ensu ing ha a ask was comple ed o
ecei ed. A unique communica ion pa e n o ROS 2 is he ac ion. Ac ions
a e goal-o ien ed and asynch onous, p o iding communica ion in e aces
30 Ad ancing Edge AI Pe cep ion Pla o ms and Senso Fusion
wi h eques - esponse capabili ies, pe iodic eedback, and he abili y o be
cancelled. The middlewa e a chi ec u e o ROS 2 consis s o se e al abs ac-
ion laye s dis ibu ed ac oss many decoupled packages. These abs ac ion
laye s enable mul iple solu ions o he equi ed unc ionali y, such as a ious
middlewa e o logging solu ions. Addi ionally, he dis ibu ion ac oss a ious
packages allows use s o eplace componen s o ake only he necessa y
pieces o he sys em, which may be impo an o ce i ica ion [11, 12].
Figu e 1.14 displays he laye s wi hin ROS 2 as i is a se o so wa e
lib a ies and ools o building obo and au onomous sys ems applica ions.
ROS2 builds upon DDS and con ains a DDS abs ac ion laye . Use s do no
need o be awa e o he DDS APIs due o his abs ac ion laye . This laye
enables ROS2 o ha e high-le el con igu a ions and op imises he u ilisa ion
o DDS. Addi ionally, due o he use o DDS, ROS2 does no equi e a mas e
p ocess [7, 11, 12].
The clien lib a ies p o ide access o he co e communica ion APIs. They
a e ailo ed o each p og amming language o make hem mo e idioma ic and
ake ad an age o language-speci ic ea u es. Communica ion is agnos ic o
how he sys em is dis ibu ed ac oss compu e esou ces, whe he hey a e in
he same p ocess, a di e en p ocess, o e en a di e en p ocessing uni . A
Figu e 1.14 ROS 2 a chi ec u e [7, 12].
1.5 Fu u e Conside a ions and Resea ch 31
use may dis ibu e hei applica ion ac oss mul iple machines and p ocesses,
and e en le e age cloud compu e esou ces, wi h minimal changes o he
sou ce code. ROS 2 can connec o cloud and edge esou ces o e he in e ne .
The clien lib a ies ely on an in e media e in e ace ha p o ides s an-
da d unc ionali y o each clien lib a y. This lib a y is w i en in C and is
used by all he clien lib a ies, al hough i is no equi ed o hei ope a ion.
The middlewa e abs ac ion laye , called RMW (ROS Middlewa e), p o ides
he essen ial communica ion in e aces. The endo s o each middlewa e
implemen he RMW in e ace and a e made in e changeable wi hou code
changes.
Use s may choose di e en RMW implemen a ions, and he eby di e en
middlewa e echnologies, based on a ious cons ain s such as pe o mance,
so wa e licensing, o suppo ed pla o ms. The ne wo k in e aces (e.g.
opics, se ices, ac ions) a e de ined, and ROS 2 de ines hese ypes using
speci ic o ma iles.
Communica ions a e agnos ic o he loca ion o endpoin s wi hin
machines and p ocesses. Nodes w i en as componen s can be alloca ed o
any p ocess as a con igu a ion, allowing mul iple nodes o be con igu ed o
sha e a p ocess, he eby conse ing sys em esou ces o educing la ency.
1.5 Fu u e Conside a ions and Resea ch
1.5.1 Deploymen Conside a ions
Au onomous ehicles o las -mile deli e y can eshape he inal s ep o
he supply chain, om he dis ibu ion cen e o cus ome s’ doo s eps. The
p ima y ad an age o au onomous deli e y is he po en ial o signi ican
cos educ ion and inc eased e iciency. These ehicles can ope a e a ound
he clock, esul ing in as e deli e y imes and imp o ed lee u ilisa ion.
They can also be designed o be mo e en i onmen ally iendly, con ibu ing
o mo e sus ainable logis ics p ac ices.
The deploymen o unmanned g ound and ae ial au onomous ehicles
and mobile obo s equi es ca e ul planning and conside a ion o a ious
ac o s such as egula o y compliance, in eg a ion wi h exis ing logis ics
in as uc u e, and public pe cep ion. Add essing hese ac o s is essen ial
o he success ul adop ion o au onomous deli e y ehicles in u ban a eas.
Pa ne ships wi h local go e nmen s and logis ics p o ide s can acili a e
smoo he in eg a ion and expansion o se ices.
32 Ad ancing Edge AI Pe cep ion Pla o ms and Senso Fusion
Figu e 1.15 The au onomous ehicle [10].
The success o he echnology depends on na iga ing complex u ban
en i onmen s, which include e e y hing om busy s ee s o unp edic able
beha iou o pedes ians and a ic pa icipan s.
The pa h o widesp ead adop ion o las -mile deli e y au onomous ehi-
cles is illed wi h challenges. The echnology is s ill e ol ing, and ensu ing
he sa e y and eliabili y o au onomous sys ems is a op p io i y as he
ehicles mus be able o na iga e a wide ange o eal-wo ld scena ios, om
inclemen wea he o unexpec ed oad closu es.
Public accep ance and he impac on he wo k o ce a e also c i ical
conside a ions. Gaining he us o consume s and in eg a ing hese ehicles
in o daily li e wi hou causing dis up ion is key.
Gene a i e AI can enhance he in e ac ion be ween au onomous ehicles
and use s h ough na u al language p ocessing, allowing use s o communi-
ca e wi h deli e y sys ems using oice commands, while acili a ing seamless
use expe iences whe e cus ome s can ack o de s o in e ac di ec ly wi h
se ice in e aces.
Add essing u u e ends and o e coming echnological, egula o y, and
social challenges, while de eloping new echnologies and AI-assis ed ools,
is key o unlocking he ull po en ial o au onomous las -mile deli e y.
1.5.2 Fu u e esea ch
Au onomous las -mile deli e y applica ions a e le e aging ad ancemen s in
edge AI pla o ms and senso usion echnologies (came a, LiDAR, IMU,

1.5 Fu u e Conside a ions and Resea ch 33
ul asound, V2X, e c.), speci ically ailo ed o mee he unique equi emen s
and cons ain s o hese applica ions, including close-qua e s na iga ion,
in e ac ion wi h VRUs, and se e e cos and powe limi a ions.
Key a chi ec u al concep s, including a ious le els and s a egies o
senso usion, a e employed alongside enabling algo i hms (e.g., deep
lea ning me hods such as CNNs and T ans o me s) and ha dwa e com-
ponen s (e.g., GPUs, specialised accele a o s, and edge compu ing pla -
o ms). Signi ican in eg a ion challenges, encompassing senso calib a ion,
da a synch onisa ion, compu a ional load managemen , powe consump ion,
cos -e ec i eness, and aul ole ance, s ill need o be add essed.
The p essing esea ch challenges a e achie ing obus pe cep ion in
ad e se condi ions, eliable VRU de ec ion, handling senso limi a ions, and
ensu ing he sa e y and alida ion o au onomous sys ems.
Key u u e esea ch di ec ions include no el usion a chi ec u es, end- o-
end lea ning, sel -supe ised me hods, enhanced V2X in eg a ion, explain-
able edge AI (XAI) and in e p e able edge AI (IAI) o usion, ligh weigh
models o edge deploymen , and ad anced simula ion o alida ion, all
aimed a ad ancing he sa e y, eliabili y, and e iciency o au onomous
las -mile deli e y.
Fu u e wo k includes he concep o a dedica ed a chi ec u e and a
mul imodal AI-based au onomous ehicle pla o m o pe cep ion, au oma ed
con ol, and decision-making in deli e ing goods in con olled en i onmen s.
The wo k add esses e alua ing he in eg a ion o da a om mul iple senso s.
Mul imodal AI and gene a i e AI enable eal- ime co ela ion and a mo e
comp ehensi e unde s anding o he ehicle’s su oundings, allowing o
ehicle con ol h ough oice and ges u e commands.
Fu u e wo k plans o add ess he adop ion o new concep s p o ided
by So wa e-De ined and AI-De ined Vehicles (SDV/AIDV) a chi ec u es,
whe e he ehicle’s ea u es and unc ionali y a e de e mined by he holis ic
in e play be ween senso s/ac ua o s, he ha dwa e, so wa e, AI pla o ms,
and da a and ROS is well-sui ed o his new pa adigm, as i p o ides a
lexible and modula pla o m o de eloping and deploying a wide ange
o applica ions.
As las -mile deli e y au onomous ehicles ope a e in lee s and a e con-
nec ed, he e is a g owing end owa ds o loading he compu a ional asks
o he edge. ROS 2’s suppo o DDS makes i easy o ex end he ehicle’s
communica ion sys em o include edge-based se ices.
Fu he esea ch is add essing he ad ancemen s o au onomous deli -
e y ehicles, ocusing on enhancing he human-machine in e aces be ween
34 Ad ancing Edge AI Pe cep ion Pla o ms and Senso Fusion
AI-d i en ehicles and humans ope a ing indi idual ehicles o lee s,
imp o ing secu i y measu es o p o ec agains ampe ing, and e ining he
sus ainabili y aspec s o au onomous ehicle ope a ions h ough a ious use
cases and business models. Human-au onomous sys em in e ac ion and he
de elopmen o new human-machine in e aces equi e u he in es iga ion
o enhance us , accep ance, and he success ul in eg a ion o las -mile
deli e y au onomous ehicles and IoRT in o daily li e.
Fu u e esea ch combining las -mile deli e y au onomous ehicles, IoRT,
and edge AI needs o ocus on ad ancing decen alised swa m in elligence,
enabling lee s o au onomous ehicles o collabo a e and make collec i e
decisions o imp o e scalabili y and esilience, allowing he lee o adap
o un o eseen e en s in eal- ime. Signi ican esea ch should be di ec ed
owa ds de eloping mo e ad anced and ene gy-e icien edge AI algo i hms
o p edic i e analysis, an icipa ing deli e y demand, and op imising V2X
communica ion.
Robus secu i y, p i acy-p ese ing p o ocols, and us wo hiness wi hin
he IoRT amewo k a e c ucial o sa egua ding sensi i e deli e y da a and
p o ec ing au onomous sys ems om cybe h ea s, making his an impo an
u u e a ea o esea ch.
Fu he esea ch in es iga ion includes he concep o a dedica ed
a chi ec u e and a mul imodal AI-based au onomous ehicle pla o m o
pe cep ion, au oma ed con ol, and decision-making in deli e ing goods in
con olled en i onmen s. The esea ch also includes e alua ing he in eg a-
ion o da a om mul iple senso s, combined wi h decision suppo ha
in eg a es small language models, ision language models, and agen ic AI.
Mul imodal AI and gene a i e AI enable eal- ime co ela ion and a mo e
comp ehensi e unde s anding o he ehicle’s su oundings, enabling he
implemen a ion o ehicle con ol h ough oice and ges u e commands.
Ano he key ocus is on s anda disa ion and he de elopmen o
AI-assis ed ools ha imp o e he capabili ies o amewo ks like ROS
and inc ease he e iciency o designing las -mile deli e y au onomous
ehicles.
1.6 Conclusion
Ea ly implemen a ions o las -mile deli e y au onomous ehicle echnolo-
gies demons a e signi ican po en ial o cos educ ion and e iciency
enhancemen in las -mile logis ics. None heless, lessons om hese ope -
a ions unde sco e he complexi y o u ban en i onmen s and he need o
1.6 Conclusion 35
con inuous adap a ion o echnologies o mee eal-wo ld challenges. Key
akeaways include p io i ising sa e y, enhancing AI-based decision-making
ac oss di e se scena ios, and main aining obus alida ion me hodologies
o au onomous sys ems.
AI in eg a es da a om mul iple senso s, including came as, LiDAR,
IMU, ul asound senso s, and GNSS, o c ea e a comp ehensi e unde -
s anding o he ehicle’s su oundings. This usion o senso y in o ma ion
enhances he ehicle’s decision-making capabili ies, as i can analyse and
in e p e complex da ase s o asce ain he sa es and mos e icien ope a ion.
AI se es as he backbone o au onomous deli e y sys ems, d i ing hei
e iciency, sa e y, and use engagemen . By ha nessing ad anced algo i hms
and echnologies, hese sys ems can p o ide as e , eliable, and mo e cos -
e ec i e deli e y solu ions, e olu ionising he logis ics and deli e y sec o s.
As AI echnology con inues o de elop, i s ole in enhancing au onomous
deli e y ope a ions is expec ed o g ow, pa ing he way o e en mo e
inno a i e solu ions in his a ea.
Acknowledgemen s
This publica ion has ecei ed unding h ough he p ojec s ECSEL Join
Unde aking (JU) AI4CSM, Chips JU EdgeAI, and Chips JU MOSAIC.
The ECSEL JU AI4CSM “Au omo i e In elligence o Connec ed Sha ed
Mobili y” p ojec was suppo ed by he ECSEL Join Unde aking and
i s membe s, including op-up unding om Aus ia, Belgium, he Czech
Republic, I aly, La ia, Li huania, he Ne he lands, and No way unde g an
ag eemen No. 101007326. The Chips JU EdgeAI, “Edge AI Technologies
o Op imised Pe o mance Embedded P ocessing,” p ojec is suppo ed by
he Chips Join Unde aking and i s membe s, including op-up unding
om Aus ia, Belgium, F ance, G eece, I aly, La ia, he Ne he lands, and
No way, unde g an ag eemen No. 101097300. The Chips JU MOSAIC
“A Mosaic o Essen ial Elec onic Componen s and Sys ems (ECS) o ou
Au oma ed Digi al Fu u e in Indus y and Mobili y” p ojec is suppo ed by
he Chips Join Unde aking and i s membe s including op-up unding by
Aus ia, Belgium, Czech Republic, Denma k, F ance, G eece, Is ael, I aly,
La ia, Ne he lands, No way, Poland, and Tü kiye unde g an ag eemen No
101194414. Funded by he Eu opean Union. Views and opinions exp essed
a e, howe e , hose o he au ho (s) only and do no necessa ily e lec hose
o he Eu opean Union o Chips Join Unde aking. Nei he he Eu opean
Union no he g an ing au ho i y can be held esponsible o hem.
36 Ad ancing Edge AI Pe cep ion Pla o ms and Senso Fusion
The au ho s would like o acknowledge he con ibu ions o Espen Teigen,
ISPAS AS, No way, o he so wa e implemen a ion o he odome y and
pa h planning modules desc ibed in his a icle du ing his enu e a NxTECH
AS, No way.
Re e ences
[1] M. G. Augus o e al., “Au onomous Van and Robo Las -Mile Logis ics
Pla o m: A Re e ence A chi ec u e and P oo o Concep Implemen a-
ion,” Logis ics, ol. 9, no. 1, pp. 10–10, Jan. 2025, h ps://doi.o g/10.3
390/logis ics9010010.
[2] Bo Penguin, “Las Mile Deli e y,” Bo penguin.com, 2024. h ps://bo p
enguin.com/glossa y/las -mile-deli e y.
[3] Wise Sys ems, “The S a e o he Au onomous Las -Mile Ma ke in
2023,” Wise Sys ems, No . 02, 2023. h ps://www.wisesys ems.co
m/blog/au onomous-las -mile-ma ke -2023/
[4] V. Engesse , E. Rombau , L. Vanha e beke, and P. Lebeau,
“Au onomous Deli e y Solu ions o Las -Mile Logis ics Ope a ions: A
Li e a u e Re iew and Resea ch Agenda,” Sus ainabili y, ol. 15, no. 3,
p. 2774, Feb. 2023, h ps://doi.o g/10.3390/su15032774.
[5] O. Ve mesan e al., “4 In e ne o Robo ic Things -Con e ging Sens-
ing/Ac ua ing, Hype connec i i y, A i icial In elligence and IoT Pla -
o ms.” A ailable a : h ps://www. i e publishe s.com/pd /ebook/chap
e /RP_9788793609105C4.pd
[6] O. Ve mesan e al., “In e ne o Robo ic Things In elligen Connec i i y
and Pla o ms,” F on ie s in Robo ics and AI, ol. 7, Sep. 2020, h ps:
//doi.o g/10.3389/ ob .2020.00104.
[7] O. Ve mesan, “Embedding ROS and Edge AI-Based Pe cep ion Capa-
bili ies in AV Pla o ms,” Eu opean Con e ence on EDGE AI Tech-
nologies and Applica ions – EEAI 2023, 17-19 Oc obe 2023, A hens,
G eece. h ps://edge-ai- ech.eu/wp-con en /uploads/2023/11/2023-10
-19_P esen a ion_C03_2_Con e ence_EDGE-AI_A hens_EEAI_2023.
pd .
[8] O. Ve mesan, “T us wo hy Edge AI Solu ions in Au onomous Vehi-
cle Pe cep ion and Senso Fusion,” Eu opean Con e ence on EDGE
AI Technologies and Applica ions – EEAI 2024, 21-23 Oc obe 2024
Caglia i, I aly. h ps://edge-ai- ech.eu/wp-con en /uploads/2024/11/202
4-10-23_P esen a ion_B03-3_Con e ence_EDGE-AI_Caglia i_EEA
I_2024.pd .
2.1 In oduc ion and Backg ound 43
In o de o be able o p oduce highly op imized and a ge -speci ic imple-
men a ions o desi ed DNNs, we de eloped Aidge, a amewo k con aining
ools and me hods ha allow use s o ac bo h a he g aph-le el and a he
ope a o le el. Aidge is hus a he same ime a neu al ne wo k g aph edi o
and a compile ha accep s high-le el desc ip ions o DNNs (e.g. in ONNX)
and p oduces low-le el code (e.g. in C o assembly) op imized and a ge ed
a chosen ha dwa e back-ends.
One o he g ea challenges in gene a ing op imized code om high-le el
desc ip ions is he ac ha di e en a chi ec u es manage ope a ions, da a
and memo y in di e en ways. Fo example, Deep Lea ning Accele a o s
(DLAs) [5] [1] [6] usually implemen op imized enso compu e p imi i es,
while GPUs [7] exploi hei massi e pa allelism, and mode n CPUs [8] [9]
[10] con ain ec o ized ins uc ions. Mo eo e , CPUs and GPUs au oma i-
cally con ol pipeline dependencies o hide memo y access la ency, while o
DLAs his has o be explici ly implemen ed by he de elope . All hese ac o s
ende he c ea ion o a mul i- a ge ool ex emely complica ed.
2.1.1 Rela ed Wo k
Al hough DL models ha e seen an inc edible ise in mul iple domains and
applica ions, he same canno be said abou amewo ks ha allow o easily
op imize and deploy hem o a wide ange o ha dwa e a ge s.
One way o ep esen and pe o m high-le el op imiza ions is h ough
compu a ion g aph domain-speci ic languages (DSLs). Examples o hese
a e Tenso low’s XLA [2], DLVM [11] and Glow [12]. Al hough hese ep-
esen a ions a e well sui ed o high-le el op imiza ions, hey a e no ap o
low-le el ope a o op imiza ion. To do his, many amewo ks eso o low-
e ing p ocedu es o di ec ly gene a e low-le el LLVM o u ilize p op ie a y
endo lib a ies. Clea ly, hese me hods equi e conside able enginee ing
e o , conside ing ha hey ha e o be done o e e y combina ion o
ha dwa e backend and ope a o a ian .
An in e es ing solu ion has been p oposed in he Halide language and
compile [13] whe e compu ing and scheduling a e sepa a ed. This allows
he au ho s o ob ain conside able simpli ica ions in p og amming and majo
speed-ups compa ed o hand- uned C, in insics, and CUDA implemen a-
ions.
A di e en op imiza ion me hod is p oposed in Weld [14] whe e di e se
unc ions can submi hei compu a ions in a simple bu gene al in e media e

44 AIDGE: A F amewo k o Deep Neu al Ne wo k De elopmen , T aining
ep esen a ion ha cap u es hei da a-pa allel s uc u e. I hen op imizes
da a mo emen ac oss hese unc ions and emi s e icien code o di e se
ha dwa e.
DnnWea e [15] is a amewo k ha au oma ically gene a es a syn hesiz-
able accele a o o a gi en (DNN, FPGA) pai om a high-le el speci ica ion
in Ca e. I uses hand-op imized design empla e o i s ansla e a gi en high-
le el DNN speci ica ion o i s no el ISA ha ep esen s a mac o da a low
g aph o he DNN, hen i iles, schedules, and ba ches DNN ope a ions
o maximize da a euse and bes u ilize a ge FPGA’s memo y and o he
esou ces.
Finally, TVM [16] is a DNN compile ha has he capabili y o op i-
mizing code by sea ching and combining he bes enso ope a o s. This
compile p o ides end- o-end compila ion and op imiza ion s acks ha allow
he deploymen o DNNs on CPUs, bu also mobile GPUs, and FPGA-based
de ices.
2.2 Ou F amewo k O e iew
In his wo k we p esen Aidge, a new end- o-end amewo k o aining,
op imizing and compiling DNNs especially o low powe edge de ices.
This ool was designed o balance e icien compila ion, lexibili y, low
le el con ol and po abili y by combining insigh s om g aph analysis and
manipula ion wi h me hods om s uc u ed and unc ional p og amming
languages.
The pla o m in eg a es da abase cons uc ion, da a p e-p ocessing, ne -
wo k building o impo ing, manipula ion, op imiza ion, quan iza ion, es ing
and ha dwa e expo unc ionali ies (see Figu e 2.1). I is pa icula ly use ul
o DNN design and explo a ion, allowing simple and as p o o yping o
di e en DNNs.
Wi h his ool i is possible o de ine and ain mul iple opology a ia ions
o a ne wo k and o au oma ically compa e hei pe o mances (in e ms o
accu acy and compu a ional cos ).
One dis inc i e aspec o Aidge is ha i is based on he p inciple o
“modula i y”, i.e. he e is a “Co e Module” (see Figu e 2.2) ha can be
ex ended by so called “plugins” ha allow o add new unc ionali ies and
o mee needs no o eseen o implemen ed du ing he ini ial design o he
amewo k.
2.2 Ou F amewo k O e iew 45
Figu e 2.1 Schema ic ep esen a ion o he Aidge F amewo k wi h i s main componen s and
unc ionali ies
The Co e Module is de eloped en i ely in C++ (14) wi h bindings o
Py hon (>3.7), and includes a se o unc ions ha enable i o:
• C ea e a compu a ional g aph ep esen ing a DNN.
• Modi y he compu a ional g aph (e.g. by dele ing, eplacing o adding a
node).
• Do g aph que ying/ma ching o ind a speci ic sequence o ope a o s in
he compu a ional g aph.
• Ins an ia e ope a o s.
• Ins an ia e da a s uc u es, such as Tenso s.
• C ea e schedule s ( o now only sequen ial) o execu e he compu a ional
g aph
• Apply g aph op imiza ion, such as usion o ope a o s
Aidge sepa a es he concep s o desc ip ion and implemen a ion. Ope -
a o s and da a desc ip ions a e abs ac , while implemen a ions a e a ge -
speci ic.
Fo example, he so wa e implemen a ion o a con olu ion may di e
on a GPU o CPU, bu he de ini ion o he con olu ion i sel (i.e. i s inpu s
and pa ame e s) does no change. Mo eo e , he implemen a ion migh also
change acco ding o he u ilized lib a y, o example on an NVIDIA GPU,
p og amming can be done ei he ia CUDA o ia Tenso RT. Fo his eason
Aidge in oduces he no ion o “Backend” o de ine bo h he ha dwa e a ge
46 AIDGE: A F amewo k o Deep Neu al Ne wo k De elopmen , T aining
Figu e 2.2 Aidge is buil upon he concep o modula i y wi h a “Co e” componen and
se e al “plugins” ha comple e and ex end he amewo k.
and he lib a y used o he implemen a ion (wi h i s da a ype and a numbe
p ecision)
Plugins allow de elope s and use s o add o adap unc ionali ies o he
pla o m. Di e en kinds o plugins can be de eloped (in C++ o Py hon)
using he Aidge API, such as:
• “Recipe plugins”, which may allow o load and sa e he ne wo k
desc ip ion in a speci ic o ma , o i may consis in a se o op imize
algo i hms, o example o educe he model’s cos in e ms o mem-
o y and compu ing complexi y, o o inc ease i s obus ness agains
ex e nal/ad e sa ial a acks.
• “Da ase plugins”, which add he capabili y o load da a and labels om
a speci ic da ase .
• “Backend plugins”, which egis e o he Co e compiled ke nel lib a ies
(e.g. C++, CUDA, HLS) allowing i o execu e he compu a ional g aph.
2.2 Ou F amewo k O e iew 47
Figu e 2.3 The image shows he cons i uen pa s o an example Con olu ion ope a o .
• “Ope a o plugins”, which adds he abili y o de ine a new ope a o in
C++ which is no a ailable in he Co e.
• “Expo plugins”, which de ine a se o ules and me hods aimed a
adap ing he g aph o he a ge ed ha dwa e, and me hods o p oduce
sou ce code co esponding o he op imized g aph.
2.2.1 In e nal G aph Rep esen a ion
Aidge’s low-le el a chi ec u e is designed o allow he highes lexibili y
in DNN ep esen a ion and compu a ion, hus DNN models a e ep esen ed
using a di ec ional “compu a ional g aph”. This g aph is composed o a se
o nodes, ep esen ing ope a ions (Ope a o s), connec ed wi h di ec ed edges,
ep esen ing he low o da a.
Nodes in his compu a ional g aph a e de ined by h ee p ope ies: he
connec i i y, he ope a ion desc ip ion, and he implemen a ion.
1. Ope a ion desc ip ion: i desc ibes he ope a ion a node will do
(e.g. Con olu ion, ReLu, Da a P o ide , e c.) and i s a ibu es (see
Figu e 2.3). This desc ip ion is agnos ic o he implemen a ion. The
a ibu es a e he ollowing:
◦The sizes o he Ke nel, Dila ion (in case o con olu ions), S ide,
e c.
◦The numbe o inpu s and hei dimensions, da a ype and p ecision;
◦The numbe o ou pu s and hei dimensions, da a ype and
p ecision;
48 AIDGE: A F amewo k o Deep Neu al Ne wo k De elopmen , T aining
◦A e e ence o a o wa d (i.e. in e ence) unc ion implemen a ion;
◦A e e ence o a backwa d (i.e. ain) unc ion implemen a ion.
2. Connec i i y: i desc ibes which nodes (p ope Ope a o s o Da a
P o ide s) a e connec ed o a gi en node.
3. Implemen a ion: i poin s o he compu a ional unc ion/ke nel used by
he Ope a o o i s o wa d and backwa d ope a ions. The selec ion o
he igh implemen a ion is made ia a egis a sys em depending on
he ollowing a ibu es:
◦The Backend, de ined by bo h he ha dwa e a ge (e.g. CPU, GPU,
...) and a ailable lib a ies (e.g OpenCV)
◦The Da aType ( loa , in , . . . ) and P ecision (8bi s, 16bi s,
32bi s,...) o he inpu s and ou pu s
◦The Da aFo ma (NCHW, NHWC, . . .)
◦The Ke nel, he algo i hm chosen o pe o m he compu a ion
This lexible compu a ional g aph desc ip ion is pai ed wi h he abili y o
use a g ea a ie y o da a ep esen a ion (e.g. Tenso s, Spa se Tenso s, E en
Based s imuli, e c.).
2.2.2 Pla o m in e ope abili y
Thanks o PyBind11, he e is a seamless in e ope abili y wi h Numpy a ays,
achie ed by de ining a bu e _p o ocol in he binding o Aidge Tenso s. This
allows o use da a impo ed om o he amewo ks ha a e compa ible wi h
Numpy.
Aidge is in e ope able wi h PyTo ch and allows:
• C ea ing an Aidge Tenso om a PyTo ch Tenso
• Running an Aidge (sub)g aph wi hin he PyTo ch en i onmen .
• Running an Aidge compu a ional g aph wi hin he PyTo ch en i on-
men .
Aidge allows in e ope abili y wi h Ke as by c ea ing a w appe om a
Ke as Model h ough a con e sion s ep ia an ONNX ile.
Simila ly o PyTo ch, Aidge can con e Ke as enso s by using he
Numpy in e ope abili y.
2.2.3 G aph Regula Exp ession (G aphRegex)
The p oposed Aidge’s in e nal g aph ep esen a ion is a powe ul ool ha
combines ca e ully chosen abs ac ion le els. The s a egy is o adap he
in e nal ep esen a ion o na ow he gap be ween a neu al ne wo k and

2.2 Ou F amewo k O e iew 49
ha dwa e de ices. Aidge p oposes an inno a i e way o acili a e he manip-
ula ion o he in e nal g aph ep esen a ion: he G aph Regula Exp ession o
G aphRegex
The G aph Regula Exp ession combines wo main inno a ions:
1. A desc ip ion o g aph pa e ns. Taking inspi a ion om egula exp es-
sions om he o mal language heo y, we in oduce a new language o
desc ibe a se o g aphs s a ing om a sequence o cha ac e s.
2. G aph ma ching. Aidge p o ides a unc ion ha allows o ex ac a
subse o he g aph ma ching he p o ided G aphRegex desc ip ion.
G aph Regula Exp ession is complemen a y o o he g aph ans o -
ma ion me hods such as adding and emo ing nodes o en i e pa s o he
g aph.
Wi h G aphRegex i is possible o wo k on wo dis inc le els in a g aph:
1. A a condi ional le el, which co esponds o checking he p esence o a
node in a de ined dic iona y.
2. A a opological le el, which allows o desc ibe he in e connec ions
be ween symbols.
The opological desc ip ion, can be compa ed o classical egula exp es-
sions, as i is a o m o symbol sequence exp ession, bu ex ended o he
de ini ion o g aphs.
A a p ac ical le el, his ma ching me hod can be subdi ided in o wo
dis inc s ages. Fi s , we desc ibe he desi ed pa e n wi h he syn ax o egula
exp essions, hen we sea ch o ha pa e n inside he g aphs. These wo s eps
oge he o m he G aphRegex Que y.
A e ex ac ing all he subg aphs co esponding o a G aphRegex Que y,
i is possible o use an in e sec ion esolu ion algo i hm o ob ain in e sec ion-
ee solu ions. Howe e , i is impo an o no e ha hese algo i hms can ha e
a high complexi y, which can make hei execu ion ime-consuming.
2.2.4 Ne wo k op imiza ion
We can de ine wo ca ego ies o op imiza ions: opological ones, which
change he s uc u e o he compu a ion g aph, and pa ame ical ones, which
change he pa ame e s o he nodes.
An example o opological modi ica ion is Tiling. This me hod spli s
con olu ions in se e al ones ( o example in 4 con olu ions, as show in
Figu e 2.4). All o hem a e compu ed independen ly and conca ena ed a
he end. This manipula ion is ma hema ically exac (lossless).
50 AIDGE: A F amewo k o Deep Neu al Ne wo k De elopmen , T aining
He e is he p ac ical implemen a ion:
Figu e 2.4 Example o ope a o iling/spli ing: a Con + Relu subg aph is spli in o a Slice
+ 4 Con + 4 Relu + Conca .
One o he key di e en ia o s compa ed o o he amewo ks such as
LLVM, is ha Aidge applies di ec ly g aph modi ica ions, which allows o
make global opological changes as opposed o only ocus on local ones.
On he o he hand, an example o pa ame ical op imiza ion is quan iza-
ion a e aining (PTQ) o du ing aining (QAT). This is a well-es ablished
me hod o educing memo y usage and in mos cases, accele a ing he
2.2 Ou F amewo k O e iew 51
in e ence. PTQ is e y use ul when one does no ha e he ime o he
possibili y o e- un he aining and does no need o quan ize o mo e han
o 8-bi s. I ewe bi s a e necessa y, s a e-o - he-a QAT me hods gi e e y
good esul s. These and o he echniques (e.g. LSQ and F acBi QAT) a e
cu en ly being inalized in Aidge.
2.2.5 Expo phase
One o he aims o Aidge is o p oduce an in e p e able, explainable and
audi able ou pu . To do his Aidge p oduces/expo s sou ce code iles and
a numbe o ela ed esou ce iles ha o m a comple e package.
In Figu e 2.5, which summa izes he expo s a egy, i is possible o see
wo phases: Expo Mapping and Expo Implemen a ion.
The i s objec i e o he Expo Mapping phase is o modi y he com-
pu a ional g aph o i he a ge ha dwa e by using se e al op imiza ion
echniques (e.g. ha dwa e mapping op imiza ion o g aph ans o ma ion).
The second objec i e is he gene a ion o he g aph scheduling con-
s ained by he a chi ec u e ules o he a ge and addi ional p ojec ules
imposed by he de elope o he use (e.g. he a ailable memo y, he a ailable
compu e esou ces o he ime alloca ed o he execu ion).
Taking in o accoun he a chi ec u e ules and he p ojec cons ain s,
he schedule will gene a e a sequen ial lis o nodes om he op imized
g aph ha will de e mine how he o wa d p ocess (i.e. he in e ence) o he
expo ed DNN will un on he a ge .
Figu e 2.5 Schema ic ep esen a ion o Aidge’s expo p ocedu e.
52 AIDGE: A F amewo k o Deep Neu al Ne wo k De elopmen , T aining
The Expo Implemen a ion phase aims a p oducing a sou ce code o
he ha dwa e- uned g aph e u ned by he schedule . The ypical s eps o
gene a ing sou ce code a e he ollowing:
1. Design and expo he compu a ion ke nels.
2. Expo he a ibu es o he nodes.
3. Expo he pa ame e s o he nodes.
Each node o he g aph mus ha e an implemen a ion o i s o wa d
me hod in o de o use i in he expo . Since only he ha dwa e de elope s
eally know he cha ac e is ics and capabili ies o hei de ices, i is hei
du y o p o ide he implemen a ions o he compu a ion ke nels. These may
be implemen ed as a ke nel lib a y, which is a collec ion o op imized unc-
ions de eloped by expe p og amme s a ge ing he a chi ec u e (compu ing
unc ions, DMA p og amming, e c...).
Toge he wi h he ke nels, Aidge gene a es he con igu a ion and pa ame-
e iles, and also he iles ha con ain he sou ce code o he o wa d unc ion
o he ha dwa e-adap ed g aph.
The de elope has also he possibili y o add iles o gene a e a whole
So wa e Toolki ha will p o ide unc ionali ies such as:
• Compila ion o p ojec iles o compile he expo
• Files o un a whole applica ion o he expo
• Se o uni a y es s ( o es he ke nels on boa d, ...)
• Inpu da a o es s
• Thi d pa y lib a ies o use boa d unc ions
• Resou ces o check o he cons ain s like secu i y ules o obus ness
di ec i es
• Memo y map iles indica ing in o ma ion abou he s a ic alloca ion o
he esou ces used by he
2.3 Conclusion and u u e wo k
In his a icle we p oposed Aidge, a amewo k ha allows end- o-end
manipula ion, op imiza ion and compila ion o DNN a chi ec u es and hei
deploymen o a as spec um o ha dwa e de ices anging CPUs o GPUs,
MCUs, DSPs, FPGAs and neu omo phic a chi ec u es. Ano he aim o his
amewo k is o de elop and p o ide eusable ha dwa e building blocks and
me hodologies ha a e ans e sal o all ypes o a chi ec u es.
3.4 In e connec -based da a low a chi ec u e 59
which is compu ed using he equa ion below:
o map_ ows = (((i maps_ ows −( il e _ ows +padding_s a
+padding_end))/s ide) + 1)
3.4.1 NGC: Neu al Global Con olle
The NGC is a i e-s age Fini e-S a e Machine (FSM) shown in Figu e 3.1 (b).
Following is he desc ip ion o each s age:
1. IDLE ep esen s he idle s a e o he NGC.
2. LOAD_con ig loads and decodes he con igu a ion line o he cu en
laye , including he size and numbe o il e s and i maps o he cu en
laye .
3. LOAD_ il e s a s sending he il e da a (i.e., payload) and he con ol
wo d om he Inpu GBs (IGBs) o he connec ed ou e s. Depending on
he opcode, ou e s ei he unicas , mul icas , o b oadcas he incoming
payload da a o he AINoC.
4. LOAD_i maps s a s sending he i maps da a and he con ol wo d om
he IGBs o he connec ed ou e . Depending on he opcode, ou e s
ei he unicas , mul icas , o b oadcas he incoming payload da a o
he AINoC. LOAD_ il e and LOAD_i maps s a es can be in e changed
o p o ide some lexibili y in he da a loading o de in he p oposed
a chi ec u e.
5. COMPUTE s a s he MAC ope a ions in he NPEs. Once a Pa ial Sum
(PSum) is compu ed in he bo om ow NPEs, he esul s a e sen o hei
espec i e no h NPEs along wi h hei con ol wo ds. The NPEs in he
uppe ow hen add he incoming esul s wi h hei locally compu ed
PSum and send he compu ed esul o hei no h NPEs. This chain o
ope a ions is execu ed un il i eaches he op NPE ow, whe e he inal
PSum is s o ed in he Ou pu GBs (OGBs) o be used in he nex laye .
3.4.2 NPE: Neu al P ocessing Elemen
The NPE is a simple ope a o con olled by he FSM o he NGC, as shown in
Figu e 3.1 (c). I emains in he idle s a e un il he NGC igge s he execu ion.
I p oceeds in his way acco ding o he con ol signals sen by he NGC:
•When he load_ il e signal is ecei ed, i s o es he incoming payload
in o he il e Regis e File (RF). Once all he NPEs ha e ecei ed hei

60 A scalable and lexible in e connec -based da a low a chi ec u e o Edge
co esponding il e da a, he op igh NPE in he NPE a ay sends a
signal o NGC o jump o he nex s a e.
•When he load_i map signal is ecei ed, i s o es he incoming payload
in o he i maps RF. Once all he NPEs ha e ecei ed hei co esponding
i maps da a, he op igh NPE in he NPE a ay sends a signal o NGC
o jump o he nex s a e.
When he s a _compu e signal is ecei ed, i begins he MAC ope a ion
on he il e and i maps da a and gene a es PSum. Then, i ei he (i) sends he
PSum o he no h NPE (i bo om ow NPE) o (ii) adds he incoming PSum
om he sou h NPE wi h local PSum and sends he esul o he no h NPE o
OGB (i op ow NPE). In his phase, communica ion-compu a ion o e lap is
also pe o med by he NPEs, which ecei ed hei equi ed da a.
A he end o he compu a ion, an end_compu e signal will be sen by he
las NPE o he a ay o in o m he NGC o he end o he execu ion, in o de
o mo e on o he nex execu ion and he loading o new da a.
Figu e 3.1 (a) The p oposed in e connec -based da a low a chi ec u e sub-sys em, (b) Neu-
al Global Con olle (NGC), (c) Neu al P ocessing Elemen (NPE), (d) Rou e in A i icial
In elligence Ne wo k on Chip (AINoC).
3.4 In e connec -based da a low a chi ec u e 61
3.4.3 AINoC: A i icial In elligence Ne wo k-on-Chip
The AINoC [9] consis s o ou e s op imised o pa allel da a low p ocessing
wi h minimal da a ans e cos o achie e ene gy-e icien CNN p ocessing
wi hou comp omising accu acy and applica ion pe o mance.
As shown in Figu e 3.1 (d), he ou ing de ice is composed o se e al
pa allel ou ing pa hs, each including a bu e , a communica ion con olle ,
an a bi e , and a swi ch. All hese pa hs a e designed o gua an ee a la ge
bandwid h and lexible communica ion. Indeed, h ough se e al bu e ing
modules, e.g. Fi s -In-Fi s -Ou (FIFO), di e en communica ion eques s
ecei ed in pa allel can be s o ed wi hou any loss. These eques s a e
hen p ocessed simul aneously in se e al con ol modules. These modules
ensu e a de e minis ic con ol o he da a ans e acco ding o a s a ic
X-Y (X-di ec ion p io i y) ou ing algo i hm and managemen o di e en
communica ions (unicas , mul icas , and b oadcas ). Pa allel a bi a ion o
he p ocessing o de o incoming da a packe s acco ding o he Round-
Robin A bi a ion (RRA) [5] based on scheduled access allows o be e
collision managemen , i.e., a eques ha has jus been g an ed, will ha e he
lowes p io i y on he nex a bi a ion cycle. Pa allel swi ching comes nex
o simul aneously ou e da a o he igh ou pu s acco ding o he Wo mhole
swi ching [11], i.e. he connec ion be ween one o he inpu s and one o he
ou pu s o a ou e is main ained un il all he elemen a y da a o a message
packe a e sen and his in a simul aneous way h ough he di e en swi ching
modules.
The da a packe o ma is shown in Figu e 3.2. A da a message consis s
o wo packe s: a con ol packe ollowed by a da a packe . A packe is
composed o a heade ( li code) and a payload. In he con ol packe , he
payload is a des ina ion o sou ce add ess, while in he da a packe , he
payload is a se o da a li s. The packe size is 32-bi . Howe e , he size
o he heade and he payload a e a iable. I depends on he size o he
in e connec ion ne wo k, as he numbe o ou ing de ices inc eases, mo e
bi s a e needed o encode he add esses o he ecei e s o sende s. Simila ly,
he li size and numbe a y wi h he size o he payloads ( il e weigh s,
ac i a ion inpu s, o PSums) o be passed h ough he ne wo k. The alue
o he heade de e mines he communica ion o be p o ided by he ou e .
The e a e h ee possible ypes o communica ion in e -PEs: unicas , mul icas
(ho izon al, e ical, and diagonal), and b oadcas . Fo memo y access, he
62 A scalable and lexible in e connec -based da a low a chi ec u e o Edge
Figu e 3.2 Packe o ma
eading om he IGB is a mul icas communica ion; howe e , he w i ing
o he OGB is a communica ion ype ha p ocesses a di ec pa allel unicas
om he i s NPEs ows, and he OGB. The ou ing de ice i s ecei es
he con ol packe con aining he ype o communica ion and he sou ce o
des ina ion add ess. The ou ing de ice decodes his con ol packe and hen
alloca es he communica ion pa h o ansmi he da a packe ha a i es a
he cycle ollowing he con ol packe . Once he da a li s a e ansmi ed, he
alloca ed pa h will be eleased o u he ans e s.
3.5 Execu ion Model 63
3.4.4 Global Bu e s
GBs a e dual-po Random Access Memo y (DPRAM) ha a e used o s o e
he inpu da a i.e., il e and i maps o ou pu da a i.e., PSum om op ow
NPEs. The size o each GB ype is de e mined acco ding o he da a size
equi emen o each laye , such ha he o e head due o GB is minimised.
3.5 Execu ion Model
The da a mo emen and execu ion pa e n in he p oposed a chi ec u e a e
p esen ed in his sec ion. Once he da a is eady in IGB, he execu ion in he
p oposed a chi ec u e can be di ided in o h ee phases, i.e., (1) load i map
da a in o hei espec i e NPEs, (2) load il e da a in o hei espec i e NPEs,
and (3) pe o m execu ion on he a ailable da a in each NPE. These s eps a e
explained below:
1. Load i map da a: In his phase, i map da a a e loaded in o hei espec-
i e NPEs. Da a om IGBs a e diagonally loaded in o he NPEs, which
ha e connec ions wi h hem h ough a single ou e , and hen da a euse
is pe o med by mo ing he da a diagonally o he a ge NPEs.
2. Load il e da a: In his phase, il e da a a e loaded in o hei espec i e
PEs. Da a om IGBs a e ho izon ally loaded in o he NPEs, which ha e
connec ions wi h hem h ough a single ou e , and hen da a euse is
pe o med by mo ing he da a ho izon ally o hei espec i e NPEs.
Du ing his phase, he o e lap be ween communica ion and compu a ion
is also pe o med. The NPEs, which ecei e he equi ed da a o compu e
he pa ial sum, begin he compu a ion phase. Pa icula ly, he i s
column o he NPE a ay ge s all he equi ed da a and jumps om
he communica ion (i.e., da a ecei ing) phase o he compu a ion phase
while o he columns s ill wai o inpu da a.
3. Execu e MAC ope a ion: When an NPE ecei es all equi ed da a,
i jumps om he communica ion phase o he compu a ion phase.
Each column is locally synch onised, whe e he bo om NPE sends
he compu ed PSum o he no h NPE. Each NPE (excep he bo om
NPE) adds he PSum ecei ed om hei sou h NPE wi h he locally
compu ed PSum be o e sending he esul o hei no h NPE. This
chain o ecei ing, adding, and sending da a is pe o med un il he da a
eaches he op NPE, whe e he compu ed esul is s o ed back in o he
OGB. NPE a ay is execu ing in he Globally Asynch onous Locally
64 A scalable and lexible in e connec -based da a low a chi ec u e o Edge
Synch onous (GALS) pa e n o enable o e lap be ween communica ion
and compu a ion in he p oposed a chi ec u e.
3.6 Expe imen s and Resul s
3.6.1 E alua ion Me hodology
In his wo k, di e en CNN algo i hms om s a e-o - he-a we e used as
case s udies. They ha e di e en sizes and include di e en ypes o laye s
and shapes. LeNe 5 [18] and MobileNe V2 [12] we e chosen o ha e a
collec ion o da a esul ing om a ange o small o la ge CNN and using a se
o laye s including classical 2D con olu ion (CONV2D) and ully connec ed
laye s (FC) bu also poin -wise (PW) and dep h-wise (DW) con olu ion
laye s in MobileNe V2. Table 3.1 de ails he cha ac e is ics o all hese
CNN algo i hms, including he ypes o laye s hey ha e and he numbe o
each laye ype. The alues in he p oposed a chi ec u e con igu a ion a e
ob ained by ollowing he calcula ion ule p esen ed in sec ion 1.4. In ou
expe imen al s udy, we chose o es he key con olu ion laye s ha emphasise
di e en il e sizes and i maps and he ully connec ed laye s ha equi e a
linea spa ial ep esen a ion o he p oposed a chi ec u e. We also no e ha
a con igu a ion o he p oposed a chi ec u e mus be gene a ed o each
Table 3.1 CNN Laye s ype
CNNs Laye
Type i map size il e shape
Con ig. o
p oposed
a chi ec u e
LeNe 5
con _1 1x32x32 1x5x5 5x28
con _2 6x14x14 6x5x5 5x10
con _3 16x5x5 16x1x1 1x5
c_1 1x1x120 1x120x84 1x84
c_2 1x1x84 1x84x10 1x10
MobileNe V2
con _1 1x128x128 8x[3x3x3] 3x126
con _2 8x64x64 8x3x3 3x62
con _3 24x64x64 24x3x3 3x62
con _4 36x32x32 36x3x3 3x30
con _5 48x16x16 48x3x3 3x14
con _6 96x8x8 96x3x3 3x6
con _7 144x8x8 144x3x3 3x6
con _8 240x4x4 240x3x3 3x2
con _9 80x4x4 256x1x1 1x4

3.6 Expe imen s and Resul s 65
e alua ed laye o espec he RS da a low execu ion mode (sec ion 1.3.2).
Howe e , he ow wid h o he FC laye (i.e., 1000) o MobileNe V2 is oo
big o he p oposed a chi ec u e, due o he limi ed space allo ed o s o e he
alue o he numbe o channels in he con igu a ion wo d, so his laye has
been excluded om ou expe imen s.
3.6.2 FPGA Implemen a ion Resul s
The e alua ion pla o m used o all es s is he Ve sal ACAP VCK190
ki [20] ea u ing an “XCVC1902-2VSVA2197” FPGA pa i ion con aining
899840 p og ammable LUTs, 899840 Flip-Flops, 1968 DSP58, and 158Mb
o URAM and BRAM. The so wa e ools used o implemen and es
di e en con igu a ions o he p oposed a chi ec u e a e:
• Ques aSim o Ques a Ad anced Simula o ( e sion 2021.4) om Men-
o G aphics is p o ided o simula e and es he p og amming and
debugging o FPGA chips.
• Vi ado Design Sui e ( e sion 2021.2) is a so wa e sui e p oduced by
Xilinx o syn hesise and analyse ha dwa e desc ip ion language (HDL)
designs.
3.6.2.1 A ea
The di e en con igu a ions o he p oposed a chi ec u e include ou main
modules: he NGC, dis ibu ed memo ies (IGB & OGB), a gi en numbe o
NPEs, and ou e s ha a e di ec ly connec ed o he NPEs. All con igu a-
ions o he p oposed a chi ec u e a e designed wi h he VHDL desc ip ion
language o be apidly implemen ed on FPGA. The implemen a ion esul s
es ima e he equency o he p oposed a chi ec u e, which is a ound 125
MHz. This equency depends on he equency o he longes c i ical pa h
in he con igu a ion. A good place and ou e o he modules o he p oposed
a chi ec u e is necessa y o educe he leng h o he c i ical pa h and accele -
a e he p opaga ion o he signals. The syn hesis esul s de ine he occupied
a ea (logic elemen s, memo y, Digi al Signal P ocessing (DSP) blocks,
e c.) and he ha dwa e esou ces consump ion o he p oposed a chi ec u e
acco ding o he di e en con igu a ions de ined in Table 3.1.
The syn hesis esul s o he di e en modules cons i u ing he p oposed
a chi ec u e a e gi en in Table 3.2. Due o he simple s uc u e o he di e en
modules, he consump ion o logic and memo y esou ces emains low. This
allows gene a ing a con igu a ion o he p oposed a chi ec u e wi h a la ge
66 A scalable and lexible in e connec -based da a low a chi ec u e o Edge
g id o compu ing elemen s o p ocess la ge con olu ion laye s. Fo he GB
memo ies, we op ed o he use o Block Random Access Memo y (BRAM)
by o cing he syn hesis ool o choose hese memo y blocks ins ead o he
con igu able logic blocks (CLB). We also no ice ha he size o he ou e
is ela i ely la ge han he NPE. This can be explained by NPE p o iding
a simple con olu ion ope a ion. A he same ime, he ou e has mul iple
ou ing pa hs o p o ide pa allel mul icas and con ol o blocking a eas in
he communica ion ne wo k. These mul iple ou ing pa hs mainly accele a e
he da a ans e and educe he ene gy consump ion du ing he execu ion o
a con olu ion laye . I is hen a ade-o be ween a ea and pe o mance in
he p oposed a chi ec u e. A ea can be ea ed as a small o e head o ensu e
a balance in he choice o he a chi ec u e and he objec i es o be achie ed.
Table 3.2 B eakdown o Ve sal ACAP VCK190 FPGA esou ces used by he modules o
he p oposed a chi ec u e a e syn hesis
Module CLB BRAM A ea occupancy (%)
NGC 12.21 0 0.02
GB 0 0.5 0.05
Rou e 76.78 0 0.13
NPE 39.10 0 0.07
Figu e 3.3 Syn hesis esul s o di e en con igu a ions o he p oposed a chi ec u e
3.6 Expe imen s and Resul s 67
Figu e 3.3 shows he pe cen ages o FPGA esou ce u iliza ion when
execu ing he di e en laye s o Lene 5 and Mobilene V2 gi en in Table 3.1.
The p ocessing o each ype o laye equi es a pa icula con igu a ion o he
p oposed a chi ec u e. A con igu a ion o he p oposed a chi ec u e depends
on he numbe o il e ows, i map ows, and o map ows. We obse e a
co ela ion be ween he a ia ion in he size o he p oposed a chi ec u e
and he consump ion o he CLBs. The la ge he con igu a ion, he g ea e
he esou ce consump ion. The consump ion o he BRAM memo y blocks
depends on he size o he inpu image o he i maps. This means ha he
memo y size emains ixed o a ixed inpu image/ ea u e-map size, and he
size o he il e . Pa icula ly, o he con _1 o MobileNe V2, we no ice ha
he numbe o CLBs exceeds he maximum numbe o CLBs a ailable in
he FPGA a ge ed in hese expe imen s. This ep esen a ion shows ha he
p oposed a chi ec u e emains lexible o suppo all con olu ion laye sizes.
We jus need o aim o a p o o yping pla o m ha p o ides he necessa y
logic esou ces o mapping all laye s.
3.6.2.2 La ency
Figu e 3.4 shows he la ency pe o mance o he p oposed a chi ec u e o
each con olu ion ype. I can be obse ed ha he p oposed a chi ec u e is up
o 71.2×(con _1, Lene 5) as e w. . . single RISC-V CPU [2]. The o al
execu ion ime o each con olu ion ype o he p oposed a chi ec u e is
di ided in o i map loading ime, il e loading ime, da a euse, and o e lap
be ween communica ion-compu a ion including ime equi ed o he PSum
o a e se ac oss hei espec i e columns o s o e he compu ed o map. The
b eakdown o la ency epo s ha da a euse and o e lap be ween communi-
ca ion and compu a ion signi ican ly imp o e he o e all execu ion ime in he
p oposed a chi ec u e. Fo la ency compa ison o he p oposed a chi ec u e
wi h RISC-V CPU, he ime equi ed o access L2 o load da a in o IGBs is
also conside ed o a ai compa ison.
The o e all speedup o MobileNe V2 con olu ion laye s is up o 2.07×
w. . . Eye iss 2 [16, 17]. He e, Eye iss 2 execu es all laye s o MobileNe V2
while he p oposed a chi ec u e execu es con olu ion laye s (Table 3.1).
These esul s a e ob ained h ough RTL simula ions.
3.6.2.3 Ene gy consump ion
Di e en ha dwa e modules o he p oposed a chi ec u e in ol ed in di e en
execu ion phases o each con olu ion ype a e shown in Table 3.3. The
explana ion o each phase is as ollows: (1) Phase A ep esen s da a loading
68 A scalable and lexible in e connec -based da a low a chi ec u e o Edge
om all IGBs, (2) Phase B ep esen s da a euse, (3) Phase C ep esen s da a
loading om ow IGBs, and (4) Phase D ep esen s compu a ion in NPE
a ay. The esul s in his sec ion a e ob ained h ough ha dwa e emula ion.
Figu e 3.4 B eakdown o la ency (ns). Fo he p oposed a chi ec u e, he con olu ion laye
includes memo y accesses and compu a ions. WORK = This Wo k, RV32 = RISC-V CPU.
Table 3.3 Di e en execu ion phases in he p oposed a chi ec u e
Execu ion
Phase
NPE
a ay
AINoC IGB
ow
IGB
column
OGB NGC
A X X X X X
B X X X
C X X X X
D X X X X
4.2 Fede a ed Lea ning and Rela ed Wo k 75
In malwa e classi ica ion, e ec i e and e icien de ec ion models equi e
signi ican amoun s o aining da a. Al hough such amoun s o da a could
be collec ed and made a ailable h ough da a sha ing among o ganiza ions,
he e a e in ellec ual p ope y and p i acy conce ns and cons ain s ha o bid
o limi such exchanges. Fede a ed Lea ning (FL) is a p omising me hod
o building e ec i e and e icien de ec ion models in a dis ibu ed ashion,
using da a o di e en o ganiza ions, because i does no equi e he exchange
o sou ce da a [9].
FL is employed in wo main con igu a ions, c oss-de ice and c oss-silo.
In c oss-de ice con igu a ions a la ge numbe o membe s (clien s) wi h
limi ed da a samples each a e coo dina ing in de eloping a model. C oss-silo
con igu a ions ha e signi ican ly less membe s (clien s), each wi h a la ge
popula ion o da a samples. FL has been employed o malwa e de ec ion in
c oss-de ice en i onmen s, ocusing on IoT and And oid de ices [18, 19, 20].
Howe e , FL in c oss-silo con igu a ions has no been explo ed. Ou wo k
ocuses on c oss-silo FL, conside ing he equi emen s o applica ions and
se ices such as Edge o nea -Edge de ices and hei coo dina ion and
collabo a ion in hie a chies ha a e being de eloped in e na ionally.
In his pape , we p esen c oss-silo FL-based malwa e de ec ion, whe e
he de ec ion model is cons uc ed exploi ing ho izon al FL and employing a
NN. Conside ing an analysis app oach analogous o he one in Sisy os, we
measu e he pe o mance in malwa e de ec ion and e alua e i s dependence
on se e al pa ame e s, such as numbe o clien s, epe i ions o agg ega ion
s eps, da ase size and he pe cen age o common aining da a. Ou esul s
demons a e ha FL enables high accu acy in malwa e de ec ion o all
membe s o he ede a ion, i espec i e o he size o hei own aining
da ase . This demons a es an impo an ad an age o FL in malwa e de ec-
ion: membe s o he ede a ion wi h small aining da ase s would ne e
achie e independen ly he high accu acy which hey achie e h ough hei
pa icipa ion in he ede a ion.
The pape is o ganized as ollows. Sec ion 1.2 p esen s an o e iew o FL
and he cu en s a e-o - he-a in i s employmen in malwa e de ec ion. Sec-
ion 1.3 p esen s ou c oss-silo FL sys em a chi ec u e. Sec ion 1.4 p esen s
ou e alua ion esul s and demons a es he e ec i eness o ou app oach.
4.2 Fede a ed Lea ning and Rela ed Wo k
Fede a ed Lea ning is an eme ging machine lea ning app oach ha enables
he aining o AI models in a decen alized manne . Pa icipa ing clien s

76 Fede a ed Lea ning o Malwa e De ec ion in Edge de ices
collabo a e o ain ML algo i hms unde he coo dina ion o a cen al se e ,
wi hou sha ing hei p i a e da ase s wi h o he pa ies [10].
To ain a ede a ed model a cen al se e dis ibu es o he pa icipa ing
clien s an ini ial model and he aining pa ame e s. Then he ollowing s eps
ake place:
1. each pa icipan ains he ecei ed model using hei p i a e da ase ,
p oducing a local model and hen sends i o he se e ;
2. he se e agg ega es all local models in o a global one;
3. he global model is dis ibu ed o all clien s.
Figu e 4.2 illus a es his p ocess which can be epea ed o mul iple
lea ning s eps and s opped when a designa ed c i e ion is me .
FL is conside ed in wo di e en con igu a ions, in gene al [10]:
•C oss-de ice: clien s a e compu ing sys ems wi h limi ed compu ing
capabili ies, a ying de ice a ailabili y and small da ase s, e.g. IoT
de ices o sma phones.
•C oss-silo: clien s a e compu ing sys ems wi h high compu a ional
powe , high eliabili y and la ge da ase s (da a silos), such as cen alized
and en e p ise sys ems ( ypically 2-100).
In adi ional cen alized machine lea ning en i onmen s, a de ice o an
o ganiza ion mus ain a model on i s own sel -collec ed da a. In p ac ice,
hese de ices o o ganiza ions may no ha e access o su icien ly la ge da a
se s and he compu ing capabili ies necessa y o ain an e ec i e model.
Addi ionally, da a p i acy conce ns and in ellec ual p ope y igh s limi
collabo a ion be ween pa ies. FL add esses hese challenges by enabling
collabo a ion be ween mul iple pa ies o join ly ain e ec i e ML models
wi h la ge, di e se da ase s collec ed om all membe s o he ede a ion
[11]. As he p oduced models a e he only in o ma ion sha ed among ed-
e a ion membe s, local da a ne e lea e he pa icipa ing de ices enabling
da a owne s o keep hei da a p i a e. Impo an ly, FL scales well because
addi ional membe s can con ibu e o model aining wi hou any bu den o
o he membe s and wi h educed da a a ic among hem.
FL is employed in se e al ope a ions o cybe secu i y such as a ack
de ec ion, anomaly de ec ion, us managemen , au hen ica ion and o he IoT
ela ed asks [12, 13, 14, 15]. FL is also e ec i e in malicious URL and
Denial-o -Se ice (DoS) a acks de ec ion [16, 17].
In malwa e de ec ion, esea ch in FL employmen is mainly ocused on
c oss-de ice FL whe e ede a ion membe s (clien s) a e sma phones [18, 19]
o IoT de ices [20], while limi ed e o has been spen on malwa e de ec ion
4.3 A chi ec u e 77
5
Figu e 1.2 Fede a ed Lea ning con igu a ion
1.3 A chi ec u e
Ou p oposed a chi ec u e consis s o a FL c oss-silo con igu a ion as shown in
Figu e 1.2. Mul iple pa icipa ing membe s, indica ed as clien s, collabo a i ely
ain a global malwa e de ec ion model, and a se e is esponsible o all
communica ions as well as he agg ega ion o he global model. Each clien owns
and ains wi h some la ge amoun o local p i a e da a which i does no sha e wi h
he o he clien s no he se e .
In aining, each clien uses he same eed o wa d neu al ne wo k, i.e. he same
a chi ec u e and aining pa ame e s, an inc easingly popula me hod o malwa e
de ec ion [8][9]. Speci ically, we employ a neu al ne wo k deployed in [21]. We
adop i s a chi ec u e because i is e sa ile, widely adop ed and can be i ed in
de ices wi h limi ed compu ing powe such as nea -edge o edge de ices. The
model consis s o 3 linea laye s and a d opou laye . The ou pu laye pe o ms
bina y classi ica ion using a So Max laye , classi ying a sample as ei he malicious
o benign. We adap he model in [21] o accommoda e ou di e en da ase :
EMBER 2 [22] ins ead o EMBER 1. EMBER 2 is an upda e on he o iginal
EMBER da ase and con ains 2381 inpu ea u es ins ead o he 2351 used in [21].
The da ase is discussed in mo e de ail in Sec ion 1.4.1.
Ou de ec ion sys em, ope a es in wo modes: (a) aining, whe e mul iple clien s
a e using FL o collabo a i ely ain he de ec ion model and (b) de ec ion, whe e
each pa icipa ing clien uses he p oduced global model o de ec malwa e.
Figu e 4.2 Fede a ed Lea ning con igu a ion.
using c oss-silo FL con igu a ions. In ou wo k, we p opose a c oss-silo FL-
based malwa e de ec ion me hod, whe e ede a ion membe s a e di e en
Edge o nea -Edge de ices deployed o p o ide secu i y o di e en o gani-
za ional ne wo ks. The de ices collec la ge amoun s o da a and ha e highe
compu a ional capabili ies ela i ely o he de ices conside ed in c oss-de ice
con igu a ions.
S a e-o - he-a malwa e de ec ion app oaches employ neu al ne wo ks
a chi ec u es o ain models o sample classi ica ion as ei he malicious o
benign [8, 9]. In ou sys em, we employ a simila neu al ne wo k [21] ha can
e ec i ely lea n o de ec malwa e om he aining da a, while being able o
i in Edge o nea -Edge de ices. Some p elimina y esul s o his wo k we e
p esen ed in [26].
4.3 A chi ec u e
Ou p oposed a chi ec u e consis s o a FL c oss-silo con igu a ion as shown
in Figu e 4.3. Mul iple pa icipa ing membe s, indica ed as clien s, collabo-
a i ely ain a global malwa e de ec ion model, and a se e is esponsible
o all communica ions as well as he agg ega ion o he global model. Each
clien owns and ains wi h some la ge amoun o local p i a e da a which i
does no sha e wi h he o he clien s no he se e .
78 Fede a ed Lea ning o Malwa e De ec ion in Edge de ices
In aining, each clien uses he same eed o wa d neu al ne wo k, i.e. he
same a chi ec u e and aining pa ame e s, an inc easingly popula me hod
o malwa e de ec ion [8, 9]. Speci ically, we employ a neu al ne wo k
deployed in [21]. We adop i s a chi ec u e because i is e sa ile, widely
adop ed and can be i ed in de ices wi h limi ed compu ing powe such as
nea -edge o edge de ices. The model consis s o 3 linea laye s and a d opou
laye . The ou pu laye pe o ms bina y classi ica ion using a So Max laye ,
classi ying a sample as ei he malicious o benign. We adap he model in [21]
o accommoda e ou di e en da ase : EMBER 2 [22] ins ead o EMBER
1. EMBER 2 is an upda e on he o iginal EMBER da ase and con ains
2381 inpu ea u es ins ead o he 2351 used in [21]. The da ase is discussed
in mo e de ail in Sec ion 1.4.1.
Ou de ec ion sys em, ope a es in wo modes: (a) aining, whe e mul iple
clien s a e using FL o collabo a i ely ain he de ec ion model and (b)
de ec ion, whe e each pa icipa ing clien uses he p oduced global model
o de ec malwa e.
In aining mode, he FL-based aining p ocess akes place in mul iple
s eps. In each s ep he ollowing p ocess occu s: (i) each clien ains a local
model using i s own p i a e da a, (ii) each clien sends he p oduced local
model o he se e , (iii) he se e agg ega es all local models, p oducing a
global model and (i ) he se e dis ibu es he global model o all clien s.
Then, he clien s can use he global model o measu e he model’s pe o -
mance agains hei p i a e da ase s. The p ocess can be epea ed o mul iple
s eps o imp o e he model’s pe o mance u he , un il a sa is ac o y model
is achie ed, conside ing he ime and p ocessing cons ain s o he clien s o
un il no u he accu acy imp o emen is achie ed.
A e aining is comple e, in de ec ion mode, all pa icipa ing clien s
ha e ecei ed a copy o he inal global model om he se e . Each clien
can use his model o de ec malwa e in hei own sys ems and ne wo ks,
independen ly om all o he clien s. Finally, he clien s can e u n o aining
mode o e esh and e ain hei model wi h new da a.
4.4 Expe imen s
We e alua e he pe o mance o c oss-silo FL measu ing he malwa e de ec-
ion accu acy on a benchma k da ase con aining ea u es om malwa e and
benign iles. To u he explo e he e ec i eness o FL, we conside mul iple
FL aining se ups measu ing how he de ec ion a e is a ec ed by he numbe
o lea ning s eps, he numbe o pa icipa ing clien s and he commonali y in
4.4 Expe imen s 79
he pa icipa ing clien s’ da ase s. Fu he mo e, o demons a e he bene i s
o FL o he pa icipan s, we also ain a cen alized model o he same
a chi ec u e and pa ame e s and e alua e i s pe o mance di e en aining
da ase sizes. To conduc he expe imen s, we employ lowe [23], a pop-
ula FL amewo k, o aining he ede a ed models, in conjunc ion wi h
Py o ch [24].
4.4.1 Da ase
In all ou expe imen s we use he EMBER 2 da ase [22], a publicly a ailable
benchma k da ase , which con ains ea u es ex ac ed om bo h malwa e
and benign samples using s a ic analysis. We use EMBER because he e
a e no widely a ailable s anda d da ase s; his is a well-known p oblem in
cybe secu i y esea ch. Al hough i does no dis ibu e he sample bina y iles,
due o p i acy conce ns, EMBER is common choice in malwa e de ec ion and
analysis because o h ee ac o s: (i) i s su icien size, (ii) i s se o ea u es
and (iii) i con ains ea u es o malwa e and benign samples.
EMBER 2 con ains 2381 ea u es pe sample, ex ac ed using s a ic
analysis om 1.1 million Windows Po able Execu ables (PE). Mo e speci -
ically o aining, he da ase con ains 600.000 samples labelled as ei he
benign o malicious (300.000 benign and 300.000 malicious), and 300.000
unlabelled samples. The da ase also con ains 200.000 samples labelled as
ei he benign o malicious (100.000 benign and 100.000 malicious) o be
used as a dedica ed benchma k es ing se . In ou expe imen s we only use
labelled samples, 600.000 o aining he neu al ne wo ks and 200.000 o
es ing he p oduced models.
4.4.2 E alua ion esul s
In he i s expe imen we conside a FL se up whe e 2 pa icipa ing clien s
ain a common model o 10 lea ning s eps using he en i e da ase . Thus,
each clien ains each local model wi h 300.000 da a samples. We measu e
he accu acy, p ecision. ecall and 1 sco e o he model on he es se o each
s ep. Table 4.1 summa izes he esul s o he expe imen . Fo compa ison we
also ain a cen alized model on he ull da ase , 600.000 da a samples and
we measu e an accu acy o 0,9338 on he es se .
Figu e 4.3 plo s he accu acy o he FL model o mul iple lea ning s eps.
The o ange line deno es he accu acy o he cen alized model as e e ence
ained wi h he en i e EMBER da ase o 600K samples. The esul s show
ha he accu acy o FL inc eases wi h he inc easing numbe o aining loops
80 Fede a ed Lea ning o Malwa e De ec ion in Edge de ices
Table 4.1 Accu acy on es se o FL model wi h 2 clien s o mul iple lea ning s eps.
Numbe o s eps Accu acy P ecision Recall F1 Sco e
1 0,8711 0,8419 0,9137 0,8763
2 0,9111 0,9012 0,9236 0,9123
3 0,9176 0,9066 0,9312 0,9187
4 0,9194 0,9091 0,9321 0,9205
5 0,9211 0,911 0,9335 0,9221
6 0,9232 0,9123 0,9365 0,9242
7 0,9242 0,9143 0,9361 0,9251
8 0,9261 0,9183 0,9355 0,9268
9 0,9247 0,9143 0,9373 0,9257
10 0,9251 0,9157 0,9365 0,926
8
Figu e 1.3 Fede a ed Lea ning model pe o mance o a iable aining loops.
Nex , we e alua e whe he he numbe o pa icipan s in luences he accu acy o he
p oduced model. We use he en i e y o he EMBER da ase (600.000 samples) and
keep he same o al da ase size o all expe imen s, dis ibu ing i equally among
he pa icipa ing clien s in e e y case (i.e. o 2 pa icipa ing clien s, each clien s
holds 300.000 samples and o 5 pa icipa ing clien s, each clien s holds 120.000
samples). We un expe imen s o 2,5,10,15 and 20 pa icipan s and measu e he
accu acy o he p oduced models on he es se . Table 1.2 summa izes he esul s
o he expe imen s o 2 lea ning s eps.
Table 1. 2 Accu acy on es se o FL model o di e en numbe o clien s o 2 lea ning
s eps
Numbe o
clien s
Accu acy
P ecision
Recall
F1 Sco e
2
0,9103
0,9015
0,9212
0,9112
5
0,9201
0,9156
0,9255
0,9205
10
0,9144
0,9091
0,921
0,915
15
0,9139
0,9089
0,92
0,9144
20
0,9175
0,913
0,9221
0,9175
0.85
0.87
0.89
0.91
0.93
0.95
0.97
1 2 3 4 5 6 7 8 9 10
Accu acy
Lea ning S eps
Accu acy on es se
Bes cen alized lea ning accu acy
Figu e 4.3 Fede a ed Lea ning model pe o mance o a iable aining loops.
and impo an ly eaches he pe o mance o he cen alized model and is on
pa wi h he esul s p esen ed in [21]. Addi ionally, we obse e ha we make
mos o he accu acy gains in he i s 2 FL aining s eps o his da ase .
Thus, in subsequen expe imen s we ain all FL models o 2 aining s eps.
Nex , we e alua e whe he he numbe o pa icipan s in luences he
accu acy o he p oduced model. We use he en i e y o he EMBER da ase
(600.000 samples) and keep he same o al da ase size o all expe imen s,
dis ibu ing i equally among he pa icipa ing clien s in e e y case (i.e. o 2

4.4 Expe imen s 81
pa icipa ing clien s, each clien s holds 300.000 samples and o 5 pa ic-
ipa ing clien s, each clien s holds 120.000 samples). We un expe imen s
o 2,5,10,15 and 20 pa icipan s and measu e he accu acy o he p oduced
models on he es se . Table 4.2 summa izes he esul s o he expe imen s
o 2 lea ning s eps.
Table 4.2 Accu acy on es se o FL model o di e en numbe o clien s o 2 lea ning
s eps.
Numbe o clien s Accu acy P ecision Recall F1 Sco e
2 0,9103 0,9015 0,9212 0,9112
5 0,9201 0,9156 0,9255 0,9205
10 0,9144 0,9091 0,921 0,915
15 0,9139 0,9089 0,92 0,9144
20 0,9175 0,913 0,9221 0,9175
Figu e 4.4 plo s he accu acy o he FL model o di e en numbe
o clien s. The o ange line deno es he accu acy o he cen alized model
as e e ence ained wi h he en i e EMBER da ase o 600K samples.
We obse e accu acy is e ec i ely independen o he numbe o clien s,
sugges ing ha he malwa e de ec ion sys em can scale o mo e and mo e
pa icipan s wi hou accu acy losses.
9
Figu e 1.4 plo s he accu acy o he FL model o di e en numbe o clien s. The
o ange line deno es he accu acy o he cen alized model as e e ence ained wi h
he en i e EMBER da ase o 600K samples. We obse e accu acy is e ec i ely
independen o he numbe o clien s, sugges ing ha he malwa e de ec ion sys em
can scale o mo e and mo e pa icipan s wi hou accu acy losses.
Figu e 1.4 Fede a ed Lea ning model accu acy o di e en numbe o clien s
Nex , we conside a case whe e di e en o ganiza ions o di e en de ices in he
same o ganiza ion ha pa icipa e in a FL se up, ha e common da a samples in hei
p i a e da a. As a acke s and malwa e au ho s use he same malwa e samples o
in ec mul iple a ge s and as o ganiza ions p ocess a la ge amoun o malwa e on
daily basis i seems a likely scena io ha pa icipan s will ha e some deg ee o
commonali y in hei p i a e da ase s. Thus, we e alua e whe he he p esence o
o e lapping samples in he pa icipan s’ aining se s in luences he accu acy o he
p oduced FL model. We conside di e en da ase o e lap pe cen ages be ween he
pa icipan s o 2 and 10 pa icipa ing clien s. Tables 1.3 and 1.4 p esen he esul s
o he expe imen s o 2 and 10 pa icipan s espec i ely o 2 FL aining s eps.
Table 1. 3 Accu acy on es se o FL models o di e en da ase o e laps o 2 clien s.
2 Clien s
0.85
0.87
0.89
0.91
0.93
0.95
0.97
2 5 10 15 20
Accu acy
Numbe o clien s
Accu acy on es se
Bes cen alized lea ning accu acy
Figu e 4.4 Fede a ed Lea ning model accu acy o di e en numbe o clien s.
82 Fede a ed Lea ning o Malwa e De ec ion in Edge de ices
Nex , we conside a case whe e di e en o ganiza ions o di e en
de ices in he same o ganiza ion ha pa icipa e in a FL se up, ha e common
da a samples in hei p i a e da a. As a acke s and malwa e au ho s use he
same malwa e samples o in ec mul iple a ge s and as o ganiza ions p ocess
a la ge amoun o malwa e on daily basis i seems a likely scena io ha pa ic-
ipan s will ha e some deg ee o commonali y in hei p i a e da ase s. Thus,
we e alua e whe he he p esence o o e lapping samples in he pa icipan s’
aining se s in luences he accu acy o he p oduced FL model. We conside
di e en da ase o e lap pe cen ages be ween he pa icipan s o 2 and 10
pa icipa ing clien s. Tables 4.3 and 4.4 p esen he esul s o he expe imen s
o 2 and 10 pa icipan s espec i ely o 2 FL aining s eps.
Table 4.3 Accu acy on es se o FL models o di e en da ase o e laps o 2 clien s.
2 Clien s
O e lap pe cen age Accu acy P ecision Recall F1 Sco e
0 0,9111 0,9012 0,9236 0,9123
5 0,9166 0,9139 0,92 0,9169
10 0,9045 0,9039 0,9054 0,9046
15 0,9129 0,9057 0,9218 0,9137
20 0,9114 0,902 0,9231 0,9124
25 0,918 0,9117 0,9256 0,9186
30 0,9125 0,9063 0,9201 0,9131
35 0,9124 0,9098 0,9157 0,9128
40 0,9173 0,9127 0,9228 0,9178
45 0,9319 0,9267 0,9378 0,9322
50 0,9262 0,9197 0,934 0,9268
Table 4.4 Accu acy on es se o FL models o di e en da ase o e laps o 10 clien s.
10 Clien s
O e lap pe cen age Accu acy P ecision Recall F1 Sco e
0 0,9144 0,9091 0,921 0,915
5 0,9162 0,9108 0,9229 0,9168
10 0,9154 0,9086 0,9238 0,9161
15 0,9167 0,9096 0,9255 0,9175
20 0,9171 0,9121 0,9233 0,9176
25 0,9169 0,909 0,9267 0,9177
30 0,9171 0,9114 0,9242 0,9177
35 0,9178 0,9138 0,9227 0,9182
40 0,9146 0,9059 0,9254 0,9155
45 0,9179 0,9093 0,9285 0,9188
50 0,9174 0,9114 0,9247 0,918
4.4 Expe imen s 83
11
da a e en in he ex eme case o a 50% o e lap, meaning ha he p oduced models
do no o e i on he common da a.
Figu e 1.5 Fede a ed model accu acy o di e en da ase o e laps.
Finally, o showcase he bene i s o FL o o ganiza ions, we conside a scena io
whe e a single o ganiza ion o de ice is no pa icipa ing in a FL se up bu ins ead
ains i s own cen alized model using i s own p i a e da a. We conside
o ganiza ions o di e en sizes ha ha e di e en aining da a a ailabili y. We
ain a cen alized model (no ede a ion p esen ) o he same a chi ec u e and
pa ame e s as in he p e ious FL se ups wi h di e en da ase sizes. Table 1.5
summa izes he esul s.
Table 1. 5 Accu acy on es se o cen alized models o di e en da ase sizes.
Numbe o
samples
Accu acy
P ecision
Recall
F1 Sco e
5000
0,8443
0,8299
0,8662
0,8477
10000
0,8482
0,8428
0,8828
0,8623
0.85
0.87
0.89
0.91
0.93
0.95
0.97
0 5 10 15 20 25 30 35 40 45 50
Accu acy
O e lap pe cen age
Accu acy on es se
Accu acy - 2 clien s
Accu acy - 10 clien s
Bes cen alized lea ning accu acy
Figu e 4.5 Fede a ed model accu acy o di e en da ase o e laps.
Figu e 4.5 plo s he accu acy on he es se o he FL models p oduced
as a unc ion o he o e lap (common subse ) o he clien s’ aining da a, i.e.
x=5 indica es 5% common da a in he clien da ase s. The blue and g een lines
depic he accu acy o he models ained by 2 and 10 clien s espec i ely. The
o ange line deno es he accu acy o he cen alized model as e e ence ained
wi h he en i e EMBER da ase o 600K samples. As we obse e in bo h
se ups, accu acy seems o be e ec i ely independen o he common samples
p esen in he pa icipan s’ p i a e da a e en in he ex eme case o a 50%
o e lap, meaning ha he p oduced models do no o e i on he common
da a.
Finally, o showcase he bene i s o FL o o ganiza ions, we conside a
scena io whe e a single o ganiza ion o de ice is no pa icipa ing in a FL
se up bu ins ead ains i s own cen alized model using i s own p i a e da a.
We conside o ganiza ions o di e en sizes ha ha e di e en aining da a
a ailabili y. We ain a cen alized model (no ede a ion p esen ) o he same
a chi ec u e and pa ame e s as in he p e ious FL se ups wi h di e en da ase
sizes. Table 4.5 summa izes he esul s.
84 Fede a ed Lea ning o Malwa e De ec ion in Edge de ices
Table 4.5 Accu acy on es se o cen alized models o di e en da ase sizes.
Numbe o samples Accu acy P ecision Recall F1 Sco e
5000 0,8443 0,8299 0,8662 0,8477
10000 0,8482 0,8428 0,8828 0,8623
50000 0,8949 0,8791 0,9156 0,897
100000 0,9126 0,904 0,9232 0,9135
600000 0,9338 0,9271 0,9417 0,9343
Figu e 4.6 plo s he accu acy o he cen alized (non- ede a ed) sys em
as a unc ion o he da ase size. The blue line deno es he bes FL accu-
acy we measu ed in ou expe imen s o e e ence. The esul s indica e
ha a da a se size o 600K is necessa y in he cen alized (non- ede a ed)
case o achie ing high accu acy ha eaches abo e 93% and ma ches he
accu acy o he FL sys em. This esul is he e e ence accu acy owa ds
which we e alua e he pe o mance o he ede a ed sys em cases. Impo -
an ly, when conside ing Figu e 4.3 as well, he plo demons a es he bene i
o FL o small o ganiza ions and nea -edge de ices wi h limi ed da a
a ailabili y ha do no ha e access o la ge da ase s o ain cen alized
models. We also no e ha e en ha e en o ganiza ions and de ices ha
12
50000
0,8949
0,8791
0,9156
0,897
100000
0,9126
»0,904
0,9232
0,9135
600000
0,9338
0,9271
0,9417
0,9343
Figu e 1.6 plo s he accu acy o he cen alized (non- ede a ed) sys em as a unc ion
o he da ase size. The blue line deno es he bes FL accu acy we measu ed in ou
expe imen s o e e ence. The esul s indica e ha a da a se size o 600K is
necessa y in he cen alized (non- ede a ed) case o achie ing high accu acy ha
eaches abo e 93% and ma ches he accu acy o he FL sys em. This esul is he
e e ence accu acy owa ds which we e alua e he pe o mance o he ede a ed
sys em cases. Impo an ly, when conside ing Figu e 1.3 as well, he plo
demons a es he bene i o FL o small o ganiza ions and nea -edge de ices wi h
limi ed da a a ailabili y ha do no ha e access o la ge da ase s o ain cen alized
models. We also no e ha e en ha e en o ganiza ions and de ices ha ha e access
o la ge da ase s can bene i om FL, as a model gene a ed wi h con ibu ions om
mul iple pa icipan s is ained on a po en ially mo e di e se da ase wi h malwa e
and benign samples coming om di e en ne wo ks.
Figu e 1.6 Cen alized lea ning model’s pe o mance o di e en da ase sizes.
0.8
0.82
0.84
0.86
0.88
0.9
0.92
0.94
0.96
5,000 10,000 50,000 100,000 600,000
Accu acy
Da ase size
Accu acy on es se
Bes ede a ed accu acy
Figu e 4.6 Cen alized lea ning model’s pe o mance o di e en da ase sizes.
5.1 In oduc ion and Backg ound 91
calls will be imp ac ically slow. As a esul , unning wi h a pa ame e
se x is ime consuming, and any sol e o uning p oblem should ind
op imal pa ame e s wi h he smalles numbe o i e a ions. An example o
op imiza ion p oblem is Linea P og amming (LP) whe e , g, and h a e linea
unc ion and de ine a polyhed on. Changing pa ame e x will mo e a linea
unc ion ac oss he easibili y se as show in Figu e 5.2.
5.1.4 S a ic and Dynamic Pa ame e s in ISP
Di e en algo i hms u ilized in ISP which ha e indi idual pa ame e s. Each
algo i hm ies o a enua e a i ac s om a speci ic sou ce:
a) S a ic Pa ame e s: Algo i hms esponsible o imp o ing a i ac s o igi-
na ed om came a senso ha e s a ic pa ame e s which should be uned
only once.
b) Dynamic Pa ame e s: Algo i hm esponsible o imp o ing a i ac s
due o ligh condi ion and en i onmen al phenomena ha e dynamic
pa ame e s which should be upda ed du ing un ime.
The s a ic pa ame e s which a e ela ed o came a senso cha ac e is ics
can be uned once o he speci ic came a. A e uning he ISP o he speci ic
came a, he pa ame e s can be ixed in con igu a ion ile o deploymen .
Tuning dynamic pa ame e s imp o es he image quali y in di e en en i-
onmen al condi ions. The dynamic pa ame e s should be upda ed du ing
un ime o gua an y bes image quali y pe o mance.
5.1.5 S a e o A
The uning p ocess is done manually by expe s. Each ISP algo i hm is
esponsible o educing speci ic a i ac in image and he expe can mea-
su e he a i ac in ensi y using a speci ic KPI. Then by changing he ISP
pa ame e s and y and e o , he expe can ind bes pa ame e s combina ion
o speci ic came a senso .
Fo uning dynamic pa ame e s, expe do he same p ocess, bu o
a ange o en i onmen al condi ions. Tha means, i s a se o inpu ISP
images (Baye Pa e n) a e cap u ed om senso in di e en en i onmen al
condi ion, hen expe should une he ISP o each one o he inpu images.
ISP has an algo i hm o ga he ing he s a is ical da a o Baye pa e n
image. Ha ing he s a ical da a o all images and co esponding op imal ISP
pa ame e s, one can make a “decision ee” which changes he pa ame e s on
ligh based on s a is ical da a p o ided by ISP.

92 Image Signal P ocesso (ISP) Tuning using Machine Lea ning (ML) me hods
5.2 Au oma ic ISP Tuning
The au oma ic uning p ocess should be done o bo h Dynamic and S a ic
pa ame e s.
5.2.1 KPIs o A i ac A enua ion
Fo measu ing ISP pe o mance, a ious KPIs should be de ined. Each KPI
measu es he in ensi y o a speci ic a i ac in image. I should be no ed ha
he e is no one o one ela ion be ween a i ac s and ISP algo i hms. In many
cases one KPI could be used o uning mul iple ISP algo i hms. The lis o
KPIs is [4]:
Table 5.1 KPIs o Measu ing Image A i ac s.
A i ac ISP block KPI
Noise Noise Reduc ion Block(s) PSNR
Loss o de ail Sha pness Co ec ion MTF50
Colo Inaccu acy Colo Co ec ion Ma ix (CCM) ∆E
Colo Cas ing Whi e Balancing ∆E
5.2.2 S a ic Pa ame e s
The se up is done once in lab and a came a is a ached o he cap u ing
de ice. The cap u ed Baye pa e n is ed o he ISP as inpu . The ISP
gene a es an image. Fo measu ing he pe o mance o speci ic ISP algo i hm
in a enua ing an a i ac , co esponding KPI in Table 1 is used. ISP une can
ack KPI alue o judging pe o mance esul o a se o pa ame e s. ISP
une changes he ISP pa ame e s in mul iple i e a ions and ies o op imize
pa ame e s based on KPI alue. The p ocess is i e a i e, and he i e a i e
p ocess is needed o be done only once o s a ic ISP pa ame e s as shown
in Figu e 5.3. The op imal alue ound by Tune will be s o ed as ixed
con igu a ion o un ime.
Figu e 5.3 Tuning ISP S a ic Pa ame e s.
5.2 Au oma ic ISP Tuning 93
5.2.3 Dynamic Pa ame e s and Run ime
Achie ing op imal pe o mance wi h dynamic pa ame e s is ha de . The
pa ame e s should be e- uned o adap en i onmen al e ec s such as ligh
condi ion, empe a u e, e c.
The e a e limi a ions in using same i e a i e app oach o uning s a ic
pa ame e s:
1. The i e a i e app oach makes i impossible o ha e op imal pa ame e s
pe ame o e en pe minu e.
2. Measu ing he KPI du ing un ime is challenging since he e is no
e e ence o he scene cap u ed by came a.
A p oposal solu ion is o use a machine lea ning model which can map
image s a is ical da a o op imal pa ame e s. All ISPs measu e image s a is-
ical da a and p o ide i pe ame. The da ase can be c ea ed by u ilizing
same uning p ocedu e men ioned o uning s a is ical pa ame e s in a loop
as demons a ed in Figu e 5.4.
E: All desi ed en i onmen al condi ions which ISP should ope a e in.
Fo e in E
{
1. Run une o op imizing ISP.
2. S o e Op imal ISP pa ame e s + ISP s a is ical
da a o he scene.
}
Figu e 5.4 Dynamic Pa ame e s Da a Gene a ion.
In heo y, a Neu al Ne wo k (NN) should be able o p edic op imal
alues o dynamic pa ame e s a e aining; howe e , an NN c ea es a high
compu a ion load du ing in e ence, so a mo e e icien solu ion is needed.
94 Image Signal P ocesso (ISP) Tuning using Machine Lea ning (ML) me hods
In his pape , a g adien boos model is p oposed o in e encing he
op imal pa ame e s. G adien boos ing models a e ainable decision ees.
Unlike NN which use Di ec ed acyclic g aphs (DAGs) as unde laying da a
s uc u e o aining and in e encing, g adien boos (GB) u ilizes ees which
a e simple da a s uc u es [5]. GB models a e as o in e ence and has simila
pe o mance as NNs o abula da a which exac ly ma ches he use case and
da ase we ha e o dynamic uning applica ion.
Figu e 5.5 S o ing Op imal Pa ame e s and ISP S a is ical Da a o T aining ML model.
5.2.4 Tes Se up
Tes ing he p oposed me hods o uning s a ic and dynamic pa ame e a e
done wi h Solec ix So ISP SXIVE1. The ISP uns on PC wi h dedica ed
g aphic ca d. The une uses 16 co es CPU o speed up he p ocess in lab o
inding s a ic pa ame e s in ISP.
Same une is u ilized o c ea e a da ase o ISP s a is ical da a and op imal
pa ame e s o en ligh ing colo empe a u e. The da ase is hen used o
aining a GB model named XGboos as un ime. The ained model hen
uns on single co e wi h minimum load on he same machine o upda e Whi e
Balancing pa ame e s.
5.2.5 Resul s
The demo so wa e (SW) is ins an ia ed wi h ini bu on. ISP wi h andom
pa ame e s is un and he ISP ou pu image is shown o he use . The use
can selec he bounda ies o he Colo Checke (CC) boa d inside he image
hen p ess “Tune”. The SW will c op he image o ind cc boa d and samples
1sxi e.com
5.2 Au oma ic ISP Tuning 95
pa ches om he boa d and shows he sampling a eas o he use (Anno a ed
CC). Then, uning p ocess begins. In Figu e 5.6 he p ocess o inding a
be e op imum poin is illus a ed as une p og ess. The measu ed ∆E o
all colo blocks in colo checke boa d is calcula ed and agg ega ed as Mean
Squa ed E o (MSE) o ∆E alues:
MSE(∆E) =
23
X
k=0
(∆Ek)2
24 .
The SW esul s in Figu e 5.6 can be in e p e ed as:
“ISP ou pu ” shows ISP ou pu image o he cu en i e a ion, and “Bes
Con ig” shows he bes esul ound by une un il he cu en i e a ion. When
une i e a es o e di e en con igu a ions, i gene a es a ious “ISP ou pu s”
Figu e 5.6 O line Tuning Resul s o Colo Co ec ion Ma ix Tuning.
96 Image Signal P ocesso (ISP) Tuning using Machine Lea ning (ML) me hods
and co esponding KPI alues; Based on obse ed KPI alue, une guesses a
be e pa ame e se o he nex i e a ion. As i e a ions go on, une can ind
be e pa ame e se s which esul s in be e KPI alues, so as i can be seen,
he “Bes Con ig” image is imp o ed when une p og esses.
Whi e balancing (WB) is he algo i hm chosen o dynamic uning. Same
une inds op imal pa ame e o a ious ligh condi ions and in ensi ies in
No WB
XGboos WB
Figu e 5.7 Run ime Resul o T ained XGboos WB.

Re e ences 97
lab in a loop as explained in Figu e 5.4. The gene a ed da ase maps image
his og am o op imal WB con igu a ion is used o ain a XGboos model.
The ISP has i s own Au oma ic Whi e Balancing (AWB), bu we u ned i
o o show he e ec i eness ou o XGboos WB. The esul s a e shown in
Figu e 5.7 wi hou and wi h whi e balancing unde blue ligh sou ce.
5.3 Conclusion
Tuning ISP was con en ionally a cumbe some, cos ly, and subop imal ask.
The i e a i e p ocess should ha e been done o combina ion o nume ous
ISPs and came as. In pu sue o a mo e au oma ed solu ion, he p oposed
me hod ies o dis inguish pa ame e s based on s a ic/dynamic na u e. The
pa ame e s which a e ela ed o speci ic came a, can be uned in lab, and
he op imal pa ame e s will be ixed as s a ic pa ame e s. Fo ISP algo i hms
which a enua es en i onmen al impac on image quali y, a un ime should
ine- une he ISP in he ield. The un ime algo i hm should be ligh enough
o un on a es ic ed HW p ocesso . Fo uning bo h s a ic and dynamic
pa ame e s, he pape p esen s a uning amewo k o au oma e he p ocess.
A une inds op imal pa ame e s o s a ic pa ame e s. A da a gene a ion
pipeline u ilizes same une in a loop o a ious en i onmen al condi ions.
The gene a ed da a maps he s a is ical da a p o ided by ISP o op imal
pa ame e s ound by une . In he nex s eps, he ained GB model based on
he gene a ed da ase is used as a ligh weigh ed un ime o uning dynamic
pa ame e s in changing en i onmen al condi ion.
Re e ences
[1] D. Molloy e al., “Impac o ISP Tuning on Objec De ec ion,” Jou nal o
Imaging, ol. 9, no. 12, pp. 260-260, No . 2023, doi: h ps://doi.o g/10.3
390/jimaging9120260.
[2] S. Boyd and L. Vandenbe ghe, “Con ex Op imiza ion,” Ma . 2004, doi:
h ps://doi.o g/10.1017/cbo9780511804441.
[3] h ps://en.wikipedia.o g/wiki/Linea _p og amming.
[4] h ps://www.ima es .com/suppo /docs/23-2/colo check/
[5] T. Chen and C. Gues in, “XGBoos : a Scalable T ee Boos ing Sys em,”
P oceedings o he 22nd ACM SIGKDD In e na ional Con e ence on
Knowledge Disco e y and Da a Mining - KDD ’16, pp. 785-794, 2016,
doi: h ps://doi.o g/10.1145/2939672.2939785.
6
Using Edge AI in IoT de ices o Sma
Ag icul u e: Au onomous Weeding
Ch is ian Ge main1,2, Ba na Ke esz es1,2, Ayme ic Deshayes1,
and Jean-Pie e Da Cos a1,2
1Uni e si y o Bo deaux, CNRS, F ance
2Bo deaux Sciences Ag o, F ance
Abs ac
This pape p esen s he e olu ion o a ision sys em dedica ed o au oma ic
weeding, ini ially implemen ed on a NVIDIA Je son Xa ie boa d. This
e olu ion aims o ake ad an age o a new compu ing boa d able o implemen
e icien a i icial in elligence o ien ed compu a ions, keeping a low powe
consump ion, and a low cos , de eloped in he ANDANTE p ojec . The
pape p esen s he au oma ic weeding ool, he exis ing ision sys em and
he weeding da a used o ain he sys em. I also desc ibes he speci ica ions
o he new boa d and he adap a ion needed in o de o in eg a e he p e ious
algo i hm in his new boa d. The esul s ob ained du ing he i s s ep o his
in eg a ion a e p esen ed and compa ed o hose ob ained wi h he p e ious
ision sys em. These new esul s a e encou aging and ich in lessons o he
u u e.
Keywo ds: edge compu ing, p ecision ag icul u e, sma ag icul u e, au o-
ma ic weeding, image p ocessing, deep lea ning.
6.1 In oduc ion
Ag icul u e has o ace many challenges in he 21s cen u y. Wi h he inc eas-
ing a i icializa ion o land and he augmen a ion o he global popula ion, we
ha e o p oduce mo e ood using less su ace. Ano he challenge in o de o
99
100 Using Edge AI in IoT de ices o Sma Ag icul u e: Au onomous Weeding
p ese e ou na u al esou ces and soil quali y o ag icul u e, is o p oduce
di e en ly, wi h less inpu s ( e ilize s, phy osani a y p oduc s, he bicides...).
In addi ion, clima e change has a huge impac on p oduc ion, yields, wa e
a ailabili y and many mo e aspec s.
To add ess hose challenges, se e al solu ions can be p oposed, among
which a e sma a ming, he usage o digi al echnologies in ag icul u e and
p ecision ag icul u e, which can be summa ised as applying o he c ops he
app op ia e ac ion, a he bes momen , and a he igh place and quan i y.
In o de o make hose imp o emen s in c op managemen , wo key
echnologies can p o ide a signi ican help: in- ield connec ed senso s and
obo ic p ocessing o he c ops.
In- ield connec ed senso s ha e p o en o be e y use ul o collec ing
da a on he plo s ( ege a ion index, soil composi ion, wea he pa ame e s...).
These da a a e hen p ocessed and in eg a ed in decision suppo sys ems o
help he a me manage he c ops. Ins alling senso s in he ield is no an
easy ask: ou doo condi ions equi e obus ma e ial ha can esis mois u e,
dus and shocks. Mo eo e , access o a powe supply is no p ac ical, so i
is impo an o ha e low consump ion de ices, which limi s he p ocessing
powe a ailable. Cloud compu ing could o e a solu ion: he senso sends he
da a ia in e ne in o de o p ocess i on a se e . Howe e , in e ne access
is o en limi ed in he ields and he amoun o da a can be la ge (images o
ideos). Ano he solu ion could be o use a long- ange echnology such as
LoRa o SigFox. Howe e , hose echnologies ha e a limi ed da a a e and
can’ suppo huge amoun s o da a o send in he cloud.
Edge compu ing is a p omising al e na i e, wi h he possibili y o make
compu ing on boa d, d ama ically educing he amoun o da a o be send
(only he esul s), ia LoRa o example. This makes i possible o use a
connec ed senso e en in a eas wi h poo ne wo k access. I also allows o
educe he powe consump ion in ol ed in he communica ion in case o
la ge amoun o da a. Howe e , his la e ad an age can be neu alized by
he ene gy cos o he calcula ions ca ied ou on boa d.
In he case o obo ic p ocessing o he c ops, embedded senso s a e
necessa y o p o ide eal- ime da a o he sys em so ha i can implemen
he ope a ions needed o p ocess he cul u e. This eal ime cons ain a ou s
he edge a chi ec u e, a oiding loss o ime in da a ans e and ecep ion o
esul s, especially o signi ican inpu da a quan i y (image o ideo).
In bo h cases, he cons ain s a e simila : p ocessing signals, images
o ideos o na u al scenes equi e complex compu a ions; he e u n on
in es men expec ed o he a me s limi s he cos o he echnologies used,
6.4 Wo k in P og ess and Fu u e Wo k 107
a ailable de elopmen ools. The Mobilene -based SSD de ec o was imple-
men ed using he Py o ch lib a y, wi h sepa a e classes o he backbone and
he de ec ion head. A simula o o he Neu oCo gi ci cui was implemen ed
on he N2D2 pla o m [14], which is a deep lea ning amewo k o c ea ing
a i icial neu al ne wo ks in ended o wo k on cons ained en i onmen s.
The SSD ne wo k head was ansla ed o he K ia KV260 FPGA using he
Vi isAI lib a y.
Figu e 6.6 The adap ed ne wo k a chi ec u e used o his applica ion. The igu e p esen s
how he duplica ed Mobilene laye s and he SSD head a e connec ed o he Neu oCo gi
backbone.
Some modi ica ions o he ne wo k s uc u e we e necessa y: as he
aining da abase o he encode con ains ew examples o plan s, he ini ial
de ec ion esul s we e inadequa e. Duplica ing some o he encode laye s on
he FPGA and making hem ainable imp o ed conside ably he de ec ion
accu acy. The esul ing ne wo k a chi ec u e is p esen ed in Figu e 6.6.
Table 6.3 shows he i s esul s om he p oposed a chi ec u e ( hese
esul s a e exp essed in e ms o loss unc ion bu he p ecision pe o mances
will be a ailable soon). These esul s a e p omising, e en i he SSD a chi-
ec u e is less p ecise ha he e e ence Yolo V4 ne wo k. They also show

108 Using Edge AI in IoT de ices o Sma Ag icul u e: Au onomous Weeding
ha i is necessa y o use a leas a pa ially e ained backbone. Howe e , he
imp o emen a e e aining he i s laye s is ma ginal.
Table 6.3 De ec ion pe o mance (loss unc ion) using he new a chi ec u e.
Ne wo k aining Loss on SSD Loss on SSD Li e
Head only 3.8 4.6
Pa ially e ained backbone 1.8 1.9
Re ained backbone 1.5 1.7
Figu e 6.7 p esen s examples o de ec ion ob ained by he e e ence
sys em and by he p oposed ne wo k.
Figu e 6.7 Resul s om he Yolo V4 ne wo k (le ) and he p oposed SSD ne wo k ( igh )
on maize. Blue ec angles show he plan s. G een ec angles show he s em loca ions.
6.4.2 Fu u e wo k
The wo pa s o he ne wo k a e cu en ly being ans e ed on he Pla o m
4.1a. The pe o mances, in e ms o accu acy, p ocessing ime and powe
6.5 Conclusion 109
consump ion will hen be measu ed and compa ed o he e e ence (NVIDIA
Je son Xa ie boa d). I he accu acy and compu ing ime a e adequa e, he
implemen a ion inside he BIPBIP weeding sys em will hen be possible,
allowing ield es ing.
6.5 Conclusion
In his pape we p esen ed a ision sys em o au oma ic weeding (BIPBIP
pla o m), and desc ibed he objec i es and he p og ess o a p ojec o
e ol e his ision sys em, h ough he in eg a ion o an A i icial In elli-
gence o ien ed compu a ion boa d wi h low cos and low powe consump ion
(ANDANTE p ojec ).
The exis ing ision sys em (BIPBIP) should allow easy ha dwa e in eg a-
ion by eplacing he NVIDIA Je son Xa ie wi h he new ci cui . Howe e ,
an adap a ion o he Con olu ional Neu al Ne wo k model appea ed o be
necessa y. Encou aging simula ions ha e shown he o e all easibili y o
he ans e , and ha e been e y in o ma i e, pa icula ly abou he need o
adap he ini ial a chi ec u e o he ci cui o achie e he expec ed p ecision
pe o mance expec ed o weed con ol applica ions.
Fu he mo e, he a ailabili y o he new ANDANTE ci cui makes i
possible o add ess o he “sma ag icul u e” use case such as a ixed ineya d
moni o ing ision senso . The in eg a ion o he new ANDANTE boa d, e en
in i s cu en a chi ec u e, should make i possible o imp o e he e y simple
ision p ocessing algo i hms ca ied ou on boa d he exis ing p o o ype,
while keeping a low powe consump ion inhe en o his ype o de ice, hus
allowing o ex end i s uses.
Acknowledgemen s
The ANDANTE p ojec has ecei ed unding om he ECSEL Join Unde -
aking (JU) unde g an ag eemen No 876925. The JU ecei es suppo om
he Eu opean Union’s Ho izon 2020 esea ch and inno a ion p og amme and
Belgium, F ance, Ge many, The Ne he lands, Po ugal, Spain, Swi ze land.
www.andan e-ai.eu.
The BIPBIP p ojec has been unded by he F ench Resea ch Agency
(ANR) (g an ANR-17-ROSE-0001 - BIPBIP) and has been suppo ed by
he o ganize s o he ROSE Challenge and all he pa ne s o he BIPBIP
p ojec .
110 Using Edge AI in IoT de ices o Sma Ag icul u e: Au onomous Weeding
Re e ences
[1] L. Lac, J-P. Da Cos a, M. Donias, B. Ke esz es, A. Ba de , “C op
s em de ec ion and acking o p ecision hoeing using deep lea ning”.
Compu e s and Elec onics in Ag icul u e, 2022, 192:106606.
[2] L. Lac, “Mé hodes de ision pa o dina eu e d’app en issage p o ond
pou la localisa ion, le sui i e l’analyse de s uc u e de plan es : applica-
ion au déshe bage de p écision”, PhD Thesis, Uni e si é de Bo deaux,
2022.
[3] Lemken. ‘IC-Weede : Au oma ic in a- ow hoeing machine o ege a-
bles’. Accessed 21 June 2024. h ps://lemken.com/en-en/ag icul u al-
machines/c opca e/weed-con ol/mechanical-weed-con ol/ic-weede .
[4] Ga o d Fa m Machine y. ‘Roboc op InRow Weede ’. Accessed 21 June
2024. h ps://ga o d.com/p oduc s/ oboc op-in ow-weede .
[5] VisionWeeding. ‘Mechanical Robo a o ’. Accessed 21 June 2024. h ps:
//www. isionweeding.com/ obo a o -mechanical/.
[6] B. Jiang, J-L. Zhang, W-H Su, and R. Hu. ‘A SPH-YOLO 5x-Based
Au oma ic Sys em o In a-Row Weed Con ol in Le uce’. Ag onomy
13, no. 12 (Dec. 2023): 2915. h ps://doi.o g/10.3390/ag onomy131229
15.
[7] M. Pé ez-Ruiz, D.C. Slaugh e , C.J. Glie e , and S.K. Upadhyaya.
‘Au oma ic GPS-Based In a-Row Weed Kni e Con ol Sys em o
T ansplan ed Row C ops’. Compu e s and Elec onics in Ag icul u e 80
(1 Janua y 2012): 41–49. h ps://doi.o g/10.1016/j.compag.2011.10.0
06.
[8] A. Bochko skiy, C.Y. Wang, H.Y.M. Liao, “YOLO 4: Op imal Speed
and Accu acy o Objec De ec ion”. in a Xi :2004.10934, 2020.
[9] I. Mi o-Panades, I. Kuche , V. Lo ain, A. Valen ian, “Mee ing he
la ency and ene gy cons ain s on iming-c i ical edge-AI sys ems”, in
In e na ional Wo kshop on Embedded A i icial In elligence De ices,
Sys ems, and Indus ial Applica ions (EAI), 2022.
[10] I. Mi o-Panades, E. Romay, L. Ma eu Saez, M. Diaz Na a “Pla o m
4.1a : A Mul i-Applica ion Pla o m Suppo ing Se e al Uses Cases
in he Domains Digi al Fa ming and T anspo and Sma Mobili y”,
Eu opean Con e ence on EDGE AI Technologies and Applica ions
-EEA, 17_10 Oc obe 2023 A hens, G eece.
[11] A.G. Howa d, M. Zhu, B. Chen, D Kalenichenko, W. Wang, T. Weyand,
M. And ee o, H. Adam, “Mobilene s: E icien con olu ional neu al
ne wo ks o mobile ision applica ions”, h ps://a xi .o g/abs/1704
.04861, 2017.
Re e ences 111
[12] W. Liu, D. Anguelo , D., E han, C. Szegedy, S. Reed, C.Y. Fu,
A.C. Be g, “SSD: Single sho mul ibox de ec o ”. In Compu e Vision–
ECCV 2016: 14 h Eu opean Con e ence, Ams e dam, The Ne he -
lands, Sp inge In e na ional Publishing, 2016, P oceedings, Pa I 14,
pp. 21-37.
[13] T.Y. Lin, M. Mai e, S. Belongie, J. Hays, P. Pe ona, D. Ramanan, P.
Dolla , C.L. Zi nick, “Mic oso COCO: Common Objec s in Con ex ”,
in Compu e Vision– ECCV 2014. Sp inge In e na ional Publishing
D. Flee , T. Pajdla, B. Schiele, T. Tuy elaa s, Eds., ol. 8693, 2014,
pp. 740–755.
[14] CEA-LIST. “N2D2: Neu al Ne wo k Design & Deploymen ”, in
gi hub.com, 2017, h ps://gi hub.com/CEA-LIST/N2D2.

Index
A
ad anced d i e assis ance sys em
(ADAS) 90
agen ic AI 34
AI-de ined ehicles (AVD) 1, 33, 39
Aidge xi, 41, 43, 45, 46, 48, 52
a i ac a enua ion 92
a i icial in elligence ne wo k on chip
(AINoC) 58, 60, 61
au oma ic weeding 99, 101, 109
au onomous ehicle xi, 1, 2, 3, 6, 11,
15, 16, 17, 32
C
came a uning 89
CNN accele a o 55
con olu ional neu al ne wo k (CNN)
54, 55, 71, 88, 103, 109
D
da a dis ibu ion se ice (DDS) 27
da a euse 44, 55, 58, 63, 69, 70
da a low a chi ec u e xi, 55, 56, 57,
58, 60
deep lea ning 33, 42, 43, 56, 86, 90,
101
deep lea ning accele a o s (DLAs) 43
deep neu al ne wo k (DNN) xi, 41,
42, 71, 73, 118
denial-o -se ice (DoS) 76
deploymen and compila ion 42
DNN op imisa ion 42
dynamic d i ing ask (DDT) 5
dynamic andom access memo y
(DRAM) 57
E
edge AI xi, 1, 3, 26, 55, 99
edge compu ing xi, 3, 15, 26, 87, 100
edge de ices xii, 44, 73, 77, 84
elec onic con ol uni (ECU) 18
embedded dynamic andom access
memo y (eDRAM) 57
ene gy consump ion 57, 58, 66, 67,
69
ene gy e iciency 26, 57, 69, 70
explainable edge AI (XAI) 33
F
ede a ed lea ning 73, 75, 77, 80, 81,
85
ield-p og ammable ga e a ay
(FPGA) 26
G
g aph manipula ion 42
g aphics p ocessing uni (GPU) 26
H
ha dwa e accele a o 54, 105
ha dwa e expo 41, 44
I
image p ocessing 54, 99
113
114 Index
image signal p ocesso (ISP) xii, 89,
90
ine ial measu emen uni (IMU) 37,
38
ine ial na iga ion sys ems (INS) 14
in e connec xi, 3, 49, 55, 56, 58
In e ne o Robo ic Things (IoRT) 3,
36
In e ne o Things (IoT) 7, 73, 87,
117
in e p e able edge AI (IAI) 33
ISP 89, 90, 91, 92, 93, 97
K
Ke as 17, 48
L
las -mile deli e y xi, 1, 15, 17, 25, 32
la ency 9, 14, 43, 57, 67, 68, 85
ligh de ec ion and anging (LiDAR)
1, 4, 8, 9, 35
long ange adio (LoRa) 100
low powe xii, 9, 44, 53, 85, 99, 101
M
machine lea ning xii, 16, 23, 28, 41,
72, 73, 75, 85, 89
malwa e de ec ion xii, 73, 74, 75, 77,
85
mic o-elec o-mechanical sys ems
(MEMS) 12
ML 16, 73, 74, 75, 89, 94
N
ne wo k op imiza ion 49
neu al global con olle (NGC) 58,
59, 60
neu al p ocessing elemen (NPE) 58,
59, 60
neu al p ocessing uni (NPU) 26
O
objec ecogni ion 2
odome y 12, 23, 24, 36
open neu al ne wo k exchange
(ONNX) 42, 43, 48, 53
OpenCV 17, 48
ope a ional design domain (ODD) 4,
5, 37
ope a ional echnology (OT) 73
P
pa h planning 23, 26, 36, 40
pe cep ion xi, 1, 4, 9, 12, 15, 16, 33,
101
pla ooning 13, 18, 19, 20
poin cloud 8, 10
p ecision ag icul u e 100
p uning 53
Py hon 17, 42, 45, 46, 103
PyTo ch 17, 26, 28, 42, 48, 79, 107
Q
quali y o se ice (QoS) 29
quan iza ion 41, 44, 50, 53
quan iza ion awa e aining (QAT) 53
R
ada 1, 4, 7, 8, 9, 11
eal- ime kinema ic (RTK) 14
eal- ime ope a ing sys em (RTOS)
26
educed ins uc ion se compu ing
(RISC) 53, 55, 67, 69
oadside uni s (RSUs) 13
obo ope a ing sys em (ROS) 4, 27,
37, 39
ROS 1 27, 29
Index 115
ROS 2 27, 29, 30, 31, 33
ROS middlewa e (RMW) 31
S
senso usion xi, 1, 4, 7, 12, 16, 23,
29, 33, 38
SimpleCV 17
simul aneous localiza ion
and mapping (SLAM) 12
small language models (SLMs) 34
sma ag icul u e xii, 99, 101, 109
so wa e-de ined ehicle (SDV) 1, 7,
33, 39
s a ic andom access memo y
(SRAM) 57
Sys em-on-a-Chip (SoC) 26
T
enso p ocessing uni (TPU) 26
Tenso Flow 17, 26, 28, 42, 43
ans o me s 33, 53
uning xii, 21, 42, 89, 90, 93, 95
V
ehicle con ol uni (VCU) 18
ehicle- o-e e y hing (V2X) 1, 7, 9,
13, 33
ision sys em 99, 101, 102, 103, 105,
106
ulne able oad use s (VRUs) 8, 10,
33
W
weeding sys em xii, 101, 102, 109
X
XGboos 94, 96, 97
Y
You Only Look Once (YOLO) 20