I
MDSAA
Mes ado em Mé odos Analí icos A ançados
Mas e P og am in Da a Science and Ad anced Analy ics
Th ea De ec ion wi h Compu e Vision
In e nship a E e is UK
Gab iel Azenha Ca doso
In e nship epo p esen ed as a equi emen o be awa ded
he mas e ’s deg ee in Da a Science and Ad anced Analy ics
II
NOVA In o ma ion Managemen School
Ins i u o Supe io de Es a ís ica e Ges ão de In o mação
Uni e sidade No a de Lisboa
Th ea De ec ion wi h Compu e Vision
by
Gab iel Azenha Ca doso
In e nship epo p esen ed as a equi emen o be awa ded he mas e ’s deg ee in Da a Science and
Ad anced Analy ics
Ad iso : P o esso Dou o Mau o Cas elli
Feb ua y 2023
III
ACKNOWLEDGEMENTS
I mus exp ess my g a i ude o my amily, especially my mo he who encou aged me o ake his mas e
cou se and ga e he suppo and condi ions I needed o ul ill his goal.
I am also hank ul o my iends om back home and all he new iendships I made along he way by
helping me going h ough some ha d momen s ha only a ew could unde s and.
I also mus hank E e is UK o he oppo uni y, and in pa icula o he Inno a ion eam o ecei ing
me wi h a ms wide open and being ex emely pa ien o answe all my ques ions and appeals o
assis ance.
A special hanks o Mau icio o guiding me h ough my i s p o essional expe ience in in o ma ion
echnology and being always kind and suppo i e and Wen o eaching me e e y hing she knew on
compu e ision and deep lea ning.
My inal acknowledgemen goes o P o esso Mau o Cas elli, o all he ad ice and sugges ions o his
pape .
IV
ABSTRACT
This documen desc ibes he wo k conduc ed du ing an in e nship expe ience a he AI Inno a ion
Depa men o E e is UK (now NTT Da a). I epo s wha was done, lea ned, and de eloped wi h he
sole objec i e o ha ing a comme cial p oduc solu ion o he company's clien s.
The p ima y goal was o implemen a solu ion in e ail s o es, o help assis he secu i y eam wi h
h ea de ec ion. To do so, he solu ion consis s in deploying ained deep lea ning models in o
ha dwa e connec ed o he CCTV secu i y came as and de ec ing in ha li e eed any po en ial h ea s.
By he ime I s a ed wo king on his p ojec , was a an ad anced s age so I had o s udy all he wo k
p e iously done o unde s and wha was needed and p ope ly in eg a e he eam ully. My
con ibu ion was ocused on he model aining p ocess, whe e I had o c ea e and s uc u e a da ase
and ain a model capable o de ec ing he a ge ed classes quickly and accu a ely.
KEYWORDS
compu e ision; deep lea ning; in e ence; secu i y
V
INDEX
LIST OF FIGURES ............................................................................................................. VII
LIST OF TABLES .............................................................................................................. VIII
LIST OF ABREVIATIONS AND ACRONYMS ....................................................................... IX
1. INTRODUCTION ........................................................................................................... 1
1.1. IDEA DESCRIPTION ............................................................................................... 2
1.2. BUSINESS OBJECTIVES .......................................................................................... 3
1.3. SITUATION ASSESSMENT ..................................................................................... 3
1.3.1. Resou ces ...................................................................................................... 3
1.3.2. Challenges and Bene i s ................................................................................ 3
1.3.3. Value P oposi ion .......................................................................................... 3
1.3.4. Business Model ............................................................................................. 4
2. THEORETHICAL FRAMEWORK ..................................................................................... 5
2.1. DEEP LEARNING ................................................................................................... 5
2.2. DEEP LEARNING APPLICATIONS IN COMPUTER VISION ....................................... 5
2.2.1. Compu e Vision In oduc ion ...................................................................... 5
2.2.2. Neu al Ne wo ks ........................................................................................... 5
2.2.3. Con olu ional Neu al Ne wo ks .................................................................... 7
2.2.4. Tenso Flow .................................................................................................... 9
2.3. NVIDIA DEEPSTREAM ......................................................................................... 11
2.3.2. Deeps eam G aph A chi ec u e ................................................................. 12
2.3.3. Key Fea u es ................................................................................................ 13
3. SOFTWARES AND TOOLS ........................................................................................... 15
3.1. COMPUTER VISION ANNOTATION TOOL ............................................................ 15
3.1.1. In oduc ion ................................................................................................ 15
3.1.2. CVAT Se up Basics ...................................................................................... 15
3.2. AZURE VIRTUAL MACHINE ................................................................................. 16
3.3. NVIDIA JETSON NANO ........................................................................................ 16
3.4. NVIDIA TRANSFER LEARNING TOOLKIT .............................................................. 17
3.4.1. O e iew ..................................................................................................... 17
3.4.2. P e- ained Models ..................................................................................... 18
3.5. PYTHON .............................................................................................................. 19
3.6. MOBAXTERM ..................................................................................................... 19
3.7. DOCKER .............................................................................................................. 20

VI
4. MODELLING ............................................................................................................... 22
4.1. STATUS OF THE PROJECT .................................................................................... 22
4.1.1. O e iew ..................................................................................................... 22
4.1.2. Sc ip s .......................................................................................................... 23
4.1.3. Model T ain S ep Guide .............................................................................. 28
4.1.4. Limi a ions ................................................................................................... 30
4.2. TRANSFER LEARNING TOOLKIT DEMO ............................................................... 31
4.3. DATA PREPARATION .......................................................................................... 41
4.3.1. KITTI Con e sion ......................................................................................... 41
4.3.2. Anno a ions and Folde S uc u e ............................................................... 43
4.4. TRAINING ........................................................................................................... 44
5. RESULTS DISCUSSION ................................................................................................ 48
6. CONCLUSIONS ............................................................................................................ 50
7. REFERENCES .............................................................................................................. 52
VII
LIST OF FIGURES
FIGURE 1 – MAIN ARCHITECTURE COMPONENTS ................................................................................................... 2
FIGURE 2 – MODEL DEVELOPMENT COMPONENTS ................................................................................................ 2
FIGURE 3 - NTT DATA VALUE PROPOSITION CANVAS ............................................................................................. 3
FIGURE 4 - BUSINESS MODEL CANVAS .................................................................................................................... 4
FIGURE 5 – BIOLOGICAL AND ARTIFICIAL NEURONS [5] .......................................................................................... 6
FIGURE 6 – ACTIVATION FUNCTIONS PLOTS ........................................................................................................... 7
FIGURE 7 – CNN REPRESENTATION ......................................................................................................................... 8
FIGURE 8 – DIAGRAM OF TENSORFLOW ARCHITECTURE ........................................................................................ 9
FIGURE 9 – A SIMPLIFIED GRAPH CORRESPONDING TO A MODEL [6] .................................................................. 10
FIGURE 10 – NVIDIA METROPOLIS ........................................................................................................................ 11
FIGURE 11 – FULL DEEPSTREAM ARCHITECTURE .................................................................................................. 12
FIGURE 12 – INFERENCE WORKFLOW ................................................................................................................... 14
FIGURE 13 – TAO TOOLKIT INTEGRATION ............................................................................................................. 14
FIGURE 14 – TLT PRE-TRAINED MODEL’S OVERVIEW ............................................................................................ 18
FIGURE 15 – PERFORMANCE OF PRE-TRAINED MODELS ..................................................................................... 19
FIGURE 16 – OBJECT DETECTION WORKFLOW ...................................................................................................... 22
FIGURE 17 – TRAIN_SSD.PY RUNNING LOG ........................................................................................................... 30
FIGURE 18 – TLT PAGE ON THE NGC CATALOG FOR VIDEO STREAMING .............................................................. 31
FIGURE 19 – DATA TREE FOR NVIDIA DEMO ......................................................................................................... 33
FIGURE 20 – TLT INFERENCE WITH MASK/NO MASK DEMO MODEL .................................................................... 40
FIGURE 21 – CVAT ANNOTATION EXAMPLE .......................................................................................................... 43
FIGURE 22 – TLT RUNNING LOG ............................................................................................................................ 45
FIGURE 23 – TRAINED MODEL 2 CLASS PRECISION ............................................................................................... 46
FIGURE 24 – TLT INFERENCE TEST ......................................................................................................................... 47
VIII
LIST OF TABLES
TABLE 1 – UK STORES INCIDENT REGISTRATION ..................................................................................................... 1
TABLE 2 – THE TRAIN_SSD.PY ARGUMENTS .......................................................................................................... 24
TABLE 3 – EVAL_SSD.PY ARGUMENTS ................................................................................................................... 27
TABLE 4 - RUN_SSD_EXAMPLE.PY ARGUMENTS ................................................................................................... 28
IX
LIST OF ABREVIATIONS AND ACRONYMS
AI A i icial In elligence
API Applica ion P og amming In e ace
CCTV Closed-Ci cui Tele ision
CLI Command Line In e ace
CNN Con olu ional Neu al Ne wo ks
CPU Cen al P ocessing Uni
CSI Came a Se ial In e ace
CUDA Compu e Uni ied De ice A chi ec u e
CVAT Compu e Vision Anno a ion Tool
DLA Deep Lea ning Accele a o
GPU G aphics P ocessing Uni
HPC High Pe o mance Compu ing
HR Human Resou ces
IT In o ma ion Technology
IVA In elligen Vi ual Agen
ML Machine Lea ning
MVP Minimum Viable P oduc
NGC NVIDIA GPU Cloud
NN Neu al Ne wo ks
NVDEC NVIDIA Video Decoding
NVENC NVIDIA Video Encoding
OEM O iginal Equipmen Manu ac u e
RTSP Real Time S eam P o ocol
SASL Simple Au hen ica ion and Secu i y Laye
SDK So wa e De elopmen Ki
SVM Suppo - ec o machine
TLS T anspo Laye Secu i y
UI Use In e ace
UK Uni ed Kingdom
USB Uni e sal Se ial Bus
VGG Visual Geome y G oup
6
mul iple laye s oge he , a neu al ne wo k can lea n complex pa e ns in da a and pe o m a wide
ange o asks, such as image classi ica ion, speech ecogni ion, and na u al language p ocessing.
Figu e 5 – Biological and a i icial neu ons [5]
The sum o he inpu s is weigh ed, so each inpu is scaled depending on i s co esponding weigh .
Toge he wi h he neu on’s bias, hey a e he main pa ame e s ha we can adjus du ing he aining
phase o op imize he inal esul s. Fo malizing he p ocess ma hema ically o one neu on ha akes
wo inpu s x0 and x1, weigh ed by a ac o w0 and w1, espec i ely, (wi h an op ional bias) we can exp ess
he inpu alues as a ho izon al ec o , he weigh s as e ical ec o and mul iply hem esul ing in
he ollowing equa ion:
So, we can ep esen a simple a i icial neu on in a ne wo k as i ollows.
He e, z is he ou pu o he neu on, x is he inpu ec o , w is he weigh ec o , and b is he bias. The
weigh ec o de e mines he s eng h o he connec ions be ween he inpu and he ou pu , and he
bias de e mines he h eshold o he ac i a ion unc ion.

7
The equa ion ep esen s he do p oduc be ween he inpu ec o and he weigh ec o , wi h he
addi ion o he bias e m. This do p oduc cap u es he weigh ed sum o he inpu s, and he bias e m
shi s he ac i a ion unc ion o he le o igh , which a ec s he ange o ou pu s ha can be
p oduced by he neu on.
Be o e he neu on ou pu s, he signal has o pass a key componen o he o iginal pe cep on, he
ac i a ion unc ion, which wi h linea inpu s akes a bina y o m, e u ning 1 o a 0 (usually wi h =
0).
In non-linea i y cases (mo e complex beha io s) and con inuous di e en iabili y he mos common
unc ions a e:
● The sigmoid unc ion, (wi h
𝑒
" he exponen ial unc ion)
● The hype bolic angen ,
● The Rec i ied Linea Uni (ReLU),
To isualize he di e ence be ween each unc ion, plo s in he ollowing igu e:
Figu e 6 – Ac i a ion unc ions plo s [5]
Wi h his logic in mind, we modeled a simple a i icial neu on, ha can ecei e a signal, p ocess i and
ou a alue ha can be o wa ded o o he neu ons, building a ne wo k [5].
2.2.3. Con olu ional Neu al Ne wo ks
Knowing wha Neu al Ne wo ks a e, we go a s ep deepe in o he concep and wen on he
unde s anding wha con olu ional neu al ne wo ks (CNNs) a e and how hese mode n me hods a e
ained o u he imp o e hei obus ness. CNNs we e in oduced o sol e some o he sho comings
o he o iginal neu al ne wo ks. The wo main d awbacks o basic ne wo ks when dealing wi h images
a e he explosi e numbe o pa ame e s and he lack o spa ial easoning.
8
Images a e e y complex s uc u es wi h a la ge numbe o alues (H x W x D alues wi h H indica ing
he image’s heigh , W wid h, and D he dep h). E en small single-channel images can ha e inpu
ec o s o size (e.g.) 28 x 28 x 1 = 784 alues each and hese numbe s simply explode when conside ing
la ge RGB images o deepe ne wo ks.
Also, because hei neu ons ecei e all he alues om he p e ious laye wi hou any dis inc ion ( hey
a e ully connec ed), hese neu al ne wo ks do no ha e he concep s o dis ance o spa iali y. Spa ial
ela ions in he da a a e los . Mul idimensional da a (e.g. images) could also be any hing om column
ec o s o dense laye s because hei ope a ions do no ake in o accoun he da a dimensionali y no
he posi ions o inpu alues, mo e p ecisely his means ha he idea o p oximi y be ween pixels is
los o ully connec ed (FC) laye s, as all pixel alues a e combined by he laye s wi hou conside ing
hei o iginal posi ions.
As i does no change he beha io o dense laye s, o simpli y, i la ens mul idimensional inpu s
be o e passing hem o hese laye s ( eshapes hem in o column ec o s). Neu al laye s would be
sma e i hey could conside he spa ial in o ma ion, meaning ha some inpu alues belong o he
same pixel (channel alues) o he same image egion (neighbo pixels).
CNNs o e s simple solu ions o hese sho comings while wo king in he same way as he ne wo ks
p e iously p esen ed. Fi s , CNNs can handle mul idimensional da a. Fo images, i akes as inpu he
same h ee-dimensional da a (H x W x D) and has i s neu ons a anged in a simila olume, and his
leads o he second imp o emen o CNNs ha unlike ully connec ed ne wo ks each neu on only has
access o some elemen s in he neighbo ing egion o he p e ious laye . This egion is called he
ecep i e ield o he neu ons:
Figu e 7 – CNN ep esen a ion [7]
By linking neu ons only o hei neighbo ing ones in he p e ious laye , CNNs no only d as ically educe
he numbe o pa ame e s o ain, bu also p ese e he localiza ion o image ea u es [7].
9
Now ha we know he basic concep and logic o CNNs we can unde s and why hey lay he obus
aining and op imiza ion ounda ions o he models used o compu e ision p ojec s and used o
he de elopmen o ou solu ion.
2.2.4. Tenso Flow
Tenso Flow is an open-sou ce lib a y de eloped by Google p ima ily o simpli y he
deploymen o machine lea ning solu ions on a ious pla o ms.
Tenso Flow’s a chi ec u e is composed o a C++ laye , a py hon low-le el API ha w aps C++ sou ces,
so when calling a py hon me hod in Tenso Flow, i usually in okes C++ code behind he scenes bu
since py hon is conside ed o be easie o use his w appe allows use s o wo k mo e quickly. A he
op laye is he high-le el API made om wo componen s, Ke as and he Es ima o API.
The Es ima o API con ains p e-made componen s ha allow you o build machine lea ning models
mo e easily, e y simila o building blocks o empla es. Ke as is a use - iendly, modula , and
ex ensible w appe o Tenso Flow ha a i s was designed as an in e ace o enable as
expe imen a ion wi h neu al ne wo ks.
Figu e 8 – Diag am o Tenso Flow a chi ec u e [6]
In his p ojec , we used Tenso Flow 2 which was eleased wi h some new ea u es and concep s. One
o hose co e concep s is he enso s, which can be desc ibed as N-dimensional a ays ha could ake
he o m o a scala , a ec o , a 3D ma ix, o an N-dimensional ma ix. This componen is used o s o e
ma hema ical alues which can be ixed alues c ea ed using .cons an o changing alues c ea ed
using . a iable.
Tenso Flow uses enso s as inpu s and as ou pu s, and a componen ha ans o ms one in o he o he
is called an ope a ion, he e o e a compu e ision model is composed o mul iple ope a ions. These
ope a ions a e ep esen ed using wha is called a di ec ed acyclic g aph (DAC), also e e ed o as a
Tenso Flow g aph.
10
Figu e 9 – A simpli ied g aph co esponding o a model [6]
Each node is composed o smalle ope a ions such as ma ix mul iplica ions and addi ions and while
e y simple his g aph ep esen s he di e en laye s o a simple model in he o m o mul iple
ope a ions. By elying on g aphs, Tenso Flow can un pa o he ope a ions on he CPU and ano he
pa on he GPU, un di e en pa s o he g aph on di e en machines in case o a dis ibu ed model,
and op imize he g aph o a oid unnecessa y ope a ions, lending o be e compu a ional
pe o mance.
Ano he use ul concep is backp opaga ing e o s using g adien ape. In sho , Tenso Flow compu es
he esul s o an ope a ion ins ead o s o ing he ope a ion, wi h no in o ma ion on he ope a ion and
i s inpu s i would be impossible o au oma ically di e en ia e he loss ope a ion. Tha is whe e he
g adien ape makes a di e ence, by unning ou loss compu a ion in he con ex o .g adien ape,
Tenso Flow will au oma ically eco d all ope a ions and allow us o eplay hem backwa d a e wa d.
de ain_s ep():
wi h .G adien Tape() as ape:
loss = .ma h.abs(A * X - B)
dX = ape.g adien (loss, X)
p in ('X = {:.2 }, dX = {:2 }'. o ma (X.numpy(), dX))
X.assign(X - dX)
o i in ange(7):
ain_s ep()
The code abo e de ines one aining s ep. E e y ime ain_s ep is called, he loss is compu ed in he
con ex o he g adien ape, hen he con ex is used o compu e he g adien . The X a iable is hen
upda ed, and we can see i con e ging owa d he alue ha sol es he equa ion:
X = 20.00, dX = 3.000000
X = 17.00, dX = 3.000000
X = 14.00, dX = 3.000000
X = 11.00, dX = 3.000000
So, o inno a i e models o when expe imen ing, he g adien ape is a powe ul ool ha allows
au oma ic di e en ia ion wi hou much e o [6].
11
2.3. NVIDIA DEEPSTREAM
2.3.1. Wha is NVIDIA Deeps eam?
“NVIDIA’s Deeps eam SDK is a comple e s eaming analy ics oolki based on GS eame o
AI-based mul i-senso p ocessing, ideo, audio, and image unde s anding. I ’s ideal o ision AI
de elope s, so wa e pa ne s, s a ups, and OEMs building IVA apps and se ices” [8]. Wi h his oolki ,
de elope s can cons uc s eam p ocessing pipelines ha include neu al ne wo ks and o he
ad anced p ocessing asks, such as acking, ideo encoding/decoding, and ideo ende ing. These
pipelines enable eal- ime analysis on ideo, image, and senso da a, acili a ing imely insigh s and
analy ics.
Figu e 10 – NVIDIA Me opolis [8]
Deeps eam akes he s eaming da a as inpu – om a USB/CSI came a, ideo om a ile, o s eams
o e RTSP (which is he a ailable s eam o ma o us) and uses “AI and compu e ision o gene a e
insigh s om pixels o a be e unde s anding o he en i onmen ” [9]. This oolki has he po en ial
o se e as a undamen al building block o a ious ideo analy ic applica ions, such as sa e y
moni o ing, e ail sel -checkou , and ou speci ic ocus, objec de ec ion.
DeepS eam suppo s applica ion de elopmen in C/C++ and Py hon, o e ing Py hon bindings o
ex a ease o use. The oolki includes p e-buil e e ence applica ions in C/C++ and Py hon o simpli y
he de elopmen p ocess. The co e SDK includes ha dwa e accele a o plugins ha le e age
accele a o s such as VIC, GPU, DLA, NVDEC, and NVENC, enabling compu e-in ensi e asks o be
o loaded o dedica ed accele a o s o op imal pe o mance in ideo analy ics applica ions.

12
A no able ea u e o DeepS eam is i s secu e bi-di ec ional communica ion be ween edge and cloud
en i onmen s. The oolki includes buil -in secu i y p o ocols such as SASL/Plain au hen ica ion using
use name/passwo d and 2-way TLS au hen ica ion o ensu e da a secu i y.
DeepS eam uses a ious NVIDIA lib a ies om he CUDA-X s ack, including CUDA, Tenso RT, NVIDIA
T i on In e ence se e , and mul imedia lib a ies. Tenso RT accele a es AI in e ence on NVIDIA GPUs,
and DeepS eam abs ac s hese lib a ies in o plugins, simpli ying he de elopmen o ideo analy ic
pipelines wi hou he need o lea n each lib a y.
DeepS eam is op imized o NVIDIA GPUs and can in eg a e on embedded edge de ices unning he
Je son pla o m, as well as da a cen e GPUs like T4. DeepS eam applica ions can be con aine ized
using NVIDIA con aine Run ime and a e a ailable on NGC, he NVIDIA GPU cloud egis y [9].
2.3.2. Deeps eam G aph A chi ec u e
“Deeps eam is an op imized g aph a chi ec u e buil using he open-sou ce GS eame
amewo k. The g aph below shows a ypical ideo analy ic applica ion s a ing om inpu ideo o
ou pu ing insigh s. All he indi idual blocks a e a ious plugins ha a e used. A he bo om a e he
di e en ha dwa e engines ha a e u ilized h oughou he applica ion. Op imum memo y
managemen wi h ze o-memo y copy be ween plugins and he use o a ious accele a o s ensu e he
highes pe o mance” [9].
Figu e 11 – Full Deeps eam A chi ec u e [9]
DeepS eam p o ides a comp ehensi e sui e o GS eame plugins ha se e as ounda ional
building blocks o cons uc ing e icien ideo analy ics pipelines. These plugins a e speci ically
designed o ha ness ha dwa e accele a ion capabili ies and op imize pe o mance in a ious ideo
p ocessing asks:
• The s eaming da a can be cap u ed om di e se sou ces such as RTSP, local ile sys ems,
o came as and p ocessed using he CPU.
• Op ionally, image p e-p ocessing can be applied a e decoding, including image de-
wa ping o co ec ing dis o ions om FishEye lenses o colo space con e sion.
13
• F ames a e hen ba ched o op imal in e ence pe o mance using he gs -n s eammux
plugin.
• In e ence can be pe o med using Tenso RT, NVIDIA's in e ence accele a o un ime, o
na i e amewo ks such as Tenso Flow o PyTo ch wi h he T i on in e ence se e . GPU
o DLA (Deep Lea ning Accele a o ) can be u ilized o in e ence on Je son AGX Xa ie and
Xa ie NX.
• Objec acking can be pe o med a e in e ence using buil -in e e ence acke s in he
SDK.
• Visualiza ion a i ac s such as bounding boxes, segmen a ion masks, and labels can be
c ea ed.
DeepS eam o e s a ious op ions o ou pu ing he esul s, such as ende ing he ou pu wi h
bounding boxes on he sc een, sa ing he ou pu o a local disk, s eaming ou o e RTSP, o sending
me ada a o he cloud. Buil -in b oke p o ocols such as Ka ka, MQTT, AMQP, and Azu e IoT a e
a ailable, and cus om b oke adap e s can be c ea ed o sui speci ic equi emen s [9].
2.3.3. Key Fea u es
DeepS eam p o ides comp ehensi e suppo o AI models, speci ically o objec de ec ion
and segmen a ion, encompassing cu ing-edge models such as SSD, YOLO, Fas e RCNN, and
MaskRCNN. Mo eo e , DeepS eam acili a es he in eg a ion o cus om unc ions and lib a ies,
ca e ing o unique equi emen s.
The e sa ili y o DeepS eam spans om apid p o o yping o ull-scale p oduc ion-le el solu ions,
a o ding he lexibili y o choose he mos sui able in e ence pa h. The pla o m seamlessly in eg a es
wi h he NVIDIA T i on™ In e ence Se e , allowing o he deploymen o models in na i e amewo ks
such as PyTo ch and Tenso Flow o in e ence. Addi ionally, DeepS eam le e ages he NVIDIA
Tenso RT™ o enable high- h oughpu in e ence wi h suppo o mul i-GPU, mul i-s eam, and
ba ching, leading o op imal pe o mance.
In addi ion o suppo ing na i e in e ence, DeepS eam's capabili ies ex end o communica ion wi h
independen / emo e ins ances o he T i on In e ence Se e ia gRPC, enabling he implemen a ion
o dis ibu ed in e ence solu ions. This empowe s e icien and scalable deploymen o AI models in
dis ibu ed en i onmen s, ensu ing obus ness and scalabili y o he in e ence p ocess.
14
Figu e 12 – In e ence wo k low [8]
Fu he mo e, DeepS eam acili a es seamless in eg a ion wi h he T ans e Lea ning Toolki (TAO
Toolki ) wi hin he NVIDIA Me opolis ecosys em, enabling accele a ed de elopmen and enhanced
eal- ime pe o mance o ision AI sys ems. The TAO Toolki enables de elope s o adap and op imize
p oduc ion-quali y ision AI models, including SSD, MaskRCNN, YOLO 4, Re inaNe , and o he s a e-
o - he-a models, while DeepS eam o e s u nkey in eg a ion o hese models o deploymen .This
in eg a ed app oach empowe s he use o p e- ained models and ans e lea ning echniques,
esul ing in a s eamlined wo k low o building end- o-end ision AI applica ions. By le e aging he
TAO Toolki in conjunc ion wi h DeepS eam, de elope s can achie e as e de elopmen cycles and
supe io pe o mance in eal- ime scena ios, making i a aluable ool o academic esea ch and
de elopmen in he ield o ision-based AI [9].
Figu e 13 – TAO Toolki in eg a ion [11]
15
3. SOFTWARES AND TOOLS
3.1. COMPUTER VISION ANNOTATION TOOL
3.1.1. In oduc ion
CVAT is a ee and open-sou ce in e ac i e ideo and image anno a ion ool o compu e
ision. I was designed o p o ide use s wi h a se o con enien ins umen s o anno a ing digi al
images and ideos since da a scien is s need lo s o anno a ed da a o ain (in ou case) he CNNs a
he co e o AI wo k lows and ob ain ha da a wi h he necessa y quali y is a huge and ime-consuming
challenge.
CVAT suppo s supe ised machine lea ning asks pe aining o objec de ec ion, image classi ica ion,
image segmen a ion, and 3D da a anno a ion. I allows use s o anno a e images wi h mul iple ools
(boxes, polygons, cuboids, ci cles, skele ons, e c).
3.1.2. CVAT Se up basics
In ou pa icula case, we used machines wi h Windows 10 and o ha , we needed o ins all
a ew o he ools o u he use:
• Ins all WSL2 (Windows Subsys em o Linux);
• Download and ins all Docke Desk op;
• Download and ins all gi o Windows;
• Download and ins all Google Ch ome, i is he only b owse ha suppo s CVAT;
Open he Gi Bash applica ion and clone he CVAT sou ce code om he Gi Hub eposi o y wi h he
ollowing command.
gi clone h ps://gi hub.com/openc /c a
cd c a
Then we mus un docke con aine s. I will ake some ime o download he la es CVAT elease and
o he equi ed images like Pos g es, Redis, e c. om Docke Hub and c ea e con aine s.
docke -compose up -d
A e his, we c ea e a supe use . A supe use can use an admin panel o assign co ec g oups o o he
use s.
# en e docke image i s
Docke exec -i c a _se e /bin/bash
# hen un
22
4. MODELLING
4.1. STATUS OF THE PROJECT
4.1.1. O e iew
By he ime I go assigned o his p ojec , he e was al eady a sys em designed and es ed. This
sys em was based on he wo k ha an N idia de elope ha goes by he name o dus y-n in Gi Hub.
On his Gi Hub, he e is a eposi o y wi h a sha ed wo k (h ps://gi hub.com/dus y-n /py o ch-
ssd/ ee/8ed842a408 8c4a8812 430c 8063e0b93a56803) ha consis s o he implemen a ion o an
SSD (Single Sho De ec o ) o objec de ec ion wi h PyTo ch and using MobilleNe a chi ec u e.
In his gi eposi o y, we ha e all he necessa y sc ip s (w i en in py hon) o he aining, e alua ion,
and expo o he ained model, we mus p epa e he inpu wi h ou labeled da a and un he sc ip s
wi h ou speci ica ions.
Figu e 16 – Objec De ec ion Wo k low
To s a he p ocess, we ha e o ge a usable inpu , meaning we ha e o label da a wi h he classes we
wan o de ec (e.g. kni e, ace mask…), which will be explained in mo e de ail ahead.
Ha ing he co ec inpu , we call di ec ly he spli _ ain es .py o sepa a e ou da a in o wo da ase s,
one o aining and he o he o es ing la e . Then we un he ain_ssd.py sc ip , wi h he selec ed
p e- ained model and subsequen speci ica ions. When he aining concludes we check he de ec ion
accu acy o ou model o each class by calling he e al_ssd.py. Wi h he ou pu , we e alua e i he
model is eady o be expo ed and es ed o i we need o e ain i wi h mo e da a labeled wi h a
speci ic class o e en wi h a di e en p e- ained model, he accu acy benchma k is 70% o each class
de ec ion.
Finally, i he accu acy is o e ou minimum benchma k, we call he onnx_expo .py, which will gi e us
a .onnx ile con aining he model eady o be es ed wi h he un_ssd_example.py, whe e we con i m
i ou model de ec s any hing in a se o andom images. A e we con i m de ec ions, he model is
s o ed in ou VM and sen o he Je son Nano, o un on he li e CCTV eed connec ed o ha machine.

23
4.1.2. Sc ip s
As men ioned abo e, in he aining p ocess wi h PyTo ch we use 5 sc ip s:
• spli _ ain es .py
• ain_ssd.py
• e al_ssd.py
• onnx_expo .py
• un_ssd_example.py
The i s is he spli _ ain es .py, which is a Py hon sc ip ha spli s anno a ed images in o aining,
alida ion, and es da ase s. The sc ip uses he A gumen Pa se module o allow he use o inpu
he pa h o he anno a ions di ec o y and he images spli ex ou pu di ec o y. The Pa h() objec is
used o o ma he inpu pa hs acco ding o he ope a ing sys em being used. The os module's lis di
unc ion is used o ge a lis o all he iles in he anno a ions di ec o y. The names o he iles a e hen
ex ac ed, spli in o aining and es da ase s using he ‘ ain_ es _spli ’ unc ion om sciki -lea n, and
w i en o sepa a e ex iles in he images spli x ou pu di ec o y.
### spli he whole da ase o ain and es : 8:2
om sklea n.model_selec ion impo ain_ es _spli
X_ ain, X_ es = ain_ es _spli (name_lis , es _size=0.20, andom_s a e=
1)
### w i e he names o spe a e x iles: ain_ al and es
ain_lis = open( ain_ ilename,'w')
o elemen in X_ ain:
ain_lis .w i e(elemen )
ain_lis .w i e(' n')
ain_lis .close()
es _lis = open( es _ ilename,'w')
o elemen in X_ es :
es _lis .w i e(elemen )
es _lis .w i e(' n')
es _lis .close()
Wi h he inpu da a o he ain_ssd.py sc ip , he ain can begin. This is a sc ip o aining and
e alua ing a single-sho mul i-box de ec ion model using he PyTo ch deep lea ning amewo k. I
suppo s se e al base a chi ec u es (VGG-16, MobileNe -V1, MobileNe -V2) and allows o aining on
di e en da ase s (PASCAL VOC, OpenImages). The sc ip co e s he ollowing s eps:
1. Se ing up logging and command line a gumen pa sing.
2. Loading da ase s and de ining da a loade s.
3. De ining he base a chi ec u e and cus omizing i wi h he desi ed pa ame e s.
4. Se ing up he loss unc ion and op imize .
5. Ini ializing he model and unning aining and e alua ion.
6. Logging aining s a is ics using Tenso Boa d.
This sc ip uses se e al lib a ies such as a gpa se, o ch, and o ch ision. The a gpa se module is used
o de ine command-line a gumen s o a ious hype pa ame e s and se ings, and once hese
hype pa ame e s and se ings a e de ined, he sc ip will ain he de ec o acco ding o hose
speci ica ions.
24
Table 2 – The ain_ssd.py a gumen s
The ‘ ain()’ is he unc ion ha ains a deep lea ning model on an objec de ec ion ask. The inpu
includes a da a loade , a ne wo k model, a loss c i e ion, an op imize , and a de ice (GPU in his case
is). Du ing each i e a ion, he g adien s o he model pa ame e s a e compu ed and upda ed using he
op imize . The unning loss, eg ession loss, and classi ica ion loss a e compu ed and logged e e y
"debug_s eps" i e a ions. The Tenso Boa d scala alues o he a e age loss, a e age eg ession loss,
and a e age classi ica ion loss a e also eco ded. Addi ionally, he ne wo k g aph is added o
Tenso Boa d a e he aining is comple ed.
de ain(loade , ne , c i e ion, op imize , de ice, debug_s eps=100, epoch
=-1):
ne . ain(T ue)
unning_loss = 0.0
unning_ eg ession_loss = 0.0
unning_classi ica ion_loss = 0.0
o i, da a in enume a e(loade ):
images, boxes, labels = da a
images = images. o(de ice)
boxes = boxes. o(de ice)
labels = labels. o(de ice)
op imize .ze o_g ad()
con idence, loca ions = ne (images)
eg ession_loss, classi ica ion_loss = c i e ion(con idence, loca i
ons, labels, boxes) # TODO CHANGE BOXES
loss = eg ession_loss + classi ica ion_loss
loss.backwa d()
op imize .s ep()
unning_loss += loss.i em()
unning_ eg ession_loss += eg ession_loss.i em()
unning_classi ica ion_loss += classi ica ion_loss.i em()
i i and i % debug_s eps == 0:
a g_loss = unning_loss / debug_s eps
a g_ eg_loss = unning_ eg ession_loss / debug_s eps
a g_cl _loss = unning_classi ica ion_loss / debug_s eps
logging.in o(
"Epoch: {epoch}, S ep: {i}/{len(loade )}, " +
"A g Loss: {a g_loss:.4 }, " +
"A g Reg ession Loss {a g_ eg_loss:.4 }, " +
"A g Classi ica ion Loss: {a g_cl _loss:.4 }"
)
25
unning_loss = 0.0
unning_ eg ession_loss = 0.0
unning_classi ica ion_loss = 0.0
b.add_scala ('T ain_Loss', a g_loss, epoch)
b.add_scala ('T ain_Reg ession_Loss', a g_ eg_loss, epoch)
b.add_scala ('T ain_Classi ica ion_Loss', a g_cl _loss, epoch)
b.add_g aph(ne , images)
Then he ‘ es ()’ unc ion is used o e alua e he pe o mance o he objec de ec ion model. I se s
he ne wo k o e alua ion mode by calling ‘ne .e al()’, and hen calcula es he loss on he alida ion
da ase . The unc ion loops o e he alida ion da a and calcula es he loss, eg ession loss and
classi ica ion loss. The loss is calcula ed by calling he ‘c i e ion’ unc ion and passing i he ne wo k's
ou pu , ‘con idence’ and ‘loca ions’, along wi h he g ound u h ‘labels ‘and ‘boxes.’ The loss is
accumula ed o each ba ch and di ided by he numbe o ba ches o ob ain he a e age loss o he
en i e alida ion da ase .
de es (loade , ne , c i e ion, de ice):
ne .e al()
unning_loss = 0.0
unning_ eg ession_loss = 0.0
unning_classi ica ion_loss = 0.0
num = 0
o _, da a in enume a e(loade ):
images, boxes, labels = da a
images = images. o(de ice)
boxes = boxes. o(de ice)
labels = labels. o(de ice)
num += 1
wi h o ch.no_g ad():
con idence, loca ions = ne (images)
eg ession_loss, classi ica ion_loss = c i e ion(con idence, lo
ca ions, labels, boxes)
loss = eg ession_loss + classi ica ion_loss
unning_loss += loss.i em()
unning_ eg ession_loss += eg ession_loss.i em()
unning_classi ica ion_loss += classi ica ion_loss.i em()
e u n unning_loss / num, unning_ eg ession_loss / num, unning_class
i ica ion_loss / num
A he end o he unc ion, i e u ns he a e age o he o al loss, a e age eg ession loss, and a e age
classi ica ion loss.
The nex s ep is o e alua e he model’s p ecision wi h he e al_ssd.py sc ip .
I has wo unc ions, he i s one is he ‘g oup_anno a ion_by_class()’, which is he unc ion ha
g oups he anno a ions in he da ase , loops o e all o hose anno a ions and ex ac s in o ma ion
such as he g ound- u h bounding boxes, he class labels, and whe he he anno a ion is conside ed
di icul o no . The ex ac ed in o ma ion is hen s o ed in h ee dic iona ies: ‘ ue_case_s a ’,
‘all_g _boxes’, and ‘all_di icul _cases’. The ‘ ue_case_s a ’ dic iona y s o es he numbe o ue (i.e.,
26
non-di icul ) cases o each class. The ‘all_g _boxes’ dic iona y s o es he g ound- u h bounding
boxes o each image and class, and he ‘all_di icul _cases’ dic iona y s o es he di icul y s a us o
each anno a ion o each class and image. Finally, he unc ion con e s he lis s o bounding boxes
and di icul ies in o enso s and e u ns he h ee dic iona ies.
Then i uns he second unc ion on his sc ip , he compu e_a e age_p ecision_pe _class(), which is
he unc ion ha compu es he a e age p ecision o objec de ec ion. The inpu a gumen s include:
• ‘num_ ue_cases’: a dic iona y wi h class index as key and numbe o ue posi i e cases o
ha class as alue.
• ‘g _boxes’: a dic iona y wi h class index as key and o each class, a dic iona y o image IDs and
hei co esponding g ound u h bounding boxes.
• ‘di icul _cases’: a dic iona y wi h class index as key and o each class, a dic iona y o image
IDs and he co esponding g ound u h "di icul y" labels.
• ‘p edic ion_ ile’: a ile con aining he model p edic ions, wi h each line ha ing he o ma
‘image_id sco e box_coo dina es’.
• ‘iou_ h eshold’: he In e sec ion o e Union (IoU) h eshold used o de e mine a ue posi i e
ma ch be ween a p edic ion and g ound u h.
• ‘use_2007_me ic’: a lag indica ing whe he o use he 11-poin in e pola ion me hod (VOC
2007) o no .
The unc ion i s eads he p edic ion ile and s o es he image IDs, boxes, and sco es in sepa a e lis s,
so ed by sco e in descending o de . Then, o each p edic ion, i inds he co esponding g ound u h
box (i he e is one) and compu es i s IoU wi h he p edic ion. I he IoU is abo e he h eshold, he
p edic ion is conside ed a ue posi i e, unless he g ound u h is ma ked as "di icul ". The ue and
alse posi i e coun s a e accumula ed as he loop p og esses and a e used o compu e p ecision and
ecall. Finally, he a e age p ecision is compu ed using ei he he VOC 2007 me hod (i
‘use_2007_me ic’ is se ) o he s anda d me hod.
de compu e_a e age_p ecision_pe _class(num_ ue_cases,g _boxes,
di icul _cases, p edic ion_ ile,
iou_ h eshold, use_2007_me ic):
wi h open(p edic ion_ ile) as :
image_ids = []
boxes = []
sco es = []
o line in :
= line. s ip().spli (" ")
image_ids.append( [0])
sco es.append( loa ( [1]))
box = o ch. enso ([ loa ( ) o in [2:]]).unsqueeze(0)
box -= 1.0
boxes.append(box)
sco es = np.a ay(sco es)
so ed_indexes = np.a gso (-sco es)
boxes = [boxes[i] o i in so ed_indexes]
image_ids = [image_ids[i] o i in so ed_indexes]
ue_posi i e = np.ze os(len(image_ids))
27
alse_posi i e = np.ze os(len(image_ids))
ma ched = se ()
o i, image_id in enume a e(image_ids):
box = boxes[i]
i image_id no in g _boxes:
alse_posi i e[i] = 1
con inue
g _box = g _boxes[image_id]
ious = box_u ils.iou_o (box, g _box)
max_iou = o ch.max(ious).i em()
max_a g = o ch.a gmax(ious).i em()
i max_iou > iou_ h eshold:
i di icul _cases[image_id][max_a g] == 0:
i (image_id, max_a g) no in ma ched:
ue_posi i e[i] = 1
ma ched.add((image_id, max_a g))
else:
alse_posi i e[i] = 1
else:
alse_posi i e[i] = 1
ue_posi i e = ue_posi i e.cumsum()
alse_posi i e = alse_posi i e.cumsum()
p ecision = ue_posi i e / ( ue_posi i e + alse_posi i e)
ecall = ue_posi i e / num_ ue_cases
i use_2007_me ic:
e u n measu emen s.compu e_ oc2007_a e age_p ecision(p ecision, e
call)
# 11 poin in e pola ion o a e age p ecision
else:
e u n measu emen s.compu e_a e age_p ecision(p ecision, ecall)
The a gspa se inpu hype pa ame e s and se ings o his sc ip .
Table 3 – e al_ssd.py a gumen s
I he p ecision ac oss all classes is o e 70%, he model is eady o be expo ed, es ed, and sen o
he eam ha will es he in e ence in he s o e’s CCTV li e eed wi h he Je son Nano.
To expo he ile he e is he onnx_expo .py sc ip . This sc ip con e s he ained PyTo ch objec
de ec ion model o ONNX o ma . I uses he ‘ o ch.onnx’ module o he con e sion and ‘a gpa se’
o pa sing he command-line a gumen s o ne wo k a chi ec u e, inpu and ou pu pa hs, labels ile,
inpu dimensions, and ba ch size. The sc ip se s he de ice o use CUDA i a ailable and speci ied by
he use , o he wise i uses he CPU.

28
The sc ip suppo s mul iple ne wo k a chi ec u es, including VGG16, MobileNe 1, MobileNe 1 Li e,
SqueezeNe Li e, and MobileNe 2 Li e.
I o ma s he inpu model pa hs and au oma ically selec s he checkpoin wi h he lowes loss i an
inpu pa h is no speci ied, de e mines he numbe o classes by coun ing he lines in he labels ile,
cons uc s he ne wo k a chi ec u e based on he speci ied a chi ec u e a gumen , hen loads he
PyTo ch model checkpoin , con e s i o ONNX o ma , and sa es i o he speci ied ou pu pa h.
Table 4 – onnx_expo .py a gumen s
The las sc ip o his model ain wo k low is he un_ssd_example.py, which is he in e ence es o
he objec de ec ion model. The sc ip loads he inpu images and he label ile and c ea es a lis o
image pa hs. A e ha , i chooses he ne wo k a chi ec u e based on he p o ided "--ne " a gumen
and c ea es he co esponding SSD ne wo k, hen i loads he ained model weigh s and uns objec
de ec ion on each image in he ‘images_pa hs’ lis . The ou pu o he de ec ion is he images wi h he
bounding boxes a ound he de ec ed objec s, which a e sa ed in he speci ied ‘ou pu _images_di ’.
Table 5 - un_ssd_example.py a gumen s
4.1.3. Model ain s ep guide
The whole model ain wo k low wi h PyTo ch can be esumed o a ew s eps. Be o e unning
he sc ip s and assuming ha he e al eady is a VOC PASCAL anno a ions da ase , we s a by accessing
he GPU machine (Azu e i ual machine wi h 4 GPUs) wi h MobaX e m.
ssh e e isai@**.***.***.**
passwo d: *********
29
Nex we ac i a e he ‘openc ’ en i onmen wi h a conda command. I allows o use a speci ic e sion
o he lib a y in isola ion om he es o ou sys em. A e execu ing his command, we should be able
o un he sc ip s ha use OpenCV.
conda ac i a e openc
Then we un a command o s a a PyTo ch Docke con aine using he N idia GPU accele a ion and
map he hos di ec o y, whe e we ha e s o ed he anno a ions da ase and he sc ip s, o he con aine
di ec o y.
sudo n idia-docke un --ipc=hos --gpus all -i -- m - /home/e e isai/py
hon/:/py hon n c .io/n idia/py o ch:21.05-py3
The ‘--ipc=hos ’ lag is used o sha e he hos in e -p ocess communica ion namespace wi h he
con aine , allowing i o access he hos 's GPU de ices, he ‘--gpus all’ lag speci ies ha all a ailable
GPUs should be used, he ‘-i ’ s a s he con aine in in e ac i e mode and he ‘— m’ au oma ically
emo es he con aine when i exi s. The ‘n c .io/n idia/py o ch:21.05-py3’ is he image name o he
PyTo ch en i onmen wi h he CUDA e sion.
The nex s ep is o ins all he lib a ies ha we need and a e missing om he con aine .
pip ins all o ch==1.7.0+cu110 o ch ision==0.8.1+cu110 o chaudio===0.7.0
- h ps://download.py o ch.o g/whl/ o ch_s able.h ml
The command will ins all (ou da ed e sions used by he ime o his p ojec ) PyTo ch e sion 1.9.0,
o ch ision 0.8.1 and o chaudio 0.7.0. The ‘+cu110’ indica es ha hese packages a e buil wi h CUDA
11.0 suppo , which equi es an N idia GPU. The ‘- ’ op ion allows you o speci y he o icial PyTo ch
eposi o y o s able eleases URL om which o download he package.
Then we se he di ec o y o he sc ip s olde and s a o un hem.
cd /py hon/ aining/de ec ion/model_ 1.0
Inside he con aine we s a by unning he ain_ssd.py sc ip . The spli _ ain es .py is an in ou local
machines and hen he sepa a ed da a is impo ed in o he con aine .
py hon3 ain_ssd.py --ne = gg16-ssd --da ase - ype= oc --p e ained-ssd=mo
dels/ gg16-ssd-mp-0_7726.p h --da a=da a/ --model-di =models/ gg16/ --ba ch
-size=30 --wo ke s 1 --lea ning- a e=0.001 --base-ne -l =0.001 --epochs 100
30
The command uns he aining sc ip o he objec de ec ion model using he a chi ec u e wi h he
VGG16 backbone. The model is se o use a ba ch size o 30, a lea ning a e o 0.001, and is ained o
100 epochs.
I speci ied he use o a single wo ke o aining wi h he ‘--wo ke s 1’. This a gumen is used o speci y
he numbe o wo ke h eads o use o da a loading, which can help speed up he aining p ocess
by loading da a in pa allel wi h he aining p ocess.
The nex s ep is he e alua ion o he objec de ec ion model wi h he e al_ssd.py sc ip . The
e alua ion sc ip will use he speci ied SSD a chi ec u e ( gg16-ssd), he ained model and he da ase
o e alua e he model's p ecision ac oss all classes.
py hon3 e al_ssd.py --ne gg16-ssd --da ase da a/ -- ained_model models/
gg16/ gg16-ssd-Epoch-93-Loss-2.5332502018321645.p h --label_ ile models/ l
abels. x
I he model has an accep able p ecision, i is expo ed wi h he onnx_expo .py sc ip .
py hon3 onnx_expo .py --model-di =models/ gg16
Then we un he ial implemen a ion wi h he un_ssd_example.py sc ip .
py hon3 un_ssd_example_WY.py gg16-ssd models/ gg16/ gg16-ssd-Epoch-93-Los
s-2.5332502018321645.p h models/ gg16/labels. x da a/JPEGImages
I he e is con i ma ion o co ec de ec ions in he es images, he model mo es o he in e ence
p ocess.
4.1.4. Limi a ions
This solu ion wo ks and has posi i e esul s, bu i also has limi a ions ha can become
liabili ies o his p ojec . The main p oblem he e was ha he ime i akes an immense amoun o
ime o ully un he ain_ssd.py sc ip . Each ain a emp wi h a da ase o app oxima ely 30
housand anno a ions ook be ween 15 and 16 hou s.
Figu e 17 – ain_ssd.py unning log
31
Tha ime was no ideal since each hou o VM usage has i s associa ed cos , and hose cos s we e o e
he budge , which aised a inancial liabili y. Wi h his code, we we e only able o ain ou model wi h
one CUDA ca d a a ime, bu since ou VM machine has ou a ailable GPU ca ds he e is a possibili y
o using mul iple ca ds o un he same sc ip .
My challenge he e is o change he wo k low and enable he mul i-GPU model aining. Tha will be
accomplished wi h he NVIDIA T ans e Lea ning Toolki since i is compa ible wi h ou ha dwa e, has
a lo o a chi ec u al amewo ks lexibili y, and has a mul i-GPU selec ion ea u e which in heo y will
accele a e he whole p ocess.
4.2. TRANSFER LEARNING TOOLKIT DEMO
The i s s ep o implemen he NVIDIA T ans e Lea ning Toolki is o ins all and con igu e an
NGC Ca alog accoun (NVIDIA GPU Cloud). This ca alog is a cu a ed se o GPU-op imized so wa e o
AI, HPC and Visualiza ion.
NGC o e s a di e se collec ion o con aine s, including deep lea ning amewo ks, ha bundle
so wa e applica ions, lib a ies, dependencies, and un- ime compile s in a sel -con ained
en i onmen o seamless deploymen ac oss di e en compu ing en i onmen s, acili a ing so wa e
po abili y and enabling easy scaling o applica ions ac oss he cloud, da a cen e , and edge
en i onmen s wi h a single command [16].
Then I had o c ea e an NGC accoun in h ps://ca alog.ngc.n idia.com/ and ge an API $KEY ha i can
be gene a ed in he accoun op ions.
Ha ing he p e ious s ep concluded, I am able o e ch and s a any NGC con aine . In he TLT
collec ion, he e’s a de ailed o e iew ha goes h ough o wha i is, how i wo ks, whe e o ge
s a ed. A e s udying he TLT a ailable op ions, I se led o he T ans e Lea ning Toolki s Fo
S eaming Video Analy ics con aine .
The nex s ep was o e ch he con aine om NGC on o he GPU ins ance, and ha is done by unning
a p o ided docke command on he VM e minal.
Figu e 18 – TLT page on he NGC ca alog o ideo s eaming [16]
38
bbox_ as e ize _con ig {
a ge _class_con ig {
key: " balacla a "
alue {
co _cen e _x: 0.5
co _cen e _y: 0.5
co _ adius_x: 0.40000000596
co _ adius_y: 0.40000000596
bbox_min_ adius: 1.0
}
}
a ge _class_con ig {
key: " kni e "
alue {
co _cen e _x: 0.5
co _cen e _y: 0.5
co _ adius_x: 1.0
co _ adius_y: 1.0
bbox_min_ adius: 1.0
}
}
Ha ing hese changes lis ed and unde s ood, e e y hing was eady o un he l - ain ea u e.
l - ain de ec ne _ 2 -e $SPECS_DIR/de ec ne _ 2_ ain_ esne 18_ki i. x
- model_unp uned
-k $KEY
-n esne 18_de ec o
--gpus $NUM_GPUS
A e he ain is comple ed, I can e alua e he model’s accu acy on each class.
l -e alua e de ec ne _ 2 -e $SPECS_DIR/de ec ne _ 2_ ain_ esne 18_ki i.
x
-m model_unp uned/weigh s/ esne 18_de ec o . l
-k $KEY
The l -e alua e de ec ne _ 2 command is used o e alua e a ained model using he de ec ne _ 2
a chi ec u e. The -e lag is speci ying he pa h o he e alua ion con igu a ion ile, is he same used o
he aining p ocess, he -m lag is speci ying he pa h o he ained model, and he -k lag is he access
key which has o be he same used o he ain p ocess.
As o his s ep, I ha e a ained model and i s de ec ion accu acy. I al eady can be expo ed o es ing
pu poses, bu be o e ha , he e is he op ional s ep o p une he model. Model p uning is he p ocess
o emo ing unnecessa y pa ame e s om a ained model in o de o educe i s size and
compu a ional equi emen s. P uning a model wi h TLT is done using he p une command, bu i s I
mus c ea e a new di ec o y o s o e he p uned model:
mkdi -p model_p uned

39
And now he TLT model p une command:
l -p une -m $USER_EXPERIMENT_DIR/model_unp uned/weigh s/ esne 18_de ec o .
l
-o $USER_EXPERIMENT_DIR/model_p uned/ esne 18_nopool_bn_de ec ne
_ 2_p uned. l
-eq union
-p h 0.8
-k $KEY
He e, he -eq lag is speci ying he p uning me hod: in his case, i is 'union', a me hod ha emo es
he weigh s wi h he lowes absolu e. The -p h lag is speci ying he p uning h eshold: in his case he
sugges ion is o use 0.8, which means ha only he weigh s ha a e below 0.8 will be p uned bu la e
I can adjus o accu acy and model size ade o . A highe p h gi es highe in e ence speed bu wo se
accu acy.
Now, he p uned model mus be e ained.
l - ain de ec ne _ 2 -e $SPECS_DIR/de ec ne _ 2_ e ain_ esne 18_ki i. x
- $USER_EXPERIMENT_DIR/model_ e ain
-k $KEY
-n esne 18_de ec o _p uned
--gpus $NUM_GPUS
This command is like he p e ious one, bu o un he e ain he e’s he need o speci y a di e en
speci ica ions ile, which is e y simila o he ain ile. The --gpus and -k inpu mus be he same used
in he ain p ocess.
When he e ain p ocess is inished, I can e-e alua e he model.
l -e alua e de ec ne _ 2 e $SPECS_DIR/de ec ne _ 2_ e ain_ esne 18_ki i.
x
-m $USER_EXPERIMENT_DIR/model_ e ain/weigh s/ e
sne 18_de ec o _p uned. l
-k $KEY
A e concluding hese s eps, I ha e a . l ile, and o es i i p ope ly de ec s he in ended classes I
can un he TLT in e ence ea u e in a da a sample wi h a leas 8 images and see i he e is any
de ec ion.
l -in e de ec ne _ 2 -e $SPECS_DIR/de ec ne _ 2_in e ence_ki i_ l . x
-o $USER_EXPERIMENT_DIR/ l _in e _ es ing
-i $DATA_DOWNLOAD_DIR/ es _images
-k $KEY
40
This command will pe o m in e ence on a ained model using he . l ile on he inpu images loca ed
in he di ec o y speci ied by he -i lag and sa e he in e ence ou pu o he di ec o y speci ied by he
-o lag. Same as be o e, he speci ica ions ile is al eady se o wo k wi h his da a and hese classes,
bu looking a i I could iden i y ha besides he class names I also will ha e o change he in e ence
dimensions o ma ch wha he model was ained o .
# In e ence dimensions.
image_wid h: 960
image_heigh : 544
Looking in o he l _in e _ es ing iles, I con i med ha he model was de ec ing bo h classes p ope ly.
Figu e 20 – TLT in e ence wi h mask/no mask demo model [18]
Finally he nex and las s ep is o expo he ile o .e l o ma . In summa y, he . l o ma is used o
s o e he model's a chi ec u e, weigh s, and o he me ada a and i can be used o bo h aining and
in e ence while he .e l o ma is an op imized o ma o deploymen and in e ence and i can be
used only o in e ence.
l -expo de ec ne _ 2
-m expe imen _di _ e ain/weigh s/ esne 18_de ec o _p uned. l
-o expe imen _di _ inal_p uned/ esne 18_de ec o _p uned.e l
-k $KEY
41
Now he model is eady o deploymen , and I can mo e on o he nex phase and apply hese TLT
ea u es wi h ou da a and de ec a di e en se o classes.
4.3. DATA PREPARATION
4.3.1. KITTI Con e sion
This is he inal s ep o ully in eg a e ou da a wi h he TLT ea u es. The TLT ain p ocess only
wo ks i he inpu da a is in KITTI o ma , which is no possible o expo di ec ly om CVAT, so I need
o expo he da a in VOC PASCAL which gi es me he anno a ions iles in .xml, and hen con e o
KITTI wi h a sc ip .
To ul ill his ask, I cloned a gi hub eposi o y ( h ps://gi hub.com/k us nic/xml2ki i ) wi h ha same
pu pose and c ea ed some ex a ea u es o cus omize he code acco ding o ou necessi ies.
Ano he TLT ain speci ica ion is ha all images in he da ase ha e o be he same size, so i in he
con igu a ion ile I se he wid h o 960 and he heigh o 544, all images mus ha e ha exac same
dimensions.
pa h = '…/JPEGImages'
c=0
o ile in os.lis di (pa h):
_img = +"/"+ ile
img = Image.open( _img)
img = img. esize((heigh ,wid h))
img.sa e( _img)
c+=1
p in ( '{c} esized images')
This code uses he Py hon imaging lib a y (PIL) o esize he se o images loca ed in a di ec o y. The
i s line assigns he pa h o he di ec o y con aining he images o he a iable ‘pa h’ hen i ini ializes
a a iable ‘c’ o ze o, which will be used o keep ack o he numbe o esized images.
In he o loop, he ‘os.lis di (pa h)’ unc ion is used o lis all he iles in he di ec o y speci ied by he
’pa h’ a iable. The loop i e a es h ough each ile in he di ec o y. Fo each i e a ion, he code opens
he ile using he PIL's ‘Image.open()’ unc ion and assigns i o he a iable 'img'. Then i uses he
‘. esize()’ me hod on he 'img' a iable o esize he image o he gi en 'heigh ' and 'wid h' size. Then
i sa es he esized image o he same ile using he ‘.sa e()’ me hod.
Finally, he 'c' a iable is inc emen ed by 1 o each i e a ion o he loop, o keep ack o he numbe
o esized images. A he end o he loop, he code p in s he o al numbe o esized images.
The xml2ki i.py success ully con e ed mos o he iles. The ou pu was sen in o he same olde o
he inpu and he e we e some con e sion excep ions, aising he issue o ha ing mo e images han
KITTI anno a ions, which mus be he same numbe . To sol e his p oblem i s I added a ile mo ing
42
unc ion o mo e he ou pu o ano he di ec o y ('.../Anno a ions_ki i') and a ile emo e unc ion
o emo e he images wi hou a co esponding KITTI anno a ion.
des _di = '.../Anno a ions_ki i'
o ile in glob.glob('.../Anno a ions/*. x '):
shu il.mo e( ile, des _di )
In his code snippe I use he py hon 'glob' and 'shu il' lib a ies o mo e a se o iles om one di ec o y
o ano he . The i s line assigns he pa h o he des ina ion di ec o y o he a iable 'des _di '. The o
loop uses he 'glob' lib a y unc ion 'glob.glob()' o sea ch o all he iles in he di ec o y
'.../Anno a ions/' ha ha e he '. x ' ex ension and hen i i e a es h ough each ile ound. Fo each
ile, he code uses he 'shu il' lib a y unc ion 'shu il.mo e()' o mo e he ile om i s cu en loca ion
o he des ina ion di ec o y speci ied by he 'des _di ' a iable.
ki i_lis =[]
c=0
o ile2 in os.lis di (".../Anno a ions_ki i"):
name2 = ile2. spli ('.',1)[0]
ki i_lis .append(name2)
o ile in os.lis di (".../JPEGImages"):
name1 = ile. spli ('.',1)[0]
ilename = ".../JPEGImages/"+ ile
i name1 no in ki i_lis :
os. emo e( ilename)
c+=1
p in ( 'Remo ed-------{ ile}')
p in ( 'Remo ed {c} iles')
He e I use py hon's os lib a y o emo e a se o iles om a di ec o y based on a compa ison wi h he
iles in ano he di ec o y. The a iable 'c' will be used o keep ack o he numbe o emo ed iles,
and he 'ki i_lis ' will ha e he names o all he KITTI iles, which a e added in he i s loop using he
‘os.lis di ()’ unc ion o lis all he iles in ".../Anno a ions_ki i" and i e a es h ough each ile in he
di ec o y. Fo each ile, I apply he ‘. spli ()’ unc ion o emo e he ile ex ension and append he name
o he a iable 'ki i_lis '. The second o loop uses he ‘os.lis di ()’ unc ion o lis all he iles in he
image di ec o y (".../JPEGImages") and i e a es h ough each ile in he di ec o y. Fo each ile, I again
apply he ‘. spli ()’ unc ion o emo e he ile ex ension and assign he name o he ile o he 'name1'
a iable o hen compa e i o he names in he ‘ki i_lis ’, i is no ound I apply he ‘os. emo e()’ o
dele e he image ile and inc emen he ‘c’ a iable by 1.
By he end, o apply hese changes I jus need o un he ollowing in he p omp .
py hon3 xml2ki i.py .../Anno a ions
43
4.3.2. Anno a ions and Folde S uc u e
The i s s ep o his segmen is he labeling p ocess in CVAT. We s a by unning he docke
CVAT image and access h ough he h p://localhos :8080/ on google ch ome. Nex I ha e o c ea e a
new ask, name i , add he labels ha will be he classes in he expo ed anno a ions, upload he
images o be labeled and manually selec in he image he objec s associa ed o he labels.
Figu e 21 – CVAT Anno a ion example
A e inishing he label p ocess in all images he e is an expo op ion o dump all he anno a ions in
PASCAL VOC o ma in o a olde . Tha olde will be he one used o con e he iles o KITTI and
impo he da a o he model aining, bu he e a e some necessa y changes o ha e he olde in
he igh s uc u e.
The dumped olde is composed by he ollowing:
• Labels ‘. x ’ ile
• ‘Anno a ions’ olde
• ‘ImageSe s’ olde
Fo he desi ed s uc u e I mus do hese changes:
• Inse he images olde wi h he name ‘JPEGImages’.
• Rename he labels ile o ‘labels’.
• In he labels ile, dele e e e y hing excep he classes names and I also need o make su e
ha he e a e no any emp y lines.
• Dele e he ‘ImageSe s’ olde .
• C ea e an emp y olde named ‘Anno a ions_ki i’.
By he end o hese changes he desi ed olde s uc u e should look like his.

44
|---Anno a ions
| image_name.xml
| ...........
| ...........
|
|---Anno a ions_ki i
| (emp y)
|
|---JPEGImages
| image_name.jpeg
| ...........
| ...........
|
|---labels. x
4.4. TRAINING
Wi h he anno a ions in KITTI o ma I inally can use he TLT ea u es. The i s da a ba ch is
composed o close o 30000 anno a ions o wo classes, balacla a, and kni e. The main goal he e is o
apply he ain ea u e and gua an ee a lowe aining ime wi h he mul iple gpus op ion and s ill
main ain an accep able p ecision (>70%).
Be o e he whole p ocess, download he l _specs olde om he NVIDIA demo o my local machine
and adjus he con igu a ion iles as lis ed abo e.
The docke image di ec o y name will be " ain_ l ", so he da a sou ces change mus be acco ding o
ha , se he a ge classes names o “balacla a” and “kni e”, selec he ou pu image wid h and heigh
o 1280 and 720 espec i ely ( he esolu ion o he inpu images) and o educe he ain ime I can se
numbe o aining epochs o 100.
Ha ing done hese changes on all iles I can s a he p ocess s eps acco ding o he NVIDIA demo.
Fi s , pull he docke TLT con aine .
docke pull n c .io/n idia/ l -s eamanaly ics: 2.0_py3
Run he docke image wi h he name “ ain_ l ”.
docke un --gpus all -i - "/pa h/ o/di /on/hos /":" ain_ l "
-p 8888:8888 n c .io/n idia/ l -s eamanaly ics: 2.0_py3 /bin/bash
The pa h on hos we should apply is he one whe e he TLT speci ica ions iles and he anno a ions a e.
Fo his i s un I am going o selec he same a chi ec u e amewo k used in he demo, he
De ec ne _ 2 and he p e ained model will be he Resne 18. La e , when he p ocess is ully wo king,
I can es ano he amewo ks and models and une i o op imize he de ec ion.
45
Wi h he docke image c ea ed and he necessa y da a he e, I mus con e he eco ds om he
KITTI anno a ions.
l -da ase -con e -d l _specs/de ec ne _ 2_ eco ds_ki i_ ain al. x -
o / ain_ l / eco ds/ki i_ ain al/
A e con i ming ha he eco ds a e in he ou pu di ec o y, I need o c ea e a a ge des ina ion
olde and download he p e ained model.
mkdi p e ained_ esne 18
ngc egis y model download- e sion n idia/ l _p e ained_de ec ne _ 2: esn
e 18
--des p e ained_ esne 18
Nex , open he de ec ne _ 2_ ain_ esne 18_ki i. x o con i m ha e e y hing is co ec ly se o un
he TLT ain ea u e. A e gua an ee ha he speci ica ions a e co ec , I can ain he model wi h my
da a. Fi s , I am going o use 2 gpus and compa e he o al aining ime wi h he pe o mance I go
using jus 1. The p e ious ain ime was a ound 15h-16h.
l - ain de ec ne _ 2 -e l _specs/de ec ne _ 2_ ain_ esne 18_ki i. x
- model_unp uned
-k $KEY
-n esne 18_de ec o
--gpus 2
Figu e 22 – TLT unning log
As obse ed in he igu e abo e, he ime i ook o comple e he model aining was 5 hou s and 22
minu es, which educes by 10 hou s he model ain ime and ul ills he main goal o his
implemen a ion. Now I will di ec ly p une and e ain he model and a e ha e alua e and y o es
he in e ence on some es images.
I will c ea e a di e en di ec o y o he p uned model and hen un he l -p une ea u e.
mkdi model_p uned
l -p une -m model_unp uned/weigh s/ esne 18_de ec o . l
-o model_p uned/ esne 18_nopool_bn_de ec ne _ 2_p uned. l
-eq union
-p h 0.01
-k $KEY
46
A e he model p une he model mus be e ained.
l - ain de ec ne _ 2 -e l _specs/de ec ne _ 2_ e ain_ esne 18_ki i. x
- model_ e ain
-k $KEY
-n esne 18_de ec o _p uned
--gpus 2
Again, he ime o e ain is a ound he 5 hou s ma k. Now I un he e alua ion and check he model’s
de ec ion quali y.
l -e alua e de ec ne _ 2 -e l _specs/de ec ne _ 2_ e ain_ esne 18_ki i.
x
-m model_ e ain/weigh s/ esne 18_de ec o _p uned. l
-k $KEY
Figu e 23 – T ained model 2 class p ecision
The esul s a e e y p omising, we can obse e ha he balacla a classes de ec ion p ecision is e y
good and well abo e he 70%, which is no he case o he kni e de ec ion which is qui e
unde s andable since a kni e is much mo e di icul o de ec .
Al hough he kni e p ecision is no ideal, is e y close o he minimum necessa y and he p ecision
ac oss all classes mee s he equi emen s, so I can p oceed o he in e ence es phase and depending
on he esul s I decide i his model will be expo ed and es ed in loco o i i needs o be imp o ed
immedia ely.
Wi h his hough , I will un he in e ence command o gene a e de ec ions on some andom images
ha I ha e pu oge he and impo ed o he docke image.
l -in e de ec ne _ 2 -e ain_ l /de ec ne _ 2_in e ence_ki i_ l . x
-o l _in e _ es ing
-i es _images
-k $KEY
Looking o he ou pu , I con i m ha he esul s a e sa is ac o y and in ac de ec ion is unning
acco dingly he expec a ions.
47
Figu e 24 – Tl in e ence es
Has obse ed he alues associa ed o he de ec ions a e acco ding o he l -e alua ion esul s. Since
hese esul s demons a e ha he model can in ac de ec I will expo and sha e wi h he local eam
o es i in he clien ’s p emises. To do so, I apply he expo command.
mkdi -p model_ inal_p uned
l -expo de ec ne _ 2
-m model_ e ain/weigh s/ esne 18_de ec o _p uned. l
-o model_ inal_p uned/ esne 18_de ec o _p uned.e l
-k $KEY
The .e l ile is he inal p oduc o he whole p ocess and is wha I will sha e wi h he in e ence eam.
54