i
Implemen a ion o Deep Lea ning models o
In o ma ion Ex ac ion on Iden i ica ion
Documen s
Hen ique Edua do Espadinha Renda
In e nship a Biome id
In e nship epo p esen ed as pa ial equi emen o
ob aining he Mas e ’s deg ee P og am in Da a Science and
Ad anced Analy ics
i
Ti le: Implemen a ion o Deep Lea ning models o In o ma ion
Ex ac ion on Iden i ica ion Documen s
In e nship a Al ice Po ugal
Hen ique Edua do Espadinha Renda
MAA
2023
i
ii
NOVA In o ma ion Managemen School
Ins i u o Supe io de Es a ís ica e Ges ão de In o mação
Uni e sidade No a de Lisboa
IMPLEMENTATION OF DEEP LEARNING MODELS FOR INFORMATION
EXTRACTION ON IDENTIFICATION DOCUMENTS
by
Hen ique Renda
In e nship epo p esen ed as pa ial equi emen o ob aining he Mas e ’s deg ee in Da a Science
and Ad anced Analy ics
Ad iso : P o Dou o Robe o Hen iques
Co Ad iso : Lucas Soa es
iii
STATEMENT OF INTEGRITY
As a membe o he Biome id eam and mas e ´s s uden a NOVA In o ma ion Managemen School,
I pledge o uphold he highes s anda ds o in eg i y and e hical conduc in he comple ion o my
in e nship epo . This includes ensu ing he accu acy and u h ulness o all in o ma ion p esen ed,
p ope ly ci ing any sou ces used, and e aining om plagia ism o any o he o m o academic
dishones y. I will also main ain con iden iali y and espec any p i acy o sensi i e in o ma ion ha
may be sha ed du ing he cou se o my in e nship. I unde s and ha any iola ion o hese p inciples
may esul in disciplina y ac ion, and I am ully commi ed o conduc ing mysel in an hones and
us wo hy manne h oughou my in e nship.
Lisbon, Feb ua y 2023

i
Feb ua y 2023
DEDICATION
I dedica e his epo o my amily and iends, who ha e been a cons an sou ce o inspi a ion and
mo i a ion. In pa icula , I would like o exp ess my deepes app ecia ion o my pa en s and
gi l iend o hei unwa e ing suppo , encou agemen , and mo i a ion ha ha e been ins umen al
in helping me pe se e e h ough he challenges and celeb a e he iumphs. Addi ionally, I ex end
my g a i ude o my men o s o p o iding in aluable guidance and suppo h oughou he esea ch
p ocess. This epo ep esen s he culmina ion o yea s o ha d wo k, and I am hono ed o sha e i
wi h hose who ha e played a signi ican ole in shaping my academic and pe sonal g ow h.
i
ABSTRACT
The de elopmen o objec de ec ion models has e olu ionized he analysis o pe sonal in o ma ion
on iden i ica ion ca ds, leading o a dec ease in ex e nal human labo . Al hough p e ious s a egies
ha e been employed o add ess his issue wi hou using machine lea ning models, hey all p esen
ce ain limi a ions, which a i icial in elligence aims o o e come. This epo del es in o he
de elopmen o a deep lea ning-based objec de ec ion capable o ecognizing ele an in o ma ion
om Po uguese iden i ica ion ca ds. All he decisions made du ing he p ojec will be accompanied
by a de ailed backg ound heo y. Addi ionally, we p o ide an in-dep h analysis o Op ical Cha ac e
Recogni ion (OCR) echnology, which was u ilized h oughou he p ojec o gene a e ex om
images. As he newes membe o he Machine lea ning Team o Biome id, I had he p i ilege o
being in ol ed in his p ojec ha led o he imp o emen o he cu en app oach ha does no
le e age machine lea ning in he de ec ion o ele an sec ions om ID ca ds. The indings o his
p ojec p o ide a ounda ion o u he esea ch in o he use o AI in iden i ica ion ca d analysis.
KEYWORDS
A i icial In elligence; Machine Lea ning; A i icial Neu al Ne wo ks; Compu e Vision; Objec
De ec ion Models; Op ical Cha ac e Recogni ion (OCR)
ii
INDEX
1. In oduc ion ................................................................................................................ 1
1.1. Company O e iew .............................................................................................. 2
1.2. P oblem De ini ion ............................................................................................... 3
1.2.1. Case S udy ..................................................................................................... 4
1.2.2. Cons ain s and Limi a ions .......................................................................... 4
1.2.3. P oposed Solu ion ......................................................................................... 5
2. Theo e ical F amewo k ............................................................................................... 6
2.1. Machine Lea ning ................................................................................................. 6
2.1.1. Supe ised Lea ning ...................................................................................... 9
2.2. A i icial Neu al Ne wo ks & CNNs ....................................................................... 9
2.2.1. Con olu ion Laye s ...................................................................................... 10
2.2.2. Pooling Laye s ............................................................................................. 11
2.2.3. Fully Connec ed Laye s ............................................................................... 12
2.3. Deep lea ning ..................................................................................................... 13
2.4. Objec De ec ion ................................................................................................ 14
2.4.1. Region-based Con olu ion Neu al Ne wo ks (R-CNN) ................................ 15
2.4.2. Fas e -RCNN ................................................................................................ 16
2.4.3. Single Sho Mul iBox De ec o (SSD) ........................................................... 17
2.4.4. Cen e Ne .................................................................................................... 18
2.5. E alua ion Me ics ............................................................................................. 19
2.5.1. Con usion Ma ix ......................................................................................... 20
2.5.2. In e sec ion o e Union (IoU) ..................................................................... 21
2.5.3. P ecision and Recall .................................................................................... 22
2.5.4. F1-Sco e: ..................................................................................................... 22
2.5.5. A e age P ecision ........................................................................................ 22
2.5.6. Mean A e age P ecision (mAP) ................................................................... 23
2.6. Op ical Cha ac e Recogni ion (OCR) ................................................................. 23
3. Me hodology ............................................................................................................. 26
3.1. Tools and Technologies ...................................................................................... 26
3.1.1. Tenso Flow .................................................................................................. 26
3.1.2. Label S udio ................................................................................................. 27
3.1.3. OpenCV ....................................................................................................... 28
1
1. INTRODUCTION
Wi h he echnological ad ances egis e ed in ecen yea s, i can be acknowledged ha we a e
cu en ly in an e a whe e da a is being p oduced a an unp eceden ed a e, compelling businesses o
ely on au oma ed me hods o da a analysis. Re ie ing use ul insigh s om his amoun o da a is
c ucial, so in olun a ily companies in his si ua ion no ice ha i is manda o y o de elop ad anced
algo i hms ha can summa ize, classi y, ex ac impo an in o ma ion, and con e hem in o an
unde s andable o m (V. Y., Ma iano e al., 2002).
To add ess his issue, he e has been a huge inc ease in he usage o A i icial In elligence (AI)
algo i hms o guide business decisions. Machine lea ning (ML) models and A i icial Neu al Ne wo ks
(ANN), in pa icula , ha e demons a ed ou s anding pe o mance in esol ing complex challenges
ac oss a wide ange o indus ies, e ec i ely e olu ionizing he way companies app oach p oblem-
sol ing (A. Zulkhaiza , 2023).
Fo example, in compu e ision asks like objec de ec ion and seman ic segmen a ion, deep
lea ning models ha e been success ully used in applica ions such as au onomous d i ing. This
echnology is apidly ad ancing, b inging sys ems close o possessing a le el o in elligence
compa able o humans in ce ain si ua ions. Acco ding o he cu en N idia CEO, Jensen Huang,
“Deep Lea ning can ain a ca o d i e, and ul ima ely pe o m a be e , and mo e sa ely, han any
human d i e could do behind he wheel.”. This was he goal o Huang’s company when de eloping
ha dwa e o he upcoming au onomous ca s (J. Huang, 2017).
Figu e 1 - Usage o Deep Lea ning models in au onomous ca s
1
In he example abo e, objec de ec ion models o e a solu ion o iden i ying ele an i ems in he
oad ha allow he ehicle o d i e sa ely. This abili y ende s hem highly e icien o ac i i ies such
as pe cep ion, decision-making, and ehicle con ol. Ne e heless, hei use ulness ex ends beyond
his use case scena io, as hey ha e he capabili y o analyze and e ie e in o ma ion om a gi en
ame.
As ano he example, h oughou his epo , we’ll highligh he expe ience o a Junio Machine
Lea ning Enginee esponsible o a p ojec ha led o he imp o emen o Biome id’s machine
lea ning in as uc u e behind i s OCR solu ion o in o ma ion e ie al o Po uguese iden i ica ion
1
Sou ce: h ps://www.cbinsigh s.com/ esea ch/s a ups-d i e-au o-indus y-dis up ion/

2
ca ds. The esponsibili ies ega ding he collec ion, p ocessing, and analysis o in o ma ion will be
desc ibed in de ail.
We’ll s a by in oducing he p oblem add essed and cla i y he challenges aced, ega ding he
de elopmen o an objec de ec ion model capable o ecognizing ele an sec ions om he
Po uguese iden i ica ion ca ds.
Nex , we’ll do a concise explana ion o how he company is composed alongside i s objec i e, be o e
del ing in o mo e speci ic and echnical aspec s ega ding he heo y behind he p ojec . Finally, his
epo will end up by clea ly exposing all he p ac ical implemen a ions o he p ojec alongside he
inal esul s.
1.1. COMPANY OVERVIEW
Figu e 2 - Biome id's Logo
2
C ea ed in 2015 unde he name Polygon, he cu en Biome id eme ged as an IT company wi h
headqua e s es ablished in Po o ha p omised o e olu ionize he way we communica e wi h
digi al sys ems. A a ime when wi h one click i is possible o open a bank accoun , egis e on a
social ne wo k o e en subsc ibe o a se ice i is no su p ise ha he digi al oo p in o all in e ne
use s has been g owing exponen ially. Wi h ha in mind, Biome id is commi ed o deli e ing a
ange o ools ha acili a e he onboa ding, au hen ica ion, and alida ion o pe sonnel in
companies o all sizes.
Figu e 3 - Biome id's Miles ones
Biome id o e s a ange o solu ions o simpli y he Know You Cus ome (KYC) p ocesses. The
company's lagship p oduc , which sha es he same name, s eamlines hese ime-consuming
ac i i ies, which a e essen ial o sa egua d inancial ins i u ions agains aud, co up ion, money
launde ing, and e o ism inancing.
2
Sou ce: h ps://biome id.com/
3
Compa ed o i s main compe i o s, Biome id's p oduc akes a unique app oach. I uses a
D ag&D op me hodology, which p o ides cus ome s wi h au onomy and lexibili y o build
pe sonalized e i ica ion p ocesses. The p oduc enables use s o open bank accoun s emo ely,
e i y P oo -o -Li e using pe sonal documen s, and gene a e digi al signa u es o public se ices.
The company has es ablished pa ne ships wi h banking ins i u ions, insu ance companies, and
Po uguese Go e nmen depa men s. I has also expanded i s ope a ions o in e na ional ma ke s,
including Poland, Hunga y, Colombia, Angola, and Mozambique. Biome id's goal is o con inue
expanding i s se ices and each in o new sec o s and egions wo ldwide.
Fo he ML eam he main esponsibili y is o c ea e and main ain all he ML models ha suppo i s
solu ions, especially he Op ical Cha ac e Recogni ion (OCR) p oduc ion pipeline o e ie ing
in o ma ion om iden i ica ion ca ds. The ocus is di ided in o wo: on one side imp o ing he OCR
pipeline, which would ecei e images o ID ca ds o passpo s, and om he e ex ac in o ma ion
ha could iden i y use s in he u u e. On he o he side, esea ching and de eloping new ML ools
o he OCR in as uc u e.
1.2. PROBLEM DEFINITION
As men ioned be o e, sys ems equipped wi h a i icial in elligence algo i hms a e now mo e p esen
han e e in ou daily li es. This pa adigm shi means ha p ocesses ha equi ed a lo o manual
wo k a e now being eplaced by models ha simula e human beha io .
One o he se e al examples whe e his change is no iceable is in he egis a ion p ocess. In he
pas , ega dless o he indus y, o pe o m he onboa ding o a new clien his one needed o ill ou
a o m wi h his pe sonal in o ma ion manually and wai o he esponsible s a o ans e he
in o ma ion in o a dedica ed da abase. This implied a a he leng hy p ocess wi h se e al dedica ed
pa s and a complex in as uc u e. Luckily, o s eamline he p ocess, a new me hod has eme ged
o ex ac ing his in o ma ion wi hou equi ing manual en y om ei he he cus ome o s a (M.
Ryan, & N. Hana iah., 2015).
A new sys em using op ical cha ac e ecogni ion (OCR) has been p oposed o ex ac cus ome
in o ma ion om iden i ica ion ca ds ins ead o manual da a en y. This sys em, in he i s ins ance,
is powe ed by an objec de ec ion model esponsible o p ocessing an ID ca d image and e ie ing
he coo dina es o he sec ions ela ed o pe sonal da a as he esul . These image coo dina es a e
hen used o c op he o iginal image and eed hose ou pu s in o an OCR applica ion ha ecognizes
he ex in he pic u e and con e s i in o machine- eadable and edi able ex .
4
1.2.1. Case S udy
This chap e will examine he applica ion o he selec ed app oach in he Biome id OCR pipeline o
au oma e he p ocess o e ie ing pe sonal in o ma ion om Po uguese ID ca ds. By le e aging
cu ing-edge op ical cha ac e ecogni ion echnology, he pipeline aims o s eamline he da a inpu
p ocess and p o ide accu a e and e icien esul s. This subsec ion will del e in o he de ails o his
s a egy and i s impac on he p ocess o e ie ing pe sonal in o ma ion om ID ca ds.
The g ow h o membe s o he ML eam led o a new p ojec which has he objec i e o imp o ing
he way he OCR da a pipeline analyses images. Cu en ly, he ex ac ion p ocess con ains se e al
laye s o p e-p ocessing:
1. The documen s a e exposed o a o a ion model ha o ien s hem ho izon ally in an
au oma ic way.
2. Nex , his esul ing image will se e as inpu o a segmen a ion model, which aims o
emo e all su ounding elemen s om he pho o and gene a e c opped images.
3. These images a e hen con e ed om an RGB Fo ma o G ayscale o imp o e he
pe o mance o he bina iza ion ha will be applied o he image la ely.
4. A e mul iple mo phological image ans o ma ions, he coo dina es om he egions o
in e es we e de ec ed by mul iple adi ional image p ocessing echniques ha use ela i e
coo dina es o cap u e in o ma ional sec ions.
5. The p ocess ends jus when all o he p oposed a eas a e analyzed by an OCR engine.
This s a egy p esen s i sel as a e y good al e na i e compa ed o a ull-manual p ocess, howe e ,
i ’s in many ways limi ed as i s esul s a e only iable in a o able condi ions whe e he documen
has a cap u e angle and a o able ligh condi ions.
As a company ha consis en ly emb aces new echnologies, Biome id ex ends his philosophy o
e e y aspec o i s p oduc . So, he main goal o his p ojec is o imp o e he OCR Pipeline by
c ea ing an a i icial neu al ne wo k capable o de ec ing all sec ions o he on o he Po uguese
iden i ica ion ca d. This way we hope o enhance he gene al pe o mance o he OCR p ocesses and
es ablish a s a egy ha deli e s a o able ou comes in all image scena ios.
1.2.2. Cons ain s and Limi a ions
In he ealm o came a-based analysis o ex and documen s, he e exis se e al challenges
associa ed wi h cap u ing images, including low esolu ion, une en ligh ing, dis o ed pe spec i es,
non-plana su aces, wide-angle lens dis o ion, clu e ed backg ounds, di icul ies wi h zooming and
ocusing (M. Ryan, & N. Hana iah., 2015). Th oughou he epo , we will explain all he s a egies
and echniques ha we will apply o sol e some o hose p oblems.
Besides ha , we also ind some cons ain s ela ed o he da ase con aining images o Po uguese
Iden i ica ion Ca ds. This is no publicly a ailable due o GDPR es ic ions, bu o una ely, Biome id
had al eady ob ained his ype o da a h ough a pa ne ship wi h he “Agência pa a a Mode nização
Adminis a i a” (AMA) and egula ly u ilized i o o he p ojec s. So, e en wi h limi ed da a we s ill
5
manage o o ganize and cu a e app oxima ely 3000 o hese images o cons uc a consis en da ase
capable o success ully aining he model. Impo an o men ion ha hese images con ain pe sonal
in o ma ion, and hei public sha e is exp essly o bidden.
1.2.3. P oposed Solu ion
To add ess he case s udy, always wi h he limi a ions and cons ain s in conside a ion, i was
decided by he eam ha he bes s a egy o imp o e he de ec ion o he sec ions o he on ace
o he Po uguese iden i ica ion ca d would go h ough he aining o a deep neu al ne wo k capable
o imp o ing he p e ious esul s and add ess si ua ions whe e he image quali y is no a o able.
P e ious s udies sugges ha a ious me hods ha e been employed o ackle his issue. One
commonly employed app oach in ol es iden i ying saliencies in he image o loca e documen s
inside a pho o o ideo ame, wi hou any p io knowledge abou he documen (F. A i issimo e
al., 2019). Howe e , his s a egy s ill p esen s some limi a ions mainly when he image con ains
noise, o he angle o cap u e is no a o able. The imp o ed solu ion o he Biome id OCR ha will
be p esen ed h oughou his epo p esen s i sel as a mo e ad anced app oach o his p oblem. I
s a s by using a de ec ion model o loca e he documen , hen c ops he image o isola e he
documen om he backg ound. A e p ope segmen a ion, he documen is classi ied using a deep
lea ning model.
Th oughou his documen , we aim o enhance he a e age success a e o de ec ing ele an
elemen s in iden i ica ion ca ds, compa ed o he esul s ha we e achie ed in he pas . The models
es ed in he de elopmen o his solu ion will be ho oughly e alua ed and compa ed o he cu en
p oduc ion models. Th ough his compa ison, we can gain a be e unde s anding o he p og ess
made in ou de elopmen and i s po en ial o u u e ad ancemen s.
Also, in he e alua ion phase, we will ocus on e alua ing he in e ence la ency since we belie e ha
his will be o u mos impo ance. The du a ion be ween cap u ing an image and p oducing a esul
has a signi ican impac on deli e ing a smoo h and seamless expe ience o ou consume s and is
he e o e a op p io i y o he company.
Figu e 4 - Expec ed esul
6
2. THEORETICAL FRAMEWORK
This chap e will p o ide a comp ehensi e o e iew o he echnical knowledge ha unde pins he
p ojec ecen ly comple ed by he Machine Lea ning eam a Biome id.
In Sec ion 2.1, a comp ehensi e o e iew o Machine Lea ning will be p esen ed. In Sec ion 2.2, we
will cla i y he heo e ical opics su ounding a i icial neu al ne wo ks, wi h a pa icula ocus on
con olu ional neu al ne wo ks (CNNs). In Sec ion 2.3, we'll del e u he in o a i icial neu al
ne wo ks by explaining he concep o Deep Lea ning. In Sec ion 2.4, we will p o ide an o e iew o
he a ious objec de ec ion model a chi ec u es ha we e es ed o e he pas ew mon hs. In
Sec ion 2.5, we will discuss he e alua ion s age o he p ojec by p esen ing a ious me ics used o
e alua e he pe o mance o objec de ec ion models. Finally, in Sec ion 2.6 o his epo , we will
p o ide an in-dep h explana ion o Op ical Cha ac e Recogni ion (OCR) echnology. This sec ion will
be c ucial o unde s and he inal phase o he p ojec as i will co e he heo y and echnical
aspec s behind OCR.
2.1. MACHINE LEARNING
Machine Lea ning (ML) is one o he subjec ields o a i icial in elligence ha makes use o s a is ical
models so ha , wi h he help o da a, i gene a es eliable p edic ions. Sys ems ha o e hese
capabili ies o e solu ions o p e ious common limi a ions, examples comp ehend a huge space o
possibili ies om co ec ly iden i ying spam emails o image/ ideo ecogni ion (P. A iwala, 2022).
This makes i possible o ind hidden insigh s and iden i y complex pa e ns wi hou explici
p og amming, no jus because o he ML de elopmen bu also due o an inc ease in compu a ional
powe egis e ed in ecen yea s (C. Janiesch e al., 2021).
Acco ding o a epo by McKinsey, machine lea ning can c ea e a po en ial economic impac o $2
illion o $10 illion pe yea ac oss indus ies (J. Bughin e al., 2017) and is expec ed o g ow a a
compound annual g ow h a e o 38.8% om 2022 h ough 2029 and each a alue o $210 billion by
he end o 2029. One o he key ac o s ueling his expansion is he g owing accep ance o machine
lea ning by majo echnology companies such as Apple, Mic oso , and many o he s ac oss a wide
ange o indus ies, including heal hca e, manu ac u ing, au omo i e, e ail, ad e ising, au oma ion,
de ense, inancial se ices, and o he s (O. Fa ooq, 2022).

7
Figu e 5 - Machine Lea ning in es men s by ca ego y
3
Compa ed o con en ional o classical p og amming, whe e sys ems we e ed by inpu da a, and a
unc ion wi h p ede ined ules, which gene a es esul s acco ding o hem. An in elligen mechanism,
ha is, a sys em equipped wi h an ML model, p esen s i sel as an al e na i e o he con en ional
enginee ing app oach which ecei es he same da a, bu unlike he p e ious one, i also ecei es he
expec ed ou pu s and hus c ea es a ma hema ical model wi h pa ame e s adjus ed o he p oblem.
(O. Simeone, 2018)
Figu e 6 - T adi ional P og amming agains Machine Lea ning app oach
4
The idea behind he de elopmen o any machine lea ning algo i hm is he p esence o a la ge
numbe o da a ha p esen a conside able le el o quali y. An o ganized collec ion o da a is
denomina ed by da ase and usually p esen a g oup o ea u es in a abula o ma . These da ase s
3
Adap ed om: h ps://www.s a is a.com/cha /17966/wo ldwide-a i icial-in elligence- unding/
4
Adap ed om: Mo oney, L. (2020). AI and Machine Lea ning o code s. O'Reilly Media.
8
can be used o ain machine lea ning models o asks such as classi ica ion, eg ession, clus e ing,
and o he s.
On he o he hand, image da ase s con ain in o ma ion in he o m o images, which a e wo-
dimensional a ays o pixel alues. These da ase s a e used o ain machine lea ning models o
asks such as image classi ica ion, objec de ec ion, seman ic segmen a ion, and o he s. Image
da ase s can be much la ge in size compa ed o o he ypes o da ase s, as hey o en con ain
hund eds o housands o images, each wi h mul iple channels (e.g., ed, g een, and blue) and high
esolu ion.
To e alua e he pe o mance o a machine lea ning model, he da a a e usually di ided in o h ee
g oups: aining, es ing, and alida ion. Only in his way, i is possible o ha e a pe cep ion o he
beha io o he model wi h da a ne e p ocessed in he aining phase. The i s g oup has he
bigges amoun o da a, ypically be ween six y o eigh y pe cen , and is used o ain he model. The
alida ion se is used as an e alua ion a he end o each epoch o e alua e he model du ing aining
and help op imize i s pa ame e s and se ings. Finally, he es se is p esen ed as a way o e alua e
he model wi h di e en da a om hose p esen ed in he p e ious se s and ha e a inal e alua ion
o he esul s.
Figu e 7 - T ain, Valida ion & Tes Spli
5
Howe e , o conclude his chap e is wo h men ioning ha , despi e he po en ial bene i s in he
indus y, ML implemen a ions ha e also challenges ha need o be add essed. Being one o he mos
impo an examples o highligh he in e p e a ion o accu acy. In mos cases, a model ha p esen s
a high accu acy is conside ed a good one, and he opposi e can lead o ha m. Fo example whe e
acial ecogni ion sys ems a e used in law en o cemen . Bu highly accu a e acial ecogni ion
sys ems can also pose isks o p i acy and indica e he p esence o mass su eillance (J. Fle che , & A.
Kos iainen., 2022).
Ano he impo an opic ela ed o accu acy and he abili y o AI o pe o m co ec p edic ions is i s
usage in a eas ha don’ ha e an objec i e g ound u h and need ex e nal human judgmen . The
impac o an inco ec ou pu depends on he con ex . Fo ins ance, p edic ing alse occu ences o
cance can inc ease cos s, bu on he o he hand, ailing o p edic a ue esul can ha e a much
mo e impac on people’s li e and delay he ea men . Ano he common example o his is using ML
in a judicial con ex , alse posi i es may send innocen s o jail, while alse nega i es may make i
ha de o con ic c iminals. (J. Fle che , & A. Kos iainen., 2022).
5
Adap ed om: h ps://communi y.al e yx.com/ 5/Da a-Science/Holdou s-and-C oss-Valida ion-Why- he-Da a-Used- o-
E alua e-you /ba-p/448982
9
O e all, is impo an o no e ha ML echnologies a e apidly shaping a new u u e o he indus y’s
pano ama. Companies emb acing his ype o AI ha e he po en ial o gain a compe i i e edge and
imp o e hei bo om line. Howe e , i is impo an o companies o unde s and he challenges and
limi a ions o machine lea ning, and o in es in he necessa y expe ise and esou ces o ensu e he
success o hei machine lea ning ini ia i es.
2.1.1. Supe ised Lea ning
Based on a gi en p oblem and he a ailable da a, di e en ypes o Machine Lea ning algo i hms can
be used, each wi h i s ad an ages and disad an ages. The i s s ep is o unde s and which s a egy
o use o analyze he da a. This da a has g ea ele ance in he esul s and can be classi ied as labeled
da a, ha is, da a ha ha e one o mo e classes and allow hei g ouping o unlabeled da a. In his
p ojec , we will ocus on he i s ype o da a which is cha ac e ized as da a ha con ains inpu and
ou pu ea u es.
Fo a ull unde s anding o his p ojec is c ucial o unde s and he heo e ical aspec s o supe ised
lea ning. This ype o machine lea ning model ecei es inpu da a (x) and ou pu da a (y). The goal is
o disco e he pa ame e s o a unc ion ha will gene a e he bes p edic ions when aced wi h
in o ma ion ha has ne e been p ocessed be o e. This is an algo i hm ha needs o ecei e labeled
da a, acqui e a deep knowledge o hem based on i s pa ame e s, and c ea e a cause-e ec
ela ionship.
We can di ide he applica ion o his ype o machine lea ning in o wo ca ego ies, classi ica ion, and
eg ession. In he case o classi ica ion, he objec i e is o p edic a ca ego ical esul o each o he
da a, based on i s independen a iables. In eg ession p oblems, he expec ed ou pu alue is o he
con inuous ype.
2.2. ARTIFICIAL NEURAL NETWORKS & CNNS
A i icial Neu al Ne wo ks (ANN) a e s a is ical models ha a e di ec ly inspi ed by he s uc u e and
unc ion o he neu ons in he b ain. The undamen al uni s o neu al ne wo ks a e also e e ed o
as neu ons, nodes, o a i icial neu ons. A g oup o connec ed neu ons c ea es ne wo ks ha a e
ained o pe o m a a ie y o asks, such as ecognizing pa e ns in da a, making p edic ions, o
making decisions. This can be achie ed by adjus ing he weigh s o he connec ions be ween he
nodes based on example inpu s and hei co esponding ou pu s (K. O'Shea, & R. Nash., 2015).
An ANN is modeled by o e laying se e al laye s o a i icial neu ons, o compu a ional uni s, o
ecei e inpu s and hen ans e hem on o he nex laye . The basic a chi ec u e consis s o a
ne wo k ha has h ee ypes o neu on laye s: inpu , hidden, and ou pu laye s (A. Ab aham, 2005).
Howe e , he e a e some con igu a ion a ian s, depending on he ou pu s and he da a p ocessing
app oach. Some common ypes o neu al ne wo ks a e Feed o wa d Neu al Ne wo ks (FNN),
Con olu ion Neu al Ne wo ks (CNN), and Recu en Neu al Ne wo ks (RNN). Howe e , h oughou
his epo , we’ll jus ocus on CNNs since his ype will be he only one ha will be used o add ess
ou p oblem.
Con olu ional Neu al Ne wo ks (CNN), which a e also called “Con Ne s”, is a ype o ANN ha , like
he o he s, consis s in neu ons ha lea n h ough sel -op imiza ion. The main di e ence be ween
his a chi ec u e and adi ional ANN a chi ec u e is ha his one is designed o deal wi h isual da a.
10
CNN p ocess images o ideos and ex ac s ele an ea u es di ec ly om each pixel alue wi hou
equi ing any hand-enginee ed ea u es o p e ious knowledge abou he wo ld.
Figu e 8 - Con olu ion Neu al Ne wo k A chi ec u e
6
To unde s and CNNs is impo an o know how i s a chi ec u e is composed and wha he ypes o
laye s ha ake pa o i a e. This ype o ANN includes se e al building blocks, such as con olu ion
laye s, pooling laye s, and ully connec ed laye s. A common s uc u e consis s o epe i ions o
se e al con olu ion laye s and a pooling laye , ollowed by one o mo e ully connec ed laye s.
2.2.1. Con olu ion Laye s
A Con olu ion Laye is a undamen al componen o he CNN a chi ec u e ha pe o ms ea u e
ex ac ion, which ypically consis s o a combina ion o linea and nonlinea ope a ions.
The i s ype o linea ope a ion is con olu ion. This ope a ion in ol es sliding a small ma ix o
weigh s (called a ke nel o il e ) o e he inpu da a (o inpu enso ) and compu ing an elemen -
wise p oduc o he weigh s and he alues in he inpu olume a each posi ion. The esul o each
pixel is hen summed o ob ain an ou pu a ay called a ea u e map. This p ocedu e is epea ed by
applying mul iple ke nels o o m an a bi a y numbe o ea u e maps ha ex ac di e en ea u es
om he inpu enso s (R. Yamashi a e al., 2018).
Two key hype pa ame e s ha de ine he con olu ion ope a ion a e size (I x J) and he numbe o
ke nels (K). The size is ypically 3 × 3, bu some imes 5 × 5 o 7 × 7. The numbe o ke nels ha a e
s acked on op o each o he will se he dep h o he ea u e maps and he complexi y o he
de ec ion.
6
Sou ce: Ho ak, K., & Sabla nig, R. (2019, Augus ). Deep lea ning concep s and da ase s o image ecogni ion: o e iew 2019.
In Ele en h in e na ional con e ence on digi al image p ocessing (ICDIP 2019) (Vol. 11179, pp. 484-491). SPIE.
17
A each sliding-window loca ion, se e al p oposals a e p edic ed, and knowing ha he eg esso has
4*k coo dina e ou pu s and he classi ica ion laye gene a es 2*k bina y esul s he au ho s
in oduce he de ini ion o “ancho s” o de ine he cen al poin o each sliding window. Fo any
image, scale, and aspec a io play a decisi e ole as impo an pa ame e s o calcula e he numbe
o ancho s (k). The de elope s chose 3 scales and 3 aspec - a io as de aul pa ame e s meaning ha
he maximum numbe o p oposals pe pixel is equal o 9, concluding ha o he whole image, k =
W*H*9.
Acco ding o he o iginal pape , he loss unc ion o he RPN is minimized by he bina y class label (o
being an objec o no ) o each ancho . To assign a posi i e label o an ancho his one has o ha e
he highes In e sec ion-o e -Union (IoU) o e lap wi h a g ound- u h box o has o ha e an IoU
o e lap highe han 0.7 (S. Ren e al., 2015).
2.4.3. Single Sho Mul iBox De ec o (SSD)
To add ess he high compu a ional p oblems caused by he Fas e R-CNN, o he objec de ec ion
sys ems, p oposed di e en echniques ha educe he in e ence ime while main aining he same
le el o accu acy. So, by he end o 2016, C. Szegedy p esen ed a pape on he implemen a ion o
Single Sho Mul iBox De ec o (SSD) eaching immedia e b eak h ough eco ds in e ms o
pe o mance and p ecision o objec de ec ion asks compa ed o he p e ious models.
This model app oach is based on a eed- o wa d con olu ional ne wo k ha p oduces a ixed
numbe o bounding boxes and c ea es a sco ing mechanism ha e alua es he p esence o objec s
in hose boxes. The a chi ec u e o he SSD uses VGG-16 a chi ec u e as i s base ne wo k inhe i ing in
his way i s s ong pe o mance in high-quali y image classi ica ion asks. F om he e, he main
di e ence s ands in he emo al o he ully connec ed laye s by a se o con olu ional laye s ha
dec ease in size p og essi ely and enable p edic ions a mul iple scales. These laye s a e hen
ollowed by a non-maximum supp ession s ep o p oduce he inal de ec ions.
Figu e 15 - SSD A chi ec u e
12
The SSD Model wo ks as ollows, he base ne wo k p ocesses an inpu image ha is di ided in o g ids
o a ious sizes, and, o each g id, de ec ion is pe o med o di e en classes and di e en aspec
a ios. A e ha , each o hese g ids is e alua ed by a sco e ha says how well he de ec ed objec
i s in ha pa icula g id. The non-maximum supp ession s ep is as s a ed be o e, esponsible o
choosing he bes inal de ec ion om he se o o e lapping de ec ions (W. Liu e al., 2016).
12 Sou ce: Liu, W., Anguelo , D., E han, D., Szegedy, C., Reed, S., Fu, C. Y., & Be g, A. C. (2016). Ssd: Single sho mul ibox de ec o .
In Compu e Vision–ECCV 2016: 14 h Eu opean Con e ence, Ams e dam, The Ne he lands, Oc obe 11–14, 2016, P oceedings, Pa I 14 (pp.
21-37). Sp inge In e na ional Publishing.

18
Figu e 16 - SSD F amewo k
13
In he yea o i s launch, he SSD model egis e ed be e esul s han he p e ious s a e-o - he-a
de ec ion algo i hms, bu in ecen yea s, new a ian s o YOLO a chi ec u e and SSD keep compe ing
o he awa d o bes objec de ec ion model in he PASCAL VOC 2007 Challenge.
2.4.4. Cen e Ne
To educe he s eps pe o med by wo-s age objec de ec ion a chi ec u es, like R-CNN o Fas e R-
CNN models, and o deal wi h he limi a ions o ancho -based models, Cen e Ne a chi ec u e was
p oposed in 2019 as an ancho less objec de ec ion algo i hm ha p o es o be an e ec i e
ligh weigh op ion (K. Duan e al., 2019).
Taking as he baseline he Co ne Ne a chi ec u e, Cen e Ne is a one-s age de ec o ha aims o
imp o e o pe o mance o i s p edecesso by using a iple o keypoin s, a he han a pai . This
a chi ec u e educes he analysis cos by only paying a en ion o he cen al in o ma ion using a
keypoin de ec ion p ocess.
In he gi en igu e, he model ep esen s each objec using a cen e keypoin and a pai o co ne s.
To gene a e a hea map, Cen e Ne uses CNNs as a model backbone o gene a e bounding boxes,
hen coun he numbe o bounding boxes ha con ain he cen e keypoin and uses his in o ma ion
o p edic he likelihood o a cen al egion in he image con aining he same class o cen e
keypoin s. Each pixel in he hea map ep esen s he p obabili y o an objec cen e being p esen a
he co esponding spa ial loca ion in he image. A high p obabili y sco e indica es a high likelihood o
an objec cen e being p esen a ha loca ion, while a low p obabili y sco e indica es a low
likelihood o an objec cen e is p esen . This app oach o p edic ing he likelihood o objec cen e s
using a hea map can imp o e he accu acy o objec de ec ion, pa icula ly o small o occluded
objec s.
13
Sou ce: Liu, W., Anguelo , D., E han, D., Szegedy, C., Reed, S., Fu, C. Y., & Be g, A. C. (2016). Ssd: Single sho mul ibox de ec o .
In Compu e Vision–ECCV 2016: 14 h Eu opean Con e ence, Ams e dam, The Ne he lands, Oc obe 11–14, 2016, P oceedings, Pa I 14 (pp.
21-37). Sp inge In e na ional Publishing.
19
Figu e 17 - Cen e Ne A chi ec u e
14
The au ho s p esen ed an a chi ec u e ha employs wo key s a egies o p edic he geome ic
cen e s and co ne s o objec s. The i s s a egy is called Cen e Pooling and s a s by p edic ing he
cen e keypoin by inding he maximum alue in bo h ho izon al and e ical di ec ions and adding
hese wo alues oge he . This helps o ob ain mo e ecognizable isual pa e ns wi hin objec s,
making i easie o pe cei e he cen al pa o he p oposal.
The second s a egy is denomina ed Cascade Co ne Pooling, which aims o ind he co ne poin s
simila ly o he p e ious one, bu now by ob aining he maximum summed esponse in bo h he
bounda y and in e nal di ec ions o objec s on a ea u e map o co ne p edic ion. Empi ically, he
au ho s claim ha he esul s a e mo e s able and obus o ea u e-le el noises, leading o imp o ed
p ecision and ecall.
In summa y, he Cen e Ne model is a s a e-o - he-a objec de ec ion a chi ec u e ha achie es
high accu acy wi h low compu a ional cos s. The model p edic s objec cen e s di ec ly using a
hea map, which educes he complexi y o he model and imp o es de ec ion accu acy o small and
occluded objec s. The model has achie ed s a e-o - he-a esul s on se e al benchma k da ase s and
is widely used in bo h esea ch and indus y o a ious compu e ision applica ions.
2.5. EVALUATION METRICS
E alua ion me ics play a c ucial ole in he de elopmen and deploymen o machine lea ning
models. Building a machine lea ning model can be seen as a ecu si e pipeline whe e he esponsible
eam ains a model, hen e alua e i using e alua ion me ics, ine une speci ic model
hype pa ame e s, and epea his p ocedu e un il he desi ed pe o mance is achie ed. So, i is
ex emely impo an o ca e ully choose and ack he igh e alua ion me ics o ensu e ha he
model is pe o ming well, o pe o m a model compa ison, and o mee he desi ed objec i es.
In his chap e , we will discuss he heo e ical backg ound o he e alua ion me ics used h oughou
he p ojec and explo e how o choose and in e p e hese me ics in a p oduc ion en i onmen . We
will also discuss he impo ance o conside ing he speci ic cha ac e is ics and goals o he ask when
selec ing an e alua ion me ic, and he ole o e alua ion me ics in model selec ion, op imiza ion,
and moni o ing.
14
Sou ce: Duan, K., Bai, S., Xie, L., Qi, H., Huang, Q., & Tian, Q. (2019). Cen e ne : Keypoin iple s o objec de ec ion.
In P oceedings o he IEEE/CVF in e na ional con e ence on compu e ision (pp. 6569-6578).
20
Since we a e dealing wi h an objec de ec ion algo i hm, is expec ed o e alua e he model by
compa ing i s ou pu wi h g ound- u h bounding boxes gene a ed by a human ope a o esponsible
o manually labeling he objec s and de ining hei bounda ies (V. Y. Ma iano e al., 2002). F om his
analysis, i ’s common o use e alua ion me ics like accu acy, p ecision, ecall, and many o he s o
unde s and how well he model can dis inguish be ween di e en class labels and he ade-o s
be ween alse posi i e and alse nega i e p edic ions.
Since 2010 challenges like VOC PASCAL Challenge, COCO, ImageNe Objec De ec ion Challenge, and
Google Open Images Challenge ha e eme ged as compe i ions ha aim o e alua e new
implemen a ions and a e now seen as aluable benchma ks o es objec de ec ion models in
speci ic scena ios by using eal-wo ld anno a ed da ase s (M. E e ingham e al., 2009). These
compe i ions con ibu e o he de ini ion o s anda d e alua ion p ocedu es wi hin he scien i ic
communi y by using popula me ics like A e age P ecision (AP), In e sec ion o e Union (IoU), o
c ea ing hei a ian s o ank he models.
In he nex sec ions, we’ll y o cla i y he me ics p e iously men ioned in a simpli ied and
o ganized s a egy whe e he eade is guided by a sequen ial p ocess illus a ed in he igu e below.
I should be no ed ha each s ep can be conside ed as a s andalone me hod o e alua ion, bu hey
a e in e dependen and build upon one ano he .
Figu e 18 - Objec De ec ion E alua ion P ocess
2.5.1. Con usion Ma ix
This able comes up as a use ul ool o an ini ial unde s anding o he ypes o e o s made by a
model and o compa ing he pe o mance o di e en models. I ’s cons uc ed by compa ing he
p edic ed class labels wi h he ue class labels o he da a.
21
T ue Class
P edic ed
Class
Posi i e
Nega i e
Posi i e
TP
FP
Nega i e
FN
TN
Table 1 - Con usion Ma ix
15
Looking a he able abo e, a con usion ma ix will be composed by ou main a eas ha help o
unde s and he s eng hs and weaknesses o a model and o iden i ying a eas o imp o emen .
Mo e p ecisely, each p edic ion can be classi ied as:
• T ue posi i e (TP) i he p edic ion con ains a co ec de ec ion o a g ound- u h bounding
box.
• False posi i e (FP) i he algo i hm pe o ms an inco ec de ec ion o a nonexis en objec o
a misplaced de ec ion o an exis ing objec .
• False nega i e (FN) i he g ound- u h objec is no de ec ed.
Bea in mind ha , wi hin he ealm o objec de ec ion, he e is no such hing as a False Posi i e (FP)
as he e a e an in ini e numbe o bounding boxes ha should no be iden i ied wi hin a gi en image.
2.5.2. In e sec ion o e Union (IoU)
Based on he Jacca d Index, his me ic indi idually e alua es he o e lap be ween a p e-anno a ed
g ound u h bounding box (g ) and he one p edic ed by he model (pd) ha ing in conside a ion a
speci ic h eshold alue.
IoU sco e anges be ween 0 and 1 whe e he close he wo boxes he highe he sco e, meaning ha
de ec ion ou pu wi h maximum IoU alue is conside ed o ha e a pe ec o e lap wi h he g ound
u h bounding box.
Figu e 19 - Fo mula and Rep esen a ion o he IoU calcula ion
16
15
Adap ed om: Jeppesen, J. H., Jacobsen, R. H., Inceoglu, F., & To egaa d, T. S. (2019). A cloud de ec ion algo i hm o
sa elli eimage y based on deep lea ning. Remo e sensing o en i onmen , 229, 247-259.
16
Adap ed om: Padilla, R., Ne o, S. L., & Da Sil a, E. A. (2020, July). A su ey on pe o mance me ics o objec -de ec ion
algo i hms. In 2020 in e na ional con e ence on sys ems, signals and image p ocessing (IWSSIP) (pp. 237-242). IEEE.
22
By compu ing he IoU sco e o each de ec ion, i ’s impo an o de ine a h eshold (a) o con e
eal- alued sco es in o classi ica ions.
2.5.3. P ecision and Recall
This pe o mance benchma k ac as a measu e o how closely a p edic ed alue ag ees wi h he ue
alue o a quan i y. I is de ined as he a io o he numbe o co ec ly p edic ed posi i e cases
(posi i e p edic ions con aining an IoU g ea e han he h eshold) o he o al numbe o p edic ed
posi i e cases (co ec ly iden i ied objec s + alse posi i e p edic ions). In o he wo ds, i is a measu e
o he p opo ion o posi i e cases ha a e ac ually posi i e.
Figu e 20 - P ecision Fo mula
P ecision is o en used in conjunc ion wi h ano he me ic called Recall (o Sensi i y) o e alua e he
pe o mance o a model. P ecision is ypically mo e ele an when he goal is o limi he numbe o
alse posi i e p edic ions, whe eas ecall is mo e ele an when he goal is o iden i y as many
posi i e cases as possible. This way we can de ine Recall as he me ic ha measu es he a io o he
numbe o co ec ly p edic ed posi i e cases o he o al numbe o ac ual posi i e cases. In o he
wo ds, i is a measu e o he p opo ion o ac ual posi i e cases ha a e co ec ly iden i ied by he
model.
Figu e 21 - Recall Fo mula
Impo an o men ion ha , in he p esence o imbalanced da ase s, whe e one class is a e, i ’s
impo an o ocus on he ecall o he model.
2.5.4. F1-Sco e:
This is he ha monic mean o p ecision and ecall, wi h a highe sco e indica ing a be e balance
be ween he wo. The F1 sco e is de ined as:
Figu e 22 - F1-Sco e Fo mula
The F1 sco e is o en used in imbalanced classi ica ion asks, whe e i is impo an o achie e a
balance be ween p ecision and ecall. I is also use ul when he cos o alse posi i e and alse
nega i e p edic ions is no he same, as i allows o he weigh ing o hese cos s o be inco po a ed
in o he e alua ion me ic.
2.5.5. A e age P ecision
In objec de ec ion algo i hms, i is essen ial o s ike a balance be ween p ecision and ecall. The
adi ional e alua ion me ic, which is he F1 sco e, can only p o ide a single sco e o a gi en

23
h eshold, making i less in o ma i e in si ua ions whe e he h eshold a ies. To add ess his issue,
he A e age P ecision (AP) me ic was in oduced, which has become a widely used e alua ion
me ic o objec de ec ion algo i hms. AP conside s he numbe o ue posi i es, alse posi i es, and
alse nega i es, making i pa icula ly use ul when he da a is imbalanced. By compu ing he a ea
unde he p ecision- ecall cu e (AUC), AP p o ides a comp ehensi e summa y o he ade-o
be ween p ecision and ecall ac oss di e en h esholds, hus p o iding a mo e in o ma i e
e alua ion me ic o objec de ec ion algo i hms.
To calcula e he AP, he p ecision- ecall cu e is i s compu ed by a ying he h eshold alues and
plo ing he p ecision on he y-axis and ecall on he x-axis. The AUC is hen compu ed, which anges
om 0 o 1, wi h highe alues indica ing be e pe o mance. AP p o ides a mo e p ecise e alua ion
me ic compa ed o adi ional me ics, making i a aluable ool o objec de ec ion algo i hms. As
such, i has become a s anda d me ic used in many objec de ec ion benchma ks and compe i ions.
2.5.6. Mean A e age P ecision (mAP)
Mul iclass Objec De ec ion models ace he challenge o de ec ing mul iple objec classes in an
image. As a esul , a me ic called Mean A e age P ecision (mAP) was de eloped as an ex ension o
he AP me ic o calcula e he a e age p ecision o each class. mAP calcula es he AP o each class
in he da ase and hen a e ages he AP alues o all he classes o ob ain he inal mAP sco e. The
esul ing alue anges om 0 o 1, wi h 1 indica ing he bes possible pe o mance. mAP is a use ul
me ic as i p o ides an o e all iew o he model's pe o mance and helps o iden i y which classes
he model is pe o ming well on and which classes i is s uggling wi h, p o iding insigh in o whe e
he model needs imp o emen .
The simplici y o mAP has made i an essen ial benchma k o e alua ing he pe o mance o objec
de ec ion models in compe i ions and challenges. The mAP allows o easy compa ison o di e en
models and p o ides a ai e alua ion me ic o all models, ega dless o he numbe o classes in
he da ase . This makes i an e ec i e ool o esea che s and de elope s o assess he pe o mance
o hei models and iden i y a eas o imp o emen .
2.6. OPTICAL CHARACTER RECOGNITION (OCR)
Op ical Cha ac e Recogni ion (OCR) is a echnology ha p o ides he capaci y o con e ing
handw i en, ypew i en, o p in ed ex in o machine- eadable images. This o e s nume ous
applica ions ha can be used o imp o e wo k low e iciency. Common examples a e indus ies like
legal, banking, and heal hca e ha a e cu en ly le e aging OCR echnology o simpli y hei
ope a ions and s eamline hei p ocesses (A. Singh, K. Bacchuwa , & A. Bhasin., 2012).
Unlike humans, machines do no ha e he capabili y o ecognize ex o cha ac e s easily om an
image, which is why signi ican esea ch e o s ha e been pu in o de eloping OCR echniques. OCR
is a complex p oblem due o he nume ous languages, on s, and s yles in which ex can be w i en,
as well as he complex ules o languages. As a esul , OCR equi es he in eg a ion o a ious
compu e science disciplines, such as image p ocessing, pa e n classi ica ion, and na u al language
p ocessing, o o e come hese challenges (N. Islam, Z. Islam, & N. Noo ., 2017).
The p ocess o OCR in ol es a se ies o dis inc phases. The i s one, denomina ed by image
acquisi ion, en ails ob aining an image om an ex e nal sou ce such as a came a o scanne , and
24
ans o ming i in o a o ma ha is compa ible wi h compu e p ocessing. This is a c ucial s ep in he
OCR p ocess as i se s he ounda ion o accu a e and e icien op ical cha ac e ecogni ion.
Once he image has been acqui ed, he subsequen s ep in he OCR p ocess is known as
p ep ocessing. Du ing his phase, a a ie y o echniques can be u ilized o enhance he quali y o he
image. These echniques may in ol e emo ing noise, se ing h esholds, and ex ac ing he baseline
o he image. By imp o ing he image quali y, he OCR algo i hm can achie e a highe le el o
accu acy when i comes o ecognizing he ex o cha ac e s con ained wi hin he image. This phase
is c i ical o ensu ing he o e all e ec i eness o he OCR p ocess.
Mo ing o he nex phase, he nex s ep in he OCR p ocess will ocus on cha ac e segmen a ion.
The goal o his phase is o isola e indi idual cha ac e s wi hin he image so ha hey can be p ope ly
iden i ied by he ecogni ion engine. While simple echniques such as connec ed componen analysis
and p ojec ion p o iles may be su icien in ce ain cases, mo e ad anced segmen a ion echniques
a e necessa y o complex si ua ions whe e cha ac e s may be o e lapping, b oken, o obscu ed by
noise wi hin he image. By e ec i ely sepa a ing he cha ac e s, he OCR sys em can accu a ely
ecognize he ex and p oduce an ou pu ha is ai h ul o he o iginal documen .
The segmen ed cha ac e s a e hen p ocessed o ex ac di e en ea u es. Based on hese ea u es,
he cha ac e s a e ecognized. Di e en ypes o ea u es ha can be used ex ac ed om images a e
momen s e c. The ex ac ed ea u es should be e icien ly compu able, minimize in a-class
a ia ions, and maximizes in e -class a ia ions.
The segmen ed cha ac e s a e hen p ocessed in o de o ex ac di e en ea u es ha will be used
o cha ac e ecogni ion. Based on hese ea u es, he cha ac e s a e ecognized. The ea u es
ex ac ed om he cha ac e s should be e icien ly compu able and should minimize in a-class
a ia ions ( a ia ions wi hin he same class o cha ac e s) while maximizing in e -class a ia ions
( a ia ions be ween di e en classes o cha ac e s). Va ious ypes o ea u es can be ex ac ed om
he cha ac e s, such as momen s and o he image-based ea u es. A e ha , he ea u es o
segmen ed images a e assigned o di e en ca ego ies o classes using di e en ypes o cha ac e
classi ica ion echniques.
Finally, pos -p ocessing echniques can be pe o med o imp o e he accu acy o OCR sys ems. These
echniques u ilize na u al language p ocessing, and geome ic and linguis ic con ex o co ec e o s
in OCR esul s.
25
Figu e 23 – Gene al OCR Wo k low
17
O e he yea s, OCR echnology has unde gone emendous imp o emen s, and mode n sys ems can
now ecognize a ious on s, sizes, and s yles o ex . Despi e his, he accu acy o OCR sys ems s ill
depends on ac o s such as image quali y, ex complexi y, and language. Fo una ely, ad ancemen s
in a i icial in elligence and machine lea ning ha e esul ed in subs an ial imp o emen s in OCR
accu acy, leading o mo e eliable and p ecise esul s (A. Ranjan, V.N.J. Behe a, & M. Reza., 2021).
Nowadays, he e a e nume ous OCR ools a ailable ha ha e made hei usage mo e common in
di e en en i onmen s. Some o he mos popula OCR ools include Tesse ac , de eloped by
Google, Mic oso Cogni i e Se ices by Mic oso , and Amazon Tex ac . Fo ou p ojec , we ha e
decided o u ilize he Py esse ac Py hon lib a y, which p o ides an easy- o-use in e ace o le e age
all he ea u es and capabili ies o Tesse ac . The Py esse ac lib a y o e s a s aigh o wa d and
e icien way o in eg a ing OCR unc ionali y in o Py hon-based applica ions, making i a g ea choice
o ou p ojec needs.
17
Adap ed om: Mudia a, I. M. D. R., A maja, I. M. D. S., Suha sana, I. K., An a a, I. W. G. S., Bha adi ya, I. W. P., Suandi a , G. A.,
& Ind awan, G. (2020, Ap il). Balinese cha ac e ecogni ion on mobile applica ion based on esse ac open-sou ce OCR engine. In Jou nal
o Physics: Con e ence Se ies (Vol. 1516, No. 1, p. 012017). IOP Publishing.
26
3. METHODOLOGY
3.1. TOOLS AND TECHNOLOGIES
In his chap e , all he ools and echnologies used h oughou his p ojec will be p esen ed. We will
explo e he easons o hei selec ion, enume a e al e na i es a ailable in he ma ke , and
unde s and he impo ance o hese ools and echnologies o he success o he p ojec . The
chap e will co e he a ious lib a ies, amewo ks, and pla o ms ha ha e been employed o
acili a e he p ocess o building an objec de ec ion model, om da a p epa a ion o model aining
and deploymen . By he end o his chap e , eade s will ha e a comp ehensi e unde s anding o he
ools and echnologies used in he p ojec , and how hey con ibu e o he o e all objec i e o he
p ojec .
3.1.1. Tenso Flow
Tenso Flow (TF) is a powe ul open-sou ce so wa e lib a y o machine lea ning and deep lea ning,
de eloped by esea che s and enginee s wo king on he Google B ain Team. A i s co e, TF uses da a
low g aphs o ep esen any compu a ion. A da a low g aph is a di ec ed acyclic g aph (DAG) whe e
he edges ep esen he low o da a, and he nodes ep esen ope a ions. This makes i easy o
implemen machine lea ning algo i hms, such as neu al ne wo ks (M. Abadi e al., 2016).
Tenso Flow also p o ides a wide ange o ools o building, aining, and deploying machine lea ning
models using a a ie y o pla o ms including CPUs, GPUs, and TPUs. Howe e , he de elopmen
eam main ains a collec ion o p e-made models on a pla o m called Tenso Flow Hub, which can be
used o asks like objec de ec ion, image classi ica ion, and ex gene a ion.
Fo isualiza ion pu poses, Tenso Flow has a isualiza ion ool called Tenso Boa d, which allows
de elope s o easily isualize and unde s and he beha io o hei models du ing aining,
e alua ion, and in e ence.
Figu e 24 - Tenso Boa d In e ace
18
The main al e na i e o Tenso Flow is PyTo ch. This py hon lib a y was de eloped by Facebook and
de ines i sel as an easie - o-use ool easy o asks such as image classi ica ion, na u al language
18
Sou ce: h ps://www. enso low.o g/ enso boa d
33
o da a o a new ask is limi ed, as ans e lea ning educes he need o ex ensi e da a collec ion
and labeling.
Luckily, Tenso Flow o e s a eposi o y ha con ains se e al model a chi ec u es p e iously ained
and accompanied by hei e alua ions when used in he COCO da ase . This way we s a ed o
esea ch which ones me ou needs seeking a model wi h consis en pe o mance bu a he same
ime no p esen ing a high in e ence ime as his would nega i ely a ec he p ocessing ime o he
OCR pipeline.
To o ganize he p ojec and obse e imp o emen s o e ime, we op o di ide i in o e sions. Each
e sion would p esen a di e en g oup o models alongside da a modi ica ions ha we belie ed
ha made sense acco ding o he ou pu s e ained. In his way, i was possible o e ol e ou wo k
while deli e ing epo s o he Biome id eam a he end o each mon h. Below is displayed a able
illus a ing all he models es ed pe e sion.
Models
Obse a ions
Ve sion 1
E icien De D1
SSD ResNe 50
SSD ResNe 101
SSD ResNe 152
Ve sion 2
SSD ResNe 152 1024x1024
SSD ResNe 50 2
Imp o ed e sion wi h
di e en hype pa ame e s
SSD ResNe 101 2
Imp o ed e sion wi h
di e en hype pa ame e s
Ve sion 3
Fas e R-CNN Incep ion
E icien De D2
Cen e Ne Hou Glass104
Ve sion 4
Fas e R-CNN Incep ion 2
Imp o ed e sion wi h
di e en hype pa ame e s
Cen e Ne Resne 50
Cen e Ne Resne 101
Table 2 - Model's a chi ec u e used pe e sion.
I 's c ucial o emphasize ha he model’s key aspec s ha e o be al e ed o adap o he app op ia e
da a o ma o ou p ojec . This in ol ed modi ying each model's con igu a ion ile o enhance he
de ec ion o objec s based on he quan i y and shape o he bounding boxes.

34
To measu e he pe o mance o ou models, we ocused on e alua ing hem using me ics explained
p e iously such as mAP, Recall, and IoU. Fo his pu pose, he e alua ion me ics g oup was se o
“coco_de ec ion_me ics” in he e al_con ig pa ame e . Al hough he lib a y p o ides o he
e alua ion me ics, we ound his o be he mos app op ia e o ou p ojec .
The model aining p ocedu e was ho oughly documen ed, making i possible o ini ia e he i s
aining i e a ion a e inse ing he da a and label maps. This ga e us a p elimina y unde s anding o
he model's capabili ies. The aining p ocess was moni o ed by no only he console logs bu also
h ough he use o Tenso Boa d. This allowed o comp ehensi e acking o he aining e olu ion.
3.2.3. Model Fine-Tuning
In he ini ial i e a ion o he models, we solely elied on he aining and es da a con e ed o
TFReco ds o ma wi hou making any modi ica ions o he model a chi ec u e. None heless, we
conduc ed esea ch o iden i y he op imal pa ame e s sui ed o ou scena io. This led o he nex
phase o he solu ion's de elopmen known as ine uning he model, whe e du ing each aining
session, small modi ica ions a e made o he model's a chi ec u e o enhance i s e icacy.
In his sec ion, he main objec i e is o discuss he a ious modi ica ions ha he eam made o he
model's con igu a ion ile o ob ain he op imal a chi ec u e o ou pa icula p oblem. The ocus
will be on he da ase composi ion, and a de ailed o e iew will be p o ided o each o he hemes
con ained wi hin he con igu a ion ile. Th oughou his p ocess, we will explain he easoning
behind all he choices and p o ide insigh in o he a ious op ions ha a e a ailable h ough his
lib a y.
3.2.3.1. Image P ep ocessing
In he con ex o any compu e ision p ojec , i is absolu ely c ucial o ha e a ull unde s anding o
he inpu da a ha is being ed in o he model. Wi hou his unde s anding, i is impossible o design
and ain a model ha is uly e ec i e a he ask a hand. Deep lea ning algo i hms ha powe his
kind o p ojec a e hea ily dependen on he inpu da a, and e en small a ia ions o inconsis encies
in he da a can g ea ly impac he pe o mance o he model. I is he e o e essen ial o ca e ully
p ep ocess he da a, ensu ing ha i is in a o ma ha is compa ible wi h he model's a chi ec u e.
Fo una ely, he TFOD API p o ides a ange o p ep ocessing ools ha can be accessed igh ou o
he box. In he ini ial sec ion o he con igu a ion ile, use s ha e he abili y o exe cise con ol o e
he image esizing p ocess, including he shape o he esul ing esized image. The lib a y o e s a
conside able amoun o lexibili y conce ning he a ious pa ame e s ha can be modi ied o achie e
he desi ed esul s. Howe e , i is impo an o no e ha be o e any modi ica ions a e applied,
Tenso Flow au oma ically handles he no maliza ion o he images. This is an impo an ea u e ha
ensu es consis ency in he da a and helps o op imize he pe o mance o he model.
When con igu ing he s a egy chosen o p ocess he inpu image o ou model, he API o e s wo
dis inc op ions: "keep_aspec _ a io_ esize " and " ixed_shape_ esize ". The o me op ion allows
use s o speci y a minimum size while main aining he aspec a io o he o iginal image. Howe e , i
is impo an o no e ha his op ion can some imes esul in ex ensi ely padded images, especially in
cases whe e he o iginal image is ec angula . In such cases, i may be mo e sui able o use he
" ixed_shape_ esize " op ion, which esizes he image o a speci ied ec angle size de ined by he
35
"heigh " and "wid h" pa ame e s. Fo ou speci ic use case, we ha e de e mined ha he
" ixed_shape_ esize " op ion is he op imal choice, as i allows us o main ain consis ency in he size
o he inpu images, which is c i ical o op imizing he pe o mance o he model.
3.2.3.2. Image Augmen a ion
Image augmen a ion is an essen ial echnique ha can g ea ly enhance he accu acy and obus ness
o he model. By applying a ious ans o ma ions and manipula ions o he inpu images,
augmen a ion p ocedu es can help o add ess issues such as o e i ing, insu icien aining da a,
and class imbalance. Fo example, lipping, o a ing, and c opping images can inc ease he a iabili y
and di e si y o he aining da a while adjus ing b igh ness, con as , and colo can help o accoun
o a ia ions in ligh ing condi ions. Addi ionally, image augmen a ion can also help o mi iga e he
e ec s o occlusions, iewpoin changes, and o he eal-wo ld ac o s ha may impac he model's
abili y o accu a ely de ec objec s in new and unseen images.
Th oughou he cou se o his p ojec , we explo ed a ious o hese me hods o augmen ou da ase
wi h syn he ic images gene a ed h ough he applica ion o image manipula ion echniques. Knowing
ha ou images we e accu a ely segmen ed and co ec ly o ien ed om he segmen a ion and
o a ion models, we made a delibe a e e o o a oid modi ying hei o ien a ion. Ins ead, we
ocused on applying echniques ha could al e he colo sys em o he da a as you can check in he
image below. This decision p o ed o be highly e ec i e, as adjus men s o ac o s such as hue,
con as , sa u a ion, and b igh ness allowed us o minimize he model's sensi i i y o colo . As a
esul , he model was able o de ec objec s mo e accu a ely ac oss a wide ange o colo s, leading
o imp o ed o e all pe o mance. The use o hese echniques demons a es he impo ance o
ca e ul and s a egic da a augmen a ion in he con ex o objec de ec ion and unde sco es he
c i ical ole ha i plays in op imizing model pe o mance.
Figu e 27 - Image Augmen a ion echniques used o he Cen e Ne con igu a ion ile
While he echniques desc ibed abo e can be highly e ec i e o augmen ing image da a in ou
con ex , i 's wo h no ing ha he Tenso Flow Objec De ec ion API o e s a much b oade ange o
augmen a ion op ions. As you can see in he image below, he e a e a wide a ie y o di e en
36
echniques ha can be used o manipula e and enhance image da a, anging om simple colo
adjus men s o mo e complex ans o ma ions like o a ion, lipping, and dis o ion. By ca e ully
conside ing he unique cha ac e is ics and equi emen s o you speci ic use case, you can le e age
hese powe ul ools o c ea e highly cus omized and e ec i e da a augmen a ion s a egies ha can
signi ican ly imp o e he accu acy and pe o mance o you objec de ec ion models.
Figu e 28 - Image Augmen a ion echniques p o ided by Tenso low Objec De ec ion API
20
3.2.3.3. Pos -p ocessing
Pos -p ocessing is ano he indispensable s ep in any objec de ec ion p ojec , as i enables he model
o gene a e accu a e and meaning ul esul s om he aw ou pu o he de ec ion algo i hm. In he
con ex o objec de ec ion, pos -p ocessing ypically in ol es analyzing he ou pu o he model and
applying a ious echniques o e ine and il e he esul s. By ca e ully ailo ing hese pos -
p ocessing echniques o he speci ic equi emen s o he p ojec , i 's possible o signi ican ly
imp o e he accu acy and p ecision o he objec de ec ion model, while also educing he isk o
alse posi i es o o he e o s.
As we a e acing an ancho less a chi ec u e, ou p ima y conce n a his s age was p e en ing
o e i ing and imp o ing model pe o mance. To achie e his, we ocused on ad anced app oaches
o lea ning a e con ol o e ime. Lea ning a e plays a c i ical ole in de e mining he a e a which
he model's in e nal pa ame e s and weigh s a e upda ed du ing aining. A ca e ully chosen lea ning
a e can signi ican ly impac he model's con e gence and i s abili y o ind he op imal se o
pa ame e s o he gi en ask. As such, we employed a ious s a egies o une he lea ning a e,
20
Sou ce: Tenso low Model Ga den Reposi o y
37
including lea ning a e schedules and op imiza ion algo i hms. Th ough hese echniques, we aimed
o s ike a balance be ween model con e gence and gene aliza ion, ul ima ely leading o imp o ed
pe o mance.
In ou case, we employed he cosine lea ning a e decay schedule o con olling he lea ning a e
o e ime. This app oach enables he lea ning a e alue o al e na e be ween inc easing and
dec easing h oughou he aining p ocess. Speci ically, he lea ning a e is g adually dec eased
owa ds ze o as he aining p og esses, which helps he model con e ge owa d an op imal solu ion
while educing he isk o o e i ing.
To co ec ly con igu e he lea ning a e schedule, i is impo an o ocus on he ollowing pa ame e s
and unde s and hei impac o hem on he pe o mance o he model:
• lea ning_ a e_base: pa ame e ha de ines he ini ial lea ning a e ha will be used o ain
you model.
• o al_s eps: pa ame e ha de ines he numbe o o al s eps you model is going o ain.
Impo an o no e ha in he las s eps o you aining job, he lea ning a e schedule will
d i e he lea ning a e alue o be close o ze o.
• wa mup_lea ning_ a e: he maximum alue ha he lea ning a e will each be o e s a ing
o dec ease.
• wa mup_s eps: de ines he numbe o s eps ha will be aken o inc ease he lea ning a e
om lea ning_ a e_base o wa mup_lea ning_ a e
3.2.4. Image Pipeline
Once we had p ope ly ained he model on ou cus om da ase and ine- uned i s pa ame e s o
enhance i s pe o mance, he subsequen phase in ol ed comp ehending he ou comes p oduced by
he model. Fo e e y examined image, he model deli e s nume ous esul s, wi h he i s ones being
he labels assigned o each sec ion. This is c ucial o linking he sec ion's name wi h he coo dina es
o he bounding boxes ha will e en ually be showcased.
The bounding box coo dina es gene a ed by he model du ing image in e ence play a i al ole in
de e mining he loca ion o he egions o in e es . U ilizing hese coo dina es makes i easible o
isola e speci ic subsec ions o he images ha only con ain he desi ed in o ma ion. In ou scena io,
we aim o ex ac sec ions o he image ha exclusi ely con ain pe sonal da a p esen ed on he on
ace o he Po uguese iden i ica ion ca d.
To achie e his objec i e, we designed a da a pipeline ha in ol es conduc ing an image in e ence
and subsequen ly u ilizing he model's ou comes o c op he o iginal image, leading o se e al sub-
images, each co esponding o a pa icula sec ion. Ou wo k low delibe a ely in ol es minimal
image manipula ion echniques since he model is adep a p ocessing ho izon al segmen ed images
and can accommoda e colo a ia ions owing o he p e-applied image augmen a ion echniques.
This ep esen s one o he key bene i s o ou implemen a ion o e he cu en app oach, as we can
simply esize he image and eed i in o he model, educing he eliance on ex ensi e image
38
p ocessing ha can lead o la ency issues. As a esul , ou ML-powe ed solu ion ope a es
independen ly o any ime-consuming image p ocessing, enhancing i s o e all e iciency.
In he ollowing pa ag aphs, we will discuss each unc ion ha cons i u es ou pipeline. Ou pipeline
was en i ely coded in Py hon and c ea ed en i ely by he Machine Lea ning eam, which modi ied he
exis ing unc ions o accommoda e he changes in oduced by he objec de ec ion algo i hm.
The ini ial unc ion in ou pipeline is called "de ec _sec ionsML" and only needs a pa ame e ha
e e s o he image ha will be analyzed. Fo he unc ion o ope a e co ec ly, i mus ecei e an
image o a Po uguese iden i ica ion ca d, which will be ed in o ou model o gene a e a p edic ion.
Al hough he model gene a es se e al pieces o in o ma ion, he unc ion is designed o solely e u n
he alues co esponding o he bounding box coo dina es and hei espec i e classes/labels.
Figu e 29 – Py hon unc ion used o make model in e ences
Wi h a clea unde s anding o he p eceding unc ion's ou comes, we will now p oceed o c op he
o iginal image u ilizing he "c op_sec ions_ML" unc ion, esul ing in a lis o images, wi h each
elemen co esponding o a pa icula sec ion.
Figu e 30 – Py hon unc ion used o c op ca d sec ions by gi en coo dina es
Concluding ou pipeline is he "ex ac _sec ions_da a_ML" unc ion, which plays a undamen al ole
in p ocessing he images o each sec ion. This unc ion employs he Py esse ac lib a y o con e
images in o ex . Howe e , as each sec ion can display da a in a ious o ma s, such as numbe s and
punc ua ion ma ks (as in he case o heigh ) o solely le e s (as in ields like i s o las name), each

39
sec ion equi es a dis inc Tesse ac con igu a ion. As a esul , his unc ion me ges he mul iple
con igu a ions wi h hei espec i e sec ions and c ea es a Py hon dic iona y o sa e he esul s.
Figu e 31 – Py hon unc ion used o apply OCR in gi en sec ions
40
4. EXPERIMENTAL STUDY
In his chap e , we will p esen he ou comes o ou expe imen a ion wi h a ious model
a chi ec u es and de e mine he op imal one o deploymen in a p oduc ion se ing. We will s a by
ou lining he e alua ion me ics used, and hei ela ionship o he concep s in oduced in chap e
2.5. Following ha , we will end by p o iding a de ailed analysis o he compa a i e pe o mance o
ou bes model agains he exis ing p oduc ion s a egy.
4.1. EVALUATION PROTOCOL
Th oughou ou objec de ec ion model aining, Tenso Flow gene a es eal- ime aining p ocess
checkpoin log iles which allow us o assess he pe o mance o ou models. These logs con ain
COCO e alua ion me ics, which include essen ial me ics like mean a e age p ecision, ecall, and
In e sec ion o e Union. These me ics aid us in quan i ying he accu acy and p ecision o ou models
and enable us o iden i y a eas ha equi e imp o emen . By using COCO e alua ion me ics, we can
ensu e ha ou models a e op imized o de ec and classi y objec s wi h high accu acy and
e iciency, allowing us o achie e ou esea ch goals.
As we ha e p e iously men ioned, we spli ou da ase in o wo pa s - a aining se and a es se .
The es se plays a c ucial ole in e alua ing he pe o mance o ou model, as i con ains da a ha
he model has ne e seen be o e. Impo an o unde s and ha he e alua ion p ocess u ilizes he
checkpoin iles gene a ed du ing he aining p ocess o assess he model's abili y o de ec objec s
in he es da ase . The e alua ion gene a es a se o me ics ha p o ide a summa y o he model's
pe o mance, enabling us o ack i s accu acy and p ecision o e ime. These me ics o e aluable
insigh s in o he s eng hs and weaknesses o ou model, allowing us o ine- une i o op imal
pe o mance. By egula ly moni o ing he e alua ion me ics, we can ensu e ha ou models a e
con inuously imp o ing and deli e ing eliable esul s (L. Vladimi o , 2020).
All hese esul s can also be isualized wi h he help o Tenso Boa d. This isualiza ion ool con e s
he e alua ion esul s ob ained om he checkpoin iles in o in ui i e and in o ma i e dashboa ds.
These dashboa ds p o ide a comp ehensi e o e iew o he model's pe o mance, enabling us o
iden i y a eas ha equi e imp o emen . In addi ion o he e alua ion me ics, Tenso Boa d also
allows us o isualize he de ec ion esul s in he es images, p o iding a clea unde s anding o how
well he bounding boxes a e de ec ing he a eas o in e es . This ea u e is pa icula ly use ul in
iden i ying alse posi i es o alse nega i es, which can be u he in es iga ed and co ec ed o
imp o e he model's accu acy. By u ilizing Tenso Boa d o analyze he e alua ion esul s and
isualize he de ec ion ou pu s, we can gain a deepe unde s anding o ou model's pe o mance and
make in o med decisions on how o op imize i u he .
Once we es ablish ha a model p oduces a o able ou comes, i becomes c ucial o assess he
la ency i in oduces while making an in e ence. This aspec holds immense signi icance since i can
ad e sely impac he pipeline's pe o mance i he esponse ime inc eases signi ican ly. The e o e, i
is essen ial o ca e ully sc u inize he model's in e ence ime o ensu e ha i mee s he
equi emen s o he in ended applica ion.
Finally, upon analyzing he model's pe o mance me ics o e he es se , checking he bounding
boxes display and he la ency, he inal e alua ion phase now shi s ocus o assessing he ou pu s
41
gene a ed by he OCR engine. In ligh o his, he p ojec eam decided o build a web applica ion
using S eamli o compa e he esul s o ou app oach agains he cu en p oduc ion en i onmen 's
ou pu . The applica ion p o ides an in e ace ha you can check below, ha allows us o b owse
h ough a olde o images depic ing iden i ica ion ca ds and manually check which sec ions
Tesse ac accu a ely con e ed o ex using ick boxes. This e alua ion phase is undoub edly he
mos ime-consuming since he e is no eco d o all he g ound u h alues o he documen s, and
he pe o mance o he OCR engine mus be assessed by manually inspec ing each sec ion o
de e mine i he ou pu ma ches he expec ed alues.
Figu e 32 - S eamli Ou pu Compa ison App
Upon e alua ing bo h me hods' esul s on ou 3000-image da ase , he applica ion gene a es a CSV
ile, which we use o p oduce a line plo ha will be used o compa e isually he wo
implemen a ions. We will p esen his plo below when analyzing he esul s.
The e alua ion p ocess o his p ojec is comp ehensi ely ou lined, highligh ing each dis inc phase.
The nex chap e will p o ide a de ailed o e iew o he ou comes ob ained in each o hese phases
and explica e he a ionale behind selec ing he inal model.
4.2. EXPERIMENTAL RESULTS AND DISCUSSION
This chap e ocuses on p esen ing he ou comes achie ed wi h ou a i icial in elligence algo i hm's
implemen a ion o de ec ing Po uguese ID ca d sec ions. We ha e di ided he esul s in o h ee
phases, as discussed in he p e ious chap e , o assess ou app oach's e ec i eness in di e en ways.
42
4.2.1. Tenso Boa d Resul s
Th oughou each phase p esen ed in Table 2, we s a ed checking he pe o mance o each model ia
Tenso Boa d by analyzing i s e alua ion me ics on he ain and es se . This app oach allowed us o
moni o he aining p og ess o a speci ic model in eal ime and compa e i o p e iously ained
models.
Ou ini ial emphasis is on he e alua ion me ic known as loss, which is a alue assigned du ing he
aining o a model ha aims o e lec he dispa i y be ween he model's p edic ed ou pu and he
eal ou pu . The undamen al objec i e o he aining p ocess is o dec ease he loss me ic,
indica ing ha he p edic ed ou pu should closely esemble he ac ual ou pu .
P esen ed in he ollowing able a e he loss alues o he mos op imized e sions o each o he
ained models. I is e iden ha all he loss alues a e qui e low, bu he Cen e Ne Resne 101
model ou pe o med he es , indica ing ha i s p edic ions a e he mos accu a e. This ini ial
e alua ion posi ions i as a s ong con ende o he inal model, al hough we s ill need o analyze
nume ous o he me ics o de e mine i i is he op imal choice.
Model
Loss alue
Cen e Ne Hou Glass104
0.01957
E icien De 1
0.02583
E icien De 2
0.02957
Fas e R-CNN Incep ion
0.01372
SSD ResNe 101
0.04287
SSD ResNe 50
0.0577
Cen e Ne Resne 50
0.01219
Cen e Ne Resne 101
0.0106
Table 3 - Loss Compa ison
Rega ding he me ics o mean a e age p ecision and ecall alues, he esul s we e ound o be
dissimila . Among he e alua ed model a chi ec u es, only h ee achie ed alues highe han 80% o
mean a e age p ecision, namely Cen e Ne Hou Glass104, Fas e R-CNN Incep ion, and Cen e Ne
Resne 50. These models also pe o med excep ionally well in e ms o ecall alues. Thus, hese
same h ee models we e conside ed he mos e ec i e ones o he gi en ask. A compa a i e iew
o he eco ded alues o bo h hese e alua ion me ics is p esen ed in he ollowing able.
49
6. BIBLIOGRAPHY
[1] Abadi, M., Ba ham, P., Chen, J., Chen, Z., Da is, A., Dean, J., ... & Zheng, X. (2016, No embe ).
Tenso low: a sys em o la ge-scale machine lea ning. In Osdi (Vol. 16, No. 2016, pp. 265-283).
[2] Ab aham, A. (2005). A i icial neu al ne wo ks. Handbook o measu ing sys em design.
[3] Alexanda i, A. M., Sh ikuma , A., & Kundaje, A. (2017). Sepa able ully connec ed laye s
imp o e deep lea ning models o genomics. BioRxi , 146431.
[4] A iwala. (2022, Oc obe ). 9 Real-Wo ld P oblems ha can be Sol ed by Machine Lea ning.
Ma u i Techlabs. h ps://ma u i ech.com/p oblems-sol ed-machine-lea ning/
[5] A i issimo, F., Giaquin o, N., Sca pe a, M., & Spada ecchia, M. (2019, Oc obe ). An
au oma ic eade o iden i y documen s. In 2019 IEEE In e na ional Con e ence on Sys ems, Man and
Cybe ne ics (SMC) (pp. 3525-3530). IEEE.
[6] Bha , D., Pa el, C., Talsania, H., Pa el, J., Vaghela, R., Pandya, S., ... & Ghay a , H. (2021). CNN
a ian s o compu e ision: his o y, a chi ec u e, applica ion, challenges and u u e
scope. Elec onics, 10(20), 2470.
[7] Bughin, J., Hazan, E., Ramaswamy, S., Chui, M., Allas, T., Dahls om, P., ... & T ench, M.
(2017). A i icial in elligence: The nex digi al on ie ?
[8] Duan, K., Bai, S., Xie, L., Qi, H., Huang, Q., & Tian, Q. (2019). Cen e ne : Keypoin iple s o
objec de ec ion. In P oceedings o he IEEE/CVF in e na ional con e ence on compu e ision (pp.
6569-6578).
[9] E e ingham, M., Van Gool, L., Williams, C. K., Winn, J., & Zisse man, A. (2009). The pascal
isual objec classes ( oc) challenge. In e na ional jou nal o compu e ision, 88, 303-308.
[10] Fa ooq, O. (2022, Oc obe 3). [10 Bes Machine Lea ning S ocks o In es In]. Yahoo Finance.
h ps:// inance.yahoo.com/news/10-bes -machine-lea ning-s ocks-165306920.h ml
[11] Fle che , J., & Kos iainen, A. (2022, No embe ). E hical P inciples o Web Machine Lea ning.
W3C. h ps://www.w3.o g/TR/webmachinelea ning-e hics
[12] F i z Labs. (2018). Objec De ec ion Guide. F i z AI. h ps://www. i z.ai/objec -de ec ion/
[13] Gi shick, R., Donahue, J., Da ell, T., & Malik, J. (2014). Rich ea u e hie a chies o accu a e
objec de ec ion and seman ic segmen a ion. In P oceedings o he IEEE con e ence on compu e
ision and pa e n ecogni ion (pp. 580-587).
[14] Good ellow, I., Bengio, Y., & Cou ille, A. (2016). Deep lea ning. MIT p ess.
[15] H. Zhu, X. Yan, H. Tang, Y. Chang, B. Li and X. Yuan, "Mo ing Objec De ec ion wi h Deep
CNNs," in IEEE Access, ol. 8, pp. 29729-29741, 2020, doi: 10.1109/ACCESS.2020.2972562.
[16] Hakim, H., & Fadhil, A. (2021, Feb ua y). Su ey: Con olu ion neu al ne wo ks in objec
de ec ion. In Jou nal o Physics: Con e ence Se ies (Vol. 1804, No. 1, p. 012095). IOP Publishing.

50
[17] Ho ak, K., & Sabla nig, R. (2019, Augus ). Deep lea ning concep s and da ase s o image
ecogni ion: o e iew 2019. In Ele en h in e na ional con e ence on digi al image p ocessing (ICDIP
2019) (Vol. 11179, pp. 484-491). SPIE.
[18] Huang, J. (2017, Augus 14). “Ou goal: E e y single ca will be au onomous.” Bosch Global.
h ps://www.bosch.com/s o ies/ hough -leade -jensen-huang/
[19] Islam, N., Islam, Z., & Noo , N. (2017). A su ey on op ical cha ac e ecogni ion sys em. a Xi
p ep in a Xi :1710.05703.
[20] Janiesch, C., Zschech, P., & Hein ich, K. (2021). Machine lea ning and deep
lea ning. Elec onic Ma ke s, 31(3), 685-695.
[21] Janke, J., Cas elli, M., & Popo ič, A. (2019). Analysis o he p o iciency o ully connec ed
neu al ne wo ks in he p ocess o classi ying digi al images. Benchma k o di e en classi ica ion
algo i hms on high-le el image ea u es om con olu ional laye s. Expe Sys ems wi h
Applica ions, 135, 12-38.
[22] K izhe sky A., Su ske e I. & Hin on G.E., 2012. Imagene Classi ica ion wi h Deep
Con olu ional Neu al Ne wo ks. Ad ances in Neu al In o ma ion P ocessing Sys ems (NIPS). :1097–
1105.
[23] Liu, W., Anguelo , D., E han, D., Szegedy, C., Reed, S., Fu, C. Y., & Be g, A. C. (2016, Oc obe ).
SSD: Single sho mul ibox de ec o . In Eu opean con e ence on compu e ision (pp. 21-37). Sp inge ,
Cham.
[24] Ma iano, V. Y., Min, J., Pa k, J. H., Kas u i, R., Mihalcik, D., Li, H., ... & D aye , T. (2002,
Augus ). Pe o mance e alua ion o objec de ec ion algo i hms. In 2002 In e na ional Con e ence on
Pa e n Recogni ion (Vol. 3, pp. 965-969). IEEE.
[25] Ma ikis, S. T., An onopoulos, C. P., Vo os, N. S., & Ke amidas, G. (2021, Sep embe ).
Compa a i e e alua ion o compu e ision echnologies, a ge ing objec iden i ica ion and
localiza ion scena ios. In 2021 6 h Sou h-Eas Eu ope Design Au oma ion, Compu e Enginee ing,
Compu e Ne wo ks and Social Media Con e ence (SEEDA-CECNSM) (pp. 1-8). IEEE.
[26] Mo oney, L. (2020). AI and Machine Lea ning o code s. O'Reilly Media.
[27] Mudia a, I. M. D. R., A maja, I. M. D. S., Suha sana, I. K., An a a, I. W. G. S., Bha adi ya, I. W.
P., Suandi a , G. A., & Ind awan, G. (2020, Ap il). Balinese cha ac e ecogni ion on mobile
applica ion based on esse ac open-sou ce OCR engine. In Jou nal o Physics: Con e ence Se ies (Vol.
1516, No. 1, p. 012017). IOP Publishing.
[28] Padilla, R., Ne o, S. L., & Da Sil a, E. A. (2020, July). A su ey on pe o mance me ics o
objec -de ec ion algo i hms. In 2020 in e na ional con e ence on sys ems, signals and image
p ocessing (IWSSIP) (pp. 237-242). IEEE.
[29] Ranjan, A., Behe a, V. N. J., & Reza, M. (2021). Oc using compu e ision and machine
lea ning. Machine Lea ning Algo i hms o Indus ial Applica ions, 83-105.
51
[30] Ren, S., He, K., Gi shick, R., & Sun, J. (2015). Fas e -cnn: Towa ds eal- ime objec de ec ion
wi h egion p oposal ne wo ks. Ad ances in neu al in o ma ion p ocessing sys ems, 28.
[31] Ryan, M., & Hana iah, N. (2015). An examina ion o cha ac e ecogni ion on ID ca d using
empla e ma ching app oach. P ocedia Compu e Science, 59, 520-529.
[32] Simeone, O. (2018). A e y b ie in oduc ion o machine lea ning wi h applica ions o
communica ion sys ems. IEEE T ansac ions on Cogni i e Communica ions and Ne wo king, 4(4), 648-
664.
[33] Singh, A., Bacchuwa , K., & Bhasin, A. (2012). A su ey o OCR applica ions. In e na ional
Jou nal o Machine Lea ning and Compu ing, 2(3), 314.
[34] Vladimi o , L. (2020). Tenso Flow 2 Objec De ec ion API Tu o ial. Tenso Flow 2 Objec
De ec ion API Documen a ion. h ps:// enso low-objec -de ec ion-api-
u o ial. ead hedocs.io/en/la es /index.h ml
[35] Yamashi a, R., Nishio, M., Do, R. K. G., & Togashi, K. (2018). Con olu ional neu al ne wo ks:
an o e iew and applica ion in adiology. Insigh s in o imaging, 9(4), 611–629.
h ps://doi.o g/10.1007/s13244-018-0639-9
[36] Zulkhaiza , A. (2023, Janua y 4). The Fu u e o AI: How A i icial In elligence is Re olu ionizing
he Way We Wo k - Digi al Fi s Magazine. Digi al Fi s Magazine.
h ps://www.digi al i s magazine.com/ he- u u e-o -ai-how-a i icial-in elligence-is- e olu ionizing-
he-way-we-wo k/