Implementation of Deep Learning models for Information Extraction on Identification Documents

i Implemen a ion o Deep Lea ning models o In o ma ion Ex ac ion on Iden i ica ion Documen s Hen ique Edua do Espadinha Renda In e nship a Biome id In e nship epo p esen ed as pa ial equi emen o ob aining he Mas e ’s deg ee P og am in Da a Science and Ad anced Analy ics i Ti le: Implemen a ion o Deep Lea ning models o In o ma ion Ex ac ion on Iden i ica ion Documen s In e nship a Al ice Po ugal Hen ique Edua do Espadinha Renda MAA 2023 i ii NOVA In o ma ion Managemen School Ins i u o Supe io de Es a ís ica e Ges ão de In o mação Uni e sidade No a de Lisboa IMPLEMENTATION OF DEEP LEARNING MODELS FOR INFORMATION EXTRACTION ON IDENTIFICATION DOCUMENTS by Hen ique Renda In e nship epo p esen ed as pa ial equi emen o ob aining he Mas e ’s deg ee in Da a Science and Ad anced Analy ics Ad iso : P o Dou o Robe o Hen iques Co Ad iso : Lucas Soa es iii STATEMENT OF INTEGRITY As a membe o he Biome id eam and mas e ´s s uden a NOVA In o ma ion Managemen School, I pledge o uphold he highes s anda ds o in eg i y and e hical conduc in he comple ion o my in e nship epo . This includes ensu ing he accu acy and u h ulness o all in o ma ion p esen ed, p ope ly ci ing any sou ces used, and e aining om plagia ism o any o he o m o academic dishones y. I will also main ain con iden iali y and espec any p i acy o sensi i e in o ma ion ha may be sha ed du ing he cou se o my in e nship. I unde s and ha any iola ion o hese p inciples may esul in disciplina y ac ion, and I am ully commi ed o conduc ing mysel in an hones and us wo hy manne h oughou my in e nship. Lisbon, Feb ua y 2023 i Feb ua y 2023 DEDICATION I dedica e his epo o my amily and iends, who ha e been a cons an sou ce o inspi a ion and mo i a ion. In pa icula , I would like o exp ess my deepes app ecia ion o my pa en s and gi l iend o hei unwa e ing suppo , encou agemen , and mo i a ion ha ha e been ins umen al in helping me pe se e e h ough he challenges and celeb a e he iumphs. Addi ionally, I ex end my g a i ude o my men o s o p o iding in aluable guidance and suppo h oughou he esea ch p ocess. This epo ep esen s he culmina ion o yea s o ha d wo k, and I am hono ed o sha e i wi h hose who ha e played a signi ican ole in shaping my academic and pe sonal g ow h. i ABSTRACT The de elopmen o objec de ec ion models has e olu ionized he analysis o pe sonal in o ma ion on iden i ica ion ca ds, leading o a dec ease in ex e nal human labo . Al hough p e ious s a egies ha e been employed o add ess his issue wi hou using machine lea ning models, hey all p esen ce ain limi a ions, which a i icial in elligence aims o o e come. This epo del es in o he de elopmen o a deep lea ning-based objec de ec ion capable o ecognizing ele an in o ma ion om Po uguese iden i ica ion ca ds. All he decisions made du ing he p ojec will be accompanied by a de ailed backg ound heo y. Addi ionally, we p o ide an in-dep h analysis o Op ical Cha ac e Recogni ion (OCR) echnology, which was u ilized h oughou he p ojec o gene a e ex om images. As he newes membe o he Machine lea ning Team o Biome id, I had he p i ilege o being in ol ed in his p ojec ha led o he imp o emen o he cu en app oach ha does no le e age machine lea ning in he de ec ion o ele an sec ions om ID ca ds. The indings o his p ojec p o ide a ounda ion o u he esea ch in o he use o AI in iden i ica ion ca d analysis. KEYWORDS A i icial In elligence; Machine Lea ning; A i icial Neu al Ne wo ks; Compu e Vision; Objec De ec ion Models; Op ical Cha ac e Recogni ion (OCR) ii INDEX 1. In oduc ion ................................................................................................................ 1 1.1. Company O e iew .............................................................................................. 2 1.2. P oblem De ini ion ............................................................................................... 3 1.2.1. Case S udy ..................................................................................................... 4 1.2.2. Cons ain s and Limi a ions .......................................................................... 4 1.2.3. P oposed Solu ion ......................................................................................... 5 2. Theo e ical F amewo k ............................................................................................... 6 2.1. Machine Lea ning ................................................................................................. 6 2.1.1. Supe ised Lea ning ...................................................................................... 9 2.2. A i icial Neu al Ne wo ks & CNNs ....................................................................... 9 2.2.1. Con olu ion Laye s ...................................................................................... 10 2.2.2. Pooling Laye s ............................................................................................. 11 2.2.3. Fully Connec ed Laye s ............................................................................... 12 2.3. Deep lea ning ..................................................................................................... 13 2.4. Objec De ec ion ................................................................................................ 14 2.4.1. Region-based Con olu ion Neu al Ne wo ks (R-CNN) ................................ 15 2.4.2. Fas e -RCNN ................................................................................................ 16 2.4.3. Single Sho Mul iBox De ec o (SSD) ........................................................... 17 2.4.4. Cen e Ne .................................................................................................... 18 2.5. E alua ion Me ics ............................................................................................. 19 2.5.1. Con usion Ma ix ......................................................................................... 20 2.5.2. In e sec ion o e Union (IoU) ..................................................................... 21 2.5.3. P ecision and Recall .................................................................................... 22 2.5.4. F1-Sco e: ..................................................................................................... 22 2.5.5. A e age P ecision ........................................................................................ 22 2.5.6. Mean A e age P ecision (mAP) ................................................................... 23 2.6. Op ical Cha ac e Recogni ion (OCR) ................................................................. 23 3. Me hodology ............................................................................................................. 26 3.1. Tools and Technologies ...................................................................................... 26 3.1.1. Tenso Flow .................................................................................................. 26 3.1.2. Label S udio ................................................................................................. 27 3.1.3. OpenCV ....................................................................................................... 28 1 1. INTRODUCTION Wi h he echnological ad ances egis e ed in ecen yea s, i can be acknowledged ha we a e cu en ly in an e a whe e da a is being p oduced a an unp eceden ed a e, compelling businesses o ely on au oma ed me hods o da a analysis. Re ie ing use ul insigh s om his amoun o da a is c ucial, so in olun a ily companies in his si ua ion no ice ha i is manda o y o de elop ad anced algo i hms ha can summa ize, classi y, ex ac impo an in o ma ion, and con e hem in o an unde s andable o m (V. Y., Ma iano e al., 2002). To add ess his issue, he e has been a huge inc ease in he usage o A i icial In elligence (AI) algo i hms o guide business decisions. Machine lea ning (ML) models and A i icial Neu al Ne wo ks (ANN), in pa icula , ha e demons a ed ou s anding pe o mance in esol ing complex challenges ac oss a wide ange o indus ies, e ec i ely e olu ionizing he way companies app oach p oblem- sol ing (A. Zulkhaiza , 2023). Fo example, in compu e ision asks like objec de ec ion and seman ic segmen a ion, deep lea ning models ha e been success ully used in applica ions such as au onomous d i ing. This echnology is apidly ad ancing, b inging sys ems close o possessing a le el o in elligence compa able o humans in ce ain si ua ions. Acco ding o he cu en N idia CEO, Jensen Huang, “Deep Lea ning can ain a ca o d i e, and ul ima ely pe o m a be e , and mo e sa ely, han any human d i e could do behind he wheel.”. This was he goal o Huang’s company when de eloping ha dwa e o he upcoming au onomous ca s (J. Huang, 2017). Figu e 1 - Usage o Deep Lea ning models in au onomous ca s 1 In he example abo e, objec de ec ion models o e a solu ion o iden i ying ele an i ems in he oad ha allow he ehicle o d i e sa ely. This abili y ende s hem highly e icien o ac i i ies such as pe cep ion, decision-making, and ehicle con ol. Ne e heless, hei use ulness ex ends beyond his use case scena io, as hey ha e he capabili y o analyze and e ie e in o ma ion om a gi en ame. As ano he example, h oughou his epo , we’ll highligh he expe ience o a Junio Machine Lea ning Enginee esponsible o a p ojec ha led o he imp o emen o Biome id’s machine lea ning in as uc u e behind i s OCR solu ion o in o ma ion e ie al o Po uguese iden i ica ion 1 Sou ce: h ps://www.cbinsigh s.com/ esea ch/s a ups-d i e-au o-indus y-dis up ion/ 2 ca ds. The esponsibili ies ega ding he collec ion, p ocessing, and analysis o in o ma ion will be desc ibed in de ail. We’ll s a by in oducing he p oblem add essed and cla i y he challenges aced, ega ding he de elopmen o an objec de ec ion model capable o ecognizing ele an sec ions om he Po uguese iden i ica ion ca ds. Nex , we’ll do a concise explana ion o how he company is composed alongside i s objec i e, be o e del ing in o mo e speci ic and echnical aspec s ega ding he heo y behind he p ojec . Finally, his epo will end up by clea ly exposing all he p ac ical implemen a ions o he p ojec alongside he inal esul s. 1.1. COMPANY OVERVIEW Figu e 2 - Biome id's Logo 2 C ea ed in 2015 unde he name Polygon, he cu en Biome id eme ged as an IT company wi h headqua e s es ablished in Po o ha p omised o e olu ionize he way we communica e wi h digi al sys ems. A a ime when wi h one click i is possible o open a bank accoun , egis e on a social ne wo k o e en subsc ibe o a se ice i is no su p ise ha he digi al oo p in o all in e ne use s has been g owing exponen ially. Wi h ha in mind, Biome id is commi ed o deli e ing a ange o ools ha acili a e he onboa ding, au hen ica ion, and alida ion o pe sonnel in companies o all sizes. Figu e 3 - Biome id's Miles ones Biome id o e s a ange o solu ions o simpli y he Know You Cus ome (KYC) p ocesses. The company's lagship p oduc , which sha es he same name, s eamlines hese ime-consuming ac i i ies, which a e essen ial o sa egua d inancial ins i u ions agains aud, co up ion, money launde ing, and e o ism inancing. 2 Sou ce: h ps://biome id.com/ 3 Compa ed o i s main compe i o s, Biome id's p oduc akes a unique app oach. I uses a D ag&D op me hodology, which p o ides cus ome s wi h au onomy and lexibili y o build pe sonalized e i ica ion p ocesses. The p oduc enables use s o open bank accoun s emo ely, e i y P oo -o -Li e using pe sonal documen s, and gene a e digi al signa u es o public se ices. The company has es ablished pa ne ships wi h banking ins i u ions, insu ance companies, and Po uguese Go e nmen depa men s. I has also expanded i s ope a ions o in e na ional ma ke s, including Poland, Hunga y, Colombia, Angola, and Mozambique. Biome id's goal is o con inue expanding i s se ices and each in o new sec o s and egions wo ldwide. Fo he ML eam he main esponsibili y is o c ea e and main ain all he ML models ha suppo i s solu ions, especially he Op ical Cha ac e Recogni ion (OCR) p oduc ion pipeline o e ie ing in o ma ion om iden i ica ion ca ds. The ocus is di ided in o wo: on one side imp o ing he OCR pipeline, which would ecei e images o ID ca ds o passpo s, and om he e ex ac in o ma ion ha could iden i y use s in he u u e. On he o he side, esea ching and de eloping new ML ools o he OCR in as uc u e. 1.2. PROBLEM DEFINITION As men ioned be o e, sys ems equipped wi h a i icial in elligence algo i hms a e now mo e p esen han e e in ou daily li es. This pa adigm shi means ha p ocesses ha equi ed a lo o manual wo k a e now being eplaced by models ha simula e human beha io . One o he se e al examples whe e his change is no iceable is in he egis a ion p ocess. In he pas , ega dless o he indus y, o pe o m he onboa ding o a new clien his one needed o ill ou a o m wi h his pe sonal in o ma ion manually and wai o he esponsible s a o ans e he in o ma ion in o a dedica ed da abase. This implied a a he leng hy p ocess wi h se e al dedica ed pa s and a complex in as uc u e. Luckily, o s eamline he p ocess, a new me hod has eme ged o ex ac ing his in o ma ion wi hou equi ing manual en y om ei he he cus ome o s a (M. Ryan, & N. Hana iah., 2015). A new sys em using op ical cha ac e ecogni ion (OCR) has been p oposed o ex ac cus ome in o ma ion om iden i ica ion ca ds ins ead o manual da a en y. This sys em, in he i s ins ance, is powe ed by an objec de ec ion model esponsible o p ocessing an ID ca d image and e ie ing he coo dina es o he sec ions ela ed o pe sonal da a as he esul . These image coo dina es a e hen used o c op he o iginal image and eed hose ou pu s in o an OCR applica ion ha ecognizes he ex in he pic u e and con e s i in o machine- eadable and edi able ex . 4 1.2.1. Case S udy This chap e will examine he applica ion o he selec ed app oach in he Biome id OCR pipeline o au oma e he p ocess o e ie ing pe sonal in o ma ion om Po uguese ID ca ds. By le e aging cu ing-edge op ical cha ac e ecogni ion echnology, he pipeline aims o s eamline he da a inpu p ocess and p o ide accu a e and e icien esul s. This subsec ion will del e in o he de ails o his s a egy and i s impac on he p ocess o e ie ing pe sonal in o ma ion om ID ca ds. The g ow h o membe s o he ML eam led o a new p ojec which has he objec i e o imp o ing he way he OCR da a pipeline analyses images. Cu en ly, he ex ac ion p ocess con ains se e al laye s o p e-p ocessing: 1. The documen s a e exposed o a o a ion model ha o ien s hem ho izon ally in an au oma ic way. 2. Nex , his esul ing image will se e as inpu o a segmen a ion model, which aims o emo e all su ounding elemen s om he pho o and gene a e c opped images. 3. These images a e hen con e ed om an RGB Fo ma o G ayscale o imp o e he pe o mance o he bina iza ion ha will be applied o he image la ely. 4. A e mul iple mo phological image ans o ma ions, he coo dina es om he egions o in e es we e de ec ed by mul iple adi ional image p ocessing echniques ha use ela i e coo dina es o cap u e in o ma ional sec ions. 5. The p ocess ends jus when all o he p oposed a eas a e analyzed by an OCR engine. This s a egy p esen s i sel as a e y good al e na i e compa ed o a ull-manual p ocess, howe e , i ’s in many ways limi ed as i s esul s a e only iable in a o able condi ions whe e he documen has a cap u e angle and a o able ligh condi ions. As a company ha consis en ly emb aces new echnologies, Biome id ex ends his philosophy o e e y aspec o i s p oduc . So, he main goal o his p ojec is o imp o e he OCR Pipeline by c ea ing an a i icial neu al ne wo k capable o de ec ing all sec ions o he on o he Po uguese iden i ica ion ca d. This way we hope o enhance he gene al pe o mance o he OCR p ocesses and es ablish a s a egy ha deli e s a o able ou comes in all image scena ios. 1.2.2. Cons ain s and Limi a ions In he ealm o came a-based analysis o ex and documen s, he e exis se e al challenges associa ed wi h cap u ing images, including low esolu ion, une en ligh ing, dis o ed pe spec i es, non-plana su aces, wide-angle lens dis o ion, clu e ed backg ounds, di icul ies wi h zooming and ocusing (M. Ryan, & N. Hana iah., 2015). Th oughou he epo , we will explain all he s a egies and echniques ha we will apply o sol e some o hose p oblems. Besides ha , we also ind some cons ain s ela ed o he da ase con aining images o Po uguese Iden i ica ion Ca ds. This is no publicly a ailable due o GDPR es ic ions, bu o una ely, Biome id had al eady ob ained his ype o da a h ough a pa ne ship wi h he “Agência pa a a Mode nização Adminis a i a” (AMA) and egula ly u ilized i o o he p ojec s. So, e en wi h limi ed da a we s ill 5 manage o o ganize and cu a e app oxima ely 3000 o hese images o cons uc a consis en da ase capable o success ully aining he model. Impo an o men ion ha hese images con ain pe sonal in o ma ion, and hei public sha e is exp essly o bidden. 1.2.3. P oposed Solu ion To add ess he case s udy, always wi h he limi a ions and cons ain s in conside a ion, i was decided by he eam ha he bes s a egy o imp o e he de ec ion o he sec ions o he on ace o he Po uguese iden i ica ion ca d would go h ough he aining o a deep neu al ne wo k capable o imp o ing he p e ious esul s and add ess si ua ions whe e he image quali y is no a o able. P e ious s udies sugges ha a ious me hods ha e been employed o ackle his issue. One commonly employed app oach in ol es iden i ying saliencies in he image o loca e documen s inside a pho o o ideo ame, wi hou any p io knowledge abou he documen (F. A i issimo e al., 2019). Howe e , his s a egy s ill p esen s some limi a ions mainly when he image con ains noise, o he angle o cap u e is no a o able. The imp o ed solu ion o he Biome id OCR ha will be p esen ed h oughou his epo p esen s i sel as a mo e ad anced app oach o his p oblem. I s a s by using a de ec ion model o loca e he documen , hen c ops he image o isola e he documen om he backg ound. A e p ope segmen a ion, he documen is classi ied using a deep lea ning model. Th oughou his documen , we aim o enhance he a e age success a e o de ec ing ele an elemen s in iden i ica ion ca ds, compa ed o he esul s ha we e achie ed in he pas . The models es ed in he de elopmen o his solu ion will be ho oughly e alua ed and compa ed o he cu en p oduc ion models. Th ough his compa ison, we can gain a be e unde s anding o he p og ess made in ou de elopmen and i s po en ial o u u e ad ancemen s. Also, in he e alua ion phase, we will ocus on e alua ing he in e ence la ency since we belie e ha his will be o u mos impo ance. The du a ion be ween cap u ing an image and p oducing a esul has a signi ican impac on deli e ing a smoo h and seamless expe ience o ou consume s and is he e o e a op p io i y o he company. Figu e 4 - Expec ed esul 6 2. THEORETICAL FRAMEWORK This chap e will p o ide a comp ehensi e o e iew o he echnical knowledge ha unde pins he p ojec ecen ly comple ed by he Machine Lea ning eam a Biome id. In Sec ion 2.1, a comp ehensi e o e iew o Machine Lea ning will be p esen ed. In Sec ion 2.2, we will cla i y he heo e ical opics su ounding a i icial neu al ne wo ks, wi h a pa icula ocus on con olu ional neu al ne wo ks (CNNs). In Sec ion 2.3, we'll del e u he in o a i icial neu al ne wo ks by explaining he concep o Deep Lea ning. In Sec ion 2.4, we will p o ide an o e iew o he a ious objec de ec ion model a chi ec u es ha we e es ed o e he pas ew mon hs. In Sec ion 2.5, we will discuss he e alua ion s age o he p ojec by p esen ing a ious me ics used o e alua e he pe o mance o objec de ec ion models. Finally, in Sec ion 2.6 o his epo , we will p o ide an in-dep h explana ion o Op ical Cha ac e Recogni ion (OCR) echnology. This sec ion will be c ucial o unde s and he inal phase o he p ojec as i will co e he heo y and echnical aspec s behind OCR. 2.1. MACHINE LEARNING Machine Lea ning (ML) is one o he subjec ields o a i icial in elligence ha makes use o s a is ical models so ha , wi h he help o da a, i gene a es eliable p edic ions. Sys ems ha o e hese capabili ies o e solu ions o p e ious common limi a ions, examples comp ehend a huge space o possibili ies om co ec ly iden i ying spam emails o image/ ideo ecogni ion (P. A iwala, 2022). This makes i possible o ind hidden insigh s and iden i y complex pa e ns wi hou explici p og amming, no jus because o he ML de elopmen bu also due o an inc ease in compu a ional powe egis e ed in ecen yea s (C. Janiesch e al., 2021). Acco ding o a epo by McKinsey, machine lea ning can c ea e a po en ial economic impac o $2 illion o $10 illion pe yea ac oss indus ies (J. Bughin e al., 2017) and is expec ed o g ow a a compound annual g ow h a e o 38.8% om 2022 h ough 2029 and each a alue o $210 billion by he end o 2029. One o he key ac o s ueling his expansion is he g owing accep ance o machine lea ning by majo echnology companies such as Apple, Mic oso , and many o he s ac oss a wide ange o indus ies, including heal hca e, manu ac u ing, au omo i e, e ail, ad e ising, au oma ion, de ense, inancial se ices, and o he s (O. Fa ooq, 2022). 7 Figu e 5 - Machine Lea ning in es men s by ca ego y 3 Compa ed o con en ional o classical p og amming, whe e sys ems we e ed by inpu da a, and a unc ion wi h p ede ined ules, which gene a es esul s acco ding o hem. An in elligen mechanism, ha is, a sys em equipped wi h an ML model, p esen s i sel as an al e na i e o he con en ional enginee ing app oach which ecei es he same da a, bu unlike he p e ious one, i also ecei es he expec ed ou pu s and hus c ea es a ma hema ical model wi h pa ame e s adjus ed o he p oblem. (O. Simeone, 2018) Figu e 6 - T adi ional P og amming agains Machine Lea ning app oach 4 The idea behind he de elopmen o any machine lea ning algo i hm is he p esence o a la ge numbe o da a ha p esen a conside able le el o quali y. An o ganized collec ion o da a is denomina ed by da ase and usually p esen a g oup o ea u es in a abula o ma . These da ase s 3 Adap ed om: h ps://www.s a is a.com/cha /17966/wo ldwide-a i icial-in elligence- unding/ 4 Adap ed om: Mo oney, L. (2020). AI and Machine Lea ning o code s. O'Reilly Media. 8 can be used o ain machine lea ning models o asks such as classi ica ion, eg ession, clus e ing, and o he s. On he o he hand, image da ase s con ain in o ma ion in he o m o images, which a e wo- dimensional a ays o pixel alues. These da ase s a e used o ain machine lea ning models o asks such as image classi ica ion, objec de ec ion, seman ic segmen a ion, and o he s. Image da ase s can be much la ge in size compa ed o o he ypes o da ase s, as hey o en con ain hund eds o housands o images, each wi h mul iple channels (e.g., ed, g een, and blue) and high esolu ion. To e alua e he pe o mance o a machine lea ning model, he da a a e usually di ided in o h ee g oups: aining, es ing, and alida ion. Only in his way, i is possible o ha e a pe cep ion o he beha io o he model wi h da a ne e p ocessed in he aining phase. The i s g oup has he bigges amoun o da a, ypically be ween six y o eigh y pe cen , and is used o ain he model. The alida ion se is used as an e alua ion a he end o each epoch o e alua e he model du ing aining and help op imize i s pa ame e s and se ings. Finally, he es se is p esen ed as a way o e alua e he model wi h di e en da a om hose p esen ed in he p e ious se s and ha e a inal e alua ion o he esul s. Figu e 7 - T ain, Valida ion & Tes Spli 5 Howe e , o conclude his chap e is wo h men ioning ha , despi e he po en ial bene i s in he indus y, ML implemen a ions ha e also challenges ha need o be add essed. Being one o he mos impo an examples o highligh he in e p e a ion o accu acy. In mos cases, a model ha p esen s a high accu acy is conside ed a good one, and he opposi e can lead o ha m. Fo example whe e acial ecogni ion sys ems a e used in law en o cemen . Bu highly accu a e acial ecogni ion sys ems can also pose isks o p i acy and indica e he p esence o mass su eillance (J. Fle che , & A. Kos iainen., 2022). Ano he impo an opic ela ed o accu acy and he abili y o AI o pe o m co ec p edic ions is i s usage in a eas ha don’ ha e an objec i e g ound u h and need ex e nal human judgmen . The impac o an inco ec ou pu depends on he con ex . Fo ins ance, p edic ing alse occu ences o cance can inc ease cos s, bu on he o he hand, ailing o p edic a ue esul can ha e a much mo e impac on people’s li e and delay he ea men . Ano he common example o his is using ML in a judicial con ex , alse posi i es may send innocen s o jail, while alse nega i es may make i ha de o con ic c iminals. (J. Fle che , & A. Kos iainen., 2022). 5 Adap ed om: h ps://communi y.al e yx.com/ 5/Da a-Science/Holdou s-and-C oss-Valida ion-Why- he-Da a-Used- o- E alua e-you /ba-p/448982 9 O e all, is impo an o no e ha ML echnologies a e apidly shaping a new u u e o he indus y’s pano ama. Companies emb acing his ype o AI ha e he po en ial o gain a compe i i e edge and imp o e hei bo om line. Howe e , i is impo an o companies o unde s and he challenges and limi a ions o machine lea ning, and o in es in he necessa y expe ise and esou ces o ensu e he success o hei machine lea ning ini ia i es. 2.1.1. Supe ised Lea ning Based on a gi en p oblem and he a ailable da a, di e en ypes o Machine Lea ning algo i hms can be used, each wi h i s ad an ages and disad an ages. The i s s ep is o unde s and which s a egy o use o analyze he da a. This da a has g ea ele ance in he esul s and can be classi ied as labeled da a, ha is, da a ha ha e one o mo e classes and allow hei g ouping o unlabeled da a. In his p ojec , we will ocus on he i s ype o da a which is cha ac e ized as da a ha con ains inpu and ou pu ea u es. Fo a ull unde s anding o his p ojec is c ucial o unde s and he heo e ical aspec s o supe ised lea ning. This ype o machine lea ning model ecei es inpu da a (x) and ou pu da a (y). The goal is o disco e he pa ame e s o a unc ion ha will gene a e he bes p edic ions when aced wi h in o ma ion ha has ne e been p ocessed be o e. This is an algo i hm ha needs o ecei e labeled da a, acqui e a deep knowledge o hem based on i s pa ame e s, and c ea e a cause-e ec ela ionship. We can di ide he applica ion o his ype o machine lea ning in o wo ca ego ies, classi ica ion, and eg ession. In he case o classi ica ion, he objec i e is o p edic a ca ego ical esul o each o he da a, based on i s independen a iables. In eg ession p oblems, he expec ed ou pu alue is o he con inuous ype. 2.2. ARTIFICIAL NEURAL NETWORKS & CNNS A i icial Neu al Ne wo ks (ANN) a e s a is ical models ha a e di ec ly inspi ed by he s uc u e and unc ion o he neu ons in he b ain. The undamen al uni s o neu al ne wo ks a e also e e ed o as neu ons, nodes, o a i icial neu ons. A g oup o connec ed neu ons c ea es ne wo ks ha a e ained o pe o m a a ie y o asks, such as ecognizing pa e ns in da a, making p edic ions, o making decisions. This can be achie ed by adjus ing he weigh s o he connec ions be ween he nodes based on example inpu s and hei co esponding ou pu s (K. O'Shea, & R. Nash., 2015). An ANN is modeled by o e laying se e al laye s o a i icial neu ons, o compu a ional uni s, o ecei e inpu s and hen ans e hem on o he nex laye . The basic a chi ec u e consis s o a ne wo k ha has h ee ypes o neu on laye s: inpu , hidden, and ou pu laye s (A. Ab aham, 2005). Howe e , he e a e some con igu a ion a ian s, depending on he ou pu s and he da a p ocessing app oach. Some common ypes o neu al ne wo ks a e Feed o wa d Neu al Ne wo ks (FNN), Con olu ion Neu al Ne wo ks (CNN), and Recu en Neu al Ne wo ks (RNN). Howe e , h oughou his epo , we’ll jus ocus on CNNs since his ype will be he only one ha will be used o add ess ou p oblem. Con olu ional Neu al Ne wo ks (CNN), which a e also called “Con Ne s”, is a ype o ANN ha , like he o he s, consis s in neu ons ha lea n h ough sel -op imiza ion. The main di e ence be ween his a chi ec u e and adi ional ANN a chi ec u e is ha his one is designed o deal wi h isual da a. 10 CNN p ocess images o ideos and ex ac s ele an ea u es di ec ly om each pixel alue wi hou equi ing any hand-enginee ed ea u es o p e ious knowledge abou he wo ld. Figu e 8 - Con olu ion Neu al Ne wo k A chi ec u e 6 To unde s and CNNs is impo an o know how i s a chi ec u e is composed and wha he ypes o laye s ha ake pa o i a e. This ype o ANN includes se e al building blocks, such as con olu ion laye s, pooling laye s, and ully connec ed laye s. A common s uc u e consis s o epe i ions o se e al con olu ion laye s and a pooling laye , ollowed by one o mo e ully connec ed laye s. 2.2.1. Con olu ion Laye s A Con olu ion Laye is a undamen al componen o he CNN a chi ec u e ha pe o ms ea u e ex ac ion, which ypically consis s o a combina ion o linea and nonlinea ope a ions. The i s ype o linea ope a ion is con olu ion. This ope a ion in ol es sliding a small ma ix o weigh s (called a ke nel o il e ) o e he inpu da a (o inpu enso ) and compu ing an elemen - wise p oduc o he weigh s and he alues in he inpu olume a each posi ion. The esul o each pixel is hen summed o ob ain an ou pu a ay called a ea u e map. This p ocedu e is epea ed by applying mul iple ke nels o o m an a bi a y numbe o ea u e maps ha ex ac di e en ea u es om he inpu enso s (R. Yamashi a e al., 2018). Two key hype pa ame e s ha de ine he con olu ion ope a ion a e size (I x J) and he numbe o ke nels (K). The size is ypically 3 × 3, bu some imes 5 × 5 o 7 × 7. The numbe o ke nels ha a e s acked on op o each o he will se he dep h o he ea u e maps and he complexi y o he de ec ion. 6 Sou ce: Ho ak, K., & Sabla nig, R. (2019, Augus ). Deep lea ning concep s and da ase s o image ecogni ion: o e iew 2019. In Ele en h in e na ional con e ence on digi al image p ocessing (ICDIP 2019) (Vol. 11179, pp. 484-491). SPIE. 17 A each sliding-window loca ion, se e al p oposals a e p edic ed, and knowing ha he eg esso has 4*k coo dina e ou pu s and he classi ica ion laye gene a es 2*k bina y esul s he au ho s in oduce he de ini ion o “ancho s” o de ine he cen al poin o each sliding window. Fo any image, scale, and aspec a io play a decisi e ole as impo an pa ame e s o calcula e he numbe o ancho s (k). The de elope s chose 3 scales and 3 aspec - a io as de aul pa ame e s meaning ha he maximum numbe o p oposals pe pixel is equal o 9, concluding ha o he whole image, k = W*H*9. Acco ding o he o iginal pape , he loss unc ion o he RPN is minimized by he bina y class label (o being an objec o no ) o each ancho . To assign a posi i e label o an ancho his one has o ha e he highes In e sec ion-o e -Union (IoU) o e lap wi h a g ound- u h box o has o ha e an IoU o e lap highe han 0.7 (S. Ren e al., 2015). 2.4.3. Single Sho Mul iBox De ec o (SSD) To add ess he high compu a ional p oblems caused by he Fas e R-CNN, o he objec de ec ion sys ems, p oposed di e en echniques ha educe he in e ence ime while main aining he same le el o accu acy. So, by he end o 2016, C. Szegedy p esen ed a pape on he implemen a ion o Single Sho Mul iBox De ec o (SSD) eaching immedia e b eak h ough eco ds in e ms o pe o mance and p ecision o objec de ec ion asks compa ed o he p e ious models. This model app oach is based on a eed- o wa d con olu ional ne wo k ha p oduces a ixed numbe o bounding boxes and c ea es a sco ing mechanism ha e alua es he p esence o objec s in hose boxes. The a chi ec u e o he SSD uses VGG-16 a chi ec u e as i s base ne wo k inhe i ing in his way i s s ong pe o mance in high-quali y image classi ica ion asks. F om he e, he main di e ence s ands in he emo al o he ully connec ed laye s by a se o con olu ional laye s ha dec ease in size p og essi ely and enable p edic ions a mul iple scales. These laye s a e hen ollowed by a non-maximum supp ession s ep o p oduce he inal de ec ions. Figu e 15 - SSD A chi ec u e 12 The SSD Model wo ks as ollows, he base ne wo k p ocesses an inpu image ha is di ided in o g ids o a ious sizes, and, o each g id, de ec ion is pe o med o di e en classes and di e en aspec a ios. A e ha , each o hese g ids is e alua ed by a sco e ha says how well he de ec ed objec i s in ha pa icula g id. The non-maximum supp ession s ep is as s a ed be o e, esponsible o choosing he bes inal de ec ion om he se o o e lapping de ec ions (W. Liu e al., 2016). 12 Sou ce: Liu, W., Anguelo , D., E han, D., Szegedy, C., Reed, S., Fu, C. Y., & Be g, A. C. (2016). Ssd: Single sho mul ibox de ec o . In Compu e Vision–ECCV 2016: 14 h Eu opean Con e ence, Ams e dam, The Ne he lands, Oc obe 11–14, 2016, P oceedings, Pa I 14 (pp. 21-37). Sp inge In e na ional Publishing. 18 Figu e 16 - SSD F amewo k 13 In he yea o i s launch, he SSD model egis e ed be e esul s han he p e ious s a e-o - he-a de ec ion algo i hms, bu in ecen yea s, new a ian s o YOLO a chi ec u e and SSD keep compe ing o he awa d o bes objec de ec ion model in he PASCAL VOC 2007 Challenge. 2.4.4. Cen e Ne To educe he s eps pe o med by wo-s age objec de ec ion a chi ec u es, like R-CNN o Fas e R- CNN models, and o deal wi h he limi a ions o ancho -based models, Cen e Ne a chi ec u e was p oposed in 2019 as an ancho less objec de ec ion algo i hm ha p o es o be an e ec i e ligh weigh op ion (K. Duan e al., 2019). Taking as he baseline he Co ne Ne a chi ec u e, Cen e Ne is a one-s age de ec o ha aims o imp o e o pe o mance o i s p edecesso by using a iple o keypoin s, a he han a pai . This a chi ec u e educes he analysis cos by only paying a en ion o he cen al in o ma ion using a keypoin de ec ion p ocess. In he gi en igu e, he model ep esen s each objec using a cen e keypoin and a pai o co ne s. To gene a e a hea map, Cen e Ne uses CNNs as a model backbone o gene a e bounding boxes, hen coun he numbe o bounding boxes ha con ain he cen e keypoin and uses his in o ma ion o p edic he likelihood o a cen al egion in he image con aining he same class o cen e keypoin s. Each pixel in he hea map ep esen s he p obabili y o an objec cen e being p esen a he co esponding spa ial loca ion in he image. A high p obabili y sco e indica es a high likelihood o an objec cen e being p esen a ha loca ion, while a low p obabili y sco e indica es a low likelihood o an objec cen e is p esen . This app oach o p edic ing he likelihood o objec cen e s using a hea map can imp o e he accu acy o objec de ec ion, pa icula ly o small o occluded objec s. 13 Sou ce: Liu, W., Anguelo , D., E han, D., Szegedy, C., Reed, S., Fu, C. Y., & Be g, A. C. (2016). Ssd: Single sho mul ibox de ec o . In Compu e Vision–ECCV 2016: 14 h Eu opean Con e ence, Ams e dam, The Ne he lands, Oc obe 11–14, 2016, P oceedings, Pa I 14 (pp. 21-37). Sp inge In e na ional Publishing. 19 Figu e 17 - Cen e Ne A chi ec u e 14 The au ho s p esen ed an a chi ec u e ha employs wo key s a egies o p edic he geome ic cen e s and co ne s o objec s. The i s s a egy is called Cen e Pooling and s a s by p edic ing he cen e keypoin by inding he maximum alue in bo h ho izon al and e ical di ec ions and adding hese wo alues oge he . This helps o ob ain mo e ecognizable isual pa e ns wi hin objec s, making i easie o pe cei e he cen al pa o he p oposal. The second s a egy is denomina ed Cascade Co ne Pooling, which aims o ind he co ne poin s simila ly o he p e ious one, bu now by ob aining he maximum summed esponse in bo h he bounda y and in e nal di ec ions o objec s on a ea u e map o co ne p edic ion. Empi ically, he au ho s claim ha he esul s a e mo e s able and obus o ea u e-le el noises, leading o imp o ed p ecision and ecall. In summa y, he Cen e Ne model is a s a e-o - he-a objec de ec ion a chi ec u e ha achie es high accu acy wi h low compu a ional cos s. The model p edic s objec cen e s di ec ly using a hea map, which educes he complexi y o he model and imp o es de ec ion accu acy o small and occluded objec s. The model has achie ed s a e-o - he-a esul s on se e al benchma k da ase s and is widely used in bo h esea ch and indus y o a ious compu e ision applica ions. 2.5. EVALUATION METRICS E alua ion me ics play a c ucial ole in he de elopmen and deploymen o machine lea ning models. Building a machine lea ning model can be seen as a ecu si e pipeline whe e he esponsible eam ains a model, hen e alua e i using e alua ion me ics, ine une speci ic model hype pa ame e s, and epea his p ocedu e un il he desi ed pe o mance is achie ed. So, i is ex emely impo an o ca e ully choose and ack he igh e alua ion me ics o ensu e ha he model is pe o ming well, o pe o m a model compa ison, and o mee he desi ed objec i es. In his chap e , we will discuss he heo e ical backg ound o he e alua ion me ics used h oughou he p ojec and explo e how o choose and in e p e hese me ics in a p oduc ion en i onmen . We will also discuss he impo ance o conside ing he speci ic cha ac e is ics and goals o he ask when selec ing an e alua ion me ic, and he ole o e alua ion me ics in model selec ion, op imiza ion, and moni o ing. 14 Sou ce: Duan, K., Bai, S., Xie, L., Qi, H., Huang, Q., & Tian, Q. (2019). Cen e ne : Keypoin iple s o objec de ec ion. In P oceedings o he IEEE/CVF in e na ional con e ence on compu e ision (pp. 6569-6578). 20 Since we a e dealing wi h an objec de ec ion algo i hm, is expec ed o e alua e he model by compa ing i s ou pu wi h g ound- u h bounding boxes gene a ed by a human ope a o esponsible o manually labeling he objec s and de ining hei bounda ies (V. Y. Ma iano e al., 2002). F om his analysis, i ’s common o use e alua ion me ics like accu acy, p ecision, ecall, and many o he s o unde s and how well he model can dis inguish be ween di e en class labels and he ade-o s be ween alse posi i e and alse nega i e p edic ions. Since 2010 challenges like VOC PASCAL Challenge, COCO, ImageNe Objec De ec ion Challenge, and Google Open Images Challenge ha e eme ged as compe i ions ha aim o e alua e new implemen a ions and a e now seen as aluable benchma ks o es objec de ec ion models in speci ic scena ios by using eal-wo ld anno a ed da ase s (M. E e ingham e al., 2009). These compe i ions con ibu e o he de ini ion o s anda d e alua ion p ocedu es wi hin he scien i ic communi y by using popula me ics like A e age P ecision (AP), In e sec ion o e Union (IoU), o c ea ing hei a ian s o ank he models. In he nex sec ions, we’ll y o cla i y he me ics p e iously men ioned in a simpli ied and o ganized s a egy whe e he eade is guided by a sequen ial p ocess illus a ed in he igu e below. I should be no ed ha each s ep can be conside ed as a s andalone me hod o e alua ion, bu hey a e in e dependen and build upon one ano he . Figu e 18 - Objec De ec ion E alua ion P ocess 2.5.1. Con usion Ma ix This able comes up as a use ul ool o an ini ial unde s anding o he ypes o e o s made by a model and o compa ing he pe o mance o di e en models. I ’s cons uc ed by compa ing he p edic ed class labels wi h he ue class labels o he da a. 21 T ue Class P edic ed Class Posi i e Nega i e Posi i e TP FP Nega i e FN TN Table 1 - Con usion Ma ix 15 Looking a he able abo e, a con usion ma ix will be composed by ou main a eas ha help o unde s and he s eng hs and weaknesses o a model and o iden i ying a eas o imp o emen . Mo e p ecisely, each p edic ion can be classi ied as: • T ue posi i e (TP) i he p edic ion con ains a co ec de ec ion o a g ound- u h bounding box. • False posi i e (FP) i he algo i hm pe o ms an inco ec de ec ion o a nonexis en objec o a misplaced de ec ion o an exis ing objec . • False nega i e (FN) i he g ound- u h objec is no de ec ed. Bea in mind ha , wi hin he ealm o objec de ec ion, he e is no such hing as a False Posi i e (FP) as he e a e an in ini e numbe o bounding boxes ha should no be iden i ied wi hin a gi en image. 2.5.2. In e sec ion o e Union (IoU) Based on he Jacca d Index, his me ic indi idually e alua es he o e lap be ween a p e-anno a ed g ound u h bounding box (g ) and he one p edic ed by he model (pd) ha ing in conside a ion a speci ic h eshold alue. IoU sco e anges be ween 0 and 1 whe e he close he wo boxes he highe he sco e, meaning ha de ec ion ou pu wi h maximum IoU alue is conside ed o ha e a pe ec o e lap wi h he g ound u h bounding box. Figu e 19 - Fo mula and Rep esen a ion o he IoU calcula ion 16 15 Adap ed om: Jeppesen, J. H., Jacobsen, R. H., Inceoglu, F., & To egaa d, T. S. (2019). A cloud de ec ion algo i hm o sa elli eimage y based on deep lea ning. Remo e sensing o en i onmen , 229, 247-259. 16 Adap ed om: Padilla, R., Ne o, S. L., & Da Sil a, E. A. (2020, July). A su ey on pe o mance me ics o objec -de ec ion algo i hms. In 2020 in e na ional con e ence on sys ems, signals and image p ocessing (IWSSIP) (pp. 237-242). IEEE. 22 By compu ing he IoU sco e o each de ec ion, i ’s impo an o de ine a h eshold (a) o con e eal- alued sco es in o classi ica ions. 2.5.3. P ecision and Recall This pe o mance benchma k ac as a measu e o how closely a p edic ed alue ag ees wi h he ue alue o a quan i y. I is de ined as he a io o he numbe o co ec ly p edic ed posi i e cases (posi i e p edic ions con aining an IoU g ea e han he h eshold) o he o al numbe o p edic ed posi i e cases (co ec ly iden i ied objec s + alse posi i e p edic ions). In o he wo ds, i is a measu e o he p opo ion o posi i e cases ha a e ac ually posi i e. Figu e 20 - P ecision Fo mula P ecision is o en used in conjunc ion wi h ano he me ic called Recall (o Sensi i y) o e alua e he pe o mance o a model. P ecision is ypically mo e ele an when he goal is o limi he numbe o alse posi i e p edic ions, whe eas ecall is mo e ele an when he goal is o iden i y as many posi i e cases as possible. This way we can de ine Recall as he me ic ha measu es he a io o he numbe o co ec ly p edic ed posi i e cases o he o al numbe o ac ual posi i e cases. In o he wo ds, i is a measu e o he p opo ion o ac ual posi i e cases ha a e co ec ly iden i ied by he model. Figu e 21 - Recall Fo mula Impo an o men ion ha , in he p esence o imbalanced da ase s, whe e one class is a e, i ’s impo an o ocus on he ecall o he model. 2.5.4. F1-Sco e: This is he ha monic mean o p ecision and ecall, wi h a highe sco e indica ing a be e balance be ween he wo. The F1 sco e is de ined as: Figu e 22 - F1-Sco e Fo mula The F1 sco e is o en used in imbalanced classi ica ion asks, whe e i is impo an o achie e a balance be ween p ecision and ecall. I is also use ul when he cos o alse posi i e and alse nega i e p edic ions is no he same, as i allows o he weigh ing o hese cos s o be inco po a ed in o he e alua ion me ic. 2.5.5. A e age P ecision In objec de ec ion algo i hms, i is essen ial o s ike a balance be ween p ecision and ecall. The adi ional e alua ion me ic, which is he F1 sco e, can only p o ide a single sco e o a gi en 23 h eshold, making i less in o ma i e in si ua ions whe e he h eshold a ies. To add ess his issue, he A e age P ecision (AP) me ic was in oduced, which has become a widely used e alua ion me ic o objec de ec ion algo i hms. AP conside s he numbe o ue posi i es, alse posi i es, and alse nega i es, making i pa icula ly use ul when he da a is imbalanced. By compu ing he a ea unde he p ecision- ecall cu e (AUC), AP p o ides a comp ehensi e summa y o he ade-o be ween p ecision and ecall ac oss di e en h esholds, hus p o iding a mo e in o ma i e e alua ion me ic o objec de ec ion algo i hms. To calcula e he AP, he p ecision- ecall cu e is i s compu ed by a ying he h eshold alues and plo ing he p ecision on he y-axis and ecall on he x-axis. The AUC is hen compu ed, which anges om 0 o 1, wi h highe alues indica ing be e pe o mance. AP p o ides a mo e p ecise e alua ion me ic compa ed o adi ional me ics, making i a aluable ool o objec de ec ion algo i hms. As such, i has become a s anda d me ic used in many objec de ec ion benchma ks and compe i ions. 2.5.6. Mean A e age P ecision (mAP) Mul iclass Objec De ec ion models ace he challenge o de ec ing mul iple objec classes in an image. As a esul , a me ic called Mean A e age P ecision (mAP) was de eloped as an ex ension o he AP me ic o calcula e he a e age p ecision o each class. mAP calcula es he AP o each class in he da ase and hen a e ages he AP alues o all he classes o ob ain he inal mAP sco e. The esul ing alue anges om 0 o 1, wi h 1 indica ing he bes possible pe o mance. mAP is a use ul me ic as i p o ides an o e all iew o he model's pe o mance and helps o iden i y which classes he model is pe o ming well on and which classes i is s uggling wi h, p o iding insigh in o whe e he model needs imp o emen . The simplici y o mAP has made i an essen ial benchma k o e alua ing he pe o mance o objec de ec ion models in compe i ions and challenges. The mAP allows o easy compa ison o di e en models and p o ides a ai e alua ion me ic o all models, ega dless o he numbe o classes in he da ase . This makes i an e ec i e ool o esea che s and de elope s o assess he pe o mance o hei models and iden i y a eas o imp o emen . 2.6. OPTICAL CHARACTER RECOGNITION (OCR) Op ical Cha ac e Recogni ion (OCR) is a echnology ha p o ides he capaci y o con e ing handw i en, ypew i en, o p in ed ex in o machine- eadable images. This o e s nume ous applica ions ha can be used o imp o e wo k low e iciency. Common examples a e indus ies like legal, banking, and heal hca e ha a e cu en ly le e aging OCR echnology o simpli y hei ope a ions and s eamline hei p ocesses (A. Singh, K. Bacchuwa , & A. Bhasin., 2012). Unlike humans, machines do no ha e he capabili y o ecognize ex o cha ac e s easily om an image, which is why signi ican esea ch e o s ha e been pu in o de eloping OCR echniques. OCR is a complex p oblem due o he nume ous languages, on s, and s yles in which ex can be w i en, as well as he complex ules o languages. As a esul , OCR equi es he in eg a ion o a ious compu e science disciplines, such as image p ocessing, pa e n classi ica ion, and na u al language p ocessing, o o e come hese challenges (N. Islam, Z. Islam, & N. Noo ., 2017). The p ocess o OCR in ol es a se ies o dis inc phases. The i s one, denomina ed by image acquisi ion, en ails ob aining an image om an ex e nal sou ce such as a came a o scanne , and 24 ans o ming i in o a o ma ha is compa ible wi h compu e p ocessing. This is a c ucial s ep in he OCR p ocess as i se s he ounda ion o accu a e and e icien op ical cha ac e ecogni ion. Once he image has been acqui ed, he subsequen s ep in he OCR p ocess is known as p ep ocessing. Du ing his phase, a a ie y o echniques can be u ilized o enhance he quali y o he image. These echniques may in ol e emo ing noise, se ing h esholds, and ex ac ing he baseline o he image. By imp o ing he image quali y, he OCR algo i hm can achie e a highe le el o accu acy when i comes o ecognizing he ex o cha ac e s con ained wi hin he image. This phase is c i ical o ensu ing he o e all e ec i eness o he OCR p ocess. Mo ing o he nex phase, he nex s ep in he OCR p ocess will ocus on cha ac e segmen a ion. The goal o his phase is o isola e indi idual cha ac e s wi hin he image so ha hey can be p ope ly iden i ied by he ecogni ion engine. While simple echniques such as connec ed componen analysis and p ojec ion p o iles may be su icien in ce ain cases, mo e ad anced segmen a ion echniques a e necessa y o complex si ua ions whe e cha ac e s may be o e lapping, b oken, o obscu ed by noise wi hin he image. By e ec i ely sepa a ing he cha ac e s, he OCR sys em can accu a ely ecognize he ex and p oduce an ou pu ha is ai h ul o he o iginal documen . The segmen ed cha ac e s a e hen p ocessed o ex ac di e en ea u es. Based on hese ea u es, he cha ac e s a e ecognized. Di e en ypes o ea u es ha can be used ex ac ed om images a e momen s e c. The ex ac ed ea u es should be e icien ly compu able, minimize in a-class a ia ions, and maximizes in e -class a ia ions. The segmen ed cha ac e s a e hen p ocessed in o de o ex ac di e en ea u es ha will be used o cha ac e ecogni ion. Based on hese ea u es, he cha ac e s a e ecognized. The ea u es ex ac ed om he cha ac e s should be e icien ly compu able and should minimize in a-class a ia ions ( a ia ions wi hin he same class o cha ac e s) while maximizing in e -class a ia ions ( a ia ions be ween di e en classes o cha ac e s). Va ious ypes o ea u es can be ex ac ed om he cha ac e s, such as momen s and o he image-based ea u es. A e ha , he ea u es o segmen ed images a e assigned o di e en ca ego ies o classes using di e en ypes o cha ac e classi ica ion echniques. Finally, pos -p ocessing echniques can be pe o med o imp o e he accu acy o OCR sys ems. These echniques u ilize na u al language p ocessing, and geome ic and linguis ic con ex o co ec e o s in OCR esul s. 25 Figu e 23 – Gene al OCR Wo k low 17 O e he yea s, OCR echnology has unde gone emendous imp o emen s, and mode n sys ems can now ecognize a ious on s, sizes, and s yles o ex . Despi e his, he accu acy o OCR sys ems s ill depends on ac o s such as image quali y, ex complexi y, and language. Fo una ely, ad ancemen s in a i icial in elligence and machine lea ning ha e esul ed in subs an ial imp o emen s in OCR accu acy, leading o mo e eliable and p ecise esul s (A. Ranjan, V.N.J. Behe a, & M. Reza., 2021). Nowadays, he e a e nume ous OCR ools a ailable ha ha e made hei usage mo e common in di e en en i onmen s. Some o he mos popula OCR ools include Tesse ac , de eloped by Google, Mic oso Cogni i e Se ices by Mic oso , and Amazon Tex ac . Fo ou p ojec , we ha e decided o u ilize he Py esse ac Py hon lib a y, which p o ides an easy- o-use in e ace o le e age all he ea u es and capabili ies o Tesse ac . The Py esse ac lib a y o e s a s aigh o wa d and e icien way o in eg a ing OCR unc ionali y in o Py hon-based applica ions, making i a g ea choice o ou p ojec needs. 17 Adap ed om: Mudia a, I. M. D. R., A maja, I. M. D. S., Suha sana, I. K., An a a, I. W. G. S., Bha adi ya, I. W. P., Suandi a , G. A., & Ind awan, G. (2020, Ap il). Balinese cha ac e ecogni ion on mobile applica ion based on esse ac open-sou ce OCR engine. In Jou nal o Physics: Con e ence Se ies (Vol. 1516, No. 1, p. 012017). IOP Publishing. 26 3. METHODOLOGY 3.1. TOOLS AND TECHNOLOGIES In his chap e , all he ools and echnologies used h oughou his p ojec will be p esen ed. We will explo e he easons o hei selec ion, enume a e al e na i es a ailable in he ma ke , and unde s and he impo ance o hese ools and echnologies o he success o he p ojec . The chap e will co e he a ious lib a ies, amewo ks, and pla o ms ha ha e been employed o acili a e he p ocess o building an objec de ec ion model, om da a p epa a ion o model aining and deploymen . By he end o his chap e , eade s will ha e a comp ehensi e unde s anding o he ools and echnologies used in he p ojec , and how hey con ibu e o he o e all objec i e o he p ojec . 3.1.1. Tenso Flow Tenso Flow (TF) is a powe ul open-sou ce so wa e lib a y o machine lea ning and deep lea ning, de eloped by esea che s and enginee s wo king on he Google B ain Team. A i s co e, TF uses da a low g aphs o ep esen any compu a ion. A da a low g aph is a di ec ed acyclic g aph (DAG) whe e he edges ep esen he low o da a, and he nodes ep esen ope a ions. This makes i easy o implemen machine lea ning algo i hms, such as neu al ne wo ks (M. Abadi e al., 2016). Tenso Flow also p o ides a wide ange o ools o building, aining, and deploying machine lea ning models using a a ie y o pla o ms including CPUs, GPUs, and TPUs. Howe e , he de elopmen eam main ains a collec ion o p e-made models on a pla o m called Tenso Flow Hub, which can be used o asks like objec de ec ion, image classi ica ion, and ex gene a ion. Fo isualiza ion pu poses, Tenso Flow has a isualiza ion ool called Tenso Boa d, which allows de elope s o easily isualize and unde s and he beha io o hei models du ing aining, e alua ion, and in e ence. Figu e 24 - Tenso Boa d In e ace 18 The main al e na i e o Tenso Flow is PyTo ch. This py hon lib a y was de eloped by Facebook and de ines i sel as an easie - o-use ool easy o asks such as image classi ica ion, na u al language 18 Sou ce: h ps://www. enso low.o g/ enso boa d 33 o da a o a new ask is limi ed, as ans e lea ning educes he need o ex ensi e da a collec ion and labeling. Luckily, Tenso Flow o e s a eposi o y ha con ains se e al model a chi ec u es p e iously ained and accompanied by hei e alua ions when used in he COCO da ase . This way we s a ed o esea ch which ones me ou needs seeking a model wi h consis en pe o mance bu a he same ime no p esen ing a high in e ence ime as his would nega i ely a ec he p ocessing ime o he OCR pipeline. To o ganize he p ojec and obse e imp o emen s o e ime, we op o di ide i in o e sions. Each e sion would p esen a di e en g oup o models alongside da a modi ica ions ha we belie ed ha made sense acco ding o he ou pu s e ained. In his way, i was possible o e ol e ou wo k while deli e ing epo s o he Biome id eam a he end o each mon h. Below is displayed a able illus a ing all he models es ed pe e sion. Models Obse a ions Ve sion 1 E icien De D1 SSD ResNe 50 SSD ResNe 101 SSD ResNe 152 Ve sion 2 SSD ResNe 152 1024x1024 SSD ResNe 50 2 Imp o ed e sion wi h di e en hype pa ame e s SSD ResNe 101 2 Imp o ed e sion wi h di e en hype pa ame e s Ve sion 3 Fas e R-CNN Incep ion E icien De D2 Cen e Ne Hou Glass104 Ve sion 4 Fas e R-CNN Incep ion 2 Imp o ed e sion wi h di e en hype pa ame e s Cen e Ne Resne 50 Cen e Ne Resne 101 Table 2 - Model's a chi ec u e used pe e sion. I 's c ucial o emphasize ha he model’s key aspec s ha e o be al e ed o adap o he app op ia e da a o ma o ou p ojec . This in ol ed modi ying each model's con igu a ion ile o enhance he de ec ion o objec s based on he quan i y and shape o he bounding boxes. 34 To measu e he pe o mance o ou models, we ocused on e alua ing hem using me ics explained p e iously such as mAP, Recall, and IoU. Fo his pu pose, he e alua ion me ics g oup was se o “coco_de ec ion_me ics” in he e al_con ig pa ame e . Al hough he lib a y p o ides o he e alua ion me ics, we ound his o be he mos app op ia e o ou p ojec . The model aining p ocedu e was ho oughly documen ed, making i possible o ini ia e he i s aining i e a ion a e inse ing he da a and label maps. This ga e us a p elimina y unde s anding o he model's capabili ies. The aining p ocess was moni o ed by no only he console logs bu also h ough he use o Tenso Boa d. This allowed o comp ehensi e acking o he aining e olu ion. 3.2.3. Model Fine-Tuning In he ini ial i e a ion o he models, we solely elied on he aining and es da a con e ed o TFReco ds o ma wi hou making any modi ica ions o he model a chi ec u e. None heless, we conduc ed esea ch o iden i y he op imal pa ame e s sui ed o ou scena io. This led o he nex phase o he solu ion's de elopmen known as ine uning he model, whe e du ing each aining session, small modi ica ions a e made o he model's a chi ec u e o enhance i s e icacy. In his sec ion, he main objec i e is o discuss he a ious modi ica ions ha he eam made o he model's con igu a ion ile o ob ain he op imal a chi ec u e o ou pa icula p oblem. The ocus will be on he da ase composi ion, and a de ailed o e iew will be p o ided o each o he hemes con ained wi hin he con igu a ion ile. Th oughou his p ocess, we will explain he easoning behind all he choices and p o ide insigh in o he a ious op ions ha a e a ailable h ough his lib a y. 3.2.3.1. Image P ep ocessing In he con ex o any compu e ision p ojec , i is absolu ely c ucial o ha e a ull unde s anding o he inpu da a ha is being ed in o he model. Wi hou his unde s anding, i is impossible o design and ain a model ha is uly e ec i e a he ask a hand. Deep lea ning algo i hms ha powe his kind o p ojec a e hea ily dependen on he inpu da a, and e en small a ia ions o inconsis encies in he da a can g ea ly impac he pe o mance o he model. I is he e o e essen ial o ca e ully p ep ocess he da a, ensu ing ha i is in a o ma ha is compa ible wi h he model's a chi ec u e. Fo una ely, he TFOD API p o ides a ange o p ep ocessing ools ha can be accessed igh ou o he box. In he ini ial sec ion o he con igu a ion ile, use s ha e he abili y o exe cise con ol o e he image esizing p ocess, including he shape o he esul ing esized image. The lib a y o e s a conside able amoun o lexibili y conce ning he a ious pa ame e s ha can be modi ied o achie e he desi ed esul s. Howe e , i is impo an o no e ha be o e any modi ica ions a e applied, Tenso Flow au oma ically handles he no maliza ion o he images. This is an impo an ea u e ha ensu es consis ency in he da a and helps o op imize he pe o mance o he model. When con igu ing he s a egy chosen o p ocess he inpu image o ou model, he API o e s wo dis inc op ions: "keep_aspec _ a io_ esize " and " ixed_shape_ esize ". The o me op ion allows use s o speci y a minimum size while main aining he aspec a io o he o iginal image. Howe e , i is impo an o no e ha his op ion can some imes esul in ex ensi ely padded images, especially in cases whe e he o iginal image is ec angula . In such cases, i may be mo e sui able o use he " ixed_shape_ esize " op ion, which esizes he image o a speci ied ec angle size de ined by he 35 "heigh " and "wid h" pa ame e s. Fo ou speci ic use case, we ha e de e mined ha he " ixed_shape_ esize " op ion is he op imal choice, as i allows us o main ain consis ency in he size o he inpu images, which is c i ical o op imizing he pe o mance o he model. 3.2.3.2. Image Augmen a ion Image augmen a ion is an essen ial echnique ha can g ea ly enhance he accu acy and obus ness o he model. By applying a ious ans o ma ions and manipula ions o he inpu images, augmen a ion p ocedu es can help o add ess issues such as o e i ing, insu icien aining da a, and class imbalance. Fo example, lipping, o a ing, and c opping images can inc ease he a iabili y and di e si y o he aining da a while adjus ing b igh ness, con as , and colo can help o accoun o a ia ions in ligh ing condi ions. Addi ionally, image augmen a ion can also help o mi iga e he e ec s o occlusions, iewpoin changes, and o he eal-wo ld ac o s ha may impac he model's abili y o accu a ely de ec objec s in new and unseen images. Th oughou he cou se o his p ojec , we explo ed a ious o hese me hods o augmen ou da ase wi h syn he ic images gene a ed h ough he applica ion o image manipula ion echniques. Knowing ha ou images we e accu a ely segmen ed and co ec ly o ien ed om he segmen a ion and o a ion models, we made a delibe a e e o o a oid modi ying hei o ien a ion. Ins ead, we ocused on applying echniques ha could al e he colo sys em o he da a as you can check in he image below. This decision p o ed o be highly e ec i e, as adjus men s o ac o s such as hue, con as , sa u a ion, and b igh ness allowed us o minimize he model's sensi i i y o colo . As a esul , he model was able o de ec objec s mo e accu a ely ac oss a wide ange o colo s, leading o imp o ed o e all pe o mance. The use o hese echniques demons a es he impo ance o ca e ul and s a egic da a augmen a ion in he con ex o objec de ec ion and unde sco es he c i ical ole ha i plays in op imizing model pe o mance. Figu e 27 - Image Augmen a ion echniques used o he Cen e Ne con igu a ion ile While he echniques desc ibed abo e can be highly e ec i e o augmen ing image da a in ou con ex , i 's wo h no ing ha he Tenso Flow Objec De ec ion API o e s a much b oade ange o augmen a ion op ions. As you can see in he image below, he e a e a wide a ie y o di e en 36 echniques ha can be used o manipula e and enhance image da a, anging om simple colo adjus men s o mo e complex ans o ma ions like o a ion, lipping, and dis o ion. By ca e ully conside ing he unique cha ac e is ics and equi emen s o you speci ic use case, you can le e age hese powe ul ools o c ea e highly cus omized and e ec i e da a augmen a ion s a egies ha can signi ican ly imp o e he accu acy and pe o mance o you objec de ec ion models. Figu e 28 - Image Augmen a ion echniques p o ided by Tenso low Objec De ec ion API 20 3.2.3.3. Pos -p ocessing Pos -p ocessing is ano he indispensable s ep in any objec de ec ion p ojec , as i enables he model o gene a e accu a e and meaning ul esul s om he aw ou pu o he de ec ion algo i hm. In he con ex o objec de ec ion, pos -p ocessing ypically in ol es analyzing he ou pu o he model and applying a ious echniques o e ine and il e he esul s. By ca e ully ailo ing hese pos - p ocessing echniques o he speci ic equi emen s o he p ojec , i 's possible o signi ican ly imp o e he accu acy and p ecision o he objec de ec ion model, while also educing he isk o alse posi i es o o he e o s. As we a e acing an ancho less a chi ec u e, ou p ima y conce n a his s age was p e en ing o e i ing and imp o ing model pe o mance. To achie e his, we ocused on ad anced app oaches o lea ning a e con ol o e ime. Lea ning a e plays a c i ical ole in de e mining he a e a which he model's in e nal pa ame e s and weigh s a e upda ed du ing aining. A ca e ully chosen lea ning a e can signi ican ly impac he model's con e gence and i s abili y o ind he op imal se o pa ame e s o he gi en ask. As such, we employed a ious s a egies o une he lea ning a e, 20 Sou ce: Tenso low Model Ga den Reposi o y 37 including lea ning a e schedules and op imiza ion algo i hms. Th ough hese echniques, we aimed o s ike a balance be ween model con e gence and gene aliza ion, ul ima ely leading o imp o ed pe o mance. In ou case, we employed he cosine lea ning a e decay schedule o con olling he lea ning a e o e ime. This app oach enables he lea ning a e alue o al e na e be ween inc easing and dec easing h oughou he aining p ocess. Speci ically, he lea ning a e is g adually dec eased owa ds ze o as he aining p og esses, which helps he model con e ge owa d an op imal solu ion while educing he isk o o e i ing. To co ec ly con igu e he lea ning a e schedule, i is impo an o ocus on he ollowing pa ame e s and unde s and hei impac o hem on he pe o mance o he model: • lea ning_ a e_base: pa ame e ha de ines he ini ial lea ning a e ha will be used o ain you model. • o al_s eps: pa ame e ha de ines he numbe o o al s eps you model is going o ain. Impo an o no e ha in he las s eps o you aining job, he lea ning a e schedule will d i e he lea ning a e alue o be close o ze o. • wa mup_lea ning_ a e: he maximum alue ha he lea ning a e will each be o e s a ing o dec ease. • wa mup_s eps: de ines he numbe o s eps ha will be aken o inc ease he lea ning a e om lea ning_ a e_base o wa mup_lea ning_ a e 3.2.4. Image Pipeline Once we had p ope ly ained he model on ou cus om da ase and ine- uned i s pa ame e s o enhance i s pe o mance, he subsequen phase in ol ed comp ehending he ou comes p oduced by he model. Fo e e y examined image, he model deli e s nume ous esul s, wi h he i s ones being he labels assigned o each sec ion. This is c ucial o linking he sec ion's name wi h he coo dina es o he bounding boxes ha will e en ually be showcased. The bounding box coo dina es gene a ed by he model du ing image in e ence play a i al ole in de e mining he loca ion o he egions o in e es . U ilizing hese coo dina es makes i easible o isola e speci ic subsec ions o he images ha only con ain he desi ed in o ma ion. In ou scena io, we aim o ex ac sec ions o he image ha exclusi ely con ain pe sonal da a p esen ed on he on ace o he Po uguese iden i ica ion ca d. To achie e his objec i e, we designed a da a pipeline ha in ol es conduc ing an image in e ence and subsequen ly u ilizing he model's ou comes o c op he o iginal image, leading o se e al sub- images, each co esponding o a pa icula sec ion. Ou wo k low delibe a ely in ol es minimal image manipula ion echniques since he model is adep a p ocessing ho izon al segmen ed images and can accommoda e colo a ia ions owing o he p e-applied image augmen a ion echniques. This ep esen s one o he key bene i s o ou implemen a ion o e he cu en app oach, as we can simply esize he image and eed i in o he model, educing he eliance on ex ensi e image 38 p ocessing ha can lead o la ency issues. As a esul , ou ML-powe ed solu ion ope a es independen ly o any ime-consuming image p ocessing, enhancing i s o e all e iciency. In he ollowing pa ag aphs, we will discuss each unc ion ha cons i u es ou pipeline. Ou pipeline was en i ely coded in Py hon and c ea ed en i ely by he Machine Lea ning eam, which modi ied he exis ing unc ions o accommoda e he changes in oduced by he objec de ec ion algo i hm. The ini ial unc ion in ou pipeline is called "de ec _sec ionsML" and only needs a pa ame e ha e e s o he image ha will be analyzed. Fo he unc ion o ope a e co ec ly, i mus ecei e an image o a Po uguese iden i ica ion ca d, which will be ed in o ou model o gene a e a p edic ion. Al hough he model gene a es se e al pieces o in o ma ion, he unc ion is designed o solely e u n he alues co esponding o he bounding box coo dina es and hei espec i e classes/labels. Figu e 29 – Py hon unc ion used o make model in e ences Wi h a clea unde s anding o he p eceding unc ion's ou comes, we will now p oceed o c op he o iginal image u ilizing he "c op_sec ions_ML" unc ion, esul ing in a lis o images, wi h each elemen co esponding o a pa icula sec ion. Figu e 30 – Py hon unc ion used o c op ca d sec ions by gi en coo dina es Concluding ou pipeline is he "ex ac _sec ions_da a_ML" unc ion, which plays a undamen al ole in p ocessing he images o each sec ion. This unc ion employs he Py esse ac lib a y o con e images in o ex . Howe e , as each sec ion can display da a in a ious o ma s, such as numbe s and punc ua ion ma ks (as in he case o heigh ) o solely le e s (as in ields like i s o las name), each 39 sec ion equi es a dis inc Tesse ac con igu a ion. As a esul , his unc ion me ges he mul iple con igu a ions wi h hei espec i e sec ions and c ea es a Py hon dic iona y o sa e he esul s. Figu e 31 – Py hon unc ion used o apply OCR in gi en sec ions 40 4. EXPERIMENTAL STUDY In his chap e , we will p esen he ou comes o ou expe imen a ion wi h a ious model a chi ec u es and de e mine he op imal one o deploymen in a p oduc ion se ing. We will s a by ou lining he e alua ion me ics used, and hei ela ionship o he concep s in oduced in chap e 2.5. Following ha , we will end by p o iding a de ailed analysis o he compa a i e pe o mance o ou bes model agains he exis ing p oduc ion s a egy. 4.1. EVALUATION PROTOCOL Th oughou ou objec de ec ion model aining, Tenso Flow gene a es eal- ime aining p ocess checkpoin log iles which allow us o assess he pe o mance o ou models. These logs con ain COCO e alua ion me ics, which include essen ial me ics like mean a e age p ecision, ecall, and In e sec ion o e Union. These me ics aid us in quan i ying he accu acy and p ecision o ou models and enable us o iden i y a eas ha equi e imp o emen . By using COCO e alua ion me ics, we can ensu e ha ou models a e op imized o de ec and classi y objec s wi h high accu acy and e iciency, allowing us o achie e ou esea ch goals. As we ha e p e iously men ioned, we spli ou da ase in o wo pa s - a aining se and a es se . The es se plays a c ucial ole in e alua ing he pe o mance o ou model, as i con ains da a ha he model has ne e seen be o e. Impo an o unde s and ha he e alua ion p ocess u ilizes he checkpoin iles gene a ed du ing he aining p ocess o assess he model's abili y o de ec objec s in he es da ase . The e alua ion gene a es a se o me ics ha p o ide a summa y o he model's pe o mance, enabling us o ack i s accu acy and p ecision o e ime. These me ics o e aluable insigh s in o he s eng hs and weaknesses o ou model, allowing us o ine- une i o op imal pe o mance. By egula ly moni o ing he e alua ion me ics, we can ensu e ha ou models a e con inuously imp o ing and deli e ing eliable esul s (L. Vladimi o , 2020). All hese esul s can also be isualized wi h he help o Tenso Boa d. This isualiza ion ool con e s he e alua ion esul s ob ained om he checkpoin iles in o in ui i e and in o ma i e dashboa ds. These dashboa ds p o ide a comp ehensi e o e iew o he model's pe o mance, enabling us o iden i y a eas ha equi e imp o emen . In addi ion o he e alua ion me ics, Tenso Boa d also allows us o isualize he de ec ion esul s in he es images, p o iding a clea unde s anding o how well he bounding boxes a e de ec ing he a eas o in e es . This ea u e is pa icula ly use ul in iden i ying alse posi i es o alse nega i es, which can be u he in es iga ed and co ec ed o imp o e he model's accu acy. By u ilizing Tenso Boa d o analyze he e alua ion esul s and isualize he de ec ion ou pu s, we can gain a deepe unde s anding o ou model's pe o mance and make in o med decisions on how o op imize i u he . Once we es ablish ha a model p oduces a o able ou comes, i becomes c ucial o assess he la ency i in oduces while making an in e ence. This aspec holds immense signi icance since i can ad e sely impac he pipeline's pe o mance i he esponse ime inc eases signi ican ly. The e o e, i is essen ial o ca e ully sc u inize he model's in e ence ime o ensu e ha i mee s he equi emen s o he in ended applica ion. Finally, upon analyzing he model's pe o mance me ics o e he es se , checking he bounding boxes display and he la ency, he inal e alua ion phase now shi s ocus o assessing he ou pu s 41 gene a ed by he OCR engine. In ligh o his, he p ojec eam decided o build a web applica ion using S eamli o compa e he esul s o ou app oach agains he cu en p oduc ion en i onmen 's ou pu . The applica ion p o ides an in e ace ha you can check below, ha allows us o b owse h ough a olde o images depic ing iden i ica ion ca ds and manually check which sec ions Tesse ac accu a ely con e ed o ex using ick boxes. This e alua ion phase is undoub edly he mos ime-consuming since he e is no eco d o all he g ound u h alues o he documen s, and he pe o mance o he OCR engine mus be assessed by manually inspec ing each sec ion o de e mine i he ou pu ma ches he expec ed alues. Figu e 32 - S eamli Ou pu Compa ison App Upon e alua ing bo h me hods' esul s on ou 3000-image da ase , he applica ion gene a es a CSV ile, which we use o p oduce a line plo ha will be used o compa e isually he wo implemen a ions. We will p esen his plo below when analyzing he esul s. The e alua ion p ocess o his p ojec is comp ehensi ely ou lined, highligh ing each dis inc phase. The nex chap e will p o ide a de ailed o e iew o he ou comes ob ained in each o hese phases and explica e he a ionale behind selec ing he inal model. 4.2. EXPERIMENTAL RESULTS AND DISCUSSION This chap e ocuses on p esen ing he ou comes achie ed wi h ou a i icial in elligence algo i hm's implemen a ion o de ec ing Po uguese ID ca d sec ions. We ha e di ided he esul s in o h ee phases, as discussed in he p e ious chap e , o assess ou app oach's e ec i eness in di e en ways. 42 4.2.1. Tenso Boa d Resul s Th oughou each phase p esen ed in Table 2, we s a ed checking he pe o mance o each model ia Tenso Boa d by analyzing i s e alua ion me ics on he ain and es se . This app oach allowed us o moni o he aining p og ess o a speci ic model in eal ime and compa e i o p e iously ained models. Ou ini ial emphasis is on he e alua ion me ic known as loss, which is a alue assigned du ing he aining o a model ha aims o e lec he dispa i y be ween he model's p edic ed ou pu and he eal ou pu . The undamen al objec i e o he aining p ocess is o dec ease he loss me ic, indica ing ha he p edic ed ou pu should closely esemble he ac ual ou pu . P esen ed in he ollowing able a e he loss alues o he mos op imized e sions o each o he ained models. I is e iden ha all he loss alues a e qui e low, bu he Cen e Ne Resne 101 model ou pe o med he es , indica ing ha i s p edic ions a e he mos accu a e. This ini ial e alua ion posi ions i as a s ong con ende o he inal model, al hough we s ill need o analyze nume ous o he me ics o de e mine i i is he op imal choice. Model Loss alue Cen e Ne Hou Glass104 0.01957 E icien De 1 0.02583 E icien De 2 0.02957 Fas e R-CNN Incep ion 0.01372 SSD ResNe 101 0.04287 SSD ResNe 50 0.0577 Cen e Ne Resne 50 0.01219 Cen e Ne Resne 101 0.0106 Table 3 - Loss Compa ison Rega ding he me ics o mean a e age p ecision and ecall alues, he esul s we e ound o be dissimila . Among he e alua ed model a chi ec u es, only h ee achie ed alues highe han 80% o mean a e age p ecision, namely Cen e Ne Hou Glass104, Fas e R-CNN Incep ion, and Cen e Ne Resne 50. These models also pe o med excep ionally well in e ms o ecall alues. Thus, hese same h ee models we e conside ed he mos e ec i e ones o he gi en ask. A compa a i e iew o he eco ded alues o bo h hese e alua ion me ics is p esen ed in he ollowing able. 49 6. BIBLIOGRAPHY [1] Abadi, M., Ba ham, P., Chen, J., Chen, Z., Da is, A., Dean, J., ... & Zheng, X. (2016, No embe ). Tenso low: a sys em o la ge-scale machine lea ning. In Osdi (Vol. 16, No. 2016, pp. 265-283). [2] Ab aham, A. (2005). A i icial neu al ne wo ks. Handbook o measu ing sys em design. [3] Alexanda i, A. M., Sh ikuma , A., & Kundaje, A. (2017). Sepa able ully connec ed laye s imp o e deep lea ning models o genomics. BioRxi , 146431. [4] A iwala. (2022, Oc obe ). 9 Real-Wo ld P oblems ha can be Sol ed by Machine Lea ning. Ma u i Techlabs. h ps://ma u i ech.com/p oblems-sol ed-machine-lea ning/ [5] A i issimo, F., Giaquin o, N., Sca pe a, M., & Spada ecchia, M. (2019, Oc obe ). An au oma ic eade o iden i y documen s. In 2019 IEEE In e na ional Con e ence on Sys ems, Man and Cybe ne ics (SMC) (pp. 3525-3530). IEEE. [6] Bha , D., Pa el, C., Talsania, H., Pa el, J., Vaghela, R., Pandya, S., ... & Ghay a , H. (2021). CNN a ian s o compu e ision: his o y, a chi ec u e, applica ion, challenges and u u e scope. Elec onics, 10(20), 2470. [7] Bughin, J., Hazan, E., Ramaswamy, S., Chui, M., Allas, T., Dahls om, P., ... & T ench, M. (2017). A i icial in elligence: The nex digi al on ie ? [8] Duan, K., Bai, S., Xie, L., Qi, H., Huang, Q., & Tian, Q. (2019). Cen e ne : Keypoin iple s o objec de ec ion. In P oceedings o he IEEE/CVF in e na ional con e ence on compu e ision (pp. 6569-6578). [9] E e ingham, M., Van Gool, L., Williams, C. K., Winn, J., & Zisse man, A. (2009). The pascal isual objec classes ( oc) challenge. In e na ional jou nal o compu e ision, 88, 303-308. [10] Fa ooq, O. (2022, Oc obe 3). [10 Bes Machine Lea ning S ocks o In es In]. Yahoo Finance. h ps:// inance.yahoo.com/news/10-bes -machine-lea ning-s ocks-165306920.h ml [11] Fle che , J., & Kos iainen, A. (2022, No embe ). E hical P inciples o Web Machine Lea ning. W3C. h ps://www.w3.o g/TR/webmachinelea ning-e hics [12] F i z Labs. (2018). Objec De ec ion Guide. F i z AI. h ps://www. i z.ai/objec -de ec ion/ [13] Gi shick, R., Donahue, J., Da ell, T., & Malik, J. (2014). Rich ea u e hie a chies o accu a e objec de ec ion and seman ic segmen a ion. In P oceedings o he IEEE con e ence on compu e ision and pa e n ecogni ion (pp. 580-587). [14] Good ellow, I., Bengio, Y., & Cou ille, A. (2016). Deep lea ning. MIT p ess. [15] H. Zhu, X. Yan, H. Tang, Y. Chang, B. Li and X. Yuan, "Mo ing Objec De ec ion wi h Deep CNNs," in IEEE Access, ol. 8, pp. 29729-29741, 2020, doi: 10.1109/ACCESS.2020.2972562. [16] Hakim, H., & Fadhil, A. (2021, Feb ua y). Su ey: Con olu ion neu al ne wo ks in objec de ec ion. In Jou nal o Physics: Con e ence Se ies (Vol. 1804, No. 1, p. 012095). IOP Publishing. 50 [17] Ho ak, K., & Sabla nig, R. (2019, Augus ). Deep lea ning concep s and da ase s o image ecogni ion: o e iew 2019. In Ele en h in e na ional con e ence on digi al image p ocessing (ICDIP 2019) (Vol. 11179, pp. 484-491). SPIE. [18] Huang, J. (2017, Augus 14). “Ou goal: E e y single ca will be au onomous.” Bosch Global. h ps://www.bosch.com/s o ies/ hough -leade -jensen-huang/ [19] Islam, N., Islam, Z., & Noo , N. (2017). A su ey on op ical cha ac e ecogni ion sys em. a Xi p ep in a Xi :1710.05703. [20] Janiesch, C., Zschech, P., & Hein ich, K. (2021). Machine lea ning and deep lea ning. Elec onic Ma ke s, 31(3), 685-695. [21] Janke, J., Cas elli, M., & Popo ič, A. (2019). Analysis o he p o iciency o ully connec ed neu al ne wo ks in he p ocess o classi ying digi al images. Benchma k o di e en classi ica ion algo i hms on high-le el image ea u es om con olu ional laye s. Expe Sys ems wi h Applica ions, 135, 12-38. [22] K izhe sky A., Su ske e I. & Hin on G.E., 2012. Imagene Classi ica ion wi h Deep Con olu ional Neu al Ne wo ks. Ad ances in Neu al In o ma ion P ocessing Sys ems (NIPS). :1097– 1105. [23] Liu, W., Anguelo , D., E han, D., Szegedy, C., Reed, S., Fu, C. Y., & Be g, A. C. (2016, Oc obe ). SSD: Single sho mul ibox de ec o . In Eu opean con e ence on compu e ision (pp. 21-37). Sp inge , Cham. [24] Ma iano, V. Y., Min, J., Pa k, J. H., Kas u i, R., Mihalcik, D., Li, H., ... & D aye , T. (2002, Augus ). Pe o mance e alua ion o objec de ec ion algo i hms. In 2002 In e na ional Con e ence on Pa e n Recogni ion (Vol. 3, pp. 965-969). IEEE. [25] Ma ikis, S. T., An onopoulos, C. P., Vo os, N. S., & Ke amidas, G. (2021, Sep embe ). Compa a i e e alua ion o compu e ision echnologies, a ge ing objec iden i ica ion and localiza ion scena ios. In 2021 6 h Sou h-Eas Eu ope Design Au oma ion, Compu e Enginee ing, Compu e Ne wo ks and Social Media Con e ence (SEEDA-CECNSM) (pp. 1-8). IEEE. [26] Mo oney, L. (2020). AI and Machine Lea ning o code s. O'Reilly Media. [27] Mudia a, I. M. D. R., A maja, I. M. D. S., Suha sana, I. K., An a a, I. W. G. S., Bha adi ya, I. W. P., Suandi a , G. A., & Ind awan, G. (2020, Ap il). Balinese cha ac e ecogni ion on mobile applica ion based on esse ac open-sou ce OCR engine. In Jou nal o Physics: Con e ence Se ies (Vol. 1516, No. 1, p. 012017). IOP Publishing. [28] Padilla, R., Ne o, S. L., & Da Sil a, E. A. (2020, July). A su ey on pe o mance me ics o objec -de ec ion algo i hms. In 2020 in e na ional con e ence on sys ems, signals and image p ocessing (IWSSIP) (pp. 237-242). IEEE. [29] Ranjan, A., Behe a, V. N. J., & Reza, M. (2021). Oc using compu e ision and machine lea ning. Machine Lea ning Algo i hms o Indus ial Applica ions, 83-105. 51 [30] Ren, S., He, K., Gi shick, R., & Sun, J. (2015). Fas e -cnn: Towa ds eal- ime objec de ec ion wi h egion p oposal ne wo ks. Ad ances in neu al in o ma ion p ocessing sys ems, 28. [31] Ryan, M., & Hana iah, N. (2015). An examina ion o cha ac e ecogni ion on ID ca d using empla e ma ching app oach. P ocedia Compu e Science, 59, 520-529. [32] Simeone, O. (2018). A e y b ie in oduc ion o machine lea ning wi h applica ions o communica ion sys ems. IEEE T ansac ions on Cogni i e Communica ions and Ne wo king, 4(4), 648- 664. [33] Singh, A., Bacchuwa , K., & Bhasin, A. (2012). A su ey o OCR applica ions. In e na ional Jou nal o Machine Lea ning and Compu ing, 2(3), 314. [34] Vladimi o , L. (2020). Tenso Flow 2 Objec De ec ion API Tu o ial. Tenso Flow 2 Objec De ec ion API Documen a ion. h ps:// enso low-objec -de ec ion-api- u o ial. ead hedocs.io/en/la es /index.h ml [35] Yamashi a, R., Nishio, M., Do, R. K. G., & Togashi, K. (2018). Con olu ional neu al ne wo ks: an o e iew and applica ion in adiology. Insigh s in o imaging, 9(4), 611–629. h ps://doi.o g/10.1007/s13244-018-0639-9 [36] Zulkhaiza , A. (2023, Janua y 4). The Fu u e o AI: How A i icial In elligence is Re olu ionizing he Way We Wo k - Digi al Fi s Magazine. Digi al Fi s Magazine. h ps://www.digi al i s magazine.com/ he- u u e-o -ai-how-a i icial-in elligence-is- e olu ionizing- he-way-we-wo k/