i
Da a Labeling ools o Compu e Vision:
Ped o Miguel Lima de Sousa Reis
a Re iew
Disse a ion p esen ed as pa ial equi emen o ob aining
he Mas e 's deg ee in Da a Science and Ad anced Analy ics
i
NOVA In o ma ion Managemen School
Ins i u o Supe io de Es a ís ica e Ges ão de In o mação
Uni e sidade No a de Lisboa
DATA LABELING TOOLS FOR COMPUTER VISION: A REVIEW
by
Ped o Miguel Lima de Sousa Reis
Disse a ion p esen ed as pa ial equi emen o ob aining he Mas e 's deg ee in Da a Science
and Ad anced Analy ics
Ad iso : Robe o And é Pe ei a Hen iques, PhD
No embe 2021
ii
To my belo ed Ca olina, who is he pu es and kindes human being I know, wi h whom my li e is
happie e e y day. To Paula and Miguel o hei endless lo e and suppo . To Rui and Ri a ha
always wo y and wonde abou wha and how I am cu en ly doing. To Manuel and Conceição Ma ia
o always being he e uncondi ionally, pushing me u he and belie ing in me since I was a child,
whose wo ds o encou agemen and enaci y s ill echo in my ea s. To Rui, who has always le me
s and on his shoulde s. To João, Alexand ino, Miguel, Pinhal, B i es and Sa abando – each one wi h
i s supe powe . This disse a ion is dedica ed o all my amily and iends. Finally, I’d also like o
acknowledge NTT DATA, o me ly known as e e is, and he Da a & Analy ics eam, o he suppo
and inspi a ion h oughou all my mas e ’s deg ee as a ull- ime wo ke -s uden .
iii
ABSTRACT
La ge olumes o labeled da a a e equi ed o ain Machine Lea ning models in o de o sol e
oday’s compu e ision challenges. The ecen exace ba ed hype and in es men in Da a Labeling
ools and se ices has led o many ad-hoc labeling ools. In his e iew, a de ailed compa ison be ween
a selec ion o da a labeling ools is amed o ensu e he bes so wa e choice o holis ically op imize
he da a labeling p ocess in a Compu e Vision p oblem. This analysis is buil on mul iple domains o
ea u es and unc ionali ies ela ed o Compu e Vision, Na u al Language P ocessing, Au oma ion,
and Quali y Assu ance, enabling i s applica ion o he mos p e alen da a labeling use cases ac oss
he scien i ic communi y and global ma ke .
KEYWORDS
Re iew; Compu e Vision; Image Anno a ion; Da a Labeling so wa e; Supe ised Machine
Lea ning; Me hodologies and Tools
i
INDEX
1. In oduc ion ........................................................................................................ 1
1.1. Da a Labeling ......................................................................................................... 2
2. His o ical O e iew ............................................................................................. 4
3. Li e a u e Re iew ............................................................................................... 9
4. Da a Labeling Tools ........................................................................................... 13
4.1. Compu e Vision ea u es.................................................................................... 16
4.2. Na u al Language P ocessing ea u es ................................................................ 18
4.3. Au oma ion and De elope - iendly ea u es ..................................................... 20
4.4. Managemen and Quali y Assu ance ea u es .................................................... 22
4.5. Gene al Compa a i e Analysis............................................................................. 24
5. Discussion ......................................................................................................... 25
6. Conclusions ....................................................................................................... 27
7. Limi a ions and ecommenda ions o u u e wo ks ....................................... 28
8. Bibliog aphy ...................................................................................................... 29
LIST OF FIGURES
Figu e 1 – E olu ion o he image gene a ion capabili ies by Gene a i e Ad e sa ial Ne wo ks
(GANs) om 2014 o 2018 (Saxena & Cao, 2020) .............................................................. 4
Figu e 2 – E olu ion o he mos ac i e esea ch opics in he Compu e Vision ield o e ime
(Szeliski, 2010) .................................................................................................................... 7
Figu e 3 – F equency ha he e m “Da a Science” was sea ched o on Google, in he wo ld
and in mul iple languages, om 2004 un il he p esen (Google, 2021) ........................... 8
Figu e 4 – Da a Labeling anno a ion ool ImageTagge (Fiedle e al., 2019) aming an
Objec De ec ion ask ......................................................................................................... 9
Figu e 5 – E olu ion o Image Classi ica ion models on ImageNe da ase : Top 1 Accu acy
(le ) and Top 5 Accu acy ( igh ) (Facebook AI Resea ch, 2021) ...................................... 10
Figu e 6 – Hype Cycle o A i icial In elligence 2021 (Ga ne Inc., 2021), as o July 2021..... 11
Figu e 7 – F equency ha he e m “Supe ised Machine Lea ning” was sea ched o on
Google, in he wo ld and in mul iple languages, om 2016 un il he p esen ................ 13
i
LIST OF TABLES
Table 1 – Manually selec ed da a anno a ion ools and espec i e de elope eams and
e e ences, i applicable ................................................................................................... 14
Table 2 – Compu e Vision ela ed ea u es in he selec ed da a anno a ion ools ................ 17
Table 3 – NLP- ela ed ea u es in he selec ed da a anno a ion ools .................................... 18
Table 4 – Au oma ion and de elope - iendly ea u es in he selec ed da a anno a ion ools20
Table 5 – Managemen and QA ea u es in he selec ed da a anno a ion ools ..................... 22
Table 6 – Compa a i e summa y o he ea u es g ouped by ca ego y ac oss he analyzed
da a anno a ion ools ....................................................................................................... 24
ii
LIST OF ABBREVIATIONS AND ACRONYMS
AGI A i icial Gene al In elligence
AI A i icial In elligence
API Applica ion P og amming In e ace
CAGR Compound Annual G ow h Ra e
CDO Chie Da a O ice
CI/CD Con inuous In eg a ion/Con inuous De elopmen
CNN Con olu ional Neu al Ne wo k
COVID-19 Co ona i us Disease 2019
CV Compu e Vision
DARPA De ense Ad anced Resea ch P ojec s Agency
FTE Full- ime Equi alen employee
GAN Gene a i e Ad e sa ial Ne wo k
GPU G aphics P ocessing Uni
HDR High Dynamic Range
HITL Human-In-The-Loop
ILSVRC ImageNe La ge Scale Visual Recogni ion Challenge
IT In o ma ion Technology
MIT Massachuse s Ins i u e o Technology
ML Machine Lea ning
NER Named En i y Recogni ion
NLG Na u al Language Gene a ion
NLP Na u al Language P ocessing
QA Quali y Assu ance
RPA Robo ic P ocess Au oma ion
SaaS So wa e-as-a-Se ice
SDK So wa e De elopmen Ki
SVR Single-View Recons uc ion
UI Use In e ace
UX Use Expe ience
1
1. INTRODUCTION
The concep o Compu e Vision (CV) has unde gone many changes since he beginning o he 21s
cen u y. In 2003, Compu e Vision was jus en isioned as an exci ing bu diso ganized ield (Fo sy h &
Ponce, 2003), bu en yea s la e his de ini ion apidly u ned in o a ield ha ocused on he
au oma ed ex ac ion o in o ma ion om images o in e some hing abou he wo ld (P ince, 2012;
Solem, 2012). Cu en ly, scien i ic and business communi ies a e al eady used o hea on he concep
o CV in a newspape headline highligh ing an in es men o millions o dolla s on he esolu ion o
cu en global challenges, such as igh ing Co ona i us Disease 2019 (COVID-19) (Ulhaq e al., 2020) o
p e en ing clima e changes (Ramachand a, 2019).
As a ield ha seeks o ex ac in o ma ion om image da a au oma ically, cu en in-use solu ions
o sys ems wo k wi h Compu e Vision h ough wo di e en app oaches. Handc a ed app oaches,
o example, apply CV echniques by using se s o ules o sol e a speci ic challenge. Some use-cases
o i a e 1) using a su eillance came a o de ec mo emen in a oom; 2) de ec ing g een o de o es ed
a eas in sa elli e images, and; 3) c opping some pa o a documen om a pic u e. On he o he hand,
Compu e Vision solu ions may also be associa ed wi h machine lea ning models o o he A i icial
In elligence (AI) applica ions, such as objec ecogni ion and de ec ion, image classi ica ion o anomaly
de ec ion solu ions. These Compu e Vision sys ems a e o en inspi ed by he p ope ies and
cha ac e is ics o human ision. Con e sely, hese algo i hms can also o e insigh s in o how he
in o ma ion ex ac ed om images is in e p e ed in he human b ain.
The abili y o a i icially in elligen sys ems o see like humans has been a subjec o inc easing
in e es and does no appea o be slowing down any ime soon. Howe e , he p ocess o deciphe ing
images, due o he mo e signi ican amoun o da a ha needs analysis, is mo e complex han
unde s anding o he o ms o bina y in o ma ion. None heless, he usage o a i icial neu al ne wo ks
is making compu e ision mo e capable o iden i ying pa e ns om images han he human isual
cogni i e sys em (Schei e e al., 2014).
Also, compu e ision echnologies will no only be less demanding o ain, bu also be capable o
pe cei ing mo e om images han hey a e doing wi hin he p esen . Toge he wi h o he echnologies
o o he subse s o AI, hese can be used in o de o build e en mo e powe ul and obus applica ions.
Fo ins ance, image cap ioning echniques can be combined wi h Na u al Language Gene a ion (NLG)
applica ions a e used o deciphe he su ounding objec s o isually impai ed indi iduals (Kim, 2020).
In a nea u u e, compu e ision will play a i al ole wi hin he de elopmen o A i icial Gene al
In elligence (AGI) and A i icial Supe in elligence by g an ing he capaci y o handle da a as simila ly o
e en be e han he human isual sys em (Pueyo, 2018). Taking his in o conside a ion, i can be ha d
o he scien i ic communi y o accep ha oday’s compu e ision capabili ies oge he wi h i s
8
(CNN), and i s b eak h ough consis ed in i s abili y o use a GPU o ain he compu e ision model
signi ican ly as e and o longe – AlexNe was ained o e 6 days on wo GPUs ha we e accessible
o he consume . Since hen, Da a Science and all i s subdomains ha e e ol ed ema kably, bo h
ma hema ically, in e ms o he in as uc u e i uses, and in e ms o compu ing and la ge-scale
p ocessing. Finally, he global ma ke has unde s ood he ad an ages o applying he da a- ela ed
echnologies and me hodological app oaches in en ed o da e, and has been able o ma e ialize hem
ei he in p oduc s o se ices, o in he in e nal p ocesses o o ganiza ions, such as in chu n p edic ion
and sales o ecas ing (Google, 2021). E en C-le el execu i e posi ions such as Chie Da a O ice s
(CDOs) ha e been c ea ed o manage he c ea ion and go e nance o da a p ocesses and o ensu e a
da a-d i en cul u e in o ganiza ions.
The Compu e Vision ield has inhe en ly bene i ed om his e olu ion, and is aced, pe haps o
he i s ime, wi h he challenge o no only con inuing o imp o e i s pe o mance, bu also o
op imizing he cos s ela ed o he necessa y e o . This is whe e he da a labeling componen comes
in, which may well be he d i ing o ce behind all u u e de elopmen s.
Figu e 3 – F equency ha he e m “Da a Science” was sea ched o on Google, in he wo ld and in
mul iple languages, om 2004 un il he p esen (Google, 2021)
9
3. LITERATURE REVIEW
(Du a & Zisse man, 2019) de ined da a labeling in he con ex o machine lea ning as he p ocess
o de ec ing and agging da a samples while a aching meaning and/o con ex o digi al da a. This
p ocess can be manual bu is usually pe o med o assis ed by so wa e. Da a labeling is an impo an
pa o da a p ep ocessing o Machine Lea ning (ML), pa icula ly o Supe ised Lea ning. Bo h inpu
and ou pu da a a e labeled o classi ica ion o p o ide a lea ning basis o u u e da a p ocessing. Fo
example, a sys em aining o iden i y animals in images migh be p o ided wi h mul iple images o
a ious ypes o animals om which i would lea n he s anda d ea u es o each, enabling i o
co ec ly iden i y he animals in unlabeled images (Why ock e al., 2021).
Machine Lea ning and Deep Lea ning sys ems o en equi e massi e amoun s o da a o es ablish
a ounda ion o eliable lea ning pa e ns. Da a acili a ed o he lea ning p ocess mus be labeled o
anno a ed, which means ha e e y hing, o some imes only he mos impo an hings, mus be
iden i ied and localized in he image. I mus also be labeled based on da a ea u es ha help he
model o ganize he da a in o pa e ns ha p oduce a desi ed answe . A p ope ly labeled da ase
p o ides a g ound u h ha he ML model uses o check i s p edic ions o accu acy and o con inue
e ining i s algo i hm. E o s in his p ocedu e impai he quali y o he aining da ase and he
pe o mance o any p edic i e models i is used o (Kshe i, 2021). To mi iga e his, many
o ganiza ions ake a Human-In-The-Loop (HITL) app oach, which is called a “da a labele ” main aining
human in ol emen in aining and es ing da a models h oughou hei i e a i e g ow h (Mona ch,
2021). The e a e se e al p ocedu es o s uc u e and label da a while main aining human in ol emen
(Fiedle e al., 2019). Ei he by using c owdsou cing, whe e a hi d-pa y pla o m gi es an en e p ise
Figu e 4 – Da a Labeling anno a ion ool ImageTagge (Fiedle e al., 2019)
aming an Objec De ec ion ask
10
access o many wo ke s a once, and/o by using con ac o s, whe e an en e p ise can hi e empo a y
eelance wo ke s o p ocess and label da a. A ecen epo om AI esea ch and ad iso y i m
Cognily ica ound ha o e 80% o he ime en e p ises spend on AI p ojec s goes owa d p epa ing,
cleaning, and labeling da a (C. Resea ch, 2019). Manual da a labeling is he mos ime-consuming and
expensi e me hod, bu i migh be wa an ed o impo an applica ions (Fiedle e al., 2019). Some
expe s do belie e ha da a labeling may p esen a new low-skilled job oppo uni y o eplace he
ones ha a e nulli ied by au oma ion, because he e is an e e -g owing su plus o da a and machines
ha need o p ocess i o pe o m he asks necessa y o ad anced ML and AI, which will c ea e mo e
and mo e low-skilled jobs and needs o hi e mo e ope a ional p o iles (Kshe i, 2021).
Apa om ha , he e olu ion o image classi ica ion models shows a clea upwa d end in he
eme gence o new image classi ica ion models, s emming om an in es men in esea ch (Facebook
AI Resea ch, 2021). This is also ue o seman ic segmen a ion, language modelling, ime se ies
o ecas ing, speech ecogni ion, among o he me hods (Wason, 2018).
E iden ly, he e olu ion in da a anno a ion echniques and so wa e ollows a simila end in
ecen yea s, since ha dwa e limi a ions a e a hind ance in inc easing he pe o mance o he me hods
lis ed abo e, and esea ch and de elopmen e o is di ec ed owa ds his a ea. Acco ding o (G. V.
Resea ch, 2021), he global da a anno a ion ma ke was alued a US$ 695.5 million in 2019, is
cu en ly alued a US$ 1.66 billion and is expec ed o each US$ 8.22 billion by 2028. The g owing da a
anno a ion indus y, which is expec ed o g ow a a Compound Annual G ow h Ra e (CAGR) o 25.6%
om 2021 o 2028 (G. V. Resea ch, 2021), is expec ed o expe ience eno mous expansion in he nea
u u e.
Plus, Ga ne classi ied Da a Labeling and Anno a ion Se ices as en e ing he T ough o
Disillusionmen (Figu e 6) as his end jus walked by he Peak o In la ed Expec a ions (Ga ne Inc.,
2021). This means ha his ield has jus su ed i s wa e o exace ba ed hype and in es men and i is
inally slowing down while implemen a ions ail o deli e . I ypically happens be o e mo e ins ances
Figu e 5 – E olu ion o Image Classi ica ion models on ImageNe da ase : Top 1 Accu acy (le )
and Top 5 Accu acy ( igh ) (Facebook AI Resea ch, 2021)
11
o how hese se ices can bene i he en e p ise s a o ake shape and become mo e b oadly
ecognized, and is c ucial o s a ga he ing mo e ma u i y in he Da a and AI ma ke . Ga ne es ima es
i o each he Pla eau o P oduc i i y in 5 o 10 yea s (Ga ne Inc., 2021). Thus, da a labeling is a
p ocess ha will cons an ly e ol e and change o mee he business and echnical objec i es, ha is,
labeling asks oday a e e y p one o be di e en in h ee mon hs. Th ough he ime, a da a labeling
eam e ol es and inds be e ways o label aining da a o imp o ed quali y and model pe o mance,
c ea ing guidelines and sha ing in o ma ion on how o deal wi h he a es use cases o scena ios.
Rela i ely o he so wa e, he bes da a labeling ools mus be use - iendly in e ms o Use
In e ace and Use Expe ience (UI/UX) and b eak he wo k down in o a omic and smalle asks o
maximize labeling quali y (Du a & Zisse man, 2019). When a complex ask is ans o med in o a se o
a omic componen s, i is easy o measu e and quan i y each o hose asks. I also allows he
iden i ica ion o which asks a e bes sui ed o humans and which ones can be au oma ed. To op imize
bo h da a quali y and he wo k o ce in es men , he e a e plen y aspec s o conside when choosing
he ideal da a labeling ool. In he ollowing chap e , an in-dep h analysis and compa ison o a hand-
picked selec ion o ools is p esen ed, ocusing on he mul iple unc ional and echnical issues ela ed
o he opics o Compu e Vision, Na u al Language P ocessing (NLP), Au oma ion, Quali y Assu ance
(QA) and Managemen (Du a & Zisse man, 2019; Said e al., 2017). These opics a e deal wi h as
clus e s o ea u es o unc ionali ies ha a e appa en ly p e alen wi hin he da a anno a ion ools
and se ices ma ke and migh de ia e sligh ly om ou ocus on he Compu e Vision ield.
Figu e 6 – Hype Cycle o A i icial In elligence 2021 (Ga ne Inc., 2021), as o July
2021
12
Las ly, (Gau e al., 2018) conduc ed a simila wo k o e iew he s a e-o - he-a in ideo
anno a ion. Howe e , gi en he la es ma ke upda es in Da a Labeling se ices, his e iew lacks a
iew o he p esen and i doesn’ analyze he p e alen concep s o ea u es ha a e common o da a
labeling ools in s uc u ed e ms.
13
4. DATA LABELING TOOLS
The ool selec ion ha ollows was pe o med based on da a ha was manually collec ed un il July
2021. These ools we e selec ed based on hei gene al epu a ion, ma ke adop ion, and he ea u es
hey p o ide o speed up and sol e he da a labeling ask in Machine Lea ning p oblems. E en hough
his disse a ion consis s o a de ailed and s uc u ed analysis, he de ini ion o a c i e ion o he
selec ion o he ools o be s udied is no i ial, o se e al easons. Fi s , a panoply o new da a
labeling ools has been c ea ed and made a ailable la ely, gi en hei demand, which makes choosing
hem di icul , as hey a e o en eleased wi h e y imma u e documen a ion. On he o he hand, he e
a e ools p oduced by la ge echnological companies wo ldwide, and he e a e o he s ha a e
de eloped by pa icula s as side-p ojec s o e en as hobbies, which makes hei compa ison
imp ac icable due o he lack o esou ces associa ed wi h he la e . The inc easing ma ke p essu e
o de eloping new da a labeling ools can be explained by i s own needs and ma e ialized in Figu e 7,
using Google T ends (Google, 2021) o he e m “Supe ised Machine Lea ning”, ha is immedia ely
associa ed o he da a labeling p ocess because o i s dependency on labeled da a. Finally, pe sonal
expe ience and p e e ence migh ha e an undesi able impac on he me iculous de ini ion o he s udy
po en ial ha ools may hold. Thus, he selec ion o he da a labeling ools was based on he
knowledge acqui ed h oughou he wo k expe ience, om he sha ing o people and o ums o
e e ence in he domain, om publica ions in scien i ic jou nals, om code sha ing pla o ms, om
news on ma ke adop ion, and om men ions in ele an con e ences in he subjec .
As such, Table 1 enume a es he conside ed ools ha we e analyzed in his disse a ion, along wi h
he espec i e e e ence and wi h each co esponding de eloping en i y o b and, i applicable.
Figu e 7 – F equency ha he e m “Supe ised Machine Lea ning” was sea ched o
on Google, in he wo ld and in mul iple languages, om 2016 un il he p esen
14
Tool Name
De elope /B and
Re e ence
Colabele
Colabele
N/A
CVAT
In el
A cos -e ec i e, as , and
obus anno a ion ool (Said
e al., 2017)
di g am
Di g am
N/A
ImageTagge
Hambu g Bi -Bo s, Uni e si y o
Hambu g
ImageTagge : An Open
Sou ce Online Pla o m o
Collabo a i e Image Labeling
(Fiedle e al., 2019)
Label S udio
Hea ex
N/A
Labelbox
LabelBox
N/A
LabelD
N/A
N/A
LabelImg
N/A
N/A
LabelMe
Compu e Science and A i icial
In elligence Labo a o y, MIT
LabelMe: A Da abase and
Web-Based Tool o Image
Anno a ion (Russell e al.,
2008)
makesense.ai
makesense.ai
N/A
Playmen
Playmen , TELUS In e na ional
N/A
Ra snake
N/A
Ra snake: A Ve sa ile Image
Anno a ion Tool wi h
Applica ion o Compu e -
Aided Diagnosis (Iako idis e
al., 2014)
Rec Label
N/A
N/A
Remo.ai
Redisco e y.io
N/A
V7 Da win
V7
N/A
VGG Image Anno a ion
Visual Geome y G oup, Uni e si y
o Ox o d
The VIA anno a ion so wa e
o images, audio and ideo
(Du a & Zisse man, 2019)
VoTT
Mic oso
N/A
COCO Anno a o
N/A
N/A
EVA
N/A
N/A
Supe Anno a e
Supe Anno a e
N/A
Table 1 – Manually selec ed da a anno a ion ools and espec i e de elope eams and e e ences,
i applicable
Du ing his e ision, i was ealized ha a la ge pa o he s udied so wa e ools does no ha e an
associa ed a icle o published da a. In o ma ion abou hei au ho s is also di icul o each, which
demons a es ha hese ools a e e iden ly di ided in o 3 g oups: 1) ools ha we e de eloped
speci ically o comme cial pu poses; 2) ools ha we e de eloped acco ding o he au ho s' own
needs, and; 3) ools ha a e ocused on scien i ic esea ch and imp o emen in o de o make he da a
anno a ion p ocess as agile as possible. Fo hese easons, Table 1 displays de elope eam/b and as
“N/A” when in o ma ion abou he so wa e’s au ho s is no publicly a ailable o when hey we e a
15
dynamic and changeable eam o con ibu o s h oughou ime and whe e e e y de elope had
speci ic In o ma ion Technology (IT) knowledge and con ibu ed o an Open-Sou ce p ojec .
Taking his in o conside a ion, his i s analysis demons a es in a glance ha he main de elope
eams o b ands in ol ed in Da a Anno a ion ools o amewo ks a e big ech companies, specialized
ech s a -ups mainly based on Silicon Valley in he Uni ed S a es o Ame ica, o globally enowned
academic esea ch g oups. I p o es ha nowadays, he da a anno a ion ools a e a p io i y o he
mos ecognized IT companies and scien i ic en i ies a ound he wo ld and hei in es men s. These
ools a e o en sold as SaaS (So wa e-as-a-Se ice), mone izing no only he p oduc i sel bu also he
op ional se ice o ou sou cing da a labele s, cons i u ing an ex emely aluable asse o p ese e,
gi en he high demand ha only ends o inc ease e en u he (Moulik, 2020).
Aiming a deep analysis o he echnical ea u es o he selec ed s a e-o - he-a da a anno a ion
ools, mul iple clus e s o unc ionali ies coupled wi h hei espec i e desc ip ions, g ouped by
unc ional ields o ac i i y a e ollowed. Each subchap e co esponds o a speci ic clus e , whe e he
analyzed ea u es help o peel o he mul iple ools and a able is p esen ed o ha pu pose, whe e
he “✓” ma k indica es he p esence o a gi en unc ionali y, “X” deno es i s absence, and he “?”
shows ha he e was no in o ma ion a ailable on ha gi en subjec a he ime o his disse a ion.
16
4.1. COMPUTER VISION FEATURES
Rega ding CV- ela ed unc ionali ies, unc ionali ies as Bounding Boxes, Polygons, Lines, Key-poin s,
Cuboids, Image classi ica ion and Video labeling we e selec ed o enable a ull compa ison be ween
he selec ed ools.
▪ Bounding Boxes: Func ionali y ha allows he use o d aw ec angula Bounding
Boxes ha de ine he loca ion o he a ge objec s in s a ic images, ypically sui able o objec
de ec ion asks.
▪ Polygons: Tool le s he use o d aw polygons o delimi objec s in s a ic images, usually
needed o ins ance segmen a ion asks.
▪ Lines: Func ionali y o d aw lines o ec o s, usually employed in au onomous d i ing
applica ions in lane when anno a ing he lanes on he highways, o ins ance.
▪ Key-poin s: Pe mi s labeling h ough he connec ion o key-poin s o build a skele on
and o unde s and mo e easily wha is labeled, widely known o mo ion acking, acial
landma k de ec ion and hand ges u e ecogni ion.
▪ Cuboids: Allows 3D da a anno a ion, sa ing he dep h and heigh o each objec o
in e es . Usually applied on Objec De ec ion o sel -d i ing ehicles.
▪ Image Classi ica ion: Func ionali y ha associa es an image as a whole o a speci ic
ca ego y.
▪ Video Labeling: Tool pe mi s an easy na iga ion be ween ames o a ideo and make
anno a ions o each sequen ial image. Can also deal wi h ideos mo e complexly, using a
model o es ima e he posi ion o a p e iously anno a ed objec in he ollowing ames.
To pe mi a be e unde s anding abou hese ools’ unc ionali ies on Compu e Vision, Table 2
shows how hey a e used h ough he p e iously selec ed da a labeling ools.
Tool Name
Bounding
Boxes
Polygons
Lines
Key-poin s
Cuboids
Image
Classi ica ion
Video
Labeling
Colabele
✓
✓
✓
X
X
X
✓
CVAT
✓
✓
✓
✓
✓
X
✓
di g am
✓
✓
✓
✓
✓
✓
✓
ImageTagge
✓
✓
✓
✓
X
X
X
Label S udio
✓
✓
✓
✓
X
✓
X
Labelbox
✓
✓
✓
✓
X
✓
✓
LabelD
✓
?
?
?
?
✓
?
LabelImg
✓
X
X
X
X
X
✓
LabelMe
✓
✓
✓
✓
X
✓
✓
makesense.ai
✓
✓
✓
✓
X
✓
✓
17
Playmen
✓
✓
✓
✓
✓
X
✓
Ra snake
✓
✓
✓
✓
X
✓
✓
Rec Label
✓
✓
✓
✓
✓
X
X
Remo.ai
✓
✓
✓
X
X
✓
X
V7 Da win
✓
✓
✓
✓
✓
✓
✓
VGG Image
Anno a ion
✓
✓
✓
✓
✓
✓
✓
VoTT
✓
✓
X
X
X
✓
✓
COCO Anno a o
✓
✓
?
✓
?
?
?
EVA
✓
X
X
X
X
X
✓
Supe Anno a e
✓
✓
✓
✓
✓
✓
✓
Table 2 – Compu e Vision ela ed ea u es in he selec ed da a anno a ion ools
Table 2 shows ha all he selec ed da a anno a ion ools con empla e he possibili y o d awing
Bounding Boxes as a labeling ask. Besides, polygons d awing o ins ance segmen a ion asks is also
e y common. On he o he hand, Cuboids, Video Labeling and Image Classi ica ion seem o be he
leas exis ing ea u es in he da a anno a ion ools. The lack o unc ionali y on Cuboids d awing and
Video Labeling can be explained by he ac ha i s use is o ien ed owa ds uncommon and e y
speci ic use cases which need in o ma ion on dep h o he objec s o eal- ime ideo p ocessing,
espec i ely. Plus, companies ha de elop da a anno a ion ools o mee his equi emen end o use
hem in-house only in o de o ge a compe i i e ad an age. The lack o he unc ionali y ha pe mi s
Image classi ica ion can mean ha he de elope b ands conside his ask as a ainable by a manual
p ocess, ha ing nea ly no cos bene i o de elop i .
24
4.5. GENERAL COMPARATIVE ANALYSIS
Table 6 is shown below as a w ap up o all he pe o med analysis, whe e he ca ego ies o ea u es
p e alen in each da a labeling ool a e easily poin ed ou . The p esen ed alues we e calcula ed based
on he a io o co e ed unc ionali ies unde each o he ou clus e s (Compu e Vision, NLP,
Au oma ion and QA) scope o each analyzed ool.
Tool Name
Compu e
Vision
Na u al
Language
P ocessing
Au oma ion
& De elope
Managemen
and QA
Colabele
57%
100%
50%
0%
CVAT
86%
0%
50%
17%
di g am
100%
100%
100%
83%
ImageTagge
57%
0%
0%
0%
Label S udio
71%
100%
100%
17%
Labelbox
86%
100%
100%
67%
LabelD
29%
0%
0%
0%
LabelImg
29%
0%
0%
0%
LabelMe
86%
0%
0%
17%
makesense.ai
86%
0%
50%
0%
Playmen
86%
0%
0%
17%
Ra snake
86%
0%
50%
17%
Rec Label
71%
0%
50%
0%
Remo.ai
57%
0%
100%
33%
V7 Da win
100%
0%
100%
67%
VGG Image
Anno a ion
100%
0%
50%
0%
VoTT
57%
0%
50%
33%
COCO Anno a o
43%
0%
50%
0%
EVA
29%
0%
0%
0%
Supe Anno a e
100%
0%
50%
0%
Table 6 – Compa a i e summa y o he ea u es g ouped by ca ego y ac oss he analyzed da a
anno a ion ools
The compa ison be ween he ela i e co e age o he unc ionali y ca ego ies by ool demons a es
only a mino i y o he ools ocus on he NLP- ela ed ea u es, unlike he unc ionali ies ela ed o
Compu e Vision. Mo eo e , he selec ed Au oma ion and De elope - iendly ea u es a e appa en ly
ep esen a i e along mos o he analyzed ools, and Managemen and QA unc ionali ies a e mo e
p esen in he ools ha we e de eloped by a company.
25
5. DISCUSSION
The objec i e o his disse a ion is o compa e he selec ed ools in a s uc u ed way, wi h a iew
o hei po en ial use o da a labeling pu poses o Compu e Vision challenges. In o de o e alua e
he ools in hei o ali y, his compa a i e analysis was ex ended o ea u es ha a e unnecessa y o
CV opics, bu ha migh e en ually suppo an o ganiza ion's decision in selec ing a ool. As such,
mul iple unc ionali ies we e highligh ed ha ocus no only on image, ideo, and ex da a, bu also
on he au oma ion and op imiza ion o he labeling p ocess and i s quali y assu ance. Howe e , Table
6 shows ha he selec ion o ools migh be biased owa ds he ocus on Compu e Vision, which
means ha his analysis has mo e signi icance in his ield.
Ini ially, i was planned o c ea e a poin sys em o e alua e each da a labeling ool ullness, whe e
he p esence o each ea u e would add up one poin o a ool, and he decision on he bes ool
would be judged only by he maximum numbe o poin s ob ained h oughou he analysis. Howe e ,
his compa ison would no be ai o ealis ic since, o example, he weigh ing o impo ance o NLP-
ela ed ea u es is likely o be lowe i he o ganiza ion unde conside a ion is a so wa e house ha
de elops deep lea ning models o in e p e images. In addi ion, he inancial ac o may also ha e an
impac on he choice and he e o e he choice o he da a labeling ool p esen ed in his disse a ion
will only be acco ding o he au ho 's pe spec i e and migh di e acco ding o he ci cums ances o a
eade who belongs o a speci ic o ganiza ion o de elopmen eam.
Gi en he di e si y o pu poses o he analyzed ools, one o mo e ools will be chosen o each
analysis conduc ed. S a ing wi h he main analysis o he unc ionali ies wi h Compu e Vision, he
ools ha p esen all he analyzed unc ionali ies s and ou e iden ly: di g am, V7 Da win, VGG Image
Anno a ion and Supe Anno a e. Rega ding he pilla o NLP unc ionali ies, he ools ha allow ex
classi ica ion and he iden i ica ion o named en i ies a e Colabele , di g am, Label S udio and
Labelbox. I should be no ed ha he pool o ools ha o e hese wo unc ionali ies is qui e di e en
when compa ed o he Compu e Vision o ien ed one, as ex ea u es a e ex emely
unde ep esen ed. This is explained by he scope es ic ions o he di e en p oduc s, which a e
p obably aimed a di e en niche ma ke s o academic acks, as p e iously explained. Conce ning
HITL and being open o he inclusion o cus omizable add-ins, di g am, Label S udio, Labelbox, Remo.ai
and V7 Da win a e he winne s, meaning ha hese a e he ools wi h he mos e sa ili y o mul iple
use cases, while ha ing a educed e o a e needed o achie e he da ase anno a ion goal. As o he
pilla ha ela es o ea u es o Managemen and QA, he ools ha s and ou he mos a e di g am,
Labelbox, Remo.ai and V7 Da win. The unc ionali ies o ien ed o p ojec managemen , ask planning
and assignmen and da a managemen a e ce ainly unde de eloped in he labeling ools ma ke
o e all, bu a e mos ly p esen in he co esponding de elopmen oadmaps, which o esee ha hey
26
will be olled ou du ing 2021 o 2022. F om he en i e pale e o e iewed ools, he e a e wo ools
ha s and ou om he compe i ion – V7 Da win and di g am.
V7 Da win has all he necessa y ea u es o da a anno a ion o a Compu e Vision ML p ojec ,
which a e de eloped in a obus , quali y and ex emely use - iendly way. I s majo ocus is on ease o
anno a ion and au oma ion o labeling using machine lea ning models, while being simplis ic and
ha ing an appa en ly sho lea ning cu e. Thus, i is one o he mos p omising ools o image and
ideo labeling, despi e being expensi e and no being open-sou ce which is a g ea disad an age pe
se o no elying on i s own communi y o e ol e.
On he o he hand, di g am is he mos comple e open-sou ce choice ha holis ically co e s he
en i e da a li ecycle, om da a inges ion and mining o in eg a ion wi h cloud o on-p emise machine
lea ning pipelines. In addi ion, i is a pla o m which is only paid o eams consis ing o mo e han 20
use s, which ensu es da a s o age and e sioning, da a labeling o mul iple asks, wo k low
managemen , and da a secu i y - all hese ea u es a e accessible di ec ly om i s pla o m o i s
Applica ion P og amming In e ace (API). Mo eo e , i is based on a e y ac i e Gi Hub eposi o y and
has mo e han 500 s a s. Plus, i ensu es in e pola ion inside ideos o absol e he labele s’ e o in
ha ing o label e e y single ame in e e y imes amp o he ideo, using an objec acke and sma
ame compa ison heu is ics. Un o una ely, i does no ye co e he unc ionali y needed o NLP
p oblems, al hough hese will be on nex yea 's de elopmen oadmap, as well as audio- ela ed
unc ionali y.
27
6. CONCLUSIONS
This disse a ion ames he e olu ion o he domain o Compu e Vision, om i s incep ion o he
p esen . Pas all he ups and downs o he ield, da a eme ges as he new oil o he 21s cen u y, so
he e is a global shi in he bleeding-edge ma ke ends o da a-d i en cul u e. As such, he need
a ises o s a op imizing he da a labeling p ocess, en isioning he gene a ion o da ase s wi h mo e
and be e da a e en as e , in o de o minimize ime e o and inancial cos s, wi hou penalizing he
labeling p ocess. Howe e , a panoply o ools was c ea ed wi h hese in en s, c ea ing he need o
cons an ly e iew he s a e-o - he-a and o s a egically pick he igh so wa e o one’s needs. Fo
his, a selec ion o he ools wi h he mos men ions in he scien i ic and indus ial communi y is
p oposed, on which a compa a i e analysis is made o elec he mos e olu iona y and comple e ools
o pe o m da a labeling and espond o Compu e Vision o Machine Lea ning p oblems in gene al, in
any o ganiza ion. Howe e , he labeling p ocess is ypically expensi e and edious and migh e en
ha m he easibili y o a p ojec . Thus, he scope o his compa ison seeks o mi iga e and add ess he
bo leneck associa ed wi h he da a labeling phase in a Machine Lea ning p ojec .
F om he echnical poin o iew, he compa a i e analysis o exis ing da a labeling amewo ks
poin ed o he ic o y o di g am so wa e, whose unc ionali ies co e p odigiously he en i e pipeline
o a da a p ojec , ea u ing da a inges ion, anno a ion, in eg a ion, explo a ion and p oduc ion in he
o m o Machine Lea ning models. Di g am is open-sou ce and main ains a public de elopmen
oadmap, le e aging he communi y pa icipa ion in i s e olu ion.
In e ms o mo e business- ela ed insigh s abou his ool, di g am has a ee e sion o eams
wi h less han 20 use s and can p e en mul iple e o s and da a edundancy while a oiding he daily
impo s and expo s o da a be ween di e en ools. By cen alizing he en i e pipeline in one ool, a
eam can: 1) sho en i s own lea ning cu e, which is a huge ad an age gi en he ma ke p essu e o
quickly ain Full- ime Equi alen employees (FTEs) in new echnologies; 2) minimize po en ial secu i y
p oblems; 3) educe licensing cos s, and; 4) a oid edundancy o s o ed da a. Mo eo e , he ac o
being open-sou ce opens a panoply o possibili ies o quickly adding and es ing new ea u es o he
p oduc .
28
7. LIMITATIONS AND RECOMMENDATIONS FOR FUTURE WORKS
A p ac ical componen was planned o accompany his disse a ion – he wo k was in ended o
consis in he c ea ion o a new da a labeling ool, which was which was quickly seen as oo ambi ious
o a mas e hesis disse a ion. Then, he p ac ical componen shaped i sel in o appending a new
ea u e/ unc ionali y o an al eady exis ing da a labeling ool. Howe e , as his wo k ad anced
h oughou he exis ing documen a ion on he mul iple analyzed ools, he mo e i allowed o ealize
ha a unc ionali y designed and de eloped in less han one yea would no be able o compe e wi h
a ool de eloped by a niche company o a la ge IT leade , since i would ine i ably all sho o quali y
and complexi y o he ea u es al eady p esen in o he ools as hey a e olled-ou and p oduc ized
by la ge, specialized, de elope eams wi h much mo e c i ical mass.
Also, he lack o p emium/en e p ise licenses in non-open-sou ce ools was one o he limi a ions
ound du ing his wo k. As a ecommenda ion o u u e wo k, i is sugges ed o con ac he owne s o
hese ools o eques and ob ain p emium/en e p ise licenses o academic pu poses. The usage o
such licenses may allow his wo k o go in o mo e de ail a he echnical le el and mo e ob ious
in e ence o how he backends o he ools wo k.
Once a license is gi en, one o he possible essen ial aspec s o explo e is he compa ison o objec
acke s ega ding ideo labeling. An objec acke ha p esen s a be e pe o mance can also
encou age he choice o i s ool, as he ideo labeling p ocess can be hugely op imized, as he acke
i sel can spa e he da a labele o anno a ing all he ames in a ideo while using in o ma ion collec ed
in p e ious ideo ames o help he consequen ones. Fu he mo e, u u e wo ks could conside using
mul iple ools o achie e a labeled da ase while main aining he same eam o da a labele s, in o de
o measu e and compa e he e iciency o each labeling ool in e ms o e o and ime consumed in
a eal-li e scena io.
29
8. BIBLIOGRAPHY
Aa s, E., & Ko s , J. (1987). Simula ed Annealing: Theo y and Applica ion. Simula ed Annealing: Theo y
and Applica ion, 7. h ps://link-sp inge -
com.ezp oxy2.lib a y.colos a e.edu/con en /pd /10.1007%2F978-94-015-7744-1_2.pd
Abbas, A., Su e , D., Zou al, C., Lucchi, A., Figalli, A., & Woe ne , S. (2021). The powe o quan um
neu al ne wo ks. Na u e Compu a ional Science, 1(6), 403–409. h ps://doi.o g/10.1038/s43588-
021-00084-1
Aga , J. O. N. (2020). Wha is science o ? The Ligh hill epo on a i icial in elligence ein e p e ed.
B i ish Jou nal o he His o y o Science, 53(3), 289–310.
h ps://doi.o g/10.1017/S0007087420000230
Aga wala, A., Don che a, M., Ag awala, M., D ucke , S., Colbu n, A., Cu less, B., Salesin, D., & Cohen,
M. (2004). In e ac i e digi al pho omon age. ACM SIGGRAPH 2004 Pape s, SIGGRAPH 2004, 294–
302. h ps://doi.o g/10.1145/1186562.1015718
Belongie, S., Malik, J., & Puzicha, J. (2002). Shape ma ching and objec ecogni ion using shape
con ex s. IEEE T ansac ions on Pa e n Analysis and Machine In elligence, 24(4), 509–522.
h ps://doi.o g/10.1109/34.993558
Be e o, M., Poggio, T. A., & To e, V. (1988). Ill-Posed P oblems in Ea ly Vision. P oceedings o he
IEEE, 76(8), 869–889. h ps://doi.o g/10.1109/5.5962
Besl, P. J., & Jain, R. C. (1985). Th ee-dimensional objec ecogni ion. ACM Compu ing Su eys (CSUR),
17(1), 75–145. h ps://doi.o g/10.1145/4078.4081
Blake, A., & Isa d, M. (1998). Ac i e Con ou s. In Ac i e Con ou s. Sp inge London.
h ps://doi.o g/10.1007/978-1-4471-1555-7
B own, M., & Lowe, D. G. (2007). Au oma ic pano amic image s i ching using in a ian ea u es.
In e na ional Jou nal o Compu e Vision, 74(1), 59–73. h ps://doi.o g/10.1007/s11263-006-
0002-3
Bu ka , N., & Hube , M. F. (2021). A Su ey on he Explainabili y o Supe ised Machine Lea ning.
Jou nal o A i icial In elligence Resea ch, 70, 245–317. h ps://doi.o g/10.1613/jai .1.12228
Chang, J. C., Ame shi, S., & Kama , E. (2017). Re ol : Collabo a i e c owdsou cing o labeling machine
lea ning da ase s. Con e ence on Human Fac o s in Compu ing Sys ems - P oceedings, 2017-May,
2334–2346. h ps://doi.o g/10.1145/3025453.3026044
Comaniciu, D., & Mee , P. (2002). Mean shi : a obus app oach owa d ea u e space analysis. IEEE
T ansac ions on Pa e n Analysis and Machine In elligence, 24(5), 603–619.
h ps://doi.o g/10.1109/34.1000236
Da id, M., & Jayan , S. (1989). Op imal App oxima ions by Piecewise Smoo h Func ions and Associa ed
30
Va ia ional P oblems. Communica ions on Pu e and Applied Ma hema ics, 42, 577–685.
Debe ec, P. E., & Malik, J. (1997). Reco e ing high dynamic ange adiance maps om pho og aphs.
P oceedings o he 24 h Annual Con e ence on Compu e G aphics and In e ac i e Techniques -
SIGGRAPH ’97, 3(1), 369–378. h ps://doi.o g/10.1145/258734.258884
DodgeSpecial, J. (2001). D ape P ize Hono s Fou “Fa he s o he In e ne .”
h ps://www.wsj.com/a icles/SB982004616905008338
Du a, A., & Zisse man, A. (2019). The VIA anno a ion so wa e o images, audio and ideo. MM 2019
- P oceedings o he 27 h ACM In e na ional Con e ence on Mul imedia, 2276–2279.
h ps://doi.o g/10.1145/3343031.3350535
E e h, J. (2018). Da aOps – Towa ds a de ini ion. CEUR Wo kshop P oceedings, 2191, 104–112.
Facebook AI Resea ch. (2021). Pape sWi hCode.com.
Fauge as, O. D., & Hebe , M. (1986). The Rep esen a ion, Recogni ion, and Loca ing o 3-D Objec s.
The In e na ional Jou nal o Robo ics Resea ch, 5(3), 27–52.
h ps://doi.o g/10.1177/027836498600500302
Fe gus, R., Pe ona, P., & Zisse man, A. (2007). Weakly supe ised scale-in a ian lea ning o models o
isual ecogni ion. In e na ional Jou nal o Compu e Vision, 71(3), 273–303.
h ps://doi.o g/10.1007/s11263-006-8707-x
Fiedle , N., Bes mann, M., & Hend ich, N. (2019). ImageTagge : An Open Sou ce Online Pla o m o
Collabo a i e Image Labeling. In Lec u e No es in Compu e Science (including subse ies Lec u e
No es in A i icial In elligence and Lec u e No es in Bioin o ma ics): Vol. 11374 LNAI (pp. 162–
169). h ps://doi.o g/10.1007/978-3-030-27544-0_13
Fo sy h, D., & Ponce, J. (2003). Compu e Vision: A Mode n App oach. P en ice Hall P o essional
Technical Re e ence.
Ga ne Inc. (2021). Ga ne . Hype Cycle o A i icial In elligence 2021.
h ps://www.ga ne .com/en/in o ma ion- echnology/insigh s/a i icial-in elligence
Gau , E., Saxena, V., & Singh, S. K. (2018). Video anno a ion ools: A Re iew. P oceedings - IEEE 2018
In e na ional Con e ence on Ad ances in Compu ing, Communica ion Con ol and Ne wo king,
ICACCCN 2018, 911–914. h ps://doi.o g/10.1109/ICACCCN.2018.8748669
Google. (2021). Google T ends. h ps://www.google.com/ ends
Hil on, A., Fua, P., & Ron a d, R. (2006). Modeling people: Vision-based unde s anding o a pe son’s
shape, appea ance, mo emen , and beha iou . Compu e Vision and Image Unde s anding,
104(2–3), 87–89. h ps://doi.o g/10.1016/j.c iu.2006.09.002
Iako idis, D. K., Goudas, T., Smailis, C., & Maglogiannis, I. (2014). Ra snake: A Ve sa ile Image
Anno a ion Tool wi h Applica ion o Compu e -Aided Diagnosis. The Scien i ic Wo ld Jou nal,
2014, 1–12. h ps://doi.o g/10.1155/2014/286856
31
Jianbo Shi, & Malik, J. (2000). No malized cu s and image segmen a ion. IEEE T ansac ions on Pa e n
Analysis and Machine In elligence, 22(8), 888–905. h ps://doi.o g/10.1109/34.868688
Jianbo Shi, & Tomasi. (1994). Good ea u es o ack. P oceedings o IEEE Con e ence on Compu e
Vision and Pa e n Recogni ion CVPR-94, 593–600. h ps://doi.o g/10.1109/CVPR.1994.323794
Kass, M., Wi kin, A., & Te zopoulos, D. (1988). Snakes: Ac i e con ou models. In e na ional Jou nal o
Compu e Vision, 1(4), 321–331. h ps://doi.o g/10.1007/BF00133570
Kim, J. (2020). Applica ion on cha ac e ecogni ion sys em on oad sign o isually impai ed: Case
s udy app oach and u u e. In e na ional Jou nal o Elec ical and Compu e Enginee ing, 10(1),
778–785. h ps://doi.o g/10.11591/ijece. 10i1.pp778-785
Kopuklu, O., Kose, N., Gunduz, A., & Rigoll, G. (2019). Resou ce e icien 3D con olu ional neu al
ne wo ks. P oceedings - 2019 In e na ional Con e ence on Compu e Vision Wo kshop, ICCVW
2019, 1910–1919. h ps://doi.o g/10.1109/ICCVW.2019.00240
K izhe sky, A., Su ske e , I., & Hin on, G. E. (2017). ImageNe classi ica ion wi h deep con olu ional
neu al ne wo ks. Communica ions o he ACM, 60(6), 84–90. h ps://doi.o g/10.1145/3065386
Kshe i, N. (2021). Da a Labeling o he A i icial In elligence Indus y: Economic Impac s in De eloping
Coun ies. IT P o essional, 23(2), 96–99. h ps://doi.o g/10.1109/MITP.2020.2967905
Lani is, A., Taylo , C. J., & Coo es, T. F. (1997). Au oma ic in e p e a ion and coding o ace images using
lexible models. IEEE T ansac ions on Pa e n Analysis and Machine In elligence, 19(7), 743–756.
h ps://doi.o g/10.1109/34.598231
Lecle c, Y. G. (1989). Cons uc ing simple s able desc ip ions o image pa i ioning. In e na ional
Jou nal o Compu e Vision, 3(1), 73–102. h ps://doi.o g/10.1007/BF00054839
Ligoza , A.-L., Le è e, J., Bugeau, A., & Combaz, J. (2021). Un a eling he hidden en i onmen al
impac s o AI solu ions o en i onmen . In P oceedings o ACM Con e ence (Con e ence’17) (Vol.
1, Issue 1). Associa ion o Compu ing Machine y. h p://a xi .o g/abs/2110.11822
Lucas, B. D., & Kanade, T. (1981). I e a i e Image Regis a ion Technique Wi h an Applica ion To S e eo
Vision. 2(Ap il 1981), 674–679.
Malladi, R., Se hian, J. A., & Vemu i, B. C. (1995). Shape Modeling wi h F on P opaga ion: A Le el Se
App oach. IEEE T ansac ions on Pa e n Analysis and Machine In elligence, 17(2), 158–175.
h ps://doi.o g/10.1109/34.368173
Ma , D. (1982). Vision: a compu a ional in es iga ion in o he human ep esen a ion and p ocessing
o isual in o ma ion. In Vision: a compu a ional in es iga ion in o he human ep esen a ion and
p ocessing o isual in o ma ion. W. H. F eeman.
Ma hews, I., Xiao, J., & Bake , S. (2007). 2D s. 3D De o mable Face Models: Rep esen a ional Powe ,
Cons uc ion, and Real-Time Fi ing. In e na ional Jou nal o Compu e Vision, 75(1), 93–113.
h ps://doi.o g/10.1007/s11263-007-0043-2
32
Ma hews, J., & Bake , S. (2004). Ac i e appea ance models e isi ed. In e na ional Jou nal o
Compu e Vision, 60(2), 135–164. h ps://doi.o g/10.1023/B:VISI.0000029666.37597.d3
Moeslund, T. B., Hil on, A., & K üge , V. (2006). A su ey o ad ances in ision-based human mo ion
cap u e and analysis. Compu e Vision and Image Unde s anding, 104(2-3 SPEC. ISS.), 90–126.
h ps://doi.o g/10.1016/j.c iu.2006.08.002
Mona ch, R. (Mun o). (2021). Human-in- he-Loop Machine Lea ning: Ac i e lea ning and anno a ion
o human-cen e ed AI. Manning Publica ions Co.
Mo i, G., Xiao eng Ren, E os, A. A., & Malik, J. (2004). Reco e ing human body con igu a ions:
combining segmen a ion and ecogni ion. P oceedings o he 2004 IEEE Compu e Socie y
Con e ence on Compu e Vision and Pa e n Recogni ion, 2004. CVPR 2004., 2, 326–333.
h ps://doi.o g/10.1109/CVPR.2004.1315182
Moulik, S. (2020). Da a as he New Cu ency—How Open Sou ce Toolki s Ha e Made Labeled Da a he
Co e Value in he AI Ma ke place. Academic Radiology, 27(1), 140–142.
h ps://doi.o g/10.1016/j.ac a.2019.09.016
Mundy, J. L. (2006). Towa d Ca ego y-Le el Objec Recogni ion (J. Ponce, M. Hebe , C. Schmid, & A.
Zisse man (eds.); Vol. 4170). Sp inge Be lin Heidelbe g. h ps://doi.o g/10.1007/11957959
Nguyen, T. Q., & Salaza , J. (2019). T ans o me s wi hou Tea s: Imp o ing he No maliza ion o Sel -
A en ion. 1. h ps://doi.o g/10.5281/zenodo.3525484
No ig, S. R. and P. (2019). A i icial In elligence A Mode n App oach 4 h Ed. In Jou nal o Chemical
In o ma ion and Modeling (Vol. 53, Issue 9).
Pape , S. (1966). The summe ision p ojec (pp. 1–6). h p://dspace.mi .edu/handle/1721.1/6125
Pe schnigg, G., Szeliski, R., Ag awala, M., Cohen, M., Hoppe, H., & Toyama, K. (2004). Digi al
pho og aphy wi h lash and no- lash image pai s. ACM T ansac ions on G aphics, 23(3), 664–672.
h ps://doi.o g/10.1145/1015706.1015777
Poggio, T., To e, V., & Koch, C. (1987). Compu a ional ision and egula iza ion heo y. Readings in
Compu e Vision, 638–643. h ps://doi.o g/10.1016/b978-0-08-051581-6.50061-1
P ince, S. D. J. (Uni e si y C. L. (2012). Compu e Vision: Models, Lea ning and In e ence.
h p://www.camb idge.o g/9781107011793
Pueyo, S. (2018). G ow h, deg ow h, and he challenge o a i icial supe in elligence. Jou nal o Cleane
P oduc ion, 197, 1731–1736. h ps://doi.o g/10.1016/j.jclep o.2016.12.138
Pul o d, G. W. (2005). Taxonomy o mul iple a ge acking me hods. IEE P oceedings: Rada , Sona
and Na iga ion, 152(5), 291–304. h ps://doi.o g/10.1049/ip- sn:20045064
Ramachand a, V. (2019). Causal in e ence o clima e change e en s om sa elli e image ime se ies
using compu e ision and deep lea ning. h p://a xi .o g/abs/1910.11492
Rehg, J. M., & Kanade, T. (1994). Visual acking o high DOF a icula ed s uc u es: An applica ion o
33
human hand acking. In Lec u e No es in Compu e Science (including subse ies Lec u e No es in
A i icial In elligence and Lec u e No es in Bioin o ma ics): Vol. 801 LNCS (pp. 35–46).
h ps://doi.o g/10.1007/BFb0028333
Resea ch, C. (2019). Da a Enginee ing, P epa a ion, and Labeling o AI 2019: Technical Repo .
Resea ch, G. V. (2021). Da a Collec ion And Labeling Ma ke Size, Sha e & T ends Analysis Repo .
h ps://doi.o g/10.1109/CVPR.2004.1315182
Robe s, L. G. (1963). Machine pe cep ion o h ee-dimensional solids. No embe .
h p://dspace.mi .edu/handle/1721.1/11589
Rosen eld, A. (1984). Some Use ul P ope ies o Py amids. 2–5. h ps://doi.o g/10.1007/978-3-642-
51590-3_1
Rosen eld, Az iel, & Kak, A. C. (2019). Digi al pic u e p ocessing 2nd edi ion. In Jou nal o Chemical
In o ma ion and Modeling (Vol. 1, Issue 1).
Rosen eld, Az iel, & P al z, J. L. (1966). Sequen ial Ope a ions in Digi al Pic u e P ocessing. Jou nal o
he ACM (JACM), 13(4), 471–494. h ps://doi.o g/10.1145/321356.321357
Russell, B. C., To alba, A., Mu phy, K. P., & F eeman, W. T. (2008). LabelMe: A Da abase and Web-
Based Tool o Image Anno a ion. In e na ional Jou nal o Compu e Vision, 77(1–3), 157–173.
h ps://doi.o g/10.1007/s11263-007-0090-8
S., L. L., Blake, A., & Zisse man, A. (1987). Visual Recons uc ion. In Ma hema ics o Compu a ion (Vol.
53, Issue 188). h ps://doi.o g/10.2307/2008745
Said, A. F., Kashyap, V., Choudhu y, N., & Akhba i, F. (2017). A cos -e ec i e, as , and obus
anno a ion ool. P oceedings - Applied Image y Pa e n Recogni ion Wo kshop, 2017-Oc ob, 1–6.
h ps://doi.o g/10.1109/AIPR.2017.8457958
Saxena, D., & Cao, J. (2020). Gene a i e Ad e sa ial Ne wo ks (GANs): Challenges, Solu ions, and Fu u e
Di ec ions. 54(3). h p://a xi .o g/abs/2005.00065
Schei e , W. J., An hony, S. E., Nakayama, K., & Cox, D. D. (2014). Pe cep ual anno a ion: Measu ing
human ision o imp o e compu e ision. IEEE T ansac ions on Pa e n Analysis and Machine
In elligence, 36(8), 1679–1686. h ps://doi.o g/10.1109/TPAMI.2013.2297711
Shapi o, L. G. (2020). Compu e ision: he las 50 yea s. In e na ional Jou nal o Pa allel, Eme gen
and Dis ibu ed Sys ems, 35(2), 112–117. h ps://doi.o g/10.1080/17445760.2018.1469018
Sidenbladh, H., Black, M. J., & Flee , D. J. (2000). S ochas ic T acking o 3D Human Figu es Using 2D
Image Mo ion. In Jou nal o Explosi es Enginee ing (Vol. 31, Issue 6, pp. 702–718).
h ps://doi.o g/10.1007/3-540-45053-X_45
Solem, J. E. (2012). P og amming Compu e Vision wi h Py hon: Tools and algo i hms o analyzing
images (Vol. 1, Issue 1). O’Reilly Media, Inc.
Szeliski, R. (2010). Compu e Vision: Algo i hms and Applica ions (Vol. 1). Sp inge -Ve lag London.
