Validación au omá ica de guías de accesibilidad en
ideojuegos
Au oma ic alida ion o accesibili y guidelines in
ideogames
T abajo de Fin de G ado
Cu so 2021–2022
Au o
Ad ián Ál a ez Be nabé
Es eban Res epo Gu ié ez
Di ec o
Guille mo Jimenez Díaz
Colabo ado
Alessand o Len ini
G ado en Desa ollo de Videojuegos
Facul ad de In o má ica
Uni e sidad Complu ense de Mad id
Validación au omá ica de guías de
accesibilidad en ideojuegos
Au oma ic alida ion o accesibili y
guidelines in ideogames
T abajo de Fin de G ado en Desa ollo de Videojuegos
Au o
Ad ián Ál a ez Be nabé
Es eban Res epo Gu ié ez
Di ec o
Guille mo Jimenez Díaz
Colabo ado
Alessand o Len ini
Con oca o ia: Junio 2022
Cali icación: No a
G ado en Desa ollo de Videojuegos
Facul ad de In o má ica
Uni e sidad Complu ense de Mad id
30 de mayo de 2022
Resumen
La accesibilidad es un aspec o undamen al de los ideojueogos. Todos los jugado es,
independien emen e de sus capacidades isicas, deben pode dis u a de ellos. Al ededo
del mundo, dis in os gobie nos y asociaciones han c eado es ánda es con los que medi la
accesibilidad de los ideojuegos pa a que puedan se dis u ados po odos los públicos.
Uno de es os casos que hay que ene en cuen a al desa olla ideojuegos, y en el cual
se cen a es e abajo, es el de las pe sonas con discapacidad isual como el dal onismo o
la miopía. Ellos pueden ene di icul ad pa a e bien el ex o en un juego cuando es e es
muy pequeño en elación al amaño de la pan alla o cuando no hay su icien e con as e de
colo en e la le a y el ondo.
A dia de hoy es comun que las emp esas cuen en con una ex ensa plan illa de es e s
que comp ueba manualmen e los eque imien os legales y la calidad del juego. Es as com-
p obaciones, aunque sencillas, esul an labo iosas y consumen un iempo muy alioso que
pod ía dedica se a p uebas sólo ealizables po humanos. Además, la endencia a c ea
juegos que ienen cada ez más con enido hace que es e sea un olumen de abajo cada
ez menos asumible po una ue za de abajo manual.
Es e abajo consis e en el desa ollo una he amien a que pe mi e au oma iza el eco-
nocimien o de ex o pa a de ec a su amaño y su con as e con el ondo, pa a asegu a nos
de que cumple una se ie de c i e ios que asegu an su accesibilidad a a ios públicos.
Palab as cla e
accesibilidad, ideojuegos, ex o, con as e, luminancia

Abs ac
Accessibili y is a undamen al aspec o ideo games. Any playe should be able o enjoy
a game, independen ly o hei physical abili y. All a ound he wo ld, go e nmen s and
associa ions ha e implemen ed s anda ds o game de elope s o make su e hei ideo
games can be enjoyed by e e yone. One o he cases o keep in mind when de eloping
games, and he one his p ojec is based on, is he case o people wi h de icien colo
ision o nea sigh edness. These use s can ha e di icul ies seeing ex in-game when i
is ela i ely small when compa ed o sc een size o when i s colo does no ha e enough
con as wi h he backg ound.
I is commonplace o companies o ha e a conside able eam o es e s who manually
check o legal accessibili y equi emen s as well as ensu ing he quali y o he p oduc .
These checks, al hough simple, a e ime consuming, aking up aluable ime ha could be
used o mo e complex checks ha could only be made by humans. Mo eo e , endency
o c ea e inc easingly con en -packed games c ea es less manageable wo kloads o manual
checks.
Fo his p ojec , a ool has been de eloped ha au oma es he ex de ec ion and
calcula es i s size and ela i e luminance con as wi h he backg ound. This ool is used
o e i y ha a se ies o accessibili y c i e ia a e being me .
Keywo ds
accessibili y, ideo games, accessibili y, ex , con as , luminance
ii
Con en s
1. In oduc ion 1
1.1. Mo i a ion..................................... 1
1.2. Objec i es..................................... 2
1.3. Wo kplan..................................... 2
1.4. Documen ou line................................. 3
2. S a e o he A 5
2.1. De icien colo ision............................... 5
2.1.1. Making media accessible o colo ision de iciency . . . . . . . . . . . 6
2.1.2. The case o ideogames . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2. Myopia ...................................... 7
2.3. Accessibili y Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.3.1. Communica ions and Video Accessibili y Ac . . . . . . . . . . . . . 8
2.3.2. Web Con en Accessibili y Guidelines . . . . . . . . . . . . . . . . . . 8
2.3.3. Mic oso guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.4. Colo ep esen a ion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.5. Tex de ec ion and ecogni ion . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.5.1. EAST - E icien and accu a e scene ex de ec o . . . . . . . . . . . 11
2.5.2. Tesse ac - Op ical cha ac e ecogni ion engine . . . . . . . . . . . . 12
2.5.3. OpenCV Tex Recogni ion models . . . . . . . . . . . . . . . . . . . 12
2.6. Conclusion..................................... 13
3. Design o a ool o au oma ically alida ing accessibili y guidelines 15
3.1. Mo i a ion..................................... 15
3.1.1. P ocess o ex size measu emen . . . . . . . . . . . . . . . . . . . 15
3.1.2. P ocess o ex con as measu emen . . . . . . . . . . . . . . . . . 16
3.2. TinEye....................................... 17
3.2.1. Tex de ec ion .............................. 17
3.2.2. GuidelineChecks............................. 18
3.2.3. Repo gene a ion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.3. Con igu a ion................................... 19
3.4. Conclusion..................................... 20
4. Tool De elopmen 21
ix

Chap e 2
S a e o he A
This chap e will de ail some o he di e en ision impai men s ha ideo game use s
su e , like colo ision de iciency (sec ion 2.1) o myopia (sec ion 2.2), and how hey can
a ec hei game expe ience. Nex i will men ion di e en s anda ds and guidelines ha
a e used in he indus y o ensu e any pe son can enjoy in sec ion 2.3, and how hey ha e
o ake in o accoun opics like colo spaces (sec ion 2.4) since hey change how colo s a e
ep esen ed on a sc een. Finally, a ious ex de ec ion and ecogni ion solu ions ha a e
a ailable and could be used o de ec sc een ex will be explained along wi h hei mos
common use cases in sec ion 2.5.
2.1. De icien colo ision
De icien colo ision, inaccu a ely called colo -blindness, occu s when someone canno
pe cei e colo as he majo i y o people. This si ua ion eme ges om he pa ial mal unc-
ion o a g oup o colo cones in he eye. Humans pe cei e colo om h ee ypes o cones
in he eye, sensible o he di e en wa eleng hs o he ligh spec um a ibu able o ed,
g een and blue (Sha pe, 1999). When one o hese ypes o cones is missing o mal unc ion-
ing, i becomes di icul o e en impossible o dis inguish be ween some colo s. Depending
on which ypes o cones a e impai ed, he di e en cases ecei e di e en names:
P o anopia: no ed cones
P o anomaly: impai ed ed cones, can pe cei e some shades o ed
Deu e anopia: no g een cones
Deu e anomaly: impai ed g een cones, can pe cei e some shades o g een
T i anopia: no blue cones
T i anomaly: impai ed blue cones, can pe cei e some shades o blue
Ach oma opsia: comple e lack o colo ision.
Deu e anopia, p o anopia and i anopia a e g ouped as dich omacy. (Tanaka e al.,
2010). Compa ison o how a pe son wi h each ype o colo ision de iciency would see he
s anda d colo wheel can be seen in igu e 2.1
5
6Chap e 2. S a e o he A
Figu e 2.1: Compa ison o he isible colo spec um in common ypes o colo ision
de iciency (Melillo e al., 2017)
2.1.1. Making media accessible o colo ision de iciency
The e a e many alid app oaches o making media accessible o people wi h de i-
cien colo ision. One o he me hods o imp o e legibili y is changing he colo ligh ness
(Tanaka e al., 2010), which in ol e small changes wi hou comple ely al e ing he colo .
This makes o an inc ease in con as ha makes wo p e iously simila colo s dis in-
guishable o people wi h de icien colo ision. The p oblem wi h ligh ness modi ica ion
is i s cos , as mos o his me hods equi e a ious i e a ions, complex algo i hm, o a se
o pa ame e s ha a y om ame o ame.
Ano he me hod widely used o making media accessible is colo il e ing. By speci y-
ing he ype o colo de iciency, p oblema ic colo alues o he speci ic de iciency can be
swapped ou o o he s so he use can enjoy he media as showcased in igu e 2.2.
Figu e 2.2: Compa ison o how colo il e ing enables colo ision de icien use s o be e
dis inguish ce ain colo s (Tycho Henzen, 2021)
2.2. Myopia 7
2.1.2. The case o ideogames
Video games p esen some unique challenges in compa ison o o he media: wha is
displayed on he sc een is no known be o ehand like a mo ie, i is being gene a ed and
ende ed up o six y imes a second.
Pe o mance is a c i ical aspec in mos ideo games o a smoo h expe ience, pa ic-
ula ly wi h he ise in popula i y o i ual eali y (VR); he high cos o he p e iously
men ioned ligh ness modi ica ion makes i a non iable solu ion o sol ing colo ision
de iciency.
In addi ion o echnical challenges, ideo games also ace a is ical ones. Colo is as
undamen al o ideogame design as i is o any o m o a . One o he main goals o playe s
is o eel some so o emo ion (Joos en e al., 2010); a is s hen use e e y a ailable ool,
including colo , o c a hese emo ional pieces. This aises a p oblem wi h colo il e ing:
he in ended ision o he designe s may be mangled by colo il e ing, as seen in igu e 2.3.
Figu e 2.3: Game O e wa ch wi h no mal colo se ings (abo e) and wi h i anopia il e
applied (below) (Ashley Wood, 2017)
2.2. Myopia
Myopia, commonly e e ed o as nea sigh edness, is a ision de iciency ha causes he
pe son o see objec s ha a e a away as blu y while s ill being able o see close objec s
clea ly.
Myopia is g ea ly p esen ac oss he wo ld, wi h he highes incidence in asian popu-
8Chap e 2. S a e o he A
la ions (84% in 16 o 18 yea olds) and wi h lesse bu s ill signi ican incidence in o he
egions: 49.7% in Swedish kids aged 12 o 13, o 37.2% in 10 o 15 yea olds in G eece and
Bulga ia (Wu e al., 2016).
People wi h myopia commonly s uggle o see a away ex and images like oad signs
o billboa ds clea ly. This may no seem oo ele an o ideo games a i s , bu consoles
( ep esen ing mo e han $50 billion in e enue in he global ma ke in 2021 (Tom Wijman,
2021)) a e ecommended o be played om a dis ance o a leas one me e (HDhes, 2014),
which o highe diop e use s al eady poses a p oblem o legibili y.
The commonly accep ed solu ion o his is ensu ing ha ex has a minimum size
depending on he sc een size so ha i is legible o mos use s. Fo example, Mic oso
has implemen ed a se o guidelines ha a games mus ollow in o de o be published o
hei console.This means ha an Xbox i le’s ex has o ha e a minimum heigh o 28
pixels in a 1080 pixel all sc een (Mic oso , 2021).
2.3. Accessibili y Guidelines
To ensu e ha any pe son ega dless o hei abili y can enjoy ideo games and o he
media di e en go e nmen al bodies and independen o ganiza ions a ound he wo ld ha e
c ea ed se s o guidelines o accessibili y. This way use s who ha e special needs a e be
able o comple ely enjoy a piece o media like he no ma i e a ge audience would be.
The ollowing subsec ions will explo e he main guidelines being ollowed.
2.3.1. Communica ions and Video Accessibili y Ac
The 21s Cen u y Communica ions and Video Accessibili y Ac (CVAA) (FCC, 2010)
comp ises a se o legal equi emen s ha media should ul ill in he Uni ed s a es, including
ideogames. This was c ea ed o o upda e he accessibili y laws c ea ed in he 1980s and
1990s o mode n digi al s anda ds.
This ac en o ces i ems like closed cap ioning a ailabili y being manda o y o any
digi al ideo dis ibu ed by companies in he Uni ed S a es. This equi emen is applicable
o ideogames. Many ideogame i les use p e- ende ed oo age known as cu -scenes o
s o y elling ha a e subjec o his law.
CVAA se s a legal s anda d o all communica ions inside a game such as in e connec ed
oice o e p o ocol (VoIP), non-in e connec ed VoIP, elec onic messaging, in e -ope able
ideo con e encing and ideo communica ions.
This law is applicable o media p oduce s and ideogame publishe s wi h su icien
esou ces o en o ce i . This p esen s a double edged-swo d o ideogame companies. The
companies ha ha e he esou ces o ake ac ion a e usually he ones who o e bigge
amoun s o con en , making hei games ha de o es .
2.3.2. Web Con en Accessibili y Guidelines
The Wo ld Wide Web Conso ium (W3C) also c ea ed hei own se o guidelines o
digi al con en called he Web Con en Accessibili y Guidelines (WCAG). They "de ine
how o make Web con en mo e accessible o people wi h disabili ies [...] including isual,
audi o y, physical, speech..." -W3C (2008b).
The web con en accessibili y guidelines ha e gone h ough a ious i e a ions, e sion
1.0 being eleased in 1999, 2.0 in 2008 and 3.0 in he wo ks, wi h a wo king d a om
2021.
2.3. Accessibili y Guidelines 9
W3C cu en ly ecommends con en c ea o s on he in e ne o adhe e o he WCAG 2
guidelines. WCAG 3.0 builds upon p e ious e sions bu i s me hods o ensu e accessibili y
change, making some con en ha passes ollowing he 2.X s anda ds o no pass anymo e,
as will be de ailed sho ly.
2.3.2.1. WCAG 2.0
The WCAG 2 guidelines de ail a ious echniques and equi emen s o ensu e ha con-
en is no only accessible o people wi h colo ision de iciency o lesse sigh impai men s
like myopia bu also o use s wi h mo e limi ing condi ions like blindness o dea ness.
The a icle ha applies o ou wo k om he WCAG 2 guidelines is 1.4.3: "The isual
p esen a ion o ex and images o ex has a con as a io o a leas 4.5:1" (W3C, 2008a).
I s a es ha ex should ha e a minimum luminance con as a io o 3 be ween ex and
backg ound when ce ain size equi emen s a e me o 4.5 when hese size equi emen s
canno be ensu ed.
Luminance (L) is used o measu e he in ensi y o ligh in a gi en a ea; measu ed
in candela ( he SI uni y o measu ing luminous in ensi y) pe squa e me e . Rela i e
luminance (Y), on he o he hand, no malizes luminance alues om 0 (absolu e black, no
ligh being e lec ed) o 1 (absolu e whi e, pe ec e lec ion) (Sik-Lányi, 2012).
Rela i e luminance is he s anda d uni o measu emen when wo king wi h digi al
media because i allows de elope s o a oid hinking abou how each colo combina ion
migh be pe cei ed by use s and jus keep in mind abou how b igh o da k a colo will
be seen when playing independen ly om he display.
The WCAG 2 s anda d (sec ion 2.3.2.1) dic a es ha a con as a io o 4.5 o mo e
be ween he backg ound colo and he ex colo ensu es ha mos use s will be able
o dis inguish he con en s, ega dless o he colo s used. This is possible because, as
exposed in sec ion 2.1, luminance is pe cei ed sepa a ely om colo . This con as a io is
calcula ed wi h equa ion 2.1.
Ra io = (L1+0.05)/(L2+0.05) (2.1)
Whe e L1 is he ela i e luminance o he ligh e colo and L2 he da ke one.
2.3.2.2. WCAG 3.0
WCAG3 upda es he calcula ion o con as checking in ex based on mo e ecen
esea ch on colo pe cep ion. I uses he Ad anced Pe cep ual Con as Algo i hm o
calcula e con as (W3C, 2021). This algo i hm, apa om he backg ound and o eg ound
colo like he WCAG 2 calcula ions do, also akes in o accoun he ex size and weigh
ela i e o he sc een space i occupies.
2.3.3. Mic oso guidelines
The ech gian Mic oso has also c ea ed i s own accessibili y guidelines o i les
eleased on hei Xbox sys ems. They de ail he s anda d ex display sizes and con as
equi emen s bu also se s anda ds o in-game wai imes, UI na iga ion, e o messages
o hap ic eedback (Mic oso , 2021).
Mic oso ’s ex size guidelines s a e he ollowing o ex heigh in console i les:
26 pixel heigh a 1080p

10 Chap e 2. S a e o he A
52 pixel heigh a 4k
2.4. Colo ep esen a ion
As explained along WCAG 2.0 (sec ion 2.3.2.1), i is impo an o make ou calcula ions
in a way ha ma ches he physical pe cep ion o ligh as accu a ely as possible. The i s
measu e was he calcula ion o ela i e luminance wi h he equa ion 2.1. Luminance allows
us o measu e how ’b igh ’ o ’da k’ a pixel is wi hou ha ing o ake in o conside a ion
how i s speci ic colo is pe cei ed. A e i is assu ed ha a colo luminance is pe cei ed
equally by e e yone, i is needed o assu e ha he colo displayed is he same among all
displays.
I is impo an o deno e he exis ence o colo spaces and hei signi icance when
wo king in his ield: he human eye can see a wide ange o colo s p esen in he wo ld,
bu when ying o p esen hese colo s h ough a sc een, no e e y colo he human eye
can see can be ai h ully ep esen ed by ha dwa e, he mo e exac you wan you colo s o
be in ela ion o eali y he mo e expensi e and di icul o come by he ha dwa e o i
becomes.
To wo k a ound his issue, colo spaces we e in en ed. They a e s anda dized collec ions
o colo s ha could be ep esen ed on sc een so ha he same colo is ensu ed o be equal
ac oss o he de ices. Thanks o colo spaces, when c ea ing an image in one sc een you
would know i would show up he same on a di e en one i bo h a e capable o showing
he speci ied colo space.
The compa ison be ween human ision and di e en colo spaces can be seen in im-
age 2.4. As can be seen, pu e g een in sRGB does no equa e o pu e g een in o he colo
spaces. I we wan ed o use his colo alue o luminance calcula ions we need i s ac ual
colo alue in ’human’ colo space; his ans o ma ion is known as linea iza ion.
The s anda d colo space o compu e s and web is sRGB, which is clea ly mo e lim-
i ed han P oPho o RGB o Adobe RGB, bu wide colo spaces equi e mo e expensi e
ha dwa e, which is he eason such a colo space is as ex ended and wide ones a e only
used in p o essional se ings whe e he mos ai h ul ep esen a ion o colo s is needed.
2.5. Tex de ec ion and ecogni ion
The au oma ion o accessibili y es ing equi es some o m o ex de ec ion o ecogni-
ion. In he ollowing sec ion, he mos popula lib a ies and echniques will be discussed.
Tex de ec ion will co e he de ec ion o ex inside an a ea, wi hou ac ually being able
o ead i s cha ac e s. On he o he hand, ex ecogni ion will be able o ell wha wo ds
a e w i en in an image.
Op ical Cha ac e Recogni ion, o OCR, is a echnology ha allows ecognizing cha -
ac e s, such as le e s and symbols, h ough an op ical mechanism, such as an image aken
o a piece o pape . Al hough much less capable han humans, his echnology is widely
used o iden i y, ecognize and digi ize ex ound in images like scans om books and
ex ac hei ex (Mi he e al., 2013).
These echnologies can be used o also iden i y ex in digi ally c ea ed images, like a
sc eensho o a scene in a ideo game.
2.5. Tex de ec ion and ecogni ion 11
Figu e 2.4: Colo space compa isons (BenRG and cmglee, 2014)
2.5.1. EAST - E icien and accu a e scene ex de ec o
Fo he pu pose o de elopmen , he echnology used o analysing ex in eal wo ld
scena ios will be esea ched. Videogames a e a much mo e complex scena io o ex
de ec ion han plain ex : hey include a a ie y o on s, colo s and complex backg ounds.
Because o his, he ad ancemen s in scene ex de ec ion can p o ide e y use ul esul s
o he de elopmen o he ool.
EAST, E icien and Accu a e Scen e Tex De ec o , is a deep lea ning based on au-
oma ic ea u e de ec ion o he di e en cha ac e s. I s de elopmen was mo i a ed by
inc easing he F-sco e and speed o he cu en solu ions. EAST app oach ied o simpli y
he pipeline, emo ing many s ages conside ed unnecessa y and using wo-s age pipeline
(see igu e 2.5). EAST’s i s s age is a ully con ec ional neu al ne wo k ha assigns a
alue anging [0,1] o each pixels and hen encloses he alues in quad angles. The second
s age is a pos -p ocessing s age, he geome ies ha "su i e" he h esholding s age a e
p ocessed by a NMS, non-max supp ession, algo i hm. NMS ensu es ha he e is a sin-
gle quad angle o e e y single piece o ex ha co e s i s whole a ea ins ead o a ious
o e lapping quad angles. (Zhou e al., 2017)
12 Chap e 2. S a e o he A
Figu e 2.5: Compa ison o di e en scene ex de ec ion algo i hms om Zhou e al. (2017)
F om ou pe spec i e o use s, when using EAST as a black box we need o keep in mind
i s expec ed inpu and ou pu . EAST inpu mus be a 3-channel image whose dimensions
a e a mul iple o 32, and i s ou pu is a se ies o quad angles. This will make i necessa y
o keep in mind because he scale will need o be es o ed in o de o he measu emen s
o be accu a e.
2.5.2. Tesse ac - Op ical cha ac e ecogni ion engine
The mos widesp ead solu ion o ex ecogni ion is Tesse ac . O iginally a Hewle
Packa d p oduc , i was eleased as open sou ce in 2005 (Communi y, 2022). Tesse ac
has been de eloped mainly as a ool o digi izing documen s; i has se e al laye s o
p ep ocessing ha a e no use ul o digi ally gene a ed images, o example s eps a e
aken o educe w inkles in pape o a i ac s ha appea when scanning a documen .
The e is a ocus on e o s na u al o ex documen s such as misaligned ex , ben pages,
b oken cha ac e s (Smi h, 2007) ha do no apply when wo king wi h digi al con en .
When unning an image h ough an OCR engine he e is usually a necessa y s ep aken
called bina iza ion, simpli ica ion o i s pixels o a alue o 1 o 0 so ha i becomes an
image composed o only black and whi e pixels o ind ou whe e he le e s a e. Tesse ac
can make mos o he wo k o i s use s. Fi s , i au oma ically bina izes he image
being p ocessed and hen denoises i , inally unning he ex ecogni ion pa o he
p ocess. When wo king wi h digi al images, easing his bina iza ion is e y impo an , as
mo e complex backg ounds (opposed o jus a whi e page o la backg ound om no mal
documen s) a e o be expec ed and he engine may no be as accu a e.
2.5.3. OpenCV Tex Recogni ion models
OpenCV o e s high le el APIs o bo h ex de ec ion and ecogni ion. Tex de ec ion
is use ul o inding whe he ex is p esen in a scene and i s posi ion. Tex ecogni ion
needs o be gi en a speci ic egion o an image known o con ain ex ou pu s a e ob aining
hese egions ough ex de ec ion. A e wa ds, i ge s he ex con ained wi hin h ough
a se ies o calcula ions using neu al ne wo ks.
Ope a ions wi h neu al ne wo ks can be e y e icien and seemingly magical, bu hey
all depend on he neu al ne wo k being co ec ly ained wi h enough examples in i s
aining da a se s.
The OpenCV documen a ion o e s esou ces o a ious ained models, each one o
a ying complexi y depending on how big an alphabe o possible cha ac e s i suppo s:
2.6. Conclusion 13
CRNN o black and whi e images, 36 cha ac e alphabe (0-9 + a-z).
CRNN o colo images, 94 cha ac e alphabe (0-9 + a-z + A+Z + punc ua ions).
CRNN o colo images, 3944 cha ac e alphabe (0-9 + a-z + A-Z + chinese cha -
ac e s + special cha ac e s).
The a ailabili y o di e se p e- ained ne wo ks and he capaci y o changing he ecog-
nized cha ac e se make OpenCV’s ex ecogni ion a use ul asse . A ool can be de eloped
ollowing only he English alphabe and, i needed, could la e be expanded o include mo e
alphabe s. This sys em also o e s mo e lexibili y ega ding i s sou ce media compa ed o
Tesse ac , which expec s o ge a scanned documen .
2.6. Conclusion
This chap e has exposed how di e en sigh p oblems can nega i ely impac playe s’
abili y o enjoy a game o ead c i ical ex . Mi iga ing his impac s is ul ima ely he
de elope ’s esponsibili y. Games should be uni e sally accessible bo h because o e hical
p inciples ega ding i s consume base and because o he big pe cen age o playe s hey
can ep esen , like myopia in Asian popula ions.
The amoun o esou ces needed o keep games accessible is conside able, and becomes
bigge as he amoun o con en o e ed by games inc ease. While a en ion is b ough
o his opic, some coun ies like he Uni ed S a es a e s a ing o en o ce laws ha all
published media should comply. Console manu ac u es like Mic oso a e also aking a
s ance owa ds accessibili y. The console gaming business model elies hea ily in wha is
conside ed "couch gaming", being able o play in a li ing oom a away om he sc een,
hus needing o ensu e i s eadabili y. The inabili y o publish a game in a console due o
accessibili y guidelines could be a huge backlash o a publishe .
The inc easing amoun o esou ces needed o keep games accessible pai ed wi h he
g owing legal p essu e makes he au oma ion o hese asks a big asse o game de elop-
men companies. Finally, he manual app oach ega ding many o hese es s make hem
sub-op imal and p one o human e o . Au oma ion o his checks could esul no only in
ime sa ed, bu also in an accu acy inc ease.
20 Chap e 3. Design o a ool o au oma ically alida ing accessibili y guidelines
•Igno e egions: Rec angles ha speci y egions o be igno ed, e en i i is inside
a ocus egion.
•Use ex ecogni ion: Tu n on o o ex ecogni ion. Tu ning o ex ecogni-
ion speeds up analysis bu wid h canno be measu ed.
•P in alues on esul : Whe he he use wan s he speci ic measu emen s ob-
ained in each check o be p esen in he ou pu image. This way he use can
easily see, i ex is ailing, by how much hey would need o inc ease con as
o size o make i pass.
3.4. Conclusion
This chap e has explo ed accessibili y guidelines and he manual p ocess ollowed
o alida e hem. Au oma ion is di ely needed in his ield. TinEye is a p oposal ha
au oma ically checks his guidelines in a way ha is bo h mo e exhaus i e and p ecise han
manual es ing. This p oposal is p omising and lexible, i can be applied o a ious o he
se ings o he han only ideo game use in e ace alida ion by changing i s con igu a ion.

Chap e 4
Tool De elopmen
This chap e desc ibes he de elopmen o TinEye o checking i a piece o gi en media,
speci ically sc eensho s o ideos om ideo games, is complian wi h a se o guidelines
ha ensu e ex is eadable o use s wi h de icien colo ision o myopia.
The whole de elopmen p ocess will be explained, s a ing ou wi h a p o o ype phase
o make su e ou app oach was iable (sec ion 4.1), ollowed by he p ope de elopmen o
he ool (sec ion 4.2) and i s subsequen i e a ions.
A e wa ds, i s inal a chi ec u e will be explained (sec ion 4.3), as well as he possible
con igu able a iables o de elope s in sec ion 4.3.3. Subsequen ly, he me hodology and
echniques used du ing de elopmen (sec ion 4.4) will be desc ibed. Finally, he a ious
es s ha we e designed o ensu e he ool wo ks as in ended will be de ailed in sec ion 4.5.
4.1. Ini ial p o o ypes
The de elopmen o he TinEye ool s a ed ou wi h a se ies o p o o ypes. These
p o o ypes helped he eam o amilia ize hemsel es wi h he chosen echnologies as well
as sol ing indi idually he pa ial p oblems he ool will ackle. The main objec i es o
he p o o ypes, aside om being a i s con ac wi h he lib a ies, we e he ollowing:
P ecisely calcula ing con as and pe cei ed luminance (subsec ion 4.1.1).
Tes ing he ex ecogni ion lib a ies o measu emen (subsec ion 4.1.2).
Seeing how EAST would beha e wi h ideogame oo age consis ing o a scene and a
HUD (subsec ion 4.1.3).
Du ing de elopmen , he eam ollowed he Agile me hodology p inciples. This was due
o he eam al eady ha ing expe ience wo king wi h agile me hodologies and ha hese
p ac ices a e be e when wo king in small mul i-disciplina eams. These p inciples a e
he ollowing:
Indi iduals and in e ac ions o e p ocesses and ools
Wo king so wa e o e comp ehensi e documen a ion
Cus ome collabo a ion o e con ac nego ia ion
Responding o change o e ollowing a plan
21
22 Chap e 4. Tool De elopmen
Addi ionally, small 2-week i e a ions (sp in s) we e planned, du ing which he highes
p io i y asks we e ca ied ou . A he end o each sp in , he s a e o he ool would be
assessed and he p io i ies o he ollowing sp in would be de e mined. In line wi h he
second Agile p inciple, a wo king e sion o he ool would always be a ailable o use.
4.1.1. Rela i e luminance p o o ype
The goal o he i s p o o ype was calcula ing he ela i e luminance a io be ween
ex and i s backg ound. This p o o ype compu ed he luminance di e ences be ween
wo colo s in an image. The esul o his calcula ions can be compa ed o he minimum
con as alue expec ed by he WCAG guidelines (subsubsec ion 2.3.2.1) o check i i
p o ides su icien con as e en o people wi h some o m o colo ision de iciency.
Gi en an image and wo pixel coo dina es, i had o calcula e he ela i e luminance
Yand con as alue be ween he colo s o he wo pixels. I s unc ionali y mus mimic
he ’Con as Ra io’ ool om sec ion 3.1.2.
Using he ed, blue, and g een componen s o he colo s o an image, ela i e luminance
Ycan be calcula ed wi h equa ion 4.1.
Y= 0.2126 ∗Rlin + 0.7152 ∗Glin+=0.0722 ∗Blin (4.1)
Knowing he ela i e luminance Yo he colo s o bo h he ex and he ex ’s back-
g ound, he con as a io be ween he wo can be calcula ed wi h equa ion 2.1.
No ice ha in equa ion 4.1, colo alues o ed, g een and blue a e s a ed as lin. This
s ands o linea ized alues, which a e no he same alues you would ge when simply
opening an image in an image edi ing so wa e. As men ioned in he explana ion o colo
spaces in sec ion 2.4, colo s in hese colo spaces such as sRGB mus be linea ized be o e
he ela i e luminance can be calcula ed. The o mula o linea izing sRGB o ITU-R
BT.709 colo spaces is equa ion 4.2 o equa ion 4.3.
Rlin =R′2.2Glin =G′2.2Blin =B′2.2(4.2)
Fo mula 4.2 p io i izes calcula ion speed o e accu acy. A mo e accu a e e sion o
ans o m om sRGB o linea ized componen s is equa ion 4.3
i R <= 0.04045 hen Rlin =R/12.92 else Rlin = ((R+ 0.055)/1.055)2.4
i G <= 0.04045 hen Glin =G/12.92 else Glin = ((G+ 0.055)/1.055)2.4
i B <= 0.04045 hen Blin =B/12.92 else Blin = ((B+ 0.055)/1.055)2.4
(4.3)
Once all componen s ha e been linea ized he ela i e luminance Ycan be calcula ed
ollowing equa ion 4.1
4.1.1.1. Resul s o he ela i e luminance p o o ype
A se ies o es s we e an wi h small, simple images. These images consis ed o a g id
o known colo s whose con as could be easily calcula ed wi h ano he ool o check o
co ec ness.
4.1. Ini ial p o o ypes 23
Figu e 4.1: Resul s o compa ing a ed pixel wi h a yellow one wi h he con as p o o ype
The inal esul s, shown in Figu e 4.1 di e om he e e enced websi e due o di e en
decimal accu acy be ween p og amming languages and a di e en alue being used in he
linea iza ion calcula ion. This is because, in equa ion 4.3 Snook.ca (2015) uses a alue o
0.03928 and he p o o ype uses 0.04045, ollowing he IEC s anda d (MozillaCo po a ion,
2022).
4.1.2. Tesse ac OCR p o o ype
The objec i e o his p o o ype is o es cha ac e ecogni ion and measu emen in
di e en scena ios using he Tesse ac OCR engine. The measu emen should no di e
mo e han 2 pixels om wha could be measu ed by a human ca e ully coun ing he pixels
using so wa e ha p o ides a g id. This ma gin o e o is accep able o a oid e o s due
o an i-aliasing and image comp ession echniques. The e ec o an i-aliasing can be seen
in igu e 4.2, he e appea o be some colo ed pixels o se om he main cha ac e ’s shape.
Figu e 4.2: Blue and ed ou line when ende ing ex wi h MS Pain
4.1.2.1. Resul s o he Tesse ac p o o ype
A e implemen ing basic cha ac e ecogni ion wi h Google’s Tesse ac , he sample
images seen in igu e 4.3 we e p ocessed. Figu e 4.3a ep esen s black ex on whi e back-
g ound, Tesse ac ’s p e e ed o ma as a s anda d OCR. Figu e 4.3b p esen s colo ed ex
on a colo ed g adien , his is ou side o Tesse ac ’s p e e ed inpu , bu as s a ed by Smi h
24 Chap e 4. Tool De elopmen
(a) Black ex on whi e backg ound (b) Wo ds wi h low con as on a g a-
dien backg ound
(c) Luminance map o igu e
Figu e 4.3: Di e en media used o Tesse ac .
(2007) he OCR is based on pa e ns and con as ; making he colo o he ex much less
ele an han in o he algo i hms. Finally, in igu e 4.3c is he luminance map calcula ed
wi h he o mula used o luminance in 4.1.2. This would be ou p e e ed inpu o e colo
images since i akes in o accoun he pe cei ed luminance ega dless o he o iginal colo .
The esul s we e he ollowing:
Figu e 4.3a was ecognized pe ec ly
Figu e 4.3b mos wo ds un ecognized, only "cyan" was ecognised.
Figu e 4.3c (Figu e 4.3b’s luminance map) did no imp o e esul s, bu in his case
only he wo d "yellow" was ecognised.
In o de o check he size o he ex acco ding o he CVAA guidelines (FCC, 2010),
we need o ge an accu a e cha ac e measu emen s. Since Tesse ac allows o pe -symbol
i e a ion, some images we e c ea ed and manually measu ed in o de o check he accu acy
o he OCR bounding-boxes o each indi idual cha ac e . The c ea ed images (Figu e 4.4)
we e all black ex o e a whi e backg ound, he objec i e was o check p ecision in op imal
condi ions. The esul s o he OCR on he p esen ed pho os was he ollowing:
4.1. Ini ial p o o ypes 25
(a) le e , 10x19
pixels
(b) 5 le e s , 47x94px each
(c) es wo d, cha ac e s measu ing
31x40 27x32 21x32 19x39 pixels
Figu e 4.4: Di e en media used o Tesse ac .
Figu e 4.4a measu emen s we e co ec , epo ing 10 pixels wid h and 19 pixels
heigh .
Figu e 4.4b measu emen s con inued o be accu a e when conca ena ing a ious cha -
ac e s, epo ing 5 cha ac e s 47 pixels wide and 94 pixels all.
Figu e 4.4c accu acy p oblems when es ing wi h eal wo ds. The i s cha ac e
om his igu e co ec ly epo s 31x40 dimensions. The measu emen s o ollowing
cha ac e s a e co ec heigh -wise, bu he wid h g ea ly di e s esul ing in 51 pixels
o wid h o |"e" ins ead o he co ec 27 o 43 o he las " " ins ead o 19.
The main akeaways om he p esen ed esul s we e ha al e na i e OCR engines
would need o be explo ed. Tesse ac is a s a e o he a OCR engine bu is hea ily
ocused on scanned documen s. A e some mo e esea ch, and inding he esul s on
low-con as discou aging, he disc epancies be ween he ac ual accu acy and wha was
expec ed om he o e iew by Smi h (2007) p opelled u he esea ch, inding ou ha
Tesse ac d opped suppo o whi e ex o e black backg ound on e sion 4. This inding
con i med Tesse ac ’s inc easing bias owa ds ex documen s and aised he need o an
al e na i e OCR solu ion ha could ecognize ex in mo e complex scenes and digi ally
c ea ed images wi h di e en backg ounds.
4.1.3. EAST p o o ype o ex de ec ion
The objec i e o he ollowing p o o ype is o see how he EAST algo i hm beha es
wi h ideo game oo age. The p o o ype will in ol e ollowing a simple example p o ided
in he EAST documen a ion (Communi y, 2020) and using i o de ec ex in a ideo
game’s use in e ace o see i all he ex is de ec ed.

26 Chap e 4. Tool De elopmen
4.1.3.1. Resul s o he EAST p o o ype
EAST was es ed by c ea ing a simple command line ool based on he p oposed usage
o he sample implemen a ion p o ided in he EAST documen a ion and elying in he
p o ided model in Zhou e al. (2018). The esul s we e e y p omising, showing ull
ecogni ion o all he ex in scene in he es cases (Figu e 4.5).
(a) Base Image
(b) EAST de ec ion esul s
Figu e 4.5: Example inpu and ou pu o he EAST CLI
4.2. I e a i e de elopmen p ocess
A e inalizing he p o o ypes, he i e a i e de elopmen o TinEyeLib s a ed. The
p ocess s a ed wi h he basic unc ionali ies es ed in he p o o ypes (sec ion 4.1). Once
he s ages we e in eg a ed, a simple clien TinEyeApp ha loads media and ou pu s a
epo using he da a om he lib a y was de eloped.
4.2. I e a i e de elopmen p ocess 27
4.2.1. I e a ion 1: TinEyeLib
The i s s age o he de elopmen consis ed in making a lib a y ha combined he pa -
ial solu ions de eloped du ing he p o o yping p ocess. This lib a y was called TinEyeLib,
i exposed he necessa y me hods o es images o ex size and con as . The combi-
na ion o all h ee s ages c ea ed he necessi y o ha ing da a, like he image’s luminance
maps o he esul s ha we e gene a ed, pe sis be ween each s age. A class ha held he
media being p ocessed and i s analysis in o ma ion was c ea ed. This was he Media class,
he de ails o TinEye’s a chi ec u e a e explained la e in sec ion 4.3.
Once loaded, he media goes h ough a ex de ec ion s age (subsec ion 3.2.1) ha elies
on he EAST algo i hm. The de ec ed ex boxes a e sa ed in o he Media class a e simple
p ocessing. EAST is made o de ec ex in eal wo ld scenes, so he ex boxes gene a ed
a e no always aligned wi h he e ical and ho izon al axis. TinEye’s ex de ec ion s age
akes he esul om EAST and c ea es a ec angle ha is axis-aligned o help he ex
measu emen asks.
A e de ec ing whe e ex is p esen , he guidelines can be checked independen ly.
The ex measu emen s age ecei es a Media e e ence and i e a es h ough all o i s ex
boxes. Du ing his i s i e a ion, he measu emen o he heigh co esponds wi h he
heigh o ex box, and he wid h o indi idual cha ac e s is calcula ed by di iding he
wid h o he ex box by he numbe o cha ac e s de ec ed by he cha ac e ecogni ion
lib a y.
A his poin , we ealized ha we we e no con en enough wi h Tesse ac ’s pe o -
mance. We had hoped ha i would ecognize in-game ex be e . This p omp ed chang-
ing he ex ecogni ion sys em o he one p o ided by OpenCV.
4.2.1.1. Changes o ex ecogni ion
As shown in subsec ion 4.1.2, he esul s o Tesse ac we e discou aging. While being
one o he mos p omising ools ega ding ex ecogni ion in images, Tesse ac ell sho
when dealing wi h ex wi h complica ed backg ounds. The p o o ype could no ecognize
ex on op o a g adien , e en in condi ions wi h good con as (Figu e 4.3b). I is sa e o
assume ha his p oblems would ansla e when dealing wi h mo e complex backg ounds
like hose in ideo games o ex ha p esen s a low con as wi h i s backg ound. Ano he
ac o agains Tesse ac is ha since i is used o scan documen s i a emp s o ecognize
eal dic iona y wo ds and no jus s ings o cha ac e s, which when a emp ing o ecognize
use names o o he use c ea ed ex s, eal wo ds may no be ound.
A e delibe a ion, i was decided ha Tesse ac ’s capabili ies would no be su icien o
ensu e he co ec unc ioning o he ool. OpenCV bundles ex ecogni ion capabili ies,
which p o ide neu al ne wo k suppo o a ious ypes o cha ac e ecogni ion. The
decision du ing de elopmen was o keep he ex measu emen s age lexible in o de o
compa e he esul s o bo h OpenCV and Tesse ac o inally choose one o he o he
depending on hei pe o mance.
When implemen ing OpenCV cha ac e ecogni ion inside he ex measu emen s age,
he esul s as ly imp o ed on hose o he Tesse ac p o o ype as can be seen in igu e 4.6.
OpenCV’s solu ion ecognizes much mo e ex , mos no ably when di e se backg ounds
a e p esen . T ansi ion o use OpenCV’s p o ided OCR echnology om Tesse ac was a
success, and now he luminance calcula ion pa o he ool could be ackled.
28 Chap e 4. Tool De elopmen
(a) De ec ed ex egions in a sc eensho o a game.
(b) Resul s o unning image Figu e 4.6a
h ough OpenCV’s OCR algo i hm
(c) Resul s o unning image Figu e 4.6a
h ough Tesse ac wi h a minimum con i-
dence o 50%
Figu e 4.6: Pe o mance compa ison o Tesse ac s OpenCV’s ex ecogni ion
4.2.1.2. Tex backg ound calcula ion app oach
The emaining s age would be ha in cha ge o calcula ing he con as o each ex
piece wi h i s backg ound. This s age needs bo h a way o calcula ing ela i e luminance
and a way o sepa a ing ex om i s backg ound. The con as calcula ion was handled in
sec ion 4.1.1. To scale he desc ibed p o o ype, he luminance alues o he whole image a e
calcula ed a once aking ad an age o OpenCV’s pa alleliza ion when applying a unc ion
o all pixels in a ma ix.
The me hod o sepa a ing ex and i s backg ound a his poin is a nai e simpli ica-
ion. The mean luminance o each ex box is conside ed o be he mean luminance o he
ex , and he ou e edge’s mean luminance o he ex box, he backg ound luminance, as
can be seen in igu e 4.7. Once he con as is calcula ed, he compa ison esul agains
he guideline h eshold is s o ed in he Media esul s.
4.2. I e a i e de elopmen p ocess 29
Figu e 4.7: Example o a nai e luminance calcula ion.
4.2.2. I e a ion 2: Inc easing accu acy
A e ha ing a i s wo king e sion o he ool, se e al imp o emen s we e made.
Some o hese imp o emen s a e aimed a imp o ing he accu acy o he calcula ions.
O he imp o emen s such as me ging ex boxes o igno ing il ed ex we e aimed owa ds
p o iding a clea e epo .
4.2.2.1. His og am app oach
A e comple ing a i s e sion o he con as checking s age, imp o emen s needed
o be made in o de o ep esen he con as be ween he ex and i s backg ound. The
p oblem wi h he nai e app oach desc ibed in sec ion 4.2.1 is ha i akes in o accoun he
immedia ely adjacen backg ound o he ex in o he mean o he ex i sel . This means
ha when wo king wi h hin on s, he luminance alue om he ex i sel is much less
signi ican han i s immedia e backg ound.
The i s al e na i e was using his og ams. His og ams a e he ep esen a ion o da a
poin s g ouped in o wha a e usually called bins, whose heigh is de e mined by hei
absolu e o ela i e incidence. They can be used o ob ain he di e ence in luminance
be ween he o eg ound and he backg ound. Thei mos ep esen a i e alues in each o
hei his og ams could be used o ep esen hei ep esen a i e luminance and calcula e
he ela i e luminance a io.
In pho og aphy and image p ocessing his og ams a e usually used o isualize he in en-
si y o he di e en colo s in an image; o colo images using h ee-channeled his og ams
and o black and whi e jus a single channel. In he case o images wi h ew alues o
in ensi y (256 in mos consume cases) he bins used encompass only one alue, bu o
highe colo dep h images bigge bins may be used. An example o he his og am o a
iple channel image can be seen in igu e 4.8.
36 Chap e 4. Tool De elopmen
be e algo i hm is de eloped, his abs ac ion would make i easie o implemen and
compa e wi h he cu en me hod.
4.3.1.6. ITex boxRecogni ion and Tex boxRecogni ionOpenCV
This in e ace is in cha ge o abs ac ing he de ec ion o ex inside a ex box. The
cu en implemen a ion ecei es a Tex box and uses OpenCV models o ecognise i , bu
mo e implemen a ions can be de eloped in he u u e.
4.3.2. IChecke , Con as Checke and SizeChecke
This in e ace allows us o abs ac he size and con as calcula ions om he main
applica ion. The main goals o his abs ac ions a e he abili y o p o ide al e na e size
and con as calcula ions in he u u e and o be able o add new checks i p o en necessa y.
4.3.3. Con igu a ion
The ool needed o be lexible and allow o di e en con igu a ions based on he lan-
guage and local accessibili y laws applied o he media being p ocessed as well as di e en
guidelines he de elope migh wan o adhe e o. Minimum equi ed alues need o be
easily upda ed, and om he echnical side, ex de ec ion and p ocessing needs o be
easily con igu able o adjus accu acy and se o used cha ac e s.
The con igu a ion comes in he o m o a .json ile, due o i s ease o use and ha i
is easy o pa se wi h a simple lib a y.
The Con igu a ion class is a simple pa se ha holds small classes wi h compa men-
alized in o ma ion o di e en s ages o he ool, shown as i s a ibu es in igu e 4.14.
This a ibu e classes a e no shown in he diag am o simplici y, since hey a e simple
classes consis ing o ge e s and se e s o he ollowing da a ha should be p esen in he
json ile:
Guideline se s he minimum s anda ds o adhe e o:
•Floa ing poin alue de e mining he minimum con as be ween ela i e lumi-
nance ha ex mus ha e wi h i s backg ound (sec ion 2.3.2.1).
•In ege alue o he adius o he ex ’s ou line ha is conside ed o be he i s
backg ound (men ioned in 4.2.2.2).
•A lis o esolu ions wi h hei speci ied sizes o minimum ex heigh and
minimum a e age cha ac e wid h as in ege s.
App Se ings con igu es wha in o ma ion is sa ed du ing execu ion o he ool, so
he de elope can isualize he in e media e s eps aken and debug as needed. I also
changes he way he ool execu es, so he use can cus omize i o hei liking
•Use ex ecogni ion: Boolean alue ha u ns on o o ex ecogni ion. I i
is u ned o , i speeds up he ool, as i does no a emp o ecognize ex in
a ex box, bu i canno measu e i s wid h.
•Sa e ex de ec ion s age esul s: Boolean alue o whe he show he unp o-
cessed esul s o he EAST ex de ec ion as seen in igu e 3.3b.
•Sa e luminance map: Boolean alue o whe he o ou pu a clone o he media
ha ep esen s he luminance (sec ion 2.3.2.1) o each indi idual pixel.

4.4. Con inuous in eg a ion 37
•Sa e luminance masks: Boolean alue o whe he o sa e he ex mask and i s
ou line o memo y.
•Focus egions: Rec angles ha de ine egions in which o ocus he analysis o
he image and igno e e e y hing ou side. They a e de ined as a lis o objec s
ha ha e x and y coo dina es as well as a wid h and heigh .
•Igno e egions: Rec angles ha speci y egions o be igno ed, e en i i is inside
a ocus egion. They a e de ined he same way as ocus egions.
•P in alues on esul : Boolean alue i he use wan s he speci ic measu emen s
ob ained in each check o be p esen in he ou pu image. I he lag is on, when
gene a ing he ou pu image, he esul alue is a emp ed o pu o he igh
o he ex box. I he ex would no i in he image when being pu on he
igh i is pu on he le .
Tex de ec ion Pa ame e s allows he use o con igu e EAST pa ame e s and
o he ope a ions ela ing o ex de ec ion, like how many deg ees a ex box can be
o a ed be o e being ma ked as en i onmen al ex .
•EAST pa ame e s: Floa ing poin alues o non max supp ession algo i hm,
minimum con idence, de ec ion scale and mean.
•Ro a ion h eshold deg ee: In ege alue exp essed in deg ees. I a de ec ed
ex box is o a ed mo e han his h eshold alue i will be disca ded.
•Me ge h eshold: Floa ing poin alues in a ange om 0 o 1 o he x and y
h eshold o me ging. I wo ex boxes o e lap o e a ce ain amoun hey will
be me ged in o a singula ex box (de ails on how his wo ks in sec ion 4.2.4).
Tex ecogni ion pa ame e s ha allow o con igu a ion o he ex ecogni ion
module’s pa ame e s
•Model: The ilename o he speci ied neu al ne wo k model o OpenCV (4.2.1.1)
ha will be loaded o ecogni ion.
•Vocabula y: The ilename o he ocabula y ile ha indica es which cha ac e s
a e expec ed o be ound in scene.
•Neu al ne wo k pa ame e s: loa ing poin alues o mean, scale, decode ype
and in ege alues o inpu size.
Linea iza ion alues o sRGB: A lis o 256 p ecalcula ed alues o he sRGB
linea iza ion. This p e en s he loa ing poin e o ha appea s when a emp ing
o linea ize sRGB alues du ing un ime.
I a child en objec is mal o med o missing om he json ile, he pa se in Con igu a ion
will c ea e one om allback alues om he sou ce code. I he whole con igu a ion ile is
missing o mal o med, all he membe classes will be buil om hese allback alues.
4.4. Con inuous in eg a ion
The e sion con ol sys em used o he p ojec was gi , due o he eams amilia i y
wi h i and i being widely used ac oss he so wa e indus y. As an ex a s ep o ensu e
ha he whole eam was always amilia wi h wha was going on in he code we ins i u ed
manda o y code e iews o e e y change o he code base. Tha way we we e always up o
38 Chap e 4. Tool De elopmen
da e wi h any hing ha changed and he code passed h ough a second se o eyes be o e
being me ged in o he main b anch, p e en ing possible e o s. This app oach was possible
because o he limi ed size o he eam, consis ing o wo membe s.
To ensu e TinEye always p esen ed a wo king e sion, bo h in a windows de elopmen
en i onmen and a linux-based dis ibu ion, he Elec onic A s membe s p o ided ou
eam wi h a con inuous in eg a ion pipeline. Usually, when wo king on so wa e in eams
o a ious people, when one pe son wan s o in oduce a change in o he codebase hey
c ea e a eques o enac hese changes. Ano he de elope has o come along and check
hei code, download i , and es i locally o make su e ha e e y hing wo ks co ec ly.
This is whe e he con inuous in eg a ion pipeline comes in. When a de elope wan s o
add hei changes o he so wa e, an au oma ed build p ocess is execu ed ha checks ha
e e y hing builds co ec ly and uns p e iously designed es s ha ensu e he p og am
wo ks as in ended. A e hese au oma ic checks a e un ano he de elope can simply
con i m he co ec ness o hese checks and me ge he changes igh away (Ma in Fowle ,
2006).
The p o ided pipeline an on a Debian docke image had 3 unc ions:
1. Building he C++ p ojec
2. Running a se o es s de ised by us
3. Checking o memo y leaks, by unning a second ime ou es sui e
The p ojec build s age loads all o he necessa y dependencies on o he docke image,
like OpenCV o he C++ compile s, and compiles all o he p ojec s ela ed o he ool:
The main lib a y, he console applica ion and he uni es s.
I he compila ion ails on any o hese p ojec s he pipeline execu ion s ops and aises
an e o .
4.5. Tes s
A ba e y o es s, anging om uni es s es ing ou speci ic unc ions in he code o
accep ance es s ha make su e ha passing es cases always pass and ailing ones always
ail, we e de ised o make su e ha a ious aspec s o he ool we e wo king co ec ly.
They ensu e ha he ool would no b eak when pe o ming inc emen al ad ances o he
code o when e ac o ing some sec ion o i .
The es s can be di ided in o he ollowing ca ego ies:
Luminance calcula ion checks
•A whi e image has a mean luminance o 1.
A comple ely whi e image should ha e a mean luminance o 1, he maximum
luminance alue. This es helped us disco e ha he e was an app oxima-
ion e o when calcula ing he luminance wi h loa ing poin decimals a e
linea izing RGB colo s. The e o was sol ed by implemen ing look up ables
wi h p ecalcula ed alues.
•Luminance mean calcula ion wi h masks.
Selec ing a egion o an image wi h a mask should only calcula e he mean o
he speci ied egions, allowing o he backg ound and o eg ound luminance
calcula ions.
4.5. Tes s 39
•Maximum con as .
The speci ied maximum ela i e luminance con as is 21, he a io be ween he
luminance alues o pu e whi e and pu e black. This es ensu es ha bo h he
luminance ex ac ion o hese colo s and he con as be ween hem a e co ec .
•Con as a io is commu a i e.
Con as a io calcula ions should be commu a i e, he esul o he a io cal-
cula ion should be he same whe he he i s ope and is he ligh es alue and
he second he da kes and ice e sa.
Image ope a ions checks
•Luminance lip o a comple e image.
When lipping an image’s luminance each o he esul ing pixels’ luminance
should be one minus hei p e ious luminance.
•Luminance lip o a egion.
I s beha io is he same as he p e ious es bu pixels ou side o he speci ied
egion should emain unchanged.
•Double luminance lip.
When lipping an image’s luminance wice i should be back o i s o iginal s a e.
•Double luminance lip o a egion.
As wi h p e ious es s, he speci ied egion o an image should be unchanged
when lipping i s luminance wice.
Con as checks
A se ies o images we e de ised wi h speci ic colo alues nea he h eshold in di e -
en ci cums ances o make su e ha he ool esponded co ec ly. High con as , low
con as , and la , g adien and s iped backg ounds we e used. These es images
can be seen in igu e 4.15
Figu e 4.15: Di e se con as checks o make su e calcula ions a e co ec
Guideline es s
The ool always ies o load a con igu a ion ile on s a up. These es s ensu e ha
his con igu a ion adhe es o legal guidelines and in case o a mal o med o missing
con igu a ion he e a e de aul allback alues ha wo k co ec ly.
Size es s
40 Chap e 4. Tool De elopmen
A se ies o images we e c ea ed o ailing and passing cases o a a ie y o on s o
he de aul esolu ions he ool suppo s (720p, 1080p and 4k). The ypes o on s
es ed include se i , sans-se i , mono-space, and special on s wi h wide and hinne
cha ac e s. An example o a ious on s es ed can be seen in igu e 4.16, he ac ual
es s only ea u e one ype o on a a ime.
Figu e 4.16: Example o a ious on s no passing size es s in 4k esolu ion
4.5.1. Memo y leaks and add ess sani a ion es s
A e passing all o he p e ious es s he applica ion uns hem all again while mon-
i o ing he memo y usage o he ool. This makes su e ha no dynamic memo y has
been le unaccoun ed o when comple ing execu ion and ha i has been eleased, as
well as making su e ha no poin e s we e le poin ing o eed egions o memo y du ing
execu ion.
4.6. Conclusions
In his chap e we ha e co e ed he p ocess o de eloping he TinEye ool. Fi s ,
some p o o ypes we e made o assess he capabili ies o he echnologies chosen o he
de elopmen and he iabili y o he app oach. Ano he objec i e o his p o o yping was
inding ou he le el o accu acy we could expec om he EAST algo i hm, de eloped o
eal li e scena ios, when wo king wi h ideogame use in e aces. The emaining objec i e
o he p ocess was o ind ou a sui able lib a y o ecognizing said ex .
A e he p o o yping p ocess was inished, he ool was de eloped ollowing an i e a i e
p ocess. The eam wo ked in wo week i e a ions ha always ocused on ha ing a wo king
p oduc o he ool. This allowed he ool o ecei e cons an eedback, bo h om ou
es s and om pee s a Elec onic A s, whose needs shaped he way he ool beha ed.
The i s i e a ions o he ool we e ocused on in eg a ing wha we lea ned du ing he
p o o yping p ocess: de ec ing ex wi h he EAST algo i hm, size and con as checks,
and using OpenCV o ex ecogni ion when necessa y o measu emen s.
A e he ool was su icien ly ad anced, a se ies o eal es cases we e un h ough i
o check i s accu acy. The ollowing i e a ions ocused on imp o emen s o said accu acy,
wi h measu es o a oid en i onmen al o alse ex de ec ion, mo e accu a e ex size
4.6. Conclusions 41
measu emen and be e selec ion o meaning ul backg ound pixels o con as checks.
TinEye’s esul ing a chi ec u e was desc ibed, explaining he di e en modules he
p ojec is spli in o and how he di e en classes ela e be ween each o he . Finally,
he es sui e comp ised o di e en uni and accep ance es o ensu e he ool’s co ec
ope a ion as he codebase changes was explained in de ail.

Chap e 5
E alua ion and discussion
This chap e goes o e he p o iling o he ool, analyzing which s ages o he p ocess
ake up mo e execu ion ime. Addi ionally, he s udy o a ious eal wo ld es cases ha
b ough up he exis ence o ce ain edge cases ha bea conside a ion when a emp ing o
au oma e he en i e p ocess.
A e wa ds, he implica ion o his e alua ion, he accu acy o he di e en analysis
he ool does, and he impac and imp o emen s o e he cu en accessibili y alida ion
me hodology he indus y uses will be discussed.
5.1. P o iling
The co e me ic o TinEye, o he han analysis co ec ness, is execu ion ime, so his
would be he ocus o he p o iling p ocess. The main objec i e o he p o iling p ocess was
o see which unc ions and s ages o he p og am we e aking he mos ime.
The i s app oach was he use o he in eg a ed p o ile in Visual S udio 2022, he IDE
o choice o he de elopmen . The in eg a ed p o ile would allow o see a call g aph wi h
in o ma ion o execu ion imes, and pe cen ages o o al un ime aking in o accoun he
ime aken by a unc ion and i s called unc ions. The esul s om his ool showed ha
mos o he execu ion ime was spen on ex e nal code implemen ed by OpenCV.
The i s un o he Visual S udio p o ile p o ed o be almos oo p ecise o ou
needs. Wha we wan ed o s udy was he p opo ional ime aken o execu e each s ep
o he analysis p ocess: Media loading, luminance map gene a ion, ex de ec ion, ex
ecogni ion, mask calcula ion, ex measu emen , con as a io calcula ion, and esul
gene a ion.
Fo his eason we op ed o a ligh weigh heade w i en by Che niko (2019). I was
chosen o i s ease o use, simple esul s o analyze a e wa d, easy se up and c oss-pla o m
compa ibili y. This simple lib a y uses he class Ins umen a ionTime o measu e he
ime i akes o a unc ion o scope o be execu ed. Once he ime s ops, he da a is
w i en in a json ile compa ible wi h Google’s Ch ome acing iew in ch ome:// acing
ha displays a hie a chy showing he execu ion ime o a ime and he ones used while i
was unning as shown in igu e 5.1
43
44 Chap e 5. E alua ion and discussion
Figu e 5.1: Example o he ch ome:// acing iewe
Once he p o iling ool was chosen, i was de e mined ha he code egions o p o ile
we e:
Ini ializa ion: Load he ool’s lib a ies and neu al ne wo ks in o memo y.
Luminance map calcula ion: Load an image and calcula e and s o e i s luminance
map in o ma ion.
Tex box de ec ion: Run he cu en image h ough he EAST algo i hm, modi y
and p une he esul s.
Size measu emen : Tex ecogni ion o ge numbe o cha ac e s and hen ob aining
ex measu emen s.
Con as calcula ion: Mask ob aining and calcula ion o a e age luminance alues
inside mask a ea.
5.1.1. Resul s and conclusions
A e se ing up he p o ile , he ool was un on elease con igu a ion analyzing a se
o images wi h 1080p esolu ion. The esul s a e shown in igu e 5.2.
Figu e 5.2: Windows elease p o iling iew
The majo i y o he execu ion ime is spen ecognizing ex ( on SizeCheck unc ion),
ollowed by de ec ing ex boxes wi h he EAST algo i hm (ge Tex boxes unc ion). Ex-
ecu ion imes o op imiza ions such as axis aligned ex box me ging, luminance map
calcula ions, o il e ing il ed ex boxes a e i ial in compa ison o he ex de ec ion
and ecogni ion. The con as checks, al hough elian on he ex box de ec ion s ep a e
also as .
To imp o e he o e all pe o mance o TinEye, he ollowing measu es we e conside ed:
Tex ecogni ion, while only being used o a e age cha ac e wid h calcula ion, akes
up 80% o execu ion ime. Du ing he es p ocess, we we e unable o ind a on ha
would ail he a e age cha ac e wid h equi emen while mee ing he heigh one.
This op imiza ion has been implemen ed as an expe imen al ea u e, i can make he
ool as e , bu i also has a mino downside: he possibili y o aising a wa ning
when ex was de ec ed bu no ecognized would be los . This wa ning would be
help ul dealing wi h edge cases whe e con as is oo low o ex is oo small o he
ex o be ecognized.
5.2. Real-wo ld-cases es ing 45
When p ocessing a ideo, since ex has o be on sc een o a ce ain amoun o ime
o a human o ead i , a lo o ames con ain he same in o ma ion, mo eso a highe
ame a es. To a oid was ing compu a ion ime on p ac ically iden ical ames, an
algo i hm could be designed o only analyze he mo e ele an ames. P oposals o
his app oach a e explained mo e in-dep h in chap e 6.
5.2. Real-wo ld-cases es ing
A se ies o images and ideos om games like Apex Legends, Knockou Ci y, Ba le ield
2042, NHL, o Need o Speed we e p o ided by a ious depa men s a EA. These pieces
o media we e used o es he ool’s pe o mance, o see i i unc ioned adequa ely and
o ind any possible bugs o unin ended beha io .
In o al, hi y- i e images and wen y-six ideos o ideo game oo age we e analyzed.
As will be de ailed in he ollowing sec ion, he ool li ed up o ou expec a ions o i s pe -
o mance. Known issues, like en i onmen al ex de ec ion o de ec ion o en i onmen al
non- ex ea u es as ex , came up, bu hey we e o be expec ed. These esul s ein-
o ced he design decisions o allowing he use o se an a ea o in e es , o he capaci y
o igno ing a ce ain a ea o he sc een.
In he cases he ool was designed o wo k in, which is accessibili y alida ion in cha
and sub i les ex , i wo ked ou s andingly well. I ma ked exhaus i ely e e y wo d ec-
ognizable in a scene and pe o med size and con as alida ion on each o hem. Va ious
examples o hese beha io can be seen in igu e 5.3.
In o he si ua ions ha we e no planned o when speci ying equi emen s, like in game
menus o ex e nal desk op applica ions, he ool also pe o med well (as seen in igu e 5.4).
I makes sense, since he con ex o he analysis o hese applica ions is simila o analyzing
a piece o oo age o a game. This opens up he doo o possible ex ensions o accessibili y
guidelines, since he echnology o check o i is a ailable he e.
Figu e 5.3: Co ec cases o image p ocessing, sub i les and cha egions in-game.
52 Chap e 6. Conclusions and Fu u e Wo k
would be.
This ool is a signi ican wo k low imp o emen o he indus y and will become e en
mo e ele an in he yea s o come. While ull-au oma ion couldn’ be eached and manual
e ision o he epo s is s ill needed, he h oughpu o a single pe son has been as ly
imp o ed. TinEye, as an assis ed ool is capable o helping a es e much mo e p oduc i e
han wi h a manual app oach. Fu he mo e, e en hough i is mainly aimed a ideogame
accessibili y guidelines alida ion, he lexible and gene ic na u e o he ool makes TinEye
able o wo k wi h any media, like GUI applica ions.
Fu u e wo k will ocus in imp o ing he ools pe o mance o deploymen . I is ex-
pec ed ha TinEye will be deployed as a back-end whe e EA es e s and s udios will upload
he media o be analysed. Feedback om his deploymen will be aken in o accoun , o-
cusing in p o iding he use s all he in o ma ion and ea u es hey need.
6.1. Fu u e wo k
TinEye can be ma ked as a success ul p ojec o he objec i es i se ou o accomplish,
bu he e a e a numbe o imp o emen s ha could enhance and acili a e he accessibili y
alida ion p ocess:
Imp o e ideo p ocessing speed by selec ing speci ic ames. Execu ion speed could
be inc eased by only selec ing speci ic ames when analyzing a ideo. Video games,
specially shoo e s o o he as -paced gen es, ha e e y ame a es, a high numbe o
ames a e being gene a ed e e y second. This means ha he con en o sequen ial
ames may be he same, mo eso when alking abou ex which humans ake ime
o ead. This means ha a lo o p ocessing ime is being spen on analyzing e y
simila ames.
An algo i hm can be de eloped ha de ec s when a new ame o be p ocessed is su -
icien ly di e en om he las one o be analyzed, i so i would un he accessibili y
checks on his ame and i no i would skip i .
Implemen a g aphical in e ace o ocus and igno e egions c ea ion. The way ha
ocus egions a e cu en ly de ined in he con igu a ion is by speci ying he posi ion
and size o he ec angles ela i e o he sc een size. Fo example, i he use wan ed
o de ine a ocus egion o he op le qua e o he sc een hey would ha e o
s a e hey wan ed a egion in he X dimension om 0 o 0.5 and likewise o he Y
dimension.
A g aphical in e ace would make his p ocess a lo mo e use iendly. A e upload-
ing he media, a so o p e iew o he image could be shown and he use would
d aw he ocus and igno e egions hey wan ed on op o his p e iew. This would
la e be ansla ed o ela i e posi ions and inpu in o he ool so i can unc ion as
no mal. This way use s and es e s do no ha e o wo y abou how he unde lying
pa sing o he ec angle de ini ions wo ks.
T ain a speci ic neu al ne wo k o ex de ec ion in a game scene. EAST was c ea ed
o de ec all kinds o ex in eal wo ld scena ios, i expec s o ind ins ances o ex
in skewed and some imes obscu ed si ua ions. This is e y p ac ical o a sys em
like a sel d i ing ca , bu ou ool’s use case is di e en . Some o he ’acciden al’
en i onmen al de ec ions could be a ibu ed o his ac .

6.1. Fu u e wo k 53
Using he same algo i hm bu aining a new neu al ne wo k wi h a lo o aining
examples om di e en ideo game use in e aces can p o e e y bene icial in making
TinEye mo e accu a e and au oma ing i .
C ea e a eal ime o e lay o es e s. I su icien pe o mance is achie ed, an o e lay
in e ace o injec in o games could be c ea ed. This o e lay would un he acces-
sibili y guidelines checks in he backg ound and ou pu in nea - eal- ime i he ex
ha is cu en ly p esen on sc een passes he guidelines’ equi emen s.
This app oach could be use ul while es ing new in e ace con igu a ions, since a
keeping ack o e e y ex box esul while playing o ganically could be di icul
o a human and ha ing o eco d he session and seeing he esul s a e could be
i esome as well.
Bibliog aphy
G ee e e y mo ning wi h a smile. Tha way i
won’ know wha you’ e planning o do o i .
The Alloy o Law
Ashley Wood. The impo ance o colo blind accessibili y in games. 2017. [Online;
accessed 5-Ap il-2022].
BenRG and cmglee. Compa ison o some gb and cmyk colou gamu on a cie 1931 xy
ch oma ici y diag am. 2014. [Online; CC BY-SA 3.0].
Che niko , Y. Basic ins umen a ion p o ile . 2019. h ps://gis .gi hub.com/
TheChe no/31 135eea6ee729ab5 26a6908eb3a5e# ile-ins umen o -h; Las ac-
cessed 28-03-2022.
Communi y, O. Openc eas algo i hm u o ial. 2020. h ps://gi hub.com/openc /
openc /blob/mas e /samples/dnn/ ex _de ec ion.cpp , las accessed 10-2-2022.
Communi y, T. Tesse ac open sou ce eposi o y. 2022.
FCC. 21s cen u y communica ions and ideo accessibili y ac (c aa). 2010. [Online;
accessed 6-Ap il-2022].
F ed ick, D. R. Myopia. BMJ, Vol. 324(7347), 1195–1199, 2002. ISSN 0959-8138.
HDhes. Hd size and dis ance calcula ions. 2014. [Online; a chi ed om o iginal].
Joos en, E.,Lank eld, G. . and Sp onck, P. Colo s and emo ions in ideo games.
61–65, 2010.
Ma in Fowle . Con inuous in eg a ion. 2006. [Online; accessed 27-Ap il-2022].
Melillo, P.,Riccio, D.,Pe na, L.,Sanni i di Baja, G.,Nino, M.,Rossi, S.,
Tes a, F.,Simonelli, F. and F ucci, M. Wea able imp o ed ision sys em o colo
ision de iciency co ec ion. IEEE Jou nal o T ansla ional Enginee ing in Heal h and
Medicine, Vol. 5, 1–7, 2017.
Mic oso . Xbox accessibili y guideline. 2021. [Online; accessed 5-Ap il-2022].
Mi he, R.,Indalka , S. and Di eka , N. Op ical cha ac e ecogni ion. In e na ional
jou nal o ecen echnology and enginee ing (IJRTE), Vol. 2(1), 72–75, 2013.
55
56 BIBLIOGRAPHY
MozillaCo po a ion. Web accessibili y: Unde s anding colo s and lumi-
nance. 2022. h ps://de elope .mozilla.o g/en-US/docs/Web/Accessibili y/
Unde s anding_Colo s_and_Luminance", las accessed 15-03-2022.
Giudi a de p a o, C. F. . J.-P. S. Inno a ions in he ideo game indus y: Changing
global ma ke s. Communica ions & S a egies, 2014.
Sambasi a ao, K. Non-maximum supp ession (nms) a echnique o il e he
p edic ions o objec de ec o s. 2019. h ps:// owa dsda ascience.com/
non-maximum-supp ession-nms-93ce178e177c; Las accessed 4-4-2022.
Sha pe, K. R. G. . L. T. Colo Vision, F om Genes o Pe cep ion. 1999.
Sik-Lányi, C. 22 - choosing e ec i e colou s o websi es. 600–621, 2012.
Smi h, R. An o e iew o he esse ac oc engine. In Nin h in e na ional con e ence on
documen analysis and ecogni ion (ICDAR 2007), Vol. 2, 629–633. IEEE, 2007.
Snook.ca. Snook.ca colou con as check. 2015.
H ps://snook.ca/ echnical/colou con as /colou .h ml g = 33FF33, bg =
333333, las accessed07/02/2022.
Tanaka, G.,Sue ake, N. and Uchino, E. Ligh ness modi ica ion o colo image o
p o anopia and deu e anopia. Op ical e iew, Vol. 17(1), 14–23, 2010.
Tom Wijman. The games ma ke and beyond in 2021: The yea in numbe s. 2021. [Online;
accessed 5-Ap il-2022].
Tycho Henzen. Wha is he poin o he colou blind il e s in some games? 2021. [Online;
accessed 5-Ap il-2022].
W3C. How o mee wcag, guideline 1.4 – dis inguishable. 2008a. [Online; accessed 6-Ap il-
2022].
W3C. Web con en accessibili y guidelines (wcag) 2.0. 2008b. [Online; accessed 6-Ap il-2022].
W3C. W3c accessibili y guidelines (wcag) 3.0. 2021. [Online; accessed 6-Ap il-2022].
Wu, P.-C.,Huang, H.-M.,Yu, H.-J.,Fang, P.-C. and Chen, C.-T. Epidemiology o
myopia. Asia-Paci ic Jou nal o Oph halmology, Vol. 5(6), 386–393, 2016.
Youse i, J. Image bina iza ion using o su h esholding algo i hm. 2015.
Zhou, X.,Yao, C.,Wen, H.,Wang, Y.,Zhou, S.,He, W. and Liang, J. Eas :
An e icien and accu a e scene ex de ec o . In P oceedings o he IEEE Con e ence on
Compu e Vision and Pa e n Recogni ion (CVPR). 2017.
Zhou, X.,Yao, C.,Wen, H.,Wang, Y.,Zhou, S.,He, W. and Liang, J. Eas
p e ained model. 2018. h ps://www.d opbox.com/s/ 2ingd0l3z 8hxs/ ozen_eas _
ex _de ec ion. a .gz?dl=1 Las accessed 2022-02-20 .
Appendix A
Indi idual Con ibu ions
A.1. Ad ián Ál a ez
Du ing he beginning o he p ojec ou side EA, we had limi ed in o ma ion abou
he cu en p ocess. We knew ha i would be equi ed o emula e a humans capaci y
o ind ex , measu e con as and measu ing ex . My wo k ocused, on esea ching he
echnologies and ools a ailable ega ding ex ecogni ion and con as checking, as well
as possible lib a ies o image and ideo manipula ion like OpenCV. The esul s o his
i s con ac mean ha ing knowledge o he key echnologies we we e going o depend on
du ing he de elopmen o he ool. Du ing his p io esea ch, I began o wo k on he s a e
o he a chap e o his memo y, aking no e o why a pa icula echnolgy is being con-
side ed and he possible limi a ions o pi alls ha could be ound du ing he de elopmen .
Once he wo k inside Elec onic A s began, I spen some ime lea ning he basics
o CMake because one o he company’s equi emen s o he ool was ha i should be
c oss-pla o m. This p ocess was slow in he s a , he i s p o o ypes we e build as sim-
ple execu ables. Since he ool would need o be a lib a y in o de o i o be used as
a backend o a u u e web applica ion. Once de elopmen mo ed away om p o o ypes,
he p ojec was buil as a s a ic lib a y and an execu able ha linked i . Las ly, ega ding
p ojec con igu a ion, I esea ched how o gene a e a es p ojec wi h Google Tes in o de
o de elop and un es s locally. Wi h some indica ions om company pee s, I looked in o
cpkg as a cpp dependency manage and used i o link he ele an dependencies such as
a json pa se o he con igu a ion iles, a logge , uni es ing and OpenCV.
Du ing he p o o yping p ojec my main ocus, o he ha p ojec building and con ig-
u a ion, was he EAST p o o ype, ollowing he documen a ion and es ing i s capabili ies
wi h ideogames. Rega ding he con as p o o ype I made su e ha he o mulas dis-
closed by Mozilla we e he sames ha we e being used by he web cu en ly being used
in he manual wo k low wi h Snook.ca (2015) and s a ed amilia ising wi h OpenCV o
wo king wi h.
Du ing he i s i e a ion o he inal ool, I s a ed ocusing on a chi ec u e, de e min-
ing how classes would be exposed and wha esponsibili ies would be le ou o he inal
use o he lib a y. Fo example, c ea ing a Media ac o y ha allows use s o he lib a y
o load he media when hey de e mine. I implemen ed a basic con igu a ion class ha
57

58 Appendix A. Indi idual Con ibu ions
would be expanded as new ea u es we e de eloped and in eg a ed he EAST p o o ype
ha would de ec ex o he ollowing s ages. To keep bo h o he eam membe s in up o
da e, i was common p ac ice o w i e es s o he o he ’s implemen a ions. I w o e some
basic es s o size measu emen s and con as calcula ions. The con as calcula ion es s
allowed us o ind a p ecision e o in he calcula ions e en when using doubles. Es eban
la e ixed his issue by using look-up ables.
Du ing he imp o emen i e a ions o he ool, I esea ched di e en p o iling ools and
ins umen alised he code o see how execu ion ime was spen . This ask p omp ed me
o look in o con inous in eg a ion asks. While he CI/CD pipeline had been p o ided
by pee s, I ealised ha i would be bene i ial o un his p o iling inside he pipeline. I
looked in o he al eady p esen implemen a ion o au oma ically unning es s and de ised
a simila job, whe e each ime he sou ce code om he lib a y changed, TinEye would be
un gene a ing a downloadable a i ac wi h he p o iling da a.
The p o iling ga e key in o ma ion o he u u e o he ool de elopmen like he pos-
sibili y o a oiding ex ecogni ion o inc ease speed. Ano he implemen ed op imisa ions
we e he il e ing o il ed ex boxes, which was p oposed by Es eban, and he me ging o
aligned ex boxes aking in o accoun ha all ex would be aligned wi h he UI, dec eas-
ing he o al numbe o ex boxes.
Finally, in pa alell o he op imiza ions, I wo ked wi h Es eban analysing he images
ha made up he inal es s. I con ibu ed o he deduc ion p ocess o his es s, poin ing
ou he main imp o emen s ha his ool is able o p o ide in compa ison o he cu en
wo k low.
Du ing he whole p ojec I made su e o ollow good agile p ac ices: keeping an open
communica ion wi h Es eban bo h o p io i ise and es ima e cu en wo k. I paid close
a en ion in weekly mee ing wi h he es o he membe s o he eam o ind ou wha
unc ionalies could make hei wo k easie . This weekly mee ings also helped he es o
he eam o see he cu en s a e o TinEye, and I answe ed wi h all he de ails needed
o hem o plan a ound TinEye’s capabili ies. I e iewed each o Es eban’s commi s, o
be e comp ehend he code ha was going in o he ool and ha I would need o wo k
wi h la e .
A.2. Es eban Res epo 59
A.2. Es eban Res epo
In he in es iga ion phase, p io o s a ing ou wo k a Elec onic A s and he p o-
o ype de elopmen s age, I mainly ocused on in es iga ing di e en indus y s anda ds
o con as calcula ion and compliance equi emen s, and how di e en s anda ds di e ed
om each o he . I also ocused on inding ou in o ma ion on di e en OCR lib a ies and
hei possibili ies, inally deciding on using Tesse ac o ou i s cha ac e ecogni ion
es s.
Du ing he p opo ype c ea ion s age I c ea ed he basic con as calcula ion p o o ype
and he Tesse ac OCR p o o ype. The con as calcula ion p o o ype aimed o emula e
he beha io o he websi e ha EA’s es e s cu en ly use o check con as be ween wo
colo s, bu ou ool ecei ed an image and wo posi ions ins ead o wo colo s.
Du ing his p ocess I amilia ized mysel wi h he Tesse ac API o make be e use o
i la e down he line, al hough Tesse ac would end up ge ing sc apped in ou p ojec . I
also c ea ed a ious es cases o he con as ool o make su e ha we we e ge ing he
alues we expec ed o ge . These es cases ac ually ga e use he i s hin s ha Tesse ac
would no i ou needs, since g adien backg ounds wi h colo ed ex would gi e lacklus e
esul s when an h ough he OCR p o o ype.
Once de elopmen s a ed a EA I ocused on how o c ea e he luminance map o
he images inpu in o he ool. I used OpenCV’s ma ix ans o ma ions o ensu e ha
all p ocessed images had h ee colo channels, linea ized all o hem and mul iplied each
linea ized channel using he luminance o mula o ob ain he inal luminance alue o a
pixel. Finally, hese alues would be added up and encoded in a single channel ma ix ha
ep esen ed he luminance map.
A his s age I also wo ked on build au oma ion, u ilizing CMake di ec i es o ecognize
when new esou ce iles had been added o he p ojec and copy hem au oma ically o he
build olde when he p ojec was compiled.
A i s , all o he image in o ma ion was inside he TinEye class alongside all o he
checks and de ec ion and ecogni ion me hods, making i e y unwieldy o wo k on, so as
he p ojec g ew I abs ac ed he image in o ma ion and speci ic me hods in o he Image
class (which would la e u n be abs ac ed in he media class a e ideo suppo was
added) and documen ed ou wo k h oughou .
A e wa ds, once he uni es amewo k was se up, I c ea ed a ious es s o size
and con as cases o make su e ha he ongoing changes in ou code would no b eak
any hing ha was al eady wo king. The size es s consis ed in passing and ailing images
each wi h ex o one o i e on ypes (se i , sans-se i , monospace, a bold on and an
ul a- hin on ) o each o he esolu ions we de eloped he ool o (720p, 1080p and 4k).
The con as ex cases consis ed o a simple wo d on op o di e en backg ounds. The
backg ounds could be ei he a solid colo , a g adien , o s ipes o di e en colo s. wo
cases we e made o each ype o backg ound, a passing one and a ailing one, as well as
high and low con as es cases.
I c ea ed he Tex box class and used he now in eg a ed EAST p o o ype o ou pu a
60 Appendix A. Indi idual Con ibu ions
lis o egula ec angles ha ep esen egions in an image whe e ex is ound. We could
use hese boxes o ex measu emen and la e o highligh he ou pu o passes and ails
on o an image by colo ing he boxes a ce ain colo depending on he esul s.
A e Tesse ac p o ed o be ine ec i e o ou use case, I esea ched and la e imple-
men ed he al e na i e, OpenCV’s ex ecogni ion solu ion. I made su e o in eg a e i
well wi h ou exis ing codebase and he be e esul s om OpenCV we e e iden imme-
dia ely.
I implemen ed he op ion o p ocess ideo. P ocessing ideo p o ed o be icky in a
ce ain way: When p ocessing images wi h OpenCV, depending on he ile ype, he p o-
g am can di ec ly impo hem as ma ices wi h speci ic p ope ies, bu when g abbing
ideo ames you ha e o manually con e hem o you desi ed ma ix ype. I had o
implemen ce ain checks du ing loading o a ideo ame o ensu e co ec con e sion o
he ame so he ool could un all o he subsequen es s and checks.
La e on I wo ked he possibili y o calcula ing he luminance his og am o a egion o
imp o e he accu acy o ou backg ound con as calcula ions, using hem o ind ou he
mo e signi ican alues ep esen ing he colo s o he ex and i s backg ound. This op ion
ended up no panning ou since an ialising and o he noise in he image made his og am
p ocessing no eally s aigh o wa d o wo k wi h.
To imp o e ou con as calcula ions I implemen ed he h eshold and masking ope -
a ions. I esea ched o op imal way o au oma ically calcula e he h eshold o di ide an
image in o egions wi h ex and egions wi h backg ound, and how o use he esul ing
mask o c ea e an ou line o he ex and ge a mo e ep esen a i e esul o he ex ’s
backg ound. This ou line ep esen s he backg ound ha jus su ounds he ex ins ead
o he backg ound o he whole ex box as we had been doing up un il ha poin . Finally,
I c ea ed a ba e y o accep ance es s o ensu e ha ou ool co ec ly passed o ailed
a ious on ypes in all o he suppo ed esolu ions and uni es s o ou execu i e class
TinEye o ensu e co ec ness o he whole p ocess.
O e all, du ing he whole p ocess I wo ked closely wi h Ad ian so we would always
know he ad ances each one was making on he ool and we always had inpu on he
design decisions o wha was going on du ing de elopmen . I also a ended a ious weekly
mee ings wi h di e en depa men s a EA o upda e hem on he p og ess o he ool and
ge eedback on he ea u es hey would wan TinEye o ha e.
The pu pose o a s o y elle is no o ell you
how o hink, bu o gi e you ques ions
o hink upon
Wi
The way o kings
B andon Sande son