Herramienta adaptativa para la creación de tests automáticos de interfaces de videojuegos

He amien a adap a i a pa a la c eación de es s au omá icos de in e aces de ideojuegos Adap a i e ool o au oma ic es c ea ion o ideo game in e aces T abajo de Fin de G ado Cu so 2022–2023 Au o C is ian Rene Cas illo de León Iago Quin as Diz Di ec o Guille mo Jiménez Díaz Colabo ado Alessand o Len ini G ado en Desa ollo de Videojuegos Facul ad de In o má ica Uni e sidad Complu ense de Mad id He amien a adap a i a pa a la c eación de es s au omá icos de in e aces de ideojuegos Adap a i e ool o au oma ic es c ea ion o ideo game in e aces T abajo de Fin de G ado en Desa ollo de Videojuegos Depa amen o de So wa e e In eligencia A i icial Au o C is ian Rene Cas illo de León Iago Quin as Diz Di ec o Guille mo Jiménez Díaz Colabo ado Alessand o Len ini Con oca o ia: Sep iemb e 2023 Cali icación: G ado en Desa ollo de Videojuegos Facul ad de In o má ica Uni e sidad Complu ense de Mad id 15 de Sep iemb e de 2023 Dedica o ia A mi amilia, po apoya me y ayuda me en odo lo posible, no lo hab ía conseguido sin oso os. A mi mad e, po da me odo lo que necesi aba pa a cumpli mis sueños. Y a Raquel, que has sido mi a o du an e an os años, po escucha me y consegui saca me una son isa odos los días. C is A Aa on, po habe es ado siemp e ahí a lo la go de es os años y po se un g an amigo. A mis pad es, po habe con iado en mí y da lo odo pa a que hoy es e aqui. A mi he mano, po habe sido siemp e mi mejo amigo e inspi a me a se mejo . Y a mi he mana, que siemp e consigue saca me una son isa. Iago Ag adecimien os G acias a odo el equipo po habe nos dado odo lo que necesi ábamos. G acias a Alessand o y Angela po sus ideas y seguimien o. A Claudio po mo i a nos a abaja odos los días a ando siemp e de apo a nos ideas y p eocupándose po noso os. A Blanca y a Rica do po se nues os e e en es y es a siemp e disponibles cuando les hemos necesi ado. A F an, po ayuda nos a adap a nos al en o no y es a ahí pa a apoya nos. A la gen e que aunque no abajase con noso os di ec amen e, se in e esó po el abajo que haciamos. Y inalmen e a Guille, po habe se p eocupado an o po el p oyec o y habe nos dado la opo unidad de accede a es as p ác icas. A odo el mundo que con ió en el p oyec o o dedicó pa e de su iempo en segui nues o p og eso, g acias. ii Resumen He amien a adap a i a pa a la c eación de es s au omá icos de in e aces de ideojuegos El es ing es una pa e undamen al du an e el desa ollo de un p og ama o aplicación de so wa e. Es e p oceso consis e en e i ica y alida las dis in as uncionalidades de un p og ama y de es a o ma consegui un p oduc o de calidad. Sin dicha pa e la mayo ía de las aplicaciones se ían inu ilizables debido a la can idad de e o es que expe imen a ía un usua io. Sin emba go, el es ing equie e de mucho iempo y dine o pa a log a unos esul ados signi ica i os, lle ando a las emp esas de ideojuegos a con a con un g an núme o de es e s que comp ueban manualmen e el uncionamien o de sus p oduc os. En el caso de in e aces de ideojuegos, las nue as ecnologías y he amien as de au oma ización pueden se ú iles pa a educi es e abajo manual y dedica lo a p uebas únicamen e ealizables po humanos. Usando ap endizaje au omá ico y p ep ocesado de imágenes, se ha hecho una he a- mien a capaz de e ique a de o ma gene al elemen os de una in e az de ideojuegos apli- cable a múl iples í ulos pa a la ealización de p uebas de na egación au omá icas. Palab as cla e au oma ización, p ep ocesado, clasi icación, de ección, in e az, ap endizaje, ideojuegos, es ing, bo des, adap able ix Lis o igu es 1.1. Technical deb inc eases o e p oduc ion ime . . . . . . . . . . . . . . . . . 1 2.1. Example o igu e-g ound p inciple (Todo o ic, 2008). . . . . . . . . . . . . 6 2.2. Example o p oximi y p inciple (Todo o ic, 2008). . . . . . . . . . . . . . . . 6 2.3. Example o simila i y p inciple (Todo o ic, 2008). . . . . . . . . . . . . . . . 7 2.4. Example o closu e p inciple (Todo o ic, 2008). . . . . . . . . . . . . . . . . 7 2.5. Example o con inui y p inciple (Todo o ic, 2008). . . . . . . . . . . . . . . 7 2.6. Compa ison be ween a s anda d HUD and a diege ic HUD . . . . . . . . . . 8 2.7. Example o es p og ammed on Selenium IDE . . . . . . . . . . . . . . . . 10 2.8. Example o es p og ammed on Sikuli IDE . . . . . . . . . . . . . . . . . . 12 2.9. Simpli ica ion done by OCRs om ex o bi images (Smi h, 1987). . . . . 13 3.1. Cogi opipeline .................................. 18 3.2. F1®22 mul iplaye menu example . . . . . . . . . . . . . . . . . . . . . . . 19 3.3. Image P ocessing pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 3.4. F1®22 menu be o e and a e bina y h eshold . . . . . . . . . . . . . . . . 20 3.5. F1®22 adap ed con as and b igh ness esul . . . . . . . . . . . . . . . . 20 3.6. Sobel in ensi y g aphic (B adski and Kaehle , 2008) . . . . . . . . . . . . . 22 3.7. Second de i a i e o an edge (B adski and Kaehle , 2008) . . . . . . . . . . 22 3.8. Gaussian il e applica ion on con as ed image, he en i e image is blu ed in o de o ob ain he mos p ominen shapes in he image. . . . . . . . . . 23 3.9. Median il e applica ion on con as ed image, a minimal blu o educe he noise in he image is applied. This modi ica ion o he image esul s in smoo hed pa s, such as he ace o he igu e. . . . . . . . . . . . . . . . . 24 x ii 3.10. Bila e al il e applica ion on con as ed image. This il e helps smoo hing ou he noise in speci ic pa s while p ese ing edges such as he backg ound and bu ons in his igu e. . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 3.11. One-dimensional edge p o iles. (Jain e al., 1995) . . . . . . . . . . . . . . . 25 3.12. HSV Mul i-Channel Canny esul . . . . . . . . . . . . . . . . . . . . . . . . 26 3.13. Compa ison o Canny applied o a Non-blu ed and Gaussian il e F1®21 image. Con ou s and edges om inside each de ec ed elemen a e mode - a ely educed on he igh a e he il e is applied. . . . . . . . . . . . . . 27 3.14. Compa ison o Canny applied o a Non-blu ed and Median il e F1®20 image. The noise is sligh ly educed o he ool o co ec ly de ec he elemen son hesc een. ............................. 27 3.15. Compa ison o Canny applied o a Non-blu ed and Bila e al il e FIFA 23 image. Mos o he noise om he backg ound is emo ed. . . . . . . . . . 28 3.16. Fil e ed con ou s o a F1®2020image..................... 28 3.17. Bounding boxes o a F1®2020image ..................... 29 3.18. Tex de ec ed on a STAR WARS Jedi: Su i o ™menu ........... 29 3.19. Icons de ec ed by machine lea ning model on a FIFA 23 se ings menu . . . 30 3.20. Tex de ec ion and icon de ec ion pe o med on a FIFA 23 menu. Le being he o iginal image and igh being a isual ep esen a ion o each kind o elemen de ec ed. ................................ 30 3.21. Exi and Se ings label dic iona y examples . . . . . . . . . . . . . . . . . . 31 3.22. Classi ied ex s on a STAR WARS Jedi: Su i o ™menu. The g een boxes ou line he de ec ed ex s and display he label classi ied on op o hem. . . 32 3.23. Classi ied icons on a FIFA 23 se ings menu. The g een boxes ou line he de ec ed icons and display he classi ied label and con idence on op o hem. 32 3.24. Ho izon al checke o menu ba de ec ion on a F1®2021 menu . . . . . . . 34 3.25. Example esul o all he inal in o ma ion ga he ed by he ool. . . . . . . . 35 4.1. Tesse ac ex de ec ion p o o ype simple example . . . . . . . . . . . . . . 37 4.2. Tesse ac ex de ec ion p o o ype FIFA 23 menu example . . . . . . . . . . 38 4.3. Sikuli es example................................ 39 4.4. Example o Rekogni ion ex de ec ion. . . . . . . . . . . . . . . . . . . . . . 40 4.5. SageMake i s bu on de ec ion model . . . . . . . . . . . . . . . . . . . . 41 4.6. Compa ison be ween AWS ex de ec ion se ices on Madden. . . . . . . . . 42 4.7. Missing con ou s in a FIFA 23 se ings menu . . . . . . . . . . . . . . . . . 43 4.8. A i icial con ou s o missing con ou s de ec ed . . . . . . . . . . . . . . . . 43 4.9. Example o gene ic icon’s da ase and i s use on FIFA 23 . . . . . . . . . . . 44 4.10. Example o sc eensho s o di e en menus om FIFA 23 . . . . . . . . . . . 46 4.11. Combined dic iona y esul example. This example shows he ype o ele- men , he ex i con ains wi h i s posi ion and dimensions as well as he posi ion and dimensions o i s con ou , and inally i s labels. . . . . . . . . . 47 4.12. Cogi o UML a chi ec u e diag am . . . . . . . . . . . . . . . . . . . . . . . . 49 4.13. Labels.json example con aining wo labels. . . . . . . . . . . . . . . . . . . . 52 4.14. Appse ings.json example. I con ains he game o be es ed, whe he lambda will be used o p ocessing o locally and whe he debug images willbegene a ed. ................................ 53 5.1. Con usion ma ix o di e en s eps o he ool h ough F1®2022 sc eensho s 57 5.2. Table o imes o each s ep o he ool applied o FIFA 2023 sc eensho s (in seconds)...................................... 61 5.3. Table o imes o each s ep o he ool applied o F1®2022 sc eensho s (in seconds)...................................... 62 5.4. G aph o imes o each s ep o he ool applied o FIFA 2023 sc eensho s (in seconds)...................................... 63 5.5. G aph o imes o each s ep o he ool applied o F1®2022 sc eensho s (in seconds)...................................... 63 5.6. Boxplo o imes o each s ep o he ool applied o FIFA 2023 sc eensho s (inseconds).................................... 63 5.7. Boxplo o imes o each s ep o he ool applied o F1®2022 sc eensho s (inseconds).................................... 64 Lis o ables xxi Chap e 1 In oduc ion “Cogi o e go sum” — René Desca es 1.1. Mo i a ion Manual es ing o ideo games can be a ime-consuming and cos ly p ocess, as i equi es es e s o play games epea edly o iden i y and epo any bugs o issues ha may a ise. The la ge scale o hese p ojec s make i a e y complex ask o de ec all hese issues. Fo ha eason, he cos o he echnical deb should be minimized (Kei h, 2015) in o de o a oid hese inc eased sol ing cos s ha accumula e be ween all de elopmen s ages. Figu e 1.1: Technical deb inc eases o e p oduc ion ime As Figu e 1.1 shows, he mo e ad anced he de elopmen s ages a e, he mo e cos i will be equi ed o sol e issues wi h he code. Fixing hese issues on ea lie s ages can ake e y li le ime once hey a e ound, Howe e , i hey a e no ixed, he cos o sol e hese bugs g ows exponen ially. Ad ancemen s in echnology ha e led o he de elopmen o e en mo e sophis ica ed au oma ed es ing ools, such as AI-powe ed es ing, which can help u he educe he cos s and ime associa ed wi h manual es ing. Di e en AI app oaches such as na u al p ocessing language o neu al ne wo ks a e in e es ing echnologies ha could be eally use ul i in eg a ed wi h an ac ual es ing wo k low, making es ing mo e e icien . 1 2Chap e 1. In oduc ion While au oma ed es ing can be a mo e cos -e ec i e and e icien op ion, i is impo - an o no e ha hese es s s ill equi e human o e sigh . Howe e , he goal is o us humans o do he inal asse ions o es s so hey can s ill be eliable, while he compu e does he es o he edious wo k, he na iga ion and all he back end implemen a ion on how o unde s and wha is happening on a speci ic momen . O e all, he use o au oma ion in ideo game es ing is becoming inc easingly popula , as i can help educe cos s and imp o e he e iciency o he es ing p ocess. Howe e , i is impo an o ha e a balance be ween au oma ed and manual es ing, as bo h ha e hei ad an ages and limi a ions. One speci ic a ea whe e es ing is needed is in e aces. I can highly bene i o au oma ic es ing due o he na u e o he es s: doing an speci ic se o ac ions o a i e o ce ain menus, check bu on unc ionali ies o e en o check i a menu has no c i ical bugs ha make he ideo game unplayable. Usually, g aphic use in e aces, o GUIs, a e designed wi h usabili y p inciples on mind. Fo his eason, i a ela ionship could be es ablished be ween hese p inciples and he es ing o wha is seen on he sc een, i is possible ha a be e unde s anding o wha is happening on he es s would be ob ained. 1.2. Objec i es The main goal o his in es iga ion is con ibu ing o he de elopmen o Cogi o, a ool capable o pe o ming au oma ic es ing o in e aces on ideo games by in oducing human like ac ions. An example could be he "Go o he Se ings Menu" ac ion, Cogi o will be able o iden i y he di e en use in e ace elemen s o he ideo game and ind he speci ic elemen s o each ha menu. The ool will iden i y he di e en elemen s using a combina ion o ex , con ou s, edges and icon de ec ion and hen i will classi y hose elemen s in di e en labels using ex and image classi ica ion. This main goal is di ided in o h ee sub-goals o be comple ed in o de : 1. Resea ch abou al eady exis ing ools and possible app oaches. Some in es iga ion will be made abou al e na i e es ing so wa e: Wha unc ionali ies do hey p o ide and how a e hose implemen ed. 2. Design and de elopmen o he ool. 3. E alua ing he e ec i eness and pe o mance o he ool. The wo k will be mainly ocused on he pa ela ed o analyze he in o ma ion on he sc een. Gi en a sc eensho o he cu en s a e o a ideo game, he ool should de ec all impo an de ec ed elemen s, being hose: Classi ying each bu on o he in e ace in o a meaning ul label abou i s ole. De ec selec ed bu ons, i any. Re u n he g ouped elemen s like: g ids, menu ba s and lis s. 1.3. Wo k plan 3 One o he mos impo an poin s on he de elopmen o he ool will be i s adap abili y, o be able o wo k in di e en ypes o games and in e aces wi h he lowes possible main enance. Since mos o he exis ing ools equi e a lo o main enance o a e buil inside he game i sel , his solu ion will wo k ou side o any engine o game. The e a e some limi a ions which de ine wha he p ojec will ocus on: A ully ocused machine lea ning solu ion is no necessa y. Some s eps o he p ojec can be achie ed h ough he use o image p ocessing algo i hms, due o he complex- i y o a ully gene ic machine lea ning elemen de ec o . The in e aces whe e he ool will be es ed will be he subsequen ones o he main menu in e ace. Those in e aces ha a e a pa o he game will no be es ed, as well as diege ic in e aces ha a e a pa o he game’s wo ld. The p ojec will be buil as a ool o imp o e au oma ic in e ace es ing o Elec onic A s and will allow es e s o ocus on pe o ming less edious and epe i i e es s. 1.3. Wo k plan This p ojec has been de eloped based on he SCRUM me hodology. SCRUM is one o he mos widely used me hodologies in he so wa e de elopmen indus y, a p ojec managemen me hod (Su he land, 2015). Following his me hodology, i has been es ab- lished ha he wo k plan will be di ided in o wo-week pe iods called sp in s, in each o he sp in s he eam will mee o assign each o he membe s a numbe o asks, each ask will ha e an es ima ed sco e ha will indica e he complexi y o he ask and a p io i y o indica e which asks should be done be o e o he s. A he end o each sp in , he esol ed asks will be closed and hose s ill in p og ess o o be s a ed will be assigned o he nex sp in , and new ones will be c ea ed. Due o he complexi y and con inuous esea ch o he p ojec , he e will no be many asks in he beginning and mo e will be c ea ed o e ime. Be o e s a ing o wo k wi h he eam o he c ea ion o he ool, he p ojec will ocus on he esea ch o he di e en exis ing ools o he c ea ion o es s and will in es iga e Amazon’s machine lea ning ools. The de elopmen o he ool will s a wi h a pe iod o a ew weeks o esea ch on SageMake and Rekogni ion se ices by c ea ing di e en image ecogni ion models and obse ing he ad an ages and disad an ages o hese models. Once he models ha e been ained and es ed wi h e ined da ase s, a p o o ype o how he ool will wo k will be de eloped, wi hou ye ocusing on na iga ion and ocusing only on he ecogni ion o in e ace elemen s. Based on he esul s o hese image ecogni ion models, he eam will s udy he di e en al e na i es o combina ions o be implemen ed o he inal ool. The eam has al eady de e mined pa o he echnologies ha will be used du ing he de elopmen o he ool, mainly Amazon Web Se ices (AWS) such as SageMake o Rekogni ion o he machine lea ning pa s o he p ojec . Py hon will be used as he de elopmen language, Mic oso Visual S udio Code as he en i onmen and he OpenCV lib a y o image p ocessing. The eason o using Py hon as a p og amming language is because i allows as i e a ion be ween p o o ypes. In addi ion, when closed i will allow 10 Chap e 2. S a e o he a Wai s, needed o he na u e o he pla o m he es s a e being un on. A e doing an ac ion, a delay should be expec ed un il he websi e loads all he equi ed elemen s. Fo his eason, he e a e wo main ways o wai : •Ac i e wai s. Wai ing o a speci ic ime o seconds. No e y eliable due o sho wai s can bo h pass o ail a es , depending on he connec ion o he use . •Wai o an elemen . Mo e e icien on success ul es s, since once he elemen is ound, he es can con inue unning. Asse s. In o de o check i a es has passed success ully, he e a e mul iple in- s uc ions o e i y he es is co ec . Some examples a e i an elemen exis s, i i is isible o i he i le o he websi e con ains speci ic wo ds. Figu e 2.7: Example o es p og ammed on Selenium IDE As Figu e 2.7 shows, Selenium has an in eg a ed de elopmen en i onmen (IDE) ha allows o easily c ea e unc ional es s. The e is a lis o commands like he ones ha ha e been desc ibed be o e, which a e pe o med in o de . 2.2.1.2. GameD i e API GameD i e (Gu ie ez e al., 2018) is a unc ional es ing au oma ion ideo game API designed o au oma e epe i i e asks a playe mus do in a ideo game. I was de eloped speci ically o he Uni y engine and allowed e ec i e communica ion wi h he elemen s o he scenes. The API enables he es e o con ol and in e oga e he game, asking abou i s p ope ies such as he sco e, posi ions o elemen s, cu en scene, and mo e. GameD i e API enables he es e o c ea e a sc ip ha wai s o a speci ic ime o an objec , equi ed o he es , o appea . The sc ip can ake con ol o he keyboa d and mouse wi h speci ic commands and allows he cap u e o images and ideo. This API 2.2. GUI es ing 11 also p o ides me hods o mo ing objec s o cha ac e s o a speci ic loca ion and de ec ing elemen s on he sc een by name and colou . 2.2.1.3. Appium Appium (Cuella , 2012) is an open-sou ce ool ha allows he de elopmen o au o- ma ic unc ional es s in bo h Mobile Web, iOS and And oid applica ions. A single API ha wo ks on bo h pla o ms. Recen ly Appium has also been able o es Windows applica ions. Appium allows he de elopmen o es s on Web Applica ions, as well as Na i e Appli- ca ions on mobile de ices. The main di e ence be ween Appium is he abili y o pe o m au oma ic es s on bo h And oid and iOS. Appium uses an HTTP se e w i en in Node.js ha c ea es and manages mul iple WebD i e sessions on And oid and iOS de ices, WebD i e is an au oma ion amewo k ha wo ks wi h di e en APIs o sending commands o and in e ac ing wi h an applica ion o web. The ypical ope a ion o Appium would be, a clien o es e sends a eques o he Appium se e , he se e will ecei e he eques and handle i by sending i o he de ice o emula o ha will execu e he eques o es cases eques ed by he clien and send he esul s o he Appium se e ha will send i o he clien o obse a ion. I wo ks simila ly o Selenium, an HTTP se e ha lis ens o and sends you eques s. 2.2.2. Visual es ing Visual es ing, also known as isual eg ession es ing, is a ype o au oma ic es ing ha e i ies he isual appea ance o a so wa e applica ion by compa ing sc eensho s o he applica ion be o e and a e changes a e made. Visual es ing ensu es ha he applica ion’s use in e ace emains consis en and isually appealing a e changes a e made. The main ad an age o isual es ing o e unc ional es ing is ha i can de ec isual de ec s ha may be missed by unc ional es s. Fo example, unc ional es ing may no de ec a change in on size o colou ha could a ec he o e all use expe ience. Visual es ing can also help ensu e b and consis ency and compliance wi h design s anda ds. I has one g ea ad an age compa ed o unc ional es ing: i only elies on he isual ou pu o an applica ion, so i can be gene ic. One example o his is Sikuli, which uses compu e ision echniques in eg a ed wi h i s es s. 2.2.2.1. Sikuli Sikuli (Chang and Yeh, 2009) is an open sou ce ool used o pe o m au oma ic isual es ing. I is an in e p e e o mul iple sc ip ing languages as Py hon, used o de ine es s cases. I also suppo s an API o use speci ic isual es ing ac ions, o ins ance, sea ching a empla e image inside ano he sc eensho . To implemen all hese ac ions, he compu e ision lib a y OpenCV is used. As i is shown on Figu e 2.8, es s a e p og ammed on py hon, wi h he addi ion o an API ha allows passing sc eensho s as pa ame e s. The e is an in eg a ion inside o he 12 Chap e 2. S a e o he a Figu e 2.8: Example o es p og ammed on Sikuli IDE ool ha allows o cap u e he egion o in e es o he sc een o speed up p og amming he es s. Jus like all he p e iously s udied ools do in unc ional es ing, i has wai ins uc ions o gi e ime o he p og am o change be ween s a es and load all he esou ces needed be o e nex ins uc ion. This kind o ins uc ions a e expanded wi h he image ecogni ion API, allowing o also wai o an speci ic image o appea . 2.3. Compu e ision echniques Compu e ision echniques in ol e he use o compu e s o iew elemen s o an image in a simila way o how an ac ual human would see hem. Tha way, compu e s can unde s and images, allowing o au oma ize asks ha p e iously only humans could do. The e a e mul iple examples o compu e ision algo i hms, such as edge de ec ion, ea u e ex ac ion, empla e ma ching o his og am compa ison. Some o hese will be explained along his chap e and he ollowing ones (see subsec ion 3.2.3 o mo e de ails). 2.3.1. Op ical cha ac e ecogni ion Op ical cha ac e ecogni ion, o OCR, is he p ocess o con e ing ex s om any kind o image da a o encoded ex ha can be ecognized by machines. I is one o he mos logical s eps o ake ela ed o isual es ing. I is used in a a ie y o si ua ions, such as i s in eg a ion wi h au oma ion ools. One o i s mos common uses is being in eg a ed in au oma ic wo k low p ocesses. AI- powe ed ools o en ely on OCR ei he o ob ain da ase s by p ocessing documen s ela ed 2.3. Compu e ision echniques 13 o one subjec , o o ob aining inpu cases o o he AI models o use hem. W i en ex de ec ion, ei he by a machine o a human, depend on a lo o ac o s. Fon size, colo , con as wi h he backg ound, alignmen o o ien a ion can a ec he whole de ec ion. The exis ence o di e en on s is a e y oublesome ac o , no o men ion how ex is ead on di e en languages. Figu e 2.9: Simpli ica ion done by OCRs om ex o bi images (Smi h, 1987). As Figu e 2.9 shows, each le e is de ec ed (a), hen an skele on o he image is ob ained (b) and inally he skele on is educed o a minimum shape and s o ed as a bi map. The s o y o OCRs has been in cons an de elopmen . When his echnology was i s in es iga ed, only compu e ision echniques we e used. The simpli ica ion seen on Figu e 2.9 was used la e in an algo i hm called ea u e ex ac ion. I consis s ob aining he mos in e es ing poin s o an image in his case, in o de o educe he amoun o da a wi h he models a e ained on. This algo i hm selec s he poin s, labeling hem acco ding o he ollowing cha ac e is ics pe sec ion. Cen e poin , conca e o con ex, dis ance o he cen al poin o leng h a e some o hese cha ac e is ics. In he pas , OCR echnology only depended on compu e ision echniques, which had limi a ions. E en i ea u e ex ac ion is a powe ul ool, adap abili y is no one o i s s eng hs. In cases like handw i en ex , whe e big enough a ia ions can occu be ween di e en cases, i can lead o alse nega i es. Also, he mo e ea u es de ec ed pe le e , he mo e challenging i will be compu a ionally o analyze (Wang e al., 2021). Nowadays e e y s a e-o - he-a OCR is based on machine lea ning. I makes he whole p ocess easie , as well as sa es compu a ional ime and e o s. Howe e , compu e ision is no excluded om he p ocess. In o de o ob ain meaning ul da a om each le e , ea u e ex ac ion is s ill used, bu now is also done by he model. This is wha is known as deep lea ning. Nex , a model is in cha ge o assigning encoded ex le e ins ead o compa ison as i was p e iously done. Rela ed o compu e ision, he e a e wo open sou ce lib a ies ha a e o en used: Tesse ac and OpenCV. Tesse ac is an OCR lib a y while OpenCV includes o he com- pu e ision echniques, s anding ou o i s wide ange o image p ocessing unc ionali ies. 14 Chap e 2. S a e o he a 2.3.2. Tesse ac Tesse ac (Smi h and Hewle -Packa d, 1984) is an open sou ce engine OCR which can be used on di e en lib a ies, depending on he language. I is cu en ly ained wi h mo e han 100 languages and mos o he common on s, and allows he use o neu al ne s o line de ec ion on i s las e sion, Tesse ac 4. I also gi es suppo o he p e ious e sion, which ecognizes cha ac e pa e ns. Fo he legacy engine, i equi es . ainedda a iles, which include he esul o he aining o each language. In addi ion o he 100 languages i is ained on, Tesse ac can be ained on o he languages as well gene a ing ano he . ainedda a as has been done wi h he o iginal suppo ed languages. Tesse ac ex wo ks in unicode (UTF-8). I suppo s PNG, JPEG and TIFF iles as inpu and mul iple ou pu s such as plain ex o PDFs. 2.3.3. OpenCV Open Sou ce Compu e Vision Lib a y, o OpenCV (B adski and Kaehle , 2008), is a powe ul open sou ce compu e ision and machine lea ning lib a y ha implemen s ime e icien ope a ions wi h images. I ac s as a common in as uc u e o machine lea ning and compu e ision algo i hms, including bo h classic algo i hms and s a e-o - he-a ones. OpenCV has been p og ammed using C++ and has bindings wi h Ja a, Py hon and MATLAB, among o he languages. This allows o use e icien in ime algo i hms while keeping he as i e a ion wo k low Py hon allows. OpenCV coun s wi h i s own deep neu al ne wo k module (dnn) ha allows o p epa e da a in o de o use hem o aining machine lea ning models. Besides ha , i also allows o pe o m image con e sion o simpli y da a, ope a ing wi h images in bo h RGB o HSV colo schemes, as well as g ayscales. This is pa icula ly use ul when i comes eed less in o ma ion o models o ob ain be e pe o ming solu ions. Howe e , his simpli ica ion can be also help ul wi h he classic compu e ision algo i hms in some cases, as i happens wi h edge de ec ion algo i hms. 2.4. Image classi ica ion In o de o au oma e isual es ing o ideo game in e aces, i is necessa y o he ool o somehow unde s and he meaning o he elemen s i will de ec on he sc een. One o he solu ions mos widely used o classi y da a a p esen is deep lea ning. Deep lea ning is a ield wi hin machine lea ning. O en deep lea ning is e e enced o images and ea u e ex ac ion (de ined on Figu e 2.3.1). Addi ionally, in deep lea ning, he models au oma ically pe o m ea u e ex ac ion o he cha ac e is ics wi h which be e esul s would be ob ained (Ma hWo ks, 2017). Deep lea ning is used on neu al ne wo ks as i will be explained in de ail on he nex subsec ion. 2.4. Image classi ica ion 15 2.4.1. Con olu ional Neu al Ne wo ks o image classi ica ion Neu al ne wo ks (NN) a e s uc u es based on how an ac ual b ain wo ks like. They a e composed o in e connec ed neu ons, he mos basic uni o da a, and hose a e g ouped in laye s, which a e communica ed wi h weigh s ha modi y he in o ma ion sha ed be ween neu ons. Each neu on is connec ed o all he neu ons om he p e ious and nex laye . Mos o en, he applica ion o hem is using hem as classi ie s, whe e i s laye con ains he inpu , and las laye is he ou pu . Be ween hese 2 laye s he e a e hidden laye s, which unc ion is o ind he co ela ions and pa e s be ween he inpu and he ou pu , modi ying he weigh s o he laye s h oughou he p ocess. The p ocess o aining a neu al ne wo k consis s on i e a e h oughou hem using unc ions called o wa d p opaga ion algo i hms. These go om he inpu laye s o he ou pu ones, measu ing he e o ob ained on he end and adjus ing he weigh s o he neu al ne wo k in o de o ge close o he desi ed esul . In addi ion o ha , he e a e some ex ensions o hese p ocess, such as back p opaga ion, which allows o check he e o in he ou pu laye and send i back o he p e ious laye s so he adjus men is mo e p ecise. Con olu ional neu al ne wo ks (CNN) di e on ypical neu al ne wo ks on he way neu ons communica e wi h each o he . Inpu neu ons a e no all assigned o e e y hidden laye ’s neu on. Ins ead, each inpu neu on is assigned o a g oup o neu ons om he hidden laye wi h weigh s ha ne e change, since each hidden laye ’s neu on is only used o a speci ic ea u e. This way mimics how an ac ual b ain wo ks, and his p ocess has been imp o ed on upda ed e sions like RNCC, Fas RNCC and Fas e RNCC. Thei e iciency has been es ed and wo k be e wi h la ge olumes o da a (Vic o Ikechukwu e al., 2021). Specially deep con olu ional neu al ne wo ks, which a e neu al ne wo ks wi h mul iple hidden laye s. As i has been said on he p e ious sec ion, using a p e ained neu al ne wo k acili a es aining, allowing aining o be done wi h some ini ial weigh s and ea u es, e en i he labels o he images a e di e en o he ones used o iginally o ain he NN. This opens he possibili y o aining o e neu al ne wo ks ha ha e al eady been ained on la ge da ase s o gene ic images, and une hem in o classi ie s wi h mo e speci ic labels. Cu en ly, he e a e wo popula ones widely used o his pu pose: Residual neu al ne wo k (ResNe -50) is a CNN al eady ained wi h mo e han one million images om he ImageNe da ase , and coun s wi h 50 hidden laye s. I ap- pea ed as a solu ion o he anishing g adien p oblem, which made he op imiza ion o neu al ne wo ks ha de as e o inc eased. I uses a echnique called skip connec- ions, which allows o omi communica ions ha would endange he accu acy o he neu al ne wo k. VGG-19 (Visual Geome y G oup) is ano he CNN, ained wi h 19 hidden laye s. I is based on a a ia ion o he same da ase used by ResNe -50. Coun s wi h 1000 labels and was ained o e 150000 images. I s s uc u e allows i o pe o m a be e ea u e ex ac ion, while using s a e-o - he-a echnologies in he ield o CNN. 16 Chap e 2. S a e o he a 2.5. Conclusion This chap e has discussed he di e en ways o es ing based on wha hey aim o , unc ional and isual es ing. Also, a esea ch has been made abou he di e en open sou ce lib a ies and machine lea ning models ha can be use ul o he de elopmen o he p ojec . Bo h hese es ing app oaches ha e hei own s eng hs, which will be discussed la e in he design chap e . Al hough he so wa e s udied pe o ms au oma ic es s ollowing a guide sc ip , hey a e no adap a i e o changes in he applica ion unde es . E en when unc ional es ing me hodology is p o en o be use ul, hey a e no compa ible wi h all es ing wo k lows due o i s p e iously men ioned limi a ions. Howe e , i is impo an o look in o de ail abou i s capabili ies gi en ha hey migh be use ul o u u e wo k. Since good ideo game GUIs ollow Ges al p inciples, hey cons i u e he en i onmen ha will appea mo e commonly on he majo i y o exis ing games. Wi h his in o ma ion a s anda d poin can be gua an eed in o de o pe o m es s on in e aces, knowing which elemen s hey ha e in common. Ha ing s udied he ele an echnologies in his chap e , i is ime o in eg a e hem in o he ool design. Chap e 3 Design o a ool o c ea ion o au oma ic es s o ideo game in e aces The s anda d way o pe o m menu na iga ion es ing is based on es ing a ideo game’s in e ace manually. O he me hods include unc ional au oma ion es ing ools (subsec ion 2.2.1) o manually p og ammed sc ip s. All hese solu ions equi e a lo o wo k and main enance, being edious as hey will ha e o be changed o es hem in each new game. The use o au oma ion in ideo game es ing has become qui e popula and menu na iga ion is no excep ion, he design o his ool akes a leap o wa d ocusing also on adap abili y, allowing au oma ic na iga ion in mul iple i les using one unique solu ion wi hou he need o edious main enance and ex ensi e sc ip ing. This chap e will discuss he design o ou ool o c ea ion o au oma ic es s o ideo game in e aces, Cogi o. Pa icula ly, i will be ocused on au oma ic menu na iga ion. This ool p ocesses images o de ec he di e en elemen s in a use in e ace using a com- bina ion o image p ocessing, op ical cha ac e ecogni ion and machine lea ning solu ions. The de ec ed elemen s a e hen classi ied in o di e en labels and based on he esul s Cog- i o inds he speci ic elemen s o each he desi ed des ina ion. Cogi o will use a sc ip ing language o enable high-le el es ing and will wo k in di e en ypes o ideo games. The design o he ool was an ongoing p ocess o esea ch ha changed signi ican ly h oughou he p ojec . The inal unc ionali y and ea u es o his ool will be explained du ing his chap e . 3.1. Cogi o pipeline Cogi o is an adap a i e ool o he au oma ic na iga ion o menus o mul iple di e en i les. This ool uses a combina ion o di e en image ecogni ion algo i hms and machine lea ning models o de ec and classi y he elemen s on a ideo game sc eensho and, based on he esul s, mo es o clicks on he i em ha is mos likely o help i each he desi ed des ina ion. The Cogi o pipeline wo ks as ollows: 17 18 Chap e 3. Design o a ool o c ea ion o au oma ic es s o ideo game in e aces 1. W i e a es ing sc ip indica ing ac ions such as "Go o Se ings". 2. Na iga ion, un he sc ip on he game: a) Take a sc eensho o he cu en s a e o he game. b) Iden i y and classi y elemen s on he sc een: 1) Image p ocessing Adap a i e con as and b igh ness. Adap a i e noise and blu il e ing. Edge de ec ion. Con ou de ec ion. Tex de ec ion. Icon de ec ion. 2) Elemen classi ica ion: Tex classi ica ion. Icon classi ica ion. Meaning ul elemen g oups. •Elemen g ouping de ec ion •Highligh ed bu on de ec o •Finding na egable menus c) Choose he elemen o in e ac wi h based on he p e ious s eps. d) Repea un il eaching des ina ion. The main ocus o his hesis has been he s ep (b): Iden i y and classi y elemen s on he sc een. Mos o he wo k has been done on he iden i ica ion and classi ica ion o da a due o he la ge scope o he p ojec . Ou side o he main ocus, s ep (c) has been b ie ly app oached in a ce ain way which will be explained la e on. Figu e 3.1 ep esen s he whole p ocess. Figu e 3.1: Cogi o pipeline The ollowing sec ions desc ibe in de ail he en i e Cogi o p ocess and ea u es. 3.2. Image p ocessing The Image P ocessing sec ion is one o he mos impo an pa s o he ool, as i helps o ind he sizes and posi ions o he di e en in e ace elemen s. I was concluded ha in o de o achie e an adap a i e ool, i was necessa y o ind a way o ocus only on he impo an elemen s o a menu, lea ing behind e e y hing unnecessa y such as dynamic elemen s o backg ounds. As seen in Figu e 3.2, mos o he 3.2. Image p ocessing 19 in e ace elemen s ha e an unnecessa y d awing o pic u e ha does no hing o indica e he meaning o he elemen , bu i is s ill equi ed o ge i s whole size and posi ion o achie e a p ope na iga ion and menu s uc u e, ha is why image p ocessing is used. Figu e 3.2: F1®22 mul iplaye menu example This p ocess consis s mainly in he de ec ion o edges and con ou s in a menu image, bu o he ool o be adap able o mul iple ypes o games, an adap a i e p ocess o di e en il e s mus be pe o med in o de o achie e he bes esul s. Figu e 3.3 ep esen s he whole Image P ocessing pipeline. Figu e 3.3: Image P ocessing pipeline The ollowing subsec ions desc ibe in dep h he Image P ocessing pipeline. 3.2.1. Adap a i e con as and b igh ness Tonal con as is he di e ence be ween b igh a eas and da k a eas in an image. To achie e a solu ion adap able o any ype o in e ace, he bes possible con as and b igh ness se ings mus be achie ed. The implemen ed p oposal is based on he alue o he onal con as calcula ion o he image, and hus be able o adjus he o e all b igh ness and con as o he image depending on ha alue. This onal con as alue is calcula ed and hen used in a combina ion o ope a ions: 1. The inpu image in RGBA is con e ed in o g ay-scale o apply a ixed-le el h eshold o ge a bina y image esul . This ixed-le el h eshold con e s any pixel ha exceeds he h eshold o whi e and hose no exceeded o black. 2. Based on he bina y esul , he numbe o black and whi e pixels and hus he a e age onal con as o he image is ob ained, as can be seen in Figu e 3.4. 3. This a e age onal con as can hen be used o ob ain a b igh ness alue o change he o iginal image’s b igh ness; he esul ing image can be ob ained by calcula ing 26 Chap e 3. Design o a ool o c ea ion o au oma ic es s o ideo game in e aces In o de o ob ain he adap i e h eshold alues, a combina ion o con e sion o he image o g ay-scale and he median o he image pixels using he o mulas 3.6 and 3.7 was used. In hese equa ions he sigma alue is se o 0.33 while he o he wo depend on i s g ay-scale median alues, his p ocess will be applied o each channel o he HSV image due o he possible loss o in o ma ion when con e ing o g ay-scale, by doing his on each channel and me ging he esul s he ool makes su e ha no in o ma ion is los and ob ains a mo e accu a e esul . Figu e 3.12: HSV Mul i-Channel Canny esul As can be seen on Figu e 3.12, he combina ion o each HSV channel Canny esul s on a bina y ma ix con aining he edges o he image. The impo ance o he b igh ness, con as and blu il e s explained abo e in pe o ming he Canny algo i hm will be app ecia ed below. Gaussian Fil e - Canny As explained abo e, he Gaussian il e is used o high con as images in o de o emo e unnecessa y noise o edges, as can be seen in Figu e 3.13, he Canny algo i hm is able o cap u e mos o he edges in a non-blu ed image bu main ains a conside able amoun o noise, so i is necessa y o apply a Gaussian il e in o de o educe such noise. Median Fil e - Canny The median il e is used o images wi h low con as and low sha pness, due o he low sha pness alue i is applied in a educed way o main ain as much edges as possible and educe he noise, as seen in Figu e 3.14. Bila e al Fil e - Canny The bila e al il e is used o images wi h low con as and high sha pness. Images wi h a high sha pness alue could indica e ha unimpo an backg ounds o ele- men s s and ou in he image. This il e is pe ec o educe his kind o noise and unnecessa y edges. As seen in Figu e 3.15 backg ound noise is conside ably educed. 3.2. Image p ocessing 27 (a) Non-blu ed image Canny (b) Gaussian il e image Canny Figu e 3.13: Compa ison o Canny applied o a Non-blu ed and Gaussian il e F1®21 image. Con ou s and edges om inside each de ec ed elemen a e mode a ely educed on he igh a e he il e is applied. (a) Non-blu ed image Canny (b) Median il e image Canny Figu e 3.14: Compa ison o Canny applied o a Non-blu ed and Median il e F1®20 image. The noise is sligh ly educed o he ool o co ec ly de ec he elemen s on he sc een. 3.2.4. Con ou de ec ion Con ou s a e cu es joined by all con inuous poin s along a bounda y ha ha e he same colo o in ensi y (B adski and Kaehle , 2008). To co ec ly de ec he con ou s o an image, some s eps mus be ollowed: 1. The image o be de ec ed mus be bina y o g ea e accu acy. To do his he canny algo i hm is used. 2. Based on he bina y image, a con ou de ec ion algo i hm is used. 3. Con ou s a e simpli ied and il e ed o keep only hose ha a e closed o con ex. (Figu e 3.16). 4. The bounding boxes a e ob ained om he con ou s in o de o ha e simpli ied shapes and hus ob ain mos o he bu ons o in e ace elemen s (Figu e 3.17). This s ep is ela i ely simple hanks o he wo k done in he p e ious phases, allowing a mo e accu a e de ec ion. Th oughou his p ocess he ool has applied di e en b igh ness, con as and blu il e s in o de o be able o use he edge and con ou de ec ion algo i hms in he image. 28 Chap e 3. Design o a ool o c ea ion o au oma ic es s o ideo game in e aces (a) Non-blu ed image Canny (b) Bila e al il e image Canny Figu e 3.15: Compa ison o Canny applied o a Non-blu ed and Bila e al il e FIFA 23 image. Mos o he noise om he backg ound is emo ed. Figu e 3.16: Fil e ed con ou s o a F1®2020 image A his poin i is necessa y o il e hese con ou s by keeping only hose ha con ain a meaning ul elemen , hese elemen s being ex o icons. This is done because a meaningless con ou is p obably no pa o a menu. By ob aining ex s and icons he ool will hen be able o classi y hem and hen ind s uc u es o na igable menu elemen s. 3.2.5. Tex de ec ion To de ec ex , an OCR machine lea ning model is used. A e ex ensi e es ing, i has p o en o be e y e ec i e and is able o de ec ex om a la ge numbe o di e en sou ces. Al hough i is a e y easible solu ion, i s ill equi es u he e inemen which will be desc ibed below: 1. The machine lea ning model ecognizes ex wo d-by-wo d due o possible e o s when de ec ing pa ag aphs o lines o ex . 2. Once all he wo ds o an image ha e been ob ained, hose ha a e no in ho izon al o ien a ion will be disca ded. Then hose wo ds ha a e su icien ly close and aligned ho izon ally will be me ged oge he and hose lines o wo ds ha a e close and 3.2. Image p ocessing 29 Figu e 3.17: Bounding boxes o a F1®2020 image aligned e ically will be joined oge he (Figu e 3.18). Wi h his p ocess he ool is able o de ec pa ag aphs and lines o ex . 3. Finally, he e u ned esul will be a lis o all he ex s wi h hei espec i e bounding boxes. Figu e 3.18: Tex de ec ed on a STAR WARS Jedi: Su i o ™menu 3.2.6. Icon de ec ion In addi ion o ex de ec ion, a gene al icon de ec ion is also pe o med. Fo his pu pose, a ained machine lea ning model is used o ecognize elemen s ha esemble an icon in a sc eensho . This model has been used because i is possible ha some bu ons only ha e an icon wi hou ex and i is necessa y o selec i o know wha i is, so i mus be de ec ed i s . This model is ained wi h a da abase o di e en images con aining icons o any ype, 30 Chap e 3. Design o a ool o c ea ion o au oma ic es s o ideo game in e aces all labeled wi h he same label. The model will ecei e an image and will de ec in i he elemen s ha ha e any esemblance simila o an icon. Figu e 3.19: Icons de ec ed by machine lea ning model on a FIFA 23 se ings menu As can be seen on Figu e 3.19, he model is able o de ec mos o he icons in an image and also conside s ex as icons, since du ing he aining p ocess some o he icon images used con ained ex . Bu due o he p e ious ex de ec ion i is possible o disca d hose ex s de ec ed by he gene al icon model and keep hose ue icons. Now ha he bounding boxes o he con ou s as well as he ex and icons o he image ha e been ob ained, i is ime o combine all he p e ious s eps. The ool will me ge bo h esul s in o de o il e ou he impo an elemen s o he image. Fo his pu pose, hose con ou s ha con ain a ex o an icon a e conside ed as ele an and because he con ou de ec ion could ail, i has been decided o keep hose ex s and icons ha a e no con ained in a bounding box, while he es o he con ou s will be disca ded. (a) FIFA 23 Se ings menu (b) Tex and icon de ec ion combined wi h he de ec ed bu ons on he image. Figu e 3.20: Tex de ec ion and icon de ec ion pe o med on a FIFA 23 menu. Le being he o iginal image and igh being a isual ep esen a ion o each kind o elemen de ec ed. As can be seen in Figu e 3.20, he ed ch oma ic boxes a e he con ou s o each elemen , he yellow boxes ep esen he ex and he pu ple boxes ep esen he icons. This esul is used o he nex s ep o he ool, classi ica ion. This is in cha ge o classi ying he ex s and icons o he da a ob ained. Once he elemen s ha e been classi ied sepa a ely, he ool will seek o g oup hese elemen s in o de o ind a na igable menu. 3.3. Elemen classi ica ion 31 3.3. Elemen classi ica ion Now ha in o ma ion like he posi ion o elemen s o hei con en s has been ob ained, his pa o he pipeline will ocus on classi ying hese elemen s. This will no only in o ex s and icons, bu also meaning ul elemen s 3.3.1. Tex classi ica ion In he case o ex classi ica ion, he ool mainly uses a dic iona y o labels (Figu e 3.21), whe e each label con ains wo ds o ph ases associa ed wi h ha label. The p ocess o classi y a ex using his dic iona y consis s o going h ough each wo d o each label in he dic iona y and making a compa ison be ween he wo d and he ex o be classi ied. This compa ison will be based on wo d simila i y and he con idence will be de e mined by ha simila i y. Figu e 3.21: Exi and Se ings label dic iona y examples This p ocess is applied o e e y ex ype en y o he p e ious phase (subsubsec- ion 4.2.3.2). Tex no classi ied co ec ly o wi h a con idence alue below a de ined h eshold will be disca ded. Figu e 3.22 shows an example o ex classi ica ion using his solu ion, using g een boxes o su ound classi ied elemen s, ma king hem wi h hei co esponding label. Al hough i has been men ioned ha his solu ion does no need edious main enance, Cogi o will s ill need o be main ained o keep he di e en labels in check wi h he ideo games o be es ed wi h. E en so, main enance will be minimal and his will be he only c i ical aspec o be upda ed. In he case o ex classi ica ion, he label dic iona y pa o he ool is he one ha needs he mos main enance, bu i can be done quickly as i only means including a wo d o ph ase in he label dic iona y. 3.3.2. Icon classi ica ion The p ocess ollowed by icon classi ica ion is simila o ha o icon de ec ion (subsub- sec ion 4.2.3.2), bu in his case applied only o speci ic ypes o menu icons. The model 32 Chap e 3. Design o a ool o c ea ion o au oma ic es s o ideo game in e aces Figu e 3.22: Classi ied ex s on a STAR WARS Jedi: Su i o ™menu. The g een boxes ou line he de ec ed ex s and display he label classi ied on op o hem. has been ained wi h mul iple examples o be able o ecognise ypical icons ha appea in mul iple ideo games such as he Se ings, Accessibili y o Con olle icons. Each ype o icon is a label o he model o ain. This solu ion, in combina ion wi h he icon de ec ion esul , can be e y use ul o speci y which unc ion a bu on has and hus be able o classi y bu ons ha do no con ain any ex . Figu e 3.23 shows an example o icon classi ica ion using he a o emen ioned model, using g een boxes o su ound classi ied elemen s on he same way ha was done wi h ex s, ma king hem wi h hei co esponding label as well. Figu e 3.23: Classi ied icons on a FIFA 23 se ings menu. The g een boxes ou line he de ec ed icons and display he classi ied label and con idence on op o hem. 3.3.3. Meaning ul elemen g oups Now ha mos o he elemen s o he sc een ha e been ob ained and hey ha e been classi ied, i is ime o check a ious aspec s o he elemen s in o de o know in which 3.3. Elemen classi ica ion 33 pa s o he sc een can he ool na iga e h ough. The s eps o pe o m a e he ollowing: Elemen g ouping de ec ion (subsubsec ion 3.3.3.1) - G oupings o elemen s o bu - ons will be de ec ed ollowing Ges al ’s p inciples o di e en ia e be ween g oups (sec ion 2.1). Highligh ed bu on de ec ion (subsubsec ion 3.3.3.2) - Inside each o hese g oups, he selec ed bu on mus be de ec ed in o de o p ope ly know whe e i is loca ed du ing na iga ion. Finding na igable menus (subsubsec ion 3.3.3.3) - I is necessa y o know which g oups o elemen s a e uly pa o he menu in e ace and can be na iga ed h ough. This case will help o es ablish he na iga ion’s en y poin and o know i we ha e eached he desi ed des ina ion. 3.3.3.1. Elemen g ouping de ec ion The objec i e o his s age is o ob ain he g oups o elemen s ha o m an aligned s uc u e because hey a e mo e likely o belong o a ideo game menu. This app oach has been decided due o he cu en s a e o ideo game in e aces, which a e la gely based on Ges al p inciples and main ain adequa e accessibili y o playe s (sec ion 2.1). The ool will sea ch o elemen s ha a e aligned bo h ho izon ally, e ically and in he o m o g id, in o de o ob ain he possible lis s o ho izon al, e ical o g id bu ons ha may be on he sc een. Following he Ges al p inciple o p oximi y, he ool assumes ha he elemen s ha a e close and aligned will belong o he same g oup. Once he g oups o ho izon ally o e ically aligned elemen s a e ound, he ool will sea ch i he e is a ela ionship be ween hese g oups, in o de o ind a g id o elemen s as well. When alking abou elemen s, he ool will mainly seek o g oup ex s and icons ha a e aligned. These elemen s may o may no be ac ually ela ed and o his he ool mus be able o keep only he co ec g oups o elemen s, ha is why he ool is equi ed o ha e a way o ind na igable menus. 3.3.3.2. Highligh ed bu on de ec ion The de ec ion o he selec ed bu on is a c ucial pa o he ool because knowing which elemen has he ocus g ea ly aids na iga ion. I we look a ideo game in e aces, he selec ed bu on usually has a di e en con as le el han he es o a di e en colo , so his in o ma ion is e y use ul when i comes o inding he di e en bu on. To do his, he ool uses an algo i hm based on his og am in e sec ion compa ison. When ge ing he g oups o bu ons he ool acqui es he his og am o each o he bu ons and compa es hem wi h he es using he in e sec ion algo i hm. I a his og am has a simila ange o colo s o con as i s in e sec ion will be g ea e while i hey a e o ally di e en he in e sec ion will be minimal (subsec ion 3.2.2). By ob aining he in e sec ions be ween e e y elemen in a g oup he ool is able o know which elemen is selec ed because i will ha e a much smalle in e sec ion han he o he s. 34 Chap e 3. Design o a ool o c ea ion o au oma ic es s o ideo game in e aces 3.3.3.3. Finding na igable menus In o de o he ool o be able o know i he e is a na igable menu on he sc een, i is necessa y o use he g oups o aligned elemen s and keep only hose g oups ha ha e a meaning. Tha is why he ool uses checke s o iden i y hose g oups, which a e in cha ge o checking ha he di e en g oupings o aligned elemen s con ain elemen s classi ied wi h labels belonging o a menu in e ace. Checke s help he ool ind he co ec g oups o elemen s and he ool will use hem o ind na igable menus o menu componen s such as menu ba s. These a e he ollowing ypes o checke s: Ho izon al and Ve ical Checke These checke s a e mainly in cha ge o inding in he ho izon al o e ical aligned g oups ob ained p e iously i hey comply wi h he ule p e iously men ioned, ha he elemen s o said g oup ha e speci ic labels. This ule is used so ha hese checke s only s o e g oups o elemen s ha can belong o an in e ac i e menu. They can wo k well when looking o menu ba s o p o ile ba s, as such menu i ems usually ha e he same ype o labels in any game. (a) F1®2021 menu (b) Menu ba de ec ed by ho izon al checke Figu e 3.24: Ho izon al checke o menu ba de ec ion on a F1®2021 menu As can be seen in Figu e 3.24, he menu ba is de ec ed co ec ly and e en includes which bu on is cu en ly selec ed. G id Checke This ype o checke uses he g id elemen de ec ion men ioned abo e which combines bo h ho izon al and e ical g oups o ind a g id which co ec ly is a menu in e ace. Uses he same ule as he p e ious checke s, i he elemen s sha e he same o some o he labels belonging o a menu in e ace. A e he p ocess o da a classi ica ion and checking, he ool e u ns all he collec ed in o ma ion o he na iga ion applica ion o i o use. This in o ma ion includes ex s, icons, con ou s and g oups o menu elemen s wi h hei selec ed elemen . Figu e 3.25 ep esen s a isual example o he inal esul . 3.4. Choose he elemen o in e ac 35 Figu e 3.25: Example esul o all he inal in o ma ion ga he ed by he ool. 3.4. Choose he elemen o in e ac The Cogi o applica ion is able o un a game, ake sc eensho s, send hose sc eensho s o he Na iga ion pipeline and based on he esul s e u ned by he de ec ion and classi ica ion wo k lows i will pe o m some inpu ac ions. I is impo an o highligh ha he na iga ion pa was no he main ocus o his p ojec . S ill, i s design is b ie ly explain as how i has been hough , Once he iden i ica ion and classi ica ion o elemen s is done, now he na iga ion s a s. I uses he da a ob ained by he p e ious s eps o pe o m an au oma ic na iga ion h ough he en i e menu o a ideo game, selec ing all he menu bu ons. This app oach is a simple s a o demons a e he po en ial o he ool. F om he da a ob ained, he inpu made by he ool changes acco ding o he ype o na igable menu in which i is loca ed. I i is ho izon al o e ical i will mo e along ha axis and i i is a g id i will mo e along bo h axis. The ool will sea ch o he des ina ion elemen on he menu and i i does no ind i , i will sea ch o elemen s wi h he same label as he des ina ion. The ool conside s ha i has eached he des ina ion i i has selec ed he elemen o i i has ound i in he sc een. In case he na iga ion ge s s uck o does no each he desi ed des ina ion i will s op and decla e he es as a ail. 3.5. Conclusion Th oughou his sec ion he pipeline o he Cogi o ool has been desc ibed. This is di ided in o mul iple pa s. The wo k done has mainly ocused on he s ep ega ding iden i ica ion and classi ica ion o elemen s, which o ms he en i e analysis pa o he ool and i s he one explained wi h mo e de ail. 42 Chap e 4. De elopmen ing. Two machine lea ning ex de ec ion models, AWS Rekogni ion and AWS Tex ac , we e es ed. Rekogni ion was shown o wo k wi h a ious ypes o on s o di e en i les, while Tex ac was a model ocused on documen ex de ec ion, so he i s ex de ec ion model was chosen. The p ocess o use he OCR is desc ibed on subsec ion 3.2.5. (a) Example o AWS Tex ac , almos none o he ex was de ec ed. (b) Example o AWS Reckogni ion, mos o he ex was co ec ly de ec ed. Figu e 4.6: Compa ison be ween AWS ex de ec ion se ices on Madden. As can be seen in Figu e 4.6, Rekogni ion’s ex de ec ion ou pe o ms Tex ac ’s in ideo game cases, de ec ing all he ele an ex o he sc een. Al hough Tesse ac OCR was in es iga ed a he beginning o he p ojec , i s esul s we e no sa is ac o y due o on de ec ion limi a ions. Addi ionally, i had he same p oblem as AWS Tex ac , i wo ked be e wi h documen s han wi h dynamic in e aces. 4.2.1.1. Tex de ec ion limi a ions and imp o emen s Rekogni ion ex de ec ion equi es a by e-encoded image as inpu and e u ns a dic io- na y objec , which indica es a ibu es abou he de ec ed ex s, such as posi ion, de ec ed wo ds, con idence and an unique id o each ph ase. I should be no ed ha ex s can be de ec ed as sepa a e wo ds o as ph ases. Howe e , i has occu ed ha wo ds ha we e pa o he same ph ase we e no always g ouped co ec ly. In addi ion, wo ds we e only g ouped ho izon ally, bu no e ically. Fo his eason, wo d de ec ion mode is used, and an in e nal adjus men is made ha o ms g oups based on he e ical alignmen o wo ds, and he dis ance be ween wo ds, based on hei on size. One p oblem encoun e ed in es ing was he one hund ed wo d limi : Rekogni ion limi es ic i s de ec ion o a hund ed wo ds pe call. This caused some images exceeding his amoun o ha e missing wo ds a he bo om o he sc een, because wo d de ec ion goes om he op le posi ion o he image o he bo om igh , eading wo ds om le o igh and op o bo om. This was sol ed by making se e al calls, adding a black box o he egions be o e whe e he las wo d was de ec ed in he p e ious call and combining he esul s om he di e en calls. Thanks o his solu ion, he ool is able o de ec all ex s co ec ly and hus also classi y hem co ec ly. 4.2.2. Second I e a ion - Image p ocessing Al hough he use o models ha de ec any ype o bu ons was disca ded, an a emp was s ill made o simpli y he da a o bo h aining and alida ion, because i wha 4.2. I e a i e de elopmen 43 hinde ed he model was unnecessa y in o ma ion, keeping only he necessa y in o ma ion would imp o e he esul s. This led o he c ea ion o an image p ocessing module, which simpli ied a sc eensho o an in e ace o ed and yellow boxes, signi ying con ou s and ex s espec i ely. Due o he g ea esul s p o ided by his solu ion, he bu on de ec ion model was ul ima ely disca ded and he p ojec ocused in o imp o ing he image p ocessing. The objec i e o his image p ocessing is o ob ain hose elemen s ha a e in he o eg ound o he image, pe o ming edge and con ou de ec ion echniques on images wi h speci ic blu and con as il e s. The whole inal p ocess is explained wi h de ail abo e (sec ion 3.2). E en so, a e he whole p ocess some cases may occu in which he elemen s s ill do no ha e hei con ou s co ec ly de ec ed (Figu e 4.7). Figu e 4.7: Missing con ou s in a FIFA 23 se ings menu 4.2.2.1. Missing con ou s solu ion The de ec ion o bounding boxes is no pe ec , so a missing con ou de ec ion algo i hm has been de eloped ha based on alignmen c ea es a new con ou o he con ou less ex o icon. The algo i hm eplica es he bounding boxes o nea aligned elemen s o i he missing space. I he elemen is no aligned wi h any o he , i is assumed ha i does no belong o a bu on lis o any hing like ha so no con ou is applied o i . Figu e 4.8: A i icial con ou s o missing con ou s de ec ed 44 Chap e 4. De elopmen 4.2.3. Thi d I e a ion - Elemen classi ica ion Once he bu on iles we e co ec ly de ec ed, he nex goal was o be able o classi y each elemen wi h a meaning ul label ha can be used in u u e s eps o na iga ion. Elemen s a e shapes eassembling bu ons on he sc een ha can con ain ei he a ex , an icon, o bo h. Mul iple app oaches we e es ed o his ask. 4.2.3.1. Gene al icon objec de ec ion model Using he AWS Rekogn ion cus om labels se ice, se e al models we e ained wi h he objec i e o de ec ing jus he icon posi ions, bu no i s meaning, as i was desc ibed on subsubsec ion 4.2.3.2. This was done in o de o acili a e he image p ocessing s ep, since he gene al posi ions a e needed o know wha is a bu on and wha is no , e en when he icon is oo abs ac o he models o de e mine i s meaning wi h he nex model. The idea was using mul iple sc eensho s om ideo games labeling all menu icons unde he same label. Howe e , in o de o achie e a good accu acy, models need om egula iza ion da a o a oid o e i ing, a p oblem ha would se e ely a ec he use ulness o his mode. The da ase used o his model includes no only ideo games sc eensho s. bu a com- bina ion o sc eensho s and a i icially c ea ed menus wi h gene ic icons (Delapoui e and Lo c, 2022) o e backg ounds. O e hese a i icial menus, mul iple hings we e es ed, o example changing i s o ien a ion, anspa ency, colo and backg ound. The inal da ase is composed o 44 game sc eensho s and 167 gene a ed images o aining, 9 game sc eensho s and 46 gene a ed images o es ing. A small sc ip was made in o de o gene a e his a i icial menus wi h a ia ions and ob ain hei posi ions o label hem au oma ically wi h he gene ic label and i s posi ion on he sc een. The eason o doing his is ha he model mus be an objec de ec o , so i needs o be ained wi h posi ions so ha la e i can also de ec icon posi ions. Excep o he modi ica ions o he anspa ency, all o he changes we e sa is ac o y imp o emen s, esul ing in a highly accu a e model. (a) A i icially c ea ed es in e ace (b) Example o gene ic icons model on FIFA 23 Figu e 4.9: Example o gene ic icon’s da ase and i s use on FIFA 23 In he p e ious igu e Figu e 4.9 (a) shows one image om he da ase wi h he p e i- ously men ioned modi ica ions, whe eas (b) shows how gene ic icons a e de ec ed wi h a con idence o e 80%, wi hou il e ing he icons no needed. 4.2. I e a i e de elopmen 45 4.2.3.2. Icon de ec ion model This model is based on pa o he da ase used o he gene ic icons model (see sub- sec ion 3.3.2). Howe e , gene ic icons can no be used o his app oach, so egula iza ion mus be done wi h speci ic icons, which causes on a la ge sized amoun o icons used. This is due o inding speci ic icons is a ha de ask which will equi e mo e e o han in he gene al model. In addi ion o ha , his model includes mul iple labels o he mos common elemen s ha ha e he same meaning be ween di e en ideo games gen es. Fo example, some labels used a e se ings, audio, accessibili y o social. I is impo an o no e ha his model will no be used on icons which meaning a ies depending on wha gen e you a e playing. An example o his issue would be an icon o a house. In he case o a gene ic game, his icon usually means " e u n o he main menu". Howe e , in a acing game, a house icon can be iden i ied as "ga age" o "cus omiza ion". Since icons ha a e om he same label can appea in di e en shapes, mul iple sub- labels o he same label a e used when needed. An example o his would be he mos common icons o sound, a speake icon, and a headphones icon. I only a label was used o each kind o icon, he model would i wo se o he al eady limi ed ain icons, which would esul on wo se p edic ion sco es. Fo bo h gene ic and speci ic icons, a SageMake model was ained as well using he ResNe -50 Con olu ional Neu al Ne wo k (see subsec ion 2.4.1) 4.2.3.3. SageMake ex classi ica ion Tex classi ica ion has been ied wi h mul iple app oaches. Th ee kinds o models we e es ed and compa ed o know which one was able o gi e be e p edic ions. Fu he mo e, aining a model om sc a ch is e y challenging, and needs o la ge amoun s o da a o be eliable. Howe e , his was ied o esea ch pu poses a his i e a ion. The accu acy o his model is e y low, ha ing he mean a e age p ecision (mAP) luc ua ing ou o con ol. Also i is no able o de ec simila seman ic ela ionships, ha ing a ha d ime o classi y e en a ia ions o wo ds wi h he same oo s unde he same label. The second app oach was aining o e an al eady ained model. SageMake allows ans e lea ning o ex on a lis o models. In he case o ex , i suppo s BERT model a ia ions. BERT (Bidi ec ional Encode Rep esen a ions om T ans o me s) p o ides dense ec o ep esen a ions o na u al language by using a deep, p e- ained neu al ne - wo k wi h he T ans o me a chi ec u e (De lin e al., 2019). Al hough i s accu acy is mo e s able compa ed o he p e ious app oach and pa o he ex is co ec ly labeled, i s ill happens ha on he same label, success ully classi ied ex sco es can a y g ea ly. This is a huge p oblem, causing he classi ica ion o be un eliable, since a low sco e can mean ei he a co ec o inco ec classi ica ion a he same ime. The hi d app oach was using a e olu iona y echnology called Ze o-Sho lea ning. I consis on models ained on na u al language p ocessing echniques wi h he di e ence o hem being able o p edic labels ha we e no speci ically ained wi h. This model by i sel is e sa ile, allowing o de ec any exis en labels i enough in o ma ion is used. A 46 Chap e 4. De elopmen disad an age ha i has compa ed o p e ious models is ha i does no allow uning o ans e lea ning using SageMake , so he esul s a e limi ed o wha he model o e s by de aul . In he end, due o he un eliabili y o hese solu ions, i was decided o implemen a dic iona y-based ex classi ica ion sys em, which is a mo e obus solu ion han he p e ious ones. 4.2.3.4. Dic iona y ex classi ica ion Dic iona ies appea as an a emp o imp o e ex classi ica ion. Each label has a lis wi h he mos common wo ds o each label ga he ed om mul iple i les om Elec onic A s. This gua an ees ph ases being co ec ly labeled when de ec ed, bu a he same ime i lacks adap abili y. The use o dic iona ies is explained on subsec ion 3.3.1. Some hing ha is done o inc ease he chances o a co ec classi ica ion is de ec ing pa ial ma ch ins ead o he ull wo d. The eason o do his is ha he OCR may ead a le e w ongly, causing i o ail in a limi ed amoun o cases. In o de o ix ha and simila issues like plu als o wo ds, his me hod goes le e by le e and e u ns a pe cen age wi h how simila o he o iginal ex a label is, doing his o each ph ase inside ha label (a) Se ings menu sc eensho o FIFA 23 (b) Gamemodes on FIFA 23 Figu e 4.10: Example o sc eensho s o di e en menus om FIFA 23 Howe e , e en wi h he subsequen imp o emen s, he e is a p oblem ha hese ex classi ie s canno a oid, which is lack o con ex . Fo example, on Figu e 4.10, on (a) he wo d "MATCH" can be in e p e ed by ained models as "play a ma ch", when igh he e i s con ex sugges s i ac ually means "ma ch se ings". In addi ion o ha , models can no de ec p ope nouns. Taking as example he wo d "SEASONS" on (b), models would assign a "season pass" label in he bes o he scena ios, ins ead o "game mode". I does no ma e how ex ensi e he aining is, i can no de ec wo ds i has no seen be o e. Figu e 4.11 is an example o an en y in he da a e u ned by he ool a e he ex classi ica ion using he dic iona y solu ion. As can be seen, he ex SOLO is classi ied as single playe . This comple es he mos i al in o ma ion abou he objec s de ec ed in he image. 4.2. I e a i e de elopmen 47 { "Type": "Tex ", "Tex ": "SOLO", "Box": { "Le ": 987, "Top": 319 "Wid h": 125, "Heigh ": 74, }, "Con ou Box": { "Le ": 967, "Top": 299, "Wid h": 358, "Heigh ": 299 }, "Labels": [ { "Label": "single_playe ", "Con idence": 1.0 } ] } Figu e 4.11: Combined dic iona y esul example. This example shows he ype o elemen , he ex i con ains wi h i s posi ion and dimensions as well as he posi ion and dimensions o i s con ou , and inally i s labels. 4.2.4. Fou h I e a ion - Checke s and Highligh ed Bu on An addi ion o he ool is he use o checke s. Checke s, as we de ine hem, a e classes ha check whe he ce ain labels a e con ained in aligned g oups o elemen s in o de o ind na igable menus. This is use ul o de ec componen s such as menu ba s, o o de ec which is he i s menu ound ha ul ills he ules o be a main menu. Thanks o he checke s, he con ex o he ool is upda ed, allowing i o unde s and be e i s cu en s a e. Fo ins ance, a common elemen in ideo game in e aces a e menu ba s. Knowing wha a menu ba is, hey will be conside ed as such, acili a ing na iga ion. In combina ion wi h checke s, he e is a second imp o emen ela ed o unde s anding he con ex o a ideo game s a e. Highligh ed bu ons a e conside ed as he mos di e en- ia ed bu ons om a g oup. Two me hods ha e been es ed, being he i s one de ec ing he highes mean de ia ion o V componen s in images wi h HSV o ma , and he second one doing his og am in e sec ions in o de o ind he lowes in e sec ion alue. Al hough he i s me hod wo ks in some F1 2022 es cases, i does no always wo k, since he a io be ween he numbe o backg ound and ex pixels is no always he same. Fo his eason, a gene ic compa ison o his og ams by applying in e sec ions wo ks be e . Bo h o hese addi ions a e explained on subsubsec ion 3.3.3.3. The basic p ocess is he opposi e: ins ead o looking o he mos di e en bu on, he 48 Chap e 4. De elopmen ool ocuses on wha makes all hese bu ons simila and hen excludes he one wi h he leas simila i y. This solu ion wo ks on all es ed games wi hou any alse de ec ion (see chap e 5). A he end o his whole Cogi o na iga ion pipeline p ocess he in o ma ion is s o ed in a dic iona y objec and used o pe o m inpu . The in o ma ion is s o ed as ollows: Da a: A lis con aining all he de ec ed ex s and icons s o ed in aw o m. Each elemen main ains he same s uc u e as Figu e 4.11. RawTex : A lis which con ains all he ex s wi hou u he in o ma ion. This lis is used in o de o de e mine i a menu has changed conside ably om one inpu o ano he in he na iga ion. MenuIn o: This en y on he dic iona y s o es he di e en ho izon al, e ical o g id menus ha we e de e mined by he checke s. Each menu con ains a lis o i s elemen s and he index o he selec ed elemen a he momen . 4.2.5. Fi h I e a ion - Lambda The ool a his poin consis s o mul iple ways o analyze wha is on sc een. In he u u e, ano he ool designed o na iga e using he in o ma ion cu en ly ob ained should be unning a he same ime. Fo his eason, and also o con enience, he objec i e o his i e a ion is po ing he code so i can un on a py hon lambda, in o de o acili a e emo e access on ano he de ice ha will be doing he na iga ion while communica ing wi h his lambda. Speci ically, he se ice used o hos his lambda is AWS Lambda, desc ibed on sub- sec ion 4.1.4. A con inuous in eg a ion sys em upda es he code wi h he newes upda es, and uploads i o he lambda. Some basic es s can be done his way o gua an ee ha he ool is wo king as in ended, bu o now he e a e no any. In he u u e i would be a good idea o ha e hem o speed up he wo k be ween i e a ions. 4.3. A chi ec u e Cogi o is a console applica ion implemen ed in Py hon and composed o wo Py hon modules: Cogi oApp, a console applica ion ha con ains he main loop o he ool. This applica ion is in cha ge o execu ing a gi en game, aking sc eensho s and sending hem o he lambda unc ion and based on he esul s o he unc ion i will pe o m inpu o ul ill i s objec i e. This applica ion uses an appse ings.json ile o de ine which game execu able o use and an inpu module o na iga e h ough a menu. The main ocus o his p ojec has been he Cogi o module and his Cogi oApp only con ains wha is necessa y o execu e a na iga ion loop. Cogi o, a module s o ed as a lambda unc ion in he AWS Lambda se ice. This module is he cen al pa o he ool, since i is in cha ge o all image p ocessing, ele- men de ec ion and classi ica ion p ocess explained be o e (sec ion 3.2). The lambda 4.3. A chi ec u e 49 unc ion akes an image as inpu and e u ns a dic iona y objec con aining all he de ec ed and classi ied elemen s, as well as he ype o menu and selec ed bu on. This module can also be used locally as a Py hon module. The a chi ec u e o he Cogi o module is di ided in o i e sub-modules, each one in cha ge o a pa o he image p ocessing, elemen de ec ion and classi ica ion p ocess ex- plained abo e (chap e 3). The ollowing UML diag am g aphically depic s he Cogi o a chi ec u e (Figu e 4.12). Figu e 4.12: Cogi o UML a chi ec u e diag am 4.3.1. Image P ocessing module This module is di ided in o se e al classes ha a e hen combined o ob ain he esul o con ou and edge de ec ion, he classes a e as ollows: B igh nessCon as Fil e e - As he name sugges s, his class is esponsible o adjus ing he b igh ness and con as o an inpu image o be e adap o con ou and edge de ec ion. Blu Au oma ion - I uses he ou pu o he B igh nessCon as Fil e e class o apply he di e en ypes o blu il e by ob aining he sha pness and con as as explained in subsec ion 3.2.2. Con ou De ec o - This class akes an image as inpu o de ec i s con ou s and con ains me hods o il e hose con ou s based on ex and icons. MissingCon ou De ec o - As explained ea lie in his sec ion, his class de ec s missing con ou s ha migh no ha e been co ec ly ecognized by he p e ious algo i hms. I de ec s hese con ou s based on alignmen , dis ances and al eady exis ing con ou s. ImageP ocesso - The ImageP ocesso class is he main ocus o his module, con- aining he es o he classes o combine hei esul s and e u n he inal ou pu . 50 Chap e 4. De elopmen EdgeDe ec ionS a egy - An abs ac class whose child en de ine he di e en Canny algo i hms o use. The di e en s a egies a e as ollows: •Mul iChannelS a egy - Applies he median Canny algo i hm o each channel o he inpu image and combines hem all. •O suS a egy - Applies he O su h esholding algo i hm which e u ns an in- ensi y h eshold o de ine he backg ound and o eg ound o an image (Sezgin and Sanku (2004)). This h eshold is used o he Canny algo i hm. •S aigh LineS a egy - Uses OpenC ’s s aigh line de ec ion algo i hm o de ec only he ho izon al and e ical lines o an image in combina ion wi h he Canny algo i hm. The s a egy used a he momen is he Mul iChannelS a egy. This class is used by he ImageP ocesso o p ope ly de ec he edges in a sc eensho . 4.3.2. Elemen De ec ion module The Elemen De ec ion module con ains he wo classes o machine lea ning used o de ec ea u es in an inpu image: Tex De ec - This class is esponsible o making calls o AWS Rekogni ion o de ec ex in an image. I has me hods ha based on he wo ds de ec ed by he model disca ds wo ds no aligned ho izon ally and combines hose wo ds in o sen ences o pa ag aphs based on alignmen , dis ance and on size. IconDe ec - Simila o he Tex De ec , is esponsible o making calls o AWS Rekogni ion o de ec icons in an image, his class needs a p ope ly ained model o wo k wi h and cu en ly uses a model ained and es ed wi h o e 250 images o gene al icon de ec ion. 4.3.3. Elemen Classi ica ion module I ollows a simila s uc u e con aining wo classes o classi y ex and icons espec- i ely. They ely on S a egy classes o es ablish di e en ypes o classi ica ion. Tex Classi ie and Tex Classi ie S a egy - These classes a e esponsible o classi ying he ex de ec ed by he Elemen De ec ion module. The Tex Classi ie class can ake he ou pu o he Tex De ec class and classi y i di ec ly, i uses a label dic iona y whe e each label con ains a lis o wo ds ela ed o i . This class uses he abs ac class Tex Classi ie S a egy o de ine wha kind o classi ica ion o use, a he momen , a Simila i yClassi ie S a egy is used o compa e he de ec ed wo ds wi h hose in he label dic iona y based on wo d simi- la i y, bu i can also be con igu ed o use an Exac Classi ie S a egy, which only labels de ec ed wo ds i hey a e exac ly in he label dic iona y. IconClassi ie and IconClassi ie S a egy - These classes ollow he same s uc- u e as he ex classi ica ion classes, bu ins ead o using a dic iona y o labels. 4.3. A chi ec u e 51 The IconClassi ie uses an IconClassi ie S a egy abs ac class which calls a model ha looks o common icons in a ideo game in e ace and classi ies hem. The S a egy used can implemen a Rekogni ion o SageMake model, which can be con igu ed on he models.py sc ip . Then he image is sen o he speci ied model, and each icon is classi ied wi h i s label. 4.3.4. Meaning ul Elemen Iden i ica ion module This module uses he classi ica ion esul s o ind signi ican elemen s o a ideo game menu. I con ains he ollowing classes: Alignmen Elemen De ec o - One o he pilla s o he module, his class uses he esul s ob ained in he de ec ion o ex s and icons o g oup hese elemen s in di e en ho izon al and e ical g oups, acco ding o hei alignmen and dis ance. This class s o es he g oups de ec ed o he o he classes o use. Checke - The Checke is an abs ac class ha akes an a ay o labels, an alignmen de ec o and a highligh ed bu on de ec o and uses hem o check o signi ican i ems in a menu. I s way o checking is o look o whe he a gi en numbe o labels om he label a ay a e ound in he g oups o aligned i ems. I also de ines unc ions o check speci ic labels in he da a ou pu e u ned by he classi ica ion and a simple se o ules o checking hese signi ican i ems. Ho izon al, Ve ical and G id Checke s - These classes inhe i om he Checke class, hey a e in cha ge o checking o each ype o g oup o elemen s i he e a e some labels o a gi en lis o labels. This solu ion is used o de e mine i an aligned g oup o elemen s is uly pa o a na igable menu, since elemen s in a menu should be ela ed o speci ic menu labels. Highligh edBu onDe ec o - This is one o he mos impo an classes o he ool, because knowing he exac bu on whe e he ool is loca ed g ea ly aids na iga ion. The Highligh edBu onDe ec o uses a g oup o aligned elemen s and he o iginal image o wo k p ope ly. This class acqui es he his og am o each elemen in he g oup and compa es i using an in e sec ion algo i hm. Once all he in e sec ions o he his og am o all he bu ons a e ob ained, he sum o all he in e sec ions o each bu on is done and he lowe one will be he selec ed bu on, since i i s in e sec ions wi h he es o he bu ons is low i means ha i does no look like he es o he bu ons which makes i he highligh ed bu on. OpenCV allows he de elope o de ine he size o he his og am o be mo e o less de ailed. In he case o his class small size his og ams a e used due o he le el o de ail in some bu ons which can complica e he de ec ion, ha ing a small his og am will keep he gene al de ails o he bu on and will allow a mo e accu a e compa ison be ween he o he bu ons. 4.3.5. Con igu a ion module In o de o p ope ly con igu e he ool a module was c ea ed. This con igu a ion module con ains wo .json iles which one s o es he ool con igu a ion (appse ings.json) and he 58 Chap e 5. E alua ion 5.2. Time pe o mance As a ma e o ac , ob aining be e imes has no been he p ima y objec i e o his p oo o concep , bu is s ill necessa y o unde s and he cu en s a e o he ool. The main goal o his e alua ion will inding he bo lenecks on he wo k low o he ool in o de o know which pa s need o be mo e ime e icien . 5.2.1. Me hodology This e alua ion has been done moni o ing he ime employed by each s ep on e e y image om he las e alua ion. In addi ion o he images om F1®2022, ime has also been moni o ed on sc eensho s om ano he ideo game, FIFA 2023. This ime measu emen has only been done once due o he amoun o images used, which should be desc ip i e enough abou he s a e o he ool. Time has been measu ed on he ollowing s eps: con as and b igh ness, blu il e , edge de ec o , con ou de ec o , ex de ec o , icon de ec o , ex classi ica ion, icon clas- si ica ion and highligh ed iles de ec ion. I is impo an o no e ha hese se ices include ne wo k la ency on he calls o he models esponsible o ex de ec ion, icon de ec ion, and icon classi ica ion, which a e cloud-based online se ices, bu he es o he s eps a e done locally on a i ual machine wi h 16 CPU, 110 GB o RAM, and an NVIDIA Tesla T4 as GPU (1 GPU and 16 GB). The me ics used o de e mine how much ime i akes o he ool o p ocess an image will he calcula ed his way: 1. Time e en s will be s o ed on a lis each ime an impo an pa o he ool s a s and inishes. This is epea ed o each image and i is done in he same way a eleme y sys em would wo k, bu a a much smalle scale. 2. When hese wo ime s amps a e ob ained, a sub ac ion will be done. The ime ob ained will be calcula ed by sub ac ing he end ime om he s a ime, and hen inal imes will be s o ed on a da a ame ( o easie la e display o he esul s) Wi h his in o ma ion, i can also be calcula ed he o al ime needed o p ocess each image. 5.2.2. Resul s and conclusions The same numbe o images has been used han in he p e ious e alua ion: 129 o F1®2022, and 72 o FIFA 20231. The ull esul s a e included on a d i e olde , bu only a ac ion o hose a e included on he ables om he igu es since no all en ies a e needed o unde s and he objec i e. As i can be seen on Figu e 5.2 and Figu e 5.3, in bo h ables each column show he ime needed o each s ep o each image o each game. All esul s a e so ed by he o al ime 1Full esul s a ea a ailable a h ps://d i e.google.com/d i e/ olde s/1on6ZYTw0sAniN4AEGEaUqITnbzMLw- dC?usp=sha ing 5.3. Conclusions 59 needed o p ocess an image, going om he lowe o he highes imes. The slowes s ep is highligh ed in ed, and he o al p ocessing ime o ha image is highligh ed in g een. Analyzing he esul s om bo h games he same limi a ion appea s. One o he mos ime consuming pa s is de ec ing he ex on he sc een using Rekogni ion’s OCR model. In addi ion o his, he e is ano he bo leneck ha appea s mos ly on FIFA 2023 some imes when adding blu il e s, which can be seen on he hi d column. Mo e speci ically, i has been obse ed ha highe imes only occu when Bila e al il e s a e applied, al hough his in o ma ion is no isible on he able. The eason why his is chosen au oma ically o he mos i ing images and why he il e akes longe is explained on subsec ion 3.2.2. This is due o i is based on a ime consuming algo i hm which akes each pixel and con e s i o he weigh ed a e age o i s neighbou s, and he ime needed eally depends on he ha dwa e on which he ool is unning. An ob ious limi a ion his p oo o concep has is wo king on a single h ead. The use o synch onous calls o he models used on ex de ec ion, icon de ec ion, and icon classi ica ion o ces he ool o wai un il models ha e done hei wo k. Howe e , he e a e pa s o he ool ha could be wo king a he same ime. I calls o hese h ee models we e done a he same ime while blu il e s a e applied, a lo o ime could be po en ially sa ed. On Figu e 5.4 and Figu e 5.5 i can be seen he esul s om he ables in he shape o g aphs. Obse ing he me ics his way shows in a mo e ob ious way how ime is was ed wai ing o he esponse o machine lea ning models, when i is no eally necessa y o wai o hei esul s since he con en o hese models is no needed o calcula e p e ious s eps. Finally, on Figu e 5.7 and Figu e 5.6 boxplo diag ams can be seen. These a e only o p o e ha he s eps o blue il e and ex de ec ion keep he alues shown on he ime ables. This can be seen on he blue lines inside he boxes, ep esen ing he median o he seconds pe s ep o each image. 5.3. Conclusions A e ob aining hese esul s, some poin s ha e been cla i ied. Fi s , he quali y o machine lea ning models ela ed o icons needs o be imp o ed. While he icon de ec ion model has shown esul s ha s a o ge close o being eliable, i s ill is no , and nei he he icon classi ica ion is. Mo e aining should be done wi h a be e da ase o inc ease hei con idence. In addi ion o his, o he s eps no ela ed o machine lea ning need mo e wo k, which is he case o he highligh ed bu ons de ec o . Now i has o be s udied he eason why hese esul s a en’ eliable. The high amoun o alse nega i es in compa ison wi h alse posi i es included on Figu e 5.1 seem o indica e some hing is ailing in on some images whe e no highligh ed bu on is de ec ed a all, so ha should be s udied. Finally, he o al imes ob ained o his p oo o concep a e easonable. Howe e , he whole p ocess o he ool is expec ed o un mul iple imes on es s. Fo his eason, ime should be educed as much as possible o a oid un easonable long es s in he u u e. An e ec i e way o elimina e he p oblem o was ing execu ion ime would be o use a mul i- h eaded implemen a ion, as i would educe he unp oduc i e wai s o a minimum, achie ing a possible imp o emen o educing he ime needed on F1®by hal in he bes 60 Chap e 5. E alua ion o si ua ions. 5.3. Conclusions 61 Figu e 5.2: Table o imes o each s ep o he ool applied o FIFA 2023 sc eensho s (in seconds) 62 Chap e 5. E alua ion Figu e 5.3: Table o imes o each s ep o he ool applied o F1®2022 sc eensho s (in seconds) 5.3. Conclusions 63 Figu e 5.4: G aph o imes o each s ep o he ool applied o FIFA 2023 sc eensho s (in seconds) Figu e 5.5: G aph o imes o each s ep o he ool applied o F1®2022 sc eensho s (in seconds) Figu e 5.6: Boxplo o imes o each s ep o he ool applied o FIFA 2023 sc eensho s (in seconds) 64 Chap e 5. E alua ion Figu e 5.7: Boxplo o imes o each s ep o he ool applied o F1®2022 sc eensho s (in seconds) Chap e 6 Conclusions and Fu u e Wo k Manual es ing is a cos ly and edious job ha in ol es a la ge numbe o es e s pe o ming simple asks ha a e pe o med epe i i ely o e a la ge numbe o ideo games. The job o hese es e s should be o ocus on es ing hose elemen s o he game ha a e complex and hus ha e a mo e polished game. This is why an au oma ic na iga ion ool would be ideal o es ing a ideo game, since he menus in he indus y a e usually simila and es ing in such menus is ela i ely simple. This p ojec has desc ibed he design and de elopmen o he Cogi o ool, which h ough he use o image p ocessing, elemen de ec ion and classi ica ion is able o iden i y he elemen s o a ideo game menu using a sc eensho and pe o m he necessa y inpu o na iga e h ough ha menu. Cogi o is di ided in o wo pa s: (1) w i ing a sc ip based on human-like commands such as Go o Se ings and (2) isual na iga ion o un he sc ip . This na iga ion is mainly ocused on de ec ing ou lines o bu ons, ex and icons, wi h ha in o ma ion i is able o classi y he elemen s and pe o ms inpu based on i . This p ojec has been mainly ocused on he na iga ion pa o he ool. The de elopmen o Cogi o s a ed wi h a p o o yping phase in which eam membe s looked o capabili ies o implemen om o he exis ing ools. Once all he necessa y in o ma ion was ga he ed, he i e a i e de elopmen p ocess began, wi h each i e a ion esul ing in an imp o ed e sion o he ool. This i e a i e p ocess was s ongly based on he SCRUM agile me hodology. Cogi o s a ed as a p oo o concep ha o e ime g ew in o a p ojec wi h a lo o po en ial, which can be wo ked on u he o co ec ly na iga e h ough a menu as a sc ip ing language o es e s o use. Now, Cogi o is a ool in de elopmen which has been implemen ed as desc ibed h ough- ou his documen , implemen ed on py hon. I makes use o a po en lib a y o image p ocessing, OpenCV, as well as implemen s machine lea ning models using Amazon Web Se ices o he speci ic asks o ex de ec ion, icon de ec ion and icon classi ica ion. An e alua ion o he ool has been done, and i has been de e mined ha i s ill has lo s o oom o imp o emen e e ing bo h o quali y and imes o each s ep. Howe e , i is clea now which s eps need o be mo e e icien , and which s eps need o imp o e hei p edic ions. A e wo king on he issues men ioned, he ool could signi ican ly imp o e. 65 66 Chap e 6. Conclusions and Fu u e Wo k The ollowing sec ion will discuss he imp o emen s ha can be applied o Cogi o and u u e wo k o he ool. 6.1. Fu u e wo k Al hough he esul s ob ained a e p omising, he ool is no pe ec . Since i is a p oo o concep , he e is plen y o oom o imp o emen . A ew i ems abou he mos impo an upg ades ha can be done would be: Con inue de eloping a sepa a ed na iga ion app ha elies on Cogi o implemen a ion o unde s and and s o e s a es o he game. This app would un locally whe e he game is unning, and using in o ma ion p o ided by he Cogi o unning on AWS Lambda, i would unde s and in wha menu he game is, which ones a e he possible inpu ac ions, and how o ind a speci ic ex by using labels. An addi ional use o he ool ha does no ely on labeling would be a menu c awle . A e ob aining he s uc u e o a menu using isual es ing echniques, unc ional es ing ones could be applied o e i in o de o c ea e a g aph ha maps each di e en node om he g aph o a speci ic menu. This would be use ul o au oma ically ind ou i all he menus be o e launching a game mode a e accessible, which is an impo an common ask done while es ing GUIs. Doing mo e esea ch abou possible ways o educe unp oduc i e wai ing imes. One way would include a mul i h ead implemen a ion, as i was men ioned on he ime pe o mance conclusions (subsec ion 5.2.2), ha calls he models in pa allel while execu ing he image p ocessing s eps on ano he h ead, and hen only wai ing when i ac ually eaches he poin hey a e needed. Tha would allow minimizing he o al ime o he p ocess, educing he unp oduc i e wai s o ze o i he models inish hei asks be o e eaching he poin hey a e needed. Imp o e e iciency in some o he mos ime consuming s eps o image p ocessing, such as applying bila e al il e s, which can ake whole seconds o be applied. A possible way ha has no been es ed on his p ojec o imp o e his is using Fou ie T ans o m, a p ocess ha is based on he idea o all unc ions can be app oxima ed as he sum o in ini e sinus and cosines. Image ope a ions may be op imized using hese ans o ma ions. Once he da ase s used by Rekogni ion models ha e shown good esul s, hese could be con e ed o he o ma SageMake uses. This can be done wi h a sc ip ha was c ea ed du ing he c ea ion o he i s SageMake models, which akes a .ls ile and con e s i o a .mani es one. The eason o do his is SageMake allows o ha e mo e con ol o model aining. I does no necessa ily mean he esul s a e going o be be e , since his has been al eady ied wi h he gene al icons model ob aining a wo se mAP. E en so, i would be wo h ying, bu i is no a c i ical issue igh now. T ain AWS Comp ehend models o be used in combina ion wi h he al eady exis ing checke s in o de o ge a be e con ex o he si ua ion o in e aces in ideo games. This could be used o e y speci ic menus, such as se ings, popups, o "p ess x bu on o play" menus. These would be good examples o use since he ex con ained 6.1. Fu u e wo k 67 in hese menus is always e y simila , allowing a possible in eg a ion o Comp ehend wi hin he ool ha uns a cus om classi ie model o hese menus. Howe e , i would only wo k wi h de ailed menus. Mo e abs ac menus such as a main menu, which can con ain wo ds ela ed o all he menus men ioned be o e, would no wo k on his model. Checke s s ill will ha e a eason o exis in hese cases, which is why a combina ion o bo h would gi e be e esul s. 74 Appendix A. Appendix A - Con ibu ions C ea ion o sc ip s o au oma e he gene a ion o da ase s sepa a ed on aining and es se s o Rekogni ion and SageMake images classi ica ion models. •In es iga ion abou he use o a g aphic anno a ion ool o ob ain labeled iles in YOLO .’ x ’ o ma . Con igu a ion o de aul labels o ma ch he ones o ou models. •Au oma ic de ec ion o labels used on he p e ious ool o use hem di ec ly, wi hou con e ing he o ma o ano he one. •Con e sion om ’. x ’ labeled iles o ’.ls ’ o SageMake and ’.mani es ’ o Rekogni ion. •Use o AWS S3 se ice o hos he images only once. The upload is done au oma ically, as well as he deploymen o he models when hey inish aining. Collabo a i e wo k on esea ch abou OpenCV and i s unc ionali ies: •In es iga ion abou he possibili y o using simple solu ions such as empla e ma ching ins ead o objec de ec ion models. •In es iga ion o mo e complex image compa ison echniques such as ea u e ex ac ion o his og am compa ison, and o wha si ua ions hey a e mo e sui able o . •Resea ch abou possible solu ions o imp o e canny edge de ec ion, such as mo - phological ope a o s, and he implemen a ion o canny edge de ec ion applied o e e y colo channel o a oid losing in o ma ion. •Resea ch abou o he edge de ec ion solu ions like s aigh lines de ec o s. •Wo k on con ou il e ing o a oid sa ing non closed shapes as iled bu ons. Collabo a i e wo k on g ouping iled bu ons based on hei alignmen , mainly o- cused on g ids. La e his was use ul o g oup elemen s in o speci ic objec s, such as menu ba s. In es iga ion o ex classi ica ion al e na i es such as ze o-sho models and o he na u al language p ocessing echniques. Also in es iga ion abou using al eady ex- is ing da ase s in o de o do ine- uning o ge cus om ex classi ica ion models, ins ead o building one om sc a ch. Tha would equi e a huge amoun o esou ces ha a e no easonable. •A he end he ex classi ica ion models we e disca ded since keeping a gene al app oach o es di e en games opposed ha ing a speci ic ained model o each game. T aining o objec de ec o and image classi ica ion models using AWS Rekogni ion and SageMake . A he end, SageMake models we e elega ed as possible u u e wo k. The eason o ha is ha Rekogni ion allowed a as e i e a ion in aining models, while SageMake cos ed no only mo e ime, bu also be e ine- uning. Rekogni ion also does his au oma ically. •T aining o icon de ec ion models, o which I ga he ed a la ge quan i y o images and uned hem o imp o e model con idence o eliable le els. This model only had one label, since i had one speci ic pu pose: inding e e y icon on he sc een, wi hou ocusing on hei meaning. This included he c ea ion o a sc ip A.2. Iago Quin as Diz 75 ha enhanced icons so he model could lea n as e wi h less da a, pe o ming ope a ions such as changing icon sizes, colo s, o ien a ions and backg ounds being some o hose. •T aining o icon classi ica ion models, which a e less eliable han he de ec ion model. This is due o inc easing he numbe o labels and he lowe amoun o images appea ing on he da ase . Ga he ing images o speci ic icons was a much mo e complex ask. In addi ion o his, i is impo an o highligh ha public icon da ase s o he speci ic icons ha a e needed do no exis . E alua ion o he ool, planning he di e en objec i es, designing he me hodology o do i and analysing he esul s a e ha . Mo e speci ically, hese we e he s eps done •Adding a simpli ied eleme y sys em o he ool o eco d ime e en s, ga he ing da a ames o each game es ed. E en s we e sen o a class o pe o m he ime measu ing ope a ion, and hen each alue was s o ed on hei co esponding image and s ep. •A e ha , he ool un h oughou all images. Then. manually classi ica ion o images was done o de e mine ue posi i es, alse posi i es, ue nega i es and alse nega i es. •Wi h he esul s ob ained, g aphics and ables we e c ea ed in o de o isualize be e he da a and each conclusions as e abou wha s eps ha e mo e issues. •Finally, wi h his in o ma ion, u u e wo k was decided abou wha s eps needed mo e wo k o imp o e and be eliable.