A Mobile Assessment Tool for Collecting Data in Large-Scale Educational Studies

A ailable online a www.sciencedi ec .com P ocedia Compu e Science 00 (2018) 000–000 www.else ie .com/loca e/p ocedia The 15 h In e na ional Con e ence on Mobile Sys ems and Pe asi e Compu ing (MobiSPC 2018) A Sma Mobile Assessmen Tool o Collec ing Da a in La ge-Scale Educa ional S udies Ke in And ewsa,∗, Michael Zimocha, Man ed Reiche a, Miles Tallonb, Ul ich F ickb, R¨ udige P yssa aUlm Uni e si y, Ins i u e o Da abases and In o ma ion Sys ems, James-F anck-Ring 1, Ulm, 89081, Ge many bHSD - Hochschule D¨op e , Uni e si y o Applied Sciences, Waldma k 3 and 9, Cologne, 50676, Ge many Abs ac Conduc ing scien i ic s udies is an o en us a ing and edious ask. To minimize ecu ing p oblems, such as lack o concen a- ion o willingness o pa icipa e, and ins ead p omo e in e es in he s udy, a sma mobile de ice assessmen ool was de eloped ocusing on educa ional s udies. The able -based assessmen ool o e s a wide ange o isual asks ha can be employed when conduc ing s udies u ilizing he Eu opean F amewo k o Visual Li e acy (ENViL). Fu he mo e, he assessmen ool is highly con- igu able in he ield by using a cen alized se e and sp eadshee -based con igu a ion iles, he eby ensu ing ha no p og amming language is equi ed o adap he asks on he mobile de ices pa icipa ing in he s udy. Finally, he p esen ed amewo k and a chi ec u e adhe e o he c oss-pla o m and c oss-de ice s yle and can be e-used and ex ended o any numbe o simila s udies. c 2018 The Au ho s. Published by Else ie L d. This is an open access a icle unde he CC BY-NC-ND license (h p://c ea i ecommons.o g/licenses/by-nc-nd/3.0/). Keywo ds: Sma Mobile Assessmen , Visual Li e acy, Cul u al Educa ion, La ge-Scale S udies 1. In oduc ion Nowadays, sma mobile de ices (e.g., sma phones and able s) a e ubiqui ous and he exploi a ion o hei cha - ac e is ics (e.g., mobili y) o e s p omising pe spec i es o a ious domains. Fo example, he applica ion o sma mobile de ices in he ga he ing and sensing o da a o e s oppo uni ies o un a el insigh s as well as indings in he ield o science and echnology [2,13,6]. Fu he mo e, he p oli e a ion and la ge-scale dis ibu ion o sma mobile de ices has os e ed a g owing in e es o esea che s o u ilize hese kind o de ices when collec ing da a [16,12]. Resea che s use sma mobile de ices as da a collec ion ools by employing speci ic applica ions (i.e., assessmen ools), he eby gaining no only aluable da a, bu also expe ience in add essing he nume ous p ac ical challenges o unde aking success ul s udies [19]. Assessmen ools on sma mobile de ices allow o an ex ensi e assessmen and ∗Co esponding au ho . Tel.: +49-731-50-24-133 ; ax: +49-731-50-24-134. E-mail add ess: ke[email p o ec ed] 1877-0509 c 2018 The Au ho s. Published by Else ie L d. This is an open access a icle unde he CC BY-NC-ND license (h p://c ea i ecommons.o g/licenses/by-nc-nd/3.0/). 2Ke in And ews e al. /P ocedia Compu e Science 00 (2018) 000–000 e alua ion o ac o s, bo h objec i e as well as subjec i e, o in e es om indi iduals. The e o e, assessmen ools a e o en applied in s udies and, hence, being sui able o he ope a ionaliza ion o hese ac o s [5]. Fo many domains, such as educa ional o social science, which o en use o dinal measu emen s, ope a ionaliza ion is essen ial, since ac o s a e s ic ly de ined as measu able a iables [1]. In he ield o educa ional science, assessmen ools enable esea che s o c ea e assessmen asks o de e mine whe e indi iduals (i.e., lea ne s) a e on he con inuum o lea ning [10]. Fu he mo e, insigh s can be ob ained abou knowledge, skills, and concep s he lea ne has mas e ed and wha may ha e been misunde s ood o no ye mas e ed [3]. In his con ex , pa icula emphasis can be pu on he •imp o emen o s a is ical and empi ical e alua ions, •ga he ing o addi ional insigh s abou lea ning s eng hs and challenges, •moni o ing o lea ne p og ess, •indi idualiza ion o ins uc ional quali y o inc ease hei e ec i eness. Conside ing hese emphases, he wo k a hand p esen s an assessmen ool o sma mobile de ices de eloped in he con ex o a p ojec in es iga ing isual li e acy in cul u al educa ion. S udies can be easily designed and conduc ed wi h he ool and adop ions can be made wi hou p og amming a line o code. Hi he o, he assessmen ool has been success ully applied o o e 500 pa icipan s. The emainde o his pape is s uc u ed as ollows: Sec ion 2 explains he p ojec con ex . In Sec ion 3, he assessmen ool design is p esen ed, ollowed by he use case desc ibed in Sec ion 4. Sec ion 5 discusses ela ed wo k and, inally, Sec ion 6 summa izes he pape and gi es an ou look. 2. Con ex o he P ojec The p esen ed assessmen ool was de eloped in he con ex o a coope a i e p ojec ocusing on isual li e acy in cul u al educa ion1. The pu pose o his p ojec is o gain insigh s on how isual li e acy can be os e ed, ope a ional- ized, and, hence, empi ically measu ed. The eby, isual li e acy is a concep ha de ines he capabili y o in e p e , unde s and, and ex ac in o ma ion p esen ed in images [17]. Fu he mo e, s udies ha e shown ha he use o ap- p op ia e images mo i a es lea ning p ocesses as well as p omo es he de elopmen o lea ning s a egies. Based on his, he p ojec analyzes he in e - (i.e., be ween indi iduals) and in a-indi idual (i.e., wi hin indi idual) a ia ion o mo i a ion in lea ne s such as pupils, s uden s, and eache s. This includes, o example, he pe cep ion o ins uc- ional quali y wi h ega d o isual li e acy. The eby, a pa icula emphasis is pu on he e ec s o cul u al capi al (i.e., social asse s like educa ion and pe sonali y) as well as cul u al-aes he ic p ac ices (e.g., eligious belie s) o lea ne s. Cul u al educa ion is assessed wi hin he Eu opean F amewo k o Visual Li e acy (ENViL) in a se ies o na ionwide s udies ac oss a ious kinds o schools (e.g., seconda y school and uni e si y) in Ge many [18]. To help conduc he s udy, an assessmen ool o sma mobile de ices was de eloped o assess he isual li e acy in cul u al educa ion. Lea ne s Wi h Di e se Cul u al Capi al Lea ne s Wi h Di e se Cul u al Capi al Collec ion o Ques ions Collec ion o Visual Tasks Va ie y o Ques ions and Visual Tasks Addi ional Da a ia Eye T acking Fig. 1. Visual Li e acy in Cul u al Educa ion Speci ically, pa icipan s (i.e., lea ne s) o he s udies we e asked o comple e a a ie y o isual asks (e.g., Comple ing Pic u es and Men- al Ro a ion) and answe a se o ques ions, while he assessmen ool cap u ed ele an in o ma ion [8]. Mo eo e , p ima ily o s eng hen he con e gen alidi y (i.e., measu es whe he applied cons uc s a e ela ed), addi ional da a will be collec ed ia eye acking. In pa icu- la , wi h e ospec i e hink-aloud (RTA) s udies and pos expe ience eye acking p o ocols (PEEP), an endea o is o ga he addi ional in- sigh s abou di e ences be ween no ices and expe s wi h ega d o i- sual li e acy [7]. The objec i e is o gain well- ounded insigh s on how ins uc ions should be shaped o os e lea ne mo i a ion and isual li e acy. These insigh s will be ed back o he pa icipa ing lea ne s and he knowledge ob ained will be implemen ed in ins uc ions a e he p ojec e m. 1h ps://www.dip .de/en/ esea ch/cu en -p ojec s/bkkb- isual-li e acy-in-cul u al-educa ion?se _language=en Ke in And ews e al. /P ocedia Compu e Science 00 (2018) 000–000 3 3. Mobile Assessmen Tool Design The mobile assessmen ool p esen ed in his pape is a highly ex ensible and con igu able amewo k o able - based isual asks. I o e s a a ie y o i ems, ca ego ized in o isual asks and ques ions ha pa icipan s may com- ple e. The isual asks ange om simple ones, such as so ing objec s and iden i ying small de ails in la ge images (c . Fig. 2), o complex ones, such as c ea ing a 2D image om a 3D image and a gi en pe spec i e (c . Fig. 12). As i was c ea ed speci ically o conduc ing isual li e acy s udies on pupils, he assessmen ool had o be buil om sc a ch o suppo no only he exac s udy equi emen s, bu also he addi ional equi emen s imposed when dealing wi h pupils. As mos o he assessmen akes place in class oom se ings wi h as many as 30 pupils and only one o wo supe iso s, he pupils ha e o be comple ely blocked om pe o ming ac ions ou side o he in ended scope. In consequence, he use in e ace o he en i e applica ion is s ipped down o an absolu e ba e minimum wi h no con igu a ion o na iga ion op ions isible o use s. Fig. 2. Visual Task “Iden i y De ails” One o he main challenges when implemen ing he assess- men ool was c oss pla o m compa ibili y, as he a ge ha d- wa e (Windows, And oid, o iOS) had no been inally speci ied when he de elopmen s a ed. Fu he mo e, ou in en was o c ea e a highly eusable amewo k ha could be u ilized o a - ious s udies, po en ially conduc ed on o he able , o e en desk- op ha dwa e. In any o he si ua ion, a simple way o ackling his challenge would be o implemen he i ems u ilized in he s udy as a web applica ion. Howe e , as mos b owse s do no allow o locking use s ou om basic na iga ion unc ionali y and he p ima y use case is conduc ing s udies wi h pupils, who a e o en unp edic able o e en mischie ous by na u e, we we e o ced o implemen he mobile assessmen ool as a c oss pla - o m mobile applica ion. This allows o ull con ol o e he displayed isual elemen s and na iga ion con ols. An example o one o he isual asks used in he s udy, p esen ed in he e y es ic i e use in e ace s yle necessa y o dealing wi h child en, is shown in Fig. 2. To keep c oss pla o m compa ibili y, in combina ion wi h ull use in e ace con ol, we op ed o he Xama in Fo ms amewo k. Xama in Fo ms allows us o euse almos all code, including he use in e ace, ac oss Windows, And oid, and iOS de ices, while s ill o e ing access o ad anced ea u es like sc een eco ding o ouch inpu logging, which a e u ilized in some o he games o collec ing addi ional da a. Howe e , implemen ing he assessmen ool as a na i e o c oss pla o m applica ion, ins ead o a web applica ion, con lic ed wi h ano he equi emen ha he ool shall be ex ensible and highly con igu able wi hou ha ing o upda e he applica ion on all 30 able s e e y ime unc ional changes a e made o indi idual i ems. Such changes, e.g., exchanging an image o changing he o de in which he i ems a e p esen ed o s udy pa icipan s, can be done cen ally i he able s a e jus displaying a web si e om a se e . Howe e , i each able has he applica ion ins alled locally, one has o upda e all able s h ough sideloading o e e y unc ional change. Un o una ely, one can no use he au o-upda ing capabili ies o he a ious app s o es, as hey do no allow apps wi h ad anced ea u es, such as sc een eco ding, which a e equi ed o collec ing he addi ional da a equi ed by ou use case. To ackle hese issues, we implemen ed a e y simple se e in ASP.NET Co e, a ligh weigh and c oss pla o m C# amewo k ha allows o hos ing REST web se ices and iles in a b owseable olde s uc u e. The se e has only wo in e aces, one ha sends a lis o all iles i is hos ing o he eques ing able and ano he o ecei e he esul da a gene a ed by pa icipan s comple ing he a ious i ems, which we will de ail subsequen ly. This allows us hos ing all unc ional esou ces on he se e , including all ex s and images, allowing hem o be upda ed o exchanged quickly in one cen alized loca ion. When he applica ion is s a ed on he a ious able s, hey each eques he comple e ile lis om he se e ia a REST call, and s a p eloading he da a o a oid communica ion o e head du ing he assessmen . The se e ollows a ligh weigh s yle and can be hos ed on any ope a ing sys em on a lap op o simila de ice when conduc ing an assessmen in he ield. As each o he isual asks can ha e mul iple a ian s, e.g., di e en images wi h a ying di icul y le els o he ask shown in Fig. 2, he se e has o ha e an ex ensi e olde s uc u e wi h all images, ex s, and o he iles so ed by ask and a ian . An example o one such olde o a speci ic a ian o one o he isual asks is shown in Fig. 3. 4Ke in And ews e al. /P ocedia Compu e Science 00 (2018) 000–000 Fig. 3. Visual Task Va ian Folde Fig. 4. I em Sequence Con igu a ion The con en s o his olde , which cons i u e all he da a ha he a ian “exhibi ion” o he ask “iden i y de ail” equi es o un, can be edi ed by manipula ing he iles on he se e . As p e iously men ioned, he applica ion will eload all iles om all o hese olde s di ec ly a e i is opened on he able s, wi hou ha ing o upda e he app package. The se ings.cs ile p esen in he olde can be used o change con igu a ion op ions o he a ian , such as ime limi s o ins uc ion ex s. CSV (Comma Sepa a ed Values) iles can be in e p e ed as ables by Excel, allowing o simple edi ing o se ings. This makes CSV iles ideal o edi ing by esea che s, as hey do no need p og amming expe ience o ools o mo e specialized ile o ma s, such as XML. Fu he mo e, CSV iles can also be used o de ine he sequence o i ems which a e p esen ed o pa icipan s, as shown in Fig. 4. The example game sequence con igu a ion ins uc s he applica ion o i s display an eye acking calib a ion page, nine a ian s o a isual ask called “dalli klick”, ano he calib a ion page, and, inally, wo a ian s o a ask called “ anking” in he exac o de o he ows. As hese con igu a ion iles a e downloaded om he se e a applica ion s a , he sequence o i ems can be changed cen ally on he se e and is e ec i e immedia ely a e e-opening he able applica ion. To acili a e his, we build a s ack o he i em sequence om he ows o he CSV ile shown in Fig. 4a applica ion s a up. This s ack con ains all in o ma ion necessa y o load a speci ic i em, i.e., he class name con aining he logic o he isual ask and he name o he olde on he se e ha con ains he da a o he speci ied ask a ian . In pa icula , his allows esea che s o add a new i em in he ield by adding ano he a ian o an exis ing isual ask. This can be achie ed by c ea ing a olde wi h he images and se ings o he a ian (c . Fig 3) as a sub- olde o he isual ask on he se e and adding he name o he new sub- olde in a new ow o he i em sequence con igu a ion CSV ile (c . Fig. 4). When he able applica ion is wo king h ough he s ack o i ems and eaches a new isual ask a ian , i can be displayed seamlessly, as he images and ex s in he a ian olde we e p eloaded om he se e on applica ion s a up. Clea ly, all a ian s o a isual ask mus use he same code. Addi ionally, some i ems ha e special con igu a ion iles ha allow o g ea e cus omiza ion o he isual ask and i s a ian s. An example o one such con igu a ion ile, o he pic u e e ealing ask “Dalli” (c . Fig. 5), in which an image ge s e ealed o e ime, is shown in Fig. 6. The ma ix con igu a ion ile de ines he o de in which he iles co e ing he pic u e a e emo ed. Again, as his is edi ed cen ally on he se e and applied o all able s on applica ion es a , his allows o con igu a ion o un- ime logic wi hou ouching code o he applica ion deployed on he able s, a huge bene i o esea che s accus omed o Excel, especially when wo king in he ield. Fig. 5. Visual Task “Dalli” Fig. 6. “Dalli” Ma ix Con igu a ion Ke in And ews e al. /P ocedia Compu e Science 00 (2018) 000–000 5 Fig. 7. “Iden i y De ails” Solu ion Con igu a ion Fig. 8. “Iden i y De ails” Resul s File Fig. 9. Touch T acking Log “MB-1 exhibi ion.cs ” Ha ing discussed he con igu a ion o he i ems and he sequence in which hese a e p esen ed o s udy pa icipan s, we u n o he ou pu ha is p oduced while he i ems a e being comple ed. As he goal is o allow o mos ly au oma ed e alua ion o esul s, some i ems need addi ional con igu a ion o de e mine which use inpu cons i u es a “co ec ” solu ion. An example o he solu ion con igu a ion o he “Iden i y De ails” isual ask is shown in Fig 7. The con igu a ion ile allows o he speci ica ion o pai s o X/Y coo dina es and a adius a ound hese ha he applica ion should a e as a co ec solu ion when pa icipan s ma k hem du ing he assessmen (c . Fig. 2). An exce p o he co esponding esul s ile ha is c ea ed on he se e a e he isual ask was comple ed by wo di e en pa icipan s, wi h he codewo ds MB-1 and MB-2, is shown in Fig. 8. The alues o C1, C2, and C3, i.e. whe he he pa icipan ci cled he i s , second, and hi d de ail a ea co ec ly, a e au oma ically calcula ed by he applica ion wi h he help o he con igu a ion ile. Fu he mo e, as each ow ep esen s a single assessmen o one pa icipan , addi ional in o ma ion, such as a ouch acking log (c . Fig. 9), can be e e enced in each ow in he FileName column. These addi ional logs can be me ged wi h he main esul s ile in ools such as SPSS o Ma lab o c ea e hea maps and s a is ical g aphs o all collec ed esul s. The inal assessmen capabili y we wish o highligh is he p esen a ion o ex ual ques ion i ems. C ea ing a gene ic me hod o p esen ing a ques ion i em as an a bi a ily sized able o mul iple choice ques ions and answe s, as shown in Fig. 10, p o ed o be qui e a challenge. To keep in line wi h he con igu a ion s yle o he isual asks, each ques ion i em is con igu ed by h ee sepa a e CSV con igu a ion iles, one con aining he ques ions, one con aining he possible answe s, and one con aining he addi ional desc ip i e ex s. As he ho izon al sc een space o he able can be used up quickly by long ques ions in he esul ing able columns, we had o de elop a simple algo i hm o scaling down he on size o he longes answe s dynamically, and inse ing line b eaks be o e he on size eaches an un eadable size. This way, hese aspec s, i.e., on size and line b eaking, do no ha e o be con igu ed by he esea che s designing he ques ion i ems, simpli ying hei wo kload signi ican ly. Fu he mo e, as we ha e an a bi a y numbe o ques ions and answe s belonging o a single ques ion i em, he design o he esul s ile o he ques ion i ems was challenging as well. As, a he end o an assessmen , he single esul s ile o he ques ion i ems should con ain he answe s o all pa icipan s o all indi idual ques ions, we op ed o ea each ques ion i em as a a ian o a base “ques ion” ask. This enables a s yle consis en wi h he esul iles o he isual asks (c . Fig. 8), wi h all answe s gi en by a single pa icipan conce ning a ques ion i em w i en in o he columns o a single ow (c . Fig. 11). Fig. 10. Ques ion I em Fig. 11. Ques ion I em Resul s File (Mul iple Columns Hidden) 6Ke in And ews e al. /P ocedia Compu e Science 00 (2018) 000–000 I em esul s, such as he ones shown in Fig. 8, Fig. 9, o Fig 11, a e gene a ed by he applica ion unning on he a ious able s du ing he assessmen . Howe e , as p e iously men ioned, he able s a e no gua an eed o ha e an In e ne connec ion in he ield, so uploading hese esul s o a se e on he In e ne is no easible. Fu he mo e, ga he ing all he Excel esul iles s o ed locally on he able s would be an ex emely edious ask. Ins ead, he able s use he same REST web se ice used o p eloading he da a and se ings iles on applica ion s a up o pe iodically ans e esul s o he se e . Addi ionally, he usage o WiFi o ans e ing game esul s allows o he ans e o la ge supplemen al iles, such as he images p oduced when playing ano he one o he games, “Iden i y Pe spec i e O ien a ion”, shown in Fig. 12. The game p o ides he s udy pa icipan s wi h a pale e o colo s, which hey ha e o use o c ea e a 2D image o a 3D model om a ce ain pe spec i e. This game ans e s no only he Excel esul s iles wi h he usual s a is ics on ime aken, ouch poin s, e c., bu also an image ile o he solu ion, which can be used o manual e alua ion la e on. An abs ac schema ic o he communica ion pa e n be ween he applica ion and se e pa s o he mobile assessmen ool is shown in Fig. 13. No e ha a di ec WiFi connec ion be ween he able s hos ing he applica ion and he lap op ac ing as he se e may no always be possible as i is shown in Fig. 13. This is due o he ac ha he Windows d i e s o many popula WiFi chipse s only suppo ad-hoc WiFi connec ions wi h up o eigh de ices, i a all. To alle ia e his issue, we allow o he u iliza ion o an addi ional WiFi access poin o ho spo , which all able s and he se e can connec o. Fu he mo e, i is possible o connec he able s and he se e o any on-p emise WiFi, e ec i ely emo ing ange es ic ions be ween he able s and he se e . This enables unning mul iple assessmen s in di e en pa s o he same building concu en ly wi h only one lap op ac ing as a se e , as long as all pa icipa ing de ices a e on he same local WiFi ne wo k. To his end, he able s show a simple IP Add ess inpu dialog on s a up i hey canno connec o hei las known se e IP add ess. Finally, in he e en ha he WiFi becomes conges ed o b eaks down, ei he du ing he p eloading phase when he applica ion is s a ed on he able s o du ing he assessmen phase when lea ne s a e comple ing he isual asks and esul s a e being sen o he se e , he e a e sa egua ds in place o p e en he applica ion om mal unc ioning. On one hand, iles ha a e no ans e ed success ully in he p eloading phase a e au oma ically downloaded om he se e once hey become necessa y o he execu ion o an i em. On he o he , esul s ha ail o ans e o he se e a e cached locally on he able and e- ansmi ed once he WiFi connec ion becomes a ailable again. Theo e ically, i he p eloading phase is comple ed success ully, he en i e assessmen can be comple ed wi hou a WiFi connec ion as he able s ha e enough memo y o cache he mos ly ex -based assessmen esul s o long pe iods o ime. In summa y, his means ha he goal o c ea ing a mobile assessmen ool ha is con igu able by non-p og amme domain expe s in he ield was achie ed. Fu he mo e, h ough ou unique a chi ec u e, u ilizing a p eloading phase o all images, ex s, and con igu a ion iles on applica ion s a up, comple e econ igu a ion o all assessmen i ems is possible in a cen alized loca ion, equi ing only simple sp eadshee edi ing so wa e such as Excel. Finally, in con as o a web applica ion, ou applica ion has he ad an age o p e en ing s udy pa icipan s om exi ing he assessmen applica ion by na iga ing o o he websi es o changing se ings in he b owse , which is c ucial when wo king wi h g oups o younge pupils in class oom se ings. Fig. 12. Visual Task “Iden i y Pe spec i e O ien a ion” Se e Table 1 P eload Da a on S a up Upload Addi ional Files Upload Excel Resul s Table nTable 2 Fig. 13. Mobile Assessmen Tool A chi ec u e Ke in And ews e al. /P ocedia Compu e Science 00 (2018) 000–000 7 4. Use Case As desc ibed in Sec ion 2, he assessmen ool was de eloped in he con ex o a p ojec ocusing on he isual li e acy in cul u al educa ion and has been success ully es ed in p ac ice. So a , a o al o 547 pa icipan s we e success ully assessed by ou pa ne s in Cologne and F ank u am Main. The long- e m objec i e is o collec da a om o e 800 pa icipan s. The pa icipan s a e ec ui ed in 20 di e en Ge man schools, loca ed in 4 ede al s a es o Ge many, i.e., No h Rhine-Wes phalia, Hesse, Schleswig Hols ein, and Ba a ia (in planning). The eby, he assess- men en i onmen is p e alen ly he isual a s class oom. I a isual a s class oom is no a ailable, he assessmen is conduc ed in a no mal class oom. Fu he , he la ges assessmen class con ains 30 pa icipan s and, on he o he , he smalles one comp ises 12 pa icipan s. In gene al, he assessmen ool comp ises 108 i ems ca ego ized in o isual asks and ques ions. In pa icula , 19 i- sual asks co e 61 i ems (i.e., each a ian o a ask ep esen s an i em) and 47 i ems a e ela ed o ques ions (i.e., each ques ion ep esen s an i em). Fu he mo e, a pape -based ques ionnai e, no co e ed by he assessmen ool, o pa en s is handed ou o he pa icipan s and needs o be answe ed by hei pa en s a home. The p ocedu e o he s udy is as ollow (c . Fig. 14): Gene ally, all pa icipan s ge an in oduc ion be o e he assess- men . In mo e de ail, one pa icipan o he class eads he in oduc ion ex aloud in on o all pa icipan s in he class. A e wa ds, he pa icipan s need o wo k on a p ede ined sequence o isual asks ela ed o isual li e acy and answe a numbe o ques ions abou pe sonal in o ma ion and cul u al capi al. The eby, he pa icipan s a e asked o pe o m he assessmen a hei own pace. All eme ging da a is s o ed anonymously. Mo eo e , he pa icipan s need o answe all ques ions hones ly and may ask ques ions a any ime i he e a e any misunde s andings o ambigui- ies. The du a ion o he assessmen is app oxima ely 40 minu es. A e comple ing all isual asks and answe ing all ques ions, he pa icipan s a e able o pe o m a d awing ask as many imes as desi ed o he assessmen o c ea i e image y abili ies in he la e analysis. Fig. 14 illus a es he used s udy design. A e comple ion, all da a is uploaded o he se e (c . Fig. 11). Besides he expec ed da a o 800 pa icipan s, in addi ion, i is planned o conduc an in- s uc ional s udy du ing he isual a s class (i.e., ambulan assessmen ). The e o e, he applica ion will be andomly s a ed du ing he class. Fu he mo e, speci ic isual asks a e de eloped in such a way ha addi ional da a can be ga he ed by means o eye acking. Acco dingly, eye acking da a will be u ilized o RTA as well as PEEP s udies showing he playbacks o eye mo emen s in uncued o cued s imuli (i.e., images). De ined Sequence o Ques ionnai es and Visual Tasks …. Lea ne s D awing Task Du a ion o Comple ion: App ox. 40 Minu es Fig. 14. P ocedu e o he S udy 5. Rela ed Wo k The ope a ionaliza ion o assessmen ools in he con ex o sma mobile de ices has been add essed by a ious o he esea ch wo ks. Fo example, he au ho s o [15] demons a e how domain expe s can be empowe ed when implemen ing obus sma mobile da a collec ion applica ions using a model-d i en amewo k ela ed o end-use p og amming. In u n, [14] p esen s he g aphical domain-speci ic language MAML, which enables he au oma ic gene a ion o na i e mobile business applica ions. Mo eo e , an app oach o he de elopmen o mobile applica ions in he con ex o heal h ca e o ganiza ion is in oduced in [11]. Wi h a s onge in ol emen o clinicians, a me hod- ology o he c ea ion o educa ional applica ions o sma mobile de ices is discussed in [20]. Wi h a ocus pu on mobile lea ning, wi hou being limi ed by ime and space, [9] illus a es he bene i s o lea ning ac i i ies using appli- ca ions on sma mobile de ices. In he con ex o usabili y s udies on sma mobile de ices, a amewo k emphasizing he ad an age o s udy design on such de ices is p esen ed in [4]. Howe e , a con igu able assessmen ool as shown he e, especially in he con ex o educa ional s udies, was no p esen ed by o he wo ks so a . 8Ke in And ews e al. /P ocedia Compu e Science 00 (2018) 000–000 6. Summa y and Ou look This pape ga e insigh s in o he a chi ec u e and amewo k o an assessmen ool de eloped in he con ex o a p ojec ocusing on isual li e acy in cul u al educa ion o sma mobile de ices. Fu he mo e, he success ul applica- ion o he assessmen ool in a la ge-scale educa ional s udy wi h mo e han 500 pa icipan s so a is shown he ein. Cu en ly, he assessmen o u he pa icipan s wi h he p esen ed ool is in p og ess wi h he objec i e o assess a leas 800 pa icipan s. Mo eo e , addi ional e ospec i e hink-aloud as well as pos expe ience eye acking p o o- cols s udies a e planned o he u u e wi h he ea u es and suppo ha a e al eady ealized in he assessmen ool. Due o he ac ha he assessmen ool, including i s a chi ec u e and employed amewo k ha e been de eloped in a c oss-pla o m and c oss-de ice ashion, i can be e-used o simila s udies in he isual li e acy o simila con ex s. Fu he mo e, he s udy design (i.e., he sequence o isual asks and ques ions) can be indi idually cus omized. Fi- nally, ex ensions and adap a ions can be implemen ed in he assessmen ool ollowing eme ging equi emen s wi hou he need o p og amming. Acknowledgmen The p ojec BKKB is unded by he Bundesminis e ium ¨ u Fo schung und Bildung (BMBF). Re e ences [1] Amo, C., Cousins, J. B., 2007. Going Th ough he P ocess: An Examina ion o he Ope a ionaliza ion o P ocess Use in Empi ical Resea ch on E alua ion. New Di ec ions o E alua ion (116), 5–26. 1 [2] Bo u , J. T., S o ie, D., 2014. Mobile De ices in Medicine: A Su ey o How Medical S uden s, Residen s, and Facul y use Sma phones and o he Mobile De ices o Find In o ma ion. Jou nal o he Medical Lib a y Associa ion: JMLA 102 (1). 1 [3] Cobc o , R. S., Towe s, S. J., Smi h, J. E., B uns, A., 2006. Mobile Lea ning in Re iew: Oppo uni ies and Challenges o Lea ne s, Teache s, and Ins i u ions. Online Lea ning and Teaching, 21–30. 1 [4] Cou sa is, C. K., Kim, D. J., 2011. A Me a-Analy ical Re iew o Empi ical Mobile Usabili y S udies. Jou nal o Usabili y S udies 6 (3), 117–171. 5 [5] Dea nley, C., Taylo , J., Hennessy, S., Pa ks, M., Coa es, C., Haigh, J., Fai hall, J., Riley, K., D ans ield, M., 2009. Using Mobile Technologies o Assessmen and Lea ning in P ac ice Se ings: Ou comes o Fi e Case S udies. In e na ional Jou nal on E-Lea ning 8 (2), 193–207. 1 [6] El-Hussein, M. O. M., C onje, J. C., 2010. De ining Mobile Lea ning in he Highe Educa ion Landscape. Jou nal o Educa ional Technology & Socie y 13 (3), 12–21. 1 [7] F eeman, B., 2011. T igge ed Think-Aloud P o ocol: Using Eye T acking o Imp o e Usabili y Tes Mode a ion. In: P oceedings o he SIGCHI Con e ence on Human Fac o s in Compu ing Sys ems. pp. 1171–1174. 2 [8] G een ield, P. M., 2009. Technology and In o mal Educa ion: Wha is Taugh , Wha is Lea ned. Science 323 (5910), 69–71. 2 [9] Hwang, G.-J., Chang, H.-F., 2011. A o ma i e assessmen -based mobile lea ning app oach o imp o ing he lea ning a i udes and achie e- men s o s uden s. Compu e s & Educa ion 56 (4), 1023–1031. 5 [10] Hwang, G.-J., Tsai, C.-C., 2011. Resea ch T ends in Mobile and Ubiqui ous Lea ning: A Re iew o Publica ions in Selec ed Jou nals om 2001 o 2010. B i ish Jou nal o Educa ional Technology 42 (4), 65–70. 1 [11] Khamba i, A., G undy, J., Wa en, J., Hosking, J., 2008. Model-D i en De elopmen o Mobile Pe sonal Heal h Ca e Applica ions. In: P oc In ’l Con on Au oma ed So wa e Enginee ing. IEEE Compu e Socie y, pp. 467–470. 5 [12] Kim, S., Manko , J., Paulos, E., 2013. Sens : E alua ing a Flexible F amewo k o Au ho ing Mobile Da a-Collec ion Tools o Ci izen Science. In: P oceedings o he 2013 con e ence on Compu e suppo ed coope a i e wo k. ACM, pp. 1453–1462. 1 [13] Lane, N. D., Miluzzo, E., Lu, H., Peebles, D., Choudhu y, T., Campbell, A. T., 2010. A Su ey o Mobile Phone Sensing. IEEE Communica ions Magazine 48 (9), 140–150. 1 [14] Riege , C., Kuchen, H., 2018. A P ocess-O ien ed Modeling App oach o G aphical De elopmen o Mobile Business Apps. Compu e Languages, Sys ems & S uc u es 53, 43–58. 5 [15] Schobel, J., P yss, R., Schickle , M., Ru -Leuschne , M., Elbe , T., Reiche , M., 2016. End-Use P og amming o Mobile Se ices: Empow- e ing Domain Expe s o Implemen Mobile Da a Collec ion Applica ions. In: In ’l Con on Mobile Se ices (MS). IEEE, pp. 1–8. 5 [16] Schobel, J., P yss, R., Wipp, W., Schickle , M., Reiche , M., 2016. A Mobile Se ice Engine Enabling Complex Da a Collec ion Applica ions. In: In e na ional Con e ence on Se ice-O ien ed Compu ing. Sp inge , pp. 626–633. 1 [17] S okes, S., 2002. Visual Li e acy in Teaching and Lea ning: A Li e a u e Pe spec i e. E J o he In o Tech in Edu 1 (1), 10–19. 2 [18] Wagne , E., Sch¨ onau, D., 2016. Common Eu opean F amewo k o Re e ence o Visual Li e acy - P o o ype. Waxmann. 2 [19] Wu, W.-H., Wu, Y.-C. J., Chen, C.-Y., Kao, H.-Y., Lin, C.-H., Huang, S.-H., 2012. Re iew o T ends om Mobile Lea ning S udies: A Me a- Analysis. Compu e s & Educa ion 59 (2), 817–827. 1 [20] Zhang, M., Cheow, E., Ho, C. S., Ng, B. Y., Ho, R., Cheok, C. C. S., 2014. Applica ion o Low-Cos Me hodologies o Mobile Phone App De elopmen . JMIR mHeal h and uHeal h 2 (4). 5