A ailable online a www.sciencedi ec .com
P ocedia Compu e Science 00 (2018) 000–000
www.else ie .com/loca e/p ocedia
The 15 h In e na ional Con e ence on Mobile Sys ems and Pe asi e Compu ing
(MobiSPC 2018)
A Sma Mobile Assessmen Tool o Collec ing Da a in La ge-Scale
Educa ional S udies
Ke in And ewsa,∗, Michael Zimocha, Man ed Reiche a, Miles Tallonb, Ul ich F ickb,
R¨
udige P yssa
aUlm Uni e si y, Ins i u e o Da abases and In o ma ion Sys ems, James-F anck-Ring 1, Ulm, 89081, Ge many
bHSD - Hochschule D¨op e , Uni e si y o Applied Sciences, Waldma k 3 and 9, Cologne, 50676, Ge many
Abs ac
Conduc ing scien i ic s udies is an o en us a ing and edious ask. To minimize ecu ing p oblems, such as lack o concen a-
ion o willingness o pa icipa e, and ins ead p omo e in e es in he s udy, a sma mobile de ice assessmen ool was de eloped
ocusing on educa ional s udies. The able -based assessmen ool o e s a wide ange o isual asks ha can be employed when
conduc ing s udies u ilizing he Eu opean F amewo k o Visual Li e acy (ENViL). Fu he mo e, he assessmen ool is highly con-
igu able in he ield by using a cen alized se e and sp eadshee -based con igu a ion iles, he eby ensu ing ha no p og amming
language is equi ed o adap he asks on he mobile de ices pa icipa ing in he s udy. Finally, he p esen ed amewo k and
a chi ec u e adhe e o he c oss-pla o m and c oss-de ice s yle and can be e-used and ex ended o any numbe o simila s udies.
c
2018 The Au ho s. Published by Else ie L d.
This is an open access a icle unde he CC BY-NC-ND license (h p://c ea i ecommons.o g/licenses/by-nc-nd/3.0/).
Keywo ds: Sma Mobile Assessmen , Visual Li e acy, Cul u al Educa ion, La ge-Scale S udies
1. In oduc ion
Nowadays, sma mobile de ices (e.g., sma phones and able s) a e ubiqui ous and he exploi a ion o hei cha -
ac e is ics (e.g., mobili y) o e s p omising pe spec i es o a ious domains. Fo example, he applica ion o sma
mobile de ices in he ga he ing and sensing o da a o e s oppo uni ies o un a el insigh s as well as indings in he
ield o science and echnology [2,13,6]. Fu he mo e, he p oli e a ion and la ge-scale dis ibu ion o sma mobile
de ices has os e ed a g owing in e es o esea che s o u ilize hese kind o de ices when collec ing da a [16,12].
Resea che s use sma mobile de ices as da a collec ion ools by employing speci ic applica ions (i.e., assessmen
ools), he eby gaining no only aluable da a, bu also expe ience in add essing he nume ous p ac ical challenges o
unde aking success ul s udies [19]. Assessmen ools on sma mobile de ices allow o an ex ensi e assessmen and
∗Co esponding au ho . Tel.: +49-731-50-24-133 ; ax: +49-731-50-24-134.
E-mail add ess: ke[email p o ec ed]
1877-0509 c
2018 The Au ho s. Published by Else ie L d.
This is an open access a icle unde he CC BY-NC-ND license (h p://c ea i ecommons.o g/licenses/by-nc-nd/3.0/).
2Ke in And ews e al. /P ocedia Compu e Science 00 (2018) 000–000
e alua ion o ac o s, bo h objec i e as well as subjec i e, o in e es om indi iduals. The e o e, assessmen ools a e
o en applied in s udies and, hence, being sui able o he ope a ionaliza ion o hese ac o s [5].
Fo many domains, such as educa ional o social science, which o en use o dinal measu emen s, ope a ionaliza ion
is essen ial, since ac o s a e s ic ly de ined as measu able a iables [1]. In he ield o educa ional science, assessmen
ools enable esea che s o c ea e assessmen asks o de e mine whe e indi iduals (i.e., lea ne s) a e on he con inuum
o lea ning [10]. Fu he mo e, insigh s can be ob ained abou knowledge, skills, and concep s he lea ne has mas e ed
and wha may ha e been misunde s ood o no ye mas e ed [3]. In his con ex , pa icula emphasis can be pu on he
•imp o emen o s a is ical and empi ical e alua ions,
•ga he ing o addi ional insigh s abou lea ning s eng hs and challenges,
•moni o ing o lea ne p og ess,
•indi idualiza ion o ins uc ional quali y o inc ease hei e ec i eness.
Conside ing hese emphases, he wo k a hand p esen s an assessmen ool o sma mobile de ices de eloped in he
con ex o a p ojec in es iga ing isual li e acy in cul u al educa ion. S udies can be easily designed and conduc ed
wi h he ool and adop ions can be made wi hou p og amming a line o code. Hi he o, he assessmen ool has been
success ully applied o o e 500 pa icipan s.
The emainde o his pape is s uc u ed as ollows: Sec ion 2 explains he p ojec con ex . In Sec ion 3, he assessmen
ool design is p esen ed, ollowed by he use case desc ibed in Sec ion 4. Sec ion 5 discusses ela ed wo k and, inally,
Sec ion 6 summa izes he pape and gi es an ou look.
2. Con ex o he P ojec
The p esen ed assessmen ool was de eloped in he con ex o a coope a i e p ojec ocusing on isual li e acy in
cul u al educa ion1. The pu pose o his p ojec is o gain insigh s on how isual li e acy can be os e ed, ope a ional-
ized, and, hence, empi ically measu ed. The eby, isual li e acy is a concep ha de ines he capabili y o in e p e ,
unde s and, and ex ac in o ma ion p esen ed in images [17]. Fu he mo e, s udies ha e shown ha he use o ap-
p op ia e images mo i a es lea ning p ocesses as well as p omo es he de elopmen o lea ning s a egies. Based on
his, he p ojec analyzes he in e - (i.e., be ween indi iduals) and in a-indi idual (i.e., wi hin indi idual) a ia ion
o mo i a ion in lea ne s such as pupils, s uden s, and eache s. This includes, o example, he pe cep ion o ins uc-
ional quali y wi h ega d o isual li e acy. The eby, a pa icula emphasis is pu on he e ec s o cul u al capi al (i.e.,
social asse s like educa ion and pe sonali y) as well as cul u al-aes he ic p ac ices (e.g., eligious belie s) o lea ne s.
Cul u al educa ion is assessed wi hin he Eu opean F amewo k o Visual Li e acy (ENViL) in a se ies o na ionwide
s udies ac oss a ious kinds o schools (e.g., seconda y school and uni e si y) in Ge many [18]. To help conduc he
s udy, an assessmen ool o sma mobile de ices was de eloped o assess he isual li e acy in cul u al educa ion.
Lea ne s Wi h
Di e se Cul u al
Capi al
Lea ne s Wi h
Di e se Cul u al
Capi al
Collec ion o
Ques ions Collec ion o
Visual Tasks
Va ie y o Ques ions and Visual Tasks
Addi ional Da a
ia Eye T acking
Fig. 1. Visual Li e acy in Cul u al Educa ion
Speci ically, pa icipan s (i.e., lea ne s) o he s udies we e asked o
comple e a a ie y o isual asks (e.g., Comple ing Pic u es and Men-
al Ro a ion) and answe a se o ques ions, while he assessmen ool
cap u ed ele an in o ma ion [8]. Mo eo e , p ima ily o s eng hen
he con e gen alidi y (i.e., measu es whe he applied cons uc s a e
ela ed), addi ional da a will be collec ed ia eye acking. In pa icu-
la , wi h e ospec i e hink-aloud (RTA) s udies and pos expe ience
eye acking p o ocols (PEEP), an endea o is o ga he addi ional in-
sigh s abou di e ences be ween no ices and expe s wi h ega d o i-
sual li e acy [7]. The objec i e is o gain well- ounded insigh s on how
ins uc ions should be shaped o os e lea ne mo i a ion and isual
li e acy. These insigh s will be ed back o he pa icipa ing lea ne s
and he knowledge ob ained will be implemen ed in ins uc ions a e he p ojec e m.
1h ps://www.dip .de/en/ esea ch/cu en -p ojec s/bkkb- isual-li e acy-in-cul u al-educa ion?se _language=en
Ke in And ews e al. /P ocedia Compu e Science 00 (2018) 000–000 3
3. Mobile Assessmen Tool Design
The mobile assessmen ool p esen ed in his pape is a highly ex ensible and con igu able amewo k o able -
based isual asks. I o e s a a ie y o i ems, ca ego ized in o isual asks and ques ions ha pa icipan s may com-
ple e. The isual asks ange om simple ones, such as so ing objec s and iden i ying small de ails in la ge images
(c . Fig. 2), o complex ones, such as c ea ing a 2D image om a 3D image and a gi en pe spec i e (c . Fig. 12). As
i was c ea ed speci ically o conduc ing isual li e acy s udies on pupils, he assessmen ool had o be buil om
sc a ch o suppo no only he exac s udy equi emen s, bu also he addi ional equi emen s imposed when dealing
wi h pupils. As mos o he assessmen akes place in class oom se ings wi h as many as 30 pupils and only one o
wo supe iso s, he pupils ha e o be comple ely blocked om pe o ming ac ions ou side o he in ended scope.
In consequence, he use in e ace o he en i e applica ion is s ipped down o an absolu e ba e minimum wi h no
con igu a ion o na iga ion op ions isible o use s.
Fig. 2. Visual Task “Iden i y De ails”
One o he main challenges when implemen ing he assess-
men ool was c oss pla o m compa ibili y, as he a ge ha d-
wa e (Windows, And oid, o iOS) had no been inally speci ied
when he de elopmen s a ed. Fu he mo e, ou in en was o
c ea e a highly eusable amewo k ha could be u ilized o a -
ious s udies, po en ially conduc ed on o he able , o e en desk-
op ha dwa e. In any o he si ua ion, a simple way o ackling
his challenge would be o implemen he i ems u ilized in he
s udy as a web applica ion. Howe e , as mos b owse s do no
allow o locking use s ou om basic na iga ion unc ionali y
and he p ima y use case is conduc ing s udies wi h pupils, who
a e o en unp edic able o e en mischie ous by na u e, we we e
o ced o implemen he mobile assessmen ool as a c oss pla -
o m mobile applica ion. This allows o ull con ol o e he displayed isual elemen s and na iga ion con ols. An
example o one o he isual asks used in he s udy, p esen ed in he e y es ic i e use in e ace s yle necessa y o
dealing wi h child en, is shown in Fig. 2. To keep c oss pla o m compa ibili y, in combina ion wi h ull use in e ace
con ol, we op ed o he Xama in Fo ms amewo k. Xama in Fo ms allows us o euse almos all code, including he
use in e ace, ac oss Windows, And oid, and iOS de ices, while s ill o e ing access o ad anced ea u es like sc een
eco ding o ouch inpu logging, which a e u ilized in some o he games o collec ing addi ional da a.
Howe e , implemen ing he assessmen ool as a na i e o c oss pla o m applica ion, ins ead o a web applica ion,
con lic ed wi h ano he equi emen ha he ool shall be ex ensible and highly con igu able wi hou ha ing o upda e
he applica ion on all 30 able s e e y ime unc ional changes a e made o indi idual i ems. Such changes, e.g.,
exchanging an image o changing he o de in which he i ems a e p esen ed o s udy pa icipan s, can be done
cen ally i he able s a e jus displaying a web si e om a se e . Howe e , i each able has he applica ion ins alled
locally, one has o upda e all able s h ough sideloading o e e y unc ional change. Un o una ely, one can no use
he au o-upda ing capabili ies o he a ious app s o es, as hey do no allow apps wi h ad anced ea u es, such as
sc een eco ding, which a e equi ed o collec ing he addi ional da a equi ed by ou use case.
To ackle hese issues, we implemen ed a e y simple se e in ASP.NET Co e, a ligh weigh and c oss pla o m C#
amewo k ha allows o hos ing REST web se ices and iles in a b owseable olde s uc u e. The se e has only
wo in e aces, one ha sends a lis o all iles i is hos ing o he eques ing able and ano he o ecei e he esul da a
gene a ed by pa icipan s comple ing he a ious i ems, which we will de ail subsequen ly. This allows us hos ing all
unc ional esou ces on he se e , including all ex s and images, allowing hem o be upda ed o exchanged quickly
in one cen alized loca ion. When he applica ion is s a ed on he a ious able s, hey each eques he comple e
ile lis om he se e ia a REST call, and s a p eloading he da a o a oid communica ion o e head du ing he
assessmen . The se e ollows a ligh weigh s yle and can be hos ed on any ope a ing sys em on a lap op o simila
de ice when conduc ing an assessmen in he ield.
As each o he isual asks can ha e mul iple a ian s, e.g., di e en images wi h a ying di icul y le els o he
ask shown in Fig. 2, he se e has o ha e an ex ensi e olde s uc u e wi h all images, ex s, and o he iles so ed
by ask and a ian . An example o one such olde o a speci ic a ian o one o he isual asks is shown in Fig. 3.
4Ke in And ews e al. /P ocedia Compu e Science 00 (2018) 000–000
Fig. 3. Visual Task Va ian Folde Fig. 4. I em Sequence Con igu a ion
The con en s o his olde , which cons i u e all he da a ha he a ian “exhibi ion” o he ask “iden i y de ail”
equi es o un, can be edi ed by manipula ing he iles on he se e . As p e iously men ioned, he applica ion will
eload all iles om all o hese olde s di ec ly a e i is opened on he able s, wi hou ha ing o upda e he app
package. The se ings.cs ile p esen in he olde can be used o change con igu a ion op ions o he a ian , such as
ime limi s o ins uc ion ex s. CSV (Comma Sepa a ed Values) iles can be in e p e ed as ables by Excel, allowing
o simple edi ing o se ings. This makes CSV iles ideal o edi ing by esea che s, as hey do no need p og amming
expe ience o ools o mo e specialized ile o ma s, such as XML.
Fu he mo e, CSV iles can also be used o de ine he sequence o i ems which a e p esen ed o pa icipan s, as
shown in Fig. 4. The example game sequence con igu a ion ins uc s he applica ion o i s display an eye acking
calib a ion page, nine a ian s o a isual ask called “dalli klick”, ano he calib a ion page, and, inally, wo a ian s
o a ask called “ anking” in he exac o de o he ows. As hese con igu a ion iles a e downloaded om he se e
a applica ion s a , he sequence o i ems can be changed cen ally on he se e and is e ec i e immedia ely a e
e-opening he able applica ion. To acili a e his, we build a s ack o he i em sequence om he ows o he CSV
ile shown in Fig. 4a applica ion s a up. This s ack con ains all in o ma ion necessa y o load a speci ic i em, i.e.,
he class name con aining he logic o he isual ask and he name o he olde on he se e ha con ains he da a o
he speci ied ask a ian . In pa icula , his allows esea che s o add a new i em in he ield by adding ano he a ian
o an exis ing isual ask. This can be achie ed by c ea ing a olde wi h he images and se ings o he a ian (c .
Fig 3) as a sub- olde o he isual ask on he se e and adding he name o he new sub- olde in a new ow o he
i em sequence con igu a ion CSV ile (c . Fig. 4). When he able applica ion is wo king h ough he s ack o i ems
and eaches a new isual ask a ian , i can be displayed seamlessly, as he images and ex s in he a ian olde we e
p eloaded om he se e on applica ion s a up. Clea ly, all a ian s o a isual ask mus use he same code.
Addi ionally, some i ems ha e special con igu a ion iles ha allow o g ea e cus omiza ion o he isual ask and
i s a ian s. An example o one such con igu a ion ile, o he pic u e e ealing ask “Dalli” (c . Fig. 5), in which
an image ge s e ealed o e ime, is shown in Fig. 6. The ma ix con igu a ion ile de ines he o de in which he
iles co e ing he pic u e a e emo ed. Again, as his is edi ed cen ally on he se e and applied o all able s on
applica ion es a , his allows o con igu a ion o un- ime logic wi hou ouching code o he applica ion deployed
on he able s, a huge bene i o esea che s accus omed o Excel, especially when wo king in he ield.
Fig. 5. Visual Task “Dalli”
Fig. 6. “Dalli” Ma ix Con igu a ion
Ke in And ews e al. /P ocedia Compu e Science 00 (2018) 000–000 5
Fig. 7. “Iden i y De ails” Solu ion Con igu a ion
Fig. 8. “Iden i y De ails” Resul s File
Fig. 9. Touch T acking Log
“MB-1 exhibi ion.cs ”
Ha ing discussed he con igu a ion o he i ems and he sequence in which hese a e p esen ed o s udy pa icipan s,
we u n o he ou pu ha is p oduced while he i ems a e being comple ed. As he goal is o allow o mos ly au oma ed
e alua ion o esul s, some i ems need addi ional con igu a ion o de e mine which use inpu cons i u es a “co ec ”
solu ion. An example o he solu ion con igu a ion o he “Iden i y De ails” isual ask is shown in Fig 7. The
con igu a ion ile allows o he speci ica ion o pai s o X/Y coo dina es and a adius a ound hese ha he applica ion
should a e as a co ec solu ion when pa icipan s ma k hem du ing he assessmen (c . Fig. 2).
An exce p o he co esponding esul s ile ha is c ea ed on he se e a e he isual ask was comple ed by
wo di e en pa icipan s, wi h he codewo ds MB-1 and MB-2, is shown in Fig. 8. The alues o C1, C2, and C3,
i.e. whe he he pa icipan ci cled he i s , second, and hi d de ail a ea co ec ly, a e au oma ically calcula ed by
he applica ion wi h he help o he con igu a ion ile. Fu he mo e, as each ow ep esen s a single assessmen o
one pa icipan , addi ional in o ma ion, such as a ouch acking log (c . Fig. 9), can be e e enced in each ow in he
FileName column. These addi ional logs can be me ged wi h he main esul s ile in ools such as SPSS o Ma lab o
c ea e hea maps and s a is ical g aphs o all collec ed esul s.
The inal assessmen capabili y we wish o highligh is he p esen a ion o ex ual ques ion i ems. C ea ing a gene ic
me hod o p esen ing a ques ion i em as an a bi a ily sized able o mul iple choice ques ions and answe s, as shown
in Fig. 10, p o ed o be qui e a challenge. To keep in line wi h he con igu a ion s yle o he isual asks, each ques ion
i em is con igu ed by h ee sepa a e CSV con igu a ion iles, one con aining he ques ions, one con aining he possible
answe s, and one con aining he addi ional desc ip i e ex s. As he ho izon al sc een space o he able can be used
up quickly by long ques ions in he esul ing able columns, we had o de elop a simple algo i hm o scaling down he
on size o he longes answe s dynamically, and inse ing line b eaks be o e he on size eaches an un eadable size.
This way, hese aspec s, i.e., on size and line b eaking, do no ha e o be con igu ed by he esea che s designing he
ques ion i ems, simpli ying hei wo kload signi ican ly. Fu he mo e, as we ha e an a bi a y numbe o ques ions
and answe s belonging o a single ques ion i em, he design o he esul s ile o he ques ion i ems was challenging
as well. As, a he end o an assessmen , he single esul s ile o he ques ion i ems should con ain he answe s o
all pa icipan s o all indi idual ques ions, we op ed o ea each ques ion i em as a a ian o a base “ques ion” ask.
This enables a s yle consis en wi h he esul iles o he isual asks (c . Fig. 8), wi h all answe s gi en by a single
pa icipan conce ning a ques ion i em w i en in o he columns o a single ow (c . Fig. 11).
Fig. 10. Ques ion I em
Fig. 11. Ques ion I em Resul s File (Mul iple Columns Hidden)

6Ke in And ews e al. /P ocedia Compu e Science 00 (2018) 000–000
I em esul s, such as he ones shown in Fig. 8, Fig. 9, o Fig 11, a e gene a ed by he applica ion unning on he
a ious able s du ing he assessmen . Howe e , as p e iously men ioned, he able s a e no gua an eed o ha e an
In e ne connec ion in he ield, so uploading hese esul s o a se e on he In e ne is no easible. Fu he mo e,
ga he ing all he Excel esul iles s o ed locally on he able s would be an ex emely edious ask. Ins ead, he able s
use he same REST web se ice used o p eloading he da a and se ings iles on applica ion s a up o pe iodically
ans e esul s o he se e . Addi ionally, he usage o WiFi o ans e ing game esul s allows o he ans e o
la ge supplemen al iles, such as he images p oduced when playing ano he one o he games, “Iden i y Pe spec i e
O ien a ion”, shown in Fig. 12. The game p o ides he s udy pa icipan s wi h a pale e o colo s, which hey ha e o
use o c ea e a 2D image o a 3D model om a ce ain pe spec i e. This game ans e s no only he Excel esul s iles
wi h he usual s a is ics on ime aken, ouch poin s, e c., bu also an image ile o he solu ion, which can be used o
manual e alua ion la e on. An abs ac schema ic o he communica ion pa e n be ween he applica ion and se e
pa s o he mobile assessmen ool is shown in Fig. 13.
No e ha a di ec WiFi connec ion be ween he able s hos ing he applica ion and he lap op ac ing as he se e
may no always be possible as i is shown in Fig. 13. This is due o he ac ha he Windows d i e s o many popula
WiFi chipse s only suppo ad-hoc WiFi connec ions wi h up o eigh de ices, i a all. To alle ia e his issue, we
allow o he u iliza ion o an addi ional WiFi access poin o ho spo , which all able s and he se e can connec o.
Fu he mo e, i is possible o connec he able s and he se e o any on-p emise WiFi, e ec i ely emo ing ange
es ic ions be ween he able s and he se e . This enables unning mul iple assessmen s in di e en pa s o he same
building concu en ly wi h only one lap op ac ing as a se e , as long as all pa icipa ing de ices a e on he same local
WiFi ne wo k. To his end, he able s show a simple IP Add ess inpu dialog on s a up i hey canno connec o hei
las known se e IP add ess.
Finally, in he e en ha he WiFi becomes conges ed o b eaks down, ei he du ing he p eloading phase when he
applica ion is s a ed on he able s o du ing he assessmen phase when lea ne s a e comple ing he isual asks and
esul s a e being sen o he se e , he e a e sa egua ds in place o p e en he applica ion om mal unc ioning. On
one hand, iles ha a e no ans e ed success ully in he p eloading phase a e au oma ically downloaded om he
se e once hey become necessa y o he execu ion o an i em. On he o he , esul s ha ail o ans e o he se e
a e cached locally on he able and e- ansmi ed once he WiFi connec ion becomes a ailable again. Theo e ically,
i he p eloading phase is comple ed success ully, he en i e assessmen can be comple ed wi hou a WiFi connec ion
as he able s ha e enough memo y o cache he mos ly ex -based assessmen esul s o long pe iods o ime.
In summa y, his means ha he goal o c ea ing a mobile assessmen ool ha is con igu able by non-p og amme
domain expe s in he ield was achie ed. Fu he mo e, h ough ou unique a chi ec u e, u ilizing a p eloading phase
o all images, ex s, and con igu a ion iles on applica ion s a up, comple e econ igu a ion o all assessmen i ems is
possible in a cen alized loca ion, equi ing only simple sp eadshee edi ing so wa e such as Excel. Finally, in con as
o a web applica ion, ou applica ion has he ad an age o p e en ing s udy pa icipan s om exi ing he assessmen
applica ion by na iga ing o o he websi es o changing se ings in he b owse , which is c ucial when wo king wi h
g oups o younge pupils in class oom se ings.
Fig. 12. Visual Task “Iden i y Pe spec i e O ien a ion”
Se e
Table 1
P eload Da a on S a up
Upload Addi ional Files
Upload Excel Resul s
Table nTable 2
Fig. 13. Mobile Assessmen Tool A chi ec u e
Ke in And ews e al. /P ocedia Compu e Science 00 (2018) 000–000 7
4. Use Case
As desc ibed in Sec ion 2, he assessmen ool was de eloped in he con ex o a p ojec ocusing on he isual
li e acy in cul u al educa ion and has been success ully es ed in p ac ice. So a , a o al o 547 pa icipan s we e
success ully assessed by ou pa ne s in Cologne and F ank u am Main. The long- e m objec i e is o collec da a
om o e 800 pa icipan s. The pa icipan s a e ec ui ed in 20 di e en Ge man schools, loca ed in 4 ede al s a es
o Ge many, i.e., No h Rhine-Wes phalia, Hesse, Schleswig Hols ein, and Ba a ia (in planning). The eby, he assess-
men en i onmen is p e alen ly he isual a s class oom. I a isual a s class oom is no a ailable, he assessmen is
conduc ed in a no mal class oom. Fu he , he la ges assessmen class con ains 30 pa icipan s and, on he o he , he
smalles one comp ises 12 pa icipan s.
In gene al, he assessmen ool comp ises 108 i ems ca ego ized in o isual asks and ques ions. In pa icula , 19 i-
sual asks co e 61 i ems (i.e., each a ian o a ask ep esen s an i em) and 47 i ems a e ela ed o ques ions (i.e.,
each ques ion ep esen s an i em). Fu he mo e, a pape -based ques ionnai e, no co e ed by he assessmen ool, o
pa en s is handed ou o he pa icipan s and needs o be answe ed by hei pa en s a home.
The p ocedu e o he s udy is as ollow (c . Fig. 14): Gene ally, all pa icipan s ge an in oduc ion be o e he assess-
men . In mo e de ail, one pa icipan o he class eads he in oduc ion ex aloud in on o all pa icipan s in he
class. A e wa ds, he pa icipan s need o wo k on a p ede ined sequence o isual asks ela ed o isual li e acy and
answe a numbe o ques ions abou pe sonal in o ma ion and cul u al capi al. The eby, he pa icipan s a e asked o
pe o m he assessmen a hei own pace. All eme ging da a is s o ed anonymously. Mo eo e , he pa icipan s need
o answe all ques ions hones ly and may ask ques ions a any ime i he e a e any misunde s andings o ambigui-
ies. The du a ion o he assessmen is app oxima ely 40 minu es. A e comple ing all isual asks and answe ing all
ques ions, he pa icipan s a e able o pe o m a d awing ask as many imes as desi ed o he assessmen o c ea i e
image y abili ies in he la e analysis. Fig. 14 illus a es he used s udy design. A e comple ion, all da a is uploaded
o he se e (c . Fig. 11). Besides he expec ed da a o 800 pa icipan s, in addi ion, i is planned o conduc an in-
s uc ional s udy du ing he isual a s class (i.e., ambulan assessmen ). The e o e, he applica ion will be andomly
s a ed du ing he class. Fu he mo e, speci ic isual asks a e de eloped in such a way ha addi ional da a can be
ga he ed by means o eye acking. Acco dingly, eye acking da a will be u ilized o RTA as well as PEEP s udies
showing he playbacks o eye mo emen s in uncued o cued s imuli (i.e., images).
De ined Sequence o Ques ionnai es
and Visual Tasks
….
Lea ne s D awing Task Du a ion o Comple ion:
App ox. 40 Minu es
Fig. 14. P ocedu e o he S udy
5. Rela ed Wo k
The ope a ionaliza ion o assessmen ools in he con ex o sma mobile de ices has been add essed by a ious
o he esea ch wo ks. Fo example, he au ho s o [15] demons a e how domain expe s can be empowe ed when
implemen ing obus sma mobile da a collec ion applica ions using a model-d i en amewo k ela ed o end-use
p og amming. In u n, [14] p esen s he g aphical domain-speci ic language MAML, which enables he au oma ic
gene a ion o na i e mobile business applica ions. Mo eo e , an app oach o he de elopmen o mobile applica ions
in he con ex o heal h ca e o ganiza ion is in oduced in [11]. Wi h a s onge in ol emen o clinicians, a me hod-
ology o he c ea ion o educa ional applica ions o sma mobile de ices is discussed in [20]. Wi h a ocus pu on
mobile lea ning, wi hou being limi ed by ime and space, [9] illus a es he bene i s o lea ning ac i i ies using appli-
ca ions on sma mobile de ices. In he con ex o usabili y s udies on sma mobile de ices, a amewo k emphasizing
he ad an age o s udy design on such de ices is p esen ed in [4]. Howe e , a con igu able assessmen ool as shown
he e, especially in he con ex o educa ional s udies, was no p esen ed by o he wo ks so a .
8Ke in And ews e al. /P ocedia Compu e Science 00 (2018) 000–000
6. Summa y and Ou look
This pape ga e insigh s in o he a chi ec u e and amewo k o an assessmen ool de eloped in he con ex o a
p ojec ocusing on isual li e acy in cul u al educa ion o sma mobile de ices. Fu he mo e, he success ul applica-
ion o he assessmen ool in a la ge-scale educa ional s udy wi h mo e han 500 pa icipan s so a is shown he ein.
Cu en ly, he assessmen o u he pa icipan s wi h he p esen ed ool is in p og ess wi h he objec i e o assess a
leas 800 pa icipan s. Mo eo e , addi ional e ospec i e hink-aloud as well as pos expe ience eye acking p o o-
cols s udies a e planned o he u u e wi h he ea u es and suppo ha a e al eady ealized in he assessmen ool.
Due o he ac ha he assessmen ool, including i s a chi ec u e and employed amewo k ha e been de eloped in a
c oss-pla o m and c oss-de ice ashion, i can be e-used o simila s udies in he isual li e acy o simila con ex s.
Fu he mo e, he s udy design (i.e., he sequence o isual asks and ques ions) can be indi idually cus omized. Fi-
nally, ex ensions and adap a ions can be implemen ed in he assessmen ool ollowing eme ging equi emen s wi hou
he need o p og amming.
Acknowledgmen
The p ojec BKKB is unded by he Bundesminis e ium ¨
u Fo schung und Bildung (BMBF).
Re e ences
[1] Amo, C., Cousins, J. B., 2007. Going Th ough he P ocess: An Examina ion o he Ope a ionaliza ion o P ocess Use in Empi ical Resea ch
on E alua ion. New Di ec ions o E alua ion (116), 5–26. 1
[2] Bo u , J. T., S o ie, D., 2014. Mobile De ices in Medicine: A Su ey o How Medical S uden s, Residen s, and Facul y use Sma phones and
o he Mobile De ices o Find In o ma ion. Jou nal o he Medical Lib a y Associa ion: JMLA 102 (1). 1
[3] Cobc o , R. S., Towe s, S. J., Smi h, J. E., B uns, A., 2006. Mobile Lea ning in Re iew: Oppo uni ies and Challenges o Lea ne s, Teache s,
and Ins i u ions. Online Lea ning and Teaching, 21–30. 1
[4] Cou sa is, C. K., Kim, D. J., 2011. A Me a-Analy ical Re iew o Empi ical Mobile Usabili y S udies. Jou nal o Usabili y S udies 6 (3),
117–171. 5
[5] Dea nley, C., Taylo , J., Hennessy, S., Pa ks, M., Coa es, C., Haigh, J., Fai hall, J., Riley, K., D ans ield, M., 2009. Using Mobile Technologies
o Assessmen and Lea ning in P ac ice Se ings: Ou comes o Fi e Case S udies. In e na ional Jou nal on E-Lea ning 8 (2), 193–207. 1
[6] El-Hussein, M. O. M., C onje, J. C., 2010. De ining Mobile Lea ning in he Highe Educa ion Landscape. Jou nal o Educa ional Technology
& Socie y 13 (3), 12–21. 1
[7] F eeman, B., 2011. T igge ed Think-Aloud P o ocol: Using Eye T acking o Imp o e Usabili y Tes Mode a ion. In: P oceedings o he SIGCHI
Con e ence on Human Fac o s in Compu ing Sys ems. pp. 1171–1174. 2
[8] G een ield, P. M., 2009. Technology and In o mal Educa ion: Wha is Taugh , Wha is Lea ned. Science 323 (5910), 69–71. 2
[9] Hwang, G.-J., Chang, H.-F., 2011. A o ma i e assessmen -based mobile lea ning app oach o imp o ing he lea ning a i udes and achie e-
men s o s uden s. Compu e s & Educa ion 56 (4), 1023–1031. 5
[10] Hwang, G.-J., Tsai, C.-C., 2011. Resea ch T ends in Mobile and Ubiqui ous Lea ning: A Re iew o Publica ions in Selec ed Jou nals om
2001 o 2010. B i ish Jou nal o Educa ional Technology 42 (4), 65–70. 1
[11] Khamba i, A., G undy, J., Wa en, J., Hosking, J., 2008. Model-D i en De elopmen o Mobile Pe sonal Heal h Ca e Applica ions. In: P oc
In ’l Con on Au oma ed So wa e Enginee ing. IEEE Compu e Socie y, pp. 467–470. 5
[12] Kim, S., Manko , J., Paulos, E., 2013. Sens : E alua ing a Flexible F amewo k o Au ho ing Mobile Da a-Collec ion Tools o Ci izen
Science. In: P oceedings o he 2013 con e ence on Compu e suppo ed coope a i e wo k. ACM, pp. 1453–1462. 1
[13] Lane, N. D., Miluzzo, E., Lu, H., Peebles, D., Choudhu y, T., Campbell, A. T., 2010. A Su ey o Mobile Phone Sensing. IEEE Communica ions
Magazine 48 (9), 140–150. 1
[14] Riege , C., Kuchen, H., 2018. A P ocess-O ien ed Modeling App oach o G aphical De elopmen o Mobile Business Apps. Compu e
Languages, Sys ems & S uc u es 53, 43–58. 5
[15] Schobel, J., P yss, R., Schickle , M., Ru -Leuschne , M., Elbe , T., Reiche , M., 2016. End-Use P og amming o Mobile Se ices: Empow-
e ing Domain Expe s o Implemen Mobile Da a Collec ion Applica ions. In: In ’l Con on Mobile Se ices (MS). IEEE, pp. 1–8. 5
[16] Schobel, J., P yss, R., Wipp, W., Schickle , M., Reiche , M., 2016. A Mobile Se ice Engine Enabling Complex Da a Collec ion Applica ions.
In: In e na ional Con e ence on Se ice-O ien ed Compu ing. Sp inge , pp. 626–633. 1
[17] S okes, S., 2002. Visual Li e acy in Teaching and Lea ning: A Li e a u e Pe spec i e. E J o he In o Tech in Edu 1 (1), 10–19. 2
[18] Wagne , E., Sch¨
onau, D., 2016. Common Eu opean F amewo k o Re e ence o Visual Li e acy - P o o ype. Waxmann. 2
[19] Wu, W.-H., Wu, Y.-C. J., Chen, C.-Y., Kao, H.-Y., Lin, C.-H., Huang, S.-H., 2012. Re iew o T ends om Mobile Lea ning S udies: A Me a-
Analysis. Compu e s & Educa ion 59 (2), 817–827. 1
[20] Zhang, M., Cheow, E., Ho, C. S., Ng, B. Y., Ho, R., Cheok, C. C. S., 2014. Applica ion o Low-Cos Me hodologies o Mobile Phone App
De elopmen . JMIR mHeal h and uHeal h 2 (4). 5