1
MESTRADO INTEGRADO EM MEDICINA
Ins i u o Ciências Biomédicas Abel Salaza
Uni e sidade do Po o
2013/2014
Analysis o a no el e-lea ning sys em in measu ing medical s uden s
ou comes in s udy ime and ecall accu acy
RUI ALEXANDRE DO PRADO COSTA
2
Rui Alexand e do P ado Cos a
Analysis o a no el e-lea ning sys em in measu ing medical s uden s
ou comes in s udy ime and ecall accu acy
Mes ado In eg ado em Medicina
Á ea: Educação Médica
3
Analysis o a no el e-lea ning sys em in measu ing medical
s uden s ou comes in s udy ime and ecall accu acy
Tiago Ta ei a-Gomes*1,2
Rui P ado Cos a*2,3
Mil on Se e o1
Ma ia Amélia Fe ei a1
1Cen e o Medical Educa ion, Facul y o Medicine o he Uni e si y o Po o, PT
2ALERT Li e Sciences Compu ing SA
3Abel Salaza Biomedical Sciences Ins i u e, Uni e si y o Po o
Email: Tiago Ta ei a-Gomes*- iago. a ei [email protected]; Rui P ado Cos a –
cos [email protected]
*Co esponding au ho s
Abs ac
Backg ound: Spaced- epe i ion and epea ed es ing wi h eedback, a e wo me hodologies
known o boos knowledge e en ion. ALERT STUDENT, an online pla o m o medical
educa ion, was de ised o implemen bo h app oaches in an in eg a ed ashion and p omo e
esea ch in his ield, by allowing c ea ion and dis ibu ion o lea ning objec s named lashca ds.
This s udy aims o demons a e how lea ning objec s and epea ed es ing using open-ended
ques ions can be b ough oge he o p o ide insigh s on ecall accu acy, and o cha ac e ize
he ex en o which s uden ac o s, con en ac o s and epe i ion a ec i .
Me hods: Th ee in-pe son sessions (s0, s1 and s2), sepa a ed by one-week in e als we e
conduc ed wi h 96 medical s uden s using he pla o m. S uden s we e andomized in o
expe imen and con ol g oups using simple andomiza ion. The s udy consis ed on a 20 min
s udy ask using 27 lashca ds abou he Golgi complex, and on a sel -assessmen ask, ha
consis ed on answe ing lashca d open-ended ques ions and g ading ecall accu acy. On s0
bo h g oups pe o med he sel -assessmen ask. On s1 and s2, he expe imen g oup
pe o med he s udy ask ollowed by he sel -assessmen ask, whe eas he con ol g oup only
pe o med he sel -assessmen ask.
4
Resul s: A s0 he e we e no di e ences in ecall accu acy be ween g oups, bu in s1 and s2
ecall accu acy di e ences we e s a is ically signi ican . The expe imen g oup achie ed a
sha pe inc ease in ecall accu acy han he con ol g oup, which was s onge be ween s0 and
s1. In he expe imen g oup, inc eases in ecall accu acy we e mainly due o he session e ec
(49.6%) ollowed by lashca d ac o s (15.3%) and s uden ac o s (5.1%). In he con ol g oup,
inc eases in ecall accu acy we e mainly accoun ed by lashca d ac o s (34.7%), ollowed by
s uden (15.9%) and session (8.2%) ac o s. Gene alizabili y sco e o he lashca d in he
con ol g oup was 0.91, indica ing almos pe ec ag eemen , and 0.47 o he expe imen
g oup, indica ing mode a e ag eemen . This inding indica es ha lashca ds can be e y well
cha ac e ized by ecall accu acy sco es by ha ing he same s uden s pe o m sequen ial sel -
assessmen asks alone. Rega ding he expe imen g oup, di e ences in s udy du a ion o
sessions s1 and s2 we e s a is ically signi ican . Mean s udy du a ion was 14:32 minu es o s1
and 12:28 minu es o s2.
Conclusions: The sel -assessmen ac i i y alone had li le e ec on ecall accu acy, howe e
s udying and sel - es ac i i ies oge he g ea ly inc eased ecall accu acy. The session e ec
was he main de e minan o ecall accu acy o he expe imen g oup. Lea ning objec s may
be able o p o ide addi ional alue when combined wi h open-ended ques ions. Insigh s abou
ecall accu acy can be use ul o eache s o assess con en di icul y, lea ning a e, and o
s uden s o p io i ize con en and make be e use o s udy sessions.
Keywo ds
Medical educa ion, Memo y e en ion, Compu e -Assis ed Ins uc ion, E-lea ning, Tailo ed-
lea ning, Spaced epe i ion
5
Backg ound
Medical educa ion is a complex ield whe e upda es in medical knowledge, educa ional
echnology and eaching s a egies in e wine in a p og essi e ashion [1, 2, 3, 4, 5]. O e he
pas decade he e has been a shi in his ield, whe e adi ional ins uc o -cen e ed eaching
is yielding o a lea ne -cen e ed model [6, 7, 8, 9], in which he lea ne has g ea e con ol o e
he lea ning me hodology and he ole o a eache becomes ha o a acili a o o knowledge
acquisi ion, eplacing he ole o an in o ma ion p o ide [10, 11, 12, 7].
The e is as li e a u e ega ding he applica ion o educa ional
s a egies [13, 14, 15, 16, 17, 18], ins uc ional design [19, 11, 20, 21, 22, 23] and cogni i e
lea ning science [24, 25, 26, 27] o he ield o medical educa ion. Two app oaches ha eme ge
om ha li e a u e a e spaced- epe i ion and epea ed es ing.
Spaced- epe i ion is a lea ning app oach g ounded in cogni i e lea ning heo y ha en o ces
con inuously e isi ing con en o e op imized ime in e als, and has been shown o imp o e
long- e m knowledge e en ion [28, 29, 30, 11, 31].
Repea ed es ing du ing lea ning is an app oach ha ex ends spaced epe i ion by applying i
o assessmen s du ing lea ning pe iods. I a gues ha he addi ion o epea ed assessmen
eedback can u he enhance lea ning [32, 33, 34]. In addi ion, he usage o open-ended
ques ions as means o assessmen has been shown o be supe io o mul iple choice ques ions
in e ms o knowledge e en ion [35].
Despi e his as li e a u e, he e a e ew epo s o implemen ing hese p inciples in an on-line
pla o m in an in eg a ed ashion [36].
ALERT STUDENT, a pla o m o medical educa ion, was de ised wi h he goal o implemen
such p inciples and p omo e esea ch in his ield [36]. The pla o m allows c ea ion and
dis ibu ion o lea ning objec s [37] named no ebooks. These a e composed o smalle lea ning
objec s named lashca ds, sel -con ained in o ma ion chunks wi h ela ed open-ended
ques ions [36]. No ebooks can be accessed using a s udy mode ha p esen s he lashca ds
in a s udy- iendly en i onmen en iched wi h no e aking, ex highligh ing and ime acking
ea u es. The quiz mode is a complemen a y en i onmen whe e lashca d knowledge can be
sel -assessed using i s open-ended ques ions. A se o ques ions om each lashca d is
sequen ially p esen ed and equi es he s uden o ac i ely ecall he answe and g ade ecall

6
accu acy using a 4 poin like scale. This in o ma ion is summa ized in o a 4 poin colo -coded
sco e ha is a ibu ed o each lashca d and displayed in he s udy mode as a s udy p io i y
cue.
Explo ing he use ulness o p o iding knowledge e en ion ools in e-lea ning sys ems is an
impo an con ibu ion o in o ma ion managemen axis o he co e-compe ences o medical
educa ion [38] and may p o ide a means o a iche and as e acquisi ion o ac ual knowledge,
which is a co ne s one o he de elopmen o cogni i e schema a, clinical easoning skills [27].
This s udy aims o demons a e how lea ning objec s and epea ed es ing using open-ended
ques ions can be b ough oge he o p o ide insigh s on ecall accu acy, and o cha ac e ize
he ex en o which s uden ac o s, con en ac o s and epe i ion a ec i .
7
Me hods
S udy design
The Facul y o Medicine o he Uni e si y o Po o (FMUP) implemen s a 6-yea g adua e
p og am. Applican s a e mainly high school g adua es. The i s h ee yea s ocus on basic
sciences while he las h ee ocus on clinical special ies.
Nine y-eigh (n=98) s uden s om he 4 h and 5 h g ades o ou school we e con ac ed ia
email and olun ee ed o pa icipa e in his s udy. The s udy was pe o med in h ee in-pe son
sessions (s0, s1 and s2) o 1 hou du a ion. Sessions we e ca ied one week apa . S uden s
we e assigned in o expe imen g oup o con ol g oup using simple andomiza ion.
The s udy consis ed o a s udy ask and a sel -assessmen ask. Du ing he s udy ask,
pa icipan s s udied a no ebook con aining lashca ds abou he Golgi complex du ing 20
minu es using he s udy mode. Du ing he sel -assessmen ask, s uden s we e p esen ed
open-ended ques ions om he no ebook on he quiz mode, and we e equi ed o g ade hei
ac i e ecall o each ques ion using a 4 poin like scale (0 - no ecall, 1 - sca ce ecall, 2 -
good ecall, 3 - ull ecall). This ask las ed 15 minu es. (S udy mode and quiz mode
en i onmen sc eensho s can be iewed on appendix C)
A pilo s udy was conduc ed using wo 2nd g ade s uden s ha had ecen ly comple ed he
Cellula and Molecula Biology class, wo 4 h g ade and wo 5 h g ade medical s uden s (n =
6). S uden s pe o med he sel -assessmen ask. The 2nd g ade s uden s’ a e age ecall
accu acy was assumed o be simila o wha would be ound a session s2. The 4 h and 5 h
g ade s uden s’ mean ecall accu acy was assumed o co espond o wha would be ound a
session s0. Based on he di e ences be ween he wo pilo g oups i was concluded ha n =
48 would be su icien o disc imina e s a is ical signi ican di e ences in ecall accu acy
be ween sessions and be ween g oups. The sample size was inc emen ed o n = 98 o ake
ad an age o he s udy enue capaci y.
On s0, bo h g oups pe o med he sel -assessmen ask. On s1 and s2, he con ol g oup
pe o med he sel -assessmen ask, and he expe imen g oup pe o med he s udy ask,
immedia ely ollowed by he sel -assessmen ask.
8
Fo each session, lashca d ecall accu acy was compu ed as he mean ecall accu acy o he
ques ions belonging o a lashca d. Fo he expe imen g oup, s udy du a ion o each use and
session was compu ed as he cumula i e sum o ime spen on each lashca d o he session.
Con en design
The con en used in his s udy was designed using o icial lec u e con en p o ided by he
Cellula and Molecula Biology Depa men o ou school, ha is esponsible o eaching he
Golgi complex in Cellula Biology class as pa o he i s g ade o he school medical cu icula.
See Appendix B o access no ebook con en . The ques ions we e pa o a no ebook abou he
Golgi complex ha con ained 27 lashca ds. In o al 42 open ended ques ions abou he 27
lashca ds we e answe ed and g aded acco ding o ecall accu acy. The lea ning ma e ial did
no change be ween sessions. Du ing each session di e en ques ions we e p esen ed, bu all
o hem co e ed he no ebook ma e ial in a comple e and homogeneous ashion.
Fi ing con en o ime cons ain
Ano he pilo s udy was pe o med o ailo he no ebook con en based on he s udy ime
cons ain o 20 minu es. Ano he se o 6 5 h g ade s uden s we e assigned o ead he
no ebook con en . F om he ini ial se o 30 lashca ds, he las 3 we e emo ed so ha he
con en could be ully ead wi hin he 20 minu e in e al. No changes we e necessa y o answe
he quiz mode ques ions wi hin he 15 minu e in e al.
Sample cha ac e iza ion
In session s0 bo h g oups illed a su ey o cha ac e ize he s uden sample. Measu ed ac o s
we e gende , cou se yea , p e e ed s udy esou ce o Cellula Biology, compu e usage
habi s, Cellula Biology g ade, mean cou se g ade, and a e age s udy session du a ion du ing
he semes e and du ing he exam season.
E alua ion
Changes ac oss sessions in ecall accu acy we e examined by uni a ia e epea ed-measu es
analysis o a iance (ANOVA). Con ol and expe imen g oups we e used as be ween-subjec s
ac o . Session and lashca d we e used as wi hin subjec ac o . Repea ed con as (s0 s s1
and s1 s s2) was used o e alua e he sessions and he session in e ac ion e ec . In o de o
de e mine he eliabili y o he a iance o ecall accu acy explained by he lashca d, he
9
gene alizabili y sco e (G-Sco e) o his a iance componen was compu ed. Guidelines o
in e p e ing G-Sco es sugges ha alues o ela i e a iance be ween 0.81 - 1.00 indica e
almos pe ec ag eemen , 0.61 - 0.80 subs an ial ag eemen , 0.41 - 0.60 mode a e ag eemen ,
0.21 - 0.40 ai ag eemen , and alues less han 0.21 depic poo o sligh ag eemen [39]. The
Wilcoxon signed- ank es was pe o med o compa e s udy du a ion dis ibu ions o he
expe imen g oup o sessions s1 and s2.
This s udy was app o ed by he Facul y o Medicine Uni e si y o Po o / São João Hospi al
E hics Commi ee in compliance wi h he Helsinki Decla a ion.
16
Conclusions
The p esen s udy ocus on measu ing ecall accu acy o con en using open ended ques ions
in a con olled se ing using he ALERT STUDENT pla o m. I was ound ha sel -assessmen
ecall ac i i y alone led o a modes inc ease on ecall accu acy, and ha s udying and sel -
assessmen oge he had high impac in ecall accu acy. I was shown ha session epe i ion
was he main de e minan o ecall accu acy when s uden s pe o m s udy ollowed by he sel -
assessmen ask, and ha when employing sel -assessmen only, con en and s uden ac o s
de e mine mos o he inc ease in ecall accu acy. These insigh s may be help ul o ailo
eaching and lea ning me hodologies o be e each lea ning goals.
The p esen indings will be explo ed in mo e de ail in u u e wo k, as hey may help u u e
physicians and medical schools mee he challenge o in o ma ion managemen [38] and
ins illing a cul u e o con inuous lea ning, unde pinning he co e compe encies ou lined o XXI
cen u y physicians [4] [3].

17
Lis o abb e ia ions
ANOVA Analysis o a iance
FMUP Facul y o Medicine, Uni e si y o Po o
G-Sco e Gene alizabili y sco e
s0 S udy session 0
s1 S udy session 1
s2 S udy session 2
Compe ing in e es s
The au ho s o he s udy decla e no compe ing in e es s.
Au ho s con ibu ions
TTG and RC o mula ed he esea ch p oblem, ca ied on-si e p ocedu es, pe o med s a is ical
analysis and con ibu ed equally o his wo k. MS Valida ed s udy design and conduc ed
s a is ical analysis. MAF o e saw he s udy and ga e inal app o al. All au ho s con ibu ed o
he s udy design and d a ed he manusc ip . All au ho s ead and app o ed he inal
manusc ip .
Acknowledgmen s
The au ho s wish o hank o he s uden s who ook pa in he s udy.
18
Re e ences
1. Qiao YQ, Shen J, Liang X, Ding S, Chen FY, Shao L, Zheng Q, Ran ZH: Using cogni i e
heo y o acili a e medical educa ion. BMC medical educa ion 2014, 14:79.
2. Schoonheim M, Heyden R, Wiecha JM: Use o a i ual wo ld compu e en i onmen
o in e na ional dis ance educa ion: lessons om a pilo p ojec using Second Li e.
BMC medical educa ion 2014, 14:36.
3. F enk J, Chen L, Bhu a ZA, Cohen J, C isp N, E ans T, Finebe g H, Ga cia P, Ke Y, Kelley
P, Kis nasamy B, Meleis A, Naylo D, Pablos-Mendez A, Reddy S, Sc imshaw S,
Sepul eda J, Se wadda D, Zu ayk H: Heal h p o essionals o a new cen u y:
ans o ming educa ion o s eng hen heal h sys ems in an in e dependen wo ld.
Lance 2010, 376(9756):1923–58, [h p://www.ncbi.nlm.nih.go /pubmed/21112623].
4. Ho on R: A new epoch o heal h p o essionals’ educa ion. Lance 2010,
376(9756):1875–7, [h p://www.ncbi.nlm.nih.go /pubmed/21112621].
5. Pa el VL, Cy yn KN, Sho li e EH, Sa an C: The collabo a i e heal h ca e eam: he
ole o indi idual and g oup expe ise. Teaching and lea ning in medicine 2000,
12(3):117–32, [h p://www.ncbi.nlm.nih.go /pubmed/11228898].
6. Bahne DP, Adkins E, Pa el N, Donley C, Nagel R, Kman NE: How we use social media
o supplemen a no el cu iculum in medical educa ion. Medical eache 2012,
[h p://in o maheal hca e.com/doi/abs/10.3109/0142159X.2012.668245].
7. Ruiz JG, Min ze MJ, Leipzig RM: The impac o E-lea ning in medical educa ion.
Academic medicine : jou nal o he Associa ion o Ame ican Medical Colleges 2006,
81(3):207–12, [h p://www.ncbi.nlm.nih.go /pubmed/16501260].
8. Eysenbach G: Medicine 2.0: social ne wo king, collabo a ion, pa icipa ion,
apomedia ion, and openness. Jou nal o medical In e ne esea ch 2008, 10(3):e22,
[h p://www.pubmedcen al.nih.go /a icle ende . cgi?a id=2626430 & ool=pmcen ez &
ende ype=abs ac ].
9. Boulos MNK, Ma amba I, Wheele S: Wikis, blogs and podcas s: a new gene a ion o
Web-based ools o i ual collabo a i e clinical p ac ice and educa ion. BMC
medical educa ion 2006, 6:41,
[h p://www.pubmedcen al.nih.go /a icle ende . cgi?a id=1564136 & ool=pmcen ez &
ende ype=abs ac ].
19
10. Koops W, Van de Vleu en C, De Leng B, Oei SG, Snoeckx L: Compu e -suppo ed
collabo a i e lea ning in he medical wo kplace: S uden s’ expe iences on
o ma i e pee eedback o a c i ical app aisal o a opic pape . Medical eache
2011, 33(6):e318–23, [h p://www.ncbi.nlm.nih.go /pubmed/21609168].
11. Cook Da, Le inson AJ, Ga side S, Dup as DM, E win PJ, Mon o i VM: Ins uc ional
design a ia ions in in e ne -based lea ning o heal h p o essions educa ion: a
sys ema ic e iew and me a-analysis. Academic medicine : jou nal o he Associa ion
o Ame ican Medical Colleges 2010, 85(5):909–22,
[h p://www.ncbi.nlm.nih.go /pubmed/20520049].
12. Ch e ien KC, G eysen SR, Ch e ien JP, Kind T: Online pos ing o unp o essional
con en by medical s uden s. JAMA : he jou nal o he Ame ican Medical Associa ion
2009, 302(12):1309–15, [h p://www.ncbi.nlm.nih.go /pubmed/19773566].
13. Wol e ing V, He le A, Spi ze K, Sp eckelsen C: Blended lea ning posi i ely a ec s
s uden s’ sa is ac ion and he ole o he u o in he p oblem-based lea ning
p ocess: esul s o a mixed-me hod e alua ion. Ad ances in heal h sciences educa ion
: heo y and p ac ice 2009, 14(5):725–38,
[h p://www.ncbi.nlm.nih.go /pubmed/19184497].
14. Sanda s J, Hay ho n hwai e C: New ho izons o e-lea ning in medical educa ion:
ecological and Web 2.0 pe spec i es. Medical eache 2007, 29(4):307–10,
[h p://www.ncbi.nlm.nih.go /pubmed/17786742].
15. Giani U, B ascio G, B uzzese D, Ga zillo C, Vigilan e S: Emo ional and cogni i e
in o ma ion p ocessing in web-based medical educa ion. Jou nal o biomedical
in o ma ics 2007, 40(3):332–42, [h p://www.ncbi.nlm.nih.go /pubmed/17208055].
16. Pe ei a JA, Pleguezuelos E, Me _ A, Molina-Ros A, Molina-Tom_s MC, Masdeu C:
E ec i eness o using blended lea ning s a egies o eaching and lea ning human
ana omy. Medical educa ion 2007, 41(2):189–95,
[h p://www.ncbi.nlm.nih.go /pubmed/17269953].
17. Ruiz JG, Min ze MJ, Leipzig RM: The impac o E-lea ning in medical educa ion.
Academic medicine : jou nal o he Associa ion o Ame ican Medical Colleges 2006,
81(3):207–12, [h p://www.ncbi.nlm.nih.go /pubmed/16501260].
18. Ta adi SK, Ta adi M, Radic K, Pok ajac N: Blending p oblem-based lea ning wi h Web
echnology posi i ely impac s s uden lea ning ou comes in acid-base physiology.
20
Ad ances in physiology educa ion 2005, 29:35–9,
[h p://www.ncbi.nlm.nih.go /pubmed/15718381].
19. Conn JJ, Lake FR, McColl GJ, Bilsz a JLC, Woodwa d-K on R: Clinical eaching and
lea ning: om heo y and esea ch o applica ion. The Medical Jou nal o Aus alia
2012, 196(8):527,
[h p://www.ncbi.nlm.nih.go /pubmed/22571313h ps://www.mja.com.au/jou nal/2012/19
6/8/clinical- eaching-and-lea ning- heo y-and- esea ch-applica ion].
20. Sanda s J, La e y N: Twel e Tips on usabili y es ing o de elop e ec i e e-lea ning
in medical educa ion. Medical eache 2010, 32(12):956–60,
[h p://www.ncbi.nlm.nih.go /pubmed/21090948].
21. Pa el VL, Yoskowi z NA, A ocha JF, Sho li e EH: Cogni i e and lea ning sciences in
biomedical and heal h ins uc ional design: A e iew wi h lessons o biomedical
in o ma ics educa ion. Jou nal o biomedical in o ma ics 2009, 42:176–97,
[h p://dx.doi.o g/10.1016/j.jbi.2008.12.002h p://www.ncbi.nlm.nih.go /pubmed/1913517
3].
22. Swelle J, an Me ienboe JJ, Paas FG: Cogni i e A chi ec u e and Ins uc ional
Design. Educa ional Psychology Re iew 1998, 10(3):251–296,
[h p://doc.u wen e.nl/58655/].
23. Swelle J: Cogni i e Load Du ing P oblem Sol ing: E ec s on Lea ning. Cogni i e
Science 1988, 12(2):257–285, [h p://doi.wiley.com/10.1207/s15516709cog1202 _4].
24. D o I, Schmid P, O’conno L: A cogni i e pe spec i e on echnology enhanced
lea ning in medical aining: g ea oppo uni ies, pi alls and challenges. Medical
eache 2011, 33(4):291–6, [h p://www.ncbi.nlm.nih.go /pubmed/21456986].
25. an Me i_nboe JJG, Swelle J, Me ie JJGV: Cogni i e load heo y in heal h
p o essional educa ion: design p inciples and s a egies. Medical educa ionVan
Me i_nboe , J. J. G., Swelle , J., & Me ie, J. J. G. Van. (2010). Cogni i e load heo y in
heal h p o essional educa ion: design p inciples and s a egies. Medical educa ion, 44(1),
85–93. doi:10.1111/j.1365-2923.2009.03498.x 2010, 44:85–93,
[h p://www.ncbi.nlm.nih.go /pubmed/20078759].
26. Khalil MK, Paas F, Johnson TE, Paye AF: In e ac i e and dynamic isualiza ions in
eaching and lea ning o ana omy: a cogni i e load pe spec i e. Ana omical eco d.
Pa B, New ana omis 2005, 286:8–14, [h p://www.ncbi.nlm.nih.go /pubmed/16177993].
21
27. Swelle J: Cogni i e load heo y, lea ning di icul y, and ins uc ional design.
Lea ning and ins uc ion 1994, 4(4):295–312.
28. Ke oo BP, Bake H: An online spaced-educa ion game o each and assess
esiden s: a mul i-ins i u ional p ospec i e ial. Jou nal o he Ame ican College o
Su geons 2012, 214(3):367–73, [h p://www.ncbi.nlm.nih.go /pubmed/22225647].
29. Shaw T, Long A, Chop a S, Ke oo BP: Impac on clinical beha io o ace- o- ace
con inuing medical educa ion blended wi h online spaced educa ion: a andomized
con olled ial. The Jou nal o con inuing educa ion in he heal h p o essions 2011,
31(2):103–8, [h p://www.ncbi.nlm.nih.go /pubmed/21671276].
30. Ke oo BP, Fu Y, Bake H, Connelly D, Ri chey ML, Genega EM: Online spaced
educa ion gene a es ans e and imp o es long- e m e en ion o diagnos ic skills:
a andomized con olled ial. Jou nal o he Ame ican College o Su geons 2010,
211(3):331–337.e1, [h p://www.ncbi.nlm.nih.go /pubmed/20800189].
31. Ke oo BP, DeWol WC, Masse BA, Chu ch PA, Fede man DD: Spaced educa ion
imp o es he e en ion o clinical knowledge by medical s uden s: a andomised
con olled ial. Medical educa ion 2007, 41:23–31,
[h p://www.ncbi.nlm.nih.go /pubmed/17209889].
32. Ke dijk W, Tio RA, Mulde BF, Cohen-Scho anus J: Cumula i e assessmen : s a egic
choices o in luence s uden s’ s udy e o . BMC medical educa ion 2013, 13:172.
33. Wood T: Assessmen no only d i es lea ning, i may also help lea ning. Medical
educa ion 2009, 43:5–6.
34. La sen DP, Bu le AC, Roedige III HL: Repea ed es ing imp o es long- e m e en ion
ela i e o epea ed s udy: a andomised con olled ial. Medical educa ion 2009,
43(12):1174–1181.
35. T ial C, Cook DA, Thompson WG, Thomas KG, Thomas MR, Pank a z VS: Impac o
sel -assessmen ques ions and lea ning s yles in Web-based lea ning: a
andomized, con olled, c osso e ial. Academic medicine : jou nal o he Associa ion
o Ame ican Medical Colleges 2006, 81(3):231–8,
[h p://www.ncbi.nlm.nih.go /pubmed/16501263].
36. Ta ei a Gomes T, Se e o M, Guima _es J, Fe ei a A: A no el pla o o m o medical
educa ion - ALERT STUDENT. BMC Medical Educa ion 2014, 1:1.

22
37. Ruiz JG, Min ze MJ, Issenbe g SB: Lea ning objec s in medical educa ion. Medical
eache 2006, 28(7):599–605, [h p://www.ncbi.nlm.nih.go /pubmed/17594550].
38. Schwa z MR, Woj czak A: Global minimum essen ial equi emen s: a oad owa ds
compe ence-o ien ed medical educa ion. Medical eache 2002, 24(2):125–9,
[h p://www.ncbi.nlm.nih.go /pubmed/12098430].
39. Landis JR, Koch GG: The Measu emen o Obse e Ag eemen o Ca ego ical Da a.
Biome ics 1977, 33:159–174.
40. La sen DP, Bu le AC, Roedige III HL: Tes -enhanced lea ning in medical educa ion.
Medical educa ion 2008, 42(10):959–966.
41. Roedige HL, Ka picke JD: The powe o es ing memo y: Basic esea ch and
implica ions o educa ional p ac ice. Pe spec i es on Psychological Science 2006,
1(3):181–210.
42. Schuwi h L, Vleu en C d, Donke s H: A close look a cueing e ec s in mul iple-
choice ques ions. Medical Educa ion 1996, 30:44–49.
43. Schuwi h LW, Van De Vleu en CP: Di e en w i en assessmen me hods: wha can
be said abou hei s eng hs and weaknesses? Medical Educa ion 2004, 38(9):974–
979.
44. Cox M, I by DM, Eps ein RM: Assessmen in medical educa ion. New England Jou nal
o Medicine 2007, 356(4):387–396.
45. Si zmann T, Ely K, B own KG, Baue KN: Sel -Assessmen o Knowledge: A Cogni i e
Lea ning o A ec i e Measu e? The Academy o Managemen Lea ning and Educa ion
2010, 9(2):169–191.
46. Da is DA, Mazmanian PE, Fo dis M, Van Ha ison R, Tho pe KE, Pe ie L: Accu acy o
Physician Sel -assessmen Compa ed Wi h Obse ed Measu es o Compe enceA
Sys ema ic Re iew. Jama 2006, 296(9):1094–1102.
47. T acey J, A oll B, Ba ham P, Richmond D: The alidi y o gene al p ac i ione s’ sel
assessmen o knowledge: c oss sec ional s udy. BMJ 1997, 315(7120):1426–1428.
48. Ha den RM, Gessne IH, Gunn M, Issenbe g SB, P ingle SD, S ewa a: C ea ing an e-
lea ning module om lea ning objec s using a commen a y o ’pe sonal lea ning
23
assis an ’. Medical eache 2011, 33(4):286–90,
[h p://www.ncbi.nlm.nih.go /pubmed/21456985].
49. Cla k R, Maye R: Applying he Segmen ing and P e aining P inciples: Managing
Complexi y by B eaking a Lesson in o Pa s. In E-Lea ning and he Science o
Ins uc ion: P o en Guidelines o Consume s and Designe s o Mul imedia, San
F ancisco: Jossey-Bass 2010:207–218.
50. Mas e s K, Ellaway R: e-Lea ning in medical educa ion Guide 32 Pa 2: Technology,
managemen and design. Medical eache 2008, 30(5):474–89,
[h p://www.ncbi.nlm.nih.go /pubmed/18576186].
51. Kim S, Song SM, Yoon YI: Sma lea ning se ices based on sma cloud compu ing.
Senso s (Basel, Swi ze land) 2011, 11(8):7835–50,
[h p://www.pubmedcen al.nih.go /a icle ende . cgi?a id=3231729 & ool=pmcen ez &
ende ype=abs ac ].
52. Ma inez M: Designing lea ning objec s o pe sonalize lea ning. PhD hesis,
Blooming on IL 2002.
53. Beux PL, Fieschi M: Vi ual biomedical uni e si ies and e-lea ning. In e na ional
jou nal o medical in o ma ics 2007, 76(5-6):331–5,
[h p://www.ncbi.nlm.nih.go /pubmed/17407747].
54. K is iansson MH, T oein M, B o sson A: We li ed and b ea hed medicine- hen li e
ca ches up: Medical s uden s’ e lec ions. BMC medical educa ion 2014, 14:66.
Appendix A
Tables
Table A1: S udy sample cha ac e iza ion
To al
Con ol
Expe imen
p
Gende
n (%)
n (%)
n (%)
Female
59 (62.8)
28 (59.6)
31 (65.9)
0.670
Male
35 (37.2)
19 (40.4)
16 (34.1)
Cou se yea
n (%)
n (%)
n (%)
4 h yea
44 (46.8)
23 (48.9)
21 (44.7)
0.836
5 h yea
50 (53.2)
24 (51.1)
26 (55.3)
P e e ed esou ce
n (%)
n (%)
n (%)
P o esso ex s
36 (38.3)
17 (36.2)
19 (40.4)
0.898
Lec u e no es
24 (25.5)
12 (25.5)
12 (25.5)
Lec u e slides
23 (24.5)
13 (27.7)
10 (21.3)
Tex book
11 (11.7)
5 (11.6)
6 (12.8)
Compu e usage
n (%)
n (%)
n (%)
E e yday
73 (77.7)
37 (78.2)
36 (76.6)
0.193
No e e yday
21 (22.3)
10 (21.2)
11 (23.4)
G ades
Mean (SD)
Mean (SD)
Mean (SD)
Cellula Biology
12.8 (1.2)
13.1 (1.6)
12.8 (1.6)
0.102
Cou se a e age
13.6 (1.1)
13.8 (1.1)
13.6 (1.1)
0.433
Daily s udy hou s
Median (IR)
Median (IR)
Median (IR)
Du ing semes e
3.0 (2.5)
3.0 (2.0)
3.0 (2.0)
0.628
Du ing exam season
9.5 (2.0)
10.0 (2.0)
8.0 (2.0)
0.307
Table A2: Recall accu acy pe session and g oup
To al
Con ol
Expe imen
Mean (SD)
Mean (SD)
Mean (SD)
p1
s0
0.76 (0.56)
0.72 (0.50)
0.81 (0.53)
0.924
s1
1.59 (0.67)
0.99 (0.54)
2.18 (0.55)
<0.001
s2
1.87 (0.65)
1.26 (0.62)
2.47 (0.45)
<0.001
p2
<0.001
<0.001
<0.001
<0.0013
SD - S anda d De ia ion; 1 Di e ences in ecall accu acy be ween expe imen and
con ol g oup; 2 Di e ences in ecall accu acy be ween pai wise sessions; 3
In e ac ion e ec be ween session and g oup.
Table A3: Componen s o a iance o ecall accu acy o he con ol g oup
Componen
n
Va iance
SD
%1
Pa icipan
47
0.165
0.406
15.1%
Flashca d
27
0.377
0.614
34.7%
Session
3
0.089
0.299
8.2%
Residual
3440
0.456
0.676
41.2%
SD - S anda d De ia ion; 1 Pe cen age o o al a iance.
Appendix C
Figu e C1: Sc eensho o s udy mode en i onmen on ALERT STUDENT
Figu e C2: Sc eensho o quiz mode en i onmen on ALERT STUDENT
Flashca d con en
Topic lis
Open sel assessmen
ques ion
Recall accu acy inpu