Analysis of a novel e-learning system in measuring medical students outcomes in study time recall accuracy.

1 MESTRADO INTEGRADO EM MEDICINA Instituto Ciências Biomédicas Abel Salazar Universidade do Porto 2013/2014 Analysis of a novel e-learning system in measuring medical students outcomes in study time and recall accuracy RUI ALEXANDRE DO PRADO COSTA 2 Rui Alexandre do Prado Costa Analysis of a novel e-learning system in measuring medical students outcomes in study time and recall accuracy Mestrado Integrado em Medicina Área: Educação Médica 3 Analysis of a novel e-learning system in measuring medical students outcomes in study time and recall accuracy Tiago Taveira-Gomes*1,2 Rui Prado Costa*2,3 Milton Severo1 Maria Amélia Ferreira1 1Center for Medical Education, Faculty of Medicine of the University of Porto, PT 2ALERT Life Sciences Computing SA 3Abel Salazar Biomedical Sciences Institute, University of Porto Email: Tiago Taveira-Gomes*- [email protected]; Rui Prado Costa – [email protected] *Corresponding authors Abstract Background: Spaced-repetition and repeated testing with feedback, are two methodologies known to boost knowledge retention. ALERT STUDENT, an online platform for medical education, was devised to implement both approaches in an integrated fashion and promote research in this field, by allowing creation and distribution of learning objects named flashcards. This study aims to demonstrate how learning objects and repeated testing using open-ended questions can be brought together to provide insights on recall accuracy, and to characterize the extent to which student factors, content factors and repetition affect it. Methods: Three in-person sessions (s0, s1 and s2), separated by one-week intervals were conducted with 96 medical students using the platform. Students were randomized into experiment and control groups using simple randomization. The study consisted on a 20 min study task using 27 flashcards about the Golgi complex, and on a self-assessment task, that consisted on answering flashcard open-ended questions and grading recall accuracy. On s0 both groups performed the self-assessment task. On s1 and s2, the experiment group performed the study task followed by the self-assessment task, whereas the control group only performed the self-assessment task. 4 Results: At s0 there were no differences in recall accuracy between groups, but in s1 and s2 recall accuracy differences were statistically significant. The experiment group achieved a sharper increase in recall accuracy than the control group, which was stronger between s0 and s1. In the experiment group, increases in recall accuracy were mainly due to the session effect (49.6%) followed by flashcard factors (15.3%) and student factors (5.1%). In the control group, increases in recall accuracy were mainly accounted by flashcard factors (34.7%), followed by student (15.9%) and session (8.2%) factors. Generalizability score for the flashcard in the control group was 0.91, indicating almost perfect agreement, and 0.47 for the experiment group, indicating moderate agreement. This finding indicates that flashcards can be very well characterized by recall accuracy scores by having the same students perform sequential selfassessment tasks alone. Regarding the experiment group, differences in study duration for sessions s1 and s2 were statistically significant. Mean study duration was 14:32 minutes for s1 and 12:28 minutes for s2. Conclusions: The self-assessment activity alone had little effect on recall accuracy, however studying and self-test activities together greatly increased recall accuracy. The session effect was the main determinant of recall accuracy for the experiment group. Learning objects may be able to provide additional value when combined with open-ended questions. Insights about recall accuracy can be useful for teachers to assess content difficulty, learning rate, and for students to prioritize content and make better use of study sessions. Keywords Medical education, Memory retention, Computer-Assisted Instruction, E-learning, Tailoredlearning, Spaced repetition 5 Background Medical education is a complex field where updates in medical knowledge, educational technology and teaching strategies intertwine in a progressive fashion [1, 2, 3, 4, 5]. Over the past decade there has been a shift in this field, where traditional instructor-centered teaching is yielding to a learner-centered model [6, 7, 8, 9], in which the learner has greater control over the learning methodology and the role of a teacher becomes that of a facilitator of knowledge acquisition, replacing the role of an information provider [10, 11, 12, 7]. There is vast literature regarding the application of educational strategies [13, 14, 15, 16, 17, 18], instructional design [19, 11, 20, 21, 22, 23] and cognitive learning science [24, 25, 26, 27] to the field of medical education. Two approaches that emerge from that literature are spaced-repetition and repeated testing. Spaced-repetition is a learning approach grounded in cognitive learning theory that enforces continuously revisiting content over optimized time intervals, and has been shown to improve long-term knowledge retention [28, 29, 30, 11, 31]. Repeated testing during learning is an approach that extends spaced repetition by applying it to assessments during learning periods. It argues that the addition of repeated assessment feedback can further enhance learning [32, 33, 34]. In addition, the usage of open-ended questions as means of assessment has been shown to be superior to multiple choice questions in terms of knowledge retention [35]. Despite this vast literature, there are few reports of implementing these principles in an on-line platform in an integrated fashion [36]. ALERT STUDENT, a platform for medical education, was devised with the goal to implement such principles and promote research in this field [36]. The platform allows creation and distribution of learning objects [37] named notebooks. These are composed of smaller learning objects named flashcards, self-contained information chunks with related open-ended questions [36]. Notebooks can be accessed using a study mode that presents the flashcards in a study-friendly environment enriched with note taking, text highlighting and time tracking features. The quiz mode is a complementary environment where flashcard knowledge can be self-assessed using its open-ended questions. A set of questions from each flashcard is sequentially presented and requires the student to actively recall the answer and grade recall 6 accuracy using a 4 point likert scale. This information is summarized into a 4 point color-coded score that is attributed to each flashcard and displayed in the study mode as a study priority cue. Exploring the usefulness of providing knowledge retention tools in e-learning systems is an important contribution to information management axis of the core-competences for medical education [38] and may provide a means to a richer and faster acquisition of factual knowledge, which is a cornerstone for the development of cognitive schemata, clinical reasoning skills [27]. This study aims to demonstrate how learning objects and repeated testing using open-ended questions can be brought together to provide insights on recall accuracy, and to characterize the extent to which student factors, content factors and repetition affect it. 7 Methods Study design The Faculty of Medicine of the University of Porto (FMUP) implements a 6-year graduate program. Applicants are mainly high school graduates. The first three years focus on basic sciences while the last three focus on clinical specialties. Ninety-eight (n=98) students from the 4th and 5th grades of our school were contacted via email and volunteered to participate in this study. The study was performed in three in-person sessions (s0, s1 and s2) of 1 hour duration. Sessions were carried one week apart. Students were assigned into experiment group or control group using simple randomization. The study consisted of a study task and a self-assessment task. During the study task, participants studied a notebook containing flashcards about the Golgi complex during 20 minutes using the study mode. During the self-assessment task, students were presented open-ended questions from the notebook on the quiz mode, and were required to grade their active recall for each question using a 4 point likert scale (0 - no recall, 1 - scarce recall, 2 - good recall, 3 - full recall). This task lasted 15 minutes. (Study mode and quiz mode environment screenshots can be viewed on appendix C) A pilot study was conducted using two 2nd grade students that had recently completed the Cellular and Molecular Biology class, two 4th grade and two 5th grade medical students (n = 6). Students performed the self-assessment task. The 2nd grade students’ average recall accuracy was assumed to be similar to what would be found at session s2. The 4th and 5th grade students’ mean recall accuracy was assumed to correspond to what would be found at session s0. Based on the differences between the two pilot groups it was concluded that n = 48 would be sufficient to discriminate statistical significant differences in recall accuracy between sessions and between groups. The sample size was incremented to n = 98 to take advantage of the study venue capacity. On s0, both groups performed the self-assessment task. On s1 and s2, the control group performed the self-assessment task, and the experiment group performed the study task, immediately followed by the self-assessment task. 8 For each session, flashcard recall accuracy was computed as the mean recall accuracy of the questions belonging to a flashcard. For the experiment group, study duration for each user and session was computed as the cumulative sum of time spent on each flashcard for the session. Content design The content used in this study was designed using official lecture content provided by the Cellular and Molecular Biology Department of our school, that is responsible for teaching the Golgi complex in Cellular Biology class as part of the first grade of the school medical curricula. See Appendix B to access notebook content. The questions were part of a notebook about the Golgi complex that contained 27 flashcards. In total 42 open ended questions about the 27 flashcards were answered and graded according to recall accuracy. The learning material did not change between sessions. During each session different questions were presented, but all of them covered the notebook material in a complete and homogeneous fashion. Fitting content to time constraint Another pilot study was performed to tailor the notebook content based on the study time constraint of 20 minutes. Another set of 6 5th grade students were assigned to read the notebook content. From the initial set of 30 flashcards, the last 3 were removed so that the content could be fully read within the 20 minute interval. No changes were necessary to answer the quiz mode questions within the 15 minute interval. Sample characterization In session s0 both groups filled a survey to characterize the student sample. Measured factors were gender, course year, preferred study resource for Cellular Biology, computer usage habits, Cellular Biology grade, mean course grade, and average study session duration during the semester and during the exam season. Evaluation Changes across sessions in recall accuracy were examined by univariate repeated-measures analysis of variance (ANOVA). Control and experiment groups were used as between-subjects factor. Session and flashcard were used as within subject factor. Repeated contrast (s0 vs s1 and s1 vs s2) was used to evaluate the sessions and the session interaction effect. In order to determine the reliability of the variance of recall accuracy explained by the flashcard, the 9 generalizability score (G-Score) for this variance component was computed. Guidelines for interpreting G-Scores suggest that values for relative variance between 0.81 - 1.00 indicate almost perfect agreement, 0.61 - 0.80 substantial agreement, 0.41 - 0.60 moderate agreement, 0.21 - 0.40 fair agreement, and values less than 0.21 depict poor or slight agreement [39]. The Wilcoxon signed-rank test was performed to compare study duration distributions of the experiment group for sessions s1 and s2. This study was approved by the Faculty of Medicine University of Porto / São João Hospital Ethics Committee in compliance with the Helsinki Declaration. 16 Conclusions The present study focus on measuring recall accuracy of content using open ended questions in a controlled setting using the ALERT STUDENT platform. It was found that self-assessment recall activity alone led to a modest increase on recall accuracy, and that studying and selfassessment together had high impact in recall accuracy. It was shown that session repetition was the main determinant of recall accuracy when students perform study followed by the selfassessment task, and that when employing self-assessment only, content and student factors determine most of the increase in recall accuracy. These insights may be helpful to tailor teaching and learning methodologies to better reach learning goals. The present findings will be explored in more detail in future work, as they may help future physicians and medical schools meet the challenge of information management [38] and instilling a culture of continuous learning, underpinning the core competencies outlined for XXI century physicians [4] [3]. 17 List of abbreviations ANOVA Analysis of variance FMUP Faculty of Medicine, University of Porto G-Score Generalizability score s0 Study session 0 s1 Study session 1 s2 Study session 2 Competing interests The authors of the study declare no competing interests. Authors contributions TTG and RC formulated the research problem, carried on-site procedures, performed statistical analysis and contributed equally to this work. MS Validated study design and conducted statistical analysis. MAF oversaw the study and gave final approval. All authors contributed to the study design and drafted the manuscript. All authors read and approved the final manuscript. Acknowledgments The authors wish to thank to the students who took part in the study. 18 References 1. Qiao YQ, Shen J, Liang X, Ding S, Chen FY, Shao L, Zheng Q, Ran ZH: Using cognitive theory to facilitate medical education. BMC medical education 2014, 14:79. 2. Schoonheim M, Heyden R, Wiecha JM: Use of a virtual world computer environment for international distance education: lessons from a pilot project using Second Life. BMC medical education 2014, 14:36. 3. Frenk J, Chen L, Bhutta ZA, Cohen J, Crisp N, Evans T, Fineberg H, Garcia P, Ke Y, Kelley P, Kistnasamy B, Meleis A, Naylor D, Pablos-Mendez A, Reddy S, Scrimshaw S, Sepulveda J, Serwadda D, Zurayk H: Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. Lancet 2010, 376(9756):1923–58, [http://www.ncbi.nlm.nih.gov/pubmed/21112623]. 4. Horton R: A new epoch for health professionals’ education. Lancet 2010, 376(9756):1875–7, [http://www.ncbi.nlm.nih.gov/pubmed/21112621]. 5. Patel VL, Cytryn KN, Shortliffe EH, Safran C: The collaborative health care team: the role of individual and group expertise. Teaching and learning in medicine 2000, 12(3):117–32, [http://www.ncbi.nlm.nih.gov/pubmed/11228898]. 6. Bahner DP, Adkins E, Patel N, Donley C, Nagel R, Kman NE: How we use social media to supplement a novel curriculum in medical education. Medical teacher 2012, [http://informahealthcare.com/doi/abs/10.3109/0142159X.2012.668245]. 7. Ruiz JG, Mintzer MJ, Leipzig RM: The impact of E-learning in medical education. Academic medicine : journal of the Association of American Medical Colleges 2006, 81(3):207–12, [http://www.ncbi.nlm.nih.gov/pubmed/16501260]. 8. Eysenbach G: Medicine 2.0: social networking, collaboration, participation, apomediation, and openness. Journal of medical Internet research 2008, 10(3):e22, [http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=2626430\&tool=pmcentrez\&r endertype=abstract]. 9. Boulos MNK, Maramba I, Wheeler S: Wikis, blogs and podcasts: a new generation of Web-based tools for virtual collaborative clinical practice and education. BMC medical education 2006, 6:41, [http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=1564136\&tool=pmcentrez\&r endertype=abstract]. 19 10. Koops W, Van der Vleuten C, De Leng B, Oei SG, Snoeckx L: Computer-supported collaborative learning in the medical workplace: Students’ experiences on formative peer feedback of a critical appraisal of a topic paper. Medical teacher 2011, 33(6):e318–23, [http://www.ncbi.nlm.nih.gov/pubmed/21609168]. 11. Cook Da, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM: Instructional design variations in internet-based learning for health professions education: a systematic review and meta-analysis. Academic medicine : journal of the Association of American Medical Colleges 2010, 85(5):909–22, [http://www.ncbi.nlm.nih.gov/pubmed/20520049]. 12. Chretien KC, Greysen SR, Chretien JP, Kind T: Online posting of unprofessional content by medical students. JAMA : the journal of the American Medical Association 2009, 302(12):1309–15, [http://www.ncbi.nlm.nih.gov/pubmed/19773566]. 13. Woltering V, Herrler A, Spitzer K, Spreckelsen C: Blended learning positively affects students’ satisfaction and the role of the tutor in the problem-based learning process: results of a mixed-method evaluation. Advances in health sciences education : theory and practice 2009, 14(5):725–38, [http://www.ncbi.nlm.nih.gov/pubmed/19184497]. 14. Sandars J, Haythornthwaite C: New horizons for e-learning in medical education: ecological and Web 2.0 perspectives. Medical teacher 2007, 29(4):307–10, [http://www.ncbi.nlm.nih.gov/pubmed/17786742]. 15. Giani U, Brascio G, Bruzzese D, Garzillo C, Vigilante S: Emotional and cognitive information processing in web-based medical education. Journal of biomedical informatics 2007, 40(3):332–42, [http://www.ncbi.nlm.nih.gov/pubmed/17208055]. 16. Pereira JA, Pleguezuelos E, Mer_ A, Molina-Ros A, Molina-Tom_s MC, Masdeu C: Effectiveness of using blended learning strategies for teaching and learning human anatomy. Medical education 2007, 41(2):189–95, [http://www.ncbi.nlm.nih.gov/pubmed/17269953]. 17. Ruiz JG, Mintzer MJ, Leipzig RM: The impact of E-learning in medical education. Academic medicine : journal of the Association of American Medical Colleges 2006, 81(3):207–12, [http://www.ncbi.nlm.nih.gov/pubmed/16501260]. 18. Taradi SK, Taradi M, Radic K, Pokrajac N: Blending problem-based learning with Web technology positively impacts student learning outcomes in acid-base physiology. 20 Advances in physiology education 2005, 29:35–9, [http://www.ncbi.nlm.nih.gov/pubmed/15718381]. 19. Conn JJ, Lake FR, McColl GJ, Bilszta JLC, Woodward-Kron R: Clinical teaching and learning: from theory and research to application. The Medical Journal of Australia 2012, 196(8):527, [http://www.ncbi.nlm.nih.gov/pubmed/22571313https://www.mja.com.au/journal/2012/19 6/8/clinical-teaching-and-learning-theory-and-research-application]. 20. Sandars J, Lafferty N: Twelve Tips on usability testing to develop effective e-learning in medical education. Medical teacher 2010, 32(12):956–60, [http://www.ncbi.nlm.nih.gov/pubmed/21090948]. 21. Patel VL, Yoskowitz NA, Arocha JF, Shortliffe EH: Cognitive and learning sciences in biomedical and health instructional design: A review with lessons for biomedical informatics education. Journal of biomedical informatics 2009, 42:176–97, [http://dx.doi.org/10.1016/j.jbi.2008.12.002http://www.ncbi.nlm.nih.gov/pubmed/1913517 3]. 22. Sweller J, van Merrienboer JJ, Paas FG: Cognitive Architecture and Instructional Design. Educational Psychology Review 1998, 10(3):251–296, [http://doc.utwente.nl/58655/]. 23. Sweller J: Cognitive Load During Problem Solving: Effects on Learning. Cognitive Science 1988, 12(2):257–285, [http://doi.wiley.com/10.1207/s15516709cog1202\_4]. 24. Dror I, Schmidt P, O’connor L: A cognitive perspective on technology enhanced learning in medical training: great opportunities, pitfalls and challenges. Medical teacher 2011, 33(4):291–6, [http://www.ncbi.nlm.nih.gov/pubmed/21456986]. 25. van Merri_nboer JJG, Sweller J, Merrie JJGV: Cognitive load theory in health professional education: design principles and strategies. Medical educationVan Merri_nboer, J. J. G., Sweller, J., & Merrie, J. J. G. Van. (2010). Cognitive load theory in health professional education: design principles and strategies. Medical education, 44(1), 85–93. doi:10.1111/j.1365-2923.2009.03498.x 2010, 44:85–93, [http://www.ncbi.nlm.nih.gov/pubmed/20078759]. 26. Khalil MK, Paas F, Johnson TE, Payer AF: Interactive and dynamic visualizations in teaching and learning of anatomy: a cognitive load perspective. Anatomical record. Part B, New anatomist 2005, 286:8–14, [http://www.ncbi.nlm.nih.gov/pubmed/16177993]. 21 27. Sweller J: Cognitive load theory, learning difficulty, and instructional design. Learning and instruction 1994, 4(4):295–312. 28. Kerfoot BP, Baker H: An online spaced-education game to teach and assess residents: a multi-institutional prospective trial. Journal of the American College of Surgeons 2012, 214(3):367–73, [http://www.ncbi.nlm.nih.gov/pubmed/22225647]. 29. Shaw T, Long A, Chopra S, Kerfoot BP: Impact on clinical behavior of face-to-face continuing medical education blended with online spaced education: a randomized controlled trial. The Journal of continuing education in the health professions 2011, 31(2):103–8, [http://www.ncbi.nlm.nih.gov/pubmed/21671276]. 30. Kerfoot BP, Fu Y, Baker H, Connelly D, Ritchey ML, Genega EM: Online spaced education generates transfer and improves long-term retention of diagnostic skills: a randomized controlled trial. Journal of the American College of Surgeons 2010, 211(3):331–337.e1, [http://www.ncbi.nlm.nih.gov/pubmed/20800189]. 31. Kerfoot BP, DeWolf WC, Masser BA, Church PA, Federman DD: Spaced education improves the retention of clinical knowledge by medical students: a randomised controlled trial. Medical education 2007, 41:23–31, [http://www.ncbi.nlm.nih.gov/pubmed/17209889]. 32. Kerdijk W, Tio RA, Mulder BF, Cohen-Schotanus J: Cumulative assessment: strategic choices to influence students’ study effort. BMC medical education 2013, 13:172. 33. Wood T: Assessment not only drives learning, it may also help learning. Medical education 2009, 43:5–6. 34. Larsen DP, Butler AC, Roediger III HL: Repeated testing improves long-term retention relative to repeated study: a randomised controlled trial. Medical education 2009, 43(12):1174–1181. 35. Trial C, Cook DA, Thompson WG, Thomas KG, Thomas MR, Pankratz VS: Impact of self-assessment questions and learning styles in Web-based learning: a randomized, controlled, crossover trial. Academic medicine : journal of the Association of American Medical Colleges 2006, 81(3):231–8, [http://www.ncbi.nlm.nih.gov/pubmed/16501263]. 36. Taveira Gomes T, Severo M, Guimar_es J, Ferreira A: A novel platoform for medical education - ALERT STUDENT. BMC Medical Education 2014, 1:1. 22 37. Ruiz JG, Mintzer MJ, Issenberg SB: Learning objects in medical education. Medical teacher 2006, 28(7):599–605, [http://www.ncbi.nlm.nih.gov/pubmed/17594550]. 38. Schwarz MR, Wojtczak A: Global minimum essential requirements: a road towards competence-oriented medical education. Medical teacher 2002, 24(2):125–9, [http://www.ncbi.nlm.nih.gov/pubmed/12098430]. 39. Landis JR, Koch GG: The Measurement of Observer Agreement for Categorical Data. Biometrics 1977, 33:159–174. 40. Larsen DP, Butler AC, Roediger III HL: Test-enhanced learning in medical education. Medical education 2008, 42(10):959–966. 41. Roediger HL, Karpicke JD: The power of testing memory: Basic research and implications for educational practice. Perspectives on Psychological Science 2006, 1(3):181–210. 42. Schuwirth L, Vleuten Cvd, Donkers H: A closer look at cueing effects in multiplechoice questions. Medical Education 1996, 30:44–49. 43. Schuwirth LW, Van Der Vleuten CP: Different written assessment methods: what can be said about their strengths and weaknesses? Medical Education 2004, 38(9):974– 979. 44. Cox M, Irby DM, Epstein RM: Assessment in medical education. New England Journal of Medicine 2007, 356(4):387–396. 45. Sitzmann T, Ely K, Brown KG, Bauer KN: Self-Assessment of Knowledge: A Cognitive Learning or Affective Measure? The Academy of Management Learning and Education 2010, 9(2):169–191. 46. Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L: Accuracy of Physician Self-assessment Compared With Observed Measures of CompetenceA Systematic Review. Jama 2006, 296(9):1094–1102. 47. Tracey J, Arroll B, Barham P, Richmond D: The validity of general practitioners’ self assessment of knowledge: cross sectional study. BMJ 1997, 315(7120):1426–1428. 48. Harden RM, Gessner IH, Gunn M, Issenberg SB, Pringle SD, Stewart a: Creating an elearning module from learning objects using a commentary or ’personal learning 23 assistant’. Medical teacher 2011, 33(4):286–90, [http://www.ncbi.nlm.nih.gov/pubmed/21456985]. 49. Clark R, Mayer R: Applying the Segmenting and Pretraining Principles: Managing Complexity by Breaking a Lesson into Parts. In E-Learning and the Science of Instruction: Proven Guidelines for Consumers and Designers of Multimedia, San Francisco: Jossey-Bass 2010:207–218. 50. Masters K, Ellaway R: e-Learning in medical education Guide 32 Part 2: Technology, management and design. Medical teacher 2008, 30(5):474–89, [http://www.ncbi.nlm.nih.gov/pubmed/18576186]. 51. Kim S, Song SM, Yoon YI: Smart learning services based on smart cloud computing. Sensors (Basel, Switzerland) 2011, 11(8):7835–50, [http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3231729\&tool=pmcentrez\&r endertype=abstract]. 52. Martinez M: Designing learning objects to personalize learning. PhD thesis, Bloomington IL 2002. 53. Beux PL, Fieschi M: Virtual biomedical universities and e-learning. International journal of medical informatics 2007, 76(5-6):331–5, [http://www.ncbi.nlm.nih.gov/pubmed/17407747]. 54. Kristiansson MH, Troein M, Brorsson A: We lived and breathed medicine-then life catches up: Medical students’ reflections. BMC medical education 2014, 14:66. Appendix A Tables Table A1: Study sample characterization Total Control Experiment p Gender n (%) n (%) n (%) Female 59 (62.8) 28 (59.6) 31 (65.9) 0.670 Male 35 (37.2) 19 (40.4) 16 (34.1) Course year n (%) n (%) n (%) 4th year 44 (46.8) 23 (48.9) 21 (44.7) 0.836 5th year 50 (53.2) 24 (51.1) 26 (55.3) Preferred resource n (%) n (%) n (%) Professor texts 36 (38.3) 17 (36.2) 19 (40.4) 0.898 Lecture notes 24 (25.5) 12 (25.5) 12 (25.5) Lecture slides 23 (24.5) 13 (27.7) 10 (21.3) Textbook 11 (11.7) 5 (11.6) 6 (12.8) Computer usage n (%) n (%) n (%) Everyday 73 (77.7) 37 (78.2) 36 (76.6) 0.193 Not everyday 21 (22.3) 10 (21.2) 11 (23.4) Grades Mean (SD) Mean (SD) Mean (SD) Cellular Biology 12.8 (1.2) 13.1 (1.6) 12.8 (1.6) 0.102 Course average 13.6 (1.1) 13.8 (1.1) 13.6 (1.1) 0.433 Daily study hours Median (IR) Median (IR) Median (IR) During semester 3.0 (2.5) 3.0 (2.0) 3.0 (2.0) 0.628 During exam season 9.5 (2.0) 10.0 (2.0) 8.0 (2.0) 0.307 Table A2: Recall accuracy per session and group Total Control Experiment Mean (SD) Mean (SD) Mean (SD) p1 s0 0.76 (0.56) 0.72 (0.50) 0.81 (0.53) 0.924 s1 1.59 (0.67) 0.99 (0.54) 2.18 (0.55) <0.001 s2 1.87 (0.65) 1.26 (0.62) 2.47 (0.45) <0.001 p2 <0.001 <0.001 <0.001 <0.0013 SD - Standard Deviation; 1 Differences in recall accuracy between experiment and control group; 2 Differences in recall accuracy between pairwise sessions; 3 Interaction effect between session and group. Table A3: Components of variance of recall accuracy for the control group Component n Variance SD %1 Participant 47 0.165 0.406 15.1% Flashcard 27 0.377 0.614 34.7% Session 3 0.089 0.299 8.2% Residual 3440 0.456 0.676 41.2% SD - Standard Deviation; 1 Percentage of total variance. Appendix C Figure C1: Screenshot of study mode environment on ALERT STUDENT Figure C2: Screenshot of quiz mode environment on ALERT STUDENT Flashcard content Topic list Open self assessment question Recall accuracy input