Error 404- Struggling Learners Not Found” Exploring the Behavior of MOOC Learners.

The final authenticated version is available online at https://doi.org/10.1007/978-3-030-29736-7_56 “Error 404Struggling Learners Not Found” Exploring the Behavior of MOOC Learners Paraskevi Topali, Alejandro Ortega-Arranz, Yannis Dimitriadis, Alejandra Martínez-Monés, Sara L. Villagrá-Sobrino, and Juan I. Asensio-Pérez GSIC-EMIC Research Group, Universidad de Valladolid, Valladolid, Spain [email protected] Abstract. Lack of timely instructors' support when the learners are struggling with the course contents and activities is one frequent problem of MOOC learners. The early identification of these learners could help instructors spend part of their limited time assisting them and avoiding potential dropouts. This paper presents a MOOC case study that explores the behavior of learners who reported problems in private messages and discussion forums. The study aimed at the identification of parameters that might allow the detection of learners struggling with different course aspects. As the results suggested, the comparison of the learners’ activity traces reveals some common sequences that in the future could facilitate the identification of learners facing problems, even without reporting them. On the other hand, statistical analyses on learners’ behavior showed nonsignificant differences between the learners reporting putting their maximum effort to overcome a problem before asking for help and the ones who did not. Keywords: MOOCs, Learner´s Problems, Learner´s Behavior. 1 Introduction Despite the learning opportunities that Massive Open Online Courses (MOOCs) offer, usually MOOC learners face problems during course runtime [1]. While some of these learners are reluctant to share their problems with course peers and instructors, others prefer to post their questions in course forums, eventually without receiving timely the expected support [2]. In both cases, some of these learners that initially showed an interest in the course, disengage due to the experienced problems and drop out. The large learners’ population and the instructors’ high workload in MOOCs make unmanageable the timely awareness and assistance of every learner facing problems [3]. The early identification of learners who face difficulties could help instructors spend part of their limited time assisting them and trying to prevent them from dropping out. Among the different forms of detecting these students, the identification of indicators of learners' behavior could help to understand whether the learner is experiencing a concrete problem without reporting it [4]. Additionally, the learners’ effort to overcome their problems could be considered as a parameter to prioritize the instructors’ limited time (i.e., assist first those students who have already tried to solve the problems). Previous studies have focused on identifying the problems of MOOC learners Topali P., Ortega-Arranz A., Dimitriadis Y., Martínez-Monés A., Villagrá-Sobrino S.L., Asensio-Pérez J.I. (2019) “Error 404Struggling Learners Not Found” Exploring the Behavior of MOOC Learners. In: Scheffel M., Broisin J., Pammer-Schindler V., Ioannou A., Schneider J. (eds) Transforming Learning with Meaningful Technologies. EC-TEL 2019. Lecture Notes in Computer Science, vol 11722. Springer, Cham. The final authenticated version is available online at https://doi.org/10.1007/978-3-030-29736-7_56 [1, 2] and on creating predictive models for detecting critical cohorts of learners at risk of dropout [5]. However, to the best of our knowledge, none of the previous works has explicitly studied activity traces of learners reporting problems nor conducted comparative analysis between different cohorts of learners based on their experiences towards solving MOOC problems. This paper presents a MOOC case study that explores the learners’ behavioral activity traces to provide useful information for the identification of learners who face problems during a MOOC. Two research questions (RQ) guided this study: (RQ·1) “To what extent is it possible to identify learners who face problems by looking at their effort before asking for help?” and (RQ·2) “Is there any kind of common behavior among the MOOC learners who reported problems before asking for help?” 2 Methodology of the Study The study was conducted in a MOOC about English-Spanish translation in the financial and business fields, launched in the Canvas Network platform by the University of Valladolid. The course consisted of seven weekly modules including video lectures, readings, extra material/resources, discussion forums and several compulsory and optional activities. A total number of 866 learners enrolled in the course, out of whom 169 obtained the certificate (19.52% completion rate). The certificate was issued to those participants completing all the compulsory activities. In order to answer the two RQs, three data sources were used: (a) self-reported data from discussion forums (N=156) and private messages (N=38); (b) learners’ trace data (number of forum posts, assignments’ submissions, pageviews and the total time spent in the course); and (c) the answers to a post-course questionnaire (N=172). 3 Results RQ·1: In the post-course questionnaire the subjects were asked about the effort they put before asking for help. Many learners (N=44) reported that they could have solved their problem by putting some personal extra effort, but more learners (N=54) claimed that they turned for assistance after putting their maximum effort to overcome their challenge. The early identification of the latter learners could help the instructor assist first the ones who need help and have put their maximum effort. For that reason, the behavioral activity of these two cohorts of learners was explored regarding the pageviews, tasks submissions, time spent in the course and their general participation (see Table1). Results show non-significant differences between the two groups for the four variables measuring behavioral activity. Z-tests (two-tailed, alpha=.05) were performed to analyze the mean differences of the previous variables between cohorts due to the large sample sizes (>30 answers). Table 1. Statistical analysis of the two different learners’ cohorts Pageviews Submissions Posts Activity Time “Min. Effort” mean 639.36 12.5 3.24 1676.21 “Max. Effort” mean 595.93 12.13 3.68 1743.25 Z-test p-value 0.463 0.2396 0.6751 0.8377 RQ·2: The previous analysis was complemented with an analysis of the activity traces of the learners who reported problems in private messages and discussion forums. For the analysis, the activity of the learners previous to the communication of the problem and two days after the communication of the problem was considered. Common activity sequences were detected among 13 (out of N=14) learners mentioning collaboration problems in discussion forums; 9 learners stated that only a few group members were active, and 4 learners reported that their group members were totally absent. The sequences of the learners’ trace activity were: (1) visiting many times the communication threads (general discussion and group discussion forums 1 ) (N=12) and (2) reporting the problem in both threads, first in the group forum making it visible to group peers (N=13), and then to the general forum to make it visible to the rest of the students and to the course team. Additionally, two learners (out of the 4 who did not find other group members) tended to (3) revisit several times the private message page probably waiting a message from the instructors. Figure 1 illustrates the activity traces of two representative learners’ cases. In Case 1 the learner was not able to find any active member of his group. According to his traces, he visited many times both the general discussion and group discussion forums, and he also posted in group discussion forums to communicate with his peers. Without receiving answer, he continued visiting both communication threads and finally he posted his problem in general discussion forums as well. In Case 2, the learner was experiencing the same problem as the student before behaving similar. However, instead of only visiting the two discussion forums (group and general), she was visiting many times the private messages' page, suggesting that she was waiting for the instructor's answer to her message. 4 Discussion & Conclusions This study attempted to shed light on the behavior of MOOC learners’ tracking their activity traces to explore indicators for the identification of learners who face problems during a MOOC. The evidence gathered from RQ·1 showed non-significant differences between the learners reporting putting their maximum effort to overcome a problem before asking for help and the ones who did not. Whilst the indicators of posts in discussion forums, time spent, pageviews and assignments’ submission applied in our 1 Learners were expected to discuss and complete the group activities by communicating in private group forums. Groups were composed by 5 or 6 learners. Topali P., Ortega-Arranz A., Dimitriadis Y., Martínez-Monés A., Villagrá-Sobrino S.L., Asensio-Pérez J.I. (2019) “Error 404Struggling Learners Not Found” Exploring the Behavior of MOOC Learners. In: Scheffel M., Broisin J., Pammer-Schindler V., Ioannou A., Schneider J. (eds) Transforming Learning with Meaningful Technologies. EC-TEL 2019. Lecture Notes in Computer Science, vol 11722. Springer, Cham. The final authenticated version is available online at https://doi.org/10.1007/978-3-030-29736-7_56 study are commonly used in the literature analyzing activity traces, probably they did not provide fruitful information regarding our question. This finding points out the need of collecting richer data, from different sources, to help instructors understand better who needs help and what kind of help. Regarding RQ·2 we identified two patterns of behavior that could help instructors or, eventually systems, to spot learners that are not finding the needed support to continue with their tasks even without communicating their problems. These patterns are the recurring visits to the forums/private messages after posting in them, and the broadcasting of help-seeking messages in all possible channels. Our findings may not be generalized but they contribute to a better understanding of learners’ course activity and open new lines of future work. Fig. 1. Representation of two learners’ activity traces Acknowledgements This research has been partially funded by the European Regional Development Fund and the National Research Agency of the Spanish Ministry of Science, Innovations and Universities under project grants TIN2017-85179-C3-2-R and TIN2014-53199C3-2R, by the European Regional Development Fund (Operational program of Castile and León) and the Regional Government of Castile and León by the Regional Ministry of Education under the grant BOCYL-D07062018-6 and the project grant VA257P18, and by the European Commission under project grant 588438-EPP-1-2017-1-EL-EPPKA2-KA. References 1. Henderikx, M., Kreijns, K., Kalz, M.: A Classification of Barriers that Influence Intention Achievement in MOOCs. In Proceedings of 13th European Conference on Technology Enhanced Learning, EC-TEL 2018, Leeds, UK (2018). 2. Almatrafi, O., Johri, A., Rangwala, H.: Needle in a haystack: Identifying learner posts that require urgent response in MOOC discussion forums. Comput. Educ. (2018). 3. Hew, K.F., Cheung, W.S.: Students’ and instructors’ use of massive open online courses (MOOCs): Motivations and challenges, (2014) 4. Teusner, R., Hille, T., Staubitz, T.: Effects of Automated Interventions in Programming Assignments : Evidence from a Field Experiment. In: Proceedings of the Fifth ACM Conference on Learning @ Scale - L@S ’18. , London, United Kingdom (2018) 5. He, J., Bailey, J., Rubinstein, B.I., Zhang, R.: Identifying At-Risk Students in Massive Open Online Courses. In: Proceedings of the Twenty Ninth AAAI Conference on Artificial Intelligence. pp. 1749–1755 (2015)