Exploring the Problems Experienced by Learners in a MOOC Implementing Active Learning Pedagogies

Exploring the Problems Experienced by Learners in a MOOC Implementing Active Learning Pedagogies Paraskevi Topali, Alejandro Ortega-Arranz, Erkan Er, Alejandra Martínez-Monés, Sara L. Villagrá-Sobrino, and Yannis Dimitriadis GSIC-EMIC Research Group, Universidad de Valladolid, Valladolid, Spain [email protected] Abstract. Although Massive Open Online Courses (MOOCs) have been reported as an effective educational tool offering numerous opportunities in online learning, the high dropout rates and the lack of learners’ motivation are factors concerning researchers and instructors. The one-size-fits-all instructional approach that most courses follow, failing to address the individual needs of learners, has been seen as their weakest point. Recent efforts focus on the inclusion of active learning pedagogies in MOOCs to stimulate the interaction among the participants and to keep them engaged. However, taking into account that in these massive contexts the learners face several issues while trying to keep up with the course, the incorporation of active learning strategies may introduce additional problems to the learning process. This study explores the problems that learners experienced in a MOOC implementing collaboration and gamification strategies. As the results reveal, the introduction of collaborative learning activities can generate additional problems to learners and for that reason, a careful design and a proper scaffolding is needed in an early stage to overcome the problems that will occur. No significant problems were reported regarding the implementation of gamification elements. Keywords: MOOC · Learners’ problems · Active learning · Gamification Collaboration 1 Introduction Massive Open Online Courses (MOOCs) have transformed online learning byoffering learning experiences without geographical and cost restrictions [1]. Although MOOCs present rich and diverse educational materials and enable connecting individuals all around the world, not all MOOC learners fully benefit from these opportunities. Many learners face difficulties in understanding the concepts and completing the assignments, thus leading to student disengagement and course abandonment [1]. It has been suggested that the aforementioned problems are related with the lack of solid pedagogical frameworks in MOOC environments [2]. Most courses follow a one-size- fits-all instructional approach and fail to address the individual needs of learners [1]. The final authenticated version is available online at https://doi.org/10.1007/978-3-030-19875-6_10 The application of more active learning pedagogies may help address the aforementioned problems in MOOCs. Active learning is a pedagogical approach defined as “the instructional activities involving learners in doing things and thinking about what they are doing” [3]. Recent efforts focus on the inclusion of active learningpedagogies in MOOCs to stimulate the interaction among the participants and promote their engagement [4]. However, these strategies may have some adverse effects; according to a study of [5], many students found the learning process more stressful and unpleasant due to their lack of experience in working with active learning strategies. Common difficulties faced by MOOC learners have been previously explored [1, 6, 7], mostly using post-course surveys [8–12] or interviews [13, 14]. According to such studies, the main reasons for learners to disengage and/or to drop out of the courses are (i) the lack of time, (ii) the absence of support and feelings of isolation, (iii) the lack of previous knowledge and learning skills, (iv) unchallenging course design, and (v) the failure to understand the course content. However, most of the reviewed works did not focus on MOOCs implementing active learning strategies beyond discussion forums and peer reviews. In this study we analyze the problems reported by the learners in a MOOC implementing a set of collaborative activities (i.e., two group activities, acollaborative glossary, and two peer reviews) and gamified activities associated with optional tasks. Collaborative learning and games (currently extended to gamification1) are two strategies promoting the learners’ active learning [3]. The underlying research question guiding this work is: Which problems do learners experience in a MOOC implementing active learning strategies? This paper is structured as follows. The next section explains the design of the study including the research methodology and the data gathering sources and techniques. Next, the findings from the analysis are presented and the results are discussed. Finally, the paper draws some conclusions along with limitations and ideas for future research. 2 Methodology 2.1 Research Design The present study attempts to address the following research question (RQ): “Which problems do learners experience in a MOOC implementing active learning strategies?”. To thoroughly explore this topic, the RQ has been subdivided into two informative questions (IQ): 1. What were the problems faced by the learners who completed the course successfully? 2. What were the problems faced by the learners who dropped out of the course? In order to answer these questions, we applied a mixed 1 Gamification is defined as the use of game design elements in non-game contexts. Therefore, nowadays, the concept of game can be extended to gamification and therefore, it can be considered as a strategy promoting active learning. method approach and more specifically, a Convergent Parallel Design [15]. According to this design, qualitative and quantitative data were collected and processed to provide a more comprehensive interpretation of the information gathered and a better exploration of the research question. 2.2 Context of the Study The study was conducted in a MOOC offered by a Spanish university and deployed in the Canvas Network platform. The course topic was translation in the field of business and economics. The course spanned eight weeks. Each module mainly consisted of video lectures, readings, extra material/resources, discussion forums, optional and mandatory individual and/or collaborative activities. The certificate was issued to the participants completing all the compulsory activities (one per week). The estimated participant workload was 3 hours per week. The course included two active learning strategies: gamification and collaborative learning. Regarding collaborative learning, the course included: one optional activity (in which learners created a collaborative glossary), two compulsory group activities (the learners were divided into groups of five-six members to prepare and submit a common proposal of (a) a set of terms and (b) a translation); and, two compulsory peer review activities (the learners had to review and evaluate activities done by other participants). Regarding the gamification component, eight badges and redeemable rewards (e.g., extra attempts in quizzes) were designed and incorporated together with optional activities (e.g., introduction of terms in the glossary, high quiz performance) in order to promote student active learning and engagement. From a total number of 866 students who enrolled in the course, 169 of them completed the compulsory assignments and received the course completion certificate (19.52% completion rate). The three course instructors provided feedback and support to participants through private messages and discussion forums. 2.3 Data Collection and Analysis In this mixed method study, multiple data sources were used (Table 1). To increase the rigor and the credibility of this study, several strategies were applied [16] such as: triangulation of six data sources, peer debriefing among the members of the research team especially during the refinement of the questionnaires’ items and provision of deep descriptions of the context of the study. Quantitative data were processed using the R Studio software and Microsoft Excel. Concerning the qualitative data, content analysis was employed using both etic and emic categories and themes during the coding process [17]. The emergent categories arose from the analysis of the answers of the learners and from the analysis of the course design. Following the Convergent Parallel Design, we triangulated the findings from both data sources to answer the two IQs that guided the analysis. These findings are presented in the following section, together with excerpts of evidence that support them. Table 1. The data sources used in the study Label Data source N Description [Post_Quest] Post-course Questionnaire 174 Questionnaire distributed at the end of the course regarding the difficulties that the learners faced. The questionnaire was composed of open-ended and closed questions, including nine 4-point Likert-scale items (ranging from I strongly disagree to I strongly agree and an I don’t know/No answer option) [Drop_Quest] Dropout Questionnaire 69 Questionnaire administered after the course to dropout learners to inquire about the reasons for such dropouts. This questionnaire consisted of two multiple choice and one openended items. The participants were required to indicate the aspects of the course that were difficult to follow and to suggest improvements that would have helped them continue with the course [General Forums] Discussion Forums’ Posts 156 Learners’ messages (entries) in the discussion forums in each module [Group_Forums] Discussion Group Forums 2.213 Learners’ messages (entries and replies) in the group discussion forums in both collaborative activities [Priv_Mess] Private Messages 39 Messages sent (asynchronously) by the participants to the instructors [Logs] Logs 69 Trace data about learners’ learning activities performed in the MOOC platform (pages visited, task submissions; time spent in the course). The data were retrieved from the Canvas Network platform. These logs were processed to generate numeric data 3 Results This section describes the main findings associated to the IQs mentioned in the previous section. Each finding is supported with different excerpts of evidence (see Table 2). I.Q.1. What were the problems faced by the learners who completed the course successfully? From the evidence gathered, most of the problems reported by the learners were related to collaborative activities. At the post-questionnaire, 64% of participants reported problems regarding collaboration. Additionally, 16,9% of the learners mentioned difficulties in completing the group activities due to absence of communication among the group members and the different time-zones (see Table 2, [Post_Quest]-A, B). The analysis of the discussion forums complemented the evidence coming from the post-questionnaire; the most intensively reported problem (N = 16 out of 29 entries) was about the collaborative activity in week 4. The fourth week was the one in which the collaborative activities were introduced. Learners indicated that only a few members of the groups were working and as a result additional effort was needed from the remaining active members (see Table 2, [GeneralForums]-A, B). However, at the sixth week, in which the second collaborative activity was conducted, there were no more posts in the general forums reporting collaboration-related problems. The problems with the collaborative activities were further explored in the group forums which were created in the fourth and sixth week of the course to ease the communication among the members of the groups. During the fourth week, 944 messages were exchanged among the group members. From those, we encountered 23 posts that referred to problems with collaboration. These were posts regarding the absence of group members (out of the six members only two/three members were active) and had to do additional work to complete the activity (see Table 2, [Group_Forum]-A, B), and posts regarding the division of the workload among the members. At the sixth week a total number of 1.219 messages were exchanged in group forums and only four of them reported problems caused by non-active group members. The workload demanded in the course was another problematic aspect reported by many learners. At the post-questionnaire, 53% of learners reported difficulties in submitting the assignments on time. 12,7% of the participants affirmed in the openended question of the final questionnaire that the mandatory activities took much time, thus hindering the completion of the optional tasks. Furthermore, the majority of private messages to instructors (N = 14 out of 39 messages) was related to submission issues, where the learners asked for deadline extensions (see Table 2, [Priv_Mess]-A and [GeneralForums]-D). Among the other problems identified by the learners, activity-related issues were prominent. In the post-questionnaire, 32% of the learners stated that they faced difficulties in understanding the course concepts. Also, 7,9% of the learners reported problems related to course assignments (such as (i) too much conceptual explanations with less practical application; (ii) difficulty to understand some concepts and complete the associated activities; (iii) problems in peer-review activities regarding the evaluation received; and (iv) the feeling of not being able to assist the group members (see Table 2, [Post_Quest]-A, B). Similarly, the second most frequent reason for contacting the instructors through private messages regarded problems with the activities (N = 8 out of 39), where the participants were asking either for clarifications of the concepts, or for corrections of the obtained scores (see Table 2, [Priv_Mess]- B). Forums also revealed posts about general course clarifications related to assignments’ tasks or to the correct answers of the activities (see Table 2, [GeneralForums]-C). Additionally, many participants reported being puzzled about their next steps in the course (see Table 2, [Priv_Mess]-C, D). Problems regarding technical difficulties (such as links that didn’t work) were also mentioned (see Table 2, [GeneralForums]-F and [Post_Quest]-C). I.Q.2 What were the problems faced by the learners who dropped out of the course? To discover if the problems mentioned above led some participants to drop out of the course in the intermediate weeks, we contacted via email with 468 dropout participants and we received 69 answers. In this study, dropout learners are considered those who filled out the first obligatory questionnaire but did not complete at least one obligatory submission by the end of the course. The dropout learners reported insufficient time as the main reason of dropping out of the course (N = 44) stating among other reasons that “they could not combine their Table 2. Selected excerpts of evidence Data source Excerpts [Post_Quest] A. The only problem I faced during the course was that coordination in the group was not an easy task, probably due to time differences between participants and the poor communication B. Group activities seem a good way to work in order to learn, but I do not think they are suitable for this type of seminar. In the end, I get the feeling that you work worse than when you work individually, due to the lack of time or the different schedules we all have C. As I mentioned, I do not know why the videos of the last two blocksdid not open and the translations neither. In the previous blocks I accessed videos and translations without problems [GeneralForums] A. Hello! What happens if from the group of 6 only two people propose terms when it is time to deliver? Are we two the responsible for gathering the 20 terms? I tried to communicate with the other members of the group, but I cannot find how to send them a message and this doubt arose for the hypothetical case that they do not appear in the group forum B. Hello! I have a question about the group task. In my group, for the moment, we are only two people participating. If we have to upload 20 terms, I’m afraid there will not be much to discuss. What do we do in this case? Regards C. Good morning, I have a question regarding the analysis of the texts. Although someone had already asked this, I still do not understand very well what we should do exactly. Do we have to do an analysis in which we compare the two texts or an analysis for each of the texts, that is, in the end we would have to complete two analyses or only one? Thanks in advance D. Hello! When I tried to answer the mini survey, it was not enabled. I thought that it could be done at any other moment and now I see that the survey has been closed. Isn’t there any other possibility to do it? Thank you, greetings! E. Hello, I have tried to insert my translation, but I do not see where to deliver the task. Can you send me the link? Regards F. I have tried to access the texts to perform the obligatory task of this block in two different computers and in different browsers, but I cannot access any text. I do not know what to do (continued) Table 2. (continued) Data source Excerpts [Group_Forum] A. As I see that nobody responds, at the risk of passing us from the delivery time limit, I will submit the following terms [..] B. Hello! Will someone in the group work on this group task? [Priv_Mess] A. I get in touch with you to indicate a problem that has arisen to two other learners of the course and me. From the 23rd to the 30th of April we have a few days of the master’s degree that we are studying in Brussels and we will not be able to complete the last task in the established time. Would there be any possibility of doing it before or after those dates? B. I would like to know why I am incorrect about the answer of the question 13, since it is what the article by Andrea Rosalia Olteanu says, page 30 C. Once I have qualified in the rubric, what do I do? D. Hello! I made the revisions; however, I do not know if they were stored and completed. Thanks for your help [Post_Quest] A. Without having experience in economic translation [..] it has been sometimes difficult to understand certain concepts/ terms. For this reason, some of the translation tasks have turned out to be more complex than expected. In general, I think it has been an intense course. [..] B. Difficulty in correcting others and knowing what was right/wrong of my corrections. It would have been useful at the end of the course or of each task to have the correct results from the teachers [Drop_Quest] A. When I started the course, I had more time but with two jobs finally I had to leave it due to lack oftime B. Unfortunately, I did not evaluate the required time correctly, since my availability was much more restricted job responsibilities with the course requirements” (see Table 2, e.g. [Drop-Quest]-A, B). The next most reported reason was the different student expectations about the course content and motivation (N = 10). A smaller group of participants (N = 9) stated learning difficulties as reasons for quitting the course such as problems with the concepts, the lower background knowledge and the need of extra support. Regarding the collaborative activities (which was reported as the main problematic issue of the learners who completed the course), posts related to that issue were not detected except from one participant expressing her preferences to work individually. To obtain more insights about the possible reasons for dropouts, we explored their behavior in the course. Results show that most of them (N = 35) completed the first and/or second compulsory activities (quizzes in week 1 and week 2). However, only 30 reached week 4, from which a few participated in the first collaborative activity (N = 4). Moreover 13 learners interacted with gamification by claiming and earning at least one reward. Finally, the private messages of dropout learners were explored as well as the answers provided by the instructors. The messages were sent at weeks 4, 5 and 6 to inform the instructors about the participants’ intention to leave the course. 4 Discussion and Conclusion This study explored the problems of MOOC learners in a course implementing active learning strategies aiming at stimulating the interaction among the participants and their general engagement. The findings suggest that (a) introducing active learning strategies can generate additional problems to learners and (b) issues that can be challenging for the learners who follow and complete a course can be different from those faced by the dropout learners. The aspect reported as most problematic from the learners who completed the course regarded the collaborative activities. Yet, we did not encounter many relevant complaints about this aspect the second time they had to submit a collaborative task. This suggests that problems regarding the organization of collaboration mainly appear the first time that the learners are exposed to this method and become less prominent after they learn how to deal with them. At the same time, given that the majority of the unmotivated or non-active learners had dropped out by the second collaborative activity, groups were probably composed of learners who engage more with the activity and with each other. Further work is needed, however, to determine whether the difference of reported problems between the course weeks is related to the composition of the groups and/or the type of collaborative activities. On the other hand, no problems were reported about the gamification implemented in the course. We need to consider, nevertheless, the fact that the gamified activities were associated with optional tasks and only the learners who were interested in getting rewards were involved in their attainment. Thus, further exploration is needed to analyze if compulsory gamified activities would pose barriers to the learners during their experience in a MOOC. The evidence gathered revealed additional problems concerning time-related, activity-related and technical issues, which are also reported in the literature [9–14]. However, while the results of the reviewed works relied on one single data source, our study followed a more thorough process, exploring the perspective of the student problems from multiple data sources. Moreover, unlike the results reported in [8, 10, 12] we found limited evidence regarding the lack of support as one of the barriers of the learners. This fact could be explained by the continuous help provided by the instructors and their timely responses to posts both in discussion forums and in private messages, an infrequent strategy in MOOCs. Among the total number of 156 messages posted in discussion forums by the learners, 269 answers were provided by the instructors with the maximum waiting time for a reply to be one day. Our study points out that although active learning strategies can be challenging during the course, their inclusion under a careful design can help to overcome these challenges. Regarding collaborative activities, several group formation aspects and their adequate deployment for such massive, diverse and varied contexts should be taken into account. For example, one issue that should be considered is the possibility of grouping active learners with non-active ones, where a satisfactory design should come with alternatives. A study carried out in parallel to ours [18] discussed the benefits of homogeneous group formation to keep students engaged in the MOOC. Their results showed that, the second iteration of the collaborative activities were better, and that this could be partially due to the fact that the groups were formed using a better-informed data analytics algorithm, with data coming from the second half of the course (when most of the dropout learners had already abandoned). Concerning gamification, the design should challenge students and keep them motivated within the course. The positive fact of non-reported problems can be associated to design decisions, such as gamifying optional and diverse activities throughout the different weeks of the course, and to implementation decisions, such as placing the course gamification page inside the course interface and giving students the possibility of claiming the rewards once they completed the gamified conditions. Finally, a careful design should provide a proper scaffolding, as well, to overcome learners’ problems. This can be achieved with the active presence of the instructor throughout the whole learning process both reactively, assisting the learners, and ideally proactively foreseeing problems that can arise and preparing the adequate reaction. This study has limitations and further empirical work is required. First, still we need additional studies to detect learners’ problems in a consistent way and to explore how to provide the adequate support. Second, self-reported data studied in this work were gathered only at the end of the course, when learners were already disengaged. During the course we only gathered evidence from the logs and the learners-to-learners and learners-to-instructor interactions. To overcome this limitation, in our next study we plan to explore learners’ issues by collecting self-reported data in real-time during the course. Finally, the collaboration activities were compulsory while the gamified activities were associated with optional tasks. In short term, we will explore MOOC learners’ problems in different contexts that implement active learning strategies. Acknowledgements. This research has been partially funded by the European Regional Development Fund and the National Research Agency of the Spanish Ministry of Science, Innovations and Universities under project grants TIN2017-85179-C3-2-R and TIN2014-53199- C3-2R, by the European Regional Development Fund and the Regional Ministry of Education of Castile and Leon under project grant VA257P18, and by the European Commission under project grant 588438-EPP-1-2017-1-EL-EPPKA2-KA. The authors thank the rest of the GSIC-EMIC and Canvas Network team for their support. Special thanks to Juan I. Asensio-Pérez (Universidad de Valladolid) for his valuable ideas and his constant guidance conducting this research. References 1. Onah, D., Sinclair, J., Boyatt, R.: Dropout rates of massive open online courses: behavioural patterns MOOC dropout and completion: existing evaluations. In: Proceedings of 6th International Conference on Education and New Learning Technologies, pp. 1–10 (2014) 2. Ferguson, R., Sharples, M.: Innovative pedagogy at massive scale: teaching and learning in MOOCs. In: Rensing, C., de Freitas, S., Ley, T., Muñoz-Merino, Pedro J. (eds.) EC-TEL 2014. LNCS, vol. 8719, pp. 98–111. Springer, Cham (2014). https://doi.org/10.1007/978-3- 319-11200-8_8