To reward and beyond: Analyzing the effect of reward-basedstrategies in a MOOC

To Reward and Beyond: Analyzing the Effect of Reward-Based Strategies in a MOOC Alejandro Ortega-Arranza,∗, Miguel L. Bote-Lorenzoa, Juan I. Asensio-P´ereza, Alejandra Mart´ınez-Mon´esb, Eduardo G´omez-S´ancheza, Yannis Dimitriadisa aSchool of Telecomm. Engineering, Universidad de Valladolid, p.ode Bel´en 15, 47011 Valladolid, Spain bSchool of Computer Engineering, Universidad de Valladolid, p.ode Bel´en 15, 47011 Valladolid, Spain Abstract Despite the benefits of MOOCs (e.g., open access to education offered by prestigious universities), the low level of student engagement remains as an important issue causing massive dropouts in such courses. The use of reward-based gamification strategies is one approach to promote student engagement and prevent dropout. However, there is a lack of solid empirical studies analyzing the effects of rewards in MOOC environments. This paper reports a between-subjects design study conducted in a MOOC to analyze the effects of badges and redeemable rewards on student retention and engagement. Results show that the implemented reward strategies had not significant effect on student retention and behavioral engagement measured through the number of pageviews, task submissions, and student activity time. However, it was found that learners able to earn badges and redeemable rewards participated more in gamified tasks than those learners in the control group. Additionally, results reveal that the participants in the redeemable reward condition requested and earned earlier the rewards than those participants in the badge condition. The potential implications of these findings in the instructional design of future gamified MOOCs are also discussed. Keywords: Gamification, MOOC, Engagement, Retention, Between-Subjects Design 1. Introduction The importance and presence of MOOCs in digital education has been growing up since their appearance in 2008 (Siemens,2013), providing multiple benefits such as access to high quality open learning, accreditation from prestigious universities, or the creation of communities around a shared topic (Siemens,2013;Ferguson & Sharples,2014;Deng et al.,5 2019). The increasing number of courses and enrollments (Shah,2017) indicates the growing interest of society in MOOCs. Nevertheless, MOOCs still fail to motivate and engage ∗Corresponding author Email addresses: [email protected] (Alejandro Ortega-Arranz), [email protected] (Miguel L. Bote-Lorenzo), [email protected] (Juan I. Asensio-P´erez), [email protected] (Alejandra Mart´ınez-Mon´es), [email protected] (Eduardo G´omez-S´anchez), [email protected] (Yannis Dimitriadis) Preprint submitted to Computers & Education, DOI: https://doi.org/10.1016/j.compedu.2019.103639 learners with course contents and learning activities (Khalil & Ebner,2014) leading to low participation and high dropout rates (Jordan,2014;Alario-Hoyos et al.,2014). Some dropouts can derive from the different profiles enrolling in MOOCs (Kizilcec et al.,10 2013;Ferguson & Clow,2015). However, many dropouts are produced as a side effect of the pedagogical models and instructional designs used (Margaryan et al.,2015;Henderikx et al., 2017). Some research works have shown the benefits of strategies promoting active learning to improve engagement (Ferguson & Sharples,2014;Hew,2016), thus trying to decrease the dropout rates and the low levels of participation. Gamification is one of these strategies15 that has attracted the attention of MOOC practitioners during the last years (Davis et al., 2018), due to the benefits already shown in other educational contexts (De Sousa Borges et al.,2014;Dicheva et al.,2015). Gamification is the application of elements and structures that frequently appear in games (e.g., narrative, rewards) into non-game contexts (Deterding et al.,2011;De Sousa Borges20 et al.,2014) such as those involving online learning. According to previous literature reviews (Hamari et al.,2014;Dicheva et al.,2015;Ortega-Arranz et al.,2017), one of the most used game elements in online educational contexts are rewards, thus generating the so-called reward-based gamification (Nicholson,2015) or incentive systems (Kyewski & Kr¨amer,2018). In this type of gamification, students are awarded with game elements (rewards) integrating25 asignifier (e.g., name, visual, description) when a completion logic (i.e., relevant actions defined by the teachers beforehand) is satisfied (Hamari & Eranti,2011;Hamari,2017). Previous studies integrating reward-based gamifications in online and blended learning courses have shown positive benefits on student retention (Khalil et al.,2017;Krause et al., 2015) and engagement (O’Donovan et al.,2013;Ding et al.,2017;Ib´a˜nez et al.,2014;Barata30 et al.,2013;Anderson et al.,2014). However, none of such studies was performed in real MOOC environments with a significantly heterogeneous set of participants, and therefore extrapolating their results to MOOCs is venturesome. Additionally, most of these studies analyze the effects of several game elements as a whole instead of isolating the effects of specific reward types, and do not compare the effects of different types of rewards on learners.35 Finally, previous literature reviews on gamification in MOOCs reflect the lack of empirical evidence regarding their effects (Ortega-Arranz et al.,2017;Antonaci et al.,2017;Khalil et al.,2018). Evidence-informed design could be greatly useful for instructional designers and MOOC instructors to better understand the effects of concrete rewards on student engagement and40 retention, and to better align their MOOC learning designs with the gamification goals. Moreover, researchers can advance in the understanding of gamification benefits for MOOC learners. Thus, the generic underlying research question that leads this study is To what extent reward-based gamification strategies can foster student retention and engagement in MOOCs?45 To address this question, we have conducted a between-subjects study (Charness et al., 2012) considering three different conditions to which participants were randomly assigned (two experimental conditions, each with a different type of reward, and one control group), in an 8-week MOOC with 866 enrolled students. In this study, students belonging to experimental conditions had to explicitly claim the rewards to earn them once they satisfied50 2 the gamified tasks. Therefore, we could better analyze the effects and motivation behind rewards and avoid influencing those students who were not interested in them. The structure of the paper is as follows. Section 2provides a brief overview of similar research works studying with the effects of rewards in MOOCs and formulates the hypotheses of this study. Section 3explains the design of the study including the sample, the context,55 the gamification design and the data sources. Then, the results are presented (Section 4) and the findings are discussed regarding the implications of rewards in the instructional design of gamified MOOCs (Section 5). Finally, conclusions, limitations and ideas for future research are introduced in the last section. 2. Reward-Based Strategies in MOOCs60 This section first describes previous studies investigating the effects of reward-based strategies on student engagement and retention. Then, the theoretical background supporting the formulation of the hypotheses addressed in this study is introduced. 2.1. Related Work There exist several previous studies dealing with the effect of gamification in online65 and blended learning environments (Dicheva et al.,2015;Khalil et al.,2018). However, to the best of our knowledge, only five of these studies have been performed in a MOOClike context1and isolate the effect of a single type of reward (typically, badges) instead of studying the effect of multiple game elements as a whole. These similar works are described below.70 Cross et al. (2014) implemented a series of badges associated with a variety of activities in two MOOCs. Badges were requested by learners and manually issued by instructors and other peers. Although results showed positive opinions and attitudes about badges, the study focused on understanding the reasons behind badge acquisition rather than in the effects on engagement and retention.75 Ortega-Arranz et al. (2019) carried out a mixed method analysis to understand the correlation between learners’ actions towards earning badges and their behavioral engagement in a MOOC. Results revealed a positive correlation between such parameters but causality was not analyzed and the effects of badges on student retention were not examined. Anderson et al. (2014) performed a quantitative analysis of the effects of badges applied80 to discussion forums in a MOOC with 112.897 enrolled students. Although results showed that forum participation was higher than in a previous run of the same MOOC, and that the participation increased when the next available badges were clearly visible, no further analysis was performed on student retention and engagement. Khalil et al. (2017) studied the effects of implementing a meter in form of a battery bar85 whose charge increases when predefined conditions are met (e.g., attempt a quiz, reading 1MOOCs present different features with respect to traditional environments which can have an impact in the expected outcomes caused by gamification strategies (e.g., learning design activities, massive and heterogeneous set of participants, lack of time to complete the course) (Ortega-Arranz et al.,2019). 3 and posting in forums) in a MOOC with 284 enrolled students. The study presented a higher percentage of active and certified students compared with two previous runs of the same MOOC without gamification. Nevertheless, the game element used was a ‘battery bar’ (non-collectable reward) and the analysis was restricted to the student retention level90 without isolating the effect caused by the context (i.e., different versions of the course). Similarly, Kyewski & Kr¨amer (2018) divided course participants into three different conditions regarding the visibility of course badges to understand the effects of badges on learners’ motivation, performance (grades) and participation (logins, quiz participations and attempts, resource access). No matter the condition, student’s motivation decreased over95 time, and learners in the experimental conditions did not participate more actively in the course nor got higher scores than those in the control group (no badges). However, the study was not performed in a real MOOC context. Instead, the context of the study was an online one-semester seminar in a higher education setting. In contrast with previous research works, the between-subjects study presented in this100 paper analyzes and compares the effects of two reward types (i.e., badges and redeemable rewards) on student retention and engagement. Additionally, this study was carried out in a real MOOC environment with an heterogeneous set of participants (e.g., age, location), where learners had to explicitly claim the rewards, which provided more variables to complement the analysis of the effects on students’ engagement in the experimental conditions105 (e.g., the number of students claiming rewards, the claiming time stamp). 2.2. Theoretical Background Although there is some controversy about the effects of rewards in educational environments (e.g.,Deci et al. (2001); Seaborn & Fels (2015)), researchers have identified several reasons behind the interest of achieving rewards in educational learning environments such110 as intrinsic and extrinsic motivation (Nicholson,2015), sense of progression and goal accomplishment (Hamari,2017) or simply fun (Codish & Ravid,2014). Reward strategies in online environments are frequently implemented through different game elements such as points, badges or levels (Dicheva et al.,2015;Ortega-Arranz et al.,2017) aiming to increase the student motivation and engagement and to mitigate the high dropout rates. To this115 end, some authors have proposed the use of specific game elements depending on the human desires that the designers want to promote (Zichermann & Cunningham,2011;Bunchball, 2010). Chang & Wei (2016) identified badges and redeemable rewards as two of the most engaging game elements to be used in MOOC environments. Badges are optional rewards, represented with graphical icons and issued when users sat-120 isfy predefined requirements typically associated with non-compulsory activities (Dom´ınguez et al.,2013;Hamari,2017). Redeemable Rewards are rewards which provide students with a certain privilege during course runtime (e.g., extra attempts in quizzes, access to extra content, join a queue to receive feedback from teachers) (O’Donovan et al.,2013;Ortega-Arranz et al.,2018). These two reward types have been also positively evaluated by students in125 other MOOC studies (Cross et al.,2014;Rizzardini et al.,2016). In this study, we will focus on these two types of rewards to understand and compare their effects on student retention and engagement. 4 2.2.1. Rewards and Retention in MOOCs Several authors have identified factors that are important to help increase student reten-130 tion in MOOCs (Adamopoulos,2013;Hone & El Said,2016), among which gamification is considered as a potential strategy to this end. For example, Rizzardini et al. (2016) proposed the addition of gamification elements in a conceptual model for MOOCs to decrease their attrition rates. Similarly, Borr´as-Gen´e et al. (2016) created a gamified cooperative MOOC model for the design of engineering education MOOCs to improve the student motivation,135 learning level and completion rate. Previous studies have shown that reward strategies can serve students as elements of progression and goal accomplishment, encouraging them to keep track of their learning and performance (Hamari,2017). Empirically, Krause et al. (2015) performed a betweensubjects design study where experimental conditions implementing reward-based strategies140 in a SPOC reduced the dropout rate in a 25%. Also, the study performed by Khalil et al. (2017) showed that the attrition rates were much lower in a MOOC with a reward element than in two previous versions of the same MOOC without gamification. Therefore, according to the previous studies and assuming students will try to earn the rewards associated with course activities as a sign of progression and goal accomplishment,145 the first hypothesis for this study is: •H1: A higher retention level will occur for participants under the Experimental conditions. 2.2.2. Rewards and Engagement in MOOCs One of the indirect causes of student retention is the student engagement with the course150 content and activities (Khalil & Ebner,2014). Several authors have proposed the inclusion of gamification strategies in online environments to promote student engagement, mainly based on two theories: the Flow Theory (Csikszentmihalyi,1991) and the Self-Determination Theory (Ryan & Deci,2000). Adapting from the former theory, when teachers design gamified activities that challenge students and keep them in the flow2zone, students will maintain155 their engagement throughout the course (e.g.,Zhu et al. (2017); Antonaci et al. (2018)). The Self-Determination Theory proposes the satisfaction of three human psychological needs to promote student motivation: competence, relatedness and autonomy. Satisfying these three needs with the gamification elements and their associated tasks is likely to increase student engagement in the course activities (e.g.,Borr´as-Gen´e et al. (2014); Seaborn & Fels (2015)).160 Therefore, if reward-based strategies are aligned with the activity difficulty level and target the previous psychological needs, they can be potentially used as motivators to engage students in MOOCs. According to Fredricks et al. (2004), there are three types of engagement: cognitive, emotional and behavioral. In this study, we will focus on behavioral engagement as the observable behaviors that represent the student progress and learning in165 the course. Previous studies have empirically tested the positive effects of reward strategies 2Flow is defined as a state of absorption characterized by intense concentration, loss of self-awareness, a feeling of being perfectly challenged and a sense that time is flying (Csikszentmihalyi,1991). 5 on behavioral engagement in online and blended educational environments (Barata et al., 2013;Ib´a˜nez et al.,2014;Ding et al.,2017;Ruip´erez-Valiente et al.,2017;Ortega-Arranz et al.,2019). Therefore, the second hypothesis of this study is: •H2: Participants in the Experimental conditions will show a higher behavioral engage-170 ment level than participants in the Control group. In the current context, the use of reward-based strategies provides researchers with more variables defining the student behavioral engagement within the course. Some of these additional variables are the completion of the task associated with rewards, the time needed to complete the task or the time needed to claim the reward. This additional set of measures175 of behavioral engagement corresponds to the so-called reward-derived student engagement. Some previous studies support the use of rewards as a mean to promote student specific actions (i.e., actions associated with reward conditions) (Anderson et al.,2014;Hakulinen et al.,2013;Hamari,2017). Learners interested in earning rewards would be expected to show higher engagement through the early completion of the reward conditions. Therefore,180 according to previous studies, the third and fourth hypothesis of this study are: •H3: A higher percentage of students in the Experimental conditions will perform the instructor predefined actions associated with reward conditions, as compared to the Control group. •H4: Students in the Experimental conditions will satisfy the gamified conditions sooner185 than the students in the Control group. Another variable representing the student engagement when rewards are implemented, is the time elapsed from the moment that a student satisfies the reward conditions to the moment s/he claims and receives such reward. This variable can potentially inform us about the student motivation and interest on earning such reward in a massive online learning190 environment (Berger et al.,2016). The evidence that a student claims the reward right after completing the conditions suggests that s/he was aware of the existence of this reward and wanted to earn it. In this regard, redeemable rewards have been ranked in the 2nd position as the most engaging game element in MOOCs over badges and trophies in the 5th position (Chang & Wei,2016). Additionally, although redeemable rewards can be collected after195 the expiration date of the associated privilege, they are expected to be earned and used before (e.g., students are expected to earn a privilege associated to the second week of the course before the end of such week). We believe that students will try to earn redeemable rewards sooner than badges due to expiration of the associated privileges in the different course modules. Therefore, the fifth hypothesis of this study is:200 •H5: Students in the Redeemable Rewards condition will request the rewards sooner than students in the Badge condition. 6 Condition Enrolled Students Participants of the Study BADGE 290 223 REDEEM 287 205 CTRL 289 220 Total 866 648 Table 1: Number of MOOC enrolled students and participants of the study per condition. 3. The Study 3.1. Sample The students of the course were randomly assigned to one of the following conditions205 once they enrolled in the course: •BADGE: Students involved in this condition were able to obtain up to 8 badges throughout the course. •REDEEM: Students involved in this condition were able to obtain up to 8 redeemable rewards whose requirements were the same as the badges.210 •CTRL: Students involved in this condition had neither rewards nor game elements implemented in the course. This condition was considered as the control group of the study. This group assignment by enrollment date avoided bias caused by those students registering late who are more prone to disengage with the course (Gurantz,2015). Among 866215 learners enrolled in the course3, 648 submitted the initial questionnaire, allowing them to access the course contents and activities. Additionally, by submitting the initial questionnaire, learners provided their consent to analyze their data with research purposes, thus allowing us to profile the participants of this study (see Table 1). According to the data reported in the initial questionnaire, the students of this study were220 mostly female (83.02%), between 20-30 years old (64.17%), from Latin America (53.86%), with an undergraduate background (56.17%), and medium knowledge level about the topic of the MOOC (39.81%), planning to actively participate in the course (58.64%), without previous MOOC and gamification experience (69.60% and 60.49% respectively), and with positive beliefs about the benefits of using gamification in educational environments (64.20%).225 Before testing the hypotheses of this study, we checked the homogeneity of the three conditions regarding the variables of the initial questionnaire including the student gender, age, background knowledge level and type of participation in the course. If groups are homogeneously distributed, the differences in the composition of the groups are not likely to influence the results of the study. To this end, we have conducted a Chi-square homogeneity230 7 Variable p-value Exclusions (number of answers) Age 0.954 DK/NA (2): R (1), C (1) Gender 0.914 DK/NA (3): B (1), R (1), C (1) Background 0.121 DK/NA (2): R (1), C (1) Location 0.875 Asia (2), DK/NA (2): R (2), C (2) Knowledge level 0.531 None (13), DK/NA (2): B (6), R (3), C (6) Participant type 0.836 - MOOC experience 0.928 DK/NA (2): R (1), C (1) Gamification experience 0.573 - Gamification beliefs 0.249 - Table 2: Chi-square test for homogeneity p-values regarding the variables of the initial questionnaire (DK/NA=Don’t Know/No Answer; B=BADGE, R=REDEEM, C=CTRL). P-value is significant at <.05 level (two-tailed). test (Kirch,2008) for every variable considered in the initial questionnaire4. According to the results (see Table 2), the p-values are much higher than the significance level (.05) for every variable. Therefore, the degree of similarity among groups regarding the variables measured in the initial questionnaire is high and the results obtained in this study are unlikely to be caused by the composition of the groups.235 3.2. Course Context The study was conducted within an 8-week instructor-led MOOC offered by a Spanish university from March, 12th to May, 6th, 2018. The course was published and launched in the Canvas Network platform5. The topic of the course was related to translation from English to Spanish in the business and economic field, and the content was divided into240 7 weekly modules plus one extra week to complete the activities. The modules included videos, content pages, recommended readings, discussion forums and individual and collaborative activities (see Fig. 1). The activities were classified into compulsory and optional. Students had to submit all the compulsory activities in order to receive the course completion certificate. For all activities, the submission was due eight days after the release of245 the activity although in some activities the due date was extended a few days according to the instructor criteria. The course team was composed of one instructor and two teacher assistants. Furthermore, the enrollment was closed in the second week of the course to avoid group management problems in the collaborative activities. Teaching and technical support was offered by the course team and the researchers respectively, through private messages250 and posts in forums. 3A simple random process was applied to assign participants to the different conditions but the number of enrolled students in every condition is different due to the removal of test users and duplicated accounts. 4The Chi-homogeneity test assumes a minimum number of frequencies of every multiple choice option (freq.>5). In our case, some questions were answered with a frequency under this value. As a consequence, these values have been removed or grouped as other answer representing a maximum number of 15 excluded answers among the three groups (i.e., 2.31% from the total number of answers) as described in Table 2. 5Canvas Network: https://www.canvas.net, last access: December, 2018. 8 Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Module Video-Introduction Video Contents + Recommended Readings Discussion Forum Content Questionnaire Module Video-Summary Peer Review: Text Translation Content Questionnaire Content Questionnaire Text Analysis Term Extraction (individual) Text Translation Text Selection (individual) Peer Review: Text Translation Glossary Parallel Text Search Term Extraction (group) Text Translation Peer Review: Text Analysis Peer Review: Text Translation Text Selection (group) Text Translation Certificate Req. Course Experience Questionnaire Self-Review: Text Translation Goodbye Forum Course Info Platform Info Twitter Account Facebook Page Social Forum Resource Forum Welcoming Questionnaire Week 0 Compulsory Activities Resources and Optional Activities Figure 1: Learning design of the MOOC under study. Group BADGE Group REDEEM Week Cond. Image Name Condition Privilege Week Priv. 0Welcome! Update your profile picture and introduce yourself in the Social Forum Get 3 more attempts in Quiz 1 and Quiz 2 1, 2 1, 2 Quiz Master! Get a score, equal or higher than 90% in Quiz 1 and Quiz 2 Get access to extra content in week 2 2 2Glossary Master! Contribute with at least 3 terms in the Glossary activity Extend the due date of the compulsory task at week 3 3 3Text Provider! Share a text in the Parallel Text Search activity and receive 5 likes from other participants Extent the due date of the text translation task at week 5 5 5Expert Reviewer! Review 2 more submissions from your colleagues (4 in total) in week 5 Join the queue so that the instructors evaluate your work and provide feedback 5 5Smartie! Get a score, equal or higher than 70% in the reviews performed by other peers regarding your submission Get 20 more minutes in Quiz 7 7 4, 6 Translation Master! Submit the optional translations: Public Descriptive text and Expositive Private text Get 3 more attempts in Quiz 7 7 6Graduated! Watch the "summary videos" in weeks 1 to 6 Get access to an exclusive video-session with the teacher and other students 7 Figure 2: Gamification design implemented in the MOOC of the study. 9 BADGE REDEEM N % Date (µ)N % Date (µ) Welcome 56 77.78 23/03 22:08:27 56 80 20/03 03:01:47 Quiz 33 64.71 31/03 00:49:54 31 77.5 27/03 20:13:57 Glossary 25 89.29 27/03 15:47:03 32 82.05 26/03 23:54:53 Text Prov. 14 82.35 07/04 11:15:09 19 86.36 07/04 15:38:37 Reviewer 33 62.26 17/04 12:21:20 23 51.11 15/04 20:07:43 Smartie 41 59.42 20/04 19:51:15 29 51.79 17/04 11:57:25 Translator 10 83.33 25/04 18:09:09 5 71.43 23/04 22:11:50 Graduated 29 82.86 25/04 11:55:54 19 63.33 23/04 04:30:04 Table 9: Statistical summary of participants claiming and earning course rewards. After performing a Wilcoxon signed-rank test, the Rvalue obtained when comparing the BADGE and REDEEM group with the CRTL group is 1 in both cases, which is over the380 critical value = 0 (two-tailed, alpha=.05, n=614) (Navidi,2008). Therefore, we can conclude that the median weights of the dates when students satisfy the gamified-task conditions in the BADGE and REDEEM groups are close to be significantly different from the median weight in the CTRL group (1 day, 9 hours and 4 minutes, and 1 day, 3 hours and 17 minutes later, respectively) with a p-value of 0.0625 for both groups. Furthermore, there are not385 significant differences between the median weights of the dates that students satisfied the reward conditions in the BADGE and REDEEM groups (p-value = 0.844). Surprisingly, results suggest that students in CTRL group performed the tasks associated to rewards earlier than the students in the experimental groups. However, this difference is not significant and therefore, Hypothesis 4 is not supported.390 4.5. Reward-derived Student Engagement: Claiming Dates (H5) Hypothesis 5 states that students in the Redeemable Rewards condition will request the rewards sooner than students in the Badge group. The statistical summary of claiming dates is presented in Table 9. After performing a Wilcoxon signed-rank test, the Rvalue obtained when comparing395 the BADGE with the REDEEM group is 1, which is under the critical value = 3 (two14In this hypothesis testing, we have removed the tasks associated to Text Provider and Smartie rewards because the fulfillment of the conditions depended on peers actions and not in the own student. 16 tailed, alpha=.05, n=8) (Navidi,2008). Therefore, we can conclude that the median weight of the dates when students claimed and earned the gamified-task conditions in the BADGE group is significantly different from the median weight in the REDEEM group (17 hours, 55 minutes and 4 seconds later) with a p-value of 0.016 and a very large effect size (r= 0.604)400 (Cohen,1988). According to the results, students from the REDEEM group claimed the rewards significantly earlier than those students from the BADGE group. Therefore, we can confirm the Hypothesis 5 and state that students in the REDEEM condition claimed and earned the rewards sooner (17 hours, 55 minutes and 4 seconds) than the students in the BADGE405 condition. 5. Discussion In this study, the gamification was co-designed with the instructor of the course to encourage learners carry out the optional activities which were considered important for their learning, and to indirectly enhance their engagement within the course.410 The analysis of the results presented in the previous section showed that the implemented reward strategies: (H1) did not lead to a higher student retention; (H2) did not increase the student behavioral engagement (measured in terms of number of pageviews, number of completed tasks, number of forum posts and activity time in the course); and (H4) did not encourage learners to perform earlier the optional activities. Although these results may415 contradict the results reported in many gamification studies in online and blended learning, they are in line with some studies performed in massive environments where gamification did not have the expected benefits on student retention and engagement (Rizzardini et al., 2016;Kyewski & Kr¨amer,2018). These results highlight the importance of the context (e.g., MOOCs) and the individuals (e.g., heterogeneity of learners) in the gamification design420 (Hamari et al.,2014;Seaborn & Fels,2015). On the other hand, the results also showed that the implemented rewards: (H3) significantly encouraged participants to satisfy the conditions associated with rewards; and (H5) affected the time when rewards were claimed (i.e., redeemable rewards were claimed sooner than badges), which could be interpreted as a higher student intentionality to earn425 such rewards. These results support the idea that rewards helped the instructor achieve her main goal when introducing gamification in the course: to encourage participants to perform optional tasks. Thus, the overall results suggest that those learners who are unlikely to complete the MOOC due to external reasons (e.g., lack of time, lack of previous knowledge, or lack430 of interest on the course contents), will neither be motivated or engaged with the reward strategies (regardless of the reward type used). However, it seems that reward strategies can potentially encourage learners who are already motivated to complete the course (e.g., interest on course topic and contents) to perform the optional tasks that would otherwise not be fulfilled.435 Finally, (H5) the sooner claiming of rewards by participants in the REDEEM condition suggests a higher extrinsic motivation caused by the associated privileges. In this case, 17 the privileges associated with deadline extensions of compulsory assignments and instructors’ feedback were the most valued by participants, which was also observed in a previous study conducted in a blended course (O’Donovan et al.,2013). This extrinsic motivation440 can be also used by MOOC designers and instructors to enhance the attainment of some specific pedagogical goals. However, the inclusion of these privileges into MOOC learning designs brings new variables in the alignment of the gamification design with the expected learning goals (e.g., what activities should incorporate privileges, which privileges should be implemented). Therefore, the development of guidelines and tools supporting practitioners445 to successfully put in practice this kind of gamification strategies in MOOCs appears as a promising line of future research. 6. Conclusions Despite the increasing number of works proposing the use of gamification strategies in MOOCs, there is a scarcity of empirical works testing the effects of such strategies (Ortega-450 Arranz et al.,2017;Antonaci et al.,2017;Khalil et al.,2018). In some cases, the literature has proposed the use of gamification as an easy-to-implement strategy to diminish some of the current drawbacks of MOOCs such as the attrition rates, the lack of motivation and engagement, the learner performance or the lack of interaction. This paper provides empirical evidence about whether reward-based gamification increases student engagement455 and retention in MOOCs. Although no significant differences were found on student general engagement and retention among the different conditions, participants in the experimental conditions (Badges and Redeemable Rewards) showed higher participation in tasks associated to rewards. This effect can be used by instructors to promote specific learning goals, such as the learners’ com-460 pletion of optional tasks beneficial for their learning (providing a good alignment between the gamification design and pedagogical goals). This study has some limitations as it is based on a unique MOOC oriented to Spanishspeaking population. It would be interesting to analyze to what extent this MOOC and gamification design (e.g., number of implemented rewards, type of activities associated to465 rewards, rewards only visible to students themselves, rewards associated with optional tasks) affected the results of this study, and if the adaptation of this design to other topics and contexts would have similar effects (Hamari et al.,2014;Seaborn & Fels,2015). Further evaluations involving reward-based strategies in other MOOCs with different topics, features, language and target population are needed to generalize the results of this study. In future470 versions of the same course, we plan to analyze the extent to which the aforementioned gamification parameters would change the effects on retention and engagement reported in this study. Participants in MOOCs can typically be classified according to specific profiles based on their background, goals and participation (Kizilcec et al.,2013). In order to avoid a bias475 caused by the different types of learners in our study, we performed a Chi-square homogeneity test, thus checking if there was any significant difference between the experimental and control conditions regarding variables such as the background, knowledge level and the 18 expected type of participation in the course. Results showed that conditions are homogeneously distributed according to these variables. As a future work, we plan to analyze480 the relationship between the different student types and the reward-derived behavioral engagement (e.g., if students with higher knowledge level of the course topic earned more rewards). Nevertheless, despite these apparent benefits, the design and implementation of such gamification strategies can be time-consuming and cognitively-costly for MOOC practition-485 ers. When using reward strategies, practitioners are responsible of (i) gamifying the learning design and aligning the pedagogical goals with the gamification intentions (e.g., create the reward-condition rules); (ii) implementing the gamification design in the platform (e.g., learn how to use the gamification platform, make changes in the gamification design according to the platforms capabilities); and (iii) managing the evolution of gamification during course490 runtime (e.g., watch over the effect of rewards on student behavior). All these gamificationrelated activities imply an extra time and effort (Dicheva et al.,2018) added to the existing work employed by practitioners to produce a MOOC. Such extra time and effort may hinder the use and adoption of these gamification strategies. As a future work, we plan to explore the affordability of GamiTool for MOOC practitioners.495 Acknowledgements This research has been partially funded by the European Regional Development Fund and the National Research Agency of the Spanish Ministry of Science, Innovations and Universities under project grants TIN2017-85179-C3-2-R and TIN2014-53199-C3-2-R; by the European Regional Development Fund and the Regional Ministry of Education of Castilla500 y Le´on under project grant VA257P18; and by the European Commission under project grant 588438-EPP-1-2017-1-ELEPPKA2-KA. The authors thank the rest of the GSICEMIC research team for their valuable ideas, and the Canvas Network team for the support received conducting this research. References505 Adamopoulos, P. (2013). What Makes a Great MOOC? An Interdisciplinary Analysis of Student Retention in Online Courses. In Proceedings of the 34th International Conference on Information Systems. Alario-Hoyos, C., P´erez-Sanagust´ın, M., Delgado Kloos, C., Parada G., H. A., & Mu˜noz-Organero, M. (2014). Delving into Participants’ Profiles and Use of Social Tools in MOOCs. IEEE Transactions on Learning Technologies,3, 260–266.510 Anderson, A., Huttenlocher, D., Kleinberg, J., & Leskovec, J. (2014). Engaging with Massive Online Courses. In Proceedings of the 23rd International Conference on World Wide Web (pp. 687–698). ACM. Antonaci, A., Klemke, R., Kreijns, K., & Specht, M. (2018). Get Gamification of MOOC right! How to Embed the Individual and Social Aspects of MOOCs in Gamification Design. International Journal of Serious Games,5, 61–78.515 Antonaci, A., Klemke, R., Stracke, C. M., & Specht, M. (2017). Gamification in MOOCs to enhance users’ goal achievement. In Proceedings of the 2017 Global Engineering Education Conference (pp. 1654–1662). IEEE. 19 Barata, G., Gama, S., Jorge, J., & Gon¸calves, D. (2013). Improving participation and learning with gamification. In Proceedings of the 1st International Conference on gameful design, research, and applications520 (pp. 10–17). ACM. Berger, P., Hennig, P., Bocklisch, T., Herold, T., & Meinel, C. (2016). A journey of bounty hunters: Analyzing the influence of reward systems on stackoverflow question response times. In 2016 IEEE/WIC/ACM International Conference on Web Intelligence (WI) (pp. 644–649). IEEE. Borr´as-Gen´e, O., Mart´ınez-N´u˜nez, M., & Fidalgo-Blanco, ´ A. (2014). Gamification in MOOC: Challenges,525 Opportunities and Proposals for Advancing MOOC Model. In Proceedings of the Second International Conference on Technological Ecosystems for Enhancing Multiculturality (pp. 215–220). ACM. Borr´as-Gen´e, O., Mart´ınez-N´u˜nez, M., & Fidalgo-Blanco, ´ A. (2016). New Challenges for the Motivation and Learning in Engineering Education Using Gamification in MOOC. International Journal of Engineering Education,32, 501–512.530 Bunchball, I. (2010). Gamification 101: An Introduction to the Use of Game Dynamics to Influence Behavior. White paper, available at http://jndglobal.com/wp-content/uploads/2011/05/gamification1011.pdf, last access: January, 2019,9. Chang, J.-W., & Wei, H.-Y. (2016). Exploring Engaging Gamification Mechanics in Massive Online Open courses. Journal of Educational Technology & Society,19 , 177–203.535 Charness, G., Gneezy, U., & Kuhn, M. A. (2012). Experimental methods: Between-subject and withinsubject design. Journal of Economic Behavior & Organization,81, 1–8. Codish, D., & Ravid, G. (2014). Academic course gamification: The art of perceived playfulness. Interdisciplinary Journal of E-Learning and Learning Objects,10 , 131–152. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd edition).540 Cross, S., Whitelock, D., & Galley, R. (2014). The use, role and reception of open badges as a method for formative and summative reward in two Massive Open Online Courses. International Journal of e-Assessment,4, 1–16. Csikszentmihalyi, M. (1991). Flow: The psychology of optimal experience. New York: HarperPerennial. Davis, D., Chen, G., Hauff, C., & Houben, G.-J. (2018). Activating learning at scale: A review of innovations545 in online learning strategies. Computers & Education,125, 327–344. De Sousa Borges, S., Durelli, V., Reis, H., & Isotani, S. (2014). A Systematic Mapping on Gamification Applied to Education. In Proceedings of the 29th Annual ACM Symposium on Applied Computing (pp. 216–222). ACM. Deci, E. L., Koestner, R., & Ryan, R. M. (2001). Extrinsic rewards and intrinsic motivation in education:550 Reconsidered once again. Review of educational research,71, 1–27. Deng, R., Benckendorff, P., & Gannaway, D. (2019). Progress and new directions for teaching and learning in MOOCs. Computers & Education,129, 48–60. Deterding, S., Dixon, D., Khaled, R., & Nacke, L. (2011). From Game Design Elements to Gamefulness: Defining Gamification. In Proceedings of the 15th ACM International Academic MindTrek Conference:555 Envisioning Future Media Environments (pp. 9–15). Dicheva, D., Dichev, C., Agre, G., & Angelova, G. (2015). Gamification in Education: A Systematic Mapping Study. Journal of Educational Technology & Society,18, 75–88. Dicheva, D., Irwin, K., & Dichev, C. (2018). Oneup: Supporting practical and experimental gamification of learning. International Journal of Serious Games,5, 5–21.560 Ding, L., Kim, C., & Orey, M. (2017). Studies of student engagement in gamified online discussions. Computers & Education,115, 126–142. Dom´ınguez, A., Saenz-de Navarrete, J., De-Marcos, L., Fern´andez-Sanz, L., Pag´es, C., & Mart´ınez-Herr´aiz, J.-J. (2013). Gamifying learning experiences: Practical implications and outcomes. Computers & Education,63, 380–392.565 Ferguson, R., & Clow, D. (2015). Examining engagement: analysing learner subpopulations in massive open online courses (MOOCs). In Proceedings of the 5th International Conference on Learning Analytics And Knowledge (pp. 51–58). ACM. Ferguson, R., & Sharples, M. (2014). Innovative pedagogy at massive scale: teaching and learning in 20 MOOCs. In Open Learning and Teaching in Educational Communities (pp. 98–111). Springer.570 Festinger, L. (1954). A theory of social comparison processes. Human relations,7, 117–140. Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School Engagement: Potential of the Concept, State of the Evidence. Review of educational research,74, 59–109. Gurantz, O. (2015). Who loses out? Registration order, course availability, and student behaviors in community college. The Journal of Higher Education,86, 524–563.575 Hakulinen, L., Auvinen, T., & Korhonen, A. (2013). Empirical Study on the Effect of Achievement Badges in TRAKLA2 Online Learning Environment. In Proceedings of the 2013 International Conference on Learning and Teaching in Computing and Engineering (pp. 47–54). IEEE. Hamari, J. (2017). Do badges increase user activity? A field experiment on the effects of gamification. Computers in human behavior,71, 469–478.580 Hamari, J., & Eranti, V. (2011). Framework for designing and evaluating game achievements. In Proceedings of the 2011 DiGRA International Conference. Hamari, J., Koivisto, J., & Sarsa, H. (2014). Does Gamification Work? - A Literature Review of Empirical Studies on Gamification. In Proceedings of the 47th Hawaii International Conference on System Sciences (pp. 3025–3034). IEEE.585 Henderikx, M. A., Kreijns, K., & Kalz, M. (2017). Refining success and dropout in massive open online courses based on the intention–behavior gap. Distance Education,38, 353–368. Henrie, C. R., Halverson, L. R., & Graham, C. R. (2015). Measuring student engagement in technologymediated learning: A review. Computers & Education,90, 36–53. Hew, K. F. (2016). Promoting engagement in online courses: What strategies can we learn from three highly590 rated MOOCS. British Journal of Educational Technology,47, 320–341. Hone, K. S., & El Said, G. R. (2016). Exploring the factors affecting MOOC retention: A survey study. Computers & Education,98, 157–168. Ib´a˜nez, M. B., Di-Serio, A., & Delgado Kloos, C. (2014). Gamification for Engaging Computer Science Students in Learning Activities: A Case Study. IEEE Transactions on Learning Technologies,7, 291–595 301. Jordan, K. (2014). Initial trends in enrolment and completion of massive open online courses. The International Review of Research in Open and Distributed Learning,15, 133–160. Khalil, H., & Ebner, M. (2014). MOOCs Completion Rates and Possible Methods to Improve Retention - A Literature Review. In Proceedings of the 2014 World Conference on Educational Multimedia, Hypermedia600 and Telecommunications (pp. 1305–1313). Khalil, M., Ebner, M., & Admiraal, W. (2017). How can Gamification Improve MOOC Student Engagement? In Proceedings of the 11th European Conference on Games Based Learning (pp. 819–828). Khalil, M., Wong, J., de Koning, B., Ebner, M., & Paas, F. (2018). Gamification in MOOCs: A Review of the State of the Art. In Proceedings of the 2018 Global Engineering Education Conference (pp. 1629–1638).605 IEEE. (2008). Test of homogeneity, chi-squaretest of homogeneity, chi-square. In W. Kirch (Ed.), Encyclopedia of Public Health (pp. 1386–1386). Springer Netherlands. Kizilcec, R. F., Piech, C., & Schneider, E. (2013). Deconstructing Disengagement: Analyzing Learner Subpopulations in Massive Open Online Courses. In Proceedings of the 3rd International Conference on610 Learning Analytics and Knowledge (pp. 170–179). ACM. Krause, M., Mogalle, M., Pohl, H., & Williams, J. J. (2015). A Playful Game Changer: Fostering Student Retention in Online Education with Social Gamification. In Proceedings of the 2nd ACM Conference on Learning @ Scale (pp. 95–102). ACM. Kyewski, E., & Kr¨amer, N. C. (2018). To gamify or not to gamify? an experimental field study of the615 influence of badges on motivation, activity, and performance in an online learning course. Computers & Education,118, 25–37. Margaryan, A., Bianco, M., & Littlejohn, A. (2015). Instructional quality of massive open online courses (MOOCs). Computers & Education,80, 77–83. Navidi, W. C. (2008). Statistics for engineers and scientists. McGraw-Hill Higher Education New York,620 21 NY, USA. Nicholson, S. (2015). A recipe for meaningful gamification. In Gamification in education and business (pp. 1–20). Springer. O’Donovan, S., Gain, J., & Marais, P. (2013). A case study in the gamification of a university-level games development course. In Proceedings of the South African Institute for Computer Scientists and Information625 Technologists Conference (pp. 242–251). ACM. Ortega-Arranz, A., Er, E., Mart´ınez-Mon´es, A., Bote-Lorenzo, M. L., Asensio-P´erez, J. I., & Mu˜nozCrist´obal, J. A. (2019). Understanding Students Behavior and Perceptions towards Earning Badges in a Gamified MOOC. Universal Access in the Information Society, (p. accepted). Ortega-Arranz, A., Kalz, M., & Mart´ınez-Mon´es, A. (2018). Creating Engaging Experiences in MOOC630 through In-Course Redeemable Rewards. In Proceedings of the 2018 Global Engineering Education Conference (pp. 1875–1882). IEEE. Ortega-Arranz, A., Mu˜noz-Crist´obal, J. A., Mart´ınez-Mon´es, A., Bote-Lorenzo, M. L., & Asensio-P´erez, J. I. (2017). How Gamification is Being Implemented in MOOCs? A Systematic Literature Review. In Proceedings of the 12th European Conference on Technology Enhanced Learning (pp. 441–447). Springer.635 Penuel, W. R., Roschelle, J., & Shechtman, N. (2007). Designing formative assessment software with teachers: An analysis of the co-design process. Research and Practice in Technology Enhanced Learning, 2, 51–74. Rizzardini, R., Chan, M., & Guetl, C. (2016). An Attrition Model for MOOCs: Evaluating the Learning Strategies of Gamification. Formative Assessment, Learning Data Analytics and Gamification: In ICT640 Education. Elsevier. Ruip´erez-Valiente, J. A., Mu˜noz-Merino, P. J., & Delgado Kloos, C. (2017). Detecting and Clustering Students by their Gamification Behavior with Badges: A Case Study in Engineering Education. International Journal of Engineering Education,33, 816–830. Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation,645 social development, and well-being. American psychologist,55, 68. Seaborn, K., & Fels, D. I. (2015). Gamification in theory and action: A survey. International Journal of Human-Computer Studies,74, 14–31. Shah, D. (2017). By The Numbers: MOOCS in 2017. How has the MOOC space grown this year? Get the facts, figures, and pie charts. Retrieved from: https://www.class-central.com/report/mooc-stats-2017/650 last access: June 2018. Siemens, G. (2013). Massive open online courses: Innovation in education. Open educational resources: Innovation, research and practice,5, 5–15. Zhu, Y., Pei, L., & Shang, J. (2017). Improving video engagement by gamification: A proposed design of MOOC videos. In International Conference on Blended Learning (pp. 433–444). Springer.655 Zichermann, G., & Cunningham, C. (2011). Gamification by design: Implementing game mechanics in web and mobile apps. O’Reilly Media, Inc. 22