Aligning learning design and learning analytics through instructor involvement: a MOOC case study

This is an Accepted Manuscript of an article published by Taylor & Francis Group in Interactive Learning Environments on 02/05/2019, available online: https://www.tandfonline.com/doi/full/10.1080/10494820.2019.1610455 1 Aligning Learning Design and Learning Analytics through Instructor Involvement: A MOOC Case Study Erkan Er Universidad de Valladolid [email protected] Eduardo Gómez-Sánchez Universidad de Valladolid edug[email protected] Yannis Dimitriadis Universidad de Valladolid yann[email protected] Miguel L. Bote-Lorenzo Universidad de Valladolid migbo[email protected] Juan I. Asensio-Pérez Universidad de Valladolid [email protected] Susana Álvarez Álvarez Universidad de Valladolid susana.alvar[email protected] Abstract. With the emergence of MOOCs, there is a growing interest in prediction research. Most existing predictive models do not consider the context for which they are intended, thus resulting in limited impact. Learning design (LD) can provide a contextual understanding for the design of predictive models in collaboration with the instructors, maximizing their potential for supporting learning. This paper presents the findings of a mixed-methods research that explored the potentials emerging from aligning LD and LA during the design of a predictive analytics solution and from involving the instructors in the design process. The context was a past MOOC, where the learner data and the instructors were accessible for posterior analysis and additional data collection. Through a close collaboration with the instructors, the details of the prediction task were identified, including the learning activity to focus on, the target variable to predict, and the practical constraints to consider. Later, two predictive models were built for the prediction task identified: LD-specific model, in which the features were based on the LD and pedagogical intentions, and a generic model, which was based on cumulative features, not informed by the LD. Although the LD-specific predictive model did not outperform the generic one, some features derived from the LD and pedagogical intentions were predictive. The quantity and the power of such features were associated with the degree to which the students acted as guided by the LD and pedagogical intentions. The leading instructor’s opinion about the importance of the learning activities in the LD was compared with the results of the feature importance analysis. This comparison helped identify the parts of the LD that need improvement. That is, the results of the LA informed back the LD, where the instructor was a mediator. The implications for improving the LD are discussed. Keywords: MOOCs, learning design, learning analytics, predictive analytics, collaborative learning 1. Introduction In the recent years, the growth of massive open online courses (MOOCs) has attracted much attention on predictive analytics, a popular area of learning analytics (LA). The unprecedented amount of learner data generated in massive online courses is providing great affordances for building accurate predictive models of student engagement and success (Moreno-Marcos, Alario-Hoyos, Muñoz-Merino, & Kloos, 2018). Although numerous predictive models and techniques have been developed, they have had a limited capacity in improving the teaching and learning processes in real-world contexts (Gardner & Brooks, 2018). One main reason for the limited impact is that many of these works do not take into account the pedagogical context (Gašević, Dawson, Rogers, & Gašević, 2016). That is, they are decoupled from the contexts for which they are intended, and therefore they offer insufficient pedagogical affordances (Gašević et al., 2016). Learning design (LD), which is the blueprint of “the sequence of learning tasks, the resources, and the sequence of teaching methods” (Lockyer, Heathcote, & Dawson, 2013, This is an Accepted Manuscript of an article published by Taylor & Francis Group in Interactive Learning Environments on 02/05/2019, available online: https://www.tandfonline.com/doi/full/10.1080/10494820.2019.1610455 2 p.1441), can provide a contextual understanding that can inform the design of predictive models and enhance their potential use for improving pedagogy. First, the features (or variables) generated to train predictive models can be decided considering the impact of the LD on student activities. For example, instead of the total number of video views, a commonly used generic feature in the literature, LD could be used to interpret the varying importance of each video on student learning and to decide on a relevant feature. Accordingly, several research works have noted the relevance of LD in building predictive models (Gašević et al., 2016; Veeramachaneni, O’Reilly, & Taylor, 2014). Yet, the role of LD during the design of predictive models remains largely unknown. Moreover, instructors may play a crucial role in aligning the predictive analytics solutions with LD and their pedagogical intentions. In order for the predictive models to be used in the targeted context, they have to serve a predetermined pedagogical need decided by the course instructors (or instructional designers). Without their involvement, predictive models are highly likely to serve only for experimentation, as in most dropout prediction models in the literature (e.g., Halawa, Greene, & Mitchell, 2014; Kurka, Godoy, & Von Zuben, 2016). Previous research noted that the close collaboration between the researchers and the designers can lead to LA interventions that connects well with particular needs of a learning context (RodríguezTriana, Martínez-Monés, Asensio-Pérez, & Dimitriadis, 2015; Rodríguez-Triana, Prieto, Martínez-Monés, & Asensio-Pérez, 2017). Thus, LD and LA can offer new opportunities for improving teaching and learning when one informs the other (Eradze, Rodríguez-Triana, & Laanpere, 2017; Mor, Ferguson, & Wasson, 2015; Schmitz, Limbeek, Greller, Sloep, & Drachsler, 2017). However, this synergic relationship has been understudied in the area of predictive analytics. Attending to this gap, this paper reports on a research investigating the opportunities emerging from aligning LD and LA in a particular MOOC context in close collaboration with the whole instructional designer team (of this particular course). The instructional designers played an active role when: (1) determining a relevant prediction task, (2) identifying the constraints imposed by the context, (3) assessing the importance of learning activities included in the LD, and (4) and refining the LD of the course by analysing the results. LD and LA were connected in two ways: (1) the LD and the pedagogical intentions were used to guide feature generation, and reciprocally (2) the results of the predictive analysis were used to review and improve the LD for the future run of the same course. This research is guided by the following research questions: • RQ1: What are the potentials emerging from aligning LA and LD during the design of a predictive analytics solution? • RQ2: What are the potentials emerging from involving the instructional designers in the design process of the predictive analytics solution? The paper is structured as follows. Section 2 discusses the related literature and identifies the need for research, and Section 3 describes the context and the research design. Then, Section 4 presents the results in three sections: the prediction task and the practical constraints (as determined by the instructor), the predictive models and their accuracies, and the comparison of instructor opinion with the results of the predictions. Section 5 discusses the findings organized around the research questions. The paper concludes with Section 6, which presents the limitation of this research work and suggest future research. 2. Background: Learning Design, Learning Analytics, and MOOCs A critical issue often echoed in the LA literature is the lack of real-time interventions that align well with instructors’ pedagogical needs (Harrer & Göhnert, 2015; Nguyen, Rienties, & Toetenel, 2017; Wise & Vytasek, 2017). As a result, the LA researched has yielded a limited impact so far. In their recent literature review of LA dashboards, Jivet, Scheffel, Drachsler, and This is an Accepted Manuscript of an article published by Taylor & Francis Group in Interactive Learning Environments on 02/05/2019, available online: https://www.tandfonline.com/doi/full/10.1080/10494820.2019.1610455 3 Specht (2017) found that LA solutions are built mainly based on the motivation to take advantage of the trace data available instead of a clear pedagogical intention. Similarly, in their systematic literature review on teaching and LA, Sergis & Sampson (2017) noted only few efforts to convert LA into actionable recommendations for instructors to refine the pedagogy based on the contextual needs. Thus, even though considerable progress has been made in conceptual development and design of LA (Bakharia et al., 2016; Lockyer et al., 2013; Manolis & Karkalasi, 2017), the products generated (e.g., data, visualizations, and tools) often ignore the influence of the pedagogical context (Eradze et al., 2017; Nguyen et al., 2017) and do not align well with instructors’ needs for refining their pedagogy and learning activities (Mor et al., 2015). For a greater impact on practice, LA solutions need to be implemented in a way that is sensitive to the needs of learning context for which they are intended (Harrer & Göhnert, 2015; Wise & Vytasek, 2017). The trace data used in LA are collected through students’ interactions with the learning systems (e.g., LMS, MOOC platforms), and these interactions are largely shaped by the LD (Gašević, Mirriahi, Dawson, & Joksimović, 2017). Therefore, understanding the LD is important for the meaningful interpretation of learning outcomes (e.g., visualizations of students’ daily course visits) generated based on the trace date (Mangaroska & Giannakos, 2017). For instance, LD would determine the roles of the instructor and students in an online discussion activity and shape their interactions (Lockyer et al., 2013). Instructors may play a central role or may intend to promote peer interactions in discussions depending on the pedagogical intentions. LD of MOOCs may play a similar role in shaping learner’ interactions with the MOOC platforms and could help identify relevant features when building predictive models in MOOC settings. However, given that MOOCs are non-formal learning settings with massive numbers and variety of participants (Kizilcec, Piech, & Schneider, 2013), previous findings from formal learning contexts should not be taken for granted. Yet, there is scarce research on the use of LD to inform predictive analytics in MOOCs and to create interventions driven by the context. Most MOOC research has exploited a posteriori student data to understand the learning behaviour of MOOC participants (Kizilcec et al., 2013; Tseng, Tsao, Yu, Chan, & Lai, 2016) and to identify the variables that can explain learners’ engagement and success (Bote-Lorenzo & Gómez-Sánchez, 2017; Er, Bote-Lorenzo, Gómez-Sánchez, Dimitriadis, & Asensio-Pérez, 2017). These research studies have substantial value in establishing the theoretical foundations of how individuals behave, engage, and learn at massive and open learning settings (Brooks & Thompson, 2017; Liu & Koedinger, 2017). However, their impact in the practice of MOOC teaching and learning is still questionable. Accordingly, in their review of the literature on MOOC prediction research, Gardner & Brooks (2017) underlined the need for practical predictive analytics research in real-world MOOC contexts in order to close the gap between LA research and MOOC practice. Thus, there is gap in the MOOC literature regarding the potentials emerging from aligning predictive analytics with LD. MOOC predictive analytics have been based on experimental studies with minor direct effects in MOOC practice (Gardner & Brooks, 2018). On the other hand, in collaboration with instructors, LD can inform the design of LA solutions and lead to predictive models for designing interventions that target the contextual needs (Mangaroska & Giannakos, 2017). 3. The Current Study 3.1. Context The context of this research was a MOOC that teaches the translation of terms from English/Spanish to Spanish/English in the domain of finance, offered by a Spanish university in the Canvas Network MOOC platform. The enrolment was closed at the end of the second week. This is an Accepted Manuscript of an article published by Taylor & Francis Group in Interactive Learning Environments on 02/05/2019, available online: https://www.tandfonline.com/doi/full/10.1080/10494820.2019.1610455 4 Total number of enrolled students was 1031. Regarding the gender distribution of the students, the majority was woman (75%). Given that the course was taught in Spanish, a great portion of the students were from Spain (57%) and Latin America (31%). Most participants were between 20-30 years old (61.2%) and holding at least Master (26 %), Bachelor (53%) or High School (16%) degrees. The lead course instructor is experienced in online teaching, mostly in formal learning settings with relatively fewer students. This MOOC was her first teaching experience in massive and open learning context. Moreover, the instructor’s field of expertise is in the domain of Translation and Interpretation. The other two instructional designers were PhD students in the domain of the educational technologies who worked with the leading instructor to design the course. The course was composed of 7 blocks (or modules or weeks). Each block was available one week after another, and once the block was published, its content remained accessible until the end of the course. Two collaborative activities were designed in this MOOC, involving the extraction of relevant terms from given documents dealing with finance. Groups of six participants were formed in both collaborative activities. The instructional designers of the course consisted of a leading instructor and two teaching assistants, all of which were involved in this research Further details about the design of the course are provided in Appendix C and D. 3.2. Method This is a mixed-methods study (Greene, 2007) that employs both qualitative and quantitative data to address the research questions. The design of the study includes three stages as delineated in Figure 1. In the first stage (see section 4.1), qualitative data were collected through a face-to-face meeting with the teaching assistants and an online survey filled by the lead instructor (see Appendix A), with the goal of identifying the learning design, pedagogical intentions and the prediction task. The LD of the course (see Appendix C) was derived from the course pages in the Canvas Platform and finalized with the help of the teaching assistants during the face-to-face meeting. The meeting notes were analysed using thematic analysis (Marshall, C. & Rossman, 1999) to identify the pedagogical intentions. The course components in the LD were used as the themes to guide the analysis. This analysis resulted in the pedagogical intentions behind each learning element (see Appendix D) and the learning activity to be targeted with predictive analytics. Moreover, in the first stage, an online survey was administered to the lead instructor to identify the target variable for the prediction along with the practical constraints. In the survey, the instructor was asked to rank the possible engagement variables to predict about the target activity according to their pedagogical usefulness. The instructor was also asked to indicate when the intended predictions would be needed (i.e., how early before the target activity starts), which was used as a practical constraint to consider in the design of the prediction models. The second stage (see section 4.2) involved building and running two models within the practical contexts for the prediction task decided in the first stage: 1) a generic model, built with cumulative features (e.g., total number of discussion posts), and 2) a LD-specific model, built with features informed by the LD and pedagogical intentions. Then, the prediction accuracies of these models were assessed and compared. The predictive features were generated from the learner activity logs (as the quantitative data source) retrieved using the API of the Canvas Network MOOC platform. Consent from the MOOC participants for the use of their activity logs was obtained through the welcome survey administered at the initial days of the course. All learner data were anonymized before the analysis. After the anonymization, the data files that contain identifying user information were deleted. Any corrupted logs were discarded. As further discussed in section 4.2., qualitative data (i.e., LD and pedagogical intentions) were used to inform the analysis of the quantitative data (i.e., feature generation) when building the LD-specific model. That is, the research design involved mixing for sequential exploration This is an Accepted Manuscript of an article published by Taylor & Francis Group in Interactive Learning Environments on 02/05/2019, available online: https://www.tandfonline.com/doi/full/10.1080/10494820.2019.1610455 5 (Leech & Onwuegbuzie, 2009). Note that the personnel who worked with the research data had undergone a training program on Human Subject Research. Figure 1. Design of the study In the last stage (see section 4.3), LD-specific features were analysed based on their predictive powers determined by the machine learning algorithm, which helped identify the important learning activities in the LD. The number of corresponding features and their predictive powers were used to identify the importance of the learning activities. Then, through an online interview (see Appendix B), instructor ranked the same learning activities based on their importance for student learning, which was compared with the algorithmic results by the researcher simultaneously during the interview and the results were shared with the instructor. Instructor reflected on the discrepancies emerging from the comparison and provided possible reasons for the differences. The interview was audio recorded and transcribed for the analysis. The interview transcript was analysed using a general inductive approach (Thomas, 2006). The general inductive approach is a simple yet effective approach to derive findings from qualitative data shaped with focused questions that require concrete answers. Similarly, the interview was guided by the discussion on specific discrepancies between the instructor and algorithmic ranking. We first formatted the transcription in a common format and then performed a close read to get familiar with the content. During the analysis, we first created the general categories (e.g., problems in conveying the pedagogical intentions to the learners) and then refine them to identify specific themes (e.g., poor assignment descriptions, low accessibility of the syllabus), which helped us capture the core message reported by the instructor. Thus, through the inductive approach, we were able to effectively convert the instructor input into a structured summary format and to establish links with particular course elements and activities (that resulted in discrepancies) for deriving useful implications (see section 5.3). Logistic regression was used as the classifier algorithm to train the predictive models (i.e., generic and LD-specific) as it has been effective in many classification tasks in the MOOC literature (Kizilcec, Pérez-Sanagustín, & Maldonado, 2016; Whitehill, Mohan, Seaton, Rosen, & Tingley, 2017b). Features were calculated based on the student activity after February 14 since the logs before that day were missing in the database of the Canvas platform. Students with no visits to any course pages were removed from the data. All features were standardized before they were used for training and testing the models. Learners with zero pageviews (n=458) were eliminated to avoid noise in the analysis. Given the unbalanced distribution of the class labels, the performance of the models were evaluated using area under the curve (AUC) as performance metric (Hanley & McNeil, 1982). 2. Build and compare the predictive models 1. Identify the LD, pedagogical intentions, and the prediction task Meeting notes Survey responses Student logs Interview transcripts LD, pedagogical intentions & constraints The engagement variable to predict LD-specific model Generic model RESEARCH DATA 3. Compare the importance of LD components Importance of LD by the algorithm Importance of LD by the instructor STAGES This is an Accepted Manuscript of an article published by Taylor & Francis Group in Interactive Learning Environments on 02/05/2019, available online: https://www.tandfonline.com/doi/full/10.1080/10494820.2019.1610455 6 The AUC score of a classifier refers to the likelihood of ranking a randomly chosen positive example higher than a randomly chosen negative example. The categorization of model performance based on AUC scores is: .9-1: excellent, .8-.9: very good, .7-.8: good, .6-.7: fair, and .5- .6: bad (Bekkar, Djemaa, & Alitouche, 2013). The research on human behavior prediction considers AUC values of .7 and higher reasonably accurate (Rice & Harris, 2005). Previous MOOC research have regarded the models with such accuracy (i.e., AUC > .7) as robust models (Boyer & Veeramachaneni, 2016). Even though the model performances will be evaluated based on the AUC scores, precision, recall, and f1-scores were also presented to provide a more comprehensive view of the model performances. Feature selection was performed using randomized logistic regression (RLR). The use of RLR for feature selection is also known as stability selection (SS) (Meinshausen & Bühlmann, 2010). The purpose of using SS in this study was to be able to interpret the feature importance rather than to select features. RLR assigns a score to each feature indicating their predictive power. SS has been used in MOOC prediction research (Taylor, Veeramachaneni, & O’Reilly, 2014; Veeramachaneni et al., 2014). The Scikit-Learn (Pedregosa et al., 2012) implementations of logistic regression and RLR were used. 4. Results The results are organized by the stages of the research (see Figure 1). 4.1. Stage 1: The prediction task and its practical requirements The first stage involved identifying prediction task and its practical requirements as determined based on the qualitative data collected from the teaching staff. According to the meeting notes with the teaching assistants, the collaborative learning activities were identified as critical for student learning for several reasons. The teaching assistants noted that the collaborative learning activities enabled students (1) “to acquire prerequisite skills necessary for translating terms in the business domain”, and (2) “to directly interact with peers”, thus allowing them “to learn from peers’ knowledge and experience” as well as “to feel more connected with other learners”. Therefore, collaborative learning activities were decided for a possible intervention through predictive analytics. Among the two activities (one in the fourth week and another in the sixth week), for the scope of this study, we focused on the first one. To determine the information to be produced by predictive analytics for intervening the activity, through an online survey the instructor ranked four specific pieces of information based on their pedagogical usefulness. According to the rankings of the instructor (see Table 1), knowing whether the number of students who participate in the group discussion will be higher than a threshold (i.e., 3) was considered the most useful. As the rationale behind this decision, the instructor noted that “by utilizing this information regarding students’ participation levels, a certain level of interaction in groups can be assured and the workload per group members would be manageable.” Table 1 The instructor’s ranking on the usefulness of information needed for the collaborative activity Ranking Information 1 Whether the number of students who participate in the group discussion will be higher than a threshold. (e.g., at least 3 students make a post in a group) 2 Whether the number of messages posted in the group discussion will be higher than a threshold (e.g., having at least 5 discussion entries) 3 Whether the collaborative group will submit their assignment or not. 4 Whether a student will post in the group discussion or not. This is an Accepted Manuscript of an article published by Taylor & Francis Group in Interactive Learning Environments on 02/05/2019, available online: https://www.tandfonline.com/doi/full/10.1080/10494820.2019.1610455 7 The prediction task was aimed at identifying if a student would participate in group discussions (i.e., post at least one entry) or not, as these predictions would be needed to obtain the number of active participants in groups as demanded by the instructor. In this prediction task, students with at least one entry are identified as contributors, and students with zero entries are identified as non-contributors. Please note that this decision on the target variable to predict was determined by what was considered pedagogically more useful from the instructor’s perspective. A different classification (e.g., very active contributors, active contributors, low active contributors and not contributors) could be preferred in a different course depending on pedagogical intentions in the context. For the predictions to be usable for intervening in real time, they would need to be available when they are still considered actionable by the instructor. For this purpose, through the online survey, the leading instructor’s preference was obtained. The instructor indicated that she would need the predictions by the 2nd of March, which was then used as the cut-off point for calculating the features. The underlying reason for the instructor’s decision was (1) to allow students to make some progress on the collaborative activity as they had probably different availability and schedule and (2) to have enough time (around 10 days) before the activity deadline to be able to intervene using the predictions. Please note that the decision of having the predictions 10 days before the target activity may not apply to different contexts. We recommend allowing instructors to decide on the best date according to contextual factors and pedagogical intentions. 4.2. Stage 2: Building and comparing the classification models (LD-driven vs generic model) The feature engineering for the LD-driven model and the generic model are provided below. The performances of the models are also compared. Feature engineering for LD-driven model (mixing for sequential exploration) The LD of the MOOC (see Appendix C) and pedagogical intentions (see Appendix D) were used to inform the feature generation (i.e., variables about student activities) that can possibly lead to an accurate model predicting participation in group discussions. Please note that all the features described below were derived by the research team, directly informed by the LD and the pedagogical intentions. Regarding the potential influence of the pedagogical intentions on students’ activities several features were derived. In particular, given the intent of the review videos to promote the selfevaluation and recap as necessary, the subsequent student activity could be an indicator of student engagement. For example, after watching the review video a student might identify a gap in his own understanding and may want to revisit the related lecture content. Accordingly, the research team identified 4 features based on the sequence of activities in which visiting a reviewvideo page is anterior, resulting in 13 sequence features in total. Moreover, learning resources and activities in a block could be important for student learning in subsequent blocks. For example, the glossary document created in Block 1 was intended to be a useful resource for students during the whole course. Accordingly, three sequence features were derived based on students’ visits to previous blocks. These features indicate the number of visits from Block 3 to preceding blocks (i.e., Block 2, Block 1, and Block 0), and from Block 2 to preceding blocks (i.e., Block 1 and Block 0). Thus, informed by the instructors’ pedagogical intentions, in total 15 sequence features were generated by the research team. The sequence of learner actions was previously noted in the MOOC literature as important to student learning (Boroujeni & Dillenbourg, 2018) Moreover, features were generated based on each particular content item in the LD. The first set of variables includes page-view features that convey student engagement in each specific This is an Accepted Manuscript of an article published by Taylor & Francis Group in Interactive Learning Environments on 02/05/2019, available online: https://www.tandfonline.com/doi/full/10.1080/10494820.2019.1610455 8 learning component. Links to PDF files (if exists within a content page) were considered separately, since they may signal higher student interest and engagement in the corresponding activity or the content. In total, 48 page-view features were generated. Last, 17 features regarding students’ active participation in the course were generated, including one or all of the following activities depending on the block: taking a quiz, posting in a discussion forum, and submitting an assignment. Please note that previous MOOC research has showed the predictive capacity of such learning activities but using cumulative features (Bote-Lorenzo & Gómez-Sánchez, 2018; Veeramachaneni et al., 2014; Whitehill et al., 2017b), as in this work to build the generic model. Table 2 Number of LD-specific features per each block Block 0 Block 1 Block 2 Block 3 Features regarding students’ sequence activities Number of sequences of review-video and lecture content visits 1 1 1 1 Number of sequences of review-video and discussion forum visits 1 1 1 1 Number of sequences of review-video and quiz visits 1 1 0 0 Number of sequences of review-video and assignment visits 0 1 1 1 Features regarding students’ page-view activities Number of page-view features 6 9 12 12 Features regarding students’ active participation Number of discussion posts 2 1 1 1 Number of quiz attempts 0 1 1 0 Time spent in quizzes 0 1 1 0 Quiz scores 0 1 1 0 Assignment-submission status 0 0 2 2 In total, 80 features were generated. The names of these features and their distribution across different blocks are provided in Table 2. These features were calculated based on the student activity logs beginning from the course start until the cut-off point (March 2), resulting in 419 students. Histogram plots along with some descriptive statistics of all features are provided in the Appendix E. Feature names are composed of two parts: the first part is the name of the associated item in the LD, and the second part is the suffix to describe the feature. This notation is used to better communicate the association of the features generated with specific components of the LD. The explanations for the suffixes are given in Table 3. Table 3 The explanations for the suffixes used in feature names Suffix Meaning *_PAGEVIEW Page-view count *_DISCENTRY Entry count in a discussion *_ASSIGNSUBM If assignment is submitted or not *_QUIZATTEMPT Number of quiz attempts *_QUIZSCORE Quiz score *_QUIZTIMESPENT Minutes spent on a quiz *_RVID_DISC Number of times visiting a discussion after a review-video page *_RVID_LECTURE Number of times visiting a lecture content page after a reviewvideo page *_RVID_QUIZ Number of times visiting a quiz page after a review-video page *_RVID_ASSIGNMENT Number of times visiting an assignment page a review-video page This is an Accepted Manuscript of an article published by Taylor & Francis Group in Interactive Learning Environments on 02/05/2019, available online: https://www.tandfonline.com/doi/full/10.1080/10494820.2019.1610455 9 *_ACROSSBLOCKS Number of visits to previous blocks Feature engineering for the generic model In the feature engineering for the generic model, no specific information about the LD and the pedagogical intentions were used. These features are widely used in the predictive analytics research. Features and their descriptions are provided in Table 4. Table 4 Descriptions of generic the features Feature name Description TTL_INFOPAGE_VIEW Number of times students visited introduction pages TTL_LECCONT_ VIEW Number of times students visited lecture-content pages TTL_DISC_ VIEW Number of times students visited discussion forums TTL_QUIZ_VIEW Number of times students visited quiz pages TTL_RVID_VIEW Number of times students visited review-video pages TTL_ASSIGN_VIEW Number of times students visited assignment-related pages TTL_PAGE_VIEW Number of times students visited any course page TTL_QUIZ_ATTEMPT Number of quiz attempts TTL_QUIZ_TIMESPENT Total time spent on quizzes TTL_QUIZ_SCORE Total quiz score TTL_ASSIGN_SBM Number of assignment submissions TTL_DISCENTRY Number of discussion posts Comparing the classification models To test the performance of the classification models, 10-fold CV was used. The accuracy of the predictions for both models are provided in Table 5. The results show that the classification models perform quite accurate (AUC scores are 0.858 and 0.822 for LD-driven and generic models, respectively). Thus, the generic model built with cumulative features has very similar predictive capacity to that of LD-drive model that is more particular to the context. Table 5 The performance scores of prediction accuracies for LD-Driven and generic models AUC Precision Recall F1-score LD-Driven Model 0.858 0.880 0. 770 0.821 Generic Model 0.822 0.862 0.737 0.794 4.3. Stage 3: Instructor opinion on the importance of learning activities and comparison with feature importance analysis (mixing for concurrent triangulation) The importance of learning activities as ranked by the instructor was compared with those determined based on the feature importance as automatically determined by the SS approach. The results are provided in Table 6. In this table, the first column indicates the name of the LD component, which are shown in Appendix C. The number of predictive features in each category are provided in the second column. These numbers are accompanied with average of the percentages of the selected features. As discussed in section 4.2, this percentage represents the predictive power. For example, 100% would indicate a very high predictive capacity as the feature would have been selected within all inspected feature subsets. In the third column, the rankings of the instructor are provided. According to the results, in general, the instructor rankings on the importance of various LD components did not match with the rankings determined by the SS approach. To begin with,