Online Self-Regulated Learning Questionnaire in a Russian MOOC

MANUS CRIP T ACCEP TED ACCEPTED MANUSCRIPT Online Self-Regulated Learning Questionnaire in a Russian MOOC Ruth Martinez-Lopezab , Carmen Yotb , Iuliia Tuovila c , Víctor-Hugo Perera-Rodríguez b a Samara National Research University, Russia b University of Seville, Faculty of Sciences of Education, Seville, Spain c Samara National Research University, department of foreign languages and Russian as a foreign language, Samara, Russia MANUS CRIP T ACCEP TED ACCEPTED MANUSCRIPT Online Self-Regulated Learning Questionnaire in a Russian MOOC ABSTRACT The learning opportunities that Massive Open Online Courses (MOOCs) promote urge more research in self-regulated learning to enhance student engagement. Since self-regulation can be determined by the Online Self-Regulating Learning Questionnaire. This research assessed its adaptation, using the modified kappa coefficient and the Content Validity Indexing (CVI) in the Russian context. Clarity, cultural equivalence and relevance of the instrument were classified by seven experts. Forty-five students confirmed the ease of understanding in a pre-test. At scale-level index, the “Goal Setting” value was below the adequate level. Three items were considered unacceptable, and one of them was identified as a potentially problematic item. Confirmatory factor analysis showed some problems in the adjustment of the questionnaire model to the data. Self-regulated learning skills of Russian engineering students in MOOCs are moderate, with high levels in terms of “Environment Structuring” and “Goal Setting”, but low in “Help Seeking”. Selfregulated learning variables indicated no gender differences. Keywords: Content validity index, cross-cultural, MOOCs, Online Self-Regulated Learning, psychometric Introduction Because of the wide use of the online environment in learning, the research of predictors like self-regulated learning strategies to success is increasing (Broadbent & Poon, 2015). According to Jou, Lin, & Wu (2014), students’ preferences in technology and skills could influence their learning achievement. That is why the engagement of students is considered relevant for a self regulated learning curriculum (McLoughlin & Lee, 2010). Online environments demand selfregulation, and at the same time they are optimal for supporting self-regulated learning (Cigdem, 2015; Kitsantas, 2013; Zheng, 2016). In addition, empirical research has proved how the context of the student influences self-regulation in MOOCs (Hood et al., 2015). Thus, further studies in self-regulation in online learning environments are required (Basol & Balgalmis, 2016). Today MOOCs are very popular, and a key trend in Russia (Sapargaliyev, 2014). Correspondingly, the necessity for more studies examining learner experiences in MOOCs is justified (Hood et al., 2015; Veletsianos, Collier, & Schneider, 2015). In the review of self-regulation in the Russian context, Morosanova, Fomina, & Bondarenko (2015) draw our attention to the personality and cognitive approaches applied in self-regulation and academic success interactions. Furthermore, Russian didactics show that a dialectical relationship rather than a difference in teachercentered guidance and self-regulated learning characterizes the instructional process (Rakhkochkine, 2012). Nonetheless, self-regulated learning profiles among Russian students, especially in the online context, require more research because training viewed as the ‘transfer of knowledge’ from the one who knows to those who do not know seems to be dominant in Russian schools, and didactics (Rakhkochkine, 2012; Uman, 2007). Furthermore, a recent study recommends Russian teachers to use learning technologies in order to help students to develop self-regulation in online environments (Khuziakhmetov & Amin, 2016). According to Littlejohn et al. (2016), self-regulated learning research focused on MOOCs is vital. Specifically, further empirical studies are needed to identify if MOOCs can facilitate self-regulatory learning skills (Barnard, Lan, To, Osland, & MANUS CRIP T ACCEP TED ACCEPTED MANUSCRIPT Lai, 2009) and the e-learning strategies (Littlejohn, Hood, Milligan, & Mustain, 2016; Wilson & Narayan, 2014). Since self-regulation measures should take into consideration the context of learning (Barnard, Lan, To, Osland, & Lai, 2009), proper questionnaires are recommended for online learning research (Broadbent & Poon, 2015). In determining the self-regulated learning behaviors of Russian students in MOOCs, the Online Self-Regulated Learning Questionnaire (OSLQ) has been chosen, which assesses self-regulated learning in online and blended environments (Barnard, Lan, & Paton, 2010). The questionnaire used has a brief 24 item scale (α= 0.92) from the original with 86 items, using a 5-point Likert-scale. Students' beliefs in self-regulated learning are measured through six sub-scale constructs “Goal Setting”, “Task Strategies”, “Time Management”, “Self- Evaluation”, “Help Seeking” and “Environment Structuring” (Barnard, Lan, Crooks, & Paton, 2008). Epstein, Santo, & Guillemin (2015) recommend the selection of a questionnaire based on its good and documented quality. OSLQ validity has been verified by confirmatory factor analysis (Ajzen & Albarracin, 2007; Barnard et al., 2009; Barnard et al., 2010; Chang et al., 2015). Moreover, OSLQ has the capability of differentiating the “Environment Structure” (α= 0.90) (Barnard et al., 2009) as MOOC requires, and it is applicable to cross-cultural research (Zolotykhin, 2015). No Russian version of OSLQ is available; consequently, an adaptation of the instrument is required (Guillemin, Bombardier & Beaton, 1993). A OSLQ Turkish adaptation (Korkmaz & Kaya, 2012), and a OSLQ Romanian version already exist (Chen & Bakker, 2014) nevertheless, Cross-Cultural Adaptation (CCA) of instruments could benefit from a different approach (Epstein et al., 2015; Lemos, Cardoso, & Sougey, 2016; Orts-Cortés et al., 2013; Schaffer & Riordan, 2003). For example, this research includes the Content Validity Index, which is vital for new contexts (Alavi-Majd & Nikanfar, 2015; Peirce, Brown, Corkish, Lane, & Wilson, 2016; Zamanzadeh, Ghahramanian, Rassouli, Abbaszadeh, 2015). This research attempts to validate the OSLQ from three different angles. The first uses CVI and the estimation of the modified kappa coefficient (k) with seven experts. The second approach is to assess the verbal understanding scale by the students in a MOOC. Thirdly, a psychometric test was carried out to validate the instrument from a crosscultural angle. Considering the validity of the OSLQ, the Russian version is expected to become an identical measure to the other language versions mentioned above (Beaton, Bombardier, Guillemin & Ferraz, 2000). Agreeing with Barnard, Lan, Crooks, & Paton (2008), due to the increased use of MOOCs, OSLQ validation in the Russian context can assess and explore how learners develop self-regulatory learning skills. Although there are studies examining the regulation of the learners in relation to process-oriented characteristics (De Backer, Van Keer & Valcke, 2016; Drozdova & Sarmanova, 2011), the identification of participant cohorts and motivations require further studies (Salmon, Pechenkina, Chase & Ross, 2016). This study is a first step in this direction. This paper is structured as follows: the next section is devoted to methodology and data analysis. This is followed by the results of the research. Next, the findings and analysis are discussed as well as the limitations of the study. Finally, implications of the research are commented on the conclusion. MANUS CRIP T ACCEP TED ACCEPTED MANUSCRIPT 2. Methods The contextualization of self-regulated learning is a critical feature (Winne, 2010), which is essential for CCA (Boekaerts, 2002; Epstein et al., 2015; Gjersing, Caplehorn & Clausen, 2010). The relevance of context in learning goals and strategies has been empirically highlighted (Volet & Renshaw, 1995). Moreover, it is argued that a sociocultural theory from the experience of self-regulated learning could be defined by the context of cross-cultural research (Shi, Frederiksen, & Muis, 2013). 2.1.Translation process The method of CCA used in the OSLQ in the Russian context (Epstein et al., 2015), was as follows: after getting the permission of the author, the OSLQ was translated from English to Russian. The translation was realized according to the adapted process of the guidance of Beaton et al. (2000), Epstein et al. (2015) and Guillemin et al. (1993): Stage 1. Original translation (T1 and T2) was done from English into Russian by two Russian natives proficient in English: one professional translator aware of the research context, and the other, who was not aware of the context. Stage 2. A synthesis of both translations (T1 and T2) and discrepancies were resolved by a Russian native with linguistic knowledge and high proficiency in English. Stage 3. Back Translation (B1T1/T2 and B2T1/T2): Two native English speakers with a knowledge of Russian, worked with the synthesis to create two back-translated documents (B1T1/T2 and B2T1/T2). Stage 4. The review of B1T1/T2 and B2T1/T2 with the original OSLQ, resulted in the pre-final Russian adaptation. This synthesized version was examined by three Russian natives fluent in English. According to this review, changes were made to Item 9 and 18. 2.2.Measures 2.2.1. Content Validity Index Once the expert committee assured conformity of the translated and the original instruments, in step 5, the pre-final version of OSLQ was assessed employing CVI by seven subject matter experts (SMEs) in the comprehensive terms of clarity, cultural equivalence and relevance. This study follows the method proposed by Seo, MacEntee, & Brondani (2015), with some adjustments. This method was chosen because the calculation is easy and provides item level results (Peirce et al., 2016). Taking into account Sireci & Faulkner-Bond’s (2014) recommendations, SMEs were selected because of their knowledge and skills in e-learning. Three of them had specific knowledge of MOOCs. The results for each item and for the whole instrument were averaged for every comprehensive sub-scale and the six scales of the OSLQ. 2.2.1.1. SMEs Procedure The link to the questionnaire was posted to the seven SMEs by email, defining the study and, explaining how to rate 1) clarity, 2) conceptual equivalence; and 3) the relevance index of the OSLQ items. The Likert type scale with four possible responses, from 0 to 3, was used. According to the procedure of Seo, MacEntee, & Brondani (2015), clarity, equivalence and relevance were assessed as follows: 1) Clarity was measured from 0 = ”not at all clear” to 3 = ”very clear”. Experts defined whether the language of the questionnaire was explicit and unambiguous. MANUS CRIP T ACCEP TED ACCEPTED MANUSCRIPT 2) With a view to acquiring equivalence of the terms, as previous research reported (Orts-Cortés et al., 2013), semantic, colloquial, experiential and conceptual assessments of the OSLQ Russian version were gleaned from experts. The score was rated from 0 “not at all equivalent” to 3 “equivalent”. 3) In the relevance index, identification of items in MOOC context was evaluated with a Likert scale from 0 to 3 (0= not relevant or unrepresentative, 1= needs extensive review to be representative, 2= little revision needs to be representative and 3= relevant or representative). 2.2.1.2. Data analysis of CVI The content validity criteria of items were determined at scale (S-CVI/AV) and at item (I-CVI) level, and inter-rater score for each item was calculated as indicated in Peirce et al. (2016). In addition, following Polit, Beck, & Owen (2007) recommendations, the I- CVI questionnaire was rated. Whereby, I-CVI refers to the item-level CVI and the S- CVI/AV is the median of I-CVIs. S-CVI measured the median of the I-CVI for each sub-scale and the mean of the I-CVI (Kassam-Adams, Marsac, Kohser, Kenardy et al., 2015; Polit & Beck, 2006). Mean scores (S-CVI) were calculated at an OSLQ scale level and for the overall instrument. Based on the methods previously described by Seo, MacEntee, & Brondani (2015), I- CVI per item was counted by experts in each of the three comprehensive sub-scales: clarity, conceptual equivalence and relevance. At the item level, as indicated in Polit et al. (2007), I-CVI measures relevance by concurrence of experts, in 2 and 3, on a scope from 0 to 3, divided by seven SMEs. Then, the median of the total I-CVIs (S-CVI/AV) was also computed for the clarity, conceptual equivalence and relevance of each subscale (Polit et al., 2007; Zamanzadeh et al., 2015). Also, scale-level index of the OSLQ (S-CVI) was calculated by the S-CVI/AV for each scale of the OSLQ (“Environment Structuring”, “Time Management”, “Help Seeking”, “Goal Setting”, “Self-Evaluation” and “Task Strategies”). This approach was supplemented with the standard deviation and mean of the scales. The values of the scale between 0 and 1 were considered “content invalid”, values 2 and 3 were regarded as “content valid”. Thus, two dichotomous categories of responses were formed for each item. Then, with the sum of items (I-CVIs), division of I-CVIs was calculated. Data management was performed with OpenOffice Calc and SPSS version 22. Items are considered acceptable if they are ≥0.78 I-CVI, and unacceptable if they are under 0.78 (Peirce et al., 2016; Polit et al., 2007). The S-CVI minimum scores of each sub-scale are ≥0.80 and ≥0.90, as Orts-Cortés et al. (2013) reported. According to this, if S-CVI>0.90 the content validity of the scale is accepted. Following other studies (Orts-Cortés et al., 2013; Seo, MacEntee, Brondani, 2015; Squires et al., 2013) and recommendations (Polit & Beck, 2006; Polit et al., 2007; Wynd, Schmidt, & Schaefer, 2003), the modified kappa coefficient (k) was used to evaluate content validity of a possible random agreement. Polit (2015) establishes multi-rater kappa as an index of item agreement among the experts, where non-relevance agreement is disregarded. According to the previous research process of Orts-Cortés et al. (2013) and Squires et al. (2013) items’ deviation of response was evaluated through the modified kappa criteria as: Insufficient (≤0.39); Passable 0.40–0.59; Good 0.60–0.73 and Excellent >0.74, in order to avoid the CVI calculation deficiency. The procedure used is as proposed by Polit et al. (2007) and reported by Orts-Cortés et al., (2013) and Zamanzadeh et al., (2015): numerical values of conformity (Pc) and I- CVI were counted as Kappa: K=(I-CVI-Pc)/(1-Pc). First, counting item accord: Pc = MANUS CRIP T ACCEP TED ACCEPTED MANUSCRIPT [N!/A!(N-A)!]*.5N, SMEs is indicated by N. Experts recognizing the importance of the item are represented by A. Negative kappas, determined previously by Wynd et al. (2003), were considered as disagreement between experts. Squires et al. (2013) has already noted an inconsistency with I-CVI results between 0.63 and 0.77, seen as good in modified kappa. Consequently, the method followed in this research was the same as used by Orts-Cortés et al. (2013) where potentially problematic items (PPI) have 0.60 or less of kappa coefficient. 2.2.2. Socio-demographic data Socio-demographic characteristics of the sample were calculated using basic descriptive analysis. The self-regulation levels of the participants were obtained from the descriptive statistical test of the 45 students. The mean (M) and standard deviation (SD) amounts for every scale of the OSLQ and for every item in the three sub-scales were calculated. Although previous research determined that only answers 4 and 5 were valid (Lemos et al., 2016), in the present study it was established that if mean score is greater than 3.5 (M> 3.5) it is maintained in the instrument and if the score is between 2.5 and 3.5 (2.5 <M <3.5) or below 2.5 (M <2.5), it is revised. The same sections were tested in the interpretation of the level of self-regulated learning. With the test of Kolmogorov–Smirnov, variables (p > 0.05) were calculated, indicating non-normality. If P is under 0.05 it reveals that the distribution is significantly different from normal. In this research, the p value was higher than 0.05 showing equalities in the variances. For analysis the ANOVA test was used in order to examine differences in self-regulated learning due to participants’ gender and age in relation to MOOCs. Gender was measured with the test of Mann-Whitney U, and age variable was calculated using Kruskal Wallis test. Lastly, in order to analyze the use of diverse scales and evidence of linear correlation in the sample, the correlation coefficient of Spearman’s rho (q) was applied (Shani & Gunawardana, 2011). Homogeneity of variances of the variables was completed by the test of Levene. For the interval data (1- 2 or 4-5) of online self-regulated learning, percentages of frequency of agreement in the students’ answers were calculated for each OSLQ sub-scale (Hallgren, 2012). 2.2.3. Pilot-test and verbal understanding scale Following Kearney’s (2016) recommendations, the content validation was explored using a more pragmatic angle, inspired by Newman, Lim, & Pineda, (2013) and Zamanzadeh et al. (2015). Thus, the pre-final Russian version of the OSLQ was tested with 45 engineering students of the MOOCs offered by Samara National Research University. Through Stepik’s MOOC platform, answers were collected in June and July 2016. An online questionnaire and a verbal understanding scale were used. This approach also allowed to find out how potential respondents understood and responded to each item (Gehlbach & Brinkworth, 2011). The online questionnaire was created using Google form and sent by email address. Participants answered anonymously with the option of indicating their e-mail. They were offered no incentive for participation. With the purpose of minimizing the effect of social desirability acquiescence, the method bias described by Classen et al. (2010) and Xu, Dinev, Smith & Hart (2011) was followed. In accordance with the procedure of Podsakoff, MacKenzie, Lee, & Podsakoff (2003), method bias can be performed by a common scale, thus the number of scale points was varied from the OSLQ scale (Likert, 1 to 5) and the verbal understanding scale (0 to 5, Likert). To engage participants and prevent boredom, demographic questions were presented at the end. MANUS CRIP T ACCEP TED ACCEPTED MANUSCRIPT Students measured the verbal understanding of each item of the Russian version and their self-regulated learning competencies in the MOOC. Lemos et al. (2016) state that the analysis of the participants’ understanding accentuates their comprehension. Indeed Artino, La Rochelle, Dezee & Gehlbach (2014) argue that the validity of the questionnaire has to be linked to participants’ understanding of each item similar to the original intention. According to Morales (2007), in relation to item analysis, it is possible to test whether subjects seem to understand the formulation with the intended meaning; otherwise, the item should be removed or reformulated. To evaluate the comprehension of the items, this research adhered to the procedure of Vallerand (1989, cited in Kristjansson, Desrochers, & Zumbo, 2003). The participants assessed the OSLQ using an adaptation of the Likert verbal understanding sub-scale previously applied by Lemos et al. (2016). Understanding was measured from 0: “I did not understand at all” to 5: “I understood completely”. 2.3.Data analysis In relation to reliability and validity, the term “estimated” was used as recommended by Newman et al. (2013). Estimated internal consistency reliability was assessed for each comprehensive sub-scale, the overall, and in addition it was calculated in the pilot testing phase by coefficient alpha. The relation of items on a scale under the same construct, item amount in a scale level and equivalences between items were measured by estimating internal consistency (Artino et al., 2014; Noar, 2003). Since Likert-type scales were used, the Cronbach’s alpha coefficient was calculated for sub-scales as indicated in Gliem & Gliem (2003). As the OSLQ has more than one scale, the estimated reliability index for each of these scales was reported (López, Valenzuela, Nussbaum, & Tsai, 2015). Although coefficient alpha varies from 0 to 1, Gliem & Gliem (2003) underlines that no lower limit is considered. In fact, Noar (2003) points out that the increase in correlation of the items impacts the coefficient alpha. Consequently, Cronbach’s alpha values of 1.0 raise items’ internal uniformity in the scale (Gliem & Gliem, 2003). For instance, Field (2003) considers 0.8 as a reliable scale; and insufficient under this value. Therefore, values 0 to 1, α ≥ 0.80 and α ≥ 0.90 illustrate good and excellent levels of internal consistency reliability (Lovelace & Brickman, 2013). Thus, this research is based on Cronbach’s alpha classification of Babbie (1992), whereby very high with 0.90-1.00, 0.70-0.89 is high, moderate is 0.30-0.69 and low is 0.00-0.30. 3. Results During the translation process some stylistic changes were made. Items that did not match the original were revised and rewritten. Thus, the term “distance learning” was substituted by “e-learning”. The global content validity of the Russian adaptation of the OSLQ (S-CVI) by the SMEs, was 0.89, the lowest amount of I-CVI was 0.66 and the greatest 1, prior to revision of items (Table 1). Furthermore, k was measured as good. Table 1 Content Validity Index of the Russian adaptation of the OSLQ. Scale Items I-CVI Pc Goal Setting 1. I set standards for my assignments in online courses. 0.76 0.127 2. I set short-term (daily or weekly) goals as well as long-term goals (monthly or for 0.80 0.091 MANUS CRIP T ACCEP TED ACCEPTED MANUSCRIPT the semester). 3. I keep a high standard for my learning in my online courses. 0.80 0.091 4. I set goals to help me manage studying time for my online courses. 0.71 0.164 5. I don’t compromise the quality of my work because it is online. 0.80 0.091 Environment Structuring 6. I choose the location where I study to avoid too much distraction. 0.95 0.023 7. I find a comfortable place to study. 0.90 0.059 8. I know where I can study most efficiently for online courses. 0.90 0.059 9. I choose a time with few distractions for studying for my online courses. 0.90 0.039 Task Strategies 10. I try to take more thorough notes for my online courses because notes are even more important for learning online than in a regular classroom. 0.80 0.091 11. I read aloud instructional materials posted online to fight against distractions. 0.95 0.023 12. I prepare my questions before joining in the chat room and discussion. 0.95 0.023 13. I work extra problems in my online courses in addition to the assigned ones to master the course content. 1 0.007 Time Management Help 14. I allocate extra studying time for my online courses because I know it is timedemanding. 0.95 0.023 15. I try to schedule the same time every day or every week to study for my online courses, and I observe the schedule. 0.95 0.023 16. Although we don’t have to attend daily classes, I still try to distribute my studying time evenly across days. 1 0.007 Help Seeking 17. I find someone who is knowledgeable in course content so that I can consult with him or her when I need help. 1 0.007 18. I share my problems with my classmates online so we know what we are struggling with and how to solve our problems. 1 0.007 19. If needed, I try to meet my classmates face-to-face. 0.95 0.023 20. I am persistent in getting help from the instructor through e-mail. 0.90 0.039 Self- Evaluation 21. I summarize my learning in online courses to examine my understanding of what I have learned. 0.95 0.023 22. I ask myself a lot of questions about the course material when studying for an online course. 0.90 0.039 23. I communicate with my classmates to find out how I am doing in my online classes. 0.66 0.200 24. I communicate with my classmates to find out what I am learning that is different from what they are learning. 0.85 0.054 MANUS CRIP T ACCEP TED ACCEPTED MANUSCRIPT S-CVI for every scale of the OSLQ resulted 0.78 in “Goal Setting” below the acceptable level of 0.80, “Environment Structuring” with 0.91, 0.84 for “Self-Evaluation”, “Task Strategies” with 0.92, 0.96 for “Help Seeking” and “Time Management” with 0.96. During the process three items 1, 4 and 23 were I-CVI under 0.78, acknowledged not acceptable. Two of them were in the scale “Goal setting”: item 1 (I-CVI = 0.76, K = 0.70) and item 4 (I-CVI = 0.71, K = 0.63). The third item was in the scale “Self- Evaluation”: item 23 (I-CVI = 0.66, K = 0.57). Moreover, item 23, was identified as PPI due modified kappa coefficient of 0.57. 3.1. Clarity of OSLQ Elements The Russian version of the OSLQ was evaluated as clear (S-CVI = 0.91), with the exception of the items 1 (I-CVI = 0.57, K = 0.41), item 3 (I-CVI = 0.71, K = 0.65), item 4 (I-CVI = 0.71, K = 0.65) and item 23 (I-CVI = 0.57, K = 0.41). The other 20 elements showed 0.85 I-CVI values of clarity. There were remarkable items with (I-CVI = 1, K = 1) 6, 7, 8, 9 from the scale “Environment Structuring”; 14, 15, 16 from “Time Management”; items 11, 12, 13 “Task Strategies”; 17, 18, 19 from “Help Seeking” and item 21, 22 “Self-Evaluation”. Two items with a k of 0.41 were highlighted as PPI: 1 from the “Goal Setting” scale and 23, from the scale “Self-Evaluation”. Closer examination of the items revealed an average score lower than two points: 23 (M = 1.71; SD = 0.76) and 1 (M = 2; SD = 1). Cronbach’s alpha (α) reliability coefficient was 0.95 which indicated that this sub-scale represents possibly higher item reliability. 3.2. Cultural Equivalence between original OSLQ and Russian version of OSLQ Two items were under the minimum CVI value and significant agreement, I-CVI of 0.71 and K = 0.65, item 23 (M = 1.85; SD = 0.69) from the scale “Self-Evaluation” and from the scale “Goal Setting”, item 2, with M = 2.14 and SD = 0.89. Item 23 is the only one that gets an average score less than two points. The standard deviation below 1, shows homogeneity in SME ratings. Items with values of 1 (I-CVI = 1, K = 1) were 6, 7, 8 from the scale “Environment Structuring”; items 11, 13 “Task Strategies”, items 14, 15 “Time Management”; items 17, 18, 20 from the scale “Help Seeking” and item 21 from “Self-Evaluation”. The rest items values, I-CVI of 0.85 and K = 0.84 were deemed content valid. The equivalence results were considered appropriate (S-CVI = 0.91) by the SMEs. This sub-scale, also, represent possibly higher item reliability (α = 0.95). 3.3. Relevance of Russian version OSLQ Items sample MOOC context was measured for relevance as S-CVI = 0.85. This subscale has an alpha coefficient of 0.93, considered convenient. Five items were unacceptable (ICV = 0.71, K = 0.65) and accounted to be reviewed: 5 (M = 2.14; SD = 0.89) from the scale “Goal Setting”, 8 (M = 2; SD = 0.81) and 7 (M = 1.85; SD = 1.06) from the scale “Environment Structuring”, 10 (M = 2.14; SD = 0.89) from the scale “Task Strategies” and from the scale “Self-Evaluation” the item 23 (M = 1.71, SD = 0.95). The item 4 (M = 2; SD = 1) from the scale “Goal Setting”, fell below with an I-CVI = 0.57 and a modified kappa value moderate (k = 0.41), determined as PPI. Of these, items 4, 7 and 2 are the only ones with a standard deviation point greater than or equal to 1. The two items with less dispersion in their valuations are 13, from “Task Strategies” and 18, from “Help Seeking”, (M = 2.71; SD = 0.48), both of them were MANUS CRIP T ACCEP TED ACCEPTED MANUSCRIPT Cigdem, H. (2015). How Does Self-Regulation Affect Computer-Programming Achievement in a Blended Context?. Contemporary Education Technology, 6(1), 19–37. Classen, S., Winter, S. M., Velozo, C. A., Bédard, M., Lanford, D. N., Brumback, B., & Lutz, B. J. (2010). Item development and validity testing for a self- And proxy report: The safe driving behavior measure. American Journal of Occupational Therapy, 64(2), 296–305. http://doi.org/10.5014/ajot.64.2.296 De Backer, L., Van Keer, H., & Valcke, M. (2016). Is collaborative learners’ adoption of metacognitive regulation related to students’ content processing strategies and the level of transactivity in their peer discussions? European Journal of Psychology of Education. http://doi.org/10.1007/s10212-016-0323-8 Drozdova M., & Sarmanova J. (2011). Experimental assessment of virtual students. In 10th European Conference on e-Learning, ECEL 201110th European Conference on e-Learning, ECEL 2011. Proceedings of the European Conference on Gamesbased Learning. Retrieved from https://www.holaportal.com/publication/experimental-assessment-of-virtual- students-48680/ Epstein, J., Santo, R. M., & Guillemin, F. (2015). A review of guidelines for crosscultural adaptation of questionnaires could not bring out a consensus. Journal of Clinical Epidemiology, 68(4), 435–441. http://doi.org/10.1016/j.jclinepi.2014.11.021 Field, A. (2003). Designing a questionnaire. Questionnaire Design, 6. http://doi.org/10.4135/9781473913943.n3 Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin. http://doi.org/10.1037/h0031619 Floyd, F. J., & Widaman, K. F. (1995). Factor analysis in the development and refinement of clinical assessment instruments. Psychological Assessment, (7(3)), 286–299. http://doi.org/10.1037/1040-3590.7.3.286 Fuster, H., Carbonell, X., Pontes, H. M., & Griffiths, M. D. (2016). Spanish validation of the Internet Gaming Disorder-20 (IGD-20) Test. Computers in Human Behavior, 56, 215–224. http://doi.org/10.1016/j.chb.2015.11.050 Gehlbach, H., & Brinkworth, M. E. (2011). Measure twice, cut down error: A process for enhancing the validity of survey scales. Review of General Psychology, 15(4), 380–387. http://doi.org/10.1037/a0025704 George, D., & Mallery, P. (2003). SPSS for Windows Step by Step: Answers to Selected Exercises. A Simple Guide and Reference, 63. http://doi.org/9780335262588 Ghorbani, F., & Montazer, G. A. (2015). E-learners’ personality identifying using their network behaviors. Computers in Human Behavior, 51(PA), 42–52. http://doi.org/10.1016/j.chb.2015.04.043 Gjersing, L., Caplehorn, J. R. M., & Clausen, T. (2010). Cross-cultural adaptation of research instruments: language, setting, time and statistical considerations. BMC Medical Research Methodology, 10, 13. http://doi.org/10.1186/1471-2288-10-13 Gliem, J. A., & Gliem, R. R. (2003). Calculating, interpreting, and reporting Cronbach’s alpha reliability coefficient for Likert-type scales. Midwest Research-to-Practice Conference in Adult, Continuing, and Community Education. Guillemin, F., Bombardier, C., & Beaton, D. (1993). Cross-cultural adaptation of healthrelated quality of life measures: Literature review and proposed guidelines. Journal of Clinical Epidemiology, 46(12), 1417–1432. http://doi.org/10.1016/0895- 4356(93)90142-N MANUS CRIP T ACCEP TED ACCEPTED MANUSCRIPT Hallgren, K. A. (2012). Computing inter-rater reliability for observational data : An overview and tutorial. Tutor Quant Methods Psychol, 8(1), 23–34. Hood, N., Littlejohn, A., & Milligan, C. (2015). Context counts: How learners’ contexts influence learning in a MOOC. Computers and Education, 91, 83–91. http://doi.org/10.1016/j.compedu.2015.10.019 Hyrkäs, K., Appelqvist-Schmidlechner, K., & Oksa, L. (2003). Validating an instrument for clinical supervision using an expert panel. International Journal of Nursing Studies, 40(6), 619–625. http://doi.org/10.1016/S0020-7489(03)00036-1 Jou, M., Lin, Y.-T., & Wu, D.-W. (2014). Effect of a blended learning environment on student critical thinking and knowledge transformation. Interactive Learning Environments, 4820(November), 1–17. http://doi.org/10.1080/10494820.2014.961485 Kassam-Adams, N., Marsac, M. L., Kohser, K. L., Kenardy, J. A., March, S., & Winston, F. K. (2015). A new method for assessing content validity in model-based creation and iteration of eHealth interventions. Journal of Medical Internet Research, 17(4), e95. http://doi.org/10.2196/jmir.3811 Kearney, M. H. (2016). Under-Appreciated Steps in Instrument Development, Part I: Starting With Validity. Research in Nursing and Health, 39(2), 81–83. http://doi.org/10.1002/nur.21713 Khuziakhmetov, A. N., & Amin, A. S. (2016). The Study of the Application Rate of Effective Learning Technologies in Self- Regulation of KFU and VIIU Students, 11(6), 1073–1078. http://doi.org/10.12973/ijese.2016.377a Kirmizi, O. (2014). Self-regulated Learning Strategies Employed by Regular , Evening , and Distance Education English Language and Literature Students. Anthropologist, 18(2), 447–460. Kitsantas, A. (2013). Fostering college students’ self regulated learning with learning technologies. Hellenic Journal of Psychology, 10(3), 235–252. Korkmaz, O., & Kaya, S. (2012). Adapting Online Self-Regulated. Turkish Online Journal of Distance Education, 13(1), 52–67. Kristjansson, E. a, Desrochers, A., & Zumbo, B. (2003). Designer ’ s Corner Translating and Adapting Research : A Guide for Practitioners. Measurement, 35(2), 127–142. Lan, W. Y., Bremer, R., Stevens, T., & Mullen, G. (2004). Self-regulated learning in the online environment. In Paper presented at the 2004 annual meeting American Educational Research Association. San Diego. Lemos, I. L., Cardoso, A., & Sougey, E. B. (2016). Cross-cultural adaptation and evaluation of the psychometric properties of the Brazilian version of the Video Game Addiction Test. Computers in Human Behavior, 55, 207–213. http://doi.org/10.1016/j.chb.2015.09.019 Littlejohn, A., Hood, N., Milligan, C., & Mustain, P. (2016). Learning in MOOCs: Motivations and self-regulated learning in MOOCs. The Internet and Higher Education, 29, 40–48. http://doi.org/10.1016/j.iheduc.2015.12.003 Lloyd, L. K., Bishop, P. A., Walker, J. L., Sharp, K. R., & Richardson, M. T. (2016). Measurement in Physical Education and Exercise Science The Influence of Body Size and Composition on FITNESSGRAM ( r ) Test Performance and the Adjustment of FITNESSGRAM ( r ) Test Scores for Skinfold Thickness in Youth, 7841(March 2014), 37–41. http://doi.org/10.1207/S15327841MPEE0704 López, X., Valenzuela, J., Nussbaum, M., & Tsai, C.-C. (2015). Some recommendations for the reporting of quantitative studies. Computers and Education, 91, 106–110. http://doi.org/10.1016/j.compedu.2015.09.010 MANUS CRIP T ACCEP TED ACCEPTED MANUSCRIPT Lovelace, M., & Brickman, P. (2013). Best practices for measuring students’ attitudes toward learning science. CBE Life Sciences Education, 12(4), 606–617. http://doi.org/10.1187/cbe.12-11-0197 MacCallum, R. C., Widaman, K. F., Preacher, K. J., & Hong, S. (2001). Sample size in factor analysis: The role of model error. Multivariate Behavioral Research, (36(4)), 611–637. http://doi.org/10.1207/S15327906MBR3604_06 McLoughlin, C., & Lee, M. J. W. (2010). Personalised and self regulated learning in the Web 2. Australasian Journal of Educational Technology, 26(1), 28–43. http://doi.org/10.1523/JNEUROSCI.4255-09.2010 Morales, P. (2007). La fiabilidad de los tests y escalas. Estadistica Aplicada a Las Ciencias Sociales, 36. Morosanova, V. I., Fomina, T. G., & Bondarenko, I. N. (2015). Academic achievement: Intelligence, regulatory, and cognitive predictors. Psychology in Russia: State of the Art, 8(3), 136–157. http://doi.org/10.11621/pir.2015.0311 Newman, I., Lim, J., & Pineda, F. (2013). Content Validity Using a Mixed Methods Approach: Its Application and Development Through the Use of a Table of Specifications Methodology. Journal of Mixed Methods Research, 7(3), 243–260. http://doi.org/10.1177/1558689813476922 Noar, S. M. (2003). The Role of Structural Equation Modeling in Scale Development. Structural Equation Modeling: A Multidisciplinary Journal, 10(4), 622–647. http://doi.org/10.1207/S15328007SEM1004 Orts-Cortés, M. I., Moreno-Casbas, T., Squires, A., Fuentelsaz-Gallego, C., Maciá- Soler, L., & González-María, E. (2013). Content validity of the Spanish version of the Practice Environment Scale of the Nursing Work Index. Applied Nursing Research, 26(4), e5–e9. http://doi.org/10.1016/j.apnr.2013.08.006 Panayides, P. (2013). Coefficient alpha: Interpret with caution. Europe’s Journal of Psychology, 9(4), 687–696. http://doi.org/10.5964/ejop.v9i4.653 Peirce, D., Brown, J., Corkish, V., Lane, M., & Wilson, S. (2016). Instrument validation process: A case study using the Paediatric Pain Knowledge and Attitudes Questionnaire. Journal of Clinical Nursing, 25(11–12), 1566–1575. http://doi.org/10.1111/jocn.13130 Podsakoff, P. M., MacKenzie, S. B., Lee, J.-Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: a critical review of the literature and recommended remedies. The Journal of Applied Psychology, 88(5), 879–903. http://doi.org/10.1037/0021-9010.88.5.879 Polit, D. F. (2015). Assessing measurement in health: Beyond reliability and validity. International Journal of Nursing Studies, 52(11), 1746–1753. http://doi.org/10.1016/j.ijnurstu.2015.07.002 Polit, D. F., & Beck, C. T. (2006). The Content Validity Index: Are You Sure You Know What’s Being Reported? Critique and Recommendations. Research in Nursing & Health, (29), 489–497. http://doi.org/10.1002/nur.20147 Polit, D. F., Beck, C. T., & Owen, S. V. (2007). Is the CVI an Acceptable Indicator of Content Validity? Appraisal and Recommendations. Research in Nursing & Health, (30), 459–467. http://doi.org/10.1002/nur.20199 Rakhkochkine, A. (2012). On the dichotomy of teacher-centred instruction and selfregulated learning in Russian didactics. Zeitschrift Fur Erziehungswissenschaft, 15(3), 555–571. http://doi.org/10.1007/s11618-012-0323-7 Rebollo-Catalán, M. Á., García-Pérez, R., Buzón-García, O., & Vega-Caro, L. (2014). Las emociones en el aprendizaje universitario apoyado en entornos virtuales: Diferencias según actividad de aprendizaje y motivación del alumnado. Revista MANUS CRIP T ACCEP TED ACCEPTED MANUSCRIPT Complutense de Educacion, 25(1), 69–93. http://doi.org/10.5209/rev_RCED.2014.v25.n1.41058 Salmon, G., Pechenkina, E., Chase, A., & Ross, B. (2016). Designing Massive Open Online Courses to take account of participant motivations and expectations. British Journal of Educational Technology (BJET), (July), 1–18. http://doi.org/10.1111/bjet.12497 Sapargaliyev, D. (2014). Forced Necessity: MOOCs in Post-Soviet Countries. Leading Issues in elearning, 2, 23. Schaffer, B. S., & Riordan, C. M. (2003). A Review of Cross-Cultural Methodologies for Organizational Research: A Best- Practices Approach. Organizational Research Methods , 6(2), 169–215. http://doi.org/10.1177/1094428103251542 Seo, J., MacEntee, M., & Brondani, M. (2015). The use of Subject Matter Experts in Validating an Oral H ealth-Related Qualit y of Li fe measure in Korean. Health and Quality of Life Outcomes, 13(1), 138. http://doi.org/10.1186/s12955-015-0335-0 Shani, G., & Gunawardana, A. (2011). Evaluating recommendation systems. Recommender Systems Handbook, 257–298. http://doi.org/10.1007/978-0-387- 85820-3_8 Shi, Y., Frederiksen, C. H., & Muis, K. R. (2013). A cross-cultural study of selfregulated learning in a computer-supported collaborative learning environment. Learning and Instruction, 23(1), 52–59. http://doi.org/10.1016/j.learninstruc.2012.05.007 Sireci, S., & Faulkner-Bond, M. (2014). Validity evidence based on test content. Psicothema, 26(1), 100–7. http://doi.org/10.7334/psicothema2013.256 Squires, A., Aiken, L. H., van den Heede, K., Sermeus, W., Bruyneel, L., Lindqvist, R., … Matthews, A. (2013). A systematic survey instrument translation process for multi-country, comparative health workforce studies. International Journal of Nursing Studies, 50(2), 264–273. http://doi.org/10.1016/j.ijnurstu.2012.02.015 Tabachnick, B., & Fidell, L. (2001). Using multivariate statistics. Needham Heights. (A. & Bacon, Ed.). Boston. Tabuenca, B., Kalz, M., Drachsler, H., & Specht, M. (2015). Time will tell : The role of mobile learning analytics in self-regulated learning. Computers & Education, 89, 53–74. http://doi.org/10.1016/j.compedu.2015.08.004 Ting, K. Y., & Chao, M. S. (2013). The application of self-regulated strategies to blended learning. English Language Teaching, 6(7), 26–32. http://doi.org/10.5539/elt.v6n7p26 Torrance, H. (2012). Triangulation, Respondent Validation, and Democratic Participation in Mixed Methods Research. Journal of Mixed Methods Research, 6(2003), 111–123. http://doi.org/10.1177/1558689812437185 Uman, A. I. (2007). Teoriya obucheniya: ot edinoĭ – k triedinoĭ didaktike. Innovatsii v Obrazovanii, (4), 4–12. Usta, E. (2011). The examination of online self-regulated learning skills in web-based learning environments in terms of different variables. Turkish Online Journal of Educational Technology, 10(3), 278–286. Vallerand, R. (19 89 ). Vers un e méthodologie de la validation trans-culturelle de questionnaires psychologiques : implications pour la recherche en langue française. Psychologie Canadienne, (30), 662–678. Veletsi anos , G., Collier, A., & Schneid er, E. (2015). Digging deeper into learners’ experiences in MOOCs: Participation in social networks outside of MOOCs, notetaking and contexts surrounding content consumption. British Journal of Educational Technology, 46(3), 570–587. http://doi.org/10.1111/bjet.12297 MANUS CRIP T ACCEP TED ACCEPTED MANUSCRIPT Vo let , S. E. , & Ren sha w, P. D . ( 19 95) . C ros s-c ul tural diferences in university students' goals and perceptions of study settings for achieving their own goals. Higher Education, 30 (4), 407–433. Wilson, K., & Narayan, A. (2014). Relationships among individual task self-efficacy, self-regulated learning strategy use and academic performance in a computersupported collaborative learning environment. Educational Psychology, 3410(July 2014), 1–18. http://doi.org/10.1080/01443410.2014.926312 Winne, P. H. (2010). Improving measurements of self-regulated learning. Educational Psychologist, 45(4), 267–276. http://doi.org/10.1080/00461520.2010.517150 Winter, J. C. F., Dodou, D., & Wieringa, P. A. (2009). Exploratory Factor Analysis With Small Sample Sizes With Small Sample Sizes. Multivariate Behavioral Research, 44, 147–181. http://doi.org/10.1080/00273170902794206 Wynd, C. a, Schmidt, B., & Schaefer, M. a. (2003). Two quantitative approaches for estimating content validity. Western Journal of Nursing Research, 25(5), 508–518. http://doi.org/10.1177/0193945903252998 Xu, H., Dinev, T., Smith, J., & Hart, P. (2011). Information Privacy Concerns : Linking Individual Perceptions with Institutional Privacy Assurances. Journal of the Association for Information Systems, 12(12), 798–824. Available at: http://aisel.aisnet.org/jais/vol12/iss12/1 Zamanzadeh, V., Ghahramanian, A., Rassouli, M., Abbaszadeh, A., Alavi-Majd, H., & Nikanfar, A.-R. (2015). Design and Implementation Content Validity Study: Development of an instrument for measuring Patient-Centered Communication. Journal of Caring Sciences, 4(2), 165–178. http://doi.org/10.15171/jcs.2015.017 Zheng, L. (2016). The effectiveness of self-regulated learning scaffolds on academic performance in computer-based learning environments: a meta-analysis. Asia Pacific Education Review, 17(2), 187–202. http://doi.org/10.1007/s12564-016- 9426-9 Zolotykhin, S. A. (2015). The Use of Learner–Focused Approach When Designing of Massive Open Online Courses. Sovremennye Issledovaniya Sotsialnykh Problem, 0(8), 243. http://doi.org/10.12731/2218-7405-2015-8-19 MANUS CRIP T ACCEP TED ACCEPTED MANUSCRIPT Acknowledgements This work was supported by the Ministry of Education and Science of the Russian Federation and the 5-100 program. The main author thanks Professor Valeriy Dmitrievich Elenev, Director of the Institute of Aviation Technology at Samara National Research University, for invaluable support of this study; Carolyn Westbrook and Nataliya Fedorovna Kochanova for providing language help and writing assistance. The main author, also, gratefully acknowledges the helpful comments and suggestions received from the referees. MANUS CRIP T ACCEP TED ACCEPTED MANUSCRIPT • The Online self-regulated learning questionnaire Russian version appears to be valid. • Evidence of learners’ ability to self-regulate in Massive Open Online Course. • Not statistically significant differences between female and male students. • The instrument demonstrated good index of reliability. • It is estimated that the scale will not have convergent validity.