The development of secondary students’ feedback literacy : peer assessment as an intervention

This is a self-archived version of an original article. This version may differ from the original in pagination and typographic details. Author(s): Title: Year: Version: Copyright: Rights: Rights url: Please cite the original version: CC BY 4.0 https://creativecommons.org/licenses/by/4.0/ The development of secondary students’ feedback literacy : peer assessment as an intervention © 2020 the Authors Published version Ketonen, Laura; Nieminen, Pasi; Hähkiöniemi, Markus Ketonen, L., Nieminen, P., & Hähkiöniemi, M. (2020). The development of secondary students’ feedback literacy : peer assessment as an intervention. Journal of Educational Research, 113(6), 407-417. https://doi.org/10.1080/00220671.2020.1835794 2020 Full Terms & Conditions of access and use can be found at https://www.tandfonline.com/action/journalInformation?journalCode=vjer20 The Journal of Educational Research ISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/vjer20 The development of secondary students’ feedback literacy: Peer assessment as an intervention Laura Ketonen , Pasi Nieminen & Markus Hähkiöniemi To cite this article: Laura Ketonen , Pasi Nieminen & Markus Hähkiöniemi (2020): The development of secondary students’ feedback literacy: Peer assessment as an intervention, The Journal of Educational Research, DOI: 10.1080/00220671.2020.1835794 To link to this article: https://doi.org/10.1080/00220671.2020.1835794 © 2020 The Author(s). Published with license by Taylor & Francis Group, LLC View supplementary material Published online: 07 Dec 2020. Submit your article to this journal View related articles View Crossmark data The development of secondary students’feedback literacy: Peer assessment as an intervention Laura Ketonen , Pasi Nieminen , and Markus H€ ahki€ oniemi Department of Teacher Education, University of Jyv€ askyl€ a, Jyv€ askyl€ a, Finland ABSTRACT A growing body of research has recognized the importance of students’having active roles in feedback processes. Feedback literacy refers to students’understandings of and participation in feedback processes, and research on students’feedback literacy has so far focused on higher education; secondary schools have not received attention. This case study investigates secondary students’feedback literacy and its development in the context of formative peer assessment. From various data sources, three categories of students’feedback literacy were identified, and criteria for the levels of literacy in each category were created. The criteria were used in the coding of seventhand eighth-grade students’skills. The results show that students were able to develop their feedback literacy skills. Thus, secondary school students should be introduced to feedback literacy via, for example, formative peer assessment. ARTICLE HISTORY Received 18 May 2020 Revised 18 August 2020 Accepted 7 October 2020 KEYWORDS Feedback literacy; peer assessment; formative assessment; feedback; secondary education; science Introduction Feedback can significantly enhance or inhibit learning (Hattie & Timperley, 2007), and it has therefore generated great interest and considerable research. Ideally, feedback helps students achieve their goals (Hattie & Timperley, 2007). The studies on feedback have mainly concentrated on the features of efficient feedback, thus implying that the responsibility of a successful feedback process lies on the provider of feedback, who must consider its content, tone, and timing. Similar attention to the quality of feedback can be seen in research on peer assessment. Only recently has receiving feedback generated steady attention from scholars (Boud & Molloy, 2013; Delva et al., 2013; Jonsson, 2013;McLeanetal.,2015;Sutton,2012;Wiliam,2012; Winstone et al., 2017). Carless and Boud (2018)introduced a framework of feedback literacy describing the competences necessary to participate in feedback processes; it has generated considerable interest and encouraged further research. Studies have investigated students of higher education (Han & Xu, 2019a,2019b; Hey-Cunningham et al., 2020; Molloy et al., 2019) and academics (Gravett et al., 2020), but so far no study has investigated the feedback literacy of a much larger group: secondary students. Exploring the feedback literacy of secondary school students is the first goal of this study. Carless and Boud (2018) expressed the need for research on the development of students’feedback literacy combined with intervention. This article is such a study, and its second goal is to investigate the development of lower secondary school students’feedback literacy, when formative peer assessment is repeatedly used in students’physics and chemistry lessons. Peer assessment is used not only as a tool to uncover students’perceptions about assessment and feedback, but also as an intervention that provides students with the opportunities to reflect and practice feedback processes and hence to develop their feedback literacy. Feedback literacy Sutton (2012) conceptualized feedback literacy as “the ability to read, interpret and use written feedback”(p. 31). Carless and Boud (2018) built on his work and defined feedback literacy as “the understandings, capacities and dispositions needed to make sense of information and use it to enhance work or learning strategies”(p. 1316). Feedback literacy highlights the need for students’own activity in feedback processes. Though teachers provide feedback, no matter how useful it is, it does not automatically benefit the receiver. The feedback needs to be accepted, processed, and acted on by the receiver. Feedback literacy denotes these competences. The framework of Carless and Boud (2018) presents four features of students’feedback literacy, which are appreciating feedback, making judgments, managing affect, and taking action. Appreciating feedback includes understanding that feedback is for improvement, understanding that the recipient must have an active role in the feedback process, appreciating different forms and sources of feedback, and the ability to use technology in feedback processes. Making CONTACT Laura Ketonen [email protected] Department of Teacher Education, University of Jyv€ askyl€ a, PO Box 35, FI-40014 University of Jyv€ askyl€ a, Jyv€ askyl€ a, Finland. Supplemental data for this article can be accessed at https://doi.org/10.1080/00220671.2020.1835794. ß2020 The Author(s). Published with license by Taylor & Francis Group, LLC This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. THE JOURNAL OF EDUCATIONAL RESEARCH https://doi.org/10.1080/00220671.2020.1835794 judgments comprises capacities of judging one’s own and others’work, participating productively in peer feedback processes and developing self-evaluative skills. Managing affect includes maintaining emotional balance, avoiding defensiveness, having dialogue about feedback, and striving for continuous improvement. These three features (appreciating feedback, making judgments, and managing affect) add students’possibilities for the fourth feature, which is taking action. Taking action comprises understanding that using feedback requires the recipient’s activity and developing strategies for acting on feedback. The work of Carless and Boud (2018) has induced further research. Molloy et al. (2019) explored higher education students’feedback literacy from the students’perspective. Their results present seven groups containing altogether 31 categories of knowledge, capabilities, and skills of feedback literacy. Hey-Cunningham et al. (2020) designed a pilot of blended learning for postgraduate research students in academic writing that contained principles, exemplars, self-assessment, and peer assessment, and noticed that it improved students’ feedback literacy. In particular, students felt that they learned skills and gained experience about how to proceed with the received feedback. Han and Xu (2019b) explored the development of higher education students’feedback literacy, and used teachers’feedback on peer feedback as an intervention. Researchers reported a growth in feedback literacy, especially in students’ability to assess their peers’ work, that is, judging the quality and providing feedback for learning. They reported a considerable individual variation in students’development. Han and Xu (2019a) investigated higher education students’feedback literacy and its impact on engagement in the context of teachers’written corrective feedback. They found that individual students’skills of feedback literacy were unequally developed, which limited students’engagement with feedback. The researchers noticed not only that students’feedback literacy was dynamic and developed in the feedback processes, but also that the development appeared rather randomly. Peer assessment Building on previous research (Carless & Boud, 2018; Sadler, 1989; Topping, 2013), formative peer assessment is defined as a process in which students evaluate or are evaluated by their peers with the intention that both the assesses and assessees enhance their work or learning strategies in the process. When discussing peer assessment, the concept of feedback must be understood broadly, as both assessing and being assessed provide opportunities for receiving feedback. Carless and Boud defined feedback as “a process through which learners make sense of information from various sources and use it to enhance their work or learning strategies”(p. 1315). In peer assessment, the potential sources of information are many, including assessment criteria, assessed work, received feedback, interaction with classmates, and coaching from the teacher. Relationship of peer assessment and feedback literacy Peer assessment and feedback literacy are inter-related (Figure 1). Participating productively in peer assessment requires feedback literacy (Han & Xu, 2019b), and peer assessment provides opportunities to advance it. In the best case, feedback literacy and peer assessment support each other, as positive experiences of peer assessment predict teachers’frequent use of peer assessment (Panadero & Brown, 2017). Students’advancement in feedback literacy makes peer assessment a more applicable tool; therefore, feedback-literate students are more likely to get further opportunities to develop their feedback literacy with peer assessment. Researchers have a common understanding that peer assessment requires training (Gielen et al., 2010; Hovardas et al., 2014; Lu & Law, 2012; Topping, 2009; van Zundert et al., 2010), which implies that a certain level of feedback literacy is needed to enable a productive peer assessment process. If students are not feedback-literate enough to benefit from peer assessment, the practice is more likely experienced as malfunctioning and rejected. The prior research on peer assessment does not generally mention feedback literacy but recognizes its features. The next sections introduce research that presents how each of four features of feedback literacy (Carless & Boud, 2018) are needed in peer assessment as well as research that shows that peer assessment has the potential to advance each feature of feedback literacy. Appreciating feedback When students understand that feedback is a learning tool, they are more likely to benefit from peer assessment (Ketonen et al., 2020). Incomprehension in this area may result in unproductive practices, such as the provision of superficial positive feedback (Tasker & Herrenkohl, 2016)or friendship marking (Foley, 2013; Panadero et al., 2013), which occurs when students let social relationships affect the feedback they provide. Peer assessment can help tackle these issues because it is a tool for discussing the different aims of summative and formative assessment (Davis et al., 2007). Tasker and Herrenkohl (2016) reported that peer assessment provides a context for training and reflection that advances Figure 1. Relationship of feedback literacy and peer assessment. 2 L. KETONEN ET AL. students’appreciation of critical feedback and leads them from providing positive, superficial peer feedback to providing guidance. Another essential component of feedback literacy in peer assessment is appreciating feedback from different sources. Students tend to undervalue and disregard feedback from their peers (Foley, 2013). Peer assessment can help with this issue as well because it advances students’ appreciation of peers as a source of feedback (Crane & Winterbottom, 2008). Making judgments Peer assessment requires the skills of judging received feedback and judging others’work. Concerning judgments, Carless and Boud’s(2018) framework for feedback literacy concentrates on judging one’sownandothers’work, and Molloy et al. (2019) further included the aspect of processing received feedback. The ability to critically interpret feedback is an important skill that enhances the benefits of peer assessment (To & Panadero, 2019).Guidingstudentstobeactiveintheassessee’sroleby evaluating received feedback improves their motivation to engagement in peer assessment (Minjeong, 2009). Students need to understand the assessment criteria in order to judge their peers’work (Cartney, 2010;Foley,2013; Panadero et al., 2013). Moreover, a shared understanding of the criteria increases students’comfort with peer assessment (Panadero et al., 2013). Peer assessment can be used to train students to deepen their understanding of the criteria (Anker-Hansen & Andr ee, 2019; Black & Wiliam, 2018) and develop the ability to judge others’work (Han & Xu, 2019b). Managing affect Peer feedback may raise negative emotions (Cartney, 2010; Panadero, 2016). Assessees can be defensive against corrective peer feedback (Anker-Hansen & Andr ee, 2019; Ketonen et al., 2020; Tasker & Herrenkohl, 2016), and assessors may worry about assessees’negative emotions (Cartney, 2010; Davis et al., 2007). However, because of peer assessment’s interactive nature, affective issues cannot be avoided (Panadero, 2016). Managing affections is hence necessary, and for productive peer assessment, students need support with the emotional aspects related to both the assessors’and assessees’roles (Cartney, 2010). Fortunately, peer assessment improves psychological safety within a group (van Gennip et al., 2010), meaning that group members feel safer sharing their opinions and asking for help. This promotes elicitation of suggestions from peers and discussions about them, which relate to managing affect in Carless and Boud’s(2018) framework. Taking action Studies have noted students’reluctance to revise their work according to peer feedback (Anker-Hansen & Andr ee, 2019; Tsivitanidou et al., 2011;2012). As theorized by Carless and Boud (2018), this reluctance is the result of insufficient feedback literacy, such as an insufficient understanding of assessment criteria (Tsivitanidou et al., 2011) or formative assessment (Ketonen et al., 2020). As explained previously, peer assessment can help address these issues and is generally considered useful in encouraging students to act on feedback (Jonsson, 2013), especially when integrated in a deeper conversation about assessment (Cartney, 2010). Research questions Feedback literacy seems an inherent requirement of productive peer assessment, but it is little researched at the secondary level. In order to investigate secondary students’ feedback literacy and its development, the following research questions were addressed: Research Question 1:What kind of skills of feedback literacy do students have in the context of formative peer assessment? Research Question 2:How does students’feedback literacy develop during one year of using peer assessment repeatedly? Method Participants The study was carried out in an ordinary urban school in Finland. The intervention was conducted in cooperation with an experienced subject teacher and her two science classes (15 and 16 students, respectively); the study started at the beginning of the students’Grade 7 year and lasted until the middle of their Grade 8 year. At the beginning of the intervention, the students did not know each other well, but most had at least one friend in the class. In Finland, students study general science in Grades 1–6 with a class teacher and begin physics and chemistry in Grade 7 with a subject teacher. This transition into more specialized learning seemed like an ideal moment to introduce a new practice. Peer assessment is not yet a well-established practice in Finland (Atjonen et al., 2019), and when asked, approximately half the students had experiences with peer assessment, though none had used it regularly. The teacher was asked to join the study because of her well-organized, thoughtful working style, which enabled the intervention’s careful co-planning, and for her considerate but not overly individual teaching style. The study aimed at understanding and explaining the phenomena of peer assessment and feedback literacy instead of making broad generalizations, and the number of participants was kept small in order to enable knowing the students and gathering of rich data. Two students chose not to participate in the study; the total number of participants was 29. Twenty-two students participated in all parts of the first peer assessment, and 15 of those 22 participated in all parts of the second peer assessment, which allowed the researcher to track their development. Two of 15 students missed three or more of the five peer assessments that were arranged between the abovementioned two peer assessments, which diluted the intervention, and they were therefore left out of the study. Altogether, 13 students provided a satisfying dataset, and they were included in the analysis of the study. THE JOURNAL OF EDUCATIONAL RESEARCH 3 Procedure The study examined students in two similar peer assessments. The first (PA1) was conducted during the fall semester of Grade 7, and the second (PA2) was conducted a year later. In PA1 and PA2, the students conducted and reported on a physics inquiry. The topics (determining the speed of an object and determining the resistance of a wire) were different due to the curricula, but the difficulty of the tasks, the form of the lab reports, and the assessment criteria were similar. The students conducted the inquiry in groups but created the reports individually. After the reports were finished, they were assessed by their peers. The researcher and the teacher planned the pairings so that students were not partnered with someone they conducted the inquiry with and with sensitivity to social issues, such as pairing vulnerable students with considerate classmates. The students were provided with premade criteria in which each criterion was assessed with a 3-point Likert scale and optional written comments (Figure 2). After the peer assessment, the reports and feedback were returned to the students, and they had time to improve their work before returning it to the teacher for summative assessment. In addition to these two peer assessments, training sessions and several other peer assessments were arranged. Before PA1, the students had six training sessions, and they practiced peer assessment once. Between PA1 and PA2, they had five more peer assessments and one training session. All parts of the interventions are presented in Table 1. The training sessions and peer assessments were embedded in the curricula and consisted of class discussions led by the researcher or the teacher, both of whom used cards displaying examples of feedback (Figure 3), principles of peer assessment, written tasks (e.g., creating or testing assessment criteria), and actual peer assessments. The training focused on issues that previous research noted as important for successful peer assessment, specifically understanding the idea of formative assessment, understanding of the qualities of useful feedback, internalizing assessment criteria, comparing the criteria with the work, social rules, and practicing actual peer assessment. The key message was that peer assessment is for helping others learn and receiving help from others. The students remained focused during the class discussions and seemed to find the topics interesting. The intention of the discussions was not to find “the right answer,” but rather to encourage the students to share their thoughts. The training was flexible, as the intent was to respond to the students’needs. For example, when peer assessment was first practiced, the students struggled to deal with critical feedback; the issue was therefore included in the training and discussed before PA1. The teacher and the researcher had prolonged experience in co-teaching at the school, which helped co-planning the interventions and made sharing the responsibility in the classroom natural. During the written tasks and peer assessments, they circulated throughout the classroom and held discussions with students who needed help, encouraging them to use and trust their own judgment. Even though class sessions were planned so that the students were equipped with the necessary information, discussions with individual students and working groups appeared necessary and influential. Figure 2. Premade assessment criteria of PA2. 4 L. KETONEN ET AL. Research design and data A case study design was adopted for the exploratory nature of the research. The case study approach allowed us to know all the students and gather versatile data. The data include field notes and audio recordings of lessons, student interviews, and students’written work and feedback. The researcher placed recorders on the tables of each student pair and took field notes on each lesson in which she participated (approximately 60 lessons of 1.5 hr each). Students’ original and revised work and provided feedback were scanned. Besides the analysis, these documents were used as a basis of conversation in semi-structured, audio-recorded student interviews soon after PA1 and PA2. Analysis The analysis comprised three parts. In the first and second parts, the data were analyzed using a thematic analysis (Braun & Clarke, 2006) to identify and report patterns in the data. A thematic analysis was chosen for its flexibility: it is not bound to any specific theoretical framework, and it works well in analyzing data consisting of various forms. It can be used to find similarities and differences within the data, and it can produce unexpected insights. The patterns can be driven by either data or theory. The first part of the analysis was driven by theory. The aim was to recognize the features of feedback literacy that appeared in the data during peer assessment and to adjust the features of Carless and Boud’s(2018) framework to the context of formative peer assessment. We looked for data extracts that contained information on the students’feedback literacy and attached codes to them. The codes described the students’actions and attitudes relating to the elements of feedback literacy: appreciating feedback, making judgments, managing affect, and acting on feedback. After coding the whole dataset, the codes were gathered into preliminary categories. The distinctiveness of the categories was then examined, and the categories were adjusted and named. The data were recoded, and categories were re-examined and readjusted until no further changes were undertaken. In the end, three categories of feedback literacy skills were discerned. The goal of the second part of the analysis was to examine the skills that students had in the three categories of feedback literacy. During the first part of the analysis, we noticed students had varying feedback literacy skills, and we defined and described them in an iterative process. The analysis was driven by data, but its scope was limited to the previously identified categories. Within each category, we formed case groups with similar skill levels. Then, we examined and described the groups and returned to the data with revised descriptions. We revised the groups until new rounds did not produce changes to the groups or their descriptions. While remaining sensitive to the theory, we organized the groups from the most basic to the most advanced, thereby creating a criteria-based rubric for feedback literacy skills. In two cases, a skill level was created from the data of only one student, and we enriched that data with interviews with additional students in the spring of Grade 7 (Table 1). This enabled us to test our rubric and find more examples to describe the levels. In the third phase of analysis, the category criteria were used in evaluating seventhand eighth-grade students’feedback literacy, and multiple data sources were also used. We coded students’“Understanding of the purpose of feedback” Table 1. Interventions of the study. Time Activity Description Grade 7 fall Six sessions of training peer assessment skills (10–45 min) Reflecting on the use of assessment Reflecting on the quality of feedback Creating assessment criteria Comparing a work to created criteria Self-assessment with created criteria Reacting to feedback Peer assessment (task 40 min, assessment 30 min, improvement 10 min) Groups assessed other groups’inquiry plans. Afterwards, groups had time to improve their work. PA1 (task 3 hr, assessment 45 min, improvement 45 min) Each student assessed another student’s lab report with provided criteria. The researcher and the teacher planned the pairings. Afterwards students had time to improve their work. Interview Discussing PA1 Grade 7 spring Five peer assessments (approximately 5 min) Working pairs provided each other with feedback based on criteria after four peer assessments. Peer assessment (task 15 min, assessment 10 min, improvement 10 min) Each student assessed another student’s inquiry plan with provided criteria. The researcher and the teacher planned the pairings. Afterwards students had time to improve their work. Interview Discussing all previous PA sessions. Grade 8 fall Peer assessment training (30 min) Students categorized the feedback comments that they provided during PA1. Intervention was adapted from Tasker and Herrenkohl (2016). PA2 (task 3 hr, assessment 30 min, improvement 30 min) Each student assessed another student’s lab report with provided criteria. The researcher and the teacher planned the pairings. Afterwards students had time to improve their work. Interview Discussing PA2 Figure 3. The feedback example cards discussed in one training session. The class agreed that the first critique was worth considering, whereas the third was unhelpful, offensive, and not worth keeping. THE JOURNAL OF EDUCATIONAL RESEARCH 5 and “Engagement and interpretation of received feedback” skills using interviews as a primary data source, and to ensure reliability, we compared students’narratives to their own work, their received and provided feedback, and the work that they had assessed. If the findings needed further affirmation, we used field notes from and audio recordings of the lessons. In two cases (Students 4 and 8), even these data did not provide enough information to determine the category level, and these skills were left out of these students’results (Figure 4). The “Engagement in making revisions”skill was coded using students’original and revised work to recognize the changes they had made. Their engagement in making these changes was interpreted using interviews with students, field notes from the lessons, and audio recordings of the lessons. This study’s authors engaged in peer negotiation to test the levels and categories. Coding students’skills was a complex task that required precise interpretation and the management of versatile data because students expressed their feedback literacy skills in diverse parts of the interview, and as explained previously, the comments were interpreted with other data. Therefore, negotiation among the authors was Figure 4. The change in students’feedback literacy. 6 L. KETONEN ET AL. considered appropriate to examine the reliability of coding. Laura Ketonen, who was the only one involved in teaching, prepared the data of three regular and two specifically challenging cases for negotiation. The data from each student were gathered into one document, and the data extracts used in defining the levels of the categories were discussed. Pasi Nieminen and Markus H€ ahki€ oniemi explained their views on the Laura Ketonen’s coding, and discrepancies were carefully discussed. After the regular cases, the challenging ones were examined. We had a mutual agreement that, due to scant interviewee input from students, these cases were considerably more challenging than the regular cases, and definitively determining the level of students’ skills was not possible. The decision to leave one skill out of the results in two cases (Students 4 and 8) was made in the meeting. Results Theresultsaretwofold.Thefirstpartoftheresultsconsists of the categories with levels of students’feedback literacy (Tables 2,3,and4), and the second part consists of the paths of development of secondary students’feedback literacy. Categories and levels for students’feedback literacy Three categories of students’skills of feedback literacy were recognized in the context of formative peer assessment: understanding of the purpose of feedback, engagement and interpretation of received feedback, and engagement in making revisions. Understanding of the purpose of feedback (Table 2) relates to students’dispositions to feedback. The students on the first level do not appreciate any kind of feedback. The second level, appreciation of positive feedback, reflects the understanding of feedback as a judgment. By this conception, corrective feedback communicates a shortcoming or a failure, and it is not wanted. Conversely, students on the third level welcome corrective feedback, as they see it as a mediator of learning and improvement. Engagement and interpretation of received feedback (Table 3) describes students’activity in judging the received feedback. The first level is not being interested in feedback Table 2. Understanding of the purpose of feedback. Label Criteria Examples Level 1 No appreciation of feedback Student expresses that he/she does not appreciate feedback. R: Could the feedback [have] been such that you would have made some changes [to your work]? S: No … R: What do you think, if the teacher had given you feedback, would it have been more effective or the same? S: Exactly the same. (Student 5, PA1) Level 2 Appreciation of positive feedback Student expresses that receiving positive feedback is a satisfying or hopedfor situation. R: Would you say that you benefited from the feedback? S: Yes. R: How? S: It was positive, like it made me feel good and then you know that you did well. (Student 6, PA1) Level 3 Appreciation of corrective feedback Student expresses that receiving corrective feedback is a hoped-for situation. R: What did you think of having positive feedback? S: Well, that did not help me much …if I, like, I was not satisfied with this last part [of my work] because I ran out of time …so I hoped for help with what to add there. (Student 11, PA2) Table 3. Engagement and interpretation of received feedback. Label Criteria Examples Level 1 No interest in feedback Student expresses that he/she is not interested in engaging with the feedback. R: Okay, you received the feedback. I have it here. Erm, so what did you think when you received the feedback? S: Nothing. R: Okay. Did you read it? S: Nope. R: Okay. So you did not think anything. Why didn’t you read [it]? S: Since I am not, like, interested what it says. (Student 5, PA for seventh grade spring) Level 2 Reading feedback Student expresses that he/she has read the feedback but does not show signals of active interpretation of feedback compared with his/ her own work OR has problems with selectively rejecting the data. R: Tell me what you did. S9: So I added there that it [speed] is distance divided by time. Then I wrote that more accurately, the calculation. Since it is there [in the original work], a little small, so … When receiving feedback, the student found the comment of missing calculation amusing, since it was clearly written on the report. Nevertheless, the student did not reject the comment but wrote the same calculation in bigger letters. (Student 9, PA1) Level 3 Active interpretation of feedback The data show that the student actively interprets the feedback by comparing it to his/her own work and, if he/she finds it necessary, rejects the feedback. S12: I took, like, at least the most important ones [feedback comments] and then … R: Yes? S12: Well, the others I ignored, there was not time for everything. (Student 12, PA2) THE JOURNAL OF EDUCATIONAL RESEARCH 7