Supporting Engineering Physics Learning with Automated Formative Feedback

Research Paper Recommended citation: Immonen, P., Äijälä, M., & Naukkarinen, J. (2025). Supporting Engineering Physics Learning with Automated Formative Feedback. In Kangaslampi, R., Langie, G., Järvinen, H.-M., & Nagy, B. (Eds.), SEFI 53rd Annual Conference. European Society for Engineering Education (SEFI), Tampere, Finland. DOI: 10.5281/zenodo.17631376. This Conference Paper is brought to you for open access by the 53rd Annual Conference of the European Society for Engineering Education (SEFI) at Tampere University in Tampere, Finland. This work is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 4.0 International License. SUPPORTING ENGINEERING PHYSICS LEARNING WITH AUTOMATED FORMATIVE FEEDBACK P. Immonen a, 1 , M. Äijälä b, J. Naukkarinen c a LUT University, Lappeenranta, Finland, 0000-0002-3286-6840 b LUT University, Lappeenranta, Finland, 0000-0002-6626-4207 c LUT University, Lappeenranta, Finland, 0000-0001-6029-5515 Conference Key Areas: Teaching mathematics and physics in engineering education, Digital tools and AI in engineering education Keywords: online-assignment, feedback, engineering physics ABSTRACT This article investigates how students' exam performance is affected by receiving interactive formative feedback on a weekly assignment. The study was conducted in a physics course for first-year engineering students. Students were divided into two groups which both acted as an intervention group and as a control group in turn. One group received a specific Q-factor assignment with automated feedback in a form of tries and hints and a wave interference assignment without feedback. The other group received a Q-factor assignment without feedback and a wave interference assignment with tries and hints. The performance of students in different groups on weekly assignments and the final exam in the corresponding subject area was examined. The results showed that the assignment with multiple tries and hints increased students' learning of the topic. 1 P. Immonen [email protected] 1 INTRODUCTION Feedback is an essential part of learning and the research on feedback in learning is long-established and diverse. On a general level it has been noted that learning is more effective the more often feedback is received (Yong et al. 2021), (Krause, Stark, and Mandl 2009). However, the effects of feedback on learning can be very different depending on the formative or summative role of feedback (Brookhart 2018), the type of feedback (Attali & van der Kleij 2017), the timing of feedback (Pol et al. 2009), the focus of feedback (Fyfe et al. 2012), the prior knowledge of the learners (Lee 2017) and many other features of the learning situation. Brookhart (2018) suggests that the power of feedback lies especially in its formative use. Shute (2008) characterizes good formative feedback as timely, valid, focused, objective, clear, and well matched to the recipient’s previous knowledge level. A feedback model formulated by Hattie and Timperley (2007) proposes relevant feedback questions for the feedback’s recipient to be: “Where am I going?” “How am I going?” and “Where to next?” The model implies that feedback should a) highlight the task’s goal, b) position how the current performance relates to this goal (i.e. the gap or discrepancy), and c) provide hints on the next steps toward the objective. Research on feedback on solving mathematical problems often distinguishes between specific, task-level feedback and more strategic process-related or selfregulating feedback and it has been suggested that especially the novices benefit from the former wand the more proficient learners of the latter (Knaut et al. 2022). Interestingly, Lee’s (2015) method of feedback prompting the use of Polya’s problem-solving methods improved the learning effectiveness of moderate or lowlevel achieving students more than the learning of the high achievers, but this was suspected to be caused by the more limited scope of improvement of the already more knowledgeable learners. Under some circumstances feedback can weaken the learning instead of supporting it. Attali and van der Kleij (2017) noticed that an elaboration presented after a student had given a correct answer to a problem did not improve but marginally lowered the performance on the similar question. This was explained by the increased cognitive load due to processing of redundant information. In a similar vein, Fyfe et al. (2012) discovered that children with some prior knowledge of a correct mathematical solution strategy benefitted more from exploring the problem without feedback than with it. Also Roll et al. (2014) discovered that with children overusing help was associated with poorer learning of mathematics, suggesting that novice learners may require several tries before being able to make sense of the feedback. Contemporary online learning solutions provide new challenges and opportunities for feedback. Despite losing information on the recipients’ situation and often seen to be lacking in diversity and responsiveness, it can have marked benefits related to timeliness, (perceived) objectiveness and clarity in delivery. E-learning platforms also allow the students to try the assignments more than once, with the option of giving immediate feedback, and if needed, re-generating or randomising the task data in between the tries (e.g. Yourstone et al., 2010). This type of e-learning assignments, i.e. multiple tries feedback with hints have been recently found to enhance learning, by enabling students to learn from their mistakes (Schwerter et al., 2022). Such a trial-and error learning method is also preferred by the students, according to Brazhkin & Strakos, 2023), whose study suggests that students prefer to have three tries to an assignment, with concrete enough hints that allow them to solve the exercise by themselves. On the other hand, research also shows that giving more than two tries may be unnecessary (Yourstone et al., 2010), and that multiple tries may lead to guessing behaviour and related optimisation strategies that do not promote learning on the desired topic (Rhodes & Sarbaum, 2015). This study explores the effect of task-specific formative feedback on learning problem solving in an engineering physics course module. The formative feedback was built into two physics assignments with different topics but equal level of difficulty, by giving the students the maximum of four consecutive hints if the student submitted a wrong answer. The research question was formulated as: How is the student performance on a specific topic in an examination effected by receiving structured formative feedback during the practice exercise? 2 METHODOLOGY 2.1 Participants and description of course implementation The study involved students from the Finnish engineering programs in the departments of Energy Technology, Mechanical Engineering, Electrical Engineering, and Sustainability Science at the LUT School of Energy Systems. These students were enrolled in the physics course Basics of Vibration and Wave Motion. The participants were primarily first and second-year students. The students who participated in the study were asked for permission to use their results in the study. The Basics of Vibration and Wave Motion course is worth two ECTS credits and it extends for seven weeks. The course includes lectures, exercises, weekly independent online assignments, and a final exam. The weekly online assignments consist of multiple-choice questions, image interpretation tasks, and calculation problems. These assignments aim to measure and deepen students' understanding of the week's topics. The tasks are automatically graded and include features such as immediate feedback and interactive hints. 2.2 Procedure In the final week of the course, students were given two more extensive assignments, the results of which are examined in this study. The assignments were voluntary, but provided extra points, making it possible to raise an accepted grade. Students were able to complete the assignments either independently or together with peers. One assignment involved determining the Q-factor (quality factor) of a damped forced oscillator, and the other involved analysing wave interference. The topics of assignments used for the study were chosen from topics that have been challenging for students in previous years. The assignments were implemented and graded using Moodle's Numerical Cloze question. Numerical Cloze questions consist of text, possible figures or tables and an answer box to where the student enters their numerical answer. (Moodle, 2003) The students in the course were divided into two groups, A1 and A2. The number of students in Group A1 who completed the final week assignment, participated in the exam, and gave permission for the study was NA1 = 32. The corresponding number in Group A2 was NA2 = 37. Each of the two assignments (Q-factor and wave interference) was implemented in two different versions. In one version, the assignment questions were divided into multiple parts and students have multiple tries with hints (MTH) if they answered incorrectly. In the other version, there was only one try and no feedback (NF). The initial values for the assignment were different for students in different groups. In the assignment with MTH, students received immediate feedback on whether their answer was correct or incorrect. Each student had five tries per question part. If the first answer was correct, the student received full points. If not, the student received a hint. If the student answered correctly: • on the second try, they received 80% of the maximum points, • on the third try, they received 60% of the maximum points, • on the fourth try, they received 40% of the maximum points, • on the fifth try, they received 20% of the maximum points. The hints were cumulative, with the hint text getting progressively longer, more elaborated, and adding visual cues at every iteration (Fig. 1). For example, in the Q-factor assignment, the student was tasked to determine from the table given in the assignment: • the deviation caused by a static constant force on a body, and • the deviation of a body when an external force oscillates at the resonant frequency. The first hint told the student that these required values could be determined using a table. It is also explained that the amplitude reaches its maximum value when the frequency of the external force is equal to the resonant frequency. The second hint additionally told the student that the frequency of the constant force is zero and that the deviation caused by the external force oscillating at the resonant frequency is the maximum deviation value in the table. In addition to the previous hints, the third and fourth hints presented the same table given in the assignment, and highlight the rows in the table from which the required values could be determined. Group A1 students received multiple parts, MTH assignment related to determining the Q-factor in the final week of the course, as well as a one try, and NF assignment related to wave interference. Group A2 students received multiple parts, MTH assignment related to wave interference, as well as a one try, and NF assignment related to the Q-factor. Both groups first completed the multiple part, MTH assignment, followed by the one try and NF assignment. Fig. 1. Q-factor assignment first part, hints for the first part and the scoring principle. The Qfactor task has been divided into six parts. Each part similarly contains five tries with hints and the same scoring principle. The first part of the Q-factor assignment asks for answers to items a) and b), the second part to item c), the third part to item d), the fourth part to item e), the fifth part to item f), and the last part to item g). 3 RESULTS 3.1 The weekly assignment The maximum points for the final week's Q-factor and wave interference assignments were 9 points and the maximum total points for the weekly assignments in the final week was 29. Table 1 shows the mean points, standard deviation, Welch’s t-test statistic values and p-values and Cohen's d effect size of the different groups in the Q-factor assignment, wave interference assignment and total points. Table 1. The mean points, standard deviation, differences of means, Welch’s t-test statistic values and p-values and Cohen's d effect size of the different groups in the Q-factor assignment, wave interference assignment and total points. A1 A2 Diff. of means Welch’ t-test Cohen’s d Mean SD Mean SD t p Q-factor (max. 9) 6.72 2.29 3.62 2.73 |3.09| 5.13 0.000 1.22 Wave interf. (max. 9) 3.04 2.50 6.77 2.61 |3.72| -6.05 0.000 1.46 Total (max. 29) 19.03 4.62 19.78 5.36 |0.75| -0.62 0.536 0.15 It can be seen from table 1 that there is a significant difference between the mean points of groups A1 and A2 on the Q-factor and wave interference assignments but there is no difference in the mean total points for the weekly assignment. Cohen's d effect size (> 1) also indicates that the difference between groups A1 and A2 is statistically highly significant in the Q-factor and wave interference assignments. Of course, it is expected that there will be differences in the mean points of the different groups, as one group has only one try per task part, so they can only get 0 or 100% of the points and another group can get 0, 20%, 40%, 60%, 80% or 100% of the points on the task part thanks to multiple tries. Figure 2 shows the percentage of students in groups A1 and A2 who received hints in the Q-factor task and the wave interference task. For both groups, the percentages of students who received hints are almost the same, except for students who received 4 hints and those who did not respond to the task. Fig. 2. The percentage of students in groups A1 and A2 who received hints in the Q-factor assignment and the wave interference assignment. Now the most interesting thing is whether multiple tries have helped students learn and internalize the things tested in the assignment, or if it can actually hurt their learning, in case of too many tries, as observed by Yourstone et al. (2010). This was examined by testing students' knowledge with a question on the corresponding topic in the exam. 3.2 The exam The final exam of the course included questions on the corresponding topics. The final exam has 5 questions, each of which is worth a maximum of 10 points, making the maximum score for the final exam 50. Question 2 of the final exam was a question related to the Q-factor and question 5 was a question related to wave interference. Table 2 presents the mean points, standard deviation, Welch’s t-test statistic values and p-values and Cohen's d effect size obtained from the exam questions of the different groups. Table 2. The mean points, standard deviation, differences of means, Welch’s t-test statistic values and p-values and Cohen's d effect size of the different groups in the exam questions, sum of points of question 1, 3 and 4 and total points of exam. A1 A2 Diff. of means Welch’ t-test Cohen’s d Mean SD Mean SD t p Q1 8.39 1.81 8.45 1.61 |0.06| -0.13 0.894 0.03 Q2 (Q-factor) 8.77 1.99 7.49 2.81 |1.28| 2.16 0.035 0.52 Q3 8.13 1.58 7.86 1.87 |0.27| 0.63 0.532 0.15 Q4 6.48 3.08 6.69 2.98 |0.21| -0.29 0.773 0.07 Q5 (Wave interference) 6.69 2.22 7.52 1.95 |0.83| -1.59 0.117 0.40 Q1+Q3+Q4 22.99 4.74 23.00 4.65 |0.01| -0.01 0.995 0.00 Total 37.28 7.62 37.81 6.95 |0.53| -0.30 0.765 0.07 It can be seen from table 1 that there is a difference between the mean points of groups A1 and A2 on the question Q2 and Q5 but in other questions or in the total exam points, there is only very little difference between the groups' mean points. Cohen's d effect size (≥ 0.4) also shows that the difference between groups A1 and A2 is noticeable in questions 2 and 5, but it is not particularly large. In the exam question, the mean points between the different groups do not differ as significantly as in the assignment of the final week. In the exam questions, the students have returned an attachment in which they have presented the solution principle they used in the question. If the student's answer has been incorrect, but the presented calculation principle is correct, there has been a possibility of getting some points for the question. 4 DISCUSSION AND CONCLUSIONS Results indicate that our model of task-specific formative feedback operationalised as consecutive hints in a Moodle assignment enhanced students learning of engineering physics topics as indicated by the examination results. The overuse of help (Roll et al.2014) and the unnecessary increase of cognitive load (Attali & van der Kleij 2017) were avoided by providing hints only when students submitted wrong answers and discouraging the unnecessary use of hints by redacting exercise points for each hint use. Although the task-specific feedback has been criticised for encouraging surface learning, we believe that in bachelor level engineering physics course many learners can still be considered novices, whose learning benefits from this kind of feedback. We also hypothesise that these kind of consecutive hints in a complex problem can help students also to acquire appropriate process-related knowledge and provide opportunities of self-reflection. This, however, needs to be examined further in another study. One significant limitation of this study is that providing feedback in form of hints meant that the students receiving the hints also received more tries than the control group. Thus it is not possible to conclusively discern between the learning effects of a) providing formative feedback and b) providing multiple tries. Since having multiple tries before providing assistance has been suggested to support learning with children (Fyfe et al. 2012, Roll et al. 2014) it would be good to examine how much multiple tries without hints, or with knowledge of response (if correct or incorrect) feedback only, affect students' success in exam questions corresponding to weekly assignment. This could be done for example by dividing students into three groups, in which students would receive the following assignment: 1. assignment with multiple tries and hints 2. assignment with multiple tries without hints 3. assignment without multiple tries or hints The obvious implication for practice from this research is to encourage teachers to utilize the means of virtual learning environments for providing structural formative feedback. With the help of technology this can be done in a timely way and regardless of the class size. According to research by Schwerter et al. (2022), assignments that allow for multiple attempts and interactions, facilitated by technology, have been found to enhance learning. Based on our previous research (Immonen at al. 2023), students are also motivated to complete voluntary online learning assignments with multiple tries and hints even though the impact of the assignments on the final course grade is small. ACKNOWLEDGEMENTS An artificial intelligence application, Google NotebookLM, was used to manage the reference literature, but not to produce the text. The AI application, Copilot, was used for English language refinement of the final version of the manuscript. REFERENCES Attali, Y. (2015). Effects of multiple-try feedback and question type during mathematics problem solving on performance in similar problems. Computers & Education, 86, 260-267. https://doi.org/10.1016/j.compedu.2015.08.011 Attali, Y., & van der Kleij, F. (2017). Effects of feedback elaboration and feedback timing during computer-based practice in mathematics problem solving. Computers & Education, 110, 154–169. https://doi.org/10.1016/j.compedu.2017.03.012 Brazhkin, V. & Strakos, J. K. (2023). Student preferences for multiple attempts and feedback on online. Intersection: A Journal at the Intersection of Assessment and Learning, 4(2). https://doi.org/10.61669/001c.77262.