Investigating The Relationship Between Mathematics and Retention and Success in Engineering Study: A South African Case Study

Research Paper Recommended citation: Prince, R. N., & Simpson, Z. (2025). Investigating The Relationship Between Mathematics and Retention and Success in Engineering Study: A South African Case Study. In Kangaslampi, R., Langie, G., Järvinen, H.-M., & Nagy, B. (Eds.), SEFI 53rd Annual Conference. European Society for Engineering Education (SEFI), Tampere, Finland. DOI: 10.5281/zenodo.17631228. This Conference Paper is brought to you for open access by the 53rd Annual Conference of the European Society for Engineering Education (SEFI) at Tampere University in Tampere, Finland. This work is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 4.0 International License. INVESTIGATING HOW MATHEMATICS SKILLS UPON ENTERING ENGINEERING STUDIES AFFECT GRADUATION AND DROPOUT RATES: A SOUTH AFRICAN CASE STUDY RN Prince a,1, ZS Simpson b a University of Johannesburg, Johannesburg, South Africa, 0000-0002-8640-799X b University of Johannesburg, Johannesburg, South Africa, 0000-0002-1263-3812 Conference Key Areas: Teaching mathematics and physics in engineering education Keywords: student success; mathematics; retention and throughput; ABSTRACT While many students are underprepared for higher education, especially in mathematics-heavy disciplines such as engineering, universities are also underprepared for the students they admit. This has resulted in low graduation and high dropout rates in engineering programs globally. In the USA, only around 30% of students who enter an engineering qualification graduate in the expected time and fewer than 60% graduate within regulation time plus two years. To this end, we seek to answer the question: “How does students’ mathematics ability upon entry into university affect graduation and dropout rates in engineering study?” We determine mathematics ability using two assessments employed in the South African context – one a statutory school-leaving assessment and the other an optional higher education admission assessment. Academic performance (graduation and dropout) in engineering study is assessed at the end of first year and again at the end of four and six years, for the same cohort of students. A sample of 525 students is selected, amongst which 23.2% dropped out at the end of first year and 30.9% dropped out at the end of four years. While only 26.5% graduated in regulation time (4 years), 60.4% graduated in regulation time plus two years (6 years). With regard to whether mathematics ability offers insight into this poor performance, we find a strong correlation between both mathematics assessments and performance in engineering study. This information can be used to inform the ways in which curriculum 1 Corresponding Author RN Prince [email protected] development and teaching and learning should be responsive to students admitted into engineering programs of study, both within South Africa and internationally. 1 INTRODUCTION AND CURRENT KNOWLEDGE Mathematics serves as the foundational language of STEM (Science, Technology, Engineering, and Mathematics) education, enabling students to engage in scientific inquiry, technological development, and engineering design. Proficiency in mathematics equips learners with problem-solving, modelling, and analytical skills essential for tackling real-world challenges. Despite its central role, research indicates that mathematics is often underemphasized in STEM curricula, pointing to the need for stronger integration and contextualization of learning experiences (Simpson & Prince, 2018). Several studies report on the relationships between numerous predictive factors and the performance and retention of engineering students (Tsui & Khan, 2023; de Winter & Dodou, 2011). However, they do not investigate how national school-leaving and university admission mathematics assessments, affects engineering graduation and dropout rates. Mathematics proficiency plays a critical role in engineering education, specifically, influencing students' academic success, confidence, and overall performance in engineering programmes. A strong foundation in mathematics enhances students' ability to grasp complex engineering concepts, while inadequate mathematical preparedness often leads to challenges in coursework. For example, insufficient mathematical skills may result in faculty dedicating additional time to revisiting fundamental topics, such as trigonometry and geometry, rather than focusing on core engineering principles (Grayson et al., 2018). Longitudinal studies further highlight a positive correlation between mathematics achievement and performance in engineering subjects like mechanics, underscoring the necessity of strong mathematical competencies for success in applied engineering disciplines (Bischof et al., 2015). Mathematics is a significant factor in engineering student success, with students who have stronger mathematical backgrounds and better performance in mathematics more likely to succeed and complete their engineering degrees (Tsui & Khan, 2023). Beyond cognitive ability, affective factors—such as attitudes, beliefs, and selfefficacy in mathematics—also influence student performance. Research on first-year engineering students suggests that a positive mathematical mindset is linked to improved academic outcomes, emphasizing the importance of fostering confidence and motivation in mathematics education (Panaoura et al., 2024). In response to these challenges, scholars advocate for innovative teaching strategies that promote active learning and real-world applications of mathematical concepts, helping students bridge the gap between theoretical knowledge and practical problemsolving (Pepin et al., 2021). In the South African context, concerns about student preparedness in engineering study persist, necessitating a deeper understanding of students’ mathematical capabilities upon entry (Scott et al., 2007; CHE, 2013). The South African National Curriculum and Assessment Policy Statement (CAPS) defines mathematics as a symbolic and notational language used to describe numerical, geometric, and graphical relationships, emphasizing its role in critical thinking, problem-solving, and access to advanced mathematical study (DBE, 2011). Similarly, the National Benchmark Tests’ (NBT) Mathematics assessment evaluates students’ ability to apply mathematical concepts from the school curriculum in higher education contexts, particularly in disciplines such as Mathematics, Chemistry, Physics, and Engineering (Bohlmann & Braun, 2006; Bohlmann et al., 2017). Given the critical role of mathematics in engineering education, this paper examines the predictive validity of two mathematics assessments for engineering programmes in South Africa, exploring their effectiveness in determining students’ likelihood of academic success and progression. 2 METHODOLOGY In South Africa, two assessments of school-leavers' mathematical proficiency are available. The first is the National Senior Certificate (NSC) assessments; these are norm-referenced, meaning scores are ‘standardised’ or ‘normed’ to a five-year rolling average. Thus, while candidates may perform well compared to the norm, they may still fail to meet a certain standard in the subject being tested. In the NSC, students are rated on the achievement scale given in Table 1. Students wishing to study engineering at university must write the NSC Mathematics assessment and obtain a rating of 5 or above. Table 1. National Senior Certificate (NSC) scale of achievement (Grades 10-12) (DBE 2009, p. 5) Rating code Description Score (%) 7 Outstanding achievement 80-100 6 Meritorious achievement 70-79 5 Substantial achievement 60-69 4 Adequate achievement 50-59 3 Moderate achievement 40-49 2 Elementary achievement 30-39 1 Not achieved 0-29 The South African National Benchmark Test (NBT) in Mathematics (MAT) determines whether school-leavers are able to transfer their school-leaving Mathematics knowledge to contexts that are found in South African higher education. The NBT MAT test aims to assess candidates’ ability with respect to various mathematical topics: problem solving and modelling, use of algebraic processes and functions; basic trigonometry; spatial perception; analytic and circle geometry; data handling and probability; and logical skills (CEA, 2024). The NBT Mathematics (MAT) assessment employs select-response items as mapped onto the respective MAT assessment specification tables. Responses are scored either as right or wrong, and the unidimensional three-parameter (a, b, c) Item Response Theory (IRT) model, where a = item discrimination, b = item difficulty, and c = item guessing/pseudochance are used to score the performance of the examinee (Yen & Fitzpatrick 2006, p. 114). Performance on the NBTs is captured as a score out of 100, which is further classified according to three benchmark levels: proficient, intermediate, and basic. These benchmarks are set through a Modified Angoff Standard Setting process (Hambleton and Pitoniak 2006). Because a large number of test-takers fall into the intermediate band, it has been found useful to use the arithmetic mean to split this band into two: upper intermediate and lower intermediate. Table 2 shows how these benchmarks are generally interpreted. Table 2. NBT MATHEMATICS Benchmark Levels (extracted from Prince 2016, p. 26) Proficient Performance suggests that academic performance will not be adversely affected, and student should be able to cope with the demands of regular programmes of study. Intermediate Upper Challenges exist such that academic progress will be affected. Students are likely to need complementary support (additional tutorials, workshops, augmented courses, language intensive work). Intermediate Lower Challenges exist such that academic progress will be affected. Students need to be placed in an extended degree programme. Basic Serious challenges exist, and students are unlikely to cope with university study. In this paper, we investigate the relationship between students’ achievement on the NSC and NBT MAT assessments, on the one hand, and their academic standing and performance in engineering study, on the other. Academic standing is measured at the end of first year, where a student has either dropped out (DROP) or is still busy studying (BUSY), and the end of four and six years, where a student may also have graduated (GRAD). One-way analysis of variance (ANOVA) was used to determine the relationship between academic standing (after one, four and six years) and performance on the NSC and NBT Mathematics assessments. Stack bar charts and Chi-squared tests of independence were used to determine the relationship between academic standing (after one, four, five and six years) and NSC and NBT Mathematics levels. Analysis is undertaken on one cohort of engineering students over a period of six years. This cohort consists of 525 first-time entering students in an engineering faculty at a higher education institution in South Africa. 3 RESULTS Of the 525 students, 437 (83.2%) achieved the NSC Mathematics with scores that were rated at Level 7 (80 – 100%). In contrast, on the NBT MAT, only 18.1% of the incoming students achieved ‘proficient’ NBT MAT scores. The majority, 81.9%, had scores in the Intermediate and Basic bands. Of more concern, 37.7% had scores in the Lower Intermediate and Basic proficiency bands. Table 3. Performance on the NSC MATHEMATICS Test NSC Mathematics Level N = 525 5 11 (2.1) 6 77 (14.7) 7 437 (83.2) Table 4. Performance on the NBT MATHEMATICS Test Mathematics Proficiency N = 525 Basic 13 (2.5) Intermediate Lower 185 (35.2) Intermediate Upper 232 (44.2) Proficient 95 (18.1) As can be seen in Table 5, nearly a quarter (23.2%) of the cohort dropped out at the end of first year and just more than a quarter (26.5%) of the cohort graduated in the regulation time of four years, by which time drop out had increased to 30.9%. At the end of regulation time plus one year, 48.8% of the cohort had graduated while at the end of regulation time plus two years, 60.4% of the cohort had graduated. At the end of regulation time plus two years four students who had previously dropped out had returned and were either still busy or had graduated. Table 5. Academic standing at the end of first year (AS1), regulation time (ASn – four years), regulation time plus one year (ASn1 – five years) and regulation time plus two years (ASn2 – six years) Characteristic AS1 N = 525 ASn N = 525 ASn1 N = 525 ASn2 N = 525 GRAD 0 (0.0) 139 (26.5) 256 (48.8) 317 (60.4) BUSY 403 (76.8) 224 (42.7) 109 (20.8) 50 (9.5) DROP 122 (23.2) 162 (30.9) 160 (30.5) 158 (30.1) One-way ANOVA was done for the NSC Mathematics scores (Fig. 1a) as well as for the NBT Mathematics scores (Fig. 1b). This analysis reveals that there is a significant difference in mean values [F(2, 522) = 55.13, p = 0.000] of the NSC Mathematics assessment scores across the three academic standing groups after four years. Fig. 1b shows the results of a Tukey HSD test for difference between the means. There was a statistically significant difference between all pairs of groups. This can be seen in the fact that all 95% confidence intervals do not contain the value zero, indicating that the differences are significant. Similar results were obtained for the NBT Mathematics assessment scores [F(2, 522) = 52.3, p = 0.000] across the three academic standing groups after four years. At the end of six years, one-way ANOVA was done for the NSC Mathematics scores (Figure 2a) and for the NBT Mathematics scores (Figure 2b). This analysis reveals that there is a significant difference in mean values [F(2, 522) = 35.28, p = 0.000] of the NSC Mathematics assessment scores across the three academic standing groups after six years. Figure 2a shows the results of a Tukey HSD test for difference between the means. There was a statistically significant difference between two of the three pairs of groups. This can be seen in the fact that all but the difference between BUSY and GRAD had 95% confidence intervals that do not contain the value zero, indicating that the differences are significant. Similar results were obtained for the NBT Mathematics assessment scores [F(2, 522) = 30.32, p = 0.000] of the NBT Mathematics assessment scores across the three academic standing groups after six years. However, in this instance, the difference between DROP and BUSY was not statistically significant. This can be seen in the fact that the 95% confidence interval contains the value zero, indicating that the differences are not significant. Fig. 1a. Differences between mean NSC Mathematics scores for Academic Standing categories after four years (with 95% confidence intervals) Fig. 1b. Differences between mean NBT Mathematics scores for Academic Standing categories after four years (with 95% confidence intervals) Fig. 2a. Differences between mean NSC Mathematics scores for Academic Standing categories after six years (with 95% confidence intervals) Fig. 2b. Differences between mean NBT Mathematics scores for Academic Standing categories after six years (with 95% confidence intervals) Figs. 3a and 3b are stack bar charts of NSC Mathematics levels and NBT Mathematics proficiency levels versus Academic Standing at the end of regulation time (four years). In Fig. 3a, it can be seen that Graduation increased from level 5 to level 7 while Dropout decreased over these levels. In Fig. 3b, it can be seen that graduation increased from lower intermediate level to proficient (basic is an exception, due to the low numbers in this category), while dropout decreased as NBT Mathematics performance increased from lower intermediate to proficient. Figs. 4a and 4b are stack bar charts of NSC Mathematics levels and NBT Mathematics proficiency levels versus Academic Standing at the end of regulation time plus two years (six years). In Fig. 4a, it can be seen that graduation increased from level 5 to level 7 while dropout decreased across these levels. In Fig. 4b, it can be seen that graduation increased from basic level to proficient while dropout decreased as performance on the NBT MAT increased from basic to proficient levels. Fig. 3a. Academic Standing categories by NSC Mathematics proficiency at the end of four years (X-squared = 40.202, df = 4, pvalue = 3.932e-08) Fig. 3b. Academic Standing categories by NBT Mathematics proficiency bands at the end of four years (X-squared = 80.688, df = 6, p-value < 2.576e-15) Fig. 4a. Academic Standing categories by NSC Mathematics proficiency bands at the end of six years (X-squared = 38.646, df = 4, p-value = 8.243e-08) Fig. 4b. Academic Standing categories by NBT Mathematics proficiency bands at the end of six years (X-squared = 47.615, df = 6, p-value = 1.411e-08) 4 DISCUSSION AND CONCLUSIONS The findings indicate a strong correlation between students' mathematics proficiency and their likelihood of persistence and graduation in engineering programmes. Students with an NSC Mathematics level 5 face a high risk of dropping out, with probabilities of 82%. Similarly, those at the NBT Mathematics Basic level have a dropout probability of 46.2%. In contrast, students with an NSC Mathematics level 7 demonstrate a 29.7% likelihood of graduating in regulation time and a 65% chance within the extended period of six years, while those with a Proficient NBT Mathematics level exhibit significantly higher success rates—60% within regulation time and 87.4% within the extended time. These results underscore the critical role of mathematical competency in engineering education, emphasizing both cognitive and affective factors. To enhance student success, institutions should be responsive to the mathematical needs of students identified through the NSC and NBT Mathematics assessments. Tsui and Khan (2023) highlights the need for better integration of mathematics teaching within engineering curricula to support student retention and understanding. The information in Table 2 also suggests that institutions need to implement innovative teaching strategies such as tutorials and workshops that integrate mathematical principles with engineering applications, fostering both conceptual understanding and practical problem-solving skills. REFERENCES Bischof, G., Zwölfer, A., & Rubeša, D. (2015). Correlation between engineering students’ performance in mathematics and academic success. 122nd ASEE Annual conference and exposition. American Society for Engineering Education, Washington D.C. Bohlmann, C. and Braun, M. (2006). Cognitive Academic Mathematics Proficiency, in Griesel, H. (ed) Access and Entry Level Benchmarks, the National Benchmark Tests Project, 35-41. Available at: http://www.hesa.org.za/sites/hesa.org.za/files/2006_HESA_Access and Entry Level Benchmarks.pdf (Accessed 24 February 2015). Bohlmann, C., Prince, R. and Deacon, A. (2017). Mathematical Errors Made by High Performing Candidates Writing the National Benchmark Tests. Pythagoras. 38. pp. 1 – 10. Centre for Educational Assessments (CEA). (2024). The National Benchmark Tests National Report: 2024 Intake Cycle Data Report. Cape Town: Centre for Educational Assessments. Available at: https://www.nbt.ac.za/content/national-report-2024 (Accessed 24 February 2025). CHE. (2013). A proposal for undergraduate curriculum reform in South Africa: The case for a flexible curriculum structure. Available at: http://www.che.ac.za/sites/default/files/publications/Full_Report.pdf (Accessed 21 May 2025). de Winter, J. C. F., & Dodou, D. (2011). Predicting academic performance in engineering using high school exam scores. International Journal of Engineering Education, 27(6), 1343–1351 Department of Basic Education. (2009). National examinations and assessment. Report on the National Senior Certificate examination results 2009. Pretoria: DBE. Available at: http://www.education.gov.za/LinkClick.aspx?fileticket=l3hlVk9sypk%3d&tabid=92&po rtalid=0&mid=4359&forcedownload=true. (Accessed 28 February 2025). DBE. (2011). National Curriculum Statement (NCS) Curriculum and Assessment Policy Statement: Grades 10‒12 (General): Mathematics. Pretoria: Department of Basic Education. Grayson, J. M., Wood, T. A., Robinson, R. J. M., Plumblee, J. M., & Prince-Nelson, S. (2018). The Influence of Mathematical Preparedness on Student Performance in an Engineering Statics Course. ASEE Southeastern Section Conference.