Do continuous assessment results affect final exam outcomes? Evidence from a microeconomics course

Multidisciplinary Journal for Education, http://dx.doi.org/10.4995/muse.2017.6548 Social and Technological Sciences EISSN: 2341-2593 Reboredo (2017) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 4 Nº 1 (2017): 88-101 | 88 Do continuous assessment results affect final exam outcomes? Evidence from a microeconomics course Juan Carlos Reboredo Dpto. de Fundamentos da Análise Económica. Universidade de Santiago de Compostela, Avda. Xoán XXIII, s/n, 15782 Santiago de Compostela, Spain. * Corresponding author: Email: juancarlos.rebo[email protected]; Phone: + 34 881811675 Received: 2016-01-09; Accepted: 2016-07-21 Abstract Continuous assessment aims to enhance student learning and understanding of a subject and so achieve better educational outcomes. We investigated how continuous assessment grades affected final exam grades. Using a dataset for six academic post-Bologna Process years (2009-2015) for a first-year undergraduate microeconomics course offered at a Spanish public university, we examined conditional dependence between continuous assessment and final exam grades. Our results would indicate a limited contribution of continuous assessment results to final exam results: the probability of the final exam performance improving on the continuous assessment grade was lower than the probability of the opposite occurring. A consistent exception, however, was students who obtained an A grade for continuous assessment. Our results would cast some doubt on the beneficial effects of continuous assessment advocated by the Bologna Process. Keywords Continuous assessment; final exam performance; conditional dependence Multidisciplinary Journal for Education, http://dx.doi.org/10.4995/muse.2017.6548 Social and Technological Sciences EISSN: 2341-2593 Reboredo (2017) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 4 Nº 1 (2017): 88-101 | 89 1. Introduction Implementation of the European Higher Education Area (Bologna Process) led to the introduction of continuous assessment as a way to enhance and measure student learning. Traditional lectures and final examinations have therefore been replaced by expositive and practical classes combined with student self-learning activities that are continuously assessed. A student’s final grade is comprised of marks for continuous assessment and for a final global exam. Grades therefore depend not only on the ability of the lecturer but also on the student’s own motivation, dedication and abilities. Examining the impact of continuous assessment on final student performance has several potentially important implications: for the efficient allocation of scarce resources in higher education institutions, for student time management aimed at maximizing academic performance and for the Bologna Process aim of enhancing student learning and understanding. Despite the fact that continuous assessment is a core principle underlying the Bologna Process, however, evidence on the impact of continuous assessment on final exam grades for different subjects is still lacking in most EU countries. For a microeconomics undergraduate course, we investigate the relationship between continuous assessment and final exam grades by examining the conditional dependence between these two kinds of grades. We considered four different grades (A, B, C and D) for both kinds of assessment and computed the probabilities of each final grade conditional on specific continuous assessment grades. We formulate three hypotheses regarding the positive, null or negative contribution of continuous assessment grades to final exam grades and tested the hypotheses using a likelihood ratio test for conditional dependence. We applied our modelling procedure to a sample of students enrolled in the same first-year undergraduate microeconomics course over six years (2009-2015). Our empirical evidence would indicate that continuous assessment grades made a limited contribution to final exam grades: the conditional probability of achieving a better grade in the final exam compared to continuous assessment was significantly lower than the conditional probability of getting Multidisciplinary Journal for Education, http://dx.doi.org/10.4995/muse.2017.6548 Social and Technological Sciences EISSN: 2341-2593 Reboredo (2017) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 4 Nº 1 (2017): 88-101 | 90 a lower grade. However, for students who obtained a grade A in continuous assessment, the conditional probability of obtaining an A in the final exam was no lower than the conditional probability of getting a lower grade. Overall, however, final exam grades were generally lower than the grades that would be expected on the basis of continuous assessment outcomes. Our results for the analysed microeconomics courses would suggest that continuous assessment aimed at fostering student learning and understanding may have neutral or even detrimental effects on student learning outcomes. Poorer performance in final exams is likely due to students not fully assimilating a subject, which, in turn, may be due to rational management of scarce study time aimed at maximizing the probability of a pass on the basis of continuous assessment and dedicating more time to subjects for which they may obtain better marks. Our results have implications for the efficiency of resource allocation to more personalized learning aimed at improving overall educational outcomes (e.g., the greater cost of reduced-size classes), which are highly dependent on student motivation and choices. The remainder of this article is organized as follows: Section 2 reviews relevant literature; Section 3 describes the data; Section 4 outlines our methodological approach; Section 5 presents and discusses the results; and, finally, Section 6 concludes. 2. Brief literature review Previous literature on student performance that examines the relationship between lecture attendance and exam performance and between study time and student grades reports mixed evidence for the impact of attendance and study time on academic performance. A seminal study by Romer (1993), based on attendance records for an intermediate-level macroeconomics course, reported attendance to have a positive and significant impact on academic performance. This conclusion, which has been corroborated by other empirical studies (see, e.g., Durden and Ellis, 1995; Devadoss and Foltz, 1996; Chan et al., 1997; Multidisciplinary Journal for Education, http://dx.doi.org/10.4995/muse.2017.6548 Social and Technological Sciences EISSN: 2341-2593 Reboredo (2017) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 4 Nº 1 (2017): 88-101 | 91 Rodgers, 2002; Kirby and McElroy, 2003; Dolton et al., 2003), would suggest that mandatory attendance policies enhance student performance. However, other studies have found otherwise, namely, that incentives to attendance have no impact on academic performance (Chen and Lin, 2015; Rodgers, 2002). Yet other studies have found that the positive link between attendance and performance may be explained by endogeneity problems. Krohn and O’Connor (2005), for a study of macroeconomics courses, found no relationship between attendance and grades when using instrumental variables to account for endogeneity. Similarly, Martin and Walker (2006), using fixed effect estimators to account for endogeneity, found that the positive and significant effect of class attendance disappeared when using panel data estimations instead of ordinary least squares (OLS). More recently, Andrietti (2014) also reported similar findings for a panel-data study. Another strand of the literature has examined the impact of study time on student grades. Several studies have reported no significant impact of study time on grades (Schuman et al., 1985; Hill, 1991; Rau and Durand, 2000) and Plant et al (2005) found that the amount of study by college students was a poor predictor of academic performance. In contrast, in a study that considered causal effects, Stinebrickner (2008) found that study time mattered, reporting that each additional study hour increased grades by 0.36 points. Similarly, Bonesrønning and Opstad (2012) found that increased study effort improved test grades. In contrast with the existing literature, we do not consider attendance or study time but, instead, outcomes for both these variables as reflected in continuous assessment grades. Continuous assessment provides useful information to both teachers and students, as teachers can identify the main areas of difficulty and students can pinpoint lack of sufficient knowledge, allowing an opportunity for both to rectify. Continuous assessment can therefore act as an early-warning system that alerts both teachers and students to the need to refocus their respective efforts in the light of results. On that basis, if the final exam covers all the topics reflected in the continuous assessment tests, its outcome will be no worse than the outcome for the continuous assessment. This is the spirit underlying Multidisciplinary Journal for Education, http://dx.doi.org/10.4995/muse.2017.6548 Social and Technological Sciences EISSN: 2341-2593 Reboredo (2017) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 4 Nº 1 (2017): 88-101 | 92 continuous assessment of student learning as introduced by the Bologna Process. We test for this hypothesis by considering the conditional dependence of final exam grades on continuous assessment results. 3. Data We collected data from assessments for students taking a first-year microeconomics course taught in the spring semester of a business administration undergraduate degree offered at a Spanish public university (Universidade de Santiago de Compostela). Course content included analyses of consumption and production decisions and competitive market functioning. The course was delivered in a two-hour theoretical lecture plus a 90-minute reduced-size, interactive practical session per week. Our database of 32 hours of theoretical classes and 17 hours of interactive classes contained data for six academic years (2009- 2015) subsequent to launch of the Bologna Process. Students were evaluated using scored tests and exercises, weighted in a similar way and with the same level of difficulty as the final exam (which students had to take at the end of the course) consisting of theoretical and practical content. The course was taught each year by the same teacher using the same syllabus and the same assessment procedures. Grading was as follows: A, 90% or more; B, 70%-89%; C, 50%-69%; and D, 49% or less. Table 1 reports descriptive statistics for our data referring to 589 students. The sample reflected academic performance data unevenly distributed over the sampling years. The percentages for students obtaining each grade point to a variation in final exam grade distribution with respect to continuous assessment grade distribution. Thus, largely the same proportions of students obtained an A in both kinds of assessments (except for 2009- 2010); the share of students who obtained a B-grade for continuous assessment and who maintained this grade in the final exam, however, was greatly reduced; and the share of C- and D-grade students receiving the same grades in the final exam increased. In other words, Multidisciplinary Journal for Education, http://dx.doi.org/10.4995/muse.2017.6548 Social and Technological Sciences EISSN: 2341-2593 Reboredo (2017) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 4 Nº 1 (2017): 88-101 | 93 the probability distribution for continuous assessment grades moved left: the probability of an A remained similar, the probability of a B dropped and the probability of a C or D rose. This descriptive evidence would indicate that continuous assessment yields confusing evidence of future student performance: the hard-working and high-performing students (A grade) performed consistently, obtaining the same top grades in both assessments; students with reasonably good continuous assessment results (B grade) would seem to have slackened off their efforts and, consequently, obtained poorer final exam grades; and students with poor continuous assessment results (C and D grades) would seem to have invested lower effort and, consequently, do not improved their final exam grades. The descriptive statistics in Table 1 also show that student performance worsened with class size: the larger the class, the greater the leftward movement of the final exam grade distribution with respect to the continuous assessment grade distribution (see, for instance, results for the 46 students of 2014-2015 compared to results for the 194 students of 2010- 2011). Table 1. Descriptive statistics. Continuous assessment grade vs final exam grade. Continuous assessment grade (% students) Final exam grade (% students) Year # Students A B C D A B C D 2009-2010 60 13.5 47.5 25.4 13.6 6.8 16.9 52.6 23.7 2010-2011 194 0.5 17.1 46.6 35.8 2.6 11.4 30.6 55.4 2011-2012 80 3.7 7.5 30.1 58.7 10 22.5 28.7 38.8 2012-2013 139 2.9 4.4 47.8 44.9 0.7 3.6 52.9 42.8 2013-2014 70 5.7 34.3 22.9 37.1 5.7 12.9 14.3 67.1 2014-2015 46 13.0 39.2 13.0 34.8 10.9 13.0 34.8 41.3 Multidisciplinary Journal for Education, http://dx.doi.org/10.4995/muse.2017.6548 Social and Technological Sciences EISSN: 2341-2593 Reboredo (2017) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 4 Nº 1 (2017): 88-101 | 94 4. Modelling conditional dependence We modelled continuous assessment and final exam grades to test for conditional dependence between grades as follows. Consider indicators for the grades that student i can obtain for the final exam (fe) and for continuous assessment (ca), fe i I and ca i I , respectively, as taking the values j=A, B, C or D. Conditional dependence of the final exam grade on the continuous assessment grade for each student is given by   fe ca ii Pr I h | I j , for h,j=A, B, C, D, and conditional dependence implies that     fe ca fe i i i Pr I h | I j Pr I h    . Conditional dependence between grades can thus be represented by the conditional dependence matrix P: AA AB AC AD BA BB BC BD CA CB CC CD DA DB DC DD P P P P P P P P PP P P P P P P P        , (1) where each element, given by   fe ca jh i i P Pr I h | I j   for h,j=A, B, C, D, reports the probability of a grade h in the final exam provided the continuous assessment grade was j. Conditional probability estimates are obtained by maximizing the maximum likelihood function under conditional dependence, given by:   jh n jh j,h PP , (2) where jh n is the number of students with continuous assessment grade ca i Ij followed by final exam grade fe i Ih . Estimated parameters arising from Eq. (2) are simply the ratio of the counts for the corresponding cells: jh jh jA jB jC jD n ˆ Pn n n n    , (3) Multidisciplinary Journal for Education, http://dx.doi.org/10.4995/muse.2017.6548 Social and Technological Sciences EISSN: 2341-2593 Reboredo (2017) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 4 Nº 1 (2017): 88-101 | 95 In light of the independence tests proposed by Christoffersen (1998) and Reboredo (2014) for indication sequences, hypotheses regarding the impact of the continuous assessment grade on final exam performance can be formulated as specific restrictions on the conditional probabilities inside matrix P in Eq. (1). These restrictions can be tested using standard likelihood ratios tests. We consider three hypotheses. First, we consider the hypothesis that final exam grade is independent of the continuous assessment grade (H1), formulated as:  Hypothesis 1: 0 jh h H : P P The conditional and unconditional probability of a specific grade h in the final exam are thus similar. Note that this hypothesis states that the conditional probabilities in the hcolumn of the matrix P in Eq. (1) are equal. Taking the likelihood function under the null hypothesis 1, we can obtain the likelihood ratio test as: jh hn n h jh hj 2log P P       LR . (4) A second hypothesis consists of testing whether the final exam grade is better than the continuous assessment grade. In this case, the conditional probability of getting a better final exam grade is greater than the conditional probability of obtaining the same or a lower continuous assessment grade. This hypothesis (H2) can be formulated as:  Hypothesis 2: 0 jh jz H : P P , where h is a better grade than j and z is a poorer grade than h. Using the likelihood function we can estimate the likelihood ratio for H2 as: jh jh nn jh jh h z h 2log P P       LR . (5) Multidisciplinary Journal for Education, http://dx.doi.org/10.4995/muse.2017.6548 Social and Technological Sciences EISSN: 2341-2593 Reboredo (2017) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 4 Nº 1 (2017): 88-101 | 96 Finally, the third hypothesis is that continuous assessment and final exam grades are the same. Thus, the conditional probability of obtaining the same grade in the continuous assessment and final exam is greater than the conditional probability of obtaining a different grade. Accordingly, each conditional probability in the diagonal of matrix P will be greater than the conditional probabilities in the same row. This hypothesis (H3) can be formulated as:  Hypothesis 3: 0 hh hz H : P P , h z Using the likelihood function under null H3, we obtain the likelihood ratio as: hh hz nn hz hz 2log P P      LR . (6) 5. Results Table 2 reports estimates for the conditional probability matrix P in Eq. (1) considering the full sample of student grades. The empirical results show that conditional dependence between continuous assessment and final exam performance varied according to grade (A, B, C or D). They also reveal substantial differences in conditional transition probabilities for the different grades. The implications for the effectiveness of continuous assessment in terms of final exam performance merit investigation, as, broadly speaking (and excluding the A grades), the conditional probabilities of getting a better final exam grade than continuous assessment grade were notably lower than the conditional probabilities of gaining a similar or lower grade. Table 2. Conditional probabilities. 0.39 0.27 0.30 0.04 0.09 0.30 0.49 0.12 ˆ P0.04 0.06 0.45 0.45 0.01 0.05 0.22 0.72       