Validity of academic work indicators in the projected European Higher Education Area

Running head: Validity of indicators in EHEA Validity of academic work indicators in the projected European Higher Education Area Rafael J. Martínez and Rafael Moreno University of Seville Address for correspondence: Department of Experimental Psychology. Calle Camilo José Cela s/n. 41018 Seville (Spain). Phone number: 34 -954551685; Fax: 34 - 954551784. Electronic correspondence: [email protected]; [email protected] The final publication is available at Springer via http://dx.doi.org/10.1007/s10734-005-4507-4 2 Validity of academic work indicators in the projected European Higher Education Area Abstract The competencies achieved, the quantity and distribution of the time employed, and the activities carried out by the student are fundamental elements of the future European Higher Education Area. The present study explores, in a specific course, the current level of some indicators of such elements and their validity. The results highlight the irrelevance of the temporal aspects with respect to final performance, and the relevance of the academic work activities to performance over the length of the course. Each indicator is discussed, and it is intended to further those which result to be relevant for the success of the student in the future, and to limit those temporal elements to a mere function of organization of the universities. Key words: competencies, credit-transfer, EHEA, higher education, indicators 3 Validity of academic work indicators in the projected European Higher Education Area After the successive accords of the Governments and Universities of Europe (the Bologna, Prague and Berlin declarations), the new European Higher Education Area (EHEA) will be initiated in the near future. This will include the European CreditTransfer System, ECTS, as a fundamental element which will permit the bestowing and standardization of each degree. The ECTS credits, designed by the European Commission (1998) within the framework of the Erasmus programme, “represents, in the form of a numerical value assigned to each unit of the course, the volume of work required of the student to pass each one of those units” (p. 3; European Commission, 31/3/98). The project defines the weight of academic work of each subject in function of the average time required by the student to successfully complete it, distributing this weight homogenously over the weeks of the course. The work of the student is generally understood to be autonomous and sustained by the philosophy of “learning to learn”; thus, the more traditional systems, based on explanations by the professor which must be reproduced later, must be largely replaced by a system centred in the activities of the student. Via those activities, and the orientation and supervision of the teaching staff, the student should acquire the competencies associated with each subject. Although defined with still insufficient precision and consensus, the notion of competency makes reference to each set of knowledge and abilities which permit the cognitive and behavioural skills considered effective by determined criteria of evaluation (McNamara 1992; Varela and Ribes 2002). Definitively, the time employed in work and its distribution over the course, the activities carried out in that time and the competencies achieved are fundamental elements of the future system. The 4 competencies constitute the objective or goal to be achieved, while the time employed, its distribution, and the academic activities carried out are thought of as antecedents or previous conditions to that achievement. At the present, close to changing over to the new system, it would be appropriate to explore those above mentioned fundamental elements, and use the knowledge gained to inform that new system. By not proceeding in this way, we would be confronted with reform insufficiently supported by data; such has been the case, to a certain extent, until now. For example, while there exist numerous analyses and theoretical discussions about the possibilities, difficulties and implications of the EHEA project (e.g. Amaral and Magalhaes 2004; Koenig 2001; Iza and Encina 2004; Neave 2003; Van Damme 2001), a search done in September of 2004 in the ISI Current Contents Connect database since 1999 found one single empirical study – and only loosely related – about the elements of the future EHEA considered here (Larsen 2000). The present study has two objectives. First, in a context limited to a single course, to probe the current level of the elements suggested by the future system: The time dedicated by the students to work, the distribution of this time over the weeks of the course, the activities carried out and the competencies achieved. Second, to explore which of those elements present during the course are related to the final results, such as the completion and performance of the final examination. It is intended to learn which elements should receive more attention during the course to optimize the final results. Both objectives are approached from the context of a pilot project for the installation of the ECTS, applied to psychology degrees in the public universities of the Autonomous Community of Andalusia, in Spain; concretely, the study was realized in a single course which, in the two previous academic years, had been developed with a methodology centred in the student, as in the future EHEA (Moreno et al., 2004), and 5 therefore constitutes an adequate testing space for the future generalization of the projected system. Method Subjects The initial population was comprised of 427 students enrolled in diverse groups of the course entitled Fundamental Methodology in 2003-2004, which took place in the first semester of the Bachelor’s degree program in psychology at the University of Seville, and which has a weight of 5.8 ECTS equivalent to 145 hours of work of the student, of which 60 must be carried out in class and the other 85 out of class. At the midway point of the semester, 39 of the students, who had failed the course in the previous year, passed the course by successfully completing a special examination. As a result, the number of students at the final examination was reduced to 388. Material A survey was developed using open and closed questions about the time employed in schoolwork and its distribution over the course. The students were offered 10 series of automatic and controlled self-evaluation exercises implemented on the program Hot Potatoes® and offered on the course website (http://www.us.es/afunmet). It was also made possible to complete three tests over the semester, having a format similar to the final examination, as practice and as feedback on the goals reached in a determined part of the syllabus. Those tests were comprised of 16, 18 and 12 multiple choice items (Livingstone reliability coefficients K2XV = 0.39, 0.64 and 0.54, respectively). For its part, the final examination, which serves to grant accreditation and 6 to establish the level obtained in the course, was composed of 30 multiple-choice items dependent on a context (K2XV =0,63) which evaluated practical competencies implied in the methodological analysis of two summaries of psychological research or practice. Procedure The temporal elements were defined by the following indicators supplied by each student: The number of hours of work dedicated to the course out of class, the number of hours estimated to be necessary to pass the course, and the distribution of the time employed, as defined by distributions that were irregular, homogenous, incremental or other. The indicators of behaviour or work activities of each student were specified as the level of attendance in class, the number of exercises completed of those offered on the course website, the number of practice tests completed and the completion of the final examination. Also, for each student, the following competency indicators were considered: The number of practice tests passed, the number of correct responses on the practice tests, whether or not they met – once the randomization correction was applied – the established performance criteria to achieve accreditation on the final examination, and the level achieved on the examination, in terms of correct responses. The temporal and behavioural indicators were collected mid-semester using an unannounced survey given to the students in the classroom at the time. The number of exercises completed on the course website was obtained by another survey given immediately following the final examination. The remaining indicators were obtained using direct data from the tests and examinations, supplied by the teaching staff. To obtain the correlations planted in the second objective, the students were asked to indicate their national identity number on the tests, the final examination and on the 7 survey. As this may have impeded the desired anonymity on the survey, the professors formally expressed the promise to use that information solely to the specified ends, respecting the privacy of personal information. Results The descriptive study of the temporal indicators shows that the mean amount of time spent working outside of class per week, as reported by the students, was 3.55 hours (SD = 1.84; n = 236) with a minimum of 1 hour spent and a maximum of 12, while the average time deemed necessary was 5.36 hours (SD = 2.60). The difference indicates that the students worked an average of 1.8 hours less than that which they considered necessary to pass (SD = 1.81), oscillating between those who dedicated five hours more than what was required to those who dedicated up to nine hours less. Concretely, 52.8% declared a deficit of between zero and two hours with respect to what they considered necessary, while for 25.1% this difference was greater than two hours. With respect to the distribution of the time employed, until at least the time of the survey, the majority indicated an irregular distribution (44.5%), followed by a large group who indicated that the distribution increased incrementally over the semester (32.2%), and a third group which considered the distribution of hours worked to be homogenous (19.9%); a residual percentage (3.4%) indicated other distributions, notably a decreasing distribution over the semester. With respect to the behavioural or activity indicators, the mean class attendance, until the halfway point of the course was 4.68 (SD = 0.70; n = 235) on a scale of 1 to 5; 78.3% of the students responded that they had nearly perfect attendance. They also responded to have completed and average of 3.79 exercises (SD = 3.49) of the total 10 available on the course website, with nearly 50% of the students not having completed 8 any. The number of students completing the practice tests was 252 (59.0% of those enrolled) on the first test, 227 (58.5%) on the second test, and 168 (43.3%) on the third. For the last two tests, the students who had passed the course by the extraordinary examination were not included in the total. Finally, 299 students (77.1% of the total enrolled students) completed the final examination. Regarding the competency indicators, the performance criteria established for the practice tests was passed by 132 students on the first test (30.9% of the total enrolled students at the time), 186 on the second (47.9%) and 107 on the third (27.6%). The mean of correct responses on all tests was 20.68 (SD = 7.99) with a range from 4 to 40. On the final examination, 45.1% of the students passed the minimum criteria required to obtain the credits assigned for the course. The mean of correct responses on the examination was 19.19 on a scale of 0 to 30 (SD = 3.98), with a range of 9 to 30 correct responses and a normal distribution (Dmax = 0.074; Z = 1.274; p =0.078; asymptotic bilateral). Table 1 shows the correlations and contingency coefficients required for each type of data, to describe the relations between the indicators during and at the end of the semester. It can be seen that of the temporal indicators, only the time dedicated to study correlated significantly and positively with another indicator at the end of the course – specifically, with the completion of the final examination. Inversely, all the behavioural indicators correlated significantly and positively with the indicators from the final examination, with the exception of the number of web-based exercises completed, whose correlation could not be calculated, as that data was obtained only from those students who had completed the final examination. Of these correlations, the size of the correlation is considerable between the number of practice tests completed by each student and the completion of the final examination, and less for the other correlations. 9 For their part, the competency indicators also significantly corresponded with the indicators associated with the final examination, occurring to a greater degree with the completion of the final examination and with the number of correct responses on the final examination. It is also noteworthy that the competency indicators at the midway point of the course presented greater correlations with the final examination indicators than with the behavioural indicators, with the exception of the number of practice tests completed. INSERT TABLE 1 Conclusions Some of the indicators in the sample studied present levels which would be considered insufficient once the future EHEA is inaugurated. A vast majority of students recognized a deficit in hours worked – especially noteworthy considering that the hours considered necessary by the students in this study was practically equal to the 4.72 hours per week estimated in the current EHEA projects for a course with the same number of credits as the one studied here. Additionally, 80% of the students claimed to distribute their time working in a style different to the homogenous style intended for the EHEA. Also, the number of exercises completed on the website is insufficient, fewer than 2 of every 5 exercises, with nearly 50% of students not having completed any. Likewise, only slightly more than one third passed the practice tests before the final examination. Other indicators show high levels, such as class attendance and the number of students completing the final examination. The remaining indicators showed intermediate values.