Competence Assessment: a Measurement System for the Subject “Applied Analytical Chemistry” of the Pharmacy Degree

“Competence Assessment: A Measurement System for the Subject “Applied Analytical Chemistry” of the Pharmacy Degree”, María Teresa Morales, Inmaculada Romero, Dolores Hernanz, María Ángeles Herrador, María Teresa Montaña, Guillermina Galán, María José Navas, Ana María Jiménez, María del Mar Orta, Diego Luis García-González, Julia Martín, Agustín García Asuero Volumen 30 | Número 2 | Páginas 100-113 | Abril 2019 DOI: 10.22201/fq.18708404e.2019.2.65623 100 COMPETENCE ASSESSMENT: A MEASUREMENT SYSTEM FOR THE SUBJECT “APPLIED ANALYTICAL CHEMISTRY” OF THE PHARMACY DEGREE Abstract The current university environment is characterised by the implementation of an education system focused on competence-based learning. The present study has concentrated on the subject Applied Analytical Chemistry of the Pharmacy Degree and has made it possible to clearly establish the specific competences of the subject, based on levels of acquisition, as well as to design a system of assessment, based on the use of rubrics, which provides the corresponding equivalences in the grading system currently used in the university. This new form of assessment allows the teacher to know where there are gaps in student learning, and this information can be subsequently used to prioritize the training of those skills. The implementation of these rubrics has made it possible to obtain relevant information related to the students’ acquisition of competences. Keywords: Competences; Applied Analytical Chemistry; Pharmacy Degree; Rubric EVALUACIÓN DE LAS COMPETENCIAS: UN SISTEMA DE MEDICIÓN PARA LA ASIGNATURA “QUÍMICA ANALÍTICA APLICADA” DEL GRADO EN FARMACIA Resumen El entorno universitario actual se caracteriza por la implementación de un sistema educativo centrado en el aprendizaje basado en competencias. Este trabajo se centra en la asignatura de Química Analítica Aplicada del Grado en Farmacia y trata de establecer claramente las competencias específicas de la asignatura, en función de los niveles de adquisición, así como diseñar un sistema de evaluación, basado en el uso de rúbricas, proporcionando las equivalencias correspondientes en el sistema de calificación actualmente utilizado en la Universidad. Esta nueva forma de evaluación permite al docente saber dónde hay vacíos en el aprendizaje de los estudiantes, de modo que esta información puede usarse posteriormente para priorizar el entrenamiento de esas habilidades. La implementación de estas rúbricas ha permitido obtener información relevante relacionada con la adquisición de competencias de los estudiantes. Palabras clave: Competencias; Química Analítica Aplicada; Grado en Farmacia; Rúbricas Autores: Mª Teresa Morales1, Inmaculada Romero1, Dolores Hernanz1, Mª Ángeles Herrador1, Mª Teresa Montaña1, Guillermina Galán1, Mª José Navas1, Ana Mª Jiménez1, Mª Mar Orta1, Diego L. García-González1, Julia Martín2, Agustín G. Asuero1 Investigación Educativa 1 Department of Analytical Chemistry, Faculty of Pharmacy, University of Seville. 2 Department of Analytical Chemistry, Escuela Politécnica Superior, University of Seville. “Competence Assessment: A Measurement System for the Subject “Applied Analytical Chemistry” of the Pharmacy Degree”, María Teresa Morales, Inmaculada Romero, Dolores Hernanz, María Ángeles Herrador, María Teresa Montaña, Guillermina Galán, María José Navas, Ana María Jiménez, María del Mar Orta, Diego Luis García-González, Julia Martín, Agustín García Asuero Volumen 30 | Número 2 | Páginas 100-113 | Abril 2019 DOI: 10.22201/fq.18708404e.2019.2.65623 101 COMPETENCE ASSESSMENT: A MEASUREMENT SYSTEM FOR THE SUBJECT “APPLIED ANALYTICAL CHEMISTRY” OF THE PHARMACY DEGREE Introduction In recent years, the concept of education has changed significantly due to the process of European convergence. On the one hand, this has meant that the university teachers of various countries have had to orient the curricula of the different degrees towards external similarities and, on the other hand, there has been a rethinking of the teachinglearning process towards a student-centred learning approach. This implies that the activities of the students have to be planned in relation to the competences that they are going to put into practice in real professional settings (Poloyac et al., 2011). This new method of teaching requires carrying out a new form of assessment, because if the proposed training model is competence-based it is, therefore, necessary to have competence-based assessment. This is not solely a question of techniques, but also involves a cultural change in the concept of assessment and of its purpose. At any educational level, issues related with assessment are those that cause more problems for teaching staff. University level teaching is made more difficult because, in many cases, teachers do not have specialized training in the field of education and other related subjects, especially in topics related to assessment (Hawes, 2005). A good student can overcome poor quality teaching, but will find it much more difficult to overcome poor quality assessment. For most students, to a considerable extent, the assessment determines the curriculum: its selection, its content, its system for ranking priorities and its learning approach (James et al., 2002). Curricular reforms oriented towards competence-based training include the problem of assessment, as would be expected with such a profound change from a framework based on objectives to one based on competences. Nevertheless, there is usually resistance to change, not only among teachers, but also among students for whom an assessment is only seen as such when it carries an associated grade (Hawes, 2005). An important aspect of this reform is related with the view of education as being focused both on the proposals set out for the subjects and, of equal importance, on the competency outcomes of the professional being trained. For this reason, there must be forward-looking management of the competences (Le Boterf, 2003). The core question to be addressed is: What are the most effective ways for verifying if the students have achieved the competences required by the training course or programme? Assessment responds basically to the need for constructing, nurturing and using a student-centred system of institutional knowledge, with the ulterior motive of certifying and accrediting to society the professional quality of the graduate. Certainly, one prerequisite for a system of assessment, irrespective of the type, is that it must have a degree of credibility in the community at large. For the knowledge management system to be relevant and able to activate the competences, it has to be envisaged as a process with the following clearly defined stages (Buck, 2003), see Figure 1. “Competence Assessment: A Measurement System for the Subject “Applied Analytical Chemistry” of the Pharmacy Degree”, María Teresa Morales, Inmaculada Romero, Dolores Hernanz, María Ángeles Herrador, María Teresa Montaña, Guillermina Galán, María José Navas, Ana María Jiménez, María del Mar Orta, Diego Luis García-González, Julia Martín, Agustín García Asuero Volumen 30 | Número 2 | Páginas 100-113 | Abril 2019 DOI: 10.22201/fq.18708404e.2019.2.65623 102 Analysing the usual practices that have been traditionally applied in higher education, it can be concluded that the first four stages are frequently completed, but the remaining three, which are more related to communication and sharing, are weakly structured (Hawes, 2005). The UNE standard 66173 (2003) defines competence as those “personal attributes and (the) demonstrated ability to apply knowledge and skills”; specifically indicating that it is a synonym for “the ability to resolve problems in a certain context”. Analysis of this regulation makes it possible to point out the following dimensions, which constitute the concept of competence and have to be considered in its assessment: a) Personal attributes or any attitude relating to the individual, that is, those characteristics that a person possesses from birth or has acquired by training, and that define who the person is (as against what the person does), such as: talent, motivation, communication skills, cognitive capacity, values, emotional intelligence and others, including knowledge (‘to know’) and abilities (‘to know how’); b) Demonstrated aptitudes (‘to do’) that make up the observable behaviour as a response to the stimuli of a real life environment. It is an attempt to give precedence to what a person does (as against who the person is), that is, to the students’ applied abilities and skills; and c) The demonstrated capacity to solve problems in any context; that capacity to be able to assume any foreknowledge and uncertainties derived from any environment in time (potential, available and required competences). All these dimensions can be summarized under the headings of ‘being’ (attitudes), ‘knowing’ (knowledge, approaches, theories) and ‘know-how’ (abilities). The teaching staff needs instruments that facilitate the assessment of the results obtained by the student in achieving the competences in terms of these dimensions. In addition to the assessment criteria, defined by stating and detailing the particular competence, the following are needed: a) measurement scales that analyse objectively and are able to rank the learning outcomes; b) indicators that make it possible to compare, Figure 1. Management system by stages. “Competence Assessment: A Measurement System for the Subject “Applied Analytical Chemistry” of the Pharmacy Degree”, María Teresa Morales, Inmaculada Romero, Dolores Hernanz, María Ángeles Herrador, María Teresa Montaña, Guillermina Galán, María José Navas, Ana María Jiménez, María del Mar Orta, Diego Luis García-González, Julia Martín, Agustín García Asuero Volumen 30 | Número 2 | Páginas 100-113 | Abril 2019 DOI: 10.22201/fq.18708404e.2019.2.65623 103 for each student, the real achievement for the competence with the pre-established level of knowledge and proficiency for this area of competence; and c) assessment techniques and methods that make it easier to obtain the data and information required —evidence— to be able to calculate the indicators (de la Mano González and Moro Cabero, 2009). In the current university environment, focused on competence-based learning, this study tackles one of the most complex aspects of the training process: assessment. The aim of this study is to propose a measurement system, based on the use of rubrics, which allows the teaching staff to assess the achievement of competences by the student. This study has two main objectives, to draw up the subject sheets on the basis of the competences included in the Accredited Degree Programme, and to carry out, for the first time, competence-based assessment of the subject. Achievement of these objectives would also lead to achieving other sub-objectives: • To clearly establish the specific competences of the subject on the basis of levels of acquisition. • To develop a relevant competence-based assessment system that will provide their corresponding equivalences in the grading method used in our university system. • To promote the acquisition of the specific competences of the subject, while simultaneously facilitating the acquisition of transversal competences, which allow the students to develop a series of professional competences related with the Analytical Chemistry from the earlier academic years of the Degree. Methods This study has been carried out during the first four-month period of the academic year 2010/2011 and has focused on the subject of Applied Analytical Chemistry (Basic level/ First four months) of the Pharmacy Degree that has 6 European Credit Transfer System (ECTS). This subject, taken in the 2nd year of the degree, has been taught for the first time in the academic year 2010/2011. It has been implemented in 5 groups, with a total of 340 students, and with 9 lecturing staff involved. The staff had prior experience in different activities related with the implementation of the EHEA (European Higher Education Area) and with many of the new teaching methodologies (Asuero et al., 2006a; Asuero et al., 2006b; Asuero et al., 2007a; Asuero et al., 2007b; Asuero et al., 2008; Montaña et al., 2008; Galán et al., 2009; Herrador et al., 2009; Navas et al., 2009; Morales et al., 2010; Navas et al., 2010) which promote the acquisition of various competences, both transversal and specific, by the students. As the subject in question is taught in the 2nd academic year of the Degree, and the teaching staff had already taught other subjects to the same students in their 1st academic year, the course started with the advantage that the students already knew the work methodology of the teaching staff. Most students had participated very actively in the innovation activities that had been carried out in the previous year, so that they were already receptive and participative. The competences that were developed in this study are considered to be performance indicators for the pharmacist as analyst, so that those students who progress in their proficiency will have greater possibilities of practicing their profession with the appropriate knowledge, abilities and values. Therefore, to carry out this study the teaching staff analysed the concept and the elements that constitute the competence, “Competence Assessment: A Measurement System for the Subject “Applied Analytical Chemistry” of the Pharmacy Degree”, María Teresa Morales, Inmaculada Romero, Dolores Hernanz, María Ángeles Herrador, María Teresa Montaña, Guillermina Galán, María José Navas, Ana María Jiménez, María del Mar Orta, Diego Luis García-González, Julia Martín, Agustín García Asuero Volumen 30 | Número 2 | Páginas 100-113 | Abril 2019 DOI: 10.22201/fq.18708404e.2019.2.65623 104 as well as the different stages that its process of assessment entails. The final aim of this assessment process was no longer to determine the student’s level of knowledge about specific thematic teaching blocks (Asuero et al., 2008), but rather to evaluate the student’s level of proficiency for a specific competence. When contemplating the teaching-learning process for competences, the teaching staff envisaged providing answers to a series of questions. A set of subject sheets about the competences were constructed, based on the course programme and according to the different thematic teaching blocks. They were structured following a basic pattern that makes it possible to give answers to the following questions: What to do?, How is it done?, Why is it done?. The competences associated to each one of the thematic teaching blocks were selected. As they are competences specific to the subject, the proposals that define units of competence mainly include aspects related to ‘knowledge’ and to ‘know-how’. In this context, an evaluation rubric of the 10 main competences of the subject was produced, as an instrument to facilitate the assessment of the achievement of competence. Assessment of the competences assigned to each one of the thematic teaching blocks was structured according to the level of knowledge and abilities acquired, ranked from 4 (the maximum level) to 1 (the minimum level). For this study, assessment matrices or rubrics were used in which, in a phased and hierarchical way, the different levels of proficiency for the competence by the student were established, as well as the corresponding equivalences in the grading method used by our university system. Once the rubric was created, it was implemented in the 5 course groups to assess the acquisition of competences, with a follow-up of the students throughout the fourmonth period. An opinion-satisfaction survey was also carried out. Finally, the grading was completed on the basis of acquisition of these competences. The timetable followed for carrying out the study is shown in Table 1, which details the 6 phases for implementing the study during the four-month period. September October November December January February Phase 1. Creation of subject sheets based on competences. Information to the students. Phase 2. Analysis and selection of competences for thematic teaching blocks. Phase 3. Creation of competence-based assessment matrices. Phase 4. Application of assessment matrices. Phase 5. Information about results to the students. Opinion/satisfaction survey. Phase 6. Grading based on the level of proficiency of the competences. Table 1. Timetable followed for carrying out the study “Competence Assessment: A Measurement System for the Subject “Applied Analytical Chemistry” of the Pharmacy Degree”, María Teresa Morales, Inmaculada Romero, Dolores Hernanz, María Ángeles Herrador, María Teresa Montaña, Guillermina Galán, María José Navas, Ana María Jiménez, María del Mar Orta, Diego Luis García-González, Julia Martín, Agustín García Asuero Volumen 30 | Número 2 | Páginas 100-113 | Abril 2019 DOI: 10.22201/fq.18708404e.2019.2.65623 105 Results and discussion Six subject sheets were drawn up, one for each one of the thematic teaching blocks of the subject, in which the competences to be acquired by the student are specified in a simple manner, presenting them in a way that gives answers to the questions: What to do?, How is it done?, Why is it done?. The first thematic teaching block (General Analytical Process, Sampling and Sample Preparation) includes three lectures about the general analytical process, as well as the study of sampling and sample preparation procedures. The competences corresponding to this block are shown in Table 2. What to do? How is it done? Why is it done? • To carry out all the stages of the general analytical process, from obtaining information to drawing up the final report. • To carry out the appropriate sampling and treatment of the sample. • Defining the analytical problem. • Choosing the most appropriate method. • Taking a representative sample. • Applying the optimized treatment of the sample. • To solve any analytical problem in any area, with different types of samples and several analytes, using the most appropriate analytical method. The second thematic teaching block (Chemometrics and Quality) consists of three lectures. The first one is aimed at understanding the variables that can affect the result obtained in the determination of an analyte, the second one covers the comparison procedures and chemometric tools necessary for such procedures, and the third one introduces the student to the importance of quality control in the analytical laboratory and how it is implemented. Table 3 shows the relevant competences assigned to this teaching block. What to do? How is it done? Why is it done? • Quality management of the laboratory and handling of the analytical data to obtain quality results. • Treatment of analytical data. • Handling of chemometric tools. • Quality management and the use of control cards. • To know the variables that can affect the result obtained in the determination of an analyte in the laboratory. • To be able to interpret the data obtained in the analysis. • To ensure the proper functioning of the laboratory. The third block (Analytical Methods of Separation and Measurement) includes four lectures about the analytical methods of separation. An introduction to the topic is followed by coverage of the analytical procedures used in non-chromatographic separation and, subsequently, by two lectures focused on the chromatographic methods (gases, liquids and supercritical fluids). Table 4 shows the subject sheet of competences associated to this block. Table 2. Subject sheet for the competences corresponding to the first thematic teaching block Table 3. Subject sheet for the competences corresponding to the second thematic teaching block “Competence Assessment: A Measurement System for the Subject “Applied Analytical Chemistry” of the Pharmacy Degree”, María Teresa Morales, Inmaculada Romero, Dolores Hernanz, María Ángeles Herrador, María Teresa Montaña, Guillermina Galán, María José Navas, Ana María Jiménez, María del Mar Orta, Diego Luis García-González, Julia Martín, Agustín García Asuero Volumen 30 | Número 2 | Páginas 100-113 | Abril 2019 DOI: 10.22201/fq.18708404e.2019.2.65623 106 What to do? How is it done? Why is it done? • To separate the components of the sample into different fractions. • By using nonchromatographic separation methods (solvent extraction, solid-phase extraction, supercritical fluid extraction, etc.) and chromatographic ones (planar chromatography, of gases, of liquids, etc.). • To separate the analyte from the rest of the sample. • To separate the interfering species. • To enrich the sample. • To identify and quantify analytes of a similar composition and structure. The fourth block (Quantitative and Qualitative Aspects of Chemical Analysis) consists of two lectures about the quantitative and qualitative aspects of Analytical Chemistry, which give an overall view of both aspects of the chemical analysis. Table 5 shows the relevant competences for this thematic teaching block. What to do? How is it done? Why is it done? • To detect and quantify the species in different types of samples. • To generate numerical data about the absolute and relative quantities of one or several analytes of a sample. • To generate information about the presence or absence of an analyte-species in the sample. • Using screening procedures, binary answers, assigning of false positives and false negatives. • Applying various analytical quantification methods. • Establishing the analytical properties, carrying out instrumental and methodological calibration and undertaking trace analysis using reference materials. • Evaluating the analytical methods. • To identify the analyte from its chemical and physicochemical characteristics or by its product reaction. • To determine the content of one or more analytes of a sample. The fifth thematic teaching block (Volumetric and Gravimetric Analysis) has a total of three lectures, in which are tackled the fundamentals and applications in the pharmaceutical area of volumetric methods, as well as those of gravimetric analysis. The competences assigned to this block are shown in Table 6. What to do? How is it done? Why is it done? • To apply absolute and stoichiometric methods (gravimetric and volumetric analysis) to the determination of analytes by weight and volume. • By using the analytical balance and volumetric material in the development and application of gravimetric and volumetric analytical methods. • To determine the major components in samples of pharmaceutical interest. Table 4. Subject sheet for the competences corresponding to the third thematic teaching block Table 5. Subject sheet for the competences corresponding to the fourth thematic teaching block Table 6. Subject sheet for the competences corresponding to the fifth thematic teaching block “Competence Assessment: A Measurement System for the Subject “Applied Analytical Chemistry” of the Pharmacy Degree”, María Teresa Morales, Inmaculada Romero, Dolores Hernanz, María Ángeles Herrador, María Teresa Montaña, Guillermina Galán, María José Navas, Ana María Jiménez, María del Mar Orta, Diego Luis García-González, Julia Martín, Agustín García Asuero Volumen 30 | Número 2 | Páginas 100-113 | Abril 2019 DOI: 10.22201/fq.18708404e.2019.2.65623 107 Ten lectures of the course programme are dedicated to the sixth thematic teaching block (Trace and Instrumental Analysis), which also involves a high number of classroom hours. The lectures cover electroanalytical methods, optical methods, mass spectrometry and coupled methods, kinetic methods of analysis and immunoassay techniques, all topics of recognized interest in the field of health analysis. Table 7 shows the competences associated to this thematic teaching block of the course programme. What to do? How is it done? Why is it done? • To carry out the chemical measurement process using (relative) instrumental methods that compare the signal obtained from the sample with those from analytical standards. • To apply analytical procedures of special interest in the pharmaceutical field. • Selecting the most appropriate instrumental technique for solving the problem. • Selecting the patterns for the analytical calibration. • Knowing the fundamentals and applications of the optical, electroanalytical, coupled and kinetic methods, etc. • For the qualitative, quantitative and structural determination of the major, minor and trace components of samples of pharmaceutical interest. • To determine the content of different elements, enzymes, scents, etc. in inorganic, organic and biological samples of medical-pharmaceutical interest. Once the subject sheets for the different thematic blocks had been created, the competences related to each one of these blocks were selected, taking into account the knowledge and abilities that should be acquired by the students. An assessment rubric was designed that, by considering 10 specific competences of the subject, made followup possible for the abilities, knowledge and competences acquired by the students throughout the four-month period. As they are competences specific to the subject, the proposals that define its units of competence mainly include aspects related to these two dimensions of the competence: ‘knowledge’ and ‘know-how’. The third dimension, ‘the being’ (the attitudes), was developed through the acquisition of transversal competences. The procedure (de la Mano González and Moro Cabero, 2009; Delgado et al., 2005) that the teaching staff has followed for drawing up the evaluation rubric consisted of several stages: • Analysis of the competences that are acquired from this subject, and of each one of the units of competence that define them, with the aim of identifying common elements in their formulation: the main actions (to know, to define, to apply, etc.) and the object of those actions (rules, terms, methodologies, etc.). • Drawing up of general proposals from these common elements, with the aim of creating models of units of competence that can be applied to the particular context of each one of the specific competences. • Classification of the proposals into two categories, ‘to know’, if they define actions related to the acquisition of knowledge, and ‘to know how’, if they define actions related to the application of practical abilities or skills. Table 7. Subject sheet for the competences corresponding to the sixth thematic teaching block “Competence Assessment: A Measurement System for the Subject “Applied Analytical Chemistry” of the Pharmacy Degree”, María Teresa Morales, Inmaculada Romero, Dolores Hernanz, María Ángeles Herrador, María Teresa Montaña, Guillermina Galán, María José Navas, Ana María Jiménez, María del Mar Orta, Diego Luis García-González, Julia Martín, Agustín García Asuero Volumen 30 | Número 2 | Páginas 100-113 | Abril 2019 DOI: 10.22201/fq.18708404e.2019.2.65623 108 • Design of an individual scale for each unit of competence, constituted by four proposals that identify four different levels of proficiency of the competence (Blommel, 2007): Level 1: the student does not make enough effort to acquire the competence and does not demonstrate having acquired it, or does so only rarely. Level 2: the student studies, is trained and shows that he/she sometimes applies the competence. Level 3: the student has learnt the competence, and by his/her performance demonstrates that he/she applies it. Level 4: the student has integrated the competence into his/her pattern of performance. • Integration of these scales for the units of competence into a general scale, organized around the four levels of the assessment system used in the university context of Spain (with their equivalences in the ECTS grading scale) (Table 8), which correspond to the levels of proficiency of the competence previously established (de la Mano González and Moro Cabero, 2009). The result of this process has been the creation of an evaluation rubric for assessment of the competences of the course (Table 9), which has been designed so that: it can be easily applied; it can be verified objectively; and it is clearly understood by both the evaluators of the competences (the teachers), and by those who are going to be evaluated (the students). Level of proficiency for the competence ECTS grading scale Spanish grading system 1F / FX (Fail) No Aprobado 2E (Sufficient) D (Satisfactory) Aprobado 3 C (Good) Notable 4B (Very good) A (Excellent) Sobresaliente / Matrícula de honor Table 8. Equivalences between the grading scales and the levels of proficiency for the competence