Statistical analysis and use of questionnaries for evaluation of the knouledge at university. A practical case

STATISTICAL ANALYSIS AND USE OF QUESTIONNAIRES FOR EVALUATION OF THE KNOWLEDGE AT UNIVERSITY. A PRACTICAL CASE. José Manuel Gómez Soberón Ph.D. in Civil Engineering. Full-time professor in the Department of Architectural II. Technical University of Catalonia. Spain. [email protected] Montserrat Bosch González Humanities Degree. Full-time professor in the Department of Architectural II. Technical University of Catalonia. Spain. [email protected] Oriol Marín i Gordi Technical Architect. Associate professor in the Department of Architectural II. Technical University of Catalonia. Spain. oriol.ma[email protected] M. Consolación Gómez Soberón Ph.D. in Civil Engineering. Full-time lecturer in the Department of Materials. Metropolitan Autonomous University. Mexico. [email protected]zc.uam.mx Luis Alberto Gómez Soberón Master in Structural Engineer. Practice Civil Engineer. Mexico. [email protected] ABSTRACT: The accomplished experience, using the data processing platform Moodle, in the execution of multiple-option questionnaires with automatic evaluation; is presented. The obtained results have been utilized subsequently for their statistical analysis by means of centered measures and basic dispersion. The documented experience was implemented in the subject: Construction of Traditional Structures and Equipments (CETE), in the Technical Architecture (AT) career of the Technical University of Catalonia (UPC) The summary of the information includes the results obtained for a total sampling of 437 students distributed in four different groups. In those groups, three different professors taught the classes for the two available schedules. The obtained results facilitate to discern among the different professors, student typologies, student gender, different levels of acquired knowledge, relation to other evaluation techniques applied, and relation to the documented prior knowledge. It is proposed and analyzed, basing on the obtained results, educational adaptations that will allow future improvements in subjects with similar requests or needs on the part of the students. Similarly, possible poor preceding formation in students or in the teaching by professors can be determined; both shall be corrected after the analysis. This work is part of the effort achieved in the educational improvement field, that is being executed inside the Upper-Technical Building College of Barcelona (EPSEB) and framed in the European Space of Higher Education [1, 2, 3, 4, 5, 6]. The objective is to show the current descriptive focal point on the learning and continuous evaluation field, applied in this subject. Keywords: Statistical analysis in teaching, Questionnaires for evaluation, Higher Education, Student Evaluation. I THEORETICAL – CONCEPTUAL FRAMEWORK. There are numerous published investigations describing the convenience to introduce teaching evaluation questionnaires by means of data processing systems. Likewise, there have been observed some advantages that those systems contribute with, such as results management, savings in the evaluation time, the not need to supply the questionnaire in paper, etc. On the other hand, the existence of some objections has been shown, such as: students ID confidentiality, the subsequent use of the information, the possible repercussion of its use in the educational process, etc. [7, 8, 9, 10, 11] The statistical analysis in the evaluation (its reliability and the surveys authority for the educational quality evaluation) has given credibility to these processes. In some occasions, and currently in a more official form; these processes have not served just as developers of educational changes, but also as promotion tools in the teaching, as criterion of academic compliance, and as an evaluator parameter for economic incentives concession for the teaching staff or for the educational institution [12, 13, 14, 15, 16] It is important to emphasize that there are nowadays tools and calculation processes which permit: multiple processes analysis, creation of simulations or hypothesis validation in the prediction of guidelines inside the teaching field [17]. It is not common to analyze the results data in students evaluations for their comprehension and subsequent analysis by professors [18]; there are just few documented cases about the use of these analysis in programs and educational systems in the national situation [19, 20, 21] As possible causes of this lack of empathy by part of the educational community, we could enumerate the following conjectures observed in daily practice of educational labor: 1. The traditional position by conventional faculty towards these new tools; since it is commonly considered that there has been already a considerable time invested in the students evaluation, and it is not necessary "to be more overwhelmed" in this "ineffective" process. 2. Since, with the evaluation of students, the course finishes, the faculty will not have again contact with the same students and, therefore, "it is not necessary to explore situations that will not be repeated again". 3. The analysis and derivations that this type of information can generate "does not contribute anything or very little to the educational process" or to the educational improvement. 4. The information generated with this type of analysis would be able to generate guidelines for the intervention and specific supports to students or groups that need it; nevertheless, "the university teaching shall not promote this type of specialization". 5. The specific actions produced by these analyses would be able "to generate needs of additional resources that the university teaching cannot assume". But the reality is that, according to the adaptation process to the European Space of Higher Education (ESHE); conceptual and deep structuring changes related to the teaching staff work, to the form in which the knowledge should be transmitted, to the way in which is easier to learn by students, and to the correct social satisfaction in a competent education required by society and by the educational institution; are being prompted [¡Error! Marcador no definido., 22] In line with these ideas, it has been begun to question and to analyze all the professors and university educators acting: incorporating the analysis and the deduction of learning results in students as a new educational tool that provides answer to the current need of analysis and deduction of possible tendencies. Similarly, as second derivation of this information, it will be able to predict and to validate the correct educational performing or the process and system to evaluate the students. In a nearby future, all this system will permit, besides, to include instruments such as [18]: 1. Specific software tools for educational institutions that permit the analysis of existing or future data. 2. Standardized methods or practices that permit to identify cases or critical situations. 3. Development of professors in specific positions with particular thematic contents. II EVALUATIVE HISTORIC PROFILE OF THE SUBJECT The study and analysis of data presented is with reference to the subject "Construction of Traditional Structures and Equipments (CETE or Construction III): this subject is part of the Technical Architecture syllabus (AT) in the Upper-Technical Building College of Barcelona (EPSEB) of the Technical University of Catalonia (Spain) (UPC) It is a fourmonth term subject in the second year of the career (Obligatory in Curriculum Block BC3) and it is taught in the four-month term 2A, its worth is 4.5 credits (one credit equals to 10 class hours) subdivided in: 3.0 theoretical credits and 1.5 credits of practices. The subject is taught simultaneously in four groups in all the four-month terms (1Q: Autumn and 2Q: Spring): two groups for students that attend class in the mornings (Groups 1M and 2M), and two groups more for students that attend class in the afternoon (3T and 4T) A global view, for the four groups, of the statistical facts for the subject is summarized in Table 1 [23]: Table 1. Academic results Course Semester Total Student Notes Distinction Exce l lent Notable A p pro ved Suspended No t submitted 2002/2003 1Q 363 0 0 10 234 105 14 2Q 343 0 0 12 233 80 18 2003/2004 1Q 315 0 0 7 174 116 18 2Q 271 0 0 10 199 64 12 2004/2005 1Q 290 0 0 7 142 132 9 2Q 285 0 0 24 180 64 3 2005/2006 1Q 302 0 0 8 170 105 19 2Q 257 0 0 11 137 95 14 2006/2007 1Q 299 0 0 11 182 89 17 2Q 205 0 0 4 149 46 6 2007/2008 1Q 273 0 0 19 172 70 12 2Q 236 0 0 13 160 61 2 TOTALS 3439 0 0 136 2132 1027 144 % ON THE TOTAL 0,0% 0,0% 4,0% 62,0% 29,9% 4,2% As shown in Table 1, the historic percentage of approved students in the subject is about 66% (Notable more Approved) Also, a simple significant correlation can be appreciated (of the direct type) as that for the four-month term 2Q there are better results than for the 1Q (with the exception of 2005/2006 school year, just for a 3%) The previous aspect is of greater importance if it is linked with the number of students registered by term: while the average of all the courses is about 72 students by class, in the 2Q the number of students is always slightly lower. As conclusion of this, generally, courses with smaller number of registered students report greater number of approved students; being the four-month term 2Q the one with, historically, lesser number of students and, consequently, more amounts of approved students. In accordance with the authors criteria and by their own experiences; it seems that, although the contributed data do not correlate in a direct form these variables (because they are not filtered), great part of the first-time registered students manage to accredit the subject; leaving as unsuccessful the students who repeat the term. It is also evident that the larger number of students in a class causes a detriment in the quality and personalized dedication that a professor can give to his or her class. On the other hand, an investigation to relate the number of students that manage to approve the subject with the ones who have prior basic knowledge acquired in approved related subjects (Construction I and Construction II) or with supposed accreditations of their knowledge (not always in a sufficiently truthful form), has been done; as it will see further on in this work. It can be observed in Table 1 that the majority of the grades to accredit the subject are concentrated in the approved item, and the second important statistical figure is for those who have suspended the subject. A conclusion of the previous paragraphs, in general conditions, is that having the subject as a one concept, it exists a high number of students that suspend it; in the cases of classes with smaller number of students, the approved figures improve sensitively by the order of the 10%; and, on the other hand, the obtained grades to accredit it have been historically near to the low limit. III METHODOLOGY III. 1 Procedure for the analysis to apply. For the design of the evaluation analysis system to utilize in this work, some general criteria and practical recommendations have followed [16, 20], in order to guarantee a correct application of the work and to avoid bias by its incorrect use. In this way, from the generic fields used for the processing of this type of information, there exist the levels of analysis and of application. For these levels, in a summarized form, there are three sections for each one of them. For analysis levels: individual, institutional and systematic. For application levels: faculty or students, educational institutions or programs, and the entirely educational system. In order to have affinity with the reality evaluated, and because the degree of takes of decisions or the conclusions and hypothesis analyzed explanation; for this work, it has prescribed some application and analysis levels in an organize form: most of the variables to be analyzed will be of individual level (variable linked to the students and to their evaluation process), which will be utilized to decide and to propose actions tending to the teaching improvement and to the cognitive development of the students. In smaller range, variables will evaluate to analyze, in an institutional level, a program of possible interventions in the educational field, an adaptive improvement of the program to obtain the degree, and a general development of the institution. In a summarized form, for both cases, the complete work is eminently of formative nature. For the report typology and content structure, it has been opted to elect, similarly to the previous case, a mixed report, this is: more scientific than evaluative, since the technical soundness sought has dominate on the need of efficient communication. Also because the information here presented is more, in a deliberate way, aimed at specialists than to not specialize audiences. The information utilized in this work also responds to the statistical answers for specific items, that is, it is utilized and analyzed, giving the items multiple option percentages or grades percentages of the students. Finally, referring to the report characteristics, it has been opted to include all the necessary information for the potential or real beneficiaries (faculty and educational institution), applying the necessary criteria of: maximum relevance, importance, conciseness, etc. III. 2 Design of the research. The analysis here presented obtains the information to process by means of the results extracted in the valuation of the studied subject. That evaluation process was defined in the school term 2008/09-2Q, it consisted of a continuous evaluation in which each two modules from the thematic content, a to-grade directed activity was developed. Also, two countable exams were complete, with the following considerations (Table 2): Table 1 Evaluation procedure of the subject. Academic Content (Modules) Cont e n ts (%) Technical Evaluation Final Grade (%) 1 y 2 25 Activity nº 1 5 3 y 4 25 Activity nº 2 5 1, 2 , 3 y 4 50 1 st Mid-term Exam 40 5 y 6 25 Activity nº 3 5 7 y 8 25 Activity nº 4 5 5, 6 ,7 y 8* 50* 2 nd Mid-term Exam* 40* * Recovery 1st Midterm Exam (optional) The activities developed by students consisted in solving problems or real cases associated to applications from the thematic contents given in classes. These activities were developed individually by students and graded according to some previously established rules (class agreements) The evaluations with grade were based on the resolution of two clearly differentiated sections, in the case of the 1st. mid-term exam they consisted in the resolution of a graphic–conceptual problem, with a specific 50% value of this exam grade, and a multiple answers test with a the other 50% value for the grade. The multiple answer test utilized was formed by a 20-question format with three possible answers to select in each one, and implemented in the subject Virtual Campus by means of the Moodle data processing platform [24]; this test was proposed to evaluate the different knowledge levels acquired by students according to the called Taxonomy of Bloom [25] In a summarized form, the subdivision of the evaluated knowledge levels is presented in Table 3, including the number of questions for each one of them. Table 2 Bloom's Taxonomy of evaluative test. Levels Type Number of questions Level 1 Knowledge 4 Level 2 Comprehension 2 Level 3 Application 7 Level 4 Analysis 4 Level 5 Synthesis 2 Level 6 Evaluation 1 TOTALS 20 For the case of the 2nd mid-term exam, the graphic-conceptual problems resolution, incorporated in this part of the thematic content, was utilized as evaluation method; permitting in this occasion (as a request from the students) to recover the grade of the first mid-term exam, resolving the proposed test in the that 1st exam. Therefore, for the analysis of this first part of the statistical study, twelve different variables were taking into account, assigning them codes and meanings that are presented in Table 4. Table 3 Nomenclature of the study variables. Nomenclature Meaning associated with the variable Range of possible values VAR01 Student Gender 1 = male, 2 = female VAR02 Groups they belong to the students 1 = 1M, 2 = 2M, 3 = 3T, 4 = 4T VAR03 Hours in which class is offered 1 = Morning, 2 = Afternoons VAR04 Activity Note nº 1 Del 0 al 10 * VAR05 Activity Note nº 2 Del 0 al 10 * VAR06 Note the 1 st midterm exam (part-conceptual graph) Del 0 al 10 * VAR07 Note the 1 st midterm exam (hand test) Del 0 al 10 * VAR08 Activity Note No 3 and No 4 Del 0 al 10 * VAR09 Note the 2 nd term exam (part-conceptual graph) Del 0 al 10 * VAR10 Note recovery of the 1 st midterm exam (test) Del 0 al 10 * VAR11 Note end of the course Del 0 al 10 * VAR12 Number of times you have registered for the course Del 1 al 5 * With accuracy of two decimal places of significance. With these criteria and variables to analyze, the data processing program of statistical analysis SPSS V17 for Windows, was utilized in order to obtain the following parameters: general statistical descriptive for each one of the variables in a isolated form, with the purpose of knowing and distinguishing the samples in an isolated form also; next an analysis of bi-varied correlation was performed to seek affinity among the values of pairs of samples and to compare among them with the aim of verifying the relation among the different variables. III. 3 Examination of external variables of the subject. With the intention of investigating the relation among the achievement of this subject knowledge (and thus to obtain the approved status) and the previously acquired knowledge (the access to the university for students and their origin), a second statistical analysis has been accomplished. For this, the grades obtained previously, corresponding to the two prior subjects with related knowledge to Construction III: Construction and materials knowledge (Construction I) and Construction of elements (Construction II); and finally, the origin of the student has been considered. In this case, the variables to enclose to the data analysis before mentioned are presented in Table 5, (adopted nomenclature, description and possible ranks that each variable can adopt) Table 4 Nomenclature of variables external analysis. Nomenclature Meaning associated with the variable Range of possible values VAR13 Note accreditation Construction I Del 0 al 10 * VAR14 Note accreditation Construction II Del 0 al 10 * VAR15 Access to college 1 = Test Access to University (PAU), 2 = Foreign selectivity, 3 = Diploma, 4 = Training (FP), 5 = Studies initiated via University Course (COU), 6 = Studies initiated via FP, 7 = Over 25 * With accuracy of two decimal places of significance. III. 4 Data analyzes. From the data universe analyzed (437 students), fourteen samples or students were separated because they did not present evaluative activity in any of the variables to analyze; understanding that these students, for a specific motive (their own decision), do not belong to the developed sampling. So, the analysis is done with a total number of 423 students. As first step, to know better the proposed variables, the process to obtain a series of parameters was developed; among them: Centering measures (Mean, Median, Mode and Sum), Dispersion measures (standard deviation, Variance, Amplitude, Minimum, Maximum and Media typical error), Samples distribution (Asymmetry and Kurtosis), and finally the Percentiles Values. In Table 6, the general results obtained for all the samples and analyzed variables, referring to its general description, are presented. The values reported for the variables: VAR01, VAR02, VAR03 and VAR15, have to be analyzed taking into account that they suffered a change from alphabetical variable to numerical, in agreement to the indications shown in Table 4 and Table 5; to permit the analytic process of them; therefore, they do not exactly determinate the habitual statistical meaning (especially the parameters of centering measures) Nevertheless, the analysis parameters of the samples can contribute with general information about to dispersion measures such as: the standard deviation and the variance, as well as their own distribution. Table 5 Descriptive statistics for study variables. VAR01 VAR02 VAR03 VAR04 VAR05 VAR06 VAR07 VAR08 VAR09 VAR10 VAR11 VAR12 VAR13 VAR14 VAR15 N Valid 422 423 423 423 423 423 422 423 423 422 423 423 322 138 423 Lost 1 0 0 0 0 0 1 0 0 1 0 0 101 285 0 Mean 1.37 2.53 1.51 6.349 6.34 4.719 5.87 6.415 4.371 8.29 6.122 1.1 5.597 5.48 2.64 Standard error of the mean 0.024 0.059 0.024 0.078 0.102 0.108 0.095 0.113 0.096 0.087 0.064 0.021 0.039 0.058 0.095 Median 1 3 2 6.5 7 5 6.25 7 4.5 8.88 6.3 1 5.4 5.2 1 Mode 1 4 2 6 7 5 6.3 7.3 5 10 5 1 5 5 1 Standard deviation 0.484 1.219 0.501 1.618 2.091 2.2205 1.955 2.325 1.979 1.779 1.321 0.434 0.695 0.677 1.962 Variance 0.234 1.487 0.251 2.62 4.37 4.931 3.82 5.406 3.915 3.165 1.744 0.188 0.484 0.458 3.848 Asymmetry 0.531 -0.033 -0.033 -2.099 -1.465 -0.017 -0.772 -1.658 -0.2 -2.573 -2.055 5.27 1.308 1.607 0.62 Standard error of asymmetry 0.119 0.119 0.119 0.119 0.119 0.119 0.119 0.119 0.119 0.119 0.119 0.119 0.136 0.206 0.119 Kurtosis -1.726 -1.574 -2.008 7.268 2.658 -0.359 0.968 2.43 -0.158 8.8 7.772 32.236 1.436 1.905 -1.108 Standard error of Kurtosis 0.237 0.237 0.237 0.237 0.237 0.237 0.237 0.237 0.237 0.237 0.237 0.237 0.271 0.41 0.237 Amplitude 1 3 1 9.5 10 10 10 10 10 10 9 4 3.5 3 6 Minimum 1 1 1 0 0 0 0 0 0 0 0 1 5 5 1 Maximum 2 4 2 9.5 10 10 10 10 10 10 9 5 8.5 8 7 Sum 579 1070 638 2685.5 2682 1996.3 2477.3 2713.8 1848.8 3498 2589.6 467 1802.1 757 1116 Percentiles 10 1.0 1.0 1.0 5.5 4.5 2.5 3.5 4.3 1.5 6.6 5.0 1.0 5.0 5.0 1.0 20 1.0 1.0 1.0 5.5 5.0 2.5 4.3 5.3 2.8 7.5 5.3 1.0 5.0 5.0 1.0 25 1.0 1.0 1.0 6.0 5.5 3.8 4.8 5.8 3.0 7.5 5.5 1.0 5.0 5.0 1.0 30 1.0 2.0 1.0 6.0 6.0 3.8 5.0 6.3 3.5 8.0 5.7 1.0 5.0 5.0 1.0 40 1.0 2.0 1.0 6.0 6.5 3.8 5.6 6.8 4.0 8.5 6.0 1.0 5.2 5.0 1.0 50 1.0 3.0 2.0 6.5 7.0 5.0 6.3 7.0 4.5 8.9 6.3 1.0 5.4 5.2 1.0 60 1.0 3.0 2.0 6.5 7.0 5.0 6.5 7.3 5.0 9.0 6.5 1.0 5.6 5.3 4.0 70 2.0 4.0 2.0 7.0 7.5 6.3 7.0 7.5 5.5 9.5 6.8 1.0 5.8 5.5 4.0 75 2.0 4.0 2.0 7.0 7.5 6.3 7.1 7.8 5.8 9.5 7.0 1.0 6.0 5.7 4.0 0 1 2 3 4 5 6 7 8 9 10 11 VAR04 VAR05 VAR06 VAR07 VAR08 VAR09 VAR10 VAR11 VAR12 VAR13 VAR14 Mean Median Mode 0 500 1000 1500 2000 2500 3000 3500 4000 VAR04 VAR05 VAR06 VAR07 VAR08 VAR09 VAR10 VAR11 VAR12 VAR13 VAR14 Sum VAR01 VAR02 VAR03 VAR04 VAR05 VAR06 VAR07 VAR08 VAR09 VAR10 VAR11 VAR12 VAR13 VAR14 VAR15 80 2.0 4.0 2.0 7.1 8.0 6.3 7.5 8.0 6.0 9.5 7.1 1.0 6.2 6.0 5.0 90 2.0 4.0 2.0 8.0 8.5 7.5 8.3 8.5 7.0 10.0 7.5 1.0 6.6 6.6 5.0 100 2.0 4.0 2.0 9.5 9.5 10.0 10.0 10.0 10.0 10.0 9.0 5.0 8.5 7.9 7.0 Now, based on Table 6, Figure 1, Figure 2 and Figure 3; some reflections that concern to the determined statistical parameters for the studied variables are mentioned. For the centering measures, the general behavior of the samples (from VAR04 to VAR09 and VAR11) can be summarized saying that the evaluated parameters are centered slightly above the center of the distributions; the same with the symmetry of its distribution. For VAR10, it is evident that the recovery process of the first mid-term exam moves away from the habitual behavior of the subject; because of different factors such as: prior knowledge of the content by students (it is easier), excessive time in test application (calibration), levels of low control (guarantee of application), etc. Being therefore necessary that these factors be calibrated in future put in practice. For VAR12, the values of the reported parameters indicate that the initially proposed hypothesis, in which the students who repeat the course were considered as the origin of the subject fails, does not have a numerical support; therefore, it is verified that the rejuvenation of students in the courses is produced in a natural and constant form. Finally, for VAR13 and VAR14, the values reported for their central parameters indicate to be much related among them; being on the other hand, of greater demand in the referring to their evaluation in comparison with the subject Construction III. This is due to factors such as: affinity in the thematic contents of the subjects Construction I and Construction II, or, by contrary, opposition of both with those of Construction III. Figure 1 Graphic representation of the behavior of the measures centering of the variables. academic level and to avoid the possible failure of a student with poor prior knowledge. The article general comments are: 1. The statistical analysis of the data generated in educational evaluations is useful to understand the behavior of variables that participate in its decision; likewise, these techniques help to take correct decisions to future actions, and permit to predict behaviors, to establish methods or techniques to reduce problems. 2. The observations of all samples have a sensitively normal distribution, linking the behavior (as a group) of the applicable variables to the teaching field with this type of distribution. 3. The analyzed data have permitted to know and to corroborate the relations of most of the variables based on initial conditions; reckoning in this way, with certainty and validity, the general behavior of the assembly of samples. 4. More investigations in this predictive and analytic teaching field are desirable, since they are productive and important for the faculty, students and the educational institution where they are accomplished. ACKNOWLEDGMENTS The authors thank to: Projects Financing Aids for the of the Educational Quality Improvement in the Catalonian Universities for the year 2008 (MQD) by the Agència of Gestió d' Ajuts Universitaris i de Recerca, to the Improvement Project Aid to the Teaching Assembly 2008 by the UPC (Sciences of the Education Institute), and to the Department of CAII of the EPSEB of UPC REFERENCES 1. Gómez Soberón, J. M.; Gómez Soberón, M. C. (2007) Aplicación de una técnica de mejora docente. ACHE. II Jornadas de Enseñanza. Asociación Científico- Técnica de Hormigón Estructural. Madrid, Spain, 30/01/2007. pp. 77-83. [online] [Accessed 1 September 2009] http://hdl.handle.net/2117/2848 2. Gómez Soberón, J. M.; Gómez Soberón, M. C. (2007) Moodle como herramienta para la creación de un campus virtual: adaptación al EEES. ACHE. II Jornadas de Enseñanza. Asociación Científico-Técnica de Hormigón Estructural. Madrid, Spain, 30/01/07. p. 173-179 [online] [Accessed 1 September 2009] http://hdl.handle.net/2117/2849 3. Gómez Soberón, J. M.; Gómez Soberón, M. C. (2007) Hormigones reciclados: nuevo Contenido Temático en asignaturas de hormigones. ACHE. II Jornadas de Enseñanza. Asociación Científico-Técnica de Hormigón Estructural. Madrid, Spain, 30/01/09. p. 337-343. [online] [Accessed 1 September 2009] http://hdl.handle.net/2117/2847 4. Gómez Soberón, J. M.; Gómez Soberón, M. C. (2008) Adaptación de las asignaturas de construcción del Departamento de Construcciones Arquitectónicas II al nuevo concepto de los créditos europeos de Educación Superior y del campus virtual Atenea. Instituto de Ciencias de la Educación, Universidad Politécnica de Cataluña. Jornada de Presentació de Resultats dels Projectes de Millora de la Docència 2008. [online] [Accessed 1 September 2009] http://hdl.handle.net/2099/4870 5. Gómez Soberón, J. M.; Gómez Soberón, M. C. (2008) Evolución hacia EEES: un caso práctico. Instituto de Ciencias de la Educación, Universidad Politécnica de Cataluña. Jornada de Presentació de Resultats dels Projectes de Millora de la Docència. 2008. [online] [Accessed 1 September 2009] http://hdl.handle.net/2099/4870 6. Gómez Soberón, J. M.; Gómez Soberón, M. C.; Gómez Soberón, L. A. (2009) Active learning and autonomous learning: an educational experience. ALE- 2009 The learning Experience. Creative Commons - UPCommons. 2009. [online] [Accessed 1 September 2009] http://hdl.handle.net/2099/7809 7. Burden, P. (2008) ELT teacher views on the appropriateness for teacher development of end of semester student evaluation of teaching in a Japanese context. System Science Direct Elsevier, 36, p. 478-491. 8. Nulty, D. D. (2008) The adequacy of response rates to online and paper surveys: what can be done? Assessment & Evaluation in Higher Education, vol. 33, nº 3, p. 301-314. 9. Huxham, M. [et al.] (2008) Collecting student feedback: a comparison of questionnaire and other method. Assessment & Evaluation in Higher Education. vol. 33, nº 6, p. 675-686. 10. Nair, Ch. S.; Adams, P.; Mertova, P. (2008) Student Engagement: The key to improving survey response rates. Quality in Higher education. vol. 14, nº 3, p. 225-232. 11. Davies, M. [et al.] (2009) A systematic analysis of quality of teaching surveys. Assessment & Evaluation in Higher Education. 1, p. 1 – 14. 12. Kember, D.; Leung, D. Y. P. (2008) Establishing the validity and reliability of course evaluation questionnaires. Assessment & Evaluation in Higher Education. Vol. 33, nº 4, p. 341- 353. 13. Gaspar, M. F. [et al.] (2008) A questionnaire for listening to students’ voices in the assessment of teaching quality in a classical medical school. Assessment & Evaluation in Higher Education. vol. 33, nº 4, p. 445 – 453. 14. Oliver, B. [et al.] (2008) eVALUate: an evaluation instrument for measuring students’ perceptions of theirs engagement and learning outcomes. Assessment & Evaluation in Higher Education. vol. 33, nº 6, p. 619 – 630. 15. Van Cleave, K. (2008) The self-study as an information literacy program assessment tool. Collage & Undergraduate Libraries. vol. 15, nº 4. p. 414 – 431. 16. Tiana, A. (1997) Tratamiento y usos de la información en evaluación. Calidad y Equidad en la Educación. Organización de Estados Iberoamericano [online] [Accessed 1 September 2009] http://www.oei.es/calidad2/tiana.htm 17. Hutchison, D. (2009) Designing your sample efficiently: clustering effects in education surveys. Educational Research. vol. 51, nº 1, p. 109 – 126. 18. Thompso, D.; Treadaway, M.; Knight, T (2005) Investing data use to identify learning needs. Department for Education and Skills Educational Research. Research Report, nº 665, p. 1 – 78. 19. Simm, C. (2006) Improving the collection of qualifications data. Department for Education and Skills Educational Research. Research Report, nº 715 p. 1 – 92. 20. Ravel, P. [et al.] (2000) Grupo de Trabajo Sobre Estándares y Evaluación. Los próximos pasos: ¿hacia dónde y cómo avanzar en la evaluación de aprendizajes en América Latina? Programa de Programación de la Reforma Educativa en América Latina y el Caribe. [online] [Accessed 30 October 2009] http://www.oei.es/calidad2/grade.PDF 21. Fuseler Avery, E. (ed.) (2003) Assessing Student Learning Outcomes for Information Literacy Instruction in Academic Institutions. Chicago: American Library Association, 2003. p. 299. ISBN 0-8389-8261-1 22. América, V.; Amaral, A. (2009) Survey on the implementation of the Bologna process in Portugal. Higher Education: The International Journal of Higher Education and Educational Planning, vol. 57, nº 1, p 57-69. 23. Escuela Politécnica Superior de Edificación de Barcelona (2009) Memoria 2007- 2008, datos estadísticos. Universidad Politécnica de Cataluña. 24. Dougiamas, M. Moodle [online]: community. [Accessed 1 September 2009] http://moodle.org/ 25. Van Niekerk, J.; Von Solms, R.; Using Bloom's Taxonomy for Information Security Education. En: World Conference on Computers in Education. Education and Technology for a Better World: 9th IFIP World Conference on Computers in Education (WCCE 2009) Bento Gonçalves, RS, Brazil, 27-31 Julio, 2009 [online]. [Accessed 1 September 2009] http://www.wcce2009.org/proceedings/papers/WISE6_Niekerk.pdf