The use of Audience Response Systems in the Faculty of Economics and Business: an case of study

THE USE OF AUDIENCE RESPONSE SYSTEMS IN THE FACULTY OF ECONOMICS AND BUSINESS: AN CASE OF STUDY José Ignacio Castillo Manzano Mercedes Castro Nuño Maria Teresa Sanz Díaz Rocío Yñiguez Ovando Dpto. Análisis Económico y Economía Política Facultad de Ciencias Económicas y Empresariales Universidad de Sevilla jignaci[email protected] ABSTRACT We have investigated the effectiveness of clickers as a support tool of the learning process of economics in higher education, specifically on the subject of Principle of Economics. We assessed whether support teaching with the use of Audience Response Systems (ARSs) increase the probability that the student passes the examination in both its theoretical and practical. We propose a mixed methodology, a bivariate probit model framed by statistical causal inference, which provides robust results. We have found strong statistical evidence, that the ARSs offer outstanding support to teaching, although with constraints. First focuses on the aforementioned theoretical and secondly, the help offered by the students ARSs is clearly a function of the frequency of use thereof cited by students. KEYWORDS: European Higher Education Area, ICT, Active Learning, Clickers, University Teaching. THE USE OF AUDIENCE RESPONSE SYSTEMS IN THE FACULTY OF ECONOMICS AND BUSINESS: AN CASE OF STUDY 86 1. INTRODUCCIÓN Nowadays, universities play a decisive role that goes beyond their traditional function as centers of teaching, specialization and research systems, in an expanding framework of market-based orientation and internationalization of studies, students, researchers, teachers and institutions (Buela-Casal et al. 2007, Leydesdorff & Meyer, 2003). In a competitive world, researchers such as Bonaccorsi & Daraio (2007) and Visser et al. (2007) analyze changes in university structures on every continent to guarantee the efficient use of public/private resources and promote a culture based on quality. During the last decade international academic rankings based on a comparison of prestige universities have proliferated as useful tools for policy makers, industries, project funding and students (Buela-Casal, et al. 2007; Cheng & Liu, 2006), although these quality quantification procedures mainly refer to scientific production and research productivity (De Filippo et al. 2012). University prestige has multidimensional fields (García et al., 2012) and the concept of quality teaching in higher education is much more difficult to define through quantitative and measurable mechanisms (Díaz-Méndez & Gummesson, 2012) because each university has its own criteria for evaluating teachers and programs that cannot be compared easily (Buela-Casal et al. 2012). Nevertheless, teaching quality is currently a core issue in the context of change and restructuring that European universities are experiencing as they become part of the European Higher Education Area (hereafter EHEA) (Jiménez & Palmero, 2007), because teachers seem to be key pieces of the so-called value creation process for students (Barile & Polese, 2010; Vargo & Lusch, 2008) in which students are not only service customers but are actively involved in a joint and interactive learning process. New teaching strategies and the use of innovative tools are required to promote the development of generic and specific competences in students at the same time that they are autonomous and can undertake independent learning (Jiménez & Palmero, 2007; Salas et al. 2012). García-Valcárcel et al. (2009) and Iniesta-Bonillo et al. (2012) argue that the use of Information and Communication Technology (here after ICT) for interactive learning is an important influence and that it is a source of innovation for all types of organizations to gain a competitive advantage. The use of ICT and interactive learning in higher education has been extensively studied by the academic literature (see e.g., Arenas-Márquez et al. 2012 or Cheung & Slavin, 2012 for a deeper analysis). New innovations and digital media have enriched the teaching and learning processes and have become commonplace among university students and lecturers during past few years (Hatakka & Lagsten, 2012; Paechter et al., 2010), with the widespread advance of e-learning with multimedia tools, such as wikis, virtual platforms (Blackboard, WebCT and Moodle, among others) and even micro-blogging networks (Delfino & Persico, 2007). THE USE OF AUDIENCE RESPONSE SYSTEMS IN THE FACULTY OF ECONOMICS AND BUSINESS: AN CASE OF STUDY 87 Many of the innovative teaching experiences have been based on the application of a kind of ICT tool called Audience Response Systems (hereafter ARSs) or clickers (Martín- Laborde, 2005). An ARS or clicker system, allows students in large classrooms to answer multiple choice questions anonymously. Questions usually take the form of a Power Point presentation. Students click on their handsets, the results are collected and displayed in chart form. Both teacher and student have feedback on how well the entire class understands the concepts being explained (see Moss & Crowley, 2011, for a fuller description of this technology). The aim of this paper is to evaluate the effectiveness of ARSs as a tool to support the learning process in economics in higher education, using a case study carried out at the University of Seville (Spain). The subject in question was, specifically, Principles of Economics taught at the School of Economics. Our goal is to evaluate whether supporting teaching with the use of ARSs increases the likelihood that students will pass the two papers that comprise the final examination in the subject, the theoretical paper (with questions where students must elaborate on their answers), and the practical paper (based on mathematical problems). Using the skills that students have acquired and the scores that they achieve in examinations as a proxy we test whether the use of ARSs in the university teaching procedure is reflected in their learning process through learning outcomes. For this we use an econometric approach, namely a bivariate probit model framed by statistical causal inference, which, apart from offering robust results, can be easily replicated in any other similar experiment. Following Arenas-Márquez et al. (2012), the effect of ICT-based methods as an alternative/complement to traditional methods for improving teaching, is a major topic of interest for researchers. Most earlier studies on ICT-based tools have demonstrated their value for the learning process from a subjective perspective measured by variables relating to students’ perceptions (Klein et al. 2006; López-Pérez et al. 2011). Nevertheless, authors such as Ginns & Ellis (2009) state that further evaluation of the impact of the use of ICT on student learning in higher education is required. Kay & LeSage (2009) point out in this regard that the impact of clickers on learning has not been studied in detail. There are many qualitative studies, but we agree with other previous works that the research done on this topic is limited (Patterson et al., 2010; Blasco et al., 2013) because there is only a small number of quantitative studies that use methodologies such as econometric models to analyze the benefits of ARSs. Besides, following Caldwell (2007) they are heterogeneous and do not enable consistent conclusions to be drawn. To shed light on the issue, this paper also includes a recent review of the latest literature on the impact of ARSs. The study is structured in five sections. After this Introduction, Section 2 presents a literature review of ARSs evaluation. Section 3 describes the data and the methodology of our research. The results are discussed in Section 4 and the Conclusions are presented in Section 5. Finally, we include the references. THE USE OF AUDIENCE RESPONSE SYSTEMS IN THE FACULTY OF ECONOMICS AND BUSINESS: AN CASE OF STUDY 88 2. LITERATURE REVIEW Today, at public universities, lecturers teach hundreds of students in huge classes and traditional teaching methodology can mainly lead to students feeling that they are passive recipients of the lecture (Valentín et al. 2013). Often students seem distracted and unmotivated in class and interaction between teacher and students is limited in traditional lecture classes. There are several types of feedback, such as hand-raising, requesting volunteers, and using cards, but the disadvantages of the traditional modes of involvement are that student’s lack anonymity, they are slow and, also, there are few participants (Kay, 2009). Therefore, several studies have analyzed the positive effects of active learning (such as ARSs) on engagement; attendance and academic performance (see Desrochers & Shelnutt, 2012; Mollborn & Hoekstra, 2010 and Moss &Crowley, 2011). The use of ARSs in higher education dates back to the 1960s at U.S. universities (Chafer, 2009) but it was in the mid 1990s that their use spread (Judson & Sawada, 2002). The ARS-based classroom has various key advantages (Moss & Crowley, 2011; Kay, 2009), such as allowing an anonymous response, allowing all the students to respond with their handsets even in large classroom, feedback is immediate and information processing is very easy and quick. Salemi (2009) also shows the positive relationship between the audience response system and engagement. However, ARSs also have some disadvantages; clickers can be expensive to purchase and technical malfunctions can occur (Guse & Zobitz, 2011, investigated the accuracy of this technology and concluded that ARSs are a valid and accurate method of response collecting). Costello (2010) offers new ideas to overcome these drawbacks, such as the implementation of new software that uses the Internet and laptops. Diverse studies at American universities seem to have demonstrated that clickers reinforce the quality of education by promoting student involvement in the classroom (Berry, 2009 University of Wisconsin; Bode et al., 2009 Northwestern University; Matesic & Adams, 2008 York University, among others). Therefore, ARSs have been used in a wide variety of disciplines, such as Physics (Lin et al. 2011); Biology (Crossgrove & Curran, 2008); Economics (Salemi, 2009); and Psychology (Desrochers & Shelnutt 2012), and for a range of subjects and different teaching levels (Moss & Crowley, 2011), such as school pupils (Kay, 2009) and undergraduates. However, few studies explore the factors that influence and determine whether or not students intend to continue using these systems in their classes and how they improve student academic performance, and only a limited number have been carried out in Social Sciences (Kay & LeSage, 2009). In general, the use of blended instruction and the application of elearning at universities have increased rapidly (Arenas-Márquez et al. 2012; García-Valcárcel et al. 2009). Nevertheless, little is known about students’ expectations and experiences, and the role of technology in students’ achievements has not been sufficiently tested from an empirical point of view: until recently, research focused on students’ experiences with specific aspects of e-learning courses, e.g., interaction with an instructor, learning with a specific learning management system, or certain characteristics of a course (Paechter et al. 2010). THE USE OF AUDIENCE RESPONSE SYSTEMS IN THE FACULTY OF ECONOMICS AND BUSINESS: AN CASE OF STUDY 89 The prior literature has in fact reported inconsistent or even contradictory results regarding the effect of technology use on student outcomes. In some cases, the use of technology has been measured by students' satisfaction with the learning experience (Lopez- Perez et al. 2011), but the use of objective data might enable its impact on outcomes to be analyzed more robustly. There are several reviews of ARSs in this respect; the most complete and recent are by Caldwell (2007) and Kay & LeSage (2009). Table 1 summarizes the main findings of the majority of studies on clickers published between these recent reviews and early 2013. 3. EMPIRICAL FRAMEWORK: SAMPLE To conduct this study, we analyzed a teaching experience using clickers in the Principles of Economics subject at the School of Economics (University of Seville) during the 2012/2013 academic year. The database was developed using student surveys and class notes. The population consisted of all the students who regularly attended class in this subject in the various groups (with different timetables) that take it. There were 441 students in all distributed between 9 first year groups on the Administration and Business Management degree course. The number of students enrolled on the course was in fact significantly higher, but all the students who did not regularly attend class were omitted (basically students repeating the course). The main characteristics of the sample used can be seen in Table 2, including socio-demographic characteristics and whether students combine their studies with some paid employment, for example. Three of these nine first year groups, with a total of 141 students, were then randomly chosen for intervention, in three knowledge tests of program content during the course. The remaining 322 students, (in six different groups taking the subject), who had no access to said ARSs, but who had to sit exactly the same final examination in the subject therefore formed a control group. This means that our sample is much larger than earlier studies with some type of methodology other than a simple Before/After analysis (Desrochers & Shelnutt, 2012; Marshall & Varnon, 2012; Patterson et al, 2010 and Powell et al, 2011). Another interesting feature that makes the results more robust is that our study presents the smallest intervention group/control group ratio of all the above-mentioned studies (specifically, Desrochers & Shelnutt, 2012, Patterson et al, 2010 and Powell et al, 2011: 1:1 or Marshall & Varnon, 2012: 1:2 while our study is: 1:3). In other words, this study has the largest control group compared to the intervention group with which to compare the real effectiveness of ARSs. Students in their first year at the School of Economics were chosen as, having just come from senior high school, it was virtually impossible for them to have had any previous experience of ARSs; at this moment in time we are not aware of ARSs being used at any high school in the proximity of the university. Clearly some repeating students might have had experience of them, but this possible bias (albeit doubtful, given the simple way that ARSs are used) has been corrected with the Freshman covariate (see Table 2). THE USE OF AUDIENCE RESPONSE SYSTEMS IN THE FACULTY OF ECONOMICS AND BUSINESS: AN CASE OF STUDY 90 Another positive aspect of our population, unlike the system in force at many American universities, is that virtually all the students are assigned to classes/groups before courses begin (there are no ‘shopping days’ to produce a bias depending on the quality of the instructor) and the students are in the same classes/groups for all the 10 subjects that comprise the first year, including Principles of Economics. Fairly random assignation of intellectual ability is therefore to be expected among the students distributed among the 9 groups. We jointly defined an experiment with approximated very well to the principles that govern simple random sampling for social science standards. As many as 10 covariates (see Table 2) have been included to prevent any unexpected bias, i.e., a clearly higher number of variables than the four studies cited above (Desrochers & Shelnutt, 2012; Marshall & Varnon, 2012; Patterson et al, 2010 and Powell et al, 2011). The interactive control ARS at the Faculty of Economic and Business Sciences can be used by all professors at the Faculty with prior notice. As previously stated, it was used for three tests on three blocks of content in the Principles of Economics program included in the first quarter of the first year of the undergraduate degree course in Administration and Business Management. The structure of the tests was the same in all three cases: 20 multiple choice questions with four possible answers. The norms for assessment were: one point for each correct answer, minus half a point for wrong answers, and zero points for answers left blank. The methodology used in our study is framed by statistical causal inference. It is based on the estimation of the causal effect (Pearl, 2000) that a specific measure or fact can have on one or more relevant variables (Dawid, 2000). This methodology allows consistent estimators of the effects of the evaluated measure to be obtained (Rotnitzky & Robins, 1995) by determining and isolating the possible impact of additional contaminating variables. Starting with an N-size random sample, we defined the binary variable D that indicates the observation corresponding to a student who has used ARS (Di =1) or a student in the control group who received traditional instruction, essentially based on lecture classes (Di=0). Thus, our N observations were divided into N1 and N0 observations (using ARS or clicker vs. traditional instruction). In our case, N1 stands for the 141 students who used ARSs or clickers, while N0 represents the remaining 322 students. Thus, the requirement that “N0 is at least the same order of magnitude of N1” is met in full (Abadie & Imbens, 2006). 4. RESULTS Table 1 summarizes the results of estimating the propensity score (Model (1)). We opted for a logit specification rather than a probit specification as it maximized the log pseudolikelihood. THE USE OF AUDIENCE RESPONSE SYSTEMS IN THE FACULTY OF ECONOMICS AND BUSINESS: AN CASE OF STUDY 91 Covariate Sevillian First option University access examinations Log pseudo likelihood Wald Chi2 Pseudo R2 No. of observations Coefficient -0.736(0.308)** 1.381(0.571)** -0.224(0.084)*** -177.017 16.99 0.052 340 Table 1. Main estimates derived from the Probit model. Note: Standard errors robust to heteroscedasticity are presented in brackets in the coefficient column. One, two, or three asterisks indicate coefficient significance at the 10percent, 5percent, and 1percent levels, respectively. We then used a bivariate probit specification to estimate the causal effects. The results are shown in Table 2. In Model 1, rows 2 and 3, we have used the basic model where D takes a value of 1 if the student was in the group that used ARSs and 0 if he/she was in the control group. In the last two rows (Model 2) we have considered Di as an increasing variable, between 0 and 3, depending on the number of tests that the student did with ARSs before sitting the final examination. Variable Clicker (Di) Number Clickers (Di) Model 1 Theory exam SIGNIFICANT AT 5% Practical exam NON-SIGNIFICANT Model 2 Theory exam SIGNIFICANT AT 5% Practical exam NON-SIGNIFICANT Table 2. Bivariate Probit estimation of relevant causal effects. As Table 2 shows both models 1 and 2, the use of clickers is significant to pass the theory exam but not in the case of the practical exam. In the model 1 we have obtained a positive coefficient sign and significant of the Clicker variable (0.38; 0.178); and in the model 2, we have found a positive coefficient sign and significant for the Number of Clickers (0.129; 0.064) variable. Furthermore, there is an accumulative effect regarding to that the more tests the student does with ARSs, the greater the likelihood is. THE USE OF AUDIENCE RESPONSE SYSTEMS IN THE FACULTY OF ECONOMICS AND BUSINESS: AN CASE OF STUDY 92 5. CONCLUSIONS After a review of recent studies that analyze and/or attempt to quantify the utility of ARSs in university teaching, it becomes clear that ARSs generally might have a positive impact on learning process. An ARS or clicker system, allows students in large classrooms to answer multiple choice questions anonymously, gives immediate feedback with very easy and quick information processing, and reinforces the quality of education by promoting student involvement in the classroom (Berry, 2009; Bode et al., 2009; Crossgrove & Curran, 2008; Desrochers & Shelnutt 2012; Lin et al. 2011; Matesic & Adams, 200; Salemi, 2008). However, previous literature seems to be very heterogeneous, usually based on qualitative methodologies and does not allow accurate conclusions to be drawn about the improvements achieved. Therefore, new empirical research needs to be done to provide concrete results on the real utility of ARSs that are, more importantly, based on robust and well-grounded methodologies that use control groups. This article puts forward a methodological, full and rigorous approach in this respect, both for the analysis and for the analysis to be applied practically. By applying it practically we were able to measure the extent to which ARSs used as a teaching support tool influenced the likelihood that a student would pass both the theoretical and practical papers of an examination. Our results show that ARS provides significant support to teaching, although with possible limitations. On the one hand, our study addresses only theoretical teaching, and we think that it would be interesting to extend the experiment through other practical parts of the subject. On the other hand, three tests with clickers had been done by the students along our intervention, so we think that the results might have been better if they had been used more regularly. 6. REFERENCES Abadie, A., Imbens, G.W. 2006. Large sample properties of Matching estimators for Average Treatment Effects. Econometrica, 74 (1), 235-267. Arenas-Márquez, F.J., Machuca, J.A.D., Medina-López, C. 2012. Interactive learning in operations management higher education. International Journal of Operations & Production Management, 32(12), 1395-1426. Barile, S., Polese, F. 2010. Linking the viable system and many-to-many networkapproaches to service-dominant logic and service science. International Journal of Quality and Service Sciences, 2(1), 23-42. Berry, J. 2009. Technology supports in Nursing education: Clickers in the classroom. Nursing education perspective, ProQuest Health and Medical Complete, 30(5), 295-298. THE USE OF AUDIENCE RESPONSE SYSTEMS IN THE FACULTY OF ECONOMICS AND BUSINESS: AN CASE OF STUDY 93 Bode, M., Drane, D., Kolikant, B-D Y, Schuller, M. 2009. A clicker approach to teaching Calculus. Notices of the AMS, 56(2), 253-256. Bonaccorsi,A., Daraio, C.(Eds). 2007. Universities and strategic knowledge creation: specialization and performance in Europe. Cheltenham: Edward Elgar Publishing. Buela-Casal, G., Gutiérrez-Martínez, O., Bermúdez-Sánchez, M.P., Vadillo-Muñoz, O. 2007. Comparative study of international academic rankings of universities. Scientometrics, 71(3), 349-365. Buela-Casal, G., Bermúdez, M.P., Sierra J.C., Quevedo-Blasco, R., Castro, A., Guillén- Riquelme, A. 2012. Ranking de 2011 en producción y productividad en investigación de las universidades públicas españolas. Psicothema, 24(4), 505-515. Caldwell, J. E. 2007. Clickers in the Large Classroom: Current Research and Best-Practice Tips. CBE- Life Sciences Education, 6 (1), 9-20. Cameron, A. C., Trivedi, P. K. 2005. Microeconometric.Methods and Applications. New York: Cambridge University Press. Chafer, E. 2009. Una introducción a los sistemas de respuestas interactivos. Electrónica y Comunicación. Monográfico TICs en las aulas. Elementos didácticos para la enseñanza, 242,56- 57. Cheng, Y., Liu, N.C. 2006. A first approach to the classification of the top 500 world universities in their disciplinary characteristics using scientometrics. Scientometrics, 68(1), 135- 150. Cheung, A.C.K., Slavin, R.E. 2012. How features of educational technology applications affect student reading outcomes: a meta-analysis. Educational Research Review, 7, 198-215. Costello, P. 2010. A cost-effective classroom response system. British Journal of Educational Technology, 41(6), 153-154. Dawid, A. P. 2000. Causal Inference Without Counterfactuals. Journal of the American Statistical Association, 95 (2), 407-448. Delfino, M., Persico, D. 2007. Online or face-to-face? Experimenting with different techniques in teacher training. Journal of Computer Assisted Learning, 23, 351-365. De Filippo, D., Casani, F., García-Zorita, C., Efraín-García, P., Sanz-Casado, E. 2012. Visibility in international rankings. Strategies for enhancing the competitiveness of Spanish universities. Scientometrics, 93, 949-966. Díaz-Méndez, M., Gummesson, E. 2012. Value co-creation and university teaching quality. Journal of Service Management, 23(4), 571-592. García, J.A., Rodríguez-Sánchez, R., Fernández-Valdivia, J., Torres-Salinas, D, Herrera, F. 2012. Ranking of research output of universities on the basis of the multidimensional prestige of influential fields: Spanish universities as a case of study. Scientometrics, 93, 1081-1099. García-Valcárcel Muñoz-Repiso, A., Tejedor Tejedor, F.J. 2009. Information and communication technologies in university teaching: Implications in European higher education space. International Journal of Human Sciences, 6(2), 684-696.