Schools´ External Evaluation: An analysis through national and international evaluation reports 1 Joana Sousa 2 José Augusto Pacheco 3 Universidade do Minho, Braga, Portugal Keywords Evaluation, Inspection, Globalization Research Question What the national and the international evaluation reports emphasize about the schools´ evaluation system in Portugal? Objectives To analyse data from national and international evaluation reports; To understand how the national and the international reports create an impact on the Portuguese schools´ evaluation system. Theoretical Framework In the globalization context evaluation, has been recognized as a key tool in education policy reform. By borrowing and lending policies (Steiner-Khamsi, 2012) there is a mindset that implies to respond to the market logic (McNamara & O’Hara, 2009; Smith, 2014). Transnational institutions such as the United Nations (UN), the Organization for Economic Cooperation and Development (OECD), the International Monetary Fund (IMF), the World Bank, the International Association for the Evaluation of Educational Achievement (IEA), the European Court of Human Rights (ECHR), the European Union (EU) and other variety of “border-crossing institutions that exert enormous influence on States and citizens around the globe” (Sperling, 2009, p.2) are increasing uniformity in the educational policies (Schwandt, 2009) reflecting it on the evaluation politics. Different European national models for the Schools’ External Evaluation (SEE) aroused based on international evaluation systems (European Commission, EACEA, Eurydice, 2015). Nowadays there are thirty-one education systems in Europe that put their schools under the spotlight through external and internal evaluations, a number which still is increasing (Puhl & Crosier, 2015). In Portugal, the Law No. 31/2002 of December the 20th defined the Portuguese system of SEE which defends that this process is a formative instrument that evaluates the quality of the schools. The school evaluation process is assured by the Portuguese 1 This work is funded by CIEd – Research Centre on Education, projects UID/CED/1661/2013 and UID/CED/1661/2016, Institute of Education, University of Minho, through national funds of FCT/MCTESPT. 2 Ph.D. student in Educational Sciences in the field of Curriculum Development at the Institute of Education at the University of Minho (Portugal) with a research studentship by the Portuguese Foundation for Science and Technology (FCT - SFRH/BD/93389/2013). Email:
[email protected] 3 Full Professor at Institute of Education in the University of Minho, Braga, Portugal. Email: [email protected]minho.pt
Inspectorate of Education and Science (IGEC) in collaboration with the Universities and it has, already, two evaluation cycles. The first Portuguese cycle of SEE (2006-2011) occurred normally and only in some public schools. As the result was considered positive, the process was extended to all national public schools in the second cycle (2012-2017), but with some changes based on the proposals made by the Portuguese Education Council (CNE) and implemented by Inspection, as occurs in other European countries (Ehren & Shakleton, 2016). The domains evaluated in the scope of the SEE process in the first cycle were five: i) results; ii) educational service performance; iii) organization and school management; iv) leadership and the capacity for self-regulation; v) school improvement. There was also a rating scale with four grades: very good; good; sufficient; insufficient. From the first to the second cycle changes were made and two of them were the adjustment from five to three evaluation domains (removing the school improvement domain and combining the leadership domain and the management domain in one) and switch a four-grade rating scale to a five-grade scale, including the “excellent” classification. The results of SEE have been published by national and international reports from different organizations. In Portugal, the IGEC has published three reports after the beginning of the SEE second cycle (IGEC, 2013; IGEC, 2015; IGEC, 2016). At the international level, the OECD has produced reports about evaluation (Santiago, et al., 2012; OECD, 2013), general overviews about the education field and in which evaluation it is focused (OECD, 2012; 2015a; OECD, 2016) and an outlook for education policy (OECD, 2015b). The Eurydice makes an overview of evaluation in Europe (European Commission/EACEA/Eurydice, 2015). These national and transnational recommendations are globally recognized and, consequently, create an impact on international and national structural policies, namely on the schools’ external evaluation system. Methodology To answer the research question, it was used a summative approach to content analysis (Krippendorff, 1990; Hsieh & Shannon, 2005; Bardin, 2013), based on reports’ unique perspectives and grounded in the actual data. The analytical process included the selection of the sample to be analysed that was developed by a computer literature search that had permitted to identify relevant sources using keywords in the OECD, the Eurydice, and the IGEC data base. In this research, it was only accepted reports there were published during the Portuguese SEE second cycle (2012-2017). The analysed documents were seven international reports (European Commission/EACEA/Eurydice, 2015; Santiago, et al., 2012; OECD, 2012; 2013; 2015a; 2015b; OECD, 2016), and three national reports from the IGEC (IGEC, 2013; IGEC 2015; IGEC, 2016). Data analysis started with computer-assisted searches of seven pre-determined terms in each report: efficacy; efficiency; quality; responsibility; inspection; accountability; success. When a term was identified in meaningful segments of data, determining its trustworthiness, the word frequency was registered by each report and compared to the total length of the respectively ten documents. The frequency of each term was calculated and compared to the total number of terms coded. The discussion of this study focused on exploring patterns in the data and to contextualize the codes by possible explanations (Morgan, 1993).
Findings Considering the analysis of the national and the transnational organizations´ reports about schools’ evaluation, we can conclude that education has become a target of evaluation. Data collected in the schools by, for instance, the school inspections, the school selfevaluations councils, the schools’ compliance with regional or national rules and regulations and the student examinations and assessments are the developing support of evaluations in the education sector (OECD, 2015a). As in many other countries, in Portugal it is used a combination of these mechanisms as a part of a larger system of accountability (OECD, 2015a). The national and international reports revealed that theories and practices constitute a pragmatic kind of globalization (Ball, 2012) which promotes a market logic (Smith, 2014) based on homogenization and standards of evaluation. According to the research, the international and national perspective is aligned with concepts that predetermine a future policy trajectory. As Ball (1997) argues, policy construction is influenced by the regulation that is exercised by transnational corporations. When comparing the coherence between the national and transnational discourses, the findings reflect the promise that Portugal assumed with the European Union on the Lisbon Council occurred in 2000 in which has accepted the commitment of helping the European Union in becoming “the most competitive and dynamic knowledge-based economy in the world” (European Parliament, 2000). Evaluation is becoming central in the education field (Steiner-Khamasi, 2012), and the focus on the quality concept, related to the SEE in the analysed documents, may reflect its relevance in both national and international education panorama. In Portugal, evaluation reports reveal that there is a concern about reaching a predetermined and globalized notion of education quality based on a specific profile of success, supported by the accountability that uses efficacy and efficiency to make the educational agents responsible for the schools’ external achievements. References Ball, S. (2012). Global Education Inc. New policy networks and the neoliberalism imaginary. New York: Routledge. Bardin, L. (2013). Análise de Conteúdo. Lisboa: Edições 70. Comissão Europeia/EACEA/Eurydice (2015). Garantia da Qualidade na Educação: Políticas e Abordagens à Avaliação das Escolas na Europa. Relatório Eurydice. Luxemburgo: Serviço de Publicações da União Europeia. Ehren, M. & Shakleton, N. (2016). Mechanisms of Change in Dutch Inspected Schools: Comparing Schools in Different Inspection Treatments. British Journal of Educational Studies, 64 (2), 185-213. DOI: 10.1080/00071005.2015.1019413 Hsieh, H. & Shannon, S. (2005). Three Approaches to Qualitative Content Analysis. Qualitative Health Research, 15 (9), 1277-1288. DOI: 10.1177/1049732305276687 IGEC (2013). Avaliação Externa das Escolas 2011-2012 — Relatório. Lisboa: InspeçãoGeral da Educação e Ciência.
IGEC (2015). Avaliação Externa das Escolas 2012-2013 — Relatório. Lisboa: InspeçãoGeral da Educação e Ciência. IGEC (2016). Avaliação Externa das Escolas 2013-2014 — Relatório. Lisboa: InspeçãoGeral da Educação e Ciência. Krippendorff, K. (1990). Content analysis: an introduction to its methodology. Newbury Park: The Sage. McNamara, G. & O’Hara, J. (2009). The importance of self-evaluation in the changing landscape of education policy. Studies in Educational Evaluation, 34, 173-179. DOI:10.1016/j.stueduc.2008.08.001 Morgan, D. (1993). Qualitative content analysis: A guide to paths not taken. Qualitative Health Research,3, 112-121 OECD (2012). Education Today 2013: The OECD Perspective. OECD Publishing. DOI: 10.1787/edu_today-2013-en OECD (2013). Synergies for Better Learning: An international perspective on evaluation and assessment. OECD Publishing. DOI: 10.1787/9789264190658-en OECD (2015a). Education at a Glance 2015: OECD Indicators. OECD Publishing. DOI: 10.1787/eag-2015-en OECD (2015b). Education Policy Outlook 2015: Making reforms happen. OECD Publishing. DOI: 10.1787/9789264225442-en OECD (2016). Education at a Glance 2016: OECD Indicators. Paris: OECD Publishing. DOI: 10.187/eag-2016-en Santiago, P. et al. (2012). OECD Reviews of Evaluation and Assessment in Education: Portugal 2012, OECD Publishing. DOI: 10.1787/9789264117020-en Schwandt, T. (2009) Globalizing influences on the western evaluation imaginary. In K. Ryan & J. Cousins (Ed.), The Sage International Handbook of Educational Evaluation (pp. 19-36). London: Sage. Smith, D. (2014). Wisdom responses to globalization. In W. Pinar (Ed.), International handbook of curriculum research (2nd ed.) (pp. 45-59). New York: Routledge. Sperling, V. (2009). Altered states: the globalization of accountability. Cambridge: Cambridge University Press. Steiner-Khamsi, G. (2012). Understanding policy borrowing and lending. Building comparative policy studies. In G. Steiner-Khamsi & F. Waldow (Eds.), World yearbook of education 2012. Policy borrowing and lending in education (pp. 517). London: Routledge.
