The effects of accountability on the allocation of school resources: Regression discontinuity evidence from Chile

Elacqua, Gregory; Jaimovich, Analía; Román, Alonso Working Paper The effects of accountability on the allocation of school resources: Regression discontinuity evidence from Chile IDB Working Paper Series, No. IDB-WP-1074 Provided in Cooperation with: Inter-American Development Bank (IDB), Washington, DC Suggested Citation: Elacqua, Gregory; Jaimovich, Analía; Román, Alonso (2019) : The effects of accountability on the allocation of school resources: Regression discontinuity evidence from Chile, IDB Working Paper Series, No. IDB-WP-1074, Inter-American Development Bank (IDB), Washington, DC, https://doi.org/10.18235/0001979 This Version is available at: https://hdl.handle.net/10419/208210 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. https://creativecommons.org/licenses/by-nc-nd/3.0/igo/legalcode The effects of accountability on the allocation of school resources: Regression discontinuity evidence from Chile Gregory Elacqua Analía Jaimovich Alonso Román IDB WORKING PAPER SERIES No IDB-WP-1074 Inter-American Development Bank Education Division October 2019 The effects of accountability on the allocation of school resources: Regression discontinuity evidence from Chile Gregory Elacqua Analía Jaimovich Alonso Román Inter-American Development Bank Education Division October 2019 Cataloging-in-Publication data provided by the Inter-American Development Bank Felipe Herrera Library Elacqua, Gregory M., 1972The effects of accountability on the allocation of school resources: regression discontinuity evidence from Chile / Gregory Elacqua, Analia Jaimovich, Alonso Román. p. cm. — (IDB Working Paper ; 1074) Includes bibliographic references. 1. Educational accountability-Chile-Econometric models. 2. Education-Chile-FinanceEconometric models. 3. Government aid to education-Chile-Econometric models. 4. Educational vouchers-Chile-Econometric models. I. Jaimovich, Analia. II. Román, Alonso. III. Inter-American Development Bank. Education Division. IV. Title. V. Series. IDB-WP-1074 http://www.iadb.org Copyright © 2019 Inter-American Development Bank. This work is licensed under a Creative Commons IGO 3.0 AttributionNonCommercial-NoDerivatives (CC-IGO BY-NC-ND 3.0 IGO) license (http://creativecommons.org/licenses/by-ncnd/3.0/igo/legalcode) and may be reproduced with attribution to the IDB and for any non-commercial purpose, as provided below. No derivative work is allowed. Any dispute related to the use of the works of the IDB that cannot be settled amicably shall be submitted to arbitration purs uant to the UNCITRAL rules. The use of the IDB’s name for any purpose other than for attribution, and the use of IDB’s logo shall be subject to a separate written license agreement between the IDB and the user and is not authorized as part of this CC-IGO license. Following a peer review process, and with previous written consent by the Inter-American Development Bank (IDB), a revised version of this work may also be reproduced in any academic journal, including those indexed by the American Economic Association’s EconLit, provided that the IDB is credited and that t he author(s) receive no income from the publication. Therefore, the restriction to receive income from such publication shall only extend to the publication’s author(s). With regard to such restriction, in case of any inconsistency between the Creative Commons IGO 3.0 Attribution-NonCommercialNoDerivatives license and these statements, the latter shall prevail. Note that link provided above includes additional terms and conditions of the license. The opinions expressed in this publication are those of the authors and do not necessarily reflect the views of the Inter-American Development Bank, its Board of Directors, or the countries they represent. Contact information: [email protected] 1 The effects of accountability on the allocation of school resources: Regression discontinuity evidence from Chile August 2019 Gregory Elacqua, Analia Jaimovich & Alonso Román Abstract This research examines the effect of accountability threats for low performing schools on resource allocation decisions and provides evidence that schools act with strategic behavior only when the accountability pressure is high. We used a generalization of a traditional regression discontinuity design, taking advantage of the sharp discontinuity in the Chilean accountability system’s ranking of schools based on performance measures, and of a unique school level expenditure data set, to make causal estimates of the effect of being ranked as “low-performing” on school spending decisions. The results indicate that, while first time low-performing schools do not change their resource allocation strategy, chronically underperforming schools are more likely to hire external technical pedagogical support and invest in teacher training that may help them boost achievement in the short and mediumterm and avoid sanctions. Keywords: School accountability, school spending, school finance, Chile, vouchers JEL Classification: I22, I28, H52 2 1. Introduction School accountability and high-stakes testing have been at the center of most major educational reform discussions over the last two decades. One innovation of accountability reforms is the use of student outcomes to evaluate teacher and school performance (Elmore et al., 1996; Figlio & Loeb, 2011; O’Day, 2002). By using different types of accountability systems, central governments set performance standards and choose between giving rewards to schools that meet the standards, and/or apply sanctions to the ones that fail to meet them. These rewards and sanctions can be explicit, through bonuses for example, or implicit through community pressure or signaling. Therefore, school accountability can work through direct government action or through the provision of information (Figlio & Loeb, 2011). Schools are often ranked or classified according to their performance and have a specific amount of time to improve their outcomes. If not, they face sanctions that range from mandatory improvement plans to closure. The implementation of accountability systems has generated a persistent debate on the effects of these systems on student performance. Advocates argue that accountability pressures should have positive effects on academic outcomes in low-performing schools. However, critics maintain that these improvements may not be explained by real progress in student learning, but rather by strategic behavior that schools develop as they internalize the incentives (see for example Koretz, 2017). While this type of discussion has focused mainly on achievement gains and “gaming” accountability incentives, studies on how these interventions modify school resource allocation decisions are scarce (Booher-Jennings, 2005). This paper contributes to this debate by analyzing the effects of accountability pressures on school spending decisions in Chile. The Chilean education system is an interesting case study to address this kind of questions since it presents a high-stakes accountability scheme within a school choice institutional arrangement. Chile is one of the few countries that, as part of a systemic reform introduced by the military regime in 1981, instituted a universal voucher program. Under this scheme, school quality was supposed to be assured by parent accountability. Similar to what Hirschman (1970) proposed for companies, when schools offer a lowquality education, parents have two options: they can leave the school (“exit”) or they can express their dissatisfaction (“voice”). In a competitive schooling market, choice advocates maintain that low-quality schools would disappear, because they will lose students as a result of the exit and voice mechanisms (Hirschman, 1970). However, in the mid-2000s, despite substantial increases in funding and parental choice, education achievement gaps compared to OECD countries continued to persist. In response to persistent low performance, in 2008 the Chilean Congress enacted the Subvención Escolar Preferencial 3 Law (Preferential School Subsidy, Ley SEP) that, among other changes, introduced a national system of accountability for schools. Similar to other school accountability systems, SEP established minimum performance standards and ranked schools based on their performance on a national standardized test and other indicators. It also established sanctions for low-performing schools, including closure when a school did not show adequate improvement. In this paper, we examine the effect of accountability threats for low performing schools on resource allocation. We use a regression discontinuity for our analysis, leveraging the sharp discontinuity in Chile’s designation of schools to in recovery status (low-performing) based on performance measures. The identification strategy allows us to make a causal estimate of the effect of being ranked as in recovery, in combination with the threat of sanctions and the stigma of being classified as chronically underperforming, on school spending. We find evidence that being assigned to the treatment (in recovery) led to strategic behavior by school owners in the allocation of resources only when the accountability pressure is high, i.e. when schools have underperformed systematically over the years. While first time low performing schools do not change their resource allocation strategy, chronically underperforming schools are more likely to hire external technical pedagogical support and invest in teacher training that may help them boost achievement in the short and medium-term. This paper is organized as follows. Section 2 lays out the theoretical framework and reviews the empirical literature on the effects of accountability pressures on different outcomes. Section 3 describes the school funding and the SEP accountability system in Chile. The next two sections discuss the methodology we employ and our data. Section 6 describes the RD design. Our results and final discussion are presented in sections 7 and 8. 2. Theoretical framework and literature Accountability mechanisms have been implemented in various educational systems around the world. The No Child Left Behind Law (2001) in the United States, the Education and Inspection Law (2006) in England, and the SEP Law (2008) in Chile are amongst the most-developed accountability systems, and consequently have been the subject of extensive academic research. In all these cases, the government established performance goals and sanctions for schools that fail to meet them. An important aspect of accountability systems is the information content that allows families, teachers and policy makers a more effective way to assess how successful a school has been in meeting the achievement goals (Figlio & Loeb, 4 2011). Thus, the identification, classification, and subsequent publication of school rankings are all key components of these accountability systems. The objective of these actions is to increase the supervision of low-performing schools by parents and the government and to increase the pressure on schools to improve outcomes (Jacob, 2005). Previous research shows that the mere identification of low-performing schools operates as a social stigma for its principals, teachers, and students, increasing pressure to improve performance (Goldhaber & Hannaway, 2004). Once low-performing schools are identified, different sanctions are often gradually introduced, with the ultimate consequence of school closure (Brady, 2003). The assumption is that closing chronically underperforming schools would operate as an incentive for other low-performing schools to improve under the threat of closure (Smarick, 2010). Many accountability studies have focused in school performance consequences (Dee & Dizon-Ross, 2017; Hanushek & Raymond, 2005; Dee & Jacob, 2011), and some have analyzed the effect of accountability pressures on school policies and practices. Rouse et al. (2013) and Elacqua et al. (2016) show that schools under high accountability pressures in Florida and Chile modified some of their internal practices and policies in educationally meaningful ways. Rouse et al. (2013) show that these changes explain performance gains in low performing schools in Florida. This is consistent with the findings of other studies in New York City, Chicago, Texas, which show that, after the introduction of accountability mechanisms, low-performing schools improved their test scores (Deming et al., 2016; Jacob, 2005; Rockoff & Turner, 2010). Critics counter that accountability pressures can also produce undesirable effects. First, given that the performance standards set by the government measure only certain subjects from the curriculum, researchers have documented that schools spend more time on subjects that are included in the accountability index (Hannaway & Hamilton, 2007; Koretz & Barron, 1998). For example, in Kentucky, where students are evaluated in fifth grade, 82% of fifth-grade teachers reported that they increased instruction time for math, compared to 14% of fourth-grade teachers (Stecher & Barron, 2001). Similar results were found in Washington, California, Florida, Georgia, North Carolina, and Pennsylvania (Deming et al. 2016; Hamilton et al., 2007; Stecher et al.,2000; Hannaway and Cohodes, 2007; Ladd and Zelli, 2002). Accountability pressures have also led teachers to try to “outsmart” standardized tests through various practices. First, some teachers alter the pool of students evaluated. For example, Figlio and Getzler (2006) find that some teachers reclassify low performing students as pupils with learning disabilities so that their scores are not counted in the assessment. Figlio (2006) finds that some schools suspend low-performing students the day of the test. Jacob and Levitt (2003) find that teachers in schools under accountability 5 pressure have a greater probability of helping students answer the tests. Pedulla et al. (2003) find that teachers provide 12% to 19% more time than stipulated for students to take the tests. There is also evidence that teachers pay more attention to students who are closer to surpassing the performance threshold established by the authorities, disregarding students who are far below or above the threshold (Booher-Jennings, 2005). In the case of Chile, specifically, most econometric research on the SEP law has focused on its effects on student outcomes. Some studies show that the SEP subsidy has improved student performance (Nielson, 2013; Navarro-Palau, 2017; Bos et al, 2017), while others find that it did not (Aguirre, 2017; Feigenberg et al., 2017). Most of these studies, however, analyze the SEP law’s general effects, regardless of the specific mechanism that may drive them. Indeed, the SEP law introduced several measures: an increase in the size of the voucher based on student characteristics, the mandatory development of School Improvement Plans, and an accountability mechanism that ranked schools based on their performance. Most of these studies cannot disentangle which of the specific mechanisms explain the results. This paper innovates and contributes to the literature in two ways. First, this research contributes to the international literature on school accountability by, instead of focusing on how accountability affects student performance, it focuses on how accountability pressures may alter school spending decisions. To the best of our knowledge, this is the first paper to explore this relationship. This is a particularly relevant issue in school systems like Chile’s where school owners and principals have significant autonomy over school budgets. Second, the paper contributes to the research on the SEP law in Chile by focusing specifically on the effects of the accountability mechanism on schools’ decisions, isolating its effects from other aspects of the SEP law. Thus, the paper analyzes the effects of the accountability system under the SEP law on schools’ spending decisions. Specifically, we analyze the effect of being classified as a low performing school on schools’ budget allocation. Faced with accountability pressures, schools may decide to respond in several ways. For instance, schools may increase investments in teachers or classroom support for teachers or spend less on inputs that may be less relevant to improve student performance. In contrast, they could also focus investments on non-classroom related activities, such as school uniforms or busing students or on expenditures that will improve the school’s image such as publicity or safety. The goal of this paper is to gain insight into the decisions school managers make when faced with accountability pressures. 12 database, collected by the School Inspection and Audit Agency of Chile. This database contains detailed income and expenditure information for 11,472 schools in 2014 and 11,424 in 2015, including school payroll and private donations. Our third dataset is the official school information record collected by the Ministry of Education, which contains information on school characteristics, such as student enrollment, school location, school curriculum, ownership status, and socio-demographic data, among other data. 4.2. Identification strategy For our identification strategy, we exploit the fact that the methodology used to rank Chilean schools in the SEP accountability system is based on a school’s position relative to a multiple set of variables and their respective thresholds. These variables include national standardized test scores, the number of students tested, the number of available measurements, and a set of indicators that measure other quality dimensions (e.g. student retention rates, student pass rates or teacher evaluation results). The multidimensional characteristic of the accountability ranking allows us to use a multivariate regressiondiscontinuity design (MRDD), where a combination of cutoffs attained in a number of variables determines treatment status (unlike traditional RDDs, where units are assigned to treatment and control conditions based on a single cutoff score on a continuous variable. See for example Papay et al. 2011; Reardon & Robinson, 2012; Wong et al., 2013). Methods to estimate average treatment effects with multiple assignment variables are based on regression models such as (Reardon & Robinson, 2012): 𝑌= 𝑚(𝑅1,𝑅2,…,𝑅𝑛)+∑𝜏𝑇 + 𝑋𝐵 + 𝑒, where {𝑅1,𝑅2,…,𝑅𝐽}∈ 𝑫 ⊂ 𝑹. 𝑅1,𝑅2,…,𝑅𝑛 correspond to the 𝑛 assignment variables and 𝑇 is a dummy variable indicating if unit 𝑖 is assigned to treatment 𝑘. The estimators of treatment effects 𝜏 differ in two important ways: i) the specification of the 𝑚 function and ii) the 𝑫 domain of observations used in estimating the model, which is a subset of the space formed by the 𝑛 assignment variables (𝑹). The inclusion of pretreatment covariates (𝑋) may increase the precision of the estimates, but is generally unnecessary, as the model is (1) 13 well identified without it (Lee & Lemieux, 2010). The choice of the functional form of 𝑚 may be important, especially when there are few observations near the frontier. In this case, it is necessary to use data further from the cutoff score and make assumptions about the functional form of the average potential outcome, but doing so increases the potential bias in the estimation. In other words, there is a trade-off between bias and precision. Reardon and Robinson (2012) present five estimation methods: response surface RD, frontier RD, fuzzy frontier RD, distance-based RD, and binding-score RD. In this paper, we use the binding-score method because it has advantages over other approaches when there are a lower number of observations and when multiple rating scores determine assignment to only two treatment conditions. The main advantage of this approach is that it allows the researcher to parsimoniously collapse scores from multiple assignment rules into a single assignment variable and therefore can use all the observations simultaneously in the estimation. The approach also generalizes well to MRDDs with more than two assignment variables and simplifies the analyses for estimating average treatment effects across multiple discontinuity frontiers, but it requires the assumption that the average treatment effect is the same at each boundary. This method has been used, for example, in the evaluation of NCLB in the United States (e.g. Gill et al., 2009). Other examples are found in Reardon et al. (2010) and Robinson (2011). One disadvantage is that it does not allow the estimation of frontier-specific effects, so we cannot test the existence of heterogeneous treatment effects9. Another disadvantage is that pooling units from different frontiers increases the heterogeneity of the outcome at the pooled cutoff, requiring a larger bandwidth for nonparametric estimates and increases the complexity of the functional form around the cutoff (Wong et al., 2013). The Binding Score method relies on the construction of a new assignment variable 𝑍 (binding-score) that sharply determines treatment assignment, so it assimilates to a traditional RDD. Let´s suppose that treatment assignment depends on two variables (R and M) and schools are assigned to a single treatment condition 𝑇 if they score below both cutoffs (𝑅≤ 𝑟 and 𝑀≤ 𝑚), and to the control condition 𝐶 if they score above either cutoff. Neither of these variables individually defines treatment allocation, but we can construct a new variable 𝑍, defined as the maximum between both assignment variables centered at its respective cutoff: 9 Based on the variables forming the binding score. 14 𝑍= 𝑚𝑎𝑥(𝑅 ,𝑀 ), where 𝑅 = 𝑅− 𝑟 and 𝑀 = 𝑀− 𝑚. By construction, 𝑇= 1 if 𝑍< 0 and 𝑇= 0 if 𝑍≥ 0 In this case, the problem becomes a traditional RDD and all the standard analytic methods can be used, defining 𝑍 as the assignment variable and zero as the cutoff. Although this transformation applies to the original assignment variables, Wong et al. (2013) show that this method estimates the same causal effect as alternative methods. We use the SEP ranking database to construct the binding score of the RDD model. First, we consider the rules under which schools are ranked in the 2015 SEP database. For instance, according to the SEP Law, schools without SIMCE data for 2 or more years or with less than 20 students taking the national test in fourth grade are not ranked, and thus, are excluded from our analytical sample. Second, we determine our treatment and control groups. The treatment group includes schools that are ranked as in recovery in 2015. The control group contains those schools that are ranked as emerging or autonomous in the 2015 classification (non-recovery). Third, we limit the school universe to those schools that could be classified in 2015, and that have reported their income and expenses information to the School Inspection and Audit Agency for the years 2014 and 2015. The total sample includes 62 (2,39%) in recovery schools and 2.534 (97,61%) non-recovery schools for 2015. Our binding-score (𝑍 ) is constructed from this final dataset by using the seven rating scores that determine assignment to the in recovery category. Details on the construction of the binding score variable are presented in Appendix 1. Table 1 presents the treatment and control group details along with the number of times schools have been classified as in recovery. (2) 15 Table 1: Autonomous, Emergent and in Recovery Schools Autonomous Emergent In Recovery Total Treatment & Control SEP Classification 2015 1,049 1,485 62 2,596 40.4% 57.2% 2.4% 100.0% In Recovery History Never in Recovery 1,040 1,359 0 2,399 99.1% 91.5% 0.0% 92.4% In Recovery once 6 72 19 97 0.6% 4.8% 30.6% 3.7% In Recovery twice 3 45 12 60 0.3% 3.0% 19.4% 2.3% In Recovery three times 0 9 17 26 0.0% 0.6% 27.4% 1.0% In Recovery four times 0 0 14 14 0.0% 0.0% 22.6% 0.5% Source: Ministerio de Educación de Chile and authors´ calculations 4.3. Variables To analyze school expenditure structures, we use data reported by schools for 2014 and 2015 captured at the National School Income and Expenditure database. We group expenditures into eight categories, each of which is used as an outcome in our regression models by analyzing the difference in percentages of school total expenditures allocated to the category in 2015, using the year 2014 as a baseline. Table 2 describes the following outcomes: Payroll expenses refers to the percentage of school total expenditures devoted to teacher and class assistant’s payroll, along with all related expenses from hiring and retiring processes, like nursery school expenses or social security and retirement funds. Teacher training and PME refers to the percentage of school total expenditures focused on quality improvements. These expenditures include teachers’ participation in professional development courses, seminars, or coaching; educational software; and external support for the development, implementation, and evaluation of the school’s improvement plan (PME). In Chile, this external support is mostly provided by ATEs. In general, the work of the ATEs focuses on providing training for school leadership teams, 16 assessing school improvement needs, providing advice on the development and implementation of the PMEs, etc. School administrators (private owners in the case of private schools, or municipalities in the case of public schools) can choose any ATE from a national registry of certified agencies. ATEs vary widely in the quality of the services they provide, and there is no specific guidance on the selection of ATE services. Pedagogical equipment expenses refer to the percentage of total expenditures devoted to purchasing technological aides for pedagogical activities, such as computers, interactive whiteboards, etc. Learning resources expenses refers to the percentage of total expenditures devoted to the acquisition of school pedagogical inputs such as school libraries, laboratories, evaluation tools, teacher guides, etc. School transportation refers to the percentage of total expenditures focused on school transportation, which may include the hiring of external school transportation services or the purchasing of school buses. This is an important expenditure item in the case of Chile given that schools receive government subsidies based on student attendance, rather than on student enrollment. School uniforms refers to the percentage of total expenditures used for school uniforms and clothing accessories like aprons. This expenditure item is optional for schools because the use of uniforms is not mandatory in Chile. Moreover, many of the schools ask the student’s family to purchase uniforms. Offering them as a benefit could help retain or attract students. Table 2: Descriptive Statistics for sample Variable 2014 Baseline USD 2014 (2) Var Mean Std Dev N Outcomes (1) School payroll expenses (%) 76.59% $938,103 1.05% 8.38% 2,596 Teacher training and PME expenses (%) 1.60% $19,597 -0.44% 2.11% 2,596 Pedagogical equipment expenses (%) 1.38% $16,903 -0.16% 1.86% 2,596 Learning resources expenses (%) 2.57% $31,478 -0.11% 2.42% 2,596 Transportation expenses (%) 1.08% $13,228 -0.10% 1.38% 2,596 School uniform expenses (%) 0.25% $3,062 0.03% 0.66% 2,596 School safety expenses (%) 0.11% $1,347 0.00% 0.33% 2,596 Publicity expenses (%) 0.02% $245 -0.01% 0.08% 2,596 Source: National School Directory of the Education Ministry of Chile and National Income and Expenditure database for 2015, collected by the School Audit Agency of Chile. (1) Percentages are calculated over Annual School Income (Including Public, Privat e, and Donations transfers) (2) USD are calculated over mean exchange rate 2014 17 School safety refers to the percentage of total expenditures invested in safety measures for the school. This could include security guards, alarm systems and/or cameras among other items. Security has been an increasing concern for families, and our hypothesis is that low performing schools could decide to invest in this category in order to be more competitive without having to substantially change staff and alter management and pedagogical practices. Finally, publicity refers to the percentage of total expenditures invested in advertising. This could include web page development, publicity campaigns, leaflets, and school open houses among others. This category may be important for schools to improve their image in a competitive market. Table 3: Descriptive Statistics for school characteristics Variable 2014 Baseline Std Dev N School Characteristics Public schools (%) 49.7% 2,596 Rural schools (%) 7.4% 2,596 Adult education (%) 8.6% 2,596 Special needs education (%) 2.6% 2,596 Students enrollment 590.7 392.4 2,596 SEP enrollment (%) 56.8% 17.8% 2,596 Number of classrooms 18.6 9.9 2,596 Number of teachers 34.8 17.1 2,596 Contracted Hours Teachers 2014 1221.8 646.6 2,596 SEP Contracted Hours Teachers 2014 44.9 72.9 2,596 Number of Assistants 22.5 13.25 2,596 Contracted Hours Assistants 2014 866.7 531.8 2,596 SEP Contracted Hours Assistants 2014 121.7 179.7 2,596 Student attendance 2014 83.8% 6.5% 2,594 Student SEP attendance 2014 83.2% 9.9% 2,592 Free disposable income (%) 53.8% 9.9% 2,596 Source: National School Directory of the Education Ministry of Chile and National Income and Expenditure database for 2015, collected by the School Audit Agency of Chile. In addition to our outcome variables , w e analyze several school characteristics. T hese include dummies indicating whether the school is a private voucher school or a municipal-public school, or whether the school is located in a rural area. We also analyze variables indicating the number of students enrolled in the school, the percentage of students that are priority students (low SES), the number of classrooms, the number and working hours of teachers and assistants, student attendance, and a variable indicating the proportion of free disposable income the school has (that is, the amount of income that is not earmarked for a specific 18 10 As explained before, the voucher income system relies on student attendance. expenditure category) among many other variables. We present a summary of the descriptive statistics for our treatment and control groups in Table 3. To test whether our treatment and control groups differ significantly in any of these school characteristics, we conduct a difference in proportion and a difference in means tests that compares the differences in each of these variables between in recovery (treatment) and non-recovery (control) schools within the 0.3 bandwidth. Table 4 indicates evidence of statistically significant differences on percentage of SEP enrollment (71.3% vs. 64%), where schools in recovery have a greater proportion of low SES students. No other significant differences were found between the groups. Beyond statistical significance, the data shows that (in means) in recovery schools are smaller in enrollment, and therefore, tend to have fewer classrooms and teachers. For income related variables, in recovery schools have lower attendance10 but almost equivalent free disposable income. Table 4: Testing differences in Recovery and Non-Recovery School groups characteristics Recovering Non-Recovering Difference in means / Proportion test Domain/Variable Mean N Mean N t / z p value Group Characteristics % of Public Schools 69.6% 46 71.2% 118 0.205 0.838 % of Rural Schools 2.2% 46 5.1% 118 0.828 0.407 % of Adult Education 10.9% 46 14.4% 118 0.597 0.550 % of Special (Disable) Education 6.5% 46 2.5% 118 -1.219 0.223 School Characteristics Mean of student enrollment 421.04 46 447.42 118 0.588 0.557 Mean of % SEP enrollment 70.5% 46 65.2% 118 -2.123 0.035* Mean of classrooms 14.98 46 15.46 118 0.374 0.709 Mean of Teachers 29.78 46 30.32 118 0.227 0.821 Mean of Contracted Hours Teachers 2014 1,041.70 46 1,047.08 118 0.060 0.952 Mean of SEP Contracted Hours Teachers 2014 62.61 46 40.78 118 -1.885 0.061 Mean of Assistants 19.41 46 19.58 118 0.101 0.919 Mean of Contracted Hours Assistants 2014 762.000 46 753.695 118 -0.126 0.900 Mean of SEP Contracted Hours Assistants 2014 111.15 46 78.03 118 -1.524 0.130 Mean of attendance 2014 75.3% 46 77.0% 118 1.421 0.157 Mean of SEP attendance 2014 76.4% 46 77.7% 118 1.063 0.289 Mean of Free disposable Income 2015 46.9% 46 49.1% 118 1.163 0.246 * p<0.05 ** p<0.01 *** p<0.001 Source: Authors’ calculations based on National School Directory and National School Income and Expense Database for 2015 19 4.4. RD Validity A key assumption of regression discontinuity analyses is that no agent can manipulate the assignment variable, thus falling on either side of the threshold could be considered random. While it is likely that schools would be motivated to score above the cutoff that places them in recovery status, it is unlikely that they can manipulate their ranking. To corroborate this empirically, we explore a standard group of tests for manipulation of the assignment. First, we plot the density function of the binding scores. Figure 2 demonstrates that there is no jump in the density after the cut point of zero. We find no evidence of bunching near the cutoff that could suggest assignment variable manipulation. Along with the two-step procedure test proposed by McCrary (2008) for discontinuity, the second stage estimates a local linear regression separately on both sides of the threshold. The test is implemented as a Wald test whose null hypothesis is that the discontinuity is zero. Table 5 presents the McCrary test of discontinuity and confirms the graphical evidence displayed in Figure 2. These tests fail to reject the null hypothesis of no discontinuity in our binding score and in every component. In sum, both the smoothness of the assignment variable’s distribution and the group and school covariate balance verify the causal assumptions of the RD design. Figure 2: McCrary Test for 2015 20 Finally, Lee and Lemieux (2010) maintain that researchers should test the continuity of the baseline covariates as an important part of assessing the validity of an RD design. We test for discontinuities in preexisting school´s characteristic prior to the classification, among them school SEP enrollment, student attendance and free disposable income, and found no discontinuities11. Table 5: McCrary Test Variable t p value Main Binding Score 2015 1.076 0.282 Detail psimce2013 0.368 0.713 psimce2012 0.410 0.682 psimce2011 0.471 0.638 p2502013 -0.595 0.552 p2502012 0.459 0.646 p2502011 0.518 0.604 Education Quality Index (ICE)2015 1.447 0.148 * p<0.05 ** p<0.01 *** p<0.001 Note: Each variable is centered on its respective cutoff and divided by its standard deviation Source: Authors’ calculations 5. Results We report regression discontinuity estimates of the effect of the 2015 in recovery classification on the decisions about school expenditure allocation across spending categories, within a bandwidth of 0.3 standard deviations relative to the binding score of zero that determined treatment status. We present the results with and without preexisting covariates in the following model that includes interactions with past SEP classifications: 𝑌 = 𝛼 + 𝜏𝑇 + 𝛽(𝑍 − 𝑧)+ 𝛽𝑇(𝑍 − 𝑧)+ 𝜏𝑇 + 𝜏𝑇𝑇 + 𝑋𝐵 + 𝜀 (3) Where 𝑌 is the expense outcome, 𝑇 takes the value of one if the school is classified as in recovery in 2015 and zero for non-recovery, 𝑇 is one if the school was classified as in recovery in 2012, 2013 or 2014 and 11 Tests and regressions available upon request 21 zero otherwise, (𝑍 − 𝑧) represents the distance from the school to the threshold of our assignment variable constructed with the binding-score method, 𝑋 represents the covariates, and 𝜀 is an error term with a normal distribution. We include as covariates a dummy for municipal-public schools (given management differences between public and private voucher schools) and the percentage of SEP enrollment at school. Table 6 presents the regression results for being classified as an in recovery school in the 2015 SEP classification, taking into account whether this was the first time the school was classified as in recovery, or if it had been previously ranked in this category. The first three rows for each expenditure category show results for our model without covariates, and the next three rows include them. For each expenditure category and model, table 6 reports the effect of being in recovery 2015, of being in recovery at least one other time, and the aggregation of these effects. For this last result, significance is calculated through a Wald test where the null hypothesis that the sum of the effects is equal to zero is tested. One of our main findings, that is consistent in subsequent analyses, is that being ranked as in recovery for the first time does not appear to change the way schools allocate their resources. For our eight expenditure outcomes there are no significant changes in the percentage of budget allocation to each spending category once the schools were publicly classified as in recovery in 2015. Nonetheless, schools that were previously classified as in recovery in either 2012, 2013 or 2014 responded to accountability pressures by investing in strategies to improve learning. The results show these schools are more likely to invest in teacher training programs and hire external support for class or school management (expenses contained in our “Teacher Training & PME” outcome). For the model without covariates, the budget shift is close to 1.6% of total expenditures and about 1.8% for the model with covariates. On the other hand, there is a significant negative effect on school payroll expenses when the school has been ranked more than once as in recovery. However, in contrast to the impact found for “Teacher Training & PME”, there is no significant joint effect of being classified as in recovery in 2015. Along with these results, we do not find any other statistically significant budget allocation change at the school level. The results suggest that in recovery schools do not react by investing in other quality measures such as pedagogical equipment or learning resources; or in student wellbeing measures, such as transportation or clothing. Finally, in recovery schools also do not appear to increase spending on safety and publicity measures. 28 Table 10: Robustness Check with false treatment schools. Model for 2014 - 2015 difference(1) School Payroll Teacher training & PME Treatment Threshold (Z) Z = 0.3 Sd Z = 0.5 Sd Z = 0.3 Sd Z = 0.5 Sd No covariates In recovery 2015 -0.0264 0.0332 0.00152 -0.0120 (0.0330) (0.0464) (0.0113) (0.0139) In recovery 2015 and before 0.0115 -0.0110 -0.00952 0.00504 (0.0192) (0.0219) (0.00653) (0.00654) Total Effect (3) -0.0149 0.0222 -0.008 -0.00696 With covariates (2) In recovery 2015 -0.0273 0.0322 0.00331 -0.0121 (0.0331) (0.0464) (0.0113) (0.0139) In recovery 2015 and before 0.00997 -0.00759 -0.00987 0.00540 (0.0192) (0.0220) (0.00654) (0.00659) Total Effect (3) -0.01733 0.02461 -0.00656 -0.0067 N 343 480 343 480 Standard errors in parentheses * p<0.10 ** p<0.05 *** p<0.01 (1) Outcomes are 2014 - 2015 difference expenses in percentages over Annual School Income (Including Public, Private, and Donations transfers). (2) The regression with covariate included dummies for Public School and % SEP enrollment (3) Effect sum between "in Recovery 2015" and "in Recovery 2015 and before". Significance Wald test for the hypothesis that the sum of both effects is 0. 29 6. Discussion One of today’s most controversial topics in education reform discussions is school accountability. Advocates argue that schools under accountability pressure have strong incentives to adjust internal practices and policies to improve student performance. Critics have countered that accountability pressures also produce undesirable effects such as teaching to the test, altering the composition of the testing pool, the overemphasis of tested material, and cheating by teachers. Accountability opponents have also argued that low performing schools will tend to focus on quick solutions that generate rapid improvements (e.g. test taking strategies) rather than on educational investments that produce longer term gains (e.g. teacher development). Skeptics are also concerned that, faced with accountability pressures, schools will have incentives to undertake “glitzy” reforms that focus on publicity and improving the school’s image. This is especially relevant in systems of school choice where parents may easily choose to exit low performing schools. While scholars have developed a substantial body of empirical research that has examined the effects of accountability on student achievement and school “gaming” of accountability incentives (e.g. Figlio & Loeb, 2011), there has been little attention paid to changes in school resource allocation resulting from school accountability. This is an important oversight since there is evidence that some resources are more likely to improve student performance than others. For example, there is a growing evidence that effective teachers can dramatically improve student achievement (e.g. Araujo et al., 2016). This dearth of research is mainly due to the lack of school level expenditure data. Our study seeks to contribute to this debate by analyzing a unique school expenditure data set in Chile, coupled with administrative data in a school accountability system. Our results indicate that low-performing schools respond timidly to the accountability pressures generated by the SEP Law. The findings show that, despite high fixed costs (teacher salaries, facilities, etc.), in recovery schools strategically focus a larger share of their variable spending on certain expenditures, but only when the accountability pressure is high. First-time in recovery schools do not show a resource allocation pattern different from similar schools just above the threshold. It is only when low-performing schools are ranked in recovery more than once that they change their resource allocation strategy, being more likely to allocate resources to professional development and to external technical assistance than similar schools just above the threshold and reducing expenses on teacher payroll. 30 Thus, recurrent low-performing schools, compared to their counterfactual, are more likely to focus spending on items that may potentially be linked to quality improvements. School administrators under systematic pressure are focusing resources on measures such as professional development and external technical assistance that aim to improve student performance in the short or medium-term. Interestingly, we also find that in recovery schools are not more likely than emerging schools to focus resources on inputs such as learning and pedagogical resources, student well-being (transportation and uniforms) and inputs that may improve the school’s image with parents such as publicity or school security. Recurrent low-performing schools seem to be responding in a way that is consistent with the design of the SEP accountability system. First-time low performing schools, in contrast, are not reacting to the accountability pressures. Whether this slow reaction is due to poor management capacity, or a perception of low risk of closing due to a single low performance classification is beyond the scope of this paper. Nevertheless, this finding highlights the importance of the design of accountability systems, because the deadlines, types of sanctions, communication strategy, and the assumptions made about the school improvement process are key in determining how schools under threat will target their resources. The literature on school improvement emphasizes the fact that low performing schools do not improve overnight; they take sometimes years to boost achievement levels. The slow reaction of first-time in recovery schools in the case of Chile calls attention to the need to critically analyze the design of accountability systems to ensure that these schools are not losing valuable time doing more of the same. Carefully targeted external support programs for first time low performing schools that recommend more effective resource allocation may be among the policy options to support these schools in this process. 31 References Aguinis, H., Gottfredson, R. K., & Culpepper, S. A. (2013). Best-practice recommendations for estimating cross-level interaction effects using multilevel modeling. Journal of Management, 39(6), 1490-1528. Araujo, M.C., P. Carneiro, Y. Cruz & N. Schady (2016) Teacher quality and learning outcomes in kindergarten. Quarterly Journal of Economics, March. Booher-Jennings, J. (2005). Below the Bubble: "Educational Triage" and the Texas Accountability System. American Educational Research Journal, 42(2), 231-268. Murnane, R., Waldman, M., Willett, J., Bos, M.S., & Vegas, E. (2017) The consequences of educational voucher reform in Chile, IDB working paper series N. 833. Brady, R. (2003). Can Failing Schools be fixed? Washington, DC: Thomas B. Fordham Foundation. Cortina, J. M. (2002). Big things have small beginnings: An assortment of “minor” methodological misunderstandings. Journal of Management, 28(3), 339-362. Dee, T., & Dizon-Ross, E. (2017). School performance, accountability and waiver reforms: Evidence from Louisiana (No. w23463). National Bureau of Economic Research. Dee, T. S., & Jacob, B. (2011). The impact of No Child Left Behind on student achievement. Journal of Policy Analysis and management, 30(3), 418-446. Deming D., S. Cohodes, J. Jennings & C. Jencks (2016) School Accountability, Postsecondary Attainment, and Earnings, The Review of Economics and Statistics, MIT Press, vol. 98(5), pages 848-862, Elacqua, G., M. Martinez, H. Santos & D. Urbina (2016) Short-run effects of accountability pressures on teacher policies and practices in the voucher system in Santiago, Chile School Effectiveness and School Improvement Vol. 27, Iss. 3, 2016 Elmore, R. F., Abelman, C. H., & Fuhrman, S. H. (1996). The new accountability in state education reform: From process to performance. In H. F. Ladd (Ed.), Holding schools accountable: Performance-based reform in education (pp. 65-98). Washington, DC: Brookings Institution. Figlio, D. N. (2006). Testing, crime and punishment. Journal of Public Economics, 90, 837-851. Figlio, D. N., & Getzler, L. S. (2006). Accountability, Ability and Disability: Gaming the System? In T.J. Gronberg, & D.W. Jansen (Eds.), Improving School Accountability (Advances in Applied Microeconomics, Volume 14) (pp.35-49). Bingley, UK: Emerald Group Publishing Limited 32 Figlio, D.N., & Loeb, S. (2011). School Accountability. In E. A.Hanushek, S. Machin, & L. Woessmann (Eds.), Economics of Education (pp. 383-421). The Netherlands: North Holland. Figlio, D. N., & Lucas, M. E. (2004). What’s in a Grade? School Report Cards and the Housing Market. The American Economic Review, 94, 591-604. Gill, B., Lockwood, J., Martorell, F., Setodji, C. M., Booker, K., Vernez, G., Birman, B., & Garet, M. S. (2009). An Exploratory Analysis of Adequate Yearly Progress, Identification for Improvement, and Student Achievement in Two States and Three Cities. (Technical Report U.S. Department of Education). Goldhaber, D., & Hannaway, J. (2004). Accountability with a kicker: Observations on the Florida A+ accountability plan. Phi Delta Kappan, 85, 598-605. Hamilton, L., Stecher, B., Marsh, J., McCombs, J., Robyn, A., Russel, J. N., & Barney, H. (2007). Implementing standards-based accountability under No Child Left Behind: Responses of superintendents, principals, and teachers in three states. Santa Monica, CA: RAND. Hannaway, J., & Cohodes, S. (2007). Trouble Even in Choice Paradise: NCLB in Miami-Dade County Public Schools. In F. M. Hess, & C. E. Finn (Eds.), No remedy left behind: Lessons from a half-decade of NCLB (pp. 244-264). Washington DC: American Enterprise Institute Press. Hannaway, J., & Hamilton, L. (2008). Performance-based accountability policies: Implications for school and classroom practices. Washington, DC: Urban Institute and RAND Corporation. Retrieved from http://www.urban.org/sites/default/files/alfresco/publication-pdfs/411779-Accountability-Policies.PDF Hanushek, E. A., & Raymond, M. E. (2005). Does school accountability lead to improved student performance? Journal of Policy Analysis and Management: The Journal of the Association for Public Policy Analysis and Management, 24(2), 297-327. Hirschman, A. O. (1970). Exit, voice, and loyalty: Responses to decline in firms, organizations, and states. Cambridge, MA: Harvard University Press. Imbens, G. W., & Lemieux, T. (2008). Regression discontinuity designs: A guide to practice. Journal of econometrics, 142(2), 615-635. Jacob, B. A. (2005). Accountability, incentives and behavior: the impact of high-stakes testing in Chicago Public Schools. Journal of Public Economics, 89, 761-796. Jacob, B. A., & Levitt, S. D. (2003). Rotten apples: An Investigation of the Prevalence and Predictors of Teacher Cheating. The Quarterly Journal of Economics, 118, 843-877. Koretz, D. (2017) The testing charade. Pretending to make schools better. Chicago: University of Chicago Press. 33 Koretz, D. M., & Barron, S. I. (1998). The Validity of Gains in Scores on the Kentucky Instructional Results Information System (KIRIS). Ladd, H. F., & Zelli, A. (2002). School-based accountability in North Carolina: The responses of school principals. Educational Administration Quarterly, 38, 494-529. Lee, D. S., & Lemieux, T. (2010). Regression Discontinuity Designs in Economics. Journal of Economic Literature, 48, 281–355 McCrary, J. (2008). Manipulation of the running variable in the regression discontinuity design: A density test. Journal of Econometrics, 142, 698-714. O'Day, J. (2002). Complexity, Accountability, and School Improvement. Harvard Educational Review, 72, 293-329. Papay, J. P., Willett, J. B., & Murnane, R. J. (2011). Extending the regression-discontinuity approach to multiple assignment variables. Journal of Econometrics, 161, 203-207. Pedulla, J., Abrams, L., Madaus, G., Russell, M., Ramos, M., & Miao, J. (2003). Perceived Effects of StateMandated Testing Programs on Teaching and Learning: Findings from a National Survey of Teachers. Boston, MA: National Board on Education Testing and Public Po Reardon, S. F., Arshan, N., Atteberry, A., & Kurlaender, M. (2010). Effects of failing a high school exit exam on course taking, achievement, persistence, and graduation. Educational Evaluation and Policy Analysis, 32, 498–520. doi:10.3102/0162373710382655 Reardon, S. F, & Robinson, J. (2012). Regression discontinuity designs with multiple rating-score variables. Journal of Research on Educational Effectiveness, 5, 83–104. doi:10.1080/19345747.2011.609583 Robinson, J. P. (2011). Evaluating criteria for English learner reclassification: A causal-effects approach using a binding score regression discontinuity design with instrumental variables. Educational Evaluation and Policy Analysis, 33, 267–292. doi:10.3102/0162373711407912 Rockoff, J., & Turner, L. (2010). Short-run impacts of accountability on school quality. American Economic Journal: Economic Policy, 2, 119–147. doi:10.1257/pol.2.4.119 Rouse, C. E., Hannaway, J., Goldhaber, D., & Figlio, D. (2013). Feeling the Florida heat? How lowperforming schools respond to voucher and accountability pressure. American Economic Journal: Economic Policy, 5(2), 251-81. Smarick, A. (2010). The turnaround fallacy. Education Next, 10, 20–26. Retrieved from http://educationnext.org/theturnaround-fallacy/ 34 Stecher, B. M., & Barron, S. (2001). Unintended consequences of test-based accountability when testing in "milepost" grades. Educational Assessment, 7(4), 259-281. Stecher, B. M., Barron, S. L., Chun, T., & Ross, K. (2000). The effects of the Washington state education reform on schools and classrooms. Santa Monica, CA: RAND Corporation. Wong, V., Steiner, P., & Cook, T. (2013). Analyzing regression-discontinuity designs with multiple assignment variables: A comparative study of four methods. Journal of Educational and Behavioral Statistics, 38, 107–141. doi:10.3102/1076998611432172 35 Appendix 1. Construction of the Binding Score Table A1 shows the seven variables that define if a school is classified as Recovery or Not In-Recovery for 2015. Using these variables, we are able to establish a unique continuous rating score (𝑍) that determine the year’s final school classification. In order to have all variables on a same scale, each variable was centered on the respective predefine cutoff and then divided by their standard deviation. For 2015 we transform its first variable 𝑝𝑠𝑖𝑚𝑐𝑒 to 𝑝𝑠𝑖𝑚𝑐𝑒  : 𝑝𝑠𝑖𝑚𝑐𝑒  =𝑝𝑠𝑖𝑚𝑐𝑒 − 220 𝜎 For 2015, we construct the first rule to be pre-classified as Recovery, which is that in two years the school average SIMCE score must be under the cutoff value, and that less than 20% of their students reach a higher score of 250. Therefore, we calculate first the maximum between 𝑝𝑠𝑖𝑚𝑐𝑒 and 𝑝250 for each year: 𝑍 = max (𝑝𝑠𝑖𝑚𝑐𝑒  ,𝑝250  ) 𝑍 = max𝑝𝑠𝑖𝑚𝑐𝑒  ,𝑝250   𝑍 = max (𝑝𝑠𝑖𝑚𝑐𝑒  ,𝑝250  ) So, as the variables are centered to their respective cutoff, if the school doesn´t meet any of the two rules in a year, the constructed value will be negative 𝑍 < 0. To capture the “two bad year” rule, we build a fourth binding value that takes the second maximum between 𝑍 , 𝑍  and 𝑍 . For 2015 is: 𝑍 = 𝑠𝑒𝑐𝑜𝑛𝑑max (𝑍 ,𝑍 ,𝑍 ) Thus, 𝑍  indicates if a school is classified as recovery according to SIMCE results. If 𝑍 < 0 then 𝑝𝑠𝑖𝑚𝑐𝑒 < 220 and 𝑝250 < 0.2 in two years, and therefore the school meets the requirements to be pre-classified as recovery. The opposite is true when 𝑍 ≥ 0. The final classification rule incorporates the Education Quality Index. If a school`s Index is below the 10th percentile, then it will be classified as Recovery, thus: 𝑍 = 𝑚𝑖𝑛 (𝑍 ,𝐼𝐶𝐸  ) This variable (binding-score) perfectly determines treatment assignment. If 𝑍< 0, 𝑖 school is classified as Recovery. If 𝑍≥ 0 it will be classified as Non-Recovery. 36 Table A1: Variables that define if a school is classified as Recovery or Non-Recovery in year 2015 Variable Description Cutoff Binding Scorey psimce2013 School average SIMCE fourth grade year 2013 score 220 psimce2012 School average SIMCE fourth grade year 2012 score 220 psimce2011 School average SIMCE fourth grade year 2011 score 220 p2502013 School average proportion of students who have scored over 250 points in SIMCE fourth grade year 2013 score 20% p2502012 School average proportion of students who have scored over 250 points in SIMCE fourth grade year 2012 score 20% p2502011 School average proportion of students who have scored over 250 points in SIMCE fourth grade year 2011 score 20% Education Quality Index (ICE)2015 Index that combines average SIMCE score of previous 3 years1 (70%) with complementary indicators2 (30%) 10th percentile Notes: The SIMCE variables takes scores in Math, Language and Science tests. 1 Average of years 2013, 2012 and 2011 2 Complementary indicators are: student´s approval and retention rates, teacher`s and family involvement is school project, school´s educational innovation, teacher´s working conditions, and public teacher`s evaluation.