Evaluation of inflation forecasting models in Guatemala

Castañeda-Fuentes, Juan Carlos et al. Working Paper Evaluation of inflation forecasting models in Guatemala IDB Working Paper Series, No. IDB-WP-882 Provided in Cooperation with: Inter-American Development Bank (IDB), Washington, DC Suggested Citation: Castañeda-Fuentes, Juan Carlos et al. (2018) : Evaluation of inflation forecasting models in Guatemala, IDB Working Paper Series, No. IDB-WP-882, Inter-American Development Bank (IDB), Washington, DC, https://doi.org/10.18235/0001266 This Version is available at: https://hdl.handle.net/10419/208107 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. https://creativecommons.org/licenses/by-nc-nd/3.0/igo/legalcode Evaluation of Inflation Forecasting Models in Guatemala Juan Carlos Castañeda-Fuentes Héctor Augusto Valle-Samayoa Juan Carlos Catalán-Herrera Juan Carlos Arriaza-Herrera Mariano José Gutiérrez-Morales Carlos Eduardo Castillo-Maldonado Douglas Napoleón Galindo-Gonzáles Guisela Hurtarte-Aguilar Edson Roger Ortiz-Cardona IDB WORKING PAPER SERIES Nº IDB-WP-882 July 2018 Department of Research and Chief Economist Inter-American Development Bank July 2018 Evaluation of Inflation Forecasting Models in Guatemala Juan Carlos Castañeda-Fuentes Héctor Augusto Valle-Samayoa Juan Carlos Catalán-Herrera Juan Carlos Arriaza-Herrera Mariano José Gutiérrez-Morales Carlos Eduardo Castillo-Maldonado Douglas Napoleón Galindo-Gonzáles Guisela Hurtarte-Aguilar Edson Roger Ortiz-Cardona Banco de Guatemala Cataloging-in-Publication data provided by the Inter-American Development Bank Felipe Herrera Library Evaluation of inflation forecasting models in Guatemala / Juan Carlos CastañedaFuentes, Héctor Augusto Valle-Samayoa, Juan Carlos Catalán-Herrera, Juan Carlos A rriaza-Herrera, Mariano José Gutiérrez-Morales, Carlos Eduardo Castillo-Maldonado, Douglas Napoleón Galindo-Gonzáles, Guisela Hurtarte-Aguilar, Edson Roger OrtizCardona. p. cm. — (IDB Working Paper Series ; 882) Includes bibliographic references. 1. Inflation (Finance)-Guatemala-Forecasting. 2. Inflation (Finance)-GuatemalaEconometric models. I. Castañeda-Fuentes, Juan Carlos. II. Valle-Samayoa, Héctor A ugusto. III. Catalán-Herrera, Juan Carlos. IV. Arriaza-Herrera, Juan Carlos. V. Gutiérrez-Morales, Mariano José. VI. Castillo-Maldonado, Carlos Eduardo. VII. Galindo-Gonzáles, Douglas Napoleón. VIII. Hurtarte-Aguilar, Guisela. IX. OrtizCardona, Edson Roger. X. Inter-American Development Bank. Department of Research and Chief Economist. XI. Series. IDB-WP-882 Copyright © Inter-American Development Bank. This work is licensed under a Creative Commons IGO 3.0 AttributionNonCommercial-NoDerivatives (CC-IGO BY-NC-ND 3.0 IGO) license (http://creativecommons.org/licenses/by-nc-nd/3.0/igo/ legalcode) and may be reproduced with attribution to the IDB and for any non-commercial purpose, as provided below. No derivative work is allowed. Any dispute related to the use of the works of the IDB that cannot be settled amicably shall be submitted to arbitration pursuant to the UNCITRAL rules. The use of the IDB's name for any purpose other than for attribution, and the use of IDB's logo shall be subject to a separate written license agreement between the IDB and the user and is not authorized as part of this CC-IGO license. Following a peer review process, and with previous written consent by the Inter-American Development Bank (IDB), a revised version of this work may also be reproduced in any academic journal, including those indexed by the American Economic Association's EconLit, provided that the IDB is credited and that the author(s) receive no income from the publication. Therefore, the restriction to receive income from such publication shall only extend to the publication's author(s). With regard to such restriction, in case of any inconsistency between the Creative Commons IGO 3.0 Attribution-NonCommercial-NoDerivatives license and these statements, the latter shall prevail. Note that link provided above includes additional terms and conditions of the license. The opinions expressed in this publication are those of the authors and do not necessarily reflect the views of the Inter-American Development Bank, its Board of Directors, or the countries they represent. http://www.iadb.org 2018 1 Abstract1 Forecasting the inflation path is an important task for central banks in an inflation targeting regime. Therefore, central banks must continuously evaluate the forecasting accuracy of the models used to generate inflation forecasts. This paper evaluates the performance of most of the models that produce either unconditional or conditional inflation forecasts at Banco de Guatemala. The paper is divided in two main parts. The first part evaluates the forecasting accuracy and efficiency of the models that produce unconditional forecasts, applying different measures as normality test, root mean square error (RMSE), mean percentage errors (MPA), and tests as Diebold-Mariano, Pesaran-Timmerman, Giacomini-Rossi, as well as both weak and efficiency tests. The second part evaluates the conditional forecasting performance of the central bank’s main macroeconomic models by generating sample forecasts in hindsight for different scenarios for exogenous and some endogenous variables. We find evidence supporting the claim that the time series models perform better in forecasting inflation for short time horizons while the structural macroeconomic models perform better in medium and long time horizons. JEL classification: C53 Keywords: Economic forecasting, Forecasting accuracy, Forecasting efficiency 1 This research project was developed at Banco de Guatemala’s Economic Research Department, within the framework of the CEMLA’s Joint Research Program 2017 that was coordinated by the Central Bank of Colombia. The authors thank counseling and technical advice provided by the Financial Stability and Development (FSD) Group of the Inter-American Development Bank in the process of writing this document. The opinions expressed herein are those of the authors and do not necessarily reflect the views of CEMLA, the FSD group, the Inter-American Development Bank or Banco de Guatemala. 2 1. Introduction The purpose of this research is to evaluate the main inflation forecasting models in use by the Banco de Guatemala. Banco de Guatemala adopted an inflation targeting regime to conduct its monetary policy in 2005. To have accurate and unbiased forecasts of inflation is a matter of paramount importance in this monetary policy framework. Currently Banco de Guatemala employs an array of models to this end. This set of models encompasses time series econometrics models, macroeconomic models and forecast combination technics. Each model provides different information on the future evolution of inflation according to its nature. The forecasts are a key element in the conduct of monetary policy. This paper is divided into two main parts. In the first part we evaluate the forecasting performance of all the models, both time-series and structural macroeconomic models, in terms of accuracy and bias. This is done based on the typical measures of evaluation, i.e., the root mean square error, the mean percentage error and normality tests. In addition, we assess the capability of each model to forecast a change in the direction of inflation using the Pesaran-Timmerman test. Also, we use the Diebold and Mariano test to compare the predictive accuracy of the forecasts between two competing models, and the Giacomini-Rossi tests to examine the performance of two competing models in the presence of possible instabilities over different evaluation rolling windows. Finally, we test the efficiency of the models with both weak and strong efficiency tests. In the second part, we assess the performance of the macroeconomic models in producing conditional forecasts. We evaluate the forecasting model’s performance by generating in-sample forecasts in hindsight for different scenarios for exogenous and some endogenous variables. Some of those scenarios involve historically observed values for the exogenous and some endogenous variables. In general, we have found that the time series models perform better for short time horizons, while the structural macroeconomic models perform better for longer time horizons, as expected. The paper is organized as follows. In Section 2, we focus on the unconditional forecast evaluation. In Section 3, we look at the conditional forecast evaluation. Finally, in Section 4, we present the main conclusions of the research paper. 3 2. Unconditional Forecasts Evaluation 2.1 The Models 2.1.1 Indicator Variable (IV) This is the inflation forecast employed at Banco de Guatemala as the main short-term forecast in the conduct of its monetary policy. It is estimated by the Department of Macroeconomic Analysis and Forecasts. The forecast is based on a set of time series models plus economic analysts’ expert knowledge about the inflation series. In particular, economists complement the inflation forecasts generated by the models with considerations of trend, seasonality and temporary shocks in addition to the overall domestic and foreign economic conditions. 2.1.2 Forecast Combination through Individual Time-Varying Efficient Weights (EFP) This model is based on assessing past forecast performance efficiency at each of eight quarters ahead, according to an algorithm called the Efficient Forecast Path (EFP), described in Castillo and Ortiz (2017). It consists of a five-step method to construct a weighting scheme for model combination that includes an ex ante endogenous procedure for model trimming. The model is discussed in Appendix 1. 2.1.3 Average Macroeconomic Models (AMM) The Economic Research Department (DIE2) uses two macroeconomic models to make forecasts: The Semi-Structural Macroeconomic Model 4.0.1 (MMS) and the Macroeconomic Structural Model (MME). The MMS 4.0.1 is a reduced form model, characterized by a difference-equations system representing the transmission mechanisms of monetary policy for quarterly data. The current version (MMS 4.0.1) is part of the set of non-micro funded general equilibrium macroeconomic models used in Banco de Guatemala that have evolved from the first version launched in 2006. The MME is a medium-scale DSGE model, built within the New Keynesian framework. It features a financial accelerator à la Bernanke et al. (1999) and other frictions relevant for emerging or developing economies, such as deviations from the law of one price and the uncovered interest parity. 2 DIE is the Spanish acronym of the Economic Research Department. 4 2.1.4 Inflation Expectations There are two measures of inflation expectations available at Banco de Guatemala. Both are updated monthly, and they are described below. Panel of Economic Experts Banco de Guatemala conducts a monthly survey with an independent panel of private-sector experts on economics, finance and business in Guatemala. The objective is to gather information on their perceptions of the future trend of inflation, economic activity and confidence in the economy. Economic Research Department The Economic Research Department carries also an inflation expectations survey among its members. 2.2 Evaluation Methodology In this section, we describe the methodology chosen to evaluate the forecasting accuracy of the unconditional forecasts models. We consider three competing models: Indicator Variable, Average Macroeconomic Models, and the Efficient Forecasting Path model. Also, we evaluate the inflation expectations of both the Panel of Economic Experts and Economic Research Department of the Central Bank. 2.2.1 Evaluation Sample First, we use quarterly data to evaluate the accuracy of the forecasts of the quantitative models. Each quarter, the competing models forecast inflation for the next eight quarters, starting at 2011q1 and finishing at 2017q2 in the case of both the Indicator Variable (IV) and the Average Macroeconomic Models (AMM). However, the Efficient Forecasting Path model forecasts inflation for the next eight quarters from 2014q2 to 2017q2. Then, we classify the forecasts of each quantitative model into different time-horizons (one, two, three, four, and eight quarters) to evaluate the forecasting performance at each time horizon and determine which model is best for forecasting the inflation patterns in every one of them. The evaluation sample is rather short, though, especially in the case of the EFP’S forecasts, for which 5 there are only 13 quarters. We additionally evaluate how well the models predict the December inflation rate for both the current and the next year. Second, we use the inflation expectations monthly data from both the economic experts’ panel and the Economic Research Department to examine the accuracy of the inflation expectations to predict December inflation at one and two-year horizons. The sample of forecasting errors is from 2015q7 to 2017q6 in the case of the one-year horizon and from 2016q7 to 2017q6 in the case of the two-year horizon. 2.2.2 Forecasting Evaluation In this study, we evaluate the key properties of the forecasting errors; i.e., we perform precision, accuracy, directional change, and efficiency tests to evaluate which model is best to predict the inflation path. We start by examining the residuals distribution of the forecast to check for normality and skewness. We then use the Root Mean Square Error (RMSE) measure to find which model best predicts the inflation rate, and the Diebold–Mariano test to examine if the difference between the MSE of the two competing models is statistically significant at least at the 10% level of significance. Also, we use the Giacomini–Rossi Fluctuation test in order to examine the forecasting accuracy between two competing models over the forecasting horizons with rolling windows of four. With this test, we examine whether the forecasts of one model are better than the other’s in every rolling window or whether there is a change (fluctuation) in the accuracy. In addition, we use the Pesaran–Timmerman test to determine whether the forecasts of the models can correctly predict the directional change of inflation, and Finally, we test the efficiency of the forecasts by examining bias, autocorrelation, and (weak and strong) efficiency tests. The tests are discussed in detail in Appendix 2. 3. Main Results This section compares forecasting performance to predict the inflation patterns of the Average Macroeconomic Models (AMM), the Indicator Variable model (IV), and the Efficient Forecasting Path framework (EFP). Also, we evaluate the forecasting performance of the respective inflation expectations generated by the Economic Experts Panel (EEP) and the Economic Research Department (DIE). 12 Second, we evaluate the performance of the AMM model. We observe from table A3.16, Appendix 3, that we can reject the null hypothesis of independence in the case of the one-, two-, three-, six-, seven-, and eight-quarter horizons; therefore, the model can predict successfully the directional change of inflation in both the short and the long run. 3.2.4 Giacomini–Rossi Fluctuation Test In this section, we use the Giacomini and Rossi fluctuation test to examine the performance of two competing models in the presence of possible instabilities. We use the IV model as the benchmark model in the case of the quantitative model, and the inflation expectations forecasts of the economic experts panel in the case of expectations forecasts. The test is only used in some of the forecasting horizons due to data availability. We set the rolling windows equal to four quarters to make the forecasting analysis. Also, we use graphic analysis to examine the performance of the forecasts of the two competing models in the different rolling windows to see whether there is a fluctuation in the forecasting accuracy. We explain in the test in detail in Appendix 2. Forecasting Horizons • Quantitative Models First, we start with the comparison of the forecasting performance of the AMM and the IV models with the use of the GR statistic. We define the loss function for the two models as follows: 𝐿𝐿𝑡𝑡�𝜃𝜃 �𝑗𝑗−ℎ,𝑅𝑅,𝛾𝛾�𝑗𝑗−ℎ,𝑅𝑅�=𝑀𝑀𝑆𝑆𝑀𝑀𝐴𝐴𝐴𝐴𝐴𝐴,𝑡𝑡 –𝑀𝑀𝑆𝑆𝑀𝑀𝐼𝐼𝐼𝐼,𝑡𝑡 (1) If the loss function turns out to be negative, we conclude that the forecasts of the AMM model are more accurate than those of the IV model. On the other hand, if the loss function turns out to be positive, the forecasts of the IV model are better at predicting inflation than those of the AMM model. From Table A3.17, Appendix 3, we observe that we reject the null hypothesis of equal forecasting accuracy over every forecasting horizon since the GR statistic is higher than its critical value. It means that one model displays better predictive ability to forecast inflation in at least one period of time. 13 The fluctuation test allows us to determine whether there is a change in predictability between the forecasts of the IV and the AMM models. Unlike the Diebold–Mariano test, the GR test compares the performance of both models in different sub-samples through rolling windows. In Table A3.18, Appendix 3, we deliver a summary of the Giacomini–Rossi fluctuation test for each forecasting horizon. We observe that the forecasts of the IV model are more accurate than those of the AMM model one step ahead. However, it seems that the forecasts of the AMM model better predict the inflation patterns at fourand eight-quarter horizons. In addition, in Appendix 4 we explain in further detail the results of the test with a graphic analysis. Second, we compare the forecasting accuracy between the EFP and the IV model with the use of the Giacomini-Rossi fluctuation test. The loss function between the two models is as follows: 𝐿𝐿𝑡𝑡�𝜃𝜃 �𝑗𝑗−ℎ,𝑅𝑅,𝛾𝛾�𝑗𝑗−ℎ,𝑅𝑅�=𝑀𝑀𝑆𝑆𝑀𝑀𝐸𝐸𝐸𝐸𝐸𝐸,𝑡𝑡 –𝑀𝑀𝑆𝑆𝑀𝑀𝐼𝐼𝐼𝐼,𝑡𝑡 (2) In the same way than the previous analysis, if the loss function is negative then the forecasts of the EFP model are more accurate than those of the IV model. On the other hand, if the loss function is positive, the forecasts of the IV model are better at predicting inflation. From Table A3.19, Appendix 3, we see that the null hypothesis of equal accuracy is rejected in every forecasting horizon since the GR–statistic is higher than the critical value. It means that, at least in one period, one model generates more accurate forecasts of inflation. In Table A3.20, Appendix 3, we show a summary of the Giacomini–Rossi test for each forecasting horizon. We observe that the forecasts of the IV model are more accurate in almost all the evaluation sample in each forecasting horizon. Therefore, the forecasts of the IV model seem to be more accurate than the EFP model at all forecasting horizons. In Appendix 4, we explain in more detail the results of the test with a graphic analysis. • Inflation Expectations We use the Giacomini-Rossi Fluctuation test to examine the performance of the inflation expectations from the Economic Research Department (DIE) and the Economic Experts Panel (EEP), where we assume the latter as a benchmark model. We estimate the loss function of the inflation expectations of the DIE and the EEP. 𝐿𝐿𝑡𝑡�𝜃𝜃 �𝑗𝑗−ℎ,𝑅𝑅,𝛾𝛾�𝑗𝑗−ℎ,𝑅𝑅�=𝑀𝑀𝑆𝑆𝑀𝑀𝐸𝐸𝐸𝐸𝐸𝐸,𝑡𝑡 –𝑀𝑀𝑆𝑆𝑀𝑀𝐷𝐷𝐼𝐼𝐸𝐸,𝑡𝑡 (3) 14 In the qualitative forecasts, we have two scenarios: one and two-year forecasting horizons. Also, if the loss function is negative then the inflation expectations of the EEE are more accurate than those of the DIE. On the contrary, if the loss function is positive, the inflation expectations of the DIE are better at predicting inflation. From Table A3.21, Appendix 3, the GR statistic is higher than its critical values at both one and two-year horizons. Therefore, we reject the null hypothesis of equal accuracy. It means that one model outperforms the other in at least one period of the time. In Table A3.22, Appendix 3, we show a summary of the Giacomini-Rossi test. From the table, there is a fluctuation of the forecasting accuracy of the inflation expectations between the two models in the case of the one-year horizon. However, the inflation expectations of the EEE better predict the inflation patterns in the case of the two-year horizon. 3.2.5 Forecasting Efficiency In this section, we test the efficiency of the unconditional forecasts of both the quantitative and the qualitative models. These tests will complement the previous tests of accuracy and precision in order to evaluate the overall forecast performance of the models. Before evaluating the efficiency of the forecasts, we start examining the absence of bias and serial autocorrelation. Then, we use the weak and strong efficiency test to evaluate the efficiency of the forecasts. We explain the tests in more detail in Appendix 2. Unbiasedness Test • Forecasting Horizons o Quantitative Models We start with the quantitative models: AMM, IV, and EFP. From Table A3.23, Appendix 3, we observe the unbiasedness tests (the p-values are shown in the parenthesis) for the quantitative models. If the p-value is lower than 0.05 then we reject the null hypothesis and the model is biased in the forecasting horizon. The forecasts of the AMM model are unbiased from one to third forecast horizons because we cannot reject the null hypothesis that the constant term is equal to zero. However, in the case of four and eight quarters ahead, the forecasts are biased. 15 We observe a similar pattern in the case of the forecasts of the IV model. In this case, the forecast are unbiased only for one and two quarters ahead. After that, the forecasts are biased because we reject the null hypothesis of unbiasedness. Finally, the forecasts of the EFP model are only biased at the one quarter ahead horizon. After that, they are unbiased. In sum, the forecasts of the AMM and the IV become biased in the middle and long run. o Inflation Expectations Table A3.24, Appendix 3 shows shown the unbiasedness tests for the inflation expectations of both the panel of economic experts (EEP) and the economic research department (DIE). Again, if the p-value is lower than 0.05, the forecasts are biased. We observe that both the EEP and DIE models are unbiased in the case of the one-year horizon because we cannot reject the null hypothesis at the 5 percent level of confidence. However, the models are biased in the case of the two-year horizon. Autocorrelation Test • Forecasting Horizons o Quantitative Models We start with the quantitative models: AMM, IV, and EFP. The null hypothesis of the test is that there is no autocorrelation. If the p-value is lower than 0.05, we reject the null hypothesis of no autocorrelation. Both the unbiasedness and the autocorrelation test represent a preliminary weak efficiency test. From table A3.25, Appendix 3, we show the autocorrelation tests where the p-values are shown in parenthesis. In the case of the forecasting errors of the AMM model, we observe autocorrelation in all forecasting horizons because we reject the null hypothesis at the 5 percent level of significance. Similarly, the forecasting errors of the IV model show serial correlation in almost all forecasting horizons with the exception of one step ahead. In the case of the EFP model, there is evidence of autocorrelation only in the case of one quarter ahead, after that; there is no serial correlation in the remaining forecasting horizons 16 o Inflation Expectations We continue with the forecasts of the inflation expectations. From A3.26, Appendix 3, we observe that both the inflation expectations of the DIE and the EEP, shows autocorrelations in all forecasting horizons. Weak Efficiency Tests • Forecasting Horizons o Quantitative Models We start with the quantitative models: AMM, IV, and EFP. Table A3.27, Appendix 3 presents appears the F-statistic of the weak efficiency test, where the p-values appear in parenthesis. From the second column, we observe that the forecast of the AMM model satisfies the weak efficiency hypothesis only in the case of one quarter ahead, because we cannot reject the null hypothesis (see Appendix 2). In the case of the remaining forecasting horizons, the model does not satisfy the weak efficiency hypothesis. Then, we analyze the weak efficiency of the IV forecasts. From the third column, we observe that forecasts of the model satisfy the weak efficiency only in the case of one and two forecasting horizons. Finally, we evaluate the weak efficiency of the EFP forecasts. From the fourth column, we observe that the forecasts of the model satisfy the weak efficiency in almost all forecasting horizons with the exception of four quarters ahead. In sum, the forecast of the EFP are more efficient than those of the other models based in the results of the weak efficiency test. Also, the forecast of the AMM and the IV are weakly efficient in the short run. o Inflation Expectations From Table A3.28, Appendix 3, we observe the weak efficiency tests of the inflation expectations. In this case, the forecasts of both the EEP and DIE model do not satisfy the weak efficiency test at the 5 percent level of significance in all forecasting horizons. Therefore, the forecasts are not weakly efficient. 17 o December Evaluation We test for weak efficiency only in the case of the AMM and the IV models because of data availability (see table A3.29, Appendix 3). In the case of the forecasts of the AMM model, we cannot reject the null hypothesis of weak efficiency only in the case of two and three quarters ahead. The null hypothesis is rejected in the remaining forecasting horizons. Also, the forecasts of the IV model satisfy the weak efficiency tests in five out of eight forecasting horizons. In sum, the forecasts of the IV model are more efficient than those of the AMM model in evaluating the December predictability of inflation. Strong Efficiency Tests • Forecasting Horizons We perform the strong efficiency test for the two econometrics models: IV and EFP in this subsection. The null hypothesis establishes that a new variable (which is not included in the econometric models) does not explain the forecasting error. Therefore, the rejection of the null hypothesis means that the errors are strongly efficient. Otherwise, if the null hypothesis is not rejected, then the inclusion of a new variable can add information to improve the forecasts and it may be included in them. We consider five variables in logs of the structural model of the Central Bank of Guatemala to make the test: Consumption, index of raw materials, investment, government spending, and credit. We start with the IV model; the tests are shown in table A3.30, Appendix 3. In the second column, it appears the coefficient of consumption. We observe that we cannot reject the null hypothesis at the 5 percent level of significance in the case of one and two quarters ahead. Therefore, the forecasts are strongly efficient at those horizons. However, from three to eight quarters ahead, consumption does explain the forecasting error, which means that they are not strongly efficient in those horizons. Similarly, in the third column, the null hypothesis is not rejected at 5% level of significance. Therefore, the forecasts are strongly efficient in those horizons. However, from three to eight quarters ahead, the inclusion of the raw material index can improve the forecasts, which mean that they are not strongly efficient. 18 Then, in the fourth column, we observe that the null hypothesis is not rejected in one, two and three quarters ahead, which mean that the forecasts are strongly efficient in those horizons. However, from four to eight quarters ahead, investment explains the forecasting errors, therefore; the forecasts are not strongly efficient. After that, in the fifth column, we observe that the null hypothesis is not rejected in all forecasting horizons, which means that the forecasts are strongly efficient, and the inclusion of government spending will not improve them. Finally, in the sixth column, we observe that the forecasts are strongly efficient from one to three quarters ahead. However, from four to eight quarters ahead, the inclusion of credit can improve the forecasts, which mean that they are not strongly efficient in those horizons. We continue with the EFP model; the tests are shown in Table A3.31, Appendix 3. We observe that we reject the null hypothesis in the case of one quarter ahead for the five variables, which mean that the forecasts of the IV model are not strongly efficient and the inclusion of the consumption, raw material index, investment, government spending, and credit can improve the forecasts in this forecasting horizon. However, the forecasts are strongly efficient in the case of the remaining forecasting horizons for the five variables, because we cannot reject the null hypothesis. In sum, the forecasts of the IV model are strongly efficient in the short run but lose the efficiency in the long run. Also, the forecasts of the EFP model are strongly efficient from two quarters ahead in all structural variables. • December Evaluation In the case of the evaluation of December, we perform the test only for the IV model due to data availability. Table A3.32, Appendix 3 shows the tests for the eight forecasting horizons. We observe that we cannot reject the null hypothesis for all forecasting horizons in the case of the raw material index, investment, government spending, and credit, at 5% level of significance, which means that the forecast are strongly efficient. However, in the case of consumption, we cannot reject the null hypothesis in all forecasting horizons with the exception of three quarters ahead, which means that the forecasts are strongly efficient in most of the forecasting horizons. 19 4. Conditional Forecast Evaluation 4.1 The Models 4.1.1 The Semi-Structural Macroeconomic Model 4.01 (MMS 4.0.1) The MMS 4.0.1 is a reduced form model, characterized by a difference-equations system, representing the transmission mechanisms of monetary policy for quarterly data. The current version (MMS 4.0.1) is part of the set of non-micro funded general equilibrium macroeconomic models used at Banco de Guatemala (Banguat) that have evolved from the first version launched in 2006. The MMS 4.0.1 was built on the basis proposed by Berg, Karam and Laxton (2006a and 2006b), who provided a practical guide to non-micro funded DSGE models and its implementation for central banks. In this regard, the MMS 4.0.1 is a semi-structural model (nonmicro funded) for a small, open economy, where monetary authorities operate policy within an inflation-targeting framework and implement monetary policy through a Taylor-type rule. All variables in the model are specified in annual growth rates. The MMS 4.0.1 has 40 equations (and 40 variables), of which 28 (70 percent) are endogenous and 12 (30 percent) are exogenous variables. The MMS 4.0.1 delivers forecasts for both core inflation and headline inflation. The model is currently used for producing inflation and monetary policy interest rate forecasts that are inputs for Banco de Guatemala’s monetary policymaking process. Variables that display higher volatility are transformed through a moving sum (average) scheme in order to reduce that volatility and avoid possible outliers. We thus obtain smoothed series. 4.1.2 Macroeconomic Model of Inflation Forecast for Guatemala (PIGU) The second model (PIGU) is also a semi structural macroeconomic model, very similar to the MMS 4.0.1. Variables in PIGU are also expressed as annual rates of change. There are three main differences between PIGU and MMS 4.0.1. are the following: the set of exogenous variables, the exogenous variables’ volatility, and the type of inflation. First, the set of exogenous variables: though some exogenous variables are common to both models, others are not. For example, foreign inflation in MMS 4.0.1 is the US Core-PCE inflation, while in PIGU is US Headline CPI inflation. 20 Second, the exogenous variables’ volatility: many MMS 4.0.1’s exogenous variables are smoothed (four-quarter averages), while PIGU uses quarterly variables. Finally, the type of inflation: MMS 4.0.1 forecasts both core and headline inflation, while PIGU forecasts headline inflation only. PIGU is currently available to all the central bank’s staff, through a custom-made interface. The interface is a tool built on MATLAB that allows the user to become familiar with the equations of the PIGU model, to create different forecasting scenarios based on alternative paths for any particular variable (endogenous or exogenous), and to modify the calibration of the model and visualize the corresponding effects on the conditional forecasts. It also shows the dynamic effects of different shocks (impulse-response functions). PIGU’s interface is designed to be a user-friendly technological tool, which does not require programming or modeling skills from the user. 4.1.3. Macroeconomic Structural Model (MME) The structural model is a medium scale DSGE model, built within the New-Keynesian framework. It features a financial accelerator à la Bernanke et al. (1999) and other frictions relevant for emerging or developing economies, such as deviations from the law of one price and the UIP. It is a model of heterogeneous agents; households supply labor services to entrepreneurs. They consume domestic and foreign goods, constitute deposits in domestic currency, take foreign debt and collect remittances from abroad. Firms, operating in a perfectly competitive market, assemble differentiated varieties to produce the home (or domestic) homogeneous final good. There are other firms producing the intermediate good, operating in a monopolistic competitive market; they buy a homogeneous wholesale good from entrepreneurs to differentiate it, and produce a particular variety. When these firms decide to change their prices, they face adjustment costs, à la Rotenberg (1982), introducing nominal price rigidities into the model. Entrepreneurs use three inputs to produce the wholesale good: capital, labor and imported raw materials. They buy capital from capital producing firms using their own wealth and loans granted by banks, since they are not able to self-finance their entire capital purchases. The financial sector is comprised by private banks divided into two activities: narrow banks that carry out passive operations gathering deposits from households and retail banks using those deposits to grant loans to entrepreneurs. 21 There is also a central bank setting the short-term interest rate—the policy rate—according to a Taylor-type rule and a central government carrying out unproductive spending. Some parameters of the model are calibrated, and a subset is estimated using Bayesian technics with quarterly data. Bayesian estimation was performed using Dynare. The first sample runs from 2001Q1 to 2010Q4. Then we add one quarter at a time, until we reach 2017Q1, the last observed data used to forecast corresponds to 2017Q2. Bayesian estimation deals with outliers by assigning a small probability (in the tails of the posterior distribution) to the parameter values that could generate such extreme values. Observable variables were introduced (for estimation) as stationary series. They were transformed, in accordance with the model, as the first difference of the natural logarithm, of seasonal adjusted data. In the case of the consumer price index (from which inflation rate is derived), there are no structural breaks. 4.2 Evaluation Methodology The quality of any variable’s conditional forecasts depends on two elements: i) the performance of the forecasting model (as such) and ii) the quality of the forecasting model inputs on which the forecasts are conditioned (e.g., the quality of the exogenous variables’ forecasts). We evaluate the forecasting model’s performance by generating in-sample forecasts in hindsight for different scenarios for the exogenous variables and for some endogenous variables as well. Some of those scenarios involve historically observed values for the exogenous and some endogenous variables, to evaluate forecasts as if we had the best possible forecast for these variables and thus eliminate one source of error. In the case of the semi-structural models (MMS and PIGU), we plug in, for each forecasted period, the historically observed values of exogenous and endogenous variables. In the case of the structural model (MME), exogenous variables are represented by stochastic processes, typically of an autoregressive nature. Therefore, alternative scenarios are only conditioned by historically observed values of two endogenous variables: inflation and output. 28 Appendix 1. Efficient Forecasting Path (EFP) The method described in this appendix was first developed in Castillo and Ortiz (2017). It is designed to generate an efficient out-of-sample forecast path (EFP) for a finite time series, based on a variety of forecasting models. The following sections summarize its methodology, the models and data employed to generate inflation forecasts. 1. Methodology Let us assume that 𝒚𝒚𝑡𝑡is an N-size vector containing a finite time series, which we are interested in forecasting for h periods ahead of the sample size. To perform this task we can use k models of the following types: ARIMA, OLS, SWLS, VAR, and VEC. In addition, let 𝒚𝒚𝑡𝑡 𝑤𝑤 be an s-size vector containing an ordered subsample of the original series, such that there could be a total of w continuous-size-s windows that can be derived from the original sample. Based on this information, the EFP algorithm can be computed through the following five steps. Step 1: Rolling regressions The first step consists of estimating rolling regressions of sample size s for each model k, so as to generate in-sample forecasts for h periods ahead. Step 2: Absolute forecast errors and model weights A different weight, 𝑞𝑞𝑠𝑠+ℎ,𝑘𝑘 𝑤𝑤 is assigned to each model k according to its performance (or forecast accuracy) to project each period s+h. Such a weight is equivalent to the inverse absolute forecast error of each model expressed as a ratio of the summation overall absolute forecast error inverse values generated by every model k when forecasting each s+h period at the specific window w: 𝑞𝑞�𝑠𝑠+ℎ,𝑘𝑘 𝑤𝑤=𝑓𝑓𝑓𝑓 �𝑠𝑠+ℎ,𝑘𝑘 𝑤𝑤 𝐸𝐸𝐼𝐼 �𝑠𝑠+ℎ 𝑤𝑤. Notice that each element of 𝑞𝑞�𝑡𝑡,𝑘𝑘 𝑤𝑤 has a value between zero and one. The higher its value, or the closer is it to one, the more accurate its forecast of s+h, and the more important would be model k when forecasting outside the sample size. Step 3: Average weight for each model For each model k, it is computed a simple average of weights, overall rolling regression windows. This term is denoted as 𝑞𝑞�𝑠𝑠+ℎ,𝑘𝑘 , and its value fluctuates between zero and one. 29 Step 4: Average weight distributions and endogenous trimming The last step involves getting rid of the models whose forecast efficiency, denoted by its average weight 𝑞𝑞�𝑆𝑆+ℎ,𝑘𝑘, is too low relatively to the rest of models. This is done by computing the mean and standard deviation for all weights that forecast the same period s+h, so as to obtain an average weight distribution, which allows us to select models based on above-average performance. Through this endogenous trimming method we could get rid of (set to zero) those average weight values at the lower extreme of each distribution. The proportion of models, whose average weights were set to zero, represented those who were below two standard deviations above the mean, so we ended up with the top 5 percent performers. Hence, we also renormalized each weight, 𝑄𝑄 �𝑠𝑠+ℎ,𝑘𝑘, so that the overall summation is still add up to one. Step 5: Efficient Forecast Path The final step in the algorithm is to generate out-of-sample forecasts for h periods ahead for each model k, and to weight each forecast by each model’s normalized weight 𝑄𝑄 �𝑠𝑠+ℎ,𝑘𝑘. Therefore, each weighted forecast, say 𝑄𝑄 �𝑠𝑠+ℎ,𝑘𝑘𝑦𝑦�𝑁𝑁+ℎ,𝑘𝑘, can be interpreted as the contribution of model k to 𝑦𝑦𝑡𝑡’s forecast of period 𝑁𝑁+ℎ. Hence, the summation overall k models is equivalent to a single forecast for 𝑦𝑦𝑡𝑡, and it is defined as the Efficient Forecast Path (EFP) of 𝑦𝑦𝑡𝑡. 2. Forecasting Models In order to proceed with the forecast combination we created a set of 1,380 models to forecast Guatemalan inflation. In particular, we employed five types of models: ARMA (p, q), Ordinary Least Squares (OLS), Stepwise Least Squares (SWLS), Vector Autoregression (VAR), and Vector Error Correction (VEC) models. Different lag and variable combinations were performed for each kind of model. All variables were transformed to their logarithmic form, and rolling estimation through moving windows was performed for each model specification. 2.1 Autoregressive Moving Average (ARMA) Models ARMA(p,q) models are univariate representations that express a variable to be modeled (𝑦𝑦𝑡𝑡) as a function of its own lags (p), as well as lags (q) of the error term (𝜀𝜀𝑡𝑡). Let Ω𝐴𝐴𝑅𝑅𝐴𝐴𝐴𝐴 = 𝐴𝐴𝐴𝐴𝑀𝑀𝐴𝐴(1,0),𝐴𝐴𝐴𝐴𝑀𝑀𝐴𝐴(2,0), … 𝐴𝐴𝐴𝐴𝑀𝑀𝐴𝐴(𝑝𝑝, 0),𝐴𝐴𝐴𝐴𝑀𝑀𝐴𝐴(1,1), … 𝐴𝐴𝐴𝐴𝑀𝑀𝐴𝐴(𝑝𝑝,𝑞𝑞), be the set of models 30 estimated per country. Therefore, the total number of models contained in such a set is equal to Ω𝐴𝐴𝑅𝑅𝐴𝐴𝐴𝐴 = 2p(1+q), where p and q varied per country according to data availability. We considered values for p and q of 12 and 11, respectively. Hence, Ω𝐴𝐴𝑅𝑅𝐴𝐴𝐴𝐴 is composed of 288 models 2.2 Ordinary Least Squares (OLS) Models The OLS models that we employed are classical econometric representations of variables as a function of their past values, and one or more independent variables and their lags. Let us assume that Ω𝑂𝑂𝑂𝑂𝑆𝑆 is the set of all OLS models estimated, which contains all possible combinations of multivariable models, along with all possible combinations of lags. In this case, the total number of OLS models contained in Ω𝑂𝑂𝑂𝑂𝑆𝑆 equals to 5𝑝𝑝0+ 4𝑝𝑝1+ 6𝑝𝑝2+ 4𝑝𝑝3+𝑝𝑝4. The sub-index in the lag expression denotes the number of inflation fundamentals included in each estimation. By letting 𝑝𝑝0,𝑝𝑝1,𝑝𝑝2,𝑝𝑝3 and 𝑝𝑝4 to be equal to 31, 21, 15, 12, and 10, respectively, it would imply that Ω𝑂𝑂𝑂𝑂𝑆𝑆 contains 387 (=155+84+90+48+10) models. 2.3 Stepwise Least Squares (SWLS) Models SWLS is an iterative algorithm proposed by Efroymson (1960) to automatically obtain the best fit of OLS regressions.5 Each model representation is identical to those of the previous section, but the final results differ, since the SWLS algorithm was established to select just those regressors whose p-value was lower or equal to 0.05. The set of all SWLS models estimated per country is equal to the OLS set (Ω𝑆𝑆𝑆𝑆𝑂𝑂𝑆𝑆 =Ω𝑂𝑂𝑂𝑂𝑆𝑆). 2.4 Vector Autoregression (VAR) Models Unrestricted VAR models are systems of equations that express each variable as a function of its own past values, and lags of the remaining variables in the system. We tried all combinations from two to five variables (domestic IPC and its four fundamentals), and different lag specifications. The set of all VAR models estimated per country, Ω𝐼𝐼𝐴𝐴𝑅𝑅, contained 4𝑃𝑃2+ 6𝑃𝑃3+ 4𝑃𝑃4+𝑃𝑃5 models. As before, the sub-index in the lag expression denotes the number of inflation fundamentals included in each VAR model. By letting 𝑃𝑃2,𝑃𝑃3,𝑃𝑃4 and 𝑃𝑃5 to be equal to 21, 15, 12, and 10, respectively, it would imply that Ω𝐼𝐼𝐴𝐴𝑅𝑅 is composed of 232 (=84+90+48+10) models. 5 Derksen and Keselman (1992), and Burnham and Anderson (1998) provide a description of the algorithm and describe some advantages and disadvantages of this methodology. 31 2.5 Vector Error Correction (VEC) Models We estimated equilibrium VEC models for one cointegrating relationship through the Johansen procedure. As with the previous case, combinations for all possible variable and lag specifications were performed. Although the procedure to compute the number of models resembles the VAR case, the number of lags considered to estimate each VEC model specification was lower: 8, 6, 4, and 2, respectively. Hence, Ω𝐼𝐼𝐸𝐸𝑉𝑉 is composed of 86 (=32+36+16+2) models. 3. Data Guatemalan inflation was estimated and forecast based on the models described above, using quarterly data from 1995Q1 to 2017Q1. The data set also included information for four wellidentified inflation fundamentals (Castillo, 2014): i) US Inflation; ii) Nominal Exchange Rate (Q/US$); iii) Real Money Supply; and iv) Real Banking Credit. As the United States is Guatemala’s main trading partner, US price fluctuations, along with Q/US$ variations are transferred quickly and almost fullly to domestic prices. Moreover, changes in domestic money supply and banking credit affect domestic inflation indirectly through their effect on domestic GDP.6 Following Clements and Hendry (1999), data were neither deseasonalized nor detrended in order to avoid missing important forecasting information. Finally, for empirical purposes all variables were transformed to their log form. The results obtained are described in the following section. Appendix 2. Forecasting Evaluation Tests In this appendix, we explain in detail the different tests considered to evaluate the forecasting performance of the unconditional models. 1. Tests of the Residuals We examine the key statistical properties of the forecasting errors: Normality and Skewness. First, the normality of the forecasting errors is evaluated with the Jarque–Bera test defined as 𝐽𝐽𝐽𝐽=�𝑛𝑛−𝑘𝑘−1 6��𝑆𝑆2+ 1 4(𝐶𝐶−3)2� (A2.1) 6 A quarterly series for Guatemalan GDP is not available for the whole period under consideration. 32 where 𝑛𝑛 is the number of observations, 𝑘𝑘 is the number of repressors, 𝑆𝑆 is the sample skewness, and 𝐶𝐶 is the sample kurtosis The null and the alternative hypotheses of the test are the following: 𝐻𝐻0: The forecast errors follow a standard normal distribution. 𝐻𝐻1: The forecast errors do not follow a standard normal distribution. Second, the skewness allows us to know whether the forecasts overestimate or underestimate the inflation patterns. 2. Root Mean Square Error and Mean Percentage Error We consider the Root Mean Square Error (RMSE) and the Mean Percentage Error (MPE) to test the precision of the forecasts of each model. The model with the lowest RMSE is considered the best. The RMSE is defined as follows: 𝐴𝐴𝑀𝑀𝑆𝑆𝑀𝑀=�1 𝑛𝑛∑(𝑦𝑦𝑖𝑖−𝑦𝑦�𝑖𝑖)2 𝑛𝑛 𝑡𝑡=1 2 (A2.2) The MPE is defined as follows: 𝑀𝑀𝑃𝑃𝑀𝑀=100 𝑛𝑛∑𝑦𝑦𝑖𝑖−𝑦𝑦 �𝑖𝑖 𝑦𝑦𝑖𝑖 𝑛𝑛 𝑡𝑡=1 (A2.3) where yi is the value of inflation, y�i is the inflation forecast, and n is the number of forecasting evaluation periods. The RMSE gives us an initial evaluation about the precision of the models to forecast inflation. After that, we use the Diebold-Mariano test to examine whether the difference between the MSE is statistically significant or not to determine which model has better forecasting accuracy. 3. Diebold-Mariano Test We use the Diebold-Mariano (DM) test developed by Diebold and Mariano (1995) to compare the predictive accuracy of two competing models in both quantitative models and inflation expectations. 33 The null and the alternative hypotheses are the following: 𝐻𝐻0: 𝑔𝑔(𝑒𝑒1𝑡𝑡 )−𝑔𝑔(𝑒𝑒2𝑡𝑡)= 0 𝐻𝐻1: 𝑔𝑔(𝑒𝑒1𝑡𝑡 )−𝑔𝑔(𝑒𝑒2𝑡𝑡)≠0 where g(eit ) is the loss function associated with the forecasting error term of each model. In our paper, we assume the mean square error as the loss function. The null hypothesis establishes that the forecasts have the same forecasting accuracy. If we reject the null hypothesis, the model with the lower mean square error is the best at forecasting inflation. The Diebold Mariano statistic (DM) is defined as 𝐷𝐷𝑀𝑀12,𝑡𝑡=𝑑𝑑12 𝜎𝜎𝑑𝑑12 → 𝒩𝒩(0,1) (A2.4) where d12 =(e1t )−g(e2t) and σd12 is a consistent estimate of the standard deviation of the DM statistic. The statistics follows a standard normal distribution with mean zero and standard deviation equal to 1. 4. Pesaran-Timmerman Test We also use the Pesaran – Timmerman (PT) test developed by Pesaran and Timmerman (1993) to examine the ability of the forecasts to predict the direction of the change of inflation in the case of the quantitative models as well as those of the inflation expectations. The null and alternative hypotheses of the test are the following: 𝐻𝐻0: The forecasts are not able to predict the change of inflation. 𝐻𝐻1: The forecasts are able to predict the change of inflation. The Pesaran – Timmerman Statistic is defined as: 𝑆𝑆𝑛𝑛=𝐸𝐸 �−𝐸𝐸 �∗ ⌊𝐼𝐼 �(𝐸𝐸 �)−𝐼𝐼 �(𝐸𝐸 �∗)⌋12 � → 𝒩𝒩(0,1) (A2.5) where 𝑃𝑃 �= 1 𝑛𝑛∑𝐼𝐼(𝑦𝑦𝑡𝑡 𝑛𝑛 𝑡𝑡=1 ,𝑥𝑥𝑡𝑡), in which 𝑦𝑦𝑡𝑡 represents the change of the consumer price index, 𝑥𝑥𝑡𝑡 represents the change of inflation forecasts of every model, and P � represents the proportion of times where both the change of inflation and the change of the forecast of inflation go in the same direction either increasing or decreasing. 34 We additionally have the following definitions: 𝑃𝑃 �∗= 𝑃𝑃 �𝑦𝑦𝑃𝑃 �𝑥𝑥+ �1−𝑃𝑃 �𝑦𝑦��1−P �x� (A2.6) P �y= 1 n∑I(yt n t=1 ) (A2.7) P �x= 1 n∑I(xt n t=1 ) (A2.8) where P �y and P �x represents the proportion of the positive changes of inflation and the forecasts of every model.; I(∙) = �1 if (∙) > 0 0 otherwise (A2.9) where I(∙) is an indicator variable which takes the value of 1 if the change is positive and 0 otherwise; and 𝑉𝑉 ��𝑃𝑃 ��=1 𝑛𝑛�𝑃𝑃 �∗�1−𝑃𝑃 �∗�� (A2.10) 𝑉𝑉 ��𝑃𝑃 �∗�=1 𝑛𝑛�2𝑃𝑃 �𝑦𝑦−1�2𝑃𝑃 �𝑥𝑥�1−𝑃𝑃 �𝑥𝑥�+1 𝑛𝑛�2𝑃𝑃 �𝑥𝑥−1�2𝑃𝑃 �𝑦𝑦�1−𝑃𝑃 �𝑦𝑦�+ 4 𝑛𝑛2𝑃𝑃 �𝑦𝑦𝑃𝑃 �𝑥𝑥�1−𝑃𝑃 �𝑦𝑦��1−𝑃𝑃 �𝑥𝑥� where V ��P �� and V ��P �� are the variance of P � and P �∗ respectively. 5. Giacomini – Rossi Fluctuation Test In addition, we use the Giacomini-Rossi Fluctuation test developed by Giacomini and Rossi (2010) to examine the performance of two competing models in the presence of instabilities. In this test, we can analyze the predictive accuracy of the models in different subsamples by using rolling windows to determine if the behavior of the forecasts is the same across the complete sample. The null and alternative hypotheses of the test are: The loss function is defined as the difference between the mean square errors of the two competing models as follows: ∆Lt�θ �j−h,R,γ�j−h,R�=MSE1,t −MSE2,t= 0 (A2.11) The GR – Statistics is defined as Ho ∶E�∆Lt(θ �j−h,R,γ�j−h,R)�= 0 for all t = R + h, … … … T 𝐻𝐻1: 𝑀𝑀�∆𝐿𝐿𝑡𝑡(𝜃𝜃 �𝑗𝑗−ℎ,𝑅𝑅,𝛾𝛾�𝑗𝑗−ℎ,𝑅𝑅)�≠0 When 𝑚𝑚𝑚𝑚𝑥𝑥𝑡𝑡�𝐹𝐹𝑡𝑡,𝑚𝑚 ∞𝑆𝑆�>𝑘𝑘𝛼𝛼 𝐹𝐹𝑡𝑡,𝑚𝑚 ∞𝑆𝑆 = 𝜎𝜎�−1𝑚𝑚12 �∑∆𝐿𝐿𝑗𝑗 𝑡𝑡+𝑚𝑚 2−1 ⁄ 𝑗𝑗=𝑡𝑡− 𝑚𝑚2 ��𝜃𝜃 �𝑗𝑗−ℎ,𝑅𝑅,𝛾𝛾�𝑗𝑗−ℎ,𝑅𝑅� (A2.12) with 𝑡𝑡=𝐴𝐴+ℎ+ 𝑚𝑚2 �+ 1 35 Giacomini and Rossi (2010) found that the rolling window, m, has to be an even and small number to preserve the power of the test. In this paper, m is set as two. 6. Efficiency Tests We examine the efficiency of the forecasts with two tests: strong and weak efficiency. Before making the efficiency tests, we also check for the presence of bias and autocorrelation in the forecasting errors. The efficiency tests complement the previous tests. They aim to verify if there are other variables not included in the models that could improve the accuracy of the forecasts. 6.1 Absence of Bias This is important to know whether the forecasts are overly optimistic or over pessimistic. To test the absence of bias, we run the following regression e�t= α + εt (A2.13) where e�t is the forecast error, εt is a normal distributed error term with zero mean and α is a constant which measures the degree of bias of the forecasts. Under the null and alternative hypothesis, H0: α = 0 H1: α ≠0 If we reject the null hypothesis, then the forecasts are biased. We test the null hypothesis with a 5 percent level of significance. 6.2 Autocorrelation Tests Another important test is to examine the possible autocorrelation between the forecasting errors of the models. We estimate the following regression e�t= α + βe�t−1 +εt (A2.14) where e�t is the forecast error, εt is a normal distributed error term with zero mean, and α is a constant which measures the degree of bias of the forecasts. Under the joint null hypothesis H0: ∝ = 0, β= 0 36 If we reject the null hypothesis, then the forecasts are biased. We test the null hypothesis with a 5 percent level of significance. 6.3 Weak Efficiency We test if the forecasts contain all the information available included in the models. We estimate the forecast error with an MA structure e�t= α + w𝑡𝑡+ βw𝑡𝑡−1 (A2.15) where e�t is the forecast error,∝is a constant, and β is a coefficient of the moving average process Under the joint null hypothesis H0: ∝ = 0, β= 0 If we do not reject the joint null hypothesis, then the forecasts are weakly efficient. 6.4 Strong Efficiency In this test, we examine if the addition of new variables different from those of the models can explain the forecasting errors. If they can, then the forecasts of the models are not strongly efficient. We estimate the following regression 𝑒𝑒𝑡𝑡= α + βzt+ εt (A2.16) where e�t is the forecast error , εt is a normal distributed error term with zero mean, ∝ is a constant, and β is the coefficient of the new variables. Under the joint null hypothesis H0: ∝ = 0, β= 0 We consider five variables of the structural model to perform the strong efficiency tests in the IV and the EFP model, which are the econometrics models. 37 Appendix 3. Tables of the Unconditional Forecast Evaluation In this appendix, we include the tables of the unconditional forecasting evaluation. 1. Statistical Properties of the Forecasting Error Table A3.1. Statistical Properties of the Forecasting Error, Quantitative Models Forecasting Horizons in quarters Skewness Normality (Jarque-Bera Test) IV AMM EFP IV AMM EFP 1 -0.15 0.21 0.32 0.90 (0.64) 0.70 (0.70) 0.805 (0.67) 2 0.09 -0.07 0.18 1.28 (0.52) 0.05 (0.97) 0.200 (0.91) 3 -0.03 0.07 0.13 3.13 (0.21) 0.31 (0.87) 2.000 (0.78) 4 -0.41 0.06 0.29 2.45(0.29) 3.15 (0.21) 0.690 (0.71) 8 -0.55 0.51 -0.31 4.50 (0.11) 4.60 (0.10) 0.345 (0.84) Source: Author’s compilation, Central Bank forecasts. Table A3.2: Statistical Properties of the Forecasting Error, Inflation Expectations Forecasting Horizons in Years Skewness Normality (Jarque-Bera Test) Economic Research Department Economic Experts’ Panel Economic Research Department Economic Experts 1 0.80 0.56 3.02 ( 0.22 ) 1.97 ( 0.37 ) 2 1.11 -0.32 2.88 (0.24) 0.39 (0.82) Source: Author’s compilation, Central Bank forecasts. Table A3.3. Statistical Properties of the Forecasting Error, Quantitative Model, December Evaluation Forecasting Horizons in quarters Skewness Normality (Jarque-Bera Test) IV AMM EFP IV AMM EFP 1 -0.39 0.53 0.71 0.42 ( 0.81 ) 0.69 ( 0.70 ) 0.53 ( 0.77 ) 2 -0.20 -0.19 0.36 1.08 ( 0.58 ) 0.82 ( 0.66 ) 0.35 ( 0.84 ) 3 0.40 -0.36 0.68 1.52 ( 0.76 ) 2.06 ( 0.36 ) 0.52 ( 0.77 ) 4 0.59 0.10 0.00 2.48 ( 0.29 ) 1.40 ( 0.50 ) 0.33 ( 0.84 ) 5 -0.28 0.09 0.00 0.86 (0.65 ) 1.04 ( 0.59 ) 0.33 ( 0.85 ) 6 -0.40 -0.25 0.00 0.95 ( 0.62 ) 1.48 ( 0.48 ) 0.33 ( 0.85 ) 7 -0.80 -0.28 0.00 1.48 ( 0.48 ) 1.85 ( 0.40 ) 0.33 ( 0.85 ) 8 -1.08 -0.10 ------- 2.93 (0.23) 1.45 (0.47) --------------- Source: Author’s compilation, Central Bank forecasts. 44 6. Efficiency Tests 6.1 Unbiasedness Tests Table A3.23. Unbiasedness Test, Quantitative Models Forecasting Horizons in Quarters Unbiasedness test (AMM) Unbiasedness test (IV) Unbiasedness test (EFP) 1 -0.03 (0.63) -0.04 (0.47) -0.45 (0.02) 2 -0.24 (0.12) -0.22 (0.12) -0.04 (0.69) 3 -0.42 (0.06) - 0.50 (0.02) -0.01 (0.92) 4 -0.58 (0.03) -0.87 (0.00) 0.02 (0.80) 8 -0.35 (0.24) -1.40 (0.01) 0.04 (0.79) Source: Author’s compilation, Central Bank forecasts. Table A3.24. Unbiasedness Test, Inflation Expectations Forecasting Horizons in Years Unbiasedness test (EEP) Unbiasedness test (DIE) 1 0.10 (0.71) 0.45 (0.10) 2 0.31 (0.00) 0.53 (0.00) Source: Author’s compilation, Central Bank forecasts. 6.1 Autocorrelation Tests Table A3.25. Autocorrelation Test, Quantitative Models Forecasting Horizons in Quarters Autocorrelation Test (AMM) Autocorrelation Test (IV) Autocorrelation Test (EFP) 1 6.67 (0.01) 0.56 (0.58) 6.68 (0.01) 2 3.81 (0.04) 4.89 (0.02) 0.07 (0.93) 3 5.87 (0.01) 13.04 (0.00) 0.03 (0.97) 4 9.34 (0.00) 19.87 (0.00) 0.05(0.96) 8 13.04 (0.00) 70.13 (0.00) 0.02 (0.98) Source: Author’s compilation, Central Bank forecasts. Table A3.26. Autocorrelation Test, Inflation Expectations Forecasting Horizons in Years Autocorrelation Test (DIE) Autocorrelation Test (EEE) 1 77.28 (0.00) 66.81 (0.00) 2 42.54 (0.00) 9.79 (0.01) Source: Author’s compilation, Central Bank forecasts. 45 C. Weak Efficiency Tests. Table A3.27. Weak Efficiency Test, Quantitative Models Forecasting Horizons in Quarters Weak Efficiency test (AMM) Weak Efficiency test (IV) Weak Efficiency test (EFP) 1 0.11 (0.89) 6.33 (0.24) 3.58 (0.08) 2 4.41 (0.02) 3.29 (0.12) 0.22 (0.89) 3 6.18 (0.00) 11.57 (0.01) 12.08 (0.97) 4 5.39 (0.01) 21.81 (0.00) ------------------- 8 104.62 (0.00) 62.16 (0.00) 0.20 (0.83) Source: Author’s compilation, Central Bank forecasts. Table A3.28. Weak Efficiency Test, Inflations Expectations Forecasting Horizons in Years Weak Efficiency test (EEP) Weak Efficiency test (DIE) 1 1.67E+11 (0.00) 184.40 (0.00) 2 6.51 (0.02) 62.77 (0.00) Source: Author’s compilation, Central Bank forecasts. Table A3.29. Weak Efficiency Test, Quantitative Models, December Evaluation Forecasting Horizons in Quarters Weak Efficiency test (AMM) Weak Efficiency test (IV) 1 83.48 (0.00) 1.17E+12 (0.00) 2 1.36 (0.35) 1.5268 (0.32) 3 1.45 (0.34) 1.2242 (0.38) 4 8.87 (0.034) 9.5156 (0.03) 5 1.71E+11 (0.00) 14.1267 (0.03) 6 1.85E+11 (0.00) 1.6197 (0.33) 7 2.03E+10 (0.00) 0.9950 (0.47) 8 1.66E+10 (0.00) 1.8451 (0.30) Source: Author’s compilation, Central Bank forecasts. 46 6.4 Strong Efficiency Tests. Table A3.30. Strong Efficiency Test, IV Model Forecasting Horizons in Quarters Strong Efficiency (consumption) Strong Efficiency (raw material index) Strong Efficiency (Investment) Strong Efficiency (Government Spending) Strong Efficiency (Credit) 1 0.47 (0.49) -0.32 (0.43) 0.20 (0.79) 0.33 (0.69) 0.26 (0.51) 2 2.83 (0.09) -1.60 (0.10) 2.51 (0.19) 0.25 (0.91) 0.71 (0.49) 3 6.17 (0.01) -2.89 (0.04) 4.86 (0.10) -0.01 (0.99) 1.64 (0.31) 4 9.36 (0.00) -4.90 (0.01) 7.47 (0.04) 1.35 (0.78) 4.42 (0.04) 8 11.91 (0.00) -5.02 (0.01) 10.79 (0.01) -9.11 (0.29) 5.40 (0.10) Source: Author’s compilation, Central Bank forecasts. Table A3.31. Strong Efficiency Test, EFP Model Forecasting Horizons in Quarters Strong Efficiency (consumption) Strong Efficiency (raw material index) Strong Efficiency (Investment) Strong Efficiency (Government Spending) Strong Efficiency (Credit) 1 8.49 (0.01) -3.3091 (0.05) 9.10 (0.02) -11.93 (0.01) 6.76 (0.09) 2 1.73 (0.37) -1.1865 (0.24) 2.89 (0.29) -3.76 (0.17) 3.54 (0.17) 3 1.16 (0.59) ------------------ ------------------ -3.27 (0.27) 3.64 (0.27) 4 -0.08 (0.98) 0.0919 (0.10) 0.97 (0.79) -2.70 (0.37) ------------------ 8 ------------------ 0.7945 (0.82) ------------------ -5.80 (0.33) ------------------ Source: Author’s compilation, Central Bank forecasts. Table A3.32: Strong Efficiency Test, IV Model, December Evaluation Forecasting Horizons in Quarters Strong Efficiency (consumption) Strong Efficiency (raw material index) Strong Efficiency (investment) Strong Efficiency (Government Spending) Strong Efficiency (Credit) 1 1.13 (0.31) -0.83 (0.23) 0.94 (0.49) 0.94 (0.95) 0.7864 (0.34) 2 7.88 (0.06) -4.46 (0.11) 6.32 (0.27) 8.55 (0.26) 4.9864 (0.13) 3 5.83 (0.04) -3.60 (0.05) 4.88 (0.23) 2.66 (0.65) 3.5045 (0.14) 4 5.90 (0.34) -3.25 (0.42) 2.13 (0.79) -6.29 (0.54) 1.7364 (0.72) 5 9.54 (0.21) -4.28 (0.41) 5.47 (0.60) 2.96 (0.92) 4.4500 (0.53) 6 11.90 (0.17) -6.16 (0.29) 8.99 (0.45) 7.23 (0.83) 7.1250 (0.37) 7 13.00 (0.13) -6.81 (0.24) 10.49 (0.37) 11.69 (0.73) 8.0250 (0.32) 8 11.59 (0.16) -6.40 (0.24) 10.78 (0.33) 15.01 (0.63) 7.9000 (0.28) Source: Author’s compilation, Central Bank forecasts. 47 Appendix 4. Graphic Analysis of Giacomini-Rossi Fluctuation Tests In this appendix, we graphically illustrate the Giacomini-Rossi fluctuation tests between two competing models. The IV model is the benchmark model in this study. If the GR statistics line is outside the critical bands, it means that the null hypothesis of equal accuracy is rejected. 1. GR - Test between the AMM and the IV Model In Figure A4.1, we observe that the GR statistic line is outside of the critical band, which means that we reject the null hypothesis of equal accuracy. Also, over 2011 the forecasts of the IV model are more accurate than those of the AMM model. Similarly, the forecasts of the former are better than the latter in June 2013 and March 2016. Figure A4.1 Source: Authors’ compilation, Central Bank Dataset. From Figure A4.2, we reject the null hypothesis of equal predictability too. In this case, from the second to the four quarter of 2011, the forecasts of the IV model are more accuracy than the AMM model. However, from the first to the second quarter of 2015, the forecasting accuracy is reverse. -4 -2 0 2 4 2011q3 2013q1 2014q3 2016q1 2017q3 Time Critical values G-R test stat. Fluctuation Test AMM - IV (One quarter ahead) 48 -10 -5 0 5 10 2012q1 2013q1 2014q1 2015q1 2016q1 2017q1 Time Critical values G-R test stat. Fluctuation Test AMM - IV (Three quarters ahead) Figure A4.2 Source: Authors’ compilation, Central Bank Dataset. From Figure A4.3, the null hypothesis of equal accuracy is rejected since the GR statistics is outside the critical bands. As in the previous graph, the forecasts of the IV model are more accurate in the beginning, from the first to the second quarter of 2012. After that, the forecasts of the AMM are better from the first to the third quarter of 2015. Figure A4.3 Source: Authors’ compilation, Central Bank Dataset. -15 -10 -5 0 5 2012q1 2013q1 2014q1 2015q1 2016q1 2017q1 Time Critical values G-R test stat. Fluctuation Test AMM - IV (Two quarters ahead) 49 From Table A4.4, the null hypothesis of equal accuracy is rejected since the GR statistic is outside the lower critical bands in two different sub periods. Therefore, the forecasts of the AMM model are more accurate in two different sub-periods of the total sample. FigureA4.4 Source: Authors’ compilation, Central Bank Dataset. From Figure A4.5, we observe that the GR statistics is below the lower critical band. In this forecasting horizon, the forecasts of the AMM model are more accurate in a significant part of the total sample. Figure A4.5 Source: Authors’ compilation, Central Bank Dataset. -10 -5 0 5 2012q3 2013q3 2014q3 2015q3 2016q3 2017q3 Time Critical values G-R test stat. Fluctuation Test AMM - IV (Four quarters ahead) -15 -10 -5 0 5 2013q3 2014q3 2015q3 2016q3 2017q3 Time Critical values G-R test stat. Fluctuation Test AMM - IV (eigth quarters ahead) 50 Therefore, there is some evidence that, in the medium and long term, the forecasts of the AMM model are more accurate than those of the IV model. 2. GR Test between the EFP and the IV Model We use the graphical analysis to examine the behavior of the forecasts in every rolling window as we did previously. From Figure A4.6, we can observe that in the beginning of the sample, specifically through 2015, the forecasts of the IV model are more accurate than those of the EFP model. After that, there is no statistical difference between them. Therefore, it seems that the IV model generates better forecasts than the EFP model in the case of the one-quarter horizon. Figure A4.6 Source: Authors’ compilation, Central Bank dataset. From Figure A4.7, we observe that the null hypothesis of equal predictability is rejected because the GR statistic line is outside the upper critical band in most of the sample. The forecasts of the IV model are more accuracy than those of the EFP model. Therefore, the IV model generates better forecast than the EFP model in the case of the two quarters horizon. -4 -2 0 2 4 6 2015q1 2015q3 2016q1 2016q3 2017q1 Time Critical values G-R test stat. Fluctuation Test EFP - IV (One quarter ahead) 51 Figure A4.7 Source: Authors’ compilation, Central Bank dataset. From Figure A4.8, we observe that in most of the sample, the GR statistic line is outside the upper critical band, which means that in these subperiods, the forecasts of the IV model are more precise than those of the EFP model. Therefore, the IV model can better predict the patterns of inflation. Figure A4.8 Source: Authors’ compilation, Central Bank dataset. -4 -2 0 2 4 6 2015q2 2015q4 2016q2 2016q4 2017q2 Time Critical values G-R test stat. Fluctuation Test EFP - IV (Two quarters ahead) -5 0 5 10 2015q3 2016q1 2016q3 2017q1 2017q3 Time Critical values G-R test stat. Fluctuation Test EFP - IV (Three quarters ahead) 52 From Figure A4.9, we observe the same pattern as before. The GR statistics line is outside the upper critical band, which means that the null hypothesis is rejected. Again, the forecasts of the IV model are more accurate than those of the EFP model in most of the sub-periods of time. Figure A4.9 Source: Authors’ compilation, Central Bank Dataset. All in all, the IV model better predicts better the inflation pattern than the EFP model in different sub-periods of time. 3. GR Test between the EEP and DIE We use the graphical analysis to examine the performance of the forecasts in different sub-periods of time. In contrast to the quantitative analysis, we have monthly inflation expectations. From Figure A4.10, we observe that the GR statistic line is outside the upper and lower critical bands in some sub-periods of time, which means that the null hypothesis of equal accuracy is rejected. From October 2015 to December 2015, the inflation expectations of the DIE are more accurate than those of the EEP. However, the trend is reversed later. From April 2016 to June 2017, the inflation expectations of the EEP are more accurate. -2 0 2 468 2016q1 2016q2 2016q3 2016q4 2017q1 2017q2 Time Critical values G-R test stat. Fluctuation Test EFP - IV (Four quarters ahead) 53 Figure A4.10 Source: Authors’ compilation, Central Bank Dataset. From Figure A4.11, we see that in almost the entire sample the GR statistic line is outside the lower critical band. Therefore, we reject the null hypothesis of equal accuracy between the inflation expectations of the DIE and the EEP. In almost the complete sample, the inflation expectations of the Economic Experts Panel are more accurate at predicting the inflation patterns than those of the Economic Research Department in the case of the two-year horizon. Figure A4.11 Source: Authors’ compilation, Central Bank Dataset. -20 -10 0 10 20 2015m10 2016m4 2016m10 2017m4 Time Critical values G-R test stat. Fluctuation Test EEP - DIE (One year ahead) -15 -10 -5 0 5 2016m10 2017m1 2017m4 2017m7 Time Critical values G-R test stat. Fluctuation Test EEFP - DIE (Two years ahead)