Density Forecasts with Midas Models

Aastveit, Knut Are; Foroni, Claudia; Ravazzolo, Francesco Working Paper Density Forecasts with Midas Models Working Paper, No. 10/2014 Provided in Cooperation with: Norges Bank, Oslo Suggested Citation: Aastveit, Knut Are; Foroni, Claudia; Ravazzolo, Francesco (2014) : Density Forecasts with Midas Models, Working Paper, No. 10/2014, ISBN 978-82-7553-818-3, Norges Bank, Oslo, https://hdl.handle.net/11250/2495975 This Version is available at: https://hdl.handle.net/10419/210059 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. https://creativecommons.org/licenses/by-nc-nd/4.0/deed.no Density forecasts with MIDAS models Norges BaNk research 10 | 2014 AuthorS: Knut Are AAStveIt ClAuDIA ForonI FrAnCeSCo rAvAzzolo WorkiNg PaPer Norges BaNk Working PaPer xx | 2014 rapportNavN 2 Working papers fra Norges Bank, fra 1992/1 til 2009/2 kan bestilles over e-post: [email protected] Fra 1999 og senere er publikasjonene tilgjengelige på www.norges-bank.no Working papers inneholder forskningsarbeider og utredninger som vanligvis ikke har fått sin endelige form. hensikten er blant annet at forfatteren kan motta kommentarer fra kolleger og andre interesserte. Synspunkter og konklusjoner i arbeidene står for forfatternes regning. Working papers from Norges Bank, from 1992/1 to 2009/2 can be ordered by e-mail: [email protected] Working papers from 1999 onwards are available on www.norges-bank.no norges Bank’s working papers present research projects and reports (not usually in their final form) and are intended inter alia to enable the author to benefit from the comments of colleagues and other interested parties. views and conclusions expressed in working papers are the responsibility of the authors alone. ISSN 1502-8143 (online) ISBN 978-82-7553-818-3 (online) Density forecasts with MIDAS models∗ Knut Are Aastveit† Norges Bank Claudia Foroni‡ Norges Bank Francesco Ravazzolo§ Norges Bank and BI Norwegian Business School This version: July 17, 2014 Abstract In this paper we derive a general parametric bootstrapping approach to compute density forecasts for various types of mixed-data sampling (MIDAS) regressions. We consider both classical and unrestricted MIDAS regressions with and without an autoregressive component. First, we compare the forecasting performance of the different MIDAS models in Monte Carlo simulation experiments. We find that the results in terms of point and density forecasts are coherent. Moreover, the results do not clearly indicate a superior performance of one of the models under scrutiny when the persistence of the low frequency variable is low. Some differences are instead more evident when the persistence is high, for which the ARMIDAS and the AR-U-MIDAS produce better forecasts. Second, in an empirical exercise we evaluate density forecasts for quarterly US output growth, exploiting information from typical monthly series. We find that MIDAS models applied to survey data provide accurate and timely density forecasts. JEL codes: C11, C53, E37 Keywords: Mixed Data Sampling, Density Forecasts, Nowcasting ∗This Working Paper should not be reported as representing the views of Norges Bank. The views expressed are those of the authors and do not necessarily reflect those of Norges Bank. We would like to thank Todd Clark, Mike Clements, Eric Ghysels, James Mitchell, Anders Rygh Swensen, Christian Schumacher and seminar and workshop participants at the Macroeconomic and Econometric Conference in Birmingham, Norges Bank, the 22nd Annual Symposium of the Society for Nonlinear Dynamics and Econometrics in New York, the IAAE 2014 Annual Conference in London and the 34th International Symposium on Forecasting in Rotterdam for useful comments. †Knut-Are.Aastv[email protected] ‡Corresponding author: Claudia.F[email protected] §Francesco.Rav[email protected] 1 1 Introduction Economic time series are often available at different frequencies. As an example, while Gross Domestic Product (GDP) is released quarterly and with a substantial delay, many indicators are available at a monthly frequency and in a more timely manner. These timely indicators often contain important indications of the current state of the economy. In order to obtain good nowcasts and short-term forecasts, it is therefore crucial to exploit the information from high-frequency data, see e.g. Evans (2005) and Banbura et al. (2011). Hence, methods that can deal with mixed-frequency data are becoming increasingly popular for nowcasting and short-term forecasting. Nowadays, there are several methods in the literature that can deal with mixed-frequency data. Factor models and mixed-frequency VARs are examples of models that can deal with mixed-frequency data when cast in a state-space form with time aggregation scheme, see e.g. Mariano and Murasawa (2010) and Kuzin et al. (2011) for examples of mixed-frequency VARs, and Giannone et al. (2008) and Banbura and Modugno (2014) for examples of mixed-frequency dynamic factor models.1An alternative univariate approach is the mixed data-sampling (MIDAS) regression proposed by Ghysels et al. (2005, 2006) in the context of financial applications and extended to a macroeconomic context by Clements and Galv˜ao (2008, 2009). Policymakers and forecasters are increasingly interested in forecast metrics that require density forecasts of macroeconomic variables, as complete probability distributions over outcomes provide information helpful for making economic decisions, see e.g. Tay and Wallis (2000), Garratt et al. (2003), Gneiting (2011) and Clark (2011). Accordingly, several central banks, including the Bank of England, Norges Bank and Sveriges Riksbank have committed to publishing density or interval forecasts for macroeconomic aggregates in recent years. However, despite the flourishing theoretical and empirical literature on the use of mixed-frequency approaches, the focus has so far mainly been on point forecasts, Aastveit et al. (2014), Mazzi et al. (2013) and Carriero et al. (2014) being notable exceptions.2 In this paper, we extend the existent literature by using the MIDAS approach to obtain 1See also Kuzin et al. (2013) and Aastveit et al. (2014) for applications that combine nowcasts from different models that use mixed-frequency data. 2Aastveit et al. (2014) study combinations of density nowcasts from a wide set of factor models, mixedfrequency VARs and bridge equation models, while the two latter studies focus on either combining density nowcasts from a small set of bridge equation models, Mazzi et al. (2013), or density nowcasts from models with time-varying parameters and volatility, Carriero et al. (2014). 2 density forecasts. As a first contribution, we compute density forecasts from different MIDAS model specifications. In particular, we consider the classical MIDAS model, introduced by Ghysels et al. (2004), and the extended version by Clements and Galv˜ao (2008), which includes an autoregressive component. These models rely on the exponential lag polynomials to exploit high-frequency information while at the same time being parsimonious. Due to the non-linearity of the lag polynomials, MIDAS regressions are typically estimated by non-linear least squares (NLS). Furthermore, we also examine the unrestricted MIDAS regressions, U-MIDAS and AR- U-MIDAS, introduced by Foroni et al. (2013). The U-MIDAS and AR-U-MIDAS approaches do not resort to functional distributed lag polynomials and can be estimated by OLS. We propose a general stochastic simulation method based on parametric bootstrapping for computing predictive densities for both MIDAS and U-MIDAS regressions. Our method accounts for both parameter and shock uncertainty and extends the bootstrapping procedures in Berkowitz and Kilian (2000) and Clements and Taylor (2001) to the class of MIDAS models. We evaluate our bootstrapping approach in Monte Carlo simulations as well as in an empirical exercise applied to US real-time data. First, we compare the forecasting performance of the different MIDAS models using Monte Carlo simulations. The basic design of the exercise is similar to that of Foroni et al. (2013), where a high-frequency VAR(1) is specified. We consider three different residual assumptions: normal distributed residuals, Student’s t-distributed residuals with 5 degrees of freedom, and residuals that have a time-varying variance modeled as a GARCH(1,1) process. We evaluate density forecasts according to two different criteria, the Logarithmic Score (LS) and the Continuous Rank Probability Score (CRPS). Point forecasts are evaluated based on root mean square forecasting errors. Our findings indicate that the forecasting results in terms of point and density forecasts are coherent. Moreover, the results doe not clearly indicate a superior performance of one of the models under scrutiny, especially when the persistence of the low-frequency variable is low. Some differences are instead more evident when the persistence is high. In this case, the AR-U-MIDAS performs somewhat better, both in terms of point and density forecasts. Second, in an empirical nowcasting exercise using US real-time data, we compute recursive density forecasts from the various MIDAS regressions for quarterly real output growth for the evaluation period 1985q2-2013q2. We consider 6 monthly explanatory variables. Following 3 Clements and Galv˜ao (2008), we use industrial production, employment (nonfarm payrolls) and capacity utilization. In addition, we include 3 survey-based measures: the Purchasing Managers Index (PMI), the Chicago Fed National Activity Index (CFNAI) and the Philadelphia Fed Business Outlook Survey for general business activity (PFBOS). The monthly real-time vintages that we use correspond to the information available around the 20-25th of each month in the quarter, i.e., just after the release of the CFNAI. Looking ahead to the results, we find that the use of within-quarter information on monthly indicators can result in marked improvements in density forecast accuracy, with reductions in CRPS and increases in LS compared to the quarterly-frequency AR and AR distributedlag (ADL) models. In particular, within the set of models that use monthly information, the survey-based measures seem to improve forecast accuracy relative to the ADL, which ignores the most timely information. However, gains are smaller when focusing on more traditional macroeconomic series such as industrial production, employment (nonfarm payrolls) and capacity utilization. The results are robust to ending the evaluation sample in 2007q4, i.e. prior to the Great Recession. The plan of the rest of our paper is as follows. Section 2 provides an overview of the different MIDAS models that we consider in our analysis. Section 3 describes our bootstrapping approach for computing density forecasts for MIDAS models and how these forecasts are evaluated. In Section 4, results from our Monte Carlo experiments are presented, while results from the empirical application are presented in Section 5. Finally, Section 6 concludes. 2 Mixed-data Sampling To take into account mixed-frequency data, Ghysels et al. (2004) introduced the mixed-data sampling (MIDAS) approach, in which the dependent variable ytq, sampled at a lower frequency, is regressed on a distributed lag of xtm, which is sampled at a higher frequency. In this section, we present the two main MIDAS approaches that have been suggested in the literature; the standard MIDAS regression introduced by Ghysels et al. (2004) and the unrestricted version as in Foroni et al. (2013). For each of the two approaches, we present the basic model and their extension with an AR component. In the rest of the paper, we define tq= 1, ...Tqas the basic (quarterly) time unit and 4 tm= 1, ...Tmas the higher frequency (monthly) time unit. In every basic time unit we observe the high frequency unit m= 3 times (e.g. 3 months in a quarter). 2.1 The MIDAS model As introduced by Ghysels et al. (2004), MIDAS regressions are reduced form regressions that involve processes sampled at different frequencies. Highly parsimonious distributed lag polynomials are used to link the higher-frequency variable to the low-frequency one. These polynomials are chosen in such a way to prevent the proliferation of parameters while at the same time being very flexible, allowing for various shapes, where only a few parameters need to be estimated. The basic MIDAS model for a single explanatory variable, and hq-step-ahead forecasting, with hq=hm/m, is given by: ytq+mhq=ytm+hm=β0+β1b(Lm;θ)x(m) tm+w+εtm+hm(1) where bL1/m;θ= K P k=0 c(k;θ)Lk m,and Lx mx(m) tm=x(m) tm−x. x(m) tm+wis skip-sampled from the high frequency indicator xtm. Different choices for c(k;θ) have been proposed in the literature. The most common one is the exponential Almon lag, defined as: c(k;θ) = exp θ1k+... +θQkQ K P k=1 exp (θ1k+... +θQkQ) (2) The key advantage of this specification is that it allows for long lags with a limited number of parameters. It also allows for different values of the parameters to take various shapes. Therefore, estimating the parameters from the data automatically determines the shape of the weights and the number of lags to be included in the regression. The MIDAS model is estimated using nonlinear least squares (NLS) in a regression of ytonto x(m) t−h. Notice that the MIDAS model is h−dependent, and thus needs to be re-estimated for each forecast horizon. It is therefore a model which provides direct and not recursive forecasts. There is no therefore need to have forecasts of the independent variable, but the model is adjusted 5 accordingly to include the very last observations of the x(m)available. The forecast is given by byTy m+hm|Tx m=b β0+b β1bLm;b θx(m) Tx m.(3) 2.2 The AR-MIDAS model The main extension to the MIDAS model is the inclusion of an autoregressive term. However, this is not straightforward, since it would result in the creation of seasonal patterns in the explanatory variables, as highlighted in Clements and Galv˜ao (2009). To avoid the inconvenience of seasonal effects, Clements and Galv˜ao (2009) suggest the introduction of the AR dynamics as a common factor: ytm=β0+λytm−hm+β1b(Lm;θ)1−λLhm mx(m) t+w−hm+εtm.(4) In this way, the response of yto x(m)remains non-seasonal. To estimate the AR-MIDAS model, we follow the procedure described in Clements and Galv˜ao (2009), for which we briefly highlight the steps. First, estimate the standard MIDAS (the basic model), take the residuals bεtmand estimate an initial value for λ, say λ0, where b λ0=Pbε2 tm+w−hm−1Pbεtmbεtm+w−hm.Then construct y∗ tm=ytm−b λ0ytm−hmand x∗(m) tm+w−hm= x(m) tm+w−hm−b λ0x(m) tm−(hm−w)−hm. The estimator b θ1is obtained by applying nonlinear least squares to: y∗ tm=β0+β1b(Lm;θ)x∗(m) tm+w−hm+εtm.(5) A new value of λ,b λ1, is obtained from the residuals of this regression. Then a new step is run, using b λ1and b θ1as the initial values. In this way, the procedure gets the estimates of b λand b θ which minimize the sum of squared residuals. 2.3 The Unrestricted MIDAS model Foroni et al. (2013) study the performance of MIDAS regressions that do not resort to functional distributed lag polynomials, referred to as the U-MIDAS. The U-MIDAS model is based on a 6 1000 draws. The forecasts are then evaluated based on MSPE (point forecasts) and LS and CRPS (density forecasts). 4.2 DGP with normally distributed shocks: results The results for the Monte Carlo forecast experiments where the DGP has normally distributed residuals, are summarized in Table 2. We compare the forecasting performance of the different MIDAS models based on their (out-of-sample) MSPEs, CRPS, and LS computed over the 50 periods of the evaluation sample. Moreover, we report the results for two standard quarterly benchmarks, an AR(1) process and a Random Walk (RW) process. The densities for the benchmark are computed using our bootstrap method without the high frequency components, therefore similar to Berkowitz and Kilian (2000) and Clements and Taylor (2001). In this way, we are also able to compare the performance of our mixed-frequency models relative to models that consider only quarterly information. From Table 2, we observe several interesting results. First, when the persistence of the low-frequency variable is low (ρ= 0.1,0.5), using high-frequency information substantially improves the forecasting performance relative to the AR and RW benchmarks. This is not very surprising as these benchmarks discard the useful information provided by the high-frequency variable. Moreover, by comparing the performance of the U-MIDAS vs. the AR-U-MIDAS and the MIDAS vs. AR-MIDAS, including an AR-component in the mixed frequency models does not substantially affect the forecasting performance. Obviously, results change when the persistence is high, i.e. setting ρ= 0.9 in our simulations. In this case, only the AR-MIDAS and the AR-U-MIDAS produce superior forecasts relative to the two benchmarks. The results in terms of point and density forecasts are coherent: whenever we find a smaller MSPE, the models also obtain an improvement in the density forecasts. We interpret this as supporting evidence for our bootstrapping approach, which is then precise not only for the mean part of the distribution but more importantly also for the full distribution. When ρ= 0.9, the results do not change in the case of the U-MIDAS, but some issues arise in the case of the AR-MIDAS. In a few of the bootstrapping replications, imposing the common factor restriction in the AR component of the AR-MIDAS model to avoid seasonality effects creates convergence issues and forecast performance deteriorates as a result. 13 Finally, ranking the mixed-frequency models is not straightforward, as results do not clearly indicate a superior performance to one of the models under scrutiny. However, we can highlight some features that emerge from the table. First of all, when the persistence of the model is low, there are no sizeable differences in performance across the models. Some differences are instead more evident when the persistence is high. The AR-U-MIDAS seems to perform slightly better, in terms of both point and density forecasts. This evidence is in line with the findings in Foroni et al. (2013) in the case of point forecasts. It is worth mentioning that the equally good or even superior performance of the AR-U- MIDAS is obtained in a much shorter computational time and with no convergence issues, since the model and bootstrap replications are estimated by a simple OLS. 4.3 DGP with Student’s t-distributed shocks: results In this subsection, we present the results of the Monte Carlo forecast experiments for the second DGP, in which the residuals are drawn from a Student’s t distribution. As in the previous simulation design, we compare the forecasting performance of the different MIDAS models based on their MSPEs, CRPS, and LS. The results of the mixed-frequency models, together with the two benchmarks mentioned in the previous subsection, are reported in Table 3. In general, the results display the same features as in the previous DGP, with shocks generated by a normal distribution. First, we can confirm that when the persistence of the low frequency variable is low, using high-frequency information hugely improves forecasting performance relative to a quarterly benchmark. Moreover, the different mixed-frequency models perform in a very similar way, and ranking them is impossible because their forecasting ability is nearly the same. In contrast, when the persistence is high, it is the AR component which turns out to be necessary to have a forecasting performance able to compete with the benchmarks. Therefore, the models that incorporate an AR term clearly outperform their counterparts that do not. The results in terms of point and density forecasts are also in this case coherent: whenever a model performs well in terms of point forecasts, it also performs well in terms of densities. 14 4.4 DGP with time-varying volatility shocks: results As a final Monte Carlo experiment, we consider a DGP where the residuals of the low-frequency variable are drawn from a GARCH(1,1) distribution. The design of the experiment is similar to the previous subsections, except for the error distribution for the variable y. The results of the mixed-frequency models, together with the two benchmarks, are reported in Table 4. Even in this case, the results share the same features as in the previous two cases, where the errors in the DGPs were distributed respectively as a Normal and as a Student’s t. Once again, therefore, we confirm the importance of mixed-frequency information in both point and density forecasting, especially in the case in which the persistence of the low frequency variable is low. 5 An empirical application: nowcasting US output growth In this section we provide an empirical application which focuses on the performance of different MIDAS models for real-time out-of-sample nowcasting of quarterly US real output growth using monthly information. By using real-time data, we take into account the publication lags of the different series and ensure that we are only using data that were available on the date of the forecast origin. We evaluate the performance of our models in terms of point and density forecasts. 5.1 The data and design of the empirical exercise Our dataset includes quarterly US real output growth and six monthly explanatory indicators, which are recognized to be important predictors. Table 1 provides an overview of the data and some of their real-time characteristics. Following Clements and Galv˜ao (2008), we consider industrial production, employment (nonfarm payrolls), and capacity utilization. The three monthly indicators are converted to monthly growth rates by taking the log-differences of the series. Quarterly real output growth is measured as the log-difference of the real GDP series. We obtained the real-time data vintages of the aforementioned variables from the Federal Reserve Bank of Philadelphia’s Real-Time Dataset for Macroeconomists (RTDSM), described in Croushore and Stark (2001). Moreover, we include three survey-based measures: the Purchasing Managers Index (PMI), the Chicago Fed National Activity Index (CFNAI) and the Philadel- 15 Table 1: Description of the data Name Description Transf First vintage Timing Publishing lag EMP Employment ∆ln 1985m1 1st Friday of month 1 month IP Industrial Production ∆ln 1985m1 15-18th of month 1 month CU Capacity Utilization ∆ln 1985m1 15-18th of month 1 month PMI NAPM Purchasing Managers Index Level 2002m1 1st business day of month 1 month CFNAI Chicago Fed National Activity Index Level 2001m1 22-25th of month Current month PFBOS Philly Fed’s Business Outlook Survey Level Last vintage 3rd Thursday of month Current month Note: Variable names and variable descriptions are reported in Column 1 and 2. Column 3 reports the transformation that is used for each of the explanatory variables in various MIDAS regressions. The first available real-time vintage for each variable is reported in Column 4. Finally, Column 5 and Column 6 indicate the official dates of the publication and the lag with which the data are reported, respectively. phia Fed Business Outlook Survey for general business activity (PFBOS). Monthly real-time vintages of PMI are available from the ALFRED (ArchivaL Federal Reserve Economic Data) database maintained by the Federal Reserve Bank of St. Louis. Real-time vintages of monthly CFNAI are collected from the website of the Federal Reserve Bank of Chicago. Finally, the PFBOS series is collected from the website of the Federal Reserve of Philadelphia. Real-time vintages of the PFBOS are not available. However, this series only undergoes revisions due to changes in seasonal factors. Since these revisions are substantially smaller than revisions in e.g. national account data, we include this series in our empirical analysis. We compute recursive point and density forecasts for US real GDP growth from the different MIDAS regressions. As benchmarks, we consider quarterly-frequency AR and autoregressive distributed lag (ADL) models.9For the AR model we select the lag length according to the BIC criterion, where the maximum lag is fixed to 4. For the ADL model, we fix the lag of GDP growth equal to one and select the lags of the exogenous variables with a BIC criterion. Our estimation sample covers the period 1970m1 to 2013m6. The full recursive forecast evaluation period runs from 1985q1 to 2013q2. As robustness, we also provide results for an evaluation sample that ends in 2007q4 (before the Great Recession). We use monthly real-time vintages from 1985m2 to 2013m7 and compute forecasts using monthly vintages that correspond to the available information between the 20th and 25th of the month.10 More precisely, this corresponds to the timing just after the release of the CFNAI. The quarterly US GDP is available as an advance estimate at the end of the first month after the end of the referring quarter. Accordingly, when 9Notice that when the monthly exogenous variable is only available for the first month or the two first months of the quarter, the quarterly ADL model ignores this information. 10For the PMI and CFNAI, real-time vintage data exist only for parts of the evaluation period, from 2002m1 and 2001m1, respectively. For these, we use the first available real-time vintage and truncate these series backward recursively. 16 computing the forecasts at the second and third month of each quarter (e.g. February and March), GDP growth for the previous quarter (e.g. q4 in our example) is available. On the other hand, when we compute forecasts in the first month of each quarter (e.g. January), we need to backcast the GDP value for the previous quarter. For each monthly indicator we specify 4 different MIDAS regressions (MIDAS, AR-MIDAS, UM-IDAS and AR-U-MIDAS). We compute a nowcast from each model specification for each monthly vintage, i.e. 3 forecasts per quarter (which correspond to month horizons hm= 1,2,3). Due to the release delay of US real output, we also compute a backcast of the previous quarter for the first month of each quarter (we refer to this as hm=−1). A key issue in this exercise is the choice of a benchmark for the “actual” measure of GDP. We follow Romer and Romer (2000) in using the second available estimate of GDP as the actual measure.11 The forecasting performance of the different models is assessed in terms of CRPS and LS for the density forecasts. To provide a rough gauge of whether the differences in forecast accuracy between the ADL model and the alternative MIDAS models are significant, we follow Clark and Ravazzolo (2014) and apply a Diebold and Mariano (1995) t-tests for equality of the average loss (with loss defined as LS, CRPS or RMSE).12 In the tables, differences in accuracy that are statistically different from zero are denoted by one, two or three asterisks, corresponding to significance levels of 10%, 5% and 1%, respectively. The underlying p-values are based on t-statistics computed with a serial correlation-robust variance, using the pre-whitened quadratic spectral estimator of Andrews and Monahan (1992).13 Since the AR-MIDAS and the AR-U- MIDAS model nests the ADL model, for the comparison of these models to the ADL, we report p-values based on one-sided tests, taking the ADL as the null and the AR-MIDAS/AR-U- MIDAS as the alternative. Because the ADL and the MIDAS/UMIDAS models are not nested, for these model comparisons we report p-values based on two-sided tests. 11For a discussion of alternative benchmark data vintages, see Stark and Croushore (2002) 12Amisano and Giacomini (2007) extend the results of Giacomini and White (2006) for point forecasts to density forecasts, developing a set of weighted likelihood ratio tests. In our application to tests for density forecasts, we do not employ a weighting scheme (or put another way, we use equal weights). 13The issue of real-time data complicates any assessment of whether the resulting differences in forecast accuracy between models are significant, see Clark and McCracken (2009). 17 5.2 Results Table 5 presents real-time density forecasting performance for the different MIDAS models for the evaluation sample 1985q2-2013q2. Density forecast performance is measured in terms of CRPS and LS. In addition, we provide results for the two quarterly benchmarks (an AR model and an ADL model). The table reveals three interesting results. First, by comparing the forecasting performance from the various MIDAS models with the AR and ADL benchmarks, the table shows sizeable reductions in the CRPS and sizeable increases in the LS for the MIDAS models when monthly data are available on the current quarter. For each model, there is a clear and steady increase in forecasting performance as more information is available, i.e. the nowcasts produced in the third month of the quarter (at horizon hm= 1) are better than the forecasts produced in the first month of the quarter (at horizon hm= 3). Compared to the quarterly-frequency ADL benchmark the density nowcasting improvements for the MIDAS models at horizon hm= 1 are of the order of 15-20 percent in terms of LS and 10-15 percent in terms of CRPS. Results based on the t-tests indicate a significant improvement for several of the models compared to the ADL benchmark.14 In particular, this is the case for model specifications using the CFNAI. Second, a comparison of the forecasting performance of the 4 different MIDAS models for each of the 6 monthly indicators reveals very small differences in terms of forecasting performance. There is a small tendency for MIDAS and AR-MIDAS models to perform relatively better than the U-MIDAS and AR-U-MIDAS when applied to survey data. However, the differences are not significant. Thus, in terms of model comparison, we do not obtain clearly a superior approach. This is consistent with results in Foroni et al. (2013) based on point forecasts comparison. Third, the CFNAI index is the single monthly indicator that improves the density forecasts the most both in terms of LS and CRPS. This holds for all the 4 MIDAS models. This is not very surprising as the CFNAI is a weighted average of 85 existing monthly indicators of national economic activity. Moreover, once the CFNAI is released it also provides information for the current month, while the other variables (except the PFBOS) provide information for the previous month. However, the PFBOS, IP and CU also provide useful information that 14In most cases, the ADL and MIDAS models provide statistically superior forecasts, in terms of the t-type test, to the AR model. 18 improves the forecasts. As a robustness for our results, Table 6 reports results when the evaluation sample ends in 2007q4, i.e., prior to the Great Recession. The absolute performance of the individual models is, as expected, superior compared with the performance when including the crisis period. The table also shows that even when excluding the crisis period, the various MIDAS models outperform the AR benchmark and in most cases also the ADL benchmark. Figures 1 and 2 show how the relative cumulative LS and CRPS from the MIDAS and ARMIDAS regressions have evolved over time, relative to the AR benchmark model. The figures describe the relative forecasting performance using each of the 6 monthly indicators and for each of the 4 forecasting horizons. The measures are constructed so that an increase in the relative value measures a relative improvement in the forecasting performance of the MIDAS and ARMIDAS regressions compared with the AR benchmark. The figures reveal three interesting results. First, for all models and indicators, except for the AR-MIDAS model with the PMI,15 there has been a steady increase in the relative cumulative LS and CRPS. Second, for both the MIDAS and the AR-MIDAS regressions the cumulative LS and CRPS increase the most for CFNAI. This also holds for all forecast horizons and for most of the recursive time period. Thus, the superior forecast performance from models that use the CFNAI is a robust finding. Third, for all indicators and for both the MIDAS and AR-MIDAS regressions, the relative cumulative LS and CRPS increase sharply during the Great Recession, but this increase is then followed by a weak, but persistent, decrease during the recovery period. This may indicate that the monthly indicators are particularly informative about sharp decreases in real output growth. Finally, Table 7 reports real-time point forecasting performance for the different MIDAS models for the evaluation samples 1985q2-2013q2 and 1985q2-2007q4. We measure point forecast accuracy in terms of RMSE. As for the results obtained with density forecasting, there are small differences between the different MIDAS specifications in terms of forecasting performance and there is a clear and steady increase in forecasting performance as more information is available. However, based on the t-test for equality of the average loss as well as the relative forecasting performance of the different MIDAS specifications compared to the benchmarks, the gains from using mixed-frequency information are smaller for point forecasting than for density 15The AR-MIDAS with PMI seems to experience convergence issues similar to what we found in the Monte Carlo experiments in the case of high persistence. 19 forecasting. This indicates that the gains in density forecast accuracy associated with having more and more information on the quarter derive not only from better centering (i.e., better point forecasts) of the forecast densities but also from improvement of the density shape. 6 Conclusions In this paper, we extend the existent literature by using the MIDAS approach to obtain density forecasts. First, we compute density forecasts from different MIDAS models, the classical MIDAS models and the unrestricted version, U-MIDAS. We derive a parametric bootstrap algorithm to compute densities valid for both classes of models when accounting for parameter uncertainty. To compare the different models, we run Monte Carlo simulations that provide evidence on the forecasting performance of the models in finite samples. We find that the results in terms of point and density forecasts are coherent, but do not clearly indicate a superior performance of one of the models under analysis. Finally, in our empirical applications, we compute density forecasts from the different mixed-frequency models for US output growth, considering three typical monthly variables: industrial production, employment, capacity utilization, and three surveys indices: the Purchasing Managers Index, the Chicago Fed National Activity Index and the Philadelphia Fed Business Outlook Survey for general business activity. The results indicate that including monthly survey-based measures largely improves density forecasting performance. 20 References Aastveit, K., K. Gerdrup, A. Jore, and L. Thorsrud (2014). Nowcasting GDP in real time: A density combination approach. Journal of Business and Economic Statistics 32 (1), 48–68. Amisano, G. and R. Giacomini (2007). Comparing density foreasts via weighted likelihood ratio tests. Journal of Business and Economic Statistics 25 (2), 177–190. Andrews, D. and J. Monahan (1992). An improved heteroskedasticity and autocorrelation consistent covariance matrix estimator. Econometrica 60, 953–966. Banbura, M., D. Giannone, and L. Reichlin (2011). Nowcasting. In M. P. Clements and D. F. Hendry (Eds.), The Oxford Handbook of Economic Forecasting. Oxford Handbooks in Economics. Banbura, M. and M. Modugno (2014). Maximum likelihood estimation of factor models on data sets with arbitrary pattern of missing data. Journal of Applied Econometrics 29(1), 133–160. Berkowitz, J. and L. Kilian (2000). Recent developments in bootstrapping time series. Econometric Reviews 19 (1), 1–48. Billio, M., R. Casarin, F. Ravazzolo, and H. K. van Dijk (2013). Time-varying combinations of predictive densities using nonlinear filtering. Journal of Econometrics 177(2), 213–232. Bose, A. (1988). Edegeworth corrections by bootstrap in autorgressive. Annals of Statistics 16(4), 1709–1722. Carriero, A., T. E. Clark, and M. Marcellino (2014). Real-time nowcasting with a Bayesian mixed frequency model with stochastic volatility. Journal of Applied Econometrics, forthcoming. Clark, T. and F. Ravazzolo (2014). The macroeconomic forecasting performance of autoregressive models with alternative specifications of time-varying volatility. Journal of Applied Econometrics (forthcoming). Clark, T. E. (2011). Real-time density forecasts from bayesian vector autoregressions with stochastic volatility. Journal of Business & Economic Statistics 29 (3), 327–341. 21 Clark, T. E. and M. W. McCracken (2009). Tests of Equal Predictive Ability With Real-Time Data. Journal of Business & Economic Statistics 27 (4), 441–454. Clements, M. P. and A. B. Galv˜ao (2008). Macroeconomic forecasting with mixed-frequency data. Journal of Business & Economic Statistics 26, 546–554. Clements, M. P. and A. B. Galv˜ao (2009). Forecasting US output growth using leading indicators: an appraisal using MIDAS models. Journal of Applied Econometrics 24 (7), 1187–1206. Clements, M. P. and N. Taylor (2001). Bootstrapping prediction intervals for autoregressive models. International Journal of Forecasting 17(2), 247–267. Croushore, D. and T. Stark (2001). A real-time data set for macroeconomists. Journal of Econometrics 105(1), 111–130. Davison, A. and D. Hinkley (1997). Bootstrap Methods and their Applications. Cambridge University Press. Diebold, F. X. and R. S. Mariano (1995, July). Comparing predictive accuracy. Journal of Business & Economic Statistics 13 (3), 253–63. Evans, M. D. (2005). Where are we now? Real-time estimates of the macro economy. International Journal of Central Banking 1 (2), 127–175. Foroni, C., M. Marcellino, and C. Schumacher (2013). U-midas: Midas regressions with unrestricted lag polynomials. Journal of the Royal Statistical Society - Series A (forthcoming). Garratt, A., K. Lee, M. H. Pesaran, and Y. Shin (2003). Forecast uncertainties in macroeconomic modeling: An application to the UK economy. Journal of the American Statistical Association 98(464), 829–38. Ghysels, E., P. Santa-Clara, and R. Valkanov (2004). The MIDAS Touch: Mixed Data Sampling Regression Models. CIRANO Working Papers 2004s-20. Ghysels, E., P. Santa-Clara, and R. Valkanov (2005). There is a risk-return trade-off after all. Journal of Financial Economics 76 (3), 509–548. 22 Table 6: Forecast accuracy for US output. Evaluation Sample: 1985q2:2007q4 CRPS Log Score hm -1 1 2 3 -1 1 2 3 ’EMP’ umidas 0.260 0.276 0.294 0.271 -0.678 -0.735 -0.794 -0.817 ar-umidas 0.258 0.272 0.297 0.271 -0.668 -0.706** -0.798 -0.817 midas 0.261 0.268 0.280 0.268 -0.708 -0.733 -0.764 -0.807 ar-midas 0.259 0.271 0.272 0.288 -0.708 -0.740 -0.730** -0.863 ADL 0.250 0.261 0.264 0.268 -0.647 -0.787 -0.791 -0.814 ’IP’ umidas 0.224 0.229 0.248 0.280 -0.570 -0.604** -0.693 -0.837 ar-umidas 0.222 0.229 0.250 0.280 -0.577 -0.602*** -0.686 -0.823 midas 0.220 0.229 0.246 0.264 -0.562 -0.601** -0.685 -0.783 ar-midas 0.220 0.226 0.243 0.276 -0.558 -0.585*** -0.658 -0.809 ADL 0.212 0.247 0.246 0.249 -0.549 -0.732 -0.721 -0.733 ’CU’ umidas 0.226 0.227 0.257 0.275 -0.589 -0.601** -0.709 -0.807 ar-umidas 0.224 0.226 0.254 0.271 -0.583 -0.589*** -0.693 -0.801 midas 0.219 0.224 0.252 0.258 -0.571 -0.595** -0.693 -0.753 ar-midas 0.223 0.224* 0.252 0.277 -0.587 -0.587** -0.692 -0.816 ADL 0.217 0.246 0.248 0.250 -0.567 -0.714 -0.728 -0.734 ’PMI’ umidas 0.261 0.275 0.275 0.305 -0.755 -0.76** -0.775** -0.858 ar-umidas 0.262 0.273* 0.278 0.315 -0.777 -0.755*** -0.77** -0.889 midas 0.24** 0.245*** 0.251** 0.262** -0.712 -0.729*** -0.790 -0.846 ar-midas 0.303 0.340 0.365 0.353 -0.927 -1.019 -1.076 -1.046 ADL 0.267 0.292 0.290 0.293 -0.758 -0.852 -0.839 -0.850 ’CFNAI’ umidas 0.210 0.198 0.206** 0.232 -0.493** -0.471** -0.505*** -0.602*** ar-umidas 0.208 0.202 0.207** 0.228 -0.477*** -0.474** -0.496*** -0.591*** midas 0.194*** 0.193** 0.199** 0.223 -0.459*** -0.463*** -0.461*** -0.556*** ar-midas 0.194*** 0.192*** 0.199*** 0.240 -0.473*** -0.45*** -0.471*** -0.661** ADL 0.222 0.219 0.251 0.252 -0.619 -0.606 -0.729 -0.742 ’PHILLY’ umidas 0.261 0.262 0.263 0.275 -0.708 -0.711 -0.740** -0.784 ar-umidas 0.262 0.263 0.271 0.275 -0.709 -0.703 -0.761* -0.796 midas 0.252 0.254 0.262 0.273 -0.681 -0.695 -0.723** -0.791 ar-midas 0.252* 0.253* 0.257 0.260 -0.675* -0.678** -0.719*** -0.772** ADL 0.263 0.261 0.271 0.274 -0.712 -0.706 -0.830 -0.834 AR 0.301 0.300 0.301 0.345 -0.972 -0.975 -0.972 -1.107 Notes: The table compares the forecasting performance of the four types of MIDAS models described in Section 2 (umidas, ar-umidas, midas, ar-midas), based on CRPS and LS for evaluating density forecasts. The criteria are described in Section 3.2. The results are computed on an evaluation sample from 1985q2 to 2007q4. The results are reported for each of the monthly indicators included in the models. Results for the corresponding ADL model are reported for each indicator and serve as a comparison in our tests. Moreover, the results from an AR process with lag length selected according to the BIC criterion are included for comparison. The table shows the results for the backcast and nowcast. Bold numbers indicate that a MIDAS provide more accurate forecasts than the ADL and AR benchmarks. Differences in accuracy that are statistically different from zero are denoted by one, two or three stars corresponding to significance levels of 10%, 5% and 1%, respectively. The underlying p-values are based on t-statistics computed with a serial correlation-robust variance, using the pre-whitened quadratic spectral estimator of Andrews and Monahan (1992). Since the ar-umidas and ar-midas models nest the ADL model, we report p-values based on one-sided tests, taking the ADL as the null and the MIDAS models as the alternative. Because the umidas and midas models are not nested, for these model comparisons we report p-values based on two-sided tests. 29 Table 7: Forecast accuracy for US output. Point forecasts. Evaluation sample: 1985q2:2013q2 and 1985q2:2007q4 1985 - 2013 1985 - 2007 hm -1 1 2 3 -1 1 2 3 ’EMP’ umidas 0.458 0.489 0.543 0.498 0.452 0.475 0.511 0.442 ar-umidas 0.457 0.494 0.547 0.505 0.452 0.477 0.521 0.446 midas 0.478 0.490 0.524 0.495 0.459 0.470 0.493 0.441 ar-midas 0.476 0.494 0.509 0.552 0.451 0.472 0.477 0.480 ADL 0.432 0.474 0.490 0.494 0.435 0.432 0.441 0.439 ’IP’ umidas 0.419 0.437 0.471 0.462 0.383 0.388 0.427 0.467 ar-umidas 0.413 0.434 0.473 0.466 0.378 0.389 0.430 0.470 midas 0.408 0.430 0.465 0.445 0.371 0.388 0.425 0.440 ar-midas 0.414 0.429 0.457 0.535 0.370 0.385 0.416 0.470 ADL 0.369 0.433 0.436 0.447 0.350 0.404 0.403 0.409 ’CU’ umidas 0.430 0.445 0.494 0.470 0.387 0.389 0.449 0.465 ar-umidas 0.421 0.436 0.484 0.466 0.381 0.384 0.440 0.460 midas 0.420 0.440 0.483 0.447 0.375 0.383 0.440 0.434 ar-midas 0.416 0.432 0.475 0.532 0.376 0.381 0.439 0.468 ADL 0.399 0.438 0.443 0.457 0.363 0.403 0.406 0.409 ’PMI’ umidas 0.466 0.500 0.520 0.558 0.441 0.482 0.482 0.540 ar-umidas 0.467 0.495 0.521 0.579 0.447 0.479 0.488 0.569 midas 0.418** 0.437** 0.45** 0.473* 0.386** 0.405** 0.393** 0.405** ar-midas 0.615 0.677 0.730 0.727 0.525 0.591 0.644 0.613 ADL 0.473 0.530 0.522 0.532 0.461 0.503 0.495 0.503 ’CFNAI’ umidas 0.388 0.377 0.392 0.440 0.367 0.344 0.354 0.406 ar-umidas 0.385 0.380 0.395 0.434 0.366 0.347 0.357 0.397 midas 0.365 0.367 0.381 0.426 0.327 0.324 0.339* 0.385 ar-midas 0.361 0.361 0.379* 0.510 0.325* 0.318* 0.338** 0.415 ADL 0.381 0.374 0.435 0.445 0.360 0.357 0.415 0.418 ’PHILLY’ umidas 0.467 0.469 0.456 0.473 0.460 0.445 0.459 0.444 ar-umidas 0.468 0.471 0.476 0.477 0.458 0.469 0.474 0.451 midas 0.439** 0.445* 0.454 0.471 0.439 0.448 0.455 0.427 ar-midas 0.447** 0.455* 0.452 0.474 0.437* 0.447 0.449 0.402 ADL 0.476 0.471 0.489 0.496 0.462 0.457 0.440 0.450 AR 0.535 0.529 0.533 0.564 0.458 0.453 0.461 0.523 Notes: The table compares the forecasting performance of the four types of MIDAS models described in Section 2 (umidas, ar-umidas, midas, ar-midas), based on RMSPE for evaluating point forecasts. The criterion is described in Section 3.2. The results are computed on an evaluation sample from 1985q2 to 2013q2 and on 1985q2 to 2007q4. The results are reported for each of the monthly indicators included in the models. Results for the corresponding ADL model are reported for each indicator and serve as a comparison in our tests. Moreover, the results from an AR process with lag length selected according to the BIC criterion are included for comparison. The table shows the results for the backcast and nowcast. Bold numbers indicate that a MIDAS provide more accurate forecasts than the ADL and AR benchmarks. Differences in accuracy that are statistically different from zero are denoted by one, two or three stars corresponding to significance levels of 10%, 5% and 1%, respectively. The underlying p-values are based on t-statistics computed with a serial correlation-robust variance, using the pre-whitened quadratic spectral estimator of Andrews and Monahan (1992). Since the ar-umidas and ar-midas models nest the ADL model, we report p-values based on one-sided tests, taking the ADL as the null and the MIDAS models as the alternative. Because the umidas and midas models are not nested, for these model comparisons we report p-values based on two-sided tests. 30 Relative Cumulative CRPS MIDAS AR-MIDAS 1985Q2 1995Q2 2005Q2 2013Q2 −5 0 5 10 15 EMP IP CU PMI CFNAI PFBOS 1985Q2 1995Q2 2005Q2 2013Q2 −5 0 5 10 15 EMP IP CU PMI CFNAI PFBOS hm=-1 1985Q2 1995Q2 2005Q2 2013Q2 −10 −5 0 5 10 15 EMP IP CU PMI CFNAI PFBOS 1985Q2 1995Q2 2005Q2 2013Q2 −10 −5 0 5 10 15 EMP IP CU PMI CFNAI PFBOS hm=1 1985Q2 1995Q2 2005Q2 2013Q2 −15 −10 −5 0 5 10 15 EMP IP CU PMI CFNAI PFBOS 1985Q2 1995Q2 2005Q2 2013Q2 −15 −10 −5 0 5 10 15 EMP IP CU PMI CFNAI PFBOS hm=2 1985Q2 1995Q2 2005Q2 2013Q2 −5 0 5 10 15 EMP IP CU PMI CFNAI PFBOS 1985Q2 1995Q2 2005Q2 2013Q2 −5 0 5 10 15 EMP IP CU PMI CFNAI PFBOS hm=3 Notes: Relative Cumulative CRPS = Pt+h(m) 1(CRP Sl t+h(m)−CRP Si t+h(m)), t = 1, ..., T,l= AR benchmark model and i= alternative information set. The first column shows the results for the MIDAS model and the second column the results for the AR-MIDAS models. The different rows show the results for different forecasting horizons. 31 Cumulative Log Score MIDAS AR-MIDAS 1985Q2 1995Q2 2005Q2 2013Q2 −10 0 10 20 30 40 50 60 EMP IP CU PMI CFNAI PFBOS 1985Q2 1995Q2 2005Q2 2013Q2 −10 0 10 20 30 40 50 60 EMP IP CU PMI CFNAI PFBOS hm=-1 1985Q2 1995Q2 2005Q2 2013Q2 −20 0 20 40 60 EMP IP CU PMI CFNAI PFBOS 1985Q2 1995Q2 2005Q2 2013Q2 −20 0 20 40 60 EMP IP CU PMI CFNAI PFBOS hm=1 1985Q2 1995Q2 2005Q2 2013Q2 −10 0 10 20 30 40 50 60 EMP IP CU PMI CFNAI PFBOS 1985Q2 1995Q2 2005Q2 2013Q2 −10 0 10 20 30 40 50 60 EMP IP CU PMI CFNAI PFBOS hm=2 1985Q2 1995Q2 2005Q2 2013Q2 −20 0 20 40 60 EMP IP CU PMI CFNAI PFBOS 1985Q2 1995Q2 2005Q2 2013Q2 −20 0 20 40 60 EMP IP CU PMI CFNAI PFBOS hm=3 Notes: Relative Log Score = Pt+h(m) 1(ln ftm+hm,tm,i (yt+hm)−ln ftm+hm,tm,l (yt+hm)), t = 1, ..., T ,l= AR benchmark model and i= alternative information set. The first column shows the results for the MIDAS model and the second column the results for the AR-MIDAS models. The different rows show the results for different forecasting horizons. 32