The IWG Forecasting Dashboard: From forecasts to evaluation and comparison

Heinisch, Katja et al. Article The IWG Forecasting Dashboard: From forecasts to evaluation and comparison Journal of Economics and Statistics Provided in Cooperation with: De Gruyter Brill Suggested Citation: Heinisch, Katja et al. (2024) : The IWG Forecasting Dashboard: From forecasts to evaluation and comparison, Journal of Economics and Statistics, ISSN 2366-049X, De Gruyter Oldenbourg, Berlin, Vol. 244, Iss. 3, pp. 277-288, https://doi.org/10.1515/jbnst-2023-0011 This Version is available at: https://hdl.handle.net/10419/333284 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. https://creativecommons.org/licenses/by/4.0/ Data Observer Katja Heinisch*, Christoph Behrens, Jörg Döpke, Alexander Foltas, Ulrich Fritsche, Tim Köhler, Karsten Müller, Johannes Puckelwald and Hannes Reichmayr The IWH Forecasting Dashboard: From Forecasts to Evaluation and Comparison https://doi.org/10.1515/jbnst-2023-0011 Received February 3, 2023; accepted February 4, 2023 Abstract: The paper describes the “Halle Institute for Economic Research (IWH) Forecasting Dashboard (ForDas)”. This tool aims at providing, on a non-commercial basis, historical and actual macroeconomic forecast data for the Germany economy to researchers and interested audiences. The database renders it possible to directly compare forecast quality across selected institutions and over time. It is partly based on data collected in the DFG-funded project “Macroeconomic forecasts in great crisis”. Keywords: forecasting, macroeconomic data We thank Ida Rockenbach, Ronny Ehlen, Aurora Li, Max Weinig, Sebastian Przetak, Christopher Gluth, Tim Bernutz, and Kay Felix Domke for data support. Further, we thank Oliver Holtemöller for his useful comments on implementing the dashboard. *Corresponding author: Katja Heinisch, Halle Institute for Economic Research (IWH), Kleine Maerkerstraße 8, D-06108 Halle (Saale), Germany, E-mail: [email protected]. https://orcid.org/ 0000-0002-2664-4950 Christoph Behrens, Freie und Hansestadt Hamburg, Hamburg, Germany, E-mail: [email protected] Jörg Döpke and Tim Köhler, Hochschule Merseburg, Merseburg, Germany, E-mail: [email protected] (J. Döpke), [email protected] (T. Köhler) Alexander Foltas, Helmut-Schmidt-Universität Hamburg, Hamburg, Germany, E-mail: [email protected]. https://orcid.org/0000-0003-2146-8158 Ulrich Fritsche, Universität Hamburg, Hamburg, Germany, E-mail: [email protected]. https://orcid.org/0000-0003-0492-8300 Karsten Müller, Deutsches Zentrum für Luftund Raumfahrt, Institut für Vernetzte Energiesysteme, Stuttgart, Germany, E-mail: [email protected] Johannes Puckelwald, Deutsches Maritimes Zentrum Hamburg, Hamburg, Germany, E-mail: [email protected] Hannes Reichmayr, Martin-Luther-Universit¨ at Halle-Wittenberg, Halle, Germany, E-mail: [email protected] Journal of Economics and Statistics 2024; 244(3): 277–288 Open Access. © 2023 the author(s), published by De Gruyter. This work is licensed under the Creative Commons Attribution 4.0 International License. JEL Classification: E32, E37, G11 1 Introduction Macroeconomic forecasting is one of the areas of economics that receives the most attention from the media and public discourse. For example, forecast competitions are quite popular: Several newspapers and database providers evaluate forecasters more or less regularly (see Döhrn 2015, for an overview of German rankings). Also, point forecasts by institutions gain a lot of attention from the media and politics. Data providers such as Consensus Economics or Focus Economics use the institutions’ forecasts to produce a mean forecast across several forecasters. As a result, a significant part of the scientific literature is focused on evaluating these forecasts. However, typically forecasts are evaluated against a theoretical benchmark like a naive forecast but rarely across forecasting institutions. One reason for the relative scarcity of such analyses is the lack of non-commercial databases that include several institutions, macroeconomic indicators, and forecast periods. The IWH Forecasting Dashboard (ForDas) by the Halle Institute for Economic Research aims to fill the data gap by providing historical and actual data for this purpose. 1 Furthermore, on can directly compare forecasting quality of institutions. The dashboard relies partly on data collected within the DFG project “Macroeconomic forecasts in great crisis”. 2 The IWH took over the data collection process in 2020, continues with it, and has implemented a tool to visualize the data. Due to data gaps and different dimensions (for example, growth vs. level predictions) provided by several institutions, the dashboard only shows a selection of variables. In the following, we describe the main contents 3 of the database and the IWH Forecasting Dashboard. Since the database was initiated first within the context of economic history, we aimed at collecting all quantitative forecasts made for the German economy by institutions relevant to economic policy. As (Antholz 2006) points out, these forecasts started in the early 1960s. While it was possible to find texts describing the prospects of the German economy before, they rarely included any concrete numbers for GDP growth or inflation. Hence, analyses and comparisons based on ForDas data can start in 1965 at the earliest for selected forecasters. 4 1https://www.iwh-halle.de/fordas. 2This project has been part of the Priority Programme “Experiences and Expectations: Historical Foundations of Economic Behaviour”(SPP 1859) funded by the Deutsche Forschungsgemeinschaft (DFG). 3Please note, that ForDas does not yet contain the full forecast text reports. This is however planned for the future. 4In the 1970s, figures were often provided as a fractional number and not decimal number, which is line with a so called forecast confidence interval. For instance, a value of 3 1 2is rewritten to 3.5 during the data collection, however, policy makers should have in mind a range between 3.35 and 3.65. 278 K. Heinisch et al. 2 Contents of the Database The main objective of the DFG project was to build a macroeconomic forecast database, including a broad set of forecasts for Germany over the longest possible period, achieving consistency within the dataset as far as possible. In combination with user-friendliness and free access to the data, the database supplies a new instrument for forecast evaluation, made for the scientific community, media, or the public in general. To this end, the forecast database covers the key macroeconomic variables at each point in time, including variables related to national accounts, financial and monetary variables, and also variables related to labor and (un-)employment. We choose the institutions according to their long-term experience in forecasting and their relevance for economic policy support in Germany. The field of macroeconomic forecasting has markedly grown over time. Therefore, the quantity of forecasts and forecasters has also increased substantially. To give an impression of the growth of the “forecasting industry”during the period covered, Figure 1 shows the sheer number of forecasts regarding the headline measure of economic growth per year included in the database. For the larger part of the period, this headline measure was the rate of change of real GDP. In a smaller part of the sample, however, real GNP growth served as the most prominent figure in this context and is, thus, also taken into account. The number of forecasts is both driven by the number of forecasting institutions as well as the forecast frequency per year. While at the beginning of the sample, forecasts are published once per year, while up to four forecasts are Figure 1: Number of forecasts for the headline figure of economic growth in the database, 1965 to 2019. Source: Own compilation. The IWH Forecasting Dashboard 279 published recently by forecasters. The number of variables of interest for the forecasters also grew over time. Definitions and names of variables varied over time as well, which to some extent reflected changes in the system of national accounts. 2.1 Forecasters Covered The database includes forecast data on the German economy covered by 15 national and international institutions with different institutional backgrounds: –The six largest German economic research institutes: German Institute for Economic Research (DIW), ifo Institute for Economic Research (ifo), RWI –Leibniz Institute for Economic Research (RWI), Halle Institute for Economic Research (IWH), Hamburg Institute of International Economics (HWWA, since 2007 HWWI), and Kiel Institute for the World Economy (IfW). –Research institutes related to trade unions or employers’associations: Macroeconomic Policy Institute (IMK) and German Economic Institute (IW). 5 –Institution related to policy or policy advice: the Deutsche Bundesbank (German central bank BBK), the joint forecast (GD) of the leading research institutes, and the German Council of Economic Experts (SVR). –International organisations, namely: the Organisation for Economic Cooperation and Development (OECD), the International Monetary Fund (IMF), and the European Commission (EC). 6 2.2 Variables Included The database covers key macroeconomic variables (Table 1). In addition, financial and monetary variables as well as data related to trade, labor, employment, and unemployment are included. 7 However, note that not all forecasting institutions 5Institutions formerly involved in forecasting are also covered, i.e. the Wirtschafts-und sozialwissenschaftliches Institut in der Hans-Boeckler-Stiftung (WSI) up to 2004. Although still existing, the WSI institute has not provided business cycle forecasts since the IMK came into existence. Second, the Hamburger Weltwirtschaftsarchiv (HWWA) until 2006. This institute was mainly funded by public money. From 2007 onwards, the institute was renamed to HWWI and functions as a privately funded institute. 6Note, that the forecasts of the EC are part of the IWH Forecasting Dashboard, but not of the data set used in the DFG project. 7For the sake of brevity, in this paper we describe the most recent version of the database only, namely the one accessible over the IWH homepage as explained below. Therefore, in the IWH Forecasting Dashboard, only a selection of variables is chosen that is covered by most of the forecasters. A version with a broader dataset, in particular older data, collected for the DFG project, can be found in the project’s data repository “Emporion”, see: https://emporion.gswg.info. 280 K. Heinisch et al. mentioned above provide forecasts for all variables in all periods. 8 Furthermore, institutions might provide forecasts either in levels or growth rates. 2.3 Realisations (Actual Data) and Forecast Horizons For the comparison of the forecast data with the actual economic development, the so-called “real-time”data problem is an important issue (Stark and Croushore 2002) in economic forecasting and forecast evaluation. Since time series are prone to revisions, the database provides realisations for selected important series in two Table :Overview of the variables in ForDas. Components of Real GDP Gross domestic product (constant prices) Final consumption expenditure of households and NPISHs (constant prices) Government final consumption expenditure (constant prices) Gross fixed capital formation (constant prices) Gross fixed capital formation in machinery and equipment (constant prices)a Construction (constant prices) Gross fixed capital formation in other fixed assets (constant prices) (since ) Exports of goods and services (constant prices) Imports of goods and services (constant prices) Labor market Persons in employment Unemployment registered (persons) according to “Bundesagentur für Arbeit” Unemployment registered (ratio) Unemployment according to international labor organisation Unemployment rate Memorandum Items Consumer price index GDP deflator Unit labor costs Net lending/net borrowing as a percentage of GDP Current account balance as a percentage of GDP Technical Assumptions Oil price Exchange rate Policy interest rate World trade growth Own compilation. aNote that forecasters have often summarised Gross fixed capital formation in machinery and equipment and gross fixed capital formation in other products until . 8See the data availability sectionof the IWH Forecasting Dashboardto see the time span andmissing observations for selected variables and institutions. The IWH Forecasting Dashboard 281 variants: first, the initial publication (first release) and, second, the revised data (current data vintage) by the German statistical office, if available. The database includes information on three possible forecast horizons: the current year, i.e. the year in which the forecast is released (labeled t 0 in the database), a forecast for the next year (t 1 ), and the year after the next year (t 2 ). Hence, following the practice of the institutions covered, the forecasts in the database are “fixed event” rather than “fixed horizon”predictions (see, e.g. Knüppel and Vladu 2016). However, most of the institutions publish multiple forecasts per year (forecast rounds). Therefore, the exact date of forecast publication is stored, to distinguish different forecast rounds (e.g. quarters, months). In addition, if available, the date on which the forecast was completed is reported. Generally, the database refers to the name of a variable as it was at the date of the production of the forecast. 9 In a similar vein, all dimensions refer to the date of the forecasts. Thus, variables are expressed in Deutsche Mark up to 2000, and in Euros after. Real variables usually refer to the respective base year. Around German reunification, the switch from forecasts referring to West Germany to predictions for Germany as a whole differs by series and by the institution and is, hence, noted similarly for each series. 10 Figure 2 shows the real GDP growth forecasts for Germany, provided by the economic research institutes as well as the realised values from 2001 to 2022. The Figure 2: Realised growth rate and forecasts, 2000 to 2022. Source: IWH Forecasting Dashboard, 2023. 9This is important for cases in which the official name in the national accounts has changed. For example, until a certain date, the database refers to “Gross national product”, followed by “Gross National Income”in later years. 10 The disentanglement from West German to German forecasts is not uniform in the period of unification across forecasters. Therefore, we excluded the year 1991 for the calculation of forecast errors in the IWH Forecasting Dashboard. 282 K. Heinisch et al. forecasts have been conducted in autumn (months 9, 10) of the previous year. The black line represents the realised growth rate. The figure illustrates, at first glance, some key insights regarding economic forecast evaluation. First, the forecasts are relatively close to each other, and it seems that they do not differ significantly over time. Second, the prediction of economic turning points (and/or recessions) is still a big challenge in economic forecasting. Different forecasting dates and diverging information sets seem to be important in determining forecast accuracy. 3 Recent and Possible Applications The IWH Forecasting Dashboard provides the basis for various potential research questions, most obviously evaluating German business cycle forecasts based on their accuracy or efficiency. Potential other uses could be the investigation of the institutes themselves, their behavior, and change after economically significant events or due to paradigmatic shifts. An additional research field concerns assessing business cycle forecasts’benefits for economic agents. Several studies have already produced scientificfindings based on the data available on the IWH Forecasting Dashboard. (Köhler and Döpke 2023) use the IWH Forecasting Dashboard to conduct an overall ranking of 14 institutions from 1993 to 2019 according to their forecast accuracy. They report substantial long-run differences in forecasting quality, which they mostly attribute to distinct average forecast horizons. Therefore, they cannot single out institutions as being superior at predicting the German economy. (Engelke et al. 2019) examine the extent to which initial assumptions that prove incorrect ex-post drive economic forecast errors. Based on an unbalanced panel of annual forecasts from different institutions forecasting German GDP and the underlying assumptions, they found that over 75% of squared errors of the GDP forecast co-move with the squared errors in their underlying assumptions. This finding implies that the accuracy of the assumptions is of great importance and that forecasters should reveal the framework of their assumptions in order to obtain useful policy recommendations based on economic forecasts. The impact of the Great Recession on forecast accuracy for growth and inflation and forecaster behaviour are investigated by (Döpke et al. 2019) using a data panel from 1971 to 2017. The authors report stable accuracy for growth forecasts, but slightly lower precision for inflation forecasts. More significantly, they report that the loss function has changed after the Great Recession, leading to more pessimistic forecasts from German professional forecasters. (Behrens et al. 2018a) evaluate whether growth and inflation forecasts are efficient or optimal, which requires that the information available at the time of forecast creation has no explanatory power for the corresponding forecast error. The The IWH Forecasting Dashboard 283 joint forecast efficiency evaluation shows heterogeneity across the institutes, with different institutes conducting inefficient forecasts for different prognosis horizons. (Behrens et al. 2020) confirm this result, which extends the previous study with various scenarios and robustness checks, rejecting strong and weak forecast efficiency of growth and inflation forecasts in multiple cases. Additionally, the authors show in an out-of-sample experiment that a Bayesian additive regression trees (BART) model produces significantly more accurate forecasts. (Behrens 2020) finds trade forecasts similarly heterogeneously inefficient. Remarkably, the forecasters include typical trade predictors more efficiently than macroeconomic variables in their export and import forecasts. Further research examines forecast efficiency while assuming a flexible instead of a symmetric (quadratic) loss function. For this purpose, the researchers test whether the set of predictors has predictive value for the sign of the forecast error using random decision forests. Re-evaluating inflation forecast optimality, (Behrens et al. 2018b) suggest that short-term inflation forecasts are suboptimal for some institutes while failing to reject the null hypothesis for long-term forecasts. Reconsidering trade forecasts, (Behrens 2019) rejects optimality only in one case, thus supporting a more favorable assessment of forecasts if flexible loss functions are assumed. Several studies have implemented textual business cycle reports using natural language processing (NLP) in their forecast efficiency analyses. While the written reports are not part of the IWH ForDas itself, they are source and rationale of the forecasts, making a combined analysis reasonable. (Müller 2022) transforms the written accounts of forecasters’expectations into sentiment indices using nine different methods. The author demonstrates that several indices can improve the accuracy of German business cycle forecasts proving that forecasters do not fully exploit the information content of their business cycle reports for their numerical point forecasts. (Foltas 2022) uses the Word2Sense-LDA topic model developed specifically for this task to measure the proportions of different economic topics in each business cycle report and uses their shift to test investment forecast efficiency. In some cases, the author rejects forecast efficiency with topics as the most important predictors supporting the thesis that institutes inefficiently incorporate qualitative information discussed in their business cycle reports into point forecasts. With an approach using topics as the sole predictors of the forecast error, (Foltas and Pierdzioch 2022a) affirm the usefulness of topic modeling for forecast efficiency analysis. The authors find several interpretable topics related to the forecast error under symmetric and flexible loss functions. Lastly, (Foltas and Pierdzioch 2022b) utilise a mixed sample of indicators and topics to predict growth forecast errors using quantile random forests and out-of-sample density forecasts. Even though none of the topics are among the top predictors, their aggregated relative importance varied between roughly 30–50%. 284 K. Heinisch et al.