Can population projections be used for sensitivity tests on policy models?

Bryant, John Working Paper Can population projections be used for sensitivity tests on policy models? New Zealand Treasury Working Paper, No. 03/07 Provided in Cooperation with: The Treasury, New Zealand Government Suggested Citation: Bryant, John (2003) : Can population projections be used for sensitivity tests on policy models?, New Zealand Treasury Working Paper, No. 03/07, New Zealand Government, The Treasury, Wellington This Version is available at: https://hdl.handle.net/10419/205512 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. https://creativecommons.org/licenses/by/4.0/ Can population projections be used for sensitivity tests on policy models? John Bryant N EW Z EALAND T REASURY W ORKING P APER 03/07 J UNE 2003 NZ TREASURY WORKING PAPER 03/07 Can population projections be used for sensitivity tests on policy models? MONTH / YEAR June 2003 AUTHOR John Bryant The Treasury PO Box 3724 Wellington New Zealand Email Telephone Fax [email protected] +64 4 917 7027 +64 4 473 1151 ACKNOWLEDGEMENTS John Creedy, Dharma Dharmalingam, John Janssen, and Veronica Jacobsen provided helpful comments on earlier versions of this paper. NZ TREASURY New Zealand Treasury PO Box 3724 Wellington 6008 NEW ZEALAND Email Telephone Website [email protected] 64 4 472 2733 www.treasury.govt.nz DISCLAIMER The views expressed in this Working Paper are those of the author and do not necessarily reflect the views of the New Zealand Treasury. The paper is presented not as policy, but with a view to inform and stimulate wider debate. WP 03/07 | CAN POPULATION PROJECTIONS BE USED FOR SENSITIVITY TESTS ON POLICY MODELS? i Abstract Many policy models require assumptions about future population trends. Sensitivity tests for these assumptions are normally carried out by comparing population projection variants. This paper outlines some of the conditions that variant-based sensitivity tests must meet if they are to be informative. It then describes four common situations where these conditions are not met, so that conventional sensitivity tests are not informative. The solution, the paper argues, is stochastic population projections. JEL CLASSIFICATION C520 - Model Evaluation and Testing E170 - Forecasting and Simulation J110 - Demographic Trends and Forecasts KEYWORDS Demography; Sensitivity testing; Population projections; Policy modelling WP 03/07 | CAN POPULATION PROJECTIONS BE USED FOR SENSITIVITY TESTS ON POLICY MODELS? ii Table of Contents Abstract...............................................................................................................................i Table of Contents ..............................................................................................................ii List of Tables......................................................................................................................ii List of Figures....................................................................................................................ii 1 Introduction ..............................................................................................................1 2 A framework for assessing demographic sensitivity tests on policy models.......................................................................................................................2 3 Fertility, mortality, and migration assumptions exclude random shocks ..........5 4 ‘Low-low or ‘high-high’ variants are not calculated..............................................6 5 Variants’ rankings differ with the outcome considered........................................7 6 Fertility only changes early in the projection period............................................8 7 A solution: Stochastic population projections....................................................10 8 References..............................................................................................................12 List of Tables Table 1 - Coverage of possible scenarios and the implications for sensitivity testing........................3 Table 2 - Interpretation of results from a sensitivity test .....................................................................4 Table 3 - Statistics New Zealand projections variants for total population and dependency ratios in 2021.............................................................................................................................8 List of Figures Figure 1 - Demographic sensitivity tests for policy models.................................................................2 Figure 2 - Estimates and 1999-base projection assumptions for net permanent and long term migration into New Zealand......................................................................................................5 Figure 3 - Simulation of the effects of a fertility decline ......................................................................8 Figure 4 - Statistics New Zealand fertility assumptions and projected values for percentage of population aged 15-64, 1999(base)-2101 projections..........................................................9 WP 03/07 | CAN POPULATION PROJECTIONS BE USED FOR SENSITIVITY TESTS ON POLICY MODELS? 1 Can population projections be used for sensitivity tests on policy models? 1 Introduction Many policy models require projections of future population size and structure. Macroeconomic models that include variables for the labour force or the size of the tax base, for instance, require data on the size and age-distribution of the working-age population. Forecasts of future needs for hospitals, schools, or prisons all require data on potential occupants. Sometimes modellers use only one projection variant, typically the ‘central’, ‘median’, or ‘medium’ series prepared by the relevant statistical agency. Often, however, modellers require some indication of how uncertainty concerning the demographic variables affects the robustness of the model results. The standard tool for doing so is projection variants. Projection variants are generated by varying assumptions about future paths for fertility, mortality, and migration. The status and interpretation of the projection variants is often ambiguous. Demographers are generally unwilling to attach explicit probabilities to the variants, and commentaries on the projections often warn the reader that the variants are hypothetical scenarios rather than predictions. However, the commentaries often refer to some variants as more plausible than others, or state that certain events, such as the population reaching a given level, are ‘likely’. Most modellers appear to take a pragmatic stance towards these conceptual ambiguities. They enter the variants into their policy models, and compare the outcomes. If the outcomes are similar for all variants, modellers state that their forecasts are insensitive to demographic uncertainty. If the outcomes differ, modellers warn their readers accordingly and call for further research. Few modellers give any indication that they are dissatisfied with this situation. This paper argues that the conventional approach is seriously flawed. It presents examples in which population variants provide a misleading indication of uncertainty about demographic variables. The paper explores the underlying reason for these problems, and argues that they prevent effective sensitivity testing. The paper considers only national projections. Projections for groups of countries or for regions within countries involve addition difficulties, discussed in Lee (1998: 164-5), Bongaarts and Bulatao (2000: 198), and Siegel (2002: 460-82). WP 03/07 | CAN POPULATION PROJECTIONS BE USED FOR SENSITIVITY TESTS ON POLICY MODELS? 2 Section 2 of the paper sets out a framework for assessing the value of demographic sensitivity tests with policy models. The four subsequent sections describe specific problems. Section 3 describes how the omission of random shocks from trajectories for fertility, mortality, and migration leads to actual population sizes exceeding all projected population sizes soon after the projections are published. Section 4 describes how the absence of low-fertility, low-mortality variants and high-fertility, high-mortality variants reduces the range covered by projected dependency ratios. Section 5 looks at how variants that bracket a substantial range for one population variable may bracket only a narrow range for another population variable. Section 6 examines how the practice of restricting fertility changes to the beginning of the projection interval can lead to confusing results for trends in age structure. The paper concludes with a discussion of stochastic population projections, which are a promising alternative to the variants approach. 2 A framework for assessing demographic sensitivity tests on policy models Figure 1 shows the steps involved in demographic sensitivity testing. The population projections are generally carried out by the relevant statistical agency. Future paths for fertility, mortality, and migration are chosen. These are entered into a population projection model, such as the standard ‘cohort components’ model,1 and future paths for population size and structure are derived. These paths are the ‘population variants’ referred to in the population projections literature. Complete descriptions of the variants consist of variables giving the size of each age-sex group, in each year of the projection. Many derived variables are, however, produced, such as dependency rates, numbers of school-age children, or total population size. Figure 1 - Demographic sensitivity tests for policy models Users of policy models generally take the population variants as given. Some models require all the detail produced in the population projections. Typical health expenditure models, for instance, require population numbers for every age-sex group. Other models require only a few derived variables. Some macroeconomic models, for instance, require nothing more than numbers for the total and working-age population. Paths for the required variables are entered into the policy model and the results compared, in an attempt to learn something about the sensitivity of the model’s results to demographic uncertainty. 1 Typically, a path for fertility, mortality, or migration is specified using a single variable: mortality paths, for instance, are often specified using life expectancy. To carry out population projections, entire schedules of age-sex-specific rates are needed. These schedules are derived from a model relating overall levels to underlying rates (see, for instance, Lee and Carter 1992). The discussion in this paper implicitly treats such models as part of the overall population projection model. Paths for fertility, mortality, and migration Population projection model Policy model Paths for population variables used in polic y model Paths for policy variables WP 03/07 | CAN POPULATION PROJECTIONS BE USED FOR SENSITIVITY TESTS ON POLICY MODELS? 3 Under what conditions does this procedure in fact provide informative results? Table 1 presents a simple typology of cases arising during sensitivity testing, and shows admissible conclusions under each case. Table 1 - Coverage of possible scenarios and the implications for sensitivity testing Coverage of empirically possible scenarios Case Fertility, mortality, and migration Population variables used in policy model Outcome variables from policy model Conclusion about sensitivity of policy model to demographic assumptions 1 Wide Wide Wide Sensitive 2 Wide Wide Narrow Insensitive 3 Narrow Wide Wide Sensitive 4 Narrow Wide Narrow Insensitive 5 Narrow Narrow Wide Sensitive 6 Narrow Narrow Narrow No conclusion possible In Case 1, coverage of empirically possible scenarios for fertility, mortality, and migration is wide. In other words, the sets of fertility, mortality, and migration assumptions that are entered into the population projection model in Case 1 jointly cover a broad range of plausible conditions. In Case 1, coverage of possible scenarios for population variables is also wide. This is likely when coverage of possible fertility, mortality, and migration scenarios is also wide, since changes in population size and age structure are completely determined by changes in fertility, mortality, and migration. Finally, in Case 1, coverage of possible outcomes from the policy model is wide. Entering different population scenarios into the policy model gives substantially different outputs. The correct conclusion is that the policy model’s results are sensitive to demographic assumptions. Uncertainty over future demographic variables carries through to uncertainty about the model results. Case 2 is identical to Case 1, except that the range of outcomes from the policy model is narrow. Entering different population inputs into the policy model has little effect on the model outcomes, even though the population inputs cover a wide range of possible cases. The model user is entitled to infer that the model is insensitive to demographic assumptions. This is the result modellers generally prefer. In Cases 3 and 4, coverage of possible fertility, mortality, and migration scenarios is narrow. Coverage of possible scenarios for the population variables used in the policy model is, however, wide. This combination of wide and narrow coverage does arise in practice. One example is when low and high migration assumptions differ markedly, and the only population variable used in the policy model is total population size. Wide and narrow coverage of the outcomes from the policy model lead to the same conclusions about the sensitivity of the model in theses cases as they do in Cases 1 and 2. In Case 5, the narrow coverage of fertility, mortality, and migration carries through to coverage of population variables. Coverage of policy model outcomes is, nevertheless, wide. The fact that outcomes from the policy model vary substantially even when the population scenarios vary relatively little implies that the model is definitely sensitive to demographic uncertainty. Finally, in Case 6, coverage is narrow for fertility, mortality, and migration, and for the population variables, and for model outcomes. The narrow coverage of model outcomes is consistent with the model being insensitive to the demographic assumptions, but it may WP 03/07 | CAN POPULATION PROJECTIONS BE USED FOR SENSITIVITY TESTS ON POLICY MODELS? 4 simply reflect the fact that the population scenarios entered into the model covered only a small proportion of the plausible range. No conclusion about the policy model’s sensitivity to demographic assumptions is therefore possible. Cases 1 and 2, in which wide coverage of possible fertility, mortality, and migration scenarios ensures wide coverage of possible population scenarios, never occurs when working with population variants. As later sections of this paper illustrate, this is because the set of plausible scenarios for fertility, mortality, and mortality trajectories is too large and multi-dimensional to be adequately represented by a small number of population variants. Work with population variants only leads, then, to Cases 3-6. Whether or not modellers draw the correct conclusions from these cases depends on which cases the modellers believe to have occurred. The extent to which the policy model produces a wide range of outcomes during demographic sensitivity testing is readily observable. Modellers who encounter Cases 3 or 5 and observe a wide range of outcomes are therefore likely to assume that one or other of these cases has occurred; if the modellers are unfamiliar with the limitations of population variants, they might also assume that Case 1 has occurred. Similarly, modellers who encounter Cases 2 or 4 and observe a narrow coverage of possible outcomes are likely to assume that one of Cases 2, 4, or 6 has occurred. Table 2 - Interpretation of results from a sensitivity test Case that actually occurred Case that modeller believes occurred Modeller’s interpretation of the sensitivity test 3 or 5 1, 3, or 5 Correctly concludes that model sensitive to demographic uncertainty 4 2 or 4 Correctly concludes that model insensitive to demographic uncertainty 4 6 Incorrectly concludes that test uninformative 6 2 or 4 Incorrectly concludes that model insensitive to demographic uncertainty 6 6 Correctly concludes that test uninformative Table 2 shows the possible combinations of cases and modellers’ beliefs, and the consequences for the correctness of the modellers’ interpretations. The first row of the table shows what happens when Cases 3 or 5 occur. Regardless of whether the modellers believe that Case 1, 3, or 5 has occurred, they still conclude, correctly, that the policy model is sensitive to demographic uncertainty. The second and third rows show combinations occurring under Case 4, when the policy model produces only a narrow range of outcomes. If the modellers assume that Cases 2 or 4 have occurred, they conclude, correctly, that the model is insensitive to demographic uncertainty. If, however, the modellers assume that Case 6 has occurred, so that the narrow range of outcomes is simply a result of a narrow range of population scenarios, they conclude, incorrectly, that the test is uninformative. Finally, the fourth and fifth rows of the table show combinations occurring under Case 6, when the range of population scenarios and model outcomes are both narrow. If modellers overlook the narrow range of population outcomes, and assume that Cases 2 or 4 have occurred, they conclude, without warrant, that the model insensitive to demographic uncertainty. If they assume that Case 6 has occurred they reach the correct but unhelpful conclusion that the test is uninformative. WP 03/07 | CAN POPULATION PROJECTIONS BE USED FOR SENSITIVITY TESTS ON POLICY MODELS? 11 such as fertility, are perfectly correlated, while adjacent years are partly correlated, and distinct variables, such as fertility and mortality, are uncorrelated (Lee 1998). The fertility, mortality, and migration trajectories are entered into standard population projection models, to produce large sets of population projections. Demographers summarize these sets by calculating means, variances, and confidence intervals for key variables such as population size and the dependency rate. Carrying out sensitivity tests on a policy model is simplest when the model requires only the key variables. In this case, users can simply enter values that, on the basis of the variance and confidence intervals, appear suitably extreme. Testing is more difficult with models that require highly detailed demographic inputs, such as models of health expenditure. Users may, in this case, need to enter the full set of population projections, rather than summary statistics. Stochastic population projections can, accordingly, be unwieldy. They are also technically demanding, and, in the case of time series methods, require long series of historical data. Furthermore, existing methods for randomly generating fertility, mortality, and migration paths are still not entirely satisfactory. Even with time series methods, for instance, users still need to specify a long-term trend level for fertility (Lee and Tuljapurkar 1994). Some demographers argue, in addition, that the assumption of perfect correlations between agespecific rates can and should be relaxed (Booth, Maindonald, and Smith 2002). Stochastic population projections do, however, allow the user to obtain Cases 1 and 2 of the typology in Table 1. These are the cases in which the plausible ranges for fertility, mortality, and migration, and hence the ranges for the population variables, are adequately covered. Adequate coverage of these ranges means that the results from sensitivity tests can be interpreted easily and safely. Stochastic population projections can put sensitivity testing on a surer footing. Applications of stochastic population projections to important policy questions have begun to appear. The United States Congressional Budget Office (2001), for instance, has used stochastic population projections to forecast social security expenditures. The New Zealand Treasury has carried out similar work for social expenditures by the New Zealand government (Creedy and Scobie 2002). Variant-based population projections are still, however, more commonly used than stochastic projections. Demographers sometimes try to promote greater use of stochastic projections by pointing out that stochastic projections have clearer conceptual status than variant-based projections, or by noting how stochastic projections can be incorporated into an elegant Bayesian decision-making framework (Tuljapurkar 1992). It seems unlikely, however, that practical minded users of policy models will be persuaded that these benefits outweigh stochastic projections’ additional costs. Users of policy models may be more interested in the capacity of variant-based and stochastic projections to support meaningful sensitivity tests. On this measure, stochastic projections clearly outperform variant-based projections. This suggests that stochastic projections will become increasingly popular. WP 03/07 | CAN POPULATION PROJECTIONS BE USED FOR SENSITIVITY TESTS ON POLICY MODELS? 12 8 References Bongaarts, J and R A Bulatao (2000) Beyond Six Billion: Forecasting the World's Population. (Washington DC: National Academy Press). Booth, Heather, John Maindonald, and Len Smith (2002) "Applying Lee-Carter under conditions of variable mortality decline." Population Studies 56(3): 325-336. Congressional Budget Office (2001) "Uncertainty in Social Security's Long-Term Finances: A Stochastic Analysis." Washington DC, Congressional Budget Office, Working Paper. Lee, Ronald and Lawrence Carter (1992) "Modelling and Forecasting US Mortality." Journal of the American Statistical Association 87(419): 659-671. Lee, Ronald D (1998) "Probabilistic approaches to population forecasting." in Wolfgang Lutz, James W Vaupel and Dennis A Ahlburg (eds) Frontiers of Population Forecasting (New York: The Population Council). Lee, R and S Tuljapurkar (1994) "Stochastic projections for the United States: Beyond high, medium, and low." Journal of the Americal Statistical Association 89(428): 1175-1189. Lutz, Wolfgang and Sergei Scherbov (1998) "An expert-based framework for national population projections: The example of Austria." European Journal of Population 14(1): 1-17. Lutz, Wolfgang, James W Vaupel and Dennis A Ahlburg (1998) Frontiers of Population Forecasting. (New York: The Population Council). Lutz, Wolfgang, Warren Sanderson and Sergei Scherbov (2001) "The end of world population growth." Nature 412: 543-545. OECD (1998) Maintaining Prosperity in an Ageing Society. (Paris and Washington, DC: Organisation for Economic Cooperation and Development). Siegel, Jacob S (2002) Applied Demography: Applications to Business, Government, Law, and Public Policy. (San Diego: Academic Press). Statistics New Zealand, ‘Demographic Trends 2001: All Tables’. Document downloaded from Statistics New Zealand website www.stats.govt.nz in August 2002. Statistics New Zealand, ‘National Population Projections, 1999(base)-2101: All Tables’ Document downloaded from Statistics New Zealand website www.stats.govt.nz in August 2002. Statistics New Zealand, ‘Key demographic indicators, 1999-2002’. Page on Statistics New Zealand website www.stats.govt.nz. Document downloaded in April 2003. Tuljapurkar, Shripad (1992) "Stochastic population forecasts and their uses." International Journal of Forecasting 8(3): 385-91.