Debiasing policymakers: The role of behavioral economics training

Rojas, Ana María; Scartascini, Carlos G. Working Paper Debiasing policymakers: The role of behavioral economics training IDB Working Paper Series, No. IDB-WP-1595 Provided in Cooperation with: Inter-American Development Bank (IDB), Washington, DC Suggested Citation: Rojas, Ana María; Scartascini, Carlos G. (2024) : Debiasing policymakers: The role of behavioral economics training, IDB Working Paper Series, No. IDB-WP-1595, Inter-American Development Bank (IDB), Washington, DC, https://doi.org/10.18235/0012888 This Version is available at: https://hdl.handle.net/10419/299407 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. https://creativecommons.org/licenses/by/3.0/igo/ Debiasing Policymakers: The Role of Behavioral Economics Training A na María Rojas M. Carlos Scartascini WORKING PAPER No IDB-WP-1595 InterA merican Development Bank Department of Research and Chief Economist April 2024 * World Bank ** Inter-American Development Bank Debiasing Policymakers: The Role of Behavioral Economics Training A na María Rojas M.* Carlos Scartascini** InterA merican Development Bank Department of Research and Chief Economist April 2024 Cataloging-in-Publication data provided by the Inter-American Development Bank Felipe Herrera Library Rojas Méndez, Ana María. Debiasing policymakers: the role of behavioral economics training / Ana María Rojas M., Carlos Scartascini. p. cm. — (IDB Working Paper Series ; 1595) Includes bibliographical references. 1. Public health-Decision making -Latin America. 2. Public health-Decision making-Caribbean Area. 3. Economics-Psychological aspects-Latin America. 4. Economics-Psychological aspects-Caribbean Area. I. Scartascini, Carlos G., 1971II. Inter-American Development Bank. Department of Research and Chief Economist. III. Title. IV. Series. IDB-WP-1595 http://www.iadb.org Copyright © 2024 Inter-American Development Bank ("IDB"). This work is subject to a Creative Commons license CC BY 3.0 IGO (https://creativecommons.org/licenses/by/3.0/igo/legalcode). The terms and conditions indicated in the URL link must be met and the respective recognition must be granted to the IDB. Further to section 8 of the above license, any mediation relating to disputes arising under such license shall be conducted in accordance with the WIPO Mediation Rules. Any dispute related to the use of the works of the IDB that cannot be settled amicably shall be submitted to arbitration pursuant to the United Nations Commission on International Trade Law (UNCITRAL) rules. The use of the IDB's name for any purpose other than for attribution, and the use of IDB's logo shall be subject to a separate written license agreement between the IDB and the user and is not authorized as part of this license. Note that the URL link includes terms and conditions that are an integral part of this license. The opinions expressed in this work are those of the authors and do not necessarily reflect the views of the Inter-American Development Bank, its Board of Directors, or the countries they represent. Abstract Behavioral biases often lead to suboptimal decisions, a vulnerability that extends to policymakers who operate under conditions of fatigue, stress, and time constraints and with significant implications for public welfare. While behavioral economics offers strategies like default adjustments to mitigate decision-making costs, deploying these policy interventions is not always feasible. Thus, enhancing the quality of policy decision-making is crucial, and evidence suggests that targeted training can boost job performance among policymakers. This study evaluates the impact of a behavioral training course on policy decision-making through a randomized experiment and a survey test that incorporates problem-solving and decision-making tasks among approximately 25,000 participants enrolled in the course. Our findings reveal a significant improvement in the treated group, with responses averaging 0.6 standard deviations better than those in the control group. Given the increasing prevalence of such courses, this paper underscores the potential of behavioral training in improving policy decisions and advocates for further research through additional experimental studies. JEL classifications: H83, Z18 Keywords: Experimental design, Behavioral economics, Training, Public policy, Government officials The authors are grateful to Karina Marquez Guerra and Andr´es Bari˜nas for superb research assistance, the many behavioral specialists at the IDB who worked with us during different s tages o f t he project, to Indhira Ramirez, Josue Mendoza, the team at eMBeD-WB, and to the LAER Special Issue workshop participants for their comments and suggestions. We are also indebted to the Knowledge, Innovation, and Communication Sector Department team that works with us in the maintenance of the online course and collects the data. The information and opinions presented herein are entirely those of the authors, and no endorsement by the Inter-American Development Bank (IDB), its Board of Executive Directors, or the countries they represent is expressed or implied. IDB management had no involvement in the study design, analysis, or interpretation of the data, in the writing of the report, or in the decision to submit the article for publication. 1 Introduction A vast literature from psychology and economics has shown that individuals tend to have nonstandard preferences (e.g., social preferences), nonstandard beliefs (e.g., overconfidence), and nonstandard decision-making (e.g., framing and limited attention) (DellaVigna, 2009). Policymakers are no exception. Research has consistently shown that policy professionals are susceptible to nonstandard beliefs and decision-making traps. Overconfidence, for example, has been observed in the judgments of physicians, clinical psychologists, lawyers, negotiators, engineers, bankers, and security analysts (Berner and Graber, 2008; Griffin and Tversky, 1992; Kovacs et al., 2020; Lambert et al., 2012; Sandroni and Squintani, 2004; Stark and Sachau, 2016). Policy professionals are further affected by framing outcomes as losses or gains and by confirmation bias (Banuri et al., 2019). These biases can have real implications. For example, U.S. judges’ opinions are significantly influenced by the political composition of judicial panels (Sunstein, 2006), and the temporal order of rulings may affect the outcomes (Danziger et al., 2011). In the case of healthcare, biases are likely to influence diagnosis and make treatment decisions and levels of care dependent on patient characteristics (FitzGerald and Hurst, 2017). In education, teachers’ unconscious biases and preferences related to students’ gender, race, sexual orientation, socio-economic background, or other aspects of identity can affect learning outcomes and perpetuate inequalities in the classroom (Farfan Bertran et al., 2021). Becoming aware of our systematic errors may help correct them (Farfan Bertran et al., 2021). There are ways to reduce overconfidence (Brookins et al., 2014) and other biases. Making individuals reflect on their choices and providing information about actual performance and the risks entailed by wrong choices helps. For example, once NBA referees are made aware of their implicit preferences, their favoritism bias disappears (Pope et al., 2018). This is particularly relevant in the context of policymaking, where biased judgment can have significant welfare consequences (Cafferatta et al., 2023). Could training help? A meta-analytic review of management training programs found that those focused on human resources, soft skills, marketing, and finance and accounting, especially when organized by local organizations, tend to result in better firm performance (Busso et al., 2023). In the case of public servants, some types of training have been found to be effective, at least in the short run. Training programs for hospital managers positively affected managerial skills, knowledge, and competencies (Ravaghi et al., 2021). Training police officers in investigation techniques and soft skills increased the satisfaction of crime victims (Banerjee et al., 2012) and reduced some types of crimes (Garcia et al., 2013). 2 In this paper, we test whether a behavioral economics (BE) online course for public officials has an effect on their decision-making process toward public policy issues, including fighting the COVID-19 pandemic. We also test whether the course improves their problemsolving skills. The experiment took place in the context of the online behavioral course provided by the Inter-American Development Bank (IDB) on its learning platform. We randomized the individuals enrolled in 16 editions of the Spanish-language version of the course into treatment and control groups (about 25,000 individuals.) The control group was asked to solve problems in a six-question questionnaire before starting the course, and the treatment group did so at the end of the course. Results indicate that the course had a positive effect on improving problem-solving and decision-making. When considering the overall score, treated individuals scored 0.6 standard deviations higher than the control group. In terms of specific questions, the impact was between 0 and an increase of 34 percentage points. The results are robust to a series of tests that exploit the fact that the control group took the test before and after the course, as well as the rollover nature of the different editions of the course. Regarding mechanisms, we added to the survey a question (not considered in the overall score calculation) that was covered in the lectures and in-course tests. Participants scored higher on that one than on the other questions (40.8 percentage points), which provides some partial evidence that the effects happened because of learning. This study complements nascent but still scant research showing that debiasing training can significantly improve decision-making, with both short-term and long-term effects (Morewedge et al., 2015; Sellier et al., 2019). While previous studies have worked with a dedicated sample of lab or student participants watching a video or a case study, we evaluate the impact of a multi-week-long course designed for policymakers that was imparted over several years. It also complements a literature that evaluates the effectiveness of online learning tools (Cristia and Vlaicu, 2023). Here, we show that online courses can improve learning outcomes and decision-making abilities. Finally, the paper complements the vast literature on behavioral science by showing that training courses could be an additional tool available for better decision-making. This study could serve as the stepping stone to experiments that test problem-solving skills more broadly and generate incentives for further replication studies using the multiple courses on behavioral science available. 3 2 The Experiment 2.1 The IDB Course on Behavioral Economics The IDB provides online education aimed at policymakers in Latin America and the Caribbean.1 In 2020, the IDB launched the first online course in Behavioral Economics offered in Spanish.2The course is interactive, self-paced, and applied to public policy design. It is offered at no cost and targets Latin American policymakers. More than 14,000 individuals registered to participate, and by the end of 2023, the number had climbed to more than 25,000. The Portuguese and English versions were launched during the second semester of 2020. The course is divided into four modules with an approximate workload of 4-5 hours per week.. It was designed to be completed within a four-week time period, but participants are allowed to finish the course in up to six weeks. The first two modules cover the main concepts of the field (main biases and behavioral insights) and explanations of how these differ from the notions of the standard economic model. For example, module 1 includes 10 activities that take between 3 to 30 minutes each to complete. Activity 1 provides an introduction to “How good are we at making decisions?” Activity 2 describes what behavioral science is. Activity 3 provides an overview of the field and applications of behavioral economics. Activity 4 teaches about examples of non-standard preferences, activity 5 about non-standard beliefs, and activity 6 about the factors that affect information processing. Activities 7 to 9 deal with the main terms used in the field, how governments use behavioral insights, and the role of behavioral economics in the design and execution of public policies. Activity 10 is the learning assessment for the module. The third module focuses on applied cases in several sectors, with a special focus on tax compliance and health, two areas in which the IDB has built a broader portfolio. Starting with session 3, a specific section on COVID-19 was added. The revised learning guide, with a full description of the contents of the course, is provided in the Online Appendix. The teaching methodology consists of providing reference materials such as videos, interactive presentations, and readings and carrying out activities and exercises using real case examples from Latin America, the Caribbean, and other parts of the world. After each module, participant knowledge is tested. There are five learning assessments or tests during 1By 2020, the IDB offered more than 200 online courses in development effectiveness, integration and trade, project management, social and environmental risks management, water and climate change, and others. The full catalog of courses is available at https://cursos.iadb.org/en/indes/programas?lang=en 2For context, the course’s first five editions or sessions were launched in Spanish on February 18, March 17, May 19, July 28, and October 6, 2020. 4 the course: Modules 1 and 2 each contribute 20% to the total assessment. Module 3 consists of two assessments, one for the tax compliance section and one for the health section, each contributing 15%. The learning assessment for Module 4 is weighted at 30%.3Although completing each questionnaire is mandatory in order to move on to the next module, passing it is not a prerequisite for advancing in the course. The passing score for each assessment and for the overall course is at least 80 percent of the total score, and the final score is calculated based on the weights assigned to each questionnaire. Those who finish the course are awarded a certificate of completion (see example in the Online Appendix), and they can also share digital badges on social media. 2.2 Experiment Design To evaluate the impact of the course, we randomized those who registered for each one of the sessions in Spanish. Once individuals register for a course, they are divided into two groups (treatment and control) and then assigned to virtual classrooms of up to 100 people (each classroom is formed by individuals from the same group: treated or control.)4 Before starting the course, students receive a questionnaire with basic demographic questions (country of origin, sex, academic degree, etc.)5. Those individuals in the control group also receive a survey test that includes 5 questions (first two sessions) or 6 questions (beginning with session 3).6Everybody received the same survey test at the end of the course. The questions included in the test were of two types: i) cognitive skills tasks: a cognitive illusion (“triangles”), a computation of compound interest (“lottery”—only in sessions 1 and 2), and an expected value question (“disease”); and ii) public policy questions that tested the individual knowledge of behavioral insights. One of these questions (“teachers’ incentives”) was explicitly considered in the set of materials provided during the course; therefore, it acts as a validation exercise. The questions included in the survey (in the order they are presented to the individuals) are the following (right answers in bold face): 3In the first two sessions, the learning assessment for Module 3 was weighted at 20%, and Module 4 was weighted at 40%. 4The purpose of the classrooms is to provide the opportunity for interaction in virtual chats. These chats are not supervised or monitored. 5This information is available only for sessions 1 to 5, for those who chose to complete the questionnaire. 6The changes in the questionnaire responded to the introduction of COVID-19 material in the course; one of the original questions was replaced to avoid extending the survey too much. 5 Table 2: Treatment Effects (all courses pooled) Test Triangles: Disease: Child Anemia: Lottery COVID-19: COVID-19: Teachers z-score Reasoning Exp Value SocNorm & Loss Av Beh Interv Social Distancing Incentives (1) (2) (3) (4) (5) (6) (7) (8) Treatment 0.600*** 0.030** -0.024* 0.340*** 0.012 0.147*** 0.290*** 0.408*** (0.027) (0.012) (0.014) (0.014) (0.032) (0.014) (0.014) (0.014) Constant -0.121*** 0.429*** 0.571*** 0.242*** 0.717*** 0.544*** 0.278*** 0.442*** (0.032) (0.018) (0.021) (0.020) (0.028) (0.016) (0.016) (0.018) Observations 5655 5655 5655 5655 864 4791 4791 5655 Clusters 247 247 247 247 32 215 215 247 Course FE Yes Yes Yes Yes Yes Yes Yes Yes Adj R-squared 0.086 0.008 0.005 0.132 0.003 0.038 0.107 0.196 Notes: each row shows the regression coefficients and the standard error in parenthesis corresponding to an OLS regression, including session fixed effects. Standard errors are clustered at the session level. *** p<0.01, ** p<0.05, * p<0.1. Differences in the number of observations across columns because COVID questions were included starting in Session 3 when the Lottery question was eliminated. Figure 1: Distribution of Correct Answers 0 1 2 3 4 5 Control 0 1 2 3 4 5 Treatment Test score distribution 12 Figure 2: Correct and Incorrect Answers per Question and Group 0% 20% 40% 60% 80% 100% Percent Incorrect Correct Question 1 - triangles 0% 20% 40% 60% 80% 100% Percent Incorrect Correct Question 2 - Lottery 0% 20% 40% 60% 80% 100% Percent Incorrect Correct Question 3 - Teachers 0% 20% 40% 60% 80% 100% Percent Incorrect Correct Question 4 - Disease Control Treatment 0% 20% 40% 60% 80% 100% Percent Incorrect Correct Question 5 - Anemic 0% 20% 40% 60% 80% 100% Percent Incorrect Correct Question 2 - COVID 1 0% 20% 40% 60% 80% 100% Percent Incorrect Correct Question 2 - COVID 2 Control Treatment 13 We found no significant heterogeneous effects7. Neither gender, academic degree, nor experience in their job at the time of the course had any differential effect. This is important, as it shows that everybody benefited equally from the course. 4 Mechanism and Robustness Can we be sure that the differences between the treatment and control groups come from the course? In order to provide some evidence in this direction, we have performed three exercises. First, we introduced one question in the test that was also part of the tests within the course. The difference between the control and treatment groups in this question is higher than for any other test questions (see column 8 in Table 2.) Second, because the individuals in the control group took the questionnaire before and after the course, we can evaluate if the quality of their answers improved. As shown in Table 3, the individuals in the control group scored much higher in the test after having taken the course than before the course. In particular, the overall score improves by 0.6 standard deviations. Table 3: Control group: Differences between before and after the course (all courses pooled) Test Triangles: Disease: Child Anemia: Lottery COVID-19: COVID-19: Teachers z-score Reasoning Exp Value SocNorm & Loss Av Beh Interv Social Distancing Incentives (1) (2) (3) (4) (5) (6) (7) (8) After taking the course 0.616*** 0.114*** -0.029*** 0.319*** 0.036 0.191*** 0.253*** 0.382*** (0.020) (0.008) (0.010) (0.014) (0.029) (0.011) (0.011) (0.014) Constant -0.308*** 0.424*** 0.550*** 0.277*** 0.719*** 0.541*** 0.295*** 0.447*** (0.033) (0.018) (0.020) (0.017) (0.029) (0.018) (0.017) (0.016) Observations 2933 2933 2933 2933 478 2455 2455 2933 Clusters 124 124 124 124 17 107 107 124 Course FE Yes Yes Yes Yes Yes Yes Yes Yes Adj R-squared 0.253 0.059 0.003 0.202 0.005 0.110 0.155 0.296 Notes: each row shows the regression coefficients and the standard error in parenthesis corresponding to an OLS regression, including session fixed effects. Standard errors are clustered at the session level. *** p<0.01, ** p<0.05, * p<0.1. ifferences in the number of observations across columns because COVID questions were included starting in Session 3 when the Lottery question was eliminated. Third, one potential issue with the current analysis is that we are comparing individuals who took the survey test at different points; that is, the control takes the test a few weeks earlier than the treatment does. In order to control for that, we exploit the recurring nature of the courses and compare groups of people who took the survey test at approximately the 7This analysis exclusively focuses on sessions 1 to 5, as they are the only sessions for which there is available information on the variables used for heterogeneity analyses. Nevertheless, the available information pertains solely to individuals who chose to complete the demographic questionnaire. 14 same time, even though they belong to different course sessions (cohorts). That way, we can compare treated individuals in session 1 with control individuals in session 2 (who eventually finished their course), treated in session 2 with controls from session 3, and so on.8Results are shown in Table 4. Each column compares the results from the treated in session twith the control group (who then went on to finish their course) in session t+ 1. The results are very similar to those presented so far. Those who took the course answered between 0.2 and 0.9 standard deviations better than those who had not taken and finished the course yet. Table 4: Treatment Effect Across Courses Course 1 T Course 2 T Course 3 T Course 4 T Course 5 T Course 6 T Course 7 T Course 2 C Course 3 C Course 4 C Course 5 C Course 6 C Course 7 C Course 8 C Test z-score Treatment 0.511*** 0.210*** 0.562*** 0.621*** 0.652*** 0.857*** 0.628*** (0.097) (0.053) (0.061) (0.060) (0.161) (0.096) (0.181) Observations 383 665 942 941 582 207 250 Clusters 15 25 35 35 24 8 11 Course 8 T Course 11 T Course 12 T Course 13 T Course 14 T Course 15 T Course 16 T Course 9 C Course 12 C Course 13 C Course 14 C Course 15 C Course 16 C Course 17 C Test z-score Treatment 0.525*** 0.741*** 0.682*** 0.460*** 0.742*** 0.516*** 0.312** (0.104) (0.118) (0.126) (0.138) (0.145) (0.162) (0.128) Observations 177 137 193 244 119 218 215 Clusters 9 8 10 13 9 18 19 Notes: each row shows the regression coefficients and the standard error in parenthesis corresponding to an OLS regression. Standard errors are robust. *** p<0.01, ** p<0.05, * p<0.1. Source: Authors’ calculations As a final analysis, we ran a robustness exercise by evaluating whether there were any differences between the treatment and the control group after both had taken the course. We present the results in Table 5. As can be observed, differences are small, and they fluctuate in terms of the sign. The treatment group scored higher than the control group after the course in only 3 of them. Overall, it is the opposite effect seems to dominate according to the composite score: the control group scored higher than the treatment group. This result 8For reference, in the first year, the sessions started on February 18, March 17, May 19, July 28, and October 6, 2020 15 could be expected given that the control group had already taken the survey test before the course, which may have led to some learning even though they did not receive feedback. Table 5: Ex-post: Differences between Treatment and Control after course (all courses pooled) Test Triangles: Disease: Child Anemia: Lottery COVID-19: COVID-19: Teachers z-score Reasoning Exp Value SocNorm & Loss Av Beh Interv Social Distancing Incentives (1) (2) (3) (4) (5) (6) (7) (8) Treatment -0.048* -0.083*** 0.006 0.026* -0.021 -0.045*** 0.038** 0.029*** (0.027) (0.011) (0.013) (0.014) (0.029) (0.014) (0.014) (0.011) Constant 0.024 0.563*** 0.571*** 0.654*** 0.760*** 0.732*** 0.551*** 0.844*** (0.030) (0.015) (0.016) (0.015) (0.024) (0.014) (0.016) (0.012) Observations 5655 5655 5655 5655 864 4791 4791 5655 Clusters 247 247 247 247 32 215 215 247 Course FE Yes Yes Yes Yes Yes Yes Yes Yes Adj R-squared 0.001 0.013 0.005 0.040 0.001 0.017 0.017 0.097 Notes: each row shows the regression coefficients and the standard error in parenthesis corresponding to an OLS regression. Standard errors are robust. *** p<0.01, ** p<0.05, * p<0.1. Source: Authors’ calculations While this set of exercises does not offer full evidence on the mechanism, they suggest that there seems to be no randomization bias. It does not appear that the treated were better than the control in answering questions for reasons other taking the course (e.g., the passage of time). Moreover, people do not seem to be learning about the right answers independently of the course, and there is no selection effect arising from the different cohorts: those who took the course answer better than those in their same cohort but also better than those in future cohorts who take the survey test at approximately the same time. 5 Conclusions Behavioral biases lead to suboptimal decisions, and policymakers are not exempt from them. Behavioral biases tend to have a larger effect when individuals are tired, have high stress, or have shorter times to decide. This is usually the environment in which policymakers have to make decisions that can have large welfare consequences. Behavioral economics has provided ways to reduce the cost of some decisions, such as changing defaults. Still, restricting the policy space is not always possible. Finding ways to improve policymaking is therefore of first-order importance. Training courses for policymakers have been shown to be effective in increasing job performance. This paper tests whether a behavioral course could improve policy decisions. We show suggestive evidence that it does. Of course, these results should not be taken as the 16 ultimate proof. For example, testing policymakers regularly after they took the course and based on real decisions is needed. This paper could be a stepping stone in that direction. 17 References Banerjee, A., Chattopadhyay, R., Duflo, E., Keniston, D., and Singh, N. (2012). Improving police performance in rajasthan, india: Experimental evidence on incentives, managerial autonomy and training. Technical report, National Bureau of Economic Research. Banuri, S., Dercon, S., and Gauri, V. (2019). Biased policy professionals. The World Bank Economic Review, 33(2):310–327. Berner, E. S. and Graber, M. L. (2008). Overconfidence as a cause of diagnostic error in medicine. The American Journal of Medicine, 121(5):S2–S23. Brookins, P., Lucas, A., and Ryvkin, D. (2014). Reducing within-group overconfidence through group identity and between-group confidence judgments. Journal of Economic Psychology, 44:1–12. Busso, M., Park, K., and Irazoque, N. (2023). The effectiveness of management training programs: A meta-analytic review. Cafferatta, F., Dominguez, P., and Scartascini, C. (2023). Overconfidence and gun preferences: How behavioral biases affect your safety. Working Paper. Inter-American Development Bank. Available at https://publications.iadb.org/en/overconfidence-and-gunpreferences-how-behavioral-biases-affect-your-safety. Cristia, J. and Vlaicu, R. (2023). Digitalizing public services opportunities for Latin America and the Caribbean. Washignton, DC: Inter-American Development Bank. Danziger, S., Levav, J., and Avnaim-Pesso, L. (2011). Extraneous factors in judicial decisions. Proceedings of the National Academy of Sciences, 108(17):6889–6892. DellaVigna, S. (2009). Psychology and economics: Evidence from the field. Journal of Economic literature, 47(2):315–72. Farfan Bertran, M. G., Holla, A., and Vakis, R. (2021). Poor expectations. FitzGerald, C. and Hurst, S. (2017). Implicit bias in healthcare professionals: a systematic review. BMC medical ethics, 18(1):19. Frederick, S. (2005). Cognitive reflection and decision making. Journal of Economic perspectives, 19(4):25–42. Fryer Jr, R. G., Levitt, S. D., List, J., and Sadoff, S. (2012). Enhancing the efficacy of teacher incentives through loss aversion: A field experiment. Technical report, National Bureau of Economic Research. Garcia, J., Mejia, D., and Ortega, D. (2013). Police reform, training and crime: experimental evidence from colombia’s plan cuadrantes. Documento CEDE, (2013-04). 18 Griffin, D. and Tversky, A. (1992). The weighing of evidence and the determinants of confidence. Cognitive Psychology, 24:411–435. Kahneman, D., Bar-Hillel, M., et al. (2020). Comment: Laplace and cognitive illusions. Statistical Science, 35(2):171–172. Kovacs, R. J., Lagarde, M., and Cairns, J. (2020). Overconfident health workers provide lower quality healthcare. Journal of Economic Psychology, 76:102213. Lambert, J., Bessi`ere, V., and N’Goala, G. (2012). Does expertise influence the impact of overconfidence on judgment, valuation and investment decision? Journal of Economic Psychology, 33(6):1115–1128. Morewedge, C. K., Yoon, H., Scopelliti, I., Symborski, C. W., Korris, J. H., and Kassam, K. S. (2015). Debiasing decisions: Improved decision making with a single training intervention. Policy Insights from the Behavioral and Brain Sciences, 2(1):129–140. Pope, D. G., Price, J., and Wolfers, J. (2018). Awareness reduces racial bias. Management Science, 64(11):4988–4995. Ravaghi, H., Beyranvand, T., Mannion, R., Alijanzadeh, M., Aryankhesal, A., and Belorgeot, V. D. (2021). Effectiveness of training and educational programs for hospital managers: a systematic review. Health services management research, 34(2):113–126. Sandroni, A. and Squintani, F. (2004). A survey on overconfidence, insurance and selfassessment training programs. Working Paper, University of Warwick. Sellier, A.-L., Scopelliti, I., and Morewedge, C. K. (2019). Debiasing training improves decision making in the field. Psychological science, 30(9):1371–1379. Stark, E. and Sachau, D. (2016). Lake wobegon’s guns: Overestimating our gun-related competences. Journal of Social and Political Psychology, 4(1):8–23. Sunstein, C. R. (2006). Infotopia: How many minds produce knowledge. Oxford University Press. Tversky, A. and Kahneman, D. (1981). The framing of decisions and the psychology of choice. science, 211(4481):453–458. Vandenbroucke, A. R., Fahrenfort, J. J., Sligte, I. G., and Lamme, V. A. (2014). Seeing without knowing: Neural signatures of perceptual inference in the absence of report. Journal of cognitive neuroscience, 26(5):955–969. 19 BEHAVIORAL ECONOMICS FOR BETTER PUBLIC POLICIES Learning Guide A Online Appendix A.1 Course Learning Guide The course website link in English is: https://indesvirtual.iadb.org/enrol/index.php?id=1960 The full course learning guide is also provided below. BEHAVIORAL ECONOMICS FOR BETTER PUBLIC POLICIES Pg. 2 CONTENTS TARGETS AND OBJECTIVES ......................................................................................................................... COURSE PACE AND METHODOLOGY ................................................................................................ "NETIQUETTE" RULES FOR FORUM PARTICIPANTS ................................................................................... OBJECTIVES OF THE MODULES .................................................................................................................. EVALUATION ............................................................................................................................................... PASS POLICY …………………………………………………………………………………………. CERTIFICATION ........................................................................................................................................... DIGITAL BADGES ......................................................................................................................................... COURSE POLICIES ....................................................................................................................................... WORK PLAN ................................................................................................................................................ CREDITS ....................................................................................................................................................... 21 BEHAVIORAL ECONOMICS FOR BETTER PUBLIC POLICIES Pg. 9 Module 3 - Applied cases (week 3) 6 hours ☐Activity 1: Read the “Tax Compliance” section 10 min ☐Activity 2: Study the “Beliefs, barriers, and examples of solutions" lesson 35 min ☐Activity 3: Study the “Preferences, barriers, and examples of solutions" lesson 35 min ☐Activity 4: Study the “Information processing, barriers and nudges” lesson 30 min ☐Activity 5: Participate in the "Should we shame tax evaders?" forum 25 min ☐Activity 6: Read “Conclusions" 5 min ☐Activity 7: Browse the “Takeaways for tax compliance" interactive summary 15 min ☐Activity 8: Take the learning assessment for the section on tax compliance 25 min ☐Activity 9: Read the “Health" section and watch the video 15 min ☐Activity 10: Study the “Frequent biases in a patient's decisions” lesson 40 min ☐Activity 11: Study the “Nudges to overcome the barriers presented" lesson 40 min ☐Activity 12: Study the “Behavioral Economics can help fight COVID19” lesson 15 min ☐Activity 13: Participate in the “The ethics of health nudges - where is the limit?” forum 25 min ☐Activity 14: Read “Conclusions” 5 min ☐Activity 15: Browse the “Takeaways on patients’ decisions" interactive summary 15 min ☐Activity 16: Take the learning assessment for the section on health 25 min Module 4: From theory to practice: An interactive game (week 4) 3 hours ☐Activity 1: Watch the “Can behavioral economics help improve vaccination rates?" video 5 min ☐Activity 2: Participate in the interactive game 120 min ☐Activity 3: Take the learning assessment for Module 4 40 min ☐Activity 4: Watch the “Course closing” video 5 min 28 BEHAVIORAL ECONOMICS FOR BETTER PUBLIC POLICIES Pg. 10 CREDITS This course was developed by IDB’s Research Department and the Knowledge, Innovation and Communication Sector, under the coordination of its Behavioral Economics Group. The following IDB staff participated in the preparation of these contents: • Carlos Scartascini, Nina Rapoport, Ana María Rojas y Cristina Parilli - Research Department • Florencia Lopez Boo and Nicolás Ajzenman - Social Sector • Carlos Gerardo Molina and Fernanda Camera - Knowledge, Innovation and Communication Sector 29 A.2 Certificate of Completion 30 A.3 Additional Tables Analysis at Course Session Level Table A1: Number of students in each course session Number of students Registered in the course Finished the course Course 1 2453 720 29.35 % Course 2 650 148 22.77 % Course 3 4126 1147 27.80 % Course 4 2712 788 29.06 % Course 5 4257 1041 24.45 % Course 6 381 138 36.22 % Course 7 1142 283 24.78 % Course 8 1023 236 23.07 % Course 9 1535 111 7.23 % Course 11 982 168 17.11 % Course 12 635 106 16.69 % Course 13 1552 319 20.55 % Course 14 894 125 13.98 % Course 15 1004 119 11.85 % Course 16 1275 151 11.84 % Course 17 568 64 11.27 % Total 25189 5664 100 % 31 Table A2: Treatment Effect Course 1 Course 2 Course 3 Course 4 Course 5 Course 6 Course 7 Course 8 Test z-score Treatment 0.645*** 0.088 0.581*** 0.599*** 0.672*** 0.757** 0.765*** 0.459*** (0.064) (0.093) (0.058) (0.060) (0.067) (0.181) (0.165) (0.140) Triangles: Reasoning Treatment 0.082** 0.113** -0.032 -0.010 0.064* 0.035 0.074 -0.018 (0.034) (0.045) (0.028) (0.031) (0.032) (0.023) (0.047) (0.050) Disease: Expected Value Treatment 0.083** 0.095 -0.036 -0.033 -0.031 -0.078 0.000 -0.055 (0.036) (0.068) (0.031) (0.031) (0.034) (0.099) (0.089) (0.059) Child Anemia: Social Norm and Loss Aversion Treatment 0.454*** -0.087* 0.346*** 0.367*** 0.320*** 0.395*** 0.414*** 0.318*** (0.032) (0.041) (0.030) (0.044) (0.027) (0.045) (0.079) (0.058) Lottery Treatment 0.025 -0.046 (0.037) (0.041) COVID-19: Behavioral interventions Treatment 0.147*** 0.158*** 0.192*** 0.239** 0.094 0.097 (0.028) (0.034) (0.027) (0.069) (0.068) (0.060) COVID-19: Social Distancing Treatment 0.296*** 0.290*** 0.265*** 0.276* 0.384*** 0.229** (0.029) (0.030) (0.027) (0.095) (0.081) (0.078) Teachers Incentives Treatment 0.440*** -0.239*** 0.452*** 0.422*** 0.417*** 0.603*** 0.350*** 0.357*** (0.031) (0.057) (0.023) (0.029) (0.033) (0.089) (0.074) (0.062) Observations 717 147 1147 788 1041 138 283 236 Clusters 24 8 42 28 43 4 12 11 Notes: each row shows the regression coefficients and the standard error in parenthesis corresponding to an OLS regression. Standard errors are robust. *** p<0.01, ** p<0.05, * p<0.1. Source: Authors’ calculations 32 Table A2: Treatment Effect Course 9 Course 11 Course 12 Course 13 Course 14 Course 15 Course 16 Course 17 Test z-score Treatment 0.722*** 0.718*** 0.852*** 0.484*** 0.538*** 0.596*** 0.382** 0.134 (0.166) (0.121) (0.092) (0.150) (0.114) (0.162) (0.168) (0.077) Triangles: Reasoning Treatment 0.116 0.118 -0.049 0.042 -0.089** 0.037 0.100 -0.008 (0.062) (0.066) (0.056) (0.065) (0.037) (0.070) (0.068) (0.117) Disease: Expected Value Treatment -0.116 0.059 -0.091* -0.097 0.124 -0.020 -0.212*** -0.369*** (0.078) (0.064) (0.042) (0.058) (0.100) (0.097) (0.065) (0.060) Child Anemia: Social Norm and Loss Aversion Treatment 0.404*** 0.341*** 0.541*** 0.214*** 0.196* 0.389*** 0.157* 0.332*** (0.050) (0.085) (0.059) (0.048) (0.099) (0.086) (0.077) (0.030) COVID-19: Behavioral interventions Treatment 0.029 0.120** 0.276** 0.111 0.081 0.084 0.146* -0.025 (0.105) (0.046) (0.103) (0.067) (0.056) (0.093) (0.069) (0.121) COVID-19: Social Distancing Treatment 0.422*** 0.248*** 0.311** 0.315*** 0.357*** 0.214** 0.254*** 0.235 (0.056) (0.060) (0.096) (0.047) (0.070) (0.076) (0.044) (0.147) Teachers Incentives Treatment 0.371*** 0.330*** 0.576*** 0.425*** 0.350*** 0.477*** 0.409*** 0.389*** (0.075) (0.045) (0.062) (0.055) (0.094) (0.048) (0.075) (0.066) Observations 111 168 106 319 120 119 151 64 Clusters 6 10 6 15 9 10 13 6 Notes: each row shows the regression coefficients and the standard error in parenthesis corresponding to an OLS regression. Standard errors are robust. *** p<0.01, ** p<0.05, * p<0.1. Source: Authors’ calculations 33