Historical Communist Party Strength and Modern Party Loyalty. A Replication Study of Barceló (PNAS, 2021)

Malesky, Edmund J.; Nguyen, Trung-Anh Article Historical Communist Party Strength and Modern Party Loyalty. A Replication Study of Barceló (PNAS, 2021) Journal of Comments and Replications in Economics (JCRE) Suggested Citation: Malesky, Edmund J.; Nguyen, Trung-Anh (2024) : Historical Communist Party Strength and Modern Party Loyalty. A Replication Study of Barceló (PNAS, 2021), Journal of Comments and Replications in Economics (JCRE), ISSN 2749-988X, ZBW - Leibniz Information Centre for Economics, Kiel, Hamburg, Vol. 3, pp. 1-27, https://doi.org/10.18718/81781.35 This Version is available at: https://hdl.handle.net/10419/299994 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. https://creativecommons.org/licenses/by/4.0/ Journal of Comments and Replications in Economics - JCRE Historical Communist Party Strength and Modern Party Loyalty A Replication Study of Barceló (PNAS, 2021) Edmund J. Malesky*Trung-Anh Nguyen† Journal of Comments and Replications in Economics, Volume 3, 2024-5, DOI: 10.18718/81781.35 JEL: P0, P2, P3, P5, N0 Keywords: Conflict, Civic engagement, Vietnam War, Historical legacies Data Availability: The R code and data to reproduce the results of this replication can be downloaded at JCRE’s data archive (DOI: 10.15456/j1.2023285.1524615125). Please Cite As: Malesky, E. J. and Nguyen T. (2024). Historical Communist Party Strength and Modern Party Loyalty: A Replication Study of Barceló (2021). Journal of Comments and Replications in Economics, Vol.3 (2024-5). DOI: 10.18718/81781.35 Abstract Does exposure to violence create more politically engaged citizens? In a PNAS paper, Barceló (2021) asks this provocative question and proposes an intricate and original socio-psychological theory to answer it. Barceló also employs a research design that seeks to account for reverse causality to address the fact that more engaged places may have been targeted. Unfortunately, even after a correction, his statistical analysis remains dogged by coding, historical, and contextual errors, which bias results and lead to misleading conclusions. When we account for these mistakes with simple and reasonable corrections, the main results no longer hold. Rather, we show that the true underlying explanation is more mundane: areas that were sources of pre-war communist insurgency strength were targeted for their activity during the conflict, and they remain more loyal to the regime today. *Corresponding author: Department of Political Science, Duke University, NC, USA [email protected] †Department of Political Science, Duke University, NC, USA Declaration: The authors declare no competing interest. Research reported in the paper was not the results of a for-pay consulting relationship. Neither the authors nor their employers have a financial interest in the topic of the paper. Received October 12, 2023; Revised March 26, 2024; Accepted May 09, 2024; Published June 18, 2024. ©Author(s) 2024. Licensed under the Creative Common License - Attribution 4.0 International (CC BY 4.0). 1 E. J. Malesky & T. Nguyen – Historical Communist Party Strength (Replication). JCRE (2024-5) 1 Introduction In “The Long-Term Effects of War Exposure on Civic Engagement,” Joan Barceló argues that citizens in Vietnam who experienced greater bombing during the war with the United States (US) (1966-1974) had higher levels of political participation, institutional trust, and civic engagement in 2001 than citizens who were less impacted by the conflict (Barceló, 2021). In this research note, we explain why Barceló’s results are an artifact of coding, historical, and contextual mistakes in his research design. Even after accounting for his recent correction and a subsequent update to the correction (Barceló, 2023), the most likely explanation for the observed correlation is that the US bombing targeted strongholds of support for the communist regime, where populations remained loyal to the regime long after the war. In the paper, Barceló uses a linear estimation to demonstrate a correlation between civic engagement today and bombing during the war. However, this correlation is threatened by endogeneity, the US may have strategically bombed places that already had high levels of pre-war political participation. To address this possibility, Barceló exploits a natural experiment in conflict exposure caused by distance from the 17th parallel, suggesting that the conflict was the heaviest around the “arbitrary” border set at the Geneva Conference in 1954. 1 Consequently, Barceló claims it is plausible that some citizens close to the 17th parallel were accidentally exposed to violence, allowing him to trace through whether this conflict exposure generated the greater political participation of his theory. Barceló does not substantiate this assumption, but relies upon a previous assertion and measurement by Miguel and Roland (2011). Based on this logic, Barceló uses the difference in latitude between pre-war provinces and the 17th parallel in a two-stage least squares analysis. In this note, we first demonstrate that Barceló makes several statistical errors that render his research design inappropriate and his conclusions problematic. We do so by reproducing the original study’s results and providing our corresponding estimates after correcting for such errors. Second, we show that even if those unfortunate errors were less consequential, the research design is invalidated by the fact that the 17th parallel ran through the stronghold of the Vietnamese communist insurgency, where the most politically engaged citizens were already located. Third, we show that Barceló’s survey-based outcome variable is not measuring independent engagement as he claims, but is in fact reflecting current communist party strength through participation in entirely party-dominated mass organizations. When we account for these issues in the statistical analysis, Barceló’s provocative conclusions no longer hold. Finally, we demonstrate through a statistical analysis of our own that the central regions of Vietnam had higher levels of communist activity before the war and continued to have higher levels of communist activity after the war. The conclusion is clear: American bombs did not intensify civic engagement; they merely targeted pre-existing bastions of communist activity and loyalty. 1 We disagree with even this initial premise. While it is true that the proposals for demarcation varied from the 13th parallel to the 18th parallel at the Geneva Conference in 1954, these proposals did not reflect negotiators throwing darts at a map. Rather, the historical record is clear that the specific suggestions reflected actors staking out maximalist negotiation positions (Asselin, 2011), responding to changes in military control on the ground (such as the North’s surprising victory at Dien Bien Phu), and accommodating the interests of international powers such as the US, USSR, and China (Waite, 2012). The selection of the 17th parallel ensured that both modern combatants kept their historical power bases intact. The French retained Hue and Da Nang, while the North Vietnamese retained the important Zone IV military area (Asselin, 2011, 177), which we describe in more detail in Section 2. 2 Journal of Comments and Replications in Economics - JCRE 2 Replication and Correction of Errors 2.1 Error 1: Unclustered Standard Errors and Collinear Covariates In a letter to the PNAS editors from March 2023 regarding Barceló’s original article (Barceló, 2021), we pointed out that Barceló’s replication files demonstrated three flaws in statistical analysis among other concerns about the paper’s understanding of Vietnam’s history and geography. These three flaws included systematic measurement error in his instrumental variable (distance from the 17th parallel), the failure to include clustered standard errors as claimed in the text, and the inclusion of a control variable (latitude) that was a nearly perfectly correlated transformation of his instrumental variable, measured as the absolute value of the distance in degrees of latitude from the 17th parallel. We showed that these mistakes were leading to biased results and calculation of standard errors, which was driving the inaccurate conclusion that exposure to bombing led to greater civic engagement in Vietnam. Our response was not published, because it was submitted to PNAS after the six-month deadline for responses to published pieces; however, Barceló subsequently introduced a correction (Barceló, 2023) on July 7th and an update to the correction on February 29, 2024 2 that purportedly addressed the concerns we had raised with him and the journal’s editors. We have now reviewed these corrections along with Barceló’s amended replication files. While this correction did fix the error in the measurement of distance, it, unfortunately, introduced a new coding error that again results in the failure to cluster standard errors, does not correct an inaccurate coding of whether a province is located in South Vietnam, and incorrectly asserts that the inclusion of a nearly perfectly collinear control variable is justified on theoretical and empirical grounds. These fundamental statistical mistakes continue to lead to inaccurate results. In this section, we highlight these statistical errors and their consequences by reproducing the original study’s results and providing our corresponding estimates after correcting for these errors. 2.1.1 Failure to include clustered standard errors Equation 1 reproduces the two-stage instrumental variable approach that Barceló uses 3 to test his theory that bombing exposure leads to greater civic engagement today. In the first stage, Barceló regresses the extent of bombing, measured as the natural log of bombs dropped on the province per capita, on distance from the 17th parallel, measured as the absolute value of the distance from the latitude at the center of the province to the 17th parallel. In the second stage, he regresses his measure of civic engagement on the predicted number of bombs from the first stage. Table 1 from Barceló (2023) presents three instrumental variable models: Model 3 presents the bivariate correlations, Model 4 adds provincial-level control of population density, and Model 5 introduces a full suite of individual (𝑋𝑖) and provincial-level controls (𝑋𝑝), including latitude. 2see https://www.pnas.org/doi/10.1073/pnas.2308785120 3 For a discussion of the exclusion and strength assumptions of the instrumental variables approach and the potential pitfalls of causal identification in this context, see Appendix A. 3 E. J. Malesky & T. Nguyen – Historical Communist Party Strength (Replication). JCRE (2024-5) Notice that in the first stage, both bombing exposure and distance from the 17th parallel are measured at the provincial level ( 𝑝=49 ) and do not vary among surveyed citizens who reside in the same province. Essentially, Barceló is using the same distance and bombing measures repeatedly for different citizens within the same location. First stage: Bombs𝑝=𝛼0+𝛼1Dist17𝑝+𝑋𝑖+𝑋𝑝+𝑒𝑖 𝑝, Second stage: Civic𝑖 𝑝 =𝛽0+𝛽1ˆ Bombs𝑝+𝑋𝑖+𝑋𝑝+𝑢𝑖 𝑝,(1) This exaggerates the statistical significance of the underlying correlation by artificially reducing the size of the standard errors (Cameron, Gelbach, and Miller, 2008). To see why this happens, let’s step away from the specific analysis and focus on a simple thought experiment. Imagine we wanted to calculate the relationship between the median income of US states and the percentage of each state that turned out to vote in the 2020 Presidential Election using regression analysis. Obviously, the sample size ( 𝑛 ) for this relationship, measured by ˆ 𝛽1 , is 50: we have one observation of median income and one observation of aggregate turnout for each state. In turn, the standard error ( 𝜎 ) for the correlation should be calculated using 𝑛=50: ˆ 𝜎=√︃ˆ 𝛽1/50 −2 The author’s mistake is akin to regressing aggregate state turnout for every voting-eligible citizen in every state on the median income of that state. This is equivalent to calculating the standard error for the correlation using a sample size equal to the total voting-eligible population in the United States (approximately 255 million). With the new larger number, the denominator for the standard error is now much larger. This will artificially reduce the standard error by orders of magnitude: ˆ 𝜎=√︃ˆ 𝛽1/255,457,000 −2 By not clustering in his PNAS analysis, Barceló has essentially taken 49 observations of the outcome and independent variables but applied them to 875 individual-level survey observations. Econometricians have long understood that such clustered data is common in statistical analysis (i.e. children nested inside treated schools) and have worked to correct the dangers of this false precision and deflated standard errors. The solution is clustering the standard errors to address the correlation (in Barceló’s case, perfect correlation) of observations within the higher-level unit (Abadie et al., 2023). Clustering corrects for the false precision by calculating standard errors within each grouping, increasing the size of the standard errors and leading to more accurate tests of statistical significance. To be sure, there are serious statistical debates about clustering and questions about the accuracy of the approach when the number of higher-level units or the number of observations within higher-level units is small (Cameron, Gelbach, and Miller, 2008). However, none of these apply in Barceló’s situation, which is the canonical case of when clustering is necessary, because there is zero within-province variation in both the dependent and independent variables of his first-stage regression model. Barceló demonstrates awareness of the issue. In both his original article and July 2023 correction, he writes that the tables report, “Province-level heteroskedastic clustered-SEs in parenthesis.” In the original paper, however, he mistakenly reported only robust standard errors by failing to define the clustering variable in his R-script. He corrected the original mistake but introduced a new one, which 4 Journal of Comments and Replications in Economics - JCRE unfortunately leads to the exact same outcome of artificially small standard errors. 4 His calculation is shown in the screenshot below. Figure 1: Screenshot from Barceló’s R-script Showing Standard Errors Were Not Clustered (line 192-7 in provided replication file, last updated May 13, 2023) When Barceló applied corrections to the standard errors, he used the command vcovHC, which is designed to calculate robust standard errors and does not allow for a cluster argument. As a result, the cluster argument shown in the script (“cluster = ∼ province_wartime”) is ignored and every model in his analysis defaults to robust standard errors. To cluster, he needed to use the command vcovCL, which takes a cluster argument. There is no doubt that Barceló’s code is not correcting for clustering, because the standard errors he produces with his calculation in every model are smaller than not clustering at all. To illustrate this point in Table 1 below, we reproduce Barceló’s estimates from the first table in his corrections with results using our new corrected code that accurately calculates the standard errors by entering vcovCL into Barceló’s original command. We use M to denote the models used in Barcelo’s 2023 correction table (i.e. M3=Model 3), while referring to our own estimates using column numbers at the top of the table. Beginning with the first stage estimates in column 1 at the bottom of the table, Barceló’s M1 displays a coefficient of -.04 (a one-unit change in distance is associated with a 4% reduction in bombing) and a standard error of .0232 which he reports is significant at the 0.1 level. The F-test for this first stage is 3.66, which is smaller than the current standard of 10 and recommended score of 104.7 to represent instrument strength (Lee et al., 2022). In the second stage, this leads to a coefficient of bombing on the civic engagement of 1.19 with a standard error of .65, which Barceló also reports as significant at the .1 level. 4 In the February 2024 update to the July 2023 correction, the table notes on standard errors were edited to: "Heteroskedasticity-consistent standard errors in parenthesis. Main models are robust to using province-level heteroskedastic clustered standard errors." This indicates that Barceló is aware that the applied corrections to the standard errors only accounted for robustness to heteroskedasticity and not clustering at the province level. Additionally, we show in our Table 1 that two out of the three main models were, in fact, not robust to using province-level heteroskedastic clustered standard errors as the table notes claim. 5 E. J. Malesky & T. Nguyen – Historical Communist Party Strength (Replication). JCRE (2024-5) Table 1: Original Model Insignificant Regardless of Approach to Clustering Variables Civic Engagement (log) in 2001 (1) (2) (3) (4) (5) (6) (7) (8) (9) Without geographic controls Original/Corrected Model in Barceló (2023) Table 1 Original M3 Original M4 Original M5 Corrected M3 Corrected M4 Corrected M5 Original M5 Corrected M5 Corrected M5 Bombs, per km2 (log) 1.19* 0.44** 0.23** 1.19 0.44 0.23** -0.72** -0.72 -0.58 (residence pre-1975) (0.65) (0.12) (0.03) (3.65) (0.46) (0.06) (0.31) (1.33) (0.97) Female=1 -0.07** -0.07** -0.02 -0.02 -0.02 (0.04) (0.03) (0.07) (0.10) (0.09) Age -0.002 -0.002 0.0003 0.0003 0.001 (0.001) (0.002) (0.002) (0.003) (0.003) Education 0.02* 0.02* 0.07** 0.07 0.06 (0.01) (0.01) (0.03) (0.10) (0.07) Population density -0.22** -0.11** -0.22 -0.10* 0.44** 0.44 0.36 (1960 to 61) (’000) (0.11) (0.03) (0.23) (0.06) (0.20) (0.86) (0.63) Average precipitation -0.22** -0.14 1.54** 1.54 1.30 (’00) (2.10) (0.21) (0.65) (2.79) (2.10) Latitude (’0) -0.02 -0.17 (0.17) (0.30) South -0.29* -0.45* (0.16) (0.26) Constant -3.23* -1.15** -0.15 -3.23 -1.15 0.04 -1.29** -1.29 -2.65 (1.66) (0.28) (0.31) (9.45) (1.05) (0.48) (0.50) (2.04) (3.64) Clustered SEs (“HC0”) N N N Y Y Y N Y Y Ethnicity FEs N N N N N N N N Y First Stage Distance to 17th Parallel -0.04* -0.11** -0.57** -0.04 -0.11 -0.59** 0.07** 0.07 0.08 (0.02) (0.023) (0.027) (0.12) (0.12) (0.1) (0.02) (0.10) (0.10) F-Statistic 2.84 24.6 683.5 0.1 0.84 38.9 9.37 0.43 0.59 Observations 875 875 862 875 875 862 862 862 856 Clusters 49 49 48 49 49 48 48 48 48 Notes: Standard errors in parentheses (** p<0.05, * p <0.1). Using the code provided in the replication package, we were unable to replicate Barceló (2023, 2)’s F-statistics in Columns 1-3 for his three IV models, which were reported to be 3.66, 23.2, and 464.4, respectively. M3, M4, and M5 denote the model numbers in Table 1 of the 2023 correction. Column numbers represent the order of our estimations. 6 Journal of Comments and Replications in Economics - JCRE The second stage result implies a 10% increase in bombing is associated with a 12% increase in civic engagement. However, caution should be taken in interpreting economic effect from these coefficients as the estimates are widely varying. Reading across his original five estimates in Table 1 of the 2023 correction Barceló (2023), we find that a 10% increase in bombing had an 0.8% (M1), 1.2% (M2), 12% (M3), 4.4% (M4), or 2.3% (M5) impact on civic engagement. When we remove the controls for South and Latitude, the effect is a 7.2% reduction in civic engagement. Column 4 reproduces the column 1 (M3) estimates with the correct standard errors. We recover the exact same coefficient size, however, the standard errors are very different. The first-stage and second-stage standard errors are .123 and 3.65 respectively, over five times the size of the original Barceló estimates. The new estimates imply first-stage and second-stage p-values of 0.75 and 0.74, which are extremely far from statistical significance. The same pattern is true of the population-density adjusted estimates that Barceló presents in column 2 (M4) and we re-analyze in column 5. Correspondingly, Barceló’s error impacts his determination of instrument strength, where an F-test of at least 10 is needed to demonstrate an instrument is a strong enough predictor of the treatment variable that it will not generate bias in the second stage. New work, however, suggests a F-test threshold of 104.7. (Lee et al., 2022). When appropriate clustering is applied, Barceló’s F-test in the first stage drops from 2.84 to 0.1 in his M3, from 24.6 to 0.84 in M4, and from 683.5 to 38.9 in M5. The bottom line is that when correct procedures are applied, neither the first nor second stages of Barceló’s estimates are statistically significant. There is no observable correlation between distance from the 17th parallel and bombs dropped, and no statistical relationship between exposure to bombs and civic engagement, invalidating his first stage and therefore his research design. Analysts disagree on the mathematical nuances of correcting for clustering. To avoid concerns that our conclusions may be affected by reliance on a particular type of clustering estimation, in Table 2 we re-estimate Barceló’s standard errors using four different estimation types (HC0, HC1, HC2, and HC3) for clustering included in R’s sandwich package (Zeileis, Köll, and Graham, 2020). All lead to the unmistakable conclusion that the estimates in M3 and M4 from the correction’s Table 1 are no longer statistically significant. In other words, there is no relationship between distance and bombing or between bombing and civic engagement in these estimations. 7 E. J. Malesky & T. Nguyen – Historical Communist Party Strength (Replication). JCRE (2024-5) Table 2: Replication of Barceló’s Table 1 with New Models Using Clustered Standard Errors Variables Civic Engagement (log) in 2001 (1) (2) (3) (4) (5) (6) Original Model 4 Non-robust SEs Clustered SEs (HC0) Clustered SEs (HC1/STATA) Clustered SEs (HC2) Clustered SEs (HC3) Bombs, per km2 (log) 0.44** 0.44** 0.44 0.44 0.44 0.44 (residence pre-1975) (0.12) (0.11) (0.46) (0.47) (0.50) (0.54) Population density -0.22** -0.22** -0.22 -0.22 -0.22 -0.22 (1960 to 61) (’000) (0.06) (0.06) (0.23) (0.23) (0.26) (0.33) Constant -1.15** -1.15** -1.15 -1.15 -1.15 -1.15 (0.28) (0.26) (1.05) (1.05) (1.13) (1.23) First Stage Distance to 17th Parallel -0.11** -0.11** -0.11 -0.11 -0.11 -0.11 (0.02) (0.02) (0.12) (0.12) (0.13) (0.15) F-statistic 24.6 70.4 0.84 0.84 0.73 0.6 Observations 875 875 875 875 875 875 Notes: Standard errors in parentheses (** p <0.05, * p <0.1). Using the code provided in the replication package, we were unable to replicate Barceló (2023, 2)’s F-statistics in Column 1 for his IV Model 4, which was reported to be 23.2. 2.1.2 Regression with highly collinear covariates It is worth noting that in our re-estimations in Table 1, the coefficients in the first and second stages from column 3 (M5) remain significant in column 6. The key difference is that in column 3, Barceló includes a full suite of individual and provincial control variables. Why do these new controls change the results of the estimation? Most immediately, Barcelo’s south control is miscoded for the two southern provinces closest to the 17th parallel. As Figure 2 shows, both Thua Thien-Hue and Quang Tri are coded as North, when they are below the 17th parallel and were part of South Vietnam. In our analysis above and in the rest of the paper, we properly code these provinces. 8 Journal of Comments and Replications in Economics - JCRE 0.45 1.13 2.35 2.62 2.17 4 3.67 6 0 2 4 6 Provinces Outside Zone IV 6 Provinces Inside Zone IV Average Number of Central Committee Members per Province Second Central Committee (1951) Third Central Committee (1960) Ninth Central Committee (2001) Twelfth Central Committee (2016) Figure 4: Party Strength was Higher in Zone IV Before Vietnam War and Remains Higher Today Critical to our alternative theory is that Zone IV was targeted by US military planners, because this underlying party strength made it a massive source of manpower and resources. To demonstrate this point, Table 4 documents sights from Zone IV listed in the recently declassified publication of the top 94 military targets selected by the US Joint Chiefs of Staff for bombing raids in 1964, known as the JCS 94-Target List (Kamps, 2001). Just under a quarter (23 targets) were directly in Zone IV, including the military barracks Chanh Hoa, Dong Hoi, Vit Thu Lu, Vinh, and Chap Le where recruits, soldiers, and leadership for the Northern Army and Viet Cong insurgency trained and lived (Kamps, 2001). This is a strong indication that the bombing did not create party strength and loyalty, but that Zone IV was deliberately targeted for its pre-existing regime support. 15 E. J. Malesky & T. Nguyen – Historical Communist Party Strength (Replication). JCRE (2024-5) Table 4: 23 Targets from the Joint Chief of Staff 94-Target List That Are in Zone IV Target number Target description 4 {R} Dong Hoi airfield [limited jet-capable] 5 {R} Vinh airfield [limited jet-capable] 14 Thanh Hoa railroad/highway bridge 24 {R} Chanh Hoa barracks SE and division headquarters 33 {R} Dong Hoi barracks WNW (probable division headquarters) 36 {B}{R} Vit Thu Lu barracks/storage area (guerrilla staging area) 38 Vinh barracks and headquarters military region IV 39 {R} Chap Le baracks NW 41 {R} Phu Van ammunition depot E (major depot) 42 {R} Phu Van POL storage and ammunition depot NE 50 Vinh POL storage 52 {R} Vinh supply depot E 53 {R} Phu Van supply depot SE 54 Thien Linh Dong supply depot S 55 {R} Vinh Son supply depot SW/SE 64 {R} Xom Bang ammunition depot 71 {R} Ben Thuy port facilities/transshipment center 74 Quang Khe Port approaches 85/86 Road/Rail Route 1 (Hamrong to Hanoi) 89 Route 7 (Laos/North Vietnam border) 90 Route 8 (vicinity Nape, Laos to Roa Qua) 91 Route 12 (Laos/North Vietnam border to Xom Ma Na) To illustrate our empirical concerns, Figure 5 looks at differences between the six Zone IV provinces and all others on Barceló’s main outcome variable (civic engagement), the confounder above that we showed is biasing results (party membership), the suggested treatment variable (logged bombs per capita), and the instrumental variable (distance from the 17th parallel). Historical party membership is strongly correlated with all four variables. Zone IV provinces have higher levels of civic engagement and dramatically more party members in the World Values Survey sample, experienced greater bombing, and are closer to the 17th parallel. 10 This previous history of historical communist activity, targeting by the US, and higher levels of party-related activity today appears to be a clear exclusion violation and is therefore likely to bias results. 10See also Appendices E, G1, and G2. 16 Journal of Comments and Replications in Economics - JCRE −0.22 0.15 −0.3 −0.2 −0.1 0.0 0.1 0.2 Outside Zone IV Inside Zone IV Civic Engagement (log) for Provinces Civic Engagement 23.53 60.45 0 20 40 60 Outside Zone IV Inside Zone IV Share of Party Members (%) in Provinces Party Members in 2001 2.48 3.09 0 1 2 3 Outside Zone IV Inside Zone IV Bombs per square KM (log) Bombs Dropped 4.92 2.23 0 2 4 Outside Zone IV Inside Zone IV Latitude Difference from 17th Parallel Distance from Border Range bars represent 95% CI with SEs clustered at wartime province level Figure 5: Historical Party Strength is Strongly Associated with All Measures 2.3 Error 3: Constructing an Index of Current Party Strength and Not Civic Engagement Barceló’s next contextual error occurs during the assembly of his outcome variable, civic engagement, where he constructs an index using respondents’ self-reported participation (volunteer work and membership) in 14 types of social groups and volunteer associations from the 2001 World Values Survey (WVS). The choice of the 2001 WVS is odd; the Provincial Administrative Performance Index (PAPI) survey is a far larger, more nationally and provincially representative, and better-translated survey of public opinion and governance that has been produced since 2009 by the United Nations Development Program and the Centre for Community Support and Development Studies. Barceló claims to use the WVS because it allows him to track respondents who migrated after the bombing. However, this group constitutes only 12.6% of the 1000 survey respondents, as most respondents remained in the province where the survey was conducted. Beyond translation errors that have been raised before (Kurzman, 2014), there are two additional problems with using this measure from WVS. First, it does not provide a nationally representative sample, as the dataset severely oversamples members of the VCP. Throughout the country in 2001, there were about 2.5 million registered party members (Phong vien tong hop, n.d.), comprising 3% of the population. However, in the 2001 WVS, the average party membership is 23.8%. Limiting the share to the 878 respondents who answered the pre-1975 location question as Barceló does, those 17 E. J. Malesky & T. Nguyen – Historical Communist Party Strength (Replication). JCRE (2024-5) who say they are members of a political party is 29.2% (V43, WVS), and those who say they do voluntary work for the party is 24.3% (V58, WVS). As such, these survey respondents’ answers about civic engagement, participation, and confidence cannot be considered representative of the 97% of non-party members who were affected by the war. These respondents, as a whole, are unusually political engaged. Secondly, the author’s measure of civic engagement, which was constructed from WVS, is not independent of political engagement (or party membership), which we have shown above was high in Zone IV both before and after the war, because 12 out of these 14 organizations are party mass organizations that fall under the Vietnamese Fatherland Front (VFF) and controlled by the party-state (See Table 5). Barceló claims that local political organizations are decentralized and independent of national organizations (Barceló, 2021, 5). However, he provides no evidence for this assertion, which contradicts existing scholarship (Bui, 2016). In Vietnam’s Leninist single-party state structure, VFF is an umbrella group of mass organizations including the Communist Party itself and some officially sanctioned religious groups. Many of the government’s social programs are conducted through VFF and it is intended to supervise governmental organizations and activities. In reality, VFF’s various organizations are made up of mostly party members and the Front itself is chaired by a high-ranking VCP official, usually a Central Committee member. Membership and participation in youth, female, labor, and community organizations were not voluntary in Vietnam in 2001 at the time of the WVS survey. Most membership at this time would have occurred through Vietnam’s party mass organizations. It was not until 2003 that the Vietnamese government promulgated Decree 88 on Associations, which provides a guiding regulatory document for the formation and operations of associations and related groups (Sidel, 2010). Essentially, Barceló’s naive models are simply regressing a measure of historical party strength (bombing) on current party strength. 18 Table 5: 14 Types of Social Groups and Volunteer Organizations Used in the WVS. For the Vietnamese names of the VFF organizations, see Appendix D Wording in English Original Vietnamese wording Most likely VFF organizations % of WVS respondents as members 1 Social welfare services Các dịch vụ phúc lợi xã hội cho người già tàn tật và người túng thiếu Red Cross Vietnam, Vietnam Blind Association, Vietnam Assoc. for Victims of Agent Orange, Assoc. in Support of Vietnam Handicapped and Orphaned, Vietnam Relief Assoc. for Handicapped Children 26.5% 2 Religious organization Các tổ chức tôn giáo Vietnam Buddhist Sangha, Committee for Solidarity Catholics of Vietnamese, Vietnamese Protestant Assoc. 10.4% 3Education, arts, music or cultural activities Các hoạt động giáo dục, nghệ thuật, âm nhạc hoặc văn hoá Vietnam Union of Literature and Arts Assoc., Vietnam Assoc. for Learning Promotion 17.3% 4 Labor unions Các liên đoàn lao động Vietnam General Confederation of Labor 11.3% 5 Political groups or organizations Các tổ chức, đoàn thể chính trị Vietnam Communist Party 28.5% 6Local community actions on social issues Hoạt động tại cộng đồng địa phương về các vấn đề như nghèo khổ, việc làm, nhà cửa, bình đẳng chủng tộc Vietnam Women’s Union 26.2% 7Third world development or human rights Các tổ chức phát triển thế giới thứ 3 1.5% 8Conservation, environment, animal rights groups Các nhóm bảo tồn, môi trường, quyền động vật Vietnam Assoc. for Safe water and Environment 7.6% 19 9 Professional associations Các hiệp hội nghề nghiệp Vietnam Farmer’s Union, Vietnam Union of Science and Technology Assoc., Vietnam Cooperative Alliance, Vietnam Lawyers Assoc., Vietnam Journalist Assoc., Vietnam Gardening Assoc., Ornamental Oganism Science Assoc., Vietnam Medical Assoc., Vietnam History Science Assoc., Vietnam Gemstones, Jewelry, and Art craft Assoc., Veteran Teacher Assoc. of Vietnam, Vietnam Publishers Assoc., Vietnam Fisheries Society, Vietnam Public Health Assoc., Veteran Volunteering Youth Assoc., Vietnam Assoc. for Invalids and Disabilities Enterprise, Vietnam Assoc. of Small and Medium Enterprises, Vietnam Orientally Traditional Medicine Assoc., Young Business People Assoc., Vietnam Bar Federation 13.3% 10 Youth work (scouts, guides, youth clubs, etc.) Công tác thanh niên (tức là các câu lạc bộ thanh niên, hướng dẫn viên, hướng đạo sinh vv) Ho Chi Minh Communist Youth Union, Vietnam Youth Federation 15.4% 11 Sports or recreation Thể thao hoặc giải trí 19.2% 12 Women’s groups Các nhóm phụ nữ Vietnam Women’s Union 28.4% 13 Peace movements Phong trào vì hoà bình Vietnam Union of Friendship Organisations 9.2% 14 Voluntary organizations concerned with health Các tổ chức tình nguyện liên quan đến sức khoẻ Family Planning Assoc. 14.8% 20 Journal of Comments and Replications in Economics - JCRE −0.34 0.25 −0.4 −0.2 0.0 0.2 Non−Party Members Communist Party Members Civic Engagement (log) Civic Engagement 0.28 0.33 0.0 0.1 0.2 0.3 Non−Party Members Communist Party Members Participatory values (Index) Participation 0.34 0.4 0.0 0.1 0.2 0.3 0.4 Non−Party Members Communist Party Members Participation in Community More Voice in Community 0.19 0.25 0.0 0.1 0.2 0.3 Non−Party Members Communist Party Members Participation in Government More Voice in Government Range bars represent 95% CI with SEs clustered at wartime province level Figure 6: Party Members are Significantly More Engaged and Interested in Political Participation than Non-Members In short, party membership is highly correlated with the organizations used to construct the civic engagement index. To demonstrate this point, Figure 6 studies differences between party and non-party members on Barceló’s main outcome variable of civic engagement, a logged index of participation in the groups listed in Table 5, and his three main mechanism measures (participatory values, preference for voice in community, and preference for voice in government). It is immediately obvious that respondents, who are members of the communist party, score significantly higher on all of Barceló’s outcome measures. Party members report participation in 3.5 non-party organizations from Barceló’s list compared to 1.4 organizations for non-party members (p<0.0001). Similarly, party members demonstrate significantly higher levels of participatory values and willingness to engage in the community and government. Figure 7d studies the relationship from the provincial level—the level of the treatment variable, demonstrating the strong bivariate correlation between average party membership and participation in the Women’s Union and even Sports associations using both WVS 2001 and PAPI 2020. Combined, the two figures clearly demonstrate that Barceló’s measure of “civic engagement” is really just a proxy for underlying party strength in 2001. Provinces with large shares of party members also had large population shares in other seemingly unrelated mass organizations. 21 E. J. Malesky & T. Nguyen – Historical Communist Party Strength (Replication). JCRE (2024-5) Figure 7: Party Membership Is Strongly Correlated with Membership in Women’s and Sports Organizations (a) Sports clubs and Communist Party membership using World Values 2001 Survey data (b) Sports clubs and Communist Party membership using PAPI 2020 data (c) Women’s union and Communist Party membership using World Values 2001 Survey data (d) Women’s union and Communist Party membership using PAPI 2020 data After accounting for both Error 2 and Error 3 in Appendix B by correctly clustering standard errors by wartime provinces, dropping the structurally collinear geographical controls of south and latitude, as well as adding in additional treatment variables of Zone IV and number of Second Central Committee members (1950), we show that the main OLS results are also statistically indistinguishable from zero - there is no evidence that bombing caused civic engagement. We further discuss these corrections and test our theory in the next section. 3 Our Modification Thus far, we have demonstrated that the author employs a research design that confuses historical party strength with exposure to bombing, tests his theory on a flawed sample of the Vietnamese population, and conflates party strength with civic engagement. Together, these errors lead to the erroneous conclusion that bombing caused civic engagement, when it only shows that communist 22 Journal of Comments and Replications in Economics - JCRE bastions were targeted for bombing. We test the theoretical extent of our views about the exclusion restriction violation in Table 6. We have argued that distance is really captured by historical party strength, because the 17th parallel ran directly through Zone IV, the epicenter of communist activity. To test this directly, Table 6 includes a direct measure of whether historically communist communities continue to be more heavily involved in party activities today. To do this, we run a linear model that replaces the bombing measure with a dummy variable for whether a province is located in Zone IV. In line with the adjustments, we show the original controlled OLS specification in Model 1 and Model 2 (without the geographic controls south and latitude) while dropping party members in Models 3-4. Table 6 finds a robust relationship between historical party membership and civic engagement today. In the fully-specified original Model 1, location in military Zone IV is associated with 34% higher civic engagement, while it is 37% in Model 2 with the trimmed controls. Dropping party members in Models 3 and 4 reduces the size of the coefficients, which is in line with our theory of the exclusion restriction violation that historical party strength is correlated with party domination today. Removing party members reduces the path dependency of party loyalty and control. In short, we observe the legacy of historical party strength: places that were more active and influential in the pre-war communist movement remain more so today. 23 E. J. Malesky & T. Nguyen – Historical Communist Party Strength (Replication). JCRE (2024-5) Table 6: Historical Party Strength is Associated with Political Activity Today Variables Civic Engagement (log) in 2001 Using Full Dataset Dropping Party Members (1) (2) (3) (4) OLS OLS OLS OLS Zone IV=1 0.34** 0.37** 0.27** 0.27** (0.07) (0.06) (0.06) (0.04) Female=1 -0.05* -0.05* -0.05 -0.05 (0.03) (0.03) (0.03) (0.03) Age -0.001 -0.001 -0.002 -0.002 (0.001) (0.001) (0.001) (0.001) Education 0.03** 0.03** 0.005 0.01 (0.01) (0.01) (0.01) (0.01) Population density 0.02 0.02 0.04* 0.05** (1960 to 61) (’000) (0.03) (0.03) (0.02) (0.02) Average precipitation 0.14 0.07 0.06 0.05 (’00) (0.13) (0.08) (0.11) (0.06) Latitude (’0) -0.29 -0.09 (0.23) (0.20) South -0.22 -0.05 (0.23) (0.23) Constant 0.08 -0.39* -0.21 -0.36** (0.46) (0.22) (0.36) (0.18) Observations 862 862 608 608 R-squared 0.09 0.08 0.06 0.05 RMSE 0.51 0.52 0.39 0.39 Notes: Standard errors, clustered at wartime province level, in parentheses (** p <0.05, * p <0.1). Model 1 replicates the fully-specified OLS model 2 in Table 1 of the original article while replacing bombing with a dummy variable of whether a province is in Zone IV. Model 2 drops the geographical controls South and Latitude. Models 3-4 are similar to Models 1-2 but uses data that excludes members of the communist party. 4 Discussion and Conclusion Does exposure to bombing create more politically engaged citizens? The evidence does not support an affirmative answer. None of the findings presented in Barceló (2021, 2023) hold in the face of rigorous scrutiny, and much of it is the artifact of elementary statistical errors. Important conclusions should not be based on such flawed foundations. Rather, we show that the true underlying explanation is more mundane: areas that were sources of communist insurgency strength from before the war were targeted for their activity during the conflict, and they remain more loyal to the regime today. Beyond the immediate findings in the paper, our reconstruction should lead scholars to exercise caution when employing sophisticated causal inference tools using historical data from countries in which they are not experts. Such analyses have become increasingly popular, as scholars scavenge the world looking for historical natural experiments to test social science theories. This is important work, but understanding these experiments requires great contextual nuance, historical research, and language ability. Without these, evidence can be easily misinterpreted. The damage is more than academic: statistical manipulations can obscure fundamental mistakes, leading to erroneous conclusions and misguided policy recommendations. 24