Irreconcilable Statistical Paradoxes Question the Validity of Kim HJ et al.'s Study on COVID-19 Vaccination and Psychiatric Outcomes (Mol Psychiatry 2024; 29: 3635–3643)

1 CORRESPONDENCE Irreconcilable Statistical Paradoxes Question the Validity of Kim HJ et al.'s Study on COVID-19 Vaccination and Psychiatric Outcomes (Mol Psychiatry 2024; 29: 3635–3643) Marco Roccetti Department of Computer Science and Engineering University of Bologna, 40126, Italy marco.r[email protected] Abstract: The study by Kim et al. [1] on psychiatric outcomes following COVID-19 vaccination is statistically invalid due to a severe, uncorrected selection bias. This methodological flaw yields irreconcilable epidemiological paradoxes, including an implausible ≈ 77% risk reduction for Schizophrenia (i.e., 1 - HR=0.231) in the highrisk group, and statistically impossible incidence rates for Bipolar and Anxiety Disorders, inconsistent with robust Korean national benchmarks [2-4]. The simultaneous occurrence of HR ≪1 and HR ≫ 1 across different pathologies proves the observed effect is a mathematical artifact of misclassification bias. A valid re-analysis using robust methods like Propensity Score Matching [5] is warranted. We read with concern the study by Kim HJ et al on the association between COVID-19 vaccination and the incidence of psychiatric disorders in the South Korean National Health Insurance Service (NHIS) cohort [1]. Given the critical nature of data concerning national and international health policies regarding COVID-19 and vaccinations, the production of statistically unsound results can pose a significant risk to public health by misinforming decision-making. Therefore, these flaws must be corrected, even a year after publication. Our analysis concludes that the study’s primary findings are statistically invalid and driven by a severe, uncorrected selection bias. This flaw manifests in a series of highly implausible Hazard Ratios (HRs) and incidence rates that are irreconcilable with established South Korean epidemiological data, as reported in [2-5]. 2 The most striking statistical anomaly of [1] is the reported false protective effect HR = 0.231 (95% CI: 0.164– 0.326) for Schizophrenia (ICD-10 F20-F29) (see Abstract of [1]). The vaccinated group is ≈10.5 years older and significantly less healthy at baseline (sourced from Table 1 of [1]). Historical data from the Korean National Health Insurance database shows an annual incidence rate for Schizophrenia of [77.6 - 88.5] per 100,000 individuals (see Abstract of [4]). This range translates to a quarterly incidence of approximately 2.0 - 2.2 per 10,000 population (e.g., (88.5/10) /4 ≈ 2.21). This range is consistent with the reported unvaccinated rate 1.98 per 10,000 of [1] (Section titled: The cumulative incidences per 10,000 of psychiatric AEs following the COVID19 vaccination). Instead, the cumulative incidence in the vaccinated, high-risk group 0.51 per 10,000 (same Section of [1] as above) is nearly four times lower than both the control group's rate and the historically expected rate. Hence, it is scientifically implausible for the incidence of a chronic disorder to be so significantly lower in the older, less healthy group compared to the younger, healthier control. This pattern is the first signature of a misclassification bias: obviously, chronic Schizophrenia patients in the high-risk vaccinated group were systematically excluded or miscategorized as "not new cases", artificially lowering the numerator and invalidating the HR. Furthermore, the incidence rate reported for Bipolar Disorder (BD) in the control group is statistically impossible when benchmarked against South Korean national epidemiological data. The incidence of BD in the unvaccinated group was given as 1.39 per 10,000 over three months in [1] (Section titled: The cumulative incidences per 10,000 of psychiatric AEs following the COVID-19 vaccination). This rate, however, exceeds the reported national prevalence (total existing cases) for Borderline Personality Disorder (a severe, chronic condition often comorbid with BD) which was 1.06 per 10,000 over an entire year (Results Section of [2]). As a fundamental epidemiological rule, in fact, the incidence of a severe chronic condition over three months cannot exceed the annual prevalence of a similar chronic condition in the same population. This absolute inconsistency confirms that the authors' methodology for defining "new cases" of BD is fundamentally flawed, rendering the calculated HR of 0.672 (Abstract of [1]) meaningless. The selection bias of [1] creates falsely low HRs for chronic diseases (Schizophrenia, BD) and simultaneously creates falsely high HRs for common diseases like Anxiety, Dissociative, Stress-Related, and Somatoform Disorders (F40-F48). The resulting HR of 1.439 [95% CI] = 1.322–1.568 (see Abstract of [1]) is a direct 3 mathematical artifact of the uncorrected baseline disparity. The incidence in the vaccinated group (28.41 per 10,000) is reported as ≈ 40% higher (28.41/20.27 ≈ 1.40) than the control group (20.27 per 10,000). All these figures are sourced from [1] (Section, titled: The cumulative incidences per 10,000 of psychiatric AEs following the COVID-19 vaccination). The uncorrected baseline disparity is demonstrated by comparing the study's rates to robust national epidemiology. The 12-month prevalence of Anxiety Disorders in Korea is 3.1% (Section Anxiety in [3]), which translates to an annual rate of 310 per 10,000. The incidence of a chronic condition (new cases appearing over time) must, by definition, be lower than its total prevalence (cases existing at a point in time). Therefore, the quarterly fraction of the national prevalence, ≈ 77.5 per 10,000 (i.e., 310/4), represents an absolute theoretical upper bound for any plausible three-month incidence rate. The fact that both the vaccinated rate (28.41) and the non-vaccinated rate (20.27) are massively lower than this expected national upper bound (77.5) proves the existence of severe misclassification bias in both cohorts (vaccinated and non-vaccinated). Indeed, this study [1] fails again to capture the vast majority of new cases. This methodological failure renders the derived HR meaningless. Not only but the simultaneous presence of HR ≪ 1 (suggesting a miraculous protective effect on chronic disorders highly prevalent in the vaccinated group) and HR ≫1 (suggesting a strong detrimental effect on common disorders linked to the uncorrected selection bias) within the same non-corrected cohort is the definitive proof that the observed effect is not biological, but the shadow of the underlying bias, reflecting differently across distinct pathologies. In closing, the three statistical paradoxes we have exposed demonstrate that the findings of [1]. are unreliable. Our final informed opinion is that the core problem lies in the construction of the control cohort: the authors state that the unvaccinated group was created by selecting 50% of unvaccinated individuals at random from the general population (see Abstract of 1). This method, while seemingly random, is statistically inadequate in a retrospective study utilizing administrative data, as it fails to correct for the vast, intrinsic differences, such as age, comorbidities, and health-seeking behavior, between those who chose to be vaccinated and those who did not. The observed anomalies are the direct consequence of this methodological error. We therefore suggest that any valid re-analysis 4 of this data must employ a statistically robust procedure, such as Propensity Score Matching (PSM) [5], to achieve true covariate balance and to obtain reliable estimates. References 1. Kim HJ, Kim MH, Choi MG, Chun EM. Psychiatric adverse events following COVID-19 vaccination: a population-based cohort study in Seoul, South Korea. Mol Psychiatry. 2024;29(12):3635–3643. doi: 10.1038/s41380-024-02627-0 2. Shin H, Lee HS, Lee BC, Park G, Uranbileg K. The Prevalence and Clinical Characteristics of Borderline Personality Disorder in South Korea Using National Health Insurance Service Customized Database. Yonsei Med J. 2023;64(9):566–72. doi: 10.3349/ymj.2023.0071. 3. Rim SJ, Hahm B-J, Seong SJ, Park JE, Chang SM, Kim B-S, et al. Prevalence of Mental Disorders and Associated Factors in Korean Adults: National Mental Health Survey of Korea 2021. Psychiatry Investig. 2023;20(3):262–272. doi:10.30773/pi.2022.0307 4. Cho SJ, Kim J, Kang YJ, Lee SY, Seo HY, Park JE, et al. Annual Prevalence and Incidence of Schizophrenia and Similar Psychotic Disorders in the Republic of Korea: A National Health Insurance Data-Based Study. Psychiatry Investig. 2020 Jan 25;17(1):61–70. doi. 10.30773/pi.2019.0041 5. D’Agostino RB Jr. Propensity score methods for bias reduction in the comparison of a treatment to a non-randomized control group. Stat Med. 1998;17(19):2265–81. doi: 10.1002/(sici)10970258(19981015)17:19<2265::aid-sim918>3.0.co;2-b Author Information Marco Roccetti: Department of Computer Science and Engineering, University of Bologna, 4016 Bologna, Italy, [email protected]. ORCID: 0000-0003-1264-8595 Contributions MR conducted the data analysis, conceptualized the statistical arguments, and wrote the paper. Corresponding author Correspondence to Marco Roccetti, [email protected]. 5 Ethics declarations: Competing interests The author declares no competing interests. Funding This research received no external funding Data Availability The data presented here is either included directly or was extracted from the referenced documents. All calculations are easily reproducible based on the definitions provided Ethics approval and consent to participate This study uses publicly available, aggregated data that contains no private information. Therefore, ethical approval is not required