The STROBE Contradiction: When Reporting Guidelines Mask, Rather Than Constrain, Inferential Overreach

1 Technical Note The STROBE Contradiction: When Reporting Guidelines Mask, Rather Than Constrain, Inferential Overreach Marco Roccetti Department of Computer Science and Engineering, University of Bologna, 40127 Bologna, Italy * Corresponding Author: Marco Roccetti Email: [email protected] Author Contributions: M.R. conceived the study, performed the analysis, and wrote the manuscript. Competing Interest Statement: The author declares no competing interests. Keywords: Observational studies; scientific reporting; external validity; STROBE; peer review. Abstract Scientific reporting guidelines such as the STROBE Statement were established to ensure epistemic discipline in observational research by transparently delineating the boundaries of legitimate inference. They serve as a crucial safeguard, guaranteeing that scientific conclusions do not exceed the evidential support provided by the data. However, analysis of recent highprofile medical observational studies reveals a critical contradiction: formal compliance is increasingly prioritized over substantive application. This technical note documents a failure in editorial enforcement, highlighting a case where published results exhibited clear, numerical nonrepresentativeness: a violation of STROBE principles concerning external validity and bias assessment (Items 21 and 12). By reconstructing crude incidence rates (CRs) directly from the published data and comparing them against national baseline statistics, we demonstrate that internally valid relative cancer risk estimates following COVID-19 vaccination were implicitly generalized as population-level signals, despite evidence of systematic sample divergence. We posit that if STROBE is consistently treated merely as a descriptive checklist rather than as an epistemically binding constraint, its presence undermines scientific integrity by creating a false sense of methodological rigor. Maintaining this symbolic framework is less honest than abolishing 2 it. The scientific community must restore STROBE's intended role as an enforced boundary on unwarranted inference Introduction Observational studies are indispensable in domains where randomized experiments may be impractical or unethical, like medicine for example. At the same time, they are intrinsically vulnerable to confounding, selection effects, and limits on generalizability. Reporting guidelines such as STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) were developed precisely to mitigate these vulnerabilities, not by guaranteeing validity, but by making the boundaries of legitimate inference explicit [1]. Central to STROBE is the requirement that authors distinguish internal statistical associations from claims that can be generalized beyond the analyzed cohort. In particular, STROBE emphasizes transparent discussion of bias and external validity when study samples diverge from the target population. Despite its widespread adoption, growing evidence suggests that STROBE is increasingly treated as a formal submission requirement rather than as an epistemic constraint. This shift has important consequences for how observational findings are interpreted, amplified, and trusted. Results: Crude incidence rates and loss of external validity A critical inconsistency emerges when crude incidence rates (CRs) are reconstructed from the raw counts reported in a recent large population-based medical observational study conducted in South Korea and presented as STROBE-compliant [2]. Although the study emphasizes post COVID-19 vaccination increased cancer risk estimates derived from regression models, its published summary statistics allow direct assessment of absolute cancer incidence levels, which are essential for evaluating representativeness. The matched cohort includes 12,133 incident cancer cases among 2,975,035 individuals observed over one year, corresponding to an overall crude incidence rate of 40.78 cases per 10,000 population. Stratified by exposure, the same data yield a CR of 42.63 per 10,000 among vaccinated individuals (10,144 cases among 2,380,028 subjects) and 33.43 per 10,000 among unvaccinated individuals (1,989 cases among 595,007 subjects) [2]. These values contrast sharply with national cancer incidence baselines for South Korea reported by the Korean Central Cancer Registry. For the years 2020–2022, national crude incidence rates averaged 52.46 cases per 10 000 population, with annual values of 48.29, 54.06, and 55.02 per 10,000, respectively [3-5]. The cohort analyzed in the study [2] therefore exhibits an overall cancer incidence more than 20% lower than the population-level baseline. This discrepancy is not marginal and cannot plausibly be attributed to short-term temporal variation. Rather, it indicates that the analyzed cohort is systematically unrepresentative of the national population with respect to cancer risk. Despite this, the study does not discuss sample representativeness, does not explain the divergence from national incidence data, and does not restrict its conclusions to the internal domain of the cohort. As demonstrated in an independent methodological reanalysis based exclusively on the published figures [6], the coexistence of hazard ratios greater than unity and an overall crude incidence substantially below the national baseline produces an inferential contradiction: 3 internally valid relative differences are implicitly interpreted as population-level risk signals despite numerical evidence that external validity does not hold. Discussion: When reporting guidelines cease to constrain inference The inconsistency documented above is not a technical subtlety but a direct consequence of neglecting external validity. Importantly, it is observable without access to individual-level data and follows directly from the descriptive statistics reported by the authors themselves. This omission constitutes a violation of STROBE Item 21, which requires explicit discussion of generalizability, and Item 12, which mandates transparent handling of bias and study limitations when these emerge from the data [1]. The issue, therefore, is not disagreement over interpretation but failure to apply the reporting standard that the journal explicitly endorses. The broader implication is editorial rather than methodological. Journals increasingly require formal adherence to STROBE, yet violations of its most consequential principles appear to carry no corrective consequence. As a result, STROBE functions rhetorically signaling rigor, while failing to operate as an epistemic boundary on inference. This shift has systemic effects. Internally valid associations are routinely interpreted as generalizable risks; exploratory observational analyses acquire quasi-confirmatory status; and the distinction between statistical modeling and population-level inference becomes blurred. Over time, reporting guidelines intended to protect scientific integrity instead mask its erosion. Materials and Methods This technical note is based on methodological analysis of published observational studies and reporting guidelines. No new human or animal data were collected. All calculations and comparisons rely exclusively on summary statistics reported in the literature and publicly available national incidence data. Conclusions: A necessary provocation The title of this technical note is deliberately provocative but logically grounded. If STROBE is treated as a checklist rather than as a boundary on legitimate inference, if clear violations of external validity do not require explicit limitation of conclusions, then STROBE no longer performs a scientific function. Under such conditions, maintaining STROBE as a formal requirement risks doing more harm than good by creating the illusion of methodological rigor where none is enforced. Abolishing it would be more honest than preserving it as a symbolic artifact. Alternatively, the scientific community must restore STROBE to its original role: not a reporting formality, but a constraint that determines what claims may, and may not, be made from observational data. The choice is not between guidelines and no guidelines, but between epistemic discipline and methodological theater [7]. Data, Materials and Software Availability All essential data are included within this manuscript. 4 Acknowledgments We thank colleagues from University of Bologna for their helpful feedback during the development of this manuscript. Statement on the Use of Artificial Intelligence We are proud to state that artificial intelligence-based tools were used (exclusively) as a dialogical aid to facilitate critical self-review, in a context where repeated attempts to engage in methodological discussion through conventional editorial correspondence with journals, editors, and responsible members of major publishing groups involved in these issues had not elicited substantive responses. Instead, the scientific ideas, methodological arguments, data analysis, and conclusions presented in this manuscript originate entirely from the author’s independent research and ongoing scholarly work. AI tools were not used to generate hypotheses, identify results, select data, or determine interpretations. The author remains fully responsible for the content of the manuscript, including all analyses, interpretations, and conclusions. References 1. E. Elm, D.G. Altman, M. Egger, S.J. Pocock, P.C. Gotzsche, J.P. Vandenbroucke, The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: guidelines for reporting observational studies. PLoS Med. 4(10):e296. (2007) DOI: 10.1016/j.jclinepi.2007.11.008. 2. H.J. Kim, M.-H. Kim, M.G. Choi, E.M. Chun, 1-year risks of cancers associated with COVID-19 vaccination: a large population-based cohort study in South Korea. Biomark Res. 13, 114 (2025). 3. M.J. Kang, K.-W. Jung, S.H. Bang, S.H. Choi, E.H. Park, E.H. Yun, H.J. Kim, H.-J. Kong, J.-S. Im, H.G. Seo, Cancer Statistics in Korea: Incidence, Mortality, Survival, and Prevalence in 2020. Cancer Res Treat. 55(2):385-399. (2023) DOI: 10.4143/crt.2023.447. 4. E.H. Park, K.-W. Jung, N.J. Park, M.J. Kang, E.H. Yun, , H.J. Kim, J.-E. Kim, H.-J. Kong, J.-S. Im, H.G. Seo, Cancer Statistics in Korea: Incidence, Mortality, Survival, and Prevalence in 2021. Cancer Res Treat. 56(2):357-371. (2024) DOI: 10.4143/crt.2024.253. 5. E.H.Park, K.-W. Jung, N.J. Park, M.J. Kang, E.H. Yun, H.J. Kim, J.-E. Kim, H.-J. Kong, J.-S. Im, H.G. Seo, Cancer Statistics in Korea: Incidence, Mortality, Survival, and Prevalence in 2022. Cancer Res Treat. 57(2):312-330. (2025) DOI: 10.4143/crt.2025.264 6. M. Roccetti, A Critical Note on Contradictions in South Korean Cancer Incidence Rates: The Paradox of Crude Rates Derived from the Kim HJ et al. Cohort (Biomark Res, 13:114, 2025). Preprints.org. (2025) DOI: 10.20944/preprints202510.0883.v1. 7. J. P. A. Ioannidis, Why most published research findings are false. PLoS Med. 2(8):e124. (2005) DOI: 10.1371/journal.pmed.0020124.