Lost in Replication - A Journey through the Scientific Replication Crisis Understanding the Replication Crisis

Lost in Replication - A Journey through the Scientific Replication Crisis Building a Culture of Reproducibility/Replication Leonardo Egidi ([email protected]) Dipartimento di Scienze Economiche, Aziendali, Matematiche e Statistiche "Bruno de Finetti" (DEAMS) Università degli Studi di Trieste 28 November 2025 Lost in Replication A Journey through the Scientific Replication Crisis Building a Culture of Reproducibility/Replication Leonardo Egidi University of Trieste, [email protected] Master in Data Management e Curation (MDMC), Area Science Park 28 November 2025 L.Egidi (UniTS) (MDMC) Lost in Replication 1 / 25 Outline 1Reproducible Workflow 2A Mini Pre-registered Experiment (with Python notebook) 3Take-home messages L.Egidi (UniTS) (MDMC) Lost in Replication 2 / 25 Reproducible? L.Egidi (UniTS) (MDMC) Lost in Replication 3 / 25 What Is a Reproducible Workflow? A reproducible workflow is not just “running the code again.” It is a structured way of organizing: data (how they were collected, cleaned, stored) code (scripts, versioning, execution order) documentation (decisions, assumptions, environment) outputs (figures, tables, reports) Gelman’s view: “Good workflow is not about perfection. It is about reducing the number of things that can go wrong.” L.Egidi (UniTS) (MDMC) Lost in Replication 4 / 25 Reproducibility vs Replicability ACM/NASEM definitions: Reproducibility: Using the same data and the same code, another researcher obtains the same results. (→computational integrity) Replicability: Collecting new data and obtaining results consistent with the original study. (→scientific credibility) Robustness: Results do not change under reasonable alternative models, priors, or assumptions. Take-home message: “Reproducibility is the minimum. Replicability is the goal.” L.Egidi (UniTS) (MDMC) Lost in Replication 5 / 25 FAIR Principles for Data Modern research follows the FAIR principles: 1Findable — metadata, identifiers, searchable repositories 2Accessible — open formats, clear access conditions 3Interoperable — standardized vocabularies, common formats 4Reusable — licenses, documentation, provenance Why it matters for reproducibility: Data cannot be reproduced if they cannot be found, understood, or reused. L.Egidi (UniTS) (MDMC) Lost in Replication 6 / 25 Tools for Reproducible Workflows Version Control: Git, GitHub, GitLab — track every change, ensure transparency. Executable Documents: R Markdown / Quarto Jupyter Notebooks Python scripts + renv / virtualenv Project Structure: Clear folder organization: data/,scripts/,output/, docs/. Advice: “Start simple, avoid magic, document everything.” L.Egidi (UniTS) (MDMC) Lost in Replication 7 / 25 OSF: The Open Science Framework OSF provides: pre-registration of studies version-controlled storage of data and code transparency in workflows easy sharing with collaborators and reviewers Why is this important? A transparent workflow prevents accidental p-hacking, selective reporting, and encourages better science. Let’s take a look at one of the examples. L.Egidi (UniTS) (MDMC) Lost in Replication 8 / 25 From Original Study to Replication Attempt Original Experiment: n= 60, random assignment to treatment/control. Data generated: y= 2 + 0.35 ×group +noise True effect simulated as δ= 0.35 (small but nonzero). Frequentist estimate + Bayesian posterior on the effect. Replication Study: Pre-registered: n= 100. Same data-generating process, new sample. Analysis: same model, same priors, same workflow. What you can do: 1Re-run the notebook (with fixed seed). 2Compare original vs replication effect sizes. 3Check whether “replication success” criteria are met. L.Egidi (UniTS) (MDMC) Lost in Replication 15 / 25 Bayesian Synthesis & Reproducible Workflow Bayesian hierarchical model: Combines both datasets while allowing for uncertainty. Estimates a common underlying “true effect”. Produces posterior intervals for the replication. Workflow principles highlighted in the notebook: Fixed random seed ⇒deterministic results. Transparent code: every step documented. Environment recorded (package versions, OS). Modular structure: data, model, results separated. Lesson: “Good workflow reduces the number of things that can go wrong.” L.Egidi (UniTS) (MDMC) Lost in Replication 16 / 25 Frequentist/Bayesian original estimation Frequentist analysis:p-value = 0.0824, estimated effect is 0.4696, CI: = (−0.0618,1.001). Bayesian analysis: posterior median for the effect is 0.4696, CI is narrower than the frequentist one. group 0.00 0.25 0.50 0.75 1.00 L.Egidi (UniTS) (MDMC) Lost in Replication 17 / 25 Frequentist/Bayesian replication estimation Frequentist analysis:p-value = 0.292, estimated effect is 0.2339, CI: = (−0.2075,0.6753). Bayesian analysis: posterior median for the effect is 0.3, CI is narrower than the frequentist one. group 0.0 0.2 0.4 0.6 L.Egidi (UniTS) (MDMC) Lost in Replication 18 / 25 Final criteria for replication in the experiment Well-posed and transparent criteria: Criterion 1: is the sign of the effect equal across the original and the replicated experiments? ⇒Yes Criterion 2: Is the product between the posterior mean of the effect in the original experiment and that in the replicated experiment greater than zero? ⇒Yes Criterion 3: is the absolute value difference between the original experiment and the replicated experiment lower than a given threshold, say 0.25? ⇒Yes Criterion 4: are both the CI intervals for the effect not containing zero in both experiments? ⇒No Comment: there is not a unique definition of replication success! You, the researcher, are responsible for this final claim (see the paper Consonni and Egidi (2025)) L.Egidi (UniTS) (MDMC) Lost in Replication 19 / 25 Center for Reproducible Science Why I started working on this field: Center for Reproducible Science in Zurich lead by Prof. Leonhard Held L.Egidi (UniTS) (MDMC) Lost in Replication 20 / 25 Open project from my network Joint work with Leo Held, Samuel Pawel, Roberto Macr`ı Demartino. Mixture priors for replication studies (under review) L.Egidi (UniTS) (MDMC) Lost in Replication 21 / 25 Andrew’s blog Many ideas of this course come from this amazing blog, for which I am a contributor: L.Egidi (UniTS) (MDMC) Lost in Replication 22 / 25 Take-home Messages 1. The replication crisis is real, but solvable. 2. Reproducible workflows are the foundation of credible science. 3. Openness (data, code, methods) is a cultural shift, not a technical one. 4. Bayesian methods offer principled ways to integrate original and replication evidence. L.Egidi (UniTS) (MDMC) Lost in Replication 23 / 25 A Reproducible Workflow at a Glance Data Collection (raw data, metadata) Data Cleaning (code, logs, scripts) Analysis (models, diagnostics) Results (figures, tables) Report (R Markdown/Quarto) Archive (OSF / GitHub) A workflow is reproducible when data, code, and documentation flow together. L.Egidi (UniTS) (MDMC) Lost in Replication 24 / 25