The State of Study Preregistration - How Science Benefits From It and How to Apply It

Preregistration 1 The State of Study Preregistration - How Science Benefits From It and How to Apply It Hilmar Brohmer Department of Psychology, Graz Open Science Initiative, Arqus Open Science Ambassador, University of Graz, [email protected], https://orcid.org/0000-0001-7763-4229 Gabriela Hofer Department of Psychology, Graz Open Science Initiative, Arqus Open Science Ambassador, University of Graz, [email protected], https://orcid.org/0000-0003-4407-1487 Sarah von Götz Department of Psychology, Graz Open Science Initiative, University of Graz, [email protected], https://orcid.org/0009-0006-9430-8867 Abstract Study preregistration – a written plan on how to conduct a future study – is an important, yet underappreciated Open Science practice. In this article, we describe the emergence of preregistration in medicine and psychology and how science as a whole may benefit from it. We also show that the uptake of preregistration is very slow thus far among researchers in the social and behavioral sciences. There may be several reservations and misconceptions for this, which we address. We hope that a short and accessible tutorial may motivate researchers also from more fields to start using preregistration in their workflow. Keywords: preregistration, registered report, clinical trial, Open Science This manuscript is currently under peer review. Cite at your own risk Preregistration 2 Introduction The concept of a preregistration is simple and straight forward: It is a written and publicly accessible plan describing why and how a researcher aims at conducting and analyzing a scientific study in the future thus providing a level of transparency of one’s research. Although this seems like a trivial task – and even one that laypersons may believe to be common in science anyways – preregistrations have not been around for long. In this article, we want to briefly review how and under which circumstances preregistration was introduced to research on human subjects, how several preregistration forms developed based on the original idea, and how effective preregistrations are to tackle widespread problems of publication bias. Next, we will present how preregistration is adopted by researchers, particularly PhD students, in psychology—a field where preregistration has recently become more common— over time. Then, we will provide a quick tutorial on how to implement preregistration in one’s research using the popular Open Science Framework repository (OSF). Finally, we will address some common critiques of preregistration. This article aims to give a quick overview on preregistration and remove barriers researchers might have towards implementing them by providing a quick and easy tutorial on how to get started, thus motivating them to integrate preregistration in their own research. Emergence of Preregistration Although researchers presumably spend a lot of time working out the details on how to set up their scientific studies, the idea of a written digital plan that lays out the specifics of a future study and that is publicly accessible is not older than a quarter century. In medical research, preregistration was first implemented around the year 2000 in response to an emerging credibility crisis: many clinical studies probing new medications (usually conducted in randomized controlled trials) seemed to show positive effects (i.e., patients became healthy quickly following a new medication compared to placebo or an older medication). However, after many initial studies, follow-up studies of medical treatments sometimes did not show the same health benefits – despite seemingly carefully conducted research. As a response, the US National Institute for Health launched the first registry for studies – ClinicalTrials.gov – where researchers had to indicate the aims and set-up of their planned studies. It was received positively and just five years later the International Committee of Medical Journal Editors of major scientific journals made registration of studies a mandatory requirement for later publication (McCray & Ide, 2000; CinicalTrials.gov, 2025). The effectiveness of this early form of a preregistration became evident in 2010s: initial studies in medical research now often did not report exaggerated health benefits anymore. Instead, findings became more nuanced and oftentimes even showed null effects (Kaplan & Irvin, 2015). The public registration of the research plan resulted in fewer researchers trying to circumvent their initial idea to publish “positive” results that turned out to be fluke findings later. The social and behavioral sciences had their own simmering credibility and replication crisis, which flared up very suddenly in the 2010s. First, concerns arose that some findings in the social Preregistration 3 and cognitive-psychological domain seemed to be incompatible with physical understandings of the world or just absurd to be taken for granted (Lakens, 2025). Most prominently, one paper in a prestigious psychology journal demonstrated that some people could “feel the future” by seemingly correctly guessing the position of erotic pictures on an empty computer screen. Not only did this finding cause some media controversy (French, 2012), but also proved to be unreliable in a followup study (Ritchie et al., 2012). More often than not did these findings show positive – statistically significant – effects that appeared to confirm the researchers’ hypotheses, implying that they are almost always right with their predictions. This bias towards positive results is common in many research fields (Fanelli, 2010). However, a group of methodologists convincingly argued that many findings could simply be fluke results stemming from undisclosed analyses steps (e.g., researchers flexibly run several analyses, but only report the one’s that show a positive finding). When carefully conducting replication studies, these results would likely vanish (Simmons et al., 2011). This was confirmed in a large-scale replication project by the Open Science Collaboration (2015) of 100 published studies from major journals could show indeed many of these findings (more than 60%) did not replicate – a major upset for researchers in the field. Simmons and colleagues (2011) argued that the issues of replicability and transparency need to be addressed far beyond researchers only sharing their data and methods, which constitute necessary, but not sufficient Open Science practices for more transparency. Instead, researchers would need to hold commit to and uphold important parts of their research idea beforehand to decrease the chance that they would later deviate too much from their initial plan through hindsight and confirmation bias in their analysis decisions. In other words, they needed to plan ahead more carefully, which part of their research is confirmatory (hypotheses testing) and which part is exploratory (hypotheses generating; see also Nosek et al., 2018). Hence, the idea of study preregistration started to become popular in psychology. Key Points of a Preregistration Simmons and colleagues (Data Colada, 2015) as well as Nosek and his Center for Open Science (n.d.) became a vanguard of preregistration by providing early online registries to popularize the concept in psychology and other social and behavioral sciences and promoting the concept among researchers. If one compares popular preregistration forms on these repositories (e.g., the popular “AsPredicted” form or the “OSF Preregistration” form), principles of what a preregistration should entail to be informative become evident (see also McPhetres, 2020; Simmons et al., 2021). These principles can be roughly subsumed under the following key points of a preregistration – parts that should be fixed beforehand to avoid flexibility in data analyses steps: a) Research question and hypotheses (i.e., what question do the researchers want to answer and what specific prediction, if any, is being made for the confirmatory part?) Preregistration 4 b) Research design and variables (i.e., how is the study conceptualized and set up and which variables are measured or experimentally manipulated?) c) Sample size justification (i.e., what is the population and the sample size drawn from it? Is this sample size sufficient to find a hypothesized effect given that it exists?) d) Analysis plan including preparation steps (i.e., how is the data being prepared to become ready for the analysis? What are the specific analysis strategies?) Development of Preregistration and Registered Reports While the initial preregistration forms focused on experimental studies in psychology, several groups of Open Science proponents from different disciplines have since updated and refined the idea of study preregistration, making it suitable for all kinds of research, including qualitative studies (Haven & van Grootel, 2019), replication studies (Brandt et al., 2014), systematic reviews (van den Akker et al., 2023), or secondary analyses of existing data (van den Akker et al., 2021). These efforts also reflect that researchers do not only recognize the non-replicability as an issue of social and cognitive psychology (in which the replication crisis started), but as a fundamental issue to all scientific disciplines that apply some kind of hypothesis testing. After all, excessive positive results are present in most research disciplines (see Fanelli, 2010), and many of these deal with own replication issues (e.g., experimental economics, Camerer et al., 2016, social sciences, Camerer et al., 2018, and pre-clinical cancer research, Errington et al., 2021). Several journals even brought the preregistration concept one step further: While preregistering a study before data collecting provides a level of transparency, there is no formal review process beforehand. The journal format of Registered Reports (Chambers, 2013, Chambers & Tzavella, 2022) addresses this problem: this format includes a formal review process before data is collected. This way, researchers write a journal-style preregistration and submit it to a journal before the begin of the study. The peer-review then critiques the theory and methods (rather than the results) and can give constructive feedback before any data is collected. This effectively allows the researchers to still implement external ideas in their study design. Registered Reports come with the premise that the research is evaluated based on its actual quality and not whether its results turn out to be positive, thus also providing a method of preventing a publication bias that is skewed towards positive results (Chambers, 2017). Similar to medical research, preregistration in the social and behavioral sciences, proved to be effective: effect sizes in published studies that utilized preregistration are often much smaller than in regular studies (Schäfer & Schwarz, 2019), implying that that the literature is less swamped with fluke findings. This is particularly true for Registered Reports, which also seem to be of higher quality in terms of their theorizing, methods, and implications (Scheel et al., 2020; Soderberg et al., 2021). Although registries like the Open Science Framework (https://osf.io/) provide researchers with a highly streamlined and low effort way of preregistering their studies, the question remains, how many researchers actually chose to apply it in their research. Preregistration 5 Uptake of Preregistration Contrary to randomized controlled research in medicine, studies in the social and behavioral sciences do not include mandatory preregistration before data collection. However, Open Science communities and initiatives have emerged across universities to promote the uptake OS practices – including preregistration. These initiatives often employ various methods such as implementing, journal clubs, organizing events, teaching courses etc. to convince researchers of the importance of OS practices to improve research as a whole as well as providing them with the skills they need to successfully implement them (e.g., Orben, 2019, Armeni et al., 2021, Brohmer et al. 2025 in this issue). Nonetheless, in the end, the success of such endeavors may be measured in how popular OS practices have become in research output, that is, in scientific publications. For preregistrations, an increase throughout the last decade may indicate that researchers started recognizing these practices being important to improve transparency in general. Some recent data from Hardwicke and colleagues (2024) suggests that there is indeed an uptake in applying preregistration: Whereas between 2016 to 2018, about 7% of research articles in psychology reported a functional link to a preregistration form, this number had increased to 20% in 2022. Brohmer and Hoffmann (2025) did something similar, but focused on PhD students, because they are assumed to just having acquired their primary research skills, which they will likely use until the end of their career. In a sample of studies (N = 379) from 91 dissertations from psychology departments at two universities, they found that 20% (n = 80) contained a preregistration, confirming the results by Hardwicke (see Figure 1). Figure 1. Use of preregistration in studies of psychology dissertations 2018-22. Preregistration 6 From all studies that used preregistration, most were based on the beginner-friendly “AsPredicted” form (16%, n = 60), few used the more comprehensive “OSF Preregistration” form (3.7%, n = 14), and n = 3 (0.8%) conducted a Registered Report (see also Figure 1). Hence, ECRs started using preregistration, but there is certainly room for improvement. If one looks at the prevalence of Registered Reports in journals, they do not appear to be very popular yet: we compared the proportion of Registered Reports in two journals from psychology: Collabra: Psychology emerged as a small journal dedicated to transparency and reproducibility throughout the last couple of years, whereas Nature Human Behavior quickly became a high-impact journal, focusing on more broadly interesting and high-impact results. Both journals introduced Registered Reports in 2017. Collabra’s rate of Registered Reports went up from zero to 16% in 2024 (14 out of 88 published research articles). For Nature Human Behavior only about 1% of published studies in 2024 were in a Registered Report format (two out of 153 published research articles). This may indicate two things: I) The few researchers, who aim at conducting a Registered Report may rather “play it safe” by submitting it to a journal that values transparency and reproducibility explicitly. A potential reason might be that they struggle with evaluating the importance and impact of their research before knowing their results. 1 II) For a similar reason, the editors of highimpact journals may reject Registered Reports more often as they fear potential null findings (which are more common in Registered Reports), which they think are less citable. Unfortunately, there is little data on the adoption of preregistration in research fields beyond psychology. But as the awareness of the importance of OS practices only recently gained traction in fields like sociology, consumer research, or education research, a slow and delayed uptake of preregistration is rather plausible. With our following tutorial, we hope to stimulate a further uptake of preregistration across fields. Four Easy Steps to Preregistration – A Brief Tutorial Certainly, science would benefit from a quicker uptake of OS practices and particularly from the uptake of study preregistration. However, it can be difficult to know where to start as this methodological skill is often not taught in university courses. Moreover, researchers at later career stages might have received their training before the emergence of preregistration. In this section, we will provide a four-step tutorial for preregistration in the hopes that this will make it easy for interested researchers to get started. We have chosen the Open Science Framework (OSF), as this is in our opinion the most accessible registry and researchers might already be familiar with it for other OS-related purposes (e.g., as a repository for data sharing). Therefore, a prerequisite is that interested readers create an account on the OSF (https://osf.io/). It is also important that you already have a relatively concrete 1 For researchers, who value OS a lot, another reason might be that the smaller, transparency-oriented journal is more in line with their values and that the open-access publication charges in that journal are not as high. Preregistration 7 study idea in their mind, including the research questions, hypotheses and methodology. The OSF provides many available preregistration formats, which differ in level of detail and type of research they can be applied to (see https://help.osf.io/article/229-select-a-registration-template). As this is a beginner’s tutorial, we will focus on the “AsPredicted” format which is relatively simple and can be applied for most experimental studies. The four steps are illustrated in Figure 2. 2 Step 1: Create a New Project “Projects” on the OSF are superordinate folders, which can be created for single studies or whole research projects (e.g., a dissertation or a post-doc grant project). Upon clicking on the button “Create New Project” (Figure 2a), one should select a short, but clearly descriptive title (e.g., “Study 1: Cooperative Behavior”). When selecting a server, it is advisable to keep local data protection regulations in mind (e.g., European researchers may select the Frankfurt server). Figure 2. Four Steps to a Preregistration via the Open Science Framework Note: red ellipses and arrows with annotation are added for clarification; form is available on https://osf.io/; screenshots were taken in July 2025 2 A video tutorial in German language is provided on the Unitube portal of the University of Graz: https://unitube.unigraz.at/portal/aufzeichnungen.html?id=e1afd6a9-c7cf-4faa-a77d-4953c501c67f Preregistration 8 Step 2: Navigate to the Preregistration Form The landing project page is now created and should look like as in Figure 2b. Note that this project is “private”, which means it is only visible to the author themselves and will remain private until the authors chose to share the project (e.g., during publication of the research paper). Following the tabs on the top, one could later add information on the project by clicking on the Wiki tab, add supplemental documents, methods, and data files via the Files tab, or add co-authors and collaborators to the project via the Contributors tab. For adding a preregistration, one should simply click on the Registrations tab and then on “create new registration”. Step 3: Choose a Preregistration Form As we mentioned above, there are many forms available, but we select the “AsPredicted” form from the list (Figure 2c). 3 Researchers with more preregistration experience may choose the “OSF Preregistration” form, which is more informative in regard to some details (i.e., option 1, see Bakker et al., 2020) or another form that closely aligns with their study design (e.g., Replication Recipe). After selection, click on “create draft”. Step 4: Fill Out the Form and Submit It This final step requires some time, depending on the study’s complexity and how well thoughtthrough all study aspects already are. For a simple study, where all methodological and analytical decisions have already been made, one may plan to invest about 20 to 30 minutes. But the more details one can add, the better for the purpose of the transparency. Following a short study description (see Figure 2d), one has to fill out eight more questions (or headings), which contain information on the above-mentioned four key points A to D. Here we describe these points for the “AsPredicted” form to provide researchers with some guidance for transparency: 1. Title and Description. Here, the purpose of the study should be contextualized and the research should be mentioned (e.g., online experiment via a survey platform). (Corresponds with key point b) 2. Has the Data Collection Started? Choose the appropriate option – usually it is “no”. If “yes” is chosen, elaborate in point 9 why you still believe that this study can be preregistered. 3. Hypotheses. For confirmatory studies, one should list all hypotheses, which will be tested in the study. Although the heading asks for hypotheses, one could also list further exploratory research questions. (Corresponds with key point a) 3 Even if the planned study at hand is not an experiment, but rather an observational / correlational study, one can use customize the content of fit the purpose. E.g., instead of “conditions”, one may simply describe all predictor variables. Preregistration 9 4. Dependent Variable. This is the variable (or variables) of interest that is measured to draw a conclusion from (i.e., the outcome variable on which two groups are compared). (Corresponds with key point b) 5. Conditions. In a simple experiment, this point constitutes the experimental and control group. It is beneficial to include some information what the treatment of the experimental group entails (e.g., a new medication). Also, one could add relevant control variables (e.g., participant gender). For correlational studies, one should add here any independent, predictor or moderation variable which is relevant for the study. (Corresponds with point b) 6. Analyses. How is the data analyzed to draw conclusions from it and what is the inferential criterium (e.g., the p-value of a test statistic)? Here, one should also add important preprocessing information, such as, how the dependent variable is transformed or how composite scores are calculated. (Corresponds with key point d) 7. Outliers and Exclusion. Sometimes, cases have to be excluded before data analysis (e.g., because of extreme values on the dependent variable, unmet inclusion criteria (e.g., related to a minimum or maximum age, language requirements, etc.), or doubts about data quality). These exclusion rules should be defined here. (Corresponds with key point d) 8. Sample Size. One should indicate an anticipated sample size that one aims to collect. It is advisable to also provide a sample size justification (e.g., via a statistical power analysis) and distinguish between collected (raw) sample size and analyzed sample size (e.g., after exclusion, see point 7). (Corresponds with key point c) 9. Other. It could be that data on additional variables is collected or the study is part of a larger research project. Such specifics, including further exploratory analysis ideas, can be optionally indicated here. Up until this point, the preregistration is still in “private” mode. Upon pressing the “submit” button, which officially registers the study, researchers can set an embargo to when the preregistration becomes visible and findable This can be up to four years in the future and gives researchers time to conduct their study and publish their findings. The link to the preregistration should then be made available as part of the final publication. This will make their study plan transparent in the context of the published study. 4 Reservations Against Preregistrations Finally, we want to address some frequently raised criticisms of preregistration that are commonly voiced and that we have encountered during our work at the Open Science initiatives (see 4 Note that for the peer-review process, researchers can also provide an anonymized view-only link to their preregistration, as a peer review is usually double-blind. To achieve this, one can follow the information on this link: https://help.osf.io/article/155-create-a-view-only-link-for-a-registration