scieee AI-readable full text Open interactive document viewer

The Impact of Open Science – what do we really know?

Ross-Hellauer, Tony; Cole, Nicki Lisa; Klebel, Thomas; Tsipouri, Lena

Publisher
Zenodo
DOI
10.7557/5.7808
Repository
zenodo.org
Language
en

Abstract

IntroductionOpen Science (OS) is increasingly mainstream, with wide adoption of practices like Open Access and FAIR data. While systems and data sources for monitoring the uptake of these practices are now relatively mature, research to establish real-world, long-term impacts is largely scattered and patchy. In this talk, I (on behalf of the PathOS project team) will present work to scope and synthesise current evidence of: Academic impact: E.g., OS contributions to research efficiency, quality, collaboration, and equity. Societal impact: E.g., effects on policy, governance, education, climate, public trust, and health. Economic impact: E.g., influence on industry productivity, innovation, and cost efficiency. We used the PRISMA-SCR methodology (Tricco et al. 2018) to systematically scope literature from 2000-2022. Starting from a total of more than 30,000 initially retrieved records, the study team identified, selected, analysed, and synthesised peer-reviewed literature from academic databases and further “grey literature”/preprints from the Web. Full results for two studies (academic and societal impact) are already available online (Cole et al. 2024; Klebel et al. 2024). The third, on economic impact, is in preparation and will soon be made available as a preprint and submitted for journal publication. Results415 studies demonstrated academic impact, 196 societal impact, and 70 economic impact. Most academic studies focus on Open Access, showing increased citation rates but raising equity concerns due to the article processing charge (APC) model. Citizen Science improves data collection and education but has varying levels of data quality. Open/FAIR data leads to reuse benefits and better reproducibility. Societal impact studies highlight Citizen Science’s role in education, climate awareness, and policy engagement, though Open Access’s societal impact remains limited. Economic studies suggest theoretical benefits like cost savings and innovation, but empirical data is scarce. Discussion and conclusionThis talk will detail and reflect upon the these findings, including the uneven distribution of research on OS impacts, difficulties in assessing OS impact and establishing causality, the tendency for research to focus on areas where data is evenly available rather than where evidence is most needed, prior focus on uptake rather than impact, and the implications of all this for our understanding OS impact. Finally we will look ahead to PathOS’ future work on OS impact indicators, methods for cost-benefit analysis, and specific case-study investigations to deepen our understanding of OS’s influence on research efficiency, quality, equity, societal engagement, policy-making, and economic outcomes. View the presentation in this video recording.

Full text

PathOS_EU THE IMPACT OF OPEN SCIENCE WHAT DO WE REALLY KNOW? TONY ROSS-HELLAUER KNOW CENTER RESEARCH GMBH CO-AUTHORS: NICKI LISA COLE, THOMAS KLEBEL, LENA TSIPOURI 19th Munin Conference on Scholarly Publishing https://pathos-project.eu/ https://doi.org/10.7557/5.7808 Programme: Horizon Europe (WIDERA) Consortium: 10 international partners. Experts in Metaresearch, Scientometrics, Econometrics, Science & Society, Science Policy, Research Infrastructures Duration: Sep 2022 –Aug 2025 (36M) Budget: €2M Website: https://pathos-project.eu/ Open Science Impact Pathways •Based at Know Center Research GmbH in Graz, Austria •Metaresearchers studying the effects and impacts of services, policies, and practices to make research more open, responsible and reproducible Open Science Impact Pathways Munin 2024 Open Science is a plurality Open Science Impact Pathways 5 •Open Science is not a unified ideology but a diverse bunch of principles and practices •There are various routes to implementation of Open Science; the “how” is crucially important •Impact comes in many shapes and forms, very often intangible •Are we investing in the right instruments to truly realize the promise? •Are we achieving expected outcomes? •Are there unintended consequences? •What key pathways and enablers are driving impact? •How can we measure and monitor impacts and accurately attribute them to Open Science? Munin 2024 Open Science Impact Pathways •Map the Causal Pathways for Open Science •Design and estimate OS Impact Indicators for selected case studies •Use data-driven, AI-assisted methodologies •Formulate a Cost-Benefit Analysis framework for Open Science EOSC Symposium 2024 | Oct 21, 2024 | Berlin, Germany 6 Identify and quantify the Key Impact Pathways of Open Science across academia, society, and the economy to enhance understanding and drive informed policy-making. Open Science Impact Pathways Example: French Open Science Monitor Impact is relatively underexplored so far The primary focus has been measuring uptake, not impact •Uptake: Monitoring and measuring whether researchers, institutions, nations implementing OS practices •Impact: Monitoring and measuring the longterm, elementary and wide-spread changes attributable to Open Science Understanding impact is essential for knowing whether the intended longer-term ambitions of transition to Open Science (greater quality, equity, reproducibility, inclusion, innovation, …) are actually being realised? Munin 2024 https://frenchopensciencemonitor.esr.gouv.fr/ Open Science Impact Pathways Munin 2024 Open Science Impact Pathways Munin 2024 Studies followed Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews (PRISMA-ScR) methodology •Step 1: Identify relevant studies oAcademic literature in Scopus, Web of Science, OpenAlex, grey literature •Step 2: Selection of eligible studies by screening titles, abstracts, then fulltexts oAcademic, Societal, Economics impacts of Open Science and its constituent practices (Open Access, Open/FAIR Data, Open Methods, Open Code, Citizen Science, Open Evaluation) •Step 3: Data extraction from included studies oKey information: methods, findings, type of impact, aspect of OS •Step 4: Synthesis of data and reporting oPre-registered protocol: https://osf.io/m4rnc oFinal results reported in 3 separate papers (Academic, Societal, Economic impacts) Open Science Impact Pathways Munin 2024 Content of the literature •Scarce company data •Many theoretical papers on expected gains, but few with real evidence •Most papers on Open Science, Open Access and Open Data, few on Citizen Science, Open Source or Open Code •Most evidence comes from the medical and biotech sector Challenges/evidence gaps •Great difficulties in identifying either business (turnover/profits) or macroeconomic impacts •A lot more case studies and broader assessments are needed to allow for meta-analyses Open Science Impact Pathways Munin 2024 •Causality/correlation: difficulty of directly measuring relationships between interventions, outcomes, and impacts •Lack of standards for defining and measuring OS impact Open Science Impact Pathways Munin 2024 •Many case studies, often from those linked to initiatives •Case studies often difficult to generalize due to local contexts. •Is there a risk of publication bias from initiatives effectively evaluating themselves? •Streetlight effect –measuring what’s easy to measure •E.g., OA impact –a huge number of studies on OA citation advantage •Are we working enough to effectively measure progress on other Open Science aims (increasing quality, reproducibility, inclusion, innovation, etc.) Open Science Impact Pathways •Lack of robust evidence, except in key areas –knowledge gaps should be addressed •Need to better orchestrate, fund and sustain impact monitoring/evaluation efforts, especially those which employ strong causal methods •Vice versa, some areas with strong evidence of impact (e.g., Citizen Science) have a lack of policy support and funding •Monitoring of Open Science should increasingly focus on impact rather than uptake •Qualitative and mixed methods approaches are needed to study impact pathways Munin 2024 Open Science Impact Pathways Munin 2024 •Covers various aspects of quantifying impacts of Open Science •Covers indicators for Open Science uptake, academic, societal, and economic impact, and reproducibility •If an indicator can be readily operationalised, we provide readyto-go recipes to support its implementation •Also include more speculative indicators, not yet easily operationalised •Includes opening chapter with introduction to causality in science studies Open Science Impact Pathways Munin 2024 •Preprint of full results for academic impact: Klebel, T., Traag, V., Grypari, I., Stoy, L., & RossHellauer, T. (2024). The academic impact of Open Science: A scoping review. OSF. https://doi.org/10.31235/osf.io/ptjub •Publication for societal impact: Cole, N. L., Kormann, E., Klebel, T., Apartis, S., & RossHellauer, T. (2024). The societal impact of Open Science: A scoping review. Royal Society Open Science, 11(6), 240286. https://doi.org/10.1098/rsos.240286 •Write-up of full results for economic impact underway •Initial report on database search results: Klebel, T., Cole, N. L., Tsipouri, L., Kormann, E., Karasz, I., Liarti, S., Stoy, L., Traag, V., Vignetti, S., & Ross-Hellauer, T. (2023). PathOS -D1.2 Scoping Review of Open Science Impact. Zenodo. https://doi.org/10.5281/zenodo.7883699 •Zotero library available: https://pathos-project.eu/os-impact-evidence-library Open Science Impact Pathways _EU Munin 2024 Email: [email protected] https://doi.org/10.7557/5.7808