Costs and Benefits of Open Science: Contributing to the Development of a Rigorous Assessment Framework

Costs and Benefits of Open Science: Contributing to the Development of a Rigorous Assessment Framework Gelsomina Catalano, Erica Delugas, and Silvia Vignetti Abstract The concept of Open Science (OS) is transforming the landscape of scientific research by promoting collaboration, transparency, and innovation. Acknowledged by policymakers and international organisations, OS is integrated into policy agendas recognising its potential to shape the future of research. Despite significant progress, Open Science faces challenges in showing economic impacts, which undermines its maximal adoption. Empirical evidence on positive economic outcomes, such as cost savings and the emergence of new products and collaborations, exist, but there is a scarcity of comprehensive economic impact studies comparing open and closed science. This article advocates for the use of Cost–Benefit Analysis (CBA) as an analytical tool to systematically assess the advantages and disadvantages of OS. CBA, traditionally applied to sectors like transport and health, can provide a structured framework for mapping and evaluating the costs and benefits of OS, contributing to a more informed understanding of its societal desirability. Keywords Open science ·Cost-benefit analysis ·Economic impact ·Efficiency · Cost savings 1 Introduction The concept of Open Science (OS) is deeply reshaping the landscape of scientific research production and dissemination. This paradigm shift in research practices is recognised for its potential to accelerate scientific progress, foster innovation, and promote transparency and collaboration. Policymakers, international organisations and the European Union have acknowledged the pivotal role of OS in shaping the future of research and have incorporated it into their policy agendas (e.g. [5,10,13]). The promotion of OS stems from the consideration of its multiple potential advantages. From increased collaboration among scientists to accelerated discoveries in G. Catalano ·E. Delugas ·S. Vignetti (B) CSIL, Milan, Italy e-mail: [email protected] © The Author(s) 2025 J. Gutleber and P. Charitos (eds.), The Economics of Big Science 2.0, Science Policy Reports, https://doi.org/10.1007/978-3-031-60931-2_10 127 128 G. Catalano et al. companies, from enhanced transparency and reproducibility through public engagement and trust in science by the society, OS benefits extend along impact pathways spreading from the academic to the economic and societal domains. Despite significant strides in the last two decades (see [11]), OS still faces obstacles undermining its full potential. One critical challenge is the lack of comprehensive evidence regarding the economic impacts of OS and how they can be maximised to accelerate its adoption. Few empirical studies are available measuring positive economic outcomes resulting from the improved accessibility and efficiency of research findings. They typically include cost savings in terms of access, labour, and transaction costs [1,2, 13]. Beyond efficiency, OS has been credited with facilitating the emergence of new products, services, companies and research collaborations [6]. Yet, the scarcity of economic impact studies comparing open and closed science is a notable issue [9]. Existing literature predominantly discusses positive or, to a lesser extent, negative effects, mechanisms, drivers, and barriers, often relying on theoretical arguments rather than empirical assessments of costs and benefits. This evidence gap has possibly led to inadequately documented high expectations regarding the impacts of OS, especially in the economic sphere. Cost–Benefit Analysis (CBA) is an analytical instrument to evaluate the societal desirability of an investment decision. It achieves this by examining the costs and benefits associated with the decision, ultimately determining the net welfare change it brings about [4]. Initially applied to traditional sectors like transport, environment, energy, health, and education, CBA has expanded its scope to encompass the economic impact of science [7]. This article explores how CBA can support a systematic mapping and assessment of advantages and disadvantages of OS. Two are the main contributions: (i) helping a rigorous structuring of the analysis and (ii) providing a framework to mapping and assessing costs and benefits. 2 Structuring the Analysis While OS is often referred to as a movement or a set of practices, for the purpose of a systematic assessment, it is necessary to shift from this theoretical concept to the practical domain of OS projects. Focusing on one specific OS project or initiative is the first necessary condition to run a realistic assessment and draw conclusive evidence on actual impacts. It can be a digital infrastructure such as an open repository designed to grant open access to journal articles, an open data platform providing free access to specific data, or free software that facilitates the processing of data and information. The selected project should deliver specific services, cater to specific user groups, have recognisable boundaries, identifiable resources, traceable effects, and exhibit a defined time horizon. Additionally, the degree of openness of the project is highlighted as a critical factor influencing impact generation, particularly relevant when identifying the counterfactual scenario. First, the extent of openness is not solely tied to providing open access Costs and Benefits of Open Science: Contributing to the Development … 129 to scientific information; it also pertains to the feasibility of comprehending, validating, and leveraging such information. For example, if complex data are presented in a simplified format and can be accessed by a broader audience rather than only by scientists with specific technical expertise, the impact is likely to be more direct and extend to diverse beneficiaries. The degree of openness can encompass the accessibility conditions of research outputs. In the case of open-source software, where users have open access to the source code and can make modifications or create routines, it should not be assumed that users have unrestricted free access to the software. Instead, open access to the software code may be available under a subscription payment. Therefore, the degree of openness of a research output is likely to influence the generation of impact. By comparing what actually occurred with what might have happened under different circumstances, the use of counterfactual enables a better understanding of the causal relationships and consequences of a particular project. For this reason, the selection of the counterfactual scenario is a critical aspect. It involves the definition of what would happen in the absence of the OS project: whether the impact would occur similarly or if the impact intensity would diminish or completely disappear. Once the counterfactual scenario is identified, all costs and benefits are identified and assessed in an incremental way (with-without project scenario). At least two counterfactual scenarios warrant consideration. The first is applicable when an OS project facilitates access to research products that would not have been available otherwise. This acknowledges that the OS project is instrumental in creating and disseminating materials that either did not exist in a comparable form previously or were entirely inaccessible. An illustration is data repositories collecting shared datasets that were previously unavailable, even though their primary sources were accessible. In this case, the incremental scenario aligns with the OS project scenario itself. In contrast, a closed scenario is apt when a scientific product is already shared, but access comes at a cost. This often pertains to scientific journals operating within closed environments, where individuals must pay a subscription fee or the price of individual articles for access. Another example is software usage that is subject to licensing fees. Here, the counterfactual scenario should encompass the costs and benefits associated with the existing status quo, providing a basis for comparison with those related to the OS project scenario to determine the net effect. Being often hypothetical and not directly observable, the selection of an appropriate counterfactual scenario involves extensive discussions with researchers and specialists involved. This is crucial because determining the counterfactual scenario, in some instances, necessitates robust assumptions. In such cases, it becomes practical to approximate and refer to the next best alternative. 130 G. Catalano et al. 3 Mapping and Measuring Costs and Benefits 3.1 Costs: Actual Use of Resources and Lost Opportunities Costs in the context of OS projects are defined as resources used in the process of structuring, operating, maintaining, and upgrading the OS project. According to economic theory, every factor of production, such as capital, labour, and knowledge, carries an associated cost. They should be measured through their opportunity costs, representing the value of the next best alternative forgone when choosing a specific resource for a particular purpose. It is advisable to start the measurement of economic costs with financial costs and then correct them as needed to get the shadow costs [8]. The main challenge related to costs is related to their attribution. Attributing costs to a single OS service can be challenging due to complex connections between costs, the involvement of numerous actors, and the common scenario where an OS service is just one component supplied by the same institution. Three types of social costs can be identified: •Set-up costs encompass resources associated with the initial establishment of the project. They are tangible and intangible assets, start-up phase expenditures, personnel, and future upgrading that require significant changes in the technical approach. Additionally, resources related to closing down or discontinuing access to a specific database or research tool may be needed. •Maintenance costs include all resources essential for the operation and upkeep of newly developed or upgraded OS projects. They can be of fixed and variable types, with fixed costs remaining constant regardless of service volume and variable costs fluctuating based on output volume. Socio-economic costs associated with OS adoption, such as additional costs borne by users, for example, the development of the necessary skills, should also be considered. •Additional socio-economic costs are associated with the preparation phase of materials to be shared on open-access platforms. These costs involve the extra time invested by researchers to prepare materials and align them with platform requirements. Other social costs on the user’s side are the opportunity cost of patenting and potential career implications of opening up the developed knowledge instead of protecting it, especially in academic fields where OS adoption is nascent or in sectors where scientific knowledge appropriation can yield significant economic benefits (e.g. pharma). 3.2 Benefits of OS: Efficiency and Enablement Gains The benefits associated with OS projects primarily centre around efficiency gains, signifying the attainment of the same research or innovation output with reduced input. The concept of cost savings within OS encompasses enhanced production efficiency. It involves saving both time and money due to OS, leading to reduced Costs and Benefits of Open Science: Contributing to the Development … 131 expenses and resource conservation in the scientific production process. The cost savings benefit, acting as a macro category, comprises four distinct benefits under which various sources and stages of savings emerge for professionals, enterprises, and researchers utilising OS outputs. 3.2.1 Cost Savings The first type of cost savings is access cost savings, capturing costs avoided when accessing essential knowledge or tools within a closed environment. It refers to avoided expenses associated with accessing proprietary or paid resources, such as subscription fees or licensing charges. From another perspective, this benefit can be seen as the cost savings achieved by not having to replicate, developing it from scratch, the same type of research output that may not be available without payment. In this case, the focus is on valuing the saved production costs (i.e. the Long-Run Marginal Cost method, [7]) rather than the monetary savings. With a focus on the users’ perspective, for example when market prices are either unavailable or do not accurately reflect the true economic value of savings, a reliable metric can be the monetary amount that individuals are willing to pay (WTP) to enjoy a specific benefit or avoid a particular cost [3]. Stated preference techniques are employed to determine WTP, allowing the elicitation of people’s preferences in hypothetical scenarios. Storage cost savings is the second type of savings. OS services enable the substitution of private data storage with open repositories or eliminate the necessity to store research outputs. The assessment approach involves a thorough examination of the expenses that are circumvented or diminished due to the utilisation of OS services, providing a quantification of the economic benefits derived from the costeffective storage alternatives offered by open repositories. The avoided cost method (if a market exists) is suitable for assessing these savings, considering factors like storage space, market prices, or LRMC (if a market does not exist and self-production is the alternative to the OS service). The third type of savings is related to labour, reflecting the gain in opportunity costs by saving working time through OS knowledge and tools. Shared codes and protocols reduce the need for coding from scratch, enabling users to build upon existing code and save working time. Sharing data mining techniques automates information collection, minimising the manual effort required for data entry. The existence of open data enhances efficiency in finding needed information, as open data is inherently more findable. OS projects also contribute to time savings in designing research projects and writing papers by facilitating the easier circulation of research outputs, reducing duplications of codes, papers, and data. Monetising this benefit involves assessing the time saved by users using methods like shadow wages or WTP. The last category is transaction cost savings, related to time spent navigating copyright agreements, negotiating access to specific data, or other research outputs. 132 G. Catalano et al. Open data, protocols, and software can reduce time spent on such procedures. The social value of this benefit is evaluated similarly to labour cost savings. While these categories often overlap, caution is needed to avoid double counting. These insights contribute to understanding the multifaceted benefits of OS adoption and provide a framework for assessing its socio-economic impact. 3.2.2 Enablement In some instances, benefits of OS can extend to enablement, encompassing activities that emerge from an open science environment and are less likely to materialise in a counterfactual scenario. Along the causal pathway timeline, enablement benefits are observed to occur subsequent to efficiency gains. However, attributing enablement gains solely to OS is not always justified. Other contributing factors or inputs likely play significant roles in the causal relationship between OS and the realisation of new products and services. Enablement often results from savings in time and money, allowing researchers and enterprises to focus on research tasks that might have been deferred if OS services had not provided access to knowledge. Enablement benefits stem from knowledge spillovers resulting from the widespread dissemination of research outputs. Scientific journals are a primary medium for the scientific community and industry to stay informed about cuttingedge research, therefore, norms and pricing governing journal access are crucial. OS practices likely expand the possibility for knowledge production within the scientific community through open access and open data, enabling individuals to access and reuse publicly funded research results and data openly. Similarly, enterprises, by accessing OS outputs and utilising OS services, are likely to foster innovation. Additionally, early and easy accessibility to knowledge is likely to enhance the likelihood of patent registrations. The rapid dissemination of knowledge facilitated by an OS environment provides a notable advantage. Open publications, codes, data, software, and research discoveries can be swiftly and widely shared, fostering a climate of innovation within enterprises. The transparency and openness allowed by OS projects lead to more efficient collaboration and knowledge exchange, reducing research and development costs and enabling exploration of new ideas without the constraints of expensive access fees or restrictive licensing agreements. This, in turn, offers opportunities for more transparent science and allows businesses to pursue innovative avenues that might have been financially prohibitive in a closed system. Enterprises gain the potential to develop novel products, services, and technologies that may not have emerged in a less collaborative and accessible environment. In cases where enterprises lack the capacity to exploit shared research outputs effectively, OS projects can play a catalytic role in the creation of start-ups and spinoffs, addressing knowledge gaps. These new enterprises emerge to bridge the gap by preparing and processing complex open data, serving as intermediaries to make valuable information accessible to a broader range of industries and professionals. Costs and Benefits of Open Science: Contributing to the Development … 133 The economic value of this benefit is typically quantified by measuring the incremental shadow profits resulting from the sale of new or improved products, services, and technologies compared to a hypothetical scenario without OS projects. Another dimension of enablement gain is associated with patents and other forms of intellectual property rights, providing a significant advantage for enterprises. Using OS services can contribute to an increase in patent registrations for innovative products, services, and technologies by enterprises. When a patent is registered, it generates private returns for the inventor and the potential for knowledge spillover to society. When evaluating this benefit, it is essential to avoid double counting of the expected shadow profit generated by the new product. The marginal social value of patents is typically associated with both a private value (for the enterprise) and an externality (for society). Careful consideration is needed to distinguish and appropriately quantify these aspects in the assessment of the overall social and economic impact of OS projects. 4 Conclusions Despite the significant progress made by the OS movement, several barriers hinder its full potential. These include the costs associated with openness, insufficient skills in data management, and diverse regulatory frameworks. To develop effective OS policies, a thorough understanding of OS practices and their impacts is crucial. While progress has been made in understanding OS dynamics within the research system, more limited evidence is available on how it affects economies and societies. Existing studies often focus on the OS movement in general, with less clarity on the role of individual OS projects or initiatives. Moreover, many studies assess the impacts of OS without comparing it to non-OS approaches, potentially leading to an overestimation of OS benefits. Adopting a CBA framework for OS would provide a systematic and comparative assessment of the socio-economic costs and benefits of OS projects. Overall, the framework can contribute to a more nuanced understanding of the economic impacts of OS, facilitating evidence-based decision-making and policy formulation in the realm of scientific research and innovation. Applying CBA to assess the socioeconomic impacts of OS requires to focus on direct, more short-term outcomes and impacts, encompassing benefits directly related to the OS project itself. For example, the availability of open data may reduce the time spent on data creation. These shortterm outcomes are within the control of the participating organisations and can be upstream (e.g., affecting scientists involved in the set-up phase) or downstream (e.g., impacting end-users). On the other hand, long-term outcomes are influenced not only by the OS project itself but also by additional factors. For instance, the introduction of new e-health technology enabled by open science research may lead to a decrease in deaths caused by strokes, but this outcome requires various additional activities, investments, and factors to materialise. 134 G. Catalano et al. Assessing long-term impacts falls outside the scope of CBA and necessitates other methods. Broader causal pathways frameworks can provide a more comprehensive perspective on the long-term outcomes of OS projects. The impact pathways identified within the framework of OS practices represent the non-linear sequences of steps connecting inputs to immediate and measurable outputs and extending to more indirect, broader impacts that may not be easily quantifiable. While CBA focuses on causal impacts through an incremental approach, impact pathways offer a progressively broader perspective, tracing the causal chain of OS impacts and identifying the mechanisms and conditions allowing these impacts to materialise. Still, taking into consideration a clear demarcation of the project boundaries, a proper counterfactual and the systematic assessment of both advantages and disadvantages, remain crucial for a proper impact assessment. Acknowledgements This paper draws from research activities carried out in the frame of the research project Open Science Impact Pathways (PathOS) addressed to identify and quantify the Key Impact Pathways of Open Science relating to the research system and its interrelations with economic and societal actors. The project is co-funded by the European Union’s Horizon Europe framework programme under the grant agreement No. 101058728, whose financial support is gratefully acknowledged. Further details can be found at: https://pathos-project.eu/ References 1. Beagrie N, Houghton J (2016) The value and impact of the European Bioinformatics 2. Beagrie N, Houghton J (2021) Data-driven discovery. https://www.embl.org/documents/wpcontent/uploads/2021/10/EMBL-EBI-impact-report-2021.pdf 3. Boadway R (2006) Principles of cost-benefit analysis. Public Policy Rev 2(1) 4. Boardman AE, Greenberg DH, Vining AR, Weimer DL (2006) Cost benefit analysis—concepts and practice, 3rd edn. Pearson Education, London 5. European Commission, Directorate-General for Research and Innovation (2015) Open innovation, open science, open to the world: a vision for Europe, Publications Office. https://data. europa.eu/doi/10.2777/061652 6. Fell MJ (2019) The economic impacts of open science: a rapid evidence assessment. MDPI. https://doi.org/10.3390/publications7030046 7. Florio M (2019) Investing in science: social cost-benefit analysis of research infrastructures. Mit Press. Koundouri P, Chatzistamoulou N, Dávila OG, Giannouli A, Kourogenis N, Xepapadeas A, Xepapadeas P (2021) Open access in scientific Information: sustainability model and business plan for the infrastructure and organisation of OpenAIRE. J Benefit-Cost Anal 12(1):170–198 8. Florio M, Pancotti C (2023) Applied welfare economics: cost-benefit analysis of projects and policies, 2nd edn. Routledge, London 9. Klebel T, Cole NL, Tsipouri L, Kormann E, Karasz I, Liarti S, Stoy L, Traag V, Vignetti S, Ross-Hellauer T (2023) PathOS—D1.2 scoping review of open science impact. Zenodo. https:// doi.org/10.5281/zenodo.7883699 10. OECD (2015) Making open science a reality. OECD Science, Technology and Industry Policy Papers, No. 25. OECD Publishing, Paris. https://doi.org/10.1787/5jrs2f963zs1-en Costs and Benefits of Open Science: Contributing to the Development … 135 11. Paic A (2021) Open science—enabling discovery in the digital age. Going Digital Toolkit Note, No. 13. https://goingdigital.oecd.org/data/notes/No13_ToolkitNote_OpenScience.pdf 12. Sweeny DK, Fridman DM, Rasmussen PB (2017) Estimating the value and impact of Nectar Virtual Laboratories 13. UNESCO Open Science Outlook (2023) Status and trends around the world. Available at https://unesdoc.unesco.org/ark:/48223/pf0000387324 Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. The images or other third party material in this chapter are included in the chapter’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.