D5.9 Holistic RRI, SDG, Open Science and Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters

Holistic RRI, SDG, Open Science and Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters Page 2 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters Document Information Project Name BlueMissionAA – Building a coordination hub to support the Mission implementation in the Atlantic and Arctic Basin Grant Number 101093962 Deliverable number 5.9 Deliverable title Holistic RRI, SDG, Open Science and Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters WP number 5 Deliverable due date 30.04.2025 Submission date 30.04.2025 Dissemination levels PU (Public) Lead beneficiary Fraunhofer (IRB) Authors Dr. Hannah Keding, Barbara Klobucaric, Denise Glatzel, Anna Baumgärtner Contributors T5.3 contributors: IPMA, PLOCAN, NIVA, SPI, UCC Internal reviewers N/A Document history and changes Version Date Authors Description 0.1 08.04.2025 Hannah Keding, Barbara Klobucaric, Denise Glatzel, Anna Baumgärtner First draft 0.2 28.04.2025 Input from: Alexander Dernild (University of Southern Denmark), Barbara Klobucaric (Fraunhofer IRB), Harald Hasler-Sheetal (ICES), Ellen Johannesen (NIVA), Tania Montoto Martinez (PLOCAN), Baiba Prūse (UCC), Catriona Iulia Reid (UCC), Eric Retzlaff (Fraunhofer IRB), Oliver Wilson (Marine Institute) Comments and edits 1 30.04.2025 Hannah Keding, Denise Glatzel Final version Page 3 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters Authors | Hannah Keding, Barbara Klobucaric, Denise Glatzel, Anna Baumgärtner Publisher | BlueMissionAA Project Consortium How to cite | 2025. BLUEMISSIONAA Deliverable 5.9. Holistic RRI, SDG, Open Science and Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters. 69 pp. Supported by/In partnership with Page 4 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters Table of Contents Document Information .................................................................................................................... 2 Document history and changes ..................................................................................................... 2 Table of Contents ............................................................................................................................. 4 List of Figures ...................................................................................................................................... 6 List of Tables ....................................................................................................................................... 6 Glossary ............................................................................................................................................. 6 1 Summary ........................................................................................................................................ 8 2 Introduction ................................................................................................................................... 8 2.1 Task 5.3: Holistic RRI, SDG, Open Science and Social Impact evaluation framework 9 2.2 Aims ........................................................................................................................................ 9 2.3 Report structure .................................................................................................................. 11 3 Concepts of the reflection tool ................................................................................................ 12 3.1 Sustainable Development Goals (SDGs) ......................................................................... 12 3.2 Responsible Research and Innovation (RRI) .................................................................... 13 3.3 SDGs and RRI ....................................................................................................................... 14 3.4 Social Impact ...................................................................................................................... 14 3.5 Open Science ..................................................................................................................... 15 3.6 Merging the four concepts ............................................................................................... 17 4 How the reflection tool works .................................................................................................... 17 4.1 Reflection tool structure ..................................................................................................... 17 4.2 Reflection tool questions .................................................................................................... 18 5 How we developed the tool: methodology ........................................................................... 19 5.1 Developing the structure ................................................................................................... 20 5.2 Adapting the structure and including missing aspects ................................................. 21 5.3 Developing the evaluation questions .............................................................................. 23 6 How we coordinated, tested and improved the reflection tool: application .................... 25 6.1 Collaborating with partners and enhancing utilisation ................................................. 25 6.2 Developing an online reflection tool ............................................................................... 26 6.3 Revising the reflection tool before its test ........................................................................ 28 6.4 Testing the reflection tool .................................................................................................. 29 6.5 Results of the reflection tool’s test .................................................................................... 33 6.6 Improving the reflection tool based on user-feedback ................................................ 40 7 Discussion, dissemination and exploitation ............................................................................. 42 7.1 Limitations ............................................................................................................................ 42 7.2 Dissemination and exploitation strategies ....................................................................... 43 7.3 Discussion and next steps .................................................................................................. 44 Page 5 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters 8 Reflection Tools ........................................................................................................................... 45 8.1 Responsible and Sustainable Research Reflection Tool for Ocean and Waters ........ 45 8.2 Responsible and Sustainable Research Reflection Tool ................................................ 55 9 References .................................................................................................................................. 65 Page 6 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters List of Figures Figure 1: Sustainble Development Goals (United Nations, n.d.a) ................................................ 12 Figure 2: Starting page of the online reflection tool in LimeSurvey .............................................. 27 Figure 3: Mean values across the 19 evaluation questions ........................................................... 36 Figure 4. Box plot across the 19 evaluation questions .................................................................... 37 Figure 5: Spiderweb diagram comparing average self-evaluations of the project ‚OLAMUR‘ with means across all 29 participants ............................................................ 39 List of Tables Table 1: Structure of the reflection tool ............................................................................................ 19 Table 2: Feedback questions ............................................................................................................ 29 Table 3: General questions ................................................................................................................ 30 Table 4: Abbreviations of evaluation questions .............................................................................. 34 Table 5: Responsible and Sustainable Research Reflection Tool for Ocean and Waters ......... 47 Table 6: Responsible and Sustainable Research Reflection Tool .................................................. 56 Glossary Acronym Meaning AREA Anticipate, reflect, engage, act (process dimensions of RRI) (ORBIT RRI, 2023) AWI Alfred-Wegener-Institut, Helmholtz-Zentrum für Polarund Meeresforschung EPSCR The Engineering and Physical Sciences Research Council EU European Union EU Mission; Mission Ocean EU Mission ‘Restore our Ocean and Waters’ Fraunhofer IRB Fraunhofer Information Center for Regional Planning and Building Construction IPMA International Project Management Association IQR Inter quartile range NIVA Norwegian Institute for Water Research PLOCAN Plataforma Oceánica de Canarias R&I Research and innovation RRI Responsible Research and Innovation Page 7 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters RSSR UNESCO Recommendation on Science and Scientific Researchers SD Standard deviation SDGs Sustainable Development Goals SPI Sociedade Portuguesa de Inovação UCC University College Cork UN United Nations UNESCO United Nations Educational, Scientific and Cultural Organization WaveLinks WaveLinks.eu, a website that maps the research and innovation landscape of the Mission Ocean to foster collaborations WP Work package Page 8 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters 1 Summary Responsible Research and Innovation (RRI), Sustainable Development Goals (SDGs), Open Science, and Social Impact are established guidelines and pursue a similar aim. They are dedicated to aligning research, innovations and products with societal and environmental needs and values. Therefore, they are very important in every research and innovation, especially when a lot of people are affected, as in coastal restoration. It is thus necessary that we make these needs and values practically applicable for every research and innovation project. Our holistic reflection tool ‘Responsible and Sustainable Research Reflection Tool for Ocean and Waters’ supports this objective, as it unites the before mentioned concepts within a common structure and reduces complexity. It helps to understand how far projects within the EU Mission ‘Restore our Ocean and Waters’ (EU Mission; Mission Ocean) and beyond utilise their total sphere of influence to address societal and planetary needs and values. The final version of the reflection tool is presented in section Error! Reference source not found. and can be found online under: https://s.fhg.de/reflectiontooloceanwaters. Based on this reflection tool we have also developed a more general ‘Responsible and Sustainable Research Reflection Tool’, employable in all thematic areas, to broaden the tool’s usability for aligning with societal and environmental needs and values (see section 8.2). An important motivation for the two reflection tools is knowledge transfer. They holistically demonstrate the manifold possibilities of a research and innovation project to have a positive societal and ecological influence on three levels beyond its main goal: the motivational, the process and the outcome level. By reflecting on their questions, users of the tools can realise positive effects and further potential of their project, as well as areas of development and conflict. In this way, one cannot merely evaluate projects, but rather investigate them. The tools should therefore offer helpful perspectives, spark discussion and iterative improvement. This report details how we have developed the two reflection tools as part of the European Union-funded project BlueMissionAA; how they work, how we have applied and subsequently improved them based on user feedback. 2 Introduction BlueMissionAA promotes the EU Mission Ocean (European Commission, 2024a) in the Atlantic and Arctic basins by building a coordination hub. This EU Mission aims to protect and restore our oceans and waters by the means of research and innovation, citizen engagement and blue investments. It also has a pivotal role in achieving climate neutrality. With four area-based ‘Mission Lighthouses’, i.e. Atlantic-Arctic, Mediterranean Sea, Baltic-North Sea, and DanubeBlack Sea, the EU Mission encompasses places to pilot and showcase, develop and apply the Mission activities (European Commission, 2024a). BlueMissionAA’s focus is on preserving and Page 9 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters restoring marine and coastal ecosystems and biodiversity to improve climate resilience (BlueMissionAA, n.d.). 2.1 Task 5.3: Holistic RRI, SDG, Open Science and Social Impact evaluation framework Within BlueMissionAA, we developed a holistic evaluation framework that incorporates aspects of RRI, the United Nation’s (UN) SDGs, Open Science, and Social Impact throughout the entire value chain 1 . This Task 5.3 is part of Work Package (WP) 5, Citizen Engagement & Knowledge Transfer. The BlueMissionAA Grant Agreement (European Climate, Infrastructure and Environment Executive Agency (CINEA), 2022, p. 84) specifies Task 5.3 as follows: “Sustainability, responsibility and ethics are the cornerstones of all current research, development and innovation. This is particularly important in the area of coastal restoration, because so many citizens are affected. To ensure that the solutions and processes developed are compliant with current standards, frameworks are needed for an analysis and assessment in this regard. Hence, this task will compile a holistic evaluation framework to measure responsibility, sustainability, knowledge transfer, social impact and ethics […] that includes the following aspects: (a) Responsible Research and Innovation (RRI) evaluation framework for innovation processes; (b) Sustainable Development Goals (SDG) evaluation framework for systems, product and solutions; (c) Social Impact evaluation framework for the whole value chain: Input and results from WP1 (D5.8).“ To achieve this task, we researched the elements of the holistic evaluation framework, how far according evaluation frameworks already existed and how we could translate them into a useful framework or tool in the area of ocean and waters. During this process, we discussed with multiple BlueMissionAA partners including AWI, IPMA, PLOCAN, NIVA, SPI and UCC, the ways in which such a tool would be useful and well-applicable for projects within the EU Mission and beyond, to also create a lasting benefit. Taking all of this into account, we set the aim of our reflection tool, as described forth following. 2.2 Aims The overarching aims of our Responsible and Sustainable Research Reflection Tool for Ocean and Waters are: 1. to provide added value to research and innovation projects by increasing their knowledge of and helping them comply with current EU, UN and other standards for responsible and sustainable research and development, Open Science and Social Impact; 2. to support the implementation of these important goals and values at the project level; 1 “A value chain is the combination of activities a company performs to create value in a product (a good or service). The value chain covers everything from product design and manufacturing to sales and post-sales servicing” (Montevirgen, 2025). Page 16 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters was no formal definition of it. Fecher and Friesike (2013) clarified the plurality of understandings of Open Science by reviewing the literature and structuring it into five Open Science schools of thought. These can be differentiated by varying assumptions about the future of knowledge creation and dissemination and foci in relation to the phases of research and the media to establish its openness. The authors carve out: • “The infrastructure school (which is concerned with the technological architecture), • the public school (which is concerned with the accessibility of knowledge creation), • the measurement school (which is concerned with alternative impact measurement), • the democratic school (which is concerned with access to knowledge) and • the pragmatic school (which is concerned with collaborative research) (Fecher and Friesike, 2013, p. 0, emphases in original). UNESCO’s Recommendation on Open Science (UNESCO, 2021) specifies the concept’s key pillars and subaspects which we list here for a better understanding – for more detail, see UNESCO (2021): • Open scientific knowledge: • Scientific publications • Open research data • Open educational resources • Open source software and source code • Open hardware • Open Science infrastructures: • Virtual • Physical • Open engagement of societal actors: • Crowdfunding • Crowdsourcing • Scientific volunteering • Citizen and participatory science • Open dialogue with other knowledge systems: • Indigenous people • Marginalised scholars • Local communities While there is significant overlap between RRI and Open Science, their emphases differ and it is therefore useful to separately integrate Open Science into our reflection tool. While RRI focuses on normative concerns and democracy deficits (“science for and with society”, Garfinkel (2020, p. 99)), Open Science emphasises efficiency and technical optimisation (Shelley-Egan, Gjefsen and Nydal, 2020) in the research process and in the disposition of findings (Garfinkel, 2020, p. 99). With this focus, Open Science depends on a “trade-off between disclosure and secrecy” (Mukherjee and Stern, 2009, p. 449), operating under the principle "as open as possible, as closed as necessary". This means that findings and data may not be made publicly accessible if it is against the researcher’s legitimate interests, e.g. commercial exploitation of research or obligations through their funding (European Commission, 2024b). Page 17 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters 3.6 Merging the four concepts An aim of the Responsible and Sustainable Research Reflection Tool for Ocean and Waters is to integrate these four previously described concepts. Their overarching commonality is that they all address societal needs. However, the similarities could give rise to questions and confusion when evaluating a project. For instance, it might be unclear whether Social Impact should be assessed independently from the impact on the SDGs. Also, there may be uncertainty about whether an SDG assessment is necessary when RRI already encompasses ecological, economic, and social values. Herein, it is important to recognise that the distinct concepts of the reflection tool differ in terms of perspectives and levels. One objective in developing the reflection tool is to identify the concepts’ overlaps, align their aspects within a unified perspective where feasible, and otherwise, establish a structure that allows for the coexistence of differing perspectives. For instance, the SDGs address all societal fields on a global scale, while Open Science is particularly relevant in the field of research. Moreover, RRI dimensions address the process level while SDGs are concerned with projects’ impact. Thus, our task includes creating a single reflection tool that accommodates all of these diverse perspectives and levels to help research and innovation projects answer the following question: To which degree does my research and innovation project in the field of ocean and waters use its total sphere of influence to address societal and planetary needs? 4 How the reflection tool works 4.1 Reflection tool structure In this section, we explain the Responsible and Sustainable Research Reflection Tool for Ocean and Waters. It combines aspects of RRI, SDGs, Open Science, and Social Impact for the whole value chain, as mentioned in section 1. The reflection tool structure we employ is based on Stahl et al.’s (2017) RRI maturity model and its three categories: Purpose, Process, and Product. This structure is suitable because it makes the distinct evaluation levels visible, as some criteria assess the process whilst others focus on the outcomes. Here we explain the three categories, along with their respective criteria: 1. Purpose: This category refers to the underlying reasons for undertaking the research and innovation endeavour (Stahl et al., 2017, p. 4). We have included two criteria in this category, adapted from Stahl et al.‘s original framework (ibid.). The first is ‘Motivation behind your project‘, which concerns the motivation for meeting societal and planetary needs as a main motivation for a project rather than others, such as financial or career objectives. Hereby, motivation concerns the outcomes or products of a project. The second criterion is ‘Motivation for engaging with RRI’. Since our tool places emphasis on the process level, a high evaluation would signify that RRI practices are pursued out of conviction rather than out of external pressures. Page 18 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters 2. Process: This category is about all activities that are part of research and innovation projects (ibid.). At the end of section 3.3, we explain the importance of assessing the process level. The Process category encompasses the RRI dimensions Anticipation, Reflection and Engagement of the AREA framework, as they “suggest processes that should be incorporated into research and innovation endeavours” (Stahl et al., 2017, p. 5). The last dimension Action of the AREA framework, which is about taking concrete steps to implement RRI, is incorporated into the other three dimensions. The Process category also contains the RRI process-related keys Research Ethics and Governance and just one SDG that overlaps with Engagement: SDG 17, Partnerships for the Goals (Bundesumweltministerium, n.d.). Overall, we have incorporated all RRI criteria of the original framework by Stahl et al. (2017). However, in the criteria names we highlighted the connection of Engagement with SDG 17 and the connection of Anticipation with Social Impact. 3. Product: This category is about the tangible outcomes, the actual consequences and the impact of a research and innovation project (Stahl et al., 2017). Accordingly, the criteria include Social Impact, SDGs and corresponding RRI keys. Here, all the RRI keys which are not part of the process-level are assigned to: Gender Equality and Diversity, Science Education, and Open Science – which we adapted from Open Access in the original framework (Stahl et al., 2017). We merged Gender Equality and Diversity with SDG 5 (Gender Equality) and Science Education with SDG 4 (Quality Education). In this category, we have also included the SDGs 6 (Clean Water and Sanitation), 9 (Industry, Innovation and Infrastructure), 10 (Reduced Inequalities), 11 (Sustainable Cities and Communities), 12 (Responsible Consumption and Production), 13 (Climate Action), 14 (Life below Water), and 15 (Life on Land) as criteria because ocean and waters projects could have potential impacts and trade-offs concerning these SDGs. This category also emphasises Social Impact as one of the four aspects by assigning it an individual criterion. 4.2 Reflection tool questions Each criterion is assessed using a single broad evaluation question, such as "To what extent is public engagement part of your research process?" These questions for all criteria are uniformly structured using the format "To what extent...?" to ensure consistency. Answers to these questions are rated on a Likert scale ranging from 1 to 5, where a rating of 1 represents "not at all" and a rating of 5 "to a great extent." We chose the Likert scale due to its simplicity and ability to capture sufficient nuances for self-reflection. This rating scale sets the foundation for a self-reflection tool, where each level on the scale represents a different maturity level. The results of the reflection tool can be visualised, which provides a comprehensive overview of maturity levels across different criteria. This will display strengths and possible areas for improvement – as well as thematic foci of a project. In the reflection tool, we have included 19 broad questions, which we regard as a reasonable number to ensure that the tool is comprehensive enough regarding its underlying concepts without becoming overly extensive. Our goal is to maximise user-friendliness, considering that the reflection tool is designed as a voluntary questionnaire to be completed alongside daily research activities. Page 19 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters Since the evaluation questions might be difficult to answer directly, we also provide reflection questions for assistance. The reflection questions are more specific and help understanding the content and depth of each criterion. For example, a question to reflect on the extent to which Research Ethics are incorporated into a project is: “Does your data collection require informed consent from the participants?” Table 1 shows the tool’s structure, section 8 the complete reflection tool. It is important to note that reflection questions relating to the SDGs should only be understood as suggestions and will not suit all projects. In this, they differ from the remaining reflection questions, i.e. regarding RRI and others, which should be considered in all research and innovation projects. Therefore, ratings with regards to SDGs might only represent thematic foci of a project and need to be understood in relation to a project’s orientation and context. The main objective of the reflection tool is to facilitate knowledge transfer and self-reflection, rather than external evaluation. That way, the tool offers added value to projects by fostering a deeper understanding of SDGs, RRI, Social Impact and Open Science, as well as their interconnectedness; and it helps projects assess their own positions within these aspects. Comparisons with one’s own position at various project stages or – with restrictions – relative to other Mission Ocean projects also become possible, particularly through the integration of results into WaveLinks.eu (see sections 6.1 and 6.5). The tool can be useful for persons fulfilling various roles within a project, for example project leaders and managers, researchers or communicators. Of course, views can vary and this variation could again give rise to discussion and reflection about responsible and sustainable practices within a project. Table 1: Structure of the reflection tool Category Evaluation question Scale Reflection questions Purpose Process Criterion 1 Criterion 2 ... Product/Outcomes Criterion 1 Criterion 2 ... 5 How we developed the tool: methodology In summary, for developing the tool in its current version, we carried out the following steps: 1) research on the concepts and their operationalisation; 2) critical reflection on existing evaluation frameworks and assessments; 3) development of one holistic evaluation framework; 4) development of an online reflection tool based on this framework; 5) Page 20 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters improvement of the reflection tool based on feedback. In the following, findings and details regarding the first three steps are explained. Thereby, we show how our tool builds on and complements existing work. 5.1 Developing the structure We began by collecting information about RRI, SDGs, Social Impact and Open Science by researching how these concepts are defined and operationalised, especially at the project level. It became clear that RRI and SDGs are the two more comprehensive aspects of the four as shown in section 3. Therefore, we continued by researching existing evaluation-oriented tools, assessments, models and frameworks regarding SDGs and RRI. We could then incorporate the narrower concepts of Social Impact and Open Science within broader frameworks. To effectively build on existing work, we paid special attention to models that already combine RRI and SDGs and found the following: Buehrer-Topcu and Roth (2023) have developed an evaluation and monitoring tool that combines RRI with the SDGs. However, there is a lack of clarity regarding its application, evaluation, and presentation of the evaluation results. This is because the questions are not formulated in a consistent manner. We also noted that the questionnaire with 41 questions, often with multiple sub-questions, could be too extensive for the purposes of many (smaller) projects. Therefore, our reflection tool takes a different approach. It focuses on self-reflection and userfriendliness by employing fewer questions. Besides, all questions in our tool are structured in a consistent manner. This allows for an easy visual evaluation of responses, based on the 5-point Likert scale. This visual representation could provide an immediate and clear overview, enabling the project to see its foci, strengths and weaknesses – as well as comparing its selfevaluations over time. Furthermore, the RRING project, funded by Horizon 2020, also explores the relation between RRI and the SDGs. Its objective is to understand how RRI can contribute to achieving the SDGs (RRING Project, 2020). The authors propose using the UNESCO Recommendation on Science and Scientific Researchers (RSSR) “as a bridge, to link RRI and SDGs on a global scale, as it shows considerable enthusiasm globally for both the SDGs and aspects of RRI” (Rosemann and Kgwadi, 2020, p. 83). The difference between RRI and RSSR lies in their scope and backgrounds. RRI is rooted in EU policy and research, while RSSR is a global policy instrument ratified by 195 member states of UNESCO in 2017. That way, RSSR provides a structure that is globally recognised and accepted for promoting principles related to RRI (Jensen, n.d.). In our reflection tool, we do not see the necessity to work with RSSR. However, since RRI encompasses topics that overlap with RSSR (Jensen et al., 2021), we used the work of the RRING project on the connection of RSSR with the SDGs (Rosemann and Kgwadi, 2020, p. 1) to create selfreflection questions to support the evaluation questions that highlight the connection between SDGs and RRI. However, the project does not offer a concrete evaluation tool for these connections. Page 21 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters As we did not find a suitable reflection tool encompassing more than one of our four concepts, we researched frameworks addressing a single concept with the intention of combining these into a holistic tool. We found Stahl et al.’s (2017) framework the most suitable to use as a foundation for our purposes (presented in section 4), as we justify forth following. Stahl et al.’s (2017) framework was designed as an RRI maturity model, but its structure allows for the seamless integration of the SDGs, namely into the Product category. This is precisely why we opted for this framework as a foundational structure. The Product category proved to be inherently suitable for incorporating SDGs and specific aspects of Social Impact, as the RRI criteria listed in this category, such as Gender Equality and Social Justice, simultaneously represent SDGs. Besides, Stahl et al. (2017) already aspired to extend RRI with more aspects that are not yet covered by the RRI keys nor the AREA framework with criteria of “Sustainability” and “Social justice/inclusion”. These serve as ideal connecting points for adding aspects of the SDGs relating to sustainability and social justice, as well as aspects of Social Impact. Thereby, we could add criteria at the Product level to create a holistic reflection tool whilst maintaining the useful structure of the RRI maturity model. We adopted most of Stahl et al.’s (2017) criteria and their location within the categories because they are comprehensive and logically organised. Their RRI framework combines criteria of the AREA framework by the EPSCR (ORBIT RRI, 2023) as well as the six RRI keys developed by the EU Commission (cf. section 3.2) and even adds further social and environmental aspects. This laid a very fitting ground for our purpose of creating a holistic reflection tool. As Stahl et al. (2017) discuss, the position of the RRI keys could be debated, for example placing Research Ethics under the Process and not the Product category. However, we agree with the authors that the exact positioning is only of secondary importance. More relevant is the inclusion of all relevant criteria into the reflection tool and a good comprehensibility. 5.2 Adapting the structure and including missing aspects To simplify and streamline the final reflection tool, we made some adjustments. In the Purpose category, we omitted the criterion of "Ethics (intended outcomes)" (Stahl et al., 2017), as we found it to be covered by the criterion "Motivation for engaging with RRI" (ibid.), since Ethics is a RRI key. Furthermore, we changed the "Motivation for doing the research" (ibid.), to Motivation behind the Project to account for research not always being the primary focus of ocean and waters research and innovation projects. Regarding the Process category, we did not include "Responsiveness" (ibid.) as a separate criterion. We consider this aspect of RRI to be part of Governance, which encompasses “[...] hard and soft multilevel structures, guidelines and arrangements with the aim to optimize our responsivity to societal challenges, needs and concerns [...]” (RIPEET Project Consortium, 2020). Therefore, we address Responsiveness through a supporting self-reflection question within the Governance criterion. Subsequently, we incorporated the SDGs into the reflection tool. While the RRI criteria are applicable to all research and innovation projects, not all SDGs hold equal relevance in every Page 22 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters project, as indicated by the SDG Compass (GRI, UN Global Compact & WBCSD, 2015). To streamline the reflection tool and reduce complexity, we only included those SDGs relevant in the context of the EU Mission Ocean. This approach is supported by Mansell et al.'s (2020) literature review, which suggests the need to simplify the complex nature of the 17 goals and prioritise a reduced selection for assessment. On the one hand, we aimed to simplify the four concept’s many aspects for our tool, particularly concerning the SDGs. On the other hand, the tool recognises that ocean and water restoration projects bear responsibility for multiple SDGs that are not directly linked to oceans or water. For this, we have explored the literature on the connection between different SDGs and ocean and water restoration projects and identified several articles that examine the link between SDGs and the maritime industry or the blue economy (Singh et al., 2017; Said and Chuenpagdee, 2019; Lee, Noh and Khim, 2020; Diniz et al., 2023). These served as sources for orientation. Therefore, the tool goes beyond evaluating the impact on SDG 14 which is “Life below water” and defined by: “Conserve and sustainably use the oceans, seas and marine resources for sustainable development” (Project Everyone, n.d.). Extending beyond this ocean and water related SDG also acknowledges the interconnected nature of the SDGs, which can result in co-benefits and trade-offs when attempting to achieve a single goal (Singh et al., 2017). As previously mentioned, we substituted the original criterion “Sustainability” in the tool by Stahl et al. (2017) with various environmentand climate-related SDGs; namely the SDGs 6, 9, 11, 12, 13, 14 and 15. Each of these SDGs was added as an individual criterion. Then, we merged existing criteria which have parallels with SDGs. The original criterion Social Justice/Inclusion also represents SDG 10 Reduced Inequalities. The RRI key Engagement correlates with SDG 17 Partnerships for the Goals; and Gender Equality has a direct parallel with SDG 5 Gender Equality (RRing Project Consortium, 2021). Science Education aligns with SDG 4 Quality Education, so we covered it in one criterion. Even though the SDG Quality Education is broader than the RRI key, we covered them in one criterion, as the impact that research and innovation projects can have on Quality Education is mainly through Science Education. While merging, we indicated the origins of the criteria from the respective RRI keys and SDGs for clarity. This approach also aims to enhance awareness and understanding of both, SDGs and RRI principles. We have organised the criteria in a manner that reflects their conceptual origins. In the Process category, the RRI dimensions are grouped together, and the RRI keys are listed consecutively. Similarly, in the Product category, criteria that combine RRI and SDGs are grouped together first, while SDGs that are not directly linked to RRI keys follow. At this stage, we had two of the four aspects included in the tool: RRI and the SDGs. In the next step, we incorporated Open Science and Social Impact into the tool. We found that Open Science is partially covered by RRI, inter alia since Open Access is an RRI key and also a component of Open Science (cf. section 3). While Stahl et al.'s (2017) framework mentions Open Access, the ORBIT Self-Assessment (Stahl, 2017) – which is an Page 23 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters advancement of Stahl et al.'s framework – replaces Open Access with Open Science as an RRI criterion. We equally used Open Science as a singular criterion since it includes Open Access. Regarding Social Impact, certain elements are already addressed by the SDGs, such as Social Inequality or Clean Water and Sanitation, which are distinct criteria in the reflection tool. However, we identified additional elements that were not fully captured by the SDG criteria in the Product category. Consequently, we created a separate criterion for Social Impact. Open Science and Social Impact as individual criteria are accommodated between RRI and SDGs, as they are connected to both concepts. 5.3 Developing the evaluation questions As this tool enables self-reflection, we added self-evaluation questions to the criteria. The structure of the evaluation questions is mainly inspired by the SDG Impact Assessment Tool (Eriksson, n.d.) because of its focus on self-reflection and simplicity. Here, the questions follow a consistent format which is “How does your case impact SDG ...?” (ibid.) Thus, only one broad question per SDG is to be answered. The five answer choices provided for each evaluation question, namely "Direct positive", "Indirect positive", "Indirect negative", "Direct negative", and "No impact" (ibid.), allow for a clear categorisation of a project’s impact. This enables a straightforward overview of the results, i.e. the case's estimated impact on the SDGs. Additional information, being descriptions of the respective SDG and the connected targets, is provided to assist respondents in answering the broad question. In our reflection tool, we have adopted the same approach of providing one evaluation question per criterion that encourages self-reflection. These evaluation questions are also accompanied by answer choices on a 5-point Likert scale, allowing for a range of options. Besides, we also assist to answer the question with cues on what the criterion is about. This is essential to our tool as one of its objectives is the transfer of knowledge. However, in contrast to the SDG Impact Assessment Tool, we have chosen not to provide extensive descriptions of the criteria in our tool, besides questions one and two on the general concepts of the SDGs and RRI. Instead, we have opted to include reflection questions that respondents can answer for themselves. They help to get an orientation about what relevant aspects a criterion includes. Besides, they aim to enable critical analysis concerning the extent to which these aspects are incorporated in a project. These reflection questions can also serve as a foundation for conducting interviews, which is one of the possible ways to apply the tool. This approach of formulating a single, broad evaluation question per criterion with clear answering options is specifically designed to prevent the tool from becoming overly extensive. Instead of requiring respondents to answer multiple questions for each criterion, we have chosen to include reflection questions as optional, potentially helpful tools. This distinguishes our tool from The ORBIT Self-Assessment Tool (Stahl, 2017), which includes multiple evaluation questions for each RRI criterion, such as Anticipation or Governance. This might be useful to assess only one of our four concepts in a deeper manner, as it is the case in the ORBIT SelfAssessment, focusing on RRI only. However, this approach is less feasible and less user-friendly if applied to four differing concepts, which in our view should be kept simpler. Still, we have Page 24 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters found value in the multiple ORBIT Self-Assessment questions and have incorporated some of them into our tool as reflection questions. The concept of including self-reflection questions into our reflection tool to enhance critical analysis was mainly inspired by the RRI self-reflection tool (RRI Tools Consortium, n.d.) and the New HoRRIzon Thinking Tool (New HoRRIzon Project Consortium, 2019). The RRI self-reflection tool is characterised by its focus on open-ended self-reflection questions and does not provide quantitative or visual outputs. In order to combine the strengths of both approaches, we integrated the structure of the SDG Impact Assessment Tool, which employs a systematic evaluation system using a Likert scale, with the RRI tool's emphasis on thought-provoking questions. To construct the self-reflection questions, we followed a specific process. Firstly, we gathered descriptions of the criteria and questions from various sources used for evaluating those criteria. We then carefully selected the most relevant and precise questions and information that best suited our purposes. Next, we rephrased the selected input into reflection questions, ensuring consistency in style and alignment with our objectives. In some cases, we adapted existing questions, while in others, we created new ones based on the provided text. Throughout this process, we made sure to maintain the integrity of the original sources while also ensuring that the questions worked cohesively as uniform reflection questions. We give an example reflection question adopted from the RRING project (Rosemann and Kgwadi, 2020) as part of the Social justice and inclusion criterion: “Do you promote participation of communities and indigenous people in your research to establish the connection between science and society?” Participatory research is a crucial element of RRI (and RSSR), representing the Engage dimension of the AREA framework (ORBIT RRI, 2023). Social Justice corresponds to SDG 10 Reduced Inqualities. In addition to fostering critical reflection on the alignment of the project with the tool’s criteria, the reflection questions also serve the purpose of highlighting the connections between different criteria and the four concepts which is a key objective of this holistic reflection tool. For instance, Social Impact plays a significant role in the RRI dimension of Anticipation, which involves anticipating the social consequences of a project prior to its implementation. To capture this, we included the question "Do you consider the impacts of the project on human activities and employment?" among others. Furthermore, we integrated SDG 2 Zero Hunger into the Social Impact criterion through a self-reflection question, recognising the importance of possible impacts on food security. As mentioned earlier, there are also linkages between RRI and SDGs. This connection is evident in criteria such as Social justice and inclusion, where we ask: "If you conduct participatory research, e.g. by including local communities or indigenous populations, does it contribute to reduced inequalities through the distribution of the benefits of the research?" These reflection questions allow for a comprehensive assessment of the project's impacts, promoting a deeper understanding of the interplay between RRI, Open Science, Social Impact, and SDGs within the reflection tool. Page 25 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters In conclusion, our reflection tool enables research and innovation projects in the EU Mission Ocean and beyond to reflect on the four discussed concepts in relation to their project by using one holistic reflection tool with manageable effort. Additionally, the reflection tool enhances knowledge about these concepts which could support responsible and ethical research and innovation in the future. Having developed the reflection tool in the first project phase, we then offered research and innovation projects to test and use it in the second phase, as we detail in the next section. 6 How we coordinated, tested and improved the reflection tool: application In the following sections, we describe how we brought the Responsible and Sustainable Research Reflection Tool for Ocean and Waters into application: we collaborated with partners within and beyond BlueMissionAA for this (section 6.1), developed the reflection tool as an online questionnaire (section 6.2), revised (section 6.3) and tested it (section 6.4). We then discuss results (section 6.5), feedback on the reflection tool and how we incorporated it to improve the tool (section 6.6). 6.1 Collaborating with partners and enhancing utilisation To enhance the utilisation of the Responsible and Sustainable Research Reflection Tool for Ocean and Waters, several steps were undertaken prior to implementing it as an online reflection tool. Key to this process was to coordinate with BlueMissionAA Task 5.3 contributors and the whole project consortium in a dedicated meeting and at the Annual Meeting in fall 2024. These meetings were essential for presenting and discussing the reflection tool, asking for improvements before its application, brainstorming dissemination strategies and receiving considerable support in its distribution by establishing contacts with relevant projects. We also discussed how to use evaluation results and connections with other Tasks in BlueMissionAA (inter alia key performance indicator development and links with WaveLinks). In the following, we organised a further broader meeting to coordinate with BlueMIssionBANOS, PREP4BLUE and BlueMissionAA partners in terms of informing about the reflection tool, receiving feedback, looking for synergies and coordinating efforts in the evaluation of sustainability, responsibility and ethics, including potentially common (key performance) indicators. All of these meetings led to very valuable input and discussions, thereby improving our work on the reflection tool. Integrating the evaluation results from the Responsible and Sustainable Research Reflection Tool for Ocean and Waters into the platform WaveLinks.eu – a website that maps the research and innovation landscape of the Mission Ocean and groups projects according to different criteria to foster collaborations –, was identified as an important strategy for several reasons. First, the results of how research and innovation projects self-evaluate their sustainability and responsibility are highly relevant for projects themselves, stakeholders, including the EU, funders, and the public. It enables transparently improving in sustainability and responsibility. Understanding these evaluations provides valuable insights into the effectiveness and impact of various projects – and enables funders to adjust calls and programmes accordingly. Second, this integration supports the aims of BlueMissionAA by collecting and making Page 32 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters according Readiness Level in the next question, opening after your selection. What is the Technical Readiness Level (TRL) of your project? TRL 1 – Basic principles observed TRL 2 – Technology concept formulated TRL 3 – Experimental proof of concept TRL 4 – Technology validated in lab TRL 5 – Technology validated in relevant environment TRL 6 – Technology demonstrated in relevant environment TRL 7 – System prototype demonstration in operational environment TRL 8 – System complete and qualified TRL 9 – Actual system proven in operational environment What is the Business Readiness Level (BRL) of your project? BRL 1 – Basic need research BRL 2 – Needs formulation BRL 3 – Needs validation BRL 4 – Small scale stakeholders offer BRL 5 – Large scale early adopter offer BRL 6 – Proof of traction BRL 7 – Proof of satisfaction BRL 8 – Proof of scalability BRL 9 – Proof of stability What is the Societal Readiness Level (SRL) of your project? SRL 1 – Societal problem in stakeholder context SRL 2 – Proposed solution in stakeholder context SRL 3 – Stakeholder context proof of concept SRL 4 – Stakeholder context validated SRL 5 – Validated in simulated stakeholder context SRL 6 – Demonstrated in simulated stakeholder context SRL 7 – Demonstrated in operational stakeholder context SRL 8 – Final testing in real stakeholder context SRL 9 – Successful deployment in real stakeholder context What is the Impact Readiness Level (IRL) of your project? IRL 1 – Conception IRL 2 – Discovery IRL 3 – Outcome: Engagement IRL 4 – Outcome: Implementation IRL 5 – Impact: Uptake Page 33 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters IRL 6 – Impact: Sustained Change Do you agree that the results of your individual project may be publicly visible under your project's name on WaveLinks.eu? (mandatory question) If you select "No", then your self-assessment results will be shown on WaveLinks.eu only as average results of all participating projects. Yes No The online reflection tool was available for approximately three weeks, from January 22 to February 7, 2025. It remained accessible beyond this period due to a subsequent call for projects to participate, which resulted in an additional contribution. To effectively disseminate the reflection tool, we employed a multi-faceted approach that targeted various stakeholders and audiences. We presented the reflection tool in a Mission Ocean Implementation Platform monthly meeting on 5/2/2025 and called for project members of BlueMissionAA to participate. Social media platforms, particularly LinkedIn, were utilised to reach a broader audience. Additionally, we sent out emails to other Mission Ocean projects and internal organisations, complemented by reminder communications to ensure maximum engagement. The six case studies selected as part of Milestone 3.2 within BlueMissionAA were also invited to contribute insights (for the rationale for this see section 2.2). Multiple newsletters were also contacted in order to request the distribution of the online reflection tool. Several specific mailing lists and newsletters were identified that focused on ocean and water issues, as well as research and innovation. These include the ‘EU Marine and Fisheries Newsletter’, the ‘Ocean Conservancy Newsletter’, the ‘UNESCO Intergovernmental Oceanographic Commission’, the ‘International Ocean Institute’, the ‘European Marine Board’, and the ‘European Commission - Horizon Europe’. Only the European Marine Board, which covers marine research and innovation answered and distributed our online reflection tool in their newsletter. The dissemination and feedback gathering strategy was designed to raise awareness and facilitate participation among a diverse range of projects and stakeholders. This was achieved by direct outreach via emails, collaborating with partners of BlueMissionAA, using social media channels and newsletters to gather insightful information for improving the reflection tool. 6.5 Results of the reflection tool’s test In this section, we first describe the data generated by testing the reflection tool through the online questionnaire. We then analyse and discuss the results. We tested our reflection tool to: 1. Receive user feedback about the tool and improve it on this basis – results discussed in section 6.6; 2. Gain insights about how far research and innovation projects in the field of ocean and waters perceive that they use their sphere of influence to address societal and planetary needs – results are discussed in this section. Page 34 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters Before we could begin analysis of the data, we groomed the data. We did this by deleting 59 test cases or cases where none or no more than two of the evaluation questions were answered. The remaining number of respondents was n=29, on which we base the following analysis of the online reflection tool’s results. The underlying population 3 is 'research and innovation projects in the field of oceans and waters'. However, their exact number is unknown to us, also since these could be located anywhere around the world. 4 The 1,236 projects supporting the Mission Ocean (European Union, 2025) might approximate the underlying population in a narrower sense. We directly invited more than 60 Mission Ocean projects in the four basins – Atlantic & Arctic, Baltic & North Sea, Danube & Black Sea, and Mediterranean. Due to research practical reasons, the 29 participants of our online reflection tool test represent a convenience sample, i.e. users participated because we could reach them and they were ready to do so; they are not randomly sampled. Our sample is therefore not representative of all projects supporting the Mission Ocean and it is not possible to draw conclusions about or evaluate these based on our sample of n=29. However, since the main purpose for testing the reflection tool was to gain feedback for its improvement and not to evaluate Mission Ocean projects, the 29 respondents are suitable to gain valuable insights into the reflection tool from a user perspective. In total, 21 distinct projects applied the online reflection tool. Some projects participated several times through different persons, ranging from two participants in two projects to seven participants in another. In case of multiple respondents per project, mean values of their responses were calculated and used, in case project results are being displayed. For a simplified overview, Table 4 displays the tool’s 19 full evaluation questions together with their respective abbreviations, which are used in the diagrams below. The following discussion of results begins with a description of the data, considering mean values and the distribution of the 19 evaluation questions, as well as responses to the general questions. The results will be interpreted in the concluding section, with the presentation of an example for the project 'OLAMUR'. Table 4: Abbreviations of evaluation questions Abbreviations Evaluation Question Purpose 1. SDGs To what extent is addressing the Sustainable Development Goals (SDGs) a main motivation of your project? 2. RRI To what extent does your project group believe in the importance of engaging in Responsible Research and Innovation (RRI) practices? Process 3 The underlying population refers to the entire group of individuals or entities that a study aims to draw conclusions about based on the responses collected from a questionnaire. 4 Although the reflection tool was developed within an EU project and includes RRI, a concept developed in and for the EU, it may be relevant to projects around the world. Page 35 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters 3. Anticipate consequences To what extent do you anticipate potential consequences of your project that extend beyond its main aim? 4. Purpose, processes, outcomes To what extent do you reflect on the purpose, processes and outcomes of your project? 5. Public engagement To what extent is public engagement part of your research process? 6. Research ethics To what extent are research ethics incorporated into your research process? 7. Governance instruments for RRI To what extent do governance instruments exist in your project, specifically to promote Responsible Research and Innovation (RRI)? Product/Outcomes 8. Gender equality and diversity To what extent does your project promote gender equality and diversity? 9. Science education for citizens To what extent does your project promote science education for citizens? 10. Open Science To what extent does your project promote Open Science (as open as possible and as closed as necessary)? 11. Improve people's lives To what extent does your project improve people's lives? 12. Social justice To what extent does your project promote social justice? 13. Clean water and sanitation To what extent does your project promote clean water and sanitation (SDG 6)? 14. Industry, innovation and infrastructure To what extent does your project support industry, innovation and infrastructure (SDG 9)? 15. Sustainable cities and communities To what extent does your project contribute to sustainable cities and communities (SDG 11)? 16. Responsible production and consumption To what extent does your project promote responsible production and consumption (SDG 12)? 17. Climate action To what extent does your project promote climate action (SDG 13)? 18. Life below water To what extent does your project promote life below water (SDG 14)? 19. Life on land To what extent does your project promote life on land (SDG 15)? Mean values (average scores derived from the Likert scale responses) for the 19 evaluation questions across the 29 participants are shown in Figure 3. The mean value across the 19 questions was 3.80. The mean value for the category Purpose was 4.39 (Standard deviation, SD=0.78), for Process 3.9 (SD=0.92) and for Product/Outcomes it was the lowest with 3.66 (SD=1.01). The range laid between 2.76 (Life of land) and 4.72 (Life below water). Page 36 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters Figure 4 shows the distribution of answers to the 19 evaluation questions. Mean values are represented by crosses inside the boxes, with the boxes themselves indicating the interquartile range (IQR) of the given answers. This range extends from the first quartile (the lowest 25% of answers) to the third quartile (the highest 75% of answers). The boxes thus represent the middle 50% of the data. The ‘whiskers’ of the box plot extend from the edges of the box (the first and third quartiles) to the smallest and largest values that are not considered outliers. Specifically, they reach out to the furthest data points that are within 1.5 times the interquartile range (IQR) from the quartiles. Any data points falling outside this range (beyond 1. 5 times the IQR) are designated as outliers and are represented as individual dots on the plot. Notable outliers exist in questions one, two, five, twelve, and 18. Many of these questions have relatively high mean values (apart from question 12 – Social justice), indicating unique responses where projects deviate from the trend of high awareness regarding evaluation criteria. However, they may also indicate a particular thematic focus, specifically regarding the SDGs, for example on life below water. It is important to note that this does not permit a 'positive' or 'negative' evaluation. However, the SDGs may offer potential avenues for identifying relevant topics. Figure 3: Mean values across the 19 evaluation questions Page 37 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters Regarding general questions (see Table 3), the distribution across Mission Lighthouse areas shows that the majority of participants contributed to projects in the Baltic and North Sea Lighthouse (n=12), followed by the Atlantic and Arctic (n=10), the Mediterranean (n=6), and lastly the Danube and Black Sea Lighthouse (n=4). The distribution across geographical areas was notably heterogeneous, with most participants’ projects operating in the North Sea (n=12), across the Atlantic (n=29) and Baltic Sea (n=10). Multiple entries were possible for these questions; thus, the number of responses exceeds the number of participants. Participants’ projects contributed similarly to the three Mission Ocean Objectives, with Objective three (“Make the sustainable blue economy carbon-neutral and circular”) receiving the highest number of answers (n=16), while both Objective one (“Protect and restore marine and freshwater ecosystems and biodiversity”) and Objective two (“Prevent and eliminate pollution of our ocean, seas, and waters”) received n=15 answers each. Participants’ projects spanned all thematic areas (see Table 3), ranging from n=6 in ‘Nature Accounting and Finance’ to n=17 in ‘Community Engagement’. No thematic area was left unrepresented, and some respondents provided open-ended answers, including topics such as ‘Marine Multi-Use’, ‘Women in the Blue Economy’ and ‘Food Security, Sustainable Food Production’. Project budget data given by participants (n=10 did not provide an answer) ranged from €0 to €10,000,000, with a mean budget of €4,599,243.69. When choosing between Technical, Business, Societal and Impact Readiness Levels to best describe their project, most participants identified primarily with the 'Technical Readiness Level' dimension (n=13). Within this, most rated themselves at 'TRL 7 - Demonstration of a system prototype in an operational environment' (n=6). For the 'Societal Readiness Level' (n=5), responses were concentrated at 'SRL 6 - Demonstrated in simulated stakeholder context' (n=2) and 'SRL 7 - Demonstrated in operational stakeholder context' (n=2). Only n=3 participants placed their projects within the 'Impact Readiness Level', with most assessing their phase as 'IRL 4 - Outcome: Implementation'. Figure 4. Box plot across the 19 evaluation questions 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11.12. 13. 14. 15. 16. 17.18. 19. Page 38 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters Notably, no participant selected the 'Business Readiness Level' as the best describing dimension. Interpreting the data reveals several key insights regarding participant engagement across various themes and objectives of the reflection tool. To understand the nature of participants’ projects, we first consider their general characteristics. The geographical distribution of projects shows a concentration in the Atlantic, the Baltic and North Sea, which points to according regional foci in project implementation. The balanced emphasis on the three Mission Ocean Objectives indicates a good representation of Mission Ocean projects across its three objectives for the reflection tool’s test, making its results relevant across the Mission Ocean. This is complemented by the relatively homogenous distribution of answers across thematic areas, showcasing broad engagement across multiple topics. Mean budget allocation suggests a significant investment in these projects, although there are substantial variability and missing values in budgets. Additionally, the predominance of responses about ‘Technical Readiness Levels’ highlights a strong emphasis on technological development at relatively high readiness levels among participants. On the other hand, the lack of responses about ‘Business Readiness Levels’ may indicate a potential gap in business-oriented projects, suggesting an area for further exploration and development in future initiatives. Second, we interpret content-related results of the reflection tool. The overall mean value of the 19 evaluation questions (see Figure 3) was 3.80, thereby lying between “to a moderate extent” (3) and “to a large extent” (4), but tending towards the latter. Deviations from mean values across the questions are moderate, with standard deviations of the 19 questions ranging from 0.53 to 1.38. This is an overall positive result, displaying a medium to high engagement of participants’ ocean and waters research and innovation projects with regards to responsibility and sustainability. Particularly relevant are the first two questions regarding the SDGs and RRI. They both show relatively high mean values of 4.33 (SDGs) and 4.45 (RRI), reflecting a strong overall understanding of and emphasis on these two important concepts among participants. Notably, the topics ‘Social justice’ and ‘Life on land’ exhibit the lowest mean values, indicating a lesser focus on these areas compared to others. In contrast, there is a marked emphasis on ‘Life under water’ and ‘Open Science,’ suggesting that participants prioritise marine and scientific initiatives over terrestrial social issues, which often is the core of projects working on ocean and water topics. These trends could therefore display mere thematic foci, particularly with regards to single SDGs (questions eleven; 13-19), for example little focus on ‘Life on land’. However, they could also be indicative of blind spots, for example with regards to ‘Social justice’, which should be a cross-cutting issue – but again might not appear as a theme in, for instance, a technical under-water restoration project. These examples show that interpretations of evaluation values should consider a project’s aim and focus and the significance of mere values should not be overstated. Even the comparison with other projects might not be meaningful in all evaluation criteria, in case they treat diverging topics, have differing project budgets and are in different project stages. However, comparisons between projects and with means across participants become possible on WaveLinks – if projects agreed to be represented on the Website by their name. Altogether, the most significant comparison may be a project’s self-reflection at various project phases, so as to display a potential development; and the most knowledgeable interpreters of results will probably be the projects themselves. Page 39 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters Lastly, as an exemplary visualisation and comparison of a project with the mean, Figure 5 presents a spiderweb diagram that compares the average responses participants gave for the project ‘OLAMUR’ with the overall mean values from all participants. We only display those answers for ‘OLAMUR’ in their mean value where participants agreed that their responses may be publicly represented. This visualisation allows for a comparison of how a single project selfevaluates against the collective responses from all 29 participants. OLAMUR (Offshore Low-trophic Aquaculture in Multi-Use offshore wind farms: an EU approach to sustainably harnessing Europe's oceans), funded under the EU HORIZON Framework Programme and contributing to the EU Mission Ocean, is an innovation project that explores how to integrate sustainable aquaculture – such as mussel and seaweed farming – within offshore wind farms. Its goal is to promote multi-use of ocean space by combining renewable energy production with environmentally friendly food production (OLAMUR, 2023). The diagram reveals that in many instances, ‘OLAMUR’ rates itself similarly to other projects, as seen in questions five, six, twelve, and 15. However, there are notable disparities; for example, from questions three, four, 14, 16, 17, the project’s ratings were higher than the overall mean, showing a higher perceived engagement or strengths in these areas. Conversely, questions eight and nine highlight areas where the project’s self-evaluation was lower than that of other projects, indicating potential opportunities for improvement. These overviews which the reflection tool facilitates, particularly if filled in by several members of a project, can provide orientations about one’s project, about current foci and strengths, as well as about areas for improvement. Objectives can be set with regards to the 19 criteria and evaluated at various project stages. Additionally, responses of project members could be compared, so as to find out how far self-perceptions are shared or various views about how the project addresses Figure 5: Spiderweb diagram comparing average self-evaluations of the project ‚OLAMUR‘ with means across all 29 participants Page 40 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters societal and planetary needs exist. Diverging views could again give rise to discussions about how a project could (better) tackle these vital needs. 6.6 Improving the reflection tool based on user-feedback The feedback questions added to the online reflection tool (see Table 2) provided very valuable findings. Participants responded to the question “To what extent did the assessment provide you with valuable insights?” with an average rating of 3.19 (SD=1.04), that is between “to a moderate extent” (3) and “to a large extent” (4). The clarity of the questions (“To what extent were the questions understandable?”) was rated notably higher, receiving a score of 4.38 (SD=0.64). Overall, the reflection tool was deemed helpful (“To what extent was the assessment helpful?”) with a score of 3.19 (SD=1.18). Its potential for iterative use – as the reflection tool was conceived – (“To what extent would you use the assessment as a tool for iterative reflection and improvements?”), was assessed with a score of 3.23 (SD=0.99). Altogether, most of these average scores are moderately positive. However, they are not very positive with five to seven low ratings of 1 (“not at all”) and 2 (“to a small extent”) for the first and the two latter discussed questions. Besides some very positive ratings, some participants did not see the reflection tool as insightful nor useful. We aim to understand reasons behind positive and negative ratings, reflect these critically and improve the reflection tool accordingly. The focus will then be to provide an added value to those who benefit from the reflection tool. The open questions in the reflection tool allowed participants to share their experiences and suggestions, revealing a varied feedback. Many respondents acknowledged the benefits they derived from the reflection tool, which are for example new knowledge, its application and reflection with regard to one’s work. This is expressed by comments such as: “Well, this assessment has awoken my consciousness to new knowledge regarding efficient project execution. […] I hope to apply my newly acquired skills in my project, "[…]".”; “I think it gave me a good opportunity to reflect on the work we are doing through the project and the areas we need to work on.”; “This assessment made me reflect on work-related issues that I had not questioned before.”; “It is comprehensive and connects the project with the SDGs, which helps again to better understand why the project is done? It shows why it is important to achieve our goals.” However, some felt that the reflection tool was less useful for existing projects, noting for instance that “[t]he benefit from the assessment is towards future investment. This reflective view is not particularly useful for existing projects.” On a positive note, participants expressed what they appreciated about the reflection tool, highlighting its comprehensive nature and connection to the SDGs. As one respondent stated, “[t]he survey is well done, in my opinion. It covers questions on all the key pillars of RRI and complements it with the relationship with the SDGs, which were not define[d] when RRI was coined.” Others noted its straightforwardness and ability to provoke thought: “Makes you think. You can use it at different stages of a same project/work.” Respondents also liked the userfriendliness and conciseness of the reflection tool, highlighting for example: “[t]he clarity of language and responsive pages”; “Good questions, focusing more on the 'social science', easy to understand and not too long.”; “Relatively quick. Well layed out and str[uc]tured.” Page 41 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters In terms of improvement, participants provided constructive feedback, suggesting enhancements like clarifying that it is a self-reflection tool rather than a test. Another participant suggested adding specificity through a drop-down menu for questions that might not apply to all projects. This was a concern shared by some respondents, for example: “It is always [that] some questions are more relevant than others. Projects are different and should be. Maybe reflect that a little better?; “And all questions do[…] not apply to all projects.” One participant wished: “to go more in depth with the questions – to make the respondent be more specific in the response”. Another user stated: “[…]it might be good to write down, even for one (not sharing it), what are you going to have in your mind to answer the following questions...” A final feedback was from a person who already worked for three years in the implementation of RRI: “I am therefore very familiar with RRI and so I did not encounter many new things in your survey.” This latter point combined with the feedback above gives hints towards the suitable audience of the reflection tool: persons who do not (yet) have in-depth knowledge and experience with SDGs, RRI, Open Science and Social Impact or who seek to apply and/or refresh it in relation to a certain project. In response to the helpful feedback received, we implemented several important changes to enhance the self-reflection tool. Firstly, the name was changed from "Responsible and Sustainable Research Assessment for Ocean and Waters" to "Responsible and Sustainable Research Reflection Tool for Ocean and Waters", to better convey the reflective nature of the tool, addressing concerns that it was perceived as a test. We also highlighted this in the introduction text to the online reflection tool. It is important to note that feedback and general questions still referred to the tool as "Assessment," as the name was changed after its testing phase. By framing it as a "Research Reflection Tool," we aim to foster a culture of accountability among researchers and innovators, encouraging the consideration of societal and environmental implications of one’s work in ocean and water contexts. Second, we made adjustments to clarify the applicability of the reflection tool and of its evaluation questions. In the online reflection tool’s introduction text, we added: “We recommend users to focus on one specific project they are working on or planning when answering the questions. However, not all questions might apply to every project. We encourage users to focus on aspects which are useful to their work.” However, we decided against going more into depth with questions to keep the accessible and user-friendly nature of the reflection tool – which several participants perceived as a strength. Finally, to make the reflection tool relevant to a much broader audience, we decided to develop another more general version of the reflection tool through changing and adding some questions. This more general ‘Responsible and Sustainable Research Reflection Tool’ (see Table 6 in section 8.2) addresses all research and innovation projects across thematic areas, not only those dealing with ocean and waters. The necessary tweaks for this laid in the last questions, tackling specific SDGs applying particularly to ocean and waters. This broader applicability makes the reflection tool much more relevant – significantly exceeding Mission Ocean projects – and enables its wider use across all research and innovation projects. It also enhances the legacy of BlueMissionAA and its Task 5.3. Altogether, these adjustments aim to create a wider applicability of the reflection tools and a more user-friendly and meaningful experience for participants, encouraging the engagement with responsible and sustainable research practices in relation to one’s project. Page 48 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters guiding principles that help researchers and innovators navigate the complexities of their work, that is the “what” of RRI.10 Furthermore, there are four RRI dimensions to provide practical approaches for implementing these principles, addressing the "how" of RRI through the AREA Framework. AREA stands for Anticipation, Reflection, Engagement, and Action which further complements RRI by guiding researchers on how to proactively consider the different keys. Together, these elements create a comprehensive strategy for fostering RRI in research and innovation.10 • Is there an awareness of RRI within your project? • Do you have a definition of RRI and/or components of RRI integrated into your research process? • Do you proactively seek to integrate RRI into your research process? • Do you believe that most of the components of RRI are vital for the success of your project? Process12 Anticipation (RRI, Social Impact & SDG 2 Zero Hunger) To what extent do you anticipate potential consequences of your project that extend beyond its main aim?11 1 (no extent) - 5 (great extent) • Do you consider positive and negative consequences of your project?11 Do you have mechanisms in place to include a broad and diverse range of stakeholders when considering the consequences of your research?11 • Do you consider the consequences of the project on different cultures?11 • Do you consider the impacts of the project on human activities and employment? • Do you consider the impacts of your project on food security (SDG 2)? • Do you consider whether your project will influence people’s behaviour, attitude, emotions, thoughts, moods, and actions?11 • Do you consider the social impact for the whole life cycle of your product (production, use, disposal) and all procedures for every step of your project (planning, process, outcome, impact)? • Do you consider all consequences that the project may have on the environment? Reflection (RRI) To what extent do you reflect on the purpose, processes and 1 (no extent) - 5 (great extent) • Do you question your axioms and basic assumptions to ensure the research is aligned with societal needs?10 Page 49 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters outcomes of your project? • Do you include reflexivity in your work? (Reflexivity is the practice of reflecting on the processes and assumptions guiding your research. It involves questioning biases either collectively or alone to ensure alignment with societal needs.10) • Does your project include mechanisms for reflection, e.g. internal workshops, sociotechnical integration, backcasting/hindsight analysis, a project advisory board, stage-gating processes, review and quality assurance steps?10, 11 • Do you consider the potential of your work to raise ethical, social or political controversy?11 • Do you think about possible alternate uses and abuses of your work?11 Engagement (RRI) & SDG 17 (Partnerships for the Goals) To what extent is public engagement part of your research process? 1 (no extent) - 5 (great extent) • Do you use participative research methods? • How might the project benefit from involving stakeholders?11 • Do you use an adequate methodology to identify stakeholders?4 • Have relevant stakeholders been involved in defining the research problem?11 • How will you ensure that all stakeholders have the information they need to engage in a meaningful dialogue?4 • Do you ensure that all stakeholders feel adequately supported to voice their opinion?4 • How will you ensure that you maintain good relations with your stakeholders?4 • Have the results been discussed with different types of stakeholders to allow for alternative interpretations?4 Ethics (RRI) To what extent are research ethics incorporated into your research process? 1 (no extent) - 5 (great extent) • Does a comprehensive ethics approval exist for your work?11 • Does your project involve any risks of breach of confidentiality and what might they be?4 • Does your data collection require informed consent from the participants?4 • Is the planned research methodology ethically acceptable, including aspects related to data collection, data management and data storage?4 • Do you hold critical peer review and internal discussion on research integrity throughout the process?7 • Do you consult an external research ethics expert or ethics committee?7 Page 50 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters • Do you share the responsibility for the impact of your research among all actors involved (researchers, policy makers, research institutions, universities, industry, civil society organisations, etc.)?7 Governance (RRI) To what extent do governance instruments exist in your project to promote Responsible Research and Innovation (RRI)? 1 (no extent) - 5 (great extent) • Do you have transparent internal procedures?10 • Are you encouraging future-oriented governance?10 • Are you investing resources to make your research more responsive to societal needs and concerns?7 • Do you ensure that your research practices can adapt to unforeseen results or societal changes (e.g. agile project management, recalibrating the vision of the project)?1, 7 • What RRI-related training opportunities do you have (regarding gender equality, ethics, Open Science, research integrity, research management, data management)? 1, 7 Product/Outcomes12 Gender Equality and Diversity (RRI and SDG 5 Gender Equality) To what extent does your project promote gender equality and diversity? 1 (no extent) - 5 (great extent) • Do you consider sex and gender in your chosen topics, methodology and data?7 • Do you have a plan to promote Equality, Diversity and Inclusion (EDI) in your project (including gender-balanced teams and management positions, equal salary guarantees and contract conditions, support of diverse working approaches, avoiding gender stereotypes, gender sensibility in publications, evaluation of gender equality)?11 • Do you consider EDI when evaluating the impact of your project?11 • Do you have a procedure in place for raising concerns regarding EDI?11 Science Education (RRI & SDG 4 Quality Education) To what extent does your project promote science education for citizens? 1 (no extent) - 5 (great extent) • Do you think some of your research outputs should be accessible to the general public?11 • Do you equip citizens with the necessary knowledge to engage in debates around research and innovation?11 • Do you actively create and exploit science education opportunities?11 Page 51 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters • Do you help people to acquire the knowledge and skills needed to promote sustainable development (including, amongst others, the criteria of this reflection tool)?8 Open Science To what extent does your project promote Open Science (as open as possible and as closed as necessary)?5 1 (no extent) - 5 (great extent) • Do you provide free and early access to your research, its data and findings in order to improve its quality and facilitate fast innovation?7 • Do you adhere to the FAIR Principles (guidelines to improve the Findability, Accessibility, Interoperability, and Reuse of digital assets, emphasising machineactionability; cf. for example: https://www.go-fair.org/fair-principles/)?5 • Is the open access information accompanied by a transparent documentation of the data structure, data editing, statistical procedures and analytical decisions made throughout the project (allowing for replication)?4 • Do you have communication channels to allow for a co-production of knowledge across different sectors and cultures?6 • Do you promote Open Science to benefit the Sustainable Development Goals (SDGs)?6 • Are you able to balance commercial interest with the need for open access?11 Social Impact (SDG 3 Good Health and WellBeing) To what extent does your project improve people's lives?3 1 (no extent) - 5 (great extent) • Does your project at any stage (planning, process, outcome, consequences) ... ... improve how people live or work together or how they interact with each other?3 ... promote culture? 3 ... improve people's living environment?3 ... improve people's economic situation?3 ... improve people's health and wellbeing?3 (SDG 3: Good Health and Well-Being) Social justice and inclusion (SDG 1 No Poverty, SDG 10 Reduced Inequalities, SDG 16 Peace, Justice To what extent does your project promote social justice? 1 (no extent) - 5 (great extent) • Do you consider social justice as part of your research focus, design and practices? • Does your research address social inequalities (e.g. in terms of economic resources, opportunities, education, health)? • Do you promote no poverty (SDG 1)? • Does your research impact social inequalities (e.g. affect particularly poorer citizens, target poorer or richer citizens more in aiming for behavioural change)? Page 52 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters and strong Institutions) • Do you promote participation of communities and indigenous people in your research to establish the connection between science and society?6 • Do you promote peace, justice and strong institutions (SDG 16)?8 • If you conduct participatory research, e.g. by including local communities or indigenous populations, does it contribute to reduced inequalities through the distribution of the benefits of the research?6 • Do you ensure that the resulting knowledge from this exchange is appropriately credited?6 • Do you have collaborations with other developed and developing countries to help bridging the knowledge gap between rich and poor countries?6 • How could your Open Science policies influence social equality, for example through access to knowledge? Clean Water and Sanitation (SDG 6) To what extent does your project promote clean water and sanitation (SDG 6)?8 1 (no extent) - 5 (great extent) • Do you contribute to good access of safe and affordable drinking water?8 • Does your project protect and restore water-related ecosystems?8 • Does your project increase water-use efficiency and ensure freshwater supplies?8 • Does your project improve water quality, wastewater treatment and safe reuse?8 • Do you support local engagement in water and sanitation management?8 • Do you implement integrated water resources management?11 Industry, Innovation and Infrastructure (SDG 9) To what extent does your project support industry, innovation and infrastructure (SDG 9)? 1 (no extent) - 5 (great extent) • Do you develop sustainable, resilient and inclusive infrastructures?8 • Do you promote inclusive and sustainable industrialization?8 • Do you upgrade industries and infrastructures for sustainability?8 • Do you enhance research and upgrade industrial technologies?8 • Do you support domestic technology development and industrial diversification?8 • Do you facilitate sustainable infrastructure development for developing countries?11 Sustainable Cities and Communities (SDG 11) To what extent does your project contribute to sustainable cities 1 (no extent) - 5 (great extent) • Do you protect the world’s cultural and natural heritage?8 • Do you foster inclusive and sustainable urbanization?8 • Do you promote decent work and economic growth (SDG 8)?8 • Does your project try to reduce the adverse effects of natural disasters?8 • Does your project reduce the environmental impact of cities?8 Page 53 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters and communities (SDG 11)? • Do you provide access to safe and inclusive green and public spaces?11 • Do you implement policies for inclusion, resource efficiency or disaster risk reduction?11 Responsible Production and Consumption (SDG 12) To what extent does your project promote responsible production and consumption (SDG 12)? 1 (no extent) - 5 (great extent) • Do you use natural resources in a sustainable way?8 • Do you have a responsible management of chemicals and waste?8 • Do you reduce waste generation?8 • Do you promote a universal understanding of sustainable lifestyles?8 • Do you support developing countries' scientific and technological capacity for sustainable production and consumption?8 • Do you evaluate the environmental impact of your research (including facilities, travels etc.)?11 Climate Action (SDG 13) To what extent does your project promote climate action (SDG 13)? 1 (no extent) - 5 (great extent) • Does your project strengthen resilience and adaptive capacity to climate related disasters?8 • Does your project build knowledge and capacity to meet climate change?8 • Does your project promote raising capacity for effective climate change-related planning and management?8 • Does your project integrate climate change measures into policies and planning?8 • Does your project support affordable and clean energy (SDG 7)?8 Life below Water (SDG 14) To what extent does your project promote life below water (SDG 14)? 1 (no extent) - 5 (great extent) • Does your project reduce marine pollution?8 • Does your project conserve, protect or restore marine or coastal ecosystems?8 • Does your project reduce ocean acidification?8 • Does your project support sustainable fishing or support small scale fishers?8 • Does your project increase scientific knowledge, research and technology for ocean health?8 • Does your project implement and enforce international sea law?8 • Does your project increase the economic benefits from a sustainable use of marine resources?8 Life on Land (SDG 15) To what extent does your project 1 (no extent) - • Does your project conserve and restore terrestrial and freshwater ecosystems? • Does your project end desertification and restore degraded land?8 Page 54 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters promote life on land (SDG 15)? 5 (great extent) • Does your project prevent invasive alien species on land and in water ecosystems?8 • Does your project eliminate poaching and trafficking of protected species?8 • Does your project protect biodiversity and natural habitats?8 Page 55 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters 8.2 Responsible and Sustainable Research Reflection Tool Table 5 shows a more general version of the Responsible and Sustainable Research Reflection Tool we developed, usable across all thematic areas. The rationale for its development is explained in section 6.6 and 7.2. The questions highlighted in blue were added or modified, compared to the Responsible and Sustainable Research Reflection Tool for Ocean and Waters in the previous section; if the entire line is blue, then the whole question was added, while if only a part is blue, that specific part was modified. For instance, the evaluation question about ‘Science Education’ was adjusted to be more generally applicable. Additionally, the reflection questions in the ‘Social Justice and Inclusion’ category were shortened, and further questions were developed to specifically address SDGs 1 (No Poverty), 10 (Reduced Inequalities), and 16 (Peace, Justice and strong Institutions). Sources we drew on to formulate the additional evaluation and reflection questions are listed below. To keep the reflection tool in a well readable format, we marked the source for each question with a superscript – corresponding to the superscripts of the following references. Questions without a superscript are developed by our team. Sources for the reflection tool’s questions are: • 1 AREA 4P framework (ORBIT RRI, 2023) • 2 FAIR Principles (GO FAIR Initiative, n.d.) • 3 International Principles For Social Impact Assessment (Vanclay, 2003) • 4 New HoRRIzon Societal Readiness Thinking Tool (New HoRRIzon Project Consortium, 2019) • 5 Road 2 Openness. Open Science Self Assessment (Stifterverband für die Deutsche Wissenschaft e.V., no date) • 6 Rosemann and Kgwadi (2020) • 7 RRI Self-Reflection Tool (RRI Tools Consortium, n.d.) • 8 SDG Impact Assessment Tool (Wexsus and SDSN Northern Europe, n.d.) • 9 SEEbalance® (Schmidt et al., 2004) • 10 The ORBIT Keys of RRI (ORBIT, 2023) • 11 The ORBIT Self-Assessment Tool (ORBIT, 2023) • 12 RRI Maturity model (Stahl, 2017) • 13 RRI-Leaders (n.d.) • 14 RRING Project. (n.d.) Page 56 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters Table 6: Responsible and Sustainable Research Reflection Tool The questions highlighted in blue were added or modified, compared to the Responsible and Sustainable Research Reflection Tool for Ocean and Waters in the previous section; if the entire line is blue, then the whole question was added, while if only a part is blue, that specific part was modified. Category & Criterion Evaluation question Scale Reflection questions Purpose12 Motivation behind your project To what extent is addressing the Sustainable Development Goals (SDGs) a main motivation of your project?11 1 (no extent) – 5 (great extent) The Sustainable Development Goals (SDGs) are a set of 17 global goals established by the United Nations in 2015. They aim to address various global challenges, including poverty, inequality, climate change, environmental degradation, peace, and justice as seen in Fig.1. The goals are designed to be achieved by 2030 and are interconnected, meaning progress in one goal can impact others. For example, climate action (Goal 13) directly affects life below water (Goal 14) and life on land (Goal 15), so by achieving one goal, others might be affected as well. • What is the aim of your project? • Does your project aim for positive social impacts? • Does your project aim for positive ecological impacts? • Are there other motivations for your project that compete with the objective of addressing societal and planetary needs (e.g. financial, reputational,....)?12 Motivation for engaging with Responsible Research and Innovation (RRI) To what extent does your project group believe in the importance of engaging in Responsible Research and Innovation (RRI) practices? 1 (no extent) – 5 (great extent) The European Commission defines Responsible Research and Innovation (RRI) as a way of doing research and innovation that proactively considers the possible effects and expectations of society14. The aim of RRI is to ensure that research and innovations are designed to be inclusive, meaning they involve various voices and perspectives, and sustainable, meaning they focus on long-lasting benefits for both society and the environment. RRI promotes inclusivity by engaging diverse stakeholders, including the public, policymakers, and researchers, in the research process. This engagement helps to align research outcomes with societal needs and values, ensuring trust and acceptance among the public.13 Page 57 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters Moreover, RRI emphasizes sustainability, encouraging practices that consider long-term impacts on society and the environment. This includes addressing issues such as climate change, social justice and access to the benefits of research and innovation.12 To support RRI, the European Commission has identified six keys: Ethics, Science Education, Gender Equality, Open Access, Governance, and Public Engagement. These keys serve as guiding principles that help researchers and innovators navigate the complexities of their work, that is the “what” of RRI.10 Furthermore, there are four RRI dimensions to provide practical approaches for implementing these principles, addressing the "how" of RRI through the AREA Framework. AREA stands for Anticipation, Reflection, Engagement, and Action which further complements RRI by guiding researchers on how to proactively consider the different keys. Together, these elements create a comprehensive strategy for fostering RRI in research and innovation.10 • Is there an awareness of RRI within your project? • Do you have a definition of RRI and/or components of RRI integrated into your research process? • Do you proactively seek to integrate RRI into your research process? • Do you believe that most of the components of RRI are vital for the success of your project? Process12 Anticipation (RRI, Social Impact & SDG 2 Zero Hunger) To what extent do you anticipate potential consequences of your project that extend beyond its main aim?11 1 (no extent) - 5 (great extent) • Do you consider positive and negative consequences of your project? 11 • Do you have mechanisms in place to include a broad and diverse range of stakeholders when considering the consequences of your research?11 • Do you consider the consequences of the project on different cultures?11 • Do you consider the impacts of the project on human activities and employment? • Do you consider whether your project will influence people’s behaviour, attitude, emotions, thoughts, moods, and actions?11 • Do you consider the social impact for the whole life cycle of your product (production, use, disposal) and all procedures for every step of your project (planning, process, outcome, impact)? • Do you consider all consequences that the project may have on the environment? Page 64 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters Life below Water (SDG 14) To what extent does your project promote life below water (SDG 14)? 1 (no extent) - 5 (great extent) • Does your project reduce marine pollution?8 • Does your project conserve, protect or restore marine or coastal ecosystems?8 • Does your project reduce ocean acidification?8 • Does your project support sustainable fishing or support small scale fishers?8 • Does your project increase scientific knowledge, research and technology for ocean health?8 • Does your project implement and enforce international sea law?8 • Does your project increase the economic benefits from a sustainable use of marine resources?8 Life on Land (SDG 15) To what extent does your project promote life on land (SDG 15)? 1 (no extent) - 5 (great extent) • Does your project conserve and restore terrestrial and freshwater ecosystems? • Does your project end desertification and restore degraded land?8 • Does your project prevent invasive alien species on land and in water ecosystems?8 • Does your project eliminate poaching and trafficking of protected species?8 • Does your project protect biodiversity and natural habitats?8 Peace, justice and strong institutions (SDG 16) To what extent does your project support peace, justice, and strong institutions (SDG 16)? 1 (no extent) - 5 (great extent • Does your project actively contribute to promoting peace and preventing conflict within the community or at a national level?8 • Does your project enhance access to justice for all individuals, particularly for marginalized or disadvantaged groups?8 • Does your project work to strengthen local or national institutions to ensure they are effective, accountable, and inclusive?8 • Does your project encourage community participation in decision-making processes to foster a culture of transparency and accountability?8 • Have you identified potential negative impacts on peace and justice in your project?8 Page 65 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters 9 References Azure OpenAI (2024) FhGenie – AI for internal use at Fraunhofer-Gesellschaft. Available at: https://fhgenie.fraunhofer.de/. Azure OpenAI (2025) FhGenie – AI for internal use at Fraunhofer-Gesellschaft. Available at: https://fhgenie.fraunhofer.de/. BioProtect Consortium (2024) Home, BioProtect. Available at: https://bioprotect-project.eu/ (Accessed: 24 April 2025). BLUE CONNECT Consortium (2025) BLUE CONNECT Project – Strict protection, restoration and comanagement of Marine Protected Areas, BLUE CONNECT. Available at: https://blueconnectproject.eu/ (Accessed: 24 April 2025). BlueMissionAA (2025) Weekly Hours – Blue Mission Atlantic & Arctic. Available at: https://bluemissionaa.eu/weekly-hours/ (Accessed: 24 March 2025). BlueMissionAA (n.d.) ‘European Mission – BlueMissionAA’. Available at: https://bluemissionaa.eu/wps/ (Accessed: 21 March 2024). Boise State University (2024) ‘Grand Challenges. Contributing to solving the most pressing issues of our time’, Research and Economic Development. Available at: https://www.boisestate.edu/research/grand-challenges/ (Accessed: 12 April 2024). Buehrer-Topcu, S. and Roth, F. (2023) ‘Development of a responsible monitoring and evaluation tool for the assessment of research projects’, 27th International Conference on Science, Technology and Innovation Indicators (STI 2023) [Preprint]. Available at: https://doi.org/10.55835/6440fdcc50b7281abb62c847. Bundesumweltministerium (n.d.) 17 Nachhaltigkeitsziele – SDGs, Bundesministerium für Umwelt, Naturschutz, nukleare Sicherheit und Verbraucherschutz. Available at: https://www.bmuv.de/WS5613 (Accessed: 19 October 2023). Caiado, R.G.G. et al. (2018) ‘A literature-based review on potentials and constraints in the implementation of the sustainable development goals’, Journal of Cleaner Production, 198, pp. 1276–1288. Available at: https://doi.org/10.1016/j.jclepro.2018.07.102. Diniz, N. et al. (2023) ‘Disclosure of the sustainable development goals in the maritime industry and port sector’, Revista de Gestão e Secretariado (Management and Administrative Professional Review), 14, pp. 8129–8149. Available at: https://doi.org/10.7769/gesec.v14i5.2185. EFFECTIVE Consortium (2025) Home - EFFECTIVE, EFFECTIVE. Available at: https://effectiveeuproject.eu/ (Accessed: 24 April 2025). Eriksson, M. (n.d.) ‘SDG Impact Assessment Tool - Guide 1.0’, Sustainable Development [Preprint]. European Climate, Infrastructure and Environment Executive Agency (CINEA) (2022) ‘Grant Agreement. Project 101093962 - BlueMissionAA’. European Commission (2024) Cluster 6: Food, Bioeconomy, Natural Resources, Agriculture and Environment - European Commission, Horizon Europe. Available at: https://research-andinnovation.ec.europa.eu/funding/funding-opportunities/funding-programmes-and-opencalls/horizon-europe/cluster-6-food-bioeconomy-natural-resources-agriculture-andenvironment_en (Accessed: 24 April 2025). Page 66 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters European Commission (2024a) EU Mission: Restore our Ocean and Waters. Available at: https://research-and-innovation.ec.europa.eu/funding/funding-opportunities/fundingprogrammes-and-open-calls/horizon-europe/eu-missions-horizon-europe/restore-our-ocean-andwaters_en (Accessed: 21 March 2024). European Commission (2024b) Open science. Available at: https://rea.ec.europa.eu/openscience_en (Accessed: 6 April 2024). European Commission (2025) Living guidelines on the responsible use of generative AI in research | Research and innovation. Available at: https://research-andinnovation.ec.europa.eu/document/2b6cf7e5-36ac-41cb-aab5-0d32050143dc_en (Accessed: 29 April 2025). European Union (2025) EU Mission Restore our Ocean and Waters Portfolio (public) - Portfolio Dashboard, EU Mission Restore our Ocean and Waters by 2030 portfolio. Available at: https://dashboard.tech.ec.europa.eu/qs_digit_dashboard_mt/public/sense/app/23f6708b-bad64147-889a-7a2205964cd0/sheet/9758dd27-b8ac-43f3-8336-8120db0ae19a/state/analysis (Accessed: 29 April 2025). Fecher, B. and Friesike, S. (2013) ‘Open Science: One Term, Five Schools of Thought’. Rochester, NY. Available at: https://doi.org/10.2139/ssrn.2272036. Garfinkel, M.S. (2020) ‘Towards the right standards: The intersection of open science, responsible research and innovation, and standards’, Mètode Revista de difusió de la investigació [Preprint], (11). Available at: https://doi.org/10.7203/metode.11.16103. GO FAIR Initiative (n.d.) FAIR Principles, GO FAIR. Available at: https://www.go-fair.org/fairprinciples/ (Accessed: 18 April 2024). GRI, UN Global Compact & WBCSD (2015) SDG Compass – A Guide for Business Action to Advance the Sustainable Development Goals. Available at: https://sdgcompass.org/ (Accessed: 13 April 2024). Jensen et al. (2021) ‘WP 4: Global Comparative analysis of SoA Deliverable 4.2: Global SDG and RRI comparative analysis report’. Jensen, E.A. (n.d.) ‘Linking policy and practice in monitoring socially responsible research and innovation (RRI): A conceptual framework to evaluate progress through the UNESCO-led Recommendation on Science and Scientific Researchers [version 2; peer review: 2 approved, 1 approved with reservations, 3 not approved]’, Open Res Europe, 2(21). Available at: https://doi.org/10.12688/openreseurope.14263.2. Klobucaric, B., Baumgärtner, A. and Keding, H. (2024) BLUEMISSIONAA Deliverable 5.8. Holistic RRI, SDG, Open Science and Social Impact evaluation framework., p. 37. Available at: https://bluemissionaa.eu/wp-content/uploads/2024/10/5.8Attachment_0.pdf. Lee, K.-H., Noh, J. and Khim, J.S. (2020) ‘The Blue Economy and the United Nations’ sustainable development goals: Challenges and opportunities’, Environment International, 137, p. 105528. Available at: https://doi.org/10.1016/j.envint.2020.105528. Lindner, R. et al. (2016) ‘»Responsible Research and Innovation« als Ansatz für die Forschungs-, Technologieund Innovationspolitik – Hintergründe und Entwicklungen’. Available at: https://doi.org/10.5445/IR/1000133371. Mansell, P., Philbin, S.P. and Konstantinou, E. (2020) ‘Redefining the Use of Sustainable Development Goals at the Organisation and Project Levels - A Survey of Engineers’, Administrative Sciences, 10(3), p. 55. Available at: https://doi.org/10.3390/admsci10030055. Page 67 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters van Mierlo, B. (2013) Reflexive Monitoring in Action, Wageningen University & Research. Available at: https://www.wur.nl/en/show/reflexive-monitoring-in-action.htm (Accessed: 29 April 2025). Montevirgen, K. (2025) What is a value chain, and how can you use it to make better investment decisions?, Britannica Money. Available at: https://www.britannica.com/money/value-chainanalysis-overview (Accessed: 28 April 2025). Mukherjee, A. and Stern, S. (2009) ‘Disclosure or secrecy? The dynamics of Open Science’, International Journal of Industrial Organization, 27(3), pp. 449–462. Available at: https://doi.org/10.1016/j.ijindorg.2008.11.005. New HoRRIzon Project Consortium (2019) Societal Readiness Thinking Tool. Available at: https://www.thinkingtool.eu/ (Accessed: 27 September 2023). OLAMUR (2023) Project Overview – OLAMUR. Available at: https://olamur.eu/project-overview/ (Accessed: 8 April 2025). ORBIT (2023) ‘The Keys of RRI | Orbit RRI’, Orbit. Available at: https://orbit-rri.org/resources/keys-ofrri/ (Accessed: 12 April 2024). ORBIT RRI (2023) AREA 4P Framework | Orbit RRI, Orbit. Available at: https://orbitrri.org/resources/area4pframework/ (Accessed: 18 April 2024). Ought (2023) Elicit – The AI Research Assistant. Available at: https://elicit.com/. PHAROS Consortium (2022) About - PHAROS Project, PHAROS. Available at: https://pharosproject.eu/about-us/ (Accessed: 24 April 2025). Project Everyone (n.d.) Goal 14: Life below water, The Global Goals. Available at: https://globalgoals.org/goals/14-life-below-water/ (Accessed: 13 April 2024). RIPEET Project Consortium (2020) Self-assessment | RIPEET Project. Available at: https://ripeet.eu/toolkit/self-assessment (Accessed: 13 April 2024). Rogers, E.M. (2003) Diffusion of Innovations. 5. Edition. New York: Free Press. Rosemann, D.A. and Kgwadi, B. (2020) ‘Deliverable D6.1 Mapping the UNESCO Recommendations on Science and Scientific Researchers to the UN Sustainable Development Goals’. RRI Tools Consortium (n. d.) Fragen und Antworten - RRI Tools. Available at: https://rritools.eu/de/fragen-und-antworten (Accessed: 27 September 2023). RRI Tools Consortium (n.d.) RRI Selbstreflexions-Tool. Available at: https://cordis.europa.eu/project/id/612393/reporting (Accessed: 27 September 2023). RRING Project (2020) ‘About Us – RRING Project’. Available at: https://cordis.europa.eu/project/id/788503 (Accessed: 12 April 2024). RRING Project (n.d.) Responsible Research and Innovation & Sustainable Development Goals. Available at: https://rring.eu/impact/ (Accessed: 19 February 2024). RRing Project Consortium (2021) ‘WP 1: Project coordination, Standardisation and Methodology. RRING Self evaluation report of RRI’. Available at: https://ec.europa.eu/research/participants/documents/downloadPublic?documentIds=080166e 5e1111976&appId=PPGMS. Page 68 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters Said, A. and Chuenpagdee, R. (2019) ‘Aligning the sustainable development goals to the smallscale fisheries guidelines: A case for EU fisheries governance’, Marine Policy, 107, p. 103599. Available at: https://doi.org/10.1016/j.marpol.2019.103599. Schmidt, I. et al. (2004) ‘SEEbalance®: Managing sustainability of products and processes with the socio-eco-efficiency analysis by BASF’, Greener Management International, 45. Shelley-Egan, C., Gjefsen, M.D. and Nydal, R. (2020) ‘Consolidating RRI and Open Science: understanding the potential for transformative change’, Life Sciences, Society and Policy, 16(1), p. 7. Available at: https://doi.org/10.1186/s40504-020-00103-5. Singh, G. et al. (2017) ‘A rapid assessment of co-benefits and trade-offs among Sustainable Development Goals’, Marine Policy, 93, pp. 223–231. Available at: https://doi.org/10.1016/j.marpol.2017.05.030. Stahl, B.C. (2017) ‘The ORBIT Self-Assessment Tool’, The ORBIT Journal, 1(2), pp. 1–12. Available at: https://doi.org/10.29297/orbit.v1i2.59. Stahl, B.C. et al. (2017) ‘The responsible research and innovation (RRI) maturity model: Linking theory and practice’, Sustainability (Basel, Switzerland), 9(6), pp. 1036-. Available at: https://doi.org/10.3390/su9061036. Stifterverband für die Deutsche Wissenschaft e.V. (n.d.) Road2Openness | Self Assessment Tool. Available at: https://road2openness.de/ (Accessed: 18 April 2024). Tash, A.P. and Jansen, E.A. (2021) ‘WP 6: D6.2. High-level strategy for various geographies, to promote implementing RRI practices globally’. Available at: https://rring.eu/wpcontent/uploads/2021/07/D6.2.pdf. UNESCO (2021) ‘UNESCO Recommendation on Open Science’. Available at: https://unesdoc.unesco.org/ark:/48223/pf0000379949 (Accessed: 3 November 2023). UNESCO (2023) ‘UNESCO Recommendation on Open Science’. Available at: https://www.unesco.org/en/open-science/about (Accessed: 7 March 2024). United Nations (n.d.a) THE 17 GOALS. Available at: https://sdgs.un.org/goals (Accessed: 19 February 2024). United Nations (n.d.b) Transforming our world: the 2030 Agenda for Sustainable Development. Available at: https://sdgs.un.org/2030agenda (Accessed: 19 February 2024). Vanclay, F. (2003) ‘International Principles For Social Impact Assessment’, Impact Assessment and Project Appraisal, 21(1), pp. 5–12. Available at: https://doi.org/10.3152/147154603781766491. Vicente-Saez, R. and Martinez-Fuentes, C. (2018) ‘Open Science now: A systematic literature review for an integrated definition’, Journal of Business Research, 88, pp. 428–436. Available at: https://doi.org/10.1016/j.jbusres.2017.12.043. Wender, B.A. et al. (2014) ‘Anticipatory life-cycle assessment for responsible research and innovation’, Journal of Responsible Innovation, 1(2), pp. 200–207. Available at: https://doi.org/10.1080/23299460.2014.920121. Wexsus and SDSN Northern Europe (n.d.) SDG Impact Assessment Tool. Available at: https://sdgimpactassessmenttool.org/en-gb (Accessed: 18 April 2024). Page 69 D5.9 Holistic RRI, SDG, Open Science, Social Impact evaluation framework (update): Responsible and Sustainable Research Reflection Tool for Ocean and Waters 2025