Open Infrastructures for Responsible Research Assessment: Principles, Framework, and Checklist for Research Organisations

Open Infrastructures for Responsible Research Assessment: Principles, Framework, and Checklist for Research Organisations 10 DECEMBER 2025 COARA WORKING GROUP - TOWARDS OPEN INFRASTRUCTURES FOR RESPONSIBLE RESEARCH ASSESSMENT (OI4RRA) 10/12/2025 Open Infrastructures for Responsible Research Assessment: Principles and Framework 1 Table of Contents Table of Contents ....................................................................................................................... 1 List of Acronyms ....................................................................................................................... 5 Preamble .................................................................................................................................... 6 Executive Summary ................................................................................................................. 8 Introduction ............................................................................................................................. 11 The Role of Open Infrastructures in Transforming Research Assessment .................... 11 Expand Research Assessment beyond Publications ........................................................ 11 Integrate Qualitative Reforms with Data-Driven Indicators ............................................. 12 Enhancing Standards and Promoting Responsible Research Culture ........................... 12 Reducing Administrative Burdens through Automation ................................................. 12 Principles for Open Infrastructures .................................................................................... 13 What is an Open Infrastructure ........................................................................................... 13 Principles of Open Infrastructures ...................................................................................... 13 Open Infrastructures Fit for Responsible Research Assessment (OI4RRA) ............... 15 Technical Robustness .......................................................................................................... 16 TR1. Data Integrity ............................................................................................................ 16 TR2. Traceability and Reproducibility ............................................................................ 17 TR3. Interoperability ......................................................................................................... 17 TR4. Infrastructure Resilience, Scalability, and Adaptive Performance ...................... 18 TR5. Technological Innovation for Assessment ............................................................ 18 Operational Capacity ........................................................................................................... 19 OC1. Open and Verifiable Workflows ........................................................................... 19 OC2. Inclusivity in Research Output Coverage............................................................ 19 OC3. Context-aware and Balanced Assessment .......................................................... 20 OC4. Operational Efficiency ........................................................................................... 20 OC5. Capacity Building and Support ............................................................................ 21 Community-Centred Practices ............................................................................................ 21 CC1. Collaborative and Participatory Governance ...................................................... 21 CC2. Stakeholder Involvement ....................................................................................... 22 CC3. Equity, Accessibility, and Diverse Representation .............................................. 22 CC4. Resilient Sustainability ............................................................................................ 23 Open Infrastructures for Responsible Research Assessment: Principles and Framework 2 Ethical and Inclusive Practices ............................................................................................ 23 EI1. Fair, Responsible, and Transparent Research Assessment .................................. 23 EI2. Secure and Ethical Data Management ................................................................... 24 EI3. Ethical Oversight and Responsible Technology Use ............................................ 24 EI4. Ethical Neutrality and Fair Responsibility ............................................................... 25 No More Half-Measures: Build the Open Infrastructure that Responsible Research Assessment Deserves ............................................................................................................ 26 Annex ........................................................................................................................................ 28 A. Methodology ................................................................................................................. 28 B. CHECKLIST FOR OPEN INFRASTRUCTURES FOR RPOs AND RFOs ..................... 29 Organisations affiliated to this WG .................................................................................... 32 CoARA....................................................................................................................................... 34 Working Group Overview .................................................................................................... 34 Open Infrastructures for Responsible Research Assessment: Principles and Framework 3 List of Figures Figure 1: A visual summary of the four key categories that define OI4RRA. Open Infrastructures for Responsible Research Assessment: Principles and Framework 4 Please cite as: Manola, N.; Tzouganatou, A.; Kuchma, I.; CoARA OI4RRA WG (2025). CoARA Output. Open Infrastructures for Responsible Research Assessment: Principles and Framework. 10.5281/zenodo.15487403. Document Information Output Title Open Infrastructures for Responsible Research Assessment: Principles and Framework Working Group Cora Working Group - Towards Open Infrastructures for Responsible Research Assessment (OI4RRA) Topic Open Infrastructures and Responsible Research Assessment Draft Version 2.0 Publication Date 08/05/2025 Output Type Policy Paper Producer(s) Natalia Manola, Angeliki Tzouganatou, Iryna Kuchma and the CoARA OI4RRA WG Project Leader(s) Natalia Manola and Clifford Tatum Editor(s) Angeliki Tzouganatou Reviewer(s) Other contributors Status Final Revision history Version Date Revised by Comments 1 10/02/2024 OI4RRA WG 2 08/05/2025 Angeliki Tzouganatou Updated after the Community Consultation 3 10/12/2025 Angeliki Tzouganatou The OI4RRA WG checklist is added in the Annex B Open Infrastructures for Responsible Research Assessment: Principles and Framework 5 List of Acronyms CoARA Coalition for Advancing Research Assessment WG Working Group OI4RRA CoARA Working Group on Towards Open Infrastructures for Responsible Research Assessment OI Open Infrastructure RRA Responsible Research Assessment FAIR Findable Accessible Interoperable Reusable Open Infrastructures for Responsible Research Assessment: Principles and Framework 6 Preamble Effective research assessment is fundamental to the advancement of scientific progress, shaping funding decisions, career trajectories, and institutional strategies. However, traditional evaluation models, often reliant on closed, proprietary infrastructures and rigid, publication-based metrics, fail to capture the full range of scholarly contributions to knowledge, policy, and society. As demands for reform grow, there is a pressing need to transition to transparent, inclusive, and Responsible Research Assessment (RRA) practices that reflect the diversity and complexity of the research enterprise as a whole. This transition cannot be built upon closed infrastructures. In an era of Open Science, relying on opaque and restrictive, proprietary systems driven by profit-oriented commercial imperatives, fundamentally contradicts the very principles of openness and RRA. It is crucial to recognize that these systems are not only proprietary but are also driven by for-profit imperatives that prioritize revenue over transparency and accountability. Commercial systems refer to proprietary systems driven by profit-oriented commercial imperatives, closed-source research platforms controlled by for-profit entities, lacking the transparency and community governance of open infrastructures. This focus on commercial imperatives often result in the concentration of power over research data and scholarly communication, thereby compromising the public interest. The increasing commercialization of academic infrastructure by major publishers raises significant concerns about their growing control over research data and scholarly communication, underscoring the need to advocate for institutional control over these critical resources 1 . Open Infrastructures (OIs) provide the only sustainable and trustworthy path forward. They embed transparency, community governance, democratization, inclusion 2 and interoperability as core attributes, ensuring that research assessment practices evolve in alignment with the needs of the scholarly community. However, for this transition to be successful, OIs must not only replace existing systems, they must surpass them. This output consolidates the foundational Principles and Framework with the corresponding Checklist for Research Organisations, providing both a conceptual 1 Aspesi, C., Allen, N. S., Crow, R., Daugherty, S., Joseph, H., McArthur, J. T., & Shockey, N. (2019, March 29). SPARC Landscape Analysis. , https://doi.org/10.31229/osf.io/58yhb 2 Leiden Madtrics. Alysson Mazoni and Rodrigo Costas. Towards the Democratisation of Open Research Information for Scientometrics and Science Policy: The Campinas Experience. June 06, 2024. Towards the democratisation of open research information for scientometrics and science policy: the Campinas experience - Leiden Madtrics; Dominique Babini, Arianna Becerril Garcia, Rodrigo Costas, Lautaro Matas, Ismael Rafols, and Laura Rovelli. Not only Open, but also Diverse and Inclusive: Towards Decentralised and Federated Research Information Sources. April 22, 2024. Not only Open, but also Diverse and Inclusive: Towards Decentralised and Federated Research Information Sources - Leiden Madtrics Open Infrastructures for Responsible Research Assessment: Principles and Framework 7 foundation and a practical mechanism to guide adoption. Together, they advance a clear, actionable vision for Open Infrastructures fit for Responsible Research Assessment. Open Infrastructures for Responsible Research Assessment: Principles and Framework 8 Executive Summary The CoARA Working Group on Open Infrastructures for Responsible Research Assessment (OI4RRA) was established in November 2023 to define the role of ΟΙs in RRA and provide recommendations for their adoption. Comprising 80 experts from 50 organizations, the group operates under a 24-month mandate to identify the key characteristics and principles that ΟΙs must meet to support RRA, address gaps and challenges in transitioning from proprietary to ΟΙs, and ensure that these systems remain transparent, inclusive, and aligned with evolving research assessment needs. This output brings together the conceptual foundations and the practical evaluation tool developed by the WG, offering a coherent pathway from principles to implementation. Building the Blueprint for Open Infrastructures Fit for Responsible Research Assessment This report is one of the first outputs of the CoARA OI4RRA Working Group, setting the foundation for the principles behind ΟΙs in RRA, their unique contributions, and their essential characteristics. It provides a structured framework for leveraging ΟΙs to support a fair, responsible, and future-proof research assessment system. Specifically, it outlines: ● The critical role of ΟΙs in ensuring transparent, equitable, and responsible assessment. ● The essential characteristics that OIs must possess to surpass the capabilities of proprietary systems. In addition to emphasizing transparency, the report strongly advocates for inclusion, democratization, and active participation 3 . This output also incorporates the OI4RRA Checklist, which operationalizes the framework into concrete, actionable criteria for Research Performing and Research Funding Organisations, enabling them to evaluate infrastructures consistently and identify risks such as black-box algorithms, vendor lock-in, or inadequate governance. This report will be followed by complementary outputs from the Working Group, including a report on transition experiences (highlighting both enablers and challenges and providing strategic recommendations), a conceptual architecture positioning institutional and national investments, and a schema to describe all key characteristics of an OI4RRA, developed in cooperation with GraspOS, which will translate this schema into a registry to help stakeholders discover, evaluate, and engage with ΟΙs that best suit their research assessment needs. 3 This reflects the vision of Sabina Leonelli, who argues for moving the reasoning and justification of open science from mere transparency to a more inclusive approach (see: https://www.cambridge.org/core/elements/philosophy-of-openscience/0D049ECF635F3B676C03C6868873E406). Open Infrastructures for Responsible Research Assessment: Principles and Framework 15 1. Community Governance: Stakeholder-driven models reflecting scholarly diversity, ensuring inclusive, representative, and responsible decision-making. 2. Transparency: Clear communication of policies, governance, and operations, alongside open access and re-use of scholarly outputs to build trust and accountability. 3. FAIR Data: Research data adheres to the FAIR principles. 4. Inclusion: proactively mitigating biases in data, algorithms, and workflows to ensure fair representation across diverse research outputs, disciplines, and geographies. 5. Openness: Access to data, metadata, operational processes, and software. Accompanied by open standards, protocols, and open-source software, ensuring interoperability and reusability under open licenses. 6. Sustainability: Long-term financial models with diverse funding, mitigating risks, ensuring independence and sovereignty from commercial entities, and maintaining equitable participation and benefits. 7. Responsibility, Integrity, and Accountability: Safeguarding the authenticity and reliability of scholarly outputs while ensuring accessible and trustworthy infrastructures. 8. Diversity: Diverse academic, indigenous, and local communities, fostering inclusion and participation across regions and social actors. 9. Equity: Universal accessibility and usability of OIs and scholarly outputs, adhering to metadata standards and advocating equitable participation. 10. Innovation: Reflection to technological and social changes, aligning with community values to enhance scholarly communication. 11. Adaptability: Responsiveness to evolving research needs, enabling systems to progress alongside emerging technologies, research practices, and policy requirements, such as by developing flexible tools and workflows that meet the specific demands of institutions, regions, and disciplines while ensuring inclusivity and equity. Open Infrastructures Fit for Responsible Research Assessment (OI4RRA) Building on the foundational principles of OIs, it is essential to define and refine the specific characteristics required for these systems to play a transformative role in enabling responsible, transparent, and equitable research assessment. Such refinements not only support the transition to fairer research assessment practices but also foster trust and accountability within the research community. To meet the diverse needs of stakeholders, OIs must integrate ethical, sustainable, and adaptable practices. These characteristics are grouped into four key categories, Technical Robustness, Operational Capacity, Open Infrastructures for Responsible Research Assessment: Principles and Framework 16 Community-Driven Practices, and Ethical and Inclusive Foundations, summarised in the following figure. Figure 1: A visual summary of the four key categories that define OI4RRA. Technical Robustness Technical Robustness (TR) for OI4RRA focuses specifically on how OIs manage, process, and utilize data to ensure transparency, fairness, and accountability in research evaluations. This involves addressing the unique requirements of research assessment processes, such as the diverse nature of research outputs, the need for traceable data, and the integration of both qualitative and quantitative information. TR1. Data Integrity Ensuring high-quality and reliable data is fundamental to credible, transparent, and equitable research assessments. Data must be accurate, consistent, and representative of diverse scientific contributions, allowing for fair and trustworthy evaluations. Key Characteristics ● Ensure Accuracy, Consistency, and Coverage – Implement systems with the goal of continuously improving the capability to provide precise, up-to-date, and uniform data across all sources, ensuring inclusivity of research contributions from diverse regions, disciplines, and contexts. ● Implement Validation Mechanisms – Use automated and manual checks to verify data quality, reducing duplication, inconsistencies, and errors in reporting. ● Guarantee Data Reliability and Trustworthiness – Establish robust data management practices to prevent corruption, maintain authenticity, and ensure long-term accessibility for research assessments. Open Infrastructures for Responsible Research Assessment: Principles and Framework 17 Why it matters: High-quality, reliable, and globally inclusive data forms the foundation of credible and equitable research assessments. Poor data quality, inconsistencies, or lack of international representation compromise fairness and erode stakeholder trust. Validated, accurate, and inclusive data strengthens confidence in the system, enabling fair, transparent, and trustworthy evaluations across diverse research communities. TR2. Traceability and Reproducibility A technically robust infrastructure for traceability and reproducibility ensures that data, processes, and research assessments can be independently verified, replicated, and transparently audited. Key Characteristics ● Data Lineage and Processing History – Implement metadata-driven tracking systems to document the complete lifecycle of data, from collection to final use in research assessment. ● Version Control for Transparency – Establish versioning mechanisms for datasets, algorithms, and methodologies to support rollback, reproducibility, and long-term reliability. ● Audit Trails and Verifiable Workflows – Integrate logging and monitoring tools that provide an immutable record of data handling and decision-making. Why this matters: A structured and transparent technical infrastructure ensures credibility, prevents opacity, and fosters trust in research assessments. By implementing systematic tracking, auditability, and version control, stakeholders can replicate findings, verify processes, and confidently rely on research evaluation outcomes. TR3. Interoperability Seamless connectivity ensures that OIs can integrate with other systems, platforms, and tools. This facilitates seamless data flow, fosters collaboration, and minimizes redundancies across the research ecosystem. Key Characteristics ● Open Standards – Adopt widely recognized open standards (e.g., metadata schemas, persistent identifiers) to ensure compatibility with both current and emerging systems. ● Persistent Identifiers – Use globally recognized identifiers such as ORCID IDs, and ROR IDs to link research outputs, contributors, and institutions precisely and reliably. ● APIs and Protocols – Provide well-documented APIs and standardized protocols to enable seamless integration with other tools, platforms, and infrastructures. Why this matters: Interoperability prevents the formation of isolated silos that hinder data sharing and collaboration. By ensuring seamless connectivity, OIs remain adaptable, efficient, and aligned with global research efforts, driving collective progress in the scientific community. Furthermore, interoperability encompasses two key aspects: Technological Interoperability, which enables systems to connect and exchange data through common protocols, standards, and data formats (dealing with the structure of data Open Infrastructures for Responsible Research Assessment: Principles and Framework 18 exchange); and Semantic Interoperability, which ensures that the meaning and context of the exchanged data are preserved and understood by all systems through the use of standardized vocabularies and metadata (addressing the meaning and context of the data). TR4. Infrastructure Resilience, Scalability, and Adaptive Performance A resilient and scalable OI provides continuous availability, efficient performance under varying demands, and adaptability to evolving research needs. Reliability ensures that research workflows remain uninterrupted, while scalability enables infrastructures to accommodate fluctuations in data volume and user activity. Adaptability allows these systems to evolve alongside emerging technologies, research practices, and policy requirements. Key Characteristics ● Fault Tolerance & Redundancy – Implement failover mechanisms, backup systems, and distributed frameworks to prevent downtime and maintain uninterrupted service. ● Real-Time & Proactively Manage Performance – Track system performance continuously to detect anomalies early, minimize disruptions, and ensure operational stability. ● Elastic Scalability to Meet Demand – Dynamically allocate resources to handle peak periods, such as grant evaluations and institutional assessments, without bottlenecks. ● Performance Optimisation & Workflow Adaptation – Manage resources efficiently to sustain performance across diverse workloads, supporting both routine and high-intensity operations. ● Future Needs Anticipation & Flexibility – Support emerging data types, interdisciplinary research metrics, and Open Science practices to ensure long-term sustainability. Why this matters: Without scalability and resilience, OIs risk disrupting critical research and assessment workflows during high-demand periods. Reliable and adaptive systems foster trust, support innovation, and ensure efficiency, aligning with the evolving global research landscape. TR5. Technological Innovation for Assessment Technology-driven innovation harnesses advanced technologies like AI, including Machine Learning/Natural Language Processing (NLP), to analyse data, support qualitative and datadriven assessments, and provide actionable insights. Key Characteristics ● Open/Explainable Artificial Intelligence (ΧAI) – Enable the analysis of vast datasets to identify patterns, automate repetitive or complex analyses, and deliver actionable insights. This is achieved by leveraging AI approaches that are open (utilizing open-source tools and/or transparent algorithms) and explainable (capable of providing understandable rationales for decisions) These capabilities improve the efficiency, depth, and precision of research assessments. Open Infrastructures for Responsible Research Assessment: Principles and Framework 19 ● Open NLP – Enhance qualitative evaluations by analysing narratives found in grant proposals, peer reviews, or impact statements using NLP tools that are openly available (open-source or developed with transparent methodologies). These tools provide nuanced insights into research quality and context, complementing quantitative metrics. Why it Matters: Without advanced and transparent technologies, OIs may struggle to meet the complexities of modern research assessment, limiting their ability to deliver timely and actionable insights. However, as traditional black-box metrics have been rightfully challenged, it is equally important to critically assess the role of AI and NLP in research assessment contexts. In this regard, XAI and auditable AI refer to established methodologies and standards that ensure algorithmic processes are transparent, interpretable, and subject to independent review. The responsible adoption of explainable and auditable AI ensures that OIs remain adaptive, accountable, and effective while avoiding the pitfalls of opaque algorithmic decision-making. Operational Capacity Operational Capacity (OC) for OI4RRA focuses on how the infrastructure supports endusers, processes, and workflows in practical, day-to-day implementation. It emphasizes usability, efficiency, and alignment with stakeholder needs. OC1. Open and Verifiable Workflows An open and verifiable workflow provides transparency, explainability, and accountability in research assessment by ensuring that every stage of data processing and decisionmaking is accessible, understandable, and subject to scrutiny. Key Characteristics ● Document All Stages of the Assessment Process – Ensure clear documentation of data collection, cleaning, processing, and analysis, making the entire workflow traceable and reproducible. ● Ensure Explainability in Algorithmic Decisions – Provide structured insights into how algorithms influence outcomes, detailing input variables, weighting factors, and decision logic. ● Facilitate Independent Audits and Stakeholder Engagement – Allow third-party verification of workflows through transparent logging, public reporting, and stakeholder feedback mechanisms. Why this matters: Without transparency, stakeholders may mistrust assessment outcomes, questioning their fairness and reliability. Clear documentation, algorithmic transparency, and auditability ensure accountability and help users feel confident in the system’s integrity. OC2. Inclusivity in Research Output Coverage Recognizing diverse contributions ensures that research assessments value diverse outputs, including non-traditional contributions such as datasets, software, and community engagement activities, alongside traditional publications. Key Characteristics Open Infrastructures for Responsible Research Assessment: Principles and Framework 20 ● Inclusivity of Outputs – Encompasses datasets, software, policy contributions, patents, and teaching activities, ensuring a comprehensive view of research outputs. ● Recognition of Non-Traditional Contributions – Values activities such as mentoring, science communication, community engagement, and interdisciplinary collaborations. ● Diverse Research Practices – Accommodates unique contributions from various disciplines, career stages, and geographic regions, fostering diversity. Why this matters: Traditional metrics often neglect valuable contributions like community engagement, mentorship, or interdisciplinary work, resulting in incomplete and biased evaluations. Recognizing diverse contributions ensures a holistic and equitable assessment of research impact. OC3. Context-aware and Balanced Assessment A user-centric and context-aware assessment framework ensures that OIs are adaptable to diverse institutional, disciplinary, and regional needs while maintaining a balance between quantitative metrics and qualitative narratives. Key Characteristics ● Customization and Local Adaptation – Allow institutions to modify dashboards, workflows, and indicators to align with their specific goals. ● Diverse Use Cases and Assessment Scenarios – Accommodate different research profiles, including early-career researchers, interdisciplinary work, and collaborative projects, ensuring inclusivity. ● Qualitative and Quantitative Evidence – Combine narrative-based insights with metrics-based indicators to provide well-rounded evaluations. ● Comprehensive and Flexible Frameworks – Develop tools that recognize nontraditional research contributions (e.g., mentoring, open science practices) while ensuring fair comparisons across different contexts. Why this matters: A rigid, metric-heavy approach risks oversimplifying research impact, while an over-reliance on narratives may lack comparability. By balancing adaptability with robust assessment methods, OIs foster trust, inclusivity, and meaningful evaluations, ensuring that research assessments are both context-sensitive and methodologically sound. OC4. Operational Efficiency A well-designed OI optimizes workflows, automates repetitive tasks, and ensures userfriendly interactions, enabling stakeholders to work efficiently and effectively. Key Characteristics ● Streamlined Workflows – Eliminate redundancies and optimize processes for data collection, integration, processing, and reporting, ensuring faster operations with reduced friction. ● Automation – Automate repetitive tasks, such as data integration and cleaning, enabling users to focus on strategic activities like analysis and decision-making. Open Infrastructures for Responsible Research Assessment: Principles and Framework 21 ● User-Friendly Interfaces – Design intuitive dashboards and tools that simplify data access and usability, catering to users with varying technical expertise. Why this matters: Inefficient workflows and complex interfaces waste time, create resource bottlenecks, and frustrate users. A streamlined and accessible system maximizes efficiency and allows stakeholders to concentrate on high-value activities like interpretation and strategic planning. OC5. Capacity Building and Support A well-supported OI ensures that stakeholders have the tools, knowledge, skills, and confidence to engage effectively, maximizing its impact. Key Characteristics ● Training Programmes – Offer workshops, tutorials, and webinars for researchers, administrators, and policymakers, covering both system functionalities and best practices in data interpretation. Institutions must proactively offer continuous training initiatives to ensure every user remains adept and empowered in navigating and leveraging the system. ● Support Networks – Provide dedicated helpdesks, active user communities, and responsive technical teams to assist users and enhance their experience. ● Skill Development – Strengthen both technical (e.g., data analysis, visualization) and contextual (e.g., responsible metric interpretation) skills to ensure meaningful engagement with the infrastructure. Why this matters: Without proper training and support, users may face difficulties navigating and utilizing the system, leading to inefficiencies and reduced impact. Continuous capacity-building fosters confidence, ensuring optimal use of the infrastructure and sustained adoption over time. Community-Centred Practices Community-Centred (CC) Practices emphasize the collaborative, participatory, and inclusive nature of OI4RRA. These principles ensure that OIs align with the diverse needs of the research community, foster shared ownership, and enable meaningful engagement. By focusing on participation, inclusivity, and sustainability, community-centred principles empower stakeholders to shape and govern OIs in a way that reflects their values and priorities. CC1. Collaborative and Participatory Governance An inclusive governance ensures that OIs are guided by diverse stakeholder perspectives, fostering transparency, accountability, and trust while aligning priorities with the needs of the research community. Key Characteristics ● Inclusive Decision-Making – Actively involve a broad range of stakeholders — researchers, administrators, funders, and policymakers — in governance to reflect varied perspectives and priorities in decision-making. Open Infrastructures for Responsible Research Assessment: Principles and Framework 22 ● Transparent Policies – Clearly document and share governance decisions, policies, and processes to build accountability and trust in the system’s leadership. ● Community Oversight – Establish governance boards or advisory committees with community representatives to guide strategy and oversee priorities, ensuring alignment with the broader research community’s goals. Why this matters: Without participatory governance, OIs risk being dominated by narrow interests, undermining their relevance and trustworthiness. Collaborative leadership ensures decisions are equitable and inclusive, and reflect the diverse values of all stakeholders. CC2. Stakeholder Involvement Community-driven design ensures that OIs are shaped collaboratively with their users, fostering shared ownership, increasing adoption, and aligning the infrastructure with the evolving needs of the research community. Key Characteristics ● Co-Creation Processes – Actively involve stakeholders in shaping the infrastructure by defining features, setting strategic priorities, and contributing to its development. This collaborative approach fosters ownership and ensures alignment with user needs and expectations. ● Feedback Mechanisms – Establish regular channels and open forums, to gather stakeholder input. These mechanisms enable users to highlight areas for improvement and guide the infrastructure's evolution. ● Outreach and Awareness – Conduct educational campaigns, workshops, and community events to raise awareness about the infrastructure’s purpose, features, and benefits. Effective outreach drives broader participation and enhances the impact of the infrastructure. Why this matters: Without active stakeholder engagement, OIs risk becoming disconnected from user needs, reducing their adoption and long-term relevance. Meaningfully engaging stakeholders builds trust, broadens participation, and fosters a shared sense of responsibility for the infrastructure’s success. CC3. Equity, Accessibility, and Diverse Representation Open Infrastructures must embrace diversity, support multilingual and multicultural participation, and ensure accessibility to create fair and representative research assessments. Diverse research practices could also include language diversity (as opposed to the dominance of English). Asides from career stages, it's also key to consider societal actors involved in research (namely, citizen science projects). By actively engaging underrepresented communities and adapting to different regional and disciplinary contexts, OIs can foster a truly inclusive and globally relevant assessment ecosystem. Key Characteristics ● Diverse Participation – Actively include underrepresented groups, such as researchers from the Global South, Indigenous communities, early-career scholars, Open Infrastructures for Responsible Research Assessment: Principles and Framework 23 researchers with disabilities, and non-English-speaking communities, ensuring broad and equitable representation. ● Multilingual and Multicultural Support – Provide support for multiple languages, cultural norms, and assessment practices, ensuring accessibility for users across regions and backgrounds. ● Contextual Adaptability – Develop flexible tools and workflows that address the specific needs of institutions, regions, and disciplines while maintaining inclusivity and equity. Why it matters: Excluding diverse voices risks marginalizing critical perspectives, reducing the system's global relevance and credibility. Equity and accessibility enhance the infrastructure’s fairness, adaptability, and ability to address the global diversity of research practices and contributions effectively. CC4. Resilient Sustainability Open Infrastructures must be financially stable, strategically adaptable, and operationally independent to support long-term research assessment needs. By securing diverse funding sources, anticipating future challenges, and maintaining autonomy from large commercial interests, governmental and other interests, OIs can ensure reliability, trust, and long-term impact. Key Characteristics ● Robust & Diverse Funding Models – Secure diverse revenue streams, including consortia memberships, public grants, and institutional contributions, to guarantee financial stability and reduce dependence on a single source. ● Long-Term Planning – Develop strategic plans that anticipate future challenges, such as growing user bases, evolving technologies, and changing policy landscapes, to ensure ongoing relevance and responsiveness. ● Independent Operations – Avoid over-reliance on commercial entities or external pressures, ensuring alignment with the research community’s values and priorities. Why it matters: Without financial and operational sustainability, OIs risk instability, diminished credibility, and loss of trust. A resilient and independent infrastructure guarantees consistent service, adaptability to emerging needs, and long-term alignment with research community goals, ensuring lasting impact. Ethical and Inclusive Practices Ethical and Inclusive (EI) foundations underpin the fairness, inclusivity, integrity and accountability of OI4RRA. They ensure that the design, operation, and governance of OIs align with the values of equity, transparency, and respect for individual and institutional rights. Ethical principles are essential for building trust among stakeholders and fostering confidence in the outcomes of research assessment. EI1. Fair, Responsible, and Transparent Research Assessment A responsible and transparent research assessment framework ensures that metrics, workflows, and evaluation processes are open, contextualized, and verifiable. Transparent Open Infrastructures for Responsible Research Assessment: Principles and Framework 24 documentation and ethical verification mechanisms foster trust, reproducibility, and stakeholder confidence in assessment outcomes. Key Characteristics ● Transparency in Metrics – Clearly document methodologies, data sources, and assumptions behind metrics to ensure openness, interpretability, and responsible use within OIs. ● Contextualized and Nuanced Evaluations – Avoid oversimplified and rigid indicators by incorporating qualitative insights, diverse contributions, and Open Science principles into assessment frameworks. ● Version Control for Traceability and Accountability – Track and make accessible all changes to datasets, algorithms, and methodologies to enable stakeholder verification and auditability. ● Verification and Ethical Compliance Mechanisms – Support independent audits, community-driven validation, and open governance to align with best practices in Open Research Infrastructures and RRA. Why this matters: Without clear documentation, contextualized assessment approaches, and independent verification, research assessments risk being opaque, biased, and unaccountable. A well-documented, reproducible, and ethically grounded evaluation system ensures that research contributions are fairly assessed, stakeholders are informed, and OIs remain trustworthy alternatives to proprietary evaluation systems. EI2. Secure and Ethical Data Management A responsible approach to data privacy and security ensures that OIs handle sensitive information ethically and in compliance with legal frameworks. By implementing strong governance, privacy safeguards, and controlled access, OIs can protect data integrity, prevent misuse, and foster trust among stakeholders. Key Characteristics ● Compliance with Regulations – Adhere to relevant legal frameworks (e.g., GDPR) to process personal data securely, minimizing legal and reputational risks. ● Identity protection through Anonymization and Aggregation – Apply anonymization and data aggregation techniques to enable meaningful insights while safeguarding individual privacy. ● Role-Based Access and Security Controls – Restrict access to sensitive data to authorized personnel, ensuring confidentiality, accountability, and prevention of misuse. Why this matters: Without robust privacy protections, breaches can harm individuals and institutions, undermining trust and exposing OIs to significant risks. Strong data security measures ensure ethical data handling, protect system integrity, and maintain stakeholder confidence. EI3. Ethical Oversight and Responsible Technology Use Open Infrastructures must operate transparently, integrate stakeholder input, and implement technology responsibly to ensure fairness and trust in research assessments. Open Infrastructures for Responsible Research Assessment: Principles and Framework 31 Open Infrastructures for Responsible Research Assessment: Principles and Framework 32 Organisations affiliated to this WG • OpenAIRE AMKE • Leiden University (CWTS) • Consiglio Nazionale delle Ricerche (CNR) • Federation of Finnish Learned Societies (TSV) • Ss Cyril and Methodius University in Skopje • University of Minho • Masaryk University • OpenCitations • University of Debrecen • Trinity College Dublin • University of Bologna • Universities Norway • NIFU (Nordic institute for studies of innovation, research and education) • EIFL • Netherlands Research Council (NWO) • COKI (Australia) * • National Institute of Informatics (NII, Japan) * • Michael J. Fox Foundation (United States) * • Technische Universiteit Delft • West and Central African Research and Education Network (WACREN, Ghana) * • Vrije Universiteit Amsterdam (Netherlands) • UEFISCDI (Romania) • University of Ljubljana (Slovenia) • Netherlands eScience Center (RSD) • UKRN • NASA TOPS* • University of South-Eastern Norway • University of Hamburg • INRIA • Athena Research Center • LMU Munich • Université Côte d’Azur • University of Novi Sad • University of Padua • Lusófona University • Petre Shotadze Tbilisi Medical Academy • CSC – IT Center for Science • Netherlands eScience Center (RSD) • Library and Information Centre of the Hungarian Academy of Sciences • Training Centre in Communication (TCC Africa) Open Infrastructures for Responsible Research Assessment: Principles and Framework 33 • University of Groningen • SPARC EUROPE • JISC Open Infrastructures for Responsible Research Assessment: Principles and Framework 34 CoARA The Coalition for Advancing Research Assessment (CoARA) is a collective of organisations committed to reforming the methods and processes by which research, researchers, and research organisations are evaluated. Current research assessment methods rely heavily on publication-based metrics such as citation counts and often fail to recognise the wide array of contributions made by researchers. Over 700 research organisations, funders, assessment authorities, professional societies, and their associations have agreed on a common direction and guiding principles to implement reform in the assessment of research, researchers, and research organisations, outlined in the Agreement on Reforming Research Assessment published in July 2022 which provides an outline for reform and implementation. Find out more about CoARA at www.coara.eu. Working Group Overview Working Groups are key communities of practice that work to implement research assessment reform in specific thematic areas. Participating members exchange knowledge, learn from each other’s experience, discuss and develop outputs to advance research assessment and support the implementation of members’ commitments. The Working Group ‘’Towards Open Infrastructures for Responsible Research Assessment’’ is committed to advancing the transition from closed, proprietary research infrastructures toward open, community-governed, and interoperable systems. This shift is essential for enabling ethical, transparent, and RRA practices. The group emphasizes the importance of making research information universally accessible, transparent, and inclusive of a wide diversity of disciplines, geographic contexts, languages, and research output types. It prioritizes the development of open infrastructures that are sustainable, interoperable, community-driven, and aligned with the principles of Open Science. Central to this work is the aim to support infrastructures that foster trustworthiness, equity, and inclusivity in research assessment processes, ensuring that the infrastructures themselves are fit for responsible research evaluation across varied global contexts.