Ethics first: rethinking research assessment in the age of AI and open science

CoARA Working Group on ‘Ethics and Research Integrity Policy in Responsible Research Assessment for Data and Artificial Intelligence (ERIP)’ Prof. Dr. Perihan Elif EKMEKCI •Workstream 1: The creation of an outreach program to shape policy, foster institutional and societal change, and create awareness, promoting a culture of responsible data and AI research. •oThe ERIP Handbook on integrity, data, and AI in research evaluation to offer a comprehensive guide to navigating ethical complexities and leveraging digital advancements in contemporary research assessment practices. •oA Series of ERIP Webinars •oStakeholder Engagement Workshops •oSocial Media Presence ••Workstream 2: The development of policy, methods, and guidelines to ensure the research ethics and integrity of scientific methods and outputs with the advancing use of data and application of AI methods and to assess digital contributions to science/knowledge in research programs and assessment procedures. •oA Policy Framework for Ethical Research Assessment in the Digital Age •oA Framework for assessing and evaluating Digital Contributions . •oMethodological Guidelines for Assessing Digital Contributions and Ethical Research Assessment in the digital age • •Workstream 3: The development of a comprehensive toolkit for the implementation of best practices in research assessment for data and AI. •oA toolkit for evaluation of Digital Contributions to Science •oA Digital Toolkit for Responsible Research Assessment •oTechnology-based Tools for Best Practices for Data and AI in Research Assessment Objectives of ERIP •to develop ethics and research integrity guidance for research assessment involving data and AI; •to promote responsible data and AI practices by advocating for responsible assessment practices that prioritize integrity, responsibility, and societal benefit, while mitigating the potential risks and harms associated with data and AI-driven research; •to foster stakeholder engagement through facilitated dialogue and collaboration among researchers, policymakers, funding agencies, and other actors to ensure diverse perspectives in the development and implementation of ethical research assessment policies; •to provide a toolkit of educational resources to raise awareness and help implement best practices in research assessment for data and AI; and •to engage in outreach to shape policy and institutional change, promoting a culture of responsible data and AI research. Developing methods and tools to ensure the research ethics and integrity of scientific outputs with the advancing use of data and the impact of AI to evaluate digital contributions to science/knowledge in research program and assessment procedures; and innovative methodologies for employing data ecosystems and AI models for research assessment in digital environments. Guiding principles to help us navigate the comp lexities of research assessment in the digital ag e. Embedding Ethics into AI Practices for Research Assessment The Modular Configurator Supported by the ENVISAGE Principles and PILOT Principles Open Science Fair Session Ethics First: Rethinking Research Assessment in the Age of AI and Open Science CERN, Geneva, Switzerland 17 September 2025 Francis P. Crawley COARA-ERIP CODATA IDPC The Why We Need an Ethical Overhaul Outdated Metrics, New Realities Traditional metrics (JIF, citations) fail to assess modern research: datasets, software, interdisciplinary work, and societal impact. The AI Revolution introduces new challenges: •Asymmetry: AI-empowered applicants vs. assessment processes lacking tools to evaluate AIgenerated content. •New artifacts: How to assess AI models, generated content, and complex digital outputs? •Ethical risks: Amplification of bias, lack of transparency, and threats to research integrity. The Core Conflict: The gap between rapid technological advancement and slow, rigid assessment frameworks risks undermining science and higher education. OSF Geneva CERN, 17 Sept 2025 Embedding Ethics into AI Practices for RA Slide 6 The Modular Assessment Configurator (MAC): A Vision for Ethical, Adaptive Research Assessment From Rigid Metrics to Responsible Configuration The MAC is a proposed digital, interactive tool designed to operationalize high-level ethical principles into practical, customizable research assessment (RA) processes. It moves beyond one-size-fits-all metrics to allow for context-sensitive, value-based evaluation across a range of research contributions and artifacts. Key Motivations: •The Problem: Traditional metrics (JIF, citation counts) are inadequate for assessing modern, collaborative, and AI-enabled research outputs like datasets, software, and AI models. •The AI Challenge: Creates an asymmetry between AI-empowered applicants and assessment processes that lack the tools to evaluate new research artifacts ethically and transparently. •The Need for Flexibility: Different disciplines, institutions, and missions require different assessment criteria. The MAC provides the needed configurability. OSF Geneva CERN, 17 Sept 2025 Embedding Ethics into AI Practices for RA Slide 7 The Modular Assessment Configurator (MAC): A Vision for Ethical, Adaptive Research Assessment From Rigid Metrics to Responsible Configuration The MAC is a proposed digital, interactive tool designed to operationalize high-level ethical principles into practical, customizable research assessment (RA) processes. It moves beyond one-size-fits-all metrics to allow for context-sensitive, value-based evaluation across a range of research contributions and artifacts. Key Motivations: •The Problem: Traditional metrics (JIF, citation counts) are inadequate for assessing modern, collaborative, and AI-enabled research outputs like datasets, software, and AI models. •The AI Challenge: Creates an asymmetry between AI-empowered applicants and assessment processes that lack the tools to evaluate new research artifacts ethically and transparently. •The Need for Flexibility: Different disciplines, institutions, and missions require different assessment criteria. The MAC provides the needed configurability. OSF Geneva CERN, 17 Sept 2025 Embedding Ethics into AI Practices for RA Slide 8 The Ethical Compass ENVISAGE Principles for DRR ENVISAGE provides the constitutive ethical ‘why’ for using AI in DRR, ensuring technology serves humanity. Critical principles for crisis contexts: •Sovereignty (S): Respects community & national rights over their data and risks. •Integrity (I): Ensures honesty and reliability of AI-driven risk information. •Access (A): Promotes equitable access to AI tools and benefits for all communities. •Engagement (E): Mandates involving local stakeholders in AI design and deployment. •Goal: Prevents digital colonialism, builds trust, and ensures AI solutions are culturally and contextually appropriate. ‘To envisage is not merely to imagine but to anticipate with ethical purpose and orient action toward just and humane outcomes.’ OSF Geneva CERN, 17 Sept 2025 Embedding Ethics into AI Practices for RA Slide 9 Ethics-Based Tools for Responsible Assessment Based on CoARA-ERIP’s workstreams: ❑How can we ensure transparency and accountability when assessing research outputs generated or assisted by AI? Should AI-generated content be evaluated differently from human-authored work, or by the same ethical standards? ❑Are our current evaluation systems equipped to fairly recognize the value of interdisciplinary and transdisciplinary research? What biases or blind spots might exist in traditional metrics when applied to collaborative or cross-sectoral work? ❑What would an ethically inclusive research assessment framework look like in practice? How can we balance qualitative and quantitative indicators to reflect integrity, equity, and societal relevance? CoARA-ERIP is developing frameworks that respond to these challenges. These include: ▪Ethical criteria for evaluating AI-generated research outputs, ensuring transparency and accountability. ▪Qualitative indicators that reflect integrity, equity, and societal relevance. ▪Tools for assessing interdisciplinary and transdisciplinary contributions, which are often overlooked in traditional models. These frameworks are not abstract ideals; they are practical instruments that institutions, funders, and evaluators can adopt to build more just and reflective assessment systems. Ethics-Based Tools for Responsible Assessment A concrete example: Imagine a university evaluating candidates for a research position. Instead of relying solely on publication counts or journal rankings, it could apply CoARA-ERIP’s ethical criteria to assess the societal impact, collaborative nature, and methodological integrity of each candidate’s work. From a bioethical standpoint, responsible research assessment must be guided by core principles: ❑Autonomy: Respecting diverse methodologies and epistemologies. ❑Justice: Ensuring fair evaluation across disciplines and geographies. ❑Beneficence and non-maleficence: Avoiding harm by recognizing and valuing contributions that may be invisible to conventional metrics. Institutions have a moral responsibility to foster a culture of ethical evaluation, one that supports innovation without compromising integrity. A Bioethical Perspective Respect for diverse methodologies, epistemologies, and research cultures. Promotes recognition of non-traditional outputs, including AI-assisted and collaborative research, ensuring researchers are not penalized for choosing innovative or inclusive approaches. Are we allowing researchers the freedom to innovate, or are we constraining them with outdated metrics? Autonomy Justice Fair evaluation across disciplines, geographies, and institutional contexts. Advocates for inclusive assessment systems that recognize interdisciplinary and transdisciplinary contributions and address systemic biases in global research evaluation. - Only 12% of existing frameworks adequately assess interdisciplinary work, revealing a justice gap in current practices. How can we redesign evaluation systems to ensure equity across diverse research landscapes? Avoiding harm by recognizing and valuing contributions that may be invisible to conventional metrics. Encourages the use of qualitative indicators that reflect societal impact, ethical integrity, and collaborative value, especially in AI-generated and data-intensive research. AI-generated content poses risks of misinformation and bias. Ethical evaluation frameworks must ensure these outputs are assessed with transparency and accountability. Are we unintentionally harming scientific progress by ignoring valuable contributions that fall outside traditional metrics? Beneficence and Non-Maleficence Institutions have a moral and strategic responsibility to raise a culture of ethical evaluation, one that supports innovation without compromising integrity. This includes: •Adopting CoARA-ERIP’s tools and frameworks; •Training evaluators in ethical and inclusive assessment; •Engaging researchers in co-creating evaluation criteria. What steps can your institution take today to embed ethics into its research assessment practices? Conclusion –Ethics First Ethics is not a barrier to progress; it is the foundation of trustworthy science. As we rethink research assessment in the age of AI and open science, we must place ethics at the center of our frameworks, policies, and practices. I invite all of you—researchers, funders, institutionsto engage with the work of CoARA-ERIP and help build systems that reflect the values we want science to uphold. To evaluate ethically is to recognize the true value of science—not just for academia, but for society. By placing ethics at the heart of research assessment, CoARA-ERIP is not just reshaping how we evaluate science, it is redefining how science serves society with integrity, inclusiveness, and purpose. Call to Institutional Action Ethics First: Rethinking Research Assessment 1517 SE P T E M BE R Authors: Mr CRAWLEY, Francis P. (CoARA Ethics and Research Integrity Policy Working Group on Responsible Research Assessment in Data and Artificial Intelligence (CoARA-ERIP)); Prof. EKMEKCI, Perihan Elif, (Associate Professor, Deputy Dean, and Head of the History of Medicine and Ethics Department, School of Medicine, TOBB ETU) Prof. DE SOUSA FREITAS, Mara (Director of the Institute of Bioethics of the Universidade Católica Portuguesa (IB-UCP));