Should we be assessing openness openly or at all?

SHOULD WE BE ASSESSING OPENNESS OPENLY OR AT ALL? GraspOS Final Conference 12 November 2025 Dr Elizabeth Gadd CoARA Vice Chair INORMS Research Evaluation Group Chair Loughborough University, UK OVERVIEW •Open research as a driver of assessment reform •Should we be assessing openness openly? oWhat level of openness do the key RRA principles demand? oTo what extent do we make good on those expectations? •Should we be assessing openness at all? oWorking through the SCOPE Framework Aubert-Bonn, Noemie. Literature Review. In: Curry, Stephen; Gadd, Elizabeth; Wilsdon, James (2022): Harnessing the Metric Tide: indicators, infrastructures & priorities for UK responsible research assessment. DRIVERS OF ASSESSMENT REFORM Open research Equity, Diversion & Inclusion Research careers Research Culture Research Integrity 4 Assessment Reform SHOULD WE ASSESS OPENNESS OPENLY? WHAT ARE THE OPENNESS EXPECTATIONS OF THE MAIN PRINCIPLES OF RRA? •DORA – “Be explicit about the criteria used to reach hiring, tenure, and promotion decisions...” (CRITERIA) •Metric Tide – “Transparency: Openly communicating methods & limitations” (METHODS & LIMITATIONS) •Leiden Manifesto – “Allow those evaluated to verify data and analysis.” (DATA & ANALYSIS) •CoARA – “Evaluate practices, criteria & tools… and make data openly available for evidence gathering and research.” (ASSESSMENT REFORM DATA) TO WHAT EXTENT ARE ASSESSMENT CRITERIA & METHODS OPEN (& TO WHOM)? https://eua.eu/downloads/publications/research%20assessment%20in%20the %20transition%20to%20open%20science.pdf Strategic Approaches to, and Research Assessment of, Open Science. James Morris & Bregt Saenen. (2024) https://zenodo.org/records/13961124 TO WHAT EXTENT ARE ASSESSMENT DATA & ANALYSIS OPEN (& TO WHOM)? INTRODUCING THE INORMS SCOPE FRAMEWORK THREE PRINCIPLE S OF SCOPE Evaluate only where necessary 19 MENTAL HEALTH COSTS THE LOGIC •Based on Campbell’s Law: What we measure is what get •Assess based on highly-cited journal articles Negative impacts on quality, integrity, cost of open research, ghost/guest/gift authorships •Assess better behaviours (open practices, integrity, collegiality) Get better behaviours? 3 PROBLEMS WITH ASSESSING TO INCENTIVISE 1. We add new things to our assessments without re-weighting the old COARA ACADEMIC CAREER ASSESSMENT WG SURVEY Credit: Janne Polonen THREE PRINCIPLE S OF SCOPE Evaluate only where necessary Evaluate with the evaluated THREE PRINCIPLE S OF SCOPE Evaluate only where necessary Evaluate with the evaluated Draw on evaluation expertise START WITH WHAT YOU VALUE S - START WITH WHAT YOU VALUE ✔ Not what others’ value ✔ Not by the availability of data THE STREETLIGHT EFFECT: MEASURING BY AVAILABLE DATA NOT BY MISSION PLOS OPEN SCIENCE INDICATORS 37 “Open science is just good science”. (Always?) “Closed science is bad science”. (Really?) “If it’s not open, is it really research?” (Erm, yes?) “We shouldn’t monitor whether there is more or less open science, but what types of OS are developed and adopted, by whom, and with what consequences.” WHAT DO WE VALUE ABOUT OPEN RESEARCH? WEIGHTING WHAT WE VALUE TENSIONS BETWEEN OPEN RESEARCH AND RESEARCH SECURITY 41 OS-CAM https://op.europa.eu/en/publication-detail/-/publication/47a3a330-c9cb-11e7-8e69-01aa75ed71a1 49 PROLIFERATING ALTERNATIVES – HELP OR HINDRANCE? MORE JOINED UP THINKING ABOUT ASSESSMENT STANDARDS? 51 CONCERNS ABOUT GLOBAL MOBILITY 52 DO WE NEED A ‘DUBLIN CORE’ FOR RESEARCHER ASSESSMENT? •Can the international community agree a core (but broader) set of dimensions on which researchers will be assessed? •Disciplines and regions may add to the core, but the essentials remain •Benefits: oStability and certainty for researchers oAutonomy for individual regions and organisation •Disadvantages: oWould it become a race to the bottom? “AS ALIGNED AS POSSIBLE, AS DIVERSE AS NECESSARY” P - PROBE 1. Who might this discriminate against? 2. What might the unintended consequences be? 3. How might this be gamed? 4. What is the cost-benefit? HOW OPEN IS OPEN TO THE ARTS & HUMANITIES? EEVALUATE YOUR EVALUATION •Evaluation is cyclical and iterative •Use SCOPE to re-evaluate your evaluation CONCLUSION •Should we assess openly? oYes oBut we need more work on ‘open evaluation’ ▪As a key element of Open Science ▪As a key element of Responsible Research Assessment •Should we assess openness at all? oYes – in line with SCOPE ▪In certain contexts ▪In line with our values ▪With the right assessment mechanisms ▪…probed carefully THANK YOU FOR LISTENING! Useful links: •CoARA website •https://coara.org •INORMS SCOPE framework •https://inorms.net/scope-framework-for -research-evaluation/ •Barcelona Declaration •https://barcelona-declaration.org/ 6666 CoARA queries @CoARAssessment Coalition for Advancing Research Assessment Contact us at [email protected]