Implementation proposal for language-aware assessments

Implementation proposal for language -aware assessments WORKING GROUP ON MULTILINGUALISM AND LANGUAGE BIASES IN RESEARCH ASSESSMENT Implementation proposal for language-aware assessments 1 Implementation proposal for language-aware assessments Table of Contents Introduction ............................................................................................................................... 5 Proposal 1: Make multilingualism a value in research assessment .............................. 5 Proposal 2: Assess and improve language diversity, competences and practices ... 6 Proposal 3: Create expectations and criteria for multilingualism ................................ 7 Proposal 4: Integrate multilingualism into assessment processes ............................... 7 Implementation proposal for language-aware assessments 2 Please cite as: CoARA WG on Multilingualism and Language Biases in Research Assessment. (3 October 2025). CoARA Output. Implementation proposal for language-aware assessments. DOI: https://doi.org/10.5281/zenodo.17256827. Document Information Output Title Implementation proposal for language-aware assessments Working Group Multilingualism and Language Biases in Research Assessment Topic Draft Version 3.0 Publication Date 3 October 2025 Output Type Actionable policy resource Producer(s) Working Group on Multilingualism and Language Biases in Research Assessment Project Leader(s) Martine Garnier-Rizet, Gian-Maria Greco, Emanuel Kulczycki, and Janne Pölönen Editor(s) Janne Pölönen Reviewer(s) WG members and Focus Group participants Other contributors WG and TF3 Focus Group members Status Draft Revision history Version Date Revised by Comments 1.0 09-05-2025 J. Pölönen, M. C. Conceição, J. M. Cots, B. Dendrinos, M. Dietl, S. Garcia, E. Gardan, M. Garnier-Rizet, G. M. Greco, E. Kulczycki, P. Laihonen, F. Mystakopoulos, S. Penati, G. Peruginelli, I. Rafols, C. Soler, M. Urbanc First draft 2.0 23-05-2025 WG members’ feedback 3.0 03-10-2025 TF3 Focus Group feedback Implementation proposal for language-aware assessments 3 List of Acronyms CoARA Coalition for Advancing Research Assessment CRIS Current Research Information System CV Curriculum Vitae WG Working Group Implementation proposal for language-aware assessments 4 Summary This implementation proposal is aimed at leadership and management of higher education institutions, research and funding organisations to facilitate implementation of CoARA commitments. The proposal has been produced by the CoARA WG on Multilingualism and language biases in research assessment. The document presents four proposals for the possible implementation of language-aware assessments, including concrete actions and steps to guide the implementation. Implementation proposal for language-aware assessments 5 Introduction This implementation proposal is aimed at leadership and management of higher education institutions, research and funding organisations to facilitate implementation of CoARA commitments by recognising and rewarding multilingualism in the practice of science, in scholarly publications and in academic communications, and by addressing language biases in assessments and evaluations. Multilingualism means consideration and use of multiple languages at individual and community level. It includes purposeful and flexible use of an individual's full linguistic repertoire and ability to facilitate understanding and coconstruction of meaning across languages, audiences, and contexts (see Annex I). The proposal has been produced by the Coalition for Advancing Research Assessment (CoARA) Working Group (WG) on Multilingualism and language biases in research assessment. The document presents four proposals for the possible implementation of language-aware assessments, including concrete actions and steps to guide the implementation. The working-group will also provide five annexes to support implementation: I. Glossary of multilingualism for research assessment II. Checklist of multilingualism policies and initiatives III. Toolbox on language-aware assessments IV. Landscape report on Multilingualism and Language Biases in Research Assessment V. Exploring Multilingualism in Research: A Survey to researchers at the National Research Council of Italy Proposal 1: Make multilingualism a value in research assessment • Action 1: set up a committee, or assign the task to an existing body, for promoting language awareness, multilingualism and addressing language bias in research assessment as outlined in this implementation proposal in your organisation (see Annex II and IV) • Action 2: incorporate multilingualism into your organisation’s CoARA Action plan o Step A: review all assessment processes of your organization to identify language requirements and barriers (including legal), adverse incentives and language biases (see Annex III and IV) o Step B: prioritise issues and create a roadmap with schedule for addressing and alleviating the identified language barriers, adverse incentives and language biases (Annex III) Implementation proposal for language-aware assessments 6 o Step C: implement the roadmap to change criteria, methods, data, guidance and support of assessment processes, monitor progress, collect feedback, and revise processes if needed • Action 3: create or update relevant strategies and policies of your organisation o Step A: review international and national policies and initiatives your organisation has signed or adhered to (see see Annex II) o Step B: consult international and national expert bodies, committees or societies on language and multilingualism policies and practices o Step C: review and update (if needed), strategies and policies of your organisation in light of the international and national policies. Proposal 2: Assess and improve language diversity, competences and practices • Action 1: create an overview of active and passive language skills, practices and needs in your organisation o Step A: create an overview of language skills and practices of the research, teaching and administrative personnel (e.g. by means of survey or collecting and analysing research information on publications, see Annex III and V) o Step B: create an overview of diverse contributions for the benefit of science and society, and of languages in which they are communicated o Step C: create an overview of the languages needed to most effectively communicate knowledge and contributions to relevant experts as well as diverse target audiences outside academia • Action 2: identify gaps and priorities for improvement o Step A: identify strengths, weaknesses, opportunities and threats in the language skills and competences, languages used and needed in your organisation o Step B: identify and prioritise areas of improvement and create a roadmap for better supporting the language skills and competences, improving language practices and meeting the language needs of target audiences • Action 3: develop capacity, services and infrastructure to support language skills and multilingualism o Step A: offer workshops, courses, or mentorship programmes to develop multilingual skills and competences o Step B: offer resources for human translation Implementation proposal for language-aware assessments 7 o Step C: point to suitable and secure resources (tools, machine translation, AI-based solutions, etc.) Proposal 3: Create expectations and criteria for multilingualism • Action 1: implement criteria that recognise and reward contributions in any language as well as in many languages (e.g. through translations or adaptations) o Step A: explicitly state that research contributions are recognised irrespective of language and that multilingual communication strategies and translanguaging practices are positively valued o Step B: avoid criteria and indicators that explicitly use or implicate communication language as a proxy of quality, impact or internationality of contributions o Step C: recognise and reward citizen and community engagement, outreach activities and collaboration, public science communication, and media presence in the language(s) of the local/regional/national communities • Action 2: positively value contributions to development of scientific terminology regardless of language • Action 3: recognise as merit or establish awards for promoting multilingualism in research process, outputs (e.g. data, publications), teams and collaboration, conferences and events, teaching and education, community engagement, publication channels (e.g. journals, books), and assessments Proposal 4: Integrate multilingualism into assessment processes • Action 1: support multilingual communication throughout assessment process o Step A: introduce options for researchers to communicate (e.g. submit forms, narrative CV’s, and other documentation), provide and receive feedback in different languages / language of their choice o Step B: introduce options for expert evaluators to use different languages or language of their choice, and provide them training to carry out languageaware assessment (see see Annex III) o Step C: communicate progress, decisions and results of assessment in different languages / language of choice • Action 2: consider language diversity and competences relevant to assessment context when selecting and instructing evaluators Implementation proposal for language-aware assessments 8 o Step A: recruit diverse pools of evaluators who are attentive of language diversity and challenges and are capable of assessing contributions in the relevant languages o Step B: make evaluators aware of language biases and disadvantages for second language users, and explicitly instruct evaluators to focus on the scientific quality rather than the language of communication • Action 3: Make multilingualism integral part of quantitative and qualitative information supporting assessment o Step A: create an overview of information currently used to support assessments and how it represents contributions and facilitates evidence in different languages o Step B: create a plan for transitioning towards more open, linguistically inclusive and multilingual research information (e.g. local CRIS and repositories / global databases OpenAlex and OpenAIRE) o Step C: offer options, templates and guidance for providing qualitative information (e.g. in form of Narrative CV or self-assessment) in different languages / language of choice o Step D: develop multilingual user-interfaces, guidance and metadata for the research information