CoARA Working Group Multilingualism: Webinar on Language-Aware Assessments

PROGRAMME 2 1. CoARA Working Group Martine Garnier-Rizet 2. Landscape Report Gian Maria Greco 3. Implementation proposal Janne Pölönen 4. Questions & Answers WG on multilingualism and language biases in research assessment Martine Garnier-Rizet French National Research Agency (ANR) REFORM OF RESEARCH ASSESSMENT 4 •The Agreement on Reforming Research Assessment (ARRA) published in 2022 has been signed by over 800 organisations from over 50 countries (Europe and beyond) •The Agreement represents a public commitment to the principles, commitments and 5-year timeframe for reforms with action plan •The agreement establishes a common direction for research assessment reform, while respecting organisations’ autonomy •CoARA (Coalition for Advancing Research Assessment) offers a space for signatories to learn from others' experiences, to advance the process of research assessment reform in Europe and beyond MULTILINGUALISM IN ARRA 5 •1st core commitment (scope): “Recognize valuable contributions that researchers make to science and for the benefit of society, including diverse outputs beyond journal publications and irrespective of the language in which they are communicated”. •3rd core commitment (scope): inappropriate uses include assessing outputs based on metrics relating to publication venue, format or language. •6.2 commitment (scope): “Criteria, tools and processes… should enable recognition of… diverse outputs in different languages” •Annex 4 – Toolbox: Value diverse outputs (FAIR data sets, replication studies, registered reports, pre-prints) in different languages in accordance with the Helsinki initiative 3. Promote language diversity in research assessment, evaluation, and funding systems. •Make sure that in the process of expert-based evaluation, high quality research is valued regardless of the publishing language or publication channel •Make sure that when metrics-based systems are utilized, journal and book publications in all languages are adequately taken into account WG MULTILINGUALISM PARTNERS 6 •21 Universities: Coimbra group, AMU, Hanken, Sorbonne, TAU, TOUR4EU, UMI, UHR, UAntwerp, LEI, UNIMIB, UTU, Nanterre, SU, JYU, Lusófona, UAB, Stockholm, Montréal, Lleida, Warwick •5 Research centres and infrastructures: OPERAS, CNR, CSIC, CNRS, NIFU •7 Academies, societies and association of researchers: TSV, MCAA, ECSPM, Eurodoc, EuroScience, ENRESSH, GYA •4 Public or private research funding organisations: ANR, cOAlition S, FWO, RCF •1 Evaluation authorities/agencies: ANVUR •2 Not-for-profit organisations: EASSH, ISE •4 Organisations outside Europe: CLACSO-FOLEC (Argentina), translatE (Australia), UNESCO Chair on Open Science (Canada), Fonds de recherche du Québec (Canada, observer) 44 organisations / 81 experts as of June 2025 Co-chairs •Emanuel Kulczycki (AMU) •Martine Garnier-Rizet (ANR) •Gian Maria Greco (MCAA) Coordination •Janne Pölönen (TSV) WG MISSION 7 •By addressing language diversity and biases in assessment, WG supports the EU (and other) institutions in fulfilling their duty to enhance, promote and uphold linguistic equity, diversity and non-discrimination in Europe and globally. •This requires fostering an academic culture that values diverse competencies, interactions and communications in all languages without exclusions or priorities. The United Nations’ Universal Declaration of Human Rights article 27 states that “everyone has the right freely to participate in the cultural life of the community, to enjoy the arts and to share in scientific advancement and its benefits”, and this according to Article 2 “without distinction of any kind, such as… language”. The Charter of Fundamental Rights of the EU also places an obligation on the Union to respect linguistic diversity (Article 22) and prohibits discrimination on grounds of language (Article 21). European charter for researchers: “Employers and funders of researchers should not discriminate against researchers in any way based on gender, age, ethnic, national or social origin, religion or belief, sexual orientation, language, disability, political opinion, social or economic condition”. WG MAIN OBJECTIVES 8 1. to raise awareness across all fields about the importance of “multilingualism in practice of science, in scientific publications and in academic communications” (UNESCO) 2. to provide institutions with guidelines, toolbox and implementation proposal for recognizing, rewarding and incentivizing research carried out and communicated in all languages, and for addressing language biases in metrics, expert-assessment and rankings Task Forces (TF) 1. TF1: Coordination 2. TF2: Landscape analysis 3. TF3: Policy advice and implementation 4. TF4: Communication Outputs to expect ●Landscape report ●Implementation proposal ●White paper ●Position paper ●Translations UNESCO RECOMMENDATION ON OPEN SCIENCE 9 •Definition: Open science is defined as an inclusive construct that combines various movements and practices aiming to make multilingual scientific knowledge openly available, accessible and reusable for everyone… •Member-state actions: (I) Promoting a common understanding of open science, associated benefits and challenges, as well as diverse paths to open science. (d) Encouraging multilingualism in the practice of science, in scientific publications and in academic communications. Key findings • Researchers report language barriers affecting productivity and well-being, with minimal institutional support for translation or multilingual dissemination. • Institutions acknowledge the problem, but few have integrated multilingualism into evaluation, peer review, or funding policies. • Literature warns of “epistemic monoculture” and calls for balanced multilingualism, a principle articulated in the WG’s shared glossary. Glossary A living glossary of 52 terms with clear definitions relevant to research evaluation. Key term: Balanced multilingualism “considering all the communication purposes in all different areas of research, and all the languages needed to fulfill these purposes, in a holistic manner without exclusions or priorities” Comprehensive Literature Review A wide literature review on multilingualism in research assessment. Five main areas in the literature •Benefits of multilingualism: Inclusivity, bibliodiversity, societal impact. •English dominance and biases: Inequalities, epistemic monoculture, invisibility. •Regional and disciplinary differences: Varied practices across fields and regions. •Policy recommendations: Reward non-English outputs, adapt metrics, narrative CVs. •Technology and AI: Machine translation, inclusion, complement to policy. Survey of Marie Curie Alumni Association members •~27% of the Marie Curie Alumni Association (MCAA) surveyed members agreed that language barriers “disrupted their workflow,” and roughly 33% felt a negative impact on their well-being. •~95% of respondents use English for peer-reviewed publications, and 94% for scholarly presentations •~49% of respondents relied on professional translation services, or automated tools or help from colleagues in publishing or communicating in English for a scholarly audience Italian National Research Council (CNR) Survey •910 respondents (13% response rate). •CNR scientists almost all work in multilingual settings. In Social Sciences and Humanities (SSH), most researchers balance Italian with other languages, reflecting local cultural engagement and global dialogue. In STEM and life sciences, English is overwhelmingly dominant for publishing and collaborations. •Translation is common in SSH: 36% of SSH respondents had their work translated. Hungarian Research Network (HUN-REN) Survey •211 responses (6% of ~3,500 researchers). •Nearly all HUN-REN researchers (95.8%) use Hungarian and other languages in their work. •English overwhelmingly dominates in publications: 40% of respondents publish entirely in English, and another 29.7% publish 90% in English. Survey of CoARA signatory organizations •81 CoARA members from 30 countries replied (14.3% of those invited). •Nearly 60% said their institution considers multilingualism “to some extent” in current assessment policies , only 8 said “to a large extent,” and about 13 said “not so much.” •61 institutions answered “No” to favoring certain languages in evaluation (i.e. they profess not to limit outputs by language). +Bibliometric analysis of global research output ● WEB OF SCIENCE ○ English: 93 % ○ Other: 7 % ● SCOPUS ○ English: 88 % ○ Other: 12 % ● OPENALEX ○ English: 75 % - 67 %* ○ Other: 25 % - 33 %* Brasil, A. (2024). Exploring the landscape of multilingualism through bibliometrics, CoARA WG Webinar 26 Sep 2024 *Céspedes, L. et al. (2025). Evaluating the Linguistic Coverage of OpenAlex, JASIST Implementation Proposal for Language-aware Assessments Janne Pölönen Federation of Finnish Learned Societies (TSV) INTRODUCTION 25 ●Implementation proposal is aimed at leadership and management of higher education institutions, research and funding organisations to facilitate implementation of CoARA commitments. ●The document presents four proposals for the possible implementation of language-aware assessments, including concrete actions and steps to guide the implementation Five annexes to support the implementation proposal ○I Glossary of multilingualism for research assessment ○II Checklist of multilingualism policies and initiatives ○III Toolbox on language-aware assessments ○IV Landscape report on Multilingualism and Language Biases in Research Assessment ○V Exploring Multilingualism in Research: A Survey to researchers at the National Research Council of Italy ANNEX II: EUROPEAN CHARTER FOR RESEARCHERS 32 ANNEX V: EXPLORING MULTILINGUALISM IN RESEARCH 33 ●A Survey to researchers at the National Research Council of Italy ●Full questionnaire for conducting an institutional survey on language skills,, practices and attitudes towards multilingualism ●The survey has been conducted in Hungarian by HUN-REN Questions & Answers Working Group on Multilingualism and Language Biases in Research Assessments OPEN CONSULTATION ON THE IMPLEMENTATION PROPOSAL FOR LANGUAGE-AWARE ASSESSMENTS PROVIDE FEEDBACK UNTIL 6 NOVEMBER 2025