INFORMATICS RESEARCH EVALUATION,
2025 REVISED REPORT
An In o ma ics Eu ope epo
endo sed by Na ional In o ma ics Associa ion membe s o
In o ma ics Eu ope
In o ma ics Resea ch E alua ion,
2025 Re ised Repo
An In o ma ics Eu ope Repo
P epa ed by he Resea ch E alua ion Recommenda ions Panel o In o ma ics Eu ope. Endo sed by
Na ional In o ma ics Associa ions o Aus ia, F ance, Ge many, I aly, Ne he lands, Spain, Swi ze land
and Uni ed Kingdom.
Edi o s:
● Ca los Baque o, Uni e sidade do Po o and INESC TEC, Po ugal
● Manuel Ca o, Uni e sidad Poli écnica de Mad id and IMDEA So wa e Ins i u e, Spain, and
In o ma ics Eu ope
● Ma in Hi z, Alpen-Ad ia-Uni e si ä Klagen u and In o ma ik Aus ia
● Pekka O ponen, Aal o Uni e si y, Finland, and In o ma ics Eu ope
● Sil io Pe oni, Uni e si y o Bologna and GRIN – G uppo di In o ma ica, I aly
● Pe e Rossmani h, RWTH Aachen Uni e si y and Fakul ä en ag In o ma ik, Ge many
Ad ise s:
● Michael Goedicke, Uni e si ä Duisbu g-Essen and Gesellscha ü In o ma ik, Ge many
● En ico Na delli, Uni e si à di Roma “To Ve ga a”, I aly, and In o ma ics Eu ope
● Pie e Pa adinas, Conse a oi e Na ional des A s e Mé ie s and Socié é In o ma ique de
F ance
● E nes Tenien e, Uni e si a Poli ècnica de Ca alunya and Sociedad Cien i ica In o má ica de
España, Spain
● Heike T au mann, Pade bo n Uni e si y and Fakul ä en ag In o ma ik, Ge many
© In o ma ics Eu ope, 2025, CC BY-SA 4.0|Page 1
In o ma ics Resea ch E alua ion, Re ised Repo
Ma ch 2025
Published by:
In o ma ics Eu ope
Binzmhles asse 14/54
8050 Zu ich, Swi ze land
www.in o ma ics-eu ope.o g
adminis a ion@in o ma ics-eu ope.o g
© In o ma ics Eu ope, 2025, CC BY-SA 4.0
O he In o ma ics Eu ope Repo s
● Su ey abou Di e si y and Inclusion Ini ia i es (2024, Elisabe a Di Ni o, An inisca Di Ma co and
he In o ma ics Eu ope Di e si y & Inclusion Wo king G oup)
● P oceedings o he 1s Ea ly Ca ee Resea che s Wo kshop a ECSS 2021 (2021, Elisabe a Di Ni o
and S anda Ži ný)
● B idging he Digi al Talen Gap: Towa ds Success ul Indus y-Uni e si y Pa ne ships (2020, En ico
Na delli, C is ina Pe ei a and appo eu s in A i icial In elligence, Cybe Secu i y and So wa e
Enginee ing, wi h he suppo o he Eu opean Commission, DG CONNECT)
● In o ma ics Educa ion in Eu ope: Ins i u ions, Deg ees, S uden s, Posi ions, Sala ies — Key Da a
2013-2018 (2019, S e lana Tikhonenko, C is ina Pe ei a)
● E hical/Social Impac o In o ma ics as a S udy Subjec in In o ma ics Uni e si y Deg ee P og ams.
(2019, Paola Mello, En ico Na delli, wi h con ibu ion om he Wo king G oup Membe s)
● The Wide Role o In o ma ics a Uni e si ies. (2019, Elisabe a Di Ni o, Susan Eisenbach,
Inmaculada Ga cía Fe nández, Edua d G ölle )
● Indus y Funding o Academic Resea ch in In o ma ics in Eu ope. Pilo S udy. (2018, Da a
Collec ion and Repo ing Wo king G oup o In o ma ics Eu ope)
● In o ma ics Educa ion in Eu ope: Ins i u ions, Deg ees, S uden s, Posi ions, Sala ies — Key Da a
2012-2017 (2018, S e lana Tikhonenko, C is ina Pe ei a)
● In o ma ics Resea ch E alua ion (2018, Resea ch E alua ion Wo king G oup o In o ma ics Eu ope)
● In o ma ics o All: The S a egy (2018, Michael E. Caspe sen, Judi h Gal-Eze , And ew McGe ick,
En ico Na delli. Join epo wi h ACM Eu ope)
● When Compu e s Decide: Recommenda ions on Machine-Lea ned Au oma ed Decision Making
(2018, James La us, Ch is Hankin, Si i G anum Ca son, Ma kus Ch is en, Sil ia C a a, Oli e G au,
Claude Ki chne , B an Knowles, And ew McGe ick, Damian And ew Tambu i, Hannes We hne
Join Repo wi h ACM Eu ope)
● In o ma ics Educa ion in Eu ope: A e We All In The Same Boa ? (2017, The Commi ee on
Eu opean Compu ing Educa ion. Join epo wi h ACM Eu ope)
All epo s can be ob ained om In o ma ics Eu ope a :
www.in o ma ics-eu ope.o g
© In o ma ics Eu ope, 2025, CC BY-SA 4.0|Page 2
Execu i e Summa y
E alua ion is an indispensable ins umen o imp o ing esea ch quali y and impac . To achie e he in ended
e ec s, esea ch e alua ion should ollow es ablished and widely accep ed p inciples, be benchma ked agains
app op ia e c i e ia, and be sensi i e o disciplina y di e ences. This epo add esses he p inciples and c i e ia
ha a e o be ollowed when indi idual esea che s a e e alua ed o hei esea ch in he ield o In o ma ics.
This epo builds on and upda es he ou comes o he 2008 and 2018 In o ma ics Eu ope epo s on Resea ch
E alua ion o Compu e Science/In o ma ics, while aligning i s ecommenda ions wi h o he ecen documen s on
esea ch e alua ion, mos no ably he CoARA Ag eemen on Re o ming Resea ch Assessmen (CoARA 2022). The
epo also con ains upda ed analyses and ecommenda ions in ou opical a eas o conce n in In o ma ics: he
esponsible use o bibliome ics and c edi assignmen in con ibu ions, assessing a e ac s, Open Science, and
in e disciplina y esea ch, oge he wi h a discussion on he ole o AI in esea ch e alua ion.
Ou key messages a e he ollowing:
1. In o ma ics is an o iginal discipline ha combines aspec s o ma hema ics, science, and enginee ing.
Resea che e alua ion mus ecognise and espec i s speci ici y.
2. A dis inc i e ea u e o publica ion in In o ma ics is he impo ance o highly selec i e con e ences.
Jou nals ha e complemen a y ad an ages bu do no necessa ily ca y mo e p es ige. Publica ion models
ha couple con e ences and jou nals, whe e he pape s o a con e ence a e published di ec ly in a jou nal,
a e a g owing end ha may b idge he cu en gap be ween hese wo o ms o publishing.
3. Open a chi es and o e lay jou nals a e ecen inno a ions in he In o ma ics publica ion cul u e ha o e
imp o ed acking in e alua ion.
4. The impac o a e ac s such as so wa e, open da ase s, and o he esea ch p oduc s such as ained
machine lea ning models can be as g ea as publica ions. The e alua ion o such objec s, which is now
conduc ed by many con e ences, should be encou aged and accep ed as an es ablished componen o
esea ch assessmen . Ano he impo an indica o o impac is ad ances ha lead o comme cial
exploi a ion o adop ion by indus y o s anda disa ion bodies.
5. Open Science and i s esea ch e alua ion p ac ices a e highly ele an o In o ma ics. In o ma ics has
played a key enabling ole in he Open Science e olu ion and should emain a i s o e on .
6. Nume ical measu emen s such as ci a ion and publica ion coun s mus ne e be used as he sole
e alua ion ins umen . They mus be il e ed h ough human in e p e a ion, speci ically o a oid e o s,
and complemen ed by pee e iew and assessmen o ou pu s o he han publica ions. In pa icula ,
nume ical measu emen s mus no be used o compa e esea che s ac oss scien i ic disciplines, including
ac oss sub ields o In o ma ics.
7. In In o ma ics, he o de o au ho s o en holds li le signi icance and a ies ac oss sub ields. Wi hou clea
guidelines, i should no be a ac o in esea che e alua ion. Ins ead, au ho s should be encou aged o
clea ly s a e he scope and ole o hei indi idual con ibu ions o mul i-au ho wo ks.
8. In assessing ins i u ions, esea che s, publica ions, and ci a ions, he use o open esea ch in o ma ion
p o ided by Open Science in as uc u es should be a ou ed and suppo ed. When using anking and
benchma king se ices p o ided by o -p o i companies, espec o open access c i e ia is manda o y.
Jou nal-based o jou nal-biased anking se ices a e inadequa e o mos o In o ma ics and mus no be
used.
9. Any e alua ion, especially quan i a i e, mus be based on clea , published c i e ia. Fu he mo e,
assessmen c i e ia mus hemsel es unde go assessmen and e ision.
10. Any use o gene a i e AI in esea ch e alua ion should inc ease he quali y o he assessmen s and educe
he e o o he human e alua o s. AI mus no be used o educe he numbe o human expe s in
assessmen panels and hei collec i e esponsibili y o he panels’ ecommenda ions.
© In o ma ics Eu ope, 2025, CC BY-SA 4.0|Page 3
Table o Con en s
1. Resea ch E alua ion ................................................................................................................................. 4
2. In o ma ics and I s Speci ici y .................................................................................................................. 5
2.1 Cha ac e is ics o In o ma ics ............................................................................................................ 5
2.2 The In o ma ics publica ion cul u e and i s e olu ion....................................................................... 5
3. Resea ch E alua ion o Inc eased Quali y and Impac ........................................................................... 6
3.1 Assessing he quali y o esea ch....................................................................................................... 6
O he indica o s o quali y ...................................................................................................................... 7
3.2 Assessing he impac o esea ch....................................................................................................... 7
4. Responsible Use o Indica o s and C edi Assignmen in Con ibu ions ................................................. 8
5. Assessing A e ac s .................................................................................................................................. 9
5.1 So wa e a e ac s ............................................................................................................................ 10
5.2 Why e alua e so wa e sys ems ...................................................................................................... 10
5.3 Ca ea s ............................................................................................................................................. 11
5.4 Recommenda ions ........................................................................................................................... 12
6. Open Science .......................................................................................................................................... 13
7. In e disciplina y Resea ch ...................................................................................................................... 15
8. The Role o AI in Resea ch E alua ions .................................................................................................. 16
9. Conclusions ............................................................................................................................................ 17
Acknowledgemen s.................................................................................................................................... 17
Endo semen s ............................................................................................................................................ 17
Re e ences ................................................................................................................................................. 18
© In o ma ics Eu ope, 2025, CC BY-SA 4.0|Page 4
1. Resea ch E alua ion
E alua ion is inna ely linked wi h esea ch. The wo k o esea che s is subjec o e alua ion om he e y
beginning o hei ca ee s. Resea ch esul s submi ed o assessmen a e subjec o a pee e iew
p ocess ha sc u inises hei scien i ic quali y and po en ial o impac . The esea che s hemsel es a e
cons an ly being e alua ed: o ec ui men , o p omo ion, o unding o esea ch p oposals, when
being conside ed o speci ic oles in commi ees, o ecogni ion by awa ds, e c. Resea che s also o en
ac as e alua o s o hei pee s.
By and la ge, hese e alua ion p ocesses a e in e nal o he esea ch communi y and aim o gua an ee
i s in e nal ai ness and in eg i y h ough sel - egula ion. Inc easingly, e alua ion is also manda ed and
egula ed by exogenous en i ies and scaled om indi iduals o en i e ins i u ions. In many cases,
go e nmen s ha e de eloped na ional e alua ion s anda ds and p ocesses. O en, hei end esul s a e
ins i u ional ankings. The main mo i a ion o hese e o s is o gua an ee ha axpaye s’ money is
spen on esea ch e icien ly and leads o socie al bene i s.
Resea ch e alua ion can hus be pe o med o di e en goals. I can a ge a speci ic piece o esea ch
(documen ed by a single o se e al a e ac s), an indi idual esea che , a esea ch g oup, o an
o ganisa ional uni (e.g., a depa men ). I may e en gene alise o en i e ins i u ions (e.g., uni e si ies)
o e en coun ies. In any case, he goal o e alua ion is o assess some explici o implici no ion o alue,
o quali y, o esea ch. To achie e he posi i e objec i es o esea ch e alua ion, he speci ic goals o any
such e alua ion e o mus be clea ly s a ed, and he way i is conduc ed mus be aligned wi h hese
goals. The e alua ion mus ollow es ablished p inciples and p ac ical c i e ia, known and sha ed by
e alua o s and esea che s, and ake in o accoun any speci ici ies o he scien i ic ield and a ea
in ol ed.
E alua ion can ha e a emendously posi i e e ec in imp o ing esea ch quali y and p oduc i i y. I is
i al o ecognise and suppo esea ch ha can lead o ad ances in knowledge and impac on socie y.
A he same ime, he e ec o ollowing ill-concei ed c i e ia o p ac ices in esea ch e alua ion, o
misusing me ics and indica o s, can ha e se iously nega i e long- e m e ec s. In pa icula , i may
g ea ly damage he po en ial o u u e gene a ions o esea che s. Fu he mo e, e y equen
e alua ion can ha e a de imen al e ec on undamen al esea ch, as he assessmen may hen
accen ua e low- isk, sho - e m de elopmen s a he expense o po en ially high-gain, long- e m wo k.
This epo ocuses on he main p inciples and c i e ia ha should be ollowed when indi idual
esea che s
1
a e e alua ed o hei esea ch ac i i y in he ield o In o ma ics,
2
add essing he
speci ici ies o his a ea. This subsumes he e alua ion o a speci ic piece o esea ch and can, o some
ex en , be gene alised o depa men s since hei esea ch pe o mance is la gely de e mined by hei
indi iduals.
This epo easse s he ecommenda ions in p e ious In o ma ics Eu ope epo s on he subjec
(In o ma ics Eu ope 2008, 2018), o which i inco po a es a numbe o obse a ions conce ning he
opical a eas o bibliome ics and c edi assignmen in con ibu ions, assessing a e ac s, Open Science,
in e disciplina y esea ch, and he ole o AI in esea ch e alua ion.
An impo an de elopmen since he publica ion o he 2018 epo has been he con e gence o se e al
epo s and ini ia i es on e ising esea ch e alua ion p ac ices in o he CoARA Ag eemen on Re o ming
Resea ch Assessmen (CoARA 2022). The ecommenda ions in his epo ha e been e iewed and
upda ed o align wi h he CoARA ecommenda ions.
1
Some aspec s o depa men e alua ion a e add essed in he publica ion (In o ma ics Eu ope 2013).
2
In e changeably “Compu e Science” o “Compu ing”.
© In o ma ics Eu ope, 2025, CC BY-SA 4.0|Page 5
2. In o ma ics and I s Speci ici y
2.1 Cha ac e is ics o In o ma ics
In o ma ics is a ela i ely young science ha is apidly e ol ing in close connec ion wi h echnology.
Beyond he wo basic and uni e sal esea ch pilla s o heo y and expe imen a ion, which in In o ma ics
esea ch a e o en bo h p esen in a ying p opo ions, a hi d impo an componen in In o ma ics is he
c ea ion o a e ac s, which p o ide new designs, ools o o he wise imp o ed suppo o in o ma ion
p ocessing asks.
In o ma ics esea ch co e s an ex emely wide and me hodologically di e se amily o a eas: om he
de elopmen o new compu ing de ices o he ma hema ical heo y o algo i hms and complexi y, om
human ac o s o big da a, machine lea ning, and a i icial in elligence, om s udies o p og amme s’
p oduc i i y o secu e enc yp ion me hods. In e disciplina i y, as in he case o bioin o ma ics, medical
in o ma ics, geo-in o ma ics, o cogni i e science, b ings in e en mo e di e si y.
Wi h he con inuing digi alisa ion o socie y and he eme gence o po en ially dis up i e compu a ional
echnologies such as a i icial in elligence and quan um compu ing, In o ma ics has an ex emely high
socie al and economic impac . Measu ing up o hese esponsibili ies no only equi es con inuous
p og ess in In o ma ics esea ch, bu also calls o mo e emphasis on In o ma ics educa ion and he
in luence o In o ma ics on socie y, including e hical conce ns.
Resea ch in In o ma ics, as in any o he science, mus be e alua ed acco ding o c i e ia ha ake in o
accoun he ield’s speci ici y. Uni e sal c i e ia do no exis o e alua ing esea ch quali y. This is also
ue o he di e en sub ields o In o ma ics. Di e ences be ween ields mus be aken in o accoun ,
and he emp a ion o adop simplis ic, one-size- i s-all c i e ia should be esis ed.
2.2 The In o ma ics publica ion cul u e and i s e olu ion
The publica ion cul u e wi hin In o ma ics di e s om mos o he ields in he p ominen ole played by
con e ence publica ions. Many sub ields o In o ma ics ha e leading con e ences wi h s a us, isibili y,
and impac compa able o o highe han hei espec i e leading jou nals. Con e ence pape s a hese
mee ings unde go a highly selec i e pee - e iew p ocess ha makes hese enues e y compe i i e and
o en leads o lowe accep ance a es han in he bes jou nals. Con e ences in In o ma ics p o ide a
as e u na ound ime han jou nals o publish esea ch esul s, ge eedback om pee s and build upon
i , and also o en ha e highe s anda ds o no el y han jou nals. These ac o s a e c ucial in a apidly
e ol ing ield like In o ma ics, and as a esul , in In o ma ics, jou nals do no necessa ily ca y mo e
p es ige han con e ences. Jou nal publica ions a e, o cou se, also impo an , especially o ga he ing
and ewo king an eceden esea ch in o an a chi al p esen a ion ha is ca e ully p epa ed, has
gua an eed long- e m a ailabili y, and is ee o he space limi a ions imposed by con e ences.
Towa ds b idging he dicho omy be ween con e ences and jou nals, new al e na i es a e now
changing he publica ion cul u e:
● Coupled con e ences and jou nals, implemen ed in di e en ways: VLDB-s yle wi h con inuous
submission o he jou nal and p esen a ion o he accep ed pape s a he con e ence; ICLP-s yle
wi h he p oceedings o he con e ence being published as a special issue o a s anda d jou nal
(in his case TPLP); and i s a ian PACMPL-s yle,
3
whe e a dedica ed jou nal is used o publish
3
VLDB: "In e na ional Jou nal on Ve y La ge Da abases"; ICLP: "In e na ional Con e ence on Logic P og amming"; TPLP:
"Jou nal on Theo y and P ac ice o Logic P og amming"; PACMPL: "P oceedings o he ACM on P og amming Languages".
© In o ma ics Eu ope, 2025, CC BY-SA 4.0|Page 6
p oceedings o se e al di e en , ela ed con e ences. These hyb id combina ions o con e ences
and jou nals a e a p omising and g owing end ha combines he ad an ages o imely
publica ion o con e ences wi h he impac acking o jou nals (Dags uhl 2012).
● Open A chi es (a Xi , HAL, Zenodo e c.) p o ide oppo uni ies o publish i s e sions o pape s
and p o ec in ellec ual p ope y o new esul s, a he same ime gi ing online access o all p oo s
and ma e ials, including da a and so wa e, ha sus ain hese esul s. A e possible eedback
om pee s, imp o ed e sions can be submi ed o o e lay jou nals acco ding o hei publica ion
cons ain s. In his model, e iewe s ha e access o he comple e his o y o he esul s and can
be e e alua e hei quali y and impac .
Books emain a speci ic and impo an o ma o p o iding a comp ehensi e iew o a gi en opic and
o con ibu ing o educa ion. He e again, p epublica ion in an open a chi e can help in sha ing d a s,
ge ing eedback om pee s be o e o icial publica ion, managing e sions, e c.
Ano he speci ici y o he In o ma ics publishing cul u e is ha unlike in o he sciences, such as Physics
o Medicine, In o ma ics does no ha e a gene ally adop ed con en ion o in e p e ing he o de in
which he au ho s o a publica ion a e lis ed, and does no especially dis inguish he “co esponding
au ho (s)”. Thus, in he absence o speci ic indica ions, au ho o de should no se e as a ac o in
indi idual esea che s’ e alua ions. This issue is discussed mo e ex ensi ely in Sec ion 4 o his epo .
3. Resea ch E alua ion o Inc eased Quali y and Impac
The undamen al goal o esea ch e alua ion is o assess he quali y and impac o esea ch o he
e en ual imp o emen o bo h. Quali y is an elusi e in insic cha ac e is ic o which a commonly
accep ed assessmen me hod, e en i impe ec , is pee e iew by a panel o in o med expe s. Impac is
an obse able ex e nal cha ac e is ic ha akes many o ms and can o some ex en be measu ed by
nume ical indica o s, bu e en hen, only wi h human expe in e p e a ion. Quali y is mos ly a good
p edic o o impac , and impac is mos ly a good indica o o quali y, bu he wo a e no coex ensi e.
The CoARA ag eemen ecommends o " ocus esea ch assessmen c i e ia on quali y [and] ecognise he
con ibu ions ha ad ance knowledge and he (po en ial) impac o esea ch esul s" (CoARA 2022, p.
3). Assessing impac can be qui e di icul , and is o en e en in easible in a sho ime ame, because he
impac o a body o esea ch can be indi ec and i may ake yea s be o e he e en ual impac can be
ecognised.
3.1 Assessing he quali y o esea ch
As ecognised by he CoARA ag eemen , “ esea ch assessmen should ely p ima ily on quali a i e
assessmen o which pee e iew is cen al, suppo ed by esponsibly used quan i a i e indica o s whe e
app op ia e” and “i is impo an ha pee e iew p ocesses a e designed o mee he undamen al
p inciples o igou and anspa ency” (CoARA 2022, p. 5).
Wi h he inc easing a ailabili y o publicly a ailable bibliome ic da a, esea ch assessmen s ha e been
eso ing mo e and mo e o bibliome ic indica o s p o ided by a a ie y o sou ces, o en e en wi hou
ques ioning he sou ces’ soundness and us abili y. While we acknowledge he use ulness o
quan i a i e da a and bibliome ic indica o s when used esponsibly (see Sec ion 4), we emphasise he
ollowing:
© In o ma ics Eu ope, 2025, CC BY-SA 4.0|Page 7
Resea ch assessmen should no ake quan i y as a p oxy o quali y and impac (c . F iedman &
Schneide 2015). Any policy ha ends o a ou quan i y o e quali y has po en ially dis up i e e ec s
and can mislead young esea che s wi h e y nega i e long- e m e ec s. Such policies can lead
esea che s o ocus on publishing he leas publishable inc emen s, and in his espec , some Eu opean
coun ies ha e es ablished na ional e alua ion p ac ices wi h indica o s ha aise se ious conce ns. To
s ess he impo ance o quali y and impac , esea che e alua ions should p e e ably ocus on a
ela i ely small numbe o high-quali y publica ions and a e ac s, ying o iden i y also hei impac
ca ie s such as no el y, suppo ing a e ac s, and in luence on o he esea che s’ wo k.
Quan i a i e da a and bibliome ic indica o s mus be in e p e ed in he speci ic con ex o he
esea ch being e alua ed. They should ne e cons i u e he sole e alua ion c i e ion. Di e en esea ch
a eas and e en sub ields o In o ma ics ha e e y di e en cha ac e is ics. E en wi hin a homogeneous
se , bibliome ic indica o s only p o ide a e y coa se assessmen . Fo example, al hough a e y high
numbe o ci a ions may indica e a po en ially impac ul piece o wo k, i may also indica e a widely
e e enced su ey a he han a no el o iginal con ibu ion (which, o cou se, also has i s own alue).
Human app aisal wi h espec o es ablished c i e ia is needed o in e p e da a and disce n quali y
and impac . Numbe s can only help; hey a e no a subs i u e.
In addi ion o being es ablished, known, and sha ed by e alua o s and esea che s, assessmen c i e ia
mus hemsel es unde go assessmen and e ision in o de o ollow he e olu ion o science.
O he indica o s o quali y
Majo con e ences and schola ly socie ies in In o ma ics o en g an “bes pape awa ds” o con ibu ions
pe cei ed o ha e he highes alue o hose accep ed a a con e ence. A u he de elopmen is he
g an ing o “mos in luen ial pape ” o “ es -o - ime” awa ds o con ibu ions pe cei ed o ha e had he
mos in luence in he a ea o e some pas numbe o yea s. These dis inguishing, pee - e iewed awa ds
should be aken in o accoun in e alua ions.
3.2 Assessing he impac o esea ch
The impac o esea ch can be assessed along many di e en dimensions. Fi s , one can dis inguish
be ween ex e nal and communi y impac . The ex e nal impac o esea ch conce ns i s e ec on socie y
a la ge. The new GenAI echniques, o example, will po en ially ha e an eno mous impac on socie y.
Likewise, he in en ion o a new secu e p o ocol may lead o he de elopmen o mo e secu e ne wo ks
on which socie y elies, and a new au oma ed de elopmen en i onmen can imp o e he indus y’s
p oduc i i y. Ad ances ha lead o comme cial exploi a ion, spin-o ac i i ies, o adop ion by indus y
(e.g., so wa e licenses) o s anda disa ion bodies, a e some highly alued indica o s o ex e nal impac .
Communi y impac , on he o he hand, e e s o he impac o one’s esea ch on o he esea che s. This
means ha o he esea che s can build on op o one’s esea ch. E alua ion o en ies o cap u e his
kind o impac by, e.g., ci a ion indices o esea ch so wa e download coun s. A di e en kind o
indica o could be p o ided by so wa e compe i ions ha a e egula ly o ganised o assess p og ess in
ools (e.g., SAT sol e s o lea ning ools). Such compe i ions a e ypically highly app ecia ed and
con ibu e o inc easing he quali y o he ools, and success in hem could be aken as an indica o , o a
leas a p edic o , o communi y impac .
An o hogonal dimension o impac conce ns he ou comes o esea ch. An ou come can ha e an impac
because i ad ances knowledge in a gi en a ea o because i ad ances p ac ice. New knowledge can be
c ea ed by heo e ical esea ch, simila o ma hema ics. A ypical example is esea ch on algo i hms and
compu a ional complexi y. New knowledge can also be gene a ed by esea ch ha pu sues empi ical
© In o ma ics Eu ope, 2025, CC BY-SA 4.0|Page 14
esea ch assessmen p ac ices o align hem wi h he p inciples o Open Science. Indeed, he o e all
di ec ion sugges ed is o build assessmen p ocesses on exis ing and well-known p ac ices and guidelines
– such as he San F ancisco Decla a ion on Resea ch Assessmen (DORA 2012), he Leiden Mani es o (Hick
e al. 2015), and he Coali ion o Ad ancing Resea ch Assessmen (CoARA 2022). The main ac ion poin s
o hese guidelines a e:
● A oid using me ics de eloped o measu ing one kind o en i y (e.g. Jou nal Impac Fac o o
jou nals) as a p oxy measu e o assessing o he kinds o en i ies (e.g. esea che s).
● Make explici he c i e ia used in he assessmen , and p e e pee - e iew e alua ion, suppo ed
by quan i a i e indica o s when app op ia e – as discussed in de ail ea lie in Sec ions 4 and 5.
● Recognise he di e si y o esea ch con ibu ions in assessmen exe cises (da ase s, da abases,
so wa e, and o he a e ac s), in addi ion o classic p in -like publica ions (jou nal a icles, books,
book chap e s, e c.) – as la gely discussed in Sec ion 5.
● Apply openness and anspa ency when p o iding da a and me hods used in he assessmen
exe cise.
● Enable anyone o e i y bo h he da a and he analysis.
● Figh agains manipula ions and inco ec uses o me ics – as an icipa ed in Sec ion 4.
● Conside he a iabili y o di e en ypes o esea ch ou pu and subjec a eas when compa ing
an en i y agains ano he en i y (e.g. esea che s).
● Focus on quali a i e judgemen o esea ch ou pu s when assessing esea che s – as discussed in
Sec ion 4.
Se e al o hese ac ions a e no implemen able wi hou he a ailabili y o app op ia e Open Science
in as uc u es such as open bibliog aphic and ci a ion da abases and me ada a eposi o ies, ins i u ional
esea ch in o ma ion sys ems, and open bibliome ics and scien ome ics sys ems o assessing and
analysing scien i ic domains. Indeed, UNESCO insis s ha in es ing in Open Science in as uc u es and
se ices is essen ial, and ha he schola ly communi y should e ain con ol and owne ship o e hese
in as uc u es. They a e a c ucial means o p o iding he da a, i.e., esea ch in o ma ion used o
de ising me ics and indica o s ha may be used o suppo pee - e iew assessmen .
Resea ch in o ma ion e e s o all he me ada a ela ed o he conduc and communica ion o esea ch,
such as bibliog aphic me ada a o esea ch ou comes (a icles, so wa e, da ase s, me hodologies, e c.),
in o ma ion on unding and g an s, and in o ma ion on o ganisa ions and esea ch con ibu o s. Se e al
ini ia i es, such as he Ba celona Decla a ion (DORI 2024), and se e al Open Science in as uc u es push
o making esea ch in o ma ion openly a ailable and eely eusable o he schola ly communi y
because a ious ac i i ies, including esea ch assessmen , a e cha ac e ised by using such in o ma ion.
Among he ele an in as uc u es, OpenCi a ions (h ps://openci a ions.ne ), OpenAIRE
(h ps://openai e.eu), So wa e He i age (h ps://www.so wa ehe i age.o g), and DBLP
(h ps://dblp.o g) also p o ide in o ma ion abou so wa e and o he In o ma ics- ela ed a e ac s in
addi ion o classic publica ions in jou nals and con e ence p oceedings. O he in as uc u es, such as
PRE e iew (h ps://p e e iew.o g), help in add essing ela ed assessmen asks in a e y anspa en
way, e.g. by enabling open pee e iew p ac ices. These open e iewing p ocesses pe mi he disclosu e
o he iden i y o he e iewe s, he publica ion o e iews, and hus, he ecogni ion o he e o ha
schola s pu in o e iewing, and he possibili y o a b oade communi y o p o ide commen s and
pa icipa e in he assessmen p ocess (UNESCO 2021).
Suppo ing hese kinds o Open Science and communi y-guided in as uc u es is key o main aining an
en i onmen ha enables anspa en assessmen wo k lows. Indeed, he e is an u gen need o make
esea ch in o ma ion o ad ance esponsible esea ch assessmen and ope a ionalise Open Science and
p omo e unbiased, anspa en , and high-quali y decision-making. The Open Science Ca ee Assessmen
Ma ix (OS-CAM) (Eu opean Commission 2017) ep esen s a possible, p ac ical mo e owa d a mo e
comp ehensi e app oach o e alua ing esea che s h ough he lens o Open Science. In addi ion, o he
© In o ma ics Eu ope, 2025, CC BY-SA 4.0|Page 15
empla es o Open Science assessmen ha in ol e In o ma ics as a domain o applica ion ha e been
s udied in he con ex o speci ic esea ch p ojec s, such as G aspOS (h ps://g aspos.eu/).
In o ma ics should acknowledge Open Science p ac ices in i s esea ch e alua ion. In o ma ics hus also
has a p ominen ole o play in he adop ion and de elopmen o he Open Science app oaches and
in as uc u es, and i s suppo is key o keep hem sus ainable in he long e m.
R6.1 Acknowledge, suppo , and adop Open Science p ac ices in In o ma ics.
R6.2 Apply p inciples de i ed om Open Science p ac ices in he e alua ion o esea ch.
R6.3 S i e o keep he con ol o Open Science in as uc u es and p ocesses wi hin he esea ch,
academic, and educa ional In o ma ics communi ies.
7. In e disciplina y Resea ch
In o ma ics has o a long ime been an impo an suppo p o ide o esea ch in o he ields, in he
o m o so wa e ools o modelling, op imisa ion and isualisa ion, da a managemen and analysis, e c.
In an in e disciplina y collabo a ion his kind o wo k is o en conside ed mundane by he companion
a ea pa ne s, and no eally compu ing esea ch by he In o ma ics communi y. And indeed, o ins ance
simply applying a known analysis me hod o a new da ase wi hou a clea scien i ic ad ance on he
In o ma ics side is jus a echnical suppo ask.
The e a e howe e also mo e ambi ious collabo a ions, whe e he In o ma ics con ibu ion is an in eg al
elemen o he esea ch agenda, and pu suing i equi es bo h compe ence in he companion a ea and
an abili y o adap exis ing and de elop new me hods in In o ma ics o he needs o he speci ic
collabo a ion. A high-p o ile example o he la e kind o collabo a ion is e.g. he wo k on compu a ional
p o ein design ha was awa ded one o he 2024 Nobel P izes in Chemis y. E en in such cases, howe e ,
c edi is commonly assigned along disciplina y lines, so ha ecogni ion goes p ima ily o he companion
a ea, and also on he In o ma ics side he compu a ional me hods con ibu ion is conside ed as “an
applica ion”.
An added challenge is ha in assessmen s ha a e based mechanically on publica ion enues and/o
indica o s, In o ma ics con ibu ions o o he a eas han co e In o ma ics a e easily o e looked, because
he ele an enues a e ei he no indexed a all in In o ma ics da abases, o a e conside ed as
“applica ion a ea ou le s”, no ma e how p es igious in he companion a ea.
The challenge o o e coming disciplina y ba ie s in he assessmen o in eg a i e in e disciplina y
esea ch is a widely ecognised p oblem (e.g. McLeish & S ang 2016). Some key ecommenda ions o
he esponsible e alua ion o In o ma ics esea ch in his con ex a e:
R7.1 Recognise he alue o in eg a ed in e disciplina y esea ch in i s own e ms, no as an
“applica ion” o In o ma ics.
R7.2 Assess (i) he dep h o he in eg a ion and (ii) he no el y and signi icance o he In o ma ics
con ibu ion o he o ali y o he wo k.
R7.3 Be wa y o nume ical indica o s and “ op enue” lis s o ien ed owa ds assessing In o ma ics
disciplina y wo k (such as e.g. he CSRankings con e ence lis ).
© In o ma ics Eu ope, 2025, CC BY-SA 4.0|Page 16
8. The Role o AI in Resea ch E alua ions
The eme gence o powe ul AI me hods and ools in luences esea ch e alua ion in wo ways: i s , how
o ega d and assess AI being used in esea ch, and second, how AI can be used o pe o m o assis in
ca ying ou esea ch e alua ions. While he i s poin is being in ensely discussed, much less discussion
exis s ega ding he second poin . We shall add ess he la e issue he e.
Whene e a ask is edious, he ques ion a ises o how i can be au oma ed o a leas made easie by
delega ing pa s o an au oma ed p ocess. A good example in he con ex o esea ch e alua ions is he
h-index, which is widely used bu also hea ily c i icised. I s ad an age lies in he ease wi h which i can
be ob ained. I is much ha de o judge he con en o publica ions han simply coun hem and hei
ci a ion numbe s. Un il now, he e was no easy way o delega e he e e eeing o publica ions o an
au oma ed p ocess. This has changed wi h he ad en o La ge Language Models (LLMs). I is ce ainly
emp ing o use such models o e alua e he ou pu o esea che s o en i e depa men s (o e en
uni e si ies). A he ime o w i ing his posi ion publica ion, he gene ally a ailable LLMs a e hesi an o
w i e en i e e alua ions. Fo example, Gemini p ima ily desc ibes he p ocess o w i ing an e alua ion
bu does no p o ide one. Cha GPT can w i e an e alua ion bu is e y cau ious in compa ing di e en
In o ma ics depa men s. Mo eo e , Cha GPT bases i s judgmen s on ci a ion numbe s and e y ague
iden i ica ion o key s eng hs.
Ne e heless, we can sa ely assume ha LLMs can, in p inciple, be used o esea ch e alua ions and ha
hey will be used o his pu pose in he u u e, i no al eady oday. The ques ion o us is o unde s and
he po en ial dange s and bene i s o his de elopmen .
As wi h any o he use o AI, some migh call o es ic ions o an ou igh ban on using AI me hods in
e alua ions ha o m he basis o impo an decisions, such as unding la ge p ojec s o g an ing enu e.
I is sho -sigh ed o ban AI al oge he : i migh ac ually help in he decision-making p ocess when used
wi h disc e ion, and a ban canno be en o ced anyway. Ra he han a ban, we ecommend he ollowing
policies o he esponsible use o such echnologies, i hey a e used a all:
R8.1 The use o gene a i e AI in esea ch e alua ions should be communica ed openly, including
de ailed logs o he communica ions and he way he in o ma ion was used in he decision-making.
R8.2 Decisions mus s ill be made by human expe s, and he use o AI should be es ic ed o he lowe
le els o he decision p ocess.
R8.3 E o s should be made o e i y c i ical da a ob ained om a compu e .
R8.4 The use o AI mus no be used o educe he numbe o human expe s in decision panels and
hei esponsibili y o he ou come o he panel.
The use o AI is a mo ing a ge , and oday we a e jus beginning o ha ness i s powe . While AI has been
ins umen al in many a eas o some ime, i s applica ions in decision-making a e s ill in hei ea ly s ages.
Resea ch e alua ion is among hese a eas, and u he discussions in he nea u u e will be impo an as
he ield con inues o e ol e. The e o e, ou conclusions a his ime can only be conside ed p elimina y.
© In o ma ics Eu ope, 2025, CC BY-SA 4.0|Page 17
9. Conclusions
Resea ch e alua ion is essen ial o main aining he igou o any scien i ic ield, and he sys em o
expec a ions and incen i es c ea ed by e alua ion p ac ices can ha e a signi ican e ec on a esea ch
ield’s de elopmen . In o ma ics has many special cha ac e is ics, including me hodological di e si y,
close connec ion be ween opical esea ch and socio-economic impac , p ominence o op con e ences
as p ima y publica ion enues, a e ac s as i s -class esea ch ou comes, e c. These may no always be
appa en om ou side o he ield, ye need special conside a ion when assessing esea ch in In o ma ics.
In a judicious e alua ion p ocess, hese speci ici ies mus be aken in o accoun , oge he wi h he gene al
p inciples o esponsible esea ch e alua ion as ad anced in, e.g., he CoARA ecommenda ions.
In his documen , we a e econ i ming and upda ing he gene al guidelines o In o ma ics esea ch
e alua ion p esen ed in he 2008 and 2018 In o ma ics Eu ope epo s on he opic. We a e also p o iding
analyses and ecommenda ions in some a eas ha we e p e iously ea ed in less de ail o ha e
inc eased in p ominence since he p epa a ion o he ea lie epo s: bibliome ics and c edi assignmen
in con ibu ions, assessing a e ac s, Open Science, in e disciplina y esea ch, and he ole o AI in
esea ch e alua ion. As wi h any o he aspec s o science and esea ch, he dynamic and changing
landscape will ce ainly wa an e isi ing hese ecommenda ions again in some yea s, especially as
pe ains o opics ha a e ela i ely ecen .
Acknowledgemen s
We would like o acknowledge he inpu om a numbe o esea che s who ha e gene ously de o ed
hei ime o answe ou ques ions: E wan Bousse, Ca los Es eban Budde, Ka ine E en-Mendoza, Hada
F enkel, Lynda Ha dman, A nd Ha manns, Tobias Kappé, Raphaël Mona , Benoi Mon agu, Ma ius
Muench, Robe o Na ella, Michael Rawson, Robe Ricci, Ma kku-Juhani Saa inen, Jesús Sánchez
Cuad ado, Alexande Se eb enik, Caleb S an o d, and Quen in S ié ena .
Endo semen s
This epo is endo sed by he ollowing Na ional In o ma ics Associa ion membe s o In o ma ics
Eu ope:
● In o ma ik Aus ia (Aus ia)
● SIF – Socie é In o ma ique de F ance (F ance)
● FTI – Fakul ä en ag In o ma ik (Ge many)
● GI – Ge man In o ma ics Socie y (Ge many)
● GRIN – GRuppo di IN o ma ica (I aly)
● IPN - ICT-Resea ch Pla o m Ne he lands (Ne he lands)
● CODDII – Con e encia de Di ec o es y Decanos de Ingenie ía In o má ica (Spain)
● SCIE – Sociedad Cien í ica In o má ica de España (Spain)
● SIRA – Swiss In o ma ics Resea ch Associa ion (Swi ze land)
● CPHC – Council o P o esso s and Heads o Compu ing (Uni ed Kingdom)
© In o ma ics Eu ope, 2025, CC BY-SA 4.0|Page 18
Re e ences
(ACM 2020) ACM, A i ac Re iew and Badging Ve sion 1.1., 2020.
h ps://www.acm.o g/publica ions/policies/a i ac - e iew-and-badging-cu en
(Bake 2016) M. Bake , 1,500 scien is s li he lid on ep oducibili y. Na u e 533 (2016), 452-454.
h ps://doi.o g/10.1038/533452a
(Ball 2023) P. Ball, Is AI leading o a ep oducibili y c isis in science? Na u e 624 (2023), 22-25.
h ps://doi.o g/10.1038/d41586-023-03817-6
(Can eau 2021) A. Can eau e al., So wa e E alua ion (Resea ch Repo ). INRIA, 2021.
h ps://in ia.hal.science/hal-03110728/documen
(CoARA 2022) Coali ion o Ad ancing Resea ch Assessmen : Ag eemen on Re o ming Resea ch
Assessmen , 2022. h ps://coa a.eu/ag eemen / he-ag eemen - ull- ex /
(Dags uhl 2012) Publica ion Cul u e in Compu ing Resea ch -- Posi ion Pape s. Dags uhl Pe spec i e
Wo kshop 12452, 2012. Eds. K. Mehlho n, M. Y. Va di, M. He bs i .
h ps://doi.o g/0.4230/DagRep.2.11.20
(Di Cosmo 2019) R. Di Cosmo. How o use So wa e He i age o a chi ing and e e encing you sou ce
code: guidelines and walk h ough. Hal-02263344, 2019. h ps://doi.o g/10.48550/a Xi .1909.10760
(Di Cosmo e al. 2020) R. Di Cosmo, M. G uenpe e , S. Zacchi oli. Re e encing sou ce code a i ac s: a
sepa a e conce n in so wa e ci a ion. Compu ing in Science and Enginee ing 22:2 (2020), 33-43.
h ps://doi.o g/10.48550/a Xi .2001.08647
(DORA 2013) San F ancisco Decla a ion on Resea ch Assessmen , 2013. h ps://s do a.o g/ ead/
(DORI 2024) Ba celona Decla a ion on Open Resea ch In o ma ion, 2024.
h ps://ba celona-decla a ion.o g/
(Eu opean Commission 2017) Eu opean Commission. Di ec o a e Gene al o Resea ch and Inno a ion,
E alua ion o esea ch ca ee s ully acknowledging Open Science p ac ices: Rewa ds, incen i es and/o
ecogni ion o esea che s p ac ising Open Science, 2017. h ps://doi.o g/10.2777/75255
(F iedman & Schneide 2015) B. F iedman, F. B. Schneide , Incen i izing Quali y and Impac : E alua ing
Schola ship in Hi ing, Tenu e, and P omo ion. CRA Bes P ac ice Memo, 2015.
h ps://www.c a.o g/c a/wp-con en /uploads/si es/10/2016/02/BP_Memo.pd
(Hicks e al. 2015) D. Hicks, P. Wou e s, L. Wal man, S. de Rijcke, I. Ra ols, Bibliome ics: The Leiden
Mani es o o esea ch me ics. Na u e 520 (2015), 429–431. h ps://doi.o g/10.1038/520429a
(In o ma ics Eu ope 2008) Resea ch E alua ion o Compu e Science, In o ma ics Eu ope Repo , 2008.
Eds. B. Meye , C. Choppy, J. an Leeuwen and J. S auns up.
h ps://www.in o ma ics-eu ope.o g/se ices/publica ions/ epo s.h ml
(In o ma ics Eu ope 2013) P o ocol o Resea ch Assessmen in In o ma ics, Compu e Science and IT
Depa men s and Resea ch Ins i u es. In o ma ics Eu ope Repo , 2013. Ed. Man ed Nagl.
h ps://www.in o ma ics-eu ope.o g/se ices/publica ions/ epo s.h ml
(In o ma ics Eu ope 2018) In o ma ics Resea ch E alua ion, In o ma ics Eu ope Repo , 2018. Eds. F.
Esposi o, C. Ghezzi, M. He menegildo, H. Ki chne , L. Ong.
h ps://www.in o ma ics-eu ope.o g/se ices/publica ions/ epo s.h ml
© In o ma ics Eu ope, 2025, CC BY-SA 4.0|Page 19
(McLeish & S ang 2016) T. McLeish, V. S ang, E alua ing in e disciplina y esea ch: he elephan in he
pee - e iewe s’ oom. Na u e Palg a e Communica ions 2 (2016), 16055.
h ps://doi.o g/10.1057/palcomms.2016.55
(Rous 2017) B. Rous, The ACM Task Fo ce on Da a, So wa e, and Rep oducibili y in Publica ion, 2017.
h ps://www.acm.o g/publica ions/ ask- o ce-on-da a-so wa e-and- ep oducibili y
(UNESCO 2021a), UNESCO Recommenda ion on Open Science, 2021.
h ps://www.unesco.o g/en/open-science
(UNESCO 2021) UNESCO Recommenda ion on Open Science (P og amme and Mee ing Documen SC-
PCB-SPP/2021/OS/UROS), 2021. h ps://unesdoc.unesco.o g/a k:/48223/p 0000379949
(USENIX 2025) USENIX Secu i y '25 Call o Pape s.
h ps://www.usenix.o g/con e ence/usenixsecu i y25/call- o -pape s
(Win e e al. 2022) S. Win e e al., A e ospec i e s udy o one decade o a i ac e alua ions. P oc.
ESEC/FSE 2022. h ps://doi.o g/10.1145/3540250.3549172
Fo enqui ies and eedback abou his epo ,
please con ac [email p o ec ed]
www.in o ma ics-eu ope.o g
© In o ma ics Eu ope, 2025
CC BY-SA 4.0
