1
Re hinking al me ics as p ocess-based indica o s: a concep ual
amewo k o cons uc cla i y
Ad ián A. Díaz-Faes1, Zoh eh Zahedi2,3
1INGENIO (CSIC-UPV), Uni e si a Poli ècnica de València, Camino de Ve a s/n, 46022
Valencia, Spain
2Depa men o In o ma ion Science, Facul y o Humani ies, Pe sian Gul Uni e si y, Busheh ,
I an
3Cen e o Science and Technology S udies (CWTS), Leiden Uni e si y, he Ne he lands
Co esponding au ho : [email p o ec ed]
ORCID: AADF: 0000-0003-1928-4608, ZZ: 0000-0001-5801-1886
Abs ac : The Al me ics Mani es o p omised o b oaden he no ion o impac . Howe e ,
eliance on ci a ion heo y has limi ed his p omise, con ining al me ics o bibliome ic
p inciples by ea ing in e ac ions wi h schola ly ou pu s ha occu ou side he no ms and alues
ha ule science h ough he same lens as ci a ions. This s udy a gues ha his cons i u es a o m
o on ological misalignmen ( allacy o ei ica ion), whe eby al me ics a e assigned en i a i e
p ope ies when hey in ac cap u e p ocessual aspec s o engagemen and use ha may p ecede
o con ibu e o socie al impac . This misalignmen is mani es ed as use o coun s o social media
and online in e ac ions wi h schola ly ou pu s as di ec indica o s o socie al impac . Aligning
on ology and epis emology equi es he ede ini ion o al me ics as p ocess-based indica o s.
We desc ibe ecen empi ical de elopmen s applying in e ac ion and ne wo ked app oaches o
ansla e his pe spec i e in o p ac ice. We a gue also ha calls o p oduc ion o socially ele an
knowledge and inc eased in e es in science communica ion ep esen a imely oppo uni y o
use o al me ics as p ocess-o ien ed moni o ing ools. To s udy socie al impac , we sugges hei
use in a mixed-me hods esea ch design o iden i y engagemen pa e ns ha can be u he
explo ed using quali a i e me hods.
Keywo ds: Science-socie y in e ac ions, on ological shi , socie al impac , engagemen , p ocess-
based indica o s.
2
1. INTRODUCTION
Al me ics eme ged 15 yea s ago as p omising web-based me ics o engagemen , use, and
impac . They we e seen as expanding he scope o wha can be measu ed sys ema ically and wha
is wo h measu ing. Expec a ions abou hei abili y o measu e impac in a b oade sense beyond
jou nal-based schola ly publishing g ew apidly (Sugimo o e al., 2017). The aim o al me ics
1
was o “ alue all esea ch p oduc s” and “gi e a ulle pic u e o how esea ch p oduc s ha e
in luenced con e sa ion, hough and beha io ” (Piwowa , 2013, p. 159). Al hough his ambi ion
esona es wi h subsequen calls o e o ms o esea ch assessmen
2
, which would acknowledge
he di e si y o he con ibu ions in science and hei impac (Rush o h & Hamma el , 2023),
he ini ial p omise and expec a ions su ounding al me ics ha e ye o ma e ialize (Wou e s e
al., 2019).
We a gue ha hese un ul illed expec a ions a e no due me ely o da a issues such as
he e ogenei y, co e age, and pe sis ence o o he absence o s anda ds bu a he s em om he
lack o a concep ual g ounding. As al me ics accumula e a a apid pace and b oaden na ow
indica o s o scien i ic impac , esea che s apidly leap in o da a analyses ansla ing ci a ion
heo y o social media, esul ing in inconsis en in e p e a ions o i s meaning (Haus ein, 2016;
Wou e s e al., 2019). A g owing numbe o ecen s udies is eschewing bibliome ic-based
models and hei applica ion o social media and ad oca ing o in e ac i e and ne wo k
app oaches o analyze al me ic da a. Examples include de elopmen o me hods o accoun o
he ci cula ion o esea ch beyond academic bounda ies (Alpe in e al., 2024; Cos as e al., 2021),
di e en ia ion be ween ypes and deg ees o engagemen (Fang e al., 2022; Haus ein e al.,
2016), mapping o scien is s’ engagemen pa e ns (Robinson-Ga cía e al., 2018; Wal e e al.,
2019) and communica ion p ac ices (Ha e e al., 2024; Mongeon, 2018), clus e ing use s based
1
Following Wou e s e al. (2019), we use “me ics” o deno e bo h da a and indica o s.
2
CoARA: h ps://coa a.eu/app/uploads/2022/09/2022_07_19_ a_ag eemen _ inal.pd
3
on hei social media beha io (A aujo, 2020; Díaz-Faes e al., 2019; Pea ce e al., 2014; Yu e
al., 2019), and acing socially ele an opics and misin o ma ion (Shao e al., 2018; Van
Schalkwyk e al., 2020). Howe e , despi e he me i s o such app oaches, he concep ual
discussion equi ed o unde pin any shi in ocus is mos ly lacking.
The p esen s udy aims o ill his gap by ad oca ing o a shi in he concep ualiza ion and
applica ion o al me ics. We a gue ha a key obs acle o al me ics ealizing hei ini ial p omise
is he absence o cons uc cla i y, speci ically he misalignmen be ween wha al me ics cap u e
(on ology) and how hei meaning is in e p e ed and applied in esea ch p ac ice (epis emology).
This a ises because al me ics emain g ounded in bibliome ic con en ions and p inciples, and
apply ci a ion lens o in e ac ions and exchanges ha occu ou side he no ms and alues ha ule
science. To imp o e cons uc cla i y and posi ion al me ics as aluable ools o in es iga e
science-socie y in e ac ions
3
, we sugges hey should be concei ed as p ocess-based indica o s.
We build on Thompson's (2011) amewo k o align on ology and epis emology o achie e
cons uc cla i y, and on Resche ’s wo k on p ocess philosophy (2006). Ou wo k is in o med
also by Haus ein e al. (2016), Robinson-Ga cía e al. (2018), and Díaz-Faes e al. (2019). The
o me wo k add esses he need o speci ic amewo ks, models, and heo ies o suppo
in e p e a ion o al me ics. The la e wo call o an in e ac i e app oach o al me ics.
Ou pape is s uc u ed as ollows: sec ion 2 discusses he p oblems in ol ed in applying
a bibliome ic model o socie al impac , and shows ha ci a ion heo y does no ansla e well o
al me ics. In his con ex , sec ion 3 no es he impo ance o aligning on ology and epis emology
and a gues ha al me ics esea ch su e s om a allacy o ei ica ion (a ibu ing en i a i e
p ope ies o p ocesses). We a gue ha he shi owa ds a p ocess iew would make al me ics
mo e e ec i e ools o s udying science-socie y in e ac ions. We also desc ibe he empi ical
de elopmen s in he ield in line wi h his iew. Sec ion 4 desc ibes how he policy and e alua ion
3
Cos as e al. (2021) sugges ha gi en he p oblems ela ed o ope a ionalizing ‘socie y’ as a well-bounded
ca ego y, he dis inc ions science and non-science could be used as al e na i es.
4
con ex s c ea e an oppo uni y space o use o al me ics as p ocess-o ien ed me ics. Sec ion 5
p esen s he main implica ions o his s udy.
2. THE NEED TO DEPART FROM CITATION THEORY
Ci a ions ha e been discussed in he con ex o di e en compe ing and coexis ing heo ies,
mainly no ma i e and cons uc i is (Moed, 2005). The no ma i e iew posi s ha ci a ions a e
an ins i u ionalized p ac ice o science ha equi es scien is s o acknowledge he sou ces and
p io wo k on which hey ha e buil upon (Me on, 1973). Ci a ions main ain “in ellec ual
adi ions and p o ide he pee ecogni ion equi ed o he e ec i e wo king o science as a
social ac i i y” (Me on, 1988, p. 621). Howe e , ci a ion beha io is complex and canno be
desc ibed unidimensionally (Bo nmann & Daniel, 2008; B ooks, 1986).
The cons uc i is iew challenges he no ma i e heo y, a guing ha ci a ions unc ion as
he o ical de ices used o pe suade — o example, o con ince eade s o he no el y o alue
o one’s indings (Gilbe , 1977) — a he han o e lec in ellec ual in luence. Howe e , as
Zucke man (1987) no es, Gilbe ’s (1977, p. 116) no ion ha some pape s a e “impo an and
co ec ”, and he e o e used as au ho i a i e e e ences o suppo au ho s’ own a gumen s, is
i sel g ounded in pee ecogni ion and ypically mani es ed in high ci a ion a es, which in u n
make hose pape s au ho i a i e.
Empi ical s udies on he alidi y o bo h app oaches o ci a ion show ha a no ma i e
accoun i s he da a be e han he pe suasion model (Moed & Ga ield, 2004; Whi e, 2004).
Besides heo ies, ci ing beha io can also be in luenced by o he ac o s, such as language biases,
a ge audience, jou nal s a us, and he scope, o ma , o leng h o he pape (C onin, 1981;
MacRobe s & MacRobe s, 2018), al hough such biases end o cancel ou a agg ega ed le els
( an Raan, 1998).
5
Despi e he di e se mo i a ions o ci ing and he p esence o dis o ing ac o s
4
, i is
gene ally ag eed ha ci a ions e lec schola ly an eceden s o wo k and se e as indica o s o
scien i ic impac (Aksnes e al., 2019; Ma in & I ine, 1983). Thus, one can assume ha highly
ci ed pape s a e gene ally scien i ically ele an , and ha ci a ions can he e o e be measu ed
linea ly and assumed o be posi i e (Bo nmann & Daniel, 2008). Howe e , such a con en ion
canno be applied o wha al me ics measu e.
The al me ics niche eme ged based on he inadequacy o bibliome ic indica o s and pee
e iew — he adi ional il e s. The Al me ics Mani es o desc ibes hese indica o s as slow,
na ow, and closed compa ed o al me ics’ imeliness, di e si y, and openness (Lecke , 2021).
Ci a ions ha e been c i icized and desc ibed as: “coun ing measu es [ ha ] a e use ul, bu no
su icien (…) in luen ial wo k may emain unci ed. These me ics a e na ow; hey neglec
impac ou side he academy and also igno e he con ex and easons o ci a ion” (P iem e al.,
2010, p. 2). The Al me ics Mani es o en isioned a change owa ds a mo e comp ehensi e
unde s anding and assessmen o science while in p ac ice, he unde lying logic does no
ep esen any depa u e om he adi ional il e s. Al me ics esea ch apidly became ocused
on coun ing
5
men ions, sha es, and downloads o schola ly ou pu s in social media as i
bibliome ic con en ions and he scien i ic e hos ( he alues and no ms go e ning scien i ic
ac i i y) could be applied beyond he bounda ies o academia. This can be seen clea ly in he
main opics add essed in al me ics esea ch.
Much o he esea ch on al me ics has ocused on wo lines o inqui y. The i s examines
he ela ionship be ween al me ics and ci a ions (Holmbe g e al., 2019; Sugimo o e al., 2017)
on he p emise ha wo indica o s o impac should be co ela ed. Since al me ics accumula e
4
The ac ha no all ci a ions a e c i ical, and ha scien is ’s eason o ci e can go beyond he me i s o he wo k
i sel highligh s he need o con ex ualized, mul idimensional, and well-cu a ed analyses.
5
Da a p o ide s seized he oppo uni y o p o ide agg ega ed indica o s such as he Al me ic A en ion Sco e which
lumps oge he di e se men ions and in e ac ions no based on any clea c i e ia (Thelwall, 2021). C i iques o his
me hod seem no o ha e p e en ed i s applica ion.
6
as e han ci a ions, i was sugges ed ha al me ics could be used as p edic o s o u u e
ci a ions. This led o esea ch ha in ol ed coun ing he numbe o imes schola ly ou pu s we e
wee ed, sha ed on Facebook, o men ioned in blogs and news media o ob ain di ec indica o s
o impac . Howe e , he e idence shows ha al me ic coun s o schola ly ou pu s a e poo
p edic o s o la e ci a ions (Cos as e al., 2015). A second line explo es he use o al me ics o
assess he socie al impac o esea ch. Ye , empi ical s udies show ha high al me ic coun s a e
no co ela ed wi h pee e iew assessmen o socie al impac
6
(Bo nmann e al., 2019; Kassab
e al., 2020; Ra ensc o e al., 2017). The e a e wo easons why hese indings a e no a su p ise.
Fi s , he e a e no g ounds o a s aigh ela ionship be ween ci a ions and al me ics since no
all ele an scien i ic esea ch (e.g. blue-sky esea ch) will necessa ily lead o socie al impac .
Second, a en ion and engagemen wi h science do no necessa ily ansla e in o socie al impac .
The quali a i e assessmen me hods li e a u e spells ou he unde lying easons: he
di icul y o linking speci ic schola ly ou pu s o obse ed socie al impac (a ibu ion p oblem),
he o en long ime lag be ween esea ch and i s speci ic impac on socie y — empo ali y issue
— (Spaapen & Van D ooge, 2011), and ha socie al impac encompasses social, cul u al,
en i onmen al and economic aspec s — mul iple dimensions (Bo nmann, 2013). Al hough he
quali a i e app oaches o measu ing socie al impac a e di e se and ocus on di e en p ocesses
c ea ing alue, he e is ag eemen abou he need o mo e om a ibu ion o con ibu ion since
socie al impac is con ex -dependen and equi es in e ac ions be ween scien is s and
s akeholde s (Smi & Hessels, 2021). Resea ch wi h socie al impac can ake he o m o a
p oduc (e.g. e idence, me hods, echnology, models), knowledge use (i.e. adop ion o
knowledge by s akeholde s), o bene i s based on i s applica ion and use (e.g. changes o p ac ice,
awa eness, heal h, economic impac ) (Bo nmann, 2013; De Jong e al., 2014; Woolley & Molas-
6
The excep ion is Mendeley o which mild-medium co ela ions a e ound when compa ed wi h pee e iew sco es
om he UK Resea ch Excellence F amewo k (Akella e al., 2021; Thelwall e al., 2023). S ill, one can a gue ha
his mainly e lec academic use.
7
Galla , 2023). This s eam o wo k o e s a new lens allowing be e use o al me ic da a
(Robinson-Ga cía e al., 2018).
Fo ins ance, he SIAMPI app oach (Spaapen & Van D ooge, 2011) de ines in e ac ions as
p oduc i e i he knowledge p oduced is scien i ically obus and socially ele an . The au ho s
ope a ionalize his by dis inguishing be ween di ec (people o people), indi ec (in e ac ion
media ed h ough a ca ie such as schola ly ou pu ), and inancial ( esea ch con ac , inancial
con ibu ion) in e ac ions. Ano he example is Mo on’s (2015) ‘con ibu ions app oach’ which
de ines a p ocess iew o socie al impac based on he no ion o esea ch up ake (i.e. engagemen
wi h esea ch), esea ch use (e.g. discussion o , sha ing, adap a ions o, applica ion o he
esea ch o in o m policy o ac ion), and esea ch impac ( esea ch esul ing in changes o
knowledge, awa eness, a i udes, o p ac ice).
Despi e he exis ence o empi ical e idence and quali a i e assessmen me hods ha
demons a e he inadequacy o applying bibliome ic con en ions o al me ics, mos al me ics
esea ch con inues o do so. To assess he ex en o his end, we que ied he Scopus da abase
7
and e ie ed all Al me ic pape s (a icles and e iews) published be ween Oc obe 2010 — he
yea o he Al me ics Mani es o — and 2024. We sys ema ically sc eened he abs ac s,
me hods, and esul s sec ions o each pape o check o any e idence o ou pu o p ocess-based
app oaches. This esul ed in ou classi ying he s udies in o one o h ee g oups: ou pu -based
(coun ing al me ic da a on pape s as di ec indica o s o impac , akin o ci a ions), p ocess-based
(applying ne wo k and in e ac ion app oaches o examine pa e ns o engagemen , dissemina ion,
and use), and o he (e.g. using al me ic da a in ields such as digi al ma ke ing o educa ion).
Figu e 1 shows ha he as majo i y o al me ics esea ch applies an ou pu -based app oach
7
(TITLE-ABS-KEY (al me ic*) OR TITLE-ABS-KEY (“social media me ic*”)) AND PUBYEAR > 2010 AND
PUBYEAR < 2024 AND (LIMIT-TO (SRCTYPE, "j”)) AND (LIMIT-TO (DOCTYPE, "a ”) OR LIMIT-TO
(DOCTYPE, " e”)) AND (LIMIT-TO (LANGUAGE, "English”)). The sea ch que y e ie ed 1,493 pape s. A e
manual e ision, 109 pape s ha we e nei he a icles no e iews we e emo ed, esul ing in a inal sample o 1,392
pape s.
8
(82%), while only 2% o pape s adop a p ocess lens. The emaining 16% co esponds o he use
o social media da a un ela ed o he scope o his s udy.
Figu e 1. Time ends in al me ics esea ch by analy ical app oach
3. IMPROVING CONSTRUCT CLARITY IN ALTMETRICS
3.1. Aligning on ology and epis emology
As explained abo e, he e is no basis o applying bibliome ic con en ions and ci a ion heo y
o cap u e a b oade no ion o impac . This misma ch is e idence o he need o e hink ou
unde s anding and use o al me ics which demands a be e cla i ica ion o al me ics as a
cons uc .
Cons uc s a e concep ual abs ac ions adop ed o in en ed o a speci ic scien i ic pu pose
(Venka aman & G an , 1986). As Suddaby (2010, p. 346) no es: “Clea cons uc s a e simply
obus ca ego ies ha dis ill phenomena in o sha p dis inc ions ha a e comp ehensible o a
communi y o esea che s.” Because cons uc cla i y p o ides a basis o heo y and p ac ice,
we a gue ha i is c ucial o align on ology and epis emology so ha knowledge gene a ed
9
h ough al me ics e lec s he phenomena al me ics seeks o obse e. On ology conce ns he
basic assump ions abou he ype o hings ha can exis , hei condi ions o exis ence and wha
coun s as ele an objec s, en i ies and p ocesses in a gi en domain. Epis emology, howe e ,
deals wi h how alid knowledge abou hese objec s, en i ies and p ocesses can be accessed and
gene a ed (Chalme s, 1999; Kildu e al., 2011).
Ou p oposed shi om a bibliome ic model o a p ocess iew d aws on Thompson’s
(2011) amewo k o p o ide cons uc cla i y and con as he classical pe spec i es ha
pe cei e eali y as an en i y o as a p ocess. An en i y lens iews eali y (in ou case al me ics)
as comp ising objec s wi h s able p ope ies whose meaning does no change ac oss mul iple
si ua ions. This applies o how ci a ions a e seen in bibliome ics and hei conside a ion as di ec
indica o s o scien i ic impac . Howe e , a p ocess lens emphasizes ha eali y is dynamic,
eme gen , and e ol ing and ha in e ac ions cons i u e a he han simply modi ying eali y
(Resche , 2006). Ra he han conside ing hings (numbe s o men ions, sha es, o commen s o
schola ly ou pu s) as meaning ul and indica i e by hemsel es, al me ics should be abou
unde s anding happenings ( he in e ac ion, engagemen wi h, and use o ).
Figu e 2 depic s how he misalignmen be ween on ology ( op hal ) and epis emology
(bo om hal ) in al me ics esea ch leads o on ological d i , speci ically he allacy o
ei ica ion. This allacy a ibu es en i a i e p ope ies o a cons uc ha is mo e adequa ely
unde s ood as a p ocess (quad an s 1 and 3). G ounded in an en i y pe spec i e bo owed om
ci a ion heo y, al me ics esea ch ea s di e se o ms o online engagemen and in e ac ions
wi h schola ly ou pu s as di ec indica o s o he b oade impac o esea ch. Tha is, he e is an
assump ion ha some speci ic socie al impac can be a ibu ed o men ions, commen s, sha es,
downloads, o o he o ms o engagemen and in e ac ions wi h schola ly ou pu s. This
on ological d i is mani es ed in he assump ion ha online engagemen wi h science by di e se
s akeholde s is media ed by sha ed no ms, alues, and epis emic con en ions simila o
16
o socially ele an knowledge equi es new ools o measu e b oade no ions o impac
10
(Eu opean Commission e al., 2017; Ra ols e al., 2024). Since many ac i i ies and in e ac ions
wi h science a e digi al, al me ics esea ch adop ing a p ocess iew can p o ide in o ma ion on
in e ac ions, exchanges, dialogues, and he ways ha di e en s akeholde s engage wi h and use
esea ch knowledge.
No e also ha he g ea e openness he o ic which unde pins claims o p oduc ion o
socially ele an knowledge igge s some impo an challenges (Mi owski, 2018). Fo ins ance,
p ep in s make scien i ic wo k a ailable as e and os e c i ique and up ake o new ideas bu
also acili a e he di usion o un e i ied and me hodologically lawed esea ch (see Caul ield e
al., 2021 o he COVID-19 case). Also, cu en open access is mo e a p o i able business o a
ew comme cial publishe s han a way o p o ide socie al access o science (Bu le e al., 2023).
Use o al me ics as p ocess indica o s allows us o examine whe he open access is achie ing i s
goals o ein o cing inequali ies and hampe ing di e si y.
4.2. The science communica ion model(s)
Besides he g ea e expec a ions abou scien i ic wo k b inging alue o socie y, scien is s a e
inc easingly being encou aged by unde s and hei ins i u ions o ake a mo e ac i e ole in
communica ing science and es ablishing a dialogue wi h public audiences (NASEM, 2017;
Weinga , 2017). As a esul o digi aliza ion, open access, and he ise o social media, science
communica ion can no longe be unde s ood as a linea and op-down p ocess (de ici model) in
which scien is s gene a e new knowledge, science jou nalis s ope a e as b oke s, and he public
passi ely consumes he in o ma ion (Bucchi, 2017; Scheu ele, 2014). This ans o ma ion o he
science-socie y in e ace has democ a ized con en c ea ion and dissemina ion and pa ed he way
10
Conce n o e he “gami ica ion” o socie al impac measu es as has happened wi h ci a ions is well- ounded
(Holmbe g e al., 2019). Howe e , hese a gumen s a e ela ed mo e o e o m o he science ewa d sys em, esea ch
cul u e, and esponsible use o indica o s han o al a oidance o quan i a i e da a (Balboa e al., 2024). Pee e iew
as a quali y con ol mechanism is subjec o bias (Ho obin, 1990), and quali a i e assessmen s can lack objec i i y
and be hinde ed by ime and esou ce cons ain s (Bo nmann, 2013).
17
o dialogue, pa icipa o y, and co-c ea ion models (T ench, 2008). Howe e , i aises ques ions
abou how gene al audiences can iden i y eliable scien i ic in o ma ion and how science
communica ion can os e a sha ed unde s anding o scien i ic indings and suppo a dialogue o
enac socie al change. Robus socially ele an knowledge does no a ise simply om scien i ic
e idence bu is spawned by con inual in e ac ion and con e sa ion be ween ac - inding and
meaning-making (Jasano , 2010).
Al hough he science, echnology and inno a ion communi y has so a paid li le a en ion
o science communica ion, ecen s udies show, o example, he u ili y o a ne wo k pe spec i e
and con en analysis o un a eling he communi ies in ol ed in an i- accine mo emen s (Van
Schalkwyk e al., 2020), and scien is s’ engagemen in he clima e change deba e (Wal e e al.,
2019). Likewise, science communica ion ou pu s — p ess eleases, news i ems — a e being
analyzed quan i a i ely and becoming ele an o unde s anding online science communica ion
(G o es e al., 2015; O duña-Malea & Cos as, 2023). Thus, al me ics ha e he po en ial o
moni o aspec s ela ed o he science communica ion p ocesses. They can be ele an o policy
o os e mo e in o med and meaning ul online in e ac ions wi h science (i.e. diminish he e ec s
o misin o ma ion, ake news, and unc i ical use o sou ces) (Díaz-Faes, 2023).
5. CONCLUDING REMARKS
This s udy con ibu es o o e coming he seclusion o da a-d i en esea ch in quan i a i e science
s udies (Ra ols, 2019) by add essing he need o a concep ual amewo k in al me ics esea ch.
Since cons uc cla i y is c ucial o heo y and p ac ice (Suddaby, 2010), we examine he
misalignmen be ween wha al me ics measu e and how hei meaning is in e p e ed and applied
in p ac ice. We also discuss cons uc alidi y o p o ide a basis o ansla ing a shi owa ds
iewing al me ics as p ocess-based me ics in o esea ch p ac ice.
The Al me ics Mani es o p omised a change by expanding he no ion o impac . Howe e ,
he concep ual lens unde pinning al me ics esea ch has emained g ounded in he p inciples o
18
bibliome ics. We a gue ha al me ics esea ch unc i ically bo owed ci a ion heo y, ea ing
in e ac ions and exchanges on social media and online pla o ms as i hey we e simila o hose
media ed by he no ms and alues uling scien i ic wo k. This leads o he unde lying assump ion
ha scien i ic and socie al impac a e somehow equi alen , despi e he ac ha he la e is
con ex dependen and di icul o a ibu e o speci ic schola ly ou pu s. This has esul ed in a
o m o on ological misalignmen ( allacy o ei ica ion) in which al me ics a e assigned
en i a i e p ope ies as i hey di ec ly indica e socie al impac , a he han ecognizing ha hey
cap u e p ocessual aspec s o engagemen ha may p ecede o con ibu e o socie al impac . We
ha e sugges ed ha aligning on ology and epis emology equi es a ede ini ion o al me ics as
p ocess indica o s.
This shi is suppo ed by he policy and e alua ion con ex which needs quan i a i e
e idence o p o ide in o ma ion on he complex and plu alis ic in e ac ions and exchanges
ela ed o he po en ial an eceden s o socie al impac and science communica ion dynamics
(Rush o h & Hamma el , 2023; Science Eu ope, 2022). Bo nmann & Leydesdo (2014) no e
ha al me ics a e in a simila eme ging s a e o ha o bibliome ic indica o s in he 1970s. I
could be a gued ha as a ield, we a e s ill na iga ing he de elopmen o a oolbox o me hods
o al me ics while ha ing emba ked on his phase wi h no consensus on al me ics as a
cons uc . In de eloping al me ics as p ocess-based indica o s o suppo assessmen o socie al
impac , we sugges a mixed-me hod app oach, in which p ocess al me ics adap o and accoun
o each pla o m’s speci ic cha ac e is ics and a o dances in o de o hin a ele an science-
socie y in e ac ions and engagemen pa e ns ha can be explo ed u he quali a i ely.
Au ho con ibu ions
AADF: Concep ualiza ion, Da a cu a ion, In es iga ion, Fo mal Analysis, Me hodology,
Visualiza ion, Funding acquisi ion, W i ing-o iginal d a , W i ing- e iew & edi ing
ZH: In es iga ion, Da a cu a ion, Fo mal analysis, W i ing- e iew & edi ing
19
Compe ing in e es s
The au ho s ha e no compe ing in e es s o disclose.
Acknowledgmen s
Ad ián A. Díaz-Faes acknowledges suppo om esea ch p ojec PID2020-112837RJ-I00,
unded by MCIN/AEI/10.13039/501100011033. We a e g a e ul o Rod igo Cos as o insigh ul
discussions, and o Maddie Ha e and Pablo D’Es e o hei in aluable eedback on an ea lie
e sion o his pape . A p elimina y e sion o his wo k was p esen ed a NWB2024 in
Reykja ík.
Re e ences
Aagaa d, K., No n, M. T., & S age, A. K. (2022). How mission-d i en policies challenge
adi ional esea ch unding sys ems [ e sion 1; pee e iew: 2 app o ed wi h ese a ions,
1 no app o ed]. F1000Resea ch, 11(949).
h ps://doi.o g/10.12688/ 1000 esea ch.123367.1
Akella, A. P., Alhoo i, H., Kondamudi, P. R., F eeman, C., & Zhou, H. (2021). Ea ly indica o s
o scien i ic impac : P edic ing ci a ions wi h al me ics. Jou nal o In o me ics, 15(2),
101128. h ps://doi.o g/h ps://doi.o g/10.1016/j.joi.2020.101128
Aksnes, D. W., Lang eld , L., & Wou e s, P. (2019). Ci a ions, Ci a ion Indica o s, and
Resea ch Quali y: An O e iew o Basic Concep s and Theo ies. SAGE Open, 9, 1–17.
h ps://doi.o g/10.1177/2158244019829575
Alpe in, J. P., Flee acke s, A., Riedlinge , M., & Haus ein, S. (2024). Second-o de ci a ions in
al me ics: A case s udy analyzing he audiences o COVID-19 esea ch in he news and
on social media. Quan i a i e Science S udies, 5(2), 366–382.
h ps://doi.o g/10.1162/qss_a_00298
A aujo, R. F. (2020). Communi ies o a en ion ne wo ks: in oducing quali a i e and
con e sa ional pe spec i es o al me ics. Scien ome ics, 124(3), 1793–1809.
20
h ps://doi.o g/10.1007/s11192-020-03566-7
Balboa, L., Gadd, E., Méndez, E., Pölönen, J., S ooban s, K., & Tó h-Czi a, E. (2024). The
ole o scien ome ics in he pu sui o esponsible esea ch assessmen . LSE Blog.
h ps://blogs.lse.ac.uk/impac o socialsciences/2024/09/04/ he- ole-o -scien ome ics-in-
he-pu sui -o - esponsible- esea ch-assessmen /
Beck, S., B asseu , T. M., Poe z, M., & Saue mann, H. (2022). C owdsou cing esea ch
ques ions in science. Resea ch Policy, 51(4), 104491.
h ps://doi.o g/h ps://doi.o g/10.1016/j. espol.2022.104491
Bo nmann, L. (2013). Wha is socie al impac o esea ch and how can i be assessed? A
li e a u e su ey. Jou nal o he Ame ican Socie y o In o ma ion Science and
Technology, 64(2), 217–233.
Bo nmann, L. (2014). Do al me ics poin o he b oade impac o esea ch? An o e iew o
bene i s and disad an ages o al me ics. Jou nal o In o me ics, 8(4), 895–903.
Bo nmann, L., & Daniel, H. (2008). Wha do ci a ion coun s measu e? A e iew o s udies on
ci ing beha io . Jou nal o Documen a ion, 64(1), 45–80.
Bo nmann, L., Haunschild, R., & Adams, J. (2019). Do al me ics assess socie al impac in a
compa able way o case s udies? An empi ical es o he con e gen alidi y o al me ics
based on da a om he UK esea ch excellence amewo k (REF). Jou nal o In o me ics,
13(1), 325–340.
Bo nmann, L., & Leydesdo , L. (2014). Scien ome ics in a changing esea ch landscape:
bibliome ics has become an in eg al pa o esea ch quali y e alua ion and has been
changing he p ac ice o esea ch. EMBO Repo s, 15(12), 1228–1232.
B ooks, T. A. (1986). E idence o complex ci e mo i a ions. Jou nal o he Ame ican Socie y
o In o ma ion Science, 37(1), 34–36.
Bucchi, M. (2017). C edibili y, expe ise and he challenges o science communica ion 2.0.
21
Public Unde s anding o Science, 26(8), 890–893.
Bu le , L.-A., Ma hias, L., Sima d, M.-A., Mongeon, P., & Haus ein, S. (2023). The
oligopoly’s shi o open access: How he big i e academic publishe s p o i om a icle
p ocessing cha ges. Quan i a i e Science S udies, 4(4), 778–799.
h ps://doi.o g/10.1162/qss_a_00272
Caul ield, T., Bubela, T., Kimmelman, J., & Ra i sky, V. (2021). Le ’s do be e : public
ep esen a ions o COVID-19 science. FACETS, 6, 403–423.
h ps://doi.o g/10.1139/ ace s-2021-0018
Chalme s, A. F. (1999). Wha is his hing called science? (3 d ed.). Open Uni e si y P es.
Cos as, R., de Rijcke, S., & Ma es, N. (2021). “He e ogeneous couplings”: Ope a ionalizing
ne wo k pe spec i es o s udy science-socie y in e ac ions h ough social media me ics.
Jou nal o he Associa ion o In o ma ion Science and Technology, 72(5), 595–610.
h ps://doi.o g/10.1002/asi.24427
Cos as, R., Zahedi, Z., & Wou e s, P. (2015). Do “al me ics” co ela e wi h ci a ions?
Ex ensi e compa ison o al me ic indica o s wi h ci a ions om a mul idisciplina y
pe spec i e. Jou nal o he Associa ion o In o ma ion Science and Technology, 66(10),
2003–2019. h ps://doi.o g/10.1002/asi.23309
C onin, B. (1981). The need o a heo y o ci a ion. Jou nal o Documen a ion, 37(1), 16–24.
h ps://doi.o g/10.1108/eb026703
C owe, S., C esswell, K., Robe son, A., Huby, G., A e y, A., & Sheikh, A. (2011). The case
s udy app oach. BMC Medical Resea ch Me hodology, 11(1), 100.
h ps://doi.o g/10.1186/1471-2288-11-100
De Jong, S., Ba ke , K., Cox, D., S einsdo i , T., & Van den Besselaa , P. (2014).
Unde s anding socie al impac h ough p oduc i e in e ac ions: ICT esea ch as a case.
Resea ch E alua ion, 23(2), 89–102. h ps://doi.o g/10.1093/ ese al/ u001
22
Díaz-Faes, A. A. (2023). Science communica ion and public policy: Imp o ing science socie y
in e ac ions – oppo uni ies and challenges. In V. Pa one, J. Molas, J. B and s, & R.
Se ano (Eds.), Science o Public Policy (CSIC, pp. 76–79). Edi o ial CSIC.
h ps://digi al.csic.es/handle/10261/337975
Díaz-Faes, A. A., Bowman, T. D., & Cos as, R. (2019). Towa ds a second gene a ion o ‘social
media me ics’: Cha ac e izing Twi e communi ies o a en ion a ound science. PLOS
ONE, 14(5), e0216408. h ps://doi.o g/doi.o g/10.1371/jou nal.pone.0216408
Eu opean Commission, B.-I. U., o Resea ch, D.-G., Inno a ion, o Technology, G. U.,
Uni e si y, K., o Economics, L. I. C., Uni e si y, L., o No h Texas, U., o She ield
(UK), U., Pe e s, I., F odeman, R., Wilsdon, J., Ba -Ilan, J., Lex, E., & Wou e s, P. (2017).
Nex -gene a ion me ics – Responsible me ics and e alua ion o open science.
Publica ions O ice. h ps://doi.o g/doi/10.2777/337729
Fang, Z., Cos as, R., & Wou e s, P. (2022). Use engagemen wi h schola ly wee s o scien i ic
pape s: a la ge-scale and c oss-disciplina y analysis. Scien ome ics.
h ps://doi.o g/10.1007/s11192-022-04468-6
Gibbons, M., Limoges, C., Nowo ny, H., Schwa zman, S., Sco , P., & T ow, M. (1994). The
new p oduc ion o knowledge: The dynamics o science and esea ch in con empo a y
socie ies. SAGE Publica ions.
Gilbe , G. N. (1977). Re e encing as Pe suasion. Social S udies o Science, 7(1), 113–122.
h ps://doi.o g/10.1177/030631277700700112
Go man, E. (1959). The p esen a ion o he sel in e e yday li e. Ancho .
G o es, T., Figue ola, C. G., & Quin anilla, M. Á. (2015). Ten yea s o science news: A
longi udinal analysis o scien i ic cul u e in he Spanish digi al p ess. Public
Unde s anding o Science, 25(6), 691–705. h ps://doi.o g/10.1177/0963662515576864
Ha e, M., K ause, G., MacKnigh , K., Bowman, T. D., Cos as, R., & Mongeon, P. (2024). Do
23
you ci e wha you wee ? In es iga ing he ela ionship be ween wee ing and ci ing
esea ch a icles. Quan i a i e Science S udies, 5(2), 332–350.
h ps://doi.o g/10.1162/qss_a_00296
Haus ein, S. (2016). G and challenges in al me ics: he e ogenei y, da a quali y and
dependencies. Scien ome ics, 108(1), 413–423. h ps://doi.o g/10.1007/s11192-016-1910-
9
Haus ein, S. (2019). Schola ly Twi e me ics. In G. Wol gang, H. F. Moed, U. Schmoch, &
M. Thelwall (Eds.), Sp inge Handbook o science and echnology indica o s (pp. 729–
760). Sp inge . h ps://link.sp inge .com/chap e /10.1007/978-3-030-%0A02511-3_28
Haus ein, S., Bowman, T. D., & Cos as, R. (2016). In e p e ing “al me ics”: iewing ac s on
social media h ough he lens o ci a ion and social heo ies. In C. R. Sugimo o (Ed.),
Theo ies o In o me ics and Schola ly Communica ion (pp. 372–406). De G uy e Be lin,
Bos on.
Holmbe g, K., Bowman, S., Bowman, T., Didegah, F., & Ko elainen, T. (2019). Wha Is
Socie al Impac and Whe e Do Al me ics Fi in o he Equa ion? Jou nal o Al me ics.
h ps://doi.o g/10.29024/joa.21
Ho obin, D. F. (1990). The Philosophical Basis o Pee Re iew and he Supp ession o
Inno a ion. JAMA, 263(10), 1438–1441.
h ps://doi.o g/10.1001/jama.1990.03440100162024
Jasano , S. (2010). A New Clima e o Socie y. Theo y, Cul u e & Socie y, 27(2–3), 233–253.
h ps://doi.o g/10.1177/0263276409361497
Kassab, O., Bo nmann, L., & Haunschild, R. (2020). Can al me ics e lec socie al impac
conside a ions?: Explo ing he po en ial o al me ics in he con ex o a sus ainabili y
science esea ch cen e . Quan i a i e Science S udies, 1(2), 792–809.
h ps://doi.o g/10.1162/qss_a_00032
24
Kildu , M., Meh a, A., & Dunn, M. B. (2011). F om Blue Sky Resea ch o P oblem Sol ing:
A Philosophy o Science Theo y o New Knowledge P oduc ion. Academy o
Managemen Re iew, 36(2), 297–317. h ps://doi.o g/10.5465/am .2009.0164
La ue, P. (2021). The design and implemen a ion o mission-o ien ed inno a ion policies.
OECD Science, Technology and Indus y Policy Pape s No 100.
h ps://doi.o g/h ps://doi.o g/h ps://doi.o g/10.1787/3 6c76a4-en
Lecke , M. (2021). (E-) Valua i e Me ics as a Con es ed Field: A Compa a i e Analysis o
he Al me ics- and he Leiden Mani es o. Scien ome ics, 126(12), 9869–9903.
h ps://doi.o g/10.1007/s11192-021-04039-1
Libe a o e, A., Bowke , E., MacLeod, C. J., Spu , E., & Longnecke , N. (2018). Social Media
as a Pla o m o a Ci izen Science Communi y o P ac ice. Ci izen Science: Theo y and
P ac ice. h ps://doi.o g/10.5334/cs p.108
MacRobe s, M. H., & MacRobe s, B. R. (2018). The mismeasu e o science: Ci a ion
analysis. Jou nal o he Associa ion o In o ma ion Science and Technology, 69(3), 474–
482. h ps://doi.o g/h ps://doi.o g/10.1002/asi.23970
Ma in, B. R., & I ine, J. (1983). Assessing Basic Resea ch: Some Pa ial Indica o s o
Scien i ic P og ess in Radio As onomy. Resea ch Policy, 12(2), 61–90.
h ps://doi.o g/10.1016/0048-7333(83)90005-7
Me on, R. K. (1973). The sociology o science: Theo e ical and empi ical in es iga ions.
Uni e si y o Chicago P ess.
Me on, R. K. (1988). The Ma hew E ec in Science, II: Cumula i e Ad an age and he
Symbolism o In ellec ual P ope y. Isis, 79(4), 606–623. h ps://doi.o g/10.1086/354848
Mi owski, P. (2018). The u u e(s) o open science. Social S udies o Science, 48(2), 171–203.
h ps://doi.o g/10.1177/0306312718772086
Moed, H. F. (2005). Ci a ion analysis in esea ch e alua ion. Sp inge .
25
Moed, H. F., & Ga ield, E. (2004). In basic science he pe cen age o “au ho i a i e”
e e ences dec eases as bibliog aphies become sho e . Scien ome ics, 60(3), 295–303.
h ps://doi.o g/10.1023/B:SCIE.0000034375.39385.84
Mongeon, P. (2018). Using social and opical dis ance o analyze in o ma ion sha ing on social
media. P oceedings o he Associa ion o In o ma ion Science and Technology, 55(1),
397–403. h ps://doi.o g/h ps://doi.o g/10.1002/p a2.2018.14505501043
Mo on, S. (2015). P og essing esea ch impac assessmen : A ‘con ibu ions’ app oach.
Resea ch E alua ion, 24(4), 405–419. h ps://doi.o g/10.1093/ ese al/ 016
NASEM. (2017). Communica ing Science E ec i ely: A Resea ch Agenda. Washing on.
h ps://doi.o g/h ps://doi.o g/10.17226/23674.
Newman, M. E. J. (2004). Coau ho ship ne wo ks and pa e ns o scien i ic collabo a ion.
P oceedings o he Na ional Academy o Sciences o he Uni ed S a es o Ame ica,
101(suppl 1), 5200–5205. h ps://doi.o g/10.1073/pnas.0307545100
No s öm, A. V, C i ano ic, C., Lö , M. F., Wes , S., Wybo n, C., Bal ane a, P., Bedna ek, A.
T., Benne , E. M., Biggs, R., de B emond, A., Campbell, B. M., Canadell, J. G.,
Ca pen e , S. R., Folke, C., Ful on, E. A., Ga ney, O., Gelcich, S., Jou ay, J.-B., Leach,
M., … Ös e blom, H. (2020). P inciples o knowledge co-p oduc ion in sus ainabili y
esea ch. Na u e Sus ainabili y, 3(3), 182–190. h ps://doi.o g/10.1038/s41893-019-0448-2
O duña-Malea, E., & Cos as, R. (2023). A Scien ome ic-inspi ed amewo k o analyze
Eu ekAle ! P ess eleases. In I. B oe , S. Lemke, A. Maza akis, I. Pe e s, & Z.-W.
Ch is ian (Eds.), The science-media in e ace: on he ela ion be ween in e nal and
ex e nal science communica ion (pp. 1–27). De G uy e .
O duña-Malea, E., & Lopezosa, C. (2024). Unco e ing he po en ial o Twi ch as a sou ce o
social media me ics. Fi s Monday, 29(1 SE-). h ps://doi.o g/10.5210/ m. 29i1.13214
Pea ce, W., Holmbe g, K., Hells en, I., & Ne lich, B. (2014). Clima e Change on Twi e :
