S ein, Ca olin; Teubne , Timm; Mo ana, S e an
A icle — Published Ve sion
Designing a con e sa ional agen o suppo ing da a
explo a ion in ci izen science
Elec onic Ma ke s
P o ided in Coope a ion wi h:
Sp inge Na u e
Sugges ed Ci a ion: S ein, Ca olin; Teubne , Timm; Mo ana, S e an (2024) : Designing a
con e sa ional agen o suppo ing da a explo a ion in ci izen science, Elec onic Ma ke s, ISSN
1422-8890, Sp inge , Be lin, Heidelbe g, Vol. 34, Iss. 1,
h ps://doi.o g/10.1007/s12525-024-00705-3
This Ve sion is a ailable a :
h ps://hdl.handle.ne /10419/315791
S anda d-Nu zungsbedingungen:
Die Dokumen e au EconS o dü en zu eigenen wissenscha lichen
Zwecken und zum P i a geb auch gespeiche und kopie we den.
Sie dü en die Dokumen e nich ü ö en liche ode komme zielle
Zwecke e iel äl igen, ö en lich auss ellen, ö en lich zugänglich
machen, e eiben ode ande wei ig nu zen.
So e n die Ve asse die Dokumen e un e Open-Con en -Lizenzen
(insbesonde e CC-Lizenzen) zu Ve ügung ges ell haben soll en,
gel en abweichend on diesen Nu zungsbedingungen die in de do
genann en Lizenz gewäh en Nu zungs ech e.
Te ms o use:
Documen s in EconS o may be sa ed and copied o you pe sonal
and schola ly pu poses.
You a e no o copy documen s o public o comme cial pu poses, o
exhibi he documen s publicly, o make hem publicly a ailable on he
in e ne , o o dis ibu e o o he wise use he documen s in public.
I he documen s ha e been made a ailable unde an Open Con en
Licence (especially C ea i e Commons Licences), you may exe cise
u he usage igh s as speci ied in he indica ed licence.
h p://c ea i ecommons.o g/licenses/by/4.0/
Vol.:(0123456789)
Elec onic Ma ke s (2024) 34:23
h ps://doi.o g/10.1007/s12525-024-00705-3
RESEARCH PAPER
Designing acon e sa ional agen o suppo ing da a explo a ion
inci izen science
Ca olinS ein1,2 · TimmTeubne 3,4· S e anMo ana5
Recei ed: 31 July 2023 / Accep ed: 14 Feb ua y 2024 / Published online: 27 Ma ch 2024
© The Au ho (s) 2024
Abs ac
Da a is ubiqui ous in oday’s digi ized socie y. Howe e , access o and li e acy in handling da a plays a pi o al ole in de e -
mining who can bene i om i and who can use—o po en ially misuse—i . To comba inequali ies and add ess issues such
as misin o ma ion, i is essen ial o enable ci izens o e ec i ely access and unde s and da a wi hin hei local ecosys ems.
To add ess his challenge, we ocus on he case o ci izen science and p opose using a con e sa ional agen o suppo da a
explo a ion and lowe ba ie s o ci izen engagemen in esea ch p ojec s. Using a design science esea ch app oach, we
de i e design p inciples and de elop a p o o ypical a i ac . Mo eo e , we conduc an expe imen al e alua ion, demons a -
ing s ong in e es among ci izens o pa icipa e in scien i ic da a analysis and ha con e sa ional agen s hold g ea po en ial
in inc easing da a li e acy.
Keywo ds Con e sa ional agen s· Ci izen science· Design science esea ch
JEL Classi ica ion O35· O36· I24
In oduc ion
The eme gence o he digi al e a has undeniably ampli-
ied he p o ound impac o da a on all aspec s o ou li es.
Technological ad ancemen s ha e enabled he collec ion and
s o age o la ge da ase s (Cla ke, 2016; Twidale e al., 2013).
Da a-d i en business models mo i a e companies o collec
and analyze inc easing amoun s o da a, in some cases e en
a he expense o hei cus ome s’ in e es s (T zaskowski,
2022). While he inc eased a ailabili y o da a can lead o
new insigh s and be e decisions, i also accen ua es issues
o inequali y and exploi a ion (D’Ignazio, 2022). Owne ship
and li e acy o da a a e c i ical ac o s in de e mining who
can e ec i ely u ilize da a o hei ad an age (D’Ignazio,
2022). This becomes pa icula ly conce ning as da a and i s
p oduc s can be misused o pe sonal, poli ical, o economic
easons (Ca mi e al., 2020; Pullinge , 2021; T zaskowski,
2022). The e ol ing landscape o gene a i e AI poses u -
he challenges wi h he p oli e a ion o disin o ma ion in
he digi al ealm (Hanley & Du ume ic, 2023). In his con-
ex , da a li e acy becomes essen ial, no jus o ac i ely
engaging in public deba es and decision-making (Deb uyne
e al., 2021; Rade mache , 2021; Schülle e al., 2019) bu
also o na iga ing he digi al landscape in gene al (Ca mi
e al., 2020). Despi e i s e iden signi icance, a subs an-
ial po ion o he popula ion s ill lacks adequa e da a li -
e acy, elega ing hem o passi e “da a subjec s” a he han
empowe ed da a use s (D’Ignazio, 2022). This da a li e acy
di ide pe pe ua es inequali ies and denies indi iduals he
Responsible Edi o : Al ed Zimme mann
* Ca olin S ein
[email p o ec ed]
Timm Teubne
[email p o ec ed]
S e an Mo ana
s e [email p o ec ed]
1 FZI Resea ch Cen e o In o ma ion Technology,
Haid-Und-Neu-S aße 10-14, 76131Ka ls uhe, Ge many
2 KIT Ka ls uhe Ins i u e o Technology,
Kaise s aße 89-93, 76133Ka ls uhe, Ge many
3 TUB Technical Uni e si y o Be lin,
S aße des 17. Juni 135, 10623Be lin, Ge many
4 Eins ein Cen e Digi al Fu u e,
Wilhelms aße 67, 10117Be lin, Ge many
5 Saa land Uni e si y,
Campus C3 1, 66123Saa b ücken, Ge many
Elec onic Ma ke s (2024) 34:2323 Page 2 o 18
agency o bene i om he da a-d i en landscape (D’Ignazio,
2022). Consequen ly, he ques o e ec i e coun e meas-
u es becomes a c i ical pu sui . A p omising use case in
his ega d is ci izen science (CS) (Twidale e al., 2013),
e e ing o he pa icipa ion o non-p o essionals in scien-
i ic esea ch ac i i ies (Shi k e al., 2012). His o ically, CS
s i ed o democ a ize science and coun e ac social ineq-
ui y (I win, 1995). Al hough pa icipa o y ac i i ies a y,
in many CS p ojec s, pa icipan s can access and wo k wi h
scien i ic da a (Na ional Academies o Sciences & Medi-
cine, 2018). I is hence a na u al i when hinking abou
con eying da a li e acy and shi ing modes o powe and
agency. Cu en ly, howe e , se e al ba ie s p e en he ull
ealiza ion o educa ional bene i s. Fi s , while many CS
p ojec s enable pa icipa ion in da a collec ion, pa icipa-
ion in consecu i e explo a ion and in e p e a ion o da a
is spa se (Monzón Al a ado e al., 2020). The complexi y
o con iden iali y o da a and asks can educe he o e o
ac i i ies (Kloe ze e al., 2021). Second, esea che s and
p ojec ini ia o s a e limi ed in esou ces, such as unding
and ime (Kloe ze e al., 2021; Wald e al., 2016) necessa y
o o ganize pa icipa ion and suppo . This makes cu en
educa ional CS ools such as (pee ) men o ing, u o ials and
ainings, o cu iculums po en ially unsui able as hey imply
addi ional e o s o esea che s o he communi y (Na ional
Academies o Sciences & Medicine, 2018). While ad iso y
bodies call o da a li e acy o be ac i ely add essed in CS
p ojec s (Na ional Academies o Sciences & Medicine,
2018), gi en he cu en challenges, app op ia e solu ions
mus i s be explo ed. Speci ically, o succeed on a la ge
scale, a lexible ye au oma ed suppo is equi ed. The e-
o e, we p opose ha con e sa ional agen s (CAs) migh be
sui able ools o his ask. CAs enable he p o ision o sup-
po and in o ma ion cos -e ec i ely (K ale e al., 2021)
and a e used in many educa ional se ings (Okonkwo &
Ade-Ibijola, 2021). They can inc ease lea ne s’ mo i a ion
and enable s uden s o access con en o ecei e help swi ly
(Okonkwo & Ade-Ibijola, 2021). As a suppo ool o da a
explo a ion, a CA could enable ci izens o pa icipa e in his
esea ch s ep wi hou p oducing addi ional wo k o cos s o
ini ia o s, such as pe sonal aining o men o ing. Also, i
would be a mo e scalable and cons an ly a ailable solu ion,
quickly p o iding ci izen scien is s wi h in o ma ion and
assis ance o enable hei pa icipa ion and lea ning. Ne -
e heless, compa ed o u o ials and cu iculums, CAs can
p o ide pe sonalized suppo ha can adap o he needs o
he indi idual ci izen.
CAs equi e conscious design o i he audience’s and he
domain’s idiosync a ic equi emen s. Cu en esea ch on
CA design and u iliza ion encompasses aspec s o educa ion
(e.g., Okonkwo & Ade-Ibijola, 2021; Pé ez e al., 2020),
wo king wi h da a (e.g., Alaaeldin e al., 2021; Na echania
e al., 2021), and CS (e.g., Holowka e al., 2021; Ta anapou
e al., 2019). Howe e , he in e sec ion o hese h ee
opics emains a esea ch gap. In pa icula , while CAs
o collec ing (e.g., Holowka e al., 2021; Lia e al., 2023;
Ta anapou e al., 2019) and accessing da a (e.g., Na echania
e al., 2021; Neumaie e al., 2017; Simud e al., 2020) ha e
ecei ed some schola ly a en ion, he consecu i e use case
o analyzing da a (i.e., an in eg al pa o s eng hen da a
li e acy) has no been explo ed ye . We hence seek o answe
he ollowing esea ch ques ion:
RQ: How should a con e sa ional agen be designed o
suppo da a explo a ion in ci izen science applica ions?
We add ess he esea ch ques ion by applying he design
science esea ch (DSR) app oach. Beyond con eying da a
li e acy, we iden i y he need o mo i a ion and empowe -
men o ci izens in he li e a u e. Based on his, we de i e
design p inciples o a CA suppo ing ci izen pa icipa ion in
da a explo a ion and implemen hem in a p o o ypical a i-
ac . E alua ing he p o o ype in an expe imen al s udy, we
ind ha using he CA can enhance da a li e acy and analysis
pe o mance among inexpe ienced use s. Wi h his esea ch,
we aim o con ibu e o he ongoing e o s in educing in o -
ma ion dispa i ies and ensu ing ha da a is le e aged o
socie al bene i . We u he iden i y oppo uni ies o u u e
esea ch by examining he limi a ions and challenges o he
a i ac and ou esea ch app oach.
Rela ed wo k
As he ounda ion o ou wo k, we e iew he li e a u e on
da a li e acy and i s ela ionship o CS. We examine exis -
ing e o s o suppo ci ic engagemen in da a analysis and
explo e he po en ial o CAs in he domains o da a li e acy,
educa ion, and CS o guide ou CA design.
Da a li e acy
Da a li e acy can be e e ed o as “ he abili y o ead, w i e
and communica e da a in con ex , including an unde s anding
o da a sou ces and cons uc s, analy ical me hods and
echniques applied, and he abili y o desc ibe he use case,
applica ion, and esul ing alue” (Pane a, 2021, pa a. 3).
Roo ing back o he no ion o in o ma ion li e acy, wi h an
inc easing amoun o da a and he eme gence o mo e and
mo e da a-d i en p o essions, da a li e acy eme ged as a
buzzwo d in esea ch and popula p ess (Schülle e al., 2019).
While Ga ne ’s de ini ion o da a li e acy ocuses p ima ily
on desc ibing a skill se (Pane a, 2021), o he de ini ions
also emphasize he abili y and mo i a ion o use hese skills
in one’s en i onmen . Bha ga a e al. (2015) de ine i as
“ he desi e and abili y o cons uc i ely engage in socie y
h ough and abou da a” (p. 24), and Schülle e al. (2019)
desc ibe i as an abili y needed o na iga e he digi alized
Elec onic Ma ke s (2024) 34:23 Page 3 o 18 23
wo ld and make in o med decisions. The e o e, e ec i e da a
li e acy p omo ion should no only ocus on skills bu also
on empowe ing and mo i a ing lea ne s o apply hese skills
in hei espec i e con ex s (Bha ga a e al., 2015). To guide
eaching and e alua ion app oaches o da a li e acy, Schülle
e al. (2019) de eloped a da a li e acy amewo k ha can be
ailo ed o di e en needs and equi emen s. They subdi ide
da a li e acy in o six co e compe encies: (1) he es ablishmen
o a da a cul u e, (2) he p o ision o da a, (3) he exploi a ion
o da a, (4) esul in e p e a ion, (5) in e p e a ion o da a, and
(6) he de i a ion o ac ions (Schülle e al., 2019). While he
amewo k p o ides in o ma ion abou he con en equi ed o
p omo e da a li e acy, i does no add ess how i can be augh .
Examples o eaching app oaches o da a li e acy comp ise
in-pe son o ma s such as wo kshops (e.g., D’Ignazio, 2022;
Deb uyne e al., 2021), school ini ia i es (e.g., Bha ga a
e al., 2016; Gould, 2021), o online o ma s such as o ums,
quizzes, and online classes (Jayawick ama e al., 2020). In
addi ion, many digi al ools acili a e da a- ela ed asks, such
as da a collec ion, p ocessing, and isualiza ion. D’Ignazio
and Bha ga a (2016) ha e mapped ools such as Excel,
ca oDB, o in og .am in iew o hei lexibili y and expe ise
equi emen s. Howe e , hey poin ed ou ha cu en ools
emphasize ou pu c ea ion a he han lea ning. They de i e
ou design p inciples o pedagogical lea ning ools: a ge ed
ocus, guidance, in i ing design, and ool expandabili y
(D’Ignazio & Bha ga a, 2016). These p inciples should
ensu e ha ools ease ba ie s o lea ning and quickly ge use s
s a ed wi h ac i i ies. While being in i ed o ollow appealing
i s ac i i ies, use s should ind addi ional in o ma ion
on mo e demanding p ac ices (D’Ignazio & Bha ga a,
2016). O he au ho s s ess ha eaching app oaches should
encompass mul iple pa hways o use s o choose om—
acco ding o hei needs (e.g., Bha ga a e al., 2015). These
should be agile, adap i e, and ocus on wha is e ec i e and
meaning ul o he lea ne s, such as wo king wi h communi y
da a (Bha ga a e al., 2015; D’Ignazio, 2022).
Ci izen science
Ci izen Science is de ined as “ he gene al public engagemen
in scien i ic esea ch ac i i ies when ci izens ac i ely con ib-
u e o science ei he wi h hei in ellec ual e o o su ound-
ing knowledge o wi h hei ools and esou ces” (Socien ize,
2014, p. 6). O iginally used in he na u al sciences, oday
CS has p o en use ul ac oss many di e en ields (Pe ibone
e al., 2017). Wi h he expansion o CS, he he e ogenei y
o pa icipa ion app oaches has also inc eased (Shi k e al.,
2012; Spasiano e al., 2021). The mos common p ojec ypes
a e con ibu o y p ojec s ha ocus on pa icipa o y da a col-
lec ion (Bowse e al., 2020; Monzón Al a ado e al., 2020).
The pa icipa o y analysis and in e p e a ion o da a is less
common and usually occu s in co-c ea ed o collegial CS
p ojec s (Shi k e al., 2012). Howe e , ci izens ha e inc eased
access o aw da a, o ins ance, h ough open (go e nmen )
da a pla o ms. They could suppo public ins i u ions in
d awing impo an insigh s, when gi en access o (gami-
ied) oolki s suppo ing da a u iliza ion (K ishnamu hy &
Awazu, 2016; Simono ski e al., 2022; Wi z e al., 2022).
On he CS pla o m Zooni e se (www. zooni e se. o g), o
ins ance, a s ong ocus in da a analysis is pu on pa ici-
pa o y image classi ica ion (Bonney e al., 2016; Simpson
e al., 2014). Mo eo e , o indi idual CS p ojec s, digi al
ools such as Google Sp eadshee a e p epa ed bu o en used
only in a class oom se ing, whe e addi ional suppo and
eaching a e p o ided (e.g., Kjel ik & Schul heis, 2019; Shah
& Ma inez, 2016). Wi hin CS p ojec s, lea ning happens
ei he on a mic o le el (e.g., h ough ac i e pa icipa ion and
he execu ion o asks) o on a mac o le el (e.g., by sha ing
ideos o online u o ials) (Jenne e al., 2016). Howe e ,
he educa ion o ci izen scien is s and, hus, he p o ision
o educa ional ools a e a ely he ocus o CS p ojec s. In
a s udy explo ing pa icipan mo i a ion and e en ion in
digi al CS p ojec s, Wald e al. (2016) epo ed ha o mos
p ojec s, scien i ic ou comes we e he ocus while “educa-
ional and social bene i s […] we e inciden al” (p. 562). Fo
esea che s, he main ba ie s o lea ning a e he necessa y
empo al, echnical, o mone a y esou ces, as well as b eak-
ing down complex asks (Kloe ze e al., 2021). Con e sely,
pa icipan s can be p e en ed om lea ning due o a lack o
con idence, skills, money, o ime (Kloe ze e al., 2021). In
addi ion, p ojec design i sel can nega i ely in luence lea n-
ing by including oo li le eedback o in e ac ion (Kloe ze
e al., 2021). These obs acles should be a s a ing poin o
echnical solu ions suppo ing pa icipa o y da a explo a ion.
Con e sa ional agen s
CAs a e applica ions ha allow use s o in e ac wi h hem
in a na u al language and can ei he be ex o speech-based
(Janssen e al., 2020; Rapp e al., 2021). They a e also dis-
cussed in ce ain domains unde he e ms cha bo o cha e -
bo (Bi ne e al., 2019). O e ing au oma ion whe e p io y
human esou ces a e needed, CAs can be a low- h eshold
solu ion, leading o la ge cos educ ions (K ale e al., 2021).
Howe e , main aining use sa is ac ion can pose majo chal-
lenges. S udies on cus ome se ice cha bo s indica e ha
ac o s such as p oblem esolu ion, answe p ecision, and
conc e eness d i e cus ome sa is ac ion (e.g., K ale e al.,
2021; an de Goo e al., 2021), while e o s and a lack o
unc ionali y can quickly de e io a e i (e.g., an de Goo
e al., 2021). Likewise, s udies on CAs in he wo kplace sug-
ges ha CA adop ion depends on use cha ac e is ics such
as indi idual ech sa iness (e.g., Gkinko & Elbanna, 2023).
The possible applica ion ields o CAs ange om econom-
ics (e.g., inance o e-comme ce) o pe sonal applica ions
Elec onic Ma ke s (2024) 34:2323 Page 4 o 18
such as heal h o emo ional suppo (Rapp e al., 2021). I
has been shown ha good CA design highly depends on
he domain i is buil o (e.g., Bi ne e al., 2019). Fu he
esea ch on he ans e abili y o design knowledge be ween
con ex s is necessa y (Diede ich e al., 2022). Fo ins ance,
while he usage o social cues is encou aged in some CA
applica ions (e.g., Holowka e al., 2021; Ta anapou e al.,
2019), i can ha e de imen al e ec s when he eliabili y
o in o ma ion is essen ial (S iegli z e al., 2022). Thus,
a one-size- i s-all app oach o CA design is un ealis ic—
con ex , s akeholde s, and unique alue p oposi ions mus
be conside ed (Janssen e al., 2020). To guide he design
o a CA o suppo ci izens in da a explo a ion, di e en
domains p o ide in e es ing insigh s. In he ollowing, we
shed ligh on insigh s om he applica ion o CAs o edu-
ca ion, (big) da a- ela ed wo k and CS (see also Table1 in
heelec onicsupplemen a y ma e ial).
In he domain o educa ion, esea ch di e en ia es
be ween eaching- and se ice-o ien ed CAs. While he
o me desc ibes CAs a ge ing knowledge gene a ion, se -
ice-o ien ed CAs p o ide adminis a i e se ices, such as
in oduc o y o lib a y se ices (Pé ez e al., 2020). When
in e ac ing wi h lea ne s, CAs usually ac as “ eache , s u-
den , o colleague” (Tamayo-Mo eno & Pé ez-Ma ín, 2016,
p. 1). In his ole, CAs ha e p o en bene icial as hey allow
o in eg a ing mul iple con en in o one ool and pa allel
access by mul iple use s (Okonkwo & Ade-Ibijola, 2021).
The possibili y o ecei ing immedia e help on demand
is con enien wi h posi i e e ec s on lea ning mo i a ion
(Okonkwo & Ade-Ibijola, 2021). CAs also p o ed sui able
o closing lea ning gaps be ween mains eam lea ne s and
lea ne s om ce ain mino i y g oups (Pé ez e al., 2020).
Howe e , h ough a s uc u ed li e a u e e iew, Pé ez e al.
(2020) ha e iden i ied bo edom and use us a ion (e.g.,
h ough leng hy messages and inadequa e eplies) as com-
mon impedimen s. Teaching CAs can a ge a ious opics
and domains, wi h language lea ning being a p ominen use
case (Pé ez e al., 2020). Ano he use case, close ela ed o
da a li e acy, is ma h educa ion whe e CAs ha e been used
(e.g., Anh & Ngan, 2021; Nguyen e al., 2019).
CAs in da a- ela ed wo k en i onmen s usually ocus
on da a p o ision and depic ion o non- echnical skilled
employees (e.g., Alaaeldin e al., 2021; Na echania e al.,
2021; Simud e al., 2020). CAs can conduc asks such as
gene a ing da abase que ies and isualiza ions in o based on
na u al language (e.g., Hoon e al., 2020; Na echania e al.,
2021; Neumaie e al., 2017; Simud e al., 2020). They can
also suppo decision-making by explaining analy ic ools
and key pe o mance indica o s o a gi en da ase (e.g.,
Alaaeldin e al., 2021). Ano he impo an applica ion is he
iden i ica ion o ele an da ase s, domain-speci ic scien-
i ic ools, and me hods (e.g., Keyne e al., 2019; Neumaie
e al., 2017; Zhang e al., 2019). O e all, CAs in da a- ela ed
wo k en i onmen s mainly ocus on o e coming mode n
da abases’ echnical complexi ies h ough na u al language
in e aces. Since suppo ing he unde s anding o da a and
analysis me hods is no he ocus o publica ions, undamen-
al da a li e acy emains a p e equisi e o use s.
Wi hin he domain o CS, he usage o CAs o di e en
ac i i ies is no ye a common p ac ice. Fi s and o emos ,
CAs ha e been used o quan i a i e and quali a i e da a
collec ion in CS p ojec s (e.g., Holowka e al., 2021; Isacco
e al., 2018; Lia e al., 2023; Tallyn e al., 2018; Ta anapou
e al., 2019). They enable pa icipan s o answe ques ion-
nai es, upload ex , pic u es, o geo ags (e.g., Isacco e al.,
2018; Lia e al., 2023; Tallyn e al., 2018) and can, in e u n,
p o ide guidance o encou agemen and suppo o sha e
da a di ec ly wi h expe s o he communi y (e.g., Holowka
e al., 2021). Ad an ages o hei use in da a collec ion can
include pe sonalized eedback and he abili y o conduc
u he inqui ies when obse a ions a e incomple e (Po -
ela, 2021). Addi ionally, hey can p o ide ci izens wi h da a
o isualiza ions (Po ela, 2021). O he wo k explo es he
ad an ages o CAs acili a ing he idea ion p ocess by col-
lec ing, s uc u ing, and p esen ing ideas (e.g., Ta anapou
e al., 2019) o suppo ing he communi y and i s in e ac ion
(e.g., A h eya e al., 2018; Po ela, 2021). O e all, he po en-
ial o using CAs o CS seems o be no ye exploi ed. Fo
example, we could no iden i y li e a u e p esen ing a CA
used o aining ci izen scien is s, al hough he sui abili y
o CAs o educa ional pu poses has been p o en in o he
domains (Okonkwo & Ade-Ibijola, 2021).
Resea ch gap
The ela ed li e a u e on da a li e acy, CS, and CAs o e s
aluable insigh s in o he possibili ies and challenges associ-
a ed wi h designing ac i i ies and ools o enhancing da a
li e acy. Howe e , i unde sco es a signi ican esea ch gap
a he in e sec ion o hese opics: he design o ools o
ac i e pa icipa ion in da a explo a ion. While he da a li -
e acy li e a u e p o ides c ucial insigh s in o ool design
o lea ne s (e.g., D’Ignazio, 2022; Schülle e al., 2019),
i emphasizes he need o mo e lea ning-o ien ed ools
embedded in a meaning ul con ex o he use (Bha ga a
e al., 2015; D’Ignazio & Bha ga a, 2016). The CS li e a u e
add esses his con ex and discusses pa icipan s’ lea ning
and ools o suppo p ojec s (e.g., Jenne e al., 2016; Liu
e al., 2021; Na ional Academies o Sciences & Medicine,
2018). Howe e , i e eals nume ous challenges in in eg a -
ing educa ional componen s (e.g., Kloe ze e al., 2021;
Wald e al., 2016) and ha he da a analysis s ep is o en no
add essed. The educa ional CA li e a u e gene ally discusses
oppo uni ies and challenges in using CAs o eaching (e.g.,
Okonkwo & Ade-Ibijola, 2021; Pé ez e al., 2020), bu in
he speci ic con ex o wo king wi h da a, he ocus emains
Elec onic Ma ke s (2024) 34:23 Page 5 o 18 23
p ima ily on disco e ing da ase s (e.g., Keyne e al., 2019;
Neumaie e al., 2017; Zhang e al., 2019) o making hem
accessible (e.g., Hoon e al., 2020; Na echania e al., 2021;
Neumaie e al., 2017). Likewise, CAs in CS do no ocus
on acili a ing da a analysis bu a he cen e on quali a i e
o quan i a i e da a collec ion (e.g., Holowka e al., 2021;
Lia e al., 2023; Tallyn e al., 2018). Addi ionally, hey do
no exploi hei eaching capabili ies.
O e all, he exis ing li e a u e p o ides c ucial insigh s
o designing CAs suppo ing ci izens in da a explo a ion,
ye speci ic design guidelines o his use case a e missing.
Conside ing he challenges on he ans e abili y o design
knowledge ac oss con ex s (Bi ne e al., 2019; Diede ich
e al., 2022; Janssen e al., 2020), his ep esen s a signi ican
esea ch gap ha should be u he explo ed and in es iga ed.
Resea ch app oach
To answe ou esea ch ques ion, we conduc ed a esea ch p o-
jec ollowing he DSR app oach o Pe e s e al. (2020). Fo
a “p oblem-cen e ed app oach” (Pe e s e al., 2020, p. 56),
he me hodology includes six s eps, s a ing wi h iden i ying a
p oblem and de i ing a mo i a ion o i s solu ion (see Fig.1).
Fi s , o elabo a e on he p oblem o sol e, we assessed he
cu en s a e o esea ch by e iewing ela ed li e a u e in he
ield o da a li e acy, CS, and CAs and pe o med a s akeholde
analysis based on insigh s om hese domains. In addi ion, we
ca ied ou an expe wo kshop on da a analysis conduc ion
wi h 12 expe s and ad anced p ac i ione s, combining ideas
om wo es ablished equi emen s elici a ion me hods “In o-
spec ion” and “B ains o ming” (Sha ma & Pandey, 2013).
While in an in ospec ion, expe s elici he use needs based on
hei domain knowledge; in b ains o ming, pa icipan s om
di e en s akeholde g oups a e in i ed o collec i ely gene a e
ideas (Pae sch e al., 2003). The expe wo kshop acili a ed
in ospec ion h ough a hink-aloud session abou “conduc -
ing a da a analysis.” Think-aloud sessions, ypically known
om usabili y es ing, enable esea che s o gain insigh s in o
pa icipan s’ hough p ocesses by neu ally obse ing hem
speaking ou hei hough s aloud while wo king on a gi en
ask (E icsson & Simon, 1984; Fan e al., 2020). A e he
indi idual hink-aloud session, b ains o ming was conduc ed
in he expe wo kshop as an open discussion be ween all pa -
icipan s, guided by he componen s o he da a li e acy ame-
wo k (Schülle e al., 2019). The app oach o he conduc o
he expe wo kshop is u he desc ibed in he “Expe wo k-
shop (ac i i y 1)” sec ion. Second, he ollowing ac i i y o he
DSR app oach comp ised he de ini ion o solu ion objec i es
by de i ing ei he quan i a i e o quali a i e equi emen s o
he a i ac (Pe e s e al., 2020). The e o e, we ansla ed he
esul s om he i s ac i i y in o a omic use needs as solu-
ion objec i es and posi ioned hem in he ela ed li e a u e.
Thi dly, we app oached he ac ual design and de elopmen
phase by ins an ia ing ou a i ac . The wo asks o his phase
we e ou lining he necessa y unc ion and design equi emen s
be o e p ac ically c ea ing he a i ac (Pe e s e al., 2020).
Using ou use needs, we de e mined design speci ica ions, ol-
lowing he schema by G ego e al. (2020), and implemen ed
hem in a p o o ypical ins an ia ion. The app oach o speci-
ying he design p inciples and implemen ing he a i ac is
u he desc ibed in he “Design p inciples o a con e sa ional
agen o public pa icipa ion in da a analysis” and “A i ac ”
sec ions, espec i ely. To close he i s DSR cycle, he a i ac
should be demons a ed and e alua ed. We co e ed hese s eps
simul aneously by designing and conduc ing a inal expe i-
men wi h 30 pa icipan s guided by bes p ac ices o e alua e
CAs and da a li e acy lea ning (Pé ez e al., 2020; Schülle
e al., 2019) and DSR a i ac s (Venable e al., 2016). Using
a be ween-subjec s design, he expe imen compa ed he da a
Fig. 1 O e iew o he DSR app oach
Elec onic Ma ke s (2024) 34:2323 Page 6 o 18
analysis pe o mances and (lea ning) expe iences o a pa ici-
pan g oup guided by he designed a i ac wi h he esul s o a
g oup ha ecei ed no explici guidance bu was allowed o use
exis ing ma e ial h ough he web. By combining quan i a i e
( ask pe o mance, empowe men , mo i a ion, pe cei ed lea n-
ing, use expe ience) and quali a i e (use expe ience, dialog
analysis) insigh s, he expe imen enabled us o e alua e he
a i ac comp ehensi ely. The app oach o he expe imen ’s
design, conduc ion, and e alua ion is desc ibed in de ail in he
“E alua ion” sec ion.
Designing acon e sa ional agen o public
pa icipa ion inda a analysis
P oblem awa eness andsolu ion objec i es
We se ou o design a CA capable o suppo ing da a explo a-
ion in CS p ojec s. This design endea o en ails unde s anding
he in icacies o da a li e acy and CS and applying his knowl-
edge o CAs. The e iew o ele an li e a u e has shown ha
os e ing da a li e acy equi es c ea i e eaching app oaches
ha conside he in e es s and eali ies o he a ge audience
(e.g., Bha ga a e al., 2015; D’Ignazio, 2022). A challenge in
his ega d is he complexi y o he abo e concep s and ha
da a li e acy is based on a se o compe encies a he han a
speci ic skill o echnique (Deb uyne e al., 2021; Schülle
e al., 2019). The in ended da a li e acy con en hus needs
o be b oken down o guide he design o he CA (ac i i y 1).
While da a collec ion is equen ly seen in CS p ojec s
(Bowse e al., 2020; Liu e al., 2021; Monzón Al a ado e al.,
2020), ew p ojec s use pa icipa o y da a analysis (beyond
classi ica ion asks). In gene al, budge s and ime cons ain s
limi such p ojec s o esea che s and ci izens (Kloe ze e al.,
2021; Wald e al., 2016). In addi ion, main aining momen-
um and keeping ci izens engaged and ac i e (beyond he
i s explo a ion) a e c ucial bu di icul (Wald e al., 2016).
The e o e, unde s anding ci izen scien is s is key o he design
o app op ia e suppo ools (ac i i y 2).
Expe wo kshop (ac i i y 1)
To unde s and he speci ic equi emen s o he CA (con en
and design), we in i ed 12 da a analys s i s o an indi idual
hink-aloud session and second o a b ains o ming session.
The pa icipan s consis ed o six Ph.D. candida es and six
Table 1 A omic use needs g ouped by pe spec i e and con as ed wi h ela ed li e a u e
Use needs de i ed om he wo kshop Concep s in he li e a u e
As a s uden , I wan o… U1 Unde s and he da a analysis p ocess Di e en ac i i ies in da a alue c ea ion (Schülle
e al., 2019)
U2 Unde s and he da ase (meaning, use ulness) Compe ence ob ain da a, p epa e da a (Schülle e al.,
2019)
U3 Know and apply me hods o da a cleaning
U4 Find a s a o he da a analysis Low en y poin o da a li e acy ools (D’Ignazio &
Bha ga a, 2016)
U5 Unde s and and selec a me hod o da a analysis Compe ence analyze da a, in e p e da a analysis,
isualize da a, in e p e da a isualiza ions (Schülle
e al., 2019)
U6 No e limi s and challenges o he da a analysis
U7 In e p e analysis esul s and c i ically ques ion indings
U8 Selec and design app op ia e da a isualiza ions
U9 Unde s and how o in e p e and check isualiza ions
U10 Ask ques ions CA’s answe ing s uden ’s ques ions (Okonkwo & Ade-
Ibijola, 2021)
U11 Decide on an analysis ac ion and pa h Enabling mul iple pa hways o lea ne s o choose om
(Bha ga a e al., 2015)
U12 Ge access o da a analy ics ools Knowledge and mas e y o ools as essen ial skills
(Schülle e al., 2019)
U13 Ge access o assis ance and help ul ma e ial CA’s in eg a ing mul iple con en s (Okonkwo & Ade-
Ibijola, 2021)
As a u o , I wan o… U14 S ee h ough he analysis p ocess Guided da a li e acy ools (D’Ignazio & Bha ga a, 2016)
U15 Poin ou missing compe encies and skills -
U16 P oceed wi h ques ions (answe , edi ec , collec ) CA’s answe ing s uden ’s ques ions (Okonkwo & Ade-
Ibijola, 2021)
U17 Fo m an in e ace o o he ma e ial/ so wa e Expandable da a li e acy ools (D’Ignazio & Bha ga a,
2016)
U18 Suppo di e en le els o p e-knowledge CAs p o iding indi idualized suppo (Okonkwo &
Ade-Ibijola, 2021)
Elec onic Ma ke s (2024) 34:23 Page 7 o 18 23
mas e s uden s majo ing in da a science- ela ed ields and
p esen ed hus ad anced o p o essional p ac i ione s in he
ield. Sessions we e conduc ed i ually using an online
con e ence ool mode a ed by one esea che , which had
some implica ions o he wo kshop conduc ion. In he
hink-aloud session, pa icipan s we e asked o sol e se e al
analy ical asks based on a gi en da ase . Ins ead o being
able o obse e hei ac ions physically, in he i ual hink-
aloud sessions, we used audio, ideo, and sc een sha ing o
enable as many insigh s as possible o he esea che . In he
ollowing g oup b ains o ming session, pa icipan s could
hen discuss app oaches and pi alls o da a explo a ion and
equi emen s o suppo based on hei own expe iences in
he hink-aloud session. Since in i ual compa ed o on-
si e g oups lowe social p esence can be a challenge o he
discussion quali y, we limi ed he size o he b ains o ming
g oups o h ee pa icipan s pe session o be e in eg a e he
indi idual pa icipan s and u ilized an online whi eboa d o
acili a e collabo a ion (Robe s e al., 2006). The esul s o
he expe wo kshop we e ansla ed in o a omic use needs
o s uden s in da a explo a ion and u o s in s uden sup-
po and con ex ualized wi h he li e a u e e iew esul s
(see Table1).
S akeholde analysis (ac i i y 2)
Mos CS ini ia i es ca e o b oad audiences (Spie s e al.,
2019). Common use cha ac e is ics include abo e-a e age
educa ion, abo e-a e age income, and abo e-a e age senio -
i y (Cia án Mac Domhnaill & Nolan, 2020; Na ional Acad-
emies o Sciences & Medicine, 2018). In addi ion, ci izen
scien is s end o “embody he cha ac e is ics o au onomy,
compe ence, and ela edness in hei hobby” (Jones e al.,
2018, p. 15). Howe e , CS emb aces he di e si y o pa -
icipan s, and p ojec o ganize s claim o s i e o mo e
di e si y in e ms o age, gende , and e hnici y (Na ional
Academies o Sciences & Medicine, 2018). Thus, he le el
o au onomy and knowledge can be assumed o be he e o-
geneous. Based on his and esea ch on da a li e acy (e.g.,
Logan, 2017; Wa son & Callingham, 2004), we dis inguish
he ollowing (b oad) use g oups o ha e in mind o he
CA a i ac : (1) beginne use s, (2) ad anced use s, and (3)
p o essional use s (see Table2).
Design p inciples o acon e sa ional agen
o public pa icipa ion inda a analysis
A design p inciple should include an aim, implemen e ,
and use ; a con ex ; mechanisms; and a a ionale (G ego
e al., 2020, p. 1634). We ollow his scheme and p opose
i e p inciples o he design o a CA o suppo in da a
analysis asks (con ex ), which can be used by esea che s
and de elope s (implemen e ) o c ea e so wa e suppo
o non-expe ci izen scien is s unde aking hei analy i-
cal ac i i ies (use s). Conside ing use needs U1, U11,
U14, and U18, he pla o m mus speci y a ce ain p ocess
s uc u e while he use emains ee o choose how o ol-
low his pa h. We ind guidance o his equi emen in he
design p inciples o Ta anapou e al. (2019), who s a e
ha a CA o idea c ea ion mus be able o ollow a gi en
con e sa ion low while s ill being able o lead he p ocess
ac i ely and D’Ignazio andBha ga a(2016), who unde -
lined, a da a li e acy ool should be guided. Addi ionally,
Po ela (2021) ad ises ha a CA should include ixed cha
commands o use o ien a ion. The e o e, we o mula e
he i s design p inciple as ollows:
DP1: In o de o s uc u e he analysis p ocess (aim), he
sys em should p o ide a so ed menu highligh ing he indi-
idual pa s o a da a analysis (mechanism), as his enables
he use o ge guidance on he p ocess and na iga e o a
speci ic opic o in e es ( a ionale).
Se e al use needs (U2, U4) exp ess ha beginne s’
en ance mus be eased. Thus, he sys em should “p o ide a
low en y poin ” (p. 87) o da a analysis (D’Ignazio & Bha -
ga a, 2016). Ne e heless, he knowledge needed o conduc
many such asks is comp ehensi e (U3, U5, U7, U8, U9).
The s akeholde analysis showed he need o accoun o
di e en use g oups, which is suppo ed by Bha ga a e al.
Table 2 Desc ip ion o use g oups o he CA a i ac
Use g oups Desc ip ion
Beginne use s No o li le knowledge abou da a li e acy (see idiosync a ic le el, in o mal le el (Wa son & Callingham, 2004))
New o CS o wi hou he ypical cha ac e is ics o an expe ienced ci izen scien is , equi ing guidance and explana ions
Ad anced use s Expe ienced ci izen scien is s o people expe ienced in da a li e acy wi h a undamen al unde s anding o he da a p ocess
(see con e sa ional le el; Logan, 2017), da a analysis, and isualiza ion me hods (see inconsis en and consis en non-
c i ical le el; Wa son & Callingham, 2004)
Knowledge is a he basic and incomple e o migh da e back a long ime ago, bu use s a e mo e au onomous and in o m
hemsel es o migh ha e speci ic ques ions
P o essional use s Familia wi h wo king wi h da a (see c i ical and c i ical ma hema ical le el; Wa son & Callingham, 2004) and wo king
la gely o comple ely au onomously
G oup is a he ou o scope o he CA
Elec onic Ma ke s (2024) 34:2323 Page 8 o 18
(2015), poin ing ou he necessi y o p o ide “mul iple pa h-
ways o people wi h di e en da a li e acy needs and capac-
i ies o in e ac wi hin a complex sys em” (p. 15). The e-
o e, he pla o m should equip use s wi h he app op ia e
backg ound knowledge based on hei needs and in e es s.
Ta anapou e al. (2019) o his end p opose a compa able
mechanism speci ying ha CAs should ha e he “capaci y
o summa ize […] in o ma ion […] and o e u he expla-
na ions, i eques ed” (p.8). We, he e o e, o mula e ou
second design p inciple as ollows:
DP2: The sys em should p o ide a ie ed knowledge
s uc u e (mechanism) o educa e he use e icien ly (aim),
as his enables he use o de e mine he dep h acco ding o
hei in e es s and skills ( a ionale).
Use needs U10 and U16 exp ess ha he use s should be
enabled o ge answe s o hei speci ic ques ions, which is a
common unc ionali y o eaching CAs (Okonkwo & Ade-
Ibijola, 2021). We, hus, o mula e he hi d design p inciple
as ollows:
DP3: The sys em should allow use s o en e ques ions
and p ocess hem (mechanism) o ge answe s o indi idual
ques ions (aim, a ionale).
Use needs U12, U13, and U17 imply ha he pla o m
should use exis ing eaching ma e ial. To his end, D’Ignazio
andBha ga a(2016) poin ou ha da a li e acy ools should
be expandable, b idging he pa hway o lea ne s o go om
one da a li e acy ool o he o he . We inco po a e hese ind-
ings in he ou h design p inciple:
DP4: To e icien ly educa e he use (aim), he sys em
should p o ide a combina ion o sel -de eloped and ex e nal
ma e ials h ough embedding o o wa ding (mechanism), as
use s ha e an in e es in a b oad o e o lea ning ma e ial
( a ionale).
Fu he mo e, he p esence o many pi alls (use needs
U6, U9, U15) equi es he pla o m o suppo use s in
unde s anding challenges and a oiding common mis akes.
We, he e o e, p opose ha :
DP5: The sys em should p o ide indica ions and wa n-
ings o challenges and common mis akes in ime (mecha-
nism) o p e en he use om ailing (aim, a ionale).
An o e iew o he design p inciples and hei de i a ion
om he a omic use needs a e ound in Table3.
A i ac
In he hi d phase o he DSR p ocess, he o mula ed design
p inciples a e ins an ia ed in an a i ac in he o m o a CA
p o o ype. The CA p o ides da ase -independen suppo o
beginne and ad anced use s in he p ocess o da a analysis. I
p o ides p ocess-o ien ed and knowledge-based ad ice h ough
messages, pic u es, links, and guidance along wo wo k lows:
Wo k low 1
The da a analysis (DA) wo k low o e s guidance o begin-
ne s and p o ides a menu o s eps (Fig.2), showing di e en
s eps o a ypical da a analysis p ocess (DP1). The menu
se es as a cen al poin o which he use e u ns wi hin he
low. The i s s ep o he DA p ocess (“Ge ing s a ed”)
e lec s DP2, DP4, and DP5. A e ecei ing in o ma ion
on how o ge s a ed, he use can eques mo e in o ma ion
(DP2) o b owse h ough ex e nal educa ion ma e ial (DP4).
To add ess DP5, he CA in i es use s o analyze hei da a
ac i ely. Upon he use ’s con i ma ion o p oceed, he bo
p o ides an o e iew o common mis akes conce ning he
ask he use has jus comple ed (DP5).
Wo k low 2
The ques ion and answe (Q&A) wo k low should a ac
use s wi h basic da a knowledge. He e, use s can speci y
opics o in e es by asking ques ions. Upon a eques , he
CA ei he ecognizes he ques ion as da ase -speci ic o
me hodological. In he o me case, he bo poin s ou ha
such ques ions a e ou o scope. In he la e case, he bo
p o ides an answe i i ecognizes he ques ion. I he ques-
ion is no ecognized, he CA o e s o o wa d he ques ion
o a supe ising esea che . Upon a i ma ion, he CA sends
ques ions and con ac de ails o an online sp eadshee , p i-
a ely accessible o he supe ising esea che .
Table 3 Design p inciples mapped o hei espec i e use needs
Design p inciples Associa ed use needs
DP1 P o ide a menu ha s uc u es he analy ics p ocess o he use o ge guidance o na iga e o a speci ic
opic
U1, U11, U14, U18
DP2 P o ide a ie ed knowledge s uc u e o le he use de e mine he dep h acco ding o hei in e es s and
skills
U2, U3, U4, U5, U7, U8, U9
DP3 Allow he use s o en e ques ions and p ocess hem by ei he ad hoc answe ing o o wa ding U10, U16
DP4 P o ide a combina ion o sel -de eloped and ex e nal ma e ials h ough embedding o o wa ding o edu-
ca e he use e icien ly
U12, U13, U17
DP5 P o ide indica ions and wa nings o challenges and common mis akes in ime U6, U9, U15
Elec onic Ma ke s (2024) 34:23 Page 15 o 18 23
knowledge. Applied o ou expe imen , his could mean he
amoun o knowledge p esen ed o he CA use s could ha e
nega i ely a ec ed he use ’s assessmen o aspec s such as
his o he eeling o empowe men o lea ning h ough he
analysis ac i i ies, compa ed o a use no exposed o his
knowledge. A hi d esea ch s ing could ocus on e alua ing
he CA in a eal-li e CS en i onmen , including assessing
who can use i and who is excluded. This pe spec i e would
be c i ical as CS s i es o inclusi i y (Na ional Academies
o Sciences & Medicine, 2018; So ensen e al., 2019), and
he unde - o o e - ep esen a ion o pa icula g oups can
ha e nega i e consequences o p ojec ou comes (So ensen
e al., 2019). The e o e, unde s anding who is excluded om
ou a i ac and wha al e na i es could be c ea ed would be
indispensable.
Conclusion
Inequali ies in da a access and li e acy pose a isk o he
indi idual and socie y as a whole. In his wo k, we he e-
o e in es iga ed he use case o CS as a means o empowe
ci izens in accessing and wo king wi h da a. We ha e p e-
sen ed esul s om he i s cycle o a DSR p ojec a ge -
ing he de elopmen o a CA o suppo da a explo a ion in
CS p ojec s. Following he six s eps o a p oblem-cen e ed
design p ocess by Pe e s e al. (2020), we ha e app oached
his challenge by s uc u ing and elabo a ing on he p oblem
space and i s associa ed s akeholde s, elici ing equi emen s
and ansla ing hem in o design p inciples o a solu ion
a i ac and inally p esen ing and es ing a p o o ypical
implemen a ion o his a i ac . The esul o ou i s design
cycle is a CA o da a analysis ac i i ies o e ing lexible
suppo on demand o mul iple use s in pa allel. Fo inexpe-
ienced use s, he ool p o ides seamless guidance h ough
da a analysis by p o iding da ase -independen knowledge
and ips, allowing use s o decide how deeply hey wan o
di e in o a pa icula opic. Ad anced use s can use a ques-
ion-and-answe p ocess o ask ques ions abou da a analysis
eely, hus en iching only hei knowledge wi h he CA and
con olling he analysis p ocess.
In i s cu en s a e, he CA shows high po en ial o ans-
e ing equi ed da a li e acy o ci izens, enabling hem o
pe o m be e in analy ical asks. Quali a i e use eedback
shows ha mul iple ci izens pe cei e he ool’s suppo as
enjoyable and use ul and poin ou po en ial applica ion
ields. Ha nessing he ad an ages o hei easy, esou ce-
e icien p o isioning, he usage o CAs in CS p ojec s seems
p omising and could posi i ely p omo e equi able access o
da a-d i en knowledge. Howe e , iden i ied challenges, such
as he pa icipan s’ mo i a ion, eeling o empowe men , and
pe cei ed lea ning e ec , could no be sol ed adequa ely by
he CA. This indica es ha u he esea ch is necessa y o
e ine he CA and i s usabili y, which we in end o accom-
plish in a p oceeding design cycle.
Supplemen a y In o ma ion The online e sion con ains supplemen-
a y ma e ial a ailable a h ps:// doi. o g/ 10. 1007/ s12525- 024- 00705-3.
Acknowledgemen s We wan o hank he 12 wo kshop pa icipan s
and he 30 expe imen pa icipan s o hei con ibu ion o ou wo k.
Funding Open Access unding enabled and o ganized by P ojek
DEAL.
Da a A ailabili y The da a ha suppo he indings o his s udy a e
a ailable om he au ho s upon easonable eques .
Decla a ions
Con lic o in e es The au ho s decla e no compe ing in e es s.
Open Access This a icle is licensed unde a C ea i e Commons A i-
bu ion 4.0 In e na ional License, which pe mi s use, sha ing, adap a-
ion, dis ibu ion and ep oduc ion in any medium o o ma , as long
as you gi e app op ia e c edi o he o iginal au ho (s) and he sou ce,
p o ide a link o he C ea i e Commons licence, and indica e i changes
we e made. The images o o he hi d pa y ma e ial in his a icle a e
included in he a icle’s C ea i e Commons licence, unless indica ed
o he wise in a c edi line o he ma e ial. I ma e ial is no included in
he a icle’s C ea i e Commons licence and you in ended use is no
pe mi ed by s a u o y egula ion o exceeds he pe mi ed use, you will
need o ob ain pe mission di ec ly om he copy igh holde . To iew a
copy o his licence, isi h p://c ea i ecommons.o g/licenses/by/4.0/.
Re e ences
Alaaeldin, R., As ou a, E., Kassem, G., & Abdel-Haq, M. S. (2021).
De eloping cha bo sys em o suppo decision making based on
big da a analy ics. Jou nal o Managemen In o ma ion and Deci-
sion Sciences, 24(2), 1–15.
Ala i, M., Ma akas, G. M., & Yoo, Y. (2002). A compa a i e s udy o
dis ibu ed lea ning en i onmen s on lea ning ou comes. In o ma-
ion Sys ems Resea ch, 13(4), 404–415. h ps:// doi. o g/ 10. 1287/
is e. 13.4. 404. 72
Anh, N., & Ngan, H. (2021). A i icial in elligence in ma hema ics edu-
ca ion: An empi ical s udy o using cha bo in eaching and lea n-
ing ma hema ics a ie namese high schools. 5 h ASIA PACIFIC
In e na ional Mode n Sciences Cong ess, 306– 317.
A h eya, R. G., NgongaNgomo, A.-C., & Usbeck, R. (2018). Enhanc-
ing communi y in e ac ions wi h da a-d i en cha bo s— he DBpe-
dia cha bo . The Web Con e ence 2018 - Companion o he Wo ld
Wide Web Con e ence. WWW, 2018, 143–146.
Ba zilai, S., & Blau, I. (2014). Sca olding game-based lea ning:
Impac on lea ning achie emen s, pe cei ed lea ning, and game
expe iences. Compu e s & Educa ion, 70, 65–79. h ps:// doi. o g/
10. 1016/j. compe du. 2013. 08. 003
Bha ga a, R., Deahl, E., Le ouzé, E., Noonan, A., Sangokoya, D., &
Shoup, N. (2015). Beyond da a li e acy: Rein en ing communi y
engagemen and empowe men in he age o da a (Da a-Pop
alliance whi e pape se ies). h ps:// da ap opall iance. o g/ i em/
be yond- da a- li e acy- ein en ing- commu ni y- engag emen - and-
empow e men - in- he- age- o - da a. Accessed 23 Jan 2024.
Bha ga a, R., Kadouaki, R., Bha ga a, E., Cas o, G., & D’Ignazio,
C. (2016). Da a mu als: Using he a s o build da a li e acy. The
Elec onic Ma ke s (2024) 34:2323 Page 16 o 18
Jou nal o Communi y In o ma ics, 12(3), 197–216. h ps:// doi.
o g/ 10. 15353/ joci. 12i3. 3285
Bi ne , E., Oes e-Reiß, S., & Leimeis e , J. M. (2019). Whe e is
he bo in ou eam? Towa d a axonomy o design op ion
combina ions o con e sa ional agen s in collabo a i e wo k.
InP oceedings o he 52nd Hawaii In e na ional Con e ence
on Sys em Sciences. h ps:// doi. o g/ 10. 24251/ HICSS. 2019. 035
Bonney, R., Phillips, T. B., Balla d, H. L., & Enck, J. W. (2016). Can
ci izen science enhance public unde s anding o science? Public
Unde s anding o Science, 25(1), 2–16. h ps:// doi. o g/ 10. 1177/
09636 62515 607406
Bowse , A., Coope , C., de She binin, A., Wiggins, A., B en on, P.,
Chuang, T.-R., Faus man, E., Haklay, M., & (Muki), & Meloche,
M. (2020). S ill in need o no ms: The s a e o he da a in ci izen
science. Ci izen Science: Theo y and P ac ice, 5(1), A icle 1.
h ps:// doi. o g/ 10. 5334/ cs p. 303
Ca mi, E., Ya es, S. J., Lockley, E., & Pawluczuk, A. (2020). Da a
ci izenship: Re hinking da a li e acy in he age o disin o -
ma ion, misin o ma ion, and malin o ma ion. In e ne Policy
Re iew, 9(2). h ps:// doi. o g/ 10. 14763/ 2020.2. 1481
Cen e o Sel -De e mina ion Theo y. (2022). In insic Mo i a ion
In en o y (IMI). h ps:// sel d e e m ina i on he o y. o g/ in i nsic-
mo i a ion- in en o y. Accessed 23 Jan 2024.
Chan, S. C. H., & Ko, S. (2021). The da k side o pe sonal esponse
sys ems (PRSs): Bo edom, eedback, pe cei ed lea ning, lea n-
ing sa is ac ion. Jou nal o Educa ion o Business, 96(7), 435–
444. h ps:// doi. o g/ 10. 1080/ 08832 323. 2020. 18487 69
Cia án Mac Domhnaill, S. L., & Nolan, A. (2020). The ci izens in
ci izen science: Demog aphic, socioeconomic, and heal h cha -
ac e is ics o biodi e si y eco de s in I eland. Ci izen Science:
Theo y and P ac ice, 5, 1–17. h ps:// doi. o g/ 10. 5334/ cs p. 283
Cla ke, R. (2016) Big da a big isks, Abs ac . In o ma ion Sys ems
Jou nal, 26(1), 77–90. h ps:// doi. o g/ 10. 1111/ isj. 12088
D’Ignazio, C., & Bha ga a, R. (2016). Da aBasic: Design p inci-
ples, ools and ac i i ies o da a li e acy lea ne s. The Jou nal
o Communi y In o ma ics, 12(3), A icle 3. h ps:// doi. o g/ 10.
15353/ joci. 12i3. 3280
D’Ignazio, C. (2022). C ea i e da a li e acy: B idging he gap
be ween he da a-ha es and da a-ha e no s. In o ma ion Design
Jou nal, 6–18. h ps:// doi. o g/ 10. 1075/ idj. 23.1. 03dig
Da is, F. D. (1989). Pe cei ed use ulness, pe cei ed ease o use,
and use accep ance o in o ma ion echnology. MIS Qua e ly,
13(3), 319–340. h ps:// doi. o g/ 10. 2307/ 249008
De Albuque que, J., & Almeida, A. (2020). Modes o engagemen :
Re aming ‘sensing’ and da a gene a ion in ci izen science o
empowe ing ela ionships. In T. Da ies & A. Mah (Eds.), Toxic
u hs: En i onmen al jus ice and ci izen science in a pos - u h
age (pp. 267–281). Manches e Uni e si y P ess.
Deb uyne, C., Kea ns, A., O’Neill, C., Colclough, M., G ehan, L.,
& O’Sulli an, D. (2021). DALIDA: Da a li e acy discussion
wo kshops o adul s. ACM In e na ional Con e ence P oceed-
ing Se ies, 23–25.
Diede ich, S., B endel, A., Mo ana, S., & Kolbe, L. (2022). On
he design o and in e ac ion wi h con e sa ional agen s: An
o ganizing and assessing e iew o human-compu e in e ac ion
esea ch. Jou nal o he Associa ion o In o ma ion Sys ems,
23(1), 96–138. h ps:// doi. o g/ 10. 17705/ 1jais. 00724
Ekinci, E., Omu ca, S., & Acun, N. (2018). A compa a i e s udy
on machine lea ning echniques using i anic da ase . In7 h
In e na ional Con e ence on Ad anced Technologies, An alya,
Tu key (pp. 411–416).
Eom, S. B., Wen, H. J., & Ashill, N. (2006). The de e minan s o s u-
den s’ pe cei ed lea ning ou comes and sa is ac ion in uni e -
si y online educa ion: An empi ical in es iga ion. Decision Sci-
ences Jou nal o Inno a i e Educa ion, 4(2), 215–235.h ps://
doi. o g/ 10. 1111/j. 1540- 4609. 2006. 00114.x
E icsson, K. A., & Simon, H. A. (1984). P o ocol analysis: Ve bal
epo s as da a. The MIT P ess.
Fan, M., Li, Y., & T uong, K. N. (2020). Au oma ic de ec ion o
usabili y p oblem encoun e s in hink-aloud sessions. ACM
T ansac ions on In e ac i e In elligen Sys ems (TiiS), 10(2),
1–24.h ps:// doi. o g/ 10. 1145/ 33857 32
Gkinko, L., & Elbanna, A. (2023). The app op ia ion o con e sa-
ional AI in he wo kplace: A axonomy o AI cha bo use s.
In e na ional Jou nal o In o ma ion Managemen , 69, 102568.
h ps:// doi. o g/ 10. 1016/j. ijin omg . 2022. 102568
Glynn, S. M., B ickman, P., A ms ong, N., & Taasoobshi azi, G.
(2011). Science mo i a ion ques ionnai e II: Valida ion wi h
science majo s and nonscience majo s. Jou nal o Resea ch in
Science Teaching, 48(10), 1159–1176.h ps:// doi. o g/ 10. 1002/
ea. 20442
Gould, R. (2021). Towa d da a-scien i ic hinking. Teaching S a is-
ics, 43, 11–22.h ps:// doi. o g/ 10. 1111/ es . 12267
G ego , S., & He ne , A. R. (2013). Posi ioning and p esen ing
design science esea ch o maximum impac . MIS Qua e ly,
37(2), 337–356. h ps:// doi. o g/ 10. 25300/ MISQ/ 2013/ 37.2. 01
G ego , S., K use, C. L., & Seidel, S. (2020). Resea ch pe spec i es:
The ana omy o a design p inciple. Jou nal o he Associa ion
o In o ma ion Sys ems, 21(6), 1622–1652. h ps:// doi. o g/ 10.
17705/ 1jais. 00649
Gup a, K., Sha ma, P., & Bouza, C. N. (2018). Su i ing he Ti anic
agedy: A sociological s udy using machine lea ning models.
Suma De Negocios Magazine, 9(20), 86–92.h ps:// doi. o g/ 10.
14349/ sumneg/ 2018. 9. n20. a2
Hanley, H. W. A., & Du ume ic, Z. (2023, May 16). Machine-made
media: Moni o ing he mobiliza ion o machine-gene a ed a i-
cles on misin o ma ion and mains eam news websi es. A Xi .
O g.h ps:// a xi . o g/ abs/ 2305. 09820 1.
Hinde ks, A., Sch epp, M., & Thomaschewski, J. (2018). A benchma k
o he sho e sion o he use expe ience ques ionnai e. P o-
ceedings o he 14 h In e na ional Con e ence on Web In o ma ion
Sys ems and Technologies (WEBIST 2018), 373–377.
Holowka, E. M., Woods, S., Pahayahay, A., Roy, M., & Khalili-Mahani,
N. (2021). P inciples o designing an mHeal h app o pa icipa-
o y esea ch and managemen o ch onic pain. In V. G. Du y
(Ed.), Digi al Human Modeling and Applica ions in Heal h,
Sa e y, E gonomics and Risk Managemen AI, P oduc and Se -
ice. Lec u e No es in Compu e Science (pp. 50–67). Sp inge ,
Cham.
Hoon, G. K., Yong, L. J., & Yang, G. K. (2020). In e acing cha bo
wi h da a e ie al and analy ics que ies o decision making. In
A. Abdul Majeed, J. A. Ma -Jiza , M. H. A. Hassan, Z. Taha,
H. L. Choi, & J. Kim (Eds.), Ri a 2018 - P oceedings o he 6 h
In e na ional Con e ence on Robo In elligence Technology and
Applica ions (pp. 385–394). Sp inge Singapo e.
I win, A. (1995). Ci izen science: A s udy o people, expe ise and
sus ainable de elopmen . Rou ledge.
Isacco, S., Claps, P., G asso, S., Fe a i, E., Gue cio, M. B., Musumeci,
R. E., Sca cella, G. E., Ve sace, P., & Laio, F. (2018). Floodbook:
A social pla o m o lood hyd ology. P oceedings o he 13 h
In e na ional Con e ence on Hyd oin o ma ics, 3, 941–949.
Janssen, A., Passlick, J., Rod íguez Ca dona, D., & B ei ne , M. H.
(2020). Vi ual assis ance in any con ex : A axonomy o design
elemen s o domain-speci ic cha bo s. Business and In o ma ion
Sys ems Enginee ing, 62(3), 211–225.h ps:// doi. o g/ 10. 1007/
s12599- 020- 00644-1
Jayawick ama, T., Abdelaal, S., & Abadia, R. (2020). Using online
lea ning en i onmen s o add ess digi al li e acy compe encies
o cons uc ion managemen g adua es. P oceedings o he 28 h
In e na ional Con e ence on Compu e s in Educa ion, ICCE 2020,
1, 670–679.
Elec onic Ma ke s (2024) 34:23 Page 17 o 18 23
Jenne , C., Kloe ze , L., Schneide , D., Iaco ides, I., Cox, A., Gold,
M., Fuchs, B., E eleigh, A., Me hieu, K., Ajani, Z., e al. (2016).
Mo i a ions, lea ning and c ea i i y in online ci izen science.
Jou nal o Science Communica ion, 15(3). A05. h ps:// doi. o g/
10. 22323/2. 15030 205
Jones, M. G., Childe s, G., And e, T., Co in, E. N., & Hi e, R. (2018).
Ci izen scien is s and non-ci izen scien is hobbyis s: Mo i a ion,
bene i s, and in luences. In e na ional Jou nal o Science Educa-
ion, Pa B, 8(4), 287–306.h ps:// doi. o g/ 10. 1080/ 21548 455.
2018. 14757 80
Keyne , S., Sa enko , V., & Vakulenko, S. (2019). Open da a cha bo .
In P. Hi zle , S. Ki ane, O. Ha ig, V. de Boe , M.-E. Vidal, M.
Maleshko a, S. Schlobach, K. Hamma , N. Lasie a, S. S ad mül-
le , K. Hose, & R. Ve bo gh (Eds.), The Seman ic Web: ESWC
2019 Sa elli e E en s (pp. 111–115). Sp inge .
Kim, H.-W., & Gup a, S. (2014). A use empowe men app oach o
in o ma ion sys ems in usion. IEEE T ansac ions on Enginee -
ing Managemen , 61(4), 656–668.h ps:// doi. o g/ 10. 1109/ TEM.
2014. 23546 93
Ki kpa ick, D. L. (1959). Techniques o e alua ing aining p og ams.
Jou nal o Ame ican Socie y o T aining Di ec o s, 13(3), 21–26.
Kjel ik, M. K., & Schul heis, E. H. (2019). Ge ing messy wi h au hen-
ic da a: Explo ing he po en ial o using da a om scien i ic
esea ch o suppo s uden da a li e acy. CBE Li e Sciences Edu-
ca ion, 18(2), A icle es2. h ps:// doi. o g/ 10. 1187/ cbe. 18- 02- 0023
Kloe ze , L., Lo ke, J., Roche, J., Golumbic, Y., Win e , S., & Jõge a,
A. (2021). Lea ning in ci izen science. In K. Vohland, A. Land-
Zands a, L. Cecca oni, R. Lemmens, J. Pe elló, M. Pon i, R. Sam-
son, & K. Wagenknech (Eds.), The Science o Ci izen Science
(pp. 283–308). Sp inge .
K ishnamu hy, R., & Awazu, Y. (2016). Libe a ing da a o public
alue: The case o Da a.go . In e na ional Jou nal o In o ma-
ion Managemen , 36(4), 668–672. h ps:// doi. o g/ 10. 1016/j. ijin
omg . 2016. 03. 002
K uge , J., & Dunning, D. (1999). Unskilled and unawa e o i : How
di icul ies in ecognizing one’s own incompe ence lead o in la ed
sel -assessmen s. Jou nal o Pe sonali y and Social Psychology,
77(6), 1121.h ps:// doi. o g/ 10. 1037/ 0022- 3514. 77.6. 1121
K ale, K., F eddi, E., Hodneb og, S., Sell, O. A., & Føls ad, A.
(2021). Unde s anding he use expe ience o cus ome se ice
cha bo s: Wha can we lea n om cus ome sa is ac ion su -
eys? In A. Føls ad, T. A aujo, S. Papadopoulos, E.L.-C. Law,
E. Luge , M. Goodwin, & P. B. B and zaeg (Eds.), Cha bo
Resea ch and Design (pp. 205–218). Sp inge .
Lewandowski, T., Kuče ić, E., Leible, S., Pose , M., & Böhmann, T.
(2023). Enhancing con e sa ional agen s o success ul ope a-
ion: A mul i-pe spec i e e alua ion app oach o con inuous
imp o emen . Elec onic Ma ke s, 33, 39.h ps:// doi. o g/ 10.
1007/ s12525- 023- 00662-3
Lia, E. H., De ien, M. M., Whi e, E. M., Winde , S. G., & Wood, S.
A. (2023). Communi y science wi h a ex -messaging cha bo
p o ides a no el app oach o moni o ing ou doo ec ea ion.
Digi al Geog aphy and Socie y, 100059. h ps:// doi. o g/ 10.
1016/j. diggeo. 2023. 100059
Liu, H.-Y., Dö le , D., Heigl, F., & G ossbe nd , S. (2021). Ci izen sci-
ence pla o ms. In K. Vohland, A. Land-Zands a, L. Cecca oni,
R. Lemmens, J. Pe elló, M. Pon i, R. Samson, & K. Wagenknech
(Eds.), The Science o Ci izen Science (pp. 439–459). Sp inge .
Logan, V. (2017). In o ma ion as a second language: Enabling da a
li e acy o digi al socie y. Ga ne . h ps:// www. ga n e . com/ en/
docum en s/ 36025 17.
Ma suu a, S., & Ishimu a, R. (2017). Cha bo and dialogue demons a-
ion wi h a humanoid obo in he lec u e class. In M. An ona & C.
S ephanidis (Eds.), Uni e sal Access in Human–Compu e In e -
ac ion. Human and Technological En i onmen s (pp. 233–246).
Sp inge , Cham.
Monzón Al a ado, C. M., Zamo a Rendon, A., & Vázquez Pé ez, A.
del S. (2020). In eg a ing public pa icipa ion in knowledge gen-
e a ion p ocesses: E idence om ci izen science ini ia i es in
Mexico. En i onmen al Science & Policy, 114, 230–241. h ps://
doi. o g/ 10. 1016/j. en sci. 2020. 08. 007.
Na echania, A., S ini asan, A., & S asko, J. (2021). NL4DV: A oolki
o gene a ing analy ic speci ica ions o da a isualiza ion om
na u al language que ies. IEEE T ansac ions on Visualiza ion and
Compu e G aphics, 27(2), 369–379.h ps:// doi. o g/ 10. 1109/
TVCG. 2020. 30303 78
Na ional Academies o Sciences, Enginee ing & Medicine. (2018).
Lea ning h ough ci izen science: Enhancing oppo uni ies by
design. The Na ional Academies P ess.
Neumaie , S., Sa enko , V., & Vakulenko, S. (2017). Talking open
da a. The Seman ic Web: ESWC 2017 Sa elli e E en s - ESWC
2017 Sa elli e E en s, 132–136.
Nguyen, H. D., Pham, V. T., T an, D. A., & Le, T. T. (2019). In el-
ligen u o ing cha bo o sol ing ma hema ical p oblems in
high-school. P oceedings o he 11 h In e na ional Con e ence
on Knowledge and Sys ems Enginee ing (KSE), 1–6.
Okonkwo, C. W., & Ade-Ibijola, A. (2021). Cha bo s applica ions in
educa ion: A sys ema ic e iew. Compu e s and Educa ion: A i-
icial In elligence, 2, 100033.
Pae sch, F., Ebe lein, A., & Mau e , F. (2003). Requi emen s engi-
nee ing and agile so wa e de elopmen . P oceedings o he 12 h
IEEE In e na ional Wo kshops on Enabling Technologies: In a-
s uc u e o Collabo a i e En e p ises, WET ICE 2003, 308–313.
Palan, S., & Schi e , C. (2018). P oli ic.ac—a subjec pool o online
expe imen s. Jou nal o Beha io al and Expe imen al Finance,
17, 22–27. h ps:// doi. o g/ 10. 1016/j. jbe . 2017. 12. 004
Pane a, K. (2021). A da a and analy ics leade ’s guide o da a li e acy.
Ga ne .h ps:// www. ga n e . com/ sma e wi hga ne /a- da a- and-
analy ics- leade s- guide o- da a- li e acy
Pe e s, K., Tuunanen, T., Gengle , C. E., Rossi, M., Hui, W., Vi anen,
V., & B agge, J. (2020). Design science esea ch p ocess: A model
o p oducing and p esen ing in o ma ion sys ems esea ch. CoRR,
abs/2006.02763.
Pé ez, J. Q., Da adoumis, T., & Puig, J. M. M. (2020). Redisco e ing
he use o cha bo s in educa ion: A sys ema ic li e a u e e iew.
Compu e Applica ions in Enginee ing Educa ion, 28(6), 1549–
1565. h ps:// doi. o g/ 10. 1002/ cae. 22326
Pe ibone, L., Vohland, K., & Ziegle , D. (2017). Unde s anding he
(in e )disciplina y and ins i u ional di e si y o ci izen science: A
su ey o cu en p ac ice in Ge many and Aus ia. PLoS ONE,
12(6), 1–6. h ps:// doi. o g/ 10. 1371/ jou n al. pone. 01787 78
Po ela, M. (2021). In e acing pa icipa ion in ci izen science p o-
jec s wi h con e sa ional agen s. Human Compu a ion, 8(2),
33–53.h ps:// doi. o g/ 10. 15346/ hc. 8i2. 114
Pullinge , J. (2021). Misuse o s a is ics: Time o speak ou . S a is i-
cal Jou nal o he IAOS, 37(1), 79–84. h ps:// doi. o g/ 10. 3233/
SJI- 210783
Rade mache , W. J. (2021). Li e acy in s a is ics o he public dis-
cou se. S a is ical Jou nal o he IAOS, 37(3), 747–752.h ps://
doi. o g/ 10. 3233/ SJI- 210852
Radziwill, N. M., & Ben on, M. C. (2017). E alua ing quali y o cha -
bo s and in elligen con e sa ional agen s ( a Xi :1704.04579).
a Xi .h ps:// doi. o g/ 10. 48550/ a Xi . 1704. 04579
Rapp, A., Cu i, L., & Boldi, A. (2021). The human side o human-
cha bo in e ac ion: A sys ema ic li e a u e e iew o en yea s o
esea ch on ex -based cha bo s. In e na ional Jou nal o Human-
Compu e S udies, 151, 102630. h ps:// doi. o g/ 10. 1016/j. ijhcs.
2021. 102630
Robe s, T. L., Low y, P. B., & Sweeney, P. D. (2006). An e alua ion
o he impac o social p esence h ough g oup size and he use
o collabo a i e so wa e on g oup membe ‘ oice’ in ace- o-
ace and compu e -media ed ask g oups. IEEE T ansac ions on
Elec onic Ma ke s (2024) 34:2323 Page 18 o 18
P o essional Communica ion, 49(1), 28–43. h ps:// doi. o g/ 10.
1109/ TPC. 2006. 870460
Ryan, R. M. (1982). Con ol and in o ma ion in he in ape sonal
sphe e: An ex ension o cogni i e e alua ion heo y. Jou nal o
Pe sonali y and Social Psychology, 43(3), 450–461.h ps:// doi.
o g/ 10. 1037/ 0022- 3514. 43.3. 450
Sch epp, M., Hinde ks, A., & Thomaschewski, J. (2017). Design and
e alua ion o a sho e sion o he use expe ience ques ionnai e
(UEQ-S). In e na ional Jou nal o In e ac i e Mul imedia and
A i icial In elligence, 4(6), 103–108. h ps:// doi. o g/ 10. 9781/
ijimai. 2017. 09. 001
Schülle , K., Busch, P., & Hindinge , C. (2019). Fu u e skills: A
amewo k o da a li e acy. Hochschul o um Digi alisie ung,
46, 1–128.
Seege , A.-M., P ei e , J., & Heinzl, A. (2021). Tex ing wi h human-
like con e sa ional agen s: Designing o an h opomo phism.
Jou nal o he Associa ion o In o ma ion Sys ems, 22(4), 931–
967. h ps:// doi. o g/ 10. 17705/ 1jais. 00685
Shah, H. R., & Ma inez, L. R. (2016). Cu en app oaches in imple-
men ing ci izen science in he class oom. Jou nal o Mic obiology
& Biology Educa ion, 17(1), 17–22. h ps:// doi. o g/ 10. 1128/ jmbe.
17i1. 1032
Sha ma, S., & Pandey, S. (2013). Re isi ing equi emen s elici a ion
echniques. In e na ional Jou nal o Compu e Applica ions,
75(12), 35–39.
Shi k, J. L., Balla d, H. L., Wilde man, C. C., Phillips, T., Wiggins,
A., Jo dan, R., McCallie, E., Mina chek, M., Lewens ein, B. V., &
E, K. M., & Bonney, R. (2012). Public pa icipa ion in scien i ic
esea ch: A amewo k o delibe a e design. Ecology and Socie y,
17, 29–48.h ps:// doi. o g/ 10. 5751/ ES- 04705- 17022
Simono ski, A., Zuide wijk, A., Cla in al, A., & Hammedi, W. (2022).
Tailo ing open go e nmen da a po als o lay ci izens: A gami ica-
ion heo y app oach. In e na ional Jou nal o In o ma ion Manage-
men , 65, 102511. h ps:// doi. o g/ 10. 1016/j. ijin omg . 2022. 102511
Simpson, R., Page, K. R., & De Rou e, D. (2014). Zooni e se: Obse -
ing he wo ld’s la ges ci izen science pla o m. P oceedings o he
23 d In e na ional Con e ence on Wo ld Wide Web, 1049–1054.
h ps:// doi. o g/ 10. 1145/ 25679 48. 25792 15
Simud, T., Ruengi inun, S., Su as adi, N., Sangle dsinlapachai, N., &
Plangp asopchok, A. (2020). A con e sa ional agen o da abase
que y: A use case o hai people map and analy ics pla o m. P o-
ceedings o he 15 h In e na ional Join Symposium on A i icial
In elligence and Na u al Language P ocessing, ISAI-NLP 2020,
pp 1–6. h ps:// doi. o g/ 10. 1109/ iSAI- NLP51 646. 2020. 93768 33
Socien ize. (2014). G een pape on ci izen science o Eu ope: Towa ds
a socie y o empowe ed ci izens and enhanced esea ch. Eu o-
pean Commission. h ps:// digi al- s a egy. ec. eu opa. eu/ en/ lib a
y/ g een- pape - ci iz en- scien ce- eu ope- owa ds- socie y- empow
e ed- ci iz ens- and- enhan ced- esea ch
So ensen, A. E., Jo dan, R. C., LaDeau, S. L., Biehle , D., Wilson,
S., Pi as, J.-H., & Leisnham, P. T. (2019). Re lec ing on e o s
o design an inclusi e ci izen science p ojec in Wes Bal imo e.
Ci izen Science: Theo y and P ac ice, 4(1).
Spasiano, A., G imaldi, S., B accini, A. M., & Na di, F. (2021). Towa ds
a ansdisciplina y heo e ical amewo k o ci izen science:
Resul s om a me a- e iew analysis. Sus ainabili y, 13(14), 7904.
Spie s, H., Swanson, A., Fo son, L., Simmons, B., T ouille, L., Blick-
han, S., & Lin o , C. (2019). E e yone coun s? Design conside a-
ions in online ci izen science. Jou nal o Science Communica-
ion, 18(1). A04. h ps:// doi. o g/ 10. 22323/2. 18010 204
S iegli z, S., Ho edi z, L., B ünke , F., Ehnis, C., Mi babaie, M., &
Ross, B. (2022). Design p inciples o con e sa ional agen s o
suppo eme gency managemen agencies. In e na ional Jou -
nal o In o ma ion Managemen , 63, 102469. h ps:// doi. o g/ 10.
1016/j. ijin omg . 2021. 102469
Tallyn, E., F ied, H., Gianni, R., Isa d, A., & Speed, C. (2018). The
E hnobo : Ga he ing e hnog aphies in he age o IoT. P oceedings
o he 2018 CHI Con e ence on Human Fac o s in Compu ing
Sys ems, CHI ‘18, 1–13.
Tamayo-Mo eno, S., & Pé ez-Ma ín, D. (2016). Adap ing he design
and he use me hodology o a pedagogical con e sa ional agen o
seconda y educa ion o childhood educa ion. In e na ional Sym-
posium on Compu e s in Educa ion (SIIE), 2016, 1–6.h ps:// doi.
o g/ 10. 1109/ SIIE. 2016. 77518 19
Ta anapou , N., Pose , M., & Bi ne , E. A. C. (2019). Suppo ing
he idea gene a ion p ocess in ci izen pa icipa ion— owa d
an in e ac i e sys em wi h a con e sa ional agen as acili a o .
InP oceedings o he 27 h Eu opean Con e ence on In o ma ion
Sys ems. S ockholm & Uppsala.
T zaskowski, J. (2022). Da a-d i en alue ex ac ion and human well-
being unde EU law. Elec onic Ma ke s, 32(2), 447–458. h ps://
doi. o g/ 10. 1007/ s12525- 022- 00528-0
Twidale, M. B., Blake, C., & Gan , J. P. (2013). Towa ds a da a li e -
a e ci izen y. ICon e ence 2013 P oceedings. h ps:// doi. o g/ 10.
9776/ 13189
an de Goo , M. J., Ha kamp, L., & Dank o , Z. (2021). Cus ome
se ice cha bo s: A quali a i e in e iew s udy in o he communi-
ca ion jou ney o cus ome s. In A. Føls ad, T. A aujo, S. Papado-
poulos, E.L.-C. Law, E. Luge , M. Goodwin, & P. B. B and zaeg
(Eds.), CONVERSATIONS 2020, 4 h in e na ional wo kshop on
cha bo esea ch (pp. 190–204). Sp inge .
Venable, J., P ies-Heje, J., & Baske ille, R. (2016). FEDS: A ame-
wo k o e alua ion in design science esea ch. Eu opean Jou nal
o In o ma ion Sys ems, 25(1), 77–89. h ps:// doi. o g/ 10. 1057/ ejis.
2014. 36
Vinuales, G., Magno a, S. R., S e es, E., & Kulka ni, G. (2019).
Desc ip ion and e alua ion o an inno a i e segmen a ion, a -
ge ing, and posi ioning ac i i y using s uden pe cei ed lea ning
and ac ual s uden lea ning. Ma ke ing Educa ion Re iew, 29(1),
24–36. h ps:// doi. o g/ 10. 1080/ 10528 008. 2018. 14939 32
Wald, D. M., Longo, J., & Dobell, A. (2016). Design p inciples o
engaging and e aining i ual ci izen scien is s. Conse a ion
Biology, 30(3), 562–570.h ps:// doi. o g/ 10. 1111/ cobi. 12627
Wa son, J., & Callingham, R. (2004). S a is ical li e acy: F om idi-
osync a ic o c i ical hinking abs ac . In G. Bu ill & M. Camden
(Eds.), Cu icula De elopmen in S a is ics Educa ion In e na-
ional Associa ion o S a is ical Educa ion (IASE) Round able
(pp. 116–162). In e na ional S a is ical Ins i u e Voo bu g.
Wi z, B. W., Weye e , J. C., Becke , M., & Mülle , W. M. (2022).
Open go e nmen da a: A sys ema ic li e a u e e iew o empi i-
cal esea ch. Elec onic Ma ke s, 32(4), 2381–2404. h ps:// doi.
o g/ 10. 1007/ s12525- 022- 00582-8
Zhang, Y., Calyam, P., Joshi, T., Nai , S., & Xu, D. (2019). Domain-
speci ic opic model o knowledge disco e y h ough con e sa-
ional agen s in da a in ensi e scien i ic communi ies. P oceedings
o 2018 IEEE In e na ional Con e ence on Big Da a, Big Da a
2018, 4886–4895
Publishe 's No e Sp inge Na u e emains neu al wi h ega d o
ju isdic ional claims in published maps and ins i u ional a ilia ions.
