Recei ed: 26 Ap il 2024 Re ised: 12 No embe 2024 Accep ed: 8 Decembe 2024
DOI: 10.1002/aaai.12208
ARTICLE
On he eliabili y o La ge Language Models o misin o med
and demog aphically in o med p omp s
Toluwani A emu1Oluwakemi Akinwehinmi2Chukwuemeka Nwagu3
Syed Ish iaque Ahmed4Ri a O ji3Ped o A nau Del Amo2
Abdulmo aleb El Saddik1,5
1Mohamed Bin Zayed Uni e si y o
A i icial In elligence, Abu Dhabi, UAE
2CIMNE, Uni e si y o Lleida, Lleida,
Spain
3Dalhousie Uni e si y, Hali ax, Canada
4Uni e si y o To on o, To on o, Canada
5Uni e si y o O awa, O awa, Canada
Co espondence
Toluwani A emu, Mohamed Bin Zayed
Uni e si y o A i icial In elligence, Abu
Dhabi, UAE.
Email: oluwani.a [email p o ec ed]
Abs ac
We in es iga e and obse e he beha io and pe o mance o La ge Language
Model (LLM)-backed cha bo s in add essing misin o med p omp s and ques-
ions wi h demog aphic in o ma ion wi hin he domains o Clima e Change and
Men al Heal h. Th ough a combina ion o quan i a i e and quali a i e me hods,
we assess he cha bo s’ abili y o disce n he e aci y o s a emen s, hei adhe -
ence o ac s, and he p esence o bias o misin o ma ion in hei esponses.
Ou quan i a i e analysis using T ue/False ques ions e eals ha hese cha -
bo s can be elied on o gi e he igh answe s o hese close-ended ques ions.
Howe e , he quali a i e insigh s, ga he ed om domain expe s, shows ha
he e a e s ill conce ns ega ding p i acy, e hical implica ions, and he neces-
si y o cha bo s o di ec use s o p o essional se ices. We conclude ha while
hese cha bo s hold signi ican p omise, hei deploymen in sensi i e a eas
necessi a es ca e ul conside a ion, e hical o e sigh , and igo ous e inemen o
ensu e hey se e as a bene icial augmen a ion o human expe ise a he han
an au onomous solu ion. Da ase and assessmen in o ma ion can be ound a
h ps://gi hub.com/ olusophy/Edge-o -Tomo ow.
INTRODUCTION
In ecen imes, he p oli e a ion o La ge Language Mod-
els (LLMs) has signi ican ly impac ed he ield o a i icial
in elligence, owing o hei excep ional capabili ies in lan-
guage comp ehension and gene a ion. These ad anced
models ha e become in eg al in a ious applica ions ac oss
mul iple indus ies. Ye , hei g owing popula i y and
u ili y b ing o h c ucial challenges and e hical conside -
a ions.
P edominan ly based on ans o ma i e deep lea ning
a chi ec u es like ans o me s, LLMs ha e e olu ionized
This is an open access a icle unde he e ms o he C ea i e Commons A ibu ion License, which pe mi s use, dis ibu ion and ep oduc ion in any medium, p o ided he
o iginal wo k is p ope ly ci ed.
© 2025 The Au ho (s). AI Magazine published by John Wiley & Sons L d on behal o Associa ion o he Ad ancemen o A i icial In elligence.
Na u al Language P ocessing (NLP). These models, cha -
ac e ized by hei as neu al ne wo ks con aining millions
o billions o pa ame e s, a e ained on ex ensi e da ase s
encompassing a wide a ay o sou ces such as in e ne
con en , li e a y wo ks, and di e se media. Such comp e-
hensi e aining enables hem o g asp and in e p e a
my iad o linguis ic pa e ns and sub le ies.
Mi o ing he his o ical eliance on sea ch engines
o in e ne que ies, use s a e now inc easingly u ning
o cha bo s powe ed by LLMs o ins an aneous and
di ec esponses. No ably, since he ad en o Cha GPT, a
a ian based on he GPT-3.5 a chi ec u e in la e 2022, he
AI Magazine. 2025;46:e12208. wileyonlinelib a y.com/jou nal/aaai 1o 15
h ps://doi.o g/10.1002/aaai.12208
2o 15 AI MAGAZINE
FIGURE 1 S a ing a con e sa ion wi h an LLM Cha bo .
de elopmen and implemen a ion o LLMs ha e apidly
expanded ac oss a ious sec o s. These models ha e been
deployed in a eas including i ual assis ance, cus ome
suppo , con en c ea ion, sea ch unc ionali y, and in
he ealms o medical, scien i ic esea ch, p og amming
assis ance, educa ional ools, and mo e. Howe e , his
expansion has simul aneously spa ked signi ican con-
ce ns ega ding he e hical use o hese echnologies, as
he e ha e been ins ances o he models exhibi ing biases,
gene a ing inaccu a e in o ma ion, making un ai judg-
men s, o inci ing e hical deba es due o po en ial misuse
(Figu e 1).
LLMs a e apidly e ol ing, aising conce ns abou hei
po en ial o gene a e and dissemina e misin o ma ion.
Biases wi hin hese models could lead o unequal in o -
ma ion access o e-in o cemen o exis ing socie al biases.
Based on hese issues, we in es iga e he beha io o cha -
bo s backed by LLMs, o answe he ollowing wo esea ch
ques ions:
Resea ch ques ions
1. When aced wi h misin o med p omp s, do LLMs
e lec , ampli y, o ec i y he misin o ma ion h ough
hei esponses?
2. Do LLMs exhibi biases when answe ing p omp s,
which con ain demog aphic in o ma ion?
To answe hese ques ions, we ocused on he implica-
ions o u ilizing hese cha bo s in discussions ela ed o
clima e change and men al heal h. We ocus ou analy-
sis on h ee LLM-powe ed cha bo s: Cha GPT, Bing Cha ,
and Google BARD, assessing whe he hey mani es biases
o p opaga e misin o ma ion. Clima e change and men al
heal h, being among he mos ex ensi ely discussed op-
ics on social media as indica ed by Google T ends1and
Exploding Topics2, a e chosen o hei ele ance and he
c i ical na u e o accu a e in o ma ion dissemina ion in
hese a eas. Fo he pu pose o ou s udy, ou main con-
ibu ions a e as ollows: We de eloped a comp ehensi e
benchma k da ase comp ising 3120 ue/ alse ques ions
on clima e change and 2762 on men al heal h. This
da ase was ins umen al o he empi ical and quan i a-
i e e alua ion o esponses om LLM-backed cha bo s.
We conduc ed an in-dep h quali a i e analysis in collab-
o a ion wi h domain expe s o sc u inize he esponses
om Cha GPT, Google BARD, and Bing Cha o po en-
ial biases. The indings om his analysis a e p esen ed
he ein. To suppo his, we cons uc ed a dedica ed bench-
ma k da ase con aining 53 ques ions on clima e change
and 40 on men al heal h, aimed a analyzing he ex en o
misin o ma ion in he esponses p o ided by hese cha -
bo s. Addi ionally, we u ilized 24 clima e change and 38
men al heal h ques ions speci ically o e alua e biases. We
p oposed and delibe a ed on se e al s a egies ha could
add ess he cu en challenges hinde ing he e ec i e and
e hical deploymen o LLM-backed cha bo s in p o id-
ing accu a e in o ma ion on clima e change and men al
heal h issues.
LITERATURE REVIEW
Backg ound
The con empo a y AI landscape, pa icula ly he ise o
LLMs has spa ked c i ical discussions a ound e hical con-
ce ns. These conce ns ex end beyond job displacemen and
p i acy iola ions o encompass he po en ial o misin o -
ma ion dissemina ion. LLMs, ained on massi e da ase s,
can unknowingly pe pe ua e biases and ac ual inaccu-
acies p esen in he aining da a. P e ious s udies says
con i ma ion bias and mo i a ed easoning can lead o
a o in o ma ion ha aligns wi h exis ing belie s (Nicke -
son 1998). This aises conce ns abou he us wo hiness
o LLMs ou pu s, especially when applied o sensi i e
domains like clima e change and men al heal h.
Massi e ounda ion models boas ing billions o lea ned
pa ame e s and ained on ex ensi e da ase s ha e ound
applica ions ac oss a wide a ay o domains and con-
ex s (Bommasani e al. 2022), and exhibi ed ema kable
e ec i eness in hei espec i e downs eam asks. Conse-
quen ly, he in eg a ion o AI in o eal-wo ld applica ions
has wi nessed a phenomenal and exponen ial su ge (Li,
Gan e al. 2023; Moo e al. 2023; Weisz e al. 2023). In
sha p con as o adi ional models, which o en su e
om inhe en cons ain s ied o hei na ow ocus, oun-
da ion models o e a e sa ile and adap able app oach.
Once hese models ha e unde gone aining, hey can
be con enien ly ine- uned o sui a di e se spec um o
applica ions, hus elimina ing he necessi y o ex ensi e
e aining. This adap i e amewo k se es as he linch-
pin o LLMs. No ably, his ine- uning capabili y has pa ed
he way o deploying hese language models in a mul i-
ude o domains, spanning heal hca e, inancial ad iso y,
clima e change analysis, and ques ion answe ing, o name
jus a ew.
23719621, 2025, 1, Downloaded om h ps://onlinelib a y.wiley.com/doi/10.1002/aaai.12208 by Readcube (Lab i a Inc.), Wiley Online Lib a y on [17/02/2025]. See he Te ms and Condi ions (h ps://onlinelib a y.wiley.com/ e ms-and-condi ions) on Wiley Online Lib a y o ules o use; OA a icles a e go e ned by he applicable C ea i e Commons License
AI MAGAZINE 3o 15
As he scope o a i icial in elligence (AI) con inues
o expand, ushe ing in cap i a ing inno a ions, i has
spa ked spi i ed deba es on a b oad ange o e hical con-
ce ns. These conce ns encompass he po en ial impac s
o hese ad ancemen s on a ious ace s o human exis-
ence, including indi idual li es, employmen , p i acy,
and issues ela ed o disc imina ion (Ragha an e al.
2020; Kelley e al. 2021). Simul aneously, ques ions ha e
eme ged ega ding he app op ia e cou se o ac ion o
he adop ion o hese ans o ma i e echnologies (A emu
2023;Liange al.2023).
Acco ding o an a icle by esea che s a Google pub-
lished in 2021 (Kelley e al. 2021) o assess public pe cep ion
o AI in eigh coun ies, people in de eloping coun ies
like Nige ia, India, and B azil a e signi ican ly mo e likely
o emb ace and adop AI compa ed o indi iduals in
de eloped coun ies. Howe e , e hical esea che s in he
AI domain ha e aised conce ns ha such AI applica-
ions may disp opo iona ely a ec people li ing in hese
egions, as mos o he da a used o ain hese models
is sou ced om de eloped coun ies (Geb u e al. 2021;
Shneide man 2020; Buolamwini and Geb u 2018;Rajie al.
2020; Mi chell e al. 2019; Mi els ad e al. 2016). The e-
o e, i comes as no su p ise ha h oughou 2023, a se ies
o pi o al go e nmen al hea ings ha e con ened, whe e
poli ical leade s engaged wi h a di e se a ay o expe s
o gain insigh s in o he o igins and implica ions o hese
echnologies. These hea ings ha e p obed c i ical aspec s,
such as he inhe en isks associa ed wi h hese AI sys-
ems, he na u e o he da a on which hey a e ained, and
he o mula ion o policies designed o sa egua d he well-
being and p i acy o use s. These discussions also conside
equi able compensa ion o he c ea o s and owne s o he
da a ha unde pin hese AI sys ems.
In his sec ion, ou ocus na ows o a icles highligh -
ing he deploymen o LLMs in he con ex s o clima e
change/sus ainabili y and men al heal h/physical heal h.
Ou objec i e he e is o demons a e he subs an ial s ides
made in he adop ion o AI echnologies, se ing he s age
o subsequen sec ions whe e we del e in o ou me hod-
ologies and p esen he esul s o expe imen s conduc ed o
e alua e biases and misin o ma ion in LLMs when applied
o bo h clima e change and men al heal h con ex s.
Clima e change
The ad en o LLMs in he ealm o clima e change
esea ch ook a signi ican leap o wa d in 2021 wi h he
in oduc ion o Clima eBERT by Webe sinke (Webe sinke
e al. 2022). Clima eBERT, a ans o me -based language
model, was p e ained on an ex ensi e da ase comp ising
o e wo million pa ag aphs sou ced om clima e- ela ed
ex s, including news, esea ch a icles, and co po a e
clima e epo s. I s p ima y pu pose was o acili a e cli-
ma e change ques ion answe ing and ex summa iza ion.
Subsequen ly, an a ay o ools (Vaghe i e al. 2023;Ni
e al. 2023; Fa d, Hasan, and Bell 2022;Li2023; Ga ido-
Me ch’an, Gonz’alez-Ba he, and Vaca 2023; K aus e al.
2023) and da ase s (Diggelmann e al. 2020;Spokoynye al.
2023;Laude al.2023) ha e been c ea ed o imp o e he
c edibili y and co ec ness o in o ma ion dissemina ed by
applica ions u ilizing such models.
While se e al empi ical s udies ha e examined he
e ec i eness o hese ools, mos ha e concen a ed on sen-
imen s (K ishnan and Anoop 2023; Sham and Mohamed
2022; Baguio, Lu, and Peña 2023; Ray and Kuma 2023)
and sus ainabili y (Jain and Padmanaban 2023). In a
closely ela ed s udy (Bulian e al. 2023), esea che s
assessed he accu acy o LLMs in handling clima e in o -
ma ion and p oposed a p ac ical p o ocol ha combines AI
assis ance wi h human a e s o mi iga e he limi a ions
encoun e ed du ing he e alua ion o LLMs. Ou wo k,
in con as , encompasses bo h quali a i e and quan i a-
i e e alua ions o LLM esponses o misin o med que ies
and conside s how hese models in e ac wi h a ious
demog aphic g oups.
Men al heal h
Language models show p omise in add essing challenges
wi hin he ield o men al heal h. Se e al o hese models,
employing smalle language models (Denecke, Vaaheesan,
and A ulna han 2020;Jie al.2021), ha e been p oposed
o applica ions in bo h men al heal h and gene al medi-
cal ca e. Mo e ecen de elopmen s ha e le e aged la ge
language models (Li, Li e al. 2023;Xue al.2023;Liu,Li
e al. 2023; Bao e al. 2023). I is impo an o no e ha
hese models in oduce e hical conce ns, as hey ha e he
po en ial o cause i e e sible ha m o use s.
Empi ical e alua ions o LLMs in his domain p ima ily
all in o wo ca ego ies: some assess LLMs’ abili y o
classi y di e en ypes o men al heal h issues using
anno a ed ex da a (Yang e al. 2023; Wang, Zhao, and
Pe zold 2023), while o he s e alua e hei pe o mance
in a ious medical examina ions (No i e al. 2023;Liu,
Zhou e al. 2023; Mana hunga and He igoda 2023; Rosol
e al. 2023; Kasai e al. 2023; Singhal e al. 2023). Ou
app oach, howe e , di e ges om hese s udies. We del e
deepe in o he analysis o he esponses gene a ed by
he models we employ, collabo a ing wi h expe s o
de e mine hei eadiness o eal-wo ld deploymen o
whe he signi ican s ides a e s ill equi ed in his a ea.
23719621, 2025, 1, Downloaded om h ps://onlinelib a y.wiley.com/doi/10.1002/aaai.12208 by Readcube (Lab i a Inc.), Wiley Online Lib a y on [17/02/2025]. See he Te ms and Condi ions (h ps://onlinelib a y.wiley.com/ e ms-and-condi ions) on Wiley Online Lib a y o ules o use; OA a icles a e go e ned by he applicable C ea i e Commons License
4o 15 AI MAGAZINE
METHODOLOGY
This s udy is s uc u ed o add ess ou cen al esea ch
objec i e which is o e alua e he le el o misin o ma-
ion and bias in LLM-powe ed cha bo s in clima e change
and men al heal h discussions. We do his h ough a
dual-p onged app oach: i s , by unde s anding and quan-
i ying misin o ma ion, and second, by e alua ing biases
in he esponses o LLM cha bo s. This sec ion de ails
he me hodologies employed in each o hese ca ego ies.
The de ails o he da ase used, cha bo s’ esponses, and
assessmen ques ions a e accessible he e.3We also p o ide
examples o ou ques ions in he appendix.
Tools and da a collec ion
In ou s udy, we e alua e he ollowing h ee cu ing-edge
mos popula and accessible LLM cha bo s: Mic oso ’s
Bing Cha , OpenAI’s Cha GPT, and Google’s Ba d (now
Gemini). To conduc an ex ensi e e alua ion, we compiled
a se o equen ly asked ques ions (FAQs) on wo c ucial
opics—Clima e Change and Men al Heal h. We p io i-
ize FAQs o e lec eal-wo ld use que ies encoun e ed
by hese LLMs. This app oach ensu es he gene alizabili y
o ou indings o eal-wo ld LLMs in e ac ions. We also
assume ha he LLMs used in hese cha bo s we e ained
on and has access o he in o ma ion. Hence, o es each
cha bo suscep ibili y o misin o ma ion, we in en ionally
al e ed a subse o he selec ed ques ions. These al e a ions
in ol ed in oducing sub le ac ual inaccu acies, changing
he one o he p omp , o inco po a ing i ele an wo ds
o he p omp s. The speci ic ype o al e a ion depended
on he misin o ma ion concep we aimed o assess. We
used he same misin o med p omp s uni o mly ac oss he
cha bo s in ended o es ing.
Misin o ma ion assessmen
Fo he quan i a i e analysis o misin o ma ion, we
cu a ed a da ase comp ising 3120 ue/ alse ques ions on
Clima e Change and 2762 on Men al Heal h. Fo he qual-
i a i e analysis, we ini ially compiled a b oade se o FAQ
p omp s om au ho i a i e sou ces such as NASA and
he CDC. We hen pa ed his down o 53 ques ions abou
Clima e Change and 40 abou Men al Heal h based on se -
e al c i e ia. Fi s , we p io i ized p omp s ha ep esen ed
a wide ange o common misconcep ions and equen ly
deba ed opics. Second, we ensu ed ha he selec ed
p omp s we e ep esen a i e o he mos pe asi e and
impac ul o ms o misin o ma ion. Las ly, we chose ques-
ions ha could e ec i ely challenge he cha bo s’ abili y
o disce n and add ess misin o ma ion wi hou inad e -
en ly ein o cing i . These p omp s we e hen ca e ully
al e ed o assess whe he he cha bo s could app op ia ely
handle misin o med s a emen s o i hey would ampli y
he misin o ma ion.
Bias assessmen
Fo he bias assessmen , we selec ed 24 ques ions on
Clima e Change and 38 on Men al Heal h, aiming o pe -
o m a comp ehensi e quali a i e analysis o he cha bo s’
esponses wi h a ocus on objec i i y and neu ali y. These
ques ions we e chosenbased on hei po en ial o highligh
biases, as hey co e key issues and a eas o po en ial con-
o e sy in each domain. To examine how demog aphic
ac o s migh in luence he cha bo s’ esponses, we in e-
g a ed de ails such as age, ace, and loca ion in o he
p omp s. This allowed us o assess whe he he cha bo s’
ou pu s a ied inapp op ia ely based on hese demo-
g aphics o i hey main ained consis en and impa ial
esponses ac oss di e en scena ios.
Cha bo que y app oach
Fo quali a i e analysis, we in e ac ed wi h he cha -
bo s using a s anda dized o ma o ensu e concise and
in o ma i e esponses. Each ques ion was o ma ed as
ollows:
“In one sho pa ag aph,[ques ion].P o ide
sou ces o you esponse.”
Fo Quan i a i e Analysis, he me hodology o quan i-
a i e in e ac ion also employed a s anda d p omp o ma ,
simpli ying he cha bo s’ esponses o a bina y choice:
“Respond wi h ei he T ue/Yes o False/No:
[ques ion].”
Analysis
This sec ion de ails he me hodological amewo k
adop ed o analyzing he misin o ma ion and bias com-
ponen s in ou s udy, di ided in o wo dis inc segmen s
as ollows:
Quan i a i e Analysis o check o misin o ma ion
The quan i a i e analysis sc u inizes he cha bo s’
esponses using a sui e o me ics: Con usion ma ix:This
ool isualizes he dis ibu ion o T ue Posi i es, False
Posi i es, T ue Nega i es, and False Nega i es o he
ue/ alse ques ions. P ecision and ecall: These me ics
e alua e he accu acy and comple eness o he classi ica-
ion model, based on he esul s o he con usion ma ix.
F1 sco e and accu acy: These indica o s p o ide insigh s
in o he model’s ha monic balance be ween p ecision
23719621, 2025, 1, Downloaded om h ps://onlinelib a y.wiley.com/doi/10.1002/aaai.12208 by Readcube (Lab i a Inc.), Wiley Online Lib a y on [17/02/2025]. See he Te ms and Condi ions (h ps://onlinelib a y.wiley.com/ e ms-and-condi ions) on Wiley Online Lib a y o ules o use; OA a icles a e go e ned by he applicable C ea i e Commons License
AI MAGAZINE 5o 15
and ecall. Simila i y index sco es:WeemployBLEU
(Papineni e al. 2002), ROGUE (Lin 2004), and METEOR
(Bane jee and La ie 2005) sco es o measu e he close-
ness o he cha bo ’s esponses o s anda d benchma k
answe s, he eby assessing he quali y and ele ance o
he con en p o ided.
Quali a i e analysis o check o misin o ma ion
We conduc a comp ehensi e quali a i e analysis o he
esponses om he cha bo s on he da ase we collec ed o
his analysis. This ace o he analysis in ol es in-dep h
in e iews wi h domain expe s and he deploymen o
specialized ques ionnai es ailo ed o hese ields.
Quali a i e analysis o check o bias
The bias analysis segmen ocuses on quan i a i ely e al-
ua ing eedback om domain expe s. These specialis s
will c i ique and p o ide pe spec i es on he ex en o
bias e iden in he cha bo s’ esponses. The aim he e is
o unco e any subjec i e biases ha migh be embed-
ded in he ou pu s ela ed o Men al Heal h and Clima e
Change opics.
Th oughou ou analysis, we some imes pe soni ied
LLM-based cha bo s (e.g., discussing wha hey “know”)
and, a o he imes, ea ed hem pu ely as unc ional
ex gene a o s. This mixed app oach was in en ional and
di ec ly ied o ou objec i e o assessing bias. Speci ically,
o e alua e how demog aphic ac o s migh in luence he
cha bo s’ esponses, we embedded demog aphic de ails
wi hin he p omp s, e ec i ely pe soni ying he cha -
bo s o simula e scena ios whe e biases migh mani es .
This app oach allowed us o explo e whe he he cha -
bo s would espond di e en ly based on he demog aphic
con ex p o ided.
Domain expe selec ion c i e ia
In he p ocess o selec ing domain expe s o ou s udy, we
es ablished speci ic c i e ia ailo ed o he dis inc ields o
Clima e Change and Men al Heal h.
Fo Clima e Change, we a ge ed academically c eden-
ialed p o essionals, including P o esso s, Pos Docs, PhD
s uden s, esea che s, o p ac i ione s holding a leas a
mas e ’s deg ee in ields such as en i onmen al science,
clima ology, me eo ology, o ecology. Thei expe ise was
alida ed h ough a demons a ed ack eco d in clima e
change esea ch, including publica ions in pee - e iewed
jou nals, con e ence p esen a ions, o signi ican con i-
bu ions o ele an indus y p ojec s. A p e equisi e was
a minimum o h ee yea s o ac i e in ol emen in a eas
such as clima e change esea ch, policy de elopmen ,
mi iga ion s a egies, adap a ion me hods, o ad ocacy.
Addi ionally, we emphasized he impo ance o in e disci-
plina y knowledge, combining insigh s om a mosphe ic
science, oceanog aphy, and social sciences, o os e a
comp ehensi e unde s anding o clima e change impac s.
Familia i y wi h clima e policies, he abili y o e ec i ely
communica e complex scien i ic concep s, and expe ience
in inno a i e solu ions and collabo a i e p ojec s we e also
deemed essen ial.
In he Men al Heal h domain, ou ocus was on p o-
essionals wi h a solid educa ional ounda ion in psy-
chology, psychia y, clinical social wo k, counseling, o
ela ed disciplines, equi ing a minimum o a mas e ’s
deg ee. We sough expe s wi h subs an ial clinical o
esea ch expe ience in men al heal h, e idenced by a his-
o y o pa ien ca e, pa icipa ion in clinical ials, esea ch
con ibu ions, o ad ocacy wo k. P o iciency in a ious
he apeu ic modali ies such as cogni i e-beha io al he -
apy, psycho he apy, and mind ulness-based in e en ions
was c ucial. Cul u al compe ence—unde s anding and
add essing he di e se cul u al and socioeconomic ac o s
in luencing men al heal h—was ano he c i ical c i e ion.
Las ly, we alued expe s open o explo ing he e hical
implica ions and po en ial applica ions o gene a i e lan-
guage echnologies in men al heal h ca e and challenges.
Limi a ions
A signi ican challenge encoun e ed was he ec ui -
men o domain expe s. Fo in e iew-based quali a i e
e iews, s anda d p ac ice ecommends a minimum o
i e expe s pe domain. Ques ionnai e-based quali a i e
e iews gene ally equi e a mo e ex ensi e pa icipan
base, ideally wi h a leas 50 esponden s. Despi e ex ensi e
ou each e o s, ou esponse a e was limi ed o 14 pa ic-
ipan s, comp ising ou in e iewees and 10 ques ionnai e
esponden s. This equa es o se en domain expe s o
each o he wo ca ego ies unde s udy. Al hough he num-
be o pa icipan s a e limi ed, he use o bo h quan i a i e
and quali a i e app oach o e oppo uni y o in-dep h
da a and insigh s. We also belie e ha he inclusion o
expe s om di e se backg ounds, cul u e, and con inen s
con ibu es posi i ely o he quali y o ou indings.
As men ioned ea lie , one o he s eng hs o his s udy
is he geog aphical di e si y o ou expe panel. We suc-
cess ully included a leas one domain expe om each
con inen ( e e o Figu e 2 o de ails), which bols e s he
alidi y o ou esul s. This di e se ep esen a ion helps
mi iga e egional biases and enhances he global ele ance
o ou indings.
Hence, we belie e ha despi e he limi a ion in e ms o
numbe s, he s udy p o ides meaning ul insigh s in o he
esea ch a ea. The limi a ions highligh a enues o u u e
23719621, 2025, 1, Downloaded om h ps://onlinelib a y.wiley.com/doi/10.1002/aaai.12208 by Readcube (Lab i a Inc.), Wiley Online Lib a y on [17/02/2025]. See he Te ms and Condi ions (h ps://onlinelib a y.wiley.com/ e ms-and-condi ions) on Wiley Online Lib a y o ules o use; OA a icles a e go e ned by he applicable C ea i e Commons License
6o 15 AI MAGAZINE
FIGURE 2 Dis ibu ion o domain expe s in he ields o
Clima e Change and Men al Heal h, ca ego ized by loca ion and
p o ession. In bo h domains, Asia and A ica p o ide he la ges
egional sha e o expe s, while No h Ame ica and Oceania
con ibu e he leas . The majo i y o expe s in bo h domains a e
om Academic & Resea ch backg ounds, accoun ing o 57% o he
o al, as opposed o 43% om Indus y.
esea ch, pa icula ly in b oadening he expe pa icipan
base o u he alida e and en ich he s udy’s conclusions.
FINDINGS
This sec ion elucida es ou s udy’s esul s, commencing
wi h a quan i a i e analysis o he cha bo s’ a e age pe o -
mance on he da ase o T ue/False ques ions wi hin he
Clima e Change and Men al Heal h domains. We subse-
quen ly p esen he simila i y sco es based on he me ics
men ioned abo e, compa ing he esponses o all h ee
cha bo s agains es ablished ac s o de e mine hei ac-
ual adhe ence. Las ly, we expand in o he quali a i e
insigh s de i ed om engaging wi h domain expe s, p o-
iding an in-dep h explo a ion o hei pe spec i es in bo h
domains o in e es . These indings a e consis en wi h
esea ch by Shneide man (2020) and Yang e al. (2023),
who ound ha LLMs ained on massi e da ase s can s ill
be suscep ible o misin o ma ion, pa icula ly when he
in o ma ion is cle e ly disguised.
Quan i a i e analysis: T ue/ alse p omp s
In his s udy, we e alua e he cha bo s’ abili y o disce n
he e aci y o s a emen s ela ed o clima e change and
TABLE 1 Compa a i e analysis o he selec ed cha bo s’
pe o mance in disce ning T ue/False s a emen s wi hin he ealms
o Clima e Change and Men al Heal h.
Domain P ecision Recall F1 sco e Accu acy
Clima e Change 88.4% 91.9% 90.1% 89.9%
Men al Heal h 90.1% 95.2% 92.6% 92.5%
men al heal h, in o de o quan i y i s le el o knowledge,
o how misin o med i migh be. U ilizing a quan i a i e
app oach, we analyze a T ue/False da ase and calcu-
la e c i ical pe o mance me ics. The analysis (Figu e 3)
includes a de ailed examina ion o ins ances whe e he
model inco ec ly classi ied ue s a emen s as alse ( alse
nega i es) and alse s a emen s as ue ( alse posi i es), as
well as accu a ely iden i ied ue ( ue posi i es) and alse
( ue nega i es) s a emen s.
An in-dep h analysis o he pe o mance me ics, as
de ailed in Table 1, shows ha in he ealm o Clima e
Change, ou cha bo s demons a es commendable accu-
acy wi h a p ecision a e o 88.4%. This indica es ha he
majo i y o he s a emen s classi ied as ue by he model
a e indeed co ec . The ecall a e o 91.9% u he sug-
ges s ha i success ully iden i ies a high pe cen age o he
ue s a emen s wi hin his domain. The F1 sco e s ands
a 90.1%, e lec ing a s ong o e all pe o mance. Howe e ,
he o e all accu acy, a 89.9%, while high, indica es he e is
oom o imp o emen in educing misin o ma ion when
i comes o clima e change.
In he Men al Heal h domain, he obse ed pe o mance
is no ably enhanced. I achie es a highe p ecision a e
o 90.1%, sugges ing ha i s capaci y o co ec ly iden i y
ue s a emen s is mo e e ined in his domain. The ecall
a e o 95.2% is pa icula ly imp essi e, indica ing ha he
model is highly e ec i e a cap u ing ue ins ances. The
F1 sco e, a an ele a ed 92.6%, poin s o a balanced and
e icien classi ica ion capabili y. Mo eo e , he accu acy
o 92.5% unde sco es a signi ican le el o eliabili y in he
Men al Heal h domain.
Quan i a i e analysis: Simila i y index
sco es
In he quan i a i e phase o ou analysis, we employed
h ee well-known me ics—BLEU, ROGUE, and
METEOR— om he machine ansla ion e alua ion
ield. These me ics adi ionally assess how closely
machine-gene a ed ex ma ches human ansla ion, in
e ms o bo h accu acy and con ex ual cohe ence. Fo his
s udy, we adap ed hese me ics o assess he pe o mance
o cha bo s, posi ing hei applicabili y beyond hei usual
con ex o ansla ion.
23719621, 2025, 1, Downloaded om h ps://onlinelib a y.wiley.com/doi/10.1002/aaai.12208 by Readcube (Lab i a Inc.), Wiley Online Lib a y on [17/02/2025]. See he Te ms and Condi ions (h ps://onlinelib a y.wiley.com/ e ms-and-condi ions) on Wiley Online Lib a y o ules o use; OA a icles a e go e ned by he applicable C ea i e Commons License
AI MAGAZINE 7o 15
FIGURE 3 Con usion ma ices depic ing he pe o mance o he selec ed cha bo s in answe ing whe he a ac gi en wi hin a p omp is
ei he ue o alse, o he Clima e Change and Men al Heal h domains. Fo Clima e Change, he e we e 1368 ue nega i es and 1436 ue
posi i es, wi h alse posi i es and nega i es a 188 and 127, espec i ely. In he Men al Heal h domain, he model p oduced 1253 ue nega i es
and 1301 ue posi i es, and lowe alse posi i es and nega i es a 143 and 65. These esul s indica e a high le el o accu acy in he model’s
knowledge ac oss bo h domains.
BLEU and ROGUE me ics a e designed o mea-
su e he p ecision and ecall o he cha bo s’ esponses
agains a benchma k o human-gene a ed ex s. METEOR
goes a s ep u he by including ad anced linguis ic
analysis—such as synonym ma ching, s emming, and
pa aph asing— o p o ide a mo e nuanced assessmen .
This me ic, he e o e, o e s a measu e o e alua ion ha
mo e closely app oxima es human judgmen by accoun -
ing o seman ic and con ex ual accu acy, in addi ion
o exac wo d co espondences. We analyzed he cha -
bo s’ ou pu s by compa ing hem o he e i iable answe s
wi hin ou da ase . To ensu e compa abili y, we no mal-
ized he esul ing simila i y sco es, aiming o a maximum
alue o 1. This s ep was c ucial, gi en ha ou “misin-
o med” p omp s o en led o cha bo esponses ha we e
sho e and subs an ially a ied om he ac ual esponses,
some imes esul ing in inaccu acies o misin o ma ion.
Al hough he e’s no absolu e h eshold se o misin o -
ma ion, hese no malized sco es se e as indica o s o
he deg ee o which he cha bo s’ esponses emula e he
ac ual da a.
Figu e 4p esen s he no malized simila i y index
sco es, which e lec he cha bo s’ accu acy in ela-
ion o he ac ual s a emen s p o ided. Wi hin he cli-
ma e change con ex , he Bing Cha bo , powe ed by
GPT-4, demons a ed he highes likelihood o deli e -
ing co ec esponses—e en when aced wi h mislead-
ing p omp s. Google’s Ba d, le e aging LaMDA, ollowed
closely, likely bene i ing om i s access o up- o-da e
online in o ma ion, in con as o Cha GPT (GPT-3.5),
which ope a es based on da a up un il 2021 and hence
o line. In he men al heal h a ena, Ba d was obse ed
o ha e he highes accu acy, sugges ing ha i s model
is well-equipped o handle e en ad e sa ially designed
p omp s.
FIGURE 4 The ba cha s illus a e he Simila i y Index
Sco es o he ollowing h ee LLM-powe ed cha bo s—Cha GPT
(GPT-3.5), Ba d (LaMDA), and Bing (GPT-4)—ac oss he ollowing
h ee e alua ion me ics: BLEU, ROUGE, and METEOR.
Quali a i e analysis
This sec ion de ails ou app oach o gauging he le els o
misin o ma ion and bias p esen in he esponses p o ided
23719621, 2025, 1, Downloaded om h ps://onlinelib a y.wiley.com/doi/10.1002/aaai.12208 by Readcube (Lab i a Inc.), Wiley Online Lib a y on [17/02/2025]. See he Te ms and Condi ions (h ps://onlinelib a y.wiley.com/ e ms-and-condi ions) on Wiley Online Lib a y o ules o use; OA a icles a e go e ned by he applicable C ea i e Commons License
8o 15 AI MAGAZINE
by h ee p ominen cha bo s—speci ically, hose ocused
on Clima e Change and Men al Heal h opics. To his
end, we sough insigh s om domain expe s in hese
espec i e ields.
Ini ially, we endea o ed o engage a b oad spec um
o specialis s o in-dep h in e iews based on he cha -
bo s’ esponses. As e ealed in Sec ion, he esponse a e
was limi ed o 14 expe s, spanning bo h domains. Ou
ini ial s a egy in ol ed o wa ding he cha bo -gene a ed
esponses o hese expe s, ollowed by in e iews a e
a week. Howe e , ime cons ain s necessi a ed a s a e-
gic swi ch a e in e iews wi h ou i s wo clima e
change expe s. We ansi ioned o a ques ionnai e-based
app oach, using simila ques ions om he in e iew
me hodology, which signi ican ly enhanced ime e i-
ciency. The ques ionnai es comp ised bo h closed- and
open-ended ques ions, adap ed o sui he expe s’ con e-
nience. This app oach was simila ly applied in he men al
heal h domain, whe e wo specialis s we e in e iewed,
and he emaining p o ided hei inpu s ia ques ion-
nai es. The expe in e iews a ied om 45 o 60 min,
encompassing 8–13 comp ehensi e ques ions, indica i e
o he dep h o hese discussions. Con e sely, he ques-
ionnai es, comp ising 11 i ems o clima e change expe s
and 16 o men al heal h expe s, equi ed 10–30 min
o comple e.
The clima e change- ocused ques ions sough expe
opinions on language-model-based cha bo s in aising
awa eness and hei e ec i eness in clima e change com-
munica ion. The assessmen co e ed di e se aspec s,
including hei ole in awa eness, challenges in p o iding
accu a e in o ma ion, po en ial biases and misin o ma-
ion, hei u ili y in adap a ion and mi iga ion s a egies,
use engagemen ea u es, in eg a ion wi h o he pla -
o ms, p omo ing sus ainable beha io s, and he ele ance
o hei in o ma ion in ligh o new scien i ic disco e ies.
In he men al heal h con ex , he ques ions e alua ed
he impac , e hical conside a ions, e ec i eness, and limi-
a ions o cha bo s. Key a eas o inqui y included hei ole
in s igma educ ion and awa eness, pe sonalized suppo
e ec i eness, speci ic men al heal h condi ions add essed,
necessi y o human in e en ion, os e ing us and
con iden iali y, ea ly de ec ion and p e en ion, inclusi -
i y, cul u al sensi i i y, po en ial d awbacks, c i e ia o
success measu emen , complemen ing exis ing se ices,
empa hy le el, and hei abili y o ecommend ailo ed
men al heal h esou ces.
Ou analysis o expe esponses was conduc ed using
a hema ic analysis app oach, adhe ing o he guidelines
ou linedbyB aunandCla ke(B aunandCla ke2006).
This me hod allowed o he sys ema ic iden i ica ion
and o ganiza ion o isible and alid pa e ns wi hin he
da a. Ini ially, we i e a i ely ead h ough he esponses o
ex ac signi ican s a emen s and concep s conce ning he
use and implica ions o cha bo s in he domains o Clima e
Change and Men al Heal h, which we hen ep esen ed
as codes. Thema ic sa u a ion was achie ed when no new
codes could be iden i ied. In he inal phase o ou analy-
sis, we syn hesized hese hemes in o a cohe en na a i e.
This in ol ed linking he hemes o ou esea ch ques ions
and d awing conclusions abou he ole and impac o cha -
bo s in he espec i e ields. Ou in e p e a ions, g ounded
in he da a, include ep esen a i e quo es om he expe s
o illus a e hei pe spec i es.
The subsequen sec ions o his pape will del e in o
hese speci ic hemes, p o iding a de ailed explo a ion o
he u ili y and e ec i eness o cha bo s in he con ex o
Clima e Change and Men al Heal h.
(I.) Expe s’ pe spec i es on misin o ma ion and bias in
clima e change
In ou in es iga ion, we p esen ed each expe wi h
ques ions conce ning he cha bo s’ esponses on Clima e
Change. The ocus o hese inqui ies was o unde s and he
expe s’ pe cep ions o he po en ial impac hese cha bo s
migh ha e on use sa e y and in o ma ion dissemina ion.
We p esen ou indings unde ou main hemes.
1. Role o LLM-based cha bo s in clima e change
awa eness: We asked he expe s abou he signi -
icance o LLM-based cha bo s in enhancing public
awa eness o clima e change. The majo i y, ba ing wo,
we e op imis ic, acknowledging ha he newe gen-
e a ion o cha bo s could play a subs an ial ole in
sp eading awa eness and dissemina ing in o ma ion.
On he con a y, one expe exp essed skep icism abou
he dep h and u ili y o he cha bo s’ esponses, liken-
ing hem o shallow in e ne sea ches. Ano he poin ed
ou no disce nible ad an age o e adi ional sea ch
engines. Despi e hese di e ing iews, he e was a con-
sensus ha mo e specialized and expe -d i en models
could yield mo e p ecise and eliable in o ma ion han
he h ee cha bo s e alua ed. The expe s sugges ed
imp o emen s such as enabling cha bo s o p o ide
simpli ied ye comp ehensi e answe s, elucida e he
easoning behind hei esponses, and anspa en ly
ci e hei in o ma ion sou ces.
2. Challenges in gene a ing accu a e clima e- ela ed
in o ma ion: We sough he expe s’ iews on he
obs acles aced by cha bo s in deli e ing p ecise and
us wo hy in o ma ion on clima e change. A sig-
ni ican po ion o he esponden s aised conce ns
abou he da a sou ces used o ain hese models.
They emphasized he lack o assu ance ega ding he
quali y and c edibili y o he sou ces ci ed by he
cha bo s. Ano he common issue highligh ed was he
23719621, 2025, 1, Downloaded om h ps://onlinelib a y.wiley.com/doi/10.1002/aaai.12208 by Readcube (Lab i a Inc.), Wiley Online Lib a y on [17/02/2025]. See he Te ms and Condi ions (h ps://onlinelib a y.wiley.com/ e ms-and-condi ions) on Wiley Online Lib a y o ules o use; OA a icles a e go e ned by he applicable C ea i e Commons License
AI MAGAZINE 9o 15
inconsis ency in he cha bo s’ esponses. Expe s no ed
ha o ce ain que ies, he models p oduced as ly
di e ing answe s, which could lead o con usion. Addi-
ionally, he gene ali y o he esponses was a poin o
con en ion. Expe s poin ed ou ha clima e change
e ec s and solu ions a e o en loca ion-speci ic, ye he
cha bo s ended o p o ide b oad, uni e sally applica-
ble answe s. This, hey sugges ed, migh s em om he
na u e o he p omp s ed o he models. Mo e accu-
a e and ailo ed esponses could po en ially be elici ed
wi h p omp s ha include mo e de ailed and speci ic
ins uc ions o con ex .
3. Expe insigh s on biases and misin o ma ion in
clima e da a dissemina ion: We inqui ed abou he
expe s’ pe cep ion o po en ial biases and misin o -
ma ion in he cha bo -gene a ed esponses on clima e-
ela ed opics. A no able po ion o he expe s, abou
hal , iden i ied ins ances o da a exagge a ion leading
o misin o ma ion. They exp essed conce ns abou he
cha bo s being ained on da ase s wi h un e i ied o
non ep oducible sou ces, a signi ican issue in a ield
p one o alse nega i es. Such p ac ices, hey cau ioned,
esul in he p opaga ion o un e i ied and po en-
ially misleading in o ma ion. Con e sely, he o he
hal acknowledged he gene al adequacy o he in o -
ma ion p o ided by he cha bo s bu s essed he need
o s ingen measu es o ensu e he e i iabili y o
dissemina ed da a. Despi e hese di e gen iews, a
unanimous conce n among all expe s was he lack o
demog aphic sensi i i y in he cha bo s’ esponses. The
expe s obse ed ha he cha bo s ended o p o ide
uni o m answe s i espec i e o a ying demog aphic
con ex s. Fo ins ance, he esponse gi en o a middle-
aged A ican male was iden ical o ha gi en o a young
Eu opean emale, o e looking he speci ic ulne abil-
i ies and con ex s o di e en demog aphic g oups.
This could mean ha he in o ma ion abou he pe -
son behind he p omp s was no conside ed much
in gene a ing exac answe s. One expe poignan ly
ema ked ha a eenage and a senio ci izen should
a he ecei e ela i e esponses based on hei p e i-
ous knowledge highligh ing he need o mo e nuanced
and demog aphic-awa e cha bo s.
4. Cha bo s as ools o p omo ing sus ainable beha -
io s: We explo ed he expe s’ iews on he po en ial
o hese cha bo s in os e ing sus ainable beha io s
and li es yle changes among use s. The esponse was
unanimously posi i e ac oss he boa d. The expe s
acknowledged he impo ance o eliable in o ma ion
sou ces bu we e op imis ic abou he ole o cha -
bo s, especially hose specialized in clima e change, in
in luencing use beha io owa ds sus ainabili y. They
concu ed ha app op ia ely designed cha bo s could
e ec i ely encou age use s o adop mo e en i on-
men ally iendly p ac ices. Fu he mo e, some expe s
p oposed speci ic ea u es ha could enhance he cha -
bo s’ capabili y o ad oca e o sus ainabili y. These
sugges ions included pe sonalized ad ice based on
use ’s li es yle, in e ac i e guides on educing ca bon
oo p in s, and imely upda es on en i onmen al issues
and solu ions, all ailo ed o engage use s ac i ely in
sus ainabili y e o s.
Discussion: Ou panel o clima e change expe s con-
cu s ha , despi e he need o conside able imp o e-
men s in sa e y and eliabili y, LLM-backed cha bo s hold
immense po en ial o impac ul applica ions. These AI-
d i en ools a e lauded o hei capaci y o e olu ionize
he dissemina ion o c ucial in o ma ion and o p omo e
en i onmen al consciousness among he public. None he-
less, expe s s ess he impe a i e o s ingen alida ion
and con inuous e inemen o hese sys ems o bols e hei
e ec i eness and c edibili y in in o ma ion dissemina ion.
One p ominen ecommenda ion om he expe s is
he s a egic cu a ion o aining da ase s o LLMs, ad o-
ca ing o he inclusion o da a p ima ily om e i i-
able and expe -endo sed sou ces. This ecommenda ion
a ises om hei obse a ion ha he cha bo s in ou
s udy o en e e enced ma e ials indisc imina ely, link-
ing o a icles ha may be obsole e o om publishe s
lacking o icial s anding in academic esea ch. Mo e-
o e , ins ances o non unc ional o po en ially decep i e
links u he highligh he isks associa ed wi h un e ed
in o ma ion sou ces.
The expe s also highligh ed a c i ical conce n ega ding
he inhe en bias wi hin p omp s hemsel es, sugges -
ing ha LLMs may p io i ize comple ing a use ’s eques
o e ensu ing he accu acy and eliabili y o he p o ided
in o ma ion. This endency unde sco es a undamen al
challenge: he need o ine- une LLMs o disce n and p io -
i ize high-quali y, us wo hy con en in hei esponses,
ega dless o he na u e o he p omp s hey ecei e.
(II.) Expe s’ pe spec i es on misin o ma ion and bias in
men al heal h
In discussions wi h domain expe s, we sough o unde -
s and he beha io and po en ial o he h ee cha bo s in
he men al heal h domain. Ou dialog ocused on se e al
key a eas:
1. Impac on s igma and awa eness: Ou inqui y in o
expe s’ pe cep ions commenced wi h ques ions abou
he po en ial impac o cha bo s on s igma educ ion
and awa eness enhancemen in men al heal h. The
esponse was uni o mly posi i e, wi h expe s ecog-
nizing he subs an ial alue o LLM-backed cha bo s
23719621, 2025, 1, Downloaded om h ps://onlinelib a y.wiley.com/doi/10.1002/aaai.12208 by Readcube (Lab i a Inc.), Wiley Online Lib a y on [17/02/2025]. See he Te ms and Condi ions (h ps://onlinelib a y.wiley.com/ e ms-and-condi ions) on Wiley Online Lib a y o ules o use; OA a icles a e go e ned by he applicable C ea i e Commons License
