Jou nal o Psychology and AI
ISSN: 2997-4100 (Online) Jou nal homepage: www. and online.com/jou nals/ pai20
F om human a e ac o machine ou pu :
au oma ing he “a ” o psychological
measu emen
Fe nando Ma molejo-Ramos, Okan Bulu , Luis Anunciaçáo, Louise Ma ques,
Abhina a Ba haku , Jose Kund a , Ka el Rečka, Özge Ka akale, Juan C.
Co ea, Luis Albe o Pinos-Ullau i, Raydonal Ospina & Julian Tejada
To ci e his a icle: Fe nando Ma molejo-Ramos, Okan Bulu , Luis Anunciaçáo, Louise
Ma ques, Abhina a Ba haku , Jose Kund a , Ka el Rečka, Özge Ka akale, Juan C. Co ea, Luis
Albe o Pinos-Ullau i, Raydonal Ospina & Julian Tejada (2025) F om human a e ac o machine
ou pu : au oma ing he “a ” o psychological measu emen , Jou nal o Psychology and AI, 1:1,
2561692, DOI: 10.1080/29974100.2025.2561692
To link o his a icle: h ps://doi.o g/10.1080/29974100.2025.2561692
© 2025 The Au ho (s). Published by In o ma
UK Limi ed, ading as Taylo & F ancis
G oup.
Published online: 09 Oc 2025.
Submi you a icle o his jou nal
View ela ed a icles
View C ossma k da a
Full Te ms & Condi ions o access and use can be ound a
h ps://www. and online.com/ac ion/jou nalIn o ma ion?jou nalCode= pai20
RESEARCH ARTICLE
F om human a e ac o machine ou pu : au oma ing he “a ” o
psychological measu emen
Fe nando Ma molejo-Ramos
a
, Okan Bulu
b
, Luis Anunciaçáo
c
, Louise Ma ques
c
, Abhina a Ba haku
d
,
Jose Kund a
e
, Ka el Rečka
e
, Özge Ka akale
, Juan C. Co ea
g
, Luis Albe o Pinos-Ullau i
h,i,j
,
Raydonal Ospina
k
and Julian Tejada
l
a
College o Educa ion, Psychology, and Social Wo k, Flinde s Uni e si y, Adelaide, Aus alia;
b
Cen e o Resea ch in Applied
Measu emen and E alua ion, Uni e si y o Albe a, Edmon on, Canada;
c
Depa men o Psychology, Ca holic Uni e si y o Rio de
Janei o, Rio de Janei o, B azil;
d
Educa ion Fu u es, Uni e si y o Sou h Aus alia, Adelaide, Aus alia;
e
Depa men o Psychology,
Uni e si y o Os a a, Os a a, Czech Republic;
School o Psychology, Uni e si y o Wollongong, Wollongong, Aus alia;
g
Resea ch
& De elopmen Uni , C i ical Cen ali y Ins i u e, Mon e ey, Mexico;
h
IMEC esea ch g oup ITEC, KU Leu en, Ko ijk, Belgium;
i
Facul y o Psychology and Educa ional Sciences, KU Leu en, Ko ijk, Belgium;
j
Cen e o Digi al Sys ems, IMT No d Eu ope, Douai,
F ance;
k
Depa men o de Es a ís ica, LInCA, Uni e sidade Fede al da Bahia, Cidade Uni e si á ia, Bahia, B azil;
l
Depa men o
Psychology, Fede al Uni e si y o Se gipe, São C is ó ão, B azil
ABSTRACT
C ea ing psychological assessmen ools is c ucial o esea ch bu adi ionally expen-
si e and ime-consuming. While La ge Language Models (LLMs) show p omise o
au oma ing his p ocess, exis ing app oaches lack sys ema ic, use - iendly me hodolo-
gies g ounded in psychome ic p inciples. This s udy p esen s an enhanced
Psychome ic I em Gene a o (PIG) me hod using con e sa ional LLMs wi h P oblem-
Sol ing Plans (PSP) and Chain-o -Though (CoT) p omp ing. Th ee demons a ions ali-
da ed he app oach: Gemini 1.5 Flash gene a ed 20 “p opensi y o us AI” i ems wi h
s ong seman ic cohe ence; Claude 3 Opus c ea ed 20 “AI anxie y” i ems ha ou pe -
o med human-gene a ed e sions linguis ically; and a 6-i em “AI adop ion in online
lea ning” scale was de eloped and alida ed wi h 1,233 pa icipan s using mul i e se
analysis. Resul s demons a e ha LLMs can p oduce psychome ically sound i ems. The
AI-gene a ed anxie y scale showed supe io linguis ic p ope ies compa ed o human
al e na i es, while he lea ning scale exhibi ed good in e nal consis ency, i em homo-
genei y, and clea wo- ac o s uc u e ac oss mul iple analy ical eams. The s udy
es ablishes a PSP-CoT amewo k ha imp o es LLM ou pu quali y, o e ing esea che s
a cos -e ec i e, accessible scale de elopmen me hodology. Howe e , indings empha-
size ha human o e sigh , igo ous alida ion, and e hical conside a ions emain essen-
ial componen s o he p ocess.
ARTICLE HISTORY
Recei ed 21 Ap il 2025
Accep ed 15 Augus 2025
KEYWORDS
La ge language models;
psychome ics; psychome ic
i em gene a o ; chain-o -
hough
1. In oduc ion
The use o su eys and/o sel - epo scales in psychology has been inc easing in ecen decades (Cla k &
Wa son, 2019). These ins umen s, a e ha ing enough sou ces o e idence o alidi y (Associa ion, 2014),
a e aluable ools o measu ing cons uc s ha a e no di ec ly obse able. Thei cons uc ion is no an easy
p ocess in ol ing a se ies o s eps, om concep ualisa ion up o alida ion, including he i em gene a ion
(Jebb e al., 2021). The la e is a special s ep in which, usually om a la ge se o i ems, hose ha bes i
he su ey objec i es a e selec ed, wi h a special ocus on i em wo ding. Cla k and Wa son (2019) desc ibe
his s ep as one o he mos c i ical in he de elopmen o a scale, because by means o psychome ic
echniques, i is possible o elimina e p oblema ic i ems, bu no o c ea e missing i ems ha should ha e
been included.
Recen ly, Gö z e al. (2023) in oduced he Psychome ic I em Gene a o (PIG), a pionee ing me hod
ha uses he GPT-2 neu al ne wo k wi hin a Py hon-based Google Colab en i onmen o au oma e i em
gene a ion. Thei con ibu ion elies on simpli ying and accele a ing he p ocess o c ea ing i ems o
psychological assessmen s by p oducing con en ha aligns wi h he in ended cons uc s, such as pe son-
ali y ai s o a i udes. While g oundb eaking, hei o iginal me hod has wo p ima y limi a ions in he
CONTACT Fe nando Ma molejo-Ramos [email p o ec ed]
JOURNAL OF PSYCHOLOGY AND AI
2025, VOL. 1, NO. 1, 2561692
h ps://doi.o g/10.1080/29974100.2025.2561692
© 2025 The Au ho (s). Published by In o ma UK Limi ed, ading as Taylo & F ancis G oup.
This is an Open Access a icle dis ibu ed unde he e ms o he C ea i e Commons A ibu ion License (h p://c ea i ecommons.o g/licenses/by/4.0/), which pe mi s
un es ic ed use, dis ibu ion, and ep oduc ion in any medium, p o ided he o iginal wo k is p ope ly ci ed. The e ms on which his a icle has been published allow
he pos ing o he Accep ed Manusc ip in a eposi o y by he au ho (s) o wi h hei consen .
cu en echnological landscape: i equi es use s o in e ac wi h and modi y Py hon code, c ea ing a ba ie
o non-p og amme s, and i is based on he now-ou da ed GPT-2 a chi ec u e (Gö z e al., 2023). Ou
s udy p oposes a subs an ial e olu ion o his concep , c ea ing a mo e accessible, lexible, and powe ul
Psychome ic I em Gene a ion a ian wi h h ee key imp o emen s. Fi s , ou app oach is en i ely code-
ee and pla o m-agnos ic, elying exclusi ely on con e sa ional p omp enginee ing. This emo es
echnical ba ie s, empowe ing any esea che o le e age au oma ed i em gene a ion using eadily a ailable
web in e aces. Second, we u ilise s a e-o - he-a LLMs (e.g. Google’s Gemini 1.5 and An h opic’s Claude 3
Opus), which o e signi ican ly mo e ad anced easoning, con ex ual unde s anding, and ex gene a ion
capabili ies han GPT-2. Thi d, we in oduce a no el P oblem-Sol ing Plan (PSP) and Chain-o -Though
(CoT) p omp ing amewo k. This s uc u ed app oach is designed o imp o e he cohe ence and psycho-
me ic ele ance o he gene a ed i ems, mo ing beyond simple one-sho p omp s o a guided, i e a i e
dialogue wi h he LLM. This combina ion o accessibili y, ad anced model usage, and a sophis ica ed
p omp ing s a egy ep esen s a signi ican and necessa y ad ancemen o e exis ing au oma ed i em
gene a ion me hods.
LLMs a e machine lea ning models able o gene a e ex in human languages o coun less ypes o
con ex s, in which he gene a ed ex esembles human esponses (Gi ay, 2023; Hen ickson &
Me oño-Peñuela, 2025). In e ac ion wi h hese models is conduc ed h ough a cha -like command line,
whe e commands a e yped as human con e sa ions, and his is he eason why he concep o p omp
enginee ing was bo n, o assis in he “con e sa ion” wi h hese sys ems. P omp enginee ing is an eme ging
ield ocused on he me hodical c ea ion and e inemen o p omp s – he ins uc ions used o in e ac wi h
LLMs – h ough i e a i e cycles o modi ica ion (see Figu e 1).
The e a e di e en ypes o p omp ing: Ze o-Sho P omp ing, in which he ins uc ions a e gi en
wi hou examples; Chain-o - hough (CoT), which inco po a es in e media e s eps o guide he easoning,
b eaking down he ask in o o de ing sub-goals and o e ing examples o s ep-by-s ep easoning (Chu e al.,
Figu e 1. Scheme o how an LLM wo ks. I all s a s wi h a p omp o inpu ex , a human con e sa ion-s yle ins uc ion ha
is seg ega ed (Tokeniza ion) in o a g oup o cha ac e s based on punc ua ion ma ks, spaces, o special cha ac e s. These
g oups can ep esen wo ds o subwo ds and a e called okens. Once he okenize algo i hm has seg ega ed he ex , each
oken is iden i ied wi h an ID, which in u n, is associa ed wi h a dense ec o o high-dimensional space (i.e. GPT-3 uses
ec o s o 12,288 dimensions), o ming an embedding ma ix. These ec o ep esen a ions a e sui able o neu al ne wo k
p ocessing and con ain all LLM aining da a and will allow he algo i hm o cap u e seman ic ela ionships and con ex . The
embedded ex sequence is p ocessed ia an encode ne wo k (a ecu en neu al ne wo k [RNN] o ans o me -based
a chi ec u e) which analyses wo d ela ionships and builds a con ex ual ep esen a ion, cap u ing knowledge abou he
o e all meaning and low o he inpu . In he con ex wi h a en ion phase, he mos signi ican pa s o he con ex
gene a ed by he encode a e highligh ed (i.e. ‘scales’ in his con ex do no e e o de ices used o measu ing a pe son’s
body mass o he small igid pla es ha g ow ou o he skin o a ish). Then, he decode ne wo k, ano he RNN o
ans o me -based a chi ec u e, uses he encode con ex and i s own in e nal s a e o gene a e he ou pu ex wo d by
wo d by p edic ing he nex wo d in he sequence based on he lea ned in o ma ion. Fo mo e de ails isi : h ps://cu .Ly/
lw8PKU7Y.
2F. MARMOLEJO-RAMOS ET AL.
2023); Few-Sho P omp ing, simila o Ze o-Sho , bu wi h some examples; and sel -consis ency (S-C) in
which simila ins uc ions a e gi en o ob ain a pool o esponses om which he mos equen is selec ed
(Ahmed & De anbu, 2023) (see also Wei e al., 2022). In ou case, we p opose o use a combina ion o CoT
wi h Few-Sho o con igu e ou adap a ion o he PIG app oach, which can be applied o any LLM a ailable.
The PSP componen , inspi ed by plan-and-sol e p omp ing s a egies (L. Wang e al., 2023), speci ically
ad ances beyond p io p omp enginee ing me hods by es ablishing an up on , op-down plan ha de ines
he scope, goals, and bounda ies o he in e ac ion be o e he CoT begins. While a s anda d CoT p omp
b eaks a ask in o s eps (Y. Wang e al., 2024), a PSP ensu es hose s eps a e logically cohe en and di ec ed
owa ds a p e-de ined psychome ic objec i e. Fo ins ance, he PSP would i s es ablish he goal: “De elop
a 5-i em scale o he ‘socio echnical blindness’ dimension o AI anxie y, ensu ing wo i ems a e e e se-
coded and all i ems a e app op ia e o a lay audience.” The subsequen CoT hen execu es his plan s ep-by-
s ep. This “plan-and-sol e” app oach educes he isk o he LLM de ia ing in o i ele an o incohe en
pa hs, a common issue wi h less-s uc u ed p omp ing, he eby making he i em gene a ion p ocess mo e
e icien , a ge ed, and aligned wi h psychome ic goals om he ou se .
Ou p oposed app oach p oposed he ein equi es no echnical expe ise and is a ailable o ee use. I
o e comes p io ba ie s o au oma ic i em gene a ion like in lexibili y, inaccessibili y, and compu a ional
demands. Addi ionally, ou app oach enables esea che s o b eak ee om eliance on manual i em
w i ing, allowing algo i hms o inally speak he language o psychome ics. O e all, ou s udy aims o
p o ide esea che s wi h a cos -e ec i e, e sa ile a i icial in elligence solu ion o psychological
measu emen .
In he nex sec ion, we illus a e h ough h ee demons a ions, ou non- echnical p oposed app oach on
gene a ing psychome ic i ems. Fi s , we employed Gemini 1.5 Flash wi h ze o-sho p omp ing o gene a e
20 ini ial ques ions measu ing “p opensi y o us AI”, which we e hen analysed using na u al language
p ocessing (NLP) me hods. Second, using Claude 3 Opus, we gene a ed and linguis ically e alua ed 20 i ems
a ge ing AI anxie y, inspi ed by Y.-Y. Wang & Wang (2022). Thi d, by in en ionally le e aging Gemini 1.5
Flash’s endency o gene a e imagina i e o ab ica ed con en om minimal p omp s, we p oduced o e 20
di e se i ems assessing AI us , wi h a ocus on adop ion and pe cep ion. Expe alida ion e ined his se
o six i ems, whose psychome ic p ope ies we e subsequen ly con i med using pa icipan esponses. The
expe s in ol ed in his p ocess we e academics specialising in educa ional echnology, a i icial in elligence,
online educa ion, da a analy ics, and psychome ics. The LLM models used in he demons a ions we e
eleased in Janua y 2024. We hen discuss how embedding p omp s wi hin a p oblem-sol ing plan can lead
o ui ul in e ac ions wi h LLMs and, consequen ly, in o ma i e ou pu s. Finally, in he discussion sec ion,
we elabo a e on psychome ics as a science and he ole o LLMs wi hin i .
2. Demons a ions
2.1. P oducing 20 usable i ems o measu e “p opensi y o us in AI”
We eso ed o a CoT con e sa ional s yle o que y Gemini. We s a ed wi h a ze o-sho p omp ing
s yle: “P o ide 10 up- o-da e psychome ic scales o measu e people’s p opensi y o us AI. P o ide
eal academic e e ences o each.” Pu pose ully, we aimed o use Gemini’s hallucina ions as a way o
ob ain esul s. Tha is, ins ead o he usual app oach o a oiding and disca ding AI hallucina ions, we
used i s po en ial insigh s as a p o ing g ound. As expec ed, Gemini p o ided some eal and some
hallucina ed e e ences (anecdo ally, howe e , yping hese illuso y e e ences in o Google p o ided
use ul eal e e ences). We hen con inued wi h he ques ion, “Based on he scales abo e, wha ’s he
o e all de ini ion o ‘p opensi y o us AI’?” Gemini hen p o ided an educa ed esponse and b oke
down he concep o p opensi y o us in AI in ou dimensions: i) us in AI compe ence, ii) us in
AI bene olence, iii) us in AI anspa ency, and i ) us in AI ai ness. Gemini also p o ided
de ini ions o each dimension and examples o how he p opensi y o us AI migh mani es i sel
in eal-wo ld beha iou s (e.g. Gemini said, “A pe son wi h a high p opensi y o us AI migh be mo e
likely o use a sel -d i ing ca ”). Then we asked Gemini: “P o ide psychome ic i ems ha will measu e
‘p opensi y o us in AI’ based on you de ini ion abo e. P o ide i e i ems o T us in AI
compe ence, 5 i ems o T us in AI bene olence, i e i ems o T us in AI anspa ency, and 5
JOURNAL OF PSYCHOLOGY AND AI 3
i ems o T us in AI ai ness”. A e Gemini’s ou pu , we asked o “ ew i e he abo e i ems so ha hey
a e o be answe ed on a con inuous likelihood scale be ween “ e y unlikely” and ‘ e y likely” and a e
Gemini’s ou pu we inally asked o “ ew i e he abo e i ems so ha hey a e wo ded in he i s
pe son”. Fou o he i ems we e:
●I am likely o us ha AI sys ems can be used o make my li e be e (i em gene a ed o he “ us in
AI compe ence” dimension)
●I am likely o belie e ha AI sys ems a e designed o ac in he bes in e es o humans (i em gene a ed
o he “ us in AI bene olence” dimension)
●I am likely o us ha AI sys ems will be used in a anspa en and accoun able manne (i em
gene a ed o he “ us in AI anspa ency” dimension)
●I am likely o belie e ha AI sys ems ea all people ai ly and equi ably (i em gene a ed o he “ us
in AI ai ness” dimension)
Upon ini ial examina ion, he gene a ed i ems appea o be ele an and align well wi h he ou
dimensions o he “p opensi y o us AI” as de ined by Gemini. Mo e speci ically, o he ini ial
human e alua ion, we selec ed se e al c i e ia: 1) he gene a ed ex mus be g amma ically cohe en , 2)
he gene a ed ex mus be ele an o he gi en opic, and 3) a esea che un amilia wi h he s udy’s
design mus be unable o disce n whe he he i ems o igina e om a human-designed es o a e AI-
gene a ed con en .
Howe e , when he i ems a e conside ed as a co pus, we can assess whe he his co pus exhibi s ex ual
dimensionali y using an NLP me ic, such as cosine simila i y (Singh & Singh, 2021). Cosine simila i y
e alua es he simila i y be ween wo non-ze o ec o s by compu ing he cosine o he angle be ween hem.
The alues ange om −1 o 1: 1 signi ies ha he ec o s a e iden ical (pe ec ly aligned), 0 indica es ha
he ec o s a e o hogonal (comple ely dissimila ), and −1 means he ec o s a e diame ically opposed. In
he esul ing ma ix, mos alues exceed 0.75, showing ha he ou i ems a e highly ela ed and sugges ing
ha he co pus canno be di ided in o dis inc dimensions.
Howe e , i is impo an o no e ha he esul s may a y sligh ly om hose epo ed he e i he p omp s
a e used again in he u u e. This is because LLMs ha e a ious hype -pa ame e s, such as empe a u e, Top
P, equency penal y, and p esence penal y, ha in luence he andomness o hei ou pu s. The i s
pa ame e is empe a u e, which con ols he andomness o he model’s ou pu . A highe empe a u e
se ing yields mo e a ied and unp edic able esponses, whe eas a lowe empe a u e se ing gene a es mo e
de e minis ic and conse a i e ou pu s. The second pa ame e is Top P, also known as nucleus sampling,
which de e mines he p obabili y h eshold o oken selec ion. A highe Top p alue allows o a wide
ange o okens o be conside ed, esul ing in mo e andom ou pu . In con as , a lowe Top p alue es ic s
he selec ion o mo e likely okens, educing andomness. The hi d pa ame e is he equency penal y,
which dec eases he likelihood o he model epea ing he same wo ds o ph ases. A highe penal y se ing
discou ages epe i ion and p omo es he use o unique wo d choices. Finally, he p esence penal y pa a-
me e encou ages he in oduc ion o new concep s o opics in he ou pu . A highe p esence penal y
se ing makes he model less likely o epea he same opics, os e ing mo e di e se con en gene a ion (see
Ding e al., 2023; Zhang e al., 2024). The same a gumen applies o he esul s o he nex demons a ion.
To con ex ualise hese indings, i is ins uc i e o compa e he gene a ed i ems o exis ing alida ed
scales. The ield has se e al es ablished measu es, such as he 12-i em T us in Au oma ion Scale (TIAS)
(McG a h e al., 2025) and he 9-i em Human-Compu e T us Scale (HCTS) (Gula i e al., 2019). The TIAS,
o example, is a obus measu e wi h high in e nal consis ency (C onbach’s α¼0:94) ha assesses us
along dimensions o pe o mance and in eg i y. A key ea u e o he TIAS is i s inclusion o i e e e se-
sco ed i ems (e.g. “The sys em is decep i e”) o mi iga e acquiescence bias. Ou LLM-gene a ed i ems o
“compe ence” and “bene olence” align concep ually wi h he TIAS’s “abili y” and “in eg i y” dimensions
(McG a h e al., 2025). Howe e , a no able di e ence is ha ou ini ial ze o-sho p omp ing did no
p oduce any e e se-sco ed i ems. This compa ison highligh s a c i ical poin : while LLMs can eadily
gene a e opically ele an i ems, achie ing he psychome ic sophis ica ion o es ablished scales like he
TIAS equi es mo e ad anced p omp ing s a egies, such as explici ly ins uc ing he model o gene a e
nega i ely-wo ded i ems o ensu e a balanced scale.
4F. MARMOLEJO-RAMOS ET AL.
2.2. P oducing 20 usable i ems o measu e AI anxie y
Y.-Y. Wang and Wang (2022) p oposed a 21-i em scale o measu e AI anxie y (AIA). These i ems
ep esen ed ou AIA dimensions: eigh i ems ela ed o lea ning, six ela ed o job eplacemen , ou
ela ed o socio echnical blindness, and h ee ela ed o AI con igu a ion. In hei s udy, hey de ined AIA as
“as an o e all, a ec i e esponse o anxie y o ea ha inhibi s an indi idual om in e ac ing wi h AI. Thus,
AIA may be ope a ionally conside ed as a gene al pe cep ion o belie wi h mul iple dimensions. Mo eo e ,
AIA in he p esen esea ch con ex ocuses on he a iable i sel , a he han he p ocess o esponse o
model e alua ion, which p omo es he ope a ion o AIA as a single a iable, independen o nume ous
an eceden s o consequences” (p. 621). Those i ems we e gene a ed in he s anda d way; esea che s c ea ed
a lis o 59 ques ions o measu e AIA and hen e iewed he ques ions wi h expe s o ensu e hey we e clea
and co e ed all aspec s o AIA.
We es ed ou p oposed app oach by cha ing wi h Claude o gene a e 20 i ems sui able o assessing he
“lea ning” dimension o AIA. Table 1 shows he p omp used. Claude gene a ed 20 usable i ems, and one
las p omp equi ed Claude o indica e how o a e he i ems, as such in o ma ion was no explici in i s i s
ou pu (“How a e hese i ems o be esponded o?”). We lis below h ee o he i ems gene a ed:
●T ying o unde s and he ad anced algo i hms behind AI sys ems makes me anxious
●Adap ing o apid changes in AI echnologies makes me anxious
●Lea ning new e ms and concep s ela ed o AI makes me anxious
I is clea ha he gene a ed i ems ollowed a linguis ic s uc u e simila o he eigh i ems c ea ed by Y.-Y.
Wang and Wang (2022), bu mos impo an ly, he Claude-gene a ed i ems we e seman ically ela ed o he
keywo ds AI, anxie y, and lea ning. When compa ing i ems gene a ed by humans wi h hose p oduced by
An h opic’s LLM, Claude, ou applica ion o in o ma ion heo y e eals ha he coe icien o mu ual
Table 1. P omp submi ed o Claude in o de o gene a e 20 i ems in he “lea ning dimension” o an AI anxie y scale.
New i ems o he lea ning dimension o an AI anxie y scale
A i icial In elligence Anxie y (AIA) can be de ined as a gene al, a ec i e eac ion o ea o anxie y ha p e en s an indi idual om in e ac ing
wi h a i icial in elligence. The ollowing eigh psychome ic i ems ep esen he lea ning dimension o AIA:
1. Lea ning o unde s and all o he special unc ions associa ed wi h AI echnique/p oduc makes me anxious.
2. Lea ning o use AI echniques/p oduc s makes me anxious.
3. Lea ning o use speci ic unc ions o an AI echnique/p oduc makes me anxious.
4. Lea ning how an AI echnique/p oduc wo ks makes me anxious.
5. Lea ning o in e ac wi h an AI echnique/p oduc makes me anxious.
6. Taking a class abou he de elopmen o AI echniques/p oduc s makes me anxious.
7. Reading an AI echnique/p oduc manual makes me anxious.
8. Being unable o keep up wi h he ad ances associa ed wi h AI echniques/p oduc s makes me anxious.
Gene a e 20 i ems simila o he examples abo e, wi h he aim o assessing he “lea ning” dimension o AIA.
I ems gene a ed:
1.T ying o unde s and he ad anced algo i hms behind AI sys ems makes me anxious.
2.Lea ning how o weak he se ings o an AI sys em o imp o e i s pe o mance makes me anxious.
3. Reading esea ch pape s abou new AI echniques makes me anxious.
4. A ending semina s on AI de elopmen me hods makes me anxious.
5. Lea ning abou di e en AI p og amming languages and amewo ks makes me anxious.
6. Figu ing ou how o in eg a e AI in o exis ing sys ems and wo k lows makes me anxious.
7. T oubleshoo ing e o s in AI sys ems makes me anxious.
8. Being asked o explain how an AI sys em wo ks makes me anxious.
9. En olling in uni e si y cou ses on AI and machine lea ning makes me anxious.
10. A emp ing o build my own basic AI p o o ype makes me anxious.
11. Reading books ha explo e he social implica ions o AI makes me anxious.
12. Lea ning abou ad ances in AI ha could displace human jobs makes me anxious.
13. T ying o unde s and an AI sys em’s aining da a and models makes me anxious.
14. Lea ning new e ms and concep s ela ed o AI makes me anxious.
15. Adap ing o apid changes in AI echnology makes me anxious.
16. Taking a p o essional de elopmen cou se on wo king alongside AI makes me anxious.
17. Lea ning how o audi AI sys ems o bias makes me anxious.
18. Figu ing ou how o in eg a e my wo k low wi h an AI assis an makes me anxious.
19. Lea ning how o cus omise an AI sys em’s beha iou s and ou pu s makes me anxious.
20. Being unable o unde s and how an AI sys em makes i s decisions makes me anxious.
JOURNAL OF PSYCHOLOGY AND AI 5
in o ma ion (mi) o i ems gene a ed by Claude was highe (mi ¼1:339) han ha o he i ems gene a ed by
humans (mi ¼0:643). Claude ends o use ewe wo ds han humans when gene a ing i ems ( isi h ps://
cu .ly/ edm74ho o de ails).
The i ems gene a ed by Claude 3 Opus can be di ec ly con ex ualised by compa ing hem o he o iginal
21-i em A i icial In elligence Anxie y Scale (AIAS) by Wang and Wang (2022), om which ou p omp
was de i ed. The o iginal AIAS is a well- alida ed ins umen wi h a con i med ou - ac o s uc u e
(Lea ning, Job Replacemen , Socio echnical Blindness, and AI Con igu a ion) and excellen eliabili y o
i s subscales ( o example, C onbach’s α was ound o be 0.964 o he whole scale and anged om 0.917 o
0.974 o he ou dimensions) (Y.-Y. Wang & Wang, 2022). Subsequen alida ion s udies in di e en
languages and popula ions ha e u he con i med i s obus psychome ic p ope ies (Te zi, 2020). Ou AI-
gene a ed i ems a e linguis ically cohe en and seman ically ela ed o he “lea ning” dimension. Howe e ,
his compa ison unde sco es ha such i ems ep esen he s a ing poin , no he end poin , o igo ous
scale de elopmen . A ull alida ion s udy would be equi ed o de e mine i hese new i ems eplica e he
es ablished ac o s uc u e and high eliabili y o he o iginal AIAS. This ein o ces he p inciple ha while
ou p oposed app oach demons a es a powe ul capaci y o apid i em pool gene a ion, i mus be
ollowed by empi ical alida ion agains es ablished benchma ks.
2.3. Six i ems p oduced o measu e adop ion and pe cep ion o AI in online lea ning
The s udy by Ma molejo-Ramos e al. (in p ess) demons a es he applica ion o he PIG p ocedu e, leading
o he de elopmen o a scale ha e alua es he inclina ion o adop and pe cei e a i icial in elligence.
Employing he PIG model, 60 i ems we e c ea ed o assess he adop ion and pe cep ion o AI in online
lea ning. The i ems we e gene a ed using a se ies o p omp s. The i s p omp asked o 10 cu en
psychome ic scales o measu e people’s pe cep ions and adop ion o (gene a i e) AI, along wi h eal
academic e e ences o each. The i ems gene a ed we e hen classi ied as ela ing o “adop ion o AI” o
“pe cep ion o AI”. Nex , a 60-ques ion su ey was c ea ed based on he i ems, wi h 30 ques ions o
“pe cep ions o AI” and 30 ques ions o “adop ion o AI”. The ques ions we e ph ased using “I” and “me”
and we e designed o be answe ed on a a ing scale om “s ongly disag ee” o “s ongly ag ee”, wi h he
scale going om 0 o 1. All ques ions we e eph ased so ha any la ge language model could unde s and
hem. These p omp s we e submi ed o Cha GPT 3.5, Claude, and Gemini, and he op ions “ egene a e
esponses”, “ e y”, and “modi y esponse” we e used o e ine he ou pu s and gene a e addi ional i ems.
Finally, wi h he goal o c ea ing a concise scale (Ramms ed & Beie lein, 2014) ha can be easily
adminis e ed online, h ee i ems we e selec ed o “adop ion o AI” and h ee o “pe cep ion o AI”
h ough expe knowledge elici a ion (see Table 2 o he selec ed i ems and Ma molejo-Ramos e al. (in
p ess) o de ails).
Mo eo e , Table 3 p esen s he speci ic models and pa ame e s used o each demons a ion. I can be
seen ha he hype -pa ame e s a e shown in de aul . This is because in adi ional web-based cha LLMs,
he e is no way o adap o modi y he hype -pa ame e s ha a ec he gene a ed ou pu . These modi ica-
ions a e no mally done ia sc ip s and APIs (e.g. he PIG me hod). Howe e , his also equi es he
in e es ed pa ies o be p og amming-sa y and modi y sou ce code, which limi s he accessibili y o he
PIG app oach. The e o e, in ou app oach, he hype -pa ame e s a e se and adap ed by he LLM i sel .
Mo eo e , o ep oducibili y he exac p omp s used o gene a e he i ems a e p o ided in supplemen a y
ma e ials.
Table 2. I ems kep o each scale a e e ining he ou pu s.
Subscale I em
Pe cep ion o IA “I am no conce ned ha AI could eplace human eache s in online educa ion.”
“I hink AI can imp o e he e iciency o online lea ning.”
“I belie e AI can imp o e he quali y o online lea ning”.
Adop ion o IA “I am open o using AI-powe ed cha bo s o online cou se suppo .”
“I am open o using AI-powe ed w i ing eedback ools in online cou ses.”
“I am willing o allow AI o collec and analyse da a abou my online lea ning beha iou .”
6F. MARMOLEJO-RAMOS ET AL.
2.3.1. Analysis o he gene a ed i ems
Employing a mul i e se analysis app oach (S eegen e al., 2016) and a c owd-sou cing da a analysis me hod
(Silbe zahn e al., 2018), six esea ch g oups, who a e also co-au ho s o his pape , we e asked wi h using
hei p e e ed psychome ic echniques o assess he quali y o he i ems. This e alua ion was based on
esponses om 1,233 pa icipan s who comple ed bo h scales in an online da a collec ion conduc ed ia
Qual ics so wa e. The da a collec ion was app o ed by he Uni e si y o Sou h Aus alia (app o al
numbe : 205473), and in o med consen was ob ained om all pa icipan s in acco dance wi h he
uni e si y’s e hical s anda ds. The indings a e summa ised in Table 4. In gene al, he esul s sugges ha
he wo scales (pe cep ion and adop ion o AI) demons a ed good in e nal consis ency (C onbach’s alpha
= 0.63 and 0.78, espec i ely) and dis inc ness om each o he . I ems we e mos ly simila in di icul y as
es ima ed by he Rasch model o a ing scales (Pe cep ion i ems 1, 2, and 3, i em di icul y pa ame e s =
−0.078, −0.291, and −0.282, espec i ely; Accep ance i ems 1, 2, and 3, i em di icul y pa ame e s = −0.597,
−0.592, and −0.274, espec i ely). Ne wo k analysis e ealed s ong connec ions be ween i ems, excep o
one in he pe cep ion sub-scale (see Figu e 2). Addi ionally, by es ima ing he Hopkins s a is ic, which
cap u es he clus e ing endency o a da ase by measu ing he p obabili y ha he da a is gene a ed by
a uni o m dis ibu ion, we ound a alue o 0.932 (be ween 0.7 and 1), indica ing clus e ed da a. This
sugges s ha he da a is likely o con ain meaning ul clus e s.
While he ini ial indings demons a e he po en ial o la ge language models like Cha GPT 3.5, Claude,
and Gemini in gene a ing eliable psychome ic i ems, he p ocess also highligh ed he challenges o
ensu ing cla i y and cohe ence in he ou pu s. This expe ience sugges s ha a mo e s uc u ed app oach
o in e ac ing wi h LLMs could u he enhance hei u ili y in esea ch se ings. By implemen ing
a p oblem-sol ing o ien ed plan and employing ailo ed p omp s, esea che s may guide hese models
mo e e ec i ely, leading o mo e consis en and meaning ul esul s. In he ollowing sec ion, we explo e
Table 4. Quali a i e summa y o mul i e se and c owdsou cing psychome ic analyses o six i ems measu ing AI adop ion
and pe cep ion in online lea ning. The da ase analysed in his s udy includes esponses om 1,233 pa icipan s o he i ems
o in e es (364 pa icipan s om Nige ia, 361 om Finland, 258 om I an, and 250 om Poland), as well as addi ional
co a ia es o in e es . Analys g oups and au ho s: g1: A. B.; g2: K. R. and J. K.; g3: L. M. and L. A.; g4: O. B.; g5: F. M-R. And
R. O.; and g6: L. A. P-U. GAMLSS: gene alised addi i e models o loca ion, scale, and shape (see (S asinopoulos e al., 2018)).
The da ase and he quan i a i e analyses ha suppo hese quali a i e summa ies can be accessed a h ps://cu .ly/
pedEKzw in R ma kdown and qua o ma kdown o ma s.
Me hods Quali a i e o e all Resul s
Rasch model as a GAMLSS jg5 O he h ee i ems in each dimension, only one was es ima ed o be mo e di icul . Howe e , he o e all
ange o i em di icul y was ela i ely homogeneous.
Ra ing scale model and Many ace
Rasch model jg6
The analysis showed a ying i em di icul y wi hin each dimension, wi h he mos challenging i ems
ela ing o conce ns abou AI eplacing human eache s and allowing AI o collec da a. Addi ionally,
indi iduals mo e in ol ed in online lea ning had a mo e open a i ude owa ds AI in bo h pe cep ion
and adop ion.
Reliabili y analysis jg4 The esul s indica ed ha he Pe cep ion o AI and Adop ion o AI we e unidimensional cons uc s,
measu ed by hei espec i e scale i ems. The e we e also s ong i em-cons uc associa ions, wi h
high ac o loadings and explained a iance.
Explo a o y Fac o Analysis jg1 The eliabili y o he AI adop ion sub-scale was good, bu he AI pe cep ion sub-scale had poo eliabili y
unless one i em was excluded.
Con i ma o y Fac o analysis jg3 The e we e s ong co ela ions among pe cep ion i ems and among adop ion i ems. Howe e , he
co ela ions be ween pe cep ion and adop ion i ems we e weake , indica ing ha hey we e sepa a e
ac o s despi e some ela ionship.
Ne wo k Analysis jg2 The ne wo k was densely connec ed, excep o one i em in he AI pe cep ion sub-scale. The s onges
edges we e obse ed be ween i ems 2 and 3 in he AI pe cep ion subscale, and be ween i ems 1 and
2 in he adop ion sub-scale.
Table 3. LLM models and pa ame e s used in demons a ions.
Pa ame e Demons a ion 1 (T us in AI) Demons a ion 2 (AI Anxie y) Demons a ion 3 (Adop ion/Pe cep ion)
LLM Used Google Gemini 1.5 Flash An h opic Claude 3 Opus Cha GPT 3.5, Claude 2.1, Gemini P o
Model Ve sion gemini-1.5- lash-001 claude-3-opus -20,240,229 gp -3.5- u bo, claude-2.1, gemini-p o
Access Da e Janua y 2024 Janua y 2024 Janua y 2024
Tempe a u e De aul De aul De aul
Top P (Nucleus Sampling) De aul De aul De aul
F equency Penal y De aul De aul De aul
P esence Penal y De aul De aul De aul
In e ac ion Me hod Web UI Web UI Web UI wi h ‘ egene a e’ op ion
JOURNAL OF PSYCHOLOGY AND AI 7
how such s a egies can op imise dialogues wi h LLMs, ul ima ely imp o ing he quali y o insigh s in he
con ex o AI adop ion and pe cep ion s udies.
3. Using p oblem-sol ing plans and ailo ed p omp s o guide chains o hough
We belie e ha s uc u ing a logical Chain-o -Though (CoT) and using ailo ed p omp s (Ps) can g ea ly
imp o e he cla i y and cohe ence o he dialogue o psychome ic i em gene a ion when aiming o
p oduc i e in e ac ion wi h an LLM. Howe e , a i s s ep is o c ea e an o ganised, p oblem-sol ing-
o ien ed plan (PSP) o p o ide op-down di ec ion and scope o he psychome ic scale be o e he s a o
a ocused CoT (L. Wang e al., 2023). These p omp enginee ing s a egies ha e e ol ed wi h he goal o
be e guiding he model in i s sea ch o he bes possible esponse. While CoT ini ially sugges ed p epa ing
mo e de ailed p omp s including examples o s ep-by-s ep easoning (Wei e al., 2022), he Ze o-sho
s a egy imp o ed upon CoT by adding “Le ’s hink s ep by s ep” (Kojima e al., 2023) be o e each answe ,
hus elimina ing he need o examples. Finally, he PSP u he enhanced he Ze o-sho app oach by
including mo e de ailed ins uc ions on how he sys em should hink s ep-by-s ep, adding: “Le ’s i s
unde s and he p oblem and de ise a plan o sol e he p oblem. Then, le ’s ca y ou he plan and sol e he
p oblem s ep by s ep” (L. Wang e al., 2023). This up on amewo k guides he subsequen s aged
de elopmen o insigh s in a mo e linea , in e connec ed low. In a psycholinguis ic con ex , p oblem-
sol ing in ol es h ee componen s: iden i ying goals, o mula ing s a egies, and moni o ing p og ess
(Ma molejo-Ramos & Ce asco, 2014). In he case o a PS-o ien ed plan, we belie e ha hese h ee
componen s can be expanded o include componen s such as goal se ing (i.e. clea ly de ining he pu pose,
0.09
0.10
0.12
0.14
0.15
0.15
0.19
0.22
0.33
0.68
P1
P2
P3
A1
A2
A3
Figu e 2. Es ima ed Gaussian g aphical model o he AI pe cep ion and adop ion scale. Doughnu cha s ep esen
p edic abili y, wi h a ully illed ing indica ing ha 100% o an i em’s a iance is explained by i s connec ions wi h o he
i ems. P1 = “I am no conce ned ha AI could eplace human eache s in online educa ion.” P2 = “I hink AI can imp o e he
e iciency o online lea ning.” P3 = “I belie e AI can imp o e he quali y o online lea ning”. A1 = “I am open o using AI-
powe ed cha bo s o online cou se suppo .” A2 = “I am open o using AI-powe ed w i ing eedback ools in online
cou ses.” A3 = “I am willing o allow AI o collec and analyse da a abou my online lea ning beha iou .”.
8F. MARMOLEJO-RAMOS ET AL.
equi alen ly ac oss di e en demog aphic g oups. This s ep is c ucial o iden i ying and ec i ying
any la en biases ha we e no caugh du ing he ini ial e iew.
This sys ema ic app oach ensu es ha e hical conside a ions a e no an a e hough bu a e wo en in o he
ab ic o he AI-assis ed scale de elopmen p ocess, aligning wi h in e na ional guidelines ha call o
ai ness, anspa ency, and accoun abili y in AI sys ems.
5. Conclusion
In his s udy, we in oduced and demons a ed a no el, code- ee e sion o he PIG me hod, which
le e ages he con e sa ional powe o mode n LLMs o au oma e he de elopmen o psychological scales.
Ou cen al con ibu ion is he p esen a ion o a sys ema ic amewo k, g ounded in a PSP and CoT
p omp ing, ha enhances he accessibili y, e iciency, and psychome ic soundness o AI-assis ed i em
gene a ion. Ou indings ac oss h ee demons a ions show ha his app oach can p oduce i ems wi h
s ong linguis ic and psychome ic p ope ies, compa able o e en supe io o human-gene a ed coun e -
pa s, o cons uc s such as AI anxie y and us in AI. The mul i e se analysis o ou 6-i em scale o AI
adop ion and pe cep ion u he alida ed he me hod, e ealing good in e nal consis ency and a clea
ac o ial s uc u e ha was obus o di e en analy ical app oaches. Howe e , we u ge ha his po en ial be
iewed wi h cau ious op imism. Ou wo k also highligh s signi ican limi a ions and e hical impe a i es ha
mus be add essed. The challenges o LLM hallucina ions, p omp sensi i i y, and inhe en algo i hmic bias
necessi a e ha his echnology be used as a powe ul assis an o, no a eplacemen o , human expe ise.
Rigo ous con en alida ion by subjec ma e expe s, p oac i e mi iga ion o esponse biases h ough
ca e ul i em design, and igilan e hical o e sigh a e non-nego iable componen s o his p ocess. Fu u e
esea ch should ocus on se e al key a eas. Fi s , a sys ema ic in es iga ion is needed o de e mine op imal
p omp ing s a egies o gene a ing di e se i em ypes, including hose ha a e e e se-coded o ha assess
di e en le els o cogni i e complexi y. Second, he de elopmen o au oma ed, AI-d i en pipelines o
alida ing i ems, such as using LLMs o lag po en ial biases o assess con en alidi y agains a de ined
seman ic space, ep esen s a p omising on ie . Finally, longi udinal s udies a e equi ed o assess he long-
e m s abili y and p edic i e alidi y o scales de eloped using hese me hods. By in eg a ing he powe o
LLMs wi h he igou o adi ional psychome ics, we can signi ican ly ad ance he science o psychological
measu emen , making i mo e e icien , inno a i e, and accessible o he b oade esea ch communi y.
Disclosu e s a emen
No po en ial con lic o in e es was epo ed by he au ho (s).
Au ho s’ con ibu ions
Concep ualisa ion: FM-R; Me hodology: FM-R and JT; Fo mal analysis: FM-R, OB, LA, LM, AB, KR JC, LAPU, RO,
and JT; In es iga ion: all au ho s; Resou ces: all au ho s; Da a cu a ion: FM-R, OB, LA, LM, AB, KR JC, LAPU, RO, and
JT; W i ing – O iginal D a : FM-R and JT; W i ing – Re iew Edi ing: all au ho s; Visualisa ion: FM-R, KR and JT;
Supe ision: FM-R; P ojec adminis a ion: FM-R.
Funding
R.O. g a e ully acknowledges he pa ial inancial suppo ecei ed om he B azilian agencies Conselho Nacional de
Desen ol imen o Cien í ico e Tecnológico (CNPq) [G an s 303192/2022-4 and 402519/2023-0] and Fundaçáo de
Ampa o á Ciência e Tecnologia do Es ado da Bahia (FAPESB) [G an APP0021/2023] o his esea ch. J. K. and
K. R. wo k was suppo ed by Ope a ional P og am Jan Ámos Komenský p ojec ‘Biog aphy o Fake News wi h a Touch
o AI: Dange ous Phenomenon h ough he P ism o Mode n Human Sciences’ ( eg. CZ.02.01.01/00/23_025/0008724).
ORCID
Fe nando Ma molejo-Ramos h p://o cid.o g/0000-0003-4680-1287
JOURNAL OF PSYCHOLOGY AND AI 15
A ailabili y o da a and ma e ials
A ailable a h ps://cu .ly/pedEKzw and h ps://cu .ly/ edm74ho
Code a ailabili y
A ailable a h ps://cu .ly/pedEKzw and h ps://cu .ly/ edm74ho
Re e ences
Ahmed, T., & De anbu, P. (2023). Be e pa ching using LLM p omp ing, ia sel -consis ency. In 2023 38 h IEEE/ACM
In e na ional Con e ence on Au oma ed So wa e Enginee ing (ASE) (pp. 1742–1746). IEEE.
Ajwani, R., Ja aji, S. R., Rudzicz, F., & Zhu, Z. (2024). Llm-gene a ed black-box explana ions can be ad e sa ially
help ul. a Xi .a Xi :2405.06800 [CS] (h ps://doi.o g/10.48550/a Xi .2405.06800
Ami ianiani, S., Zhou, Y., Su esh, H., Schwe mann, S., & Ghassemi, M. (2024). Llmaudi o : A amewo k o audi ing
la ge language models. a Xi P ep in a Xi , 2402.09346.
Associa ion, A. E. R. (2014). S anda ds o educa ional and psychological es ing. Ame ican Educa ional Resea ch
Associa ion.
Ba on, E. N. (2024). Game heo y: An in oduc ion. John Wiley & Sons.
Be ea, E., & Zai , A. (2013). Scale alidi y in explo a o y s ages o esea ch. Managemen & Ma ke ing Jou nal, (1),
38–46.
Boa eng, G. O., Neilands, T. B., F ongillo, E. A., Melga -Quiñonez, H. R., & Young, S. L. (2018). Bes p ac ices o
de eloping and alida ing scales o heal h, social, and beha io al esea ch: A p ime . F on ie s in Public Heal h, 6,
149. h ps://doi.o g/10.3389/ pubh.2018.00149 .
Bos om, N., & Yudkowsky, E. (2018). The e hics o a i icial in elligence. In Roman V. Yampolskiy (Ed.), A i icial
in elligence sa e y and secu i y (pp. 57–69). Chapman and Hall/CRC.
Chu, Z., Chen, J., Chen, Q., Yu, W., He, T., Wang, H., Peng, W., Liu, M., Qin, B., & Liu, T. (2023). A su ey o chain o
hough easoning: Ad ances, on ie s and u u e. h ps://doi.o g/10.48550/a Xi .2309.15402
Cla k, L. A., & Wa son, D. (2019). Cons uc ing alidi y: New de elopmen s in c ea ing objec i e measu ing
ins umen s. Psychological Assessmen , 31(12), 1412–1427. h ps://doi.o g/10.1037/pas0000626
Coni ze , V., Sinno -A ms ong, W., Schaich Bo g, J., Deng, Y., & K ame , M. (2017). Mo al decision making
amewo ks o a i icial in elligence. P oceedings o he AAAI Con e ence on A i icial In elligence (Vol. 31(1)).
h ps://doi.o g/10.1609/aaai. 31i1.11140
Da is, L. L. (1992). Ins umen e iew: Ge ing he mos om a panel o expe s. Applied Nu sing Resea ch, 5(4),
194–197. h ps://doi.o g/10.1016/S0897-1897(05)80008-4 .
Da is, R. E., Lee, S., Johnson, T. P., Con ad, F., Resnicow, K., Th ashe , J. F., Mesa, A., & Pe e son, K. E. (2020). The
in luence o i em cha ac e is ics on acquiescence among La ino su ey esponden s. Field Me hods, 32(1), 3–22.
h ps://doi.o g/10.1177/1525822x19873272 .
Ding, N., Qin, Y., Yang, G., Wei, F., Yang, Z., Su, Y., Hu, S., Chen, Y., Chan, C.-M., Chen, W., Yi, J., Zhao, W.,
Wang, X., Liu, Z., Zheng, H.-T., Chen, J., Liu, Y., Tang, J., Li, J., & Sun, M. (2023). Pa ame e -e icien ine- uning o
la ge-scale p e- ained language models. Na u e Machine In elligence, 5(3), 220–235. h ps://doi.o g/10.1038/s42256-
023-00626-4
Dumas, D., G ei , S., & We zel, E. (2025). Ten guidelines o sco ing psychological assessmen s using a i icial
in elligence. Eu opean Jou nal o Psychological Assessmen , 41(3), 169–173. h ps://doi.o g/10.1027/1015-5759/
a000904
Elson, M., Hussey, I., Alsal i, T., & A slan, R. (2023). Psychological measu es a en’ oo hb ushes. Communica ions
Psychology, 1(25). h ps://doi.o g/10.1038/s44271-023-00026-9
E zioni, A., & E zioni, O. (2017). Inco po a ing e hics in o a i icial in elligence. The Jou nal o E hics, 21(4), 403–418.
h ps://doi.o g/10.1007/s10892-017-9252-2
Fa me , R. L., Lockwood, A. B., Go o h, A., & Thomas, C. (2025). A i icial in elligence in p ac ice: Oppo uni ies,
challenges, and e hical conside a ions. P o essional Psychology, Resea ch and P ac ice, 56(1), 19–32. h ps://doi.o g/
10.1037/p o0000595
Floyd, M. W., Ka neeb, J., & Aha, D. W. (2017, June 26–28). Case-based eam ecogni ion using lea ned opponen
models. In Da id W. Aha, & Jean Liebe (Eds.), Case-Based Reasoning Resea ch and De elopmen : 25 h In e na ional
Con e ence, ICCBR 2017 (Vol. 25., pp. 123–138). Sp inge .
F eye , O., Wies , I. C., Ka he , J. N., & Gilbe , S. (2024). A u u e ole o heal h applica ions o la ge language models
depends on egula o s en o cing sa e y s anda ds. Lance Digi al Heal h, 6(9), 662–672. h ps://doi.o g/10.1016/
S2589-7500(24)00124-9
Geisslinge , M., Poszle , F., Be z, J., Lü ge, C., & Lienkamp, M. (2021). Au onomous d i ing e hics: F om olley
p oblem o e hics o isk. Philosophy & Technology, 34(4), 1033–1055. h ps://doi.o g/10.1007/s13347-021-00449-4
16 F. MARMOLEJO-RAMOS ET AL.
Gi ay, L. (2023). P omp enginee ing wi h Cha GPT: A guide o academic w i e s. Annals o Biomedical Enginee ing,
51(12), 2629–2633. h ps://doi.o g/10.1007/s10439-023-03272-4
Gö z, F. M., Mae ens, R., Loomba, S., & Linden, S. (2023). Le he algo i hm speak: How o use neu al ne wo ks o
au oma ic i em gene a ion in psychological scale de elopmen . Psychological Me hods, 29(3), 494–518. h ps://doi.
o g/10.1037/me 0000540
Gula i, S., Sousa, S., & Lamas, D. (2019). Design, de elopmen and e alua ion o a human-compu e us scale.
Beha iou & In o ma ion Technology, 38(10), 1004–1015. h ps://doi.o g/10.1080/0144929X.2019.1656779
Hen ickson, L., & Me oño-Peñuela, A. (2025). P omp ing meaning: A he meneu ic app oach o op imising p omp
enginee ing wi h Cha GPT. AI & Socie y, 40(2), 903–918. h ps://doi.o g/10.1007/s00146-023-01752-8
Hu nyan, M., & Go lieb, M. C. (2025). A i icial in elligence in psychological p ac ice: Applica ions, e hical conside a-
ions, and ecommenda ions. In P o essional psychology: Resea ch and p ac ice. Ad ance online publica ion. h ps://
doi.o g/10.1037/p o0000631
I an, D., & Tang, X. (2025). E alua ing China’s elec ic ehicle adop ion wi h PESTLE: S akeholde pe spec i es on
sus ainabili y and adop ion ba ie s. Sus ainabili y, 17(14), 6258. h ps://doi.o g/10.3390/su17146258
Jebb, A. T., Ng, V., & Tay, L. (2021). A e iew o key Like scale de elopmen ad ances: 1995–2019. F on ie s in
Psychology, 12. h ps://doi.o g/10.3389/ psyg.2021.637547
Jin, Z., Kleiman-Weine , M., Pia i, G., Le ine, S., Liu, J., Gonzalez, F., O u, F., S ausz, A., Sachan, M., Mihalcea, R.,
Choi, Y., & Schölkop , B. (2024). Language model alignmen in mul ilingual olley p oblems. h ps://a xi .o g/
abs/2407.02273
Jobin, A., Ienca, M., & Vayena, E. (2019). The global landscape o AI e hics guidelines. Na u e Machine In elligence, 1
(9), 389–399. h ps://doi.o g/10.1038/s42256-019-0088-2
Kale, A., Nguyen, T., Ha is, F. C., Li, C., Zhang, J., & Ma, X. (2023). P o enance documen a ion o enable explainable
and us wo hy AI: A li e a u e e iew. Da a In elligence, 5(1), 139–162. h ps://doi.o g/10.1162/din _a_00119
Kama a, A., & Vaughn, B. K. (2004). An in oduc ion o di e en ial i em unc ioning analysis. Lea ning Disabili ies
a Con empo a y Jou nal, 2(2), 49–69.
Kojima, T., Gu, S. S., Reid, M., Ma suo, Y., & Iwasawa, Y. (2023). La ge language models a e ze o-sho easone s. A Xi .
a Xi :2205.11916 [CS] (h ps://doi.o g/10.48550/a Xi .2205.11916
Kuma , C. V., U lana, A., Kanumolu, G., Ga lapa i, B. M., & Mish a, P. (2025). No LLM is ee om bias:
A comp ehensi e s udy o bias e alua ion in la ge language models. a Xi . a Xi :2503.11985 [cs] (h ps://doi.o g/
10.48550/a Xi .2503.11985
Lawshe, C. H. (1975). A quan i a i e app oach o con en alidi y 1. Pe sonnel Psychology, 28(4), 563–575. h ps://doi.
o g/10.1111/j.1744-6570.1975. b01393.x
Lo è, N., & Heyda i, B. (2024). S a egic beha io o la ge language models and he ole o game s uc u e e sus
con ex ual aming. Scien i ic Repo s, 14(1), 18490. h ps://doi.o g/10.1038/s41598-024-69032-z
Lynn, M. R. (1986). Accessed 2025-07-15 De e mina ion and quan i ica ion o con en alidi y. Nu sing Resea ch, 35
(6), 382–385. h ps://doi.o g/10.1097/00006199-198611000-00017
Ma molejo-Ramos, F., Abadia, R., Ka akale, O., Ba e a-Causil, C., Männikkö, N., Gho bani, B., S zelecki, A.,
Nwizu, S., Ta a es, C., Cas illo, M., Som, B., Ngo, G., Özdoğ u, A., A iyabuddhiphongs, K., & Tejada J. (in p ess).
Pe cep ions and adop ions o a i icial in elligence in online lea ning. Disco e A i icial In elligence.
Ma molejo-Ramos, F., & Ce asco, J. (2014). Tex comp ehension as a p oblem sol ing si ua ion. Uni e si as
Psychologica, 13(2), 725–743. h ps://doi.o g/10.11144/Ja e iana.UPSY13-2. cps
McG a h, M. J., Lack, O., Tisch, J., & Duense , A. (2025). Measu ing us in a i icial in elligence: Valida ion o an
es ablished scale and i s sho o m. F on ie s in A i icial In elligence, 8. h ps://doi.o g/10.3389/ ai.2025.1582880
Mo, S., Salakhu dino , R., Mo ency, L.-P., & Liang, P. P. IoT-lm: La ge mul isenso y language models o he In e ne
o Things. A Xi P ep in A Xi :2407.09801 (2024.) h ps://doi.o g/10.48550/a Xi .2407.09801
Oeljeklaus, L., Hö , S., & Danne , D. (2025). Compa ing psychome ic p ope ies o expe -de eloped and
AI-gene a ed pe sonali y scales. Psychological Tes Adap a ion and De elopmen , 6, 29–43. h ps://doi.o g/10.1027/
2698-1866/a000095
O’Hagan, A. (2019). Expe knowledge elici a ion: Subjec i e bu scien i ic. The Ame ican S a is ician. Ame ican
S a is ician, 73(sup1), 69–81. h ps://doi.o g/10.1080/00031305.2018.1518265
Pelle , M., Lechne , C. M., Wagne , C., Ramms ed , B., & S ohmaie , M. (2024). AI psychome ics: Assessing he
psychological p o iles o la ge language models h ough psychome ic in en o ies. Pe spec i es on Psychological
Science, 19(5), 808–826. h ps://doi.o g/10.1177/17456916231214460
P imi, R., Hauck-Filho, N., Valen ini, F., & San os, D. (2020). Classical pe spec i es o con olling acquiescence wi h
balanced scales. In M. Wibe g, D. Molenaa , J. González, U. Böckenhol & J.-S. Kim (Eds.), Quan i a i e
psychology (pp. 333–345). Sp inge . h ps://doi.o g/10.1007/978-3-030-43469-4_25
Ramms ed , B., & Beie lein, C. (2014). Can’ we make i any sho e ? Jou nal o Indi idual Di e ences, 35(4), 212–220.
h ps://doi.o g/10.1027/1614-0001/a000141
Rape , R. (2024). A su ey o machine e hics. In Raising obo s o be good: A p ac ical o ay in o he a and science o
machine e hics (pp. 19–33). Sp inge . h ps://link.sp inge .com/book/10.1007/978-3-031-75036-6
Raza i, A., Sol angheis, M., A abzadeh, N., Salama , S., Zihaya , M., & Baghe i, E. (2025). Benchma king p omp
sensi i i y in la ge language models. In C. Hau , C. Macdonald, D. Jannach, G. Kazai, F. M. Na dini, F. Pinelli,
JOURNAL OF PSYCHOLOGY AND AI 17
F. Sil es i & N. Tonello o (Eds.), Ad ances in in o ma ion e ie al (pp. 303–313). Sp inge . h ps://doi.o g/10.
1007/978-3-031-88714-7_29
Rome o Jeld es, M., Díaz Cos a, E., & Faouzi Nadim, T. (2023). A e iew o Lawshe’s me hod o calcula ing con en
alidi y in he social sciences. F on ie s in Educa ion, 8. h ps://doi.o g/10.3389/ educ.2023.1271335 .
Sallam, M. (2023). Cha GPT u ili y in heal hca e educa ion, esea ch, and p ac ice: Sys ema ic e iew on he p omising
pe spec i es and alid conce ns. Heal hca e, 11(6), 887. h ps://doi.o g/10.3390/heal hca e11060887
Sen, A., Mainali, M., Rauch, C. B., Addison, U., Floyd, M. W., Goel, P., Ka neeb, J., Kulhanek, R., La ue, O., Ménage ,
D., Molineaux, M., Tu ne , J. T., & Webe , R. O. (2024). Coun e ac ual-based syn he ic case gene a ion. In
In e na ional Con e ence on Case-BasedReasoning (pp. 388–403). Sp inge .
Silbe zahn, R., Uhlmann, E. L., Ma in, D. P., Anselmi, P., Aus , F., Aw ey, E., Bahník, S., Bai, F., Banna d, C.,
Bonnie , E., Ca lsson, R., Cheung, F., Ch is ensen, G., Clay, R., C aig, M. A., Rosa, A. D., Dam, L., E ans, M. H.
Ce an es, I. F., . . . Nosek, B. A. (2018). Many analys s, one da a se : Making anspa en how a ia ions in analy ic
choices a ec esul s. Ad ances in Me hods and P ac ices in Psychological Science, 1(3), 337–356. h ps://doi.o g/10.
1177/2515245917747646
Sil a, A. (2024). La ge language models playing mixed s a egy Nash equilib ium games. a Xi p ep in a Xi :2406.10574.
Singh, R., & Singh, S. (2021). Tex simila i y measu es in news a icles by ec o space model using NLP. Jou nal o he
Ins i u ion o Enginee s (India): Se ies B, 102(2), 329–338. h ps://doi.o g/10.1007/s40031-020-00501-5. Accessed
2025-04-03.
S ahl, B. C. (2021). E hical issues o AI. In Sp inge (Ed.), A i icial in elligence o a be e u u e: An ecosys em
pe spec i e on he e hics o AI and eme ging digi al echnologies (pp. 35–53). h ps://link.sp inge .com/book/10.1007/
978-3-030-69978-9
S asinopoulos, M. D., Rigby, R. A., & Bas iani, F. D. (2018). GAMLSS: A dis ibu ional eg ession app oach. S a is ical
Modelling, 18(3–4), 248–273. h ps://doi.o g/10.1177/1471082X18759144
S eegen, S., Tue linckx, F., Gelman, A., & Vanpaemel, W. (2016). Inc easing anspa ency h ough a mul i e se
analysis. Pe spec i es on Psychological Science, 11(5), 702–712. h ps://doi.o g/10.1177/1745691616658637. PMID:
27694465.
Takemo o, K. (2024). The mo al machine expe imen on la ge language models. Royal Socie y Open Science, 11(2),
231393. h ps://doi.o g/10.1098/ sos.231393
Tan, B., A moush, N., Mazzullo, E., Bulu , O., & Gie l, M. (2025). A e iew o au oma ic i em gene a ion echniques
le e aging la ge language models. In e na ional Jou nal o Assessmen Tools in Educa ion, 12(2), 317–340. h ps://doi.
o g/10.21449/ija e.1602294 .
Te zi, R. (2020). An adap a ion o a i icial in elligence anxie y scale in o Tu kish: Reliabili y and alidi y s udy.
In e na ional Online Jou nal o Educa ion and Teaching, 7(4), 1501–1515. h p://ioje .o g/index.php/IOJET/a icle/
iew/103 .
UNESCO. (2022). Unesco: Recommenda ion on he e hics o a i icial in elligence. UNESCO. h ps://unesdoc.unesco.
o g/a k:/48223/p 0000380455. Accessed 2025-07-15
Wang, L., Xu, W., Lan, Y., Hu, Z., Lan, Y., Lee, R. K.-W., & Lim, E.-P. (2023). Plan-and-sol e p omp ing: Imp o ing
ze o-sho chain-o - hough easoning by la ge language models. a Xi . a Xi :2305.04091 [CS] (h ps://doi.o g/10.
48550/a Xi .2305.04091 .
Wang, Y., Su, Z., Guo, S., Dai, M., Luan, T. H., & Liu, Y. (2023). A su ey on digi al wins: A chi ec u e, enabling
echnologies, secu i y and p i acy, and u u e p ospec s. IEEE In e ne o Things Jou nal, 10(17), 14965–14987.
h ps://doi.o g/10.1109/JIOT.2023.3263909
Wang, Y.-Y., & Wang, Y.-S. (2022). De elopmen and alida ion o an a i icial in elligence anxie y scale: An ini ial
applica ion in p edic ing mo i a ed lea ning beha io . In e ac i e Lea ning En i onmen s, 30(4), 619–634. h ps://
doi.o g/10.1080/10494820.2019.1674887
Wang, Y., Zhao, S., Wang, Z., Huang, H., Fan, M., Zhang, Y., Wang, Z., Wang, H., & Liu, T. (2024). S a egic chain-o -
hough : Guiding accu a e easoning in LLMs h ough s a egy elici a ion. a Xi . a Xi :2409.03271 [cs] (h ps://doi.
o g/10.48550/a Xi .2409.03271
Wei, J., Wang, X., Schuu mans, D., Bosma, M., Ich e , B., Xia, F., Chi, E. H., Le, Q. V., & Zhou, D. (2022). Chain-o -
hough p omp ing elici s easoning in la ge language models. h ps://a xi .o g/abs/2201.11903
Weij e s, B., Baumga ne , H., & Schillewae , N. (2013). Re e sed i em bias: An in eg a i e model. Psychological
Me hods. Psychological Me hods, 18(3), 320–334. h ps://doi.o g/10.1037/a0032121 .
Zeeshan, M., Iqbal, M., Shamim-U -Rasul, S., Sami, F., Malik, G. M., Imdad, D., & She azi, G. Z. (2024). A compa a i e
analysis o psychome ic p ope ies in AI-gene a ed and eache -made MCQs es . Ku dish S udies, 12(4),
1808–1820. h ps://doi.o g/10.53555/ks. 12i4.3653
Zhang, S., Bao, Y., & Huang, S. (2024). Imp o ing la ge language models’ gene a ion by en opy-based dynamic
empe a u e sampling. h ps://a xi .o g/abs/2403.14541
18 F. MARMOLEJO-RAMOS ET AL.
