Heikkilä, Jussi T. S.
A icle — Published Ve sion
Human in elligence e sus a i icial in elligence in
classi ying economics esea ch a icles: explo a o y
e idence
Jou nal o Documen a ion
Sugges ed Ci a ion: Heikkilä, Jussi T. S. (2024) : Human in elligence e sus a i icial in elligence in
classi ying economics esea ch a icles: explo a o y e idence, Jou nal o Documen a ion, ISSN
1758-7379, Eme ald, Bingley, Vol. 81, Iss. 7, pp. 18-30,
h ps://doi.o g/10.1108/JD-05-2024-0104
This Ve sion is a ailable a :
h ps://hdl.handle.ne /10419/307995
S anda d-Nu zungsbedingungen:
Die Dokumen e au EconS o dü en zu eigenen wissenscha lichen
Zwecken und zum P i a geb auch gespeiche und kopie we den.
Sie dü en die Dokumen e nich ü ö en liche ode komme zielle
Zwecke e iel äl igen, ö en lich auss ellen, ö en lich zugänglich
machen, e eiben ode ande wei ig nu zen.
So e n die Ve asse die Dokumen e un e Open-Con en -Lizenzen
(insbesonde e CC-Lizenzen) zu Ve ügung ges ell haben soll en,
gel en abweichend on diesen Nu zungsbedingungen die in de do
genann en Lizenz gewäh en Nu zungs ech e.
Te ms o use:
Documen s in EconS o may be sa ed and copied o you pe sonal
and schola ly pu poses.
You a e no o copy documen s o public o comme cial pu poses, o
exhibi he documen s publicly, o make hem publicly a ailable on he
in e ne , o o dis ibu e o o he wise use he documen s in public.
I he documen s ha e been made a ailable unde an Open Con en
Licence (especially C ea i e Commons Licences), you may exe cise
u he usage igh s as speci ied in he indica ed licence.
h ps://c ea i ecommons.o g/licenses/by/4.0/legalcode
Human in elligence e sus a i icial
in elligence in classi ying economics
esea ch a icles: explo a o y e idence
Jussi T.S. Heikkil€
a
LUT Uni e si y, Lah i Campus, Lah i, Finland and
Jy €
askyl€
aUni e si y School o Business and Economics, Jy €
askyl€
a, Finland
Abs ac
Pu pose –We compa e human in elligence o a i icial in elligence (AI) in he choice o app op ia e Jou nal o
Economic Li e a u e (JEL) codes o esea ch pape s in economics.
Design/me hodology/app oach –We compa e he JEL code choices ela ed o a icles published in he ecen
issues o he Jou nal o Economic Li e a u e and he Ame ican Economic Re iew and compa e hese o he o iginal
JEL code choices o he au ho s in ea lie wo king pape e sions and JEL codes ecommended by a ious gene a i e
AI sys ems (OpenAI’s Cha GPT, Mic oso ’s Copilo , Google’s Gemini) based on he abs ac s o he a icles.
Findings –The e a e signi ican disc epancies and o en limi ed o e lap be ween au ho s’ choices o JEL codes,
edi o s’ choices as well as he choices by con empo a y widely used AI sys ems. Howe e , he obse a ions
sugges ha gene a i e AI can augmen human in elligence in he mic o- ask o choosing he JEL codes and,
hus, sa e esea che s ime.
Resea ch limi a ions/implica ions –Rapid de elopmen o AI sys ems makes he indings quickly obsole e.
P ac ical implica ions –AI sys ems may economize on classi ica ion cos s and (semi-)au oma e he choice o
JEL codes by ecommending he mos app op ia e ones. Fu u e s udies may apply he p esen ed app oach o
analyze whe he he JEL code choices be ween au ho s, edi o s and AI sys ems con e ge and become mo e
consis en as humans inc easingly in e ac wi h AI sys ems.
O iginali y/ alue –We assume ha he choice o JEL codes is a mic o- ask in which boundedly a ional
decision-make s a he sa is ice han op imize. This explo a o y expe imen is among he i s o compa e
human in elligence and gene a i e AI in choosing and jus i ying he choice o op imal JEL codes.
Keywo ds JEL codes, A i icial in elligence, La ge language models, Sea ch cos s, Bounded a ionali y
Pape ype Resea ch pape
In oduc ion
The Jou nal o Economic Li e a u e (JEL) classi ica ion codes sys em main ained and
published by he Ame ican Economic Associa ion is he de ac o s anda d o classi ying
esea ch pape s in economics (Che ie , 2017;Heikkil€
a, 2021,2022;Bo nmann and
Wohl abe, 2024). Da a on JEL classi ica ion codes has been u ilized o analyze he e olu ion o
published economics pape s by ields and s yles (e.g. Ca d and DellaVigna, 2013;Ang is
e al., 2017;Bo nmann and Wohl abe, 2024). Kosnik (2018) has documen ed ha he e can be
di e ences in he au ho -assigned and edi o -assigned JEL codes (in he Ame ican Economic
Re iew jou nal) and Heikkil€
a(2022) u he illus a ed ha JEL codes o economics wo king
pape s can di e om hose o he inal pee - e iewed and published a icles. Concu en ly,
he mic o ask o classi ying economics esea ch pape s has become inc easingly complex. Fo
ins ance, Bo nmann and Wohl abe (2024) epo ha he a e age numbe o JEL codes pe
pape has inc eased s eadily om abou 1.9 in 1991 o 4.3 in 2021.
Using “human in elligence” may lead o subjec i e and boundedly a ional choices
(A inge e al., 2022) and di e en esea che s choose di e en JEL codes ha in hei
JD
81,7
18
© Jussi T.S. Heikkil€
a. Published by Eme ald Publishing Limi ed. This a icle is published unde he
C ea i e Commons A ibu ion (CC BY 4.0) licence. Anyone may ep oduce, dis ibu e, ansla e and
c ea e de i a i e wo ks o his a icle ( o bo h comme cial and non-comme cial pu poses), subjec o ull
a ibu ion o he o iginal publica ion and au ho s. The ull e ms o his licence may be seen a h p://
c ea i ecommons.o g/licences/by/4.0/legalcode
I hank wo anonymous e iewe s o hei help ul commen s. Financial suppo om he P€
aij€
a -H€
ame
Regional Fund o he Finnish Cul u al Founda ion is g a e ully acknowledged.
The cu en issue and ull ex a chi e o his jou nal is a ailable on Eme ald Insigh a :
h ps://www.eme ald.com/insigh /0022-0418.h m
Recei ed 8 May 2024
Re ised 20 Oc obe 2024
Accep ed 24 Oc obe 2024
Jou nal o Documen a ion
Vol. 81 No. 7, 2025
pp. 18-30
Eme ald Publishing Limi ed
e-ISSN: 1758-7379
p-ISSN: 0022-0418
DOI 10.1108/JD-05-2024-0104
subjec i e opinion gi en hei e ol ing belie s and expec a ions a e he mos app op ia e
ones. I is an empi ical ques ion, o wha ex en esea che s in eali y amilia ize hemsel es
wi h he JEL Classi ica ion Codes guide when choosing JEL codes (Heikkil€
a, 2022) o
hei a icles and how much he e is a ional ina en ion (Ma�
ckowiak e al., 2023). The guide
is a ailable online [1] and i has qui e de ailed ins uc ions o he use o speci ic JEL codes
so ha i equi es a cos ly and ime-consuming e o o lea n and ollow he guide’s
ecommenda ions. P esumably, he e is a non-negligible amoun o a ional ina en ion
a ound he use o JEL codes: esea che s may hink ha he expec ed bene i s o choosing
app op ia e JEL codes may no exceed he cos s o lea ning he de ailed ins uc ions
p o ided by he JEL codes guide.
I is impo an o keep in mind ha o iginally, JEL codes classi ica ion sys em and i s
p edecesso s we e de eloped o dec ease sea ch cos s in he “pape e a” in he 20 h cen u y
(Che ie , 2017), bu now ha we li e in he e a o digi ized in o ma ion, esea che s can easily
conduc hei sea ches using online sea ch engines ha enable keywo d sea ches om ull ex s
o a icles which has emendously dec eased sea ch cos s (c . Gold a b and Tucke , 2019). As
in ecen yea s, we ha e seen signi ican p og ess in he ield o la ge language models (LLM)
[2] and gene a i e AI [3], we a e expec ing an inc easing numbe o asks o be au oma ed
(B ynjol sson e al., 2023;Eloundou e al., 2024). P esumably, sea ch cos s will u he
dec ease and ime sa ings ela ed o li e a u e e iews inc ease as con inuously imp o ing AI
sys ems can e iew and p ocess whole da abases o esea ch pape s. Recen ly, Ko inek (2023)
a gued ha “economis s can eap signi ican p oduc i i y gains by aking ad an age o
gene a i e AI o au oma e mic o- asks” and i seems ha choosing JEL codes o esea ch
pape s is one such ask. Thus, he simple esea ch ques ion ha is explo ed in his pape is he
ollowing: Can gene a i e AI sys ems augmen human in elligence in he choice o app op ia e
JEL codes? The answe is posi i e and his is demons a ed wi h a simple expe imen nex .
Human e sus a i icial in elligence in he choice o JEL codes: an expe imen
Do esea che s s udy AEA’s JEL codes guide when submission sys ems o jou nals equi e hem
o assign app op ia e JEL codes o hei a icles? P esumably, in his mic o ask, many
esea che s do no op imize bu a he sea ch JEL codes un il hey a e sa is ied (mee ing
sa is icing aspi a ion le el, c . Simon, 1955;A inge e al., 2022) wi h hei “good enough” JEL
code choices and s op sea ching (c . Caplin e al., 2011). Due o echnological p og ess, AI can
nowadays be u ilized ei he o eplace human choice o JEL codes al oge he o complemen and
augmen human in elligence in such a classi ica ion ask as illus a ed in Figu e 1. On he one
hand, AI sys ems can base hei choices on a huge and accumula ing amoun o aining da a
which p ocessing is beyond human capaci y. On he o he hand, AI sys ems may no be able o
adap o changes (e.g. new classi ica ion codes) in he JEL codes classi ica ion sys em as lexibly
as humans, o ins ance when no exis ing a icles in he aining da a a e associa ed wi h no el
JEL codes. As bo h human in elligence and a i icial in elligence ha e hei p os and cons,
augmen ed in elligence migh be he p e e ed op ion – a leas o now.
Howe e , he e is no ag eed concep o “op imal choice o JEL codes” – as he e is no clea
and consis en guidelines o he choice o keywo ds o a icles (c . Lu e al., 2020). I is no
un easonable o assume ha he e is no common knowledge o JEL code choice c i e ia.
The e o e, we in en ionally do no de ine he concep “op imal” (o app op ia e) he e - ha is,
wha and whose p e e ences ough o de ine wha should be op imized in he choice o JEL
codes. Howe e , we p omp ed selec ed AI sys ems (OpenAI’s Cha GPT, Mic oso ’s Copilo ,
Google’s Gemini) o explain wha hey conside o be impo an using he ollowing p omp :
Wha should esea che s conside when choosing he op imal JEL codes o hei esea ch a icles?
Wha should hey op imize?
Table A1 in he Appendix summa izes he answe s o AI sys ems. To summa ize, hey gene ally
no e he goal o he JEL code choice o be o maximize isibili y, disco e abili y and impac
Jou nal o
Documen a ion
19
(Cha GPT 3.5 and 4 e e explici ly o ci a ions) among app op ia e and in ended academic
audiences. All selec ed AI sys ems lis ele ance and accu acy - JEL codes should accu a ely
e lec he con en ( opics, ields, hemes, ocus, scope, me hodologies) o he a icles - as key
ac o s o conside . Speci ici y is also lis ed, bu i s de ini ion is sligh ly ambiguous. Cha GPT 4
highligh s ha choosing speci ic JEL codes “can inc ease he isibili y o he a icle among
esea che s who a e wo king on he same niche” while Gemini ecommends o “a oid o e ly
gene al seconda y codes”. Copilo and Gemini ecommend consul ing AEA’s JEL codes guide.
While Copilo ad ises no o o e use JEL codes, a bi su p isingly Gemini ecommends limi ing
he choice o a maximum o wo JEL codes (howe e , i ecommends mo e codes i sel as
demons a ed in Table 1a and 2a below). Cha GPT 3.5 conside s consis ency wi h he JEL
codes used in he exis ing li e a u e and main aining cohe ence wi hin he academic discou se
impo an and Gemini ecommends conside ing jou nal audience and look a he ypical JEL
codes used in you a ge jou nal (e en discuss he JEL code choice wi h colleagues). Copilo
links he choice o JEL codes o he choice o keywo ds by no ing ha “look o keywo ds and
ph ases wi hin he a icle ha ma ch he JEL code desc ip ions”. O he ac o s lis ed include
in e disciplina i y, cu en ends and eme ging opics.
In o de o compa e he JEL code choices be ween human in elligence (by esea che s) and
a i icial in elligence, we chose a se o a icles, he la es published issue o Jou nal o
Economic Li e a u e as o 12 Ap il 2024. This is he Ma ch 2024 issue, issue 1 o olume 62. I
includes se en a icles ha a e shown in Table 1. The Jou nal o Economic Li e a u e jou nal is
pa icula ly app op ia e case o s udy he choice o JEL codes as i is he ou le whe e he JEL
classi ica ion sys em was in oduced in 1969 and has been publishing he o icial classi ica ion
sys em e e since [4]. We p oceeded by p omp ing h ee gene a i e AI sys ems (OpenAI’s
Cha GPT, Mic oso ’s Copilo , Google’s Gemini) o assign JEL codes o he a icles using he
ollowing p omp [5]:
Sou ce(s): Au ho ’s illus a ion
Figu e 1. Compa ing human in elligence, augmen ed in elligence and AI in he choice o JEL codes
JD
81,7
20
Table 1. The choices o JEL codes based on human in elligence and a i icial in elligence, Jou nal o Economic
Li e a u e 62(1)
No e(s): The lis ed a icles appea ed in he Ma ch 2024 issue (Vol. 62 No. 1) o he Jou nal o
Economic Li e a u e which was he cu en issue as o Ap il 2024, a ailable a : h ps://www.
aeaweb.o g/issues/755. The i s numbe in pa en heses unde assigned JEL codes is he o e lap
wi h he inal JEL codes and he second numbe is he di e ence in he numbe o assigned
JEL codes compa ed o he inal ones. Used AI sys ems: Cha GPT, h ps://cha .openai.com/;
Copilo , copilo .mic oso .com; Gemini, h ps://gemini.google.com/. JEL codes a e p esen ed
in he sequence p o ided in pape s and
ecommended by AI sys ems. Da a (incl. in o ma ion
abou selec ed wo king pape s) and ull
esponses o AI sys ems a e a ailable in he
Appendix/Supplemen a y Ma e ial
Sou ce(s): Table by he au ho
Jou nal o
Documen a ion
21
Table 2. The choices o JEL codes based on human in elligence and a i icial in elligence, Ame ican Economic
Re iew 114(4)
No e(s): The lis ed a icles appea ed in he Ap il 2024 issue (Vol. 114 No. 4) o he Ame ican
Economic Re iew which was he cu en issue as o Ap il 2024, a ailable a : h ps://www.aea
web.o g/issues/757. The i s numbe in pa en heses unde assigned JEL codes is he o e lap
wi h he inal JEL codes and he second numbe is he di e ence in he numbe o assigned JEL
codes compa ed o he inal ones. Used AI sys ems: Cha GPT, h ps://cha .openai.com/; Copilo ,
copilo .mic oso .com; Gemini, h ps://gemini. google.com/. Da a (incl. in o ma ion abou
selec ed wo king pape s) and ull esponses o AI sys ems a e a ailable in he Appendix/
Supplemen a y Ma e ial
Sou ce(s): Table by he au ho
JD
81,7
22
Please, assign op imal JEL codes o he ollowing abs ac and explain why hey a e op imal:
[Abs ac he e]
Table 1 compa es he JEL codes assigned o he a icles (shaded columns) o he JEL codes ha
AI sys ems assigned o he a icles based on hei abs ac s only (Table 1a) and based on ull
ex s (Table 1b) o ea lie wo king pape e sions o he same a icles as well as he inal
e sions. The op ion o p o ide he AI sys em wi h he ull ex is no a ailable o all he used
AI sys ems and he ull ex s o he a icles a e mainly a ailable o subsc ibe s only. Thus, o
enable compa isons be ween di e en AI sys ems we used only abs ac s in Table 1a since he
abs ac s a e publicly a ailable online o anyone.
Since he e a e o en mul iple e sions o ea lie wo king pape s, in Table 1b we chose he
ea lies wo king pape e sions ha we ound om IDEAS RePEc and Google Schola and
p io i ized majo es ablished wo king pape se ies in economics (e.g. NBER, CEPR, c .
Baumann and Wohl abe, 2020)[6]. This illus a es how he se o JEL codes assigned o a
wo king pape – p esumably, ypically by au ho s hemsel es – can be signi ican ly di e en
om he ones assigned o he pee - e iewed inal pape .
Se e al pa e ns can be obse ed e en om his limi ed se o a icles. Fi s , bo h AI
sys ems and au ho s assign ypically sys ema ically less JEL codes compa ed o he inal se
and au ho s o en assign less JEL codes han AI. In he case o AI sys ems, his could be
explained by he ac ha we p omp ed he AIs o sugges JEL codes based on he abs ac only
and no based on he whole a icle. Fo ins ance, Gemini ypically sugges s h ee JEL codes,
one p ima y and wo seconda y ones. The jus i ica ion o he choice o he speci ic JEL codes
by he AIs is easonable and Gemini e en p o ides “jus i ica ion o excluding o he JEL
codes” and explains why some selec ed JEL codes would no be app op ia e.
Second, he o e lap be ween he ones sugges ed by AI sys ems based on abs ac s and inal
JEL codes a ies in he ange o 10%–30% (Table 1a). This may seem low, bu i should be
no ed ha AI sys ems (as well as au ho s) sugges sys ema ically less JEL codes. Unlike o he
selec ed AI sys ems, Copilo p o ides by de aul he in o ma ion sou ces unde lying i s
easoning o he JEL codes. The in es iga ion o hese sou ces e eals ha in mul iple cases
Copilo e e s o he publicly a ailable wo king pape s o he websi e o he inal e sion o he
unde lying a icle whe e he abs ac is a ailable. Some imes Copilo ends i s answe : “Fo
mo e de ails on he pape , you can e e o he [link o he a icle he e].”
Thi d, we also es ed which JEL codes Cha GPT 4 would assign o he selec ed wo king
pape e sion and he inal pee - e iewed a icles based on he ex o he whole a icle
(Table 1b). Despi e he ac ha he di e ences be ween he con en , ocus and scope o he
wo king pape e sion and he inal a icle a e ela i ely mino , Cha GPT 4 assigns qui e
di e en JEL codes o hem. I seems ha in mos cases Cha GPT 4 ecommends exac ly he
JEL codes lis ed in he a icles and a gues why hey a e app op ia e.
To conclude, gene a i e AI sys ems can augmen human in elligence in choosing he JEL
codes by p o iding easoned sugges ions based on he a icle abs ac s only.
Nex , in o de o es he obus ness o ou obse a ion ega ding disc epancies o JEL code
choices be ween human and a i icial in elligence in he Jou nal o Economic Li e a u e, we
applied he same me hod o he nine a icles published in he la es (as o Ap il 2024) issue o
he Ame ican Economic Re iew (AER).
Again, as Table 2 p esen s, we ind ha he gene a i e AI can sugges easonable JEL codes
and p o ide he easoning why hese could be he op imal se o JEL codes (see Supplemen a y
ma e ial). Again, in line wi h Kosnik’s (2018) and Heikkil€
a’s (2022) obse a ions, we ind ha
au ho -assigned JEL codes o he wo king pape e sions (Table 2b) o en di e om he inal
assigned JEL codes.
Again, au ho s (Table 2b) and AI sys ems assign ypically less JEL codes and Table 2a
indica es ha he o e lap be ween he JEL codes assigned by AI sys ems based on abs ac and
he inal JEL codes anges be ween ca. 18% (Gemini) and 65% (Copilo ). Fo bo h Cha GPT
3.5 and 4 he o e lap is abou 40%. While Copilo some imes e e s o he websi es whe e he
Jou nal o
Documen a ion
23
abs ac o he a icle - and ela ed JEL codes - a e publicly a ailable, i o en does no pick
hose exac JEL codes bu a he assigns a smalle numbe o JEL codes. Fo his se o a icles,
he ecommended JEL codes by Cha GPT4 based on he whole a icles ha e lowe o e lap (ca.
70%) wi h he inal ones compa ed o he o e lap epo ed in Table 1b (ca. 93%).
In he p io li e a u e i is common o ocus on mo e agg ega ed le els o JEL codes ins ead o
he mos g anula ones as we did in Tables 1 and 2. Fo ins ance, Ca d and Della Vigna (2013)
classi ied a icles based on hei own classi ica ion whe e JEL codes we e agg ega ed in o 14 ield
ca ego ies and ecen ly Bo nmann and Wohl abe (2024) ocused in hei analyses on he le el o
20 p ima y JEL code ca ego ies (see also Kosnik, 2018). In Tables A2 and A3 in he Appendix,
we apply his agg ega ed app oach based on he 20 p ima y JEL code ca ego ies o he a icles
p esen ed in Tables 1 and 2. These u he analyses show a highe le el o o e lap as expec ed
( anging be ween 50–100%) indica ing ha while he AI sys ems may no ecommend exac ly he
same JEL codes, hey a leas ecommend JEL codes om he same JEL code ca ego ies in mos
o he cases. I seems ha he impe ec o e lap s ems mainly om he ac ha AI sys em as well
as au ho s selec ewe JEL codes compa ed o he inal edi o -assigned JEL codes o he a icles.
The explo a o y e idence p esen ed he e indica es ha gene a i e AI may help esea che s
and augmen human in elligence in he choice o app op ia e JEL codes o hei a icles. A
minimum, gene a i e AI can be used o c oss-check he choices o esea che s i no o ully
au oma e he mic o ask. Thus, augmen ed in elligence may make he use o JEL classi ica ion
codes mo e e icien and consis en and sa e esea che s’ ime.
Discussion
Since lea ning he nuances o he AEA’s JEL codes guide equi es cos ly and ime-consuming
e o , i seems economic o u ilize AI sys ems in (pa ially) au oma ing he mic o- ask o
choosing JEL codes. Ou simple expe imen indica es ha gene a i e AI may help esea che s
and augmen hei boundedly a ional human in elligence in choosing app op ia e JEL codes
o hei a icles based on abs ac s. Howe e , we also documen ed ha he e a e signi ican
disc epancies be ween he chosen JEL codes by humans and he selec ed gene a i e AI
sys ems as well as be ween he AI sys ems.
I he scien i ic communi y and esea ch publishe s wan o con inue classi ying esea ch
pape s using JEL codes, hen he use o AI may help make he human choice o JEL codes less
boundedly a ional and mo e consis en (c . Kosnik, 2018). While i emains an open ques ion
how o de ine “app op ia e” o “op imal” choice o JEL codes, AI can sa e ime and help in
inding mo e “sa is icing” (Simon, 1955;Caplin e al., 2011;A inge e al., 2022) se s o JEL
codes. Mo e consis en use o JEL codes imp o es he aining da a o AI sys ems. When his is
complemen ed wi h au oma ed ecommenda ion sys ems ha sugges JEL codes bes
desc ibing esea ch con en , i could u he dec ease he sea ch cos s o he audiences as well
as p omo e he analysis o esea ch ends based on JEL codes (c . Ca d and DellaVigna, 2013;
Ang is e al., 2017;Bo nmann and Wohl abe, 2024).
We acknowledge ha he p esen ed p elimina y obse a ions ha e se e al limi a ions. Fi s ,
he analysis ocuses on only wo ecen issues o leading economics jou nals, so he ex e nal
alidi y o he obse a ions is limi ed. Mo e ex ensi e analyses o la ge numbe s o a icles
ac oss a mo e di e se se o jou nals would lead o mo e c edible and gene alizable obse a ions.
Second, he e is a hallucina ion p oblem wi h la ge language models – ha is, hey may
gene a e ex ha is no ue (Zhai, 2024). In his analysis we did no y o de ec hallucina ion,
bu a mo e igo ous analysis o JEL code choices wi h la ge se s o a icles should be
accompanied wi h he check o easoning o each selec ed JEL code ( o con i m ha he AI
sys ems do no come up wi h any hallucina ed JEL codes).
Thi d, “model collapse” (Shumailo e al., 2024) is ano he de imen al phenomenon
which e e s o he degene a i e ecu si e p ocess whe e AI sys ems ained wi h pollu ed
(incl. hallucina ed) da a end up aining he nex gene a ion o AI sys ems wi h model-
gene a ed pollu ed da a. Simila ly, in he con ex o JEL codes, i AI sys ems a e again and
JD
81,7
24
again ained wi h da a we e inapp op ia e o hallucina ed JEL codes a e s a is ically linked o
speci ic a icles, his p obably compounds he biases dynamically.
Fou h, he analysis used only a e y limi ed se o AI sys ems. As he de elopmen o AI
sys ems con inues, au ho s can consul an inc easing numbe o con inuously imp o ing AI
sys ems o c oss-check hei ecommenda ions o op imal JEL codes.
While he p elimina y indings p esen ed he e will become quickly obsole e as AI sys ems
(incl. aining da a) imp o e and a e inc easingly u ilized, u u e s udies may apply he
p esen ed app oach o analyze whe he he JEL code choices be ween au ho s, edi o s and AI
sys ems con e ge and become mo e consis en o e ime.
No es
1. See h ps://www.aeaweb.o g/jel/guide/jel.php Accessed 10 Ap il 2024. The guide also lis s “Ca ea s”
which, o ins ance, in he case o JEL code D82 “Asymme ic and P i a e In o ma ion: Mechanism
Design” men ion ha “S udies abou in o ma ion in gene al no asymme ic o p i a e should be
classi ied unde D83 [Sea ch: Lea ning; In o ma ion and Knowledge; Communica ion; Belie ;
Unawa eness]. Theo e ical s udies abou con ac heo y should be classi ied unde D86 [Economics
o Con ac s: Theo y]”. Then, D86 lis s addi ional ca ea s ha should be aken in o accoun in
classi ica ion.
2. See, e.g. Zhai (2024) o a ecen e iew o he oppo uni ies and challenges in he con ex o la ge
language models and in o ma ion e ie al. Zhai (2024, p. 481) no es ha “While s a is ical language
models ha e been applied o in o ma ion e ie al (IR) since many decades ago, hese new LLMs go
a beyond adi ional language models in hei ep esen a ion lea ning capaci y, which enabled hem
o bo h unde s and na u al language seman ically and gene a e luen meaning ul na u al
language ex .”
3. OpenAI launched Cha GPT 3.5 in No embe 2022 and GPT4 in Ma ch 2023. Google launched Ba d
in Feb ua y 2023 and eb anded i as Gemini in Feb ua y 2024. Mic oso launched Copilo in
Feb ua y 2023 ( o me ly Bing Cha ). The e a e also se e al o he gene a i e AI sys ems and new ones
a e launched, bu we in en ionally ocus on his limi ed se o popula ones (as o Ap il 2024).
4. See Che ie (2017) and Heikkil€
a(2022) o addi ional in o ma ion abou he his o y o he JEL codes
classi ica ion sys em.
5. We also expe imen ed by sligh ly changing he p omp s and a i ed a simila se s o JEL codes ha
bes i he pape s. Fo ins ance, asking he gene a i e AI sys ems o “choose JEL codes ha minimize
sea ch cos s” leads, gene ally, o a smalle numbe o ecommended JEL codes.
6. IDEAS is a la ge bibliog aphic da abase (mo e han 4.7 million i ems) dedica ed o Economics based on
Resea ch Pape s in Economics (RePEc) da a. A ailable a : h ps://ideas. epec.o g/. A signi ican sha e o
wo king pape s in he ield o economics a e indexed in RePEc (c . Baumann and Wohl abe, 2020).
Re e ences
Ang is , J., Azoulay, P., Ellison, G., Hill, R. and Lu, S. (2017), “Economic esea ch e ol es: ields and
s yles”, The Ame ican Economic Re iew, Vol. 107 No. 5, pp. 293-297, doi: 10.1257/ae .
p20171117.
A inge , F., Gige enze , G. and Jacobs, P. (2022), “Sa is icing: in eg a ing wo adi ions”, Jou nal o
Economic Li e a u e, Vol. 60 No. 2, pp. 598-635, doi: 10.1257/jel.20201396.
Baumann, A. and Wohl abe, K. (2020), “Whe e ha e all he wo king pape s gone? E idence om ou
majo economics wo king pape se ies”, Scien ome ics, Vol. 124 No. 3, pp. 2433-2441, doi:
10.1007/s11192-020-03570-x.
Bo nmann, L. and Wohl abe, K. (2024), “Recen empo al dynamics in economics: empi ical analyses
o annual publica ions in economic ields”, Jou nal o Documen a ion, Vol. 80 No. 4,
pp. 824-856, doi: 10.1108/JD-10-2023-0201.
B ynjol sson, E., Li, D. and Raymond, L. (2023), “Gene a i e AI a wo k”, NBER Wo king Pape No.
31161, a ailable a : h p://www.nbe .o g/pape s/w31161
Jou nal o
Documen a ion
25
