Idea Evaluation for Solutions to Specialized Problems: Leveraging the Potential of Crowds and Large Language Models

Gimpel, Henne ; Laubache , Robe ; P obos , Fabian; Schä e , Rica da; Schoch,
Man ed
A icle — Published Ve sion
Idea E alua ion o Solu ions o Specialized P oblems:
Le e aging he Po en ial o C owds and La ge Language
Models
G oup Decision and Nego ia ion
P o ided in Coope a ion wi h:
Sp inge Na u e
Sugges ed Ci a ion: Gimpel, Henne ; Laubache , Robe ; P obos , Fabian; Schä e , Rica da; Schoch,
Man ed (2025) : Idea E alua ion o Solu ions o Specialized P oblems: Le e aging he Po en ial o
C owds and La ge Language Models, G oup Decision and Nego ia ion, ISSN 1572-9907, Sp inge
Ne he lands, Do d ech , Vol. 34, Iss. 4, pp. 903-932,
h ps://doi.o g/10.1007/s10726-025-09935-y
This Ve sion is a ailable a :
h ps://hdl.handle.ne /10419/330757
S anda d-Nu zungsbedingungen:
Die Dokumen e au EconS o dü en zu eigenen wissenscha lichen
Zwecken und zum P i a geb auch gespeiche und kopie we den.
Sie dü en die Dokumen e nich ü ö en liche ode komme zielle
Zwecke e iel äl igen, ö en lich auss ellen, ö en lich zugänglich
machen, e eiben ode ande wei ig nu zen.
So e n die Ve asse die Dokumen e un e Open-Con en -Lizenzen
(insbesonde e CC-Lizenzen) zu Ve ügung ges ell haben soll en,
gel en abweichend on diesen Nu zungsbedingungen die in de do
genann en Lizenz gewäh en Nu zungs ech e.
Te ms o use:
Documen s in EconS o may be sa ed and copied o you pe sonal
and schola ly pu poses.
You a e no o copy documen s o public o comme cial pu poses, o
exhibi he documen s publicly, o make hem publicly a ailable on he
in e ne , o o dis ibu e o o he wise use he documen s in public.
I he documen s ha e been made a ailable unde an Open Con en
Licence (especially C ea i e Commons Licences), you may exe cise
u he usage igh s as speci ied in he indica ed licence.
h ps://c ea i ecommons.o g/licenses/by/4.0/
Vol.:(0123456789)
G oup Decision and Nego ia ion (2025) 34:903–932
h ps://doi.o g/10.1007/s10726-025-09935-y
Idea E alua ion o Solu ions oSpecialized P oblems:
Le e aging hePo en ial o C owds andLa ge Language
Models
Henne Gimpel1,2,3· Robe Laubache 5· FabianP obos 1,2,3·
Rica daSchä e 1,4· Man edSchoch1,3,6
Accep ed: 21 May 2025 / Published online: 28 June 2025
© The Au ho (s) 2025
Abs ac
Complex p oblems such as clima e change pose se e e challenges o socie ies wo ld-
wide. To o e come hese challenges, digi al inno a ion con es s ha e eme ged as a
p omising ool o idea gene a ion. Howe e , assessing idea quali y in inno a ion
con es s is becoming inc easingly p oblema ic in domains whe e specialized knowl-
edge is needed. T adi ionally, expe ju ies a e esponsible o idea e alua ion in
such con es s. Howe e , expe s a e a subs an ial bo leneck as hey a e o en sca ce
and expensi e. To assess whe he expe ju ies could be eplaced, we conside wo
app oaches. We le e age c owdsou cing and a La ge Language Model (LLM) o
e alua e ideas, wo app oaches ha a e simila in e ms o he agg ega ion o col-
lec i e knowledge and could he e o e be close o expe knowledge. We compa e
expe ju y e alua ions om inno a ion con es s on clima e change wi h c owd-
sou ced and LLM’s e alua ions and assess pe o mance di e ences. Resul s indi-
ca e ha c owds and LLMs ha e he abili y o e alua e ideas in he complex p ob-
lem domain while con es specializa ion— he deg ee o which a con es ela es o a
knowledge-in ensi e domain a he han a b oad ield o in e es —is an inhibi o o
c owd e alua ion pe o mance bu does no in luence he e alua ion pe o mance o
LLMs. Ou con ibu ion lies wi h demons a ing ha c owds and LLMs (as opposed
o adi ional expe ju ies) a e sui able o idea e alua ion and allows inno a ion
con es ope a o s o in eg a e he knowledge o c owds and LLMs o educe he
esou ce bo leneck o expe ju ies.
Keywo ds Idea e alua ion· C owdsou cing· La ge language model· Specialized
knowledge
Ex ended au ho in o ma ion a ailable on he las page o he a icle
904
H.Gimpel e al.
1 In oduc ion
Complex p oblems such as clima e change o social inequali y a e some o he
mos p essing challenges o ou ime. Complex p oblems sha e common cha ac-
e is ics: hey a e ( om he cu en poin o iew) unique, and humankind has li -
le expe ience wi h sol ing hem (Ri el and Webbe 1973). E en i solu ions a e
implemen ed, i o en akes yea s o decades un il he solu ion quali y becomes
isible and measu able (Gimpel e al. 2020). Thus, complex p oblems pose g ea
ques ions o humani y and equi e sophis ica ed p oblem-sol ing abili ies, in pa -
icula , in ol ing a b oad ange o a ec ed s akeholde s (Head 2008).
Online inno a ion con es s a e one ool used o os e inno a i e p oblem-sol -
ing by o e ing a pla o m ha b ings oge he people wi h di e en knowledge
(Chesb ough 2006). Such pla o ms o en add ess complex p oblems and can
con ain con es s o di e en sub-p oblems. These sub-p oblems a e di e en in
hei le el o specializa ion and hus equi e di e en le els o expe ise o sol e
hem. We de ine he e m specializa ion as he deg ee o which a con es ’s opic
belongs o one pa icula , knowledge-in ensi e domain a he han a b oad ield
o in e es .
The ease o accessing online inno a ion con es s usually leads o an abundance
o submi ed ideas, esul ing in he challenge o idea e alua ion (Blohm e  al.
2013). The e is as ag eemen ha g oups o expe ju ies a e he mos sui able
decision make s in he absence o an objec i ely known solu ion quali y (Klein
and Ga cia 2015; Blohm e al. 2016; Gö zen and Kundisch 2016; Naga e al.
2016). In his con ex , decision make e e s o he ins ance ha e alua es he
idea. Howe e , he esou ce o expe ju ies is sca ce and expensi e, c ea ing a
se e e ade-o be ween e iciency and quali y o e alua ions. This bo leneck is
one o he cen al challenges o using inno a ion con es s o sol e complex p ob-
lems (Naga e al. 2016).
In his pape , we he e o e ocus on he pa o he decision-making p ocess
ha deals wi h he e alua ion o he ideas. Speci ically, we s udy wo p omis-
ing al e na i es o expe ju ies in he e alua ion p ocess, namely e alua ion
by c owds and La ge Language Models (LLMs). These wo app oaches ha e
been s udied oge he since he eme gence o LLMs, as bo h a e simila in
e ms o he agg ega ion o collec i e knowledge. C owdwo king o idea e al-
ua ion is an app oach ha has gained ecogni ion om esea che s and p ac i-
ione s (Oos e man e al. 2014; Klein and Ga cia 2015; Gö zen and Kundisch
2016; Magnusson e al. 2016; Mollick and Nanda 2016; Wimbaue e al. 2019).
C owds e e o he gene al public (o selec ed communi ies he eo ) bu do no
equi e pa icipan s o possess speci ic skills o expe ise, in con as o expe
ju ies. The e m “wisdom o he c owd” desc ibes he phenomenon whe e he
a e age o seemingly unin o med indi idual opinions can lead o collec i e
in elligence when hey wo k oge he as a g oup (Mollick and Nanda 2016).
Exis ing esea ch compa es c owd and expe idea e alua ions in mainly co po-
a e con ex s and inds mixed esul s (Magnusson e al. 2016; Wimbaue e al.
905
Idea E alua ion o Solu ions oSpecialized P oblems:…
2019). Ye , mos co po a e idea e alua ions do no belong o he class o com-
plex p oblems o wo easons: (1) idea quali y can o en be de e mined quickly
in co po a e en i onmen s and (2) specialized knowledge is o en no equi ed
as he specializa ion o co po a e p oblems ypically is low (Magnusson e al.
2016; Wimbaue e al. 2019). So a , esea ch has no in es iga ed he po en ial
o c owdsou ced idea e alua ions in con ex s wi h in anspa en solu ion quali y
and a ying specializa ion.
Coming o he second app oach, he e alua ion ask can be au oma ed en i ely by
a i icial in elligence (AI), o example, machine lea ning models (Naga e al. 2016).
Fo ins ance, Naga e al. (2016) o e a compu a ional app oach whe e a machine
lea ning classi ie analyzes di e en cha ac e is ics o ex s o assess ideas. Howe e ,
he decisions o hese models a e based on a ious pa ame e s (such as he leng h o
he ex ) and a e no based on knowledge abou he con en o he ex s o be e alu-
a ed. Cu en ly, a p omising AI-based app oach o e alua ion a e LLMs ha gained
signi ican ecogni ion wi h he elease o Cha GPT a he end o 2022, pa icula ly
gi en ha hey a e capable o gene a ing ou pu o a highe quali y han p e ious
machine lea ning models in ce ain asks (Gao e al. 2023; Wang e al. 2023). Ini ial
esea ch on he capabili ies o hese models ocuses on simple e alua ion asks such
as s ance de ec ion o emo ion ecogni ion (Kocoń e al. 2023; Zhang e al. 2023).
Some o hese s udies show ha LLMs also ha e he abili y o execu e a ious asks
o which hey a e no ini ially ained (Kocmi and Fede mann 2023). Due o his
abili y, LLMs may be p omising o idea e alua ion. Howe e , p e ious esea ch on
LLMs’ capabili ies inco po a es nei he complex p oblems no specialized knowl-
edge in de ail.
Fo applying c owdsou cing o LLMs in inno a ion con es s o complex p oblems,
assessing he impac o specializa ion on idea e alua ion quali y is u gen ly needed, as
complex p oblems a e specialized by de ini ion, and specializa ion could push decision
make s o hei limi s as hey a e un amilia wi h such opics. We need o unde s and
be e which decision make can be used bes in which se ing in he p esence o spe-
cializa ion. The e o e, we add ess he ollowing esea ch ques ion:
Wha is he po en ial o c owdsou cing and LLMs o e alua ing solu ion ideas
o complex p oblems in he p esence o specializa ion?
To answe his ques ion, we compa ed c owd and LLM e alua ions o 104 ideas
om ou ea men g oups wi h di e en le els o specializa ion hos ed by he pla -
o m MIT Clima e CoLab wi h expe e alua ions. Ou esul s show ha depending on
he specializa ion c owds and LLMs ha e he abili y o e alua e ideas in he complex
p oblem domain. Con es specializa ion is a key inhibi o o c owd pe o mance ha
we can coun e ac by adap ing he e alua ion p ocedu e. In con as , we ind ha LLMs
can be e handle di e en deg ees o specializa ion and yield highe pe o mance
when using anking asks. Fo p ac ice, we ecommend ha o eplacing expe ju ies
employing LLMs a he han a c owd migh be app op ia e.
906
H.Gimpel e al.
2 Theo e ical Backg ound
2.1 E alua ing Solu ion Ideas o Complex P oblems
Complex p oblems such as clima e change, o en also e e ed o as “wicked
p oblems” a e desc ibed as “complex, in ol ing mul iple possible causes and
in e nal dynamics ha could no [be] assumed o be linea , and ha e e y neg-
a i e consequences o socie y i no add essed p ope ly” (Pe e s 2017). The e
is no clea pa h o sol ing hese p oblems. Howe e , a ew p inciples guide he
con inuing e o s: (1) he need o coo dina ion ac oss loca ions, (2) he need o
in ol e s akeholde s om all ypes o in e es g oups, and (3) he need o a high
deg ee o specialized knowledge (S e n 2006; Head 2008; Ka onen and B and
2009). Complex p oblems can be add essed h ough ideas and inno a ion— he
cen al ac o o sol ing p oblems ( on Hippel 1994).
Sol ing complex p oblems is a decision-making p oblem as i equi es bo h
he gene a ion o mul iple al e na i e ideas and he selec ion o he bes solu-
ion unde unce ain y and limi ed esou ces (Simon 1960). The concep ualiza-
ion o he decision-making p ocess de eloped by Simon (1960) di ides i in o
h ee phases: in elligence, design and choice. Fi s , he p oblem is analyzed, and
ele an in o ma ion is ga he ed (‘in elligence’). Then po en ial solu ions a e
de eloped (‘design’) and inally he bes op ion o he bes op ions a e selec ed
(‘choice’).
To acili a e idea gene a ion, which encompasses bo h he sys ema ic analy-
sis o he p oblem (‘in elligence’) and he c ea i e de elopmen o solu ions
(‘design’), he concep o IT-enabled idea gene a ion h ough open inno a ion
con es s has eme ged (Han e  al. 2020). These con es s p o ide a s uc u ed
app oach o collec and e ine ideas om di e se con ibu o s. Wi h he o en
abundan ideas gene a ed in inno a ion con es s, he challenge o e ec i ely e al-
ua ing and p ocessing hem a ises (‘choice’) (Blohm e al. 2013; Özaygen and
Balagué 2018). This challenge becomes e en mo e demanding when conside -
ing he cha ac e is ics o complex p oblems. In o de o e alua e solu ion ideas
o complex p oblems success ully, decision make s ely on hei own expe ience
and knowledge o make a quali ied decision (Ge lach e al. 2019). Making his
decision in highly specialized a eas equi es deep knowledge o speci ic domains
(Fische e al. 2012).
To app oach idea e alua ion in he ligh o complex p oblems, we e e o
knowledge ega ding idea e alua ion in gene al. Resea che s s udied he e alu-
a ion p ocedu e, wi h absolu e a ing and ela i e anking being wo o he bes -
known app oaches. In absolu e a ing, he decision make a es al e na i es
independen ly o each o he , o example, on a Like scale. In con as , ela i e
anking desc ibes decision make s’ di ec hie a chical a angemen o al e na-
i es (O adia 2004). Besides he e alua ion p ocedu es, esea che s s udied
he di e en ypes o decision make s (e.g., expe ju ies, ex e nal c owds, and
a ious echnical solu ions) (Klein and Ga cia 2015; Blohm e al. 2016; Naga
e al. 2016). P io esea ch sugges s ha expe ju ies gene ally ou pe o m o he

907
Idea E alua ion o Solu ions oSpecialized P oblems:…
decision make s and, hus, ha e been conside ed closes o he “g ound u h”
(Klein and Ga cia 2015; Blohm e  al. 2016; Naga e  al. 2016). In con as ,
c owds and echnical solu ions a e mo e cos - and ime-e icien han expe ju ies
(Vukice ic e al. 2022). Consequen ly, idea e alua ion is cha ac e ized by a ade-
o be ween e iciency and quali y.
The e o e, o le e age he po en ial o inno a ion con es s, he possibili y o
eplacing expe ju ies wi h c owds o LLMs in a way ha quali y is no impai ed
has gained in e es in p io decision-making li e a u e (Klein and Ga cia 2015;
Gö zen and Kundisch 2016; Magnusson e al. 2016; Mollick and Nanda 2016; Wim-
baue e al. 2019; Gao e al. 2023; Pe es e al. 2023; Wang e al. 2023).
2.2 C owd‑based Idea E alua ion
C owdsou cing desc ibes open calls o con ibu ions o selec ed ac i i ies o ben-
e i om human collec i e in elligence and allows o in eg a ing a di e se c owd
wi h di e en specialized knowledge, backg ounds, and expe iences (Howe 2006).
When a g oup o people, i.e. a c owd, e alua es ideas, he alue o he wo k o he
indi idual c owdwo ke s eme ges in he agg ega ion o all con ibu ions oge he .
Each indi idual con ibu ion o a c owdwo ke con ibu es o he c ea ion o a
la ge , eme gen alue ha is only ealized h ough he o ali y and agg ega ion o all
con ibu ions (Geige e al. 2012). Se e al s udies analyzed eplacing expe ju ies
wi h c owdsou ced idea e alua ions, o example, in co po a e inno a ion con es s
(Klein and Ga cia 2015; Gö zen and Kundisch 2016; Magnusson e al. 2016; Wim-
baue e al. 2019) o e alua ing p ojec s in he a s indus y (Oos e man e al. 2014;
Mollick and Nanda 2016). Gene ally, he esul s ega ding c owd pe o mance
a e mixed. Oos e man e al. (2014) see signi ican ly be e pe o mance in expe
ju ies on an image anno a ion ask. On he con a y, se e al s udies see cong uence
be ween expe and c owd e alua ions and, hus, see high po en ial in c owd e alu-
a ions—a leas o speci ic domains (Magnusson e al. 2016; Mollick and Nanda
2016; Wimbaue e al. 2019). G oups end o be mo e e icien o complex asks
han indi iduals (Almaa ouq e al. 2020). To adequa ely add ess c owd e alua ions
and ind gene alizable esul s, an unde s anding o he unde lying e alua ion p ocess
is necessa y.
In sum, exis ing esea ch es ablished c owd e alua ions as a p omising way o
e alua e ideas (Magnusson e al. 2016). Howe e , esea ch has no ye in es iga ed
he po en ial o c owd e alua ions unde he conside a ion o con es specializa ion.
While specialized knowledge has been men ioned as an essen ial ac o o idea
e alua ion (Gö zen and Kundisch 2016), unde s anding i s impac on c owd pe o -
mance is s ill needed.
2.3 LLM‑based Idea E alua ion
LLMs a e compu a ional models designed o analyze and gene a e ex (Susa la
e al. 2023). LLMs may ely on he Gene a i e P e- ained T ans o me (GPT) a chi-
ec u e, known o i s “a en ion mechanism” (Vaswani e al. 2017). The a en ion
908
H.Gimpel e al.
mechanism enables he model o lea n di e en posi ions o wo ds in a sen ence
du ing he p ocessing o inpu sequences by assigning indi idual weigh s o each
elemen in he inpu , acili a ing s ong con ex ualiza ion (Wol am 2023). Such
models a e p e- ained on ex ensi e ex da ase s, c ea ing a deep ep esen a ion o
he seman ic s uc u e o na u al language. While LLMs yield ascina ing esul s
ha a e well-documen ed, he knowledge embedded in such models is b oad and
no ask-speci ic. Howe e , LLMs can be ine- uned o speci ic asks o domains
(Ray 2023) and migh be coupled wi h o he knowledge sou ces such as knowledge
g aphs.
Se e al empi ical s udies ha e been conduc ed o in es iga e he capabili ies o
LLMs (Kocoń e al. 2023; Zhang e al. 2023). Fo ins ance, Kocoń e al. (2023)
in es iga ed an LLM’s capabili ies on 25 di e en Na u al Language P ocessing
(NLP) asks, such as sen imen analysis, emo ion ecogni ion, and s ance de ec ion.
Thei s udy compa es he LLM wi h he bes exis ing ask-speci ic au oma ion solu-
ion. They showed ha he LLM, on a e age, pe o med 25% wo se and concluded
ha he LLM can only cope wi h speci ic asks o a limi ed ex en . In con as ,
Zhang e al. (2023) examine LLMs’ capabili ies in s ance de ec ion. In such asks,
a subjec ’s s andpoin ( o , agains , o nei he ) o a claim in a ex is analyzed. The
s udy ound ha he LLM can keep up wi h he pe o mance o ask-speci ic s a e-
o - he-a solu ions.
In addi ion, se e al s udies ha e been published showing ha LLMs ha e expe
le el knowledge in domains such as oph halmology, law o ope a ions managemen
(Te wiesch 2023; Ma in e al. 2024; Thi una uka asu e al. 2024). Fo example,
Ma in e al. (2024) benchma k LLMs agains a g ound u h p o ided by senio
lawye s in he ask o con ac e iew. Thei indings indica e ha LLMs ma ch o
su pass human accu acy in iden i ying legal issues while being signi ican ly as e
and mo e cos -e ec i e. These s udies demons a e ha LLMs a e al eady capa-
ble o pe o ming many asks in knowledge wo k mo e accu a ely, e icien ly, and
cos -e ec i ely han humans. Building on hese indings, we aim o explo e whe he
LLMs can also make decisions in mo e complex and uns uc u ed domains.
Mos published s udies ha e been in he ze o-sho ange, desc ibing an LLM’s
abili y o execu e a new ask wi hou speci ic aining (Ray 2023). As exis ing s ud-
ies demons a ed good LLM pe o mance in ze o-sho asks, he e a e i s indi-
ca ions ha LLMs possess ze o-sho quali ies ac oss domains (Kocmi and Fede -
mann 2023; Kocoń e al. 2023). In addi ion o ze o-sho p omp ing, mo e complex
p omp ing s a egies, such as ew-sho p omp ing and chain o hough p omp ing,
can inc ease he quali y o LLM ou pu (Kocmi and Fede mann 2023).
E en hough he examples abo e do no di ec ly ela e o he ealm o complex
p oblem idea e alua ion, hey hin a LLMs’ po en ial in his a ea. Fi s , hey show
ha LLMs can succeed in ze o-sho asks. Second, he NLP skills examined in hese
s udies include bo h Na u al Language Unde s anding and Na u al Language Gen-
e a ion. These skills a e equi ed when e alua ing solu ion ideas o unde s and he
idea i sel and o gene a e an e alua ion. Ye , he e is also esea ch on LLMs’ abili y
o e alua e ex (Gao e al. 2023; Kocmi and Fede mann 2023; Wang e al. 2023).
P omp ing an LLM o a e he quali y o ex summa ies om di e en da ase s
showed ha an LLM can e alua e ex and, in some ins ances, is close o human
909
Idea E alua ion o Solu ions oSpecialized P oblems:…
expe ju ies han p e ious echnological e alua ion models (Gao e al. 2023). While
his ep esen s an impo an s ep owa ds le e aging LLMs o e alua ion asks,
e alua ing solu ions o complex p oblems equi es addi ional specialized domain
knowledge (Ka onen and B and 2009). In es iga ing he po en ial o LLMs in his
ega d is essen ial because i may help close one o he mos se e e bo lenecks o
inno a ion con es s.
3 Hypo hesis De elopmen
We in es iga e whe he c owds o LLMs can eplace expe ju ies in idea e alua-
ion o complex p oblems. The e o e, c owd and LLM pe o mance is assessed as
a measu e o how close he e alua ion is o he expe ju y’s decisions. An expe
ju y’s decision is no an objec i e measu e o idea quali y. Howe e , in he absence
o such a measu e, he expe s’ assessmen is as close o a g ound u h as one can
ge o he con ex o ou s udy (Klein and Ga cia 2015; Blohm e al. 2016). Ou
in e es lies in analyzing he e ec o con es specializa ion and he e alua ion p o-
cedu e unde which each decision make can be u ilized mos .
3.1 E ec o Specializa ion
The ype o ask decision make s ace s ongly impac s hei e alua ion pe o mance
(Poole e al. 1985). Con es specializa ion is a ele an ask cha ac e is ic because
many opics add essed in inno a ion con es s o complex p oblems a e highly spe-
cialized and, hus, o li le amilia i y o a gene al c owd. Fo human decision mak-
e s, li le amilia i y wi h a opic indica es ha indi iduals canno use long- e m
memo y esou ces o ackle a ask and, hus, a e bound o he es ain s o he wo k-
ing memo y (Kalyuga and Singh 2016). Thus, high con es specializa ion inc eases
he e alua ion ask’s complexi y (Song and B uning 2016). Consequen ly, highe
complexi y leads o wo se pe o mance by c owdwo ke s on a gi en ask (Mayna d
and Hakel 1997). This is caused by high complexi y causing a misma ch be ween
a ailable in o ma ion and ask equi emen s in ela ion o one’s a ailable p ocessing
capaci y (Cheng e al. 2020). We hus hypo hesize:
H1a Fo absolu e a ing, c owd e alua ion pe o mance is lowe o inno a ion
con es s o highe specializa ion.
As seen in he H1a, ou de aul o examining specializa ion is he absolu e a ing
o ideas. This app oach is also he de aul o LLM assessmen s. Gene ally, LLMs
should be able o e alua e due o hei abili y in ze o-sho ange (Gao e al. 2023;
Kocmi and Fede mann 2023; Wang e al. 2023). In addi ion, p e ious esea ch also
poin s o he abili y o e alua e. Resea ch is less clea ega ding he ela ionship
be ween con es specializa ion and he e alua ion pe o mance o LLMs. LLMs a e
ained on a la ge co pus o aining da a om a b oad ange o opics and, hus, pos-
sess a wide- anging knowledge base ha includes specialized knowledge in many
910
H.Gimpel e al.
di e en domains (Feue iegel e al. 2023; Ray 2023). Wi h ega d o complex asks,
he ans o me a chi ec u e o LLMs enables hem o p ocess complex con ex s
h ough di e en weigh ings in he inpu — he a en ion mechanism (Vaswani e al.
2017). This is c i ical o an LLM’s abili y o answe specialized ques ions app op i-
a ely by conside ing he nuances o a specialized domain. Consequen ly, LLMs a e
p obably amilia wi h highly specialized opics, and hei e alua ion pe o mance
does no depend on specializa ion le els. We hus hypo hesize:
H1b Fo absolu e a ing, LLM e alua ion pe o mance is no impac ed by con es
specializa ion.
3.2 In e ac ion E ec o Compu ed Ranking andSpecializa ion
Hypo hesizing he nega i e e ec o specializa ion on c owd pe o mance (H1a), he
ques ion o how o coun e ac his e ec eme ges. One possibili y lies in speci i-
cally adap ing he e alua ion p ocedu e o accoun o he challenges associa ed wi h
high con es specializa ion. Absolu e a ing pe o mance, as a gued in he de i a-
ion o H1a, likely dec eases wi h highe specializa ion. Howe e , e en i a decision
make e alua es se e al ideas independen ly on an absolu e scale, he p e ious e al-
ua ions will subconsciously impac he subsequen e alua ion sco es (Mussweile
and Englich 2005). In he ield o decision-making, his e ec is called ancho ing o
ancho ing bias (T e sky and Kahneman 1974), which, in ou case, e e s o he ac
ha he i s p oposals‘ e alua ions in luence he e alua ion o subsequen p oposals.
The esul ing subliminal compa ison be ween p oposals c ea es an implici ancho
ha helps o decide wha cons i u es a good o a bad idea. By a anging he abso-
lu e e alua ion sco es om highes o lowes and making decisions based solely on
he anking posi ion a he han he magni ude o he sco es, a compu ed anking is
o med. This compu ed anking allows o mo e consis en e alua ions, as he indi-
idual e alua ions a e seen in a con ex o ela i e posi ions. We hus hypo hesize:
H2a Compu ed anking educes he nega i e impac o specializa ion on c owd pe -
o mance as compa ed o absolu e a ing.
LLMs a e no explici ly de eloped o e alua ion bu a he conduc he e alu-
a ion based on hei b oad language-based aining da a (Ray 2023). These da a
included in he LLM p o ide an implici ancho wi h which he ideas can be com-
pa ed. When compu ing a anking o he absolu e a ings, di e ences be ween he
indi idual p oposals a e p esen ed in a mo e di e en ia ed way, as hey a e no only
conside ed in ela ion o he aining da a bu also in ela ion o each o he . Thus, a
compu ed anking should inc ease pe o mance ac oss asks o e e y specializa ion.
In line wi h ou assump ion in H1b, which posi s ha he e alua ion pe o mance is
equal o a ying deg ees o specializa ion, we con inue o no assume an in e ac ion
e ec o compu ed anking and specializa ion:
917
Idea E alua ion o Solu ions oSpecialized P oblems:…
homogenei y o he pa icipan s conce ning gende , age, educa ion, and p o ession
ac oss he ou g oups o ensu e compa abili y. Fo disc e e a iables, we use χ2
es s o homogenei y, and o con inuous a iables, we use ANOVA (Hai e al.
2010). Based on hose es s, pa icipan s do no di e signi ican ly be ween he
g oups (5% signi icance le el).
Nex , we p esen desc ip i e s a is ics o he e alua ions o he LLM. Figu e2
illus a es he dis ibu ion o he e alua ion sco es o each p oposal ac oss he ou
g oups. The boxplo s show he dis ibu ion o he 30 e alua ion sco es o each p o-
posal. The black line ep esen s he median, he black do ep esen s he mean, and
he whi e do s show ou lie s.
We can obse e a s ikingly low a iance in he e alua ion sco es in he boxplo s,
wi h mean and median alues clus e ed a ound 3.5. Thus, we conduc ed an ANOVA
o ind ou whe he signi ican di e ences exis be ween he mean e alua ion sco es
o he p oposals wi hin each g oup. We conduc ed he ANOVA o each ea men
g oup wi h 26 g oups (since each g oup con ains 26 p oposals). G oup 1 and 2
exhibi signi ican di e ences in hei e alua ion sco es (p- alue < 0.05). G oup 3
and 4 do no show signi ican di e ences in hei e alua ion sco es, as hei p- alues
a e > 0.05. Howe e , he pos -hoc Tukey es (Abdi and Williams 2010) shows sig-
ni ican di e ences be ween se e al p oposals wi h ega d o he mean e alua ion
sco es o all ou g oups. This enables an in e p e able anking o p oposals, dem-
ons a ing ha he LLM does no a e all p oposals equally.
5.2 Hypo heses Tes s
Table2 p esen s c owd and LLM pe o mance measu emen s o each g oup’s abso-
lu e a ing, compu ed anking, and ela i e anking. Nume ically, balanced accu acy
is bes o each g oup when using he LLM’s compu ed anking. Likewise, PPV is
bes o each g oup when using he LLM’s compu ed anking. Fo NPV, di e en
combina ions o e alua ion me ic and decision make pe o m bes .
To es he hypo heses, we use wo s a is ical es s. Fi s , we use Fishe ’s exac es
o see i he pe o mance in he g oups changes signi ican ly by changing he e alu-
a ion p ocedu e. We conduc he es wi h wo e alua ion p ocedu es as he ows
o he 2 × 2 ma ix, and p oposals anked iden ical as well as p oposals anked di -
e en ly han he expe ju y as i s columns (Up on 1992). Second, we use a χ2 es
o independence o see i he e is a signi ican e ec o specializa ion. The es is
Fig. 2 Boxplo s o he E alua ion Sco es o he LLM o each o he 104 P oposals

918
H.Gimpel e al.
pe o med in a 4 × 2 ma ix wi h he ou g oups o one e alua ion p ocedu e as he
ows. The wo columns indica e whe he a p oposal was e alua ed as iden ical o
di e en om he expe ju y as he second a iable (McHugh 2013).
5.2.1 E ec o Specializa ion onAbsolu e Ra ing Pe o mance (H1a andH1b)
Assessing balanced accu acy alues o an absolu e a ing o he c owd, we see a
pe o mance decline om he leas specialized g oup 1 (79%) o he mos special-
ized g oup 4 (50%). In g oup 1, he c owd a ed ideas a he simila o he expe
ju y. Wi h ising g oup specializa ion, he c owd’s p oposal e alua ions di e mo e
s ongly om he expe s. The esul ing p- alue o a χ2 es o independence is
0.015, indica ing ha he g oups’ obse ed pe o mance is signi ican ly di e en
be ween a leas wo o he g oups. A pai wise compa ison sugges s ha his is due
o signi ican ly highe pe o mance in g oup 1 compa ed o he h ee o he g oups.
Fo absolu e a ings, ad ance classi ica ions by he c owd we e mo e eliable o
he li le specialized g oups as PPV dec eases wi h g oup specializa ion. NPV al-
ues a e a 100% o g oups 1 and 2, indica ing ha all p oposals he c owd a ed
nega i ely we e also a ed he same way by he expe ju y. The c owd a ed e e y
p oposal as ad ance o he wo mo e specialized g oups, which also explains he
low PPV in hose g oups (no applicable (N/A) o NPV). Balanced accu acy and
NPV bo h suppo H1a. The lowe sco es o PPV as compa ed o NPV indica e ha
he c owd a) ends o a e p oposals oo lenien ly and ma ks oo many p oposals as
Table 2 Pe o mance Measu emen s o Absolu e Ra ing, Compu ed Ranking and Rela i e Ranking wi h
ascending Specializa ion om G oup 1 (G1) o G oup 4 (G4)
Balanced Accu acy PPVNPV
Absolu e
Ra ing
Compu e
d Ranking
Rela i e
Ranking
█= C owd
█= LLM
79%
56%
50%
50%
50%
50%
50%
50%
0%
20%
40%
60%
80%
100%
G1 G2 G3 G4
74%
38%
50%
42%
54%
35%
50%
42%
0%
20%
40%
60%
80%
100%
G1 G2 G3 G4
100% 100%
0%
20%
40%
60%
80%
100%
G1 G2 G3 G4
77%
66%
54%
69%
80%
74%
69%
76%
0%
20%
40%
60%
80%
100%
G1 G2 G3 G4
79%
56%
54%
64%
80%
67%
69%
73%
0%
20%
40%
60%
80%
100%
G1 G2 G3 G4
75%
76%
54%
73%
82%
82%
69%
80%
0%
20%
40%
60%
80%
100%
G1 G2 G3 G4
49%
62%
69%
74%
62%
53%
0%
20%
40%
60%
80%
100%
G1 G2 G3 G4
33%
62%
71%
67%
62%
45%
0%
20%
40%
60%
80%
100%
G1 G2 G3 G4
65%
62%
67%
82%
62%
60%
0%
20%
40%
60%
80%
100%
G1 G2 G3 G4
919
Idea E alua ion o Solu ions oSpecialized P oblems:…
ad ance, especially in highly specialized g oups. Also, b) he c owd is mo e eliable
in hei nega i e judgmen s.
Analyzing he balanced accu acy alues o he LLM o absolu e a ing, we see
a mo e consis en pic u e. Ac oss specializa ion le els, we can obse e a balanced
accu acy o 50%. As he pe o mance is cons an , we conclude ha i is independ-
en o specializa ion, suppo ing H1b. NPV alues o he LLM all show N/A. This
esul s om he LLM ma king e e y single p oposal as ad ance. The PPV alues
a e also in luenced by he ad ancemen o all p oposals. Like he c owd, we obse e
ha oo many p oposals a e e alua ed as ad ance. Fo absolu e a ing, LLM pe o -
mance is a he poo , independen om g oup specializa ion.
5.2.2 In e ac ion E ec o Compu ed Ranking andSpecializa ion (H2a andH2b)
The second se o hypo heses conce ns he compu ed anking p ocess. Fo he c owd,
we see a subs an ial pe o mance inc ease o balanced accu acy wi h a del a o up
o 19% (g oup 4) when shi ing om absolu e a ing o compu ed anking. Robus
ac oss all ou g oups, PPV alues inc ease be ween 4% (g oup 3) and 22% (g oup
4) compa ed o absolu e a ing. In compu ed anking, he c owd’s ad ance classi-
ica ions become mo e eliable. Fo he less specialized g oups, NPV pe o mance
dec eases as he c owd a es mo e ideas as no ad ance wi h a ew o hem being
alse classi ica ions. We conduc Fishe ’s exac es be ween absolu e a ing and com-
pu ed anking. Pe o mance does no di e signi ican ly be ween absolu e a ing and
compu ed anking (p- alues om g oup 1 o 4: 1.000, 0.093, 1.000, 0.093).
Nex , we assess he in e ac ion e ec o compu ed anking and specializa ion. Fo
he c owd, he impac o compu ed anking on balanced accu acy is mos mino o
g oup 1 (−2%) and la ges o g oup 4 (+ 19%). Acco dingly, wi h highe specializa-
ion, he e ec o compu ed anking on c owd pe o mance inc eases (χ2 p = 0.345),
indica ing ha he pe o mance is no signi ican ly di e en be ween he g oups.
Consequen ly, by changing he e alua ion p ocedu e, he specializa ion e ec disap-
pea s, which suppo s H2a.
When compa ing absolu e a ing and compu ed anking o he LLM, a clea
inc ease in pe o mance can be seen ac oss all g oups (del a be ween 19% o g oup
3 up o 30% o g oup 1). Fishe ’s exac es indica es ha he dis ibu ion o equal
and di e en a ings does di e signi ican ly be ween absolu e a ing and compu ed
anking (p = 0.075 o g oup 1, p = 0.004 o g oup 2, p = 0.023 o g oup 4, wi h
one ou lie in g oup 3: p = 0.258). This inc ease is p ima ily because he GPT model
allows all p oposals o ad ance in he absolu e a ing and is mo e nuanced in he
compu ed anking. PPV and NPV also show consis en ly high alues ac oss special-
iza ion le els. NPV alues a e he same o sligh ly highe han PPV (del a be ween
0% o g oup 3 up o 15% o g oup 2). Thus, he LLM is mo e eliable in iden i y-
ing bad p oposals. Balanced accu acy o compu ed anking is in a a he na ow
ange be ween 69% (g oup 3) and 81% (g oup 1) and no signi ican ly di e en (χ2
p = 0.801) ac oss he ou g oups. O e all, his means compu ed anking inc eases
pe o mance ac oss all g oups bu does no c ea e an e ec o specializa ion on pe -
o mance, which suppo s H2b.
920
H.Gimpel e al.
5.2.3 In e ac ion E ec o Rela i e Ranking andSpecializa ion (H2c andH2d)
The las se o hypo heses conce ns he ela i e anking p ocedu e. Fo he c owd,
balanced accu acy o he ela i e anking is 49% o g oup 2 and 62% o g oup
3. Since NPV is equal o o highe han PPV, he c owd is also mo e eliable in
iden i ying poo p oposals han good p oposals in he ela i e anking. The p- alues
o Fishe ’s exac es be ween compu ed anking and ela i e anking a e 0.393 o
g oup 2 and 0.779 o g oup 3, showing ha he dis ibu ion o equal and di e -
en a ings does no di e signi ican ly. The esul ing p- alue o a χ2 es o inde-
pendence o 0.779 shows ha he obse ed pe o mance is no signi ican ly di e en
be ween he g oups. Thus, as ou esul s show no u he inc ease in pe o mance,
bu he specializa ion e ec is also no longe p esen , H2c is suppo ed.
Fo he LLM, balanced accu acy o he ela i e anking anges om 53% (g oup
4) o 74% (g oup 2). Analogous o he compu ed anking, he NPV alues a e, on
a e age, be e han he PPV alues. This means he LLM is also be e a iden i-
ying poo p oposals when applying ela i e anking. The di e ence in balanced
accu acy be ween compu ed and ela i e anking is smalles o g oup 2 (0%) and
highes o g oup 4 (-23%). The p- alues o Fishe ’s exac es be ween compu ed
anking and ela i e anking show ha he dis ibu ion o equal and di e en a -
ings does no di e signi ican ly (p- alues om g oup 1 o 4: 0.532, 1.000, 0.771,
0.144). To examine he specializa ion e ec , we conduc a χ2 es o independence.
The esul ing p- alue o 0.334 indica es ha he g oups’ obse ed pe o mance is
no signi ican ly di e en be ween he g oups. Hence, he LLM’s a ing pe o mance
wi h ela i e anking does no depend on specializa ion, which suppo s H2d.
Table3 summa izes ou hypo heses and he espec i e empi ical esul s.
Table 3 Resea ch Hypo heses O e iew and Resul s
# Desc ip ion Empi ical
Resul s
H1a Fo absolu e a ing, c owd e alua ion pe o mance is lowe o inno a ion
con es s o highe specializa ion
Suppo ed
H1b Fo absolu e a ing, LLM e alua ion pe o mance is no impac ed by con es
specializa ion
Suppo ed
H2a Compu ed anking educes he nega i e impac o specializa ion on c owd
pe o mance as compa ed o absolu e a ing
Suppo ed
H2b The compu ed anking does no impac he e ec o specializa ion on LLM
pe o mance
Suppo ed
H2c Rela i e anking educes he nega i e impac o specializa ion on c owd pe o -
mance compa ed o compu ed anking
Suppo ed
H2d The ela i e anking does no impac he e ec o specializa ion on LLM
pe o mance
Suppo ed
921
Idea E alua ion o Solu ions oSpecialized P oblems:…
5.3 Analysis o Di e en ia ed P oposal E alua ions
To be e unde s and he decision make s’ easons o ad ancing o no ad ancing a
p oposal, we pe o m a logis ic eg ession o he ela ionship be ween he independ-
en a iables no el y, easibili y, impac , and p esen a ion (measu ed on a 4-poin Lik-
e scale) and he bina y dependen a iable (0 o no ad ance, 1 o ad ance) (Win-
ship and Ma e 1984). As he compu ed anking leads o he highes accu acy alues,
we ocus on his e alua ion p ocedu e. The assump ions o he logis ic eg ession,
pa icula ly he mul icollinea i y be ween he independen a iables, we e checked.
We i 15 eg ession models in o al: Fo each o he h ee decision make s, we i one
model pe g oup and an addi ional o e a ching model in eg a ing all da a. Exponen i-
a ed coe icien s a e used o in e p e a ion, indica ing a change in odds ( o assessing
p oposals as ad ance) when inc easing he espec i e independen a iable by one
uni . McFadden’s Pseudo R2 is compu ed as a goodness-o - i measu e, whe e alues
a ound 0.3 ep esen s ong model i (McFadden 1979). Table4 p esen s he esul s.
McFadden’s Pseudo R2 alues a e high (be ween 0.347 and 0.694) o he c owd and
ju y, indica ing s ong p edic abili y o he decision make s’ decision based on he
ou dimensions. The LLM alues a e lowe (be ween 0.066 and 0.267), indica ing
only a mode a e i o he logis ic eg ession model.
Fo eg essions wi h all g oups, all ou dimensions signi ican ly and posi i ely
impac he decision o classi y a p oposal as ad ance. Consequen ly, o he expe
ju y, he c owd, and he LLM, a highe e alua ion in he indi idual dimensions
leads o a highe p obabili y o ad ancing he p oposal. Fo he c owd, p esen a ion
impac s he decision subs an ially mo e han he o he dimensions. When looking a
he alues o all g oups o he LLM, p esen a ion, and impac ha e a high impac ,
Table 4 Logis ic Reg essions
*p < 0.05; **p < 0.01
All G oup 1 G oup 2 G oup 3 G oup 4
C owd No el y 1.66** 2.02** 3.20** 2.17* 1.05
Feasibili y 1.67** 1.86* 0.68 0.73 4.17**
Impac 1.69** 2.11** 1.81 0.88 2.56*
P esen a ion 4.80** 3.89** 5.30** 8.35** 11.47**
McFadden’s Pseudo R20.371 0.368 0.35 0.347 0.598
LLM No el y 1.74** 2.07** 1.53** 1.61** 1.47*
Feasibili y 1.57** 1.49 1.05 3.85 3.8**
Impac 3.48** 3.18** 2.93** 2.88** 11.50**
P esen a ion 4.85** 5.35** 2.56** 4.95** 6.42**
McFadden’s Pseudo R20.145 0.169 0.066 0.147 0.267
Expe Ju y No el y 3.54** 5.8* 2.05 5.31* 5.02
Feasibili y 2.59** 10.10** 2.41 1.61 1.12
Impac 3.11** 9.58** 2.73* 3.90* 1.03**
P esen a ion 4.20** 3.65* 3.36* 2.99 3.35*
McFadden’s Pseudo R20.515 0.694 0.445 0.468 0.614
922
H.Gimpel e al.
no el y, and easibili y, a mino impac . In con as , he expe ju y’s eg ession
coe icien s a e mo e balanced. Analyzing he indi idual g oup eg ession, an e ec
o specializa ion can be ound o he c owd’s e alua ion beha io . The mo e spe-
cialized a g oup is, he mo e s ongly he c owd bases hei decision on he p oposal
p esen a ion a he han impac , no el y, o easibili y. No specializa ion e ec s can
be iden i ied o he expe ju y and he LLM.
6 Discussion
6.1 Theo e ical Implica ions
Idea e alua ion by expe ju ies is a se e e bo leneck in inno a ion con es s, as
empo al and inancial esou ces a e o en sca ce. Consequen ly, we in es iga e he
po en ial o c owd and LLM e alua ions o elie e expe ju ies o he bu den o
e alua ing solu ions o complex p oblems.
Exis ing s udies on c owd e alua ions ocused on speci ic idea domains mainly
in a co po a e se ing, and hei esul s ega ding c owd pe o mance we e mixed
(Klein and Ga cia 2015; Gö zen and Kundisch 2016). Mollick and Nanda (2016)
highligh he impo ance o be e unde s anding he ci cums ances unde which he
c owd is a sui able decision make . Ou s udy p o ides e idence ha con es special-
iza ion is a key inhibi o o c owd pe o mance. Acco dingly, he c owd pe o ms
poo ly in he e alua ion o highly specialized opics. Howe e , less specialized op-
ics a e qui e sui able o he c owd, as he c owd is mo e amilia wi h hese op-
ics and can use hei long- e m memo y esou ces. Ou indings on lenien c owd
e alua ions and o e sco ing u he indica e ha c owdwo ke s a e hesi an o a e
a p oposal nega i ely. This e ec is pa icula ly s ong wi h high con es specializa-
ion, whe e he p esen a ion o a p oposal appea s o inapp op ia ely ou weigh o he
ele an cha ac e is ics. The c owd likely s uggles o ejec ideas when hey lack
he specialized knowledge equi ed o unde s and wha cons i u es a good o a poo
idea. Fo esea ch, hese con ibu ions shed ligh on e alua ion pa e ns in c owd-
sou cing and e eal a po en ial e o sou ce in c owdsou ced idea e alua ions.
In line wi h Magnusson e al. (2016), anking compa ed o absolu e a ing can
coun e ac he hesi ancy o ejec p oposals, pa icula ly in he p esence o high spe-
cializa ion. Ranking akes he esponsibili y o ejec ing a p oposal away om he
c owd, as he p oposals a e so ed in he o de o hei quali y. Con es ope a o s
a e hen esponsible o se ing a h eshold and making he inal decision ega ding
ad ancing and no ad ancing ideas. Ranking hus educes he bu den o unde s and-
ing wha good and bad p oposals cons i u e on a uni e sal scale o all exis ing ideas
in a o o a compa a i e app oach be ween he speci ic ideas in he espec i e con-
es . The e o e, he app oach appea s p omising and highly ele an o inno a ion
con es s in he complex p oblem domain. This highligh s ha con es ope a o s can
a ec he c owd’s abili y o e alua e ideas based on an adequa e e alua ion p oce-
du e design. Ou esul s show ha which anking app oach is used is less ele an ,
as he pe o mance does no di e signi ican ly be ween he compu ed anking and
he ela i e anking. The o iginal assump ion ha ela i e anking leads o be e

923
Idea E alua ion o Solu ions oSpecialized P oblems:…
pe o mance was no suppo ed he e. One possible eason o his could be ha he
subliminal compa ison be ween p oposals in he compu ed anking is al eady su -
icien o e alua e he p oposals app op ia ely.
Mos p e ious esea ch on he capabili ies o LLMs has been ela ed o NLP asks
(Kocoń e al. 2023; Zhang e al. 2023) which we ha e now ex ended o a p ac ical
e alua ion ask in he a ea o complex p oblems. In line wi h s udies o LLM pe o -
mance in NLP asks e e ing o e alua ion (Gao e al. 2023; Kocmi and Fede mann
2023; Wang e al. 2023), we show ha LLMs possess e alua ion capabili ies unde
he igh ci cums ances. The e alua ion p ocedu e in pa icula should be ep esen ed
by a ( ela i e o compu ed) anking, as he LLM pe o ms consis en ly and accu a ely
when applying a anking. In con as , he LLM pe o ms consis en ly bu no neces-
sa ily accu a ely in he absolu e a ing. A high consis ency canno be used o in e
pe o mance because consis ency e lec s s abili y, no necessa ily he quali y o he
ou comes. When conside ing he specializa ion, he e alua ion pe o mance o LLMs
seems independen o specializa ion o each e alua ion p ocedu e. This adds o he
heo e ical ounda ions o LLMs, showing hei inhe en capaci y o inco po a e a
wide a ay o di e se specialized knowledge o ca y ou e alua ion asks e ec i ely.
A second no ewo hy inding is ha he LLM in ou s udy ends o assign only
e y good a ings wi h minimal a iance ac oss p oposals on an absolu e scale. This
sugges s ha sco ing p oposals based on an absolu e a ing may no cap u e nuanced
di e ences in ask complexi y and comp ehension. A i s sigh , no clea h esh-
old di ides he e alua ion sco es in o good and bad. Compa ing he compu ed wi h
he ela i e anking, he LLM pe o ms wo se in ela i e anking. As his di e ence
has no s a is ical signi icance, we canno say which anking app oach is supe io .
The compu a ional complexi y o he compu ed anking is in he class o linea ime
(O(n)), while i is quad a ic ime (O(n2)) o he ela i e anking based on pai wise
compa isons. This means ha i p oposals a e compa ed in pai s, each p oposal
mus be compa ed wi h each o he and he e o e he e o is highe han i you only
e alua e a single p oposal. Fo a la ge numbe o p oposals, his migh lead o a o -
ing he compu ed anking. The e o e, esea che s in he ield o LLMs should ca e-
ully conside adop ing anking app oaches o gain mo e meaning ul insigh s in o
he model’s pe o mance on di e en asks.
Based on a logis ic eg ession, we we e able o un angle he basis on which he
LLM makes i s decision. The mode a e i o he eg ession models o he LLM
shows ha we can only explain a small pa o he a iance o he LLM’s decision
o ad ance p oposals. LLMs, he e o e, emain a kind o black box o us in hei
mode o ope a ion (Feue iegel e al. 2023). This inding unde lines he impo ance
o Explainable AI o be e unde s and how AI models make decisions (Ray 2023).
Syn hesizing he implica ions o bo h decision make s, i can be concluded ha
c owds and LLMs ha e he abili y o e alua e ideas in he complex p oblem domain.
The pe o mance o he LLM is no dependen on he deg ee o specializa ion. In
con as , i ’s a key inhibi o o he c owd, bu i can disappea h ough he choice o
e alua ion p ocedu es. Fu he analysis show ha bo h decision make s e alua e he
ew e y bes and wo s ideas mos accu a ely. The e o e, hey a e pa icula ly help-
ul in he p eselec ion phase o inno a ion con es s o weed ou bad ideas and educe
he bu den on expe ju ies.
924
H.Gimpel e al.
6.2 P ac ical Implica ions
Se e al p ac ical implica ions can be in e ed om ou s udy. Fi s , ou esul s show
ha , in gene al, c owds and LLMs can be used as decision make s in inno a ion
con es s ha add ess complex p oblems. Howe e , he impac o specializa ion high-
ligh s he need o con es ope a o s o know he specializa ion o he opic o be
e alua ed. I in o ma ion on he specializa ion is a ailable, a c owd can be used o
he e alua ion o less specialized a eas. I he specializa ion is no known o c i -
ically high, an LLM e alua ion migh be a mo e sui able solu ion as i e alua es
ega dless o specializa ion le els.
Second, ou indings on he e ec i eness o anking e alua ion p ocedu es se e
as guidelines o designing e alua ion p ocedu es in inno a ion con es s. Inno a ion
con es s should use anking asks a he han absolu e a ings o bo h c owds and
LLMs o achie e assessmen s close o expe ju ies. This holds in pa icula o
e alua ions by he c owd in con es s whe e he domain is highly specialized o spe-
cializa ion le els a e unknown.
Thi d, we obse e ha he p esen a ion o he p oposals s ongly impac s he deci-
sion make ’s decision o ad ance a p oposal, pa icula ly o he c owd. This inding
could be conside ed when designing he pla o m by poin ing idea gene a o s o he
impo ance o he p esen a ion and s anda dizing he p esen a ion’s le el o de ail
and p o essional appea ance. Fu he , he c owd migh be explici ly in o med no o
o e ly ocus on he p esen a ion o a p oposal bu o ocus on aspec s such as no -
el y, easibili y, and impac .
The undamen al challenge o balancing e iciency and quali y canno be sol ed
en i ely. Howe e , ou s udy sugges s ha when an expe ju y is no a ailable and
he s akes in he decision a e no oo high, employing LLMs a he han a c owd
migh be app op ia e. We speci ically poin o LLMs he e a he han a c owd as—
conside ing he nume ical alues o balanced accu acy— he compu ed anking om
he LLM ou pe o ms o he me ics and he c owd ( he ou pe o mance is nume ical
bu no s a is ically signi ican ly di e en om ze o in all cases). The LLM e alu-
a ion quali y ends no o de e io a e wi h con es specializa ion, and—gi en some
echnical expe ise o assessing an LLM—e alua ion is quicke and cheape wi h
an LLM as compa ed o a c owd. Fu he , GPT-3.5 allowed us o achie e oughly
70–80% balanced accu acy compa ed o he expe ju y. Using GPT-4 o o he mo e
ad anced models ha will become a ailable o e ime will likely inc ease accu acy.
A inal wo d o cau ion: o high-s akes decisions, 80% accu acy migh no su ice,
and expe ju ies migh be p e e able.
6.3 Limi a ions & Fu u e Resea ch
Ou esea ch has limi a ions, which highligh he need o u u e esea ch. Fi s , he
e alua ion o he c owd ook place in 2020, and he knowledge base o he LLM
ex ends o 2021. In his pe iod, he e migh ha e been echnological o poli ical p o-
g ess in some o he idea a eas discussed in he inno a ion con es s. Consequen ly,
hese di e ences may impac he pe cep ion o an idea.
925
Idea E alua ion o Solu ions oSpecialized P oblems:…
Second, ou esea ch builds upon he co e assump ion ha expe ju y e alua ions
a e closes o he unknown g ound u h o idea quali y. Acco dingly, we assess he
decision make s’ pe o mance by compa ing hei e alua ion decision o he expe
ju y. This assump ion is based on ex ensi e esea ch in he domain (Klein and Ga cia
2015; Blohm e al. 2016; Naga e al. 2016). Howe e , o ully alida e his assump ion
and be e unde s and he decision make s’ e alua ion beha io , i would be bene icial
o u u e esea ch o ack he implemen a ion p og ess and esul ing bene i s o he
e alua ed ideas. Simila o Wimbaue e al. (2019) esea ch could hen assess hei
ac ual quali y and, e en be e , e alua e he idea p e e ences o he decision make s.
A e demons a ing he gene al sui abili y o LLMs in he complex p oblem
domain, ou s udy o e s some pa hways o imp o ing e alua ion pe o mance.
Fu he esea ch should in es iga e mo e complex p omp ing s a egies o achie e
highe -quali y ou pu . In addi ion o adap ing he p omp ing, ine- uning an LLM o
he speci ic ask and knowledge a ea is possible o achie e be e pe o mance.
In addi ion, we a e endea o ing o ob ain objec i e e alua ions o inno a i e
ideas om an LLM and compa e hem wi h expe e alua ions. Ano he ele an
esea ch s eam deals wi h he modeling o human opinions by LLMs o speci ic
popula ions in social science (A gyle e al. 2023; Dominguez-Olmedo e al. 2024;
Lee e al. 2024). This esea ch s eam is pa icula ly ele an o ou wo k as i con-
ibu es o a be e unde s anding o how LLMs can eplica e objec i e and subjec-
i e judgmen s. The esea ch sugges s ha LLMs could model c owds wi h di e se
opinions ha may be well sui ed o e alua e ideas in combining bo h LLMs and
c owd esea ch. Me ging insigh s om bo h esea ch s eams could open up u he
pe spec i es on he e alua ion o ideas.
The e ol ing capabili ies o LLMs ha e been demons a ed in many s udies, and
i seems ine i able ha decision-making will become inc easingly au oma ed in he
u u e. This is due o a combina ion o ac o s, including echnological ad ances, he
g owing a ailabili y o da a and he need o mo e e icien solu ions. As a esul ,
many decision-making p ocesses will inc easingly be delega ed o machines. Ou
s udy highligh s ha ull au oma ion is no ye achie able, unde sco ing he con in-
ued necessi y o in e ac ion be ween humans and echnology. The e o e, i is c ucial
o complemen he pu ely echnical pe spec i e wi h a socio echnical app oach ha
conside s his in e play. Thus, u u e decision-making li e a u e should ake a holis ic
iew o he en i e decision-making sys em, ocusing on bo h e ec i eness and e i-
ciency (S o ey e al. 2024). I is he e o e impe a i e o add ess human compu e in e -
ac ion and he e ec i e delega ion be ween he wo in he decision-making p ocess in
o de o op imally u ilize hei espec i e s eng hs (Bai d and Ma uping 2021).
7 Conclusion
Designing idea e alua ions in inno a ion con es s o complex p oblems mo e e i-
cien ly is inc easingly impo an , as adi ional expe ju ies o en cons i u e a se e e
inancial and empo al bo leneck. This s udy p o ides he i s de ailed assessmen o
he po en ial o c owdsou ced idea e alua ions and LLMs’ idea e alua ions in he com-
plex p oblem domain. Ou esul s sugges ha specializa ion and e alua ion p ocedu es
926
H.Gimpel e al.
a ec he c owd’s po en ial ac oss con ex s and domains. Wi h ega d o he LLM, ou
esul s show ha i can p ocess asks e en in he ze o-sho ange, ega dless o speciali-
za ion. Conce ning p ac ice, we show ha o some deg ee c owds and especially LLMs
a e indeed sui able o idea e alua ions, e en in he complex p oblem domain. C owds
and LLMs a e impo an con ibu o s in ackling he bo leneck o expe ju ies.
Appendix A: Su ey I ems
No el y (sou ce: Clima e CoLab)
Scale Label
1 Common, mundane, bo ing
2 In e es ing, bu no unhea d o
3 Unusual, in e es ing, imagina i e
4 Ra e, su p ising, challenging pa adigms
Feasibili y (sou ce: Clima e CoLab)
Scale Label
1 In easible socially, poli ically, legally o echnically
2 Challenging, easibili y is ques ionable
3 Accep able; Objec ions & ba ie s pa ially
add essed
4 Appealing; Po en ial objec ions & ba ie s well
add essed
Impac (sou ce: Clima e CoLab)
Scale Label
1 Bene i s/ impac no clea
2 Limi ed bene i s/small impac
3 Pa ial solu ion/ mode a e impac
4 La ge, di ec , & posi i e impac
P esen a ion (sou ce: Clima e CoLab)
Scale Label
1 Nei he clea , pe suasi e no appealing
2 Only 1 o he ollowing 3 applies: Clea , Pe suasi e,
Appealing
3 Only 2 o he ollowing 3 apply: Clea , Pe suasi e,
Appealing
4 All 3 o he ollowing 3 apply: Clea , Pe suasi e,
Appealing
O e all Sco e (sou ce: e.g. Gö zen e al.. 2016): Gi en he quali y o he idea p oposal you jus saw and
he a ing you sco ed on all ou dimensions, do you hink he idea should ad ance o he nex ound?
Scale Label
1 Absolu ely no
2 Ra he no
3 Ra he yes
4 Absolu ely yes