Incentive alignment in conjoint analysis: a meta-analysis on predictive validity

Sch amm, Joshua Benjamin
A icle — Published Ve sion
Incen i e alignmen in conjoin analysis: a me a-analysis
on p edic i e alidi y
Ma ke ing Le e s
P o ided in Coope a ion wi h:
Sp inge Na u e
Sugges ed Ci a ion: Sch amm, Joshua Benjamin (2025) : Incen i e alignmen in conjoin analysis: a
me a-analysis on p edic i e alidi y, Ma ke ing Le e s, ISSN 1573-059X, Sp inge US, New Yo k, NY,
Vol. 36, Iss. 3, pp. 533-546,
h ps://doi.o g/10.1007/s11002-025-09764-8
This Ve sion is a ailable a :
h ps://hdl.handle.ne /10419/330642
S anda d-Nu zungsbedingungen:
Die Dokumen e au EconS o dü en zu eigenen wissenscha lichen
Zwecken und zum P i a geb auch gespeiche und kopie we den.
Sie dü en die Dokumen e nich ü ö en liche ode komme zielle
Zwecke e iel äl igen, ö en lich auss ellen, ö en lich zugänglich
machen, e eiben ode ande wei ig nu zen.
So e n die Ve asse die Dokumen e un e Open-Con en -Lizenzen
(insbesonde e CC-Lizenzen) zu Ve ügung ges ell haben soll en,
gel en abweichend on diesen Nu zungsbedingungen die in de do
genann en Lizenz gewäh en Nu zungs ech e.
Te ms o use:
Documen s in EconS o may be sa ed and copied o you pe sonal
and schola ly pu poses.
You a e no o copy documen s o public o comme cial pu poses, o
exhibi he documen s publicly, o make hem publicly a ailable on he
in e ne , o o dis ibu e o o he wise use he documen s in public.
I he documen s ha e been made a ailable unde an Open Con en
Licence (especially C ea i e Commons Licences), you may exe cise
u he usage igh s as speci ied in he indica ed licence.
h ps://c ea i ecommons.o g/licenses/by/4.0/
Vol.:(0123456789)
Ma ke ing Le e s (2025) 36:533–546
h ps://doi.o g/10.1007/s11002-025-09764-8
ORIGINAL RESEARCH
Incen i e alignmen inconjoin analysis: ame a‑analysis
onp edic i e alidi y
JoshuaBenjaminSch amm1
Accep ed: 9 Janua y 2025 / Published online: 20 Janua y 2025
© The Au ho (s) 2025
Abs ac
Conjoin analysis is a widely used me hod in ma ke esea ch o p edic ing con-
sume pu chases, making p edic i e alidi y a cen al ene . Conjoin analyses, how-
e e , a e ypically conduc ed in hypo he ical se ings, making hem suscep ible o
hypo he ical bias. One solu ion is incen i e-aligning conjoin s udies o igge u h-
ul answe ing beha io , he eby inc easing he accu acy o p edic ions. Howe e ,
despi e incen i e alignmen ’s concep ual appeal, p ac i ione s a ely use i . One ea-
son o his is he unce ain y o i s e ec i eness. This esea ch sys ema ically in es-
iga es he gains in p edic i e alidi y employing a me a-analysis o 134 e ec sizes
om 34 a icles (N = 12,980). Incen i e alignmen inc eases he p edic i e alid-
i y (i.e., hi a e) by 12%, p o iding a signi ican inc ease in accu acy. In addi ion,
i s e ec i eness is ampli ied when esea ching du able and se ice goods ( s. non-
du able goods) and when he payou p obabili y ises. In con as o con en ional
wisdom, indi ec ( s. di ec ) incen i e p ocedu es do no mi iga e he posi i e e ec s
on p edic i e alidi y. We hope o s imula e a e hink in p ac ice o make mo e use
o incen i e alignmen and help decide whe he incen i e alignmen is wo h he
addi ional e o .
Keywo ds Conjoin analysis· Hypo he ical bias· Incen i e alignmen · Ma ke
esea ch· Mul i a ia e me a-analysis· P edic i e alidi y
1 In oduc ion
Conjoin analysis is a me hod o assessing p e e ences among p oduc a ibu es
and hei co esponding le els (Pa k e al., 2008). I is widely used, o example,
o de ine p omising p oduc s (Gilb ide e al., 2008), o measu e willingness- o-pay
* Joshua Benjamin Sch amm
joshua.sch amm@o gu.de
1 Facul y o Economics andManagemen , O o on Gue icke Uni e si y Magdebu g, Magdebu g,
Saxony-Anhal , Ge many
534
Ma ke ing Le e s (2025) 36:533–546
(WTP; Schmid & Bijmol , 2020), o o de e mine he impo ance o a ibu es in
he decision-making p ocess (S eine & Meißne , 2018). Conjoin analysis’ ul ima e
goal is o p edic consume s’ pu chase decisions (e.g., Ding, 2007; G een & S ini-
asan, 1990). Measu ing p e e ences accu a ely is c ucial because only p oduc s ha
ma ch consume s’ ue as e will be success ul (Hause e al., 2014).
P ac i ione s and academics can choose om a wide a ie y o conjoin analyses
(see, e.g., S eine & Meißne , 2018). The mos commonly applied conjoin analysis
me hod is he choice-based conjoin (CBC; Saw oo h So wa e Inc., 2024) due o i s
simila i y o ac ual choice beha io (Lou ie e & Woodwo h, 1983). A CBC su -
ey ypically includes a sequence o choice asks in which esponden s choose he
al e na i e hey would mos likely buy (Yang e al., 2018) and alida ion asks (i.e.,
asks excluded o u ili y es ima ion, also called holdou asks).1 Valida ion asks a e
impo an o assessing he alidi y o he esponden s’ choices and he esul s o he
conjoin analysis (see, e.g., G een & S ini asan, 1990).
Despi e he widesp ead use o conjoin analysis, esea che s exp ess conce ns
ha he esul s ob ained may no e lec ac ual p e e ences (Gilb ide e al., 2008), as
conjoin analyses a e ou inely conduc ed in hypo he ical se ings (Ding e al., 2005;
Pachali e al., 2023). In hese se ings, esponden s may no exe he same cogni i e
e o as in an ac ual pu chase si ua ion (Toubia e al., 2012), which ul ima ely can
lead o a disc epancy be ween s a ed and ac ual p e e ences (i.e., hypo he ical bias;
see, e.g., Ding, 2007; Ho s e e e al., 2021).
To adequa ely add ess hypo he ical bias, esea che s ha e in oduced incen i e
alignmen o conjoin analysis. Ins ead o a ixed paymen , as is common in ma ke
esea ch, esponden s’ s udy disbu semen depends on hei choices in he choice
asks (seee.g., Ding e al., 2005). Mul iple s udies ha e shown i s e ec i eness in
conjunc ion wi h p e e ence measu emen echniques, such as (adap i e) CBC (see,
e.g., Ding, 2007; Sablo ny-Wacke shause e al., 2024).
Haghani e al. (2021) and Schmid and Bijmol (2020) examine hypo he ical bias
and p edic i e alidi y in mo e de ail. While Haghani e al. (2021) conduc ed a sys-
ema ic li e a u e e iew on hypo he ical bias in di e en disciplines and show ha
hypo he ical bias is an issue in 11 o he 18 s udies examined in consume econom-
ics, Schmid and Bijmol (2020) conduc ed a me a-analysis.2 The la e esea che s
show ha hypo he ical and ac ual WTP di e by 21% and ha he e ec on hypo-
he ical bias is g ea e o indi ec ( s. di ec ) me hods such as conjoin analysis
(Schmid & Bijmol , 2020). Howe e , de e mining WTP is only one ield o applica-
ion o conjoin analysis, and p ice is no necessa ily an included a ibu e in e e y
ma ke esea ch p oblem. Fo example, esea che s a e some imes no p ima ily
in e es ed in de e mining he op imal p ice bu in he choice sha es o same-p iced
p oduc s. The e o e, he p esen me a-analysis aims o shed ligh on ano he o m
o p edic i e alidi y, namely, whe he incen i e alignmen can imp o e he co ec
p edic ion o p oduc decisions.
1 We use he e m alida ion asks o bo h alida ion asks and holdou asks.
2 Hypo he ical bias was absen in only 3 o he 18 s udies (Haghani e al., 2021).
535
Ma ke ing Le e s (2025) 36:533–546
This inqui y con ibu es o he li e a u e in he ollowing ways. Fi s , we examine
incen i e alignmen ’s e ec i eness in imp o ing conjoin analysis’ p edic i e alid-
i y. Recen ly, Pachali e al. (2023, p. 970) showed ha 96% o he conjoin s ud-
ies ca ied ou by ma ke esea ch companies a e hypo he ical. We demons a e he
po en ial bene i s p ac i ione s can gain by showing incen i e alignmen ’s o e all
e ec on p edic i e alidi y. Second, we p o ide esea che s wi h an o e iew o
wha o expec when applying incen i e alignmen in conjoin analysis (e.g., a e age
addi ional cos s). Bo h may alle ia e doub s abou incen i e alignmen among p ac-
i ione s. Thi d, we examine po en ial mode a o s ha migh in luence he p edic i e
alidi y imp o emen ’s magni ude (e.g., payou p obabili y).
Ou me a-analysis assesses he p edic i e alidi y in e ms o he hi a e (i.e.,
he pe cen age o co ec ly p edic ed choices in alida ion asks; Ding e al., 2005),
as is common in he ield (see, e.g., Wlöme & Egge s, 2016). Based on 134 e ec
sizes om 34 a icles and a o al o 12,980 esponden s, we conclude ha incen i e-
aligning conjoin analysis inc eases he hi a e by 12%.
2 Theo e ical backg ound andhypo heses de elopmen
2.1 Conjoin analysis
Conjoin analysis is one o he mos equen ly used ma ke esea ch me hods
(Pachali e al., 2023). Fo ins ance, ma ke esea che s apply his class o me hods o
measu e WTP o p oduc s o speci ic p oduc a ibu es (Schmid & Bijmol , 2020),
o o assess he p oduc a ibu es’ impo ance in he decision-making p ocess
(S eine & Meißne , 2018). In i s classical o m, esponden s ei he a e o ank di -
e en p oduc p o iles, o hey decide on one p o ile in a pai ed compa ison (S eine
& Meißne , 2018). Nowadays, CBC supe seded hese o ms o conjoin analysis
(Saw oo h So wa e Inc., 2024). CBC enjoys equen usage due o i s close mim-
ics o eal pu chase decisions (Lou ie e & Woodwo h, 1983). Responden s ace a
sequence o choice asks in which hey op o he al e na i e hey would mos likely
buy (S eine & Meißne , 2018). To enhance ealism, CBC o en includes a no-buy
al e na i e as ano he al e na i e in he choice ask (see Fig. A1 in he Appendix o
an exempla y ask). Some imes esea che s include he no-buy al e na i e in a dual-
esponse logic ins ead, in which esponden s answe a o ced choice ask i s and
hen decide whe he he chosen al e na i e is an ac ual pu chase op ion (see, e.g.,
B azell e al., 2006).
CBC is a s a ic me hod. Choice asks do no change based on he esponden ’s
indica ed p e e ences, which migh esul in i ele an choices o some espond-
en s (Sablo ny-Wacke shause e al., 2024). To o e come his s a ic na u e, esea ch-
e s ha e de eloped adap i e designs ha use he esponden ’s answe s o p e ious
choice asks o c ea e new ones (see, e.g.,Johnson & O me, 2007). The mos com-
monly used adap i e me hod is he adap i e CBC (ACBC; see, e.g., Johnson &
O me, 2007; Sablo ny-Wacke shause e al., 2024).
Las ly, esea che s ha e ied o implemen mo e gami ica ion aspec s in o con-
join analysis. One p ime example is conjoin poke in oduced by Toubia e  al.
536
Ma ke ing Le e s (2025) 36:533–546
(2012). Fo an o e iew o di e en (adap i e) conjoin me hods, see, o example,
Sablo ny-Wacke shause e al. (2024) and S eine and Meißne (2018).
2.2 Incen i e alignmen
All conjoin analyses aim a p edic ing consume pu chases in he ma ke place
(Ding, 2007). I he ex ac ed p e e ences de ia e om ac ual p oduc choices due o
he hypo he ical esea ch aming (i.e., hypo he ical bias), esea che s and p ac i ion-
e s become skep ical o he me hod and i s esul s (Gilb ide e al., 2008). In hypo-
he ical se ings, esponden s a e no mo i a ed o exe he same cogni i e e o as
hey would in eal-li e si ua ions (Ding e al., 2005; Toubia e al., 2012). One way o
educe hypo he ical bias in conjoin analysis is o make he esponden s’ disbu se-
men dependen on hei choices (i.e., incen i e alignmen ), which should mo i a e
esponden s o answe u h ully (Ding e al., 2005). Figu e1 highligh s some o
incen i e alignmen ’s key e ec s on he esul s ob ained om conjoin analysis.
Fo example, Ding e  al. (2005) demons a e incen i e alignmen ’s e ec i e-
ness in inc easing p edic i e alidi y. In addi ion, esponden s a e mo e likely o
answe u h ully and mo e consis en ly in incen i e-aligned se ings (i.e., highe
scale; Hause e al., 2019). Finally, esponden s’ highe le el o u h ulness u he
e eals a highe p ice sensi i i y, less no el y-seeking beha io , and lowe likelihood
o adhe e o social no ms in incen i e-aligned se ings (see, e.g., Ding e al., 2005;
Yang e al., 2018).
2.3 P edic i e alidi y andi s e alua ion inconjoin analysis
This me a-analysis exclusi ely ocuses on p edic i e alidi y in e ms o p oduc
choice. E alua ions o p edic i e alidi y can be subdi ided in o in-sample and
ou -o -sample p ocedu es (see Fig. 2). While in he o me , esea che s use he
same esponden s o bo h u ili y es ima ion and alida ion, in he la e , he es i-
ma ion esul s a e used o p edic he decisions o di e en esponden s (e.g., p e-
dic ing ac ual ma ke igu es o choices o a alida ion sample; see, e.g., Gensle
e al., 2012; G een & S ini asan, 1990). Such an ou -o -sample alida ion is o en
Fig. 1 E ec s o incen i e alignmen on he esul s ob ained om conjoin analysis

537
Ma ke ing Le e s (2025) 36:533–546
no accessible and/o oo cos ly, hus, esea che s usually use in-sample alida ion
me hods. In-sample p edic i e alidi y can be assessed by benchma king p edic ed
choices wi h ma ke beha io o wi h choices in ixed alida ion asks ha a e usu-
ally he same o e e y esponden . These alida ion asks can be u he classi ied
in o asks o he same layou (i.e., in his me a-analysis, de ined as he same o -
ma and same numbe o al e na i es as hecalib a ion asks) o a di e en layou .
Bo h op ions ha e hei me i s. On he one hand, p esen ing alida ion asks in he
same layou as egula calib a ion asks ob usca es hei special ole om espond-
en s. On he o he hand, esea che s o en employ a di e en layou o p esen many
mo e op ions o esponden s as in egula calib a ion asks, which makes p edic ions
ha de (Sablo ny-Wacke shause e al., 2024).
This me a-analysis solely ocuses on he in-sample s and. Based on he eason-
ing abo e, we hypo hesize:
Hypo hesis 1 (H1): Incen i e alignmen inc eases conjoin analysis’ p edic i e
alidi y.
2.4 E ec o incen i e alignmen onp edic i e alidi y: mode a ing mechanisms
2.4.1 Type o incen i e mechanism
Ding e al. (2005) ha e implemen ed he di ec mechanism in CBC, in which espond-
en s ecei e one o hei choices as s udy disbu semen . In he di ec mechanism, all
possible p oduc combina ions need o be a ailable o s udy disbu semen (Dong
Fig. 2 An o e iew o po en ial ways o es ing p edic i e alidi y in a conjoin analysis
538
Ma ke ing Le e s (2025) 36:533–546
e al., 2010). In con as , he e a e indi ec mechanisms, he wo mos well-known o
which a e he RankO de and he WTP mechanisms. In he o me , a p ede ined lis
o p oduc s (unknown o he esponden ) is anked based on he esponden ’s u ili y
es ima es, and he esponden ecei es he highes - anked p oduc as a disbu semen
(Dong e al., 2010). In he la e , he WTP is de e mined based on he u ili y es ima es
and compa ed o a andomly d awn p ice (Becke Deg oo -Ma schak p ocedu e;
Becke e al., 1964; Ding, 2007). While a leas wo p oduc s mus be a ailable o he
RankO de mechanism, he indi ec mechanism equi es only one (Dong e al., 2010).
Rega ding comp ehensibili y, he payou mechanism is easy o unde s and o he
di ec mechanism (Ding e al., 2005), while i is a he opaque o he indi ec mecha-
nisms (Dong e al., 2010). F om he esponden ’s pe spec i e, con ol o e he inal
payou is educed as i emains unspeci ic. The e o e, we expec :
Hypo hesis 2 (H2): Using a di ec ( s. indi ec ) incen i e-aligned mechanism
mo e s ongly inc eases p edic i e alidi y.
2.4.2 Payou p obabili y
In incen i e-aligned condi ions, he payou p obabili y o en a ies depending on he p ice
o he s imuliand he numbe o esponden s in he s udy. Fo example, in one s udy by
Ding e al. (2005) on Chinese dinne op ions, he esea che s disbu sed e e y esponden .
This was di e en , o example, in he s udy by Hause e al. (2019, p. 1070) on sma -
wa ches, whe e he esea che s disbu sed 1 in 500 esponden s. Yang e al. (2018) show
ha se ing he payou p obabili y close o 1 (1 esembles an ac ual pu chase si ua ion)
inc eases esponden s’ e o . Hence H3 s a es:
Hypo hesis 3 (H3): An inc easing payou p obabili y inc eases he posi i e e ec s
o incen i e alignmen .
2.4.3 Adap i e e suss a ic conjoin me hod
In s a ic conjoin me hods (e.g., classical conjoin analysis o CBC), esponden s
may answe choice asks ha a e no ele an o hem. In con as , adap i e conjoin
me hods (e.g., adap i e conjoin analysis, ACA, o ACBC) use esponden s’ answe s
o ea ly asks o in o m he compila ion o new ones (see, e.g., Johnson & O me,
2007). The goal is o imp o e he p ecision o he pa ame e s (Sablo ny-Wacke -
shause e al., 2024), which should ul ima ely also bene i he p edic i e alidi y.
Sablo ny-Wacke shause e al. (2024) compa e bo h me hods (CBC s. ACBC) in an
incen i e-aligned se ing, in which incen i e-aligned ACBC p edic s be e . To es
whe he adap i e designs’ e ec on p edic i e alidi y can be u he enhanced by
incen i e alignmen , H4 s a es:
Hypo hesis 4 (H4): Incen i e alignmen posi i ely mode a es he inc ease in p e-
dic i e alidi y ob ained om he applica ion o adap i e ins ead o s a ic con-
join designs.
539
Ma ke ing Le e s (2025) 36:533–546
2.4.4 Layou o  alida ion ask
P edic ing alida ion asks wi h he same layou as he calib a ion asks is usually
easie gi en he common-me hod a iance and he se up o he alida ion asks (i.e.,
usually a lowe numbe o al e na i es; O me & Ch zan, 2021). The la e makes i
di icul o incen i e alignmen o e eal i s posi i e e ec on p edic i e alidi y. In
con as , esea che s design alida ion asks wi h a di e en layou o be e e lec
eal pu chase decisions and o gain mo e in o ma ion (i.e., di e en o ma and/o
mo e al e na i es; Ding, 2007; Pa k e al., 2008). Thus, H5 s a es:
Hypo hesis 5 (H5): Incen i e alignmen inc eases he p edic i e alidi y mo e
s ongly o alida ion asks wi h a di e en ( s. same) layou han he calib a ion
asks.
2.4.5 P oduc ype
Finally, we es whe he incen i e alignmen is mo e e ec i e depending on he ype
o p oduc assessed in he conjoin analysis. The e o e, we di ide he s imuli in o
du able, non-du able, and se ice goods.3
3 Da a and esea ch me hod
3.1 Da a collec ion
We pe o med a keywo d sea ch in he ollowing da abases: JSTOR, Web o Science,
Scopus, and P oQues . We also used backwa d snowballing in he a icles sc eened
o eligibili y. Finally, we con ac ed o he esea che s in he ield ia email and by
sending a newsle e o he Ad ances in Consume Resea ch Lis se (ACR-L).
Fo ou li e a u e sea ch, we used he ollowing keywo ds (“conjoin *” OR
“*cbc” OR “disc e e choice expe imen ” OR “DCE” OR “choice-based conjoin ”
OR “s a ed p e e ence”) AND (“p edic i e alidi y” OR “p ognos ic alidi y” OR
“hi a e” OR “hi p edic ion” OR “incen i e?align*”). "The Appendix p o ides
mo e de ails" o "All sea ch e ms a e lis ed in an addi ional Open Science F ame-
wo k (he ea e OSF) eposi o y, whe e we also p o ide he da a and he R analysis
sc ip (h ps:// os . io/ 9n ph/)". We included a icles published back o he yea 2000
and adhe ed o he PRISMA guidelines o documen ing he sc eening p ocess (see
Fig. A2 in he WA; Page e al., 2021).
3.2 Inclusion c i e ia
The me a-analysis includes a icles and wo king pape s ha assess he p edic-
i e alidi y o conjoin analysis and ul ill he ollowing c i e ia: (1) espond-
en s answe ed a leas one alida ion ask, (2) au ho s epo hi a e o ele an
3 Wewould like o hank an anonymous e iewe o his sugges ion.
540
Ma ke ing Le e s (2025) 36:533–546
pa ame e s o calcula e i , and (3) an own s udy was conduc ed. We ound 136
e ec sizes (35 pape s), o which wo e ec sizes we e excluded because we could
no ex ac in o ma ion on he mode a o s and con ol a iables, espec i ely (see
Appendix). Thus, he inal se includes 134 e ec sizes o 34 (wo king) pape s, wi h
a o al sample size o 12,980 esponden s, o which 4165 belong o incen i e-aligned
and 8815 o hypo he ical conjoin analysis.
3.3 Da a coding
Fi s , we ex ac ed in o ma ion abou he s udy design (incen i e-aligned o hypo-
he ical) and he conjoin me hod used, which we hen ca ego ized in o s a ic o
adap i e design. Fo he incen i e-aligned e ec sizes, we u he coded he mecha-
nism (di ec s. indi ec ), he lo e y (i.e., payou p obabili y), and he wo h o he
p ice esponden s could win. Second, we ex ac ed in o ma ion abou he alida ion
ask, namely, he ac ual hi a e, he layou (same as calib a ion choice asks o di -
e en ), and he numbe o al e na i es (i.e., chance le el o p edic ing co ec ly).
Thi d, we acqui ed in o ma ion on he s imuli hemsel es and classi ied i in o he
p oduc ca ego ies o du able, non-du able, o se ice goods. Finally, o he con-
ol a iables, we eco ded in o ma ion on he sample (s uden s s. mixed), he loca-
ion o he s udy (No h Ame ica s. o he ), and he publica ion yea . Table A1 in
he Appendix p o ides addi ional in o ma ion.All included e ec sizes a e lis ed on
OSF.The es is s a ed ea lie (see commen in "3.1 Da a collec ion").
3.4 Me a‑analy ical p ocedu e
The obse ed hi a e se ed as he dependen e ec size in all me a-analy ical
eg ession models. I is de ined as he pe cen age o co ec ly p edic ed choices in
he alida ion asks (e.g., Ding e al., 2005). Mo e speci ically, we used he loga-
i hm o he hi a e o simpli y in e p e a ion and o accoun o non-linea i y (see
Schmid & Bijmol , 2020). To assess he e ec o incen i e alignmen , we added he
condi ion (hypo he ical s. incen i e-aligned) as a mode a o (i.e., independen a i-
able in me a- eg essions). We also added he numbe o choice al e na i es in he
alida ion asks as a mode a o (see in o ma ion on he model below).4
Some e ec sizes a e nes ed because pape s epo mo e han one e ec size and
some s udies epo mul iple e ec sizes (i.e., hi a es) o he same esponden s bu
di e en alida ion asks (Glese & Olkin, 2009). To accoun o hese dependen-
cies, we un a mul i a ia e me a-analysis wi h andom e ec s o he s udy le el and
he e ec size le el (Knapp e al., 2017; Kons an opoulos, 2011). All analyses we e
implemen ed in R using he “me a o ” package (Viech baue , 2010). In addi ion, he
4 Since he in e ac ion e ec was insigni ican , we epo he main e ec s only. We use his app oach o
all epo ed models.