Incentive alignment in conjoint analysis: a meta-analysis on predictive validity

Schramm, Joshua Benjamin Article — Published Version Incentive alignment in conjoint analysis: a meta-analysis on predictive validity Marketing Letters Provided in Cooperation with: Springer Nature Suggested Citation: Schramm, Joshua Benjamin (2025) : Incentive alignment in conjoint analysis: a meta-analysis on predictive validity, Marketing Letters, ISSN 1573-059X, Springer US, New York, NY, Vol. 36, Iss. 3, pp. 533-546, https://doi.org/10.1007/s11002-025-09764-8 This Version is available at: https://hdl.handle.net/10419/330642 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. https://creativecommons.org/licenses/by/4.0/ Vol.:(0123456789) Marketing Letters (2025) 36:533–546 https://doi.org/10.1007/s11002-025-09764-8 ORIGINAL RESEARCH Incentive alignment inconjoint analysis: ameta‑analysis onpredictive validity JoshuaBenjaminSchramm1 Accepted: 9 January 2025 / Published online: 20 January 2025 © The Author(s) 2025 Abstract Conjoint analysis is a widely used method in market research for predicting consumer purchases, making predictive validity a central tenet. Conjoint analyses, however, are typically conducted in hypothetical settings, making them susceptible to hypothetical bias. One solution is incentive-aligning conjoint studies to trigger truthful answering behavior, thereby increasing the accuracy of predictions. However, despite incentive alignment’s conceptual appeal, practitioners rarely use it. One reason for this is the uncertainty of its effectiveness. This research systematically investigates the gains in predictive validity employing a meta-analysis of 134 effect sizes from 34 articles (N = 12,980). Incentive alignment increases the predictive validity (i.e., hit rate) by 12%, providing a significant increase in accuracy. In addition, its effectiveness is amplified when researching durable and service goods (vs. nondurable goods) and when the payoutprobability rises. In contrast to conventional wisdom, indirect (vs. direct) incentive procedures do not mitigate the positive effects on predictive validity. We hope to stimulate a rethink in practice to make more use of incentive alignment and help decide whether incentive alignment is worth the additional effort. Keywords Conjoint analysis· Hypothetical bias· Incentive alignment· Market research· Multivariate meta-analysis· Predictive validity 1 Introduction Conjoint analysis is a method for assessing preferences among product attributes and their corresponding levels (Park etal., 2008). It is widely used, for example, to define promising products (Gilbride etal., 2008), to measure willingness-to-pay * Joshua Benjamin Schramm [email protected] 1 Faculty ofEconomics andManagement, Otto von Guericke University Magdeburg, Magdeburg, Saxony-Anhalt, Germany 534 Marketing Letters (2025) 36:533–546 (WTP; Schmidt & Bijmolt, 2020), or to determine the importance of attributes in the decision-making process (Steiner & Meißner, 2018). Conjoint analysis’ ultimate goal is to predict consumers’ purchase decisions (e.g., Ding, 2007; Green & Srinivasan, 1990). Measuring preferences accurately is crucial because only products that match consumers’ true taste will be successful (Hauser etal., 2014). Practitioners and academics can choose from a wide variety of conjoint analyses (see, e.g., Steiner & Meißner, 2018). The most commonly applied conjoint analysis method is the choice-based conjoint (CBC; Sawtooth Software Inc., 2024) due to its similarity to actual choice behavior (Louviere & Woodworth, 1983). A CBC survey typically includes a sequence of choice tasks in which respondents choose the alternative they would most likely buy (Yang etal., 2018) and validation tasks (i.e., tasks excluded for utility estimation, also called holdout tasks).1 Validation tasks are important for assessing the validity of the respondents’ choices and the results of the conjoint analysis (see, e.g., Green & Srinivasan, 1990). Despite the widespread use of conjoint analysis, researchers express concerns that the results obtained may not reflect actual preferences (Gilbride etal., 2008), as conjoint analyses are routinely conducted in hypothetical settings (Ding etal., 2005; Pachali etal., 2023). In these settings, respondents may not exert the same cognitive effort as in an actual purchase situation (Toubia etal., 2012), which ultimately can lead to a discrepancy between stated and actual preferences (i.e., hypothetical bias; see, e.g., Ding, 2007; Hofstetter etal., 2021). To adequately address hypothetical bias, researchers have introduced incentive alignment to conjoint analysis. Instead of a fixed payment, as is common in market research, respondents’ study disbursement depends on their choices in the choice tasks (seee.g., Ding etal., 2005). Multiple studies have shown its effectiveness in conjunction with preference measurement techniques, such as (adaptive) CBC (see, e.g., Ding, 2007; Sablotny-Wackershauser etal., 2024). Haghani etal. (2021) and Schmidt and Bijmolt (2020) examine hypothetical bias and predictive validity in more detail. While Haghani etal. (2021) conducted a systematic literature review on hypothetical bias in different disciplines and show that hypothetical bias is an issue in 11 of the 18 studies examined in consumer economics, Schmidt and Bijmolt (2020) conducted a meta-analysis.2 The latter researchers show that hypothetical and actual WTP differ by 21% and that the effect on hypothetical bias is greater for indirect (vs. direct) methods such as conjoint analysis (Schmidt & Bijmolt, 2020). However, determining WTP is only one field of application of conjoint analysis, and price is not necessarily an included attribute in every market research problem. For example, researchers are sometimes not primarily interested in determining the optimal price but inthe choice shares of same-priced products. Therefore, the present meta-analysis aims to shed light on another form of predictive validity, namely, whether incentive alignment can improve the correct prediction of product decisions. 1 We use the term validation tasks for both validation tasks and holdout tasks. 2 Hypothetical bias was absent in only 3 of the 18 studies (Haghani etal., 2021). 535 Marketing Letters (2025) 36:533–546 This inquiry contributes to the literature in the following ways. First, we examine incentive alignment’s effectiveness in improving conjoint analysis’ predictive validity. Recently, Pachali etal. (2023, p. 970) showed that 96% of the conjoint studies carried out by market research companies are hypothetical. We demonstrate the potential benefits practitioners can gain by showing incentive alignment’s overall effect on predictive validity. Second, we provide researchers with an overview of what to expect when applying incentive alignment in conjoint analysis (e.g., average additional costs). Both may alleviate doubts about incentive alignment among practitioners. Third, we examine potential moderators that might influence the predictive validity improvement’s magnitude (e.g., payout probability). Our meta-analysis assesses the predictive validity in terms of the hit rate (i.e., the percentage of correctly predicted choices in validation tasks; Ding etal., 2005), as is common in the field (see, e.g., Wlömert & Eggers, 2016). Based on 134 effect sizes from 34 articles and a total of 12,980 respondents, we conclude that incentivealigning conjoint analysis increases the hit rate by 12%. 2 Theoretical background andhypotheses development 2.1 Conjoint analysis Conjoint analysis is one of the most frequently used market research methods (Pachali etal., 2023). For instance, market researchers apply this class of methods to measure WTP for products or specific product attributes (Schmidt & Bijmolt, 2020), or to assess the product attributes’ importance in the decision-making process (Steiner & Meißner, 2018). In its classical form, respondents either rate or rank different product profiles, or they decide on one profile in a paired comparison (Steiner & Meißner, 2018). Nowadays, CBC superseded these forms of conjoint analysis (Sawtooth Software Inc., 2024). CBC enjoys frequent usage due to its close mimics of real purchase decisions (Louviere & Woodworth, 1983). Respondents face a sequence of choice tasks in which they opt for the alternative they would most likely buy (Steiner & Meißner, 2018). To enhance realism, CBC often includes a no-buy alternative as another alternative in the choice task (see Fig. A1 in the Appendix for an exemplary task). Sometimes researchers include the no-buy alternative in a dualresponse logic instead, in which respondents answer a forced choice task first and then decide whether the chosen alternative is an actual purchase option (see, e.g., Brazell etal., 2006). CBC is a static method. Choice tasks do not change based on the respondent’s indicated preferences, which might result in irrelevant choices for some respondents (Sablotny-Wackershauser etal., 2024). To overcome this static nature, researchers have developed adaptive designs that use the respondent’s answers to previous choice tasks to create new ones (see, e.g.,Johnson & Orme, 2007). The most commonly used adaptive method is the adaptive CBC (ACBC; see, e.g., Johnson & Orme, 2007; Sablotny-Wackershauser etal., 2024). Lastly, researchers have tried to implement more gamification aspects into conjoint analysis. One prime example is conjoint poker introduced by Toubia et al. 536 Marketing Letters (2025) 36:533–546 (2012). For an overview of different (adaptive) conjoint methods, see, for example, Sablotny-Wackershauser etal. (2024) and Steiner and Meißner (2018). 2.2 Incentive alignment All conjoint analyses aim at predicting consumer purchases in the marketplace (Ding, 2007). If the extracted preferences deviate from actual product choices due to the hypothetical research framing (i.e., hypothetical bias), researchers and practitioners become skeptical of the method and its results (Gilbride etal., 2008). In hypothetical settings, respondents are not motivated to exert the same cognitive effort as they would in real-life situations (Ding etal., 2005; Toubia etal., 2012). One way to reduce hypothetical bias in conjoint analysis is to make the respondents’ disbursement dependent on their choices (i.e., incentive alignment), which should motivate respondents to answer truthfully (Ding etal., 2005). Figure1 highlights some of incentive alignment’s key effects on the results obtained from conjoint analysis. For example, Ding et al. (2005) demonstrate incentive alignment’s effectiveness in increasing predictive validity. In addition, respondents are more likely to answer truthfully and more consistently in incentive-aligned settings (i.e., higher scale; Hauser etal., 2019). Finally, respondents’ higher level of truthfulness further reveals a higher price sensitivity, less novelty-seeking behavior, and lower likelihood to adhere to social norms in incentive-aligned settings (see, e.g., Ding etal., 2005; Yang etal., 2018). 2.3 Predictive validity andits evaluation inconjoint analysis This meta-analysis exclusively focuses on predictive validity in terms of product choice. Evaluations of predictive validity can be subdivided into in-sample and out-of-sample procedures (see Fig. 2). While in the former, researchers use the same respondents for both utility estimation and validation, in the latter, the estimation results are used to predict the decisions of different respondents (e.g., predicting actual market figures or choices of a validation sample; see, e.g., Gensler etal., 2012; Green & Srinivasan, 1990). Such an out-of-sample validation is often Fig. 1 Effects of incentive alignment on the results obtained from conjoint analysis 537 Marketing Letters (2025) 36:533–546 not accessible and/or too costly, thus, researchers usually use in-sample validation methods. In-sample predictive validity can be assessed by benchmarking predicted choices with market behavior or with choices in fixed validation tasks that are usually the same for every respondent. These validation tasks can be further classified into tasks of the same layout (i.e., in this meta-analysis, defined as the same format and same number of alternatives as thecalibration tasks) or a different layout. Both options have their merits. On the one hand, presenting validation tasks in the same layout as regular calibration tasks obfuscates their special role from respondents. On the other hand, researchers often employ a different layout to present many more options to respondents as in regular calibration tasks, which makes predictions harder (Sablotny-Wackershauser etal., 2024). This meta-analysis solely focuses on the in-sample strand. Based on the reasoning above, we hypothesize: Hypothesis 1 (H1): Incentive alignment increases conjoint analysis’ predictive validity. 2.4 Effect ofincentive alignment onpredictive validity: moderating mechanisms 2.4.1 Type ofincentive mechanism Ding etal. (2005) have implemented the direct mechanism in CBC, in which respondents receive one of their choices as study disbursement. In the direct mechanism, all possible product combinations need to be available for study disbursement (Dong Fig. 2 An overview of potential ways of testing predictive validity in a conjoint analysis 538 Marketing Letters (2025) 36:533–546 etal., 2010). In contrast, there are indirect mechanisms, the two most well-known of which are the RankOrder and the WTP mechanisms. In the former, a predefined list of products (unknown to the respondent) is ranked based on the respondent’s utility estimates, and the respondent receives the highest-ranked product as a disbursement (Dong etal., 2010). In the latter, the WTP is determined based on the utility estimates and compared to a randomly drawn price (Becker Degroot-Marschak procedure; Becker etal., 1964; Ding, 2007). While at least two products must be available for the RankOrder mechanism, the indirect mechanism requires only one (Dong etal., 2010). Regarding comprehensibility, the payout mechanism is easy to understand for the direct mechanism (Ding etal., 2005), while it is rather opaque for the indirect mechanisms (Dong etal., 2010). From the respondent’s perspective, control over the final payout is reduced as it remains unspecific. Therefore, we expect: Hypothesis 2 (H2): Using a direct (vs. indirect) incentive-aligned mechanism more strongly increases predictive validity. 2.4.2 Payout probability In incentive-aligned conditions, the payout probability often varies depending on the price of the stimuliand the number of respondents in the study. For example, in one study by Ding etal. (2005) on Chinese dinner options, the researchers disbursed every respondent. This was different, for example, in the study by Hauser etal. (2019, p. 1070) on smartwatches, where the researchers disbursed 1 in 500 respondents. Yang etal. (2018) show that setting the payout probability closer to 1 (1 resembles an actual purchase situation) increases respondents’ effort. Hence H3 states: Hypothesis 3 (H3): An increasing payout probability increases the positive effects of incentive alignment. 2.4.3 Adaptive versusstatic conjoint method In static conjoint methods (e.g., classical conjoint analysis or CBC), respondents may answer choice tasks that are not relevant to them. In contrast, adaptive conjoint methods (e.g., adaptive conjoint analysis, ACA, or ACBC) use respondents’ answers to early tasks to inform the compilation of new ones (see, e.g., Johnson & Orme, 2007). The goal is to improve the precision of the parameters (Sablotny-Wackershauser etal., 2024), which should ultimately also benefit the predictive validity. Sablotny-Wackershauser etal. (2024) compare both methods (CBC vs. ACBC) in an incentive-aligned setting, in which incentive-aligned ACBC predicts better. To test whether adaptive designs’ effect on predictive validity can be further enhanced by incentive alignment, H4 states: Hypothesis 4 (H4): Incentive alignment positively moderates the increase in predictive validity obtained from the application of adaptive instead of static conjoint designs. 539 Marketing Letters (2025) 36:533–546 2.4.4 Layout ofvalidation task Predicting validation tasks with the same layout as the calibration tasks is usually easier given the common-method variance and the setup of the validation tasks (i.e., usually a lower number of alternatives; Orme & Chrzan, 2021). The latter makes it difficult for incentive alignment to reveal its positive effect on predictive validity. In contrast, researchers design validation tasks with a different layout to better reflect real purchase decisions and to gain more information (i.e., different format and/or more alternatives; Ding, 2007; Park etal., 2008). Thus, H5 states: Hypothesis 5 (H5): Incentive alignment increases the predictive validity more strongly for validation tasks with a different (vs. same) layout than the calibration tasks. 2.4.5 Product type Finally, we test whether incentive alignment is more effective depending on the type of product assessed in the conjoint analysis. Therefore, we divide the stimuli into durable, non-durable, and service goods.3 3 Data andresearch method 3.1 Data collection We performed a keyword search in the following databases: JSTOR, Web of Science, Scopus, and ProQuest. We also used backward snowballing in the articles screened for eligibility. Finally, we contacted other researchers in the field via email and by sending a newsletter to the Advances in Consumer Research Listserv (ACR-L). For our literature search, we used the following keywords (“conjoint*” OR “*cbc” OR “discrete choice experiment” OR “DCE” OR “choice-based conjoint” OR “stated preference”) AND (“predictive validity” OR “prognostic validity” OR “hit rate” OR “hit prediction” OR “incentive?align*”). "The Appendix provides more details" to "All search terms are listed in an additional Open Science Framework (hereafter OSF) repository, where we also provide the data and the R analysis script (https:// osf. io/ 9ntph/)". We included articles published back to the year 2000 and adhered to the PRISMA guidelines for documenting the screening process (see Fig. A2 in the WA; Page etal., 2021). 3.2 Inclusion criteria The meta-analysis includes articles and working papers that assess the predictive validity of conjoint analysis and fulfill the following criteria: (1) respondents answered at least one validation task, (2) authors report hit rate or relevant 3 Wewould like to thank an anonymous reviewer for this suggestion. 540 Marketing Letters (2025) 36:533–546 parameters to calculate it, and (3) an own study was conducted. We found 136 effect sizes (35 papers), of which two effect sizes were excluded because we could not extract information on the moderators and control variables, respectively (see Appendix). Thus, the final set includes 134 effect sizes of 34 (working) papers, with a total sample size of 12,980 respondents, of which 4165 belong to incentive-aligned and 8815 to hypothetical conjoint analysis. 3.3 Data coding First, we extracted information about the study design (incentive-aligned or hypothetical) and the conjoint method used, which we then categorized into static or adaptive design. For the incentive-aligned effect sizes, we furthercoded the mechanism (direct vs. indirect), the lottery (i.e., payout probability), and the worth of the price respondents could win. Second, we extracted information about the validation task, namely, the actual hit rate, the layout (same as calibration choice tasks or different), and the number of alternatives (i.e., chance level of predicting correctly). Third, we acquired information on the stimuli themselves and classified it into the product categories of durable, non-durable, or service goods. Finally, for the control variables, we recorded information on the sample (students vs. mixed), the location of the study (North America vs. other), and the publication year. Table A1 in the Appendix provides additional information.All included effect sizes are listed on OSF.The rest is stated earlier (see comment in "3.1 Data collection"). 3.4 Meta‑analytical procedure The observed hit rate served as the dependent effect size in all meta-analytical regression models. It is defined as the percentage of correctly predicted choices in the validation tasks (e.g., Ding etal., 2005). More specifically, we used the logarithm of the hit rate to simplify interpretation and to account for non-linearity (see Schmidt & Bijmolt, 2020). To assess the effect of incentive alignment, we added the condition (hypothetical vs. incentive-aligned) as a moderator (i.e., independent variable in meta-regressions). We also added the number of choice alternatives in the validation tasks as a moderator(see information on the model below).4 Some effect sizes are nested because papers report more than one effect size and some studies report multiple effect sizes (i.e., hit rates) for the same respondents but different validation tasks (Gleser & Olkin, 2009). To account for these dependencies, we run a multivariate meta-analysis with random effects for the study level and the effect size level (Knapp etal., 2017; Konstantopoulos, 2011). All analyses were implemented in R using the “metafor” package (Viechtbauer, 2010). In addition, the 4 Since the interaction effect was insignificant, we report the main effects only. We use this approach for all reported models.