Idea Evaluation for Solutions to Specialized Problems: Leveraging the Potential of Crowds and Large Language Models

Gimpel, Henner; Laubacher, Robert; Probost, Fabian; Schäfer, Ricarda; Schoch, Manfred Article — Published Version Idea Evaluation for Solutions to Specialized Problems: Leveraging the Potential of Crowds and Large Language Models Group Decision and Negotiation Provided in Cooperation with: Springer Nature Suggested Citation: Gimpel, Henner; Laubacher, Robert; Probost, Fabian; Schäfer, Ricarda; Schoch, Manfred (2025) : Idea Evaluation for Solutions to Specialized Problems: Leveraging the Potential of Crowds and Large Language Models, Group Decision and Negotiation, ISSN 1572-9907, Springer Netherlands, Dordrecht, Vol. 34, Iss. 4, pp. 903-932, https://doi.org/10.1007/s10726-025-09935-y This Version is available at: https://hdl.handle.net/10419/330757 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. https://creativecommons.org/licenses/by/4.0/ Vol.:(0123456789) Group Decision and Negotiation (2025) 34:903–932 https://doi.org/10.1007/s10726-025-09935-y Idea Evaluation forSolutions toSpecialized Problems: Leveraging thePotential ofCrowds andLarge Language Models HennerGimpel1,2,3· RobertLaubacher5· FabianProbost1,2,3· RicardaSchäfer1,4· ManfredSchoch1,3,6 Accepted: 21 May 2025 / Published online: 28 June 2025 © The Author(s) 2025 Abstract Complex problems such as climate change pose severe challenges to societies worldwide. To overcome these challenges, digital innovation contests have emerged as a promising tool for idea generation. However, assessing idea quality in innovation contests is becoming increasingly problematic in domains where specialized knowledge is needed. Traditionally, expert juries are responsible for idea evaluation in such contests. However, experts are a substantial bottleneck as they are often scarce and expensive. To assess whether expert juries could be replaced, we consider two approaches. We leverage crowdsourcing and a Large Language Model (LLM) to evaluate ideas, two approaches that are similar in terms of the aggregation of collective knowledge and could therefore be close to expert knowledge. We compare expert jury evaluations from innovation contests on climate change with crowdsourced and LLM’s evaluations and assess performance differences. Results indicate that crowds and LLMs have the ability to evaluate ideas in the complex problem domain while contest specialization—the degree to which a contest relates to a knowledge-intensive domain rather than a broad field of interest—is an inhibitor of crowd evaluation performance but does not influence the evaluation performance of LLMs. Our contribution lies with demonstrating that crowds and LLMs (as opposed to traditional expert juries) are suitable for idea evaluation and allows innovation contest operators to integrate the knowledge of crowds and LLMs to reduce the resource bottleneck of expert juries. Keywords Idea evaluation· Crowdsourcing· Large language model· Specialized knowledge Extended author information available on the last page of the article 904 H.Gimpel et al. 1 Introduction Complex problems such as climate change or social inequality are some of the most pressing challenges of our time. Complex problems share common characteristics: they are (from the current point of view) unique, and humankind has little experience with solving them (Rittel and Webber 1973). Even if solutions are implemented, it often takes years or decades until the solution quality becomes visible and measurable (Gimpel etal. 2020). Thus, complex problems pose great questions to humanity and require sophisticated problem-solving abilities, in particular, involving a broad range of affected stakeholders (Head 2008). Online innovation contests are one tool used to foster innovative problem-solving by offering a platform that brings together people with different knowledge (Chesbrough 2006). Such platforms often address complex problems and can contain contests for different sub-problems. These sub-problems are different in their level of specialization and thus require different levels of expertise to solve them. We define the term specialization as the degree to which a contest’s topic belongs to one particular, knowledge-intensive domain rather than a broad field of interest. The ease of accessing online innovation contests usually leads to an abundance of submitted ideas, resulting in the challenge of idea evaluation (Blohm et al. 2013). There is vast agreement that groups of expert juries are the most suitable decision makers in the absence of an objectively known solution quality (Klein and Garcia 2015; Blohm etal. 2016; Görzen and Kundisch 2016; Nagar etal. 2016). In this context, decision maker refers to the instance that evaluates the idea. However, the resource of expert juries is scarce and expensive, creating a severe trade-off between efficiency and quality of evaluations. This bottleneck is one of the central challenges of using innovation contests to solve complex problems (Nagar etal. 2016). In this paper, we therefore focus on the part of the decision-making process that deals with the evaluation of the ideas. Specifically, we study two promising alternatives to expert juries in the evaluation process, namely evaluation by crowds and Large Language Models (LLMs). These two approaches have been studied together since the emergence of LLMs, as both are similar in terms of the aggregation of collective knowledge. Crowdworking for idea evaluation is an approach that has gained recognition from researchers and practitioners (Oosterman etal. 2014; Klein and Garcia 2015; Görzen and Kundisch 2016; Magnusson etal. 2016; Mollick and Nanda 2016; Wimbauer etal. 2019). Crowds refer to the general public (or selected communities thereof) but do not require participants to possess specific skills or expertise, in contrast to expert juries. The term “wisdom of the crowd” describes the phenomenon where the average of seemingly uninformed individual opinions can lead to collective intelligence when they work together as a group (Mollick and Nanda 2016). Existing research compares crowd and expert idea evaluations in mainly corporate contexts and finds mixed results (Magnusson etal. 2016; Wimbauer etal. 905 Idea Evaluation forSolutions toSpecialized Problems:… 2019). Yet, most corporate idea evaluations do not belong to the class of complex problems for two reasons: (1) idea quality can often be determined quickly in corporate environments and (2) specialized knowledge is often not required as the specialization of corporate problems typically is low (Magnusson etal. 2016; Wimbauer etal. 2019). So far, research has not investigated the potential of crowdsourced idea evaluations in contexts with intransparent solution quality and varying specialization. Coming to the second approach, the evaluation task can be automated entirely by artificial intelligence (AI), for example, machine learning models (Nagar etal. 2016). For instance, Nagar etal. (2016) offer a computational approach where a machine learning classifier analyzes different characteristics of texts to assess ideas. However, the decisions of these models are based on various parameters (such as the length of the text) and are not based on knowledge about the content of the texts to be evaluated. Currently, a promising AI-based approach for evaluation are LLMs that gained significant recognition with the release of ChatGPT at the end of 2022, particularly given that they are capable of generating output of a higher quality than previous machine learning models in certain tasks (Gao etal. 2023; Wang etal. 2023). Initial research on the capabilities of these models focuses on simple evaluation tasks such as stance detection or emotion recognition (Kocoń etal. 2023; Zhang etal. 2023). Some of these studies show that LLMs also have the ability to execute various tasks for which they are not initially trained (Kocmi and Federmann 2023). Due to this ability, LLMs may be promising for idea evaluation. However, previous research on LLMs’ capabilities incorporates neither complex problems nor specialized knowledge in detail. For applying crowdsourcing or LLMs in innovation contests for complex problems, assessing the impact of specialization on idea evaluation quality is urgently needed, as complex problems are specialized by definition, and specialization could push decision makers to their limits as they are unfamiliar with such topics. We need to understand better which decision maker can be used best in which setting in the presence of specialization. Therefore, we address the following research question: What is the potential of crowdsourcing and LLMs for evaluating solution ideas for complex problems in the presence of specialization? To answer this question, we compared crowd and LLM evaluations of 104 ideas from four treatment groups with different levels of specialization hosted by the platform MIT Climate CoLab with expert evaluations. Our results show that depending on the specialization crowds and LLMs have the ability to evaluate ideas in the complex problem domain. Contest specialization is a key inhibitor of crowd performance that we can counteract by adapting the evaluation procedure. In contrast, we find that LLMs can better handle different degrees of specialization and yield higher performance when using ranking tasks. For practice, we recommend that for replacing expert juries employing LLMs rather than a crowd might be appropriate. 906 H.Gimpel et al. 2 Theoretical Background 2.1 Evaluating Solution Ideas forComplex Problems Complex problems such as climate change, often also referred to as “wicked problems” are described as “complex, involving multiple possible causes and internal dynamics that could not [be] assumed to be linear, and have very negative consequences for society if not addressed properly” (Peters 2017). There is no clear path to solving these problems. However, a few principles guide the continuing efforts: (1) the need for coordination across locations, (2) the need to involve stakeholders from all types of interest groups, and (3) the need for a high degree of specialized knowledge (Stern 2006; Head 2008; Karvonen and Brand 2009). Complex problems can be addressed through ideas and innovation—the central factor for solving problems (von Hippel 1994). Solving complex problems is a decision-making problem as it requires both the generation of multiple alternative ideas and the selection of the best solution under uncertainty and limited resources (Simon 1960). The conceptualization of the decision-making process developed by Simon (1960) divides it into three phases: intelligence, design and choice. First, the problem is analyzed, and relevant information is gathered (‘intelligence’). Then potential solutions are developed (‘design’) and finally the best option or the best options are selected (‘choice’). To facilitate idea generation, which encompasses both the systematic analysis of the problem (‘intelligence’) and the creative development of solutions (‘design’), the concept of IT-enabled idea generation through open innovation contests has emerged (Han et al. 2020). These contests provide a structured approach to collect and refine ideas from diverse contributors. With the often abundant ideas generated in innovation contests, the challenge of effectively evaluating and processing them arises (‘choice’) (Blohm etal. 2013; Özaygen and Balagué 2018). This challenge becomes even more demanding when considering the characteristics of complex problems. In order to evaluate solution ideas for complex problems successfully, decision makers rely on their own experience and knowledge to make a qualified decision (Gerlach etal. 2019). Making this decision in highly specialized areas requires deep knowledge of specific domains (Fischer etal. 2012). To approach idea evaluation in the light of complex problems, we refer to knowledge regarding idea evaluation in general. Researchers studied the evaluation procedure, with absolute rating and relative ranking being two of the bestknown approaches. In absolute rating, the decision maker rates alternatives independently of each other, for example, on a Likert scale. In contrast, relative ranking describes decision makers’ direct hierarchical arrangement of alternatives (Ovadia 2004). Besides the evaluation procedures, researchers studied the different types of decision makers (e.g., expert juries, external crowds, and various technical solutions) (Klein and Garcia 2015; Blohm etal. 2016; Nagar etal. 2016). Prior research suggests that expert juries generally outperform other 907 Idea Evaluation forSolutions toSpecialized Problems:… decision makers and, thus, have been considered closest to the “ground truth” (Klein and Garcia 2015; Blohm et al. 2016; Nagar et al. 2016). In contrast, crowds and technical solutions are more costand time-efficient than expert juries (Vukicevic etal. 2022). Consequently, idea evaluation is characterized by a tradeoff between efficiency and quality. Therefore, to leverage the potential of innovation contests, the possibility of replacing expert juries with crowds or LLMs in a way that quality is not impaired has gained interest in prior decision-making literature (Klein and Garcia 2015; Görzen and Kundisch 2016; Magnusson etal. 2016; Mollick and Nanda 2016; Wimbauer etal. 2019; Gao etal. 2023; Peres etal. 2023; Wang etal. 2023). 2.2 Crowd‑based Idea Evaluation Crowdsourcing describes open calls for contributions for selected activities to benefit from human collective intelligence and allows for integrating a diverse crowd with different specialized knowledge, backgrounds, and experiences (Howe 2006). When a group of people, i.e. a crowd, evaluates ideas, the value of the work of the individual crowdworkers emerges in the aggregation of all contributions together. Each individual contribution of a crowdworker contributes to the creation of a larger, emergent value that is only realized through the totality and aggregation of all contributions (Geiger etal. 2012). Several studies analyzed replacing expert juries with crowdsourced idea evaluations, for example, in corporate innovation contests (Klein and Garcia 2015; Görzen and Kundisch 2016; Magnusson etal. 2016; Wimbauer etal. 2019) or evaluating projects in the arts industry (Oosterman etal. 2014; Mollick and Nanda 2016). Generally, the results regarding crowd performance are mixed. Oosterman etal. (2014) see significantly better performance in expert juries on an image annotation task. On the contrary, several studies see congruence between expert and crowd evaluations and, thus, see high potential in crowd evaluations—at least for specific domains (Magnusson etal. 2016; Mollick and Nanda 2016; Wimbauer etal. 2019). Groups tend to be more efficient for complex tasks than individuals (Almaatouq etal. 2020). To adequately address crowd evaluations and find generalizable results, an understanding of the underlying evaluation process is necessary. In sum, existing research established crowd evaluations as a promising way to evaluate ideas (Magnusson etal. 2016). However, research has not yet investigated the potential of crowd evaluations under the consideration of contest specialization. While specialized knowledge has been mentioned as an essential factor for idea evaluation (Görzen and Kundisch 2016), understanding its impact on crowd performance is still needed. 2.3 LLM‑based Idea Evaluation LLMs are computational models designed to analyze and generate text (Susarla etal. 2023). LLMs may rely on the Generative Pre-trained Transformer (GPT) architecture, known for its “attention mechanism” (Vaswani etal. 2017). The attention 908 H.Gimpel et al. mechanism enables the model to learn different positions of words in a sentence during the processing of input sequences by assigning individual weights to each element in the input, facilitating strong contextualization (Wolfram 2023). Such models are pre-trained on extensive text datasets, creating a deep representation of the semantic structure of natural language. While LLMs yield fascinating results that are well-documented, the knowledge embedded in such models is broad and not task-specific. However, LLMs can be fine-tuned to specific tasks or domains (Ray 2023) and might be coupled with other knowledge sources such as knowledge graphs. Several empirical studies have been conducted to investigate the capabilities of LLMs (Kocoń etal. 2023; Zhang etal. 2023). For instance, Kocoń etal. (2023) investigated an LLM’s capabilities on 25 different Natural Language Processing (NLP) tasks, such as sentiment analysis, emotion recognition, and stance detection. Their study compares the LLM with the best existing task-specific automation solution. They showed that the LLM, on average, performed 25% worse and concluded that the LLM can only cope with specific tasks to a limited extent. In contrast, Zhang etal. (2023) examine LLMs’ capabilities in stance detection. In such tasks, a subject’s standpoint (for, against, or neither) to a claim in a text is analyzed. The study found that the LLM can keep up with the performance of task-specific stateof-the-art solutions. In addition, several studies have been published showing that LLMs have expert level knowledge in domains such as ophthalmology, law or operations management (Terwiesch 2023; Martin etal. 2024; Thirunavukarasu etal. 2024). For example, Martin etal. (2024) benchmark LLMs against a ground truth provided by senior lawyers in the task of contract review. Their findings indicate that LLMs match or surpass human accuracy in identifying legal issues while being significantly faster and more cost-effective. These studies demonstrate that LLMs are already capable of performing many tasks in knowledge work more accurately, efficiently, and cost-effectively than humans. Building on these findings, we aim to explore whether LLMs can also make decisions in more complex and unstructured domains. Most published studies have been in the zero-shot range, describing an LLM’s ability to execute a new task without specific training (Ray 2023). As existing studies demonstrated good LLM performance in zero-shot tasks, there are first indications that LLMs possess zero-shot qualities across domains (Kocmi and Federmann 2023; Kocoń etal. 2023). In addition to zero-shot prompting, more complex prompting strategies, such as few-shot prompting and chain of thought prompting, can increase the quality of LLM output (Kocmi and Federmann 2023). Even though the examples above do not directly relate to the realm of complex problem idea evaluation, they hint at LLMs’ potential in this area. First, they show that LLMs can succeed in zero-shot tasks. Second, the NLP skills examined in these studies include both Natural Language Understanding and Natural Language Generation. These skills are required when evaluating solution ideas to understand the idea itself and to generate an evaluation. Yet, there is also research on LLMs’ ability to evaluate text (Gao etal. 2023; Kocmi and Federmann 2023; Wang etal. 2023). Prompting an LLM to rate the quality of text summaries from different datasets showed that an LLM can evaluate text and, in some instances, is closer to human 909 Idea Evaluation forSolutions toSpecialized Problems:… expert juries than previous technological evaluation models (Gao etal. 2023). While this represents an important step towards leveraging LLMs for evaluation tasks, evaluating solutions for complex problems requires additional specialized domain knowledge (Karvonen and Brand 2009). Investigating the potential of LLMs in this regard is essential because it may help close one of the most severe bottlenecks of innovation contests. 3 Hypothesis Development We investigate whether crowds or LLMs can replace expert juries in idea evaluation for complex problems. Therefore, crowd and LLM performance is assessed as a measure of how close the evaluation is to the expert jury’s decisions. An expert jury’s decision is not an objective measure of idea quality. However, in the absence of such a measure, the experts’ assessment is as close to a ground truth as one can get for the context of our study (Klein and Garcia 2015; Blohm etal. 2016). Our interest lies in analyzing the effect of contest specialization and the evaluation procedure under which each decision maker can be utilized most. 3.1 Effect ofSpecialization The type of task decision makers face strongly impacts their evaluation performance (Poole etal. 1985). Contest specialization is a relevant task characteristic because many topics addressed in innovation contests for complex problems are highly specialized and, thus, of little familiarity to a general crowd. For human decision makers, little familiarity with a topic indicates that individuals cannot use long-term memory resources to tackle a task and, thus, are bound to the restraints of the working memory (Kalyuga and Singh 2016). Thus, high contest specialization increases the evaluation task’s complexity (Song and Bruning 2016). Consequently, higher complexity leads to worse performance by crowdworkers on a given task (Maynard and Hakel 1997). This is caused by high complexity causing a mismatch between available information and task requirements in relation to one’s available processing capacity (Cheng etal. 2020). We thus hypothesize: H1a For absolute rating, crowd evaluation performance is lower for innovation contests of higher specialization. As seen in the H1a, our default for examining specialization is the absolute rating of ideas. This approach is also the default for LLM assessments. Generally, LLMs should be able to evaluate due to their ability in zero-shot range (Gao etal. 2023; Kocmi and Federmann 2023; Wang etal. 2023). In addition, previous research also points to the ability to evaluate. Research is less clear regarding the relationship between contest specialization and the evaluation performance of LLMs. LLMs are trained on a large corpus of training data from a broad range of topics and, thus, possess a wide-ranging knowledge base that includes specialized knowledge in many 910 H.Gimpel et al. different domains (Feuerriegel etal. 2023; Ray 2023). With regard to complex tasks, the transformer architecture of LLMs enables them to process complex contexts through different weightings in the input—the attention mechanism (Vaswani etal. 2017). This is critical to an LLM’s ability to answer specialized questions appropriately by considering the nuances of a specialized domain. Consequently, LLMs are probably familiar with highly specialized topics, and their evaluation performance does not depend on specialization levels. We thus hypothesize: H1b For absolute rating, LLM evaluation performance is not impacted by contest specialization. 3.2 Interaction Effect ofComputed Ranking andSpecialization Hypothesizing the negative effect of specialization on crowd performance (H1a), the question of how to counteract this effect emerges. One possibility lies in specifically adapting the evaluation procedure to account for the challenges associated with high contest specialization. Absolute rating performance, as argued in the derivation of H1a, likely decreases with higher specialization. However, even if a decision maker evaluates several ideas independently on an absolute scale, the previous evaluations will subconsciously impact the subsequent evaluation scores (Mussweiler and Englich 2005). In the field of decision-making, this effect is called anchoring or anchoring bias (Tversky and Kahneman 1974), which, in our case, refers to the fact that the first proposals‘ evaluations influence the evaluation of subsequent proposals. The resulting subliminal comparison between proposals creates an implicit anchor that helps to decide what constitutes a good or a bad idea. By arranging the absolute evaluation scores from highest to lowest and making decisions based solely on the ranking position rather than the magnitude of the scores, a computed ranking is formed. This computed ranking allows for more consistent evaluations, as the individual evaluations are seen in a context of relative positions. We thus hypothesize: H2a Computed ranking reduces the negative impact of specialization on crowd performance as compared to absolute rating. LLMs are not explicitly developed for evaluation but rather conduct the evaluation based on their broad language-based training data (Ray 2023). These data included in the LLM provide an implicit anchor with which the ideas can be compared. When computing a ranking of the absolute ratings, differences between the individual proposals are presented in a more differentiated way, as they are not only considered in relation to the training data but also in relation to each other. Thus, a computed ranking should increase performance across tasks of every specialization. In line with our assumption in H1b, which posits that the evaluation performance is equal for varying degrees of specialization, we continue to not assume an interaction effect of computed ranking and specialization: 917 Idea Evaluation forSolutions toSpecialized Problems:… homogeneity of the participants concerning gender, age, education, and profession across the four groups to ensure comparability. For discrete variables, we use χ2 tests for homogeneity, and for continuous variables, we use ANOVA (Hair etal. 2010). Based on those tests, participants do not differ significantly between the groups (5% significance level). Next, we present descriptive statistics for the evaluations of the LLM. Figure2 illustrates the distribution of the evaluation scores for each proposal across the four groups. The boxplots show the distribution of the 30 evaluation scores for each proposal. The black line represents the median, the black dot represents the mean, and the white dots show outliers. We can observe a strikingly low variance in the evaluation scores in the boxplots, with mean and median values clustered around 3.5. Thus, we conducted an ANOVA to find out whether significant differences exist between the mean evaluation scores of the proposals within each group. We conducted the ANOVA for each treatment group with 26 groups (since each group contains 26 proposals). Group 1 and 2 exhibit significant differences in their evaluation scores (p-value < 0.05). Group 3 and 4 do not show significant differences in their evaluation scores, as their p-values are > 0.05. However, the post-hoc Tukey test (Abdi and Williams 2010) shows significant differences between several proposals with regard to the mean evaluation scores for all four groups. This enables an interpretable ranking of proposals, demonstrating that the LLM does not rate all proposals equally. 5.2 Hypotheses Tests Table2 presents crowd and LLM performance measurements for each group’s absolute rating, computed ranking, and relative ranking. Numerically, balanced accuracy is best for each group when using the LLM’s computed ranking. Likewise, PPV is best for each group when using the LLM’s computed ranking. For NPV, different combinations of evaluation metric and decision maker perform best. To test the hypotheses, we use two statistical tests. First, we use Fisher’s exact test to see if the performance in the groups changes significantly by changing the evaluation procedure. We conduct the test with two evaluation procedures as the rows of the 2 × 2 matrix, and proposals ranked identical as well as proposals ranked differently than the expert jury as its columns (Upton 1992). Second, we use a χ2 test for independence to see if there is a significant effect of specialization. The test is Fig. 2 Boxplots of the Evaluation Scores of the LLM for each of the 104 Proposals 918 H.Gimpel et al. performed in a 4 × 2 matrix with the four groups of one evaluation procedure as the rows. The two columns indicate whether a proposal was evaluated as identical or different from the expert jury as the second variable (McHugh 2013). 5.2.1 Effect ofSpecialization onAbsolute Rating Performance (H1a andH1b) Assessing balanced accuracy values for an absolute rating of the crowd, we see a performance decline from the least specialized group 1 (79%) to the most specialized group 4 (50%). In group 1, the crowd rated ideas rather similar to the expert jury. With rising group specialization, the crowd’s proposal evaluations differ more strongly from the experts. The resulting p-value of a χ2 test for independence is 0.015, indicating that the groups’ observed performance is significantly different between at least two of the groups. A pairwise comparison suggests that this is due to significantly higher performance in group 1 compared to the three other groups. For absolute ratings, advance classifications by the crowd were more reliable for the little specialized groups as PPV decreases with group specialization. NPV values are at 100% for groups 1 and 2, indicating that all proposals the crowd rated negatively were also rated the same way by the expert jury. The crowd rated every proposal as advance for the two more specialized groups, which also explains the low PPV in those groups (not applicable (N/A) for NPV). Balanced accuracy and NPV both support H1a. The lower scores for PPV as compared to NPV indicate that the crowd a) tends to rate proposals too leniently and marks too many proposals as Table 2 Performance Measurements for Absolute Rating, Computed Ranking and Relative Ranking with ascending Specialization from Group 1 (G1) to Group 4 (G4) Balanced Accuracy PPVNPV Absolute Rating Compute d Ranking Relative Ranking █= Crowd █= LLM 79% 56% 50% 50% 50% 50% 50% 50% 0% 20% 40% 60% 80% 100% G1 G2 G3 G4 74% 38% 50% 42% 54% 35% 50% 42% 0% 20% 40% 60% 80% 100% G1 G2 G3 G4 100% 100% 0% 20% 40% 60% 80% 100% G1 G2 G3 G4 77% 66% 54% 69% 80% 74% 69% 76% 0% 20% 40% 60% 80% 100% G1 G2 G3 G4 79% 56% 54% 64% 80% 67% 69% 73% 0% 20% 40% 60% 80% 100% G1 G2 G3 G4 75% 76% 54% 73% 82% 82% 69% 80% 0% 20% 40% 60% 80% 100% G1 G2 G3 G4 49% 62% 69% 74% 62% 53% 0% 20% 40% 60% 80% 100% G1 G2 G3 G4 33% 62% 71% 67% 62% 45% 0% 20% 40% 60% 80% 100% G1 G2 G3 G4 65% 62% 67% 82% 62% 60% 0% 20% 40% 60% 80% 100% G1 G2 G3 G4 919 Idea Evaluation forSolutions toSpecialized Problems:… advance, especially in highly specialized groups. Also, b) the crowd is more reliable in their negative judgments. Analyzing the balanced accuracy values of the LLM for absolute rating, we see a more consistent picture. Across specialization levels, we can observe a balanced accuracy of 50%. As the performance is constant, we conclude that it is independent of specialization, supporting H1b. NPV values for the LLM all show N/A. This results from the LLM marking every single proposal as advance. The PPV values are also influenced by the advancement of all proposals. Like the crowd, we observe that too many proposals are evaluated as advance. For absolute rating, LLM performance is rather poor, independent from group specialization. 5.2.2 Interaction Effect ofComputed Ranking andSpecialization (H2a andH2b) The second set of hypotheses concerns the computed ranking process. For the crowd, we see a substantial performance increase for balanced accuracy with a delta of up to 19% (group 4) when shifting from absolute rating to computed ranking. Robust across all four groups, PPV values increase between 4% (group 3) and 22% (group 4) compared to absolute rating. In computed ranking, the crowd’s advance classifications become more reliable. For the less specialized groups, NPV performance decreases as the crowd rates more ideas as not advance with a few of them being false classifications. We conduct Fisher’s exact test between absolute rating and computed ranking. Performance does not differ significantly between absolute rating and computed ranking (p-values from group 1 to 4: 1.000, 0.093, 1.000, 0.093). Next, we assess the interaction effect of computed ranking and specialization. For the crowd, the impact of computed ranking on balanced accuracy is most minor for group 1 (−2%) and largest for group 4 (+ 19%). Accordingly, with higher specialization, the effect of computed ranking on crowd performance increases (χ2 p = 0.345), indicating that the performance is not significantly different between the groups. Consequently, by changing the evaluation procedure, the specialization effect disappears, which supports H2a. When comparing absolute rating and computed ranking for the LLM, a clear increase in performance can be seen across all groups (delta between 19% for group 3 up to 30% for group 1). Fisher’s exact test indicates that the distribution of equal and different ratings does differ significantly between absolute rating and computed ranking (p = 0.075 for group 1, p = 0.004 for group 2, p = 0.023 for group 4, with one outlier in group 3: p = 0.258). This increase is primarily because the GPT model allows all proposals to advance in the absolute rating and is more nuanced in the computed ranking. PPV and NPV also show consistently high values across specialization levels. NPV values are the same or slightly higher than PPV (delta between 0% for group 3 up to 15% for group 2). Thus, the LLM is more reliable in identifying bad proposals. Balanced accuracy for computed ranking is in a rather narrow range between 69% (group 3) and 81% (group 1) and not significantly different (χ2 p = 0.801) across the four groups. Overall, this means computed ranking increases performance across all groups but does not create an effect of specialization on performance, which supports H2b. 920 H.Gimpel et al. 5.2.3 Interaction Effect ofRelative Ranking andSpecialization (H2c andH2d) The last set of hypotheses concerns the relative ranking procedure. For the crowd, balanced accuracy for the relative ranking is 49% for group 2 and 62% for group 3. Since NPV is equal to or higher than PPV, the crowd is also more reliable in identifying poor proposals than good proposals in the relative ranking. The p-values of Fisher’s exact test between computed ranking and relative ranking are 0.393 for group 2 and 0.779 for group 3, showing that the distribution of equal and different ratings does not differ significantly. The resulting p-value of a χ2 test for independence of 0.779 shows that the observed performance is not significantly different between the groups. Thus, as our results show no further increase in performance, but the specialization effect is also no longer present, H2c is supported. For the LLM, balanced accuracy for the relative ranking ranges from 53% (group 4) to 74% (group 2). Analogous to the computed ranking, the NPV values are, on average, better than the PPV values. This means the LLM is also better at identifying poor proposals when applying relative ranking. The difference in balanced accuracy between computed and relative ranking is smallest for group 2 (0%) and highest for group 4 (-23%). The p-values of Fisher’s exact test between computed ranking and relative ranking show that the distribution of equal and different ratings does not differ significantly (p-values from group 1 to 4: 0.532, 1.000, 0.771, 0.144). To examine the specialization effect, we conduct a χ2 test for independence. The resulting p-value of 0.334 indicates that the groups’ observed performance is not significantly different between the groups. Hence, the LLM’s rating performance with relative ranking does not depend on specialization, which supports H2d. Table3 summarizes our hypotheses and the respective empirical results. Table 3 Research Hypotheses Overview and Results # Description Empirical Results H1a For absolute rating, crowd evaluation performance is lower for innovation contests of higher specialization Supported H1b For absolute rating, LLM evaluation performance is not impacted by contest specialization Supported H2a Computed ranking reduces the negative impact of specialization on crowd performance as compared to absolute rating Supported H2b The computed ranking does not impact the effect of specialization on LLM performance Supported H2c Relative ranking reduces the negative impact of specialization on crowd performance compared to computed ranking Supported H2d The relative ranking does not impact the effect of specialization on LLM performance Supported 921 Idea Evaluation forSolutions toSpecialized Problems:… 5.3 Analysis ofDifferentiated Proposal Evaluations To better understand the decision makers’ reasons for advancing or not advancing a proposal, we perform a logistic regression of the relationship between the independent variables novelty, feasibility, impact, and presentation (measured on a 4-point Likert scale) and the binary dependent variable (0 for not advance, 1 for advance) (Winship and Mare 1984). As the computed ranking leads to the highest accuracy values, we focus on this evaluation procedure. The assumptions for the logistic regression, particularly the multicollinearity between the independent variables, were checked. We fit 15 regression models in total: For each of the three decision makers, we fit one model per group and an additional overarching model integrating all data. Exponentiated coefficients are used for interpretation, indicating a change in odds (for assessing proposals as advance) when increasing the respective independent variable by one unit. McFadden’s Pseudo R2 is computed as a goodness-of-fit measure, where values around 0.3 represent strong model fit (McFadden 1979). Table4 presents the results. McFadden’s Pseudo R2 values are high (between 0.347 and 0.694) for the crowd and jury, indicating strong predictability of the decision makers’ decision based on the four dimensions. The LLM values are lower (between 0.066 and 0.267), indicating only a moderate fit of the logistic regression model. For regressions with all groups, all four dimensions significantly and positively impact the decision to classify a proposal as advance. Consequently, for the expert jury, the crowd, and the LLM, a higher evaluation in the individual dimensions leads to a higher probability of advancing the proposal. For the crowd, presentation impacts the decision substantially more than the other dimensions. When looking at the values for all groups of the LLM, presentation, and impact have a high impact, Table 4 Logistic Regressions *p < 0.05; **p < 0.01 All Group 1 Group 2 Group 3 Group 4 Crowd Novelty 1.66** 2.02** 3.20** 2.17* 1.05 Feasibility 1.67** 1.86* 0.68 0.73 4.17** Impact 1.69** 2.11** 1.81 0.88 2.56* Presentation 4.80** 3.89** 5.30** 8.35** 11.47** McFadden’s Pseudo R20.371 0.368 0.35 0.347 0.598 LLM Novelty 1.74** 2.07** 1.53** 1.61** 1.47* Feasibility 1.57** 1.49 1.05 3.85 3.8** Impact 3.48** 3.18** 2.93** 2.88** 11.50** Presentation 4.85** 5.35** 2.56** 4.95** 6.42** McFadden’s Pseudo R20.145 0.169 0.066 0.147 0.267 Expert Jury Novelty 3.54** 5.8* 2.05 5.31* 5.02 Feasibility 2.59** 10.10** 2.41 1.61 1.12 Impact 3.11** 9.58** 2.73* 3.90* 1.03** Presentation 4.20** 3.65* 3.36* 2.99 3.35* McFadden’s Pseudo R20.515 0.694 0.445 0.468 0.614 922 H.Gimpel et al. novelty, and feasibility, a minor impact. In contrast, the expert jury’s regression coefficients are more balanced. Analyzing the individual group regression, an effect of specialization can be found for the crowd’s evaluation behavior. The more specialized a group is, the more strongly the crowd bases their decision on the proposal presentation rather than impact, novelty, or feasibility. No specialization effects can be identified for the expert jury and the LLM. 6 Discussion 6.1 Theoretical Implications Idea evaluation by expert juries is a severe bottleneck in innovation contests, as temporal and financial resources are often scarce. Consequently, we investigate the potential of crowd and LLM evaluations to relieve expert juries of the burden of evaluating solutions to complex problems. Existing studies on crowd evaluations focused on specific idea domains mainly in a corporate setting, and their results regarding crowd performance were mixed (Klein and Garcia 2015; Görzen and Kundisch 2016). Mollick and Nanda (2016) highlight the importance of better understanding the circumstances under which the crowd is a suitable decision maker. Our study provides evidence that contest specialization is a key inhibitor of crowd performance. Accordingly, the crowd performs poorly in the evaluation of highly specialized topics. However, less specialized topics are quite suitable for the crowd, as the crowd is more familiar with these topics and can use their long-term memory resources. Our findings on lenient crowd evaluations and overscoring further indicate that crowdworkers are hesitant to rate a proposal negatively. This effect is particularly strong with high contest specialization, where the presentation of a proposal appears to inappropriately outweigh other relevant characteristics. The crowd likely struggles to reject ideas when they lack the specialized knowledge required to understand what constitutes a good or a poor idea. For research, these contributions shed light on evaluation patterns in crowdsourcing and reveal a potential error source in crowdsourced idea evaluations. In line with Magnusson etal. (2016), ranking compared to absolute rating can counteract the hesitancy to reject proposals, particularly in the presence of high specialization. Ranking takes the responsibility of rejecting a proposal away from the crowd, as the proposals are sorted in the order of their quality. Contest operators are then responsible for setting a threshold and making the final decision regarding advancing and not advancing ideas. Ranking thus reduces the burden of understanding what good and bad proposals constitute on a universal scale of all existing ideas in favor of a comparative approach between the specific ideas in the respective contest. Therefore, the approach appears promising and highly relevant for innovation contests in the complex problem domain. This highlights that contest operators can affect the crowd’s ability to evaluate ideas based on an adequate evaluation procedure design. Our results show that which ranking approach is used is less relevant, as the performance does not differ significantly between the computed ranking and the relative ranking. The original assumption that relative ranking leads to better 923 Idea Evaluation forSolutions toSpecialized Problems:… performance was not supported here. One possible reason for this could be that the subliminal comparison between proposals in the computed ranking is already sufficient to evaluate the proposals appropriately. Most previous research on the capabilities of LLMs has been related to NLP tasks (Kocoń etal. 2023; Zhang etal. 2023) which we have now extended to a practical evaluation task in the area of complex problems. In line with studies of LLM performance in NLP tasks referring to evaluation (Gao etal. 2023; Kocmi and Federmann 2023; Wang etal. 2023), we show that LLMs possess evaluation capabilities under the right circumstances. The evaluation procedure in particular should be represented by a (relative or computed) ranking, as the LLM performs consistently and accurately when applying a ranking. In contrast, the LLM performs consistently but not necessarily accurately in the absolute rating. A high consistency cannot be used to infer performance because consistency reflects stability, not necessarily the quality of the outcomes. When considering the specialization, the evaluation performance of LLMs seems independent of specialization for each evaluation procedure. This adds to the theoretical foundations of LLMs, showing their inherent capacity to incorporate a wide array of diverse specialized knowledge to carry out evaluation tasks effectively. A second noteworthy finding is that the LLM in our study tends to assign only very good ratings with minimal variance across proposals on an absolute scale. This suggests that scoring proposals based on an absolute rating may not capture nuanced differences in task complexity and comprehension. At first sight, no clear threshold divides the evaluation scores into good and bad. Comparing the computed with the relative ranking, the LLM performs worse in relative ranking. As this difference has no statistical significance, we cannot say which ranking approach is superior. The computational complexity of the computed ranking is in the class of linear time (O(n)), while it is quadratic time (O(n2)) for the relative ranking based on pairwise comparisons. This means that if proposals are compared in pairs, each proposal must be compared with each other and therefore the effort is higher than if you only evaluate a single proposal. For a large number of proposals, this might lead to favoring the computed ranking. Therefore, researchers in the field of LLMs should carefully consider adopting ranking approaches to gain more meaningful insights into the model’s performance on different tasks. Based on a logistic regression, we were able to untangle the basis on which the LLM makes its decision. The moderate fit of the regression models for the LLM shows that we can only explain a small part of the variance of the LLM’s decision to advance proposals. LLMs, therefore, remain a kind of black box for us in their mode of operation (Feuerriegel etal. 2023). This finding underlines the importance of Explainable AI to better understand how AI models make decisions (Ray 2023). Synthesizing the implications of both decision makers, it can be concluded that crowds and LLMs have the ability to evaluate ideas in the complex problem domain. The performance of the LLM is not dependent on the degree of specialization. In contrast, it’s a key inhibitor for the crowd, but it can disappear through the choice of evaluation procedures. Further analysis show that both decision makers evaluate the few very best and worst ideas most accurately. Therefore, they are particularly helpful in the preselection phase of innovation contests to weed out bad ideas and reduce the burden on expert juries. 924 H.Gimpel et al. 6.2 Practical Implications Several practical implications can be inferred from our study. First, our results show that, in general, crowds and LLMs can be used as decision makers in innovation contests that address complex problems. However, the impact of specialization highlights the need for contest operators to know the specialization of the topic to be evaluated. If information on the specialization is available, a crowd can be used for the evaluation of less specialized areas. If the specialization is not known or critically high, an LLM evaluation might be a more suitable solution as it evaluates regardless of specialization levels. Second, our findings on the effectiveness of ranking evaluation procedures serve as guidelines for designing evaluation procedures in innovation contests. Innovation contests should use ranking tasks rather than absolute ratings for both crowds and LLMs to achieve assessments closer to expert juries. This holds in particular for evaluations by the crowd in contests where the domain is highly specialized or specialization levels are unknown. Third, we observe that the presentation of the proposals strongly impacts the decision maker’s decision to advance a proposal, particularly for the crowd. This finding could be considered when designing the platform by pointing idea generators to the importance of the presentation and standardizing the presentation’s level of detail and professional appearance. Further, the crowd might be explicitly informed not to overly focus on the presentation of a proposal but to focus on aspects such as novelty, feasibility, and impact. The fundamental challenge of balancing efficiency and quality cannot be solved entirely. However, our study suggests that when an expert jury is not available and the stakes in the decision are not too high, employing LLMs rather than a crowd might be appropriate. We specifically point to LLMs here rather than a crowd as— considering the numerical values of balanced accuracy—the computed ranking from the LLM outperforms other metrics and the crowd (the outperformance is numerical but not statistically significantly different from zero in all cases). The LLM evaluation quality tends not to deteriorate with contest specialization, and—given some technical expertise for assessing an LLM—evaluation is quicker and cheaper with an LLM as compared to a crowd. Further, GPT-3.5 allowed us to achieve roughly 70–80% balanced accuracy compared to the expert jury. Using GPT-4 or other more advanced models that will become available over time will likely increase accuracy. A final word of caution: for high-stakes decisions, 80% accuracy might not suffice, and expert juries might be preferable. 6.3 Limitations & Future Research Our research has limitations, which highlight the need for future research. First, the evaluation of the crowd took place in 2020, and the knowledge base of the LLM extends to 2021. In this period, there might have been technological or political progress in some of the idea areas discussed in the innovation contests. Consequently, these differences may impact the perception of an idea. 925 Idea Evaluation forSolutions toSpecialized Problems:… Second, our research builds upon the core assumption that expert jury evaluations are closest to the unknown ground truth of idea quality. Accordingly, we assess the decision makers’ performance by comparing their evaluation decision to the expert jury. This assumption is based on extensive research in the domain (Klein and Garcia 2015; Blohm etal. 2016; Nagar etal. 2016). However, to fully validate this assumption and better understand the decision makers’ evaluation behavior, it would be beneficial for future research to track the implementation progress and resulting benefits of the evaluated ideas. Similar to Wimbauer etal. (2019) research could then assess their actual quality and, even better, evaluate the idea preferences of the decision makers. After demonstrating the general suitability of LLMs in the complex problem domain, our study offers some pathways for improving evaluation performance. Further research should investigate more complex prompting strategies to achieve higher-quality output. In addition to adapting the prompting, fine-tuning an LLM to the specific task and knowledge area is possible to achieve better performance. In addition, we are endeavoring to obtain objective evaluations of innovative ideas from an LLM and compare them with expert evaluations. Another relevant research stream deals with the modeling of human opinions by LLMs for specific populations in social science (Argyle etal. 2023; Dominguez-Olmedo etal. 2024; Lee etal. 2024). This research stream is particularly relevant to our work as it contributes to a better understanding of how LLMs can replicate objective and subjective judgments. The research suggests that LLMs could model crowds with diverse opinions that may be well suited to evaluate ideas in combining both LLMs and crowd research. Merging insights from both research streams could open up further perspectives on the evaluation of ideas. The evolving capabilities of LLMs have been demonstrated in many studies, and it seems inevitable that decision-making will become increasingly automated in the future. This is due to a combination of factors, including technological advances, the growing availability of data and the need for more efficient solutions. As a result, many decision-making processes will increasingly be delegated to machines. Our study highlights that full automation is not yet achievable, underscoring the continued necessity for interaction between humans and technology. Therefore, it is crucial to complement the purely technical perspective with a sociotechnical approach that considers this interplay. Thus, future decision-making literature should take a holistic view of the entire decision-making system, focusing on both effectiveness and efficiency (Storey etal. 2024). It is therefore imperative to address human computer interaction and the effective delegation between the two in the decision-making process in order to optimally utilize their respective strengths (Baird and Maruping 2021). 7 Conclusion Designing idea evaluations in innovation contests for complex problems more efficiently is increasingly important, as traditional expert juries often constitute a severe financial and temporal bottleneck. This study provides the first detailed assessment of the potential of crowdsourced idea evaluations and LLMs’ idea evaluations in the complex problem domain. Our results suggest that specialization and evaluation procedures 926 H.Gimpel et al. affect the crowd’s potential across contexts and domains. With regard to the LLM, our results show that it can process tasks even in the zero-shot range, regardless of specialization. Concerning practice, we show that to some degree crowds and especially LLMs are indeed suitable for idea evaluations, even in the complex problem domain. Crowds and LLMs are important contributors in tackling the bottleneck of expert juries. Appendix A: Survey Items Novelty (source: Climate CoLab) Scale Label 1 Common, mundane, boring 2 Interesting, but not unheard of 3 Unusual, interesting, imaginative 4 Rare, surprising, challenging paradigms Feasibility (source: Climate CoLab) Scale Label 1 Infeasible socially, politically, legally or technically 2 Challenging, feasibility is questionable 3 Acceptable; Objections & barriers partially addressed 4 Appealing; Potential objections & barriers well addressed Impact (source: Climate CoLab) Scale Label 1 Benefits/ impact not clear 2 Limited benefits/small impact 3 Partial solution/ moderate impact 4 Large, direct, & positive impact Presentation (source: Climate CoLab) Scale Label 1 Neither clear, persuasive nor appealing 2 Only 1 of the following 3 applies: Clear, Persuasive, Appealing 3 Only 2 of the following 3 apply: Clear, Persuasive, Appealing 4 All 3 of the following 3 apply: Clear, Persuasive, Appealing Overall Score (source: e.g. Görzen etal.. 2016): Given the quality of the idea proposal you just saw and the rating you scored on all four dimensions, do you think the idea should advance to the next round? Scale Label 1 Absolutely no 2 Rather no 3 Rather yes 4 Absolutely yes