Document [original]

REGULAR ARTICLE

Understanding crowdsourcing projects: A review on the key

design elements of a crowdsourcing initiative

Rea Karachiwalla | Felix Pinkow

Institute of Technology and Management,

Technische Universität Berlin, Berlin, Germany

Correspondence

Rea Karachiwalla, Institute of Technology and

Management, Technische Universität Berlin,

Straße des 17. Juni 135, 10623 Berlin,

Germany.

Email: [email protected]

Crowdsourcing has gained considerable traction over the past decade and has

emerged as a powerful tool in the innovation process of organizations. Given its

growing significance in practice, a profound understanding of the concept is crucial.

The goal of this study is to develop a comprehensive understanding of designing

crowdsourcing projects for innovation by identifying and analyzing critical design

elements of crowdsourcing contests. Through synthesizing the principles of the social

exchange theory and absorptive capacity, this study provides a novel conceptual

configuration that accounts for both the attraction of solvers and the ability of the

crowdsourcer to capture value from crowdsourcing contests. Therefore, this paper

adopts a morphological approach to structure the four dimensions, namely, (i) task,

(ii) crowd, (iii) platform and (iv) crowdsourcer, into a conceptual framework to present

an integrated overview of the various crowdsourcing design options. The morpholog-

ical analysis allows the possibility of identifying relevant interdependencies between

design elements, based on the goals of the problem to be crowdsourced. In doing so,

the paper aims to enrich the extant literature by providing a comprehensive overview

of crowdsourcing and to serve as a blueprint for practitioners to make more informed

decisions when designing and executing crowdsourcing projects.

KEYWORDS

absorptive capacity, crowdsourcing contest, crowdsourcing design, morphological framework,

open innovation, social exchange theory, systematic literature review

1|INTRODUCTION

Innovations are considered a cornerstone of achieving and

maintaining competitive advantage (Salomo et al., 2008). However,

the ways how organizations innovate experienced fundamental

changes in the last two decades. Enkel et al. (2009) highlight that

many organizations are compelled to shift their focus from exclusive

internal research and development (R&D) to cooperation with

external partners. This understanding is rooted in the open innovation

paradigm coined by Chesbrough (2003). The concept of open innova-

tion assumes that knowledge is widely distributed, and organizations

seeking external knowledge for their own innovation purposes engage

in open innovation practices (Bogers & West, 2012; Chesbrough &

Bogers, 2014). Chesbrough and Bogers (2014) thereby explicitly high-

light that the rise of the Internet contributes to an ongoing paradigm

shift in innovation.

The emergence of Web 2.0 has enabled enterprises, people and

societies across the globe to connect and collaborate easily

(Vukovic, 2009; Zhao & Zhu, 2014b). In this context, crowdsourcing has

emerged as an effective problem-solving approach, attracting firms to

tap into a global pool of expertise, knowledge and creativity at substan-

tially lower costs (Afuah & Tucci, 2012; Boudreau & Lakhani, 2013;

Received: 14 June 2020 Revised: 14 May 2021 Accepted: 21 June 2021

DOI: 10.1111/caim.12454

This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium,

provided the original work is properly cited.

Creat Innov Manag. 2021;30:563–584. wileyonlinelibrary.com/journal/caim 563

Brabham, 2008; Jeppesen & Lakhani, 2010; Vukovic, 2009). Over the

past decade, many organizations have benefited from crowdsourcing-

based business models to solve internal problems, adapt to rapidly

evolving customer needs, shorten product lifecycles and increase overall

innovation efficiency (Brabham, 2008; Kohler, 2015).

Since the term was first coined by Howe (2006), crowdsourcing

has emerged as a complex, multidisciplinary concept with applications

in a wide variety of domains, including computer science, public

health, disaster and crisis management, information technology, engi-

neering, business and management (Afuah & Tucci, 2012;

Brabham, 2008, 2009; Gao et al., 2011; Hossain, 2015).

Crowdsourcing for innovation primarily refers to innovation contests,

also called tournament-based crowdsourcing or broadcast search

(Afuah & Tucci, 2012; Boudreau & Lakhani, 2013; Terwiesch &

Xu, 2008). Innovation contests are typically used to solve innovative,

challenging or creative problems in the form of an open call to large

network of potential contributors (Afuah & Tucci, 2012; Blohm

et al., 2018; Boudreau & Lakhani, 2013; Jeppesen & Lakhani, 2010). In

such contests, contributors self-select into participating and compete

with each other to generate the best solution(s). Consequently, the

best solution(s) is awarded by the seeking firm, typically in the form of

monetary awards (Afuah & Tucci, 2012; Blohm et al., 2018). The pri-

mary essence of such contests lies in mobilizing knowledge and exper-

tise that is otherwise distributed among the crowd to obtain novel

solutions beyond the traditional boundaries of an organization (Blohm

et al., 2013).

Despite the widespread adoption of crowdsourcing and the

many advantages it offers, there are many managerial challenges in

running crowdsourcing contests, and consequently, many companies

do not use the crowd effectively (Boudreau & Lakhani, 2013). In

particular, managers are concerned about executing crowdsourcing

challenges at reasonable costs that deliver appropriate solutions,

which are ultimately implementable in their organizations

(Acar, 2019; Afuah & Tucci, 2012; Boudreau & Lakhani, 2013).

Addressing these managerial challenges, the plethora of literature on

crowdsourcing concerns two central aspects. On the one hand, a

crowdsourcer must motivate the crowd to develop solutions

(Acar, 2019; Zhao & Zhu, 2014a; Zheng et al., 2011), and on the

other hand, the crowdsourcing firm must ensure that it can imple-

ment and capture value from the crowdsourced solutions (Blohm

et al., 2013; Ghezzi et al., 2018).

Therefore, when setting up and planning a crowdsourcing initia-

tive, crowdsourcing firms must consider these two central aspects,

which requires crowdsourcing managers to make informed decisions

that account for both aspects. In the course of this study, we define

decisions that relate to both crowd motivation or engagement and

capturing value as design-related decisions. The scope of the design

of a crowdsourcing challenge thus refers to the complete set of deci-

sions managers must address when designing a crowdsourcing con-

test. In this context, crowdsourcing research often focuses on

individual crowdsourcing design elements, such as the motivation of

crowd members (Leimeister et al., 2009; Zhao & Zhu, 2014a), task

design (Nakatsu et al., 2014; Zheng et al., 2011) and communication

and feedback mechanisms (Camacho et al., 2019; Piezunka &

Dahlander, 2019; Schäfer et al., 2017). However, these studies have

primarily focused on addressing single or specific design elements,

without developing an integrated picture of the overall crowdsourcing

system. As a result, there is still a lack of standardization for designing

crowdsourcing projects, and a conceptual framework representing the

important elements has yet to be established (Neto & Santos, 2018;

Zheng et al., 2011). Amrollahi (2015, p. 2) also points out that the

‘crowdsourcing literature lacks a comprehensive guideline through

which practitioners can initiate and manage their crowdsourcing pro-

jects’. As emphasized in the context of innovation contests by

Adamczyk et al. (2012), the design of a contest must be tailored for its

individual purpose. As such, from a practical standpoint, it is crucial to

have a comprehensive and standardized blueprint, which allows

to efficiently address the elaborated managerial challenges when

setting up crowdsourcing contests.

The central research question derived from this gap in crowdsourcing

literature therefore is: Which decisions must crowdsourcing managers

take during the design process of a crowdsourcing initiative in order

to both motivate the crowd to develop solutions and ensure that the

solutions can be implemented and provide value to the

crowdsourcing firm? In particular, what are the attributes that man-

agers can choose from within these design-related decisions? In con-

sideration of the two central managerial challenges, attracting the

crowd and capturing value from crowdsourced solutions, we seek to

answer this call for research through synthesizing existing research

on crowdsourcing design. This paper is organized as follows. First,

we elaborate on two major theoretical considerations that conceptu-

ally relate to the derived managerial challenges and that outline the

central dimensions along which design-related decisions must be

taken. To identify the concrete design-related decisions, we con-

ducted a systematic literature review in order to capture a compre-

hensive overview of the current state of research in the field of

crowdsourcing. As a result, a decision-centric overview of design

elements for crowdsourcing contests for innovation is developed and

discussed, and promising avenues for future research based on the

findings are presented.

2|CONCEPTUAL BACKGROUND

2.1 |Defining crowdsourcing contests for

innovations

As proposed originally by Howe (2006), the underlying premise of

crowdsourcing is that an organization outsources a task to a large,

undefined external group of individuals in the form of an open call. In

the context of crowdsourcing for innovation, the crowd typically

solves problems through creating prototypes, contributing ideas

in ideation contests or developing intellectual property for

crowdsourcing firms. Therefore, the individual solvers who decide to

develop a solution compete with each other. Since its emergence,

research on crowdsourcing has identified a range of elements defining

564 KARACHIWALLA AND PINKOW

the process of crowdsourcing for innovations. A common denomina-

tor in the vast majority of crowdsourcing literature is that the

crowdsourcing environment encompasses four distinct dimensions

that Hosseini et al. (2014) classify as the four fundamental pillars

of crowdsourcing: the crowdsourcing firm, the crowdsourced task, the

crowd and the system or platform used to connect the crowd and the

crowdsourcing firm (e.g. Afuah & Tucci, 2012; Brabham, 2008, 2009;

Estellés-Arolas & González-Ladr

on-de-Guevara, 2012; Kazman &

Chen, 2009; Pedersen et al., 2013; Vukovic, 2009).

Assuming this classification of the crowdsourcing environment,

many of the characteristics of crowdsourcing challenges that have

already been identified in extant crowdsourcing literature can be sub-

sumed under these four pillars, or dimensions, of crowdsourcing. For

instance, the task dimension includes certain characteristics such as

the task specificity and the degree of idea elaboration (Leimeister

et al., 2009) or the task definition (Blohm et al., 2018). The

crowdsourcing firm is characterized, for instance, by factors contribut-

ing to the quality assurance concerning the received solutions (Blohm

et al., 2018) or how firms evaluate the submitted ideas from the

crowd (Muhdi et al., 2011). The crowd, in turn, can be characterized

by the type of target group the crowdsourcer seeks to address

(Leimeister et al., 2009), which determines the specific skills and

knowledge the crowd requires to develop solutions (Blohm

et al., 2018). Against this backdrop, the four fundamental pillars of

crowdsourcing are considered a robust classification of the

crowdsourcing environment, encompassing four distinct dimensions

that allow to clearly distinguish and categorize a vast majority of more

specific characteristics of crowdsourcing. In the following, we will use

these four dimensions to derive their linkage to the stated two central

managerial challenges for conducting crowdsourcing challenges.

2.2 |Motivating and encouraging the crowd

The fundamental mechanism that enables successful crowdsourcing

initiatives is the participation of individual crowd members. Therefore,

crowdsourcing firms must convince the crowd to develop solutions by

conveying the task to be solved through a suitable platform. Thus, the

crowdsourcing firm and the solvers engage in an exchange process—

the solvers put effort into developing solutions and expect to receive

rewards for their efforts. This exchange process reflects the basic

notion of the social exchange theory introduced by Blau (1964). The

exchange process hereby ‘refers to voluntary actions of individuals

that are motivated by the returns they are expected to bring and

typically do in fact bring from others’(Blau, 1964, p. 91). In a

crowdsourcing context, potential solvers screen the task provided by

the crowdsourcing firm and evaluate both the expected benefits and

the related costs.

In fact, previous research on crowdsourcing participation primarily

focuses on factors motivating the crowd to participate. Individuals

may be motivated to develop solutions based on intrinsic

motives such as altruism, working on an interesting project, being

creative or demonstrating their skills (Afuah & Tucci, 2012;

Garcia Martinez, 2017; Schäper et al., 2021). These intrinsic motiva-

tors illustrate that crowdsourcing is not exclusively an economic rela-

tionship and exchange process (Allon & Babich, 2020). Solvers may

enjoy the very process of developing solutions merely based on the

required creativity and the individual autonomy to solve the given prob-

lems (Garcia Martinez, 2017). Although intrinsic motivation plays a cen-

tral role for crowdsourcing participation, crowdsourcing firms also offer

extrinsic motivation by providing monetary rewards for the best

solution(s) (Afuah & Tucci, 2012). Solvers who provide solutions hence

provide knowledge and ideas in return for an expected outcome, which

can be either monetary or non-monetary (Afuah & Tucci, 2012). Ye and

Kankanhalli (2017) acknowledge the central role of motivators for the

crowd to engage in this exchange process but highlight the lack of

research on possible deterrents of participation and thus introduce the

social exchange perspective to the context of crowdsourcing. More spe-

cifically, individuals who develop solutions also face costs in terms of

required time and effort. Ultimately, individuals only engage in develop-

ing solutions when they expect a positive net reward from a cost–

benefit analysis (Ye & Kankanhalli, 2017), which is reflecting the central

notion of the social exchange theory (Blau, 1964).

Therefore, the description of the task to be crowdsourced pro-

vides the relevant information for solvers to create solutions and con-

veys potential motivators and costs. This constitutes the central

interdependence required for a social exchange process—the outcome

(the solutions developed by the crowd) depends on both (i) the

crowdsourcing firm through providing a sufficiently detailed task

description and defining solution requirements and (ii) the knowledge

and skills of the crowd to interpret the task and to develop solutions.

This interdependence is a fundamental requirement of social

exchange (Cropanzano & Mitchell, 2005). We build on the argumenta-

tion provided by Ye and Kankanhalli (2017) and seek to identify the

factors in a crowdsourcing contest that determine the benefits and

costs of the participating solvers that must be considered during the

design phase of a crowdsourcing campaign. In particular, this high-

lights that the crowdsourced task must be sufficiently delineated in

order to provide adequate information to potential solvers. Applying

the social exchange perspective further emphasizes that managers

must not only take extrinsic motivators, in terms of monetary awards,

into consideration but must also deliberately determine which poten-

tial intrinsic motivation, and which costs, the task description conveys

to the crowd. This theoretical perspective thereby captures and

motivates three of the fundamental pillars of crowdsourcing that are

the communication of the crowdsourced task, which provides

the required information for the cost–benefit analysis, and thus

conveys motivational and cost factors to the crowd through a chosen

crowdsourcing platform.

2.3 |Capturing value from crowdsourced ideas

and solutions

Besides the elements that support attracting potential solvers to

develop solutions, firms must also take into account that the

KARACHIWALLA AND PINKOW 565

crowdsourced solutions ultimately should provide value to the

firm (Cappa et al., 2019). Malone et al. (2010) raise the central

question of why crowdsourcers engage in crowdsourcing projects

in the first place, emphasizing the need to define how solutions

can eventually be utilized to provide value. Recent crowdsourcing

research therefore increasingly focuses on the absorptive capacity

of organizations in the context of crowdsourcing (e.g. Afuah &

Tucci, 2012; Boons & Stam, 2019; Gassenheimer et al., 2013; Ruiz

et al., 2020).

In its core, absorptive capacity relates to an organization's ability

to recognize the value of external information, the assimilation of said

value and the implementation and application to commercial ends

(Cohen & Levinthal, 1990). In the context of crowdsourcing, absorp-

tive capacities can include the platform that is used to connect the

crowdsourcer and the crowd, filtering processes that enable

the crowdsourcer to exclude weak solutions quickly, establishing

information exchange processes between the crowd and the

crowdsourcer and attracting a critical mass of contributors (Blohm

et al., 2013). Furthermore, gaining crowdsourcing experience and

thereby creating knowledge on how to conduct crowdsourcing pro-

jects ultimately can positively affect the absorptive capability for

future crowdsourcing projects and knowledge exchange processes

(Pollok et al., 2019b).

These approaches to build absorptive capacities to capture

value from crowdsourcing demonstrate that this is primarily the

task of the crowdsourcing firm. Given the solutions provided by

the crowd are contingent on the description of the crowdsourcing

task, the crowdsourcer can already account for creating absorptive

capacities during the design phase of a crowdsourcing contest. As

such, defining certain success metrics to evaluate solutions (Ford

et al., 2015), estimating the costs of required resources such as

personnel (Muhdi et al., 2011) and deliberate risk management (Liu,

Xia, Zhang, & Wang, 2016) can positively contribute to

crowdsourcing success. These exemplary issues facilitate to receive

solutions that ultimately can provide value to the crowdsourcing

firm. We hereby emphasize the importance of the early design

phase of a crowdsourcing contest to determine whether a firm can

benefit from the received solutions. Moreover, this consideration

goes beyond the scope of the introduced exchange process

between the crowd and the crowdsourcer. Crowdsourcing firms

have to determine internal organizational factors that are not

directly linked or perceived by the crowd, but that contribute to

the ability to capture value. For instance, firms must determine the

internal costs of executing a crowdsourcing campaign and subse-

quently determine whether the expected benefits of the solutions

exceed the internal costs. This cost–benefit analysis is a prerequi-

site for firms to ultimately capture value from crowdsourcing. This

second theoretical perspective relates to two of the mentioned

pillars of crowdsourcing, which are the crowd, in terms of decisions

related to which type of crowd to attract, and the crowdsourcing

firm, in terms of internal organizational capacities that enable the

firm to capture value.

2.4 |Theoretical framework and contributions

Although these distinct theoretical perspectives, the social

exchange theory and absorptive capacity, have been investigated

separately in the context of crowdsourcing, we aim to integrate

these perspectives from both a theoretical and practical point of

view. On the one hand, this integration allows to provide design-

related implications for crowdsourcing managers to make more

informed decisions when designing and executing crowdsourcing

projects for innovation by offering a decision-centric blueprint for

crowdsourcing challenge design. On the other hand, the synthesis

of the different theoretical perspectives—social exchange theory

and absorptive capacity—constitutes a novel conceptual approach

in crowdsourcing literature. As indicated above, only this integra-

tive perspective allows to address all four fundamental dimensions

of crowdsourcing and thereby captures the two main challenges

for crowdsourcing managers to attract and motivate the crowd and

capture value from crowdsourcing.

Amidst the plethora of literature on crowdsourcing, this con-

ceptual paper is positioned in the context of crowdsourcing con-

tests for innovation. As Hosseini et al. (2014) elaborate on a

general perspective on the four dimensions of crowdsourcing, their

categorization seeks to maintain a rather multidisciplinary perspec-

tive. With this paper, we refine and enhance the conceptual under-

standing of crowdsourcing contests, adopting the perspective of

the crowdsourcing firm. The central contribution of this perspective

to the extant literature is twofold. First, the elaboration of design-

related decisions along the four crowdsourcing dimensions

contributes to a unitary understanding of the process of designing

a crowdsourcing contest. Future research can thus benefit from

this refined understanding as further insights on crowdsourcing can

be clearly positioned within this framework, resulting in more

coherent research designs. Second, the novel conceptual approach

emphasizes that in order to advance our understanding of

crowdsourcing for innovation, research results must be discussed in

the context of all four dimensions. As Ghezzi et al. (2018) outline,

there is still need for further research on both the mechanisms

that allow firms to effectively integrate solvers ideas and practices

that enable firms to increase solver participation through intrinsic

and extrinsic motivational factors. While investigating factors that

impact the extrinsic or intrinsic motivation of the crowd, not

relating these factors to the ability of firms to effectively utilize

the crowd to create value, or vice versa, fails to address

fundamental managerial concerns. This paper therefore provides a

groundwork for discussing specific findings in the broader context

of the crowdsourcing environment. Furthermore, by adopting a

decision-centric approach to conceptualize crowdsourcing project

design, this paper highlights the overall design space opportunities

available to firms engaging in crowdsourcing for innovation.

Because planning and framing a crowdsourcing contest is rather

cost intensive, it is of particular importance to cautiously define all

fundamental aspects and decision to be taken (Paik et al., 2020).

566 KARACHIWALLA AND PINKOW

This study, therefore, contributes to this central managerial

challenge by providing an integrative overview on the different

possible campaign configurations for a crowdsourcing initiative.

3|RESEARCH METHODOLOGY

Building on the four pillars of crowdsourcing proposed by Hosseini

et al. (2014), we review existing literature and broadly classify previ-

ous research into the following four dimensions: (i) the task to be

crowdsourced, (ii) the crowd, (iii) the platform and (iv) the

crowdsourcer. Next, individual design elements corresponding to each

of the four dimensions are extracted and analysed. The introduced

theoretical background provides further guidance to structure the

reviewed literature. Against this backdrop, for example, the motiva-

tion of the crowd to participate in a crowdsourcing initiative could be

intrinsic, extrinsic or both. In order to consolidate the findings from

the literature, a morphological framework considering the different

design elements is proposed. The morphological approach accommo-

dates multiple alternative configurations because it allows the possi-

bility of choosing different combinations of attributes for each design

element.

This paper uses a systematic literature review as the research

method to identify and analyse the different design elements of a

crowdsourcing project. A systematic literature review is a structured

analysis of previous work done in a field by evaluating and assimilating

extant research using a concept-centric approach (Webster &

Watson, 2002). Because prior research has explored individual, spe-

cific elements of a crowdsourcing project, a systematic literature

review is an appropriate research method for extracting and synthe-

sizing the literature to develop a comprehensive overview of a given

field of research. We followed the four essential stages of a system-

atic literature review, namely, (i) plan the review, (ii) conduct search,

(iii) extract data and (iv) report results, as proposed by Okoli and

Schabram (2010).

3.1 |Systematic review

The first stage of a systematic literature review is to meticulously

plan the research strategy. This comprises defining the research

questions to be addressed and outlining the search strategy,

including identifying appropriate databases, defining search terms

and setting selection criteria for the search. As discussed in the

previous section, the goal of this paper is to develop a profound

conceptual understanding of crowdsourcing projects by answering

the central research question which key elements of a

crowdsourcing project can be identified in consideration of a holis-

tic perspective on crowdsourcing contests for innovations. In

particular, this study aims to develop a comprehensive overview of

all aspects that need to be considered when designing a

crowdsourcing contest.

3.1.1 | Databases

As a preliminary step, suitable databases for the search process were

selected. EBSCO Business Source Complete is a leading scholarly

business database, with content from over 10,000 well-established

academic journals. Because the focal point of this paper is to better

understand crowdsourcing projects in the business context, this data-

base seemed appropriate. As a second source, the Web of Science

database was selected because of its breadth of interdisciplinary

research literature from over 30,000 peer-reviewed scientific journals.

As a third source, the ABI/Inform was used, because it offers a pleth-

ora of research literature on business trends, corporate strategy and

management theory, which are relevant for this paper.

3.1.2 | Search terms

In order to get a complete overview of prior work done in the field,

we intentionally chose broad keywords. Because this study focuses

on exploring crowdsourcing in the context of sourcing innovations,

the search terms included crowdsourc* or crowd sourc* restricted to

business and management literature. This restriction was made in

order to exclude other forms of crowdsourcing, such as

crowdsourcing for software engineering, and to account for the scope

of this study considering the focus on crowdsourcing contests for

innovation. In line with Snyder (2019), we developed predefined

criteria to determine which articles to include in the final analysis from

the initial pool of articles that have been identified through searching

for the keywords. These criteria include (i) articles that focus on

crowdsourcing in the business or management context (not NGOs,

social context or non-business organizations), (ii) articles that focus on

crowdsourcing innovation contests, (iii) articles that must include

design elements of the crowdsourcing concept, (iv) primary focus on

articles from high-quality academic journals (peer-reviewed),

(v) articles within the time frame 2006 (when the term was first

coined) until 2019 and (vi) articles written in English.

3.1.3 | Conduct search

As a preliminary step, the keywords crowdsourc* and crowd sourc*

were used in all three selected databases, applying the following first-

level criteria: articles published in the time frame January 2006 and

April 2021 and articles published in English. This resulted in a total

population of 22,178 results. A broad search was intentionally con-

ducted at first in order to generate a wide range of results and to get

an overview of prior work done in the field. Because this study

focuses on understanding crowdsourcing projects in the business and

management context specifically, the search was narrowed down by

applying the following second-level criteria: articles focused on

crowdsourcing in the business or management fields and articles pub-

lished in academic or scholarly journals. In doing so, the population of

KARACHIWALLA AND PINKOW 567

articles was reduced to 1,859 articles. The significant reduction in

articles shows that there has been comparatively little crowdsourcing

research in the business and management context. As a final step, the

following third-level criteria were applied: articles that focus particu-

larly on innovation contests or tournament-based crowdsourcing and

articles that include at least one dimension or design element. During

this step, the relevance of the articles was determined by reading the

abstract, introduction and conclusion and in some cases by examining

the paper. Our search resulted in a total of 94 articles that we

identified as eligible for further review, which is accordingly illustrated

in Table 1.

After removing duplicates, a total of 55 articles were left. In order

to identify additional relevant articles, a backward and forward search

was conducted (Webster & Watson, 2002). In this step, relevant

conference papers were additionally included. This yielded another

11 journal and conference articles, resulting in a final pool of 66 articles.

3.2 |Data extraction

In this stage, data were extracted from the final pool of articles to

identify the different elements of a crowdsourcing initiative. Building

on the ‘four pillars of crowdsourcing’proposed by Hosseini

et al. (2014), the existing literature was first classified and coded into

the following four fundamental dimensions of crowdsourcing: (i) the

task to be crowdsourced, (ii) the crowd, (iii) the crowdsourcer and

(iv) the platform. For each of the articles, the main findings

and corresponding design elements were recorded.

Because the four selected dimensions are relatively broad and

contain multiple elements within them (for instance, the task

dimension includes different elements such task delineation, task

modularity and task granularity), many articles address more than one

dimension. It is worth pointing out that a relatively permissive

approach was adopted when classifying the literature, meaning that

even articles that vaguely related to any crowdsourcing design ele-

ments were initially considered. The purpose in doing so was to

ensure that the review took into account different findings previously

suggested in the literature in order to develop a comprehensive and

cohesive picture of the key design elements of crowdsourcing.

3.3 |Descriptive results

Including the final pool of 66 articles, Figure 1 illustrates the number

of publications per year as a result of the systematic literature search.

The distribution shows that the number of studies related specifically

to crowdsourcing contests for innovation is relatively low, in contrast

to the literature in the crowdsourcing field in general. In the first years

since the coined was termed, most studies focused on exploring the

general crowdsourcing concept and its potential applications in other

fields. However, in the past few years, there have been an increasing

number of publications per year with regard to crowdsourcing con-

tests, which indicates the growing relevance of crowdsourcing for

innovation in the business and management context.

As previously mentioned, the articles in the final pool were

coded based on the four fundamental dimensions of crowdsourcing.

Papers that dwelled upon any of the design elements corresponding

to the four dimensions were included. For each of the selected

articles, the main findings and corresponding design elements were

recorded.

FIGURE 1 Publications per year

(journal and conference articles)

TABLE 1 Database search results

Database First-level criteria Second-level criteria Third-level criteria

Business Source Complete 3,811 802 38

Web of Science 10,813 477 32

ABI/Inform 7,554 580 24

Total 22,178 1,859 94

568 KARACHIWALLA AND PINKOW

Consistent with prior research, the results show that the task to

be crowdsourced is one of the most critical factors influencing the

overall success of crowdsourcing projects (Afuah & Tucci, 2012;

Blohm et al., 2018; Ghezzi et al., 2018; Gillier et al., 2018; Nakatsu

et al., 2014; Zheng et al., 2011). As the nature and complexity of the

task has a significant impact on elements in the other dimensions such

as crowd participation, incentive design and intellectual property

mechanisms, effective task design is crucial for crowdsourcing pro-

jects. The results of the literature review also demonstrate that the

crowdsourcer dimension is highly important for the success of

crowdsourcing contests. Because the crowdsourcer is responsible for

initiating and operating the project, several key decisions need to be

made, for instance, how to manage risk, allocate resources and evalu-

ate and implement crowdsourced ideas. From the crowd perspective,

the success of any crowdsourcing project largely depends on the

knowledge and diversity, motivation and size of the crowd. Therefore,

crowdsourcing firms must consider the characteristics of the crowd

when designing a crowdsourcing project. The platform dimension

seems to have received relatively less attention. However some prior

studies point out that the decision to set up an own platform versus

contract an intermediary is an important decision for crowdsourcing

firms (Ford et al., 2015; Thuan et al., 2016).

4|LITERATURE SYNTHESIS AND

ANALYSIS

In this section, the results of the literature review are presented and

discussed. The first subsection essentially summarizes prior work

related to design elements for innovation contests. Subsequently, the

individual design elements are analysed and discussed in more detail.

4.1 |Literature synthesis

A concept-centric approach is used to present a summary of the rele-

vant findings (Webster & Watson, 2002). More specifically, Table 2

maps the various design elements identified in the literature to the

four fundamental dimensions as guided by the introduced theoretical

background. As previously mentioned, articles that explicitly dwelled

upon any of the design elements corresponding to the four dimen-

sions were coded and included in the context of this study. In the con-

cept matrix below, studies corresponding to each of the design

elements are also highlighted. In total, 20 design elements were

extracted.

4.2 |Literature analysis

4.2.1 | Task

The task or problem to be crowdsourced is one of the most important

aspects of a crowdsourcing initiative. The task is usually the first point

of contact between the crowdsourcer and the crowd, based on which

solvers decide to self-select into participating or not (Afuah &

Tucci, 2012; Steils & Hanine, 2016). Based on an analysis of prior liter-

ature, seven essential design elements related to the task dimension

are identified, and their relevance for the design phase is illustrated in

Table 3.

Task delineation

The delineation of a task refers to how well the crowdsourced prob-

lem is described and formulated (Afuah & Tucci, 2012). A well-

articulated problem statement is one of the most fundamental steps

in the crowdsourcing process. Prior research suggests that the formu-

lation of the problem statement has a direct impact on the quality of

the solutions received (Allahbakhsh et al., 2013; Gillier et al., 2018;

Hetmank, 2013; Jespersen, 2018; Lee et al., 2015; Thuan et al., 2016).

Well-delineated problems are easier to understand and interpret, but

on the other hand, problems that are not clearly described can

increase the chances of being misinterpreted (Afuah & Tucci, 2012;

Muhdi et al., 2011; Natalicchio et al., 2017; Schenk & Guittard, 2011).

In order to formulate a well-delineated problem statement, the follow-

ing characteristics should be considered in more detail.

Task specificity

Task specificity refers to the scope of the problem to be solved. It

could range from highly specific tasks to open-ended tasks for which

no particular problem solving approach or solution is known

(Jespersen, 2018; Leimeister et al., 2009; Nakatsu et al., 2014; Ren

et al., 2021). Jespersen (2018) argues that highly specific tasks reduce

the solution space and often lead to an incremental nature of solu-

tions submitted. On the other hand, unstructured, open-ended tasks

foster creativity and could lead to the discovery of new knowledge. In

particular, tasks with a rather broad scope may attract more solvers

but in turn lead to higher firm efforts considering the evaluation of a

more heterogeneous pool of solutions (Christensen & Karlsson, 2019).

The specificity of the task reflects whether the company searches

for solutions that are ‘local’or ‘distant’from its exiting knowledge

base (Afuah & Tucci, 2012; Jespersen, 2018). For instance, if a com-

pany searches for improvements to an existing technology, it is con-

ducting a local search, but on the other hand, if a company is

searching for new technologies that it is unfamiliar with, it is con-

ducting a distant search (Afuah & Tucci, 2012). Pollok et al. (2019a)

suggest that in order to attract an optimal number of solutions, the

seeker should have sufficient domain knowledge to formulate a com-

prehensive problem statement, but at the same time not be too spe-

cific such that the problem-solving space is overly constrained. Given

that the task briefing should contain all the necessary information

needed by the solvers to develop a solution (Colombo et al., 2013;

Schenk & Guittard, 2011; Zheng et al., 2011), tasks with highly confi-

dential information are not ideal for crowdsourcing because sensitive

components are made unavailable to the solvers, which in turn may

affect the quality of the solutions (Afuah & Tucci, 2012; Ghezzi

et al., 2018; Hetmank, 2013; Nakatsu et al., 2014; Natalicchio

et al., 2017).

KARACHIWALLA AND PINKOW 569

TABLE 2 Crowdsourcing design elements

Dimension Design element Sources

Task Task delineation (10) Afuah & Tucci, 2012; Allahbakhsh

et al., 2013; Gillier et al., 2018;

Hetmank, 2013; Jespersen, 2018; Lee

et al., 2015; Muhdi et al., 2011;

Natalicchio et al., 2017; Schenk &

Guittard, 2011; Thuan et al., 2016

Task specificity (13) Afuah & Tucci, 2012; Christensen &

Karlsson, 2019; Colombo et al., 2013;

Ghezzi et al., 2018; Hetmank, 2013;

Jespersen, 2018; Leimeister et al., 2009;

Nakatsu et al., 2014; Natalicchio

et al., 2017; Pollok et al., 2019a; Ren

et al., 2021; Schenk & Guittard, 2011;

Zheng et al., 2011

Task granularity (10) Afuah & Tucci, 2012; Garcia

Martinez, 2017; Ghezzi et al., 2018; Lee

et al., 2015; Liu, Xia, Zhang, Pan, &

Zhang, 2016; Muhdi et al., 2011;

Natalicchio et al., 2017; Rouse, 2010;

Zhao & Zhu, 2014a; Zheng et al., 2011

Task modularity (6) Afuah & Tucci, 2012; Blohm et al., 2018;

Liu, Xia, Zhang, Pan, & Zhang, 2016;

Nakatsu et al., 2014; Natalicchio

et al., 2017; Pee et al., 2018

Solution requirements (8) Afuah & Tucci, 2012; Blohm et al., 2018;

Ford et al., 2015; Ghezzi et al., 2018;

Koh, 2019; Mazzola et al., 2018; Steils &

Hanine, 2016; Zheng et al., 2011

Task allocation (10) Afuah & Tucci, 2012; Blohm et al., 2018;

Boudreau & Lakhani, 2013;

Brabham, 2008; Christensen &

Karlsson, 2019; Geiger & Schader, 2014;

Hetmank, 2013; Jeppesen &

Lakhani, 2010; Leimeister et al., 2009;

Piezunka & Dahlander, 2015

Contest duration (4) Ayaburi et al., 2020; Chen et al., 2021;

Leimeister et al., 2009; Muhdi et al., 2011

Crowd Motivation and incentives (21) Acar, 2019; Afuah & Tucci, 2012; Blohm

et al., 2018; Brabham, 2010; Chen

et al., 2021; de Beer et al., 2017; Frey

et al., 2011; Garcia Martinez, 2017;

Ghezzi et al., 2018; Görzen, 2021; Hanine

& Steils, 2019; Jeppesen &

Lakhani, 2010; Lee et al., 2015;

Leimeister et al., 2009; Li & Hu, 2017;

Mazzola et al., 2018; Mazzola

et al., 2020; Pee et al., 2018; Schenk &

Guittard, 2011; Zhao & Zhu, 2014a;

Zheng et al., 2011

Knowledge diversity (8) Afuah & Tucci, 2012; Blohm et al., 2018;

Boons & Stam, 2019; Ford et al., 2015;

Frey et al., 2011; Natalicchio et al., 2017;

Steils & Hanine, 2016; Thuan et al., 2016

Size (2) Afuah & Tucci, 2012; Boudreau et al., 2011

Platform Ownership (9) Blohm et al., 2018; Colombo et al., 2013;

Diener & Piller, 2013; Ford et al., 2015;

Leicht et al., 2016; Schenk et al., 2019;

Thuan et al., 2016; Zhao & Zhu, 2014a;

Zogaj et al., 2014

570 KARACHIWALLA AND PINKOW

Therefore, it is highly important to distinguish between

crowdsourcing specific problems for which a predefined solution or

approach exists and open-ended problems for which creative, innova-

tive solutions are needed.

Task granularity

Task granularity refers to the degree of complexity of the problem

to be solved. Rouse (2010) distinguishes three classifications of

tasks to be crowdsourced: simple tasks,moderate tasks and sophisti-

cated tasks. Simple tasks refer to those of low complexity that do

not require specific skills or expertise, whereas sophisticated tasks,

on the other hand, are more complex in nature and demand cer-

tain competencies and substantial domain knowledge. Moderate

tasks refer to tasks that involve a moderate level of skill and

knowledge.

Zheng et al. (2011) further define two components of task com-

plexity: analysability and variability. Analysability refers to the difficulty

of the knowledge search process in developing the solution. Variabil-

ity refers to the amount of new knowledge required to solve a prob-

lem. Similarly, Natalicchio et al. (2017) highlight that task complexity is

represented by the number of knowledge components involved in the

problem.

TABLE 2 (Continued)

Dimension Design element Sources

Crowdsourcer Solution evaluation (10) Afuah & Tucci, 2012; Blohm et al., 2013;

Blohm et al., 2018; Chen et al., 2020;

Geiger & Schader, 2014; Ghezzi

et al., 2018; Mack & Landau, 2020;

Muhdi et al., 2011; Piezunka &

Dahlander, 2015; Ye & Kankanhalli, 2015

Implementation potential (5) Afuah & Tucci, 2012; Blohm et al., 2013;

Ford et al., 2015; Lüttgens et al., 2014;

Muhdi et al., 2011

Feedback and communication (11) Blohm et al., 2013; Blohm et al., 2018;

Camacho et al., 2019; Chan et al., 2021;

Jian et al., 2019; Leimeister et al., 2009;

Muhdi et al., 2011; Piezunka &

Dahlander, 2019; Schäfer et al., 2017;

Wooten & Ulrich, 2017; Zheng

et al., 2014

Incentives and awards (11) Afuah & Tucci, 2012; Blohm et al., 2018;

Boudreau et al., 2011; Geiger &

Schader, 2014; Lee et al., 2015;

Leimeister et al., 2009; Mazzola

et al., 2018; Muhdi et al., 2011; Schenk &

Guittard, 2011; Zhao & Zhu, 2014a;

Zheng et al., 2011

Resource planning (10) Afuah & Tucci, 2012; Boudreau &

Lakhani, 2013; Brabham, 2008; Ford

et al., 2015; Jeppesen & Lakhani, 2010;

Lüttgens et al., 2014; Muhdi et al., 2011;

Thuan et al., 2016; Vukovic, 2009; Ye &

Kankanhalli, 2015

Risk management (10) Afuah & Tucci, 2012; de Beer et al., 2017;

Ford et al., 2015; Ghezzi et al., 2018; Liu,

Xia, Zhang, Pan, & Zhang, 2016; Liu, Xia,

Zhang, & Wang, 2016; Nakatsu

et al., 2014; Natalicchio et al., 2017;

Nirosh Kannangara & Uguccioni, 2013;

Sauerwein et al., 2016

Legal and intellectual property management (4) Blohm et al., 2018; de Beer et al., 2017;

Foege et al., 2019; Mazzola et al., 2018

Brand image and trust (4) Blohm et al., 2013; Garcia Martinez, 2017;

Liu, Xia, Zhang, Pan, & Zhang, 2016; Ye &

Kankanhalli, 2015

Success metrics (2) Blohm et al., 2013; Ford et al., 2015

KARACHIWALLA AND PINKOW 571

Prior studies suggest that the complexity of the task strongly

influences the motivation of the crowd to participate in

crowdsourcing contests (Ghezzi et al., 2018; Lee et al., 2015; Zhao &

Zhu, 2014a; Zheng et al., 2011). For instance, a complex task often

requires a higher level of specific knowledge and skill and is more

likely to satisfy an intrinsic need to further develop one's compe-

tences (Garcia Martinez, 2017). However, as the task complexity

increases, the level of involvement, effort and cognitive skills needed

to solve the problem also increase, and therefore, solvers need to be

incentivized appropriately. During the task design phase, it is crucial

to assess the level of complexity of the task and the effort required to

solve it (Lee et al., 2015; Liu, Xia, Zhang, Pan, & Zhang, 2016). Afuah

and Tucci (2012) highlight that because highly complex problems can

often be difficult to delineate, they increase the chances of being mis-

interpreted (Lee et al., 2015; Muhdi et al., 2011). Therefore,

crowdsourcing companies should take into account the level of com-

plexity and ease of delineation when deciding to crowdsource a

problem.

Task modularity

Task modularity refers to the decomposition of the task into smaller

sub-tasks. Though not all tasks can be decomposed, it is important to

point out that modularity is effective only when there is a low degree of

interdependence among the task components (Afuah & Tucci, 2012;

Natalicchio et al., 2017). Modular tasks with a low level of

interdependence can be easier to delineate and hence easier for the

solvers to interpret. It also provides the opportunity for individuals to

work on sub-tasks for which they have high levels of expertise and skills

(Afuah & Tucci, 2012). However, for problems that require a high level

of interaction among the task components, decomposing the problem

can increase the overall complexity, because different

knowledge components need to be combined to develop the solution.

Furthermore, such tasks require individuals to collaborate and share

knowledge (Pee et al., 2018). As a result, any missing information or lack

of knowledge for any of the sub-tasks can negatively influence the qual-

ity of the solution. Some prior studies suggest that task decomposition

is more appropriate for collaborative crowdsourcing that leverages col-

lective intelligence (Afuah & Tucci, 2012; Blohm et al., 2018; Nakatsu

et al., 2014) than tournament-based crowdsourcing for which individ-

uals compete individually to develop the best solutions (Liu, Xia, Zhang,

Pan, & Zhang, 2016). Therefore, when delineating the problem,

crowdsourcing firms should take into consideration the degree of inter-

actions between the sub-tasks when deciding whether to decompose it

or not.

Solution requirements

Solution requirements refer to the criteria that must be fulfilled in

developing a solution. Defining the contribution requirements is con-

sidered to be one of the most crucial steps because its influences the

solvers decision to participate (Afuah & Tucci, 2012; Blohm

et al., 2018; Zheng et al., 2011). Seekers should ensure that contribu-

tion requirements are explicitly defined because they serve as a guide

for solvers to develop solutions (Steils & Hanine, 2016). Furthermore,

the contribution requirements stated in the project briefing serve as

an indicator for how the solutions will be evaluated and assessed by

the seeker. Some studies suggest providing examples to improve the

quality and effectiveness of solutions (Ghezzi et al., 2018; Koh, 2019).

Afuah and Tucci (2012) further highlight that firms must take into

account the potential for integrating solutions into the company's

existing value chain. This becomes especially relevant for organiza-

tions that search for ‘distant’solutions beyond their current trajecto-

ries. On the other hand, integrating ‘local’alternative solutions that

are incremental in nature is relatively less complicated. Therefore, to

benefit from crowdsourcing initiatives, it is important to consider the

implementation potential when delineating the problem and defining

requirements (Ford et al., 2015).

Prior research also indicates that firms must clearly indicate the

expectations regarding intellectual property rights (IPR) in the problem

statement, because this has a significant impact on participation in

crowdsourcing contests (Mazzola et al., 2018; Steils & Hanine, 2016).

Intellectual property arrangements should be defined based on the

level of complexity of the problem and expected solution require-

ments (Mazzola et al., 2018).

Task allocation

Task allocation refers to allocation of the task to a specific group of

individuals in the crowd, depending on the expertise required to solve

TABLE 3 Design-related elements in the task dimension

Task

dimension Design-related relevance

Task

delineation

Problem and task articulation

Facilitate interpretation of crowdsourced task

Task specificity Defines the scope and solution space of the task

Determines whether the task requires local or

distant knowledge

Task

granularity

Defines the complexity of the task

Conveys the required skills to solve the task and

thereby impacts crowd motivation to engage in

solving the task

Task

modularity

Decomposition of tasks into sub-tasks (if

applicable)

Illustrate interdependence of sub-tasks, which can

increase the task complexity

Solution

requirements

Criteria that a solution must fulfil

Guide solvers during solution development

Indicate the evaluation procedure by the

crowdsourcer

Indicate expectations on intellectual property

rights

Task allocation Specifies the target group contingent on required

expertise and skills

Contest

duration

Time frame defining the deadline to submit

solutions

Depends on the defined complexity of the task

572 KARACHIWALLA AND PINKOW

the problem. For instance, crowdsourcing companies can target con-

tributors with specific knowledge that may be better suited to

develop a solution (Blohm et al., 2018; Leimeister et al., 2009). Blohm

et al. (2018) highlight that crowdsourcing firms can target crowd con-

tributors based on specific skills, demography and performance in

prior contests.

Although tournament-based crowdsourcing leverages the diver-

sity of the crowd to solve problems (Afuah & Tucci, 2012; Boudreau &

Lakhani, 2013; Brabham, 2008; Jeppesen & Lakhani, 2010), it can also

result in a lot of ‘crowding’from low-quality and irrelevant solutions

(Piezunka & Dahlander, 2015). Therefore, targeting crowd contribu-

tors with the appropriate knowledge and expertise could be an effec-

tive approach to reduce the ‘noise’and generate higher quality

solutions (Blohm et al., 2018; Christensen & Karlsson, 2019; Geiger &

Schader, 2014; Hetmank, 2013).

Contest duration

Contest duration refers to the time frame during which solvers can

actively submit solutions. When designing a crowdsourcing project,

it is important to consider the contest duration, because this could

affect the overall quality of solutions (Ayaburi et al., 2020;

Leimeister et al., 2009). In a study of 12 crowdsourcing projects

for idea generation, Muhdi et al. (2011) find that most of the ideas

were submitted within the first 4 weeks of the contest being

online. However, it is important to point out that these findings

could differ with the nature of the problem. For instance, for

problems with a higher level of complexity and specific solution

requirements, solvers may need more time and effort to develop

high-quality solutions (Ayaburi et al., 2020). Chen et al. (2021) find

that although a higher contest duration increases the amount of

solvers and thus the likelihood of receiving high-quality solutions,

they also report a decrease in the attraction of high-quality contes-

tants. Ultimately, it is crucial for managers to take into account the

difficulty of the problem, the specificity of the solution require-

ments and the level of skill and expertise required when defining

the duration of the contest.

4.2.2 | Crowd

The crowd refers to the people that participate in a crowdsourcing

activity. The crowd is one of the most important actors in the

crowdsourcing system (Zhao & Zhu, 2014a). The success of any

crowdsourcing initiative largely depends on the ability to attract and

motivate a crowd to develop solutions (Ford et al., 2015). Based on a

literature analysis, three crucial characteristics of the crowd are

identified, and the central points are summarized in Table 4.

Crowd motivation and incentives

Crowd motivation refers to the motivation of the crowd to participate

in innovation contests. Many previous studies have investigated

the motivation aspect of crowdsourcing (Acar, 2019; Brabham, 2010;

Frey et al., 2011; Garcia Martinez, 2017; Leimeister et al., 2009;

Pee et al., 2018; Zhao & Zhu, 2014a; Zheng et al., 2011). Because

incentives are an inherent component of tournament-based

crowdsourcing (Blohm et al., 2018; Leimeister et al., 2009), a thorough

understanding of what motivates the crowd is important for

crowdsourcing firms when designing incentive mechanisms. The

award money, as a central extrinsic motivator, increases both the

number of solutions and the solution quality (Chen et al., 2021). How-

ever, as Leimeister et al. (2009) explore different incentives that moti-

vate individuals to participate in ideas competitions, they find that

incentives providing direct compensation (extrinsic) are not the only

motivating factor. Other forms of motivation such as appreciation and

learning through interaction with knowledge experts and mentors

(intrinsic) are also important. Similarly, Brabham (2010) identifies four

primary motivators for participation in crowdsourcing initiatives: the

opportunity to make money, the opportunity to develop one's skills,

the opportunity to take up full-time work and the love of community.

Hanine and Steils (2019) state that negative feelings must be avoided,

for instance, through transparency and encouraging a positive and

respectful climate. In particular, perceived fairness of a crowdsourcing

contest increases the likelihood of crowd participation (Mazzola

et al., 2020). Görzen (2021) complements these findings on perceived

feelings and reports that a meaningful task can stimulate positive

mood among solvers, which positively impacts the creativity of

solutions, and as such, task meaningfulness is considered an indirect

motivator (Görzen, 2021).

Prior research has further linked the nature and complexity of the

crowdsourced problem with the motivation of solvers to participate

(Afuah & Tucci, 2012; Garcia Martinez, 2017; Ghezzi et al., 2018; Li &

Hu, 2017; Pee et al., 2018; Zheng et al., 2011). For instance, Pee

et al. (2018) demonstrate that participants that are motivated to

develop competence focus on high-commitment tasks, whereas those

motivated by the ‘love of community’focus on tasks that require

interaction between solvers. Zheng et al. (2011) state that tasks that

are ill-structured and poorly delineated may have a negative influence

TABLE 4 Design-related elements in the crowd dimension

Crowd dimension Design-related relevance

Crowd motivation

and incentives

Indication of incentives for the crowd to

participate in the contest

Conveys monetary and/or non-monetary

incentives

Incentives should consider emerging costs

(time and effort) for the crowd to develop

solutions

Knowledge

diversity

Crowdsourcer needs to specify the required

knowledge diversity

Determines the crowd to be targeted

Crowd size Crowdsourcer needs to determine the amount

of required solvers

Address an open crowd or selected

participants (experts) contingent on

required knowledge diversity

KARACHIWALLA AND PINKOW 573

the motivation to participate. On the other hand, well-structured tasks

with high level of autonomy have a positive influence on participation

in crowdsourcing contests (Garcia Martinez, 2017; Lee et al., 2015).

Another crucial aspect that plays a significant role in participant

motivation is the treatment of intellectual property. Stringent intellec-

tual property arrangements could significantly discourage participa-

tion in crowdsourcing contests (Mazzola et al., 2018). Prior studies

indicate that intellectual property decisions are typically dependent

on the complexity and stage of development of the problems (de Beer

et al., 2017; Mazzola et al., 2018). Crowdsourcing firms should con-

sider the negative influence on participation when designing intellec-

tual property arrangements. For instance, one possible way to

motivate solvers that have to fully transfer rights is to offer signifi-

cantly higher monetary rewards (Mazzola et al., 2018).

Although prior research suggests that both intrinsic and extrinsic

motivation is important in crowdsourcing contests (Zhao &

Zhu, 2014a), crowdsourcing firms must take into account the nature

and complexity of the problem when designing incentives. For prob-

lems with a higher complexity that require more time and effort,

financial incentives are particularly important (Afuah & Tucci, 2012;

Blohm et al., 2018; Jeppesen & Lakhani, 2010; Schenk &

Guittard, 2011; Zhao & Zhu, 2014a). Managers should also find ways

to incentivize solvers that are intrinsically motivated to develop com-

petencies and learn from experience. For instance, Leimeister

et al. (2009) show that solvers that are motivated to develop their

skills appreciate feedback from experts. Therefore, incorporating

feedback mechanisms can be very helpful to foster learning and

competence development.

Knowledge diversity

Knowledge diversity refers to the range of knowledge, skills and

expertise of the crowd members. The required knowledge to solve a

problem is closely related to the complexity of the task, which is rep-

resented by the number of knowledge components involved in the

problem (Ford et al., 2015; Natalicchio et al., 2017; Thuan

et al., 2016). Some tasks such as software testing require highly spe-

cialized knowledge (Afuah & Tucci, 2012; Blohm et al., 2018), whereas

other generic problems rely on the heterogeneity of the crowd

(Steils & Hanine, 2016). Boons and Stam (2019) argue that the ability

to combine and integrate ‘related’(specific domain knowledge) and

‘unrelated’(other domain knowledge) perspectives are key in generat-

ing high-quality, novel solutions. Similarly, Frey et al. (2011) highlight

that individuals with knowledge in diverse areas are better able to

combine knowledge and make connections. Though knowledge diver-

sity plays an important role in crowdsourcing contests, making the dis-

tinction between problems that require specific knowledge versus

generic problems is essential for crowdsourcing companies to benefit

from crowdsourcing initiatives.

Crowd size

Crowd size refers to the number of solvers participating in a

crowdsourcing contest. Crowd size is an important element of

a crowdsourcing initiative, as it has a direct impact on the quantity

and quality of solutions received. Afuah and Tucci (2012) highlight

that because knowledge and expertise are widely dispersed among

the crowd, a larger solver base increases the possibility of receiving

higher quality solutions. Boudreau et al. (2011) point out that in

tournament-based crowdsourcing contests in which few winning

solutions are selected, the larger the number of participants, the

less likely it is for individual contestants to win, which in turn

reduces the effort exerted by individuals in developing solutions.

However, for problems that draw on multiple knowledge domains

and that do not have a specific problem-solving approach, this

effect is reversed. In other words, a large (diverse) crowd could

lead to better performance for problems with greater knowledge

uncertainty. On the other hand, for problems in which a specific

knowledge domain is required and predefined solutions exist,

targeting professionals with the appropriate skills and expertise

could be a more effective approach. Therefore, managers should

take into account the nature of the problem when deciding

whether to address an open crowd (unlimited contestants) or to

target individuals with specific expertise to develop solutions.

4.2.3 | Platform

The platform refers to the interface through which a firm broadcasts

the problem to be solved, and the design-related considerations are

indicated in Table 5.

Organizations can either develop their own crowdsourcing plat-

form or use third-party (intermediary-based crowdsourcing) platforms.

Although many renowned enterprises (including Dell, SAP, Google,

LEGO and Procter & Gamble) have successfully developed their own

crowdsourcing platforms (eYeka, 2015), it is important to take into

account the costs of setting up, operating and managing such plat-

forms (Ford et al., 2015; Schenk et al., 2019). Blohm et al. (2018) fur-

ther highlight that developing scalable platforms with appropriate

governance mechanisms can be very challenging for firms with no

prior experience in crowdsourcing. Another crucial aspect is the

access to a large crowd with diverse skills and expertise. The success

of any crowdsourcing initiative largely depends on the ability to

attract and motivate a crowd to develop solutions (Ford et al., 2015).

This is especially relevant for tournament-based crowdsourcing for

challenging, innovative problems.

TABLE 5 Design-related elements in the platform dimension

Platform

dimension Design-related relevance

Platform Connects crowdsourcer and crowd

Considers the costs associated with developing an

own platform or using existing (external)

crowdsourcing platforms as intermediary

Depends on existing crowdsourcing experience

Platform specifies the size of the network of

potential solvers

574 KARACHIWALLA AND PINKOW

Over the past decade, the market for crowdsourcing intermedi-

aries has grown significantly (Diener & Piller, 2013). Some well-known

examples include InnoCentive, NineSigma, IdeaConnection and Yet2.

These intermediaries have a large global network of experts and pro-

fessionals in diverse fields and play a mediating role by connecting the

seeker firm with external solvers via their own web-based platform

(Diener & Piller, 2013; Leicht et al., 2016). Because crowdsourcing

intermediaries differ in expertise (Colombo et al., 2013; Diener &

Piller, 2013), firms must select the right one, based on the nature and

complexity of problem to be solved.

Recent developments indicate that many organizations have

turned to intermediary-based crowdsourcing to broadcast innovation

problems. Intermediaries play a key role in managing the

crowdsourcing process (Zogaj et al., 2014), including formulating

the problem statement, broadcasting the task to their solver commu-

nity, preselecting appropriate solutions and providing feedback to

solvers. Furthermore, intermediaries support seeker firms by providing

advice, managing intellectual property and associated risks and track-

ing overall crowdsourcing performance (Colombo et al., 2013;

Diener & Piller, 2013; Leicht et al., 2016). Prior research suggests that

using intermediaries can significantly decrease development costs and

other risks associated with crowdsourcing, therefore making it an

attractive problem-solving approach for firms (Ford et al., 2015;

Zhao & Zhu, 2014a). Therefore, the choice between establishing an

internal platform and using an external intermediary is a crucial deci-

sion in the crowdsourcing process (Ford et al., 2015; Schenk

et al., 2019; Thuan et al., 2016; Zhao & Zhu, 2014a).

4.2.4 | Crowdsourcer

The crowdsourcer refers to the organization seeking to solve a task

through crowdsourcing. The crowdsourcing firm is responsible for

designing the overall contest starting from formulating the problem

statement to attracting solvers and finally evaluation and implementa-

tion of crowdsourced ideas. Based on a review of prior literature, nine

design elements corresponding to the crowdsourcing firm are

extracted and illustrated in Table 6.

Solution evaluation

Solution evaluation refers to how firms assess the solutions developed

by the crowd. In tournament-based crowdsourcing, participants typi-

cally compete with each other to generate solutions to a defined

problem. Crowdsourcing firms then screen and evaluate the set of

solutions received to select the best one(s), which are ultimately

rewarded (Geiger & Schader, 2014).

Although crowdsourcing provides the opportunity to tap into

diverse knowledge that is distributed among the crowd, it can also

lead to a state of ‘crowding’, in which organizations received a large

number of solutions (Mack & Landau, 2020; Piezunka &

Dahlander, 2015). Because organizations have limited resources,

evaluating large sets of solutions can be tedious and increase the

overall transaction cost of crowdsourcing (Afuah & Tucci, 2012;

Blohm et al., 2013; Ye & Kankanhalli, 2015). Piezunka and

Dahlander (2015) further highlight that as crowding increases, firms

tend to limit their attention to solutions that are familiar and within

TABLE 6 Design-related elements in the crowdsourcer dimension

Crowdsourcer

dimension Design-related relevance

Solution

evaluation

Internal evaluation process of received

solutions

Determine how to deal with a large number of

solutions

Implementation

potential

Determine how to transform solutions into

valuable information

Establish criteria to determine the technical and

economic feasibility of solutions

Top management commitment

Communication

and feedback

Create communication channels to

communicate with the crowd

Determine adequate communication forms and

flows

Create feedback structures and channels

Incentives and

rewards

Define potential intrinsic motivators that are

conveyed through the nature of the task

Provide external motivators (e.g. monetary

rewards)

Align rewards with required effort and time the

crowd needs to develop solutions

Consider amount of potentially winning

solution(s)

Resource planning Assess internal expertise and experience on

crowdsourcing

Determine all associated costs (financial, time,

personnel)

Allocate sufficient resources in advance

Create informal organizational roles

(gatekeepers, champions)

Risk management Identify potential uncertainties (e.g. receiving

only inferior solutions)

Identify sources of potential threats to

intellectual property (IP) that may be exposed

through the crowdsourcing contest

Consider data protection/data privacy

Legal and IP

management

Determine the legal terms and conditions, and

IP rights arrangements

Determine the ownership of the IP created

through the crowdsourced solutions

Brand image and

trust

Consider marketing-related aspects of

crowdsourcing

Determine the influence crowdsourcing firm's

brand on the perception of the crowd with

regards to the task

Success metrics Assess the overall success of a crowdsourcing

contest

Recognize failures and encourage learning

KARACHIWALLA AND PINKOW 575

the local knowledge domain. Consequently, when crowding occurs,

organizations tend to filter out solutions that include distant knowl-

edge that could be potentially relevant. Therefore, in order to benefit

from the diversity (local and distant solutions), firms should establish

clear evaluation criteria through which relevant solutions are selected.

Defining clear guidelines for solution requirements in the problem

statement can be helpful for solvers to develop solutions that better

meet the expectations of seeking firms (Afuah & Tucci, 2012; Blohm

et al., 2018). Previous studies also indicate that different evaluation

tools and methods can be used when assessing crowdsourced ideas

(Blohm et al., 2018; Geiger & Schader, 2014; Ghezzi et al., 2018;

Muhdi et al., 2011). Blohm et al. (2018) point out that in tournament-

based crowdsourcing, manual assessment is crucial because auto-

mated tools could potential overlook relevant contributions. To

reduce the transaction costs from crowding, firms can integrate peer

assessment techniques in the contest design, in which other crowd

contributors can also rate the quality of contributions. As such, the

jury evaluating solution may be composed of the end users of the

solution, who can judge the value of solutions based on their own

needs and requirements (Afuah & Tucci, 2012). In this context, Chen

et al. (2020) find the benefit that when the crowd itself can vote for

the winning solutions, the overall motivation to participate in a con-

test can be increased. For solutions that primarily will be used by the

crowdsourcing firm, and not by other end users, the firm must employ

internal evaluators with sufficient knowledge (Afuah & Tucci, 2012).

As such, internal experts that have sufficient domain knowledge

should be involved in evaluating ideas received from crowdsourcing

contests, if the firm evaluates the received solutions internally.

Implementation potential

Implementation potential refers to the ability to utilize the solutions

received by the crowd. Crowdsourcing contests often generate an

overwhelming number of solutions, and therefore, organizations must

be prepared to effectively transform relevant solutions into valuable

information for the company (Afuah & Tucci, 2012; Blohm

et al., 2013). Prior studies suggest that in order to benefit from

crowdsourcing initiatives, firms must develop distinct capabilities to

integrate and transfer externally developed solutions (Afuah &

Tucci, 2012; Ford et al., 2015). Firms need to carefully assess the

technical and economic feasibility of crowdsourced solutions. Blohm

et al. (2013) further highlight that solutions received by the crowd

may need to be modified and tailored to fit the exact internal needs of

the company.

Muhdi et al. (2011) stress the importance of a concrete imple-

mentation plan to better manage and transfer crowdsourced ideas

either into existing projects or to initiate new projects. Communica-

tion of crowdsourcing project results to other business units within

the organization can also increase the potential for implementation

(Blohm et al., 2013). Finally, the support and commitment from the

top management is instrumental for overcoming internal resistance

towards externally developed solutions, making the transformation of

crowdsourced ideas faster and easier (Ford et al., 2015; Lüttgens

et al., 2014).

Communication and feedback

Communication and feedback refer to how crowdsourcing firms com-

municate with the crowd at different stages of the contest. Many

studies affirm that communication and feedback to solvers is a critical

component, which can significantly influence the quality of solutions

received (Blohm et al., 2013; Camacho et al., 2019; Chan et al., 2021;

Jian et al., 2019; Leimeister et al., 2009; Piezunka & Dahlander, 2019;

Schäfer et al., 2017; Wooten & Ulrich, 2017).

Schäfer et al. (2017) distinguish three types of communication

flows in crowdsourcing contests: unidirectional (suggestion boxes),

bidirectional (e-mail) and multidirectional (forums, wikis). The authors

further outline which communication flow is best suited for different

stages of the contest. The study revealed that unidirectional and mul-

tidirectional communication are most valuable before the contest and

bidirectional communication during and after the contest (Schäfer

et al., 2017). Communicating with crowd members during the runtime

of the contest increases the chances of high-quality submissions

(Blohm et al., 2018; Zheng et al., 2014) because, in some cases, solvers

may require additional information about the problem to develop

appropriate solutions (Jian et al., 2019; Schäfer et al., 2017). A further

feedback mechanism that firms can employ is creating peer-feedback

structures, such that members of the crowd can provide feedback to

other members (Chan et al., 2021). In fact, both peer- and firm-

feedback impact solvers' motivation to improve solutions and lead to

high-quality ideas (Chan et al., 2021). Though in-process communica-

tion is important, feedback after the contest, especially when solu-

tions are rejected, are particularly important (Piezunka &

Dahlander, 2019; Schäfer et al., 2017). For solvers that are intrinsically

motivated to learn and develop competencies (Leimeister et al., 2009),

providing constructive feedback can play a critical role in participation

in future contests (Piezunka & Dahlander, 2019).

In order to benefit from crowdsourcing initiatives, crowdsourcing

firms should establish appropriate communication tools for different

stages of the crowdsourcing process. Additionally, firms should allo-

cate time to communicate and answer questions from solvers (Muhdi

et al., 2011). Trained moderators to provide constructive feedback at

appropriate phases of the contest can highly influence the participa-

tion and success rates of crowdsourcing projects (Camacho

et al., 2019).

Incentives and rewards

Incentives and rewards refer to the remuneration that solvers receive

in exchange for winning solutions. As previously discussed, incentives

are one of the most critical components in tournament-based

crowdsourcing (Blohm et al., 2018; Leimeister et al., 2009). Although

prior research indicates that both intrinsic and extrinsic motivation

influence participation in crowdsourcing contests (Lee et al., 2015;

Zhao & Zhu, 2014a; Zheng et al., 2011), the nature and complexity of

the problem plays a critical role when designing effective incentive

structures. For problems with a higher complexity that require partici-

pants to invest substantial time and effort, financial incentives are par-

ticularly important (Afuah & Tucci, 2012; Blohm et al., 2018;

Schenk & Guittard, 2011; Zhao & Zhu, 2014a). Furthermore, for

576 KARACHIWALLA AND PINKOW

problems that require solvers to transfer IPR, firms must ensure that

the rewards are sufficiently high (Mazzola et al., 2018).

Though intrinsic motivators such as passion or personal achieve-

ment cannot be directly controlled (Geiger & Schader, 2014;

Leimeister et al., 2009), crowdsourcing firms should try to incorporate

elements that promote learning and competence development in

incentive mechanisms, for instance, the possibility to communicate

and receive feedback from experts or potential collaboration opportu-

nities to further develop promising solutions (Leimeister et al., 2009).

Tournament-based crowdsourcing is typically associated with

success-based remuneration, which means that only successful contri-

butions are rewarded (Geiger & Schader, 2014). As a result, the higher

the number of potential contributors, the smaller the chance of win-

ning for individual contributors. This could potentially discourage

solvers from investing time and effort into developing solutions,

which in turn could negatively influence the performance of the con-

test (Boudreau et al., 2011). Therefore, instead of following the ‘win-

ner takes it all’approach, crowdsourcing firms should consider

distributing awards among the top contributors (Blohm et al., 2018;

Muhdi et al., 2011).

From a managerial perspective, designing appropriate incentive

structures are very important for the success of crowdsourcing pro-

jects. Crowdsourcing firms must take into account the nature and

complexity of the problem crowdsourced, the type of intellectual

property arrangement and the different intrinsic and extrinsic motiva-

tors when defining awards and incentives.

Resource planning

Resource planning refers to management of resources (time, human

capital and financial capital) in a crowdsourcing project. Prior

crowdsourcing literature suggests that when making the decision to

crowdsource, one of the most important factors that must be consid-

ered is whether an organizational has sufficient internal expertise to

solve the problem (Thuan et al., 2016; Ye & Kankanhalli, 2015). Orga-

nizations typically engage in crowdsourcing initiatives when they do

not possess the required expertise or knowledge (Afuah &

Tucci, 2012). Another critical factor that influences the decision to

crowdsource is the cost of running crowdsourcing projects. Whereas

many studies argue that crowdsourcing can significantly reduce devel-

opment costs (Afuah & Tucci, 2012; Boudreau & Lakhani, 2013;

Brabham, 2008; Jeppesen & Lakhani, 2010; Vukovic, 2009), some

studies suggest that dedicated budgets are required to effectively

carry out crowdsourcing projects (Ford et al., 2015; Thuan

et al., 2016). Therefore, crowdsourcing firms must consider the differ-

ent transaction costs involved, for instance, the cost to develop a plat-

form (or hire an intermediary), the cost of internal human resources

and the cost of incentivizing the crowd (Ford et al., 2015; Ye &

Kankanhalli, 2015).

Managing crowdsourcing initiatives is similar to managing pro-

jects. Allocating sufficient resources and defining a concrete project

plan with clear milestones are important (Ford et al., 2015; Muhdi

et al., 2011). Furthermore, competent managers with crowdsourcing

experience are critical for the success of crowdsourcing projects

(Ford et al., 2015). Lüttgens et al. (2014) recommend that in addition

to getting the strong commitment from the top management, creating

informal organizational roles such as gatekeepers and champions can

be particularly beneficial in overcoming organizational resistance and

barriers in crowdsourcing projects. Therefore, from a resource per-

spective, it is crucial for firms to weigh the costs and benefits when

deciding whether to crowdsource or not and to ensure the commit-

ment of employees to effectively support and manage crowdsourcing

initiatives.

Risk management

Risk management refers to the management of uncertainties in

the context of crowdsourcing projects. Although most previous

research has focused on the different benefits of crowdsourcing,

it is equally important for firms to consider the potential

risks involved in crowdsourcing projects. One of the most promi-

nent risks in crowdsourcing projects is the possibility of receiving

inferior solutions of low quality (Liu, Xia, Zhang, Pan, &

Zhang, 2016; Liu, Xia, Zhang, & Wang, 2016; Nirosh Kannangara &

Uguccioni, 2013). Because crowd members are not confined to

employment contracts, firms may not have effective control

over the quality of the output (Nirosh Kannangara &

Uguccioni, 2013). However, by defining the problem explicitly,

providing clear solution requirements and addressing an appropriate

crowd, organizations can reduce the risk of receiving poor quality

solutions.

Another critical risk in crowdsourcing initiatives is the loss of

intellectual property and knowhow. Crowdsourcing projects run the

risk of revealing too much information when delineating a problem,

which could negatively impact competitive advantage (Ford

et al., 2015; Nirosh Kannangara & Uguccioni, 2013). Prior studies sug-

gests that tasks with highly confidential information are not ideal for

crowdsourcing (Afuah & Tucci, 2012; Ghezzi et al., 2018; Nakatsu

et al., 2014; Natalicchio et al., 2017). Therefore, crowdsourcing firms

should be cautious when defining problem statements and ensure that

no sensitive information is revealed. Including non-disclosure agree-

ments could also be effective when working with external crowds

(de Beer et al., 2017).

Other risks related to information and data security include

violation of personal data and malicious activity on crowdsourcing

platforms (Sauerwein et al., 2016). To ensure platform security,

performing penetration tests to evaluate the vulnerability of the

system can be an effective measure.

Similar to any other projects, crowdsourcing projects also involve

risks. Therefore, organizations must carefully assess potential risks,

and define measures to mitigate them, before embarking on

crowdsourcing initiatives.

Legal and IPR management

Legal and IPR management refer to the legal terms and conditions and

intellectual property mechanisms in crowdsourcing projects.

IPR arrangements can significantly impact the participation and

performance in crowdsourcing contests (de Beer et al., 2017;

KARACHIWALLA AND PINKOW 577

Foege et al., 2019; Mazzola et al., 2018); therefore, firms need to

carefully decide which intellectual property treatment to use when

designing crowdsourcing projects.

de Beer et al. (2017) distinguish four different approaches to

manage intellectual property in crowdsourcing projects based on the

degree of ownership and the potential to reduce liabilities associated

with crowdsourced solutions. The degree of ownership refers to the

degree to which organizations acquire IPR of crowdsourced solutions.

In the case of high degree of ownership, seekers have exclusive con-

trol over the intellectual property, but in the case of low degree of

ownership, the solvers retain exclusive rights, which means that the

IPR can be licensed out to other parties. Reducing liabilities refers to

the degree to which organizations protect themselves from liabilities

associated with crowdsource solutions, for instance, third-party intel-

lectual property that may possibly be embedded in the solutions.

Mazzola et al. (2018) highlight that firms should also consider the

complexity and the stage of development of the crowdsourced prob-

lem when deciding the degree of ownership. For problems with higher

complexity or those related to the later stages of development, seek-

ing firms generally prefer to have a high degree of ownership because

the potential value generated is larger. In such situations,

crowdsourcing firms should ensure that contributors are adequately

compensated (Blohm et al., 2018; Foege et al., 2019). From a solver

perspective, when IPR arrangements are too stringent, individuals are

less motivated to participate in crowdsourcing contests because they

are not always willing to give up ownership (de Beer et al., 2017;

Mazzola et al., 2018). Because crowds are not protected by employ-

ment regulations, a fair and balanced approach to manage intellectual

property is important (de Beer et al., 2017). Therefore, firms should

prioritize in which circumstances retaining exclusive control can be

beneficial and define appropriate remuneration (Foege et al., 2019).

By defining explicit terms and conditions, including non-

disclosure agreements and outlining appropriate intellectual property

arrangement in the problem statement, firms can ensure widespread

participation as well as protect themselves from legal contamination

(de Beer et al., 2017).

Brand image and trust

Crowdsourcing provides the opportunity for firms to gain visibility

and advertise their brand (Ye & Kankanhalli, 2015). The Lego Ideas

platform is an excellent example of creating a strong brand image

and enhancing customer loyalty through crowdsourcing. With an

established online community of almost one million members, Lego

actively works together with the crowd to convert promising ideas

into tangible products. Through its crowdsourcing platform, Lego

has attracted an increasing number of loyal customers and

fans, who are enthusiastic about developing new Lego products

(PD., 2018).

Another related aspect is developing a sense of trust among

solvers (Blohm et al., 2013; Garcia Martinez, 2017; Liu, Xia, Zhang,

Pan, & Zhang, 2016). When solvers associate with the brand, they

may be more motivated to develop solutions. Therefore, when design-

ing crowdsourcing projects, firms should ensure a sense of trust and

perceived fairness. Not only does this encourage participation but also

provides the opportunity to create a strong brand image.

Success metrics

Success metrics are important to evaluate the overall performance

and effectiveness of crowdsourcing initiatives. Crowdsourcing firms

should develop specific metrics to track the success of crowdsourcing

outcomes (Blohm et al., 2013; Ford et al., 2015). Although it is clear

that not every crowdsourcing project may result in high-quality solu-

tions, establishing success criteria can be helpful for organizations to

recognize failures and foster learning (Ford et al., 2015).

5|A MORPHOLOGICAL FRAMEWORK ON

CROWDSOURCING CONTEST DESIGN

In this section, the results of the literature analysis are consolidated to

present a concrete, holistic overview. In particular, a morphological

framework is developed to better structure and investigate the differ-

ent design elements in a crowdsourcing project. The morphological

approach was first popularized by Zwicky (1969) to ‘study the more

abstract structural interrelations among phenomena, concepts, and

ideas’(Ritchey, 2013, p. 3). A morphological analysis is essentially a

systematic approach to structure and analyse multidimensional prob-

lems by identifying and investigating possible relationships or configu-

rations in the system (Ritchey, 2013). This approach is an alternative

to other quantitative modeling methods and is particularly useful

when organizing and synthesizing qualitative aspects to determine dif-

ferent possible outcomes.

The development of a morphological framework typically

begins by identifying important ‘dimensions’of the overall system.

Consequently, the dimensions can be further broken down into sub-

dimensions (in this case elements and sub-elements). For each

dimension (and sub-dimension), possible ‘values’or attributes are

identified and structured into a matrix, known as a morphological box

(Ritchey, 2013). By organizing the different design parameters and

possible options in a morphological box, it presents an integrative,

visual representation of the overall solution space (Ritchey, 2006). In

other words, the box seeks to uncover different possible solutions to

a problem by accommodating multiple alternative perspectives rather

than prescribing a single solution (Ritchey, 2006). This allows the

possibility of choosing different combinations of options for each

design parameter, best suited to the goals of the problem.

A morphological framework is specifically chosen for this study

because it provides a cohesive, integrative picture of the findings in

the literature. Table 7 presents the morphological framework devel-

oped in the context of crowdsourcing contest design. The matrix com-

prises the four fundamental pillars of crowdsourcing: the task, the

crowd, the platform and the crowdsourcer (Hosseini et al., 2014). For

each dimension, corresponding design parameters (16 elements and

12 sub-elements) are identified through a rigorous systematic litera-

ture review. Consequently, for each parameter, possible ‘values’or

attributes are outlined. By assigning options to each parameter, the

578 KARACHIWALLA AND PINKOW

morphological box serves as a well-structured framework with differ-

ent potential configurations, allowing managers to make well-

informed decisions when designing crowdsourcing contests.

6|DISCUSSION

Crowdsourcing contests are a promising and innovative approach in

the field of open innovation to tap into a global pool of widely distrib-

uted external knowledge. In consideration of the fundamental phase

of designing a crowdsourcing contest, this paper sought to answer the

question that design-related factors organizations must consider.

Existing literature informs practitioners about specific factors that

help to succeed with crowdsourcing a task to an external crowd.

However, the lack of a theory-based yet practicably applicable frame-

work on crowdsourcing renders efficient crowdsourcing contest

design a time-consuming effort. Therefore, by applying a decision-

centric view, we provide a systematic review on design-related

elements that help in guiding practitioners and future research.

6.1 |Theoretical implications

From a theoretical perspective, this study responds to the calls for fur-

ther research regarding the ‘lack of standardization’and the need for

a‘comprehensive guideline’to better understand and manage

crowdsourcing projects (Amrollahi, 2015; Neto & Santos, 2018). By

synthesizing two distinct theoretical perspectives on crowdsourcing,

social exchange theory and absorptive capacity, we provide an inte-

grative conceptual approach that pursues the goal to determine which

factors contribute to executing successful crowdsourcing contests.

Though many studies focus on the crowd participation motiva-

tion, and as such on the expected benefits (e.g. Acar, 2019; Zhao &

Zhu, 2014a; Zheng et al., 2011), the crowdsourcing firm must also

consider which costs it signals to the crowd by delineating the task

(Ye & Kankanhalli, 2015). We draw on the principles of the social

exchange theory to elaborate which potential costs the crowd per-

ceives and argue that only when individual crowd members perceive a

positive expected net gain from a cost–benefit analysis based on the

information conveyed through the task description, they engage in

TABLE 7 A morphological framework of crowdsourcing design elements

Dimensions Elements

Sub-elements

(if applicable) Attributes

Task Task delineation Task specificity Low Moderate High

Task granularity Simple Moderate Complex

Task modularity Decomposed Undecomposed

Solution requirements Bounded Unbounded

Task allocation Performance

based

Demographic based Skill based Open

Contest duration Short Medium Long

Crowd Motivation Intrinsic Extrinsic Both

Knowledge diversity Related Unrelated Both

Size Small Medium Large

Platform Ownership Own platform Intermediary

platform

Crowdsourcer Solution evaluation Expert evaluation Peer evaluation Both

Implementation potential Technical feasibility Low Moderate High

Economic feasibility Low Moderate High

Feedback and

communication

Uni-directional Bi-directional Multi-

directional

All

Incentives and awards Financial Non-financial Both

Resource planning Financial resources Low Moderate High

Time resources Low Moderate High

Personnel resources Low Moderate High

Risk management Solution quality risk Low Moderate High

Loss of IP risk Low Moderate High

Platform & Security

Risk

Low Moderate High

Legal and IP management Passive Prudent Persuasive Possessive

Brand image impact Low Medium High

Success metrics Quantitative Qualitative Both

KARACHIWALLA AND PINKOW 579

developing solutions. Thereby, we add to extant literature that the

identified design elements cannot be considered sequentially, but our

results indicate that they are closely related. Although a crowdsourcer

must determine internal factors such as deciding on the evaluation

process or a deliberate risk management, these factors impact the

way how the crowdsourcer can describe the task and thereby convey

the costs and benefits to the crowd in terms of time and effort versus

intrinsic and extrinsic motivation.

Bayus (2013) shows that some firms are able to establish own

crowdsourcing platforms and communities over time and thus estab-

lish long-term relationships with potential solvers who provide solu-

tions to several crowdsourced tasks over time. This consideration

adds to the idea of social exchange to develop a lasting relationship

between two parties (Blau, 1964), and our results indicate that

crowdsourcing is a learning experience for firms as well. Although

crowdsourcers may initially utilize intermediary platforms to connect

to the crowd, they gain experience in executing crowdsourcing con-

tests and may establish their own platform over time. However, this

implies that the crowdsourcer can benefit, or capture value, from the

received solutions.

Thus, we add an absorptive capacity perspective in order to

account for the value that organizations aim to create through

crowdsourcing contests. Bloodgood (2013) emphasizes that capturing

value is a fundamental issue that impacts the decision how to solve

problems, with crowdsourcing being one approach. Thus, effective

crowdsourcing requires certain organizational capabilities to capture

value. We accounted for this consideration by integrating an absorptive

capacity perspective to our research design. First, absorptive capacity is

built through deliberate internal planning, referring to the crowdsourcer

dimension in our morphological approach. Second, though the

absorptive capacity primarily emerges from internal processes of the

crowdsourcer, it impacts the way how the task can be communicated to

the crowd. As such, crowdsourcers must be aware that building

absorptive capacity should be part of the design process of a

crowdsourcing contest, as it impacts the crowdsourced task and contrib-

utes to defining the solution space for the task. Thus, the concept of

absorptive capacity is implicitly included in the task dimension, because

task-related design decision may emerge or be based on internal

decisions that aim to create the ability to capture value from the

submitted solutions. This finding extends previous research that

investigated the task-related elements by the perspective that the task

description and delineation not only conveys the motivation to engage

in crowdsourcing and provide high-quality solutions but also determines

whether crowdsourcers can ultimately benefit from these solutions.

6.2 |Managerial implications

From a practical perspective, this paper provides several valuable

insights for practitioners and managers undertaking crowdsourcing

projects. In recent years, crowdsourcing contests have become an

increasingly popular innovation management practice to leverage the

knowledge and wisdom of external crowds. In spite of its widespread

adoption in practice, it is important to point out that not all

crowdsourcing initiatives are an immediate success, and therefore, a

thorough understanding of the underlying dynamics is crucial. This

study specifically aims to bridge this gap by presenting an integrative

account of the key design elements of a crowdsourcing contest. By

adopting a morphological approach, this paper provides different

configurations of crowdsourcing projects, allowing managers to

choose alternative solutions for each design parameter. The findings

of this study aim to serve a comprehensive guideline and blueprint

through which managers can effectively carry out crowdsourcing

contests and use this guideline as a checklist. Using the morphological

approach presented in this study allows practitioners to consider

elements that relate to both the crowd's motivation to engage

in the crowdsourcing contest and the internal management of

crowdsourcing. There are two central managerial implications that we

derive from this study.

First, attracting the crowd requires a comprehensive perspective

on the crowd's perception on associated costs and benefits for solving

a task. Although the crowdsourcing firm can explicitly determine the

extrinsic motivation in form of monetary rewards, the intrinsic motiva-

tion is more difficult to assess. However, intrinsic motivation plays a

fundamental role, and the crowdsourcer can emphasize certain issues

in the task description that relate to intrinsic motivation, such as

promising feedback to submitted solutions (Hanine & Steils, 2019).

Whereas motivation is one aspect that the crowd considers for partic-

ipation in crowdsourcing, another aspect is the related costs in terms

of time and effort that occur in the solution development. The

crowdsourcer must make sure to provide adequate benefits, such that

contingent on the task complexity, the rewards and incentives must

be adapted accordingly. This requires that the crowdsourcing firm is

aware of the complexity of the task and the potential expectations of

the crowd. Sourcing specific knowledge to a complex problem might

be more costly than sourcing diverse knowledge to a rather simple

problem. This consideration must be addressed during the early

design phase of a crowdsourcing contest in order to attract the right

crowd members to develop high-quality solutions.

Second, crowdsourcing firms must define the factors that deter-

mine whether crowdsourcing a specific task ultimately can provide

value to the firm. It may be particularly difficult for the crowdsourcer

to determine the internal workload associated with conducting a

crowdsourcing campaign (Hanine & Steils, 2019). Thus, a first step is

to explicitly consider the related internal costs of resources, such as

personnel. Moreover, the crowdsourcer must be aware whether the

required knowledge sourced from the crowd should be local or distant

to the firm, because the resulting crowdsourcing design is contingent

on this requirement (Jespersen, 2018). When crowdsourcing a task is

evaluated as promising based on an internal evaluation, the second

requirement to capture value from crowdsourcing is to make sure to

receive high-quality solutions. Thus, from a general perspective, cap-

turing value comprises both the decision taken internally related to

the organizational processes associated with crowdsourcing and the

decisions taken related to the way how the task is communicated to

the crowd.

580 KARACHIWALLA AND PINKOW

6.3 |Limitations and future research

As with any systematic literature review, this paper is not without lim-

itations. Although the morphological approach presents a structured

overview of the potential design configurations, we acknowledge that

it does not capture potential relationships or interdependencies

between individual design options. Although this is a relevant aspect,

we perceive these interdependencies to be beyond the scope of the

current study due to the chosen morphological approach. Because

designing a crowdsourcing campaign entails multifold interrelated

decisions, future research is encouraged to explore the interrelation-

ships among the various design elements, as certain decision taken

might confine the decision opportunities for other elements. In partic-

ular, further research should investigate whether there is a certain

hierarchy between design choices such that certain decisions taken

early on consequently influence other decisions and thereby limit the

decision space of other design elements.

A further limitation is related to crowd motivation. Though crowd

motivation as a central factor for crowdsourcing is accounted for in

this study, there are more potential advancements that can be made

to the suggested morphology. We suggest that it could be interesting

to investigate how the crowdsourcer can foster rewards that corre-

spond to the intrinsic motivation of individuals to participate in

crowdsourcing, for instance, how crowdsourcers can implement

rewards in the design phase of a crowdsourcing contest that relate to

the motivation of social recognition as a central intrinsic motivator

(Hanine & Steils, 2019). This requires the crowdsourcer to understand

the crowd composition and what determines prestige within the

crowd communities. These rather subjective factors perceived by

the crowd, for example, prestige within communities, cannot be

directly designed by the crowdsourcer and thus are not included in

the morphology. Although we are well aware that such scoping deci-

sions also go along with excluding potentially relevant but more indi-

rect effects of crowdsourcing design approaches, we perceive this to

be beyond the scope of this study. We acknowledge that collabora-

tive crowdsourcing models, which enable active discussions of partici-

pants and sharing of solutions and encourage community building,

may increase community belonging and promote higher levels of

solver activity and better solutions (Bayus, 2013; Boudreau &

Lakhani, 2013; Hutter et al., 2011; Vuculescu & Bergenholtz, 2014).

Though these aspects of design effects are certainly relevant, in the

context of this paper, we primarily focus on innovation contests,

wherein individuals compete with one another to develop the best

solutions. Therefore, we chose to focus on central design aspects and

their systematic development rather than aiming at the discussion of

all potentially indirect effects of campaigns. The latter certainly

deserves attention from further research. Our study thus provides an

avenue for exploring relevant but more indirect effects of design

decisions.

Moreover, because the primary focus of this study is on

crowdsourcing contests for innovation, the results may be different in

other crowdsourcing settings, such as open collaboration, micro-

tasking or information pooling. Though we provide an initial starting

point, we encourage future research to investigate other types of

crowdsourcing using the morphological box and adapt the box accord-

ingly, if required.

Lastly, the morphological framework needs to be validated in a

practical setting. Although the morphological approach is theory-

based and derived from extant research, we see great potential in

applying this approach to practice. Future research could test the

framework on crowdsourcing contests to fill any gaps and strengthen

the validity of the framework. Moreover, the morphological approach

could serve to identify archetypes of crowdsourcing contests, which

potentially demonstrate certain patterns across the morphological box

presented in this study. This could serve to facilitate future

crowdsourcing contests by defining the nature of the task, determin-

ing the crowdsourcing archetype and following the corresponding

suggestions of the morphological framework.

DATA AVAILABILITY STATEMENT

Data sharing not applicable - no new data generated

ORCID

Rea Karachiwalla https://orcid.org/0000-0002-1852-8888

Felix Pinkow https://orcid.org/0000-0002-3680-5682

REFERENCES

Acar, O. A. (2019). Motivations and solution appropriateness in

crowdsourcing challenges for innovation. Research Policy,48(8),

103716. https://doi.org/10.1016/j.respol.2018.11.010

Adamczyk, S., Bullinger, A. C., & Möslein, K. M. (2012). Innovation

contests: A review, classification and outlook. Creativity and

Innovation Management,21(4), 335–360. https://doi.org/10.1111/

caim.12003

Afuah, A., & Tucci, C. L. (2012). Crowdsourcing as a solution to distant

search. Academy of Management Review,37(3), 355–375. https://doi.

org/10.5465/amr.2010.0146

Allahbakhsh, M., Benatallah, B., Ignjatovic, A., Motahari-Nezhad, H. R.,

Bertino, E., & Dustdar, S. (2013). Quality control in crowdsourcing

systems: Issues and directions. IEEE Internet Computing,17(2), 76–81.

https://doi.org/10.1109/MIC.2013.20

Allon, G., & Babich, V. (2020). Crowdsourcing and crowdfunding in the

manufacturing and services sectors. Manufacturing & Service Opera-

tions Management,22(1), 102–112. https://doi.org/10.1287/msom.

2019.0825

Amrollahi, A. (2015). A process model for crowdsourcing: Insights from the

literature on implementation. In: Australasian Conference on Informa-

tion Systems, Adelaide. https://arxiv.org/pdf/1605.04695

Ayaburi, E. W., Lee, J., & Maasberg, M. (2020). Understanding

crowdsourcing contest fitness strategic decision factors and perfor-

mance: An expectation-confirmation theory perspective. Information

Systems Frontiers,22(5), 1227–1240. https://doi.org/10.1007/

s10796-019-09926-w

Bayus, B. L. (2013). Crowdsourcing new product ideas over time: An analy-

sis of the Dell IdeaStorm community. Management Science,59(1),

226–244. https://doi.org/10.1287/mnsc.1120.1599

Blau, P. M. (1964). Exchange and power in social life. John Wiley &

Sons, Inc.

Blohm, I., Leimeister, J. M., & Krcmar, H. (2013). Crowdsourcing: How to

benefit from (too) many great ideas. MIS Quarterly Executive,12(4),

199–211.

Blohm, I., Zogaj, S., Bretschneider, U., & Leimeister, J. M. (2018). How

to manage crowdsourcing platforms effectively? California

KARACHIWALLA AND PINKOW 581

Management Review,60(2), 122–149. https://doi.org/10.1177/

0008125617738255

Bloodgood, J. (2013). Crowdsourcing: Useful for problem solving, but what

about value capture? Academy of Management Review,38(3), 455–457.

https://doi.org/10.5465/amr.2012.0318

Bogers, M., & West, J. (2012). Managing distributed innovation: Strategic

utilization of open and user innovation. Creativity and Innovation

Management,21(1), 61–75. https://doi.org/10.1111/j.1467-8691.

2011.00622.x

Boons, M., & Stam, D. (2019). Crowdsourcing for innovation: How related

and unrelated perspectives interact to increase creative performance.

Research Policy,48(7), 1758–1770. https://doi.org/10.1016/j.respol.

2019.04.005

Boudreau, K. J., Lacetera, N., & Lakhani, K. R. (2011). Incentives and

problem uncertainty in innovation contests: An empirical analysis.

Management Science,57(5), 843–863. https://doi.org/10.1287/mnsc.

1110.1322

Boudreau, K. J., & Lakhani, K. R. (2013). Using the crowd as an innovation

partner. Harvard Business Review,91(4), 60–69, 140.

Brabham, D. C. (2008). Crowdsourcing as a model for problem solving.

Convergence: The International Journal of Research into New Media Tech-

nologies,14(1), 75–90. https://doi.org/10.1177/1354856507084420

Brabham, D. C. (2009). Crowdsourcing the public participation process for

planning projects. Planning Theory,8(3), 242–262. https://doi.org/10.

1177/1473095209104824

Brabham, D. C. (2010). Moving the crowd at Threadless. Information,

Communication & Society,13(8), 1122–1145. https://doi.org/10.1080/

13691181003624090

Camacho, N., Nam, H., Kannan, P. K., & Stremersch, S. (2019). Tourna-

ments to crowdsource innovation: The role of moderator feedback

and participation intensity. Journal of Marketing,83(2), 138–157.

https://doi.org/10.1177/0022242918809673

Cappa, F., Oriani, R., Pinelli, M., & de Massis, A. (2019). When does

crowdsourcing benefit firm stock market performance? Research

Policy,48(9), 103825. https://doi.org/10.1016/j.respol.2019.103825

Chan, K. W., Li, S. Y., Ni, J., & Zhu, J. J. (2021). What feedback matters?

The role of experience in motivating crowdsourcing innovation. Pro-

duction and Operations Management,30(1), 103–126. https://doi.org/

10.1111/poms.13259

Chen, L., Xu, P., & Liu, D. (2020). Effect of crowd voting on participation in

crowdsourcing contests. Journal of Management Information Systems,

37(2), 510–535. https://doi.org/10.1080/07421222.2020.1759342

Chen, P.-Y., Pavlou, P., Wu, S., & Yang, Y. (2021). Attracting high-quality

contestants to contest in the context of crowdsourcing contest plat-

form. Production and Operations Management. Advance Online Publica-

tion. https://doi.org/10.1111/poms.13340

Chesbrough, H. W. (2003). Open innovation: The new imperative for creating

and profiting from technology. Harvard Business School Press.

Chesbrough, H. W., & Bogers, M. (2014). Explicating open innovation:

Clarifying an emerging paradigm for understanding innovation. In

H. W. Chesbrough, W. Vanhaverbeke, & J. West (Eds.), New frontiers

in open innovation (pp. 3–28). Oxford University Press.

Christensen, I., & Karlsson, C. (2019). Open innovation and the effects of

crowdsourcing in a pharma ecosystem. Journal of Innovation & Knowl-

edge,4(4), 240–247. https://doi.org/10.1016/j.jik.2018.03.008

Cohen, W. M., & Levinthal, D. A. (1990). Absorptive capacity: A new per-

spective on learning and innovation. Administrative Science Quarterly,

35(1), 128. https://doi.org/10.2307/2393553

Colombo, G., Buganza, T., Klanner, I.-M., & Roiser, S. (2013).

Crowdsourcing intermediaries and problem typologies: An explorative

study. International Journal of Innovation Management,17(02),

1350005. https://doi.org/10.1142/S1363919613500059

Cropanzano, R., & Mitchell, M. S. (2005). Social exchange theory: An inter-

disciplinary review. Journal of Management,31(6), 874–900. https://

doi.org/10.1177/0149206305279602

de Beer, J., McCarthy, I. P., Soliman, A., & Treen, E. (2017). Click here to

agree: Managing intellectual property when crowdsourcing solutions.

Business Horizons,60(2), 207–217. https://doi.org/10.1016/j.bushor.

2016.11.002

Diener, K., & Piller, F. (2013). The market for open innovation. Raleigh, NC,

USA: Lulu. https://doi.org/10.13140/RG.2.2.24961.35682

Enkel, E., Gassmann, O., & Chesbrough, H. W. (2009). Open R&D and open

innovation: Exploring the phenomenon. R&D Management,39(4),

311–316. https://doi.org/10.1111/j.1467-9310.2009.00570.x

Estellés-Arolas, E., & González-Ladr

on-de-Guevara, F. (2012). Towards an

integrated crowdsourcing definition. Journal of Information Science,

38(2), 189–200. https://doi.org/10.1177/0165551512437638

eYeka. (2015). The state of crowdsourcing in 2015. https://eyeka.pr.co/

99215-eyeka-releases-the-state-of-crowdsourcing-in-2015-trend-

report

Foege, J. N., Lauritzen, G. D., Tietze, F., & Salge, T. O. (2019). Recon-

ceptualizing the paradox of openness: How solvers navigate sharing-

protecting tensions in crowdsourcing. Research Policy,48(6),

1323–1339. https://doi.org/10.1016/j.respol.2019.01.013

Ford, R. C., Richard, B., & Ciuchta, M. P. (2015). Crowdsourcing: A new

way of employing non-employees? Business Horizons,58(4), 377–388.

https://doi.org/10.1016/j.bushor.2015.03.003

Frey, K., Lüthje, C., & Haag, S. (2011). Whom should firms attract to open

innovation platforms? The role of knowledge diversity and motivation.

Long Range Planning,44(5–6), 397–420. https://doi.org/10.1016/j.lrp.

2011.09.006

Gao, H., Wang, X., Barbier, G., & Liu, H. (2011). Promoting coordination for

disaster relief –From crowdsourcing to coordination. In J. Salerno,

S. J. Yang, D. Nau, & S.-K. Chai (Eds.), Social computing, behavioral-

cultural modeling and prediction: 4th international conference, SBP 2011,

College Park, MD, USA, March 29-31, 2011; proceedings (Vol. 6589).

Lecture Notes in Computer Science. (pp. 197–204). Springer.

Garcia Martinez, M. (2017). Inspiring crowdsourcing communities to create

novel solutions: Competition design and the mediating role of trust.

Technological Forecasting and Social Change,117, 296–304. https://

doi.org/10.1016/j.techfore.2016.11.015

Gassenheimer, J. B., Siguaw, J. A., & Hunter, G. L. (2013). Exploring moti-

vations and the capacity for business crowdsourcing. AMS Review,

3(4), 205–216. https://doi.org/10.1007/s13162-013-0055-8

Geiger, D., & Schader, M. (2014). Personalized task recommendation

in crowdsourcing information systems—Current state of the art.

Decision Support Systems,65,3–16. https://doi.org/10.1016/j.dss.

2014.05.007

Ghezzi, A., Gabelloni, D., Martini, A., & Natalicchio, A. (2018).

Crowdsourcing: A review and suggestions for future research. Interna-

tional Journal of Management Reviews,20(2), 343–363. https://doi.org/

10.1111/ijmr.12135

Gillier, T., Chaffois, C., Belkhouja, M., Roth, Y., & Bayus, B. L. (2018). The

effects of task instructions in crowdsourcing innovative ideas. Techno-

logical Forecasting and Social Change,134,35–44. https://doi.org/10.

1016/j.techfore.2018.05.005

Görzen, T. (2021). “What's the point of the task?”exploring the influence

of task meaning on creativity in crowdsourcing. International Journal of

Innovation Management,25(01), 2150007. https://doi.org/10.1142/

S1363919621500079

Hanine, S., & Steils, N. (2019). Ideation contests: Crowd management and

valorization to avoid negative feelings of participants. Creativity

and Innovation Management,28(4), 425–435. https://doi.org/10.1111/

caim.12325

Hetmank, L. (2013). Components and functions of crowdsourcing systems

- A systematic Literature review. In: Wirtschaftsinformatik Proceed-

ings 2013.

Hossain, M. (2015). Crowdsourcing in business and management disci-

plines: An integrative literature review. Journal of Global Entrepreneur-

ship Research,5(1). https://doi.org/10.1186/s40497-015-0039-2

582 KARACHIWALLA AND PINKOW

Hosseini, M., Phalp, K., Taylor, J., & Ali, R. (2014). The four pillars of

crowdsourcing: A reference model. In M. Bajec (Ed.), 2014 IEEE eighth

international conference on research challenges in information science

(RCIS): Marrakech, Morocco, 28-30 May 2014 (pp. 1–12). IEEE.

Howe, J. (2006). The rise of crowdsourcing. Wired Magazine,14(6), 1–4.

Hutter, K., Hautz, J., Füller, J., Mueller, J., & Matzler, K. (2011).

Communitition: The tension between competition and collaboration in

community-based design contests. Creativity and Innovation Manage-

ment,20(1), 3–21. https://doi.org/10.1111/j.1467-8691.2011.

00589.x

Jeppesen, L. B., & Lakhani, K. R. (2010). Marginality and problem-solving

effectiveness in broadcast search. Organization Science,21(5),

1016–1033. https://doi.org/10.1287/orsc.1090.0491

Jespersen, K. R. (2018). Crowdsourcing design decisions for optimal inte-

gration into the company innovation system. Decision Support Systems,

115,52–63. https://doi.org/10.1016/j.dss.2018.09.005

Jian, L., Yang, S., Ba, S., Lu, L., & Jiang, L. C. (2019). Managing the crowds:

The effect of prize guarantees and in-process feedback on participa-

tion in crowdsourcing contests. MIS Quarterly,43(1), 97–112. https://

doi.org/10.25300/MISQ/2019/13649

Kazman, R., & Chen, H.-M. (2009). The metropolis model a new logic for

development of crowdsourced systems. Communications of the ACM,

52(7), 76–84. https://doi.org/10.1145/1538788.1538808

Koh, T. K. (2019). Adopting seekers' solution exemplars in crowdsourcing

ideation contests: Antecedents and consequences. Information Systems

Research,30(2), 486–506. https://doi.org/10.1287/isre.2018.0810

Kohler, T. (2015). Crowdsourcing-based business models: How to create

and capture value. California Management Review,57(4), 63–84.

https://doi.org/10.1525/cmr.2015.57.4.63

Lee, C., Chan, C. Y., Ho, S., Choy, K. L., & Ip, W. H. (2015). Explore the fea-

sibility of adopting crowdsourcing for innovative problem solving.

Industrial Management & Data Systems,115(5), 803–832. https://doi.

org/10.1108/IMDS-09-2014-0249

Leicht, N., Durward, D., Haas, P., Zogaj, S., Blohm, I., & Leimeister, J. M.

(2016). An empirical taxonomy of crowdsourcing intermediaries. Acad-

emy of Management Proceedings,2016(1), 17518. https://doi.org/10.

5465/ambpp.2016.17518abstract

Leimeister, J. M., Huber, M., Bretschneider, U., & Krcmar, H. (2009).

Leveraging crowdsourcing: Activation-supporting components for IT-

based ideas competition. Journal of Management Information Systems,

26(1), 197–224. https://doi.org/10.2753/mis0742-1222260108

Li, D., & Hu, L. (2017). Exploring the effects of reward and competition

intensity on participation in crowdsourcing contests. Electronic Mar-

kets,27(3), 199–210. https://doi.org/10.1007/s12525-017-0252-7

Liu, S., Xia, F., Zhang, J., Pan, W., & Zhang, Y. (2016). Exploring the trends,

characteristic antecedents, and performance consequences of

crowdsourcing project risks. International Journal of Project Manage-

ment,34(8), 1625–1637. https://doi.org/10.1016/j.ijproman.2016.

09.002

Liu, S., Xia, F., Zhang, J., & Wang, L. (2016). How crowdsourcing risks

affect performance: An exploratory model. Management Decision,54

(9), 2235–2255. https://doi.org/10.1108/MD-12-2015-0604

Lüttgens, D., Pollok, P., Antons, D., & Piller, F. (2014). Wisdom of the

crowd and capabilities of a few: Internal success factors of

crowdsourcing for innovation. Journal of Business Economics,84(3),

339–374. https://doi.org/10.1007/s11573-014-0723-7

Mack, T., & Landau, C. (2020). Submission quality in open innovation con-

tests - An analysis of individual-level determinants of idea innovative-

ness. R&D Management,50(1), 47–62. https://doi.org/10.1111/radm.

12345

Malone, T., Laubacher, R., & Dellarocas, C. (2010). The collective intelli-

gence genome. IEEE Engineering Management Review,38(3), 38–52.

https://doi.org/10.1109/EMR.2010.5559142

Mazzola, E., Acur, N., Piazza, M., & Perrone, G. (2018). “To own or not to

own?”A study on the determinants and consequences of alternative

intellectual property rights arrangements in crowdsourcing for innova-

tion contests. Journal of Product Innovation Management,35(6),

908–929. https://doi.org/10.1111/jpim.12467

Mazzola, E., Piazza, M., Acur, N., & Perrone, G. (2020). Treating the crowd

fairly: Increasing the solvers' self-selection in idea innovation contests.

Industrial Marketing Management,91,16–29. https://doi.org/10.1016/

j.indmarman.2020.07.019

Muhdi, L., Daiber, M., Friesike, S., & Boutellier, R. (2011). The

crowdsourcing process: An intermediary mediated idea generation

approach in the early phase of innovation. International Journal of

Entrepreneurship and Innovation Management,14(4), 43052. https://

doi.org/10.1504/ijeim.2011.043052

Nakatsu, R. T., Grossman, E. B., & Iacovou, C. L. (2014). A taxonomy of

crowdsourcing based on task complexity. Journal of Information Sci-

ence,40(6), 823–834. https://doi.org/10.1177/0165551514550140

Natalicchio, A., Messeni Petruzzelli, A., & Garavelli, A. C. (2017). Innovation

problems and search for solutions in crowdsourcing platforms –A sim-

ulation approach. Technovation,64-65,28–42. https://doi.org/10.

1016/j.technovation.2017.05.002

Neto, F. R. A., & Santos, C. A. (2018). Understanding crowdsourcing pro-

jects: A systematic review of tendencies, workflow, and quality man-

agement. Information Processing & Management,54(4), 490–506.

https://doi.org/10.1016/j.ipm.2018.03.006

Nirosh Kannangara, S., & Uguccioni, P. (2013). Risk management in

crowdsourcing-based business ecosystems. Technology Innovation

Management Review,3(12), 32–38. https://doi.org/10.22215/

timreview/751

Okoli, C., & Schabram, K. (2010). A guide to conducting a systematic

literature review of information systems research. SSRN Electronic

Journal. Advance Online Publication. https://doi.org/10.2139/ssrn.

1954824

Paik, J. H., Scholl, M., Sergeev, R., Randazzo, S., & Lakhani, K. R. (2020).

Innovation contests for high-tech procurement. Research-Technology

Management,63(2), 36–45. https://doi.org/10.1080/08956308.2020.

1707007

PD. (2018). Building together: How LEGO leverages crowdsourcing to sus-

tain both innovation and brand love. HBS Digital Innovation and

Transformation. https://digital.hbs.edu/platform-digit/submission/

building-together-how-lego-leverages-crowdsourcing-to-sustain-both-

innovation-and-brand-love/

Pedersen, J., Kocsis, D., Tripathi, A., Tarrell, A., Weerakoon, A.,

Tahmasbi, N., Xiong, J., Deng, W., Oh, O., & de Vreede, G.-J. (2013).

Conceptual foundations of crowdsourcing: A review of IS research. In:

2013 46th Hawaii International Conference on System Sciences

(pp. 579–588). IEEE. https://doi.org/10.1109/HICSS.2013.143

Pee, L. G., Koh, E., & Goh, M. (2018). Trait motivations of crowdsourcing

and task choice: A distal-proximal perspective. International Journal of

Information Management,40,28–41. https://doi.org/10.1016/j.

ijinfomgt.2018.01.008

Piezunka, H., & Dahlander, L. (2015). Distant search, narrow attention:

How crowding alters organizations' filtering of suggestions in

crowdsourcing. Academy of Management Journal,58(3), 856–880.

https://doi.org/10.5465/amj.2012.0458

Piezunka, H., & Dahlander, L. (2019). Idea rejected, tie formed: Organiza-

tions' feedback on crowdsourced ideas. Academy of Management Jour-

nal,62(2), 503–530. https://doi.org/10.5465/amj.2016.0703

Pollok, P., Lüttgens, D., & Piller, F. T. (2019a). Attracting solutions in

crowdsourcing contests: The role of knowledge distance, identity dis-

closure, and seeker status. Research Policy,48(1), 98–114. https://doi.

org/10.1016/j.respol.2018.07.022

Pollok, P., Lüttgens, D., & Piller, F. T. (2019b). How firms develop capabili-

ties for crowdsourcing to increase open innovation performance: The

interplay between organizational roles and knowledge processes.

Journal of Product Innovation Management,36(4), 412–441. https://

doi.org/10.1111/jpim.12485

KARACHIWALLA AND PINKOW 583

Ren, J., Han, Y., Genc, Y., Yeoh, W., & Popovicˇ, A. (2021). The boundary of

crowdsourcing in the domain of creativity. Technological Forecasting

and Social Change,165, 120530. https://doi.org/10.1016/j.techfore.

2020.120530

Ritchey, T. (2006). Problem structuring using computer-aided morphologi-

cal analysis. The Journal of the Operational Research Society,57(7),

792–801. https://doi.org/10.1057/palgrave.jors.2602177

Ritchey, T. (2013). General morphological analysis: A general method for non-

quantified modelling. Swedish Morphological Society.

Rouse, A. C. (2010). A preliminary taxonomy of crowdsourcing. In: ACIS

2010 Proceedings. http://aisel.aisnet.org/acis2010/76

Ruiz,



E., Brion, S., & Parmentier, G. (2020). Absorbing knowledge in the

digital age: The key role of integration mechanisms in the context of

crowdsourcing for innovation. R&D Management,50(1), 63–74.

https://doi.org/10.1111/radm.12349

Salomo, S., Talke, K., & Strecker, N. (2008). Innovation field orientation

and its effect on innovativeness and firm performance. Journal of Prod-

uct Innovation Management,25(6), 560–576. https://doi.org/10.1111/

j.1540-5885.2008.00322.x

Sauerwein, C., Gander, M., Felderer, M., & Breu, R. (2016). A systematic

literature review of crowdsourcing-based research in information secu-

rity. In 2016 IEEE Symposium on Service-Oriented System Engineering

(SOSE) (pp. 364–371). IEEE. https://doi.org/10.1109/SOSE.2016.67

Schäfer, S., Antons, D., Lüttgens, D., Piller, F., & Salge, T. O. (2017). Talk to

your crowd. Research-Technology Management,60(4), 33–42. https://

doi.org/10.1080/08956308.2017.1325689

Schäper, T., Foege, J. N., Nüesch, S., & Schäfer, S. (2021). Determinants of

idea sharing in crowdsourcing: Evidence from the automotive industry.

R&D Management,51(1), 101–113. https://doi.org/10.1111/radm.12429

Schenk, E., & Guittard, C. (2011). Towards a characterization of

crowdsourcing practices. Journal of Innovation Economics,7(1), 93.

https://doi.org/10.3917/jie.007.0093

Schenk, E., Guittard, C., & Pénin, J. (2019). Open or proprietary? Choosing

the right crowdsourcing platform for innovation. Technological Fore-

casting and Social Change,144, 303–310. https://doi.org/10.1016/j.

techfore.2017.11.021

Snyder, H. (2019). Literature review as a research methodology: An over-

view and guidelines. Journal of Business Research,104, 333–339.

https://doi.org/10.1016/j.jbusres.2019.07.039

Steils, N., & Hanine, S. (2016). Creative contests: Knowledge generation

and underlying learning dynamics for idea generation. Journal of Mar-

keting Management,32(17–18), 1647–1669. https://doi.org/10.1080/

0267257X.2016.1251956

Terwiesch, C., & Xu, Y. (2008). Innovation contests, open innovation, and

multiagent problem solving. Management Science,54(9), 1529–1543.

https://doi.org/10.1287/mnsc.1080.0884

Thuan, N. H., Antunes, P., & Johnstone, D. (2016). Factors influencing the

decision to crowdsource: A systematic literature review. Information

Systems Frontiers,18(1), 47–68. https://doi.org/10.1007/s10796-015-

9578-x

Vuculescu, O., & Bergenholtz, C. (2014). How to solve problems with

crowds: A computer-based simulation model. Creativity and Innovation

Management,23(2), 121–136. https://doi.org/10.1111/caim.12059

Vukovic, M. (2009). Crowdsourcing for enterprises. In: 2009 Congress on

Services - I (pp. 686–692). IEEE. https://doi.org/10.1109/SERVICES-I.

2009.56

Webster, J., & Watson, R. T. (2002). Analyzing the past to prepare for the

future: Writing a literature review. MIS Quarterly,26(2), xiii–xxiii.

Wooten, J. O., & Ulrich, K. T. (2017). Idea generation and the role of feed-

back: Evidence from field experiments with innovation tournaments.

Production and Operations Management,26(1), 80–99. https://doi.org/

10.1111/poms.12613

Ye, H., & Kankanhalli, A. (2015). Investigating the antecedents of organiza-

tional task crowdsourcing. Information & Management,52(1), 98–110.

https://doi.org/10.1016/j.im.2014.10.007

Ye, H., & Kankanhalli, A. (2017). Solvers' participation in crowdsourcing

platforms: Examining the impacts of trust, and benefit and cost factors.

The Journal of Strategic Information Systems,26(2), 101–117. https://

doi.org/10.1016/j.jsis.2017.02.001

Zhao, Y. C., & Zhu, Q. (2014a). Effects of extrinsic and intrinsic motivation

on participation in crowdsourcing contest. Online Information Review,

38(7), 896–917. https://doi.org/10.1108/OIR-08-2014-0188

Zhao, Y. C., & Zhu, Q. (2014b). Evaluation on crowdsourcing research: Cur-

rent status and future direction. Information Systems Frontiers,16(3),

417–434. https://doi.org/10.1007/s10796-012-9350-4

Zheng, H., Li, D., & Hou, W. (2011). Task design, motivation, and participa-

tion in crowdsourcing contests. International Journal of Electronic Com-

merce,15(4), 57–88. https://doi.org/10.2753/jec1086-4415150402

Zheng, H., Xie, Z., Hou, W., & Li, D. (2014). Antecedents of solution quality

in crowdsourcing: The sponsor's perspective. Journal of Electronic Com-

merce Research,15(3), 212–224.

Zogaj, S., Bretschneider, U., & Leimeister, J. M. (2014). Managing

crowdsourced software testing: A case study based insight on the

challenges of a crowdsourcing intermediary. Journal of Business Eco-

nomics,84(3), 375–405. https://doi.org/10.1007/s11573-014-0721-9

Zwicky, F. (1969). Discovery, invention, research through the morphological

approach. Macmillan.

AUTHOR BIOGRAPHIES

Rea Karachiwalla holds a Master of Science from the European

School of Management and Technology in Berlin. Since 2018, she

is an industrial PhD candidate at the Chair of Technology and

Innovation Management at Technische Universität Berlin. Work-

ing alongside in a large automotive company, her primary interests

are the operationalization of new innovation methods and tools.

Her main research focus is open innovation and crowdsourcing, in

particular exploring the design of crowdsourcing projects for inno-

vation. Her work was presented at international conferences, such

as the Innovation and Product Development Management Con-

ference and the Open and User Innovation Conference.

Felix Pinkow holds a Master of Science in Management from Uni-

versity of Mannheim. Since 2018, he works as a Research Fellow

and PhD candidate at the Chair of Technology and Innovation

Management at Technische Universität Berlin. His primary

research interests concern creativity and creative thinking pro-

cesses, in particular the relationship between affect and creativity.

Further research topics include open innovation, crowdsourcing,

and crowdfunding. His work was presented at international con-

ferences, such as the Innovation and Product Development Man-

agement Conference, the JPIM Research Forum, and the Open

and User Innovation Conference.

How to cite this article: Karachiwalla R, Pinkow F.

Understanding crowdsourcing projects: A review on the key

design elements of a crowdsourcing initiative. Creat Innov

Manag. 2021;30:563–584. https://doi.org/10.1111/caim.

12454

584 KARACHIWALLA AND PINKOW