Gamified crowdsourcing : Conceptualization, literature review, and future agenda

1 Gamified Crowdsourcing: Conceptualization, Literature Review, and Future Agenda Benedikt Morschheuser Institute of Information Systems and Marketing, Karlsruhe Institute of Technology, Germany Corporate Research, Robert Bosch GmbH, Germany benedikt.morsc[email protected] Juho Hamari Gamification Group, Tampere University of Technology, Finland Gamification Group, University of Turku, Finland [email protected] Jonna Koivisto Gamification Group, Tampere University of Technology, Finland [email protected] Alexander Maedche Institute of Information Systems and Marketing, Karlsruhe Institute of Technology, Germany [email protected] Correspondence: Benedikt Morschheuser Karlsruhe Institute of Technology (KIT) Institute of Information Systems and Marketing (IISM) Fritz-Erler-Straße 23 76131 Karlsruhe benedikt.morsc[email protected] T: +49 177 347 4435 Cite: Morschheuser, B., Hamari, J., Koivisto, J., & Maedche, A. (2017). Gamified crowdsourcing: Conceptualization, literature review, and future agenda. International Journal of Human-Computer Studies, 106, 26-43. doi:https://doi.org/10.1016/j.ijhcs.2017.04.005 This is the accepted manuscript of the article, which has been published in International Journal of Human-Computer Studies. 2017, 106, 26-43. https://doi.org/10.1016/j.ijhcs.2017.04.005 2 ABSTRACT Two parallel phenomena are gaining attention in human-computer interaction research: gamification and crowdsourcing. Because crowdsourcing’s success depends on a mass of motivated crowdsourcees, crowdsourcing platforms have increasingly been imbued with motivational design features borrowed from games; a practice often called gamification. While the body of literature and knowledge of the phenomenon have begun to accumulate, we still lack a comprehensive and systematic understanding of conceptual foundations, knowledge of how gamification is used in crowdsourcing, and whether it is effective. We first provide a conceptual framework for gamified crowdsourcing systems in order to understand and conceptualize the key aspects of the phenomenon. The paper’s main contributions are derived through a systematic literature review that investigates how gamification has been examined in different types of crowdsourcing in a variety of domains. This meticulous mapping, which focuses on all aspects in our framework, enables us to infer what kinds of gamification efforts are effective in different crowdsourcing approaches as well as to point to a number of research gaps and lay out future research directions for gamified crowdsourcing systems. Overall, the results indicate that gamification has been an effective approach for increasing crowdsourcing participation and the quality of the crowdsourced work; however, differences exist between different types of crowdsourcing: the research conducted in the context of crowdsourcing of homogenous tasks has most commonly used simple gamification implementations, such as points and leaderboards, whereas crowdsourcing implementations that seek diverse and creative contributions employ gamification with a richer set of mechanics. Keywords: gamification, crowdsourcing, literature review, research agenda, human computation, persuasive technology 3 1 INTRODUCTION During recent years, modern ICT technologies have spawned two parallel phenomena: gamification and crowdsourcing. Today, many different organizations employ crowdsourcing as a way to outsource various tasks to be carried out by ‘the crowd’: a mass of people reachable through the Internet (Howe, 2006). The rapid diffusion of these technologies can be seen both in practice and in academia (Estellés-Arolas and González-Ladrón-de-Guevara, 2012; Hamari et al., 2014; IEEE, 2014; Seaborn and Fels, 2015). As of December 2015, almost 3,000 crowdsourcing-related examples are listed at crowdsourcing.org, a leading crowdsourcing industry portal. In parallel, business analysts have estimated that at least 50% of all organizations that manage innovation processes have gamified some of their processes by 2015 (Gartner, 2011). The primary general goals of crowdsourcing are either cost savings or the possibility to handle tasks that would be difficult to perform without human support. However, crowdsourcing relies on the existence of a reserve of people willing to take on tasks for free or for little monetary compensation. Along this reasoning, crowdsourcing systems are increasingly gamified (Hamari et al., 2014; Seaborn and Fels, 2015), that is, organizations seek to make the crowdsourced work activity more like playing a game in order to provide other motives for working than just monetary compensation. Such gamified crowdsourcing systems are increasing, and are a major application area of gamification (Hamari et al., 2014). However, while the new phenomenon seems intuitively appealing, there is little coherent understanding of the characteristic features of gamified crowdsourcing systems. Although there are singular scattered empirical pieces on the topic, no efforts have yet been made to collate and synthesize this body of knowledge. Further, both crowdsourcing and gamification can take a variety of forms, and it would be myopic to assume that differing gamification implementations would function similarly across different crowdsourcing approaches. This lack of comprehensive understanding of the phenomenon inhibits us from designing effective incentive systems for crowdsourcing and therefore to optimally harness the potential of the crowd and to derive the most successful solutions and innovations. In this paper, we provide a comprehensive review, overview, and future outlook on the usage and study of gamification in crowdsourcing systems. We first provide an integrated conceptual framework 4 for gamified crowdsourcing systems (Figure 3), based on the extant literature on crowdsourcing (Geiger and Schader, 2014; Prpić et al., 2015) and gamification (Hamari et al., 2014; Seaborn and Fels 2015). This framework remedies existing conceptual hurdles and scantness in how gamification, crowdsourcing, and their combinations are generally perceived, and acts both as a framework to direct this review and as an anchor point for further studies. The primary contribution of the paper is a systematic literature review of 110 papers that investigates how gamification is being studied and implemented in crowdsourcing research. Specifically, we review the use of different forms of gamification in different types of crowdsourcing, as well as the interplay of gamification and monetary rewards, the types of work being crowdsourced, the types of crowdsourcees, the domains where gamification in crowdsourcing have been applied, and empirical results of studies on the effectiveness of gamification in crowdsourcing. This meticulous mapping enables us to 1) infer what kinds of gamification efforts are effective in different kinds of crowdsourcing approaches, 2) derive recommendations for designers of gamified crowdsourcing systems, and 3) outline a research agenda for future research. 2 CONCEPTUAL FOUNDATIONS 2.1 Crowdsourcing Generally, crowdsourcing can be seen as an online, distributed problem-solving approach that transforms problems and tasks into solutions by harnessing the potential of large groups of crowdsourcees via the Web rather than traditional employees or suppliers (Brabham, 2008a; Doan et al., 2011; Estellés-Arolas and González-Ladrón-de-Guevara, 2012; Howe, 2006; Nakatsu et al., 2014; Pedersen et al., 2013; Prpić et al., 2015; Zuchowski et al., 2016). Via the rise of online collaboration technologies and Web2.0, it has become fairly easy to reach large groups of people. Thus, the concept of crowdsourcing has become increasingly popular (Gatautis and Vitkauskaite, 2014; Geiger and Schader, 2014; Rouse, 2010; Zuchowski et al., 2016). There has been an increase in the number of startups with crowdsourcing-based business models (Brabham, 2010, 2008b) and many companies have begun to invest in internal and external crowdsourcing (Leimeister et al., 2009; Schlagwein and BjørnAndersen, 2014; Zuchowski et al., 2016). Crowdsourcing is considered a particularly useful way to 5 coordinate work for tasks that can benefit from collective intelligence (Leimeister, 2010) or that are hard to process by computers and are therefore outsourced to people (Von Ahn, 2009). Figure 1. Four Archetypes of Crowdsourcing Systems (based on Geiger and Schader, 2014) Following the conceptual works of Geiger and Schader (2014) and Prpić et al. (2015)1, crowdsourcing systems can be categorized into four categories, depending on the characteristics of the crowdsourced work (see Figure 1). First, crowdprocessing approaches rely on the crowd to perform large quantities of homogeneous tasks. Identical contributions are a quality attribute of the work’s validity. The value is derived directly from each isolated contribution (non-emergent) (e.g. Mechanical Turk or Galaxy Zoo) (Lintott et al., 2008). Second, crowdsolving approaches use the diversity of the crowd to find a huge number of heterogeneous solutions to a given problem. The value of this approach results directly from each isolated contribution (non-emergent). Crowdsolving is often used for very complex problems (e.g. Foldit, a game-based approach to optimize protein folding) (Cooper et al., 2010) or if no pre-definable solution exists (e.g. ideation contests). Third, crowdrating systems commonly seek to harness the so-called wisdom of crowds (Surowiecki, 2005) to perform collective assessments or predictions. In this case, the emergent value arises from a huge number of homogeneous ‘votes’ (e.g. 1 The frameworks of Geiger and Schader (2014) as well as Prpić et al. (2015) classify crowdsourcing into four categories that are comparable at their core. For clarity, we employed Geiger and Schader’s (2014) terminology. 6 NASA Clickworkers, in which the clicks/votes of a crowd were used to identify craters on asteroids) (Kanefsky et al., 2001). Fourth, crowdcreating solutions seek to create comprehensive (emergent) artifacts based on a variety of heterogeneous contributions. Typical examples include all kinds of usergenerated content (e.g. YouTube) or knowledge derived from collaborative aggregation (e.g. Wikipedia). 2.2 Gamification Since an active crowd of participants is crucial for successful crowdsourcing, the motivation of crowdsourcees is crucial (Zhao and Zhu, 2014a). Although much research has been done in the area of crowdsourcing, only a few studies have comprehensively investigated participants’ motivations (e.g. Brabham, 2010, 2008b; Kaufmann et al., 2011; Zhao and Zhu, 2014b; Zheng et al., 2011) and incentive design (e.g. Harris et al., 2015; Leimeister et al., 2009; Straub et al., 2015). Studies have shown that a wide variety of reasons and motivations, ranging from intrinsic to extrinsic, lead people to participate in crowdsourcing and related online work and economic coordination (Hamari et al., 2016; Kaufmann et al., 2011; Straub et al., 2015; Zhao and Zhu, 2014b; Zheng et al., 2011). For instance, intrinsic motivation – caused by tasks that allow a participant to be creative and experience autonomy, to develop own skills and feel competent, to enjoy a pastime, or to achieve social recognition – can in some cases be dominated by extrinsic motivation evoked by financial payoffs or external social reasons (Kaufmann et al., 2011). Further, task characteristics (Kaufmann et al., 2011; Zheng et al., 2011), task granularity (Nakatsu et al., 2014; Zhao and Zhu, 2014b), or perceived motivational affordances (Zhao and Zhu, 2014b) can further influence an individual’s motivation. Thus, one major challenge in motivating people to participate is to design a crowdsourcing system that promotes and enables the formation of positive motivations towards crowdsourcing work and fits the type of the activity. For instance, while some crowdsourcing approaches aim for systematically derived contributions, others may call for incentive structures that promote creativity. In other words, since crowdsourcing activities can differ dramatically, so can the means to motivate crowdsourcees in a crowdsourcing initiative. 7 In incentive design, an important part of human-computer interaction research, one of the most popular developments in recent years has commonly been called gamification (Hamari et al., 2014; Hamari et al., 2015; Seaborn and Fels 2015). Gamification refers to design that seeks to, first, increase the motivation of users or participants to engage in an activity or behavior and, second, to increase or otherwise change a given behavior. The concept of gamification stems from the notion that games are a pinnacle form of hedonic self-purposeful systems (Hamari and Koivisto, 2015a). Most gamification applications borrow design patterns from (video) games, and, consequently, aim to give rise to similar experiences as games commonly do, for instance, feelings of mastery, autonomy, flow, or suspense (see e.g. Huotari and Hamari, 2016; Seaborn and Fels, 2015). If we consider gamification in the context of crowdsourcing, it can be seen as an attempt to redirect crowdsourcees’ motivations from purely rational gain-seeking to self-purposeful, intrinsically motivated activity: “Transforming Homo Economicus into Homo Ludens” (Hamari, 2013). Through this redirection of motivations, the goal is to influence crowdsourcees’ behaviors (e.g. participation, concentration, work duration, engagement, or work quality) in the execution of the crowdsourced work. In other words, elements known from games act as motivational affordances (Huotari and Hamari, 2016; Jung et al., 2010; Zhang, 2008) for intrinsic motivations. Points, badges, leaderboards, avatars, and stories are frequently used motivational affordances in gamification (Hamari et al., 2014). The extant literature has conceptualized gamification into a few key aspects: 1) the design (gamification affordances), 2) the psychological outcomes of gamification, and 3) the behavioral outcomes of gamification (Huotari and Hamari, 2016) (Figure 2). As in classical, non-gamified crowdsourcing systems, gamification can be combined with additional incentives, typically monetary rewards, for instance, piece rate payments or a tournament prize that might have additional effects on crowdsourcees’ motivations (Straub et al., 2015; Zhao and Zhu, 2014a). Existing empirical works also suggest that contextual factors, such as the domain (Hamari, 2013), and aspects relating to the user, have an effect (Koivisto and Hamari, 2014). Gamification has thus far been researched in a variety of areas, such as health (Jones et al., 2014), exercise (Hamari and Koivisto, 2014, 2015a, 2015b; Chen and Pu, 2014; Koivisto and Hamari, 2014), education (Bonde et al., 2014; Christy and Fox, 2014; Domínguez et al., 2013; De-Marcos et al., 2014; 8 Denny, 2013; Morschheuser et al., 2014), commerce (Hamari, 2013, 2015), intra-organizational communication and activities (Morschheuser et al., 2017, 2015), government services (Bista et al., 2014), public engagement (Tolmie et al., 2013), environmental behavior (J. J. Lee et al., 2013; Lounis et al., 2014), and marketing and advertising (Terlutter and Capella, 2013; Cechanowicz et al., 2013). A review on empirical studies on gamification (Hamari et al., 2014) indicated that most gamification studies reported positive effects from the gamification implementations. However, there is still a sizeable gap in our knowledge on the effectiveness of gamification in crowdsourcing, how the results pertaining to gamification differ across domains, which gamification strategies have been used in which environments and towards which kinds of goals. Even though crowdsourcing systems are one of the most researched application areas of gamification (Hamari et al., 2014), the literature is currently fragmented, and no comprehensive conceptualization of gamified crowdsourcing systems exist. Figure 2. Abstract conceptualization of gamification according to Hamari et al. (2014); Huotari and Hamari (2016) 2.3 An integrated conceptual framework for gamified crowdsourcing systems To map the existing literature on gamified crowdsourcing, conceptualizations are needed to guide the mapping so that all the key aspects can be accounted for. Thus, by building on existing work on crowdsourcing (Geiger and Schader, 2014; Pedersen et al., 2013; Zuchowski et al., 2016) and gamification (Hamari et al., 2014) above, we suggest an integrated conceptual framework (as depicted in Figure 3). The framework represents all core aspects of gamified crowdsourcing systems outlined above and provides structure to investigate the phenomenon holistically, along its key components. Our literature review is guided by this framework and investigates both the empirical results on the effectiveness of gamification in crowdsourcing, as well as the variety of concrete manifestations of gamified crowdsourcing systems in the current literature, with a focus on incentive orchestrations of gamification affordances and additional (i.e. monetary) rewards that could lead to several motivational and behavioral outcomes. 9 Figure 3. Conceptual Framework of Gamified Crowdsourcing Systems 3 RESEARCH METHODOLOGY Following the guidelines of Webster and Watson (2002), Boell and Cecez-Kecmanovic (2015), and Ellis (2010), we began the literature review with a literature search. We used the Scopus database as our source of data, since it indexes all other potentially relevant databases, for instance, ACM, IEEE, Springer, and the DBLP Computer Science Bibliography. Since all these individual databases differ in their search functions and algorithms, focusing the search on only one database has ensured that the procedure is replicable, rigorous, and transparent (Boell and Cecez-Kecmanovic, 2015). The literature search in the Scopus database was conducted in October 2016 using the search query TITLE-ABS-KEY(GAMIF* AND CROWD*). The results included any permutation of the terms gamification and crowdsourcing in the entry metadata (title, abstract, or keywords). We intentionally limited the search to the metadata, since searching for the terms in all the text would result in a relatively large amount of false positives, since many papers refer to gamification and/or crowdsourcing in passing. We did not restrict the search to specific outlets or disciplines, for two reasons. First, crowdsourcing is a socio-technical approach and is therefore applied in various contexts. Second, due to the novelty of the 16 location tagging, reporting of location-based information, on-location experience, taking location-based photos 2016; Preist et al., 2014; Sheng, 2013; Simões and De Amicis, 2016; Talasila et al., 2016; Uzun et al., 2013 Answering questions/sharing knowledge answering user-generated questions, providing feedback, knowledge-sharing in communities Ipeirotis and Gabrilovich, 2014; Inaba et al., 2015; Y. Liu, Alexandrova, Nakajima et al., 2011*; Machnik et al., 2015; Pothineni et al., 2014; Vasilescu et al., 2014 6 Creative creation work idea creation, algorithm development, requirements elicitation Bentzien et al., 2013; Choi et al., 2014; Dos Santos et al., 2015; Lauto and Valentin, 2016; Snijders et al., 2015; Yakushin and Lee, 2014 6 Text annotation work text annotation, medical text annotation, biological data annotation Cao et al., 2015; Chamberlain, 2014; Dumitrache et al., 2013; Nose and Hishiyama, 2013; Ustalov, 2015 5 Assessment work relationship building, relevance assessment, classification work, decision-making Eickhoff et al., 2012; Harris, 2014; Melenhorst et al., 2015; Prestopnik and Tang, 2015; Yu et al., 2015 5 Searching for and/or optimization of tasks document searching, searching for digital profiles, finding optimal solutions He et al., 2014; T. Y. Lee et al., 2013; Nunzio et al., 2016; Sørensen et al., 2016; Tinati et al., 2016 5 Transcription work video captioning Kacorri et al., 2014, 2015; Saito et al., 2014 3 Translation work translating sentences Packham and Suleman, 2015 1 N/A no clear work description provided, usergenerated tasks, social activities Cucari et al., 2016; J. J. Lee et al., 2013; Nagai et al., 2014; Sakamoto and Nakajima, 2014 4 References in bold refer to studies in which empirical results about gamification have been reported. * Mentioned twice, because the core task of that crowdsourcing system is the answering of location-based questions. By analyzing the value creation (emergent or non-emergent solution) and the contribution type (homogeneous or heterogeneous contribution) according to our framework (Figure 3) and Geiger and Schader (2014), we found that most cases in the reviewed literature can be classified as gamified crowdprocessing systems (homogenous tasks, non-emergent outcome). Cases with gamified crowdsolving and crowdrating were also present. However, very few cases described gamified crowdcreating systems (see Table 5). We identified 12 categories of gamification affordances (design elements, known from video games) in the reviewed body of literature (see Table 5). Points (in 53 cases) were clearly the most reported gamification components and usually provided the basis for other affordances. Commonly, points were combined with leaderboards (in 45 cases) to create competition between participants. Points 17 were also combined with further elements in diverse ways across implementations; they were used in combination with, for instance, time limits (e.g. Harris, 2014; Kacorri et al., 2014), they were used as a basis for calculating the level of crowdsourcees in a level system (e.g. T. Y. Lee et al., 2013; Saito et al., 2014), with the ability to compare them between team members and peers (e.g. T. Y. Lee et al., 2013; Saito et al., 2014), as well as with badges and missions to visualize specific goals (e.g. Bowser et al., 2013; J. J. Lee et al., 2013; Massung et al., 2013; Preist et al., 2014; Vasilescu et al., 2014). Looking at the relative shares of affordances reported in all the reviewed papers, we found the largest variety of affordances in studies that investigated solving-related crowdsourcing work, while papers on crowdprocessing and crowdrating reported simpler forms of gamification such as simple combinations of points and leaderboards. Crowdsourcing types of crowdcreating and crowdsolving differ from crowdrating and crowdprocessing in that the participation at crowdsourcing work depends on a variety of heterogeneous contributions. Our review showed that studies in the areas of crowdcreating and crowdsolving reported the use of more manifold sets of gamification affordances. These approaches employed not only points and leaderboards, but also, for instance, storytelling, missions, and avatars. Especially crowdsourcing approaches that sought heterogeneous location-based information or sought to solve complex problems based on creative and diverse contributions often applied rich gamification designs. For instance, Tinati et al. (2016) applied points, badges, progress statistics, virtual teams, and leaderboards to engage users to find patterns in 3-D maps of neuro-scans, while Prandi et al. (2016) created an augmented reality with zombies and virtual weapons as a playground for creating a usergenerated map of heterogeneous accessibility barriers. Since most studies provided comprehensive information on the applied game mechanics and rules, we also analyzed and classified the gamification approaches along their applied goal structures (Morschheuser et al., 2017) into competitive, cooperative, and individualistic gamification designs (Table 6). Crowdsourcing types of creating and rating differ from solving and processing in that the end goal of the crowdsourced work is the emergent value from all the contributions. Therefore, it could be assumed that designers of gamified crowdsourcing systems with emergent outcomes would rather use cooperative gamification designs compared to designs of non-emergent approaches. However, when analyzing the goal structures used in these types, no notable differences could be found. Competition- 18 based designs with points and leaderboards that encourage individual work rather than cooperative work were used very often in all four crowdsourcing types. However, the scoring approaches differed based on how points were awarded and from which actions they could be earned. In crowdprocessing approaches, where the sheer number of contributions is often more important than quality (Geiger and Schader, 2014), users were commonly rewarded for general participation (e.g. number of completed tasks (Itoko et al., 2014), number of correct answers (Ipeirotis and Gabrilovich, 2014), or the number of visited locations (Uzun et al., 2013)). While in crowdrating approaches, where the output is more emergent, users were also rewarded for the quality of their contributions (e.g. the quality of contributions rated by others (Dumitrache et al., 2013), or similarity/agreement with other crowdsourcees’ contributions (Eickhoff et al., 2012; Goncalves et al., 2014; Harris, 2014; Saito et al., 2014)). Such scoring mechanisms, which depend on the extent of agreement with other crowdsourcees’ contributions, seem to be suitable for motivating users to emulate others and to “think and act like the community”. In crowdsolving approaches, both forms occurred equally (e.g. the number of completed tasks (Y. Liu, Alexandrova, Nakajima et al., 2011; Yakushin and Lee, 2014), and the quality of contributions rated by others (J. J. Lee et al., 2013; Vasilescu et al., 2014)). Unfortunately, the small amount of studies investigating gamification in the crowdcreating approaches limits the identification of a clear pattern in their gamification implementations. Table 5. Gamification Affordances per Crowdsourcing Type Crowdsourcing type/affordances Processing (N = 27) Rating (N = 12) Solving (N = 17) Creating (N = 7) Frequency (total 63) Points/Scores Brenner et al., 2014; Carlier et al., 2016; Cao et al., 2015; Cucari et al., 2016; Deng et al., 2016; Dergousoff and Mandryk, 2015; Feyisetan et al., 2015; Inaba et al., 2015; Ipeirotis and Gabrilovich, 2014; Kawajiri et al., 2014; Kobayashi et al., 2015; Kurita et al., 2016; T. Y. Lee et al., 2013; Melenhorst et al., 2015; Nose and Hishiyama, 2013; Packham and Suleman, 2015; Prestopnik and Tang, 2015; Riegler et al., 2015; Roengsamut et al., 2015; Rosani et al. 2015; Runge Altmeyer et al., 2016; Dumitrache et al., 2013; Eickhoff et al., 2012; Goncalves et al., 2014; Harris, 2014; Kacorri et al., 2014; Kacorri et al., 2015; Lessel et al., 2015; Mason et al., 2012; Massung et al., 2013; Preist et al., 2014; Saito et al., 2014 Choi et al., 2014; Dos Santos et al., 2015; De Franga et al., 2015; He et al., 2014; Lauto and Valentin, 2016; J. J. Lee et al., 2013; Y. Liu, Alexandrova, Nakajima et al., 2011; Nunzio et al., 2016; Simões and De Amicis, 2016; Sørensen et al., 2016; Tinati et al., 2016; Vasilescu et al., 2014; Yakushin and Lee, 2014 Brito et al., 2015; Martella et al., 2015; Pothineni et al., 2014; Prandi et al., 2016; Sheng, 2013; Snijders et al., 2015 54 19 et al., 2015; Talasila et al., 2016; Uzun et al., 2013 Leaderboards/ Rankings Brenner et al., 2014; Cao et al., 2015; Cucari et al., 2016; Dergousoff and Mandryk, 2015; Feyisetan et al., 2015*; Inaba et al., 2015; Ipeirotis and Gabrilovich, 2014*; Itoko et al., 2014; Kawajiri et al., 2014; Kobayashi et al., 2015; T. Y. Lee et al., 2013*; Machnik et al., 2015; Melenhorst et al., 2015; Packham and Suleman, 2015; Riegler et al., 2015; Roengsamut et al., 2015; Rosani et al. 2015; Talasila et al., 2016; Uzun et al., 2013 Altmeyer et al., 2016; Chamberlain, 2014; Dumitrache et al., 2013; Eickhoff et al., 2012; Goncalves et al., 2014; Harris, 2014; Kacorri et al., 2015; Lessel et al., 2015; Massung et al., 2013; Preist et al., 2014; Saito et al., 2014 Bentzien et al., 2013; De Franga et al., 2015; Dos Santos et al., 2015; He et al., 2014; Lauto and Valentin, 2016; J. J. Lee et al., 2013; Y. Liu, Alexandrova, Nakajima et al., 2011; Nunzio et al., 2016; Tinati et al., 2016; Ustalov, 2015; Vasilescu et al., 2014; Yakushin and Lee, 2014 Bowser et al., 2013; Martella et al., 2015; Snijders et al., 2015 45 Badges/ Achievements Cao et al., 2015; Feyisetan et al., 2015*; Itoko et al., 2014; Kobayashi et al., 2015; T. Y. Lee et al., 2013*; Melenhorst et al., 2015; Talasila et al., 2016; Uzun et al., 2013 Altmeyer et al., 2016; Mason et al., 2012; Massung et al., 2013; Preist et al., 2014 De Franga et al., 2015; Y. Liu, Alexandrova, Nakajima et al., 2011; Tinati et al., 2016; Vasilescu et al., 2014 Bowser et al., 2013; Martella et al., 2015; Sheng, 2013 19 Levels Brenner et al., 2014; Feyisetan et al., 2015*; T. Y. Lee et al., 2013*; Riegler et al., 2015; Roengsamut et al., 2015; Talasila et al., 2016; Yu et al., 2015 Dumitrache et al., 2013; Saito et al., 2014 De Franga et al., 2015; Nagai et al., 2014; Nunzio et al., 2016; Yakushin and Lee, 2014 Martella et al., 2015; Sheng, 2013 15 Progress Cao et al., 2015; Feyisetan et al., 2015*; Itoko et al., 2014; T. Y. Lee et al., 2013* J. J. Lee et al., 2013; Nagai et al., 2014; Tinati et al., 2016; Vasilescu et al., 2014 Brito et al., 2015 9 Feedback Brenner et al., 2014; Deng et al., 2016; Feyisetan et al., 2015*; Ipeirotis and Gabrilovich, 2014*; Melenhorst et al., 2015 Kacorri et al., 2015; J. J. Lee et al., 2013; Y. Liu, Alexandrova, Nakajima et al., 2011 8 Virtual objects/ resources (e.g. weapons, materials) Dergousoff and Mandryk, 2015; Prestopnik and Tang, 2015* ; Talasila et al., 2016 Lauto and Valentin, 2016; Nunzio et al., 2016; Simões and De Amicis, 2016 Prandi et al., 2016*; Snijders et al., 2015 8 Storytelling Nose and Hishiyama, 2013; Prestopnik and Tang, 2015* Sakamoto and Nakajima, 2014; Simões and De Amicis, 2016 Brito et al., 2015; Prandi et al., 2016*; Sheng, 2013 7 Virtual territories Talasila et al., 2016 Y. Liu, Alexandrova, Nakajima et al., 2011; Simões Brito et al., 2015; Martella et al., 2015; Prandi 7 20 and De Amicis, 2016 et al., 2016*; Sheng, 2013 Teams Saito et al., 2014; Kacorri et al., 2014; Kacorri et al., 2015 Bentzien et al., 2013; Tinati et al., 2016; Ustalov, 2015 6 Missions Cucari et al., 2016 J. J. Lee et al., 2013; Sakamoto and Nakajima, 2014 3 Avatars/Virtual characters Dergousoff and Mandryk, 2015; Talasila et al., 2016 De Franga et al., 2015; Nagai et al., 2014 4 References in bold refer to studies in which empirical results about gamification have been reported. * In this paper the affordance is used as experimental condition in a comparison of different gamification affordances. Table 6. Gamification Design Approaches per Crowdsourcing Type Crowdsourcing type/design approach Processing Rating Solving Creating Frequency Competitive 16 (+2)* 9 10 3 38 (+2) Cooperative / Intergroup competition 2 2 5 3 12 Individualistic 4 (+2)* 1 - 1 6 (+2) Not clear (due to missing details) 3 - 2 - 5 * Two papers compared an individual with a competitive approach and found that competitions seem to be more effective. In most of the studies, the incentives were solely based on gamification (Table 7). Some studies additionally employed financial rewards, for instance, a small monetary task-based compensation or a prize for the leaders on a high-score list, to motivate participants. Table 7. Incentive Orchestration Incentive Literature # Gamification Altmeyer et al., 2016; Bentzien et al., 2013; Bowser et al., 2013; Cao et al., 2015; Chamberlain, 2014; Cucari et al., 2016; De Franga et al., 2015; Dergousoff and Mandryk, 2015; Dumitrache et al., 2013; Goncalves et al., 2014; He et al., 2014; Itoko et al., 2014; Kacorri et al., 2014, 2015; Kobayashi et al., 2015; Kurita et al., 2016; Lauto and Valentin, 2016; J. J. Lee et al., 2013; T. Y. Lee et al., 2013; Lessel et al., 2015; Y. Liu, Alexandrova, Nakajima et al., 2011; Martella et al., 2015; Mason et al., 2012; Nagai et al., 2014; Nose and Hishiyama, 2013; Nunzio et al., 2016; Pothineni et al., 2014; Prestopnik and Tang, 2015; Roengsamut et al., 2015; Rosani et al., 2015; Runge et al., 2015; Saito et al., 2014; Sakamoto and Nakajima, 2014; Sheng, 2013; Simões and De Amicis, 2016; Snijders et al., 2015; Sørensen et al., 2016; Tinati et al., 2016; Ustalov, 2015; Uzun et al., 2013; Vasilescu et al., 2014; Yakushin and Lee, 2014; Yu et al., 2015 43 Gamification + monetary rewards Brenner et al., 2014; Brito et al., 2015; Choi et al., 2014; Deng et al., 2016; Dos Santos et al., 2015; Harris, 2014; Inaba et al., 2015; Kawajiri et al., 2014; Melenhorst et al., 2015; Riegler et al., 2015 10 Gamification + other rewards Machnik et al., 2015 (reward: access to specific information) 1 21 Both as an experimental condition Carlier et al., 2016; Eickhoff et al., 2012; Feyisetan et al., 2015; Ipeirotis and Gabrilovich, 2014; Massung et al., 2013; Packham and Suleman, 2015; Prandi et al., 2016; Preist et al., 2014; Talasila et al., 2016 9 References in bold refer to studies in which empirical results about gamification have been reported. As seen in Table 8, most studies combining crowdsourcing and gamification were not targeted to any specific types of crowds but rather described implementations that are agnostic as to who the crowdsourcees should be. However, interestingly a few implementations were designed with a specific crowdsourcee segment in mind. For instance, Yakushin and Lee (2014) crowdsourced the development of algorithms for humanoid robots to a network of specialists in a competitive way, while for instance, T. Y. Lee et al. (2013) motivated employees to search for and identify Twitter accounts. These examples demonstrate that gamification is usable in a variety of usage cases with different target groups. However, to date, we have seen little research into whether there are differences between user groups or which affordances should be used to support different motivations of crowdworkers. However, first empirical studies suggest that the effectiveness of gamification may differ according to crowdsourcees’ personal characteristics, such as the contributors’ ages (Itoko et al., 2014; Kobayashi et al., 2015). Based on Eickhoff et al. (2012) and Itoko et al. (2014), gamification has great potential for young and senior crowdsourcees, although competition-based gamification might be more effective with young participants. Table 8. Crowdsourcees Participants # Unspecified crowd (all other empirical papers) 44 Students Bowser et al., 2013; Kawajiri et al., 2014; J. J. Lee et al., 2013; Nunzio et al., 2016; Talasila et al., 2016 5 Experts Cao et al., 2015; Dumitrache et al., 2013; Mason et al., 2012; Melenhorst et al., 2015; Ustalov, 2015 5 Researchers Yakushin and Lee, 2014 1 Employees Lauto and Valentin, 2016; T. Y. Lee et al., 2013; Machnik et al., 2015; Pothineni et al., 2014; Snijders et al., 2015 5 The elderly Nagai et al., 2014 1 Citizens Dos Santos et al., 2015; Goncalves et al., 2014 2 References in bold refer to studies in which empirical results about gamification have been reported. 22 4.5 Psychological and behavioral outcomes Finally, we examined the psychological and behavioral outcomes described in the empirical papers and associated with the use of gamification affordances. The psychological outcomes were not commonly measured using comprehensive measurement instruments; they were mostly examined via simple questionnaires or qualitative observations, or the observations of how participants behaved was used as a proxy for psychological aspects. Currently, only four studies used validated psychometric measurement instruments (Kobayashi et al., 2015; Melenhorst et al., 2015; Prestopnik and Tang, 2015; Runge et al., 2015). Table 9 provides an overview of the literature in which results about psychological outcomes were reported. In most studies, the behavioral outcomes of gamification are related to the participation of crowdsourcees in a specific task (Figure 3). Several studies that directly compared a gamified and nongamified approach (Table 10) report positive outcomes, such as increases in (long-term) participation (e.g. Eickhoff et al., 2012; Kawajiri et al., 2014; T. Y. Lee et al., 2013), output quality (Eickhoff et al., 2012; Goncalves et al., 2014; T. Y. Lee et al., 2013), and reduction in cheating compared to traditional paid crowdsourcing (Eickhoff et al., 2012). However, gamification does not necessarily lead to an increase in participation. Massung et al. (2013) measured very small differences compared to a control group without gamification, while Packham and Suleman (2015) found that simple gamification approaches (points and leaderboards) cannot replace financial incentives in crowdprocessing. Overall, three studies reported more negative effects than positive (Table 10). In addition to the above studies that employed direct comparisons, 10 studies reported positive results based on users’ perceptions of the gamified crowdsourcing system (Bowser et al., 2013; Dumitrache et al., 2013; J. J. Lee et al., 2013; Saito et al., 2014) or based on the measured user engagement (Pothineni et al., 2014). These – mostly descriptively reported – results showed no effects of gamification per se, but can be seen as positive indicators for the acceptance of gamification in the context of crowdsourcing (Table 10). 23 Some studies even compared different gamification designs and provided first empirical results for designing gamified crowdsourcing approaches in order to achieve positive psychological and behavioral outcomes (Table 10). For instance, Choi et al. (2014) showed in an experiment that explicitly expressed gamification rewards before the task phase can increase the quality of crowdsourcing work and crowdsourcees’ engagement levels. The empirical findings of T. Y. Lee et al. (2013) indicate that social achievements seem to be a bit more effective than individual ones (see also Feyisetan et al., 2015; Runge et al., 2015). The authors examine this by comparing the effects of public participation rankings that encourage workers to compare their efforts with others and level systems that motivate via the visualization of individual achievements. Ipeirotis and Gabrilovich (2014) showed that the concrete design of a leaderboard or ranking can have significant effects on the participation. Based on their findings, the authors recommend to use ‘all-time’ leaderboards prudently, since they may demotivate low-ranked participants and newcomers. Massung et al. (2013) and Preist et al. (2014) showed demotivating effects of leaderboards and possible negative effects on the overall outcome; they propose a set of design principles for designers of gamified crowdsourcing systems and suggest mixing several motivational affordances for different target groups to increase the overall outcome. However, T. Y. Lee et al. (2013) and Dumitrache et al. (2013) indicate that adding more motivational affordances does not always increase motivation and that to date we have too little knowledge to be able to explain effectiveness of affordances for a specific user group (Itoko et al., 2014). Prestopnik and Tang (2015) highlighted the effects of storytelling in gamified crowdsourcing. By comparing two gamified crowdprocessing approaches, the researchers identified that storytelling can transform perceptions of a crowdsourcing task from work-related to play-related. Taken together, these three categories of empirical studies on the effectiveness of gamification in crowdsourcing, more than 90% of the analyzed studies reported positive or predominantly positive outcomes of gamification in crowdsourcing (Table 10). Most cases reported positive effects on quantitative contributions (Table 11). However, qualitative and long-term effects could also be achieved, which strongly depends on the context and concrete implementation of gamification affordances. Table 9. Psychological Outcomes Reported in the Literature 24 Psychological outcome Literature # Motivation Altmeyer et al., 2016; Bowser et al., 2013; Eickhoff et al., 2012; Itoko et al., 2014; Kawajiri et al., 2014; Kobayashi et al., 2015; Y. Liu, Alexandrova, Nakajima et al., 2011; Machnik et al., 2015; Massung et al., 2013; Nose and Hishiyama, 2013; Preist et al., 2014; Prestopnik and Tang, 2015; Roengsamut et al., 2015; Runge et al., 2015; Tinati et al., 2016 15 Attitudes Bowser et al., 2013; Dergousoff and Mandryk, 2015; Itoko et al., 2014; Kobayashi et al., 2015; Martella et al., 2015; Preist et al., 2014; Prestopnik and Tang, 2015; Roengsamut et al., 2015; Runge et al., 2015; Tinati et al., 2016 10 Fun/Enjoyment Altmeyer et al., 2016; Bowser et al., 2013; Choi et al., 2014; Dumitrache et al., 2013; Kobayashi et al., 2015; J. J. Lee et al., 2013; Melenhorst et al., 2015; Prandi et al., 2016; Prestopnik and Tang, 2015; Roengsamut et al., 2015; Runge et al., 2015; Sheng, 2013; Tinati et al., 2016 13 Engagement Altmeyer et al., 2016; Bowser et al., 2013; Y. Liu, Alexandrova, Nakajima et al., 2011; Snijders et al., 2015 4 Other (e.g. appeal, interest, immersion) Cucari et al., 2016; Kobayashi et al., 2015; Melenhorst et al., 2015; Prestopnik and Tang, 2015; 4 References in bold refer to studies in which empirical results about gamification have been reported. Table 10. Results on Gamified Crowdsourcing Results Compared a gamified approach with a non-gamified one No comparison (interviews, user feedback, perceptions, time series analysis, influence of context factors) Comparisons between different gamification designs # Quantitative - inferential Eickhoff et al., 2012; Nose and Hishiyama, 2013; Dergousoff and Mandryk, 2015 Melenhorst et al., 2015 Choi et al., 2014; Ipeirotis and Gabrilovich, 2014; T. Y. Lee et al., 2013; Runge et al., 2015 8 Quantitative - descriptive Carlier et al., 2016*; De Franga et al., 2015; Dumitrache et al., 2013*; Kobayashi et al., 2015; Y. Liu, Alexandrova, Nakajima et al., 2011; Simões and De Amicis, 2016; Sørensen et al., 2016; Talasila et al., 2016 Pothineni et al., 2014; Roengsamut et al., 2015 Feyisetan et al., 2015; Packham and Suleman, 2015* 12 Qualitative Kacorri et al., 2015; Martella et al., 2015 Machnik et al., 2015; Saito et al., 2014; Tinati et al., 2016 Preist et al., 2014; Prestopnik and Tang, 2015 7 Mixed - inferential Altmeyer et al., 2016; Vasilescu et al., 2014 Bowser et al., 2013; Itoko et al., 2014 Kawajiri et al., 2014; Massung et al., 2013; Prandi et al., 2016 7 Mixed - descriptive Goncalves et al., 2014 J. J. Lee et al., 2013; Snijders et al., 2015 3 Total More positive (14) / negative (2) More positive (10) More positive (10) / negative (1) 37 * Studies that reported negative effects of gamification, for instance compared to paid crowdsourcing or non-gamified approaches Table 11. Positive Effects of Gamification in Crowdsourcing Reported in the Literature Outcomes Literature # 25 Positive effects on the quantitative contribution / willingness to contribute Altmeyer et al., 2016; Bowser et al., 2013; De Franga et al., 2015; Dergousoff and Mandryk, 2015; Eickhoff et al., 2012; Feyisetan et al., 2015; Ipeirotis and Gabrilovich, 2014; Itoko et al., 2014; Kawajiri et al., 2014; Kobayashi et al., 2015; J. J. Lee et al., 2013; T. Y. Lee et al., 2013; Y. Liu, Alexandrova, Nakajima et al., 2011; Martella et al., 2015; Massung et al., 2013; Nose and Hishiyama, 2013; Pothineni et al., 2014; Prandi et al., 2016; Preist et al., 2014; Prestopnik and Tang, 2015; Roengsamut et al., 2015; Simões and De Amicis, 2016; Snijders et al., 2015; Talasila et al., 2016; Tinati et al., 2016; Vasilescu et al., 2014 26 Positive effects on the qualitative contribution Dergousoff and Mandryk, 2015; Eickhoff et al., 2012; Feyisetan et al., 2015; Goncalves et al., 2014; Ipeirotis and Gabrilovich, 2014; Kawajiri et al., 2014; Kobayashi et al., 2015; T. Y. Lee et al., 2013; Massung et al., 2013; Prestopnik and Tang, 2015; Runge et al., 2015; Simões and De Amicis, 2016; Sørensen et al., 2016 13 Positive effects on continued work / long-term engagement Itoko et al., 2014; Kawajiri et al., 2014; Kobayashi et al., 2015; T. Y. Lee et al., 2013; Massung et al., 2013; Prestopnik and Tang, 2015 6 5 DISCUSSION In this study, we have provided a comprehensive review and overview of the use of gamification in crowdsourcing in the current body of literature. Following an integrated conceptual framework (Figure 3), we analyzed characteristic features of gamified crowdsourcing systems. Especially, we reviewed the use of different forms of gamification in different types of crowdsourcing (crowdprocessing, crowdsolving, crowdrating, and crowdcreating), as well as the interplay between gamification and additional monetary rewards, the types of work that have been crowdsourced, the types of crowdsourcees, and the domains in which gamification in crowdsourcing has been applied. Furthermore, we investigated the results of empirical studies on the psychological and behavioral outcomes of gamification in crowdsourcing systems. This meticulous mapping enabled us to discuss recommendations for designing gamified crowdsourcing systems as well as limitations, emerging issues, and future research directions. 5.1 Recommendations for designing gamified crowdsourcing systems We form recommendations by triangulating from the results in the body of the reviewed literature and the results of this review. One of the overall primary findings of our review is that gamification positively affects crowdsourcing work, either in the form of increased crowdsourcee motivations or contributions. Thus, it is less important to investigate whether gamification works as a whole; instead, we need to delve deeper to explore which specific design choices are successful in the various crowdsourcing types. 32 5.2.2 Theoretical agenda Most of the reviewed studies with empirical results on gamification in crowdsourcing focused on the effectiveness of gamification. Most of these studies lacked theory to ground the research, were rudimentary, or were disconnected from the applied work. By paying attention to these theoretical limitations, future research could provide valuable contributions to better understand and explain gamification in crowdsourcing. We recommend borrowing theoretical perspectives (Whetten, 1989) from psychology, philosophy, or marketing to serve as a basis for study design and to explain psychological effects and behavioral outcomes. Especially, we recommend drawing on Csíkszentmihályi’s (1990) theory of flow and self-determination theory (Ryan and Deci, 2000), when investigating the motivational effects of gamification affordances. These two theoretical perspectives are frequently used to investigate motivational effects in crowdsourcing (Zhao and Zhu, 2014b; Zheng et al., 2011) and gamification (Hamari and Koivisto, 2014; Hamari et al., 2016), since they provide insights into inducing and achieving intrinsic motivation. Considering gamification elements as motivational affordances (Huotari and Hamari 2016) that are designed to stimulate motivational needs, goals achievement, and help people to achieve their personal goals, goal-setting theory and the affordance concept provide essential foundations (cf. Huotari and Hamari, 2016; Jung et al., 2010; Morschheuser et al., 2017). Finally, to understand the effects of gamification and gamification rewards on attitudes and behavioral outcomes, we recommend that researchers draw on the theory of planned behavior (Ajzen, 1991) and self-efficacy theory (Bandura, 1977), which are often applied in general gamification research (Hamari and Koivisto, 2015a, 2015b). Agenda point 5: Future research should increasingly employ theory from (motivational) psychology to justify research activities, operationalize research, and interpret results. 5.2.3 Thematic agenda Previous research on the motivation of crowdsourcees has primarily analyzed motivations in nongamified crowdsourcing platforms with financial incentives. Commonly, the findings have indicated that users are driven by a mixture of intrinsic motivation and monetary rewards (Brabham, 2010, 2008b; 33 Kaufmann et al., 2011; Leimeister et al., 2009; Zhao and Zhu, 2014a; Zheng et al., 2011). Our overview demonstrated that 68% of the analyzed gamified crowdsourcing cases used only gamification to incentivize crowdworkers (Table 7). This indicated that gamification could not only be used in addition to financial rewards to increase positive experiences (e.g. engagement or enjoyment); rather, it provides a cost-effective opportunity to entirely replace financial incentives. Some studies demonstrated the complex interplays between financial and gamified incentive structures (Massung et al., 2013; Preist et al., 2014). To date, it is unclear for which crowdsourcing system type, crowdsourcee type, and task type the use of gamification is more beneficial compared to financial incentives, or when the combination of the two is the best approach. Future research should compare different incentive mechanisms (see Straub et al., 2015; Harris et al., 2015) and should consider contextual factors and user characteristics. Furthermore, the economic value of gamification also requires further research. Future research could examine the development costs in relation to the effects of gamification, to evaluate the value and to provide insights into gamification-based business models. Agenda point 6: Research into gamified crowdsourcing should explore optimal incentive orchestrations for different crowdsourcing contexts and should provide insights into the overall cost efficiency of gamified crowdsourcing. The findings summarized in Table 6 demonstrate that cooperative approaches, such as gamified crowdcreating systems, are currently receiving less attention from scholars compared to the other system types. This is surprising, since several popular crowdcreating examples, such as Google Ingress, Dell’s Ideastorm, or Threadless (Kavaliova et al., 2016) have implemented various gamification approaches. Further, notably, all reviewed empirical studies that have measured the effects of gamification on participation have analyzed the effects on the intention of an individual to participate, but have neglected that crowdsourcees can form groups with collective intentions (Tsai and Bagozzi, 2014). Studies have shown that collective intentions play a key role in cooperative crowdsourcing (A. X. L. Shen et al., 2009; X.-L. Shen et al., 2014). Finally, we identified that social factors, which have been identified as an essential aspect of gamification (Hamari and Koivisto, 2015a) and could gauge cooperation (such as trust, reciprocity, and sense of community), have been neglected in the literature. 34 Future research that continues ideas from previous studies about virtual teams (Jarvenpaa and Leidner, 1998; Powell et al., 2004), collective intentions in virtual communities (Tsai and Bagozzi, 2014), cooperative games design (Morschheuser et al., 2017), and social factors of gamification (Hamari and Koivisto, 2015b) could provide new insights into the effects of gamification on collective intentions, relationships between crowdworkers, social identities, or collaborative behavior. Future research could utilize established social psychological theories that have evaluated the effects of competition, cooperation, and the combination of the two on enjoyment or performance as a basis for examining the motivational effects of different goal structures in gamification approaches (Tauer and Harackiewicz, 2004; Morschheuser et al., 2017). In this context, the use of cooperative gamification approaches such as virtual teams, cooperative missions, or shared goals that empower the formation of groups and collective intentions could be analyzed to expand the mainly competition-focused gamification conceptions and that help to design effective gamified crowdsourcing communities. Agenda point 7: Future research should seek to investigate the design and effects of cooperative gamification and consider social factors in crowd communities. Crowdsourcing as a problem-solving concept is a multifaceted phenomenon and can be applied in various contexts. Marginal differences can be found in the reviewed studies regarding the domain in which the systems are applied (Table 3), the crowd characteristics (Table 4, Table 8), and the media (e.g. mobile apps (Bowser et al., 2013; Uzun et al., 2013), website (Choi et al., 2014; T. Y. Lee et al., 2013; Y. Liu, Alexandrova, Nakajima et al., 2011), or local installations (Goncalves et al., 2014)). Future research is needed to understand how contextual factors affect gamified crowdsourcing systems. Optimally, studies could apply one gamified crowdsourcing system in a variety of contexts. Since this would be a rather sizeable undertaking, we might have to wait for the accumulating literature to cover more ground. Agenda point 8: Research is needed to understand how contextual factors, such as the domain, the media, and crowd characteristics affect gamified crowdsourcing systems. 35 Our overview indicated that gamification implementations differ in the context of crowdsolving, crowdrating, and crowdprocessing approaches (Table 5). Finally, we identified different recommendations for designers of gamified crowdsourcing systems. Further work is needed to evaluate and extend these recommendations and to study the potentials of different design approaches. Especially manifold designs with for instance avatars, storytelling, or virtual teams provide opportunities for future research. Furthermore, advanced gamification approaches that automatically consider user characteristics and context characteristics should be examined. Building on the results of Itoko et al. (2014) and Koivisto and Hamari (2014), individual adaptive incentive orchestrations might increase effectiveness, acceptance, and long-term motivations. Such adaptive gamification design that goes beyond the current rewards mechanisms used in gamification could utilize recent developments of individualization in crowdsourcing (Geiger and Schader, 2014) and games design (Prakash et al., 2009). Finally, recent technology trends such as virtual realities (Prandi et al., 2016), connected everything, artificial intelligence, and sharing economies are influencing current developments in game design and crowdsourcing. These trends also provide new spaces for gamified crowdsourcing systems that should be studied. Agenda point 9: Future research should expand the design space used in current gamified crowdsourcing systems and should consider novel trends in games design and crowdsourcing. 5.2.4 Future research In this review of applied research and theoretical papers, we were particularly interested in the use of gamification in crowdsourcing systems. However, it is possible that related research has been conducted, also under other conceptual developments such as serious games, games-with-a-purpose, pervasive games, human-based computation, or persuasive technology. Some of these related research areas might be investigating similar phenomena, but were not included in this study. Therefore, future efforts could compare these approaches and their contributions to gamified crowdsourcing. Relatedly, we conducted the literature searches intentionally with a set of keywords to find particularly studies on gamification and crowdsourcing. In our view, our selection of search keywords and data sources was successful for the review’s intended breadth. The choice of a systematic literature study is the reason 36 for some of these limitations (Boell and Cecez-Kecmanovic, 2015). However, in our view, the benefits of a structured summary and a clear aggregation of previous findings outweighed the disadvantages in our case. Future efforts could go beyond these limitations and could extend our findings. 6 CONCLUSIONS Along with the emergence of the interwoven phenomena of gamification and crowdsourcing, gamified crowdsourcing systems have drawn scientific attention and have led to a continuously rising number of research publications. In this review, we sought to provide a comprehensive conceptualization and a structured overview that compared the different characteristics of gamified crowdsourcing systems, examined the results on the effectiveness of gamification in crowdsourcing, and highlighted starting points for future research. We found a wide array of different gamification implementations in different types of crowdsourcing in the literature. However, the literature seems to be unanimous; gamification does seem to work with a majority of configurations and can positively affect the motivations of crowdsourcees, their participation, and output quality. Depending on the type of crowdsourcing (crowdcreating, crowdsolving, crowdprocessing, and crowdrating), we identified patterns in the use of gamification affordances. In the context of crowdsourcing initiatives that provide homogenous and often more monotonous tasks such as crowdprocessing and crowdrating, authors commonly report the use of simple forms of gamification such as points and leaderboards (Table 5). Conversely, crowdsourcing studies with crowdcreating and crowdsolving work that seek diverse and creative contributions employ gamification in more manifold ways with a richer set of mechanics. Generally, gamification is used to promote a kind of competition between the participants rather than a collaborative experience. Monetary rewards could be used as an addition in gamified crowdsourcing systems, but most of the analyzed cases did not apply supplementary financial incentives. However, at this early stage, the literature is still fairly fragmented, and too little research has been conducted to draw clear conclusions on which specific implementations would work better or worse in certain situations. It is clear that contextual factors and factors related to crowdsourcees play a role, but to what extents and how are still unclear. These and further aspects that would help us to understand and design successful gamified crowdsourcing systems provide much room for future research. 37 7 ACKNOWLEDGMENTS This work was supported by the Robert Bosch GmbH, the Finnish Funding Agency for Technology and Innovation (TEKES - project numbers 40111/14, 40107/14 and 40009/16) and participating partners, as well as Satakunnan korkeakoulusäätiö and its collaborators. The authors also wish to thank the editors, reviewers and proofreaders for their time and effort. 8 REFERENCES Ahmed, N., Mueller, K., 2014. Gamification as a paradigm for the evaluation of visual analytics systems, in: Proceedings of the 5th Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization. ACM, Paris, France, pp 78–86. doi:10.1145/2669557.2669574 Ajzen, I., 1991. The theory of planned behavior. Organ. Behav. Hum. Decis. Process. 50, 179–211. doi:10.1016/0749-5978(91)90020-T AlRouqi, H., Al-Khalifa, H.S., 2014. Making Arabic PDF books accessible using gamification, in: Proceedings of the 11th Web for All Conference on - W4A ’14. ACM Press, Seoul, Republic of Korea, pp. 1–4. doi:10.1145/2596695.2596712 Altmeyer, M., Lessel, P., Krüger, A., 2016. Expense control: A gamified, semi-automated, crowdbased approach for receipt capturing, in: Proceedings of the 21st International Conference on Intelligent User Interfaces - IUI ’16. ACM Press, New York, USA, pp. 31–42. doi:10.1145/2856767.2856790 Ansari, S., Kleiman, R., Binder, J., Hayes, W., Hoeng, J., Iskandar, A., Rhrissorrakrai, K., Norel, R., O’Neel, B., Peitsch, M., Poussin, C., Talikka, M., Schlage, W., Stolovitzky, G., DiFabio, A., Pratt, D., Boue, S., 2013. On crowd-verification of biological networks. Bioinform. Biol. Insights 7, 307–325. doi:10.4137/BBI.S12932 Armisen, A., Majchrzak, A., 2015. Tapping the innovative business potential of innovation contests. Bus. Horiz. 58, 389–399. doi:10.1016/j.bushor.2015.03.004 Bainbridge, D., 2015. And we did it our way: A case for crowdsourcing in a digital library for musicology, in: Proceedings of the 2nd International Workshop on Digital Libraries for Musicology - DLfM ’15. ACM Press, Knoxville, TN, USA, pp. 1–8. doi:10.1145/2785527.2785529 Bandura, A., 1977. Self-efficacy: Toward a unifying theory of behavioral change. Psychol. Rev. 84, 191–215. doi:10.1037/0033-295X.84.2.191 Benjamin, M., 2016. Problems and procedures to make wordnet data (retro)fit for a multilingual dictionary, in: Proceedings of the 8th Global WordNet Conference (GWC). Bucharest, pp. 27–33. Bentzien, J., Muegge, I., Hamner, B., Thompson, D.C., 2013. Crowd computing: Using competitive dynamics to develop and refine highly predictive models. Drug Dis. Tod. 18, pp. 472–478. doi:10.1177/1354856507084420 Biegel, B., Beck, F., Lesch, B., Diehl, S., 2014. Code tagging as a social game, in: Proceedings of the 30th International Conference on Software Maintenance and Evolution (ICSME’14). IEEE, Victoria, Canada, pp. 411–415. doi:10.1109/ICSME.2014.64 38 Bista, S.K., Nepal, S., Paris, C., Colineau, N., 2014. Gamification for online communities: A case study for delivering government services. Int. J. Coop. Inf. Syst. 23. Blohm, I., Leimeister, J.M., Bretschneider, U., 2010. Does collaboration among participants lead to better ideas in IT-based idea competitions? An Empirical Investigation, in: Proceedings of the 43rd Hawaii International Conference on System Sciences – HICSS. IEEE, Honolulu, HI, USA, pp. 1–10. Bockes, F., Edel, L., Ferstl, M., Schmid, A., 2015. Collaborative landmark mining with a gamification approach, in: Proceedings of the 14th International Conference on Mobile and Ubiquitous Multimedia - MUM ’15. ACM Press, Linz, Austria, pp. 364–367. doi:10.1145/2836041.2841209 Boell, S.K., Cecez-Kecmanovic, D., 2015. On being “systematic” in literature reviews in IS. J. Inf. Technol. 30, 161–173. Bonde, M.T., Makransky, G., Wandall, J., Larsen, M. V, Morsing, M., Jarmer, H., Sommer, M.O.A., 2014. Improving biotech education through gamified laboratory simulations. Nat. Biotechnol. 32, 694–7. doi:10.1038/nbt.2955 Bowser, A., Hansen, D., He, Y., Boston, C., Reid, M., Gunnell, L., Preece, J., 2013. Using gamification to inspire new citizen science volunteers, in: Proceedings of the 1st International Conference on Gameful Design, Research, and Applications – Gamification’13. ACM, Stratford, Ontario, Canada, pp. 18–25. doi:10.1145/2583008.2583011 Brabham, D.C., 2010. Moving the crowd at threadless. Information, Commun. Soc. 13, 1122–1145. Brabham, D.C., 2008a. Moving the crowd at iStockphoto: The composition of the crowd and motivations for participation in a crowdsourcing application by. First Monday 13. Brabham, D.C., 2008b. Crowdsourcing as a model for problem solving: An introduction and cases. Converg. Int. J. Res. into New Media Technol. 14, 75–90. doi:10.1177/1354856507084420 Brandtner, P., Auinger, A., Helfert, M., 2014. Principles of human computer interaction in Crowdsourcing to foster motivation in the context of Open Innovation, in: Proceedings of HCIB 2014. Springer, Heraklion, Crete, Greece, pp. 585–596. doi:10.1007/978-3-319-07293-7_57 Brenner, M., Mirza, N., Izquierdo, E., 2014. People recognition using gamified ambiguous feedback, in: Proceedings of the First International Workshop on Gamification for Information Retrieval - GamifIR ’14. ACM, Amsterdam, Netherlands, pp. 22–26. doi:10.1145/2594776.2594781 Brito, J., Vieira, V., Duran, A., 2015. Towards a framework for gamification design on crowdsourcing systems: The G.A.M.E. approach, in: Proceedings of the 12th International Conference on Information Technology - New Generations. IEEE, Las Vegas, Nevada, USA, pp. 445–450. doi:10.1109/ITNG.2015.78 Bullinger, A.C., Neyer, A.-K., Rass, M., Moeslein, K.M., 2010. Community-based innovation contests: Where competition meets cooperation. Creat. Innov. Manag. 19, 290–303. doi:10.1111/j.1467-8691.2010.00565.x Burnett, D., Lochrie, M., Coulton, P., 2012. “CheckinDJ” using check-ins to crowdsource music preferences, in: Proceeding of the 16th International Academic MindTrek Conference on - MindTrek ’12. ACM Press, Tampere, Finland, pp. 51–54. doi:10.1145/2393132.2393143 Cao, H.-A., Wijaya, T.K., Aberer, K., Nunes, N., 2015. A collaborative framework for annotating energy datasets, in: Proceedings of the International Conference on Big Data. IEEE, Santa Clara, CA, USA, pp. 2716–2725. doi:10.1109/BigData.2015.7364072 39 Carlier, A., Salvador, A., Cabezas, F., Giro-i-Nieto, X., Charvillat, V., Marques, O., 2016. Assessment of crowdsourcing and gamification loss in user-assisted object segmentation. Multimed. Tools Appl. 23. doi:10.1007/s11042-015-2897-6 Cechanowicz, J., Gutwin, C., Brownell, B., Goodfellow, L., 2013. Effects of gamification on participation and data quality in a real-world market research domain, in: Proceedings of the First International Conference on Gameful Design, Research, and Applications - Gamification ’13. ACM Press, New York, New York, USA, pp. 58–65. doi:10.1145/2583008.2583016 Chamberlain, J., 2014. The annotation-validation (AV) model: rewarding contribution using retrospective agreement, in: Proceedings of the First International Workshop on Gamification for Information Retrieval - GamifIR ’14. ACM, Amsterdam, Netherlands, pp. 12–16. doi:10.1145/2594776.2594779 Chen, Y., Pu, P., 2014. HealthyTogether: exploring social incentives for mobile fitness applications, in: Proceedings of the Second International Symposium of Chinese CHI on - Chinese CHI ’14. pp. 25–34. doi:10.1145/2592235.2592240 Cherinka, R., Miller, R., Prezzama, J., 2013. Emerging trends, technologies and approaches impacting innovation, in: Proceedings of the 6th International Multi-Conference on Engineering and Technological Innovation - IMETI 2013. Orlando, Florida, USA, pp. 92–97. Choi, J., Choi, H., So, W., Lee, J., You, J., 2014. A study about designing reward for gamified crowdsourcing system, in: Proceedings of the 3rd International Conference, DUXU 2014, Held as Part of HCI International 2014. Springer International Publishing, Heraklion, Crete, Greece, pp. 678–687. doi:10.1007/978-3-319-07626-3-64 Christy, K.R., Fox, J., 2014. Leaderboards in a virtual classroom: A test of stereotype threat and social comparison explanations for women’s math performance. Comput. Educ. 78, 66–77. doi:10.1016/j.compedu.2014.05.005 Cooper, S., Khatib, F., Treuille, A., Barbero, J., Lee, J., Beenen, M., Leaver-Fay, A., Baker, D., Popović, Z., 2010. Predicting protein structures with a multiplayer online game. Nature 466, 756– 760. doi:10.1038/nature09304 Csikszentmihalyi, M., 1990. Flow: The psychology of optimal experience. Harper and Row, New York, NY, USA. Cucari, G., Leotta, F., Mecella, M., Vassos, S., 2016. Collecting human habit datasets for smart spaces through gamification and crowdsourcing, in: De Gloria, A., Veltkamp, R. (Eds.), Proceedings of the 4th Games and Learning Alliance Conference (GALA), Lecture Notes in Computer Science. Springer International Publishing, Rome, Italy, pp. 208–217. doi:10.1007/978-3319-40216-1_22 Dai, W., Wang, Y., Jin, Q., Ma, J., 2016. An integrated incentive framework for mobile crowdsourced sensing. Tsinghua Sci. Technol. 21, 146–156. doi:10.1109/TST.2016.7442498 De Franga, F.A., Vivacqua, A.S., Campos, M.L.M., 2015. Designing a gamification mechanism to encourage contributions in a crowdsourcing system, in: Proceedings of the 19th International Conference on Computer Supported Cooperative Work in Design (CSCWD). IEEE, Calabria, Italy, pp. 462–466. doi:10.1109/CSCWD.2015.7231003 De-Marcos, L., Domínguez, A., Saenz-de-Navarrete, J., Pagés, C., 2014. An empirical study comparing gamification and social networking on e-learning. Comput. Educ. 75, 82–91. doi:10.1016/j.compedu.2014.01.012 40 Deci, E.L., 1971. Effects of externally mediated rewards on intrinsic motivation. J. Pers. Soc. Psychol. 18, 105–115. Deci, E.L., Koestner, R., Ryan, R.M., 1999. A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation. Psychol. Bull. 125, 627–668. doi:10.1037/0033-2909.125.6.627 Deng, J., Krause, J., Stark, M., Fei-Fei, L., 2016. Leveraging the wisdom of the crowd for finegrained recognition. IEEE Trans. Pattern Anal. Mach. Intell. 38, 666–676. doi:10.1109/TPAMI.2015.2439285 Denny, P., 2013. The effect of virtual achievements on student engagement, in: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI ’13. ACM Press, New York, New York, USA, p. 763. doi:10.1145/2470654.2470763 Dergousoff, K., Mandryk, R.L., 2015. Mobile gamification for experiment data collection : Leveraging the freemium model, in: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI 2015). ACM, Seoul, Republic of Korea, pp. 1065–1074. Doan, A., Ramakrishnan, R., Halevy, A.Y., 2011. Crowdsourcing systems on the World-Wide Web. Commun. ACM 54, 86–96. doi:10.1145/1924421.1924442 Domínguez, A., Saenz-de-Navarrete, J., De-Marcos, L., Fernández-Sanz, L., Pagés, C., MartínezHerráiz, J.-J., 2013. Gamifying learning experiences: Practical implications and outcomes. Comput. Educ. 63, 380–392. doi:10.1016/j.compedu.2012.12.020 Dos Santos, A.C., Zambalde, A.L., Veroneze, R.B., Botelho, G.A., de Souza Bermejo, P.H., 2015. Open innovation and social participation: A case study in public security in Brazil, in: Proceedings of the 4th International Conference on Electronic Government and the Information Systems Perspective (EGOVIS 2015). Springer International Publishing, Valencia, Spain, pp. 163– 176. doi:10.1007/978-3-319-22389-6_12 Dumitrache, A., Aroyo, L., Welty, C., Sips, R.-J., Levas, A., 2013. Dr. Detective: Combining gamification techniques and crowdsourcing to create a gold standard for the medical domain, in: Proceedings of the 1st International Workshop on Crowdsourcing the Semantic Web (Crowd Sem2013). Sydney, Australia, pp. 16–31. Eickhoff, C., Harris, C.G., de Vries, A.P., Srinivasan, P., 2012. Quality through flow and immersion, in: Proceedings of the 35th International ACM SIGIR Conference - SIGIR ’12, SIGIR ’12. ACM Press, Portland, Oregon, USA, pp. 871–880. doi:10.1145/2348283.2348400 Ellis, P.D., 2010. The essential guide to effect sizes: Statistical power, meta-analysis, and the interpretation of research results. Cambridge University Press, Cambridge, UK. Ermi, L., Mäyrä, F., 2005. Fundamental components of the gameplay experience: Analysing immersion, in: Proceedings of DiGRA 2005 Conference: Changing Views – Worlds in Play. Vancouver, British Columbia, Canada. Estellés-Arolas, E., González-Ladrón-de-Guevara, F., 2012. Towards an integrated crowdsourcing definition. J. Inf. Sci. 38, 189–200. doi:10.1177/0165551512437638 Fava, D., Signoles, J., Lemerre, M., Schäf, M., Tiwari, A., 2015. Gamifying program analysis, in: Proceeding of the 20th International Conference on Logic for Programming, Artificial Intelligence, and Reasoning (LPAR). Springer International Publishing, Suva, Fiji, pp. 591–605. doi:10.1007/978-3-662-48899-7_41 41 Fedorov, R., Fraternali, P., Pasini, C., 2016. SnowWatch: A multi-modal citizen science application, in: Proceedings of the 16th International Conference on Web Engineering (ICWE). Springer International Publishing, Lugano, Switzerland, pp. 538–541. doi:10.1007/978-3-319-387918_43 Feyisetan, O., Simperl, E., Van Kleek, M., Shadbolt, N., 2015. Improving paid microtasks through gamification and adaptive furtherance incentives, in: Proceedings of the 24th International Conference on World Wide Web - WWW ’15. ACM Press, Florence, Italy, pp. 333–343. doi:10.1145/2736277.2741639 Gartner, 2011. Gartner says by 2015, more than 50 percent of organizations that manage innovation processes will gamify those processes. http://www.gartner.com/it/page.jsp?id=1629214 (accessed 7.6.12). Gatautis, R., Vitkauskaite, E., 2014. Crowdsourcing application in marketing activities. Procedia - Soc. Behav. Sci. 110, 1243–1250. doi:10.1016/j.sbspro.2013.12.971 Geiger, D., Schader, M., 2014. Personalized task recommendation in crowdsourcing information systems - Current state of the art. Decis. Support Syst. 65, 3–16. doi:10.1016/j.dss.2014.05.007 Goncalves, J., Hosio, S., Ferreira, D., Kostakos, V., 2014. Game of words: tagging places through crowdsourcing on public displays, in: Proceedings of the 2014 Conference on Designing Interactive Systems - DIS ’14. ACM, Vancouver, BC, Canada, pp. 705–714. doi:10.1145/2598510.2598514 Greenhill, A., Holmes, K., Woodcock, J., Lintott, C., Simmons, B.D., Graham, G., Cox, J., Ohlsson, E., Masters, K., 2016. Playing with science: Exploring how game activity motivates users participation on an online citizen science platform. Aslib J. Inf. Manag. 68, 306–325. doi:10.1108/AJIM-11-2015-0182 Hamari, J., 2015. Why do people buy virtual goods? Attitude toward virtual good purchases versus game enjoyment. Int. J. Inf. Manage. 35, 299–308. Hamari, J., 2013. Transforming homo economicus into homo ludens: A field experiment on gamification in a utilitarian peer-to-peer trading service. Electron. Commer. Res. Appl. 12, 236–245. doi:10.1016/j.elerap.2013.01.004 Hamari, J., Koivisto, J., 2015a. Why do people use gamification services? Int. J. Inf. Manage. 35, 419–431. doi:10.1016/j.ijinfomgt.2015.04.006 Hamari, J., Koivisto, J., 2015b. “Working out for likes”: An empirical study on social influence in exercise gamification. Comput. Human Behav. 50, 333–347. doi:10.1016/j.chb.2015.04.018 Hamari, J., Koivisto, J., 2014. Measuring flow in gamification: Dispositional Flow Scale-2. Comput. Human Behav. 40, 133–143. doi:10.1016/j.chb.2014.07.048 Hamari, J., Koivisto, J., Sarsa, H., 2014. Does gamification work? A literature review of empirical studies on gamification, in: Proceedings of the 47th Hawaii International Conference on System Sciences - HICSS. IEEE, Waikoloa, HI, pp. 3025–3034. doi:10.1109/HICSS.2014.377 Hamari, J., Shernoff, D.J., Rowe, E., Coller, B., Asbell-Clarke, J., Edwards, T., 2016. Challenging games help students learn: An empirical study on engagement, flow and immersion in gamebased learning. Comput. Human Behav. 54, 170–179. doi:10.1016/j.chb.2015.07.045 Hamari, J., Sjöklint, M., Ukkonen, A., 2016. The sharing economy: Why people participate in collaborative consumption. J. Assoc. Inf. Sci. Technol. 67, 2047-2059. doi:10.1002/asi.23552 48 Schlagwein, D., Bjørn-Andersen, N., 2014. Organizational learning with crowdsourcing: The revelatory case of LEGO. J. Assoc. Inf. Syst. 15, 754–778. Seaborn, K., Fels, D.I., 2015. Gamification in theory and action: A survey. Int. J. Hum. Comput. Stud. 74, 14–31. doi:http://dx.doi.org/10.1016/j.ijhcs.2014.09.006 Shen, A.X.L., Lee, M.K.O., Cheung, C.M.K., Chen, H., 2009. An investigation into contribution IIntention and We-Intention in open web-based encyclopedia: Roles of joint commitment and mutual agreement, in: Proceeding of the 13th International Conference on Information Systems. AIS, Phoenix, Arizona, USA, pp. 1–7. Shen, X.-L., Lee, M.K.O., Cheung, C.M.K., 2014. Exploring online social behavior in crowdsourcing communities: A relationship management perspective. Comput. Human Behav. 40, 144–151. doi:10.1016/j.chb.2014.08.006 Sheng, L.Y., 2013. Modelling learning from Ingress (Google’s augmented reality social game), in: Proceedings of the 2013 IEEE 63rd Annual Conference International Council for Education Media, ICEM. IEEE, Singapore, pp. 1–8. doi:10.1109/CICEM.2013.6820152 Sigala, M., 2015. Gamification for crowdsourcing marketing practices: Applications and benefits in tourism, in: Garrigos-Simon, F.J., Gil-Pechuán, I., Estelles-Miguel, S. (Eds.), Advances in Crowdsourcing. Springer International Publishing, pp. 129–145. doi:10.1007/978-3-319-183411_11 Silva, A.C., Lopes, C.T., 2016. Health translations: A crowdsourced, gamified approach to translate large vocabulary databases, in: Proceedings of 11th Iberian Conference on Information Systems and Technologies (CISTI). IEEE, Gran Canaria, Spain, pp. 1–4. doi:10.1109/CISTI.2016.7521479 Simões, B.., Aksenov, P.., Santos, P.., Arentze, T.., De Amicis, R.., 2015. C-space: Fostering new creative paradigms based on recording and sharing “casual” videos through the internet, in: Proceedings of the International Conference on Multimedia and Expo Workshops (ICMEW). IEEE, Torino, Italy, pp. 1–4. Simões, B., De Amicis, R., 2016. Gamification as a key enabling technology for image sensing and content tagging, in: Giuseppe De Pietro, Gallo, L., Howlett, R.J., Jain, L.C. (Eds.), Intelligent Interactive Multimedia Systems and Services 2016. Springer International Publishing, pp. 503– 513. doi:10.1007/978-3-319-39345-2_44 Simperl, E., 2015. How to use crowdsourcing effectively: Guidelines and examples. Lib. Q. 25, 18– 39. Smith, R., Kilty, L.A., 2014. Crowdsourcing and gamification of enterprise meeting software quality, in: Proceedings of the 7th International Conference on Utility and Cloud Computing. IEEE, London, UK, pp. 611–613. doi:10.1109/UCC.2014.95 Snijders, R., Dalpiaz, F., Brinkkemper, S., Hosseini, M., Ali, R., Ozum, A., 2015. REfine: A gamified platform for participatory requirements engineering, in: Proceedings of the 1st International Workshop on Crowd-Based Requirements Engineering (CrowdRE). IEEE, pp. 1–6. doi:10.1109/CrowdRE.2015.7367581 Snijders, R., Dalpiaz, F., Hosseini, M., Shahri, A., Ali, R., 2014. Crowd-centric requirements engineering, in: Proceedings of the 7th International Conference on Utility and Cloud Computing. IEEE, London, UK, pp. 614–615. doi:10.1109/UCC.2014.96 49 Sørensen, J.J.W.H., Pedersen, M.K., Munch, M., Haikka, P., Jensen, J.H., Planke, T., Andreasen, M.G., Gajdacz, M., Mølmer, K., Lieberoth, A., Sherson, J.F., 2016. Exploring the quantum speed limit with computer games. Nature 532, 210–213. doi:10.1038/nature17620 Stannett, M., Legg, C., Sarjant, S., 2013. Massive ontology interface, in: Proceedings of the 14th Annual conference on Computer-Human Interaction (SIGCHI). ACM, Christchurch, New Zealand. doi:10.1145/2542242.2542251 Straub, T., Gimpel, H., Teschner, F., Weinhardt, C., 2015. How (not) to incent crowd workers. Bus. Inf. Syst. Eng. 57, 167–179. doi:10.1007/s12599-015-0384-2 Supendi, K., Prihatmanto, A.S., 2015. Design and implementation of the assesment of publik officers web base with gamification method, in: Proceedings of the 4th International Conference on Interactive Digital Media (ICIDM). IEEE, Bandung, Indonesia. doi:10.1109/IDM.2015.7516353 Supriadi, I., Prihatmanto, A.S., 2015. Design and implementation of Indonesia united portal using crowdsourcing approach for supporting conservation and monitoring of endangered species, in: Proceedings of the 4th International Conference on Interactive Digital Media (ICIDM). IEEE, Bandung, Indonesia. doi:10.1109/IDM.2015.7516354 Surowiecki, J., 2005. The wisdom of crowds. Anchor Books, New York. Susumpow, P., Pansuwan, P., Sajda, N., Crawley, A.W., 2014. Participatory disease detection through digital volunteerism: How the doctorme application aims to capture data for faster disease detection in Thailand, in: Proceedings of the 23rd International Conference on World Wide Web (WWW’14). ACM, Seoul, Korea. doi:10.1145/2567948.2579273 Talasila, M., Curtmola, R., Borcea, C., 2016. Crowdsensing in the wild with aliens and micropayments. IEEE Pervasive Comput. 15, 68–77. doi:10.1109/MPRV.2016.18 Tauer, J.M., Harackiewicz, J.M., 2004. The effects of cooperation and competition on intrinsic motivation and performance. J. Pers. Soc. Psychol. 86, 849–861. doi:10.1037/0022-3514.86.6.849 Terlutter, R., Capella, M.L., 2013. The gamification of advertising: Analysis and research directions of in-game advertising, advergames, and advertising in social network games. J. Advert. 42, 95–112. doi:10.1080/00913367.2013.774610 Tinati, R., Luczak-Roesch, M., Simperl, E., Hall, W., 2016. Because science is awesome, in: Proceedings of the 8th ACM Conference on Web Science - WebSci ’16. ACM Press, Hannover, Germany, pp. 45–54. doi:10.1145/2908131.2908151 Tolmie, P., Chamberlain, A., Benford, S., 2013. Designing for reportability: Sustainable gamification, public engagement, and promoting environmental debate. Pers. Ubiquitous Comput. 1–12. doi:10.1007/s00779-013-0755-y Tsai, H., Bagozzi, R.P., 2014. Contribution behavior in virtual communities: Cognitive, emotional, and social influences. Manag. Inf. Syst. Q. 38, 143–163. Ustalov, D., 2015. Towards crowdsourcing and cooperation in linguistic resources, in: Proccedings of the 9th Russian Summer School in Information Retrieval (RuSSIR 2015). Springer International Publishing, Saint Petersburg, Russia, pp. 348–358. doi:10.1007/978-3-319-25485-2_14 Uzun, A., Lehmann, L., Geismar, T., Küpper, A., 2013. Turning the OpenMobileNetwork into a live crowdsourcing platform for semantic context-aware services, in: Proceedings of the 9th International Conference on Semantic Systems - I-SEMANTICS ’13. ACM, Graz, Austria, pp. 89– 96. doi:10.1145/2506182.2506194 50 Vasilescu, B., Serebrenik, A., Devanbu, P., Filkov, V., 2014. How social Q&A sites are changing knowledge sharing in open source software communities, in: Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing - CSCW ’14. ACM, Baltimore, MD, USA, pp. 342–354. doi:10.1145/2531602.2531659 Von Ahn, L., 2009. Human computation, in: Proceedings of the 46th Annual Design Automation Conference - DAC ’09. IEEE, San Francisco, CA, USA, pp. 418–419. doi:10.1145/1629911.1630023 Von Ahn, L., 2008. Designing games with a purpose. Commun. of t. ACM. 51, 58–67.doi: 10.1145/1378704.1378719 in: Proceedings of the 46th Annual Design Automation Conference - DAC ’09. IEEE, San Francisco, CA, USA, pp. 418–419. doi:10.1145/1629911.1630023 Wang, Y., Jia, X., Jin, Q., Ma, J., 2015. QuaCentive: A quality-aware incentive mechanism in mobile crowdsourced sensing (MCS). J. Supercomput. 1–18. doi:10.1007/s11227-015-1395-y Webster, J., Watson, R.T., 2002. Analyzing the past to prepare for the future: Writing a literature review. MIS Q. 26, xiii–xxiii. Whetten, D.A., 1989. What constitutes a theoretical contribution? Acad. Manag. J. 14, 490–495. doi:10.5465/AMR.1989.4308371 Wu, F.-J., Luo, T., 2014. WiFiScout: A crowdsensing WiFi advisory system with gamification-based incentive, in: Proceedings of the 11th International Conference on Mobile Ad Hoc and Sensor Systems (MASS). IEEE, Philadelphia, USA, pp. 533–534. doi:10.1109/MASS.2014.32 Xie, T., Bishop, J., Horspool, R.N., Tillmann, N., Halleux, J. De, 2015. Crowdsourcing code and process via Code Hunt, in: Proceedings of the 2nd International Workshop on CrowdSourcing in Software Engineering. IEEE, Florence, Italy, pp. 15–16. doi:10.1109/CSI-SE.2015.10 Yakushin, D., Lee, J., 2014. Cooperative robot software development through the internet, in: Proceedings of the 2014 IEEE/SICE International Symposium on System Integration (SII). IEEE, Tokyo, Japan, pp. 577–582. doi:10.1109/SII.2014.7028103 Yee, N., 2006. Motivations for play in online games. Cyberpsychol. Behav. 9, 772–775. doi:10.1089/cpb.2006.9.772 Yu, H., Lin, H., Lim, S.F., Lin, J., Shen, Z., Miao, C., 2015. Empirical analysis of reputation-aware task delegation by humans from a multi-agent game, in: Bordini, E., Weiss, Y. (Eds.), Proceedings of the 14th International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS). IFAAMAS, Istanbul, Turkey, pp. 1687–1688. Zhang, P., 2008. Motivational Affordances: Reasons for ICT design and use. Commun. ACM 51, 145–147. doi:10.1145/1400214.1400244 Zhao, Y., Zhu, Q., 2014a. Evaluation on crowdsourcing research: Current status and future direction. Inf. Syst. Front. 16, 417–434. doi:10.1007/s10796-012-9350-4 Zhao, Y., Zhu, Q., 2014b. Effects of extrinsic and intrinsic motivation on participation in crowdsourcing contest. Online Inf. Rev. 38, 896–917. doi:10.1108/OIR-08-2014-0188 Zheng, H., Li, D., Hou, W., 2011. Task design, motivation, and participation in crowdsourcing contests. Int. J. Electron. Commer. 15, 57–88. doi:10.2753/JEC1086-4415150402 51 Zuchowski, O., Posegga, O., Schlagwein, D., Fischbach, K., 2016. Internal crowdsourcing: Conceptual framework, structured review and research agenda. J. Inf. Technol. 31, 166–184. doi: 10.1057/jit.2016.14 52 Author biographies Benedikt Morschheuser is working as a researcher at the Robert Bosch GmbH and the Karlsruhe Institute of Technology (KIT). His work focuses on the use of gamification in collaborative environments, especially crowdsourcing systems and online communities. Most of his scientific contributions are empirical papers on the effects of gamification in different contexts and the designs of gamified systems. https://issd.iism.kit.edu/21_126.php Juho Hamari is a Professor of Gamification (Associate & tenure-track) and leads the Gamification Group across Tampere University of Technology, University of Turku and University of Tampere. Dr. Hamari has authored several seminal scholarly articles on games and gamification from perspective of consumer behavior, human-computer interaction and information systems science. His research has been published in a variety of prestigious venues such as Organization Studies, JASIST, IJIM, Computers in Human Behavior, Internet Research, Electronic Commerce Research and Applications, Simulation & Gaming as well as in books published by e.g. MIT Press. http://juhohamari.com Jonna Koivisto is a researcher at the Gamification Group / Tampere University of Technology. Her research focuses especially on motivations and behavior online. She has authored several seminal empirical works on gamification as well as spearheaded efforts to synthesize existing academic literature on the topic. In addition to gamification, her research interests include consumer behavior in games as well as new online business models. Koivisto’s research has been published in internationally respected scholarly journals and conferences, as well as in the popular media. http://jonnakoivisto.com Alexander Maedche is Full Professor of Information Systems at the Karlsruhe Institute of Technology (KIT) and Managing Director of the Institute of Enterprise Systems at the University of Mannheim, Germany. His research focuses on designing user-centered and intelligent digital service systems. He has published more than 100 papers in journals and conferences, such as the Journal of the AIS, IEEE Internet Computing, and Information and Software Technology. https://issd.iism.kit.edu