Players' progression through GraphoGame, an early literacy game : influence of game design and context of play

ISSN: 1795-6889 www.humantechnology.jyu.fi Volume 15(2), June 2019, 226–255 226 PLAYERS’ PROGRESSION THROUGH GRAPHOGAME, AN EARLY LITERACY GAME: INFLUENCE OF GAME DESIGN AND CONTEXT OF PLAY Abstract: Researchers of serious games frequently investigate outcomes of play but overlook the underlying game-design components that drive those outcomes. In this paper, I aim to show how game design and context of play influence progression through GraphoGame, an early-literacy game. This is done by means of two intersecting studies. The first study shows how the game can be represented by a model that explicitly hypothesizes how the interaction between the player and the game drives progression. The second study explores user data generated by first graders (N = 137) who played the game over a period of 25 weeks as part of early literacy instruction. The juxtaposition of these two studies reveals factors that influence progression. I also highlight an underdeveloped area within the research field and point to the benefits that bridging game design and outcomes of play may hold for researchers, game developers, and educators. Keywords: serious games, adaptive learning, game design, literacy games, GraphoGame ©2019 Morten Njå and the Open Science Centre, University of Jyväskylä DOI: https://doi.org/10.17011/ht/urn.201906123157 This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. Morten Njå Norwegian Centre for Reading Education and Research University of Stavanger Norway Factors Influencing Progression Through GraphoGame 227 INTRODUCTION Digital games are now an integral part of popular culture, and, in recent years, the presence of such games in educational settings has grown as well. The educational setting includes not only entertainment games repurposed for educational use but also games specifically designed to promote specific learning outcomes, which are commonly referred to as serious games (Wouters, van Nimwegen, van Oostendorp, & van der Spek, 2013). In these games, the entertainment factors typically associated with digital games are secondary to the instructional ones yet employed to motivate players to achieve better learning outcomes (Wouters et al., 2013). Although serious games do not necessarily increase players’ motivation and engagement levels, these games may lead to positive learning outcomes, especially if used as part of other instructional activities (Boyle et al., 2016; Connolly, Boyle, MacArthur, Hainey, & Boyle, 2010). To date, most researchers of serious games have investigated the outcomes of play (Boyle et al., 2016; Connolly et al., 2010), whereas less attention has been paid to how these games work (Gaydos, 2015; Lämsä, Hämäläinen, Aro, Koskinaa, & Äyrämö, 2018). Although it is quite possible to conduct such research without a comprehensive understanding of how these games work, this lack of attention to the workings of games means that the specific components driving outcomes of play may be undistinguishable from one another. Hence any insights pertaining to the game itself will be restricted to the game as a whole. Clark, Tanner-Smith, and Killingsworth (2014) argued that the design of digital tools, not the digital medium itself, is the strongest predictor of learning outcomes, and they call for research exploring how design choices in digital games influence learning outcomes. This view is shared by Lämsä et al. (2018), who called for research into how game design influences learning in children with learning difficulties. This paper contributes to this underdeveloped area within research into serious games by investigating possible connections between game design and user progression in the Norwegian version of GraphoGame. This game is described as “a technology-enhanced learning environment for learning to read” for children who are “in the early stages of their formal education” (U. Richardson & Lyytinen, 2014, pp. 39-40). For my research of game design and user progression, I conducted two studies. Based on an investigation of the literature, the first study presented a theoretical model for how progression may occur as a result of playing the game, whereas empirical research in the second study generated user data from the gameplay. Juxtaposing the findings from those two studies makes it possible to identify some underpinning factors related to game design and the context of play that drive the outcome measured—in this case, progression through the game. Researching Serious Games Part of the reason for the predominance of studies investigating outcomes of play may be that serious games are designed to promote specific learning outcomes, and so researchers may be inclined to examine whether the games do what they are supposed to do. Further, stakeholders and decision makers in the educational sector often call for research that investigates “what works” and best practices in relation to digital learning technologies (Biesta, Edwards, & Allan, 2014; Eisenhart, 2006), which may further influence researchers’ choice of focus. Because of this emphasis on measuring output, experimental research designs are commonly used in Njå 228 research into serious games. In the case of research into language learning, for example, the question asked may be whether a particular teaching strategy or activity enhances students’ learning performance (Phakiti, 2014). A systematic review of earlier research into the specific serious game studied here, GraphoGame, showed that most studies used experimental designs geared toward measuring reading outcomes from play (McTigue, Zimmer, Solheim, & Uppstad, 2019). As mentioned above, such research may be carried out without any scientific knowledge of the components that constitute the game because the object of study is the game itself, not its specific individual design elements. Hence the underpinning components that drive the output measured may remain unexplored. A good metaphor for this is that of the black box, as explained by Latour (1987). This provides a useful framework when discussing how researchers deal with complex systems such as serious games. A black box metaphor is used here in the sense of an artifact for which knowledge exists about how to use it and about the outcome of using it, but for which there is a lack of understanding about how it works. Mark Richardson (2016, p. 661), in reference to Latour’s (1987) use of the metaphor, explained that black boxes are found “where the outer skin [of an artifact] masks the inner workings and obstructs comprehension.” The use of experimental research approaches may “thicken” this outer skin in that the game as a whole may be considered the object of study. What is more, in one common view, digital learning technologies are seen as tools designed to generate outcomes (Furberg & Lund, 2016). Hence, the tool itself will be of interest only in terms of the results generated by its use. Latour (1999, p. 304) touched on this phenomenon in his claim that the success of technology actually deters comprehension: “When a machine runs efficiently, when a matter of fact is settled, one need focus only on its inputs and outputs and not on its internal complexity. Thus, paradoxically, the more science and technology succeed, the more opaque and obscure they become.” Price, Jewitt, and Brown (2013) pointed out that the research approaches taken to digital technology are trailing because of the rapid changes seen in digital technology. The lack of effective investigative approaches capable of piercing the outer skin to unpack the components of a serious game is another reason why the scope of research may have to be restricted to observing the output (e.g., learning outcomes) from playing serious games. Researchers thus may be forced to use theoretical approaches not specifically designed with the affordances of digital technology in mind. For example, a vast amount of user data is generated by players’ interaction with GraphoGame, but these data are rarely explored in-depth (McTigue et al., 2019). In this case, the sheer amount and complexity of the data, in and of itself, may obscure insight into the black box. Indeed, Saarela and Kärkkäinen (2017) pointed out that, even where high-quality data sets such as those from the Programme for International Student Assessment (PISA) are readily available, research making proper use of those data sets is surprisingly rare. However, in two related fields—learning analytics and educational data mining—approaches are being developed to facilitate the use of big data in educational settings, but the tools used have not become standard yet in researchers’ toolboxes. An Activity-Theory Perspective on Serious Games Playing serious games involves a complex interplay of factors, such as patterns for players’ interaction with the game and its components, cognitive processing during play, and factors Factors Influencing Progression Through GraphoGame 229 relating to the social context of play. Each of these may influence the outcome of use to varying degrees. Activity theory (see, e.g., Engeström, 1999; Leontiev, 1977; Vygotsky, 1978) may provide a lens through which to explore such factors. Nardi and Kaptelinin (2006, p. 10) laid out the premise of this theory: “People act as subjects in the world, constructing and instantiating their intentions and desires as objects. Activity theory casts the relationship between people and tools as one of mediation; tools mediate between people and the world.” This premise proposes that technology reaches beyond an isolated interaction between a person and an artifact (such as GraphoGame); rather, the use of technology, and by extension playing of serious games, embodies complex social dynamics. When it comes to the use of digital technology (e.g., serious games), this theory offers an approach enabling the coordination of various aspects of technology, such as physical interaction, conceptual interaction, and socialcontext interaction (Nardi & Kaptelinin, 2006). The application of activity theory to research into serious games requires a change of perspective compared with typical experimental research. As Bannon and Bødker (1991, p. 241) pointed out regarding studying artifacts (e.g., GraphoGame), “We cannot study them as things, [sic] we need to look at how they mediate use.” In other words, the focus of research must shift from the game itself to the actions taken when users interact with the game. The application of this theory creates a need to pinpoint exactly what players are doing in their interactions with a serious game. As a result, aspects influencing gameplay that otherwise tend to be hidden inside the black box may be revealed. At the core of this theory is the concept of activity. Leontiev (1977) broke down activities into three interconnected hierarchical levels. At the top of this hierarchy is the activity that a person engages in (such as playing a serious game). Actions are segments consciously chosen by a person (the subject) to attain goals (the object) that typically relate to the motivation underpinning the undertaken activity (such as solving a task in the game), whereas operations are aspects of engagement whose performance does not require any conscious thought (such as clicking a mouse button or drawing on tacit knowledge). Su, Feng, Hsu, and Yang (2013, p. 2577) stated that activity theory provides “a useful framework for conceptualizing technology as a dynamic mechanism that conditions and enables development and change in learners and in the mechanism itself.” Thus, the perception of the object of study as “a whole” limits the research approaches taken. The activity theorybased model for analyzing serious games and conceptual design proposed by Carvalho et al. (2015) made a distinction among gaming, learning, and instructional activities, further highlighting how the process of playing the game involves actors other than the player, such as game designers and teachers. The player engages in gaming and learning activities, which have separate tools and objectives. Game designers influence the game intrinsically, whereas the teachers responsible for deploying the game influence it extrinsically. Hence, this model provides a way to break down the components of the serious game and identify how players, designers, and teachers engage with it. The premise of this approach—that human activities take place in social contexts—enables not only exploration of the various actors involved with the game (i.e., researchers, designers, and others) and their interaction with the player through the game’s components but also exploration of the agents actually present during the playing of the game (i.e., teachers and students). Njå 230 Learning and Entertainment in Serious Games Balancing learning and entertainment in a game can be a complex prospect. Arnab et al. (2015) called attention to the challenge of incorporating established pedagogical approaches into serious games, that is, accounting for differences in perspectives among game designers and educators of what a learning game should be. With game designers leading the development process, games may be entertaining but lack essential processes for knowledge acquisition. However, when educators are in charge, the game may be efficient as a learning tool but not fun or motivating to play (Marne, Wisdom, Huynh-Kim-Bang, & Labat, 2012). Nothing prevents these two approaches from coexisting, but it would require stakeholders (i.e., game experts and pedagogical experts) to share a common language (Marne et al., 2012). Even so, as serious games are intended to promote knowledge, pedagogical approaches should underpin the design. GraphoGame emphasizes the serious element of serious games but also makes use of game elements intended to engage and motivate players. A synthesis of research on Graphogame (McTigue et al., 2019) showed that GraphoGame may support users in developing sublexical skills and improving letter–sound knowledge and phonological processing. However, these authors found that the game supported better word reading only with strong adult interaction during play. McTigue et al. (2019) also provided an overview of Graphogame’s theoretical grounding, which included the simple view (Gough & Tunmer, 1986) word reading (Ehri, 2005), psycholinguistic grain size theory (Ziegler & Goswami, 2005), and orthographic depth hypothesis (Katz & Frost, 1992). The Present Studies The present paper is based on two studies. The aim of the first study was to investigate how game design influenced players’ progression through the game. The related research question was how the interaction between player and game was operationalized. The second study drew upon the exposition of game elements discovered in the first study and sought to investigate the direct consequences of this game design. The aim was to document any progress that became evident in 137 Norwegian first graders playing GraphoGame four times a week for 25 weeks and to relate this progress to the design elements of the game. The related research question regarded what differences in progress could be seen between a group of students initially identified as being at risk of reading and writing difficulties and a group initially identified as not being at risk of such difficulties. User data were collected at 5-week intervals. They showed how far the students progressed through the game during the timeline of play. The division of students into groups was based on the results of a screening test administered at the onset of schooling. The at-risk group consisted of students identified as being in danger of developing reading difficulties (n = 17) whereas the regular (not at-risk) group consisted of the remaining students (n = 130). This division was made to explore the possibility of different progression trajectories based on students’ starting point. Factors Influencing Progression Through GraphoGame 231 STUDY 1: OPERATIONALIZING GRAPHOGAME The aim of this first study was to explore the influence of GraphoGame’s design on user progression. Hence, the primary research question sought to articulate how the interaction between player and game could be operationalized. In response to that question, a conceptual model detailing how progression occurs through GraphoGame was developed. This model pinpoints central design elements of the game, identifies how players interact with the game, and demonstrates how this interaction drives progression. The main function of this model is to make explicit certain components that would otherwise be hidden in the black box. Theoretical Grounding for the Model Generally speaking, the ways in which information flows between the player and the game influences how the interaction ultimately drives progression through play. Earlier models exploring the interaction between computers and humans provided inspiration for the operationalization of GraphoGame. Schomaker et al. (1995) presented a model providing a bird’s-eye view of the interaction information flow as it alternates between human and computer through the interface (Figure 1). In brief, the computer captures human output (e.g., touch, voice) from its input modalities (e.g., mouse, keyboard, touch screen, microphone), processes this information according to its programming (marked as “cognition” in the model), and produces output media. The user perceives those media through human input channels (e.g., visual, audible, and tactile) and then mentally processes that information, which leads to the next cycle of information flow. The computer, the interface, and the human can be considered separate spaces in this interaction. Although this is a simplistic model, it provided a starting point for laying out the various spaces and stages in the model presented in this study. Figure 1. Basic model of human–computer interaction (adapted from Schomaker et al., 1995). This figure illustrates how information flows between computers and humans through various input and output channels, modalities, and media. Humans use their cognitive abilities to interpret the computer output and then take action by using their output channels. The computer processes this output through its programming (“cognition”) and provides output media through the interface. Njå 232 Bienkowski, Feng, and Means (2012) described the structure of an adaptive-learning system reflecting the continuous interaction between a student and that system (Figure 2). This interaction generates user data that are stored in a database. Those data are analyzed by a predictive model that aggregates the data, presents the aggregate to teachers through a dashboard, and feeds data about the student to its adaptation engine. This engine generates output content that suits the individual requirements of the student. (Some adaptive-learning systems, but not that of the present version of GraphoGame, also include an intervention engine that allows teachers or administrators to directly influence the content delivered to the student.) Interactive Model of Progression Through GraphoGame GraphoGame provides a learning environment where players may practice sound–letter correspondences. The content is presented as multiple-choice tasks, which represent the learning activity. A game starts with the player selecting one of of the nine possible game modes.1 A game mode represents a series of tasks that share the same theme. Figure 3 shows an example of a task from the Balloon Game. Figure 2. Diagram depicting data flow (the arrows) in a typical adaptive-learning system (ALS; Bienkowski et al., 2012). The model shows how students, administrators, and teachers interact with the various components of an ALS. The student interacts with the content (Line 1, bidirectional to emphasize the interaction aspect) and the outcome of this interaction is stored as student learning data (Line 2). These learning data, along with other student information, are funneled to the ALS’s predictive model (Line 3). This combined information is used to create new content through the adaption engine (Line 4), which completes the cycle between student and ALS. In addition, this information can also be accessed by students, teachers, and administrators through a dashboard (Line 5), providing insight into students’ performances. In addition, teachers and administrators may influence the content directly through the ALS’s intervention engine. Factors Influencing Progression Through GraphoGame 233 As the player is presented with the graphical elements, as seen in Figure 3, the sound representing the word gå [walk] is played out loud. The player’s task is to select the word that corresponds with this sound from three possible choices (i.e., one target and two distractors). Following this task, the player is presented with another task until all tasks in the game level have been completed. The yellow bar at the bottom of the screen indicates how far through this game the player has progressed. At completion, the player gets to choose a new game mode. The player may complete several game modes during a single play session. In the following, I propose a model that details how the interaction between players and the game drives progression. The model produced during the first study (Figure 4) consists of five stages of a cycle that are positioned across three spaces: the game space represents the programming and data-access layers of the game, the interface space details the way players and the game interact, and the mind space deals with how the player processes the information presented through the user interface. The five stages of interaction are distributed across three independent but interlinked spaces. The game state represents the values stored in the game to represent players’ current progress through Figure 3. An example of a multiple-choice task in Graphogame. Figure 4. Five stages show the interaction between player and GraphoGame that drives progression through the game. The stages move from the program (game design), through the interface, and into the mind of the user, resulting in user reaction within the interface and a change in the game state. The cycle then repeats. Njå 234 the game at any given time. Those values are used to generate trials (the learning task; trial generation), which are presented visually and aurally to the player through the user interface (trial presentation). The player then internally processes those visual and auditory cues (mental processing), which causes the player to take an action (i.e., a response to the perceived task) through interacting with the user interface (user action). Finally, the result of this action feeds and alters the game state, which in turn establishes the basis for the next trial, and so on. Progression through the game then takes place through continuous repetition of these stages of the model. As an example of how these activity stages unfold, I present a case where the player first selects the Pirate Game on the selection screen shown between games. The first trial (Figure 5) is generated based on data relating to the player’s current progression (i.e., based on prior play or from an initial level), taken from the game state. At the start of this trial, a recording of the sound /e/ is played and the letters e, a, u, b and r are shown on the screen. The player’s task is to identify the letter (i.e., click e) that corresponds to the sound played. In this example, the player clicks on the correct item, and the game state is altered to reflect this. A new trial is then generated (Figure 6). The words riv [tear], mur [wall], rot [root] and rim [rhyme] are shown on the screen. The sound /riv/ is played and the player again makes the right choice by clicking on the balloon containing this word. After successfully completing the task, the player earns a reward (indicated by the coin appearing above the word riv after selecting it). The student then performs six more trials to complete the selected game mode. At this point, the player is presented with a new gameselection screen and may select a new game mode. In the following subsections, this example will be discussed in greater detail against the background of the individual stages of the model. Game State The game state represents the current state of various variables detailing a player’s progression and performance in the game at any specific time during play. As the player interacts with the game, these values change to reflect the outcome of the player’s actions. Two clusters of variables are discussed as part of the adaptive interaction cycle: player knowledge and player performance. The first of these clusters, player knowledge, includes variables that reflect how well the player knows the content of the game (a value assigned by the game that may or may not reflect a user’s actual knowledge). These variables persist between game sessions. Each item in the game is Figure 5. GraphoGame trial, player’s initial trial. Figure 6. GraphoGame trial, player’s subsequent trial. Factors Influencing Progression Through GraphoGame 241 are referred to as P1 … P5, whereas measurement points (snapshots) of progression are referred to as M1 … M5. Results: Play Sessions, Trials Played, and Time Spent Playing Table 2 shows that the regular group played more sessions (M = 83.7, SD = 7.1) than the atrisk group (M = 79.1, SD = 8.0). However, this difference was not significant (p = .039). Table 3 shows the total number of minutes spent playing in the different measuring periods and in total across the timeline of play. The regular group (M = 516, SD = 82) played the game for slightly longer each session than the at-risk group (M = 479, SD=76), but this difference was not significant (p = .078). Table 4 shows the number of trials completed by the students during each measuring period and in total. The regular group played more trials (M = 7260, SD = 1864) than the atrisk group (M = 5948, SD = 847), and this difference was strongly significant at p < .001. During the intervention timeline, the at-risk group actually kept pace with the regular group during the first two periods (P1: p = .557 and P2: p = .946), but the difference between the groups was strongly significant throughout the rest of the intervention (P3: p = .002, P4: p < .001, P5: p < .001). Table 5 shows the average number of minutes spent playing during each play session. The regular group (M = 6.15, SD = 0.72) and the at-risk group (M = 6.04, SD = 0.57) played for a similar duration per session; there is strong statistical support for this conclusion (p = .481). Table 2. Number of Play Sessions in Each Measuring Period (P1 to P5) and in Total. Group P1 P2 P3 P4 P5 TOTAL Regular M SD 17.4 1.6 15.2 1.7 18.3 2.0 18.1 2.5 14.5 3.0 83.7 7.1 At Risk M SD 16.8 2.5 15.8 1.3 17.3 2.1 16.7 3.0 12.6 4.1 79.1 8.0 Note. M = mean; SD = standard deviation Table 3. Time Played, in Minutes, in Each Measuring Period (P1 to P5) and in Total. Group P1 P2 P3 P4 P5 TOTAL Regular M SD 114 18 92 17 115 21 109 27 86 29 516 82 At Risk M SD 109 20 94 12 106 19 99 30 71 32 479 76 Note. M = mean; SD = standard deviation. Njå 242 Table 4. Number of Trials Played in Each Measuring Period (P1 to P5) and in Total. Group P1 P2 P3 P4 P5 TOTAL Regular M SD 1648 355 1256 360 1567 489 1560 673 1229 525 7260 1864 At Risk M SD 1598 323 1251 268 1298 272 1035 330 766 310 5948 847 Note. M = mean; SD = standard deviation. Table 5. Time Played per Session, in Minutes, in Each Measuring Period (P1 to P5) and in Total. Group P1 P2 P3 P4 P5 TOTAL Regular M SD 6.51 0.80 6.02 0.88 6.25 0.93 5.98 1.04 5.86 1.28 6.15 0.72 At Risk M SD 6.51 0.55 5.98 0.74 6.10 0.80 5.83 0.98 5.56 1.13 6.04 0.57 Note. M = mean; SD = standard deviation. Results: Longitudinal Progression Overall Progression Overall progression (see Appendix B, Table B1) was measured in terms of the number of known items out of the total number of items in the game (N = 730) at each measuring point along the timeline. The boxplots shown in Figure 9 give an overview of total progression across the intervention period. The boxplots indicate the percentages of known items (y-axis) for the regular group and the at-risk group at the five measuring points (x-axis). There were no outliers in either group. The regular group had progressed further than the at-risk group at all measuring points, and this difference was strongly significant at p > .001 in all cases. The interquartile range was fairly consistent along the timeline for both groups, with the at-risk group more closely clustered together. For the regular group as a whole, the range of progression covered almost the full inventory of items (especially towards the end of the intervention period), and a steady rise in median progression was seen throughout the period; the at-risk group progressed more slowly and had a more limited overall range of progression. Comparison of the two groups shows that regular group accelerated away from the at-risk group throughout the intervention: At M1, the average difference between the groups was 61 known items, but at M5 this had increased to 141 known items. Factors Influencing Progression Through GraphoGame 243 Figure 9. The figure shows boxplots of percentage overall progression for regular and at-risk students through GraphoGame at each of the five measuring points (M1 … M5). Letter Content Progression Letter content (see Appendix A, Letter Items, and Appendix B, Table B2) is the first type of content encountered in the game, and progression to other types of content is held back until a player knows at least 40% of the letters. Given that M1 occurred after approximately 18 play sessions, the specific point where the students progressed from only letter content to letter and syllable content was not captured. The boxplots in Figure 10 show the median and percentile progression for known letters throughout the intervention period. Most students in the regular group knew all letters at M1 and onwards (Mdn = 24). The at-risk group progressed more slowly, with a median of 20 known letters (83.3%) at the first measuring point (M1) and 23 (95.8%) at M2. Further along the timeline, most at-risk students knew all the letters (Mdn = 24). It should be noted that the outliers who can be seen in the regular group represent a small group of students (less than 5%) who progressed significantly more slowly than the rest of that group. There was a significant difference between the groups at all measuring points along the timeline (p < .001). Syllable Content Progression Syllable content (see Appendix A, Syllable Items, and Appendix B, Table B3) is the second content type encountered in the game. The boxplots in Figure 11 show the different patterns for the two groups’ progression with regard to syllable content. At M1, the regular group knew a median of 90 syllables (33.1%). This increased to 169 known syllables (62.1%) at M2 and to 251 known syllables (92.3%) at M3, at which point the regular group had thus largely mastered all of the syllable content. After this there was a plateau during the rest of the intervention period. For the at-risk group, the progression curve is rather linear throughout the intervention period. At M1, the median was 23 known syllables (8.5%). At M2, this had increased slightly to 37 known syllables (13.6%), and further along the timeline there Njå 244 Figure 10. The figure shows boxplots of letter–content progression for regular and at-risk students through Graphogame at five measuring points (M1 … M5). The y-axis shows how many letters were known out of a maximum of 24 letters. Figure 11. The figure shows boxplots of syllable-content progression for regular and at-risk students at five measuring points (M1 … M5). The y-axis shows how many syllables were known out of a maximum of 272 syllables. there was a steady increase: 83 known syllables (30.5%) at M3, 111 known syllables (40.8%) at M4, and 142 known syllables (52.2%) at M5. The interquartile range narrowed in the regular group, largely as a result of the plateau, but remained similar across the intervention period in the linearly developing at-risk group. Notably, syllable content was the content type where the members of the at-risk group spent most of their game-playing time, whereas most members of the regular group progressed past the syllable stage during the third measuring period. There was a significant difference between the groups in all five periods (p < .001). In contrast to letter content, which most players learned during the first period, progression with respect to syllable content was spread out across the periods to a greater extent. Factors Influencing Progression Through GraphoGame 245 Word Content Progression Word content (see Appendix A, Word Items, and Appendix B, Table B4) is the last type of content encountered. This content type also accounts for most of the content in GraphoGame, which has 60% more word items than syllable items. The boxplots in Figure 12 show that only the regular group made any significant progress with word content. At the first two measurement points, only a few students in the regular group (the outliers) had made any significant progress in terms of words known. At M3, the median for the regular group was 6 known words (1.4%), whereas at M4 the median had increased to 35 known words (8.1%). However, at that point, the upper quartile was already at 197 known words (45.8%). At M5, the median was 76 known words (17.6%) and there had been a slight increase in the upper quartile to 214 words (49.8%). The at-risk group made only negligible progress with respect to word content throughout the intervention period. Figure 12. The figure shows boxplots of word-content progression for regular and at-risk at five measuring points (M1 … M5). The y-axis shows how many words were known out of a maximum of 434 words. GENERAL DISCUSSION Players’ progression through GraphoGame is inherently tied both to features of game design and to the context of play: The game design establishes the premises for progression, whereas playing the game realizes those premises. In the following subsections, I discuss the three overlapping factors that influence progression, namely content design, play time, and adaptation design. These factors were identified as a result of applying the knowledge obtained from Study 1 in order to contextualize the findings made in Study 2. In reading the analysis below, I must emphasize that I do not make any strong claims about learning outcomes as a result of this research. It is impossible to know exactly, from the data collected, what the participants have internalized as a result of playing even though progression is operationalized using the measurements of the number of known items (out of all 730 items Njå 246 available). Further, the factors that emerged from the juxtaposition of the two studies are not intended as a definite exploration of possible factors. Progression Through GraphoGame Content Design Most students did not encounter the full range of content available during the play period, yet even so, progression unfolded differently in the regular group as compared to the at-risk group. At the last measuring point, after 25 weeks of play, the overall progression of the regular group fell just short of the halfway point in terms of items known (49.7%), with a mean of 7,292 trials played across 85 play sessions. For the at-risk group, progression at that point was less than a quarter (22.7%), with a mean of 5,986 trials played across 79.8 play sessions. A closer look at progression for the individual content types provides a more nuanced picture. The transitions between content types show how quickly players moved from letter content via syllable content (two- or three-letter words) to word content (four- to six-letter words and minimal pairs). Most students in the regular group knew all of the letter content at the first measuring point (i.e., after an average of approximately 17 play sessions across 5 weeks), whereas many students in the at-risk group were still engaged with letter content at the second measuring point. In other words, it took the average at-risk student more than 30 play sessions across 10 weeks to progress past the letters. For syllable content, the regular group saw steady progression from the start, leading to a plateau, reflecting the fact that the majority knew most or all syllable content at the third measuring point (about 50 play sessions across 15 weeks). By contrast, the at-risk group manifested a linear progression throughout the play period, with a median progression just past the halfway point for syllables at the end of the 25-week period. When it comes to word content, only the regular group achieved any noticeable progression, and it was not until the third measuring point that the majority had made any significant progress. At the end of the play period, the median proportion of known word items was 17.6% for the regular group and zero for the at-risk group. One possible interpretation of these results is that there is too much content in the game for the average pupil to encounter during an extended play period. However, other factors also need to be considered. The time spent playing and the workings of the adaptation engine cannot be separated from content design when discussing progression. In reality, both at-risk and regular students should be able to complete the game, given enough time, and progression might have been faster if the adaptive algorithms had been adjusted. That said, faster progression does not equal a better game. Even though no measurement specifically pertains to learning, the content of GraphoGame is designed and distributed in line with established theories of language learning that suggest that a slower pace of progression may be better in regard to long-term learning outcomes. In other words, content design is not just about the amount of content included, but also about instructional aspects leading to mastery of fundamental elements essential to reading. The time factor can be seen as consisting of two aspects: how much time is spent playing and how the adaptation engine adjusts the measure of player performance between game sessions. In this case, playing GraphoGame was an activity carried out as part of regular classes, with 10 minutes allocated to play and 5 minutes for the teacher to start up the activity. The data collected show that of their 10 play minutes, players devoted approximately 6 minutes to engaging in trials. The data do not show how much time students devoted to other activities in Factors Influencing Progression Through GraphoGame 247 the game, such as browsing the in-game store or customizing their avatar, but this should account for a significant part of the remaining four minutes. Adding a single minute of effective gameplay to each play session would increase the total number of trials played during the entire play period by 800–1200, if all other factors remained the same. The short bursts of play a few times a week are engulfed in large chunks of time when the students are not playing. During that time, knowledge may regress, meaning that what was known at the end of the last play session may not have been retained over time. As mentioned above, although the game stores knowledge about each item between play sessions, player performance resets at the start of each new session. Although player performance is not the only factor influencing content generation, this does mean that a player will have to perform a few trials before attaining his or her previous performance level. The likelihood of being presented with easier content from lower dimensions and content types is thus higher at the beginning of a new session. Considering that a typical play session involves only around 70–90 trials, this design choice may play a significant role for the pace of progression. Adaptation Design The adaptive algorithm in GraphoGame is calibrated to let players encounter items and correctly identify them enough times for it to be reasonably certain that the item in question has been mastered. In order to progress through the game, a player needs to know most items from lower dimensions and to perform better than the target percentage for each content type. This means that if a player makes a few wrong choices, the adaptation engine will provide already-known content from lower dimensions. This slows down the pace of progression but may result in better long-term learning outcomes as the adaptive algorithm ensures that the player knows all of the letters that make up items in the syllable and word content. This principle may reflect a good design choice, as it ensures, for example, that players’ progression is not overextended. However, the adaptation engine needs to serve two distinct purposes. Besides preventing students from progressing too fast, it must also correctly and effectively identify items as known to ensure a steady rate of progression and avoid students being held back (which may cause them to become bored). GraphoGame has a single algorithm for all players, which means that this serves as a significant factor in progression. The players’ progression as observed over the 25-week play period opens up interesting inquiries, such as how much content should be included in relation to the total play period and how long and how frequently should the students play the game. The discrepancy found here between the regular group and the at-risk group brings additional questions to the fore, such as whether, and if so how, the adaptation engine could/should be altered to promote better learning in players of varying skill levels, or whether the learners’ progression would be better served by developing multiple versions of that engine. Although answering those questions falls outside the scope of this paper, my surfacing the questions has resulted from the juxtaposition of the study of game design and content with the study of outcomes of use, suggesting that the overall design of the present paper, with two studies of different types, has been a fruitful one. The Activity of Playing GraphoGame Activity theory establishes that human activity occurs in social contexts. From this perspective, the activity of playing GraphoGame may expand beyond the direct interaction between Njå 248 students and the game. On the one hand, the teachers who used GraphoGame as part of early reading instruction extrinsically influenced the activity, as did the researchers who determined how GraphoGame should be engaged (e.g., 10-minute play sessions 3-4 times a week). On the other hand, GraphoGame’s developers and content designers intrinsically influenced the activity through the design of the game (e.g., adaptation design, trial presentation, content organization). In other words, some actors influenced the immediate environment of play, whereas others extended their influence through GraphoGame’s code. A shared motive between the influencing actors may be that playing GraphoGame helps students on their path toward learning to read by strengthening their knowledge of letter–sound correspondences. As the students act with the game by following their own objectives (e.g., solving tasks, earning coins) and using the various tools the game offers, they, perhaps inadvertently, work towards this motive. CONCLUSIONS The median progression through GraphoGame in terms of items known was approximately 50%, with the at-risk group progressing significantly more slowly than the regular group. The juxtaposition of the two studies carried out yielded three factors that may provide part of the explanation for this finding: (a) the amount and structure (types and dimensions) of the content established the boundaries for maximum progression, whereas (b) the time spent playing the game and (c) the way the adaptation engine worked influenced how far through the game the students progressed. The factors discussed are not weighted in terms of their influence on progression. This does not mean that they are of equal influence, but rather that their relative importance is not considered here. Further, the progression as measured is not intended to reflect the quality of the factors discussed. This approach to research is important yet underrepresented in the research field, where the prevailing approach is to measure output from use without regard to the underpinning factors behind this output. Drawing connections between specific game-design components and output can be beneficial in many ways. For teachers, insights of such connections may inform them of better ways of deploying the game in the classroom. For example, knowing that at-risk pupils progress significantly slower may be used to provide them with additional play time or adult supervision during play. For game developers, there are insights that may inform of ways to improve the game. For example, the content in the game may benefit from being restructured and the possibility of different versions of the game to better suit different players may also be considered. To draw such connections, game design needs to be researched and understood with equal emphasis as on designing appropriate research means to capture data about players’ interaction with specific game components. As this research has demonstrated, such connections can reveal important and actionable data that may be implemented to improve similar interventions. Further, this research revealed knowledge that may lead to better informed decisions by those who use GraphoGame, for example, as part of language instruction in the classroom or when designing studies. Although various factors influencing progression have been brought to light, much still resides unknown in the black box. For instance, what activities are the students engaged with in the 4 minutes during each play session when they are not actively engaged with trials? Further, does the knowledge GraphoGame holds in its data banks align with knowledge measured outside of the game? Last, what are the reasons for the discrepancy in progression Factors Influencing Progression Through GraphoGame 249 between the regular and at-risk students? These are questions where the answers are still obscured, and thus warrant further research. IMPLICATIONS FOR RESEARCH This paper provides an example of how studies of game design and outcome from use may be juxtaposed. This research also highlights how such an approach may yield insights about a game that are beneficial to researchers, game developers, and educators. Although my data highlight the benefits for GraphoGame, other serious games could benefit as well. Similar studies should be carried out to establish a broader understanding of which components in a game are linked to outcome as measured. Even mainly experimental studies can be expanded in scope to include knowledge that may feed back into design or be applied in educational settings. Researchers should strive to develop methodologies specifically designed to enable the drawing of parallels between game design and outcomes from use. ENDNOTES 1. The nine game modes are balloon game, fishing game, flow game, basic game, ladder game, pirate game, race game, star game, and basic GraphoGame. Each game mode has slight variations in game play and how the multiple-choice tasks work. 2. The likelihood of letter content is between 33% and 100%. If the player knows all the letters, the likelihood is between 5% and 100%. 3. If the player performance is higher than 0.7, the likelihood of each type of content is recalculated in favor of word content. Further, if the performance is even better (higher than 0.9), the likelihood of syllable content is capped at 30%. 4. The recalculated formula for the likelihood of words takes into consideration the percentage of known syllables and the current performance with syllable content. Higher values indicate higher probability for word content. 5. The chance for word content equals 1 minus the chance for letters and syllables. REFERENCES Arnab, S., Lim, T., Carvalho, M. B., Bellotti, F., Freitas, S., Louchart, S., Suttie, N., Berta, R., & De Gloria, A. (2015). Mapping learning and game mechanics. British Journal of Educational Technology, 46, 391–411. https://doi.org/10.1111/bjet.12113 Bannon, L. J., & Bødker, S. (1991). Beyond the interface: Encountering artifacts in use. In J. M. Carroll (Ed.), Designing interaction: Psychological theory at the human-computer interface (pp. 1–32). New York, NY, USA: Cambridge University Press. Bienkowski, M., Feng, M., & Means, B. (2012). Enhancing teaching and learning through educational data mining and learning analytics: An issue brief. Washington DC, USA: SRI International. Retrieved from https://tech.ed.gov/wp-content/uploads/2014/03/edm-la-brief.pdf Biesta, G., Edwards, R., & Allan, J. (2014). Introduction: The theory question in education and the education question in theory. In G. Biesta, J. Allan, & R. Edwards (Eds.), Making a difference in theory: The theory question in education and the education question in theory (pp. 1–8). London, UK: Routledge. Njå 250 Boyle, E. A., Hainey, T., Connolly, T. M., Gray, G., Earp, J., Ott, M., Lim, T., Ninaus, M., Ribeiro, C., & Pereira, J. (2016). An update to the systematic literature review of empirical evidence of the impacts and outcomes of computer games and serious games. Computers & Education, 94, 178–192. https://doi.org/10.1016/j.compedu.2015.11.003 Carvalho, M. B., Bellotti, F., Berta, R., De Gloria, A., Sedano, C. I., Hauge, J. B., Hu, J., & Rauterberg, M. (2015). An activity theory-based model for serious games analysis and conceptual design. Computers & Education, 87, 166–181. https://doi.org/10.1016/j.compedu.2015.03.023 Clark, D. B., Tanner-Smith, E. E., & Killingsworth, S. (2014). Digital games, design, and learning: A systematic review and meta-analysis [executive summary]. Retrieved from http://www.sri.com/sites/default/files/publications/digital-games-design-and-learning- executive_summary.pdf Connolly, T. M., Boyle, E., MacArthur, E., Hainey, T., & Boyle, J. M. (2010). A systematic literature review of empirical evidence on computer games and serious games. Computers & Education, 59, 661–686. https://doi.org/10.1016/j.compedu.2012.03.004 Ehri, L. C. (2005). Learning to read words: Theory, findings, and issues. Scientific Studies of Reading, 9(2), 167– 188. https://doi.org/10.1207/s1532799xssr0902_4 Eisenhart, M. (2006). Qualitative science in experimental time. International Journal of Qualitative Studies in Education, 19(6), 697–707. https://doi.org/10.1080/09518390600975826 Engeström, Y. (1999). Activity theory and individual and social transformation. In Y. Engeström, R. Miettinen, & R. Punamäki (Eds.), Perspectives on activity theory (pp. 19–38). Cambridge, UK: Cambridge University Press. Furberg, A., & Lund, A. (2016). En profesjonsfaglig digitalt kompetent lærer? Muligheter og utfordringer i teknologirike læringsomgivelser [A professional digital competent teacher? Possibilities and challenges in technology rich learning environments]. In R. J. Krumsvik (Ed.), Digital læring i skole og lærerutdanning (pp. 26–43). Oslo, Norway: Universitetsforlaget. Gaydos, M. (2015). Seriously considering design in educational games. Educational Researcher, 44(9), 478–483. https://doi.org/10.3102/0013189X15621307 Gough, P. B., & Tunmer, W. E. (1986). Decoding, reading, and reading disability. Remedial and Special Education, 7(1), 6–10. Katz, L., & Frost, R. (1992). The reading process is different for different orthographies: The orthographic depth hypothesis. In R. Frost & L. Katz (Eds.), Advances in psychology: Orthography, phonology, morphology, and meaning (pp. 67–84). Amsterdam, the Netherlands: Elsevier North Holland Press. https://doi.org/10.1016/S0166-4115(08)62789-2 Lämsä, J., Hämäläinen, R., Aro, M., Koskimaa, R., & Äyrämö, S. M. (2018). Games for enhancing basic reading and maths skills: A systematic review of educational game design in supporting learning by people with learning disabilities. British Journal of Educational Technology, 49(4), 596–607. https://doi.org/10.1111/bjet.12639 Latour, B. (1987). Science in action: How to follow scientists and engineers through society. Cambridge, MA, USA: Harvard University Press. Latour, B. (1999). Pandora’s hope: Essays on the reality of science studies. Cambridge, MA, USA: Harvard University Press. Leontiev, A. (1977). Activity and consciousness. In Philosophy in the USSR: Problems of dialectical materialism (pp. 180–202). Moscow, Russia: Progress Publishers. Lundetræ, K., Solheim, O. J., Schwippert, K., & Uppstad, P. H. (2017). Protocol: “On Track,” a group-randomized controlled trial of an early reading intervention. International Journal of Educational Research, 86, 87–95. https://doi.org/10.1016/j.ijer.2017.08.011 Marne, B., Wisdom, J., Huynh-Kim-Bang, B., & Labat, J.-M. (2012). The six facets of serious game design: A methodology enhanced by our design pattern library. In A. Ravenscroft, S. Lindstaedt, C. D. Kloos, & D. Hernández-Leo (Eds.), 21st century learning for 21st century skills (pp. 208–221). Heidelberg, Germany: Springer.