Video game creation by students: a proposal for a systematic review of the literature

9th International Conference of Education, Research and Innovation, 2016 Video game creation by students: a proposal for a systematic review of the literature Ana C.R. Martins, Simone Maneira, Lia Oliveira Universidade do Minho, Centro de Investigação em Educação, Instituto de Educação (PORTUGAL) Abstract In order to better understand the potential benefits of the use of video game creation as a pedagogical strategy, it is important to know and to have a critical overview of the available research findings on this subject matter. Our research question is thus the following: What is the reported research concerning video game creation by students? The main goals of our study are: 1. To identify and map the research conducted so far within this subject area; 2. To search for and analyse evidence of relations between video game creation by students and behavioural and/or learning outcomes. In this paper we present a protocol for a systematic review of the literature intended to address the proposed research aims. Keywords: Constructionism, Video Games, Pedagogical Strategies, Educational Technology, Student Authoring, Systematic Review of the Literature 1. INTRODUCTION In the last decades video games have been under investigation as teaching and learning tools [1] [2] [3]. Though the most common approach to the use of video games in education has been embedded in instructivist pedagogical strategies (placing students in the role of players), there is a growing interest in the use of constructionist strategies (placing students in the role of educational game designers). Constructionist approaches are supported by the educational theory of Constructionism [4], are backedup by the benefits of learning-by-design [5], and generate activities well aligned with the so-desired 21st century skills. Adding to that, the increase of availability of tools that allow to design and develop games in simpler ways, and the contemporary culture of user-generated content, make it for an interesting time to use game design in educational settings. To understand the potential benefits of the use of game creation as a pedagogical strategy, it is important to identify and analyse the existing evidence published in this field. We aim to know what is the reported research concerning video game creation by students. To address this question we opted for a research methodology that has been historically connected to issues of evidence, decision and intervention in Health Sciences: a Systematic Review of the Literature [6] [7] [8]. With this paper we intend to present a protocol proposal for a systematic review of the literature with the aim of responding to two main goals: 1. Identify and map the research conducted so far in this subject area in terms of objectives, methodology, context, participants, duration, categorization of games, software used, results; 2. Search for and analyse evidence of relations between video game creation by students and behavioural or learning outcomes. Although the present study is still in progress we have decided to present the protocol in use as we believe it will be of great interest to those concerned with conducting a systematic review in educational sciences. In this paper we will also present fragments of initial findings. We intend to conclude the systematic review and present the full spectrum of results in a forthcoming paper with the aim of contributing to a better understanding of the research related to a constructionist approach of the use of video games in education, worldwide, in order to inform and guide teachers, researchers, decision-makers and future research initiatives. 2. METHODOLOGY In this systematic review of the literature we decided to use as guide the nine steps suggested by Boland, Cherry and Dickson [7], which we will also use to structure this section: 1. Performing scoping searches, identifying the review question and writing the protocol; 2. Literature searching; 3. Screening titles and 9th International Conference of Education, Research and Innovation, 2016 abstracts; 4. Obtaining papers; 5. Selecting full-text papers; 6. Quality assessment; 7. Data Extraction; 8. Analysis and synthesis; 9. Writing and editing. 2.1 Performing scoping searches, identifying the review question and writing the protocol (Step 1) In the initial stage we performed scoping searches using Google Scholar, ERIC and EBSCO. The scoping searches allowed us to identify background literature, to refine the review question, and to refine the search strategy, identifying keywords, creating search strings, and defining inclusion criteria. In this phase we have created a protocol that set out the approach used to answer our review question: What is the reported research concerning video game creation (video game design, modification and/or development) by students (in K-12 educational stages)? In order to answer the review question we identified two main research goals: 1. To identify and map the research conducted so far concerning video game creation by students; 2. To search for and analyse evidence of relations between video game creation by students and behavioural or learning outcomes. The proposed systematic review is based on the following inclusion criteria: 1. Published literature (journal articles and conference proceedings); 2. English language; 3. Empirical studies concerning video game creation (video game design, modification and/or development) by students of any age and educational stage (K-12), in formal or informal learning settings; 4. Peer-reviewed publications. While in this stage, in order to refine the review question and inclusion criteria for studies to be considered in the review, we have developed a PICO (population, intervention, comparator and outcome) table, including also Setting and Study Design, to clearly define the key components of the research question (see Table 1). Table 1 - A detailed PICO table Review question What is the reported research concerning video game creation by students? Population Students in any of the (K-12) educational stages. Intervention Video game design / ideation, video game modification, and/or video game development. Comparator The stated interventions compared with other pedagogical strategies, placebo or no intervention. Outcomes Any positive or adverse intervention-based outcome, any objective educational-based outcome measure (behavioural or learning outcome). Any subjective outcome (behavioural or learning outcome) whether identified through thematic analysis or quantitative data collection methods. Setting Formal and informal learning settings such as educational institutions, student homes, after-school clubs, holiday camps, ... Study design All. The process to identify the keywords and search strings to use was long and iterative. We started by identifying keywords within the fields of Population, Intervention and Setting. After several refining cycles through scoping searches, we have come up with three main thematic fields: video game, creation, and student. For each field we have produced an exhaustive list of relevant keywords. This list was computed in four stages: 1. Looking up words in dictionaries and respective thesaurus, using at least two different sources; 2. Adding words the researchers felt as relevant due to their own experience and due to the scoping searches; 3. Adding keywords retrieved from relevant literature in the field of the research question; 4. Looking up at every word, searching it using Google, Google Scholar and ERIC, and eliminating the less relevant ones (due to their definition, for being less frequent or not usually used within this context). 9th International Conference of Education, Research and Innovation, 2016 2.2 Literature searching (Step 2) The electronic databases searched included those identified as relevant to education, information technology and social science. A total of nine databases were used during this step: ACM (Association for Computing Machinery) Digital Library, EBSCOhost Online Research Databases, Emerald, ERIC (Education Resources Information Centre), IEEE (Institute of Electrical and Electronics Engineers) Xplore Digital Library, ISI Web of Knowledge, ProQuest, Science Direct and Scopus. For each database a specific string was constructed to accommodate to different database search specifications (string size, search processes e.g. such as using lemmatization functions or not, fields available to search, use of different quotation marks …). Each string used included keywords from the three selected thematic fields: “video game”, “creation” and “student”. Below we exemplify by showing the search string used in ISI Web of Knowledge, where we searched within Topic Subject (TS), which includes Title, Abstract and Keywords: TS=("computer game" OR "computer games" OR "video game" OR "video games" OR "serious game" OR "serious games" OR "simulation game" OR "simulation games" OR "online game" OR "online games" OR "digital game" OR "digital games" OR "virtual game" OR "virtual games" OR "educational game" OR "educational games" OR MMOG OR MMO OR MMORPG OR RPG) AND TS=(Create OR creation OR creating OR created OR creator OR design OR designing OR designed OR designer OR game-design OR game-designer OR construct OR construction OR constructing OR constructed OR constructor OR develop OR development OR developing OR developed OR developer OR build OR building OR built OR builder OR produce OR production OR producing OR produced OR producer OR author OR authorship OR authoring OR authored OR conceptualize OR conceptualise OR conception OR conceptualizing OR conceptualising OR conceptualized OR conceptualised OR constructionism OR constructionist) AND TS=(student OR pupil OR scholar OR learner OR undergraduate OR postgraduate OR fresher OR tutee OR trainee OR apprentice OR schoolchild OR schoolboy OR schoolgirl OR kindergartener OR "high schooler" OR "high schoolers" OR middler OR "middle schooler" OR "middle schoolers" OR freshman OR sophomore OR underclassman OR undergrad OR upperclassman OR graduate) The other search parameters used in this database were “Idiom = English”, “Type of documents = Article, Abstract of published item, Proceedings paper”, “Indexes=SCI-EXPANDED, SSCI, A&HCI, CPCI-S, CPCI-SSH, ESCI, CCR-EXPANDED, IC”, “Time frame=All years”. After searching each database, the results obtained were saved with the information of reference, title and abstract. For most of the databases the search results were saved in RIS format; in ERIC the format was NBIB File, text document in ACM and ISI Common Export Format in ISI Web of Knowledge. EndNote X7.5 was used to create a library with all the search results from all the databases used. 2.3 Screening titles and abstracts (Step 3) EndNote X7.5 was also used to remove duplicate entries, both within the results of each database and between databases. Duplicates were found, at first, using EndNote embedded options by searching references with exact matches in the fields Author, Title, Year. The highlighted results were screened and we choose to maintain the records which had more complete information, eliminating the remaining duplicate records. After that initial triage, we noted that there were still some duplicate records so we used again EndNote embedded options by searching references with the same Title, and this time not selecting the exact match option (making it non case-sensitive). Again the highlighted results were read and triaged, eliminating the records with less complete information. After removal of duplicate entries, all records, by year of publication, were exported from EndNote to Microsoft Excel Office 2013 (a file for each set of records from the same year), by creating a customized Endnote Export Output Style containing only the desired information (Record Number, Year, Author, Title, Abstract and Reference Type) and saving it as an Extensible Mark-up Language (XML) file, that was then imported into Microsoft Excel Office 2013. Each record was carefully read to decide on its inclusion or exclusion in the systematic review, based on the title and abstract. Due to the large number of results to screen, records were divided between the first and second author to be evaluated with regards to the inclusion criteria. Records that raised doubts were discussed between the authors until there was a consensus decision. All records selected for inclusion by each author were then reviewed by the other authors to evaluate if there was agreement. 9th International Conference of Education, Research and Innovation, 2016 It is important to stress that systematic review protocols are iterative in their nature. Regarding the study at hand, it is of relevance to note that only during this step did the researchers decide to exclude students from educational stages other than K12, and that is why in the previous step (step 2 – literature searching) we included words like kindergartener, undergraduate, postgraduate, fresher, freshman, sophomore, underclassman, undergrad, upperclassman or graduate, which became obsolete once we opted to limit this systematic review to K12 students. 2.4 Obtaining papers (Step 4) Papers in Portable Document Format (PDF) were obtained via online search through Google and Google Scholar using the institutional credentials of University of Minho (Universidade do Minho, Portugal). Papers that could not be freely obtained that way, whenever existing on Academia.edu (https://www.academia.edu/) or ResearchGate (https://www.researchgate.net/), were kindly asked for access, i.e., to be made available by the authors via those websites. Whenever that was not the case a search for the authors’ contacts was made in order to directly ask the authors permission to access fulltext. All Portable Document Format (PDF) files were saved in a Google Drive folder shared by the authors of the study, identified with the corresponding Record Number, Year, Author and Title. 2.5 Selecting full-text papers (Step 5) Full-text papers were then read in order to confirm its inclusion in the study. While being read to assess inclusion in the systematic review, each selected study was also initially assessed for quality (see step 6) and the researchers started open coding as well, as a way to assist the quality assessment process, and also to save time in a subsequent process of content analysis (see step 7). 2.6 Quality assessment (Step 6) Quality assessment was carried out using an adaptation of both Critical Appraisal Skills Programme (CASP) [9] and Dyba [10] quality assessment instruments. The eleven criteria considered relevant for the quality assessment, grouped in four dimensions (Minimum quality threshold, Rigor, Credibility and Relevance), guided the quality evaluation. Each reference that had passed the previous 5 steps was at this stage assessed for each criterion of the CASP/Dyba instrument [9] [10]. Each one of the first two authors would independently answer the quality assessment questions for each study and afterwards the responses would be cross-checked. Whenever there were doubts or discordance, consensus meetings took place. 2.7 Data extraction (Step 7) As stated before, during quality assessment, we performed initial coding based on the study characteristics to assist the process but also to better understand trends for future data extraction. Both data coding and data extraction were carried out using MAXQDA 12. While creating code categories we used and adapted a coding proposal by Connolly, Boyle, MacArthur and Hainey [11], which focuses on three main categories: 1. Categorisation of video games; 2. Categorisation of effects of video games, which we herein adapt to Behavioural and learning outcomes and impacts of video game making and 3. Coding of methods. Connolly et al. [11] propose the following subcategories within the first main category: Primary purpose of the game, Game genre, Subject discipline and Platform/delivery. For the second main category, the authors suggest dividing it in: Knowledge acquisition/content understanding, Perceptual and cognitive skills, Motor skills, Physiological outcomes, Affective and motivational outcomes, Behaviour change, Soft skills and Social outcomes. The third main category contain the following subcategories: Study design, Sampling, Sample, Between-group comparisons, Data collection, Data analysis, Results and Conclusions. Connolly et al. [11] was used as an initial reference, not as a closed category index. There were emergent additional categories such as Research question, Objectives, Context, Participants, Duration, Software used, Domain/Content, Intervention, Ethical considerations, Researcher role, Relevance, Limitations and Future research suggestions. 9th International Conference of Education, Research and Innovation, 2016 2.8 Analysis and synthesis (Step 8) Collected data will be scrutinized and synthesized, either narratively or through meta-analysis, depending on the data obtained through coding and extraction, which is still ongoing. 2.9 Writing up and editing (Step 9) We will use the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [12] to support writing up the upcoming paper once the study is finished, presenting background, methods and results, discussing findings and drawing conclusions from this systematic review. 3. INITIAL RESULTS It is important to note that the focus of this paper is to present a protocol to conduct a systematic review of the literature to address the question “What is the reported research concerning video game creation by students?” Since the systematic review study is still ongoing we will only present here a fragment of the initial results obtained. After performing step 2 of the systematic review protocol (Literature searching), we obtained 15.237 references. Since we performed an exhaustive search in several databases to reduce the possibility of missing relevant results, we obtained almost a third of duplicate results. In order to have a perception of the overlap of results between the different databases / search portals, see Table 2. Table 2 - Duplicate results between each two databases / search portals used Once all the duplicates were removed, 10.297 references remained for analysis. More than a third of those, 5.202, are references with publication dates from 2011 to 2015. For the purpose of this paper we will focus on that segment. After reading all titles and abstracts of the papers published between 2011 and 2015 (step 3 of the protocol) only 64 references passed all inclusion criteria and remained for analysis. When performing step 4 (obtaining papers), six papers could not be freely acquired in time and thus were excluded at this phase (will be considered once obtained). 25 studies passed steps 5 (selecting full-text papers) and 6 (performing quality assessment). Even though step 7 and onwards of the protocol are still in progress, we present the list of studies selected so far for content analysis (published between 2011 and 2015) and the corresponding research questions or objectives in Table 3. Table 3 - studies selected for content analysis (published between 2011 and 2015) 9th International Conference of Education, Research and Innovation, 2016 9th International Conference of Education, Research and Innovation, 2016 4. DISCUSSION AND FUTURE WORK Though conducting a systematic review of the literature can be a difficult and time consuming activity [7], we believe that due to its power of analysis, synthesis and objectivity, it should be more widely used in Educational Research. In this section, we will briefly outline some of the difficulties encountered so far and what still lies ahead concerning the present study. From our point of view steps 1 and 2 of the proposed protocol are the most overwhelming. It takes a great amount of time, in an iterative process, to get to know how the different databases work and how to construct an adequate search string. From the experience we had while conducting this study, databases are not transparent as to how they operate, where they search, what they search, and how they search it, creating black boxes right at the beginning of a process that is wanted and needed as clear and reproducible as possible. Many days are spent trying to understand how each database works by experimentation, many times by trial and error (the only way we were able to find out that not all quotation mark symbols are accepted in some databases). To this adds the fact that it is not possible to use the same search string with every database, due to their different specificities. On another regard, we found it very difficult to narrow the search string in order to better exclude irrelevant results and reduce the quantity of references to screen during step 3. This has to do mainly with the fact that the thematic area where this systematic review is inserted, video games and education, is vast and multidisciplinary. Due to the enormous number of references to process during step 3, more than 10.000 records, we had to divide the work amongst researchers thus not being able to calculate the coefficient of agreement between the researchers’ evaluations. We intend to mitigate this issue by selecting a representative sample of single evaluations from each researcher and give them to the other researchers to blindly cross-check them. As noted before this is a study in progress. We still have to pass all references from all publication years through the several steps of the protocol herein depicted, in order to be able to respond to our research question and attain the objectives we set out to accomplish. We believe that the final results of a thorough systematic review like the one here proposed will provide valuable contributions to the research area of video games and education, in a narrow but ever growing topic, video game creation, that will be significant to inform and guide teachers, researchers, decisionmakers and future research initiatives. ACKNOWLEDGEMENTS The work of the first author is funded by the Portuguese national funding agency for science, research and technology (Fundação para a Ciência e Tecnologia, IP - FCT) with a research scholarship (PD/BD/127783/2016) under the PhD Programme in Technology Enhanced Learning and Societal Challenges. The authors would like to thank their colleague Sandro Monteiro for his advice and expertise with regards to protocols for systematic reviews of the literature. REFERENCES [1] Gee, J. (2003). What video games have to teach us about learning and literacy. Computers in Entertainment (CIE), 1(1), 20-20. [2] Egenfeldt-Nielsen, S. (2006) Overview of research on the educational use of video games. Digital Kompetanse, 3 (1), pp. 184-213. [3] Perrotta, C., Featherstone, G., Aston, H. and Houghton, E. (2013) Game-based Learning: Latest Evidence and Future Directions. NFER Research Programme: Innovation in Education. Slough: NFER. [4] Papert, S. (1971). Teaching Children Thinking. Programmed Learning and Educational Technology, 9(5), 245–255. [5] Resnick, M., Rusk, N., and Cooke, S. (1998). The Computer Clubhouse: Technological Fluency in the Inner City. In Schon, D., Sanyal, B., and Mitchell, W. (eds.), High Technology and LowIncome Communities, pp. 266-286. Cambridge: MIT Press. 9th International Conference of Education, Research and Innovation, 2016 [6] Petticrew, M., Roberts, H. (2006). Systematic Reviews in the Social Sciences: A Practical Guide. Cebma.Org. [7] Boland, A., Cherry, M. G., Dickson, R. (eds.). (2013). Doing a systematic review: A student's guide. Sage. [8] Costa, A. B., Zoltowiski, A. P. C. (2014) Como escrever um artigo de revisão sistemática In: Koller, S.H., Couto, M. C. P., Hohendorff, J. V. (eds.). Manual de Produção Científica. Porto Alegre: Penso. [9] Dybå, T., Dingsøyr, T. (2008). Empirical studies of agile software development: A systematic review. Information and software technology,50(9), 833-859. [10] CASP - Critical Appraisal Skills Programme. 10 questions to help you make sense of qualitative research (2013). Retrieved from http://media.wix.com/ugd/dded87_29c5b002d99342f788c6ac670e49f274.pdf. Accessed in: 10 Aug. 2016. [11] Connolly, T., Boyle, E., Macarthur, E., Hainey E T., Boyle, J. (2012). A systematic literature review of empirical evidence on computer games and serious games. Computers & Education, Vol. 59, pp 661–686. [12] Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., The PRISMA Group (2009). Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med 6(7): e1000097. [13] Akcaoglu, M. (2014). Learning problem-solving through making games at the game design and learning summer program. Educational Technology Research and Development, 62(5), 583– 600. [14] Akcaoglu, M., Koehler, M. J. (2014). Cognitive outcomes from the Game-Design and Learning (GDL) after-school program. Computers and Education, 75, 72–81. [15] Akpinar, Y., Aslan, M. (2015). Supporting Children’s Learning of Probability through Video Game Programming. Journal of Educational Computing Research, 53(2), 228–259. [16] Baytak, A., Land, S. M. (2011). An investigation of the artifacts and process of constructing computers games about environmental science in a fifth grade classroom, 765–782. [17] Baytak, A., Land, S. M., Smith, B. K. (2011). Children as educational computer game designers: An Exploratory study. Turkish Online Journal of Educational Technology, 10(4), 84–92. [18] Chu, H. C., Hung, C. M. (2015). Effects of the digital game-development approach on elementary school students’ learning motivation, problem solving, and learning achievement. International Journal of Distance Education Technologies, 13(1), 87–102. [19] Reis, T., Silva, D., Henrique, E., Aranha, S. (2015). Online Game-based Programming Learning for High School Students – a Case Study. [20] Games, A., Michigan S. U., Kane, L. (Michigan S. U. (2011). Exploring adolescent’s STEM learning through scaffolded game design. Proceedings of the 6th International Conference on Foundations of Digital Games, 1–8. [21] Garneli, V., Giannakos, M. N., Chorianopoulos, K., Jaccheri, L. (2015). Serious Game Development as a Creative Learning Experience: Lessons Learnt. 2015 IEEE/ACM 4th International Workshop on Games and Software Engineering, 36–42. [22] Hwang, G. J., Hung, C. M., Chen, N. S. (2014). Improving learning achievements, motivations and problem-solving skills through a peer assessment-based game development approach. Educational Technology Research and Development, 62(2), 129-145. [23] Hwang, R., Wu, J., Tsai, Z., Yu, P., Lai, C. (2015). Enhancing the programming skill in high school engineering education via flipped classroom and peer assessment. [24] Kafai, Y. B., Vasudevan, V. (2015). Constructionist gaming beyond the screen: Middle school students’ crafting and computing of touchpads, board games, and controllers. Proceedings of the 2015 Workshop in Primary and Secondary Computing Education, 49–54. 9th International Conference of Education, Research and Innovation, 2016 [55] Kafai, Y., Vasudevan, V. (2015). Hi-Lo tech games: Crafting, Coding and Collaboration of Augmented Board Games by High School Youth. Proceedings of the 14th International Conference on Interaction Design and Children, 130–139. [26] Ke, F. (2014). Computers & Education An implementation of design-based learning through creating educational computer games : A case study on mathematics learning during design and computing. Computers & Education, 73, 26–39. [27] Ke, F., Im, T. (2014). A case study on collective cognition and operation in team-based computer game design by middle-school children. International Journal of Technology and Design Education, 24(2), 187–201. [28] Koutsikos, L., Holeva, V., Zourelidis, S., Dova, M., Patrikakis, C. (2012). Information and Communication Technology in Greek Primary Schools : A Pilot Application. Proceedings of the European Conference on Games Based Learning, 262–268. [29] Lamb, R., Annetta, L., Vallett, D. (2015). The interface of creativity, fluency, lateral thinking, and technology while designing Serious Educational Games in a science classroom. Electronic Journal of Research in Educational Psychology, 13(2), 219–242. [30] Ouahbi, I., Kaddari, F., Elachqar, A. (2015). Learning Basic Programming Concepts by Creating Games with Scratch Programming Environment, 191, 1479–1482. [31] Pinto, A., Escudeiro, P. (2014, June). The use of Scratch for the development of 21st century learning skills in ICT. In 2014 9th Iberian Conference on Information Systems and Technologies (CISTI) (pp. 1-4). IEEE. [32] Repenning, A. et al. (2015). Scalable Game Design : A strategy to bring systemic Computer Science Education to schools through game design and simulation creation. ACM Transactions on Computing Education, 15(2), 1–31. [33] Siko, J. P. (2013). Are They Climbing the Pyramid? Rating Student-Generated Questions in a Game Design Project. Canadian Journal of Learning and Technology, 39(1), n1. [34] Siko, J. P., Barbour, M. K. (2015). The sum is greater than the parts: deconstructing homemade PowerPoint games. Computers in the Schools,32(3-4), 167-182. [35] Sim, G., Read, J. C., Gregory, P., Xu, D. (2015). From England to Uganda: Children Designing and Evaluating Serious Games. Human–Computer Interaction, 30(3-4), 263–293. [36] Werner, L., Denner, J., Campe, S. (2015). Children programming games: a strategy for measuring computational learning. ACM Transactions on Computing Education (TOCE), 14(4), 24. [37] Wilson, A., Hainey, T., Connolly, T. (2012). Evaluation of Computer Games Developed by Primary School Children to Gauge Understanding of Programming Concepts. Proceedings of the European Conference on Games Based Learning, 549–558.