Assessing 4th Grade Students’ Computational Thinking through Scratch Programming Projects
Full text
This is a self-archived version of an original article. This version may differ from the original in pagination and typographic details. Author(s): Title: Year: Version: Copyright: Rights: Rights url: Please cite the original version: CC BY 4.0 https://creativecommons.org/licenses/by/4.0/ Assessing 4th Grade Students’ Computational Thinking through Scratch Programming Projects © 2020 Vilnius University, ETH Zürich Published version Fagerlund, Janne; Häkkinen, Päivi; Vesisenaho, Mikko; Viiri, Jouni Fagerlund, J., Häkkinen, P., Vesisenaho, M., & Viiri, J. (2020). Assessing 4th Grade Students’ Computational Thinking through Scratch Programming Projects. Informatics in Education, 19(4), 611-640. https://doi.org/10.15388/infedu.2020.27 2020
Informatics in Education, 2020, Vol. 19, No. 4, 611–640 © 2020 Vilnius University, ETH Zürich DOI: 10.15388/infedu.2020.27 611 Assessing 4th Grade Students’ Computational Thinking through Scratch Programming Projects Janne FAGERLUND1*, Päivi HÄKKINEN2, Mikko VESISENAHO1, Jouni VIIRI1 1Department of Teacher Education, University of Jyväskylä, Jyväskylä, Finland 2Finnish Institute for Educational Research, University of Jyväskylä, Jyväskylä, Finland e-mail: {janne.fagerlund, paivi.m.hakkinen, mikko.vesisenaho, jouni.p.t.viiri}@jyu.fi Received: June 2020 Abstract. Computational thinking (CT) has been introduced in primary schools worldwide. However, rich classroom-based evidence and research on how to assess and support students’ CT through programming are particularly scarce. This empirical study investigates 4th grade students’ (N=57)CTinacomparativelycomprehensiveandfine-grainedmannerbyassessingtheirScratch projects (N = 325) with a framework that was revised from previous studies to aim towards enhancing CT. The results demonstrate in detail the various coding patterns and code constructs the students programmed in assorted projects throughout a programming course and the extent to which they had conceptual encounters with CT. Notably, the projects indicated CT diversely, and the students altogether encountered dissimilar areas in CT. To target the acquisition of CT broadly, manifold programming activities are necessary to introduce in the classroom. Furthermore, we discuss the possibilities of applying the assessment framework employed herein to support CT education through Scratch in classrooms. Keywords: computational thinking; Scratch; assessment; primary school; education. Introduction Computational thinking (CT) has been increasingly incorporated in primary schools across the world often by means of graphical programming (Bocconi et al., 2018; Mannila et al., 2014). Using Scratch to design interactive games, animations, and stories that are thematically connected to different curricular areas is especially popular among the age group (Garneli et al., 2015; Moreno-León et al., 2017). Being a new topic in primary education, however, CT involves areas that necessitate research-based pedagogical knowledge. In particular, it is vital to more intricately spotlight what CT students * Corresponding author. Postal address: Ruusupuisto, P.O. Box 35, 40014 University of Jyväskylä, Finland. e-mail:[email protected]Tel.+358408054711
J. Fagerlund et al. 612 can gain in various ways and how their CT learning could be pedagogically supported while programming (Lye and Koh, 2014; Shute et al., 2017). On a related note, although previous literature (e.g., Barr and Stephenson, 2011) describes what kinds of skills and knowledge CT involves, the term still has no universally accepted depiction. Adopting a relatively extensive view of CT (as opposed to an overly concise one merely embodying tool-specificprogrammingskills)enablesinvestigatingskillsthatarepotentiallytransferable across problem-solving domains. To that end, this study investigates primary school students’ CT deeply based on Scratch projects they programmed in naturalistic classroom situations. In the process, we also aim to develop ways to support students’ learning of CT in this context. These aspirations led us to “assessment for learning”, in particular, formative assessment, which can support students’ learning performance and their beliefs about their own capabilities (Black and Wiliam, 1998; 2009). When outlining CT rather extensively, though, various types of programming contents in Scratch can indicate CT (Seiter and Foreman, 2013). Our review of relevant studies shows that although several assessments of CT in Scratch projects exist, they mostly cover contents that indicate partial areas in CT, such as its certain core concepts or principles in particular learning scenarios. The objective of this study is to gain rich empirical insight of 4th grade students’ CT by assessing Scratch projects that they designed during a programming course. In order to assess the students’ CT extensively through their Scratch projects and set a stage to facilitateknownlearningbenefitsinformativeassessmentintheclassroominthefuture, we were encouraged to build on existing works to revise an especially profound assessment framework. This article reports on the preparatory use of the framework in assessing programming contents and indicative CT in the students’ different kinds of Scratch projectsinacomparativelycomprehensiveandfine-grainedmanner.Bycomprehensiveness, we refer to the wide-ranging categorization embodying what students can learn in CTandhowmanifoldprogrammingcontentsindicateCT.Byfinegranularity,werefer to the way of systematically analyzing small-scale programmatic evidence in Scratch projects for advantages in research and learning-support alike. We evaluate and discuss thesignificanceoftheattainedevidenceinCTeducationandthenextstepsofdeveloping formative assessment of CT in schools. Assessing CT in Scratch Projects Positioning CT in Primary Education The term computational thinking (CT) was popularized by Jeannette Wing (2006; 2011) as “the thought processes involved in formulating a problem and expressing its solution(s) in such a way that a computer – human or machine – can effectively carry out.” CT provides competencies for adapting to the digitalized world and solving problems across disciplines by applying computational tools, models, and ideas (Denning
Assessing 4th Grade Students’ Computational Thinking through Scratch ... 613 and Tedre, 2019). Broadly viewed as a competence that, apart from mere programming skills, involves broader concepts and practices, such as algorithms, data, and problem decomposition, which describe manifold skills and areas of understanding that are expected to transfer to different problem-solving domains (Angeli et al., 2016; Barr and Stephenson, 2011; Csizmadia et al., 2015; Grover and Pea, 2018; Hsu et al., 2018; Shute et al., 2017). Amid continuing efforts to conclusively capsulize the exact nature of CT, wedefinethekindsofskillsandknowledgeitcaninvolveinarelativelyinclusiverather than overly condensed manner. In several countries, CT or its proximal topics (e.g., informatics, computer science, programming) are integrated in the learning of different curricular areas (Heintz et al., 2016). At the primary school level, block-based programming has been an especially popular way to promote CT (Grover and Pea, 2013). Meaningful learning can occur in multidisciplinary project-based settings in which students have autonomy regarding, for instance, how they learn (Lonka, 2018). Scratch has been often used in teaching and learning practice, for example, through creative game design, storytelling, or animation while the substance of other curricular areas is being processed (Garneli et al., 2015; Moreno-León et al., 2017). The focus of this study is in assessment for learning, which aims to promote learning rather than merely rank or certify it. In particular, the processes associated with formative assessment can support classroom learning (Black and Wiliam, 2009). Programming is cognitively complex, and support for learning CT through programming is vital for making learning more effective (Lye and Koh, 2014). However, CT involves several aspects, and it should be assessed from several entry points (Grover et al., 2017). Earlier research has indicated that assessing students’ Scratch projects is among essential entry points because programmed projects are rich, concrete, and contextualized approximations of the students’ conceptual encounters with CT (Brennan and Resnick, 2012; Román-González et al., 2019; Seiter and Foreman, 2013). Therefore, whilst adopting a comparatively inclusive view of CT, in the following sections we review previous studies on the assessment of students’ Scratch projects from the viewpoint three formative assessment processes: what to teach and learn (i.e., clarifying learning objectives), estimating students’ current level of understanding, and providing relevant feedback (Black and Wiliam, 2009). Assessment for Learning in Scratch What to Teach and Learn CT can be concretized in programming through core educational principles, such as “algorithm control structures”, “parallel execution”, and “Boolean logic”. These principles can be manifested in Scratch in at least three categories: code constructs, coding patterns, and other programming contents. (Fagerlund et al., 2020.) Code constructs,akintolanguageprimitives,suchasthoseforcontrollingtheflow in programs (i.e., “sequence”, “conditional”, “loop”) can be observed directly as Scratch blocks. For instance, Moreno-León et al. (2015) used Dr. Scratch, an automated analysis tool, to examine the presence of sequences of blocks, “repeat” blocks and “if” blocks in
J. Fagerlund et al. 614 Scratch projects. Other studies (Basu, 2019; Franklin et al., 2013; 2017; Maloney et al., 2008; Meerbaum-Salant et al., 2013) additionally examined projects manually for certain blocks representing constructs, such as “initialization” and “coordination.” Additionally, akin to classes, projects can comprise coding patterns: semantically meaningfulcombinationsortemplatesofconstructsthatachievespecificfunctionalities (e.g., animation of size). For instance, the Progression for Early Computational Thinking (PECT) model assesses project-wide patterns, such as “Maintain score” and “User interaction”, which incorporate different types of templates that can be programmed by combiningspecificcodeconstructsinspecificways(SeiterandForeman,2013).Franklin et al. (2013; 2017), Ota et al. (2016), and Seiter (2015) examined similar patterns to those in PECT, such as “Count up score” and “Multi-sprite synchronization.” Concerning the more social elements of programming and metaprogrammatic elements, projects can also contain other programming contents, such as “project use instructions” and the “appropriate naming of sprites” (Basu, 2019; Funke et al., 2017; Wilson et al., 2012). Manipulating such contents in Scratch simultaneously fosters and demonstrates CT (Brennan and Resnick, 2012). Learning goals and intentions (Black and Wiliam, 2009) regardingstudents’skillsandareasofunderstandinginCTcanthusbeclarifiedindirectly as meaningful and reasonably demanding contents for students to creatively design in projects. Aggregating the various contents distributed among previous studies and assessment frameworks can establish systematic and comprehensive (albeit not necessarily all-encompassing) coverage of CT-fostering programming contents in Scratch. Estimating Current Level of Understanding Students’ learning can be enhanced by guiding them to perceive a gap between set learning goals and their own present skills or understanding. Information regarding this gap can be generated by students, peers, or instructors. (Black and Wiliam, 1998.) In CT, evidence that points towards students’ skills and understanding can be elicited by examining what they have programmed in their projects (Grover and Pea, 2013; RománGonzález et al., 2019; Seiter and Foreman, 2013). More tangibly, programmed contents in projects indicate conceptual encounters with CT’s core educational principles. However, it is crucial that learning tasks generate and display relevant evidence of learning (Black and Wiliam, 1998). Evincing CT through programmed projects can be risky for two reasons. First, static artifacts are not direct measurements of thinking. In particular, blockbased programming environments can present validity concerns, as students can drag and drop blocks without knowing what they are doing (Lye and Koh, 2014). Hence, observational methods should involve rigorously considering the circumstance of the evidence. Analyzing contents primarily through coding patterns improves validity by ensuring that the implemented blocks achieve semantically meaningful computational models (Seiter and Foreman, 2013). Second, Scratch enables various project design opportunities: projects can be designed in different genres, such as animation, game, and story (Maloney et al., 2010),
Assessing 4th Grade Students’ Computational Thinking through Scratch ... 615 which typically contain different programmatic characteristics and, respectively, indications for CT (Moreno-León et al., 2017). Programming can also be effectuated through activities such as remixing or debugging preexisting contents or designing something new (Lee et al., 2011). Moreover, individual experiences may vary when students are activated as instructional resources for each other, for instance, in pair programming. Suchcontextualfactorsinfluencingwhatconstitutesrelevantevidenceoflearningmay increase the more students are activated in owning their own learning. (Black and Wiliam, 2009.) In summary, to advocate the relevance of the examined contents, it is preferable to consider the context of the contents (e.g., recognizing the learning assignment) andinterprettheirsemanticalsignificanceratherthanmerelytechnicalone. Providing Feedback After a gap between learning goals and the students’ current knowledge has been highlighted, the students are guided to take action to close that gap (Black and Wiliam, 1998). While students program, meaningful and authentic feedback can be provided with respect to the contents in their projects. The feedback should facilitate the correctionofspecificerrorsorpoorstrategieswithsuggestionsonhowtoimprovethework (e.g., by providing scaffolding), based on individual progress toward achieving goals (BlackandWiliam,1998).Inthisrespect,alternativelytoevincingthemostproficient segments in students’ projects, assigning scores, and providing generic feedback (see Moreno-León et al., 2015; Seiter and Foreman, 2013), project contents can be examined “micro-programmatically” (e.g., Vihavainen et al., 2013). In Scratch, the scripts of a project can comprise individually instantiated coding patterns and their underlying code constructs. For instance, sprites’ properties, such as location or size, are animated distinctly from one another (see Franklin et al., 2013, 2017; Meerbaum-Salant et al.,2013).Suchfine-grainedevidence,thatis,specificmicroprogrammaticprojectparts,infact,benefitsallthreeformativeassessmentprocesses: operating as meaningful learning goals for students to program in projects, semantically meaningful evidence of their understanding, and meaningful targets for feedback. However, examining instantiated coding patterns is an overlooked analytical approach when applied to CT-fostering programming contents comprehensively and systematically. The Current Study The purpose of this study was to gain rich empirical insight of 4th grade students’ (N = 57) CT by assessing the programming contents in Scratch projects (N = 325) that they designed during a programming course. Prompted by the means established above to attain rich evidence of students’ CT through their Scratch projects along with setting a stage to facilitate formative assessment in the future, we were encouraged to revise an assessment framework based on previous studies. The research questions (RQs) are as follows: What programming contents did the students’ Scratch projects contain?(1) What core educational principles in CT did the students conceptually encounter?(2)
J. Fagerlund et al. 616 Methods Research Design This empirical study had an embedded single-case design. The studied case was an introductory programming course organized for primary school students, and the units of analysis were the Scratch projects that the students made during the course. This study intended to immerse in the particular case deeply rather than acquire data to generalize or represent the population for widespread decision-making. Previous studies have assessed students’ Scratch projects in this age group and in compulsory education (e.g., Funke et al., 2017). However, this study adopted novel theoretical and analytical approaches that contributed in uncovering in-depth knowledge. This knowledge, attained from one situational context, was intended for wider comparison, creation of theoretical models, stimulation of hypotheses for experimentation, and further methodological development. On that account, the employed research method was that of a descriptive case study. Description relied on the theoretical premises regarding the assessment of CT in Scratch projects, as outlined in the previous sections. The main theoretical concepts and developed framework were determined through a thorough literature review to reinforce external and code construct validity. (Yin, 2012.) Participants The students of three 4th grade classes from an average-sized Finnish municipal primary school participated in this study. The classes were selected because the students were surveyed as generally inexperienced at programming. Also, a prequestionnaire confirmedthatthestudentswerelargelynovicesatprogramming,apartfromafewprevious programming experiences. The classes comprised 22, 21, and 26 students, from which 57 of them (62% girls and 38% boys) had informed consent provided by their legal guardians. The students were between 10 and 11 years old during data collection. Two of the participants were nonnative Finnish speakers. The classes also included students with special needs who participated in the programming activities but chose not to participate in data collection. Data Collection The data collected in this study was the Scratch projects the students programmed during a programming course that followed general guidelines in the Finnish primary school core curriculum. Each class attended the course separately one lesson per week for 4 months (13 lessons in total) in early 2017. The course was piloted with one class in another school to estimate and develop the employed pedagogical methods and data collection methods. The lessons were conducted mainly in the school’s computer lab,
Assessing 4th Grade Students’ Computational Thinking through Scratch ... 617 which had 15 functional computers. The teachers grouped the students (1-3 students per group) at the start of the course based on perceived shared skill levels or similar interestareas.Thefirstauthorwastheprimaryinstructorofthecourseduetotheregular teachers’ lack of experience in programming education. The regular teacher of each class was always present, and a research assistant and learning assistant for special needs students were present during most of the lessons. All the teachers participated in guiding the students’ work. The course design was inspired by previous studies (Grover et al., 2014; MeerbaumSalant et al., 2013). As the students were relatively young and new to programming, the main objective of the course was to introduce fundamental Scratch features and CT through perceivably introductory programming contents and activities to the students. The course began by discussing applications of programming in the world and “unplugged”exercisesoveronelesson.Subsequently,lesson-specificlearninggoalswere targeted by programming Scratch projects (see below), which included selections from the Creative Computing guide (Brennan et al., 2014). The student groups programmed different kinds of projects during the course (Table 1). “Tutorial”, “Debugging”, and “Remix” projects involved preset objectives that guidedtowarddesigning,remixing,ordebuggingspecificcontents.Typically,theselessons began with a teacher-led demonstration of a feature (e.g., sprites sprint-racing) or an incomplete program that required implementing or error-correcting particular contents (e.g., “event-sync” code construct as the opening shot). Subsequently, the students were guided to follow the tutorial or remix and complete and creatively extend their Table 1 Projects the students programmed during the course used as data Project Name Type Objective Key contents N P1 “Scratch surprise” Design Create and modify sprites and scripts with blocks. Scratch GUI (e.g., logging in, using blocks); experimenting 33 P2 “Cat dance” Tutorial Program a dance performance. Scripting, iteration, “sequence”, “event” 28 P3 “10 blocks” Design Plan and program your own series of instructions. Planning, animating, “wait”, “loop” 22 P4 “Debugging”, part 1 Debug Debug up to four faulty programs. Code-reading, debugging 64 P5 “Dinosaur race” Remix Remixafaultyprogramandfixan animation. Remixing, “initialization”, “event-sync”, “parallelism” 26 P6 “Riddler game” Design Program a game that asks questions, receives keyboard inputs and checks the correctness of answers. “Variable”, “conditional”, “user interaction” 30 P7 “Debugging”, part 2 Debug Debug up to four faulty programs. Code-reading, debugging 96 P8 Final projects Design Design an interactive game, story, or animation. Planning, creative design 26 Total: 325
J. Fagerlund et al. 618 own project. By contrast, the programmatic requirements of “Design” projects were less rigorously set: the students imagined and programmed projects within certain negotiated boundaries (e.g., a riddler game), having an opportunity to search for ideas from the Internet and, with projects P3 and P8, plan their projects with pen and paper over one lesson. All projects that the students had assigned to provided studios once the course ended were collected as data. Projects made outside the lessons were excluded because they were mainly incomplete drafts. Data Analysis Revising the Rubrics As asserted previously, our priority was to aggregate manifold programming contents indicating CT thoroughly and systematically in Scratch projects. To prioritize gaining especially rich insight on the two essentially interconnected content areas, individually instantiated coding patterns and their underlying code constructs, the examinations of “other programming contents” (see Appendix C) are omitted here. We selected the PECT model’s (Seiter and Foreman, 2013) voluminous rubrics as a baseline for our rubrics (described below). Several revisions to expand and regularize PECT’s rubrics to patterns and constructs and convert its project-wide categorization to an instance-based one were made based on other previous studies, our initial reviews of the students’ projects, and our personal experiences in Scratch as follows. Concerning coding patterns, for instance, “Animate Motion” and “Animate Looks” were merged because sprites’ all properties (e.g., position, size) are animated with the same constructs. “Conversate” was revised into “Speech and Sound” to examine separately programmed conversations using text, sound, or both. “Maintain score”, which originally focused on manipulating score-like integers, was revised into a more general “Data manipulation” to also reveal manipulations of other variables, such as strings (Ericson and McKlin, 2012). Moreover, we added new ways to program the patterns, such as video/audio sensing (see Moreno-León et al., 2015) and extensions (e.g., “Makey Makey”) in “User interaction.” Concerning code constructs, for instance, we renamed “sequencing and looping” to “control” (Moreno-León et al., 2015) and included conditional structures in it (Grover and Pea, 2018). “Parallelism” was split into parallelism “within” and “across” sprites (Meerbaum-Salant et al., 2013). “Initialization” was revised to function correctly on any event if a sprite was hidden until then. “Coordination” with timing was revised to functioncorrectlywithanyblockwithaduration.WealsoaddednewScratch-specific constructs: “pen” (Ericson and McKlin, 2012), “I/O” (Moreno-León et al., 2015), and “make-a-block” (Basu, 2019; Ota et al., 2016). Analyzing Programming Contents In short, the analysis of programming contents in Scratch projects began from examining individually instantiated coding patterns. Each instance was analyzed in terms of
Assessing 4th Grade Students’ Computational Thinking through Scratch ... 625 for question-asking and “if-else” blocks for answer checking, but these blocks were unscripted,renderingthemdysfunctionalandindicatingthattheprojectswereunfinished. All functional answer tests were conditional structures in “Test value” (DM–2). In addition to the instructed requirements, all the projects entailed other instance types (Mdn = 5, Max = 11) exclusively in the “Animation” (total: 55), “Speech and sound” (62), and “User Interaction” (49) patterns. The students had more creative freedom for the other Design projects. These projectsincluded“ScratchSurprise”(P1),thestudents’firstself-designedScratchproject; “10 Blocks” (P3) as exercises in script planning; and interactive games, stories, or animations(P8)asfinalprojectassignments.Thedivisionoffunctionalinstancesin these projects (Table 4) revealed that “Animation” was substantially the most commonly instantiated pattern (49% of all instances), followed by “User Interaction” (25%) and “Speech and sound” (20%). “Data Manipulation” (4%) and “Collision” (2%) were rarely instantiated. The P1 projects (N = 33) typically comprised various blocks as nascent scripts but without events to start them (see examples in Fig. 7). As a result, 58% of all instances were dysfunctional, suggesting that the students did not spontaneously grasp event-driven scripting entirely or merely experimented with different features. The most common functional instance types were “Monologue” (SS–1) (total: 28, in 45% of the projects), “Greenflag”(UI–1)(total:22,in42%),and“Timedanimation”(total:21,in30%). Table 4 The numbers of instantiated coding patterns in the three open-ended design projects that students programmed during the course Coding pattern Instantiated in open-ended design projects Total P1 “Scratch surprise” (N = 33) P3 “10 blocks” (N = 22) P8 Final projects (N = 26) Animation 83 77 417 577 Speech and sound 64 50 127 241 Collision 3 0 36 39 Data Manipulation 17 0 40 57 User Interaction 30 27 200 257 Fig. 7. Sample unscripted blocks from three P1 projects.
J. Fagerlund et al. 626 The P3 projects (N = 22) had typically one or two sprites performing simple behaviors, such as introducing themselves with “Timed animation” (AN–1) (total: 45, in 86% of the projects), “Monologue” (SS–1) (total: 39, in 86%), and “Click/Key press” (UI–1) (total: 15, in 59%) (see Fig. 8). However, in contrast to the P1 projects, only 12% of all instances in these projects were dysfunctional, indicating that the students had begun internalizing the idea behind scripting. Thefinalprojectassignments,theP8projects(N = 26), entailed thematically and programmatically versatile game, animation and story-like projects (see Fig. 9 and Fig. 8. An example P3 project. Fig. 9. The components and scripts in a relatively complex P8 project.
Assessing 4th Grade Students’ Computational Thinking through Scratch ... 627 Fig. 10). 9% of the instances in these projects were dysfunctional, suggesting that the students still struggled with implementing contents or that the projects were not entirelyfinished.Ofthefunctionalinstancetypes,themostcommonwere“Event-sync animation”(AN–5)(total:258,presentin89%oftheprojects),“Greenflag”(UI–1) (total: 113, in 92%), and “Monologue” (SS–1) (total: 57, present in 81% of the projects). Project Portfolios The students’ portfolios (N = 57) contained between 2 to 14 projects (Mdn = 10), suggesting that few students participated in designing only two projects or that all portfolios did not include all programmed projects. Nevertheless, the portfolios demonstrated the varying numbers of instance types that the students programmed during the course (Table 5). The most common types were “Event-sync animation” (AN–5), “Monologue” (SS–1),and“Greenflag”(UI–1).Morethanhalfoftheinstancetypesreceivedamedian of zero, potentially highlighting more advanced contents. Similar statistics were computable for the code constructs as well, but their high number rendered reporting inappropriate; however, the most common constructs were “sequence” (Mdn = 55), “repeat”/”forever” (Mdn = 22), and “green flag” (Mdn = 16). Fig. 10. The components and scripts in a relatively simple P8 project.
J. Fagerlund et al. 628 Table 5 Minimum, maximum, and median numbers of instance types in students’ project portfolios Instance types of coding patterns Numbers in project portfolios Min Max Median Animation (AN) 1–Timed animation 2–Looped animation 3–State-sync animation (repeat until) 4–State-sync animation (wait until) 5–Event-sync animation 0 0 0 0 0 32 20 11 11 146 4 6 0 0 9 Speech and sound (SS) 1–Monologue 2–Time-sync dialogue 3–State-sync (repeat until) dialogue 4–State-sync (wait until) dialogue 5–Event-sync dialogue 2 0 0 0 0 18 6 0 7 15 7 0 – 0 0 Collision (CO) 1–Test collision separately 2–Test collision in loop 3–Wait for collision 0 0 0 5 5 3 0 1 0 Data manipulation (DM) 1–Use/modify variable 2–Test value 3–Loop until value 4–Wait for value 0 0 0 0 6 5 2 0 0 1 1 – User interaction (UI) 1–Greenflag 2–Click/key press 3–Mouse use 4–Keyboard input 5–Video/audio 6–Extensions 1 0 0 0 0 0 25 20 4 5 0 0 6 2 0 1 – – Missing Code constructs Examining code constructs in the instances revealed that the P1 projects were often missing the “control” and “coordination” constructs (Table 6), which essentially rendered most instances dysfunctional. Although the lack of these constructs decreased greatly in subsequent projects, they were still occasionally missing, indicating recurring difficultiesorunfinishedprojects.“Initialization”remainedasafrequentlymissingconstruct throughout the course. Students’ Conceptual Encounters in CT (RQ2) According to the students’ conceptual encounters in CT, as indicated by the programming contents in their project portfolios (Table 7), all the students re-instantiated the coding patterns and code constructs (Patterns) and decomposed the projects into smaller
Assessing 4th Grade Students’ Computational Thinking through Scratch ... 629 parts (Problem decomposition). Nearly all (>90%) the students designed complex projects (Abstraction), implemented algorithm control structures and “initialization” (Algorithms), remixed (Collaboration), and utilized logical operators (Logic). Table 6 Code constructs missing from coding pattern instances in the students’ projects Code construct Missing from projects P1 P2 P3 P4 P5 P6 P7 P8 “Control” Number 99 8 7 4 2 15 2 30 Percentage 50% 5% 5% 4% 2% 7% 1% 4% “Coordination” Number 113 15 16 6 3 24 6 54 Percentage 57% 10% 10% 7% 3% 11% 3% 7% “Initialization”(*) Number 24 41 61 73 37 33 587 131 Percentage 83% 100% 91% 92% 45% 73% 100% 35% *Only in functional Animation instances. Table 7 Presence of and variation among conceptual encounters with CT’s core educational principles in students’ project portfolios CT concept/practice Core educational principle Indication in project portfolios Presence Coefficientofvariation Abstraction Abstractions of behaviors 34% 277.4 Abstractions of properties 63% 66.3 Abstractions of states 54% 291.2 Algorithms Algorithm control 91% 84.6 Procedures 34% 277.4 Starting from initial state 93% 216.6 Recursion 0% – Automation I/O devices 33% 152.1 Coordination Coordinating scripts 57% 183.8 Synchronizing scripts 18% 437.5 Creativity Modifying remixes 81% 69.8 Data Storing and manipulating data 50% 240.6 Logic Boolean logic 0% – Conditional structures 44% 239.0 Operations 96% 104.5 Modeling and design Algorithm animation 66% 58.6 Patterns Re-instantiated coding patterns/code constructs 100% 23.1 Problem decomposition Decomposition 100% 76.3 Modularized features 63% 153.9
J. Fagerlund et al. 630 However, less than half (<50%) of the students abstracted behaviors for sprites (Abstraction), used procedures (Algorithms), utilized I/O devices (Automation), and synchronized parallel scripts (Coordination). None of the students implemented recursive solutions (Algorithms) or Boolean logic (Logic). Variation was large among instantiating synchronized parallel scripts (Coordination), demonstrating that the few students who encountered this principle did so several times. Discussion CT through programming is a new topic in primary education that necessitates evidencebased pedagogical knowledge, especially regarding assessment that enhances learning (Lye and Koh, 2014). This study assessed 4th grade students’ CT by focusing on their Scratch projects designed in naturalistic classroom situations. We adopted a comparatively inclusive view of what students can learn in CT through Scratch and, by revising a profound assessment framework, focused uniquely on individually instantiated coding patterns and their underlying code constructs, that is, relatively fine-grained evidence.Theframeworkuncoveredampleandmanifoldempiricalfindingsofcontents programmed by the students and respective indications of their conceptual encounters withCT.Next,wediscussthesignificancethatthisevidenceandemployingtheassessment framework may have in teaching and learning CT in Scratch, highlighting also limitations that our analysis poses. Moreover, we address our outlying goal: developing formative assessment systems in schools. Programming Contents Indicating CT Coding Patterns The students implemented instances of “Animation”, “Speech and Sound”, and “User Interaction”byfarthemost,specifyingpreviousfindings(SeiterandForeman,2013) concerning that these patterns are altogether most typically present in students’ projects. These patterns were also exclusively volitionally designed. Concerning conceptual encounters with CT, these contents indicated that the students repeatedly experienced abstracting properties and behaviors (Abstraction), designing procedures (Algorithms), animating algorithms (Modeling and design), manipulating pre-provided data (Data), decomposing projects into coding patterns and code constructs (Problem decomposition), and reinstantiating patterns and constructs (Patterns). These experiences could be expected to occur somewhat naturalistically in Scratch, which is essentially a tool for designing interactive media (Brennan and Resnick, 2012). Perhaps more intriguing and relevant for pedagogical consideration is that, by contrast, “Datamanipulation”and“Collision”wereseldomdesigned.Thisinfluencedthestudents’ conceptual encounters with CT mainly via the relative scarcity of variables, conditionals,andlogicaloperations(specifiedinthefollowingsections),whichcan,however,be considered as fairly fundamental computational concepts (Grover and Pea, 2018). An
Assessing 4th Grade Students’ Computational Thinking through Scratch ... 631 underlying cause may concern the designed types of projects: Moreno-León et al. (2017) showed that the presence of certain constructs typically varies between projects in different genres. We perceived the students’ projects most akin to animations and stories with little interactivity. Therefore, they may have lacked opportunities to explore supplementary genres, such as simulations or more sophisticated games to which Data Manipulation and Collision may be more typical (see Seiter and Foreman, 2013). Facilitating the design ofsuchpresumablymorecomplexprojectscanbejustifiedinadvancedstagesoflearning CT. These patterns were also not systematically introduced during the course, proposing that students may be inclined to volitionally designing familiar contents and that they couldbenefitfromdeliberateguidancetowardsunfamiliarcontents. Instance Types Our systematic categorization of instance types in the coding patterns allowed analyzing the students’ CT in novel detail. The most often instantiated instance types, such as“event-syncanimation”(AN–5),“monologue”(SS–1),and“greenflag”(UI–1)(see Table 5), indicated that the students repeatedly experienced coordinating scripts with timing and events (Coordination), controlling algorithms by sequencing and looping (Algorithms), and modularizing animations and speaker roles (Problem decomposition). The prominence of these experiences may stem from the nature of event-driven programming in Scratch (Maloney al., 2010) and blocks representing code constructs that noviceprogrammerstypicallyfirstlearntouse(seeGroveret al., 2014). Again, perhaps more interesting and allusive in terms of CT pedagogy was that the students sporadically experienced utilizing conditional logic and arithmetic operations (Logic), abstracting program states with continuous events (Abstraction), coordinating scripts with states (Coordination), modularizing data manipulation and collision detection (Problem decomposition), and utilizing key pressing, clicking, and keyboard inputs (Automation). Supplementing prior studies (Burke, 2012; Franklin et al., 2013; Maloney et al., 2008), these findingsspecifiedexactly how studentsimplementuser interactionintheirprojects:inthisstudy,theymainlyimplemented“greenflag”instead of different I/O devices, indicating that the projects typically lacked usability and, consequently, resembled projects more for viewing than playing. As discussed above, the other features, such as logical operations and collision detection, may be more typical to presumably more advanced game-like projects (Moreno-León et al., 2017), proposing a need for educators to purposefully introduce game-like features in Scratch and thus CT more extensively. Despite few examples in prior studies (e.g., Burke, 2012; SáezLópez et al., 2016), how the substance of different curricular topics could be processed while creatively designing usable Scratch projects, such as games and simulations, is not extensively known. The integration of CT in Scratch thoroughly across the curriculum at the primary school level presents a fruitful opportunity for pedagogical planning and further research. Several instance types were instantiated infrequently or never. The students therefore experienced little if any abstracting program states with discrete events (Abstraction), utilizing Boolean logic (Logic), modularizing behaviors with state-sync (Problem decomposition), and utilizing mouse, video/audio, and extensions (Automation). Devices,
J. Fagerlund et al. 632 such as microphones or extensions like Makey Makey, which could have promoted exploring these areas in CT, were not available in the school. As the use of various I/O devices is key in CT, schools could acquire such physical add-ons for learning purposes, and their meaningful use in cross-curricular programming with Scratch could also be diversely considered. Implementing mouse use (UI–3) could also again be more typical to game-like projects, although the students may have also disregarded it because it was not explicitly taught or demonstrated. For the abundance of creative opportunities in Scratch, students could be guided to browse existing projects in the Scratch repository to gain ideas and knowledge of what possibilities exist altogether. In turn, code blocks, such as the “wait until” and Boolean operations, which essentially relate to the other above-mentionedlessencounteredareasofCT,arespecifiedbelow. Code Constructs Several previous studies have examined students’ use of code constructs. However, assessing them within instantianted coding patterns allowed us to gain insight regarding their use in diverse creative circumstances that were, perhaps most importantly, semantically meaningful, thus also favoring the legitimacy of the examination. Among notablefindingswasthatthestudents’firstprojects(P1)comprisedmainlyunscripted blocks and parameter state changes without “coordination”, suggesting that the students did not intrinsically grasp controlling algorithms (Algorithms) and coordinating them with, for instance, timing (Coordination). Control and coordination became greatly more prevalent after the students had completed the scripting tutorial (see Table 6), suggesting that direct instruction can be effective for learning these fundamentals of programming and an effective way to launch especially introductory courses in schools. However,theseconstructswereoccasionallystillmissinginthefinalprojects(P8),possibly exhibiting “bad programming habits” (Moreno-León et al., 2015), situated here under other programming contents (see Appendix C), and highlighting a need to remind students to maintain their use. However, the projects may have been incomplete, suggesting a lack of time and underlining the ever-challenging need for educators to ensure sufficienttimefordesigning. The students typically controlled the programmed instances with “sequences” and “loops” and coordinated them with “timing” and “event-sync” (see Table 5). “Conditional looping” (i.e., the “repeat until” block) and “conditional structures” were rare in control whereas “state-sync”, “blocking”, and “stopping” were rare in coordination. Consequently, the students seldom encountered the different ways to control algorithms (Algorithms) and coordinate automated processes (Coordination). Coordination by stopping and blocking may be somewhat exceptional in Scratch: stopping causes repeatinganimationstohalt,beingrelevantmainlyinprojectsinvolvinginfiniteloopingin “looped animation” (AN–2) (e.g., stopping a sprite from moving forever), and blocking is established with the “ask/set and wait” block, being relevant mainly in projects where “keyboard input” (UI–4) blocks the execution of an “Animation” pattern (e.g., sprite motiontemporarilystoppedtoreceiveaspecificinput).However,becausethesecontents are relevant for CT, an opportunity remains to consider pedagogically meaningful ways to incorporate them in programming tasks.
Assessing 4th Grade Students’ Computational Thinking through Scratch ... 633 Then again, “repeat until” and “wait until”, which represent “state-sync”, are blocks thathavebeennotedtobedifficultforstudentstouse(Basu,2019;SeiterandForeman, 2013). These blocks can be used, for instance, to synchronize collision detection (e.g., CO–2, CO–3) and speaker roles in dialogues (e.g., AN–3, AN–4), highlighting gamelike and story-like projects with colliding and conversing sprites as potentially meaningful – although presumably more advanced – contexts to introduce these constructs. However, most students debugged the “repeat until” block correctly in a structured debugging challenge (P7.4), suggesting that such challenges could offer a viable route between direct instruction and more open-ended design to teach students to understand and use even more advanced constructs. Concurringwith the findingsof Franklin et al. (2013, 2017), “initialization” was missing to varying degrees throughout the course. Most students debugged initialization correctly in P4, however, few students demonstrated avoiding it by programming the sprite to glide back to the starting location. Initialization was explicitly instructed with P5,whichmayhavereflectedonitshighpresence(seeTable6).Nevertheless,wefound that its presence subsequently decreased and varied, suggesting that a conceptual encounter may not self-evidently guarantee gaining a deep understanding. Instead, encountering contents repeatedly over time can be necessary for enhancing understanding and developing more rigorous skills. However, initialization is not mandatory for programs to execute in Scratch, contesting whether Scratch facilitates conceptually encountering it consistently and demonstrating students’ understanding reliably in it. The students manipulated exclusively Scratch variables, resulting in no experiences with abstracting properties as custom variables and lists and manipulating them (Abstraction, Data). Similarly, the lack of arithmetic and Boolean operations revealed that the core educational principles in Logic remained largely unencountered. Moreover, the students rarely implemented “parallelism”, resulting in sparse experiences in synchronization (Coordination). Variables, Boolean operations, and parallelism have been previouslydiscoveredtobesomewhatdifficultforstudentstounderstandanduse(Basu, 2019; Maloney et al., 2008; Meerbaum-Salant et al., 2013; Seiter and Foreman, 2013). In Scratch, parallelism could be introduced meaningfully when synchronizing animations (e.g., AN–4), establishing dialogues (e.g., SS–2), or waiting for sprites to collide (e.g., CO-3) especially in game-like projects. For variables and logical operations, students could design Data Manipulation with comparisons (DM–2, DM–3, or DM–4) in, for instance, a math quiz project. Lastly, the students never used “pen”, which can visualize sprites’ movement paths (Ericson and McKlin, 2012) and, therefore, animate algorithms (Modeling and design). However, algorithm animation occurs naturalistically through most programmed featuresinScratch,contestingthesignificanceofthisconstruct.Moreover,“make-ablocks” were nonexistent, and only one student used “cloning” once. Consequently, the students mainly never abstracted and programmed custom behaviors or clones’ behaviors (Abstraction, Algorithms) or implemented recursive solutions (Algorithms). These constructs have been rarely addressed in prior K–9 studies, suggesting that they, in addition to other contents that were rarely implemented, may better suit more experienced programmers.
J. Fagerlund et al. 634 Implications for Research and Practice Despite the prevalent ideology of interest-driven design and discovery-based learning in Scratch (Brennan and Resnick, 2012), direct instruction and structured debugging could effectively introduce students to fundamental contents and contents which they cannot manage to implement or fail to realize as hidden possibilities. These approaches can be purposeful when students begin to learn scripting in Scratch, become later introduced with such fundamental constructs as “initialization”, and are guided to realize previously unknown creative opportunities, such as mouse use (UI–3). Investigating how the instruction of particular contents (e.g., user interaction) could pave way for students’ constructive less-structured explorations (e.g., moving from greenflagtootherkindsofinteractivity)andhowstudents’interactionswithvarious resources in different tasks could lead to successful content implementations could be pedagogically informative. The model of scope of autonomy recently introduced by Carlborg et al. (2019) could provide a vignette through which to examine such issues. Moreover, our results suggest that a mere conceptual encounter may not assure gaining robust knowledge, and learning through implementing contents requires repetitions. Therefore, we restate known concerns (e.g., Lye & Koh, 2014) providing reason to meticulously examine when and how students gain genuine skills and deep understanding in CT while programming. For learning CT comprehensively, it can be important to design various, presumably more complex kinds of projects, including narratives with several speakers, games with colliding objects and score count, projects with data manipulation, and, altogether, projects that are usable with different I/O devices. Creative contexts in which students could implement such contents, especially the seemingly more advanced ones (e.g., Data Manipulation, Collision, coordination by stopping and blocking, custom variables, Boolean operations), could be adapted from the rubrics in future empirical studies. This could be to examine their feasibility along with considering how to organize compact yet fruitful programmingcoursesinschools.Itseemsespeciallyimportantforpractitionerstofind time to introduce the potentially more complex contents through more complex projects (e.g., games and simulations). The rubrics employed herein may suggest some content organization and the results may suggest the kinds of programming capabilities that students may gain more intrinsically than they do others. However, developing rigid learning trajectories applying, for instance, the Bloom/SOLO taxonomy (e.g., Meerbaum-Salant et al., 2013) for contents would require more studies. In practice, however, itispedagogicallyjustifiabletooffera“highceiling”forstudentstopotentiallyreach (Brennan and Resnick, 2012). Limitations in Analysis Although programmed artefacts are latent manifestations of thinking, evidence to reinforce their validity in analyzing CT has begun to emerge. For instance, analysis by Dr.