Investigating the Impact of Programming Activities on Computational Thinking and AI Literacy in Spanish Schools

Investigating the Impact of Programming Activities on Computational Thinking and AI Literacy in Spanish Schools Jesús Moreno-León Universidad de Sevilla Seville, Spain [email protected] Margarita Vasco-González Universidad Antonio de Nebrija Madrid, Spain [email protected] Marcos Román-González Universidad Nacional de Educación a Distancia Madrid, Spain [email protected] Gregorio Robles Universidad Rey Juan Carlos Fuenlabrada, Spain [email protected] ABSTRACT This article presents research conducted within the School of Computational Thinking and Artificial Intelligence (EPCIA in its original form in Spanish) project, an initiative developed by the Spanish Ministry of Education and Vocational Training during the 2020-21 academic year. The study investigated whether it is possible to contribute to the development of students’ computational thinking skills and artificial intelligence (AI) literacy through programming activities using the Scratch language and the Machine Learning for Kids platform. The study involved 133 teachers and more than 2,000 students, from 4 th to 8 th grade, with representation from 17 of the 19 Spanish regions. Students took a quantitative test on computational thinking and another on AI before and after the intervention, and teachers and students also took a questionnaire at the end of it. The results showed that: i) quantitative tests can be used to reliably measure, in terms of internal consistency, the level of development of computational thinking and AI literacy in the Spanish school population of these ages; ii) it is possible to develop students’ computational thinking skills and raise their level of AI literacy through programming activities; iii) the intervention contributed to demystifying the main prejudices regarding AI; iv) a gender gap was found in relation to previous knowledge and experiences with these topics. Overall, the study concluded that introducing the foundations of AI in primary and secondary education in Spain is appropriate, feasible, and relevant. CCS CONCEPTS •Social and professional topics → Computational thinking; K-12 education;Computing literacy;•Computing methodologies →Artificial intelligence. KEYWORDS K-12, computational thinking, artificial intelligence literacy This work is licensed under a Creative Commons Attribution-Share Alike International 4.0 License. WiPSCE ’24, September 16–18, 2024, Munich, Germany ©2024 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-1005-6/24/09 https://doi.org/10.1145/3677619.3678111 ACM Reference Format: Jesús Moreno-León, Margarita Vasco-González, Marcos Román-González, and Gregorio Robles. 2024. Investigating the Impact of Programming Activities on Computational Thinking and AI Literacy in Spanish Schools. In The 19th WiPSCE Conference on Primary and Secondary Computing Education Research (WiPSCE ’24), September 16–18, 2024, Munich, Germany. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3677619.3678111 1 INTRODUCTION During the 2020-2021 academic year, the Ministry of Education and Professional Training of Spain, through the National Institute of Educational Technologies and Teacher Training, launched the School of Computational Thinking and Artificial Intelligence, EPCIA in its original form in Spanish [ 36 ]. This initiative aimed to expose non-university level students to artificial intelligence (AI), thus contributing to their AI literacy [35,57]. AI, a broad field, encompasses aspects related to logic, probability, mathematics, perception, reasoning, learning, ethics, and safety, among others [ 46 ]. In line with [ 23 , 27 ], in the framework of this project AI is defined as the science and engineering of developing systems that solve problems that would require intelligence if performed by a human. In the EPCIA project, computational thinking is regarded as a skill that enables problem-solving and the communication of ideas by leveraging the power offered by computers. This definition is rooted in the foundational proposals of Papert [ 38 ] and Wing [ 56 ], as well as recent reviews of scientific literature [33]. Specifically, in the EPCIA project computational thinking is defined as the human ability to solve problems and to express ideas by relying on concepts, practices and perspectives of computer science. Consequently, computational thinking is a dynamic psychological construct that must evolve to the same rhythm of computer science [ 42 ]. In other words, the evolution of computer science involves the parallel expansion of CT. As Tedre argued [ 51 ], since machine learning has triggered important changes in most computing fields, computational thinking must also evolve to a machine learning-enhanced version, "computational thinking 2.0" [52]. Therefore, taking into account that AI literacy not only includes aspects regarding what to know and understand about AI, but also how to use and apply AI and how to evaluate and create AI WiPSCE ’24, September 16–18, 2024, Munich, Germany Jesús Moreno-León, Margarita Vasco-González, Marcos Román-González, and Gregorio Robles systems [ 35 ], AI literacy could be considered as part of this computational thinking 2.0, serving as a different tool for communicating ideas and solving different kinds of problems [17]. In the EPCIA project, the main goal was to enhance the AI literacy of the participating students through the development of software projects. This means that the use of computer programming—and the subsequent development of computational thinking 1.0 skills—was a subordinate element to that end. 1.1 Computational thinking and AI literacy in pre-university studies The notion that computational thinking education should be introduced at the pre-university level has gained significant traction in recent years [ 1 , 7 , 20 , 47 , 48 ]. The research literature reveals successful cases illustrating the integration of programming activities in the final years of primary education and the initial years of secondary education to develop computational thinking [ 8 , 13 , 16 , 32 ]. It also showcases how it can be transversally applied to different subjects [ 5 , 30 , 50 , 58 ]. However, clear conclusions are yet to be drawn regarding the feasibility of using programming activities to enhance AI literacy at these educational levels, as this possibility has only recently begun to be explored. A meta-review by Su, Zhong & Ng studied 14 interventions that were carried out from 2018 to 2021 in the Asia-pacific region on AI curriculum for K-12 classrooms, concluding that most studies "showed a positive influence on students’ learning outcomes with various AI concepts such as AI, machine learning, neural networks, and deep learning and improve students’ interest in AI courses, develop AI-related skills, and strengthen their learning attitude" [ 49 ]. Rizvi, Waite & Sentance [ 40 ] performed a review that "included 28 student-centred studies looking into the evidence of AI learning and teaching in the K-12 setting"; their conclusion is that "[w]hile there is some evidence of successful implementation of AI and ML teaching in K-12, the topic remains open for iterative refinement". Mariescu-Istodor & Jormanainen designed a two-hour interactive lesson -first with secondary and later with primary school studentswhere pupils applied computer and mathematics knowledge to develop an object detection application, demonstrating that students were motivated and could undertake the lesson irrespective of their prior programming level [ 28 ]. Estevez, Garate & Graña [ 15 ] organized a workshop with 37 students aged 16-17, where participants programmed two AI algorithms with Scratch [ 39 ], demonstrating that the activity made a distinctive impact on participants’ understanding of AI. Van Brummelen, Heng & Tabunshchyk created a series of two-and-a-half-hour programming workshops for conversational systems with nine teachers and 47 students, concluding that participating students developed the 8 AI competencies defined in their curriculum, although ethics and machine learning proved to be the most complex [ 55 ]. Similarly, Zhu & Van Brummelen conducted a three-day workshop with 15 early secondary education students, where participants learned about conversational AI and gained confidence in their ability to create such systems [60]. In the online mode, several noteworthy interventions in this field have also been conducted. Martins, von Wangenheim, Rauber & Hauck developed and evaluated an online course to teach AI to 108 secondary students, yielding very positive results both in terms of student learning and their perception of it [ 29 ]. Rodríguez- García, Moreno-León, Román-González & Robles conducted a study with 135 students aged 10 to 16 that had a positive impact on the participants’ AI knowledge, as they programmed a project utilizing a machine learning model created by themselves [41]. On other hand, an active line of research studies learners’ conceptions of AI and its pedagogical implications for AI literacy [ 6 , 24 , 31 , 34 , 37 ]. Some of the main myths, prejudices and misconceptions identified in the literature include fear of a terminal danger for humanity, tedium and difficulty to learn AI, and a necessary link between AI and robots. 1.2 Purpose of the study Although existing studies offer promising insights regarding AI education, it is important to note that the small sample sizes they involved warrant caution in interpreting their conclusions. This brings to light the potential significance of the research presented in this article, which involves over 2000 primary and secondary students from different regions of Spain, making it one of the largest AI literacy studies conducted globally. Thus, the findings of this investigation could hold substantial value for both educators and researchers. To anticipate the potential for result generalization later in the discussion, it is essential to emphasize that this research employed two internationally recognized, used, and validated tools for primary and secondary stages: the visual programming language Scratch [ 32 , 39 ] and the Machine Learning for Kids platform [ 18 , 25 ] (ML4K). Considering that both computational thinking and artificial intelligence, our two primary study variables, remain emerging and developing constructs, our research questions aim not only to assess changes in these variables due to the educational intervention but also to determine if they can be reliably measured —a certainty not guaranteed when dealing with novel constructs and, specifically, when it comes to AI literacy [ 26 ]. Moreover, since computational thinking and artificial intelligence, although mainly cognitive constructs, have evident psychosocial ramifications, this study also addresses research questions oriented towards this dimension. Consequently, the specific research questions this study sought to answer are as follows: RQ1: Is it possible to achieve reliable measurements, in terms of internal consistency, of computational thinking and AI literacy among Spanish students from 4 th primary grade (K4) to 2 nd secondary grade (K8)? RQ2: Can the students of these ages develop their computational thinking skills and enhance their AI literacy through programming activities with Scratch and ML4K? RQ3: Do programming activities with Scratch and ML4K contribute to dispelling the primary misconceptions associated with AI? RQ4: Is there a gender gap concerning the knowledge and prior experiences with computational thinking and AI? Investigating the Impact of Programming Activities on Computational Thinking and AI Literacy in Spanish Schools WiPSCE ’24, September 16–18, 2024, Munich, Germany 2 MATERIALS AND METHODS 2.1 Design The research design follows a mixed-method approach. On the quantitative side, it is a pre-experimental study (with only an experimental group and no control group), with pre- and post-measures of two variables (development of “computational thinking” and knowledge about “artificial intelligence”). This is complemented by a qualitative component approached through surveys. The fact that the intervention did not include a control group limits the possibility of generalization and causality. However the use of multiple, rich sources of data -both quantitative and qualitativeallows for the triangulation of results obtained from both types of instruments. In addition, the huge sample size contributes to enhancing the concluding power of the design. Therefore, solid conclusions could be derived from this study. The following sections provide an overview of the context in which the intervention was conducted, outline the procedure, and the instruments used for data collection, and describe the sample of teachers and students who participated in the study. 2.2 Context The project unfolded in two phases. Initially, teachers underwent 30 hours of online training from November 2020 to February 2021. This training included a warmup unit on programming with Scratch, then an introduction to AI based on the AI4K12 framework [ 53 ] and a unit that equipped them with the skills to program projects incorporating AI to solve various types of problems, utilizing Scratch and ML4K. As a final project, participating teachers were tasked with designing a didactic proposal by planning a set of activities to be later implemented with their students in the classroom. Figure 1illustrates one of the projects tackled by the teachers during the training phase. It comprises a virtual assistant capable of understanding instructions written in natural language to manage home lighting and temperature. In this figure, the user introduces the prompt "I have finished reading and am going to bed" and the assistant, utilizing the AI model previously trained with ML4K, understands that this phrase belongs to the Turn off the light class and acts accordingly. As shown in Figure 1, the project does not include a list of predefined commands that the user must adhere to; instead, the model recognizes similarities with the examples used during its training to classify each received instruction, offering greater flexibility in using different phrases to communicate with the assistant. The next phase of the study, carried out between March and May 2021, involved teachers implementing the didactic proposal they designed during the prior training course in the classroom with their respective students. Only those who had successfully completed the course participated in this phase. The minimum implementation required during this stage was five one-hour classroom sessions, during which students created a machine learning model on the ML4K platform and programmed a project with Scratch that utilizes this model. Activities were therefore designed and implemented by the teachers with their students, not the researchers. Consequently, students participated in different kind of activities, depending on their teachers, although all the activities followed a common structure proposed during the training phase, all used Scratch and ML4K, and all were approved by the training coordinators. The primary objective of these activities was for students to grasp "how the computer systems they use daily employ AI to perceive the world using sensors, reason, learn, and interact with humans. Additionally, students should contemplate the impact AI can have on society, identifying both positive and negative effects" [36, p.10]. The article’s replication package 1 provides an extremely illustrative and relevant set of actions undertaken by teachers and students during the implementation phase. This list of projects includes a healthy food detector, a historical events classifier, a COVID-19 prevention recommender system , and a decision-making assistant for body care, among other examples where students utilize AI to program projects that solve problems of their interest. Along with the objectives, session structures, and evaluation rubrics, participating teachers share first-hand reflections on the encountered challenges and achieved successes. 2.3 Instruments and Procedure The research employs various techniques and instruments, collecting information from different types of sources, including both students and teachers. On one hand, one of the quantitative tests used is the Computational Thinking Test (CTt), a 28-item test designed to measure the development of computational thinking in individuals between 10 and 16 years old. Numerous pieces of evidence for its content validity can be found in the scientific literature [ 19 ], as well as its reliability and criterion validity [ 44 ], predictive validity [ 45 ], and concurrent validity [ 43 ]. Additionally, the CTt has been translated into several languages and has been employed by the international scientific community in various studies, which have proved its unidimensionality and scalability from the Item Response Theory (IRT) approach [10,21]. Figure 2shows one of the items from the CTt, in which students must find a mistake in one of the instructions of the given program. This illustrates that the CTt was developed for a computational thinking 1.0, in Tedre’s vision [ 51 ], which relates to rule-based programming. Therefore, CTt assesses CT skills from a deductive thinking (top-down) point of view, in which all the rules and steps that are executed by the machine must be previously and explicitly declared. The CTt was administered twice: initially just before the start of the didactic sequence implementation (pre-test) and again right after this phase concluded (post-test), with an 8-week interval between the two applications. Participating students were also required to undertake the AI Knowledge Test (AIKT). This instrument, consisting of 14 multiplechoice items, had been previously validated for students aged between 10 and 16 years in [ 41 ], an investigation that, in turn, was influenced by earlier works yielding positive outcomes [15,17]. The AIKT was designed following the 5 big ideas of AI proposed in the AI4K12 guidelines [ 54 ], with a special focus on idea 3—computers can learn from data— and idea 5—AI can impact society in both 1https://sites.google.com/view/replicationpackageai/ WiPSCE ’24, September 16–18, 2024, Munich, Germany Jesús Moreno-León, Margarita Vasco-González, Marcos Román-González, and Gregorio Robles Figure 1: A home lighting and temperature control assistant programmed with Scratch and ML4K. positive and negative ways—. What follows is an example of one of the items of the AIKT. Question: Which of the following strategies would be most appropriate for teaching a computer to recognize photos of apples? Possible answers: a) Train the computer with photos of dogs; b) Train the computer with several photos of different apples, taken in different places and contexts; c) Train the computer with several similar photos of the same apple, taken in the same place; d) Train the computer with several identical copies of the same photo of an apple. As can be seen, the AIKT evaluates aspects related to the construction and validation of AI/ML models, which are not programmed in the traditional way, but they are progressively built and trained throughout the collection and labeling of data sets (bottom-up). Consequently, the AIKT incorporates elements of Tedre’s computational thinking 2.0 vision [ 51 ]. For clarity, though, in this paper we refer as computational thinking to Tedre’s computational thinking 1.0 and as AI literacy to some of the elements considered in Tedre’s computational thinking 2.0. The AIKT test was also administered before (pre-test) and after (post-test) the implementation of the didactic sequence in the classroom. Finally, both the participating students and teachers were required to respond to a questionnaire. The students completed it after taking the two post-tests, while the teachers did so after concluding the intervention. These questionnaires were specifically constructed by the research team for this particular study. All Likert scales used in the questionnaires range from ’0=Not at all agree’ to ’10=Totally agree’; only the meaning of the two extreme poles of the scale is explicitly indicated, with no reference to the intermediate points. Both the quantitative tests and the utilized questionnaires are available to the scientific and educational community for free upon request to the authors of this study and have been included in the article’s replication package to provide readers with a clearer view of their structure and scope. 2.4 Participants The study involved 133 teachers and 2453 students from 17 of the 19 regions of Spain. Regarding the student participants, they ranged from 4 th grade in primary school (K4) to 2 nd grade in secondary school (K8). There was a slight over representation of males, who made up 50.6% of the sample. In terms of the Grade Level variable, there was a certain balance, with representation ranging from 13.6% in K7 to 28.6% in K5. Investigating the Impact of Programming Activities on Computational Thinking and AI Literacy in Spanish Schools WiPSCE ’24, September 16–18, 2024, Munich, Germany Figure 2: An item from the CTt in which students must find a mistake in one of the instructions of the loop. However, not all students correctly completed the CTt and AIKT in both applications. Consequently, the valid sample for these tests, consisting of participants whose responses could be paired, amounted to 2217 students for the CTt and 2088 for the AIKT. The response rate was therefore 90.37% for the CTt and 85.12% for the AIKT, generally maintaining the sex and grade distribution of the total sample. Concerning the teachers, there was a slight over representation of females, who constituted 54.14% of the participants. In terms of the educational level from which the teachers participated in the intervention, primary education was the most represented at 62.4%. The technical report of the research [ 36 ] provides detailed information on the composition and characteristics of the participant sample, covering both students and teachers. 3 RESULTS 3.1 Computational thinking test (CTt) 3.1.1 Pre and Post-Test Reliability. In the pre-test, a Cronbach’s Alpha value of .791 was obtained, which represents a reliable measurement scale [ 11 ]. It is worth noting that the values obtained increase as the grade levels progress, ranging from .622 in K4 to .811 in K8. On the other hand, in the post-test, Cronbach’s Alpha value rose to .820. Additionally, except for K4 where the value is .682, the values for the remaining grades are very close to or even exceed .80. Therefore, it can be stated that the measurement instrument is also reliable in this instance. In our research the validity of the CTt was not explicitly studied since there was previous evidence of it, as reported in section 2.3. 3.1.2 Analysis of Score Differences Between the Pre-Test and Post- Test. In general terms, the collected data follows an expected normal distribution pattern, suggesting that the measurements have been conducted accurately. No significant outliers were identified in the results, which indicates that the test difficulty is well-suited for the evaluated population. In the pre-test, the average score was 14.6464 out of a maximum possible score of 28. In the post-test, the average increased to 15.7415, meaning the mean score rose by 1.0951 points between the pre-test and post-test phases. Cohen’s d, a measure of the effect size of the intervention, stands at 0.271, with a statistically significant difference between the two instances (p < .001). Upon examining the results in relation to both gender and academic grade, as depicted in Figure 3, there is a noticeable improvement for both sexes, albeit a slightly greater increase among females, whose starting point was also marginally lower than that of the males. In both instances, the increments are statistically significant (p < .001). However, in both test applications, the mean scores for boys are higher than for girls. While these differences are statistically significant, the effect size is small, with Cohen’s d ranging between 0.2 and 0.3. These findings are consistent with previous studies using CTt, where males generally perform somewhat better [44]. As expected, considering that computational thinking is a cognitive skill influenced by an individual’s developmental maturity, a significant increase in CTt scores is observed as students progress through academic grades. This fact reinforces the conclusion that the measurements have been conducted appropriately. 3.2 Artificial Intelligence Knowledge Test (AIKT) 3.2.1 Measurement Reliability. The AIKT is an objective test assessing knowledge on AI. At the time of the intervention, AI literacy was not part of the school curriculum, so students are not expected WiPSCE ’24, September 16–18, 2024, Munich, Germany Jesús Moreno-León, Margarita Vasco-González, Marcos Román-González, and Gregorio Robles Figure 3: CTt total score pre-test vs. post-test by gender and grade. to already possess this knowledge prior to participating in the study. As a consequence, the reliability of this test should be measured exclusively after the intervention, during the post-test application. After an initial analysis of the test’s reliability considering the original 14 multiple-choice items, it was found that one of the questions was neither functioning correctly nor consistently. In particular it was question number 14 in the replication package—As a user, the legal regulation that is approved regarding AI systems will affect my life—, which presented an item-total correlation of 0.023 (< 0.2). As a result, this item was removed from the test, leaving the final version of the AIKT with only 13 items. The reliability score obtained for Cronbach’s Alpha is .701, which can be considered sufficient. In our research the validity of the AIKT was not explicitly studied since there was previous evidence of it, as reported in section 2.3. 3.2.2 Analysis of Score Differences Between the Pre-Test and Post- Test. The results generally follow an expected pattern of normal distribution, suggesting that the measurements have been conducted accurately. From the perspective of test difficulty, the absence of both floor and ceiling effects suggests that the difficulty level is appropriate for the evaluated subject population. The average score at the pre-test phase was 7.06 out of a maximum of 13. At the post-test phase, the average increased to 8.11. There is a statistically significant difference (p < .001), with an effect size of d=0.419, which according to Hattie’s Barometer [ 22 ], falls within the range of desired effects. The analysis by gender and academic grade of the differences in scores between the pre-test and post-test phases is presented in Figure 4. An increase is observed in all cases, with scores rising as students advance through academic grades. This data reinforces the conclusion that the measurements have been executed correctly, as it aligns with the expectation that older students would have more prior knowledge of AI. Both males and females show a statistically significant increase in scores (p < .001), although the improvement is slightly higher for females. This increment is systematically observed across all academic grades for both genders. Figure 4: AIKT total score pre-test vs. post-test by gender and grade. 3.3 Questionnaires by Students Approximately 43% of participating students indicated that they had heard of computational thinking prior to their involvement in the study. However, there is a statistically significant gender difference, as the figure is 48.4% for boys and only 36.3% for girls. Regarding AI, around 81% of students stated that they had heard of this term before the intervention. Here too, gender differences exist, with a rate that rises to 85.1% for boys and barely exceeds 76% for girls. As for programming activities with Scratch, nearly 68% of students had prior experience. The gender gap, although also present in this case, is smaller than in the previous cases, with percentages ranging from 69.9% for boys to 64.5% for girls. The use of ML4K is not as common, as less than 35% of the students indicate having used this tool at some point. The figure for boys stands at 36.9%, while the percentage for girls who affirm this drops to 31.6%, a difference that is, again, statistically significant. Table 1compiles student responses to a series of questions aimed at detecting shifts in their initial ideas and perceptions about AI. These questions were introduced with the following text: "After having carried out activities on computational thinking and AI in recent weeks, perhaps some of your ideas and perceptions about AI have changed. Compared to your prior ideas and perceptions, how would you say your current ideas and perceptions about AI are?" Overall, as can be observed in Table 1, the majority of students state that after participating in the intervention, they consider AI to be "less complicated", "less boring" and "less dangerous", and that AI is "more important to their lives" than they previously thought. However, most students also assert that they now believe AI is "more related to robots" than they had earlier thought. Consequently, the results show that the project activities have succeeded in dispelling most of the commonly held myths about AI in society, although this is not the case for the myth linking AI mostly to robots and not to software systems. Participating students generally rate the various project activities very positively. On a scale of 0 to 10, the average score is 7.82, with a median of 8 and a mode of 10. Additionally, only 11% of participating Investigating the Impact of Programming Activities on Computational Thinking and AI Literacy in Spanish Schools WiPSCE ’24, September 16–18, 2024, Munich, Germany Table 1: Student Responses to Questions on Prior Ideas and Perceptions Response Frequency Percentage AI is LESS COMPLICATED than you thought before 1651 67.3 AI is MORE COMPLICATED than you thought before 802 32.7 AI is LESS DANGEROUS than you thought before 1711 69.8 AI is MORE DANGEROUS than you thought before 742 30.2 AI is LESS IMPORTANT to your life than you thought before 654 26.7 AI is MORE IMPORTANT to your life than you thought before 1799 73.3 AI is LESS RELATED to ROBOTS than you thought before 819 33.4 AI is MORE RELATED to ROBOTS than you thought before 1634 66.6 AI is LESS BORING than you thought before 1933 78.8 AI is MORE BORING than you thought before 520 21.2 students indicate that they would not want to continue engaging in AI or computational thinking activities in future courses. 3.4 Questionnaires by Teachers Table 2displays the central tendency and dispersion metrics for teacher responses in a questionnaire aimed at capturing their assessment of their participation in the study. For each statement, teachers were asked to indicate the extent to which they agreed, using a scale ranging from "0=Strongly Disagree" to "10=Strongly Agree". The statements were as follows: "Students found it easy/simple to complete the instructional unit with Scratch and ML4K", "Students were motivated during the instructional unit with Scratch and ML4K", "Students had fun during the instructional unit with Scratch and ML4K", "Students developed their computational thinking skills as a result of the instructional unit with Scratch and ML4K", "Students enhanced their AI knowledge as a result of the instructional unit with Scratch and ML4K". As can be seen, teachers indicate that students were highly "motivated" (mode=10, median=9) and that they had "fun" with the various programming activities using Scratch and ML4K (mode=9, median=9). Regarding the difficulty of the activities, both the mode and the median drop to 7 when stating that the activities have been "easy/simple" for the students. These results align with the objectives set when designing the intervention, as activities that are either too easy or too difficult might not have been as motivating or enjoyable for the students. Teachers predominantly believe that students have gained AI knowledge (mode=8, median=8) and developed their computational thinking skills (mode=7, median=8) as a result of the activities in the implemented instructional unit. Table 3, meanwhile, captures teachers’ perceptions of the relevance that AI and computational thinking will have in students’ lives, both personally and professionally. In all four cases, the mode is 10 and the mean is around 9, making it clear that teachers consider these skills and knowledge to be highly relevant for their students’ futures. To conclude the questionnaire, teachers were asked to globally assess their participation as educators in the intervention project. Both the mode and the median rise to 9, while the average rating stands at 8.72 (out of a maximum of 10 points). 4 DISCUSSION AND CONCLUSIONS Regarding RQ1—Is it possible to achieve reliable measurements, in terms of internal consistency, of computational thinking and AI literacy in the Spanish student population from K4 to K8?—the data from the quantitative instruments used, CTt and AIKT, indicate that the measurements obtained have been reliable for both tests, especially between K5 and K8. The explanation for the reliability values obtained by CTt in K4, which are slightly lower than in other grades, is that CTt is an instrument that has been validated with students aged 10 to 16—a year older than the age group in that grade. Consequently, future work could consider the use of other derivative instruments that have been validated for that age, such as BCTt [59] or cCTt [14]. As for RQ2—Is it possible to develop the computational thinking of students of these ages and enhance their AI literacy through programming activities with Scratch and ML4K?—the answer is also clearly provided in the obtained results. Specifically, regarding the development of computational thinking, a positive impact has been achieved with a "small-moderate" effect size (d=0.27). Meanwhile, concerning the knowledge of AI, the effect size is "moderate" (d=0.42). The differences in the positive impact between both variables can be explained by the design of the intervention, as the focus of the research was on AI content. In other words, the development of computational thinking through programming activities using the Scratch language was merely the means used for creating practical projects in which AI content was integrated. Thus, the ultimate goal of the research was to improve the AI literacy of the students, with computer programming (and the subsequent development of computational thinking) being a subordinate element to that end. The results in the quantitative tests are also reflected in the responses from teachers and students in the questionnaires. On one hand, students believe that participating in the research has WiPSCE ’24, September 16–18, 2024, Munich, Germany Jesús Moreno-León, Margarita Vasco-González, Marcos Román-González, and Gregorio Robles Table 2: Teacher Perceptions on the Impact of the Intervention on Students Found it easy/simple Were motivated Had fun CT skills AI literacy N 133 133 133 133 133 Mean 6.28 8.63 8.38 7.51 7.51 Median 7.00 9.00 9.00 8.00 8.00 Mode 7 10 9 7 8 Std. deviation 2.123 1.384 1.358 1.837 1.841 Minimum 0 3 2 1 1 Maximum 10 10 10 10 10 Table 3: Perceptions on the Importance of Computational Thinking and Artificial Intelligence for the Future of Their Students Relevance of CT skills Relevance of AI Literacy Professional career Personal life Professional career Personal life N 133 133 133 133 Mean 9.22 8.98 9.05 8.91 Median 10.00 9.00 9.00 9.00 Mode 10 10 10 10 Std. deviation 1.032 1.234 1.176 1.258 Minimum 5 5 3 5 Maximum 10 10 10 10 helped them acquire knowledge in AI and develop their computational thinking skills, something that is notably valued. Additionally, participating students indicate in the questionnaires that both computational thinking and AI will have significant importance in their future lives, from both a personal and professional standpoint. On the other hand, teachers overwhelmingly consider to a significant degree that participating students have developed their computational thinking skills and acquired knowledge in AI throughout the intervention. Regarding the importance of these skills and knowledge for their students’ personal and professional future, teachers view them as highly relevant. These results align with the promising conclusions presented in the literature reviewed in the introduction of the article. However, unlike the small-scale studies available to date, this research includes a sample size of over 2,000 students and over 100 teachers, making it one of the largest studies conducted worldwide with primary and secondary school students in relation to AI literacy [ 40 ]. Additionally, in most reviewed investigations, it is the research team who conducts the workshops or activities with the students. In contrast, in this study, the teachers themselves, after receiving 30 hours of training, carry out the activities to work on AI with their students, which provides ecological validity to the research and is highly significant for its potential incorporation into educational curricula since the "success of AI education is closely dependent on the readiness of teachers" [2]. Regarding RQ3—Do programming activities with Scratch and ML4K help demystify the main prejudices associated with AI?—the obtained results show that indeed, most of the prejudices that have been addressed, such as "difficulty", "tedium", and "danger", have been demystified. However, the myth of the necessary link between AI and robots has not been eliminated, suggesting that future work should use different approaches and increased focus to address this relationship, which is a pervasive theme in related research [ 6 , 24 , 31,34,37]. As for RQ4—Is there a gender gap in relation to prior knowledge and experiences with computational thinking and AI?—the results show that girls have had fewer prior experiences and possess less prior knowledge about both topics, as well as about the programming tools used in the intervention. These findings align with those obtained in previous interventions [ 12 ], despite this being an issue that has been studied and attempts have been made to mitigate it for at least two decades [ 3 , 4 , 9 ]. Future research may aim to look further into the causes of this gap and potential solutions to contribute to its minimization. This study has some limitations that primarily stem from the fact that EPCIA is a program in which teachers participate voluntarily. Therefore, although the sample size of teachers is large, it is an intentional, non-random sample. The fact that 67.7% of the students had previous experiences with Scratch and 34.7% with ML4K suggests that the student sample may also not be representative of the Spanish student population at these educational levels in terms of their exposure to computer science and technology-related activities and experiences. Future work should aim to obtain a more representative sample of teachers and students through random selection. Another limitation worth noting is the absence of control groups. Aiming to mitigate the impact of this limitation, we used a mixedmethod approach —with both quantitative and qualitative instruments— enabling the triangulation of results. And we also increased the sample size, consequently enhancing the significance of the conclusions. The instruments used to assess computational thinking and AI literacy also presents some limitations. The main one is that both tools, the CTt and the AIKT, lack the capacity to evaluate students’ transference of these skills to other domains, a fundamental aspect Investigating the Impact of Programming Activities on Computational Thinking and AI Literacy in Spanish Schools WiPSCE ’24, September 16–18, 2024, Munich, Germany in Wing’s view [ 56 ]. In addition, the AIKT is mainly focused on ideas 3 and 5 from the AI4K12 guidelines [ 54 ], but ideas 1—computers perceive the world using sensors—, 2—agents maintain representations of the world and use them for reasoning—and 4—intelligent agents require many kinds of knowledge to interact naturally with humans—are barely studied. Future versions of the AIKT should include questions to assess all 5 big ideas of AI. Overall, the study shows that introducing the foundations of AI in primary and secondary education is not only appropriate and relevant, as teachers and students expressed in their questionnaires, but also feasible, since regular, in-service teachers were ready to do so after participating in a 30-hour online training, and the provided tests can be used to reliably measure, in terms of internal consistency, the level of development of computational thinking and AI literacy. ACKNOWLEDGMENTS This work has received public funding within the framework of the collaboration signed between the Ministry of Education and Vocational Training of Spain and Universidad Nacional de Educación a Distancia, through Professor M. Román-González. Parts of the results of this research have been previously published by the Ministry of Education and Vocational Training in the EPCIA project report (in Spanish) [ 36 ]. That report, aimed at non-university teaching staff, serves merely informative purposes and has not undergone peer review. It does not adhere to scientific presentation standards in terms of rigor, accuracy, or critical discussion of the results, among others. The work of G. Robles has been partially supported by the Spanish Ministry of Science, Innovation and Universities under the Excellence Network AI4Software (Red2022-134647-T). The work of J. Moreno-León was partially supported by FEDER/Ministry of Science, Innovation and Universities/Junta de Andalucía/State Research Agency/CDTI with the following grants: Data-pl (PID2022- 138486OB-I00), TASOVA PLUS research network (RED2022-134337- T), MIDAS (IDI-20230256) and AquaIA (GOPG-SE-23-0011). REFERENCES [1] Ruedi Arnold, Barbara Amstalden, and Jörg Bader. 2022. Enhancing the Role of Computational Thinking in Primary and Secondary Education in Switzerland. In Proceedings of the 17th Workshop in Primary and Secondary Computing Education (, Morschach, Switzerland,) (WiPSCE ’22). Association for Computing Machinery, New York, NY, USA, Article 24, 2 pages. https://doi.org/10.1145/3556787.3556874 [2] Musa Adekunle Ayanwale, Ismaila Temitayo Sanusi, Owolabi Paul Adelana, Kehinde D. Aruleba, and Solomon Sunday Oyelere. 2022. Teachers’ readiness and intention to teach artificial intelligence in schools. Computers and Education: Artificial Intelligence 3 (2022), 100099. https://doi.org/10.1016/j.caeai.2022.100099 [3] Bettina Bair and J. McGrath Cohoon. 2004. Special issue on gender-balancing computing education. J. Educ. Resour. Comput. 4, 1 (mar 2004), 1–es. https: //doi.org/10.1145/1060071.1060072 [4] Catherine Ball, Faron Moller, and Reena Pau. 2012. The mindstorm effect: a gender analysis on the influence of LEGO mindstorms in computer science education. In Proceedings of the 7th Workshop in Primary and Secondary Computing Education (Hamburg, Germany) (WiPSCE ’12). Association for Computing Machinery, New York, NY, USA, 141–142. https://doi.org/10.1145/2481449.2481483 [5] Laura Benton, Celia Hoyles, Ivan Kalas, and Richard Noss. 2016. Building mathematical knowledge with programming: insights from the ScratchMaths project. Suksapattana Foundation. [6] Arne Bewersdorff, Xiaoming Zhai, Jessica Roberts, and Claudia Nerdel. 2023. Myths, mis- and preconceptions of artificial intelligence: A review of the literature. Computers and Education: Artificial Intelligence 4 (2023), 100143. https://doi.org/ 10.1016/j.caeai.2023.100143 [7] Stefania Bocconi, Augusto Chioccariello, Panagiotis Kampylis, Valentina Dagien ˙ e, Patricia Wastiau, Katja Engelhardt, Jeffrey Earp, Milena Horvath, Egl ˙ e Jasut ˙ e, Chiara Malagoli, et al . 2022. Reviewing computational thinking in compulsory education: State of play and practices from computing education. (2022). [8] Quinn Burke and Yasmin B Kafai. 2012. The writers’ workshop for youth programmers: digital storytelling with scratch in middle school classrooms. In Proceedings of the 43rd ACM technical symposium on Computer Science Education. 433–438. [9] Deborah Butler. 2000. Gender, Girls, and Computer Technology: What’s the Status Now? The Clearing House 73, 4 (2000), 225–229. http://www.jstor.org/ stable/30189551 [10] Shiau-Wei Chan, Chee-Kit Looi, and Bambang Sumintono. 2021. Assessing computational thinking abilities among Singapore secondary students: A Rasch model measurement analysis. Journal of Computers in Education 8, 2 (2021), 213–236. [11] Lee J Cronbach. 1970. Essentials of psychological testing. (1970). [12] Jie Du and Hayden Wimmer. 2019. Hour of Code: A study of gender differences in computing. Information Systems Education Journal 17, 4 (2019), 91. [13] Caitlin Duncan and Tim Bell. 2015. A Pilot Computer Science and Programming Course for Primary School Students. In Proceedings of the Workshop in Primary and Secondary Computing Education (London, United Kingdom) (WiPSCE ’15). Association for Computing Machinery, New York, NY, USA, 39–48. https://doi. org/10.1145/2818314.2818328 [14] Laila El-Hamamsy, María Zapata-Cáceres, Pedro Marcelino, Barbara Bruno, Jessica Dehler Zufferey, Estefanía Martín-Barroso, and Marcos Román-González. 2022. Comparing the psychometric properties of two primary school Computational Thinking (CT) assessments for grades 3 and 4: The Beginners’ CT test (BCTt) and the competent CT test (cCTt). Frontiers in Psychology 13 (2022), 1082659. [15] Julian Estevez, Gorka Garate, and Manuel Graña. 2019. Gentle Introduction to Artificial Intelligence for High-School Students Using Scratch. IEEE Access 7 (2019), 179027–179036. https://doi.org/10.1109/ACCESS.2019.2956136 [16] Janne Fagerlund, Päivi Häkkinen, Mikko Vesisenaho, and Jouni Viiri. 2021. Computational thinking in programming with Scratch in primary schools: A systematic review. Computer Applications in Engineering Education 29, 1 (2021), 12–28. [17] Juan David Rodríguez García, Jesús Moreno-León, Marcos Román-González, and Gregorio Robles. 2020. LearningML: a tool to foster computational thinking skills through practical artificial intelligence projects. Revista de Educación a Distancia (RED) 20, 63 (2020). [18] Juan David Rodríguez García, Jesús Moreno León, Marcos Román González, and Gregorio Robles. 2019. Developing Computational Thinking at School with Machine Learning: An exploration. In 2019 International Symposium on Computers in Education (SIIE). 1–6. https://doi.org/10.1109/SIIE48397.2019.8970124 [19] M Román González. 2015. Computational thinking test: Design guidelines and content validation. In EDULEARN15 Proceedings. IATED, 2436–2444. [20] Nataša Grgurina, Erik Barendsen, Bert Zwaneveld, Wim van de Grift, and Idzard Stoker. 2013. Computational thinking skills in Dutch secondary education. In Proceedings of the 8th Workshop in Primary and Secondary Computing Education (Aarhus, Denmark) (WiPSE ’13). Association for Computing Machinery, New York, NY, USA, 31–32. https://doi.org/10.1145/2532748.2532768 [21] Josef Guggemos, Sabine Seufert, and Marcos Román-González. 2023. Computational thinking assessment–towards more vivid interpretations. Technology, Knowledge and learning 28, 2 (2023), 539–568. [22] John Hattie. 2016. Visible learning. A synthesis of over 800 metaanaly (2016). [23] Gwo-Jen Hwang, Haoran Xie, Benjamin W Wah, and Dragan Gašević. 2020. Vision, challenges, roles and research issues of Artificial Intelligence in Education. Computers and Education: Artificial Intelligence 1 (2020). [24] Moritz Kreinsen and Sandra Schulz. 2021. Students’ Conceptions of Artificial Intelligence. In Proceedings of the 16th Workshop in Primary and Secondary Computing Education (Virtual Event, Germany) (WiPSCE ’21). Association for Computing Machinery, New York, NY, USA, Article 14, 2 pages. https://doi.org/10.1145/3481312.3481328 [25] Dale Lane. 2021. Machine learning for kids: A project-based introduction to artificial intelligence. No Starch Press. [26] Matthias Carl Laupichler, Alexandra Aster, and Tobias Raupach. 2023. Delphi study for the development and preliminary validation of an item set for the assessment of non-experts’ AI literacy. Computers and Education: Artificial Intelligence 4 (2023), 100126. https://doi.org/10.1016/j.caeai.2023.100126 [27] Bruno Lepri, Nuria Oliver, and Alex Pentland. 2021. Ethical machines: The human-centric use of artificial intelligence. IScience 24, 3 (2021). [28] Radu Mariescu-Istodor and Ilkka Jormanainen. 2019. Machine Learning for High School Students. In Proceedings of the 19th Koli Calling International Conference on Computing Education Research (Koli, Finland) (Koli Calling ’19). Association for Computing Machinery, New York, NY, USA, Article 10, 9 pages. https: //doi.org/10.1145/3364510.3364520 [29] Ramon Mayor Martins, Christiane Gresse von Wangenheim, Marcelo Fernando Rauber, and Jean Carlo Rossa Hauck. 2022. Evaluation of the course Machine Learning for All! Technical Report. [30] Monica M. McGill, Laycee Thigpen, and Alaina Mabie. 2023. Emerging Practices for Integrating Computer Science into Existing K-5 Subjects in the United