Using Artificial Intelligence to Assess the Level of Cognitive Complexity Involved in Didactic Tasks. A case study on Historical Thinking

Artificial Intelligence and Education Enhancing Human Capabilities, Protecting Rights, and Fostering Effective Collaboration between Humans and Machines in Life, Learning, and Work M. Dolores Díaz-Noguera Carlos Hervás-Gómez Fulgencio Sánchez-Vera (Coords.) Horizontes Universidad Horizontes Universidad Artificial Intelligence and Education Enhancing Human Capabilities, Protecting Rights, and Fostering Effective Collaboration between Humans and Machines in Life, Learning, and Work This book offers a comprehensive view of how AI is transforming education, promoting accessibility and digital well-being. Through innovative studies, the book examines the responsible implementation of AI as a pedagogical tool, highlighting its inclusive potential for individuals with disabilities. With practical examples, such as the use of chatbots and digital simulations, it invites educators and developers to explore new learning methods. This work is a call to harness AI to enrich education and create a more accessible and equitable future. María Dolores Díaz-Noguera is a Full-Time Professor in the Department of Didactics and Educational Organization at the University of Seville (Spain). Her teaching is closely linked to the disciplines of Educational Organization and Management. In recent years, she has made significant contributions to educational knowledge, with a particular focus on technology integration in education and educational inclusion. She has over twenty years of experience teaching doctoral courses at the University of Seville and other Andalusian universities, and has supervised numerous international doctoral theses. Carlos Hervás-Gómez is a Full-Time Professor at the University of Seville (Spain), affiliated with the Faculty of Education Sciences in the Department of Didactics and Educational Organization. He teaches courses related to ICT. He is a member of the editorial boards of several journals, including the European Journal of Educational Research, International Journal of Educational Methodology, Journal of Research in Science, Mathematics and Technology Education, The European Educational Researcher, and Psychreg Journal of Psychology. His research focuses on the integration of ICT in teaching and learning processes through active and innovative methodologies, with a particular emphasis on emerging technologies and artificial intelligence in education. Fulgencio Sánchez-Vera is an Assistant Professor in the Department of Didactics and School Organization at the University of La Laguna (Spain). He holds a PhD in Social and Cultural Anthropology from the University of Murcia, where he also earned a degree in Computer Engineering. Additionally, he has a Master’s in Mindfulness from the University of Zaragoza and a University Expert title in Online Teaching from the International University of La Rioja. He has been a speaker at numerous courses on Educational Technology. Currently, he is the director of the journal “Cultura y Conciencia” and serves on the evaluation boards of various scientific journals. His scientific contributions include over fifty publications and participation in educational innovation and research projects, focusing on cyberspace, cyber culture, digital competence, and artificial intelligence in education. ARTIFICIAL INTELLIGENCE AND EDUCATION M. Dolores Díaz, Carlos Hervás, Fulgencio Sánchez (Coords.) H U Artificial Intelligence and Education Enhancing Human Capabilities, Protecting Rights, and Fostering Effective Collaboration between Humans and Machines in Life, Learning, and Work M. Dolores Díaz-Noguera Carlos Hervás-Gómez Fulgencio Sánchez-Vera (Coords.) Artificial Intelligence and Education Enhancing Human Capabilities, Protecting Rights, and Fostering Effective Collaboration between Humans and Machines in Life, Learning, and Work First published: november 2024 © M. Dolores Díaz-Noguera, Carlos Hervás-Gómez, Fulgencio Sánchez-Vera (coords.) © From this edition: Ediciones OCTAEDRO, S.L. C/ Bailén, 5 – 08010 Barcelona Tel.: 93 246 40 02 [email protected] www.octaedro.com This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License: https://creativecommons.org/licenses/by-sa/4.0/ ISBN: 978-84-10282-45-2 DOI: https://doi.org/10.36006/09643-1 Typeset by: Fotocomposición gama, sl Production: Octaedro Editorial Open Access Collection Horizontes-Universidad Title: Artificial Intelligence and Education: Enhancing Human Capabilities, Protecting Rights, and Fostering Effective Collaboration between Humans and Machines in Life, Learning, and Work 7 Summary Prologue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1. Impact of Artificial Intelligence on Academic Integrity: Perspectives of Faculty Members in Spain . . 13 Fulgencio Sánchez-Vera; Inmaculada Perdomo Reyes; Betty Estévez Cedeño 2. Analysis of an Artificial Intelligence Training Program in University Students: Perspectives and Horizons . . . 31 PhD Alejandro López-García; PhD José María Campillo- Ferrer; PhD María Victoria Zaragoza-Vidal; PhD Pedro Miralles-Sánchez 3. Validation of the Attitude Scale on the Perceptions of Future Teachers about Artificial Intelligence and its Consequences on their Motivation, Critical Thinking and Improvement of their Learning . . . . . . . . . . . . . . . . 51 PhD María Dolores Díaz-Noguera; PhD Carlos Hervás- Gómez; Eduardo Puraivan; Mg. Macarena Gema Astudillo-Vásquez; Mg. Connie Cofré-Morales 4. Artificial Intelligence as a Teaching Tool to Promote the Development of Critical Thinking in Primary Education Students . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 PhD Gloria Morales-Pérez; PhD Pedro Román-Graván; María de los Ángeles Domínguez-González; Manuel Reina-Parrado 8Artificial Intelligence and Education 5. AI as a Tool for Educational Transformation: Keys for Responsible Implementation Fostering Digital Wellbeing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 PhD Ortí Martínez; PhD Pardo Ríos; Jorge Burgueño López; Javier Plitt Stevens 6. Artificial Intelligence Tools to Improve Accessibility in Education for People with Disabilities . . . . . . . . . . . 93 PhD Carlos Hervás-Gómez; PhD Ángela Martín- Gutiérrez; María de los Ángeles Domínguez-González; Carmen Manzanares-Castillo; PhD Hăisan Angel-Alex; PhD Nadia Barkoczi 7. Design, Implementation, and Evaluation of a Chatbot to Enhance Inclusive Learning through Universal Design for Learning in University Students . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 PhD Judit Ruiz-Lázaro; PhD Eva Jiménez-García; PhD Sonia Martínez-Requejo; PhD Sara Redondo- Duarte 8. Society and Artificial Intelligence: Chatbots in the Educational Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 PhD Marta Aguilar-Gil 9. I Am not as Rich as Alexander the Great, but I Can Have my own AI-RISTOTLE . . . . . . . . . . . . . . . . . . . . . . . 145 PhD Jonatan Castaño-Muñoz; PhD Celia Moreno- Morilla; Manuel Reina-Parrado 10. The Use of Artificial Intelligence Tools among University Students and its Association with Personality Traits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 PhD Joaquín Rodríguez-Ruiz; PhD Raquel Espejo-Siles; PhD Inmaculada Marín-López 11. Emotions in Digital Learning: Assessing the Emotional and Pedagogical Impact of Historical Simulations through ChatGPT in University Classrooms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 PhD Pablo Rosser; PhD Seila Soler 9Summary 12. Training GPT as a Standardized Patient . . . . . . . . . . . . . 189 Mercedes Lorena Pedrajas López; Ana Sanz Cortés; Eva García Carpintero-Blas; Esther Martínez Miguel; Sara Uceda Gutiérrez 13. Using Artificial Intelligence to Assess the Level of Cognitive Complexity Involved in Didactic Tasks. A case study on Historical Thinking . . . . . . . . . . . . . . . . 205 Mg. Connie Cofré-Morales; Eduardo Puraivan; Ing. Karina Huencho-Iturra; Mg. Macarena Astudillo- Vásquez; PhD Carlos Hervás-Gómez; PhD M. Dolores Díaz-Noguera 14. Space Design Assisted by Artificial Intelligence in Early Childhood Education Classrooms . . . . . . . . . . . . 219 PhD Rosario Ordóñez-Sierra; PhD Margarita Rodríguez-Gallego; Cristina De-Cecilia-Rodríguez 15. New Technologies Applied to the Improvement of Human Anatomy Learning . . . . . . . . . . . . . . . . . . . . . . . . . 237 PhD Soledad Aguado-Henche; PhD Rafael Moreno- Gómez-Toledano; PhD Ana María Slocker de Arce; PhD Gabriel de Arriba de la Fuente; PhD Miguel A. Ortega; PhD Soledad Cristóbal Aguado; PhD Mónica Grande Alonso 205 13 Using Artificial Intelligence to Assess the Level of Cognitive Complexity Involved in Didactic Tasks. A case study on Historical Thinking Mg. Connie Cofré-Morales Universidad Viña del Mar, Chile [email protected] https://orcid.org/0009-0000-1442-1218 Eduardo Puraivan Universidad Viña del Mar, Chile Universidad de Playa Ancha, Chile [email protected] https://orcid.org/0000-0003-2134-8922 Ing. Karina Huencho-Iturra Universidad Viña del Mar, Chile [email protected] https://orcid.org/0009-0009-7043-2929 Mg. Macarena Astudillo-Vásquez Universidad Viña del Mar, Chile [email protected] https://orcid.org/0000-0001-5840-0865 PhD Carlos Hervás-Gómez Universidad de Sevilla, Spain [email protected] https://orcid.org/0000-0002-0904-9041 PhD M. Dolores Díaz-Noguera Universidad de Sevilla, Spain [email protected] https://orcid.org/0000-0002-0624-4079 13. Using Artificial Intelligence to Assess the Level of Cognitive Complexity Involved... https://doi.org/10.36006/09643-1-13 206 Artificial Intelligence and Education Abstract The design of didactic activities is aimed at developing learning objectives at different levels of cognitive complexity. In the field of History teaching, one of the purposes is to develop Historical Thinking, which requires specific tasks. However, several investigations report that many of the activities presented in school textbooks are limited to the use of the lowest cognitive levels, not achieving the intended Historical Thinking of students. Faced with this singularity, it is valid to ask how we can use some tools to validate that the activities fulfill the expectations of cognitive complexity. In this paper, we discuss the use of an AI to evidence whether it can help as a support tool for this task. The results show that the task of assigning the predominant cognitive level is challenging even for highly qualified experts, and that the AI results match at least with expert’s assessment using some of the taxonomies considered, except for one activity. On the other hand, there is a high appreciation by experts of the potential of ChatGPT to both classify and argue its decisions, although there are also some risks to be considered. Keywords: Artificial Intelligence, ChatGPT, cognitive complexity, historical thinking. 13.1. Introduction In the subject of History, from the perspective of the development of skills and competencies, in recent decades, the focus has been placed on students being able to develop “historical thinking”, which is defined as the deployment of a set of specific skills of the discipline to ensure that the historical facts of the subject of History are not memorized, but constructed by the students themselves through the use of historical evidence or sources to generate an interpretation of the facts, emulating the work of a historian (Wineburg, 2001; Kitson, etal., 2015; Sáez-Rosenk- ranz, 2017). In this sense, the teaching of historical thinking consists in initially posing a problem or research question that guides the inquiry that students will carry out in the historical sources, in order to solve the problem posed or provide an answer to the question (Seixas & Morton, 2013; Kitson etal., 2015; Henríquez etal., 2018). The levels of cognitive complexity identifiable in didactic tasks that seek to develop specific objectives and skills, as is the case of historical thinking, can be identified through the use of 21313. Using Artificial Intelligence to Assess the Level of Cognitive Complexity Involved... Table 13.1. Experts and ChatGPT results (Ai: Activity “i”, T: Taxonomy, Ej: Expert “j”. A T ChatGPT E1E2E3E4E5E6 A1Bloom Analyze (50%) Analyze Understand Sáiz High (50%) High Medium A2Bloom Analyze (0%) Evaluate Remember Sáiz High (50%) High Medium A3Bloom Evaluate (50%) Evaluate Remember Sáiz High (50%) High Low A4Bloom Analyze (50%) Analyze Remember Sáiz Medium (50%) Medium Low A5Bloom Analyze (0%) Evaluate Understand Sáiz High (0%) Medium Medium A6Bloom Evaluate (100%) Evaluate Evaluate Sáiz High (50%) Medium High Source: developed by author. Assessment of the work performed by ChatGPT As was previously mentioned, once their classification was completed, the experts were shown the results obtained with Chat- GPT, asking them to rate (on a scale of 1 to 5) the classification and arguments provided by the chat. Table 13.2 shows an example of the classification and general arguments provided by ChatGPT. 214 Artificial Intelligence and Education Table 13.2. Example of classification and arguments provided by ChatGPT Taxonomy Assigned level Arguments Bloom (Anderson and Krathwohl’s revision) Analyze The task requires the student to interpret the symbolism of a public act, understanding the meaning behind the selection of the date and the relationship to a historical event. This demands analytical skills to connect the visual information with previous knowledge of history and politics. Model of Sáiz (2013) High The student must go beyond the mere identification of visual elements or textual comprehension (low and medium levels). They are expected to make a critical evaluation of the act in its historical and political context, which implies the creation of new information from a deep reflection on the symbology and intentionality behind the selection of the date. The average rating is shown in Table 13.3. It is interesting to note that there is a high valuation in both dimensions. On the other hand, the experts based their ratings both on the classification and on the arguments provided by ChatGPT. Among their justifications (free text), based on identifying benefits and risks, the experts show interest and surprise in the classification made by ChatGPT, but they also state as a risk that teachers may slightly accept the results and rely too much on the tool. They make explicit reference to the fact that the tool can be wrong, and that some historical contexts may not be well understood, which can lead to errors in the results. Table 13.3. Average ratings in the classification and arguments provided by ChatGPT (Ai: Activity “i”). INDICATOR A1A2A3A4A5A6 Average rating in the classification 445535 Average rating in the argument 5 4.5 5 5 4 5 Source: developed by author. 13.5. Conclusions and Projections The achievement of the learning objectives is of utmost relevance for the quality of the teaching that is delivered. In this sense, one 21513. Using Artificial Intelligence to Assess the Level of Cognitive Complexity Involved... of the aspects to consider and review in teaching is the complexity of the didactic activities proposed to students, to ensure that they are appropriate to the expected learning, for which Bloom’s taxonomy (Wang etal., 2021), or others specific to the subject, can be used. If the subject of History is intended to develop historical thinking, the most appropriate levels of complexity should be the highest (Sáiz, 2014; Palacios, 2019; Bramann, 2021). The use of AI as a tool for the task of classifying activities can be valuable and contribute greatly to quality teaching. The results of our experiments indicate an initial finding: the difficulty of assigning a specific cognitive level to didactic activities, despite the fact that these were selected directly from school textbooks and are well defined. This complexity is evidenced by the lack of consensus among experts, even using a commonly employed taxonomy such as Bloom’s. This finding suggests the importance of expanding research to other areas of knowledge to better understand these challenges. A second significant finding of our study is that, for each didactic activity (except for Activity 5), the ChatGPT’s results match at least the classification provided by one of the experts, when considering the taxonomies individually. This coincidence indicates that the ChatGPT’s classifications are in line with the experts’ evaluations, showing relevant consistency. This result suggests that the tool, in terms of classification, does not present significant deviations or obvious errors that could be considered as “hallucinations” in its responses. The third finding of our study is the experts’ favorable assessment towards the use of ChatGPT in the assigned task, both in terms of classification and argumentation. However, experts also warn about the need for caution when using this tool. They highlight the importance of avoiding overconfidence of teachers and recall that, like any automated tool, ChatGPT is not errorfree (Hashem etal., 2024; Kwan, 2024; Gill etal., 2024), e.g., in challenging contexts such as assigning a level of cognitive complexity where there are various reasons that can make this task difficult. For instance, in some cases, an activity can be associated with more than one level of Bloom’s taxonomy (Rawat etal., 2023) or the amount of time and resources associated with this task when it is performed considering a big amount of activities (Rawat etal., 2023; Wang etal., 2021), which is why it is necessary to work on an automated tool to support it. This balance 216 Artificial Intelligence and Education between recognition of its usefulness and awareness of its limitations is crucial for its effective application in educational contexts. The current results lead us to plan future research in two key directions: to extend the empirical evidence with a wider range of didactic activities and to diversify the areas of knowledge studied. In the medium term, our goal is to develop a software tool based on recent advances in AI. This tool will support decision making in assigning levels of cognitive complexity to didactic activities and will be aimed primarily at trainee and novice teachers as initial users. References Bramann, C. (2021). Fostering Historical Thinking with Textbooks. A case study of tasks in Austrian history textbooks (conference). International Conference on Textbooks and Educational Media, Berlin, Germany. https://doi.org/10.1007/978-3-030-80346-9_31 Bolaño-García, M., & Duarte-Acosta, N. (2024). Una revisión sistemática del uso de la inteligencia artificial en la educación. Revista Colombiana de Cirugía, 39(1), 51-63. https://doi.org/10.30944/20117582.2365 Förster, C., & Rojas-Barahona, C. (2017). Aprendizaje y evaluación: lo que no se evalúa, no se aprende. In C. Förster (Ed.). El poder de la evaluación en el aula. Mejores decisiones para promover aprendizajes (pp. 43-74). Universidad Católica de Chile. Gill, S., Xu, M., Patros, P., Wu, H., Kaur, R., Kaur, K., Fuller, S., Singh, M., Arora, P., Kumar, A., Stankovski, V., Abraham, A., Ghosh, S., Lutfiyya, H., Kanhere, S., Bahsoon, R., Rana, O., Dustdar, S., Sakellariou, R. ..., & Buyya, R. (2024). Transformative effects of ChatGPT on modern education: Emerging era of AI chatbots. Internet of Things and Cyber-Physical Systems, 4, 19-23. https://doi.org/10.1016/j. iotcps.2023.06.002 Gómez, C. (2014). Pensamiento histórico y contenidos disciplinares en los libros de texto. Un análisis exploratorio de la Edad Moderna en 2.º de la ESO. Ensayos. Revista de la Facultad de Educación de Albacete, 29(1), 131-158. https://revista.uclm.es/index.php/ensayos/ article/view/498 Gómez, C., & Miralles, P. (2016). Développement et évaluation des compétences historiques dans les manuels scolaires. Une étude 21713. Using Artificial Intelligence to Assess the Level of Cognitive Complexity Involved... comparative France-Espagne. Spirale. Revue de Recherches en Education, 2(58), 53-66. https://www.cairn.info/revue-spirale-revue-de- recherches-en-education-2016-2-page-53.htm Hashem, R., Ali, N., El Zein, F., Fidalgo, P., & Abu Khurma, O. (2024). AI to the rescue: Exploring the potential of ChatGPT as a teacher ally for workload relief and burnout prevention. Research and Practice in Technology Enhanced Learning, 19. https://doi.org/10.58459/ rptel.2024.19023 Henríquez, R., Carmona, A., Quinteros, A., & Garrido, M. (2018). Leer y escribir para aprender Historia. Secuencias para la enseñanza y el aprendizaje del pensamiento histórico. Universidad Católica de Chile. Jiménez, S., Juárez-Ramírez, R., Castillo, V., & Tapia, J. (2018). Affective Feedback in Intelligent Tutoring Systems. A practical approach. Springer. https://doi.org/10.1007/978-3-319-93197-5 Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A. Seidel, T. ..., & Kasneci, G. (2023). Chat- GPT for Good? On opportunities and challenges of large language models for education. Journal of Psychology and Education, 103. https://doi.org/10.35542/osf.io/5er8f Kitson, A., Steward, S., & Husbands, C. (2015). Didáctica de la Historia en Secundaria Obligatoria y Bachillerato. Comprender el pasado. Morata. Küchemann, S., Steinert, S., Revenga, N., Schweinberger, M., Dinc, Y., Avila, K., & Kuhn, J. (2023). Can ChatGPT support prospective teachers in physics task development? Physical Review Physics Education Research, 19(2). https://doi.org/10.1103/PhysRevPhysEducRes. 19.020128 Kwan, C. (2024). Exploring ChatGPT-Generated Assessment Scripts of Probability and Engineering Statistics from Bloom’s Taxonomy (conference). International Conference on Technology in Education, Hong Kong, China. https://doi.org/10.1007/978-981-99-8255-4_24 Martínez, M., & Gómez, C. (2018). Cognitive level and thinking historically competencies in history textbooks from Spain and England: A comparative study. Revista de Educación, 379, 136-159. https://doi.org/10.4438/1988-592X-RE-2017-379-364 Palacios, N. (2019). Teaching violence, drug trafficking and armed conflict in Colombian schools: Are history textbook deficient? Issues in Educational Research, 29(3), 899-922. https://www.iier.org.au/iier 29/palacios.pdf 218 Artificial Intelligence and Education Rawat, A., Kumar, S., & Samant, S. (2023). A Systematic Review of Question Classification Techniques Based on Bloom’s Taxonomy (conference). International Conference on Computing Communication and Networking Technologies, Delhi, India. https://doi.org/10.11 09/ICCCNT56998.2023.10308403 Sáez-Rosenkranz, I. (2017). La enseñanza de la historia en los libros de texto de educación básica en Chile. Revista Enseñanza de las Ciencias Sociales, 16, 27-40. https://doi.org/10.1344/ECCSS2017.16.3 Sáiz, J. (2013). Alfabetización histórica y competencias básicas en libros de texto de historia y en aprendizajes de estudiantes. Revista Didáctica de las Ciencias Experimentales y Sociales, 27, 43-66. http:// dx.doi.org/10.7203/dces.27.2648 Sáiz, J. (2014). Fuentes históricas y libros de texto en secundaria: Una oportunidad perdida para enseñar competencias de pensamiento histórico. Ensayos. Revista de la Facultad de Educación de Albacete, 29(1), 83-99. https://revista.uclm.es/index.php/ensayos/article/ view/503/458 Seixas, P., & Morton, T. (2013). The Big Six Historical Thinking Concepts. Nelson Education. https://bibliotecadigital.mineduc.cl/handle/20. 500.12365/17630 Wang, Z., Manning, K., Mallick, D., & Baraniuk, R. (2021). Towards Blooms Taxonomy Classification Without Labels (conference). Artificial Intelligence in Education: 22nd International Conference, Utrecht, Netherlands. https://doi.org/10.1007/978-3-030-78292-4_35 Wineburg, S. (2001). Historical thinking and other unnatural acts: Charting the future of teaching the past. Temple University.