Guide for Expert Review of the GAI-SciTeach Focus Group Semi-Structured Interview Script

1/23 Guide for Expert Review of the GAI-SciTeach Focus Group SemiStructured Interview Script - Task 2 Authors: Juliana Monteiro, Margarida M. Marques This document is organized as following: 1. Contextualization of the Study ............................................................................................ 1 1. Research question ........................................................................................................... 1 2. Research objectives ......................................................................................................... 2 3. Target audience ............................................................................................................... 2 2. Materials for review ............................................................................................................. 2 1. Construct definitions ....................................................................................................... 2 2. Practical instructions for the review, please: ................................................................. 3 1. Focus group interview guide and form ........................................................................... 4 Introduction to the focus group ............................................................................................ 4 Development of the focus group .......................................................................................... 6 Closing of the focus group ................................................................................................... 22 Other Suggestions – point 4 of the "Practical Instructions for Revision" ............................. 23 1. References .......................................................................................................................... 23 1. Acknowledgements ............................................................................................................ 23 We ask you to critically analyse the interview script in light of the study objectives and the defined constructs, using the feedback form to record your observations. Thank you for your availability and collaboration in the revision of this script. 1. Contextualization of the Study 1. Research question What are science education stakeholders’ perspectives about the integration of Generative Artificial Intelligence (GAI) in science education and in science teacher education? Using the PICo framework (Hosseini et al., 2024): 2/23 • Population: stakeholders in science education (researchers and teacher educators, (future) teachers) • Intervention/phenomenon: educational integration of GAI • Context: science teaching, science teacher training 2. Research objectives ⎯ To collect indicators of respondents' literacy in GAI in science education ⎯ To assess the GAI perceived ease of use in science education and GAI tools learning curve ⎯ To assess the perceived usefulness of GAI tools in enhancing science education ⎯ To assess the perceived trust in GAI tools for content creation for science education and factors influencing it ⎯ To assess perceptions on knowledge needed for GAI integration in science education ⎯ To assess perceptions on the acceptance of GAI tools in science education ⎯ To assess perceptions regarding GAI ethical issues in science education 3. Target audience A focus group with each of the profiles (estimated 4-6 respondents) is planned: ⎯ Education researchers/trainers of future science teachers; ⎯ Future Science Teachers; ⎯ Science teachers of the 3rd cycle of basic education / secondary education. 2. Materials for review 1. Construct definitions GAI literacy: knowledge and understanding of what Generative Artificial Intelligence is, how it works, its applications, capabilities and limitations, including the respondent's ability to distinguish GAI-based tools and to choose the most appropriate ones for specific tasks. Ease of use of GAI: The degree to which an individual believes that using GAI tools would be free of physical and mental exertion. It includes the perception of the clarity and simplicity of the interaction with the tools and the ease of acquiring the necessary skills for their effective use in teaching. Usefulness of GAI: insight into whether the use of GAI tools can improve work performance, increase productivity, help to perform teaching tasks more effectively, and achieve higher quality outcomes in the context of science teaching and learning. Trust in GAI: belief in the reliability, security, and accuracy of information and content generated by GAI tools. It includes confidence that tools meet expectations and consideration of risks to teacher and student privacy. Knowledge for GAI integration: knowledge that science teachers need to effectively integrate GAI tools into their pedagogical practices. This involves the ability to properly combine specific disciplinary content (Content Knowledge), pedagogical strategies (Pedagogical Knowledge) and the functioning of GAI (Technological Knowledge) tools, also considering the capabilities and limitations of AI and its ethical implications. It includes the knowledge to use GAI for learning monitoring and adaptive feedback, for example. 3/23 Acceptance of GAI: intention and willingness to adopt and use GAI tools in the teaching and learning of science, as well as their support for their integration into the education system. Ethical Issues of GAI use: perceptions and concerns about the ethical aspects related to the use of AGI in teaching, including the ability to teach students about ethics in GAI, safe and responsible behavior, protection of sensitive data (exams, grades, personal data), privacy, information security, and well-being when using these tools. 2. Practical instructions for the review, please: 1) Read each interview guide question: major and extension (defined as per Amado (2014)). 2) Evaluate the following criteria per question: • Clarity: The wording is understandable and appropriate to the target audience? • Relevance: The question effectively addresses the proposed construct (coherence of the constructs, objectives, and original formulation of the items in the baseline studies1)? (Al-Abdullatif, 2024; Chiu et al., 2024) 3) Evaluate the following criteria by dimension: • Comprehensiveness: Does the dimension include enough questions to cover the construct? There are unnecessary redundancies? 4) The sequence of questions and dimensions seems logical and appropriate for a focus group? 5) Using the following form, which is part of the interview guide, indicate suggestions for reformulation or exclusion. Identify gaps or redundancies. 1 The baseline studies were: Al-Abdullatif, A. M. (2024). Modeling Teachers’ Acceptance of Generative Artificial Intelligence Use in Higher Education: The Role of AI Literacy, Intelligent TPACK, and Perceived Trust. Education Sciences, 14(11), 1209. https://doi.org/10.3390/EDUCSCI14111209 Chiu, T. K. F., Ahmad, Z., & Çoban, M. (2024). Development and validation of teacher artificial intelligence (AI) competence self-efficacy (TAICS) scale. Education and Information Technologies, 30(5), 6667–6685. https://doi.org/10.1007/S10639-024-13094Z/FIGURES/1 4/23 1. Focus group interview guide and form The discussion group is organized into three sections: Introduction, Development (with the dimensions of analysis) and Closing. Introduction to the focus group Objectives Information Does it require reformulation? (Yes or No) Qualitative comments (to be filled in only if you mentioned "yes" in the previous column; indicating lack of adequacy to the objectives, making suggestions, etc.) Introducing the GAI-SciTeach project, the interviewer, and the objectives of the focus group Project brief description and main expected results. Interviewer’s name, affiliation, and area of work. We want to know your opinion on the use of generative artificial intelligence (GAI) in science education, namely with regard to GAI literacy, its ease of use, its usefulness, confidence in the content it generates (and factors that influence it), the knowledge that is necessary for its use, its acceptance and the ethical questions it raises. Thank the interviewees and recognize the value of their contributions Inform about GDPR and ethical procedures With your permission, the discussion group will be audio-recorded and transcribed. Although the collection of personal data is not the objective, such information may be disclosed during the interview. In accordance with the confidentiality provisions of the General Data Protection Regulation (GDPR), any personal data will be removed from the transcripts. 5/23 Objectives Information Does it require reformulation? (Yes or No) Qualitative comments (to be filled in only if you mentioned "yes" in the previous column; indicating lack of adequacy to the objectives, making suggestions, etc.) You will receive anonymized transcripts for analysis and will have two weeks to review them and suggest any relevant corrections. The remaining data (non personal) will be used for research and may be shared in data repositories. Do you need further clarifications? Do you accept this procedure? 6/23 Development of the focus group Dimension / Objective Original questions (from the baseline studies) Main questions Probing questions Evaluation (Likert scale: fill from 1 - nothing to 5 - totally) Does it require reformulation? (Yes or No) Qualitative comments (to be filled in only if you mentioned "yes" in the previous column, indicating lack of adequacy to the objectives, making suggestions, etc.) GAI Literacy / To collect indicators of respondents' literacy in GAI in science education I can explain what AI is [AIK3]. (Chiu et al. 2024) 1. Can you explain what Generative Artificial Intelligence is? Pode explicar o que é a Inteligência Artificial Generativa? - Clarity Relevance 1a. Please define the concept as you understand it. Por favor, defina o conceito como o entende. Clarity Relevance 1b. What examples would you use to illustrate what Generative Artificial Intelligence is ? Que exemplos usaria para ilustrar o que é Inteligência Artificial Generativa? Clarity Relevance I can distinguish whether a tool is AI-based or not. [AIK1] (Chiu et al. 2024) 2. How can people determine if new tools are based, or not, in Generative Artificial Intelligence ? Como saber se uma ferramenta é baseada em Inteligência Artificial Generativa ou não? - Clarity Relevance 2a. Can you illustrate with an example? Clarity 7/23 Pode ilustrar com um exemplo? Relevance 2b. What characteristics indicate that a tool uses Generative Artificial Intelligence? Que características indicam que uma ferramenta utiliza Inteligência Artificial Generativa? Clarity Relevance I can use AI applications or products to improve my work efficiency. (AlAbdullatif 2024) I can create content with AI. [AIK2] (Chiu et al. 2024) 3. How can people use GAI tools in science education for greater efficiency? Como usar ferramentas de Inteligência Artificial Generativa na educação em ciências tendo em vista uma maior eficiência? - Clarity Relevance 3a. Illustrate, by referring to concrete tasks and objectives (in science education) that can be accomplished/achieved more efficiently through the use of GAI tools. Ilustre referindo tarefas concretas e objetivos (em educação de ciências) que podem ser realizadas/alcançados com maior eficiência através da utilização de ferramentas IAG. Clarity Relevance 8/23 I know how to choose the right AI tools to effectively complete a task. [AIK4] (Chiu et al. 2024) I can evaluate the capabilities and limitations of an AI application or product after using it for a while. (Al-Abdullatif 2024) 4. How can people choose GAI tools that are adequate for carrying out a given task in science education ? Como escolher ferramentas de IAG adequadas à realização de uma dada tarefa em educação de ciências? - Clarity Relevance Does the scope of the dimension require reformulation? Yes/No Space for comments on whether the set of questions effectively covers the construct. 9/23 Dimension / Objective Original questions (from the baseline studies) Main questions Probing questions Evaluation (Likert scale: fill from 1 - nothing to 5 - totally) Does it require reformulation? (Yes or No) Qualitative comments (to be filled in only if you mentioned "yes" in the previous column, indicating lack of adequacy to the objectives, making suggestions, etc.) Perceived Ease of Use of GAI tools / To assess the GAI perceived ease of use in science education and GAI tools learning curve My interaction with GenAI tools is clear and simple. (Al-Abdullatif 2024) 5. How easy or difficult is it to use IAG tools and what factors influence this? Em que medida é fácil ou difícil usar ferramentas de IAG e que fatores o influenciam? - Clarity Relevance 5a. Can you illustrate with an example? Pode ilustrar com um exemplo? Clarity Relevance 5b. What made it easy or difficult for you or for others you know? O que tornou isso fácil ou difícil para si ou para outras pessoas que conhece? Clarity Relevance Learning how to use GenAI tools is easy for me. (AlAbdullatif 2024) It is easy for me to become skillful in using GenAI tools. (Al-Abdullatif 2024) 6. How easy or difficult is it to learn to use GAI tools? Em que medida é fácil ou difícil aprender a usar ferramentas IAG? - Clarity Relevance 6a. Can you describe your experience of building skills with GAI? Pode descrever sua experiência de desenvolvimento de competências de utilização das ferramentas IAG? Clarity Relevance 16/23 I can design an assessment approach to improve student learning in an AIbased environment (e.g., learning with ChatGPT). [AIA2] (Chiu et al. 2024) I can choose an AI tool to foster student selfassessment. [AIA4] (Chiu et al. 2024) aprendizagem e avaliar processos? 12a. E.g., what GAI tools support monitoring students’ learning, to provide real-time feedback and personalized learning to students, or to foster students’ selfassessment? E.g., Que ferramentas de GAI apoiam a monitorização da aprendizagem dos alunos, para fornecer feedback em tempo real e aprendizagem personalizada aos alunos, ou para fomentar a autoavaliação dos alunos? Clarity Relevance 12b. What challenges or limitations can happen when using GAI tools this way? E.g., could it be relevant to know how to use Large Language Models (LLM) to fill in a draft of students’ assessment report? Que desafios ou limitações podem surgir ao usar ferramentas de IAG desta forma? E.g., poderá ser relevante saber como usar LLMs (Modelos de Linguagem de Grande Escala) para preencher um rascunho de relatórios de avaliação dos alunos? Clarity Relevance In teaching my field, I know how to use different GenAI tools for generate quiz and assessments. (Al-Abdullatif 2024) 13. What type of knowledge science teachers need to use GAI tools of teaching and learning to evaluate students’ learning? - Clarity 17/23 Que tipo de conhecimento os professores de ciências precisam para usar as ferramentas de IAG de ensino e aprendizagem para avaliar a aprendizagem dos alunos? Relevance 13a. What GAI tools can work well in this area (e.g., to generate quizzes)? Que ferramentas de IAG podem funcionar bem nesta área (por exemplo, para gerar questionários)? Clarity Relevance 13b. What challenges or limitations can happen when using GAI tools this way? Que desafios ou limitações podem surgir ao usar ferramentas de IAG desta forma? Clarity Relevance Does the scope of the dimension require reformulation? Yes/No Space for comments on whether the set of questions effectively covers the construct. 18/23 Dimension / Objective Original questions (from the baseline studies) Main questions Probing questions Evaluation (Likert scale: fill from 1 - nothing to 5 - totally) Does it require reformulation? (Yes or No) Qualitative comments (to be filled in only if you mentioned "yes" in the previous column, indicating lack of adequacy to the objectives, making suggestions, etc.) GAI Acceptance / To assess perceptions on the acceptance of GAI tools in science education I look forward to use GenAI tools in my teaching. (AlAbdullatif 2024) I intend to use GenAI tools in my future teaching. (AlAbdullatif 2024) I plan to use GenAI tools in my future teaching. (AlAbdullatif 2024) I support the adoption of GenAI tools in higher education. (Al-Abdullatif 2024) 14. What are the perspectives among science teachers and students, parents and school heads about integrating GAI in teaching practice? Quais são as perspetivas entre professores e alunos de ciências, pais e diretores escolares sobre a integração da IAG na prática docente? Clarity Relevance 14a. E.g., how important is it to integrate GAI in science teaching and why? E.g., qual a importância de integrar IAG no ensino das ciências e porquê? Clarity Relevance 14b. Can you illustrate with examples of situations when people revealed reservations or excitement about using these tools in science education? Pode ilustrar com exemplos de situações em que as pessoas revelaram reservas ou entusiasmo Clarity Relevance 19/23 em usar essas ferramentas na educação de ciências? 14c. Should GAI adoption in science education be supported? How? Why? A adoção das IAG no ensino de ciências deve ser apoiada? Como ? Porquê? Clarity Relevance Does the scope of the dimension require reformulation? Yes/No Space for comments on whether the set of questions effectively covers the construct. 20/23 Dimension / Objective Original questions (from the baseline studies) Main questions Probing questions Evaluation (Likert scale: fill from 1 - nothing to 5 - totally) Does it require reformulation? (Yes or No) Qualitative comments (to be filled in only if you mentioned "yes" in the previous column, indicating lack of adequacy to the objectives, making suggestions, etc.) GAI Ethics / To assess perceptions regarding GAI ethical issues in science education I can teach students ethics [AIE1] (Chiu et al. 2024) I teach students how to behave safely and responsibly when learning with AI tools. [AIE4] (Chiu et al. 2024) I can ensure my health and well-being while using AI tools. [AIE3] (Chiu et al. 2024) 15. What topics do you consider important to include when discussing GAI ethics with students and colleagues? Que tópicos considera importantes incluir ao discutir a ética no uso da IAG com os alunos e com colegas? - Clarity Relevance 15a. What kind of guidelines or expectations should be set for responsible and safe GAI use, e.g., to ensure people’s health and wellbeing while using it? Que tipo de diretrizes ou expectativas devem ser estabelecidas para o uso responsável e seguro da IAG, por exemplo, para garantir a saúde e o bem-estar das pessoas ao usá-lo? Clarity Relevance 21/23 I always comply with ethical principles when using AI applications or products. (Al-Abdullatif 2024) I am alert to privacy and information security issues when using AI applications or products. (Al-Abdullatif 2024) I can protect sensitive content from AI tools (e.g., exams, students’ grades and personal data). [AIE2] (Chiu et al. 2024) 16. What kinds of privacy and information security issues are important to consider when using GAI tools in science education? Que tipo de questões de privacidade e segurança da informação são importantes considerar ao usar ferramentas de IAG na educação em ciências? - Clarity Relevance 16a. What steps should be taken to ensure that sensitive materials — like exams, grades, or student data — are protected when using AI tools? Que medidas devem ser tomadas para garantir que materiais confidenciais, como exames, notas ou dados de alunos, sejam protegidos ao usar ferramentas de IAG? Clarity Relevance Does the scope of the dimension require reformulation? Yes/No Space for comments on whether the set of questions effectively covers the construct. 22/23 Closing of the focus group Objectives Information Does it require reformulation? (Yes or No) Qualitative comments (to be filled in only if you mentioned "yes" in the previous column; indicating lack of adequacy to the objectives, making suggestions, etc.) Collect potential unanticipated contributions Are there other training needs in GAI, enabling factors, barriers or other aspects that you wish to address? Deseja abordar outras necessidades de formação em IAG, fatores favoráveis, barreiras ou outros aspetos? Thank respondents for participating and recognize the value of their contribution 23/23 Other Suggestions – point 4 of the "Practical Instructions for Revision" (Space to comment on the sequence of questions and dimensions and any other suggestions or observations that you consider pertinent. Optional) Thank you once again for your time and your contribution to the quality of this research instrument. 1. References Al-Abdullatif, A. M. (2024). Modeling Teachers’ Acceptance of Generative Artificial Intelligence Use in Higher Education: The Role of AI Literacy, Intelligent TPACK, and Perceived Trust. Education Sciences, 14(11), 1209. https://doi.org/10.3390/EDUCSCI14111209 Amado, J. (2014). Manual de investigação qualitativa em educação. Imprensa da Universidade de Coimbra. https://doi.org/http://dx.doi.org/10.14195/978-989-26-0879-2 Chiu, T. K. F., Ahmad, Z., & Çoban, M. (2024). Development and validation of teacher artificial intelligence (AI) competence self-efficacy (TAICS) scale. Education and Information Technologies, 30(5), 6667–6685. https://doi.org/10.1007/S10639-024-13094Z/FIGURES/1 Hosseini, M.-S., Jahanshahlou, F., Akbarzadeh, M. A., Zarei, M., & Vaez-Gharamaleki, Y. (2024). Formulating research questions for evidence-based studies. Journal of Medicine, Surgery, and Public Health, 2, 100046. https://doi.org/10.1016/J.GLMEDI.2023.100046 1. Acknowledgements The GAI-SciTeach project is funded by National Funds through FCT - Foundation for Science and Technology, I.P., within the scope of the project 2023.13203.PEX.