International Perspectives on Educational Tool Design: A User Testing Toolkit Case.

Practice Paper Recommended citation: Figueiredo, S., Kirjavainen, S., Kuikka, M., Ocampo, C. C., Devia, G., Gárate, F., Marín, S. M., Packalén, A., & Riveros, D. (2025). International Perspectives on Educational Tool Design: A User Testing Toolkit Case.. In Kangaslampi, R., Langie, G., Järvinen, H.-M., & Nagy, B. (Eds.), SEFI 53rd Annual Conference. European Society for Engineering Education (SEFI), Tampere, Finland. DOI: 10.5281/zenodo.17631585. This Conference Paper is brought to you for open access by the 53rd Annual Conference of the European Society for Engineering Education (SEFI) at Tampere University in Tampere, Finland. This work is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 4.0 International License. International perspectives on educational tool design: a user testing toolkit case. S Figueiredoa,1, S Kirjavainen a, 2, M Kuikka a,3, and C Comas Ocampo b, G Devia c.1, F Gárate a, S Merino Marín d, A Packalén a, D Riveros c.2 a Aalto Design Factory, Department of Energy and Mechanical Engineering, Aalto University, Espoo, Finland, a,1 0009-0003-9725-3900, a2 0000-0002-8049-6557, a30009-0008-8940-850X b Universidad del Desarrollo, Santiago, Chile c Campus Nova Center for Innovation and Entrepreneurship, Pontificia Universidad Javeriana, Cali, Colombia, c.1 0009-0007-2751-7818 , c.2 0009-0001-9716-6677 d Universidad del Desarrollo, Concepción, Chile Conference Key Areas: Engineer as a social debater – new skills needed?, Building the capacity and strengthening the educational competencies of engineering educators Keywords: User testing, Design education, Engineering pedagogy, Practice-based learning, Teaching resources ABSTRACT Applying user-centered design skills is a critical aspect of engineering practice, yet integrating it into engineering education remains challenging. We have observed a need for straightforward, structured tools that facilitate early-stage engagement in user testing. In response to this need, the User Testing Toolkit was developed at Aalto university as a part of a collaboration of three universities located in Chile, Colombia and Finland. The partnership was set up to produce teaching tools that balance global relevance with local adaptability. The toolkit includes modular templates designed to guide students through the pre-testing, during-testing, and post-testing phases. Additionally, an expansion pack introduces fictional users and products, enabling practicing and refining testing methodologies in diverse scenarios. Piloted in multidisciplinary settings in Finland and Chile, the toolkit provided practical insights not only into student learning but also into how educators adopt and adapt new pedagogical tools. This practice paper presents the User 1 Corresponding Author S Figueiredo [email protected] Testing Toolkit as a concrete example of cross-institutional collaboration and highlights its dual role: enhancing student learning and strengthening the educational competencies of engineering educators in user-centered, practice-based learning contexts. 1 INTRODUCTION User testing is often overlooked or poorly implemented in early-stage student design projects. Novices use prototypes early to define user requirements and understand their needs less frequently than experts, and instead tend to test with users only later in the design process for feedback and validation (Deininger et al, 2017). Students tend to rush to develop design requirements while they should focus on gathering diverse knowledge from a variety of informants, and this should be taken into account in curriculum design (Mohedas et al., 2015) in order to help students develop their ability to apply user centered design skills and knowledge. Especially in the context of project and problem based learning, students need timely and structured guidance and facilitation from the teacher (Doulougeri et al., 2024) and different forms of scaffolding can offer the needed support (Martin et al., 2019), applying also to how and why to test their ideas and concepts with users early on. While students need guidance and support, educators also face challenges in implementing project based learning practices, and supporting students in engaging in user centered design practices. These challenges include the lack of resources and materials supporting the activities (Chen et al., 2021), that the current paper responds to. International university collaboration helps in fostering an interconnected, innovative, and environmentally and socially responsible approach to higher education benefitting the institutions but also society at large (de Wit & Altbach, 2021). Therefore, the current work introduces an international collaboration where materials supporting teaching were co-developed as part of a broader international project. The toolkit introduced here, is designed to ease the educators work and support student learning around user testing. It is aimed to be used in engineering education, when introducing user testing to students as a new topic, or when guiding crossdisciplinary teams in their work. 2 CONTEXT AND PRACTICAL WORK 2.1 Global Collaboration This project was part of an international collaboration between institutions in Colombia, Chile and Finland, focused on developing teaching tools to support engineering and design education. Each partner created one tool, based on their local educational and industrial contexts while working within a shared pedagogical framework. The development process was collaborative and iterative, with partners exchanging feedback and refining each other’s work to ensure alignment across the different toolkits. Although each toolkit was developed independently, the collaboration ensured pedagogical coherence and adaptability to different learning environments, increasing the overall usefulness of the tools and toolkits. Also, while the tools can be used independent from each other, they can be used across the learning journey, forming a continuum of support. This model also reflected a shared objective of preparing students to work across diverse, international, and interdisciplinary settings. The User Testing Toolkit, developed by the Finnish partner, is one example of this collaboration. While it focused specifically on user testing, its structure and adaptability were informed by insights and feedback gathered across the partnership. 2.2 User testing toolkit educational context The User Testing Toolkit was developed by one of the partner institutions to be used in multidisciplinary project courses such as product development and innovation projects. These courses bring together students from engineering, design, business, and other fields, often working in teams with varying levels of experience in usercentered design and user research. In these types of courses students often postpone user testing until late in the process. Created in response to this common challenge, the toolkit encourages formative testing and offers practical guidance on how to conduct user testing and what to consider at different stages of the process. It also serves as a professional development tool for educators, enabling them to confidently teach user testing methods through a clear, evidence-informed structure. By offering visual guides and session-ready materials, the toolkit lowers the threshold for integrating design education and user research into engineering curricula. It provides educators with a concrete means to support practice-based learning, regardless of their prior experience with design pedagogy or user testing. The toolkit was piloted in the intended use context, to collect feedback from educators and students for further development. The results from this piloting will be shared and discussed in Section 3.2. 2.3 User testing toolkit structure The User Testing Toolkit2 is structured around three key phases of the testing process: Pre-testing, During testing, and After testing (see Figure 1). Each phase includes clearly defined sub-activities and visual templates that guide students through planning, execution, and reflection. The toolkit is designed to be modular and flexible, allowing using individual tools for short sessions, or applying the full set across longer courses and projects. Students with ongoing course projects can apply the toolkit to their projects. For use cases where the toolkit is used without a specific project in mind, an expansion pack is available, including fictional user profiles, company briefs, and physical products (e.g. a citrus squeezer or potato peeler). These allow students to role-play both facilitator and user roles, enabling them to practice key techniques in a pre-designed setting. Overall, the toolkit helps support early-stage, iterative testing and makes the process more accessible for students with limited experience in user research or prototyping. It provides clear, practical guidance on what to do, how to do it, what to consider at each stage of user testing, and how to learn from these activities. 2 For more details on the User Testing Toolkit, see https://designfactory.aalto.fi/toolkits/ First, the pre-testing activities support students in preparing for a testing session. This includes tools that help define testing goals, and in selecting and prioritizing test participants. Additional templates guide students in choosing a testing method, planning logistics, and mapping out the session flow. In the following during testing phase, the developed template offers structured guidance on what to observe and how to engage with users. It includes prompts for spoken and unspoken insights, helping students capture user behaviour, emotions, and feedback. Finally, the after testing phase focuses on making sense of the gathered data. Students are guided through identifying patterns, translating insights into ideas, and prioritising improvements. Final templates support the definition of concrete next steps. Fig. 1. User testing toolkit materials 3 REFLECTION AND INSIGHTS FROM DEVELOPMENT AND PILOTING 3.1 Reflections on international collaboration and teaching tool development This section explores the experiences of the educators involved in the co-creation of teaching tools across Colombia, Chile and Finland. Reflections focus on how working collaboratively with international partners shaped their approach to educational design and practice. Cultural relevance and adaptability were emphasised by educators across the three countries. The Colombian team emphasised that using local examples is fundamental to the effectiveness of a tool because it ensures that content resonates with learners, and bridges the gap between theoretical concepts and real-world applications. They also noted that while global frameworks provide structure, it is important to create flexible enough tools to allow teachers to incorporate diverse learning styles, languages, and contextual needs. The Finnish educators similarly reflected on how the collaboration helped them consider how tools can serve diverse learning environments, particularly through regionally grounded examples that allowed students to understand how user needs, industry, and culture intersect. Intercultural and interdisciplinary learning was a key benefit of the collaboration. The Chilean team described how working with other countries helped them gain perspective on different approaches to educational design, not only from an intercultural standpoint, but also from an interdisciplinary one. Through a series of meeting points among the institutions, they were able to transfer knowledge and new insights into the development of each tool. The Colombian team also reflected on how the collaboration addressed challenges specific to their context, such as conflicting interests among stakeholders and the interaction between the rural and urban world. These challenges sparked creative solutions that enriched the international co-creation process. Collaboration practices and values were consistently highlighted by all three partners. The Colombian educators emphasised the value of open communication, mutual respect, and iterative feedback when working across countries and institutions. Chilean reflections also highlighted this sentiment, noting that an open attitude and embracing different perspectives with curiosity greatly enriched the collaborative process. The Finnish team also emphasised the relevance of creating space for open exchange early in the process not only about content, but also about pedagogical values. Broader educational goals also emerged as a shared point of reflection. The Chilean team pointed out that education must be based on the needs of society, which interrelate in a dynamic process. Reviewing international examples helped broaden their vision of educational innovation, enabling the exchange of practices and testing in different environments. The Finnish team connected these reflections to student learning, noting that as graduates are expected to work across borders, cultures, and disciplines, exposing them to different ways of thinking during their studies is essential. Together, these reflections show that international collaboration not only contributed to the development of the teaching tools, but also functioned as a platform for shared educator learning, where participants exchanged instructional strategies, codeveloped resources, and refined their understanding of engineering pedagogy. 3.2 Reflections on local implementation of the user testing toolkit Fig. 2. Implementation one in Finland Fig. 3. Implementation two in Chile The User Testing Toolkit was piloted in two distinct educational settings: a multidisciplinary product development course in Finland and an international creativity and innovation management course in Chile. Both implementations were time-limited and used fictional products from the Expansion Pack to simulate a predesigned testing environment. Insights from first implementation: The User Testing Toolkit was implemented in a 2.5-hour session in Finland. Students worked with tools from all three phases, pre, during, and post-testing, including the Look Feel Usability model, user testing guide, and Impact vs. Effort matrix. Students tested fictional products like a garlic press or potato peeler, along with predefined user profiles and briefs. This created a shared, low-pressure environment that supported participation across disciplines and experience levels. Educators observed that students quickly grasped the toolkit’s structure and appreciated the visual frameworks. However, some noted the example products were too simple to explore user insights in depth. While a more complex product could offer richer testing opportunities, it would also require more time and readiness to unpack meaningfully. This highlighted a key trade-off, simplified scenarios promote accessibility and fit short sessions, while more nuanced cases demand deeper engagement and time. Insights from second implementation: In Chile, the User Testing Toolkit was implemented in an hour and 10-minute session, using only four templates: two from the pre-test, one during, and one post-testing. Similar to insight from Finland, the students tested fictional products, such as the head and facial massagers. To keep the exercise straightforward, the predefined user profiles were not implemented, as the educators observed that the dynamics were already established among the students and they organically started creating their own profiles. The testing was organised into two cycles, so every student could experience being a tester / user and an interviewer / observer. Educators noted that the toolkit's visual framework made it easier for the students to understand the content, the goal of the session and the tasks, regardless of the language barriers. The products used for the testing were unfamiliar to most students, which was intentional. The intended goal was for students not to be biased by previous experiences they had with the product, and the testing could be purely focused on the interaction between the tester - object - observer during the session. 4 CONCLUSIONS AND IMPLICATIONS This practice paper presents the User Testing Toolkit and reflects on its development and testing through international collaboration.The research gap addressed by this paper is the lack of tools and resources to help educators integrate project-based learning and user-centered design practices into their teaching. Knowing that material scaffolds complemented by educators’ active engagement support learning activities (Martin et al., 2019), the toolkit was designed to enable students to engage with early-stage user feedback, through structured, modular templates that guide students through three testing phases. The paper also highlights the value of international collaboration in developing the tool, with institutions from Colombia, Chile and Finland contributing their perspectives. Feedback from educators and students in diverse cultural and educational settings provided valuable insights into the toolkit's effectiveness and highlighted areas for improvement, ensuring that the toolkit is globally relevant and adaptable to different cultural and educational contexts. For educators, the toolkit offers a practical and adaptable resource to integrate user testing into their courses. It provides a step-by-step guide for students to conduct user testing, making the process more accessible, especially for those with limited previous experience often leading to insufficient testing in early phases (Deininger et al, 2017). This is particularly valuable in project-based and interdisciplinary settings, where students often come from diverse backgrounds and may lack a shared understanding of user-centered design practices. The toolkit’s structure also allows educators to adapt it to different course durations and project types, increasing its usefulness in varied teaching contexts. Additionally, it enhances educators’ instructional competencies in user-centered design, especially for those from engineering backgrounds less familiar with participatory and qualitative research methods.The toolkit can thus be seen not only as a student-facing learning resource, but also as a tool that supports the professional development of teaching staff in engineering and design education. Intentional iterations in early phases of development projects create value through, for example, building knowledge and clarifying needs and goals (Wynn & Eckert, 2017). For students, the presented toolkit offers a hands-on approach to learning user-centered design, and encourages early engagement with users, thus helping students gather valuable feedback and refine their designs through iterations. The toolkit also supports interdisciplinary collaboration, enabling students from diverse fields to work together effectively. The opportunity to apply these concepts to both real and fictional products also allows students to build confidence in their user research skills, making them more prepared for their future careers. From an industry perspective, the toolkit bridges the gap between academic learning and professional practice by providing students with the skills to conduct meaningful user testing and design products that better meet user needs. As industry increasingly demands engineers who can integrate user feedback throughout development, the toolkit provides students with a strong foundation to meet these expectations. Moving forward, the toolkit holds the potential for further refinement and adaptation in a wider range of educational settings. Continued testing in diverse environments will strengthen its effectiveness and ensure its broader applicability. Ultimately, the toolkit contributes to preparing students for the collaborative, interdisciplinary, and user-focused work required in the global engineering landscape. 5 ACKNOWLEDGEMENTS We thank the Aalto Design Factory research team, our partner institutions in Chile and Colombia, and the students who participated in the pilot sessions. This work was developed as part of a TFK funded initiative, supporting international collaboration in engineering education. REFERENCES Chen J., Kolmos, A., & Du, X. (2021). Forms of implementation and challenges of PBL in engineering education: a review of literature. European Journal of Engineering Education, 46(1), 90-115, https://doi.org/10.1080/03043797.2020.1718615 Deininger, M., Daly, S. R., Sienko, K. H., & Lee, J. C. (2017). Novice designers' use of prototypes in engineering design. Design studies, 51, 25-65. https://doi.org/10.1016/j.destud.2017.04.002 De Wit H., & Altbach, P.G. (2021). Internationalization in higher education: global trends and recommendations for its future. Policy Reviews in Higher Education, 5(1), 28-46. https://doi.org/10.1163/9789004462717_016 Doulougeri, K., Vermunt, J. D., Bombaerts, G., & Bots, M. (2024). Challenge-based learning implementation in engineering education: A systematic literature review. Journal of Engineering Education, 113(4), 1076-1106. https://doi.org/10.1002/jee.20588 Martin, N. D., Dornfeld Tissenbaum, C., Gnesdilow, D., & Puntambekar, S. (2019). Fading distributed scaffolds: The importance of complementarity between teacher and material scaffolds. Instructional Science, 47(1), 69-98. https://doi.org/10.1007/s11251-018-9474-0 Mohedas, I., Daly, S. R., & Sienko, K. H. (2015). Requirements development: Approaches and behaviors of novice designers. Journal of Mechanical Design, 137(7), 071407. https://doi.org/10.1115/1.4030058 Wynn, D. C., & Eckert, C. M. (2016). Perspectives on iteration in design and development. Research in Engineering Design, 28(2), 153–184. https://doi.org/10.1007/s00163-016-0226-3