Decisions, decisions: Methodological approaches in Engineering Education Research

Invited Workshop Recommended citation: Ventura-Medina, E., De Laet, T., de Andrade, M., Chance, S., & Berhard, J. (2025). Decisions, decisions: Methodological approaches in Engineering Education Research. In Kangaslampi, R., Langie, G., Järvinen, H.-M., & Nagy, B. (Eds.), SEFI 53rd Annual Conference. European Society for Engineering Education (SEFI), Tampere, Finland. DOI: 10.5281/zenodo.17631734. This Conference Paper is brought to you for open access by the 53rd Annual Conference of the European Society for Engineering Education (SEFI) at Tampere University in Tampere, Finland. This work is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 4.0 International License. Decisions, decisions: Methodological approaches in Engineering Education Research Esther Ventura-Medina 1 Eindhoven School of Education TU Eindhoven Eindhoven, The Netherlands 0000-0002-1041-945X. Tinne De Laet Leuven Engineering and Science Education Center KU Leuven Leuven, Belgium 0000-0003-0624-3305. Matheus de Andrade Centre for Engineering Education University College London London, UK 0000-0001-5682-0460. Shannon Chance School of Multidisciplinary Technologies TU Dublin Dublin, Ireland 0000-0001-5598-7488. Jonte Benhard Department of Behavioural Sciences and Learning Linköping University Linköping, Sweden 0000-0002-7708-069X. Conference Key Areas: Building the capacity and strengthening the educational competences of engineering educators, Improving Higher Engineering Education through researching engineering education Keywords: Engineering Education Research, Methods, Methodologies 1 Esther Ventura-Medina e.ventura.medi[email protected] ABSTRACT Engineering Education Research is a fast growing field. The goal of the SEFI Special Interest Group (SIG) of Engineering Education Research (EER) is to support the development and maturing of the engineering education research community in Europe and beyond. The EER SIG also aims to support educators and practitioners more broadly with engaging with the EER literature and implementing evidenceinformed practice. This workshop focuses on methods and methodologies for effective engineering education research. The workshop aims to provide a space for researchers and educators to come together and explore research design as a basis for their scholarly work. 1 BACKGROUND AND RATIONALE 1.1 Motivation Engineering education has emerged as a dynamic field of practice and inquiry over the past few decades (Borrego and Bernhard, 2011). As EER gained traction globally with cross-national variations in funding infrastructures and networks of support (Edström et al. 2016; Paretti et al, 2023), it has been argued that the growth of the field depends on navigating contextual priorities, values, and understandings of rigour and quality in research (Borrego and Bernhard, 2011; Riley, 2017). In the European context, EER faces persistent challenges such as varying perceptions of the legitimacy of the field, scarce funding opportunities, and challenges in developing sustainable research capacity (Wint and Nyamapfene, 2022; van Hattum-Janssen et. al, 2015; Edström et. al, 2016). The latter challenge was echoed by feedback from participants of the 2024 EER Special Interest Group workshop, which highlighted a need to support a better understanding of different research methodologies. Although there is a strong tradition of research in science and engineering on positivist approaches, social science research is marked by pluralism in paradigms, acknowledging that the complexity of learning requires a diverse and wide range of approaches in research. In this sense, the transition to EER can be particularly difficult because it involves an epistemological shift, i.e. a reorganisation of one’s beliefs about knowledge and its construction, combined to a transformation in professional values and identity which are affected by institutional constraints and affordances (Dart et. al, 2021). In an exploration of conceptual barriers faced by trained engineers in learning education research methods, Borrego (2007) concluded that technical disciplinary traditions influenced key aspects of the transition to EER such as (i) the framing of research questions and objectives, (ii) the use of theoretical frameworks, (iii) appropriate choice of research methods, (iv) different understandings of measurement, and (v) the need for multi-disciplinary expertise. To address these challenges, this workshop will focus on discussing the framing of effective research questions and their alignment to methodological and analytical approaches in EER. Participants will explore the importance of coherence in research design and execution, which involves choosing from different theoretical frameworks, data collection methods, and analytical approaches. Practical examples (e.g. published papers, reports) will be used to highlight aspects of research design that are relevant to methodology selection and alignment with the research questions. 1.2 Research methodologies in EER When designing research, it is important to consider issues such as how to choose a research question, how to conduct a literature review, and how to ensure that the project is feasible with available resources (Cohen et. al, 2018). To this end, the workshop content is developed around the idea of coherence within research design and the fact that research is an iterative endeavour. Different research designs can be used to guide inquiry on a specific topic, typically falling under quantitative, qualitative, and mixed method approaches. The chosen design approach then provides directions for ensuring the research process is transparent, credible, ethical, and inclusive (Creswell & Creswell, 2017). Methods for data collection and analysis are intrinsically linked to the research objectives, one cannot decouple them without risking being unable to answer the research question. Therefore, it is essential to consider what are the research questions (main and associated) before deciding on the methodological approach (Creswell & Creswell, 2017; Borrego, Douglas & Amelink, 2009). In this sense, no methodology can be seen as preferable because the choices made depend on the formulated research question. In turn, the research questions are also linked to the theory or theoretical framework that serve to focus the inquiry. Quantitative designs are generally used when the research objective is to describe, explain, or predict certain quantifiable empirical features of a population, or to test statistical hypotheses derived from a certain theory. Alternatively, qualitative designs are preferred when exploration, understanding or discovery is the main aim of the research. Finally, mixed approaches can be used to combine separate aspects of the research or to integrate them (Creswell & Creswell, 2017). Ultimately, there is no one-size-fits-all or a correct approach to the research as there are myriads of possibilities within a research design. However, what is important is that the research is coherent and that the theory, research questions and methods are fully aligned with the inquiry and purpose of research. In this workshop, we will discuss the following questions: • What approaches are more suited to what kind of questions? e.g. questions of what, how, why? • What happens when more than one approach is suitable? How to make decisions about approaches and research design? These questions will provide a basis to explore with participants the formulation of research questions, data to be collected including ethical considerations (e.g. surveys, interviews, observations (both videos and in-situ) and analytical approaches (e.g. statistical analysis, Content Analysis, Thematic Analysis, Discourse Analysis) that align with the inquiry aims. 2 WORKSHOP OBJECTIVES The goals of the workshop are to equip participants with the knowledge to make effective choices in relation to research they wish to conduct in engineering education settings. 2.1 Target audience The workshop is aimed primarily at education practitioners interested in assessing the impact of their own teaching activities and more generally to researchers at all levels of experience. 2.2 Expected learning outcomes We will look at the range of research designs, methods and analytical methodologies both for quantitative, qualitative and mixed methods research. Participants will learn about: • Formulating research questions, • Evaluating and selecting research methods in line with research questions Throughout the workshop the participants will be able to: • Reflect upon and consider their own research aims, • Share (and learn from) experiences in research design, • Formulate research questions suitable to different aims, • Evaluate and choose methodologies that are appropriate to the research questions and practicable within their own context • 3 WORKSHOP DESIGN 3.1 Time plan The following set of activities are planned for the session of 1 hour. Table 1. Workshop timeline Run time Activity Notes 15 min - Groups formed (3-5 participants per group). Facilitators to lead. Introductory presentation and orientation to activities 1 and 2. Intro to research design: questions and methodologies. Examples of good alignment and coherence between research questions and approaches. 30 min - Participants in their small groups practice on proposing and selecting methodologies that are coherent with the proposed research questions. Discussion as to suitable methodologies. Identification of ‘coherence’ between questions and research approaches. Based on provided examples, 1-2 research questions per group, participants will develop research design plans (e.g. possible data type, collection mode and analytical approach). Research questions example will be shared in an online board (i.e. Padlet/Miro board) for all to participate and have access. 10 min - Plenary discussion and reflections on coherence in research design as a whole. Based on small group discussions a round of key take-home messages and challenges will be shared in the online board. 5min - Closing and proposals for next year activities, Feedback for EER SIG on future events and topics. Participants to provide feedback and ideas. 3.2 Interactivity The workshop will be facilitated by experienced researchers and practitioners that will support group discussions. Participants will have the opportunity to engage in both plenary and small group discussions. An online board (Padlet/Miro) will be shared for participants to access the examples provided in the introduction and those used in the hands-on activity. The outcomes from the plenary discussion will also be added to the board which will be shared widely. 4 WORKSHOP RESULTS The workshop was very well attended and received by participants with over 30 people actively engaged in both small group and plenary discussions. The participants had a diverse range of experiences in engineering education research (from those starting to engage in scholarly work to very experienced researchers) and roles (e.g. practitioners, research students). After a short introduction depicting the aim, intended outcomes and structure for the workshop, working groups of about 4-5 participants were formed. This was followed by a short introduction to the core topic of the workshop regarding research question formulation, commonly found types and the importance of aligning the questions to appropriate research methods for data collection and analysis in order to provide coherence to the inquiry process. Groups of participants were given the opportunity to practice on selecting methods of data collection for a pre-determined set of questions before a short plenary discussion that considered the answers provided by groups on each type of question. Groups were asked to answer the following; • What kind of data (source/collection method) would be good to collect to answer these questions? • How would this data help in answering the research question (directly or indirectly)? • Is the data sufficient? Is it appropriate? Following this groups were asked to consider the following two questions that linked their selected data collection approach to its subsequent analysis; • How do you gain the insight you need to answer the research question(s)? • Does your analysis answer the research question? This group discussion was followed by a short plenary discussion where some of the salient contributions regarding the link between the data and the analytical approach with emphasis on the importance of good alignment and coherence. Further, participants were encouraged to consider and discuss in their groups how their chosen data collection and analytical approaches would link with theories that can ground the work and further support research coherence. Participants recorded their group discussions in a Padlet https://padlet.com/eventuramedina2/decisions-decisions-eer-sig-workshop-sefi2025-e129ncnngteh1ttd that served as the basis for the content of the plenary discussions and for sharing the knowledge gained in different groups. Small group discussions were supported by the workshop organisers who also engaged in the plenary discussions providing different perspectives. Two key messages were part of the concluding reflections in the final plenary discussion; • there are no right or wrong approaches but it is important to be aware and mindful of the need for coherence in research and, • good research is not about using good methods but rather about selecting approaches that serve well in answering the research questions. This workshop and its results have served to broaden the support provided by the EER SIG to the SEFI community and to strengthen the research capacity in both practitioners and researchers. REFERENCES Borrego, M. (2007), Conceptual Difficulties Experienced by Trained Engineers Learning Educational Research Methods. Journal of Engineering Education, 96: 91102. https://doi.org/10.1002/j.2168-9830.2007.tb00920.x Borrego, M., Douglas, E. P., & Amelink, C. T. (2009). Quantitative, qualitative, and mixed research methods in engineering education. Journal of Engineering education, 98(1), 53-66. Borrego, M. and Bernhard, J. (2011), The Emergence of Engineering Education Research as an Internationally Connected Field of Inquiry. Journal of Engineering Education, 100: 14-47. https://doi.org/10.1002/j.2168-9830.2011.tb00003.x Creswell, J. W., & Creswell, J. D. (2017). Research design: Qualitative, quantitative, and mixed methods approaches. Sage publications. Dart, S., Trad, S., Blackmore, K. (2021). Navigating the path from technical engineering to engineering education research: a conceptual model of the transition process, European Journal of Engineering Education, 46:6, 1076-1091, DOI: 10.1080/03043797.2021.1992609 Edström, K., Kolmos, A., Malmi, L., Bernhard, J., & Andersson, P. (2016). A bottomup strategy for establishment of EER in three Nordic countries – the role of networks. European Journal of Engineering Education, 43(2), 219–234. https://doi.org/10.1080/03043797.2016.1190956 van Hattum-Janssen, N., Oliveira, J., Williams, B. (2015). Engineering Education Research in Portugal, an Emerging Field. International Journal of Engineering Education. 31. 764-684. Paretti, M. C., Case, J. M., Benson, L., Delaine, D. A., Jordan, S., Kajfez, R. L., … Zastavker, Y. V. (2023). Building capacity in engineering education research through collaborative secondary data analysis. Australasian Journal of Engineering Education, 28(1), 8–16. https://doi.org/10.1080/22054952.2023.2214462 Wint, N., & Nyamapfene, A. (2022). The development of engineering education research: a UK based case study. European Journal of Engineering Education, 48(2), 197–220. https://doi.org/10.1080/03043797.2022.2121686