Adapting Objective Structured Practical Examinations for Summative Assessment in Engineering and Science: Evaluating Practical and Professional Skills

Practice Paper Recommended citation: Rieckmann, L. M., & Baumann, M. (2025). Adapting Objective Structured Practical Examinations for Summative Assessment in Engineering and Science: Evaluating Practical and Professional Skills. In Kangaslampi, R., Langie, G., Järvinen, H.-M., & Nagy, B. (Eds.), SEFI 53rd Annual Conference. European Society for Engineering Education (SEFI), Tampere, Finland. DOI: 10.5281/zenodo.17631776. This Conference Paper is brought to you for open access by the 53rd Annual Conference of the European Society for Engineering Education (SEFI) at Tampere University in Tampere, Finland. This work is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 4.0 International License. ADAPTING OBJECTIVE STRUCTURED PRACTICAL EXAMINATIONS FOR SUMMATIVE ASSESSMENT IN ENGINEERING AND SCIENCE: EVALUATING PRACTICAL AND PROFESSIONAL SKILLS LM Rieckmann a, M Baumann b, 1 a Institute of Applied Medical Engineering at RWTH Aachen University, Aachen, Germany, 0009-0003-2123-0147 b Institute of Applied Medical Engineering at RWTH Aachen University, Aachen, Germany, 0000-0002-0360-3005 Conference Key Areas: Engineering skills, professional skills, and transversal skills; Engineer as a social debater – new skills needed? Keywords: OSPE (Objective Structured Practical Examination), assessment, professional skills, competence orientation, STEM ABSTRACT Engineering graduates often struggle to transition from education to practice, since they do not meet employers’ expectations regarding professional skills, such as communication and problem-solving. In this respect, engineering practice and education remain misaligned, as traditional summative assessments in higher engineering education typically fail to capture the full spectrum of professional skills on an individual level. To address these limitations, we developed a comprehensive summative assessment that tests professional and practical skills alongside knowledge by adapting the format Objective Structured Practical Examination (OSPE) from healthcare education to engineering and science education. OSPE consists of a series of examination stations at each of which the examinee is required to perform process-based tasks within realistic simulated scenarios. This practice paper presents the development and the ongoing implementation of an OSPE combined with written tasks. Using an evidence-based, four-stage approach, we first completed the development process, including blueprinting, defining standard settings, creating a standardised evaluation tool and a standardised station writing template, and designing OSPE stations and written tasks. In the second 1 Corresponding Author M Baumann [email protected] stage, the designed stations and written tasks were piloted by a mock test, refined, and finalised accordingly. While the final implementation stages of administration and evaluation are still ongoing, our preliminary insights identified the following crucial steps: blueprinting, designing whole-task stations, ensuring feasibility, and recruiting human resources. Although the educational impact of the new assessment format remains to be validated, our four-stage approach offers a development process and standardised tools to transfer OSPEs in other courses within engineering education. 1 INTRODUCTION Engineers require technical knowledge and professional skills to succeed in practice. Communication, collaboration and problem-solving were rated as the most important professional skills by 6,063 practising engineers, 7,934 alumni of undergraduate engineering programs, and 432 engineering faculty members. Engineers spend an average of up to 60 % of their working time communicating (Passow & Passow, 2017). However, several studies have reported that many engineering graduates do not meet employers’ expectations regarding proficiency in these skills (Hirudayaraj et al., 2021; Nair et al., 2009). Consequently, education and practice are misaligned in engineering, resulting in difficulties for graduates in transitioning from education to practice (Flening et al., 2022). Therefore, integrating professional skills into engineering education remains an ongoing challenge. To equip engineers with professional skills during their education, numerous efforts have been made to introduce competency-based engineering education (Winberg et al., 2020). As learning is influenced by assessment (Kromann et al., 2009), practices have been developed for assessing professional skills. One approach was to develop an assessment tool that evaluates problem-solving skills (Garay-Rondero et al., 2024). Other assessment formats have shifted their focus from contentto process-based tasks. For example, group discussions are evaluated after an unsolved engineering scenario is presented to students (Ater Kranov et al., 2008), and Constructed Response Tasks associated with engineering scenarios have been introduced in written exams (Pearce, 2015). However, written exams and group discussions fail to assess practical skills. Group discussions only assess individual students to a limited extent and in a single scenario. Therefore, these approaches fail to assess the full range of each student’s performance and consequently do not meet the curricular requirements of summative assessments. To overcome these limitations, a new format is required that can comprehensively assess knowledge, professional and practical skills on an individual level across various scenarios. One such practice is the assessment format Objective Structured Clinical Examination (OSCE), a widely adopted format in healthcare education. OSCEs assess performance in a process-based manner across the cognitive, the affective, and the psychomotor domain, and up to the “shows how” level according (acc.) to Miller's pyramid (Majumder et al., 2019). An OSCE consists of a series of stations through which the examinees rotate (Khan, et al., 2013). A typical OSCE station requires the examinee to perform tasks in a clinical scenario, for example (e.g.) an anamnesis. Examiners rate the examinee's performance objectively, using a checklist (Agarwal et al., 2010; Bogo et al., 2012). OSCEs have been shown to improve the depth of learning, enhance the retention and lead to a positive educational impact (Abdelaziz et al., 2016; Ataro et al., 2020; Bashir et al., 2016; Ha & Lim, 2023; Jacobson et al., 2018; Khan, et al., 2013). An adapted format, the Objective Structured Practical Examination (OSPE) has emerged within healthcare education, which incorporates non-clinical simulated scenarios (Casey et al., 2009; Khan, et al., 2013). Despite its high potential for higher education, OSPEs are not yet widely used outside of healthcare education. In engineering education, the use of OSPE-derived formats has only been reported sparsely. Once, it was conducted as a formative assessment to assess basic electrical engineering skills. The topics covered by the OSPE stations of this assessment included e.g. “construction of electronic circuit on breadboard from a circuit diagram”, and “adjust oscilloscope settings to display properly an unknown signal”. It was reported that 93.5 % of the students and 87.9 % of the examiners perceived this assessment as beneficial, e.g. to gain confidence in practical skills (Alinier & Alinier, 2006; Guillaume Alinier, 2005). Furthermore, an OSPE-derived format within a module called control technology encouraged biomedical engineering students to work more actively and continuously throughout the semester (Gräf et al., 2018). To our knowledge, neither of them assessed professional skills. To assess practical and professional skills alongside knowledge on an individual level in a process-based manner, we adapted the OSPE format for STEM students. 2 CONTEXT AND PRACTICAL WORK 2.1 Course Context The assessment is implemented in an introductory medicine course at the RWTH Aachen University, Germany. This two-semester course is part of engineering degree programmes, e.g. mechanical engineering and electrical engineering, and natural science degree programmes, e.g. mathematics and information technology. The main course objective is to lay the foundations for applying one’s own discipline in the medical field. Students learn medical knowledge through lectures and train practical and professional skills through interactive workshops and project-based, problem-oriented group work. These focus mainly on training in communication, teamwork, problem-solving, and scientific presentation. However, these course objectives were not reflected by the previous written content-based exam with openended questions. This ongoing project introduces a new process-based assessment format by adapting the OSPE structure to engineering and science education. 2.2 Practical Work Stage 1: Development of the assessment To ensure Constructive Alignment (Biggs, 1996) and content validity (Sireci, 1998), we conducted systematic blueprinting, adapting steps from Ismail et al., 2020; Raymond & Grande, 2019. The intended learning outcomes of this course and their behavioural objectives were defined and weighted acc. to their teaching frequency and criticality to calculate each one’s contribution to the assessment. The behavioural objectives were categorised to the levels of Miller's pyramid (Miller, 1990), Bloom’s learning domains (Bloom, 1986), and the learning objective taxonomy (Krathwohl, 2002). This revealed that some intended learning outcomes included behavioural objectives up to the level of “shows how” or belonging to the psychomotor domain. These can be appropriately assessed by an OSPE station. Others contained behavioural objectives, primarily at the “knows” level and within the cognitive domain, which can be appropriately assessed by a content-based written task. Furthermore, studies have shown that the reliability of OSCEs increases when combined with multiple-choice questions (Newble & Swanson, 1988; Verhoeven et al., 2000) or various types of written tasks (Wass et al., 2001). Therefore, we combined OSPE stations and written tasks in the form of open-ended questions. The quantity and duration of OSPE stations impact the reliability and feasibility of the assessment. OSCEs consisting of five to ten stations, with each station lasting ten minutes or less, have shown to be reliable, with a cumulative estimate of Cronbach’s alpha over 0.88 (Peng et al., 2025). Since feasibility is a critical factor, we decided on five OSPE stations, each initially lasting six minutes. These were combined with written tasks of an equal duration and quantity. Defining standard settings ensures the reliability and credibility of assessments. For OSCEs, the standard settings are most commonly defined using the Angoff and borderline methods (Gupta, Dewan, & Singh, 2010; Heal et al., 2019). For Angoff methods, experts judge the criteria of a borderline candidate for each station in advance (Angoff, 1984), whereas for borderline methods, examiners judge borderline candidates on the assessment day. Studies have shown that borderline methods are less resource-intensive (Gupta et al., 2010) and more reliable and credible than the Angoff methods (Boursicot et al., 2007; Kramer et al., 2003; Schoonheim-Klein et al., 2009). Of the borderline methods, the borderline regression method is the most precise, as it performs a linear regression on the scores of all examinees at each station (Wood et al., 2006). Moreover, a study reported that a partially compensating model led to more credible results than a fully compensating model (Schoonheim-Klein et al., 2009). Therefore, we adopted a borderline regression method with a partially compensating model, introducing the additional criterion that students must pass one OSPE station and one written task to pass the entire assessment. Our additional criterion may encourage students to learn professional skills when considering the pre-assessment effects on learning (Cilliers et al., 2012). As objectivity and reliability of OSCEs are influenced by the standardisation of the stations (Khan, et al., 2013), we developed a standardised evaluation tool for the OSPE stations, adapting those from Austin et al., 2003; Höfer et al., 2013. The evaluation tool for the stations consists of an analytical checklist and ratings of professional skills and overall performance (see Fig. 1). We defined the scoring for the analytical checklist of each station and the ratings of professional skills and overall performance, which act as guidelines for examiners to increase consistency of examiners' judgment (see Fig. 2). Similarly, a standardised evaluation tool was developed for the written tasks. Another part is the design of the OSPE stations and the written tasks. To ensure standardisation also in this process, we designed a standardised station writing template (see Table 1). One key challenge in writing stations was to create realistic and transferable scenarios for students from a wide range of STEM disciplines, which was required to ensure face validity (Abdelaziz et al., 2016). Furthermore, the scenarios and task demands should reflect whole-task OSCEs, since this enables a positive educational impact (Lafleur et al., 2015). Accordingly, the stations should assess a set of integrated, coordinated practical and professional skills and knowledge taught in the course, preferably from separate sessions. Ideas for the station scenarios were developed by a heterogeneous group of students, health care and STEM professionals during a creative workshop. We designed the stations using these ideas and our station writing template. For example, in one OSPE scenario, the examinee finds himself in the role of an engineer who has developed a patientspecific implant to replace the temporomandibular joint. The examinee instructs a laboratory engineer on the mechanical testing of the implant. Station materials include inter alia the implant, a 3D-model of the patient’s mandible and a natural temporomandibular joint. The tasks at this station are designed to assess the examinee’s communication and problem-solving skills, and medical knowledge on the musculoskeletal and gastrointestinal systems. In the end, each station was described with an average of 3014 words. Stage 2: Mock test of the assessment We piloted a mock test of the developed assessment to test the validity of the assessment and the adequacy of the designed station materials and instructions (Austin et al., 2003). The examinees for the mock test were students who had already passed the course. The examiners and actors for the stations were given written guidelines and had completed a training, as scheduled for the upcoming combined assessment. The mock test was evaluated using surveys that addressed the perceptions of the examinees and the examiners. The mock test revealed that the initial station time was inadequate to fulfil the tasks of the analytical checklist under assessment conditions. However, we had to balance increasing the station time with feasibility. The duration of the station time was increased from six to eight minutes and the time between stations was reduced to ensure a feasible schedule for the assessment day. Furthermore, we modified the tasks while ensuring that they continue to correspond to the demands of a wholetask station (Lafleur et al., 2015). We narrowed the tasks down and added cues for the examiner to use if an examinee does not complete a station’s key task within the prior specified time. Stage 3: Administration of the assessment The ongoing administration of the examination involves establishing a legitimate legal framework and performing several organisational tasks. The preliminary plan allows us to assess 80 students in four cohorts of 20 students each in one day (see Table 2). The timeframe of a single day for the assessment necessitated the design of only five stations, as the scheduling was conducted in a manner that precludes the dissemination of the assessment content between the cohorts. We have allocated the resources, including a venue, station equipment, catering and human resources. The project's progress demonstrated that recruiting human resources was crucial. Twelve examiners, four actors and ten members for the organising team will be required to assess 80 students. It has been reported that the OSPE-derived formats in engineering education required eleven examiners to assess 27 students (Guillaume Alinier, 2005) or twelve examiners to assess 83 students (Gräf et al., 2018). Our administrative plan indicates that it can be applied efficiently with minimal resource demands and may be scaled up for larger courses by running more OSPE circles in parallel or administering the assessment over more than one day. The examiners will be briefed on the general assessment procedure and trained in marking in a one-hour seminar some weeks before the assessment. The seminar will include interactive practice in assessing sample examinee performance. Furthermore, each examiner will receive written guidelines and will undergo training on the specific content and scoring definitions of the station they are assessing. These trainings will be conducted on an individual basis in 30-minute sessions. Furthermore, there will be briefings for the actors on the script of their performance at the stations. As the examinees move from one OSPE station to another in several parallel cycles, these standardisation measures ensure the reliability of the assessment. Stage 4: Evaluation of the assessment The quality criteria will be evaluated using perception-based and empirical methods. Firstly, the examinees will be surveyed to obtain their perception of the assessment, providing inter alia insights into the assessment’s validity and objectivity. Second, the examinees’ scores will be analysed statistically to evaluate the reliability with metrics such as Cronbach’s alpha. These findings will validate our development process and will be used to improve the assessment continuously. Furthermore, the educational impact of the combined examination will be evaluated. A case-control study will compare learning behaviour and knowledge retention between examinees who completed the combined assessment and those who completed the previous written examination. A second case-control study will investigate long-term knowledge transfer and professional skills application. This later study will survey and compare graduates from the two groups five years postassessment to determine the extent to which the format OSPE supports their transition from education to engineering practice. 3 RESULTS AND INSIGHTS A total of 250 students are enrolled in the course, and it is expected that 80 students will be assessed. The assessment is one day long, acc. to the timetable shown in Table 2. A station writing template (see Table 1) and an evaluation tool (see Fig. 1 and Fig. 2) were developed. 3.1 Tables Table 1. Structure of the station writing template for the OSPE station Section Content of the section Blueprint to the OSPE station Specification of Intended Learning Outcome, behavioural objectives, categories of examined level, learning domains and taxonomies (Bloom, 1986; Krathwohl, 2002; Miller, 1990) in tabular format Description of the scenario Description of the scenario which examinees receive during their reading time Instructions for the examinee Clear instructions, including tasks that need to be conducted and fulfilled, which the examinees receive during their reading time Clear instructions for the examiner Description of the examiner's role, details on cues permitted during the assessment, standardised wording, e.g. when greeting, and answering questions not permitted to the examinee Evaluation tool Specified for the station using the template (see Fig. 1) Scoring definitions for the evaluation Clear guidelines for the evaluation through scoring definitions for the analytical checklist, global rating, and evaluation of overall performance (see Fig. 2) Section Content of the section Clear instructions for the actor (if applicable) Brief description of the actor's role, standardised wording for the conversation with the examinee List of required materials Detailed list of all required materials for reading and examination time, and backup materials Table 2. Preliminary planning of the assessment day; students are assessed in cohorts consisting of 20 students each; the OSPE is conducted in two parallel running circles Time Schedule 07:00-08:00 Last preparations, arrival of examiners, actors and the catering 08:00-08:20 Briefing of the 1st and 2nd cohorts 08:20-08:30 Room changes for the 1st and 2nd cohorts 08:30-10:30 OSPE for the 1st cohort, written exam tasks for the 2nd cohort 10:30-10:40 Room changes for the 1st and 2nd cohorts 10:40-12:40 OSPE for the 2nd cohort, written exam tasks for the 1st cohort 12:40-13:30 Lunch break for examiners and actors 12:40-13:10 Debriefing of the 1st and 2nd cohorts separated from the briefing of the 3rd and 4th cohorts 13:00-13:20 Briefing of the 3rd and 4th cohorts 13:20-13:30 Room changes for the 3rd and 4th cohorts 13:30-15:30 OSPE for the 3rd cohort, written exam tasks for the 4th cohort 15:30-15:40 Room changes for the 3rd and 4th cohorts 15:40-17:40 OSPE of the 4th cohort, written exam tasks for the 3rd cohort 17:40-18:00 Debriefing of the 3rd and 4th cohorts 17:40-18:00 Farewell and thank you to the examiners, actors and the organising team 3.2 Figures Fig. 1. Template of the developed evaluation tool; for each OSPE station with 10 process-based tasks listed in the analytical checklist, and two of the four professional skills (communication, presentation skills, empathy, problemsolving skills) in the global rating scale; the remaining professional skills are deleted from the template. Fig. 2. Scoring definitions for the ratings of professional skills and overall performance; these applied for all OSPE stations, the scorings for the analytical checklists were defined for each OSPE station 4 CONCLUSIONS AND IMPLICATIONS This practice paper described the four stages of the ongoing implementation of a novel, comprehensive summative assessment. We have successfully adapted the format OSPE to assess not only knowledge, but also professional and practical skills in science and engineering education. We applied an evidence-based approach, drawing on assessment practices in healthcare education. The final format of assessment was determined through systematic blueprinting and consisted of equal portions of OSPE and written tasks. While this distribution suits this course context, the blueprinting process may lead to different proportions for other courses depending on the categorisation of the behavioural objectives. Although the final stages are still ongoing, our preliminary insights have identified further crucial steps for successful implementation: designing whole-task stations, recruiting human resources and ensuring feasibility. Our administrative plan indicates that our implementation process can be applied efficiently with minimal resource demands. Although the educational impact of the new assessment remains to be validated, our four-stage approach offers a development process and standardised tools that will enable teachers to transfer OSPEs to other engineering courses. We have presented an assessment practice that objectively assesses knowledge, practical and professional skills, through process-based tasks at an individual level. This practice may support students in developing critical professional skills for succeeding in their engineering practice. 5 ACKNOWLEDGEMENTS Above all, we would like to thank the students who participated in the course and enriched it with their engagement. Thanks are also expressed to the Dean of Studies’ office of the Medical Faculty of Uniklinik RWTH Aachen, Germany, in particular Melanie Simons, Johann Arias, and Henning Schenkat, for their support in administering the assessment day. We would also like to thank the company KLS Martin Group for providing the implant and the model of the patient’s mandible for the OSPE station described here. This project was funded by the Stiftung für Innovation in der Hochschullehre (project no. FR-299/2023).