General Engineering Non-Technical Skills Behavioral Marking System: Video-Simulation Validation Workshop

Workshop Recommended citation: Jaberi, A., Lucas, C., & Ciriello, F. (2025). General Engineering Non-Technical Skills Behavioral Marking System: Video-Simulation Validation Workshop. In Kangaslampi, R., Langie, G., Järvinen, H.-M., & Nagy, B. (Eds.), SEFI 53rd Annual Conference. European Society for Engineering Education (SEFI), Tampere, Finland. DOI: 10.5281/zenodo.17631345. This Conference Paper is brought to you for open access by the 53rd Annual Conference of the European Society for Engineering Education (SEFI) at Tampere University in Tampere, Finland. This work is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 4.0 International License. GENERAL ENGINEERING NON-TECHNICAL SKILLS BEHAVIORAL MARKING SYSTEM: VIDEO-SIMULATION VALIDATION WORKSHOP A. Jaberi a, C. Lucas b, 1 , F. Ciriello c, a King’s College London, London, United Kingdom, 0009-0007-5279-805X b King’s College London, London, United Kingdom, 0000-0003-2284-272X c King’s College London, London, United Kingdom, 0000-0003-1288-3114 Conference Key Areas: Engineering skills, professional skills, and transversal skills, Engineer as a social debater – new skills needed? Keywords: Engineering Non-Technical Skills, Behavioral Marker System, Observational Assessment, Inter-Rater Reliability ABSTRACT This workshop aims to validate and refine the General Engineering Non-technical Skills Behavioural Marking System (GENTS-BMS). Developed from a doctoral research , through literature reviews, policy documents, and expert interviews, GENTS-BMS assesses UK engineers’ non-technical skills through observable behaviours. The workshop will engage participants in evaluating video scenarios of workplace situations showcasing high-performing engineering behaviours in categories including teamwork, contextual awareness, sustainability, engineering management, and communication. The primary objectives are to assess the system’s reliability through Inter-Rater Reliability (IRR) analysis and gather qualitative insights on its practical utility and behavioural marker distinctiveness. Participants will receive training on the GENTS-BMS framework before evaluating video clips using provided behavioural markers and a binary assessment scale. Their feedback will be documented to enhance GENTS-BMS’s effectiveness as an assessment tool. The workshop aims to strengthen the system’s empirical foundation through expert evaluation and feedback. 1 Corresponding Author C. Lucas [email protected] 1 BACKGROUND AND RATIONALE A Behavioral Marker System (BMS) is an observational framework designed to assess non-technical skills (NTS) by identifying specific and observable behaviours (Hamlet et al. 2023; Butler 2021). Behavioural marking systems are used across various industries and professions to support the observation, evaluation, and training of key non-technical skills (Irwin, Tone, and Sedlar 2023; Hamlet et al. 2023). Behavioural marking systems provide a structured method for measuring performance, offering feedback, and guiding individual, team, and organisational improvement (Tefera 2019; Ravindran et al. 2021; Crichton, Moffat, and Crichton 2017). Behavioural marking systems are context-specific and developed through a rigorous research process including but not limited to a review of existing literature to identify relevant NTS and behavioural markers (Ravindran et al. 2021; O’Connor and Max Long 2011), using methods like cognitive task analysis, interviews, and focus groups to identify critical skills and behaviours, developing a taxonomy that categorises skills and elements with specific, observable behaviours, engaging subject matter experts to review and refine the system, and Identifying and define specific behavioural markers that indicate good and poor performance (da ConceiçÌo et al. 2017; Ravindran et al. 2021; Irwin, Tone, and Sedlar 2023). Building on research conducted within the Engineering Department at King’s College London, our doctoral project has developed a behavioural taxonomy of non-technical skills for UK engineers. This taxonomy emerged from a review of 93 academic publications, policy documents from the Royal Academy of Engineering, UK Engineering Council, and surveys from the Institution of Engineering and Technology (IET). We also conducted 18 semi-structured interviews with 14 expert engineers across the UK, employing knowledge elicitation techniques. Thematical and content analysis of the interviews extracted behavioural indicators that distinguish between good and poor performance across NTS categories for UK engineers. We are now entering the crucial validation phase of our research. This final stage aims to establish the observability and utility of our behavioural marking system by evaluating video scenarios. We will also assess whether these video clips are suitable for training future assessors. The scenarios simulate workplace situations that challenge engineers’ non-technical skills and showcase the identified highperforming behaviours across skill categories. We are seeking attendees to report if they can observe poor or good performance behaviours in video clips and help us strengthen the empirical foundation of our behavioural marking system. 2 WORKSHOP OBJECTIVES Through structured engagement, this workshop validates and refines the General Engineering Non-technical Skills Behavioural Marking System (GENTS-BMS). First, we aim to assess the system’s reliability through Inter-Rater Reliability (IRR) analysis, establishing quantitative measures of consistency across evaluations. This statistical validation is crucial for establishing the robustness of behavioural markers (Thompson 2013). Beyond statistical validation, we seek to gather qualitative insights from observers regarding the practical utility and effectiveness of the observation process. This feedback will inform potential refinements to the system’s implementation in real-world settings. We also examine the distinctiveness of individual behavioural markers, ensuring they effectively capture discrete competencies without overlap. Throughout the workshop, we will document participant feedback, including specific comments, queries, and observations about the marking system. This feedback will provide valuable insights for enhancing the effectiveness of the GENTS-BMS framework as an assessment tool. 2.1 Target audience The target audience may include: • Engineering lecturers interested in assessing and training engineering students’ non-technical skills in learning by doing and project-based learning environments. • Industry leaders interested in upskilling newly recruited engineers on nontechnical skills via training and performance evaluation. • Engineering education policymakers interested in incorporating assessing and training non-technical skills in higher education accreditation policies. • Educators and curriculum developers interested in the use of observationalbased assessments. 2.2 Expected learning outcomes Upon completion of this workshop, participants are expected to: • Develop practical experience applying the GENTS-BMS framework by evaluating video scenarios depicting engineering workplace situations. • Contribute to the validation of the GENTS-BMS by providing evaluations of the observability, utility, and distinctiveness of the identified behavioural markers. • Gain insights into the process of developing and validating a behavioural marking system. • Have the opportunity to share their perspectives and feedback on the GENTS-BMS framework, contributing to its refinement for potential real-world implementation. 3 WORKSHOP DESIGN 3.1 Time plan Participants will receive a short training in behavioural observation using the GENTS-BMS framework. Observers will then evaluate a sample of good and poor performance behaviours shown in video clips using the behavioural markers in GENTS-BMS and contribute to a round table. To ensure clarity of purpose and effective engagement with the material, the workshop will follow a structured format: • Welcome and Workshop Overview: Introducing the workshop’s purpose as validating a BMS for assessing and training UK engineers’ NTS using simulations. • Introduction to Behavioral Marking Systems (BMS): Defining a BMS as a structured tool for identifying, rating, and training observable NTS behaviours that contribute to excellent or substandard performance, providing a common language and framework for discussion and feedback (Mitchell et al. 2012). • System Overview: Introducing the specific BMS for UK engineers (GENTSBMS), outlining its categories (e.g., contextual awareness, decision-making, engineering management, communication, teamwork, leadership) and the 2- point rating scale (‘poor’ or ‘good’ with ‘not applicable’). Participants will receive a summary sheet. • Guided Practice Session: Participants will independently complete a practice rating sheet. A group discussion of observations and initial ratings will follow. • Independent Rating: Participants will watch up to 3 experimental simulation videos (2-3 minutes each) and rate the observed NTS independently and without discussion (Flin et al. 2006). The marking sheets will be collected at the end of this part. • Overall Impressions and Challenges: Facilitating a discussion on participants’ experiences rating NTS, focusing on ease and challenges, as well as clarity of the system. The interactive design elements, workshop plans, and timing are summarised in Table 1. Table 1. Time plan Activity Delivery Method Time (mins) Cumulative Time (mins) Participant Engagement Introduction and Rationale Welcome, verbal presentation with slides 5 5 Participants listen and reflect BMS Introduction and Training Presentation, handbook distribution 5 10 Participants receive structured material and definitions to support understanding Guided Practice Video Video playback (2–3 mins), independent rating, group discussion 15 25 Active engagement through watching, rating, and sharing interpretations in discussion Experimental Video Rating Sessions Video playback, independent rating 20 45 Participants focus on observation and apply GENTS-BMS independently without influence from others Discussion and Feedback Group discussion led by facilitators 15 60 Participants reflect on the challenges of Observing behaviours and suggest improvements 3.2 Interactivity The workshop format is designed to be highly interactive, providing a dynamic environment beyond the scope of a static research paper. Participants will actively engage in the validation process by observing and evaluating video scenarios showcasing engineering non-technical skills using the provided GENTS-BMS framework. This direct application of the system allows for immediate feedback on the clarity and applicability of the behavioural markers. Furthermore, including a round table discussion provides a crucial opportunity for attendees to share their insights, queries, and observations about the marking system. 3.3 Video Content Generation Real-world cases illustrating good and poor non-technical skills (NTS) were extracted from a thematic analysis of semi-structured interviews with 14 expert engineers in the UK, providing rich narratives and critical cases that grounded the simulations in professional experiences. These insights informed the development of loosely scripted scenarios highlighting specific NTS elements shortlisted for the behavioural marking system. To enhance the video clip production process, AI was employed to generate detailed line-by-line transcripts for the actors of the video clips, which were then carefully refined and validated by the research team to ensure accuracy and completeness. 4 WORKSHOP RESULTS The validation workshop utilised a structured survey instrument to gather quantitative and qualitative feedback on the General Engineering Non-technical Skills Behavioural Marking System from ten participants. This section summarises the results related to the framework’s scope, clarity of definitions, and the appropriateness of the observation and rating scales used. 4.1 Scope and Comprehensiveness Assessment of the framework’s coverage revealed mixed feedback regarding potential omissions. Six out of ten participants indicated that a category or element was omitted, while two stated the coverage was adequate. Suggested additions included skills such as “Systems Thinking” and “Technical Skills”. Furthermore, specific professional domains, such as “DEI, Inclusion,” and a greater focus on “Reflection and growth,” encompassing self-learning and career planning, were recommended for incorporation. Conversely, when asked regarding unnecessary inclusions, there was unanimous agreement: all ten respondents confirmed that no included category or element was considered irrelevant for a graduate-level engineer. 4.2 Utility of Behavioural Markers The categories of skills and elements were deemed clearly defined by seven respondents, with two disagreeing. Analysis of the comments revealed that the marking system was regarded as “clear and logical,” with differentiation notably enhanced by the inclusion of both positive and negative behavioural examples. However, some conceptual overlap was noted, for instance, goal-setting under both “teamwork” and “leadership,” while task organisation was categorised with “task management.” The behavioural markers were uniformly validated for utility. All ten participants found the ‘good practice’ behavioural markers useful, and nine found the ‘poor practice’ markers useful. Challenges arose from implementation constraints: observers noted that there was insufficient time to fully absorb the markers before attempting the observational tasks. 4.3 Rating Scale Usability and Cognitive Load Nine participants found the rating scale clearly defined, but concerns were raised regarding its practical application, particularly when attempting to rate multiple categories simultaneously. The five-point rating scale (1: Poor – 5: Very good, plus N/A) was rejected by a majority of respondents (six ‘no’ responses, zero ‘yes’ responses). Participants characterised the scale as either too broad or too granular, leading to “fuzzy” divisions that were challenging to identify during a single observation run-through. This difficulty was explicitly linked to high cognitive load, as observers struggled to hold five distinct behavioural descriptions in mind while assessing performance. In response to the difficulty of using the five-point scale, six participants indicated a preference for a simpler, two-point rating scale (competent/not competent). Multiple respondents proposed that a three-point scale would be ideal, offering sufficient nuance without imposing the cognitive load associated with a five-point scale. This preference for reduced complexity reflects a consensus that a 2or 3-point scale would be more effective than the current five-point system. 4.4 Future Course of Actions and Revisions The most critical course of action is revising the rating scale, as the current five-point scale was largely rejected. Future steps will involve piloting and implementing a simpler scale, ideally a three-point scale, which was suggested by multiple respondents as a compromise that provides necessary nuance without the cognitive load associated with holding five descriptions in mind. This would address the difficulty observers had when simultaneously rating multiple categories The framework’s structure and utility also require refinement: 1. Reducing Overlap: Addressing the perceived conceptual overlap, such as the differentiation between goal setting in “teamwork” and “leadership” categories. 2. Increase Scaffolding: To mitigate the reported high cognitive load, future trainings and calibration will include greater scaffolding, ensuring sufficient time for observers to absorb the handbook and markers before attempting observational assessment tasks. REFERENCES Butler, Philip Carl. 2021. “Development and Evaluation of a Behavioural Marker System for UK Fire and Rescue Service Incident Commanders.” ConceiçÌo, Victor Fernando PlÁcido da, JoÌo Cruz Basso, Custodio Lopes, and Joakim Dahlman. 2017. “Development of a Behavioural Marker System for Rating Cadet’s Non-Technical Skills.” TransNav, the International Journal on Marine Navigation and Safety of Sea Transportation 11 (2): 69–76. https://doi.org/10.12716/1001.11.02.07. Crichton, Margaret T., Scott Moffat, and Lauren Crichton. 2017. “Developing a Team Behavioural Marker Framework Using Observations of SimulatorBased Exercises to Improve Team Effectiveness: A Drilling Team Case Study.” Simulation and Gaming 48 (3): 299–313. https://doi.org/10.1177/1046878117693266. Flin, R, S Yule, S Paterson-Brown, N Maran, D Rowley, and G Youngson. 2006. “Experimental Evaluation of a Behavioural Marker System for Surgeons’ Non-Technical Skills (NOTSS).” Proceedings of the Human Factors and Ergonomics Society Annual Meeting 50 (10): 969–72. https://doi.org/10.1177/154193120605001036. Flin, R, Lucy Mitchell, Kathy Coutts, George Youngson, and Janet Mitchell. 2009. “Education for Scotland Scrub Practitioners’ List of Intraoperative Non-Technical Skills (SPLINTS) Structuring Observation, Rating and Feedback of Scrub Practitioners’ Behaviours in the Operating Theatre.” www.abdn.ac.uk/iprc/splints. Hamlet, Oliver E.D., Amy Irwin, Rhona Flin, and Nejc Sedlar. 2023. “The Development of the Helicopter Non-Technical Skills (HeliNOTS) Behavioural Marker Systems.” Ergonomics 66 (12): 2232–41. https://doi.org/10.1080/00140139.2023.2194591. Irwin, A., I. R. Tone, and N. Sedlar. 2023. “Developing a Prototype Behavioural Marker System for Farmer Non-Technical Skills (FLINTS).” Journal of Agromedicine 28 (2): 199–207. https://doi.org/10.1080/1059924X.2022.2089420. Mitchell, Lucy, Rhona Flin, Steven Yule, Janet Mitchell, Kathy Coutts, and George Youngson. 2012. “Evaluation of the Scrub Practitioners’ List of Intraoperative Non-Technical Skills (SPLINTS) System.” International Journal of Nursing Studies 49 (2): 201–11. https://doi.org/https://doi.org/10.1016/j.ijnurstu.2011.08.012. O’Connor, Paul, and W Max Long. 2011. “The Development of a Prototype Behavioral Marker System for US Navy Officers of the Deck.” Safety Science 49 (10): 1381–87. https://doi.org/https://doi.org/10.1016/j.ssci.2011.05.009. Ravindran, Srivathsan, Adam Haycock, Katherine Woolf, and Siwan ThomasGibson. 2021. “Development and Impact of an Endoscopic Non-Technical Skills (ENTS) Behavioural Marker System.” BMJ Simulation and Technology Enhanced Learning 7 (1): 17–25. https://doi.org/10.1136/bmjstel-2019-000526. Tefera, Tegegne. 2019. “Behavioural Marker Systems for Evaluation of Norwegian Merchant Shipping Bridge Resource Management Training.” Thompson, D J. 2013. Behavioural Markers of Air Traffic Controller Development (Doctoral). AI USE AI was used to correct potential grammar mistakes in the first author’s writing and helped provide an initial abstract, which was subsequently edited by the first author.