Wo kshop
Recommended ci a ion: Jabe i, A., Lucas, C., & Ci iello, F. (2025). Gene al
Enginee ing Non-Technical Skills Beha io al Ma king Sys em: Video-Simula ion
Valida ion Wo kshop. In Kangaslampi, R., Langie, G., Jä inen, H.-M., & Nagy, B.
(Eds.), SEFI 53 d Annual Con e ence. Eu opean Socie y o Enginee ing
Educa ion (SEFI), Tampe e, Finland. DOI: 10.5281/zenodo.17631345.
This Con e ence Pape is b ough o you o open access by he 53 d Annual Con e ence
o he Eu opean Socie y o Enginee ing Educa ion (SEFI) a Tampe e Uni e si y in
Tampe e, Finland. This wo k is licensed unde a C ea i e Commons
A ibu ion-NonComme cial-Sha e Alike 4.0 In e na ional License.
GENERAL ENGINEERING NON-TECHNICAL SKILLS BEHAVIORAL
MARKING SYSTEM: VIDEO-SIMULATION VALIDATION WORKSHOP
A. Jabe i a, C. Lucas b,
1
, F. Ci iello c,
a King’s College London, London, Uni ed Kingdom, 0009-0007-5279-805X
b King’s College London, London, Uni ed Kingdom, 0000-0003-2284-272X
c King’s College London, London, Uni ed Kingdom, 0000-0003-1288-3114
Con e ence Key A eas: Enginee ing skills, p o essional skills, and ans e sal skills,
Enginee as a social deba e – new skills needed?
Keywo ds: Enginee ing Non-Technical Skills, Beha io al Ma ke Sys em,
Obse a ional Assessmen , In e -Ra e Reliabili y
ABSTRACT
This wo kshop aims o alida e and e ine he Gene al Enginee ing Non- echnical
Skills Beha iou al Ma king Sys em (GENTS-BMS). De eloped om a doc o al
esea ch , h ough li e a u e e iews, policy documen s, and expe in e iews,
GENTS-BMS assesses UK enginee s’ non- echnical skills h ough obse able
beha iou s. The wo kshop will engage pa icipan s in e alua ing ideo scena ios o
wo kplace si ua ions showcasing high-pe o ming enginee ing beha iou s in
ca ego ies including eamwo k, con ex ual awa eness, sus ainabili y, enginee ing
managemen , and communica ion. The p ima y objec i es a e o assess he
sys em’s eliabili y h ough In e -Ra e Reliabili y (IRR) analysis and ga he
quali a i e insigh s on i s p ac ical u ili y and beha iou al ma ke dis inc i eness.
Pa icipan s will ecei e aining on he GENTS-BMS amewo k be o e e alua ing
ideo clips using p o ided beha iou al ma ke s and a bina y assessmen scale.
Thei eedback will be documen ed o enhance GENTS-BMS’s e ec i eness as an
assessmen ool. The wo kshop aims o s eng hen he sys em’s empi ical
ounda ion h ough expe e alua ion and eedback.
1
Co esponding Au ho
C. Lucas
clai [email protected]
1 BACKGROUND AND RATIONALE
A Beha io al Ma ke Sys em (BMS) is an obse a ional amewo k designed o
assess non- echnical skills (NTS) by iden i ying speci ic and obse able beha iou s
(Hamle e al. 2023; Bu le 2021). Beha iou al ma king sys ems a e used ac oss
a ious indus ies and p o essions o suppo he obse a ion, e alua ion, and
aining o key non- echnical skills (I win, Tone, and Sedla 2023; Hamle e al.
2023). Beha iou al ma king sys ems p o ide a s uc u ed me hod o measu ing
pe o mance, o e ing eedback, and guiding indi idual, eam, and o ganisa ional
imp o emen (Te e a 2019; Ra ind an e al. 2021; C ich on, Mo a , and C ich on
2017).
Beha iou al ma king sys ems a e con ex -speci ic and de eloped h ough a igo ous
esea ch p ocess including bu no limi ed o a e iew o exis ing li e a u e o iden i y
ele an NTS and beha iou al ma ke s (Ra ind an e al. 2021; O’Conno and Max
Long 2011), using me hods like cogni i e ask analysis, in e iews, and ocus g oups
o iden i y c i ical skills and beha iou s, de eloping a axonomy ha ca ego ises skills
and elemen s wi h speci ic, obse able beha iou s, engaging subjec ma e expe s
o e iew and e ine he sys em, and Iden i ying and de ine speci ic beha iou al
ma ke s ha indica e good and poo pe o mance (da ConceiçÌo e al. 2017;
Ra ind an e al. 2021; I win, Tone, and Sedla 2023).
Building on esea ch conduc ed wi hin he Enginee ing Depa men a King’s College
London, ou doc o al p ojec has de eloped a beha iou al axonomy o non- echnical
skills o UK enginee s. This axonomy eme ged om a e iew o 93 academic
publica ions, policy documen s om he Royal Academy o Enginee ing, UK
Enginee ing Council, and su eys om he Ins i u ion o Enginee ing and Technology
(IET). We also conduc ed 18 semi-s uc u ed in e iews wi h 14 expe enginee s
ac oss he UK, employing knowledge elici a ion echniques. Thema ical and con en
analysis o he in e iews ex ac ed beha iou al indica o s ha dis inguish be ween
good and poo pe o mance ac oss NTS ca ego ies o UK enginee s.
We a e now en e ing he c ucial alida ion phase o ou esea ch. This inal s age
aims o es ablish he obse abili y and u ili y o ou beha iou al ma king sys em by
e alua ing ideo scena ios. We will also assess whe he hese ideo clips a e
sui able o aining u u e assesso s. The scena ios simula e wo kplace si ua ions
ha challenge enginee s’ non- echnical skills and showcase he iden i ied high-
pe o ming beha iou s ac oss skill ca ego ies. We a e seeking a endees o epo i
hey can obse e poo o good pe o mance beha iou s in ideo clips and help us
s eng hen he empi ical ounda ion o ou beha iou al ma king sys em.
2 WORKSHOP OBJECTIVES
Th ough s uc u ed engagemen , his wo kshop alida es and e ines he Gene al
Enginee ing Non- echnical Skills Beha iou al Ma king Sys em (GENTS-BMS). Fi s ,
we aim o assess he sys em’s eliabili y h ough In e -Ra e Reliabili y (IRR)
analysis, es ablishing quan i a i e measu es o consis ency ac oss e alua ions. This
s a is ical alida ion is c ucial o es ablishing he obus ness o beha iou al ma ke s
(Thompson 2013). Beyond s a is ical alida ion, we seek o ga he quali a i e
insigh s om obse e s ega ding he p ac ical u ili y and e ec i eness o he
obse a ion p ocess. This eedback will in o m po en ial e inemen s o he sys em’s
implemen a ion in eal-wo ld se ings. We also examine he dis inc i eness o
indi idual beha iou al ma ke s, ensu ing hey e ec i ely cap u e disc e e
compe encies wi hou o e lap.
Th oughou he wo kshop, we will documen pa icipan eedback, including speci ic
commen s, que ies, and obse a ions abou he ma king sys em. This eedback will
p o ide aluable insigh s o enhancing he e ec i eness o he GENTS-BMS
amewo k as an assessmen ool.
2.1 Ta ge audience
The a ge audience may include:
• Enginee ing lec u e s in e es ed in assessing and aining enginee ing
s uden s’ non- echnical skills in lea ning by doing and p ojec -based lea ning
en i onmen s.
• Indus y leade s in e es ed in upskilling newly ec ui ed enginee s on non-
echnical skills ia aining and pe o mance e alua ion.
• Enginee ing educa ion policymake s in e es ed in inco po a ing assessing and
aining non- echnical skills in highe educa ion acc edi a ion policies.
• Educa o s and cu iculum de elope s in e es ed in he use o obse a ional-
based assessmen s.
2.2 Expec ed lea ning ou comes
Upon comple ion o his wo kshop, pa icipan s a e expec ed o:
• De elop p ac ical expe ience applying he GENTS-BMS amewo k by
e alua ing ideo scena ios depic ing enginee ing wo kplace si ua ions.
• Con ibu e o he alida ion o he GENTS-BMS by p o iding e alua ions o
he obse abili y, u ili y, and dis inc i eness o he iden i ied beha iou al
ma ke s.
• Gain insigh s in o he p ocess o de eloping and alida ing a beha iou al
ma king sys em.
• Ha e he oppo uni y o sha e hei pe spec i es and eedback on he
GENTS-BMS amewo k, con ibu ing o i s e inemen o po en ial eal-wo ld
implemen a ion.
3 WORKSHOP DESIGN
3.1 Time plan
Pa icipan s will ecei e a sho aining in beha iou al obse a ion using he
GENTS-BMS amewo k. Obse e s will hen e alua e a sample o good and poo
pe o mance beha iou s shown in ideo clips using he beha iou al ma ke s in
GENTS-BMS and con ibu e o a ound able. To ensu e cla i y o pu pose and
e ec i e engagemen wi h he ma e ial, he wo kshop will ollow a s uc u ed o ma :
• Welcome and Wo kshop O e iew: In oducing he wo kshop’s pu pose as
alida ing a BMS o assessing and aining UK enginee s’ NTS using
simula ions.
• In oduc ion o Beha io al Ma king Sys ems (BMS): De ining a BMS as a
s uc u ed ool o iden i ying, a ing, and aining obse able NTS beha iou s
ha con ibu e o excellen o subs anda d pe o mance, p o iding a common
language and amewo k o discussion and eedback (Mi chell e al. 2012).
• Sys em O e iew: In oducing he speci ic BMS o UK enginee s (GENTS-
BMS), ou lining i s ca ego ies (e.g., con ex ual awa eness, decision-making,
enginee ing managemen , communica ion, eamwo k, leade ship) and he 2-
poin a ing scale (‘poo ’ o ‘good’ wi h ‘no applicable’). Pa icipan s will
ecei e a summa y shee .
• Guided P ac ice Session: Pa icipan s will independen ly comple e a
p ac ice a ing shee . A g oup discussion o obse a ions and ini ial a ings will
ollow.
• Independen Ra ing: Pa icipan s will wa ch up o 3 expe imen al simula ion
ideos (2-3 minu es each) and a e he obse ed NTS independen ly and
wi hou discussion (Flin e al. 2006). The ma king shee s will be collec ed a
he end o his pa .
• O e all Imp essions and Challenges: Facili a ing a discussion on
pa icipan s’ expe iences a ing NTS, ocusing on ease and challenges, as
well as cla i y o he sys em.
The in e ac i e design elemen s, wo kshop plans, and iming a e summa ised in
Table 1.
Table 1. Time plan
Ac i i y
Deli e y Me hod
Time
(mins)
Cumula i e
Time (mins)
Pa icipan
Engagemen
In oduc ion
and
Ra ionale
Welcome, e bal
p esen a ion wi h slides
5
5
Pa icipan s lis en
and e lec
BMS
In oduc ion
and T aining
P esen a ion, handbook
dis ibu ion
5
10
Pa icipan s ecei e
s uc u ed ma e ial
and de ini ions o
suppo
unde s anding
Guided
P ac ice
Video
Video playback (2–3
mins), independen
a ing, g oup discussion
15
25
Ac i e engagemen
h ough wa ching,
a ing, and sha ing
in e p e a ions in
discussion
Expe imen al
Video Ra ing
Sessions
Video playback,
independen a ing
20
45
Pa icipan s ocus
on obse a ion and
apply GENTS-BMS
independen ly
wi hou in luence
om o he s
Discussion
and
Feedback
G oup discussion led by
acili a o s
15
60
Pa icipan s e lec
on he challenges o
Obse ing
beha iou s and
sugges
imp o emen s
3.2 In e ac i i y
The wo kshop o ma is designed o be highly in e ac i e, p o iding a dynamic
en i onmen beyond he scope o a s a ic esea ch pape . Pa icipan s will ac i ely
engage in he alida ion p ocess by obse ing and e alua ing ideo scena ios
showcasing enginee ing non- echnical skills using he p o ided GENTS-BMS
amewo k. This di ec applica ion o he sys em allows o immedia e eedback on
he cla i y and applicabili y o he beha iou al ma ke s. Fu he mo e, including a
ound able discussion p o ides a c ucial oppo uni y o a endees o sha e hei
insigh s, que ies, and obse a ions abou he ma king sys em.
3.3 Video Con en Gene a ion
Real-wo ld cases illus a ing good and poo non- echnical skills (NTS) we e
ex ac ed om a hema ic analysis o semi-s uc u ed in e iews wi h 14 expe
enginee s in he UK, p o iding ich na a i es and c i ical cases ha g ounded he
simula ions in p o essional expe iences. These insigh s in o med he de elopmen o
loosely sc ip ed scena ios highligh ing speci ic NTS elemen s sho lis ed o he
beha iou al ma king sys em. To enhance he ideo clip p oduc ion p ocess, AI was
employed o gene a e de ailed line-by-line ansc ip s o he ac o s o he ideo clips,
which we e hen ca e ully e ined and alida ed by he esea ch eam o ensu e
accu acy and comple eness.
4 WORKSHOP RESULTS
The alida ion wo kshop u ilised a s uc u ed su ey ins umen o ga he quan i a i e
and quali a i e eedback on he Gene al Enginee ing Non- echnical Skills
Beha iou al Ma king Sys em om en pa icipan s. This sec ion summa ises he
esul s ela ed o he amewo k’s scope, cla i y o de ini ions, and he
app op ia eness o he obse a ion and a ing scales used.
4.1 Scope and Comp ehensi eness
Assessmen o he amewo k’s co e age e ealed mixed eedback ega ding
po en ial omissions. Six ou o en pa icipan s indica ed ha a ca ego y o elemen
was omi ed, while wo s a ed he co e age was adequa e. Sugges ed addi ions
included skills such as “Sys ems Thinking” and “Technical Skills”. Fu he mo e,
speci ic p o essional domains, such as “DEI, Inclusion,” and a g ea e ocus on
“Re lec ion and g ow h,” encompassing sel -lea ning and ca ee planning, we e
ecommended o inco po a ion. Con e sely, when asked ega ding unnecessa y
inclusions, he e was unanimous ag eemen : all en esponden s con i med ha no
included ca ego y o elemen was conside ed i ele an o a g adua e-le el
enginee .
4.2 U ili y o Beha iou al Ma ke s
The ca ego ies o skills and elemen s we e deemed clea ly de ined by se en
esponden s, wi h wo disag eeing. Analysis o he commen s e ealed ha he
ma king sys em was ega ded as “clea and logical,” wi h di e en ia ion no ably
enhanced by he inclusion o bo h posi i e and nega i e beha iou al examples.
Howe e , some concep ual o e lap was no ed, o ins ance, goal-se ing unde bo h
“ eamwo k” and “leade ship,” while ask o ganisa ion was ca ego ised wi h “ ask
managemen .” The beha iou al ma ke s we e uni o mly alida ed o u ili y. All en
pa icipan s ound he ‘good p ac ice’ beha iou al ma ke s use ul, and nine ound he
‘poo p ac ice’ ma ke s use ul. Challenges a ose om implemen a ion cons ain s:
obse e s no ed ha he e was insu icien ime o ully abso b he ma ke s be o e
a emp ing he obse a ional asks.
4.3 Ra ing Scale Usabili y and Cogni i e Load
Nine pa icipan s ound he a ing scale clea ly de ined, bu conce ns we e aised
ega ding i s p ac ical applica ion, pa icula ly when a emp ing o a e mul iple
ca ego ies simul aneously. The i e-poin a ing scale (1: Poo – 5: Ve y good, plus
N/A) was ejec ed by a majo i y o esponden s (six ‘no’ esponses, ze o ‘yes’
esponses). Pa icipan s cha ac e ised he scale as ei he oo b oad o oo g anula ,
leading o “ uzzy” di isions ha we e challenging o iden i y du ing a single
obse a ion un- h ough. This di icul y was explici ly linked o high cogni i e load, as
obse e s s uggled o hold i e dis inc beha iou al desc ip ions in mind while
assessing pe o mance.
In esponse o he di icul y o using he i e-poin scale, six pa icipan s indica ed a
p e e ence o a simple , wo-poin a ing scale (compe en /no compe en ). Mul iple
esponden s p oposed ha a h ee-poin scale would be ideal, o e ing su icien
nuance wi hou imposing he cogni i e load associa ed wi h a i e-poin scale. This
p e e ence o educed complexi y e lec s a consensus ha a 2- o 3-poin scale
would be mo e e ec i e han he cu en i e-poin sys em.
4.4 Fu u e Cou se o Ac ions and Re isions
The mos c i ical cou se o ac ion is e ising he a ing scale, as he cu en i e-poin
scale was la gely ejec ed. Fu u e s eps will in ol e pilo ing and implemen ing a
simple scale, ideally a h ee-poin scale, which was sugges ed by mul iple
esponden s as a comp omise ha p o ides necessa y nuance wi hou he cogni i e
load associa ed wi h holding i e desc ip ions in mind. This would add ess he
di icul y obse e s had when simul aneously a ing mul iple ca ego ies
The amewo k’s s uc u e and u ili y also equi e e inemen :
1. Reducing O e lap: Add essing he pe cei ed concep ual o e lap, such as he
di e en ia ion be ween goal se ing in “ eamwo k” and “leade ship” ca ego ies.
2. Inc ease Sca olding: To mi iga e he epo ed high cogni i e load, u u e
ainings and calib a ion will include g ea e sca olding, ensu ing su icien
ime o obse e s o abso b he handbook and ma ke s be o e a emp ing
obse a ional assessmen asks.
REFERENCES
Bu le , Philip Ca l. 2021. “De elopmen and E alua ion o a Beha iou al Ma ke
Sys em o UK Fi e and Rescue Se ice Inciden Commande s.”
ConceiçÌo, Vic o Fe nando PlÁcido da, JoÌo C uz Basso, Cus odio Lopes, and
Joakim Dahlman. 2017. “De elopmen o a Beha iou al Ma ke Sys em o
Ra ing Cade ’s Non-Technical Skills.” T ansNa , he In e na ional Jou nal
on Ma ine Na iga ion and Sa e y o Sea T anspo a ion 11 (2): 69–76.
h ps://doi.o g/10.12716/1001.11.02.07.
C ich on, Ma ga e T., Sco Mo a , and Lau en C ich on. 2017. “De eloping a
Team Beha iou al Ma ke F amewo k Using Obse a ions o Simula o -
Based Exe cises o Imp o e Team E ec i eness: A D illing Team Case
S udy.” Simula ion and Gaming 48 (3): 299–313.
h ps://doi.o g/10.1177/1046878117693266.
Flin, R, S Yule, S Pa e son-B own, N Ma an, D Rowley, and G Youngson. 2006.
“Expe imen al E alua ion o a Beha iou al Ma ke Sys em o Su geons’
Non-Technical Skills (NOTSS).” P oceedings o he Human Fac o s and
E gonomics Socie y Annual Mee ing 50 (10): 969–72.
h ps://doi.o g/10.1177/154193120605001036.
Flin, R, Lucy Mi chell, Ka hy Cou s, Geo ge Youngson, and Jane Mi chell.
2009. “Educa ion o Sco land Sc ub P ac i ione s’ Lis o In aope a i e
Non-Technical Skills (SPLINTS) S uc u ing Obse a ion, Ra ing and
Feedback o Sc ub P ac i ione s’ Beha iou s in he Ope a ing Thea e.”
www.abdn.ac.uk/ip c/splin s.
Hamle , Oli e E.D., Amy I win, Rhona Flin, and Nejc Sedla . 2023. “The
De elopmen o he Helicop e Non-Technical Skills (HeliNOTS)
Beha iou al Ma ke Sys ems.” E gonomics 66 (12): 2232–41.
h ps://doi.o g/10.1080/00140139.2023.2194591.
I win, A., I. R. Tone, and N. Sedla . 2023. “De eloping a P o o ype Beha iou al
Ma ke Sys em o Fa me Non-Technical Skills (FLINTS).” Jou nal o
Ag omedicine 28 (2): 199–207.
h ps://doi.o g/10.1080/1059924X.2022.2089420.
Mi chell, Lucy, Rhona Flin, S e en Yule, Jane Mi chell, Ka hy Cou s, and
Geo ge Youngson. 2012. “E alua ion o he Sc ub P ac i ione s’ Lis o
In aope a i e Non-Technical Skills (SPLINTS) Sys em.” In e na ional
Jou nal o Nu sing S udies 49 (2): 201–11.
h ps://doi.o g/h ps://doi.o g/10.1016/j.ijnu s u.2011.08.012.
O’Conno , Paul, and W Max Long. 2011. “The De elopmen o a P o o ype
Beha io al Ma ke Sys em o US Na y O ice s o he Deck.” Sa e y
Science 49 (10): 1381–87.
h ps://doi.o g/h ps://doi.o g/10.1016/j.ssci.2011.05.009.
Ra ind an, S i a hsan, Adam Haycock, Ka he ine Wool , and Siwan Thomas-
Gibson. 2021. “De elopmen and Impac o an Endoscopic Non-Technical
Skills (ENTS) Beha iou al Ma ke Sys em.” BMJ Simula ion and
Technology Enhanced Lea ning 7 (1): 17–25.
h ps://doi.o g/10.1136/bmjs el-2019-000526.
Te e a, Tegegne. 2019. “Beha iou al Ma ke Sys ems o E alua ion o
No wegian Me chan Shipping B idge Resou ce Managemen T aining.”
Thompson, D J. 2013. Beha iou al Ma ke s o Ai T a ic Con olle
De elopmen (Doc o al).
AI USE
AI was used o co ec po en ial g amma mis akes in he i s au ho ’s w i ing
and helped p o ide an ini ial abs ac , which was subsequen ly edi ed by he i s
au ho .
