Moving Beyond Standardized Metrics: Rethinking Criteria for Evaluating Educational Innovations

Wo kshop
Recommended ci a ion: Engelb ech , E., S obel, J., & Rooij, R. (2025). Mo ing
Beyond S anda dized Me ics: Re hinking C i e ia o E alua ing Educa ional
Inno a ions. In Kangaslampi, R., Langie, G., Jä inen, H.-M., & Nagy, B. (Eds.),
SEFI 53 d Annual Con e ence. Eu opean Socie y o Enginee ing Educa ion
(SEFI), Tampe e, Finland. DOI: 10.5281/zenodo.17631659.
This Con e ence Pape is b ough o you o open access by he 53 d Annual Con e ence
o he Eu opean Socie y o Enginee ing Educa ion (SEFI) a Tampe e Uni e si y in
Tampe e, Finland. This wo k is licensed unde a C ea i e Commons
A ibu ion-NonComme cial-Sha e Alike 4.0 In e na ional License.
MOVING BEYOND STANDARDIZED METRICS: RETHINKING
CRITERIA FOR EVALUATING EDUCATIONAL INNOVATIONS
E. Engelb ech a,1, J. M. S obel b, R. M. Rooij c
a Del Uni e si y o Technology, Del , NL, 0000-0001-8731-347X
b The Uni e si y o Texas a El Paso, El Paso, USA, 0000-0002-2124-1116
c Del Uni e si y o Technology, Del , NL, 0000-0002-2784-1892
Con e ence Key A eas: Building he capaci y and s eng hening he educa ional
compe ences o enginee ing educa o s; Imp o ing highe enginee ing educa ion
h ough esea ching enginee ing educa ion
Keywo ds: educa ion inno a ion; educa ional in e en ions; e alua ion c i e ia; alue
indica o s
ABSTRACT
The ield o inno a ion e alua ion in enginee ing educa ion is g owing, ye he
pe spec i es o educa o s and educa ional suppo s a a e o en o e looked. As a
esul , c i ical ac o s such as ele ance, con ex , and long- e m alue can go
un ecognized, limi ing bo h he e ec i eness o e alua ions and s akeholde
engagemen . This wo kshop builds on indings om a G oup Concep Mapping
(GCM) s udy conduc ed a a Eu opean uni e si y o echnology, whe e enginee ing
educa o s and suppo s a collabo a i ely iden i ied and anked 104 c i e ia o
e alua ing educa ional inno a ions. Ea ly indings e ealed a s ong emphasis on
quali a i e aspec s, such as wo kload, s uden well-being, and alignmen wi h
s akeholde needs, anked abo e conce ns such as scalabili y and egula o y
compliance. The wo kshop has wo aims: o b oaden he s udy by in ol ing an
in e na ional g oup o pa icipan s om di e se p o essional oles, and o in oduce
he enginee ing educa ion communi y o a mo e inclusi e and s akeholde -in o med
iew o ‘quali y’. Pa icipan s will collabo a i ely e lec on and u he de elop he lis
o e alua ion c i e ia, and g oup hem in o concep ual hemes. By he end o he
session, pa icipan g oups will ha e explo ed and de eloped ca ego iza ions o
e alua ion c i e ia.
1 Co esponding Au ho
E. Engelb ech
e.engelb ech [a ] udel .nl
1 BACKGROUND AND RATIONALE
1.1 Cu en pe spec i es on ‘quali y’
The e is limi ed esea ch on how educa o s and educa ional suppo s a , he key
s akeholde s in implemen a ion, de ine and p io i ize e alua ion c i e ia. Thei
pe spec i es a e essen ial o de eloping meaning ul, con ex -sensi i e amewo ks,
ye bo h g oups a e o en excluded om o mal e alua ion p ocesses while acul ies
all back on s anda dized me ics o quali y e alua ion. This limi s ele ance and
buy-in (A ey e al., 2009; Bajada e al., 2019; Michie e al., 2002). Wi hou hei inpu ,
c i ical ac o s in luencing he long- e m success o inno a ions emain o e looked.
Co-c ea ion wi h s akeholde s is one o he p inciples o cu iculum agili y (B ink e
al., 2024). This amewo k also highligh s he impo ance o ha ing s uc u es in
place o suppo inno a ion. The ul ima e goal o ou esea ch is o de elop one such
s uc u e by de eloping a amewo k o e alua ing inno a ion wi h academic
publica ion in mind.
1.2 Disco e ing new pe spec i es on ‘quali y’
This wo kshop d aws on a p e ious s udy (in e iew) whe e da a collec ed using he
me hod o G oup Concep Mapping (GCM) (T ochim, 1989). Enginee ing educa o s
and educa ional suppo s a b ains o med, so ed and anked 104 c i e ia o
e alua ing educa ional inno a ions a a uni e si y o echnology in Eu ope. The
c i e ia we e so ed based on simila i y o o m concep ual hemes. The anking was
done based on impo ance. One o ou p elimina y indings showed ha quali a i e
c i e ia ela ed o wo kload, s uden well-being and he needs and goals o
s akeholde s we e p io i ized o e scalabili y, and egula ion and compliance.
1.3 Towa ds an in e na ional iew
Du ing he ini ial s udy, he s akeholde g oup was limi ed o hose in ou mo e
immedia e, local p o essional ne wo k. This wo kshop ex ends ou p e ious wo k and
se es wo pu poses. Fi s ly, o expand on his s udy by including a mo e
in e na ional g oup o s akeholde s, occupying a wide ange o p o essional oles.
Secondly, o expose he la ge enginee ing communi y o a di e en iew on ‘quali y’
in educa ion.
The add essed he esea ch ques ion: How do educa ion s akeholde s concep ualize
e alua ion c i e ia o educa ional inno a ions? Pa icipan s join ly e lec ed on hose
c i e ia and ca ego ize hem.
2 WORKSHOP OBJECTIVES
2.1 Ta ge audience
The session is open o anyone in e es ed in he opic o educa ion inno a ion
e alua ion and educa ional quali y. The wo kshop will be mos bene icial o hose
wan ing o e alua e educa ional in e en ions and inno a ions o imp o ing
ins uc ional design. The e alua ion c i e ia can be used du ing egula quali y
managemen p ocesses, o o educa ional esea ch ini ia i es.
2.2 Expec ed lea ning ou comes
By he end o he wo kshop, pa icipan s should be able o concep ualize ele an
e alua ion c i e ia ha align wi h speci ic ypes o inno a ions.
3 WORKSHOP DESIGN
3.1 Time plan
The schedule o he wo kshop was as ollows:
Table 1. Time plan
Run ime
Ac i i y
No es
10 min In oduc ion In oducing opic and me hod
10 min B ains o ming Discussing s a emen s
25 min So ing and g oup
discussion So ing c i e ia o c ea e ca ego iza ion
15 min Plena y discussion
Sha ing esul s and w ap-up
3.2 In e ac i i y
A he s a o he wo kshop, we b ie ly sha ed in o ma ion abou ou p ojec o
de eloping a amewo k o e alua ing educa ional inno a ions and sha ed he
p ocess we wen h ough o comple ing he GCM s udy.
Then, pa icipan s o med g oups. Each g oup ecei ed a wo kshop pack wi h he
ollowing i ems:
 Discussion p omp ca ds o b ains o ming. The discussions gi e pa icipan s
a b ie in oduc ion o some o he e alua ion c i e ia and help hem
con ex ualize he c i e ia.
 Ins uc ions ca d o guide he ac i i y.
 Pe sona ca ds which pa icipan s use o selec hei oles du ing he
wo kshop. The oles assigned we e Coo dina o ( o managing he g oup and
wo kshop ma e ials), P esen e ( o sha ing he esul s om he discussions),
and Academics ( hei main unc ion being b ains o ming) assigned o he
emaining g oup membe s.
 C i e ia ca ds con aining he c i e ia collec ed du ing he GCM s udy.
 Clips, s icky no es and pens o g ouping and ca ego izing he c i e ia.
Fi s , pa icipan s b ains o med c i e ia using he discussion p omp ca ds. This also
se ed as a wa m-up ac i i y. Nex , hey wo ked h ough he c i e ia ca ds and
g ouped hem acco ding o simila i y (concep ually). Each g oup was assigned a
label o ca ego ize hem. Du ing he plena y discussion, he ca ego ies we e
compa ed, and we had u he discussions abou c i e ia and e alua ion o
inno a ions.
4 WORKSHOP RESULTS
F om he wo kshop, h ee se s o ca ego iza ions eme ged. The ca ego ies a e lis ed
in Table 2, alongside he ca ego ies om he o iginal GCM s udy.
Table 2. Time plan
G oup 1 G oup 2 G oup 3 CGM s udy
 Implemen a ion
(long- e m
ope a ion)
 Mo i a ional
e ec s
 Sus ainable
De elopmen
Goals
 S uden well-
being

S uden lea ning
 Legal, p ac ical
and
s akeholde s
 Scalabili y
 Facul y
(indi idual
educa o s)
 S uden lea ning
 S uden a ec i e
 Design
p inciples o
inno a ion
(educa ional
design)
 Vague (could be
in mul iple
ca ego ies)
 Wellbeing and
inclusi eness
 Educa o loss
and bene i s
 Lea ning
ou come e ec s
 In ended
implemen a ion
 Sus ainabili y
(long- e m)
 Rules,
egula ions,
policies
 Suppo
 Rele ance o
socie y/
au hen ici y
 Unin ended/
miscellaneous
 Execu ion
 Complica ions
 Mo i a ion and
engagemen
 Wellbeing
 Cou se pu pose
 S akeholde s
 Cu iculum
 Res ic ions
 E iciency
 S uden success
 Impac on use s
 Con ex o
inno a ion
 Risks and
Bene i s
 Scalabili y and
esilience
 T ade-o s
 S akeholde
needs
 Sui abili y
 Regula ion
Some no able simila i ies and di e ences ha can be obse ed. All g oups iden i ied
he s uden lea ning expe ience and included mo i a ion, wellbeing and engagemen ,
indica ing a sha ed belie ha success ul inno a ions should ha e a posi i e e ec on
s uden s. This was also e lec ed in he GCM s udy, whe e s uden wellbeing,
s uden lea ning and s uden success we e iden i ied as dis inc ca ego ies.
Likewise, each g oup ecognized he impo ance o inno a ions o be p ac ical and
manageable o e ime, wi h c i e ia such as scalabili y and sus ainabili y. In his
ega d, pa icipan s in he GCM s udy iden i ied e iciency, isks and bene i s, as well
as ade-o s.
Fu he mo e, con ex ual ac o s such as a ious s akeholde s and legisla i e aspec s
appea ed ac oss all g oups. In he o iginal s udy, he con ex o inno a ion, scalabili y
and esilience, sui abili y, as well as egula ion we e iden i ied.
Howe e , he g oups di e ed in he p ima y ocus. G oup 1 seemed o emphasize
ope a ional and design aspec s, c ea ing ca ego ies ha ela ed o implemen a ion
and educa ional design. G oup 2 included mo e human-cen e ed ca ego ies, such as
mo i a ion, inclusion, wellbeing and suppo . G oup 3, on he o he hand, adop ed a
b oade lens, connec ing e alua ion o inno a ion o Sus ainable De elopmen
Goals, ac o s ha could hinde he inno a ion (complica ions and es ic ions), and
cu iculum- ela ed aspec s.
All ca ego ies c ea ed du ing he GCM s udy we e seen again du ing his wo kshop
in some o m o ano he , excep o wo ca ego ies: con ex o inno a ion and ade-
o s. Con ex o inno a ion included c i e ia such as ‘ ans e abili y’ ( o o he
cou ses/ acul ies/disciplines) and ‘impac o he inno a ion on o he pa s o he
cu iculum’. The wo kshop pa icipan s associa ed hese c i e ia wi h cu iculum
design and implemen a ion- ela ed ca ego ies. T ade-o s included ‘ ealis ic

ime ame o implemen a ion’, and ‘ ime spen on cou se adminis a ion’, while he
wo kshop pa icipan s g ouped hese c i e ia unde es ic ions, o educa o - ela ed
ca ego ies.
Each pa icipan b ough wi h hem hei own pe spec i es ha a e shaped by hei
p o essional oles and pe sonal alues. Addi ionally, each inno a ion is implemen ed
o mee speci ic pu poses in i s own unique con ex . Consequen ly, i migh be
nei he possible no desi able o p esc ibe a ixed lis o e alua ion c i e ia. Ins ead,
e alua ion guidelines should emain lexible and esponsi e o hei con ex s o
implemen a ion o acili a e e alua ions ha a e no only in o ma i e, bu also
meaning ul.
Howe e , e alua o s can use hese ca ego iza ions o pinpoin he main pu pose o
ha ing implemen ed an inno a ion, and so ocus hei e alua ion e o s o ob ain
mo e use ul and ele an da a ha can in o m decision making when implemen ing
o imp o ing hei educa ional inno a ions.
REFERENCES
A ey, J. B., A olio, B. J., C ossley, C. D., & Lu hans, F. (2009). Psychological
owne ship: Theo e ical ex ensions, measu emen and ela ion o wo k ou comes.
Jou nal o O ganiza ional Beha io , 30(2), 173–191. h ps://doi.o g/10.1002/job.583
Bajada, C., Kandlbinde , P., & T ayle , R. (2019). A gene al amewo k o cul i a ing
inno a ions in highe educa ion cu iculum. Highe Educa ion Resea ch &
De elopmen , 38(3), 465–478. h ps://doi.o g/10.1080/07294360.2019.1572715
B ink, S. C., De Hei, M., Sjoe , E., Ca lsson, C. J., Geo gsson, F., Kelle , E.,
McCa an, C., Enelund, M., Lyng, R., & Admi aal, W. (2024). Cu iculum Agili y
p inciples o ans o ma i e inno a ion in enginee ing educa ion. Eu opean Jou nal
o Enginee ing Educa ion, 1–17. h ps://doi.o g/10.1080/03043797.2024.2398165
Michie, J., Ough on, C., & Bennion, Y. (2002). Employee Owne ship, Mo i a ion and
P oduc i i y. h p:// hewo k ounda ion.co.uk/ esea ch/publica ions/emp_own.jsp
T ochim, W. M. K. (1989). An in oduc ion o concep mapping o planning and
e alua ion. E alua ion and P og am Planning, 12(1), 1–16.
h ps://doi.o g/10.1016/0149-7189(89)90016-5