AHFE 2026: Vol. XX, 2026
doi: 10.54941/ah eXXXX
© 2025. Published by AHFE Open Access. All igh s ese ed. 1
Explo ing Usabili y And Use -expe ience
Me ics Wi h A No el AR App In The
MASTERLY P ojec
Ch is ophe G. Bu ns1, Sa ah Fle che 1, Apos olis Papa asileiou2,
Themis Anas asiou2, Geo ge Michalos2, So i is Mak is2
1 C an ield Uni e si y, College Rd, Wha ley End, Bed o d, MK43 0AL, Uni ed Kingdom
2 Labo a o y o Manu ac u ing Sys ems & Au oma ion (LMS), Depa men o Mechanical
Enginee ing and Ae onau ics, Uni e si y Campus Rio, Pa as 26504, G eece
ABSTRACT
The p esen s udy desc ibes an ini ial use -expe ience (UX) e alua ion o p o o ype
augmen ed eali y (AR) in e ace which in e ac s wi h a no el indus ial human- obo
collabo a i e sys em. Se en een pa icipan s wi h a ying le els o expe ience wi h AR
sys ems a he Uni e si y o Pa as de elopmen si e we e guided h ough he sys em’s
unc ions be o e comple ing a sho manual assembly ask di ec ed by he AR sys em.
Pa icipan s e alua ed hei expe ience ia a ques ionnai e comp ising s anda dised
psychome ics (NASA TLX, UEQ, mCSE, SUS, and he Ten-I em Pe sonali y in en o y o
TiPi), while addi ional ques ions pe mi ed ee esponses ega ding us in he sys em,
u ili y, and use p e e ences. Two inal i ems in es iga ed aes he ic and unc ional aspec s
o he isual in e ace, and he o e all ease o i s - ime usage. Using co ela ion, we
examined expec ed consis encies ac oss di e en UX me ics and a sho - o m pe sonali y
in en o y. Ini ial indings om he su ey a e epo ed on he o e all s a e o he UX, and
modi ica ions o he su ey o u u e use in he MASTERLY p ojec ’s o he use-cases.
Pa icipan s epo ed widely posi i e in e ac ions, and hei esponses also p o ided
sugges ions well imp o emen s o he inal ques ionnai e o subsequen es ing.
Keywo ds: obo s collabo a ion use in e aces UI end-e ec o psychome ics HCI UX
INTRODUCTION
Use expe iences, so wa e and ha dwa e usabili y and hei associa ed es ing
me hods ha e become inc easingly i al in mode n p oduc de elopmen in all
o ms (Saga & Saha 2017) and ha e become de ined in e ol ing IEEE s anda ds
(e.g. IEEE s d. 610.12-1990). These a ied measu es se e as ools and echniques
o he e alua ion o he quali y and usabili y o use in e aces, so wa e and
ha dwa e, and has e en ually esul ed in he b oade ield o use expe ience, o
UX – a ela i ely newe ex ension o he ield o psychome ics (Lewis 2015). Use
expe ience has become a key ea u e o wo k in almos all sys ems whe e humans
in e ac wi h echnology and especially p oduc s, whe e enhanced usabili y can
inc ease p oduc e enues by 10-35% (Be oa, T oya and Vallecillo 2005). The
human ac o s componen o he MASTERLY p ojec has concen a ed on
e alua ing use accep ance and ope a o expe ience wi h hese echnologies. In
2 Bu ns e al.
Janua y 2025, an ini ial s udy employed a mixed-me hod ques ionnai e combining
quan i a i e and quali a i e measu es o assess use expe ience wi h an ea ly
p o o ype o he indus ial assembly sys em. This e alua ion will also se e as a
empla e o assessing he emaining use cases. An addi ional esea ch in e es was
he inclusion o a sho pe sonali y assessmen , ocusing on Openness o
Expe ience (e.g. McC ae & Su in, 2009), o explo e po en ial co ela ions wi h
usabili y sco es and a i udes owa d no el echnologies.
The MASTERLY p ojec co e s h ee indus ial use cases: ae onau ics, logis ics,
and assembly. This s udy ocuses on assembly, whe e ope a o s build elec onic
panels wi h nume ous componen s. The solu ion combines a mobile obo (AGV)
wi h a obo ic a m and cus om g ippe , managed by back-end so wa e. Humans
and obo s collabo a e o place pa s and e ie e ex as as needed. An AR headse
p o ides isual o e lays o componen selec ion and placemen , o e ing wo
modes: de ailed aining o no ices and simpli ied guidance o expe ienced use s.
This lexibili y suppo s equen design changes. Assembly ypically akes o e
an hou , s a ing wi h a p e-selec ed ki bu o en equi ing addi ional i ems.
MASTERLY aims o educe e o , imp o e accu acy, and enhance adap abili y.A
he cu en s age o de elopmen , only he i ual eali y (VR) in e ace
ep esen ing he ope a o ’s in e ac ion wi h he assembly p ocess has been
implemen ed. This simula ed en i onmen allows ea ly e alua ion o sys em
usabili y be o e ull ha dwa e in eg a ion. The VR p o o ype eplica es he planned
wo kspace, enabling pa icipan s o expe ience ask low, in e ace layou , and
in e ac ion eedback unde con olled condi ions.
In Janua y 2025, a pilo s udy was conduc ed o e alua e he usabili y and use
expe ience o his VR in e ace. A mixed-me hod ques ionnai e combined
quan i a i e and quali a i e measu es o assess pe cei ed ease o use, ask cla i y,
and o e all accep ance. To explo e indi idual di e ences in in e ac ion a i udes,
a b ie pe sonali y measu e ( he Ten I em Pe sonali y In en o y (TIPI) was
included) was also included. The aim o he p esen s udy is o pilo and alida e
he p ocedu e o manu ac u ing ope a o es ing wi hin he MASTERLY p ojec
by assessing he usabili y and use expe ience o he VR-based p o o ype in e ace,
which will se e as a me hodological ounda ion o subsequen e alua ions o he
comple e sys em.
METHODS
This s udy was conduc ed as a single mixed-me hods session using quali a i e and
quan i a i e psychome ic su ey ins umen s, wi h he main aim being o assess
he u ili y o he me hods hemsel es a he han he s ic e icacy o he
MASTERLY p o o ypes a his s age. E hical app o al was ob ained using
C an ield Uni e si y’s CURES sys em unde e e ence CURES/24227/2025.
Pa icipan s
Pa icipan s we e olun ee s d awn om an enginee ing class module a he
Uni e si y o Pa as including bo h s uden s and esea ch s a . Comple e da a was
Bu ns e al. 3
a ailable om 17 male pa icipan s, comp ising nine indi iduals aged 18-25, six
indi iduals aged 25-30, and wo indi iduals aged 30-35. All pa icipan s we e
luen in English.
Ma e ials
Pa icipan s accessed p o o ype ha dwa e and so wa e om he MASTERLY
p ojec , using cus om AR so wa e unning on a Mic oso Hololens 2 headse o
con ol a obo a m. The wi eless headse s eams AR isuals om a PC, weighs
566 g, and has a 1440×936 esolu ion. The AR so wa e ope a ed a cus om g ippe
o handling elec onic componen s. In e ac ion elied on hand- acking ges u es
simila o mobile aps and pinches. Figu es 1 and 2 show sample image y.
Figu e 1: Examples o he image y displayed using he “inexpe ienced use ” mode,
whe e images o each componen and i s i ing loca ion on he panel a e p esen ed
o he use . The use manually acknowledges each componen in u n be o e
i ing.
Figu e 2: Two me hods o manually p og amming he a m and g ippe , using
ei he speci ica ion o indi idual join angles o adjus he a m’s posi ion, o by
i ually g asping and posing he a m in posi ion.
Pa icipan s comple ed a ques ionnai e ia Qual ics’ XM online pla o m
(h ps://www.qual ics.com) comp ising demog aphic in o ma ion, he NASA
TLX (Ha , S a eland, & Lowell 1988), he UEQ (Laugwi z, Held & Sch epp,
2008), he mCSE Scale (La e , Ra cli e, & C o y, 2012), he SUS (B ooke 1986),
and he TiPi pe sonali y in en o y (Gosling, Ren ow & Swann, 2003). Addi ional
ques ions allowed ee esponses on he pa icipan ’s expe ience o he sys em’s
us and p ospec i e use ulness. Pa icipan s we e addi ionally asked abou hei
le el o expe ience in using AR o VR echnologies; in u u e samples sizes, his
could allow he c ea ion o sub-g oups o compa isons. Ques ion i ems used a
Like - ype scale unless ee ex esponses we e equi ed.
4 Bu ns e al.
P ocedu e
Tes ing ook place he Uni e si y o Pa as enginee ing depa men . Pa icipan s
we e in oduced o he sys em by membe s o he de elopmen eam. The
in e ac ion wi h he sys em comp ised:
Pa icipan s calib a ed he headse using a QR code, hen amilia ized hemsel es
wi h he Hololens’ own w is -moun ed menu. They assembled componen s on he
backpla e in wo modes: inexpe ienced use , which displayed images and AR
o e lays o p ecise placemen , and expe ienced use , which p o ided b ie ex
p omp s. Finally, hey manipula ed he obo a m bo h manually and i ually,
adjus ing join angles o guide posi ioning (Figu e 3). As he sys em was no ye
comple e a he ime o es ing, some s ages we e manually ac i a ed in u n by he
de elopmen eam whils he pa icipan iewed he in e ac i e p omp s om
inside he Hololens headse . Each session equi ed app oxima ely 20 minu es o
in e ac ion ime, a e which each pa icipan was gi en a QR code o he
ques ionnai e o e alua e hei expe ience. A C an ield esea che was in
a endance o answe any ques ions ega ding he con en o consen aspec s o he
ques ionnai e. C an ield Uni e si y’s adhe ence o GDPR and e hical p inciples
we e explained o pa icipan s be o e hey began he ques ionnai e. Pa icipan s
had he op ion o iew he ques ionnai e in G eek o English, and ypically chose
English. Pa icipan s we e ee o ask any ques ions abou hei pa icipa ion o he
me hods a he end o he session.
RESULTS
Nume ical and s a is ical da a we e analysed using Mic oso Excel and SPSS
S a is ics ( 29.0.1.0(171)). P ima y hemes we e iden id ied om he quali a i e
esponses ia Mic oso ’s CoPilo gene a i e AI (Mic oso ,
h ps://m365.cloud.mic oso /cha ) and checked o accu acy by he p ima y
au ho .
NASA TLX
Use a ings o he AR in e ac ions ac oss he TLX’s six dimensions we e b oadly
a ou able, wi h mean anks o all ac o s excep Pe sonal Sa is ac ion sco ing
less han hal he maximum a ing. As a single ins ance o o dinal sel - epo da a,
F iedman’s Chi and Wilcoxon es s we e used o analysis. A main e ec ac oss
all ac o s was p esen (χ2 (5) = 38.502, p < .001), wi h Pe sonal Sa is ac ion being
a ed mo e highly han all o he ac o s. These e ec s a e de ailed in Table 1, and
illus a ed g aphically in Figu e 3.
Table 1: Pos -hoc compa isons o NASA TLX, Bon e oni co ec ed o mul iple
compa isons.
Bu ns e al. 5
Figu e 3: Mean a ings o NASA TLX ac o s (+/- SEM).
Sys em Usabili y Sco es
Sco es o SUS usabili y a ings we e simila ly high, wi h a median sco e o 80,
and he 75 h pe cen ile a 87.5. Al hough SUS sco es can possess ela i ely li le
inhe en con ex , highe sco es e lec be e usabili y a ings, and Bango , Ko um
and Mille (2008) desc ibe median SUS sco es abo e 70 as “accep able”, while
Damyano e al. (2024) desc ibe such sco es as “A-g ade” o “Good”.
Modi ied Compu e Sel -e icacy sco es (mcSES)
Pa icipan s we e gene ally con iden o hei abili ies in b oad use o mode n
echnology, wi h a mean mCSES sco e o 8.57 and low dispe sal o sco es a ound
he mean (SD = 1.42).
Use Expe ience Ques ionnai e sco es
The UEQ’s au oma ed sco ing ecommended he emo al o wo pa icipan s due
o low in e nal consis ency in hei sco es. A e emo al, mean UEQ sco es
emained wi hin he accep able/ a ou able ange (highligh ed by he g een band in
Figu e 6), wi h sco es o Pe spicui y (i.e. gene al ease o use, and ease o lea ning
how o use he sys em) showing he highes a ings (See Table 2 and Figu e 4).
6 Bu ns e al.
Table 2: Mean sco es pe a ing om he UEQ
UEQ Scales
Mean
Va iance
A ac i eness
1.81
0.67
Pe spicui y
2.03
0.34
E iciency
1.44
0.72
Dependabili y
1.54
0.66
S imula ion
1.94
0.70
No el y
1.49
0.78
Figu e 4: Visual desc ip i es om he UEQ au oma ed sco ing.
Bu ns e al. 7
Co ela ions among measu es and a ings wi h TiPi sco es
Co ela ions among he s a e measu es and Openness sco es om he TiPi
in en o y p oduced only a single signi ican co ela ion wi h NASA TLX sco es
o Tempo al Demand ( = -.536, p < .027) om he en i e g oup o N = 17. I was
hoped ha sco es ac oss he domains o he o he psychome ics desc ibed hus a
would show signi ican co ela ions wi h openness o expe ience sco es om he
Ten I em Pe sonali y In en o y (i.e. as pa icipan s we e expe iencing no el
in e ac ions wi h simila ly no el ha dwa e and so wa e), bu he single co ela ion
p esen ed wi h he NASA TLX is concep ually di icul o explain and po en ially
spu ious.
Quali a i e esponses
Pa icipan s’ esponses o he ou quali a i e ques ions analysed hema ically
using Mic oso ’s CoPilo AI, and we e manually checked o accu acy and
consis ency by e e ence o he o iginal quo a ions.
Use eedback hemes o us
Pa icipan s gene ally us ed he sys em, gi ing posi i e eedback and ci ing
anspa ency and p edic able ou comes despi e some mino p o o ype gli ches
(e.g., calib a ion issues causing isual elemen s o appea o -sc een). T us was
ein o ced by di ec eedback and clea obo isibili y: “I us ed i because I had
di ec eedback… I had a clea iew o he physical wo ld.” Usabili y and in ui i e
design we e p aised: “Ve y esponsi e and easy o use” and “The so wa e was
e y well in eg a ed…”. Ins uc ional cla i y was ano he heme, wi h use s aluing
s ep-by-s ep guidance: “Ins uc ions we e clea and no complica ed” and “The
sys em su icien ly guided me…”. Reliabili y was also no ed: “Because i wo ks
co ec ly” and “The sys em was esponsi e o my inpu s.”
Use eedback hemes o pe cei ed use ulness
Pa icipan s el he sys em mainly bene i s less-expe ienced use s by isually
guiding asks, making i a s ong aining ool. I was seen as “use ul in
manu ac u ing sys ems; helps lea n new asks; ele an o obo ics esea ch.” Clea
ins uc ions and isual cues educed e o s and e o : “Clea ins uc ions o
newcome s; educes e o s; easie o inexpe ienced wo ke s.” Use s p aised he
in e ace and guidance: “Easy access o con ols; clea isualiza ion; esponsi e
AR in e ace.” T aining po en ial was emphasized: “Excellen o aining; helps
unde s and p ocesses; use ul o lea ning new asks.” Some sugges ed
imp o emen s: “Needs u he imp o emen s; could make asks easie wi h
enhancemen s.”
8 Bu ns e al.
Use eedback hemes o “Leas Liked” aspec s o UI
Pa icipan s iden i ied se e al laws and sugges ed imp o emen s. Many
commen s ocused on UI adjus men s, such as mo ing bu ons o o se ges u e-
acking inaccu acies. Use s no ed AR sys ems a e uncom o able o long wea
due o physical s ain and bulky headse s: “I is equi ed o bend sligh ly o ack
he pa ha I had g asped o he assembly” and “Wea ing he de ice o long
pe iods o ime can become uncom o able.” O he s men ioned e gonomics: “AR
applica ions in gene al can be 'non e gonomic' i used o a e y long pe iod o
ime” and “Some imes I had o posi ion my head/POV in a speci ic way o he
sys em o wo k.”. UI issues included slide s and bu on placemen : “The slide use
o con ol he obo a m,” “The sys em's close bu on o he end e ec o ,” and
“Maybe some bu ons I wan ed o be close .” Visual conce ns in ol ed “The
b igh ness o he panels” and “Di icul in e ac ion wi h small UI elemen s such as
he slide s.” Ges u e ecogni ion and came a acking we e also c i icized: “The
ecogni ion o he came a when I clicked in he applica ion,” and “The con ol and
acking o my hands could be be e .”. Posi i e eedback highligh ed e o
p e en ion: “I liked ha i wa ned me when I picked he w ong componen ” and
“The sys em can guide he asks equi ed by he job and in o m me when I o go
some hing.”.
Use Feedback hemes o “Mos Liked” aspec s o he UI
Pa icipan s alued he in e ace’s lexibili y and eal-wo ld isualiza ion o
componen placemen , no ing i was especially help ul o less-expe ienced
ope a o s. Use s p aised i s in ui i e design and cus omiza ion: “Highly in ui i e
and cus omizable o whe e I wan hings o be in my ield o iew” and “La ge and
easy- o- ead in e aces and UI elemen s. Simplici y and in ui i eness.” Visual
guidance was highligh ed: “The isual assis ance on whe e I should pu he
componen ” and “The isualiza ion o how o pe o m he nex assembly asks.”
Manual obo con ol was app ecia ed: “Tha I el I could accu a ely con ol he
obo wi h jus my hand mo emen s.” Inno a i e ea u es and g aphics ea ned
posi i e ema ks: “The inno a ion and he inc eased g aphics. Also I app ecia ed
he ac ion- esponse on app” and “The clea ins uc ion and use in e ace, he
inno a i e AR g aphics and seamless in eg a ion wi h he obo .” Finally, suppo
o no ices s ood ou : “I was e y help ul o an inexpe ienced ope a o ” and
“Clea ins uc ions, in ui i e isuals and guidelines.”
DISCUSSION
Bu ns e al. 9
The pilo e alua ion o he MASTERLY VR in e ace showed posi i e use
expe iences, con i ming he easibili y o planned usabili y and UX es ing o
manu ac u ing ope a o s. Quan i a i e esul s indica ed high usabili y, low
pe cei ed wo kload, and s ong use con idence. NASA-TLX a ings sugges ed
modes men al and physical demands bu high E o sco es. SUS sco es exceeded
accep ed usabili y benchma ks (Bango e al., 2008; Damyano e al., 2024), and
UEQ a ings desc ibed he in e ace as a ac i e, clea , and s imula ing. O e all,
indings sugges he p o o ype VR en i onmen suppo ed in ui i e, engaging
in e ac ion.
The e en ual use o he ques ionnai e in he p esen s udy will be o assess he use
expe ience o indus ial ope a o s (wi h a ying le els o expe ience) o he
p o o ype sys ems om he MASTERLY p ojec . A he ime o w i ing, he
p o o ype de ices om he MASTERLY p ojec ha e expanded in unc ionali y
and will p o ide a mo e “comple e” use expe ience du ing o mal es ing. In his
ega d, he ques ionnai e appea s o pe o m adequa ely, wi h some addi ional
insigh s p o ided by use commen s. The a ious me ics did no co ela e wi h
each o he in meaning ul ways, bu none heless s and as use ul indi idual measu es
o wo kload, sel -e icacy, usabili y and accep ance. This ques ionnai e also
complemen s addi ional wo k no de ailed he e which as included di ec ed and
media ed discussions, whe e ope a o s ha e desc ibed mo e ully, and in o mally,
desc ip ions o ea u es o di icul ies hey ace in he exis ing wo king
en i onmen and p ocedu es hey use.
In pa icula , gi ing pa icipan s he oppo uni y o exp ess quali a i e opinions
ga e insigh s in o imp o ing he ques ionnai e as whole. The i s modi ica ion
was he emo al o he TiPi in en o y. The comple e ques ionnai e ea u es
asso ed measu es o b oad usabili y, and, as i will be used o e alua e no el
de ices and sys ems, i was hypo hesised ha he inclusion o a sho pe sonali y
in en o y ( he TiPi), wi h ocus on he “openness o expe iences” ac o could
p o ide addi ional insigh in o he ope a o s who ul ima ely use hese de ices as
well as he unc ionali y o he p o o ypes; Huang e al. (2017) no ed ha
ex a e sion and in o e sion played a ole in he UX design, wi h ex a e s
ending o p e e g ea e le els o in e ac i i y and isual s imula ion. Simila ly, i
would no be un easonable o expec indi iduals wi h highe measu ed “openness”
o be mo e engaged in he use o no el sys ems.
In he p esen s udy, his was no he case, wi h he only s a is ically signi ican
inding being an in e se co ela ion be ween openness and Tempo al Demand as
measu ed by he NASA TLX, a he han e.g. addi ional signi ican posi i e
associa ions be ween openness and he SUS o mCSES scales.
