RESEARCH ARTICLE
SLAM-based augmen ed eali y o he
assessmen o sho - e m spa ial memo y. A
compa a i e s udy o isual e sus ac ile
s imuli
F ancisco Munoz-Mon oyaID
1
, M.-Ca men JuanID
1
*, Magdalena Mendez-Lopez
2
,
Ramon Molla
1
, F ancisco Abad
1
, Camino Fidalgo
2
1Ins i u o Uni e si a io de Au oma
´ ica e In o ma
´ ica Indus ial, Uni e si a Poli ècnica de València, Valencia,
Spain, 2Depa amen o de Psicologı
´a y Sociologı
´a, IIS A ago
´n, Uni e sidad de Za agoza, Te uel, Spain
*[email p o ec ed].es
Abs ac
The assessmen o human spa ial sho - e m memo y has mainly been pe o med using
isual s imuli and less equen ly using audi o y s imuli. This pape p esen s a amewo k o
he de elopmen o SLAM-based Augmen ed Reali y applica ions o he assessmen o
spa ial memo y. An AR mobile applica ion was de eloped o his ype o assessmen in ol -
ing isual and ac ile s imuli by using ou amewo k. The ask o be ca ied ou wi h he AR
applica ion is di ided in o wo phases: 1) a lea ning phase, in which pa icipan s physically
walk a ound a oom and ha e o emembe he loca ion o simple geome ical shapes; and
2) an e alua ion phase, in which he pa icipan s a e asked o ecall he loca ion o he
shapes. A s udy o compa ing he pe o mance ou comes using isual and ac ile s imuli
was ca ied ou . Fi y- h ee pa icipan s pe o med he ask using he wo condi ions (Tac ile
s Visual), bu wi h mo e han wo mon hs o di e ence (wi hin-subjec design). The numbe
o shapes placed co ec ly was simila o bo h condi ions. Howe e , he g oup ha used he
ac ile s imulus spen signi ican ly mo e ime comple ing he ask and equi ed signi ican ly
mo e a emp s. The pe o mance ou comes we e independen o gende . Some signi ican
co ela ions among a iables ela ed o he pe o mance ou comes and o he es s we e
ound. The ollowing signi ican co ela ions among a iables ela ed o he pe o mance ou -
comes using isual s imuli and he pa icipan s’ subjec i e a iables we e also ound: 1) he
g ea e he numbe o co ec ly placed shapes, he g ea e he pe cei ed compe ence; 2)
he mo e a emp s equi ed, he less he pe cei ed compe ence. We also ound ha pe -
cei ed enjoymen was highe when a highe sense o p esence was induced. Ou esul s
sugges ha ac ile s imuli a e alid s imuli o exploi o he assessmen o he abili y o
memo ize spa ial- ac ile associa ions, bu ha he abili y o memo ize spa ial- isual associa-
ions is dominan . Ou esul s also show ha gende does no a ec hese ypes o memo y
asks.
PLOS ONE
PLOS ONE | h ps://doi.o g/10.1371/jou nal.pone.0245976 Feb ua y 4, 2021 1 / 30
a1111111111
a1111111111
a1111111111
a1111111111
a1111111111
OPEN ACCESS
Ci a ion: Munoz-Mon oya F, Juan M-C, Mendez-
Lopez M, Molla R, Abad F, Fidalgo C (2021) SLAM-
based augmen ed eali y o he assessmen o
sho - e m spa ial memo y. A compa a i e s udy o
isual e sus ac ile s imuli. PLoS ONE 16(2):
e0245976. h ps://doi.o g/10.1371/jou nal.
pone.0245976
Edi o : Guido Maiello, Jus us Liebig Uni e si a
Giessen, GERMANY
Recei ed: June 30, 2020
Accep ed: Janua y 11, 2021
Published: Feb ua y 4, 2021
Pee Re iew His o y: PLOS ecognizes he
bene i s o anspa ency in he pee e iew
p ocess; he e o e, we enable he publica ion o
all o he con en o pee e iew and au ho
esponses alongside inal, published a icles. The
edi o ial his o y o his a icle is a ailable he e:
h ps://doi.o g/10.1371/jou nal.pone.0245976
Copy igh : ©2021 Munoz-Mon oya e al. This is an
open access a icle dis ibu ed unde he e ms o
he C ea i e Commons A ibu ion License, which
pe mi s un es ic ed use, dis ibu ion, and
ep oduc ion in any medium, p o ided he o iginal
au ho and sou ce a e c edi ed.
Da a A ailabili y S a emen : All ele an da a will
be wi hin he manusc ip and i s Suppo ing
In o ma ion iles.
1. In oduc ion
People ha e he abili y o s o e and emembe ep esen a ions o spa ial s imuli o sho pe i-
ods o ime. This abili y is known as spa ial sho - e m memo y [1]. F om neu obiological and
cogni i e pe spec i es, mos o he in o ma ion ha humans explici ly s o e in spa ial memo y
comes om he isual and audi o y modali ies [2–5]. Howe e , he human b ain has he abili y
o s o e in o ma ion om all senso y modali ies [6]. One o hese senso y modali ies is he
sense o ouch, which e en hough i is no he mos s udied sense, i is ce ainly a sense o
exploi . Spa ial memo y is used o s o e and emembe he ou e o ind a p e iously isi ed
place o o ind whe e we ha e le ou belongings (e.g., keys o glasses) [7]. This ype o mem-
o y is in ol ed in e e yday asks and allows na iga ion and sol ing spa ial asks. Thus, impai -
men s in spa ial memo y ha e se ious consequences in daily li e. Di e en memo y- ela ed
help ools can be used o assessmen as well as aining. Fo assessmen , hese ools can help
in de e mining he di icul ies ha may a ec people’s independence [8]. Fo aining, hese
ools can con ibu e o spa ial o ien a ion by imp o ing co ec way- inding beha iou s [9].
O ien a ion di icul ies usually occu due o impai men s o diseases, such as s oke [10], Alz-
heime ’s disease [11], acqui ed b ain inju y [12], and heal hy aging [13]. Human spa ial o ien-
a ion in eal li e is based on bo h sel -mo ion and isual cues (including p op iocep i e and
es ibula cues) [14]. Two echnologies ha can g ea ly assis in he de elopmen o ools o
he assessmen and aining o spa ial memo y a e Vi ual Reali y (VR) and Augmen ed Real-
i y (AR). These wo echnologies can be o g ea help when he subjec has o pe o m physical
mo emen in he eal wo ld. The ac ha he subjec s ha e o mo e a ound he eal wo ld o
pe o m he asks esembles wha hey expe ience in eal li e. Mo eo e , physical displacemen
has been shown o be impo an in acqui ing spa ial abili y skills [15].
This wo k p esen s a amewo k o he de elopmen o AR based on SLAM (Simul aneous
Localiza ion and Mapping). An AR mobile applica ion was de eloped o his ype o assess-
men in ol ing isual and ac ile s imuli by using ou amewo k. Ou AR applica ion can be
used in any indoo en i onmen (e.g., a hospi al wa d, he he apis ’s o ice, o he pa ien ’s
home) and equi es he subjec o physically walk a ound he eal wo ld. The ask ha he pa -
icipan s mus pe o m is di ided in o wo phases, a lea ning phase and an e alua ion phase.
In he lea ning phase, he pa icipan s physically walk a ound a oom, and ha e o emembe
he loca ion o i ual geome ical shapes. In he e alua ion phase, he pa icipan s a e asked o
ecall he loca ions whe e he shapes we e in he oom and place hem co ec ly using AR. Ou
s udy compa es sho - e m spa ial memo y in ol ing isual ( isual condi ion) and ac ile ( ac-
ile condi ion) senses. A simila p o ocol was designed o he sense o ouch, bu using eal
objec s. In he lea ning phase, he subjec s ouch eal objec s ha a e placed inside boxes in a
eal oom, and hey mus emembe hei loca ion. In he e alua ion phase, he pa icipan s
ouch eal objec s ha a e hidden inside boxes and hey ha e o place a i ual box ha ep e-
sen s he ouched objec in he co ec loca ion. To ou knowledge, his is he i s wo k ha
uses AR o compa e isual and ac ile s imuli o he assessmen o sho - e m spa ial memo y.
2. Assessmen o spa ial memo y using i ual and augmen ed
eali y
Spa ial memo y is ypically assessed in he isual modali y, using pape and pencil es s [16,
17]. The de elopmen o compu e ized ools ha use VR o AR p o ides some ad an ages
o e adi ional es s. VR and AR applica ions allow objec i e indica o s o he spa ial lea ning
o an indi idual (e.g., successes- ailu es, eac ion imes, speed, dis ance a elled, e c.) o be
ob ained and s o ed o la e s udy. The p esen a ion o s imuli can be a ied and con olled
[18–20]. Since VR and AR applica ions p o ide ad an ages ega ding he e alua ion and
PLOS ONE
Augmen ed eali y o assess sho - e m spa ial memo y using isual and ac ile s imuli
PLOS ONE | h ps://doi.o g/10.1371/jou nal.pone.0245976 Feb ua y 4, 2021 2 / 30
Funding: This wo k was unded mainly by FEDER/
Minis e io de Ciencia e Inno acio
´n – Agencia
Es a al de In es igacio
´n/AR3Senses (TIN2017-
87044-R); o he suppo was ecei ed om he
Gene ali a Valenciana/Fondo Social Eu opeo
(ACIF/2019/031); he Gobie no de A ago
´n
( esea ch g oup S31_20D) and FEDER 2020-2022
“Cons uyendo Eu opa desde A ago
´n”. The unde s
had no ole in s udy design, da a collec ion and
analysis, decision o publish, o p epa a ion o he
manusc ip .
Compe ing in e es s: The au ho s ha e decla ed
ha no compe ing in e es s exis .
aining o people in a eal en i onmen (i.e., lowe economic and ime cos s, e c.), he use o
hese echnologies in he s udy o human abili y has inc eased.
Di e en wo ks ha e used VR o assess spa ial memo y in humans [21–24]. In he i s VR
applica ions o he assessmen o spa ial memo y, he use s we e si ing in on o a compu e
sc een wi hou pe o ming physical displacemen . The use s explo ed a VR en i onmen du -
ing he ask [18,19,25]. Howe e , physical displacemen is an impo an aspec o conside o
spa ial abili y [15]. Some wo ks ha e included physical displacemen in VR en i onmen s [26–
28]. Rod ı
´guez-And e
´s e al. [26] s udied he in luence o wo di e en ypes o in e ac ion on
a VR ask (an ac i e physical condi ion e sus an inac i e physical condi ion). F om hei
esul s, no s a is ically signi ican di e ence in ask ou comes was ound o he wo ypes o
in e ac ion. They concluded ha he ype o in e ac ion did no a ec he child en’s pe o -
mance in he VR ask. E en hough he physical mo emen was pe o med on he Wii balance
boa d, he use s did no ha e he eeling o walking in he eal wo ld. Ca
´ denas e al. [28] de el-
oped a maze-based VR ask in which he pa icipan s had o pe o m physical mo emen . As
in he p e ious wo k, hey conside ed wo di e en ypes o in e ac ion (an ac i e physical
condi ion e sus an inac i e physical condi ion). The pe o mance ou comes on hei ask
we e be e o he inac i e physical condi ion. As in he p e ious wo k, he physical mo e-
men was pe o med on a bicycle and he pa icipan s did no ha e he eeling o iding a bike
in he eal wo ld, e en when he use had o pedal on a s a iona y bike and had he HMD on.
Ve y ew wo ks ha e been p esen ed o he s udy o he pe o mance o spa ial memo y
using AR. Ou esea ch g oup has p esen ed mos o hese wo ks. Ou g oup has pionee ed
he de elopmen and alida ion o AR applica ions o he assessmen o spa ial memo y [20,
29–31]. In he i s wo k [20,29], we de eloped an AR applica ion ha used image a ge s ha
we e dis ibu ed in he eal wo ld. In he second wo k [30,31], we p esen ed an AR applica ion
based on SLAM o isual s imuli. The main di e ence wi h he p e ious wo k was ha no
addi ional elemen s we e added o he eal en i onmen . The applica ion used in his second
wo k was c ea ed using he amewo k ha is p esen ed in his pape , bu i only ocuses on
isual s imuli. Ou g oup has also s udied audi o y s imuli o he assessmen o spa ial mem-
o y [32]. Meanwhile, Keil e al. [33] ha e ecen ly p esen ed a de elopmen and a s udy ha
a e closely ela ed o ou wo k. They de eloped an AR sys em o loca ion memo y and dis-
ance es ima ion. The sys em shows holog aphic g ids on he loo . They ca ied ou a s udy o
de e mine he e ec s o g ids on he loo on loca ion memo y and dis ance es ima ion in an
indoo en i onmen . The esul s o hei s udy showed ha dis ance es ima ions we e mo e
accu a e when a g id was displayed. The pe o mance o loca ion memo y was wo se when a
g id was displayed.
O he wo ks, which a e no as closely ela ed o ou s, ha e ocused on he use o AR o help
in na iga ion. Chu e al. [34] p esen ed a mobile AR ool o help in na iga ion. The scene cap-
u ed by he came a was augmen ed wi h i ual 2D icons o guide use s o hei des ina ion.
Thei s udy compa ed an AR ool and wo map na iga ion modes. Thei conclusion was ha
he AR ool was be e han he wo map na iga ion modes. Rehman and Cao [35] p esen ed
an AR applica ion o help people na iga e indoo s. They compa ed he na iga ion using Goo-
gle Glass, a sma phone, and a pape map. Thei esul s showed ha he pa icipan s pe cei ed
Google Glass o be he mos accu a e ool. No s a is ical di e ences we e ound be ween Goo-
gle Glass and he sma phone in e ms o wo kload. Google Glass and he sma phone we e
be e han he pape map in e ms o lowe wo kload and less ime o execu ion. Howe e , he
ou e e en ion was be e wi h he pape map. Peleg e al. [36] p esen ed a ool o ou e plan-
ning in public anspo a ion. The pe o mance using a mobile AR applica ion and a non-AR
applica ion was compa ed. The augmen ed scene showed he expec ed imes ha buses go
h ough each s a ion on he map. They ca ied ou a s udy in which hey compa ed he
PLOS ONE
Augmen ed eali y o assess sho - e m spa ial memo y using isual and ac ile s imuli
PLOS ONE | h ps://doi.o g/10.1371/jou nal.pone.0245976 Feb ua y 4, 2021 3 / 30
pe o mance o olde pa icipan s and younge pa icipan s. The pa icipan s who used he AR
applica ion comple ed he ask in less ime, bu wi h highe e o a es. These esul s we e
independen o he age o he pa icipan s. The younge pa icipan s showed signi ican ly
as e pe o mance compa ed o he olde pa icipan s while using he AR applica ion. How-
e e , he e we e no signi ican di e ences ega ding e o a es.
3. Design and de elopmen
In his sec ion, we desc ibe he SLAM-based AR applica ion used o he assessmen o spa ial
sho - e m memo y and how i has been used o con igu e he en i onmen and in he isual
condi ion. We desc ibe he s eps ollowed in he ac ile condi ion and how he use o he AR
applica ion and ouching eal objec s ha e been combined. We also p esen he h ee applica-
ions ha we e de eloped o au oma e asks ha ha e adi ionally been e alua ed using pen
and pape . We inally p esen he ha dwa e and so wa e used o he de elopmen o ou
applica ions and he a chi ec u e o he amewo k p esen ed in his pape .
3.1. Con igu a ion o he en i onmen
The AR applica ion based on SLAM is designed o assess spa ial sho - e m memo y o shape
loca ion. The AR applica ion uns on a sma phone. The pa icipan s ha e o pe o m a ask
ha consis s o walking a ound a eal oom. The dimensions o he oom a e 5 x 7.5 (a ound
38 m
2
). The pa icipan s mus look o i ual geome ical shapes and emembe hei loca-
ions. These geome ical shapes can only be seen h ough he mobile sc een. Fo his pu pose,
he scene mus be con igu ed in wo phases: oom scanning and shape con igu a ion.
The oom scanning phase. In his con igu a ion phase, he supe iso scans he eal oom.
This scan s o es he geome y o he oom in he sma phone (Fig 1A and 1B). This scan
includes he walls as well as ables, chai s, compu e s, e c.
The shape con igu a ion phase. The supe iso places he di e en geome ical shapes in
he eal scanned oom (Fig 1C) using he geome y s o ed in he sma phone. The applica ion
has abou i y 3D models o di e en ypes. In ou s udy, we used eigh geome ical shapes.
These eigh shapes a e: cube, ec angula p ism, wide cylinde , na ow cylinde , sphe e, semi-
sphe e, cone, and py amid. The applica ion allows objec s o be placed on plana su aces (ho -
izon al o e ical). In ou case, hey we e all placed on ables. The bo om side o he geome ic
shape is a ached o he de ec ed su ace. The shapes canno be o a ed. The shapes a e always
acing he came a posi ion o he sma phone, and hei up ec o s a e pe pendicula o he
plane o he su ace.
3.2. The isual condi ion
Fo he isual condi ion, o use he AR applica ion, he use mus comple e he ollowing wo
phases. Fig 2 g aphically shows he s eps ollowed by he use s in he isual condi ion.
The lea ning phase. The main objec i e o his phase is o allow he subjec o explo e he
oom sea ching o i ual geome ical shapes and o emembe hei loca ions. The supe iso
guides he use s h ough he oom. The use s see he shapes in he same o de and ollow he
same pa h. The es ablished o de , which was he mos sui able o seeing all o he shapes in a
con inuous ou , was: cone, na ow cylinde , cube, py amid, sphe e, ec angula p ism, wide
cylinde , and semi-sphe e. The pa icipan s ha e o indica e on he applica ion ha hey ha e
seen a shape by apping on i . A ha momen , a box appea s o show ha his shape has been
ma ked as seen. Once he shape has been ma ked by he use , i canno be seen anymo e.
The e alua ion phase. This phase e alua es he pa icipan s’ abili y o emembe he loca-
ion o he i ual geome ical shapes in he eal oom. The use s a e asked o place all o he
PLOS ONE
Augmen ed eali y o assess sho - e m spa ial memo y using isual and ac ile s imuli
PLOS ONE | h ps://doi.o g/10.1371/jou nal.pone.0245976 Feb ua y 4, 2021 4 / 30
PLOS ONE
Augmen ed eali y o assess sho - e m spa ial memo y using isual and ac ile s imuli
PLOS ONE | h ps://doi.o g/10.1371/jou nal.pone.0245976 Feb ua y 4, 2021 5 / 30

eigh shapes in hei co ec loca ions using he applica ion. The shapes ha e a p e-es ablished
andom o de so ha all o he pa icipan s ca y ou he placemen in he same o de . This
o de is di e en om he o de ollowed in he lea ning phase: cube, ec angula p ism, wide
cylinde , na ow cylinde , sphe e, semi-sphe e, cone, and py amid. The placemen o he
shapes does no ha e o be exac . The e is a p oximi y ole ance. The poin whe e he shape is
placed by he use can be up o hal a me e away om he exac poin . The pa icipan s ha e
Fig 1. Con igu a ion phase. (A) and (B) oom scanning phase. (C) Shape con igu a ion phase (an example o he
supe iso placing a py amid in he eal scanned oom).
h ps://doi.o g/10.1371/jou nal.pone.0245976.g001
Fig 2. The s eps ollowed by he use s in he isual condi ion. The s udy wi h he isual condi ion is pe o med in wo phases:
lea ning and e alua ion. Du ing he lea ning phase, he subjec has o ollow a p ede ined pa h, sea ching o he eigh i ual
shapes placed on he desks in he oom. When a shape is ound, he subjec acknowledges i by apping he sc een. In he
e alua ion phase, subjec s a e eques ed o place each o he eigh i ual shapes in i s o iginal loca ion. Subjec s ha e h ee
a emp s o place each shape.
h ps://doi.o g/10.1371/jou nal.pone.0245976.g002
PLOS ONE
Augmen ed eali y o assess sho - e m spa ial memo y using isual and ac ile s imuli
PLOS ONE | h ps://doi.o g/10.1371/jou nal.pone.0245976 Feb ua y 4, 2021 6 / 30
h ee a emp s o place each shape co ec ly. They a e in o med o whe he o no hey ha e
placed he shape co ec ly in each y. On he las a emp , he shape emains ixed in he las
posi ion whe e i was placed, and he use is in o med whe he i was a success o ailu e. The
pa icipan s a e no helped ega ding whe e he shapes a e loca ed.
3.3. The ac ile condi ion
The ask ela ed o ac ile s imuli combines he use o he AR applica ion wi h ouching eal geo-
me ical shapes inside opaque boxes. The geome ical shapes used a e he ones al eady men-
ioned: cube, ec angula p ism, wide cylinde , na ow cylinde , sphe e, semi-sphe e, cone, and
py amid (Fig 3A). The dimensions o he geome ical shapes a e be ween 4 and 25 cm
2
a he
base and be ween 3 and 10 cm in heigh . The boxes a e closed. Thei size is 31x22x25 cm wi h a
capaci y o 18 li e s. The in e io o he box is accessed h ough wo small windows on he
lowe -le and lowe - igh sides, which a e co e ed wi h s ips o ab ic. The subjec s can in o-
duce hei wo hands inside he box and ouch he geome ical shapes, bu hey canno see
inside. Fig 3B shows he box. The e a e eigh boxes. Each o he eigh boxes had a di e en
shape inside, and each box was iden i ied using a ba code. The phases a e he same as in he
isual condi ion. Fig 4 g aphically shows he s eps ollowed by he use s in he ac ile condi ion.
The lea ning phase. The boxes a e placed in he oom in he same loca ions as he i ual
geome ical shapes we e placed in he isual condi ion. In o he wo ds, he box con aining he
eal sphe e is placed in he same loca ion whe e he i ual sphe e was in he isual condi ion.
The main goal o his phase is o allow he subjec o walk a ound he oom, o explo e he eal
geome ical shapes by ouch and o emembe hei loca ions. Since he eal shapes a e he
same as he i ual ones used by he isual condi ion, hey a e loca ed in he same loca ions.
The subjec s ollow a p e-se ou , and hey all pe o m he same ask. When a subjec eaches
a box, he o she pu s his/he hands in i and ouches he shape o iden i y i and o emembe
i s loca ion. The AR applica ion is used o scan he ba code and o s o e a imes amp o he
momen ha he shape was examined (Fig 3C).
The e alua ion phase. A e inishing he lea ning phase, he subjec is in i ed o lea e
he oom o a momen . Meanwhile he supe iso emo es e e y eal box om i s place and
piles hem al oge he on a wo king able in a gi en o de . The wo king able is ano he able
ha is no used in he lea ning phase. The subjec comes back o he oom again and s ands in
on o he wo king able. The supe iso akes a eal box (wo king box) and pu s i on he
wo king able. The subjec ouches i s shape by in oducing his o he hands inside o he
wo king box wi hou seeing he shape. Using he AR applica ion, he supe iso scans he ba -
code ha is a ached o he box being used, he eby in o ming he AR app o he shape inside.
Then, he supe iso gi es he AR applica ion o he subjec . The subjec has o place a i ual
box in he eal posi ion whe e he o she had ouched ha shape be o e. The e a e no physical
boxes on he ables. The i ual boxes a e only seen h ough he sc een o he mobile de ice.
While he use is pe o ming he placemen o he box, he supe iso akes he cu en
wo king box and pu s i on a new pile. Then, he supe iso akes he nex wo king box and
pu s i on he wo king able o he nex e alua ion s ep. When he use has placed he cu en
box a he desi ed place, he o she e u ns o he wo king able wi h he AR de ice. The p ocess
is epea ed o all eigh boxes. The o de is he same o all o he subjec s. Fig 3D shows a sub-
jec placing a i ual box in he oom.
3.4. The shape- ecogni ion applica ion
The shape- ecogni ion applica ion allows he pa icipan s o selec he shapes memo ized in
he p e ious asks om among di e en shapes. The pa icipan s a e asked o iden i y he
PLOS ONE
Augmen ed eali y o assess sho - e m spa ial memo y using isual and ac ile s imuli
PLOS ONE | h ps://doi.o g/10.1371/jou nal.pone.0245976 Feb ua y 4, 2021 7 / 30
PLOS ONE
Augmen ed eali y o assess sho - e m spa ial memo y using isual and ac ile s imuli
PLOS ONE | h ps://doi.o g/10.1371/jou nal.pone.0245976 Feb ua y 4, 2021 8 / 30
geome ical shapes ha hey ha e ouched o seen among a collec ion o di e en 3D shapes
ha appea on he sc een o a able (Fig 5A). To acili a e he ecogni ion o he shapes, hey
a e anima ed and o a e a ound hei e ical axes. The subjec s can selec and deselec as
many shapes as hey desi e. Once he subjec s ha e inished wi h he selec ion o shapes, hey
no i y he supe iso , who ends he es . The applica ion shows i een objec s on he sc een, o
Fig 3. Ma e ials used in he ac ile condi ion. (A) Physical geome ical shapes used in ou s udy. (B) Box used in he
lea ning and e alua ion phases o he ac ile condi ion. (C) An example o he AR applica ion o scanning he ba code.
(D) An example o a pa icipan placing a i ual box in he e alua ion phase using he AR applica ion.
h ps://doi.o g/10.1371/jou nal.pone.0245976.g003
Fig 4. The s eps ollowed by he use s in he ac ile condi ion. The s udy wi h he ac ile condi ion is pe o med in wo phases:
lea ning and e alua ion. Du ing he lea ning phase, he subjec has o ollow a p ede ined pa h wi h eigh physical boxes. Each
box has a di e en geome ical shape inside. Fo each box, he subjec has o inspec he shape inside by ouch h ough wo holes
in he sides o he box. Then, he AR applica ion is used o scan he ba code and o s o e a imes amp o he momen ha he
shape was examined. In he e alua ion phase, he boxes a e emo ed om hei ini ial posi ion and shown o he subjec one by
one. A e inspec ing i s shape, he subjec has o place a i ual box wi h he AR applica ion in he posi ion whe e he shape was
loca ed du ing he lea ning phase. The subjec s ha e h ee a emp s o place he i ual box co ec ly.
h ps://doi.o g/10.1371/jou nal.pone.0245976.g004
PLOS ONE
Augmen ed eali y o assess sho - e m spa ial memo y using isual and ac ile s imuli
PLOS ONE | h ps://doi.o g/10.1371/jou nal.pone.0245976 Feb ua y 4, 2021 9 / 30
1. Spa ial o ien a ion es . The s imuli in he spa ial o ien a ion es a e aised-line igu es made
up o one, wo, o h ee ec ilinea segmen s. Each segmen has a speci ic o ien a ion ( e i-
cal, ho izon al, o oblique). Each segmen is 3 cm long. Each es includes a p ac ice i em and
six i ems wi h inc easing complexi y. Ou es includes wo i ems wi h one segmen , wo
i ems wi h wo segmen s, and wo i ems wi h h ee segmen s. The subjec s mus explo e he
igu es wi h he index inge o hei dominan hand. We used he ins uc ions p oposed by
Mazella e al. [39]. Each i em consis s o a e e ence elemen and ou simila elemen s. The
pa icipan s mus inspec he e e ence elemen by ouch, and hen ouch he ollowing ou
elemen s one by one, wi hou being able o go back. Fo each o he ou elemen s, he pa ic-
ipan s mus indica e e bally whe he o no he igu e has he same spa ial o ien a ion as he
e e ence. The supe iso w i es down hei answe s wi hou gi ing any eedback. Two
poin s a e awa ded o each co ec ly comple ed i em. The maximum sco e is 12 poin s.
2. Shape disc imina ion es . The p o ocol is simila o he spa ial o ien a ion es . The only
di e ence is he ype o igu es, which a e geome ical shapes. The size o he shapes is
adjus ed so ha hey occupy he maximum size in an a ea o 9 cm
2
. These 2D shapes a e:
squa e, ec angle, hombus, sphe e, semi-sphe e, ellipse, pen agon, hexagon, igh iangle,
equila e al iangle, and i e-poin ed s a . Fig 7B shows an i em used in his es . As in he
p e ious es , he pa icipan s mus inspec he e e ence elemen by ouch. Then, o each
o he ou elemen s, he pa icipan s mus indica e e bally whe he o no he shape o he
igu e is he same as he e e ence elemen .
3. Spa ial loca ion es . The p o ocol is simila o he wo p e ious es s. The only di e ence is
he ype o igu es. The s imuli in his es a e aised-line igu es inside a ci cle (diame e = 4
cm). The ci cle may con ain one, wo, o h ee small elemen s inside (ci cle, s a , o squa e).
The size o hese small elemen s a ies om 7 o 10 mm. Th ee o he i ems ha e one inne
elemen , wo i ems ha e wo inne elemen s, and one i em has h ee inne elemen s. As in
p e ious es s, he pa icipan s mus inspec he e e ence elemen by ouch. Then, o each
o he ou elemen s, he pa icipan s mus indica e e bally whe he o no he spa ial loca-
ion o he inne elemen s is he same spa ial loca ion as in he e e ence elemen .
The swell pape es con ains a p ac ice i em and six i ems (Fig 7C). This es includes wo
i ems o spa ial o ien a ion, wo i ems o shape disc imina ion, and wo i ems o spa ial
loca ion. The p o ocol is simila o he h ee p e ious es s. Howe e , he e a e wo di e ences:
he ype o ma e ial used o build he es , and he o al numbe o i ems (6).
A e building a box plo wi h he sco es ob ained by he pa icipan s in all o he hap ic
es s, i e ou lie s we e iden i ied. These i e ou lie s we e emo ed om he sample (N = 48).
We checked whe he o no he e we e di e ences o he hap ic es ba e y and o gende .
To de e mine whe he o no he e we e di e ences o he o al sco e o he h ee plas ic es s
o ou ba e y be ween he g oup o women (Mdn = 28; IQR = 3.5) and he g oup o men
(Mdn = 28; IQR = 2), we applied he Mann-Whi ney U es (U = 200, Z = -.740, p = .467, =
.108). To de e mine whe he he e we e di e ences o he o al sco e o he swell pape es
sco e be ween he g oup o women (Mdn = 10; IQR = 0) and he g oup o men (Mdn = 10;
IQR = 4), we applied he Mann-Whi ney U es (U = 246.5, Z = .381, p = .712, = .056). These
esul s indica e ha he women and men o ou sample had he same ac ile abili ies.
We checked he o al sco e o he pa icipan s in he PTSOT es as an inclusion c i e ion
(Fig 7D). We used he p e iously published PTSOT sco es as he uppe limi o men and
women [30]. These uppe limi s a e 57.61˚ o men and 92.03˚ o women. One male was
excluded om he sample (N = 47). We checked whe he o no he e we e di e ences o he
PLOS ONE
Augmen ed eali y o assess sho - e m spa ial memo y using isual and ac ile s imuli
PLOS ONE | h ps://doi.o g/10.1371/jou nal.pone.0245976 Feb ua y 4, 2021 16 / 30

PLOS ONE
Augmen ed eali y o assess sho - e m spa ial memo y using isual and ac ile s imuli
PLOS ONE | h ps://doi.o g/10.1371/jou nal.pone.0245976 Feb ua y 4, 2021 17 / 30
PTSOT a iables and o gende . To de e mine whe he o no he e we e di e ences o he
PTSOT sco e be ween he g oup o women (12.65; 10.555) and he g oup o men (12.86;
11.56), we applied he Mann-Whi ney U es (U = 212, Z = -.442, p = .671, = .064). To analyze
he una emp ed i ems on he PTSOT o he g oup o women (33.333; 20.833) and he g oup
o men (33.333; 16.667), we applied he Mann-Whi ney U es (U = 243, Z = .284, p = .785, =
.041). These esul s indica e ha he women and men o ou sample had simila spa ial o ien-
a ion abili ies.
4.4. Measu es
Pe o mance wi h isual and ac ile condi ions. The isual applica ion and he ac ile
applica ion s o e he ollowing a iables: he o al numbe o shapes loca ed co ec ly du ing
he e alua ion phase (LocShapes); he o al numbe o a emp s made while placing he shape
in he co ec loca ion (A emp S); and he o al ime equi ed o comple e he lea ning phase
(TTimeL) and he e alua ion phase (TTimeE).
The shape- ecogni ion ask. We de eloped he shape- ecogni ion applica ion o es he
abili y o he pa icipan s o ecognize he shapes ha hey inspec ed du ing he ask. The shape-
ecogni ion ask s o es wo a iables: he numbe o shapes ha he pa icipan s selec ed co -
ec ly (RecogS), and he numbe o shapes ha he pa icipan selec ed inco ec ly (RecogI).
The map-poin ing ask. We de eloped he map-poin ing applica ion o es he abili y o
he pa icipan s o ead a bi-dimensional map o he oom in which he main ask wi h he
isual/ ac ile applica ion occu ed. The map-poin ing applica ion s o ed one a iable: he num-
be o shapes ha he pa icipan placed co ec ly on he map (MapShapes). A shape is consid-
e ed o be co ec ly loca ed on he map when he pa icipan places i wi hin a adius o wha
would be hal a me e in he eal wo ld, which is he equi alen o 3 cm on he i ual map.
Ques ionnai es. The pa icipan s illed ou a ques ionnai e abou hei subjec i e expe i-
ence wi h he AR applica ion. The ques ionnai e consis s o 21 ques ions ha we g oup in o
he ollowing a iables: enjoymen , concen a ion, usabili y, compe ence, calmness, expe ise,
non-men al e o , non-physical e o , p esence, and sa is ac ion. This ques ionnai e was
designed speci ically o his s udy, and some o he ques ions we e adap ed om commonly
used ques ionnai es [40–43] and based on ou p e ious expe iences [31,44]. The i ems we e
on a 7-poin Like scale, anging om 1 “To ally disag ee” o 7 “To ally ag ee”. All o he
i ems we e o mula ed in a posi i e manne excep o i ems #9 and #10. Sco es o he a i-
ables we e ob ained by calcula ing he mean alue o he associa ed ques ions. This ques ion-
nai e is a ailable in he S1 Appendix.
4.5. P ocedu e
The s udy was ca ied ou in a oom o 37.5 squa e me e s. The oom had he u ni u e com-
monly ound in an o ice o in a uni e si y labo a o y. Fig 5C shows a schema ic op iew o
he oom and he loca ion o he eigh i ual geome ical shapes.
The pa icipan s we e in ol ed in he wo condi ions ( ac ile and isual). Bo h condi ions
we e es ed in sessions ha we e held mo e han wo mon hs apa . A e mo e han wo
mon hs, i was e y di icul o he pa icipan s o emembe he loca ion o he shapes.
Fig 7. Tes s used as inclusion c i e ia. (A) S uc u e o p e en he pa icipan om seeing he i ems used in he
ba e y o hap ic es s. (B) An example o one i em included in he shape disc imina ion es (p in ed using ABS whi e
ma e ial). (C) An example o one shape disc imina ion i em included in he swell pape es . (D) An example o a use
in e ac ing wi h he PTSOT applica ion.
h ps://doi.o g/10.1371/jou nal.pone.0245976.g007
PLOS ONE
Augmen ed eali y o assess sho - e m spa ial memo y using isual and ac ile s imuli
PLOS ONE | h ps://doi.o g/10.1371/jou nal.pone.0245976 Feb ua y 4, 2021 18 / 30
Fu he mo e, he oom was no amilia o he pa icipan s since hey had no had p e ious
access be o e he s udy. The wo condi ions we e he ollowing:
• The ac ile condi ion: pa icipan s who lea n he loca ion o he physical geome ical shapes
ha we e placed inside o physical boxes dis ibu ed h oughou he oom. In he lea ning
phase, he boxes we e placed in he oom as shown in Fig 3D. The pa icipan s had o ouch
he shapes inside he boxes. The applica ion was used o s o e which shapes we e ouched
and when. In he e alua ion phase, he boxes we e no dis ibu ed h oughou he oom. The
boxes we e all on a able. The use s had o ouch he shapes one by one, in he same o de .
A e ouching a physical geome ical shape, he use s we e asked o place a i ual box using
he AR applica ion in he loca ion ha hey hough hey had ouched he eal objec in he
lea ning phase.
• The isual condi ion: pa icipan s who lea n he loca ion o he i ual shapes placed in he
eal oom using ou AR applica ion. The pa icipan s used he AR applica ion o he lea n-
ing and e alua ion phases. The loca ion o he geome ical shapes ( eal o i ual) was he
same in bo h condi ions (Fig 5C).
The adminis a ion p o ocol was comp ised o wo sessions. In he i s session, he pa ici-
pan s comple ed he ba e y o hap ic es s and he PTSOT es . Then, hey pe o med he asks
in he ac ile condi ion: he sho - e m memo y ask o he loca ion o geome ical shapes, he
shape- ecogni ion ask using he shape- ecogni ion applica ion, and he map-poin ing ask
using he map-poin ing applica ion. A e mo e han wo mon hs, he pa icipan s comple ed
he asks again using he isual condi ion: he sho - e m memo y ask wi h he AR applica ion,
he shape- ecogni ion ask, and he map-poin ing ask. Finally, he pa icipan s illed ou an
online ques ionnai e abou hei subjec i e pe cep ion ega ding he use o he AR applica ion.
5. Resul s
5.1. Pe o mance ou comes
We compa ed he pe o mance ou comes be ween he wo condi ions (Tac ileCondi ion s.
VisualCondi ion) (wi hin-subjec s analysis) o de e mine how he use o ac ile o isual s i-
muli a ec s memo y o he loca ion o s imuli. Fig 8 shows box plo s o he pe o mance ou -
come a iables and o he ac ile and isual condi ions. Fi s , we analyzed he a iable ha
indica es he o al numbe o successes o he eigh s imuli used. This a iable coun s he
numbe o shapes placed co ec ly (up o eigh shapes). To de e mine whe he o no he e
we e di e ences in emembe ing and placing shapes in hei co ec loca ions be ween Tac ile-
Condi ion (Mdn = 7; IQR = 1) and VisualCondi ion (Mdn = 8; IQR = 1), we applied he Wil-
coxon Signed- ank es (W= 123, Z= -1.739, p= .098, = .179). This esul indica es ha he e
a e no s a is ically signi ican di e ences be ween he wo condi ions.
Second, we analyzed he a iable ha ep esen s he o al numbe o a emp s made when
placing he shape in he co ec loca ion. No e ha a maximum o h ee a emp s we e allowed
o place a shape in he co ec loca ion. To de e mine whe he o no he e we e di e ences o
his a iable be ween Tac ileCondi ion (Mdn = 5; IQR = 4) and VisualCondi ion (Mdn = 3;
IQR = 5), we applied he Wilcoxon Signed- ank es (W= 492, Z= 2.860, p= .004, = .295).
This esul indica es ha he e a e signi ican di e ences be ween he wo condi ions in a o
o he isual condi ion, which equi ed ewe a emp s o place he shape co ec ly.
Thi d, we analyzed he a iable ha ep esen s he o al ime equi ed o comple e he
lea ning phase. To de e mine whe he o no he e we e di e ences o his a iable be ween
Tac ileCondi ion (Mdn = 123.54; IQR = 44.64) and VisualCondi ion (Mdn = 128.02;
PLOS ONE
Augmen ed eali y o assess sho - e m spa ial memo y using isual and ac ile s imuli
PLOS ONE | h ps://doi.o g/10.1371/jou nal.pone.0245976 Feb ua y 4, 2021 19 / 30
IQR = 56.84), we applied he Wilcoxon Signed- ank es (W= 454.5, Z= -1.159, p= .249, =
.120). This esul indica es ha he e a e no signi ican di e ences be ween he wo condi ions.
Fou h, we analyzed he a iable ha ep esen s he o al ime equi ed o comple e he
e alua ion phase. To de e mine whe he o no he e we e di e ences o his a iable be ween
Tac ileCondi ion (Mdn = 356.23; IQR = 132.325) and VisualCondi ion (Mdn = 156.11;
IQR = 130.44), we applied he Wilcoxon Signed- ank es (W= 1127, Z= 5.958, p<.001, =
.614). This esul indica es ha he e a e signi ican di e ences be ween he wo condi ions in
a o o he g oup wi h isual s imuli, who spen less ime o comple e he e alua ion phase.
We also analyzed he a iable o he shape- ecogni ion es ha ep esen s he numbe o
shapes ha he pa icipan emembe s a e pe o ming he ask. To de e mine whe he he e
Fig 8. Box plo s o he pe o mance ou come a iables and o he ac ile and isual condi ions. (A) To al numbe o shapes placed co ec ly (LocShapes a iable).
(B) To al numbe o a emp s equi ed o co ec ly place he shapes (A emp S a iable). (C) To al ime in seconds equi ed o comple e he lea ning phase (TTimeL
a iable). (D) To al ime in seconds equi ed o comple e he e alua ion phase (TTimeE a iable).
h ps://doi.o g/10.1371/jou nal.pone.0245976.g008
PLOS ONE
Augmen ed eali y o assess sho - e m spa ial memo y using isual and ac ile s imuli
PLOS ONE | h ps://doi.o g/10.1371/jou nal.pone.0245976 Feb ua y 4, 2021 20 / 30
we e di e ences o his a iable be ween Tac ileCondi ion (Mdn = 8; IQR = 0) and Visual-
Condi ion (Mdn = 8; IQR = 0), we applied he Mann-Whi ney U es (W= 28, Z= -1.387, p=
.178, = .143). This esul indica es ha he e a e no signi ican di e ences be ween he wo
condi ions. We also analyzed he a iable ha ep esen s he numbe o shapes ha he pa ici-
pan selec ed inco ec ly o Tac ileCondi ion (Mdn = 0; IQR = 0) and VisualCondi ion
(Mdn = 0; IQR = 0), and we applied he Wilcoxon Signed- ank es (W= 15, Z= 2.236, p=
.037, = .231). This esul indica es ha he e a e signi ican di e ences be ween he wo con-
di ions in a o o he g oup wi h isual s imuli, which selec ed ewe inco ec shapes.
We also analyzed he a iable ha ep esen s he numbe o shapes ha he pa icipan
placed co ec ly in he map-poin ing ask. To de e mine whe he he e we e di e ences o
his a iable be ween Tac ileCondi ion (Mdn = 4; IQR = 3) and VisualCondi ion (Mdn = 6;
IQR = 1), we applied he Mann-Whi ney U es (W= 127, Z= -4.250, p<.001, = .438). This
esul indica es ha he e a e signi ican di e ences be ween he wo condi ions in a o o he
g oup wi h isual s imuli, which placed mo e shapes co ec ly.
5.2. Gende analysis and subjec i e pe cep ion
To de e mine i gende in luences he a iable ha indica es he numbe o co ec ly selec ed
shapes, we analyzed he g oup o women (Mdn = 8; IQR = 1) and he g oup o men (Mdn = 7;
IQR = 2) o Tac ileCondi ion, and we applied he Mann-Whi ney U es (U= 289, Z= 1.438,
p= .154, = .210). Fo VisualCondi ion in he g oup o women (Mdn = 8; IQR = 1) and he
g oup o men (Mdn = 8; IQR = 1), we also applied he Mann-Whi ney U es (U= 221, Z=
-.262, p= .803, = .038).
We hen analyzed he o al numbe o a emp s. Fo Tac ileCondi ion in he g oup o
women (Mdn = 4; IQR = 3.5) and he g oup o men (Mdn = 5; IQR = 4), we applied he
Mann-Whi ney U es (U= 151, Z= -1.874, p= .063, = .273). Fo VisualCondi ion in he
g oup o women (Mdn = 3; IQR = 5) and he g oup o men (Mdn = 3; IQR = 6), we applied
he Mann-Whi ney U es (U= 235, Z= .094, p= .934, = .014).
We analyzed he ime equi ed o comple e he e alua ion phase. Fo Tac ileCondi ion in he
g oup o women (Mdn = 343.915; IQR = 64.173) and he g oup o men (Mdn = 395.23; IQR = 162),
we applied he Mann-Whi ney U es (U= 203, Z= -.651, p= .527, = .095). Fo VisualCondi ion
in he g oup o women (Mdn = 166.255; IQR = 105.653) and he g oup o men (Mdn = 123.83;
IQR = 132.58), we applied he Mann-Whi ney U es (U= 252, Z= .489, p= .637, = .071).
Since no s a is ically signi ican di e ences we e ound in any o hese analyses, we can con-
clude ha he pe o mance esul s we e independen o he pa icipan s’ gende .
The ques ionnai e abou subjec i e expe ience was used o measu e he pa icipan s’ subjec-
i e pe cep ion o he AR applica ion and hei pe o mance in he Visual Condi ion. The
ques ions we e g ouped in he ollowing a iables: enjoymen , concen a ion, usabili y, com-
pe ence, calmness, expe ise, non-men al e o , non-physical e o , sa is ac ion, and p esence.
No s a is ically signi ican di e ences we e ound in any o hese a iables; he e o e, we can
conclude ha he subjec i e pe cep ion was independen o he pa icipan s’ gende
(U �169.5, Z�-1.513, p �.134).
5.3. Co ela ions
This sec ion p esen s he signi ican co ela ions ob ained a e he applica ion o he Spea -
man ank co ela ion and he Bon e oni co ec ion o he isual condi ion. Fo he ac ile
condi ion, no signi ican co ela ions we e ound be ween he AR spa ial ask and o he asks.
We used he Spea man ank co ela ion o es he associa ions among a iables o he pe -
o mance in he isual condi ion (LocShapes, A emp S) and among a iables in he shape-
PLOS ONE
Augmen ed eali y o assess sho - e m spa ial memo y using isual and ac ile s imuli
PLOS ONE | h ps://doi.o g/10.1371/jou nal.pone.0245976 Feb ua y 4, 2021 21 / 30

ecogni ion ask (RecogS) and he map-poin ing ask (MapShapes). As he e a e mul iple com-
pa isons, he Bon e oni co ec ion was applied. I we apply he Bon e oni co ec ion o he
ma ix o med by hese ou a iables, LocShapes co ela es wi h MapShapes ( = .49, p<.01),
and A emp S also co ela es wi h MapShapes ( = -.48, p<.01). This sensibly means ha pa -
icipan s who placed ewe shapes co ec ly du ing he e alua ion phase o equi ed mo e
a emp s o place he shapes co ec ly also made mo e mis akes on he map-poin ing ask. Fig
9A and 9B show hese wo co ela ions g aphically.
I we apply he Bon e oni co ec ion o he ma ix o med by he ou a iables (TTimeL,
TTimeE, LocShapes, and A emp S), he TTimeE co ela es wi h he LocShapes a iable ( =
-.51, p<.01) and wi h he A emp S a iable ( = .48, p<.01). This means ha pa icipan s
who equi ed mo e ime du ing he e alua ion phase placed ewe shapes co ec ly and
equi ed mo e a emp s. Fig 9C and 9D show hese wo co ela ions g aphically.
We also used he Spea man ank co ela ion o es he associa ions among he en subjec-
i e a iables (enjoymen , concen a ion, e c.) and he a iables o he pe o mance ou comes
(LocShapes and A emp S). I we apply he Bon e oni co ec ion o he ma ix o med by
hese 12 a iables, he LocShapes a iable co ela es wi h pe cei ed compe ence ( = .49, p =
.03), and he A emp S a iable co ela es nega i ely wi h pe cei ed compe ence ( = -.50, p=
.03). This means ha pa icipan s who placed a g ea e numbe o shapes co ec ly el g ea e
compe ence. In con as , he g ea e he numbe o a emp s, he less he pe cei ed compe-
ence. Fig 9E and 9F show hese wo co ela ions g aphically.
I we apply he Bon e oni co ec ion o he ma ix o med by he en subjec i e a iables,
he e is only a signi ican co ela ion ha co esponds o he enjoymen and p esence a iables
( = .47, p= .04). This means ha pa icipan s who expe ienced mo e enjoymen epo ed a
highe sense o p esence. Fig 9G shows his co ela ion g aphically.
6. Discussion
In s udies ela ed o wo king o sho - e m spa ial memo y, mainly isual o audi o y modali-
ies ha e been conside ed [20,26,29,32,33]. In his wo k, we con ibu e wi h a s udy in which
he ac ile modali y is conside ed. To ou knowledge, his is he i s s udy in which ac ile s i-
muli a e used o he assessmen o sho - e m spa ial memo y o shape-loca ion in a h ee-
dimensional a ay. In ou s udy, he pa icipan s na iga ed he en i onmen o lea n he spa ial
loca ion o each ac ile s imulus.
The e a e subs an ial di e ences in e ms o he s udies ca ied ou in ou p e ious AR
wo ks and he s udy p esen ed in his pape . In he s udy o Munoz-Mon oya e al. [31], only
ou objec s we e used and he pa icipan s we e di ided in o wo g oups in which he lea ning
phase was di e en (AR s. Non-AR). The pa icipan s who used AR lea ned he placemen o
he i ual elemen s in he eal wo ld using AR. The pa icipan s who used Non-AR lea ned
he loca ion o he i ual elemen s h ough pho og aphs. The augmen ed pho og aphs showed
he eal en i onmen wi h he i ual objec s supe imposed on i . In he s udy o Munoz-Mon-
oya e al. [30], eigh i ual objec s we e used (ho se, wall clock, elephone, oy ca , bus , oun-
ain pen, sailboa , and sc ewd i e ). The objec s we e dis ibu ed in a building on wo di e en
loo s. Tha s udy analyzed he co ela ions be ween he esul s o he pe o mance ou comes
wi h he AR app aking in o accoun gende and di e en measu es o emo ional and spa ial
ac o s. These measu es we e he le els o ai anxie y, way inding anxie y, and di e en spa-
ial s a egies o way inding.
F om ou esul s, he e was no signi ican di e ence o he numbe o shapes placed co -
ec ly be ween he wo condi ions ( isual s. ac ile). Howe e , he pa icipan s in he isual
condi ion equi ed signi ican ly less ime o comple e he e alua ion phase and also equi ed
PLOS ONE
Augmen ed eali y o assess sho - e m spa ial memo y using isual and ac ile s imuli
PLOS ONE | h ps://doi.o g/10.1371/jou nal.pone.0245976 Feb ua y 4, 2021 22 / 30
PLOS ONE
Augmen ed eali y o assess sho - e m spa ial memo y using isual and ac ile s imuli
PLOS ONE | h ps://doi.o g/10.1371/jou nal.pone.0245976 Feb ua y 4, 2021 23 / 30
signi ican ly ewe a emp s. The e was a signi ican di e ence in he numbe o shapes ha
he pa icipan s placed co ec ly in he map-poin ing ask a e pe o ming he ask in a ou
o he isual condi ion. The e was no signi ican di e ence o he shape- ecogni ion es ,
which ep esen s he numbe o shapes ha he pa icipan s emembe a e pe o ming he
ask be ween he wo condi ions ( isual s. ac ile). Howe e , he a iable ha ep esen s he
numbe o shapes ha he pa icipan s selec ed inco ec ly showed ha he pa icipan s in he
isual condi ion selec ed ewe inco ec shapes. F om all o hese esul s, we can conclude ha
isual and ac ile s imuli can be used o assess spa ial memo y and ha isual s imuli posi i ely
a ec s sho - e m, spa ial memo y ou comes. This suppo s p e ious esea ch s a ing ha he
sense o sigh is he dominan sense in humans, and, consequen ly, his dominance acili a es
he lea ning o spa ial- isual associa ions [45,46]. Ou esul s also indica e ha isual and ac-
ile s imuli a e sui able o he assessmen o spa ial memo y in na iga ion ega dless o he
gende o he pa icipan s. These esul s a e in line wi h p e ious wo ks [20].
The ollowing co ela ions we e iden i ied be ween he o al numbe o shapes co ec ly
emembe ed and loca ed using isual and ac ile s imuli (LocShapes) and he o al numbe o
shapes co ec ly placed in he poin ing-map ask (MapShapes). Fo he Visual Condi ion, Loc-
Shapes co ela ed wi h MapShapes. This esul shows ha spa ial- isual associa ions we e
lea ned and ans e ed om he h ee-dimensional a ay o he eal oom o he bi-dimen-
sional a ay o he map. In con as , spa ial- ac ile associa ions we e lea ned, bu ailed o
achie e his ans e . These esul s a e in line wi h p e ious wo ks, in which he memo y o
he loca ion o i ual objec s in he isual modali y in a na iga ional space co ela ed posi-
i ely wi h he poin ing o hese objec s on a map [30].
We also s udied he co ela ions be ween he subjec i e pe cep ion and he pe o mance
measu es o he pa icipan s o he AR spa ial ask (Visual Condi ion). The o al numbe o
shapes (LocShapes) co ela ed wi h pe cei ed compe ence, which indica es ha he g ea e he
numbe o co ec ly placed shapes, he g ea e he pe cei ed compe ence. The A emp S a i-
able (a emp s equi ed o placing he shapes) co ela ed nega i ely wi h pe cei ed compe ence.
This co ela ion indica es ha he mo e a emp s equi ed, he less compe ence was pe cei ed.
Fo he AR applica ion, he subjec i e pe cep ion o he pa icipan s was independen o
hei gende . On a scale om 1 o 7, he medians we e e y high: equal o o abo e 6 in all
cases (excep o one wi h a alue o 5.3). The co ela ions among he subjec i e a iables we e
analyzed o he whole sample, and hi een posi i e co ela ions we e ound. I is wo h
highligh ing ha he sense o p esence co ela ed wi h he enjoymen a iable. The pa ici-
pan s ha epo ed a highe sense o p esence also expe ienced mo e enjoymen .
The de eloped amewo k allows di e en AR engines/SDKs o be used. Fo example, hese
di e en AR engines could be ARCo e o ARKi , which a e w apped in a module wi h he same
in e ace o e ing, and he e o e ha e he same unc ionali ies. This design allows he es o he
Fig 9. Sca e plo s o signi ican co ela ions o he isual condi ion (n = 47). (A) Sca e plo o he o al
numbe o shapes placed co ec ly (LocShapes a iable) and he sco e ob ained in he map-poin ing ask (MapShapes
a iable). (B) Sca e plo o he o al numbe o a emp s o place he shapes co ec ly (A emp S a iable) and he
sco e ob ained in he map-poin ing ask (MapShapes a iable). (C) Sca e plo o he o al numbe o shapes placed
co ec ly (LocShapes a iable) and he ime spen in he e alua ion phase (TTimeE). (D) Sca e plo o he o al
numbe o a emp s o place he shapes co ec ly (A emp S a iable) and he ime spen in he e alua ion phase
(TTimeE). (E) Sca e plo o he o al numbe o shapes placed co ec ly (LocShapes a iable) and he pe cei ed
compe ence a iable. (F) Sca e plo o he o al numbe o a emp s o place he shapes co ec ly (A emp S a iable)
and he pe cei ed compe ence a iable. (G) Sca e plo o he p esence and enjoymen a iables. Th ee sizes o ci cles
appea in he plo s ha ep esen a low numbe o occu ences, an in e media e numbe o occu ences, and a high
numbe o occu ences. The dashed ed lines a e bes i ing linea eg ession lines. In panels C, D and G close alues
we e g ouped o clea e isualiza ion. The g ey a ea ep esen s a 95% con idence le el in e al o p edic ions om a
linea model.
h ps://doi.o g/10.1371/jou nal.pone.0245976.g009
PLOS ONE
Augmen ed eali y o assess sho - e m spa ial memo y using isual and ac ile s imuli
PLOS ONE | h ps://doi.o g/10.1371/jou nal.pone.0245976 Feb ua y 4, 2021 24 / 30
applica ion o be independen o speci ic AR SDKs. To ake in o accoun ha di e en AR
engines may ha e di e en unc ionali ies, we ha e de ined a se o basic unc ionali ies. Ano he
se o op ional unc ionali ies was also de ined, which can be ac i a ed o deac i a ed depending
on he engine used. We ha e ied o minimize his las se o unc ionali ies so ha he impac
on he es o he applica ion code is minimized. A his ime, he Tango SDK could be changed
o ano he equi alen engine wi hou al e ing he applica ion code. Un il he elease o ARKi
3.5 in Ma ch 2020, nei he ARKi no ARCo e o e ed enough unc ionali ies o a o d he de el-
opmen o applica ions like he one p esen ed in his wo k. ARKi 3.5 inco po a es he possibil-
i y o using he new LiDAR scanne and dep h-sensing sys em ha a e buil in o he iPad P o.
Thanks o hese new senso s, i is possible o use he geome y o he scene, jus as is done wi h
Tango SDK. The e o e, applica ions such as hose p esen ed in his wo k can be c ea ed by
using ARKi in suppo ed de ices. Tango SDK has been dep eca ed since Ma ch, 2018. Since
hen, wi h he discon inua ion o compa ible de ices on he ma ke (Leno o Phab 2 P o and
ASUS Zen one AR), he e has been a gap ha was no co e ed. The e o e, he appea ance o he
iPad P o is especially ele an because i opens up many possibili ies o de elope s.
Ou p oposal and o he simila ools can g ea ly help in aining sho - e m spa ial mem-
o y. Di icul ies in spa ial memo y a e usually associa ed o diso de s o diseases (e.g., [47,
48]). Fo example, se e al s udies ha e epo ed ha hippocampal damage may p o oke
impai men s in sho - e m memo y (e.g., Koen e al. [47]). Pa ien s wi h amnes ic mild cogni-
i e impai men ha e di icul y emembe ing whe e he objec s a e loca ed (e.g., Hamps ead
e al. [48]). In an ongoing wo k, pa ien s wi h acqui ed b ain inju y pe o m a ask wi h isual
s imuli wi h ou AR applica ion. F om he p elimina y esul s, we ha e obse ed ha hese
pa ien s can pe o m he ask wi h he AR applica ion. The e o e, ou AR applica ion could be
used as a help ool o g oups wi h impai men s in sho - e m memo y.
This wo k s udies he di e ences in seeing and ouching objec s and ecognizing hei
shapes by ouch. To ou knowledge, no wo k has been p esen ed in which a hap ic sys em
allows he eco ding o use esponses ela ed o a na iga ional ask o he lea ning o spa ial-
ac ile associa ions. Howe e , di e en comme cial hap ic de ices o p o o ypes de eloped by
esea che s ha e been p esen ed o hei use in VR and help use s o ha e a highe le el o
p esence. Dangxiao e al. [49] classi ied he pa adigm o hap ic HCI in o h ee s ages (desk op
hap ics, su ace hap ics, and wea able hap ics). They p esen ed a axonomy o desk op hap ic
de ices and also a classi ica ion o ac ile eedback de ices. One o he mos widely used wea -
able hap ic de ices a e hap ic glo es. These glo es can include hand mo ion acking and p o-
ide dis ibu ed o ce and ac ile eedback in he palm and in he inge ips. They compa ed 10
exis ing comme cial hap ic eedback glo es. Exis ing comme cial VR con olle s (e.g., HTC
Vi e) only p o ide global ib o ac ile eedback, which g ea ly limi s he hap ic expe ience. The
exis ing hap ic echnology does no allow a use o ouch a i ual objec and ecognize i s
shape. An open challenge is how o e icien ly include localized and di e se spa ial- empo al
ib o ac ile pa e ns, he mal eedback, ex u e eedback, con ac , so ness, skin igh ening, e c.
in a hap ic de ice [49]. In he s udy ca ied ou in his wo k, he pa icipan s ouched eal
objec s, and he use o hap ic de ices is p oposed as u u e wo k.
We buil a hap ic es ba e y o assess ac ile abili ies. We used his ba e y as an inclusion
c i e ion in o de o assu e ha he pa icipan s in ol ed in ou s udy had simila ac ile abili-
ies. We checked whe he o no he e we e di e ences in he hap ic es ba e y (plas ic and
swell pape ) and o gende . The esul s indica ed ha he hap ic es ba e y was independen
o he pa icipan s’ gende .
We de eloped applica ions o he asks and es s ha a e adi ionally done on pape . Spe-
ci ically, we de eloped he shape- ecogni ion applica ion, he map-poin ing-applica ion, and
he PTSOT applica ion. The supe iso ’s wo k is acili a ed by being able o pe o m hese
PLOS ONE
Augmen ed eali y o assess sho - e m spa ial memo y using isual and ac ile s imuli
PLOS ONE | h ps://doi.o g/10.1371/jou nal.pone.0245976 Feb ua y 4, 2021 25 / 30