Recei ed Janua y 31, 2021, accep ed Feb ua y 5, 2021, da e o publica ion Feb ua y 11, 2021, da e o cu en e sion Feb ua y 23, 2021.
Digi al Objec Iden i ie 10.1109/ACCESS.2021.3058664
In elligen Bio eedback Augmen ed Con en
Comp ehension (TellBack)
HAYTHAM HIJAZI 1, RICARDO COUCEIRO1, JOÃO CASTELHANO2, PAULO DE CARVALHO1,
MIGUEL CASTELO-BRANCO2, AND HENRIQUE MADEIRA1
1Cen e o In o ma ics and Sys ems, Uni e si y o Coimb a (CISUC), 3000-214 Coimb a, Po ugal
2Coimb a Ins i u e o Biomedical Imaging and T ansla ional Resea ch (CIBIT), Ins i u e o Nuclea Sciences Applied o Heal h (ICNAS), Uni e si y o Coimb a,
3000-214 Coimb a, Po ugal
Co esponding au ho : Hay ham Hijazi ([email p o ec ed])
This wo k was pa ially unded by he BASE p ojec unde G an POCI - 01-0145 - FEDER- 031581, in pa by he Cen o de In o m ica e
Sis emas da Uni e sidade de Coimb a (CISUC), and also in pa by Coimb a Ins i u e o Biomedical Imaging and T ansla ional Resea ch
(CIBIT), Ins i u e o Nuclea Sciences Applied o Heal h (ICNAS), Uni e si y o Coimb a unde G an PTDC/PSI-GER/30852/2017 |
CONNECT-BCI.
ABSTRACT Assessing comp ehension di icul ies equi es he abili y o assess cogni i e load. Changes
in cogni i e load induced by comp ehension di icul ies could be de ec ed wi h an adequa e ime eso-
lu ion using di e en bio eedback measu es (e.g., changes in he pupil diame e ). Howe e , iden i ying
he Spa io- empo al sou ces o con en comp ehension di icul ies (i.e., when, and whe e exac ly he
di icul y occu s in con en egions) wi h a ine g anula i y is a big challenge ha has no been explici ly
add essed in he s a e-o - he-a . This pape p oposes and e alua es an inno a i e app oach named In elligen
Bio eedbackAugmen ed Con en Comp ehension (TellBack) o explici ly add ess his challenge. The goal
is o au onomously iden i y egions o digi al con en ha cause use ’s comp ehension di icul y, opening he
possibili y o p o ide eal- ime comp ehension suppo o use s. TellBack is based on assessing he cogni i e
load associa ed wi h con en comp ehension h ough non-in usi e cheap bio eedback de ices ha acqui e
measu es such as pupil esponse o Hea Ra e Va iabili y (HRV). To iden i y when exac ly he di icul y in
comp ehension occu s, physiological mani es a ions o he Au onomic Ne ous Sys em (ANS) such as he
pupil diame e a iabili y and he modula ion o HRV a e exploi ed, whe eas he ine spa ial esolu ion (i.e.,
he egion o con en whe e he use is looking a ) is p o ided by eye- acking. The e alua ion esul s o his
app oach show an accu acy o 83.00% ±0.75 in classi ying egions o con en as di icul o no di icul
using Suppo Vec o Machine (SVM), and p ecision, ecall, and mic o F1-sco e o 0.89, 0.79, and 0.83,
espec i ely. Resul s ob ained wi h 4 o he classi ie s, namely Random Fo es , k-nea es neighbo , Decision
T ee, and Gaussian Nai e Bayes, showed a sligh ly lowe p ecision. TellBack ou pe o ms he s a e-o - he-a
in p ecision & ecall by 23% and 17% espec i ely.
INDEX TERMS Biomedical measu emen , cogni i e load, con en comp ehension, eye- acking, hea a e
a iabili y, machine lea ning.
I. INTRODUCTION
Imagine a so wa e echnology ha augmen s you abili y o
comp ehend complex concep s and ideas. You a e eading a
sen ence in a echnical documen and he so wa e ins alled
in you able , lap op, o sma phone au oma ically de ec s
speci ic passages o wo ds in a pa ag aph ha make he
en i e pa ag aph cumbe some o you and p omp ly displays
an explana ion, shows an example, o p o ides you wi h a
de ini ion ha will make you unde s and he whole idea.
We call his app oach InTelligen Bio eedBack Augmen ed
The associa e edi o coo dina ing he e iew o his manusc ip and
app o ing i o publica ion was Giuseppe Desolda .
Con en Comp ehension (TellBack), and he p esen pape
p oposes he idea and e alua es he accu acy o TellBack
in he iden i ica ion o speci ic con en egions (e.g., line,
exp ession, e c.) ha a e conside ed di icul o unde s and
by he use . The accu a e iden i ica ion o con en loca ions
ha cause use ’s comp ehension di icul ies is a c ucial s ep
o show he easibili y o he p oposed app oach.
TellBack uses cogni i e load as a key elemen o iden i y
speci ic pa s o digi al con en ha cause comp ehension
di icul y. The goal is o p edic ha he use is needing
suppo in unde s anding he con en elemen /passage ha
is causing comp ehension di icul ies while eading digi al
con en .
VOLUME 9, 2021 This wo k is licensed unde a C ea i e Commons A ibu ion 4.0 License. Fo mo e in o ma ion, see h ps://c ea i ecommons.o g/licenses/by/4.0/ 28393
H. Hijazi e al.: In elligen Bio eedback Augmen ed Con en Comp ehension (TellBack)
Digi ized eading esou ces a e immensely g owing. Due
o he popula i y o handheld po able de ices, such as
e- eade s and able s, eading is becoming mo e on-sc een
han on p in ed ma e ial. Howe e , a ious s udies, such
as [1], showed ha on-sc een eading is associa ed wi h sho
a en ion and poo comp ehension. Comp ehension o digi al
esou ces dese es mo e esea ch a en ion, especially wi h
he e e -g owing eliance on digi al esou ces comp ehen-
sion in lea ning, aining, and indus y. Fo ins ance, in he
so wa e indus y, code e iew is one o he essen ial asks
in so wa e enginee ing o ensu e he quali y o he deli e ed
se ice. Code e iew equi es ull code comp ehension by he
e iewe . The code and i s desc ip ion a e gene ally digi ized
esou ces ha equi e he e iewe o immensely ead and
comp ehend.
Digi al esou ces comp ehension equi es he abili y o
use s o lea ne s o in eg a e and e alua e in o ma ion ac oss
di e en ypes o ep esen a ions. The in eg a ion and e alua-
ion a e o en men ally demanding asks, which impose ex a
cogni i e load on lea ne s.
Comp ehension, in gene al, is a highe cogni i e p ocess
ha imposes a cogni i e load and in ol es di e en cog-
ni i e s a es, which we hypo hesize in TellBack ha hese
s a es can be accu a ely cap u ed by non-in usi e bio eed-
back de ices h ough physiological mani es a ions o he
Au onomous Ne ous Sys ems (ANS). In he con ex o his
pape , bio eedback is de ined as he p ocess o p o iding
ask- ele an eedback o he use based on he /his cogni i e
s a e (cap u ed h ough physiological ANS eac ions) using
comme cially a ailable wea ables such as b acele s, wa ches,
and ings.
Ou main con ibu ions in his wo k can be summa ized by
he ollowing poin s:
•The pape in oduces a new echnique o de ec he cog-
ni i e load and he eby he comp ehension di icul ies a
elemen al pa s o he con en (e.g., lines, exp essions,
e c.) and no a global assessmen o cogni i e load as
in es iga ed in he s a e-o - he-a .
•The use o eye- acking is no o assess he cogni i e load
as i is gene ally used in he s a e-o - he-a (which has
he disad an age o a slow esponse ime) bu o p o ide
an accu a e spa ial esolu ion whe e he use is looking
a when he HRV and pupillome y bioma ke s indica e
peaks in he cogni i e load.
•The e alua ion o he imp o emen s in p ecision and
ecall achie ed wi h TellBack, when compa ed wi h a
e y ecen ela ed s udy in [12], shows 23% be e
p ecision and 17% be e ecall.
•This wo k add esses he limi a ions o a single modal-
i y using da a usion o non-in usi e biosenso s and
eal- ime bioma ke s om HRV and pupillome y.
Unlike o he ecen s udies [2], [3], [12] ha exploi he
mul imodali y o biosenso s, we used simple ea u es (in
e ms o compu a ion), bu disc iminan enough o de ec
he inc ease o he use ’s cogni i e load/men al e o in
eal- ime.
•The use o machine lea ning and AI echniques o op i-
mize he p edic ion o ‘‘when and whe e’’ he use is
encoun e ing di icul y on sc een.
The inal goal o Tellback is o augmen he use ’s abili y o
comp ehend complex concep s and ideas h ough a new ype
o in elligen in e ace ha seems o guess when and whe e
he use is making an unusual e o o g asp he meaning and
o p o ide he use wi h accu a e con ex ual help. This would
simpli y he li e o eade s, lea ne s, ainees, and many
o he s, and would po en ially accele a e he lea ning cu e
o in ellec ual skills. P o iding such unc ionali y in ol es
building an in elligen p edic i e model ha can ake human
bioma ke s as inpu , p ocess hose bioma ke s, and p o ide a
Spa io- empo al indica o o comp ehension di icul y as an
ou pu .
Bio eedback de ices ha e been ex ensi ely used in
esea ch o s udy he cogni i e p ocess in ol ed in a ious
ac i i ies such as so wa e de elopmen (e.g., see a ecen
su ey on measu ing he cogni i e load o so wa e de elop-
e s [9]). Recen wo k in he con ex o p og amme ’s e o s
in so wa e de elopmen [4], [5], [6] showed ha he in e-
g a ion o HRV, pupillome y, and eye- acking allows he
iden i ica ion o he code lines ha co espond o an inc ease
o cogni i e load o indi idual p og amme s. Al hough he
goal o he esea ch published in [4], [5], [6] is he anno a ion
o sou ce code o he p edic ion o p og amme s’ e o s,
i shows ha i is possible o associa e p og amme s’ men al
load in eal- ime o speci ic lines o code o lexical okens [6].
This has mo i a ed us o explo e he same concep in he
much b oade con ex o gene al con en comp ehension and
in elligen use in e aces, as p oposed in TellBack.
The e alua ion p esen ed in his pape is in ended o p o-
ide a i s answe o he ollowing ques ion: How accu a e
and p ecise a e bioma ke s ex ac ed om HRV, pupil-
lome y, and eye- acking in de ec ing, in eal- ime, whe e
exac ly he con en comp ehension di icul ies occu on
sc een?
De ec ing cogni i e load in eal- ime using mul imodal
sou ces o biosenso s has ecen ly eme ged as one o
he p omising app oaches in adap i e and cogni ion-awa e
e-lea ning [2], [3], [12], [13]. Those e y ecen s udies
we e mainly ocusing on p o iding eedback on lea ne s’
engagemen and cogni i e load changes. Howe e , he idea o
TellBack is o localize he con en (ei he in lea ning con ex
o o he eal-li e con ex s) o be able o p o ide con ex ual
help p omp ly.
The s uc u e o his pape is as ollows: he second sec ion
discusses he backg ound concep s and he ela ed wo k. The
hi d sec ion in oduces he p oposed app oach and he me h-
ods ha we e applied. The ou h sec ion add esses he da ase
and he expe imen al p o ocol ha was ollowed. Resul s
and discussions a e explo ed in he i h sec ion. Whe eas
he six h sec ion discusses he limi a ions and h ea s o he
alidi y o his app oach. Finally, he conclusion and he
u u e di ec ions a e p esen ed in he se en h sec ion o his
pape .
28394 VOLUME 9, 2021
H. Hijazi e al.: In elligen Bio eedback Augmen ed Con en Comp ehension (TellBack)
II. BACKGROUND AND RELATED WORK
This sec ion p o ides backg ound in o ma ion ela ed o he
biome ic measu es ha we e exploi ed in his wo k and
p o ides an o e iew o he ela ed wo k. Sec ion II-A b ie ly
in oduces HRV and i s use in cogni i e load de ec ion and
assessmen . Sec ion II-B concisely p esen s pupillog aphy
and eye- acking and hei use in cogni i e load assessmen .
Sec ion II-C discusses he ela ed wo k.
A. HEART RATE VARIABILITY (HRV)
Changes in he cogni i e load in luence he way he ANS
egula es he ca dio ascula sys em and causes de ec able
a ia ions o he hea a e known as HRV (Hea Ra e Va i-
abili y). HRV is modula ed by he wo componen s o he
ANS (i.e., he sympa he ic and he pa asympa he ic ne ous
sys ems) and i is based on calcula ing he changes o pe iods
be ween consecu i e ca diac cycles. HRV can be assessed in
bo h he ime and he equency domain. In he equency
domain, o example, he Low F equency (LF) a iabili y o
he hea a e is associa ed wi h he blood p essu e con ol
(i.e., sympa he ic), whe eas High F equency (HF) o he hea
a e a iabili y is associa ed wi h espi a o y sinus a hy hmia
(i.e., pa asympa he ic).
The idea o using HRV o de ec changes in he cogni i e
load is no new [21], [29]. Howe e , as indica ed in [22],
he wi hin-subjec measu emen s o HRV a e s ill unce ain
because each subjec exhibi s dis inc HRV hy hms. Thus,
non-linea me hods o HRV analysis would be p ominen
in un eiling he complexi y o he HRV hy hms. HRV is
also shown o be sensi i e o many ac o s such as gende ,
ci cadian hy hm, age, p io ac i i ies o he subjec s, and
b ea hing condi ions. Al hough hese ac o s may change
om indi idual o indi idual, his is no a p oblem o he use
o HRV in TellBack. We use LF/HF a io means and spikes
o de ec peaks in he cogni i e load o speci ic indi iduals,
and hus be ween-subjec measu emen s a e no ele an o
TellBack.
B. PUPILLOGRAPHY AND EYE-TRACKING
Eye pupil esponse has been ecognized as an indica-
o o cogni i e and a en ional e o s. Va ious esea ch
a emp s such as [23], [24] es ablished he e iden ela ion-
ship be ween pupil ac i i y and a en ional cogni i e e o s.
Bea y desc ibed in [23] ha when a pe son ecalls some-
hing om memo y o a emp s o pa se sen ences, he pupil
dila es sligh ly and e u ns o i s no mal size a e he ask is
done. This eac ion was called ask-e oked pupilla y esponse
(TEPR) [25]. The spec al analysis o he pupil diame e (PD)
is conside ed a good index o bo h he men al e o s and
a igue s a e.
Eye- acking de ices ha e also been used in esea ch o
s udy he eye gaze du ing dis inc men al asks such as code
comp ehension [4], [5], [6]. In p inciple, eye- acking is he
p ocess o acking he eye mo emen and de e mining whe e
he use is looking a o he absolu e poin o gaze (POG),
which e e s o he poin in he isual scene a which he use ’s
gaze is ocused on. Some echnologies like in [7] and [8]
used eye gaze o pinpoin di icul ies in con en . Fo example,
he iDic echnology which is desc ibed in [8] employs he eye
gaze ea u es o assis nonna i e English use s who encoun e
di icul ies in in e p e ing English wo ds. The assis i e appli-
ca ion acco dingly ansla es ha wo d based on he eye
mo emen s. In TellBack he eye- acke is used o localize he
egional/elemen al pa s o con en ha migh ha e caused he
comp ehension di icul ies.
C. RELATED WORK
Table 1 below includes a ela ed wo k compa ison. How-
e e , e y ew esea ch a emp s ha e been conduc ed o
add ess he p oblem o p edic ion o speci ic passages in
ex s ha could be di icul o comp ehend by use s. An ea ly
a emp was made by Sibe and Jacob [7]. These au ho s
de eloped a gaze mo emen s-based assis an ha deli e s
isual and audi o y p omp ing con ols o help ecogni ion
and p onuncia ion o wo ds o emedial eading ins uc ion
whene e a disabled eade encoun e s a di icul y. La e ,
Hy skyka i in oduced iDic [8] which analyzes eye gaze o
p edic di icul ies in English wo ds o nonna i e speak-
e s and acco dingly ansla e o p onounce di icul wo ds.
Those ela ed wo ks [7], [8] lack he use o he in o ma ion
sou ces mul imodali y (i.e., including di e en bioma ke s).
Mo eo e , hey lack he use o machine lea ning echniques
o p edic he di icul y. Due o he non-linea i y o cog-
ni i e load measu emen s, he use o s a is ical analysis is
no enough. In TellBack, we u ilize di e en AI echniques
such as ea u e-le el usion, ensemble decisions, and a ious
classi ica ion models.
Table 1 (below) p esen s s a e-o - he-a wo ks ha used
mul imodal biosenso s in assessing cogni i e load. Some o
hese wo ks we e men ioned in a mapping s udy [9] ha
p o ided a hema ic analysis o measu ing he cogni i e load
o so wa e de elope s while pe o ming men al asks. The
s udy included 33 a icles om 11 sea ch engines.
The au ho s in [9] showed ha 55% o he s udies used
EEG in moni o ing he cogni i e load, 36% used a combina-
ion o senso s, 6% used eye- acking, and 3% used MRI.
The ecen wo ks s a ed o in es iga e he mul imodali y o
he inpu sou ces in di e en scopes. Table 1 (abo e) summa-
izes hose s udies and o he s udies o show he bioma ke s
ha we e used o assess he cogni i e load, he me hodology,
he esul s achie ed, and he limi a ions.
Since he idea o TellBack is new, i was no easy o pe o m
a meaning ul compa ison be ween TellBack and he p e ious
s udies. Ne e heless, Table 1 b ie ly discusses he limi a ions
o he p e ious wo ks in a possible u iliza ion scena io o
iden i ica ion o use ’s con en comp ehension di icul ies.
As no iced, mos o he s udies ha used mul imodal inpu
senso s add ess so wa e de elopmen , code comp ehension,
and e-lea ning scena ios. The e a e e y ew s udies ha
add essed o he scopes and o he ypes o con en wi h he
limi a ion o a single modali y o assess he cogni i e load.
VOLUME 9, 2021 28395
H. Hijazi e al.: In elligen Bio eedback Augmen ed Con en Comp ehension (TellBack)
TABLE 1. Rela ed s udies compa ison.
28396 VOLUME 9, 2021
H. Hijazi e al.: In elligen Bio eedback Augmen ed Con en Comp ehension (TellBack)
Fo example, [10] used EEG o assess he cogni i e load o
use s while in e p e ing di e en ep esen a ions o da a isu-
aliza ions (i.e., da a dis ibu ions). Ano he example, pupil
dila ion was used in [11] as a me ic o assess linguis ics
ep esen a ions comp ehension.
In summa y, he s udies ha ha e in oduced he idea
o de ec ing he Spa io- empo al sou ces o comp ehension
di icul y in digi al con en lack he usion om mul imodal
biosenso as well as he gene ali y o con en being add essed
in comp ehension di icul y. The ele an s udy in [12] used
he mul imodal sou ces o in o ma ion o de ec he mind
wande ing momen s o eade s and no hei comp ehension
di icul ies. The s udies ha used he mul imodali y model
o senso s such as [13], [14], [15], [17] we e in gene al sup-
po ing so wa e de elopmen and na ow speci ic domains.
E en in o he domains, he e was no such abili y o localize
he con en ha caused he comp ehension di icul y a a e y
ine g anula i y le el.
Ano he e iden aspec is ha he use o ad anced AI
echniques is no commonly used in such p oposals. Fo
example, and back o he mapping s udy in [9], 40% o he
s udies ha assessed he cogni i e load o so wa e de elope s
do no use any so o machine lea ning echniques, whe eas
he es a e limi ed by using only classi ica ion echniques o
di e en se s o biosensing ea u es. Mos used classi ica ion
echniques a e (acco ding o [9]) Suppo Vec o Machine
(SVM) 15%, Nai e Bayes 15%; Mul i-algo i hms o clas-
si ica ion 9%; K-means 3%; Decision T ee 3%; Logis ic
Reg ession 3%; Neu al Ne wo k 3%; Random Fo es 3%;
Linea Reg ession 3%, and Rele ance Vec o 3%.
To measu e he cogni i e load associa ed wi h men al
e o s (e.g., con en comp ehension), we mus dis inguish
be ween he objec i i y (subjec i e s objec i e) me hods
and he causali y (di ec s indi ec ) me hods. In ou e alu-
a ion expe imen s, he subjec i e pe cep ion o he cogni i e
load was assessed using sel - epo ing h ough he NASA-
TLX [18] ques ionnai e, whe eas objec i e measu emen was
ob ained using pe iphe al physiological esponses d i en by
he au onomic ne ous sys em (ANS) cap u ed by bio eed-
back low-in usi e senso s.
In TellBack we p opose using HRV and eye- acking
(including pupillog aphy) since he e a e mul iple low- in u-
si e solu ions o implemen hese senso s. A o al o 83% o
he explo ed s udies used eye- acke and eye- ela ed ea u es
(e.g., blink), 66% used skin conduc ance o EDA measu es,
and 50% included EEG. The compa ison wi h TellBack was
no easy because he ideas p esen ed we e no simila . How-
e e , we used [12] as a good example o a ela ed ecen s udy
o compa e TellBack wi h. The p ecision and Recall we e he
alid measu es o pe o m he compa ison which is shown
in Table1.
III. METHODOLOGY
As desc ibed ea lie , TellBack is a new idea o au oma i-
cally iden i y a use ’s di icul ies in comp ehending speci ic
passages/elemen s du ing he eading o con en s on able s,
lap ops, o sma phones. Ou ision o TellBack is a u u e
in elligen and a en i e use -in e ace [16] o enhancing
comp ehension using, o example, pop-ups con aining el-
e an in o ma ion on he opic ex ac ed om he web could.
People could indeed s op eading when encoun e ing a com-
p ehension di icul y and could google he meaning o he
obscu e elemen s in he con en . Howe e , in p ac ice, his
a ely happens, due o he ush o he momen o due o he
in e ac ion limi a ions o small de ices such as sma phones,
which is he de ac o compu a ional pla o m in bo h he
de eloped and de eloping wo ld. The e o e, he key message
is o en los because o hese unsol ed di icul ies, which hin-
de s p ope decision making and leads o all so s o mis akes,
d as ically dis u bing he lea ning p ocess and ha ing a huge
nega i e impac on socie y.
Figu e 1 below shows a schema ic ep esen a ion o Tell-
Back main componen s. This ep esen a ion can be di ided
in o h ee dis inc phases: bio-signals acquisi ion, signal p o-
cessing, machine lea ning models, and con en anno a ion
based on comp ehension di icul y classi ica ion.
Recen esul s appea ed in [6] in he con ex o assessing
cogni i e s a e (mainly high cogni i e load and dis ac ion
s a es) o so wa e p og amme s show ha i is possible o
associa e such cogni i e in o ma ion, in eal- ime, o speci ic
lines o code o lexical okens, o iden i y such code lines
as being mo e suscep ible o ha ing so wa e bugs. In he
wo k epo ed he ein, we will expand such p elimina y esul s
ob ained in he speci ic con ex o p e en ion o so wa e
bugs, o a much b oade con ex o gene al con en comp e-
hension. The app oach ollowed in TellBack can be summa-
ized in he ollowing componen s as shown in Figu e 1:
1) Real- ime ANS bioma ke s measu emen s (HRV,
Pupillog aphy).
2) Fea u es ex ac ion and selec ion o assess cogni i e
load spikes and pa e ns.
3) Machine lea ning echniques o imp o e de ec ion o
high men al e o /cogni i e load om he ex ac ed
ea u es.
4) Eye- acke o localize con en egions ha a e associ-
a ed wi h high men al e o .
ANS signals a e cap u ed om subjec s while pe o ming
con en comp ehension on sc een. The signals used o assess
he cogni i e load a e sampled wi h imes amps allowing
he iden i ica ion o he momen s when he use is pe o m-
ing he comp ehension ask. The signals ob ained a e ed
in o he ea u e ex ac ion and selec ion module. In his mod-
ule, he domain knowledge is used o choose he mos ele-
an bioma ke s ou o he signals. Using hose disc iminan
ea u es ex ac ed om he collec ed pupillog aphy and HRV
synch onized wi h eye- acke ( ha will be illus a ed in he
nex sec ion), elemen al digi al con en s associa ed wi h such
ea u es a e classi ied in o wo main bina y classes, i.e., ‘‘Di -
icul ’’ and ‘‘No Di icul ’’. Those classes a e conside ed he
comp ehension di icul y s a e as shown in Figu e 1.
VOLUME 9, 2021 28397

H. Hijazi e al.: In elligen Bio eedback Augmen ed Con en Comp ehension (TellBack)
FIGURE 1. A schema ic ep esen a ion o TellBack.
The eye- acke is employed as he in o ma ion sou ce
o p o ide gaze geome ical in o ma ion (i.e., spa ial in o -
ma ion) ela ed o elemen s classi ied wi h comp ehension
‘‘di icul y’’.
In he classi ica ion module, a se o classi ie s a e sug-
ges ed o ob ain he bes esul s in e ms o accu acy, p eci-
sion, and ecall. To ensu e he in e p e abili y o he decision,
we included he use o he Decision T ee classi ie among
o he classi ie s, ega dless o i s known low accu acy in
some p oblems.
I is well known ha isola ed ANS mani es a ions a e non-
speci ic, i.e., mul iple s imulus and physiological p ocesses
migh induce simila ANS modula ion o a speci ic ANS
physiological mani es a ion such as HRV o pupil diame e
a iabili y. To educe his non-speci ici y, we p opose o
combine dis inc ANS mani es a ions using a da a usion
app oach o exploi speci ic pa e ns o ac i a ion.
In his sense, he e alua ion p esen ed in his pape ep-
esen s a less ich scena io ha only uses HRV, pupillome y,
and eye- acking, which sugges s ha ou i s esul s p o ide
a conse a i e iew on he accu acy, p ecision, and ecall o
TellBack.
IV. DATASET, PROTOCOL AND METHOD
This sec ion desc ibes he da ase ha was used o p o e he
concep o TellBack and discusses he expe imen al p o ocol
and he me hods used o analyze he esul ing da a.
A. DATASET
The da ase ha was used o his s udy is an al eady exis ing
publicly a ailable da ase , which was o igina ed om he
BASE p ojec men ioned in [4], [5], [6]. The da ase was
de eloped in one o he p ojec ’s s udies o moni o physio-
logic eac ions and men al e o ha a e associa ed wi h code
comp ehension in di e en complexi ies using non-in usi e
biosenso s. The biosenso s ha we e used in he expe imen
a e he ECG and Eye- acking wi h Pupillog aphy. The con-
olled expe imen in ol ed 30 subjec s expe ienced in Ja a.
The subjec s expe imen ed in he same oom wi h he same
condi ions while being equipped wi h men ioned wea able
non-in usi e senso s. The expe imen p o ocol o he BASE
s udy included he ollowing s eps:
1) Baseline ac i i y o le he subjec s look a an emp y
g ey sc een wi h a black c oss in he cen e o
30 seconds o de ach pa icipan s om any ac i i y.
2) Re e ence Ac i i y ha comp ises a sc een wi h a
Na u al Language ex o 60 seconds.
3) Baseline ac i i y o le he subjec s look a an emp y
g ey sc een wi h a black c oss in he cen e o
30 seconds o de ach pa icipan s om any ac i i y.
4) Code comp ehension comp ises h ee code p og ams
wi h di e en complexi ies (c1, c2, c3).
5) Baseline ac i i y o le subjec s look a an emp y g ey
sc een wi h a black c oss in he cen e o 30 seconds
o de ach pa icipan s om any ac i i y.
6) Su ey using NASA-TLX o assess he subjec s’ men-
al e o s while comp ehending he code.
7) Con ol ques ions o check i he subjec unde s ood
he codes co ec ly.
This expe imen yielded 90 da ase s co esponding o he
30 olun ee s wi h 3 asks (i.e., c1, c2, c3). In his expe imen ,
he changes in HRV and Pupil Diame e (PD) bioma ke s
we e analyzed du ing code comp ehension asks o de ec
momen s ha co espond o he high men al e o o he
p og amme . Usually, hose momen s co espond o HRV
and pupil spikes (i.e., peaks). These momen s we e mapped
o co esponding spa ial coo dina es o code loca ions using
eye- acking in o ma ion.
B. PROTOCOL AND METHOD
The main goal o he BASE p ojec was o assess he cogni i e
load o p og amme s and anno a e he code lines ha a e
mo e p one o ha e so wa e bugs. The anno a ion o hose
po en ially p oblema ic code lines would ale he p og am-
me s online o e ise hose code lines. Howe e , in ou s udy,
TellBack uses he same echnique (i.e., using bio eedback
28398 VOLUME 9, 2021
H. Hijazi e al.: In elligen Bio eedback Augmen ed Con en Comp ehension (TellBack)
wea ables o assess cogni i e load), bu wi h a di e en pu -
pose and scope. The main goal o TellBack is o p edic
con en comp ehension di icul y based on assessing he use
cogni i e load in eal- ime and o use he eye- acke syn-
ch onously o localize he con en elemen al pa s ha caused
he comp ehension di icul y.
Since he p o ocol used o p oduce he da ase a ailable
om he BASE s udy was no designed o he iden i ica-
ion o use ’s comp ehension di icul ies while eading he
code ( he goal in he BASE p ojec was p og amme ’s e o s
and bugs), we es ablished a speci ic p o ocol (adding a new
laye o he ini ial p o ocol) ha u ilizes he da ase al eady
a ailable and enhances such da ase wi h addi ional da a o
e alua e he accu acy o ou new app oach. Mo e speci ically,
he addi ional da a consis s o he iden i ica ion o he code
egions ha a e conside ed easy/di icul o unde s and by he
subjec s. Ou speci ic p o ocol assumes he ollowing, which
akes he o m o a da a-ga he ing in e iew:
1) The same subjec s o he BASE s udy we e in i ed o a
emo e da a ga he ing in e iew.
2) The selec ion o he subjec s was na owed down o
hose who ha e mos likely he same expe ience in Ja a
p og amming compa ed wi h he expe imen ime (i.e.,
hey ha e no gained addi ional expe ience in Ja a).
3) The selec ed subjec s we e asked o ill a o m indi-
ca ing hei cu en posi ion, pe sonal e alua ion o he
gained expe ience in p og amming languages since he
ime o he BASE s udy, and he cu en p og amming
languages ha hey ha e been p ac icing o he ime.
4) The same code snippe s (c1, c2, c3) ha we e used in
he BASE s udy we e p esen ed o he subjec s. These
snippe s ep esen codes o di e en complexi y le els
anging om Vg =3 o c1 o Vg =14 o c3 based on
McCabe me ic.
5) The main ask was asking he subjec s o ead and
comp ehend he code snippe s again wi h he same
amoun o ime gi en in he o iginal s udy.
6) An addi ional laye o he o iginal p o ocol was added
by asking he subjec s o manually label he code lines
wi h h ee colo s: ed o mos di icul , o ange o
di icul , and g een o easy. Highligh ing he code lines
was based on he subjec i ely pe cei ed di icul y le el.
7) The same con ol ques ions we e p esen ed o he sub-
jec s o check hei code comp ehension.
8) The da a analysis was ini ially pe o med by ex ac ing
he pupillog aphy and HRV signals ha a e associa ed
wi h he ime window o each anno a ed egion in poin
6 based on he gaze in o ma ion.
9) By epea ing hose s eps o each selec ed subjec ,
a new da ase was de eloped, which comp ises he
pupillog aphy and HRV ea u es ha co espond o
di e en code lines and egions labeled by he subjec s
acco ding o he di icul y.
Fo simplici y in he analysis phase, we conside ed o ange
and ed colo s indica ed in poin 6 abo e as ‘‘Di icul ’’,
whe eas g een was conside ed as ‘‘No Di icul ’’.
We hypo hesize ha i he subjec in he new da a-ga he ing
in e iew highligh ed a ce ain segmen as ‘‘Di icul ’’, hen
i would ha e been di icul a he ime o he expe imen o
him o he o comp ehend ha segmen o code, and hus
his would enable us o compa e be ween he code segmen s
highligh ed ‘‘di icul ’’ and he HRV and pupilome e signals
acqui ed a he o iginal s udy ime. Likewise, labeled code
segmen s wi h ‘‘No Di icul ’’ a e also compa ed wi h he
co esponding HRV and pupillog aphy signals. This would
lead us o p o e whe he HRV and pupillog aphy ea u es
could be used o iden i y wi h adequa e ime esolu ion di -
e en con en pieces whe e use s a e encoun e ing comp e-
hension di icul ies. The new laye o p o ocol men ioned in
poin 6 abo e would enable us o obse e o which ex en he
eye- acke was able o iden i y he egions/lines o code ha
co espond o high men al e o .
The da a-ga he ing in e iew ook place h ough Zoom
due o he Co id-19 ou b eak. A o al o 11 subjec s we e
in e iewed, as shown in Table 2, and 30 di e en samples
we e ob ained. Only one o he subjec s was excluded because
he e was no co esponding su icien in o ma ion om he
o iginal s udy. Table 2 shows he subjec s’ in o ma ion p o-
ile. The a e age ime o in e iews was 60 minu es. A i s ,
subjec s we e o ien ed o he ask ha is equi ed om hem
h ough an explana o y ideo. A e ha , hey we e p o ided
wi h he codes (i.e., c1, c2, and c3) o ead and comp ehend
hem sepa a ely. A s opwa ch ime was used in each s ep.
The subjec s we e cons an ly asked o highligh he code lines
whene e hey eel ha hese code lines needed mo e men al
e o s o g asp.
TABLE 2. Subjec s in o ma ion p o ile.
VOLUME 9, 2021 28399
H. Hijazi e al.: In elligen Bio eedback Augmen ed Con en Comp ehension (TellBack)
The subjec s we e asked abou he cu en p og amming
languages ha hey ha e been wo king on since he ime o
he o iginal s udy. We we e conse a i e abou hose who
acqui ed excessi e expe ience in Ja a (i.e., he language o
he code snippe s) because hey may no expe ience he same
di icul y ha hey encoun e ed a he ime o he o iginal
expe imen .
A e ga he ing he needed in o ma ion, he analysis o
he esul s was ca ied ou as ollows and as illus a ed in
pseudo-code 1:
1) Ex ac ing he ime ins an s o pupillog aphy and HRV
singles o each labeled code line o egion by he
subjec .
2) Calcula ing he ollowing ea u es (numbe o HRV
LF/HF a io spikes which ep esen s he a io be ween
he low and he high equency o he HRV signal,
he numbe o pupillog aphy LF/HF spikes, mean o
LF/HF spikes alues o pupillog aphy, mean o LF/HF
spikes alues o HRV).
3) Labeling hose ea u es wi h a bina y di icul y le el
(Di icul , no Di icul ) as anno a ed by he subjec s in
he da a-ga he ing session.
4) Pe o ming unpai ed - es o examine i he e is a
signi ican s a is ical di e ence be ween he wo di i-
cul y s a es acco ding o he gi en ea u es in poin 2
(assuming a no mal dis ibu ion).
5) Applying G id Sea ch o ind he op imum hype pa-
ame e s.
6) Using lea e-one-ou c oss- alida ion o e alua e he
esul ed model.
7) Using K-Fold c oss- alida ion o compa e wi h poin 6.
Pseudo-Code 1 Mapping Ex ac ed Fea u es wi h Regional
Pa s o Con en on Sc een
Inpu : local_maxima(signal): signal {HRV, pupil}, S,
S=(s1, s2,..,sn)
S. ime: ins an ime o S ec o
EyeGazeTime =Time_ins an (signal) signal {eye
acke }, eye gaze ime ins an s – ime ins an s o subjec s
looking a sc een.
Ou pu : eye_gaze_coo dina es associa ed wi h S
1: Fo each s S do
2: T(s)←S. ime
3: Fo each s S do
4: EGT(T(s)) ←EyeGazeTime
5: Find samples n whe e
6: EGT(T(s))<=T(s)+δAND EGT(T(s))
>=T(s)- δ);
δis he sample ange whe e T(s) spans.
7: Fo each sample ange n EGT(T(s)) do
8: Find eye_gaze_coo dina es(n) whe e subjec s we e
looking a ha ime ins an (when spikes occu ).
9: Re u n (eye_gaze_coo dina es(n))
8) Iden i ying he signals pa e ns (i.e., HRV and pupil-
log aphy) ha a e associa ed wi h he di icul y in
unde s anding using di e en classi ie s (SVM bo h he
linea and he adial basis unc ion, Random Fo es ,
KNN, Na´
’ı e Bayes, Decision T ee).
9) Running he model 50 imes and calcula ing he accu-
acy mean and he accu acy s anda d de ia ion.
10) Calcula ing he p ecision, ecall, mic o F1-sco e,
mac o F1-sco e.
The ollowing Pseudo-Code ep esen s s eps 1-3 in de ail.
V. RESULTS AND DISCUSSION
As p e ious s udies ha e shown ha pupillog aphy and HRV
a e imely manne indica o s o he cogni i e load (and a e
non-in usi e), TellBack s a ed wi h hose measu es o assess
he cogni i e load associa ed wi h he unde s anding di i-
cul y. Howe e , hose measu es a e insu icien o anno a e
he elemen al pa s o con en ha caused he comp ehension
di icul y. The e o e, eye- acking was in oduced o ell us
whe e he use is looking whene e he pupil o HRV signal
spikes abo e he h eshold, indica ing an ab up inc ease in
he use ’s cogni i e load.
A e pe o ming he analysis, he p elimina y esul s
showed ha mos o he code egions ha we e di icul o
he subjec s o unde s and we e also men ally demanding
a he ime o he in e iew (shown by sel -anno a ions).
Figu e 2 (abo e) shows an example o one o he subjec s.
In his Figu e, we can see ha he men ally demanding code
egions (Figu e 2-D) co espond in gene al o high gaze den-
si y (Figu e 2-B and C). Simila ly, he pupil and HRV spikes
ended o inc ease abo e a ce ain h eshold (Figu e 2-E)
when he ask is conside ed di icul o mos di icul . The
h eshold was de e mined based on da a obse a ions.
Likewise, he pupil and HRV spikes we e de ined based on
empi ic da a obse a ion and analysis o di e en subjec s.
Using gaze and saccades o in e he use ’s cogni i e load
equi es he in eg a ion in ime, which means ha he p e-
cision in he ime domain is poo . On he con a y, pupil, and
HRV ea u es p o ided an accu a e assessmen o cogni i e
load in he ime domain (momen s when he spikes occu ).
As obse ed in Figu e 2, he code lines anno a ed as di -
icul o mos di icul (o ange and ed colo espec i ely)
by he subjec in ou da a-ga he ing in e iews co espond
o a high numbe o HRV and pupillog aphy spikes (o ange
appea s in his example). Tha means he highe numbe o
HRV and pupil spikes as seen in (Figu e 2-E) wi hin a speci ic
ime window, he mo e po en ial e idence o high cogni-
i e load induced on he subjec by unde s anding e o s.
No e ha o iden i y he poin in he code ha co esponds
o an HRV o pupil spike (i.e., he sc een a ea whe e he
use was looking a when he spike occu ed) we needed o
ind he gaze egion in pa B o Figu e 2 ha is e ically
aligned wi h he spike, and hen o go o he igh and ind
he code lines (pa C and D o he igu e) ha ep esen
he eye- acking in o ma ion and he sel -labeling o code
28400 VOLUME 9, 2021
H. Hijazi e al.: In elligen Bio eedback Augmen ed Con en Comp ehension (TellBack)
FIGURE 2. Pupil and HRV signals o he subjec no 9 in compa ison wi h his sel -labeling: A) Eye gaze densi y co esponding o he y-axis o he
code. B) Clus e s o e ime and gaze eloci y; b) he pupillog aphy and HRV ex ac LH ea u es ex ac ed. C) The ed do s ha ep esen he eye
gaze geome ical dis ibu ion supe imposed on he code. D) The addi ional laye o he o iginal p o ocol shows labeled code lines by subjec s.
E) The co esponding pupillog aphy and HRV signals and spikes.
egions, espec i ely. Be o e applying he empi ical analysis,
we could see om Figu e 2 (C and D) ha he e is a clea
mapping be ween wha eye- acke ells us abou he men al
e o in speci ic lines/ egions o code and he labeled code
lines/ egions by subjec s as appea ed in pa D o he same
Figu e. To e alua e he bioma ke s ha co espond o each
egion, he ollowing ea u es we e analyzed: he numbe
o HRV LF/HF spikes, he numbe o pupillog aphy LF/HF
spikes, he mean o LF/HF a io spikes alues o pupillog-
aphy, mean o LF/HF a io spikes alues o HRV based
on he ime ins an s window o he subjec s’ gaze ha is
co esponding o he anno a ed code egions.
An unpai ed - es wi h a con idence in e al o 95% was
pe o med o examine i he e is a s a is ically signi ican
di e ence be ween he numbe o pupillog aphy LF/HF a io
spikes in he wo di icul y g oups (i.e., di icul , no di icul ).
The unpai ed - es wi h p − alue =0.0095 shows ha
he means a e s a is ically di e en . In he ‘‘di icul ’’ g oup
(M =9.5,SD =6.5), whe eas in he ‘‘no di icul ’’
g oup (M =3.055556,SD =6.5). Likewise, he unpai ed
- es shows ha he means o he numbe o HRV LF/HF
a io spikes wi h p − alue =0.0002975 a e s a is ically
di e en in he wo di icul y s a es., The same es shows
also ha he e is a s a is ically signi ican di e ence be ween
he means o pupillog aphy LF/HF a io spikes alues in he
wo di icul y g oups wi h (M =5.10200,SD =2.88)
in he ‘‘di icul ’’ g oup, and wi h (M =4.07444,SD =
2.86) in he ‘‘no di icul ’’ g oup wi h p- alue =0.3733.
The e o e, he al e na i e hypo hesis is ue which assumes
he s a is ically signi ican di e ences. The ea u es co e-
la ion shown in Figu e 3 abo e shows clea disc imina ion
be ween he ‘‘Di icul y’’ and ‘‘No di icul y’’ beha io . The
co ela ion sco es be ween he bioma ke s such as HRV and
Pupil spikes numbe and he di icul y pe cei ed beha io o
subjec s shown by hei sel -anno a ions o he code egions
indica e a s ong co ela ion as shown in Figu e 3.
Likewise, he box plo in Figu e 4 shows selec ed ea u es
dis ibu ions agains he di icul y g oups o egions o codes
as pe cei ed by subjec s. Figu e 4a shows ha he highe
he numbe o pupil spikes wi hin a code egion, he highe
he di icul y pe cei ed by he subjec . Simila ly, Figu e 4b
indica es ha he inc easing numbe o HRV spikes is p opo -
ional o he di icul y s a e in comp ehending egions o code.
F om his Figu e, we can see he di e ences in he means,
medians, and minimum alues o he wo di e en g oups.
The ela i ely high sp ead o HRV da a om he cen e can
be explained in e ms o he high sensi i i y o he
HRV signal o di e en cogni i e s a es such as anxi-
e y, s ess, and a igue. Those bioma ke s (i.e., ea u es)
we e ed in o 5 di e en classi ie s namely, Suppo Vec-
o Machine (SVM) wi h bo h a linea ke nel and a Radial
Basis Func ion ke nel, Random Fo es , K-Nea es Neighbo
(KNN), Decision T ee, and Gaussian Na´
’ı e Bayes. Due o
he limi ed da ase , and o a oid any so o da a o e i ing,
his wo k used he lea e-one-ou c oss- alida ion (LOOCV)
me hod [19]. The LOOCV uses one sample o he aining
da a o alida ion and uses he es o he da a o aining
un il i co e s all he da a eco ds. Howe e , o making
u he compa isons wi h o he me hods o c oss- alida ion,
K- old c oss- alida ion [26] was also examined.
All classi ie s we e un 50 imes o calcula e he mean
and he s anda d de ia ion o he accu acies as shown
VOLUME 9, 2021 28401