Intelligent Biofeedback Augmented Content Comprehension (TellBack)

Recei ed Janua y 31, 2021, accep ed Feb ua y 5, 2021, da e o publica ion Feb ua y 11, 2021, da e o cu en e sion Feb ua y 23, 2021. Digi al Objec Iden i ie 10.1109/ACCESS.2021.3058664 In elligen Bio eedback Augmen ed Con en Comp ehension (TellBack) HAYTHAM HIJAZI 1, RICARDO COUCEIRO1, JOÃO CASTELHANO2, PAULO DE CARVALHO1, MIGUEL CASTELO-BRANCO2, AND HENRIQUE MADEIRA1 1Cen e o In o ma ics and Sys ems, Uni e si y o Coimb a (CISUC), 3000-214 Coimb a, Po ugal 2Coimb a Ins i u e o Biomedical Imaging and T ansla ional Resea ch (CIBIT), Ins i u e o Nuclea Sciences Applied o Heal h (ICNAS), Uni e si y o Coimb a, 3000-214 Coimb a, Po ugal Co esponding au ho : Hay ham Hijazi ([email p o ec ed]) This wo k was pa ially unded by he BASE p ojec unde G an POCI - 01-0145 - FEDER- 031581, in pa by he Cen o de In o m ica e Sis emas da Uni e sidade de Coimb a (CISUC), and also in pa by Coimb a Ins i u e o Biomedical Imaging and T ansla ional Resea ch (CIBIT), Ins i u e o Nuclea Sciences Applied o Heal h (ICNAS), Uni e si y o Coimb a unde G an PTDC/PSI-GER/30852/2017 | CONNECT-BCI. ABSTRACT Assessing comp ehension di icul ies equi es he abili y o assess cogni i e load. Changes in cogni i e load induced by comp ehension di icul ies could be de ec ed wi h an adequa e ime eso- lu ion using di e en bio eedback measu es (e.g., changes in he pupil diame e ). Howe e , iden i ying he Spa io- empo al sou ces o con en comp ehension di icul ies (i.e., when, and whe e exac ly he di icul y occu s in con en egions) wi h a ine g anula i y is a big challenge ha has no been explici ly add essed in he s a e-o - he-a . This pape p oposes and e alua es an inno a i e app oach named In elligen Bio eedbackAugmen ed Con en Comp ehension (TellBack) o explici ly add ess his challenge. The goal is o au onomously iden i y egions o digi al con en ha cause use ’s comp ehension di icul y, opening he possibili y o p o ide eal- ime comp ehension suppo o use s. TellBack is based on assessing he cogni i e load associa ed wi h con en comp ehension h ough non-in usi e cheap bio eedback de ices ha acqui e measu es such as pupil esponse o Hea Ra e Va iabili y (HRV). To iden i y when exac ly he di icul y in comp ehension occu s, physiological mani es a ions o he Au onomic Ne ous Sys em (ANS) such as he pupil diame e a iabili y and he modula ion o HRV a e exploi ed, whe eas he ine spa ial esolu ion (i.e., he egion o con en whe e he use is looking a ) is p o ided by eye- acking. The e alua ion esul s o his app oach show an accu acy o 83.00% ±0.75 in classi ying egions o con en as di icul o no di icul using Suppo Vec o Machine (SVM), and p ecision, ecall, and mic o F1-sco e o 0.89, 0.79, and 0.83, espec i ely. Resul s ob ained wi h 4 o he classi ie s, namely Random Fo es , k-nea es neighbo , Decision T ee, and Gaussian Nai e Bayes, showed a sligh ly lowe p ecision. TellBack ou pe o ms he s a e-o - he-a in p ecision & ecall by 23% and 17% espec i ely. INDEX TERMS Biomedical measu emen , cogni i e load, con en comp ehension, eye- acking, hea a e a iabili y, machine lea ning. I. INTRODUCTION Imagine a so wa e echnology ha augmen s you abili y o comp ehend complex concep s and ideas. You a e eading a sen ence in a echnical documen and he so wa e ins alled in you able , lap op, o sma phone au oma ically de ec s speci ic passages o wo ds in a pa ag aph ha make he en i e pa ag aph cumbe some o you and p omp ly displays an explana ion, shows an example, o p o ides you wi h a de ini ion ha will make you unde s and he whole idea. We call his app oach InTelligen Bio eedBack Augmen ed The associa e edi o coo dina ing he e iew o his manusc ip and app o ing i o publica ion was Giuseppe Desolda . Con en Comp ehension (TellBack), and he p esen pape p oposes he idea and e alua es he accu acy o TellBack in he iden i ica ion o speci ic con en egions (e.g., line, exp ession, e c.) ha a e conside ed di icul o unde s and by he use . The accu a e iden i ica ion o con en loca ions ha cause use ’s comp ehension di icul ies is a c ucial s ep o show he easibili y o he p oposed app oach. TellBack uses cogni i e load as a key elemen o iden i y speci ic pa s o digi al con en ha cause comp ehension di icul y. The goal is o p edic ha he use is needing suppo in unde s anding he con en elemen /passage ha is causing comp ehension di icul ies while eading digi al con en . VOLUME 9, 2021 This wo k is licensed unde a C ea i e Commons A ibu ion 4.0 License. Fo mo e in o ma ion, see h ps://c ea i ecommons.o g/licenses/by/4.0/ 28393 H. Hijazi e al.: In elligen Bio eedback Augmen ed Con en Comp ehension (TellBack) Digi ized eading esou ces a e immensely g owing. Due o he popula i y o handheld po able de ices, such as e- eade s and able s, eading is becoming mo e on-sc een han on p in ed ma e ial. Howe e , a ious s udies, such as [1], showed ha on-sc een eading is associa ed wi h sho a en ion and poo comp ehension. Comp ehension o digi al esou ces dese es mo e esea ch a en ion, especially wi h he e e -g owing eliance on digi al esou ces comp ehen- sion in lea ning, aining, and indus y. Fo ins ance, in he so wa e indus y, code e iew is one o he essen ial asks in so wa e enginee ing o ensu e he quali y o he deli e ed se ice. Code e iew equi es ull code comp ehension by he e iewe . The code and i s desc ip ion a e gene ally digi ized esou ces ha equi e he e iewe o immensely ead and comp ehend. Digi al esou ces comp ehension equi es he abili y o use s o lea ne s o in eg a e and e alua e in o ma ion ac oss di e en ypes o ep esen a ions. The in eg a ion and e alua- ion a e o en men ally demanding asks, which impose ex a cogni i e load on lea ne s. Comp ehension, in gene al, is a highe cogni i e p ocess ha imposes a cogni i e load and in ol es di e en cog- ni i e s a es, which we hypo hesize in TellBack ha hese s a es can be accu a ely cap u ed by non-in usi e bio eed- back de ices h ough physiological mani es a ions o he Au onomous Ne ous Sys ems (ANS). In he con ex o his pape , bio eedback is de ined as he p ocess o p o iding ask- ele an eedback o he use based on he /his cogni i e s a e (cap u ed h ough physiological ANS eac ions) using comme cially a ailable wea ables such as b acele s, wa ches, and ings. Ou main con ibu ions in his wo k can be summa ized by he ollowing poin s: •The pape in oduces a new echnique o de ec he cog- ni i e load and he eby he comp ehension di icul ies a elemen al pa s o he con en (e.g., lines, exp essions, e c.) and no a global assessmen o cogni i e load as in es iga ed in he s a e-o - he-a . •The use o eye- acking is no o assess he cogni i e load as i is gene ally used in he s a e-o - he-a (which has he disad an age o a slow esponse ime) bu o p o ide an accu a e spa ial esolu ion whe e he use is looking a when he HRV and pupillome y bioma ke s indica e peaks in he cogni i e load. •The e alua ion o he imp o emen s in p ecision and ecall achie ed wi h TellBack, when compa ed wi h a e y ecen ela ed s udy in [12], shows 23% be e p ecision and 17% be e ecall. •This wo k add esses he limi a ions o a single modal- i y using da a usion o non-in usi e biosenso s and eal- ime bioma ke s om HRV and pupillome y. Unlike o he ecen s udies [2], [3], [12] ha exploi he mul imodali y o biosenso s, we used simple ea u es (in e ms o compu a ion), bu disc iminan enough o de ec he inc ease o he use ’s cogni i e load/men al e o in eal- ime. •The use o machine lea ning and AI echniques o op i- mize he p edic ion o ‘‘when and whe e’’ he use is encoun e ing di icul y on sc een. The inal goal o Tellback is o augmen he use ’s abili y o comp ehend complex concep s and ideas h ough a new ype o in elligen in e ace ha seems o guess when and whe e he use is making an unusual e o o g asp he meaning and o p o ide he use wi h accu a e con ex ual help. This would simpli y he li e o eade s, lea ne s, ainees, and many o he s, and would po en ially accele a e he lea ning cu e o in ellec ual skills. P o iding such unc ionali y in ol es building an in elligen p edic i e model ha can ake human bioma ke s as inpu , p ocess hose bioma ke s, and p o ide a Spa io- empo al indica o o comp ehension di icul y as an ou pu . Bio eedback de ices ha e been ex ensi ely used in esea ch o s udy he cogni i e p ocess in ol ed in a ious ac i i ies such as so wa e de elopmen (e.g., see a ecen su ey on measu ing he cogni i e load o so wa e de elop- e s [9]). Recen wo k in he con ex o p og amme ’s e o s in so wa e de elopmen [4], [5], [6] showed ha he in e- g a ion o HRV, pupillome y, and eye- acking allows he iden i ica ion o he code lines ha co espond o an inc ease o cogni i e load o indi idual p og amme s. Al hough he goal o he esea ch published in [4], [5], [6] is he anno a ion o sou ce code o he p edic ion o p og amme s’ e o s, i shows ha i is possible o associa e p og amme s’ men al load in eal- ime o speci ic lines o code o lexical okens [6]. This has mo i a ed us o explo e he same concep in he much b oade con ex o gene al con en comp ehension and in elligen use in e aces, as p oposed in TellBack. The e alua ion p esen ed in his pape is in ended o p o- ide a i s answe o he ollowing ques ion: How accu a e and p ecise a e bioma ke s ex ac ed om HRV, pupil- lome y, and eye- acking in de ec ing, in eal- ime, whe e exac ly he con en comp ehension di icul ies occu on sc een? De ec ing cogni i e load in eal- ime using mul imodal sou ces o biosenso s has ecen ly eme ged as one o he p omising app oaches in adap i e and cogni ion-awa e e-lea ning [2], [3], [12], [13]. Those e y ecen s udies we e mainly ocusing on p o iding eedback on lea ne s’ engagemen and cogni i e load changes. Howe e , he idea o TellBack is o localize he con en (ei he in lea ning con ex o o he eal-li e con ex s) o be able o p o ide con ex ual help p omp ly. The s uc u e o his pape is as ollows: he second sec ion discusses he backg ound concep s and he ela ed wo k. The hi d sec ion in oduces he p oposed app oach and he me h- ods ha we e applied. The ou h sec ion add esses he da ase and he expe imen al p o ocol ha was ollowed. Resul s and discussions a e explo ed in he i h sec ion. Whe eas he six h sec ion discusses he limi a ions and h ea s o he alidi y o his app oach. Finally, he conclusion and he u u e di ec ions a e p esen ed in he se en h sec ion o his pape . 28394 VOLUME 9, 2021 H. Hijazi e al.: In elligen Bio eedback Augmen ed Con en Comp ehension (TellBack) II. BACKGROUND AND RELATED WORK This sec ion p o ides backg ound in o ma ion ela ed o he biome ic measu es ha we e exploi ed in his wo k and p o ides an o e iew o he ela ed wo k. Sec ion II-A b ie ly in oduces HRV and i s use in cogni i e load de ec ion and assessmen . Sec ion II-B concisely p esen s pupillog aphy and eye- acking and hei use in cogni i e load assessmen . Sec ion II-C discusses he ela ed wo k. A. HEART RATE VARIABILITY (HRV) Changes in he cogni i e load in luence he way he ANS egula es he ca dio ascula sys em and causes de ec able a ia ions o he hea a e known as HRV (Hea Ra e Va i- abili y). HRV is modula ed by he wo componen s o he ANS (i.e., he sympa he ic and he pa asympa he ic ne ous sys ems) and i is based on calcula ing he changes o pe iods be ween consecu i e ca diac cycles. HRV can be assessed in bo h he ime and he equency domain. In he equency domain, o example, he Low F equency (LF) a iabili y o he hea a e is associa ed wi h he blood p essu e con ol (i.e., sympa he ic), whe eas High F equency (HF) o he hea a e a iabili y is associa ed wi h espi a o y sinus a hy hmia (i.e., pa asympa he ic). The idea o using HRV o de ec changes in he cogni i e load is no new [21], [29]. Howe e , as indica ed in [22], he wi hin-subjec measu emen s o HRV a e s ill unce ain because each subjec exhibi s dis inc HRV hy hms. Thus, non-linea me hods o HRV analysis would be p ominen in un eiling he complexi y o he HRV hy hms. HRV is also shown o be sensi i e o many ac o s such as gende , ci cadian hy hm, age, p io ac i i ies o he subjec s, and b ea hing condi ions. Al hough hese ac o s may change om indi idual o indi idual, his is no a p oblem o he use o HRV in TellBack. We use LF/HF a io means and spikes o de ec peaks in he cogni i e load o speci ic indi iduals, and hus be ween-subjec measu emen s a e no ele an o TellBack. B. PUPILLOGRAPHY AND EYE-TRACKING Eye pupil esponse has been ecognized as an indica- o o cogni i e and a en ional e o s. Va ious esea ch a emp s such as [23], [24] es ablished he e iden ela ion- ship be ween pupil ac i i y and a en ional cogni i e e o s. Bea y desc ibed in [23] ha when a pe son ecalls some- hing om memo y o a emp s o pa se sen ences, he pupil dila es sligh ly and e u ns o i s no mal size a e he ask is done. This eac ion was called ask-e oked pupilla y esponse (TEPR) [25]. The spec al analysis o he pupil diame e (PD) is conside ed a good index o bo h he men al e o s and a igue s a e. Eye- acking de ices ha e also been used in esea ch o s udy he eye gaze du ing dis inc men al asks such as code comp ehension [4], [5], [6]. In p inciple, eye- acking is he p ocess o acking he eye mo emen and de e mining whe e he use is looking a o he absolu e poin o gaze (POG), which e e s o he poin in he isual scene a which he use ’s gaze is ocused on. Some echnologies like in [7] and [8] used eye gaze o pinpoin di icul ies in con en . Fo example, he iDic echnology which is desc ibed in [8] employs he eye gaze ea u es o assis nonna i e English use s who encoun e di icul ies in in e p e ing English wo ds. The assis i e appli- ca ion acco dingly ansla es ha wo d based on he eye mo emen s. In TellBack he eye- acke is used o localize he egional/elemen al pa s o con en ha migh ha e caused he comp ehension di icul ies. C. RELATED WORK Table 1 below includes a ela ed wo k compa ison. How- e e , e y ew esea ch a emp s ha e been conduc ed o add ess he p oblem o p edic ion o speci ic passages in ex s ha could be di icul o comp ehend by use s. An ea ly a emp was made by Sibe and Jacob [7]. These au ho s de eloped a gaze mo emen s-based assis an ha deli e s isual and audi o y p omp ing con ols o help ecogni ion and p onuncia ion o wo ds o emedial eading ins uc ion whene e a disabled eade encoun e s a di icul y. La e , Hy skyka i in oduced iDic [8] which analyzes eye gaze o p edic di icul ies in English wo ds o nonna i e speak- e s and acco dingly ansla e o p onounce di icul wo ds. Those ela ed wo ks [7], [8] lack he use o he in o ma ion sou ces mul imodali y (i.e., including di e en bioma ke s). Mo eo e , hey lack he use o machine lea ning echniques o p edic he di icul y. Due o he non-linea i y o cog- ni i e load measu emen s, he use o s a is ical analysis is no enough. In TellBack, we u ilize di e en AI echniques such as ea u e-le el usion, ensemble decisions, and a ious classi ica ion models. Table 1 (below) p esen s s a e-o - he-a wo ks ha used mul imodal biosenso s in assessing cogni i e load. Some o hese wo ks we e men ioned in a mapping s udy [9] ha p o ided a hema ic analysis o measu ing he cogni i e load o so wa e de elope s while pe o ming men al asks. The s udy included 33 a icles om 11 sea ch engines. The au ho s in [9] showed ha 55% o he s udies used EEG in moni o ing he cogni i e load, 36% used a combina- ion o senso s, 6% used eye- acking, and 3% used MRI. The ecen wo ks s a ed o in es iga e he mul imodali y o he inpu sou ces in di e en scopes. Table 1 (abo e) summa- izes hose s udies and o he s udies o show he bioma ke s ha we e used o assess he cogni i e load, he me hodology, he esul s achie ed, and he limi a ions. Since he idea o TellBack is new, i was no easy o pe o m a meaning ul compa ison be ween TellBack and he p e ious s udies. Ne e heless, Table 1 b ie ly discusses he limi a ions o he p e ious wo ks in a possible u iliza ion scena io o iden i ica ion o use ’s con en comp ehension di icul ies. As no iced, mos o he s udies ha used mul imodal inpu senso s add ess so wa e de elopmen , code comp ehension, and e-lea ning scena ios. The e a e e y ew s udies ha add essed o he scopes and o he ypes o con en wi h he limi a ion o a single modali y o assess he cogni i e load. VOLUME 9, 2021 28395 H. Hijazi e al.: In elligen Bio eedback Augmen ed Con en Comp ehension (TellBack) TABLE 1. Rela ed s udies compa ison. 28396 VOLUME 9, 2021 H. Hijazi e al.: In elligen Bio eedback Augmen ed Con en Comp ehension (TellBack) Fo example, [10] used EEG o assess he cogni i e load o use s while in e p e ing di e en ep esen a ions o da a isu- aliza ions (i.e., da a dis ibu ions). Ano he example, pupil dila ion was used in [11] as a me ic o assess linguis ics ep esen a ions comp ehension. In summa y, he s udies ha ha e in oduced he idea o de ec ing he Spa io- empo al sou ces o comp ehension di icul y in digi al con en lack he usion om mul imodal biosenso as well as he gene ali y o con en being add essed in comp ehension di icul y. The ele an s udy in [12] used he mul imodal sou ces o in o ma ion o de ec he mind wande ing momen s o eade s and no hei comp ehension di icul ies. The s udies ha used he mul imodali y model o senso s such as [13], [14], [15], [17] we e in gene al sup- po ing so wa e de elopmen and na ow speci ic domains. E en in o he domains, he e was no such abili y o localize he con en ha caused he comp ehension di icul y a a e y ine g anula i y le el. Ano he e iden aspec is ha he use o ad anced AI echniques is no commonly used in such p oposals. Fo example, and back o he mapping s udy in [9], 40% o he s udies ha assessed he cogni i e load o so wa e de elope s do no use any so o machine lea ning echniques, whe eas he es a e limi ed by using only classi ica ion echniques o di e en se s o biosensing ea u es. Mos used classi ica ion echniques a e (acco ding o [9]) Suppo Vec o Machine (SVM) 15%, Nai e Bayes 15%; Mul i-algo i hms o clas- si ica ion 9%; K-means 3%; Decision T ee 3%; Logis ic Reg ession 3%; Neu al Ne wo k 3%; Random Fo es 3%; Linea Reg ession 3%, and Rele ance Vec o 3%. To measu e he cogni i e load associa ed wi h men al e o s (e.g., con en comp ehension), we mus dis inguish be ween he objec i i y (subjec i e s objec i e) me hods and he causali y (di ec s indi ec ) me hods. In ou e alu- a ion expe imen s, he subjec i e pe cep ion o he cogni i e load was assessed using sel - epo ing h ough he NASA- TLX [18] ques ionnai e, whe eas objec i e measu emen was ob ained using pe iphe al physiological esponses d i en by he au onomic ne ous sys em (ANS) cap u ed by bio eed- back low-in usi e senso s. In TellBack we p opose using HRV and eye- acking (including pupillog aphy) since he e a e mul iple low- in u- si e solu ions o implemen hese senso s. A o al o 83% o he explo ed s udies used eye- acke and eye- ela ed ea u es (e.g., blink), 66% used skin conduc ance o EDA measu es, and 50% included EEG. The compa ison wi h TellBack was no easy because he ideas p esen ed we e no simila . How- e e , we used [12] as a good example o a ela ed ecen s udy o compa e TellBack wi h. The p ecision and Recall we e he alid measu es o pe o m he compa ison which is shown in Table1. III. METHODOLOGY As desc ibed ea lie , TellBack is a new idea o au oma i- cally iden i y a use ’s di icul ies in comp ehending speci ic passages/elemen s du ing he eading o con en s on able s, lap ops, o sma phones. Ou ision o TellBack is a u u e in elligen and a en i e use -in e ace [16] o enhancing comp ehension using, o example, pop-ups con aining el- e an in o ma ion on he opic ex ac ed om he web could. People could indeed s op eading when encoun e ing a com- p ehension di icul y and could google he meaning o he obscu e elemen s in he con en . Howe e , in p ac ice, his a ely happens, due o he ush o he momen o due o he in e ac ion limi a ions o small de ices such as sma phones, which is he de ac o compu a ional pla o m in bo h he de eloped and de eloping wo ld. The e o e, he key message is o en los because o hese unsol ed di icul ies, which hin- de s p ope decision making and leads o all so s o mis akes, d as ically dis u bing he lea ning p ocess and ha ing a huge nega i e impac on socie y. Figu e 1 below shows a schema ic ep esen a ion o Tell- Back main componen s. This ep esen a ion can be di ided in o h ee dis inc phases: bio-signals acquisi ion, signal p o- cessing, machine lea ning models, and con en anno a ion based on comp ehension di icul y classi ica ion. Recen esul s appea ed in [6] in he con ex o assessing cogni i e s a e (mainly high cogni i e load and dis ac ion s a es) o so wa e p og amme s show ha i is possible o associa e such cogni i e in o ma ion, in eal- ime, o speci ic lines o code o lexical okens, o iden i y such code lines as being mo e suscep ible o ha ing so wa e bugs. In he wo k epo ed he ein, we will expand such p elimina y esul s ob ained in he speci ic con ex o p e en ion o so wa e bugs, o a much b oade con ex o gene al con en comp e- hension. The app oach ollowed in TellBack can be summa- ized in he ollowing componen s as shown in Figu e 1: 1) Real- ime ANS bioma ke s measu emen s (HRV, Pupillog aphy). 2) Fea u es ex ac ion and selec ion o assess cogni i e load spikes and pa e ns. 3) Machine lea ning echniques o imp o e de ec ion o high men al e o /cogni i e load om he ex ac ed ea u es. 4) Eye- acke o localize con en egions ha a e associ- a ed wi h high men al e o . ANS signals a e cap u ed om subjec s while pe o ming con en comp ehension on sc een. The signals used o assess he cogni i e load a e sampled wi h imes amps allowing he iden i ica ion o he momen s when he use is pe o m- ing he comp ehension ask. The signals ob ained a e ed in o he ea u e ex ac ion and selec ion module. In his mod- ule, he domain knowledge is used o choose he mos ele- an bioma ke s ou o he signals. Using hose disc iminan ea u es ex ac ed om he collec ed pupillog aphy and HRV synch onized wi h eye- acke ( ha will be illus a ed in he nex sec ion), elemen al digi al con en s associa ed wi h such ea u es a e classi ied in o wo main bina y classes, i.e., ‘‘Di - icul ’’ and ‘‘No Di icul ’’. Those classes a e conside ed he comp ehension di icul y s a e as shown in Figu e 1. VOLUME 9, 2021 28397 H. Hijazi e al.: In elligen Bio eedback Augmen ed Con en Comp ehension (TellBack) FIGURE 1. A schema ic ep esen a ion o TellBack. The eye- acke is employed as he in o ma ion sou ce o p o ide gaze geome ical in o ma ion (i.e., spa ial in o - ma ion) ela ed o elemen s classi ied wi h comp ehension ‘‘di icul y’’. In he classi ica ion module, a se o classi ie s a e sug- ges ed o ob ain he bes esul s in e ms o accu acy, p eci- sion, and ecall. To ensu e he in e p e abili y o he decision, we included he use o he Decision T ee classi ie among o he classi ie s, ega dless o i s known low accu acy in some p oblems. I is well known ha isola ed ANS mani es a ions a e non- speci ic, i.e., mul iple s imulus and physiological p ocesses migh induce simila ANS modula ion o a speci ic ANS physiological mani es a ion such as HRV o pupil diame e a iabili y. To educe his non-speci ici y, we p opose o combine dis inc ANS mani es a ions using a da a usion app oach o exploi speci ic pa e ns o ac i a ion. In his sense, he e alua ion p esen ed in his pape ep- esen s a less ich scena io ha only uses HRV, pupillome y, and eye- acking, which sugges s ha ou i s esul s p o ide a conse a i e iew on he accu acy, p ecision, and ecall o TellBack. IV. DATASET, PROTOCOL AND METHOD This sec ion desc ibes he da ase ha was used o p o e he concep o TellBack and discusses he expe imen al p o ocol and he me hods used o analyze he esul ing da a. A. DATASET The da ase ha was used o his s udy is an al eady exis ing publicly a ailable da ase , which was o igina ed om he BASE p ojec men ioned in [4], [5], [6]. The da ase was de eloped in one o he p ojec ’s s udies o moni o physio- logic eac ions and men al e o ha a e associa ed wi h code comp ehension in di e en complexi ies using non-in usi e biosenso s. The biosenso s ha we e used in he expe imen a e he ECG and Eye- acking wi h Pupillog aphy. The con- olled expe imen in ol ed 30 subjec s expe ienced in Ja a. The subjec s expe imen ed in he same oom wi h he same condi ions while being equipped wi h men ioned wea able non-in usi e senso s. The expe imen p o ocol o he BASE s udy included he ollowing s eps: 1) Baseline ac i i y o le he subjec s look a an emp y g ey sc een wi h a black c oss in he cen e o 30 seconds o de ach pa icipan s om any ac i i y. 2) Re e ence Ac i i y ha comp ises a sc een wi h a Na u al Language ex o 60 seconds. 3) Baseline ac i i y o le he subjec s look a an emp y g ey sc een wi h a black c oss in he cen e o 30 seconds o de ach pa icipan s om any ac i i y. 4) Code comp ehension comp ises h ee code p og ams wi h di e en complexi ies (c1, c2, c3). 5) Baseline ac i i y o le subjec s look a an emp y g ey sc een wi h a black c oss in he cen e o 30 seconds o de ach pa icipan s om any ac i i y. 6) Su ey using NASA-TLX o assess he subjec s’ men- al e o s while comp ehending he code. 7) Con ol ques ions o check i he subjec unde s ood he codes co ec ly. This expe imen yielded 90 da ase s co esponding o he 30 olun ee s wi h 3 asks (i.e., c1, c2, c3). In his expe imen , he changes in HRV and Pupil Diame e (PD) bioma ke s we e analyzed du ing code comp ehension asks o de ec momen s ha co espond o he high men al e o o he p og amme . Usually, hose momen s co espond o HRV and pupil spikes (i.e., peaks). These momen s we e mapped o co esponding spa ial coo dina es o code loca ions using eye- acking in o ma ion. B. PROTOCOL AND METHOD The main goal o he BASE p ojec was o assess he cogni i e load o p og amme s and anno a e he code lines ha a e mo e p one o ha e so wa e bugs. The anno a ion o hose po en ially p oblema ic code lines would ale he p og am- me s online o e ise hose code lines. Howe e , in ou s udy, TellBack uses he same echnique (i.e., using bio eedback 28398 VOLUME 9, 2021 H. Hijazi e al.: In elligen Bio eedback Augmen ed Con en Comp ehension (TellBack) wea ables o assess cogni i e load), bu wi h a di e en pu - pose and scope. The main goal o TellBack is o p edic con en comp ehension di icul y based on assessing he use cogni i e load in eal- ime and o use he eye- acke syn- ch onously o localize he con en elemen al pa s ha caused he comp ehension di icul y. Since he p o ocol used o p oduce he da ase a ailable om he BASE s udy was no designed o he iden i ica- ion o use ’s comp ehension di icul ies while eading he code ( he goal in he BASE p ojec was p og amme ’s e o s and bugs), we es ablished a speci ic p o ocol (adding a new laye o he ini ial p o ocol) ha u ilizes he da ase al eady a ailable and enhances such da ase wi h addi ional da a o e alua e he accu acy o ou new app oach. Mo e speci ically, he addi ional da a consis s o he iden i ica ion o he code egions ha a e conside ed easy/di icul o unde s and by he subjec s. Ou speci ic p o ocol assumes he ollowing, which akes he o m o a da a-ga he ing in e iew: 1) The same subjec s o he BASE s udy we e in i ed o a emo e da a ga he ing in e iew. 2) The selec ion o he subjec s was na owed down o hose who ha e mos likely he same expe ience in Ja a p og amming compa ed wi h he expe imen ime (i.e., hey ha e no gained addi ional expe ience in Ja a). 3) The selec ed subjec s we e asked o ill a o m indi- ca ing hei cu en posi ion, pe sonal e alua ion o he gained expe ience in p og amming languages since he ime o he BASE s udy, and he cu en p og amming languages ha hey ha e been p ac icing o he ime. 4) The same code snippe s (c1, c2, c3) ha we e used in he BASE s udy we e p esen ed o he subjec s. These snippe s ep esen codes o di e en complexi y le els anging om Vg =3 o c1 o Vg =14 o c3 based on McCabe me ic. 5) The main ask was asking he subjec s o ead and comp ehend he code snippe s again wi h he same amoun o ime gi en in he o iginal s udy. 6) An addi ional laye o he o iginal p o ocol was added by asking he subjec s o manually label he code lines wi h h ee colo s: ed o mos di icul , o ange o di icul , and g een o easy. Highligh ing he code lines was based on he subjec i ely pe cei ed di icul y le el. 7) The same con ol ques ions we e p esen ed o he sub- jec s o check hei code comp ehension. 8) The da a analysis was ini ially pe o med by ex ac ing he pupillog aphy and HRV signals ha a e associa ed wi h he ime window o each anno a ed egion in poin 6 based on he gaze in o ma ion. 9) By epea ing hose s eps o each selec ed subjec , a new da ase was de eloped, which comp ises he pupillog aphy and HRV ea u es ha co espond o di e en code lines and egions labeled by he subjec s acco ding o he di icul y. Fo simplici y in he analysis phase, we conside ed o ange and ed colo s indica ed in poin 6 abo e as ‘‘Di icul ’’, whe eas g een was conside ed as ‘‘No Di icul ’’. We hypo hesize ha i he subjec in he new da a-ga he ing in e iew highligh ed a ce ain segmen as ‘‘Di icul ’’, hen i would ha e been di icul a he ime o he expe imen o him o he o comp ehend ha segmen o code, and hus his would enable us o compa e be ween he code segmen s highligh ed ‘‘di icul ’’ and he HRV and pupilome e signals acqui ed a he o iginal s udy ime. Likewise, labeled code segmen s wi h ‘‘No Di icul ’’ a e also compa ed wi h he co esponding HRV and pupillog aphy signals. This would lead us o p o e whe he HRV and pupillog aphy ea u es could be used o iden i y wi h adequa e ime esolu ion di - e en con en pieces whe e use s a e encoun e ing comp e- hension di icul ies. The new laye o p o ocol men ioned in poin 6 abo e would enable us o obse e o which ex en he eye- acke was able o iden i y he egions/lines o code ha co espond o high men al e o . The da a-ga he ing in e iew ook place h ough Zoom due o he Co id-19 ou b eak. A o al o 11 subjec s we e in e iewed, as shown in Table 2, and 30 di e en samples we e ob ained. Only one o he subjec s was excluded because he e was no co esponding su icien in o ma ion om he o iginal s udy. Table 2 shows he subjec s’ in o ma ion p o- ile. The a e age ime o in e iews was 60 minu es. A i s , subjec s we e o ien ed o he ask ha is equi ed om hem h ough an explana o y ideo. A e ha , hey we e p o ided wi h he codes (i.e., c1, c2, and c3) o ead and comp ehend hem sepa a ely. A s opwa ch ime was used in each s ep. The subjec s we e cons an ly asked o highligh he code lines whene e hey eel ha hese code lines needed mo e men al e o s o g asp. TABLE 2. Subjec s in o ma ion p o ile. VOLUME 9, 2021 28399 H. Hijazi e al.: In elligen Bio eedback Augmen ed Con en Comp ehension (TellBack) The subjec s we e asked abou he cu en p og amming languages ha hey ha e been wo king on since he ime o he o iginal s udy. We we e conse a i e abou hose who acqui ed excessi e expe ience in Ja a (i.e., he language o he code snippe s) because hey may no expe ience he same di icul y ha hey encoun e ed a he ime o he o iginal expe imen . A e ga he ing he needed in o ma ion, he analysis o he esul s was ca ied ou as ollows and as illus a ed in pseudo-code 1: 1) Ex ac ing he ime ins an s o pupillog aphy and HRV singles o each labeled code line o egion by he subjec . 2) Calcula ing he ollowing ea u es (numbe o HRV LF/HF a io spikes which ep esen s he a io be ween he low and he high equency o he HRV signal, he numbe o pupillog aphy LF/HF spikes, mean o LF/HF spikes alues o pupillog aphy, mean o LF/HF spikes alues o HRV). 3) Labeling hose ea u es wi h a bina y di icul y le el (Di icul , no Di icul ) as anno a ed by he subjec s in he da a-ga he ing session. 4) Pe o ming unpai ed - es o examine i he e is a signi ican s a is ical di e ence be ween he wo di i- cul y s a es acco ding o he gi en ea u es in poin 2 (assuming a no mal dis ibu ion). 5) Applying G id Sea ch o ind he op imum hype pa- ame e s. 6) Using lea e-one-ou c oss- alida ion o e alua e he esul ed model. 7) Using K-Fold c oss- alida ion o compa e wi h poin 6. Pseudo-Code 1 Mapping Ex ac ed Fea u es wi h Regional Pa s o Con en on Sc een Inpu : local_maxima(signal): signal {HRV, pupil}, S, S=(s1, s2,..,sn) S. ime: ins an ime o S ec o EyeGazeTime =Time_ins an (signal) signal {eye acke }, eye gaze ime ins an s – ime ins an s o subjec s looking a sc een. Ou pu : eye_gaze_coo dina es associa ed wi h S 1: Fo each s S do 2: T(s)←S. ime 3: Fo each s S do 4: EGT(T(s)) ←EyeGazeTime 5: Find samples n whe e 6: EGT(T(s))<=T(s)+δAND EGT(T(s)) >=T(s)- δ); δis he sample ange whe e T(s) spans. 7: Fo each sample ange n EGT(T(s)) do 8: Find eye_gaze_coo dina es(n) whe e subjec s we e looking a ha ime ins an (when spikes occu ). 9: Re u n (eye_gaze_coo dina es(n)) 8) Iden i ying he signals pa e ns (i.e., HRV and pupil- log aphy) ha a e associa ed wi h he di icul y in unde s anding using di e en classi ie s (SVM bo h he linea and he adial basis unc ion, Random Fo es , KNN, Na´ ’ı e Bayes, Decision T ee). 9) Running he model 50 imes and calcula ing he accu- acy mean and he accu acy s anda d de ia ion. 10) Calcula ing he p ecision, ecall, mic o F1-sco e, mac o F1-sco e. The ollowing Pseudo-Code ep esen s s eps 1-3 in de ail. V. RESULTS AND DISCUSSION As p e ious s udies ha e shown ha pupillog aphy and HRV a e imely manne indica o s o he cogni i e load (and a e non-in usi e), TellBack s a ed wi h hose measu es o assess he cogni i e load associa ed wi h he unde s anding di i- cul y. Howe e , hose measu es a e insu icien o anno a e he elemen al pa s o con en ha caused he comp ehension di icul y. The e o e, eye- acking was in oduced o ell us whe e he use is looking whene e he pupil o HRV signal spikes abo e he h eshold, indica ing an ab up inc ease in he use ’s cogni i e load. A e pe o ming he analysis, he p elimina y esul s showed ha mos o he code egions ha we e di icul o he subjec s o unde s and we e also men ally demanding a he ime o he in e iew (shown by sel -anno a ions). Figu e 2 (abo e) shows an example o one o he subjec s. In his Figu e, we can see ha he men ally demanding code egions (Figu e 2-D) co espond in gene al o high gaze den- si y (Figu e 2-B and C). Simila ly, he pupil and HRV spikes ended o inc ease abo e a ce ain h eshold (Figu e 2-E) when he ask is conside ed di icul o mos di icul . The h eshold was de e mined based on da a obse a ions. Likewise, he pupil and HRV spikes we e de ined based on empi ic da a obse a ion and analysis o di e en subjec s. Using gaze and saccades o in e he use ’s cogni i e load equi es he in eg a ion in ime, which means ha he p e- cision in he ime domain is poo . On he con a y, pupil, and HRV ea u es p o ided an accu a e assessmen o cogni i e load in he ime domain (momen s when he spikes occu ). As obse ed in Figu e 2, he code lines anno a ed as di - icul o mos di icul (o ange and ed colo espec i ely) by he subjec in ou da a-ga he ing in e iews co espond o a high numbe o HRV and pupillog aphy spikes (o ange appea s in his example). Tha means he highe numbe o HRV and pupil spikes as seen in (Figu e 2-E) wi hin a speci ic ime window, he mo e po en ial e idence o high cogni- i e load induced on he subjec by unde s anding e o s. No e ha o iden i y he poin in he code ha co esponds o an HRV o pupil spike (i.e., he sc een a ea whe e he use was looking a when he spike occu ed) we needed o ind he gaze egion in pa B o Figu e 2 ha is e ically aligned wi h he spike, and hen o go o he igh and ind he code lines (pa C and D o he igu e) ha ep esen he eye- acking in o ma ion and he sel -labeling o code 28400 VOLUME 9, 2021 H. Hijazi e al.: In elligen Bio eedback Augmen ed Con en Comp ehension (TellBack) FIGURE 2. Pupil and HRV signals o he subjec no 9 in compa ison wi h his sel -labeling: A) Eye gaze densi y co esponding o he y-axis o he code. B) Clus e s o e ime and gaze eloci y; b) he pupillog aphy and HRV ex ac LH ea u es ex ac ed. C) The ed do s ha ep esen he eye gaze geome ical dis ibu ion supe imposed on he code. D) The addi ional laye o he o iginal p o ocol shows labeled code lines by subjec s. E) The co esponding pupillog aphy and HRV signals and spikes. egions, espec i ely. Be o e applying he empi ical analysis, we could see om Figu e 2 (C and D) ha he e is a clea mapping be ween wha eye- acke ells us abou he men al e o in speci ic lines/ egions o code and he labeled code lines/ egions by subjec s as appea ed in pa D o he same Figu e. To e alua e he bioma ke s ha co espond o each egion, he ollowing ea u es we e analyzed: he numbe o HRV LF/HF spikes, he numbe o pupillog aphy LF/HF spikes, he mean o LF/HF a io spikes alues o pupillog- aphy, mean o LF/HF a io spikes alues o HRV based on he ime ins an s window o he subjec s’ gaze ha is co esponding o he anno a ed code egions. An unpai ed - es wi h a con idence in e al o 95% was pe o med o examine i he e is a s a is ically signi ican di e ence be ween he numbe o pupillog aphy LF/HF a io spikes in he wo di icul y g oups (i.e., di icul , no di icul ). The unpai ed - es wi h p − alue =0.0095 shows ha he means a e s a is ically di e en . In he ‘‘di icul ’’ g oup (M =9.5,SD =6.5), whe eas in he ‘‘no di icul ’’ g oup (M =3.055556,SD =6.5). Likewise, he unpai ed - es shows ha he means o he numbe o HRV LF/HF a io spikes wi h p − alue =0.0002975 a e s a is ically di e en in he wo di icul y s a es., The same es shows also ha he e is a s a is ically signi ican di e ence be ween he means o pupillog aphy LF/HF a io spikes alues in he wo di icul y g oups wi h (M =5.10200,SD =2.88) in he ‘‘di icul ’’ g oup, and wi h (M =4.07444,SD = 2.86) in he ‘‘no di icul ’’ g oup wi h p- alue =0.3733. The e o e, he al e na i e hypo hesis is ue which assumes he s a is ically signi ican di e ences. The ea u es co e- la ion shown in Figu e 3 abo e shows clea disc imina ion be ween he ‘‘Di icul y’’ and ‘‘No di icul y’’ beha io . The co ela ion sco es be ween he bioma ke s such as HRV and Pupil spikes numbe and he di icul y pe cei ed beha io o subjec s shown by hei sel -anno a ions o he code egions indica e a s ong co ela ion as shown in Figu e 3. Likewise, he box plo in Figu e 4 shows selec ed ea u es dis ibu ions agains he di icul y g oups o egions o codes as pe cei ed by subjec s. Figu e 4a shows ha he highe he numbe o pupil spikes wi hin a code egion, he highe he di icul y pe cei ed by he subjec . Simila ly, Figu e 4b indica es ha he inc easing numbe o HRV spikes is p opo - ional o he di icul y s a e in comp ehending egions o code. F om his Figu e, we can see he di e ences in he means, medians, and minimum alues o he wo di e en g oups. The ela i ely high sp ead o HRV da a om he cen e can be explained in e ms o he high sensi i i y o he HRV signal o di e en cogni i e s a es such as anxi- e y, s ess, and a igue. Those bioma ke s (i.e., ea u es) we e ed in o 5 di e en classi ie s namely, Suppo Vec- o Machine (SVM) wi h bo h a linea ke nel and a Radial Basis Func ion ke nel, Random Fo es , K-Nea es Neighbo (KNN), Decision T ee, and Gaussian Na´ ’ı e Bayes. Due o he limi ed da ase , and o a oid any so o da a o e i ing, his wo k used he lea e-one-ou c oss- alida ion (LOOCV) me hod [19]. The LOOCV uses one sample o he aining da a o alida ion and uses he es o he da a o aining un il i co e s all he da a eco ds. Howe e , o making u he compa isons wi h o he me hods o c oss- alida ion, K- old c oss- alida ion [26] was also examined. All classi ie s we e un 50 imes o calcula e he mean and he s anda d de ia ion o he accu acies as shown VOLUME 9, 2021 28401