Virtual reality and machine learning in the automatic photoparoxysmal response detection

S.I.: COMPUTATIONAL-BASED BIOMARKERS FOR MENTAL AND EMOTIONAL HEALTH(CBMEH2021) Vi ual eali y and machine lea ning in he au oma ic pho opa oxysmal esponse de ec ion Fe nando Moncada 1 •So ı ´a Ma ı ´n 2 •Vı ´c o M. Gonza ´lez 1 •Vı ´c o M. A ´l a ez 2 •Bea iz Ga cı ´a-Lo ´pez 3 • Ana Isabel Go ´mez-Mene ´ndez 3 •Jose ´R. Villa 2 Recei ed: 9 No embe 2021 / Accep ed: 4 Janua y 2022 The Au ho (s) 2022 Abs ac Pho osensi i i y, in ela ion o epilepsy, is a gene ically de e mined condi ion in which pa ien s ha e epilep ic seizu es o di e en se e i y p o oked by isual s imuli. I can be diagnosed by de ec ing epilep i o m discha ges in hei elec- oencephalog am (EEG), known as pho opa oxysmal esponses (PPR). The mos accep ed PPR de ec ion me hod—a manual me hod—conside ed as he s anda d one, consis s in submi ing he subjec o in e mi en pho ic s imula ion (IPS), i.e. a lashing ligh s imula ion a inc easing and dec easing licke ing equencies in a hospi al oom unde con olled ambien condi ions, while a he same ime eco ding he /his b ain esponse by means o EEG signals. This esea ch ocuses on in oducing i ual eali y (VR) in his con ex , adding, o he con en ional in as uc u e a mo e lexible one ha can be p og ammed and ha will allow de eloping a much wide and iche se o expe imen s in o de o de ec neu ological illnesses, and o s udy subjec s’ beha iou s au oma ically. The loop includes he subjec , he VR de ice, he EEG in as uc u e and a compu e o analyse and moni o he EEG signal and, in some cases, p o ide eedback o he VR. As will be shown, AI modelling will be needed in he au oma ic de ec ion o PPR, bu i would also be used in ex ending he unc ionali y o his sys em wi h mo e ad anced ea u es. This sys em is cu en ly in s udy wi h subjec s a Bu gos Uni e si y Hospi al, Spain. Keywo ds Elec oence alog am Vi ual eali y Pho opa oxysmal esponse Machine lea ning &Vı ´c o M. Gonza ´lez [email p o ec ed] Fe nando Moncada [email p o ec ed] So ı ´a Ma ı ´n [email p o ec ed] Vı ´c o M. A ´l a ez [email p o ec ed] Bea iz Ga cı ´a-Lo ´pez [email p o ec ed] Ana Isabel Go ´mez-Mene ´ndez [email p o ec ed] Jose ´R. Villa [email p o ec ed] 1 Elec ical Enginee ing Depa men , Uni e si y o O iedo, As u ias, Spain 2 Compu e Science Depa men , Uni e si y o O iedo, As u ias, Spain 3 Neu ophysiology Depa men , Bu gos Uni e si y Hospi al, Bu gos, Spain 123 Neu al Compu ing and Applica ions h ps://doi.o g/10.1007/s00521-022-06940-z(0123456789().,- olV)(0123456789().,- olV) 1 In oduc ion Vi ual eali y (VR) is inc easingly becoming pa o ou daily li es and i s applicabili y is widening [29]: om ideo games and en e aining, o educa ion, medical ea men and ehabili a ion, mili a y applica ions, a chi- ec u al and u ban design, digi al ma ke ing and ac i ism, enginee ing and obo ics, ine a s, he i age and a chaeol- ogy, occupa ional sa e y, social sciences, psychology and many mo e. Fo his eason, i is no only impo an o unde s and how his echnology can a ec ou b ains, bu also i s po en ial o ex end he cu en scope o neu ological diseases de ec ion me hods. Using VR in new speci ic igge ing scena ios will allow con olling and ex ending he a ailable unc ionali ies o he isual s imula ion sys- ems used o he de ec ion o such pa hologies [4]. This e lec ion is no new. Back in he 1990s, esea ch pe o med o de ec he isk o ele ision and ideo games exposu e esul ed in a se o ecommenda ions o TV manu ac u e s and ideo games de elope s. Simul ane- ously, he s udies pe o med we e he basis o many new uses o hese echniques in educa ion [9] o in ehabili a ion [8], o name a ew. Pho osensi i i y is an abno mal isual sensi i i y o he b ain esul ing in a pho opa oxysmal esponse (PPR), i.e. a b ain epilep ic discha ge consis ing o a co ical spike o a spike-and-wa e p o oked by a lash o a isual s imuli [26]. The e a e ou di e en ypes o PPR esul ing om di e en b ain esponses o in e mi en ligh s imula ion [31]. E en hough PPR can be ound in non-epilep ic elec oencephalog am (EEG) eco dings, i is s ongly associa ed wi h epilepsy [16]. The ele ance o he PPR elies on i s associa ion wi h speci ic epilep ic synd omes [34] and moni o ing o ea men in a clinical con ex [20]. The pho osensi i i y ange is ela ed o he likelihood o occu ence o e lex seizu es in daily li e; hus, i is o c ucial in e es o ea ly de ec PPR. The mos commonly used p ocedu e o de ec PPR, known as in e mi en pho ic s imula ion (IPS), is desc ibed in [23]. I p oposes o sub- mi he subjec o a se ies o ligh lashes while simul a- neously moni o ing he b ain ac i i y using EEG signals, acco ding o he Eu opean consensus g oup me hodology o isual s imula ion de ined in 2012 [27]. The ligh la- shes equency is i s inc eased om a minimum o a maximum o un il a PPR is obse ed (wha e e happen i s ). I no PPR is obse ed, he p ocess inishes. Should a PPR occu , he p ocedu e is epea ed in an in e se manne , i.e. s a ing wi h a op licke ing equency ha is g adually dec eased un il a minimum is eached o a PPR happen. The aim is o de ec he minimum and he maximum e- quencies a which he subjec shows PPR, i any, wi h he minimal exposu e. The de ec ion o PPR is usually pe o med by physi- cians, i.e. clinical neu ophysiologis s and nu ses, who manually e iew he EEG signals’ a iabili y in sea ch o PPR [2,14,22], aking in o accoun each subjec ’s clinical con ex such as age, seizu e and amily his o y. To ou bes knowledge, no au oma ed me hod o he de ec ion o PPR has been de eloped so a . This esea ch is mainly ocused on he design o a new and sa e p ocedu e o he au oma ic de ec ion o PPR wi hin EEG signals using digi al bio- ma ke s implemen ed using VR and AI echniques. In his sense, [25] designed a PPR de ec ion me hod by analysing he po en ial and oscilla ion o he esponse p o oked by a lashing s imula ion, bu ollowing a di - e en s imula ion pa e n om he s anda d one. The e a e o he ecen s udies ha analyse he pho osensi i i y and epilepsy based on o he gene alized discha ges o seizu es han PPRs: in [18], a de ec ion me hod based on he band ampli ude luc ua ion compu ed om a high- equency and a low- equency componen s o he EEG windows in each EEG channel is p oposed; [30] applied he ex eme g adi- en boos echnique o he classi ica ion o seizu es in wo di e en ways (applying a s anda d pa i ioning o he da a and applying a lea e-one-ou c oss- alida ion scheme), while a channel-independen long sho - e m memo y ne wo k is used in [5]; he in o ma ion ex ac ed om EEG and elec oca diog am (ECG) signals is used in [35]ina mul i-modal neu al ne wo k which analyse he da a in h ee di e en ways (only EEG da a wi h a con olu ional LSTM ne wo k; only ECG da a wi h a esidual con olu ional ne wo k; and a used ne wo k which combines he ou pu s o he indi idual ne wo ks o pe o m he inal classi ica- ion); in [6], K-nea es neighbou s and a i icial neu al ne wo ks a e used o he de ec ion o ic al discha ges and in e -ic al s a es; [32] p oposed an EEG single-channel analysis applying h ee ypes o isibili y g aphs (basic, ho izon al and di e ence) o ep esen di e en EEG pa e ns. O he s udies make use o addi ional and di e en bio- me ic measu es o he same pu pose, such us elec o- ca diog ams (ECG) [10,11,28], elec omyog ams (EMG) [3,36] o magne oence alog ams (MEG) [24]. This s udy p oposes an al e na i e o he con en ional IPS p ocedu e o PPR de ec ion using VR and machine lea ning (ML). This esea ch is ocused on he mos e- quen PPR ype; hus, he PPR de ec ion s ill needs mo e esea ch wo k as i is no comple ely sol ed. Howe e , in oducing VR would e en ually allow o s udy and o de elop new and sa e p ocedu es o PPR de ec ion. Since he VR in as uc u e is much mo e lexible han he s anda d one, i can be easily con igu ed o ca y ou new assays and s imula ion pa adigms, allowing o an ad anced IPS/Visual s imula ion sys em. Ou p oposal includes a VR de ice wi h a wi eless connec ion o a Neu al Compu ing and Applica ions 123 compu e ha has access o he da a ga he ed by he EEG senso s. The subjec mus wea a he same ime bo h he EEG cap and a head moun ed display (HDM). Fu he - mo e, a plausible solu ion o he au oma ic PPR de ec ion is p oposed using some ea u es ex ac ed om he EEG signals in an a e age mon age, and classic ML echniques. An a e age mon age is used so ha he elec onega i e PPR discha ge will exp ess wi h an upwa d de lec ion o he EEG signal in he a ec ed channel. The main con ibu ions o his esea ch a e: – To in oduce VR in a close loop wi h AI and ML models, so medical p ocedu es could be e isi ed and enhanced. This con ibu ion can lead in he nea u u e o mo e ad anced diagnose es s and p ocedu es. – To p o ide he neu ophysiology depa men a Bu gos Uni e si y Hospi al wi h a no el ins umen o analyse he impac o VR in ela ion o pho osensi i i y by in eg a ing his solu ion in i s daily wo k. – To de elop ML models o de ec anomalies in he EEG eco dings when he pa ien is lashed using ei he VR- ML IPS o con en ional IPS. The s uc u e o his s udy is as ollows. The nex sec ion ocuses on he desc ip ion o he p oposal, de ailing he di e en elemen s in he loop. Sec ion 3gi es de ails o he expe imen a ion se -up ha has been ca ied ou a he p oposal, while Sec . 4includes all he ob ained esul s and he discussion on hem. The inal sec ion d aws he con- clusion o his esea ch. 2 A p o o ype o VR-ML IPS p ocedu e The p oposed solu ion complemen s he con en ional se - up by means o in oducing a VR de ice ha he subjec mus wea along wi h an EEG cap (see Fig. 1). The signals om he EEG senso s a e analysed using well-known ML echniques. The in elligen module will e en ually con ol he VR con en s o gain inc eased capabili ies and o pe - o m mo e complex assays. This sec ion gi es de ails on each o he main modules: he VR pa (nex subsec ion) and he ML module (Sec . 2.2). 2.1 VR design o IPS and PPR de ec ion Flashing ligh s a e one o he main igge s o pho osen- si i e esponses. VR-Pho osense [15] is a so wa e designed o de ec pho ic-d i ing and PPR while using VR and wea ing a head-moun ed display (HMD). VR-Pho o- sense o e s a VR scena io wi h IPS in o de o measu e b ain esponses o lashing ligh s a a ious equencies and using di e en sequences. The main goal o his so wa e is o simula e con en ional IPS es s in a i ual eali y en i onmen . Con en ional IPS places he ligh s imula o a a e y sho dis ance om he pa ien ’s eyes, c ea ing high exposu e o licke ing ligh s which a e pe cei ed wi h in ensi y e en when he pa ien ’s eyes a e closed. In o de o emula e he exposu e and senso y e ec caused by he con en ional IPS ligh s imula o , a i ual eali y scene has been designed as a 3D enclosing sphe ical dome en i onmen wi h he pa ien ’s ision placed a i s cen e (see Fig. 4). This design supp esses pa ien ’s pe iphe al ision and inc eases he ocus on he isual s imuli e en wi h he eyes closed. Fig. 1 To he le , he con en ional se o IPS p ocedu e. To he igh , he new VR-ML se o IPS p ocedu e, including au oma ic EEG analysis and PPR de ec ion Neu al Compu ing and Applica ions 123 The VR-Pho osense’s se -up is ai ly simple. A e downloading and s a ing he app, he ca dboa d iewe , in o which a sma phone is inse ed, is secu ed o he subjec ’s head using he adap able s aps, lea ing he uppe pa o he head clea . Then, he EEG cap is easily se up on he subjec ’s head co e ing all necessa y poin s o con ac as shown in Fig. 2. The so wa e sys em is di ided in o wo pa s: he ligh s imula ion and he moni o ing so wa e. The VR-Pho o- sense’s de aul con igu a ion s imula es wi h whi e ligh combined wi h da k black backg ound, emula ing he con en ional IPS. In oducing an inno a ion o con en- ional IPS, VR-Pho osene allows he colou o he licke - ing ligh and backg ound o be changed om he de aul con igu a ion, so we designed wo colou ed se ings in addi ion o he whi e one: i) one wi h b igh ed lashes and deep blue backg ound; ii) and ano he wi h deep blue la- shes wi h da k black backg ound. These ones may in lu- ence he b ain discha ges, and combined may be mo e o less p o oca i e when compa ed o he de aul one, allowing o s udy b ain beha iou eac ing o he s imula- ion wi h bo h o hese scena ios. VR-Pho osense has also been designed o esemble he con en ional s imula ion se -up and o acili a e he wo k o physicians, bo h clinical neu ophysiologis s and nu ses, while conduc ing pho osensi i i y es s a he hospi al. This sys em includes a moni o ing ea u e ha allows o obse e in eal ime wha is happening on he VR s imula ion ia a web se e . This is hanks o he use o Websocke s, a communica ions p o ocol ha o e s ull-duplex commu- nica ion channels o e a single TCP connec ion making i as e and wi h low la ency o upda e sys em. This moni o ing ea u e along wi h EEG eco dings ansla es in o a ull co e age IPS scena io ha closely esembles he con en ional se -up, wi h he di e ence o eplacing he adi ional s imula ion de ice wi h a low cos VR headse and he VR-Pho osense so wa e. The EEG se - up o ca y ou VR-Pho osense es ing a he hospi al consis s o using Na us B ain moni o ing and Neu owo ks so wa e o EEG eco ding. Fu he mo e, he di e en ha dwa e and so wa e elemen s can be easily mixed and eplaced as hey a e comple ely independen om each o he . Fo example, he VR-Pho osense can be con igu ed o wo k wi h highe end HDMs such as Oculus; o EEG eco ding can be pe o med using a di e en de ice such us OpenBCI 3D p in ed de ice and open sou ce so wa e, which we e used a he uni e si y EEG labo a o y o p elimina y expe imen s. All in all, VR-Pho osense o e s an inno a i e, low-cos and c oss-pla o m IPS scena io in i ual eali y, wi h mo e upcoming ea u es o be included aimed o achie e a mo e de ailed diagnosis. This ligh s imula ion so wa e has been implemen ed in Uni y 3D using he p og amming language C#. Uni y is a ideo game de elopmen engine ha allows designing 3D scenes by means o a isual edi o and he p og amming o gameplay e en s ia sc ip ing. These sc ip s a e associa ed wi h he game objec s included in he scene so ha hey beha e in he desi ed way. Wi hin he scene, he asse s included a e s uc u ed in so wa e componen s as shown in Fig. 3. – Con ols: au o-gene a ed sc ip and inpu sys em. The new Inpu Sys em 1.0.2 o Uni y has been used, which allows o se up he desi ed inpu s h ough an in e ace. Di e en inpu s can be en e ed om di e en de ices o he same ac ion. This makes con igu a ion and connec i i y wi h di e en ypes o inpu ha dwa e, such as keyboa ds and game con olle s, seamless. Fig. 2 VR-Pho osense se -up wi h a ca dboa d iewe and an OpenBCI EEG headse Fig. 3 VR-Pho osense so wa e package diag am Neu al Compu ing and Applica ions 123 – GoogleVR: an SDK o And oid ha allows he c ea ion o i ual eali y applica ions o be used wi h Google Ca dboa d HMD. – Plugins: se o And oid plugins needed o expo applica ions o de ices using his ope a ing sys em. – Resou ces: all he sc ip s de eloped o ligh ing and connec ion wi h he moni o ing side. This package also includes shade s and ma e ials used in he scene. – Scenes: he designed i ual eali y scene (Fig. 4). – XR: de aul Uni y con igu a ion o ex ended ( i ual and augmen ed) eali y apps. The de eloped scene has ou objec s o GameObjec s: he came a, wo poin ligh s and a sphe e (see Fig. 4). The pu pose o he sphe e is o p o ide a black backg ound, placing he came a inside i and in e ing he no mals, hus a oiding anspa ency. This in e sion p ocess is done by adding a cus om shade o a new ma e ial assigning i o he sphe e. As o he spo ligh s, one o hem is he main whi e ligh and he o he p o ides a blue backg ound o he cus om colou unc ionali y. All de eloped sc ip s use he MonoBeha iou buil -in class, since Uni y uses a s anda d Mono un- ime imple- men a ion. These classes a e conside ed bluep in s and each ime hey a e associa ed wi h a GameObjec , a new ins ance o he objec de ined by ha bluep in is c ea ed. In hem, wo me hods a e p ede ined: S a and Upda e. In his case, only S a is used, which will be called by Uni y when loading he scene. I is also impo an o no e he use o co u ines. Co u ines a e unc ions ha allow pausing and esuming he execu ion in he ame in which i was paused. In his case hey a e used o implemen he lick- e ing e ec in he ligh s and he s imula ion sequences. As o he inpu sys em, wo op ions we e conside ed: emo e con ol and keyboa d. Conside ing he numbe o con igu able pa ame e s and he con olled ac ions o be pe o med by he pe son in cha ge o he s imula ion, he use o only a VR emo e con ol was conside ed insu i- cien . Bea ing in mind he end use and he echnology hey use on a daily basis, he use o a Blue oo h keyboa d was chosen. As a esul , mo e con igu a ions a e a ailable and he commands a e se in he way hese expe s conside mo e com o able o pe o m he s imula ion in he mos e icien possible manne . Howe e , in o de o enable he use o he s imula ion wi h he Oculus Ques 2 glasses, his con igu a ion has also been adap ed o he wo emo e con ols associa ed wi h his HMD. Finally, he lis o commands a ailable is shown in Table 1. 2.1.1 Moni o ing In addi ion o simula ing he IPS en i onmen , i is deemed necessa y o know wha is happening inside he scene du ing he es s. To add ess his p oblem, a moni o ing componen —a web page (h ps:// pho osense.he okuapp. com)—was added o he sys em. This solu ion allows he physicis s o see in eal ime wha is happening in he VR s imula ion, e.g. he e olu ion o he sequences and he commands en e ed by he use . Ano he possibili y was o implemen a sound sys em o communica e he si ua ion h ough audio, bu i was con- side ed less e icien as i was ola ile and did no ha e a isual eco d o he simula ion. To implemen he moni o ing sys em, a WebSoke s componen was used, which allows eal- ime and as communica ion be ween he sma phone applica ion and he web page a oiding he use o a da abase. The desi ed unc ionali y is as shown in Fig. 5. Clien 1, he sma - phone applica ion, sends in o ma ion abou he s imula ion o he se e , and he se e passes i o clien 2, he web page, which displays i on he sc een along wi h a ime s amp. WebSocke s is a p o ocol ha p o ides bidi ec ional, ull duplex communica ion o e a single TCP socke . The en i onmen in which he VR-Pho osense applica ion is used is a ai ly sensi i e one o medical es ing, hus imposing an as as as possible exchange o in o ma ion exigence. WebSocke s is p obably he mos popula p o- ocol o his ype o use cases whe e da a needs o be sen in eal ime. As we ha e seen be o e, Uni y is based on he Mono pla o m as sc ip ing engine, which means ha i wo ks in .NET (C#). This amewo k p o ides a de aul suppo o his p o ocol ha is also suppo ed by Mono, Fig. 4 Scene diag am implemen ed in Uni y 3D Neu al Compu ing and Applica ions 123 h ough he Sys em.Ne .WebSocke s namespace. The ope a ion in his case would be as shown in Fig. 6. The se e has been de eloped using Node. I is a simple se e ha ecei es in o ma ion om he clien and e u ns i . In his case, no hing is done in Uni y wi h he e u ned Table 1 Lis o a ailable commands in VR-pho osense applica ion A ailable commands Ac ion Command (keyboa d) Command (Ques 2) S a En e Righ igge Pause Space ba Righ g ip Rese Backspace Seconda y ouched (R) Inc emen Hz Numpad ?P ima y ouched (R) Whi e ligh B S a (L) Blue ligh A Le g ip Red-blue combina ion R Le igge Upwa d pa e n Up a ow P ima y ouched (L) Downwa d pa e n Down a ow Seconda y ouched (L) Exi Escape S a (R) Fig. 5 Theo e ical unc ionali y o he moni o ing sys em Fig. 6 Sequence diag am o he sys em using WebSocke s Neu al Compu ing and Applica ions 123 in o ma ion; i is he moni o ing clien ha makes use o i . This clien is an HTML page ha connec s o he se e , opens a connec ion, ge s he in o ma ion coming om he se e ha is pa o a message and displays i on he sc een. This HTML clien has been implemen ed in he simples possible way since i is an add-on o he main wo k and i s appea ance in e ms o design is no ele an o he EEG labo a o y s a . 2.2 ML-based PPR de ec ion PPR can be ound in epilep ic synd omes ha p esen seizu es wi h o wi hou isual s imulus igge , and in some subjec s bo h ypes o seizu es a e obse ed. Wal z classi ica ion [31] is used o de ine he exp ession o he PPR om an elec oencephalog aphic poin o iew, in oducing up o ou di e en ypes o PPR: •Type-1: spikes wi hin he occipi al hy hm. •Type-2: pa ie o-occipi al spikes wi h a biphasic slow wa e. •Type-3: pa ie o-occipi al spikes wi h a biphasic slow wa e and sp ead o he on al egion. •Type-4: gene alized spikes and wa es o polyspikes and wa es. All o hem a e depic ed in Fig. 7. Type-4 PPR is he mos equen ly ound in epilep ic synd omes whe e pho osen- si i i y cons i u es a clinical conce n, as i seems o ha e a s ong associa ion—highe han 90%—wi h epilep ic sei- zu es; he de ec ion o his ype o PPR ep esen s he challenge ocused on his esea ch. Besides, in clinical p ac ice, i is equen ha exp ession o PPR is a iable, and in many imes we ob ain PPR ha no necessa ily i wi hin only one ca ego y o hose ini ially de ined by Wal z. E en mo e, he mo phological cha ac e is ics o a gi en subjec ’s PPR may a y because o clinical a iables as doses o an i-epilep ic ea men , sleep quali y, e c. To iden i y Type-4 PPR we p opose he use o sliding windows—one second leng h and a en h o a second shi — ollowed by a p e-p ocessing s ages ha sub ac s he window a e age and pe o ms a ea u e ex ac ion. Well-known ML echniques conside hese ea u es o p opose a inal label o he window. These ML echniques a e applied in h ee pa s ha will be desc ibed in his sec ion. Fi s , he se o ans o ma ions ha will be applied and hei a ionale a e explained in Sec . 2.2.1. Then, he design and deploymen o he ML models a e de ailed in Sec . 2.2.2 and, inally, he aining o he ML pa is desc ibed in Sec . 2.2.3. Fig. 7 The ou ypes o PPR: a ype-1: spikes wi hin he occipi al hy hm; b ype-2: pa ie o-occipi al spikes wi h biphasic slow wa e; c ype-3: pa ie o-occipi al spikes wi h biphasic slow wa e and sp ead o he on al egion; d ype-4: gene alized spikes and wa es Neu al Compu ing and Applica ions 123 2.2.1 Fea u e ex ac ion The selec ion o ma hema ical ans o ma ions o signals ep esen a ion is, pe se, a p oblem ha needs o be ca e- ully add essed. The main poin is o selec ea u es ha , in conjunc ion, include, as a whole, he in o ma ion ha he expe s use o make a decision. The e o e, all he possible windows mus be analysed and signi ican di e ences mus be s a ed be ween he anomalies and he no mal signal s a e. Nowadays, he p oblem o ea u e ans o ma ion is deal wi h deep lea ning (DL) and, mo e speci ically, wi h au o-encode s; howe e , due o he lack o da a o ain he ne wo ks, we le his issue o u u e esea ch. Fo his s udy, we ocused on channels Fz and O2 because hese a e he channels we e PPRs mo e equen ly appea ega dless hei ype. We pay a en ion o hei no mal and abno mal beha iou o he conside ed PPR ype. Some examples o he windows ha migh be aced a e depic ed in Fig. 8. F om he analysis o he signals, he ollowing ea u es se has been selec ed o ep esen each EEG da a window, whe e wis he wid h o he window, e e s he ime s amp o which he ea u e is compu ed, c is he channel—ei he Fz o O2—and dc is he EEG cchannel’s signal wi h an a e age mon age— he PPR exp esses wi h an upwa d de lec ion—a ime s amp : •Cumula i e Fi s de i a i e, also known as he in ensi y o he signal, compu ed as CFDc ¼1 wPw1 i¼0jdc þiþ1dc þij=D. This ea u e has been chosen because PPR p esen high a e o change in he alue o he channel. D ep esen s he in e al be ween consecu i e samples, which is kep cons an . •Cumula i e Second de i a i e compu ed as CSDc ¼1 wPw1 i¼0jdc þiþ22dc þiþ1þdc þij=D2.This ea u e has been selec ed because he e is also a high a e o change in he i s de i a i e, bu no so high as o a e ac s. •Numbe o ele an peaks using he S1 measu emen p oposed in [19] and compu ed as ollows. Equa ion 1 de ines he calcula ion o S1, whe e kis he p ede ined numbe o samples and pis he cu en sample imes amp o which we a e de e mining whe he i is a peak o no . The S1 ans o ma ion ep esen s a scaling o he TS, which makes he peak de ec ion easie using a p ede ined h eshold a. S1ðpÞ¼1 2max p1 i¼pkðdc pdc iÞþmax pþk i¼pþ1ðdc pdc iÞ  ð1Þ So, o each poin o which S1ðpÞcan be compu ed wi hin he EEG sliding window we compu e S1ðpÞ;a peak occu s in ime pi he alue Spis highe han aand is he highes in i s 2kneighbou hood. In he o iginal epo , all he pa ame e s (k,a) whe e ca e ully de e - mined o each p oblem in o de o op imize he peak de ec ion. In his esea ch, kis se o 10 and ais se o h ee imes he s anda d de ia ion o he EEG channel alues when no ac i i y is shown (uppe igh co ne in Fig. 8). •Sum o he absolu e alues, o measu e he a ea unde he cu e o he EEG signal. Fig. 8 Th ee EEG agmen s om di e en condi ions: he le -mos and he cen e eco dings a e conside ed no mal condi ions, while he eco ding a he igh shows a PPR. The eco dings include, om op o bo om, signals om he F3-AVG, Fz-AVG, F4-AVG, O1-AVG and O2-AVG channels, espec i ely, whe e AVG s ands o he a e age o all eco ded elec odes Neu al Compu ing and Applica ions 123 •Maximum di e ences, also known in some ields as amoun o mo emen [1], which measu es he di e - ences be ween he highes and he smalles alues— expec ed o be a high alue. I is calcula ed as MDc ¼jmaxi2½ ; þwðdc iÞmini2½ ; þwðdc iÞj. •A e age Ene gy as p oposed in [33], as he sum o he squa ed disc e e FFT componen s magni udes o he signal. All he ea u es a e s anda dized; gi en a da a se , he a e age and he s anda d de ia ion a e compu ed and used o ans o m he alues in o a no mal dis ibu ion wi h mean 0.0 and s anda d de ia ion 1.0. 2.2.2 Designing and deploying he ML pa The goal in his s age is o ob ain models able o label he p e-p ocessed EEG signal windows as no mal o as PPR. Due o he da a imbalance, he mos in e es ing app oach is o use unsupe ised lea ning, so anomalies can be de ec ed. Howe e , his migh gene a e oo many alse posi i e, so i could be in e es ing o also de elop a complemen a y supe ised lea ning solu ion. The e o e, o his esea ch we p oposed o use, i s , unsupe ised lea ning o ob ain a model ha signals hose anomalous windows and hen o classi y he anomalous windows as PPR o no mal using a supe ised app oach. The unsupe ised lea ning is speci ic o a gi en subjec , while he supe ised lea ning is a gene alised model. Fo his s age, we will de elop models ollowing he wo k low p oposed in Fig. 9. Da a om he EEG senso s is windowed as explained be o e. Fo each window, he a e age is calcula ed and sub ac ed; he ans o ma ions om he p e ious subsec ion a e calcula ed a e wa ds. A one-class classi ie , lea ned o he cu en subjec da a, labels he window as no mal o no . In case a window is labelled as an anomaly, hen he wo-class classi ie , lea ned om o he subjec s, labels he window as PPR o no . Fo he one-class classi ie , an unsupe ised one-class k-nea es neighbou s model (1C-KNN) [12,17] is es ed: his model has been selec ed o i s as aining and e alua ion imes while s ill pe o ming su icien ly accu a e. Fo he wo-class classi ie we p opose K-nea es neighbou (2C-KNN) due o he small numbe o ins ances in he a ailable da a se . Bo h classi ie s a e om he sciki -lea n lib a y o Phy on [21]. 2.2.3 T aining he ML pa T aining he models has wo main s ages as can be seen in Fig. 10: (i) aining he one-class model and (ii) aining he wo-class model. A his momen we ha e wo collec- ions o da a: (a) a collec ion o windows om he cu en subjec (CPDa a), all labelled as no mal, and (b) a collec- ion o windows om he his o ical eco ds (HRDa a), each window wi h i s co esponding no mal o PPR label. CPDa a is used in he one-class aining, while HRDa a is used in he wo-class aining. The i s pa o he aining is he 2C-KNN lea ning using he HRDa a; in case o highly imbalance o he da a se , SMOTE will be used. Di e en alues o he pa ame e K a e es ed o bo h classi ie s o ind he bes pe o ming model. When analysing he da a eco ded o he cu en sub- jec , an inc emen al aining is p oposed. The idea is epea ing he one-class classi ie aining un il a eal PPR be de ec ed a a ce ain lashing equency. Tha is, in case he equency o be es ed is inc eased o example om 4 o 6 Hz, i no PPR is de ec ed in his new s imula ion equency, hen he 1C-KNN is ained including he win- dows ga he ed om he i s equency ange (1–6 Hz), and hen, he nex lashing equency is e alua ed (8 Hz) and he p ocess is epea ed again un il a PPR is de ec ed. The s imula ion equency a which he i s PPR is de ec ed is he cu equency ( c). The p ocess is illus a ed in Fig. 11. This p ocess has been desc ibed o he ollowing lashing equency inc ease sequence—s anda d Fig. 9 The wo k low o he designed app oach. The da a ga he ed om he cu en subjec a e p e-p ocessed. When a da a window comes om equencies smalle o equal o c ( he cu equency, which is he s imula ion equency alue a which he i s PPR appea s), he window is p ese ed o he aining o he one-class models; o he wise, he window is labelled as no mal o as anomaly. In his la e case, he wo-class classi ie labels he window as PPR o no . Howe e , when no window is labelled as including a PPR o he cu en equency, he windows o his equency a e also conside ed and he one class model is e- ained Neu al Compu ing and Applica ions 123 amoun o a ailable da a allows us o do so, such as au o- encode s plus dense laye s o long sho - e m memo y ne wo ks. All o hese imp o emen s ep esen s u u e esea ch wo k. Acknowledgemen s This esea ch has been unded by he Spanish Minis y o Science and Inno a ion unde p ojec MINECO- TIN2017-84804-R, PID2020-112726RB-I00 and he S a e Resea ch Agency (AEI, Spain) unde g an ag eemen No RED2018-102312-T (IA-Biomed). Addi ionally, by he Council o Gijo ´n h ough he Uni e si y Ins i u e o Indus ial Technology o As u ias g an SV-21- GIJON-1-19. Funding Open Access unding p o ided hanks o he CRUE-CSIC ag eemen wi h Sp inge Na u e. Con lic o in e es The au ho s decla e ha hey ha e no con lic o in e es . Open Access This a icle is licensed unde a C ea i e Commons A ibu ion 4.0 In e na ional License, which pe mi s use, sha ing, adap a ion, dis ibu ion and ep oduc ion in any medium o o ma , as long as you gi e app op ia e c edi o he o iginal au ho (s) and he sou ce, p o ide a link o he C ea i e Commons licence, and indica e i changes we e made. The images o o he hi d pa y ma e ial in his a icle a e included in he a icle’s C ea i e Commons licence, unless indica ed o he wise in a c edi line o he ma e ial. I ma e ial is no included in he a icle’s C ea i e Commons licence and you in ended use is no pe mi ed by s a u o y egula ion o exceeds he pe mi ed use, you will need o ob ain pe mission di ec ly om he copy igh holde . To iew a copy o his licence, isi h p://c ea i ecommons. o g/licenses/by/4.0/. Re e ences 1. A ´l a ez A, T i in ˜o G, Co do ´n O (2011) Body pos u e ecogni ion by means o a gene ic uzzy ini e s a e machine. In: IEEE 5 h in e na ional wo kshop on gene ic and e olu iona y uzzy sys- ems (GEFs), pp 60–65 2. Beniczky S, Au lien H, F ancesche i S, da Sil a AM, Bisulli F, Ben es C, Cana oglia L, Fe i L, K y ´sl D, Pe al a AR, Ra ´cz A, C oss JH, A zimanoglou A (2020) In e a e ag eemen o clas- si ica ion o pho opa oxysmal elec oencephalog aphic esponse. Epilepsia. h ps://doi.o g/10.1111/epi.16655 3. Beniczky S, Con adsen I, Henning O, Fab icius M, Wol P (2018) Au oma ed eal- ime de ec ion o onic-clonic seizu es using a wea able EMG de ice. Neu ology. h ps://doi.o g/10.1212/WNL. 0000000000004893 4. B ooks AL, B ahman S, Kap alos B, Nakajima A, Tye man J, Jain LC (2021) Recen ad ances in echnologies o inclusi e well-being. Vi ual pa ien s, gami ica ion and simula ion. In el- ligen sys ems e e ence lib a y se ies, 196, 1 edn. Sp inge 5. Chak aba i S, Swe apadma A, Pa naik PK (2021) A channel independen gene alized seizu e de ec ion me hod o pedia ic epilep ic seizu es. Compu Me hods P og ams Biomed. h ps:// doi.o g/10.1016/j.cmpb.2021.106335 6. Choubey H, Pandey A (2021) A combina ion o s a is ical pa ame e s o he de ec ion o epilepsy and eeg classi ica ion using ann and knn classi ie . Signal Image Video P ocess. h ps:// doi.o g/10.1007/s11760-020-01767-4 7. Cobb S (1947) Pho ic d i ing as a cause o clinical seizu es in epilep ic pa ien s. A ch Neu ol Psych 58(1):70–71 8. Hall C, Vi an i A, Abbey K (2019) Impac o ele ision on nu i ional in ake in communal dining oom se ings among hose wi h acqui ed b ain inju y a pilo s udy. Nu i ion and die e ics. J Die Aus alia 77(4):444–448. h ps://doi.o g/10.1111/1747- 0080.12526 9. Ho man BL, Ho man R, Wessel CB, Shensa A, Woods MS, P imack BA (2018) Use o ic ional medical ele ision in heal h sciences educa ion: a sys ema ic e iew. Ad Heal h Sci Educ 23(1):201–216. h ps://doi.o g/10.1007/s10459-017-9754-5 10. Jahanbekam A, Baumann J, Nass RD, Bauckhage C, Hill H, Elge CE, Su ges R (2021) Pe o mance o ecg-based seizu e de ec ion algo i hms s ongly depends on aining and es con- di ions. Epilepsia Open. h ps://doi.o g/10.1002/epi4.12520 11. Jeppesen J, Fuglsang-F ede iksen A, Johansen P, Ch is ensen J, Wu ¨s enhagen S, Tankisi H, Qe ama E, Hess A, Beniczky S (2019) Seizu e de ec ion based on hea a e a iabili y using a wea able elec oca diog aphy de ice. Epilepsia 60:2105–2113. h ps://doi.o g/10.1111/epi.16343 12. Khan SS, Ahmad A (2018) Rela ionship be ween a ian s o one- class nea es neighbou s and c ea ing hei accu a e ensembles. IEEE T ans Knowl Da a Eng 30:1796–1809 13. Kiloh LG (2013) and McComas. Clinical elec oencephalog a- phy. Bu e wo h-Heinemann, A.J., Ossel on, J.W 14. La sen PM, Wu ¨s enhagen S, Te ney D, Ga della E, Al ing J, Au lien H, Beniczky S (2021) Pho opa oxysmal esponse and i s cha ac e is ics in a la ge eeg da abase using he sco e sys em. Clin Neu ophysiol. h ps://doi.o g/10.1016/j.clinph.2020.10.029 15. Ma ı ´nS,A ´l a ez V, Ga cı ´a-Lo ´pez B, Gonza ´lez VM, Vila JR (2021) V -pho osense: a i ual eali y pho ic s imula ion in e - ace o he s udy o pho osensi i i y. In: P oceedings o 16 h in e na ional con e ence on so compu ing models in indus ial and en i onmen al applica ions (SOCO 2021) so compu ing models in indus ial and en i onmen al applica ions, pp 178–186. Sp inge 16. Michelucci R, Pasini E, Riguzzi P, Ande mann E, Ka ¨l ia ¨inen R, Gen on P (2016) Myoclonus and seizu es in p og essi e myo- clonus epilepsies: pha macology and he apeu ic ials. Epilep Diso de . h ps://doi.o g/10.1684/epd.2016.0861 17. Mun oe D, Madden MG (2005) Mul i-class and single-class classi ica ion app oaches o ehicle model ecogni ion om images. In: P oceedings o AICS-05: I ish con e ence on a i icial in elligence and cogni i e science, pp 1–10 18. Omid a nia A, Wa en AE, Dalic LJ, Pede sen M, Jackson G (2021) Au oma ic de ec ion o gene alized pa oxysmal as ac i i y in in e ic al eeg using ime- equency analysis. Compu Biol Med. h ps://doi.o g/10.1016/j.compbiomed.2021.104287 19. Palshika GK (2009) Simple algo i hms o peak de ec ion in ime-se ies. Technical epo , Ta a Resea ch De elopmen and Design Cen e 20. Panayio opoulos CP (2010). A clinical guide o epilep ic syn- d omes and hei ea men . h ps://doi.o g/10.1007/978-1-84628- 644-5 21. Ped egosa F, Va oquaux G, G am o A, Michel V, Thi ion B, G isel O, Blondel M, P e enho e P, Weiss R, Dubou g V, Vande plas J, Passos A, Cou napeau D, B uche M, Pe o M, Duchesnay E (2011) Sciki -lea n: machine lea ning in Py hon. J Mach Lea n Res 12:2825–2830 22. Ra ho e C, P akash S, Makwana P (2020) P e alence o pho- opa oxysmal esponse in pa ien s wi h epilepsy: e ec o he unde lying synd ome and ea men s a us. Seizu e. h ps://doi. o g/10.1016/j.seizu e.2020.09.006 23. Rubboli G, Pa a J, Se i S, Takahashi T, Thomas P (2004) Eeg diagnos ic p ocedu es and special in es iga ions in he assess- men o pho osensi i i y. Epilepsia 45(5):35–39. h ps://doi.o g/ 10.1111/j.0013-9580.2004.451002.x Neu al Compu ing and Applica ions 123 24. So iano MC, Niso G, Clemen s J, O ı ´n S, Ca asco S, Gudı ´nM, Mi asso CR, Pe eda E (2017) Au oma ed de ec ion o epilep ic bioma ke s in es ing-s a e in e ic al meg da a. F on Neu oin- o m. h ps://doi.o g/10.3389/ nin .2017.00043 25. S iga o G, Go i B, Va asi C, Flee wood T, Can ello G, Can ello R (2021) Flash-e oked high- equency eeg oscilla ions in pho- osensi i e epilepsies. Epilep Res. h ps://doi.o g/10.1016/j.eplep sy es.2021.106597 26. T eni e DKN (2019) Pho osensi i i y and epilepsy. In: Clinical elec oencephalog aphy, pp 487–495. Sp inge 27. T eni e ´DKN, Rubboli G, Hi sch E, Ma ins Da Sil a A, Se i S, Wilkins A, Pa a J, Co anis A, Elia M, Capo illa G, S ephani U, Ha ding G (2012) Me hodology o pho ic s imula ion e isi ed: upda ed Eu opean algo i hm o isual s imula ion in he eeg labo a o y. Epilepsia. h ps://doi.o g/10.1111/j.1528-1167.2011. 03319.x 28. U ongene C, A ache RE, Loddenkempe T, Meisel C (2020) Elec oca diog aphic changes associa ed wi h epilepsy beyond hea a e and hei u iliza ion in u u e seizu e de ec ion and o ecas ing me hods. Clin Neu ophysiol. h ps://doi.o g/10.1016/ j.clinph.2020.01.007 29. Vailshe y LS (2021) A / headse shipmen s wo ldwide 2020–2025. h ps://www.s a is a.com/s a is ics/653390/wo ld wide- i ual-and-augmen ed- eali y-headse -shipmen s/ 30. Vanabelle P, Handschu e PD, Tah y RE, Benjelloun M, Boukhebouze M (2020) Epilep ic seizu e de ec ion using eeg signals and ex eme g adien boos ing. J Biomed Res. h ps://doi. o g/10.7555/JBR.33.20190016 31. Wal z S, Ch is en HJ, Doose H (1992) The di e en pa e ns o he pho opa oxysmal esponse—a gene ic s udy. Elec oen- cephalog Clin Neu ophysiol 83(2):138–145. h ps://doi.o g/10. 1016/0013-4694(92)90027-F.h ps://www.sciencedi ec .com/sci ence/a icle/pii/001346949290027F 32. Wang L, Long X, A ends JB, Aa s RM (2017) Eeg analysis o seizu e pa e ns using isibili y g aphs o de ec ion o gene al- ized seizu es. J Neu osci Me hod. h ps://doi.o g/10.1016/j.jneu me h.2017.07.013 33. Wang S, Yang J, Chen N, Chen X, Zhang Q (2005) Human ac i i y ecogni ion wi h use - ee accele ome e s in he senso ne wo ks. In: P oceedings o in e na ional con e ence on neu al ne wo ks and b ain ICNN&B’05, pp 1212–1217 34. Wol P, Goosses R (1986) Rela ion o pho osensi i i y o epilep ic synd omes. J Neu ol Neu osu g Psych. h ps://doi.o g/ 10.1136/jnnp.49.12.1386 35. Yang Y, T uong N, Mahe C, Ka ehei O, T uong ND, Esh aghian JK, Nikpou A (2021) A mul imodal ai sys em o ou -o -dis i- bu ion gene aliza ion o seizu e de ec ion. bioXRi . h ps://doi. o g/10.1101/2021.07.02.450974 36. Zib and sen IC, Kidmose P, Kjae TW (2018) De ec ion o gene alized onic-clonic seizu es om ea -eeg based on emg analysis. Seizu e. h ps://doi.o g/10.1016/j.seizu e.2018.05.001 Publishe ’s No e Sp inge Na u e emains neu al wi h ega d o ju isdic ional claims in published maps and ins i u ional a ilia ions. Neu al Compu ing and Applica ions 123