S.I.: COMPUTATIONAL-BASED BIOMARKERS FOR MENTAL AND EMOTIONAL
HEALTH(CBMEH2021)
Vi ual eali y and machine lea ning in he au oma ic
pho opa oxysmal esponse de ec ion
Fe nando Moncada
1
•So ı
´a Ma ı
´n
2
•Vı
´c o M. Gonza
´lez
1
•Vı
´c o M. A
´l a ez
2
•Bea iz Ga cı
´a-Lo
´pez
3
•
Ana Isabel Go
´mez-Mene
´ndez
3
•Jose
´R. Villa
2
Recei ed: 9 No embe 2021 / Accep ed: 4 Janua y 2022
The Au ho (s) 2022
Abs ac
Pho osensi i i y, in ela ion o epilepsy, is a gene ically de e mined condi ion in which pa ien s ha e epilep ic seizu es o
di e en se e i y p o oked by isual s imuli. I can be diagnosed by de ec ing epilep i o m discha ges in hei elec-
oencephalog am (EEG), known as pho opa oxysmal esponses (PPR). The mos accep ed PPR de ec ion me hod—a
manual me hod—conside ed as he s anda d one, consis s in submi ing he subjec o in e mi en pho ic s imula ion (IPS),
i.e. a lashing ligh s imula ion a inc easing and dec easing licke ing equencies in a hospi al oom unde con olled
ambien condi ions, while a he same ime eco ding he /his b ain esponse by means o EEG signals. This esea ch
ocuses on in oducing i ual eali y (VR) in his con ex , adding, o he con en ional in as uc u e a mo e lexible one
ha can be p og ammed and ha will allow de eloping a much wide and iche se o expe imen s in o de o de ec
neu ological illnesses, and o s udy subjec s’ beha iou s au oma ically. The loop includes he subjec , he VR de ice, he
EEG in as uc u e and a compu e o analyse and moni o he EEG signal and, in some cases, p o ide eedback o he VR.
As will be shown, AI modelling will be needed in he au oma ic de ec ion o PPR, bu i would also be used in ex ending
he unc ionali y o his sys em wi h mo e ad anced ea u es. This sys em is cu en ly in s udy wi h subjec s a Bu gos
Uni e si y Hospi al, Spain.
Keywo ds Elec oence alog am Vi ual eali y Pho opa oxysmal esponse Machine lea ning
&Vı
´c o M. Gonza
´lez
[email p o ec ed]
Fe nando Moncada
[email p o ec ed]
So ı
´a Ma ı
´n
[email p o ec ed]
Vı
´c o M. A
´l a ez
[email p o ec ed]
Bea iz Ga cı
´a-Lo
´pez
[email p o ec ed]
Ana Isabel Go
´mez-Mene
´ndez
[email p o ec ed]
Jose
´R. Villa
[email p o ec ed]
1
Elec ical Enginee ing Depa men , Uni e si y o O iedo,
As u ias, Spain
2
Compu e Science Depa men , Uni e si y o O iedo,
As u ias, Spain
3
Neu ophysiology Depa men , Bu gos Uni e si y Hospi al,
Bu gos, Spain
123
Neu al Compu ing and Applica ions
h ps://doi.o g/10.1007/s00521-022-06940-z(0123456789().,- olV)(0123456789().,- olV)
1 In oduc ion
Vi ual eali y (VR) is inc easingly becoming pa o ou
daily li es and i s applicabili y is widening [29]: om
ideo games and en e aining, o educa ion, medical
ea men and ehabili a ion, mili a y applica ions, a chi-
ec u al and u ban design, digi al ma ke ing and ac i ism,
enginee ing and obo ics, ine a s, he i age and a chaeol-
ogy, occupa ional sa e y, social sciences, psychology and
many mo e. Fo his eason, i is no only impo an o
unde s and how his echnology can a ec ou b ains, bu
also i s po en ial o ex end he cu en scope o neu ological
diseases de ec ion me hods. Using VR in new speci ic
igge ing scena ios will allow con olling and ex ending
he a ailable unc ionali ies o he isual s imula ion sys-
ems used o he de ec ion o such pa hologies [4].
This e lec ion is no new. Back in he 1990s, esea ch
pe o med o de ec he isk o ele ision and ideo games
exposu e esul ed in a se o ecommenda ions o TV
manu ac u e s and ideo games de elope s. Simul ane-
ously, he s udies pe o med we e he basis o many new
uses o hese echniques in educa ion [9] o in ehabili a ion
[8], o name a ew.
Pho osensi i i y is an abno mal isual sensi i i y o he
b ain esul ing in a pho opa oxysmal esponse (PPR), i.e. a
b ain epilep ic discha ge consis ing o a co ical spike o a
spike-and-wa e p o oked by a lash o a isual s imuli
[26]. The e a e ou di e en ypes o PPR esul ing om
di e en b ain esponses o in e mi en ligh s imula ion
[31]. E en hough PPR can be ound in non-epilep ic
elec oencephalog am (EEG) eco dings, i is s ongly
associa ed wi h epilepsy [16]. The ele ance o he PPR
elies on i s associa ion wi h speci ic epilep ic synd omes
[34] and moni o ing o ea men in a clinical con ex [20].
The pho osensi i i y ange is ela ed o he likelihood o
occu ence o e lex seizu es in daily li e; hus, i is o
c ucial in e es o ea ly de ec PPR. The mos commonly
used p ocedu e o de ec PPR, known as in e mi en pho ic
s imula ion (IPS), is desc ibed in [23]. I p oposes o sub-
mi he subjec o a se ies o ligh lashes while simul a-
neously moni o ing he b ain ac i i y using EEG signals,
acco ding o he Eu opean consensus g oup me hodology
o isual s imula ion de ined in 2012 [27]. The ligh la-
shes equency is i s inc eased om a minimum o a
maximum o un il a PPR is obse ed (wha e e happen
i s ). I no PPR is obse ed, he p ocess inishes. Should a
PPR occu , he p ocedu e is epea ed in an in e se manne ,
i.e. s a ing wi h a op licke ing equency ha is g adually
dec eased un il a minimum is eached o a PPR happen.
The aim is o de ec he minimum and he maximum e-
quencies a which he subjec shows PPR, i any, wi h he
minimal exposu e.
The de ec ion o PPR is usually pe o med by physi-
cians, i.e. clinical neu ophysiologis s and nu ses, who
manually e iew he EEG signals’ a iabili y in sea ch o
PPR [2,14,22], aking in o accoun each subjec ’s clinical
con ex such as age, seizu e and amily his o y. To ou bes
knowledge, no au oma ed me hod o he de ec ion o PPR
has been de eloped so a . This esea ch is mainly ocused
on he design o a new and sa e p ocedu e o he au oma ic
de ec ion o PPR wi hin EEG signals using digi al bio-
ma ke s implemen ed using VR and AI echniques.
In his sense, [25] designed a PPR de ec ion me hod by
analysing he po en ial and oscilla ion o he esponse
p o oked by a lashing s imula ion, bu ollowing a di -
e en s imula ion pa e n om he s anda d one. The e a e
o he ecen s udies ha analyse he pho osensi i i y and
epilepsy based on o he gene alized discha ges o seizu es
han PPRs: in [18], a de ec ion me hod based on he band
ampli ude luc ua ion compu ed om a high- equency and
a low- equency componen s o he EEG windows in each
EEG channel is p oposed; [30] applied he ex eme g adi-
en boos echnique o he classi ica ion o seizu es in wo
di e en ways (applying a s anda d pa i ioning o he da a
and applying a lea e-one-ou c oss- alida ion scheme),
while a channel-independen long sho - e m memo y
ne wo k is used in [5]; he in o ma ion ex ac ed om EEG
and elec oca diog am (ECG) signals is used in [35]ina
mul i-modal neu al ne wo k which analyse he da a in h ee
di e en ways (only EEG da a wi h a con olu ional LSTM
ne wo k; only ECG da a wi h a esidual con olu ional
ne wo k; and a used ne wo k which combines he ou pu s
o he indi idual ne wo ks o pe o m he inal classi ica-
ion); in [6], K-nea es neighbou s and a i icial neu al
ne wo ks a e used o he de ec ion o ic al discha ges and
in e -ic al s a es; [32] p oposed an EEG single-channel
analysis applying h ee ypes o isibili y g aphs (basic,
ho izon al and di e ence) o ep esen di e en EEG
pa e ns.
O he s udies make use o addi ional and di e en bio-
me ic measu es o he same pu pose, such us elec o-
ca diog ams (ECG) [10,11,28], elec omyog ams (EMG)
[3,36] o magne oence alog ams (MEG) [24].
This s udy p oposes an al e na i e o he con en ional
IPS p ocedu e o PPR de ec ion using VR and machine
lea ning (ML). This esea ch is ocused on he mos e-
quen PPR ype; hus, he PPR de ec ion s ill needs mo e
esea ch wo k as i is no comple ely sol ed. Howe e ,
in oducing VR would e en ually allow o s udy and o
de elop new and sa e p ocedu es o PPR de ec ion. Since
he VR in as uc u e is much mo e lexible han he
s anda d one, i can be easily con igu ed o ca y ou new
assays and s imula ion pa adigms, allowing o an
ad anced IPS/Visual s imula ion sys em. Ou p oposal
includes a VR de ice wi h a wi eless connec ion o a
Neu al Compu ing and Applica ions
123
compu e ha has access o he da a ga he ed by he EEG
senso s. The subjec mus wea a he same ime bo h he
EEG cap and a head moun ed display (HDM). Fu he -
mo e, a plausible solu ion o he au oma ic PPR de ec ion
is p oposed using some ea u es ex ac ed om he EEG
signals in an a e age mon age, and classic ML echniques.
An a e age mon age is used so ha he elec onega i e
PPR discha ge will exp ess wi h an upwa d de lec ion o
he EEG signal in he a ec ed channel.
The main con ibu ions o his esea ch a e:
– To in oduce VR in a close loop wi h AI and ML
models, so medical p ocedu es could be e isi ed and
enhanced. This con ibu ion can lead in he nea u u e
o mo e ad anced diagnose es s and p ocedu es.
– To p o ide he neu ophysiology depa men a Bu gos
Uni e si y Hospi al wi h a no el ins umen o analyse
he impac o VR in ela ion o pho osensi i i y by
in eg a ing his solu ion in i s daily wo k.
– To de elop ML models o de ec anomalies in he EEG
eco dings when he pa ien is lashed using ei he VR-
ML IPS o con en ional IPS.
The s uc u e o his s udy is as ollows. The nex sec ion
ocuses on he desc ip ion o he p oposal, de ailing he
di e en elemen s in he loop. Sec ion 3gi es de ails o he
expe imen a ion se -up ha has been ca ied ou a he
p oposal, while Sec . 4includes all he ob ained esul s and
he discussion on hem. The inal sec ion d aws he con-
clusion o his esea ch.
2 A p o o ype o VR-ML IPS p ocedu e
The p oposed solu ion complemen s he con en ional se -
up by means o in oducing a VR de ice ha he subjec
mus wea along wi h an EEG cap (see Fig. 1). The signals
om he EEG senso s a e analysed using well-known ML
echniques. The in elligen module will e en ually con ol
he VR con en s o gain inc eased capabili ies and o pe -
o m mo e complex assays. This sec ion gi es de ails on
each o he main modules: he VR pa (nex subsec ion)
and he ML module (Sec . 2.2).
2.1 VR design o IPS and PPR de ec ion
Flashing ligh s a e one o he main igge s o pho osen-
si i e esponses. VR-Pho osense [15] is a so wa e
designed o de ec pho ic-d i ing and PPR while using VR
and wea ing a head-moun ed display (HMD). VR-Pho o-
sense o e s a VR scena io wi h IPS in o de o measu e
b ain esponses o lashing ligh s a a ious equencies and
using di e en sequences. The main goal o his so wa e is
o simula e con en ional IPS es s in a i ual eali y
en i onmen .
Con en ional IPS places he ligh s imula o a a e y
sho dis ance om he pa ien ’s eyes, c ea ing high
exposu e o licke ing ligh s which a e pe cei ed wi h
in ensi y e en when he pa ien ’s eyes a e closed. In o de
o emula e he exposu e and senso y e ec caused by he
con en ional IPS ligh s imula o , a i ual eali y scene
has been designed as a 3D enclosing sphe ical dome
en i onmen wi h he pa ien ’s ision placed a i s cen e
(see Fig. 4). This design supp esses pa ien ’s pe iphe al
ision and inc eases he ocus on he isual s imuli e en
wi h he eyes closed.
Fig. 1 To he le , he
con en ional se o IPS
p ocedu e. To he igh , he new
VR-ML se o IPS p ocedu e,
including au oma ic EEG
analysis and PPR de ec ion
Neu al Compu ing and Applica ions
123
The VR-Pho osense’s se -up is ai ly simple. A e
downloading and s a ing he app, he ca dboa d iewe ,
in o which a sma phone is inse ed, is secu ed o he
subjec ’s head using he adap able s aps, lea ing he uppe
pa o he head clea . Then, he EEG cap is easily se up on
he subjec ’s head co e ing all necessa y poin s o con ac
as shown in Fig. 2.
The so wa e sys em is di ided in o wo pa s: he ligh
s imula ion and he moni o ing so wa e. The VR-Pho o-
sense’s de aul con igu a ion s imula es wi h whi e ligh
combined wi h da k black backg ound, emula ing he
con en ional IPS. In oducing an inno a ion o con en-
ional IPS, VR-Pho osene allows he colou o he licke -
ing ligh and backg ound o be changed om he de aul
con igu a ion, so we designed wo colou ed se ings in
addi ion o he whi e one: i) one wi h b igh ed lashes and
deep blue backg ound; ii) and ano he wi h deep blue la-
shes wi h da k black backg ound. These ones may in lu-
ence he b ain discha ges, and combined may be mo e o
less p o oca i e when compa ed o he de aul one,
allowing o s udy b ain beha iou eac ing o he s imula-
ion wi h bo h o hese scena ios.
VR-Pho osense has also been designed o esemble he
con en ional s imula ion se -up and o acili a e he wo k
o physicians, bo h clinical neu ophysiologis s and nu ses,
while conduc ing pho osensi i i y es s a he hospi al. This
sys em includes a moni o ing ea u e ha allows o obse e
in eal ime wha is happening on he VR s imula ion ia a
web se e . This is hanks o he use o Websocke s, a
communica ions p o ocol ha o e s ull-duplex commu-
nica ion channels o e a single TCP connec ion making i
as e and wi h low la ency o upda e sys em.
This moni o ing ea u e along wi h EEG eco dings
ansla es in o a ull co e age IPS scena io ha closely
esembles he con en ional se -up, wi h he di e ence o
eplacing he adi ional s imula ion de ice wi h a low cos
VR headse and he VR-Pho osense so wa e. The EEG se -
up o ca y ou VR-Pho osense es ing a he hospi al
consis s o using Na us B ain moni o ing and Neu owo ks
so wa e o EEG eco ding. Fu he mo e, he di e en
ha dwa e and so wa e elemen s can be easily mixed and
eplaced as hey a e comple ely independen om each
o he . Fo example, he VR-Pho osense can be con igu ed
o wo k wi h highe end HDMs such as Oculus; o EEG
eco ding can be pe o med using a di e en de ice such
us OpenBCI 3D p in ed de ice and open sou ce so wa e,
which we e used a he uni e si y EEG labo a o y o
p elimina y expe imen s.
All in all, VR-Pho osense o e s an inno a i e, low-cos
and c oss-pla o m IPS scena io in i ual eali y, wi h
mo e upcoming ea u es o be included aimed o achie e a
mo e de ailed diagnosis.
This ligh s imula ion so wa e has been implemen ed in
Uni y 3D using he p og amming language C#. Uni y is a
ideo game de elopmen engine ha allows designing 3D
scenes by means o a isual edi o and he p og amming o
gameplay e en s ia sc ip ing. These sc ip s a e associa ed
wi h he game objec s included in he scene so ha hey
beha e in he desi ed way.
Wi hin he scene, he asse s included a e s uc u ed in
so wa e componen s as shown in Fig. 3.
– Con ols: au o-gene a ed sc ip and inpu sys em. The
new Inpu Sys em 1.0.2 o Uni y has been used, which
allows o se up he desi ed inpu s h ough an in e ace.
Di e en inpu s can be en e ed om di e en de ices
o he same ac ion. This makes con igu a ion and
connec i i y wi h di e en ypes o inpu ha dwa e,
such as keyboa ds and game con olle s, seamless.
Fig. 2 VR-Pho osense se -up wi h a ca dboa d iewe and an
OpenBCI EEG headse Fig. 3 VR-Pho osense so wa e package diag am
Neu al Compu ing and Applica ions
123
– GoogleVR: an SDK o And oid ha allows he
c ea ion o i ual eali y applica ions o be used wi h
Google Ca dboa d HMD.
– Plugins: se o And oid plugins needed o expo
applica ions o de ices using his ope a ing sys em.
– Resou ces: all he sc ip s de eloped o ligh ing and
connec ion wi h he moni o ing side. This package also
includes shade s and ma e ials used in he scene.
– Scenes: he designed i ual eali y scene (Fig. 4).
– XR: de aul Uni y con igu a ion o ex ended ( i ual
and augmen ed) eali y apps.
The de eloped scene has ou objec s o GameObjec s:
he came a, wo poin ligh s and a sphe e (see Fig. 4). The
pu pose o he sphe e is o p o ide a black backg ound,
placing he came a inside i and in e ing he no mals, hus
a oiding anspa ency. This in e sion p ocess is done by
adding a cus om shade o a new ma e ial assigning i o he
sphe e. As o he spo ligh s, one o hem is he main whi e
ligh and he o he p o ides a blue backg ound o he
cus om colou unc ionali y.
All de eloped sc ip s use he MonoBeha iou buil -in
class, since Uni y uses a s anda d Mono un- ime imple-
men a ion. These classes a e conside ed bluep in s and
each ime hey a e associa ed wi h a GameObjec , a new
ins ance o he objec de ined by ha bluep in is c ea ed.
In hem, wo me hods a e p ede ined: S a and Upda e. In
his case, only S a is used, which will be called by Uni y
when loading he scene. I is also impo an o no e he use
o co u ines. Co u ines a e unc ions ha allow pausing
and esuming he execu ion in he ame in which i was
paused. In his case hey a e used o implemen he lick-
e ing e ec in he ligh s and he s imula ion sequences.
As o he inpu sys em, wo op ions we e conside ed:
emo e con ol and keyboa d. Conside ing he numbe o
con igu able pa ame e s and he con olled ac ions o be
pe o med by he pe son in cha ge o he s imula ion, he
use o only a VR emo e con ol was conside ed insu i-
cien . Bea ing in mind he end use and he echnology hey
use on a daily basis, he use o a Blue oo h keyboa d was
chosen. As a esul , mo e con igu a ions a e a ailable and
he commands a e se in he way hese expe s conside
mo e com o able o pe o m he s imula ion in he mos
e icien possible manne . Howe e , in o de o enable he
use o he s imula ion wi h he Oculus Ques 2 glasses, his
con igu a ion has also been adap ed o he wo emo e
con ols associa ed wi h his HMD. Finally, he lis o
commands a ailable is shown in Table 1.
2.1.1 Moni o ing
In addi ion o simula ing he IPS en i onmen , i is deemed
necessa y o know wha is happening inside he scene
du ing he es s. To add ess his p oblem, a moni o ing
componen —a web page (h ps:// pho osense.he okuapp.
com)—was added o he sys em. This solu ion allows he
physicis s o see in eal ime wha is happening in he VR
s imula ion, e.g. he e olu ion o he sequences and he
commands en e ed by he use .
Ano he possibili y was o implemen a sound sys em o
communica e he si ua ion h ough audio, bu i was con-
side ed less e icien as i was ola ile and did no ha e a
isual eco d o he simula ion.
To implemen he moni o ing sys em, a WebSoke s
componen was used, which allows eal- ime and as
communica ion be ween he sma phone applica ion and
he web page a oiding he use o a da abase. The desi ed
unc ionali y is as shown in Fig. 5. Clien 1, he sma -
phone applica ion, sends in o ma ion abou he s imula ion
o he se e , and he se e passes i o clien 2, he web
page, which displays i on he sc een along wi h a ime
s amp.
WebSocke s is a p o ocol ha p o ides bidi ec ional,
ull duplex communica ion o e a single TCP socke . The
en i onmen in which he VR-Pho osense applica ion is
used is a ai ly sensi i e one o medical es ing, hus
imposing an as as as possible exchange o in o ma ion
exigence. WebSocke s is p obably he mos popula p o-
ocol o his ype o use cases whe e da a needs o be sen
in eal ime. As we ha e seen be o e, Uni y is based on he
Mono pla o m as sc ip ing engine, which means ha i
wo ks in .NET (C#). This amewo k p o ides a de aul
suppo o his p o ocol ha is also suppo ed by Mono,
Fig. 4 Scene diag am implemen ed in Uni y 3D
Neu al Compu ing and Applica ions
123

h ough he Sys em.Ne .WebSocke s namespace. The
ope a ion in his case would be as shown in Fig. 6.
The se e has been de eloped using Node. I is a simple
se e ha ecei es in o ma ion om he clien and e u ns
i . In his case, no hing is done in Uni y wi h he e u ned
Table 1 Lis o a ailable
commands in VR-pho osense
applica ion
A ailable commands
Ac ion Command (keyboa d) Command (Ques 2)
S a En e Righ igge
Pause Space ba Righ g ip
Rese Backspace Seconda y ouched (R)
Inc emen Hz Numpad ?P ima y ouched (R)
Whi e ligh B S a (L)
Blue ligh A Le g ip
Red-blue combina ion R Le igge
Upwa d pa e n Up a ow P ima y ouched (L)
Downwa d pa e n Down a ow Seconda y ouched (L)
Exi Escape S a (R)
Fig. 5 Theo e ical unc ionali y
o he moni o ing sys em
Fig. 6 Sequence diag am o he
sys em using WebSocke s
Neu al Compu ing and Applica ions
123
in o ma ion; i is he moni o ing clien ha makes use o i .
This clien is an HTML page ha connec s o he se e ,
opens a connec ion, ge s he in o ma ion coming om he
se e ha is pa o a message and displays i on he
sc een. This HTML clien has been implemen ed in he
simples possible way since i is an add-on o he main
wo k and i s appea ance in e ms o design is no ele an
o he EEG labo a o y s a .
2.2 ML-based PPR de ec ion
PPR can be ound in epilep ic synd omes ha p esen
seizu es wi h o wi hou isual s imulus igge , and in
some subjec s bo h ypes o seizu es a e obse ed. Wal z
classi ica ion [31] is used o de ine he exp ession o he
PPR om an elec oencephalog aphic poin o iew,
in oducing up o ou di e en ypes o PPR:
•Type-1: spikes wi hin he occipi al hy hm.
•Type-2: pa ie o-occipi al spikes wi h a biphasic slow
wa e.
•Type-3: pa ie o-occipi al spikes wi h a biphasic slow
wa e and sp ead o he on al egion.
•Type-4: gene alized spikes and wa es o polyspikes and
wa es.
All o hem a e depic ed in Fig. 7. Type-4 PPR is he mos
equen ly ound in epilep ic synd omes whe e pho osen-
si i i y cons i u es a clinical conce n, as i seems o ha e a
s ong associa ion—highe han 90%—wi h epilep ic sei-
zu es; he de ec ion o his ype o PPR ep esen s he
challenge ocused on his esea ch. Besides, in clinical
p ac ice, i is equen ha exp ession o PPR is a iable,
and in many imes we ob ain PPR ha no necessa ily i
wi hin only one ca ego y o hose ini ially de ined by
Wal z. E en mo e, he mo phological cha ac e is ics o a
gi en subjec ’s PPR may a y because o clinical a iables
as doses o an i-epilep ic ea men , sleep quali y, e c.
To iden i y Type-4 PPR we p opose he use o sliding
windows—one second leng h and a en h o a second
shi — ollowed by a p e-p ocessing s ages ha sub ac s
he window a e age and pe o ms a ea u e ex ac ion.
Well-known ML echniques conside hese ea u es o
p opose a inal label o he window.
These ML echniques a e applied in h ee pa s ha will
be desc ibed in his sec ion. Fi s , he se o ans o ma ions
ha will be applied and hei a ionale a e explained in
Sec . 2.2.1. Then, he design and deploymen o he ML
models a e de ailed in Sec . 2.2.2 and, inally, he aining
o he ML pa is desc ibed in Sec . 2.2.3.
Fig. 7 The ou ypes o PPR:
a ype-1: spikes wi hin he
occipi al hy hm; b ype-2:
pa ie o-occipi al spikes wi h
biphasic slow wa e; c ype-3:
pa ie o-occipi al spikes wi h
biphasic slow wa e and sp ead
o he on al egion; d ype-4:
gene alized spikes and wa es
Neu al Compu ing and Applica ions
123
2.2.1 Fea u e ex ac ion
The selec ion o ma hema ical ans o ma ions o signals
ep esen a ion is, pe se, a p oblem ha needs o be ca e-
ully add essed. The main poin is o selec ea u es ha , in
conjunc ion, include, as a whole, he in o ma ion ha he
expe s use o make a decision. The e o e, all he possible
windows mus be analysed and signi ican di e ences mus
be s a ed be ween he anomalies and he no mal signal
s a e. Nowadays, he p oblem o ea u e ans o ma ion is
deal wi h deep lea ning (DL) and, mo e speci ically, wi h
au o-encode s; howe e , due o he lack o da a o ain he
ne wo ks, we le his issue o u u e esea ch.
Fo his s udy, we ocused on channels Fz and O2
because hese a e he channels we e PPRs mo e equen ly
appea ega dless hei ype. We pay a en ion o hei
no mal and abno mal beha iou o he conside ed PPR
ype. Some examples o he windows ha migh be aced
a e depic ed in Fig. 8. F om he analysis o he signals, he
ollowing ea u es se has been selec ed o ep esen each
EEG da a window, whe e wis he wid h o he window,
e e s he ime s amp o which he ea u e is compu ed, c
is he channel—ei he Fz o O2—and dc
is he EEG
cchannel’s signal wi h an a e age mon age— he PPR
exp esses wi h an upwa d de lec ion—a ime s amp :
•Cumula i e Fi s de i a i e, also known as he
in ensi y o he signal, compu ed as
CFDc
¼1
wPw1
i¼0jdc
þiþ1dc
þij=D. This ea u e has
been chosen because PPR p esen high a e o change in
he alue o he channel. D ep esen s he in e al
be ween consecu i e samples, which is kep cons an .
•Cumula i e Second de i a i e compu ed as
CSDc
¼1
wPw1
i¼0jdc
þiþ22dc
þiþ1þdc
þij=D2.This
ea u e has been selec ed because he e is also a high
a e o change in he i s de i a i e, bu no so high as
o a e ac s.
•Numbe o ele an peaks using he S1 measu emen
p oposed in [19] and compu ed as ollows. Equa ion 1
de ines he calcula ion o S1, whe e kis he p ede ined
numbe o samples and pis he cu en sample
imes amp o which we a e de e mining whe he i is
a peak o no . The S1 ans o ma ion ep esen s a
scaling o he TS, which makes he peak de ec ion
easie using a p ede ined h eshold a.
S1ðpÞ¼1
2max
p1
i¼pkðdc
pdc
iÞþmax
pþk
i¼pþ1ðdc
pdc
iÞ

ð1Þ
So, o each poin o which S1ðpÞcan be compu ed
wi hin he EEG sliding window we compu e S1ðpÞ;a
peak occu s in ime pi he alue Spis highe han aand
is he highes in i s 2kneighbou hood. In he o iginal
epo , all he pa ame e s (k,a) whe e ca e ully de e -
mined o each p oblem in o de o op imize he peak
de ec ion. In his esea ch, kis se o 10 and ais se o
h ee imes he s anda d de ia ion o he EEG channel
alues when no ac i i y is shown (uppe igh co ne in
Fig. 8).
•Sum o he absolu e alues, o measu e he a ea unde
he cu e o he EEG signal.
Fig. 8 Th ee EEG agmen s om di e en condi ions: he le -mos
and he cen e eco dings a e conside ed no mal condi ions, while he
eco ding a he igh shows a PPR. The eco dings include, om op
o bo om, signals om he F3-AVG, Fz-AVG, F4-AVG, O1-AVG
and O2-AVG channels, espec i ely, whe e AVG s ands o he
a e age o all eco ded elec odes
Neu al Compu ing and Applica ions
123
•Maximum di e ences, also known in some ields as
amoun o mo emen [1], which measu es he di e -
ences be ween he highes and he smalles alues—
expec ed o be a high alue. I is calcula ed as
MDc
¼jmaxi2½ ; þwðdc
iÞmini2½ ; þwðdc
iÞj.
•A e age Ene gy as p oposed in [33], as he sum o he
squa ed disc e e FFT componen s magni udes o he
signal.
All he ea u es a e s anda dized; gi en a da a se , he
a e age and he s anda d de ia ion a e compu ed and used
o ans o m he alues in o a no mal dis ibu ion wi h
mean 0.0 and s anda d de ia ion 1.0.
2.2.2 Designing and deploying he ML pa
The goal in his s age is o ob ain models able o label he
p e-p ocessed EEG signal windows as no mal o as PPR.
Due o he da a imbalance, he mos in e es ing app oach is
o use unsupe ised lea ning, so anomalies can be de ec ed.
Howe e , his migh gene a e oo many alse posi i e, so i
could be in e es ing o also de elop a complemen a y
supe ised lea ning solu ion. The e o e, o his esea ch
we p oposed o use, i s , unsupe ised lea ning o ob ain a
model ha signals hose anomalous windows and hen o
classi y he anomalous windows as PPR o no mal using a
supe ised app oach. The unsupe ised lea ning is speci ic
o a gi en subjec , while he supe ised lea ning is a
gene alised model.
Fo his s age, we will de elop models ollowing he
wo k low p oposed in Fig. 9. Da a om he EEG senso s is
windowed as explained be o e. Fo each window, he
a e age is calcula ed and sub ac ed; he ans o ma ions
om he p e ious subsec ion a e calcula ed a e wa ds. A
one-class classi ie , lea ned o he cu en subjec da a,
labels he window as no mal o no . In case a window is
labelled as an anomaly, hen he wo-class classi ie ,
lea ned om o he subjec s, labels he window as PPR o
no .
Fo he one-class classi ie , an unsupe ised one-class
k-nea es neighbou s model (1C-KNN) [12,17] is es ed:
his model has been selec ed o i s as aining and
e alua ion imes while s ill pe o ming su icien ly
accu a e.
Fo he wo-class classi ie we p opose K-nea es
neighbou (2C-KNN) due o he small numbe o ins ances
in he a ailable da a se .
Bo h classi ie s a e om he sciki -lea n lib a y o
Phy on [21].
2.2.3 T aining he ML pa
T aining he models has wo main s ages as can be seen in
Fig. 10: (i) aining he one-class model and (ii) aining
he wo-class model. A his momen we ha e wo collec-
ions o da a: (a) a collec ion o windows om he cu en
subjec (CPDa a), all labelled as no mal, and (b) a collec-
ion o windows om he his o ical eco ds (HRDa a), each
window wi h i s co esponding no mal o PPR label.
CPDa a is used in he one-class aining, while HRDa a is
used in he wo-class aining. The i s pa o he aining
is he 2C-KNN lea ning using he HRDa a; in case o
highly imbalance o he da a se , SMOTE will be used.
Di e en alues o he pa ame e K a e es ed o bo h
classi ie s o ind he bes pe o ming model.
When analysing he da a eco ded o he cu en sub-
jec , an inc emen al aining is p oposed. The idea is
epea ing he one-class classi ie aining un il a eal PPR
be de ec ed a a ce ain lashing equency. Tha is, in case
he equency o be es ed is inc eased o example om 4
o 6 Hz, i no PPR is de ec ed in his new s imula ion
equency, hen he 1C-KNN is ained including he win-
dows ga he ed om he i s equency ange (1–6 Hz), and
hen, he nex lashing equency is e alua ed (8 Hz) and
he p ocess is epea ed again un il a PPR is de ec ed. The
s imula ion equency a which he i s PPR is de ec ed is
he cu equency ( c). The p ocess is illus a ed in Fig. 11.
This p ocess has been desc ibed o he ollowing
lashing equency inc ease sequence—s anda d
Fig. 9 The wo k low o he designed app oach. The da a ga he ed
om he cu en subjec a e p e-p ocessed. When a da a window
comes om equencies smalle o equal o c ( he cu equency,
which is he s imula ion equency alue a which he i s PPR
appea s), he window is p ese ed o he aining o he one-class
models; o he wise, he window is labelled as no mal o as anomaly. In
his la e case, he wo-class classi ie labels he window as PPR o
no . Howe e , when no window is labelled as including a PPR o he
cu en equency, he windows o his equency a e also conside ed
and he one class model is e- ained
Neu al Compu ing and Applica ions
123
amoun o a ailable da a allows us o do so, such as au o-
encode s plus dense laye s o long sho - e m memo y
ne wo ks. All o hese imp o emen s ep esen s u u e
esea ch wo k.
Acknowledgemen s This esea ch has been unded by he Spanish
Minis y o Science and Inno a ion unde p ojec MINECO-
TIN2017-84804-R, PID2020-112726RB-I00 and he S a e Resea ch
Agency (AEI, Spain) unde g an ag eemen No RED2018-102312-T
(IA-Biomed). Addi ionally, by he Council o Gijo
´n h ough he
Uni e si y Ins i u e o Indus ial Technology o As u ias g an SV-21-
GIJON-1-19.
Funding Open Access unding p o ided hanks o he CRUE-CSIC
ag eemen wi h Sp inge Na u e.
Con lic o in e es The au ho s decla e ha hey ha e no con lic o
in e es .
Open Access This a icle is licensed unde a C ea i e Commons
A ibu ion 4.0 In e na ional License, which pe mi s use, sha ing,
adap a ion, dis ibu ion and ep oduc ion in any medium o o ma , as
long as you gi e app op ia e c edi o he o iginal au ho (s) and he
sou ce, p o ide a link o he C ea i e Commons licence, and indica e
i changes we e made. The images o o he hi d pa y ma e ial in his
a icle a e included in he a icle’s C ea i e Commons licence, unless
indica ed o he wise in a c edi line o he ma e ial. I ma e ial is no
included in he a icle’s C ea i e Commons licence and you in ended
use is no pe mi ed by s a u o y egula ion o exceeds he pe mi ed
use, you will need o ob ain pe mission di ec ly om he copy igh
holde . To iew a copy o his licence, isi h p://c ea i ecommons.
o g/licenses/by/4.0/.
Re e ences
1. A
´l a ez A, T i in
˜o G, Co do
´n O (2011) Body pos u e ecogni ion
by means o a gene ic uzzy ini e s a e machine. In: IEEE 5 h
in e na ional wo kshop on gene ic and e olu iona y uzzy sys-
ems (GEFs), pp 60–65
2. Beniczky S, Au lien H, F ancesche i S, da Sil a AM, Bisulli F,
Ben es C, Cana oglia L, Fe i L, K y
´sl D, Pe al a AR, Ra
´cz A,
C oss JH, A zimanoglou A (2020) In e a e ag eemen o clas-
si ica ion o pho opa oxysmal elec oencephalog aphic esponse.
Epilepsia. h ps://doi.o g/10.1111/epi.16655
3. Beniczky S, Con adsen I, Henning O, Fab icius M, Wol P (2018)
Au oma ed eal- ime de ec ion o onic-clonic seizu es using a
wea able EMG de ice. Neu ology. h ps://doi.o g/10.1212/WNL.
0000000000004893
4. B ooks AL, B ahman S, Kap alos B, Nakajima A, Tye man J,
Jain LC (2021) Recen ad ances in echnologies o inclusi e
well-being. Vi ual pa ien s, gami ica ion and simula ion. In el-
ligen sys ems e e ence lib a y se ies, 196, 1 edn. Sp inge
5. Chak aba i S, Swe apadma A, Pa naik PK (2021) A channel
independen gene alized seizu e de ec ion me hod o pedia ic
epilep ic seizu es. Compu Me hods P og ams Biomed. h ps://
doi.o g/10.1016/j.cmpb.2021.106335
6. Choubey H, Pandey A (2021) A combina ion o s a is ical
pa ame e s o he de ec ion o epilepsy and eeg classi ica ion
using ann and knn classi ie . Signal Image Video P ocess. h ps://
doi.o g/10.1007/s11760-020-01767-4
7. Cobb S (1947) Pho ic d i ing as a cause o clinical seizu es in
epilep ic pa ien s. A ch Neu ol Psych 58(1):70–71
8. Hall C, Vi an i A, Abbey K (2019) Impac o ele ision on
nu i ional in ake in communal dining oom se ings among hose
wi h acqui ed b ain inju y a pilo s udy. Nu i ion and die e ics.
J Die Aus alia 77(4):444–448. h ps://doi.o g/10.1111/1747-
0080.12526
9. Ho man BL, Ho man R, Wessel CB, Shensa A, Woods MS,
P imack BA (2018) Use o ic ional medical ele ision in heal h
sciences educa ion: a sys ema ic e iew. Ad Heal h Sci Educ
23(1):201–216. h ps://doi.o g/10.1007/s10459-017-9754-5
10. Jahanbekam A, Baumann J, Nass RD, Bauckhage C, Hill H,
Elge CE, Su ges R (2021) Pe o mance o ecg-based seizu e
de ec ion algo i hms s ongly depends on aining and es con-
di ions. Epilepsia Open. h ps://doi.o g/10.1002/epi4.12520
11. Jeppesen J, Fuglsang-F ede iksen A, Johansen P, Ch is ensen J,
Wu
¨s enhagen S, Tankisi H, Qe ama E, Hess A, Beniczky S
(2019) Seizu e de ec ion based on hea a e a iabili y using a
wea able elec oca diog aphy de ice. Epilepsia 60:2105–2113.
h ps://doi.o g/10.1111/epi.16343
12. Khan SS, Ahmad A (2018) Rela ionship be ween a ian s o one-
class nea es neighbou s and c ea ing hei accu a e ensembles.
IEEE T ans Knowl Da a Eng 30:1796–1809
13. Kiloh LG (2013) and McComas. Clinical elec oencephalog a-
phy. Bu e wo h-Heinemann, A.J., Ossel on, J.W
14. La sen PM, Wu
¨s enhagen S, Te ney D, Ga della E, Al ing J,
Au lien H, Beniczky S (2021) Pho opa oxysmal esponse and i s
cha ac e is ics in a la ge eeg da abase using he sco e sys em.
Clin Neu ophysiol. h ps://doi.o g/10.1016/j.clinph.2020.10.029
15. Ma ı
´nS,A
´l a ez V, Ga cı
´a-Lo
´pez B, Gonza
´lez VM, Vila JR
(2021) V -pho osense: a i ual eali y pho ic s imula ion in e -
ace o he s udy o pho osensi i i y. In: P oceedings o 16 h
in e na ional con e ence on so compu ing models in indus ial
and en i onmen al applica ions (SOCO 2021) so compu ing
models in indus ial and en i onmen al applica ions, pp 178–186.
Sp inge
16. Michelucci R, Pasini E, Riguzzi P, Ande mann E, Ka
¨l ia
¨inen R,
Gen on P (2016) Myoclonus and seizu es in p og essi e myo-
clonus epilepsies: pha macology and he apeu ic ials. Epilep
Diso de . h ps://doi.o g/10.1684/epd.2016.0861
17. Mun oe D, Madden MG (2005) Mul i-class and single-class
classi ica ion app oaches o ehicle model ecogni ion om
images. In: P oceedings o AICS-05: I ish con e ence on a i icial
in elligence and cogni i e science, pp 1–10
18. Omid a nia A, Wa en AE, Dalic LJ, Pede sen M, Jackson G
(2021) Au oma ic de ec ion o gene alized pa oxysmal as
ac i i y in in e ic al eeg using ime- equency analysis. Compu
Biol Med. h ps://doi.o g/10.1016/j.compbiomed.2021.104287
19. Palshika GK (2009) Simple algo i hms o peak de ec ion in
ime-se ies. Technical epo , Ta a Resea ch De elopmen and
Design Cen e
20. Panayio opoulos CP (2010). A clinical guide o epilep ic syn-
d omes and hei ea men . h ps://doi.o g/10.1007/978-1-84628-
644-5
21. Ped egosa F, Va oquaux G, G am o A, Michel V, Thi ion B,
G isel O, Blondel M, P e enho e P, Weiss R, Dubou g V,
Vande plas J, Passos A, Cou napeau D, B uche M, Pe o M,
Duchesnay E (2011) Sciki -lea n: machine lea ning in Py hon.
J Mach Lea n Res 12:2825–2830
22. Ra ho e C, P akash S, Makwana P (2020) P e alence o pho-
opa oxysmal esponse in pa ien s wi h epilepsy: e ec o he
unde lying synd ome and ea men s a us. Seizu e. h ps://doi.
o g/10.1016/j.seizu e.2020.09.006
23. Rubboli G, Pa a J, Se i S, Takahashi T, Thomas P (2004) Eeg
diagnos ic p ocedu es and special in es iga ions in he assess-
men o pho osensi i i y. Epilepsia 45(5):35–39. h ps://doi.o g/
10.1111/j.0013-9580.2004.451002.x
Neu al Compu ing and Applica ions
123

24. So iano MC, Niso G, Clemen s J, O ı
´n S, Ca asco S, Gudı
´nM,
Mi asso CR, Pe eda E (2017) Au oma ed de ec ion o epilep ic
bioma ke s in es ing-s a e in e ic al meg da a. F on Neu oin-
o m. h ps://doi.o g/10.3389/ nin .2017.00043
25. S iga o G, Go i B, Va asi C, Flee wood T, Can ello G, Can ello
R (2021) Flash-e oked high- equency eeg oscilla ions in pho-
osensi i e epilepsies. Epilep Res. h ps://doi.o g/10.1016/j.eplep
sy es.2021.106597
26. T eni e DKN (2019) Pho osensi i i y and epilepsy. In: Clinical
elec oencephalog aphy, pp 487–495. Sp inge
27. T eni e
´DKN, Rubboli G, Hi sch E, Ma ins Da Sil a A, Se i S,
Wilkins A, Pa a J, Co anis A, Elia M, Capo illa G, S ephani U,
Ha ding G (2012) Me hodology o pho ic s imula ion e isi ed:
upda ed Eu opean algo i hm o isual s imula ion in he eeg
labo a o y. Epilepsia. h ps://doi.o g/10.1111/j.1528-1167.2011.
03319.x
28. U ongene C, A ache RE, Loddenkempe T, Meisel C (2020)
Elec oca diog aphic changes associa ed wi h epilepsy beyond
hea a e and hei u iliza ion in u u e seizu e de ec ion and
o ecas ing me hods. Clin Neu ophysiol. h ps://doi.o g/10.1016/
j.clinph.2020.01.007
29. Vailshe y LS (2021) A / headse shipmen s wo ldwide
2020–2025. h ps://www.s a is a.com/s a is ics/653390/wo ld
wide- i ual-and-augmen ed- eali y-headse -shipmen s/
30. Vanabelle P, Handschu e PD, Tah y RE, Benjelloun M,
Boukhebouze M (2020) Epilep ic seizu e de ec ion using eeg
signals and ex eme g adien boos ing. J Biomed Res. h ps://doi.
o g/10.7555/JBR.33.20190016
31. Wal z S, Ch is en HJ, Doose H (1992) The di e en pa e ns o
he pho opa oxysmal esponse—a gene ic s udy. Elec oen-
cephalog Clin Neu ophysiol 83(2):138–145. h ps://doi.o g/10.
1016/0013-4694(92)90027-F.h ps://www.sciencedi ec .com/sci
ence/a icle/pii/001346949290027F
32. Wang L, Long X, A ends JB, Aa s RM (2017) Eeg analysis o
seizu e pa e ns using isibili y g aphs o de ec ion o gene al-
ized seizu es. J Neu osci Me hod. h ps://doi.o g/10.1016/j.jneu
me h.2017.07.013
33. Wang S, Yang J, Chen N, Chen X, Zhang Q (2005) Human
ac i i y ecogni ion wi h use - ee accele ome e s in he senso
ne wo ks. In: P oceedings o in e na ional con e ence on neu al
ne wo ks and b ain ICNN&B’05, pp 1212–1217
34. Wol P, Goosses R (1986) Rela ion o pho osensi i i y o
epilep ic synd omes. J Neu ol Neu osu g Psych. h ps://doi.o g/
10.1136/jnnp.49.12.1386
35. Yang Y, T uong N, Mahe C, Ka ehei O, T uong ND, Esh aghian
JK, Nikpou A (2021) A mul imodal ai sys em o ou -o -dis i-
bu ion gene aliza ion o seizu e de ec ion. bioXRi . h ps://doi.
o g/10.1101/2021.07.02.450974
36. Zib and sen IC, Kidmose P, Kjae TW (2018) De ec ion o
gene alized onic-clonic seizu es om ea -eeg based on emg
analysis. Seizu e. h ps://doi.o g/10.1016/j.seizu e.2018.05.001
Publishe ’s No e Sp inge Na u e emains neu al wi h ega d o
ju isdic ional claims in published maps and ins i u ional a ilia ions.
Neu al Compu ing and Applica ions
123